U "quora-duplicates-dev_average_precision" print(results[paraphrase_mining_evaluator.primary_metric]) # => 0.5650940787776353 NF順 dT) sentences_mapduplicates_listduplicates_dictadd_transitive_closurequery_chunk_sizecorpus_chunk_size max_pairstop_kshow_progress_bar batch_sizename write_csv truncate_dimcstg|_g|_|D] \}}|j||j|q| |_| |_| |_||_ ||_ ||_ ||_ | |_ |dk r||n tdd|_|dk r|D]4\}}||kr||krd|j||<d|j||<q|r||j|_t}|jD]X}|j|D]H}||kr||kr|j||s*|j||r|tt||gqqt||_| r\d| } d| d|_dd d d d d dg|_| |_d|_dS)a Initializes the ParaphraseMiningEvaluator. Args: sentences_map (Dict[str, str]): A dictionary that maps sentence-ids to sentences. For example, sentences_map[id] => sentence. duplicates_list (List[Tuple[str, str]], optional): A list with id pairs [(id1, id2), (id1, id5)] that identifies the duplicates / paraphrases in the sentences_map. Defaults to None. duplicates_dict (Dict[str, Dict[str, bool]], optional): A default dictionary mapping [id1][id2] to true if id1 and id2 are duplicates. Must be symmetric, i.e., if [id1][id2] => True, then [id2][id1] => True. Defaults to None. add_transitive_closure (bool, optional): If true, it adds a transitive closure, i.e. if dup[a][b] and dup[b][c], then dup[a][c]. Defaults to False. query_chunk_size (int, optional): To identify the paraphrases, the cosine-similarity between all sentence-pairs will be computed. As this might require a lot of memory, we perform a batched computation. query_chunk_size sentences will be compared against up to corpus_chunk_size sentences. In the default setting, 5000 sentences will be grouped together and compared up-to against 100k other sentences. Defaults to 5000. corpus_chunk_size (int, optional): The corpus will be batched, to reduce the memory requirement. Defaults to 100000. max_pairs (int, optional): We will only extract up to max_pairs potential paraphrase candidates. Defaults to 500000. top_k (int, optional): For each query, we extract the top_k most similar pairs and add it to a sorted list. I.e., for one sentence we cannot find more than top_k paraphrases. Defaults to 100. show_progress_bar (bool, optional): Output a progress bar. Defaults to False. batch_size (int, optional): Batch size for computing sentence embeddings. Defaults to 16. name (str, optional): Name of the experiment. Defaults to "". write_csv (bool, optional): Write results to CSV file. Defaults to True. truncate_dim (Optional[int], optional): The dimension to truncate sentence embeddings to. `None` uses the model's current truncation dimension. Defaults to None. NcSsttS)N)rboolr!r!^/tmp/pip-unpacked-wheel-i7fohqg6/sentence_transformers/evaluation/ParaphraseMiningEvaluator.pyz4ParaphraseMiningEvaluator.__init__..T_Zparaphrase_mining_evaluationz _results.csvepochsteps precisionrecallf1 thresholdaverage_precision)super__init__ sentencesidsitemsappendrrrrrrrrr duplicatesrsetaddtuplesortedlentotal_num_duplicatescsv_file csv_headersrZprimary_metric)selfrrrrrrrrrrrrridZsentenceid1id2Zpositive_key_pairskey1key2 __class__r!r"r.?sR/       z"ParaphraseMiningEvaluator.__init__r )model output_pathr&r'returnc Cs |dkr0|dkrd|}q4d|d|d}nd}|jdk rP|d|jd7}td |jd |d |jdkrztn ||j*t||j|j|j |j |j |j |j }W5QRXtd tt|d }}d } d } } } d } tt|D]}||\}}}|j|}|j|}|d7}|j||s@|j||r|d7}||}||j}d||||}| |7} || kr|} |} |} ||d |t|dt|dd d} q| |j} td| dtd| td| dtd| dtd| d|dk r|jrtj||j}tj|st|dddd4}t|}||j |||| | | | | gW5QRXn>t|dddd(}t|}|||| | | | | gW5QRX| | | | | d}|!||j}|"|||S)NrDz after epoch z in epoch z after z stepsrz (truncated to )z1Paraphrase Mining Evaluation of the model on the z dataset:zNumber of candidate pairs: rzAverage Precision: {:.2f}rzOptimal threshold: {:.4f}zPrecision: {:.2f}zRecall: {:.2f}z F1: {:.2f} wzutf-8)newlinemodeencodinga)r,r*r(r)r+)#rloggerinforrZtruncate_sentence_embeddingsr r/rrrrrrstrr8ranger0r3r9minformatrospathjoinr:isfileopencsvwriterwriterowr;Zprefix_name_to_metricsZ store_metrics_in_model_card_data)r<rErFr&r'Zout_txtZ pairs_listZ n_extractZ n_correctr+Zbest_f1Z best_recallZbest_precisionr,idxZscoreijr>r?r(r)r*Zcsv_pathfr]Zmetricsr!r!r"__call__s       0   $ " z"ParaphraseMiningEvaluator.__call__cCst}t|D]}||krt}||t||}t|dkrr|d}||kr:|||||q:t|}tt|dD]`}t|dt|D]H}d|||||<d|||||<||||||qqq|S)NrrJT)r4listkeysr5r8popextendrT)graphZ nodes_visitedrPZconnected_subgraph_nodesZneighbor_nodes_queuenoder`rar!r!r"rs&     z0ParaphraseMiningEvaluator.add_transitive_closure) NNFr rrrFrrTN)NrDrD)__name__ __module__ __qualname____doc__rrSrrr intrr.floatrc staticmethodr __classcell__r!r!rBr"r sP0 ` Wr )r\loggingrW collectionsr contextlibrtypingrrrrrZ2sentence_transformers.evaluation.SentenceEvaluatorr Zsentence_transformers.utilr Z)sentence_transformers.SentenceTransformerr getLoggerrjrQr r!r!r!r"s