U Msz/RerankingEvaluator.__init__..r _rz _results_@z.csvepochstepsZMAPzMRR@{}zNDCG@{}map)super__init__samplesrloggerwarningrrrrrr isinstancedictlistvaluescsv_fileformat csv_headersrZprimary_metric) selfr)rrrrrrrrr __class__r r!r(,s4  $  zRerankingEvaluator.__init__r )model output_pathr$r%returnc Cs|dkr0|dkrd|}q4d|d|d}nd}|jdk rP|d|jd 7}td |jd |d ||}|d }|d}|d} dd|jD} dd|jD} tdt|jt | t | t | t | t | t | td|dtd|j |dtd|j | d|dk r|j rtj||j} tj| } t| d| rtdnddd6}t|}| s||j|||||| gW5QRXd |d|j |d|j | i}|||j}||||S)a Evaluates the model on the dataset and returns the evaluation metrics. Args: model (SentenceTransformer): The SentenceTransformer model to evaluate. output_path (str, optional): The output path to write the results. Defaults to None. epoch (int, optional): The current epoch number. Defaults to -1. steps (int, optional): The current step number. Defaults to -1. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. r6z after epoch z in epoch z after z stepsrNz (truncated to )z0RerankingEvaluator: Evaluating the model on the z dataset:r&mrrndcgcSsg|]}t|dqS)rrrr r r!r"}sz/RerankingEvaluator.__call__..cSsg|]}t|dqS)rrrr r r!r"~szmQueries: {} Positives: Min {:.1f}, Mean {:.1f}, Max {:.1f} Negatives: Min {:.1f}, Mean {:.1f}, Max {:.1f}z MAP: {:.2f}dzMRR@{}: {:.2f}zNDCG@{}: {:.2f}awzutf-8)newlinemodeencodingzmrr@zndcg@)rr*inforcompute_metricesr)r1rnpminmeanmaxrrospathjoinr0isfileopencsvwriterwriterowr2Zprefix_name_to_metricsZ store_metrics_in_model_card_data)r3r7r8r$r%Zout_txtZscoresmean_apmean_mrr mean_ndcgZ num_positivesZ num_negativesZcsv_pathZoutput_file_existsfrPZmetricsr r r!__call__\s\          zRerankingEvaluator.__call__cCs|jr||S||S)a Computes the evaluation metrics for the given model. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. )rcompute_metrices_batchedcompute_metrices_individual)r3r7r r r!rEs  z#RerankingEvaluator.compute_metricesc Csg}g}g}|jdkrtn ||jj|jdd|jDd|j|jd}g}|jD] }||d||dqV|j|d|j|jd}W5QRXd\} } |jD]"} || } | d 7} t| d} t| d}|| | | |}| | |7} | d ks|d krq| | |}t|j d kr,|d }t | }| }d g| d g|}d }t|d |jD]&\}}||rnd |d }qqn|||t|g|g|jd |t||qt|}t|}t|}|||d S) aE Computes the evaluation metrics in a batched way, by batching all queries and all documents together. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. NcSsg|] }|dqS)queryr rr r r!r"sz?RerankingEvaluator.compute_metrices_batched..TZconvert_to_tensorrrrr)rrrkr&r<r=)rrtruncate_sentence_embeddingsencoder)rrextendrrshapetorchargsortcputolist enumeraterappendrrrFrH)r3r7all_mrr_scoresall_ndcg_scores all_ap_scoresZall_query_embsZall_docsrZ all_docs_embsZ query_idxZdocs_idxinstance query_embnum_posZnum_negdocs_emb pred_scorespred_scores_argsort is_relevant mrr_scorerankindexrRrSrTr r r!rWs`                z+RerankingEvaluator.compute_metrices_batchedc Csg}g}g}tj|j|j ddD]`}|d}t|d}t|d}t|dks"t|dkrbq"||} dgt|dgt|} |jdkrtn ||j0|j|gd |j d d } |j| d |j d d } W5QRX| | | } t| j dkr| d} t | }| } d}t|d|jD]&\}}| |r(d|d}qPq(|||t| g| g|jd |t| | q"t|}t|}t|}|||d S)aO Computes the evaluation metrics individually by embedding every (query, positive, negative) tuple individually. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. ZSamples)disabledescrYrrrr[NTFrZr\r^)tqdmr)rr.rrrr_r`rrrbrcrdrerfrgrrhrrrFrH)r3r7rirjrkrlrYrrZdocsrrrmrorprqrsrtrurRrSrTr r r!rXsR             z.RerankingEvaluator.compute_metrices_individual)Nr6r6)__name__ __module__ __qualname____doc__r intstrboolrrcZTensorrr(rfloatrVrErWrX __classcell__r r r4r!r sD1 GJr )rOloggingrJ contextlibrtypingrrrrZnumpyrFrcrxZsklearn.metricsrrZ2sentence_transformers.evaluation.SentenceEvaluatorr Zsentence_transformers.utilr Z)sentence_transformers.SentenceTransformerr getLoggerryr*r r r r r!s