U DAfZO@slddlmZddlmZmZmZmZmZmZddl m Z ddl m Z mZmZeeZeGdddZdS)) defaultdict)AnyCallableDictListLiteralOptional)parse)Document componentloggingc @sBeZdZdZd"eeeeededed eed d d d Z eeeededed eed dddZ e j e edd#e eeeeeeedeedeed eed dddZe eeed e edddZe ee eeede edddZed$eeedddZeeeedd d!ZdS)%MetaFieldRankerav Ranks Documents based on the value of their specific meta field. The ranking can be performed in descending order or ascending order. Usage example: ``` from haystack import Document from haystack.components.rankers import MetaFieldRanker ranker = MetaFieldRanker(meta_field="rating") docs = [ Document(content="Paris", meta={"rating": 1.3}), Document(content="Berlin", meta={"rating": 0.7}), Document(content="Barcelona", meta={"rating": 2.1}), ] output = ranker.run(documents=docs) docs = output["documents"] assert docs[0].content == "Barcelona" ?Nreciprocal_rank_fusion descendingbottomr linear_scoreZ ascendingrZdroptopr)floatintdate) meta_fieldweighttop_k ranking_mode sort_order missing_metameta_value_typecCsN||_||_||_||_||_||_|j|j|j|j|j|j|d||_dS)a Creates an instance of MetaFieldRanker. :param meta_field: The name of the meta field to rank by. :param weight: In range [0,1]. 0 disables ranking by a meta field. 0.5 ranking from previous component and based on meta field have the same weight. 1 ranking by a meta field only. :param top_k: The maximum number of Documents to return per query. If not provided, the Ranker returns all documents it receives in the new ranking order. :param ranking_mode: The mode used to combine the Retriever's and Ranker's scores. Possible values are 'reciprocal_rank_fusion' (default) and 'linear_score'. Use the 'linear_score' mode only with Retrievers or Rankers that return a score in range [0,1]. :param sort_order: Whether to sort the meta field by ascending or descending order. Possible values are `descending` (default) and `ascending`. :param missing_meta: What to do with documents that are missing the sorting metadata field. Possible values are: - 'drop' will drop the documents entirely. - 'top' will place the documents at the top of the metadata-sorted list (regardless of 'ascending' or 'descending'). - 'bottom' will place the documents at the bottom of metadata-sorted list (regardless of 'ascending' or 'descending'). :param meta_value_type: Parse the meta value into the data type specified before sorting. This will only work if all meta values stored under `meta_field` in the provided documents are strings. For example, if we specified `meta_value_type="date"` then for the meta value `"date": "2015-02-01"` we would parse the string into a datetime object and then sort the documents by date. The available options are: - 'float' will parse the meta values into floats. - 'int' will parse the meta values into integers. - 'date' will parse the meta values into datetime objects. - 'None' (default) will do no parsing. rrrrrr N)rrrrrr_validate_paramsr )selfrrrrrrr r$J/tmp/pip-unpacked-wheel-z752163x/haystack/components/rankers/meta_field.py__init__'s2zMetaFieldRanker.__init__r!cCs|dk r|dkrtd||dks,|dkr8td||dkrLtd||dkr`td||d krttd ||d krtd |dS) Nrztop_k must be > 0, but got %saCParameter must be in range [0,1] but is currently set to '%s'. '0' disables sorting by a meta field, '0.5' assigns equal weight to the previous relevance scores and the meta field, and '1' ranks by the meta field only. Change the parameter to a value in range 0 to 1 when initializing the MetaFieldRanker.rzThe value of parameter must be 'reciprocal_rank_fusion' or 'linear_score', but is currently set to '%s'. Change the value to 'reciprocal_rank_fusion' or 'linear_score' when initializing the MetaFieldRanker.rzThe value of parameter must be 'ascending' or 'descending', but is currently set to '%s'. Change the value to 'ascending' or 'descending' when initializing the MetaFieldRanker.rzThe value of parameter must be 'drop', 'top', or 'bottom', but is currently set to '%s'. Change the value to 'drop', 'top', or 'bottom' when initializing the MetaFieldRanker.)rrrNzThe value of parameter must be 'float', 'int', 'date' or None but is currently set to '%s'. Change the value to 'float', 'int', 'date' or None when initializing the MetaFieldRanker.) ValueError)r#rrrrrr r$r$r%r"is@  z MetaFieldRanker._validate_params) documents)r)rrrrrr c sb|s dgiS|pj}|dk r"|nj}|p0j}|p:j}|pDj}|pNj}j||||||d|dkr~d|d|iSfdd|D}fdd|D} t|dkrtj dj d d d|Dd d|d|iSt| dkrTd j d d dd| Dd} |dkr,tj d| dn(|dkrFtj d| dntj d| dj ||d} t t| |} |dk} zt| dd| d}WnXtk r}z8tj dd dd|D|dd|d|iWYSd}~XYnXdd|D}|dkr|| }||||}n8|dkr>| |}||||}n|}||||}d|d|iS)a Ranks a list of Documents based on the selected meta field by: 1. Sorting the Documents by the meta field in descending or ascending order. 2. Merging the rankings from the previous component and based on the meta field according to ranking mode and weight. 3. Returning the top-k documents. :param documents: Documents to be ranked. :param top_k: The maximum number of Documents to return per query. If not provided, the top_k provided at initialization time is used. :param weight: In range [0,1]. 0 disables ranking by a meta field. 0.5 ranking from previous component and based on meta field have the same weight. 1 ranking by a meta field only. If not provided, the weight provided at initialization time is used. :param ranking_mode: (optional) The mode used to combine the Retriever's and Ranker's scores. Possible values are 'reciprocal_rank_fusion' (default) and 'linear_score'. Use the 'score' mode only with Retrievers or Rankers that return a score in range [0,1]. If not provided, the ranking_mode provided at initialization time is used. :param sort_order: Whether to sort the meta field by ascending or descending order. Possible values are `descending` (default) and `ascending`. If not provided, the sort_order provided at initialization time is used. :param missing_meta: What to do with documents that are missing the sorting metadata field. Possible values are: - 'drop' will drop the documents entirely. - 'top' will place the documents at the top of the metadata-sorted list (regardless of 'ascending' or 'descending'). - 'bottom' will place the documents at the bottom of metadata-sorted list (regardless of 'ascending' or 'descending'). If not provided, the missing_meta provided at initialization time is used. :param meta_value_type: Parse the meta value into the data type specified before sorting. This will only work if all meta values stored under `meta_field` in the provided documents are strings. For example, if we specified `meta_value_type="date"` then for the meta value `"date": "2015-02-01"` we would parse the string into a datetime object and then sort the documents by date. The available options are: -'float' will parse the meta values into floats. -'int' will parse the meta values into integers. -'date' will parse the meta values into datetime objects. -'None' (default) will do no parsing. :returns: A dictionary with the following keys: - `documents`: List of Documents sorted by the specified meta field. :raises ValueError: If `top_k` is not > 0. If `weight` is not in range [0,1]. If `ranking_mode` is not 'reciprocal_rank_fusion' or 'linear_score'. If `sort_order` is not 'ascending' or 'descending'. If `meta_value_type` is not 'float', 'int', 'date' or `None`. r)Nr!rcsg|]}j|jkr|qSr$rmeta.0docr#r$r% s z'MetaFieldRanker.run..csg|]}j|jkr|qSr$r*r,r/r$r%r0s a7The parameter is currently set to '{meta_field}', but none of the provided Documents with IDs {document_ids} have this meta key. Set to the name of a field that is present within the provided Documents. Returning the of the original Documents since there are no values to rank.,cSsg|] }|jqSr$idr,r$r$r%r0sr document_idsz0The parameter is currently set to 'z' but the Documents with IDs cSsg|] }|jqSr$r2r,r$r$r%r0sz don't have this meta key. rz{warning_start}Because the parameter is set to 'bottom', these Documents will be placed at the end of the sorting order.) warning_startrz{warning_start}Because the parameter is set to 'top', these Documents will be placed at the top of the sorting order.z{warning_start}Because the parameter is set to 'drop', these Documents will be removed from the list of retrieved Documents.)docs_with_meta_fieldr rcSs|dS)Nrr$)xr$r$r%&z%MetaFieldRanker.run..keyreversezTried to sort Documents with IDs {document_ids}, but got TypeError with the message: {error} Returning the of the original Documents since meta field ranking is not possible.cSsg|] }|jqSr$r2r,r$r$r%r0,sr5errorcSsg|] \}}|qSr$r$)r-r+r.r$r$r%r02s)rrrrrr r"lenloggerwarningrjoin _parse_metalistzipsorted TypeError_merge_rankings)r#r)rrrrrr r7Zdocs_missing_meta_fieldr6Z parsed_metaZtuple_parsed_meta_and_docsr=Ztuple_sorted_by_metar?Zsorted_by_metasorted_documentsr$r/r%runsE       "  &  zMetaFieldRanker.run)r7r returnc s|dkrfdd|DSfdd|D}tdd|Dsptjd|d d d|Dd fd d|DS|d kr~tn|dkrtntzfdd|D}WnTtk r}z6tjdd dd|D|dfdd|D}W5d}~XYnX|S)z| Parse the meta values stored under `self.meta_field` for the Documents provided in `docs_with_meta_field`. Ncsg|]}|jjqSr$r+rr-dr/r$r%r0Fsz/MetaFieldRanker._parse_meta..csh|]}|jjqSr$rMr,r/r$r% Hsz.MetaFieldRanker._parse_meta..css|]}t|tVqdS)N) isinstancestr)r-Z meta_valuer$r$r% Isz.MetaFieldRanker._parse_meta..a"The parameter is currently set to '{meta_field}', but not all of meta values in the provided Documents with IDs {document_ids} are strings. Skipping parsing of the meta values. Set all meta values found under the parameter to strings to use .r1cSsg|] }|jqSr$r2r,r$r$r%r0Psr4csg|]}|jjqSr$rMrNr/r$r%r0Rsrrcsg|]}|jjqSr$rMrNZparse_fnr#r$r%r0]szTried to parse the meta values of Documents with IDs {document_ids}, but got ValueError with the message: {error} Skipping parsing of the meta values.cSsg|] }|jqSr$r2r,r$r$r%r0csr>csg|]}|jjqSr$rMrNr/r$r%r0fs)allrArBrCrr date_parser()r#r7r Zunique_meta_valuesZ meta_valuesr?r$rTr%rD?s2$zMetaFieldRanker._parse_meta)r)rJrrrLc CsNtt}|dkrntt||D]L\}\}}||j|j|dd|7<||j|j|d|7<qn|dkr tt||D]\}\}}td} |jdkrt dn0|jdks|jdkrtj d|j|jd n|j} ||j| d|7<||j|j |t |d |7<q|D]}||j|_q$t |d d d d} | S)zv Merge the two different rankings for Documents sorted both by their content and by their meta field. r)rankr'rrNz+The score wasn't provided; defaulting to 0.zXThe score {score} for Document {document_id} is outside the [0,1] range; defaulting to 0)scoreZ document_idrWamountcSs|jr |jSdS)N)rX)r.r$r$r%r9r:z1MetaFieldRanker._merge_rankings..Tr;) rr enumeraterFr3_calculate_rrfrrXrArB_calc_linear_scorer@rG) r#r)rJrrZ scores_mapidocumentZ sorted_docrXZnew_sorted_documentsr$r$r%rIjs. ""   &zMetaFieldRanker._merge_rankings=)rWkrLcCs d||S)a Calculates the reciprocal rank fusion. The constant K is set to 61 (60 was suggested by the original paper, plus 1 as python lists are 0-based and the [paper](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) used 1-based ranking). r'r$)rWrbr$r$r%r]szMetaFieldRanker._calculate_rrf)rWrZrLcCs |||S)a\ Calculate the meta field score as a linear score between the greatest and the lowest score in the list. This linear scaling is useful for: - Reducing the effect of outliers - Creating scores that are meaningfully distributed in the range [0,1], similar to scores coming from a Retriever or Ranker. r$rYr$r$r%r^s z"MetaFieldRanker._calc_linear_score)rNrrrN)NNNNNN)ra)__name__ __module__ __qualname____doc__rRrrrrr&r"r Z output_typesrr rKrrDrI staticmethodr]r^r$r$r$r%r sl D 3    $  - ' r N) collectionsrtypingrrrrrrZdateutil.parserr rVZhaystackr r r getLoggerrcrAr r$r$r$r%s