U 4Af?@s6ddlZddlZddlmZddlmZddlmZddlmZm Z m Z m Z m Z m Z mZmZddlZddlmZmZddlmZdd lmZe d eZe d e e ege eeffZGd d d ZddZd5e ee eefdddZeGdddeZe ee eefdddZ e ee eefdddZ!e ee eefdddZ"eGdddZ#eGdddeZ$d6e e%d d!d"Z&d7e e%d d#d$Z'd8e e%d d%d&Z(d'd(Z)eGd)d*d*Z*eGd+d,d,eZ+eGd-d.d.e+Z,eGd/d0d0e+Z-eGd1d2d2eZ.eGd3d4d4eZ/dS)9N)Mapping) dataclass)randint)AnyCallableDictListNewTypeOptionalTupleUnion) BertTokenizerBertTokenizerFast)PreTrainedTokenizerBase)PaddingStrategyInputDataClass DataCollatorc@seZdZdddZdS)DataCollatorMixinNcCsX|dkr|j}|dkr ||S|dkr2||S|dkrD||Std|ddS)Ntfptnpz Framework 'z' not recognized!)return_tensorstf_call torch_call numpy_call ValueErrorselffeaturesrr C/tmp/pip-unpacked-wheel-zw5xktn0/transformers/data/data_collator.py__call__'s   zDataCollatorMixin.__call__)N)__name__ __module__ __qualname__r"r r r r!r&src OsPt|ds|j||S|jdd}d|jd<z|j||}W5||jd<X|S)zz Pads without triggering the warning about how using the pad function is sub-optimal when using a fast tokenizer. deprecation_warningszAsking-to-pad-a-fast-tokenizerFT)hasattrpadr&get) tokenizerZpad_argsZ pad_kwargsZ warning_stateZpaddedr r r!"pad_without_fast_tokenizer_warning4s    r+rrreturncCs4|dkrt|S|dkr t|S|dkr0t|SdS)a Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. rrrN)torch_default_data_collatortf_default_data_collatornumpy_default_data_collator)rrr r r!default_data_collatorJs r1c@sDeZdZUdZdZeed<deeee feee fdddZ dS) DefaultDataCollatora* Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. This is an object (like other data collators) rather than a pure function like default_data_collator. This can be helpful if you need to set a return_tensors value at initialization. Args: return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". rrNr,cCs|dkr|j}t||SN)rr1rr r r!r"yszDefaultDataCollator.__call__)N) r#r$r%__doc__rstr__annotations__rrrr"r r r r!r2cs  r2csddl}t|dts$dd|D}|d}i}d|kr|ddk rt|d|jr`|dn|d}t|trx|jn|j}|jdd|D|d|d<n|d|kr|ddk rt|d|jr| d d|D|d<n:t|ddtr|jn|j}|jd d|D|d|d<| D]\}d kr |dk r t|t s t||jrt| fd d|D|<nNt|t j r|t fd d|D|<n|fdd|D|<q |S)NrcSsg|] }t|qSr vars.0fr r r! sz/torch_default_data_collator..labelcSsg|] }|dqSr=r r9r r r!r<sdtypelabels label_idscSsg|] }|dqSrBr r9r r r!r<scSsg|] }|dqSrCr r9r r r!r<sr=rBcsg|] }|qSr r r9kr r!r<scsg|] }|qSr r r9rEr r!r<scsg|] }|qSr r r9rEr r!r<s)torch isinstancerTensoritemintlongfloattensorstackitemsr5rndarray)rrGfirstbatchr=r@vr rEr!r.s,$ $ r.csddl}t|dts$dd|D}|d}i}d|krJ|ddk rJdn8d|krd|ddk rddnd|kr~|ddk r~dnddk rjt||jr|jjr|jn|j}nt|tj st|tj rt |jtj r|jn|j}nRt|t tfr.t|dtr&|jn|j}nt|trD|jn|j}|jfdd|D|d|d<|D]x\}d krr|dk rrt|tsrt||jtj fr|fd d|D|<n|fd d|D|<qr|S) NrcSsg|] }t|qSr r7r9r r r!r<sz,tf_default_data_collator..r=rBrAcsg|] }|qSr r r9)label_col_namer r!r<sr?)r=rBrAcsg|] }|qSr r r9rEr r!r<scsg|] }|qSr r r9rEr r!r<s) tensorflowrHrrIr@ is_integerint64float32rrQZgenericZ issubdtypeintegertuplelistrKconvert_to_tensorrPr5rO)rrrRrSr@rTr )rFrUr!r/s6  ""   r/cst|dtsdd|D}|d}i}d|kr|ddk rt|dtjrX|dn|d}t|trptjntj}tjdd|D|d|d<n|d|kr|ddk rt|dtjrt d d|D|d<n:t|ddtrtjntj}tjd d|D|d|d<| D]r\}d kr|dk rt|t st|tjrlt fd d|D|<ntfd d|D|<q|S)NrcSsg|] }t|qSr r7r9r r r!r<sz/numpy_default_data_collator..r=cSsg|] }|dqSr>r r9r r r!r<sr?rArBcSsg|] }|dqSrCr r9r r r!r<scSsg|] }|dqSrCr r9r r r!r<srDcsg|] }|qSr r r9rEr r!r<scsg|] }|qSr r r9rEr r!r<s) rHrrrQrJrKrXrYarrayrOrPr5)rrRrSr=r@rTr rEr!r0s&$  r0c@seZdZUdZeed<dZeee e fed<dZ e e ed<dZe e ed<dZe ed <eee efee efd d d ZdS) DataCollatorWithPaddinga Data collator that will dynamically pad the inputs received. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". r*TpaddingN max_lengthpad_to_multiple_ofrrr,cCsVt|j||j|j|j|jd}d|kr8|d|d<|d=d|krR|d|d<|d=|S)Nr`rarbrr=rArB)r+r*r`rarbr)rrrSr r r!r"s  z DataCollatorWithPadding.__call__)r#r$r%r4rr6r`r boolr5rrar rKrbrrrrr"r r r r!r_s  r_c@seZdZUdZeed<dZeee e fed<dZ e e ed<dZe e ed<dZe ed <d Ze ed <d d ZddZddZdS)"DataCollatorForTokenClassificationa Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta). label_pad_token_id (`int`, *optional*, defaults to -100): The id to use when padding the labels (-100 will be automatically ignore by PyTorch loss functions). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". r*Tr`Nrarblabel_pad_token_idrrcsddld|dkrdnd|dkrBfdd|Dnd}fdd|D}tj|jjjdd}|dkr|S|d jd jj}fd d |d krƇfdd|D|<nfdd|D|<j |j d|<|S)Nrr=rAcsg|] }|qSr r r:Zfeature label_namer r!r<IszADataCollatorForTokenClassification.torch_call..cs"g|]}fdd|DqS)csi|]\}}|kr||qSr r r:rFrTrir r! KszLDataCollatorForTokenClassification.torch_call...rPrhrir r!r<Ksrrc input_idscst|jr|St|Sr3)rHrItolistr\)Ztensor_or_iterablerGr r!to_list\s z>DataCollatorForTokenClassification.torch_call..to_listrightcs(g|] }|jgt|qSr rglenr:r=rsequence_lengthrrr r!r<bscs(g|] }jgt||qSr rtrvrwr r!r<fsr?) rGkeysr+r*r`rarbshape padding_siderNrX)rrrAZno_labels_featuresrSr{r )rjrrxrrrGr!rEs4&    z-DataCollatorForTokenClassification.torch_callcsddld|dkrdnd|dkrBfdd|Dnd}tj|jjj|dkrfdndd}|dkrz|S|djd jj }|d krfd d|D|d<nfd d|D|d<fd d| D}|S)Nrr=rAcsg|] }|qSr r rhrir r!r<qsz>DataCollatorForTokenClassification.tf_call..rrcrnrorscs(g|] }t|jgt|qSr r\rgrurvrrxr r!r<scs(g|] }jgt|t|qSr rgrur\rvr}r r!r<scs"i|]\}}|j|jdqSr?r]rXrkrr r!rlsz>DataCollatorForTokenClassification.tf_call..) rVryr+r*r`rarbr]rzr{rPrrrArSr{r )rjrrxrr!rms0&     z*DataCollatorForTokenClassification.tf_callcsd|dkrdnd|dkr:fdd|Dnd}tj|jjj|dkr^dndd}|dkrr|St|djd jj }|d krfd d|D|d<nfd d|D|d<d d| D}|S)Nr=rrAcsg|] }|qSr r rhrir r!r<szADataCollatorForTokenClassification.numpy_call..rrcrnrorscs(g|] }t|jgt|qSr r|rvr}r r!r<scs(g|] }jgt|t|qSr r~rvr}r r!r<scSs"i|]\}}|tj|tjdqSrrr^rXrkr r r!rlszADataCollatorForTokenClassification.numpy_call..) ryr+r*r`rarbrr^rzr{rPrr )rjrrxr!rs.&     z-DataCollatorForTokenClassification.numpy_call)r#r$r%r4rr6r`r rdr5rrar rKrbrgrrrrr r r r!re s   ( rerbcs:ddlt|dtttjfr0fdd|D}|ddtfdd|D}|rz|dksl|dkrzj|ddS|j dkrt d|j j d t d d|D}|dk r||dkr||d |}|dt||g|j}t|D]D\}}|jd kr|||d|jdf<q||||jd df<q|S) _Collate `examples` into a batch, using the information in `tokenizer` for padding if necessary.rNcsg|]}j|jdqSr)rNrLr:erqr r!r<sz(_torch_collate_batch..c3s|]}|dkVqdSrNsizer:xlength_of_firstr r! sz'_torch_collate_batch..)ZdimCYou are attempting to pad samples but the tokenizer you are using () does not have a pad token.css|]}|dVqdSrrrr r r!rsrors)rGrHr\r[rrQrallrO _pad_tokenr __class__r#maxZnew_fullru pad_token_id enumerater{rzexamplesr*rbare_tensors_same_lengthraresultiexampler )rrGr!_torch_collate_batchs(  rc sLddlt|dttfr,fdd|D}t|dtfdd|D}|rt|dksf|dkrtj|ddS|jdkrtd|j j dt d d|D}|dk r||dkr||d |}g} |d}t j|d ft jd }|D]J}|jd kr|t||d<n|t||d<|j|||jdqj|ddS)Nrcsg|]}j|jdqSrrrrr r!r<sz%_tf_collate_batch..c3s|]}t|kVqdSr3rurrr r!rsz$_tf_collate_batch..Zaxisrrcss|]}t|VqdSr3rrr r r!rsror r?rs)rro)rr)Zconstant_values)rVrHr\r[rurrOrrrr#rrankrzerosZint32r{appendr(r) rr*rbrrarrZpaddingsrr )rrr!_tf_collate_batchs.   rcs.t|dttfr dd|D}t|dtfdd|D}|rh|dksZ|dkrhtj|ddS|jdkrtd|j j d t d d|D}|dk r||dkr||d |}tj t||f|j |djd }t|D]D\}}|jd kr|||d|jdf<q||||jd df<q|S)rrcSsg|]}tj|tjdqSrrrr r r!r<sz(_numpy_collate_batch..c3s|]}t|kVqdSr3rrrr r!rsz'_numpy_collate_batch..Nrrrcss|]}t|VqdSr3rrr r r!r sro)rzZ fill_valuer@rs)rHr\r[rurrrOrrrr#rfullrr@rr{rzrr rr!_numpy_collate_batchs&    rcCs(t|tr|St|dr |}|S)Nnumpy)rHr\r'rrp)rr r r!rps   rpc@seZdZUdZeed<dZeeed<dZ e e e e fed<dZeeed<dZeeed<d Zeed <d Ze ed <dd dZdS)DataCollatorForSeq2Seqa Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. model ([`PreTrainedModel`], *optional*): The model that is being trained. If set and has the *prepare_decoder_input_ids_from_labels*, use it to prepare the *decoder_input_ids* This is useful when using *label_smoothing* to avoid calculating loss twice. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta). label_pad_token_id (`int`, *optional*, defaults to -100): The id to use when padding the labels (-100 will be automatically ignored by PyTorch loss functions). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". r*NmodelTr`rarbrfrgrrc sl|dkrj}d|dkr"dnd|dkrHfdd|Dnd}|dk rjtdd|Drjd}fdd|D}tj|jjj|d }jd kpjtj k}|dk r|rt |dt rt ||d<nd d|D|d<njtj kojdk }|s t d d|Dnjjdk rLjd jjjjt |dt rfdd|D|d<nfdd|D|d<|dddk r"|dkrddl}|j|d|jd|d<nD|dkrddl} | j|d| jd|d<ntj|dtjd|d<nd|d<|dk rhjdk rhtjdrhjj|dd} | |d<|S)Nr=rrAcsg|] }|qSr r rhrir r!r<Nsz3DataCollatorForSeq2Seq.__call__..css|]}|dkVqdSr3r rvr r r!rQsz2DataCollatorForSeq2Seq.__call__..cs"g|]}fdd|DqS)csi|]\}}|kr||qSr r rkrir r!rlSsz>DataCollatorForSeq2Seq.__call__...rmrhrir r!r<SsrcFcSsg|]}t|ggqSr )r concatenatervr r r!r<fscss|]}t|VqdSr3r)r:lr r r!risrocsDg|]<}dkr(|jgt|njgt||qS)rsrtrvmax_label_lengthr{rr r!r<ssc shg|]`}dkr:t|tjjgt|tjdgn(ttjjgt|tjd|gqS)rsr?)rrr^rgrurXrvrr r!r<zsrr?r%prepare_decoder_input_ids_from_labels)rAdecoder_input_ids)rryrr+r*r`rarbrZ DO_NOT_PADrHr\Z MAX_LENGTHrr{r)rGrNrXrVZconstantrr^rr'r) rrrrAZnon_labels_featuresrSZ no_paddingZ max_paddingrGrrr )rjrr{rr!r"Isp&         zDataCollatorForSeq2Seq.__call__)N)r#r$r%r4rr6rr rr`r rdr5rrarKrbrgrr"r r r r!rs !  rc@sdeZdZUdZeed<dZeed<dZe ed<dZ e e ed<d Z eed <d Zeed <d dZeddZdee eeeefdddZeeee eeeeffeeefdddZeeee eeeeffeeefdddZd ee eeeefdddZeeee eeeeffeeefdddZd!ee eeeefdddZdS)"DataCollatorForLanguageModelingal Data collator used for language modeling. Inputs are dynamically padded to the maximum length of a batch if they are not all of the same length. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. mlm (`bool`, *optional*, defaults to `True`): Whether or not to use masked language modeling. If set to `False`, the labels are the same as the inputs with the padding tokens ignored (by setting them to -100). Otherwise, the labels are -100 for non-masked tokens and the value to predict for the masked token. mlm_probability (`float`, *optional*, defaults to 0.15): The probability with which to (randomly) mask tokens in the input, when `mlm` is set to `True`. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. return_tensors (`str`): The type of Tensor to return. Allowable values are "np", "pt" and "tf". For best performance, this data collator should be used with a dataset having items that are dictionaries or BatchEncoding, with the `"special_tokens_mask"` key, as returned by a [`PreTrainedTokenizer`] or a [`PreTrainedTokenizerFast`] with the argument `return_special_tokens_mask=True`. r*Tmlmg333333?mlm_probabilityNrbFtf_experimental_compilerrcCs>|jr|jjdkrtd|jr:ddl}|j|jdd|_dS)NzThis tokenizer does not have a mask token which is necessary for masked language modeling. You should pass `mlm=False` to train on causal language modeling instead.rT)Z jit_compile)rr* mask_tokenrrrVfunctiontf_mask_tokens)rrr r r! __post_init__sz-DataCollatorForLanguageModeling.__post_init__cCs6ddl}|||}|||j|dddk|jS)Nrro)rVfillcastrandomuniformrd)rzZ probabilityrZ prob_matrixr r r! tf_bernoullis z,DataCollatorForLanguageModeling.tf_bernoulli)inputsspecial_tokens_maskr-c Csddl}|||j}||}|||j|@}|||d}||d|@} || ||}||d|@| @} |jj|||jd} || | |}||fS)w Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. rNrf皙??maxvalr@) rVrr@rzrrwhererr) rr vocab_size mask_token_idrr input_shapemasked_indicesrAindices_replacedindices_random random_wordsr r r!rs z.DataCollatorForLanguageModeling.tf_mask_tokensrr-cs ddl}t|dtr,tj|djd}ndt|jjdi}|dd}jr|dkrfdd|d D}| |j ||j d |j}n| ||j}j| |d|j |jjtjd \|d<|d <n@|d}jjdk r ||jjkd |}n ||}||d <|S) Nrrrrbrnrrcsg|]}jj|ddqST)Zalready_has_special_tokensr*Zget_special_tokens_maskr:valrr r!r< sz;DataCollatorForLanguageModeling.tf_call..r?)rrrrArf)rVrHrr+r*rbrpoprrrprr]rXrdrrrurridentity)rrrrSrrAr rr!rs>   z'DataCollatorForLanguageModeling.tf_callcCst|dtr$t|j|d|jd}ndt||j|jdi}|dd}|jrl|j|d|d\|d<|d<n0|d }|jj dk rd |||jj k<||d<|S) NrrrrnrrrrArf) rHrr+r*rbrrrtorch_mask_tokensclonerrrrSrrAr r r!r#s*   z*DataCollatorForLanguageModeling.torch_callc sddl}|}||jj}|dkrPfdd|D}|j||jd}n|}|j|dd| |}d||<| ||jd |@}j j j ||<| ||jd |@|@}|j tj |j|jd} | |||<||fS) rrNcsg|]}jj|ddqSrrrrr r!r<EszEDataCollatorForLanguageModeling.torch_mask_tokens..r?valuerfrr)rGrrrzrrprNrd masked_fill_ bernoullir*convert_tokens_to_idsrrrurL) rrrrGrAprobability_matrixrrrrr rr!r;s$  " z1DataCollatorForLanguageModeling.torch_mask_tokenscCst|dtr$t|j|d|jd}ndt||j|jdi}|dd}|jrl|j|d|d\|d<|d<n2t |d}|jj dk rd |||jj k<||d<|S) NrrrrnrrrrArf) rHrr+r*rbrrrnumpy_mask_tokensrcopyrrr r r!r\s*  z*DataCollatorForLanguageModeling.numpy_callc st|}t|jj}|dkrHfdd|D}tj|td}n |t}d||<tj j d||jdt}d||<tj j dd |jdt|@}j j ||<tj j dd |jdt|@|@}tj j dtj t|tjd }|||<||fS) rNcsg|]}jj|ddqSrrrrr r!r<|szEDataCollatorForLanguageModeling.numpy_mask_tokens..r?rrorrfrrlowhighrr@)rrrrzrrpr^rdastyperbinomialr*rrruZ count_nonzerorX) rrrrArrrrrr rr!rts.     "z1DataCollatorForLanguageModeling.numpy_mask_tokens)N)N)N)r#r$r%r4rr6rrdrrMrbr rKrrr5r staticmethodrrr rrr rrrrrrr r r r!rs*        2)2 !2rc@seZdZdZeeeeeee effee efdddZ eeeeeee effee efdddZ eeeeeee effee efdddZ dee d d d Z eeeeefd ddZeeeeefd ddZeeeeefd ddZdS)DataCollatorForWholeWordMaska# Data collator used for language modeling that masks entire words. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses batches for masked language modeling This collator relies on details of the implementation of subword tokenization by [`BertTokenizer`], specifically that subword tokens are prefixed with *##*. For tokenizers that do not adhere to this scheme, this collator will produce an output that is roughly equivalent to [`.DataCollatorForLanguageModeling`]. rcCst|dtrdd|D}n|}dd|D}t||j|jd}g}|D]}g}t|dD]}|j|}||q^d|krt|d} t|d} t | D]} | | krd|| || <q|| |qJt||j|jd} | || \} }| |d S) NrcSsg|] }|dqSrnr rr r r!r<sz;DataCollatorForWholeWordMask.torch_call..cSsg|] }d|iqSrr rr r r!r<srrn chinese_ref##rnrA) rHrrr*rbrp_convert_id_to_tokenrrurange_whole_word_maskrrrrn batch_input mask_labelsr ref_tokensidtokenref_poslen_seqr batch_maskrrAr r r!rs*     z'DataCollatorForWholeWordMask.torch_callcCsddl}t|dtr&dd|D}n|}dd|D}t||j|jd}g}|D]}g}t|dD]}|j|} || qfd|krt|d} t |d} t | D]} | | krd|| || <q|| |qRt||j|jd} | | ||j| \}}||d S) NrcSsg|] }|dqSrr rr r r!r<sz8DataCollatorForWholeWordMask.tf_call..cSsg|] }d|iqSrr rr r r!r<srrnrrr)rVrHrrr*rbrprrrurrrrrX)rrrrnrrrrrrrrrrrrAr r r!rs,     z$DataCollatorForWholeWordMask.tf_callcCst|dtrdd|D}n|}dd|D}t||j|jd}g}|D]}g}t|dD]}|j|}||q^d|krt|d} t|d} t | D]} | | krd|| || <q|| |qJt||j|jd} | || \} }| |d S) NrcSsg|] }|dqSrr rr r r!r<sz;DataCollatorForWholeWordMask.numpy_call..cSsg|] }d|iqSrr rr r r!r<srrnrrr) rHrrr*rbrprrrurrrrr r r!rs*     z'DataCollatorForWholeWordMask.numpy_call) input_tokensc sXt|jttfstdg}t|D]L\}}|dks&|dkr@q&t|dkrf|drf|d |q&| |gq&t |t |t dttt||j}g}t|D]p}t||krq t|t||krqd} |D]} | krd} qq| rq|D]} | | | qqtt|kr:td fd d tt|D} | S) zM Get 0/1 labels for masked tokens with whole word mask proxy zDataCollatorForWholeWordMask is only suitable for BertTokenizer-like tokenizers. Please refer to the documentation for more information.z[CLS]z[SEP]rorFTz?Length of covered_indexes is not equal to length of masked_lms.csg|]}|krdndqS)rorr )r:rZcovered_indexesr r!r<,szADataCollatorForWholeWordMask._whole_word_mask..)rHr*rrwarningswarnrru startswithrrshuffleminrrKroundrsetaddrr) rrZmax_predictionsZ cand_indexesrrZnum_to_predictZ masked_lmsZ index_setZis_any_index_coveredindexrr rr!rsD "  z-DataCollatorForWholeWordMask._whole_word_mask)rrr-c sddl}jjdkrtd|}|}fdd|D}|j|j||jdddjj dk r| jj }|j|dd|}d ||<| | |jd |@} jjj|| <| | |jd |@| @} |jtj|j|jd} | | || <||fS) Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. Set 'mask_labels' means we use whole word mask (wwm), we directly mask idxs according to it's ref. rNThis tokenizer does not have a mask token which is necessary for masked language modeling. Remove the --mlm flag if you want to use this tokenizer.csg|]}jj|ddqSrrrrr r!r<@szBDataCollatorForWholeWordMask.torch_mask_tokens..r?rrrfrr)rGr*rrrrprrNrdreqrrrrzrrrurL) rrrrGrArr padding_maskrrrrr rr!r/s,    " z.DataCollatorForWholeWordMask.torch_mask_tokensc sddl}||}jjdkr&td||}|||j}fdd|D}||j||jd@}jjdk r|jj k}||@}| ||d} |d|@} | | jj |} |d |@| @} |j j|tj|jd } | | | |}||fS) rrNrcsg|]}jj|ddqSrrrrr r!r<isz?DataCollatorForWholeWordMask.tf_mask_tokens..r?rfrrr)rVrzr*rrrrrdrrrrrrrrurX) rrrrrrArrr rrrr rr!rWs,       z+DataCollatorForWholeWordMask.tf_mask_tokensc sjjdkrtdt|}|t}fdd|D}d|tj|td<jj dk rp|jj k}d||<d||<tj j dd |j d t|@}jjj||<tj j dd |j d t|@|@}tj jdtj|j tjd } | |||<||fS) rNrcsg|]}jj|ddqSrrrrr r!r<szBDataCollatorForWholeWordMask.numpy_mask_tokens..rr?rfrorrrr)r*rrrrrrdrpr^rrrrrzrrrurX) rrrrArrr rrrr rr!rs*       " z.DataCollatorForWholeWordMask.numpy_mask_tokensN)r)r#r$r%r4rr rKrrr5rrrrr rrrr r r r!rs222/(*rc@sVeZdZdZddZeeeefeeefdddZ ee eeefddd Z d S) DataCollatorForSOPz Data collator used for sentence order prediction task. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses batches for both masked language modeling and sentence order prediction cOstdtdS)NzDataCollatorForSOP is deprecated and will be removed in a future version, you can now use DataCollatorForLanguageModeling instead.)rr FutureWarning)rargskwargsr r r!__init__szDataCollatorForSOP.__init__rc Csddl}ddlm}dd|D}t||j}||\}}}dd|D}||d|jjd}dd|D}||} ||||| d S) Nr) pad_sequencecSsg|] }|dqSrr r:rr r r!r<sz/DataCollatorForSOP.__call__..cSsg|] }|dqS)token_type_idsr rr r r!r<sT)Z batch_firstZ padding_valuecSsg|] }|dqS)sentence_order_labelr rr r r!r<s)rnrAattention_maskrr)rGZtorch.nn.utils.rnnrrr* mask_tokensrrO) rrrGrrnrArrZsop_label_listrr r r!r"s   zDataCollatorForSOP.__call__rr-c sXddl}jjdkrtd|}||jj}fdd|D}|j |j ||j dddjj dk r| jj}|j |dd|| }|}jj dk r| jj} |j | d dd ||<|||jd  |@} jjj|| <|||jd  |@| @} |jtj|j|jd} | | || <|||fS) z Prepare masked tokens inputs/labels/attention_mask for masked language modeling: 80% MASK, 10% random, 10% original. N-gram not applied yet. rNrcsg|]}jj|ddqSrrrrr r!r<sz2DataCollatorForSOP.mask_tokens..r?rrg?rfrr)rGr*rrrrrzrrprrNrdrrrrrMrrrurL) rrrGrArrr rrZattention_padding_maskrrrr rr!rs4      " zDataCollatorForSOP.mask_tokensN) r#r$r%r4rrrr5rr"r rr r r r!r s$r c@s(eZdZUdZeed<dZeed<dZe ed<dZ e ed<e e e e eee effee efd d d Ze e e e eee effee efd d d Ze e e e eee effee efd ddZeeeeeefdddZeeeeeefdddZeeeeeefdddZdS)*DataCollatorForPermutationLanguageModelingz Data collator used for permutation language modeling. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses batches for permutation language modeling with procedures specific to XLNet r*gUUUUUU?plm_probabilitymax_span_lengthrrrcCsHt|dtrdd|D}t||j}||\}}}}||||dS)NrcSsg|] }|dqSrr rr r r!r< szIDataCollatorForPermutationLanguageModeling.torch_call..rn perm_masktarget_mappingrA)rHrrr*rrrrSrrrrAr r r!r s  z5DataCollatorForPermutationLanguageModeling.torch_callcCsHt|dtrdd|D}t||j}||\}}}}||||dS)NrcSsg|] }|dqSrr rr r r!r<szFDataCollatorForPermutationLanguageModeling.tf_call..r)rHrrr*rrr r r!rs  z2DataCollatorForPermutationLanguageModeling.tf_callcCsHt|dtrdd|D}t||j}||\}}}}||||dS)NrcSsg|] }|dqSrr rr r r!r<szIDataCollatorForPermutationLanguageModeling.numpy_call..r)rHrrr*rrr r r!rs  z5DataCollatorForPermutationLanguageModeling.numpy_callrcsddl}jjdkrtd|dddkr6td|}|j|jd|jd}|j |d|d|df|j d}t |dD]}d}|d}||kr| dj dd} t| j} || | | dd} d||| | | f<|| 7}q||d||<q|jfd d |D|jd} |j| d d jjdk rt|jj} |j| d d | | B}jj||<d ||<|j |d|d|df|j d}t |dD]}||d}|d|ddfdd}|||dd}||dd}|||||@d||ddf|d|dfk||@||<q||||fS)g The masked tokens to be predicted for a particular sequence are determined by the following algorithm: 0. Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far). 1. Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked) 2. Reserve a context of length `context_length = span_length / plm_probability` to surround span to be masked 3. Sample a starting point `start_index` from the interval `[cur_len, cur_len + context_length - span_length]` and mask tokens `start_index:start_index + span_length` 4. Set `cur_len = cur_len + context_length`. If `cur_len < max_len` (i.e. there are tokens remaining in the sequence to be processed), repeat from Step 1. rNThis tokenizer does not have a mask token which is necessary for permutation language modeling. Please add a mask token if you want to use this tokenizer.ror This collator requires that sequence lengths be even to create a leakage-free perm_mask. Please see relevant comments in source code for details.r?)rocsg|]}jj|ddqSrrrrr r!r<UszPDataCollatorForPermutationLanguageModeling.torch_mask_tokens..rrrfr)rGr*rrrrrrzrdrrYrrrrJrKreyerNrprrrrrarangereshape transposeZrandpermflattenrL)rrrGrArrrcur_lenmax_len span_lengthcontext_length start_indexrr  non_func_maskr perm_indexr rr!r sX  (      (  & z.rfr)rr)"rVr*rrrzrrrZas_listrdrrYrrurrrKrr!rr]rrprrrrr$r#rrrrOrX)rrrrArZ labels_shaperrr&r'r(r)r*rr r+rr,r rr!rsh    "           &z9DataCollatorForPermutationLanguageModeling.tf_mask_tokenscsPjjdkrtd|jdddkr.tdt|}tj|jdtd}tj|jd|jd|jdftj d}t |jdD]}d}|jd}||krt dj d}t |j} |t d| |d} d||| | |f<|| 7}qt|jd||<qtjfdd |Dtd} d|| <jjdk rL|jjk} d || <| | B} jj||<d ||<tj|jd|jd|jdftj d}t |jdD]}t|jd}|d |jddfj}tj||j}d |||| |@<||jddf|d|jdfk||@||<q|tj|||tjfS) rNrror rr r?csg|]}jj|ddqSrrrrr r!r<!szPDataCollatorForPermutationLanguageModeling.numpy_mask_tokens..rrfr)r*rrrzrrrrdrrYrrrrKrr!r^rprrrr"r#Trrr%rrX)rrrArrrr&r'r(r)r*rr r+rr,r rr!rsV  (      (   & zs\   (    $) 3 $& tRT