U 4Af6@sddlZddlZddlZddlmZddlmZmZmZddlm Z m Z m Z e e ZeGdddZedd Gd d d ZGd d d ZGdddeZdS)N) dataclass)ListOptionalUnion)is_tf_availableis_torch_availableloggingc@sJeZdZUdZeed<eed<dZeeed<dZeeed<ddZ dS) InputExamplea5 A single training/test example for simple sequence classification. Args: guid: Unique id for the example. text_a: string. The untokenized text of the first sequence. For single sequence tasks, only this sequence must be specified. text_b: (Optional) string. The untokenized text of the second sequence. Only must be specified for sequence pair tasks. label: (Optional) string. The label of the example. This should be specified for train and dev examples, but not for test examples. guidtext_aNtext_blabelcCstjt|dddS)*Serializes this instance to a JSON string.)indent jsondumps dataclassesZasdictselfrF/tmp/pip-unpacked-wheel-zw5xktn0/transformers/data/processors/utils.pyto_json_string1szInputExample.to_json_string) __name__ __module__ __qualname____doc__str__annotations__r rrrrrrrr s  r T)frozenc@sfeZdZUdZeeed<dZeeeed<dZ eeeed<dZ ee ee fed<ddZ dS) InputFeaturesa A single set of features of data. Property names are the same names as the corresponding inputs to a model. Args: input_ids: Indices of input sequence tokens in the vocabulary. attention_mask: Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: Usually `1` for tokens that are NOT MASKED, `0` for MASKED (padded) tokens. token_type_ids: (Optional) Segment token indices to indicate first and second portions of the inputs. Only some models use them. label: (Optional) Label corresponding to the input. Int for classification problems, float for regression problems. input_idsNattention_masktoken_type_idsrcCstt|dS)rrrrrrrrKszInputFeatures.to_json_string)rrrrrintr!r%rr&rrfloatrrrrrr#6s  r#c@sNeZdZdZddZddZddZdd Zd d Zd d Z e dddZ dS) DataProcessorzEBase class for data converters for sequence classification data sets.cCs tdS)z Gets an example from a dict with tensorflow tensors. Args: tensor_dict: Keys and values should match the corresponding Glue tensorflow_dataset examples. NNotImplementedError)rZ tensor_dictrrrget_example_from_tensor_dictSsz*DataProcessor.get_example_from_tensor_dictcCs tdS)z8Gets a collection of [`InputExample`] for the train set.Nr*rdata_dirrrrget_train_examples]sz DataProcessor.get_train_examplescCs tdS)z6Gets a collection of [`InputExample`] for the dev set.Nr*r-rrrget_dev_examplesaszDataProcessor.get_dev_examplescCs tdS)z7Gets a collection of [`InputExample`] for the test set.Nr*r-rrrget_test_exampleseszDataProcessor.get_test_examplescCs tdS)z*Gets the list of labels for this data set.Nr*rrrr get_labelsiszDataProcessor.get_labelscCs(t|dkr$|t|j|_|S)z Some tensorflow_datasets datasets are not formatted the same way the GLUE datasets are. This method converts examples to the correct format. )lenr2r'r)rexamplerrrtfds_mapmszDataProcessor.tfds_mapNc Cs:t|ddd"}ttj|d|dW5QRSQRXdS)z!Reads a tab separated value file.rz utf-8-sig)encoding ) delimiter quotecharN)openlistcsvreader)clsZ input_filer;frrr _read_tsvvszDataProcessor._read_tsv)N) rrrrr,r/r0r1r2r6 classmethodrBrrrrr)Ps  r)c@sdeZdZdZdddZddZd d ZedddZedddZ dddZ dddZ dddZ dS)%SingleSentenceClassificationProcessorz@Generic processor for a single sentence classification data set.NclassificationFcCs4|dkr gn||_|dkrgn||_||_||_dSN)labelsexamplesmodeverbose)rrGrHrIrJrrr__init__sz.SingleSentenceClassificationProcessor.__init__cCs t|jSrF)r4rHrrrr__len__sz-SingleSentenceClassificationProcessor.__len__cCs(t|trt|j|j|dS|j|S)N)rGrH) isinstanceslicerDrGrH)ridxrrr __getitem__s z1SingleSentenceClassificationProcessor.__getitem__rr3c Ks(|f|}|j||||||ddd|S)NT) split_name column_label column_text column_idskip_first_rowoverwrite_labelsoverwrite_examples)add_examples_from_csv) r@ file_namerRrSrTrUrVkwargs processorrrrcreate_from_csvs  z5SingleSentenceClassificationProcessor.create_from_csvcKs|f|}|j||d|S)N)rG) add_examples)r@texts_or_text_and_labelsrGr[r\rrrcreate_from_exampless z:SingleSentenceClassificationProcessor.create_from_examplesc Cs||} |r| dd} g} g} g} t| D]`\} }| ||| |||dk rj| ||q.|r||d| nt| }| |q.|j| | | ||dS)Nr3-)rWrX)rB enumerateappendr r^)rrZrRrSrTrUrVrWrXlinesZtextsrGidsiliner rrrrYs(   z;SingleSentenceClassificationProcessor.add_examples_from_csvc CsB|dk r4t|t|kr4tdt|dt||dk rht|t|krhtdt|dt||dkr~dgt|}|dkrdgt|}g}t}t|||D]N\}} } t|ttfr| dkr|\} } n|} || |t | | d| dq|r||_ n |j ||r&t||_ ntt|j  ||_ |j S)Nz(Text and labels have mismatched lengths z and z%Text and ids have mismatched lengths )r r r r)r4 ValueErrorsetziprMtupler=addrcr rHextendrGunion) rr_rGrerWrXrHZ added_labelsZtext_or_text_and_labelrr textrrrr^s2    z2SingleSentenceClassificationProcessor.add_examplesTcs|dkr|j}ddt|jD}g}t|jD]H\} } | ddkrTtd| |j| jdt||jd} | | q0t d d |D} gtt ||jD]\} \} } | ddkrtd | d t |j|rd ndgt | } | t | }|r&|g|| } |rdnd g|| } n&| |g|} | |r@dnd g|} t | | krrt dt | d| t | | krt dt | d| |jdkr|| j}n"|jdkrt| j}n t |j| dkrZ|jrZtdtd| jtdddd| Dtdddd| Dtd| jd|d t| | |dq|dkr~S|dkrtstd ddl}fd!d"}|jj||j|jd#|jf|dg|dgd#|gf}|S|d$krts td%ddl}dd&l m!}|j"d'dD|j#d(}|j"d)dD|j#d(}|jdkr||j"d*dD|j#d(}n&|jdkr|j"d+dD|jd(}||||}|St d,dS)-a Convert examples in a list of `InputFeatures` Args: tokenizer: Instance of a tokenizer that will tokenize the examples max_length: Maximum example length pad_on_left: If set to `True`, the examples will be padded on the left rather than on the right (default) pad_token: Padding token mask_padding_with_zero: If set to `True`, the attention mask will be filled by `1` for actual values and by `0` for padded values. If set to `False`, inverts it (`1` for padded values, `0` for actual values) Returns: If the `examples` input is a `tf.data.Dataset`, will return a `tf.data.Dataset` containing the task-specific features. If the input is a list of `InputExamples`, will return a list of task-specific `InputFeatures` which can be fed to the model. NcSsi|]\}}||qSrr).0rfrrrr szFSingleSentenceClassificationProcessor.get_features..i'rzTokenizing example T)Zadd_special_tokens max_lengthcss|]}t|VqdSrF)r4)rpr$rrr szESingleSentenceClassificationProcessor.get_features..zWriting example /r3zError with input length z vs rEZ regressionz*** Example ***zguid: z input_ids:  cSsg|] }t|qSrr rpxrrr 6szFSingleSentenceClassificationProcessor.get_features..zattention_mask: cSsg|] }t|qSrrwrxrrrrz7szlabel: z (id = )r$r%rtfz?return_tensors set to 'tf' but TensorFlow 2.0 can't be importedc3s$D]}|j|jd|jfVqdS)Nr$r%r|)exfeaturesrrgenCsz?SingleSentenceClassificationProcessor.get_features..genr~ptz8return_tensors set to 'pt' but PyTorch can't be imported) TensorDatasetcSsg|] }|jqSr)r$rprArrrrzSs)ZdtypecSsg|] }|jqSr)r%rrrrrzTscSsg|] }|jqSrrrrrrrzVscSsg|] }|jqSrrrrrrrzXsz,return_tensors should be one of 'tf' or 'pt')$max_lenrbrGrHloggerinfoencoder minrcmaxrjr4rhrIrr(rJr joinr#r RuntimeErrorZ tensorflowdataZDatasetZfrom_generatorZint32Zint64Z TensorShapertorchZtorch.utils.datarZtensorlong)r tokenizerrrZ pad_on_leftZ pad_tokenZmask_padding_with_zeroZreturn_tensorsZ label_mapZ all_input_idsZex_indexr5r$Z batch_lengthr%Zpadding_lengthrr}rZdatasetrrZall_attention_maskZ all_labelsrrr get_featuress                "     z2SingleSentenceClassificationProcessor.get_features)NNrEF)rQrr3NF)N)rQrr3NFFF)NNFF)NFrTN) rrrrrKrLrPrCr]r`rYr^rrrrrrD}s@     (rD)r>rrrtypingrrrutilsrrr Z get_loggerrrr r#r)rDrrrrs  -