U 4Af&@sbddlmZmZmZddlmZddlmZddlm Z e e Z ddd d Z Gd d d eZd S))ListOptionalTuple)PreTrainedTokenizerFast)logging)HerbertTokenizerz vocab.jsonz merges.txtztokenizer.json) vocab_file merges_filetokenizer_filecseZdZdZeZeZdfdd Zde e e e e e e d d d Z de e e e e e e e dfdd Zde e e e e e e d ddZdee eeedddZZS)HerbertTokenizerFastam Construct a "Fast" BPE tokenizer for HerBERT (backed by HuggingFace's *tokenizers* library). Peculiarities: - uses BERT's pre-tokenizer: BertPreTokenizer splits tokens on spaces, and also on punctuation. Each occurrence of a punctuation character will be treated separately. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the methods. Users should refer to the superclass for more information regarding methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. Nc s(tj||f||||||d| dS)N)r cls_token unk_token pad_token mask_token sep_token)super__init__) selfr r r rrrrrkwargs __class__Y/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/herbert/tokenization_herbert_fast.pyr2s  zHerbertTokenizerFast.__init__) token_ids_0 token_ids_1returncCs8|jg}|jg}|dkr$|||S|||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An HerBERT, like BERT sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N) cls_token_id sep_token_id)rr r!clsseprrr build_inputs_with_special_tokensJs  z5HerbertTokenizerFast.build_inputs_with_special_tokensF)r r!already_has_special_tokensr"csf|rtj||ddS|dkr8dgdgt|dgSdgdgt|dgdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)r r!r(Nrr)rget_special_tokens_masklen)rr r!r(rrrr)esz,HerbertTokenizerFast.get_special_tokens_maskcCsV|jg}|jg}|dkr.t|||dgSt|||dgt||dgS)a{ Create a mask from the two sequences passed to be used in a sequence-pair classification task. HerBERT, like BERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). Nrr)r$r#r*)rr r!r&r%rrr$create_token_type_ids_from_sequencess z9HerbertTokenizerFast.create_token_type_ids_from_sequences)save_directoryfilename_prefixr"cCs|jjj||d}t|S)N)name) _tokenizermodelsavetuple)rr,r-filesrrrsave_vocabularysz$HerbertTokenizerFast.save_vocabulary)NNNrrrrr)N)NF)N)N)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESZvocab_files_namesr Zslow_tokenizer_classrrintrr'boolr)r+strrr4 __classcell__rrrrr sB     r N)typingrrrZtokenization_utils_fastrutilsrZtokenization_herbertr Z get_loggerr5loggerr9r rrrrs