U
4Af, @ s d Z ddlZddlZddlmZmZmZmZmZ ddl Z
ddl m Z m
Z
mZ erle
r^ddlZe rlddlZddlmZ ddlmZ ddlmZ d d
lmZ eeZ d d d
dZ G dd deZ dS )z$Tokenization classes for OpenAI GPT. N)
TYPE_CHECKINGListOptionalTupleUnion )is_tf_availableis_torch_availablelogging)pre_tokenizers)
BatchEncoding)PreTrainedTokenizerFast )CodeGenTokenizerz
vocab.jsonz
merges.txtztokenizer.json)
vocab_file merges_filetokenizer_filec s e Zd ZdZeZddgZeZd fdd Z e
d fd
d Z e
d fd d
Z d e
e ee
e e
e dddZd eee ee dddZd eee
e dddf eeee
e ed fdd
Zdd Z ZS ) CodeGenTokenizerFastas
Construct a "fast" CodeGen tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level
Byte-Pair-Encoding.
This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will
be encoded differently whether it is at the beginning of the sentence (without space) or not:
```python
>>> from transformers import CodeGenTokenizerFast
>>> tokenizer = CodeGenTokenizerFast.from_pretrained("Salesforce/codegen-350M-mono")
>>> tokenizer("Hello world")["input_ids"]
[15496, 995]
>>> tokenizer(" Hello world")["input_ids"]
[18435, 995]
```
You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer, but since
the model was not pretrained this way, it might yield a decrease in performance.
When used with `is_split_into_words=True`, this tokenizer needs to be instantiated with `add_prefix_space=True`.
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`, *optional*):
Path to the vocabulary file.
merges_file (`str`, *optional*):
Path to the merges file.
tokenizer_file (`str`, *optional*):
Path to [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that
contains everything needed to load the tokenizer.
unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
bos_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
The beginning of sequence token.
eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
The end of sequence token.
add_prefix_space (`bool`, *optional*, defaults to `False`):
Whether or not to add an initial space to the input. This allows to treat the leading word just as any
other word. (CodeGen tokenizer detect beginning of words by the preceding space).
return_token_type_ids (`bool`, *optional*, defaults to `False`):
Whether to return token type IDs.
Z input_idsZattention_maskN
<|endoftext|>Fc
s || _ | j r| jd t j||f||||||d| | ddrj| dd}
td|
d|
d t| j j
} | d
||krt
t| d } || d
< | f | | j _
|| _d S ) NZtoken_type_ids)r unk_token bos_token eos_tokenadd_prefix_spacereturn_token_type_idsZ
add_bos_tokenFZ name_or_path zCurrenty GPT2's fast tokenizer does NOT support adding a BOS token. Instead you should use GPT2's slow tokenizer class `CodeGenTokenizer` as follows:
`CodeGenTokenizer.from_pretrained('z'')`
or
`AutoTokenizer.from_pretrained('z', use_fast=False)`
This issue will be fixed soon, see: https://github.com/huggingface/tokenizers/pull/1005. so that the fast tokenizer works correctly.r type)r model_input_namesappendsuper__init__pop
ValueErrorjsonloadsZbackend_tokenizerZ
pre_tokenizer __getstate__getgetattrr r )
selfr r r r r r r r kwargsZmodel_idZ
pre_tok_stateZ
pre_tok_class __class__ Y/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/codegen/tokenization_codegen_fast.pyr e s6 z CodeGenTokenizerFast.__init__)returnc s8 | dd}| js*|r*td| jj dt j||S Nis_split_into_wordsFzYou need to instantiate z? with add_prefix_space=True to use it with pretokenized inputs.)r% r AssertionErrorr* __name__r _batch_encode_plusr' argsr( r/ r) r+ r, r2 s
z'CodeGenTokenizerFast._batch_encode_plusc s8 | dd}| js*|r*td| jj dt j||S r. )r% r r0 r* r1 r _encode_plusr3 r) r+ r, r5 s
z!CodeGenTokenizerFast._encode_plus) token_ids_0 token_ids_1r- c C sr | j dk r| j gng }| j dk r(| jgng }|dkrJt|| | dg S t|| | dg t|| dg S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. A sequence
pair mask has the following format:
```
0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence | second sequence |
```
If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).
Nr r )Z sep_token_idZ cls_token_idlen)r' r6 r7 sepclsr+ r+ r, $create_token_type_ids_from_sequences s
z9CodeGenTokenizerFast.create_token_type_ids_from_sequences)save_directoryfilename_prefixr- c C s | j jj||d}t|S )N)name)
_tokenizermodelsavetuple)r' r< r= filesr+ r+ r, save_vocabulary s z$CodeGenTokenizerFast.save_vocabularyz
np.ndarrayz torch.Tensorz tf.Tensor) token_idsskip_special_tokens clean_up_tokenization_spacestruncate_before_patternr- c s> t jf |||d|}|dk r:t|dkr:| ||}|S )a
Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special
tokens and clean up tokenization spaces.
Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`.
Args:
token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`):
List of tokenized input ids. Can be obtained using the `__call__` method.
skip_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not to remove special tokens in the decoding.
clean_up_tokenization_spaces (`bool`, *optional*):
Whether or not to clean up the tokenization spaces. If `None`, will default to
`self.clean_up_tokenization_spaces` (available in the `tokenizer_config`).
truncate_before_pattern (`List[str]`, *optional*, defaults to `None`):
A list of regular expression strings that will be used to truncate the returned string. This can be
used to remove extra pieces of code (e.g. truncate if observing a comment symbol "#" at the beginning
of a new line). An example pattern could be `["^#", re.escape("<|endoftext|>"), "^'''", "
"]`.
kwargs (additional keyword arguments, *optional*):
Will be passed to the underlying model specific decode method.
Returns:
`str`: The decoded sentence.
)rE rF rG Nr )r decoder8 truncate)r' rE rF rG rH r( Z decoded_textr) r+ r, rI s ! zCodeGenTokenizerFast.decodec s dd dd |D }t td tj}t|dkrJ d |d t td tj}t|dkr~ d |d dd d fd
d|D D }t|dkr d t| S S d S ) Nc S s | | |}|r| S dS )N)searchstart)stringpattern start_posmr+ r+ r, find_re s z.CodeGenTokenizerFast.truncate..find_rec S s g | ]}t |t jqS r+ )recompile MULTILINE).0rO r+ r+ r,
s z1CodeGenTokenizerFast.truncate..z^printr z^defr c S s g | ]}|d kr|qS )rK r+ )rV posr+ r+ r, rW s c s g | ]} |qS r+ r+ )rV Zterminal
completionrR rP r+ r, rW
s )listrS finditerrU r8 rM min)r' rZ rH Z terminalsZprintsZdefsZ
terminals_posr+ rY r, rJ s z CodeGenTokenizerFast.truncate)NNNr r r FF)N)N)FNN)r1
__module__ __qualname____doc__VOCAB_FILES_NAMESZvocab_files_namesr r Zslow_tokenizer_classr r r2 r5 r intr r; strr rD r boolrI rJ
__classcell__r+ r+ r) r, r , sB 4 /
-r ) r` r" rS typingr r r r r Znumpynputilsr r r
ZtorchZ
tensorflowtfZ
tokenizersr Ztokenization_utils_baser Ztokenization_utils_fastr
Ztokenization_codegenr Z
get_loggerr1 loggerra r r+ r+ r+ r, s"