U 4Af@X@sdZddlmZmZmZmZddlmZddlm Z ddl m Z ddl m Z mZmZmZddlmZmZervdd lZerdd lZd Zdd d ZddZddZddZddZGddde Zd S)z Processor class for IDEFICS. )CallableListOptionalUnion)urlparse) BatchFeature)ProcessorMixin) BatchEncodingPaddingStrategy TextInputTruncationStrategy)is_tf_availableis_torch_availableNcCs|dkr8|dkrd|||k<n|dkr8t||kd|}|dkrt|dk}d||<tjjj||d}d||ddf<nP|dkrt|d}t|d|}tj||d}t|d}t|t||}|S)Nrpttfr num_classes)depth) rwheretorchnnZ functionalZone_hotequalZ expand_dimsZ zeros_like)Zincremental_maskreturn_tensorsrZ negativesZ attn_maskZnegatives_expandedrR/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/idefics/processing_idefics.py$incremental_to_binary_attention_mask&s"  rcCs(|dkrt||S|dkr$t||SdS)Nrr),image_attention_mask_for_packed_input_ids_pt,image_attention_mask_for_packed_input_ids_tf) input_ids tokenizerrrrr)image_attention_mask_for_packed_input_ids@s r#c Csztj|dd}tj|dd}|t}|j}t|dD]l}d}d}t||D]R\} } | |kr||d7}|||| <d}n |||| <|rd||| <| |krRd}qRq:t|dD]}d}d}t||ddddD]^} ||| } | |kr|d7}|||| <d}n |||| <| |kr*d}|rd||| <q||dk} ||| |8<||| d9<q||fS)Nr)Z fill_valuerFT)rZ full_likeconvert_tokens_to_ids IMAGE_TOKEN eos_token_idrangesize enumerate) r!r"image_attention_masknext_image_attention_maskimage_token_id eod_token_id batch_idxcountseen_eodidxtoken_idZnon_negative_indicesrrrrGsF          rcCs.|t}|j}t|d}tt|d}tt|d}t|D]}d}d} t|d} t| dddD]} ||| f} | |kr|d7}|| gg} |g}t|| |}t|| |}n2| |kr| sd} d}|| gg} |g}t|| |}| rt| |krt|| gg} dg}t|| |}qtqJ||fS)NrrFr$T) r%r&r'rshapefillr(Znumpytensor_scatter_nd_update)r!r"r-r.Z batch_sizer+r,r/r0r1Z seq_lengthr2r3indicesupdatesrrrr vs8       r cCs$d|kr dSt|}t|j|jgS)zChecks if the passed string contains a valid url and nothing else. e.g. if space is included it's immediately invalidated the url F)rallschemenetloc)stringresultrrris_urlsr?cseZdZdZddgZddgZdZdZdfd d Zde e e e e e fe e e efe e e efeeeedddZddZddZeddZZS)IdeficsProcessora Constructs a IDEFICS processor which wraps a LLama tokenizer and IDEFICS image processor into a single processor. [`IdeficsProcessor`] offers all the functionalities of [`IdeficsImageProcessor`] and [`LlamaTokenizerFast`]. See the docstring of [`~IdeficsProcessor.__call__`] and [`~IdeficsProcessor.decode`] for more information. Args: image_processor (`IdeficsImageProcessor`): An instance of [`IdeficsImageProcessor`]. The image processor is a required input. tokenizer (`LlamaTokenizerFast`): An instance of [`LlamaTokenizerFast`]. The tokenizer is a required input. image_size (`int`, *optional*, defaults to 224): Image size (assuming a square image) image_processorr" image_sizeadd_end_of_utterance_tokenZIdeficsImageProcessorZLlamaTokenizerFastNc s||dkrtd|dkr tdt|||j|_|t|_|jj|jj |jj f|_ d|j j dgkrrdnd|_dS)Nz)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.Zadditional_special_tokensTF) ValueErrorsuper__init__rAZcurrent_processorr%r&r-Zimage_num_channelsrBdefault_image_dimsr"Zspecial_tokens_mapget1tokenizer_was_trained_with_end_of_utterance_token)selfrAr"rBrCkwargs __class__rrrHs zIdeficsProcessor.__init__longestFr)promptspadding truncation max_length transformreturnc -s:|dkr|j}tdd|Ds&|g}ddd} fdd} g} g} |D] }|jj}g}d }d }t|D]\}}|d kr|sd nd }t|tr|d }t|r|j |}|| |7}| |d }n|r|r|| 7}||7}d }qp|| |7}| |d }qp|r||jj 7}|d kr4t d ||j ||| d}| || |qL|j| d |||d}|d}|d}tdd| D}td|}tdd| Dd k}g}g}g}t||| D]\}}} |}!|!|j}"t|"|}#| d|#}$t|$d kr| dkrBtj|f|$dd}%|$|%d|$d <nr| dkrt|$dd}&tj|g|&gd d}'tj|'|$jd}%t|$d }(tt|(d})|$}*t|%|)|*}%n:| dkrtj|f|j}%n| dkrt|f|j}%| |%| dkr&| t |!| t |n*| dkr| tj!|!tj"d| |q| dkr~t#|}t#|}t#|}n(| dkrt#|}t#|}t#|}|rt$||j| \}+},t%|+| |d}+nX| dkrtj|jd |jddtj&d}+n,| dkr&tj|jd |jddftj&d}+t'||||+ddS)aThis method takes batched or non-batched prompts made of text and images and converts them into prompts that the model was trained on and prepares the image pixel values for the model to process. Args: prompts (`Union[List[TextInput], [List[List[TextInput]]]]`): either a single prompt or a batched list of prompts - see the detailed description immediately after the end of the arguments doc section. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `"longest"`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding. This will raise an error if the input sequences are of different lengths. Note: Unlike most processors, which set padding=`False` by default, `IdeficsProcessor` sets `padding="longest"` by default. See https://github.com/huggingface/transformers/pull/29449#pullrequestreview-1925576061 for why. max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). truncation (`bool`, *optional*): Activates truncation to cut input sequences longer than `max_length` to `max_length`. transform (`Callable`, *optional*): A custom transform function that accepts a single image can be passed for training. For example, `torchvision.Compose` can be used to compose multiple functions. If `None` a preset inference-specific set of transforms will be applied to the images add_eos_token (`bool`, *optional*, defaults to `False`): Adds `eos_token` at the end of the final prompt if True` add_end_of_utterance_token (`bool`, *optional*) Whether to automatically add `` after each prompt's text input (unless followed by an image). If `None` the tokenizer will be checked instead and if this token is found in `additional_special_tokens` then the value will be `True`. debug (`bool`, *optional*, defaults to `False`): `True` value will help debug prompt generation by dumping useful information return_tensors (`str` or `TensorType`, *optional*, defaults to `TensorType.PYTORCH`): The type of tensors to return. Can be one of: - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. Returns: a dict with entries: `input_ids`, `attention_mask`, `pixel_values`, `image_attention_mask` which can be directly passed to `model.generate` Detailed explanation: Each entry in `prompts` is either a text to be passed as is or an image that will be processed. An image can be either an image object (`PIL.Image`) or a url from which the image can be retrieved. When the processor encounters an image it'll inject `` entry into the prompt. Example: ```python checkpoint = "HuggingFaceM4/idefics-9b" processor = AutoProcessor.from_pretrained(checkpoint) url = "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg" img = processor.image_processor.fetch_images([url])[0] prompts = [ "User:", img, "Describe this image. Assistant: An image of two kittens in grass. ", "User:", "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg", "Describe this image. Assistant:", ] inputs = processor(prompts, return_tensors="pt") generated_ids = model.generate(**inputs, max_length=100) generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] ``` In this example the `prompts` will be converted into: ``` User:Describe this image. Assistant: An image of two kittens in grass. User:Describe this image. Assistant:' ``` and the two images will be massaged using [`IdeficsImageProcessor.__call__`] method and placed inside the `pixel_values` dict entry of the return value. This example also examplifies that images can be passed as objects or as text urls. It can be seen that the first image is passed as object and the second one as a url. To do training do: ```python image_transform = transforms.Compose( [ transforms.RandomResizedCrop( (w, h), scale=(0.9, 1.0), interpolation=transforms.InterpolationMode.BICUBIC ), transforms.ToTensor(), transforms.Normalize(mean=self.image_mean, std=self.image_std), ] ) inputs = processor(prompts, transform=image_transform, return_tensors="pt") ``` In order to help debug prompt generation enable `debug=True` which will show you what's happening. Ncss|]}t|tVqdSN) isinstancelist).0irrr Esz,IdeficsProcessor.__call__..zrrEcs|r SSdSrWr)last_was_imageZ fake_tokenZ image_tokenrr image_tokensLsz/IdeficsProcessor.__call__..image_tokensFrTr9z full_text=)rUr)textZadd_special_tokensrRrSrTr!attention_maskcss|]}t|VqdSrWlenrZxrrrr\sr$css|]}t|VqdSrWrbrdrrrr\srr)Zaxis)dtype)rr$r)r!raZ pixel_valuesr+)data)(rKanyr"Z bos_tokenr*rXstrstripr?rAZ fetch_imagesappendZ eos_tokenprintmaxsumzipr0r-minrcrzerosr)rr4concatrfZreshaper(r6rIZtensorZconvert_to_tensorZint32stackr#rboolr)-rLrQrRrSrTrUZ add_eos_tokenrCdebugrZend_of_utterance_tokenr_Z all_promptsZ all_imagessampleZ full_textZ image_objectsr]Z last_was_textr[itemimageZ text_encodingZ all_textsZall_attention_masksZmax_num_imagesZat_least_one_imageZoutput_input_idsZ output_imagesZoutput_attention_masksr`raZimagesZpadded_input_idsZ image_countZlocal_max_num_imagesZcurrent_imagesZpadded_image_tensorZ image_shapeZ padded_shapeZ num_imagesr7r8r+_rr^r__call__sx                                   zIdeficsProcessor.__call__cOs|jj||S)z This method forwards all its arguments to LlamaTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r" batch_decoderLargsrMrrrr{szIdeficsProcessor.batch_decodecOs|jj||S)z This method forwards all its arguments to LlamaTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )r"decoder|rrrr~szIdeficsProcessor.decodecCs"|jj}|jj}tt||SrW)r"model_input_namesrArYdictfromkeys)rLZtokenizer_input_namesZimage_processor_input_namesrrrrsz"IdeficsProcessor.model_input_names)NrDN)rPNNNFNFr)__name__ __module__ __qualname____doc__ attributesZ valid_kwargsZimage_processor_classZtokenizer_classrHrrr rtrir r rintrr rzr{r~propertyr __classcell__rrrNrr@s8   r@)r)rtypingrrrr urllib.parserZfeature_extraction_utilsrZprocessing_utilsr Ztokenization_utils_baser r r r utilsrrrZ tensorflowrr&rr#rr r?r@rrrrs"    /!