U 4Af]9@sfdZddlmZddlmZddlmZmZmZm Z ddl Z ddl m Z ddlmZdd lmZdd lmZmZmZmZmZdd lmZmZdd lmZmZmZmZd dl m!Z!d dl"m#Z#d dl$m%Z%e&e'Z(dZ)eGdddeZ*eGdddeZ+dBddZ,ddZ-dCddZ.gfddZ/Gdd d e j0j1j2Z3Gd!d"d"e j0j1j4Z5dDd#d$Z6dEd%d&Z7Gd'd(d(e j0j1j4Z8Gd)d*d*e j0j1j4Z9d+d,Z:d-d.Z;Gd/d0d0e j0j1j4ZGd5d6d6e j0j1j4Z?d7Z@ed8e@Gd9d:d:e ZAd;ZBed8e@eGdd?d?eAZDGd@dAdAe eZEdS)FzTF 2.0 Idefics model.) annotations) dataclass)ListOptionalTupleUnionN)TFPreTrainedModel)get_tf_activation) ModelOutput)TFCausalLanguageModelingLossTFModelInputTypekeras_serializable shape_list unpack_inputs)invert_attention_maskscaled_dot_product_attention)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) IdeficsConfig)TFIdeficsPerceiverResampler)TFIdeficsVisionTransformerrc@sNeZdZUdZdZded<dZded<dZded<dZded <dZ ded <dS) TFIdeficsBaseModelOutputWithPasta Base class for Idefics model's outputs that may also contain a past key/values (to speed up sequential decoding). Args: last_hidden_state (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. past_key_values (`tuple(tuple(tf.Tensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(tf.Tensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and optionally if `config.is_encoder_decoder=True` 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(tf.Tensor)`, *optional*): Tuple of `tf.Tensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver N tf.Tensorlast_hidden_statez!Optional[Tuple[Tuple[tf.Tensor]]]past_key_valuesOptional[Tuple[tf.Tensor]] hidden_states attentionsimage_hidden_states) __name__ __module__ __qualname____doc__r__annotations__rr r!r"r(r(S/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/idefics/modeling_tf_idefics.pyr8s $    rc@sZeZdZUdZdZded<dZded<dZded<dZd ed <dZ d ed <dZ d ed <dS) TFIdeficsCausalLMOutputWithPasta Base class for Idefics causal language model (or autoregressive) outputs. Args: loss (`tf.Tensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`tf.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(tf.Tensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(tf.Tensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(tf.Tensor)`, *optional*): Tuple of `tf.Tensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver NOptional[tf.Tensor]lossrlogitsOptional[List[tf.Tensor]]rrr r!r") r#r$r%r&r,r'r-rr r!r"r(r(r(r)r*es      r*Fc Ks:tttt|d|dg}t||}|dd|d<|dd|d<|dd|d<|dd|d<d|kr|d}t|||d<|dk rt|||d<|ddk rt|d||d<|ddk rt|d||d<nF|ddk rt|d||d<n"|ddk r2t|d||d<||fS) Nr pixel_valuesimage_encoder_embeddingsperceiver_embeddingsimage_attention_masktoken_type_idsattention_mask)tfreshaperepeatrangeshapegatherget) input_idsZ expand_sizeis_encoder_decoderr5Zencoder_outputs model_kwargsZexpanded_return_idxr4r(r(r)expand_inputs_for_generations.&    r@cCsd|kr|j|d<nd|d<d|krT|d}tj||dddddfgdd|d<d|kr|d}tj|t|dddddfgdd|d<d|kr|d}|dddddf}||d<|j|d<|S) Nrr4r/.axisr5r3r")rr6concatZ ones_liker")outputsr?r4r5r3Z last_maskr(r(r)"update_model_kwargs_for_generations$ (  rEc Ks|dd}|dk rD|ddddf}|dk rD|ddddf}|dd}|dd}|dk r|dkrtjjtj|tjdddd}t|dkd|}|dk r|ddddf}|d d}|d d}|d d}|d d} |d d} |||d||||||| | d S)Nr4r/r5 position_idsdtyperArrr0r1r2r3interpolate_pos_encodingF use_cache) r=rrJrFr5r4r0r1r2r3rI)r<r6mathcumsumcastZint64where) r=rkwargsr4r5rFr0r1r2r3rIr(r(r)prepare_inputs_for_generations:         rPcs|tjjjtjjjtjjjdfdd|D}t|dsDd|_|S|jD],|rptfdd|Drpd_qJd_qJ|S) N)Z LayerNormDense Embeddingcsg|] }|qSr(r().0m)mappingr(r) sz freeze_model..layersFc3s|]}t|VqdSN) isinstance)rSt)layerr(r) szfreeze_model..T) r6kerasrWZLayerNormalizationrQrRhasattr trainableany)modelmodule_exceptionsZmodule_exceptions_mappedr()r[rUr) freeze_models  rccsDeZdZdZddddfdd Zfd d Zd d d dZZS)TFIdeficsDecoupledEmbeddinga Implements a decoupling of parameters to allow freezing (or not) a subset of the embeddings. In practise, the regular `weight` can be trained or frozen (i.e. `partially_freeze=True`), and if `num_additional_embeddings` > 0, then it will create `num_additional_embeddings` additional parameters that are always trained. If `num_additional_embeddings=0`, then the module defaults back to the regular behavior of `tf.keras.layers.Embedding`. FNOptional[bool]None)partially_freezereturnc s^tjf|||d|||_||_||_|r6d|_|jdkrZtjjj |j||dd|_ dS)a Args: num_embeddings (`int`): Size of the dictionary of embeddings num_additional_embeddings (`int`): Number of additional embeddings. Only useful when you `partially_freeze=True`. embedding_dim (`int`): The size of each embedding vector partially_freeze: (`bool`, *optional*, defaults to `False`): If `True`, the regular `weight` will be frozen. `additional_weight` is never frozen. Note: there are a lot of other parameters to initialize a standard `tf.keras.layers.Embedding` such as `mask_zero`, `input_length` or `embeddings_initializer`. We are not supporting these. ) input_dim output_dimrHFradditional_embedding)rirjrHnameN) super__init__num_embeddingsnum_additional_embeddingsrgr_r6r]rWrRrk)selfrorp embedding_dimrgrHrO __class__r(r)rns& z$TFIdeficsDecoupledEmbedding.__init__cs|jdkrt|St|}t||jk}t||}|||j}t ||tj t |d|j d}t|}t |||}|S)a we have 2 embeddings, with different indices - one pretrained self.weight and another self.additional_embedding.weight that is being trained. in order to make a lookup of the input ids, we: 1. find out the indices of the entries belonging to the 2nd embedding 2. extract those values while subtracting the size of the first embedding (num_embeddings), since the 2nd embedding starts from 0 and not num_embeddings 3. perform the 2nd embedding lookup 4. now we handle the 1st embedding, we overwrite indices belonging to the 2nd embedding with a padding index 5. perform the 1st embedding lookup 6. now we overwrite the values in the 1st embedding lookup with the values of the 2nd embedding lookup note: for the 1st embedding lookup we could have looked up only the low indices and not do the padding, but then we have to create a new tensor and populate it with 2 tensors that are spread out across various indices - i.e. not a simple concat - I haven't benchmarked the complex case if it's any faster, given that seqlens are usually relatively short it's probably not faster or if faster not by much - but might be a good idea to measure. rrG) rprmcallr6identityrNroZ gather_ndrkZtensor_scatter_nd_updatezerosr:rH)rqr=Zadditional_vocab_indicesZinput_ids_additional_vocabZadditional_embeddingsZ full_vectorrsr(r)ru@s     z TFIdeficsDecoupledEmbedding.callstrrhcCsd|j|j|j|jS)NzVnum_embeddings={}, num_additional_embeddings={}, embedding_dim={}, partially_freeze={})formatrorprjrgrqr(r(r) extra_reprls z&TFIdeficsDecoupledEmbedding.extra_repr)FN)r#r$r%r&rnrur| __classcell__r(r(rsr)rd s  , ,rdcsreZdZdZddddddddfdd Zd d d d d ZfddZddddZeddZ dddZ Z S)TFIdeficsDecoupledLineara Implements a decoupling of parameters to allow freezing (or not) a subset of the parameters. In practise, the regular `weight` can be trained or frozen (i.e. `partially_freeze=True`), and if `out_additional_features` > 0, then it will create `out_additional_features * in_features` additional parameters that are always trained. If `out_additional_features=0`, then the module defaults back to the regular behavior of `tf.keras.layers.Dense`. rTintboolrf) in_features out_featuresout_additional_featuresbiasrgrhc sNtjf|||_||_||_||_||_|dkrJtjj j ||dd|_ dS)aS out_additional_features: int. Number of additional trainable dimensions. Only makes sense when `partially_freeze=True`. partially_freeze: bool. If True, the regular `weight` will be frozen and extra parameters (if any) will be trainable. If False, default to the regular behavior of tf.keras.layers.Dense. r additional_fc)Zunitsuse_biasrlN) rmrnrrgrrrr6r]rWrQr)rqrrrrrgrOrsr(r)rn}sz!TFIdeficsDecoupledLinear.__init__r)inputsrhcCsXtjj||jdd}|jdk r.tj||j}|jdkrT||}tj ||gdd}|S)NT)abZ transpose_brr/rA) r6ZlinalgmatmulweightrnnZbias_addrrrC)rqroutputZadditional_featuresr(r(r)rus   zTFIdeficsDecoupledLinear.callcs2t}||j|j|j|jdk |jd|S)N)rrrrrg)rm get_configupdaterrrrrgrqconfigrsr(r)rs  z#TFIdeficsDecoupledLinear.get_configrxrycCs d|j|j|j|jdk |jS)z=Overwriting `nn.Linear.extra_repr` to include new parameters.zYin_features={}, out_features={}, out_additional_features={}, bias={}, partially_freeze={}N)rzrrrrrgr{r(r(r)r|sz#TFIdeficsDecoupledLinear.extra_reprcCs |f|SrXr()clsrr(r(r) from_configsz$TFIdeficsDecoupledLinear.from_configNc Cs|jr dSd|_|j|j|jf|j dd|_|jrP|j|jf|j dd|_nd|_t|dddk rt |j j |j |jW5QRXdS)NTr)r:r_rlrr)built add_weightrrrgrrrgetattrr6 name_scoperrlbuildrq input_shaper(r(r)rs zTFIdeficsDecoupledLinear.build)rTT)N) r#r$r%r&rnrurr| classmethodrrr}r(r(rsr)r~us    r~cCs|\}}t||ftj|j}t|}t|dddf|dddfkd|}|dkr~tjtj||f|d|gdd}|dkrt |d}t |d}t ||dddg}n,t |ddddddf|d|||f}|S)zm Make causal mask used for bi-directional self-attention, supporting both static and dynamic shapes. NrrGr/rAr) r6fillZdtypesZas_dtypeminr9rNrCrw expand_dimsZtile broadcast_to)Zinput_ids_shaperHpast_key_values_lengthbsztgt_lenmaskZ mask_condr(r(r)_make_causal_masks *   ,rcCs|t|\}}|dk r|n|}tt|dd}t||d||g}dt||}tt|ttjt|tjj d|S)z_ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nr?)Zdimsvalue) rr6rrrMrNrrfloat32r)rrHrrZsrc_lenZ expanded_maskZ inverted_maskr(r(r) _expand_masks  rcs2eZdZdfdd ZfddZddZZS) TFIdeficsRMSNormư>c stjf|||_||_dS)z? TFIdeficsRMSNorm is equivalent to T5LayerNorm N)rmrn hidden_sizevariance_epsilon)rqrepsrOrsr(r)rnszTFIdeficsRMSNorm.__init__cs6|jr dSd|_|jd|jgdd|_t|dS)NTrones)rlr: initializer)rrrrrmrrrsr(r)rs zTFIdeficsRMSNorm.buildcCshtjjtjt|tjddd}|tj||j}|jj tj tj fkr^t||jj }|j|S)Nr/T)rBZkeepdims) r6rKZ reduce_meanZsquarerMrZrsqrtrrrHZfloat16Zbfloat16)rqr Zvariancer(r(r)ru s $zTFIdeficsRMSNorm.call)rr#r$r%rnrrur}r(r(rsr)rs rcs0eZdZd fdd ZddZd dd ZZS) TFIdeficsEmbedding'c sRtjf|||_||_||_td|jtjd|jdtjd|j|_ dS)Nrr)startlimitdeltarH) rmrndimmax_position_embeddingsbaser6Zconstantr9rinv_freq)rqrrrrOrsr(r)rns$zTFIdeficsEmbedding.__init__cCsHtj||jjd}td||j}tj||fdd}t|t|fS)NrGz i, j -> ijr/rA)r6r9rrHZeinsumrCcossin)rqseq_lenrZZfreqsZembr(r(r)_compute_cos_sin!sz#TFIdeficsEmbedding._compute_cos_sinNcCs |dkrt|d}|j|dS)Nr)r)rr)rqxrr(r(r)ru(s zTFIdeficsEmbedding.call)rr)N)r#r$r%rnrrur}r(r(rsr)rs rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr/rrA)r:r6rC)rx1Zx2r(r(r) rotate_half/srcCs`t||}t||}t|d}t|d}||t||}||t||}||fS)Nr)r6r;rr)qkrrrFZq_embedZk_embedr(r(r)apply_rotary_pos_emb6s    rcs8eZdZddddfdd ZddZd d d ZZS) TFIdeficsMLPrrx)rintermediate_size hidden_actc sjtjf|tjjj|ddd|_tjjj|ddd|_tjjj|ddd|_t ||_ ||_ ||_ dS)NF gate_projrrl down_projup_proj) rmrnr6r]rWrQrrrr act_fnrr)rqrrrrOrsr(r)rnAs zTFIdeficsMLP.__init__cCs ||||||SrX)rrrr)rqrr(r(r)ruPszTFIdeficsMLP.callNc Cs|jr dSd|_t|dddk rHt|jj|j|jW5QRXt|dddk rt|jj|j|j W5QRXt|dddk rt|j j|j |jW5QRXdS)NTrrr) rrr6rrrlrrrrrrr(r(r)rSszTFIdeficsMLP.build)Nr#r$r%rnrurr}r(r(rsr)r@src sleZdZdZdddddddd fd d Zd ddd ddZdd ddddddddddZdddZZS)TFIdeficsAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFNrfloatrr)r num_headsdropoutis_cross_attentionrqk_layer_normsc stjf|||_||_|||_||_||_d|_|j||jkr^td|jd|d||_ t j j j ||jddd|_t j j j ||jddd|_t j j j ||jdd d|_t j j j |dd d|_t|jd d |_||_|jrt|j|jd d|_t|j|jdd|_dS)NTz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).Fq_projrk_projv_projo_proj rotary_embrl q_layer_normrrl k_layer_norm)rmrnrrhead_dimrr is_causal ValueErrorrr6r]rWrQrrrrrrrr rms_norm_epsrr)rqrrrrrrrOrsr(r)rnesJ  zTFIdeficsAttention.__init__r)tensorrrcCs*tjt||||j|jfddddgdS)Nrrrrperm)r6 transposer7rr)rqrrrr(r(r)_shapeszTFIdeficsAttention._shaper+rzATuple[tf.Tensor, Optional[tf.Tensor], Optional[Tuple[tf.Tensor]]])r key_value_statesr5rFpast_key_valueoutput_attentionsrJrhc Cs^|jp |dk }t|\} } } |||| | } |s^|||| | } |||| | }n6t|\} }} ||||| } ||||| }t| d}|dk r|t|dd7}|st|rt|| }n t || }| ||\}}t | | |||\} } |dk r>tj |d| gdd} tj |d|gdd}|rL| |fnd}|j rl|| } || } tjjt|| d| |gd| d| |gdt|dt| | |||jo|dko| dkd }tjjt|| |j| |jgd| |j| |jgdt|dttj|dddd gd | | |jf}||}d}|rTtd |||fS) NrrrArz$Attention weights should be of size z , but is )message)Z attn_maskrrrz^attn_weights are not extracted in scaled_dot_product_attention. The model returns None instead)rrrrrrr6Z is_tensorZ reduce_maxmaxrrrCrrr debuggingZ assert_equalr:rrrrr7rrrlogger warning_once)rqr rr5rFrrrJrrZq_len_Z query_statesZ key_statesZ value_statesZkv_lenZ kv_seq_lenrrrZ attn_outputZ attn_weightsr(r(r)rusd          & zTFIdeficsAttention.callc Cs\|jr dSd|_|jr6t|jjds*|jn|jjj}n|j}t|dddk rzt |j j |j |j |jW5QRXt|dddk rt |jj |j |jW5QRXt|dddk rt |jj |j |W5QRXt|dddk r t |jj |j |W5QRXt|dddk rXt |jj |j dW5QRXdS)NT embed_dimrrrrr)rrr^r vision_configrrrr6rrrlrrrrrrr)rqrZ kv_input_dimr(r(r)rs0 zTFIdeficsAttention.build)rFNF)NNNNFF)N) r#r$r%r&rnrrurr}r(r(rsr)rbs5Qrc sHeZdZddfdd Zddddd d d d d d dZdddZZS)TFIdeficsDecoderLayerrrc s|tjf||j|_t|j|j|j|dd|_t|j|j|j dd|_ t |j|j dd|_ t |j|j dd|_|j|_dS)N self_attn)rrrrrlmlprrrrlinput_layernormrpost_attention_layernorm)rmrnrrnum_attention_headsrrrrrrrrrrrqrrOrsr(r)rn s,zTFIdeficsDecoderLayer.__init__NFrr+rre7Tuple[tf.Tensor, Optional[Tuple[tf.Tensor, tf.Tensor]]])r r5rFrrrJrhc Cs|}||}|j||||||d\}} } tjj||jd}||}|}||}||}tjj||jd}||}|f} |r| | f7} |r| | f7} | S)aE Args: hidden_states (`tf.Tensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`tf.Tensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(tf.Tensor)`, *optional*): cached past key and value projection states )r r5rFrrrJZrate)rrr6rrrr) rqr r5rFrrrJtrainingresidualself_attn_weightspresent_key_valuerDr(r(r)ru"s.      zTFIdeficsDecoderLayer.callc Cs|jr dSd|_t|dddk rFt|jj|jdW5QRXt|dddk r|t|jj|jdW5QRXt|dddk rt|jj|jdW5QRXt|dddk rt|j j|j dW5QRXdS)NTrrrr) rrr6rrrlrrrrrr(r(r)r[szTFIdeficsDecoderLayer.build)NNNFFF)Nrr(r(rsr)r s9rc sNeZdZddfdd ZfddZdd d d d d d d d d d ddZZS)!TFIdeficsGatedCrossAttentionLayerrrc stjf||j|_t|j|jd|j||jdd|_t|j|j |j dd|_ t |j|j dd|_t |j|j dd|_|j|_tjjj|_tjjj|_|j|_|j|_|j|_dS) NT cross_attn)rrrrrrrlrrrrr)rmrnrrrrrrrrrrrrrrrr6r]Z activationstanhact_cross_attn act_densealpha_initializer alpha_typealphas_initializer_rangerrsr(r)rnns:   z*TFIdeficsGatedCrossAttentionLayer.__init__c s|jr dSd|_|jdkr|jdkr^|jdd|jfdddd|_|jdd|jfdddd|_nF|jdkr|jd dddd|_|jd dddd|_ntd |jd n|jd krB|jdkr|jdd|jfd ddd|_|jdd|jfd ddd|_nH|jdkr.|jd d ddd|_|jd d ddd|_ntd |jd n|jd kr|jdkr|jdd|jftj j j d|j dddd|_|jdd|jftj j j d|j dddd|_nh|jdkr |jd tj j j d|j dddd|_|jd tj j j d|j dddd|_ntd |jd nt d|jdt|drHt|dsPtdt|jj|jdW5QRXt|jj|jdW5QRXt|jj|jdW5QRXt|jj|jdW5QRXt|dS)NTrwZvectorralpha_cross_attn)r:rr_rl alpha_denser)rz Unknown value for `alpha_type` ()r>gaussiannormalrandomr)Zmeanstddevr zAlpha initialization scheme z not yet implemented!z+Alpha parameters not initialized correctly!)rrr rrr r rr6r]Z initializersZ RandomNormalr NotImplementedErrorr^rrrlrrrrrmrrsr(r)rs                   z'TFIdeficsGatedCrossAttentionLayer.buildNFrr+rerr) r r5r"r3cross_attention_gaterrJrrhc Cs*|dkrtd|dkr td|dk r0td|} ||}|j||||d\}} } tjj||jd}tj|dk|j d} t | d } t t | t |d kt||}| ||j|}|} ||}||}tjj||jd}| ||j|}|f} |r| | f7} |r&| | f7} | S) a Args: hidden_states (`tf.Tensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`tf.Tensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(tf.Tensor)`, *optional*): cached past key and value projection states no_images (`bool`, *optional*, defaults to `False`): If `True` the vision part is ignored Nzt`image_hidden_states` is required for Idefics cross attention module which are visual features to be conditioned on.z`cross_attention_gate` is required for Idefics cross attention module to zero-out the cross-attention hidden_states attending to no images.zMPast key value states are not implemented for Idefics cross attention module.)r rr5rrrrGr/r)rrrrr6rrrrMrHrrNrr:Z zeros_likerr rrrr )rqr r5r"r3rrrJrrrrrrDr(r(r)rusL       z&TFIdeficsGatedCrossAttentionLayer.call)NNNNFFNrr(r(rsr)rms Trag This model inherits from [`TFPreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a TensorFlow [tf.keras.layers.Layer](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer) subclass. Use it as a regular TensorFlow Layer and refer to the TensorFlow documentation for all matter related to general usage and behavior. Parameters: config ([`IdeficsConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~TFPreTrainedModel.from_pretrained`] method to load the model weights. zSThe bare LLaMA Model outputting raw hidden-states without any specific head on top.c@s eZdZeZdZdZddgZdS)TFIdeficsPreTrainedModelraTrrN)r#r$r%r config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesr(r(r(r)r=sra  Args: input_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`tuple(tuple(tf.Tensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(tf.Tensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cseZdZdZeZd dddfdd Zd!d d Zgfd d Zgfd dZ ddZ ddZ ddZ e eed"dddddddddddddddddddZd#ddZZS)$TFIdeficsMainLayerz Transformer decoder consisting of `config.num_hidden_layers` layers. Each layer is a [`IdeficsDecoderLayer`] Args: config: IdeficsConfig Trr)radd_pooling_yearc stjf||_j|_j|_tjjjj dd|_ j j |_ j |_ t j dd|_jrj}tj j|j|j|j|jdd|_fddtjD|_j|_j|j}fddt|D|_d |_tjjd d |_ d |_|!dS) N embed_tokens)rorprrrgrl vision_modelrperceiver_resamplercsg|]}td|dqS)zlayers.r)rrSirr(r)rVsz/TFIdeficsMainLayer.__init__..csg|]}td|dqS)zgated_cross_attn_layers.r)rrrr(r)rVsFnormr)"rmrnrZ pad_token_idZ padding_idx vocab_sizerdadditional_vocab_sizerfreeze_text_layersrrZ image_sizerr use_resamplerperceiver_configrrZresampler_depthZresampler_n_headsZresampler_head_dimZresampler_n_latentsrr9Znum_hidden_layersdecoder_layerscross_layer_intervalgated_cross_attn_layersgradient_checkpointingrrrfreeze_relevant_params)rqrrrOr"Znum_cross_layersrsrr)rnsJ    zTFIdeficsMainLayer.__init__NcCs:|dkr|j}|jr ||j|jr6t|j|jddSN)rb)rr Zfreeze_text_module_exceptionsfreeze_vision_layersrcrZfreeze_vision_module_exceptionsrr(r(r)r's  z)TFIdeficsMainLayer.freeze_relevant_paramscCs"|j|jfD]}t||dq dSr()r#rrc)rqrbmoduler(r(r)r sz%TFIdeficsMainLayer.freeze_text_layerscCst|j|ddSr()rcr)rqrbr(r(r)r)sz'TFIdeficsMainLayer.freeze_vision_layerscCs|jSrXrr{r(r(r)get_input_embeddingssz'TFIdeficsMainLayer.get_input_embeddingscCs ||_dSrXr+rqrr(r(r)set_input_embeddingssz'TFIdeficsMainLayer.set_input_embeddingscCsHd}t||j|d}|dk rDt||j|dd}|dkr<|n||}|S)N)rr/)r)rrHr)rqr5r inputs_embedsrZcombined_attention_maskZexpanded_attn_maskr(r(r)_prepare_decoder_attention_masksz2TFIdeficsMainLayer._prepare_decoder_attention_maskFTFModelInputType | Noner+r.re9Union[TFIdeficsBaseModelOutputWithPast, Tuple[tf.Tensor]]r=r5rFrr/r0r1r2r3rJroutput_hidden_statesrI return_dictrrhc)Cs| dk r | n|jj} | dk r | n|jj} | dk r4| n|jj} |dk rH|n|jj}|dk rj|dk rjtdn6|dk rt|\}}n |dk rt|\}}}ntd|}d}|dk rt|ddd}||}|dk r|dkrtjj tj |tj dddd}t |dkd|}n,|dkr@tj |||tj d}t|d}d }tt|dkt|dkt|dkfdkrztd n|dk r.ttj |tj ddk}tj ||jd}t|jd krt|d}t|d}nDt|jd krt|dd\}}t|||f|jdd}|j|| d j}n@|dk rnt|\}}}}tj ||jd}t|||||f}|jjr|dkr||}t|dd\}}nt|\}}}}|}n(|dkrt|dd\}}ntdt|||||f}|dk r0t|jd kr0| dkr0tj||dftj d} t| d}t| d} tj| |d} t| ||||f} |dk rt|\}}}||f}| dkrtj|tj d} t| } nd} tjtj tj| dkdd|jddd}|dkr| |}|dkrtj||ftj!d}|"|||f||}|}|j#rH|rH| rHt$%dd } | rRdnd} | r`dnd}!| rndnd}"t&|j'D]\}#}$| r| |f7} |dk r||#nd}%dd}&|j#r|rd}%| rt$%dd } t(|&|$||||%|| | | ||#|j)|j*}'n&|&|$||||%|| || | |#|j)|j*d }'|'d}| rN|"|'| rDdndf7}"| r||!|'df7}!q||+|}| r| |f7} | r|"nd}(t|||||f}|st,dd||(| |!|fDSt-||(| |!|dS)NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timezEYou have to specify either decoder_input_ids or decoder_inputs_embedsrrrGr/rArFz_Exactly 1 of pixel_values, image_encoder_embeddings or perceiver_embeddings has to be not-None.)r0rIrzBIf `perceiver_embeddings` are passed, use_resampler should be True)ZrepeatszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...r(c  SsP| | dkr8| | | } | ||||||| dd}|d}||||||| d}|S)Nr)r5r"r3rrrJr)r5rFrrrJr()Z main_blockr r5rFrr"r3rrrJ layer_idxr$r%ZxblockrD layer_outputsr(r(r)vblocks,    z'TFIdeficsMainLayer.call..vblock) r5rFrr"r3rrrJr8r$r%css|]}|dk r|VqdSrXr()rSvr(r(r)r\sz*TFIdeficsMainLayer.call..)rrr r!r").rrr4rJuse_return_dictrrr6rKrLrMZint32rNr9rsumrZ reduce_sumrHlenr:r7rrr!rrwr8rrZsqueezeZ reduce_anyrrr0r&rr enumerater#Zrecompute_gradr$r%rtupler))rqr=r5rFrr/r0r1r2r3rJrr4rIr5rZ batch_sizeZ seq_lengthrZseq_length_with_pastrZ no_imagesZ num_imagesr"Z image_seq_lenZimage_hidden_sizeZ text_seq_lenZimage_batch_sizeZimage_sequence_lengthZimage_hidden_shaperr Zall_hidden_statesZall_self_attnsZnext_decoder_cacheidxZ decoder_layerrr:r9Z next_cacher(r(r)rusP    $           $         )    zTFIdeficsMainLayer.callc Csp|jr dSd|_t|dddk rFt|jj|jdW5QRXt|dddk r|t|jj|jdW5QRXt|dddk rt|jj|jdW5QRXt|dddk rt|j j|j dW5QRXt|dddk r*|j D](}t|j|dW5QRXqt|dddk rl|j D](}t|j|dW5QRXqBdS)NTrrrrr#r%) rrr6rrrlrrrrr#r%)rqrr[r(r(r)rs.  zTFIdeficsMainLayer.build)T)N)NNNNNNNNNNNNFNN)N)r#r$r%r&rrrnr'r r)r,r.r0rrLLAMA_INPUTS_DOCSTRINGrurr}r(r(rsr)rs<0  2 rcsZeZdZddfdd Zddddd dddddd d d d d d d d d dZdddZZS)TFIdeficsModelrrcs&tj|f||t|dd|_dS)Nrar)rmrnrra)rqrrrOrsr(r)rnszTFIdeficsModel.__init__NFr1r+r.rer2r3cCs,|j||||||||| | | | | ||d}|S)Nr=r5rFrr/r0r1r2r3rJrr4rIr5rra)rqr=r5rFrr/r0r1r2r3rJrr4rIr5rrDr(r(r)rus$zTFIdeficsModel.callc CsJ|jr dSd|_t|dddk rFt|jj|jdW5QRXdS)NTra)rrr6rrarlrrr(r(r)rBs zTFIdeficsModel.build)NNNNNNNNNNNNFNN)Nrr(r(rsr)rCs$.%rCcseZdZdgZddgZeZd(fdd ZddZdd Z d d Z d d Z ddZ ddZ ddZeeeeeedd)dddddddddddddddddddZd*ddZed d!Zed"d#Zed$d%Zd+d&d'ZZS),TFIdeficsForVisionText2Textzlm_head.weightzmodel.embed_tokens.weightNc s@tj|f|t|dd|_t|j|j|jd|jdd|_ dS)NrarFlm_head)rrgrl) rmrnrrar~rrrZfreeze_lm_headrG)rqrrrOrsr(r)rnPsz$TFIdeficsForVisionText2Text.__init__cCs|jjSrXrarr{r(r(r)r,\sz0TFIdeficsForVisionText2Text.get_input_embeddingscCs ||j_dSrXrHr-r(r(r)r._sz0TFIdeficsForVisionText2Text.set_input_embeddingscCs|jSrXrGr{r(r(r)get_output_embeddingsbsz1TFIdeficsForVisionText2Text.get_output_embeddingscCs ||_dSrXrI)rqZnew_embeddingsr(r(r)set_output_embeddingsesz1TFIdeficsForVisionText2Text.set_output_embeddingscCs ||_dSrXrE)rqdecoderr(r(r) set_decoderhsz'TFIdeficsForVisionText2Text.set_decodercCs|jSrXrEr{r(r(r) get_decoderksz'TFIdeficsForVisionText2Text.get_decodercCs|}|}t|jddrL|j|_|jdkrL|j|jks@t|jj|j _t |drt |dr|j |_ t |drt |dr|j|_dS) z Overwrite `transformers.modeling_utils.PreTrainedModel.tie_weights` to handle the case of IdeficsDecoupledLinear and IdeficsDecoupledEmbedding. Ztie_word_embeddingsTrrrorrpN) rJr,rrrrprAssertionErrorrkrr^ror)rqZoutput_embeddingsZinput_embeddingsr(r(r) tie_weightsns   z'TFIdeficsForVisionText2Text.tie_weights) output_typerFr1r+r.rez8Union[TFIdeficsCausalLMOutputWithPast, Tuple[tf.Tensor]])r=r5rFrr/r0r1r2r3labelsrJrr4rIr5rhcCsp| dk r | n|jj} | dk r | n|jj} |dk r4|n|jj}|j||||||||| | | | |||d}|d}||}d}| dk r$|dk r|dddf}|dddddf|dk}| dddf|dk}n&|dddddf}| dddf}|jt|dgt|d|j dgd}|sT|f|dd}|dk rP|f|S|St |||j |j |j |jdS) a Args: labels (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Returns: Example: ```python >> from transformers import AutoTokenizer, TFIdeficsForVisionText2Text >> model = TFIdeficsForVisionText2Text.from_pretrained("HuggingFaceM4/idefics-9b") >> tokenizer = AutoTokenizer.from_pretrained("HuggingFaceM4/idefics-9b") >> prompt = "Hey, are you consciours? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="tf") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you consciours? Can you talk to me?\nI'm not consciours, but I can talk to you." ```NrDr.rr/)rRr-)r,r-rr r!r")rrr4r<rarGZhf_compute_lossr6r7r:r*rr r!r")rqr=r5rFrr/r0r1r2r3rRrJrr4rIr5rrDr r-r,Zshift_attention_maskZ shift_logitsZ shift_labelsrr(r(r)rus\0   z TFIdeficsForVisionText2Text.callcKsj|dd}|dk r6|jjr&||d<n||d<d|d<t|fd|i|}dg}|D]}||dqT|S)Nr"r2r1r0pastr4)poprr!rP)rqr=rSrOr"rZunwanted_kwargskwargr(r(r)rPs  z9TFIdeficsForVisionText2Text.prepare_inputs_for_generationcOs t||SrX)r@)argsr?r(r(r)_expand_inputs_for_generationsz9TFIdeficsForVisionText2Text._expand_inputs_for_generationcCs t||SrX)rE)rDr?r>r(r(r)#_update_model_kwargs_for_generationsz?TFIdeficsForVisionText2Text._update_model_kwargs_for_generationcs.d}|D] }|tfdd|Df7}q|S)Nr(c3s|]}t|VqdSrX)r6r;)rSZ past_statebeam_idxr(r)r\sz=TFIdeficsForVisionText2Text._reorder_cache..)r@)rSrZZreordered_pastZ layer_pastr(rYr)_reorder_cachesz*TFIdeficsForVisionText2Text._reorder_cachec Cs|jr dSd|_t|dddk rFt|jj|jdW5QRXt|dddk r|t|jj|jdW5QRXdS)NTrarG)rrr6rrarlrrGrr(r(r)r sz!TFIdeficsForVisionText2Text.build)N)NNNNNNNNNNNNNFNF)N)N)r#r$r%Z_keys_to_ignore_on_load_missingZ_tied_weights_keysrrrnr,r.rJrKrMrNrPrrrBrr*_CONFIG_FOR_DOCrurP staticmethodrWrXr[rr}r(r(rsr)rFKsN  4e    rF)rFNN)N)r)N)Fr& __future__rZ dataclassesrtypingrrrrZ tensorflowr6r Zactivations_tfr Zmodeling_tf_outputsr Zmodeling_tf_utilsr r rrrZtf_utilsrrutilsrrrrZconfiguration_ideficsrZ perceiver_tfrZ vision_tfrZ get_loggerr#rr\rr*r@rErPrcr]rWrRrdZLayerr~rrrrrrrrrrZLLAMA_START_DOCSTRINGrrBrrCrFr(r(r(r)sv         ,+ ' ' iY   "*b@>4