U 4AfW@sdZddlZddlmZddlmZmZmZddlZddl Zddlm Z ddl m Z ddl mZmZdd lmZmZd d lmZeeZeGd d d eZGddde jZGddde jZGddde jZGddde jZGddde jZGddde jZdS)zTPyTorch IdeficsVision model: a copy of CLIPVisionModel using a simpler config objectN) dataclass)OptionalTupleUnion)nn)ACT2FN)BaseModelOutputBaseModelOutputWithPooling) ModelOutputlogging)IdeficsVisionConfigc@sfeZdZUdZdZeejed<dZ ejed<dZ ee ejdfed<dZ ee ejdfed<dS)IdeficsVisionModelOutputa Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrrrF/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/idefics/vision.pyr"s rcsReZdZedfdd ZejeeejdddZd ej e ejdd d Z Z S) IdeficsVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF)Z in_channelsZ out_channelsZ kernel_sizeZstrideZbiasr position_ids)r ) persistent)super__init__r hidden_size embed_dim image_size patch_sizer ParameterrZrandnclass_embeddingZConv2d num_channelspatch_embedding num_patches num_positionsZ Embeddingposition_embeddingZregister_bufferZarangeexpandselfr __class__rrr%As"  z IdeficsVisionEmbeddings.__init__) embeddingsheightwidthreturncCs|jdd}||j}|jdd}||kr<||kr<|S|dddf}|ddddf}|jd} ||jj} ||jj} | d| d} } t|} |dt| t| | }| dddd}|j t j k} | rt d|t j}tjj|| | | | fd d d }| r|t j }t| |jd ksFt| |jdkrxtd t| t| fd|jd |jdfd| dddddd| }t j|d|fddS)a# This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. Source: https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174 r Nrr"g?rr zUpcasting patch_pos_embed to fp32 for interpolation since `upsample_bicubic2d_out_frame` in nn.functional.interpolate is not implemented for 'torch.bfloat16' dtype. This will result in a slight overhead.ZbicubicF)Z scale_factormodeZ align_cornerszNumber of patches for images (z/) don't match the shape of position embedding ()Zdim)shaper0r!rr)mathsqrtreshapeintZpermutedtyperZbfloat16loggerZ warning_oncetofloatr functionalZ interpolate ValueErrorviewcatZ unsqueeze)r3r6r7r8r.Z pos_embedr/Zclass_pos_embedZpatch_pos_embedr'Z num_h_patchesZ num_w_patchesZsqrt_num_positionsZfp32_upcastingrrrinterpolate_pos_encodingXsD         (,z0IdeficsVisionEmbeddings.interpolate_pos_encodingF) pixel_valuesrKr9c Cs|j\}}}}|sL||jks&||jkrLtd|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r| | | ||} n| ||j} | S) NzInput image size (*z) doesn't match model (z8). You should try to set `interpolate_pos_encoding=True`)rCr r r"r=)r>r(rHr-ZweightrCrEflatten transposer+r1rrJrKr0r!) r3rLrKZ batch_sizer,r7r8Z target_dtypeZ patch_embedsZ class_embedsr6rrrforwards  zIdeficsVisionEmbeddings.forward)F) rrrrr%rTensorrBrKrboolrP __classcell__rrr4rr@s1rc sneZdZdZfddZejeedddZd eje eje eje e e eje ejfd d d Z Z S) IdeficsVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ |j |_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g)r$r%rr&r'Znum_attention_heads num_headshead_dimrHscaleZattention_dropoutdropoutrLineark_projv_projq_projout_projr2r4rrr%s  zIdeficsVisionAttention.__init__)tensorseq_lenbszcCs ||||j|jddS)Nr r )rIrUrVrO contiguous)r3r^r_r`rrr_shapeszIdeficsVisionAttention._shapeNFrattention_maskcausal_attention_maskoutput_attentionsr9cCs|\}}}|||j}|||d|} |||d|} ||jd|jf} ||||j| }| j| } | j| } | d} t || dd} | ||j|| fkrt d||j|| fd| |dk rD||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } |dk r||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } t jj| dd} |r| ||j|| }|||j|| } nd}t jj| |j|jd }t || }|||j||jfkrRt d ||j||jfd||||j||j}| dd}||||}||}||fS) z#Input shape: Batch x Time x Channelr"r r z$Attention weights should be of size z , but is Nz!Attention mask should be of size r=)ptrainingz `attn_output` should be of size )sizer\rWrbrZr[rUrVrIrZbmmrOrHrrGZsoftmaxrXrhrAr])r3rrdrerfr`Ztgt_lenr'Z query_statesZ key_statesZ value_statesZ proj_shapeZsrc_len attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputrrrrPsX         zIdeficsVisionAttention.forward)NNF)rrrrr%rrQrBrbrrRrrPrSrrr4rrTs rTcs0eZdZfddZejejdddZZS)IdeficsVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) r$r%rrZ hidden_act activation_fnrrYr&Zintermediate_sizefc1fc2r2r4rrr%s   zIdeficsVisionMLP.__init__)rr9cCs"||}||}||}|Srl)rnrmro)r3rrrrrPs   zIdeficsVisionMLP.forward)rrrr%rrQrPrSrrr4rrk s rkcsJeZdZedfdd Zdejejejeee ej dddZ Z S) IdeficsVisionEncoderLayerrcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)Zeps) r$r%r&r'rT self_attnr LayerNormlayer_norm_eps layer_norm1rkmlp layer_norm2r2r4rrr%s    z"IdeficsVisionEncoderLayer.__init__FrccCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r`||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrdrerf)rurrrwrv)r3rrdrerfZresidualrjoutputsrrrrP&s"     z!IdeficsVisionEncoderLayer.forward)F) rrrrr%rrQrrRrrrPrSrrr4rrps rpc s`eZdZdZedfdd Zd eejeejee ee ee e e e fdddZ ZS) IdeficsVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`IdeficsVisionEncoderLayer`]. Args: config: IdeficsVisionConfig rcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr)rp).0_rrr \sz1IdeficsVisionEncoder.__init__..F) r$r%rrZ ModuleListrangeZnum_hidden_layerslayersgradient_checkpointingr2r4rrr%Ys  zIdeficsVisionEncoder.__init__N)rdrerfoutput_hidden_states return_dictr9c Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|rDdnd}|rPdnd}|} t|jD]b\} } |rx|| f}|jr|jr|| j | |||} n| | |||d} | d} |rb|| df}qb|r|| f}|st dd| ||fDSt | ||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr)rfrr css|]}|dk r|VqdSrlr)rzvrrr sz/IdeficsVisionEncoder.forward..)rrr) rrfruse_return_dict enumerater~rrhZ_gradient_checkpointing_func__call__tupler ) r3 inputs_embedsrdrerfrrZencoder_statesZall_attentionsridxZ encoder_layerZ layer_outputsrrrrP_sH&     zIdeficsVisionEncoder.forward)NNNNN)rrrrrr%rrrQrRrrr rPrSrrr4rryPs  ryc sZeZdZedfdd Zd eejeeeeeeeee e e fdddZ Z S) IdeficsVisionTransformerrcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSrq) r$r%rr&rr6rrsrt pre_layrnormryencoderpost_layernorm)r3rr'r4rrr%s   z!IdeficsVisionTransformer.__init__NF)rLrfrrKrr9c Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd|j||d}||}|j||||d}|d}|dddddf} || } |s|| f|ddSt || |j |j dS)z Returns: Nz You have to specify pixel_values)rK)rrfrrrr )rZ pooler_outputrr) rrfrrrHr6rrrr rr) r3rLrfrrKrrZencoder_outputsrZ pooled_outputrrrrPs2   z IdeficsVisionTransformer.forward)NNNFN)rrrrr%rrrrRrrr rPrSrrr4rrs  r) rr?Z dataclassesrtypingrrrrZtorch.utils.checkpointrZ activationsrZmodeling_outputsr r utilsr r Zconfiguration_ideficsrZ get_loggerrrDrModulerrTrkrpryrrrrrs&     di3b