U 4Af0@sdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddl Z ddlmZddlmZmZmZmZmZddlmZdd lmZmZmZdd lmZmZmZm Z d d l!m"Z"m#Z#m$Z$e%e&Z'd Z(dZ)dZ*dZ+dZ,dZ-e j.e j.dddZ/e j.e j.dddZ0eGdddeZ1Gddde j2Z3Gddde j2Z4Gdd d e j2Z5d!e4iZ6Gd"d#d#e j2Z7Gd$d%d%e j2Z8Gd&d'd'e j2Z9Gd(d)d)e j2Z:Gd*d+d+e j2Z;Gd,d-d-e j2ZGd2d3d3e j2Z?Gd4d5d5e j2Z@Gd6d7d7e j2ZAGd8d9d9eZBGd:d;d;e j2ZCGdd?d?eBZEGd@dAdAeBZFGdBdCdCeBZGdFdDdEZHdS)GzPyTorch AltCLIP model.N) dataclass)AnyListOptionalTupleUnion)ACT2FN)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions'BaseModelOutputWithPoolingAndProjection)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutput%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) AltCLIPConfigAltCLIPTextConfigAltCLIPVisionConfigz BAAI/AltCLIPra= This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`CLIPConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. aE Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. )logitsreturncCstj|tjt||jdS)Ndevice)nn functionalZ cross_entropytorcharangelenr)rr$P/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/altclip/modeling_altclip.pycontrastive_losssr&) similarityrcCs t|}t|}||dS)Ng@)r&t)r'Z caption_lossZ image_lossr$r$r% clip_losss r)c@seZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ eed<dZeed <eed d d ZdS) AltCLIPOutputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image:(`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text:(`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`AltCLIPTextModel`]. image_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`AltCLIPVisionModel`]. text_model_output(`BaseModelOutputWithPooling`): The output of the [`AltCLIPTextModel`]. vision_model_output(`BaseModelOutputWithPooling`): The output of the [`AltCLIPVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrcstfddDS)Nc3s,|]$}|dkr|n t|VqdS))r0r1N)getattrto_tuple).0kselfr$r% sz)AltCLIPOutput.to_tuple..)tuplekeysr7r$r7r%r4s zAltCLIPOutput.to_tuple)__name__ __module__ __qualname____doc__r+rr! FloatTensor__annotations__r,r-r.r/r0r r1rrr4r$r$r$r%r*s   r*cs2eZdZdZfddZd ddZdd ZZS) AltRobertaEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dd|j|_tj|j|j|jd|_ dS) N) padding_idxZepsposition_embedding_typeabsolute position_idsrF persistenttoken_type_idsdtype)super__init__r EmbeddingZ vocab_size hidden_sizeZ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutr3rEregister_bufferr!r"expandzerosrGsizelongrCr8config __class__r$r%rPs. zAltRobertaEmbeddings.__init__Nrc Cs|dkr*|dk r t||j|}n ||}|dk r<|}n|dd}|d}|dkrt|dr|jddd|f}||d|} | }ntj|tj |j j d}|dkr| |}| |} || } |jdkr||} | | 7} || } || } | S)NrIrrLrrNrrF)"create_position_ids_from_input_idsrC&create_position_ids_from_inputs_embedsr_hasattrrLr]r!r^r`rGrrSrVrErUrWr[) r8 input_idsrLrG inputs_embedspast_key_values_length input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrV embeddingsrUr$r$r%forwards0         zAltRobertaEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrIrrer)r_r!r"rCr`rZ unsqueezer])r8rjrlZsequence_lengthrGr$r$r%rg s  z;AltRobertaEmbeddings.create_position_ids_from_inputs_embeds)NNNNr)r<r=r>r?rPrqrg __classcell__r$r$rcr%rBs  (rBc seZdZd fdd ZejejdddZd ejeejeejeejeejee e ejee e ejdd d Z Z S) AltRobertaSelfAttentionNcst|j|jdkr>t|ds>td|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|pt|dd|_|jdks|jd kr|j|_t d |jd |j|_|j|_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rErF relative_keyrelative_key_queryr)rOrPrRnum_attention_headsrh ValueErrorintattention_head_size all_head_sizerLinearquerykeyvaluerYZattention_probs_dropout_probr[r3rErTrQdistance_embedding is_decoderr8rbrErcr$r%rPs* z AltRobertaSelfAttention.__init__)xrcCs6|dd|j|jf}||}|ddddS)NrIrrwrr)r_rxr{viewpermute)r8rZ new_x_shaper$r$r%transpose_for_scores9s z,AltRobertaSelfAttention.transpose_for_scoresF hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsrcCs||}|dk } | r4|dk r4|d} |d} |}n| r^|||} |||} |}nv|dk r|||} |||} tj|d| gdd} tj|d| gdd} n |||} |||} ||} |dk } |jr| | f}t| | dd}|j dks |j dkr| j d| j d}}| r^tj |dtj |j d dd}ntj|tj |j d dd}tj|tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}||}n4|j dkrtd | |}td | |}|||}|t|j}|dk r:||}tjj|dd}||}|dk rf||}t|| }|dddd }|dd|jf}||}|r||fn|f}|jr||f}|S)NrrrwdimrIrurvrerMzbhld,lrd->bhlrzbhrd,lrd->bhlrr) r~rrrr!catrmatmul transposerEshapetensorr`rrr"rrTtorNZeinsummathsqrtr{rr softmaxr[r contiguousr_r|)r8rrrrrrrZmixed_query_layerZis_cross_attentionZ key_layerZ value_layerZ query_layer use_cacheZattention_scoresZ query_lengthZ key_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZ context_layerZnew_context_layer_shapeoutputsr$r$r%rq>sp             zAltRobertaSelfAttention.forward)N)NNNNNF) r<r=r>rPr!Tensorrrr@rboolrqrrr$r$rcr%rss$rscs4eZdZfddZejejejdddZZS)AltRobertaSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrD) rOrPrr}rRdenserWrXrYrZr[rarcr$r%rPs zAltRobertaSelfOutput.__init__r input_tensorrcCs&||}||}|||}|SNrr[rWr8rrr$r$r%rqs  zAltRobertaSelfOutput.forwardr<r=r>rPr!rrqrrr$r$rcr%rs reagerc sveZdZd fdd ZddZd ejeejeejeejeejee e ejee e ejddd Z Z S) AltRobertaAttentionNcs4tt|j||d|_t||_t|_dS)NrE) rOrP"ALT_ROBERTA_SELF_ATTENTION_CLASSESZ_attn_implementationr8routputset pruned_headsrrcr$r%rPs  zAltRobertaAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)r#rr8rxr{rrr~rrrrr|union)r8headsindexr$r$r% prune_headss zAltRobertaAttention.prune_headsFrc Cs<||||||||}||d|} | f|dd} | S)Nrr)r8r) r8rrrrrrrZ self_outputsattention_outputrr$r$r%rqs  zAltRobertaAttention.forward)N)NNNNNF) r<r=r>rPrr!rrr@rrrqrrr$r$rcr%rs$rcs0eZdZfddZejejdddZZS)AltRobertaIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) rOrPrr}rRintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrarcr$r%rPs   zAltRobertaIntermediate.__init__rrcCs||}||}|Sr)rrr8rr$r$r%rqs  zAltRobertaIntermediate.forwardrr$r$rcr%rs rcs4eZdZfddZejejejdddZZS)AltRobertaOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rOrPrr}rrRrrWrXrYrZr[rarcr$r%rPs zAltRobertaOutput.__init__rcCs&||}||}|||}|Srrrr$r$r%rqs  zAltRobertaOutput.forwardrr$r$rcr%rs rc steZdZfddZd ejeejeejeejeejeeeejee eejdddZ dd Z Z S) AltRobertaLayercsrt|j|_d|_t||_|j|_|j|_|jrZ|jsLt|dt|dd|_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is addedrFr)rOrPchunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionrycrossattentionr intermediaterrrarcr$r%rP s   zAltRobertaLayer.__init__NFrc Cs|dk r|ddnd}|j|||||d} | d} |jrP| dd} | d} n | dd} d} |jr|dk rt|dstd|d|dk r|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) Nrw)rrrrrIrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrhryrrfeed_forward_chunkrr)r8rrrrrrrZself_attn_past_key_valueZself_attention_outputsrrZpresent_key_valueZcross_attn_present_key_valueZcross_attn_past_key_valueZcross_attention_outputs layer_outputr$r$r%rqsV         zAltRobertaLayer.forwardcCs||}|||}|Sr)rr)r8rZintermediate_outputrr$r$r%r\s  z"AltRobertaLayer.feed_forward_chunk)NNNNNF) r<r=r>rPr!rrr@rrrqrrrr$r$rcr%r s$  ArcseZdZfddZd ejeejeejeejeejeeeejee ee ee ee e eeje fd ddZ Z S) AltRobertaEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr$)rr5_rbr$r% gsz.AltRobertaEncoder.__init__..F) rOrPrbr ModuleListrangenum_hidden_layerslayergradient_checkpointingrarcrr%rPds  zAltRobertaEncoder.__init__NFT) rrrrrpast_key_valuesrroutput_hidden_states return_dictrc  Csb| rdnd} |rdnd} |r(|jjr(dnd} |jrJ|jrJ|rJtdd}|rRdnd}t|jD]\}}| rv| |f} |dk r||nd}|dk r||nd}|jr|jr||j |||||||}n||||||||}|d}|r||df7}|r`| |df} |jjr`| |df} q`| r.| |f} | sPt dd ||| | | fDSt ||| | | d S) Nr$zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FrrIrrwcss|]}|dk r|VqdSrr$r5vr$r$r%r9sz,AltRobertaEncoder.forward..)last_hidden_staterr attentionscross_attentions) rbrrtrainingloggerZ warning_once enumerater_gradient_checkpointing_func__call__r:r )r8rrrrrrrrrrZall_hidden_statesZall_self_attentionsZall_cross_attentionsZnext_decoder_cacheiZ layer_moduleZlayer_head_maskr layer_outputsr$r$r%rqjsx          zAltRobertaEncoder.forward) NNNNNNFFT)r<r=r>rPr!rrr@rrrr rqrrr$r$rcr%rcs. rcs0eZdZfddZejejdddZZS)AltRobertaPoolercs*tt|j|j|_t|_dSr)rOrPrr}rRrZTanh activationrarcr$r%rPs zAltRobertaPooler.__init__rcCs(|dddf}||}||}|S)Nr)rr)r8rZfirst_token_tensor pooled_outputr$r$r%rqs  zAltRobertaPooler.forwardrr$r$rcr%rs rc sneZdZdZfddZejeedddZd eje eje eje e e eje ejfd d d Z Z S) AltCLIPAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ |j |_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).)rOrPrbrR embed_dimrx num_headshead_dimryscaleZattention_dropoutr[rr}k_projv_projq_projout_projrarcr$r%rPs  zAltCLIPAttention.__init__)rseq_lenbszcCs ||||j|jddS)Nrrw)rrrrr)r8rrrr$r$r%_shapeszAltCLIPAttention._shapeNFrrcausal_attention_maskrrcCs|\}}}|||j}|||d|} |||d|} ||jd|jf} ||||j| }| j| } | j| } | d} t || dd} | ||j|| fkrt d||j|| fd| |dk rD||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } |dk r||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } t jj| dd} |r| ||j|| }|||j|| } nd}t jj| |j|jd }t || }|||j||jfkrRt d ||j||jfd||||j||j}| dd}||||}||}||fS) z#Input shape: Batch x Time x ChannelrIrrwz$Attention weights should be of size z , but is Nz!Attention mask should be of size r)prz `attn_output` should be of size )r_rrrrrrrrr!Zbmmrryrr rr[rZreshaper)r8rrrrrZtgt_lenrZ query_statesZ key_statesZ value_statesZ proj_shapeZsrc_len attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr$r$r%rqsX         zAltCLIPAttention.forward)NNF)r<r=r>r?rPr!rrzrrrrrqrrr$r$rcr%rs rcs0eZdZfddZejejdddZZS) AltCLIPMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) rOrPrbr r activation_fnrr}rRrfc1fc2rarcr$r%rP;s   zAltCLIPMLP.__init__rcCs"||}||}||}|Sr)rrrrr$r$r%rqBs   zAltCLIPMLP.forwardrr$r$rcr%r:s rcsJeZdZedfdd Zdejejejeee ej dddZ Z S) AltCLIPEncoderLayerrcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSr) rOrPrRrr self_attnrrWrX layer_norm1rmlp layer_norm2rarcr$r%rPJs    zAltCLIPEncoderLayer.__init__FrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r`||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrr)rrrr)r8rrrrZresidualrrr$r$r%rqRs"     zAltCLIPEncoderLayer.forward)F) r<r=r>rrPr!rrrrr@rqrrr$r$rcr%rIs rc s`eZdZdZedfdd Zd eejeejee ee ee e e e fdddZ ZS) AltCLIPEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`AltCLIPEncoderLayer`]. Args: config: AltCLIPConfig rcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr$)rrrr$r%rsz+AltCLIPEncoder.__init__..F) rOrPrbrrrrlayersrrarcrr%rPs  zAltCLIPEncoder.__init__N)rrrrrrc Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|rDdnd}|rPdnd}|} t|jD]b\} } |rx|| f}|jr|jr|| j | |||} n| | |||d} | d} |rb|| df}qb|r|| f}|st dd| ||fDSt | ||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr$)rrrcss|]}|dk r|VqdSrr$rr$r$r%r9sz)AltCLIPEncoder.forward..)rrr) rbrruse_return_dictrrrrrrr:r ) r8rjrrrrrZencoder_statesZall_attentionsridxZ encoder_layerrr$r$r%rqsH&     zAltCLIPEncoder.forward)NNNNN)r<r=r>r?rrPrr!rrrrr rqrrr$r$rcr%r{s  rcs6eZdZedfdd ZejejdddZZ S)AltCLIPVisionEmbeddingsrcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF)Z in_channelsZ out_channelsZ kernel_sizeZstridebiasrwrrGrHrJ)rOrPrbrRrZ image_sizeZ patch_sizer Parameterr!Zrandnclass_embeddingZConv2dZ num_channelspatch_embeddingZ num_patchesZ num_positionsrQposition_embeddingr\r"r]rarcr$r%rPs"  z AltCLIPVisionEmbeddings.__init__) pixel_valuesrcCsn|jd}|jjj}||j|d}|ddd}|j|dd}t j ||gdd}|| |j }|S)NrrMrwrrIr) rrweightrNrflattenrrr]r!rrrG)r8r batch_sizeZ target_dtypeZ patch_embedsZ class_embedsrpr$r$r%rqs  zAltCLIPVisionEmbeddings.forward) r<r=r>rrPr!r@rrqrrr$r$rcr%rsrc@s(eZdZdZeZdZdZgZddZ dS)AltCLIPPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. ZaltclipTcCs|jj}t|trt|jj}tjj|jd|jd|dtjj|j j |jj |dtjj|j j |jj |dnt|t r|jj}|jdd|jjd|}|jd|}tjj|jj |dtjj|jj |dtjj|jj |dtjj|jj |dnt|tr||jj}|jjdd|jjd|}d|jjd|}tjj|jj |dtjj|jj |dn t|trtjj|jj |jd|jjdd|j_tjj|jj |jd|jjdd|j_nt|tjr|jj |j j!dnt|tj"rF|j jjd|jjd|jdk r|jj nBt|tj#r|j jjd|jjd|j$dk r|j j|j$ dS) zInitialize the weightsgr)Zmeanstd)r rwTg?N)%rbZinitializer_factorrrrinitZnormal_rrrr Zinitializer_rangerrrrrrrrrRrr AltCLIPModeltext_projectiontext_embed_dimZ_is_hf_initializedvisual_projectionvision_embed_dimrWrdataZzero_Zfill_r}rQrC)r8moduleZfactorZ in_proj_stdZ out_proj_stdZfc_stdr$r$r% _init_weights sT          z$AltCLIPPreTrainedModel._init_weightsN) r<r=r>r?r config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulerr$r$r$r%r s r c sheZdZedfdd Zeeeeedd e e j e e e e e e e eefdddZZS) AltCLIPVisionTransformerrcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) rOrPrbrRrrprrWrX pre_layrnormrencoderpost_layernorm)r8rbrrcr$r%rP:s   z!AltCLIPVisionTransformer.__init__ output_typerNrrrrrc Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd||}||}|j||||d}|d}|dddddf}||}|s||f|ddSt |||j |j dS)z Returns: Nz You have to specify pixel_values)rjrrrrrr pooler_outputrr) rbrrrryrprrrr rr) r8rrrrrencoder_outputsrrr$r$r%rqDs2    z AltCLIPVisionTransformer.forward)NNNN)r<r=r>rrPrALTCLIP_VISION_INPUTS_DOCSTRINGrr rr!r@rrrrqrrr$r$rcr%r9s   rc seZdZeZdZedfdd ZejdddZ e e e e edd eejeeeeeeeee fd d d ZZS)AltCLIPVisionModelrrcs"t|t||_|dSr)rOrPr vision_model post_initrarcr$r%rPws  zAltCLIPVisionModel.__init__r2cCs |jjjSr)r$rprr7r$r$r%get_input_embeddings}sz'AltCLIPVisionModel.get_input_embeddingsrNrcCs&|dk r |n|jj}|j||||dS)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AltCLIPVisionModel >>> model = AltCLIPVisionModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```Nrrrr)rbrr$)r8rrrrr$r$r%rqszAltCLIPVisionModel.forward)NNNN)r<r=r>rrZmain_input_namerPrModuler&rr"rr rr!r@rrrrqrrr$r$rcr%r#ss"  r#cseZdZdZeZdfdd ZddZddZd d Z de e j e e j e e j e e j e e j e e j e e j e e j e e e je ee ee ee eeee j efd d dZZS)AltRobertaModela* The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. .. _*Attention is all you need*: https://arxiv.org/abs/1706.03762 TcsDt|||_t||_t||_|r2t|nd|_| dSr) rOrPrbrBrprrrpoolerr%)r8rbadd_pooling_layerrcr$r%rPs    zAltRobertaModel.__init__cCs|jjSrrprSr7r$r$r%r&sz$AltRobertaModel.get_input_embeddingscCs ||j_dSrr,r8rr$r$r%set_input_embeddingssz$AltRobertaModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r8Zheads_to_prunerrr$r$r% _prune_headsszAltRobertaModel._prune_headsN)rirrLrGrrjrrrrrrrrc Cs^| dk r | n|jj} | dk r | n|jj} | dk r4| n|jj} |jjrZ| dk rP| n|jj} nd} |dk rx|dk rxtdn@|dk r||||}n"|dk r|dd}ntd|\}}|dk r|j n|j }| dk r| ddj dnd}|dkrt j |||f|d}|dkrft |jd rT|jjddd|f}|||}|}nt j|t j|d }|||}|jjr|dk r|\}}}||f}|dkrt j ||d}||}nd}|||jj}|j|||||d }|j|||||| | | | | d }|d}|jdk r$||nd}| sB||f|d dSt|||j|j|j|jdS)a  encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timerIz5You have to specify either input_ids or inputs_embedsrrwrrLre)rirGrLrjrk) rrrrrrrrrr)rr rrrr)rbrrrrrryZ%warn_if_padding_and_no_attention_maskr_rrr!ZonesrhrprLr]r^r`Zget_extended_attention_maskZinvert_attention_maskZ get_head_maskrrr*r rrrr)r8rirrLrGrrjrrrrrrrrlr rmrrkrnroZextended_attention_maskZencoder_batch_sizeZencoder_sequence_lengthrZencoder_hidden_shapeZencoder_extended_attention_maskZembedding_outputr!sequence_outputrr$r$r%rqs$          zAltRobertaModel.forward)T) NNNNNNNNNNNNN)r<r=r>r?rrrPr&r.r0rr!rrr@rrrr rqrrr$r$rcr%r)sF   r)cseZdZeZfddZejdddZej dddd Z de e ej d fd d Z eeeeed de eje eje eje eje eje eje eje eje ee ee eeeefd ddZZS)AltCLIPTextModelcsLt|t|dd|_t|j|j|_tj |j|j d|_ | dS)NF)r+rD) rOrPr)robertarr}rR project_dimtransformationrWrXpre_LNr%rarcr$r%rP]s  zAltCLIPTextModel.__init__r2cCs |jjjSrr3rprSr7r$r$r%r&dsz%AltCLIPTextModel.get_input_embeddingsN)rrcCs||jj_dSrr7r-r$r$r%r.gsz%AltCLIPTextModel.set_input_embeddings)new_num_tokensrcs t|Sr)rOresize_token_embeddings)r8r8rcr$r%r9jsz(AltCLIPTextModel.resize_token_embeddingsr) rirrLrGrrjrrrrrrc  Cs| dk r | n|jj} |j||||||||| | | d } | d} || } || }|dddf}| sx||f| ddSt||| j| jdS)a= Returns: Examples: ```python >>> from transformers import AutoProcessor, AltCLIPTextModel >>> model = AltCLIPTextModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> texts = ["it's a cat", "it's a dog"] >>> inputs = processor(text=texts, padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```N) rirrLrGrrjrrrrrrrwr)rbrr3r6r5rrr)r8rirrLrGrrjrrrrrrr1Zprojection_stater r$r$r%rqms4$  zAltCLIPTextModel.forward)N) NNNNNNNNNNN)r<r=r>rrrPrr(r&rQr.rrzr9rALTCLIP_TEXT_INPUTS_DOCSTRINGrrr!rrrrrqrrr$r$rcr%r2Zs@   r2cs eZdZeZedfdd Zeedee j ee j ee j ee ee ee e j dddZ eedee j ee ee ee e j dd d Zeeeeed dee jee j ee j ee jee j ee ee ee ee eeefd d dZZS)rrcst|t|jts.tdt|jdt|jtsPtdt|jd|j}|j}|j |_ |j |_ |j |_ t||_t||_tj|j |j dd|_tj|j |j dd|_tt|jj|_|dS)NzRconfig.vision_config is expected to be of type AltCLIPVisionConfig but is of type .zNconfig.text_config is expected to be of type AltCLIPTextConfig but is of type F)r)rOrPr vision_configr TypeErrortype text_configrZprojection_dimr4rrRrr2 text_modelrr$rr}rrrr!rrbZlogit_scale_init_value logit_scaler%)r8rbr@r=rcr$r%rPs(     zAltCLIPModel.__init__N)rirrGrrrrc Csj|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j|||||||d}|d} || } | S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`AltCLIPTextModel`]. Examples: ```python >>> from transformers import AutoProcessor, AltCLIPModel >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```N)rirrGrLrrrr)rbrrrrAr) r8rirrGrLrrr text_outputsrZ text_featuresr$r$r%get_text_featuress   zAltCLIPModel.get_text_featuresrcCsd|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||d}|d}||}|S)a* Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`AltCLIPVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AltCLIPModel >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```Nr'r)rbrrrr$r)r8rrrrvision_outputsrZimage_featuresr$r$r%get_image_featuress zAltCLIPModel.get_image_featuresr) rirrrGrL return_lossrrrrc  Cs*|dk r |n|jj}|dk r |n|jj}| dk r4| n|jj} |j||||||| d} |j|||| d} | d} || } | d} || } | | jdddd} | | jdddd} |j }t | | |}|j}d}|rt|}| s||| | | | f}|dk r|f|S|St|||| | | | d S) al Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AltCLIPModel >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```N)rirrLrGrrrr'rrwrIT)rrZkeepdim)r+r,r-r.r/r0r1)rbrrrrAr$rrZnormrBexpr!rr(Tr)r*)r8rirrrGrLrGrrrrCrEr/r.rBr-r,r+rr$r$r%rq5sV$    zAltCLIPModel.forward)NNNNNNN)NNNN) NNNNNNNNN)r<r=r>rrrPrr;rr!rrr@rDr"rFALTCLIP_INPUTS_DOCSTRINGrr*Z LongTensorrrrqrrr$r$rcr%rsl.-  rcCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)nerzr!ZcumsumZtype_asr`)rirCrkmaskZincremental_indicesr$r$r%rfs rf)r)Ir?rZ dataclassesrtypingrrrrrr!Ztorch.nnrZtorch.utils.checkpointZ activationsr Zmodeling_outputsr r r r rZmodeling_utilsrZ pytorch_utilsrrrutilsrrrrZconfiguration_altcliprrrZ get_loggerr<rZ_CHECKPOINT_FOR_DOCZ_CONFIG_FOR_DOCZALTCLIP_START_DOCSTRINGr;r"rJrr&r)r*r(rBrsrrrrrrrrrrrrrr rr#r)r2rrfr$r$r$r%sd      (%Z4W^i2b#9:62\^