U 4Af@sVdZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZmZdd lmZmZdd lmZdd lmZmZmZmZmZd d lmZmZm Z e!e"Z#dZ$e j%e j%dddZ&e j%e j%dddZ'eGdddeZ(eGdddeZ)eGdddeZ*Gddde j+Z,Gddde j+Z-Gdd d e j+Z.Gd!d"d"e j+Z/Gd#d$d$e j+Z0Gd%d&d&eZ1d'Z2d(Z3d)Z4d*Z5Gd+d,d,e j+Z6Gd-d.d.e j+Z7Gd/d0d0e1Z8Gd1d2d2e j+Z9Gd3d4d4e1Z:ee2Gd5d6d6e1Z;Gd7d8d8e j+ZdS)>zPyTorch CLIPSeg model.N) dataclass)AnyOptionalTupleUnion)nn)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) CLIPSegConfigCLIPSegTextConfigCLIPSegVisionConfigzCIDAS/clipseg-rd64-refined)logitsreturncCstj|tjt||jdS)Ndevice)r functionalZ cross_entropytorcharangelenr)rr P/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/clipseg/modeling_clipseg.pycontrastive_loss0sr") similarityrcCs t|}t|}||dS)Ng@)r"t)r#Z caption_lossZ image_lossr r r! clipseg_loss5s r%c@seZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ eed<dZeed <eed d d ZdS) CLIPSegOutputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image:(`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text:(`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegTextModel`]. image_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegVisionModel`]. text_model_output(`BaseModelOutputWithPooling`): The output of the [`CLIPSegTextModel`]. vision_model_output(`BaseModelOutputWithPooling`): The output of the [`CLIPSegVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrcstfddDS)Nc3s,|]$}|dkr|n t|VqdS))r,r-Ngetattrto_tuple.0kselfr r! [sz)CLIPSegOutput.to_tuple..tuplekeysr5r r5r!r1Zs zCLIPSegOutput.to_tuple)__name__ __module__ __qualname____doc__r'rr FloatTensor__annotations__r(r)r*r+r,r r-rrr1r r r r!r&;s   r&c@sLeZdZUdZdZejed<dZe e ejed<dZ e e ejed<dS)CLIPSegDecoderOutputa Args: logits (`torch.FloatTensor` of shape `(batch_size, height, width)`): Classification scores for each pixel. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr hidden_states attentions) r;r<r=r>rrr?r@rBrrrCr r r r!rAas  rAc@sxeZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ e ed<dZeed<eed d d ZdS) CLIPSegImageSegmentationOutputa, Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. ... vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPSegVisionModel`]. Nr'rconditional_embeddings pooled_outputr-decoder_outputr.cstfddDS)Nc3s,|]$}|dkr|n t|VqdS))r-rGNr/r2r5r r!r7sz:CLIPSegImageSegmentationOutput.to_tuple..r8r5r r5r!r1s z'CLIPSegImageSegmentationOutput.to_tuple)r;r<r=r>r'rrr?r@rrErFr-r rGrArrr1r r r r!rDus    rDcs>eZdZedfdd ZddZejejdddZ Z S) CLIPSegVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF)Z in_channelsZ out_channels kernel_sizestridebiasr position_idsr persistent)super__init__rJ hidden_size embed_dimZ image_size patch_sizer ParameterrZrandnclass_embeddingConv2dZ num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferrexpandr6rJ __class__r r!rUs"  z CLIPSegVisionEmbeddings.__init__cCst|dkrtdt|jd}|jjddjd|jj ||}t j j ||ddd d|jj |d|dj}t|jjdd|g}|S) NrNz#new_size should consist of 2 valuesg?rZbicubicF)modeZ align_cornersr)r ValueErrorintr]r`weightTviewrJrVrrZ interpolatesqueezercat)r6Znew_sizeZnum_patches_one_directionabresultr r r!interpolate_position_embeddingss$ z7CLIPSegVisionEmbeddings.interpolate_position_embeddings) pixel_valuesrcCs|jd}||}|ddd}|j|dd}tj||gdd}|jd|jkrt t |jdd}|| ||f}| |j}n|||j}|S)NrrNrrQdim)shaper\flatten transposerZrbrrmr^rhmathsqrtrqtodtyper`rO)r6rr batch_sizeZ patch_embedsZ class_embeds embeddingsZ new_shaper r r!forwards  zCLIPSegVisionEmbeddings.forward) r;r<r=rrUrqrr?Tensorr~ __classcell__r r rdr!rHsrHcsLeZdZedfdd Zdeejeejeejej dddZ Z S) CLIPSegTextEmbeddingsrIcsRt|j}t|j||_t|j||_|j dt |j ddddS)NrOrPFrR) rTrUrVrr_Z vocab_sizetoken_embeddingZmax_position_embeddingsr`rarrrbr6rJrWrdr r!rUs zCLIPSegTextEmbeddings.__init__N) input_idsrO inputs_embedsrcCsb|dk r|jdn|jd}|dkr:|jddd|f}|dkrL||}||}||}|S)NrQ)rurOrr`)r6rrOrZ seq_lengthZposition_embeddingsr}r r r!r~s  zCLIPSegTextEmbeddings.forward)NNN) r;r<r=rrUrr LongTensorr?rr~rr r rdr!rsrc sneZdZdZfddZejeedddZd eje eje eje e e eje ejfd d d Z Z S) CLIPSegAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ |j |_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).)rTrUrJrVrWnum_attention_heads num_headshead_dimrgscaleZattention_dropoutdropoutrLineark_projv_projq_projout_projrcrdr r!rUs  zCLIPSegAttention.__init__)tensorseq_lenbszcCs ||||j|jddS)NrrN)rkrrrw contiguous)r6rrrr r r!_shapeszCLIPSegAttention._shapeNFrBattention_maskcausal_attention_maskoutput_attentionsrcCs|\}}}|||j}|||d|} |||d|} ||jd|jf} ||||j| }| j| } | j| } | d} t || dd} | ||j|| fkrt d||j|| fd| |dk rD||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } |dk r||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } t jj| dd} |r| ||j|| }|||j|| } nd}t jj| |j|jd }t || }|||j||jfkrRt d ||j||jfd||||j||j}| dd}||||}||}||fS) z#Input shape: Batch x Time x ChannelrQrrNz$Attention weights should be of size z , but is Nz!Attention mask should be of size rs)ptrainingz `attn_output` should be of size )sizerrrrrrrrkrZbmmrwrgrrZsoftmaxrrZreshaper)r6rBrrrrZtgt_lenrWZ query_statesZ key_statesZ value_statesZ proj_shapeZsrc_len attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr r r!r~sX         zCLIPSegAttention.forward)NNF)r;r<r=r>rUrrrhrrboolrr~rr r rdr!rs rcs0eZdZfddZejejdddZZS) CLIPSegMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rTrUrJr hidden_act activation_fnrrrVintermediate_sizefc1fc2rcrdr r!rUXs   zCLIPSegMLP.__init__)rBrcCs"||}||}||}|Sr)rrr)r6rBr r r!r~_s   zCLIPSegMLP.forward)r;r<r=rUrrr~rr r rdr!rWs rcsJeZdZedfdd Zdejejejeee ej dddZ Z S) CLIPSegEncoderLayerrIcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)Zeps rTrUrVrWr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rcrdr r!rUhs    zCLIPSegEncoderLayer.__init__FrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r`||f7}|SaI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rBrrr)rrrrr6rBrrrZresidualroutputsr r r!r~ps"     zCLIPSegEncoderLayer.forward)F) r;r<r=rrUrrrrrr?r~rr r rdr!rgs rc@s$eZdZdZeZdZdZddZdS)CLIPSegPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. clipTcCsX|jj}t|trF|jjjjd|dd|jjjjd|ddnt|t r|jj}t j j|j d|j d|dt j j|jj|jj|dt j j|jj|jj|dnTt|trD|jj}|j dd|jjd|}|j d|}t j j|jj|dt j j|jj|dt j j|jj|dt j j|jj|dnt|tr|jj}|jjdd|jjd|}d|jjd|}t j j|jj|dt j j|jj|dnPt|trt j j|jj|jd|jjdt j j|jj|jd|jjdt|t jr.|j j!|jj"dt|t j#rT|j dk rT|j j!dS) zInitialize the weightsgg{Gz?)Zmeanstdr)rrNg?N)$rJZinitializer_factor isinstancerrridataZnormal_r`rHrinitrZrWr\Zinitializer_rangernum_hidden_layersrrrrrrVrr CLIPSegModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimrrMZzero_Zfill_r)r6moduleZfactorZ in_proj_stdZ out_proj_stdZfc_stdr r r! _init_weightssJ        z$CLIPSegPreTrainedModel._init_weightsN) r;r<r=r>r config_classZbase_model_prefixZsupports_gradient_checkpointingrr r r r!rs raI This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`CLIPSegConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. aE Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. c s`eZdZdZedfdd Zd eejeejee ee ee e e e fdddZ ZS) CLIPSegEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPSegEncoderLayer`]. Args: config: CLIPSegConfig rIcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr )rr3_rIr r! :sz+CLIPSegEncoder.__init__..F) rTrUrJr ModuleListrangerlayersgradient_checkpointingrcrdrIr!rU7s  zCLIPSegEncoder.__init__N)rrroutput_hidden_states return_dictrc Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|rDdnd}|rPdnd}|} t|jD]b\} } |rx|| f}|jr|jr|| j | |||} n| | |||d} | d} |rb|| df}qb|r|| f}|st dd| ||fDSt | ||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr )rrrcss|]}|dk r|VqdSrr r3vr r r!r7sz)CLIPSegEncoder.forward..)last_hidden_staterBrC) rJrruse_return_dict enumeraterrrZ_gradient_checkpointing_func__call__r9r ) r6rrrrrrZencoder_statesall_attentionsrBidxZ encoder_layer layer_outputsr r r!r~=sH&     zCLIPSegEncoder.forward)NNNNN)r;r<r=r>rrUrrrrrrr r~rr r rdr!r.s  rc sxeZdZedfdd Zeeeeedd e e j e e j e e j e e e e e e e eefdddZZS) CLIPSegTextTransformerrIcsHt||_|j}t||_t||_tj ||j d|_ |j |_ dSr) rTrUrJrVrr}rencoderrrrfinal_layer_norm eos_token_idrrdr r!rUs   zCLIPSegTextTransformer.__init__ output_typerNrrrOrrrrc Csn|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd|}|d|d}|j||d}t||j |j d} |dk rt ||j }|j ||| |||d} | d} | | } |jdkr| tj| jd| j d|jtj| j d jdd f} n>| tj| jd| j d|jtj| j d |jkjdd f} |sZ| | f| d dSt| | | j| jd S)  Returns: NzYou have to specify input_idsrQ)rrOr)rrrrrrrrN)r{rrsrr pooler_outputrBrC)rJrrrrgrrkr}r r{rr rrrrrrurzrhZargmaxr rBrC) r6rrrOrrrZ input_shaperBrencoder_outputsrrFr r r!r~s^     zCLIPSegTextTransformer.forward)NNNNNN)r;r<r=rrUrCLIPSEG_TEXT_INPUTS_DOCSTRINGrr rrrrrrr~rr r rdr!rs$   rc seZdZeZddgZedfdd ZejdddZ d d Z e e e eed deejeejeejeeeeeeeeefd ddZZS)CLIPSegTextModelrrrIcs"t|t||_|dSr)rTrUr text_model post_initrcrdr r!rUs  zCLIPSegTextModel.__init__r.cCs |jjjSrrr}rr5r r r!get_input_embeddingssz%CLIPSegTextModel.get_input_embeddingscCs||jj_dSrr)r6valuer r r!set_input_embeddingssz%CLIPSegTextModel.set_input_embeddingsrNrcCs|j||||||dS)aM Returns: Examples: ```python >>> from transformers import AutoTokenizer, CLIPSegTextModel >>> tokenizer = AutoTokenizer.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegTextModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrOrrr)r)r6rrrOrrrr r r!r~szCLIPSegTextModel.forward)NNNNNN)r;r<r=rrZ_no_split_modulesrUrModulerrrrrr rrrrrrr~rr r rdr!rs,  rc sheZdZedfdd Zeeeeedd e e j e e e e e e e eefdddZZS) CLIPSegVisionTransformerrIcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) rTrUrJrVrHr}rrr pre_layrnormrrpost_layernormrrdr r!rU)s   z!CLIPSegVisionTransformer.__init__rNrrrrrrc Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd||}||}|j||||d}|d}|dddddf}||}|s||f|ddSt |||j |j dS)rNz You have to specify pixel_values)rrrrrrr) rJrrrrgr}rrrr rBrC) r6rrrrrrBrrrFr r r!r~3s2   z CLIPSegVisionTransformer.forward)NNNN)r;r<r=rrUrCLIPSEG_VISION_INPUTS_DOCSTRINGrr rrr?rrrr~rr r rdr!r's   rc seZdZeZdZedfdd ZejdddZ e e e e edd eejeeeeeeeee fd d d ZZS)CLIPSegVisionModelrrrIcs"t|t||_|dSr)rTrUr vision_modelrrcrdr r!rUgs  zCLIPSegVisionModel.__init__r.cCs |jjjSr)rr}r\r5r r r!rmsz'CLIPSegVisionModel.get_input_embeddingsrNrcCs|j||||dS)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegVisionModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegVisionModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrrr)r)r6rrrrrr r r!r~ps zCLIPSegVisionModel.forward)NNNN)r;r<r=rrZmain_input_namerUrrrrrrr rrr?rrrr~rr r rdr!rcs"  rcseZdZeZedfdd Zeedee j ee j ee j ee ee ee e j dddZ eedee j ee ee ee e j dd d Zeeeeed dee jee j ee j ee jee ee ee ee eeefd d dZZS)rrIcst|t|jts.tdt|jdt|jtsPtdt|jd|j}|j}|j |_ |j |_ |j |_ t ||_t||_tj|j |j dd|_tj|j |j dd|_tt|jj|_|dS)NzNconfig.text_config is expected to be of type CLIPSegTextConfig but is of type .zRconfig.vision_config is expected to be of type CLIPSegVisionConfig but is of type F)rM)rTrUr text_configr TypeErrortype vision_configrprojection_dimrVrrrrrrrrrrrYrrrJZlogit_scale_init_value logit_scaler)r6rJrrrdr r!rUs(     zCLIPSegModel.__init__Nrc Csh|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||||d}|d}||} | S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegTextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, CLIPSegModel >>> tokenizer = AutoTokenizer.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```Nrr)rJrrrrr) r6rrrOrrr text_outputsrFZ text_featuresr r r!get_text_featuress  zCLIPSegModel.get_text_featuresrcCsd|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||d}|d}||}|S)aI Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```Nrr)rJrrrrr)r6rrrrrvision_outputsrFZimage_featuresr r r!get_image_featuress zCLIPSegModel.get_image_featuresr) rrrrrO return_lossrrrrc  Cs*|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|j||||d} |j||||||d} | d} || } | d} || } | | jdddd} | | jdddd} |j } t | | | }| }d}|rt|}|s||| | | | f}|dk r|f|S|St|||| | | | d S) a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```NrrrrNrQT)rrtZkeepdim)r'r(r)r*r+r,r-)rJrrrrrrrZnormrexprmatmulr$r%r&)r6rrrrrOrrrrrrr+r*rr)r(r'outputr r r!r~sT&    zCLIPSegModel.forward)NNNNNN)NNNN)NNNNNNNN)r;r<r=rrrUrrrrrrr?rrrCLIPSEG_INPUTS_DOCSTRINGrr&rrrr~rr r rdr!rsf .0  rcsNeZdZdZedfdd Zd ejejejee e ej dddZ Z S) CLIPSegDecoderLayerz CLIPSeg decoder layer, which is identical to `CLIPSegEncoderLayer`, except that normalization is applied after self-attention/MLP, rather than before. rIcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSrrrcrdr r!rUs    zCLIPSegDecoderLayer.__init__FrcCsd|}|j||||d\}}||}||}|}||}||}||}|f}|r`||f7}|Sr)rrrrrr r r!r~s"     zCLIPSegDecoderLayer.forward)F)r;r<r=r>rrUrrrrrr?r~rr r rdr!rys rcsNeZdZedfdd Zd eejejee ee ee dddZ Z S) CLIPSegDecoderrIc sXtj|_tjj|_tjj|_j rj j dj j df}t tj jjdddttjjjd|d|ddttjjdd|d|dd|_ntjjdj j j j d|_tj}tfd d t|D|_tj j_j_j_d _tfd d ttjD|_dS) Nrr)rKpaddingrNr)rKrL)rLcsg|]}tjjjqSr )rrrrV reduce_dimrrIr r!rsz+CLIPSegDecoder.__init__..Zrelucsg|] }tqSr )rr)decoder_configr r!rs)rTrUconditional_layerrrrrfilm_mulfilm_addZ"use_complex_transposed_convolutionrrXZ Sequentialr[ZReLUZConvTranspose2dtransposed_convolutionrextract_layersrrreducescopydeepcopyrVZdecoder_num_attention_headsrZdecoder_intermediate_sizerrr)r6rJZtransposed_kernelsdepthrd)rJr r!rUsN   zCLIPSegDecoder.__init__NT)rBrErrrcCsr|rdnd}|rdnd}|ddd}d} tt||j|jD]\} \} } } | dk rb| | | } n| | } | |jkr||| ddd||} | ddd} | | dd|d}|d} |r|| f7}|r>||df7}q>| ddddddfddd} tt | j d}|j d}| || j d||} | | d}|sdtdd|||fDSt|||d S) Nr rQrrrN)rrrcss|]}|dk r|VqdSrr rr r r!r7 sz)CLIPSegDecoder.forward..)rrBrC)rziprrr r Zpermuter rhrxryrurkr rlr9rA)r6rBrErrrZall_hidden_statesr activationsriZ activationZlayerreducerrr|rr r r!r~sH  "  $ zCLIPSegDecoder.forward)NNT) r;r<r=rrUrrrrrr~rr r rdr!rs.rzn CLIPSeg model with a Transformer-based decoder on top for zero-shot and one-shot image segmentation. cseZdZeZedfdd Zd eeej eej eej eej dddZ e e e eedd eejeejeejeejeej eejeejeeeeeeeeefd d d ZZS)CLIPSegForImageSegmentationrIcs:t|||_t||_|j|_t||_|dSr) rTrUrJrrrrdecoderrrcrdr r!rUs    z$CLIPSegForImageSegmentation.__init__Nr|rrrOconditional_pixel_valuesc Cs|dk rDt||krtdt|jj|||d}W5QRXnF|dk rt||kr`tdt|j|}W5QRXntd|S)Nz@Make sure to pass as many prompt texts as there are query images)rrOzAMake sure to pass as many prompt images as there are query imagesz[Invalid conditional, should be either provided as `input_ids` or `conditional_pixel_values`)rrgrno_gradrrr)r6r|rrrOrrEr r r!get_conditional_embeddings,s$    z6CLIPSegForImageSegmentation.get_conditional_embeddingsr) rrrrrErrOlabelsrrrrc  s| dk r | n|jj} t|jj||d| d} |j| d} | rL| jn| dfdd|jD} | rt | j | j | r| jnd| j d} n | s| dd| d dn| } W5QRX|dkr|j |jd ||||d }n8|jd |jd krtd |jd|jjkrtd |j| ||| | d}| r4|jn|d }d}|dk rh||j}t}|||}| s||| | |f}|dk r|f|S|St|||| | |dS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoProcessor, CLIPSegForImageSegmentation >>> from PIL import Image >>> import requests >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["a cat", "a remote", "a blanket"] >>> inputs = processor(text=texts, images=[image] * len(texts), padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> print(logits.shape) torch.Size([3, 352, 352]) ```NTrrrNcsg|]}|dqS)rr )r3rrBr r!rsz7CLIPSegForImageSegmentation.forward..rrrrzWMake sure to pass as many conditional embeddings as there are query images in the batchzcMake sure that the feature dimension of the conditional embeddings matches `config.projection_dim`.)rrr)r'rrErFr-rG)rJrrrrrrrBrr rrrCrrurgrrrrzrrZBCEWithLogitsLossrD)r6rrrrrErrOrrrrrrFrZdecoder_outputsrr'Zloss_fnrr rr!r~Isx,      z#CLIPSegForImageSegmentation.forward)NNNNN) NNNNNNNNNN)r;r<r=rrrUrhrrrrrrrrDrr?rrrrr&r~rr r rdr!rsN   r)?r>rrxZ dataclassesrtypingrrrrrZtorch.utils.checkpointrrr Zmodeling_attn_mask_utilsr r Zmodeling_outputsr r Zmodeling_utilsrutilsrrrrrZconfiguration_clipsegrrrZ get_loggerr;loggerZ_CHECKPOINT_FOR_DOCrr"r%r&rArDrrHrrrrrZCLIPSEG_START_DOCSTRINGrrrrrrrrrrrrr r r r!s`     $>"i24  'aa7<4b9d