U 5Af=@sDdZddlmZmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZmZmZdd lmZdd lmZmZmZmZmZd d lmZeeZd Z dZ!d dddgZ"dZ#dZ$d?e%e%ee%e%dddZ&e'de'dfe'e'e'e'dddZ(Gdddej)Z*Gdddej)Z+Gdddej)Z,Gd d!d!ej)Z-Gd"d#d#ej)Z.Gd$d%d%ej)Z/Gd&d'd'ej)Z0Gd(d)d)ej)Z1Gd*d+d+ej)Z2Gd,d-d-eZ3d.Z4d/Z5ed0e4Gd1d2d2e3Z6ed3e4Gd4d5d5e3Z7Gd6d7d7ej)Z8Gd8d9d9ej)Z9Gd:d;d;ej)Z:edd>e3Z;dS)@zPyTorch MobileViTV2 model.)OptionalTupleUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputWithNoAttention(BaseModelOutputWithPoolingAndNoAttention$ImageClassifierOutputWithNoAttentionSemanticSegmenterOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)MobileViTV2Configrz$apple/mobilevitv2-1.0-imagenet1k-256ztabby, tabby cat)valuedivisor min_valuereturncCsF|dkr |}t|t||d||}|d|kr>||7}t|S)a Ensure that all layers have a channel count that is divisible by `divisor`. This function is taken from the original TensorFlow repo. It can be seen here: https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/mobilenet.py Ng?)maxint)rrr new_valuer!X/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/mobilevitv2/modeling_mobilevitv2.pymake_divisible<s  r#z-infinf)rmin_valmax_valrcCst|t||SN)rminrr%r&r!r!r"clipKsr*c sTeZdZd eeeeeeeeeeeefdd fdd Ze j e j dd d Z Z S) MobileViTV2ConvLayerrFTN) config in_channels out_channels kernel_sizestridegroupsbiasdilationuse_normalizationuse_activationrc stt|dd|} ||dkr@td|d|d||dkrbtd|d|dtj||||| |||dd |_| rtj|d d d d d |_nd|_| rt | t rt | |_ qt |j t rt |j |_ q|j |_ nd|_ dS)NrrrzInput channels (z) are not divisible by z groups.zOutput channels (zeros) r-r.r/r0paddingr3r1r2Z padding_modegh㈵>g?T)Z num_featuresepsZmomentumZaffineZtrack_running_stats)super__init__r ValueErrorrConv2d convolutionZ BatchNorm2d normalization isinstancestrr activationZ hidden_act) selfr,r-r.r/r0r1r2r3r4r5r7 __class__r!r"r:QsB         zMobileViTV2ConvLayer.__init__featuresrcCs6||}|jdk r||}|jdk r2||}|Sr')r=r>rA)rBrFr!r!r"forwards      zMobileViTV2ConvLayer.forward)rrFrTT) __name__ __module__ __qualname__rrboolrr@r:torchTensorrG __classcell__r!r!rCr"r+Ps( 6r+csFeZdZdZd eeeeeddfdd Zejejddd Z Z S) MobileViTV2InvertedResidualzQ Inverted residual block (MobileNetv2): https://arxiv.org/abs/1801.04381 rN)r,r-r.r0r3rc stttt||jd}|dkr:td|d|dkoH||k|_t|||dd|_ t|||d|||d|_ t|||dd d |_ dS) Nr)rrzInvalid stride .r)r-r.r/r )r-r.r/r0r1r3Fr-r.r/r5) r9r:r#rroundZ expand_ratior; use_residualr+ expand_1x1conv_3x3 reduce_1x1)rBr,r-r.r0r3Zexpanded_channelsrCr!r"r:s6  z$MobileViTV2InvertedResidual.__init__rEcCs4|}||}||}||}|jr0||S|Sr')rTrUrVrS)rBrFZresidualr!r!r"rGs    z#MobileViTV2InvertedResidual.forward)r rHrIrJ__doc__rrr:rLrMrGrNr!r!rCr"rOs!rOcsBeZdZd eeeeeddfdd ZejejdddZZ S) MobileViTV2MobileNetLayerrN)r,r-r.r0 num_stagesrcsRtt|_t|D]0}t||||dkr4|ndd}|j||}qdS)Nrr)r-r.r0)r9r:r ModuleListlayerrangerOappend)rBr,r-r.r0rZir\rCr!r"r:s    z"MobileViTV2MobileNetLayer.__init__rEcCs|jD] }||}q|Sr'r\)rBrF layer_moduler!r!r"rGs  z!MobileViTV2MobileNetLayer.forward)rr rHrIrJrrr:rLrMrGrNr!r!rCr"rYsrYcs>eZdZdZeeddfdd ZejejdddZ Z S) MobileViTV2LinearSelfAttentionaq This layer applies a self-attention with linear complexity, as described in MobileViTV2 paper: https://arxiv.org/abs/2206.02680 Args: config (`MobileVitv2Config`): Model configuration object embed_dim (`int`): `input_channels` from an expected input of size :math:`(batch_size, input_channels, height, width)` N)r, embed_dimrc s\tt||dd|ddddd|_tj|jd|_t|||ddddd|_||_dS)NrrTF)r,r-r.r2r/r4r5p) r9r:r+qkv_projrDropout attn_dropoutout_projrd)rBr,rdrCr!r"r:s*    z'MobileViTV2LinearSelfAttention.__init__ hidden_statesrc Cs||}tj|d|j|jgdd\}}}tjjj|dd}||}||}tj|ddd}tjj || |}| |}|S)Nr)Zsplit_size_or_sectionsdimrmTrmZkeepdim) rgrLsplitrdr functionalZsoftmaxrisumreluZ expand_asrj) rBrlZqkvquerykeyrZcontext_scoresZcontext_vectoroutr!r!r"rGs    z&MobileViTV2LinearSelfAttention.forwardrWr!r!rCr"rcs rccs@eZdZd eeeeddfdd ZejejdddZ Z S) MobileViTV2FFNN)r,rdffn_latent_dim ffn_dropoutrc sZtt|||dddddd|_t||_t|||dddddd|_t||_dS)NrTF)r,r-r.r/r0r2r4r5) r9r:r+conv1rrhdropout1conv2dropout2)rBr,rdrzr{rCr!r"r:s.   zMobileViTV2FFN.__init__rkcCs,||}||}||}||}|Sr')r|r}r~r)rBrlr!r!r"rG9s     zMobileViTV2FFN.forward)ry rHrIrJrrfloatr:rLrMrGrNr!r!rCr"rxs rxcs@eZdZd eeeeddfdd ZejejdddZ Z S) MobileViTV2TransformerLayerryN)r,rdrzdropoutrcsbttjd||jd|_t|||_tj|d|_ tjd||jd|_ t ||||j |_ dS)NrZ num_groups num_channelsr8re)r9r:r GroupNormlayer_norm_epslayernorm_beforerc attentionrhr}layernorm_afterrxr{ffn)rBr,rdrzrrCr!r"r:Bs   z$MobileViTV2TransformerLayer.__init__rkcCs<||}||}||}||}||}||}|Sr')rrrr)rBrlZlayernorm_1_outZattention_outputZ layer_outputr!r!r"rGPs    z#MobileViTV2TransformerLayer.forward)ryrr!r!rCr"rAsrcs<eZdZeeeddfdd ZejejdddZZ S)MobileViTV2TransformerN)r,n_layersd_modelrcsft|j}||g|}dd|D}t|_t|D]"}t||||d}|j|q>dS)NcSsg|]}t|ddqS))r).0dr!r!r" esz3MobileViTV2Transformer.__init__..)rdrz) r9r:ffn_multiplierrr[r\r]rr^)rBr,rrrZffn_dimsZ block_idxtransformer_layerrCr!r"r:]s   zMobileViTV2Transformer.__init__rkcCs|jD] }||}q|Sr'r`)rBrlrar!r!r"rGns  zMobileViTV2Transformer.forwardrbr!r!rCr"r\src seZdZdZdeeeeeeeddfdd Zeje eje eeffdd d Z eje eefejd d d Z ejejdddZ Z S)MobileViTV2Layerz= MobileViTV2 layer: https://arxiv.org/abs/2206.02680 rrN)r,r-r. attn_unit_dim n_attn_blocksr3r0rc st|j|_|j|_|}|dkr\t||||dkr:|nd|dkrL|dndd|_|}nd|_t||||j|d|_ t|||dddd|_ t |||d|_ t jd||jd|_t|||dd dd|_dS) Nrr)r-r.r0r3)r-r.r/r1F)r-r.r/r4r5)rrrT)r9r: patch_size patch_width patch_heightrOdownsampling_layerr+Zconv_kernel_sizeconv_kxkconv_1x1r transformerrrr layernormconv_projection) rBr,r-r.rrr3r0Z cnn_out_dimrCr!r"r:ysN  zMobileViTV2Layer.__init__) feature_maprcCsT|j\}}}}tjj||j|jf|j|jfd}||||j|jd}|||ffS)N)r/r0rn)shaperrrZunfoldrrreshape)rBr batch_sizer-Z img_heightZ img_widthpatchesr!r!r" unfoldings  zMobileViTV2Layer.unfolding)r output_sizercCsH|j\}}}}|||||}tjj|||j|jf|j|jfd}|S)N)rr/r0)rrrrrfoldrr)rBrrrZin_dimrZ n_patchesrr!r!r"foldings  zMobileViTV2Layer.foldingrEcCs`|jr||}||}||}||\}}||}||}|||}||}|Sr')rrrrrrrr)rBrFrrr!r!r"rGs       zMobileViTV2Layer.forward)rrr)rHrIrJrXrrr:rLrMrrrrGrNr!r!rCr"rts" =$  rcsDeZdZeddfdd Zd ejeeee e fddd Z Z S) MobileViTV2EncoderNr,rcst||_t|_d|_d}}|jdkrMsz-MobileViTV2Encoder.forward..)last_hidden_staterl) enumerater\rZtrainingZ_gradient_checkpointing_func__call__tupler )rBrlrrZall_hidden_statesr_rar!r!r"rG8s   zMobileViTV2Encoder.forward)FT) rHrIrJrr:rLrMrKrrr rGrNr!r!rCr"rsT rc@sFeZdZdZeZdZdZdZdgZ e e j e j e jfdddd ZdS) MobileViTV2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. mobilevitv2 pixel_valuesTrN)modulercCsjt|tjtjfr@|jjjd|jjd|j dk rf|j j n&t|tj rf|j j |jj ddS)zInitialize the weightsry)meanZstdNg?) r?rLinearr<ZweightdataZnormal_r,Zinitializer_ranger2Zzero_ LayerNormZfill_)rBrr!r!r" _init_weights_s   z(MobileViTV2PreTrainedModel._init_weights)rHrIrJrXr config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesrrrr<rrr!r!r!r"rSsraM This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MobileViTV2Config`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aF Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`MobileViTImageProcessor.__call__`] for details. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zYThe bare MobileViTV2 model outputting raw hidden-states without any specific head on top.c steZdZd eedfdd ZddZeee e e e de ddeejeeeeeee fd d d ZZS)MobileViTV2ModelT)r, expand_outputc sft|||_||_ttd|jdddddd}t||j|ddd d d |_ t ||_ | dS) Nrrrr)rrr rTr-r.r/r0r4r5) r9r:r,rr#r*rr+r conv_stemrencoder post_init)rBr,rrrCr!r"r:s&  zMobileViTV2Model.__init__cCsF|D]8\}}|jj|}t|tr|jjD]}|j|q.qdS)zPrunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrr\r?rrrZ prune_heads)rBZheads_to_pruneZ layer_indexZheadsZmobilevitv2_layerrr!r!r" _prune_headss    zMobileViTV2Model._prune_headsZvision) checkpoint output_typerZmodalityexpected_outputN)rrrrc Cs|dk r |n|jj}|dk r |n|jj}|dkr8td||}|j|||d}|jrv|d}tj|ddgdd}n |d}d}|s|dk r||fn|f}||ddSt |||j d S) Nz You have to specify pixel_valuesrrrrnFrpr)r pooler_outputrl) r,ruse_return_dictr;rrrrLrr rl) rBrrrZembedding_outputZencoder_outputsr pooled_outputoutputr!r!r"rGs0 zMobileViTV2Model.forward)T)NNN)rHrIrJrrKr:rrMOBILEVITV2_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErrLrMrrrGrNr!r!rCr"rs&   rz MobileViTV2 model with an image classification head on top (a linear layer on top of the pooled features), e.g. for ImageNet. c speZdZeddfdd Zeeeee e e dd e e je ee e je eeee fdddZZS) !MobileViTV2ForImageClassificationNrcs`t||j|_t||_td|jdd}|jdkrJtj||jdnt |_ | dS)Nrrrr)Z in_featuresZ out_features) r9r: num_labelsrrr#rrrZIdentity classifierr)rBr,r.rCr!r"r:s  z*MobileViTV2ForImageClassification.__init__)rrrr)rrlabelsrrc Csl|dk r |n|jj}|j|||d}|r.|jn|d}||}d}|dk r,|jjdkr|jdkrnd|j_n4|jdkr|jtj ks|jtj krd|j_nd|j_|jjdkrt } |jdkr| | | }n | ||}nN|jjdkrt } | |d|j|d}n|jjdkr,t} | ||}|s\|f|dd} |dk rX|f| S| St|||jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss). If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrZ regressionZsingle_label_classificationZmulti_label_classificationrnr)losslogitsrl)r,rrrrZ problem_typerZdtyperLlongrrZsqueezerviewrr rl) rBrrrroutputsrrrloss_fctrr!r!r"rGs>     "     z)MobileViTV2ForImageClassification.forward)NNNN)rHrIrJrr:rrr_IMAGE_CLASS_CHECKPOINTr r_IMAGE_CLASS_EXPECTED_OUTPUTrrLrMrKrrrGrNr!r!rCr"rs& rcs<eZdZeeeddfdd ZejejdddZZ S)MobileViTV2ASPPPoolingN)r,r-r.rc s4ttjdd|_t|||ddddd|_dS)Nr)rTrtr)r9r:rZAdaptiveAvgPool2d global_poolr+r)rBr,r-r.rCr!r"r:5s zMobileViTV2ASPPPooling.__init__rEcCs:|jdd}||}||}tjj||ddd}|S)NrbilinearFsizemodeZ align_corners)rrrrrr interpolate)rBrFZ spatial_sizer!r!r"rGDs   zMobileViTV2ASPPPooling.forwardrbr!r!rCr"r4srcs<eZdZdZeddfdd ZejejdddZZ S) MobileViTV2ASPPzs ASPP module defined in DeepLab papers: https://arxiv.org/abs/1606.00915, https://arxiv.org/abs/1706.05587 Nrcsttdjdd}|jtjdkr.re)r9r:r#raspp_out_channelslenZ atrous_ratesr;rr[convsr+r^extendrprojectrhZaspp_dropout_probr)rBr,Zencoder_out_channelsZ in_projectionZ pool_layerrCrr"r:Qs<      zMobileViTV2ASPP.__init__rEcCsDg}|jD]}|||q tj|dd}||}||}|S)Nrro)rr^rLcatrr)rBrFZpyramidconvZpooled_featuresr!r!r"rG}s   zMobileViTV2ASPP.forward rHrIrJrXrr:rLrMrGrNr!r!rCr"rLs,rcs<eZdZdZeddfdd ZejejdddZZ S) MobileViTV2DeepLabV3zB DeepLabv3 architecture: https://arxiv.org/abs/1706.05587 Nrc sBtt||_t|j|_t||j |j ddddd|_ dS)NrFT)r-r.r/r4r5r2) r9r:raspprZ Dropout2dZclassifier_dropout_probrr+rrrrBr,rCr!r"r:s  zMobileViTV2DeepLabV3.__init__rkcCs&||d}||}||}|S)Nrn)rrr)rBrlrFr!r!r"rGs  zMobileViTV2DeepLabV3.forwardrr!r!rCr"rsrzZ MobileViTV2 model with a semantic segmentation head on top, e.g. for Pascal VOC. c sleZdZeddfdd Zeeeee dd e e j e e j e e e e eeefdddZZS) "MobileViTV2ForSemanticSegmentationNrcs8t||j|_t|dd|_t||_|dS)NF)r)r9r:rrrrsegmentation_headrrrCr!r"r:s   z+MobileViTV2ForSemanticSegmentation.__init__)rr)rrrrrc Cs|dk r |n|jj}|dk r |n|jj}|dk rD|jjdkrDtd|j|d|d}|r^|jn|d}||}d}|dk rtj j ||j ddddd } t |jj d } | | |}|s|r|f|dd} n|f|d d} |dk r|f| S| St|||r|jnddd S) a labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> import requests >>> import torch >>> from PIL import Image >>> from transformers import AutoImageProcessor, MobileViTV2ForSemanticSegmentation >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("apple/mobilevitv2-1.0-imagenet1k-256") >>> model = MobileViTV2ForSemanticSegmentation.from_pretrained("apple/mobilevitv2-1.0-imagenet1k-256") >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```Nrz/The number of labels should be greater than oneTrrrFr)Z ignore_indexr)rrrlZ attentions)r,rrrr;rrlrrrrrrrZsemantic_loss_ignore_indexr) rBrrrrrZencoder_hidden_statesrrZupsampled_logitsrrr!r!r"rGsB'   z*MobileViTV2ForSemanticSegmentation.forward)NNNN)rHrIrJrr:rrrrrrrLrMrKrrrGrNr!r!rCr"rs   r)rN)s`      "A1?)rm  TQ=