U 5AfB@sdZddlmZmZmZmZddlZddlmZddlm Z ddl m Z ddl m Z dd lmZmZmZdd lmZd d lmZd ZGdddejZGdddejZGdddejZGdddejZGdddejZGddde ZdZdZedeGdddeZ dS)zrPyTorch UperNet model. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation.)ListOptionalTupleUnionN)nn)CrossEntropyLoss)SemanticSegmenterOutput)PreTrainedModel)add_start_docstrings%add_start_docstrings_to_model_forwardreplace_return_docstrings) load_backbone) UperNetConfigrc szeZdZdZd eeeeeeeffeeeeefefeeeeeeffddfdd Z e j e j d d d Z Z S) UperNetConvModulez A convolutional block that bundles conv/norm/activation layers. This block simplifies the usage of convolution layers, which are commonly used with a norm layer (e.g., BatchNorm) and activation layer (e.g., ReLU). rFrN) in_channels out_channels kernel_sizepaddingbiasdilationreturncs<ttj||||||d|_t||_t|_dS)N)rrrrrr) super__init__rConv2dconvZ BatchNorm2d batch_normZReLU activation)selfrrrrrr __class__P/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/upernet/modeling_upernet.pyr(s  zUperNetConvModule.__init__inputrcCs"||}||}||}|SN)rrr)rr%outputr"r"r#forward=s   zUperNetConvModule.forward)rFr)__name__ __module__ __qualname____doc__intrrstrboolrtorchTensorr( __classcell__r"r"r r#r"s rcs<eZdZeeeddfdd ZejejdddZZS)UperNetPyramidPoolingBlockN) pool_scalerchannelsrcsLtt|t||ddg|_t|jD]\}}|t||q.dS)Nrr) rrrZAdaptiveAvgPool2drlayers enumerate add_moduler.)rr4rr5ilayerr r"r#rFs   z#UperNetPyramidPoolingBlock.__init__r$cCs|}|jD] }||}q |Sr&)r7)rr%Z hidden_stater;r"r"r#r(Os  z"UperNetPyramidPoolingBlock.forward) r)r*r+r-rr0r1r(r2r"r"r r#r3Es r3csNeZdZdZeedfeeeddfdd Zej e ej ddd Z Z S) UperNetPyramidPoolingModulea} Pyramid Pooling Module (PPM) used in PSPNet. Args: pool_scales (`Tuple[int]`): Pooling scales used in Pooling Pyramid Module. in_channels (`int`): Input channels. channels (`int`): Channels after modules, before conv_seg. align_corners (`bool`): align_corners argument of F.interpolate. .N) pool_scalesrr5 align_cornersrcsht||_||_||_||_g|_t|D]2\}}t|||d}|j || t ||q0dS)N)r4rr5) rrr=r>rr5blocksr8r3appendr9r.)rr=rr5r>r:r4blockr r"r#res  z$UperNetPyramidPoolingModule.__init__)xrcCsHg}|jD]8}||}tjj||ddd|jd}||q |S)Nbilinearsizemoder>)r?r functional interpolaterFr>r@)rrBZppm_outsppmZppm_outZupsampled_ppm_outr"r"r#r(qs  z#UperNetPyramidPoolingModule.forward) r)r*r+r,rr-r/rr0r1rr(r2r"r"r r#r<Vs" r<csLeZdZdZfddZddZddZdd Zej ej d d d Z Z S) UperNetHeadz Unified Perceptual Parsing for Scene Understanding. This head is the implementation of [UPerNet](https://arxiv.org/abs/1807.10221). cst||_|j|_||_|j|_d|_tj |j|j dd|_ t |j|jd|j|jd|_ t|jdt|j|j|jddd|_t|_t|_|jddD]@}t||jdd}t|j|jddd}|j||j|qtt|j|j|jddd|_dS)NFrr6)r>rrr)rrconfigr=rZ hidden_sizer5r>rr num_labels classifierr< psp_modulesrlen bottleneckZ ModuleList lateral_convs fpn_convsr@fpn_bottleneck)rrNrZl_convZfpn_convr r"r#rs@    zUperNetHead.__init__cCs||jdSr&apply _init_weightsrr"r"r# init_weightsszUperNetHead.init_weightscCs<t|tjr8|jjjd|jjd|jdk r8|jj dSNg)ZmeanZstd isinstancerrZweightdataZnormal_rNZinitializer_rangerZzero_rmoduler"r"r#rYs  zUperNetHead._init_weightscCs:|d}|g}|||tj|dd}||}|S)NrLrZdim)extendrQr0catrS)rinputsrBZpsp_outsr'r"r"r# psp_forwards  zUperNetHead.psp_forwardencoder_hidden_statesrcsfddtjDt}t|dddD]H}|djdd}|dtjj ||dj d|d<q@fd dt|dD}|dt|dddD]0}tjj |||djdddj d||<qt j |dd } |}|}|S) Ncsg|]\}}||qSr"r").0r:Z lateral_conv)rhr"r# sz'UperNetHead.forward..rrrLrCrDrEcsg|]}j||qSr")rU)rir:)lateralsrr"r#rjsrb)r8rTr@rfrRrangeshaperrHrIr>r0rdrVrP)rrhZused_backbone_levelsr:Z prev_shapeZfpn_outsr'r")rhrkrr#r(s0   zUperNetHead.forward) r)r*r+r,rr[rYrfr0r1r(r2r"r"r r#rK|s  ' rKcsbeZdZdZdeeeeeeeffddfdd Zd d Zd d Z e j e j d ddZ Z S)UperNetFCNHeada Fully Convolution Networks for Semantic Segmentation. This head is the implementation of [FCNNet](https://arxiv.org/abs/1411.4038>). Args: config: Configuration. in_channels (int): Number of input channels. kernel_size (int): The kernel size for convs in the head. Default: 3. dilation (int): The dilation rate for convs in the head. Default: 1. rCrrN)in_indexrrrc st||_|j|_|j|_|j|_|j |_ ||_ |d|}g}| t |j|j|||dt|jdD] }| t |j|j|||dqp|jdkrt|_n tj||_|j rt |j|j|j||dd|_tj|j|jdd|_dS)NrC)rrrrrrMr6)rrrNZauxiliary_in_channelsrZauxiliary_channelsr5Zauxiliary_num_convsZ num_convsZauxiliary_concat_input concat_inputror@rrlrZIdentityconvsZ Sequentialconv_catrrOrP)rrNrorrZ conv_paddingrqr:r r"r#rsN      zUperNetFCNHead.__init__cCs||jdSr&rWrZr"r"r#r[ szUperNetFCNHead.init_weightscCs<t|tjr8|jjjd|jjd|jdk r8|jj dSr\r]r`r"r"r#rYs  zUperNetFCNHead._init_weightsrgcCs@||j}||}|jr2|tj||gdd}||}|S)Nrrb)rorqrprrr0rdrP)rrh hidden_statesr'r"r"r#r(s    zUperNetFCNHead.forward)rCrr)r)r*r+r,r-rrrr[rYr0r1r(r2r"r"r r#rns$rnc@s,eZdZdZeZdZgZddZddZ dS)UperNetPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. pixel_valuescCs6t|tr2|j|j|jdk r2|jdSr&)r^rtbackboner[ decode_headauxiliary_headr`r"r"r#rY*s     z$UperNetPreTrainedModel._init_weightscCs,|j|j|jdk r(|jdS)zInitialize the weightsN)rvr[rwrxrZr"r"r#r[1s   z#UperNetPreTrainedModel.init_weightsN) r)r*r+r,r config_classZmain_input_nameZ_no_split_modulesrYr[r"r"r"r#rt s rtaI Parameters: This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. config ([`UperNetConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. ax Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`SegformerImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers in case the backbone has them. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers of the backbone. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zMUperNet framework leveraging any vision backbone e.g. for ADE20k, CityScapes.c speZdZfddZeedeee dd e e j e e e e e e j e e eeefdddZZS) UperNetForSemanticSegmentationcsHt|t||_t||jjd|_|jr6t|nd|_ | dS)N)r) rrrrvrKr5rwZuse_auxiliary_headrnrxZ post_init)rrNr r"r#rXs   z'UperNetForSemanticSegmentation.__init__zbatch_size, sequence_length) output_typeryN)ruoutput_attentionsoutput_hidden_stateslabels return_dictrcCsv|dk r|jjdkrtd|dk r(|n|jj}|dk r<|n|jj}|dk rP|n|jj}|jj|||d}|j}| |}t j j ||j ddddd}d} |jdk r||} t j j | |j ddddd} d} |dk rt|jjd } | ||} | dk r| | |} | |jj| 7} |sb|r8|f|dd} n|f|dd} | dk r^| f| S| St| ||j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, UperNetForSemanticSegmentation >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> image_processor = AutoImageProcessor.from_pretrained("openmmlab/upernet-convnext-tiny") >>> model = UperNetForSemanticSegmentation.from_pretrained("openmmlab/upernet-convnext-tiny") >>> filepath = hf_hub_download( ... repo_id="hf-internal-testing/fixtures_ade20k", filename="ADE_val_00000001.jpg", repo_type="dataset" ... ) >>> image = Image.open(filepath).convert("RGB") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits # shape (batch_size, num_labels, height, width) >>> list(logits.shape) [1, 150, 512, 512] ```Nrz/The number of labels should be greater than one)r}r|rCrDFrE)Z ignore_index)losslogitsrs attentions)rNrO ValueErrorZuse_return_dictr}r|rvZforward_with_filtered_kwargsZ feature_mapsrwrrHrIrmrxrZloss_ignore_indexZauxiliary_loss_weightr rsr)rrur|r}r~routputsfeaturesrZauxiliary_logitsrZloss_fctZauxiliary_lossr'r"r"r#r(dsR'        z&UperNetForSemanticSegmentation.forward)NNNNN)r)r*r+rr UPERNET_INPUTS_DOCSTRINGformatr r _CONFIG_FOR_DOCrr0r1r/rtupler(r2r"r"r r#rzSs    rz)!r,typingrrrrr0rZtorch.nnrZmodeling_outputsr Zmodeling_utilsr utilsr r r Zutils.backbone_utilsrZconfiguration_upernetrrModulerr3r<rKrnrtZUPERNET_START_DOCSTRINGrrzr"r"r"r#s.      #&]G