U 5Af- @sdZddlZddlZddlmZmZmZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZmZmZmZmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!m"Z"m#Z#m$Z$d d l%m&Z&e#'e(Z)dZ*dZ+ddZ,Gddde j-Z.Gddde j-Z/Gddde j-Z0Gddde j-Z1Gddde j-Z2Gddde j-Z3Gddde j-Z4Gd d!d!e j-Z5Gd"d#d#e j-Z6Gd$d%d%e j-Z7Gd&d'd'e j-Z8Gd(d)d)e j-Z9Gd*d+d+eZ:d,Z;d-Ze!d4e;Gd5d6d6e:Z?e!d7e;Gd8d9d9e:Z@e!d:e;Gd;d<dd?d?e:ZBe!d@e;GdAdBdBe:ZCdS)CzPyTorch RemBERT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) RemBertConfigrgoogle/rembertc szddl}ddl}ddl}Wn tk r<tdYnXtj|}t d||j |}g}g} |D]X\} t fdddDrqrt dd | |j |} || | qrt|| D]"\} d d d t d dDr"t dd q|} D]} |d| rL|d| }n| g}|ddksn|ddkrzt| d} n|ddks|ddkrt| d} nz|ddkrt| d} n`|ddkrt| d} nFzt| |d} Wn2tk rt dd Yq*YnXt|dkr*t|d}| |} q*| dddkr`t| d} n| dkrt|| } z,| j| jkrtd| jd | jd!Wn<tk r}z|j| j| jf7_W5d}~XYnXt d"t| | _ q|S)#z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from c3s|]}|kVqdSN).0Zdenynamer P/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/rembert/modeling_rembert.py Msz-load_tf_weights_in_rembert..)adam_vadam_mZoutput_embeddingclszLoading TF weight z with shape zbert/zrembert//css|]}|dkVqdS))r&r'ZAdamWeightDecayOptimizerZAdamWeightDecayOptimizer_1Z global_stepNr )r!nr r r$r%^sz Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweightZ output_biasbetabiasZoutput_weightsZsquad classifierz Skipping {}riZ _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )!reZnumpyZ tensorflow ImportErrorloggererrorospathabspathinfoZtrainZlist_variablesanyZ load_variableappendzipreplacesplitjoin fullmatchgetattrAttributeErrorformatlenint transposeshape ValueErrorAssertionErrorargstorchZ from_numpydata)modelconfigZtf_checkpoint_pathr2nptfZtf_pathZ init_varsnamesZarraysrGarraypointerZm_nameZ scope_namesnumer r"r$load_tf_weights_in_rembert7s~                    rVcsTeZdZdZfddZd eejeejeejeeje ej dddZ Z S) RemBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|jdt|jddddS)N) padding_idxZeps position_ids)rF) persistent)super__init__r Embedding vocab_sizeinput_embedding_size pad_token_idword_embeddingsZmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutZregister_bufferrKZarangeexpandselfrN __class__r r$r^s zRemBertEmbeddings.__init__Nr) input_idstoken_type_idsrZ inputs_embedspast_key_values_lengthreturnc Cs|dk r|}n|dd}|d}|dkrL|jdd|||f}|dkrjtj|tj|jjd}|dkr|||}||}||} ||} | | 7} | | } | | } | S)Nr[rdtypedevice) sizerZrKzeroslongrwrcrerdrfrj) rmrprqrZrrrs input_shape seq_lengthre embeddingsrdr r r$forwards"      zRemBertEmbeddings.forward)NNNNr) __name__ __module__ __qualname____doc__r^rrK LongTensor FloatTensorrETensorr~ __classcell__r r rnr$rWs rWcs0eZdZfddZejejdddZZS) RemBertPoolercs*tt|j|j|_t|_dSr)r]r^rLinear hidden_sizedenseZTanh activationrlrnr r$r^s zRemBertPooler.__init__ hidden_statesrtcCs(|dddf}||}||}|S)Nr)rr)rmrZfirst_token_tensor pooled_outputr r r$r~s  zRemBertPooler.forwardrrrr^rKrr~rr r rnr$rs rc sfeZdZfddZddZd ejeejeejeejeeje e eje e ddd Z Z S) RemBertSelfAttentioncst|j|jdkr>t|ds>td|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|j|_dS)NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())r]r^rnum_attention_headshasattrrHrEattention_head_size all_head_sizerrquerykeyvaluerhZattention_probs_dropout_probrj is_decoderrlrnr r$r^s zRemBertSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)Nr[rr1rr )rxrrviewpermute)rmxZ new_x_shaper r r$transpose_for_scoress z)RemBertSelfAttention.transpose_for_scoresNFrattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsrtcCs||}|dk } | r4|dk r4|d} |d} |}n| r^|||} |||} |}nv|dk r|||} |||} tj|d| gdd} tj|d| gdd} n |||} |||} ||} |jr| | f}t| | dd} | t |j } |dk r"| |} t j j| dd}||}|dk rN||}t|| }|dddd}|dd|jf}|j|}|r||fn|f}|jr||f}|S)Nrrr1Zdimr[r )rrrrrKcatrmatmulrFmathsqrtrrZ functionalZsoftmaxrjr contiguousrxrr)rmrrrrrrrZmixed_query_layerZis_cross_attentionZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr r r$r~sH         zRemBertSelfAttention.forward)NNNNNF) rrrr^rrKrrrrboolr~rr r rnr$rs$  rcs4eZdZfddZejejejdddZZS)RemBertSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrY) r]r^rrrrrfrgrhrirjrlrnr r$r^;s zRemBertSelfOutput.__init__r input_tensorrtcCs&||}||}|||}|Srrrjrfrmrrr r r$r~As  zRemBertSelfOutput.forwardrr r rnr$r:s rc steZdZfddZddZd ejeejeejeejeejee e ejee e ejddd Z Z S) RemBertAttentioncs*tt||_t||_t|_dSr)r]r^rrmroutputset pruned_headsrlrnr r$r^Is   zRemBertAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)rDrrmrrrrrrrrrrunion)rmheadsindexr r r$ prune_headsPs zRemBertAttention.prune_headsNFrc Cs<||||||||}||d|} | f|dd} | S)Nrr)rmr) rmrrrrrrrZ self_outputsattention_outputrr r r$r~cs  zRemBertAttention.forward)NNNNNF) rrrr^rrKrrrrrr~rr r rnr$rHs$ rcs0eZdZfddZejejdddZZS)RemBertIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) r]r^rrrintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrlrnr r$r^}s   zRemBertIntermediate.__init__rcCs||}||}|Sr)rrrmrr r r$r~s  zRemBertIntermediate.forwardrr r rnr$r|s rcs4eZdZfddZejejejdddZZS) RemBertOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r]r^rrrrrrfrgrhrirjrlrnr r$r^s zRemBertOutput.__init__rcCs&||}||}|||}|Srrrr r r$r~s  zRemBertOutput.forwardrr r rnr$rs rc steZdZfddZd ejeejeejeejeejeeeejee eejdddZ dd Z Z S) RemBertLayercsnt|j|_d|_t||_|j|_|j|_|jrV|jsLt|dt||_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is added)r]r^chunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionrHcrossattentionr intermediaterrrlrnr r$r^s    zRemBertLayer.__init__NFrc Cs|dk r|ddnd}|j|||||d} | d} |jrP| dd} | d} n | dd} d} |jr|dk rt|dstd|d|dk r|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) Nr1)rrrrr[rz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrrHrrfeed_forward_chunkrr)rmrrrrrrrZself_attn_past_key_valueZself_attention_outputsrrZpresent_key_valueZcross_attn_present_key_valueZcross_attn_past_key_valueZcross_attention_outputs layer_outputr r r$r~sV         zRemBertLayer.forwardcCs||}|||}|Sr)rr)rmrZintermediate_outputrr r r$rs  zRemBertLayer.feed_forward_chunk)NNNNNF) rrrr^rKrrrrrr~rrr r rnr$rs$  BrcsteZdZfddZd ejeejeejeejeejeeeejee e e e e ee fd ddZ Z S) RemBertEncodercsLt|_tjj|_tfddt j D|_ d|_ dS)Ncsg|] }tqSr )r)r!_rNr r$ sz+RemBertEncoder.__init__..F) r]r^rNrrrarembedding_hidden_mapping_inZ ModuleListrangenum_hidden_layerslayergradient_checkpointingrlrnrr$r^s   zRemBertEncoder.__init__NFT) rrrrrpast_key_values use_cacheroutput_hidden_states return_dictrtc  Csl|jr|jr|rtdd}||}| r0dnd} |r.)last_hidden_staterr attentionscross_attentions) rZtrainingr4Z warning_oncerrNr enumeraterZ_gradient_checkpointing_func__call__tupler )rmrrrrrrrrrrZall_hidden_statesZall_self_attentionsZall_cross_attentionsZnext_decoder_cacheiZ layer_moduleZlayer_head_maskrZ layer_outputsr r r$r~sz           zRemBertEncoder.forward) NNNNNNFFT)rrrr^rKrrrrrrr r~rr r rnr$rs.  rcs0eZdZfddZejejdddZZS)RemBertPredictionHeadTransformcsVtt|j|j|_t|jtr6t |j|_ n|j|_ tj |j|j d|_ dSr) r]r^rrrrrrrr transform_act_fnrfrgrlrnr r$r^Ss   z'RemBertPredictionHeadTransform.__init__rcCs"||}||}||}|Sr)rrrfrr r r$r~\s   z&RemBertPredictionHeadTransform.forwardrr r rnr$rRs rcs0eZdZfddZejejdddZZS)RemBertLMPredictionHeadcsRtt|j|j|_t|j|j|_t |j |_ tj |j|j d|_ dSr)r]r^rrrZoutput_embedding_sizerr`decoderr rrrfrgrlrnr r$r^ds   z RemBertLMPredictionHead.__init__rcCs,||}||}||}||}|Sr)rrrfrrr r r$r~ks     zRemBertLMPredictionHead.forwardrr r rnr$rcs rcs0eZdZfddZejejdddZZS)RemBertOnlyMLMHeadcstt||_dSr)r]r^r predictionsrlrnr r$r^us zRemBertOnlyMLMHead.__init__)sequence_outputrtcCs||}|Sr)r)rmrprediction_scoresr r r$r~ys zRemBertOnlyMLMHead.forwardrr r rnr$rts rc@s(eZdZdZeZeZdZdZ ddZ dS)RemBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. rembertTcCst|tjr:|jjjd|jjd|jdk r|jj nft|tj rz|jjjd|jjd|j dk r|jj|j  n&t|tj r|jj |jj ddS)zInitialize the weightsg)ZmeanZstdNg?)rrrr-rLZnormal_rNZinitializer_ranger/Zzero_r_rXrfZfill_)rmmoduler r r$ _init_weightss      z$RemBertPreTrainedModel._init_weightsN) rrrrr config_classrVZload_tf_weightsZbase_model_prefixZsupports_gradient_checkpointingrr r r r$r~s raJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`RemBertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a5 Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zaThe bare RemBERT Model transformer outputting raw hidden-states without any specific head on top.cseZdZdZdfdd ZddZddZd d Zee d e d e e d dejeejeejeejeejeejeejeejeeeejeeeeeeeeeeefdddZZS) RemBertModela The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. TcsDt|||_t||_t||_|r2t|nd|_| dSr) r]r^rNrWr}rencoderrpooler post_init)rmrNadd_pooling_layerrnr r$r^s    zRemBertModel.__init__cCs|jjSrr}rcrmr r r$get_input_embeddingssz!RemBertModel.get_input_embeddingscCs ||j_dSrr)rmrr r r$set_input_embeddingssz!RemBertModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rmZheads_to_prunerrr r r$ _prune_headsszRemBertModel._prune_headsbatch_size, sequence_lengthr checkpoint output_typerN)rprrqrZrrrrrrrrrrrtc Cs&| dk r | n|jj} | dk r | n|jj} | dk r4| n|jj} |jjrZ| dk rP| n|jj} nd} |dk rx|dk rxtdn@|dk r||||}n"|dk r|dd}ntd|\}}|dk r|j n|j }| dk r| ddj dnd}|dkrt j |||f|d}|dkr.t j |t j|d }|||}|jjr|dk r|\}}}||f}|dkr|t j ||d}||}nd}|||jj}|j|||||d }|j|||||| | | | | d }|d}|jdk r||nd}| s ||f|d dSt|||j|j|j|jd S)a  encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timer[z5You have to specify either input_ids or inputs_embedsrr1)rwru)rprZrqrrrs) rrrrrrrrrr)rZ pooler_outputrrrr)rNrruse_return_dictrrrHZ%warn_if_padding_and_no_attention_maskrxrwrGrKZonesryrzZget_extended_attention_maskZinvert_attention_maskZ get_head_maskrr}rrr rrrr)rmrprrqrZrrrrrrrrrrr{Z batch_sizer|rwrsZextended_attention_maskZencoder_batch_sizeZencoder_sequence_lengthrZencoder_hidden_shapeZencoder_extended_attention_maskZembedding_outputZencoder_outputsrrr r r$r~s|)         zRemBertModel.forward)T) NNNNNNNNNNNNN)rrrrr^rrrrREMBERT_INPUTS_DOCSTRINGrCrr _CONFIG_FOR_DOCrKrrrrrrr r~rr r rnr$rsP    rz5RemBERT Model with a `language modeling` head on top.cseZdZdgZfddZddZddZee de d e e d de jee jee jee jee jee jee jee jee jeeeeeeeee fd d dZdddZZS)RemBertForMaskedLMcls.predictions.decoder.weightcs@t||jrtdt|dd|_t||_| dS)NznIf you want to use `RemBertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Fr r]r^rr4warningrrrr(rrlrnr r$r^s  zRemBertForMaskedLM.__init__cCs |jjjSrr(rrrr r r$get_output_embeddingssz(RemBertForMaskedLM.get_output_embeddingscCs||jj_dSrrrmZnew_embeddingsr r r$set_output_embeddingssz(RemBertForMaskedLM.set_output_embeddingsrrrN) rprrqrZrrrrrlabelsrrrrtc  Cs| dk r | n|jj} |j||||||||| | | d } | d}||}d}| dk rtt}||d|jj| d}| s|f| dd}|dk r|f|S|St||| j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. N) rrqrZrrrrrrrrrr[r1losslogitsrr) rNrrr(rrr`rrr)rmrprrqrZrrrrrr rrrrrrZmasked_lm_lossloss_fctrr r r$r~s: zRemBertForMaskedLM.forwardcKs~|j}|d}|jjdk s"tdtj|||jddfgdd}tj|df|jjtj|j d}tj||gdd}||dS)Nrz.The PAD token should be defined for generationrr[rru)rpr) rGrNrbrIrKrZ new_zerosfullrzrw)rmrpr model_kwargsr{Zeffective_batch_sizeZ dummy_tokenr r r$prepare_inputs_for_generations"z0RemBertForMaskedLM.prepare_inputs_for_generation) NNNNNNNNNNNN)N)rrr_tied_weights_keysr^rr rrrCrrrrKrrrrrrr~rrr r rnr$rsL   8rzIRemBERT Model with a `language modeling` head on top for CLM fine-tuning.cseZdZdgZfddZddZddZee de e e d de jee jee jee jee jee jee jee jeeee jee jeeeeeeeeeee fd d d ZdddZddZZS)RemBertForCausalLMrcs@t||jstdt|dd|_t||_| dS)NzOIf you want to use `RemBertForCausalLM` as a standalone, add `is_decoder=True.`Frrrlrnr r$r^s    zRemBertForCausalLM.__init__cCs |jjjSrrrr r r$rsz(RemBertForCausalLM.get_output_embeddingscCs||jj_dSrrr r r r$r sz(RemBertForCausalLM.set_output_embeddingsr)rrN)rprrqrZrrrrrrr rrrrrtcCs|dk r |n|jj}|j||||||||| | | | |d }|d}||}d}| dk r|ddddddf}| ddddf} t}||d|jj| d}|s|f|dd}|dk r|f|S|St|||j |j |j |j dS)a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Returns: Example: ```python >>> from transformers import AutoTokenizer, RemBertForCausalLM, RemBertConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/rembert") >>> config = RemBertConfig.from_pretrained("google/rembert") >>> config.is_decoder = True >>> model = RemBertForCausalLM.from_pretrained("google/rembert", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```N) rrqrZrrrrrrrrrrrr[rr1)r rrrrr) rNrrr(rrrr`r rrrr)rmrprrqrZrrrrrrr rrrrrrrZlm_lossZshifted_prediction_scoresrrr r r$r~sF< zRemBertForCausalLM.forwardcKst|j}|dkr||}|dk rh|ddjd}|jd|krF|}n|jdd}|dd|df}|||dS)Nrr1r)rprr)rGZnew_ones)rmrprrrr{Z past_lengthZremove_prefix_lengthr r r$rhs z0RemBertForCausalLM.prepare_inputs_for_generationcsBd}|D]4}|tfdd|ddD|ddf7}q|S)Nr c3s"|]}|d|jVqdS)rN)Z index_selecttorw)r!Z past_statebeam_idxr r$r%sz4RemBertForCausalLM._reorder_cache..r1)r)rmrrZreordered_pastZ layer_pastr rr$_reorder_cache~s z!RemBertForCausalLM._reorder_cache)NNNNNNNNNNNNNN)NN)rrrrr^rr rrrCrr rrKrrrrrrr~rrrr r rnr$rsN    d rz RemBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZfddZeededee dd e j e e j e e j e e j e e j e e j e e j e ee ee eeeefd dd ZZS) RemBertForSequenceClassificationcsJt||j|_t||_t|j|_t |j |j|_ | dSr r]r^ num_labelsrrrrhclassifier_dropout_probrjrrr0rrlrnr r$r^s   z)RemBertForSequenceClassification.__init__rrrN rprrqrZrrrr rrrrtc  Cs|| dk r | n|jj} |j|||||||| | d } | d} || } || } d}|dk r8|jjdkr|jdkrzd|j_n4|jdkr|jtj ks|jtj krd|j_nd|j_|jjdkrt }|jdkr|| | }n || |}nN|jjdkrt }|| d|j|d}n|jjdkr8t}|| |}| sh| f| dd}|dk rd|f|S|St|| | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrqrZrrrrrrrZ regressionZsingle_label_classificationZmulti_label_classificationr[r1r )rNrrrjr0Z problem_typerrvrKrzrErsqueezerrrrrr)rmrprrqrZrrrr rrrrrrr rrr r r$r~sV       "     z(RemBertForSequenceClassification.forward) NNNNNNNNNN)rrrr^rrrCrrrrKrrrrrrr~rr r rnr$rs<  rz RemBERT Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. cseZdZfddZeededee dd e j e e j e e j e e j e e j e e j e e j e ee ee eeeefd dd ZZS) RemBertForMultipleChoicecs@t|t||_t|j|_t|j d|_ | dS)Nr) r]r^rrrrhrrjrrr0rrlrnr r$r^s   z!RemBertForMultipleChoice.__init__z(batch_size, num_choices, sequence_lengthrrNrc  Cst| dk r | n|jj} |dk r&|jdn|jd} |dk rJ|d|dnd}|dk rh|d|dnd}|dk r|d|dnd}|dk r|d|dnd}|dk r|d|d|dnd}|j|||||||| | d } | d} || } || }|d| }d}|dk r0t}|||}| s`|f| dd}|dk r\|f|S|St ||| j | j dS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr[rrr1r ) rNrrGrrxrrjr0rrrr)rmrprrqrZrrrr rrrZ num_choicesrrrZreshaped_logitsr rrr r r$r~sL      z RemBertForMultipleChoice.forward) NNNNNNNNNN)rrrr^rrrCrrrrKrrrrrrr~rr r rnr$r s<  r z RemBERT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZfddZeededee dd e j e e j e e j e e j e e j e e j e e j e ee ee eeeefd dd ZZS) RemBertForTokenClassificationcsNt||j|_t|dd|_t|j|_t |j |j|_ | dSNFrrrlrnr r$r^Ks  z&RemBertForTokenClassification.__init__rrrNrc  Cs| dk r | n|jj} |j|||||||| | d } | d} || } || } d}|dk rxt}|| d|j|d}| s| f| dd}|dk r|f|S|St|| | j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr[r1r ) rNrrrjr0rrrrrr)rmrprrqrZrrrr rrrrrrr rrr r r$r~Vs8   z%RemBertForTokenClassification.forward) NNNNNNNNNN)rrrr^rrrCrrrrKrrrrrrr~rr r rnr$r!Cs<  r!z RemBERT Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cseZdZfddZeededee dd e j e e j e e j e e j e e j e e j e e j e e j e ee ee eeeefd dd ZZS) RemBertForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSr") r]r^rrrrrr qa_outputsrrlrnr r$r^s  z$RemBertForQuestionAnswering.__init__rrrN) rprrqrZrrrstart_positions end_positionsrrrrtc  CsD| dk r | n|jj} |j||||||| | | d } | d} || }|jddd\}}|d}|d}d}|dk r|dk rt|dkr|d}t|dkr|d}|d}|d||d|t |d}|||}|||}||d}| s.||f| dd}|dk r*|f|S|St |||| j | j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrr[r)Z ignore_indexr1)r  start_logits end_logitsrr) rNrrr$r>rrDrxZclamp_rrrr)rmrprrqrZrrrr%r&rrrrrrr'r(Z total_lossZ ignored_indexrZ start_lossZend_lossrr r r$r~sP             z#RemBertForQuestionAnswering.forward) NNNNNNNNNNN)rrrr^rrrCrrrrKrrrrrrr~rr r rnr$r#s@  r#)Drrr6typingrrrrKZtorch.utils.checkpointrZtorch.nnrrrZ activationsr Zmodeling_outputsr r r rrrrrZmodeling_utilsrZ pytorch_utilsrrrutilsrrrrrZconfiguration_rembertrZ get_loggerrr4rZ_CHECKPOINT_FOR_DOCrVModulerWrrrrrrrrrrrrZREMBERT_START_DOCSTRINGrrrrrr r!r#r r r r$s  (   S7j4X`  2*eYTG