U 4A·fm(ã@sÚdZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZmZmZddlmZmZdd lmZmZmZmZmZmZmZmZdd lmZmZdd l m!Z!m"Z"m#Z#dd l$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*d dl+m,Z,e) -e.¡Z/dZ0dZ1dIdd„Z2Gdd„de j3ƒZ4Gdd„de j3ƒZ5Gdd„de j3ƒZ6de5iZ7Gdd„de j3ƒZ8Gdd„de j3ƒZ9Gdd „d e j3ƒZ:Gd!d"„d"e j3ƒZ;Gd#d$„d$e j3ƒZGd)d*„d*eƒZ?eGd+d,„d,e%ƒƒZ@d-ZAd.ZBe'd/eAƒGd0d1„d1e?ƒƒZCGd2d3„d3e j3ƒZDe'd4eAƒGd5d6„d6e?ƒƒZEe'd7eAƒGd8d9„d9e?ƒƒZFe'd:eAƒGd;d<„dd?„d?e?ƒƒZHe'd@eAƒGdAdB„dBe?ƒƒZIe'dCeAƒGdDdE„dEe?ƒƒZJe'dFeAƒGdGdH„dHe?ƒƒZKdS)JzPyTorch ELECTRA model.éN)Ú dataclass)ÚListÚOptionalÚTupleÚUnion)Únn)ÚBCEWithLogitsLossÚCrossEntropyLossÚMSELossé)ÚACT2FNÚget_activation)Ú"BaseModelOutputWithCrossAttentionsÚ)BaseModelOutputWithPastAndCrossAttentionsÚ!CausalLMOutputWithCrossAttentionsÚMaskedLMOutputÚMultipleChoiceModelOutputÚQuestionAnsweringModelOutputÚSequenceClassifierOutputÚTokenClassifierOutput)ÚPreTrainedModelÚSequenceSummary)Úapply_chunking_to_forwardÚ find_pruneable_heads_and_indicesÚprune_linear_layer)Ú ModelOutputÚadd_code_sample_docstringsÚadd_start_docstringsÚ%add_start_docstrings_to_model_forwardÚloggingÚreplace_return_docstringsé)Ú ElectraConfigz"google/electra-small-discriminatorr"Ú discriminatorc Cs2zddl}ddl}ddl}Wn tk r<t d¡‚YnXtj |¡}t  d|›¡|j   |¡}g} g} |D]@\} } t  d| ›d| ›¡|j   || ¡} |   | ¡|   | ¡qrt| | ƒD]l\} } | }zt|tƒræ|  dd¡} |d kr|  d d ¡} |  d d ¡} |  d d¡} |  dd¡} |  d¡} tdd„| DƒƒrRt  d|›¡Wq¾|}| D]Ü}| d|¡rz| d|¡}n|g}|ddksœ|ddkr¨t|dƒ}nj|ddksÄ|ddkrÐt|dƒ}nB|ddkrêt|dƒ}n(|ddkrt|dƒ}nt||dƒ}t|ƒd krZt|d!ƒ}||}qZ| d"¡rPt|dƒ}n|dkrd| | ¡} z,|j| jkrŽtd#|j›d$| j›d%ƒ‚Wn<tk rÌ}z|j|j| jf7_‚W5d}~XYnXtd&| ›|ƒt | ¡|_ Wq¾t!k r*}ztd|›| |ƒWY¢q¾W5d}~XYq¾Xq¾|S)'z'Load tf checkpoints in a pytorch model.rNz™Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape zelectra/embeddings/zgenerator/embeddings/Ú generatorzelectra/zdiscriminator/z generator/Zdense_1Údense_predictionz!generator_predictions/output_biaszgenerator_lm_head/biasú/css|]}|dkVqdS))Z global_stepZ temperatureN©)Ú.0Únr'r'úP/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/electra/modeling_electra.pyÚ csz-load_tf_weights_in_electra..z Skipping z [A-Za-z]+_\d+z_(\d+)ÚkernelÚgammaÚweightZ output_biasÚbetaÚbiasZoutput_weightsZsquadÚ classifierér!Z _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )"ÚreZnumpyZ tensorflowÚ ImportErrorÚloggerÚerrorÚosÚpathÚabspathÚinfoZtrainZlist_variablesZ load_variableÚappendÚzipÚ isinstanceÚElectraForMaskedLMÚreplaceÚsplitÚanyÚ fullmatchÚgetattrÚlenÚintÚendswithÚ transposeÚshapeÚ ValueErrorÚargsÚprintÚtorchZ from_numpyÚdataÚAttributeError)ÚmodelÚconfigZtf_checkpoint_pathZdiscriminator_or_generatorr3ÚnpÚtfZtf_pathZ init_varsÚnamesZarraysÚnamerHÚarrayÚ original_nameÚpointerZm_nameZ scope_namesÚnumÚer'r'r*Úload_tf_weights_in_electra9s„ ÿ                       rZcsTeZdZdZ‡fdd„Zd eejeejeejeeje ej dœdd„Z ‡Z S) ÚElectraEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csºtƒ ¡tj|j|j|jd|_t |j|j¡|_ t |j |j¡|_ tj |j|j d|_ t |j¡|_|jdt |j¡ d¡ddt|ddƒ|_|jd tj|j ¡tjd dddS) N)Ú padding_idx©ZepsÚ position_ids)r!éÿÿÿÿF)Ú persistentÚposition_embedding_typeÚabsoluteÚtoken_type_ids©Údtype)ÚsuperÚ__init__rÚ EmbeddingÚ vocab_sizeÚembedding_sizeZ pad_token_idÚword_embeddingsÚmax_position_embeddingsÚposition_embeddingsZtype_vocab_sizeÚtoken_type_embeddingsÚ LayerNormÚlayer_norm_epsÚDropoutÚhidden_dropout_probÚdropoutZregister_bufferrLÚarangeÚexpandrCraÚzerosr^ÚsizeÚlong©ÚselfrP©Ú __class__r'r*rgŽs" ÿÿzElectraEmbeddings.__init__Nr)Ú input_idsrcr^Ú inputs_embedsÚpast_key_values_lengthÚreturnc Csø|dk r| ¡}n| ¡dd…}|d}|dkrL|jdd…|||…f}|dkr t|dƒrŠ|jdd…d|…f}| |d|¡} | }ntj|tj|jjd}|dkr²|  |¡}|  |¡} || } |j dkrà|  |¡} | | 7} |  | ¡} | | ¡} | S)Nr_r!rcr©reÚdevicerb)rwr^ÚhasattrrcrurLrvrxr‚rkrnrarmrors) rzr}rcr^r~rÚ input_shapeÚ seq_lengthÚbuffered_token_type_idsÚ buffered_token_type_ids_expandedrnÚ embeddingsrmr'r'r*Úforward£s,        zElectraEmbeddings.forward)NNNNr) Ú__name__Ú __module__Ú __qualname__Ú__doc__rgrrLZ LongTensorÚ FloatTensorrEÚTensorr‰Ú __classcell__r'r'r{r*r[‹s úùr[c s‚eZdZd ‡fdd„ Zejejdœdd„Zd ejeejeejeejeejee e ejee e ejdœd d „Z ‡Z S) ÚElectraSelfAttentionNcsþtƒ ¡|j|jdkr>t|dƒs>td|j›d|j›dƒ‚|j|_t|j|jƒ|_|j|j|_t   |j|j¡|_ t   |j|j¡|_ t   |j|j¡|_ t  |j¡|_|p¸t|ddƒ|_|jdksÐ|jd krò|j|_t  d |jd |j¡|_|j|_dS) NrrjzThe hidden size (z6) is not a multiple of the number of attention heads (ú)rarbÚ relative_keyÚrelative_key_queryr2r!)rfrgÚ hidden_sizeÚnum_attention_headsrƒrIrEÚattention_head_sizeÚ all_head_sizerÚLinearÚqueryÚkeyÚvaluerqZattention_probs_dropout_probrsrCrarlrhÚdistance_embeddingÚ is_decoder©rzrPrar{r'r*rgÏs* ÿÿzElectraSelfAttention.__init__)Úxr€cCs6| ¡dd…|j|jf}| |¡}| dddd¡S)Nr_rr2r!r )rwr–r—ÚviewÚpermute)rzr Z new_x_shaper'r'r*Útranspose_for_scoresés z)ElectraSelfAttention.transpose_for_scoresF©Ú hidden_statesÚattention_maskÚ head_maskÚencoder_hidden_statesÚencoder_attention_maskÚpast_key_valueÚoutput_attentionsr€cCsÒ| |¡}|dk } | r4|dk r4|d} |d} |}n | r^| | |¡¡} | | |¡¡} |}nv|dk r´| | |¡¡} | | |¡¡} tj|d| gdd} tj|d| gdd} n | | |¡¡} | | |¡¡} | |¡} |dk } |jrô| | f}t | |  dd¡¡}|j dks |j dkr| j d| j d}}| r^tj |dtj |j d  dd¡}ntj|tj |j d  dd¡}tj|tj |j d  dd¡}||}| ||jd¡}|j| jd }|j dkrät d | |¡}||}n4|j dkrt d | |¡}t d | |¡}|||}|t |j¡}|dk r:||}tjj|dd}| |¡}|dk rf||}t || ¡}| dddd ¡ ¡}| ¡dd…|jf}| |¡}|r¶||fn|f}|jrÎ||f}|S)Nrr!r2©Zdimr_éþÿÿÿr“r”rrdzbhld,lrd->bhlrzbhrd,lrd->bhlrr ) ršr£r›rœrLÚcatržÚmatmulrGrarHZtensorrxr‚r¡rtrrlÚtoreZeinsumÚmathÚsqrtr—rZ functionalZsoftmaxrsr¢Ú contiguousrwr˜)rzr¥r¦r§r¨r©rªr«Zmixed_query_layerZis_cross_attentionZ key_layerZ value_layerZ query_layerÚ use_cacheZattention_scoresZ query_lengthZ key_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZ context_layerZnew_context_layer_shapeÚoutputsr'r'r*r‰îsp   ÿ          zElectraSelfAttention.forward)N)NNNNNF) rŠr‹rŒrgrLrr£rrŽrÚboolr‰rr'r'r{r*r‘Îs$ø÷r‘cs4eZdZ‡fdd„Zejejejdœdd„Z‡ZS)ÚElectraSelfOutputcsBtƒ ¡t |j|j¡|_tj|j|jd|_t |j ¡|_ dS©Nr]) rfrgrr™r•Údenserorprqrrrsryr{r'r*rgVs zElectraSelfOutput.__init__©r¥Ú input_tensorr€cCs&| |¡}| |¡}| ||¡}|S©N©r¹rsro©rzr¥r»r'r'r*r‰\s  zElectraSelfOutput.forward©rŠr‹rŒrgrLrr‰rr'r'r{r*r·Us r·Úeagerc sveZdZd ‡fdd„ Zdd„Zd ejeejeejeejeejee e ejee e ejdœdd „Z ‡Z S) ÚElectraAttentionNcs4tƒ ¡t|j||d|_t|ƒ|_tƒ|_dS)N©ra) rfrgÚELECTRA_SELF_ATTENTION_CLASSESZ_attn_implementationrzr·ÚoutputÚsetÚ pruned_headsrŸr{r'r*rgjs ÿ zElectraAttention.__init__cCs²t|ƒdkrdSt||jj|jj|jƒ\}}t|jj|ƒ|j_t|jj|ƒ|j_t|jj |ƒ|j_ t|j j |dd|j _ |jjt|ƒ|j_|jj|jj|j_ |j  |¡|_dS)Nrr!r¬)rDrrzr–r—rÆrršr›rœrÄr¹r˜Úunion)rzÚheadsÚindexr'r'r*Ú prune_headsrs ÿzElectraAttention.prune_headsFr¤c Cs<| |||||||¡}| |d|¡} | f|dd…} | S)Nrr!)rzrÄ) rzr¥r¦r§r¨r©rªr«Z self_outputsÚattention_outputrµr'r'r*r‰„s ù zElectraAttention.forward)N)NNNNNF) rŠr‹rŒrgrÊrLrrrŽrr¶r‰rr'r'r{r*rÁis$ø÷rÁcs0eZdZ‡fdd„Zejejdœdd„Z‡ZS)ÚElectraIntermediatecsBtƒ ¡t |j|j¡|_t|jt ƒr6t |j|_ n|j|_ dSr¼) rfrgrr™r•Úintermediate_sizer¹r=Ú hidden_actÚstrr Úintermediate_act_fnryr{r'r*rgžs   zElectraIntermediate.__init__)r¥r€cCs| |¡}| |¡}|Sr¼)r¹rÐ)rzr¥r'r'r*r‰¦s  zElectraIntermediate.forwardr¿r'r'r{r*rÌs rÌcs4eZdZ‡fdd„Zejejejdœdd„Z‡ZS)Ú ElectraOutputcsBtƒ ¡t |j|j¡|_tj|j|jd|_t  |j ¡|_ dSr¸) rfrgrr™rÍr•r¹rorprqrrrsryr{r'r*rg®s zElectraOutput.__init__rºcCs&| |¡}| |¡}| ||¡}|Sr¼r½r¾r'r'r*r‰´s  zElectraOutput.forwardr¿r'r'r{r*rÑ­s rÑc steZdZ‡fdd„Zd ejeejeejeejeejeeeejee eejdœdd„Z dd „Z ‡Z S) Ú ElectraLayercsrtƒ ¡|j|_d|_t|ƒ|_|j|_|j|_|jrZ|jsLt|›dƒ‚t|dd|_ t |ƒ|_ t |ƒ|_ dS)Nr!z> should be used as a decoder model if cross attention is addedrbrÂ)rfrgÚchunk_size_feed_forwardÚ seq_len_dimrÁÚ attentionržÚadd_cross_attentionrIÚcrossattentionrÌÚ intermediaterÑrÄryr{r'r*rg½s   zElectraLayer.__init__NFr¤c Cs|dk r|dd…nd}|j|||||d} | d} |jrP| dd…} | d} n | dd…} d} |jrÞ|dk rÞt|dƒsˆtd|›dƒ‚|dk rœ|d d…nd}| | ||||||¡}|d} | |dd…} |d} | | } t|j|j|j| ƒ}|f| } |jr| | f} | S) Nr2)r«rªrr!r_r×z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r­) rÕržrƒrIr×rÚfeed_forward_chunkrÓrÔ)rzr¥r¦r§r¨r©rªr«Zself_attn_past_key_valueZself_attention_outputsrËrµZpresent_key_valueZcross_attn_present_key_valueZcross_attn_past_key_valueZcross_attention_outputsÚ layer_outputr'r'r*r‰ËsV û     ÿù ÿ  zElectraLayer.forwardcCs| |¡}| ||¡}|Sr¼)rØrÄ)rzrËZintermediate_outputrÚr'r'r*rÙ s  zElectraLayer.feed_forward_chunk)NNNNNF) rŠr‹rŒrgrLrrrŽrr¶r‰rÙrr'r'r{r*rÒ¼s$ ø÷ ArÒcs†eZdZ‡fdd„Zd ejeejeejeejeejeeeejee ee ee ee e eeje fdœ dd„Z ‡Z S) ÚElectraEncodercs:tƒ ¡ˆ|_t ‡fdd„tˆjƒDƒ¡|_d|_dS)Ncsg|] }tˆƒ‘qSr')rÒ)r(Ú_©rPr'r*Ú sz+ElectraEncoder.__init__..F) rfrgrPrZ ModuleListÚrangeÚnum_hidden_layersÚlayerÚgradient_checkpointingryr{rÝr*rgs  zElectraEncoder.__init__NFT) r¥r¦r§r¨r©Úpast_key_valuesr´r«Úoutput_hidden_statesÚ return_dictr€c  Csb| rdnd} |rdnd} |r(|jjr(dnd} |jrJ|jrJ|rJt d¡d}|rRdnd}t|jƒD]¼\}}| rv| |f} |dk r†||nd}|dk rš||nd}|jrÆ|jrÆ| |j |||||||¡}n||||||||ƒ}|d}|rô||df7}|r`| |df} |jjr`| |df} q`| r.| |f} | sPt dd „||| | | fDƒƒSt ||| | | d S) Nr'zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frr_r!r2css|]}|dk r|VqdSr¼r')r(Úvr'r'r*r+\søz)ElectraEncoder.forward..)Zlast_hidden_staterãr¥Ú attentionsÚcross_attentions) rPrÖrâZtrainingr5Z warning_onceÚ enumerateráZ_gradient_checkpointing_funcÚ__call__Útupler)rzr¥r¦r§r¨r©rãr´r«räråZall_hidden_statesZall_self_attentionsZall_cross_attentionsZnext_decoder_cacheÚiZ layer_moduleZlayer_head_maskrªZ layer_outputsr'r'r*r‰sx   ÿ   ø ù  ûþ ûzElectraEncoder.forward) NNNNNNFFT)rŠr‹rŒrgrLrrrŽrr¶rrr‰rr'r'r{r*rÛs. õôrÛcs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚElectraDiscriminatorPredictionszEPrediction module for the discriminator, made up of two dense layers.csBtƒ ¡t |j|j¡|_t|jƒ|_t |jd¡|_ ||_ dS©Nr!) rfrgrr™r•r¹r rÎÚ activationr%rPryr{r'r*rgss   z(ElectraDiscriminatorPredictions.__init__cCs(| |¡}| |¡}| |¡ d¡}|S)Nr_)r¹rïr%Úsqueeze)rzÚdiscriminator_hidden_statesr¥Úlogitsr'r'r*r‰{s  z'ElectraDiscriminatorPredictions.forward©rŠr‹rŒrrgr‰rr'r'r{r*ríps rícs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚElectraGeneratorPredictionszAPrediction module for the generator, made up of two dense layers.cs>tƒ ¡tdƒ|_tj|j|jd|_t |j |j¡|_ dS)NÚgelur]) rfrgr rïrrorjrpr™r•r¹ryr{r'r*rg†s  z$ElectraGeneratorPredictions.__init__cCs"| |¡}| |¡}| |¡}|Sr¼)r¹rïro)rzÚgenerator_hidden_statesr¥r'r'r*r‰s   z#ElectraGeneratorPredictions.forwardrór'r'r{r*rôƒs rôc@s(eZdZdZeZeZdZdZ dd„Z dS)ÚElectraPreTrainedModelz† An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. ÚelectraTcCs¤t|tjƒr:|jjjd|jjd|jdk r |jj  ¡nft|tj ƒrz|jjjd|jjd|j dk r |jj|j   ¡n&t|tj ƒr |jj  ¡|jj  d¡dS)zInitialize the weightsg)ZmeanZstdNgð?)r=rr™r.rMZnormal_rPZinitializer_ranger0Zzero_rhr\roZfill_)rzÚmoduler'r'r*Ú _init_weights¡s      z$ElectraPreTrainedModel._init_weightsN) rŠr‹rŒrr"Ú config_classrZZload_tf_weightsÚbase_model_prefixZsupports_gradient_checkpointingrúr'r'r'r*r÷•s r÷c@s^eZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejed<dS)ÚElectraForPreTrainingOutputaä Output type of [`ElectraForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss of the ELECTRA objective. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Prediction scores of the head (scores for each token before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. NÚlossròr¥rç) rŠr‹rŒrrþrrLrŽÚ__annotations__ròr¥rrçr'r'r'r*rý²s rýaA This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ElectraConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aÛ Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. encoder_hidden_states (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a]The bare Electra Model transformer outputting raw hidden-states without any specific head on top. Identical to the BERT model except that it uses an additional linear layer between the embedding layer and the encoder if the hidden size and embedding size are different. Both the generator and discriminator checkpoints may be loaded into this model.csÒeZdZ‡fdd„Zdd„Zdd„Zdd„Zee  d ¡ƒe e e e d deejeejeejeejeejeejeejeejeeejeeeeeeeeeeeje fd œd d„ƒƒZ‡ZS)Ú ElectraModelcsPtƒ |¡t|ƒ|_|j|jkr4t |j|j¡|_t |ƒ|_ ||_ |  ¡dSr¼) rfrgr[rˆrjr•rr™Úembeddings_projectrÛÚencoderrPÚ post_initryr{r'r*rg$s    zElectraModel.__init__cCs|jjSr¼©rˆrk©rzr'r'r*Úget_input_embeddings0sz!ElectraModel.get_input_embeddingscCs ||j_dSr¼r)rzrœr'r'r*Úset_input_embeddings3sz!ElectraModel.set_input_embeddingscCs*| ¡D]\}}|jj|j |¡qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)ÚitemsrrárÕrÊ)rzZheads_to_prunerárÈr'r'r*Ú _prune_heads6szElectraModel._prune_headsúbatch_size, sequence_length©Ú checkpointÚ output_typerûN)r}r¦rcr^r§r~r¨r©rãr´r«rärår€c Csô| dk r | n|jj} | dk r | n|jj} | dk r4| n|jj} |dk rV|dk rVtdƒ‚n@|dk rt| ||¡| ¡}n"|dk rŽ| ¡dd…}ntdƒ‚|\}}|dk r¬|jn|j}| dk rÌ| ddjdnd}|dkræt j ||d}|dkr:t |j dƒr(|j j dd…d|…f}| ||¡}|}nt j|t j|d}| ||¡}|jjr”|dk r”| ¡\}}}||f}|dkrˆt j ||d}| |¡}nd}| ||jj¡}|j |||||d }t |d ƒrÒ| |¡}|j|||||| | | | | d }|S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer_z5You have to specify either input_ids or inputs_embedsrr2)r‚rcr)r}r^rcr~rr) r¦r§r¨r©rãr´r«rärå)rPr«räÚuse_return_dictrIZ%warn_if_padding_and_no_attention_maskrwr‚rHrLZonesrƒrˆrcrurvrxZget_extended_attention_maskržZinvert_attention_maskZ get_head_maskràrr)rzr}r¦rcr^r§r~r¨r©rãr´r«rärår„Z batch_sizer…r‚rr†r‡Zextended_attention_maskZencoder_batch_sizeZencoder_sequence_lengthrÜZencoder_hidden_shapeZencoder_extended_attention_maskr¥r'r'r*r‰>slÿ        û  ö zElectraModel.forward) NNNNNNNNNNNNN)rŠr‹rŒrgrrr rÚELECTRA_INPUTS_DOCSTRINGÚformatrÚ_CHECKPOINT_FOR_DOCrÚ_CONFIG_FOR_DOCrrLrrrŽr¶rrr‰rr'r'r{r*rsN  ýò ñrcs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚElectraClassificationHeadz-Head for sentence-level classification tasks.cs^tƒ ¡t |j|j¡|_|jdk r,|jn|j}tdƒ|_ t  |¡|_ t |j|j ¡|_ dS)Nrõ)rfrgrr™r•r¹Úclassifier_dropoutrrr rïrqrsÚ num_labelsÚout_proj©rzrPrr{r'r*rg¡s ÿ  z"ElectraClassificationHead.__init__cKsL|dd…ddd…f}| |¡}| |¡}| |¡}| |¡}| |¡}|S)Nr)rsr¹rïr)rzÚfeaturesÚkwargsr r'r'r*r‰«s     z!ElectraClassificationHead.forwardrór'r'r{r*ržs rzŸ ELECTRA Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cs¤eZdZ‡fdd„Zee d¡ƒedee dddd e e j e e j e e j e e j e e j e e j e e j e e e e e e eee j efd œ d d „ƒƒZ‡ZS) Ú ElectraForSequenceClassificationcs:tƒ |¡|j|_||_t|ƒ|_t|ƒ|_| ¡dSr¼) rfrgrrPrrørr1rryr{r'r*rg½s    z)ElectraForSequenceClassification.__init__r z$bhadresh-savani/electra-base-emotionz'joy'g¸…ëQ¸®?©r r rûÚexpected_outputÚ expected_lossN© r}r¦rcr^r§r~Úlabelsr«rärår€c  Csr| dk r | n|jj} |j|||||||| | d } | d} | | ¡} d}|dk r.|jjdkr¤|jdkrpd|j_n4|jdkrœ|jtjks’|jtj krœd|j_nd|j_|jjdkràt ƒ}|jdkrÔ||   ¡|  ¡ƒ}n || |ƒ}nN|jjdkrt ƒ}||   d|j¡|  d¡ƒ}n|jjdkr.tƒ}|| |ƒ}| s^| f| dd…}|dk rZ|f|S|St|| | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N©r¦rcr^r§r~r«rärårr!Z regressionZsingle_label_classificationZmulti_label_classificationr_©rþròr¥rç)rPrrør1Z problem_typerrerLrxrEr rðr r¡rrr¥rç)rzr}r¦rcr^r§r~rr«rärårñÚsequence_outputròrþÚloss_fctrÄr'r'r*r‰ÇsT÷      "     üz(ElectraForSequenceClassification.forward) NNNNNNNNNN)rŠr‹rŒrgrrrrrrrrLrr¶rrr‰rr'r'r{r*rµs@ û õôrzÊ Electra model with a binary classification head on top as used during pretraining for identifying generated tokens. It is recommended to load the discriminator checkpoint into that model. csžeZdZ‡fdd„Zee d¡ƒeee dd e e j e e j e e j e e j e e j e e j e e j e e e e e e eee j efdœ dd„ƒƒZ‡ZS) ÚElectraForPreTrainingcs,tƒ |¡t|ƒ|_t|ƒ|_| ¡dSr¼)rfrgrrøríÚdiscriminator_predictionsrryr{r'r*rgs   zElectraForPreTraining.__init__r ©r rûNrc  Csþ| dk r | n|jj} |j|||||||| | d } | d} | | ¡} d}|dk r¾t ¡}|dk r¢| d| jd¡dk}|  d| jd¡|}||}||| ¡ƒ}n||  d| jd¡| ¡ƒ}| sê| f| dd…}|dk ræ|f|S|St || | j | j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the ELECTRA loss. Input should be a sequence of tokens (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates the token is an original token, - 1 indicates the token was replaced. Returns: Examples: ```python >>> from transformers import ElectraForPreTraining, AutoTokenizer >>> import torch >>> discriminator = ElectraForPreTraining.from_pretrained("google/electra-base-discriminator") >>> tokenizer = AutoTokenizer.from_pretrained("google/electra-base-discriminator") >>> sentence = "The quick brown fox jumps over the lazy dog" >>> fake_sentence = "The quick brown fox fake over the lazy dog" >>> fake_tokens = tokenizer.tokenize(fake_sentence, add_special_tokens=True) >>> fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt") >>> discriminator_outputs = discriminator(fake_inputs) >>> predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2) >>> fake_tokens ['[CLS]', 'the', 'quick', 'brown', 'fox', 'fake', 'over', 'the', 'lazy', 'dog', '[SEP]'] >>> predictions.squeeze().tolist() [0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0] ```Nr rr_r!r!) rPrrør%rrr¡rHÚfloatrýr¥rç)rzr}r¦rcr^r§r~rr«rärårñÚdiscriminator_sequence_outputròrþr#Z active_lossZ active_logitsZ active_labelsrÄr'r'r*r‰'s@0÷  üzElectraForPreTraining.forward) NNNNNNNNNN)rŠr‹rŒrgrrrr rýrrrLrr¶rrr‰rr'r'r{r*r$s4   õôr$zý Electra model with a language modeling head on top. Even though both the discriminator and generator may be loaded into this model, the generator is the only model of the two to have been trained for the masked language modeling task. cs¼eZdZdgZ‡fdd„Zdd„Zdd„Zee  d¡ƒe d e e d d d d de eje eje eje eje eje eje eje ee ee eeeeje fdœ dd„ƒƒZ‡ZS)r>úgenerator_lm_head.weightcs>tƒ |¡t|ƒ|_t|ƒ|_t |j|j ¡|_ |  ¡dSr¼) rfrgrrørôÚgenerator_predictionsrr™rjriÚgenerator_lm_headrryr{r'r*rg‹s    zElectraForMaskedLM.__init__cCs|jSr¼©r+rr'r'r*Úget_output_embeddings•sz(ElectraForMaskedLM.get_output_embeddingscCs ||_dSr¼r,)rzrkr'r'r*Úset_output_embeddings˜sz(ElectraForMaskedLM.set_output_embeddingsr zgoogle/electra-small-generatorz[MASK]z'paris'g…ëQ¸…ó?)r r rûÚmaskrrNrc  Cs¼| dk r | n|jj} |j|||||||| | d } | d} | | ¡} | | ¡} d}|dk r|t ¡}||  d|jj¡| d¡ƒ}| s¨| f| dd…}|dk r¤|f|S|St || | j | j dS)a¢ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nr rr_r!r!) rPrrør*r+rr r¡rirr¥rç)rzr}r¦rcr^r§r~rr«räråröZgenerator_sequence_outputÚprediction_scoresrþr#rÄr'r'r*r‰›s8÷   üzElectraForMaskedLM.forward) NNNNNNNNNN)rŠr‹rŒÚ_tied_weights_keysrgr-r.rrrrrrrrLrr¶rrr‰rr'r'r{r*r>sH   ú õôr>z‰ Electra model with a token classification head on top. Both the discriminator and generator may be loaded into this model. cs¤eZdZ‡fdd„Zee d¡ƒedee dddd e e j e e j e e j e e j e e j e e j e e j e e e e e e eee j efd œ d d „ƒƒZ‡ZS) ÚElectraForTokenClassificationcs^tƒ |¡|j|_t|ƒ|_|jdk r.|jn|j}t |¡|_ t  |j |j¡|_ |  ¡dSr¼)rfrgrrrørrrrrqrsr™r•r1rrr{r'r*rgäs  ÿ z&ElectraForTokenClassification.__init__r zDbhadresh-savani/electra-base-discriminator-finetuned-conll03-englishzK['B-LOC', 'B-ORG', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'B-LOC', 'I-LOC']g)\Âõ(¼?rNrc  Cs¸| dk r | n|jj} |j|||||||| | d } | d} | | ¡} | | ¡} d}|dk rxtƒ}||  d|j¡| d¡ƒ}| s¤| f| dd…}|dk r |f|S|St|| | j | j dS)zÛ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nr rr_r!r!) rPrrørsr1r r¡rrr¥rç)rzr}r¦rcr^r§r~rr«rärårñr(ròrþr#rÄr'r'r*r‰ñs8÷   üz%ElectraForTokenClassification.forward) NNNNNNNNNN)rŠr‹rŒrgrrrrrrrrLrr¶rrr‰rr'r'r{r*r2Ûs@ û õôr2zà ELECTRA Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs¸eZdZeZdZ‡fdd„Zee  d¡ƒe de e dddd d de eje eje eje eje eje eje eje eje ee ee eeeeje fd œ d d„ƒƒZ‡ZS)ÚElectraForQuestionAnsweringrøcs<tƒ |¡|j|_t|ƒ|_t |j|j¡|_|  ¡dSr¼) rfrgrrrørr™r•Ú qa_outputsrryr{r'r*rg8s   z$ElectraForQuestionAnswering.__init__r z#bhadresh-savani/electra-base-squad2é é z'a nice puppet'g…ëQ¸@)r r rûZqa_target_start_indexZqa_target_end_indexrrN) r}r¦rcr^r§r~Ústart_positionsÚ end_positionsr«rärår€c  CsN| dk r | n|jj} |j||||||| | d} | d} | | ¡}|jddd\}}| d¡ ¡}| d¡ ¡}d}|dk r|dk rt| ¡ƒdkr | d¡}t| ¡ƒdkrº| d¡}| d¡}|  d|¡}|  d|¡}t |d}|||ƒ}|||ƒ}||d}| s8||f| dd…}|dk r4|f|S|St |||| j | j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. N)r¦rcr^r§r~r«rärr!r_r¬)Z ignore_indexr2)rþÚ start_logitsÚ end_logitsr¥rç)rPrrør4r@rðr³rDrwÚclampr rr¥rç)rzr}r¦rcr^r§r~r7r8r«rärårñr"ròr9r:Z total_lossZ ignored_indexr#Z start_lossZend_lossrÄr'r'r*r‰BsV"ø           þ ýûz#ElectraForQuestionAnswering.forward) NNNNNNNNNNN)rŠr‹rŒr"rûrürgrrrrrrrrLrr¶rrr‰rr'r'r{r*r3-sL ù ôór3z¨ ELECTRA Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. cs eZdZ‡fdd„Zee d¡ƒeee e dd e e j e e j e e j e e j e e j e e j e e j e ee ee eeee j e fdœ dd„ƒƒZ‡ZS) ÚElectraForMultipleChoicecs<tƒ |¡t|ƒ|_t|ƒ|_t |jd¡|_ |  ¡dSrî) rfrgrrørÚsequence_summaryrr™r•r1rryr{r'r*rg¡s    z!ElectraForMultipleChoice.__init__z(batch_size, num_choices, sequence_lengthr Nrc  Cst| dk r | n|jj} |dk r&|jdn|jd} |dk rJ| d| d¡¡nd}|dk rh| d| d¡¡nd}|dk r†| d| d¡¡nd}|dk r¤| d| d¡¡nd}|dk rÊ| d| d¡| d¡¡nd}|j|||||||| | d } | d} | | ¡}| |¡}| d| ¡}d}|dk r0tƒ}|||ƒ}| s`|f| dd…}|dk r\|f|S|St ||| j | j dS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr!r_r­r rr!) rPrrHr¡rwrør=r1r rr¥rç)rzr}r¦rcr^r§r~rr«räråZ num_choicesrñr"Z pooled_outputròZreshaped_logitsrþr#rÄr'r'r*r‰«sLÿý÷      üz ElectraForMultipleChoice.forward) NNNNNNNNNN)rŠr‹rŒrgrrrrrrrrrLrr¶rrr‰rr'r'r{r*r<™s< ýõôr<zIELECTRA Model with a `language modeling` head on top for CLM fine-tuning.csèeZdZdgZ‡fdd„Zdd„Zdd„Zee  d¡ƒe e e d de eje eje eje eje eje eje eje eje eje eeje ee ee ee eeeeje fd œd d „ƒƒZddd„Zdd„Z‡ZS)ÚElectraForCausalLMr)csNtƒ |¡|jst d¡t|ƒ|_t|ƒ|_t   |j |j ¡|_ | ¡dS)NzOIf you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.`)rfrgržr5Úwarningrrørôr*rr™rjrir+Z init_weightsryr{r'r*rgús    zElectraForCausalLM.__init__cCs|jSr¼r,rr'r'r*r-sz(ElectraForCausalLM.get_output_embeddingscCs ||_dSr¼r,)rzZnew_embeddingsr'r'r*r. sz(ElectraForCausalLM.set_output_embeddingsr r&N)r}r¦rcr^r§r~r¨r©rrãr´r«rärår€cCs|dk r |n|jj}| dk r d} |j||||||||| | | | |d }|d}| | |¡¡}d}| dk rÀ|dd…dd…dd…f ¡}| dd…dd…f ¡} tƒ}|| d|jj¡|  d¡ƒ}|sì|f|dd…}|dk rè|f|S|St |||j |j |j |j dS)aJ encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Returns: Example: ```python >>> from transformers import AutoTokenizer, ElectraForCausalLM, ElectraConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/electra-base-generator") >>> config = ElectraConfig.from_pretrained("google/electra-base-generator") >>> config.is_decoder = True >>> model = ElectraForCausalLM.from_pretrained("google/electra-base-generator", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NF) r¦rcr^r§r~r¨r©rãr´r«rärårr_r!)rþròrãr¥rçrè)rPrrør+r*r³r r¡rirrãr¥rçrè)rzr}r¦rcr^r§r~r¨r©rrãr´r«rärårµr"r0Zlm_lossZshifted_prediction_scoresr#rÄr'r'r*r‰ sJ>óúzElectraForCausalLM.forwardcKst|j}|dkr| |¡}|dk rh|ddjd}|jd|krF|}n|jdd}|dd…|d…f}|||dœS)Nrr2r!)r}r¦rã)rHZnew_ones)rzr}rãr¦Z model_kwargsr„Z past_lengthZremove_prefix_lengthr'r'r*Úprepare_inputs_for_generationws z0ElectraForCausalLM.prepare_inputs_for_generationcs.d}|D] }|t‡fdd„|Dƒƒf7}q|S)Nr'c3s"|]}| dˆ |j¡¡VqdS)rN)Z index_selectr°r‚)r(Z past_state©Úbeam_idxr'r*r+‘sz4ElectraForCausalLM._reorder_cache..)rë)rzrãrBZreordered_pastZ layer_pastr'rAr*Ú_reorder_caches ÿz!ElectraForCausalLM._reorder_cache)NNNNNNNNNNNNNN)NN)rŠr‹rŒr1rgr-r.rrrr rrrrLrrr¶rrr‰r@rCrr'r'r{r*r>ôsN   ñ ði r>)r#)Lrr±r7Z dataclassesrÚtypingrrrrrLZtorch.utils.checkpointrZtorch.nnrr r Z activationsr r Zmodeling_outputsrrrrrrrrZmodeling_utilsrrZ pytorch_utilsrrrÚutilsrrrrrr Zconfiguration_electrar"Z get_loggerrŠr5rrrZÚModuler[r‘r·rÃrÁrÌrÑrÒrÛrírôr÷rýZELECTRA_START_DOCSTRINGrrrrr$r>r2r3r<r>r'r'r'r*Úsœ  (     RCÿ4W]<ú{ûZúaù SúJûeûTÿ