U 5Af @sJdZddlZddlmZmZmZmZddlZddlm m Z ddl Zddlm Z ddl mZddlmZddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&e!rddl'm(Z(e#)e*Z+dZ,ej-e.e.ej/ej0e1ej-e.dddZ2Gddde j3Z4e5e4Gddde j3Z6Gddde6Z7Gddde6Z8ddZ9d8dd Z:Gd!d"d"e j3Z;ej-e.ej-d#d$d%ZGd*d+d+e=Z?e=e>e?d,Z@Gd-d.d.e j3ZAd/ZBed0eBGd1d2d2eZCd3ZDed0eBGd4d5d5eCZEGd6d7d7eCZFdS)9zPyTorch OLMo model.N)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCache StaticCache)AttentionMaskConverter)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)ALL_LAYERNORM_LAYERS)add_start_docstrings%add_start_docstrings_to_model_forwardis_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10loggingreplace_return_docstrings) OlmoConfig)_flash_attention_forwardr)attention_masksequence_length target_lengthdtypedevice min_dtypecache_position batch_sizec Cs.|dk r|dkr|}ntj||f|||d}|dkrHtj|dd}|tj||d|ddk9}|ddddddf|ddd}|dk r*|}|jd} |ddddddd| f|ddddddf} | dk} |ddddddd| f | ||ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. device (`torch.device`): The device to plcae the 4D attention mask on. min_dtype (`float`): The minimum value representable with the dtype `dtype`. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuerrr)Zdiagonalrr) dimtorchfullZtriuarangereshapeexpandcloneshapeZ masked_fill) rrrrrr r!r" causal_maskZ mask_lengthZ padding_maskr/J/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/olmo/modeling_olmo.py5_prepare_4d_causal_attention_mask_with_cache_position=s" $  8  r1cs<eZdZdZeddfdd ZejejdddZZ S) OlmoLayerNormz/LayerNorm but with no learnable weight or bias.N) hidden_sizereturncst|f|_dSN)super__init__normalized_shape)selfr3 __class__r/r0r7us zOlmoLayerNorm.__init__) hidden_statesr4cCs,|j}tj|jtjd|jdddd|S)Nrgh㈵>)Zeps)rFZ layer_normtor'float32r8)r9r<Z orig_dtyper/r/r0forwardys zOlmoLayerNorm.forward) __name__ __module__ __qualname____doc__intr7r'TensorrA __classcell__r/r/r:r0r2rsr2cs.eZdZd fdd ZeddZZS) OlmoRotaryEmbedding'N?cslt||_||_||_||_d|jtjd|jdtjd ||j}|j d|dd||_ dS)NrLrr=inv_freqF persistent) r6r7scaling_factorr&max_position_embeddingsbaser'r)int64floatr?register_bufferZmax_seq_len_cached)r9r&rRrSrrQrNr:r/r0r7s 0zOlmoRotaryEmbedding.__init__c Cs|jddddf|jddd}|dddddf}|jj}t|tr`|dkr`|nd}tj |dd@|| dd}tj ||fdd }| }| } W5QRX|j|jd | j|jd fS) Nrr%rZmpscpuF) device_typeZenabledrMr&r=)rNrUr+r-rtype isinstancestrr'Zautocast transposecatcossinr?r) r9x position_idsZinv_freq_expandedZposition_ids_expandedrXZfreqsZembr_r`r/r/r0rAs(zOlmoRotaryEmbedding.forward)rJrKNrL)rBrCrDr7r'Zno_gradrArHr/r/r:r0rIs rIcs eZdZdZfddZZS) OlmoLinearScalingRotaryEmbeddingzZOlmoRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendevcs(||j}t||\}}||fSr5)rUrQr6rA)r9rarbr_r`r:r/r0rAsz(OlmoLinearScalingRotaryEmbedding.forwardrBrCrDrErArHr/r/r:r0rcsrccs eZdZdZfddZZS)$OlmoDynamicNTKScalingRotaryEmbeddingzlOlmoRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozillacst|d}||jkr|j|j||j|jd|j|jd}d|tjd|jdtjd |j |j}|j d|ddt ||\}}||fS) NrrMrLrr=rNFrO)r'maxrRrSrQr&r)rTrUr?rrVr6rA)r9rarbZseq_lenrSrNr_r`r:r/r0rAs *z,OlmoDynamicNTKScalingRotaryEmbedding.forwardrdr/r/r:r0resrecCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr%rMrY)r-r'r^)rax1Zx2r/r/r0 rotate_halfsrhcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerh)qkr_r`rbZ unsqueeze_dimZq_embedZk_embedr/r/r0apply_rotary_pos_embs   rlcs$eZdZfddZddZZS)OlmoMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFbias) r6r7configr3Zintermediate_sizerLinear gate_projup_proj down_projr Z hidden_actact_fnr9rqr:r/r0r7s zOlmoMLP.__init__cCs ||||||Sr5)rurvrsrt)r9rar/r/r0rAszOlmoMLP.forward)rBrCrDr7rArHr/r/r:r0rms rm)r<n_repr4cCs^|j\}}}}|dkr|S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r-r+r*)r<rxbatchnum_key_value_headsslenhead_dimr/r/r0 repeat_kvs 0r}cseZdZdZd eeedfdd ZddZd e j ee j ee j ee e e ee j ee j ee j eee j fd d d ZZS) OlmoAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNrq layer_idxcs0t||_||_|dkr4td|jjd|j|_|j |_ |j |_ |j |j |_ |j |_ |j |j |_|j|_|j|_d|_|j |j |j krtd|j d|j dtj|j |j |j |jd|_tj|j |j |j |jd|_tj|j |j |j |jd|_tj|j |j |jd|_|dS)NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).ro)r6r7rqrlogger warning_oncer;rBattention_dropoutr3Znum_attention_heads num_headsr|rznum_key_value_groupsrR rope_theta is_causal ValueErrorrrrZattention_biasq_projk_projv_projo_proj _init_roper9rqrr:r/r0r7s2 zOlmoAttention.__init__cCs|jjdkr$t|j|j|jd|_nj|jjd}|jjd}|dkr^t|j|j||jd|_n0|dkrt|j|j||jd|_nt d|dS)N)rRrSrZZfactorZlinear)rRrQrSZdynamiczUnknown RoPE scaling type ) rqZ rope_scalingrIr|rRr rotary_embrcrer)r9Z scaling_typerQr/r/r0r)s.      zOlmoAttention._init_ropeFr<rrbpast_key_valueoutput_attentions use_cacher!r4cKsL|\} } } ||} ||} ||}|jjdk r| j|jj |jjd| j|jj |jjd|j|jj |jjd| | | |j|j  dd} | | | |j |j  dd} || | |j |j  dd}| ||\}}t | | ||\} } |dk r"|||d}|| ||j|\} }t| |j} t||j}t| | ddt|j }|dk r|ddddddd| jdf}||}tjj|dtjd| j}tjj||j|jd }t||}|| |j| |j fkrt d | |j| |j fd || dd!}|"| | |j#}|$|}|sBd}|||fS) NminrfrrMr`r_r!rr%)r&r)ptrainingz `attn_output` should be of size z , but is )%sizerrrrqclip_qkvclamp_viewrr|r]rzrrlupdaterr}rr'matmulmathsqrtr-r functionalZsoftmaxr@r?rdropoutrrr contiguousr*r3r)r9r<rrbrrrr!kwargsbszq_len_ query_states key_states value_statesr_r` cache_kwargs attn_weightsr. attn_outputr/r/r0rADsF           &  zOlmoAttention.forward)N)NNNFFN)rBrCrDrErrrFr7rr'rG LongTensorr boolrrArHr/r/r:r0r~s&!r~csreZdZdZfddZd ejeejeejee e e eeje ejeejee ejfdddZ Z S) OlmoFlashAttention2aD OLMo flash attention module. This module inherits from `OlmoAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cstj||t |_dSr5)r6r7r_flash_attn_uses_top_left_mask)r9argsrr:r/r0r7szOlmoFlashAttention2.__init__NFrc Ks$d}|\} } } ||} ||} ||}|jjdk r| j|jj |jjd| j|jj |jjd|j|jj |jjd| | | |j|j  dd} | | | |j |j  dd} || | |j |j  dd}| ||\}}t | | ||\} } |dk r&|||d}|| ||j|\} }| dd} | dd} | dd}|jrX|jnd}| j}|tjkrtrt}n"t|jdr|jj}n |jjj}td|d | |} | |} ||}t| | ||| |||j|jd }| | | |j!"}|#|}|sd}|||fS) NFrrrMr_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)rbrZuse_top_left_maskr)$rrrrrqrrrrr|r]rzrrlrrrrrr'r@Zis_autocast_enabledZget_autocast_gpu_dtypehasattrrweightrrr?rrrr*r3rr)r9r<rrbrrrr!rrrrrrrr_r`rZ dropout_rateZ input_dtypeZ target_dtyperrr/r/r0rAsd                     zOlmoFlashAttention2.forward)NNNFFN)rBrCrDrEr7r'rGrrr rrrArHr/r/r:r0rs$ rcsjeZdZdZdejeejeejeee e eeje ejeejee ejfdfdd Z Z S)OlmoSdpaAttentionz OLMo attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `OlmoAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. NFrc s>|r(tdtj|||||||dS|\}} } ||} ||} ||} |jj dk r| j |jj |jj d| j |jj |jj d| j |jj |jj d| || |j |j dd} | || |j|j dd} | || |j|j dd} || |\}}t| | ||\} } |dk rJ|||d}|| | |j|\} } t| |j} t| |j} |}|dk r|ddddddd| jdf}| jjdkr|dk r| } | } | } |dkr| dkrd nd }tjjj| | | ||jr|jnd |d }|dd}| || |j }|!|}|d|fS) NaOlmoModel is using OlmoSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r<rrbrrrr!rrrMrrcudaTFr)Z attn_maskZ dropout_pr)"rrr6rArrrrrqrrrrr|r]rzrrlrrr}rr-rrZrr'rrZscaled_dot_product_attentionrrr3r)r9r<rrbrrrr!rrrrrrr_r`rr.rrr:r/r0rAsf           &  zOlmoSdpaAttention.forward)NNNFFN) rBrCrDrEr'rGrrr rrrArHr/r/r:r0rs" r)eagerflash_attention_2sdpacs|eZdZeedfdd Zd ejeejeej ee ee ee eej e ej ee ej ej ffdddZZS) OlmoDecoderLayerrcsLt|j|_t|j||d|_t||_t|j|_ t|j|_ dS)Nr) r6r7r3OLMO_ATTENTION_CLASSES_attn_implementation self_attnrmmlpr2input_layernormpost_attention_layernormrr:r/r0r7Ys    zOlmoDecoderLayer.__init__NFrc Ks|} ||}|jf|||||||d|\}} } | |}|} ||}||}| |}|f} |rp| | f7} |r~| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model r)rrrr) r9r<rrbrrrr!rZresidualZself_attn_weightsZpresent_key_valueoutputsr/r/r0rAes0      zOlmoDecoderLayer.forward)NNNFFN)rBrCrDrrFr7r'rGrrr rr FloatTensorrArHr/r/r:r0rXs"raI This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`OlmoConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zRThe bare Olmo Model outputting raw hidden-states without any specific head on top.c@s@eZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ ddZdS)OlmoPreTrainedModelmodelTrpast_key_valuescCs||jj}t|tjr>|jjjd|d|jdk rx|jj n:t|tj rx|jjjd|d|j dk rx|jj|j  dS)Nr)Zmeanstd) rqZinitializer_ranger[rrrrdataZnormal_rpZzero_ Embedding padding_idx)r9modulerr/r/r0 _init_weightss    z!OlmoPreTrainedModel._init_weightsN)rBrCrDr config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attn_2Z_supports_sdpaZ_supports_cache_classZ_supports_quantized_cacheZ_supports_static_cacherr/r/r/r0rsra Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*): Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Two formats are allowed: - a [`~cache_utils.Cache`] instance; - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy cache format. The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned. If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. cseZdZdZedfdd ZddZddZee de j e e j e e j e eeee jfe e je ee ee ee ee e j eeefd d d Ze j e j e j eed ddZZS) OlmoModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`OlmoDecoderLayer`] Args: config: OlmoConfig rqcsptj|_j|_tjj|j|_t fddt j D|_ t j|_d|_|dS)Ncsg|]}t|qSr/)r).0rrr/r0 0sz&OlmoModel.__init__..F)r6r7Z pad_token_idr vocab_sizerrr3 embed_tokensZ ModuleListrangeZnum_hidden_layerslayersr2normgradient_checkpointing post_initrwr:rr0r7)s  zOlmoModel.__init__cCs|jSr5rr9r/r/r0get_input_embeddings8szOlmoModel.get_input_embeddingscCs ||_dSr5rr9valuer/r/r0set_input_embeddings;szOlmoModel.set_input_embeddingsN) input_idsrrbr inputs_embedsrroutput_hidden_states return_dictr!r4c  CsF|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}| dk rH| n|jj} |dk|dk Arhtd|jr|jr|rt dd}|dkr| |}d} |rt |t s|jsd} t |}t d| dkr|dk r|nd} tj| | |jd|jd} |dkr| d}|||| ||} |}|r6d nd}|rDd nd}d}|jD]}|rf||f7}|jr|jr||j|| ||||| }n||| ||||| d }|d}|r||rd nd}|rR||df7}qR||}|r||f7}|r|nd}| r|}| s6td d ||||fDSt||||dS)NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onezX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FTzWe detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)rrr$r/)rrbrrrr!rMcss|]}|dk r|VqdSr5r/)rvr/r/r0 sz$OlmoModel.forward..)Zlast_hidden_staterr< attentions)rqrrruse_return_dictrrrrrrr[r r Zfrom_legacy_cacheget_seq_lengthr'r)r-rri_update_causal_maskrZ_gradient_checkpointing_func__call__rZto_legacy_cachetupler)r9rrrbrrrrrrr!Zreturn_legacy_cachepast_seen_tokensr.r<Zall_hidden_statesZall_self_attnsZnext_decoder_cacheZ decoder_layerZ layer_outputsZ next_cacher/r/r0rA>s            zOlmoModel.forward)r input_tensorr!rrc Cs|jjdkr$|dk r d|kr |SdS|dk r4|nd}t|t}|jjdkrn|sn|sntj||||jdrndS|j|j }} t |j } |j d} |r|} n"t|t jr|j dn || d} t|| | || | ||j dd} |jjdkr|dk r|j jd kr|st| | } | S) Nrrrr)rZpast_key_values_lengthZ is_trainingrr%rrrrr r!r"r)rqrrr[r r Z_ignore_causal_mask_sdparrrr'finforr-get_max_lengthrGr1rZZ_unmask_unattended)r9rrr!rrrZusing_static_cacherrr rrr.r/r/r0rsV          zOlmoModel._update_causal_mask) NNNNNNNNNN)rBrCrDrErr7rrrOLMO_INPUTS_DOCSTRINGr'rrrGrr rrrrrrArrHr/r/r:r0rsD yrcseZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e e e eeddejeejeejeeejeejeejeeeeeeeeeejeeefd ddZdddZZS)OlmoForCausalLMzlm_head.weightcs@t|t||_|j|_tj|j|jdd|_| dSrn) r6r7rrrrrrr3lm_headrrwr:r/r0r7s   zOlmoForCausalLM.__init__cCs|jjSr5rrrr/r/r0r sz$OlmoForCausalLM.get_input_embeddingscCs ||j_dSr5rrr/r/r0rsz$OlmoForCausalLM.set_input_embeddingscCs|jSr5rrr/r/r0get_output_embeddingssz%OlmoForCausalLM.get_output_embeddingscCs ||_dSr5r)r9Znew_embeddingsr/r/r0set_output_embeddingssz%OlmoForCausalLM.set_output_embeddingscCs ||_dSr5r)r9decoderr/r/r0 set_decoderszOlmoForCausalLM.set_decodercCs|jSr5rrr/r/r0 get_decoderszOlmoForCausalLM.get_decoder) output_typerN) rrrbrrlabelsrrrrr!r4c  Cs,|dk r |n|jj}| dk r | n|jj} | dk r4| n|jj} |j|||||||| | | d } | d} || }|}d}|dk r|dddddf}|dddf}t}| d|jj }| d}| |j }|||}| s|f| dd}|dk r|f|S|St ||| j| j| jdS)u Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Returns: Example: ```python >>> from transformers import AutoTokenizer, OlmoForCausalLM >>> model = OlmoForCausalLM.from_pretrained("allenai/OLMo-1B-hf") >>> tokenizer = AutoTokenizer.from_pretrained("allenai/OLMo-1B-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] 'Hey, are you conscious? Can you talk to me?\nI’m not sure if you’re conscious of this, but I’m' ``` N) rrrbrrrrrrr!r.r%r)losslogitsrr<r)rqrrrrrrUrrrrr?rrrr<r)r9rrrbrrrrrrrr!rr<rrZ shift_logitsZ shift_labelsZloss_fctoutputr/r/r0rAsL+     zOlmoForCausalLM.forwardTc Ksh|dk rR|dk r.|dd|jd df}n$|jd|jdkrR|dd|f}|dk r|dkr|dd}||dkd|r|dd|jd df}|jtjd}|dk r|ddkrd|i} nd|i} t|trN|j dkrN|dk r |j\} } |j } n|j\} } |j } |j j j } t| j}t|| || | ||| d}| |||||d | S) Nrrr%)Z memory_formatrrrMr)rbr!rrr)r-longZcumsumZ masked_fill_r,r'Zcontiguous_formatr[r ndimrrrrrrr1rr)r9rrrrr!rbrrZ model_inputsr"rrrr r/r/r0prepare_inputs_for_generation{sR        z-OlmoForCausalLM.prepare_inputs_for_generation) NNNNNNNNNNN)NNNNNT)rBrCrDZ_tied_weights_keysr7rrrrrrrrrr_CONFIG_FOR_DOCr'rrrGrrrrrrArrHr/r/r:r0rsT    \r)Nr)GrErtypingrrrrr'Ztorch.nn.functionalrrr>Ztorch.utils.checkpointZtorch.nnrZ activationsr Z cache_utilsr r r Zmodeling_attn_mask_utilsr Zmodeling_outputsrrZmodeling_utilsrZ pytorch_utilsrutilsrrrrrrZconfiguration_olmorZmodeling_flash_attention_utilsrZ get_loggerrBrrrGrFrrrUr1Moduler2appendrIrcrerhrlrmr}r~rrrrZOLMO_START_DOCSTRINGrrrrr/r/r/r0sx           5    n^MJa