U 5Afh@sdZddlmZddlmZmZmZddlZddlZddlm Z ddl m Z m Z ddl mZmZmZmZd d lmZeeZd Zd ZeGd dde ZGddde jZGddde jZGddde jZGddde jZdZdZ edeGddde Z!dS)zPyTorch UnivNetModel model.) dataclass)OptionalTupleUnionN)nn) ModelOutputPreTrainedModel)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) UnivNetConfigrzdg845/univnet-devc@s.eZdZUdZdZejed<dZejed<dS)UnivNetModelOutputa Output class for the [`UnivNetModel`], which includes the generated audio waveforms and the original unpadded lengths of those waveforms (so that the padding can be removed by [`UnivNetModel.batch_decode`]). Args: waveforms (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Batched 1D (mono-channel) output audio waveforms. waveform_lengths (`torch.FloatTensor` of shape `(batch_size,)`): The batched length in samples of each unpadded waveform in `waveforms`. N waveformswaveform_lengths) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrP/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/univnet/modeling_univnet.pyr$s  rcsFeZdZdZedfdd ZejdddZdd Z d d Z Z S) #UnivNetKernelPredictorResidualBlockz Implementation of the residual block for the kernel predictor network inside each location variable convolution block (LVCBlock). Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. configcst|j|_|j|_|j|_|j|_|jdd}t |j|_ t j |j|j|j|dd|_ t j |j|j|j|dd|_dS)NrTpaddingbias)super__init__model_in_channelsZchannelskernel_predictor_conv_size kernel_sizeZkernel_predictor_dropoutZ dropout_probleaky_relu_sloperZDropoutdropoutConv1dconv1conv2)selfrr! __class__rrr$?s z,UnivNetKernelPredictorResidualBlock.__init__) hidden_statescCsJ|}||}||}tj||j}||}tj||j}||SN)r)r+r functional leaky_relur(r,)r-r0residualrrrforwardOs   z+UnivNetKernelPredictorResidualBlock.forwardcCs tj|jtj|jdSr1)rutils weight_normr+r,r-rrrapply_weight_normYsz5UnivNetKernelPredictorResidualBlock.apply_weight_normcCs tj|jtj|jdSr1)rr6remove_weight_normr+r,r8rrrr:]sz6UnivNetKernelPredictorResidualBlock.remove_weight_norm) rrrrrr$rrr5r9r: __classcell__rrr.rr5s   rcsLeZdZdZdeeedfdd Zejddd Z d d Z d d Z Z S)UnivNetKernelPredictora  Implementation of the kernel predictor network which supplies the kernel and bias for the location variable convolutional layers (LVCs) in each UnivNet LVCBlock. Based on the KernelPredictor implementation in [maum-ai/univnet](https://github.com/maum-ai/univnet/blob/9bb2b54838bb6d7ce767131cc7b8b61198bc7558/model/lvcnet.py#L7). Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. conv_kernel_size (`int`, *optional*, defaults to 3): The kernel size for the location variable convolutional layer kernels (convolutional weight tensor). conv_layers (`int`, *optional*, defaults to 4): The number of location variable convolutional layers to output kernels and biases for. r)rconv_kernel_size conv_layerscstj|_dj|_||_||_|j|j|j|j|_|j|j|_j |_ j |_ j |_j|_j|_|jdd}tj|j |j dddd|_tfddt|jD|_tj|j |j|j|dd|_tj|j |j|j|dd|_dS)NrrTr csg|] }tqSr)r).0_rrr sz3UnivNetKernelPredictor.__init__..)r#r$model_hidden_channelsconv_in_channelsconv_out_channelsr>r?Zkernel_channelsZ bias_channelsZ num_mel_binsZresnet_in_channelsZ kernel_predictor_hidden_channelsZresnet_hidden_channelsr&Zresnet_kernel_sizeZkernel_predictor_num_blocks num_blocksr(rr* input_conv ModuleListrange resblocks kernel_conv bias_conv)r-rr>r?r!r.rrr$ss<   zUnivNetKernelPredictor.__init__) spectrogramc Cs|j\}}}||}tj||j}|jD] }||}q,||}||}| ||j |j |j |j |} | ||j |j |} | | fS)a Maps a conditioning log-mel spectrogram to a tensor of convolutional kernels and biases, for use in location variable convolutional layers. Note that the input spectrogram should have shape (batch_size, input_channels, seq_length). Args: spectrogram (`torch.FloatTensor` of shape `(batch_size, input_channels, seq_length)`): Tensor containing the log-mel spectrograms. Returns: Tuple[`torch.FloatTensor, `torch.FloatTensor`]: tuple of tensors where the first element is the tensor of location variable convolution kernels of shape `(batch_size, self.conv_layers, self.conv_in_channels, self.conv_out_channels, self.conv_kernel_size, seq_length)` and the second element is the tensor of location variable convolution biases of shape `(batch_size, self.conv_layers. self.conv_out_channels, seq_length)`. )shaperHrr2r3r(rKrLrMviewr?rErFr> contiguous) r-rNZ batch_sizerBZ seq_lengthr0resblockZkernel_hidden_statesZbias_hidden_stateskernelsbiasesrrrr5s,      zUnivNetKernelPredictor.forwardcCsBtj|j|jD] }|qtj|jtj|jdSr1)rr6r7rHrKr9rLrMr-Zlayerrrrr9s   z(UnivNetKernelPredictor.apply_weight_normcCsBtj|j|jD] }|qtj|jtj|jdSr1)rr6r:rHrKrLrMrUrrrr:s   z)UnivNetKernelPredictor.remove_weight_norm)rr= rrrrrintr$rrr5r9r:r;rrr.rr<bs&.r<csbeZdZdZeeedfdd ZdddZdej ej ej eed d d Z d d Z ddZ Z S)UnivNetLvcResidualBlocka Implementation of the location variable convolution (LVC) residual block for the UnivNet residual network. Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. kernel_size (`int`): The kernel size for the dilated 1D convolutional layer. dilation (`int`): The dilation for the dilated 1D convolutional layer. )rr'dilationcs\t|j|_||_||_|j|_|j|jdd}tj|j|j|j||jd|_ dS)Nrr)r!rY) r#r$rDhidden_channelsr'rYr(rr*conv)r-rr'rYr!r.rrr$s z UnivNetLvcResidualBlock.__init__cCs|}tj||j}||}tj||j}|j||||d}t|ddd|jddft |dd|jdddf}||}|SN)hop_size) rr2r3r(r[location_variable_convolutionrZsigmoidrZtanh)r-r0kernelr"r^r4rrrr5s $zUnivNetLvcResidualBlock.forwardr)r0rar"rYr^cCsB|j\}}}|j\}}} } } || |krBtd| |d|d|t| dd} tj|| | fdd}|d|d| |}||krtj|d|fdd}|d||}|d d d d d d d d d |f}|dd }|d | d}t d ||} | j tj d } | d  d j tj d }| |} | || d } | S)u Performs location-variable convolution operation on the input sequence (hidden_states) using the local convolution kernel. This was introduced in [LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation](https://arxiv.org/abs/2102.10815) by Zhen Zheng, Jianzong Wang, Ning Cheng, and Jing Xiao. Time: 414 μs ± 309 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each), test on NVIDIA V100. Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, in_channels, in_length)`): The input sequence of shape (batch, in_channels, in_length). kernel (`torch.FloatTensor` of shape `(batch_size, in_channels, out_channels, kernel_size, kernel_length)`): The local convolution kernel of shape (batch, in_channels, out_channels, kernel_size, kernel_length). bias (`torch.FloatTensor` of shape `(batch_size, out_channels, kernel_length)`): The bias for the local convolution of shape (batch, out_channels, kernel_length). dilation (`int`, *optional*, defaults to 1): The dilation of convolution. hop_size (`int`, *optional*, defaults to 256): The hop_size of the conditioning sequence. Returns: `torch.FloatTensor`: the output sequence after performing local convolution with shape (batch_size, out_channels, in_length). z#Dim 2 of `hidden_states` should be z ) but got zX. Please check `hidden_states` or `kernel` and `hop_size` to make sure they are correct.rrZconstantrrNr=zbildsk,biokl->bolsd)Z memory_format)rO ValueErrorrWrr2padZunfold transposerZeinsumtoZchannels_last_3d unsqueezerQrP)r-r0rar"rYr^batchrBZ in_lengthZ out_channelsr'Z kernel_lengthr!Zoutput_hidden_statesrrrr_ s*  & z5UnivNetLvcResidualBlock.location_variable_convolutioncCstj|jdSr1)rr6r7r[r8rrrr9Jsz)UnivNetLvcResidualBlock.apply_weight_normcCstj|jdSr1)rr6r:r[r8rrrr:Msz*UnivNetLvcResidualBlock.remove_weight_norm)r\)rr\)rrrrrrWr$r5rrr_r9r:r;rrr.rrXs"  ArXcsPeZdZdZd eeedfdd ZejejdddZ d d Z d d Z Z S)UnivNetLvcBlocka# Implementation of the location variable convolution (LVC) residual block of the UnivNet residual block. Includes a `UnivNetKernelPredictor` inside to predict the kernels and biases of the LVC layers. Based on LVCBlock in [maum-ai/univnet](https://github.com/maum-ai/univnet/blob/9bb2b54838bb6d7ce767131cc7b8b61198bc7558/model/lvcnet.py#L98) Parameters: config (`UnivNetConfig`): Config for the `UnivNetModel` model. layer_id (`int`): An integer corresponding to the index of the current LVC resnet block layer. This should be between 0 and `len(config.resblock_stride_sizes) - 1)` inclusive. lvc_hop_size (`int`, *optional*, defaults to 256): The hop size for the location variable convolutional layers. r\)rlayer_id lvc_hop_sizecstj_j|_j|_j|_ |_ j _ t j _ tjjjdjjjdjdjdd_tjj _tfddtj D_dS)Nr)strider!Zoutput_paddingcs g|]}tjj|qSr)rXr' dilationsrAirr-rrrC~sz,UnivNetLvcBlock.__init__..)r#r$rDrZresblock_kernel_sizesr'resblock_stride_sizesrlZresblock_dilation_sizesrmcond_hop_lengthr(lenrGrConvTranspose1d convt_prer<kernel_predictorrIrJrK)r-rrjrkr.rprr$cs(      zUnivNetLvcBlock.__init__)r0rNc Cstj||j}||}||\}}t|jD]^\}}|dd|ddddddddf}|dd|ddddf}|||||jd}q2|Sr]) rr2r3r(rvrw enumeraterKrs) r-r0rNrSrTrorRrar"rrrr5s (zUnivNetLvcBlock.forwardcCs0tj|j|j|jD] }|qdSr1)rr6r7rvrwr9rKrUrrrr9s  z!UnivNetLvcBlock.apply_weight_normcCs0tj|j|j|jD] }|qdSr1)rr6r:rvrwrKrUrrrr:s  z"UnivNetLvcBlock.remove_weight_norm)r\rVrrr.rriQsriaL This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`UnivNetConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aR Converts a noise waveform and a conditioning spectrogram to a speech waveform. Passing a batch of log-mel spectrograms returns a batch of speech waveforms. Passing a single, un-batched log-mel spectrogram returns a single, un-batched speech waveform. Args: input_features (`torch.FloatTensor`): Tensor containing the log-mel spectrograms. Can be batched and of shape `(batch_size, sequence_length, config.num_mel_channels)`, or un-batched and of shape `(sequence_length, config.num_mel_channels)`. noise_sequence (`torch.FloatTensor`, *optional*): Tensor containing a noise sequence of standard Gaussian noise. Can be batched and of shape `(batch_size, sequence_length, config.model_in_channels)`, or un-batched and of shape (sequence_length, config.model_in_channels)`. If not supplied, will be randomly generated. padding_mask (`torch.BoolTensor`, *optional*): Mask indicating which parts of each sequence are padded. Mask values are selected in `[0, 1]`: - 1 for tokens that are **not masked** - 0 for tokens that are **masked** The mask can be batched and of shape `(batch_size, sequence_length)` or un-batched and of shape `(sequence_length,)`. generator (`torch.Generator`, *optional*): A [torch generator](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. return_dict: Whether to return a [`~utils.ModelOutput`] subclass instead of a plain tuple. zUnivNet GAN vocoder.c seZdZeZdZedfdd Zeee e e dde j ee j ee j ee jeeeee j e fddd Zd d Zd d ZddZZS) UnivNetModelinput_featuresrcsttj|_j|_tjjj ddddd|_ tj }d}gj D]}||} |qTt fddt|D|_tjj ddddd|_|dS) NrrZreflect)r'rlr! padding_modecsg|]}t||dqS))rjrk)rirnrZ hop_lengthsrrrCs z)UnivNetModel.__init__..)r!r|)r#r$rtrqZ num_kernelsr(rr*r%rDconv_prerrappendrIrJrK conv_postZ post_init)r-rZ num_layersZ hop_lengthrlr.r}rr$s0       zUnivNetModel.__init__) output_type config_classN)rznoise_sequence padding_mask generator return_dictreturncCs|dk r |n|jj}|dk}|s.|d}|j\}}} |dk r^|dk} | s|d}n$|||jjf} tj| ||j|j d}|jd} |dkr| dkr| |dd}n| dkr|dkr| | dd}| |krt d| d|d|dk r4|dkr |d}|jd} | |kr4t d | d|d| d d}| d d}| |}|jD]}|||}q\tj||j}||}t|}|d}d}|dk rtj|dd }|s||f}|St||d S) a Returns: Example: ```python >>> from transformers import UnivNetFeatureExtractor, UnivNetModel >>> from datasets import load_dataset, Audio >>> model = UnivNetModel.from_pretrained("dg845/univnet-dev") >>> feature_extractor = UnivNetFeatureExtractor.from_pretrained("dg845/univnet-dev") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> # Resample the audio to the feature extractor's sampling rate. >>> ds = ds.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate)) >>> inputs = feature_extractor( ... ds[0]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="pt" ... ) >>> audio = model(**inputs).waveforms >>> list(audio.shape) [1, 140288] ``` Nrr)rdtypedevicerz&The batch size of `noise_sequence` is z+ and the batch size of `input_features` is z', but the two are expected to be equal.z$The batch size of `padding_mask` is r)dim)rr)rZuse_return_dictrrgrOr%rZrandnrrrepeatrcrer~rKrr2r3r(rr`Zsqueezesumr)r-rzrrrrZspectrogram_batchedZspectrogram_batch_sizeZspectrogram_lengthrBZnoise_sequence_batchedZnoise_sequence_shapeZnoise_sequence_batch_sizeZpadding_mask_batch_sizer0rRZwaveformroutputsrrrr5sh!                  zUnivNetModel.forwardcCsFt|tjtjtjfrB|jjjd|jj d|j dk rB|j j dS)zInitialize the weights.g)ZmeanZstdN) isinstancerZLinearr*ruZweightdataZnormal_rZinitializer_ranger"Zzero_)r-modulerrr _init_weightsfs zUnivNetModel._init_weightscCs4tj|j|jD] }|qtj|jdSr1)rr6r7r~rKr9rrUrrrr9ms  zUnivNetModel.apply_weight_normcCs4tj|j|jD] }|qtj|jdSr1)rr6r:r~rKrrUrrrr:ss  zUnivNetModel.remove_weight_norm)NNNN)rrrrrZmain_input_namer$r UNIVNET_INPUTS_DOCSTRINGr r_CONFIG_FOR_DOCrrr Generatorboolrrr5rr9r:r;rrr.rrys(' jry)"rZ dataclassesrtypingrrrrZtorch.utils.checkpointrZmodeling_utilsrr r6r r r r Zconfiguration_univnetrZ get_loggerrloggerrZ_CHECKPOINT_FOR_DOCrModulerr<rXriZUNIVNET_START_DOCSTRINGrryrrrrs0    -t{L