U
5Af @ s^ d dl mZmZ ddlmZ ddlmZ e rFddlmZ ddl m
Z
dZ G d d
d
eZ d S ) )ListUnion )is_torch_available )Pipeline)%MODEL_FOR_TEXT_TO_SPECTROGRAM_MAPPING)SpeechT5HifiGanzmicrosoft/speecht5_hifiganc sh e Zd ZdZddd fdd
Zdd Zdd Zeee e f d
fd d Z
dd
dZ dd Z Z
S )TextToAudioPipelinea5
Text-to-audio generation pipeline using any `AutoModelForTextToWaveform` or `AutoModelForTextToSpectrogram`. This
pipeline generates an audio file from an input text and optional other conditional inputs.
Example:
```python
>>> from transformers import pipeline
>>> pipe = pipeline(model="suno/bark-small")
>>> output = pipe("Hey it's HuggingFace on the phone!")
>>> audio = output["audio"]
>>> sampling_rate = output["sampling_rate"]
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial)
You can specify parameters passed to the model by using [`TextToAudioPipeline.__call__.forward_params`] or
[`TextToAudioPipeline.__call__.generate_kwargs`].
Example:
```python
>>> from transformers import pipeline
>>> music_generator = pipeline(task="text-to-audio", model="facebook/musicgen-small", framework="pt")
>>> # diversify the music generation by adding randomness with a high temperature and set a maximum music length
>>> generate_kwargs = {
... "do_sample": True,
... "temperature": 0.7,
... "max_new_tokens": 35,
... }
>>> outputs = music_generator("Techno music with high melodic riffs", generate_kwargs=generate_kwargs)
```
This pipeline can currently be loaded from [`pipeline`] using the following task identifiers: `"text-to-speech"` or
`"text-to-audio"`.
See the list of available models on [huggingface.co/models](https://huggingface.co/models?filter=text-to-speech).
N)vocoder
sampling_ratec s t j|| | jdkr tdd | _| jjt krX|d krRt
t | jj
n|| _|| _| jd k rt| jjj| _| jd kr| jj}| jjdd }|d k r|| dD ] }t||d }|d k r|| _qd S )Ntfz5The TextToAudioPipeline is only available in PyTorch.generation_config)Z sample_rater )super__init__Z framework
ValueErrorr model __class__r valuesr Zfrom_pretrainedDEFAULT_VOCODER_IDtodevicer config__dict__getupdateZto_dictgetattr)selfr r argskwargsr Z
gen_configZsampling_rate_namer H/tmp/pip-unpacked-wheel-zw5xktn0/transformers/pipelines/text_to_audio.pyr L s*
z TextToAudioPipeline.__init__c K sd t |tr|g}| jjjdkrJ| jjjddddddd}|| |}| j |f|dd i}|S )
NZbarkZmax_input_semantic_length FT
max_length)r$ Zadd_special_tokensZreturn_attention_maskZreturn_token_type_idspaddingZreturn_tensorspt)
isinstancestrr r Z
model_typer Zsemantic_configr r tokenizer)r textr Z
new_kwargsoutputr! r! r"
preprocessk s
z TextToAudioPipeline.preprocessc K s | j || jd}|d }|d }| j rX| j || jd}|| | jjf ||}n.t|rrtd| | jf ||d }| j d k r| |}|S )N)r forward_paramsgenerate_kwargsa\ You're using the `TextToAudioPipeline` with a forward-only model, but `generate_kwargs` is non empty.
For forward-only TTA models, please use `forward_params` instead of of
`generate_kwargs`. For reference, here are the `generate_kwargs` used here:
r )
Z_ensure_tensor_on_devicer r Z can_generater generatelenr keysr )r Z model_inputsr r- r. r+ r! r! r" _forward s"
z TextToAudioPipeline._forward) text_inputsc s t j|f|S )a
Generates speech/audio from the inputs. See the [`TextToAudioPipeline`] documentation for more information.
Args:
text_inputs (`str` or `List[str]`):
The text(s) to generate.
forward_params (`dict`, *optional*):
Parameters passed to the model generation/forward method. `forward_params` are always passed to the
underlying model.
generate_kwargs (`dict`, *optional*):
The dictionary of ad-hoc parametrization of `generate_config` to be used for the generation call. For a
complete overview of generate, check the [following
guide](https://huggingface.co/docs/transformers/en/main_classes/text_generation). `generate_kwargs` are
only passed to the underlying model if the latter is a generative model.
Return:
A `dict` or a list of `dict`: The dictionaries have two keys:
- **audio** (`np.ndarray` of shape `(nb_channels, audio_length)`) -- The generated audio waveform.
- **sampling_rate** (`int`) -- The sampling rate of the generated audio waveform.
)r __call__)r r3 r- r r! r" r4 s z TextToAudioPipeline.__call__c C s4 |r|ni |r|ni d}|d kr&i }i }|||fS )N)r- r. r! )r Zpreprocess_paramsr- r. paramsZpostprocess_paramsr! r! r" _sanitize_parameters s
z(TextToAudioPipeline._sanitize_parametersc C sL i }t |tr|d }nt |tr*|d }| |d< | j|d< |S )Nwaveformr audior )r' dicttuplecpufloatZnumpyr )r r7 Z output_dictr! r! r" postprocess s
z TextToAudioPipeline.postprocess)NNN)__name__
__module__ __qualname____doc__r r, r2 r r( r r4 r6 r=
__classcell__r! r! r r" r
s 0
r
N)
typingr r utilsr baser Zmodels.auto.modeling_autor Z!models.speecht5.modeling_speecht5r r r
r! r! r! r" s