U
5Af @ s d dl mZ d dlmZ d dlZd dlZddlmZm Z ddl
m Z ddl m
Z
mZ e eZeed d d
G d d d e
ZdS )
)UserDict)UnionN )add_end_docstringslogging ) ffmpeg_read)Pipelinebuild_pipeline_init_argsT)Zhas_feature_extractorZ
has_tokenizerc s` e Zd ZdZ fddZeejee f d fdd Z
dd Z dd d Z d
d Z
dd Z ZS )#ZeroShotAudioClassificationPipelinea
Zero shot audio classification pipeline using `ClapModel`. This pipeline predicts the class of an audio when you
provide an audio and a set of `candidate_labels`.
The default `hypothesis_template` is : `"This is a sound of {}."`. Make sure you update it for your usage.
Example:
```python
>>> from transformers import pipeline
>>> from datasets import load_dataset
>>> dataset = load_dataset("ashraq/esc50")
>>> audio = next(iter(dataset["train"]["audio"]))["array"]
>>> classifier = pipeline(task="zero-shot-audio-classification", model="laion/clap-htsat-unfused")
>>> classifier(audio, candidate_labels=["Sound of a dog", "Sound of vaccum cleaner"])
[{'score': 0.9996, 'label': 'Sound of a dog'}, {'score': 0.0004, 'label': 'Sound of vaccum cleaner'}]
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This audio
classification pipeline can currently be loaded from [`pipeline`] using the following task identifier:
`"zero-shot-audio-classification"`. See the list of available models on
[huggingface.co/models](https://huggingface.co/models?filter=zero-shot-audio-classification).
c s. t jf | | jdkr*td| j dd S )NptzThe z is only available in PyTorch.)super__init__ framework
ValueError __class__)selfkwargsr Y/tmp/pip-unpacked-wheel-zw5xktn0/transformers/pipelines/zero_shot_audio_classification.pyr ? s
z,ZeroShotAudioClassificationPipeline.__init__)audiosc s t j|f|S )a
Assign labels to the audio(s) passed as inputs.
Args:
audios (`str`, `List[str]`, `np.array` or `List[np.array]`):
The pipeline handles three types of inputs:
- A string containing a http link pointing to an audio
- A string containing a local path to an audio
- An audio loaded in numpy
candidate_labels (`List[str]`):
The candidate labels for this audio
hypothesis_template (`str`, *optional*, defaults to `"This is a sound of {}"`):
The sentence used in cunjunction with *candidate_labels* to attempt the audio classification by
replacing the placeholder with the candidate_labels. Then likelihood is estimated by using
logits_per_audio
Return:
A list of dictionaries containing result, one dictionary per proposed label. The dictionaries contain the
following keys:
- **label** (`str`) -- The label identified by the model. It is one of the suggested `candidate_label`.
- **score** (`float`) -- The score attributed by the model for that label (between 0 and 1).
)r
__call__)r r r r r r r F s z,ZeroShotAudioClassificationPipeline.__call__c K s6 i }d|kr|d |d< d|kr,|d |d< |i i fS )Ncandidate_labelshypothesis_templater )r r Zpreprocess_paramsr r r _sanitize_parameters^ s z8ZeroShotAudioClassificationPipeline._sanitize_parametersNThis is a sound of {}.c s t |trJ|ds |dr,t|j}n t|d}| }W 5 Q R X t |trbt || j
j }t |t j
svtdt|jdkrtd| j
|g| j
j dd}| jdkr|| j}||d < fd
d |D }| j|| jd d
}|g|d< |S )Nzhttp://zhttps://rbz"We expect a numpy ndarray as inputr zNWe expect a single channel audio input for ZeroShotAudioClassificationPipeliner )
sampling_ratereturn_tensorsr c s g | ]} |qS r )format).0xr r r
s zBZeroShotAudioClassificationPipeline.preprocess..T)r padding text_inputs)
isinstancestr
startswithrequestsgetcontentopenreadbytesr Zfeature_extractorr npndarray TypeErrorlenshaper r toZ torch_dtype tokenizer)r audior r finputs sequencesr& r r# r
preprocessg s.
z.ZeroShotAudioClassificationPipeline.preprocessc C sX | d}| d}t|d tr,|d }n |d d }| jf ||}||jd}|S )Nr r& r )r logits)popr' r modelZlogits_per_audio)r Z model_inputsr r& outputs
model_outputsr r r _forward s
z,ZeroShotAudioClassificationPipeline._forwardc C sb | d}|d d }| jdkr6|jdd}| }ntddd tt||d d
d D }|S ) Nr r< r r )Zdimz `tf` framework not supported.c S s g | ]\}}||d qS ))scorelabelr )r! rB Zcandidate_labelr r r r$ s zCZeroShotAudioClassificationPipeline.postprocess..c S s
| d S )Nr r )r" r r r zAZeroShotAudioClassificationPipeline.postprocess..)key)r= r Zsoftmaxtolistr sortedzip)r r@ r r< ZprobsZscoresresultr r r postprocess s
z/ZeroShotAudioClassificationPipeline.postprocess)Nr )__name__
__module__ __qualname____doc__r r r0 r1 r/ r( r r r; rA rK
__classcell__r r r r r s
r ) collectionsr typingr Znumpyr0 r* utilsr r Zaudio_classificationr baser r
Z
get_loggerrL loggerr r r r r s