U 5Af@sddlmZddlmZmZddlmZmZmZm Z m Z m Z ddl m Z mZe rjddlmZddlmZerdd lZdd lmZerdd lmZdd lmZe eZeed dGddde Zd S))UserDict)ListUnion)add_end_docstringsis_tf_availableis_torch_availableis_vision_availableloggingrequires_backends)Pipelinebuild_pipeline_init_args)Image) load_imageN)6MODEL_FOR_ZERO_SHOT_IMAGE_CLASSIFICATION_MAPPING_NAMES)9TF_MODEL_FOR_ZERO_SHOT_IMAGE_CLASSIFICATION_MAPPING_NAMES)stable_softmaxT)Zhas_image_processorcsheZdZdZfddZeeeededfdfdd Zdd Z dd d Z ddZ ddZ Z S)#ZeroShotImageClassificationPipelineaL Zero shot image classification pipeline using `CLIPModel`. This pipeline predicts the class of an image when you provide an image and a set of `candidate_labels`. Example: ```python >>> from transformers import pipeline >>> classifier = pipeline(model="google/siglip-so400m-patch14-384") >>> classifier( ... "https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png", ... candidate_labels=["animals", "humans", "landscape"], ... ) [{'score': 0.965, 'label': 'animals'}, {'score': 0.03, 'label': 'humans'}, {'score': 0.005, 'label': 'landscape'}] >>> classifier( ... "https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png", ... candidate_labels=["black and white", "photorealist", "painting"], ... ) [{'score': 0.996, 'label': 'black and white'}, {'score': 0.003, 'label': 'photorealist'}, {'score': 0.0, 'label': 'painting'}] ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This image classification pipeline can currently be loaded from [`pipeline`] using the following task identifier: `"zero-shot-image-classification"`. See the list of available models on [huggingface.co/models](https://huggingface.co/models?filter=zero-shot-image-classification). c s4tjf|t|d||jdkr*tntdS)NZvisiontf)super__init__r Zcheck_model_type frameworkrr)selfkwargs __class__Y/tmp/pip-unpacked-wheel-zw5xktn0/transformers/pipelines/zero_shot_image_classification.pyrBs z,ZeroShotImageClassificationPipeline.__init__r)imagesc stj|f|S)a Assign labels to the image(s) passed as inputs. Args: images (`str`, `List[str]`, `PIL.Image` or `List[PIL.Image]`): The pipeline handles three types of images: - A string containing a http link pointing to an image - A string containing a local path to an image - An image loaded in PIL directly candidate_labels (`List[str]`): The candidate labels for this image hypothesis_template (`str`, *optional*, defaults to `"This is a photo of {}"`): The sentence used in cunjunction with *candidate_labels* to attempt the image classification by replacing the placeholder with the candidate_labels. Then likelihood is estimated by using logits_per_image timeout (`float`, *optional*, defaults to None): The maximum time in seconds to wait for fetching images from the web. If None, no timeout is set and the call may block forever. Return: A list of dictionaries containing result, one dictionary per proposed label. The dictionaries contain the following keys: - **label** (`str`) -- The label identified by the model. It is one of the suggested `candidate_label`. - **score** (`float`) -- The score attributed by the model for that label (between 0 and 1). )r__call__)rrrrrrr Lsz,ZeroShotImageClassificationPipeline.__call__cKsJi}d|kr|d|d<d|kr,|d|d<d|kr@|d|d<|iifS)Ncandidate_labelstimeouthypothesis_templater)rrZpreprocess_paramsrrr_sanitize_parametersms   z8ZeroShotImageClassificationPipeline._sanitize_parametersNThis is a photo of {}.c st||d}|j|g|jd}|jdkr4||j}||d<fdd|D}|jjjdkr`dnd }|j||j|d }|g|d <|S) N)r")rreturn_tensorsptr!csg|]}|qSr)format).0xr#rr ~szBZeroShotImageClassificationPipeline.preprocess..siglip max_lengthT)r&padding text_inputs) rZimage_processorrtoZ torch_dtypemodelconfig model_type tokenizer) rimager!r#r"inputs sequencesr/r0rr+r preprocessxs    z.ZeroShotImageClassificationPipeline.preprocesscCsX|d}|d}t|dtr,|d}n |dd}|jf||}||jd}|S)Nr!r0r)r!logits)pop isinstancerr2Zlogits_per_image)rZ model_inputsr!r0outputs model_outputsrrr_forwards    z,ZeroShotImageClassificationPipeline._forwardcCs|d}|dd}|jdkrX|jjjdkrXt|d}|}t |t s|g}nj|jdkr|j ddd}|}t |t s|g}n4|jdkrt |dd }| }ntd |jd d tt||d ddD}|S)Nr!r:rr'r-)Zdimr)ZaxiszUnsupported framework: cSsg|]\}}||dqS))scorelabelr)r)rAZcandidate_labelrrrr,szCZeroShotImageClassificationPipeline.postprocess..cSs |d S)Nrr)r*rrrzAZeroShotImageClassificationPipeline.postprocess..)key)r;rr2r3r4torchZsigmoidZsqueezetolistr<listZsoftmaxrZnumpy ValueErrorsortedzip)rr>r!r:ZprobsZscoresresultrrr postprocesss(       z/ZeroShotImageClassificationPipeline.postprocess)Nr%N)__name__ __module__ __qualname____doc__rrstrrr r$r9r?rM __classcell__rrrrr s &! r) collectionsrtypingrrutilsrrrr r r baser rZPILrZ image_utilsrrFZmodels.auto.modeling_autorZmodels.auto.modeling_tf_autorZtf_utilsrZ get_loggerrNloggerrrrrrs