U 4AfC=@sdZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z ddl mZddlmZdd lmZdd lmZdd lmZmZmZmZd d lmZd dlmZmZmZm Z d dl!m"Z"d dl#m$Z$d dl%m&Z&e'e(Z)eddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;dd?d@dAdBdCdDdEdFdGdHdIdJdKdLdMg<Z*eee*Z+e,dNdOdPZ-GdQdRdRZ.dS)SzAutoProcessor class.N) OrderedDict)PretrainedConfig)get_class_from_dynamic_moduleresolve_trust_remote_code)FeatureExtractionMixin)ImageProcessingMixin)ProcessorMixin)TOKENIZER_CONFIG_FILE)FEATURE_EXTRACTOR_NAMEPROCESSOR_NAMEget_file_from_repologging)_LazyAutoMapping)CONFIG_MAPPING_NAMES AutoConfigmodel_type_to_module_name!replace_list_option_in_docstrings)AutoFeatureExtractor)AutoImageProcessor) AutoTokenizer)ZalignZAlignProcessor)ZaltclipZAltCLIPProcessor)ZbarkZ BarkProcessor)ZblipZ BlipProcessor)zblip-2ZBlip2Processor)Z bridgetowerZBridgeTowerProcessor)Z chameleonZChameleonProcessor)Z chinese_clipZChineseCLIPProcessor)ZclapZ ClapProcessor)Zclip CLIPProcessor)ZclipsegZCLIPSegProcessor)ZclvpZ ClvpProcessor)ZflavaZFlavaProcessor)ZfuyuZ FuyuProcessor)gitZ GitProcessor)zgrounding-dinoZGroundingDinoProcessor)Zgroupvitr)ZhubertWav2Vec2Processor)ZideficsZIdeficsProcessor)Zidefics2ZIdefics2Processor)Z instructblipZInstructBlipProcessor)ZinstructblipvideoZInstructBlipVideoProcessor)zkosmos-2ZKosmos2Processor)Z layoutlmv2ZLayoutLMv2Processor)Z layoutlmv3ZLayoutLMv3Processor)ZllavaLlavaProcessor)zllava-next-videoZLlavaNextVideoProcessor)Z llava_nextZLlavaNextProcessor)ZmarkuplmZMarkupLMProcessor)ZmctctZMCTCTProcessor)zmgp-strZMgpstrProcessor)Z oneformerZOneFormerProcessor)Zowlv2ZOwlv2Processor)ZowlvitZOwlViTProcessor)Z paligemmaZPaliGemmaProcessor)Z pix2structZPix2StructProcessor)Z pop2pianoZPop2PianoProcessor)ZsamZ SamProcessor)Z seamless_m4tZSeamlessM4TProcessor)Zsewr)zsew-dr)ZsiglipZSiglipProcessor)Zspeech_to_textZSpeech2TextProcessor)Zspeech_to_text_2ZSpeech2Text2Processor)Zspeecht5ZSpeechT5Processor)ZtrocrZTrOCRProcessor)ZtvltZ TvltProcessor)ZtvpZ TvpProcessor)Z unispeechr)z unispeech-satr)Z video_llavaZVideoLlavaProcessor)ZviltZ ViltProcessor)Zvipllavar)zvision-text-dual-encoderZVisionTextDualEncoderProcessor)Zwav2vec2r)z wav2vec2-bertr)zwav2vec2-conformerr)Zwavlmr)ZwhisperZWhisperProcessor)ZxclipZXCLIPProcessor) class_namec CstD]T\}}||krt|}td|d}zt||WStk rZYqYqXqtj D]}t|dd|krh|Sqhtd}t ||rt||SdS)N.ztransformers.models__name__Z transformers) PROCESSOR_MAPPING_NAMESitemsr importlib import_modulegetattrAttributeErrorPROCESSOR_MAPPINGZ_extra_contentvalueshasattr)r module_nameZ processorsmodule processorZ main_moduler+L/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/auto/processing_auto.pyprocessor_class_from_namers     r-c@s:eZdZdZddZeeeddZe d ddZ d S) AutoProcessora This is a generic processor class that will be instantiated as one of the processor classes of the library when created with the [`AutoProcessor.from_pretrained`] class method. This class cannot be instantiated directly using `__init__()` (throws an error). cCs tddS)Nz}AutoProcessor is designed to be instantiated using the `AutoProcessor.from_pretrained(pretrained_model_name_or_path)` method.)EnvironmentError)selfr+r+r,__init__szAutoProcessor.__init__c sdd}|dk r@tdtdddk r8td|d<dd}dd}dd <d}d}fd d ttj D}t|t f|} | dk rt j |f\} } | d d}d | dikr| dd }|dkrt|tf|} | dk r4tj|f\} } | d d}d | dikr4| dd }| dk r|dkrtj|f\} } | d d}d | dikr| dd }|dkrt|tf|} | dk rt| dd}t|} W5QRX| d d}d | dikr| dd }|dkrLt|tstj|fd|i}t|d d}t|drLd |jkrL|jd }|dk r^t|}|dk }|dk pzt|t k}t!||||}|r|rt"||f}dd} t#j$%|r|&|j|fd|iS|dk r|j|fd|iSt|t kr t t|j|fSzt'j|fd|iWSt(k rzt)j|fd|iWYSt(k r|YnXzt*j|fd|iWYSt(k rYnXYnXtd|ddS)a Instantiate one of the processor classes of the library from a pretrained model vocabulary. The processor class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible): List options Params: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on huggingface.co. - a path to a *directory* containing a processor files saved using the `save_pretrained()` method, e.g., `./my_model_directory/`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model feature extractor should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the feature extractor files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. return_unused_kwargs (`bool`, *optional*, defaults to `False`): If `False`, then this function returns just the final feature extractor object. If `True`, then this functions returns a `Tuple(feature_extractor, unused_kwargs)` where *unused_kwargs* is a dictionary consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of `kwargs` which has not been used to update `feature_extractor` and is otherwise ignored. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. kwargs (`Dict[str, Any]`, *optional*): The values in kwargs of any keys which are feature extractor attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are *not* feature extractor attributes is controlled by the `return_unused_kwargs` keyword parameter. Passing `token=True` is required when you want to use a private model. Examples: ```python >>> from transformers import AutoProcessor >>> # Download processor from huggingface.co and cache. >>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-base-960h") >>> # If processor files are in a directory (e.g. processor was saved using *save_pretrained('./test/saved_model/')*) >>> # processor = AutoProcessor.from_pretrained("./test/saved_model/") ```use_auth_tokenNzrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.tokenzV`token` and `use_auth_token` are both specified. Please set only the argument `token`.configtrust_remote_codeTZ _from_autocsi|]}|kr||qSr+r+).0keykwargsr+r, sz1AutoProcessor.from_pretrained..processor_classr.auto_mapzutf-8)encodingZ code_revisionz!Unrecognized processing class in z. Can't instantiate a processor, a tokenizer, an image processor or a feature extractor for this model. Make sure the repository contains the files of at least one of those processing classes.)+popwarningswarn FutureWarningget ValueErrorinspect signaturer parameterskeysr r Zget_processor_dictr rZget_image_processor_dictrZget_feature_extractor_dictr openjsonload isinstancerrfrom_pretrainedr#r'r<r-typer%rrospathisdirZregister_for_auto_classr Exceptionrr)clsZpretrained_model_name_or_pathr9r2r4r5r;Zprocessor_auto_mapZget_file_from_repo_kwargsZprocessor_config_fileZ config_dict_Zpreprocessor_config_fileZtokenizer_config_filereaderZhas_remote_codeZhas_local_coder+r8r,rLs"E                            zAutoProcessor.from_pretrainedFcCstj|||ddS)a Register a new processor for this class. Args: config_class ([`PretrainedConfig`]): The configuration corresponding to the model to register. processor_class ([`FeatureExtractorMixin`]): The processor to register. )exist_okN)r%register)Z config_classr;rUr+r+r,rVcs zAutoProcessor.registerN)F) r __module__ __qualname____doc__r1 classmethodrrrL staticmethodrVr+r+r+r,r.s Jr.)/rYr!rDrIrNr? collectionsrZconfiguration_utilsrZdynamic_module_utilsrrZfeature_extraction_utilsrZimage_processing_utilsrZprocessing_utilsr Ztokenization_utilsr utilsr r r rZ auto_factoryrZconfiguration_autorrrrZfeature_extraction_autorZimage_processing_autorZtokenization_autorZ get_loggerrloggerrr%strr-r.r+r+r+r,s           A