U 4Af LE@sdZddlZddlZddlZddlZddlmZddlmZm Z m Z ddl m Z ddl mZmZddlmZdd lmZmZmZmZd d lmZd d lmZmZmZmZeeZ ed ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;dd?d@dAdBdCdDdEdFdGdHdIdJdKdLdMdNdOdPgDZ!eee!Z"e#dQdRdSZ$dZe e#ej%fe e e#ej%fe&e e&e ee#e#fe e e&e#fe e#e&dUdVdWZ'GdXdYdYZ(dS)[zAutoFeatureExtractor class.N) OrderedDict)DictOptionalUnion)PretrainedConfig)get_class_from_dynamic_moduleresolve_trust_remote_code)FeatureExtractionMixin) CONFIG_NAMEFEATURE_EXTRACTOR_NAMEget_file_from_repologging)_LazyAutoMapping)CONFIG_MAPPING_NAMES AutoConfigmodel_type_to_module_name!replace_list_option_in_docstrings)zaudio-spectrogram-transformerZASTFeatureExtractor)ZbeitBeitFeatureExtractor)Z chinese_clipZChineseCLIPFeatureExtractor)ZclapZClapFeatureExtractor)ZclipCLIPFeatureExtractor)ZclipsegViTFeatureExtractor)ZclvpZClvpFeatureExtractor)Zconditional_detrZConditionalDetrFeatureExtractor)ZconvnextConvNextFeatureExtractor)cvtr)zdata2vec-audioWav2Vec2FeatureExtractor)zdata2vec-visionr)Zdeformable_detrZDeformableDetrFeatureExtractor)ZdeitZDeiTFeatureExtractor)ZdetrDetrFeatureExtractor)Zdinatr)z donut-swinZDonutFeatureExtractor)ZdptZDPTFeatureExtractor)ZencodecZEncodecFeatureExtractor)ZflavaZFlavaFeatureExtractor)ZglpnZGLPNFeatureExtractor)Zgroupvitr)Zhubertr)ZimagegptZImageGPTFeatureExtractor)Z layoutlmv2ZLayoutLMv2FeatureExtractor)Z layoutlmv3ZLayoutLMv3FeatureExtractor)ZlevitZLevitFeatureExtractor)Z maskformerZMaskFormerFeatureExtractor)ZmctctZMCTCTFeatureExtractor)Z mobilenet_v1ZMobileNetV1FeatureExtractor)Z mobilenet_v2ZMobileNetV2FeatureExtractor)Z mobilevitZMobileViTFeatureExtractor)Znatr)ZowlvitZOwlViTFeatureExtractor)Z perceiverZPerceiverFeatureExtractor)Z poolformerZPoolFormerFeatureExtractor)Z pop2pianoZPop2PianoFeatureExtractor)Zregnetr)Zresnetr)Z seamless_m4tSeamlessM4TFeatureExtractor)Zseamless_m4t_v2r)Z segformerZSegformerFeatureExtractor)Zsewr)zsew-dr)Zspeech_to_textZSpeech2TextFeatureExtractor)Zspeecht5ZSpeechT5FeatureExtractor)Z swiftformerr)Zswinr)Zswinv2r)ztable-transformerr)Z timesformerVideoMAEFeatureExtractor)ZtvltZTvltFeatureExtractor)Z unispeechr)z unispeech-satr)ZunivnetZUnivNetFeatureExtractor)Zvanr)Zvideomaer)ZviltZViltFeatureExtractor)Zvitr)Zvit_maer)Zvit_msnr)Zwav2vec2r)z wav2vec2-bertr)zwav2vec2-conformerr)Zwavlmr)ZwhisperZWhisperFeatureExtractor)Zxclipr)ZyolosZYolosFeatureExtractor) class_namec CstD]T\}}||krt|}td|d}zt||WStk rZYqYqXqtjD] \}}t|dd|krh|Sqhtd}t ||rt||SdS)N.ztransformers.models__name__Z transformers) FEATURE_EXTRACTOR_MAPPING_NAMESitemsr importlib import_modulegetattrAttributeErrorFEATURE_EXTRACTOR_MAPPINGZ_extra_contenthasattr)r module_nameZ extractorsmodule_Z extractorZ main_moduler,T/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/auto/feature_extraction_auto.py!feature_extractor_class_from_namets     r.F)pretrained_model_name_or_path cache_dirforce_downloadresume_downloadproxiestokenrevisionlocal_files_onlyc Ks|dd} | dk r4tdt|dk r0td| }t|t|||||||d } | dkrdtdiSt | dd} t | W5QRSQRXdS) a2 Loads the tokenizer configuration from a pretrained model tokenizer configuration. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, will only try to load the tokenizer configuration from local files. Passing `token=True` is required when you want to use a private model. Returns: `Dict`: The configuration of the tokenizer. Examples: ```python # Download configuration from huggingface.co and cache. tokenizer_config = get_tokenizer_config("google-bert/bert-base-uncased") # This model does not have a tokenizer config so the result will be an empty dict. tokenizer_config = get_tokenizer_config("FacebookAI/xlm-roberta-base") # Save a pretrained tokenizer locally and you can reload its config from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased") tokenizer.save_pretrained("tokenizer-test") tokenizer_config = get_tokenizer_config("tokenizer-test") ```use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.)r0r1r2r3r4r5r6zdCould not locate the feature extractor configuration file, will try to use the model config instead.zutf-8)encoding) popwarningswarn FutureWarning ValueErrorr r loggerinfoopenjsonload) r/r0r1r2r3r4r5r6kwargsr7Zresolved_config_filereaderr,r,r-get_feature_extractor_configs6E  rGc@s:eZdZdZddZeeeddZe d ddZ d S) AutoFeatureExtractora+ This is a generic feature extractor class that will be instantiated as one of the feature extractor classes of the library when created with the [`AutoFeatureExtractor.from_pretrained`] class method. This class cannot be instantiated directly using `__init__()` (throws an error). cCs tddS)NzAutoFeatureExtractor is designed to be instantiated using the `AutoFeatureExtractor.from_pretrained(pretrained_model_name_or_path)` method.)EnvironmentError)selfr,r,r-__init__szAutoFeatureExtractor.__init__c Ks|dd}|dk r@tdt|dddk r8td||d<|dd}|dd}d|d <tj|f|\}}|d d}d} d |d ikr|d d } |dkr| dkrt|t st j |f|}t |d d}t |d rd |jkr|jd } |dk rt|}| dk } |dk pt|tk} t||| | }| rx|rxt| |f|}|d d}tj|rj||j|f|S|dk r|j|f|St|tkrtt|}|j|f|Std|dtdtdtddddtD dS)a Instantiate one of the feature extractor classes of the library from a pretrained model vocabulary. The feature extractor class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: List options Params: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on huggingface.co. - a path to a *directory* containing a feature extractor file saved using the [`~feature_extraction_utils.FeatureExtractionMixin.save_pretrained`] method, e.g., `./my_model_directory/`. - a path or url to a saved feature extractor JSON *file*, e.g., `./my_model_directory/preprocessor_config.json`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model feature extractor should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the feature extractor files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. return_unused_kwargs (`bool`, *optional*, defaults to `False`): If `False`, then this function returns just the final feature extractor object. If `True`, then this functions returns a `Tuple(feature_extractor, unused_kwargs)` where *unused_kwargs* is a dictionary consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of `kwargs` which has not been used to update `feature_extractor` and is otherwise ignored. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. kwargs (`Dict[str, Any]`, *optional*): The values in kwargs of any keys which are feature extractor attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are *not* feature extractor attributes is controlled by the `return_unused_kwargs` keyword parameter. Passing `token=True` is required when you want to use a private model. Examples: ```python >>> from transformers import AutoFeatureExtractor >>> # Download feature extractor from huggingface.co and cache. >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h") >>> # If feature extractor files are in a directory (e.g. feature extractor was saved using *save_pretrained('./test/saved_model/')*) >>> # feature_extractor = AutoFeatureExtractor.from_pretrained("./test/saved_model/") ```r7Nr8r4r9configtrust_remote_codeTZ _from_autoZfeature_extractor_typerHauto_mapZ code_revisionz"Unrecognized feature extractor in z4. Should have a `feature_extractor_type` key in its z of z3, or one of the following `model_type` keys in its z: z, css|] }|VqdS)Nr,).0cr,r,r- sz7AutoFeatureExtractor.from_pretrained..)r;r<r=r>getr?r Zget_feature_extractor_dict isinstancerrfrom_pretrainedr%r(rNr.typer'r rospathisdirZregister_for_auto_class from_dictr r joinr!keys) clsr/rEr7rLrMZ config_dictr+feature_extractor_classZfeature_extractor_auto_mapZhas_remote_codeZhas_local_coder,r,r-rTslI             4z$AutoFeatureExtractor.from_pretrainedFcCstj|||ddS)a0 Register a new feature extractor for this class. Args: config_class ([`PretrainedConfig`]): The configuration corresponding to the model to register. feature_extractor_class ([`FeatureExtractorMixin`]): The feature extractor to register. )exist_okN)r'register)Z config_classr]r^r,r,r-r_s zAutoFeatureExtractor.registerN)F) r __module__ __qualname____doc__rK classmethodrr!rT staticmethodr_r,r,r,r-rHs rH)NFNNNNF))rbr#rCrVr< collectionsrtypingrrrZconfiguration_utilsrZdynamic_module_utilsrr Zfeature_extraction_utilsr utilsr r r rZ auto_factoryrZconfiguration_autorrrrZ get_loggerr r@r!r'strr.PathLikeboolrGrHr,r,r,r-s     I   d