U 4Afg@s dZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z ddl mZmZmZmZmZmZmZmZdd lmZmZmZeeZd Zd Zd Zd Z d Z!ddZ"GdddZ#Gddde#Z$d ddZ%d!ddZ&ddZ'ddZ(GdddeZ)dS)"z-Factory function to build auto-model classes.N) OrderedDict)PretrainedConfig)get_class_from_dynamic_moduleresolve_trust_remote_code) CONFIG_NAME cached_file copy_funcextract_commit_hashfind_adapter_config_fileis_peft_availableloggingrequires_backends) AutoConfigmodel_type_to_module_name!replace_list_option_in_docstringsaJ This is a generic model class that will be instantiated as one of the model classes of the library when created with the [`~BaseAutoModelClass.from_pretrained`] class method or the [`~BaseAutoModelClass.from_config`] class method. This class cannot be instantiated directly using `__init__()` (throws an error). ax Instantiates one of the model classes of the library from a configuration. Note: Loading a model from its configuration file does **not** load the model weights. It only affects the model's configuration. Use [`~BaseAutoModelClass.from_pretrained`] to load the model weights. Args: config ([`PretrainedConfig`]): The model class to instantiate is selected based on the configuration class: List options attn_implementation (`str`, *optional*): The attention implementation to use in the model (if relevant). Can be any of `"eager"` (manual implementation of the attention), `"sdpa"` (using [`F.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html)), or `"flash_attention_2"` (using [Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention)). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual `"eager"` implementation. Examples: ```python >>> from transformers import AutoConfig, BaseAutoModelClass >>> # Download configuration from huggingface.co and cache. >>> config = AutoConfig.from_pretrained("checkpoint_placeholder") >>> model = BaseAutoModelClass.from_config(config) ``` ac Instantiate one of the model classes of the library from a pretrained model. The model class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: List options The model is set in evaluation mode by default using `model.eval()` (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with `model.train()` Args: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - A path or url to a *tensorflow index checkpoint file* (e.g, `./tf_model/model.ckpt.index`). In this case, `from_tf` should be set to `True` and a configuration object should be provided as `config` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards. model_args (additional positional arguments, *optional*): Will be passed along to the underlying model `__init__()` method. config ([`PretrainedConfig`], *optional*): Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when: - The model is a model provided by the library (loaded with the *model id* string of a pretrained model). - The model was saved using [`~PreTrainedModel.save_pretrained`] and is reloaded by supplying the save directory. - The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a configuration JSON file named *config.json* is found in the directory. state_dict (*Dict[str, torch.Tensor]*, *optional*): A state dictionary to use instead of a state dictionary loaded from saved weights file. This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using [`~PreTrainedModel.save_pretrained`] and [`~PreTrainedModel.from_pretrained`] is not a simpler option. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. from_tf (`bool`, *optional*, defaults to `False`): Load the model weights from a TensorFlow checkpoint save file (see docstring of `pretrained_model_name_or_path` argument). force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. output_loading_info(`bool`, *optional*, defaults to `False`): Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (e.g., not try downloading the model). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. code_revision (`str`, *optional*, defaults to `"main"`): The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. kwargs (additional keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: - If a configuration is provided with `config`, `**kwargs` will be directly passed to the underlying model's `__init__` method (we assume all relevant updates to the configuration have already been done) - If a configuration is not provided, `kwargs` will be first passed to the configuration class initialization function ([`~PretrainedConfig.from_pretrained`]). Each key of `kwargs` that corresponds to a configuration attribute will be used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model's `__init__` function. Examples: ```python >>> from transformers import AutoConfig, BaseAutoModelClass >>> # Download model and configuration from huggingface.co and cache. >>> model = BaseAutoModelClass.from_pretrained("checkpoint_placeholder") >>> # Update configuration during loading >>> model = BaseAutoModelClass.from_pretrained("checkpoint_placeholder", output_attentions=True) >>> model.config.output_attentions True >>> # Loading from a TF checkpoint file instead of a PyTorch model (slower) >>> config = AutoConfig.from_pretrained("./tf_model/shortcut_placeholder_tf_model_config.json") >>> model = BaseAutoModelClass.from_pretrained( ... "./tf_model/shortcut_placeholder_tf_checkpoint.ckpt.index", from_tf=True, config=config ... ) ``` a Instantiate one of the model classes of the library from a pretrained model. The model class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: List options Args: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - A path or url to a *PyTorch state_dict save file* (e.g, `./pt_model/pytorch_model.bin`). In this case, `from_pt` should be set to `True` and a configuration object should be provided as `config` argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards. model_args (additional positional arguments, *optional*): Will be passed along to the underlying model `__init__()` method. config ([`PretrainedConfig`], *optional*): Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when: - The model is a model provided by the library (loaded with the *model id* string of a pretrained model). - The model was saved using [`~PreTrainedModel.save_pretrained`] and is reloaded by supplying the save directory. - The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a configuration JSON file named *config.json* is found in the directory. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. from_pt (`bool`, *optional*, defaults to `False`): Load the model weights from a PyTorch checkpoint save file (see docstring of `pretrained_model_name_or_path` argument). force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. output_loading_info(`bool`, *optional*, defaults to `False`): Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (e.g., not try downloading the model). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. code_revision (`str`, *optional*, defaults to `"main"`): The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. kwargs (additional keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: - If a configuration is provided with `config`, `**kwargs` will be directly passed to the underlying model's `__init__` method (we assume all relevant updates to the configuration have already been done) - If a configuration is not provided, `kwargs` will be first passed to the configuration class initialization function ([`~PretrainedConfig.from_pretrained`]). Each key of `kwargs` that corresponds to a configuration attribute will be used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model's `__init__` function. Examples: ```python >>> from transformers import AutoConfig, BaseAutoModelClass >>> # Download model and configuration from huggingface.co and cache. >>> model = BaseAutoModelClass.from_pretrained("checkpoint_placeholder") >>> # Update configuration during loading >>> model = BaseAutoModelClass.from_pretrained("checkpoint_placeholder", output_attentions=True) >>> model.config.output_attentions True >>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower) >>> config = AutoConfig.from_pretrained("./pt_model/shortcut_placeholder_pt_model_config.json") >>> model = BaseAutoModelClass.from_pretrained( ... "./pt_model/shortcut_placeholder_pytorch_model.bin", from_pt=True, config=config ... ) ``` cCs|t|}t|ttfs|Sdd|D}t|dg}|D]X}||krT||Sd||krt|d|Sd||kr<|d|Sq<|dS)NcSsi|] }|j|qS__name__).0modelrrI/tmp/pip-unpacked-wheel-zw5xktn0/transformers/models/auto/auto_factory.py sz$_get_model_class.. architecturesTFFlaxr)type isinstancelisttuplegetattr)config model_mappingZsupported_modelsZ name_to_modelrarchrrr_get_model_classs   r%c@s>eZdZdZddZeddZeddZed d d ZdS) _BaseAutoModelClassNcOs*t|jjd|jjd|jjddS)Nz+ is designed to be instantiated using the `z5.from_pretrained(pretrained_model_name_or_path)` or `z.from_config(config)` methods.)EnvironmentError __class__r)selfargskwargsrrr__init__s z_BaseAutoModelClass.__init__c Ks2|dd}t|do |j|jk}t||jk}t||j||}|r|r|j|j}d|krp| d\}}n|j }t ||f|}t j |jr||jn|j|j|dd|dd} |j|f|St||jkrt||j}|j|f|Std|jd|jd d d d |jDd dS)Ntrust_remote_codeauto_mapz--Texist_ok code_revision!Unrecognized configuration class  for this kind of AutoModel: . Model type should be one of , css|] }|jVqdSNrrcrrr sz2_BaseAutoModelClass.from_config...)pophasattrrr.r_model_mappingkeysrZ _name_or_pathsplitZ name_or_pathrospathisdirregister_for_auto_classregisterr(Z _from_configr% ValueErrorjoin) clsr"r+r-has_remote_codehas_local_code class_refZrepo_id model_class_rrr from_configs4    0z_BaseAutoModelClass.from_configc s`dd}dd}dd<ddddd d d d d g }fdd|D}dd}dd} dd} |d d} |d d} | dk rtdt| dk rtd| } | dk r| |d <| dkrt|tst|tfdddd|} t | | } n t |dd} t r|| dkr*i} | dk r*| | d <t |fd| i| }|dk r|t |ddd }t|}|| d<|d}W5QRXt|ts$t}dddkrd}dddk rʈd}tj|fd||| d|\}|dddkrdd<|dddk r$|dd<t|d o:|j|jk}t||jk}t||||}| d<|r|r|j|j}t||fd|i|}|dd}tj|r| |jn|j!|j"|dd!|j|f|d|i|St||jkr&t#||j}|j|f|d|i|Std"|j"d#|jd$d%$d&d'|jDd(dS))Nr"r-TZ _from_auto cache_dirZforce_downloadZlocal_files_onlyproxiesZresume_downloadrevisionZ subfolderuse_auth_tokentokencs i|]}|kr||qSr)r;)rnamer+rrrsz7_BaseAutoModelClass.from_pretrained..r1 _commit_hashadapter_kwargszrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.zV`token` and `use_auth_token` are both specified. Please set only the argument `token`.F)Z _raise_exceptions_for_gated_repoZ%_raise_exceptions_for_missing_entriesZ'_raise_exceptions_for_connection_errorsrzutf-8)encodingZ_adapter_model_pathZbase_model_name_or_pathZ torch_dtypeautoZquantization_config)Zreturn_unused_kwargsr-r1rUr.r/r2r3r4r5css|] }|jVqdSr6rr7rrrr98sz6_BaseAutoModelClass.from_pretrained..r:)%r;warningswarn FutureWarningrErrrrr r!r r openjsonloadcopydeepcopygetrfrom_pretrainedr<rr.rr=r>rrr@rArBrCrDr(r%rF)rGpretrained_model_name_or_path model_argsr+r"r-Zhub_kwargs_namesZ hub_kwargsr1Z commit_hashrVrRrQZresolved_config_fileZmaybe_adapter_pathfZadapter_configZ kwargs_origrLrHrIrJrKrrTrrcs                          0z#_BaseAutoModelClass.from_pretrainedFcCsJt|dr4t|jt|kr4td|jd|d|jj|||ddS)a Register a new model for this class. Args: config_class ([`PretrainedConfig`]): The configuration corresponding to the model to register. model_class ([`PreTrainedModel`]): The model to register. config_classzThe model class you are passing has a `config_class` attribute that is not consistent with the config class you passed (model has z and you passed z!. Fix one of those so they match!r/N)r<strrgrEr=rD)rGrgrKr0rrrrD<s z_BaseAutoModelClass.register)F) r __module__ __qualname__r=r, classmethodrMrcrDrrrrr&s  ~r&cs4eZdZdZefddZefddZZS)_BaseAutoBackboneClassNc st|ddgddlm}|d|}|dddk r@td|dd rTtd |d |j}|d |j}|d |j}|d|j } |||||| d}t j |f|S)NZvisionZtimmr)TimmBackboneConfigr"Z out_featuresz0Cannot specify `out_features` for timm backbonesZoutput_loading_infoFz@Cannot specify `output_loading_info=True` when loading from timm num_channels features_onlyuse_pretrained_backbone out_indices)Zbackbonernrorprq) rZmodels.timm_backbonermr;rbrErnrorprqsuperrM) rGrdrer+rmr"rnrorprqr(rr#_load_timm_backbone_from_pretrainedTs&  z:_BaseAutoBackboneClass._load_timm_backbone_from_pretrainedcs6|dd}|r"|j|f||Stj|f||S)Nuse_timm_backboneF)r;rtrrrc)rGrdrer+rursrrrcns z&_BaseAutoBackboneClass.from_pretrained)rrirjr=rkrtrc __classcell__rrrsrrlPs rlcCs,t|dkr |dd|dS|ddS)Nrz(one of the model classes of the library z0one of the model classes of the library (with a z head) z-one of the base model classes of the library )lenreplace)Z docstringhead_docrrrinsert_head_docws  r{google-bert/bert-base-casedc Cs|j}|j}tt|d}|d||_ttj}tt |d}|d|}|d|}||_t |jdd|}t ||_| drt }n| drt}nt}ttj} t||d}|d|}|d|}|dd d d } |d | }|| _t |j| } t | |_|S) N)rzZBaseAutoModelClassZcheckpoint_placeholderF)Zuse_model_typesrr/-rZshortcut_placeholder)r=rr{CLASS_DOCSTRINGry__doc__r r&rMFROM_CONFIG_DOCSTRINGrrk startswithFROM_PRETRAINED_TF_DOCSTRINGFROM_PRETRAINED_FLAX_DOCSTRINGFROM_PRETRAINED_TORCH_DOCSTRINGrcr?) rGZcheckpoint_for_examplerzr#rSZclass_docstringrMZfrom_config_docstringZfrom_pretrained_docstringrcZshortcutrrrauto_class_updates4              rcCs<g}|D]*}t|ttfr,|t|7}q ||q |Sr6)valuesrrr append)r#resultrrrr get_valuess   rc s|dkr dSt|tr,tfdd|DSt|r@t|Std}|krz t||WStk rtd|dd|dYqXntd|d|ddS) Nc3s|]}t|VqdSr6)getattribute_from_module)ramodulerrr9sz+getattribute_from_module..Z transformerszCould not find z neither in z nor in !z in )rr r<r! importlib import_modulerrE)rattrZtransformers_modulerrrrs     $rc@sreZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ ddZ ddZdddZdS)_LazyAutoMappinga " A mapping config to object (model or tokenizer for instance) that will load keys and values when it is accessed. Args: - config_mapping: The map model type to config class - model_mapping: The map model type to model (or tokenizer) class cCs8||_dd|D|_||_||j_i|_i|_dS)NcSsi|]\}}||qSrrrkvrrrrsz-_LazyAutoMapping.__init__..)_config_mappingitems_reverse_config_mappingr=_extra_content_modules)r)Zconfig_mappingr#rrrr,s z_LazyAutoMapping.__init__cCs,t|j|j}t|t|jSr6)setrr> intersectionr=rxr)r)Z common_keysrrr__len__sz_LazyAutoMapping.__len__cs|jkr|jS|jj}||jkr@|j|}|||Sfdd|jD}|D](}||jkr\|j|}|||Sq\tdS)Ncsg|]\}}|jkr|qSrrrkeyrr s z0_LazyAutoMapping.__getitem__..)rrrr=_load_attr_from_modulerrKeyError)r)r model_typeZ model_nameZ model_typesmtyperrr __getitem__s        z_LazyAutoMapping.__getitem__cCs:t|}||jkr*td|d|j|<t|j||S)Nr:ztransformers.models)rrrrr)r)rr module_namerrrrs z'_LazyAutoMapping._load_attr_from_modulecs*fddjD}|tjS)Ncs*g|]"\}}|jkr||qSr)r=r>rrrrSr)rrrsz)_LazyAutoMapping.keys..)rrrrr>)r)Z mapping_keysrrrr>s z_LazyAutoMapping.keyscCs*z ||WStk r$|YSXdSr6)rr)r)rdefaultrrrrbs z_LazyAutoMapping.getcCs t|Sr6)boolr>rrrr__bool__sz_LazyAutoMapping.__bool__cs*fddjD}|tjS)Ncs*g|]"\}}|jkr||qSr)rr>rrrrrrsz+_LazyAutoMapping.values..)r=rrrr)r)Zmapping_valuesrrrrs z_LazyAutoMapping.valuescs*fddjD}|tjS)Ncs>g|]6}|jkr|j||j|fqSr)rr>rr=)rrrrrr s z*_LazyAutoMapping.items..)r=r>rrr)r)Z mapping_itemsrrrrs z_LazyAutoMapping.itemscCs t|Sr6)iterr>rrrr__iter__sz_LazyAutoMapping.__iter__cCs>||jkrdSt|dr$|j|jkr(dS|j|j}||jkS)NTrF)rr<rrr=)r)itemrrrr __contains__s   z_LazyAutoMapping.__contains__FcCsRt|drD|j|jkrD|j|j}||jkrD|sDtd|d||j|<dS)z7 Register a new model in this mapping. r'z*' is already used by a Transformers model.N)r<rrr=r>rEr)r)rvaluer0rrrrrDs  z_LazyAutoMapping.registerN)F)rrirjrr,rrrr>rbrrrrrrDrrrrrs r)rw)r|rw)*rr`rr^r@rZ collectionsrZconfiguration_utilsrZdynamic_module_utilsrrutilsrrr r r r r rZconfiguration_autorrrZ get_loggerrloggerrrrrrr%r&rlr{rrrrrrrrs2  (  lcd>' #