U 4Af[ @s2ddlmZmZddlmZmZmZddlmZm Z m Z ddl m Z ddl mZzLddlmZmZmZdd lmZdd lmZdd lmZdd lmZd ZWn(eefk reZddZdZYnXe dZ edddZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%GdddeZ&dS) )ArgumentParser Namespace)AnyListOptional)Pipelineget_supported_taskspipeline)logging)BaseTransformersCLICommand)BodyFastAPI HTTPException)APIRoute) BaseModel) JSONResponse)runTcOsdS)N)xyrrA/tmp/pip-unpacked-wheel-zw5xktn0/transformers/commands/serving.pyr"srFztransformers-cli/serving)argscCs:t|j|jr|jnd|j|j|jd}t||j|j|j S)z~ Factory function used to instantiate serving server from provided command line arguments. Returns: ServeCommand N)taskmodelconfig tokenizerdevice) r rrrrr ServeCommandhostportworkers)rZnlprrrserve_command_factory+sr#c@seZdZUdZeed<dS)ServeModelInfoResultz" Expose model information infosN)__name__ __module__ __qualname____doc__dict__annotations__rrrrr$;s r$c@s.eZdZUdZeeed<eeeed<dS)ServeTokenizeResultz Tokenize result model tokens tokens_idsN) r&r'r(r)rstrr+rintrrrrr,Cs  r,c@seZdZUdZeed<dS)ServeDeTokenizeResultz! DeTokenize result model textN)r&r'r(r)r/r+rrrrr1Ls r1c@seZdZUdZeed<dS)ServeForwardResultz Forward result model outputN)r&r'r(r)rr+rrrrr3Ts r3c@seZdZeedddZeeeedddZ ddZ d d Z e d d d e dd d fee dddZe d d d e dd d e d d d feee e dddZe d d d fddZd S)r)parsercCs|jddd}|jdttdd|jdtdd d |jd td d d |jdtddd |jdtdd|jdtdd|jdtdd|jdtddd |jtddS)z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments ZservezFCLI tool to run inference requests through REST and GraphQL endpoints.)helpz--taskzThe task to run the pipeline on)typechoicesr6z--host localhostz$Interface the server will listen on.)r7defaultr6z--porti"z Port the serving will listen to.z --workersr zNumber of http workersz--modelz%Model's name or path to stored model.)r7r6z--configz,Model's config name or path to stored model.z --tokenizerzTokenizer name to use.z--devicezSIndicate the device to run onto, -1 indicates CPU, >= 0 indicates GPU (default: -1))funcN) add_parser add_argumentr/r r0 set_defaultsr#)r5Z serve_parserrrrregister_subcommand]s.z ServeCommand.register_subcommand)r r r!r"c Cs||_||_||_||_ts&tdnttd|d|tt d|j t t dgdt d|j tt dgdt d |jtt dgdt d |jtt dgdgd d |_dS) NzUsing serve command requires FastAPI and uvicorn. Please install transformers with [serving]: pip install "transformers[serving]". Or install FastAPI and uvicorn separately.zServing model over :/GET)Zresponse_modelresponse_classmethodsz /tokenizePOSTz /detokenizez/forwardiX)Zroutestimeout) _pipeliner r!r"_serve_dependencies_installed RuntimeErrorloggerinforr model_infor$rtokenizer, detokenizer1forwardr3_app)selfr r r!r"rrr__init__|sRzServeCommand.__init__cCst|j|j|j|jddS)N)r r!r")rrQr r!r"rRrrrrszServeCommand.runcCstt|jjjdS)N)r%)r$varsrHrrrTrrrrMszServeCommand.model_infoNT)embedF) text_input return_idsc Cs|z>|jj|}|r0|jj|}t||dWSt|dWSWn8tk rv}ztddt|ddW5d}~XYnXdS)z Tokenize the provided input and eventually returns corresponding tokens id: - **text_input**: String to tokenize - **return_ids**: Boolean flags indicating if the tokens have to be converted to their integer mapping. )r-r.)r-rerror status_codeZdetailN)rHrrNZconvert_tokens_to_idsr, Exceptionrr/)rRrWrXZ tokens_txtr.errrrNszServeCommand.tokenize)r.skip_special_tokenscleanup_tokenization_spacesc Cs^z |jj|||}td|dWStk rX}ztddt|ddW5d}~XYnXdS)a: Detokenize the provided tokens ids to readable text: - **tokens_ids**: List of tokens ids - **skip_special_tokens**: Flag indicating to not try to decode special tokens - **cleanup_tokenization_spaces**: Flag indicating to remove all leading/trailing spaces and intermediate ones. rZ)rr2rYr[r]N)rHrdecoder1r_rr/)rRr.rarbZ decoded_strr`rrrrOs zServeCommand.detokenizec sht|dkrtggdSz||}t|dWStk rb}ztddt|iW5d}~XYnXdS)zF **inputs**: **attention_mask**: **tokens_type_ids**: r)r4Z attention)r4rYr\N)lenr3rHr_rr/)rRinputsr4r`rrrrPs    zServeCommand.forward)r&r'r( staticmethodrr@rr/r0rSrrMrboolrNrrOrPrrrrr\s1&    rN)'argparserrtypingrrrZ pipelinesrr r utilsr rZr ZfastapirrrZfastapi.routingrZpydanticrZstarlette.responsesrZuvicornrrI ImportErrorAttributeErrorobjectZ get_loggerrKr#r$r,r1r3rrrrrs,