U DAf? @sddlZddlmZddlmZmZmZmZmZddl m Z m Z m Z m Z mZddlmZmZddlmZddlmZeeZedZdd lmZW5QRXe Gd d d ZdS) N)Path)AnyDictListOptionalUnion)Document componentdefault_from_dictdefault_to_dictlogging)get_bytestream_from_sourcenormalize_metadata) ByteStream) LazyImportzRun 'pip install trafilatura')extractc @seZdZdZdeeeeeeeefdddZ eeefdddZ e eeefdd d d Z e jeed deeeeefeeeeefeeeeffeeeefd ddZdS)HTMLToDocumentae Converts an HTML file to a Document. Usage example: ```python from haystack.components.converters import HTMLToDocument converter = HTMLToDocument() results = converter.run(sources=["path/to/sample.html"]) documents = results["documents"] print(documents[0].content) # 'This is a text from the HTML file.' ``` N)extractor_type try_othersextraction_kwargscCs>t|dk rtdt|dk r0tdt|p6i|_dS)a Create an HTMLToDocument component. :param extractor_type: Ignored. This parameter is kept for compatibility with previous versions. It will be removed in Haystack 2.4.0. To customize the extraction, use the `extraction_kwargs` parameter. :param try_others: Ignored. This parameter is kept for compatibility with previous versions. It will be removed in Haystack 2.4.0. :param extraction_kwargs: A dictionary containing keyword arguments to customize the extraction process. These are passed to the underlying Trafilatura `extract` function. For the full list of available arguments, see the [Trafilatura documentation](https://trafilatura.readthedocs.io/en/latest/corefunctions.html#extract). NzThe `extractor_type` parameter is ignored and will be removed in Haystack 2.4.0. To customize the extraction, use the `extraction_kwargs` parameter.zMThe `try_others` parameter is ignored and will be removed in Haystack 2.4.0. )trafilatura_importcheckwarningswarnDeprecationWarningr)selfrrrrG/tmp/pip-unpacked-wheel-z752163x/haystack/components/converters/html.py__init__%szHTMLToDocument.__init__)returncCst||jdS)z{ Serializes the component to a dictionary. :returns: Dictionary with serialized data. )r)r r)rrrrto_dictDszHTMLToDocument.to_dict)datarcCs t||S)z Deserializes the component from a dictionary. :param data: The dictionary to deserialize from. :returns: The deserialized component. )r )clsr!rrr from_dictMs zHTMLToDocument.from_dict) documents)sourcesmetarc Cs|j|p i}g}t|t|d}t||D]\}}zt|d} Wn:tk r|} ztjd|| dWYq,W5d} ~ XYnXzt| j df|} Wn:tk r} ztjd|| dWYq,W5d} ~ XYnXt | | j |d} | | q,d |iS) ap Converts a list of HTML files to Documents. :param sources: List of HTML file paths or ByteStream objects. :param meta: Optional metadata to attach to the Documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If `sources` contains ByteStream objects, their `meta` will be added to the output Documents. :param extraction_kwargs: Additional keyword arguments to customize the extraction process. :returns: A dictionary with the following keys: - `documents`: Created Documents )r&Z sources_count)sourcez4Could not read {source}. Skipping it. Error: {error})r'errorNzutf-8zAFailed to extract text from {source}. Skipping it. Error: {error})contentr&r$)rrlenzipr Exceptionloggerwarningrr!decoderr&append)rr%r&rZmerged_extraction_kwargsr$Z meta_listr'metadataZ bytestreametextZ conversion_edocumentrrrrunYs* zHTMLToDocument.run)NNN)NN)__name__ __module__ __qualname____doc__rstrboolrrrr classmethodr#r Z output_typesrrrrrr5rrrrrs(   "r)rpathlibrtypingrrrrrZhaystackrr r r r Z$haystack.components.converters.utilsr rZhaystack.dataclassesrZhaystack.lazy_importsr getLoggerr6r-rZ trafilaturarrrrrrs