U DAf; @sddlZddlmZddlmZmZmZmZmZddl m Z m Z m Z ddl mZmZddlmZddlmZed"Zdd lmZdd lmZmZW5QRXe eZe Gd d d ZdS) N)Path)AnyDictListOptionalUnion)Document componentlogging)get_bytestream_from_sourcenormalize_metadata) ByteStream) LazyImportzRun 'pip install pdfminer.six') extract_pages)LAParamsLTTextContainerc @seZdZdZdeeeeeeeeddd d Zed d d Z e j e edde e eeefee eeefe eeeffdddZdS)PDFMinerToDocumenta Converts PDF files to Documents. Uses `pdfminer` compatible converters to convert PDF files to Documents. https://pdfminersix.readthedocs.io/en/latest/ Usage example: ```python from haystack.components.converters.pdfminer import PDFMinerToDocument converter = PDFMinerToDocument() results = converter.run(sources=["sample.pdf"], meta={"date_added": datetime.now().isoformat()}) documents = results["documents"] print(documents[0].content) # 'This is a text from the PDF file.' ``` ?@皙?TFN) line_overlap char_margin line_margin word_margin boxes_flowdetect_vertical all_textsreturnc Cs$tt|||||||d|_dS)ai Create a PDFMinerToDocument component. :param line_overlap: This parameter determines whether two characters are considered to be on the same line based on the amount of overlap between them. The overlap is calculated relative to the minimum height of both characters. :param char_margin: Determines whether two characters are part of the same line based on the distance between them. If the distance is less than the margin specified, the characters are considered to be on the same line. The margin is calculated relative to the width of the character. :param word_margin: Determines whether two characters on the same line are part of the same word based on the distance between them. If the distance is greater than the margin specified, an intermediate space will be added between them to make the text more readable. The margin is calculated relative to the width of the character. :param line_margin: This parameter determines whether two lines are part of the same paragraph based on the distance between them. If the distance is less than the margin specified, the lines are considered to be part of the same paragraph. The margin is calculated relative to the height of a line. :param boxes_flow: This parameter determines the importance of horizontal and vertical position when determining the order of text boxes. A value between -1.0 and +1.0 can be set, with -1.0 indicating that only horizontal position matters and +1.0 indicating that only vertical position matters. Setting the value to 'None' will disable advanced layout analysis, and text boxes will be ordered based on the position of their bottom left corner. :param detect_vertical: This parameter determines whether vertical text should be considered during layout analysis. :param all_texts: If layout analysis should be performed on text in figures. )rrrrrrrN)pdfminer_importcheckr layout_params)selfrrrrrrrr"K/tmp/pip-unpacked-wheel-z752163x/haystack/components/converters/pdfminer.py__init__(s+zPDFMinerToDocument.__init__)rcCsPg}|D]2}d}|D]}t|tr||7}q||qd|}t|dS)z Extracts text from PDF pages then convert the text into Documents :param extractor: Python generator that yields PDF pages. :returns: PDF text converted to Haystack Document  )content) isinstancerZget_textappendjoinr)r!Z extractorZpagespagetext containerconcatr"r"r#Z __converter_s    zPDFMinerToDocument.__converter) documents)sourcesmetac Csg}t|t|d}t||D]\}}z t|}Wn:tk rl}ztjd||dWYqW5d}~XYnXz$tt |j |j d} | | } Wn:tk r}ztjd||dWYqW5d}~XYnX|j |} | | _ || qd|iS)a Converts PDF files to Documents. :param sources: List of PDF file paths or ByteStream objects. :param meta: Optional metadata to attach to the Documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If `sources` contains ByteStream objects, their `meta` will be added to the output Documents. :returns: A dictionary with the following keys: - `documents`: Created Documents )Z sources_countz4Could not read {source}. Skipping it. Error: {error})sourceerrorN)ZlaparamszECould not read {source} and convert it to Document, skipping. {error}r/)r lenzipr ExceptionloggerwarningrioBytesIOdatar _PDFMinerToDocument__converterr1r)) r!r0r1r/Z meta_listr2metadataZ bytestreameZ pdf_readerdocumentZmerged_metadatar"r"r#runws,   zPDFMinerToDocument.run)rrrrrTF)N)__name__ __module__ __qualname____doc__floatrboolr$rr<r Z output_typesrrstrrr rrr@r"r"r"r#rs2 7"r)r9pathlibrtypingrrrrrZhaystackrr r Z$haystack.components.converters.utilsr r Zhaystack.dataclassesr Zhaystack.lazy_importsrrZpdfminer.high_levelrZpdfminer.layoutrr getLoggerrAr7rr"r"r"r#s