Extracting text from PDFs

Text extraction turns a page's positioned glyphs back into readable text: spans are grouped into lines, lines are joined with \n, and spaces are inserted at horizontal gaps. By default reading order follows the content stream: a typeset document writes each column whole before the next begins, so the text comes out column by column, and a figure caption or a footnote reads where the producer placed it. Geometry corrects the streams that write across two columns row by row (a page with a clear gutter still reads column-major) and takes over entirely when a stream was not written in reading order at all, which then reads top to bottom. Two other orders can be selected; see Reading order. Whole-document extraction joins pages with a form feed (\f). For structured output (headings, lists, tables), see Markdown output; for the spans themselves, with fonts, sizes and positions, see Styled spans.

Text drawn turned a quarter, such as a landscape exhibit set on a portrait page or a label running up a chart's side, is laid out in its own upright frame: each span's baseline direction is snapped to the nearest quarter turn, the spans sharing one are grouped into lines and tables the way upright text is, and the groups follow one another top to bottom by where they sit on the page. Text at other angles, such as a diagonal watermark, is treated as the nearest quarter turn. A page with no turned text skips this step.

Reading order

Every extraction entry point takes a reading order. There are three:

pdfboss text report.pdf --reading-order structure-tree
pdfboss md report.pdf --reading-order geometric

from pdfboss import Document, ReadingOrder

doc = Document("report.pdf")
doc.extract_text(reading_order=ReadingOrder.STRUCTURE_TREE)
doc[0].extract_markdown(reading_order="geometric")   # the string value works too

In Rust the order is the last argument of every entry point:

use pdfboss_core::Document;
use pdfboss_output::ReadingOrder;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let doc = Document::open("report.pdf")?;
    let page = doc.page(0)?;
    let (text, report) =
        pdfboss_output::extract_text_reporting(&doc, &page, ReadingOrder::StructureTree)?;
    println!("{text}");
    println!("read in {} order", report.order);
    Ok(())
}

ExtractReport::order says which order the page actually came out in: the one requested, or Content when the structure tree was asked for and the page has no entry in it. The content-order walk is compiled without the marked-content bookkeeping the structure-tree walk needs, so the default costs nothing extra.

Annotations in the reading order

Annotations live in the page's /Annots array, not in the content stream, so their place in the reading order comes from the structure tree alone (ISO 32000-1 14.8.2.3.2): a Link, Form or Annot element holds the annotation through an object reference (/Type /OBJR, 14.7.4.3), and the annotation precedes or follows the marked-content sequences its element sits between. Page.content_items() returns the page's content items in that order, sequences and annotations ranked together, each with the element holding it. Text extraction returns no annotation text, so this is where an annotation's slot among the text is read off:

page = doc[0]
annotations = {a.ref: a for a in page.annotations()}
for item in page.content_items():
    if item.kind == "object":
        print(item.rank, item.standard_type, annotations[item.ref].subtype)
    else:
        print(item.rank, item.standard_type, "marked content", item.mcid)

In Rust Document::content_items(&page) returns the same list as PlacedItem records, and StructureTree::place_items_with places any set of ContentItems (sequences by their /StructParents key and MCID, objects by reference) in one shared rank space; the spans of pdfboss_text under structure-tree order carry the holding element in structure.path, which is how a span and an item of the same Link element are matched. Only the page's own content stream and its /Annots are enumerated: the sequences of a form XObject with its own /StructParents are placed when handed in. The Python spans expose the element's description and language but not its reference, so in Python the match between a span and an item goes through the annotation's rectangle.

CLI

pdfboss text report.pdf

prints every page, separated by form feeds. --page selects one page, 1-based:

pdfboss text --page 4 report.pdf

Content that cannot be read is reported on stderr, one warning per skipped stream, with the same 1-based page numbers:

warning: page 17: skipped a form XObject (form limit exceeded)

stdout carries only the extracted text, so the output stays safe to pipe. Encrypted files take --password. See Encrypted documents.

Python

from pdfboss import Document

doc = Document("report.pdf")

text = doc.extract_text()           # all pages, joined by "\f"
page_text = doc[3].extract_text()   # one page (0-based index)

Document.extract_text fans the pages out across the machine's cores: each worker thread holds its own fork of the document (the immutable parsed core is shared, the caches are private), and one font cache serves every worker so each font loads once per document. Both calls release the GIL while they run, so other Python threads keep making progress during long extractions.

Rust

Per page, with pdfboss_output::extract_text:

use pdfboss_core::Document;
use pdfboss_output::ReadingOrder;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let doc = Document::open("report.pdf")?;
    for i in 0..doc.page_count() {
        let page = doc.page(i)?;
        let text = pdfboss_output::extract_text(&doc, &page, ReadingOrder::Content)?;
        println!("{text}");
    }
    Ok(())
}

For a whole document, pdfboss_core::map_pages runs a closure over every page in parallel and returns the results in page order. It fans out over std::thread::available_parallelism() threads, each holding its own document fork; workers pull page indexes from a shared counter, so pages of uneven cost cannot strand a fast core behind a slow stripe. Pass one FontCache to the _cached variant so fonts load once per document rather than once per page:

use pdfboss_core::{map_pages, Document};
use pdfboss_output::{FontCache, ReadingOrder};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let doc = Document::open("report.pdf")?;
    let fonts = FontCache::default();
    let outcomes = map_pages(&doc, |doc, page| {
        let (text, _) = pdfboss_output::extract_text_reporting_cached(
            doc,
            page,
            &fonts,
            ReadingOrder::Content,
        )?;
        Ok(text)
    });
    let texts = outcomes.into_iter().collect::<Result<Vec<String>, _>>()?;
    println!("{}", texts.join("\u{c}"));
    Ok(())
}

Under ReadingOrder::StructureTree the _cached call still loads the structure tree on every page, and loading reads the whole parent tree. A document walk loads it once with structure_for, the optional-content state once with oc_state, and drives the _with twin over Immediate from each worker. The tree keeps what each page walk read for the next page, so map_pages_forking gives every worker its own fork of it (the same tree over an empty cache) rather than one tree the workers would queue on:

use pdfboss_core::{map_pages_forking, source::block_on, Document, Immediate};
use pdfboss_output::{structure_for, FontCache, ReadingOrder};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let doc = Document::open("report.pdf")?;
    let order = ReadingOrder::StructureTree;
    let fonts = FontCache::default();
    let oc = doc.oc_state();
    let structure = structure_for(&doc, order);
    let outcomes = map_pages_forking(
        &doc,
        || structure.clone(),
        |structure, doc, page| {
            let (text, _) = block_on(pdfboss_output::extract_text_reporting_cached_with(
                Immediate(doc),
                page,
                &fonts,
                oc.as_ref(),
                structure.as_ref(),
                order,
            ))?;
            Ok(text)
        },
    );
    let texts = outcomes.into_iter().collect::<Result<Vec<String>, _>>()?;
    println!("{}", texts.join("\u{c}"));
    Ok(())
}

Asynchronous callers use extract_text_with against any object source. See Async and remote documents.

Lenient semantics and reporting

Extraction is lenient the way rendering is: content that will not fetch, decode, or parse yields no text rather than an error, so one unreadable stream never costs the rest of the document. extract_text_reporting is what keeps that leniency accountable. It returns the text together with an ExtractReport:

use pdfboss_core::Document;
use pdfboss_output::ReadingOrder;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let doc = Document::open("report.pdf")?;
    let page = doc.page(0)?;
    let (text, report) =
        pdfboss_output::extract_text_reporting(&doc, &page, ReadingOrder::Content)?;
    for skip in &report.skipped {
        eprintln!("skipped {} ({})", skip.kind, skip.cause);
    }
    println!("{text}");
    Ok(())
}

report.skipped names each stream that yielded no text: the kind (the page's own contents, a form XObject, an unresolvable XObject, a font's CMap encoding) and the cause (an unsupported filter, a stream that would not read, content that would not parse, a missing resource, an exhausted form nesting or invocation limit). report.hidden counts content the document's optional-content configuration turns off; that is configured behavior, not a loss, so report.is_complete() ignores it and is true exactly when nothing was left out. An empty text with an empty report really is an empty page. The CLI warnings above are this report, printed. Layers the document turns off on screen are excluded from the text: the default optional-content configuration with the View usage application applied, so a print-only layer is left out while a group of another /Intent is kept. The Rust _with entries take the state explicitly, so doc.oc_state_for(Some(OcEvent::Print)) extracts the text as printed.

Encodings

Character decoding covers ToUnicode CMaps, the WinAnsi, MacRoman and Standard simple-font encodings, the built-in encoding of an embedded Type 1 program (the base table of a simple font that names no /Encoding of its own, which is how TeX's symbol and math fonts arrive), and CID-keyed Type0 fonts: embedded /Encoding CMap streams parse, the predefined ISO 32000 CJK CMap set is compiled in (behind the predefined-cmaps feature, on by default in the CLI and the Python wheel), and when /ToUnicode is absent CIDs map to Unicode through the font's character collection. Glyph names resolve through the full Adobe Glyph List plus the TeX symbol-font names the list lacks, so ligatures (fi, fl, …), small-caps variants and math symbols (∀, ∃, ↦, ′) decode to their proper Unicode text.