2026-08-23 15:13:42 +02:00

25 KiB

Raw Blame History

Datasheet → Markdown: process

A repeatable runbook for turning a large vendor datasheet PDF into per-chapter Markdown with extracted figures, a table of contents, and an accuracy audit.

Worked examples in this repo:

Renesas RAA489400, 80 pages → 9 chapters, ~200 KB of Markdown, 25 extracted figures.
USB Power Delivery Specification r3.2 v1.2, 416 pages → 10 chapters + 2 appendices, ~1.1 MB of Markdown, 178 extracted figures. This one stress-tests the process at 5x the page count and surfaced the gaps folded into this document — see the caption-position note in step 4, the seam-defect table in 3c, and the audit-scaling notes in step 7.

The approach is: split the PDF, transcribe chapters in parallel with agents that read the rendered pages, extract figures mechanically, then audit each chapter against the PDF with a second, independent set of agents.

0. Why it is shaped this way

Two constraints drive the whole design.

Text extraction is not enough. pdftotext loses table structure, and pdfimages finds nothing at all in a FrameMaker-authored datasheet because the figures are vector line art, not embedded rasters. So transcription agents must read the rendered pages as images, and figures must be rendered-and-cropped rather than extracted.

An agent cannot hold a 45-page chapter. Context runs out silently — you get a completed task with no file written. Large chapters must be split into ~8-page chunks processed in parallel, which then introduces seam defects that have to be cleaned up afterwards. Budget for this.

1. Prerequisites

Everything lives in the flake devshell:

nix develop

Provides qpdf, poppler-utils (pdftoppm, pdftotext, pdfinfo, pdfimages), mupdf (mutool), imagemagick (magick), and python3.

If nix develop errors with "Path 'flake.nix' … is not tracked by Git", either git add flake.nix flake.lock, or use nix develop "path:$PWD" to bypass the git-tracking requirement.

2. Split the source PDF into chapters

Find the chapter boundaries from the PDF bookmarks or the printed contents page, then cut losslessly with qpdf:

pdfinfo REN_RAA489400_DST_20240827.pdf          # page count
pdftotext -q REN_RAA489400_DST_20240827.pdf - | grep -n "^[0-9]\+\. "

mkdir -p chapters
qpdf source.pdf --pages . 5-10  -- chapters/01-overview.pdf
qpdf source.pdf --pages . 11-13 -- chapters/02-pin-information.pdf
# ...

Name files NN-slug.pdf. Everything downstream keys off that stem: the transcription becomes NN-slug.md and figures go in NN-slug/.

Record the printed page range of each chapter — the number in the page footer, not the position in the file. You need it in step 5.

for f in chapters/*.pdf; do
  t=$(pdftotext -q "$f" - | grep -oE "Page [0-9]+" | sed "s/Page //")
  echo "$(basename "$f") : $(echo $t | tr ' ' '\n' | head -1)-$(echo $t | tr ' ' '\n' | tail -1)"
done

3. Transcribe each chapter with a parallel agent

One agent per chapter PDF, all launched in a single message so they run concurrently. Chapters over ~10 pages need chunking first — see step 3b.

The prompt template

Convert a datasheet chapter PDF into Markdown.

Source: /abs/path/chapters/NN-slug.pdf
Output: /abs/path/chapters/NN-slug.md

Steps:
1. Use the native `read` tool with filePath=<source>. The read tool returns
   PDFs as attachments so you can see the rendered pages directly. DO NOT use
   pdftotext, python, or any bash-based extraction - use the read tool.
2. Transcribe the full contents of every page into GitHub-flavored Markdown,
   then write it with the `write` tool to the output path.

Transcription rules:
- Transcribe all body text VERBATIM. Do NOT summarize, paraphrase, or omit.
- Preserve the heading hierarchy with #/##/###, keeping printed section numbers.
- Reproduce ALL tables as GFM tables. Every row and column, including units and
  conditions. Do not truncate. GFM has no rowspan: where the source merges cells
  vertically, repeat the value on each row.
- Preserve footnotes as a numbered list, keeping the markers in the cells.
- Transcribe printed errors, typos and inconsistencies EXACTLY as printed. Do
  not silently correct them. They are load-bearing for an audit.
- Equations: use LaTeX in $...$, keeping printed equation numbers.
- Use inline code for register names, pin names, bit fields and hex values.
- Add a `<!-- page N -->` comment at each page boundary, placed BEFORE that
  page's content, using the PRINTED page number. This chapter covers pages A-B.

FIGURES - IMPORTANT:
- For each figure, emit ONLY the printed caption, as bold text, exactly as it
  appears (e.g. `**Figure 12. Startup from VBUS**`), plus an image link of the
  form `![<printed caption>](NN-slug/figure-<printed number>.png)`.
- Do NOT write any description of what the figure shows. Do not describe axes,
  curve labels, block names, signal paths, colours or annotations. The images
  are extracted separately and the description is not in the datasheet.
- If a figure has no printed number or caption, say so plainly and do NOT
  invent one.

Reply with: output path, pages transcribed, number of tables, number of figures.

Why the figure rule matters

The first run of this project asked agents to describe each figure "for accessibility". Every such description was invented text indistinguishable from datasheet prose, and three of them were factually wrong:

a waveform described as settling high when it actually falls back to baseline (the exact behaviour the plot's measurement arrow was annotating);
a 32-pin QFN described as "8 pins per side" when it is 11/5/11/5;
an invented sentence about "RC filter components near the controller" for a drawing that contains no such components.

In one chapter the invented prose was ~90% of the file. Removing it later cost a full audit pass. Do not ask for figure descriptions.

3b. Chunking large chapters

An agent will silently fail on a big chapter — it reports success and writes nothing. Pre-split anything over ~10 pages:

qpdf chapters/07-registers.pdf --pages . 1-8   -- /tmp/reg/07-1-8.pdf
qpdf chapters/07-registers.pdf --pages . 9-16  -- /tmp/reg/07-9-16.pdf
# ...

Give each chunk agent the same prompt plus:

IMPORTANT: this is chunk N of M and will be concatenated. Do NOT add a document
title, intro, or closing summary. Start directly with the content of page 1 of
this chunk. Do not write commentary into the file. If the chunk starts or ends
mid-table or mid-register, transcribe the partial content; do not invent the
missing part. This chunk covers printed pages A-B.

Then concatenate in order:

cat /tmp/reg/07-part{1,2,3,4,5,6}.md > chapters/07-registers.md

3c. Clean up the chunk seams — always required

Parallel chunks drift. Every one of these occurred in this project:

Seam defect	Detection	Fix
Duplicate headings (source heading and the agent's own)	`rg "^#{3} " file.md` and look for adjacent pairs naming the same thing	Collapse to one; demote genuine sub-headings to `####`
Heading levels differ between chunks	compare `^#+` counts per chunk range	Normalize with a scripted pass — see below
A chunk emits valid heading text with no leading `#` at all	`rg '^[0-9]+\.[0-9]+(\.[0-9]+){0,4}\. [A-Z]' file.md \| rg -v '^#'` — matches printed-numbered lines that aren't markdown headings	Prepend the correct number of `#` (see the level formula below)
A table caption is written as a heading instead of bold text (`##### Table 3.7. ...` instead of `Table 3.7. ...`)	`rg -n '^#+ Table ' file.md`	`sed -E 's/^#+ (Table [0-9]+\.[0-9]+\. .+)$/\1/'`, then re-check table row/figure counts are unchanged
Derived metadata lines in 3 different formats (`\|` vs `·` vs `—`)	`rg -o "^Address:.*" \| sort \| uniq -c`	Pick one canonical form, rewrite
Some chunks bold table captions, others don't	`rg -c '^\\Table' ; rg -c '^Table '`	Bold them all
Address style `0x50` in one chunk, `50h` in another	`rg "0x[0-9A-F]" file.md`	Normalize to the printed style
RFC2119 keywords (Shall/Should/May/Shall Not/...) bold in some chunks, plain in others	count `\bShall\b` vs `\\Shall\\` per file	See "Fixing missing bold emphasis" below — this one has sharp edges

Normalizing heading levels from printed section numbers

Given a heading whose text starts with a printed number (9.2.5. Title), the correct level is (number of dots) + 1: 9.1 → ##, 9.2.5 → ###, 9.2.5.1 → ####. Chapter N. / Appendix X. headings are always #.

def desired_level(text):
    if re.match(r'^(Chapter \d+|Appendix [A-Z])\.', text):
        return 1
    m = re.match(r'^(\d+(?:\.\d+)*)\.?\s', text)
    return m and m.group(1).count('.') + 1

Sharp edge: if you rewrite a heading line with a regex substitution, make sure the replacement keeps the line's trailing newline. A pattern like ^(#{1,6})\s+(.*)$ followed by f"{new_hashes} {text}" silently drops the \n for every line that actually changes, which merges that heading into the following paragraph on the same line — invisible in a diff of line counts, very visible once you view the file. Capture the newline explicitly and put it back:

heading_re = re.compile(r'^(#{1,6})[ \t]+(.*?)([\r\n]*)$')
# ... hashes, text, eol = heading_re.match(line).groups()
# ... out.append(f"{new_hashes} {text}{eol}")

Run the script twice — a correct normalizer is idempotent (reports 0 changes on the second pass). If it isn't, something upstream (the level formula, the newline handling) is still wrong.

Fixing missing bold emphasis

Before trusting the text layer to tell you whether a word is bold, check a rendered crop. mutool draw -F stext's per-line <font> attribute is not reliable for detecting an inline bold sub-run in the middle of a line — a sentence like "...capacitance change Shall occur..." can report a single non-bold font for the whole line even when "Shall" is visibly bold on the page. Render the page and crop the relevant lines to confirm before writing a fix:

pdftoppm -png -r 200 -f <page> -l <page> -singlefile source.pdf /tmp/page
magick /tmp/page.png -crop <w>x<h>+<x>+<y> /tmp/crop.png   # then view it

If it's genuinely missing, a scripted fix needs two more guards or it corrupts the file:

Don't regex-exclude only the exact wrapped form. Excluding **Shall** but not **Shall Discard** means the bare Shall inside an already-bold multi-word phrase gets wrapped again — **Shall Discard** → ****Shall**** Discard**. Split each line on ** first and only touch the even-indexed (outside-any-bold-span) segments:
```
parts = line.split('**')
for i in range(0, len(parts), 2):   # even = outside bold, odd = inside
    parts[i] = pattern.sub(lambda m: f'**{m.group(0)}**', parts[i])
line = '**'.join(parts)
```
Watch for bold spans that cross a line break (**Shall at the end of one line, Not** at the start of the next — valid Markdown, renders fine). The even/odd split above is computed fresh per line, so a line that starts mid-bold-span (because the previous line left it open) gets the parity backwards. This is rare — find candidates first with awk '{n=gsub(/\*\*/,"**"); if (n%2!=0) print FNR}' file.md (an odd count on a line is expected only for these) — and handle them by hand rather than generalizing the script further for a handful of cases.
Don't apply it to keyword/glossary-definition tables. If a chapter defines what "Shall" means (a keywords/conventions table), the term is usually printed in plain text even though the same word is bold everywhere else as normative emphasis — check the specific table's rendering before assuming the chapter-wide convention applies inside it too.

After any scripted rewrite, prove you changed nothing real

rg '^\|' before.md > /tmp/a; rg '^\|' after.md > /tmp/b; diff /tmp/a /tmp/b

An empty diff means no table cell moved. Also compare counts of headings and **Figure captions before/after. This applies to every fix in this section — heading-level normalization, missing-bold fixes, and any other scripted pass.

4. Extract the figures

pdfimages will not work on vector art. Use the caption-anchored cropper:

python3 tools/extract-figures.py --dry-run chapters/*.pdf   # inspect first
python3 tools/extract-figures.py chapters/*.pdf             # write PNGs

Before running at scale, check the template's caption convention. Render one page that has a figure on it and look at where the caption sits relative to the drawing:

pdftoppm -png -r 150 -f <page> -l <page> -singlefile source.pdf /tmp/page

The tool defaults to the Renesas-style convention — caption below the figure, sequential integer numbers (Figure 12.). If the datasheet instead prints the caption above the figure (common in USB-IF/standards-body documents) and/or numbers figures per-chapter (Figure 4.1, 4.2, ...), pass:

python3 tools/extract-figures.py --caption-position above chapters/*.pdf

--caption-position above flips the band direction (bounded above by the caption, below by the next body text or caption) and the caption regex accepts Figure \d+(\.\d+)?\., writing figure-4.1.png instead of zero-padded integers. It also requires the caption line's font to be bold — without that check, a plain prose sentence like "Figure 4.2 illustrates the application of..." false-matches as a caption and produces a SKIP: no ink found in band entry (or worse, a bogus crop) once you get past the first one.

It locates Figure N. captions in the text layer via mutool draw -F stext, derives a band bounded by the caption and the nearest body-text line, renders the page, tightens the band to the actual ink, and crops at 300 dpi into chapters/<stem>/figure-NN.png (or figure-N.M.png) using the datasheet's own figure numbers.

Always --dry-run first and sanity-check the reported band sizes, then eyeball the output as a contact sheet:

magick montage $(find chapters -name 'figure-*.png' | sort -t- -k2 -n) \
  -tile 5x -geometry 300x300+4+4 -background gray70 /tmp/contact.png

Three tuning constants in the script are page-layout specific and are the first things to adjust for a different vendor's template:

HEADER_BOTTOM (62 pt) — must clear the running head and the horizontal rule beneath it, which otherwise gets caught as figure ink.
BODY_GAP (5 pt) — stext line boxes clip descenders, so without clearance the tail of the heading above bleeds into the crop.
PAD (6 pt) — clamped to the band so it cannot reach into the caption.

Also note captions wrap: bound the next figure with the caption's last line, not its first, or the wrapped remainder lands at the top of the following crop.

For a figure the datasheet never captioned, crop it explicitly rather than inventing a number — and record the invocation:

python3 tools/extract-figures.py \
  --region '1:195,550,410,640:current-sensing-trace-routing.png' \
  chapters/08-layout.pdf

Delegate this whole step to a subagent for big documents

For a document with 50+ figures, run the dry-run → adjust → contact-sheet → re-dry-run loop in a subagent rather than the main thread. Each contact sheet is a sizeable image, and getting the band-detection constants (or the caption-position convention) right typically takes several rounds — none of that iteration needs to live in the orchestrating conversation, only the outcome does.

Give the subagent:

the list of chapter PDFs and the expected figure count per chapter (count ^\*\*Figure in each transcribed .md — the subagent should treat a mismatch against this count as a bug to chase down, not a fact to report);
explicit instructions to render one figure page first and report which caption convention applies before running the tool at scale;
permission to extend tools/extract-figures.py if the template doesn't fit (new flag, not a fork), with an explicit instruction to re-run the tool against the existing worked-example chapter(s) afterwards to confirm the change didn't break the default convention;
a requirement to flag anything that looks visually wrong in a contact sheet rather than silently accept it, and to use --region (recording the exact invocation) for any figure the automatic detection can't handle.

Have it report back: final figure counts per chapter (matching the expected counts above), which figures (if any) needed --region, any tool changes made and the backward-compatibility check result, and one or two contact sheets for you to spot-check before moving on to the TOC/audit phases.

5. Normalize the page markers

Chunk agents tend to number pages from 1 locally instead of using the printed number. Check, then offset each file by the difference:

for f in chapters/*.md; do
  echo "$f $(rg -o '<!-- page ([0-9]+) -->' -r '$1' "$f" | head -1)-$(rg -o '<!-- page ([0-9]+) -->' -r '$1' "$f" | tail -1)"
done

# e.g. chapter 1 printed pages start at 5 but markers start at 1 -> add 4
perl -i -pe "s/<!-- page (\d+) -->/'<!-- page '.(\$1+4).' -->'/ge" chapters/01-overview.md

Markers must sit before the content of the page they name. Anything that later slices the file by page depends on this — see the pitfall in step 7.

6. Build the table of contents

Generate anchors mechanically; do not hand-write them. GitHub's slug rule is: lowercase, strip punctuation except hyphens/underscores, spaces to hyphens.

def anchor(h):
    s = h.strip().lower().replace("`", "")
    s = re.sub(r"[^\w\s-]", "", s, flags=re.UNICODE)
    return re.sub(r"\s+", "-", s.strip())

A useful TOC has: a chapter table (file, printed page range, source PDF link, figure count), the numbered section tree, a figure index, and a section listing known defects in the source datasheet so nobody later "fixes" a faithful transcription. Cap the depth — a 63-register chapter should link to its section, not enumerate every register.

Verify:

python3 tools/check-links.py chapters

This checks every relative link, every anchor against real headings, and flags orphaned PNGs. Exit code is non-zero on failure, so it can gate a commit.

7. Audit: transcription vs PDF

Do not skip this, and use fresh agents. The agent that wrote a file is not a reliable auditor of it.

One agent per chapter, launched in parallel, large chapters split the same way as in step 3b — but the chunk size doesn't have to match. Transcription chunks are sized for writing a large amount of new content (~8 pages); audit chunks only need to read and compare, so they tolerate roughly 2x the page count before running into the same silent-context-loss failure mode. On a 400+ page document, auditing in ~15-page chunks instead of ~8-page chunks roughly halves the number of audit agents needed with no loss of quality.

You are auditing a Markdown transcription of a datasheet chapter against the
original PDF. This is a READ-ONLY audit. Do NOT edit, write, or create files.

PDF (ground truth): <path>.pdf
Markdown to audit:  <path>.md

Method:
1. Use the native `read` tool on the PDF. It returns rendered pages as an
   attachment. DO NOT use pdftotext/python/bash extraction - compare against
   what is actually printed.
2. Use the `read` tool on the .md.
3. Compare page by page.

Check:
- Missing or extra sections; wrong heading text or numbers.
- Body text summarized or paraphrased rather than transcribed verbatim.
- Every table: all rows and columns present, every value correct digit by
  digit including sign, decimal point and unit.
- Figure captions: correct number, exact caption text.
- Any hallucinated content in the MD that is not in the PDF. THIS IS THE MOST
  IMPORTANT THING TO CATCH.

For each finding give: severity (CRITICAL = wrong/invented, MAJOR = missing,
MINOR = formatting), printed page number, what the PDF says, what the MD says.
If a category is clean, say so in one line. End with a one-line verdict:
ACCURATE / MINOR ISSUES / SIGNIFICANT ISSUES. Do not fix anything.

Add per-chapter emphasis: numeric accuracy for a specifications chapter, every pin row for a pinout chapter, bit-level detail for a register chapter, field order and ACK/NACK bit values for protocol frame diagrams.

Feed forward any suspected source defects and ask the auditor to confirm what the PDF actually prints. This is how you separate a transcription bug from a vendor typo. In this project every such flag turned out to be a genuine defect in the datasheet: a threshold printed in mV with volt-magnitude values, a pin listed as both NC and GPIO2, TYPE_C_PRAMETER vs TYPE_C_PARAMETER between a heading and its own table caption.

Prefer letting the audit agent navigate by marker, not pre-slicing

The simplest way to avoid the slicing pitfall below is to not slice at all: point the audit agent at the full chapter .md and tell it which  markers bound its assigned range ("only inspect the portion between  and "). The agent can read the whole file and navigate to the right markers itself. This trades a larger read for eliminating an entire class of off-by-one bugs in a slicing script, and for the sizes involved (a chapter .md is at most a few thousand lines) the read is cheap.

Reach for scripted pre-slicing only when a downstream tool (not an agent) needs an exact byte range — and if you do, remember markers precede their page:

start = marks[a]        # NOT marks[a-1] + 1
end   = marks[b + 1]

Getting this wrong shifts every slice one page and produces six confident, identical, false "MAJOR: last page is missing" reports. Before believing any missing-content finding, grep the real file:

rg -o "Table ([0-9]+)\." -r '$1' chapters/07-registers.md | sort -nu | tr '\n' ' '
rg -o "^### 7\.2\.([0-9]+) " -r '$1' chapters/07-registers.md | sort -n | tr '\n' ' '

Contiguous sequences mean nothing is missing.

8. Act on the findings

Triage before editing. In this project, all datasheet data — every table, value, register and pin — was accurate; every real defect was in prose the agents added around the figures. Expect the same shape.

Distinguish:

Invented content → delete.
Derived content (e.g. an address/reset summary computed from the register map) → keep if useful, but normalize it and add a note saying it is derived, not printed. Do not let it masquerade as transcription.
Faithful transcription of a source defect → leave alone, record in the TOC's known-issues section.

When stripping generated text, anchor the pattern tightly. Figure descriptions were italic paragraphs directly following an image line; that shape can be matched without touching legitimate printed *Note:* paragraphs elsewhere. Verify afterwards that the printed ones survived:

echo "before: $(rg -c '^\*Note' backup.md)  after: $(rg -c '^\*Note' file.md)"

Checklist

[ ] Split PDF into chapters/NN-slug.pdf; record printed page ranges
[ ] Transcribe in parallel, one agent per chapter (NO figure descriptions)
[ ] Chunk chapters over ~10 pages; concatenate; clean the seams
[ ] If normalizing heading levels with a script, verify it's idempotent
    (0 changes on a second run) and preserves trailing newlines
[ ] Verify table-row diff is empty after every scripted rewrite
[ ] Check the figure caption convention (above/below the figure) on one
    rendered page before extracting at scale; consider a subagent for the
    extract/tune/contact-sheet loop on documents with 50+ figures
[ ] Extract figures; --dry-run, then contact-sheet review
[ ] Normalize <!-- page N --> to printed numbers
[ ] Build TOC with generated anchors; record known source defects
[ ] tools/check-links.py passes
[ ] Audit every chapter with fresh agents (chunks can be ~2x transcription
    chunk size, since audits only read+compare)
[ ] Verify "missing content" findings against the real file before acting
[ ] Delete invented content; normalize derived content; keep source defects

Tools in this repo

Tool	Purpose
`tools/extract-figures.py`	Caption-anchored figure cropping from vector PDFs. `--caption-position {below,above}` (default `below`) and decimal figure numbers (`4.1`) support both the Renesas-style and USB-IF-style caption conventions
`tools/check-links.py`	Link, anchor and orphan-image validation. Anchor generation replicates GitHub's `-1`/`-2` disambiguation for duplicate headings (e.g. two companies with near-identical names in a contributors list)
`chapters/TOC.md`	Worked example of the generated TOC

25 KiB Raw Blame History