// BEFORE THE PIPELINE
If the source text is wrong, the persona is wrong, although everything downstream is correct. This page records what we learned when we prepared the corpus for the ensemble's eighth mind: 21 sources and 664,758 words of Walter Russell's books and Home Study Course. 18 of the sources arrived as damaged OCR. We verified each number on this page against the running system on July 27, 2026.
The stakes
Grounding is the central function of Revive. Answers come from the actual pages, and they are not invented. This works in both directions. What is on those pages is what the persona believes, says, and reads aloud.
A costume persona invents. It makes quotes that were never written, and it holds positions the person never held. The Architecture page shows how grounding prevents this. But grounding has its own failure mode, and it is quieter.
Give a persona the OCR of a 1950s typewritten course, and it will retrieve "slectric" for electric, "Ming" for Mind, and "cesire" for desire. The voice pipeline will pronounce them. The chat bot will quote them. Retrieval also degrades. A misspelled word embeds to the wrong place in vector space, so damaged passages stop being findable. The corpus is not raw material for the persona. The corpus is the persona.
The Architecture page describes the cosine migration: a retrieval bug that caused no error and made the personas generic. Corpus damage behaves the same way. Nothing fails, and no log complains. The persona sometimes says garbled words, or it cannot find a passage that it owns. You will not know why until you read the source text closely. The procedures below exist because we did.
Lesson one
The cheapest OCR error to repair is in a file you never use. Before you spend hours on corrections, spend two minutes to find a version that was never OCR'd.
A PDF records its own history. A document produced from Word or a typesetting tool is born-digital, and its text is exact. A document whose creator field says "Scan Plug-in" or "Paper Capture," or whose pages are each one full-page image, is a scan with an OCR text layer. That layer is guesswork.
two minutes of forensics per source
$ pdfinfo source.pdf | grep -i 'creator\|producer'
Creator: Microsoft Word # born-digital: text is exact
Creator: Adobe Acrobat Scan Plug-in # scan: text layer is OCR guesswork
$ pdfimages -list source.pdf | head
# one full-page image per page = a scan, whatever the text layer claimsThis check paid off twice in the Russell corpus. Two of his books circulated mainly as bad scans. But the archive.org listings also held volunteer retypes, visible by their small "Text PDF" and .doc files. These retypes are born-digital, word-perfect transcriptions. We used them, and a parsing script replaced hours of correction per book. The junk rate for The Universal One fell from 1.97% to 0.35% in one step, with no risk that a corrector "fixes" a word the author wrote.
The order is: born-digital text, then a retype you can verify, then your own OCR from page images, then another person's OCR text layer. Each corpus intake should start at the top of this list and stop at the first item that exists. For 18 sources, we started at the bottom, because the text files were available and looked usable. That choice cost a week.
Lesson two
"This text looks clean" is not a measurement. A junk rate is. Ours is simple: the fraction of words a spellchecker rejects, minus a whitelist of the author's own vocabulary.
the junk gate, in one line of thinking
junk_rate = aspell_nonwords(text) − author_whitelist
──────────────────────────────────────────
total_words
# Russell's whitelist: archaic verbs (knoweth, dost), his coinages
# (regiving, decentration, inbreathing), his invented elements
# (alphanon, tomion, urium). ~50 stems and words, built once,
# reused for every source. Anchor the regexes: an unanchored
# pattern silently excused garbled words that merely CONTAINED
# a coinage, and we shipped that bug for two sources.The metric made each decision clear. It ranked the 21 sources by damage, so triage was simple. It showed whether each repair pass worked. The first attempt on one source moved the rate from 3.5% to only 3.0%, which showed that the source needed a different method. It also defined completion. Each source in the corpus now sits between 0.20% and 1.02%. The remainder is mostly proper nouns and rare but real words that the checker does not know.
Non-words: "slectric," "whieh," "Greator," and digit hybrids such as "8ravitation." This is most OCR damage by volume, and the metric catches it at almost no cost.
Real-word errors: "lime" for time, "Sate" for state, "bat" for but, "cut" for out. Also structural damage, where each word is valid but the lines are in the wrong order. Both need a careful read by a person or a model. If you trust the metric alone, you ship the errors it cannot see.
This is the corpus-side twin of groundedness scoring. A persona without a groundedness score drifts, and no one notices. A corpus without a damage score does the same thing one layer down. Both metrics are cheap, and both are imperfect. Both exist so that decay must announce itself.
Lesson three
We repaired eighteen sources one at a time, and each one taught the next. For the final sources, the repair agents started with a list of known error families. Their first pass caught what the early sources needed three passes to find.
| Family | Examples from this corpus | How it is caught |
|---|---|---|
| Letter confusion | r↔c, c↔e, b↔h: "slectric," "cach," "whieh," "foree," "Greator" | Spellcheck flags it; fix in context, never globally |
| Digit homoglyphs | 1→i, 8→g, 0→O: "1s" for is, "8ravitation," "i46" in figure numbers | Any token mixing letters and digits, unless it is real notation |
| Real-word traps | "lime" for time, "Sate" for state, "bat" for but, "aver" for over, "toe" for to | Invisible to spellcheck; needs a dedicated read with a running trap list |
| Word splits | Line-break hyphens leaving both halves: "KNOWL-EDGE" surviving as "knowledges" AND "EDGE" | Merge across breaks, then re-scan: the second fragment often stays hidden |
| Page furniture | Running headers bleeding into prose, "Page 827," decorative "eee" line debris | Pattern-match and delete whole lines; never try to "fix" them into words |
| Figure garble | Chart labels shredded into pseudo-words no one can restore | Bulk-remove and log; guessing chart text manufactures fake quotes |
| The author's own spelling | Russell consistently wrote "repetative" and "etherial" | Not an error. A corrector that fixes the author is corrupting the corpus |
We never use a global find-and-replace. Each correction applies to a unique surrounding string in one record. Each correction goes to a fix log, and the untouched original stays beside the clean file. When a validation pass questions a fix, the log shows what changed, and the backup shows the previous text. We tried a generic auto-corrector once. It made words out of debris, and we disabled it.
Lesson four
One source did not come clean. Its junk rate almost did not move under the treatment that repaired seventeen others. The reason is the most transferable lesson on this page.
Russell's 1930 engineering monograph, The Genero-Radiative Concept, arrived as an archive.org text layer with 3.5% junk. Token-level repair got it to 3.0% and stopped. The remainder showed why. The damage was not misspelled words. It was broken sentences. The original OCR had read a two-column layout straight across. It mixed column fragments and figure captions into the prose, line by line. Each word was correct. The order was destroyed, and a spelling fix cannot repair word order.
the same passage, before and after re-OCR
BEFORE (inherited text layer):
"nstra of a system would not fall into ae primaries if their
motion were stayed by e gi and, : tke stele which illustrates
this text was performed..."
AFTER (tesseract, layout-aware, from the page images):
"...that the planets and satellites of a system would not fall
into their primaries if their motion were stayed by some giant
hand. The experiment which illustrates this text was performed
for the purpose of proving..."The repair took one afternoon. We extracted the 64 page images from the scan PDF. We ran tesseract with automatic layout analysis. We rebuilt the source from the new OCR, and we ran the normal token-level pass on top. The final junk rate was 0.53%. About 150 words were lost to physical scan damage, confirmed missing from both OCRs. Everything else was recovered in full.
A junk rate that continues to fall under repair means spelling damage. Continue the repair. A junk rate that stops high means layout damage. The only cure is to return to the page images with a better OCR pass. Text patches cannot repair a page the OCR read in the wrong order.
The strangest find in the corpus was page 50 of A New Concept of the Universe. The master scan holds this page upside-down, and each circulating copy descends from that scan. Thus the page, and its two chapter openings, are unreadable in each version you can download. The page-image rule recovered it. We rendered the page, turned it 180 degrees, and ran OCR again. Two chapters that were lost to each digital reader of that book are now in the corpus, and the persona can quote them.
Lesson five
Most of the repair ran as delegated agent work, with one agent per source. Each agent reported success. Several were wrong in ways their own reports could not show. Thus an independent gate stood between each "done" and the index.
The gate computes the junk rate before and after, from the untouched backup, and never from the fixer's own numbers. It caps the word-count change at 1.5%, so a "fix" that deletes prose is caught. It samples random fixes from the log and shows them in context for review. It also runs full coverage checks on the chunking and headers that follow.
It caught fixes whose replacement spans covered an adjacent error and left it uncorrected. It caught a whitelist regex that excused garbled words. It caught a hyphenation artifact that was approved as a "fix." The retrieval spot-check asked the live API questions whose answers were in repaired passages. This found remaining damage that the junk metric could not see, in a source that had already passed.
Each problem an agent found became a line in the next agent's brief: the anchored-regex bug, the covered-fix pattern, the real-word trap list, and the rule about author spellings. Early sources took three passes. The last sources passed the gate on the first try. The agents did not become smarter. The brief did. If you repair a corpus source by source, the order is an asset. Repair a small, damaged source first, and give its lessons to the second source.
The receipts
Junk rate is the spellcheck-minus-whitelist measure described above. The validation gate computes it against each source's preserved original. The three sources that arrived clean are not listed.
| Source | Before | After | Method |
|---|---|---|---|
| The Universal One | 1.97% | 0.35% | Born-digital retype found |
| The Secret of Light | 1.52% | 0.31% | Born-digital retype found |
| The Man Who Tapped the Secrets of the Universe | 0.47% | 0.36% | Adjudicated in place |
| A New Concept of the Universe | 2.45% | 0.28% | Adjudicated + upside-down page recovered |
| The Optic Dynamo-Generator | 0.73% | 0.23% | Adjudicated in place |
| The Genero-Radiative Concept | 3.51% | 0.53% | Rebuilt from layout-aware re-OCR |
| Home Study Course, units 1–12 | 1.09–4.46% | 0.20–1.02% | Adjudicated, one agent per unit, gated |
And after intake
Three smaller lessons come from the same build, on the query side of the pipeline. Each one shows how a grounded answer can still fail.
This corpus adds a one-sentence situating header to each chunk before embedding. This helps retrieval. But the header is for the search engine, not for the listener. Before the voice pipeline learned to remove it, the persona could open an answer with "Context: Home Study Course, Unit 7." Remove all added scaffolding before the persona speaks.
A text bot can end with a source list. A voice persona that says "as I recorded in the 1946 interview" stops being a man who remembers, and it becomes a scholar who cites. The persona prompt now states the rule. Build each answer from the retrieved passages, and never name a book, lesson, or year aloud. Grounding should support the answer and stay invisible.
The voice pipeline's generation model reasons before it answers. On a small token budget, it sometimes spent the full budget on reasoning and sent nothing. The persona then went silent in conversation. The repair was to cap the reasoning effort, and to retry once on empty output. If a reasoning model sits in your response path, an empty answer is a failure mode you must handle.
A fourth query-side lesson has its own section. Broad survey questions, such as "what are the top ten things I should know?", defeat similarity search in any corpus. The answer is a property of the full collection, not of a passage in it. The repair is a curated study-guide digest layer, tested on this persona. The Architecture page documents it.
This is Step 1 of Build It, as this project taught it. Ten minutes per source here saves days later.
before any source enters the index
1. PROVENANCE pdfinfo + pdfimages: born-digital, retype, or scan?
Walk the quality ladder; take the highest rung that exists.
2. MEASURE Junk rate with an author whitelist. Record it.
It is the baseline every later claim is checked against.
3. REPAIR Anchored, logged, reversible fixes. Hunt error families.
A dedicated pass for real-word traps the metric cannot see.
If the rate stalls high: back to the page images, re-OCR.
4. GATE Independent re-measure from the untouched backup.
Word-count delta bounded. Sampled fixes read in context.
5. SPOT-CHECK After indexing, query the live API for passages you
repaired. The metric passing is not the persona working.Grounding moves trust from the model to the text. Intake is where you learn whether the text deserves it.