I Pointed Karpathy's LLM Wiki at the D&D Rulebooks

My D&D side project is the sandbox where I test RAG ideas — chunkers, retrievers, embeddings, evaluators, all of it pointed at the rulebooks until something useful falls out. The long game is a fully AI-driven Dungeon Master. The post you're about to read is one piece of that arc.

Meet Macho. Chaotic Neutral, Human Barbarian, Path of the Berserker. He's the test rig — a character generated end-to-end by the system and used to stress every part of the pipeline that has to actually read the rulebooks correctly. Level 1 on the left, Level 15 on the right. If the wiki gets grapple wrong, Macho can't grapple right. That's the loop.

Earlier this month, Karpathy tweeted about something he was calling an "LLM Knowledge Base." Two days later, after the tweet went viral, he followed up with an idea file on GitHub — not code, not an app, just a markdown document describing the pattern in enough detail that you could hand it to your agent and have something working by lunch.

The pitch, stripped to its load-bearing parts:

• You have a corpus.
• An LLM reads it once and writes a wiki page per concept, with citations.
• Users (and other LLMs) read the wiki, not the corpus.
• The wiki is the artifact. It's in git. You can cat it. You can grep it. A human can audit any page in 30 seconds.

The argument underneath: most RAG systems are doing synthesis at query time when they should be doing it at ingest time. Every time you ask "what changed about grappling between editions?", a typical RAG pipeline retrieves chunks, throws them at a model, and re-derives the same answer it derived for the last person who asked. The synthesis is the expensive part, and you're paying for it on every query.

I read this, nodded, closed the tab, and then could not stop thinking about it. So I built the thing. The corpus I picked: the 2014 and 2024 D&D Player's Handbooks. Two editions, ten years apart, same chapter taxonomy, thousands of small mechanical diffs that DMs have been arguing about on Reddit since the 2024 edition shipped. If the wiki pattern works anywhere, it should work here.

It worked. Here's what happened, what it cost, what I'd do differently, and where I think this pattern eats RAG's lunch — and where it absolutely doesn't.

The premise, in one paragraph

D&D 5e (2014) and D&D 5e (2024) are technically the same edition. Wizards calls the 2024 books "fully compatible." They are, in the sense that you can play with both at the table without your character sheet bursting into flames. They are emphatically not compatible in the sense that hundreds of small things changed: how Heroic Inspiration works, what counts as a Bonus Action for spellcasting, how grapple resolves (it flipped from a contested Athletics check to a saving throw on initiation), what Backgrounds give you, the entire feat structure, the Ranger, especially the Ranger. The diff is the value. A regular wiki could capture this if you had a small army of volunteers willing to cross-reference both books page by page. I had an LLM and a weekend.

What I actually built

The whole thing is one Python module — llm_wiki.py — with three operations: ingest, ask, lint. That's it. Karpathy's framing maps cleanly onto code:

The schema file is the load-bearing piece and the part that surprised me most. It's not the model that makes the pages reliable. It's the prompt telling the model exactly which sections to write, which frontmatter fields are required, that every claim must cite a chapter and page from retrieved chunks, and — critically — that if the 2014 book has no equivalent rule, you write differs: true with an empty books_2014 and say "No 2014 equivalent located" rather than confabulating one. (And if you genuinely couldn't tell either way, differs: unknown — the absence of a verdict is also recorded as a verdict.) I called this the Prime Directive in the prompt and it earned its caps. RAG fails silently. The wiki fails loudly, in writing, on the page, with a disclaimer.

Here's what a page actually looks like — this is the top of wiki/rules/grapple.md:

---
title: Grapple
type: rule
differs: true
change_summary: "Initiating a grapple is now a saving throw vs a static
  DC; 2014 used contested Athletics."
---

## What Changed
- 2024: target makes **Str or Dex save** vs DC `8 + Str + PB`; escape
  is a check vs the same DC
- 2014: contested **Strength (Athletics)** check both directions

## Sources
- 2024 PHB · "Rules Glossary" → Unarmed Strike → Grapple
- 2014 PHB · p. 196 ("Grappling" / "Escaping a Grapple")

Frontmatter, diff, citation. Reads in five seconds; settles the table argument in six. Every page in the wiki has this shape. (The actual frontmatter has more fields — slug, tags, books_2014, books_2024, last_updated — abbreviated above for the example.) The frontmatter is queryable (grep -l "differs: true" wiki/**/*.md is your "what changed" report). The body is human-readable. The sources are auditable. The whole thing is just files.

That's the source. Here's how it renders in the actual wiki UI — same content, with the CHANGED badge and the diff panel doing visual work the markdown can't:

Stack underneath:

• Extraction: pypdf plus a custom pdf_clean.py to strip chrome lines, rejoin hyphenated words across line breaks, collapse blank-line runs. PDFs are a war crime against text, and the 2014 PHB in particular has a layout that hates you.
• Indexing: text-embedding-3-large into FAISS, plus a rank_bm25 keyword index, fused with LangChain's EnsembleRetriever at 0.7 BM25 / 0.3 vector. Hybrid retrieval is just better. I stopped fighting this two years ago.
• Models: gpt-5.5 for ingest (precision matters, we're writing the artifact), gpt-5.4-mini for the streaming Q&A panel (latency matters, the user is waiting).
• Backend: FastAPI, SSE for streaming.
• Frontend: React + Vite + Tailwind, two routes, search/tag/sort/sibling-nav, the usual.
• Storage: the wiki is markdown files. Committed to git. That's the whole database. You can grep it.

The last point is the one I want to underline. The wiki is the artifact. Not a vector store, not a fine-tune, not a knowledge graph. Files. In a folder. With frontmatter. Whatever model writes them next year will read the same files. Whatever editor I use to fix a hallucination edits the same files. The output of the system is a thing a human can read.

What it cost

I'll show you the receipts because that's the only part of any AI build post worth trusting.

Corpus sizes:

The wiki is roughly 40% the size of the 2024 source and bigger than the 2014 source. That's the synthesis tax — every page has to carry its own context, its own citations, its own diff block. You are deliberately denormalizing for read-time efficiency.

Ingest economics, per page:

• Input: system prompt (~1.7K) + 20 retrieved 2024 chunks (~3K) + 20 retrieved 2014 pages (~18K — pages, not chunks, because the 2014 source pages are short and self-contained) + the existing index + the page being rewritten. Total: ~25K input tokens.
• Output: page (~3.5K) + index rewrite (~1K) + log entry (~0.5K). Total: ~5K output tokens.
• Cost per ingest: ~$0.15–$0.45 depending on which model and how much reasoning it burned.

Total run:

Money: ~$25–$80 in OpenAI spend, plus another $5–$15 in debugging waste. Embeddings were a rounding error (~$0.50). Call it $50 all-in.

Time: ~7 hours of unattended LLM time for the productive stretch. Curated batch averaged 95 seconds per page, subprocess-isolated with a 5-minute wall-clock kill switch because gpt-5.5 will occasionally just sit and think about its life choices and you have to be willing to kill it.

The final wiki: 277 pages, 99% non-stub, 64% top-tier by a grader I wrote that checks for differs: flag set, citation count, and presence of an actual diff block when differs: true. The two remaining stubs are content-correct edge cases — weapon-mastery is a 2024-only mechanic and skilled-feat is a 2024 Origin Feat with no clean 2014 equivalent. The grader is too strict, not the pages.

What I'd do differently

Three things I burned time on that I shouldn't have:

Default k to 20, not 8. The first batch of smoke tests came back with thin, hedge-y pages and I spent an afternoon convinced the model was the problem. It wasn't. Retrieval was returning eight chunks for things that needed twenty, the model was correctly noting that it didn't have enough source material to make a confident claim, and I was reading "the model is being lazy" into "the retriever is starving it." Bumping k to 20 fixed almost everything. If I were starting over, I'd start there and tune down only if costs hurt.

Build the kill switch on day one, not day three. Bard. Of all the classes, bard hung the API. Twice. Twenty-five minutes each time, the spinner spinning, my coffee getting cold. LangChain's timeout=180 parameter turns out to be, charitably, a suggestion the underlying httpx call can ignore. After the second hang I refactored the batch loop to spawn each ingest as a subprocess with a hard 5-minute wall-clock kill. SIGKILL is not a suggestion. Bard finished in well under a minute when I got it back its turn. The whole patch is fifteen lines of code and would have saved me an evening.

Write the grader before the batch run, not after. I ran the curated batch of 156 pages, then wrote the stub-detection grader, then discovered I needed to re-ingest 15 of them. If the grader had existed first, I would have caught the under-retrieval problem at page 10 instead of page 156, and the stub re-ingest would have been part of the batch instead of a separate phase. This is the dumbest one. Evals first. Always evals first. I know this. I have known this for years. I still didn't do it.

Why D&D was the right pilot, and where this idea generalizes

I keep seeing people try the LLM-wiki pattern on their company's Notion and bouncing off, and I think it's because Notion is the worst possible corpus to start with. D&D worked because it has five properties Notion doesn't:

• Two parallel corpora that beg to be diffed. Same game, two editions, ten years apart, same chapter taxonomy. The diff is the product. Most "build a wiki over our docs" use cases don't have a natural axis of comparison.
• A finite, knowable question space. Every rule. Every class. Every spell. Every monster. Bounded. Enumerable in advance. You can actually pre-compute the answers because you can pre-compute the questions.
• Many small mechanical diffs no human will track. This is the work nobody does because it's tedious and low-status. LLMs eat tedious and low-status for breakfast. 178 of these in seven hours.
• Ground truth exists. I could grade page quality because the source books are authoritative and I could spot-check. Most enterprise corpora don't have this. Your Notion has three contradictory PRDs from 2022 and someone's vacation notes.
• High user value per page. "Did Heroic Inspiration change?" is a real question DMs ask at the table and Reddit threads die over. Each page does work.

Generalize this and the wiki pattern shines anywhere you have:

• A stable, bounded, valuable corpus (legal codes, regulatory filings, API docs, game rules, product specs at version boundaries, tax law, medical guidelines).
• Recurring questions you can enumerate (what does this clause mean, how did this regulation change, what's the diff between v1 and v2).
• Cross-document synthesis that's expensive and worth caching (anything where the answer requires reading multiple sources and noticing something).
• Provenance pressure — you actually care that every claim has a citation, and "the LLM said so" isn't an acceptable answer.

The amortization is the whole argument, so it's worth showing the math. Conservative napkin numbers — $50 one-time build, $0.05 per RAG synthesis at query time, ~$0.002 per cached wiki read once the page exists:

Crossover sits somewhere between 500 and 1,000 questions. Below that, plain RAG is cheaper. Above it, wiki wins outright and the gap widens roughly linearly. (At my actual table — 10ish questions per session, two sessions a month — the crossover is "next year, probably." The build cost was small enough I didn't care.)

Where regular RAG still wins

I am not here to tell you to throw away your retriever. The wiki pattern is not a RAG-killer; it's a layer on top of RAG that pre-computes the predictable 80% so retrieval can do the unpredictable 20%. Where plain RAG still beats this:

• Unbounded query space. "What about X?" questions you can't enumerate. Open-ended research. Exploratory analytics over logs. If you can't list the questions in advance, you can't pre-compute the answers.
• Fast-changing corpora. If your docs update daily, re-ingesting daily is going to hurt. The wiki amortizes — it gets cheaper per query the more queries you run — but only if the underlying corpus is stable enough to amortize against.
• Multi-attribute queries the wiki author didn't anticipate. "What spells require Concentration AND have a Bonus Action casting time AND are on the Cleric list?" A static page structure doesn't pre-compute that. RAG does, kind of, badly, but does.
• Low query volume. If you're going to ask five questions, plain RAG is cheaper. The wiki is a fixed-cost ingest amortized over query volume. Below some break-even, don't bother.

The split rule I'd write on a whiteboard: wiki for the predictable 80% of questions about a stable corpus, RAG for the unpredictable long tail. That's why my system has both — the /api/wiki/ask/stream endpoint reads the wiki first and only falls back to raw retrieval when the wiki is silent. The wiki is the cache. RAG is the cache miss path.

The thing that surprised me

I expected the cost analysis to be the punchline. It's not. The punchline is that I trust the wiki pages more than I'd trust a RAG response to the same question, and the reason is the schema file.

When a RAG pipeline fails — when retrieval misses, or the chunks contradict each other, or the answer just isn't in the corpus — the model usually plows ahead and produces something plausible. That's its training. It's polite. It's helpful. It's wrong.

When the wiki ingest fails the same way, the schema forces it to write differs: unknown and "No 2014 equivalent located" into the page. The failure is now a permanent, auditable artifact. I can grep for it. I can grade it. I can hand a single page to a human and say "is this right?" and they can answer in 30 seconds because the citations are right there.

That's not a retrieval improvement. That's a forcing function on honesty, encoded in a prompt, and the model can't wriggle out of it because the output format demands it. RAG outputs are conversations, which are easy to bullshit through. Wiki pages are documents, which are harder.

I think that's the real Karpathy insight, dressed up as a wiki. The format you make the LLM write into is doing more work than the model is.

What's next

The obvious move is running lint across the existing 277 pages. Right now I trust each page in isolation because the schema makes it audit itself, but I haven't asked the corpus to audit itself — to find contradictions between pages, orphan references that point at slugs I never created, or claims about the same rule that drift between the page where it lives and the page where it's mentioned. That's where I expect to find the real bugs, and it's exactly the kind of work the wiki pattern is supposed to be good at.

After that, a proper eval set so I can swap models and actually measure whether gpt-5.6 is better than gpt-5.5 for this specific job, instead of guessing from vibes. Right now my "is the page good" signal is a grader I wrote in an afternoon and my own eyeballs. Both are biased. Neither scales.

The bigger thing I keep poking at: the schema file is doing more work than the model. That suggests the next interesting experiment isn't a better model or a better retriever — it's a better schema. What does the prompt look like for a wiki over SEC filings? Over a codebase? Over a deposition transcript? The mechanical bits transfer; the schema is the part that has to be designed for the corpus. If anyone wants to compare notes on what their schema looks like, I'm interested.

If you want to try this on your own corpus, the gating question is: can I list the questions my users will ask in advance? If yes, build the wiki. If no, you want RAG. If "kind of," build both and let the wiki be the cache.

Karpathy was right. I have receipts. The receipts cost $50.

References

• Andrej Karpathy — original tweet on the LLM Knowledge Base pattern
• Andrej Karpathy — LLM Wiki idea file (GitHub Gist)