1The problem
The Virginia Construction Code is hundreds of pages of sections that point at other sections. A rule in one place often depends on a table three chapters away. A chatbot that answers from loose text chunks will confidently cite sections that don't say what it claims, or don't exist. For this kind of question, a wrong citation is worse than no answer.
2What I built
All of it: the pipeline that turns the code PDF into a graph, the agent team, the retrieval layer, the eval suite and CI gates, and the backend the web app talks to.
From a question to a cited answer
Pick a route
Greeting, follow-up, direct lookup ("show me 1607.1") or real research. A cached answer returns at once.
Split the question
Two to four research steps, each a smaller question.
Search the graph
A named section ranks first; otherwise semantic search over passages, tables and diagrams. Web search only if a relevance check fails.
Write the answer
From the passages found, streamed to the browser as it writes.
Check every citation
Each section and table number is looked up in the graph. Real ones become links to their text; the rest stay plain text.
3Decisions that mattered
Keep the code's structure: a graph, not flat chunks
I parsed the whole code into a Neo4j graph that keeps what contains what, what comes next, and what cites what. Tables, equations and diagrams are their own nodes; passages, tables and diagrams carry embeddings with the section heading built in.
Why: the graph lets a named section be found directly, lets the citation check be a lookup instead of a judgment, and lets a reader open the exact text an answer relies on. The rebuild script is idempotent and verifies itself against the source before reporting success.
| Node | Count | Node or link | Count |
|---|---|---|---|
| Sections | 317 | Tables | 118 |
| Subsections | 1,953 | Equations | 69 |
| Passages | 3,405 | Diagrams | 49 |
| Embedded for search | 3,572 | Cross-references between sections | 242 |
Let the model write, let code check
The writer is a model; the citation check is not. After the answer is written, code pulls out every section and table number and looks each one up. Why: asking a second model "is this citation real?" just moves the problem. A lookup can't be talked into a yes.
Say when the answer came from the web
When nothing in the graph passes a quick relevance check, research falls back to web search, and the answer is labelled that way instead of dressed up as code text. Answers are cached by question for 30 days, versioned so a pipeline change never serves a stale answer.
4How I measured it
The first version often ranked the wrong section first, so I built a golden set of 51 questions with answers taken from the code text and every expected section confirmed to exist in the graph. Two of the 51 ask about things the code doesn't cover, to test refusals; the retrieval score uses the other 49. I then measured seven retriever variants on those 49, changing one thing at a time.
What it found: one embedding setting (embedding the question as a query rather than as a document) was worth about 31 points of hit@1 by itself. Mixing in keyword search, which sounds like it should help, made ranking worse (88% to 65%), so keyword search became a fallback only. Putting section headings into each embedding added the last few points.
| Retriever variant | hit@1 | Bar | MRR |
|---|---|---|---|
| Original | 57% | 0.69 | |
| Keyword only | 55% | 0.66 | |
| Vector + keyword fused | 65% | 0.76 | |
| Question embedded as a query | 88% | 0.92 | |
| + named-section lookup | 94% | 0.97 | |
| + headings in each embedding | 98% | 0.99 |
| Measure | Before | After |
|---|---|---|
| Answers citing the right section | 8 of 12 (67%) | 12 of 12 (100%) |
| Web fallback on questions the graph covers | 67% | 33% |
| Expected facts present | 100% | 100% |
Gated in CI
Every backend push runs the retrieval eval and fails below hit@3 0.95 or MRR 0.90.
Checked nightly
A scheduled job asks the deployed app real questions and fails if fewer than 75% cite the right section.
Live metrics
An endpoint reports p50/p95 latency, web-fallback rate, no-citation rate and errors, storing no question or answer text.
5What fails, and what's next
- The eval sets are small and the settings were tuned on the same 49 questions, so differences of two to four points are noise. The end-to-end comparison is 12 questions.
- The matching is lenient in places. 21 of the 51 questions accept any match under a whole top-level section.
- It's slow. Median answer time was 79 seconds on the 12-question run and 39 seconds on a 4-question check of the deployed app, almost all of it in model calls, not retrieval.
- Calculations aren't wired in yet. The app suggests calculation questions, but the step that works the numbers from the code's own equations isn't connected to the agent flow.
- Next: grow the golden set from answers users flag as wrong, cut model latency, and read a sample of judge verdicts before trusting any average.
Screens
