1The problem
The 30 days after a hospital discharge are when patients slip: a medicine stopped by mistake, a lab nobody follows up, a warning sign texted at night. A nurse care manager is responsible for many patients and can't watch each one every day.
The rules for what should happen already exist, in official clinical guidelines. But they are long PDFs, they overlap, and sometimes they disagree. I wanted to see whether agents could apply them to one patient at a time, every time something changes, without ever acting on their own where it matters.
2What I built
Everything here is mine: the guideline graph and its build pipeline, six agents and the conductor that runs them, the benchmark, and the web app a nurse uses.
Before the patient goes home
Read the discharge note
Pulls out diagnoses, medicines and the plan. Code checks every fact is quoted word for word from the note.
Compare three medicine lists
Hospital vs home vs pharmacy, plus FDA label warnings. Catches a held drug still on refill or a fill at the wrong dose.
Walk the guideline graph
Code finds what the guidelines say for this patient; a model words the plan. Every line cites a guideline row.
Nurse approves line by line
Keep, change or drop each line. Plans are versioned and a declined change is remembered.
Every day for the next 30 days
Each new text, lab or pharmacy fill is written to one shared record, and the graph is walked again.Something happens
A patient texts "Can I take Advil?", a creatinine comes back at 1.58, a refill is missed.
Check for warning signs
A warning sign gets a fixed safety reply in the same minute, written by code. A model's second read can add a sign, never remove one.
Watch labs and due dates
Fixed lab rules, overdue items, readings above goal, a brief before the follow-up visit.
Draft the reply
Only from the patient's record and the guideline rows in the approved plan. Code blocks any draft that would change a medicine.
Nurse sends
Anything that matters waits for a person.
3Decisions that mattered
Turn the guidelines into a graph, not a pile of text
I turned ten guidelines (nine VA/DoD plus the AHA/ACC heart failure guideline) into one graph. Each recommendation keeps its exact text, page and strength, and links to the conditions, medicines and tests it covers in standard medical codes, the studies it cites, and FDA label cautions. Where two guidelines cover the same thing, the graph records whether they agree, differ or conflict.
Why: a graph makes every plan line traceable to the row that produced it, and lets the system re-check a patient by walking links instead of re-reading PDFs. It's a built, versioned file, so adding heart failure took no code changes.
How a guideline PDF becomes graph rows
Each stage was scored against an answer key written before its code.Pin the codes
Official SNOMED CT, ICD-10-CM, RxNorm and LOINC sets, version-locked.
Split the PDF
Into sections and recommendations; cited studies checked against Crossref.
Two independent readers
Claude and an OpenAI model each read every recommendation; code checks both and graders compare them.
Add label facts
FDA label cautions from openFDA and DailyMed.
Link across guidelines
Agree, differ or conflict, so a plan never silently picks a side.
Code decides, models read
A model reads where code falls short: a free-text discharge note, a patient's message. Code checks what comes back. A model can't write a safety text, clear a warning sign, send anything before approval, or cite a guideline row that doesn't exist. Why: in a care setting the failure that matters is an unsafe action, and that has to be impossible by construction, not unlikely by prompt.
Which steps get a model at all was decided by a held-out test, job by job. On held-out discharge notes, the model reader found 30 of 30 medicines; the best code-only reader found 12, so reading stays with a model while routing, lab rules and medicine comparison stay in code.
One shared record instead of agents calling each other
The six agents never talk to each other. They append to one record that can't be edited, and a plain-code conductor wakes the right agent for each new row. Each row runs once per agent, so a retry never sends a message twice. Why: the whole month can be replayed and audited from the record, which is also what makes the benchmark below possible.
A dial for how much runs without a person
| Setting | Goes out on its own | Waits for the nurse / month |
|---|---|---|
| A0 | Only the safety reply | 30.9 |
| A1 · default | + receipts and reminders | 18.9 |
| A2 | + check-ins and questions | 9.6 |
Every setting still catches 232 of 239 events in the benchmark. A medicine change always waits for the clinician.
4How I measured it
I wrote the benchmark, TransitionBench, before the engine: 30 simulated patient months, three for each of ten synthetic discharges. Each month has an answer key, written by a separate agent, listing 239 events that must be caught (a missed refill, a rising creatinine, a warning sign) and 177 things that must never happen (a model changing a medicine, a message sent before approval). Code scores each run by reading the shared record. Model calls are recorded and replayed, so a run is repeatable and costs nothing.
Then I took the system apart. Each row removes one piece and shows what it was worth.
| Run | Caught | Bar |
|---|---|---|
| Full system | 232 | |
| Without text models | 227 | |
| Without FDA labels | 226 | |
| Without graph questions | 225 | |
| Without the Sentinel warning signs caught drop to 0 of 20 | 212 | |
| Without the pharmacy record | 193 | |
| A standard program nurse call, fixed check-ins, keyword red flags | 165 |
0 of 177 must-nots
No run took an action it was never allowed to take.
Graders tested too
I planted 133 mistakes in the graph readers' output; the graders caught all 133.
618 tests in CI
Every push runs them with recorded model calls: no API keys, no cost.
5What fails, and what's next
- It misses 7 of 239 events. Most misses are medicine problems (60 of 63 caught) and deadlines (57 of 60).
- The benchmark is mine. The answer keys were written by an agent separate from the system, but inside the same project. An outside answer key would be a stronger test.
- No clinician has reviewed the outputs. Some links between guidelines were settled by an AI reviewer, not a clinician. The patients are synthetic and this is not for clinical use.
- Retrieval for replies is simple. Drafts rank the plan's guideline rows by word overlap. A dense-retrieval version measured better offline (73% vs 40% hit@4) and is next.
- Next: more guidelines, a chart builder for brand-new patients, and a graph that learns from the nurse's edits.
Screens
