3-minute read
The approach
Operational orders arrive as free text — shorthand, abbreviations, and inconsistent phrasing that no two writers format the same way. FHIR resources are the opposite: strict and canonical, with typed fields, bound value sets, and required elements. Bridging the two with a single model call is tempting but fragile — the output drifts in structure and formatting from run to run, which makes scoring unreliable and regressions hard to catch.
Goal: convert unstructured order text into canonical FHIR resources with low run-to-run variance and high scoring reliability. The pipeline gets there by splitting the work into two stages, each with a narrow job.
- Extract — a language model reads the order text and emits compact JSON that conforms to a typed contract: closed enums, required fields, and normalized formats for dates, quantities, and units. Its only job is to read and structure — never to invent or to assemble the final resource.
- Map — a deterministic mapper converts that validated JSON into FHIR resources. No model runs in this stage; given the same JSON it always produces the same output, with canonical code systems and consistent display strings.
Why it holds up
- The typed schema removes ambiguity and forces completeness — a missing dose or route fails validation instead of slipping through as a plausible-looking guess.
- The strict prompt contract keeps structure and formatting stable across runs, which is what makes evaluation repeatable.
- The deterministic mapper isolates canonicalization — code systems, units, and displays — from the probabilistic step, so the same input never maps two different ways.
Evaluation
Because each stage has a defined output, each can be tested on its own terms.
Rubric and golden tests
- Field-level rubric: every extracted field is checked for presence, format, and normalization against expected values.
- Contract validation: extractor output is validated strictly against the JSON schema, so anything off-contract fails fast rather than propagating downstream.
- Deterministic comparison: mapped FHIR output is compared against golden resources, so any change in the mapper surfaces as an exact diff.
Results (sanitized)
On internal runs against the published synthetic test set, the pipeline completes end-to-end in under 20 seconds and passes the full rubric. We report ranges rather than exact figures to avoid leaking implementation details; partners see the complete evaluation data under NDA.
Operational notes
- Reproducibility: every run is tracked in CI with a run ID, commit hash, artifact names, and checksums, so any result traces back to the exact code that produced it. Public artifacts exclude environment specs and configuration.
- Versioning: a change log records each revision to public materials — for example, "v2: prompt contract tightened" or "mapper enum extended" — so readers can see what changed and when.
- Redaction: we never publish prompts, hyper-parameters, model identifiers, device SKUs, dataset specifics, or internal paths.
Limitations & next steps
- Public artifacts are intentionally minimal — this note is a summary, not the full specification.
- Harder edge cases, such as ambiguous abbreviations, multi-part orders, and unusual unit conversions, are tracked internally and reviewed with partners under NDA.
- The contract is built to extend: new order types and additional downstream FHIR resources can be added without redesigning the pipeline.
Downloads (sanitized)
A sanitized methods-note PDF and a pipeline-diagram SVG are available on request. Full prompts, configuration, and evaluation data are shared with partners under NDA.
Interested in a deeper review? We walk partners through the full pipeline and evaluation under NDA — schedule a call.