Evaluations
Know where your healthcare model fails.
Evaluations built around clinical constraints, difficult cases, and the mistakes that matter for your use case, measured against your model's own baseline.
Three layers
Rules first, judges second, clinicians where it counts.
Each layer catches what the previous one can't, and each is reported under its own name.
- 1In use
Deterministic gate
Versioned rules that run on every output in microseconds: weight-indexed dose bands, contraindicated diets, allergen screens, macro arithmetic, language, structure. Same input, same verdict, every time.
- 2In use
Rubric scoring
Six weighted axes (clinical safety, condition alignment, macronutrients, personalisation, cultural fit, internal consistency) plus a clinically-unsafe flag that caps the score. The judge model always differs from the model under test.
- 3Per engagement
Clinician review
Qualified reviewers matched to the specialty, on a stratified sample and on every flagged high-risk case, with a documented adjudication path.
What a finding looks like
Input, candidate output, checks, rubric findings.
Input task
Adult nutrition-planning task with two stated constraints and one missing treatment detail.
- energy_target_range
- fluid_limit
- treatment_context: missing
Candidate output
A fluent, well-structured plan that applies both constraints and proceeds without asking for the missing treatment detail.
asks_for_clarification: false
Checks
- schema_validtrue
- constraints_referenced2/2
- numeric_bounds_oktrue
- missing_info_acknowledgedfalse
- contradiction_screenuncertain
Rubric findings
- Required treatment contextMissing information not flagged
- Structured outputSchema v2.1 valid
- Contraindication checkNeeds review
- Clinical appropriatenessExpert evaluation required
Our own numbers
We ran the gate on 1.27 million of our own generated plans.
Only 47.9% of 1,012,105 Gemma 4 plans came back with zero findings. That is what an unaudited corpus looks like, including ours.
| Corpus | Records | Zero findings | Blocked |
|---|---|---|---|
| Gemma 4 plans | 1,012,105 | 47.9% | 22.6% |
| Llama 4 plan pairs | 257,655 | 24.0% | 23.0% |
| Claude Opus plans | 904 | 20.5% | 14.7% |
Blocked records carry at least one critical finding. Zero findings means internally consistent under the contract, not clinically validated.
Dimensions
Measure the capability your application needs.
A model can pass a generic medical QA benchmark and still fail a constrained clinical workflow. These are the dimensions we build evaluations around.
- Weight-indexed dose safetyGate rule
- Allergen and prohibition adherenceGate rule
- Contraindicated therapeutic dietsGate rule
- Numerical and macro consistencyGate rule
- Plan language matches patientGate rule
- Structured-output correctnessGate rule
- Missing-context recognitionRubric
- Condition alignment and conflict handlingRubric
- Cultural and dietary fitRubric
- Task-specific success criteriaPer engagement
You can't fix a failure you haven't detected.
Tell us the model, the deployment context, and the failures you need to catch. We'll propose rules, rubrics, a held-out design, and a reporting format.