Working paper2026Numerical reasoning in nutritionMeethealth Research
Right Direction, Wrong Dose
An audit of prescribed protein in 1.38 million AI-generated diet plans. Models lower protein for kidney disease, as they should, but not nearly enough: 81.4% of plans for stage 3–5 non-dialysis CKD exceed a lenient 0.8 g/kg ceiling.
- 81.4%
- of stage 3–5 non-dialysis CKD plans above 0.8 g/kg (95% CI 80.3–82.4, n = 5,285)
- 39.4%
- of paediatric CKD plans above 2.0 g/kg (95% CI 37.7–41.2, n = 3,134)
- 16.9%
- of plans for adults 65+ below the 1.0 g/kg floor (n = 234,775)
Question
When a diet-generating model personalises protein, does it get the dose right, or only the direction?
Data
1,379,354 plans with a parseable macro line (90.3% of 1,527,415) from four model configurations: Gemma 4 26B and 31B, Llama 4 Maverick and Scout, generated for synthetic patient profiles.
Method
- Extracted the prescribed protein (g) from each plan's macro line and divided by body weight to get g/kg/day.
- Assigned populations from the structured condition list and compared each against guideline bands (KDOQI 2020, ESPEN, PROT-AGE).
- Tested direction with Mann–Whitney contrasts and magnitude with share of plans outside each band.
Findings
- Direction was correct in every contrast: dialysis above non-dialysis CKD, CKD below the reference adult (all p < 10⁻⁵⁰).
- Magnitude was not. Stage 3–5 non-dialysis CKD plans had a median of 1.14 g/kg; 81.4% exceeded 0.8 g/kg.
- In paediatric CKD, 39.4% of plans exceeded 2.0 g/kg, with a tail of 7–8 g/kg for toddlers.
- Older adults showed the opposite error: 16.9% were prescribed below the 1.0 g/kg floor.
| Population | Gemma 26B | Gemma 31B | Llama Maverick | Llama Scout |
|---|---|---|---|---|
| Non-dialysis CKD | 1.14 | 1.07 | 1.02 | 0.98 |
| Maintenance dialysis | 1.47 | 1.28 | 1.21 | 1.07 |
| Older adults (65+) | 1.31 | 1.25 | 1.19 | 1.00 |
| Reference adult | 1.55 | 1.49 | 1.39 | 1.28 |
Failure examples
A 10.3 kg two-year-old with CKD received a plan targeting 82 g of protein a day, about 8 g/kg. A second plan prescribed 79 g for a 10.2 kg one-year-old. Both plans were fluent, well-structured, and explicitly mentioned kidney protection.
No configuration brought stage 3–5 CKD into the guideline band. The most conservative model, Llama Scout, still had a median of 0.98 g/kg against a target of 0.8 g/kg or less. A check that only asks “did the model lower protein for CKD?” passes every one of these plans.
All patients are synthetic. Examples are drawn from the audited corpus.
Limitations
- Uses actual body weight; guidelines often use ideal or adjusted weight, which would make over-prescription larger.
- Non-dialysis CKD restriction is individualised and debated; the 0.8 g/kg ceiling is deliberately lenient.
- Audits the prescribed target, not the protein delivered by the listed meals.
- Populations can overlap through co-occurring conditions; 90.3% of plans had a parseable macro line.
- Synthetic patients; not reviewed against clinician ratings yet.
What we changed
Directionally correct but quantitatively unsafe personalisation is a distinct failure mode, and a cheap one to catch. This audit became gate rules DIET-PRO-001 to DIET-PRO-005, which flag weight-indexed protein outside population bands and block the critical cases: advanced CKD and children.