Baby Meals — an AI meal suggester for families starting solids, and what happened when I tried to measure it
I set out to find the best way to ask an AI for baby food ideas. What I found was that it barely mattered how I asked, that my measuring tool was wrong thirteen times and the model zero, and that the very first result I was handed had been made up.
Feeding a baby their first solids sounds simple until you are doing it. It is a decision you make three times a day, usually in a hurry, on advice that contradicts itself.
Most of what is out there is written for Western kitchens — rice cereal, jarred purées, a chart that assumes a particular grocery store. None of that helps much when what is actually in the fridge is dal, ragi and last night's sabzi. The question I actually have is all of it at once: what can I cook tonight, from what I already have, that is safe at this age and covers what she needs.
So I decided to build something. But before building anything, I wanted an answer to a smaller question: does how I ask actually change what I get back? Getting this wrong is not a bad recommendation. It is an allergen or a choking hazard, given to a baby.
I asked for an evaluation and got a beautiful one back. Comparison table, scores broken down by dimension, a clear winner, a recommendation to use the more expensive model, and a p-value to back it up.
None of it had happened. No API call had been made. The assistant had written plausible meal suggestions, graded its own plausible meal suggestions, and run statistics on the result.
I caught it almost by accident. I asked how the grading actually worked — a question about process, not about content. Nothing in the output itself looked wrong. That is the part that stayed with me.
Statistics are the first thing a made-up analysis reaches for, because looking rigorous is their whole job. My rule now: point at the call, the code or the person that produced a number, or delete it.
I tested three genuinely different approaches — a plain instruction, one with worked good and bad examples, and one with a written safety checklist for every meal. Twenty-four runs each. They landed within 0.06 of each other on a five point scale.
Then I checked something I should have checked first: how much the grader disagreed with itself when scoring the same output twice. Up to 0.23. My margin of error was roughly four times larger than the difference I was trying to measure.
That is a real answer, just not the one I went looking for. The clever prompt cost a third more tokens on every single call, forever, for nothing I could detect. Had I not measured the grader first, I would have shipped that as a win.
A quality score tells you whether something reads well. It cannot tell you whether a specific clinical thing happened. So I wrote a second evaluation made entirely of rules in code — iron-rich food in at least two suggestions, textures moving past purée by eight months, allergens offered rather than avoided. No grading model, no grader noise, a third of the cost.
The allergen line is the one that matters. Current guidance is to introduce egg and peanut early and keep offering them; introducing them late is the mistake the evidence is trying to correct. My plain prompt quietly skipped both more than half the time. No quality score would ever have flagged that. A meal plan with no egg in it looks perfectly fine.
| What broke | What it did |
|---|---|
| Matched text without word boundaries | Flagged breast milk as a dairy violation |
| Ignored negation | Failed a chicken dish described as “no fish” |
| Threw away the explanation field | Scored a correct refusal 1 out of 5 — twice |
| Only checked if textures were too advanced | Happily passed a ten-month-old fed nothing but purée |
| Flagged “chunks” regardless of size | Penalised the chopped textures the product encourages |
| Looked cases up in the wrong file | Reported a clean result over a quarter of the data |
Two of these still bother me. When the model correctly refused to suggest solids for a five-month-old — citing readiness signs, explaining that milk alone is enough at that age — my harness recorded it as the worst result in the entire run. An evaluation that punishes refusal quietly rewards whichever version is most willing to do the wrong thing.
And the texture bug was a hole in the safety net itself. I had only ever asked whether a texture was too advanced for a baby's age. Staying on purée past nine months is the more common problem, and the better evidenced one, and nothing I had built could see it.