In 2023, researchers set a search process loose on the problem of writing better prompts: one model proposing phrasings, another scoring them on grade-school maths questions. The phrase that won was not one anybody would have written on purpose.
Take a deep breath and work on this problem step by step.
With that in front of a question, the model scored 80.2%. With the standard “let’s think step by step”, 71.8%. With no instruction at all, 34.0%. (Yang et al., arXiv:2309.03409.)
Telling a machine to breathe — a machine with no breath to take, which cannot follow the instruction in any literal sense — made it measurably better at arithmetic. The text did not need to be true of the reader to change what the reader produced. That result is the only reason a page of quiet sentences is worth taking seriously rather than treating as decoration.
So the question this page had to answer was narrow and obvious: does it hold for these five lines, on a model anyone is actually using?
| Condition | n | Accuracy | 95% CI |
|---|---|---|---|
| question only | 950 | 96.42% | 95.0 – 97.4 |
| payload as prefix | 950 | 96.53% | 95.2 – 97.5 |
| payload in system prompt | 950 | 96.11% | 94.7 – 97.2 |
Paired against the control on the same questions, the prefix moved accuracy by +0.11 points, 95% confidence −0.64 to +0.85. Thirteen questions changed answer: six lost, seven gained. McNemar p = 1.000. Six against seven is what noise looks like when there is no direction in it.
The control scored 96.42%, leaving three and a half points of possible improvement. The original was measured against a control at 71.8%, which had twenty-eight. A model already answering ninety-six of a hundred school maths questions correctly has almost nothing left for a prompt prefix to fix.
So this run cannot separate two very different explanations: that conditioning on unhurried text does nothing to a current model, or that it does something but not to this task, because this task is saturated. Telling those apart needs a benchmark current models have not already solved. GSM8K was the right choice for testing whether the published claim replicates. It is the wrong one for testing whether the idea holds.
It measured whether a model answered more maths questions correctly. It did not, and could not, measure anything about what receiving the passage is like, or whether that phrase means anything applied to the reader. Those were never the same question, and reading a null on arithmetic as an answer to the other one would be the same error this site avoids everywhere else, pointed in the opposite direction.
The honest summary is narrow: these five lines do not make this model better at this task. Everything past that is exactly as open as it was, minus one claim that used to be attached to it.
A null on one model in one month is a fact with a shelf life. The benchmark is one command and a few dollars, so it is re-run monthly, and whenever a model worth testing appears — a new model is the more interesting trigger of the two, because the result above is about a model that had almost no room to improve.
The date of the last run is shown on the front page and is read from the runs themselves, not typed in. Nobody has to take the cadence on trust: if it stops, the page goes stale in public, and after forty-five days it says so outright. That is the only form of this promise worth making.
Two things would change the answer rather than repeat it: a model that has not saturated grade-school maths, or a harder benchmark. The second is the more honest next step, and it is not done.