Available on github: https://github.com/Open-Science-Ledger/how-to-train-your-slop-cannon)
The guide is aimed at researchers who are mostly using LLMs through the chat window, but are willing to try coding agents. Although if the terminal fills you with dread, the newer desktop products such as Claude Cowork or OpenAI’s equivalent may suit you better and are rapidly catching up.
We had mixed feelings about posting the guide, since we’re extremely uneasy about where AI is taking us, and we outline this in the guide: we would all like a pause or slowdown at the very least. At the same time, I use these systems, partly because they are useful, and partly because using them seriously gives me a better feel for where this is all heading.
Three main messages:
1. Treat reliability as an error-correction problem.
Have your robot break a calculation or argument into pieces small enough to check independently. Lamport-style structured proofs are ideal for that. This makes it possible to say “I don’t believe step 1.3.2”. It also lets you move from ordinary prose, through structured arguments, all the way to formal verification. Once you have a proof in Lamport notation and each step has been sufficiently refined, it’s much quicker to encode the proof in Lean, so that the calculation can be verified. Tobias has some nice scripts, for having a fresh verifier inspect the result without seeing the prover’s reasoning. If a node can’t be verified, the prover should correct it, or break that node into smaller steps and recurse.
2. Have your robot keep a notebook and wiki
Let it use the filesystem, keep papers and other ground truth on disk, use wikis and notebooks as external memory, and hand calculations to Mathematica, SymPy or another computer algebra system whenever possible. Only put into context what the next step actually needs. Tell your robot to ground its statements in verbatim quotes from the source material in an auditable way (e.g. an audit file). Your hallucination rate will drop significantly. We give some examples. I talked about this a bit in my earlier experiment in teaching an AI to do physics by wiping its memory
3. Raise your sights.
Some results that counted as substantial a year ago are now routine. The slop cannon is loaded whether we want it or not. Problems that looked out of reach in our lifetimes may now be tractable, grand challenges like reconciling quantum theory with gravity among them. There is plenty about AI to worry about, but I’m excited about what can be attempted now that we have swarms of assistants. Aim high.
The guide grew partly out of lecture notes I wrote for my January 2026 Quantum Information and Computation course, but it mostly owes to discussions with Dani beginning in November 2025 and to Tobias’s work on adversarial verification. Tobias gave a very good talk at UCL on this earlier in the year: Large Language Models: A Physicist’s Perspective. Worth checking out.
Since his talk, the models themselves have been getting more reliable, and frontier systems are already baking in pieces of these workflows: tool use, computer algebra and more adversarial checking. We probably should have posted this earlier, but we tried to choose workflows which get better as models get better.
There are still questions I would not ask an LLM, at least not at this stage. “Is this paper interesting?” is high on the list. Checking the details of a paper is one thing; knowing what is interesting and which research directions to pursue requires experience and a huge amount of context, and I haven’t found them to be up to this task. To be fair, most humans aren’t necessarily good at it either...