Practical Failure Modes in RL Training for LLMs

July 15, 2026

I’ve only been doing RL training for a few months, mostly during my Anthropic fellowship project. These are debugging notes from that work, not established best practices. Some are low-confidence, and corrections are welcome.

One general tip: get your agent to read the official documentation or a reference implementation before it writes any training code. The quality of the generated code is much better than the Claude Code or Codex default. This applies to methods too, with a caveat: a lot of this is genuinely unsettled, and a search-grounded agent will happily hand you one paper’s framing as if it were consensus. Use it to find out what the options are, not to find out which one is right.

Rollouts

• If you’re mixing tasks, check that each task’s output is going to the correct reward model.

• With multiple tasks, individual rewards might increase at different rates depending on task difficulty and the underlying model’s propensities. This is not always a problem, but log each reward separately so you can see where the improvements are actually coming from and whether you’re getting better at the task you care about.

• If you’re mixing thinking and non-thinking examples, I’ve found half and half to work okay. Low confidence: this is probably worth further exploration.

• Set max_tokens high enough for rollouts to actually complete. Run several examples from the prompt dataset you’re training on to choose this number.

• Save your rollouts, and actually read them. They’re a great debugging surface. Reward going up is completely compatible with behavior getting worse, and reading samples at a few checkpoints is often how you find out.

• Watch for other sources of weird truncation. One of my least favorite agentic-coding behaviors is taking a string and being “efficient” by slicing it, for example with string[:3000], for no reason. This can silently corrupt your findings.

• Use the same chat-template path for training, rollouts, and evaluation, and inspect a few rendered examples. Small serialization differences can matter.

Masking and thinking templates

• For completion-only objectives, verify that prompt tokens are masked. This is usually standard in the framework you’re using, but it is worth checking.

• For non-thinking training, consider masking a closed empty think block rather than leaving it out of the example. Match the exact template generated by the tokenizer, including whitespace. For example, some Qwen3 templates represent non-thinking generation with <think>\n\n</think>\n\n; don’t reproduce this from memory when you can render it with the model’s apply_chat_template.

• When training or evaluating with thinking on, prefilling with an open think tag seems to help the model know it is supposed to think. Low confidence: I don’t know whether this is generally useful. Check whether your chat template already appends one, or you’ll end up with <think><think>.

Judging and rewards

• Sanity-check the reward model before training on it. Score a handful of known-good and known-bad completions by hand and see whether it agrees with you.

• If preserving chain-of-thought monitorability matters, keep outcome judges blind to the thinking trace unless you’re deliberately doing process supervision. Scoring the trace applies optimization pressure to it, which may make it less monitorable or less representative of behavior. The exact effect depends on the training incentive; see Reasoning Under Pressure for one empirical treatment.

• Put strong checks around stray thinking tags and handle them aggressively. This becomes surprisingly tricky as the model trains and its outputs start coming out malformed.

• For non-thinking RL rollouts, one thing I’ve tried is penalizing stray thinking tags. It cleaned up the behavior, especially when prior SFT or SDF had damaged the model’s thinking behavior. Very low confidence: I’m still figuring out whether this is the right intervention; much longer, lower-learning-rate SFT might be better.

Optimization

• Do some learning-rate sweeps. RL does not necessarily have the same learning-rate characteristics as SFT.

• Consider constant learning rates and other schedules rather than defaulting to whatever the framework provides.

• Consider Dr. GRPO rather than assuming standard GRPO is neutral. Dr. GRPO removes per-response length normalization and per-question reward-standard-deviation normalization. The former can bias response length, particularly for incorrect answers; the latter changes how groups with different reward variance are weighted. I’m not sure how large the practical improvement is across settings, but these defaults are worth choosing deliberately.

• Think about group size. If every rollout in a group gets the same reward, the advantage is zero and the group teaches you nothing. This connects to the multi-task point above: a task that’s too easy or too hard can end up contributing no gradient at all.

Infrastructure

Consider a service like Tinker. I used their product during a fellowship and like it because:

• You retain control over your data and training algorithms.
• You don’t have to worry about getting a GPU or serving a checkpoint.
• Its cookbook and documentation are useful starting points.