How Do Reasoning Models Work? (Test-Time Compute Scaling, Tree-of-Thought & RL Verification)

Beyond Chinchilla: The Emergence of the Test-Time Scaling Frontier

From 2020 to 2024, state-of-the-art AI development adhered strictly to the Chinchilla / Kaplan pre-training scaling laws: performance improved as a power-law function of model parameters ($N$) and pre-training dataset tokens ($D$) [1,2]. However, as frontier models ingested trillions of tokens and faced data wall limits and exponential compute training costs, a new scaling frontier emerged: Test-Time Compute Scaling [1,3].

Rather than predicting the next token in a single, fixed forward pass ($O(1)$ thinking time), reasoning models (such as OpenAI o1/o3 and DeepMind AlphaProof) are trained with large-scale reinforcement learning to generate extensive, hidden Chain-of-Thought (CoT) tokens at inference time [1,3,4]. By allocating seconds or minutes of compute to search, verify, and deliberate before generating the user-visible response, models convert inference compute directly into higher reasoning accuracy on complex STEM, mathematical, and coding problems [1,3,4].

"Reasoning models trade inference compute for intelligence: spending 30 seconds of test-time search solves problems that fail on instant forward passes."

Process Reward Models (PRMs) vs. Outcome Reward Models (ORMs)

The core technical breakthrough enabling reliable reasoning is the transition from Outcome Reward Models (ORMs) to Process Reward Models (PRMs) [1,4,5]. In traditional RLHF (like InstructGPT), an ORM evaluates only the final answer ($y$), assigning a scalar reward based on whether the final conclusion is correct [1,5]. For complex multi-step proofs, ORMs suffer from credit assignment failure—rewarding lucky guesses or flawed intermediate steps that happen to land on the correct answer [1,5].

In contrast, PRMs evaluate each individual step ($s_1, s_2, dots, s_k$) in a chain of reasoning: $$R( ext{CoT}) = prod_{i=1}^{k} P( ext{step } i ext{ is correct})$$ [1,5,6]. By training on dense, step-by-step verification datasets, PRMs enable the model to identify the exact step where an error occurred, prune bad branches, and search over high-probability paths using Monte Carlo Tree Search (MCTS) and beam search algorithms [1,5,6].

Reinforcement Learning for Self-Correction and Backtracking

Through large-scale policy optimization (PPO / DPO) trained on verifiable environments (formal proof languages like Lean, competitive programming sandboxes, and symbolic math engines), the model naturally develops sophisticated cognitive strategies [1,3,7].

When the policy encounters a contradiction in its internal scratchpad, it emits tokens that execute deliberate backtracking: "Wait, let me recalculate that assumption," "This derivative has an extra factor of two, let me start over" [1,3,7]. These self-reflective behaviors are not hardcoded rules, but emergent, optimal policy behaviors discovered by reinforcement learning to maximize downstream verification reward [1,3,7].

System 1 vs. System 2 AI Architectures

This paradigm mirrors Daniel Kahneman’s dual-process cognitive framework: standard autoregressive LLMs operate as fast, intuitive System 1 pattern matchers, while test-time search reasoning engines represent deliberate, logical System 2 problem solvers [1,3,4].

The implications for scientific discovery are profound: reasoning models can synthesize multi-page biochemical synthesis routes, formal mathematical proofs, and architectural software refactors with verified grounding and near-zero logical hallucination rates [1,3,4,7].