Category
Machine Learning
Posts and notes in Machine Learning.
- November 5, 202638 min read
Formal Verification for LLM Reasoning: Solvers and Theorem Provers
How solvers and proof assistants check LLM reasoning: solver feedback on partial programs, formal logic checks inside a chain of thought, and LLM theorem provers for Lean, Isabelle and Coq that retrieve, search, repair and learn from the checker.
- October 29, 202644 min read
Reward Models for Reasoning: ORMs and PRMs
How outcome and process reward models judge reasoning: scoring the final answer or every step, where their step labels come from, why GRPO already contains one, how verifiers that write and run checks work, and how to train them with few or no labels.
- October 21, 202612 min read
Contrastive Perception Policy Optimization (CPPO)
How CPPO improves perception in vision-language models during reinforcement learning: it finds the tokens whose entropy rises most when the image is hidden, and trains them with a contrastive loss to stay stable under harmless image changes and to shift when the image's content is removed.
- October 16, 202611 min read
Perception-Aware Policy Optimization (PAPO)
How PAPO teaches multimodal models to look at the image while they learn to reason: it adds a loss that rewards responses whose probability drops when the image is masked, and keeps that loss from being gamed with two entropy terms.
- October 5, 202613 min read
Visually-Perceptive Policy Optimization (VPPO)
How VPPO makes reinforcement learning for vision-language models pay attention to the image: it measures how much each token depends on the picture, scales each response's advantage by that dependency, and updates only the most image-dependent tokens.
- September 30, 202612 min read
Adversarial Defense with Preference Optimization (AdPO)
How AdPO turns adversarial training into a preference optimization problem: a vision-language model learns, with DPO, to prefer its own description of a clean image over its description of an attacked one, and to describe the attacked image as if it were clean.
- September 19, 202611 min read
Adversarial Policy Optimization (APO)
How APO makes a reinforcement learning agent robust to high-dimensional and noisy observations: a perturber network learns small changes to the observation that change the agent's actions the most, and the policy network learns to ignore them, in a max-min game on top of PPO.
- September 13, 202611 min read
Group Sequence Policy Optimization (GSPO)
How GSPO fixes the instability of GRPO by moving the importance ratio from tokens to whole responses: one length-normalized sequence ratio, sequence-level clipping, and equal weights for every token, which stabilizes large Mixture-of-Experts training and powered Qwen3.
- September 3, 202610 min read
Clipped IS-weight Policy Optimization (CISPO)
How CISPO, from MiniMax-M1, keeps every token in the gradient: instead of clipping token updates like PPO and GRPO, which silently drops rare reasoning tokens such as 'Wait' and 'However', it clips the importance sampling weight and lets every token keep learning.
- August 27, 202613 min read
Value-Model-Based Augmented PPO (VAPO)
How VAPO makes PPO with a value model work for long chain-of-thought reasoning: pretraining the value model, decoupling and adapting GAE to response length, and borrowing DAPO's fixes for sparse rewards, to beat value-free methods like GRPO and DAPO.
- August 16, 202612 min read
Dr. GRPO: Group Relative Policy Optimization Done Right
How Dr. GRPO fixes two biases in GRPO: dividing by the response length makes long wrong answers cheap, and dividing by the group's standard deviation overweights questions that are too easy or too hard. Removing both recovers the unbiased policy gradient and stops responses from growing needlessly long.
- August 7, 202613 min read
Generative Adversarial Policy Optimization (GAPO)
How GAPO teaches language models to follow constraints: instead of pairing one prompt with a good and a bad response, it pairs one response with a matching and a violating prompt, and trains the policy against an encoder-only reward model in a GAN-style adversarial loop.
- July 29, 202616 min read
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)
How DAPO fixes the failures of GRPO in long chain-of-thought reasoning: Clip-Higher against entropy collapse, dynamic sampling against zero-gradient prompts, a token-level loss for long responses, and overlong reward shaping for truncated answers.
- July 21, 202614 min read
Simple Preference Optimization (SimPO)
How SimPO simplifies DPO: it replaces DPO's reward with the average log-likelihood that the model uses to generate text, which removes the reference model, and adds a target reward margin to separate preferred and rejected responses.
- June 20, 202616 min read
REINFORCE Leave-One-Out (RLOO)
How RLOO replaces PPO in RLHF with the simplest policy gradient: treating the whole response as one action, using REINFORCE, and comparing each response with the average reward of the other responses to the same prompt.