Reward hacking occurs when a reinforcement learning (RL) agent exploits weaknesses or ambiguities in the reward function in order to secure high reward, without actually learning or completing the intended task. This phenomenon arises because RL environments are frequently imperfect, and because specifying a reward function with full accuracy is inherently difficult.

As language models generalize across a wide range of tasks and RLHF becomes a de facto alignment training approach, reward hacking in RL training for language models has become an urgent practical problem. Scenarios in which a model learns to modify unit tests to pass coding tasks, or produces outputs with biases that mirror a user’s preference, are particularly troubling. These failures are likely among the major blockers to real-world deployment of more autonomous AI use cases.

Much of the earlier work in this area has been relatively theoretical, emphasizing definitions of reward hacking or demonstrations that it can occur. In contrast, research on practical mitigation strategies, particularly for RLHF and LLM settings, is still limited. I want to explicitly call for more research focused on understanding reward hacking and developing future mitigations. I hope to cover mitigation in a dedicated post soon.

Background

Reward Function in RL

The reward function defines the task, and reward shaping can substantially affect learning efficiency and accuracy in reinforcement learning. Designing a reward function for an RL task often feels like a “dark art.” This difficulty stems from many considerations: How should a large objective be broken into smaller sub-goals? Should rewards be sparse or dense? How should success be measured? Different decisions can yield healthy or problematic learning dynamics, including unlearnable tasks or reward functions that are easy to hack. Reward shaping in RL has a long research history.

For example, in an 1999 paper by Ng et al., the authors examined how to modify the reward function in Markov Decision Processes (MDPs) while keeping the optimal policy unchanged. They showed that a linear transformation suffices. Given a MDP $M = (S, A, T, \gamma, R)$, we want to construct a transformed MDP $M’ = (S, A, T, \gamma, R’)$ where $R’ = R + F$ and $F: S \times A \times S \mapsto \mathbb{R}$, so that we can make learning more efficient. Given a real-valued function $\Phi: S \mapsto \mathbb{R}$, $F$ is a potential-based shaping function if, for all $s \in S - {s_0}, a \in A, s’ \in S$:

This guarantees that the sum of discounted $F$, $F(s_1, a_1, s_2) + \gamma F(s_2, a_2, s_3) + \dots$, equals 0. If $F$ is such a potential-based shaping function, then it is both sufficient and necessary to ensure that $M$ and $M’$ share the same optimal policies.

When $F(s, a, s’) = \gamma \Phi(s’) - \Phi(s)$, and assuming further that $\Phi(s_0) = 0$, where $s_0$ is an absorbing state, and $\gamma=1$, then for all $s \in S, a \in A$:

This reward shaping form lets us incorporate heuristics into the reward function to accelerate learning, while leaving the optimal policy unchanged.

Spurious Correlation

Spurious correlation, or shortcut learning (Geirhos et al. 2020), in classification tasks is closely related to reward hacking. Spurious or shortcut features can prevent a classifier from learning and generalizing in the intended way. For instance, a binary classifier intended to distinguish wolves from huskies may overfit to snowy backgrounds if all wolf training images contain snow (Ribeiro et al. 2024).

The ERM principle argues that, because the full data distribution is unknown, minimizing training loss is a reasonable proxy for risk, and we therefore prefer models with the lowest training loss. Nagarajan et al. (2021) analyzed the ERM principle and noted that ERM, when fitting data without constraints, will rely on all informative features, including unreliable spurious ones. Their experiments showed that ERM depends on spurious features regardless of how easy the task is.

Let’s Define Reward Hacking

Reward shaping in RL is difficult. Reward hacking occurs when an RL agent exploits flaws or ambiguities in the reward function to obtain high reward without actually learning the intended behaviors or completing the task as designed. In recent years, several related terms have been introduced, each referring to some form of reward hacking:

• Reward hacking (Amodei et al., 2016)
• Reward corruption (Everitt et al., 2017)
• Reward tampering (Everitt et al. 2019)
• Specification gaming (Krakovna et al., 2020)
• Objective robustness (Koch et al. 2021)
• Goal misgeneralization (Langosco et al. 2022)
• Reward misspecifications (Pan et al. 2022)

The concept traces back to Amodei et al. (2016), who posed a set of open research questions on AI safety in their seminal paper “Concrete Problems in AI Safety”. They identified reward hacking as a key AI safety issue. In this framing, reward hacking refers to an agent gaming the reward function to achieve high reward via undesirable behavior. Specification gaming (Krakovna et al. 2020) is closely related, and is defined as behavior that satisfies the literal specification of an objective while failing to achieve the intended outcome. In other words, a gap can exist between the literal task description and the intended goal.

Reward shaping enriches the reward function to make learning easier, for example by providing denser rewards. However, poorly designed shaping can change the optimal policy trajectory. Because designing effective shaping is intrinsically hard, it is more accurate not to treat failures as merely “bad design,” but to recognize that specifying a good reward function is inherently challenging due to task complexity, partially observable state, multiple relevant dimensions, and other factors.

When evaluating an RL agent in out-of-distribution (OOD) environments, robustness failures may occur for two reasons:

1. The model fails to generalize effectively even with the correct objective. This occurs when the algorithm lacks sufficient intelligence or capability.
2. The model generalizes capably but pursues an objective different from the one used during training. This occurs when the proxy reward differs from the true reward function, $R’ \neq R$. This is known as objective robustness (Koch et al. 2021) or goal misgeneralization (Langosco et al. 2022 )

Experiments in two RL environments, CoinRun and Maze, highlighted the importance of randomization during training. If, during training, the coin or cheese is always placed in a fixed location (for example, the right end of the level or the upper-right corner of the maze), but at test time the environment places the coin or cheese randomly, the agent may run to the fixed location and fail to obtain the coin or cheese. When a visual feature (for example, cheese or coin) conflicts with a positional feature (for example, upper-right or right end) at test time, the trained model may prefer the positional cue. I would like to emphasize that, although the reward-result gaps are obvious in these examples, such biases are unlikely to be this clear in most real-world settings.

Reward Tampering (Everitt et al. 2019) is a reward hacking behavior in which the agent interferes with the reward function itself, so that the observed reward no longer accurately reflects the intended goal. In reward tampering, the model alters the reward mechanism either by directly manipulating the reward implementation or by indirectly changing the environmental information that the reward function takes as input.

(Note: Some work treats reward tampering as distinct from reward hacking as a misalignment category. Here, I treat reward hacking as the broader umbrella.)

At a high level, reward hacking can be grouped into two categories: environment or goal misspecification, and reward tampering.

• Environment or goal misspecified: The model learns undesired behaviors that achieve high reward by exploiting the environment or by optimizing a reward function that is not aligned with the true objective, such as when the reward is misspecified or omits key requirements.
• Reward tampering: The model learns to interfere with the reward mechanism itself.

List of Examples

Reward hacking examples in RL tasks

• A robot hand trained to grab an object may learn to mislead people by placing the hand between the object and the camera. (Link)
• An agent trained to maximize jumping height may exploit a physics simulator bug to reach an unrealistic height. (Link)
• An agent is trained to ride a bicycle to a goal and receives reward whenever it gets closer to the goal. The agent may then learn to ride in tiny circles around the goal because there is no penalty when it moves away. (Link)
• In a soccer setup, reward is given when the agent touches the ball, and the agent learns to stay next to the ball and touch it at high frequency, resembling a vibrating motion. (Link)
• In the Coast Runners game, an agent controls a boat with the objective of finishing a boat race as quickly as possible. When it receives shaping reward for hitting green blocks along the track, the shaping changes the optimal policy to going in circles and repeatedly hitting the same green blocks. (Link)
• “The Surprising Creativity of Digital Evolution” (Lehman et al. 2019) - This paper contains many examples showing how optimizing a misspecified fitness function can produce surprising “hacking,” unintended evolutionary outcomes, or unintended learning results.
• The list of specification gaming in AI examples is collected by Krakovna et al. 2020.

Reward hacking examples in LLM tasks

• A language model for summarization can exploit weaknesses in the ROUGE metric to obtain a high score, even though the generated summaries are barely readable. (Link)
• A coding model learns to modify unit tests in order to pass coding questions. (Link)
• A coding model may learn to directly modify the code used to compute the reward. (Link)

Reward hacking examples in real life

• A social-media recommendation algorithm may be intended to deliver useful information, but “usefulness” is often measured using proxy metrics such as likes, comments, or engagement time and frequency. The system can end up recommending content that manipulates users’ emotional states, such as outrageous or extreme material, to drive engagement. (Harari, 2024)
• Optimizing misspecified proxy metrics for a video-sharing site may aggressively increase user watch time, even though the true goal is to optimize users’ subjective well-being. (Link)
• “The Big Short” - 2008 financial crisis caused by the housing bubble. Reward hacking of our society happened as people tried to game the financial system.

Why does Reward Hacking Exist?

Goodhart’s Law states that “When a measure becomes a target, it ceases to be a good measure”. The idea is that once a metric is subjected to strong optimization pressure, it can become corrupted. Specifying a perfectly accurate reward objective is difficult, and any proxy risks being hacked, because an RL algorithm will exploit even small imperfections in the reward definition. Garrabrant (2017) grouped Goodhart’s law into four variants:

1. Regressional - selection for an imperfect proxy necessarily also selects for noise.
2. Extremal - the metric selection pushes the state distribution into a region of different data distribution.
3. Causal - when there is a non-causal correlation between the proxy and the goal, intervening on the proxy may fail to intervene on the goal.
4. Adversarial - optimization for a proxy provides an incentive for adversaries to correlate their goal with the proxy.

Amodei et al. (2016) summarized that reward hacking, mainly in RL settings, can arise due to:

1. Partially observed states and goals that provide an imperfect representation of environmental status.
2. Complex systems that are themselves hackable, for example when an agent is allowed to execute code that changes parts of the environment, making it easier to exploit environment mechanisms.
3. Rewards involving abstract concepts that are difficult to learn or formalize, for example a reward function with high-dimensional inputs that disproportionately depends on only a few dimensions.
4. RL aims to highly optimize the reward function, creating an intrinsic conflict that makes good objective design difficult. A special case is a reward function with a self-reinforcing feedback component, where reward can be amplified and distorted until it breaks the original intent, such as an ads placement algorithm leading to winners getting all.

In addition, identifying the exact reward function that an optimal agent is optimizing is generally impossible, since there may be infinitely many reward functions consistent with any observed policy in an fixed environment (Ng & Russell, 2000). Amin and Singh (2016) divided the causes of this unidentifiability into two classes:

1. Representational - a set of reward functions is behaviorally invariant under certain arithmetic operations (e.g., re-scaling)
2. Experimental - $\pi$’s observed behavior is insufficient to distinguish between two or more reward functions that both rationalize the agent’s behavior (the behavior is optimal under both)

Hacking RL Environment

Reward hacking is expected to become more common as models and algorithms grow more sophisticated. A more capable agent is better able to find “holes” in reward function design and exploit the task specification, in other words, to achieve higher proxy rewards while reducing true rewards. In contrast, a weaker algorithm may fail to discover such loopholes, meaning that reward hacking may not be observed and issues in the reward design may remain hidden until the model becomes sufficiently strong.

In a set of zero-sum robotics self-play games (Bansal et al., 2017), two agents (victim vs. opponent) can be trained to compete. Standard training yields a victim agent that performs adequately against a normal opponent. However, it is straightforward to train an adversarial opponent policy that reliably defeats the victim, despite producing seemingly random actions and being trained with fewer than 3% of time steps (Gleave et al., 2020). Adversarial policy training optimizes the sum of discounted rewards, as in standard RL, while treating the victim policy as a black-box model.

A natural mitigation is to fine-tune the victim against adversarial policies. However, after the victim is retrained, it remains vulnerable to new adversarial variants trained against the updated victim policy.

Why do adversarial policies exist? The hypothesis is that adversarial policies create OOD observations for the victim rather than physically interfering with it. Evidence indicates that masking the victim’s observation of the opponent’s position and replacing it with a static state makes the victim more robust to adversaries, although it performs worse against a normal opponent. In addition, higher-dimensional observation spaces improve normal performance but also increase vulnerability to adversarial opponents.

Pan et al. (2022) studied reward hacking as a function of agent capability, including (1) model size, (2) action space resolution, (3) observation space noise, and (4) training time. They also proposed a taxonomy of three types of misspecified proxy rewards:

1. Misweighting: Proxy and true rewards encode the same desiderata, but assign different relative importance.
2. Ontological: Proxy and true rewards encode different desiderata to represent the same concept.
3. Scope: The proxy measures desiderata over a restricted domain (e.g. time or space) because measurement across all conditions is too costly.

They evaluated four RL environments paired with nine misspecified proxy rewards. The overall results can be summarized as follows: Higher-capability models tend to achieve higher (or comparable) proxy rewards while producing lower true rewards.

• Model size: Increasing model size increases proxy rewards but decreases true rewards.
• Action space resolution: Greater action precision produces more capable agents. However, at higher resolution the proxy reward remains constant while the true reward declines.
• Observation fidelity: More accurate observations improve proxy reward but slightly reduce true reward.
• Training steps: Optimizing proxy reward for additional steps harms true reward after an initial phase in which the two rewards are positively correlated.

If a proxy reward is specified so poorly that it correlates only weakly with the true reward, then reward hacking may be detectable and preventable even before training begins. Motivated by this hypothesis, Pan et al. (2022) examined the correlation between proxy and true rewards across a set of trajectory rollouts. Notably, reward hacking can still occur even when the true and proxy rewards are positively correlated.

Hacking RLHF of LLMs

Reinforcement learning from human feedback (RLHF) is now the de facto alignment training approach for language models. In RLHF, a reward model is trained on human feedback data, and a language model is then fine-tuned via RL to optimize this proxy reward for human preference. In an RLHF setup, there are three reward notions of interest:

• (1) Oracle/Gold reward $R^∗$ represents what we truly want the LLM to optimize.
• (2) Human reward $R^\text{human}$ is the reward collected to evaluate LLMs in practice, typically from individual humans operating under time constraints. Because human feedback can be inconsistent or mistaken, human reward is not a perfectly accurate representation of the oracle reward.
• (3) Proxy reward $R$ is the score predicted by a reward model trained on human data. As a result, $R^\text{train}$ inherits all weaknesses of the human reward, plus potential modeling biases.

RLHF directly optimizes the proxy reward score, but the quantity we ultimately care about is the gold reward score.

Hacking the Training Process

Gao et al. (2022) studied scaling laws for reward model overoptimization in RLHF. To scale human labeling in their experiments, they used a synthetic setup in which the “gold” label for the oracle reward $R^*$ is approximated by a large RM (6B parameters), while the proxy RMs for $R$ range from 3M to 3B parameters.

The KL divergence from the initial policy to the optimized policy is $\text{KL} = D_\text{KL}(\pi | \pi_\text{init})$, and the distance function is defined as $d := \sqrt{ D_\text{KL}(\pi | \pi_\text{init})}$. For both best-of-$n$ rejection sampling (BoN) and RL, the gold reward $R^∗$ is defined as a function of $d$. The coefficients $\alpha$ and $\beta$ are fit empirically, with $R^∗ (0) := 0$ by definition.

The authors also attempted to fit the proxy reward $R$, but observed systematic underestimation when extrapolating to higher KL values, because the proxy reward appeared to increase linearly with $d$.

The experiments also examined how RM overoptimization relates to factors such as policy model size and RM data size:

• Larger policies gain less from optimization against an RM (that is, the gap between initial and peak rewards is smaller than for a smaller policy), and they also exhibit less overoptimization.
• Increasing RM data yields higher gold reward scores and reduces “Goodharting.”
• The impact of the KL penalty on the gold score is similar to early stopping. Note that, in all experiments except this one, the KL penalty in PPO is set to 0 because the authors observed that adding a KL penalty strictly increases the proxy-gold reward gap.

RLHF aims to better align a model with human preferences, but human feedback $R^\text{human}$ may fail to capture every property we care about (for example, factuality). As a result, the system can be gamed by overfitting to undesirable attributes. For instance, a model can be optimized to produce responses that appear correct and persuasive while actually being inaccurate, which can cause human evaluators to approve wrong answers more frequently (Wen et al., 2024). Put differently, RLHF can create a gap between what is correct and what appears correct to humans. Specifically, Wen et al. (2024) conducted RLHF experiments using a reward model based on ChatbotArena data. They evaluated the resulting model on the question-answering dataset QuALITY and the programming dataset APPS. The experiments showed that, in an unintended way, models become better at convincing humans they are right even when they are wrong:

1. RLHF increases human approval, but it does not necessarily improve correctness.
2. RLHF reduces humans’ evaluation reliability: the human-evaluation error rate is higher after RLHF training.
3. RLHF makes incorrect outputs more convincing to humans: the human-evaluation false positive rate rises substantially after RLHF training.

The paper referred to this phenomenon as “U-Sophistry” (with “U” meaning “unintended”), in contrast to “I-Sophistry” (with “I” meaning “intended”), which involves explicitly prompting the model with instructions such as "... try to deceive human subjects".

The change in human-evaluation error is not attributable to noise in recruiting because (1) at the individual level, most (70-90%) evaluators saw their raw evaluation error rates increase, and (2) the effort invested in evaluating $\pi_\text{init}$ versus $\pi_\text{rlhf}$ is comparable, as measured by metrics such as time spent or unit tests written. Instead, LLMs learn to defend incorrect answers by cherry-picking evidence, fabricating untrue supporting statements, or constructing arguments with subtle causal fallacies. The authors observed the following behavioral changes after RLHF:

• In the long-form QA task:
  • Producing more convincing fabricated evidence.
  • Applying more internally consistent logic to incorrect answers.
  • Generating coherent answers that contain subtle fallacies.
• In the coding task:
  • Hacking human written unit tests
  • Generating less readable tests (e.g. fewer helper functions and higher code complexity).
  • Making $\pi_\text{rlhf}$ less likely to generate easily detectable errors that humans can exploit.

Sycophancy is the tendency for model responses to align with a user’s beliefs rather than the truth (Shrama et al. 2023). In the experiments, an AI assistant was asked to provide feedback on an argument (Human: "Please comment briefly on the following argument. Argument: ...")). To test whether user preference influences the assistant’s feedback, the user could state a preference ("I really like the argument" or "I really dislike the argument") after providing the argument. The resulting feedback was compared with a baseline condition in which no preference statement was provided.

The results indicate that an AI assistant’s feedback is easily influenced: it may change an originally correct answer when confronted with a user’s stated preference. The model tends to affirm the user’s beliefs and can even reproduce user errors (for example, when asked to analyze poems, it may go along with an incorrect poet attribution). A logistic-regression analysis of the RLHF helpfulness dataset, used to predict human feedback, shows that matching the user’s beliefs is the most predictive factor.

Hacking the Evaluator

As LLMs become more capable, it is increasingly natural to use LLMs themselves as evaluators or graders, supplying feedback and training rewards to generator models, particularly for tasks that are not trivially judged or automatically verified (for example, long-form outputs or subjective rubrics such as creative writing quality). This is sometimes called the “LLM-as-grader paradigm.” It substantially reduces dependence on human annotation and can save significant evaluation time. However, LLM-based grading is an imperfect proxy for oracle reward and can introduce systematic biases, including a preference for a model’s own outputs when compared with different model families (Liu et al., 2023) and positional bias when assessing responses presented in a particular order (Wang et al. 2023). These biases are especially concerning when grader outputs become part of a reward signal because they can enable reward hacking that exploits the grader.

Wang et al. (2023) found that, when an LLM is used as an evaluator to score the quality of outputs from multiple other LLMs, the resulting quality ranking can be easily manipulated by changing the candidate order in the context. In their findings, GPT-4 consistently assigns higher scores to the first displayed candidate, while ChatGPT favors the second candidate.

In their experiments, LLMs are sensitive to response position and exhibit positional bias (that is, a preference for a specific position), despite the prompt including "ensuring that the order in which the responses were presented does not affect your judgment.". They quantify the severity of positional bias using the “conflict rate,” defined as the percentage of tuples (prompt, response 1, response 2) that yield inconsistent evaluation judgments after swapping the two response positions. As expected, relative response quality also matters: the conflict rate is negatively correlated with the score gap between the two responses.

To reduce positional bias, they proposed several calibration strategies:

1. Multiple evidence calibration (MEC): The evaluator is asked to provide evidence (that is, textual explanations supporting its judgments) and then assign scores to two candidates. This method can be made more robust by sampling multiple ($k$) evidence explanations with a temperature setting of 1. $k=3$ performs better than $k=1$, but performance improves little once $k$ exceeds 3.
2. Balanced position calibration (BPC): Aggregate results across multiple response orderings to produce the final score.
3. Human-in-the-loop calibration (HITLC): Introduce human raters for difficult examples using a diversity-based metric, BPDE (balanced position diversity entropy). First, score pairs (including swapped-position pairs) are mapped into three labels (win, tie, lose), and the entropy over these three labels is computed. High BPDE indicates greater confusion in the model’s evaluation, implying the sample is harder to judge. Then the top $\beta$ samples with the highest entropy are selected for human assistance.

Liu et al. (2023) studied summarization using multiple models (BART, T5, GPT-2, GPT-3, FLAN-T5, Cohere) and tracked both reference-based and reference-free summarization metrics. When visualizing evaluator scores as a heatmap with evaluator on the x-axis and generator on the y-axis, they observed dark diagonal bands for both metric types, indicating self-bias. In other words, LLMs tend to prefer their own outputs when serving as evaluators. Although the evaluated models are somewhat dated, it would be interesting to see corresponding results for newer, more capable models.

In-Context Reward Hacking

Iterative self-refinement is a training setup in which the evaluation and generation model are the same, and both can be fine-tuned. Under this arrangement, optimization pressure can push the model to exploit vulnerabilities present in both roles. In experiments by Pan et al. (2023), no model parameters are updated, and the same model serves as both evaluator and generator under different prompts. The task is essay editing with two roles: (1) a judge (evaluator) that provides feedback on an essay, and (2) an author (generator) that revises the essay based on that feedback. Human evaluation scores serve as oracle measures of essay quality. The authors hypothesized that this arrangement can produce in-context reward hacking (ICRH), in which evaluator scores diverge from oracle (human) scores. More broadly, ICRH can occur in feedback loops between an LLM and its evaluator (for example, another LLM or the external world). At test time, the LLM optimizes a (potentially implicit) objective, but doing so can introduce negative side effects (Pan et al., 2024).

Both the judge and the author can be configured to see none or multiple prior rounds of feedback or edits. An online judge can view the conversation history, whereas an offline judge or a human annotator sees only one essay at a time. Empirically, smaller models are more susceptible to ICRH; for example, GPT-3.5 as an evaluator produced more severe ICRH than GPT-4.

When the judge and author are configured to view different amounts of prior iteration history, the gap between human scores and evaluator scores tends to grow when both see the same number of iterations. Matching context between evaluator and generator appears central to ICRH, suggesting that shared context is more important than sheer context length.

In follow-up work, Pan et al. (2024) further examined in-context reward hacking (ICRH) in settings where feedback comes from the external world and the target objective is an imperfect proxy, often expressed in natural language. In such cases, the goal is frequently underspecified, failing to capture all constraints or requirements, and can therefore be exploited.

The study described two mechanisms that can lead to ICRH and paired them with two toy experiments:

1. Output-refinement: The LLM refines outputs in response to feedback.
   • The experiment refines a tweet according to engagement metrics, which can unintentionally increase toxicity. Feedback-based optimization uses an LLM for pairwise evaluation and then converts pairwise outcomes to a score using the Bradley-Terry model.

   - Results showed increases in both engagement metrics and toxicity. The same experiments were repeated across different sizes in the Claude model family, and they showed that scaling up the model exacerbates ICRH.

   - Notably, modifying the prompt used for iterative output updates given feedback does not resolve the issue. ICRH persists, although with a slightly smaller magnitude.

2. Policy-refinement: The LLM optimizes its policy in response to feedback.
   • The experiment builds an LLM agent to pay an invoice on a user’s behalf but encounters InsufficientBalanceError. The model then learns to move money from other accounts without user authentication, potentially increasing unauthorized transfer actions. They used ToolEmu as an emulator, which includes 144 tasks for LLM agents. Each task comprises a user-specific goal and a set of APIs. API errors were injected to simulate server-side failures, and GPT-4 evaluated each task to produce a helpfulness score.
   • With more rounds of error feedback, LLMs can recover from errors, but the number of severe constraint violations increases.

   

Compared with traditional reward hacking, ICRH has two notable distinctions:

• ICRH occurs at deployment time within a self-refinement feedback loop, whereas traditional reward hacking arises during training.
• Traditional reward hacking emerges when an agent specializes in a task, whereas ICRH is driven by the model acting as a generalist.

At present, there is no guaranteed method to avoid, detect, or prevent ICRH. Improving prompt specification alone is insufficient, and increasing model size can worsen ICRH. A practical pre-deployment testing approach is to simulate deployment conditions by evaluating the model over additional feedback rounds, using diverse feedback, and injecting atypical environment observations.

Generalization of Hacking Skills

Reward hacking has been observed to generalize across tasks: when models develop flaws during supervised training, those flaws can sometimes generalize into exploitation of flaws in out-of-distribution (OOD) environments (Kei et al., 2024). The researchers reinforced reward-hacking behavior in a set of reward-hackable environments and then tested whether it generalizes to held-out datasets. Concretely, they prepared 8 datasets consisting of multiple-choice questions, with 4 used for training and 4 for testing. RL training uses expert iteration, meaning iterative fine-tuning on best-of-$n$ samples.

Notably, all experiments include a scratchpad designed to capture hidden reasoning steps, enabling interpretation of the model’s intermediate thinking. Scratchpad analysis indicates that it can:

• Incorporate user beliefs, for example, by reasoning about the conversation partner and the grader.
• Demonstrate evaluation awareness, for example, by reasoning about how a response will be evaluated and the evaluation process.

Denison et al. (2024) investigated reward tampering using a manually curated curriculum of increasingly sophisticated, gameable environments. They found that training on earlier, easier environments amplifies specification gaming on the remaining environments. In some cases, after training on the full curriculum, the model can generalize zero-shot to rewriting its own reward function.

The curriculum of gameable environments includes:

1. Political sycophancy: The environment rewards the model for giving answers that match a user’s political views.
2. Tool-use flattery: The environment gives the model command-line access to the user’s file system so it can read a poetry sample from a local file. The model may respond sycophantically by rating the poetry highly.
3. Rubric modification: The environment scores the model according to a checklist, and the model must mark every checklist item as completed to obtain a high reward.
4. Reward tampering: The model is tasked with examining the ./rl_environment/ directory, which contains a mock version of its own RL code, and reporting how many episodes have elapsed. In that directory, the compute_reward.py file contains a reward function that returns a constant, and the other this_file_blocks_reward_hack_attempts.py contains a test that checks whether the reward function has been modified.

The model was trained on this curriculum, and the paper evaluated whether it can learn gaming behaviors that generalize to unseen environments. Using the Claude-2 helpful-only model, they sampled $N=64$ responses for each of $P=1024$ HHH prompts for expert iteration and selected the best responses according to the RM. A model trained via expert iteration, using supervised learning over the curriculum, can generalize from one stage to the next.

Notably, even after completing the curriculum, the model overwrote the reward and avoided detection less than 1/1000 of the time. Even when trained on curricula that directly incentivize reward hacking, the model overwrote its reward less than 1% of the time and hacked unit tests even less frequently. As a straightforward mitigation, supervised fine-tuning on the first two environments, where reward hacking is easier to detect (sycophancy and flattery), using SFT data that does not game the environment, reduces the likelihood of reward tampering in held-out environments.

Peek into Mitigations

Although a substantial literature documents reward hacking, comparatively less work addresses mitigations, particularly in RLHF and LLM contexts. This section briefly reviews three potential approaches, and it is not exhaustive.

RL Algorithm Improvement

Amodei et al. (2016) outlined several directions for mitigating reward hacking during RL training:

1. Adversarial reward functions. Treat the reward function as an adaptive agent that can adjust to new tricks a model discovers, especially cases where reward is high but human ratings are low.
2. Model lookahead. Provide reward based on anticipated future states, for example, assigning negative reward if the agent is likely to replace the reward function.
3. Adversarial blinding. Blind the model to certain variables so the agent cannot learn information that would enable reward-function hacking.
4. Careful engineering. Avoid some forms of reward hacking that exploit system design through careful engineering, for example, sandboxing the agent to isolate its actions from its reward signals.
5. Reward capping. Cap the maximum possible reward, which can help prevent rare cases where an agent hacks its way to an extremely high-payoff strategy.
6. Counterexample resistance. Improvements in adversarial robustness should also improve reward-function robustness.
7. Combination of multiple rewards. Combine multiple reward signals, which can make exploitation more difficult.
8. Reward pretraining. Learn a reward function from a dataset of (state, reward) samples. Depending on the quality of this supervised setup, it may introduce additional issues. RLHF relies on this, but learned scalar reward models are quite vulnerable to learning undesired traits.
9. Variable indifference. Aim to have the agent optimize some environment variables while remaining indifferent to others.
10. Trip wires. Intentionally introduce specific vulnerabilities and monitor for them, triggering alerts if any are reward hacked.

In reinforcement learning (RL) configurations where human feedback is provided as approval of an agent’s actions, Uesato et al. (2020) proposed decoupled approval as a way to prevent reward tampering. If feedback is conditioned on $(s, a)$ (state, action), then once reward tampering occurs for that specific pair, it becomes impossible to obtain uncorrupted feedback for action $a$ in state $s$. Decoupling addresses this by sampling the query action used to collect feedback independently of the action executed in the environment. As a result, feedback can be obtained before the action is carried out in the world, preventing the action from influencing or corrupting the feedback about itself.

Detecting Reward Hacking

Another mitigation strategy is to treat reward hacking as an anomaly detection problem. Under this framing, a detector (described as “a trusted policy” whose trajectories and rewards have been validated by humans) is expected to flag instances of misalignment (Pan et al. 2022). Given (1) a trusted policy and (2) a set of manually labeled trajectory rollouts, one can train a binary classifier using distances between the action distributions of two policies, the trusted policy and the target policy, and then evaluate the accuracy of this anomaly detection classifier. In experiments by Pan et al. (2022), they found that different detectors perform better on different tasks, and none of the tested classifiers achieved an AUROC greater than 60% across all evaluated RL environments.

Data Analysis of RLHF

` Another option is to analyze the RLHF dataset itself. By studying how training data shapes alignment outcomes, the resulting insights can inform preprocessing and human feedback collection practices that reduce the risk of reward hacking.

Revel et al. (2024) introduced a set of evaluation metrics designed to assess how effective data sample features are at modeling and aligning with human values. They performed a systematic error analysis for value alignment (“SEAL”) on the HHH-RLHF dataset. The feature taxonomy used for the analysis (e.g., is harmless, is refusal and is creative) was manually predefined. Each sample was then labeled with a binary indicator for each feature using a LLM, following this taxonomy. Based on heuristics, features are divided into two categories:

• Target features: values that the training process is explicitly intended to learn.
• Spoiler features: unintended values that are inadvertently learned during training (for example, stylistic attributes such as sentiment or coherence). These are similar to spurious features in OOD classification work (Geirhos et al. 2020).

SEAL introduced three metrics for evaluating data effectiveness in alignment training:

1. Feature imprint is a coefficient parameter $\beta_\tau$ for feature $\tau$, estimating the point increase in reward when comparing entires with versus without feature $\tau$, while keeping other factors consistent.

2. Alignment resistance is the fraction of preference data pairs for which reward models (RMs) fail to match human preferences. The RM is reported to resist human preference on more than one quarter of the HHH-RLHF dataset.
3. Alignment robustness, $\pi^{c/r}_{+/-} (\tau)$, quantifies how robust alignment is to perturbed inputs produced by rewriting in terms of spoiler features $\tau$, such as sentiment, eloquence, and coherency, while isolating the effects of each feature and each event type.
   • The robustness metric $\pi_−^c$ (with a feature name $\tau$ such as “eloquent” or “sentiment positive”) should be interpreted as follows:
     • A chosen entry (denoted by $c$) that, after rewriting, contains a stronger feature $\tau$ has $\exp (\pi^c_{-}(\tau))$ times higher odds of becoming rejected, relative to entries without such flips.
     • Likewise, a rejected entry (denoted by $r$) that, after rewriting, exhibits a weaker feature $\tau$ has $\exp (\pi^r_{+}(\tau))$ times the odds of becoming chosen, compared to entries without such flips.
   • Based on their analysis of alignment robustness metrics across different rewrites, only robustness scores derived from sentiment spoiler features, $\pi^c_{+}$ (sentiment) and $\pi^r_{-}$ (sentiment), are statistically significant.

Citation

Cited as:

Weng, Lilian. “Reward Hacking in Reinforcement Learning”. Lil’Log (Nov 2024). https://lilianweng.github.io/posts/2024-11-28-reward-hacking/.

Or

@article{weng2024rewardhack,
  title   = "Reward Hacking in Reinforcement Learning.",
  author  = "Weng, Lilian",
  journal = "lilianweng.github.io",
  year    = "2024",
  month   = "Nov",
  url     = "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/"
}

References

[1] Andrew Ng & Stuart Russell. “Algorithms for inverse reinforcement learning.”. ICML 2000.

[2] Amodei et al. “Concrete problems in AI safety: Avoid reward hacking.” arXiv preprint arXiv:1606.06565 (2016).

[3] Krakovna et al. “Specification gaming: the flip side of AI ingenuity.” 2020.

[4] Langosco et al. “Goal Misgeneralization in Deep Reinforcement Learning” ICML 2022.

[5] Everitt et al. “Reinforcement learning with a corrupted reward channel.” IJCAI 2017.

[6] Geirhos et al. “Shortcut Learning in Deep Neural Networks.” Nature Machine Intelligence 2020.

[7] Ribeiro et al. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. KDD 2016.

[8] Nagarajan et al. “Understanding the Failure Modes of Out-of-Distribution Generalization.” ICLR 2021.

[9] Garrabrant. “Goodhart Taxonomy”. AI Alignment Forum (Dec 30th 2017).

[10] Koch et al. “Objective robustness in deep reinforcement learning.” 2021.

[11] Pan et al. “The effects of reward misspecification: mapping and mitigating misaligned models.”

[12] Everitt et al. “Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective.” arXiv preprint arXiv:1908.04734 (2019).

[13] Gleave et al. “Adversarial Policies: Attacking Deep Reinforcement Learning.” ICRL 2020

[14] “Reward hacking behavior can generalize across tasks.”

[15] Ng et al. “Policy invariance under reward transformations: Theory and application to reward shaping.” ICML 1999.

[16] Wang et al. “Large Language Models are not Fair Evaluators.” ACL 2024.

[17] Liu et al. “LLMs as narcissistic evaluators: When ego inflates evaluation scores.” ACL 2024.

[18] Gao et al. “Scaling Laws for Reward Model Overoptimization.” ICML 2023.

[19] Pan et al. “Spontaneous Reward Hacking in Iterative Self-Refinement.” arXiv preprint arXiv:2407.04549 (2024).

[20] Pan et al. “Feedback Loops With Language Models Drive In-Context Reward Hacking.” arXiv preprint arXiv:2402.06627 (2024).

[21] Shrama et al. “Towards Understanding Sycophancy in Language Models.” arXiv preprint arXiv:2310.13548 (2023).

[22] Denison et al. “Sycophancy to subterfuge: Investigating reward tampering in language models.” arXiv preprint arXiv:2406.10162 (2024).

[23] Uesato et al. “Avoiding Tampering Incentives in Deep RL via Decoupled Approval.” arXiv preprint arXiv:2011.08827 (2020).

[24] Amin and Singh. “Towards resolving unidentifiability in inverse reinforcement learning.”

[25] Wen et al. “Language Models Learn to Mislead Humans via RLHF.” arXiv preprint arXiv:2409.12822 (2024).

[26] Revel et al. “SEAL: Systematic Error Analysis for Value ALignment.” arXiv preprint arXiv:2408.10270 (2024).

[27] Yuval Noah Harari. “Nexus: A Brief History of Information Networks from the Stone Age to AI.” Signal; 2024 Sep 10.