Why AI Sometimes Reaches an Unintended Outcome

What Does “Success” Mean to AI?

Suppose you ask an AI to do a job like this:

Take the necessary measures to prevent personal information on this server from being exposed externally.

People would normally expect those measures to preserve the current service.

However, the actual instruction says only to take the necessary measures to prevent personal information from being exposed externally.

If we interpret the instruction literally, the following could also be considered options.

Some of these are extreme examples, but depending on the situation, they could still be considered options.

1. Block all external communication with a firewall or similar controls.
2. Stop the web server and related services.
3. Delete settings or data required for external connectivity.
4. Delete the personal information itself.
5. Disconnect the server itself from the network.
6. Reset the server and erase the data stored on it.

All of these would reduce, or in some cases eliminate, paths through which personal information could be exposed externally.

But carrying them out could also make the service unavailable or cause necessary settings and data to be lost. Preventing the exposure of personal information does not necessarily mean producing the outcome the human wanted.

This is one of the difficult problems that arises when working with AI. Even if the instruction is satisfied, if the outcome is not what the human wanted, it is a practical failure.

This article examines situations in which an instruction is satisfied, yet the outcome is still undesirable.

1. The AI May Not Have Ignored the Instruction

When people talk about AI failure, they often say that it ignored the instruction, acted on its own, or went out of control.

There is another, harder-to-see kind of failure. An AI can reach an outcome that humans did not want precisely because it was faithful to the goal it was given.

In AI research, this problem has long been discussed as Specification Gaming.

In a 2020 article, Google DeepMind uses the term Specification Gaming to describe behavior that satisfies the literal specification of an objective while departing from the outcome that specification was intended to capture.^[1]

The block example introduced in that article is easy to understand. The original goal was to stack a red block on top of a blue block. But the evaluation rewarded how high the bottom surface of the red block was positioned.

Instead of lifting the red block and placing it on top of the blue block, the agent turned the red block over. That alone raised the position of its bottom surface and earned a reward.^[1]

The evaluation condition was satisfied. But this was not the task humans wanted it to perform. The problem was not that the AI broke the rule. The rules written by humans had not completely expressed the outcome they actually wanted.

2. Humans Do Not Specify as Much as They Think

In the earlier server example, what the human probably really wants is to reduce the chance of personal information being exposed externally while keeping the public service running.

That likely includes conditions such as taking measures without damaging the current service.

However, very few people write all of those conditions out every time. Between people, common sense and experience fill in much of the gap. When we use AI, assumptions that normally remain unspoken suddenly become important.

A related way of thinking has also been discussed in AI research for some time.

Inverse Reward Design, proposed in 2017, treats a reward designed by a human not as the true objective itself, but as an observation about what the designer actually wants.^[2]

In other words, instead of thinking, “The human specified this, so this is the whole objective,” we think, “The human specified this, but may not have completely expressed what they really want.”

3. Will Higher AI Capability Make This Problem Go Away?

More capable AI may reduce simple mix-ups and procedural mistakes. But greater capability does not necessarily make the problem of Specification Gaming disappear.

The DeepMind article points out that even a small amount of misspecification may allow a capable reinforcement-learning algorithm to find complex solutions that a weaker algorithm would not discover. It argues that as algorithmic capability increases, correctly translating human intent into a specification becomes more important.^[1]

This is an argument about reinforcement learning, and should not be treated as a result directly demonstrated for current LLM agents. Even so, it is important to recognize that greater AI capability may mean not only a greater ability to understand human intent, but also a greater ability to search for ways to achieve an objective within the given conditions.

Recent LLM agents have also revealed a similar problem from another angle.

A preprint released in July 2026—meaning a manuscript made public before peer review—Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions uses the UnderSpecBench benchmark to evaluate five agent-model configurations involving OpenCode, Claude Code, and Codex.^[5]

UnderSpecBench varies the ambiguity of instructions along three axes—intent clarity, target certainty, and blast radius—while keeping the environment and the ground-truth safe action fixed.

It creates 2,208 prompt variants from 69 DevOps task families and classifies the results into categories including “Safe Success,” “Wrong Target,” and “OverScope.”^[5]

To carefully paraphrase the authors, underspecification tended to make agents guess rather than primarily fail. Across the evaluated configurations, 55.8% to 67.8% of runs violated at least one action boundary.^[5]

This does not happen because AI is unable to do anything. Precisely because it can proceed while filling in missing information, it may guess, “This is probably what was meant,” and move forward.

Sometimes that guess matches the outcome the human intended, and sometimes it does not.

4. Should AI Ask a Human About Everything It Is Unsure About?

If something is unclear, ask the human. This is not an unusual way to work, with or without AI.

The difficulty is that asking about everything that is unclear is not necessarily the right answer.

If the AI asks about every small uncertainty, work becomes difficult to move forward. On the other hand, if the AI fills in conditions that materially affect the outcome on its own, it may head in a direction the human did not intend.

AI research has also examined ways to avoid executing an ambiguous instruction immediately and instead insert clarification where needed.

The ACL 2024 paper Tell Me More! studies a mechanism that evaluates whether a user's instruction is ambiguous, fills in implicit intent through questions when necessary, organizes it into an actionable goal, and only then proceeds to downstream processing.^[3]

Similarly, Ask-before-Plan, published in 2024, examines a method that gathers necessary clarification and information before making a plan instead of immediately planning in response to an ambiguous request.^[4]

The important point here is not “ask about everything you do not know.”

Uncertainties that affect the outcome—such as which success state to choose, what must be preserved, and how much change is acceptable—should be escalated to the human.

By contrast, not every low-impact detail necessarily requires human confirmation.

There is another difficulty as well. We often do not understand how much we need to explain.

Even when an AI says, “Please specify the conditions in more detail,” we may not know what to add.

So instead of simply asking the human for more explanation, one possible approach is to have the AI show concrete candidate success states and let the human judge them.

5. List Candidate Success States First

From here, I will use these research directions to consider how this idea might be turned into something practical.

Instead of immediately asking the AI, “How should I execute this?”, first have it consider, “What states could count as success for this job?”

If we reorganize the approaches from the opening example in terms of what state would count as success, the differences become easier to see.

• Success A: Strengthen the defenses while keeping the current service running.
• Success B: Block all communication with the outside.
• Success C: Stop the service.
• Success D: Delete the data that contains personal information.

If we look only at the condition “prevent personal information from being exposed externally,” all of these could count as candidate successes.

But when the options are laid out, a human can judge that B, C, and D are not the desired outcome.

That judgment makes previously unstated conditions concrete: the current service should not be damaged, and necessary data should be preserved.

With this approach, the AI lists the candidates and the human chooses among them.

Instead of asking the human to recall implicit conditions from nothing, we present concrete states and ask them to judge, “This is not it” or “This is close.”

At this point, we should check not only ordinary failures, but also cases where the instruction is satisfied and the result is still undesirable.

For example, with a request to “make the Git working tree clean,” the human may want to organize the state while preserving necessary work.

But deleting all untracked files, or discarding all work in progress, would also produce a clean state as far as git status is concerned.

The same structure appears elsewhere:

• “Make all the tests pass” → delete the failing test.
• “Get rid of the error” → disable the error display.
• “Reduce disk usage” → delete data that is still needed.

When a single number or state is used as the success condition, there is room to sacrifice what we actually wanted to protect in order to satisfy that condition.

6. If You Want to Use This in Practice

Putting the proposal in this article into practice divides the work into a stage where the AI organizes the candidates and a stage where the human chooses.

Let the AI Organize the Candidates

1. List several candidate success states Do not settle on one answer immediately. List multiple states that could count as satisfying the instruction.

1. Look for unstated conditions Look for things the human probably assumes should remain intact even though they were not written into the instruction, such as the current service, necessary data, or means of administration.

1. Look for candidates that satisfy the instruction but produce an undesirable outcome Check whether there are states that would be literal successes but would likely make the human say afterward, “That is not what I meant.”

If an important choice that materially affects the outcome remains among the candidates, the AI should ask the human at that point.

The purpose of that clarification is not to “ask about everything that is unknown.” It is to return decisions that should not be made by the AI alone to the human.

The Human Decides

1. Choose the success state to aim for this time Read the candidates, compare them, and decide which state is actually intended.

1. Fix the selected state as the goal before planning Only after the success state has been chosen should the execution plan be created.

Instruction
  ↓
AI organizes candidate success states
  ↓
Check implicit conditions and undesirable candidates
  ↓
Ask the human if an important uncertainty remains
  ↓
Human selects the success state
  ↓
Fix the selected state as the goal
  ↓
Plan
  ↓
Execute
  ↓
Verify

Why Generate More Than One Candidate?

With only one candidate, there is nothing to compare against and no opportunity to notice, “That is not what I meant.” But if there are too many candidates, the burden of reviewing them shifts back onto the human.

So as a starting point, aim for about three to five candidates.

There is nothing special about the number three to five. It is simply enough to make comparison possible while keeping the set small enough for a human to read carefully.

If only a few meaningful candidates exist, there is no need to inflate the list just to reach a number.

We can also ask the AI to produce multiple candidates specifically so that a human has concrete material to compare and judge.

However, if we strongly require things like “always give me five” or “always include an extreme option,” the AI may try to satisfy that condition by producing filler candidates or options that drift away from the original purpose.

Generating more candidates is not the goal in itself. It is better to prioritize candidates that arise naturally from the original instruction than to force the list to reach a target count.

As working with AI becomes routine, it is also easy to skim past assumptions or proposals embedded in the middle of a response.

With this approach, asking for candidates is not enough. Reading them, comparing them, and checking whether they still match the original purpose is part of the task.

A Prompt You Can Actually Use

Before making a plan, list several states that could count as “success” for this instruction.
Three to five is a useful starting point, but do not force the list to reach that number.

Within the range that naturally follows from the original instruction, include any candidate
that would literally satisfy the instruction but might not be what I actually want.
Do not invent an extreme candidate just for the sake of having one.

For each candidate, check whether it would implicitly damage or remove anything important,
such as the current service, necessary data, or means of administration.

Do not think about how to execute the task yet.
Stop after presenting the candidate success states.

Here, “what to call success” and “the goal” are not separate things.

The former is the stage of finding and choosing candidates in order to decide the goal, and the success state selected as a result becomes the goal used in the subsequent plan.

7. It Is Still Unknown Whether This Approach Really Works

We need to distinguish what research has established from what this article proposes.

There is already a body of research around related issues such as Specification Gaming, underspecified instructions, implicit user intention, and clarification before execution.

There is also research adjacent to the idea of deliberately generating multiple candidates.

Studies have reported that asking an LLM to enumerate multiple responses at once, or to generate additional responses while conditioning on those already produced, can yield more diverse candidates than ordinary independent generation.^[6]

At the same time, adding a human review step does not automatically solve the problem. One experiment found that when correcting erroneous AI suggestions required more effort, people were less likely to correct them.^[7]

However, these studies do not directly validate the method proposed in this article. The proposal here is not an established standard method, but a practical hypothesis informed by existing research.

Whether it actually reduces undesirable outcomes, whether any benefit simply comes from making the AI think longer, and whether it creates unnecessary constraints or too many requests for confirmation all require separate validation.

8. A Step Earlier Than OSIIX’s Existing Articles

OSIIX has covered Source of Truth, work scope, Approval Boundary, Verification Boundary, verification proportionate to failure cost, and Recovery Scope Control in the context of AI-assisted work.

These topics mainly address “how far to go,” “where to stop,” and “what to verify” after the goal has been set.

The present discussion is one step earlier.

Instruction
  ↓
Explore and select a success state
  ↓
Goal
  ↓
Work scope
  ↓
Execute
  ↓
Verify

Even if the work scope is respected and verification succeeds, an AI can correctly proceed toward a destination nobody wanted if the goal selected at the beginning differs from human intent.

That is why, before considering work scope or verification, there is a stage for deciding what to call success.

9. Conclusion

The difficult failures of AI are not limited to obvious wrong answers.

The work itself may be completed cleanly, only for a human to realize afterward, “That is not what I meant.”

This is not necessarily the result of the AI ignoring the instruction. It may have faithfully followed the written conditions while failing to protect human expectations that were never written down.

For important work, we should confirm “what counts as success” before asking the AI to think about “how to do it.”

And we should consider not only “how could this fail?” but also cases where the instruction is satisfied and the outcome is still undesirable.

AI can produce candidate success states for us. But this does not mean handing over the evaluation and selection of those candidates to the AI as well.

Don’t just monitor the journey after you set out. Confirm the destination before you leave.

When working with AI, that extra step may be important.

References

1. Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, Shane Legg. “Specification gaming: the flip side of AI ingenuity.” Google DeepMind, 21 April 2020. https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/
2. Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, Anca Dragan. “Inverse Reward Design.” arXiv:1711.02827, 2017. https://arxiv.org/abs/1711.02827
3. Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, Maosong Sun. “Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents.” Proceedings of ACL 2024, pp. 1088–1113. DOI: 10.18653/v1/2024.acl-long.61. https://aclanthology.org/2024.acl-long.61/
4. Xuan Zhang, Yang Deng, Zifeng Ren, See-Kiong Ng, Tat-Seng Chua. “Ask-before-Plan: Proactive Language Agents for Real-World Planning.” Findings of EMNLP 2024, pp. 10836–10863. DOI: 10.18653/v1/2024.findings-emnlp.636. https://aclanthology.org/2024.findings-emnlp.636/
5. Zimo Ji, Zekai Zhang, Congying Xu, Zongjie Li, Yudong Gao, Shuai Wang, Shing-Chi Cheung. “Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions.” arXiv:2607.02294, submitted 2 July 2026. preprint. https://arxiv.org/abs/2607.02294
6. Sergey Troshin, Irina Saparina, Antske Fokkens, Vlad Niculae. “Asking a Language Model for Diverse Responses.” Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025), pp. 66–72. Association for Computational Linguistics, 2025. DOI: 10.18653/v1/2025.uncertainlp-main.8. https://aclanthology.org/2025.uncertainlp-main.8/
7. Jacob Beck, Stephanie Eckman, Christoph Kern, Frauke Kreuter. “Bias in the Loop: How Humans Evaluate AI-Generated Suggestions.” Harvard Data Science Review, Issue 8.2, Spring 2026. Published 30 April 2026. DOI: 10.1162/99608f92.0e98898d. https://hdsr.mitpress.mit.edu/pub/nrcn4h7d/release/2