Chapter 10 — Problems That Remain in Today's AI

"AI has advanced" and "the problems are gone" are different things

AI has become practically usable for many tasks. But becoming more capable = the problems have gone away does not hold.

became highly capable
    =
the problems are gone

Current AI still has multiple remaining challenges related to capability, evaluation, safety, and explainability, among other things. Rather than covering all of them, this chapter takes up the problems most consistently important for understanding AI-assisted development.

Factuality

Even if generated text is natural and readable, that doesn't mean the content is factually correct. Factuality is the perspective of how well the claims in an output correspond to confirmable facts or reality. What matters here is:

fluency != factuality
confidence wording != factuality

Even if an AI writes with confidence, you cannot know whether the content matches the facts unless it's checked against external facts.

Hallucination

The term Hallucination is also used for generative AI. This word is used to refer to cases where a model's generated output is:

• incorrect
• unsupported by Evidence
• inconsistent with the given context

However, the boundary of this term varies by source and system. Also, Hallucination = every AI error does not hold. It doesn't mean the same thing as intentional deception either.

Grounding does not fully guarantee correctness

Grounding an output using external information or a given document can help reduce ungrounded generation. RAG is sometimes used for this purpose too. However, having grounded something, having used RAG, or citations being present:

grounded
used RAG
has a citation

does not mean every claim is correct. The retrieved source itself may be wrong, or the correspondence between the source and the generated text may be incomplete, leaving incorrect content behind.

The difficulty of Evaluation

Understanding AI's performance requires Evaluation. Evaluation uses things such as:

• tasks
• datasets
• metrics
• Human judgment
• Benchmarks

to examine a model's or system's performance. But an Evaluation result always comes with conditions attached.

A Benchmark is not the same as overall capability

A Benchmark is convenient for comparing models or systems. But:

Benchmark score
    =
total real-world capability

does not hold. A Benchmark result is performance under that particular task, dataset, metric, and condition. Getting a high score on one fixed Benchmark does not let you generalize to "this model is excellent at every task."

Benchmark Contamination

Evaluation is also related to Training Data. If a benchmark, or information closely related to it, ends up included in Training or adaptation, the independence of the evaluation can be compromised. This is sometimes called Benchmark Contamination. So beyond the Benchmark score itself, it matters to look at:

what was measured
under what conditions
what data was used

Robustness

Robustness is a property indicating how much a model or system can maintain its intended performance or behavior when input, environment, or conditions change. For example, a system whose results swing wildly from a slight change in phrasing might have an issue with Robustness in one sense. However:

Robustness = Security
Robustness = Correctness
Robustness = Safety

does not hold. They may be related, but they are separate problems.

Nondeterminism / Stochasticity

With generative AI, the same input doesn't always produce the same output. Probabilistic processing, such as Sampling, can be involved. So it does not hold that:

same prompt
    ↓
always exactly the same output

This does not automatically mean "AI cannot be trusted." But it is an important property to keep in mind when thinking about testing or reproducibility.

Transparency

For an AI system, it's an important question how much a user can know about things such as:

• which model is being used
• what data was used
• what tools were used
• what limitations exist

This kind of disclosure and visibility is treated as Transparency. However, Transparency = Explainability does not hold.

Explainability

Explainability is about whether a model or system can explain, in a way a human can understand, why it produced that particular output or decision. However:

an explanation exists
    =
the explanation is correct

does not hold either. Nor does:

Explainability = factual correctness

An AI-generated "explanation-sounding piece of text" cannot be treated as a complete account of the internal computation either.

Security

Since an AI system is also a software system, it has Security concerns. That said, this chapter doesn't cover AI Security as a whole. What matters is that:

highly capable
    =
safe

does not hold. Robustness, Factuality, and Security can be related, but they are each separate perspectives.

Tool / Agent Safety

With Tool Use and AI Agents, as seen in Chapter 8, the range in which a model can act on the outside world expands — for example:

read
write
execute
API call

As Tool access increases, capability broadens, but so does the potential scope of impact when something fails. In particular, read-only action and state-changing action need to be distinguished. An operation that only looks at information and an operation that rewrites a file or changes a service's state have different consequences when something goes wrong.

Permission

What a Tool or Agent can do also depends on the permission granted to it. For example, the strength of permission varies as in:

read-only
write
delete
execute
administrative

Being an AI Agent doesn't mean it necessarily needs strong permission — the important idea is only granting the actions that are actually needed. This book doesn't cover specific sandbox architecture or access-control design.

Reversibility

When an AI system performs an external action, whether that action can be undone also matters. For example, reading a file and deleting a file differ in how hard it is to undo them. When thinking about Tool/Agent systems, points such as:

• is it state-changing
• is it reversible
• is Human approval needed

matter. How these are actually handled in practice is covered in Chapter 13's discussion of AI-assisted development.

Resource

AI also has a physical resource problem. As seen in Chapter 9, a model's Training and Inference involve things such as:

• compute
• memory
• network
• electrical power
• cooling

When thinking about a large-scale AI system, resource use becomes an important constraint alongside performance. That said, this chapter doesn't center its teaching on figures like the latest power-consumption rankings or vendor efficiency comparisons, which change over time.

Problems can't be reduced to one score

For an AI system, a single result like:

the Benchmark score is high

doesn't let you judge things such as:

• Factuality
• Robustness
• Security
• Transparency
• Explainability
• Agent Safety
• Resource efficiency

because they're each a different question. When evaluating an AI system, you need to keep asking separately:

"what specifically is being called good here?"

Unresolved problems are a bit different from UNKNOWN

The "unresolved problems" referred to in this chapter don't mean "nothing at all is understood." There is substantial research and technology around Factuality, Robustness, Evaluation, and more. But it means the problems are not fully solved, and depending on the system and the use case, ongoing effort is still required.

On the other hand, when a specific vendor's undisclosed internal structure is unknown, this book treats that as UNKNOWN / UNDISCLOSED. The two are kept distinct.

Chapter summary

Current AI has at least the following problems:

• Factuality → whether the output content matches the facts
• Hallucination → the problem of generating content that doesn't match facts, Evidence, or given Context
• Evaluation / Benchmark limits → an evaluation result needs to be read within the scope of the task, data, and conditions used
• Robustness → how well performance or behavior is maintained when input or conditions change
• Nondeterminism / Stochasticity → the property that the same input doesn't always produce the same output
• Transparency → how much can be known from the outside about the model, data, tool use, and more
• Explainability → whether the reason for an output or decision can be explained in a way a human can understand
• Security → safety issues related to misuse, permissions, data, and external operations of an AI system
• Tool / Agent Safety → how to contain the impact of misoperation or permission overreach when AI can act externally
• Resource constraints → physical constraints such as compute, memory, power, and cooling

What matters is:

• a natural-sounding sentence is not necessarily factually correct
• Hallucination does not mean every AI error
• Grounding or RAG is not a complete correctness guarantee
• a Benchmark result is a result limited to specific conditions
• don't generalize from one Benchmark to overall capability
• don't treat Robustness, Security, and Factuality as the same concept
• don't treat Transparency and Explainability as the same concept
• an explanation existing and that explanation being correct are separate things
• Tool access can broaden risk at the same time as capability
• separate read-only actions from state-changing actions
• be conscious of permission and reversibility
• AI is also subject to physical resource constraints
• leave a specific vendor's undisclosed state as UNKNOWN, rather than guessing at it

From the next chapter on, taking these limitations as a premise, we organize how to confirm AI-generated output from the perspective of Claim, Evidence, INFERENCE, and UNKNOWN.