OSIIX

OSIIX Library

When AI Can't Read a Website

A troubleshooting record that separates website-side issues from AI-side fetch controls

Type
Case Study / Engineering Note
Author
mars70
Read in Japanese

1. What happened

In September 2026, while using AI as an aid during debugging, I ran into cases where a publicly accessible website could not be retrieved through an AI web-fetching function.

The same site opened normally in a browser and through ordinary HTTP clients. Similar behavior appeared with more than one target, while other websites could be fetched by the AI without issue.

My first suspicion was the website itself.

There were several possible causes: DNS, IPv4 / IPv6, HTTPS, robots.txt, the web server, or some other part of the delivery path. Rather than assume a cause, I started with the parts I could observe directly and worked through them one by one.

I did not identify the direct cause of the failure.

What the investigation did provide was a practical stopping point: a basis for deciding when it no longer made sense to keep digging into the website simply because the AI could not fetch it.

The checks included DNS, IPv4 / IPv6, GET requests, access from another network, robots.txt, and the web server access log.
When those checks are normal, an AI fetch failure alone is not enough reason to keep investigating the website indefinitely.
Other possibilities, including controls in the AI-side fetch path, can remain part of the troubleshooting scope.

2. Isolating the website side

The examples below replace the actual domain name and server-specific paths with placeholders.

DNS

I started by checking the A and AAAA records.

dig A <target-domain>
dig AAAA <target-domain>

Both IPv4 and IPv6 name resolution worked normally.

At this point, there was no evidence that a simple DNS failure was the primary cause.

Testing IPv4 and IPv6 separately

Next, I checked whether only one address family was failing.

curl -4 -I https://<target-domain>/
curl -6 -I https://<target-domain>/

Both returned normal HTTP responses.

A site can appear to work normally while one path, such as IPv6, is broken. Testing the address families separately helps remove that ambiguity.

Testing GET as well as HEAD

For a quick reachability check, it is common to use HEAD alone. But a successful HEAD request does not guarantee that the GET request used to retrieve the actual page will also succeed.

So I tested both.

curl -I https://<target-domain>/

curl -sS -o /dev/null \
  -w '%{http_code}\n' \
  https://<target-domain>/

Both HEAD and GET returned 200, and the content itself could be retrieved with GET.

That is why I did not treat a successful HEAD request alone as proof that the website path was healthy.

Testing from another network

Being able to reach the site from the server itself, or from the same network, is not enough to show that it is publicly reachable in the same way from outside.

I therefore repeated HEAD and GET from an ordinary client on a separate external network.

Those requests also succeeded.

At minimum, this ruled out the site being reachable only from inside the server.

robots.txt

I also checked robots.txt.

curl -fsS https://<target-domain>/robots.txt

For targets that had a robots.txt file, I did not find an explicit rule denying general crawlers.

One example looked like this:

User-agent: *
Allow: /

Another target did not have a robots.txt file and returned 404.

A 404 response for robots.txt does not, by itself, mean that web fetching is prohibited. In the cases I checked, robots.txt did not explain the failure.

3. Checking reachability with the access log

The web server access log provided the clearest clue in this troubleshooting process.

I monitored the log in real time.

sudo tail -F <web-server-access-log>

First, I sent HEAD and GET requests from a client on another network.

Those normal external requests appeared in the access log.

"HEAD / HTTP/1.1" 200 ... "curl/..."
"GET / HTTP/1.1"  200 ... "curl/..."

That established that, at least for ordinary external HTTP requests, the log I was watching was capable of showing incoming traffic.

With the same log still being monitored, I then asked the AI web-fetching function to retrieve the same site.

The AI still failed to fetch it. For the targets where I could inspect the relevant log, however, I could not find an HTTP request at the corresponding time that I could reasonably match to the AI fetch attempt.

There is an important limit to that observation.

The absence of an entry in the access log does not prove that no network traffic from the AI service reached the server at all.

What I could say was narrower:

the access log that recorded normal external requests did not contain a request corresponding to the AI fetch attempt.

4. Results

The troubleshooting results were as follows.

CheckResult
DNS (A / AAAA)Resolved normally
HTTPS over IPv4200
HTTPS over IPv6200
HEAD200
GET200, content retrievable
HEAD / GET from another external network200
Explicit denial in robots.txtNot found
Normal external accessRecorded in the access log
AI fetch attemptNo corresponding HTTP request found in the normal access log

At the time of the investigation, I found no clear problem in the basic site-side path—DNS, IPv4 / IPv6, HTTPS, or ordinary GET requests—that would explain the AI fetch failure.

At that point, the priority of continuing to investigate only the website side became lower.

5. AI web fetching is not necessarily the same as a browser request

The public security designs described by AI providers are useful background here.

An AI agent does more than simply display a page. Depending on the environment, it may be operating while it has access to conversation data or other tools. External URL fetching can therefore become a path for data exfiltration.

In January 2026, OpenAI published a description of a mitigation for URL-based data exfiltration.

In that design, before an agent automatically fetches a URL, the system checks whether that exact URL has previously been observed in a web index that is independent of the user's conversation. If the URL is not recognized as previously observed, the system may avoid treating it as immediately trusted, try another source, or require explicit user interaction.

The unit of comparison is the individual URL, not simply whether the domain is well known. The published description is not equivalent to saying that "minor domains that do not appear in search results are never fetched."

Google has also described layered prompt-injection defenses for Gemini, including mechanisms that use Google Safe Browsing to detect suspicious URLs. This is not the same mechanism as OpenAI's URL-observation design.

Anthropic has likewise described browser-using agents as operating in an environment where every visited web page can become a prompt-injection attack surface, and has discussed defenses that include model training, classifiers, and red teaming.

The implementations differ.

Still, these public descriptions show why AI web access should not always be assumed to behave like a normal browser or curl request. Additional safety controls can exist somewhere in the path.

6. The cause in this case remains unknown

This investigation did not identify the direct cause of the failed AI fetch.

Ordinary external access worked. I found no clear fault in the basic site-side checks. And, for the targets where I could inspect the relevant log, I could not find a corresponding HTTP request in the normal access log when the AI fetch failed.

That leaves possibilities other than a website fault, including processing or controls that occur before an HTTP request reaches the web server.

This does not prove that any specific OpenAI or other provider safety mechanism caused the observed behavior.

The published security designs are useful here only as evidence that such controls exist and can be part of the overall path.

7. Deciding when to stop digging into the site

The most useful outcome was not a root-cause finding. It was a practical stopping point for site-side troubleshooting.

DNS resolves. HTTPS works over both IPv4 and IPv6. GET retrieves the content. The site works from another external network. robots.txt does not explicitly deny access.

Normal external requests are visible in the web server log, but there is no corresponding HTTP request for the AI fetch attempt.

Once those conditions are in place, I think it is reasonable to lower the priority of continuing to investigate only the website just because the AI cannot read it.

Depending on the architecture, there may still be more to check—CDN, WAF, network ACLs, TLS behavior, or other layers.

But "the AI cannot fetch it, therefore something on the site must be broken" is not a sufficient reason to keep digging in the same direction.

When AI is used as an aid during development and debugging, it is worth keeping in mind that the fetch function itself may have constraints or safety boundaries. In this investigation, that possibility was one of the things that kept the troubleshooting from going too deep into a layer where the actual problem might not be.

References

8. References