Amazon and Google: why HTTP 200 can still mean failure

An HTTP 200 from Amazon or Google is not evidence that a scrape worked. Both sites can answer a bot-like request with a challenge page, a search shell that needs JavaScript to fill in, or a full page of markup that still lacks the records a parser needs, and all three arrive with a 200 status. This article describes those response patterns, why status alone is a poor success check, and the rules and official interfaces to review before collecting anything. It does not report measurements of either site and does not establish how residential or mobile proxies would perform.

What these responses look like in practice

Three response classes account for most of the confusion, and none of them is distinguishable by status code.

A challenge page. The status is 200, the body is small, the title is blank or generic, and the HTML consists mainly of JavaScript that triggers an interstitial or bot-management check. Amazon's challenge markup, for example, carries a challenge function and a verification marker rather than store content. A status-only success counter accepts this response even though the parser finds nothing.

Search markup. The status is 200, the body is large, the title echoes the query, and result-item containers repeat through the page. This looks like success, but a count of a container class name is not a validated record set. Counting a string is not the same as parsing each product card and confirming that its identifier, price and title are present.

A page shell. The status is 200, the body has a moderate size, the title is the generic site name, and the HTML contains a script that asks the browser to enable JavaScript alongside a noscript block. No result headings exist for a parser to match. This is consistent with markup that expects further browser processing, but it does not tell you whether a browser would succeed, or whether a different exit address would receive the same shell.

The same address can receive different classes of response on consecutive requests to different paths. An IP address alone therefore does not predict the response: path, timing, headers, cookies and server-side decisions all contribute, and none of them is controlled by changing the exit.

The useful conclusion is narrow: the status line does not tell a parser whether it has the data it needs. Treat 200 as "a response arrived", not "the scrape succeeded".

Check the body, not the status

A content check needs at least four questions answered per response.

• Does the title match what the page should be, rather than a blank, a generic site name or a verification page?
• Does the body contain a challenge or bot-management script marker? If so, classify the response as a challenge regardless of status.
• How many records parsed, and does each one carry the fields the application needs? Record the parsed count, not the count of a marker string.
• Is the HTML a shell that expects JavaScript to populate it? A noscript block or an enable-JavaScript script is a strong signal.

Log the answers next to the HTTP status, curl's exit code and the transfer timing. Curl's --write-out fields such as size_download and time_starttransfer are documented in the curl manual; they describe the transfer, not the application result, and the two must be recorded side by side.

Robots.txt and service rules

The Robots Exclusion Protocol describes crawler instructions. RFC 9309 explicitly distinguishes those instructions from access authorisation. An absent disallow rule is not, by itself, permission to collect or reuse content.

At the 11 September 2026 review, Google's robots.txt disallowed /search in its general user-agent group, with specific exceptions including /search/about and /search/howsearchworks. Evaluate the actual path and applicable group; a rule count or line number is not a useful permission check.

For Amazon, check the applicable marketplace's robots.txt and Conditions of Use, along with any programme-specific agreement. This article does not establish that any search or product path is authorised for a particular collection or reuse.

Google's terms effective 30 July 2026 restrict automated access that violates its machine-readable instructions. Its Search policy on machine-generated traffic separately covers automated Search access without express permission, including scraping results for rank checking. Review the current documents for your service and region; changing an exit address does not change those requirements.

Official data-access options

Choose an interface by the data and permitted use your application needs. Availability below was reviewed on 11 September 2026.

• Amazon Creators API provides product-catalog access for publishers and affiliate partners under its programme requirements. It is the supported successor to Product Advertising API 5. Amazon's deprecation notice says continued calls to PA-API 5 receive a 403 response with AccessDeniedException. Start with the Creators API documentation.
• Amazon Selling Partner API supports authorised applications working with selling-partner data, such as listings and orders. Its overview explains the scope; access depends on the application, permissions and selling partner.
• Google Custom Search JSON API is closed to new customers. Existing customers must transition to an alternative by 1 January 2027. Google's overview points to Vertex AI Search for up to 50 domains and an inquiry route for full-web search needs. These options require a separate fit and availability check; the legacy API is not an available starting point for a new customer.
• Google Search Console reports your own site's search performance, including clicks, impressions and average position. Those metrics can answer questions about your verified properties, but do not provide a complete competitor ranking feed.

Official interfaces have eligibility requirements, permissions, quotas and usage terms. Compare their coverage with the actual fields, freshness and markets your application requires. A documented interface still needs error handling and does not guarantee unrestricted data access.

When a location-specific connection may help

An authorised location check might investigate how your listing appears in another market, whether a regional campaign is visible, or why a customer sees different content. These are reasons to evaluate a connection from the relevant region, subject to the service's rules.

An exit IP is only one input to such a test. Keep the browser, language, account state and session settings consistent, and record what you changed. A single request or low volume does not itself establish permission, and a regional exit does not reproduce every customer's personalised view.

The rotating versus sticky guide explains session choices. The residential versus datacenter comparison describes how to evaluate candidate pools. This article does not measure either type; choose between them with an authorised comparison of your own workload.

What to record in an authorised comparison

Define application success before making requests. For product data, that can mean a matching product identifier and all required fields; for search data, a parsed result structure and the expected query context. A title or marker count can be a useful diagnostic, but should not stand in for validating the records you will use.

Record the exact public test URL and query, UTC timestamp, client version, request settings, redirect policy, gateway or direct route, and content-check method. Keep HTTP and CONNECT statuses, curl's exit code and timing with the application result. Retain enough sanitised response evidence to explain a challenge, missing fields or a changed parser result.

For a fair comparison, hold the client and request settings constant, sample multiple addresses from each candidate network type, and interleave the requests over the same observation window. Different outcomes justify investigation; they do not alone prove that network type caused the difference. Any sustained test also needs a permitted request rate and a defined stopping condition.

The benchmark method provides a starting point for transport measurements and explains where application-specific checks are needed. If CONNECT returns 407, investigate proxy authentication with the 407 guide. The curl setup guide shows how to record CONNECT and HTTP status separately.

What this article does not establish

This article does not compare responses across residential or mobile exits, test browser rendering, validate complete product records, or measure sustained challenge rates. No request count from a single address can estimate a provider's success rate or predict how a different request volume would behave. Use the patterns above to design a content check; base a proxy choice on a separate, authorised comparison of the workload you need to run.

Tagged:ProxiesChoosing proxies