技術ログ

Wayback CDX API 504 timeout: what it looks like and how to handle it

公開: 2026-09-13 · 著者: GRAMSHIFT

What 504, 503 and 403 answers look like, and what to change in your client

The Wayback Machine's CDX server can answer with a 504 timeout or a 503 page. Here is what to do, based on ten requests on 2026-09-13. Dates like (2026-09-10) mark our earlier measurement notes.

Check the HTTP status before you parse. The 504 and 503 were HTML, even with output=json.
Retry a one-off 504 or 503 a few times. Wait longer each time, and cap the total wait.
If the same query keeps getting 504, change the query. Narrow the date range or turn off collapse (one capture per period), and always send limit.
Check host names and dates before sending. Bad input does not come back as an error.
Send status filters to the server. Filtering one-per-month results yourself would lose 5 of 24 months.
Treat status - as a revisit, and do not retry a 403 refusal.

Check the status first, then check your input

Read res.status first and parse JSON only on a 200. Otherwise a SyntaxError hides the real problem.

With output=json, the 504 came back with content-type: text/html and this body:

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>

The two 503 answers were HTML too: 11,832-character pages titled Internet Archive: Temporarily Offline.

Both 503s answered the same malformed URL, http://[bad, sent twice. On 2026-09-10 that URL got HTTP 200 with three captures of a different address, bad. Neither answer said "malformed". A broken date (from=abcd) got HTTP 200 with [] (2026-09-10), which looks like "never archived". So check host names and dates before you send. A 503 on a valid URL is different: retry it.

Retry a one-off 504, but change a query that keeps timing out

A 504 means the server gave up after about 60 seconds. There are two cases.

Case 1: a one-off 504 or 503. Retry a few times with longer waits and a cap on the total time. Summary of two requests about the same second of apify.com/ today:

request 5   from=to=20190711224257   limit=1000                      HTTP 200   14,700 ms   2 captures
request 6   from=to=20190711224257   limit=1   showResumeKey=true    HTTP 504   60,346 ms   HTML error page

Request 5, sent about 21 seconds before the 504, got an answer for the same second in 14.7 seconds. The failed request was the smaller one, so size is not the only cause of a 504. The server was slow then: the next query took 44.9 seconds. In production runs (2026-09-10), a 503 and a 504 both went through when fetched again.

Why cap the time: on 2026-09-11, google.com got 504, 504, 504, 503 and 504, using 285 seconds of retries with no data.

Case 2: the same query gets 504 again and again. Retrying usually does not help. Change the query:

Stop collapsing, or narrow from/to. Collapsing means asking the server to keep only one capture per period. With one capture per month, newest first, google.com, bbc.com and facebook.com returned 504 repeatedly (2026-09-10/11). Queries without collapse, or with a narrower date range, went through. This is not guaranteed: for google.com, even single-year lookups failed 4 times in 7.
Always send limit. Without it, one page of example.com returned 768,693 rows in 60 seconds (2026-09-10).
Group URLs yourself. Listing a domain with collapse=urlkey (one row per URL) timed out with 504 even for 20 rows (2026-09-10), although today it answered in 874 ms for apify.com. Asking for raw captures and grouping them in your code avoids that request.

Send the status filter with the request

For only successful captures, add filter=statuscode:2.. to the request. Do not filter collapsed results in your code.

We asked for one capture per month of apify.com/ for the latest 24 months (collapse=timestamp:6, limit=-24), twice:

Without the filter: 24 rows, 19 with status 200 and 5 with 301.
With filter=statuscode:2..: 24 rows, all 200, for the same 24 months.

The five months that differed were 2024-11, 2024-12, 2025-01, 2025-02 and 2026-08. In those months, the capture the server chose was a redirect from another spelling of the host (www.apify.com four times, Apify.com once). If you keep only the 200 rows of the first answer, 19 months are left. The second answer has a 200 capture for all 24, which means the server filters before it picks one per month.

Status "-" is a revisit, not an error

Store a - status as null and mark the row as a revisit. Do not turn it into 0.

In the first 3,000 captures listed under apify.com, 2,135 had status 200, 377 had - (12.6%), 274 had 404, 179 had 301, 34 had 308 and 1 had 414. There were also 377 rows with the mimetype warc/revisit. A revisit means the crawler found content it already had. parseInt("-") || 0 turns it into status 0, which looks like a network failure in a report.

mimetype can also be unk (34 of the 3,000 rows) or unknown (2026-09-10); store those as null too. And two captures can share one second, so do not use (urlkey, timestamp) as a unique key. This was the full answer for request 5 above:

[["urlkey","timestamp","original","mimetype","statuscode","digest","length"],
["com,apify)/","20190711224257","https://apify.com/","text/html","200","O757GCDO6S3XYZID6SWCJ35FPWJDKAW7","11621"],
["com,apify)/","20190711224257","https://www.apify.com/","text/html","301","QH732FYSV7UM34JYWVYMB7EZGR2CYM6B","711"]]

This matters when paging with showResumeKey (a key to continue from). On 2026-09-10, fetching this second with limit=1 and then continuing with the key returned [], so the 301 was never delivered. If you page, ask again for the last second of each page (from and to set to that timestamp) and add any capture the page did not include.

A 403 on a domain listing is a refusal

If a listing returns 403 with "This type of CDX query requires authorization", stop. Do not retry it, and do not report it as "no captures".

A domain listing for nytimes.com came back in 371 ms with content-type: text/html; charset=utf-8 and this body:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>403 Forbidden</title>
<h1>Forbidden</h1>
<p>This type of CDX query requires authorization.</p>

The site is archived: a yearly history of www.nytimes.com/ returned three rows, the newest from 2026-09-12. The same refusal was seen for domain, host and prefix listings of nytimes.com (2026-09-10). When the archive has nothing to show, it answers 200 with [], so a 403 is a different answer.

If you don't want to maintain this layer

These rules are built into our tool (checked against its source code): Wayback Machine Scraper validates host names and dates before sending; stores revisits with a null statusCode and captureType: "revisit"; lists URLs from raw captures and re-asks the last second of each page; sends the status filter to the archive; returns a 403 as a refused-by-archive row without retrying; and retries 503, 504 and missing responses after 2, 5, 10 and 20 seconds. If it still gets no data and 504s were among the answers, it returns a failed row that suggests narrowing the date range or turning collapse off.

Published by GRAMSHIFT. The 2026-09-13 figures come from ten requests to web.archive.org/cdx/search/cdx between 06:56 and 07:00 UTC, one at a time, at least 6.5 seconds apart, with a 120-second timeout. Each comparison rests on one request per variant. Eight of the ten URLs were built with the tool's query-building function; the two malformed-URL requests were built directly. A follow-up request with the resume key was not sent after the 504. Figures dated 2026-09-10 and 2026-09-11 come from our earlier measurement notes and production runs of the tool, and were not re-measured. This article is not affiliated with the Internet Archive. On the use of AI: the probe script, this article, the tool and the earlier measurement notes were written by Claude, an AI model, working for GRAMSHIFT; every 2026-09-13 figure was compared against the saved responses and the counts the probe script saved (for the 3,000-row response and the 503 pages it kept counts and the start of the body only).