Only the first 1000 search results are available: GitHub Search API HTTP 422 at page 11

I was rewriting the part of a scraper that talks to GitHub's search API when the row count stopped adding up. The API says it matched 7,705 repositories. After reading every page I was allowed to read, I had 1,000 of them on disk. No exception, no non-2xx status, nothing in the log.

On 2026-09-13 I sat down and measured the wall: where exactly it is, when you get told about it, and how many requests it takes to get around it. Every number below is from that session, using plain unauthenticated requests (no token).

The cap is 1,000 results, and only page 11 complains

The query is language:javascript stars:>1000 — JavaScript repositories with at least a thousand stars.

$ curl -s "https://api.github.com/search/repositories?q=language:javascript+stars:%3E1000&per_page=1"
{
  "total_count": 7705,
  "incomplete_results": false,
  ...

7,705 matches. Reading forward with per_page=100, page 10 (results 901–1,000) comes back HTTP 200 like every page before it. Page 11 is where it stops:

$ curl -s -w "HTTP %{http_code}" ".../search/repositories?q=...&per_page=100&page=11"
HTTP 422
{
  "message": "Only the first 1000 search results are available",
  "documentation_url": "https://docs.github.com/v3/search/",
  "status": "422"
}

The message is exact about what it means: only the first 1000. This is a cap on results, not on page numbers. With per_page=50 the same 422 arrives at page 21; with per_page=30 it arrives at page 34. The boundary sits at result 1,000 regardless of how you slice the pages.

So a 422 from this endpoint means one of two very different things — either your query is malformed, or your query was fine and you simply walked off the end of the retrievable set. Worth splitting in your error handling, because one of them is your bug and the other one never will be.

The dangerous case is not the 422. It is the request you never make

422 only fires when you ask for result 1,001 or later. Which means this loop will never see it:

const pages = Math.min(Math.ceil(wanted / 100), 10);  // stop at 10 pages
for (let p = 1; p <= pages; p++) { ... }

Clamping to ten pages is the normal, defensive thing to write, and it is exactly what makes the truncation invisible. Every response is a 200. Every body parses. Nothing throws. The only trace is that your dataset ends at 1,000 rows, and 6,705 of the 7,705 matches quietly never existed.

This is not a swallowed error — there is no error to swallow. Wrapping the loop in try/catch catches nothing, because nothing is thrown. There is exactly one way to notice: compare the count the API declared against the count you actually received, every single run, and write both numbers down.

In my own scraper the cap lives in a named constant (SEARCH_CAP = 1000), and the first line of the log, written as soon as page 1 comes back, reads like 7,705 matched (this API can return at most 1,000). When a 422 does come back, the exception text carries the workaround — split by date — instead of just the status code. Finishing early in silence is the most expensive outcome of the three.

Splitting by date works. Then check that the parts add up

The documented advice is to narrow the query, and in practice the most mechanical way to narrow a repository search is created:, because you can generate the intervals programmatically. I split the same query five ways and measured each total_count:

The five parts sum to exactly 7,705, the number the unsplit query reported. That sum is the whole point of the exercise — it is the only cheap check that the intervals are correct. If the sum comes out lower than the original count, your intervals have a hole in them (an off-by-one-day boundary is the usual cause). If it comes out higher, they overlap and you will be paying twice for the same rows. Both failures are silent, so do the addition every time you generate intervals.

Also notice that three of the five parts are still over 1,000. "Split it by year" is not a rule that terminates — you have to keep splitting until every interval is under the cap, recursively. The 2,566 matches in 2014–2016 need roughly one interval per year before they drop into the 800s. And because the cap is per query, the number of intervals is the number of extra searches you have to make, which runs straight into the next section.

One character wrong in the range and you get zero rows, not an error

A trap I walked into during this measurement. For the open-ended first interval I originally wrote created:..2013-12-31:

Omitting the lower bound is not rejected as a syntax error. It returns HTTP 200 with zero results. Put the wildcard in (*..date) or use the comparison form (<date) and the same interval holds 1,851 repositories.

For a few minutes I was holding the finding that there are no JavaScript repositories with 1,000+ stars created before 2014. Zero is a plausible-looking answer, which is what makes it dangerous — it reads as "old repos are rare, makes sense". There were 1,851. An empty result set is sometimes not an answer at all; it is the sign that your question never arrived. If you generate query strings in code, it is worth re-asking any zero-result query one time in a different syntax before you believe it.

The search endpoint has its own rate limit: 10 per minute unauthenticated

While measuring the intervals above, the fifth slice came back with total_count as undefined. The reason was in the response headers:

X-RateLimit-Limit: 10
X-RateLimit-Remaining: 0
X-RateLimit-Used: 10
X-RateLimit-Resource: search
X-RateLimit-Reset: 1789231660

The limit on X-RateLimit-Resource: search is 10. Unauthenticated search is ten requests per minute, and it is accounted separately from the general REST limit of 60 requests per hour — two different buckets that exhaust independently. Six searches spaced seven seconds apart were enough to empty it.

The useful part is that your own limit is in every response. Reading X-RateLimit-Remaining and sleeping until X-RateLimit-Reset when it reaches zero is more robust than hard-coding the documented number, because the documented number depends on how you authenticated. GitHub documents 30 searches per minute with a token; I did not measure that, because I run this without one. Code that reads the headers behaves correctly either way, which is the reason to write it that way.

One thing not to merge: the rate-limit response is 403 or 429, and it is a completely different situation from the 422. Both look like "I did not get the data", but one is fixed by waiting and the other will never be fixed by waiting. My client waits for the reset on the first, and on the second returns immediately with the split-by-date instruction, because retrying a 422 is just a slower way to fail.

A column that would have shipped a fabricated number: watchers_count

One more mine, if you pipe search rows straight into a table. watchers_count is unusable.

facebook/react-devtools
  stargazers_count : 11010
  watchers_count   : 11010   <- identical to the star count
  subscribers_count:     5   <- the actual number of watchers

For historical reasons — starring used to be called watching — watchers_count is an alias of the star count, and the search response carries only that field. The number of accounts actually subscribed to notifications (subscribers_count) was 5. Shipping watchers_count under a column header called "watchers" means publishing a number that is 2,202x off, with no indication that anything is wrong.

subscribers_count is not present in search result rows at all; getting it means one extra request per repository against /repos/{owner}/{repo}. For my own tool I decided that a column I cannot measure correctly does not get emitted. A missing column makes someone ask a question. A wrong column makes someone build on it.

The shape of this wall is not specific to result counts

The number 1,000 turned out to be the least interesting part. What made this expensive is the shape: hitting the limit is delivered as a successful response. I have now run into the same shape from several unrelated sources — a shared WHOIS/RDAP redirector that returns 403 to some clients in a way that is indistinguishable from "this domain is not registered", and web pages where the server returns HTTP 200 and 5,856 bytes of HTML containing 17 characters of text.

Across all of them only three habits actually helped:

• Compare the count they declared with the count you received, on every run. If they differ, log it. If you cannot reconcile them, say so in the output rather than in a comment.
• Do not accept zero as an answer. Keep "no matches", "capped", "refused" and "the query never parsed" as distinct states, because they need different fixes.
• When you split a query, verify the parts add up. Splitting fails silently in both directions: gaps lose rows, overlaps duplicate and double-bill them.

The tool I built around the wall

I packaged these behaviours into something public: GitHub Repo Search searches through the official REST API, prints the declared-vs-retrieved gap as the first line of the log, and on a 422 stops with the date-splitting syntax written into the message instead of a bare status code. There is no watchers column, for the reason above. In its place are daysSinceLastPush (0 means touched today, 900 means abandoned) and starsPerYear, floored at one month of age so a repository created last week cannot report an absurd rate. Forty thousand stars over ten years and forty thousand over one year are the same number and not the same project.

Written by GRAMSHIFT — an independent developer building Android apps and automation tools. The figures in this article were obtained on 2026-09-13 by sending unauthenticated requests to api.github.com; the measurement script and the first draft of this text were produced while working with Claude Code, and the numbers were then checked against the raw responses. total_count moves from day to day, so the queries are included alongside the results rather than just the results.