DevOps NetworkDevOpsNetwork
Join Free
Practical infra coding

Fetch every page of a cursor-paginated API

easyAPIs and HTTP Must-do

Problem statement

An inventory service lists hosts with cursor pagination: each response holds one page of items and a next_cursor. You pass that cursor back to get the next page, and a null (or empty) cursor means you have reached the last page. A report script needs every host, not just the first page.

Write fetch_all(client, path) that returns all items across all pages, in order. The client has one method, client.get(path, cursor=None), which returns the parsed JSON as a dict. For the exercise, use a fake client that returns these three pages, so the code runs offline:

Mock API responses

◈ DIAGRAM
GET /hosts -> {"items": [{"name": "web-1", "env": "prod"}, {"name": "web-2", "env": "prod"}], "next_cursor": "c2"}
GET /hosts?cursor=c2 -> {"items": [{"name": "db-1", "env": "prod"}, {"name": "ci-1", "env": "dev"}], "next_cursor": "c3"}
GET /hosts?cursor=c3 -> {"items": [{"name": "cache-1", "env": "staging"}], "next_cursor": null}

Then print how many hosts were fetched and in how many calls, one line per host, and a count per environment sorted by name.

Your function must also survive a buggy server. If a cursor comes back that you have already used, the server is looping and a naive while loop would run forever; raise RuntimeError instead. Put a hard cap on the number of pages as a second safety net, and raise ValueError if a page has no items list.

Examples

Example 1

Input: The three mock pages above, then a second fake client that always returns `next_cursor: "c2"`.

Output: fetched 5 hosts in 3 calls web-1 prod web-2 prod db-1 prod ci-1 dev cache-1 staging by env: dev=1, prod=3, staging=1 error: cursor 'c2' repeated, server is looping

Explanation: The looping client returns c2 on the first call and c2 again on the second, so the second sighting raises instead of spinning forever.

Hints

Approach

Use one loop for every request, starting from cursor = None, so there is no special first call.

  1. Request the page and check that items is a list; raise ValueError with the page in the message if not.
  2. Extend the result list, then read next_cursor with .get(). if not cursor treats both None and "" as the end.
  3. Before using a cursor, check it against a set of cursors already seen. A repeat raises RuntimeError, which turns an infinite loop into a clear error.
  4. The for _ in range(max_pages) loop is the backstop: even a server that invents a new cursor every time cannot keep the script running forever.

n is the total number of items. The fake client counts its calls, which lets the script print 3 calls and lets a test check that no extra request was made after the last page. The per-environment counts come from Counter, sorted so the output is stable.

ComplexityTime O(n)Space O(n)
Python
from collections import Counter
class FakeInventoryAPI:
# Stands in for GET /hosts?cursor=... so the code runs offline.
PAGES = {
None: {"items": [{"name": "web-1", "env": "prod"}, {"name": "web-2", "env": "prod"}],
"next_cursor": "c2"},
"c2": {"items": [{"name": "db-1", "env": "prod"}, {"name": "ci-1", "env": "dev"}],
"next_cursor": "c3"},
"c3": {"items": [{"name": "cache-1", "env": "staging"}], "next_cursor": None},
}
def __init__(self):
self.calls = 0
def get(self, path, cursor=None):
self.calls += 1
return self.PAGES[cursor]
class LoopingAPI:
# A buggy server that keeps handing back the same cursor.
def get(self, path, cursor=None):
return {"items": [{"name": "x", "env": "dev"}], "next_cursor": "c2"}
def fetch_all(client, path, max_pages=1000):
items = []
cursor = None
seen = set()
for _ in range(max_pages):
page = client.get(path, cursor=cursor)
batch = page.get("items")
if not isinstance(batch, list):
raise ValueError(f"page has no 'items' list: {page!r}")
items.extend(batch)
cursor = page.get("next_cursor")
if not cursor: # None or "" both mean "last page"
return items
if cursor in seen:
raise RuntimeError(f"cursor {cursor!r} repeated, server is looping")
seen.add(cursor)
raise RuntimeError(f"stopped after {max_pages} pages")
api = FakeInventoryAPI()
hosts = fetch_all(api, "/hosts")
print(f"fetched {len(hosts)} hosts in {api.calls} calls")
for h in hosts:
print(f" {h['name']:<8} {h['env']}")
by_env = Counter(h["env"] for h in hosts)
print("by env:", ", ".join(f"{env}={n}" for env, n in sorted(by_env.items())))
try:
fetch_all(LoopingAPI(), "/hosts")
except RuntimeError as e:
print("error:", e)

Follow-up questions

  • Add retries with backoff around each page request, so one failed page does not throw away the pages already fetched.
  • The API also returns a Link: <...>; rel="next" header instead of a cursor field. How do you follow that?
  • Turn fetch_all into a generator and explain what changes for the caller.

Frequently asked questions

Almost every cloud and SaaS API that lists things (instances, DNS records, users, pipeline runs) paginates. A script that reads only the first page silently reports on part of the fleet, and that kind of bug usually goes unnoticed until an audit misses something.

Offset pagination asks for ?page=3 or ?offset=200. If items are added or removed while you page through, you can skip or repeat items. A cursor is an opaque token the server uses to continue exactly where the last page ended, so it is stable under changes. Your loop looks the same for both; only the value you pass back differs.

A list is fine for thousands of items. For very large result sets, make it a generator (yield from batch) so the caller can start processing and memory stays flat. The loop and the safety checks do not change.