Web Page Crawler Revealed

Scraper Junkyard

The Scraper Junkyard web pages are all generated by a PHP program. Each web page has a “node number”, and the PHP program generates HTML with links to a parent node, left child node and right child node. If N is a page’s node number, the left child link is to a page with node number 2N+1, the right child link is to a page with node number 2N+2. The parent link is to node number (N-1)/2. Since the PHP program does integer arithmetic, the parent link node number is always an integer. The parent, left and right child hyperlinks of the generated web pages form a large binary tree, with the node number of each web page appearing in breadth first order.

The Scraper Junkyard entry point, and root node of the binary tree is node number 0. It had the URL /node_0.html.

The breadth-first web page node numbers become important further along, so here’s an elaboration:

---

That’s the first four levels of the binary tree. The maximum generated node number was 1048578, so the tree had 20 full levels, and 4 nodes on level 21. I didn’t correctly calculate what node number to stop generating parent and child links.

---

Internal Structure of Contabo Crawler Requests

The gold and purple points and lines aren’t amenable to a whole lot of interpretation. There’s nothing more to think about. Luckily, the web pages have node numbers, arranged by hyperlink in a very specific manner.

Looking at what node numbers Contabo IP addresses request shows a lot of structure. Here’s the node number of each request, against the Apache log file time stamp.

That’s a whole lot of structure. I named the structures like this:

What I’ve labeled “Series D” is three intertwined set of requests. I would list them as three separate series, but the log file makes it hard to separate the requests into sub-series. I can distinguish sub-series starting and ending timestamps due to the “node numbering” of individual web pages.

Above, the Series D node numbers with respect to time of request, visualized with smaller points to see some detail in the series of requests.

The structure of all the series and sub-series is surprisingly consistent, with the exception of the “Irregulars”.

That’s all the “node numbers” of generated web pages that the Contabo IP addresses asked for. I used smaller dots than in the linear scale visualization. The Y-axis is log scale (base 2). The Y-tics are at node numbers of the start of each rank of the binary tree. This visualization shows the smaller-node-number structure, and illustrates that the Contabo Crawler appears to traverse each rank of the binary tree concurrently. This visualization shows that the Contabo Crawler started two more sub-series of requests, one between the starts of sub-series 2 and 3, and another at the end of Series E requests. That detail was not visible in the linear scale visualizations.

Characterizing Series of Requests

The Scraper Junkyard PHP program generates 1048578 distinct web pages reachable via hyperlinks. The Contabo Crawler requests the entire Scraper Junkyard during each of the series of requests.

Because Series D has about 3 times the number of HTTP requests as Series B, C and E, I feel that it’s fair to divide the 3137988 Series D requests by 3 and say that each of the three sub-series had approximately 1045996 requests. That means all 6 series requests were made at a rate of 4 per second. I don’t think the 4 requests per second is a limit of my $6.47 a month VPS because the “initial probes” happened at 3.3 requests per second, while Series D happened at 5.5 requests per second overall. The two overlapping periods happened at an even faster rate. The first overlap, running from 2026-09-12T15:16:24Z to 2026-09-14T08:14:59Z, contains 1193110 requests and lasted 147515 seconds, making 8.1 requests per second. Even though I didn’t sort out sub-series 1 and 2 requests, Contabo IP addresses made twice the usual rate of requests during this overlap. This validates my divide-by-three assumption.

The Contabo Crawler really does hold a close to constant rate of requests. Apache combined format log files have a 1-second resolution. Timestamps look like 24/Oct/2026:00:00:07 +0000. Because I can distinguish Contabo lines in Apache log files, I wrote a little shell script that counted the number of requests that got logged in each log file second. Out of 262,802 log file seconds (not elapsed seconds, the number of seconds that lexically occurs in the log files), 260,308 had 2, 3, 4 or 5 requests. I think this is strong evidence for a more-or-less constant rate of 4 requests per second.

One additional aspect of the Contabo Crawler’s Scraper Junkyard traversal that’s worth mentioning is the number of URLs any given IP address asks for. The vast majority of the IP addresses make a multiple of 101 requests. A lot make 101 requests, somewhat less make 202, 303, 404, etc. This looks like the programmer wanted a worker to make a nice round number of requests, 100 would be nice, but made an off-by-1 error in the loop.

The Shape of the Scraper Junkyard Traverse

The node numbers, unique numbers in each virtual web page generated by the PHP program provide even more insight into the workings of the Contabo Crawler.

The Series B, C, E and three Series D sub-series all look like the Contabo Crawler makes concurrent batches of requests at the binary tree rank level.

Above all the Series B requests. The Y-axis is log2 scale. Node 0 doesn’t appear on the log-scale plot. The Y-axis tics are at the beginning node number of binary tree “ranks”. The web page with node 0 is rank 1, web pages with node numbers 1 and 2 are rank 2. Node number 3, 4, 5 and 6 are rank 3 and so forth. Each rank has twice as many elements as the one before it. I put a rank number on the right side of each row of dots.

You can see a few apparently stray rank 20 requests. I wrote a small program to check if any web pages requested had “out of sequence” node numbers. An out of sequence request would be for a node number whose parent node number had not previously been requested. It’s possible to check this because of the breadth-first node numbering of web pages. If a node number N got requested, it’s parent is (N-1)/2. I found 5 out of sequence requests in Series B, node numbers 888, 637, 867725, 1022113, and 998913. The largest 3 are the strays showing up in the visualization. These 5 node numbers are the only duplicate node numbers requested in Series B. I suspect they’re manual checks, but that raises the question of why would someone let their crawler do about one million web page requests five more times. That’s a huge waste of resources on their part, and a triumph for my Scraper Junkyard. Regardless, the lack of any requests for node numbers whose parent nodes haven’t been requested shows that the Contabo Crawler is working solely by following child hyperlinks. It feeds the URLs in the links on any web pages it retrieves back for later requests.

The Contabo Crawler appears to request node numbers in ranks higher than 5 (web pages with node numbers 15 through 30 inclusive) more-or-less concurrently. Because of that, the “feeding URLs in hyperlinks back for later request” part becomes more complicated to understand.

Above, the first 18 minutes of Series B requests, about 4200 requests. The Contabo Crawler retrieves web pages with node number 1 and 2, then 3 through 6, 7 through 14, and 15 through 30. Those are the first 5 ranks of the binary tree of web pages.

It only retrieves rank 6 nodes 31 through 46. The crawler asks for the web page with node number 47 much later, about halfway through series B. Roughly the same pattern occurs at the start of Series C, D, and E: requests for web pages of all node numbers up to 45, 46 or 47, then asking for web page with a node number in rank 7. The request for the next node number in rank 6 occurs about half way through the series.

Even the 3365 requests I’ve labeled “Initial Probes” follow this pattern.

Since the initial probing of my Scraper Junkyard has the same “shape” as all other series of requests, I have to conclude that this is some aspect of how the crawler sends URLs to workers to retrieve.

Simulation

I wrote a simulation to try to understand how the Contabo Crawler retrieves web pages with node numbers in ranks 6 through 20 concurrently. Remember that each rank has twice as many nodes in it as the next-lower-numbered rank. In the linear scaled Y-axis graph, each rank has a slope twice as much as the rank below. Somehow, the Contabo Crawler managed to make requests at higher and higher rates for each rank.

This article describes the common design of several open source web crawlers.

1. A queue of URLs

2. Workers that take URLs from the head of the queue,

   and give new hyperlinks discovered in the URLs back to a server.

3. The server de-duplicates URLs given to it,

   placing any unique URLs on the back of the queue.

If only a single worker requests URLs and gives hyperlink sources to the server, requests for my Scraper Junkyard would end up in log files in numerical order. If two or more workers pop URLs from the head of the queue, and suffer indeterminate delays because of network routing, contention at my Scraper Junkyard, it won’t look so neat and tidy. The Contabo Crawler does de-duplicate URLs it receives: every series asks for every web page in the binary tree once, and web pages only get requested after their parent pages have been requested. I wrote my simulation in the Go programming language, which has easy concurrency, and primitives like channels that make the simulation easier to get correct.

Simulation code repo for your inspection.

My simulation starts a conceptual server, which comprises two goroutines. One of the goroutines reads node numbers from a channel, and pushes them on the end of a queue (FIFO). The second goroutine reads node numbers from the front of the queue, and writes them to a second channel. After creating channels, the queue and starting the server goroutines, the simulation starts a user-specifiable number of “worker” goroutines. Each worker reads a node number from the second channel. The worker goroutines use the node number to calculate the left and right child node numbers, which are written into the first channel. The Go runtime schedules goroutines indeterminately, but I also had worker goroutines call runtime.Gosched() before writing left and right child node numbers to the first channel. I piously hoped that rescheduling goroutines would simulate the effects of many machines and network delays. Workers write out a timestamp and the child node numbers, which is more-or-less what Apache httpd does in the real world. Timestamps are integers, which increment before each write. The timestamps have no relationship to actual clock seconds.

This looks nothing like the Contabo Crawler’s concurrent rank node number requests. There’s a hint of numerically consecutive requests, but no series of requests as consistent and long lasting as Contabo Crawler’s.

I tried varying the channel sizes (only changed simulation run time) and the number of workers. Since the Contabo Crawler only does 4 requests a second, I did not anticipate number of workers doing much, and indeed, varying that doesn’t change anything obvious.

I also tried a stack of queues - when the queue of “to do” node numbers gets too big, a server goroutine pushes the current queue on a stack. The server goroutine creates and starts using a new queue. When a queue is emptied, pop the stack and start working on those saved node numbers. This did give the effect of working on all ranks nearly concurrently, but the progression of node numbers was “fuzzier” than the Contabo Crawler.

I suspect that the Contabo Crawler’s “to do” apparatus is some combination of queues and distributed system. Perhaps a queue with enough “to do” URLs in it gets handed off to another server which has its own workers. Maintaining a constant rate of requests becomes more difficult in this scenario.

Conclusion

It seems evident that there’s a centrally-coordinated web site scraper operating out of Contabo GmbH IP address spaces.

1. HTTP requests from Contabo GmbH IP address spaces ask for

   every single web page provided by my Scraper Junkyard in an an orderly traversal via hyperlinks in the web pages.

2. The overall rate appears to be limited to 4 requests per second,

   and doesn’t vary too much instantaneously.

3. Web pages get requested once. No duplicate requests happen.

4. Roughly the same order of requests happens six times.

   My Scraper Junkyard gets crawled the same way every time.

5. Over 4 series of requests (separated by time),

   7690 IP address make requests. 2964 IP addresses participate in all four series.

Since Contabo rents virtual private servers and whole machines, whoever is running the crawler is spending a decent amount of money to do so, if each IP address represents a different VPS or machine.

Four requests per second isn’t an impossible rate. A single VPS could probably keep up four requests per second. Contabo GmbH would have to facilitate this with some method of rapidly rotating IP addresses used by a machine.