---
title: "Web Page Crawler Revealed"
slug: web-page-crawler-revealed
url: https://listedarticles.com/articles/web-page-crawler-revealed
canonical_url: https://bruceediger.com/posts/contabo-crawler/
content_type: blog_post
language: en
published_at: 2026-10-25T00:00:00.000Z
updated_at: 2026-10-26T23:14:34.220Z
author: "Bruce Ediger"
authored_by: human
publisher: "Information Camouflage"
publisher_url: https://bruceediger.com/
topics: ["Security", "Web Scraping"]
license: all-rights-reserved
word_count: 2633
reading_minutes: 11
citation: "Bruce Ediger, Information Camouflage. \"Web Page Crawler Revealed.\" 25 Oct 2026. https://bruceediger.com/posts/contabo-crawler/ (all-rights-reserved)"
# The full text follows. The web page shows an extract and sends readers
# to the source above; quote the citation and link the canonical URL.
---

# Web Page Crawler Revealed

> Bruce Ediger digs into Apache logs from his "Scraper Junkyard" honeypot site and confirms that at least one centrally controlled web crawler is operating out of thousands of Contabo GmbH cloud IP addresses, analysing its request patterns and rates.

In my post about
[Scraper Junkyard results](https://bruceediger.com/posts/my-own-on-bots/),
I hypothesized a single, centrally-controlled web crawler
that made requests from [Contabo GmbH](https://contabo.com/en-us/)
IP address ranges.
After a closer look at the Apache combined format log files,
I can confirm that at least one web crawler
works out of Contabo IP address space.

Contabo is a cloud provider, renting virtual private servers,
and dedicated servers.
This isn’t someone taking advantage of residential proxies.

![Visualization of web traffic to this server, top ISPs](https://bruceediger.com/images/contabo_requests.png)

Above, a visualization of web traffic to my Scraper Junkyard,
a large number of interlinked
[pragmatically generated URLs](https://codeberg.org/bediger4000/scraper-junkyard)
for the period 2026-08-01,
when I made the scraper junkyard available,
through 2026-09-30,
when I lost patience with the experiment.
The golden squares plot a subset of all the requests,
the requests from Contabo IP addresses.
Lines join actual data points to make it easier for
human eyes to follow trends.
The lines don’t mean anything.

### Scraper Junkyard

The Scraper Junkyard web pages are all generated by
a PHP program.
Each web page has a “node number”,
and the PHP program generates HTML with links to
a parent node, left child node and right child node.
If `N` is a page’s node number,
the left child link is to a page with node number `2N+1`,
the right child link is to a page with node number `2N+2`.
The parent link is to node number `(N-1)/2`.
Since the PHP program does integer arithmetic,
the parent link node number is always an integer.
The parent, left and right child hyperlinks
of the generated web pages form a large binary tree,
with the node number of each web page appearing in
breadth first order.

The Scraper Junkyard entry point,
and root node of the binary tree is node number 0.
It had the URL `/node_0.html`.

The breadth-first web page node numbers become important
further along, so here’s an elaboration:

---

![binary tree labeled with generated URLs](https://bruceediger.com/images/breadth_first65.png)

That’s the first four levels of the binary tree.
The maximum generated node number was 1048578,
so the tree had 20 full levels,
and 4 nodes on level 21.
I didn’t correctly calculate what node number
to stop generating parent and child links.

---

### Internal Structure of Contabo Crawler Requests

The gold and purple points and lines aren’t amenable
to a whole lot of interpretation.
There’s nothing more to think about.
Luckily, the web pages have node numbers,
arranged by hyperlink in a very specific manner.

Looking at what node numbers Contabo IP addresses request
shows a lot of structure.
Here’s the node number of each request,
against the Apache log file time stamp.

![node numbers vs when they were asked for](https://bruceediger.com/images/contabo_numbers_series.png)

That’s a whole lot of structure.
I named the structures like this:

|  | Start | End |
| --- | --- | --- |
| **Initial probes** | 2026-08-04T08:04:01Z | 2026-08-04T08:20:54Z |
| **Irregulars** | 2026-08-15T21:14:32Z | 2026-09-01T11:54:45Z |
| **Series B** | 2026-09-01T14:39:42Z | 2026-09-04T15:45:55Z |
| **Series C** | 2026-09-05T13:02:42Z | 2026-09-08T13:22:34Z |
| **Series D** | 2026-09-11T08:08:04Z | 2026-09-17T21:50:57Z |
| Sub-series 1 | 2026-09-11T08:08:04Z | 2026-09-14T08:14:59Z |
| Sub-series 2 | 2026-09-12T15:16:24Z | 2026-09-15T14:56:25Z |
| Sub-series 3 | 2026-09-14T22:08:25Z | 2026-09-17T21:50:57Z |
| **Series E** | 2026-09-25T04:59:46Z | 2026-09-28T04:08:41Z |

What I’ve labeled “Series D” is three intertwined
set of requests.
I would list them as three separate series,
but the log file makes it hard to separate
the requests into sub-series.
I can distinguish sub-series starting and ending
timestamps due to the “node numbering” of individual web pages.

![Contabo series D requests detail showing start and stop times](https://bruceediger.com/images/contabo_numbers_series_d.png)

Above, the Series D node numbers with respect to time of request,
visualized with smaller points to see some detail in the series of requests.

The structure of all the series and sub-series is surprisingly consistent,
with the exception of the “Irregulars”.

![log scale visualization of “node numbers” of generated web pages vs time](https://bruceediger.com/images/contabo_numbers_all_logscale.png)

That’s all the “node numbers” of generated web pages that the Contabo IP addresses
asked for.
I used smaller dots than in the linear scale visualization.
The Y-axis is log scale (base 2).
The Y-tics are at node numbers of the start of each rank
of the binary tree.
This visualization shows the smaller-node-number structure,
and illustrates that the Contabo Crawler appears to traverse each rank of
the binary tree concurrently.
This visualization shows that the Contabo Crawler
started two more sub-series of requests,
one between the starts of sub-series 2 and 3,
and another at the end of Series E requests.
That detail was not visible in the linear scale visualizations.

### Characterizing Series of Requests

|  | Elapsed Time, Seconds | Requests | Reqs/sec | Unique Node Numbers | IP addresses |
| --- | --- | --- | --- | --- | --- |
| **Initial probes** | 1013 | 3365 | 3.3 | 3365 | 2804 |
| **Irregulars** |  | 194 |  |  | 185 |
| **Series B** | 263173 | 1044633 | 4.0 | 1044628 | 5632 |
| **Series C** | 260392 | 1045841 | 4.0 | 1045840 | 5693 |
| **Series D** | 567773 | 3137988 | 5.5 | 1048579 | 7509 |
| Sub-series 1 | 259615 | ~1045996 | ~4.0 |  |  |
| Sub-series 2 | 258001 | ~1045996 | ~4.1 |  |  |
| Sub-series 3 | 258152 | ~1045996 | ~4.1 |  |  |
| **Series E** | 256135 | 1046350 | 4.1 | 1046349 | 5606 |

The Scraper Junkyard PHP program generates 1048578
distinct web pages reachable via hyperlinks.
The Contabo Crawler requests the entire Scraper Junkyard
during each of the series of requests.

Because Series D has about 3 times the number of HTTP requests
as Series B, C and E, I feel that it’s fair to divide the 3137988
Series D requests by 3 and say that each of the three sub-series had
approximately 1045996 requests.
That means all 6 series requests were made at a rate of 4 per second.
I don’t think the 4 requests per second is a limit of my $6.47 a month VPS
because the “initial probes” happened at 3.3 requests per second,
while Series D happened at 5.5 requests per second overall.
The two overlapping periods happened at an even faster rate.
The first overlap, running from 2026-09-12T15:16:24Z to 2026-09-14T08:14:59Z,
contains 1193110 requests and lasted 147515 seconds, making 8.1 requests per second.
Even though I didn’t sort out sub-series 1 and 2 requests,
Contabo IP addresses made twice the usual rate of requests during this overlap.
This validates my divide-by-three assumption.

The Contabo Crawler really does hold a close to constant
rate of requests.
Apache combined format log files have a 1-second resolution.
Timestamps look like `24/Oct/2026:00:00:07 +0000`.
Because I can distinguish Contabo lines in Apache log files,
I wrote a little shell script that counted the number of requests
that got logged in each log file second.
Out of 262,802 log file seconds
(not elapsed seconds, the number of seconds that lexically occurs in the log files),
260,308 had 2, 3, 4 or 5 requests.
I think this is strong evidence for a more-or-less constant
rate of 4 requests per second.

One additional aspect of the Contabo Crawler’s Scraper Junkyard traversal
that’s worth mentioning is the number of URLs any given IP address asks for.
The vast majority of the IP addresses make a multiple of 101 requests.
A lot make 101 requests, somewhat less make 202, 303, 404, etc.
This looks like the programmer wanted a worker
to make a nice round number of requests, 100 would be nice,
but made an off-by-1 error in the loop.

### The Shape of the Scraper Junkyard Traverse

The node numbers, unique numbers in each virtual web page
generated by the PHP program provide even more insight into the
workings of the Contabo Crawler.

The Series B, C, E and three Series D sub-series all
look like the Contabo Crawler makes concurrent
batches of requests at the binary tree rank level.

![Series B node numbers vs time, log scale](https://bruceediger.com/images/contabo_numbers_series_b.png)

Above all the Series B requests.
The Y-axis is log2 scale.
Node 0 doesn’t appear on the log-scale plot.
The Y-axis tics are at the beginning node number of binary tree “ranks”.
The web page with node 0 is rank 1, web pages with node numbers 1 and 2 are rank 2.
Node number 3, 4, 5 and 6 are rank 3 and so forth.
Each rank has twice as many elements as the one before it.
I put a rank number on the right side of each row of dots.

You can see a few apparently stray rank 20 requests.
I wrote a small program to check if any web pages requested had
“out of sequence” node numbers.
An out of sequence request would be for a node number whose
parent node number had not previously been requested.
It’s possible to check this because of the breadth-first node numbering
of web pages.
If a node number `N` got requested, it’s parent is `(N-1)/2`.
I found 5 out of sequence requests in Series B,
node numbers 888, 637, 867725, 1022113, and 998913.
The largest 3 are the strays showing up in the visualization.
These 5 node numbers are the only duplicate node numbers
requested in Series B.
I suspect they’re manual checks,
but that raises the question of why would someone let
their crawler do about one million web page requests five
more times.
That’s a huge waste of resources on their part,
and a triumph for my Scraper Junkyard.
Regardless, the lack of any requests for node numbers
whose parent nodes haven’t been requested shows that
the Contabo Crawler is working solely by following child hyperlinks.
It feeds the URLs in the links on any web pages it retrieves
back for later requests.

The Contabo Crawler appears to request node numbers in ranks
higher than 5 (web pages with node numbers 15 through 30 inclusive)
more-or-less concurrently.
Because of that, the “feeding URLs in hyperlinks back for later request”
part becomes more complicated to understand.

![Detail of beginning of Series B node numbers vs time, log scale](https://bruceediger.com/images/contabo_numbers_series_b_detail.png)

Above, the first 18 minutes of Series B requests,
about 4200 requests.
The Contabo Crawler retrieves web pages with
node number 1 and 2, then 3 through 6, 7 through 14,
and 15 through 30.
Those are the first 5 ranks of the binary tree of web pages.

It only retrieves rank 6 nodes 31 through 46.
The crawler asks for the web page with node number 47
much later, about halfway through series B.
Roughly the same pattern occurs at the start of Series C, D, and E:
requests for web pages of all node numbers up to 45, 46 or 47,
then asking for web page with a node number in rank 7.
The request for the next node number in rank 6 occurs about
half way through the series.

Even the 3365 requests I’ve labeled “Initial Probes” follow this pattern.

![Detail visualization of initial burst of Contabo Crawler requests](https://bruceediger.com/images/contabo_initial_burst.png)

Since the initial probing of my Scraper Junkyard has the same “shape”
as all other series of requests,
I have to conclude that this is some aspect of how the crawler
sends URLs to workers to retrieve.

### Simulation

I wrote a simulation to try to understand how the Contabo Crawler retrieves
web pages with node numbers in
ranks 6 through 20 concurrently.
Remember that each rank has twice as many nodes in it
as the next-lower-numbered rank.
In the linear scaled Y-axis graph,
each rank has a slope twice as much as the rank below.
Somehow, the Contabo Crawler managed to make requests
at higher and higher rates for each rank.

[This article](https://endler.dev/2026/how-other-link-checkers-recurse/)
describes the common design of several open source web crawlers.

1. A queue of URLs
2. Workers that take URLs from the head of the queue,
   and give new hyperlinks discovered in the URLs back to a server.
3. The server de-duplicates URLs given to it,
   placing any unique URLs on the back of the queue.

If only a single worker requests URLs
and gives hyperlink sources to the server,
requests for my Scraper Junkyard would end up in log files
in numerical order.
If two or more workers pop URLs from the head of the queue,
and suffer indeterminate delays because of network routing,
contention at my Scraper Junkyard, it won’t look so neat and tidy.
The Contabo Crawler does de-duplicate URLs it receives:
every series asks for every web page in the binary tree once,
and web pages only get requested after their parent pages
have been requested.
I wrote my simulation in the Go programming language,
which has easy concurrency, and primitives like channels that
make the simulation easier to get correct.

[Simulation code repo](https://codeberg.org/bediger4000/contabosim) for your inspection.

My simulation starts a conceptual server,
which comprises two goroutines.
One of the goroutines reads node numbers from a channel,
and pushes them on the end of a queue (FIFO).
The second goroutine
reads node numbers from the front of the queue,
and writes them to a second channel.
After creating channels, the queue and starting the server goroutines,
the simulation starts a user-specifiable number
of “worker” goroutines.
Each worker reads a node number from the second channel.
The worker goroutines use the node number to calculate
the left and right child node numbers,
which are written into the first channel.
The Go runtime schedules goroutines indeterminately,
but I also had worker goroutines call `runtime.Gosched()`
before writing left and right child node numbers to the first channel.
I piously hoped that rescheduling goroutines would simulate
the effects of many machines and network delays.
Workers write out a timestamp and the child node numbers,
which is more-or-less what Apache `httpd` does in the real world.
Timestamps are integers, which increment before each write.
The timestamps have no relationship to actual clock seconds.

![node numbers vs time, simulated contabo crawler](https://bruceediger.com/images/simulated_contabo_crawler.png)

This looks nothing like the Contabo Crawler’s
concurrent rank node number requests.
There’s a hint of numerically consecutive requests,
but no series of requests as consistent
and long lasting as Contabo Crawler’s.

I tried varying the channel sizes (only changed simulation run time)
and the number of workers.
Since the Contabo Crawler only does 4 requests a second,
I did not anticipate number of workers doing much, and indeed,
varying that doesn’t change anything obvious.

I also tried a stack of queues - when the queue of “to do” node numbers gets
too big,
a server goroutine pushes the current queue on a stack.
The server goroutine creates and starts using a new queue.
When a queue is emptied, pop the stack and
start working on those saved node numbers.
This did give the effect of working on all ranks nearly concurrently,
but the progression of node numbers was “fuzzier”
than the Contabo Crawler.

I suspect that the Contabo Crawler’s “to do” apparatus is some combination
of queues and distributed system.
Perhaps a queue with enough “to do” URLs in it gets handed off to another
server which has its own workers.
Maintaining a constant rate of requests becomes more difficult in this scenario.

### Conclusion

It seems evident that there’s a centrally-coordinated web site scraper
operating out of Contabo GmbH IP address spaces.

1. HTTP requests from Contabo GmbH IP address spaces ask for
   every single web page provided by my Scraper Junkyard in an
   an orderly traversal via hyperlinks in the web pages.
2. The overall rate appears to be limited to 4 requests per second,
   and doesn’t vary too much instantaneously.
3. Web pages get requested once. No duplicate requests happen.
4. Roughly the same order of requests happens six times.
   My Scraper Junkyard gets crawled the same way every time.
5. Over 4 series of requests (separated by time),
   7690 IP address make requests.
   2964 IP addresses participate in all four series.

Since Contabo rents virtual private servers and whole machines,
whoever is running the crawler is spending a decent amount of money
to do so, if each IP address represents a different VPS or machine.

Four requests per second isn’t an impossible rate.
A single VPS could probably keep up four requests per second.
Contabo GmbH would have to facilitate this with some method
of rapidly rotating IP addresses used by a machine.