![\"Visualization](\"https://bruceediger.com/images/contabo_requests.png\")
In my post about\nScraper Junkyard results,\nI hypothesized a single, centrally-controlled web crawler\nthat made requests from Contabo GmbH\nIP address ranges.\nAfter a closer look at the Apache combined format log files,\nI can confirm that at least one web crawler\nworks out of Contabo IP address space.
\nContabo is a cloud provider, renting virtual private servers,\nand dedicated servers.\nThis isn’t someone taking advantage of residential proxies.
\nAbove, a visualization of web traffic to my Scraper Junkyard,\na large number of interlinked\npragmatically generated URLs\nfor the period 2026-08-01,\nwhen I made the scraper junkyard available,\nthrough 2026-09-30,\nwhen I lost patience with the experiment.\nThe golden squares plot a subset of all the requests,\nthe requests from Contabo IP addresses.\nLines join actual data points to make it easier for\nhuman eyes to follow trends.\nThe lines don’t mean anything.
\nThe Scraper Junkyard web pages are all generated by\na PHP program.\nEach web page has a “node number”,\nand the PHP program generates HTML with links to\na parent node, left child node and right child node.\nIf N is a page’s node number,\nthe left child link is to a page with node number 2N+1,\nthe right child link is to a page with node number 2N+2.\nThe parent link is to node number (N-1)/2.\nSince the PHP program does integer arithmetic,\nthe parent link node number is always an integer.\nThe parent, left and right child hyperlinks\nof the generated web pages form a large binary tree,\nwith the node number of each web page appearing in\nbreadth first order.
The Scraper Junkyard entry point,\nand root node of the binary tree is node number 0.\nIt had the URL /node_0.html.
The breadth-first web page node numbers become important\nfurther along, so here’s an elaboration:
\n![\"binary](\"https://bruceediger.com/images/breadth_first65.png\")
That’s the first four levels of the binary tree.\nThe maximum generated node number was 1048578,\nso the tree had 20 full levels,\nand 4 nodes on level 21.\nI didn’t correctly calculate what node number\nto stop generating parent and child links.
\nThe gold and purple points and lines aren’t amenable\nto a whole lot of interpretation.\nThere’s nothing more to think about.\nLuckily, the web pages have node numbers,\narranged by hyperlink in a very specific manner.
\nLooking at what node numbers Contabo IP addresses request\nshows a lot of structure.\nHere’s the node number of each request,\nagainst the Apache log file time stamp.
\n![\"node](\"https://bruceediger.com/images/contabo_numbers_series.png\")
That’s a whole lot of structure.\nI named the structures like this:
\n| Start | End | |
|---|---|---|
| Initial probes | 2026-08-04T08:04:01Z | 2026-08-04T08:20:54Z |
| Irregulars | 2026-08-15T21:14:32Z | 2026-09-01T11:54:45Z |
| Series B | 2026-09-01T14:39:42Z | 2026-09-04T15:45:55Z |
| Series C | 2026-09-05T13:02:42Z | 2026-09-08T13:22:34Z |
| Series D | 2026-09-11T08:08:04Z | 2026-09-17T21:50:57Z |
| Sub-series 1 | 2026-09-11T08:08:04Z | 2026-09-14T08:14:59Z |
| Sub-series 2 | 2026-09-12T15:16:24Z | 2026-09-15T14:56:25Z |
| Sub-series 3 | 2026-09-14T22:08:25Z | 2026-09-17T21:50:57Z |
| Series E | 2026-09-25T04:59:46Z | 2026-09-28T04:08:41Z |
What I’ve labeled “Series D” is three intertwined\nset of requests.\nI would list them as three separate series,\nbut the log file makes it hard to separate\nthe requests into sub-series.\nI can distinguish sub-series starting and ending\ntimestamps due to the “node numbering” of individual web pages.
\n![\"Contabo](\"https://bruceediger.com/images/contabo_numbers_series_d.png\")
Above, the Series D node numbers with respect to time of request,\nvisualized with smaller points to see some detail in the series of requests.
\nThe structure of all the series and sub-series is surprisingly consistent,\nwith the exception of the “Irregulars”.
\n![\"log](\"https://bruceediger.com/images/contabo_numbers_all_logscale.png\")
That’s all the “node numbers” of generated web pages that the Contabo IP addresses\nasked for.\nI used smaller dots than in the linear scale visualization.\nThe Y-axis is log scale (base 2).\nThe Y-tics are at node numbers of the start of each rank\nof the binary tree.\nThis visualization shows the smaller-node-number structure,\nand illustrates that the Contabo Crawler appears to traverse each rank of\nthe binary tree concurrently.\nThis visualization shows that the Contabo Crawler\nstarted two more sub-series of requests,\none between the starts of sub-series 2 and 3,\nand another at the end of Series E requests.\nThat detail was not visible in the linear scale visualizations.
\n| Elapsed Time, Seconds | Requests | Reqs/sec | Unique Node Numbers | IP addresses | |
|---|---|---|---|---|---|
| Initial probes | 1013 | 3365 | 3.3 | 3365 | 2804 |
| Irregulars | 194 | 185 | |||
| Series B | 263173 | 1044633 | 4.0 | 1044628 | 5632 |
| Series C | 260392 | 1045841 | 4.0 | 1045840 | 5693 |
| Series D | 567773 | 3137988 | 5.5 | 1048579 | 7509 |
| Sub-series 1 | 259615 | ~1045996 | ~4.0 | ||
| Sub-series 2 | 258001 | ~1045996 | ~4.1 | ||
| Sub-series 3 | 258152 | ~1045996 | ~4.1 | ||
| Series E | 256135 | 1046350 | 4.1 | 1046349 | 5606 |
The Scraper Junkyard PHP program generates 1048578\ndistinct web pages reachable via hyperlinks.\nThe Contabo Crawler requests the entire Scraper Junkyard\nduring each of the series of requests.
\nBecause Series D has about 3 times the number of HTTP requests\nas Series B, C and E, I feel that it’s fair to divide the 3137988\nSeries D requests by 3 and say that each of the three sub-series had\napproximately 1045996 requests.\nThat means all 6 series requests were made at a rate of 4 per second.\nI don’t think the 4 requests per second is a limit of my $6.47 a month VPS\nbecause the “initial probes” happened at 3.3 requests per second,\nwhile Series D happened at 5.5 requests per second overall.\nThe two overlapping periods happened at an even faster rate.\nThe first overlap, running from 2026-09-12T15:16:24Z to 2026-09-14T08:14:59Z,\ncontains 1193110 requests and lasted 147515 seconds, making 8.1 requests per second.\nEven though I didn’t sort out sub-series 1 and 2 requests,\nContabo IP addresses made twice the usual rate of requests during this overlap.\nThis validates my divide-by-three assumption.
\nThe Contabo Crawler really does hold a close to constant\nrate of requests.\nApache combined format log files have a 1-second resolution.\nTimestamps look like 24/Oct/2026:00:00:07 +0000.\nBecause I can distinguish Contabo lines in Apache log files,\nI wrote a little shell script that counted the number of requests\nthat got logged in each log file second.\nOut of 262,802 log file seconds\n(not elapsed seconds, the number of seconds that lexically occurs in the log files),\n260,308 had 2, 3, 4 or 5 requests.\nI think this is strong evidence for a more-or-less constant\nrate of 4 requests per second.
One additional aspect of the Contabo Crawler’s Scraper Junkyard traversal\nthat’s worth mentioning is the number of URLs any given IP address asks for.\nThe vast majority of the IP addresses make a multiple of 101 requests.\nA lot make 101 requests, somewhat less make 202, 303, 404, etc.\nThis looks like the programmer wanted a worker\nto make a nice round number of requests, 100 would be nice,\nbut made an off-by-1 error in the loop.
\nThe node numbers, unique numbers in each virtual web page\ngenerated by the PHP program provide even more insight into the\nworkings of the Contabo Crawler.
\nThe Series B, C, E and three Series D sub-series all\nlook like the Contabo Crawler makes concurrent\nbatches of requests at the binary tree rank level.
\n![\"Series](\"https://bruceediger.com/images/contabo_numbers_series_b.png\")
Above all the Series B requests.\nThe Y-axis is log2 scale.\nNode 0 doesn’t appear on the log-scale plot.\nThe Y-axis tics are at the beginning node number of binary tree “ranks”.\nThe web page with node 0 is rank 1, web pages with node numbers 1 and 2 are rank 2.\nNode number 3, 4, 5 and 6 are rank 3 and so forth.\nEach rank has twice as many elements as the one before it.\nI put a rank number on the right side of each row of dots.
\nYou can see a few apparently stray rank 20 requests.\nI wrote a small program to check if any web pages requested had\n“out of sequence” node numbers.\nAn out of sequence request would be for a node number whose\nparent node number had not previously been requested.\nIt’s possible to check this because of the breadth-first node numbering\nof web pages.\nIf a node number N got requested, it’s parent is (N-1)/2.\nI found 5 out of sequence requests in Series B,\nnode numbers 888, 637, 867725, 1022113, and 998913.\nThe largest 3 are the strays showing up in the visualization.\nThese 5 node numbers are the only duplicate node numbers\nrequested in Series B.\nI suspect they’re manual checks,\nbut that raises the question of why would someone let\ntheir crawler do about one million web page requests five\nmore times.\nThat’s a huge waste of resources on their part,\nand a triumph for my Scraper Junkyard.\nRegardless, the lack of any requests for node numbers\nwhose parent nodes haven’t been requested shows that\nthe Contabo Crawler is working solely by following child hyperlinks.\nIt feeds the URLs in the links on any web pages it retrieves\nback for later requests.
The Contabo Crawler appears to request node numbers in ranks\nhigher than 5 (web pages with node numbers 15 through 30 inclusive)\nmore-or-less concurrently.\nBecause of that, the “feeding URLs in hyperlinks back for later request”\npart becomes more complicated to understand.
\n![\"Detail](\"https://bruceediger.com/images/contabo_numbers_series_b_detail.png\")
Above, the first 18 minutes of Series B requests,\nabout 4200 requests.\nThe Contabo Crawler retrieves web pages with\nnode number 1 and 2, then 3 through 6, 7 through 14,\nand 15 through 30.\nThose are the first 5 ranks of the binary tree of web pages.
\nIt only retrieves rank 6 nodes 31 through 46.\nThe crawler asks for the web page with node number 47\nmuch later, about halfway through series B.\nRoughly the same pattern occurs at the start of Series C, D, and E:\nrequests for web pages of all node numbers up to 45, 46 or 47,\nthen asking for web page with a node number in rank 7.\nThe request for the next node number in rank 6 occurs about\nhalf way through the series.
\nEven the 3365 requests I’ve labeled “Initial Probes” follow this pattern.
\n![\"Detail](\"https://bruceediger.com/images/contabo_initial_burst.png\")
Since the initial probing of my Scraper Junkyard has the same “shape”\nas all other series of requests,\nI have to conclude that this is some aspect of how the crawler\nsends URLs to workers to retrieve.
\nI wrote a simulation to try to understand how the Contabo Crawler retrieves\nweb pages with node numbers in\nranks 6 through 20 concurrently.\nRemember that each rank has twice as many nodes in it\nas the next-lower-numbered rank.\nIn the linear scaled Y-axis graph,\neach rank has a slope twice as much as the rank below.\nSomehow, the Contabo Crawler managed to make requests\nat higher and higher rates for each rank.
\nThis article\ndescribes the common design of several open source web crawlers.
\nWorkers that take URLs from the head of the queue,
and give new hyperlinks discovered in the URLs back to a server.
The server de-duplicates URLs given to it,
placing any unique URLs on the back of the queue.
If only a single worker requests URLs\nand gives hyperlink sources to the server,\nrequests for my Scraper Junkyard would end up in log files\nin numerical order.\nIf two or more workers pop URLs from the head of the queue,\nand suffer indeterminate delays because of network routing,\ncontention at my Scraper Junkyard, it won’t look so neat and tidy.\nThe Contabo Crawler does de-duplicate URLs it receives:\nevery series asks for every web page in the binary tree once,\nand web pages only get requested after their parent pages\nhave been requested.\nI wrote my simulation in the Go programming language,\nwhich has easy concurrency, and primitives like channels that\nmake the simulation easier to get correct.
\nSimulation code repo for your inspection.
\nMy simulation starts a conceptual server,\nwhich comprises two goroutines.\nOne of the goroutines reads node numbers from a channel,\nand pushes them on the end of a queue (FIFO).\nThe second goroutine\nreads node numbers from the front of the queue,\nand writes them to a second channel.\nAfter creating channels, the queue and starting the server goroutines,\nthe simulation starts a user-specifiable number\nof “worker” goroutines.\nEach worker reads a node number from the second channel.\nThe worker goroutines use the node number to calculate\nthe left and right child node numbers,\nwhich are written into the first channel.\nThe Go runtime schedules goroutines indeterminately,\nbut I also had worker goroutines call runtime.Gosched()\nbefore writing left and right child node numbers to the first channel.\nI piously hoped that rescheduling goroutines would simulate\nthe effects of many machines and network delays.\nWorkers write out a timestamp and the child node numbers,\nwhich is more-or-less what Apache httpd does in the real world.\nTimestamps are integers, which increment before each write.\nThe timestamps have no relationship to actual clock seconds.
![\"node](\"https://bruceediger.com/images/simulated_contabo_crawler.png\")
This looks nothing like the Contabo Crawler’s\nconcurrent rank node number requests.\nThere’s a hint of numerically consecutive requests,\nbut no series of requests as consistent\nand long lasting as Contabo Crawler’s.
\nI tried varying the channel sizes (only changed simulation run time)\nand the number of workers.\nSince the Contabo Crawler only does 4 requests a second,\nI did not anticipate number of workers doing much, and indeed,\nvarying that doesn’t change anything obvious.
\nI also tried a stack of queues - when the queue of “to do” node numbers gets\ntoo big,\na server goroutine pushes the current queue on a stack.\nThe server goroutine creates and starts using a new queue.\nWhen a queue is emptied, pop the stack and\nstart working on those saved node numbers.\nThis did give the effect of working on all ranks nearly concurrently,\nbut the progression of node numbers was “fuzzier”\nthan the Contabo Crawler.
\nI suspect that the Contabo Crawler’s “to do” apparatus is some combination\nof queues and distributed system.\nPerhaps a queue with enough “to do” URLs in it gets handed off to another\nserver which has its own workers.\nMaintaining a constant rate of requests becomes more difficult in this scenario.
\nIt seems evident that there’s a centrally-coordinated web site scraper\noperating out of Contabo GmbH IP address spaces.
\nHTTP requests from Contabo GmbH IP address spaces ask for
every single web page provided by my Scraper Junkyard in an\n an orderly traversal via hyperlinks in the web pages.
The overall rate appears to be limited to 4 requests per second,
and doesn’t vary too much instantaneously.
Roughly the same order of requests happens six times.
My Scraper Junkyard gets crawled the same way every time.
Over 4 series of requests (separated by time),
7690 IP address make requests.\n 2964 IP addresses participate in all four series.
Since Contabo rents virtual private servers and whole machines,\nwhoever is running the crawler is spending a decent amount of money\nto do so, if each IP address represents a different VPS or machine.
\nFour requests per second isn’t an impossible rate.\nA single VPS could probably keep up four requests per second.\nContabo GmbH would have to facilitate this with some method\nof rapidly rotating IP addresses used by a machine.
","headings":[{"level":3,"text":"Scraper Junkyard","id":"scraper-junkyard"},{"level":3,"text":"Internal Structure of Contabo Crawler Requests","id":"internal-structure-of-contabo-crawler-requests"},{"level":3,"text":"Characterizing Series of Requests","id":"characterizing-series-of-requests"},{"level":3,"text":"The Shape of the Scraper Junkyard Traverse","id":"the-shape-of-the-scraper-junkyard-traverse"},{"level":3,"text":"Simulation","id":"simulation"},{"level":3,"text":"Conclusion","id":"conclusion"}]}}