{"article":{"slug":"scanning-7-25-million-public-repositories-on-chinas-largest-git-platform-for-secrets","title":"Scanning 7.25 Million Public Repositories on China's Largest Git Platform for Secrets","subtitle":null,"summary":"Independent security researcher Ben Zimmermann, working with MongoDB's Kingfisher team, scanned 7,256,531 public repositories on Gitee with TruffleHog and Kingfisher and found 26,400 unique verified-live credentials, including an active org-admin key for a major enterprise AI platform and thousands of live cloud keys, at a per-repo density above published GitLab and Bitbucket scans.","content_type":"research","language":"en","canonical_url":"https://www.mongodb.com/company/blog/technical/scanning-7-point-25-million-public-repositories-chinas-largest-git-platform-for-secrets","author":{"name":"Ben Zimmermann","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"MongoDB","url":"https://www.mongodb.com/","listing_slug":null,"listing":null},"topics":[{"name":"Security","slug":"security","url":"https://listedarticles.com/topics/security"},{"name":"Secrets Management","slug":"secrets-management","url":"https://listedarticles.com/topics/secrets-management"},{"name":"Open Source","slug":"open-source","url":"https://listedarticles.com/topics/open-source"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":2056,"reading_minutes":9,"published_at":"2026-10-28T00:00:00.000Z","added_at":"2026-10-28T17:17:53.658Z","updated_at":"2026-10-28T17:17:53.658Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/scanning-7-25-million-public-repositories-on-chinas-largest-git-platform-for-secrets","markdown_url":"https://listedarticles.com/articles/scanning-7-25-million-public-repositories-on-chinas-largest-git-platform-for-secrets.md","example":false,"citation":"Ben Zimmermann, MongoDB. \"Scanning 7.25 Million Public Repositories on China's Largest Git Platform for Secrets.\" 28 Oct 2026. https://www.mongodb.com/company/blog/technical/scanning-7-point-25-million-public-repositories-chinas-largest-git-platform-for-secrets (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://www.mongodb.com/company/blog/technical/scanning-7-point-25-million-public-repositories-chinas-largest-git-platform-for-secrets"},"body_markdown":"*Ben Zimmermann · MongoDB Blog · October 28, 2026*\n\n*This guest post by independent security researcher Ben Zimmermann was developed in collaboration with the Kingfisher team at MongoDB. Ben specializes in large-scale credential exposure research across public source-hosting platforms.*\n\n**TL;DR:** I scanned **7,256,531 public repositories** on Gitee (China's largest indigenous Git hosting platform), using both TruffleHog and [Kingfisher](https://github.com/mongodb/kingfisher), and found **26,400 unique verified-live credentials**. Findings included an active organization-admin credential for a major enterprise AI platform and thousands of live cloud provider access keys.\n\n## What is Gitee?\n\nGitee, launched in 2013 by OSChina, was selected by China's Ministry of Industry and Information Technology as the country's national open-source code hosting platform. Its public API is severely constrained relative to GitHub and GitLab: no paginated projects endpoint, and the on-site code-search feature appears to have been retired years ago. A [2021 attempt](https://chinggg.github.io/post/git-leaks/) to study credential exposure on Gitee concluded that its APIs were too limited to scale the research.\n\nLike any large Git platform, it's an attractive surface for credential exposure:\n\n- It is Git-based, so secrets buried deep in commit history persist.\n- It hosts millions of public repositories with active commit volume.\n- It has no public equivalent of the platform-vendor secret-scanning partnerships that have become standard on GitHub.\n\n## Enumerating millions of Gitee repositories\n\nGitee's public API is severely limited, so enumerating millions of repositories required combining many surfaces: the only official REST endpoint that returned useful data at scale, the platform's Explore feed, per-user repository and stargazer pages on the website itself, and Gitee's internal Elasticsearch widget that backs the on-site search UI.\n\n### Bootstrapping a username corpus\n\nThe single official REST endpoint that returned useful data at scale was GET /api/v5/search/users?q=. I hit it with three-character prefix queries (aaa, aab, ..., zzz) and combined the results with usernames extracted from the public Explore feed (/explore/all?order=starred|popular|updated). I then expanded that set via each user's following list and the stargazers of popular repositories. This produced a seed corpus of **1,704,759 usernames**.\n\n### Scraping per-user pages on the frontend\n\nWith the username corpus in hand, two per-user pages yielded repositories: gitee.com/{user}/projects (a user's own repos) and gitee.com/{user}/stared\\_projects (repos the user has starred). Page 1 of /projects returns full HTML; subsequent pages are AJAX-paginated (X-Requested-With: XMLHttpRequest). The stargazer-list pages used during username bootstrap paginate the same way, additionally requiring a CSRF token extracted from the page-1 HTML.\n\n### Querying the on-site search widget\n\nA large fraction of the platform sits outside the social graph. To reach it, I used the same internal Elasticsearch endpoint that backs Gitee's on-site search UI: so.gitee.com/v1/search/widget/wong1slagnlmzwvsu5ya. It takes a query ID, a q parameter, and pagination via from + size:\n\nUnformatted\n\nCode Snippet# elastic\\_scrape\\_repos.py\nAPI\\_URL = \"https://so.gitee.com/v1/search/widget/wong1slagnlmzwvsu5ya\"\nQUERY\\_ID = \"1048\"\nPAGE\\_SIZE = 20\nMAX\\_RESULTS = 200\nasync def fetch\\_page(session, username, offset=0, language=None):\nparams = {\n\"query\": QUERY\\_ID,\n\"q\": username,\n\"from\": str(offset),\n\"size\": str(PAGE\\_SIZE),\n\"sort\\_by\\_f\": \"\",\n}\nif language:\nparams[\"编程语言\"] = language # \"programming language\"\nasync with session.get(\nAPI\\_URL,\nparams=params,\nheaders={\\*\\*HEADERS, \"userid\": str(uuid.uuid4())},\n) as resp:\nresp.raise\\_for\\_status()\nreturn await resp.json()\n\nEach query returned at most 20 results per page, with a hard cap of **200 results total** (from + size <= MAX\\_RESULTS) regardless of how many matches actually existed. For specific usernames, this was usually fine; for common usernames and trigram prefix queries against the widget, the cap silently truncated results.\n\n### Escaping the 200-result cap with bucket splitting\n\nThe widget's response includes Elasticsearch aggregations that bucket results by language (编程语言) and license (许可证). For the per-username scrape, when a query was capped, I split it by the language buckets the response already returned:\n\nUnformatted\n\nCode Snippet# elastic\\_scrape\\_repos.py\nMAX\\_RESULTS = 200\nPAGE\\_SIZE = 20\ndef get\\_total(data):\ntotal = data.get(\"hits\", {}).get(\"total\", {})\nreturn total.get(\"value\", 0), total.get(\"relation\", \"eq\")\ndef get\\_language\\_buckets(data):\naggs = data.get(\"aggregations\", {})\nreturn [b[\"key\"] for b in aggs.get(\"编程语言\", {}).get(\"buckets\", [])]\n# After fetching page 1 for a username:\ntotal, relation = get\\_total(data)\nif total <= PAGE\\_SIZE:\nreturn all\\_records\n# Within the cap: paginate normally\nif not (total >= MAX\\_RESULTS and relation == \"gte\"):\nfor offset in range(PAGE\\_SIZE, min(total, MAX\\_RESULTS), PAGE\\_SIZE):\ndata = await fetch\\_page(session, username, offset=offset)\ncollect(parse\\_hits(data, username))\nreturn all\\_records\n# Capped: split by language buckets that Elastic already returned\nfor lang in get\\_language\\_buckets(data):\noffset = 0\nwhile offset < MAX\\_RESULTS:\ndata = await fetch\\_page(session, username, offset=offset, language=lang)\nhits = parse\\_hits(data, username)\ncollect(hits)\nif len(hits) < PAGE\\_SIZE:\nbreak\noffset += PAGE\\_SIZE\n\nFor trigram prefix queries against the widget (used to surface repos not associated with any user I'd discovered), I extended this into a BFS over a query tree: capped nodes split first by language, then by license, then by deeper prefix until each branch fit within 200 results.\n\nAfter deduplicating repo IDs across all sources, this produced a working set of **7,256,531** unique public Gitee repositories.\n\n## Wiring up the scan pipeline\n\nThe next challenge was cloning and scanning the 7.25 million repositories. To do this in a single Python script, I used [Modal](https://modal.com), a serverless compute platform popular for AI workloads but well-suited to most parallel jobs. I ran TruffleHog and Kingfisher concurrently against every repository for increased coverage. Each surface verified credentials that the other missed, so running both caught more exposed credentials than either alone.\n\n**Figure 1.** Dual-scanner pipeline architecture.\n\n![A flowchart showing a Modal worker executing a bare git clone that concurrently feeds into TruffleHog and Kingfisher scanners, both writing their findings to a shared Modal Volume.](https://images.contentstack.io/v3/assets/blt7151619cb9560896/blt91ca45d76abd18ed/6ae0b99cf59c3f199fce5dec/internal-mdb-copy-gitee-blog-draft-updated.png)\n\n### The Docker image\n\nThe container image is a multi-stage build that pulls both scanner binaries from their upstream releases into a single Python base, plus git and OpenSSH for cloning:\n\nUnformatted\n\nCode SnippetFROM trufflesecurity/trufflehog:latest AS trufflehog\nFROM ghcr.io/mongodb/kingfisher:latest AS kingfisher\nFROM python:3.11-slim\nCOPY --from=trufflehog /usr/bin/trufflehog /usr/local/bin/trufflehog\nCOPY --from=kingfisher /usr/local/bin/kingfisher /usr/local/bin/kingfisher\nRUN apt-get update && apt-get install -y git openssh-client \\\n&& rm -rf /var/lib/apt/lists/\\*\n\n### The per-repo scan function\n\nA single Modal function bare-clones each repo over SSH and runs both scanners concurrently as subprocesses against the same on-disk clone:\n\nUnformatted\n\nCode Snippet@app.function(\nimage=image,\nvolumes={\"/findings\": volume},\ntimeout=300,\ncpu=(0.125, 2.0),\nmemory=(256, 4096),\nmax\\_containers=350,\n)\ndef scan(repo\\_url: str):\nrepo\\_name = repo\\_url.rstrip(\"/\").split(\"/\")[-1].removesuffix(\".git\")\nclone\\_path = f\"/tmp/{repo\\_name}.git\"\nssh\\_url = repo\\_url.replace(\"https://gitee.com/\", \"git@gitee.com:\")\nsubprocess.run(\n[\"git\", \"clone\", \"--bare\", \"--quiet\", ssh\\_url, clone\\_path],\ncapture\\_output=True, check=True,\n)\nth\\_proc = subprocess.Popen([\n\"trufflehog\", \"git\", \"--bare\", f\"file://{clone\\_path}\",\n\"--json\", \"--only-verified\", \"--allow-verification-overlap\",\n], stdout=subprocess.PIPE, text=True)\nkf\\_proc = subprocess.Popen([\n\"kingfisher\", \"scan\", clone\\_path,\n\"--format\", \"jsonl\", \"--only-valid\",\n], stdout=subprocess.PIPE, text=True)\nth\\_output = th\\_proc.communicate()[0].strip()\nkf\\_output = kf\\_proc.communicate()[0].strip()\n# ... write hits to the shared findings volume\n\n### Dispatching 7.25M repos in batches\n\nI split the corpus into 50 batches of ~150K repos and ran them sequentially at a steady 350 concurrent containers. Smaller batches were safer at this scale and easier to resume after interruptions; running larger batches was unstable and would hit clone-side blocks.\n\n### Cost and scale\n\nTotal wall time: roughly one month of continuous execution. Per-repo clone times were a major contributor: Gitee's servers sit in China, and cross-Pacific network distance plus SSH protocol overhead made each clone materially slower than an equivalent GitHub clone would be. The total cost was $3,097.64, fully absorbed by the $5,000 compute-credit sponsorship from Modal that supported this research. The same Modal + dual-scanner architecture works against most Git hosts with little more than a clone-URL substitution. I used a smaller-scale version of this approach in a [prior investigation of Algolia DocSearch admin keys](https://benzimmermann.dev/blog/algolia-docsearch-admin-keys).\n\n## The results\n\nAcross the 7,256,531 public Gitee repositories scanned, the dual-scanner pipeline surfaced **26,400 unique, verified-live credentials** after deduplicating by token value.\n\nBoth scanners authenticate each candidate against its vendor's API at scan time (TruffleHog's --only-verified and Kingfisher's --only-valid modes), discarding anything that pattern-matches but fails to authenticate. The 26,400 figure is deduplicated by raw secret value, so duplicates across repositories or scanners count once.\n\n| Scanner | Unique Findings |\n| --- | --- |\n| TruffleHog | 12,737 |\n| Kingfisher | 14,664 |\n| Caught by both | 1,001 |\n| Total unique credentials | 26,400 |\n\nEither scanner alone would have caught only about half of these credentials. The two tools' detector coverage was largely complementary: TruffleHog dominated Alibaba Cloud key detection (9,428 vs. 993), while Kingfisher dominated DeepSeek (2,940 vs. 230), OpenAI (1,198 vs. 150), HuggingFace (154 vs. 1), and Chinese-AI services like Zhipu (1,489 vs. 0) that TruffleHog has no detector for.\n\n### Secrets by type\n\nThe single most common leaked credential type was Alibaba Cloud AccessKey pairs: roughly **1 in every 766 Gitee repositories** contained a verified-live Alibaba Cloud credential.\n\n**Figure 2.** Credentials by vendor.\n\n![A bar chart illustrating the number of unique verified-live credentials found per vendor, led by Alibaba Cloud (9,473), YouTube API (3,805), and DeepSeek (3,170), with a long tail of other service providers.](https://images.contentstack.io/v3/assets/blt7151619cb9560896/blt31af8a3bc3afccd4/6ae0bc6109de0e19ab7ad120/figure3.png)\n\nA long tail of additional vendors (LangSmith, OpenWeather, Algolia, PostHog, OpenRouter, MySQL, Stripe, Slack, Twitter, and others) accounts for the rest.\n\n### Secrets committed by date\n\nThe vast majority of verified-live credentials were committed in the last two years. 2025 alone accounts for 12,242 unique credentials, nearly half the entire dataset.\n\n**Figure 3.** Unique credentials committed by year.\n\n![A line chart displaying the number of unique verified-live credentials committed each year from 2008 to 2025, showing a sharp exponential increase in recent years and peaking at 12,242 in 2025.](https://images.contentstack.io/v3/assets/blt7151619cb9560896/bltfbc653e59fa14cad/6ae0bd2d332c2f7b0ca5d929/figure4.png)\n\n### Secrets by committer email domain\n\nMost credentials were committed from personal mail services (qq.com, 163.com, gmail.com), making per-company attribution impossible at scale. The corporate-domain numbers below are a floor; even so, the names span major Chinese and Western tech companies alike.\n\n**Figure 5.** Unique credentials by committer email domain.\n\n![A bar chart showing the distribution of unique verified-live credentials by committer email domain, led by [link removed] (66), huawei.com (32), microsoft.com (22), and bytedance.com (22), followed by other major corporate domains.](https://images.contentstack.io/v3/assets/blt7151619cb9560896/blt1b8175fe75482468/6ae0bdd21a8b5a0e7477135d/figure5.png)\n\nSmaller corporate footprints also appear from Oracle, Apple, Meta, Anthropic, and others, plus Chinese commercial and academic domains.\n\n### Comparing Gitee with GitLab and Bitbucket\n\nThe closest prior data points come from [TruffleSecurity's scan of 5.6 million public GitLab repositories](https://trufflesecurity.com/blog/scanning-5-6-million-public-gitlab-repositories-for-secrets), and a companion Bitbucket scan referenced in that post.\n\n| Platform | Repos scanned | Verified-live credentials | Per 1K repos |\n| --- | --- | --- | --- |\n| Bitbucket | ~2,600,000 | 6,212 | 2.39 |\n| GitLab | ~5,600,000 | 17,430 | 3.11 |\n| Gitee | 7,256,531 | 26,400 | 3.64 |\n\nGitee tops both Western platforms on raw count and per-repository density. One methodology note: the TruffleSecurity scans used TruffleHog alone, while this one used TruffleHog and Kingfisher together. TruffleHog's detector library skews toward Western vendors, and Gitee has heavy Chinese-vendor activity (Alibaba Cloud, DeepSeek, Zhipu, and others) that Kingfisher's coverage helps to surface; stripping our findings to TruffleHog-only puts Gitee at 1.76 per 1,000 repos.\n\n## Disclosure\n\nI emailed the vendors with the highest exposure volume in this scan, providing each a coordinated report containing the full list of verified-live credentials associated with their service, along with source-repository metadata to help them attribute affected customers.\n\nPer-developer notification (contacting the committer of each leaked credential individually) was not pursued at this scale. Vendor-side revocation is the more reliable path, since it doesn't require any action from the user who leaked the credential.\n\nAt the time of writing, revocation status for the disclosed credentials has not yet been confirmed.\n\n## Summary\n\n- **Gitee is a meaningful credential-exposure surface.** 26,400 verified-live credentials across 7.25M repositories. Per-repository density higher than comparable published scans of GitLab and Bitbucket.\n- **Major tech companies appear in the committer metadata, Chinese and Western alike.** The breadth includes Huawei, ByteDance, Google, Microsoft, Oracle, and others, suggesting this is not a purely-Chinese-developer phenomenon.\n- **Disclosure paths are asymmetric.** Western companies typically have bug bounty programs or accessible security channels. Most Chinese companies don't, leaving leaked credentials without a clear remediation path.\n- **The architecture is portable.** The same Modal + dual-scanner pipeline works on most Git hosts with little more than a clone-URL substitution; the full scan cost roughly $3,000.\n\n---\n\n*Originally published at [the MongoDB Blog](https://www.mongodb.com/company/blog/technical/scanning-7-point-25-million-public-repositories-chinas-largest-git-platform-for-secrets).*\n","body_html":"<p><em>Ben Zimmermann · MongoDB Blog · October 28, 2026</em></p>\n<p><em>This guest post by independent security researcher Ben Zimmermann was developed in collaboration with the Kingfisher team at MongoDB. Ben specializes in large-scale credential exposure research across public source-hosting platforms.</em></p>\n<p><strong>TL;DR:</strong> I scanned <strong>7,256,531 public repositories</strong> on Gitee (China&#39;s largest indigenous Git hosting platform), using both TruffleHog and <a href=\"https://github.com/mongodb/kingfisher\" rel=\"nofollow ugc noopener\">Kingfisher</a>, and found <strong>26,400 unique verified-live credentials</strong>. Findings included an active organization-admin credential for a major enterprise AI platform and thousands of live cloud provider access keys.</p>\n<h2 id=\"what-is-gitee\">What is Gitee?</h2>\n<p>Gitee, launched in 2013 by OSChina, was selected by China&#39;s Ministry of Industry and Information Technology as the country&#39;s national open-source code hosting platform. Its public API is severely constrained relative to GitHub and GitLab: no paginated projects endpoint, and the on-site code-search feature appears to have been retired years ago. A <a href=\"https://chinggg.github.io/post/git-leaks/\" rel=\"nofollow ugc noopener\">2021 attempt</a> to study credential exposure on Gitee concluded that its APIs were too limited to scale the research.</p>\n<p>Like any large Git platform, it&#39;s an attractive surface for credential exposure:</p>\n<ul><li>It is Git-based, so secrets buried deep in commit history persist.</li><li>It hosts millions of public repositories with active commit volume.</li><li>It has no public equivalent of the platform-vendor secret-scanning partnerships that have become standard on GitHub.</li></ul>\n<h2 id=\"enumerating-millions-of-gitee-repositories\">Enumerating millions of Gitee repositories</h2>\n<p>Gitee&#39;s public API is severely limited, so enumerating millions of repositories required combining many surfaces: the only official REST endpoint that returned useful data at scale, the platform&#39;s Explore feed, per-user repository and stargazer pages on the website itself, and Gitee&#39;s internal Elasticsearch widget that backs the on-site search UI.</p>\n<h3 id=\"bootstrapping-a-username-corpus\">Bootstrapping a username corpus</h3>\n<p>The single official REST endpoint that returned useful data at scale was GET /api/v5/search/users?q=. I hit it with three-character prefix queries (aaa, aab, ..., zzz) and combined the results with usernames extracted from the public Explore feed (/explore/all?order=starred|popular|updated). I then expanded that set via each user&#39;s following list and the stargazers of popular repositories. This produced a seed corpus of <strong>1,704,759 usernames</strong>.</p>\n<h3 id=\"scraping-per-user-pages-on-the-frontend\">Scraping per-user pages on the frontend</h3>\n<p>With the username corpus in hand, two per-user pages yielded repositories: gitee.com/{user}/projects (a user&#39;s own repos) and gitee.com/{user}/stared_projects (repos the user has starred). Page 1 of /projects returns full HTML; subsequent pages are AJAX-paginated (X-Requested-With: XMLHttpRequest). The stargazer-list pages used during username bootstrap paginate the same way, additionally requiring a CSRF token extracted from the page-1 HTML.</p>\n<h3 id=\"querying-the-on-site-search-widget\">Querying the on-site search widget</h3>\n<p>A large fraction of the platform sits outside the social graph. To reach it, I used the same internal Elasticsearch endpoint that backs Gitee&#39;s on-site search UI: so.gitee.com/v1/search/widget/wong1slagnlmzwvsu5ya. It takes a query ID, a q parameter, and pagination via from + size:</p>\n<p>Unformatted</p>\n<p>Code Snippet# elastic_scrape_repos.py\nAPI_URL = &quot;<a href=\"https://so.gitee.com/v1/search/widget/wong1slagnlmzwvsu5ya\" rel=\"nofollow ugc noopener\">https://so.gitee.com/v1/search/widget/wong1slagnlmzwvsu5ya</a>&quot;\nQUERY_ID = &quot;1048&quot;\nPAGE_SIZE = 20\nMAX_RESULTS = 200\nasync def fetch_page(session, username, offset=0, language=None):\nparams = {\n&quot;query&quot;: QUERY_ID,\n&quot;q&quot;: username,\n&quot;from&quot;: str(offset),\n&quot;size&quot;: str(PAGE_SIZE),\n&quot;sort_by_f&quot;: &quot;&quot;,\n}\nif language:\nparams[&quot;编程语言&quot;] = language # &quot;programming language&quot;\nasync with session.get(\nAPI_URL,\nparams=params,\nheaders={**HEADERS, &quot;userid&quot;: str(uuid.uuid4())},\n) as resp:\nresp.raise_for_status()\nreturn await resp.json()</p>\n<p>Each query returned at most 20 results per page, with a hard cap of <strong>200 results total</strong> (from + size &lt;= MAX_RESULTS) regardless of how many matches actually existed. For specific usernames, this was usually fine; for common usernames and trigram prefix queries against the widget, the cap silently truncated results.</p>\n<h3 id=\"escaping-the-200-result-cap-with-bucket-splitting\">Escaping the 200-result cap with bucket splitting</h3>\n<p>The widget&#39;s response includes Elasticsearch aggregations that bucket results by language (编程语言) and license (许可证). For the per-username scrape, when a query was capped, I split it by the language buckets the response already returned:</p>\n<p>Unformatted</p>\n<p>Code Snippet# elastic_scrape_repos.py\nMAX_RESULTS = 200\nPAGE_SIZE = 20\ndef get_total(data):\ntotal = data.get(&quot;hits&quot;, {}).get(&quot;total&quot;, {})\nreturn total.get(&quot;value&quot;, 0), total.get(&quot;relation&quot;, &quot;eq&quot;)\ndef get_language_buckets(data):\naggs = data.get(&quot;aggregations&quot;, {})\nreturn [b[&quot;key&quot;] for b in aggs.get(&quot;编程语言&quot;, {}).get(&quot;buckets&quot;, [])]</p>\n<h1 id=\"after-fetching-page-1-for-a-username\">After fetching page 1 for a username:</h1>\n<p>total, relation = get_total(data)\nif total &lt;= PAGE_SIZE:\nreturn all_records</p>\n<h1 id=\"within-the-cap-paginate-normally\">Within the cap: paginate normally</h1>\n<p>if not (total &gt;= MAX_RESULTS and relation == &quot;gte&quot;):\nfor offset in range(PAGE_SIZE, min(total, MAX_RESULTS), PAGE_SIZE):\ndata = await fetch_page(session, username, offset=offset)\ncollect(parse_hits(data, username))\nreturn all_records</p>\n<h1 id=\"capped-split-by-language-buckets-that-elastic-already-returned\">Capped: split by language buckets that Elastic already returned</h1>\n<p>for lang in get_language_buckets(data):\noffset = 0\nwhile offset &lt; MAX_RESULTS:\ndata = await fetch_page(session, username, offset=offset, language=lang)\nhits = parse_hits(data, username)\ncollect(hits)\nif len(hits) &lt; PAGE_SIZE:\nbreak\noffset += PAGE_SIZE</p>\n<p>For trigram prefix queries against the widget (used to surface repos not associated with any user I&#39;d discovered), I extended this into a BFS over a query tree: capped nodes split first by language, then by license, then by deeper prefix until each branch fit within 200 results.</p>\n<p>After deduplicating repo IDs across all sources, this produced a working set of <strong>7,256,531</strong> unique public Gitee repositories.</p>\n<h2 id=\"wiring-up-the-scan-pipeline\">Wiring up the scan pipeline</h2>\n<p>The next challenge was cloning and scanning the 7.25 million repositories. To do this in a single Python script, I used <a href=\"https://modal.com\" rel=\"nofollow ugc noopener\">Modal</a>, a serverless compute platform popular for AI workloads but well-suited to most parallel jobs. I ran TruffleHog and Kingfisher concurrently against every repository for increased coverage. Each surface verified credentials that the other missed, so running both caught more exposed credentials than either alone.</p>\n<p><strong>Figure 1.</strong> Dual-scanner pipeline architecture.</p>\n<figure><img src=\"https://images.contentstack.io/v3/assets/blt7151619cb9560896/blt91ca45d76abd18ed/6ae0b99cf59c3f199fce5dec/internal-mdb-copy-gitee-blog-draft-updated.png\" alt=\"A flowchart showing a Modal worker executing a bare git clone that concurrently feeds into TruffleHog and Kingfisher scanners, both writing their findings to a shared Modal Volume.\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<h3 id=\"the-docker-image\">The Docker image</h3>\n<p>The container image is a multi-stage build that pulls both scanner binaries from their upstream releases into a single Python base, plus git and OpenSSH for cloning:</p>\n<p>Unformatted</p>\n<p>Code SnippetFROM trufflesecurity/trufflehog:latest AS trufflehog\nFROM ghcr.io/mongodb/kingfisher:latest AS kingfisher\nFROM python:3.11-slim\nCOPY --from=trufflehog /usr/bin/trufflehog /usr/local/bin/trufflehog\nCOPY --from=kingfisher /usr/local/bin/kingfisher /usr/local/bin/kingfisher\nRUN apt-get update &amp;&amp; apt-get install -y git openssh-client <br />\n&amp;&amp; rm -rf /var/lib/apt/lists/*</p>\n<h3 id=\"the-per-repo-scan-function\">The per-repo scan function</h3>\n<p>A single Modal function bare-clones each repo over SSH and runs both scanners concurrently as subprocesses against the same on-disk clone:</p>\n<p>Unformatted</p>\n<p>Code Snippet@app.function(\nimage=image,\nvolumes={&quot;/findings&quot;: volume},\ntimeout=300,\ncpu=(0.125, 2.0),\nmemory=(256, 4096),\nmax_containers=350,\n)\ndef scan(repo_url: str):\nrepo_name = repo_url.rstrip(&quot;/&quot;).split(&quot;/&quot;)[-1].removesuffix(&quot;.git&quot;)\nclone_path = f&quot;/tmp/{repo_name}.git&quot;\nssh_url = repo_url.replace(&quot;<a href=\"https://gitee.com/\" rel=\"nofollow ugc noopener\">https://gitee.com/</a>&quot;, &quot;git@gitee.com:&quot;)\nsubprocess.run(\n[&quot;git&quot;, &quot;clone&quot;, &quot;--bare&quot;, &quot;--quiet&quot;, ssh_url, clone_path],\ncapture_output=True, check=True,\n)\nth_proc = subprocess.Popen([\n&quot;trufflehog&quot;, &quot;git&quot;, &quot;--bare&quot;, f&quot;file://{clone_path}&quot;,\n&quot;--json&quot;, &quot;--only-verified&quot;, &quot;--allow-verification-overlap&quot;,\n], stdout=subprocess.PIPE, text=True)\nkf_proc = subprocess.Popen([\n&quot;kingfisher&quot;, &quot;scan&quot;, clone_path,\n&quot;--format&quot;, &quot;jsonl&quot;, &quot;--only-valid&quot;,\n], stdout=subprocess.PIPE, text=True)\nth_output = th_proc.communicate()[0].strip()\nkf_output = kf_proc.communicate()[0].strip()</p>\n<h1 id=\"write-hits-to-the-shared-findings-volume\">... write hits to the shared findings volume</h1>\n<h3 id=\"dispatching-7-25m-repos-in-batches\">Dispatching 7.25M repos in batches</h3>\n<p>I split the corpus into 50 batches of ~150K repos and ran them sequentially at a steady 350 concurrent containers. Smaller batches were safer at this scale and easier to resume after interruptions; running larger batches was unstable and would hit clone-side blocks.</p>\n<h3 id=\"cost-and-scale\">Cost and scale</h3>\n<p>Total wall time: roughly one month of continuous execution. Per-repo clone times were a major contributor: Gitee&#39;s servers sit in China, and cross-Pacific network distance plus SSH protocol overhead made each clone materially slower than an equivalent GitHub clone would be. The total cost was $3,097.64, fully absorbed by the $5,000 compute-credit sponsorship from Modal that supported this research. The same Modal + dual-scanner architecture works against most Git hosts with little more than a clone-URL substitution. I used a smaller-scale version of this approach in a <a href=\"https://benzimmermann.dev/blog/algolia-docsearch-admin-keys\" rel=\"nofollow ugc noopener\">prior investigation of Algolia DocSearch admin keys</a>.</p>\n<h2 id=\"the-results\">The results</h2>\n<p>Across the 7,256,531 public Gitee repositories scanned, the dual-scanner pipeline surfaced <strong>26,400 unique, verified-live credentials</strong> after deduplicating by token value.</p>\n<p>Both scanners authenticate each candidate against its vendor&#39;s API at scan time (TruffleHog&#39;s --only-verified and Kingfisher&#39;s --only-valid modes), discarding anything that pattern-matches but fails to authenticate. The 26,400 figure is deduplicated by raw secret value, so duplicates across repositories or scanners count once.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Scanner</th><th>Unique Findings</th></tr></thead><tbody><tr><td>TruffleHog</td><td>12,737</td></tr><tr><td>Kingfisher</td><td>14,664</td></tr><tr><td>Caught by both</td><td>1,001</td></tr><tr><td>Total unique credentials</td><td>26,400</td></tr></tbody></table></div>\n<p>Either scanner alone would have caught only about half of these credentials. The two tools&#39; detector coverage was largely complementary: TruffleHog dominated Alibaba Cloud key detection (9,428 vs. 993), while Kingfisher dominated DeepSeek (2,940 vs. 230), OpenAI (1,198 vs. 150), HuggingFace (154 vs. 1), and Chinese-AI services like Zhipu (1,489 vs. 0) that TruffleHog has no detector for.</p>\n<h3 id=\"secrets-by-type\">Secrets by type</h3>\n<p>The single most common leaked credential type was Alibaba Cloud AccessKey pairs: roughly <strong>1 in every 766 Gitee repositories</strong> contained a verified-live Alibaba Cloud credential.</p>\n<p><strong>Figure 2.</strong> Credentials by vendor.</p>\n<figure><img src=\"https://images.contentstack.io/v3/assets/blt7151619cb9560896/blt31af8a3bc3afccd4/6ae0bc6109de0e19ab7ad120/figure3.png\" alt=\"A bar chart illustrating the number of unique verified-live credentials found per vendor, led by Alibaba Cloud (9,473), YouTube API (3,805), and DeepSeek (3,170), with a long tail of other service providers.\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<p>A long tail of additional vendors (LangSmith, OpenWeather, Algolia, PostHog, OpenRouter, MySQL, Stripe, Slack, Twitter, and others) accounts for the rest.</p>\n<h3 id=\"secrets-committed-by-date\">Secrets committed by date</h3>\n<p>The vast majority of verified-live credentials were committed in the last two years. 2025 alone accounts for 12,242 unique credentials, nearly half the entire dataset.</p>\n<p><strong>Figure 3.</strong> Unique credentials committed by year.</p>\n<figure><img src=\"https://images.contentstack.io/v3/assets/blt7151619cb9560896/bltfbc653e59fa14cad/6ae0bd2d332c2f7b0ca5d929/figure4.png\" alt=\"A line chart displaying the number of unique verified-live credentials committed each year from 2008 to 2025, showing a sharp exponential increase in recent years and peaking at 12,242 in 2025.\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<h3 id=\"secrets-by-committer-email-domain\">Secrets by committer email domain</h3>\n<p>Most credentials were committed from personal mail services (qq.com, 163.com, gmail.com), making per-company attribution impossible at scale. The corporate-domain numbers below are a floor; even so, the names span major Chinese and Western tech companies alike.</p>\n<p><strong>Figure 5.</strong> Unique credentials by committer email domain.</p>\n<p><img src=\"https://images.contentstack.io/v3/assets/blt7151619cb9560896/blt1b8175fe75482468/6ae0bdd21a8b5a0e7477135d/figure5.png\" alt=\"A bar chart showing the distribution of unique verified-live credentials by committer email domain, led by [link removed] (66), huawei.com (32), microsoft.com (22), and bytedance.com (22), followed by other major corporate domains.\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></p>\n<p>Smaller corporate footprints also appear from Oracle, Apple, Meta, Anthropic, and others, plus Chinese commercial and academic domains.</p>\n<h3 id=\"comparing-gitee-with-gitlab-and-bitbucket\">Comparing Gitee with GitLab and Bitbucket</h3>\n<p>The closest prior data points come from <a href=\"https://trufflesecurity.com/blog/scanning-5-6-million-public-gitlab-repositories-for-secrets\" rel=\"nofollow ugc noopener\">TruffleSecurity&#39;s scan of 5.6 million public GitLab repositories</a>, and a companion Bitbucket scan referenced in that post.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Platform</th><th>Repos scanned</th><th>Verified-live credentials</th><th>Per 1K repos</th></tr></thead><tbody><tr><td>Bitbucket</td><td>~2,600,000</td><td>6,212</td><td>2.39</td></tr><tr><td>GitLab</td><td>~5,600,000</td><td>17,430</td><td>3.11</td></tr><tr><td>Gitee</td><td>7,256,531</td><td>26,400</td><td>3.64</td></tr></tbody></table></div>\n<p>Gitee tops both Western platforms on raw count and per-repository density. One methodology note: the TruffleSecurity scans used TruffleHog alone, while this one used TruffleHog and Kingfisher together. TruffleHog&#39;s detector library skews toward Western vendors, and Gitee has heavy Chinese-vendor activity (Alibaba Cloud, DeepSeek, Zhipu, and others) that Kingfisher&#39;s coverage helps to surface; stripping our findings to TruffleHog-only puts Gitee at 1.76 per 1,000 repos.</p>\n<h2 id=\"disclosure\">Disclosure</h2>\n<p>I emailed the vendors with the highest exposure volume in this scan, providing each a coordinated report containing the full list of verified-live credentials associated with their service, along with source-repository metadata to help them attribute affected customers.</p>\n<p>Per-developer notification (contacting the committer of each leaked credential individually) was not pursued at this scale. Vendor-side revocation is the more reliable path, since it doesn&#39;t require any action from the user who leaked the credential.</p>\n<p>At the time of writing, revocation status for the disclosed credentials has not yet been confirmed.</p>\n<h2 id=\"summary\">Summary</h2>\n<ul><li><strong>Gitee is a meaningful credential-exposure surface.</strong> 26,400 verified-live credentials across 7.25M repositories. Per-repository density higher than comparable published scans of GitLab and Bitbucket.</li><li><strong>Major tech companies appear in the committer metadata, Chinese and Western alike.</strong> The breadth includes Huawei, ByteDance, Google, Microsoft, Oracle, and others, suggesting this is not a purely-Chinese-developer phenomenon.</li><li><strong>Disclosure paths are asymmetric.</strong> Western companies typically have bug bounty programs or accessible security channels. Most Chinese companies don&#39;t, leaving leaked credentials without a clear remediation path.</li><li><strong>The architecture is portable.</strong> The same Modal + dual-scanner pipeline works on most Git hosts with little more than a clone-URL substitution; the full scan cost roughly $3,000.</li></ul>\n<hr />\n<p><em>Originally published at <a href=\"https://www.mongodb.com/company/blog/technical/scanning-7-point-25-million-public-repositories-chinas-largest-git-platform-for-secrets\" rel=\"nofollow ugc noopener\">the MongoDB Blog</a>.</em></p>","headings":[{"level":2,"text":"What is Gitee?","id":"what-is-gitee"},{"level":2,"text":"Enumerating millions of Gitee repositories","id":"enumerating-millions-of-gitee-repositories"},{"level":3,"text":"Bootstrapping a username corpus","id":"bootstrapping-a-username-corpus"},{"level":3,"text":"Scraping per-user pages on the frontend","id":"scraping-per-user-pages-on-the-frontend"},{"level":3,"text":"Querying the on-site search widget","id":"querying-the-on-site-search-widget"},{"level":3,"text":"Escaping the 200-result cap with bucket splitting","id":"escaping-the-200-result-cap-with-bucket-splitting"},{"level":1,"text":"After fetching page 1 for a username:","id":"after-fetching-page-1-for-a-username"},{"level":1,"text":"Within the cap: paginate normally","id":"within-the-cap-paginate-normally"},{"level":1,"text":"Capped: split by language buckets that Elastic already returned","id":"capped-split-by-language-buckets-that-elastic-already-returned"},{"level":2,"text":"Wiring up the scan pipeline","id":"wiring-up-the-scan-pipeline"},{"level":3,"text":"The Docker image","id":"the-docker-image"},{"level":3,"text":"The per-repo scan function","id":"the-per-repo-scan-function"},{"level":1,"text":"... write hits to the shared findings volume","id":"write-hits-to-the-shared-findings-volume"},{"level":3,"text":"Dispatching 7.25M repos in batches","id":"dispatching-7-25m-repos-in-batches"},{"level":3,"text":"Cost and scale","id":"cost-and-scale"},{"level":2,"text":"The results","id":"the-results"},{"level":3,"text":"Secrets by type","id":"secrets-by-type"},{"level":3,"text":"Secrets committed by date","id":"secrets-committed-by-date"},{"level":3,"text":"Secrets by committer email domain","id":"secrets-by-committer-email-domain"},{"level":3,"text":"Comparing Gitee with GitLab and Bitbucket","id":"comparing-gitee-with-gitlab-and-bitbucket"},{"level":2,"text":"Disclosure","id":"disclosure"},{"level":2,"text":"Summary","id":"summary"}]}}