--- title: "Token-space font compiler" slug: token-space-font-compiler url: https://listedarticles.com/articles/token-space-font-compiler canonical_url: https://ampdot.mesh.host/token-space-fonts.html content_type: blog_post language: en published_at: 2026-09-27T03:09:40.436Z updated_at: 2026-09-27T03:09:40.436Z author: "ampdot" authored_by: human publisher: "ampdot" publisher_url: https://ampdot.mesh.host topics: ["LLMs", "Programming", "Developer Tools"] license: all-rights-reserved word_count: 1527 reading_minutes: 7 citation: "ampdot, ampdot. \"Token-space font compiler.\" 27 Sept 2026. https://ampdot.mesh.host/token-space-fonts.html (all-rights-reserved)" # The full text follows. The web page shows an extract and sends readers # to the source above; quote the citation and link the canonical URL. --- # Token-space font compiler > A font compiler that makes every LLM token the same width—why monospace-per-token helps visualize chain-of-thought, plus an interactive preview of fonts built from a font + tokenizer pair. # Token-space font compiler font + tokenizer → token-space font, where each token is equal width ## Try it! Type or paste your text above. Compiling a new font switches this preview to it. ## Compile your own [Download font](https://ampdot.mesh.host/token-space-fonts/compiler/default.ttf?v=noto6) · [CSS](https://ampdot.mesh.host/token-space-fonts/compiler/default.css) · [Build report](https://ampdot.mesh.host/token-space-fonts/compiler/default-build.json) ## Use it in Discord with Vesktop Theme font: DeepSeek V4.1 Flash × Inter Leave blank to apply to all messages. We strongly recommend entering a specific em’s ID so only its messages use the token font. ## Setup instructions 1. To use gg sans, select **gg sans (Discord)** above, choose a tokenizer preset or upload, and click**Compile font** . The theme download always matches the font currently in the preview. 2. Download the font and open the TTF in Font Book. Click **Install** . Install the compiled font, even if you already have ordinary gg sans. 3. We strongly recommend entering a specific em’s Discord ID above; leaving it blank applies to all messages. Download the `.theme.css` file. To find an ID, enable Discord’s**Settings → Advanced → Developer Mode** , then right-click the user and choose**Copy User ID** . 4. If you entered an ID, open Vesktop’s **Settings → Vencord → Plugins** and enable[ThemeAttributes](https://vencord.dev/plugins/ThemeAttributes) . Restart if prompted. 5. Open **Settings → Vencord → Themes → Local Themes → Open Themes Folder** . Move the downloaded`.theme.css` file into that folder. Return to Themes, click**Load Missing Themes** if needed, and enable the theme. 6. Fully quit Vesktop with **⌘Q** and reopen it so it sees the installed font. Keep Vencord Location at its current setting. This is a theme, so no custom Vencord build or UserFonts plugin is needed. Disable older font themes for the same user if they conflict. If messages look unchanged, check that the compiled font is installed and this theme is enabled. If targeting an ID, also check that ThemeAttributes is enabled and the ID is correct. Code blocks keep Discord’s normal monospace font. Extra token spacing defaults to zero; line height and word spacing follow Discord. ## Use it in Slack Slack font: DeepSeek V4.1 Flash × Inter Use the exact display name shown above each agent’s messages, or a Slack member ID such as U012345678. One per line. Only matching messages change. ## Setup instructions 1. Download the font above and install its TTF on the device where you read Slack. 2. Enter the agents’ Slack display names or member IDs, then download the `.user.js` file. 3. Import the file into a userscript manager in your browser, enable it, and reload web Slack. The userscript reads sender labels in the page and styles matching message text, including new messages and threads. It makes no network requests and does not modify message text. Code blocks keep a monospace font. If you compile another font, download and install that font and its newly generated userscript together. Disable an older Token Mono Slack script if it targets the same agent. This works in web Slack in a browser with a userscript manager. Browser shaping runs and formatting spans can change token boundaries. ## What works? The listed presets use tokenizer-specific compiler paths. These include DeepSeek, OpenAI, Kimi, Qwen, GLM, Llama 3, Trinity and Laguna; Gemma and Gemini have experimental space-marker/byte-fallback support. ctok uses a separate minimum-cost backend. Raw ByteLevel BPE is also accepted with `use_regex: false` and `add_prefix_space: false`. Uploaded pipelines must match a supported profile; arbitrary normalizers, WordPiece and Unigram are not generally supported. Acceptance does not imply exact tokenization for every input. The output is a single TTF with the tokenizer built into its shaping rules. No tokenizer script is needed to display it. It uses your font’s default variation and character coverage. Missing characters may use missing-glyph outlines or system fallback unless you enable fallback fonts, which embeds selected fallback outlines in the same token font. Coverage still depends on the selected sources. Noto Emoji is the default emoji fallback; Mutant Standard is an optional color source with separate license requirements. The precompiled DeepSeek + Inter preview already includes Noto Emoji. Within a supported shaping run, each emitted token occupies 3 em. Extra token spacing defaults to zero and is capped at 0.5 em. The generated CSS leaves word spacing and line height alone. Browser tab stops, hard line breaks and whitespace processing do not behave like ordinary token cells. N.B. Browser shaping runs, line breaks, normalization and complex scripts can change the result. Supported horizontal pair kerning is retained, but contextual shaping is not universally preserved. This is a token font, not an exact whole-message token counter. ## Known discrepancies and experimental features The latest whitespace audit compared 17,904 BPE checks across six available font binaries and 2,355 ctok cases against local tokenizer references. Browser checks covered 60 widths and 20 wrapping samples. These are targeted tests, not a guarantee for all text; failures and coverage/capacity limits remain in the reports. - **DeepSeek whitespace:** U+001C–U+001F controls are incorrectly classified as whitespace. For example, two spaces followed by U+001C merge into one space token. The directed audit found 59 ID mismatches in 2,984 checks. Ordinary isolated whitespace runs and CRLF cases passed; the control-separator bug remains unfixed. - **GLM-5.3 and Llama 3:** the experimental presets use font-only regex boundaries, ranked BPE and whole-piece shortcuts. The tested Gutenberg passages and covered prose, code, whitespace and punctuation fuzz cases pass. Other pieces have a 64-byte / 32-round limit and display`[BPE limit]` if exceeded. The new directed whitespace audit reached that limit in 353 GLM and 354 Llama checks out of 2,984 each; the other checks passed. Browser shaping, source glyph coverage and formatting boundaries remain limitations. - **Trinity Large Thinking and Laguna M.1:** experimental presets preserve the original preprocessing pipeline and ordered merges. The compiler validates those profiles instead of reducing them to raw BPE. Both passed all 2,984 directed whitespace checks. Browser shaping, source glyph coverage and formatting boundaries remain limitations; special tokens and automatic message framing are excluded. - **Hugging Face links and uploaded tokenizers:** accepting a file does not establish compatibility with every tokenizer pipeline. Unsupported pre-tokenizers, normalizers and model types may be rejected; arbitrary uploads have not received the preset-specific audits. - **Claude / ctok:** ctok is an unofficial approximation, not a verified implementation of Claude’s tokenizer. The full ctok vocabularies now fit, but the shipped minimum-cost backend has a 256-step search limit per connected component and displays`[limit]` on overflow. Long wrapped text can still render slowly. Invisible characters such as zero-width space, word joiner, BOM and soft hyphen can suppress word boundaries and add a token. v3/v4.7 also have quote-framing and leading-space normalization errors. Of 785 directed cases per version, v3 had 29 count/width and 45 semantic-only failures; v4.7 had 25 and 49; v4.8 had 16 and 8. Ordinary whitespace and runs through 257 characters passed. Equal counts alone can hide incorrect normalization. These failures remain unfixed; passing ctok comparisons does not establish agreement with Claude, and this audit made no Claude calls. - **Normalization and Unicode:** a renderer can normalize text before the font sees it, changing token boundaries. General combining sequences remain limited. Qwen’s NFC support handles at most eight consecutive nonstarters after decomposition and displays`[NFC limit]` beyond that. Browser and native compiler Unicode versions can also differ for newly assigned characters. - **Rendering and font coverage:** fonts cannot tokenize across separate shaping runs. Formatting, links, script changes, bidirectional text, fallback fonts and hard breaks can split those runs. Missing source-font glyphs and fallback may defeat equal token widths; complex-script shaping, ligatures and mark positioning are not universally preserved. Single-run tests do not guarantee identical results in every browser or app. - **Browser whitespace and wrapping:** a leading space can form a separate shaping run under`pre-wrap` or`break-spaces` . At 24px, `Hello world` measured 216px rather than the whole-run 144px. Of 60 width comparisons, 43 matched, 15 differed because of tab/newline layout, and two exposed that leading-space boundary. Tested trailing/repeated spaces, NBSP, narrow NBSP and ideographic space matched whole-run widths. The earlier duplicated-token wrap example did not repeat in the checked 180px/320px screenshots, although wrapping can change tokenization. This does not establish correctness in every browser. - **Emoji and invisible characters:** the current default Noto font passed 5,250 supported emoji checks for token identity, spacing and drawing. Missing artwork, private-use characters and emoji/text run boundaries remain limitations. Spaces and control characters in uploaded emoji sources are now excluded from emoji joining, fixing false emoji classification and glyph-count inflation. Non-emoji split-byte text and complex scripts remain limited. - **Message counts:** special tokens, chat templates, multimodal content and automatic message overhead are not universally represented. Empty text cannot draw a framing token. These fonts visualize ordinary text, not exact API billing counts. See the [whitespace audit](https://ampdot.mesh.host/token-space-fonts/compiler/whitespace-audit.md), [ctok reproductions](https://ampdot.mesh.host/token-space-fonts/compiler/ctok-whitespace-findings.md), and [detailed discrepancy tracker](https://ampdot.mesh.host/token-space-fonts/compiler/discrepancies.md) for test results, repaired issues and remaining limitations. Compiled locally with [Pyodide](https://pyodide.org/en/0.29.2/) and [fontTools](https://fonttools.readthedocs.io/). [Inter license](https://ampdot.mesh.host/token-space-fonts/licenses/OFL-Inter.txt). [Noto Emoji license](https://ampdot.mesh.host/token-space-fonts/licenses/OFL-NotoEmoji.txt). Optional Mutant Standard emoji by Caius Nocturne: [CC BY-NC-SA 4.0](https://ampdot.mesh.host/token-space-fonts/compiler/fonts/emoji/LICENSE-MutantStandard.txt). [1](https://ampdot.mesh.host#font-expertise-reference) If you have more expertise in how fonts work, I encourage you to implement this project properly.