let v = VALENCE[word] ?? 0; // great -> 0.75 if (negatorWithin(3)) v = -v * 0.74; // not great -> -0.55 v *= gain; // really great -> 0.98

Salience is how much the word is worth looking at, 0 to 1. A frequency list gives each word a rarity, 0 for one of the hundred most common English words and 1 for one it has never seen. Rarity alone is not enough, so the score also rises with how often the word repeats in this particular text: an uncommon word you keep saying is what the text is about.

`const seen = Math.min(1, (repeats - 1) / 2); const repetition = 0.45 + 0.55 * seen; let s = SALIENCE[word] ?? 0; s = Math.max(s, 0.55 * rarity * repetition);

// rarity('the') 0.00, rarity('kubelet') 0.93 // kubelet said once -> 0.23 // kubelet said 3 times -> 0.51 `

Surprise is where the sentence turns, 0 to 1. Some words announce it on their own, like suddenly or ironically. Otherwise it comes from position: everything for six words after a contrast word gets it, decaying with distance, and so does any word much rarer than the rest of the passage.

`let s = SURPRISE[word] ?? 0; // suddenly -> 0.85 if (afterContrast) { s = Math.max(s, 0.45 * 0.82 ** (distance - 1)); } s += 0.3 * Math.max(0, rarity - passageMeanRarity - 0.25);

// 'The tests failed' -> failed 0.16 // 'It compiled, but the tests failed' -> failed 0.44 `

Certainty is how sure the writer sounds, -1 hedged to 1 asserted. Words like probably and definitely are in a table. But if you write The build probably failed, you are not unsure about the word probably, you are unsure about whether it failed. So the hedge keeps its own score and every other word in the sentence gets 55% of it, and the whole line leans a little instead of one word in the middle of it.

`c = CERTAINTY[word] ?? sentenceCertainty * 0.55;

// 'The build probably failed.' // probably -0.40, every other word -0.22 `

Then a theme maps each score to one typographic axis: valence to colour, salience to weight, surprise to a highlight, certainty to slant. Each axis has a threshold, so most words come out untouched.

Improving the algorithm

Those rules only look a few words either side, and that window has a blind spot. Write I would not go so far as to call the new editor great and great stays green, because the not that cancels it sits nine words back. Write We fixed the crash and you get one green word and one red one, because nothing connects fixed to the thing it fixed.

So a second pass now runs after the window rules and reads each clause as a whole. A negator reaches to the end of its clause and fades with distance, which turns great red. A verb like fixed, recovered or avoided marks whatever s it as the thing that got better, which turns crash green. The same pass reads less broken and fewer complaints as improvements, too simple as a complaint, and a lone Great, in front of bad news as sarcasm.

Every change it makes is recorded on the word, so you can ask why a word came out the colour it did:

analyze('We fixed the crash.').tokens[6]; // { text: 'crash', valence: 0.44, // notes: ['resolved by "fixed"'] }

It costs about as much as the first pass and stays inside the budget, so there is no switch to flip. These are the ten sentences that led to it, including those two. Left is the first version, right is now.

beforenow Great, another outage. Just what I needed today.Great, another outage. Just what I needed today. We fixed the crash and closed the security hole before anyone noticed.We fixed the crash and closed the security hole before anyone noticed. The cluster recovered from the crash in under a minute.The cluster recovered from the crash in under a minute. We avoided a catastrophic outage by catching the bug in staging.We avoided a catastrophic outage by catching the bug in staging. I would not go so far as to call the new editor great.I would not go so far as to call the new editor great. Less broken than last week, and far fewer complaints.Less broken than last week, and far fewer complaints. The memory leak is gone.The memory leak is gone. The reviewer called it "terrible", which is wrong.The reviewer called it "terrible", which is wrong. The API is too simple and the docs are too clever.The API is too simple and the docs are too clever.

Using it

`import { SemanticText } from 'semfont';

<SemanticText as="p"> The migration ran clean on staging. In production it deleted the index, and the rollback failed too. </SemanticText> `

Pick a theme, or only the channels you want:

<SemanticText text={incident} theme="monochrome" /> <SemanticText text={incident} channels={['valence']} />

Teach it your own vocabulary:

<SemanticText lexicon={{ valence: { flaky: -0.7, oncall: -0.4 }, salience: { rollback: 0.8 } }} text={incident} />

Now for the case I started this for. AI system stream in a large amount of text and its hard to read all of it so the intention of this is a library that can easily be tossed into most streaming components to make the text easier to read:

`import { useChat } from '@ai-sdk/react'; import { SemanticText } from 'semfont';

function Chat() { const { messages } = useChat(); return messages.map((m) => { const text = m.parts.filter((p) => p.type === 'text').map((p) => p.text).join(''); return m.role === 'assistant' ? <SemanticText key={m.id} as="p" text={text} /> : <p key={m.id}>{text}</p>; }); } `

plain text

semfont

Or skip React and take the scores. analyze is the engine alone, four numbers per word, no CSS, and these imports work with no React installed:

`import { analyze } from 'semfont/analyze'; import { styleFor, themes } from 'semfont/theme';

const { tokens } = analyze('The rollback failed too.'); tokens[4]; // { text: 'failed', valence: -0.7, salience: 0.23, surprise: 0.06, certainty: 0, ... } styleFor(tokens[4], themes.editorial).style; // { color: 'color-mix(in oklab, currentColor, oklch(0.58 0.19 25) 53%)' } `

That last form is how this page works. There is no bundler here, so one import map tells the browser where semfont/analyze and semfont/theme live, pinned to a version on npm, and the demo box above is the same three lines as the React component: analyze, style, render.

<script type="importmap"> { "imports": { "semfont/analyze": "https://cdn.jsdelivr.net/npm/semfont@0.2.0/src/analyze.js", "semfont/theme": "https://cdn.jsdelivr.net/npm/semfont@0.2.0/src/theme.js" } } </script> <script type="module"> import { analyze } from 'semfont/analyze'; import { styleFor, themes } from 'semfont/theme'; </script>

Next Steps

As you can probably guess based on the implementation it will be essentially impossible to get perfect classification while also being fast enough to not slow down the streaming. However for the purpose of making text easier to read it doesn't have to be perfect and some simple heuristics may end up taking us far enough away. That being said there are some case like sarcasm which would be interesting to try to tackle with heuristics and some cases like negation with embedded clauses which may be possible to parse out. Furthermore, for streaming coding agents there are technical words which could be worth including in the vocabulary.

Code and demo at github.com/RohanAdwankar/semfont.