The obvious replacement, an open NER model called GLiNER run zero-shot, cut the cost to nothing and the accuracy to about 0.65 F1 against Gemini's answers. That gap is what the rest of this is about: could Gemini label 4,290 comments once and teach GLiNER to close it?

• What: Fine-tuned GLiNER large v2.5 (459M) to tag brands, models and materials in Reddit comments, on labels Gemini 3.1 Pro wrote once.
• Why: Zero-shot GLiNER scored about 0.65 F1 (est.). Gemini scored well and billed every comment for as long as the scraper ran.
• Approach: Ask Gemini for strings, not offsets. Compute offsets in code. Add comments with no products in them as negatives. Lock a 225-comment validation set before the second run.
• Problems: Five of ten runs produced no usable model. Three failed on configuration. Two failed on a tensor called words_mask that I filled the way you fill an attention mask.
• Result: 0.83 F1 against Gemini's labels after 24 minutes on a Tesla T4. $9 of labels, about $2.50 of GPU time, and days of debugging.

What I set out to do

The plan had three steps. Have Gemini label a few thousand Reddit comments once, marking every brand, model and steel. Train GLiNER on those labels. Then run GLiNER on my own machine for every comment after that, and stop calling Gemini.

Gemini labeled 4,290 comments for $9, or $0.0021 a comment. That means the trained model pays for itself at roughly comment 4,291, as long as later comments are about the same length and it runs on a GPU I already own. The test of success was simple: on 225 comments the model had never seen, how often does it tag the same words Gemini tagged? One catch to keep in mind for every score in this article. Nobody checked Gemini's labels by hand, so the model is graded against Gemini, not against the truth. Where Gemini was wrong, the model gets marked right for copying the mistake and wrong for fixing it.

The approach

Gemini labeled the comments through OpenRouter at temperature 0 in 25 minutes. The prompt decision that mattered most was to never ask the model for character offsets. It counts characters badly and returns spans off by two or three positions. The prompt asks for the exact substring and a label, and TypeScript finds the offsets. If the string is not in the comment, the entity is dropped and logged.

// The model returns strings. Code computes the offsets.

{ "entities": [

{ "text": "Benchmade", "label": "knife brand" },

{ "text": "940", "label": "knife model" },

{ "text": "S30V", "label": "knife steel" }

] }

Product names are full of punctuation a generic tokenizer splits, so a regex keeps VG-10, CPM-154 and 1.4116 whole and emits every other non-space character as its own token. Spans that still miss a token boundary are dropped rather than guessed. About 30% of the training set is comments that contain a known false-positive trigger (gyuto, carbon, handle, patina) and no product, labeled as empty. Before the second run I set aside 225 comments as a validation set and never touched them again. Training ran on a Tesla T4 on Modal with the HF Trainer.

per_device_train_batch_size = 2

gradient_accumulation_steps = 8

learning_rate = 1e-5

threshold = 0.45

What went wrong

For five runs the model learned nothing. The first three failed on configuration, and anyone using the HF Trainer with GLiNER will hit them in an afternoon.

| Run | What went wrong | Fix |

|---|---|---|

| 1 | GLiNER's default max_steps=10000 overrode num_train_epochs=3; trained 39 epochs | Set max_steps explicitly |

| 2 | load_best_model_at_end without eval_strategy throws | Set eval_strategy="steps" |

| 3 | Trainer saved state-dict keys without the "model." prefix GLiNER's loader expects | Put the prefix back on save |

| 4 | ner_labels missing on negative examples | Set the label list on every example |

| 4–5 | words_mask built as binary; loss flat at 70–130 | Emit incremental word indices |

Runs 4 and 5 were the expensive ones. GLiNER's tokenize_inputs crashed on broken Reddit emoji, so I had patched it, and the patch has to fill a tensor called words_mask. It sits next to attention_mask, has the same shape, and every attention mask I have ever built is ones for real tokens and zeros for padding. I built it that way. Nothing about the name or the shape says it is anything other than an attention mask.

Training ran to completion. Loss started around 130, drifted to about 70 and stayed there. No crash, no warning, no NaN, gradients of ordinary size, checkpoints saved on schedule, eval F1 near zero. I blamed the label list first, because run 4 also had negatives with no labels set. Fixing that and rerunning gave the same flat loss. The only thing wrong with run 5 was words_mask, a tensor I had never looked at.

How I fixed it

I read GLiNER's training loop instead of its docstrings. words_mask is not a mask. It is a word index: 0 for special, prompt and padding tokens, then 1, 2, 3 for the first sub-token of each real word. The span-scoring head uses it to pool sub-tokens back into words. Filled with ones it says the whole comment is a single word, so the model is asked to find brand and material spans inside one enormous token. It cannot, so the loss stays flat, and a flat loss does not show which input is wrong.

what I wrote # what GLiNER expects

words_mask = [1,1,1,1,1] words_mask = [0,1,2,2,3]

[CLS] Mazaki wh ##ite #2

With the index fixed, run 6 learned on the first try. The rest was tuning against the locked set. The 209M medium model reached 0.800; the 459M large model, which fits a T4 only with gradient accumulation, reached 0.83. Ten times more adversarial negatives (510 instead of 51) dropped F1 to 0.799, so run 10 went back to 51. One threshold per class instead of a global cutoff took material recall from 0.787 to 0.911, because steel names like MagnaCut, S35VN and HAP40 score lower confidence than brands and a single cutoff dropped them. Every large run bottoms out at epoch 2 and overfits after; with 2,000 examples that is a dataset-size problem, and early stopping is the fix.

What I learned

It worked. The model runs locally, matches Gemini's labels at 0.83 F1 on comments it never saw, and knows that "carbon steel" is a category rather than a steel and that PM2 sometimes means the Spyderco Paramilitary 2 and sometimes is just letters. One earlier run scored 0.879 on a random split. I do not count it as the result: on random splits, two drops in F1 that I had blamed on my changes turned out to come from which comments landed in the validation set.

On paper the project cost less than lunch: $9 of labels, $2.50 of GPU. What it cost me was the days spent on a tensor that passed every check the code had and was still wrong. I think the lost days are the normal case for small fine-tuning jobs. The model and the data are rarely the problem; the code between them fails, and a flat loss from a wrong input tensor looks the same as a flat loss from hard data. If I had to choose between a better label set and an assertion on every tensor I hand-build, I would take the assertion.

This model powers New Knife Day, which tracks what knife people on Reddit are buying and arguing about. The knife-side write-up there has the full run log. Both are linked below.

At a glance

• Problem
• Find the brand, model and material names in Reddit comments, and skip the generic words around them, without paying an LLM per comment
• Approach
• Have Gemini label the comments once, then fine-tune GLiNER large v2.5 (a DeBERTa-v3-large encoder) on those labels and run it locally
• Result
• 0.83 F1 on a 225-comment validation set fixed before training; material recall 0.911 with a per-class threshold
• Cost
• $9 in Gemini labels plus about $2.50 of T4 time across ten runs
• Stack
• TypeScript and MongoDB for the scraper and labels, Python and PyTorch for training on Modal, FastAPI to serve the model