The claim behind TabPFN and TabICL is that they predict on a table without ever training on it and still beat tuned boosting. I measured it on fourteen datasets from the Grinsztajn benchmark, with the same split and the same clock for everyone. The one that does not train wins, the advantage holds up to 32,000 rows instead of breaking, and the most-cited model can no longer be downloaded without an account.
\nThe claim has been going around for months and it is concrete enough to be measurable: a tabular foundation model predicts on a table without ever having trained on it and still beats tuned boosting.
\nIf that is true, half a decade of practice changes shape. Searching hyperparameters stops being a mandatory step and becomes a luxury that sometimes does not pay off.
\nSo I put it to the test on my own card, with fourteen datasets, four contenders and the same stopwatch for everyone.
\nA tabular foundation model is pretrained on millions of synthetic tables generated on purpose. When a new table arrives, it does not adjust a single weight: it receives the training rows as context and produces the predictions in one forward pass.
\nIt is the same in-context learning idea we already know from language models, moved from words to columns. That is why the verb “train” sits oddly: the code still calls it fit, but inside there is no gradient descent, there is a copy of data to the card.
That also explains why the cost shows up where you do not expect it. Fitting is nearly free and prediction is what pays, exactly the reverse of a tree.
\nPut that way it sounds abstract, so here is the full journey of one row: a request arrives with an empty cell, the model weighs it against everything that already happened, resolves it in one pass and returns a future. Pick any of the six cases I measured to see it with their real data:
\nThe request
\nThe context
\nA single pass
\nThe prediction
\nA credit application arrives
\n2 100 applications already settled · 10 columnsclf_num/credit.csv
TabICL · nothing tuned · 0.8 s
\nWill they repay?
\nrepaysdefaults
\nAUC 0.8528
\ncredit
\nA contact enters the list
\n2 100 calls already made · 7 columnsclf_num/bank-marketing.csv
TabICL · nothing tuned · 0.8 s
\nIs it worth calling them?
\nsigns updoes not
\nAUC 0.8699
\nbank-marketing
\nA patient is discharged
\n2 100 previous discharges · 7 columnsclf_num/Diabetes130US.csv
TabICL · nothing tuned · 0.8 s
\nWill they be readmitted?
\nreturnsdoes not
\nAUC 0.6484
\nDiabetes130US
\nA case is assessed
\n2 100 cases already closed · 11 columnsclf_cat/compas-two-years.csv
TabICL · nothing tuned · 0.6 s
\nWill they reoffend?
\nreoffendsdoes not
\nAUC 0.7329
\ncompas-two-years
\nA market period closes
\n2 100 previous periods · 7 columnsclf_num/electricity.csv
TabICL · nothing tuned · 0.8 s
\nDoes the price go up or down?
\nupdown
\nAUC 0.8873
\nelectricity
\nA candidate molecule arrives
\n2 100 molecules already assayed · 419 columnsclf_num/Bioresponse.csv
TabICL · nothing tuned · 6.0 s
\nDoes it trigger a biological response?
\nactiveinert
\nAUC 0.8667
\nBioresponse
\nThe request
\nThe context
\nA single pass
\nThe prediction
\nA credit application arrives
\n2 100 applications already settled · 10 columnsclf_num/credit.csv
TabICL · nothing tuned · 0.8 s
\nWill they repay?
\nrepaysdefaults
\nAUC 0.8528
\ncredit
\nA contact enters the list
\n2 100 calls already made · 7 columnsclf_num/bank-marketing.csv
TabICL · nothing tuned · 0.8 s
\nIs it worth calling them?
\nsigns updoes not
\nAUC 0.8699
\nbank-marketing
\nA patient is discharged
\n2 100 previous discharges · 7 columnsclf_num/Diabetes130US.csv
TabICL · nothing tuned · 0.8 s
\nWill they be readmitted?
\nreturnsdoes not
\nAUC 0.6484
\nDiabetes130US
\nA case is assessed
\n2 100 cases already closed · 11 columnsclf_cat/compas-two-years.csv
TabICL · nothing tuned · 0.6 s
\nWill they reoffend?
\nreoffendsdoes not
\nAUC 0.7329
\ncompas-two-years
\nA market period closes
\n2 100 previous periods · 7 columnsclf_num/electricity.csv
TabICL · nothing tuned · 0.8 s
\nDoes the price go up or down?
\nupdown
\nAUC 0.8873
\nelectricity
\nA candidate molecule arrives
\n2 100 molecules already assayed · 419 columnsclf_num/Bioresponse.csv
TabICL · nothing tuned · 6.0 s
\nDoes it trigger a biological response?
\nactiveinert
\nAUC 0.8667
\nBioresponse
\nthe foundation model wins\nBanking’s most repeated case: deciding who gets lent to.
\n| dataset | size | TabICL | TabPFN | tuned XGB |
|---|---|---|---|---|
| credit | 3000 × 10 | 0.7667 | 0.7578 | 0.7533 |
| heloc | 3000 × 22 | 0.7222 | 0.7300 | 0.7078 |
| default-of-credit | 3000 × 20 | 0.6956 | 0.6967 | 0.6944 |
All three datasets go to the foundation model. On heloc TabPFN takes 0.022 and TabICL 0.014: in credit risk that is not decoration.
\nContext: clf_num/credit.csv from the inria-soda/tabular-benchmark suite, trimmed to 3,000 rows with seed 0 and split 70/30 with stratification.
sha256 22a759296600c39a56884dafd84eb346be018c537ff096adc8c2ae7e0520f2a9
the foundation model wins\nWho to call first when there are a thousand contacts and time for a hundred.
\n| dataset | size | TabICL | TabPFN | tuned XGB |
|---|---|---|---|---|
| bank-marketing | 3000 × 7 | 0.7944 | 0.7967 | 0.7833 |
The only case where TabPFN ends up ahead of TabICL. Both beat tuned boosting.
\nContext: clf_num/bank-marketing.csv from the inria-soda/tabular-benchmark suite, trimmed to 3,000 rows with seed 0 and split 70/30 with stratification.
sha256 3433fecd416ce949692442882669881746e8a6b24b904a5f808cdcb196443e7e
the foundation model wins\nA real hospital record: which patient gets readmitted.
\n| dataset | size | TabICL | TabPFN | tuned XGB |
|---|---|---|---|---|
| Diabetes130US | 3000 × 7 | 0.5911 | 0.5933 | 0.5900 |
On accuracy they nearly tie, but the area under the curve opens up sharply: 0.6484 against 0.6264. The risk ordering, which is what a triage uses, improves considerably more than accuracy suggests.
\nContext: clf_num/Diabetes130US.csv from the inria-soda/tabular-benchmark suite, trimmed to 3,000 rows with seed 0 and split 70/30 with stratification.
sha256 9be384ad7edbb9a98b509adb9cc08578e63bd87545b05d97d5147aa15b71385a
the foundation model wins\nThe dataset that opened the debate on algorithmic bias in the courts.
\n| dataset | size | TabICL | TabPFN | tuned XGB |
|---|---|---|---|---|
| compas-two-years | 3000 × 11 | 0.6733 | 0.6767 | 0.6700 |
The foundation model wins, and it is worth saying that a better model here does not make the use legitimate: this dataset’s argument was never about accuracy.
\nContext: clf_cat/compas-two-years.csv from the inria-soda/tabular-benchmark suite, trimmed to 3,000 rows with seed 0 and split 70/30 with stratification.
sha256 7e0bcef09ae633ad81e26b0a8f8f96dbc4ac3c29b9da0760d38ef2a75adde8d8
a tie\nConsumption and price series from an electricity market.
\n| dataset | size | TabICL | TabPFN | tuned XGB |
|---|---|---|---|---|
| electricity | 3000 × 7 | 0.8178 | 0.8111 | 0.8178 |
The tie. TabICL matches tuned boosting to the fourth decimal and TabPFN falls below. It is one of the two cases where the advantage does not show up.
\nContext: clf_num/electricity.csv from the inria-soda/tabular-benchmark suite, trimmed to 3,000 rows with seed 0 and split 70/30 with stratification.
sha256 d6005007c4b1f7ba88cf91cb1f231969cce3e49c98d445338c0ab0342ead5d7e
depends on the model\n419 columns of chemical descriptors: the wide table.
\n| dataset | size | TabICL | TabPFN | tuned XGB |
|---|---|---|---|---|
| Bioresponse | 3000 × 419 | 0.7922 | 0.7567 | 0.7744 |
This is where TabPFN breaks: 0.7567 is worse than even untuned XGBoost, and it costs 18.1 seconds. TabICL holds up and comes first. The table’s width, not its length, is what squeezes.
\nContext: clf_num/Bioresponse.csv from the inria-soda/tabular-benchmark suite, trimmed to 3,000 rows with seed 0 and split 70/30 with stratification.
sha256 bd3d277821eb949df41219363f0386549019249776fccf7a9f08d3bec10b8727
The six cases in the diagram are not brochure examples: they are the datasets I measured with. But it is worth widening the map, because “tabular foundation model” sounds like a laboratory and the problem it solves is one of the most common there is.
\nA table is a spreadsheet: rows that are cases and columns that are attributes. And the task is always the same, predict one column from the others:
\ncredit ,heloc anddefault-of-credit .Diabetes130US , another of the datasets, is a real hospital record.That is half the data work done in any company. It is also the ground where deep learning had been losing for years: for tables, a good gradient-boosted tree ensemble was still the right answer, and the research that assembled these datasets was titled exactly that way, asking why trees still won.
\nToday, solving any of those cases has a ritual: prepare the data, choose a model, search hyperparameters, validate, repeat. The search is the boring part and the one that eats machine and human hours. In this benchmark, XGBoost’s search took up to 50 seconds per dataset; on a real problem with more rows and more combinations, it is minutes or hours.
\nA tabular foundation model proposes skipping that ritual entirely. You hand it the table, it answers in a second, and that is that. No tree depth to choose, no learning rate, no cross-validation to decide among twenty-five candidates.
\nPut in concrete terms: instead of spending the afternoon tuning a churn model, you have an answer in the time it takes to make coffee, and only then do you decide whether anything is worth refining. For exploring a table that just arrived, or for having an honest baseline before investing time, it is hard to beat.
\nThe question, then, is not whether the idea is attractive. It is whether the result holds up when measured.
\nA badly built benchmark says whatever you want. These are the rules I imposed on myself before seeing a single number:
\nThe data is third-party and from the field where this argument is fought. I used Grinsztajn’s tabular benchmark, the suite gathered for the work asking why trees still beat deep learning on tables. Choosing the datasets yourself is the easiest way to manufacture the result you like.
\nXGBoost goes in tuned, not for decoration. The claim says “tuned boosting”, so comparing against factory parameters would be a straw man. I ran two versions: one with fixed, reasonable parameters and another with a random search over twenty-five combinations and three-fold cross-validation.
\nSame split, same seed, same columns for everyone. Categoricals are integer-encoded. It is not the best possible treatment, but it is identical for all four, which is what makes the number comparable.
\nFive seeds per dataset, and the median is reported. A single split cannot tell signal from luck.
\nPrediction time includes two inference passes, the class one and the probability one, because the benchmark needs both to compute accuracy and area under the curve. That makes things more expensive precisely for the foundation models, which is where their cost lives, so the seconds I publish for them are inflated in nobody’s favour.
\nTwo cores are left free. The machine has other work on it, and a benchmark that eats the whole CPU measures the fight with the scheduler, not the model.
\nEverything ran on a 16 GB RTX 4070 Ti SUPER, with fourteen cores for XGBoost.
\nPublishing only the final table would be lying by omission. This is what it cost to get there.
\nI set up the environment, ran the benchmark, and the first line said device: cpu. The card was free and visible. The reason:
2.13.0+cu130 driver 570.144 torch.cuda.is_available() → False\nPyTorch had resolved to a build for CUDA 13.0 while the machine’s driver exposes 12.8. Instead of failing, it silently turns the GPU off and carries on with the CPU. The warning only shows up if you inspect torch.cuda.is_available() by hand.
It is fixed by pinning the version to the right index:
\npip install --no-cache-dir \\\n --index-url https://download.pytorch.org/whl/cu128 torch==2.9.1+cu128This is the second time this year the same trap has cost me a whole run. If a GPU benchmark gives suspiciously slow numbers, that is the first place to look.
\nThe original plan was to take the datasets from OpenML, which is the canonical source for these comparisons. It failed completely during the run:
\nhttps://api.openml.org/api/v1/json/data/31 → 504 (16.9 s)\nhttps://www.openml.org/api/v1/json/data/31 → 504 (15.7 s)\nhttps://api.openml.org/api/v1/json/data/features/31 → 504 (16.5 s)Four retries with growing backoff made no difference: it was not a rate problem, it was the gateway being down. I switched the source to the same benchmark’s CSVs hosted on HuggingFace, which resolved in 250 milliseconds. It is an uncomfortable reminder of how much reproducible research depends on a single service.
\nThis is the finding that interests me most, because it is not technical.
\nTabPFN is the name that appears in nearly all coverage of this topic. I installed the current version, 8.3.0, and on the first fit:
TabPFNLicenseError: TabPFN requires a one-time license acceptance\nto download model weights for local inference, but no interactive\nterminal is available.To fetch the weights you have to open a browser, register, accept the license in a tab on the vendor’s site and export an account token. The best-known tabular foundation model stopped being something you install and run.
\nThere are two ways out, and I tried both:
\nFourteen datasets, all trimmed to 3,000 rows so the comparison is homogeneous. Accuracy as the median of five seeds. The seconds column sums fitting and prediction, which is each one’s honest cost:
\n| dataset | rows × cols | TabICL | TabPFN 2.2.1 | XGBoost | Tuned XGB | s TabICL | s tuned XGB |
|---|---|---|---|---|---|---|---|
| bank-marketing | 3000 × 7 | 0.7944 | 0.7967 | 0.7711 | 0.7833 | 0.8 | 1.8 |
| credit | 3000 × 10 | 0.7667 | 0.7578 | 0.7478 | 0.7533 | 0.8 | 2.2 |
| heloc | 3000 × 22 | 0.7222 | 0.7300 | 0.7022 | 0.7078 | 0.8 | 1.6 |
| pol | 3000 × 26 | 0.9844 | 0.9833 | 0.9778 | 0.9756 | 0.8 | 1.0 |
| eye_movements | 3000 × 20 | 0.6100 | 0.6111 | 0.5944 | 0.5833 | 0.8 | 3.0 |
| california | 3000 × 8 | 0.8967 | 0.8978 | 0.8833 | 0.8767 | 0.6 | 1.6 |
| house_16H | 3000 × 16 | 0.8811 | 0.8744 | 0.8722 | 0.8711 | 1.1 | 3.0 |
| MagicTelescope | 3000 × 10 | 0.8778 | 0.8622 | 0.8489 | 0.8456 | 1.3 | 2.4 |
| electricity | 3000 × 7 | 0.8178 | 0.8111 | 0.8156 | 0.8178 | 0.8 | 2.4 |
| Diabetes130US | 3000 × 7 | 0.5911 | 0.5933 | 0.5644 | 0.5900 | 0.8 | 1.4 |
| default-of-credit | 3000 × 20 | 0.6956 | 0.6967 | 0.6878 | 0.6944 | 0.9 | 4.5 |
| compas-two-years | 3000 × 11 | 0.6733 | 0.6767 | 0.6444 | 0.6700 | 0.6 | 0.7 |
| Bioresponse | 3000 × 419 | 0.7922 | 0.7567 | 0.7844 | 0.7744 | 6.0 | 27.2 |
| albert | 3000 × 31 | 0.6500 | 0.6611 | 0.6511 | 0.6611 | 1.8 | 2.6 |
Against tuned XGBoost, by accuracy:
\nAccuracy, however, is a coarse metric: it depends on the threshold and punishes differently depending on how the classes are balanced. Area under the curve is more informative, and there the result gets sharper:
\n| model | AUC wins | mean difference |
|---|---|---|
| TabICL | 14 of 14 | +0.0114 |
| TabPFN 2.2.1 | 13 of 14 | +0.0089 |
TabICL beats tuned XGBoost on every dataset without exception. Even on albert, where it loses on accuracy, it has a better AUC (0.7101 against 0.7094). That detail is exactly why both metrics are worth looking at: accuracy said “loss” where the probability ordering said “win by a hair”.
\nThat said, the enthusiasm needs calibrating. Winning fourteen of fourteen is a strong signal of consistency, not of crushing superiority: the mean difference is one hundredth. Nobody is going to notice that on a dashboard. What does get noticed is the other thing.
\nLook again at the table’s last two columns. TabICL resolves a dataset in under a second without tuning anything. Tuned XGBoost takes between 0.7 and 27.2 seconds searching hyperparameters, and still comes out below.
\nThat is the real argument, and it is not accuracy. It is that the expensive part of the work, the one that consumes human and machine time, simply disappears.
\nThis is where I expected to dismantle the promise. The standard objection to these models is that they only work on toy tables, because the training rows have to fit inside the transformer’s context.
\nI took jannis, which has 57,580 rows, and trimmed it to growing sizes. Three seeds per size:
\n| rows | TabICL | XGBoost | Tuned XGB | s TabICL | s tuned XGB |
|---|---|---|---|---|---|
| 500 | 0.7533 | 0.7733 | 0.7533 | 0.7 | 3.2 |
| 1,000 | 0.7733 | 0.7467 | 0.7533 | 0.9 | 4.9 |
| 2,000 | 0.7800 | 0.7583 | 0.7667 | 1.1 | 9.4 |
| 4,000 | 0.7817 | 0.7692 | 0.7725 | 1.8 | 21.1 |
| 8,000 | 0.7958 | 0.7658 | 0.7583 | 2.6 | 19.7 |
| 16,000 | 0.8090 | 0.7785 | 0.7852 | 5.3 | 34.6 |
| 32,000 | 0.8230 | 0.7894 | 0.7929 | 11.7 | 50.3 |
The advantage does not break, and at the large end it consolidates: +0.030 at 32,000 rows. And the split of times opens in the direction opposite to intuition: 11.7 seconds against 50.3.
\nIt is worth saying precisely what grows and what does not, because the difference matters. What rises steadily is TabICL’s absolute accuracy: 0.7533, 0.7733, 0.7800, 0.7817, 0.7958, 0.8090, 0.8230, monotone across all seven measurements. The advantage over XGBoost, by contrast, is irregular: 0.000, +0.020, +0.013, +0.009, +0.038, +0.024, +0.030. It rises and falls.
\nAnd at 4,000 rows that +0.009 advantage is smaller than the spread across the three seeds (TabICL ranges from 0.768 to 0.799; tuned XGBoost from 0.758 to 0.790), so at that point it is indistinguishable from noise and I do not count it as a win.
\nWhere it is solid is at the top: at 32,000 rows TabICL’s worst seed (0.8216) sits above XGBoost’s best (0.7950). There is no possible overlap there.
\nThe only place XGBoost wins cleanly is the small end, at 500 rows, where the variability between seeds is so high that I would not bet anything on that difference.
\nIf you were expecting, as I was, the curve to flip at some point, it does not do so here. You would have to go considerably higher to find it.
\nBioresponse is the telltale dataset: 419 columns.
\nTabPFN 2.2.1 falls to 0.7567, the worst of the four contenders, below even untuned XGBoost. And it costs it 18.1 seconds, twenty-five times more than on a normal dataset. TabICL holds up much better (0.7922, the best of the four) but pays too: 6.0 seconds against the usual 0.8.
\nThe reading is that the table’s width, not its length, is the axis that genuinely squeezes. That makes sense: the number of columns enters the transformer’s attention cost, and the synthetic pretraining covers tables of tens of columns well, not hundreds.
\nI would rather list the limits than pretend they do not exist:
\nDiabetes130US is won by 0.0011 when the per-seed difference ranges from −0.011 to +0.049: there the sign depends on which seed comes up. The area-under-the-curve count does hold seed by seed; the accuracy one, on three or four datasets, does not.Bioresponse is the only one with hundreds of columns, so “width is what squeezes” is a hypothesis with one observation, not a rule.I would use it on any table between one thousand and thirty thousand rows with fewer than a hundred columns, where a person’s time is worth more than the last hundredth. A second of compute, zero tuning, and a result that in my measurement was consistently better. For an initial exploration it is hard to justify not doing it.
\nI would not use it on very wide tables, where it degrades and gets expensive. Nor in production without a GPU, nor where you need a small artifact that can be inspected and deployed in a lightweight container: a trained tree weighs kilobytes and runs anywhere, whereas here you have to load a transformer.
\nAnd if the criteria include being able to audit where the weights came from, today the answer is TabICL. Not for performance, though it was also the better of the two, but because it is the one you can still download and run without asking anyone’s permission.
\nMeasured on 16 August 2026 on a 16 GB RTX 4070 Ti SUPER, with torch 2.9.1+cu128, XGBoost 3.4.1, TabICL 2.1.1 and TabPFN 2.2.1. Data from the inria-soda/tabular-benchmark tabular suite. The main table took 585 seconds and the scale sweep 514.
No comments yet. The first one is yours.
","headings":[{"level":1,"text":"TabPFN and TabICL against tuned XGBoost: the model that does not train won on fourteen tables out of fourteen","id":"tabpfn-and-tabicl-against-tuned-xgboost-the-model-that-does-not-"},{"level":2,"text":"What exactly these models do","id":"what-exactly-these-models-do"},{"level":3,"text":"Default risk","id":"default-risk"},{"level":3,"text":"Sales campaign","id":"sales-campaign"},{"level":3,"text":"Clinical readmission","id":"clinical-readmission"},{"level":3,"text":"Recidivism","id":"recidivism"},{"level":3,"text":"Electricity demand","id":"electricity-demand"},{"level":3,"text":"Molecular screening","id":"molecular-screening"},{"level":2,"text":"What this is for, concretely","id":"what-this-is-for-concretely"},{"level":3,"text":"What changes if the promise holds","id":"what-changes-if-the-promise-holds"},{"level":2,"text":"The benchmark’s rules","id":"the-benchmark-s-rules"},{"level":2,"text":"Three stumbles before the first number","id":"three-stumbles-before-the-first-number"},{"level":3,"text":"PyTorch turned the GPU off without saying so","id":"pytorch-turned-the-gpu-off-without-saying-so"},{"level":3,"text":"OpenML’s API returned 504 on every endpoint","id":"openml-s-api-returned-504-on-every-endpoint"},{"level":3,"text":"The most-cited model no longer downloads without an account","id":"the-most-cited-model-no-longer-downloads-without-an-account"},{"level":2,"text":"The numbers","id":"the-numbers"},{"level":2,"text":"The cost is the reverse of what you expect","id":"the-cost-is-the-reverse-of-what-you-expect"},{"level":2,"text":"The surprise: the advantage does not break with size","id":"the-surprise-the-advantage-does-not-break-with-size"},{"level":2,"text":"Where they do break","id":"where-they-do-break"},{"level":2,"text":"What this measurement does not say","id":"what-this-measurement-does-not-say"},{"level":2,"text":"When I would use it and when not","id":"when-i-would-use-it-and-when-not"},{"level":2,"text":"Sources","id":"sources"},{"level":2,"text":"Comments","id":"comments"}]}}