I ran 120B MoE models on my 32GB M1 Max, using under 16 GiB

I've been trying to run large language models on my MacBook Pro. It has an M1 Max and 32GB of RAM, so a 60GB model obviously won't fit entirely in memory.

I wanted to see how far I could get with a different approach: keep most of a mixture-of-experts (MoE) model on the SSD, bring in what it needs, and put an actual limit on how much memory the inference process can use. Not just a number in a configuration file, but something I could check against macOS's own memory counters.