*Sep 22, 2026
\nI'm self-hosting some open LLM models on AWS Spot EC2 instances for personal use.
\nIt wasn't my primary goal, but I'm happy enough with my setup to downgrade my Claude subscription from Max ($100/mo) to the Pro plan ($20/mo).
\nHere is what I learned along the way.
\nWhat model do we need for software development?
\nYou don't need the most expensive GPU, but one with enough virtual memory
\nRenting GPU under their market value: AWS Spot Instances
\nFrom Claude Code to Hermes
\nWhat about parallel work and sub-agents?
\nWhat's next?
\nTake Llama 3.2 (3 billion parameters - 2 GB of memory), a Small Language Model (SLM) that runs on any device, laptop, even your smartphone.
\n`ollama run llama3.2\npulling manifest\npulling dde5aa3fc5ff: 100% ▕███████████████████▏ 2.0 GB\nverifying sha256 digest\nwriting manifest\nsuccess
\n\nsay hi\nHi!\n`
This model is great to do basic tasks, offline, directly on the device like text summarization, rewriting, etc.
\nBut to do software development, like you would do with Claude Code or Codex, you need specialized models that can think about complex tasks and work with a very large context, having hundreds of billions of parameters. Or so I thought.
\nProviders don't reveal the details of their models, but in 2024 it was estimated that Sonnet 3.5 had ~175 billion and GPT-4o had ~200 billion parameters. The latest models are estimated to have trillions.
I wanted to try some open models with the same order of magnitude of capacity as Sonnet, and there are some available on Ollama:
\nornith 1.5 model trained to "think" and self-improve (397b params / 242GB memory)
\nqwen3-coder meant to produce code (30b params / 19GB memory)
\nI don't even have 300 GB on any of my disk, so I quickly chose the 10x-reduced version of ornith 1.5 to see how good it is (34b params / 23 GB of memory).
Do I even need a Large Language Model (LLM), or would some "medium" language models do the job for personal use? Some answers on this talk we had at the Tech F'Est 2026.
\nOllama models work on a CPU as long as you have enough memory to fit the entire model. The generation speed depends on your computation power.
\nYou can try to run ollama run qwen3-coder, if you have more than 20 GB of memory on your device it will work, but the generation speed will be very slow because of the number of parameters.
What you need is a GPU because they can do a lot more parallel computation.\nBut not all GPUs are equal.
\nYou need to have a GPU with enough virtual memory to fit the whole model in memory + the whole context size you want.
\nAt home, I have an old gaming PC with a NVIDIA GeForce RTX 2060 card I bought in 2020. I tried it, but it only has 6 GB of virtual memory, which is not enough.
Ollama is smart enough to use the device memory as a failover when GPU VRAM is too small, but it means falling back on the CPU which defeats the purpose of using a GPU.
\nApple Silicon devices have shared memory between CPU and GPU which is super helpful, because the GPU can read the device memory instead of its own VRAM.
\nAlso, we could fit more context in the virtual memory with quantization but it may impact the model response quality.
\nLatest gaming GPUs can go up to 32 GB of VRAM, which is great to run those medium-sized models. But I don't want to buy one, which is only leaving me one solution.
\nAWS has a range of GPU instances with different cards available.
\nThey're also offering a reduced price for unused capacity on a service called AWS Spot Instances.
\nYou choose the instance you want, they offer a reduced rate but the price is variable so you need to set a maximum price.\nWhen maximum price is reached, the instance is stopped, and you have to request a new one.
\nTo enable EC2 Spot Instance, you first need to request a quota increase. It's usually approved in a few days max.
\naws service-quotas request-service-quota-increase --service-code ec2 --quota-code L-3819A6DF --desired-value 32
The instances with a single GPU are ranging from $0.5/h to $10/h, depending on the availability.
\nI wrote a helper to spin up a spot instance, select the pricing, choose the Ollama models and spin it up: aws-llm-sandbox.
\n`./up.sh\n==> my public IP: xx.xx.xx.xx\n==> security group sg-0d910bcfdb06e201e: 22 + 11434 open to xx.xx.xx.xx/32 only\n==> no existing instance - launching a new one
\nAvailable GPUs (single-GPU instances). Pick by VRAM: the model + its KV cache must fit in it.
\n16 GB VRAM NVIDIA T4 2018 gen (g4dn). Cheapest; fits <=14B models at q4. Too small for 30B.
types: g4dn.xlarge, g4dn.2xlarge, g4dn.4xlarge, g4dn.8xlarge, g4dn.16xlarge24 GB VRAM NVIDIA A10G 2021 gen (g5). Same 24 GB tier as L4, ~same speed for Ollama; a second pool when g6 has no capacity.
types: g5.xlarge, g5.2xlarge, g5.4xlarge, g5.8xlarge, g5.16xlarge24 GB VRAM NVIDIA L4 2023 gen (g6, gr6 = same GPU with 2x system RAM). 24 GB: one 30B model at q4 (~18 GB) + KV cache; only one big model resident at a time.
types: g6.xlarge, g6.2xlarge, g6.4xlarge, gr6.4xlarge, g6.8xlarge, gr6.8xlarge, g6.16xlarge48 GB VRAM NVIDIA L40S 2023 gen (g6e). 48 GB, ~2-3x L4 throughput: two 30B models resident, or 70B at q4, or 30B with a huge context.
types: g6e.xlarge, g6e.2xlarge, g6e.4xlarge, g6e.8xlarge, g6e.16xlarge96 GB VRAM NVIDIA RTX PRO Server 6000 2025 gen (g7e, Blackwell). 96 GB, often priced like the L40S: 70B at q8, 120B-class MoE at q4, or several 30B models resident.
types: g7e.2xlarge, g7e.4xlarge, g7e.8xlargeGPU? [1-5] 4\n==> fetching spot prices for: g6e.xlarge g6e.2xlarge g6e.4xlarge g6e.8xlarge g6e.16xlarge\n==> fetching spot placement scores
\nMARKET TYPE AZ $/h SCORE vCPU RAM
SCORE: spot placement score 1-10 (10 = capacity very likely, 1 = unlikely).\n spot: market price, varies; AWS stops the instance if it rises above your cap. Needs the spot quota.\n on-demand: fixed price, never interrupted, no quota issue; AZ chosen by AWS where capacity exists.\nWhich one? [1-20]\n`
\nI tested various instances, and there are family types (g5, g6, g6e) whose prices range from $0.5/h to $10/h, and I found my sweet spot with a g6.4xlarge instance to run my two models.
Cost Management = Just-in-time provisioning
\nA good way to save on the bill is to just stop the instance when you don't use it.
\nBut when it's turned on, you have virtually no limit of tokens!
\nHere's my AWS bill from last month, and it was before I used Spot instances.
\nI'm still using Fable for most complex planning so I'm keeping Claude Pro for now. I tried all that to understand better and check if it was possible, but I'm saving a little bit of money, which is great. Obviously, it's not applicable in all situations.
\nBut it's totally possible to orchestrate a fleet of low-cost GPU instances, with a routine that track the cheapest available spot instances on multiple regions for your own usage.
\nIt is possible to run Ollama models with Claude Code.\nBut I found it easier with Hermes.
\nFrom Hermes, you can easily add your own custom models and switch provider and model in one command.
\n`hermes model
\nCurrent model: ornith-1.5:35b\n Active provider: llm-sandbox
\nCustom OpenAI-compatible endpoint configuration:
\nAPI base URL [e.g. https://api.example.com/v1]:\n`
\nThere are a lot of cool features with Hermes that can add up.
\nLike running multiple Profiles. One profile is a "Product Manager agent" running with ornith, a second is a "Coder agent" with qwen3, and a last "Architect" profile running Fable whose goal is to write the plan and test the feature in the end.
\nAnd you can then run a Kanban where the "Product agent" creates tickets, and some coder agents automatically pick up their tasks and implement them.
\nOllama can totally handle concurrent requests, even with multiple models, if you have enough memory.
\nHere is an entire blog post describing how it works and how to configure it (fr).
\nI'm easily running 3 ornith agents in parallel on a g7e.4xlarge instance.
Pretty much nothing, I'm happy with my setup for now and it allows me to test a LOT of models available on Ollama with different setups and automations.
\nI'm surprised how far I've got, I was expecting to hit a performance wall or spend way too much on GPU.
\nI didn't entirely get rid of the Claude models and their 1M context windows, but it was not the goal either.
\nBy paying a Claude / Codex subscription we're not even paying for a fraction of the compute required, so it's still very cheap to keep your subscription.
\nFor programmatic use, burst usage or a certain type of compliance need, though, you'll still find self-hosting useful. And just-in-time provisioning of GPU instances will probably come handy.
\nThere are plenty of other things to play with, like an Ollama load balancer over multiple instances.
\nI'm curious how this could fit in some other contexts. Could a well-orchestrated fleet save the bill and footprint of a small/medium company?
","headings":[{"level":1,"text":"Self-hosting LLM models for software development","id":"self-hosting-llm-models-for-software-development"},{"level":2,"text":"What model do we need for software development?","id":"what-model-do-we-need-for-software-development"},{"level":2,"text":"You don't need the most expensive GPU, but one with enough virtual memory","id":"you-don-t-need-the-most-expensive-gpu-but-one-with-enough-virtua"},{"level":2,"text":"Renting GPU under their market value: AWS Spot Instances","id":"renting-gpu-under-their-market-value-aws-spot-instances"},{"level":2,"text":"From Claude Code to Hermes","id":"from-claude-code-to-hermes"},{"level":2,"text":"What about parallel work and sub-agents?","id":"what-about-parallel-work-and-sub-agents"},{"level":2,"text":"What's next?","id":"what-s-next"}]}}