Sipho takes the train into Cape Town every morning. Somewhere along the line it drops into a tunnel and the signal dies, and that is usually when he opens his laptop to plan his day. This morning he asks his AI assistant what is ahead of him, and it answers straight away, even with no connection: three meetings, two code reviews he left open last night, and a reminder to call his sister before lunch.
The reason it still works underground is that the assistant is not in the cloud. It is a small AI model living on his laptop, trained on his own notes, calendars, and messages, so it knows him and nothing else.
He built it himself in a single afternoon. It answers for free, it works in the tunnel, it works during load-shedding, and not one word of his life ever leaves the machine.
This is what it feels like to own your AI instead of renting it. No monthly subscription. No usage limits. No quiet phone-home to someone else's server. It is simply a product that is yours.
3.8B
parameters, Phi-4 mini base
~4 hrs
LoRA fine-tune on one Spark
12 tok/s
inference on a Raspberry Pi 5
0
cloud calls at runtime
Why small, and why now
For a long time the honest answer to "which model should I use" was "the biggest one you can afford." Frontier models are impressive because they can do almost anything, badly. Ask one to summarise a legal contract, then triage a support ticket, then write a bedtime story, and it will do all three at roughly the same level of competence. That is what you are paying for.
You do not need any of that if what you need is one thing done reliably. A support agent that only ever talks about your product. A triage model that only ever routes tickets in your queue. A voice assistant that only ever answers the top forty questions your customers ask. For a job that narrow, a small model tuned on your data will out-perform a frontier model at a fraction of the compute, on hardware you already own.
Small models have quietly become good. The ones we reach for are Phi-4 mini at 3.8B, Gemma 3 in its 1B and 4B sizes, and nanochat, Andrej Karpathy's sub-2B model you can train from scratch for about a hundred dollars. Any of them, fine-tuned properly on the right data, will beat a stock frontier model at a narrow task. And every one of them fits comfortably in the memory budget of consumer-class hardware, right down to a Raspberry Pi.
When a job genuinely needs a bigger brain, we still stay open-weights and self-hosted. We run large mixture-of-experts models like MiniMax M2, 230B parameters with only a fraction active per token, on the four-node Spark cluster. That is the top of our range, not something that fits a Pi, but it never leaves the building either.
“A model you trained on your data, running in your building, is a different category of product from one you rent by the token from someone else's data centre. It is yours.”
Small enough to be personal
The size story is not just about running on cheap hardware. It is about how cheap it becomes to train. A four-hour training run in your own building costs a rounding error. That changes the unit of personalisation.
Right now, most teams have one model and hand every user the same thing. If they want it to feel personal, they bolt on RAG: a vector store, a retrieval pipeline, a prompt template that stitches your history into the context on every call. It is fine. It is also complexity you have to run forever, and every answer still passes through a model that does not actually know you.
When training is this cheap, you skip all of that. You train a model per person. Or per team. Or per household. The knowledge is baked into the weights, not retrieved on the side. There is no vector store to maintain, no reranker to tune, no cache to invalidate when the user changes their mind. The model IS the personalisation.
A Sunday afternoon
Ninety minutes on the box, then dinner.
Sipho has a personal one running on his laptop. It knows what he is working on this quarter, which library versions he actually uses, the way he writes commit messages, the fact that he does not put hot sauce on his eggs and prefers rooibos to coffee after 4 PM.
He trained it on his own notes, calendars, and messages one Sunday afternoon in about ninety minutes, then sat down to dinner.
Back on that train, Sipho keeps going. He asks the model to rewrite a paragraph of a blog draft he was stuck on, and it comes back in his own voice, using the long semicolon-heavy sentences he actually writes, because his own writing is what it learned from. Then he asks it to prepare talking points for a call with a client, Andile at Standard Bank. It gives him three, and one of them recalls the exact pricing objection Andile raised three weeks ago, because that conversation is in Sipho's notes and his notes are in the model.
None of this touches a server. There is no lookup step running in the background, no context being stitched together on the fly. The answer is simply there, faster than he can finish typing, and it costs him nothing because he already owns the laptop and already paid the electricity bill.
What the training loop actually looked like
One box, one afternoon
What the client already had was the dataset.
The client task in the pipeline is inbound customer support for a mid-sized retailer. The training material is what the client already has: five years of chat transcripts, an internal policy document, a returns-and-refunds SOP. No labels, no crowdsourcing, no synthetic data.
We pick Phi-4 mini as the base. It is one of the strongest small open models available in mid-2026, sitting right alongside Gemma 3 4B and nanochat in our regular stable, it is permissively licensed, and its footprint fits inside the working memory of everything we want to deploy on, from a Spark to a laptop to a Pi.
Fine-tuning is LoRA, or its quantised sibling QLoRA when memory pressure demands it. On a single Spark, a full LoRA pass on a dataset this size takes about four hours. Peak memory sits well under the 128 GB unified pool, which leaves room for an evaluation harness resident in the same process running structured evals every few hundred steps. The final adapters weigh a few hundred megabytes, sit on top of the base model, and get merged in for deployment.
The eval plan is thirty real support conversations the client's team has already resolved. The hypothesis, borrowed straight from the published literature: a small model fine-tuned on your data will match or beat a stock frontier model at your specific task, at a fraction of the cost per query.
Med-PaLM 2 showed this pattern in medicine, a small fine-tuned model beat GPT-4 zero-shot on the medical exam benchmark. The original Phi papers showed the same for reasoning. Domain-adaptive continued pretraining papers have been showing it for years across biomedical, legal, and financial NLP. The recipe is boring by now.
The pattern in the literature
Small, fine-tuned, on your data. Beats big and general.
The shape of results reported across the fine-tuning literature on narrow domain tasks. Our small models tuned on task data, against stock frontier models called zero-shot.
Where it runs
The deployment story is the point. The same model, quantised to four bits and served via llama.cpp, runs on:
- The Spark clusterThe full range, from Phi-4 mini and Gemma 3 at full precision up to a 230B model like MiniMax M2 across four nodes for heavier agentic work. Room for concurrent sessions and live RAG lookups.
- An old ThinkPad or a mid-range laptopGemma 3 4B or Phi-4 mini, 4-bit quantised, around 35 tok/s. Comfortable for a single agent in a back office.
- A Raspberry Pi 5Phi-4 mini, Gemma 3 1B, or nanochat, 4-bit quantised, around 12 tok/s. Slower, but fully offline, drawing under 10 watts, small enough to sit inside a kiosk.
- Whatever the client already ownsThe whole point is that we do not need to sell you new hardware to make this useful.
The Pi number is not a stunt. It is the most interesting number of the four. A model that runs on a Pi runs anywhere. It runs in a kiosk on a shop floor with intermittent connectivity. It runs in a service van doing offline diagnostics. It runs on the same laptop the call-centre agent already uses.
We keep one on the desk as a demo. Pull the ethernet cable out of the little black box, ask it a support question a real customer wrote last week, and it answers correctly, in the client's tone, citing a returns policy no cloud model has ever read. Under a second. No internet in the loop. That is the whole pitch in one gesture.
Same class of model, four devices
Inference speed by hardware.
Tokens per second in a single-user session, running a small fine-tuned model in the Phi-4 mini and Gemma 3 4B class. The Cloud row is a network round-trip from Cape Town for reference.
The three-day timeline
Day 1
Data preparation. Cleaning, deduplicating, formatting five years of support transcripts and the policy document into the training format. This is the slow, boring, valuable part.
Day 2, morning
LoRA fine-tune on one Spark. Four hours of training, evals running in parallel. The Spark stays warm, the office does not.
Day 2, afternoon
Structured evals against the thirty resolved conversations. Adjust rank, redo one epoch, re-evaluate. Merge adapters into the base weights.
Day 3
Quantise to 4-bit GGUF, deploy on Spark for scale, deploy the same weights on the Pi for the offline demo. Hand over the Pi, hand over the runbook.
Why this matters if you are in South Africa
There is a cloud-first version of this story where you fine-tune on someone else's H100s, host on someone else's inference platform, and pay per token forever. That version has a monthly bill, a data-residency conversation, and an outage clause. It is also the only version most South African teams have ever been shown.
In your building
The model is a file you hold.
Training happens on hardware you own, or hardware we lease you, and the data never leaves the building. Deployment lands on whatever machines you already run.
When the internet drops, work keeps going. When Anthropic has a bad week, your customers do not notice.
The most common objection to this approach is that a fine-tuned small model "cannot possibly" match a frontier model. That is true if you ask it to do everything. It is not true if you ask it to do one thing on your data. The literature has been quietly saying this for years, from Med-PaLM 2 in medicine to the domain-adaptive-pretraining line of work across law and finance. The recipe is not exotic.
If your business has a defined AI task and your own data to train on, we can build you a small model that fits on hardware you already own, and run the eval that tells you honestly whether it beats a frontier model at your job. Talk to us about the task.