A private AI, small enough to run on a Raspberry Pi.

A lot of AI infrastructure starts with the wrong question:

What is the biggest model we can run?

For a narrow workload, the better question is usually:

What is the smallest model that can do this job well enough?

That changes the architecture.

A smaller model can be cheaper to train, cheaper to serve, easier to evaluate, and practical to deploy on hardware that already exists inside the organisation. In some cases, the deployment target can be a laptop, a compact edge computer, or even a Raspberry Pi-class device.

A model does not need to know everything to be useful.

Our view is that narrow workloads should be matched to models small enough to train, evaluate, and run where the work happens.

Modern open models in the 1B to 4B range are now credible candidates for task-specific systems, especially after adaptation.

The important benchmark is not whether the model wins generally. It is whether it wins at your task.

The challenge

Frontier models are designed to handle a very wide range of requests.

That generality is valuable when the workload is genuinely open-ended. It is less valuable when the job is tightly bounded.

A support assistant may only need to answer questions about one product. A classifier may only need to route requests into a known set of queues. A field-service assistant may only need to work from one organisation's manuals and procedures.

Using a frontier model for every one of those cases can create unnecessary cost, network dependency, and operational complexity.

The model may also be much harder to deploy privately.

Our view

We do not think small models should be positioned as weaker substitutes for frontier models.

They are a different engineering option.

The useful comparison is not:

small model versus frontier model

It is:

model versus workload

Microsoft's Phi-4-mini is a 3.8B-parameter model. Google publishes Gemma 3 in compact variants including 1B and 4B. These models are small enough to make local adaptation and local inference practical on hardware classes that are out of reach for much larger models.

That does not mean a fine-tuned small model will automatically outperform a frontier model.

It means the comparison is worth running.

For a constrained task with good training data and a clear evaluation set, specialisation can matter more than raw model size.

The constraints

• Task definition — the workload has to be narrow enough that success can be measured clearly.
• Training data — adaptation is only useful when the organisation has representative examples of the task.
• Evaluation quality — a small model is not better because it is cheaper; it is better only if it meets the required quality bar.
• Memory — deployment hardware determines which model size and quantisation level are realistic.
• Latency — edge inference has to be measured on the target device, not inferred from desktop or cloud benchmarks.
• Privacy — local deployment helps only if the surrounding data pipeline also stays local.

What we changed

Our model-selection process starts from the deployment target and the evaluation set, not from a leaderboard.

For a narrow private-AI workload, that means:

1. define the task;
2. assemble representative examples;
3. choose the smallest credible base model;
4. adapt it;
5. evaluate it against the actual task;
6. quantise only when necessary;
7. run the same evaluation again on the deployment build.

LoRA is useful here because it reduces the amount of trainable model state required during adaptation. The original LoRA paper showed that low-rank adaptation can reduce the trainable parameter count dramatically while preserving model quality on the tasks it evaluated.

That matters on compact private compute.

A DGX Spark has 128 GB of unified memory. That is enough room to work with models far larger than the final edge target, while keeping the training and evaluation environment on owned hardware.

Deployment is a separate step.

A small model can be quantised and served through a runtime such as llama.cpp on ARM64 or x86 hardware. The target might be a workstation, mini-PC, laptop, or Raspberry Pi-class device depending on the workload.

Architecture and approach

The useful pipeline is simple:

Owned data → task-specific adaptation → held-out evaluation → quantised model → local inference

The important part is the evaluation gate.

We do not want a model to reach the edge because it is small.

We want it to reach the edge because it is small and still passes the task.

What we tested

The current NPU Labs approach treats model size, accuracy, latency, and memory as separate measurements.

A model can win on one and lose badly on another.

Evidence

We do not treat a benchmark from another machine as evidence for our deployment.

If a model is intended for a Raspberry Pi, the final measurement has to happen on the Raspberry Pi.

What we learned

Smaller models make the model lifecycle more accessible.

They reduce the gap between experimentation and deployment because the same organisation can often train, evaluate, and serve the model without building a large GPU estate.

They also make bad decisions cheaper to discover.

If a model is small enough to retrain quickly, model selection becomes an engineering loop rather than a procurement event.

The lesson

Choose the smallest model that passes the real evaluation.

A 3.8B model that reliably solves the task is more useful than a much larger model that adds cost without changing the outcome.

The reverse is also true.

If the small model fails the evaluation, use a larger one.

There is no prize for being small.

What this demonstrates

This work demonstrates our ability to:

• select models from the workload backwards — the deployment target and evaluation set shape the model choice;
• adapt open models on owned compute — training does not have to start in a public cloud;
• evaluate before deployment — model quality is measured against the actual task rather than assumed from general benchmarks;
• move inference closer to the user — compact models can make laptops, mini-PCs, and edge devices realistic deployment targets;
• separate training hardware from serving hardware — a model can be trained on a Spark and deployed somewhere much smaller.

Where this applies

This approach fits workloads where the scope is controlled and the cost of local deployment is lower than the cost of repeatedly calling a general model.

Examples include:

• support classification;
• internal knowledge assistants;
• field-service tools;
• document triage;
• routing;
• small voice-agent components;
• policy and risk classifiers.

It is less suitable when the task requires broad world knowledge, complex open-ended reasoning, or capabilities that only larger models currently provide.

The point is not to avoid frontier models.

It is to stop using them by default.

What comes next

• Build repeatable evaluation sets for each narrow workload.
• Benchmark the same model before and after adaptation.
• Measure quantised builds on their actual deployment hardware.
• Compare local inference against hosted inference on total latency, operating cost, privacy, and support burden.
• Promote a model only when the evaluation says it is good enough.

Related reading

• The South African GPU map. What is actually available for AI here. — regional cloud availability changes the economics of local training and inference.
• The week the cloud blinked. Why private AI just got urgent. — private inference also changes the failure domain.

References

• Phi-4-Mini Technical Report — Microsoft Research.
• Gemma 3 model card — Google DeepMind.
• LoRA: Low-Rank Adaptation of Large Language Models — Hu et al.
• NVIDIA DGX Spark hardware overview — NVIDIA.
• llama.cpp — CPU and edge inference runtime.