Testing and evaluation, as a service
Proving an AI system works. Then proving it again.
Anyone can demo an agent once. We build the repeatable quality system that catches regressions and turns every production failure into permanent test coverage.
Change events
Golden scenarios
Verify refund policy answer
Expected: Policy summary
Compare features across plans
Expected: Table of key differences
Extract order details from email
Expected: JSON with fields
Evaluation engine
Release gates
Gate policy
Production quality
Production sampling
Plan comparison request
SampledRefund policy question
SampledUpgrade eligibility check
SampledHuman review
Answer missed key plan limitations.
Reviewer
Clear and accurate.
Reviewer
Regression library
Continuous improvement
Audit and evidence
Change events
Golden scenarios
Verify refund policy answer
Expected: Policy summary
Compare features across plans
Expected: Table of key differences
Extract order details from email
Expected: JSON with fields
Evaluation engine
Release gates
Gate policy
Production quality
Production sampling
Plan comparison request
SampledRefund policy question
SampledUpgrade eligibility check
SampledHuman review
Answer missed key plan limitations.
Reviewer
Clear and accurate.
Reviewer
Regression library
Continuous improvement
Audit and evidence
From a prompt change to an audited, released version.
Release quality gates
Every meaningful change passes evaluation before release.
A prompt, model, retrieval, tool, policy or workflow change enters the pipeline, runs the full evaluation suite, and is held, risk-accepted with an owner, or approved. Nothing reaches production on a hunch.
- 01
Change intake
Prompt, model, retrieval-augmented generation (RAG), workflow, tool, policy, or data update.
- 02
Evaluation suite
Golden scenarios, RAG checks, tool checks, safety checks, and conversation tests.
- 03
Gate review
Blocker, warning and advisory results reviewed against thresholds.
- 04
Controlled release
Approved version moves through staging, pilot, or production rollout.
- 05
Production quality
Real traces are sampled, scored, trended, and routed for review.
- 06
Learning loop
Confirmed issues update datasets, thresholds, and backlog priorities.
Release hold
Blocker failures, severe regressions, approval bypasses, or hard policy violations.
Risk accepted
Non-blocking warnings proceed only with a named owner and target date.
Release approved
Quality, safety, workflow, and monitoring criteria are ready for rollout.
What we test
Nine test groups, each with a job.
We test the whole system: retrieval, model output, tool use, workflow routing, policy enforcement, voice stages, and the final response. Every group is reported on its own, because a single blended score hides the one thing you needed to see.
Testing portfolio overview
Unified view of quality, risk, and readiness across the AI system.
Functional tasks
Validate core capabilities and task success.
Tests
412
Pass rate
92%
Retrieval and grounding
Ensure answers are grounded and well-cited.
Tests
358
Pass rate
88%
Tool use
Verify safe and effective tool execution.
Tests
296
Pass rate
90%
Policy and guardrails
Confirm policy adherence and safe behaviour.
Tests
275
Pass rate
95%
Multi-turn conversation
Evaluate context handling across turns.
Tests
322
Pass rate
91%
Voice experience
Validate the whole spoken interaction.
Tests
268
Pass rate
89%
Edge and adversarial cases
Stress the system with hard and risky scenarios.
Tests
310
Pass rate
87%
Human review
Human-in-the-loop validation for high-impact areas.
Reviews
128
Agreement
94%
Regression suite
Detect regressions and prevent quality drift.
Tests
1,024
Pass rate
93%
Release readiness summary
Overall quality signals and the next steps.
Coverage
2,865 / 3,150 tests
Across 9 domains
Risk mix
- High18%
- Medium37%
- Low45%
Critical gaps
- Hallucination risk7
- Tool permission gaps5
- Adversarial jailbreaks4
View all gaps
Recommended action
- Address hallucination riskHigh
- Tighten tool permissionsHigh
- Expand adversarial coverageMedium
Create action plan
Functional tasks
Validate core capabilities and task success.
Tests
412
Pass rate
92%
Retrieval and grounding
Ensure answers are grounded and well-cited.
Tests
358
Pass rate
88%
Tool use
Verify safe and effective tool execution.
Tests
296
Pass rate
90%
Policy and guardrails
Confirm policy adherence and safe behaviour.
Tests
275
Pass rate
95%
Multi-turn conversation
Evaluate context handling across turns.
Tests
322
Pass rate
91%
Testing portfolio overview
Unified view of quality, risk, and readiness across the AI system.
Voice experience
Validate the whole spoken interaction.
Tests
268
Pass rate
89%
Edge and adversarial cases
Stress the system with hard and risky scenarios.
Tests
310
Pass rate
87%
Human review
Human-in-the-loop validation for high-impact areas.
Reviews
128
Agreement
94%
Regression suite
Detect regressions and prevent quality drift.
Tests
1,024
Pass rate
93%
Release readiness summary
Overall quality signals and the next steps.
Coverage
2,865 / 3,150 tests
Across 9 domains
Risk mix
- High18%
- Medium37%
- Low45%
Critical gaps
- Hallucination risk7
- Tool permission gaps5
- Adversarial jailbreaks4
View all gaps
Recommended action
- Address hallucination riskHigh
- Tighten tool permissionsHigh
- Expand adversarial coverageMedium
Create action plan
And the suites that cover them
Nine test suites, each running at its own point in the lifecycle: some on every commit, some only at the release gate, some continuously against production traffic.
Foundation tests
Always onPrompt contract, schema validation, rules, policy, tool contracts, and deterministic checks.
Golden scenario tests
Release gateKnown journeys with expected outcomes, evidence requirements, approval paths, and failure examples.
Conversation simulation
RegressionMulti-turn journeys, memory behaviour, interruptions, ambiguous inputs, and persona drift.
RAG and knowledge tests
High valueRetrieval quality, source grounding, freshness, conflicting content, and unsupported claims.
Agent workflow tests
CriticalRouting, tool sequence, tool result use, exception handling, and escalation behaviour.
Safety and privacy tests
BlockerPersonally identifiable information (PII) exposure, unsafe requests, refusal quality, policy coverage, and jailbreak resistance.
Voice interaction tests
VoiceSpeech-to-text quality, voice activity and end-of-utterance timing, barge-in, silence, text-to-speech quality, and transfer hand-off.
Load and resilience tests
OperationalLatency, concurrency, failures, retries, back pressure, cost, and queue behaviour.
Production replay tests
Learning loopReal traces promoted to offline tests after review, annotation, and root-cause tagging.
The reusable asset
The golden scenario library.
This is the centre of repeatable AI testing. It is a product asset rather than a temporary quality assurance (QA) file. Every important behaviour lives here as a versioned, reusable scenario: happy path, edge case, adversarial, policy, tool failure, voice issue, escalation, and every known production regression.
When we work with you, this library is yours.
Each scenario captures
- Scenario name
- Journey type
- User input
- Expected outcome
- Required evidence
- Prohibited behaviour
- Required tools
- Escalation rule
- Evaluation metrics
- Owner
- Version metadata
Scoring and routing
Every release decision is explained.
Weighted score rails, decision zones, and routing outcomes, with a policy override that forces a block or a human handover on any critical safety or compliance failure.
Score rails
(Dimensions)Quality
Safety
Retrieval
Tool use
Policy
Business criticality
Overall score
Decision zones
Below 60
Block
High risk
Requires human review
60 – 84
Warn
Moderate risk
Proceed with caution
85 – 100
Pass
Low risk
Safe to automate
Weighted factors · Calibrated to your risk profile
Quality
25%
Safety
25%
Retrieval
15%
Tool use
15%
Policy
10%
Business criticality
10%
Routing outcomes
Auto approve
Release automatically
Score ≥ 85
All rails ≥ 60
Deploy with watchlist
Monitor closely in production
Score 70 – 84
No critical risks
Require human review
Queue for expert review
Score 60 – 69
Or any rail < 60
Block release
Do not release
Score < 60
Or critical failure
Hand over to a person
Escalate before any action
High-risk or
Policy override
Human handover triggers
High-risk intent
Detected in prompt
Policy uncertainty
Ambiguous or unclear
Low confidence
Model confidence low
Conflicting retrieval
Sources disagree
Failed tool action
Tool error or exception
Sensitive customer impact
PII, financial, legal, safety
Repeated low score
Consistently below threshold
Policy override
(Always on)Critical safety or compliance failures bypass score averages and force a block or a mandatory handover.
Examples
What we do for you
We run the quality system, or we assess yours.
- Golden scenario libraryStand up a golden scenario library for your agents, versioned and maintained as a product asset.
- Release quality gatesWire quality gates into your delivery so every change clears the bar before release.
- Production samplingRun production sampling and evaluation against live traffic.
- Assurance assessmentAssess an existing AI system against our ISO/IEC 42001-aligned assurance framework. NPU Labs is not an accredited certification body.
Assessed against our ISO/IEC 42001-aligned assurance framework
ISO/IEC 42001
AI management system
POPIA
South Africa
GDPR
EU and UK
NIST AI RMF
Risk framework
Common questions
What does AI testing and evaluation actually cover?
Retrieval quality, model output, tool use, workflow routing, policy enforcement, voice stages, and the final response. Each group is scored and reported on its own.
How is this different from normal software QA?
An AI system is not deterministic, so a pass or fail assertion does not survive contact with it. We score behaviour against graded expectations using a golden scenario library, run every meaningful change through the same suite, and use weighted scoring with explicit decision zones instead of a binary result.
What is a golden scenario library?
A versioned, reusable set of the behaviours that matter to your business: happy path, edge case, adversarial, policy, tool failure, voice and escalation cases, plus every production regression we have ever confirmed. It is a product asset rather than a throwaway QA file, and when we work with you it is yours to keep.
Can you assess an AI system we already built?
Yes. An assessment is one of the two ways we work: we evaluate an existing system against real scenarios, safety and privacy checks, and our ISO/IEC 42001-aligned assurance framework, then give you a report and a plan to make it release-ready. The alternative is that we run the quality system for you on an ongoing basis.
What happens when a release fails its quality gate?
One of three things, and all of them are recorded. The change is blocked, or it is risk-accepted with a named owner and a stated reason, or it is approved. Any critical safety or compliance failure triggers a policy override that forces a block or a human handover regardless of the score average.
Do you replace our engineers?
No. We build the quality system and the evidence trail around your team's work. Human review stays in the loop by design, particularly on high-impact and high-risk decisions, and accountable people sign off rather than a model doing it silently.
Not the question you came with?
Ask us about your setupBring us your AI system. We will tell you what works and what does not.
An evaluation against your scenarios, safety and privacy checks, and an ISO/IEC 42001-aligned framework, then a plan to make it release-ready.