Testing and evaluation, as a service
Proving an AI system works. Then proving it again.
Anyone can demo an agent once. We build the repeatable quality system that catches regressions and turns every production failure into permanent test coverage.
Change Events
Golden Scenarios
Verify refund policy answer
Expected: Policy summary
Compare features across plans
Expected: Table w/ key diff
Extract order details from email
Expected: JSON with fields
Evaluation Engine
Release Gates
Gate Policy
Production Quality
Production Sampling
Plan comparison request
SampledRefund policy question
SampledUpgrade eligibility check
SampledHuman Review
Answer missed key plan limitations.
Reviewer
Clear and accurate.
Reviewer
Regression Library
Continuous Improvement
Audit & Evidence
Change Events
Golden Scenarios
Verify refund policy answer
Expected: Policy summary
Compare features across plans
Expected: Table w/ key diff
Extract order details from email
Expected: JSON with fields
Evaluation Engine
Release Gates
Gate Policy
Production Quality
Production Sampling
Plan comparison request
SampledRefund policy question
SampledUpgrade eligibility check
SampledHuman Review
Answer missed key plan limitations.
Reviewer
Clear and accurate.
Reviewer
Regression Library
Continuous Improvement
Audit & Evidence
End to end: from a prompt change to an audited, released version.
Release quality gates
Every meaningful change passes evaluation before it ships.
A prompt, model, retrieval, tool, policy or workflow change enters the pipeline, runs the full evaluation suite, and is held, risk-accepted with an owner, or approved. Nothing reaches production on a hunch.
- 01
Change intake
Prompt, model, RAG, workflow, tool, policy or data update.
- 02
Evaluation suite
Golden scenarios, RAG checks, tool checks, safety checks and conversation tests.
- 03
Gate review
Blocker, warning and advisory results reviewed against thresholds.
- 04
Controlled release
Approved version moves through staging, pilot or production rollout.
- 05
Production quality
Real traces are sampled, scored, trended and routed for review.
- 06
Learning loop
Confirmed issues update datasets, thresholds and backlog priorities.
Release hold
Blocker failures, severe regressions, approval bypasses or hard policy violations.
Risk accepted
Non-blocking warnings proceed only with a named owner and target date.
Release approved
Quality, safety, workflow and monitoring criteria are ready for rollout.
What we test
Nine test groups, each with a job.
We test the system, not just the prompt: retrieval, model output, tool use, workflow routing, policy enforcement, voice stages, and the final response. Every group is reported on its own, because a single blended score hides the one thing you needed to see.
Testing Portfolio Overview
Unified view of quality, risk, and readiness across the AI system.
Functional Tasks
Validate core capabilities and task success.
Tests
412
Pass Rate
92%
Retrieval & Grounding
Ensure answers are grounded and well-cited.
Tests
358
Pass Rate
88%
Tool Use
Verify safe and effective tool execution.
Tests
296
Pass Rate
90%
Policy & Guardrails
Confirm policy adherence and safe behavior.
Tests
275
Pass Rate
95%
Multi-turn Conversation
Evaluate context handling across turns.
Tests
322
Pass Rate
91%
Voice Experience
Validate spoken interactions end-to-end.
Tests
268
Pass Rate
89%
Edge Cases & Adversarial
Stress the system with hard and risky scenarios.
Tests
310
Pass Rate
87%
Human Review
Human-in-the-loop validation for high-impact areas.
Reviews
128
Agreement
94%
Regression Suite
Detect regressions and prevent quality drift.
Tests
1,024
Pass Rate
93%
Release Readiness Summary
Overall quality signals and actionable next steps.
Coverage
2,865 / 3,150 tests
Across 9 domains
Risk Mix
- High18%
- Medium37%
- Low45%
Critical Gaps
- Hallucination risk7
- Tool permission gaps5
- Adversarial jailbreaks4
View all gaps
Recommended Action
- Address hallucination riskHigh
- Tighten tool permissionsHigh
- Expand adversarial coverageMedium
Create action plan
Functional Tasks
Validate core capabilities and task success.
Tests
412
Pass Rate
92%
Retrieval & Grounding
Ensure answers are grounded and well-cited.
Tests
358
Pass Rate
88%
Tool Use
Verify safe and effective tool execution.
Tests
296
Pass Rate
90%
Policy & Guardrails
Confirm policy adherence and safe behavior.
Tests
275
Pass Rate
95%
Multi-turn Conversation
Evaluate context handling across turns.
Tests
322
Pass Rate
91%
Testing Portfolio Overview
Unified view of quality, risk, and readiness across the AI system.
Voice Experience
Validate spoken interactions end-to-end.
Tests
268
Pass Rate
89%
Edge Cases & Adversarial
Stress the system with hard and risky scenarios.
Tests
310
Pass Rate
87%
Human Review
Human-in-the-loop validation for high-impact areas.
Reviews
128
Agreement
94%
Regression Suite
Detect regressions and prevent quality drift.
Tests
1,024
Pass Rate
93%
Release Readiness Summary
Overall quality signals and actionable next steps.
Coverage
2,865 / 3,150 tests
Across 9 domains
Risk Mix
- High18%
- Medium37%
- Low45%
Critical Gaps
- Hallucination risk7
- Tool permission gaps5
- Adversarial jailbreaks4
View all gaps
Recommended Action
- Address hallucination riskHigh
- Tighten tool permissionsHigh
- Expand adversarial coverageMedium
Create action plan
And the suites that cover them
Nine test suites, each running at its own point in the lifecycle: some on every commit, some only at the release gate, some continuously against production traffic.
Foundation tests
Always onPrompt contract, schema validation, rules, policy, tool contracts and deterministic checks.
Golden scenario tests
Release gateKnown journeys with expected outcomes, evidence requirements, approval paths and failure examples.
Conversation simulation
RegressionMulti-turn journeys, memory behaviour, interruptions, ambiguous inputs and persona drift.
RAG and knowledge tests
High valueRetrieval quality, source grounding, freshness, conflicting content and unsupported claims.
Agent workflow tests
CriticalRouting, tool sequence, tool result use, exception handling and escalation behaviour.
Safety and privacy tests
BlockerPII exposure, unsafe requests, refusal quality, policy coverage and jailbreak resistance.
Voice interaction tests
VoiceSTT quality, VAD and EOU timing, barge-in, silence, TTS quality and transfer hand-off.
Load and resilience tests
OperationalLatency, concurrency, failures, retries, back pressure, cost and queue behaviour.
Production replay tests
Learning loopReal traces promoted to offline tests after review, annotation and root-cause tagging.
The reusable IP
The golden scenario library.
This is the centre of repeatable AI testing, and it is a product asset, not a temporary QA file. Every important behaviour, happy-path, edge case, adversarial, policy, tool failure, voice issue, escalation, and every known production regression, lives here as a versioned, reusable scenario.
When we work with you, this library is yours. It compounds: every failure we find becomes coverage that protects you forever.
Each scenario captures
- Scenario name
- Journey type
- User input
- Expected outcome
- Required evidence
- Prohibited behaviour
- Required tools
- Escalation rule
- Evaluation metrics
- Owner
- Version metadata
Scoring and routing
Every release decision is explained.
Weighted score rails, decision zones, and routing outcomes, with a policy override that forces a block or a human handover on any critical safety or compliance failure.
Score Rails
(Dimensions)Quality
Safety
Retrieval
Tool Use
Policy
Business Criticality
Overall Score
Decision Zones
Below 60
RED
High Risk
Requires human review
60 – 84
AMBER
Moderate Risk
Proceed with caution
85 – 100
GREEN
Low Risk
Safe to automate
Weighted factors · Calibrated to your risk profile
Quality
25%
Safety
25%
Retrieval
15%
Tool Use
15%
Policy
10%
Business Criticality
10%
Routing Outcomes
Auto Approve
Release automatically
Score ≥ 85
All rails ≥ 60
Deploy with Watchlist
Monitor closely in production
Score 70 – 84
No critical risks
Require Human Review
Queue for expert review
Score 60 – 69
Or any rail < 60
Block Release
Do not release
Score < 60
Or critical failure
Mandatory Human Handover
Escalate before any action
High-risk or
Policy override
Human Handover Triggers
High-risk intent
Detected in prompt
Policy uncertainty
Ambiguous or unclear
Low confidence
Model confidence low
Conflicting retrieval
Sources disagree
Failed tool action
Tool error / exception
Sensitive customer impact
PII, financial, legal, safety
Repeated low score
Consistently below threshold
Policy Override
(Always On)Critical safety or compliance failures bypass score averages and force Block or Mandatory Handover.
Examples
What we do for you
We run the quality system, or we assess yours.
- Stand up a golden scenario library for your agents, treated as a product asset, not a throwaway QA file.
- Wire release quality gates into your delivery so unsafe changes cannot ship.
- Run production sampling and evaluation so real failures become permanent test coverage.
- Assess an existing AI system against our ISO/IEC 42001-aligned assurance framework.
Assessed against our ISO/IEC 42001-aligned assurance framework
ISO/IEC 42001
AI management system
POPIA
South Africa
GDPR
EU and UK
NIST AI RMF
Risk framework
Common questions
Questions we get asked about AI testing.
What does AI testing and evaluation actually cover?
We test the system, not just the prompt: retrieval quality, model output, tool use, workflow routing, policy enforcement, voice stages and the final response. Each group is scored and reported on its own, because a single blended score hides the one thing you needed to see.
How is this different from normal software QA?
An AI system is not deterministic, so a pass or fail assertion does not survive contact with it. We score behaviour against graded expectations using a golden scenario library, run every meaningful change through the same suite, and use weighted scoring with explicit decision zones instead of a binary result.
What is a golden scenario library?
A versioned, reusable set of the behaviours that matter to your business: happy path, edge case, adversarial, policy, tool failure, voice and escalation cases, plus every production regression we have ever confirmed. It is a product asset rather than a throwaway QA file, and when we work with you it is yours to keep.
Can you assess an AI system we already built?
Yes. An assessment is one of the two ways we work: we evaluate an existing system against real scenarios, safety and privacy checks and our ISO/IEC 42001-aligned assurance framework, then give you an honest report and a plan to make it release-ready. The alternative is that we run the quality system for you on an ongoing basis.
What happens when a release fails its quality gate?
One of three things, and all of them are recorded. The change is blocked, or it is risk-accepted with a named owner and a stated reason, or it is approved. Any critical safety or compliance failure triggers a policy override that forces a block or a human handover regardless of the score average.
Do you replace our engineers?
No. We build the quality system and the evidence trail around your team's work. Human review stays in the loop by design, particularly on high-impact and high-risk decisions, and accountable people sign off rather than a model doing it silently.
Not the question you came with?
Ask us about your AI systemBring us your AI. We will tell you the truth about it.
An honest evaluation against real scenarios, real safety checks, and an ISO/IEC 42001-aligned framework. Then a plan to make it release-ready.