Monitoring
You cannot fix what you cannot see. Every AI system we run is instrumented first.
- Request and step tracingRecords every prompt, answer, tool call and agent step, so any result can be explained later.
- Uptime and error trackingWatches whether the system is up, and which requests fail and why.
- Speed trackingMeasures how long users wait, including the time to the first word of an answer.
- Usage analyticsShows who uses the AI, how often and what they ask.
- Live dashboardsQuality, speed, cost and volume on one screen for your team and ours.
- Alerts and on-callSends a problem to a named person, with the details needed to act.
- User feedback captureLinks each thumbs-down, comment and escalation to the exact conversation behind it.
- Conversation reviewA person reads a sample of conversations each week to catch what numbers miss.
- Open-standard telemetryUses OpenTelemetry, so the data works with tools such as Datadog, Grafana and Azure Monitor.
Your team decides: what counts as an incident and who is called.
Measured by: uptime, response time, time to detect a problem.
Evaluation and quality
“It seems fine” is not a measurement. Evaluation turns answer quality into a number you can track.
- Test sets from real casesHundreds of real questions with the right answers, built with your experts and kept current.
- Scheduled quality testsThe full test set runs every night, so a silent change is caught by morning.
- Release testsEvery change to a prompt, model or data source is tested before it goes live.
- AI judges with human checksA second model scores answers at scale, and people check a sample to keep the scoring honest.
- Grounding checksCompares each answer with its source documents to catch made-up facts.
- Search quality checksMeasures whether the search step finds the right documents before the model answers.
- Agent task successChecks whether agents finish the task correctly and how many steps they need.
- Fairness checksTests whether results differ unfairly between groups of people.
- Live A/B testsTries a change on a share of real traffic and keeps it only if it wins.
Your team decides: the pass mark for each test and whether a release goes ahead.
Measured by: accuracy, share of answers backed by a source, task success rate.
Drift and data
The world moves on after launch. Drift is the gap between what the AI learned and what is true now.
- Input drift detectionNotices when the questions or data coming in differ from what the system was built for.
- Answer drift detectionNotices when answers change, for example after a provider quietly updates a model.
- Knowledge freshnessFlags out-of-date documents and refreshes the search index on a schedule.
- Data quality checksTests the feeds the AI depends on for gaps, duplicates and errors.
- Retraining and tuningRetrains or re-tunes a model when its accuracy falls below the agreed level.
- Feedback loopsTurns corrections from your staff into new test cases and training examples.
- Search index upkeepRebuilds the index when documents, permissions or the underlying model change.
- Peak readinessTests the system before seasonal peaks, launches and policy changes.
Your team decides: when a model is retrained and which data it may learn from.
Measured by: drift score, share of answers from current sources, accuracy over time.
Cost and speed
AI is billed by use, so the bill grows with success. The aim is a lower cost for each successful task, not just a lower bill.
- Cost per task trackingShows spend by feature, team and customer, down to a single request.
- Model routingSends easy requests to a smaller, cheaper model and keeps the largest for hard ones.
- CachingReuses answers and repeated prompt content instead of paying for them again.
- Prompt trimmingCuts instructions and context the model does not need.
- Budget alerts and capsWarns before a budget is reached and stops runaway spend.
- Batch and off-peak processingMoves work that can wait to cheaper batch pricing.
- Capacity planningForecasts usage, so rate limits and reserved capacity are sized before they are needed.
- Speed tuningShortens waits with streaming, parallel steps and faster models where quality allows.
- Provider price reviewCompares providers and pricing plans as prices change.
Your team decides: budgets, and the balance between cost, speed and quality.
Measured by: cost per successful task, cache hit rate, spend against budget.
Security and safety
Attack methods against AI change month to month. Controls that passed at launch need testing again.
- Prompt-injection detectionCatches instructions hidden in messages, documents and web pages.
- Data leak preventionStops personal data and secrets from leaving in prompts or answers.
- Harmful content filteringBlocks unsafe, offensive or off-brand output.
- Misuse monitoringSpots scraping, attempts to bypass the rules and unusual account activity.
- Scheduled attack testsAttacks the system on a schedule and after each major change, then fixes what gets through.
- Access reviewsChecks what each user and each agent can reach, and removes what is no longer needed.
- Key rotationReplaces keys and passwords on a schedule and after any exposure.
- PatchingKeeps AI libraries, connectors and plug-ins up to date.
- Supplier checksReviews the models, tools and data sources your AI depends on.
Your team decides: which risks to accept and any change to what the AI may access.
Measured by: attacks blocked, time to fix a weakness, open findings.
Models and prompts
Providers release and retire models on their own schedule. Changes are planned, tested and reversible.
- Prompt version controlKeeps every version of every prompt, with who changed it and why.
- Model inventoryLists which model and version each system uses.
- Change approvalRoutes each change to the right person before it goes live.
- Gradual rolloutsReleases a change to a small share of traffic first and widens it as results hold.
- Shadow testingRuns a new version beside the old one on real requests, without showing users its answers.
- One-step rollbackReturns to the last good version in minutes.
- Model retirement planningTracks provider retirement dates and has a tested replacement ready in time.
- Backup providersKeeps a second model ready to take over during an outage.
Your team decides: every production change above the level you set.
Measured by: share of changes that cause a problem, time to roll back, notice before a model retires.
Agent operations
Agents take actions, not just answers. They need closer supervision than a chatbot.
- Agent step tracingShows the plan, each tool used and the reason for every action.
- Tool permission auditsConfirms each agent can use only the tools and data its job needs.
- Loop and stall detectionStops an agent that repeats itself or gets stuck.
- Approval queue managementTracks actions waiting for a person and escalates the ones left too long.
- Spending and action limitsCaps what each agent can spend or do in a day.
- Emergency stopPauses one agent or all of them at once.
- Memory upkeepReviews what agents remember and clears what is wrong or out of date.
- Handoff checksVerifies that work passed between agents arrives complete.
- Connector healthMonitors the APIs and tool servers that agents rely on.
Your team decides: which actions need sign-off and when an agent is paused.
Measured by: task success rate, share of tasks needing a person, cost per completed task.
Incidents and reliability
Something will go wrong eventually. What matters is how fast it is noticed and put right.
- Round-the-clock monitoringAutomated checks run day and night and raise an alert when a limit is crossed.
- Incident responseAn engineer assesses the problem, contains it and keeps you informed.
- RunbooksWritten steps for each known failure, so the response does not depend on one person.
- Automatic fallbacksSwitches to a backup model, a saved answer or a person when the main path fails.
- Root-cause analysisFinds why it happened, not only what happened.
- Post-incident reviewsA short written review with the fix and the change that prevents a repeat.
- Recovery testsProves that prompts, indexes and settings can be restored from backup.
- Status updatesKeeps your team informed during an incident and after it.
Your team decides: what customers are told and whether a system is switched off.
Measured by: time to respond, time to restore, repeat incidents.
Governance and reporting
When an auditor, a customer or your board asks how the AI is controlled, the answer should already be written down.
- AI inventoryA register of every AI system, its owner, its data and its risk level.
- Audit trail retentionKeeps records of requests, decisions and approvals for as long as you require.
- Monthly evidence packTest results, incidents, changes and access reviews in one place.
- Risk register upkeepKeeps each risk current, with its owner and its status.
- Regulation watchTracks new AI rules and tells you which systems they affect.
- Assessment refreshUpdates risk and impact assessments when a system changes.
- Model documentationWhat each system is for, its limits, its data and its change history.
- Retention and deletionApplies your data retention rules to logs, prompts and stored conversations.
Your team decides: risk levels, policy and what is reported to regulators or customers.
Measured by: evidence delivered on time, audit findings, overdue reviews.
Continuous improvement
Running AI well means it gets better, not just that it stays up.
- Failure analysisGroups wrong, unhelpful and abandoned answers by cause.
- Prompt and tool tuningFixes the largest causes first and tests each fix before release.
- Knowledge gap fillingLists the questions the AI could not answer and adds the missing content.
- Adoption analyticsShows which teams use the AI and where people give up on it.
- Return reportingTracks hours saved and cost per task against the baseline set before launch.
- Monthly reviewOne meeting with your owners to go through results and agree priorities.
- Improvement backlogA ranked list of the next fixes and features.
- Team playbooksShort guides and training so your staff get more from the AI.
Your team decides: the priorities for the next month.
Measured by: quality trend, adoption, return against the baseline.