Short answer
Measure an AI employee the way you would a new hire: against the outcomes of its role. Track speed of response, work completed, bookings or revenue it touched, escalations and corrections, and cost per task. Start with a baseline from before it began, review a short weekly report, and spot-check conversations yourself.
Start with the job, not the dashboard
The mistake is to open the reporting screen first and admire whatever numbers it shows. Counting messages sent is easy and says almost nothing. Start by writing the job in one sentence per task, in outcomes: 'every new lead gets a reply within five minutes', 'no call goes unanswered', 'quotes get three follow-ups'.
Those sentences become your measures. If you cannot say what good looks like for a task, the agent cannot either, and neither can you when something goes wrong.
Then record a baseline before the AI starts: how fast you replied, how many calls you missed, how many quotes were followed up. Without a baseline, any improvement is a feeling.
The measures that matter, by type of work
| Area | Measure | How to read it |
|---|---|---|
| Speed | Time from a lead or call to first response | Compare with your baseline; sudden jumps point to a broken connection or rule |
| Coverage | Calls answered, messages replied to, after-hours share | Anything unanswered should be explained in the report |
| Output | Appointments booked, quotes followed up, invoices sent, posts published | Compare against the plan you gave it, not against zero |
| Outcome | Bookings kept, quotes accepted, invoices paid, reviews received | Use your CRM or accounting data, and watch the trend over months |
| Quality | Share of conversations rated fine on your spot-checks; customer complaints about it | Sample a fixed number each week yourself |
| Escalation | How often it handed off to a human, and whether it should have | Too few means it may be guessing; too many means rules are vague |
| Corrections | How often you had to fix or reverse its work | Should fall steadily as it learns your rules |
| Cost | Hours billed and cost per completed task | Check against the published task-time chart |
Do not copy someone else's targets. A plumber's acceptable response time and a law office's are different, and any benchmark you read online was probably measured on somebody else's business. Your own baseline and your own goals are the only fair standard.
The weekly report you should expect
Oraculo works like an operations manager as well as a front desk: it tracks goals, flags what is slipping and sends a weekly KPI report. A useful report is short enough to read on your phone in two minutes.
- What was asked of it this week and what it completed.
- The core numbers: contacts handled, response time, bookings, follow-ups sent, invoices out.
- What went to humans and why, including anything still waiting for you.
- What it got wrong or was corrected on, so you can see the learning.
- What is slipping: stuck quotes, unpaid invoices, tasks assigned to staff and overdue.
- Questions for you: decisions it needs, rules it would like clarified.
More on report design is in AI KPI reports for small business owners. If you also use it to supervise staff goals, see the AI operations manager guide.
Spot-checks beat dashboards
Numbers tell you what happened. Only reading and listening tells you how it felt to the customer.
- Read ten threads a week at the start. Pick them at random, plus any that ended in a complaint or escalation.
- Listen to a few calls. Tone and pacing show up in audio and not in the transcript.
- Mystery shop it. Call or text as a customer would, including an awkward question. Try one at midnight.
- Check the boundaries. Ask for a discount, a refund or advice it should refuse, and confirm it hands off instead of improvising.
- Ask your staff. The people who work next to it spot problems before the report does.
Spot-checks are also how you decide how much approval to require. Important actions can require your approval first; as the correction rate falls, you can loosen the rules on the categories that have been clean for a month.
Judging value over 30, 60 and 90 days
A fair review has a calendar.
| When | Question | Evidence |
|---|---|---|
| First 2 weeks | Is it doing the basics correctly? | Spot-checks, response times, no dropped calls |
| Day 30 | Is it saving time and catching what used to slip? | Hours you got back, follow-ups completed, corrections trending down |
| Day 60 | Is it affecting outcomes? | Bookings, quote acceptances, paid invoices versus baseline |
| Day 90 | Is it worth its cost? | Compare billed hours with the value of work done and any hires or services it replaced |
Attributing revenue to an AI is rarely clean. Prefer simple claims you can defend, such as 'every lead was contacted within minutes, and more of them booked', over precise dollar figures. The ROI calculator guide shows a careful way to do the math, and AI employee mistakes covers what to fix when a number is off.
What it costs to run this review
The review itself is mostly your time: a few minutes a week once things settle. Oraculo is billed by hours of work using a published task-time chart, so cost per task is visible; Starter is $12 per hour with a six-hour weekly minimum ($312 a month), setup is $499 one-time with four one-hour Zoom sessions with an engineer, and there is a 30-day money-back guarantee plus three free billing disputes a month. See pricing for the Business tier.
Because of the guarantee, the first month works as a real trial with your own numbers. To walk through a sample weekly report and choose measures for your business, book a 20-minute demo.
Key takeaway
Write down the job, record a baseline, read a short weekly report, and spot-check the conversations yourself. That is enough to judge whether it earns its hours.
Frequently asked questions
What KPIs should I track for an AI employee?
Response time, calls and messages handled, appointments booked, follow-ups completed, escalations, corrections you had to make, and cost per task. Add outcomes like accepted quotes and paid invoices from your CRM and accounting data.
How do I know it isn't making mistakes?
Spot-check random threads and calls weekly at first, mystery shop it, and track how often you correct its work. Important actions can require your approval until corrections drop.
What is a good response time?
That depends on your business and your baseline. Measure how fast you replied before, set a target that suits your customers, and check that the agent meets it at all hours.
Can I see how many hours I'm billed for?
Billing is by hours of work using a published task-time chart, so each task type has a set value. There are three free billing disputes a month if something looks wrong.
When should I decide if it's working?
Check basics after two weeks, time saved and follow-up completion at 30 days, outcomes at 60, and cost against value at 90. The 30-day money-back guarantee covers the first trial.