Friday, the demo works. Your chatbot answers every test question.
Monday, a real customer asks it for a refund. It tries.
Nobody on your team has read one chat yet. The one person who knows the refund steps is on leave.
That gap is where I see pilots stall. Michael Fauscette asked me about it on his show, the Disambiguation Podcast. His question was how to move an AI pilot to production. I have built 500+ projects with my team at AI Data House. These are the six things I check.
Short answer:
- Write the process down before anyone builds.
- Gather real past examples for the AI to learn from.
- Give it read access first. Edit access later.
- Decide what happens when each step fails.
- Name one person who reads the log every day.
- In healthcare, log who opened what before any model work.
Want the other side of this talk? Michael was a guest on my show too. That one covers why AI projects fail in the first place.
1. The process lives in one person's head
Michael asked what owners must get right before they scale. My answer was the process. Most owners cannot tell you every step.
mainly they don't know like okay uh I don't know how it is working um this um uh [does] John know how this work I I I didn't interfere in that so if John is not there nobody on on that process
John (a made-up name) is the one person who knows. You have a John. When John is sick, the task stops. An AI can't learn a task nobody wrote down.
before even going for AI, you should list all your processes and the steps and SOPs, guides, working instructions, all that.
Michael put it in one line:
I love that line and I I use it all the time. You know, you can't automate a broken process.
Do this Monday: Pick the one task only one person does. Ask them to write it in 10 numbered steps. Read it back to them. Fix what they skipped.
How we do it: We list every process and every step in one table. Then we pick the first one to build.
we generate the complete table by listing all the processes all the steps and then uh we select the best one
Not sure which row goes first? Read what to automate first.
2. The data the AI needs was never kept
Michael asked what data problems I see when I walk in. Usually the examples don't exist. Or they exist and nobody owns them.
for example I want to automate like uh uh my meetings [and] analysis process. So if I have recordings that are recordings of my meetings of my team meetings of last one month that would have very easily uh speed up the process
Recordings, emails, forms. Real ones. And someone has to own the process.
if the process is written structured and uh they know like who own it then it become a little easier
Do this Monday: Take the task from lesson 1. Find last month's examples. Emails, call recordings, forms. Count them. Start saving every new one from today.
How we do it: Before we build, we ask for real past examples. And one named owner for the process.
3. It gets every permission on day one
Michael asked how a team learns to trust an agent. He calls it moving from human in the loop to human in the lead. My answer: small steps, one permission at a time.
A chatbot starts on common questions only.
It will not be processing refunds or payment related stuff. It will be only answering FAQs.
Reporting works the same way.
So that is a read only access like the AI can read the data but it cannot edit or optimize anything. Once they are comfortable with that after a month they see okay it is generating good reports it is not hallucinating and now we should test one more step
Picking between a chatbot and a phone agent? See AI chatbot vs voice agent.
Do this Monday: List every AI tool you run. Next to each, write read, edit or send. Switch one "send" back to "draft" this week.
How we do it: Phase one is read only, or FAQs only. Each new permission is a separate step. Your owner turns each one on.
4. Nobody wrote what happens when a step fails
Michael made a point I agree with. A person can patch a broken step by hand. At machine speed, the mistake repeats fast.
My answer was gates.
You have to define like the gates on that if it if this step goes wrong what will be happening. If this step goes wrong what will be happening. So all the cases all the edge cases all the if and else conditions are defined then the process if if it even if it is automated it become very safe to run.
Do this Monday: Take one automation you run. Write 3 lines: "If this step fails, ___ gets told." Put a real name in each blank.
How we do it: Every step in the process table gets a fail path. And a person who hears about it. Before we build it.
5. Nobody reads the log after go-live
This was Michael's main question. What breaks between a working pilot and production? The biggest one: people think the job is done.
they just need to plug in that and it will keep working automatically. It's not like that. You need to monitor that process
Two people watch it. One from each side.
there should be a developer who will be monitoring that and there should be a person from the company who will be monitoring the output of that process and then you will finetune it. you will improve it uh to the actual process with for 15 days 20 days and then it become mature
And the log has to be read by someone.
how many actions it performed how many it times it was accurate how many time it didn't went well so that log is important and who is analyzing that log that is also important
Do this Monday: Paper sheet, 3 columns: did, right, wrong. One person fills it from your AI tool's output. Five days. Then look at the wrong column together.
How we do it: After go-live, one developer on our side and one person on yours read the output together. Every action the AI takes goes in the log.
6. In healthcare, the boring work is the real work
Michael asked what it takes where mistakes are serious. Most of that work is logs.
the first thing is very fancy where we are playing with models but uh there is very boring [work] as well and that is like you have to manage the all the logs who is doing what who is accessing patient information who is archiving that.
A phone agent needs one more rule.
you have to be sure like someone cannot ask a question. I mean it just ask a question about another patient and it answer that.
Do this Monday: Email each software vendor that touches patient data. Ask one thing in writing: can you show who opened which record?
How we do it: The access log and the "never read one patient's details to another caller" rule get built before any model work. More on clinic automation.
The half nobody covers
Most pilots start inside the office. When budget is tight, I start somewhere else.
if it is regarding the budget then we start with mostly with the uh the lead handling processes because leads are uh like the pipeline a very very important key point of the business.
The cheapest first fix is the lead that came in and never got answered.
We'll tell you if it's ready to automate.
Tell us the one task only one person on your team knowsThings said on the call, and what we would add
| Said on the call | What we would add |
|---|---|
| "It will keep working automatically." Watch at 6:15 | Name a developer and one of your people to read the output before it goes live. |
| "the AI can read the data but it cannot edit" Watch at 10:54 | Write down each permission your tool has. Turn on one new one at a time. |
| "you should automate one process" Watch at 5:51 | Pick the one your team complains about most. Show them the result. |
| "they know like who own it" Watch at 8:43 | Put one name next to each process before anyone builds. |
| "speed is the major benefit there" Watch at 16:07 | Keep the first build small enough for one or two of your people to explain. |
That card is where I started. An Excel tool for a manual check my classmates did by hand. Two to three days per product, done by hand. What slow, repeated task sits in your week?
We'll show you which of the six checks it missed.
Send us the pilot that worked in the demo and stalled afterYour Monday list
| # | Do this |
|---|---|
| 1 | One person writes their solo task in 10 steps |
| 2 | Count last month's examples of that task |
| 3 | Mark every AI tool read, edit or send |
| 4 | Write 3 "if this fails, ___ gets told" lines |
| 5 | Start a did, right, wrong sheet for 5 days |
| 6 | Healthcare: ask vendors for the access log |
FAQ
How do I move an AI pilot to production in a small business?
Write the process first. Give the AI read access only. Set what happens when a step fails. Name who reads the log.
Who on my team should read the AI's output?
The person who did the task by hand. They spot a wrong answer fastest.
Do I need clean data before I start?
You need real past examples and one owner for the process. Messy examples are fine to start with.
What should an AI tool be allowed to do first?
Read data and answer common questions. Edits, refunds and payments come later, one at a time.
More on how we build these: AI workflow automation. My own show is at /podcast.
Thanks to Michael Fauscette and the Disambiguation Podcast for having me. Full episode on his channel and the show page.
