Why do AI employee demos all look the same?
AI employee demos look alike because every vendor demonstrates the happy path, and on the happy path capable models produce similar output. A clean input produces a clean result in almost any serious product, which is why an hour of demo can leave a buyer with no basis for a decision.
The differences live in the awkward cases, the controls, and the record. A structured evaluation moves the conversation there deliberately, which is what the four areas below are for.
Area one: what remit can it actually hold?
The first area to score is remit: how much of a job the product can own without a person carrying work between steps. Ask what it does on Monday morning without being prompted, and what it decides for itself within its scope.
A product that owns a function will describe a rhythm: it reviews what changed, plans the week, produces, and brings proposals. A product that owns tasks will describe features. Both answers are honest; only one is buying you capacity.
Area two: what control do you get?
The second area is control: which actions are gated, who approves them, and whether you can change that list. Ask to see the approval screen for a real action and check whether it shows a true preview rather than a description.
Then ask the reverse question: what can it do without asking, today, out of the box? A vendor whose default is broad autonomy is asking you to trust a system you have not yet seen work. A vendor whose default is propose-only is giving you the safer starting point and letting you promote deliberately.
- Which actions are gated by default, and can we change the list?
- Does the approval screen show a true preview of the result?
- Can autonomy be granted per class of action and stepped back down?
- What are the spend caps, and are they hard limits?
Area three: what evidence does it produce?
The third area is evidence: whether every claim the system makes carries a source you can check. This is what separates a product you can defend in a review from one you have to take on faith.
Ask how it handles a claim it cannot verify. The good answer is that it flags the claim and refuses to publish it. The concerning answer is that it produces confident text either way, because that failure is invisible until it is expensive.
Area four: how cleanly can you leave?
The fourth area is exit, and it is the one buyers most often skip. Ask what happens to your content, your data, and your connected accounts if you stop paying, and how the access you granted gets revoked.
Check that the work product lives in your systems rather than theirs. A Digital FTE that publishes into your content management system and writes into your ledger leaves you with the output when it goes. One that keeps everything in its own dashboard leaves you with an export file and a migration project.
How should you score the four areas?
Score each area independently rather than forming an overall impression, because an impressive demo distorts everything downstream. Write the four scores before you discuss price, and record the specific answer that produced each one.
Weight control and exit higher than remit and evidence for a first deployment. A narrower remit with strong controls is a good place to start; a broad remit with weak controls is how pilots become incidents.
What should the evaluation cost you, and how long should it take?
A proper evaluation of an AI employee vendor costs a few hours of a manager's time, one written questionnaire, and a pilot. The hours go to watching the vendor's product with the four areas in mind: remit, control, evidence, and exit. The questionnaire goes to the vendor in writing and should come back in writing, with mechanisms rather than assurances; the security questions guide on ftes.ai is a ready template. The pilot is where the answer actually comes from.
The pilot should be one function with a recorded baseline, run for two full cycles, because the first cycle includes setup and remit tuning and is not representative. Its cost is the plan block, bounded by the wallet, plus the manager's approval time, which is real and should be logged. Its output is a decision: outcomes in Targets against the baseline, approval rate with reasons, tokens per role, and human hours, compared honestly. An evaluation that skips the pilot produces an opinion; one that runs it without a baseline produces a story; one that does both produces something a finance leader can sign.