To scope an AI agent project that reaches production, write down six things before anyone builds: the exact job the agent does, what a correct result looks like, which data and tools it can reach, what it may never do alone, what a single run may cost and how you will know it is still working. Most agent projects that stall were scoped as demos. These six answers turn a demo into a system.
The stakes are real. Reuters reported in June 2025 that Gartner expects more than two in five agentic AI projects to be cancelled by the end of 2027, citing rising costs and unclear business value. MIT's report The GenAI Divide found that nearly all corporate generative AI pilots, about 19 in 20, stall before they reach scaled adoption, and its lead author put that down to a gap between the tools and how organisations actually work, more than to the quality of the models.
Does the job need an agent at all?
Start here, because the cheapest agent is the one you do not build. Anthropic draws a useful line between workflows, where a language model follows a path written in code, and agents, where the model decides its own steps and which tools to use. Its advice is to use the simplest approach that works and add autonomy only when it clearly improves the result.
OpenAI's practical guide to building agents points to three situations where an agent earns its place: decisions full of exceptions, rule sets that have become expensive to maintain, and work that depends on unstructured material such as emails, documents or claims. If your job fits none of these, a well-built workflow will usually be cheaper, faster and easier to trust.
What exactly is the job, and who does it today?
Describe the job the way the person who does it would. Not "improve customer operations", but "read each new supplier invoice, match it to a purchase order, flag mismatches over 2% and draft the query email". Then record how that person does it today: the inputs they use, the decisions they make, how long it takes and how often it happens.
That current process is your baseline. Without it, nobody can say whether the agent is better, worse or simply different. It also tells you whether the agent assists a person, who still decides, or replaces a step entirely. Those are different projects with different risks, and they should be scoped differently.
What does a correct result look like?
Agree this before the build, in writing. The most practical way is a set of golden outputs: real examples, taken from your own work, each paired with the answer the best person on your team would give. A few dozen well-chosen cases, including the awkward ones, are worth more than a long list of adjectives.
Then pick a small number of success measures you can count: accuracy against the golden set, the share of cases escalated to a person, time saved per case and cost per run. On our own product, Vyera, every module is checked against golden outputs written for it, and a module is not considered finished until its real outputs hold up against them.
What data and tools will the agent need?
List every system the agent must read from or write to, and how it will reach each one. This is where many pilots fail. The MIT report found that enterprise AI rarely scales without proper integration into existing systems, and that tools bought from specialist suppliers or built through partnerships succeeded about twice as often as purely internal builds.
Treat each tool the agent uses as an interface that deserves design. Anthropic recommends putting as much care into the instructions and definitions an agent sees as you would into a screen for a human user: clear descriptions, examples, edge cases and inputs that are hard to get wrong. A vague tool produces a confused agent, however capable the model.
What is the agent allowed to do on its own?
Write a permissions table before the build. It forces the uncomfortable conversations early, when they are cheap.
| Type of action | Example | Starting permission |
|---|---|---|
| Read | Look up an order or a customer record | Allowed |
| Draft | Prepare a reply, a report or a refund request | Allowed, a person reviews |
| Act, low risk | Tag a ticket, update a status | Allowed once accuracy is proven |
| Act, high risk | Issue a refund, make a payment, delete data | A person approves every time |
| Never | Change permissions, contact people outside agreed lists | Blocked in code |
Read
Starting permission: Allowed
Example: Look up an order or a customer record
Draft
Starting permission: Allowed, a person reviews
Example: Prepare a reply, a report or a refund request
Act, low risk
Starting permission: Allowed once accuracy is proven
Example: Tag a ticket, update a status
Act, high risk
Starting permission: A person approves every time
Example: Issue a refund, make a payment, delete data
Never
Starting permission: Blocked in code
Example: Change permissions, contact people outside agreed lists
OpenAI's guide recommends layered guardrails rather than a single check, including relevance and safety filters, protection against prompt injection, filters for personal data, a risk rating for every tool and validation of outputs. It also recommends human oversight for sensitive, irreversible or high-value actions, especially early on.
One lesson from building Vyera applies here directly. When we traced the faults in one module, most came from a single cause: the model was writing fields it had no authority over. Give the model one defined job, let code own everything around it, and many errors never happen.
How should you test it before building the whole thing?
Build the evaluation before the agent. OpenAI's guide advises setting up evaluations first to establish a baseline, prototyping with the most capable model available, then testing smaller and cheaper models on each task to see where they still meet the standard. Anthropic stresses extensive testing in a sandbox, with guardrails in place, because errors compound when an agent acts on its own output.
In practice, that means running the agent against your golden outputs in a safe copy of your systems, reading the failures one by one and fixing the cause rather than patching the symptom.
What will each run cost, and which model should do what?
Estimate the cost of a single run before you commit: how many model calls it makes, how much text goes in and out of each, and what that costs at current prices. Multiply by your expected volume. Gartner's forecast names escalating costs as a main reason agent projects are abandoned, and it is far easier to design for a cost ceiling than to cut costs after launch.
The usual answer is to match the model to the step. Simple classification and routing rarely need the most capable model, while hard judgements often do. Set a cost limit per run and decide in advance what happens when a run reaches it. On Vyera, the AI Search module runs on its own cost model and limits, separate from the rest of the product, so one heavy feature cannot quietly consume the budget of everything else.
How should it roll out?
In stages, with evidence at each step. A sensible sequence looks like this:
- Shadow mode. The agent works on real cases, but a person still does the job and compares results.
- Assisted. The agent drafts and a person approves before anything happens.
- Autonomous for low-risk actions. The agent acts alone where the permissions table allows it, and escalates everything else.
- Expand. Widen what the agent may do only when the measures from earlier stages support it.
Start small, validate with real users, and grow capabilities over time.
How will you know it is still working next month?
Plan monitoring as part of the scope, not after launch. Log every run, review a sample each week, and track the measures you agreed at the start. Set failure thresholds that hand a case to a person automatically, for example after repeated failed attempts to understand a request.
Re-run the golden set every time something changes: a new model version, a revised prompt, a new data source. Agents drift quietly. On Vyera, a single module's build carries more than 2,500 automated tests for exactly this reason.
A one-page scope checklist
If you can fill in all ten, you have a project that can reach production. If you cannot, you have found the work that has to happen first.
Sources
- Building effective agents, Anthropic, December 2024
- A practical guide to building agents, OpenAI, 2025
- Over 40% of agentic AI projects will be scrapped by 2027, Gartner says, Reuters, June 2025
- MIT report: 95% of corporate generative AI pilots fail to deliver returns, Computing, on MIT's The GenAI Divide: State of AI in Business 2025
Written by
Rana Daniyal
Chief Executive Officer, Linvex Solutions
