Why Your Outsourcing Partner Can't Build Your AI Product

Most outsourcing partners struggle with AI products for a reason that has little to do with their engineers. The outsourcing model was built for software you can fully specify upfront. AI behaviour can't be specified that way. It has to be discovered, tested against real cases and watched in production, and a fixed-scope, ticket-driven engagement has no place for any of that.

We see this from both sides. Product companies come to us after a year of AI work with a vendor that delivers well on everything else. And we've learned the same lessons ourselves, building and running Avesta AgentOS in production for the last two years. This article explains what's going wrong, the five gaps to check, and what to change.

The signs you're probably seeing

If two or three of these sound familiar, the problem is likely the model, not the people:

• The demo worked months ago, and it still isn't live. Every review shows progress; nothing reaches real users.
• Estimates keep moving. Each sprint uncovers "one more edge case" in how the AI handles real data.
• Nobody can answer "how do you know it's working?" You hear about accuracy in demos, not about test suites run on every change.
• Fixing one thing breaks another. A prompt change improves one case and quietly breaks three others.
• Costs are a surprise. Model and infrastructure bills grow, and nobody can say what each task costs.
• When the AI gets something wrong, nobody can tell you why. There's no trace of what the agent saw, decided or did.

These aren't signs of a careless vendor. They are what happens when AI work is run like ordinary software work.

Why the outsourcing model breaks for AI products

Traditional outsourcing is built on a simple chain: requirements become a spec, the spec becomes tickets, tickets become code, and code is done when its tests pass. That chain works because ordinary software is predictable. The same input gives the same output, so you can describe the right answer in advance.

AI products break every link in that chain:

Research on AI project failure points the same way. A 2024 RAND study, based on interviews with 65 experienced data scientists and engineers, found the most common cause of failure was misunderstanding or miscommunicating the problem AI was meant to solve. Ten of the 50 industry interviewees also said rigid interpretations of agile processes are a poor fit for AI projects, which need room to explore and experiment before the work can be planned in fixed sprints.

A fixed-scope contract makes both problems worse. The vendor is paid to build what the spec says, not to question whether the spec solves the user's problem. And every week spent exploring looks like a slipped deadline.

A fixed-scope contract pays a vendor to build the spec. An AI product needs someone to find out whether the spec is right.

The five gaps to check

1. Product sense

AI makes code cheap. What's scarce is knowing which job the AI should do for the user, and whether it creates value.

Jobs-to-be-Done gives a practical way to think about this. People switch to a new solution when two forces push them forward: friction with how they work today, and the pull of a better way. Two other forces hold them back: anxiety about the new solution (will it get things wrong? will I lose control?) and the habits of the current way.

Most AI features are designed only for the first two. They promise a better way and ignore the anxiety and the habit, so users try them once and go back to what they know.

A product-minded team designs for all four. For an AI feature, that might mean showing sources so users can check the answer, letting a person approve before the agent acts, or fitting the agent into the tool users already open every morning instead of asking them to learn a new one. A ticket-driven vendor builds what's in the spec. A product-minded team asks why users would adopt it, runs small experiments, and drops the ones that don't land.

At Avesta, our whole team is trained in Jobs-to-be-Done, so this thinking doesn't depend on one product manager. Engineers know the customer's struggle, the forces behind it and the habits they're designing around. We cover this in more depth in what makes a team AI-native.

2. Production experience with agents

Building a demo and running agents for real users are different skills. Production means messy inputs, slow or failing tools, partial completions, rate limits and cost spikes. A team that has only built prototypes will discover all of this on your project, at your expense. We've written about the gap between prototype and production before.

Ask what AI systems they have running in production today, for how long, and what broke. For us, that answer includes F&F's order agent, which has handled 500 to 1,000 dealer orders a day since going live, and India Avenue's investor support assistant, which resolves over 80% of enquiries without the team.

3. Evals

Unit tests check code. Evals check behaviour: a growing set of real cases with expected outcomes, run on every change. Without them, nobody knows whether a prompt tweak or model upgrade made things better or worse, which is why fixes keep breaking other cases.

If your partner can't show you an eval suite for your product, with the results from the last change, this gap is open.

4. Guardrails and access control

Once an agent can read your data or take actions in other systems, you need limits: what it can see, what it can do, for which user, and when a person must approve. Partners new to agents often treat this as a security review at the end. It has to be designed in, because retrofitting it usually means rebuilding. An email or document the agent reads can even carry hidden instructions, so the limits must hold when the agent itself is fooled.

In AgentOS, access control works per user, tenant and tool, and higher-risk actions route to a person for approval. Those weren't add-ons. They came from running agents for enterprise clients.

5. Agent behaviour analytics

In a web or mobile product, the product team learns from user behaviour analytics: clicks, page views, funnels, which features people find and how engaged they are. That feedback loop is how products improve.

AI-native products need a second loop, because much of the experience happens inside a conversation or an agent's work, where clicks and page views see nothing. The product team needs to know:

• Is the agent doing what we expect? How often it answers correctly, makes something up, or fails on a step.
• Did the user get what they needed? Whether they had to rephrase, repeat or retry, or gave up.
• What are users actually asking for? The topics and requests in conversations, which tell you what to build next.
• What did it cost? Model and tool spend per task.

That is agent behaviour analytics, with conversation analytics alongside it. Without it, the team improves agents by guesswork and finds problems through customer complaints, even while the dashboard KPIs look green. With it, every week of real use tells you which behaviour to fix.

We built this into AgentOS as Metricsense, our agent behaviour analytics module, so the team can see where agents fall short and improve them. Ask your partner what they would use to see how your agents behave with real users. If the answer is page views or "we check the logs when someone complains", this gap is open.

What to change

You don't have to replace a partner who serves you well elsewhere. There are three options, from lightest to heaviest:

1. Change the engagement, not the partner. Rewrite acceptance criteria around evals on real cases, not feature lists. Fund a short discovery phase before committing to scope. This works if the partner already has production AI experience and the problem is the contract shape.
2. Go hybrid. Keep your current partner on core systems, integrations and maintenance, where fixed scope works. Bring in an AI-native team for the AI workflows, embedded with your product owner. This is often the best fit, because it keeps continuity where you need it.
3. Move the AI work entirely. If the AI is the product and your partner has none of the five capabilities, a hybrid only delays the decision.

Outsourcing is still the right call for plenty of work: well-specified features, platform maintenance, integrations and migrations. AI products simply need a different way of working.

Ten questions to ask your partner this week

If most answers land in the right-hand column, the problem isn't effort. It's experience your partner doesn't have yet.

Your vendor isn't bad at engineering. The model they work in is bad at uncertainty.

Where to start

Pick one AI workflow that matters, define what "working" means as a set of real test cases, and run your partner's current build against them. The results will tell you which of the three options above you need.

If you want a second opinion, that's what our discovery sprint is for. Avesta's AI-native product engineering team has already built the production layer in AgentOS. In a short, fixed-scope sprint, we show you where your AI work stands and leave you with an eval baseline and a build plan. Book a discovery call to start.

Sources

• James Ryseff, Brandon De Bruhl and Sydne J. Newberry, The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed, RAND Corporation, August 13, 2024