How to Vet an AI Development Agency: 12 Questions to Ask Before You Sign (2026)

Before you sign with an AI development agency, ask for concrete evidence in five areas: model evaluation and guardrails, data security, model-agnostic architecture, transparent cost controls, and clear IP and handoff terms. If an agency answers with vague reassurance rather than specifics — a benchmark, a security certificate, a contract clause — you are very likely looking at a prompt-wrapper dressed up as an "AI-native" studio.

That distinction is worth taking seriously in 2026. MIT's NANDA initiative found that 95% of enterprise generative AI pilots deliver no measurable financial return, despite $30–40 billion in enterprise AI investment across the projects it studied. The 5% that did succeed shared a pattern: they were built by external partners whose tools succeeded roughly twice as often as internally built systems, deeply integrated into an existing workflow rather than bolted on as a standalone demo. In other words, who you hire to build the thing matters more than the model you pick.

Why "AI expertise" claims deserve more scrutiny now

Every studio, freelancer and agency now lists "AI" on their homepage. Some of that is genuine capability. A meaningful amount of it is not. Documented AI-washing cases include Nate Inc., which raised more than $42 million on largely fabricated AI claims, and Builder.ai, once valued at $445 million while much of its "AI" development work was reportedly performed by human engineers behind the scenes. The FTC has brought more than a dozen enforcement actions against inflated AI claims since 2024 under its "Operation AI Comply" initiative, and the SEC opened a dedicated Cyber and Emerging Technologies Unit in February 2025 partly to pursue the same pattern.

The commercial fallout is measurable too: AI-labelled startups are shutting down at roughly 2.5 times the prior year's rate, with unsustainable unit economics and "no moat beyond a GPT wrapper" cited as the leading causes. None of this means every AI agency is dishonest — most aren't. It means the burden of proof has shifted. A founder evaluating a partner in 2026 needs a checklist, not a gut feeling.

The 12 questions to ask before you sign

Group your due diligence into four areas. Ask each question directly, in writing, and expect a specific answer — not a slide deck full of logos.

Evals, guardrails and reliability

1. How do you evaluate model outputs before shipping to production? A real answer names a framework or test suite, references pass/fail thresholds, and distinguishes offline evals from live monitoring.
2. What guardrails stop the system from doing something costly or embarrassing? Look for concrete mechanisms — content filters, tool-call allowlists, human-in-the-loop approval for high-stakes actions — not "we prompt it carefully."
3. Can you show a benchmark or eval result on a dataset relevant to my use case? If they can't produce a number, they haven't measured it, which means you'll be the first person to find out it doesn't work.

Data security and compliance

4. Where does our data go, and who can see it? You want a clear data-flow diagram: which model providers receive your data, whether it's used for training, and where it's stored.
5. Do you hold SOC 2, ISO 27001, or an equivalent certification — or a credible plan to get there? Early-stage agencies may not have certification yet, but they should have a documented security posture and be willing to sign a proper data processing agreement.
6. What happens to our data and model access if the engagement ends? This should be a contract clause, not a verbal promise.

Model-agnosticism and architecture

7. What is your inference infrastructure — proprietary models, or an API wrapper? Both are legitimate business models, but the honest answer tells you what you're actually paying for.
8. What happens if our primary model provider raises prices threefold or is temporarily unavailable? This isn't hypothetical: a 2026 export-control order briefly took a frontier model offline mid-year, and it returned at double the price. A 2025 enterprise survey found 81% of US executives are at least somewhat concerned about dependency on a single AI vendor, and 47% said losing their primary vendor would disrupt a key business function. An agency that can't swap providers without a rebuild is building you into that risk.
9. Is the architecture model-agnostic by design? The same survey found the number of enterprises running five or more models in production rose from 29% to 37% year over year — multi-model is becoming the default, not the exception, for exactly this reason.

Cost controls and IP/handoff

10. How do you control and forecast our ongoing inference cost? Ask for caching strategy, model-routing (cheaper models for simple tasks, frontier models for hard ones), and a monthly cost estimate tied to expected usage.
11. Who owns the code, prompts, fine-tuned weights and evaluation data at the end of the contract? This needs to be an explicit IP assignment clause. Verbal assurances of "of course it's yours" are not enforceable.
12. Can you hand off a working system to another team, or does it only run inside your tooling? A genuine build should be portable. If it only works because of undocumented tribal knowledge on the agency's side, you've bought a dependency, not an asset.

Quick-reference vetting table

Area	What a real answer sounds like	Red flag
Evals & guardrails	Named framework, specific benchmark, monitoring in production	"We test it thoroughly" with no specifics
Data security	Data-flow diagram, DPA, certification or a dated plan for one	"Your data is safe with us," no documentation
Model-agnosticism	Architecture that can swap providers without a rebuild	Hard-coded to a single API with no abstraction layer
Cost controls	Caching, model-routing, a monthly cost forecast	No idea what your inference bill will look like at scale
IP & handoff	Explicit IP assignment clause; portable, documented system	Vague ownership language; system only runs on their tooling

Red flags that separate real builders from prompt-wrappers

A few patterns show up again and again in the agencies worth avoiding:

• They can't name the model provider or explain why they chose it over alternatives.
• Every demo is impressive; no one can produce a metric, an eval score, or a cost number.
• The proposal is priced entirely on hours, with no mention of ongoing inference cost or who bears it if usage spikes.
• IP and data ownership are addressed in a single vague sentence rather than a contract schedule.
• They resist a technical reference call with a past client who shipped a comparable system to production, not just a pilot.

None of these disqualify an agency on their own. Together, they're the pattern behind most of the "AI" failures making headlines this year.

Frequently asked questions

How long should AI agency vetting take?

Budget one to two weeks for a proper technical and commercial review, including at least one reference call. Rushing this stage is a false economy — the MIT NANDA data above shows most AI investment fails at the deployment stage, not the build stage, and vetting is where you catch that risk early.

Is it a red flag if an agency doesn't have SOC 2 certification?

Not on its own. Many strong, early-stage AI studios haven't completed formal certification yet. It becomes a red flag when they also can't describe their security posture, won't sign a data processing agreement, or can't say where your data is stored and processed.

Should I insist on a model-agnostic build even for an early prototype?

For a scrappy proof of concept, no — speed matters more, and you can rebuild the abstraction layer later. For anything heading toward production or fundraising diligence, yes: with 37% of enterprises now running five or more models and price shocks already a lived reality for some vendors, a hard-coded single-provider build is a liability you're choosing on purpose.

Who should own the IP if an agency builds our AI product?

In almost every case, you should own the code, fine-tuned weights, prompts and evaluation data your fee pays for — the agency retains only its own pre-existing tools and frameworks. This needs to be a specific clause in the statement of work, not an assumption.

What's the fastest way to spot a "prompt-wrapper" agency?

Ask for a benchmark result on a dataset relevant to your use case. A genuine AI builder has one, or can produce one quickly. An agency reselling a thin layer over a single API usually can't, because they've never measured performance beyond "it looked good in the demo."

Where this fits if you're evaluating a build

Vetting the partner is only half the decision — the other half is knowing what a properly scoped AI build should actually cost, so you can spot a quote that's wildly over or suspiciously under market. Our guide to what it costs to build an AI MVP in 2026 covers realistic tiers and timelines.

At UZO Lab, our AI Prototyping & R&D Sprint is deliberately structured to make this diligence easy: day-rate transparency (£500/day or £2,000/week), model-agnostic architecture from day one, and a written IP handoff as standard — not an afterthought. If you're weighing up a build and want a second opinion on a proposal you've already received, or want to scope one properly before you talk to anyone, book a free 30-minute workshop and bring your questions.

Sources: MIT NANDA GenAI Divide report via Legal.io; Morph, "AI Washing: A B2B Buyer's Guide"; IdeaProof, AI Startup Failure Analysis 2023–2026; AvePoint, AI Vendor Lock-In and Multi-Model Strategy (citing a16z Enterprise AI 2025 and a Zapier enterprise survey).