Where should you start when adding AI to an existing product?

Start from a workflow your users already repeat, not from a chat box. A chat interface makes scope impossible to define and puts the burden of discovery on the user, while a repeated workflow tells you what the feature has to do, what a correct output looks like, and how you will know it worked. What decides whether an AI feature survives is rarely the model.

What are the integration patterns for AI features?

Four patterns cover nearly every AI feature, and they build on each other. A single model call inside an existing flow is the simplest. Retrieval over customer data, or RAG, grounds answers in your own content. Tool calling lets the model act against your own API. Multi-step agents chain those together. Pick the simplest pattern that fits the workflow, because every step up adds failure modes, latency and cost.

How do you stop an AI feature leaking data between customers?

Enforce tenant isolation and permissions in your code, never in the prompt. A prompt instruction is a request the model can be argued out of, and indirect prompt injection is a live attack against exactly that. Filter at the data layer so retrieval can only ever return records the requesting user is entitled to see, and treat cross-tenant leakage and excessive agency as launch requirements rather than post-launch hardening.

When should you build the evaluation harness for an AI feature?

Before the feature, not after it. Build the test set from real product data, evaluate retrieval and generation separately so you know which half failed, use the cheapest check that works, and run the suite in CI. Treat the pass threshold as a product decision rather than a technical one. Without evals, a feature that is quietly degrading looks exactly like one that is working.

How to Add AI Features to an Existing Software Product

Your customers now expect AI inside the software they already pay for, and they are spending real money on it. Menlo Ventures estimates enterprises spent $37 billion on generative AI in 2025, with roughly $19 billion of that going to user-facing applications rather than infrastructure [6]. Gartner expected the vast majority of enterprise applications to carry embedded AI assistants by the end of 2025, and projected that 40% would include task-specific AI agents by the end of 2026, up from under 5% a year earlier [1].

Buyers are also getting stricter. In July 2026, Gartner reported that enterprise AI budgets are under growing scrutiny, with spending shifting toward vendors that can prove value on cost, latency, performance, and reliability [2].

That combination is the real challenge. An AI feature is easy to demo and hard to make worth paying for. MIT’s NANDA initiative found that only about 5% of the generative AI pilots it studied produced rapid revenue gains, with most showing little measurable impact on the P&L [3]. Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027 because of rising costs, unclear business value, or weak risk controls [4].

The difference between the features that stick and the ones that get quietly removed is rarely the model. It is product and engineering discipline: picking the right workflow, building on an architecture that respects your existing data and tenancy model, measuring quality before shipping, and engineering for cost and failure. This guide walks through those decisions in the order we recommend making them.

Why AI features are harder than the demo suggests

A prototype that sends a good prompt to a model API can be built in an afternoon. Almost all of the real effort sits between that prototype and a feature you can sell, because LLM features break assumptions your existing software quietly relies on.

Outputs are non-deterministic, so the same input can produce different results and a passing manual test proves very little. Quality is a spectrum rather than pass or fail. Cost scales with every request instead of staying roughly flat with hosting. Latency is measured in seconds, not milliseconds. And the model can be manipulated by the text it reads, which opens a security surface your current threat model does not cover.

The organizations getting value have adapted to this. In McKinsey’s 2025 State of AI survey, 88% of organizations reported using AI in at least one business function, but only about a third had begun scaling it. The small group of high performers were nearly three times as likely as everyone else to fundamentally redesign workflows instead of layering AI on top of old ones [5]. MIT’s researchers landed in the same place: tools stalled mainly because they did not fit into how work actually gets done, not because the underlying models were weak [3].

Organizations using AI in at least one business function

88%

Adoption is effectively universal. It is not the hard part.

Generative AI pilots producing rapid revenue gains

~5%

The distance between those two numbers is what this guide is about.

Two tiles rather than one chart, because these are two different studies and putting them on a shared axis would imply a precision neither has. Adoption is McKinsey’s 2025 State of AI survey; the pilot figure is MIT’s NANDA research, which is self-reported and measures rapid revenue gains rather than all value. Read together they frame the problem the rest of the guide addresses: almost everyone has shipped something, and very little of it pays.

Step 1: Start from a workflow, not a chat box

The most common first AI feature is a generic chat panel in the corner of the app. It is also one of the least likely to get used. Users do not know what to ask it, it sits outside the task they came to do, and it competes with general-purpose assistants they already have open in another tab.

A better starting point is a specific task your users already perform inside your product that is repetitive, heavy on text or documents, and depends on data you already hold. Embed the AI directly into that step.

Two questions filter candidates quickly. First, can a user verify the output faster than they could produce it themselves? A drafted reply, a summary that links back to source records, or an extracted field shown next to the original document all pass. Second, what does a wrong answer cost? If the answer is “the user edits it,” you can ship early and learn. If the answer is “money moves to the wrong account,” the feature needs a human confirmation step or a narrower scope.

Feature type	What it does	Example	Build complexity	Cost of a wrong output
Summarize or draft	Generates text the user reviews and edits	Draft a support reply from ticket history	Low	Low: the user edits it
Extract or classify	Turns unstructured input into fields or labels	Pull line items from uploaded invoices, route tickets	Low to medium	Medium: verify against the source
Search and Q&A over customer data	Answers questions with citations from tenant data	“Which contracts renew next quarter?”	Medium	Medium: citations make it checkable
Copilot actions	Proposes actions through your API, executes after confirmation	“Create follow-up tasks for these five accounts”	Medium to high	Higher: it writes data
Autonomous agents	Plans and runs multi-step work with little supervision	Reconcile records across systems overnight	High	Highest

Most products should start in the top three rows. Even Gartner notes that many use cases marketed as agentic do not actually need agentic implementations [4]. Begin with assistance that makes an existing task faster, then graduate to actions and agents once you have evals, telemetry, and user trust in place. We cover interface patterns that drive adoption in AI copilots inside B2B SaaS.

Step 2: Pick the integration pattern

Four patterns cover nearly every AI feature, and they build on each other.

A single model call inside an existing flow. Your backend sends the relevant context and asks for structured output that conforms to a schema. Your code validates the result and uses it like any other data. Most summarization, classification, extraction, and drafting features live here. It is the cheapest pattern to run and the easiest to test, so start here whenever the use case allows.

Retrieval over customer data (RAG). When answers depend on a tenant’s own documents or records, you retrieve the relevant content, pass it to the model, and instruct it to answer only from that content with citations. In practice, retrieval quality usually matters more than model choice. Our enterprise RAG guide covers the architecture, and reducing hallucinations in enterprise RAG covers the failure modes.

Tool calling against your own API. The model decides which of your API operations to call and with what arguments, and your code executes those calls. This is how copilots take actions. The Model Context Protocol standardizes how tools are exposed to AI clients, which also lets external assistants work with your product.

Multi-step agents. The model runs in a loop: plan, call a tool, inspect the result, continue. This unlocks bigger tasks, but cost, latency, and error rates compound with every step. See AI agents in the enterprise and single vs multi-agent systems before committing to this pattern.

For multi-tenant products, one rule is non-negotiable across all four patterns: tenant isolation and permissions are enforced in your code, never in the prompt. Retrieval queries filter by tenant and user permissions at the database layer before anything reaches the model, and tool calls execute with the requesting user’s credentials. A system prompt that says “only use this customer’s data” is not an access control.

Step 3: Choose model access you can change later

Three decisions matter here.

Access path. You can call model providers directly or go through a cloud platform such as Azure OpenAI, Amazon Bedrock, or Google Cloud’s Vertex AI. Cloud platforms often make enterprise sales easier: data stays inside a cloud your customers have already reviewed, spend can count against existing commitments, and regional deployments help with data residency. Direct provider APIs can expose new models and features sooner, though the gap varies by provider. We compare the options in detail in Azure OpenAI vs OpenAI API vs AWS Bedrock.

Model selection. Do not look for “the best model.” For each task, pick the smallest and cheapest model that passes your evals. A single feature can reasonably use a small model for classification and a larger one for final generation. Choosing an LLM for your product walks through the tradeoffs.

An abstraction layer. Models improve and prices fall fast. Epoch AI found that the price of reaching a fixed level of model performance has dropped by roughly 9x to 900x per year, depending on the benchmark and threshold [7]. The model you launch with is unlikely to be the one you run a year from now. Put a thin internal gateway between product code and providers: one interface for model calls, prompts versioned as configuration rather than strings scattered across the codebase, and a log of every request with the model, prompt version, token counts, latency, and cost. With that in place, switching models becomes an eval run instead of a rewrite.

Step 4: Build the evaluation harness before the feature

Without evals, every prompt tweak and model upgrade is a guess. Hamel Husain, who advises teams building LLM products, argues that failed AI products almost always share one root cause: the lack of a robust evaluation system [9]. That matches what we see.

A practical evaluation setup for a first feature looks like this:

Build the test set from real product data, anonymized where needed, rather than invented examples. Fifty to two hundred cases covering common inputs and the known edge cases is enough to start.
Use the cheapest check that works. Code assertions handle format, schema validity, required fields, and forbidden content. Use an LLM as a judge only for fuzzier criteria like faithfulness to sources, and spot-check its verdicts against human review before trusting it.
Evaluate retrieval and generation separately for RAG features, so a failure points to the component that caused it.
Treat the pass threshold as a product decision. Unlike unit tests, you rarely need a 100% pass rate; the right bar depends on how costly a wrong output is [9].
Run the suite in CI whenever a prompt, model, or retrieval setting changes.
Review production traces regularly after launch. Every real failure you find becomes a new test case.

Step 5: Engineer for cost, latency, and failure

Cost. Falling token prices do not guarantee falling bills. Usage grows as adoption grows, and agent loops multiply calls per task. The controls that matter most:

Route by task. Small models handle classification and extraction; larger models run only where evals show they are needed.
Cache aggressively. Use provider-side prompt caching for long, repeated context such as system prompts and reference material, plus application-level caching for repeated requests.
Trim context. Send what the task needs, not the entire record history.
Batch background work. Nightly enrichment and backfills can run through batch APIs, which providers typically offer at a discount.
Enforce budgets. Set per-tenant and per-user limits. OWASP lists unbounded consumption among the top ten risks for LLM applications [8].

Decide early how the feature will be priced, whether bundled, tiered, or sold as usage credits, because that decision determines whether your unit economics work. The full breakdown is in LLM API costs in production.

Latency. Stream tokens for anything user-facing so the wait feels shorter. Move long-running tasks to background jobs with a notification when they finish. Use smaller models on latency-sensitive paths, and run independent calls in parallel.

Failure. Model providers have outages and rate limits like any dependency. Set timeouts and retries with backoff, keep a fallback model or provider that has passed your eval suite, validate structured outputs and handle malformed responses, and design the product so core functionality keeps working when the AI feature does not. We go deeper in making LLM features reliable.

Step 6: Treat security and governance as launch requirements

The OWASP Top 10 for LLM Applications ranks prompt injection as the leading risk, alongside sensitive information disclosure, improper output handling, excessive agency, system prompt leakage, and weaknesses in vector and embedding stores [8]. For an existing product, the realistic threats look like this:

Indirect prompt injection. Instructions hidden inside content your feature reads, such as an uploaded document or an inbound email, attempt to hijack the model. Treat all retrieved text as untrusted.
Excessive agency. A copilot with broad tool access can be manipulated into causing damage. Give tools the narrowest permissions possible, require user confirmation for writes and anything irreversible, and log every action.
Cross-tenant leakage. Prevented by the architecture rule in Step 2, not by prompt wording.
Improper output handling. Treat model output as untrusted input. Sanitize it before rendering, and never execute it directly as SQL, code, or shell commands.

Enterprise customers will also raise procurement questions: which model providers process their data, whether that data is retained or used for training, where processing happens, and whether admins can switch AI features off. Have written answers and admin controls ready before launch, not during your first security review. For the broader picture, see enterprise RAG security and governance, enterprise AI governance, governing AI agents, and our EU AI Act guide.

Step 7: Design for trust and measure adoption

An AI feature that customers do not use is pure cost. The interface decisions that most affect adoption are simple:

Put AI output where the work happens, pre-filled into the field or draft, rather than in a separate window.
Show sources for anything factual, linked to the underlying records.
Make outputs editable and actions reversible.
Label AI-generated content clearly.
Capture feedback from behavior. Accept, edit, discard, and regenerate are more reliable signals than thumbs-up buttons most users ignore.

Then measure what matters to the business, not just whether the feature launched: the share of eligible users who use it weekly, how often outputs are accepted, how heavily users edit outputs before using them, time to complete the workflow compared with the baseline, cost per successful outcome, and retention or expansion among accounts that use the feature.

A realistic rollout plan

For a single, well-scoped feature, this is the shape we plan against. Treat the durations as rough ranges; the state of your data and permissions model is the biggest variable.

Phase	Typical duration	What happens	Exit criteria
Discovery	1 to 2 weeks	Choose the workflow, audit data access and permissions, draft the eval set, estimate unit cost	Test set exists and feasibility is confirmed
Prototype and evals	2 to 4 weeks	Build the chosen pattern, iterate on prompts and retrieval against evals	Pass threshold met on the test set
Private beta	3 to 6 weeks	Feature-flagged release to design-partner customers with tracing, cost dashboards, and feedback capture	Quality and cost on real usage within targets
General availability	Ongoing	Budgets, rate limits, fallbacks, admin controls, security documentation	Adoption and unit economics tracked against goals

Once the first feature is stable, the gateway, eval harness, and telemetry you built become shared infrastructure, and each subsequent AI feature ships faster.

Build in-house, hire, or partner

Once you know what to build, the next question is who builds it. MIT’s research found that AI tools built with external partners reached deployment roughly twice as often as purely internal builds, about 67% versus 33% [3]. The authors cautioned that these outcomes were self-reported and do not prove that partnering causes success [10]. The pattern is still plausible: a first production AI feature requires evals, retrieval engineering, cost controls, and LLM security practices that most product teams have not needed before.

A practical rule: if AI will become a core differentiator for your product, you need to own the capability long term, so any partner engagement should be structured to hand over the eval harness, gateway, and working knowledge to your team. If AI is a supporting feature rather than the product’s core, a partner build with ongoing support is often cheaper than hiring a dedicated team. We break down the options in build in-house, hire, or partner, and typical budgets in what it costs to build an enterprise AI assistant.

Pre-launch checklist

The feature targets a specific workflow, and users can verify its output faster than doing the task themselves.
Tenant isolation and permissions are enforced in code for every retrieval and tool call.
An eval suite built from real data runs in CI, with an agreed pass threshold.
Model calls go through a gateway that logs model, prompt version, tokens, latency, and cost.
Per-tenant budgets, rate limits, timeouts, and a tested fallback are in place.
Writes and irreversible actions require user confirmation, and every action is logged.
Written answers exist for data processing, retention, and training questions, and admins can disable the feature.
Adoption, acceptance, and cost-per-outcome metrics are instrumented from day one.

Keep reading

This guide is the hub for our series on adding AI to existing products:

CMW AI Labs builds production AI features for software companies, from retrieval-grounded assistants to voice agents, with the evaluation, cost, and security engineering covered in this guide. If you are scoping your first AI feature and want a second opinion on the workflow or architecture, talk to our team.

Sources

Gartner, “Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026, Up from Less Than 5% in 2025,” August 26, 2025. https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025
IT Brief UK, “Gartner forecasts AI spending to hit USD $64bn in 2026,” July 21, 2026. https://itbrief.co.uk/story/gartner-forecasts-ai-spending-to-hit-usd-64bn-in-2026
Fortune, “MIT report: 95% of generative AI pilots at companies are failing,” August 18, 2025 (covering MIT NANDA, The GenAI Divide: State of AI in Business 2025). https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/
Gartner, “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027,” June 25, 2025. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
McKinsey & Company, “The state of AI in 2025: Agents, innovation, and transformation,” November 2025. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
Menlo Ventures, “2025: The State of Generative AI in the Enterprise,” December 9, 2025. https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/
Epoch AI, “LLM inference prices have fallen rapidly but unequally across tasks.” https://epoch.ai/data-insights/llm-inference-price-trends
OWASP GenAI Security Project, “OWASP Top 10 for LLM Applications 2025.” https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/
Hamel Husain, “Your AI Product Needs Evals.” https://hamel.dev/blog/posts/evals/
Let’s Data Science, “MIT NANDA’s 2025 Report Says 95% of Organizations Saw No GenAI Return” (methodology and caveats). https://letsdatascience.com/news/mit-report-documents-genai-pilot-roi-gap-e0924d7d