Most organizations estimate the cost of their AI agent by adding up API calls. A quick back-of-the-envelope calculation, a few cents per request, a monthly total that seems manageable, and the decision to go ahead gets made on that basis. Six months later, the actual bill is two to four times higher than the initial estimate, and no one on the team can explain where the money went.
The problem is not the API. LLM inference costs are today the most visible line item and, for most mid-sized deployments, far from the largest. What drives overruns are the layers the team never budgeted for: RAG infrastructure, integration development, ongoing maintenance, production monitoring. These costs are not new or unusual. They are simply rarely estimated with the same rigor as the API cost.
This article presents a complete breakdown of a real, anonymized case: a 200-person industrial subcontractor that deployed an AI agent for supplier contract processing and analysis. We walk through every cost line, compare year 1 to year 2, and examine the specific question of GPU self-hosting. The goal is not to discourage deployment. It is to budget it correctly.
The concrete case: a mid-market manufacturer, a contract agent, 50 active users
Context: a 200-employee industrial subcontractor with a 3-person procurement team and a 2-person legal team. Before the agent, reviewing supplier contracts required manually searching through 8,000 PDF documents stored in SharePoint, with an average response time of 30 to 45 minutes per query.
The deployed agent is a conversational RAG agent. A user asks a question in plain language ("What is the notice period in the Dupont Industries contract?"), the agent retrieves the relevant passages from the document base, synthesizes them, and replies with source references. The v1 scope covers search, extraction and synthesis. No write-back to third-party systems.
Production volumes: 50 active users, 250 requests per day on average, 8,000 indexed documents (approximately 40 million embedding tokens at launch). The agent runs on GPT-4o via the OpenAI API. The vector store is hosted on Pinecone Serverless. Orchestration is handled by LangChain. Monitoring runs through Langfuse Cloud.
LLM inference: the most visible cost, rarely the largest
GPT-4o is priced at 2.50 dollars per million input tokens and 10 dollars per million output tokens. For a well-architected RAG agent, a single user request involves roughly 800 tokens of system prompt, 2,000 tokens of retrieved context (4 to 5 passages of around 400 tokens each), 300 tokens of question and conversation history, and 700 tokens of generated response. That is 3,100 input tokens and 700 output tokens per LLM call. A multi-step agent averages 3 calls per user request: question reformulation, retrieval and reranking, final synthesis.
Cost per user request: 3 x ((3,100 x 2.50 / 1,000,000) + (700 x 10 / 1,000,000)) = 3 x (0.00775 + 0.007) = $0.044.
250 requests per day x $0.044 = $11 per day, roughly $330 per month or around $4,000 per year.
This figure rises when usage increases, but also when context grows without discipline. An agent that injects long, poorly filtered chunks, fails to compress conversation history, or chains unnecessary LLM calls can easily double or triple its inference bill with no improvement in output quality. Prompt engineering has a direct, measurable impact on this line item.
A note on model optimization: routing reformulation and retrieval calls to Claude Haiku 4.5 (1 dollar / 5 dollars per million input / output tokens) while keeping GPT-4o only for the final synthesis step reduces inference costs by 40 to 50% with no noticeable degradation in perceived quality. This is one of the first adjustments we document on every engagement.
The invisible layers: RAG, infrastructure and monitoring
Embeddings. Indexing 8,000 documents (roughly 40 million tokens) with text-embedding-3-small at $0.02 per million tokens costs $0.80 at initialization. Inference-time embeddings (vectorizing each user question) add just a few cents per month at this scale. This line item is negligible.
Vector database. Splitting 8,000 documents into 4 to 5 chunks each produces roughly 35,000 to 40,000 stored vectors. At that scale, Pinecone Serverless bills between $20 and $40 per month depending on read volume. This is a stable, predictable cost that only grows if the document base expands significantly.
Application infrastructure. The agent runs on a lightweight cloud backend (2 to 4 vCPUs) at $80 to $150 per month. A PostgreSQL instance for session data and metadata adds $40 to $60. Total infrastructure: $150 to $200 per month, or around $2,000 per year.
Monitoring and observability. Langfuse Cloud Team tier is priced at $59 per seat per month. For a 3-person team operating the agent (2 developers and 1 business owner), the monthly cost is $177, roughly $2,100 per year. The open-source alternative is self-hosted Langfuse: zero licensing cost, but additional infrastructure to operate and setup time to plan for.
Total for all invisible layers, excluding development: $4,500 to $5,500 per year. Inference accounts for roughly 70 to 75% of that total. It is manageable, predictable, and adjustable in either direction through architecture choices.
The real dominant cost: development and integration
This is where initial estimates go wrong. API costs are calculable, tangible, measurable. Development costs are diffuse, spread over months, and systematically underestimated at project start.
For this case, the development scope spans several distinct phases.
Architecture and design (2 to 3 weeks): model selection, document chunking strategy, hybrid retrieval approach, agent tool design, SharePoint integration plan.
RAG pipeline development (3 to 4 weeks): document ingestion, semantic chunking, indexing, retrieval pipeline, retrieval quality evaluation on a 200-question golden set representative of real user queries.
Agent and prompt system development (3 to 4 weeks): system prompt structure, multi-step reasoning logic, error handling, response guardrails, out-of-scope query filtering.
SharePoint and SSO integration (3 to 4 weeks): almost always the longest and least anticipated phase. Existing document connectors are rarely reusable as-is. Enforcing source-level permissions, meaning ensuring the agent only surfaces documents the requesting user is authorized to access, adds a layer of complexity that no tutorial covers.
Testing, business validation and production deployment (2 to 3 weeks): building the evaluation set with procurement staff, quality iteration cycles, load testing, production rollout.
Total: 14 to 18 weeks of development. With a team of 2 senior developers at $600 per day on external contract:
- Low scenario: 14 weeks x 2 devs x 5 days x $600 = $84,000
- High scenario: 18 weeks x 2 devs x 5 days x $600 = $108,000
On top of that, internal team time (project manager, business stakeholder, CIO): 0.3 to 0.5 FTE over 4 months, adding $15,000 to $25,000 in fully loaded cost.
Year 1 development budget: $100,000 to $133,000.
In year 2, this line drops sharply. Ongoing maintenance of a stable agent (prompt updates, adding new document sources, adjustments following quality drift detected in monitoring) amounts to 1 to 2 months of development per year: $12,000 to $24,000.
Year 1 vs. year 2: the full picture
The table below consolidates costs for the described deployment.
| Line item | Year 1 | Year 2 |
|---|---|---|
| LLM inference | ~$4,000/yr | ~$5,500/yr |
| Embeddings | less than $100/yr | less than $100/yr |
| Vector database | ~$440/yr | ~$440/yr |
| Application infrastructure | ~$2,000/yr | ~$2,000/yr |
| Monitoring (Langfuse) | ~$2,100/yr | ~$2,100/yr |
| Initial development | $100,000-133,000 | 0 |
| Maintenance and updates | ~$5,000 (stabilization) | $12,000-24,000 |
| Indicative total | $113,000-146,000 | $22,000-34,000 |
Two realities stand out that teams consistently discover too late.
First: operating costs (infrastructure, API, monitoring) stay below $9,000 per year. They are predictable, adjustable, and reducible through architecture optimization. They are not what sinks projects.
Second: development accounts for 85 to 90% of year 1 spend. This is structural. A properly integrated agent, with tested retrieval quality, operational guardrails, and a proprietary evaluation set, takes 3 to 5 months of qualified work. Anyone promising to deliver that in 3 weeks for $15,000 is delivering a POC, not a production system.
Year 2 changes the picture entirely. Development is absorbed, and ROI begins to materialize. In this case, procurement staff estimated a 60 to 70% reduction in contract research time. Three buyers at $55,000 fully loaded annual cost each saving 4 hours per week represents roughly 30 FTE-weeks recovered annually, about $80,000 in value created. Breakeven is reached during year 2.
When GPU self-hosting changes the equation (and when it does not)
The question comes up regularly in client engagements: "Rather than paying OpenAI's API, would we be better off hosting an open-source model on our own GPU infrastructure?"
The answer depends entirely on volume. For the deployment described here, the monthly LLM bill is $330. Renting an H100 on Lambda Labs costs roughly $2.50 to $3.44 per hour, or $1,800 to $2,500 per month if the machine runs continuously. Self-hosting costs 5 to 8 times more than the API at this load level. There is no decision to make.
Self-hosting starts to make economic sense when the API bill consistently exceeds $5,000 to $7,000 per month. The FinOps Foundation has quantified this threshold: for workloads below 500,000 tokens per day, public API pricing is almost always cheaper than self-hosting on rented GPU. Above 2 to 3 million tokens per day with GPU utilization above 70%, the balance starts to shift.
At those volumes, an open-source model such as Mistral Large 2 or Llama 4 Scout hosted on a dedicated GPU cluster can reduce the marginal cost per token by 60 to 80%. But the full picture requires adding: GPU rental cost ($1,800 to $5,000 per month depending on configuration), operational overhead (0.3 to 0.5 FTE of MLOps engineering), and model update cycles that do not manage themselves. Self-hosting is not a default cost saving. It is an advanced option that makes sense in specific contexts of high sustained volume or hard data sovereignty requirements.
What we put in place on engagements
When we work with an organization on AI agent budgeting, we always start by separating three time horizons with very different cost structures.
The POC (4 to 8 weeks): near-zero infrastructure cost, low development cost, no structured monitoring, no deep integration into the core IT stack. It is a validation tool, not a product. Typical budget: $15,000 to $30,000.
The production pilot (3 to 6 months): real integration into existing systems, monitoring in place, evaluation set built, load testing completed. This is where the budget jumps. Typical budget: $80,000 to $140,000 depending on integration complexity.
Scaling (from year 2 onward): initial development is amortized, operating costs dominate and remain low. This is also when FinOps optimization questions become relevant: can certain query types be routed to a cheaper model? Should prompt caching be activated? Are some API calls batchable to reduce costs by 40 to 50%?
The cost that organizations almost universally underestimate at the start: integration into the existing IT stack. On the projects we take over, this line item averages 35 to 45% of total development budget. The SharePoint connector, enterprise SSO, document-level permission enforcement at the source: these are real engineering workstreams that take time, and they must be in the initial estimate, not surface as surprises halfway through.
The second insight this kind of breakdown consistently produces: the ROI of an AI agent is not measured in API bill savings. It is measured in human time recovered, better-informed decisions, and accelerated processes. That requires defining, before work starts, the business value metrics that will be measured in production. Without that anchor, the project remains a cost center with no visible return. With it, two-year payback is almost always demonstrable.
Want to work through your numbers together? Book a slot and we will spend 30 minutes breaking down the real costs of your AI agent project and checking whether your budget is aligned with what the market actually looks like.