Costly and complex?
Agentic AI has dominated tech conversations over the past few years, but its popularity truly spiked with an emerging consensus: frontier models are capable enough to complete large, complex and even multi-step pieces of work; especially in the software development space.
As the overall performance of Large Language Models (LLMs) has started to plateau, focus has shifted towards broad adoption, and the ecosystem required to support generative AI solutions. The opportunity to leverage agentic AI workflows (which are autonomous, to varying degrees) extends far beyond simple productivity enhancements for employees. Organisations are already embedding generative AI into products and internal platforms, powering everything from multi-modal client interfaces to intelligent workflows, and automated system maintenance through operational agents.
But is it really as simple as setting up your own agentic systems? Many companies are finding that relying solely on massive, general-purpose LLMs is proving slow, expensive and risky for their enterprise grade workflows. The true frontier of agentic is proving to be less about size and more about strategic deployment: deploying fine-tuned Small Language Models (SLMs) and calling them through tailored agentic harnesses.
The question now for most companies is how agentic AI can be adopted in a scalable, secure, compliant, and cost-effective manner. This is a non-trivial problem and one that industry is actively tackling across several clients and sectors.
The 3 barriers to enterprise adoption
There are a variety of barriers which are present at any scale, and some that become particularly notable in an enterprise context.
- Beyond security vulnerabilities such as prompt injection, tool-call poisoning and other similar attacks, organisations need to be mindful of their identity access and management, supply chain security, and just-in-time access grants, ensuring agents have access to only the systems and context they need over a limited time frame, and nothing else.
- Compliance is another consideration, particularly in highly legislated areas like the UK and Europe. Correctly navigating emerging policy changes (such as the EU AI Act and the UK AI Governance Framework) can mean shifting goals mid-development.
- One of the most important factors to consider when deploying agentic solutions at scale is cost. The economics of any agentic solution are often opaque, making it hard to justify implementation and difficult to prove any benefit. Cost concerns are becoming more relevant as foundational model providers appear to be reducing token subsidisation, leading to large jumps in AI cost for the same usage.
A platform engineering approach
The good news is that, for the most part, there are established principles and practices which may be applied to help deploy agentic AI solutions while keeping security, compliance and cost concerns in check.
These are often broadly categorised under the discipline of platform engineering – providing capabilities as a platform to a broader organisation. Here are some key considerations:
- From the start, the platform should have policy guardrails in place. These guardrails should be applied automatically for any consumers (whether internal or external) and should be easily configurable to support changes in legislation.
- To avoid compliance issues, it is often necessary for the platform to support hosting of LLMs and the agentic workloads which use them within specific geographical regions. This is possible through the offerings of large cloud services such as AWS and Azure.
- For agents to be able to perform any meaningful work, they generally need access to an organisation’s data lake and credentials (often stored in a secure vault). The platform should facilitate this while enforcing security through role-based or attribute-based control with identity tied to the agent itself instead of a human.
- To centralise model access control and provide some enhanced technical functionality (such as an LLM API translation layer, model load balancing, caching, etc.), it is useful for the platform to include some kind of AI gateway. In this case, all AI traffic (LLM, Model Context Protocol, Agent to Agent, etc.) flows through the gateway. An AI gateway also enables a hard budget cap per user (whether human or agent), avoiding runaway costs and allowing for more predictable budgeting.
- The platform should have comprehensive monitoring and observability throughout for audit, cost, and troubleshooting purposes.
SLMs and agentic harnesses
To address many of the above issues, we at BBD are increasingly seeing the potential for the adoption of Small Language Models (SLMs) and agentic harnesses as an alternative to the common brute-force approach of relying on frontier models.
Unlike an LLM, an SLM is a compact, efficient model trained on a narrower body of knowledge. In some cases, an SLM which has been fine-tuned on a specific topic (a so-called ‘expert SLM’) can achieve results comparable to those of a frontier LLM for that specific task or context. Once fine-tuned, SLMs open up more flexible compliance options and cost a fraction of the price to host. This can stretch AI budgets further by reducing the cost per token (for SaaS SLM solutions) or infrastructure costs (in the case of self-hosted SLMs).
Over and above this, performance (of both LLMs and SLMs) may be improved through the use of tailored agentic harnesses. The tailoring process may include carefully crafting lean system prompts, granting access to an enhanced but limited set of tools, configuring targeted event hooks, and focusing on a comprehensive context and memory management strategy. These changes to the execution context can increase the reliability of responses and assist with finer-grained control of model usage.
Driving success at scale
The maturity of an organisation’s platform is directly correlated with how quickly and successfully they can deploy agentic AI solutions within the constraints mentioned above.
Aside from ‘dog fooding’ (using) our own solutions internally, BBD is successfully assisting a large European insurance firm with building and scaling out an enterprise agentic AI platform that is rooted in the platform engineering approach above. If you’re looking for help with setting up a secure, cost-controlled agentic platform, reach out to the team making it happen at BBD.
