Skip to content
← Back to all sessions

Beyond MCP: The Missing Layers of Production-Ready Agentic AI

2027 Edition AI Security, Infrastructure & DevOps Talk 45 mins

Model Context Protocol (MCP) and modern LLM frameworks make it straightforward to run an agent prototype on a developer’s laptop. Keeping that agent operational, secure, and financially viable in enterprise production—facing concurrent users, non-deterministic inputs, and real-world side effects—is a fundamentally harder engineering problem.

AI agents behave like distributed systems with non-deterministic components. They suffer from failure modes standard unit tests cannot detect: silent prompt regressions, tool execution loops, unpredictable latency spikes, prompt injection attacks, and idempotent state corruption.

This session maps the critical architecture layers required around the model and its tool runtime: real-time telemetry, semantic prompt versioning, automated evaluation harnesses, defensive guardrails, and enterprise LLM gateways. Using a production assistant that performs real side-effecting operations as a running example, we break down what fails when each layer is missing and compare modern open-source and ecosystem solutions (including Phoenix, LiteLLM, NeMo Guardrails, and LangGraph).

Key Takeaways

  • The Agentic Production Stack: Understand the essential infrastructure layers required between raw LLM APIs and client applications.
  • Observability & Telemetry: Instrument tool calls, prompt tokens, latency drift, and non-deterministic agent trajectories.
  • Defensive Guardrails & Gateways: Enforce rate-limits, cost ceilings, PII redaction, and prompt injection defenses before tool invocation.
  • Evaluation & Regression Testing: Build automated evaluation harnesses that continuously test agent decisions against historical baselines.

Target Audience

  • Tech leads, system architects, and platform engineers tasked with shipping LLM agents into production environments.

Prerequisites

  • Experience with web APIs, distributed systems, and basic familiarity with LLM tool invocation patterns.

Presenting Speakers

Vivek Dhayalan
Vivek Dhayalan
Engineering Leader & Founder · Distributed Systems & Applied AI

Vivek Dhayalan is an engineering leader with over 18 years of experience building and running mission-critical …

// PRESENTED WITH COMMUNITY