LLM Gateway for AI Agent & Model Governance | LangSmith
LangSmith LLM Gateway
Control every model call
LLM Gateway is the governance layer between your agents and model providers. Enforce spend limits, rate limits, model fallbacks, and sensitive data protection across any model or provider.
LangSmith helps top teams ship great agents
Three ways to build, from full harness to full control
Each package solves a different problem. Choose where you want to start, then compose the rest of the stack around it.
Control Spend
Keep model costs within budget
Agent behavior can be hard to predict, and small changes in traffic or tool use can quickly increase model spend. LLM Gateway helps teams monitor usage, set limits, and catch spikes before they become larger problems.
LLM Gateway can:
- Set spending caps for individual or grouped organizations, workspaces, users, and API keys
- Monitor spend at each of these levels in real time
- Record policy violations as traces within LangSmith for later review and resolution
Ensure Reliability
Keep agents running when model providers fail
Model outages, rate limits, and spend limits shouldn't take your agents down with them. With LLM Gateway, teams can configure fallback models and automatically reroute requests when the primary provider fails or a policy blocks a call.
LLM Gateway can:
- Set up fallbacks across models, providers, and hosts for high availability
- Trigger fallbacks on configurable criteria like rate limits or spend control violations
- Set rate limits at user, workspace, and API key levels
Protect Data
Keep sensitive data from reaching model providers
Agents often handle PII, credentials, and other sensitive data that shouldn’t leave your environment. LLM Gateway scans model requests and redacts sensitive data before it reaches the provider, so you can meet data handling requirements without disrupting agent behavior.
LLM Gateway can:
- Detect and redact PII and secrets like tokens, API keys, and social security numbers from model calls
- Restore original values in the model provider’s response, so redaction doesn't break agent functionality
Provider Flexibility
Enforce controls across models
LLM Gateway provides a single interface for routing calls between your agents and model providers, so your controls apply consistently on every call. Choose the best model for each use case while keeping the same governance layer in place.
LLM Gateway can:
- Support a range of models with BYO API key or using Gateway Credits for instant access to models powered by Fireworks
- Route calls to commercial providers or custom models using OpenAI- and Anthropic-compatible endpoints
- Extend governance to coding workflows in Claude Code, Codex, Gemini, and Deep Agents Code (dcode)
FAQs for LangSmith LLM Gateway
How do I get access to LLM Gateway?
LLM Gateway is currently available in Public Beta. Sign up or log in to LangSmith, where you can configure LLM Gateway by selecting it in the sidebar.
Is LLM Gateway available as a standalone product?
LLM Gateway is currently available with LangSmith Plus and Enterprise plans.
How is LLM Gateway priced?
LangSmith LLM Gateway is included with LangSmith Plus and Enterprise plans during Public Beta.
Where can I host the LLM Gateway?
For SaaS and BYOC customers, LLM Gateway is hosted in LangChain’s data centers, with regional options spanning NAMER, EMEA, and APAC.
Which model providers does LLM Gateway support?
LLM Gateway supports Anthropic, AWS Bedrock, Baseten, Fireworks, Google Gemini, Google Vertex AI, and OpenAI out of the box. Gateway also supports any OpenAI- and Anthropic-compatible endpoints.
Does LangChain train on data sent through LLM Gateway?
We will not train on your data, and you own all rights to your data. See LangSmith’s Terms of Service for more information.