LangSmith Tuned Evaluators — Custom Evaluator Request

Tuned Evaluators are LangChain-managed judges that attach feedback to production traces. They help teams find agent conversations that need attention, understand what went wrong, and turn those traces into useful examples for improving their agents.

We’re starting with Perceived Error, which detects conversations where an agent appears to have made a mistake, misunderstood a request, or failed to resolve the user’s issue.

Tuned Evaluators can:

• Evaluate eligible production traces with LangChain-managed judges
• Attach scores and explanations directly in LangSmith
• Surface repeated failure modes that may not produce system errors or user ratings
• Help teams build datasets, route traces for review, and validate agent changes

If there is a specific issue category you want LangSmith to evaluate, fill out this form to request a custom Tuned Evaluator.

Trusted by the best teams building agents