Tools for monitoring LLM applications in production, managing and versioning prompts, and evaluating model outputs. Includes tracing, logging, cost tracking, prompt engineering platforms, automated evaluation frameworks, and human annotation workflows.
38 tools compared · Layer 4 · Updated March 27, 2026
Ranked by community traction, recent activity, and popularity.
Respan
Nested traces for every LLM call, tool run, and agent turn
Respan is the AI router with built-in observability and automated evals, and its [AI observability](https://www.respan.ai/ai-observability) turns every LLM call, tool run, retrieval, and agent turn into a span in one nested trace, each with its own input, output, latency, and cost. Threads group multi-turn sessions, so a bad output on turn five traces back to the context set on turn two.
LangSmith
Native tracing for LangChain and LangGraph
LangSmith is LangChain's commercial platform for building and running AI agents. It covers observability, evaluation, prompt management, and deployment, and its tracing works natively with LangChain and LangGraph and with other frameworks through SDKs for Python, TypeScript, Go, and Java plus OpenTelemetry. Teams already on LangChain get trace structure for every chain, tool, and agent step without adding instrumentation.
Arize AI
ML observability with LLM support
Traceloop
OpenTelemetry
Datadog LLM
LLM monitoring within Datadog platform
Helicone
HoneyHive
Prompt management
Braintrust
Experiments comparing prompts and models
Phoenix
OpenTelemetry-based LLM and agent tracing
Patronus AI
Automated LLM evaluation platform
Promptfoo
Portkey
Humanloop
Ragas
RAG-specific evaluation framework
Sentry
DeepEval
LangWatch
Multi-turn agent simulation testing
Galileo AI
LLM output quality evaluation
PromptLayer
Confident AI
DeepEval open-source evaluation framework
Maxim AI
Distributed tracing for LLM and agent apps
Opik
Agenta
Future AGI
Multimodal evaluation (text, image, audio, video)
Lunary
Parea AI
Ashr
Multi-modal synthetic testing
Sentrial
Agent failure root cause analysis
Chamber
ML infrastructure automation
Moda
Hallucination detection
Athina AI
Laminar
Agent tracing with a readable transcript view
OpenObserve
Logs, metrics, traces, RUM, and session replay
AgentOps
Session replay with time-travel debugging
Fiddler AI
Agentic observability across multi-step agent workflows
Respan lets you trace LLM and agent calls across any model or framework, A/B test prompts on production traffic, and route requests across 1,000+ models through one gateway.
Try Respan free