Langfuse
Open-source tracing, evals, prompt management and metrics for LLM apps and agents.
Website captured Oct 19
Evals & observability
See what went into the context and whether it worked: tracing, evals, regression tests, token and cost tracking.
19 listed entries
tool
promptfoo
Test prompts, agents and RAG from a CLI, compare models, and red-team for vulnerabilities.
Website captured Oct 19
Opik
Open-source tracing, automated evaluation and production monitoring for LLM apps, RAG and agents (Comet).
Website captured Oct 19
DeepEval
Pytest-style framework for evaluating LLM apps with ready-made metrics.
Website captured Oct 19
Arize Phoenix
AI observability and evaluation: traces, evals and datasets built on OpenTelemetry.
Website captured Oct 19
OpenLLMetry
OpenTelemetry-based instrumentation for LLM apps that sends traces to any OTel backend.
Website captured Oct 19
Helicone
Open-source LLM observability: one line of code to log, monitor and evaluate requests.
Website captured Oct 19
Laminar
Open-source observability platform built for AI agents: traces and evals for agent runs.
Website captured Oct 19
Inspect
The UK AI Security Institute's framework for large language model evaluations.
Website captured Oct 19
W&B Weave
Weights & Biases' toolkit for tracing, evaluating and iterating on AI applications.
Website captured Oct 19
Braintrust
Evals and observability platform (closed source), measured here through its official SDKs.
Website captured Oct 19
Context clash
When new information or tools in the context conflict with other information already there.
Context confusion
When superfluous content in the context is used by the model to produce a low-quality response.
Context distraction
When context grows so long that the model over-focuses on it and neglects what it learned in training.
Context poisoning
When a hallucination or other error makes it into the context and keeps getting referenced.
Context rot
The drop in a model's ability to recall and use information as the number of tokens in its context grows.
Diagnose context failures
Check long-context failures by type: poisoning, distraction, confusion and clash.
LangSmith
LangChain's observability and evaluation platform (closed source), measured here through its SDKs.
Website captured Oct 19
Test at the context lengths you run
Evaluate your agent with inputs as long as production ones; performance drops as input grows, even on simple tasks.
How they connect
- Opik alternative to Langfuse
- promptfoo alternative to DeepEval
- promptfoo alternative to Inspect
- Test at the context lengths you run implements promptfoo
- Test at the context lengths you run implements DeepEval
- Test at the context lengths you run implements Inspect
- Laminar alternative to Langfuse
- Helicone alternative to Langfuse
- OpenLLMetry alternative to Laminar
- Arize Phoenix alternative to Langfuse
- Braintrust alternative to W&B Weave
- Braintrust alternative to LangSmith
- W&B Weave alternative to LangSmith
- LangSmith alternative to Langfuse
- Diagnose context failures implements promptfoo
- Diagnose context failures implements Arize Phoenix
- Diagnose context failures implements Langfuse