Monitor and Debug LLM Applications
Langfuse LLM observability expert: tracing, prompt versioning, LLM-as-judge evaluation, datasets, and OpenAI/LangChain integration.
Why it matters
Become an expert in LLM observability with Langfuse. Debug, monitor, and enhance your LLM applications by leveraging tracing, prompt management, and evaluation strategies.
Outcomes
What it gets done
Implement LLM tracing and observability.
Manage and version prompts effectively.
Conduct systematic evaluation and scoring of LLM outputs.
Track costs and monitor performance of LLM applications.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/ag-langfuse | bash Overview
Langfuse
Langfuse LLM observability expertise covering tracing (traces, spans, generations), automatic OpenAI/LangChain instrumentation, prompt versioning across environments, manual and LLM-as-judge evaluation scoring, dataset-based testing, and the @observe decorator pattern. Use when adding LLM observability, tracing, prompt management, or evaluation to an LLM application; use companion skills like langgraph or structured-output to build the agent/pipeline being monitored.
What it does
Acts as an LLM Observability Architect expert in Langfuse, the open-source LLM observability platform - covering tracing, prompt management and versioning, evaluation and scoring, dataset management, and integration with LangChain, LlamaIndex, and OpenAI. It thinks in terms of traces, spans, and metrics, treating cost, quality, and latency as first-class monitoring dimensions for LLM applications, and uses data to drive prompt improvements and catch regressions.
When to use - and when NOT to
Use when the user mentions or implies Langfuse, LLM observability, LLM tracing, prompt management, LLM evaluation, monitoring an LLM, or debugging an LLM. Prerequisites: Python or TypeScript/JavaScript, a Langfuse account (cloud or self-hosted), and LLM API keys. Related skills handle adjacent build work: langgraph or crewai for building the agent to monitor, structured-output for building the extraction pipeline to monitor - Langfuse itself is the monitoring layer, not the agent framework.
Inputs and outputs
Basic tracing: initialize a Langfuse client, create a trace grouped by session_id/user_id with metadata and tags, log an LLM call as a generation with model/parameters/input, complete it with output and token usage, and score the trace - always calling langfuse.flush() before exit in serverless environments.
OpenAI integration is a drop-in client replacement (from langfuse.openai import openai) that automatically traces calls, including streaming and async variants, with Langfuse-specific parameters (name, session_id, tags) passed alongside normal OpenAI arguments.
LangChain integration uses a CallbackHandler passed via config={"callbacks": [langfuse_handler]} on any chain, agent, or retriever invocation, or set as the default handler so all subsequent calls trace automatically.
Prompt management fetches versioned prompts (langfuse.get_prompt("customer-support-v2")), compiles them with variables, links generations to the specific prompt version used, and creates/updates prompts with labels like "production" or "staging" for environment-specific deployment.
Evaluation and scoring covers manual in-code scoring (0-1 scale or boolean), LLM-as-judge evaluation (using a cheaper model like gpt-4o-mini to rate response quality against a rubric prompt), and full dataset-based evaluation - creating a dataset, adding input/expected-output items, running generation against each item, scoring similarity, and linking the resulting trace back to the dataset item.
The decorator pattern (@observe()) offers clean function-based instrumentation - nested @observe() calls automatically become spans under the parent trace, @observe(as_type="generation") marks an LLM call span specifically, and langfuse_context.update_current_trace()/score_current_trace() attach metadata and scores from within a decorated function, including async handlers.
Integrations
Integrates with LangChain, LlamaIndex, the OpenAI SDK, Anthropic SDK, and Vercel AI SDK, running on any Python/JS backend, serverless functions, or Jupyter notebooks, against either Langfuse Cloud or a self-hosted instance. Documented multi-skill workflows pair it with langgraph for observable agent monitoring and structured-output for monitored RAG pipelines.
Who it's for
Developers running LLM applications in production who need trace-level visibility into cost, quality, and latency, systematic prompt versioning across environments, and data-driven evaluation - via manual scoring, LLM-as-judge, or dataset-based testing - to catch regressions before they ship.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.