Monitoring & Observability
Monitor your AI analyst's performance with comprehensive diagnostics that track agent behavior, context quality, and instruction effectiveness.
Overview
MetricChat's monitoring provides the tools to understand not just whether the system works, but why. Move between aggregate trends and detailed inspection of individual operations.
Key Metrics
Accuracy
Track the success rate of queries producing correct, acceptable results across your organization.
Instruction Effectiveness
Automatic per-prompt scoring of how well your instructions influenced the agent's output.
Query & Message Volume
Monitor adoption and usage patterns across teams and data sources.
User Feedback
Capture human-driven signals — thumbs up/down, corrections, and comments — to reinforce or correct behavior.
Drill-Down Diagnostics
For any agent run, administrators can inspect:
- Context blocks — The specific instructions, schema, and lineage deployed
- Tools triggered — Actions taken (query creation, clarification, context search)
- Reasoning steps — The agent's sequential thinking and reflection process
- Failed queries — Error details with categorization (execution failures, context gaps, data invalidity)
Lineage & Root Cause
Every query connects to table- and column-level lineage. When results are incorrect, trace the issue back to:
- Problematic schema elements
- Incorrect joins or relationships
- Missing or wrong definitions
- Data quality issues
What You Can Do With Monitoring
- Refine context — Identify which instructions need improvement or addition
- Catch regressions — Spot performance degradation early after changes
- Assign errors — Route problems to the correct source (schema, instructions, data, or model)
- Build governance — Establish organizational standards for AI data analysis
Best Practices
- Review monitoring dashboards regularly, especially after instruction or schema changes
- Investigate low-scoring instructions and update or replace them
- Use lineage tracing to find the root cause of recurring errors
- Track user feedback trends to identify areas needing improvement
- Set up alerts for accuracy drops below acceptable thresholds