Product Analytics & AI Evaluation

The problem

Two questions go unanswered in most companies. What are people actually doing in our product, and does our new AI feature actually work? Teams guess at the first, because tracking was added late and inconsistently. They guess at the second, because AI output reads convincingly whether it is right or wrong. Both guesses get expensive: you invest in features nobody uses, and you learn about an AI defect from a customer.

What we build

Measurement you can act on. On the product side, a tracking plan with named events, funnels for the journeys that matter, and session insight for the moments where people get stuck. On the AI side, an evaluation harness: a corpus of real cases with expected answers, scored automatically on every change, so quality becomes a number in your pipeline instead of an opinion in a meeting.

What’s included

• An event taxonomy your team agrees on, implemented consistently across the product
• Funnels, retention views, and dashboards built around decisions you actually make
• Session insight and heatmaps for the screens where behavior is unclear
• An AI evaluation corpus drawn from real usage, with scoring and agreed thresholds
• Regression gates, so a prompt or model change that lowers quality is caught before release
• Cost and latency tracking for AI features, reported next to quality

Technologies

PostHog and GA4 for product analytics. Custom evaluation harnesses for AI features, run locally and in your build pipeline, with results stored so quality is visible over time.

FAQ

We already have analytics. Why redo it? Usually the events exist but nobody trusts them. Fixing the taxonomy once is cheaper than arguing about dashboards for another year.

Can you evaluate an AI feature we built ourselves? Yes. Evaluation is independent of the implementation and works against any model or vendor.

How many test cases do we need? Fewer than people expect. A few dozen real, well chosen cases catch most regressions, and the corpus grows as usage teaches you what to add.