digest
Short Summary
Main recommendation: Do not trust AI-generated answers on the basis of model capability alone; build a tri-layer evaluation architecture (LLM as a judge filter, calibrated human validation, and distal telemetry) around the AI to hold its outputs accountable to real-world business context. The speaker argues that a model's "knowledge" (passing exams, writing code) does not equal reliable "life" performance, and that evaluation is the bridge between the two. The talk draws on his work on automated context sensing and the "semantic gap" in AI-generated business queries.
Key Points
- Semantic gap: AI can write syntactically correct queries but lacks your business definitions, such as what counts as an "active user" or whether international refunds are included, so it can return a wrong number that looks correct.
- Why LLM as a judge is not enough alone: Using one AI to grade another does not solve the trust problem; models favor their own style and miss their own hallucinations, so the judge only serves as a fast first filter.
- Probabilistic outputs break A/B testing: Generative AI is non-deterministic (same question, different answers), so traditional deterministic testing methods no longer measure truth reliably.
- Automated context sensing: Instead of waiting for humans to update the AI's business context, monitor metric code, documentation, data structures, and discussion threads, and treat changes there as signals.
- Human correction as training data: Engineers fix AI queries in their own workspace, and the system captures the "delta" between the AI's query and the human's working query, feeding it back as a labeled training example.
- Measuring real-world impact: Distal telemetry asks whether an output actually moved a decision forward, using behavioral signals and causal inference comparisons of decision speed with and without AI assistance.
Details Worth Noting
- Concrete instructions: The speaker describes a build sequence rather than consumer steps: (1) Layer one: have a secondary model check each generated query against the business context layer, assign a confidence score, pass high scores, block low scores, and escalate the ambiguous middle. (2) Layer two: escalate uncertain queries to a human data scientist, let them fix the query in their own workspace, capture the final working code on execution, compute the delta from the AI's query, and feed that delta back as a labeled training example. (3) Layer three: track follow-up queries, abandoned threads, whether a dashboard reached the decision meeting, and whether a metric led to a downstream decision; apply causal inference, comparing baseline human decision speed against AI-assisted decision speed, to isolate the AI's effect. (4) Keep the context layer updated automatically by monitoring metric codebases, documentation, data structures, and discussion threads, rather than relying on manual human updates.
- The speaker cites a Meta internal data science blog reporting that 88% of a data scientist's queries touch tables they had already queried in the last 90 days, and describes their layered context system called "cookbook, recipes, and ingredients."
- The speaker frames the core problem as "knowledge versus life": raw model knowledge does not automatically handle real-world friction such as edge cases, billing errors, or diagnoses.
- Caveat: The speaker notes that layer three is the hardest to build and is where he is currently focused; layers one and two are already running at many companies.
- The speaker's examples (revenue query, customer billing, clinician diagnosis) are illustrative; the talk does not provide specific quantitative results for the tri-layer system itself.
Bottom Line
Reliable AI in business depends less on how smart the model is and more on whether its outputs are continuously checked against real business context and real decision outcomes.
