Trustworthy AI analytics is a context problem, not a model problem

[[SignalHero]]

Anthropic’s data science team recently published its internal playbook for self-service analytics: how they let AI agents answer business questions straight from the data warehouse. The headline result is that 95% of their internal analytics questions are now answered by an agent, at roughly 95% accuracy. Impressive. But the numbers are not the interesting part.

The interesting part is the diagnosis. The thing that made their system trustworthy was not a better model. It was everything they built around the model. We reached the same engineering conclusion when we designed ActInside, and it has been the core of the product since day one. This post walks through the framework in plain language, shows how each layer is solved inside ActInside, and ends with the one place we deliberately point the same architecture in a different direction.

[[PullQuote: The bottleneck for trustworthy AI analytics is not the model. It is the context around it.]]

The problem

Data is not software

When an AI agent writes code, its mistakes surface. The code fails to compile, a test goes red, a reviewer frowns. When an agent answers a business question, there is usually one correct answer from one correct source, and nothing that automatically catches a wrong one. A wrong number looks exactly like a right number. Confident, nicely formatted, plausible.

[[CodeVsDataPipes]]

Anthropic names three reasons agents get those numbers wrong, and they match what we see in wholesale and distribution data every day:

[[FailureModes]]

Notice that none of these are model problems. A smarter model picks the wrong revenue column with more confidence. We learned this early, in production, on real distribution data. It is why every layer of ActInside’s architecture exists to kill one of these three.

The framework

Four layers, not a bigger model

Anthropic’s answer is a stack of four layers. Strip away the internal tooling and each layer is a simple idea. Below is each layer, with how it lives inside ActInside on the right. Click a failure mode to see which layers attack it.

[[LayerStackDiagram]]

[[EvalGauge]]

One number from their write-up is worth holding on to: 21% without this stack, above 95% with it, on the same model. That is the whole argument in one chart.

Engineered in

What this looks like when you use ActInside

Frameworks are abstract. Trust is concrete. We did not read this playbook and adopt it; we built it independently, layer by layer, because nothing less survives contact with real operational data. Here is the shape of it, end to end:

[[ArchFlow]]

Inside the product, that architecture surfaces as three things you can see and one thing you never will.

Every number carries its receipt. Each figure in an answer links back to the exact tables, joins and definitions it came from. You should never have to take an AI’s word for a number that steers your business.

Definitions are agreed, then enforced. During onboarding we sit with your team and pin down what “active customer,” “margin” and “fill rate” mean in your business. The agent does not improvise around those definitions. It uses them, the same way, every time. AI helps draft the documentation; a human owns the definition. It is telling that Anthropic landed on exactly this rule after first trying the opposite.

When the context is not good enough, it says so. Rather than hedge with a confidence percentage, the product refuses with a plain explanation of what is missing and what would fix it. An honest “I can’t answer this yet” builds more trust than a confident guess that turns out wrong.

And the thing you never see: the loop, the graders, the gates. They run in the backend so that what reaches you has already been challenged.

Where we point it differently

Before anyone asks

Here is the one place we diverge, and it is the reason ActInside exists. Anthropic built an exceptional system for answering questions people ask. Self-service analytics, done right. But in wholesale and distribution, the expensive miss is rarely a question someone asked and got a wrong answer to. It is the question nobody thought to ask. The product line drifting to zero inside a healthy-looking customer total. The supplier whose delays started three weeks before anyone noticed.

So we take the same architecture and run it proactively. The loop scans continuously: it detects the shift, explains the driver, lets you interrogate it, recommends the move, and routes it to the person who can act. The context layer that makes an answer trustworthy is the same thing that makes a proactive signal trustworthy enough to hand to a colleague with a deadline attached.

[[ProactiveLoop]]

[[PullQuote: Insight without action is worthless. And action without trust never happens.]]

The industry’s best data team just published the foundation. We run every layer of it in production. And we built the floor above: the one where the system does not wait for your question.