# AI tutoring: the measurement problem nobody wants to solve

- Published: 2026-07-01
- Authors: CORTEXA
- Category: Analysis
- HTML: https://researchhub-vert.vercel.app/blog/bytedance-gauth-ai-tutoring

The question is not whether students use these tools. It is whether they know more afterwards — and engagement metrics cannot tell you.

A discussion in r/MachineLearning about ByteDance's push into AI education with Gauth frames it as a fair question: helpful tutoring, or another engagement product.

The framing is right, and the reason it is hard to settle is that the industry has systematically avoided the measurement that would settle it.

## Two things that look identical in the metrics

**A genuine tutor** produces a student who can solve the next problem without help. Success looks like *decreasing* reliance over time.

**An answer engine** produces a student who gets the homework done. Success looks like *increasing* usage.

From a product dashboard, the second is indistinguishable from a healthy growing product. From an educational standpoint it is the failure case. And every standard engagement metric — sessions, retention, time in app — rewards the second.

That is not a conspiracy. It is what happens when you optimise the metrics you have.

```mermaid
flowchart LR
  S[Student stuck] --> T[AI help]
  T --> A[Homework completed]
  A --> M1[Engagement up ✓]
  A --> Q{Can they solve<br/>the next one alone?}
  Q -->|yes| L[Learning ✓]
  Q -->|no| D[Dependency ✗]
  L -.same dashboard.- D
```

## The study that would answer it

This is not an unanswerable question. Education research has established methods:

- Randomise access at the student or classroom level.
- Measure on a **delayed, unassisted assessment** — weeks later, no tool available.
- Compare against both no-help and human-tutoring controls.
- Report by prior attainment, since these tools plausibly help the middle and harm the tail.

The reason this is rare is not cost. It is that the result might be unfavourable, and a null or negative finding on your own product is expensive to own.

## What is actually known

The honest summary of the current evidence:

- **Short-term task completion** improves. This is well-established and unsurprising.
- **Immediate post-test performance** often improves modestly.
- **Delayed retention** is where results get mixed, and where the studies are thinnest.
- **Effects by prior attainment** are barely studied at all, and are the most policy-relevant question.

Anyone claiming confident knowledge in either direction is going beyond the evidence.

## The design question that matters

There are product choices that push toward tutoring rather than answering — withholding the final step, requiring an attempt first, asking the student to explain, spacing repetition of missed concepts. All of these *reduce* short-term engagement.

Whether a company adopts them is the most informative signal available about which product they are actually building. It is visible from the outside, and it is a better test than anything in the marketing.

## Why the scale makes it urgent

An education product reaching tens of millions of students is running an uncontrolled experiment on a cohort, and the outcome will not be visible for years. That is an argument for independent evaluation now, not for waiting to see.