The same customers, measured two ways

The worklist

A retention model is validated, and it reports that 31 of the 100 highest-risk accounts will churn in the next six months. The team staffs against that number.

Rerun on data that only knows what was knowable on the day, the same model finds 9.

Both figures come from the same model specification, the same accounts, the same churn events and the same train and test split. The team was not staffed for a list that is three and a half times weaker than the one they were shown.

The retention curve

The same distortion turns up before any model is built, in the number a board sees.

Rebuild a three year cohort from the customer table as it stands today and the accounts that were purged under an ordinary data-retention policy are simply absent from the denominator. They were customers. They left. The table no longer holds them, so the cohort appears to have been made of survivors from the beginning.

Look at the first two years. The reconstructed curve shows a company losing customers steadily from month one. The curve built from today's table shows a company that lost nobody at all until month 23. Not a slower decline. A flat line at 100 percent.

That flat segment is not a data error and nobody falsified anything. It is what happens when the only accounts you can still see are the ones that stayed.

Model quality, and how much of it was real

Two standard measures, same rows, same labels, same split. The only difference is where the five account fields were read from.

An AUC of 0.92 is the kind of figure that ends the discussion in a model review. It is also, here, mostly an artifact. The honest figure of 0.69 is a real model doing real work, and it is a different conversation about what the team should expect.

What was actually different

Five account fields. None of them is a bug, and none of the writes below is unusual. Every one of them puts information from after the prediction date into a field that carries no history.

Any single one of these is recoverable. Together they are why a retention model that reviewed well does not reproduce in production, and why a retention figure in a diligence pack is not the figure the business actually ran.

The spread, across 25 independent runs

One dataset could be a lucky draw. The entire world generation and both analyses were rerun under 25 independent seeds. The ranges below are the observed minimum and maximum, not confidence intervals.

The ranges do not overlap on any of the four measures. The effect is a property of the mechanism, not of one draw.

One more thing the number depends on. Everything above dates a churn to the month it happened. Most CRMs date it to the month it was recognised and written down, which is later. Applying a recognition lag to both arms and re-measuring, the gap is 10.8 points at zero lag and 9.2 points at three months. It narrows, and it never closes or changes sign. So the honest way to state it is about ten points, and between nine and eleven depending on which date convention your book uses.

Reproduce it

The generative process is written before either analysis, and neither analysis can see it. Roughly 200 lines, numpy and scikit-learn, about seven seconds for all 25 runs.

def features_as_of(w, acc, mon):
    """Point-in-time fields: the value each field held on the prediction date."""
    return np.column_stack(_history_features(w, acc, mon) + [
        w["usage"][acc, mon], w["tickets"][acc, mon], w["seats"][acc, mon],
        np.ones(len(acc)), w["plan"][acc].astype(float),
    ])


def features_from_today(w, crm, acc, mon):
    """The same five fields, read from the current export instead.
    Identical rows, identical labels, identical trend features."""
    return np.column_stack(_history_features(w, acc, mon) + [
        crm["usage"][acc], crm["tickets"][acc], crm["seats"][acc],
        crm["csm"][acc], crm["plan"][acc],
    ])

# Ordinary operational hygiene. None of it malicious. All of it writes the
# future into fields that carry no history.
cur["seats"] = np.where(churned & (RNG.random(n) < 0.72), 0.0, cur["seats"])
cur["csm"]   = np.where(churned & (RNG.random(n) < 0.66), 0.0, cur["csm"])
cur["plan"]  = np.where(churned & (RNG.random(n) < 0.58), 0.0, cur["plan"])

months_since = np.where(churned, (N_MONTHS - 1) - cm, -1)
cur["purged"] = churned & (months_since > PURGE_AFTER_MONTHS)

Every figure on this page is synthetic. No customer data of any kind was used. The point of generating the book rather than using a real one is that the truth is known in advance, so the gap between the two analyses is measured rather than estimated. Run it against your own export and the gap will be different. It will not be zero.

Why this is the whole problem

Every retention figure in a board pack, a diligence file or a model review was produced by someone who already knew which customers churned. That is not an accusation. It is the default condition of any analysis run against a live system, because live systems overwrite state and delete rows, and doing both is correct behaviour.

Indexxero reconstructs what was knowable on each historical date and reruns the analysis against that. Where the data cannot support the reconstruction, it reports that instead of producing a number. Two of the eight sections in a typical report come back empty, with an explanation of what would fill them.