The Five-Signal Model: A Neural Net Your CRO Will Actually Approve

The conversation that doesn't go well

Every CRO has had this exchange. They ask the CS lead what next quarter's renewal book looks like. The CS lead says something like "we're at 87% to plan." The CRO asks how confident. The CS lead says "high." The CRO asks what the model says.

There is no model.

This piece is about building the model. Not because models replace judgment — they emphatically do not — but because the model gives the CSM a defensible second opinion that lives outside any individual operator's confirmation bias. The CRO asks "what does the model say?" and the CSM answers: "The model says 76%. Here are the three accounts driving the gap between my 87% and the model's 76%. Here's why I'm holding at 87%."

That is a conversation.

The version without a model is a posture-off.

What we'll build:

• A neural network with five inputs, two hidden layers (eight neurons each), three outputs.
• Trained on a synthesized dataset of 312 enterprise CS accounts spanning eighteen months.
• Outputs: probability of churn, probability of expansion within twelve months, predicted days-to-escalate.
• Open weights. Open notebook. Open dataset.

The model is small. Intentionally. The smallest model that beats your senior CSM's gut is the right model. Anyone selling you a seventy-billion-parameter foundation model for renewal forecasting is selling you something you don't need.

The architecture: why 5 → 8 → 8 → 3

The five inputs

These five signals were chosen because each one captures something the others miss, and because each one is reliably available in a modern CS stack. The selection of inputs matters more than any other choice in the model — bad inputs produce bad predictions, no matter how sophisticated the model is downstream.

1. Product usage depth. Not "did they log in this week" but "what fraction of the contracted use cases are running, at what maturity?" If a customer bought your platform to manage three use cases (ticket triage, knowledge base, customer self-service), depth measures how many of the three are live and operating. A customer at full depth on one use case but absent on the other two is at ~33% depth, not 100% — even if their login activity is high, because one team is the only team using it.

Data sources: product telemetry, mapped to your contracted use cases per account. Most teams don't have this mapping. Build it. It is the foundation for everything.

2. Sponsor engagement. A composite of executive touch frequency and multithread depth — how often the senior stakeholder on the customer side engages with you, and how many roles they have you connected to inside the account. An account with deep multithreading is harder to lose than an account with a single point of contact, even if the single point of contact is enthusiastic.

Data sources: CRM activity log, calendar metadata, multithreading map (from the Multithreading Map framework).

3. Support velocity. Frequency of P1 incidents and time-to-close on customer escalations. This signal captures the operational friction of the relationship — whether the customer is encountering serious issues, and how fast you're resolving them. High P1 frequency with slow time-to-close is the leading indicator of trust erosion.

Data sources: support ticket system. Be careful about P1 classification drift — different support teams classify severity differently, and the signal becomes noisy across data sources. Normalize internally.

4. Sentiment delta. The change in customer sentiment over the trailing 90 days. Critically, this is a delta, not a level. An account at 0.7 sentiment that has always been at 0.7 is stable. An account at 0.5 sentiment that dropped from 0.9 is in trouble. Levels are misleading; deltas carry signal. (The QBR Pipeline piece walks the construction of this signal in detail.)

Data sources: NPS surveys, QBR transcripts processed through a sentiment-extraction pipeline.

5. Renewal proximity. Days until the contract term ends, combined with the auto-renew flag status. Customers behave differently as the renewal window approaches — and the auto-renew flag is a binary that materially affects renewal probability.

Data sources: CRM contract data, billing system.

What's not in the model

Worth being explicit about what we deliberately excluded:

• The CSM's own forecast score. This is the most important exclusion. If you train the model on the CSM's prior predictions, you create a circular reasoning loop — the model learns to confirm the CSM's biases. We want the model and the CSM to be independent estimators, combined at decision time, not training time.
• Customer-stated intent to renew. The single most-collected and least-predictive signal in CS. Customers who say they intend to renew often don't; customers who hedge often do. The signal is too noisy to feed into the model.
• Deal-stage information from sales. This is a post-close model. Sales-stage data conflates new business dynamics with renewal dynamics.

The three outputs

1. Churn risk. Probability the account does not renew. Sigmoid output. Binary classification problem at heart.

2. Expansion fit. Probability the account is a candidate for meaningful expansion within twelve months of renewal. Sigmoid output. Separate from churn — an account can score low on both (zombie), high on both (best-case), or split (typical).

3. Days-to-escalate. Predicted days until the account should hit a manager's desk for intervention. Regression output (ReLU). Operationalizes the timing of intervention, not just the need for one.

Three outputs instead of one composite "health score" because operators need three different decisions. A single number hides which decision is being made. The model produces three numbers that map to three different actions.

Why a neural net at all

Honest answer: for this problem, a neural net is not obviously better than a logistic regression. We use the neural net because of interaction effects — the relationship between sentiment delta and renewal proximity is non-linear in ways that linear models miss. We tested both architectures on the held-out validation set. The neural net beats logistic regression by ~8 percentage points on classification accuracy.

That difference justifies the additional complexity. But it does not justify reaching for a transformer architecture or a billion-parameter language model. Most CS prediction tasks don't need a fancy model. They need clean data, a small neural net or logistic regression, and a CSM who reads the output. If a vendor pitches you on a fancy model when they can't show you why a simpler model fails, they're selling you sophistication, not predictive power.

The training data

How we built the dataset is the part that separates this piece from vendor copy.

Where the data came from

The dataset in the notebook is 312 synthesized account records, designed to match the statistical shape of real CS account data. Each record has the five input signals (sampled from distributions that reflect what you'd see in a real portfolio), the three outcome variables (renewal status, expansion outcome, escalation timing), and interaction effects between signals (poor sentiment + close renewal proximity carries disproportionate churn risk, etc.).

The shape is calibrated to what a real CS dataset looks like for a typical mid-market SaaS company with enterprise accounts ranging from $50K to $2.5M ARR. The class imbalance — 75% renewal rate, 25% churn, ~38% expansion among renewers — matches what most healthy CS organizations see.

Why synthesized rather than real? Two reasons. First, there's no legitimate way to use a real CS dataset in a public piece without violating customer NDAs and creating immediate IP problems. Second, the synthesized dataset is more useful for the operator reading this piece — you can run the same code against your own real data and see whether your real-world results match the patterns shown here. If they diverge sharply, that itself is signal worth investigating.

The de-identification process (for your own data)

When you retrain this model on your actual data, follow this process:

• Replace customer names with hashes (consistent across rows for the same customer, but irreversible).
• Band dollar amounts into ranges ($50K-$100K, $100K-$250K, etc.) rather than exact values.
• Convert calendar dates to relative offsets (days-from-contract- start, days-to-renewal) rather than absolute dates.
• Map specific product features to generic categories.
• Map industry to broad categorical bins.

This process takes ~2 hours for a portfolio of a few hundred accounts. The model trained on de-identified data performs equivalently to the model trained on raw data. The de-identification is for your security team's peace of mind, not for the model's benefit.

Class balance

In the synthesized dataset:

• 234 accounts renewed (75%)
• 78 accounts churned (25%)
• 119 of the renewers expanded within 12 months (38% of renewers)

The 75/25 split is the realistic class balance for a healthy CS portfolio. Modeling decisions accommodate this imbalance: we use class weights (where the minority class receives higher weight during loss calculation) rather than SMOTE-style oversampling, because class weights are simpler, more interpretable, and don't fabricate synthetic minority-class examples.

Train / validation / test split

70 / 15 / 15, stratified by churn outcome. This is the conventional choice for a dataset this size. Anything fancier (k-fold cross-validation, leave-one-out) is overkill for 312 accounts and would slow down iteration.

The validation set is used for hyperparameter tuning during model development. The test set is held out until the final model is locked, and only used once to report the model's accuracy. This discipline matters — if you tune on the test set, you've leaked it into the training process and the reported accuracy is inflated.

The calibration curve

This is the section most ML writeups skip and operators most need.

The model outputs a probability. The CSM uses the probability to make a decision. The question is: when the model says "73% churn risk," does that account actually have a 73% probability of churning?

If yes, the model is calibrated. If the model says 73% but the actual churn rate at that score is 90%, the model is under-confident. If the model says 73% but the actual rate is 40%, the model is over-confident. Calibration is the test.

Here's how we measure it. Bucket the validation predictions into deciles (0-10% predicted, 10-20%, 20-30%, etc.). For each decile, plot the model's average predicted probability against the actual churn rate of accounts in that decile. Plot all ten points. Draw the y=x line. The closer your points lie to y=x, the better calibrated the model.

The notebook produces this chart. For this synthesized dataset, the model is well-calibrated in the 20-80% range and slightly over-confident at the extremes. Operationally, this means:

• Trust the model's outputs in the middle of the probability range. A score of 50% really does mean roughly even odds.
• Treat extreme outputs with skepticism. A score of 95% likely means "very high probability of churn," but probably not literally 95%. Use it as a strong signal, not a literal probability.
• Recalibrate quarterly. Calibration drifts as your customer behavior and product evolve.

Why calibration matters

Calibration is the dimension that separates a useful model from a useless one. Copilot C's failure ( from the Copilot Teardown piece) was an uncalibrated pre-trained model. The model produced confident-looking numbers — 73.412% churn risk — that bore no relationship to actual outcomes. The numbers looked authoritative. They were noise.

A calibrated model is humble about the things it's not sure about and confident about the things it is. An uncalibrated model is confident about everything. Confidence without calibration is dangerous. The calibration curve is the test that catches it.

The CRO conversation, scripted

The model exists. The calibration is good. Now the moment that justifies the entire build: the CSM says 87%, the model says 76%, the CRO is asking.

The bad version

CRO: "Why is the model saying 76% if you're saying 87%?"

CSM: "The model doesn't have all the context."

CRO: "What context?"

CSM: "I know these accounts."

That's a bad conversation. The "I know these accounts" response is exactly what the model was supposed to bypass. The CRO isn't asking for reassurance; they're asking for triangulation.

The good version

CRO: "Why is the model saying 76% if you're saying 87%?"

CSM: "The model is at 76% because three accounts are flagging high on sentiment delta — their NPS dropped from 9 to 7 over the last two quarters. I'm at 87% because I know those three accounts personally. Two of them had a specific incident that's been resolved. The third has a new sponsor we just installed. So I'm overriding the model on those three. The gap between us is exactly those three accounts."

CRO: "Show me."

CSM: opens the model output, points at the three flagged accounts, walks through the override reasoning for each.

CRO: "Got it."

The difference is structural. The model has produced something specific the CSM can override with specific reasons. The override isn't an argument against the model — the override is the output of using the model correctly. The CSM and the model are co-authors of the final forecast.

The override pattern

Three rules for overriding the model:

Override on specific accounts, not aggregate scores. "I'm overriding 87% vs. 76%" is too vague to defend. "I'm overriding three named accounts" is auditable. Always force the override down to the account level.

Document the override reason in advance, not after the renewal. If you only document override reasons after you find out you were wrong, you're rationalizing, not forecasting. The discipline is that the override goes into your tracking system the day you make it, not after the outcome.

Track your override accuracy over time. Build a table: every override, the model's prediction, the CSM's override, the actual outcome. After 50 overrides, you'll know whether you're systematically over- or under-confident. After 100, the trend is calibrated.

This is the meta-skill the model is teaching. It's not about getting the right number. It's about building the discipline of productive disagreement with a model.

Three failure modes, and how to spot them

1. Distribution shift. The model was trained on data covering a specific time window. Twelve months from now, customer behavior may have shifted — a new product launch, a market downturn, a regulatory change. The model's accuracy degrades silently. Spot it by tracking the model's predictions against actuals quarterly. If the spread is widening, retrain.

2. Whale-account miscalibration. A model trained on a portfolio where most accounts are $50K-$500K ARR will systematically misjudge accounts above $2M ARR. The whales aren't in the training distribution; the model's predictions for whales are priors, not predictions. Treat the model's output for the top decile of ARR as one input among many, not as the authoritative signal.

3. Override confirmation bias. CSMs systematically override the model in the direction of their pre-existing beliefs. Optimistic CSMs override upward; pessimistic CSMs override downward. The override-tracking table catches this — if your overrides are always in one direction, you've found the bias. The fix is awareness; the bias is fixable once you measure it.

Reproduce the whole thing

The companion notebook reproduces this entire piece. It runs on a laptop in fifteen minutes.

Files in the repo:

• five_signal_model.ipynb — the notebook, with code and explanation cells
• data/csm_accounts.csv — the 312-row synthesized dataset
• requirements.txt — Python dependencies (scikit-learn, pandas, matplotlib)
• README.md — quickstart instructions

What the notebook walks through, in order:

1. Loading and exploring the dataset
2. Preprocessing — feature engineering, normalization
3. Train/validation/test split
4. Defining the network architecture in scikit-learn's MLPClassifier
5. Training with class weights to handle the imbalance
6. Producing the calibration curve
7. Predictions on three sample held-out accounts. The model running on this synthesized dataset produces predictions in the ranges of 23-64% churn risk, 23-64% expansion fit, and 8-95 days to escalate. The model is appropriately conservative on a 312-account dataset — it doesn't produce extreme predictions because it doesn't have enough data to be confident at the extremes. A model trained on a real-world dataset of several thousand accounts produces sharper predictions, in the ranges shown on the neural-network diagram at the top of the AI pillar. The lesson there is real: model confidence is a function of dataset size, and the small-dataset version of any model will hedge toward the base rate. Bring your own data and you'll see the prediction range widen.

The notebook uses scikit-learn rather than PyTorch deliberately. Scikit-learn is more accessible for operators who are not full-time ML practitioners. Production-grade implementations of this kind of model would typically use PyTorch, but scikit-learn's MLPClassifier is a faithful implementation of a small neural network and produces equivalent results for a model this size.

The repo is on GitHub. Fork it. Run it on your laptop. Then retrain it on your own data. The notebook has comments showing exactly which files to swap and which lines to change.

What this model is and isn't

What the model is:

• A second opinion that lives outside the CSM's confirmation bias.
• A defensible artifact for the CRO conversation.
• A discipline-builder for the override process.

What the model isn't:

• A replacement for the CSM's judgment. The CSM owns the forecast. The model is an input.
• A predictor for whale accounts outside the training distribution.
• A static thing. Retrain quarterly or it decays.

The model isn't the product. The discipline of training, calibrating, and overriding your own model is the product. Vendors will sell you their model. The model in this piece is the receipt that you can build your own. Once you can, the vendor pitches look very different.

Loop back to the AI tab

The neural network diagram at the top of the AI pillar on this site is not decoration. The 5 → 8 → 8 → 3 architecture is the architecture above. The output numbers on the diagram (23% churn risk, 71% expansion fit, 9d to escalate) illustrate what a well-trained version of this model produces on a real-world dataset. Our synthesized notebook produces predictions in a narrower range — that's the honest result of training on 312 accounts rather than thousands. The model in this notebook is the starting point. The model on your data is the destination.

What's next

This is the technical anchor of the AI pillar. The final piece, The QBR Transcript Pipeline, is the companion — it walks the construction of the sentiment-delta input that this model depends on. Together, the two pieces are the answer to "how do we build our own AI tools as a CS function?" The matrix gave us the operating model. The prompts gave us the day-to-day workflow. The safety rules gave us the floor. The teardown gave us the vendor skepticism. These last two pieces give us the build.

The QBR pipeline is shorter than this piece and easier to deploy. Most CS Ops leads can stand up a working version in a week. The pipeline produces one of the five inputs to this model — which means once you have the QBR pipeline running, you have the output of one of the five signals available for the next time you train this network on your own data.

The pieces compound. That is the point of the pillar.

Download the Five-Signal Model notebook — the full reproducible model, synthesized dataset, and README. Runs on your laptop in two minutes.