How it works
The plain version first, then the model card: what the model is, what it learned, how well it works, and where it can be wrong. Everything is trained and measured when the server starts (seed 20261102); no number on this page is typed in by hand.
In plain terms
When a shopper answers the exit survey, Qualm sees both their clicks and their reason. From 222 such answers it learns which clicks go with which worry: re-opening a size guide points to fit, checking a ZIP code points to delivery.
It applies that to the 3,778 shoppers who didn't answer. It also learns from their clicks without any labels, by repeatedly refining its guesses, which makes it better than learning from the answers alone.
Each guess is a probability across all six worries, tuned so that “80% sure” is right about 80% of the time (within about 5 points on the simulated data). Every guess lists the clicks that drove it, and how much more common each click is among shoppers with that worry.
Model card: how Qualm infers doubt
Semi-supervised multinomial naive Bayes, trained with EM (Nigam et al., 2000). Raw storefront events are mapped to 38 evidence tokens by a prefix-consistent featurizer, so training and live inference see identical features.
Labels are the 222 exit and post-purchase survey answers in the training split. The other 3,778 sessions enter as unlabeled data, contributing fractional counts weighted by λ = 0.7.
log P(c | x, p) ∝ log πp,c + τ · Σt nt log θc,t
- Product priors πp,c: empirical-Bayes shrinkage toward the global mix (κ = 25).
- Temperature τ = 1.00, fitted on 80 held-out labels with a survey-noise term (ε = 0.25 estimated misreport rate).
- Explanations: each token's likelihood ratio θc,t / meanc′≠c θc′,t.
Data
- Sessions
- 4,000
- Survey-labeled (train / calib / test)
- 222 / 80 / 127
- Label coverage
- 10.7%
- EM iterations to converge
- 14
- Evaluated vs. synthetic truth
- 3,778
- Abandonment model rows (prefixes)
- 8,979
Does it beat the obvious alternatives?
Two test sets. Held-out survey answers are the only check available on real data: small and noisy, so read the intervals. Synthetic ground truth is large and exact, but exists only in simulation.
| Held-out survey answers (n=127) | Synthetic ground truth (n=3,778) | ||||||
|---|---|---|---|---|---|---|---|
| Approach | Accuracy (95% CI) | Macro-F1 | Log-loss* | Accuracy | Macro-F1 | Log-loss | |
Survey majority Assume every shopper has the most common survey answer: what a team reading survey results alone would conclude. | 14.2% (9%–21%) | 0.035 | 1.950 | 15.5% | 0.038 | 1.969 | |
Last-signal rule Attribute the session to the doubt of the last doubt-related interaction (e.g. last opened section). | 64.6% (56%–72%) | 0.638 | n/a | 67.1% | 0.665 | n/a | |
Supervised naive Bayes Same model family trained only on survey-labeled sessions (no unlabeled data, no EM). | 74.0% (66%–81%) | 0.727 | 0.929 | 82.3% | 0.824 | 0.570 | |
Qualm (semi-supervised EM) ours Survey labels + all unlabeled sessions via EM, product-level shrinkage priors, calibrated temperature. | 77.2% (69%–84%) | 0.769 | 0.884 | 84.5% | 0.847 | 0.538 | |
* Survey log-loss is scored under the fitted answer-noise model (ε = 0.25), the same for every model. Survey answers include mis-clicks and secondary reasons, which caps achievable survey accuracy below 100%. The rule baseline gives no probabilities, so it has no log-loss and cannot split a session across doubts.
Calibration
When Qualm says 80%, is it right ~80% of the time? Expected calibration error 0.048 (synthetic truth).
EM convergence
Weighted log-likelihood of labeled + unlabeled data per iteration. It rises quickly, then flattens once the estimates stop changing.
Temperature fit
Held-out negative log-likelihood vs. τ. Minimum at τ = 1.00.
Confusion matrix
Rows: true primary doubt (synthetic). Columns: Qualm's top doubt. Row-normalised.
| Fit | Suitability | Durability | Value | Delivery | Choice | No | |
|---|---|---|---|---|---|---|---|
| Fit | 83 | 2 | 1 | 3 | 1 | 3 | 6 |
| Suitability | 1 | 86 | 1 | 2 | 1 | 3 | 7 |
| Durability | 1 | 2 | 88 | 1 | 2 | 6 | |
| Value | 94 | 1 | 3 | 2 | |||
| Delivery | 2 | 1 | 2 | 2 | 86 | 2 | 5 |
| Choice | 3 | 4 | 1 | 88 | 4 | ||
| No clear doubt | 8 | 5 | 3 | 3 | 5 | 6 | 69 |
What the model learned
Top evidence per doubt, by likelihood ratio vs. the other classes. Learned from data; nothing hand-assigned.
- Changed size selection106.9×
- Re-opened the size guide90.5×
- Filtered reviews by fit24.4×
- Selected a size12.4×
- Filtered reviews by performance73.8×
- Searched for a use-case or spec17.5×
- Expanded technical specs6.9×
- Opened reviews1.7×
- Filtered reviews by durability95.7×
- Searched for durability / warranty38.6×
- Expanded warranty15.6×
- Expanded materials6.6×
- Removed from cart46.2×
- Clicked into the promo-code field40.8×
- Searched for deals / price30.7×
- Filtered reviews by value18.9×
- Filtered reviews by shipping42.3×
- Searched for shipping / delivery38.1×
- Checked delivery date by ZIP21.1×
- Expanded shipping info13.0×
- Opened the comparison57.4×
- Searched to compare options39.5×
- Returned to a product viewed earlier22.8×
- Added a product to compare11.5×
Abandonment model
L2-regularised logistic regression on token counts, trained on random session prefixes (early-warning setting).
- ROC AUC at 50% of session
- 0.661
- mid-session: modest, honest signal
- ROC AUC at session end
- 0.991
- inflated: includes checkout steps
Held-out sessions: 1,007. Used for the Lens's purchase-likelihood readout only; resolutions trigger on doubt, not on this score.
Assumptions & limitations
- All sessions are synthetic. The generator's structure (latent doubt → behavior) matches the model family, which flatters accuracy. Real behavior will be messier.
- Naive Bayes assumes evidence tokens are conditionally independent given the doubt. The fitted temperature partially corrects the resulting over-confidence.
- Survey answers are treated as noisy labels. Selection into answering is assumed to depend on device, not on the doubt itself.
- Inference shows association between behavior and stated doubt. It is not a causal claim that answering the doubt will convert the shopper; that needs a live A/B test.
- One primary doubt per session is modeled; real shoppers can hold several. Secondary doubts appear as split posterior mass.