Business example: does the coupon pay for itself?¶
An online retailer sends a coupon to part of its customers. Does the campaign pay for itself, and for which customers?
The coupon was not sent at random: the CRM favours active app users, who buy a lot anyway, and price sensitive customers. A naive comparison, and even a regression with all customer characteristics, therefore overstates the effect and leads to the wrong decision.
The data is simulated, so we know the true effect and can check that DoubleML finds it.
To run it yourself, first install the package.
import os
import numpy as np
import pandas as pd
from pyspark.ml.classification import GBTClassifier
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import GBTRegressor, LinearRegression
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from sklearn.preprocessing import SplineTransformer
from doubleml_pyspark import plotting
from doubleml_pyspark.datasets import make_coupon_data
from doubleml_pyspark.irm import DoubleMLIRM
from doubleml_pyspark.utils import kfold_split
spark = SparkSession.getActiveSession() or SparkSession.builder.getOrCreate()
plotting.apply_style()
N_CUSTOMERS = 200_000 # millions on a Databricks cluster
SEED = 42
# Serverless has no cache(): data and predictions are stored once as Delta tables instead.
# Default: the current catalog and schema; set e.g. "main.default.coupon_customers" to choose.
DATA_TABLE = None
COUPON_COST, GROSS_MARGIN = 3.0, 0.35 # USD per coupon, profit share of extra revenue
BREAK_EVEN = COUPON_COST / GROSS_MARGIN # the coupon pays for itself above 8.57 USD
SERVERLESS = os.environ.get("DATABRICKS_RUNTIME_VERSION", "").startswith("client")
if SERVERLESS and not DATA_TABLE:
DATA_TABLE = f"{spark.catalog.currentCatalog()}.{spark.catalog.currentDatabase()}.doubleml_coupon_customers"
def materialize(df, suffix=""):
# Compute a DataFrame once: cache on a cluster, a Delta table on serverless (no cache() there)
if not SERVERLESS:
return df.cache()
if DATA_TABLE:
# repartition, not coalesce: few output files, while the scoring before it stays parallel
df.repartition(8).write.mode("overwrite").option("overwriteSchema", "true").saveAsTable(DATA_TABLE + suffix)
return spark.table(DATA_TABLE + suffix)
return df
Data¶
One row per customer: tenure, past spend, visits, share of orders with a discount (price
sensitivity), app use and region, the CRM segment, whether the customer received the coupon,
and the revenue in the following 30 days. true_effect is only known because the data is
simulated. The model never sees it.
df, covariates = make_coupon_data(spark, n_obs=N_CUSTOMERS, seed=SEED)
df = materialize(df)
df.limit(5).toPandas()
| ID | tenure_months | log_past_spend | visits_90d | discount_share | app_user | region_south | region_east | region_west | segment | coupon | basket_value_30d | true_effect | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 7 | 27.0 | 6.039443 | 3.0 | 0.756089 | 0.0 | 0.0 | 0.0 | 0.0 | bargain_hunter | 0.0 | 33.278422 | 14.015327 |
| 1 | 15 | 51.0 | 5.874954 | 6.0 | 0.195709 | 1.0 | 1.0 | 0.0 | 0.0 | regular | 0.0 | 47.823843 | 5.002206 |
| 2 | 23 | 62.0 | 5.957320 | 4.0 | 0.710911 | 0.0 | 0.0 | 0.0 | 0.0 | bargain_hunter | 0.0 | 44.456820 | 14.646008 |
| 3 | 31 | 7.0 | 4.986485 | 7.0 | 0.323852 | 0.0 | 1.0 | 0.0 | 0.0 | new | 0.0 | 31.801401 | 7.393460 |
| 4 | 39 | 53.0 | 5.529802 | 3.0 | 0.199738 | 0.0 | 0.0 | 0.0 | 1.0 | regular | 1.0 | 30.538853 | 3.249106 |
Naive comparison and regression¶
The usual answers: the difference in spending between customers with and without coupon, and a linear regression (OLS) on the coupon and all customer characteristics.
means = df.groupBy("coupon").agg(F.avg("basket_value_30d")).toPandas().set_index("coupon").iloc[:, 0]
naive = means[1.0] - means[0.0]
true_ate = df.agg(F.avg("true_effect")).first()[0]
ols_data = VectorAssembler(inputCols=["coupon"] + covariates, outputCol="features").transform(df)
ols = LinearRegression(featuresCol="features", labelCol="basket_value_30d", solver="normal").fit(ols_data)
try:
se = ols.summary.coefficientStandardErrors[0]
ols_estimate = (ols.coefficients[0], ols.coefficients[0] - 1.96 * se, ols.coefficients[0] + 1.96 * se)
print(f"OLS 95 % interval {ols_estimate[1]:.2f} to {ols_estimate[2]:.2f} USD")
except Exception: # model summaries are not available on every Spark Connect version
ols_estimate = ols.coefficients[0]
print(f"naive {naive:.2f} USD, OLS {ols.coefficients[0]:.2f} USD, true {true_ate:.2f} USD, break-even {BREAK_EVEN:.2f} USD")
OLS 95 % interval 10.17 to 10.39 USD naive 19.24 USD, OLS 10.28 USD, true 6.17 USD, break-even 8.57 USD
DoubleML¶
The interactive regression model (IRM) learns, with gradient boosted trees, who receives a
coupon (ml_m) and how much a customer would spend without and with it (ml_g0, ml_g1).
Cross-fitting keeps the estimate unbiased. The predictions are computed once and stored, so the
following steps do not score the models again.
The tree settings are fixed values chosen beforehand and two folds are used for cross-fitting, so the example runs in a few minutes. The commented block shows how Optuna would tune the trees on your own data. Tuning and more folds cost extra compute.
dml = DoubleMLIRM(data_schema=df.schema, data_field_map={
"id_column": "ID", "outcome": "basket_value_30d", "treatment": "coupon", "covariates": covariates,
})
# Pre-chosen settings: few boosting rounds and two folds keep the number of Spark jobs low
ml_models = {
"ml_m": GBTClassifier(maxIter=10, maxDepth=5, stepSize=0.3, seed=SEED),
"ml_g0": GBTRegressor(maxIter=10, maxDepth=5, stepSize=0.3, seed=SEED),
"ml_g1": GBTRegressor(maxIter=10, maxDepth=5, stepSize=0.3, seed=SEED),
}
splits = kfold_split(df, n_folds=2, n_rep=1, shuffle=True, seed=SEED)
# Optional: tune the trees with Optuna on a sample first (costs extra compute)
# space = lambda trial: {"maxDepth": trial.suggest_int("maxDepth", 3, 6),
# "maxIter": trial.suggest_int("maxIter", 10, 40),
# "stepSize": trial.suggest_float("stepSize", 0.05, 0.3, log=True)}
# tune_split = kfold_split(df.sample(fraction=min(1.0, 50_000 / N_CUSTOMERS), seed=SEED),
# n_folds=3, n_rep=1, shuffle=True, seed=SEED)
# ml_models = dml.tune_ml_models(spark_session=spark, data_split=tune_split[0][0], ml_models=ml_models,
# ml_param_space=dict.fromkeys(ml_models, space),
# optuna_settings={"n_trials": 10, "gc_after_trial": True})
df_nuisance, extra = dml.fit_ml_models(spark_session=spark, kfold_split=splits[0], ml_models=ml_models)
df_nuisance = materialize(df_nuisance, "_predictions")
dml.estimate_causal_parameters(spark_session=spark, df_nuisance=df_nuisance, additional_nuisance_elements=extra)
dml.summary
| coef | std err | t | P>|t| | 2.5 % | 97.5 % | |
|---|---|---|---|---|---|---|
| coupon | 6.186095 | 0.055375 | 111.712841 | 0.0 | 6.077562 | 6.294628 |
fig = plotting.plot_effect_comparison(
{"Naive comparison": naive, "Regression (OLS)": ols_estimate, "DoubleML": dml},
true_effect=true_ate, threshold=BREAK_EVEN, unit="USD",
threshold_text=f"break-even {BREAK_EVEN:.2f}: to the right the coupon pays for itself",
xlabel="Additional revenue per customer in 30 days (USD), with 95 % CI",
title="Effect of the coupon on spending",
)
DoubleML finds the true effect, below break-even: the campaign loses money. The naive comparison is about three times too high, and the regression lies above break-even, so it would keep a campaign that loses money. The sensitivity analysis shows how strong an unobserved confounder would have to be to change this.
dml.sensitivity_analysis(cf_y=0.03, cf_d=0.03)
print(dml.sensitivity_summary)
================== Sensitivity Analysis ==================
------------------ Scenario ------------------
Significance Level: level=0.95
Sensitivity parameters: cf_y=0.03; cf_d=0.03, rho=1.0
------------------ Bounds with CI ------------------
CI lower theta lower theta theta upper CI upper
coupon 5.280311 5.371446 6.186095 7.000745 7.092034
------------------ Robustness Values ------------------
H_0 RV (%) RVa (%)
coupon 0.0 20.609452 20.321539
Effect by customer segment (GATE)¶
New customers: first order less than 12 months ago. VIP: more than 600 USD revenue in 12 months. Bargain hunters: more than 45 % of orders with a discount. Regular: all others.
gate = dml.gate(spark_session=spark, df_groups=df.select("ID", "segment"), df_nuisance=df_nuisance)
true_gate = df.groupBy("segment").agg(F.avg("true_effect")).toPandas().set_index("segment").iloc[:, 0]
fig, gate_table = plotting.plot_gate(
gate, spark, threshold=BREAK_EVEN, true_effects=true_gate.to_dict(), unit="USD",
labels={"vip": "VIP", "regular": "Regular", "new": "New customers", "bargain_hunter": "Bargain hunters"},
decision_labels=("pays for itself", "does not pay for itself", "not clear"),
xlabel="Additional revenue per customer in 30 days (USD), joint 95 % CI",
title="Coupon effect by customer segment",
)
gate_table.round(2)
| group | lower | effect | upper | true_effect | decision | |
|---|---|---|---|---|---|---|
| 0 | vip | 1.41 | 1.79 | 2.18 | 1.68 | does not pay for itself |
| 1 | regular | 3.53 | 3.71 | 3.89 | 3.69 | does not pay for itself |
| 2 | new | 8.20 | 8.66 | 9.11 | 8.70 | not clear |
| 3 | bargain_hunter | 12.32 | 12.58 | 12.84 | 12.58 | pays for itself |
The coupon pays for itself for bargain hunters, not for regular and VIP customers, and for new customers the data cannot tell: the confidence interval contains the break-even point.
Effect along price sensitivity (CATE with splines)¶
Cubic B-splines approximate how the effect changes with the share of orders that used a discount, $\tau(x) \approx \sum_j \beta_j B_j(x)$ (Semenova and Chernozhukov, 2021).
A pointwise interval gives coverage for each estimated effect one at a time, so with many groups some intervals will miss even if each has 95% confidence.
A joint (simultaneous) interval widens the band so that, with the stated confidence, all the true effects are covered at once, it is for more honest bands around CATE curves.
sample = (df.select("ID", "discount_share", "visits_90d", "coupon", "true_effect")
.sample(fraction=min(1.0, 100_000 / N_CUSTOMERS), seed=SEED).toPandas())
spline = SplineTransformer(n_knots=5, degree=3).fit(sample[["discount_share"]])
cols = [f"spline_{j}" for j in range(spline.n_features_out_)]
basis = pd.DataFrame(spline.transform(sample[["discount_share"]]), columns=cols).assign(ID=sample["ID"].values)
cate = dml.cate(spark_session=spark, df_basis=spark.createDataFrame(basis), df_nuisance=df_nuisance)
x = np.linspace(0.02, 0.88, 120)
fig, cate_table = plotting.plot_cate(
cate, spark, pd.DataFrame(spline.transform(pd.DataFrame({"discount_share": x})), columns=cols), x,
threshold=BREAK_EVEN, threshold_text=f"break-even {BREAK_EVEN:.2f}",
region_text="pays for itself\nfrom {x:.0f} %", x_scale=100,
xlabel="Share of past orders with a discount (%)", ylabel="Additional revenue in 30 days (USD)",
title="Coupon effect by price sensitivity",
)
Effect along price sensitivity and activity (3D)¶
A tensor product of two spline bases gives a surface over price sensitivity and the number of visits. Rotate the chart and hover over it.
spline_v = SplineTransformer(n_knots=4, degree=3).fit(np.log1p(sample[["visits_90d"]]))
spline_d = SplineTransformer(n_knots=4, degree=3).fit(sample[["discount_share"]])
def surface_basis(discount, visits):
a = spline_d.transform(pd.DataFrame({"discount_share": discount}))
b = spline_v.transform(pd.DataFrame({"visits_90d": np.log1p(visits)}))
return pd.DataFrame(np.einsum("ni,nj->nij", a, b).reshape(len(a), -1)).add_prefix("surface_")
basis_2d = surface_basis(sample["discount_share"], sample["visits_90d"])
cate_2d = dml.cate(spark_session=spark, df_basis=spark.createDataFrame(basis_2d.assign(ID=sample["ID"].values)),
df_nuisance=df_nuisance)
gx, gv = np.meshgrid(np.linspace(0.02, 0.88, 30), np.linspace(0, 20, 30))
plotting.plot_cate_surface(
cate_2d, spark, surface_basis(gx.ravel(), gv.ravel()), gx, gv, threshold=BREAK_EVEN, x_scale=100,
unit="USD", xlabel="discount orders (%)", ylabel="visits in 90 days", zlabel="effect",
title="Coupon effect by price sensitivity and activity",
).show()
The effect is largest for price sensitive customers with a few visits: customers who hardly visit rarely notice the coupon, very active customers buy anyway.
Decision: whom to send the coupon to¶
Naive comparison and OLS give one average effect above break-even, so they would send the coupon to everybody. DoubleML gives an effect per customer, so the coupon can go only where it exceeds break-even. The chart evaluates each rule with the true simulated effects.
sample["estimated_effect"] = (cate_2d.confint(spark_session=spark, basis=spark.createDataFrame(basis_2d).coalesce(1))
.toPandas()["effect"].to_numpy())
rules = {
"Naive and OLS: coupon to everyone": np.ones(len(sample), dtype=bool),
"Current CRM rule": sample["coupon"].to_numpy() == 1,
"DoubleML: effect above break-even": sample["estimated_effect"].to_numpy() > BREAK_EVEN,
}
fig, profit_table = plotting.plot_targeting_curve(
sample["estimated_effect"], value=sample["true_effect"], cost=COUPON_COST, margin=GROSS_MARGIN,
rules=rules, scale=N_CUSTOMERS / len(sample), unit="USD",
xlabel="share of customers who receive the coupon (%)",
title="Campaign profit by decision rule (evaluated with the true simulated effects)",
)
fig.add_annotation(x=100 * rules["Current CRM rule"].mean(), y=0, ax=150, ay=-40, showarrow=True, arrowhead=0,
text="about the same number of coupons,<br>but to different customers")
fig.show()
profit_table.round(0)
but to different customers") fig.show() profit_table.round(0)
| rule | share (%) | treated | profit | |
|---|---|---|---|---|
| 0 | Naive and OLS: coupon to everyone | 100.0 | 200000.0 | -167949.0 |
| 1 | Current CRM rule | 29.0 | 57300.0 | -38633.0 |
| 2 | DoubleML: effect above break-even | 30.0 | 59237.0 | 81885.0 |
Summary
- On average the coupon raises spending by less than it costs: the current campaign loses money.
- Naive comparison and regression say the opposite, because the coupon was targeted and its effect differs between customers.
- The current CRM rule sends about as many coupons as the DoubleML rule, but to customers who buy anyway. Targeting by the estimated effect turns the loss into a profit with the same budget.
On your own data: replace make_coupon_data with your customer table and keep the rest.
References: Chernozhukov et al. (2018), Double/debiased machine learning for treatment and structural parameters, The Econometrics Journal 21(1). Semenova and Chernozhukov (2021), Debiased machine learning of conditional average treatment effects and other causal functions, The Econometrics Journal 24(2). Chernozhukov et al. (2022), Long story short: omitted variable bias in causal machine learning, NBER WP 30302.