Installation¶
On Azure Databricks, serverless or cluster, subscribe in the Microsoft Marketplace and store your license key as a secret, as described in Install from the Microsoft Marketplace. Then run these cells in a notebook:
key = dbutils.secrets.get("doubleml", "license-key")
%pip install -q --index-url https://license:$key@license.doubleml-cloud.com/simple doubleml-pyspark
dbutils.library.restartPython()
Using it elsewhere? See Other platforms. For the runtimes we support, see Supported environments.
Quick start¶
Did a coupon campaign increase spending? The dataset is simulated, so we know the true effect and can check the result. The coupon was not sent at random, which is why a simple comparison would be wrong.
On Databricks the Spark session already exists, elsewhere this line starts one.
from pyspark.ml.classification import GBTClassifier
from pyspark.ml.regression import GBTRegressor
from pyspark.sql import SparkSession
from doubleml_pyspark.datasets import make_coupon_data
from doubleml_pyspark.irm import DoubleMLIRM
from doubleml_pyspark.utils import kfold_split
spark = SparkSession.getActiveSession() or SparkSession.builder.getOrCreate()
df, covariates = make_coupon_data(spark, n_obs=20_000, seed=42)
df.select("ID", "coupon", "basket_value_30d", *covariates[:3]).limit(5).toPandas()
| ID | coupon | basket_value_30d | tenure_months | log_past_spend | visits_90d | |
|---|---|---|---|---|---|---|
| 0 | 0 | 0.0 | 45.542386 | 96.0 | 5.562929 | 6.0 |
| 1 | 1 | 0.0 | 17.317332 | 79.0 | 5.624077 | 2.0 |
| 2 | 2 | 0.0 | 52.940097 | 30.0 | 5.330364 | 12.0 |
| 3 | 3 | 0.0 | 47.888208 | 25.0 | 6.608067 | 4.0 |
| 4 | 4 | 0.0 | 33.960504 | 77.0 | 6.131646 | 1.0 |
Declare the role of each column, choose the machine learning models and fit. ml_m learns
who receives a coupon, ml_g0 and ml_g1 learn spending without and with it.
dml = DoubleMLIRM(
data_schema=df.schema,
data_field_map={
"id_column": "ID",
"outcome": "basket_value_30d",
"treatment": "coupon",
"covariates": covariates,
},
)
ml_models = {
"ml_m": GBTClassifier(maxIter=10, maxDepth=5, stepSize=0.3, seed=42),
"ml_g0": GBTRegressor(maxIter=10, maxDepth=5, stepSize=0.3, seed=42),
"ml_g1": GBTRegressor(maxIter=10, maxDepth=5, stepSize=0.3, seed=42),
}
splits = kfold_split(df, n_folds=2, n_rep=1, shuffle=True, seed=42)
dml.fit(spark_session=spark, kfold_split=splits[0], ml_models=ml_models)
print(dml.summary)
coef std err t P>|t| 2.5 % 97.5 % coupon 6.50926 0.215646 30.184875 0.0 6.086601 6.93192
coef is the estimated effect of the coupon on spending per customer in USD, with its
standard error and confidence interval. We can compare it with the true effect, which real
data would not have:
from pyspark.sql import functions as F
print(f"estimated {dml.coef[0]:.2f} USD, true {df.agg(F.avg('true_effect')).first()[0]:.2f} USD")
estimated 6.51 USD, true 6.16 USD
Next steps¶
- Workflow: each step and its options on this dataset, including tuning and sensitivity analysis
- Business example: effects by customer group, charts and who should get the coupon
- API reference