Conditional average treatment effects with IRM¶
This notebook estimates heterogeneous treatment effects using the Best Linear Predictor (BLP) approach (Semenova & Chernozhukov, 2021) on top of an Interactive Regression Model (IRM).
The IRM model is: $$Y = g_0(D, X) + U, \qquad \mathbb{E}(U | D, X) = 0,$$ $$D = m_0(X) + V, \qquad \mathbb{E}(V | X) = 0,$$
where $D \in \{0,1\}$ is a binary treatment. Unlike PLR, IRM allows the treatment effect to enter non-linearly through separate outcome models $g_0(0, X)$ and $g_0(1, X)$, making it more appropriate when effect heterogeneity is expected.
The BLP projects the IRM score onto flexible basis functions $B(X)$ to recover conditional treatment effects $\hat\tau(x)$ along any covariate or pair of covariates.
Workflow
- Spark session + data loading
- Nuisance fitting (keeping
df_nuisance) - ATE estimation + sensitivity analysis
- 1-D CATE along a single covariate (B-spline BLP)
- 2-D CATE surface over two covariates (tensor-product B-splines)
- Cleanup
1. Imports and Configuration¶
import os
from glob import glob
import numpy as np
import pandas as pd
import patsy
from scipy.stats import norm
from matplotlib import pyplot as plt
from doubleml_pyspark.irm import DoubleMLIRM
from doubleml_pyspark.utils import kfold_split
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.regression import LinearRegression
from pyspark.sql import SparkSession
from pyspark.sql.functions import monotonically_increasing_id
plt.rcParams['figure.figsize'] = (12, 7)
plt.rcParams['font.size'] = 12
plt.rcParams['axes.grid'] = True
plt.rcParams['grid.alpha'] = 0.3
# ── Configuration ──────────────────────────────────────────────────────────
DATA_PATH = os.path.abspath("../data/irm_data/")
N_CHUNKS = 1
N_FEATURES = 5
N_FOLDS = 2
SAMPLE_FRACTION = None # set e.g. 0.1 for 10% during development
print(f"Data path : {DATA_PATH}")
print(f"Chunks : {N_CHUNKS}")
print(f"Features : {N_FEATURES}")
print(f"CV folds : {N_FOLDS}")
print(f"Sample fraction: {SAMPLE_FRACTION or 'full dataset'}")
Data path : /workspace/sandbox/data/irm_data Chunks : 1 Features : 5 CV folds : 2 Sample fraction: full dataset
2. Spark Session¶
A local session with a broadcast threshold tuned so small DataFrames (like the BLP basis) get broadcast automatically during joins.
Increase driver_memory if you hit OOM on large datasets.
# Reset stale Spark state after a crash before creating a new session
from pyspark import SparkContext
SparkSession._instantiatedSession = None
SparkSession._activeSession = None
try:
sc = SparkContext._active_spark_context
if sc is not None:
sc.stop()
except Exception:
pass
SparkContext._active_spark_context = None
SparkContext._gateway = None
print("Stale Spark state cleared. Ready to create a new session.")
Stale Spark state cleared. Ready to create a new session.
def initialize_spark_session(
app_name="DoubleMLIRM_CATE",
driver_memory="4g",
executor_memory="4g",
max_result_size="2g",
):
"""
Build a SparkSession tuned for DoubleML IRM workloads.
Parameters
----------
app_name : Spark application name
driver_memory : JVM heap for the driver (increase if OOM)
executor_memory : JVM heap per executor
max_result_size : cap on data collected to the driver at once
"""
spark = (
SparkSession.builder
.appName(app_name)
.config("spark.driver.memory", driver_memory)
.config("spark.executor.memory", executor_memory)
.config("spark.driver.maxResultSize", max_result_size)
# More partitions keeps per-task data small
.config("spark.sql.shuffle.partitions", "200")
.config("spark.sql.files.maxPartitionBytes", "32m")
.config("spark.default.parallelism", "200")
# AQE: let Spark coalesce small shuffle partitions automatically
.config("spark.sql.adaptive.enabled", "true")
.config("spark.sql.adaptive.coalescePartitions.enabled", "true")
.config("spark.sql.adaptive.coalescePartitions.minPartitionSize", "1m")
.config("spark.sql.adaptive.advisoryPartitionSizeInBytes", "32m")
# Small DataFrames (e.g. BLP basis) are broadcast rather than shuffled
.config("spark.sql.autoBroadcastJoinThreshold", "50m")
# Faster serialisation + G1GC to reduce GC pauses
.config("spark.serializer",
"org.apache.spark.serializer.KryoSerializer")
.config("spark.driver.extraJavaOptions", "-XX:+UseG1GC")
.config("spark.executor.extraJavaOptions", "-XX:+UseG1GC")
.getOrCreate()
)
return spark
spark = initialize_spark_session()
print("Spark session created successfully.")
WARNING: Using incubator modules: jdk.incubator.vector Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 26/04/27 11:08:52 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 26/04/27 11:08:54 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 26/04/27 11:08:54 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
Spark session created successfully.
3. Data Loading and Preparation¶
def load_data(data_path, n_chunks=1, sample_fraction=None):
"""Load parquet chunks; optionally subsample for development."""
files = glob(os.path.join(data_path, "chunk_*.parquet"))
if not files:
raise FileNotFoundError(f"No chunk_*.parquet files found in {data_path}")
files = files[:min(n_chunks, len(files))]
print(f"Loading {len(files)} file(s)...")
df = spark.read.parquet(*files)
if sample_fraction and sample_fraction < 1.0:
print(f"Sampling {sample_fraction * 100:.0f}% of data (seed=42)...")
df = df.sample(fraction=sample_fraction, seed=42)
n = df.count()
print(f"Rows: {n:,} | Columns: {df.columns}")
if n > 100_000:
n_parts = min(20, max(2, n // 50_000))
print(f"Repartitioning to {n_parts} partitions...")
df = df.repartition(n_parts)
return df
def prepare_data(df):
"""
Attach a unique ID column and cache the DataFrame.
Note: `monotonically_increasing_id()` is globally unique but non-contiguous
and not stable across re-executions — use only as a join key within a session.
"""
df = df.withColumn("ID", monotonically_increasing_id())
df = df.cache()
print(f"DataFrame cached ({df.count():,} rows).")
df.printSchema()
return df
df_raw = load_data(DATA_PATH, N_CHUNKS, SAMPLE_FRACTION)
df_with_id = prepare_data(df_raw)
Loading 1 file(s)...
Rows: 1,000,000 | Columns: ['X1', 'X2', 'X3', 'X4', 'X5', 'y', 'd'] Repartitioning to 20 partitions...
[Stage 6:======================================================> (19 + 1) / 20]
DataFrame cached (1,000,000 rows). root |-- X1: double (nullable = true) |-- X2: double (nullable = true) |-- X3: double (nullable = true) |-- X4: double (nullable = true) |-- X5: double (nullable = true) |-- y: double (nullable = true) |-- d: double (nullable = true) |-- ID: long (nullable = false)
4. Model Configuration¶
IRM requires three nuisance models, because the treatment can enter the outcome non-linearly. The first, ml_m, targets E[D | X] and represents the propensity score estimated with LogisticRegression. The second, ml_g0, targets E[Y | D=0, X] and is the outcome model for the control group. The third, ml_g1, targets E[Y | D=1, X] and is the outcome model for the treated group.
covariate_cols = [f"X{i+1}" for i in range(N_FEATURES)]
data_field_map = {
"id_column" : "ID",
"outcome" : "y",
"treatment" : "d",
"covariates": covariate_cols,
}
ml_models = {
"ml_m" : LogisticRegression(maxIter=50, regParam=0.01), # P(D=1|X)
"ml_g0": LinearRegression(maxIter=50, regParam=0.01), # E[Y|D=0,X]
"ml_g1": LinearRegression(maxIter=50, regParam=0.01), # E[Y|D=1,X]
}
# Validate required columns exist in data
required_cols = set(covariate_cols + ["y", "d"])
missing = required_cols - set(df_with_id.columns)
if missing:
raise ValueError(f"Missing columns in data: {missing}")
print("✓ All required columns found.")
print("Data field map:", data_field_map)
print("ML models :", {k: type(v).__name__ for k, v in ml_models.items()})
✓ All required columns found.
Data field map: {'id_column': 'ID', 'outcome': 'y', 'treatment': 'd', 'covariates': ['X1', 'X2', 'X3', 'X4', 'X5']}
ML models : {'ml_m': 'LogisticRegression', 'ml_g0': 'LinearRegression', 'ml_g1': 'LinearRegression'}
5. Cross-Fitting: Nuisance Models + ATE¶
We call fit_ml_models() and estimate_causal_parameters() separately so that
df_nuisance stays cached for the downstream cate() calls.
(fit() would unpersist it automatically at the end.)
kfold_split returns a list of length n_rep, each element being n_folds tuples
of (train_ids, test_ids). For each fold, nuisance models are trained on the
out-of-fold rows and predict only on the held-out rows, avoiding overfitting bias
in the final θ estimate.
print("=" * 70)
print("STARTING ESTIMATION (retaining nuisance for CATE)")
print("=" * 70)
kf_split = kfold_split(
df_with_id, n_folds=N_FOLDS, n_rep=1, shuffle=True, seed=42
)
dml_model = DoubleMLIRM(
data_schema=df_with_id.schema,
data_field_map=data_field_map,
)
df_nuisance = None
estimation_successful = False
try:
# Step 1: fit nuisance models across folds, keep df_nuisance alive
print("Fitting nuisance ML models...")
df_nuisance, additional_nuisance = dml_model.fit_ml_models(
spark_session=spark,
kfold_split=kf_split[0],
ml_models=ml_models,
)
df_nuisance = df_nuisance.cache()
print(f"df_nuisance cached ({df_nuisance.count():,} rows).")
# Step 2: solve the IRM estimating equation for the ATE
print("Estimating causal parameters (ATE)...")
dml_model.estimate_causal_parameters(
spark_session=spark,
df_nuisance=df_nuisance,
additional_nuisance_elements=additional_nuisance,
)
# Step 3: sensitivity analysis (robustness to hidden confounders)
dml_model.sensitivity_analysis()
estimation_successful = True
print("\n✓ Model fitting completed successfully!")
except Exception as e:
print(f"\n✗ Error: {e}")
if "Connection refused" in str(e) or "Py4JNetworkError" in str(e):
print("Tip: set SAMPLE_FRACTION=0.1 or increase Spark driver memory.")
====================================================================== STARTING ESTIMATION (retaining nuisance for CATE) ====================================================================== Fitting nuisance ML models...
df_nuisance cached (1,000,000 rows). Estimating causal parameters (ATE)...
[Stage 325:=============================================> (171 + 8) / 200]
✓ Model fitting completed successfully!
ATE Results¶
if estimation_successful:
theta_ate = dml_model.thetas[0]
ci = dml_model.confint()
print("=" * 55)
print("ATE RESULTS (IRM)")
print("=" * 55)
print(f" θ̂ (ATE) : {theta_ate:.4f}")
print(f" 95% CI : [{ci.iloc[0,0]:.4f}, {ci.iloc[0,1]:.4f}]")
print("\nConfidence Interval:")
display(ci)
print("\nSensitivity Analysis Summary:")
print(dml_model.sensitivity_summary)
else:
print("⚠ Skipping results — estimation failed.")
======================================================= ATE RESULTS (IRM) ======================================================= θ̂ (ATE) : 2.0033 95% CI : [1.9973, 2.0093] Confidence Interval:
| 2.5 % | 97.5 % | |
|---|---|---|
| d | 1.997283 | 2.009338 |
Sensitivity Analysis Summary: ================== Sensitivity Analysis ================== ------------------ Scenario ------------------ Significance Level: level=0.95 Sensitivity parameters: cf_y=0.03; cf_d=0.03, rho=1.0 ------------------ Bounds with CI ------------------ CI lower theta lower theta theta upper CI upper d 1.894359 1.899365 2.003311 2.107256 2.112628 ------------------ Robustness Values ------------------ H_0 RV (%) RVa (%) d 0.0 43.950639 43.6009
6. Conditional Average Treatment Effects (CATEs)¶
We use the Best Linear Predictor (BLP) approach from Semenova & Chernozhukov (2021). The CATE along covariate Xj is recovered by regressing the score on a B-spline basis B(Xj), giving a predicted treatment effect as the inner product of the basis evaluated at xj and the estimated BLP coefficients.
6.1 Confidence Intervals via DoubleMLBLP.confint()¶
DoubleMLBLP.confint(spark_session, basis, level) returns pointwise confidence intervals for the predicted CATE on a Spark DataFrame evaluation grid, using the coefficient covariance precomputed by fit(). We call it directly on each grid below — no local helper, no manual covariance reconstruction. The returned Spark DataFrame has columns for the lower bound, the effect, and the upper bound, which we convert with toPandas() for plotting with matplotlib or Plotly.
The grid is pushed through spark.createDataFrame(...).coalesce(1) so that toPandas() returns rows in the same order as the input grid, allowing us to align the result with the original x_vals array for plotting.
6.2 Collect a Covariate Sample to Pandas¶
Pulling the full dataset to the driver with .toPandas() can OOM the Spark JVM on large data. We instead sample a fixed number of rows within Spark first, then collect. 100k rows is sufficient for a consistent BLP estimate.
# Sample within Spark before collecting — avoids pulling millions of rows to the driver
N_SAMPLE_BLP = 100_000
n_total = df_with_id.count()
sample_frac = min(1.0, N_SAMPLE_BLP / n_total)
print(f"Dataset has {n_total:,} rows — sampling {sample_frac * 100:.1f}% ({N_SAMPLE_BLP:,} rows) for BLP basis.")
df_cov_pd = (
df_with_id
.select("ID", *covariate_cols)
.sample(fraction=sample_frac, seed=42)
.toPandas()
)
print(f"Collected {len(df_cov_pd):,} rows.")
df_cov_pd.head(3)
Dataset has 1,000,000 rows — sampling 10.0% (100,000 rows) for BLP basis. Collected 100,453 rows.
| ID | X1 | X2 | X3 | X4 | X5 | |
|---|---|---|---|---|---|---|
| 0 | 7 | -0.391026 | -1.486012 | -1.865118 | -1.446912 | -1.353748 |
| 1 | 16 | 0.158469 | -0.353274 | 0.700180 | 0.805646 | -0.179496 |
| 2 | 18 | -0.325976 | -0.792757 | -0.325140 | 0.537305 | -1.088154 |
6.3 One-Dimensional CATE along a Single Covariate¶
We build a degree-2 B-spline basis on the chosen covariate, fit the BLP model,
then evaluate on a fine grid with uniform 95% confidence bands.
Change cov to inspect a different covariate.
cov = "X5" # ← change to any covariate in covariate_cols
# ── Build B-spline training basis ─────────────────────────────────────────
dm = patsy.dmatrix(
"bs(x, df=5, degree=2)",
{"x": df_cov_pd[cov]},
return_type="matrix", # retains .design_info for grid evaluation
)
bcols = [f"basis_{j}" for j in range(dm.shape[1])]
bpd = pd.DataFrame(np.asarray(dm), columns=bcols)
bpd["ID"] = df_cov_pd["ID"].values
print(f"Basis: {len(bcols)} cols × {len(bpd):,} rows")
# ── Fit BLP ───────────────────────────────────────────────────────────────
blp = dml_model.cate(
spark_session=spark,
df_basis=spark.createDataFrame(bpd),
df_nuisance=df_nuisance,
)
print("BLP fitted.")
# ── Evaluate on grid via DoubleMLBLP.confint() ────────────────────────────
x_vals = np.linspace(
df_cov_pd[cov].quantile(0.05),
df_cov_pd[cov].quantile(0.95),
100,
)
grid = pd.DataFrame(
patsy.build_design_matrices([dm.design_info], {"x": x_vals})[0],
columns=bcols,
)
df_ci = (
blp.confint(
spark_session=spark,
basis=spark.createDataFrame(grid).coalesce(1),
level=0.95,
)
.toPandas()
)
df_ci["x"] = x_vals
# ── Plot ──────────────────────────────────────────────────────────────────
fig, ax = plt.subplots(figsize=(9, 5))
ax.plot(df_ci["x"], df_ci["effect"],
color="#2563eb", linewidth=2, label="CATE")
ax.fill_between(df_ci["x"], df_ci["2.5 %"], df_ci["97.5 %"],
color="#2563eb", alpha=0.15, label="Pointwise 95% CI")
if estimation_successful:
ax.axhline(theta_ate, color="#dc2626", linestyle="--",
linewidth=1.5, alpha=0.8, label=f"ATE = {theta_ate:.3f}")
ax.set_xlabel(cov)
ax.set_ylabel("Treatment Effect")
ax.set_title(f"CATE along {cov} (IRM · BLP)")
ax.legend()
plt.tight_layout()
plt.show()
Basis: 6 cols × 100,453 rows
26/04/27 11:12:44 WARN Instrumentation: [15e01846] regParam is zero, which might cause numerical instability and overfitting. 26/04/27 11:12:45 WARN Instrumentation: [15e01846] Cholesky solver failed due to singular covariance matrix. Retrying with Quasi-Newton solver. 26/04/27 11:12:48 WARN PearsonCorrelation: Pearson correlation matrix contains NaN values.
BLP fitted.
7. Cleanup¶
# Unpersist nuisance DataFrame now that all CATE estimation is done
if df_nuisance is not None:
df_nuisance.unpersist()
print("✓ df_nuisance unpersisted.")
try:
spark.stop()
print("✓ Spark session stopped.")
except Exception as e:
print(f"⚠ Cleanup warning: {e}")
✓ df_nuisance unpersisted. ✓ Spark session stopped.
Memory Optimization Tips¶
If you encounter OOM errors or a Py4JNetworkError, try these in order:
- Sample the data: set
SAMPLE_FRACTION = 0.1to work with 10% of the data - Reduce CV folds: use
N_FOLDS = 2(already set as default) - Increase Spark memory: raise
driver_memory/executor_memoryininitialize_spark_session() - Use fewer chunks: reduce
N_CHUNKSto load less data - Reduce grid size: lower
GRID_SIZEfor the 2-D CATE evaluation
References¶
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018).
Double/debiased machine learning for treatment and structural parameters.
The Econometrics Journal, 21(1), C1–C68.Semenova, V., & Chernozhukov, V. (2021).
Debiased machine learning of conditional average treatment effects and other causal functions.
The Econometrics Journal, 24(2), 264–289.