Average treatment effect with IRM¶
This notebook demonstrates a basic Average Treatment Effect (ATE) estimation using the DoubleMLIRM class.
The IRM model is:
$$Y = g_0(D, X) + U, \qquad \mathbb{E}(U | D, X) = 0,$$
$$D = m_0(X) + V, \qquad \mathbb{E}(V | X) = 0,$$
where $D \in \{0,1\}$ is a binary treatment. Unlike PLR, IRM allows the treatment effect to enter non-linearly through separate outcome models $g_0(0, X)$ and $g_0(1, X)$, making it more appropriate when effect heterogeneity is expected.
1. Setup and Imports¶
# Import required libraries
import os
from glob import glob
from doubleml_pyspark.irm import DoubleMLIRM
from doubleml_pyspark.utils import kfold_split
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.regression import LinearRegression
from pyspark.sql import SparkSession
from pyspark.sql.functions import monotonically_increasing_id, col, round
import pandas as pd
import numpy as np
from pyspark.sql.types import DoubleType, FloatType, DecimalType
2. Initialize Spark Session¶
# Create Spark session with basic configuration
def initialize_spark_session(
app_name="DoubleMLIRM_Optimized",
driver_memory="2g",
executor_memory="2g",
max_result_size="2g",
):
"""
Initialize Spark session with optimized memory and serialization
configurations.
"""
# Start building the session
builder = SparkSession.builder.appName(app_name)
# Apply memory and optimization configs
builder = (
builder.config("spark.driver.memory", driver_memory)
.config("spark.executor.memory", executor_memory)
.config("spark.driver.maxResultSize", max_result_size)
.config("spark.sql.shuffle.partitions", "20")
.config("spark.sql.files.maxPartitionBytes", "128m")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.adaptive.enabled", "true")
.config("spark.sql.adaptive.coalescePartitions.enabled", "true")
.config("spark.driver.extraJavaOptions", "-XX:+UseG1GC")
.config("spark.executor.extraJavaOptions", "-XX:+UseG1GC")
)
# Create the Spark session
spark = builder.getOrCreate()
return spark
spark = initialize_spark_session()
print("Spark session created successfully")
Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 25/09/08 09:17:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark session created successfully
3. Load Data¶
# Specify data path
data_path = os.path.abspath("data/irm_data/")
# Find and load parquet files
file_pattern = os.path.join(data_path, "chunk_*.parquet")
files = glob(file_pattern)
if not files:
raise FileNotFoundError(f"No files found at {file_pattern}")
# Load first chunk only for simplicity
df = spark.read.parquet(files[0])
print(f"Loaded data from: {files[0]}")
print(f"Number of rows: {df.count():,}")
print(f"Columns: {df.columns}")
Loaded data from: /workspace/sandbox/data/irm_data/chunk_7.parquet Number of rows: 1,000,000 Columns: ['X1', 'X2', 'X3', 'X4', 'X5', 'y', 'd']
4. Prepare Data¶
# Add unique ID column
df_with_id = df.withColumn("ID", monotonically_increasing_id())
# Round all numeric columns to 4 decimal places
numeric_columns = []
for field in df_with_id.schema.fields:
if isinstance(field.dataType, (DoubleType, FloatType, DecimalType)):
numeric_columns.append(field.name)
# Apply rounding to numeric columns (excluding ID which should remain as integer)
for col_name in numeric_columns:
if col_name != "ID": # Don't round the ID column
df_with_id = df_with_id.withColumn(col_name, round(col(col_name), 4))
# Cache for performance
df_with_id = df_with_id.cache()
print("Data prepared with ID column")
df_with_id.show(5)
Data prepared with ID column
+-------+-------+-------+-------+-------+-------+---+-----------+ | X1| X2| X3| X4| X5| y| d| ID| +-------+-------+-------+-------+-------+-------+---+-----------+ | 2.203| 0.6545|-1.3522| -1.084|-0.3557| 4.408|1.0|42949672960| |-0.0544|-0.1065|-0.1715| 0.1225| 0.5807| 0.0417|0.0|42949672961| | 0.7788|-0.8826| 0.2032|-0.0088|-1.6356| 2.0093|1.0|42949672962| |-0.0091| 0.19| 1.9802| 1.6362| 2.6721|-1.7635|0.0|42949672963| |-1.7022|-1.6882|-0.3741|-0.9821|-0.8385| 1.0388|0.0|42949672964| +-------+-------+-------+-------+-------+-------+---+-----------+ only showing top 5 rows
5. Configure DoubleML Model¶
# Define data field mapping
data_field_map = {
"id_column": "ID",
"outcome": "y", # Outcome variable
"treatment": "d", # Treatment variable
"covariates": ["X1", "X2", "X3", "X4", "X5"] # Control variables
}
print("Data field mapping configured:")
for key, value in data_field_map.items():
print(f" {key}: {value}")
Data field mapping configured: id_column: ID outcome: y treatment: d covariates: ['X1', 'X2', 'X3', 'X4', 'X5']
# Define ML models for nuisance estimation
ml_models = {
"ml_m": LogisticRegression(maxIter=100), # Propensity score model
"ml_g0": LinearRegression(maxIter=100), # Outcome model for control group
"ml_g1": LinearRegression(maxIter=100), # Outcome model for treatment group
}
print("ML models configured:")
for name, model in ml_models.items():
print(f" {name}: {type(model).__name__}")
ML models configured: ml_m: LogisticRegression ml_g0: LinearRegression ml_g1: LinearRegression
6. Create Cross-Validation Splits¶
# Create k-fold cross-validation splits
n_folds = 3 # Number of folds
n_rep = 1 # Number of repetitions
kf_split = kfold_split(
df_with_id,
n_folds=n_folds,
n_rep=n_rep,
shuffle=True,
seed=42
)
print(f"Created {n_folds}-fold cross-validation splits")
Created 3-fold cross-validation splits
7. Initialize DoubleML Model¶
# Initialize DoubleMLIRM for ATE estimation
dml_model = DoubleMLIRM(
data_schema=df_with_id.schema,
data_field_map=data_field_map,
target_parameter="ATE", # Average Treatment Effect
n_rep=n_rep
)
print("DoubleML model initialized")
print(f"Target parameter: {dml_model.target_parameter}")
print(f"Treatment variable: {dml_model.treatment}")
print(f"Outcome variable: {dml_model.outcome}")
DoubleML model initialized Target parameter: ATE Treatment variable: d Outcome variable: y
8. Fit the Model¶
# Fit the DoubleML model
print("Starting model fitting...")
print("This may take a few minutes depending on data size.\n")
try:
dml_model.fit(
spark_session=spark,
ml_models=ml_models,
kfold_split=kf_split
)
print("\n✓ Model fitting completed successfully!")
except Exception as e:
print(f"\nError during model fitting: {e}")
print("\nTroubleshooting tips:")
print("- Check that all required columns exist in the data")
print("- Verify Spark has sufficient memory")
print("- Try reducing the number of folds or data size")
raise
Starting model fitting... This may take a few minutes depending on data size.
25/09/08 09:17:14 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(G1 Concurrent GC), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors 25/09/08 09:17:21 WARN InstanceBuilder: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS 25/09/08 09:17:21 WARN InstanceBuilder: Failed to load implementation from:dev.ludovic.netlib.blas.VectorBLAS 25/09/08 09:17:28 WARN Instrumentation: [fa97d7f3] regParam is zero, which might cause numerical instability and overfitting. 25/09/08 09:17:31 WARN InstanceBuilder: Failed to load implementation from:dev.ludovic.netlib.lapack.JNILAPACK 25/09/08 09:17:33 WARN Instrumentation: [f3c035c7] regParam is zero, which might cause numerical instability and overfitting. 25/09/08 09:17:43 WARN Instrumentation: [b1c23a89] regParam is zero, which might cause numerical instability and overfitting. 25/09/08 09:17:47 WARN Instrumentation: [0ab21005] regParam is zero, which might cause numerical instability and overfitting. 25/09/08 09:17:55 WARN Instrumentation: [77b0f847] regParam is zero, which might cause numerical instability and overfitting. 25/09/08 09:17:59 WARN Instrumentation: [0e82ba9a] regParam is zero, which might cause numerical instability and overfitting. [Stage 162:=================================> (12 + 8) / 20]
✓ Model fitting completed successfully!
9. Display Results¶
# Display the estimated ATE
print("="*60)
print("ESTIMATION RESULTS")
print("="*60)
print(f"\nEstimated Average Treatment Effect (ATE): {dml_model.thetas[0]:.4f}")
============================================================ ESTIMATION RESULTS ============================================================ Estimated Average Treatment Effect (ATE): 2.0031
# Get confidence interval
conf_int = dml_model.confint()
print("\n95% Confidence Interval:")
print(conf_int)
95% Confidence Interval:
2.5 % 97.5 %
d 1.996058 2.010097
# Display sensitivity analysis if available
if hasattr(dml_model, 'sensitivity_summary'):
print("\nSensitivity Analysis:")
print(dml_model.sensitivity_summary)
Sensitivity Analysis: ================== Sensitivity Analysis ================== ------------------ Scenario ------------------ Significance Level: level=0.95 Sensitivity parameters: cf_y=0.03; cf_d=0.03, rho=1.0 ------------------ Bounds with CI ------------------ CI lower theta lower theta theta upper CI upper d 1.887758 1.893835 2.003077 2.11232 2.11876 ------------------ Robustness Values ------------------ H_0 RV (%) RVa (%) d 0.0 42.392005 41.799463
10. Interpretation¶
The Average Treatment Effect (ATE) represents the expected difference in outcomes between the treatment and control groups, after controlling for confounding variables.
- Positive ATE: Treatment has a positive effect on the outcome
- Negative ATE: Treatment has a negative effect on the outcome
- Confidence Interval: If it doesn't include zero, the effect is statistically significant at 5% level
def interpret_ate(est, ci: pd.DataFrame):
"""
Interpret the Average Treatment Effect (ATE) using a provided confidence interval.
Parameters
----------
est : float or numpy.ndarray
The estimated treatment effect
ci : pd.DataFrame
Confidence interval DataFrame, as returned by self.confint().
"""
lower = ci.iloc[:, 0].to_numpy()
upper = ci.iloc[:, 1].to_numpy()
# Extract scalar values if arrays have single elements
if isinstance(lower, np.ndarray) and lower.size == 1:
lower = lower.item()
if isinstance(upper, np.ndarray) and upper.size == 1:
upper = upper.item()
if isinstance(est, np.ndarray) and est.size == 1:
est = est.item()
# Check statistical significance
if lower > 0:
sig_text = "The treatment has a **positive and statistically significant** effect."
elif upper < 0:
sig_text = "The treatment has a **negative and statistically significant** effect."
else:
sig_text = "The treatment effect is **not statistically significant** (CI includes 0)."
print(f"ATE = {est:.3f}, CI = [{lower:.3f}, {upper:.3f}] → {sig_text}")
# Call the function
interpret_ate(dml_model.thetas, conf_int)
ATE = 2.003, CI = [1.996, 2.010] → The treatment has a **positive and statistically significant** effect.
11. Cleanup¶
# Stop Spark session
spark.stop()
print("Spark session closed")
Spark session closed
Summary¶
This notebook demonstrated a simple ATE analysis using DoubleML with PySpark:
- Data Loading: Loaded data from parquet files
- Data Preparation: Added ID column for tracking
- Model Configuration: Set up nuisance models (propensity score and outcome models)
- Cross-Validation: Used k-fold splitting for robust estimation
- Estimation: Fitted the DoubleML model
- Results: Obtained ATE estimate with confidence intervals
The DoubleML approach provides a robust way to estimate causal effects by:
- Using machine learning for flexible nuisance estimation
- Applying cross-fitting to avoid overfitting bias
- Providing valid statistical inference for the treatment effect