Average treatment effect with PLR¶
This notebook demonstrates Average Treatment Effect (ATE) estimation using the DoubleMLPLR class. PLR is simpler than IRM as it requires fewer nuisance models.
1. Setup and Imports¶
Partially linear regression (PLR) models take the form
$$ Y = D\theta_0 + g_0(X) + \zeta, \qquad \mathbb{E}(\zeta | D, X) = 0, $$
$$ D = m_0(X) + V, \qquad \mathbb{E}(V | X) = 0, $$
where $Y$ is the outcome variable and $D$ is the policy variable of interest. The high-dimensional vector $X = (X_1, \ldots, X_p)$ consists of other confounding covariates, and $\zeta$ and $V$ are stochastic errors.
# Import required libraries
import os
from glob import glob
from doubleml_pyspark.plm.plr import DoubleMLPLR
from doubleml_pyspark.utils import kfold_split
from pyspark.ml.regression import LinearRegression, RandomForestRegressor
from pyspark.sql import SparkSession
from pyspark.sql.functions import monotonically_increasing_id, col, round
from pyspark.sql.types import DoubleType, FloatType, DecimalType
2. Initialize Spark Session¶
A local Spark session with memory capped at 2 GB (driver + executor), shuffle partitions reduced from the default 200 to 20, adaptive query execution enabled, and KryoSerializer for faster object serialization. Increase memory settings if hitting OOM errors on larger datasets.
# Create Spark session with basic configuration
def initialize_spark_session(
app_name="DoubleMLIRM_Optimized",
driver_memory="2g",
executor_memory="2g",
max_result_size="2g",
):
"""
Initialize Spark session with optimized memory and serialization
configurations.
"""
# Start building the session
builder = SparkSession.builder.appName(app_name)
# Apply memory and optimization configs
builder = (
builder.config("spark.driver.memory", driver_memory)
.config("spark.executor.memory", executor_memory)
.config("spark.driver.maxResultSize", max_result_size)
.config("spark.sql.shuffle.partitions", "20")
.config("spark.sql.files.maxPartitionBytes", "128m")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.adaptive.enabled", "true")
.config("spark.sql.adaptive.coalescePartitions.enabled", "true")
.config("spark.driver.extraJavaOptions", "-XX:+UseG1GC")
.config("spark.executor.extraJavaOptions", "-XX:+UseG1GC")
)
# Create the Spark session
spark = builder.getOrCreate()
return spark
spark = initialize_spark_session()
print("Spark session created successfully")
WARNING: Using incubator modules: jdk.incubator.vector Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 26/03/19 17:23:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark session created successfully
3. Load and Prepare Data¶
# Specify data path
data_path = os.path.abspath("../data/plr_data/") # Adjust path as needed
# Find parquet files
file_pattern = os.path.join(data_path, "*.parquet")
files = glob(file_pattern)
if not files:
# If no PLR data, try using IRM data (it should work for PLR too)
data_path = os.path.abspath("../data/irm_data/")
file_pattern = os.path.join(data_path, "chunk_*.parquet")
files = glob(file_pattern)
if not files:
raise FileNotFoundError(f"No data files found")
# Load first file
df = spark.read.parquet(files[0])
print(f"Loaded data from: {files[0]}")
print(f"Number of rows: {df.count():,}")
print(f"Columns: {df.columns}")
Loaded data from: /workspace/sandbox/data/plr_data/chunk_0.parquet
[Stage 1:===================================================> (7 + 1) / 8]
Number of rows: 1,000,000 Columns: ['X1', 'X2', 'X3', 'X4', 'X5', 'y', 'd']
We attach a unique row identifier required by the cross-fitting routine, round all floating-point columns to 4 decimal places for numerical stability, and persist the DataFrame in memory to avoid recomputation across folds.
Note:
monotonically_increasing_id()produces globally unique but non-contiguous IDs that are not stable across re-executions — do not rely on them for row ordering or reproducibility outside of a single session.
# Add unique ID column
df_with_id = df.withColumn("ID", monotonically_increasing_id())
# Round all numeric columns to 4 decimal places
numeric_columns = []
for field in df_with_id.schema.fields:
if isinstance(field.dataType, (DoubleType, FloatType, DecimalType)):
numeric_columns.append(field.name)
# Apply rounding to numeric columns (excluding ID which should remain as integer)
for col_name in numeric_columns:
if col_name != "ID": # Don't round the ID column
df_with_id = df_with_id.withColumn(col_name, round(col(col_name), 4))
# Cache for performance
df_with_id = df_with_id.cache()
print("Data prepared with ID column")
df_with_id.show(5)
Data prepared with ID column
+-------+-------+-------+-------+-------+-------+-------+-----------+ | X1| X2| X3| X4| X5| y| d| ID| +-------+-------+-------+-------+-------+-------+-------+-----------+ |-0.5305|-0.0293|-0.7574|-0.8054| 0.2615|-0.9593|-0.6535|25769803776| | 1.5042| 0.5167| 0.0462|-0.5098|-0.5239| 3.71| 2.5428|25769803777| | 0.3699| -0.002|-0.1616| 1.4997| 0.1174| 5.1316| 1.5267|25769803778| |-0.0316| 0.0718|-0.0187| 1.4367| 0.7678| 1.9758| 0.4895|25769803779| |-0.8809|-1.6079|-1.4581|-0.5893|-1.2095|-2.5507|-0.4727|25769803780| +-------+-------+-------+-------+-------+-------+-------+-----------+ only showing top 5 rows
4. Configure PLR Model¶
PLR requires only two nuisance models:
ml_m: Model for treatment given covariates (E[D|X])ml_l: Model for outcome given covariates (E[Y|X])
# Define data field mapping
data_field_map = {
"id_column": "ID",
"outcome": "y", # Outcome variable
"treatment": "d", # Treatment variable
"covariates": ["X1", "X2", "X3", "X4", "X5"] # Control variables
}
print("Data field mapping:")
for key, value in data_field_map.items():
print(f" {key}: {value}")
Data field mapping: id_column: ID outcome: y treatment: d covariates: ['X1', 'X2', 'X3', 'X4', 'X5']
# Define ML models for nuisance estimation
# PLR only needs two models (simpler than IRM!)
ml_models = {
"ml_m": LinearRegression(maxIter=100), # Model for E[D|X]
"ml_l": LinearRegression(maxIter=100), # Model for E[Y|X]
}
# Alternative: Use more flexible models for better performance
# ml_models = {
# "ml_m": RandomForestRegressor(numTrees=100, maxDepth=5),
# "ml_l": RandomForestRegressor(numTrees=100, maxDepth=5),
# }
print("ML models configured:")
for name, model in ml_models.items():
print(f" {name}: {type(model).__name__}")
ML models configured: ml_m: LinearRegression ml_l: LinearRegression
5. Create Cross-Validation Splits¶
# Create k-fold cross-validation splits
n_folds = 3
n_rep = 1
kf_split = kfold_split(
df_with_id,
n_folds=n_folds,
n_rep=n_rep,
shuffle=True,
seed=42
)
print(f"Created {n_folds}-fold cross-validation splits")
print(f"Number of repetitions: {n_rep}")
Created 3-fold cross-validation splits Number of repetitions: 1
6. Initialize and Fit PLR Model¶
# Initialize DoubleMLPLR
dml_plr = DoubleMLPLR(
data_schema=df_with_id.schema,
data_field_map=data_field_map,
)
print("DoubleML PLR model initialized")
print(f"Treatment variable: {dml_plr.treatment}")
print(f"Outcome variable: {dml_plr.outcome}")
print(f"Covariates: {dml_plr.covariates}")
DoubleML PLR model initialized Treatment variable: d Outcome variable: y Covariates: ['X1', 'X2', 'X3', 'X4', 'X5']
The fit method orchestrates the full DML procedure: for each cross-fitting fold, it trains the nuisance models (ml_l, ml_m) on the training split, generates out-of-sample residuals on the held-out split, then aggregates across folds and solves the final partialling-out estimating equation to recover θ.
# Fit the model
print("Starting PLR model fitting...")
print("This should be faster than IRM as it uses fewer models.\n")
try:
dml_plr.fit(
spark_session=spark,
ml_models=ml_models,
kfold_split=kf_split[0]
)
print("\n✓ Model fitting completed successfully!")
except Exception as e:
print(f"\nError during model fitting: {e}")
raise
Starting PLR model fitting... This should be faster than IRM as it uses fewer models.
26/03/19 17:23:59 WARN Instrumentation: [70f2fd7d] regParam is zero, which might cause numerical instability and overfitting. 26/03/19 17:24:13 WARN Instrumentation: [c7f4e47c] regParam is zero, which might cause numerical instability and overfitting. 26/03/19 17:24:30 WARN Instrumentation: [42792260] regParam is zero, which might cause numerical instability and overfitting. 26/03/19 17:24:40 WARN Instrumentation: [2b470875] regParam is zero, which might cause numerical instability and overfitting. 26/03/19 17:24:54 WARN Instrumentation: [856b170a] regParam is zero, which might cause numerical instability and overfitting. 26/03/19 17:25:03 WARN Instrumentation: [d191e99f] regParam is zero, which might cause numerical instability and overfitting. [Stage 189:=======================================> (7 + 3) / 10]
✓ Model fitting completed successfully!
7. Display Results¶
# Display the estimated treatment effect
print("="*60)
print("ESTIMATION RESULTS")
print("="*60)
theta = dml_plr.thetas[0] if hasattr(dml_plr, 'thetas') else dml_plr.theta
print(f"\nEstimated Treatment Effect: {theta:.4f}")
print("\nInterpretation:")
print(f"A one-unit increase in the treatment (d) is associated with")
print(f"a {theta:.4f} unit change in the outcome (y), holding covariates constant.")
============================================================ ESTIMATION RESULTS ============================================================ Estimated Treatment Effect: 1.9991 Interpretation: A one-unit increase in the treatment (d) is associated with a 1.9991 unit change in the outcome (y), holding covariates constant.
conf_int = dml_plr.confint()
conf_int
| 2.5 % | 97.5 % | |
|---|---|---|
| d | 1.9971 | 2.001018 |
# Get confidence interval
print("\n95% Confidence Interval:")
print(f" Lower bound: {conf_int["2.5 %"][0]:.4f}")
print(f" Upper bound: {conf_int["97.5 %"][0]:.4f}")
# Check statistical significance
if conf_int["2.5 %"][0] > 0:
print("\n✓ The treatment effect is positive and statistically significant (p < 0.05)")
elif conf_int["97.5 %"][0] < 0:
print("\n✓ The treatment effect is negative and statistically significant (p < 0.05)")
else:
print("\n⚠ The treatment effect is not statistically significant at the 5% level")
95% Confidence Interval: Lower bound: 1.9971 Upper bound: 2.0010 ✓ The treatment effect is positive and statistically significant (p < 0.05)
/tmp/ipykernel_88696/1789045467.py:4: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
print(f" Lower bound: {conf_int["2.5 %"][0]:.4f}")
/tmp/ipykernel_88696/1789045467.py:5: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
print(f" Upper bound: {conf_int["97.5 %"][0]:.4f}")
/tmp/ipykernel_88696/1789045467.py:8: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
if conf_int["2.5 %"][0] > 0:
# Display standard error if available
if hasattr(dml_plr, 'se'):
print(f"\nStandard Error: {dml_plr.se[0]:.4f}")
# Calculate t-statistic
t_stat = theta / dml_plr.se[0]
print(f"t-statistic: {t_stat:.4f}")
# Rough p-value interpretation
if abs(t_stat) > 2.576:
print("p-value: < 0.01 (highly significant)")
elif abs(t_stat) > 1.96:
print("p-value: < 0.05 (significant)")
elif abs(t_stat) > 1.645:
print("p-value: < 0.10 (marginally significant)")
else:
print("p-value: > 0.10 (not significant)")
8. Model Diagnostics (Optional)¶
# Check residuals distribution (if we have access to nuisance predictions)
print("\nModel Diagnostics:")
print("-" * 40)
# Get basic statistics about the data with rounded values
stats_df = df_with_id.select(
col("d").alias("treatment"),
col("y").alias("outcome")
).describe()
# Round the numeric columns in the stats DataFrame to 4 decimal places
# The describe() function creates columns with string values, so we need to handle this differently
stats_rounded = stats_df.select(
col("summary"),
round(col("treatment").cast("double"), 4).alias("treatment"),
round(col("outcome").cast("double"), 4).alias("outcome")
)
# Show the formatted statistics
stats_rounded.show()
print("\nNote: For more detailed diagnostics, you would need to:")
print("- Check residual plots")
print("- Verify overlap in propensity scores")
print("- Test for covariate balance")
print("- Examine first-stage prediction quality")
Model Diagnostics: ---------------------------------------- +-------+---------+---------+ |summary|treatment| outcome| +-------+---------+---------+ | count|1000000.0|1000000.0| | mean| 0.1243| 0.7507| | stddev| 1.4335| 3.2805| | min| -7.7917| -16.7858| | max| 7.0449| 16.64| +-------+---------+---------+ Note: For more detailed diagnostics, you would need to: - Check residual plots - Verify overlap in propensity scores - Test for covariate balance - Examine first-stage prediction quality
9. Cleanup¶
# We stop Spark session
spark.stop()
print("Spark session closed")
Spark session closed
Summary¶
What is PLR (Partially Linear Regression)?¶
PLR estimates the model: Y = θD + g(X) + ε
Where:
- Y is the outcome
- D is the treatment (can be binary or continuous)
- X are covariates
- θ is the treatment effect (what we want to estimate)
- g(X) is an unknown function of covariates
Advantages of PLR over IRM:¶
- Simpler: Only needs 2 nuisance models vs 3 for IRM
- Faster: Less computational overhead
- Works for continuous treatments: Not limited to binary treatments
- More efficient: Often has smaller standard errors when assumptions hold
When to use PLR vs IRM:¶
- Use PLR when: Treatment effect is homogeneous (same for everyone)
- Use IRM when: Treatment effect may vary across individuals (heterogeneous effects)
Key Assumptions:¶
- Unconfoundedness: All confounders are observed and included in X
- Overlap: Every unit has positive probability of treatment
- Linearity: Treatment enters the outcome equation linearly
The DoubleML approach ensures valid inference even when using flexible ML methods for nuisance estimation.