DoubleML Tuning Edge Cases & Advanced Patterns¶
This notebook explores edge cases and advanced usage patterns for hyperparameter tuning with Optuna in doubleml-pyspark. Each section is self-contained and demonstrates a specific scenario you may encounter in practice.
Contents:
| # | Edge Case | Key Idea |
|---|---|---|
| 1 | Setup | Creating the data for the experiment |
| 2 | Partial Tuning | Tune only a subset of learners |
| 3 | No-Tuning Baseline | Empty param space → original models returned unchanged |
| 4 | Explicit None Skipping |
Mark specific learners as skip with None |
| 5 | Model-Specific Optuna Overrides | Different trial counts, directions, and scoring per learner |
| 6 | Memory-Constrained Sequential Tuning | Tune one learner at a time to reduce memory pressure |
| 7 | Default Settings Fallback | optuna_settings=None uses sensible defaults |
This notebook is a companion to the Tuning Walkthrough. That page covers the standard end-to-end workflow; here we focus on non-default tuning scenarios you'll hit in practice such as partial tuning, skipping learners, per-model overrides, and memory-constrained setups.
Each section is self-contained and uses the same synthetic IRM dataset.
Drops the big table-of-contents table (the sections themselves are already numbered and titled — the TOC just duplicates them).
1 · Setup¶
import optuna
import numpy as np
import pandas as pd
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.regression import LinearRegression
from pyspark.sql import SparkSession
from pyspark.sql.functions import monotonically_increasing_id
from doubleml_pyspark.irm import DoubleMLIRM
from doubleml_pyspark.utils import kfold_split
optuna.logging.set_verbosity(optuna.logging.WARNING)
spark = (
SparkSession.builder
.appName("DoubleML_EdgeCases")
.config("spark.driver.memory", "2g")
.getOrCreate()
)
print(f"Spark {spark.version}")
WARNING: Using incubator modules: jdk.incubator.vector Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 26/03/25 15:15:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 26/03/25 15:15:24 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Spark 4.0.1
Synthetic IRM Data¶
We generate a simple Interactive Regression Model dataset with a known treatment effect (theta = 0.5) so we can verify that tuning choices lead to reasonable estimates.
TRUE_THETA = 0.5
def make_irm_data(n_obs=10_000, dim_x=5, theta=TRUE_THETA, seed=42):
np.random.seed(seed)
X = np.random.normal(0, 1, size=(n_obs, dim_x))
propensity = 1 / (1 + np.exp(-(X[:, 0] + 0.5 * X[:, 1])))
d = np.random.binomial(1, propensity)
g0 = X[:, 0] + 0.5 * X[:, 1] ** 2 + X[:, 2]
g1 = g0 + theta
y = d * g1 + (1 - d) * g0 + np.random.normal(0, 0.5, n_obs)
df = pd.DataFrame(X, columns=[f"X{i+1}" for i in range(dim_x)])
df["d"] = d
df["y"] = y
return df
df = (
spark.createDataFrame(make_irm_data())
.withColumn("ID", monotonically_increasing_id())
.cache()
)
data_schema = df.schema
data_field_map = {
"id_column": "ID",
"outcome": "y",
"treatment": "d",
"covariates": ["X1", "X2", "X3", "X4", "X5"],
}
data_split = kfold_split(df, n_folds=2, n_rep=1, shuffle=True, seed=42)[0][0]
print(f"Observations: {df.count():,} | True ATE: {TRUE_THETA}")
[Stage 0:=======> (1 + 7) / 8]
Observations: 10,000 | True ATE: 0.5
Reusable Parameter Spaces & Model Dict¶
We define these once and reuse them across edge cases.
def logreg_space(trial):
return {
"maxIter": trial.suggest_int("maxIter", 50, 200),
"regParam": trial.suggest_float("regParam", 1e-6, 1.0, log=True),
}
def linreg_space(trial):
return {
"maxIter": trial.suggest_int("maxIter", 50, 200),
"regParam": trial.suggest_float("regParam", 1e-6, 1.0, log=True),
}
def make_models():
"""Fresh model instances (important: don't reuse fitted estimators)."""
return {
"ml_m": LogisticRegression(),
"ml_g0": LinearRegression(),
"ml_g1": LinearRegression(),
}
2 · Partial Tuning¶
Scenario: You want to tune only the outcome models (ml_g0, ml_g1) while keeping the propensity model (ml_m) at its default parameters.
How: Simply omit the learner key from ml_param_space.
dml = DoubleMLIRM(data_schema, data_field_map)
best_models, results = dml.tune_ml_models(
spark,
data_split,
make_models(),
ml_param_space={
"ml_g0": linreg_space,
"ml_g1": linreg_space,
# ml_m intentionally omitted → uses defaults
},
optuna_settings={"n_trials": 5},
return_tune_res=True,
)
for name, r in results.items():
print(f" {name}: tuned={r.tuned}")
assert not results["ml_m"].tuned
assert results["ml_g0"].tuned and results["ml_g1"].tuned
print("\n✓ Only outcome models were tuned.")
Optuna study direction set to 'minimize' for learner 'ml_g0'.
Optuna study direction set to 'minimize' for learner 'ml_g1'.
ml_m: tuned=False ml_g0: tuned=True ml_g1: tuned=True ✓ Only outcome models were tuned.
3 · No-Tuning Baseline¶
Scenario: You want a quick baseline with default parameters — no Optuna optimization at all.
How: Pass an empty dict {} as ml_param_space.
models = make_models()
dml = DoubleMLIRM(data_schema, data_field_map)
baseline, results = dml.tune_ml_models(
spark,
data_split,
models,
ml_param_space={}, # ← empty
optuna_settings={"n_trials": 5},
return_tune_res=True,
)
for name, r in results.items():
print(f" {name}: tuned={r.tuned}, study={r.study}")
assert all(not r.tuned for r in results.values())
assert all(r.study is None for r in results.values())
assert baseline == models # same objects returned
print("\n✓ No tuning performed — original models returned unchanged.")
ml_m: tuned=False, study=None ml_g0: tuned=False, study=None ml_g1: tuned=False, study=None ✓ No tuning performed — original models returned unchanged.
4 · Explicit None Skipping¶
Scenario: Same effect as omitting a learner from the dict, but more explicit in code — useful when you want to make it obvious that skipping was intentional.
How: Set the learner's value to None.
dml = DoubleMLIRM(data_schema, data_field_map)
best_models, results = dml.tune_ml_models(
spark,
data_split,
make_models(),
ml_param_space={
"ml_m": None, # ← explicitly skip
"ml_g0": linreg_space,
"ml_g1": linreg_space,
},
optuna_settings={"n_trials": 5},
return_tune_res=True,
)
for name, r in results.items():
print(f" {name}: tuned={r.tuned}")
assert not results["ml_m"].tuned
assert results["ml_g0"].tuned and results["ml_g1"].tuned
print("\n✓ Explicit None correctly skips ml_m.")
Optuna study direction set to 'minimize' for learner 'ml_g0'.
Optuna study direction set to 'minimize' for learner 'ml_g1'.
ml_m: tuned=False ml_g0: tuned=True ml_g1: tuned=True ✓ Explicit None correctly skips ml_m.
5 · Model-Specific Optuna Overrides¶
Scenario: Different learners need different optimization strategies:
- The propensity model (
ml_m) is a classifier → maximizeareaUnderROCwith more trials. - The outcome models are regressors → minimize
mae/ maximizer2with fewer trials.
How: Nest learner-specific dicts inside optuna_settings.
dml = DoubleMLIRM(data_schema, data_field_map)
best_models, results = dml.tune_ml_models(
spark,
data_split,
make_models(),
ml_param_space={
"ml_m": logreg_space,
"ml_g0": linreg_space,
"ml_g1": linreg_space,
},
scoring_methods={
"ml_m": "areaUnderROC",
"ml_g0": "mae",
"ml_g1": "r2",
},
optuna_settings={
# Global defaults
"n_trials": 3,
"direction": "minimize",
# Per-learner overrides
"ml_m": {"n_trials": 8, "direction": "maximize"},
"ml_g0": {"n_trials": 4},
"ml_g1": {"n_trials": 5},
},
return_tune_res=True,
)
for name, r in results.items():
n = len(r.study.trials)
print(f" {name}: trials={n}, scoring={r.scoring_method}, best={r.best_score:.4f}")
assert len(results["ml_m"].study.trials) == 8
assert len(results["ml_g0"].study.trials) == 4
assert len(results["ml_g1"].study.trials) == 5
assert results["ml_m"].scoring_method == "areaUnderROC"
print("\n✓ Each learner used its own trial count, direction, and scoring metric.")
Optuna study direction set to 'maximize' for learner 'ml_m'.
Optuna study direction set to 'minimize' for learner 'ml_g0'.
Optuna study direction set to 'minimize' for learner 'ml_g1'.
ml_m: trials=8, scoring=areaUnderROC, best=0.7593 ml_g0: trials=4, scoring=mae, best=0.6146 ml_g1: trials=5, scoring=r2, best=0.6901 ✓ Each learner used its own trial count, direction, and scoring metric.
6 · Memory-Constrained Sequential Tuning¶
Scenario: Limited memory or compute. Tuning all learners at once may be too expensive.
How: Tune one learner at a time in separate tune_ml_models calls with conservative settings (gc_after_trial=True, n_jobs_optuna=1, fewer trials).
mem_settings = {
"n_trials": 3,
"gc_after_trial": True,
"n_jobs_optuna": 1,
}
def minimal_space(trial):
return {
"maxIter": trial.suggest_int("maxIter", 50, 150),
"regParam": trial.suggest_float("regParam", 1e-6, 0.1, log=True),
}
combined_best = {}
models = make_models()
for learner in ["ml_m", "ml_g0", "ml_g1"]:
print(f" Tuning {learner} ...")
dml = DoubleMLIRM(data_schema, data_field_map)
best, _ = dml.tune_ml_models(
spark,
data_split,
models,
ml_param_space={learner: minimal_space},
optuna_settings=mem_settings,
return_tune_res=True,
)
combined_best[learner] = best[learner]
print(f"\n✓ Sequential tuning complete. Lower peak memory at the cost of wall time.")
print(f" Combined best models: {list(combined_best.keys())}")
Tuning ml_m ... Optuna study direction set to 'minimize' for learner 'ml_m'.
Tuning ml_g0 ... Optuna study direction set to 'minimize' for learner 'ml_g0'. Tuning ml_g1 ... Optuna study direction set to 'minimize' for learner 'ml_g1'.
✓ Sequential tuning complete. Lower peak memory at the cost of wall time. Combined best models: ['ml_m', 'ml_g0', 'ml_g1']
7 · Default Settings Fallback (optuna_settings=None)¶
Scenario: You want tuning but don't care about custom Optuna configuration.
How: Pass optuna_settings=None (or omit it). The library applies sensible defaults (n_trials=5, direction="minimize", etc.).
dml = DoubleMLIRM(data_schema, data_field_map)
best_models, results = dml.tune_ml_models(
spark,
data_split,
make_models(),
ml_param_space={
"ml_m": logreg_space,
"ml_g0": linreg_space,
"ml_g1": linreg_space,
},
optuna_settings=None, # ← defaults
return_tune_res=True,
)
for name, r in results.items():
print(f" {name}: tuned={r.tuned}, trials={len(r.study.trials)}")
assert all(r.tuned for r in results.values())
print("\n✓ All learners tuned with library defaults (n_trials=5).")
Optuna study direction set to 'minimize' for learner 'ml_m'.
Optuna study direction set to 'minimize' for learner 'ml_g0'.
Optuna study direction set to 'minimize' for learner 'ml_g1'.
ml_m: tuned=True, trials=5 ml_g0: tuned=True, trials=5 ml_g1: tuned=True, trials=5 ✓ All learners tuned with library defaults (n_trials=5).
Summary¶
| Edge Case | Pattern | Key Takeaway |
|---|---|---|
| Partial tuning | Omit learner from ml_param_space |
Focus budget on critical models |
| No-tuning baseline | ml_param_space={} |
Quick comparison benchmark |
Explicit None |
"ml_m": None |
Self-documenting skip |
| Per-learner overrides | Nested dicts in optuna_settings |
Different trials / direction / scoring per learner |
| Memory-constrained | Tune one learner per call | Lower peak memory |
| Default fallback | optuna_settings=None |
Library defaults (n_trials=5) |
spark.stop()