DoubleMLPLR¶
doubleml_pyspark.plm.plr.DoubleMLPLR
¶
Bases: DoubleMLScalar
DoubleML class for partially linear regression with PySpark.
This class implements the Double Machine Learning (DML) estimator for partially linear regression (PLR) models using PySpark DataFrames.
Methods:
| Name | Description |
|---|---|
cate |
CATE estimation. |
confint |
Confidence intervals for DoubleML models. |
estimate_causal_parameters |
Estimate the causal parameters on a single repetition. |
fit |
Fit the DoubleML model. |
fit_ml_models |
Train and cross-fit the nuisance ML models and collect predictions. |
gate |
GATE estimation (Group Average Treatment Effects). |
sensitivity_analysis |
Sensitivity analysis. |
Attributes:
| Name | Type | Description |
|---|---|---|
additional_nuisance_elements |
dict[str, Any]
|
Additional, estimator-specific low-dimensional nuisance elements. |
all_coef |
ndarray
|
Estimates for each repetition, shape (n_thetas, n_rep). Same as :attr: |
all_se |
ndarray
|
Standard errors for each repetition, shape (n_thetas, n_rep). Same as :attr: |
all_ses |
ndarray
|
All standard errors across repetitions and folds. |
all_thetas |
ndarray
|
All estimated parameters across repetitions and folds. |
coef |
ndarray
|
Estimated causal parameter(s), shape (n_thetas,). Same as :attr: |
covariates |
list[str]
|
The covariates. |
data_field_map |
dict[str, str | list[str]]
|
The data field map. |
data_schema |
StructType
|
The schema of the input data. |
id_col |
str
|
The id column. |
n_obs |
int
|
The number of observations used for estimation. |
n_rep |
int
|
The number of repetitions for cross-validation. |
outcome |
str
|
The outcome variable. |
p_val |
ndarray
|
The p-values. |
pval |
ndarray
|
p-value(s), shape (n_thetas,). Same as :attr: |
score_elements |
list[Any]
|
The score elements. |
se |
ndarray
|
Standard error(s), shape (n_thetas,). Same as :attr: |
sensitivity_params |
dict[str, Any] | None
|
The sensitivity parameters. |
sensitivity_summary |
str
|
Summary for the sensitivity analysis after calling sensitivity_analysis. |
sensitivity_var_elements |
dict[str, ndarray] | None
|
The sensitivity variance elements. |
ses |
ndarray
|
The standard errors. |
summary |
DataFrame
|
Summary of the estimated causal parameter(s) after calling |
t_stat |
ndarray
|
The t-statistics. |
thetas |
ndarray
|
The estimated parameters. |
treatment |
str
|
The treatment variable. |
var_scaling |
ndarray
|
The variance scaling factors. |
Attributes¶
doubleml_pyspark.plm.plr.DoubleMLPLR.additional_nuisance_elements
property
writable
¶
Additional, estimator-specific low-dimensional nuisance elements.
This dictionary stores extra quantities required by a concrete estimator beyond the standard nuisances.
Notes
- For some estimators this may be empty (e.g., IRM with ATE).
- For others it may contain specific entries (e.g., IRM with ATT requires {'subgroup_prob': float in (0, 1]}).
- When set via the property setter, inputs are merged with existing values, validated, and sanitized to keep only accepted keys; missing required keys raise ValueError.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
The currently stored, validated additional nuisance elements, or None if they have not been estimated/set yet. |
doubleml_pyspark.plm.plr.DoubleMLPLR.all_coef
property
¶
Estimates for each repetition, shape (n_thetas, n_rep). Same as :attr:all_thetas.
doubleml_pyspark.plm.plr.DoubleMLPLR.all_se
property
¶
Standard errors for each repetition, shape (n_thetas, n_rep). Same as :attr:all_ses.
doubleml_pyspark.plm.plr.DoubleMLPLR.all_ses
property
¶
All standard errors across repetitions and folds.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas, n_rep) with all standard errors. |
doubleml_pyspark.plm.plr.DoubleMLPLR.all_thetas
property
¶
All estimated parameters across repetitions and folds.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas, n_rep) with all estimated parameters. |
doubleml_pyspark.plm.plr.DoubleMLPLR.coef
property
¶
Estimated causal parameter(s), shape (n_thetas,). Same as :attr:thetas.
doubleml_pyspark.plm.plr.DoubleMLPLR.covariates
property
¶
The covariates.
Returns:
| Type | Description |
|---|---|
list of str or None
|
The list of covariate names, or None if not set. |
doubleml_pyspark.plm.plr.DoubleMLPLR.data_field_map
property
¶
The data field map.
Returns:
| Type | Description |
|---|---|
dict of str
|
The mapping of field names. |
doubleml_pyspark.plm.plr.DoubleMLPLR.data_schema
property
¶
The schema of the input data.
Returns:
| Type | Description |
|---|---|
StructType
|
The schema of the input data. |
doubleml_pyspark.plm.plr.DoubleMLPLR.id_col
property
¶
The id column.
Returns:
| Type | Description |
|---|---|
str
|
The name of the id column. |
doubleml_pyspark.plm.plr.DoubleMLPLR.n_obs
property
¶
The number of observations used for estimation.
Returns:
| Type | Description |
|---|---|
int or None
|
The number of observations, or None if not set. |
doubleml_pyspark.plm.plr.DoubleMLPLR.n_rep
property
¶
The number of repetitions for cross-validation.
Returns:
| Type | Description |
|---|---|
int
|
The number of repetitions. |
doubleml_pyspark.plm.plr.DoubleMLPLR.outcome
property
¶
The outcome variable.
Returns:
| Type | Description |
|---|---|
str
|
The name of the outcome variable. |
doubleml_pyspark.plm.plr.DoubleMLPLR.p_val
property
¶
The p-values.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas,) with p-values. |
doubleml_pyspark.plm.plr.DoubleMLPLR.pval
property
¶
p-value(s), shape (n_thetas,). Same as :attr:p_val.
doubleml_pyspark.plm.plr.DoubleMLPLR.score_elements
property
¶
The score elements.
Returns:
| Type | Description |
|---|---|
Any
|
The score elements. |
doubleml_pyspark.plm.plr.DoubleMLPLR.se
property
¶
Standard error(s), shape (n_thetas,). Same as :attr:ses.
doubleml_pyspark.plm.plr.DoubleMLPLR.sensitivity_params
property
¶
The sensitivity parameters.
Returns:
| Type | Description |
|---|---|
dict or None
|
The sensitivity parameters, or None if not set. |
doubleml_pyspark.plm.plr.DoubleMLPLR.sensitivity_summary
property
¶
Summary for the sensitivity analysis after calling sensitivity_analysis.
Returns:
| Type | Description |
|---|---|
str
|
Summary for the sensitivity analysis. |
doubleml_pyspark.plm.plr.DoubleMLPLR.sensitivity_var_elements
property
¶
The sensitivity variance elements.
Returns:
| Type | Description |
|---|---|
dict or None
|
Each element is an array of shape (n_thetas, n_rep), or None if not set. |
doubleml_pyspark.plm.plr.DoubleMLPLR.ses
property
¶
The standard errors.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas,) with standard errors. |
doubleml_pyspark.plm.plr.DoubleMLPLR.summary
property
¶
Summary of the estimated causal parameter(s) after calling fit().
Returns:
| Type | Description |
|---|---|
DataFrame
|
Columns |
doubleml_pyspark.plm.plr.DoubleMLPLR.t_stat
property
¶
The t-statistics.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas,) with t-statistics. |
doubleml_pyspark.plm.plr.DoubleMLPLR.thetas
property
¶
The estimated parameters.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas,) with estimated parameters. |
doubleml_pyspark.plm.plr.DoubleMLPLR.treatment
property
¶
The treatment variable.
Returns:
| Type | Description |
|---|---|
str
|
The name of the treatment variable. |
doubleml_pyspark.plm.plr.DoubleMLPLR.var_scaling
property
¶
The variance scaling factors.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas, n_rep) with variance scaling factors. |
Functions¶
doubleml_pyspark.plm.plr.DoubleMLPLR.cate
¶
cate(spark_session: SparkSessionType, df_basis: DataFrame, df_nuisance: DataFrame, cov_type: str = 'HC0', num_partitions: int | None = None, is_gate: bool = False, use_persist: bool = False) -> DoubleMLBLP
CATE estimation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_session
|
SparkSession or SparkSession
|
The active Spark session (supports both local and connect modes). |
required |
df_basis
|
DataFrame
|
DataFrame with basis functions. |
required |
df_nuisance
|
DataFrame
|
DataFrame with nuisance parameter predictions. |
required |
cov_type
|
str
|
Covariance estimator used for the BLP. Supported values are
|
'HC0'
|
num_partitions
|
int
|
Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration. |
None
|
use_persist
|
bool
|
Whether to cache intermediate DataFrames. Not supported on Databricks serverless compute. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
DoubleMLBLP
|
The fitted BLP model for CATE estimation. |
doubleml_pyspark.plm.plr.DoubleMLPLR.confint
¶
Confidence intervals for DoubleML models.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
joint
|
bool
|
Indicates whether joint confidence intervals are computed. Default is False. |
False
|
level
|
float
|
The confidence level. Default is 0.95. |
0.95
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
A data frame with the confidence interval(s). |
Raises:
| Type | Description |
|---|---|
TypeError
|
If input types are incorrect. |
ValueError
|
If confidence level is not in (0, 1). |
NotImplementedError
|
If joint confidence intervals are requested. |
doubleml_pyspark.plm.plr.DoubleMLPLR.estimate_causal_parameters
¶
estimate_causal_parameters(spark_session: SparkSessionType, df_nuisance: DataFrame, additional_nuisance_elements: dict[str, Any] | None = None, num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_checkpoint: bool = False, use_persist: bool = False) -> DoubleMLScalar
Estimate the causal parameters on a single repetition.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_session
|
SparkSession or SparkSession
|
The active Spark session (supports both local and connect modes). |
required |
df_nuisance
|
DataFrame
|
DataFrame with nuisance parameter predictions. |
required |
additional_nuisance_elements
|
dict of str to Any
|
Additional nuisance elements. |
None
|
num_partitions
|
int
|
Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration. |
None
|
storage_level
|
StorageLevel
|
Storage level for caching intermediate DataFrames. |
MEMORY_AND_DISK_DESER
|
use_checkpoint
|
bool
|
Whether to use checkpointing. |
False
|
use_persist
|
bool
|
Whether to persist DataFrames for caching. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
DoubleMLScalar
|
The fitted estimator. |
doubleml_pyspark.plm.plr.DoubleMLPLR.fit
¶
fit(spark_session: SparkSessionType, kfold_split: list[tuple[DataFrame, DataFrame]], ml_models: dict[str, Any], num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_checkpoint: bool = False, use_persist: bool = False) -> DoubleMLScalar
Fit the DoubleML model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_session
|
SparkSession or SparkSession
|
The active Spark session (supports both local and connect modes). |
required |
kfold_split
|
list of tuples of (DataFrame, DataFrame)
|
List of (train, test) splits for cross-fitting. |
required |
ml_models
|
dict of str to Any
|
Dictionary of ML models. |
required |
num_partitions
|
int
|
Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration. |
None
|
storage_level
|
StorageLevel
|
Storage level for caching intermediate DataFrames. |
MEMORY_AND_DISK_DESER
|
use_checkpoint
|
bool
|
Whether to use checkpointing df_nuisance and df_psi. |
False
|
use_persist
|
bool
|
Whether to persist DataFrames for caching. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
DoubleMLScalar
|
The fitted estimator. |
doubleml_pyspark.plm.plr.DoubleMLPLR.fit_ml_models
¶
fit_ml_models(spark_session: SparkSessionType, kfold_split: list[tuple[DataFrame, DataFrame]], ml_models: dict[str, Any], num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_persist: bool = False) -> tuple[DataFrame, dict[str, Any]]
Train and cross-fit the nuisance ML models and collect predictions.
This runs each estimator on the provided k-fold splits (train, test), unions the fold-wise predictions into a single nuisance DataFrame conforming to _get_df_nuisance_schema(), and estimates any additional low-dimensional nuisance elements required by the concrete estimator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_session
|
SparkSession or SparkSession
|
The active Spark session (supports both local and connect modes). |
required |
kfold_split
|
list[tuple[DataFrame, DataFrame]]
|
Sequence of (train_df, test_df) splits used for cross-fitting. Each DataFrame must contain the columns specified in data_field_map. |
required |
ml_models
|
dict[str, Any]
|
Mapping from required nuisance model names to Spark Estimators as specified by _get_ml_models_structure(). |
required |
num_partitions
|
int
|
Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration. |
None
|
storage_level
|
StorageLevel
|
Storage level for caching intermediate DataFrames. |
MEMORY_AND_DISK_DESER
|
use_persist
|
bool
|
Whether to persist DataFrames for caching. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
tuple[DataFrame, dict[str, Any]]
|
|
Notes
- This method does not modify internal state beyond logging.
doubleml_pyspark.plm.plr.DoubleMLPLR.gate
¶
gate(spark_session: SparkSessionType, df_groups: DataFrame, df_nuisance: DataFrame, cov_type: str = 'HC0', num_partitions: int | None = None, use_persist: bool = False) -> DoubleMLBLP
GATE estimation (Group Average Treatment Effects).
The group variant of :meth:cate: builds a mutually exclusive dummy
(one-hot) basis from df_groups and fits the best linear predictor on
it with is_gate=True, so that every coefficient is the average
treatment effect within a group. Like :meth:cate, it works for any
model that implements :meth:_get_blp_orth_signal_and_basis (e.g. PLR
and IRM), so it lives on the shared base class rather than per model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_session
|
SparkSession or SparkSession
|
The active Spark session (supports both local and connect modes). |
required |
df_groups
|
DataFrame
|
Group membership, including the id column (
|
required |
df_nuisance
|
DataFrame
|
DataFrame with nuisance parameter predictions. |
required |
cov_type
|
str
|
Covariance estimator used for the BLP. Supported values are
|
'HC0'
|
num_partitions
|
int
|
Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration. |
None
|
use_persist
|
bool
|
Whether to cache intermediate DataFrames. Not supported on Databricks serverless compute. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
DoubleMLBLP
|
The fitted BLP model for GATE estimation (with |
doubleml_pyspark.plm.plr.DoubleMLPLR.sensitivity_analysis
¶
sensitivity_analysis(cf_y: float = 0.03, cf_d: float = 0.03, rho: float = 1.0, level: float = 0.95, null_hypothesis: float = 0.0) -> DoubleMLBase
Sensitivity analysis.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cf_y
|
float
|
Confounding strength for the outcome. Default is 0.03. |
0.03
|
cf_d
|
float
|
Confounding strength for the treatment. Default is 0.03. |
0.03
|
rho
|
float
|
Correlation parameter. Default is 1.0. |
1.0
|
level
|
float
|
Confidence level. Default is 0.95. |
0.95
|
null_hypothesis
|
float
|
Value for the null hypothesis. Default is 0.0. |
0.0
|
Returns:
| Type | Description |
|---|---|
DoubleMLBase
|
The fitted instance. |
tune_ml_models¶
Tune hyperparameters for the PLR nuisance models using Optuna. DoubleMLPLR
uses two models: ml_m (treatment model) and ml_l (outcome model). See the
Hyperparameter Tuning API for the full parameter reference.
from pyspark.ml.regression import RandomForestRegressor
from doubleml_pyspark.utils import kfold_split
# PLR uses ml_m (treatment) and ml_l (outcome)
ml_models = {
"ml_m": RandomForestRegressor(), # treatment model
"ml_l": RandomForestRegressor(), # outcome model
}
ml_param_space = {
"ml_m": lambda trial: {"maxDepth": trial.suggest_int("maxDepth", 3, 10)},
"ml_l": lambda trial: {"maxDepth": trial.suggest_int("maxDepth", 3, 10)},
}
splits = kfold_split(df, n_folds=3, n_rep=1, shuffle=True, seed=42)
ml_models = dml.tune_ml_models(
spark_session=spark,
data_split=splits[0][0],
ml_models=ml_models,
ml_param_space=ml_param_space,
)