Skip to content

DoubleMLPLR

doubleml_pyspark.plm.plr.DoubleMLPLR

Bases: DoubleMLScalar

DoubleML class for partially linear regression with PySpark.

This class implements the Double Machine Learning (DML) estimator for partially linear regression (PLR) models using PySpark DataFrames.

Methods:

Name Description
cate

CATE estimation.

confint

Confidence intervals for DoubleML models.

estimate_causal_parameters

Estimate the causal parameters on a single repetition.

fit

Fit the DoubleML model.

fit_ml_models

Train and cross-fit the nuisance ML models and collect predictions.

gate

GATE estimation (Group Average Treatment Effects).

sensitivity_analysis

Sensitivity analysis.

Attributes:

Name Type Description
additional_nuisance_elements dict[str, Any]

Additional, estimator-specific low-dimensional nuisance elements.

all_coef ndarray

Estimates for each repetition, shape (n_thetas, n_rep). Same as :attr:all_thetas.

all_se ndarray

Standard errors for each repetition, shape (n_thetas, n_rep). Same as :attr:all_ses.

all_ses ndarray

All standard errors across repetitions and folds.

all_thetas ndarray

All estimated parameters across repetitions and folds.

coef ndarray

Estimated causal parameter(s), shape (n_thetas,). Same as :attr:thetas.

covariates list[str]

The covariates.

data_field_map dict[str, str | list[str]]

The data field map.

data_schema StructType

The schema of the input data.

id_col str

The id column.

n_obs int

The number of observations used for estimation.

n_rep int

The number of repetitions for cross-validation.

outcome str

The outcome variable.

p_val ndarray

The p-values.

pval ndarray

p-value(s), shape (n_thetas,). Same as :attr:p_val.

score_elements list[Any]

The score elements.

se ndarray

Standard error(s), shape (n_thetas,). Same as :attr:ses.

sensitivity_params dict[str, Any] | None

The sensitivity parameters.

sensitivity_summary str

Summary for the sensitivity analysis after calling sensitivity_analysis.

sensitivity_var_elements dict[str, ndarray] | None

The sensitivity variance elements.

ses ndarray

The standard errors.

summary DataFrame

Summary of the estimated causal parameter(s) after calling fit().

t_stat ndarray

The t-statistics.

thetas ndarray

The estimated parameters.

treatment str

The treatment variable.

var_scaling ndarray

The variance scaling factors.

Attributes

doubleml_pyspark.plm.plr.DoubleMLPLR.additional_nuisance_elements property writable

additional_nuisance_elements: dict[str, Any]

Additional, estimator-specific low-dimensional nuisance elements.

This dictionary stores extra quantities required by a concrete estimator beyond the standard nuisances.

Notes
  • For some estimators this may be empty (e.g., IRM with ATE).
  • For others it may contain specific entries (e.g., IRM with ATT requires {'subgroup_prob': float in (0, 1]}).
  • When set via the property setter, inputs are merged with existing values, validated, and sanitized to keep only accepted keys; missing required keys raise ValueError.

Returns:

Type Description
dict[str, Any]

The currently stored, validated additional nuisance elements, or None if they have not been estimated/set yet.

doubleml_pyspark.plm.plr.DoubleMLPLR.all_coef property

all_coef: ndarray

Estimates for each repetition, shape (n_thetas, n_rep). Same as :attr:all_thetas.

doubleml_pyspark.plm.plr.DoubleMLPLR.all_se property

all_se: ndarray

Standard errors for each repetition, shape (n_thetas, n_rep). Same as :attr:all_ses.

doubleml_pyspark.plm.plr.DoubleMLPLR.all_ses property

all_ses: ndarray

All standard errors across repetitions and folds.

Returns:

Type Description
ndarray

Array of shape (n_thetas, n_rep) with all standard errors.

doubleml_pyspark.plm.plr.DoubleMLPLR.all_thetas property

all_thetas: ndarray

All estimated parameters across repetitions and folds.

Returns:

Type Description
ndarray

Array of shape (n_thetas, n_rep) with all estimated parameters.

doubleml_pyspark.plm.plr.DoubleMLPLR.coef property

coef: ndarray

Estimated causal parameter(s), shape (n_thetas,). Same as :attr:thetas.

doubleml_pyspark.plm.plr.DoubleMLPLR.covariates property

covariates: list[str]

The covariates.

Returns:

Type Description
list of str or None

The list of covariate names, or None if not set.

doubleml_pyspark.plm.plr.DoubleMLPLR.data_field_map property

data_field_map: dict[str, str | list[str]]

The data field map.

Returns:

Type Description
dict of str

The mapping of field names.

doubleml_pyspark.plm.plr.DoubleMLPLR.data_schema property

data_schema: StructType

The schema of the input data.

Returns:

Type Description
StructType

The schema of the input data.

doubleml_pyspark.plm.plr.DoubleMLPLR.id_col property

id_col: str

The id column.

Returns:

Type Description
str

The name of the id column.

doubleml_pyspark.plm.plr.DoubleMLPLR.n_obs property

n_obs: int

The number of observations used for estimation.

Returns:

Type Description
int or None

The number of observations, or None if not set.

doubleml_pyspark.plm.plr.DoubleMLPLR.n_rep property

n_rep: int

The number of repetitions for cross-validation.

Returns:

Type Description
int

The number of repetitions.

doubleml_pyspark.plm.plr.DoubleMLPLR.outcome property

outcome: str

The outcome variable.

Returns:

Type Description
str

The name of the outcome variable.

doubleml_pyspark.plm.plr.DoubleMLPLR.p_val property

p_val: ndarray

The p-values.

Returns:

Type Description
ndarray

Array of shape (n_thetas,) with p-values.

doubleml_pyspark.plm.plr.DoubleMLPLR.pval property

pval: ndarray

p-value(s), shape (n_thetas,). Same as :attr:p_val.

doubleml_pyspark.plm.plr.DoubleMLPLR.score_elements property

score_elements: list[Any]

The score elements.

Returns:

Type Description
Any

The score elements.

doubleml_pyspark.plm.plr.DoubleMLPLR.se property

se: ndarray

Standard error(s), shape (n_thetas,). Same as :attr:ses.

doubleml_pyspark.plm.plr.DoubleMLPLR.sensitivity_params property

sensitivity_params: dict[str, Any] | None

The sensitivity parameters.

Returns:

Type Description
dict or None

The sensitivity parameters, or None if not set.

doubleml_pyspark.plm.plr.DoubleMLPLR.sensitivity_summary property

sensitivity_summary: str

Summary for the sensitivity analysis after calling sensitivity_analysis.

Returns:

Type Description
str

Summary for the sensitivity analysis.

doubleml_pyspark.plm.plr.DoubleMLPLR.sensitivity_var_elements property

sensitivity_var_elements: dict[str, ndarray] | None

The sensitivity variance elements.

Returns:

Type Description
dict or None

Each element is an array of shape (n_thetas, n_rep), or None if not set.

doubleml_pyspark.plm.plr.DoubleMLPLR.ses property

ses: ndarray

The standard errors.

Returns:

Type Description
ndarray

Array of shape (n_thetas,) with standard errors.

doubleml_pyspark.plm.plr.DoubleMLPLR.summary property

summary: DataFrame

Summary of the estimated causal parameter(s) after calling fit().

Returns:

Type Description
DataFrame

Columns coef, std err, t, P>|t| and the 95 % confidence interval, one row per parameter. Empty before fit().

doubleml_pyspark.plm.plr.DoubleMLPLR.t_stat property

t_stat: ndarray

The t-statistics.

Returns:

Type Description
ndarray

Array of shape (n_thetas,) with t-statistics.

doubleml_pyspark.plm.plr.DoubleMLPLR.thetas property

thetas: ndarray

The estimated parameters.

Returns:

Type Description
ndarray

Array of shape (n_thetas,) with estimated parameters.

doubleml_pyspark.plm.plr.DoubleMLPLR.treatment property

treatment: str

The treatment variable.

Returns:

Type Description
str

The name of the treatment variable.

doubleml_pyspark.plm.plr.DoubleMLPLR.var_scaling property

var_scaling: ndarray

The variance scaling factors.

Returns:

Type Description
ndarray

Array of shape (n_thetas, n_rep) with variance scaling factors.

Functions

doubleml_pyspark.plm.plr.DoubleMLPLR.cate

cate(spark_session: SparkSessionType, df_basis: DataFrame, df_nuisance: DataFrame, cov_type: str = 'HC0', num_partitions: int | None = None, is_gate: bool = False, use_persist: bool = False) -> DoubleMLBLP

CATE estimation.

Parameters:

Name Type Description Default
spark_session SparkSession or SparkSession

The active Spark session (supports both local and connect modes).

required
df_basis DataFrame

DataFrame with basis functions.

required
df_nuisance DataFrame

DataFrame with nuisance parameter predictions.

required
cov_type str

Covariance estimator used for the BLP. Supported values are 'nonrobust', 'HC0', 'HC1', 'HC2', and 'HC3'.

'HC0'
num_partitions int

Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration.

None
use_persist bool

Whether to cache intermediate DataFrames. Not supported on Databricks serverless compute. Defaults to False.

False

Returns:

Type Description
DoubleMLBLP

The fitted BLP model for CATE estimation.

doubleml_pyspark.plm.plr.DoubleMLPLR.confint

confint(joint: bool = False, level: float = 0.95) -> pd.DataFrame

Confidence intervals for DoubleML models.

Parameters:

Name Type Description Default
joint bool

Indicates whether joint confidence intervals are computed. Default is False.

False
level float

The confidence level. Default is 0.95.

0.95

Returns:

Type Description
DataFrame

A data frame with the confidence interval(s).

Raises:

Type Description
TypeError

If input types are incorrect.

ValueError

If confidence level is not in (0, 1).

NotImplementedError

If joint confidence intervals are requested.

doubleml_pyspark.plm.plr.DoubleMLPLR.estimate_causal_parameters

estimate_causal_parameters(spark_session: SparkSessionType, df_nuisance: DataFrame, additional_nuisance_elements: dict[str, Any] | None = None, num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_checkpoint: bool = False, use_persist: bool = False) -> DoubleMLScalar

Estimate the causal parameters on a single repetition.

Parameters:

Name Type Description Default
spark_session SparkSession or SparkSession

The active Spark session (supports both local and connect modes).

required
df_nuisance DataFrame

DataFrame with nuisance parameter predictions.

required
additional_nuisance_elements dict of str to Any

Additional nuisance elements.

None
num_partitions int

Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration.

None
storage_level StorageLevel

Storage level for caching intermediate DataFrames.

MEMORY_AND_DISK_DESER
use_checkpoint bool

Whether to use checkpointing.

False
use_persist bool

Whether to persist DataFrames for caching. Defaults to False.

False

Returns:

Type Description
DoubleMLScalar

The fitted estimator.

doubleml_pyspark.plm.plr.DoubleMLPLR.fit

fit(spark_session: SparkSessionType, kfold_split: list[tuple[DataFrame, DataFrame]], ml_models: dict[str, Any], num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_checkpoint: bool = False, use_persist: bool = False) -> DoubleMLScalar

Fit the DoubleML model.

Parameters:

Name Type Description Default
spark_session SparkSession or SparkSession

The active Spark session (supports both local and connect modes).

required
kfold_split list of tuples of (DataFrame, DataFrame)

List of (train, test) splits for cross-fitting.

required
ml_models dict of str to Any

Dictionary of ML models.

required
num_partitions int

Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration.

None
storage_level StorageLevel

Storage level for caching intermediate DataFrames.

MEMORY_AND_DISK_DESER
use_checkpoint bool

Whether to use checkpointing df_nuisance and df_psi.

False
use_persist bool

Whether to persist DataFrames for caching. Defaults to False.

False

Returns:

Type Description
DoubleMLScalar

The fitted estimator.

doubleml_pyspark.plm.plr.DoubleMLPLR.fit_ml_models

fit_ml_models(spark_session: SparkSessionType, kfold_split: list[tuple[DataFrame, DataFrame]], ml_models: dict[str, Any], num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_persist: bool = False) -> tuple[DataFrame, dict[str, Any]]

Train and cross-fit the nuisance ML models and collect predictions.

This runs each estimator on the provided k-fold splits (train, test), unions the fold-wise predictions into a single nuisance DataFrame conforming to _get_df_nuisance_schema(), and estimates any additional low-dimensional nuisance elements required by the concrete estimator.

Parameters:

Name Type Description Default
spark_session SparkSession or SparkSession

The active Spark session (supports both local and connect modes).

required
kfold_split list[tuple[DataFrame, DataFrame]]

Sequence of (train_df, test_df) splits used for cross-fitting. Each DataFrame must contain the columns specified in data_field_map.

required
ml_models dict[str, Any]

Mapping from required nuisance model names to Spark Estimators as specified by _get_ml_models_structure().

required
num_partitions int

Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration.

None
storage_level StorageLevel

Storage level for caching intermediate DataFrames.

MEMORY_AND_DISK_DESER
use_persist bool

Whether to persist DataFrames for caching. Defaults to False.

False

Returns:

Type Description
tuple[DataFrame, dict[str, Any]]
  • df_nuisance: DataFrame with per-row nuisance predictions; schema is given by _get_df_nuisance_schema() and includes the id_column.
  • additional_nuisance_elements: Dictionary with any extra low-dimensional quantities needed by the estimator; may be empty ({}).
Notes
  • This method does not modify internal state beyond logging.

doubleml_pyspark.plm.plr.DoubleMLPLR.gate

gate(spark_session: SparkSessionType, df_groups: DataFrame, df_nuisance: DataFrame, cov_type: str = 'HC0', num_partitions: int | None = None, use_persist: bool = False) -> DoubleMLBLP

GATE estimation (Group Average Treatment Effects).

The group variant of :meth:cate: builds a mutually exclusive dummy (one-hot) basis from df_groups and fits the best linear predictor on it with is_gate=True, so that every coefficient is the average treatment effect within a group. Like :meth:cate, it works for any model that implements :meth:_get_blp_orth_signal_and_basis (e.g. PLR and IRM), so it lives on the shared base class rather than per model.

Parameters:

Name Type Description Default
spark_session SparkSession or SparkSession

The active Spark session (supports both local and connect modes).

required
df_groups DataFrame

Group membership, including the id column (self.id_col). Either

  • a single group-label column (string/numeric) that is one-hot encoded into mutually exclusive Group_<value> dummy columns; or
  • several already dummy-coded columns (boolean or integer 0/1), one per group.
required
df_nuisance DataFrame

DataFrame with nuisance parameter predictions.

required
cov_type str

Covariance estimator used for the BLP. Supported values are 'nonrobust', 'HC0', 'HC1', 'HC2', and 'HC3'.

'HC0'
num_partitions int

Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration.

None
use_persist bool

Whether to cache intermediate DataFrames. Not supported on Databricks serverless compute. Defaults to False.

False

Returns:

Type Description
DoubleMLBLP

The fitted BLP model for GATE estimation (with is_gate=True).

doubleml_pyspark.plm.plr.DoubleMLPLR.sensitivity_analysis

sensitivity_analysis(cf_y: float = 0.03, cf_d: float = 0.03, rho: float = 1.0, level: float = 0.95, null_hypothesis: float = 0.0) -> DoubleMLBase

Sensitivity analysis.

Parameters:

Name Type Description Default
cf_y float

Confounding strength for the outcome. Default is 0.03.

0.03
cf_d float

Confounding strength for the treatment. Default is 0.03.

0.03
rho float

Correlation parameter. Default is 1.0.

1.0
level float

Confidence level. Default is 0.95.

0.95
null_hypothesis float

Value for the null hypothesis. Default is 0.0.

0.0

Returns:

Type Description
DoubleMLBase

The fitted instance.

tune_ml_models

Tune hyperparameters for the PLR nuisance models using Optuna. DoubleMLPLR uses two models: ml_m (treatment model) and ml_l (outcome model). See the Hyperparameter Tuning API for the full parameter reference.

from pyspark.ml.regression import RandomForestRegressor

from doubleml_pyspark.utils import kfold_split

# PLR uses ml_m (treatment) and ml_l (outcome)
ml_models = {
    "ml_m": RandomForestRegressor(),  # treatment model
    "ml_l": RandomForestRegressor(),  # outcome model
}
ml_param_space = {
    "ml_m": lambda trial: {"maxDepth": trial.suggest_int("maxDepth", 3, 10)},
    "ml_l": lambda trial: {"maxDepth": trial.suggest_int("maxDepth", 3, 10)},
}
splits = kfold_split(df, n_folds=3, n_rep=1, shuffle=True, seed=42)
ml_models = dml.tune_ml_models(
    spark_session=spark,
    data_split=splits[0][0],
    ml_models=ml_models,
    ml_param_space=ml_param_space,
)