Skip to content

DoubleMLScalar

doubleml_pyspark.doubleml_scalar.DoubleMLScalar

Bases: DoubleMLBase, ABC

Abstract base class for DoubleML estimation of a scalar parameter.

This class provides the core logic for Double Machine Learning (DML) estimators that estimate a scalar causal parameter using PySpark.

Methods:

Name Description
cate

CATE estimation.

confint

Confidence intervals for DoubleML models.

estimate_causal_parameters

Estimate the causal parameters on a single repetition.

fit

Fit the DoubleML model.

fit_ml_models

Train and cross-fit the nuisance ML models and collect predictions.

gate

GATE estimation (Group Average Treatment Effects).

sensitivity_analysis

Sensitivity analysis.

tune_ml_models

Tune hyperparameters for multiple ML models using Optuna.

Attributes:

Name Type Description
additional_nuisance_elements dict[str, Any]

Additional, estimator-specific low-dimensional nuisance elements.

all_coef ndarray

Estimates for each repetition, shape (n_thetas, n_rep). Same as :attr:all_thetas.

all_se ndarray

Standard errors for each repetition, shape (n_thetas, n_rep). Same as :attr:all_ses.

all_ses ndarray

All standard errors across repetitions and folds.

all_thetas ndarray

All estimated parameters across repetitions and folds.

coef ndarray

Estimated causal parameter(s), shape (n_thetas,). Same as :attr:thetas.

covariates list[str]

The covariates.

data_field_map dict[str, str | list[str]]

The data field map.

data_schema StructType

The schema of the input data.

id_col str

The id column.

n_obs int

The number of observations used for estimation.

n_rep int

The number of repetitions for cross-validation.

outcome str

The outcome variable.

p_val ndarray

The p-values.

pval ndarray

p-value(s), shape (n_thetas,). Same as :attr:p_val.

score_elements list[Any]

The score elements.

se ndarray

Standard error(s), shape (n_thetas,). Same as :attr:ses.

sensitivity_params dict[str, Any] | None

The sensitivity parameters.

sensitivity_summary str

Summary for the sensitivity analysis after calling sensitivity_analysis.

sensitivity_var_elements dict[str, ndarray] | None

The sensitivity variance elements.

ses ndarray

The standard errors.

summary DataFrame

Summary of the estimated causal parameter(s) after calling fit().

t_stat ndarray

The t-statistics.

thetas ndarray

The estimated parameters.

treatment str

The treatment variable.

var_scaling ndarray

The variance scaling factors.

Attributes

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.additional_nuisance_elements property writable

additional_nuisance_elements: dict[str, Any]

Additional, estimator-specific low-dimensional nuisance elements.

This dictionary stores extra quantities required by a concrete estimator beyond the standard nuisances.

Notes
  • For some estimators this may be empty (e.g., IRM with ATE).
  • For others it may contain specific entries (e.g., IRM with ATT requires {'subgroup_prob': float in (0, 1]}).
  • When set via the property setter, inputs are merged with existing values, validated, and sanitized to keep only accepted keys; missing required keys raise ValueError.

Returns:

Type Description
dict[str, Any]

The currently stored, validated additional nuisance elements, or None if they have not been estimated/set yet.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.all_coef property

all_coef: ndarray

Estimates for each repetition, shape (n_thetas, n_rep). Same as :attr:all_thetas.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.all_se property

all_se: ndarray

Standard errors for each repetition, shape (n_thetas, n_rep). Same as :attr:all_ses.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.all_ses property

all_ses: ndarray

All standard errors across repetitions and folds.

Returns:

Type Description
ndarray

Array of shape (n_thetas, n_rep) with all standard errors.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.all_thetas property

all_thetas: ndarray

All estimated parameters across repetitions and folds.

Returns:

Type Description
ndarray

Array of shape (n_thetas, n_rep) with all estimated parameters.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.coef property

coef: ndarray

Estimated causal parameter(s), shape (n_thetas,). Same as :attr:thetas.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.covariates property

covariates: list[str]

The covariates.

Returns:

Type Description
list of str or None

The list of covariate names, or None if not set.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.data_field_map property

data_field_map: dict[str, str | list[str]]

The data field map.

Returns:

Type Description
dict of str

The mapping of field names.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.data_schema property

data_schema: StructType

The schema of the input data.

Returns:

Type Description
StructType

The schema of the input data.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.id_col property

id_col: str

The id column.

Returns:

Type Description
str

The name of the id column.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.n_obs property

n_obs: int

The number of observations used for estimation.

Returns:

Type Description
int or None

The number of observations, or None if not set.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.n_rep property

n_rep: int

The number of repetitions for cross-validation.

Returns:

Type Description
int

The number of repetitions.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.outcome property

outcome: str

The outcome variable.

Returns:

Type Description
str

The name of the outcome variable.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.p_val property

p_val: ndarray

The p-values.

Returns:

Type Description
ndarray

Array of shape (n_thetas,) with p-values.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.pval property

pval: ndarray

p-value(s), shape (n_thetas,). Same as :attr:p_val.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.score_elements property

score_elements: list[Any]

The score elements.

Returns:

Type Description
Any

The score elements.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.se property

se: ndarray

Standard error(s), shape (n_thetas,). Same as :attr:ses.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.sensitivity_params property

sensitivity_params: dict[str, Any] | None

The sensitivity parameters.

Returns:

Type Description
dict or None

The sensitivity parameters, or None if not set.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.sensitivity_summary property

sensitivity_summary: str

Summary for the sensitivity analysis after calling sensitivity_analysis.

Returns:

Type Description
str

Summary for the sensitivity analysis.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.sensitivity_var_elements property

sensitivity_var_elements: dict[str, ndarray] | None

The sensitivity variance elements.

Returns:

Type Description
dict or None

Each element is an array of shape (n_thetas, n_rep), or None if not set.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.ses property

ses: ndarray

The standard errors.

Returns:

Type Description
ndarray

Array of shape (n_thetas,) with standard errors.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.summary property

summary: DataFrame

Summary of the estimated causal parameter(s) after calling fit().

Returns:

Type Description
DataFrame

Columns coef, std err, t, P>|t| and the 95 % confidence interval, one row per parameter. Empty before fit().

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.t_stat property

t_stat: ndarray

The t-statistics.

Returns:

Type Description
ndarray

Array of shape (n_thetas,) with t-statistics.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.thetas property

thetas: ndarray

The estimated parameters.

Returns:

Type Description
ndarray

Array of shape (n_thetas,) with estimated parameters.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.treatment property

treatment: str

The treatment variable.

Returns:

Type Description
str

The name of the treatment variable.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.var_scaling property

var_scaling: ndarray

The variance scaling factors.

Returns:

Type Description
ndarray

Array of shape (n_thetas, n_rep) with variance scaling factors.

Functions

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.cate

cate(spark_session: SparkSessionType, df_basis: DataFrame, df_nuisance: DataFrame, cov_type: str = 'HC0', num_partitions: int | None = None, is_gate: bool = False, use_persist: bool = False) -> DoubleMLBLP

CATE estimation.

Parameters:

Name Type Description Default
spark_session SparkSession or SparkSession

The active Spark session (supports both local and connect modes).

required
df_basis DataFrame

DataFrame with basis functions.

required
df_nuisance DataFrame

DataFrame with nuisance parameter predictions.

required
cov_type str

Covariance estimator used for the BLP. Supported values are 'nonrobust', 'HC0', 'HC1', 'HC2', and 'HC3'.

'HC0'
num_partitions int

Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration.

None
use_persist bool

Whether to cache intermediate DataFrames. Not supported on Databricks serverless compute. Defaults to False.

False

Returns:

Type Description
DoubleMLBLP

The fitted BLP model for CATE estimation.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.confint

confint(joint: bool = False, level: float = 0.95) -> pd.DataFrame

Confidence intervals for DoubleML models.

Parameters:

Name Type Description Default
joint bool

Indicates whether joint confidence intervals are computed. Default is False.

False
level float

The confidence level. Default is 0.95.

0.95

Returns:

Type Description
DataFrame

A data frame with the confidence interval(s).

Raises:

Type Description
TypeError

If input types are incorrect.

ValueError

If confidence level is not in (0, 1).

NotImplementedError

If joint confidence intervals are requested.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.estimate_causal_parameters

estimate_causal_parameters(spark_session: SparkSessionType, df_nuisance: DataFrame, additional_nuisance_elements: dict[str, Any] | None = None, num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_checkpoint: bool = False, use_persist: bool = False) -> DoubleMLScalar

Estimate the causal parameters on a single repetition.

Parameters:

Name Type Description Default
spark_session SparkSession or SparkSession

The active Spark session (supports both local and connect modes).

required
df_nuisance DataFrame

DataFrame with nuisance parameter predictions.

required
additional_nuisance_elements dict of str to Any

Additional nuisance elements.

None
num_partitions int

Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration.

None
storage_level StorageLevel

Storage level for caching intermediate DataFrames.

MEMORY_AND_DISK_DESER
use_checkpoint bool

Whether to use checkpointing.

False
use_persist bool

Whether to persist DataFrames for caching. Defaults to False.

False

Returns:

Type Description
DoubleMLScalar

The fitted estimator.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.fit

fit(spark_session: SparkSessionType, kfold_split: list[tuple[DataFrame, DataFrame]], ml_models: dict[str, Any], num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_checkpoint: bool = False, use_persist: bool = False) -> DoubleMLScalar

Fit the DoubleML model.

Parameters:

Name Type Description Default
spark_session SparkSession or SparkSession

The active Spark session (supports both local and connect modes).

required
kfold_split list of tuples of (DataFrame, DataFrame)

List of (train, test) splits for cross-fitting.

required
ml_models dict of str to Any

Dictionary of ML models.

required
num_partitions int

Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration.

None
storage_level StorageLevel

Storage level for caching intermediate DataFrames.

MEMORY_AND_DISK_DESER
use_checkpoint bool

Whether to use checkpointing df_nuisance and df_psi.

False
use_persist bool

Whether to persist DataFrames for caching. Defaults to False.

False

Returns:

Type Description
DoubleMLScalar

The fitted estimator.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.fit_ml_models

fit_ml_models(spark_session: SparkSessionType, kfold_split: list[tuple[DataFrame, DataFrame]], ml_models: dict[str, Any], num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_persist: bool = False) -> tuple[DataFrame, dict[str, Any]]

Train and cross-fit the nuisance ML models and collect predictions.

This runs each estimator on the provided k-fold splits (train, test), unions the fold-wise predictions into a single nuisance DataFrame conforming to _get_df_nuisance_schema(), and estimates any additional low-dimensional nuisance elements required by the concrete estimator.

Parameters:

Name Type Description Default
spark_session SparkSession or SparkSession

The active Spark session (supports both local and connect modes).

required
kfold_split list[tuple[DataFrame, DataFrame]]

Sequence of (train_df, test_df) splits used for cross-fitting. Each DataFrame must contain the columns specified in data_field_map.

required
ml_models dict[str, Any]

Mapping from required nuisance model names to Spark Estimators as specified by _get_ml_models_structure().

required
num_partitions int

Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration.

None
storage_level StorageLevel

Storage level for caching intermediate DataFrames.

MEMORY_AND_DISK_DESER
use_persist bool

Whether to persist DataFrames for caching. Defaults to False.

False

Returns:

Type Description
tuple[DataFrame, dict[str, Any]]
  • df_nuisance: DataFrame with per-row nuisance predictions; schema is given by _get_df_nuisance_schema() and includes the id_column.
  • additional_nuisance_elements: Dictionary with any extra low-dimensional quantities needed by the estimator; may be empty ({}).
Notes
  • This method does not modify internal state beyond logging.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.gate

gate(spark_session: SparkSessionType, df_groups: DataFrame, df_nuisance: DataFrame, cov_type: str = 'HC0', num_partitions: int | None = None, use_persist: bool = False) -> DoubleMLBLP

GATE estimation (Group Average Treatment Effects).

The group variant of :meth:cate: builds a mutually exclusive dummy (one-hot) basis from df_groups and fits the best linear predictor on it with is_gate=True, so that every coefficient is the average treatment effect within a group. Like :meth:cate, it works for any model that implements :meth:_get_blp_orth_signal_and_basis (e.g. PLR and IRM), so it lives on the shared base class rather than per model.

Parameters:

Name Type Description Default
spark_session SparkSession or SparkSession

The active Spark session (supports both local and connect modes).

required
df_groups DataFrame

Group membership, including the id column (self.id_col). Either

  • a single group-label column (string/numeric) that is one-hot encoded into mutually exclusive Group_<value> dummy columns; or
  • several already dummy-coded columns (boolean or integer 0/1), one per group.
required
df_nuisance DataFrame

DataFrame with nuisance parameter predictions.

required
cov_type str

Covariance estimator used for the BLP. Supported values are 'nonrobust', 'HC0', 'HC1', 'HC2', and 'HC3'.

'HC0'
num_partitions int

Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration.

None
use_persist bool

Whether to cache intermediate DataFrames. Not supported on Databricks serverless compute. Defaults to False.

False

Returns:

Type Description
DoubleMLBLP

The fitted BLP model for GATE estimation (with is_gate=True).

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.sensitivity_analysis

sensitivity_analysis(cf_y: float = 0.03, cf_d: float = 0.03, rho: float = 1.0, level: float = 0.95, null_hypothesis: float = 0.0) -> DoubleMLBase

Sensitivity analysis.

Parameters:

Name Type Description Default
cf_y float

Confounding strength for the outcome. Default is 0.03.

0.03
cf_d float

Confounding strength for the treatment. Default is 0.03.

0.03
rho float

Correlation parameter. Default is 1.0.

1.0
level float

Confidence level. Default is 0.95.

0.95
null_hypothesis float

Value for the null hypothesis. Default is 0.0.

0.0

Returns:

Type Description
DoubleMLBase

The fitted instance.

doubleml_pyspark.doubleml_scalar.DoubleMLScalar.tune_ml_models

tune_ml_models(spark_session: SparkSessionType, data_split: tuple[DataFrame, DataFrame], ml_models: dict[str, Any], ml_param_space: dict[str, Callable[[Any], dict[str, Any]] | None], scoring_methods: dict | None = None, num_partitions: int | None = None, return_tune_res: bool = False, optuna_settings: dict | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_persist: bool = False) -> dict[str, Any] | tuple[dict[str, Any], dict[str, DMLOptunaResult]]

Tune hyperparameters for multiple ML models using Optuna.

This is the public entry point for hyperparameter tuning. It orchestrates the tuning of all models and returns comprehensive results.

Parameters:

Name Type Description Default
spark_session SparkSessionType

Active Spark session.

required
data_split tuple[DataFrame, DataFrame]

Tuple of (train, test) DataFrames for cross-validation.

required
ml_models dict[str, Any]

Dictionary mapping model names to Spark ML estimators.

required
ml_param_space dict[str, Callable | None]

Dictionary mapping model names to parameter space functions. Models in ml_models but not in ml_param_space will be fitted without tuning.

required
scoring_methods dict[str, str]

Dictionary mapping model names to scoring methods. For regressors: 'rmse', 'r2', 'mae', 'mse' For classifiers: 'areaUnderROC', 'areaUnderPR', 'f1', 'accuracy' Defaults to 'rmse' for regressors, 'logLoss' for classifiers.

None
num_partitions int

Number of partitions for data repartitioning.

None
return_tune_res bool

If False, returns only dict[str, Estimator] with the best models. If True, returns a tuple (best_models_dict, results_dict).

False
optuna_settings dict

Global Optuna settings with learner-specific overrides.

None
storage_level StorageLevel

Storage level for data.

MEMORY_AND_DISK_DESER
use_persist bool

Whether to persist DataFrames for caching. Defaults to False.

False

Returns:

Type Description
dict[str, DMLOptunaResult | None] | tuple[dict[str, Any], dict[str, DMLOptunaResult | None]]

If return_models_only=False (default): Returns dict mapping model names to DMLOptunaResult objects or None. None indicates the model was fitted without tuning.

If return_models_only=True: Returns tuple of: - dict[str, Estimator]: Best estimators ready to use (fitted models for non-tuned) - dict[str, DMLOptunaResult | None]: Full tuning results (None for non-tuned models)

Examples:

>>> # Models with partial tuning
>>> ml_models = {
...     'ml_m': LogisticRegression(),
...     'ml_g0': RandomForestRegressor(),
...     'ml_g1': RandomForestRegressor()
... }
>>> ml_param_space = {
...     'ml_g0': lambda trial: {'maxDepth': trial.suggest_int('maxDepth', 3, 10)},
...     'ml_g1': lambda trial: {'maxDepth': trial.suggest_int('maxDepth', 3, 10)}
...     # ml_m will be fitted without tuning
... }
>>> results = dml.tune_ml_models(
...     spark_session=spark,
...     data_split=splits,
...     ml_models=ml_models,
...     ml_param_space=ml_param_space
... )