DoubleMLScalar¶
doubleml_pyspark.doubleml_scalar.DoubleMLScalar
¶
Bases: DoubleMLBase, ABC
Abstract base class for DoubleML estimation of a scalar parameter.
This class provides the core logic for Double Machine Learning (DML) estimators that estimate a scalar causal parameter using PySpark.
Methods:
| Name | Description |
|---|---|
cate |
CATE estimation. |
confint |
Confidence intervals for DoubleML models. |
estimate_causal_parameters |
Estimate the causal parameters on a single repetition. |
fit |
Fit the DoubleML model. |
fit_ml_models |
Train and cross-fit the nuisance ML models and collect predictions. |
gate |
GATE estimation (Group Average Treatment Effects). |
sensitivity_analysis |
Sensitivity analysis. |
tune_ml_models |
Tune hyperparameters for multiple ML models using Optuna. |
Attributes:
| Name | Type | Description |
|---|---|---|
additional_nuisance_elements |
dict[str, Any]
|
Additional, estimator-specific low-dimensional nuisance elements. |
all_coef |
ndarray
|
Estimates for each repetition, shape (n_thetas, n_rep). Same as :attr: |
all_se |
ndarray
|
Standard errors for each repetition, shape (n_thetas, n_rep). Same as :attr: |
all_ses |
ndarray
|
All standard errors across repetitions and folds. |
all_thetas |
ndarray
|
All estimated parameters across repetitions and folds. |
coef |
ndarray
|
Estimated causal parameter(s), shape (n_thetas,). Same as :attr: |
covariates |
list[str]
|
The covariates. |
data_field_map |
dict[str, str | list[str]]
|
The data field map. |
data_schema |
StructType
|
The schema of the input data. |
id_col |
str
|
The id column. |
n_obs |
int
|
The number of observations used for estimation. |
n_rep |
int
|
The number of repetitions for cross-validation. |
outcome |
str
|
The outcome variable. |
p_val |
ndarray
|
The p-values. |
pval |
ndarray
|
p-value(s), shape (n_thetas,). Same as :attr: |
score_elements |
list[Any]
|
The score elements. |
se |
ndarray
|
Standard error(s), shape (n_thetas,). Same as :attr: |
sensitivity_params |
dict[str, Any] | None
|
The sensitivity parameters. |
sensitivity_summary |
str
|
Summary for the sensitivity analysis after calling sensitivity_analysis. |
sensitivity_var_elements |
dict[str, ndarray] | None
|
The sensitivity variance elements. |
ses |
ndarray
|
The standard errors. |
summary |
DataFrame
|
Summary of the estimated causal parameter(s) after calling |
t_stat |
ndarray
|
The t-statistics. |
thetas |
ndarray
|
The estimated parameters. |
treatment |
str
|
The treatment variable. |
var_scaling |
ndarray
|
The variance scaling factors. |
Attributes¶
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.additional_nuisance_elements
property
writable
¶
Additional, estimator-specific low-dimensional nuisance elements.
This dictionary stores extra quantities required by a concrete estimator beyond the standard nuisances.
Notes
- For some estimators this may be empty (e.g., IRM with ATE).
- For others it may contain specific entries (e.g., IRM with ATT requires {'subgroup_prob': float in (0, 1]}).
- When set via the property setter, inputs are merged with existing values, validated, and sanitized to keep only accepted keys; missing required keys raise ValueError.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
The currently stored, validated additional nuisance elements, or None if they have not been estimated/set yet. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.all_coef
property
¶
Estimates for each repetition, shape (n_thetas, n_rep). Same as :attr:all_thetas.
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.all_se
property
¶
Standard errors for each repetition, shape (n_thetas, n_rep). Same as :attr:all_ses.
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.all_ses
property
¶
All standard errors across repetitions and folds.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas, n_rep) with all standard errors. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.all_thetas
property
¶
All estimated parameters across repetitions and folds.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas, n_rep) with all estimated parameters. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.coef
property
¶
Estimated causal parameter(s), shape (n_thetas,). Same as :attr:thetas.
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.covariates
property
¶
The covariates.
Returns:
| Type | Description |
|---|---|
list of str or None
|
The list of covariate names, or None if not set. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.data_field_map
property
¶
The data field map.
Returns:
| Type | Description |
|---|---|
dict of str
|
The mapping of field names. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.data_schema
property
¶
The schema of the input data.
Returns:
| Type | Description |
|---|---|
StructType
|
The schema of the input data. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.id_col
property
¶
The id column.
Returns:
| Type | Description |
|---|---|
str
|
The name of the id column. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.n_obs
property
¶
The number of observations used for estimation.
Returns:
| Type | Description |
|---|---|
int or None
|
The number of observations, or None if not set. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.n_rep
property
¶
The number of repetitions for cross-validation.
Returns:
| Type | Description |
|---|---|
int
|
The number of repetitions. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.outcome
property
¶
The outcome variable.
Returns:
| Type | Description |
|---|---|
str
|
The name of the outcome variable. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.p_val
property
¶
The p-values.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas,) with p-values. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.pval
property
¶
p-value(s), shape (n_thetas,). Same as :attr:p_val.
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.score_elements
property
¶
The score elements.
Returns:
| Type | Description |
|---|---|
Any
|
The score elements. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.se
property
¶
Standard error(s), shape (n_thetas,). Same as :attr:ses.
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.sensitivity_params
property
¶
The sensitivity parameters.
Returns:
| Type | Description |
|---|---|
dict or None
|
The sensitivity parameters, or None if not set. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.sensitivity_summary
property
¶
Summary for the sensitivity analysis after calling sensitivity_analysis.
Returns:
| Type | Description |
|---|---|
str
|
Summary for the sensitivity analysis. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.sensitivity_var_elements
property
¶
The sensitivity variance elements.
Returns:
| Type | Description |
|---|---|
dict or None
|
Each element is an array of shape (n_thetas, n_rep), or None if not set. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.ses
property
¶
The standard errors.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas,) with standard errors. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.summary
property
¶
Summary of the estimated causal parameter(s) after calling fit().
Returns:
| Type | Description |
|---|---|
DataFrame
|
Columns |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.t_stat
property
¶
The t-statistics.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas,) with t-statistics. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.thetas
property
¶
The estimated parameters.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas,) with estimated parameters. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.treatment
property
¶
The treatment variable.
Returns:
| Type | Description |
|---|---|
str
|
The name of the treatment variable. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.var_scaling
property
¶
The variance scaling factors.
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape (n_thetas, n_rep) with variance scaling factors. |
Functions¶
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.cate
¶
cate(spark_session: SparkSessionType, df_basis: DataFrame, df_nuisance: DataFrame, cov_type: str = 'HC0', num_partitions: int | None = None, is_gate: bool = False, use_persist: bool = False) -> DoubleMLBLP
CATE estimation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_session
|
SparkSession or SparkSession
|
The active Spark session (supports both local and connect modes). |
required |
df_basis
|
DataFrame
|
DataFrame with basis functions. |
required |
df_nuisance
|
DataFrame
|
DataFrame with nuisance parameter predictions. |
required |
cov_type
|
str
|
Covariance estimator used for the BLP. Supported values are
|
'HC0'
|
num_partitions
|
int
|
Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration. |
None
|
use_persist
|
bool
|
Whether to cache intermediate DataFrames. Not supported on Databricks serverless compute. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
DoubleMLBLP
|
The fitted BLP model for CATE estimation. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.confint
¶
Confidence intervals for DoubleML models.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
joint
|
bool
|
Indicates whether joint confidence intervals are computed. Default is False. |
False
|
level
|
float
|
The confidence level. Default is 0.95. |
0.95
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
A data frame with the confidence interval(s). |
Raises:
| Type | Description |
|---|---|
TypeError
|
If input types are incorrect. |
ValueError
|
If confidence level is not in (0, 1). |
NotImplementedError
|
If joint confidence intervals are requested. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.estimate_causal_parameters
¶
estimate_causal_parameters(spark_session: SparkSessionType, df_nuisance: DataFrame, additional_nuisance_elements: dict[str, Any] | None = None, num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_checkpoint: bool = False, use_persist: bool = False) -> DoubleMLScalar
Estimate the causal parameters on a single repetition.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_session
|
SparkSession or SparkSession
|
The active Spark session (supports both local and connect modes). |
required |
df_nuisance
|
DataFrame
|
DataFrame with nuisance parameter predictions. |
required |
additional_nuisance_elements
|
dict of str to Any
|
Additional nuisance elements. |
None
|
num_partitions
|
int
|
Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration. |
None
|
storage_level
|
StorageLevel
|
Storage level for caching intermediate DataFrames. |
MEMORY_AND_DISK_DESER
|
use_checkpoint
|
bool
|
Whether to use checkpointing. |
False
|
use_persist
|
bool
|
Whether to persist DataFrames for caching. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
DoubleMLScalar
|
The fitted estimator. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.fit
¶
fit(spark_session: SparkSessionType, kfold_split: list[tuple[DataFrame, DataFrame]], ml_models: dict[str, Any], num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_checkpoint: bool = False, use_persist: bool = False) -> DoubleMLScalar
Fit the DoubleML model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_session
|
SparkSession or SparkSession
|
The active Spark session (supports both local and connect modes). |
required |
kfold_split
|
list of tuples of (DataFrame, DataFrame)
|
List of (train, test) splits for cross-fitting. |
required |
ml_models
|
dict of str to Any
|
Dictionary of ML models. |
required |
num_partitions
|
int
|
Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration. |
None
|
storage_level
|
StorageLevel
|
Storage level for caching intermediate DataFrames. |
MEMORY_AND_DISK_DESER
|
use_checkpoint
|
bool
|
Whether to use checkpointing df_nuisance and df_psi. |
False
|
use_persist
|
bool
|
Whether to persist DataFrames for caching. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
DoubleMLScalar
|
The fitted estimator. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.fit_ml_models
¶
fit_ml_models(spark_session: SparkSessionType, kfold_split: list[tuple[DataFrame, DataFrame]], ml_models: dict[str, Any], num_partitions: int | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_persist: bool = False) -> tuple[DataFrame, dict[str, Any]]
Train and cross-fit the nuisance ML models and collect predictions.
This runs each estimator on the provided k-fold splits (train, test), unions the fold-wise predictions into a single nuisance DataFrame conforming to _get_df_nuisance_schema(), and estimates any additional low-dimensional nuisance elements required by the concrete estimator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_session
|
SparkSession or SparkSession
|
The active Spark session (supports both local and connect modes). |
required |
kfold_split
|
list[tuple[DataFrame, DataFrame]]
|
Sequence of (train_df, test_df) splits used for cross-fitting. Each DataFrame must contain the columns specified in data_field_map. |
required |
ml_models
|
dict[str, Any]
|
Mapping from required nuisance model names to Spark Estimators as specified by _get_ml_models_structure(). |
required |
num_partitions
|
int
|
Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration. |
None
|
storage_level
|
StorageLevel
|
Storage level for caching intermediate DataFrames. |
MEMORY_AND_DISK_DESER
|
use_persist
|
bool
|
Whether to persist DataFrames for caching. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
tuple[DataFrame, dict[str, Any]]
|
|
Notes
- This method does not modify internal state beyond logging.
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.gate
¶
gate(spark_session: SparkSessionType, df_groups: DataFrame, df_nuisance: DataFrame, cov_type: str = 'HC0', num_partitions: int | None = None, use_persist: bool = False) -> DoubleMLBLP
GATE estimation (Group Average Treatment Effects).
The group variant of :meth:cate: builds a mutually exclusive dummy
(one-hot) basis from df_groups and fits the best linear predictor on
it with is_gate=True, so that every coefficient is the average
treatment effect within a group. Like :meth:cate, it works for any
model that implements :meth:_get_blp_orth_signal_and_basis (e.g. PLR
and IRM), so it lives on the shared base class rather than per model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_session
|
SparkSession or SparkSession
|
The active Spark session (supports both local and connect modes). |
required |
df_groups
|
DataFrame
|
Group membership, including the id column (
|
required |
df_nuisance
|
DataFrame
|
DataFrame with nuisance parameter predictions. |
required |
cov_type
|
str
|
Covariance estimator used for the BLP. Supported values are
|
'HC0'
|
num_partitions
|
int
|
Number of partitions for processing DataFrames. If None, uses the spark_session's default configuration. |
None
|
use_persist
|
bool
|
Whether to cache intermediate DataFrames. Not supported on Databricks serverless compute. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
DoubleMLBLP
|
The fitted BLP model for GATE estimation (with |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.sensitivity_analysis
¶
sensitivity_analysis(cf_y: float = 0.03, cf_d: float = 0.03, rho: float = 1.0, level: float = 0.95, null_hypothesis: float = 0.0) -> DoubleMLBase
Sensitivity analysis.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cf_y
|
float
|
Confounding strength for the outcome. Default is 0.03. |
0.03
|
cf_d
|
float
|
Confounding strength for the treatment. Default is 0.03. |
0.03
|
rho
|
float
|
Correlation parameter. Default is 1.0. |
1.0
|
level
|
float
|
Confidence level. Default is 0.95. |
0.95
|
null_hypothesis
|
float
|
Value for the null hypothesis. Default is 0.0. |
0.0
|
Returns:
| Type | Description |
|---|---|
DoubleMLBase
|
The fitted instance. |
doubleml_pyspark.doubleml_scalar.DoubleMLScalar.tune_ml_models
¶
tune_ml_models(spark_session: SparkSessionType, data_split: tuple[DataFrame, DataFrame], ml_models: dict[str, Any], ml_param_space: dict[str, Callable[[Any], dict[str, Any]] | None], scoring_methods: dict | None = None, num_partitions: int | None = None, return_tune_res: bool = False, optuna_settings: dict | None = None, storage_level: StorageLevel = StorageLevel.MEMORY_AND_DISK_DESER, use_persist: bool = False) -> dict[str, Any] | tuple[dict[str, Any], dict[str, DMLOptunaResult]]
Tune hyperparameters for multiple ML models using Optuna.
This is the public entry point for hyperparameter tuning. It orchestrates the tuning of all models and returns comprehensive results.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spark_session
|
SparkSessionType
|
Active Spark session. |
required |
data_split
|
tuple[DataFrame, DataFrame]
|
Tuple of (train, test) DataFrames for cross-validation. |
required |
ml_models
|
dict[str, Any]
|
Dictionary mapping model names to Spark ML estimators. |
required |
ml_param_space
|
dict[str, Callable | None]
|
Dictionary mapping model names to parameter space functions. Models in ml_models but not in ml_param_space will be fitted without tuning. |
required |
scoring_methods
|
dict[str, str]
|
Dictionary mapping model names to scoring methods. For regressors: 'rmse', 'r2', 'mae', 'mse' For classifiers: 'areaUnderROC', 'areaUnderPR', 'f1', 'accuracy' Defaults to 'rmse' for regressors, 'logLoss' for classifiers. |
None
|
num_partitions
|
int
|
Number of partitions for data repartitioning. |
None
|
return_tune_res
|
bool
|
If False, returns only dict[str, Estimator] with the best models. If True, returns a tuple (best_models_dict, results_dict). |
False
|
optuna_settings
|
dict
|
Global Optuna settings with learner-specific overrides. |
None
|
storage_level
|
StorageLevel
|
Storage level for data. |
MEMORY_AND_DISK_DESER
|
use_persist
|
bool
|
Whether to persist DataFrames for caching. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
dict[str, DMLOptunaResult | None] | tuple[dict[str, Any], dict[str, DMLOptunaResult | None]]
|
If return_models_only=False (default): Returns dict mapping model names to DMLOptunaResult objects or None. None indicates the model was fitted without tuning. If return_models_only=True: Returns tuple of: - dict[str, Estimator]: Best estimators ready to use (fitted models for non-tuned) - dict[str, DMLOptunaResult | None]: Full tuning results (None for non-tuned models) |
Examples:
>>> # Models with partial tuning
>>> ml_models = {
... 'ml_m': LogisticRegression(),
... 'ml_g0': RandomForestRegressor(),
... 'ml_g1': RandomForestRegressor()
... }
>>> ml_param_space = {
... 'ml_g0': lambda trial: {'maxDepth': trial.suggest_int('maxDepth', 3, 10)},
... 'ml_g1': lambda trial: {'maxDepth': trial.suggest_int('maxDepth', 3, 10)}
... # ml_m will be fitted without tuning
... }
>>> results = dml.tune_ml_models(
... spark_session=spark,
... data_split=splits,
... ml_models=ml_models,
... ml_param_space=ml_param_space
... )