Skip to content

Troubleshooting

Common issues and solutions when working with DoubleML PySpark.

Memory Issues

If you encounter OutOfMemoryError during model fitting, try the following:

  • Increase the Spark driver and executor memory:
spark = SparkSession.builder \
    .config("spark.driver.memory", "16g") \
    .config("spark.executor.memory", "8g") \
    .getOrCreate()
  • Enable checkpointing for recovery from executor failures:
spark.sparkContext.setCheckpointDir("/tmp/spark-checkpoint")
dml_model.fit(..., use_checkpoint=True)
  • Reduce the number of partitions or process smaller chunks of data.

Slow Performance

If model fitting is taking too long:

# Increase parallelism
dml_model.fit(..., num_partitions=32)

# Use fewer folds
kfold_splits = kfold_split(df, n_folds=2, ...)

# Cache frequently reused DataFrames
df = df.cache()

On a single machine, keep num_partitions at 2-4x your CPU core count — over-partitioning adds overhead without benefit.

Installation Issues

Package installation fails with dependency conflicts.

DoubleML PySpark requires Python 3.12 and PySpark 3.5.2 or newer (PySpark 3.5 only in classic mode, not over Spark Connect). On Databricks, do not install PySpark yourself (the runtime provides it). Outside Databricks, install doubleml-pyspark[spark] in a fresh virtual environment; with PySpark 3.5 on Python 3.12 also install setuptools (PySpark 3.5 still imports distutils). The example data generators need the datasets extra: doubleml-pyspark[datasets]. More cases: Install from the Microsoft Marketplace.

Spark Session Issues

Spark session is not available or fails to initialize.

Make sure your Spark environment is properly configured. If running locally, ensure that Java is installed and JAVA_HOME is set. On Databricks, the Spark session is automatically available.

Session connection errors on Databricks: Verify you are using the correct session type for your cluster configuration (single user vs. shared).

Data Issues

  • Ensure your data files exist in the specified path.
  • Verify column names match the data_field_map.
  • Check for data quality issues (nulls, infinite values).
  • Check that all required columns exist in the data.

Databricks-Specific Issues

Library not found: Install the package as described in Install from the Microsoft Marketplace, then run dbutils.library.restartPython().

Data access issues: Check that your data paths are correct and that you have appropriate permissions to access the data sources.

Further Help

If your issue is not listed here, see Support or write to cloud@economicai.com.