Skip to content

Using doubleml_pyspark in Databricks Notebooks

This guide explains how to use the doubleml_pyspark package in Databricks notebooks after completing the initial setup.

Creating or Opening a Notebook

Once doubleml-pyspark is installed (see Install from the Microsoft Marketplace):

  1. Navigate to your Databricks workspace
  2. Either create a new notebook or open an existing one
  3. Attach the notebook to serverless compute or to your cluster

Connecting to the Cluster

Single User Cluster

For single user clusters, you'll use the classic Spark session:

from pyspark.sql import SparkSession

# Use the classic Spark session
spark = SparkSession.builder.getOrCreate()

This provides access to spark.classic.SparkSession which is the default for single user environments.

Shared Cluster

For shared clusters, you need to use Spark Connect:

from pyspark.sql import SparkSession

# Use Spark Connect session for shared clusters
spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

This creates a spark.connect.SparkSession which is required for shared cluster environments to ensure proper isolation between users.

Note: The type of cluster you're using (single user vs. shared) determines which session type you should initialize. Make sure to use the appropriate session for your cluster configuration.

Using doubleml_pyspark

After establishing your Spark session, you can import and use the package:

import doubleml_pyspark

# Your doubleml_pyspark code here
# Example: Initialize models, run analyses, etc.

Loading Data

You can load data into your notebooks using several methods:

Using DBFS (Databricks File System)

# Read data from DBFS
df = spark.read.csv("dbfs:/path/to/your/data.csv", header=True, inferSchema=True)

# Or using the /dbfs/ path
df = spark.read.parquet("/dbfs/mnt/your-data/")

Direct File Upload

  • Click "Data" in the Databricks workspace sidebar
  • Upload files directly to DBFS
  • Reference them using the dbfs:/ or /dbfs/ prefix

External Storage

# From Azure Blob Storage
df = spark.read.csv("wasbs://container@account.blob.core.windows.net/path/to/file.csv")

# From S3
df = spark.read.parquet("s3a://bucket-name/path/to/data/")

Complete Example

Here's a complete example of setting up and using doubleml_pyspark:

# Step 1: Initialize Spark session (choose based on cluster type)
from pyspark.sql import SparkSession

# For single user cluster:
spark = SparkSession.builder.getOrCreate()

# OR for shared cluster:
# spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

# Step 2: Import doubleml_pyspark
import doubleml_pyspark

# Step 3: Load your data
df = spark.read.csv("dbfs:/FileStore/my_data.csv", header=True, inferSchema=True)

# Step 4: Use doubleml_pyspark with your data
# Your analysis code here

Troubleshooting

Library not found: Install the package as described in Install from the Microsoft Marketplace, then run dbutils.library.restartPython().

Session connection errors: Verify you're using the correct session type for your cluster configuration (single user vs. shared).

Data access issues: Check that your data paths are correct and that you have appropriate permissions to access the data sources.


You're now ready to run doubleml_pyspark analyses in your Databricks notebooks!