Using doubleml_pyspark in Databricks Notebooks¶
This guide explains how to use the doubleml_pyspark package in Databricks notebooks after completing the initial setup.
Creating or Opening a Notebook¶
Once doubleml-pyspark is installed (see Install from the Microsoft Marketplace):
- Navigate to your Databricks workspace
- Either create a new notebook or open an existing one
- Attach the notebook to serverless compute or to your cluster
Connecting to the Cluster¶
Single User Cluster¶
For single user clusters, you'll use the classic Spark session:
from pyspark.sql import SparkSession
# Use the classic Spark session
spark = SparkSession.builder.getOrCreate()
This provides access to spark.classic.SparkSession which is the default for single user environments.
Shared Cluster¶
For shared clusters, you need to use Spark Connect:
from pyspark.sql import SparkSession
# Use Spark Connect session for shared clusters
spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
This creates a spark.connect.SparkSession which is required for shared cluster environments to ensure proper isolation between users.
Note: The type of cluster you're using (single user vs. shared) determines which session type you should initialize. Make sure to use the appropriate session for your cluster configuration.
Using doubleml_pyspark¶
After establishing your Spark session, you can import and use the package:
import doubleml_pyspark
# Your doubleml_pyspark code here
# Example: Initialize models, run analyses, etc.
Loading Data¶
You can load data into your notebooks using several methods:
Using DBFS (Databricks File System)¶
# Read data from DBFS
df = spark.read.csv("dbfs:/path/to/your/data.csv", header=True, inferSchema=True)
# Or using the /dbfs/ path
df = spark.read.parquet("/dbfs/mnt/your-data/")
Direct File Upload¶
- Click "Data" in the Databricks workspace sidebar
- Upload files directly to DBFS
- Reference them using the
dbfs:/or/dbfs/prefix
External Storage¶
# From Azure Blob Storage
df = spark.read.csv("wasbs://container@account.blob.core.windows.net/path/to/file.csv")
# From S3
df = spark.read.parquet("s3a://bucket-name/path/to/data/")
Complete Example¶
Here's a complete example of setting up and using doubleml_pyspark:
# Step 1: Initialize Spark session (choose based on cluster type)
from pyspark.sql import SparkSession
# For single user cluster:
spark = SparkSession.builder.getOrCreate()
# OR for shared cluster:
# spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
# Step 2: Import doubleml_pyspark
import doubleml_pyspark
# Step 3: Load your data
df = spark.read.csv("dbfs:/FileStore/my_data.csv", header=True, inferSchema=True)
# Step 4: Use doubleml_pyspark with your data
# Your analysis code here
Troubleshooting¶
Library not found: Install the package as described in Install from the Microsoft Marketplace, then run dbutils.library.restartPython().
Session connection errors: Verify you're using the correct session type for your cluster configuration (single user vs. shared).
Data access issues: Check that your data paths are correct and that you have appropriate permissions to access the data sources.
You're now ready to run doubleml_pyspark analyses in your Databricks notebooks!