Skip to content

Setting Up Databricks

This guide walks you through the required steps to configure Databricks on Azure for use with doubleml_pyspark.

Prerequisites

Requirements: - Azure account with appropriate permissions - Pay-as-you-go billing enabled - Sufficient CPU quota in your desired Azure region

Setup Steps

1. Create Azure Account

If you don't already have one, set up an Azure account at azure.microsoft.com.

2. Configure Pay-as-You-Go Resource Group

Create a new resource group in Azure with pay-as-you-go pricing enabled. This will be used to host your Databricks workspace.

3. Enable Databricks Service

Within your resource group, enable the Azure Databricks service. This may require additional permissions or subscription configurations.

4. Increase CPU Quota (if needed)

Depending on your Azure region, you may need to request additional CPU quota:

  • Navigate to Azure portal → Quotas
  • Select your region
  • Request an increase for the appropriate VM family
  • Wait for approval (typically 1-3 business days)

Note: Different Azure regions have different availability and quota limits. If you encounter quota issues, consider selecting a different region.

5. Access Databricks Workspace

Once Databricks is enabled, launch your Databricks workspace from the Azure portal.

6. Create a Cluster with Required Runtime

You can use serverless compute (environment version 4, 5 or 6) or a cluster. For a cluster, choose:

  • Databricks Runtime: 17.3 LTS, 18 LTS or 19 (Python 3.12, Spark 4), or 16.4 LTS (Spark 3.5) in dedicated access mode
  • Access mode: dedicated (single user) or standard

7. Install doubleml_pyspark Package

Store your license key in a Databricks secret and install the package in a notebook, as described in Install from the Microsoft Marketplace.

Verification

Once all steps are complete, you can verify your setup by:

  • Running import doubleml_pyspark; print(doubleml_pyspark.__version__) in a notebook
  • Confirming the runtime version matches the requirements above

Success! Your Databricks environment is now ready to use with doubleml_pyspark.