Setting Up Databricks¶
This guide walks you through the required steps to configure Databricks on Azure for use with doubleml_pyspark.
Prerequisites¶
Requirements: - Azure account with appropriate permissions - Pay-as-you-go billing enabled - Sufficient CPU quota in your desired Azure region
Setup Steps¶
1. Create Azure Account¶
If you don't already have one, set up an Azure account at azure.microsoft.com.
2. Configure Pay-as-You-Go Resource Group¶
Create a new resource group in Azure with pay-as-you-go pricing enabled. This will be used to host your Databricks workspace.
3. Enable Databricks Service¶
Within your resource group, enable the Azure Databricks service. This may require additional permissions or subscription configurations.
4. Increase CPU Quota (if needed)¶
Depending on your Azure region, you may need to request additional CPU quota:
- Navigate to Azure portal → Quotas
- Select your region
- Request an increase for the appropriate VM family
- Wait for approval (typically 1-3 business days)
Note: Different Azure regions have different availability and quota limits. If you encounter quota issues, consider selecting a different region.
5. Access Databricks Workspace¶
Once Databricks is enabled, launch your Databricks workspace from the Azure portal.
6. Create a Cluster with Required Runtime¶
You can use serverless compute (environment version 4, 5 or 6) or a cluster. For a cluster, choose:
- Databricks Runtime:
17.3 LTS,18 LTSor19(Python 3.12, Spark 4), or16.4 LTS(Spark 3.5) in dedicated access mode - Access mode: dedicated (single user) or standard
7. Install doubleml_pyspark Package¶
Store your license key in a Databricks secret and install the package in a notebook, as described in Install from the Microsoft Marketplace.
Verification¶
Once all steps are complete, you can verify your setup by:
- Running
import doubleml_pyspark; print(doubleml_pyspark.__version__)in a notebook - Confirming the runtime version matches the requirements above
Success! Your Databricks environment is now ready to use with doubleml_pyspark.