Install from the Microsoft Marketplace¶
This page shows how to install doubleml-pyspark after subscribing in the
Microsoft Marketplace. It works on Databricks serverless compute and on classic
clusters.
What you need¶
- A subscription to DoubleML for PySpark in the Azure portal, and the license
key from the activation page (it starts with
DL-) - An Azure Databricks workspace with a supported runtime (see Supported environments)
- Internet access from the workspace. If your workspace restricts outbound traffic, allow
license.doubleml-cloud.com,pypi.organdfiles.pythonhosted.org.
Supported environments¶
| Environment | Spark | Python | Status | Notes |
|---|---|---|---|---|
| Databricks serverless, environment version 4, 5 or 6 | 4.x (Spark Connect) | 3.12 | Supported | |
| Databricks Runtime 19 | 4.2 | 3.12 | Supported | |
| Databricks Runtime 18 LTS | 4.1 | 3.12 | Supported | |
| Databricks Runtime 17.3 LTS | 4.0 | 3.12 | Supported | |
| Databricks Runtime 16.4 LTS | 3.5 | 3.12 | Supported | Dedicated access mode only, from version 0.1.3 |
| Databricks Runtime 15.4 LTS and older | 3.5 | 3.10, 3.11 | Not supported | |
| Other Spark platforms and local use | 3.5.2 or newer | 3.12 | Supported | Linux x86_64, including WSL |
| Windows and macOS (native) | Not supported | Use WSL or Databricks |
With Spark 4, both access modes work on clusters: dedicated (single user) and standard (shared). On Databricks Runtime 16.4 LTS (Spark 3.5), use a cluster in dedicated access mode: Spark 3.5 runs Spark ML only there, not in standard access mode or on serverless compute. Support for more Python versions is planned. If you need another environment, write to cloud@economicai.com.
1. Get your license key¶
- In the Azure portal, open SaaS and select your subscription.
- Click Open SaaS Account on publisher's site and sign in with your work account.
- Confirm the plan and click Activate. The next page shows your license key.
You can come back to this page at any time to see the key again.
2. Store the key in a Databricks secret¶
Store the key once per workspace, with the Databricks CLI:
Paste the key at the prompt. doubleml-pyspark reads the key from this secret
when you fit a model, so it never has to appear in a notebook.
3. Install the package¶
In a notebook (serverless or cluster), run these cells:
The package index uses your license key as password. Other packages (numpy, pandas, ...) are taken from PyPI as usual.
To pin a version, use for example doubleml-pyspark==0.1.2.
4. Check the installation¶
import doubleml_pyspark
from doubleml_pyspark.datasets import make_plr_data
print(doubleml_pyspark.__version__)
df = make_plr_data(spark, n_obs=100_000)
df.show(5)
Then continue with Getting Started.
Other platforms¶
On a local machine or another Spark platform, install from our package index with your license key. The spark option also installs PySpark, leave it out if Spark is already there:
pip install --index-url https://license:<KEY>@license.doubleml-cloud.com/simple "doubleml-pyspark[spark]"
export DOUBLEML_LICENSE_KEY=<KEY>
With uv, the same in one line:
uv pip install --index-url https://license:<KEY>@license.doubleml-cloud.com/simple "doubleml-pyspark[spark]"
In a uv project, take only doubleml-pyspark from our index and everything else from
PyPI. Add to pyproject.toml:
[[tool.uv.index]]
name = "doubleml"
url = "https://license.doubleml-cloud.com/simple"
explicit = true
[tool.uv.sources]
doubleml-pyspark = { index = "doubleml" }
[tool.uv]
# the package is built for Linux x86_64
environments = ["sys_platform == 'linux' and platform_machine == 'x86_64'"]
and pass the key as environment variables (it is not written to uv.lock):
export UV_INDEX_DOUBLEML_USERNAME=license
export UV_INDEX_DOUBLEML_PASSWORD=<KEY>
uv add "doubleml-pyspark[spark]"
This needs Python 3.12 and Linux x86_64, including WSL. PySpark 3.5.2 works in classic mode only.
Troubleshooting¶
| Message | Cause | What to do |
|---|---|---|
No doubleml-pyspark license key found |
Secret missing or named differently | Create doubleml / license-key (step 2), or set the environment variable DOUBLEML_LICENSE_KEY |
pip reports 401 |
Key not passed or mistyped | Check cell 1. The key starts with DL- |
pip reports 403, or License invalid |
Subscription cancelled, suspended or expired | Check the subscription status in the Azure portal |
No matching distribution found |
Python version is not 3.12 | Use serverless environment 4 to 6 or Databricks Runtime 16.4 LTS or newer |
| Installation times out | The workspace cannot reach the package index | Allow outbound traffic to license.doubleml-cloud.com, pypi.org and files.pythonhosted.org |
If the license service is briefly unreachable, a successful check from the last 24 hours lets your jobs continue.
Questions: see Support.