Skip to content

Install from the Microsoft Marketplace

This page shows how to install doubleml-pyspark after subscribing in the Microsoft Marketplace. It works on Databricks serverless compute and on classic clusters.

What you need

  • A subscription to DoubleML for PySpark in the Azure portal, and the license key from the activation page (it starts with DL-)
  • An Azure Databricks workspace with a supported runtime (see Supported environments)
  • Internet access from the workspace. If your workspace restricts outbound traffic, allow license.doubleml-cloud.com, pypi.org and files.pythonhosted.org.

Supported environments

Environment Spark Python Status Notes
Databricks serverless, environment version 4, 5 or 6 4.x (Spark Connect) 3.12 Supported
Databricks Runtime 19 4.2 3.12 Supported
Databricks Runtime 18 LTS 4.1 3.12 Supported
Databricks Runtime 17.3 LTS 4.0 3.12 Supported
Databricks Runtime 16.4 LTS 3.5 3.12 Supported Dedicated access mode only, from version 0.1.3
Databricks Runtime 15.4 LTS and older 3.5 3.10, 3.11 Not supported
Other Spark platforms and local use 3.5.2 or newer 3.12 Supported Linux x86_64, including WSL
Windows and macOS (native) Not supported Use WSL or Databricks

With Spark 4, both access modes work on clusters: dedicated (single user) and standard (shared). On Databricks Runtime 16.4 LTS (Spark 3.5), use a cluster in dedicated access mode: Spark 3.5 runs Spark ML only there, not in standard access mode or on serverless compute. Support for more Python versions is planned. If you need another environment, write to cloud@economicai.com.

1. Get your license key

  1. In the Azure portal, open SaaS and select your subscription.
  2. Click Open SaaS Account on publisher's site and sign in with your work account.
  3. Confirm the plan and click Activate. The next page shows your license key.

You can come back to this page at any time to see the key again.

2. Store the key in a Databricks secret

Store the key once per workspace, with the Databricks CLI:

databricks secrets create-scope doubleml
databricks secrets put-secret doubleml license-key

Paste the key at the prompt. doubleml-pyspark reads the key from this secret when you fit a model, so it never has to appear in a notebook.

3. Install the package

In a notebook (serverless or cluster), run these cells:

key = dbutils.secrets.get("doubleml", "license-key")
%pip install -q --index-url https://license:$key@license.doubleml-cloud.com/simple doubleml-pyspark
dbutils.library.restartPython()

The package index uses your license key as password. Other packages (numpy, pandas, ...) are taken from PyPI as usual.

To pin a version, use for example doubleml-pyspark==0.1.2.

4. Check the installation

import doubleml_pyspark
from doubleml_pyspark.datasets import make_plr_data

print(doubleml_pyspark.__version__)
df = make_plr_data(spark, n_obs=100_000)
df.show(5)

Then continue with Getting Started.

Other platforms

On a local machine or another Spark platform, install from our package index with your license key. The spark option also installs PySpark, leave it out if Spark is already there:

pip install --index-url https://license:<KEY>@license.doubleml-cloud.com/simple "doubleml-pyspark[spark]"
export DOUBLEML_LICENSE_KEY=<KEY>

With uv, the same in one line:

uv pip install --index-url https://license:<KEY>@license.doubleml-cloud.com/simple "doubleml-pyspark[spark]"

In a uv project, take only doubleml-pyspark from our index and everything else from PyPI. Add to pyproject.toml:

[[tool.uv.index]]
name = "doubleml"
url = "https://license.doubleml-cloud.com/simple"
explicit = true

[tool.uv.sources]
doubleml-pyspark = { index = "doubleml" }

[tool.uv]
# the package is built for Linux x86_64
environments = ["sys_platform == 'linux' and platform_machine == 'x86_64'"]

and pass the key as environment variables (it is not written to uv.lock):

export UV_INDEX_DOUBLEML_USERNAME=license
export UV_INDEX_DOUBLEML_PASSWORD=<KEY>
uv add "doubleml-pyspark[spark]"

This needs Python 3.12 and Linux x86_64, including WSL. PySpark 3.5.2 works in classic mode only.

Troubleshooting

Message Cause What to do
No doubleml-pyspark license key found Secret missing or named differently Create doubleml / license-key (step 2), or set the environment variable DOUBLEML_LICENSE_KEY
pip reports 401 Key not passed or mistyped Check cell 1. The key starts with DL-
pip reports 403, or License invalid Subscription cancelled, suspended or expired Check the subscription status in the Azure portal
No matching distribution found Python version is not 3.12 Use serverless environment 4 to 6 or Databricks Runtime 16.4 LTS or newer
Installation times out The workspace cannot reach the package index Allow outbound traffic to license.doubleml-cloud.com, pypi.org and files.pythonhosted.org

If the license service is briefly unreachable, a successful check from the last 24 hours lets your jobs continue.

Questions: see Support.