Connect to a Databricks Workspace

Defining Credentials

The {brickster} package connects to a Databricks workspace in three ways:

  1. OAuth user-to-machine (U2M) authentication, either directly or through credentials managed by the Databricks CLI
  2. OAuth machine-to-machine (M2M) authentication
  3. Personal Access Tokens (PAT)

It’s recommended to use option (1) when using {brickster} interactively. If you need to run code via an automated process, use option (2) or (3).

{brickster} will automatically detect when a session has Posit Workbench managed Databricks OAuth credentials enabled. For more information about this authentication flow see the section Posit Workbench Managed Databricks OAuth Credentials.

Personal Access Tokens can be generated in a few steps, for a step-by-step breakdown refer to the documentation.

Once you have a token you’ll be able to store it alongside the workspace URL in an .Renviron file. The .Renviron is used for storing the variables, such as those which may be sensitive (e.g. credentials) and de-couple them from the code additional reading.

To get started add the following to your .Renviron:

DATABRICKS_WSID is only required for the RStudio IDE integration with the connection pane.

Example of entries in .Renviron:

DATABRICKS_HOST=xxxxxxx.cloud.databricks.com
DATABRICKS_TOKEN=dapi123456789012345678a9bc01234defg5
DATABRICKS_WSID=123123123123123

For OAuth M2M:

DATABRICKS_HOST=xxxxxxx.cloud.databricks.com
DATABRICKS_CLIENT_ID=11111111-2222-3333-4444-555555555555
DATABRICKS_CLIENT_SECRET=abcdefg1234567890
DATABRICKS_WSID=123123123123123

For Azure service principal OAuth M2M:

DATABRICKS_HOST=adb-xxxxxx.xx.azuredatabricks.net
ARM_CLIENT_ID=11111111-2222-3333-4444-555555555555
ARM_CLIENT_SECRET=abcdefg1234567890
ARM_TENANT_ID=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee

With no auth_type in a selected .databrickscfg profile, {brickster} attempts Databricks OAuth M2M (DATABRICKS_CLIENT_*), then Azure service principal OAuth M2M (ARM_*), then OAuth U2M. When using .databrickscfg, set auth_type = azure-client-secret in the profile to force Azure service principal authentication.

Note: Recommend creating an .Renviron for each project. You can create .Renviron within your user home directory if required.

Restarting your R session will allow those variable to be picked up via the {brickster} package.

Using Credentials with {brickster}

Authentication should now be possible without specifying the credentials in your R code. You can load {brickster} and list the clusters within the workspace using db_cluster_list(), to access the host/token use db_host()/db_token() respectively.

library(brickster)

# using db_host() and db_token() to get credentials
clusters <- db_cluster_list(host = db_host(), token = db_token())

All {brickster} functions have their host/token parameters default to calling db_host()/db_token() therefore we can omit explicit calls to the functions.

# all host/token parameters default to db_host()/db_token()
clusters <- db_cluster_list()

When using OAuth U2M or OAuth M2M authentication you don’t define a token in .Renviron and therefore db_token() will return NULL.

Managing Multiple Credentials

There are two methods that {brickster} supports to simplify switching of credentials within an R project/session:

  1. Adding multiple credentials to .Renviron, each additional set of credentials is differentiated via a suffix (e.g. DATABRICKS_TOKEN_DEV)
  2. Using a .databrickscfg file (primary method in Databricks CLI)

The use_databrickscfg option selects which configuration source {brickster} reads. It does not sign in to the Databricks CLI or select a named profile. The following example switches the session from .Renviron to .databrickscfg:

# uses the CLI-selected default profile, or `DEFAULT` when none is selected
options(use_databrickscfg = TRUE)

# values are read from `.databrickscfg`
db_host()
db_token()

With its default value of FALSE, {brickster} does not discover profiles created by databricks auth login and does not enable the databricks-cli authentication provider. Set the option in .Rprofile when .databrickscfg should be the normal configuration source. Posit Workbench managed OAuth sessions enable .databrickscfg automatically.

Switching Between Credentials

When .databrickscfg is enabled, the selected profile controls the host, credentials, authentication type, and the profile passed to databricks auth token. {brickster} resolves the profile in this order:

  1. DATABRICKS_CONFIG_PROFILE
  2. The db_profile option
  3. The CLI default selected by databricks auth switch
  4. The DEFAULT profile

DATABRICKS_CONFIG_PROFILE is the standard Databricks unified authentication setting. db_profile is the {brickster} session-level equivalent.

Profiles enable you to switch contexts between:

This behaviour works when using credentials specified in either .Renviron or .databrickscfg:

# using .Renviron
db_host() # returns `DATABRICKS_HOST` (.Renviron)

# switch profile to 'prod'
options(db_profile = "prod")
db_host() # returns `DATABRICKS_HOST_PROD` (.Renviron)

# clear the session-specific profile selection
options(db_profile = NULL)
# use .databrickscfg
options(use_databrickscfg = TRUE)
db_host() # returns the CLI-selected default, otherwise `DEFAULT` (.databrickscfg)

options(db_profile = "prod")
db_host() # returns host from `prod` profile (.databrickscfg)

It is expected that profiles in .Renviron will adhere to the same naming convention as default but add an additional suffix.

Here is an example of an .Renviron file that has three profiles (default, dev, prod):

# default
DATABRICKS_HOST=xxxxxxx.cloud.databricks.com
DATABRICKS_TOKEN=dapixxxxxxxxxxxxxxxxxxxxxxxxx
DATABRICKS_WSID=123123123123123
# dev
DATABRICKS_HOST_DEV=xxxxxxx-dev.cloud.databricks.com
DATABRICKS_TOKEN_DEV=dapixxxxxxxxxxxxxxxxxxxxxxxxx
DATABRICKS_WSID_DEV=123123123123124
# prod
DATABRICKS_HOST_PROD=xxxxxxx-prod.cloud.databricks.com
DATABRICKS_TOKEN_PROD=dapixxxxxxxxxxxxxxxxxxxxxxxxx
DATABRICKS_WSID_PROD=123123123123125

Configuring .databrickscfg

For details on configuring please refer to documentation from Databricks CLI.

There is only one {brickster} specific feature and it is the inclusion of wsid alongside host/token.

Supported OAuth fields in a .databrickscfg profile include:

Databricks CLI OAuth profiles

Profiles created by databricks auth login use auth_type = databricks-cli. The login can be performed from any terminal; the credentials are not scoped to that terminal session. {brickster} must run as the same operating-system user and use the same .databrickscfg and CLI credential storage.

The databricks-cli provider is intentionally available only when the selected .databrickscfg profile contains auth_type = databricks-cli. Environment configuration cannot activate this provider independently.

Enable .databrickscfg and select the profile before making requests:

options(
  use_databrickscfg = TRUE,
  db_profile = "e2-demo"
)

# The host and CLI token are resolved from the same profile.
db_sql_warehouse_list()

Instead of db_profile, set DATABRICKS_CONFIG_PROFILE, or select the CLI default outside R:

databricks auth switch --profile e2-demo

When a profile is selected, {brickster} calls databricks auth token --profile <profile>. Without a selected profile, {brickster} follows the Databricks SDK behavior and asks the CLI for the resolved host. If multiple profiles contain the same host, the CLI refuses to guess; select a profile rather than signing in again. Avoid supplying a different host argument while a profile is selected, because the request host and CLI credential would no longer describe the same configuration.

Token acquisition is deferred until an API request is performed. {brickster} keeps only the short-lived access token in memory until it expires, then calls the CLI again. The Databricks CLI remains responsible for durable credential storage and refresh. The CLI must be available on PATH, or its executable can be provided with DATABRICKS_CLI_PATH. DATABRICKS_CONFIG_FILE is inherited by the CLI when using a non-default config location.

The oauth-u2m mode remains {brickster}’s direct browser-based OAuth flow; it does not reuse credentials created by databricks auth login.

wsid is used by the connections pane integration in RStudio as the underlying API’s require it.

Posit Workbench Managed Databricks OAuth Credentials

Posit Workbench has a managed Databricks OAuth credentials feature, which allows users to sign into a Databricks workspace from the home page of Workbench when launching a session and then access Databricks resources as their own identity. When in an RStudio Pro session running on Posit Workbench with managed Databricks OAuth credentials selected, {brickster} functions using db_host()/db_token() respectively should just work without needing to specify any credentials in your R code. See the code below as an example.

library(brickster)
db_cluster_list()

{brickster} will automatically detect when a session has Workbench managed OAuth credentials and then use the workbench profile defined in a .databrickscfg file at the DATABRICKS_CONFIG_FILE specified location. Workbench generates this .databrickscfg file in a temporary directory and should not be modified directly.

To use an alternative .databrickscfg file, a different profile, an alternative env variable DATABRICKS_HOST or set an env variable DATABRICKS_TOKEN, launch an RStudio Pro session without the Databricks managed credentials box selected.