Deployment

Deploying on Kubernetes/SLURM/Cloud sucks. There's a lot to figure out.

Dask already figured this out, so we cheat and hijack Dask clusters.

Hijack

Frisky can deploy itself on top of any running Dask cluster.

# Set up Dask on whatever kind of cluster you have (k8s, AWS, SLURM)
from dask_whatever import WhateverCluster

dask_cluster = WhateverCluster(...)
dask_client = dask_cluster.get_client()

# Hijack that cluster with Frisky
import frisky

client = frisky.hijack(dask_client)

That's it. Frisky loads everything it needs to on top of Dask processes. You get the same hardware, the same security, and even the dashboard gets upgraded.

For Coiled *.dask.host dashboards, hijack uses public DNS to dodge stale local negative caches. Other dashboard hosts use the system resolver by default, which matters for Kubernetes and split-horizon DNS. Pass use_public_dns=False to disable the Coiled workaround, or True to force it.

Plus, we didn't have to write a single line of yaml.

The Hard Way

Or maybe you like setting things up yourself. No worries, we get you.

Frisky is simple to set up. Deploy a scheduler and many workers pointing to that scheduler.

$ frisky scheduler
Scheduler running at localhost:8786

$ frisky worker localhost:8786
$ frisky worker localhost:8786
$ frisky worker localhost:8786

Then connect to that scheduler with a Client from Python.

from frisky import Client

client = Client("127.0.0.1:8786")

Across machines, bind the scheduler somewhere the workers can reach and point them at it:

$ frisky scheduler --address 0.0.0.0:8786

$ frisky worker scheduler.example.com:8786   # on each worker machine

Each worker listens for peer data fetches on the local address that routes to the scheduler, so other workers get an address they can reach. On a multi-homed machine where peers should use a different interface, name it with --listen-address; binding a wildcard (0.0.0.0) is fine, and the worker advertises a concrete address in its place. When the address peers should dial is not one the worker can bind at all — NAT, port forwarding, or a Kubernetes Service in front of the worker — advertise it separately with --contact-address.

That port carries data between workers and has no authentication of its own — it is open to anything that can reach the address it binds. On a private network that is the usual arrangement, and it matches what Dask does; anywhere less trusted, use TLS below, which authenticates peers by certificate.

TLS

If your cluster runs somewhere untrusted, prefix addresses with tls:// and point everything at your certificates.

$ frisky scheduler --address tls://0.0.0.0:8786 \
    --tls-cert cert.pem --tls-key key.pem --tls-ca-file ca.pem

$ frisky worker tls://scheduler.example.com:8786 \
    --tls-cert cert.pem --tls-key key.pem --tls-ca-file ca.pem

client = Client(
    "tls://scheduler.example.com:8786",
    tls_cert="cert.pem",
    tls_key="key.pem",
    tls_ca_file="ca.pem",
)

The CA file is what everyone verifies against, so workers and clients always need it — without it they trust nobody and every handshake fails. On the scheduler it does double duty: giving it a CA also turns on mutual TLS, so workers and clients must present a certificate signed by that CA. Give the scheduler a cert and key but no CA and you get encryption and server authentication, but anyone who can reach the port can connect.

Three things to watch out for.

A worker's advertised address is its identity to other workers. By default it picks the local address that routes to the scheduler, which is what you want almost everywhere, but each worker's certificate has to cover whatever address it ends up advertising — including the case where that turns out to be an IPv6 address. Override it with --listen-address when peers should dial a different interface, or with --contact-address when the advertised name should differ from the bound one — for example a per-worker DNS name that the certificate actually covers.

If issuing a certificate per host is awkward, share one certificate and pass --tls-server-name-override so peers verify against a fixed name. Note that this overrides the name on every outbound connection from that worker, including the one to the scheduler — so the scheduler's certificate has to cover the shared name too, and the scheduler will see the shared name as the SNI rather than its own hostname.

The dashboard is always plain HTTP. Put a TLS-terminating proxy in front of it if it needs to be reachable over an untrusted network.

Outbound connections use the address hostname as the TLS server name, so SNI-routing gateways see the name you dialed. This only works for DNS names: dialing an IP literal sends no SNI at all, and the certificate then needs an IP entry in its subject alternative names.

LocalCluster

There's a convenient LocalCluster that does this for you on one machine:

import frisky

cluster = frisky.LocalCluster()
client = cluster.get_client()

Cloud / Coiled

Frisky is mostly developed on Coiled, a for-profit Dask service that runs on cloud.

Frisky and Coiled are owned by the same person, so it's likely that they'll continue to work well together.

Here's a simple case:

import coiled, frisky

cluster = coiled.Cluster()
client = frisky.hijack(cluster.get_client())

And a more evocative one:

import coiled
import frisky

# Set up Dask cluster on the cloud
cluster = coiled.Cluster(
    n_workers=20,
    worker_memory="16 GiB",
    region="us-east-1",
    arm=True,
)
dask_client = cluster.get_client()

# Take it over with Frisky
client = frisky.hijack(dask_client)