Two months ago I upgraded the motherboard in my home Linux PC to accommodate an Intel i9 CPU, specifically the Intel Core i9-10900 Comet Lake 10-Core CPU running at 2.8 GHz paired with the Cooler Master MasterLiquid ML240R CPU liquid cooler. This box runs production market data capture as well as a test trading strategy that I am working on, but I am still doing backtesting on my little Mac Mini desktop. That just won't do: those 10 cores are just begging to be used for accelerating backtests as well. To tackle this we'll revisit Kubernetes and add Dask cluster support.
Dask has native Kubernetes support, but we want to take more control over the scheduler and worker setup. This can be done easily with a simple YAML file:
apiVersion: v1
kind: Service
metadata:
name: daskd-scheduler
labels:
app: daskd
role: scheduler
spec:
ports:
- port: 8786
targetPort: 8786
name: scheduler
- port: 8787
targetPort: 8787
name: bokeh
- port: 9786
targetPort: 9786
name: http
selector:
app: daskd
role: scheduler
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: daskd-scheduler
spec:
replicas: 1
selector:
matchLabels:
app: daskd
role: scheduler
template:
metadata:
labels:
app: daskd
role: scheduler
spec:
containers:
- name: scheduler
image: cloudwallcapital/serenity:2020.10.23-b47
imagePullPolicy: Always
command: ["/app/venv-py3/bin/dask-scheduler"]
resources:
requests:
cpu: 1
memory: 10000Mi # set aside some extra resources for the scheduler
ports:
- containerPort: 8786
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: daskd-worker
spec:
replicas: 9
selector:
matchLabels:
app: daskd
role: worker
template:
metadata:
labels:
app: daskd
role: worker
spec:
containers:
- name: worker
image: cloudwallcapital/serenity:2020.10.23-b47
imagePullPolicy: Always
command: [
"/bin/bash",
"-cx",
"/app/venv-py3/bin/dask-worker $DASKD_SCHEDULER_SERVICE_HOST:$DASKD_SCHEDULER_SERVICE_PORT_SCHEDULER --nthreads 1"
]
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: database-secret-config
key: password
- name: AZURE_CONNECT_STR
valueFrom:
secretKeyRef:
name: azure-secret-config
key: connect-str
resources:
requests:
cpu: 1
memory: 6000Mi
To make this work our Dockerfile needs to build the virtualenv:
FROM python:3.8-slim-buster
RUN apt-get update && apt-get install --yes gcc libpq-dev virtualenv
COPY $PWD/src /app
COPY $PWD/strategies /strategies
COPY $PWD/requirements.txt /app
WORKDIR /app
RUN virtualenv venv-py3 --python=/usr/local/bin/python3
RUN /app/venv-py3/bin/pip install --upgrade pip
RUN /app/venv-py3/bin/pip install -r requirements.txt
ENV PYTHONPATH "${PYTHONPATH}:/app"
After building this setup you can run
$ microk8s.kubectl apply -f kubernetes/dask.yaml
to instantiate the cluster and its host:port mappings. And now with the awesome power of Kubernetes let's increase the number of workers by 10x with a single command:
$ microk8s.kubectl scale deployment daskd-worker --replicas=100
deployment.apps/daskd-worker scaled
and then scale down to three workers with another command:
$ microk8s.kubectl scale deployment daskd-worker --replicas=3
deployment.apps/daskd-worker scaled
We can now hit the external NodePort we configured for the dask dashboard (http://charger:30787/workers in my case):
We now have a running Dask cluster. Let's try to connect to it from Jupyter and send some work; note here I'm using the Jupyter Lab Kubernetes instance we set up before, so the host:port is the one local to Kubernetes:
Finally, let's send a Serenity backtest to Dask. To do this we need to invoke the backtester programmatically, which we can do easily by importing the main function:
from serenity.algo.backtester import main
# define the backtest parameters
def run_backtest():
main('/strategies/bbands1_backtest_config.yaml', '2020-09-12T00:00:00', '2020-09-13T00:00:00', '/strategies')
# go!
bt = client.submit(run_backtest)
bt.result()
aaaaaaand ... we crash the workers. But have no fear, Kubernetes makes it very easy for us to boost the memory available to each worker:
resources:
requests:
cpu: 1
memory: 10000Mi
and we need to add --memory-limit 0 to the worker command line so Dask will not try to preemptively limit memory usage; since Kubernetes caps the container size this is unnecessary.
This is enough to run a backtest remotely, but it's difficult to extract the results of the backtest with the current Serenity AlgoBacktester API -- you can only get it out of the Kubernetes logs. We'll revisit this in a future post when we look at strategy analytics.
