Temporary Page To Record JW Overhead before merging with Overhead Section - casangi/RADPS GitHub Wiki

#Kubernetes Experiment on MacOS. (using readme instructions)

sudo port install k3d
sudo port install kubectl
sudo port select --set kubectl kubectl1.32
sudo port install helm-3.18
sudo port select --set helm helm3.18

Run k3d version:

k3d version v5.8.3
k3s version v1.31.5-k3s1 (default)

Run kubectl version:

Client Version: v1.32.2
Kustomize Version: v5.5.0

Run helm version:

version.BuildInfo{Version:"v3.18.1", GitCommit:"f6f8700a539c18101509434f3b59e6a21402a1b2", GitTreeState:"clean", GoVersion:"go1.24.3"}

Create Cluster

Ensure docker is running and then create cluster

k3d cluster create radps-demo --servers 1 --agents 0 --port "6200:4200@loadbalancer" --port "8000:80@loadbalancer" --port "30786:8787@loadbalancer"  

  --port "6200:4200@loadbalancer" \      # Prefect 
  --port "8000:80@loadbalancer" \      # JupyterHub 
  --port "30787:8787@loadbalancer"      # Dask dashboard 

Confirm cluster is running

The docker interface:

Run kubectl cluster-info:

Run kubectl get nodes for available nodes (should display one node (the server)):

Install & Configure Helm in the Cluster

Add the Prefect and JupyterHub Helm repositories

helm repo add prefect https://prefecthq.github.io/prefect-helm
helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
helm repo update

Create namespaces

kubectl create namespace prefect
kubectl create namespace jupyterhub

Deploy Prefect v3 Infrastructure

Create a values file for customization

# Prefect Server v3 Helm Values for k3d cluster
# Configured for external access via laptop IP

# Service configuration (top level)
service:
  type: LoadBalancer
  port: 4200
  targetPort: 4200
  # Annotations for k3d LoadBalancer
  annotations: {}

# Server image and configuration
server:
  # Environment variables for Prefect server
  env:
    - name: PREFECT_SERVER_API_HOST
      value: "0.0.0.0"
    - name: PREFECT_SERVER_API_PORT
      value: "4200"
    - name: PREFECT_UI_URL
      value: "http://10.2.113.66:4200"
    - name: PREFECT_API_URL
      value: "http://10.2.113.66/:4200/api"

  # Resource limits (adjust based on your needs)
  resources:
    requests:
      memory: "512Mi"
      cpu: "250m"
    limits:
      memory: "1Gi"
      cpu: "500m"

# PostgreSQL database configuration
postgresql:
  enabled: true
  auth:
    postgresPassword: "prefect"
    database: "prefect"
  primary:
    persistence:
      enabled: true
      size: 8Gi
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"
        cpu: "500m"

# Ingress configuration (disabled in favor of LoadBalancer)
ingress:
  enabled: false

Install the chart in your prefect namespace

helm install prefect-server prefect/prefect-server \
  --namespace prefect \
  --values prefect-values.yaml

To uninstall prefect

helm uninstall prefect-server --namespace prefect

Wait a minute or two for all pods to come up and run kubectl -n prefect get pods:

make sure the services are up with kubectl get svc -n prefect:

Deploy Jupyter Hub Infrastructure

Create jupyter-values.yaml

# jupyterhub-values.yaml
proxy:
  # The Hub’s proxy (configurable-http-proxy) will listen on 0.0.0.0:80 inside cluster
  service:
    type: ClusterIP
    port: 8000

hub:
  image:
    name: jupyterhub/k8s-hub
    tag: "2.3.0"   # adjust to latest stable as of mid-2025
  cookieSecret: "THIS_SHOULD_BE_RANDOM_FOR_PROD"
  # For simplicity: use a dummy authenticator so any username/password works
  config:
    Authenticator:
      className: dummy
      DummyAuthenticator:
        password: "zinc"   # anyone who uses any username but this password can log in
  service:
    type: ClusterIP
    port: 8081

singleuser:
  image:
    name: jupyter/base-notebook
    tag: "python-3.13.5"    # a recent Python 3.13 image
  storage:
    type: dynamic   # use default StorageClass in k3s (local-path-storage)
    dynamic:
      storageClass: local-path
      pvc:
        accessModes: ["ReadWriteOnce"]
  service:
    type: ClusterIP
    port: 8888

# Use a dummy external proxy (we rely on k3d's port mapping instead of an Ingress)

Install the chart in your Jupyter namespace

helm install jupyterhub jupyterhub/jupyterhub \
  --namespace jupyterhub \
  --values jupyter-values.yaml \
  --debug

helm uninstall jupyterhub --namespace jupyterhub

jupyterhub

###############################

Setup Conda/Mamba environment

mamba create -n zinc python=3.13
mamba activate zinc
mamba install prefect matplotlib prefect-sqlalchemy psycopg2-binary postgresql asyncpg

Create folder for Postgres (database will also reside here)

# cordierite MacOS
mkdir -p /Users/jsteeb/postgres_data 
initdb -D /Users/jsteeb/postgres_data

# Gravitas RedHat
mkdir -p /export/data_1/jsteeb/postgres_data 
initdb -D /export/data_1/jsteeb/postgres_data

# CVPost Master
mkdir -p /.lustre/cv/users/jsteeb/postgres_data 
initdb -D /.lustre/cv/users/jsteeb/postgres_data

Note: Don't use the cvfiler system as the database grows quickly (disk quota exceeded errors).

Modify /.lustre/cv/users/jsteeb/postgres_data/postgresql.conf

listen_addresses = 'localhost'
port = 5432 
max_connections = 1000 
shared_buffers = 2GB 
work_mem = 64MB 
maintenance_work_mem = 512MB 
effective_cache_size = 6GB

I suspect tuning this for our use case is going to be important.

Start Postgres and create database

pg_ctl -D /Users/jsteeb/postgres_data -l /Users/jsteeb/postgres_logfile start
createdb prefect_db
prefect config set PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://$(whoami)@localhost:5432/prefect_db"

pg_ctl -D /export/data_1/jsteeb/postgres_data -l /export/data_1/jsteeb/postgres_data/postgres_logfile start
createdb prefect_db
prefect config set PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://$(whoami)@localhost:5432/prefect_db"

pg_ctl -D /.lustre/cv/users/jsteeb/postgres_data -l /.lustre/cv/users/jsteeb/postgres_logfile start
createdb prefect_db
prefect config set PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://$(whoami)@localhost:5432/prefect_db"

Start prefect in screen

screen -R prefect
prefect server start

Note of useful commands:

ctrl + a + d      # exit screen
screen -ls        # see all available screens
screen -r prefect #reconect to screen

If you want to modify the postgress config

ctrl + c          # Kill prefect
dropdb prefect_db # Optionally delete database if it is getting large.
Killall postgres  # Should be a beter way
pg_ctl -D /.lustre/cv/users/jsteeb/postgres_data -l /.lustre/cv/users/jsteeb/postgres_logfile start
createdb prefect_db
prefect config set PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://$(whoami)@localhost:5432/prefect_db"
prefect server start

Using cvpost018 (16 cores), Postgress database and ThreadPoolTaskRunner: