Cluster Manager

Goals:

• Invoke on demand
• Mount individual user's home (NFS)
• MPI compatible
• Handle partial fleet startup (failures or allocation)
• Secured through OAuth and/or JupyterHub
• Results accessible in Jupyter
• Custom Docker image
• Secure to user
• VPC security
• User can monitor real-time
• Disk usage limits for shared partitions
• User directed cancel/restart (same cluster)
• Results archive bound with container (id)
• API based
• In-browser visualization (Jupyter?)

Workflow:

• User specified command and directory
• Local and shared scratch
• Shared results
• User notified when cluster starts
• User monitors results via Jupyter
• User can cancel or edit and restart

GIBBERISH FOLLOWS

Enough FOSS choices so only looked at those.

not IT focused (no single cluster or provider) MPI out of the box no needuser empowered (docker) web focused (not bound to unix user ids)

Docker for all components

Limit access to NFS mounts

Bare metal access Infiniband, shared memory

two classes: web app and hpc

MPI needs bare mmetal for RDMA over Infiniband

not run as root

We would like to use MPI on Docker with arbitrarily configured clusters (e.g. created with StarCluster or bare metal). What I'm curious about is if there is a queue manager that understands Docker, file systems, MPI, and OpenAuth. JupyterHub does a lot of this, but it doesn't interface with MPI. Ideally, we'd like users to be able to queue up jobs directly from JupyterHub. Currently, we can configure and initiate an MPI-compatible Docker cluster running on a VPC using Salt. What's missing is the ability to manage a queue of these clusters. Here's a list of requirements:

• JupyterHub users do not have Unix user ids
• Containers must be started as a non-root guest user (--user)
• JupyterHub user's data directory is mounted in container
• Data is shared via NFS or other cluster file system
• sshd runs in container for MPI as guest user
• Results have to be reported back to GitHub user
• MPI network must be visible (--net=host)
• Queue manager must be compatible with the above
• JupyterHub user is not allowed to interact with Docker directly
• Docker images are user selectable (from an approved list)
• Jupyter and MPI containers started from same image Know of a system which supports this? Our code and config are open source, and your feedback would be greatly appreciated. Salt configuration: https://github.com/radiasoft/salt-conf Container builders: https://github.com/radiasoft/containers/tree/master/radiasoft Early phase wiki: https://github.com/radiasoft/devops/wiki/DockerMPI

Thanks, Ralph. I'm not sure I explained the problem clearly. Salt and JupyterHub are distractions, sorry. I have code which "wires up" a cluster for MPI. What I need is scheduler that allows users to:

• Select which Docker image they'd like to wire up
• Request a number of nodes/cores
• Understands that clusters can be dynamically created
• Invokes an external command to create the cluster Here's what I'd like the user be able to do: $ queue-job --image=radiasoft/beamsim --cores=5000 my-script.sh queue-job would then have to be able to call a 3rd-party module to get the user:

3rd-party-environ-collector

This command would return a logical user and a network-accessible directory. This info would be added to the queue, and then when the scheduler decided to start the job, it would call:

3rd-party-start --image=radiasoft/beamsim --cores=5000 --user=robnagler

--mount=nfs:// intra.server.com/var/nfs/bla/robnagler/foo/bar:/home/docker-user/run my-script.sh The bit that I need is the scheduling. For example, the scheduler would have give the user a maximum number of core hours. It would maybe give the job a unique group id (a la Grid Engine) to manage disk quotas. These constraints would need to be passed to the 3rd-party programs so they could constrain the doctor container. What I have now is 3rd-party-environ-collector (JupyterHub) and 3rd-party-start (Salt). What I need is a scheduler that has an architecture that supports dynamic clusters and users who have no local credentials (non-Unix user -- just a name and a home directory).