Khushnaseeb Log - VIDA-NYU/reprozip GitHub Wiki
On Fridays, please devote at least a half hour to reflect on the work you've done over the previous week. Please use this format:
Week-1
Monday, May 21, 2018
- Took a Reprozip overview and tried unnpacking the digits-sklearn-opencv example on the local machine from the reprozip-examples repository.
- Went through Reprozip developer doc and tried to understand how to write an unpacker. Reported a broken link in the documentation on the Dynamic discovery of Services and Plugins.
Tuesday, May 22, 2018
- Went through the following links and videos to understand Containers and Docker
Wednesday, May 23, 2018
- Jumpstarted with the official Docker documentation to get hands-on on Docker.
- Learned to how to create images and containers, run a container and publishing an image to Docker Hub.
- Created services using docker-compose and scaled the application using it.
- Learned how to put placement constraints and attach volumes so as to persist data even when the container is destroyed
- Learned what a swarm is, how nodes in swarms can be managers or workers, created a swarm, and deployed an application on it
- Created and configure a Stack(a group of interrelated services) using Docker compose and added services to it
Friday, May 25, 2018
- Tried to get a basic understanding of Singularity via following videos:
- Tried to understand the key differences between Singularity and Docker and why the former is a good choice for HPC. Related links:
- Went through Singularity official documentation to try out some examples.
Week-2
Tuesday, May 29, 2018
- Jump started with Singularity official documentation to try out the following
- building a singularity container image fro docker, shub, and directory
- building a writable image
- Understanding and writing a singularity recipe file
- Understanding the singularity workflow and building apps
- creating and running singularity services/instances
Wednesday, May 31, 2018
- Making persistent singularity overlays and learning how to bind mounts and paths to enable read and write files on the host system from within the container.
- learned how to make singularity apps and how they share environment variables
- understanding singularity's environment variables
- had a discussion to come with a design to leverage singularity for reprozip: -- reproducing singularity image directory from reperozip by adding additional singularity files --- using app structure in singularity for replicating runs in reprozip
Week-3
Monday, June 04, 2018 - Wednesday, June 06, 2018
- Made and ran a singularity image by hand from a reprozip package in a sandbox format.
Week-4
Monday, June 11, 2018 - Tuesday, June 12, 2018
- Wrote a bash script to automatically create and run the singularity image for a single run reprozip package
Wednesday, June 13, 2018
- Tried emulating multiple runs via singularity apps feature.
Week-5
Monday, June 18, 2018
- Converted the bash script to python.
Tuesday, June 19, 2018 - Wednesday, June 20, 2018
- Modified the script for multiple runs feature (without using singularity apps) and successfully tested it on digits_sklearn_opencv.rpz package.
Week-6
Monday, June 25, 2018 - Tuesday, June 26, 2018
- Tried to make the singularity image for the reprozip package in the TAR format.
Wednesday, June 27, 2018
- Successfully used Singularity's app feature to emulate multiple runs in a reprozip package for example in digits_sklearn_openc.rpz example - two separate apps for classification and prediction both which can be run independently (but in order, if no dependencies are present)
Week-7
Monday, July 02, 2018 - Tuesday, July 03, 2018
- Successfully wrote a script to automatically create and run the singularity image for the reprozip package in a tar format.
- Tested the script on various packages - single and multiple runs.
Week-8
Monday, July 08, 2018 - Wednesday, July 10, 2018
- Submitted for code review and implemented code review changes
Week-9
Monday, July 15, 2018 - Thursday, July 18, 2018
- Reimplemented the tar functionality:
- removed any files that were generated on disk. Wrote code for creating everything in memory and adding the contents as a file to the tar file using BytesIO.
- Added empty folders directly to the tar file without having the need of them on the disk.
Week-10
Monday, July 22, 2018 - Tuesday, July 23, 2018
- Added the functionality of running the tar image of multiple runs directly and not just running the individual run as an app. This allows one to upload the singularity image of singularity hub and run without the reprozop unpacker script.
Week-11
Monday, July 29, 2018 - Tuesday, July 30, 2018
- Tested the unpacker ( both single and multiple run) on HPC and it worked!
Week-12
Monday, Aug 05, 2018 - Wednesday, Aug 07, 2018
- Fixed the upload and download issue - now able to download/upload to the singularity image from anywhere to the host to anywhere in the container and vice versa.
- Removed manual adding of bin/sh. Added the functionality to download the busybox url in memory and then add it as a file to the tar file.
Week-13
Monday, Aug 12, 2018 - Thursday, Aug 16, 2018
- Understanding MPI and how MPI applications run with singularity. Tried running a reprozip package with MPI.
- Documented singularity unpacker for future use with reprozip.