WG meetings 2024 - mesoscope/cellpack GitHub Wiki

October 29th, 2024

Updates and agenda:

  1. PR open for job_id storage

TODO:

  1. start looking at querying firebase for the job id and url

October 15th, 2024

Updates and agenda:

  1. APIs are working now. We can get logs, get packing status and submit packing
    1. Alli has created github issues.
    2. Using AWS API Gateway to trigger batch jobs
  2. Currently getting simularium path from logs
    1. Could get it from packing status?
    2. Could use the jobid to read status?

TODO:

  1. Saurabh: connect with Nivi to go over talk outline for ASCB subgroup
  2. Saurabh: Bring up discussion of potential new data collection for cellPACK data analysis at next scisim WG
  3. Saurabh: fix scaled distance calculations
  4. Saurabh: occupancy plots using different distance metrics
  5. Saurabh: use images to calculate distance distributions directly
  6. Saurabh: [stretch] compare occupancy across cell stages - discuss at next scisim WG?
  7. Use prenv + pdm/poetry to install cellPACK and move away from conda
  8. Megan: continue clustering to convert pdbs to spheretrees

October 1st, 2024

Updates and agenda:

  1. Go over cell morph WGF notes by Jess: https://github.com/mesoscope/cellpack-analysis/issues/27
  2. Share slides with initial results for ER, peroxisome, and endosome correlation
  3. Set up three endpoints on API gateway
  4. Another PR to address the dependabot secrets access issue
  5. WIP on a clustering algorithm to convert pdbs to spheretrees
    • There is an issue converting units to be usable with cellPACK

TODO:

  1. Saurabh: connect with Nivi to go over talk outline for ASCB subgroup
  2. Saurabh: get average PILRs for mitochondria, microtubules, and Golgi for overlap analysis with peroxisomes and endosomes - DONE
  3. Saurabh: calculate individual PILR correlations - DONE
  4. Saurabh: Schedule cellPACK discussion at SciSim WG with Clare - DONE
  5. Alli: issue #289
  6. Alli and Ruge: issue #288
  7. Megan: continue clustering to convert pdbs to spheretrees

September 17th, 2024

Updates and agenda:

  1. cell morph WGF done
  2. cellPACK selected for talk at ASCB!
  3. Ruge and Alli demo'd the server!
  4. Correction for structure positions outside membrane
  5. currently running calculations to get occupancy for all shapes

TODO:

  1. Working on getting the API gateway up and running.
    1. coordinating with the software team to get things into Terraform
  2. Ruge and Alli: Update PRs and issues in the cellPACK repo
  3. Look into dependabot PR

(Before next WG meeting with Megan)

  1. Outline talk for ASCB - brainstorming at SciSim WG
  2. Look up who spoke at ASCB subgroup last year and get tips re: structuring the talk
  3. Saurabh - Occupancy plots with non-normalized distance, scaled distance from nucleus - DONE
  4. Saurabh - Radial occupancy for ER
    1. At least based on PILR - DONE

August 20th, 2024

Updates and agenda:

  1. Peroxisome packings done, waiting for endosomes.
  2. Planning meeting with Graham, Megan, Saurabh to discuss next steps for cellPACK analysis.
  3. We're now able to run any local recipe on AWS Batch.
  4. PR for cellPACK dockerfile in review.

TODO:

  1. Saurabh - Reach out to Graham and work on updating schematics for distance measures
  2. Make and share slides for cell morph WGF
    • Add Chris' suggestions for PCNA organization
    • Add discussion points from planning meeting for ER/excluded volume analysis
    • Add plots with different normalizations
    • (stretch) Tune nuclear bias to match observed data
  3. Schedule small group meeting with Chris and other scientists to discuss new analyses and experiments if needed after the WGF meeting.
  4. Ruge and Alli - finalize plan to submit batch jobs through http requests. Consider cost considerations for lambda vs other tools.
  5. Megan - Pair programming session for spheretree generation
  6. Megan - Review PR for fixing dependabot AWS permissions

August 13th, 2024

Updates and agenda:

  1. PR submitted for dependabot access to AWS
  2. Presented at SciSim WG - updates related to new distances measures and analyses

TODO:

  1. Alli & Ruge - check with DevOps re: permissions
  2. Megan - bring up in RASA meeting as needed
    1. Alli - write up a list of requirements for RASA
  3. Set up demo of cellPACK server by end-of-month if possible
  4. Megan - look into k means clustering for automated spheretree generator
  5. Saurabh - Try excluded volume packing to reproduce occupancy curve for peroxisomes
  6. Saurabh - Try tuning nucleus bias to match peak in observed distribution
  7. Saurabh - update occupancy plot with units/try with non-scaled distances

August 6th, 2024

Updates and agenda:

  1. ASCB abstract submitted
  2. Markdown writer PR merged
  3. Ruge and Alli have set up a milestone for the cellPACK server
  4. PR for accessing AWS in docker CLI - can pack recipes and save result files to s3 and open results in simularium
  5. cellpack analysis - packings for n=1997 peroxisomes and n=~2500 endosome data started (will take ~5-7 days)
  6. New project for cellpack analysis for better tracking of tasks

TODO:

  1. Review open PR for accessing AWS in docker CLI
  2. Review packings and run punctate structure analysis on the outputs for the full variance data
  3. Dependabot issue for updated actions

July 23th, 2024

Updates and agenda:

  1. Lysosomes are not punctate - not worth focusing on in initial analysis
  2. Multiple gradient PRs merged
  3. Markdown Class PR!
  4. Firebase supports gradient arrays!
  5. cellPACK server work: got access to AWS Batch (can run jobs) and ECR (can push images)
  6. Dockerized version of cellPACK is running
  7. Setup hydra for cellpack analysis pipeline
  8. Github workflows and linting for cellpack analysis
  9. ASCB abstract drafted
  10. New packings with peroxisomes and endosomes with count obtained from observed data

TODO:

  1. Saurabh: Review gradient arrays in Firebase PR
  2. Ruge and Alli: Working on getting recipes to pack using image on ECR
  3. Saurabh: Run punctate analysis on new packings
  4. Ruge: Troubleshoot MarkdownWriter class github workflow

June 25th, 2024

Updates and agenda:

  1. PR open for grid_file_path upload
  2. endosome analysis is done
  3. pair programming on mixed/combined gradients

TODO:

  1. review PR
  2. in progress: uploading refactor
  3. discuss lysosomes (in SciSim meeting)

June 11th, 2024

Updates and agenda

  1. PR for loading grid files from s3: need get it to work with firebase upload
  2. SAC is over
  3. Discussed grid file paths

TODO:

  1. upload grid file path to firebase on upload
  2. need a second approval on open PR
  3. revisit upload algorithm (debug)
  4. refactor cellpack analysis repo, and do packing with more variation: a more faithful version of the distribution

May 28th, 2024

Updates and agenda

  1. the slides for SAC are done
  2. plots for cellpack results in simularium file is working
  3. two prs for cleanup
  4. https links work for loading meshes from s3
  5. merged github action cache so the actions run faster

TODO (next meeting):

  1. discuss feasibility of uploading a whole bunch of recipes
  2. discuss grid file paths in firebase?

May 14th, 2024

Updates and agenda

  1. SAC writeup and slides
  2. Loading meshes from s3 does not work - will need to update MeshStore.get_mesh_filepath_and_extension(filename)
  3. github action cache - discuss upgrading to pyproject.toml and python 3.10
  4. discuss cleaning up unused recipes and config options from repo
  5. if time: discuss questions from the previous server chat

Notes

  1. Use https:// link instead of s3 URIs for loading meshes from s3

Action Items:

  1. Saurabh: loading grid files from s3
  2. Ruge: add plots to result files
  3. Saurabh: check priority packing (removed functions included some priority calculations) and other functions moved to utils.py. Get rid of unused code completely and move older analyses to the cellpack-analysis repository.
  4. Saurabh: test pdm build backend to manage dependencies, will need to update test workflows as well
  5. Megan: do a first pass of the cellPACK 1 page summary
  6. Saurabh: Incorporate edits from Megan and Graham into cellPACK 1 page summary

April 30th, 2024

Updates

  1. Presented at cell morph WGF meeting on 25th April
  2. Saurabh was able to upload and run a peroxisome recipe through firebase
  3. Lunch meeting presentation prep

TODO:

  1. Saurabh: test loading meshes from s3
  2. Saurabh: loading grid files from s3
  3. Saurabh: look at Ruge's PR

April 16th, 2024

Updates

  1. Completed: Normalize obtained nuclear distance distribution with the available space to see if we recover a flat line
  2. Progress in analysis pipeline
    • Question around using firebase:
      • first need to upload local recipes using the upload script
      • also possible to run a json from AWS? worth pursuing?
  3. Ruge has open PR for skipping db system in the case that a user is only doing local packing
  4. last presentation April 9: available space investigations
  5. Proposal for cellPACK server (talk about more today?)

TODO:

  1. for normalized distribution: 300 cells data pooled together. Try doing it per cell and then plot ratio
  2. ideas: two theories for the observed plot ratio: is it just excluded space? Ie the ER is keeping the peroxisomes from being close to the nucleus? or do they actually have a preference for the edge of ER

April 2nd, 2024

Updates

  1. Discussion on grid point selection for biased packings

TODO:

  1. compare distance distributions across elongation shape mode bins

  2. Normalize obtained nuclear distance distribution with the available space to see if we recover a flat line

    a. Can also be used as a test for deviation from randomness

  3. Use z position as a metric to look at endosome distribution

March 19th, 2024

Updates

  1. Set up meeting with Alli re: cellPACK server on 20th March
  2. Merged PR 224, 237
  3. Firebase packing path updates
  4. Published a new version on PyPI
  5. removed cvapipe analysis dependency from the cellpack-analysis repo
  6. Presented updates at the scientific simulations WG - punctate analysis on observed peroxisome data

TODO:

  1. Saurabh: review PR #230
  2. Saurabh/Karthik: Install cellPACK from PyPI in the cellpack-analysis repository
  3. Lunch meeting updates on April 17th - Follow up and outline on 2nd March
  4. Ruge: issue #238

March 5th, 2024

Updates

  1. Presented at BPS
  2. Started cellpack infrastructure meetings (pipeline optimization/tooling)
  3. generated synthetic data for representation learning group

TODO:

  1. talk to Alli (working with Ruge to make a cellpack server)

Feb 2nd, 2024

Updates

  1. removed panda3d (in review)
  2. Met with Jessica to discuss pipeline optimization/tooling, going to have an ongoing meeting to coordinate efforts

TODO:

  1. Path validation PR to open
  2. PR comment
  3. see if we can take a cellMorph meeting after BPS to have Julie see new data

Jan 26th, 2024

Updates

  1. fixed issue with surface packing
  2. Reviewed open gradient image PR (still needs second review, and requested one small change)
  3. Started reading staging firebase setup PR, but need to finish review

TODO:

  1. remove panda3d
  2. start on some issues towards making cellpack usable by other people

Jan 19th, 2024

Update

  1. we work on #213 but didn't finish, some meshes seem to not be loading correctly
  2. Worked on exploring the baseline/null hypothesis of packing many replicates and running the analysis on the set to see what we can define as "same" in our system.
  3. In review: staging firebase setup

TODO:

  1. some known speed ups (#219)
  2. PR review (saving gradient data as image)
  3. #212: firebase validation and deleting metadata