Sprint Ending 2025-05-01, 2:05pm PDT

Hot Map

Demos

User demos 2:10 - 2:40pm PDT

Intended for gaining feedback and inputs from all stakeholders, including end users.

• Data Upload Manager, upload status tracking (Scott)
• API improvements (Alex, Al, Thomas)
• Web portal updates (Anil, Eddie, Sean, Sagar)
• Install peppi in conda (Scott)

Technical demos 2:40 - 3:10pm PDT

• tagging strategy for planetary data cloud (Sagar)
• ds-view security fixes (Jordan, depending on availability)
• SLOC statistics and development with Cursor (Jordan, , depending on availability, any other experience on coding with AI)
• Proposed PDS Tracking and Monitoring System initial solution (Ramesh, if time permits)

Sprint Ending 2025-04-10, 2:05pm PDT

Hot Map

Demos

User demos 2:10 - 2:40pm PDT

Intended for gaining feedback and inputs from all stakeholders, including end users.

• fix on registry-manager (Al)
• validate updates (Al)
• deep-archive fixes (Sean)
• note on management of bundles or collections with duplicate lids, but different vid (Thomas)
• new test in registry (Alex)
• registry-sweepers in production (Sagar)
• Nucleus compressed file handling and other updates (Ramesh)
• drupal web site (Web mod), updates (Anil, Eddie)
• testrail plans for TRR (Thomas, Vivian, on behalf of Catherine)

Technical demos 2:40 - 3:10pm PDT

Deep dive into technical demos/discussions around internal system components.

• Roundup CICD supports any version of Sphinx (Sean)
• terraform vs opentofu (Thomas)
• port 443 vs 80 on AWS components (Thomas)

Sprint Ending 2025-03-20, 2:05pm PDT

Hot Map

Demos

User demos 2:10 - 2:40pm PDT

Intended for gaining feedback and inputs from all stakeholders, including end users.

• Migrate 1.6 PB of data in 2 days (Vivian, Jordan)
• Validate-ref works with new Registry (Al, Jordan)
• New feedback widget for IMG node (Catherine)
• Nucleus backlog process (Ramesh)

Technical demos 2:40 - 3:10pm PDT

Deep dive into technical demos/discussions around internal system components.

• S3 batch operations (Ramesh)
• Sweepers in production (Alex, Sagar)

Sprint Ending 2025-02-27, 2:05pm PDT

Hot Map

Demos

User demos 2:10 - 2:40pm PDT

Intended for gaining feedback and inputs from all stakeholders, including end users.

• Web portal, user facing updates (Anil, Eddie)
• peppi, online documentation updates (Thomas)
• Search-API, PDS4+xml and json re-implemented (Thomas)
• Search-API, Facet interface design (Thomas, Alex)
• Data Upload Manager improvements (Scott)

Technical demos 2:40 - 3:10pm PDT

Deep dive into technical demos/discussions around internal system components.

• summer internship project (Thomas)
• How to re-use a module in your terraform script, ECS example (Sagar)
• Nucleus, terraform script break up (Ramesh)
• Data Volume Estimates (Vivian)
• Web portal, drupal integration strategy (Anil, Sean, Eddie)

Sprint Ending 2025-02-06, 2:05pm PDT

Hot Map

Demos

User Demos

Intended for gaining feedback and inputs from all stakeholders, including end users.

Web, 2:10pm PDT

• Web modernization status (Anil, Sean, Eddie)

Node Tools and Services 2:20pm PDT

• Nucleus with Cognito ready for production (Ramesh)

Technical Demos

Deep dive into technical demos/discussions around internal system components.

Registry 2:25pm PDT

• Registry index structure (Alex, Sagar, Thomas)
• Web cache poisoning in the registry-api (Thomas)

Infrastructure (Systems and CICD)

• no updates this sprint

Sprint Ending 2025-01-16

TBD - Actions from last retro:

• try to set a specific agenda grouping by component with time slots identified
• Put user services / front end updates at the front of the review to engage users, community, and non-technical people
• Move technical/developer focused talks towards the end and clearly delineate these tasks

Sprint Ending 2025-01-16

Hot Map

Demos

• Harvest log summary fixes (Al)
• New terraform modules (Sagar, Vivian)
• Nucleus cognito authentication (Ramesh)
• Registry: Read only access to all indexes, write in only one (Sagar)
• Registry scrubbed doc (Thomas)
• Registry-sweeper, fix LDDs (Alex)
• Peppi, target from keyword, mission specific implementation / cross mission interface (Scott)
• Web portal Drupal design (Anil, Sean)
• Web Portal updates (Eddie, Anil)
• Current Website Architecture (Jordan)
• Summer internship proposals (Thomas)
• Copilot demo

Sprint Ending 2024-12-26

Review anticipated on 2024-12-19, also known as the Ugly Sweater Sprint Review

Hot Map:

Demos

• new registry tools versions (Thomas)
• OL8 tests (Catherine)
• Peppi, upgraded notebooks for AGU workshops (Scott, Thomas)
• Web Mod Portal (Anil, Eddie)
• Memory leak solution for legacy registry (Jordan)
• Cloud cost management (Jordan)
• SonarCloud demo (Jordan)

Sprint Ending 2024-12-05

Hot map:

Demos

• one cloud, shared by all nodes, for the PDS why ? (Jordan)
• Information model updates (Steve, Ron, Jordan)
• Peppi and its sparkling notebooks (Scott, Thomas)
• Web portal facets filters (Eddie)
• Web portal, Drupal integration, devops analysis (Sean)

Sprint Ending 2024-11-14

Hot map:

Demos

• Web portal updates (Anil, Eddie, Sean):
  • improvments to MVP
  • drupal
• peppi (Scott)
• Rings ontology (Sara, Jordan)
• Information model upgrades for build 15.0 (Steve, Jordan)
• New registry tools release and node on-boarding status (Al, Alex, Sagar, Thomas)
• Harvest in nucleus (Ramesh)
• SBN Catch foreseen deployments (Vivian)

Sprint Ending 2024-10-24

Demos

• Localstack for DUM (Scott)
• Registry, new release after beta test phase (Thomas, Al)
• TRR (Vivian)
• Testrail, more automation (Thomas)
• Data curation, migration for the Web Mod portal (Jordan, Sean, Thomas)
• Web Mod Portal, MVP +14 days (Anil, Eddie)

Sprint Ending 2024-10-03

Demo

• Harvest/registry improvements during beta-tests (on-going) (Al, Thomas, Sagar):
  • harvest discipline node name https://github.com/NASA-PDS/harvest/issues/186
  • number of fields updated in schema https://github.com/NASA-PDS/harvest/issues/190
  • documentation, for registry and registry-client
• trailing / in cloudfront services (Thomas) https://github.com/NASA-PDS/pds-mcp-infra/pull/14
• registry-api, page size and performances (Thomas) https://github.com/NASA-PDS/registry-api/issues/552
• Planetary Data Cloud tenant roles and Dev/Test/Prod policies (Vivian)
• Testrail, more automation (Thomas)
• Data Upload Manager on LocalStack (Scott)
• Access Nucleus Airflow UI with Cognito credentials and web token (Ramesh)
• Solr Registry Search (Jordan)
• Web Mod Portal, MVP -7 days (Anil, Eddie)

Sprint Ending 2024-09-12

Demo

• Web modernization:

  • Search (Eddie)
  • Home page (Anil)
  • Deployment on MCP Dev ? (Sean, Eddie)

• deep-archive bug fixes (Sean)

• Cognito service for real users (Ramesh)

• registry-sweeper for multi-tenant registry (Alex, Sagar)

• API doc, clients updates (Thomas)

• Seismic use case (Thomas)

• Discipline node registry beta test (Thomas)

Sprint Ending 2024-08-22

Demo

• Web modernization (Anil, Eddie, Sean):
  • Search data
  • build workflow for Continuous Deployment
• GIS service integration in PDS, Matt's internship, final presentation recorded https://teams.microsoft.com/l/meetup-join/19%3ameeting_Y2M5NDNiNTctNTQ3Ni00MDA5LWIxM2ItOTNhNzI2MWFkYmQ5%40thread.v2/0?context=%7b%22Tid%22%3a%22545921e0-10ef-4398-8713-9832ac563dad%22%2c%22Oid%22%3a%22273d84fc-30d4-42f7-93b2-eaefd00ae828%22%7d
• registry-api in production (Alex, Thomas)
• registry-mgr/harvest (Al):
  • renew credentials
  • set-archive-status/delete data for many products (from -packageId option)
  • default archive status is an harvest argument
  • works in production
• validate-refs with opensearch serverless (Al)
• DUM: improvements after feedback from SBN (Scott)
• nucleus store copy of data in S3 bucket as hot copy and cold archive. (Ramesh)

Sprint Ending 2024-08-01

Demos

• AWS Cost management per node (Ramesh)
• Cognito based authentication for Nucleus (Ramesh)
• --S3 metadata updated by Data Upload Manager (Scott)--
• cloud front deployment with terraform (Thomas, Vivian, Sagar)
• local stack, a solution for integration tests ? (Scott)
• registry manager can set archive status and delete data (Al)
• registry API, latest development and non-regression automated tests, credentials renewals (Alex, Thomas)
• New continuous monitoring (Sean)
• GIS web demo with new layers, QGIS export (Matt)
• automated tests for PDS4 Information Model and LDDTool (Miguel, Jordan)
• Updated context product for web modernization (Steve, Sophia)
• web modernization updates (Anil, Eddie)

Sprint Ending 2024-07-11

Demos

• Status on registry migration, from registry user workshop (Thomas)
• Issue with OpenSearch serverless (Al, Thomas)
• s3-browser on non-root URLs (Eddie)
• Editorial working group, what is it ? (Jordan, Mike, Ed - Jordan will do this presentation if the MC has ended in time)
• Web modernization updates (Anil, Eddie)
• GIS Treks Titan (Matt)

Sprint Ending 2024-06-20

Demos

• new test data for deep archive (Sean)
• DUM updates (Scott)
• Nucleus updates (Ramesh)
• Registry loader for multi-tenant registry, docker image (Thomas)
• GIS internship introduction (Matt)
• Authentication for Opensearch serverless and Java clients (Sagar)
• Command line tool to query the multitenant registry (Alex)
• terraform script for registry-api on MCP (Thomas)
• web modernization updates (Anil, Eddie)

Sprint Ending 2024-05-30

Demos

• DDR (Jordan)
• Web modernization (Eddie)
• Continuous Deployment in Dev venue (Sean)
• Continuous Integration for web development (Eddie based on Sean development)
• Validate-ref and other validate updates (Al)
• Nucleus multitenant (Ramesh)
• Multitenant registry migration (Thomas, Sagar, Al, Alex)
  • infrastructure (Sagar)
  • opensearch/data (Sagar)
  • harvest/registry-mgr (Al, Thomas)
  • registry-sweeper (Alex)
  • command line tool (Alex)
  • Search API (Thomas)

Sprint Ending 2024-05-09

Demos

• DUM upgrades (Scott)
• Nucleus automated deployment (Ramesh)
• Multitenant registry
  • Plan/schedule/communication with the nodes (Jordan, Thomas)
  • Data migration (Sagar)
  • harvest/registry-mgr (Al)
  • sweeper (Alex)
  • api (Thomas)
• NASA SMD Software workshop, API lightning talk and overall report (Thomas)
• PDS4 Information Model upgrades (Steve)
• Web analytics pipelines improvments (Kai)
• Web modernization implementation (Anil, Eddie)

Sprint Ending 2024-04-18

Demos

• Baseline IAM policies (Sagar)
• Data Upload Manager upgrades (Scott)
• Loader tools (harvest/registry-mgr) for multi-tenant registry (Al)
• Nucleus updates (Ramesh)
• S3 browser (Eddie)
• Web modernization implementation updates (Anil, Eddie)
• NASA A&A (Jordan)
• Keyword search on legacy registry (Jordan)

Sprint Ending 2024-03-28

Demos

• Continuous Integration for nodejs, on dummy repository (Sean)
• LDD bug fixes (Steve, Jordan)
• Nucleus updates (Ramesh)
• Multitenant updates (Sagar, Al)
• Registry-api, integration of new OpenSearchClient, first step (Thomas)
• Web modernization implementation updates (Anil, Eddie)

Sprint Ending 2024-03-07

Demos (5–10 minutes each)

• Upload CSS data on the cloud with DUM (Vivian, Ramesh)
• add deep-archive client to registry-api branch CICD test (Jordan)
• CodeQL static analysis (Jordan)
• Summer Internship on GIS (Thomas)
• Web modernization implementation status (Anil, Eddie)

Bug fixes (1 minute each)

• Registry-Sweepers improvements, memory, ... (Alex)
• Deep-archive bug fix and api dependency upgrade (Sean)
• RDD automated generation fix (Thomas, Jordan)
• 500 error on registry-api fix (Alex)
• MI-label bug fix (Jordan)

Status/Planning, mostly planning (20–30 minutes each, as needed)

• Nucleus (Ramesh, all) 20'
  • sprint objectives:
    • DUM load to S3 staging bucket and synchronize to ODR s3 bucket
    • (out of scope for now) at bundle level: pds-deep-archive, registry-mgr release.
    • DAG at product level:

For CSS:

1. validate
2. validate-ref (on-hold)
3. harvest
4. glacier
5. copy data to ODR
6. remove data from EFS
7. remove staging s3 bucket

• Multitenant Registry (Sagar, Al, all) 20'

Sprint Ending 2024-02-15

Demos

• Open Data Registry, what is it ? How is it useful for PDS/CSS (Vivian)

• Incident Response Plan (Jordan)

• Build 14.1 plan update submitted to Software Working Group (Jordan)

• Validate on Windows (Jordan)

• Multitenants Registry updates (Sagar)

• Documentation of archive status for registry (Thomas)

• Nucleus udpdates (Ramesh)

• property list in pds-api-client (Thomas)

• Web modernization updates (Anil, Eddy)

• What are your takes on the Software Symposium (all)

• Other

Sprint Ending 2024-01-25

Recording: TBD

Demos

• PDS Engineering Node web site migrated and 508 compliance (Catherine)

• Archive Browser with AWS SDK, progress so far (Eddie)

• New web design implementation, progress so far (Anil)

• Nodejs repo template and roundup integration (Sean)

• CORS issues on feedback widget (Thomas)

• Harvest updated job configuration for multi-tenant opensearch (Al, Jordan, Thomas)

• Nucleus documentation on gh-pages (Ramesh):

Planning

Web implementation sprint to be detailed

Sprint Ending 2023-12-14

Recording: https://jpl365prod-my.sharepoint.com/:v:/g/personal/jordan_h_padams_jpl_nasa_gov/EXzQ04eWiWRDm3BFugAaapUBYusRLpK8prID-tTgcr0OKQ?referrer=Teams.TEAMS-ELECTRON&referrerScenario=MeetingChicletGetLink.view.view

Demos

• Press It Up: Web modernization development initial setup with Wordpress (Anil)
• Buckets for Days: Latest greatest web directory page for S3 buckets (Eddie)
• Paging 1995: New pagination interface and implementation for the registry-api (Alex)
• Sweepers Creepers: Consistent properties as list vs string in registry with sweepers (Alex)
• Checking In: Review the latest updates to Validate (Jordan)
• State of Nucleus Address: Latest Nucleus Updates and "end-to-end" execution (Ramesh)
• Fresh Rolls: Best Practices in creating IAM Roles and Security Groups (Sagar)
• Registry Design-a-thon: Multi-tenant Registry Architecture Updates for supporting signed URLs (Sagar)

Sprint Ending 2023-11-23

Meeting held on 2023-11-30 due to holiday Recording: https://jpl365prod-my.sharepoint.com/:v:/g/personal/jordan_h_padams_jpl_nasa_gov/EYw3imfWNp9IlGgVIQqJk4MBv79cyNb5NvQJp39JQDi7Xg?referrer=Teams.TEAMS-ELECTRON&referrerScenario=MeetingChicletGetLink.view.view

Demos

• Web modernization design update (Mike/Edward)
• API wrapper PoC (Thomas)
• NAIF node missing in legacy dashboards (Thomas)
• Nucleus with fitz files (compressed), Nucleus cost issue and alternate design proposal (Ramesh)
• Re-try behavior on harvest (Alex)
• Build 14.0 I&T, TRR, DDR (Vivian, Miguel, Gary, Thomas)
• Don't leak your secrets (Sean)

Sprint Ending 2023-11-02

Recording: https://jpl365prod-my.sharepoint.com/:v:/g/personal/jordan_h_padams_jpl_nasa_gov/EYrNv4VEvKJHnIuSHzr6ulwBpFny9mCsMCKcHyJU49qBjw?referrer=Teams.TEAMS-ELECTRON&referrerScenario=MeetingChicletGetLink.view.view

Demos:

Legacy

• Updates on MCP (Sagar)
  • Encryption of EC2 instances
  • OIDC service
• Data Upload Manager status and documentation (Scott)
• Nucleus integration with Data Upload Manager (Ramesh), successful workflow with Messenger as planned for the CCS meeting
• First iteration of the provenance model in PDS4 (Steve)
• Release note with known bugs, for example https://github.com/NASA-PDS/registry-api/releases/tag/v1.3.1 (Thomas)
• Deep archive, installation instructions for Windows (Sean)

Sprint Ending 2023-10-12

Recording: TBD

Demos:

Data curation

• Cutover to using new Registry API for generating context products json (Al, Jordan): cancelled, not working yet because API is slow over a few pages of results.
• Avoid race condition in the product checker (Ramesh)

Search and web

• DPS presentations (Jordan: slides, Thomas: poster)
• Sinequa useability analysis (Joe)
• Navigation app bar in new web (Anil)

Other

• Create a stable release for the registry, what does that mean ? (Thomas)
• ISA status (Jordan)

Crowning piece

• Web Prototype (Mike)

Sprint Ending 2023-09-21

Demos:

• Web analytics (Kai)
• Web portal architecture with WordPress (Anil)
• Vulnerability fix from dependabot alerts on web development (Anil)
• Request the Sinequa API for search (Joe)
• Registry Automated Tests maintenance (Thomas)
• Registry-sweepers/RepairKit tuning for production (Alex, Sagar)
• DUM: Infrastructure as Code (Scott)

Sprint Ending 2023-08-31

Recording: TBD

Demos:

• Product Completion Checker lambda for Nucleus (Ramesh)
• New LDD (LRO, mgn) (Catherine, Steve)
• RepairKit optimization for production (Alex)
• Registry loading status dasboard in Kibana (Thomas)
• CloudWatch log monitoring for Registry-Sweepers & Registry-API using AWS Lambda / SNS (Sagar)
• Messenger data visualization with the PDS Web API and LeafMap (Joaquin)
• IPDA new web site published, the steps between after development to publication (Sean, video link (JPL only), PDF of slides)
• DUM: Parallelization of upload (Scott)

Sprint Ending 2023-08-10

Recording: TBD

Demos:

• The balkanization of pds-github-utils (Sean)
• Add TermMaps to default output JSON (Steve)
• Harvest tags the loaded products with its version number (Al, Thomas)
• Performance improvement on registry-sweepers (Alex)
• New alerts on registry-api and sweepers (Sagar)
• Nucleus, workflow triggers (Ramesh)
• Registry-api, JSON for everyone (Al, Thomas)
• Read images with the registry-api (Joaquin)
• Published results on the LLM tests for PDS Search (Amanda)

Sprint Ending 2023-07-20

Recording: https://jpl365prod-my.sharepoint.com/:v:/r/personal/thomas_g_loubrieu_jpl_nasa_gov/Documents/Recordings/PDS%20Sprint%20Review-20230720_141111-Meeting%20Recording.mp4?csf=1&web=1&e=KTgX2m&nav=eyJyZWZlcnJhbEluZm8iOnsicmVmZXJyYWxBcHAiOiJTdHJlYW1XZWJBcHAiLCJyZWZlcnJhbFZpZXciOiJTaGFyZURpYWxvZyIsInJlZmVycmFsQXBwUGxhdGZvcm0iOiJXZWIiLCJyZWZlcnJhbE1vZGUiOiJ2aWV3In19

Zenhub says:

Demos:

• Updated npm package architecture for web UIs (Eddie, Anil)
• Updates and plans for Magnetometers analysis from the registry-api (Joaquin)
• Semantic Search/LLM, similarities results (Amanda)
• Analysis for product-triggered workflow for nucleus (Ramesh)
• Standards for describing AirFlow DAGs, options (Ramesh)
• Alerts on almost full storage on AWS (Sagar)
• Registry Docker compose for multiple architectures (Sean); presentation available as video (JPL-only), or as PDF

Sprint Ending 2023-06-29

Recording: https://jpl365prod-my.sharepoint.com/:v:/r/personal/thomas_g_loubrieu_jpl_nasa_gov/Documents/Recordings/PDS%20Sprint%20Review-20230629_143149-Meeting%20Recording.mp4?csf=1&web=1&e=NkeQbS

• Continuous Deployment with terraform on NGAP (Sagar) 5′
• Registry-api postman tests reports in TestRail (Sean) 5′
• Updated Sitemap for PDS web site (Catherine, Sean) 5′
• Use signed-url/cognito to connect to opensearch (Alex) 5′
• Registry-sweeper, what it does, how it is deployed (Alex) 5′
• Registry-api OVIRS notebook, OVIRS data loaded and new pds.api-client (Thomas) 5′
• Registry-api notebook: plot Galileo magnetometer observations extrcted from the registry-api with pyWWT (Joaquin) 5′
• Data Upload Manager: monitor the client commands launched by the Discipline Nodes (Scott) 5′
• Validate validates video labels and files (Al, Jordan) 5′
• Registry loader procedures documented , see https://github.com/NASA-PDS/operations/issues/359 (Vivian) 5′
• Semantic Search: Compute embedding vectors on PDS4 labels and user's search term, status and plan for next sprint (Amanda) 5′
• Sinequa Search (Joe) 5′
• Other:
  • release build 14.0 notification
  • PDS Phone book
  • Registry architecture diagram https://lucid.app/lucidchart/d2d7eb2a-352b-4105-81d6-5694f3d50f8a/edit?viewport_loc=-238%2C-107%2C2440%2C1269%2C66uWzmvPQEeN&invitationId=inv_0ca74609-afa7-44b5-ae44-2a1c095f8f16

Open ticket/planning

• jenkins vs harness (vs github actions) ?
• Messenger data loaded by Nucleus/Ramesh (to be used by Joaquin)
• DOI-Editor, soon available in production, what are the next steps ?
• Schedule a point build and deployment for registry-api (after bug fixes)
• Search architecture, how to move forward, sinequa/opensearch/semantic search/graph database (https://github.com/NASA-PDS/portal-tasks/issues/72) ...
• Start archive viewer development

Sprint Ending 2023-06-08

Recording: https://jpl365prod-my.sharepoint.com/:v:/r/personal/jordan_h_padams_jpl_nasa_gov/Documents/Recordings/PDS%20Sprint%20Review-20230608_142204-Meeting%20Recording.mp4?csf=1&web=1&e=sbjoHr

Demos

List to be completed the morning of the Sprint review.

The demos are 5 minutes max but less is also ok, with short word on something which was completed, a screenshot, an explanation of what it is useful for:

• Latest development of the Search Prototype (Eddie, presented by Thomas) 5'

• LLMs for PDS, internship plan (Amanda) 5'

• Nucleus (Ramesh): 5' [] share data between steps [] logging organization and presentation to users/operators

• New test dataset for Registry integration tests (Thomas) 5'

• Registry-api notebooks reaching our user's community (Joaquin) 5'

• New health check end-point (Jimmie) 5'

• Miscellaneous, 1 screenshot each: 5'

  • B13.1 software release (Thomas)
  • JPL logo on PDS ENG wordpress (Catherine)
  • Organize a happy hour and hurt oneself so not to be sure to attend (Jordan)

Pushed to next sprint review because of training or sickness:

• Registry-sweeper, what it does, how it is deployed (Alex) 5'
• Continuous Deployment with terraform on NGAP (Sagar) 5'
• Registry-api postman tests reports in TestRail (Sean) 5'

Other significant blocked tasks

Detailed review of the remaining open ticket by Thomas and Jordan. Next sprint only blockers should be raised by the team. Thomas and Jordan will review the remaining tickets offline or in a different meeting.