08.01.2022 TT Agenda - QualitativeDataRepository/TechnicalTeam GitHub Wiki

Admin

  • First of month - please submit hours to CP at Syracuse

Jim + Seba

  • Outage last week

Seba:

  • Did you get connected to SYR IT about remote access?
  • If so can you verify the new Prod is synching backups at SYR?

Jim:

  • Discuss 5.11 and steps for QDR curators
  • Straigtening out download count and guestbook

Notes:

  • When using dvuploader (og version) and we had full text index running creates an n^2 problem. This took down storj. We didn't run out of memory, or sockets. New DVUploader is every n file to trigger index (right now we have it set at 10)... DV is not a very smart way to qeue jobs for indexing - which is a known problem but wasn't a major priority for
  • Seba observation - perhaps storj has throttle for API posts (for security)
  • Next steps - JM to reproduce on Dev for Storj - this would help us identify if for example its the size of storage pool or if we are being throttled when indexing (which is our problem)

Seba

  • Paging system and escalation works (!)
  • Still needs to reach out to SYR to verify prod sync
  • To do: Needs to figure out how to retain higher verbosity of logging (and store it on our servers) without log aggregation going overboard - idea here is that when we have a production downturn we can have a number of people looking at the logs. EG when we get paged we turn up verbosity of logging
  • Sentry.dev.datavers - Seba wonders if we should revisit monitoring and dashboard ui so that we have stack traves, etc accssible to dev team
    • Jim DV logging is very very messy. So it might be an investment of time to figure out how to get some utility out of it for DV prod (Jim can handle the java configuration)

Jim

  • Wants sanity check on "autovacuum threshold on guestbook response table to hopefully limit inaccuracy in download count"
    • No issue from Seba - JM will push the fix back to the DV community
  • QDAS previewer now using javascript library - project file and zip file (two separate) previewers now in operation
    • We need a draft project with graph table or source table
    • SK and NW need to do some QA and give feedback
    • Open DP - borrow logic, where QDAS zip would be able to save the AUX file so you could NOT destroy the original ...