8 1 2022 Tech Team Report - QualitativeDataRepository/TechnicalTeam GitHub Wiki

8-1-2022

Logged Tasks

                            Date             Task Hours (Main) Hours (EOLS) Hours (PII) Hours (QDAS)
25-Jul-2022 Report, investigate software field difference from IQSS 1
27-Jul-2022 Investigate prod failure, storj issue w.r.t add file/reindex/full index. 3
28-Jul-2022 Coord re: storJ issue 1
29-Jul-2022 Report 5.11 data deletion bugs to team, add fixes for data deletion bugs 1

Operations

  • Investigated production failure - related to lots of files uploaded to a storj store.
    • Following up with storj w.r.t. whether this was in part from overwhelming their service or just related to overflowing our configured connection pool.
    • Recommended several changes (increase connection pool for storj store, turn off full-text indexing while doing uploads of many files, use newest DVUploader which triggers fewer dataset updates/reindex cycles than prior versions). Further changes to indexing could help (i.e. not queuing more reindexing of the same dataset if prior reindexes aren't done).

Drupal

Dataverse

  • Investigated #8857 w.r.t. whether QDR has any fields marked as controlled vocab that are actually plain text (none found)
  • Reported two bugs in v5.11 to team - they can both result in data loss.
    • Related to deleting files and to a superuser deleting a draft dataset when a draft version doesn't exist (part of the bug allows that "delete Draft Version' button to appear).
    • cherry-picked fixes from GDCC/IQSS effort to analyze/fix them to the develop branch and today, the v5.11-qdr branch

Discussion

  • Would like to try lower autovacuum threshold on guestbookresponse table to hopefully limit inaccuracy in download count. This is reversible (i.e. you can remove it) and autovacuum/analyze should not be doing anything to the entries (it may reshuffle things on disk and updates query planning stats). This is something where we could/should report results back to the community.
  • QDAS Previewer - next priorities? W.r.t. edit - would writing a lower-sensitivity version of the project file as an auxiliary object make sense (as the Open Differential Privacy project does?)

Plans

  • AnnoRep - continue to explore/fix docx/pdf github issues
    • Deploy updates to dev/stage/prod
  • Dataverse
    • Deploy 5.11.1 fixes ASAP
    • Popup info accessibility - IQSS likes the recommendations from the source I linked to, so this can be implemented along those lines.
    • Still want to investigate the guestbook responses re version info not being included.
  • QDAS Previewer
    • Updates per request/Investigate writing aux file/previewing lower-sensitivity version
  • TBD: FRDR Security
  • Other tasks as discussed in strategic planning