11 6 2023 Tech Team Report - QualitativeDataRepository/TechnicalTeam GitHub Wiki

11-6-2023

Logged Tasks

                            Date             Task Hours (Main) Hours (EOLS) Hours (PII) Hours (QDAS)
30-Oct-2023 Reporting, investigate/fix slow guestbook creation, turn off reindexing for this, fix synchronous indexing for other collection update cases., update webforms, update webforms, cherry-pick rel=nofollow for facet links 5
31-Oct-2023 HEAL OIDC mtg & prep, investigate/fix privateURL issues w.r.t. files, OIDC, coord re: Drupal bug, update webforms, investigate csv update & conflicts 4
1-Nov-2023 Investigate apache2 config change, fix once/jquery issue, investigate/fix jquery autocomplete issue w.r.t. module_filter in D10, update D10 to 10.1.6, start adding signedUrl support for privateUrlUsers and update SignedUrl part of api AuthFilter, fix login_history reports - module was referencing an old module in the view definitions 6
2-Nov-2023 Finish privateUrl/signedUrl support, fix previewer error display with signedUrl changes, deploy to dev/stage, update previewers config on stage to use privateUrls, update v5.14 branch from develop, deploy recent changes to stage, update apache to allow local previewers 5
3-Nov-2023 Investigate prod outage, add workaround, fix issue, create issue/PR #10095/10096, refactor to move auth calls to authSvc for PrivateUrl work, create issue/PR #10093/#10098, test changes on dev, update stage/5.14 branch 5

Operations

  • Investigate prod outage, find issue, deploy work-around, update Dataverse with bug fix (see Dataverse section)
  • Deploy locally hosted signedUrl-based Previewers on stage, update apache2 with ProxyPass for them

Drupal

  • Update webforms
  • Update D10 to 10.1.6
  • Fix two remaining issues w.r.t. D10 - coding change in QDR theme to avoid using deprecated jquery-once library, turn on jquery-autocomplete to avoid module_filter warning
  • Fixed login_history report on all machines - the views defined for this module referenced the old qdr_sso module instead of the new qdr_oidc_sso module, which was missed during the upgrade

Dataverse

  • Investigate/fix slow guestbook creation issue - Dataverse was reindexing all datasets in the root collection after any change to the collection. Coordinating with Leonid/IQSS, this has been turned off for any operations where it wasn't needed. Further, despite the code marking the indexing as asynchronous in some places, the reindexing was happening before the guestbook creation call returned. That issue has also be fixed for any time the collection still causes reindexing.
  • Fix download estimate query bug, create issue/PR ##10095/10096
  • Restore PrivateUrl functionality with OIDC SSO
  • Enable PrivateUrlUsers to have signedUrls for previewers, create issue/PR #10093/10098
  • Test, deploy to dev, sync changes between develop (dev) and v5.14 (stage) branches, deploy to stage
  • Cherry-pick change to add rel=nofollow attribute on facet links to stop (nice) robots from retrieving subsets of datasets this way.

HEAL

  • Attended meeting to discuss how to allow Gen3 to leverage our SSO, plan to coordinate over next ~2 weeks

AnnoRep

Discussion

  • The changes to apache2 to increase the number of servers/threads that is needed for the Keycloak admin interface to work well (and for Drupal when not merging all the css/js) appears to auto-revert. The values in /etc/apache2/mods-available/mpm_prefork.conf need to change to 10/20/50/300/20/10000
  • The mysql 'max_allowed_packet' setting appears to have be 4M on dev/stage/prod. That was causing a problem in generating a ~350KB login history csv file on stage/prod. I adjusted stage to 16MB and prod to 64MB which appears to be the default in new installs according to online docs. I made changes in /etc/mysql/conf.d/mysql.cnf - not sure where it was set before as there was no entry on dev/stage. A few questions:
    • Is this a good place to change it? Is there somewhere else to remove a 4M value?
    • Will it stay changed or get reset?
    • On prod, there was a [mysql]max_allowed_packet setting (100M) but no [mysqld]max_allowed_packet setting. I was guessing the setting was a mistake (only applies to the client) but perhaps we need it for some reason (backups?). Should I restore it?

Plans

  • Deploy updates/bug fixes to prod
  • Deploy Drupal 10 upgrade to stage/prod
  • Deploy rich HTML previewer/new signedUrl previewers to prod
  • Investigate options for HEAL to participate in SSO
  • Fix Stata-14 ingest by allowing file inspection during direct upload or adjusting the Stata ingester.
  • Fix #113 if possible
  • Matomo - investigate event-level tracking via tag manager, remove non-working google scripts
  • AnnoRep - explore round-trip, configure auto-start and log rotation
  • Ops
    • check missing globalidcreationdates and fix via /modifyRegistration or alternative
  • Dataverse
    • Make PR for guestbook adding datasetversion fix
    • Popup info accessibility - IQSS likes the recommendations from the source I linked to, so this can be implemented along those lines.
  • QDAS Previewer
    • Updates per request
    • Investigate writing aux file/previewing lower-sensitivity version and/or other write options
  • TBD: FRDR Security