12 4 2023 Tech Team Report - QualitativeDataRepository/TechnicalTeam GitHub Wiki

12-4-2023

Logged Tasks

                            Date             Task Hours (Main) Hours (EOLS) Hours (PII) Hours (QDAS)
27-Nov-2023 Reporting, meeting, fix payara service on dev, test dv, solr, annorep with j17, update stage domain.xml, try j17, deploy Dataverse v6.0-qdr-dev on dev, fix flyway merge issues, reindex, start investigating index failure for one dataset, start investigating test fails. 5
28-Nov-2023 Add J17 specific jvm options on prod, restart, coord re J17 install, verify prod using J17 fo DV, find/fix indexing error (missing schema.xml field), investigate test errors - some Unix v. Windows, fix OAuth ones, update thumbnail PR per reviewer comments. 4
29-Nov-2023 Checkin test/other fixes, update to Keycloak 23 on dev/stage, adapt qdr theme, investigate show/hide password changes, update stage to P6/DV~6.1, create v6.1-qdr branch, update solr to 9.4.0, document extra steps from 6.x release notes, fix null pointer in indexing when stores are not configured, fix store config, reindex, update matomo module on dev/stage. 6
30-Nov-2023 Merge solr optimizations from latest merge, use Http2SolrClient, fix tests, deploy to dev/stage 3

Operations

  • Added required jvm options to Dataverse to allow Java 17 to be used. Verified Java 17 was working on prod as the default after Seba made the change
  • Updated Keycloak to v23 on dev/stage - adjusted style to show new hide/show password box.

Drupal

  • Updated matomo module on dev/stage

Dataverse

  • Finish ~6.1 deployment on dev and deploy new payara, solr, Dataverse on stage
    • fix flyway issues
    • investigate/fix some failing tests - Oauth tests needed updates for our SSO, other tests were failing locally for me due to /r/n vs /n issues (tests assume \n as the line ending) - I fixed some and left others (they all work on Jenkins)
    • investigated/fixed failing indexing of 1 dataset - an old dataset using a test metadata field, indexing fails if any field is not in the schema, on stage some files run out of memory when full-text indexing, which is ~normal there. I completed reindexing on stage with full-text off to avoid these.
    • created v6.1-qdr branch for stage
    • fixed issue with indexing failing when stores are misconfigured, and fixed store config on stage
    • merged solr optimizations going into 6.1. Also changed to the new Http2SolrClient class (HttpSolrClient is deprecated as of solr 9.x) - reported the issue
  • Updated the thumbnail optimization PR per reviewer comments

HEAL

AnnoRep

  • tested server launching with Java 17 as the default (works)

Discussion

  • Is the hide/show password box useful? (If your browser prefills your password, this allows that prefilled password to be seen. While concerning, this is true on other sites and the password is visible if you edit the DOM anyway.)

Plans

  • Await final 6.1 release, merge and deploy for testing
  • Work on authentication issue #43(non-email account id, MFA, etc.)
  • Work on metadata issue #44 (more metadata to DataCite, etc.)
  • Fix Stata-14 ingest by allowing file inspection during direct upload or adjusting the Stata ingester.
  • Fix #113 if possible
  • Matomo - investigate event-level tracking via tag manager, remove non-working google scripts
  • AnnoRep - explore round-trip, configure auto-start and log rotation
  • Ops
    • check missing globalidcreationdates and fix via /modifyRegistration or alternative
  • Dataverse
    • Make PR for guestbook adding datasetversion fix
    • Popup info accessibility - IQSS likes the recommendations from the source I linked to, so this can be implemented along those lines.
  • QDAS Previewer
    • Updates per request
    • Investigate writing aux file/previewing lower-sensitivity version and/or other write options
  • TBD: FRDR Security