8 30 2021 Tech Team Report - QualitativeDataRepository/TechnicalTeam GitHub Wiki

8-30-2021

Logged Tasks

                            Date             Task Hours (Main) Hours (EOLS) Hours (PII)
23-Aug-2021 Report, meeting, deploy 5.6 and D8 updates to prod with payara 5.2021.5, update settings, fix bug in dataset created email (dev/stage/prod), experiment with including Symphony ldap to query for a user. 6
24-Aug-2021 Change ldap reads to symphony, add modify, test, check pg13 on dev 4
25-Aug-2021 Finish ldap updates for list, modify, create, delete; test, investigate other module patches/errors, include D9 patches, make custom patches as needed, remove old missing module msg fixer, fix qdr_sso, require symphony/ldap 6
26-Aug-2021 Investigate D9 deploy - process, change php version, make patched local modules for those not ^9 ready, update other modules, restore missing module meg fixer, update hypothesis module, debug/turn on book module, run db updates, cr, refresh sitemaps, document. Investigate postgres config/transfer dvndb to pg 13, swap ports, test with Dataverse, turn off 9.6. 7
27-Aug-2021 Check dev, install graphmagick, test, coord re: pg13 1

Summary

Drupal

  • Worked through remaining Drupal 9 upgrade issues
    • Rewrite qdr modules to use base symphony ldap instead of contributed ldap v4.x modules: create, get, list, modify, delete calls, ldap password mgmt
    • Dropped content_sync and ldap modules
    • For other modules:
      • update to D9 compatible version
      • temporarily remove D8 version and re-add D9-only version after upgrade
      • apply existing patches via composer to get D9 compatibility
      • for those requiring a patch to get D9 compatible info.yml files - since composer checks compatibility before patching, apply patches on D8 and then copy source as a custom module for use in D9
      • for those requiring custom patches - also create D9 compatible custom modules
    • End result: 4 new custom modules, 1 dropped custom module, 1 return to using standard version, updates to 3 custom modules. Of the 4 new custom modules, ~3 may be ones where we can return to using the standard versions after patches are accepted.

Operations:

  • Deployed D8 sec. updates, Dataverse 5.6-qdr and payara5.2021.5 to prod
  • Switch to php 7.3 on dev and update to Drupal 9 via composer, run db updates, refresh permissions, sitemaps, copy final composer json/lock files to github
  • Investigated postgres 13 on dev - running but not in use. Transferred Dataverse db from 9.6 to 13, shut off 9.6, changed pg 13 to use default port, restarted postgres and Dataverse
  • Installed GraphMagick (proposed replacement for ImageMagick)

Dataverse:

  • Fixed issue with dataset notification emails where the creator of the dataset was not getting identified correctly
  • updated jsoup library (security issue w.r.t. html cleaning) and the jai image library (thumbnails)

Discussion

  • mysql upgrade issues - ability to dump/backup database is broken after 5.7 update. Noticed with Drupal installs (rollback sql files are almost empty) and investigated more today on dev/stage. Looks like there's an upgrade util that can fix this - not sure if there was something else that could/should be done differently when upgrading to avoid that (still need to upgrade prod prior to D9).
  • FYI: I've been summoned for Grand Jury duty starting Sept 9th. It isn't clear yet what the commitment might be - I may not be chosen. If I am, the term is up to 4 months, but it sounds like it would be as needed, with a limited number of days any given week (~2?) and they are apparently limiting hours per day due to mask requirements. I probably won't know more until Sept. 9th.

Plans

  • Deploy Drupal 9 and Dataverse 5.6-qdr2 with postgres 13 once testing is complete
  • Dataverse
    • still want to investigate the guestbook responses re version info not being included.
  • Anno-Rep work
    • Help with deployment to dev
  • TBD: FRDR Security