08 31 2020 Tech Team Report - QualitativeDataRepository/TechnicalTeam GitHub Wiki
8-31-2020
Logged Tasks
| Date | Task | Hours (Main) | Hours (EOLS) | Hours (PII) |
|---|---|---|---|---|
| 24-Aug-2020 | Report, meeting, fix metrics bugs, complete graph displays, document Drupal metrics | 7 | ||
| 25-Aug-2020 | Metrics graph styling, order unique downloads by count, add max to # displayed in graph, refactor/cleanup js | 6 | ||
| 26-Aug-2020 | Refactor/cleanup js, test | 4 | ||
| 27-Aug-2020 | Remove filenames from scripts, add db update script, update 4.20-qdr8 version, deploy scripts to stage, do initial merge pull for 5.0 | 3 | ||
| 28-Aug-2020 | Merging v5.0 | 6 | ||
| 29-Aug-2020 | Finish first pass of merge, start debugging | 3 |
Summary
Metrics:
- finished work on the graphics/downloads - refactored/cleaned up javascript to remove old methods, make time series all use the same code, added a max # of datasets param to use for unique downloads display, added colors
- Minor updates to API to order unique downloads by count, set filenames for csv downloads, and added the db update script needed (adds a column to cache metrics per sub-dataverse)
- (with GDCC support - added some documentation and refactored to put local and global (any/all dataverse metrics) on separate pages)
- Merged API updates with the v4.20 branch to create v4.20-qdr8 that is ready to test/deploy to stage/prod
- Deployed the final dataverse-metrics to stage - ready to use once Dataverse is updated.
v5.0 Updates
- Did an initial merge with the v5.0 code. Found 23 files with conflicts.
- Resolved conflicts in the Java code and, for .xhtml conflicts, essentially reapplied QDR changes to the v5.0 code rather than merging due to the scope of changes
- Updated my build environment to handle the changes in the Maven manifest (pom.xml) (v5.0 removes several jars and relies on payara to supply them, and payara isn't available for a local build, so those have to be added manually in Eclipse)
- With a working environment, was able to find/fix remaining merge issues that broke compilation.
Plans
- Deploy v4.20-qdr8 and 5.0 to dev
- Work through v5.0 issues (see discussion)
- Start work to update the /replace API
and possibly:
- file DOI reservations
- Drupal 9
For Discussion
- Seba: Payara deployment is the next step for testing in dev. I can run through the upgrade instructions from IQSS on dev.
- The metrics API should be ready to test. If we deploy v4.20-qdr8 to stage, we should have most of prod's content to test against. Some of the queries involved are complex, so some ~manual sanity testing against prod content and/or test content would be useful. (My guess is that things are working except for potential issues with old DB entries - on dev I found some dataset table entries that had no matching entry in the dvobject table (basically a corrupt entry) that could either add a count or break a query. It's possible that stage/prod also need some db cleanup, or adjustments to queries to handle older entries.)
- v4.20-qdr8 - the only significant change is the new metrics API. I found two minor changes that also hadn't been merged from our develop branch - an update that fixes file names with a space being downloaded with a '+' instead (for S3 direct download) and a change to sort all date facets. I think this means that v4.20-qdr8 should only need basic testing aside from metrics testing for it to get deployed out to prod.
- deploying v4.20-qdr8 to prod could be done on it's own, running a metrics display app on stage for it, or with the metrics app also publicly available on prod. We could test with the displays private and then move them, or keep them private, or start public, etc.
- v5.0 - The main issue with the merge is that there are so many display changes that it is not clear whether our customizations, including Schema's layout/styling changes are still relevant or if they are still working when they are. I've tried to keep changes that I think are still relevant, but I think that, once I have it running on dev, we'll essentially need to go through and compare v4.20 styling and that on v5.0 to see if QDR customizations are working. It may also be worthwhile to look at demo.dataverse.org to see if the new styling is OK/better than keeping a customization of an old style. It's also possible that some underlying functionality has changed, e.g. some QDR functionality has been dropped, but it was easier to merge the Java code, so I'm more hopeful that once it runs, the functionality will be correct. That should also be checked, but I think styling/layout is probably the main focus.