ZXINFO (STEP 01) ‐ Elasticsearch ‐ How to v2 - thomasheckmann/zxinfo-es GitHub Wiki

How to create the Elasticsearch instance for ZXINFO

Basic requirements:

  • Docker Desktop
  • nodeJS v18 - (optional) nvm - for managing multiple version of nodeJS
  • local updated version of ZXDB

Creating the Elasticsearch instance for ZXInfo API requires the following steps:

  • populate mariaDB with latest ZXDB
  • add ZXINFO additional info, such as ZXDB version and TOSEC table
  • create JSON for all entries
  • update JSON with pre-generated screenshots (convert from SCR to PNG/GIF)
  • calculate md5hash for all files in ZXDB (for filecheck endPoint)

(1) Automated process for creating new Elasticsearch instance

To create a new instance for ZXInfo:

cd ~/Public/HETZNER_SITES/ZXINFO/elastic && ./create-new-zxinfo-es.sh --version 1.0.231

See below for detailed instructions, in case something fails and debugging is required.

(2) PREPARE new release

Update swagger (api)

cd ~/Public/HETZNER_SITES/ZXINFO/backend/zxinfo-api-v3/public && open -a "Visual Studio Code" swagger_v3.yaml
- change update date

- check branch is v4...
git branch
git add swagger_v3.yaml && git commit -m "ZXDB update" && git push

Update WebApp (Vue) with What is new

Creating 'what's new' requires the previous ES instance, which is compared to the newly generated JSON documents. It uses the environment variables ZXDB_NEW & ZXDB_OLD

cd ~/Public/HETZNER_SITES/ZXINFO/elastic/zxinfo-es/ && node change-log.js && cp news.json ~/Public/HETZNER_SITES/ZXINFO/frontend/zxinfo-vue/src/
cd ~/Public/HETZNER_SITES/ZXINFO/frontend/zxinfo-vue && open -a "Visual Studio Code" src/views/Home.vue
- (update date)
-- check branch is v4...
git branch
git add src/news.json src/views/Home.vue && git commit -m "ZXDB update" && git push

(3) Publish new Elasicsearch Index to ZXInfo

# update assets, e.g. new generated screenshots
cd ~/Public/HETZNER_SITES/ZXINFO/elastic/assets && ./update_ZXINFO.sh

# export ES instance and transfer to host
cd ~/Public/HETZNER_SITES/ZXINFO/elastic/es-import && ES_PORT=9400 ./export_zxinfo.sh --games

# Elasticsearch files to import can be found on the server in ~docker/ZXINFO/es-import

# remote to ES host and import new ES instance
ssh [email protected]
cd ~/ZXINFO/es-import
ES_PORT=9000 ./import_zxinfo.sh

# update swagger/API info (v4)
cd ~/ZXINFO/zxinfo-api-v3
git branch (check for v4)
git pull

cd ~/ZXINFO
UID_GID="$(id -u):$(id -g)" docker compose -p zxinfo up -d --no-deps --build zxinfo-api-v3

# web app
cd ~/zxinfo.dk/zxinfo-vue
git branch (check for v4)
git pull

cd ~/zxinfo.dk/zxinfo-vue
UID_GID="$(id -u):$(id -g)" docker compose up -d --no-deps --build zxinfo-vue
Details for the update process: ## Setup Elasticsearch container A running instance of Elasticsearch must be running, for example use the setup supplied and if upgrading a cleanup is recommended.

For setting up recommended local environment using docker, start a new terminal:

cd ~/Public/HETZNER_SITES/NEW_ZXINFO
UID_GID="$(id -u):$(id -g)" docker compose -f docker-compose.local.yml -p zxinfo_local up -d

Elasticsearch is now running on port 9400. NOTE If intended to fully run/develop API v5, add zxinfo as alias on the zxinfo-* index when data has been imported.

Setup NodeJS environment

The programs needed to create the data files, needs to know what version of ZXDB it's building for - as well as what was the last published version.

Editing the file zxinfo-es/.env to match new version of ZXDB and make environment ready to build:

vi ~/Public/ZXINFO/zxinfo-es/.env
cd ~/Public/ZXINFO && source ./setENV.sh && echo $ZXDB_NEW

# in case nvm is used, make sure the correct version of nodeJS is being used. Currently the v18.x is being used
nvm use v22.19.0

Check $ZXDB_NEW is set correct as output from the above command.

Create new directory for generated files for this release:

mkdir ~/Public/ZXINFO/zxinfo-data/release-$ZXDB_NEW

Synchronize assets and generate MD5 hash

Data depend on updated assets from old WoS, SpectrumComputing, TOSEC and a few other sources.

Update assets from SpectrumComputing

cd ~/Public/ZXINFO/assets && ./syncSC.sh
nvm use v22.19.0

Generate MD5 for TOSEC 2020 & 2023 (should only be done once!):

cd generate-hash
node index-tosec.js --file tosec_2020.txt --tosec 2020 2>skipped-tosec-2020.txt | tee output.txt && sort output.txt >output-tosec-2020.txt
node index-tosec.js --dir ../TOSEC_2023_ZXDB --tosec 2023 2>skipped-tosec-2023.txt | tee output.txt && sort output.txt >output-tosec-2023.txt

Generate MD5 for ZXDB assets (after each ZXDB update):

cd generate-hash
node index-zxdb.js --all 2>skipped-zxdb-$ZXDB_NEW.txt | tee output.txt && sort output.txt >output-zxdb-$ZXDB_NEW.txt

Generate MD5 json for everything (merges with existing and removes duplicates):

mkdir ~/Public/ZXINFO/zxinfo-data/release-$ZXDB_NEW/md5hash

node md5hash2json.js --input output-tosec-2020.txt --output ~/Public/ZXINFO/zxinfo-data/release-$ZXDB_NEW/md5hash
node md5hash2json.js --input output-tosec-2023.txt --output ~/Public/ZXINFO/zxinfo-data/release-$ZXDB_NEW/md5hash
node md5hash2json.js --input output-zxdb-$ZXDB_NEW.txt --output ~/Public/ZXINFO/zxinfo-data/release-$ZXDB_NEW/md5hash
node md5hash2json.js --input zx81stuff.co.uk.tsv --output ~/Public/ZXINFO/zxinfo-data/release-$ZXDB_NEW/md5hash

Populate mariaDB with latest ZXDB

It is assumed the local ZXDB is already available as part of SC update, other wise see section for SPECTRUMCOMPUTING ZXDB update in SPECTRUMCOMPUTING main repository.

Adding ZXINFO specific tables

Add tables for ZXDB version, adjust as needed for latest version.

-- Adjust for ZXDB version
-- Run in myphpAdmin
CREATE TABLE `zxdb-1.0.216`.zxinfo_version ( version VARCHAR(16) NOT NULL );
INSERT INTO zxinfo_version (version) VALUES ('1.0.216');

Add tables and import data for TOSEC information.

SET character_set_client = 'utf8';
-- create table for TOSEC reference import
DROP TABLE IF EXISTS tmp_tosec;
CREATE TABLE tmp_tosec (
  zxdb_id int(11) DEFAULT NULL,
  path varchar(250) COLLATE utf8_bin
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

Now use phpMyAdmin to import the ./zxinfo-db/ZXInfoExt/TOSEC/tosec_2023.txt into the table 'tmp_tosec' with the following settings:

  • CVS using LOAD DATA
  • Columns separated with: '\t'

Create entries

mkdir ~/Public/ZXINFO/zxinfo-data/release-$ZXDB_NEW/entries
cd ~/Public/ZXINFO/zxinfo-es && node --max-old-space-size=8192 create-entries-documents.js --all --output ~/Public/ZXINFO/zxinfo-data/release-$ZXDB_NEW/

NOTE:

  • Running the create-zxinfo-documents is known to fail, but all documents are created
  • zxscreens.txt contains filenames for screenshots that needs to be converted (later step)

Mapping and import into Elasicsearch

Import the generated JSON into Elasticsearch (check esConfig and settings.sh for PORT and HOST)

(cd ./Scripts/ && ./createEntries.sh)

Convert entry screenshots and import md5hash

Screenshot references in the zxscreen.txt file (create when generation JSON documents) needs to be converted from .scr to .png/gif (.gif if flash is used). Cleanup before processing screens are recommended.

mkdir ~/Public/ZXINFO/zxinfo-data/release-$ZXDB_NEW/screens

(cd UpdateScreens && php convert.php) && node update-new-screens.js ~/Public/ZXINFO/zxinfo-data/release-$ZXDB_NEW/screens/ && node update-md5hash.js ~/Public/ZXINFO/zxinfo-data/release-$ZXDB_NEW/md5hash/

Text scanning (BETA)

Make sure fontFiles[] in screen_scanner.js contains all fonts.

cd ~/Public/ZXINFO/zxinfo-text-scan/
node index.js --all 2>info.txt
node update-textscan.js output/textscan/

Starting ZXINFO & Run API Tests

API v3 (v4 branch) - used by Website

nvm use v18.20.3 && cd ~/Public/HETZNER_SITES/ZXINFO/backend/zxinfo-api-v3
DEBUG=zxinfo-api-v3:* NODE_ENV=development PORT=8300 nodemon --ignorpublic/javascripts/config.js --exec 'yarn start'

API v3 now runs on port 8300 - test by pointing your browser to: http://localhost:8300/v3/search?query=head&titlesonly=false&mode=compact&size=25&offset=0&sort=rel_desc

Run automated test to ensure Elasicsearch has been populated correctly:

nvm use v18.20.3 && cd ~/Public/HETZNER_SITES/ZXINFO/backend/zxinfo-api-v3 && node --test

API v5 - new typeahead, used by spectrumcomputing.co.uk (requires zxinfo-search index populated)

nvm use v22 && cd ~/Public/HETZNER_SITES/ZXINFO/backend/zxinfo-api-v5
DEBUG=zxinfo-api* NODE_ENV=development PORT=8400 node node_modules/nodemon/bin/nodemon.js

Start WebApp

cd ~/Public/HETZNER_SITES/ZXINFO/frontend/zxinfo-vue/ && NODE_OPTIONS=--openssl-legacy-provider yarn run serve

Point your browser to localhost:8080 - check date on frontpage and what's new

⚠️ **GitHub.com Fallback** ⚠️