Packaging System - AudiovisualMetadataPlatform/amp_documentation GitHub Wiki
2022
The AMP Packaging System creates a consistent environment to:
- Build the code in AMP repositories and creating distribution packages
- Bootstrap a new AMP installation
- Install distribution packages
- Manage the AMP system configuration
- Start/Stop/Restart the AMP system
The examples below were run on a large server with a high bandwidth network connection. Building in other environments may take substantially longer
Each AMP github repository contains an amp_build.py script which is used to build the code in the repository.
Generally, the usage for the script is:
usage: amp_build.py [-h] [--debug] [--version VERSION] [--package] destdir
positional arguments:
destdir Output directory for package or tool directory root
options:
-h, --help show this help message and exit
--debug Turn on debugging
--version VERSION Package Version
--package build a package instead of installing
But some repository build scripts have more options that are specific for their usage.
The galaxy repo at https://github.com/AudiovisualMetadataPlatform/galaxy is the AMP fork of the galaxy project.
At the time of this writing, galaxy requires Python <= 3.9 to build and run correctly.
The build will (as needed):
- download a galaxy-compatible python venv
- download node.js and modules
- rebuild the client code
The first build will take a considerable amount of time and/or bandwidth.
If a package is not desired, the build script will ignore the destdir parameter and build galaxy in the source directory.
When a package is created, the default version is pulled from the galaxy source code.
[bdwheele@capybara sources]$ git clone https://github.com/AudiovisualMetadataPlatform/galaxy.git
Cloning into 'galaxy'...
remote: Enumerating objects: 497325, done.
remote: Counting objects: 100% (29/29), done.
remote: Compressing objects: 100% (23/23), done.
remote: Total 497325 (delta 10), reused 16 (delta 6), pack-reused 497296
Receiving objects: 100% (497325/497325), 643.16 MiB | 44.48 MiB/s, done.
Resolving deltas: 100% (387756/387756), done.
[bdwheele@capybara sources]$ cd galaxy
[bdwheele@capybara galaxy]$ time ./amp_build.py --package ../../packages/
2022-05-11 10:27:51,987 [INFO ] (amp_build.py:32) Building environment
Initializing tool-data/shared/ucsc/builds.txt from builds.txt.sample
Initializing tool-data/shared/ucsc/manual_builds.txt from manual_builds.txt.sample
Initializing static/welcome.html from welcome.html.sample
Creating Python virtual environment for Galaxy: .venv
... lots and lots of messages ...
Done in 1206.76s.
/srv/scratch/AMP/sources/galaxy
2022-05-11 11:04:12,930 [INFO ] (amp_build.py:53) Copying instance to /tmp/galaxy-build-6y0ukfbl
2022-05-11 11:04:38,752 [INFO ] (amp_build.py:57) Removing non-dist directories
2022-05-11 11:04:52,349 [INFO ] (amp_build.py:83) Creating amp_galaxy package for with version 21.01 in /srv/scratch/AMP/packages
2022-05-11 11:04:53,425 [INFO ] (amp_build.py:109) Package has been created /srv/scratch/AMP/packages/amp_galaxy-21.01.tar
real 37m3.375s
user 21m1.486s
sys 3m59.330s
The amp_mgms repo at https://github.com/AudiovisualMetadataPlatform/amp_mgms contains the sources to all of the Metadata Generating Modules used by AMP. Unlike the other repositories, it pulls in several git submodules so after cloning the repository the other repos have to be pulled in via
git submodule update --init --recursive
The first time building the MGMs uses a lot of disk space, bandwidth, and time. Subsequent builds are substantially faster, but will vary based on the changes made since the last build.
When not building a package, the destdir parameter can point to a galaxy instance to install the MGMs directly into a running instance of galaxy.
When --package is specified, multiple packages are created, one per directory in tools (providing it contains an mgm_build.py script). The packages are split to reduce update download sizes since most smaller MGMs are updated more frequently than the very large ones. At the time of writing, these MGM packages are created:
- amp_mgms-applause_detection
- amp_mgms-gentle
- amp_mgms-ina_speech_segmenter
- amp_mgms-kaldi
- amp_mgms-mgm_python
- amp_mgms-mgms
The version number for each of these MGMs is taken from the mgm_version file that's in each of the tool source directories.
[bdwheele@capybara sources]$ git clone https://github.com/AudiovisualMetadataPlatform/amp_mgms.git
Cloning into 'amp_mgms'...
remote: Enumerating objects: 949, done.
remote: Counting objects: 100% (268/268), done.
remote: Compressing objects: 100% (249/249), done.
remote: Total 949 (delta 189), reused 71 (delta 19), pack-reused 681
Receiving objects: 100% (949/949), 7.56 MiB | 27.35 MiB/s, done.
Resolving deltas: 100% (624/624), done.
[bdwheele@capybara sources]$ cd amp_mgms/
[bdwheele@capybara amp_mgms]$ git submodule update --init --recursive
Submodule 'tools/applause_detection/applause-detection' (https://github.com/AudiovisualMetadataPlatform/applause-detection.git) registered for path 'tools/applause_detection/applause-detection'
Submodule 'tools/gentle/gentle-singularity' (https://github.com/AudiovisualMetadataPlatform/gentle-singularity.git) registered for path 'tools/gentle/gentle-singularity'
Submodule 'tools/ina_speech_segmenter/ina-speech-tools-singularity' (https://github.com/AudiovisualMetadataPlatform/ina-speech-tools-singularity.git) registered for path 'tools/ina_speech_segmenter/ina-speech-tools-singularity'
Submodule 'tools/kaldi/kaldi-pua-singularity' (https://github.com/AudiovisualMetadataPlatform/kaldi-pua-singularity.git) registered for path 'tools/kaldi/kaldi-pua-singularity'
Cloning into '/srv/scratch/AMP/sources/amp_mgms/tools/applause_detection/applause-detection'...
Cloning into '/srv/scratch/AMP/sources/amp_mgms/tools/gentle/gentle-singularity'...
Cloning into '/srv/scratch/AMP/sources/amp_mgms/tools/ina_speech_segmenter/ina-speech-tools-singularity'...
Cloning into '/srv/scratch/AMP/sources/amp_mgms/tools/kaldi/kaldi-pua-singularity'...
Submodule path 'tools/applause_detection/applause-detection': checked out '5afd6a8d17ec8c7344505c09de8cc980a0759520'
Submodule path 'tools/gentle/gentle-singularity': checked out '09cf51d6e9764e2abbc6c87d7ef4d3b57b640bcb'
Submodule path 'tools/ina_speech_segmenter/ina-speech-tools-singularity': checked out '031b70b2e6276bfcddf3ee3c5fb0c1c886ab5732'
Submodule path 'tools/kaldi/kaldi-pua-singularity': checked out '0f0575466423c00c96ff86efc92f849e61e1193e'
[bdwheele@capybara amp_mgms]$ time ./amp_build.py --package ../../packages/
2022-05-11 10:53:03,012 [INFO ] (amp_build.py:37) Setting tempdir to /srv/scratch/AMP/sources/amp_mgms/temp
2022-05-11 10:53:03,012 [INFO ] (amp_build.py:42) Setting SINGULARITY_TMPDIR = /srv/scratch/AMP/sources/amp_mgms/temp
2022-05-11 10:53:03,012 [INFO ] (amp_build.py:46) Setting SINGULARITY_CACHEDIR = /srv/scratch/AMP/sources/amp_mgms/temp/singularity_cache
2022-05-11 10:53:03,012 [INFO ] (amp_build.py:50) Building MGMs
2022-05-11 10:53:03,013 [INFO ] (amp_build.py:61) Running build script /srv/scratch/AMP/sources/amp_mgms/tools/applause_detection/mgm_build.sh
cd applause-detection && singularity build --force --fakeroot ../applause_detection.sif applause-detection.def
INFO: Starting build...
... lots of messages ....
INFO: Build complete: ../applause_detection.sif
mkdir -p /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-87zylfy8/tools/amp_mgms
cp applause_detection.sif applause_detection.py applause_detection.xml /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-87zylfy8/tools/amp_mgms
2022-05-11 10:57:41,259 [INFO ] (amp_build.py:76) Copying . to /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_pkg-fxon8tix
2022-05-11 10:57:42,384 [INFO ] (amp_build.py:93) Creating package for applause_detection with version 20220511_105742 in /srv/scratch/AMP/packages
2022-05-11 10:57:43,807 [INFO ] (amp_build.py:119) New package /srv/scratch/AMP/packages/amp_mgms-applause_detection-20220511_105742.tar
2022-05-11 10:57:44,060 [INFO ] (amp_build.py:61) Running build script /srv/scratch/AMP/sources/amp_mgms/tools/gentle/mgm_build.sh
cd gentle-singularity && singularity build --force --fakeroot ../gentle_forced_alignment.sif Singularity.recipe
INFO: Starting build...
... lots of messages ...
INFO: Build complete: ../gentle_forced_alignment.sif
mkdir -p /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-ix4_qa5j/tools/amp_mgms
cp gentle_forced_alignment.sif gentle_forced_alignment.py gentle_forced_alignment.xml /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-ix4_qa5j/tools/amp_mgms
2022-05-11 11:40:59,902 [INFO ] (amp_build.py:76) Copying . to /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_pkg-sadl5phb
2022-05-11 11:41:00,876 [INFO ] (amp_build.py:93) Creating package for gentle with version 20220511_105742 in /srv/scratch/AMP/packages
2022-05-11 11:41:02,170 [INFO ] (amp_build.py:119) New package /srv/scratch/AMP/packages/amp_mgms-gentle-20220511_105742.tar
2022-05-11 11:41:02,514 [INFO ] (amp_build.py:61) Running build script /srv/scratch/AMP/sources/amp_mgms/tools/ina_speech_segmenter/mgm_build.sh
cd ina-speech-tools-singularity && singularity build --force --fakeroot ../ina_speech_segmenter.sif Singularity.recipe
INFO: Starting build...
... lots of messages ...
INFO: Build complete: ../ina_speech_segmenter.sif
mkdir -p /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-m_bhsw2p/tools/amp_mgms
cp ina_speech_segmenter.sif ina_speech_segmenter.py ina_speech_segmenter.xml /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-m_bhsw2p/tools/amp_mgms
2022-05-11 11:46:18,624 [INFO ] (amp_build.py:76) Copying . to /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_pkg-76l2c6mr
2022-05-11 11:46:19,773 [INFO ] (amp_build.py:93) Creating package for ina_speech_segmenter with version 20220511_105742 in /srv/scratch/AMP/packages
2022-05-11 11:46:21,407 [INFO ] (amp_build.py:119) New package /srv/scratch/AMP/packages/amp_mgms-ina_speech_segmenter-20220511_105742.tar
2022-05-11 11:46:21,859 [INFO ] (amp_build.py:61) Running build script /srv/scratch/AMP/sources/amp_mgms/tools/kaldi/mgm_build.sh
cd kaldi-pua-singularity && if [ ! -e exp2.tar.gz ] || ! sha1sum -c exp2.tar.gz.sha1 ; then rm -f exp2.tar.gz ; wget https://master.dl.sourceforge.net/project/popuparchive-kaldi/exp2.tar.gz ; fi
... lots of messages ...
INFO: Build complete: ../kaldi-pua-cpu.sif
mkdir -p /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-ll0ra43h/tools/amp_mgms
cp kaldi-pua-cpu.sif kaldi.py kaldi.xml kaldi_transcript_to_amp_transcript.py /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-ll0ra43h/tools/amp_mgms
2022-05-11 11:54:06,231 [INFO ] (amp_build.py:76) Copying . to /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_pkg-zw7gff6z
2022-05-11 11:54:10,614 [INFO ] (amp_build.py:93) Creating package for kaldi with version 20220511_105742 in /srv/scratch/AMP/packages
2022-05-11 11:54:16,177 [INFO ] (amp_build.py:119) New package /srv/scratch/AMP/packages/amp_mgms-kaldi-20220511_105742.tar
2022-05-11 11:54:17,621 [INFO ] (amp_build.py:61) Running build script /srv/scratch/AMP/sources/amp_mgms/tools/mgm_python/mgm_build.sh
singularity build --force --fakeroot mgm_python.sif mgm_python.recipe
INFO: Starting build...
... lots of messages ...
INFO: Build complete: mgm_python.sif
mkdir -p /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-jzx26a_c/tools/amp_mgms
cp mgm_python.sif /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-jzx26a_c/tools/amp_mgms
2022-05-11 11:59:07,546 [INFO ] (amp_build.py:76) Copying . to /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_pkg-35o3rziv
2022-05-11 11:59:08,291 [INFO ] (amp_build.py:93) Creating package for mgm_python with version 20220511_105742 in /srv/scratch/AMP/packages
2022-05-11 11:59:09,518 [INFO ] (amp_build.py:119) New package /srv/scratch/AMP/packages/amp_mgms-mgm_python-20220511_105742.tar
2022-05-11 11:59:09,956 [INFO ] (amp_build.py:61) Running build script /srv/scratch/AMP/sources/amp_mgms/tools/mgms/mgm_build.sh
'amp' -> '/srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-k77t12o_/tools/amp_mgms/amp'
... lots of messages ...
'vocabulary_tagging.xml' -> '/srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-k77t12o_/tools/amp_mgms/vocabulary_tagging.xml'
'vocr_to_csv.xml' -> '/srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_build-k77t12o_/tools/amp_mgms/vocr_to_csv.xml'
2022-05-11 11:59:10,016 [INFO ] (amp_build.py:76) Copying . to /srv/scratch/AMP/sources/amp_mgms/temp/amp_mgms_pkg-vpgu0ozu
2022-05-11 11:59:10,025 [INFO ] (amp_build.py:93) Creating package for mgms with version 20220511_105742 in /srv/scratch/AMP/packages
2022-05-11 11:59:10,047 [INFO ] (amp_build.py:119) New package /srv/scratch/AMP/packages/amp_mgms-mgms-20220511_105742.tar
real 66m7.436s
user 144m11.803s
sys 11m13.561s
The kaldi MGM package only builds the CPU-enabled version, not the GPU enabled version. For some reason the upstream repo cannot regenerate the GPU version.
The MGM versions above are packaging dates because the repo branch which sets the mgm_version for each of them was not available on the build machine. They are functionally equivalent.
The AMPPD repo at https://github.com/AudiovisualMetadataPlatform/amppd contains the REST backend of the AMP system that does all of the heavy lifting.
At the time of writing, Java 11 is required for the build
The build uses maven to do the actual war file build and adds a --clean option to amp_build.py to clean up previous builds. If a package is not desired, the build script will print out where the generated war file is located.
The package version is pulled from the pom.xml file.
[bdwheele@capybara sources]$ git clone https://github.com/AudiovisualMetadataPlatform/amppd.git
Cloning into 'amppd'...
remote: Enumerating objects: 17812, done.
remote: Counting objects: 100% (4319/4319), done.
remote: Compressing objects: 100% (1205/1205), done.
remote: Total 17812 (delta 1908), reused 4095 (delta 1871), pack-reused 13493
Receiving objects: 100% (17812/17812), 10.57 MiB | 42.28 MiB/s, done.
Resolving deltas: 100% (8039/8039), done.
[bdwheele@capybara sources]$ cd amppd
[bdwheele@capybara amppd]$ ./amp_build.py --help
usage: amp_build.py [-h] [--debug] [--version VERSION] [--package] [--clean] [destdir]
positional arguments:
destdir Output directory for package or webserver path root
optional arguments:
-h, --help show this help message and exit
--debug Turn on debugging
--version VERSION Package Version
--package build a package instead of installing
--clean Clean previous build & dependencies
[bdwheele@capybara amppd]$ time ./amp_build.py --package ../../packages/
2022-05-11 11:02:51,301 [INFO ] (amp_build.py:52) Building REST WAR
[INFO] Scanning for projects...
... lots of messages ...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:44 min
[INFO] Finished at: 2022-05-11T11:06:37-04:00
[INFO] ------------------------------------------------------------------------
2022-05-11 11:06:37,638 [INFO ] (amp_build.py:70) Building package
2022-05-11 11:06:37,704 [INFO ] (amp_build.py:106) Build complete. Package is in: /srv/scratch/AMP/packages/amp_rest-0.0.1-SNAPSHOT.tar
real 3m46.705s
user 1m59.683s
sys 0m10.813s
The AMPPD-UI repository at https://github.com/AudiovisualMetadataPlatform/amppd-ui contains the source for the UI component of AMP
At the time of writing node.js v12 is required for building
Like the AMPPD repo, there's a --clean option, but in this case it will only remove the downloaded node modules. The destdir parameter is ignored unless the --package option is specified. When just building the repository, the output is in the dist directory.
The package version is derived from the package.json file.
[bdwheele@capybara sources]$ git clone https://github.com/AudiovisualMetadataPlatform/amppd-ui.git
Cloning into 'amppd-ui'...
remote: Enumerating objects: 5695, done.
remote: Counting objects: 100% (160/160), done.
remote: Compressing objects: 100% (94/94), done.
remote: Total 5695 (delta 93), reused 88 (delta 66), pack-reused 5535
Receiving objects: 100% (5695/5695), 6.75 MiB | 21.79 MiB/s, done.
Resolving deltas: 100% (3799/3799), done.
[bdwheele@capybara sources]$ cd amppd-ui/
[bdwheele@capybara amppd-ui]$ ./amp_build.py --help
usage: amp_build.py [-h] [--debug] [--version VERSION] [--package] [--clean] [destdir]
positional arguments:
destdir Output directory for package or webserver path root
optional arguments:
-h, --help show this help message and exit
--debug Turn on debugging
--version VERSION Package Version
--package build a package instead of installing
--clean Clean previous build & dependencies
[bdwheele@capybara amppd-ui]$ time ./amp_build.py --package ../../packages/
2022-05-11 11:15:03,380 [INFO ] (amp_build.py:38) Building Site
2022-05-11 11:15:03,381 [INFO ] (amp_build.py:42) Installing npm modules
... lots of messages ...
2022-05-11 11:21:32,431 [INFO ] (amp_build.py:47) Building distribution
... lots of messages ...
2022-05-11 11:22:49,792 [INFO ] (amp_build.py:119) Build complete. Package is in: /srv/scratch/AMP/packages/amp_ui-0.1.0.tar
real 7m46.636s
user 7m50.885s
sys 1m1.095s
The full set of packages should look something like this:
-rw-rw-r--. 1 bdwheele bdwheele 207M May 11 11:04 amp_galaxy-21.01.tar
-rw-rw-r--. 1 bdwheele bdwheele 1.6G May 11 10:57 amp_mgms-applause_detection-20220511_105742.tar
-rw-rw-r--. 1 bdwheele bdwheele 1.2G May 11 11:41 amp_mgms-gentle-20220511_105742.tar
-rw-rw-r--. 1 bdwheele bdwheele 1.7G May 11 11:46 amp_mgms-ina_speech_segmenter-20220511_105742.tar
-rw-rw-r--. 1 bdwheele bdwheele 3.9G May 11 11:54 amp_mgms-kaldi-20220511_105742.tar
-rw-rw-r--. 1 bdwheele bdwheele 1.4G May 11 11:59 amp_mgms-mgm_python-20220511_105742.tar
-rw-rw-r--. 1 bdwheele bdwheele 430K May 11 11:59 amp_mgms-mgms-20220511_105742.tar
-rw-rw-r--. 1 bdwheele bdwheele 67M May 11 11:06 amp_rest-0.0.1-SNAPSHOT.tar
-rw-rw-r--. 1 bdwheele bdwheele 4.0M May 11 11:22 amp_ui-0.1.0.tar
The packages are simple tarballs with a bit of metadata and the payload in the data directory:
amp_mgms-mgm_python-20220511_105742/
├── amp_package.yaml
└── data
└── tools
└── amp_mgms
└── mgm_python.sif
The metadata is a simple yaml file to inform the installer that it is a valid package and some information about it:
build_date: '20220511_115908'
install_path: galaxy
name: amp_mgms-mgm_python
version: '20220511_105742'
These packages are required to run AMP:
- PostgreSQL >= 12(?)
- ffmpeg (for MediaProbe -- should be removed in the future)
- Java 11
- Python <= 3.9
- git
Installing these are distribution specific and beyond the scope of this document.
- Create a new user and an empty database. Note the username, password, and database.
- Make sure that local connections use the md5 authentication in pg_hba.conf
The AMP bootstrap repository at https://github.com/AudiovisualMetadataPlatform/amp_bootstrap contains the code to bootstrap and manage an AMP installation. The include amp_control.py script is used to manage an AMP installation:
usage: amp_control.py [-h] [--debug] [--config CONFIG]
{init,start,stop,restart,configure,install} ...
positional arguments:
{init,start,stop,restart,configure,install}
Program action
init Initialize the AMP installation
start Start one or more services
stop Stop one or more services
restart Restart one or more services
configure Configure AMP
install Install a service
options:
-h, --help show this help message and exit
--debug Turn on debugging
--config CONFIG Configuration file to use
Create an empty directory, clone the repo, and initialize the installation:
[bdwheele@fedora AMP]$ mkdir test_installation
[bdwheele@fedora AMP]$ cd test_installation
[bdwheele@fedora test_installation]$ git clone https://github.com/AudiovisualMetadataPlatform/amp_bootstrap.git
Cloning into 'amp_bootstrap'...
remote: Enumerating objects: 78, done.
remote: Counting objects: 100% (78/78), done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 78 (delta 42), reused 56 (delta 23), pack-reused 0
Receiving objects: 100% (78/78), 37.87 KiB | 1.31 MiB/s, done.
Resolving deltas: 100% (42/42), done.
[bdwheele@fedora amp_bootstrap]$ ./amp_control.py init
2022-05-11 12:27:22,278 [INFO ] (amp_control.py:89) Installing tomcat 9.0.62 as /home/bdwheele/work_projects/AMP/test_installation/tomcat
2022-05-11 12:27:30,147 [INFO ] (amp_control.py:102) Creating /home/bdwheele/work_projects/AMP/test_installation/packages
2022-05-11 12:27:30,147 [INFO ] (amp_control.py:102) Creating /home/bdwheele/work_projects/AMP/test_installation/galaxy
2022-05-11 12:27:30,147 [INFO ] (amp_control.py:102) Creating /home/bdwheele/work_projects/AMP/test_installation/data
2022-05-11 12:27:30,147 [INFO ] (amp_control.py:102) Creating /home/bdwheele/work_projects/AMP/test_installation/data/symlinks
2022-05-11 12:27:30,147 [INFO ] (amp_control.py:102) Creating /home/bdwheele/work_projects/AMP/test_installation/data/config
2022-05-11 12:27:30,147 [INFO ] (amp_control.py:107) Checking out mediaprobe repository
Cloning into 'MediaProbe'...
remote: Enumerating objects: 63, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 63 (delta 6), reused 17 (delta 4), pack-reused 42
Receiving objects: 100% (63/63), 25.73 KiB | 585.00 KiB/s, done.
Resolving deltas: 100% (21/21), done.
To make things easier, download the AMP packages from wherever they were built (or a distribution website) and place them into the newly-created packages directory. Once they are downloaded they can be installed via amp_control.py.
The usage of the install mode is:
[bdwheele@fedora amp_bootstrap]$ ./amp_control.py install --help
usage: amp_control.py install [-h] [--yes] package [package ...]
positional arguments:
package Package file(s) to install
options:
-h, --help show this help message and exit
--yes Automatically answer yes to questions
The amp_galaxy package must be installed first, and then the rest of them can be installed.
[bdwheele@fedora amp_bootstrap]$ ./amp_control.py install ../packages/amp_galaxy-21.01.tar
Package Data:
Name: amp_galaxy
Version: 21.01
Build date: 20220511_110452
Installation path: /home/bdwheele/work_projects/AMP/test_installation/galaxy
Continue? y
2022-05-11 13:56:56,509 [INFO ] (amp_control.py:170) Installation complete
[bdwheele@fedora amp_bootstrap]$ ./amp_control.py install --yes ../packages/amp_[mru]*
Package Data:
Name: amp_mgms-applause_detection
Version: 20220511_105742
Build date: 20220511_105742
Installation path: /home/bdwheele/work_projects/AMP/test_installation/galaxy
2022-05-11 13:57:28,835 [INFO ] (amp_control.py:170) Installation complete
Package Data:
Name: amp_mgms-gentle
Version: 20220511_105742
Build date: 20220511_114100
Installation path: /home/bdwheele/work_projects/AMP/test_installation/galaxy
2022-05-11 13:57:29,725 [INFO ] (amp_control.py:170) Installation complete
Package Data:
Name: amp_mgms-ina_speech_segmenter
Version: 20220511_105742
Build date: 20220511_114619
Installation path: /home/bdwheele/work_projects/AMP/test_installation/galaxy
2022-05-11 13:57:31,093 [INFO ] (amp_control.py:170) Installation complete
Package Data:
Name: amp_mgms-kaldi
Version: 20220511_105742
Build date: 20220511_115410
Installation path: /home/bdwheele/work_projects/AMP/test_installation/galaxy
2022-05-11 13:57:36,326 [INFO ] (amp_control.py:170) Installation complete
Package Data:
Name: amp_mgms-mgm_python
Version: 20220511_105742
Build date: 20220511_115908
Installation path: /home/bdwheele/work_projects/AMP/test_installation/galaxy
2022-05-11 13:57:40,808 [INFO ] (amp_control.py:170) Installation complete
Package Data:
Name: amp_mgms-mgms
Version: 20220511_105742
Build date: 20220511_115910
Installation path: /home/bdwheele/work_projects/AMP/test_installation/galaxy
2022-05-11 13:57:40,981 [INFO ] (amp_control.py:170) Installation complete
Package Data:
Name: amp_rest
Version: 0.0.1-SNAPSHOT
Build date: 20220511_110637
Installation path: /home/bdwheele/work_projects/AMP/test_installation/tomcat/webapps
2022-05-11 13:57:41,105 [INFO ] (amp_control.py:170) Installation complete
2022-05-11 13:57:41,105 [INFO ] (amp_control.py:178) Deploying war file
Package Data:
Name: amp_ui
Version: 0.1.0
Build date: 20220511_112249
Installation path: /home/bdwheele/work_projects/AMP/test_installation/tomcat/webapps
2022-05-11 13:57:41,219 [INFO ] (amp_control.py:170) Installation complete
2022-05-11 13:57:41,219 [INFO ] (amp_control.py:178) Deploying war file
[bdwheele@fedora amp_bootstrap]$ cat ../install.log
20220511-135656: Package: amp_galaxy Version: 21.01 Build Date: 20220511_110452
20220511-135728: Package: amp_mgms-applause_detection Version: 20220511_105742 Build Date: 20220511_105742
20220511-135729: Package: amp_mgms-gentle Version: 20220511_105742 Build Date: 20220511_114100
20220511-135731: Package: amp_mgms-ina_speech_segmenter Version: 20220511_105742 Build Date: 20220511_114619
20220511-135736: Package: amp_mgms-kaldi Version: 20220511_105742 Build Date: 20220511_115410
20220511-135740: Package: amp_mgms-mgm_python Version: 20220511_105742 Build Date: 20220511_115908
20220511-135740: Package: amp_mgms-mgms Version: 20220511_105742 Build Date: 20220511_115910
20220511-135741: Package: amp_rest Version: 0.0.1-SNAPSHOT Build Date: 20220511_110637
20220511-135741: Package: amp_ui Version: 0.1.0 Build Date: 20220511_112249
[bdwheele@fedora amp_bootstrap]$
All of the amp configuration is stored in amp_bootstrap/amp.yaml. There is an amp_bootstrap/amp.yaml.default that can be used as a starting point. The file is YAML format where there is a different section for each component. Below are some of the configuration settings that are often modified. For paths, a relative directory name will be relative to the "amp_root" value, which is the directory above amp_bootstrap.
Section Key Description
amp host This is the hostname for the AMP URL. For installing on a workstation, "localhost" will work
\ port The base port for the AMP system. It defaults to 8080.
\ https If the AMP url is proxied via https. If true, the AMP URL is: https://<host>/ if false, https://<host>:<port>/
\ data_root this is the toplevel directory used for AMP data storage. It should be "data"
galaxy host This is the hostname that galaxy will bind to. Generally it should be localhost, regardless of the AMP url.
\ root This is URL base for galaxy. It should be at /root/galaxy
\ admin_username The email address for the administration user (will be set up during configuration)
\ admin_password The password for the administration user
\ id_secret This value is used to obfuscate some of the URLs. It should be set to a random bit of text, but should not be changed because it will hide any previous data
\ toolbox This is the menu structure for the workflow editor. Generally it shouldn't be modified
\ uwsgi These are values used for the UWSGI webserver that serves galaxy. Generally these shouldn't be modified
\ galaxy These are values used by galaxy itself. Generally these shouldn't be modified.
ui user_guide_url This URL is used for a custom user guide pages. The default should be https://wiki.dlib.indiana.edu/display/AMP/
\ unit Use this unit name as the default unit for the administration user. It (should be|is) setup during first startup
rest db_host The hostname for the postgres database
\ db_name The database name in postgres
\ db_user The postgres username
\ db_pass The postgres password
\ admin_username Administration username (use this when logging in to AMP as an admin)
\ admin_password Administration password
\ admin_email Administration email (valid email address for the administrator. Not used for AMP login, unlike other accounts)
\ avalon_url URL for Avalon integration
\ avalon_token The authentication token for Avalon access
\ logging_path Path for logging. Default is "logs"
\ storage_path Path for media storage. Default is "storage"
\ dropbox_path Path for AMP dropbox. Default is "dropbox"
\ encryption_secret Text for internal encryption purposes. Set this to some random text
\ jwt_secret Text for jwt token encryption. Set this to some random text.
\ properties These values are used internally and shouldn't generally be modified.
mgms \ These are settings for individual MGMS (see below)
MGM Key Description
aws_comprehend default_bucket Default AWS bucket to use for storage
\ default_access_arn AWS ARN used to access AWS comprehend
aws_transcribe default_bucket Default AWS bucket to use for storage
\ default_directory ?
jira server URL for Jira integration
\ username Jira Username
\ password Jira Password
\ project Project KEY
azure accountId Azure account ID
\ apiKey Azure API key
\ s3Bucket Azure Bucket used for storage
aws aws_access_key_id AWS key id for access
\ aws_secret_access_key AWS secret ID
\ region_name Cloud region to use for AWS services
hmgm auth_key A random string identifying authorized access
\ auth_string ?
\ transcript_api URL root for transcript tool
\ transcript_input ?
\ transcript_media ?
\ ner_api URL root for NER
\ ner_input ?
\ segmentation_api URL root for segmentation
\ segmentation_input ?
Once all of the configuration has been set up, the configuration files can be generated using amp_control.py. On the first configure, the galaxy venv and node.js are initialized so it will take more time than subsequent configurations.
[bdwheele@fedora amp_bootstrap]$ ./amp_control.py configure
2022-05-11 14:39:26,726 [INFO ] (amp_control.py:187) Configuring Galaxy
2022-05-11 14:39:26,728 [INFO ] (amp_control.py:243) Installing galaxy python and node
Creating Python virtual environment for Galaxy: .venv
using Python: python3
... lots of messages ...
* Appending data to /home/bdwheele/work_projects/AMP/test_installation/galaxy/.venv/bin/activate.fish
2022-05-11 14:41:36,832 [INFO ] (amp_control.py:256) Creating galaxy database
Activating virtualenv at .venv
INFO:galaxy.model.migrate.check:Creating database for URI [sqlite:///./database/universe.sqlite?isolation_level=IMMEDIATE]
INFO:galaxy.model.migrate.check:Creating new database from scratch, skipping migrations
INFO: Using database at sqlite:////home/bdwheele/work_projects/AMP/test_installation/galaxy/database/universe.sqlite?isolation_level=IMMEDIATE
/home/bdwheele/work_projects/AMP/test_installation/galaxy/.venv/lib/python3.9/site-packages/sqlalchemy/sql/sqltypes.py:661: SAWarning: Dialect sqlite-pysqlite does *not* support Decimal objects natively, and SQLAlchemy must convert from floating point - rounding errors and other issues may occur. Please consider storing Decimal numbers as strings or integers on this platform for lossless storage.
util.warn(
INFO: Creating user
INFO: Resetting password
INFO: Creating a new API key -- d7d10b43dfcf3318ea98e88e067dc475
INFO: Generating the encoded user_id value
f35eca4491d7f2d6
2022-05-11 14:41:41,965 [INFO ] (amp_control.py:275) Creating the galaxy toolbox configuration
2022-05-11 14:41:41,965 [INFO ] (amp_control.py:287) Creating the MGM configuration file
2022-05-11 14:41:41,966 [INFO ] (amp_control.py:189) Configuring Tomcat
2022-05-11 14:41:41,966 [INFO ] (amp_control.py:191) Configuring UI
2022-05-11 14:41:41,966 [INFO ] (amp_control.py:193) Configuring Backend
2022-05-11 14:41:41,969 [INFO ] (amp_control.py:195) Configuration complete
[bdwheele@fedora amp_bootstrap]$
At this time the installation may require a C-compiler to create the galaxy python venv.
Do not reconfigure AMP while the system is running! At best case it will create system instability. I don't even want to think about worst case.
After the AMP stack has been configured, it can be managed via the amp_control.py script:
[bdwheele@fedora amp_bootstrap]$ ./amp_control.py start --help
usage: amp_control.py start [-h] service
positional arguments:
service AMP service to start, or 'all' for all services
optional arguments:
-h, --help show this help message and exit
[bdwheele@fedora amp_bootstrap]$ ./amp_control.py stop --help
usage: amp_control.py stop [-h] service
positional arguments:
service AMP service to stop, or 'all' for all services
optional arguments:
-h, --help show this help message and exit
[bdwheele@fedora amp_bootstrap]$ ./amp_control.py restart --help
usage: amp_control.py restart [-h] service
positional arguments:
service AMP service to restart, or 'all' for all services
optional arguments:
-h, --help show this help message and exit
At this time, there are two services: galaxy and tomcat. Generally one would use 'all' for the service name:
[bdwheele@fedora amp_bootstrap]$ ./amp_control.py start all
2022-05-11 14:46:11,982 [INFO ] (amp_control.py:508) Starting Galaxy
Activating virtualenv at .venv
Requirement already satisfied: pip>=8.1 in ./.venv/lib/python3.9/site-packages (22.0.4)
... lots of messages ...
*** daemonizing uWSGI ***
2022-05-11 14:48:30,524 [INFO ] (amp_control.py:512) Starting Tomcat
Using CATALINA_BASE: /home/bdwheele/work_projects/AMP/test_installation/tomcat
Using CATALINA_HOME: /home/bdwheele/work_projects/AMP/test_installation/tomcat
Using CATALINA_TMPDIR: /home/bdwheele/work_projects/AMP/test_installation/tomcat/temp
Using JRE_HOME: /usr/lib/jvm/java-11
Using CLASSPATH: /home/bdwheele/work_projects/AMP/test_installation/tomcat/bin/bootstrap.jar:/home/bdwheele/work_projects/AMP/test_installation/tomcat/bin/tomcat-juli.jar
Using CATALINA_OPTS:
Tomcat started.
[bdwheele@fedora amp_bootstrap]$
On the first start up, galaxy will download and install a bunch of packages -- it can take > 5 minutes on the first startup
The admin unit in AMP is not initialized on first startup. Until that gets resolved, the bootstrap_rest_init.py script can be run after the stack is started to generate it
Likewise, one can shut down the amp stack:
[bdwheele@fedora amp_bootstrap]$ ./amp_control.py stop all
2022-05-11 14:50:18,434 [INFO ] (amp_control.py:518) Stopping Tomcat
Using CATALINA_BASE: /home/bdwheele/work_projects/AMP/test_installation/tomcat
... lots of log messages ...
2022-05-11 14:50:18,592 [INFO ] (amp_control.py:521) Stopping Galaxy
Activating virtualenv at .venv
... lots of log messages ...
Setting AMP python path to /home/bdwheele/work_projects/AMP/test_installation/galaxy/tools/amp_schema:/home/bdwheele/work_projects/AMP/test_installation/galaxy/tools/amp_util:
~/work_projects/AMP/test_installation/galaxy/amp_metrics ~/work_projects/AMP/test_installation/galaxy
make: Nothing to be done for 'all'.
~/work_projects/AMP/test_installation/galaxy
Executing: .venv/bin/uwsgi --stop "galaxy.pid"
[bdwheele@fedora amp_bootstrap]$
image2022-5-11_14-53-19.png
(image/png)\
Document generated by Confluence on Feb 25, 2025 10:39