About add_data.sh - OpenDRR/opendrr-api GitHub Wiki
TODO(Anthony): Move the following to their own wiki pages.
Quick way to test python/add_data.sh during development
docker cp python/add_data.sh opendrr-api_python-opendrr_1:/usr/src/app/add_data.sh && \
docker compose up python-opendrr
docker compose up --build python-opendrr
is helpful too, but it cannot be used to update the add_data.sh
that was previously copied to an existing volume.
FAQ
Where are Elasticsearch data stored? There is no docker volume associated with it?
TODO
RUN mapfile -t ARRAY_NAME < <(jq -r '......' input.json)
(Issue #111)
Regarding First of all, this is related to ShellCheck https://github.com/koalaman/shellcheck/wiki/SC2207 "Prefer mapfile or read -a to split command output (or quote to avoid splitting)". Code like array=( $(mycommand) )
is considered problematic, and mapfile -t array < <(mycommand)
is offered as as correct code for Bash 4.x and up.
RUN mapfile -t ARRAY_NAME < <(jq -r '......' input.json)
truly working and safe?
1. Is the syntax In particular, ss using the RUN function here OK?
The answer is: Yes!
Even though the log only records mapfile -t ARRAY_NAME
and nothing after the <
operator, the redirection is fully functional.
2. jq errors did not stop the build. Is that good?
The <(command_list)
syntax is known as process substitution. It does not stop at errors. set -o pipefail
does not affect this behaviour. And the use of our custom RUN function has nothing to do with it either.
For more information and workarounds (e.g. to get the error code), see the discussion "How to detect an error using process substitution" at https://unix.stackexchange.com/questions/376114/how-to-detect-an-error-using-process-substitution
For our purposes though, I think jq errors not stopping the build is good. :-) It is probably also good to keep a log file and have add_data.sh automatically show warnings and errors at the end.
Notes on optimization, refactoring, testing, debugging, etc.
docker compose logs python-opendrr | egrep '\[add_data|maxresident'
docker compose logs pygeoapi-opendrr | grep '| NotFoundError' | uniq