How to use Docker‐compose with a windows Client - USACE/cwms-data-api GitHub Wiki
Purpose
This page has the instructions for setting up your own Cwms-Data-API (CDA) in docker, for testing purposes. The setup assumes you have VWMare running on windows. Inside VMWare you will have CDA, running in an Ubuntu Linux VM. With this setup you will be able to write programs that work with CDA and access CDA in a browser.
Install and configure Ubuntu
Download from https://ubuntu.com/download/server Note: this memo was written using: Ubuntu 22.04.3 LTS • optional: Configure vmware for no-graphic UI • During Ubuntu install o User: hec o Password: hec o Packages: openssh
Install optional software in Ubuntu
sudo apt install tmux
Allow all ports (such as 8444 used for testing cwms-data-api)
echo "net.ipv4.ip_unprivileged_port_start=0" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
Install and configure docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh ./get-docker.sh
sudo usermod -aG docker $USER
Setup Ubuntu Networking
Setting a fixed IP address of 192.168.206.100
sudo vi /etc/netplan/00-installer-config.yaml
network:
version: 2
ethernets:
ens33:
addresses:
- 192.168.206.100/24
gateway4: 192.168.206.2
nameservers:
addresses:
- 192.168.206.2
After editing the netplan Yaml config run:
sudo netplan apply
Add the following entry to /etc/hosts
192.168.206.100 cwms-data.test auth.test traefik.test
That entry above is needed for networking between different CDA services.
Configure networking in your Windows Computer
In your Windows Host modify your c:\windows\system32\drivers\etc\hosts file so it has these lines:
192.168.206.100 cwms-data.test
192.168.206.100 auth.test
192.168.206.100 traefik.test
**** WARNING ****
Some IT Departments are strongly opposed to users editing the windows hosts file. If you are unsure if this is the case DO NOT do this, and instead just create a Windows Virtual Machine that operates on the same VMware network that can talk to the VM running CDA.
Test your Ubuntu Server
When your Ubuntu is running - connect like this:
C:\>ssh [email protected]
[email protected]'s password:hec
Configure Docker to find Oracle and the CWMS database
Replace your-docker-registry with your actual registry server.
NOTE If you are a USACE employee search discourse for: "Using docker and HEC Nexus" to retrieve the registry domain you should use here. You will also need a login.
docker login https://registry.your-docker-registry
Username: username
Password: ******
create a file /home/hec/cda.env as shown below:
ORACLE_DOCKER_IMAGE=registry.your-docker-registry/oracle/database:19.3.0-ee
CWMS_SCHEMA_INSTALLER_IMAGE=registry.your-docker-registry/cwms/schema_installer:latest-dev
Download and configure CDA Source Code
git clone https://github.com/USACE/cwms-data-api.git
Follow pre-steps in https://github.com/USACE/cwms-data-api/wiki/Docker-Compose
sudo apt-get install openjdk-8-jdk (needed for keytool)
cd compose_files/pki && ./genall.sh
Initial Startup of CDA
From the root of the cwms-data-api project you git clone earlier
Build the docker images using this command:
docker compose --env-file ~/cda.env build
This command Downloads and installs Oracle, downloads, and installs the CWMS schema, and compiles the CDA source code.
Oracle Configuration
Setup SESSION_KEY
When saving data, CDA checks if you have permission using the table AT_API_KEYS. The following command inserts the key ‘sessionkey-for-testing’ into Oracle. A web client using that key has permissions. They key works as both a password and username from the client perspective.
insert into CWMS_20.AT_api_keys VALUES ('L2HECTEST', 'sessionkey-for-testing', 'sessionkey-for-testing', TO_DATE('2023-01-01', 'YYYY-MM-DD'), TO_DATE('2025-01-01', 'YYYY-MM-DD'));
commit;
L2HECTEST is the oracle username associated with this key Key001 is the name of the key sessionkey-for-testing is the secret that is sent with CDA HTTPS commands The first date is the creation date The second date is when the key will expire.
That command can be run by connecting to oracle, running under docker, from your windows computer using sqldeveloper like this:
If you don’t have sqldeveloper, you may run the sql command on the command line inside the container that is running oracle by following the example below. type ‘docker ps’ as shown below to find the container ID for Oracle. It is bold below:
hec@cda:~$ docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Status}}"
CONTAINER ID IMAGE STATUS
ca9eac82bf84 cwms-rest-api:3.0-compose Up 16 minutes (healthy)
0947fd555bf3 quay.io/keycloak/keycloak:19.0.1 Up 17 minutes
**d95f086cca11** registry.your-docker-registry/oracle/database:19.3.0-ee Up 18 minutes (healthy)
7e92d188c379 traefik:v2.9 Up 18 minutes (healthy)
Using the first few characters of the ID, ‘d95’ -- connect to a command prompt inside the oracle container like this:
docker exec -it d95 /bin/bash
start the sqlplus command line tool like this:
sqlplus CWMS_20/simplecwmspasswD1@db:1521/CWMS
Which yields:
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jan 3 21:22:56 2024
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Last Successful login time: Wed Jan 03 2024 20:59:56 +00:00
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SQL>
Run the following command to insert the key like this:
SQL> insert into CWMS_20.AT_api_keys VALUES ('L2HECTEST', 'key001', 'sessionkey-for-testing', TO_DATE('2023-01-01', 'YYYY-MM-DD'), TO_DATE('2025-01-01', 'YYYY-MM-DD'));
1 row created.
SQL>
Setup Queue
This SQL below used to setup a queue:
BEGIN
cwms_20.cwms_msg.create_queues('SPK');
END;
/
Running docker-compose
| Command | Purpose |
|---|---|
docker compose --env-file ../cda.env build |
Compile CDA source and create images |
docker compose --env-file ../cda.env up --force-recreate |
Starts up the containers |
Debugging Techniques
To Debug You may need to look at docker logs, CDA responses, and oracle tables and views.
For example, if you are trying to save data using a HTTP – client, Here are some sources for debug information to help you understand the error.
Sure, here's the information presented in a GitHub Markdown table:
| Debug Source | Example output |
|---|---|
| Return response from CDA | { "message": "Invalid user", "incidentIdentifier": "user input error", "details": {} } |
| This response from CDA is not very helpful. It is trying to avoid disclosing information to hackers. | { "message": "failed to process request", "incidentIdentifier": "5867183783595083007", "details": {} } |
| docker-compose command line output. This error mentions CWMS_MSG package. This error happened when an expected queue did not exist. | ```Validated Api Key for user=L2HECTEST [CONTEXT ratelimit_period="5 SECONDS" ] data-api_1 |
| Cwms Database table: AT_LOG_Message (and AT_LOG_MESSAGE_PROPERTIES for stack) | STORE_TS ERROR on Mount Morris.Elev.Inst.30Minutes.0.GOES-NGVD29-Rev-test ORA-20019: INVALID_ITEM: "SPK_ts_stored" is not a valid message queue name. ORA-06512: at "CWMS_20.CWMS_MSG", line 143 ORA-06512: at "CWMS_20.CWMS_ERR", line 59 ORA-06512: at "CWMS_20.CWMS_MSG", line 117 ORA-06512: at "CWMS_20.CWMS_TS", line 2267 ORA-06512: at "CWMS_20.CWMS_TS", line 2052 ORA-06512: at "CWMS_20.CWMS_TS", line 5294 |
| locating Package source code | select * from user_source where upper(TEXT) like '%DISTINCT%'; |
Items of interest in CWMS
| Item | Notes |
|---|---|
| cwms_20.cwms_db_change_log | CWMS schema version table |
| cwms_20.at_sec_cwms_users | table |
| cwms_20.at_sec_session | table |
| CWMS_20.AT_api_keys | table |
| exec cwms_20.cwms_msg.create_queues('SAS'); | create queue |
| exec cwms_20.cwms_loc.create_location('MyLocation-Sub1',p_db_office_id=>'SAS'); | create a location |
| exec cwms_sec.add_user_to_group('L2HECTEST','CWMS Users', 'SAS'); | add user to group/office |
| exec cwms_sec.add_user_to_group('L2HECTEST','CWMS DBA Users', 'SAS'); | add user to DBA |
| test users | default users |
How can I connect a Java application to the service?
The developer certificate needs to be trusted by importing to your apps JRE.
The file rootca.der (generated by the pre-steps) is copied from compose_files/pki/certs/rootca.der to c:\tmp
C:\tmp>keytool -importcert -alias data-api-dev-rootca -keystore C:\my-app\java64\lib\security\cacerts -storepass changeit -file rootca.der