This document contains the installation and configuration information required to deploy the Data Exchange (DX) ACL APD Server.

In order to connect the DX ACL APD Server with PostgreSQL, RabbitMQ, Email Service, DX Catalogue Server, DX AAA Server, etc please refer Configurations. It contains appropriate information which shall be updated as per the deployment.

In this section we explain about the dependencies and their scope. It is expected that the dependencies are met before starting the deployment of DX ACL APD Server.

• The trustee user must be registered on Keycloak as a user
  • This can be done via the keycloak admin panel, or by using Data Exchange (DX) UI
  • The trustee user need not have any roles beforehand
• The COS admin user must call the create APD API : Link to the API with the name as the name of the APD, owner as email address of trustee (as registered on Keycloak) and URL as the domain of the APD
• Once the APD has been successfully registered, the trustee user will gain the trustee role scoped to that particular APD.
  • They can verify this by calling the list user roles API : Link to the API
• The trustee can get client credentials to be used in APD Operations by calling the get default client credentials API : Link to the API

• To setup RabbitMQ refer the setup and installation instructions available here
• After deployment of RabbitMQ, we need to ensure that there are certain prerequisites met. Incase if it is not met, please login to RabbitMQ management portal using the RabbitMQ management UI and create a the following

Create a DX ACL APD user using the RabbitMQ management UI and set write permission. This user will be used by DX ACL APD server to publish audit data

Body for the API request

 "permissions": [
        {
          "vhost": "IUDX-INTERNAL",
          "permission": {
            "configure": "^$",
            "write": "^auditing$",
            "read": "^$"
          }
        }
]

• To setup PostgreSQL refer setup and installation instructions available here
• Note : PostgreSQL database should be configured with a RBAC user having CRUD privileges

• Auditing is done using the DX Auditing Server which uses Immudb and Postgres for storing the audit logs
• To setup immuclient for immudb please refer immudb setup guide
• The schema for auditing table in PostgreSQL is present here - postgres auditing table schema
• The schema for Immudb table, index for the table is present here - immudb schema in DX Auditing Server

• Note : User ID, endpoint and epoch time are indexed in Immudb to retrieve the logs faster

• Database flyway migrations help in updating the schema, permissions, grants, triggers etc., with the latest version

• Each flyway schema file is versioned with the format V<majorVersion>_<minorVersion>__<description>.sql, ex : V1_1__init-tables.sql

• Schemas for PostgreSQL tables are present here - Flyway schema

• Values like DB URL, database user credentials, user and schema name should be populated in flyway.conf

• The following commands shall be executed

  • mvn flyway:info -Dflyway.configFiles=flyway.conf To get the flyway schema history table
  • mvn clean flyway:migrate -Dflyway.configFiles=flyway.conf To migrate flyway schema
  • mvn flyway:repair To resolve some migration errors during flyway migration

• Please find the reference to Flyway migration here

1. Install java 11 and maven
2. Use the maven exec plugin based starter to start the server. Goto the root folder where the pom.xml file is present and run the below command. mvn clean compile exec:java@acl-apd-server

1. Install java 11 and maven
2. Set Environment variables

export ACL_APD_URL=https://<acl-apd-domain-name>
export LOG_LEVEL=INFO

3. Use maven to package the application as a JAR. Goto the root folder where the pom.xml file is present and run the below command. mvn clean package -Dmaven.test.skip=true
4. 2 JAR files would be generated in the target/ directory
   • iudx.iudx.apd.acl.server-cluster-0.0.1-SNAPSHOT-fat.jar - clustered vert.x containing micrometer metrics
   • iudx.iudx.apd.acl.server-dev-0.0.1-SNAPSHOT-fat.jar - non-clustered vert.x and does not contain micrometer metrics

Note: The clustered JAR requires Zookeeper to be installed. Refer here to learn more about how to set up Zookeeper. Additionally, the zookeepers key in the config being used needs to be updated with the IP address/domain of the system running Zookeeper. The JAR requires 3 runtime arguments when running:

• --config/-c : path to the config file
• --host/-i : the hostname for clustering
• --modules/-m : comma separated list of module names to deploy

e.g. java -jar target/iudx.iudx.apd.acl.server-cluster-0.0.1-SNAPSHOT-fat.jar --host $(hostname) -c secrets/all-verticles-configs/config.json -m iudx.apd.acl.server.authentication.AuthenticationVerticle, iudx.apd.acl.server.apiserver.ApiServerVerticle, iudx.apd.acl.server.policy.PolicyVerticle, iudx.apd.acl.server.notification.NotificationVerticle, iudx.apd.acl.server.auditing.AuditingVerticle

Use the --help/-h argument for more information. You may additionally append an ACL_APD_JAVA_OPTS environment variable containing any Java options to pass to the application.

e.g.

$ export ACL_APD_JAVA_OPTS="Xmx40496m"
$ java $ACL_APD_JAVA_OPTS -jar target/iudx.iudx.apd.acl.server-cluster-0.0.1-SNAPSHOT-fat.jar ...

The JAR requires 1 runtime argument when running

• --config/-c : path to the config file

e.g. java -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4j2LogDelegateFactory -jar target/iudx.iudx.apd.acl.server-cluster-0.0.1-SNAPSHOT-fat.jar -c secrets/all-verticles-configs/config.json

Use the --help/-h argument for more information. You may additionally append an RS_JAVA_OPTS environment variable containing any Java options to pass to the application.

e.g.

$ export ACL_APD_JAVA_OPTS="-Xmx1024m"
$ java ACL_APD_JAVA_OPTS -jar target/iudx.iudx.apd.acl.server-cluster-0.0.1-SNAPSHOT-fat.jar ...

1. Install docker and docker-compose
2. Clone this repo
3. Build the images ./docker/build.sh
4. Modify the docker-compose.yml file to map the config file
5. Start the server in production (prod) or development (dev) mode using docker-compose docker-compose up prod

• For asynchronous logging, logging messages to the console in a specific format, Apache's log4j 2 is used
• For log formatting, adding appenders, adding custom logs, setting log levels, log4j2.xml could be updated : link
• Please find the reference to log4j 2 : here

• Micrometer is used for observability of the application
• Reference link: vertx-micrometer-metrics
• The metrics from micrometer is stored in Prometheus which can be used to alert, observe, take steps towards the current state of the application
• The data sent to Prometheus can then be visualised in Grafana
• Reference link: vertx-prometheus-grafana
• DX Deployment repository references for Prometheus, Loki, Grafana

1. Run the server through either docker, maven or redeployer

2. Run the unit tests and generate a surefire report mvn clean test-compile surefire:test surefire-report:report

3. Jacoco reports are stored in ./target/

Integration tests are through Postman/Newman whose script can be found from here.

1. Install prerequisites

• postman + newman
• newman reporter-htmlextra

2. Example Postman environment can be found here

• Please find the README to setup postman environment file here

3. Run the server through either docker, maven or redeployer

4. Run the integration tests and generate the newman report newman run <postman-collection-path> -e <postman-environment> --insecure -r htmlextra --reporter-htmlextra-export .

5. Command to store report in target/newman: newman run <postman-collection-path> -e <postman-environment> --insecure -r htmlextra --reporter-htmlextra-export ./target/newman/report.html

• JMeter is for used performance testing, load testing of the application
• Please find the reference to JMeter : here
• Command to generate HTML report at target/jmeter

rm -r -f target/jmeter && jmeter -n -t jmeter/<file-name>.jmx -l target/jmeter/sample-reports.csv -e -o target/jmeter/

• For security testing, Zed Attack Proxy(ZAP) Scanning is done to discover security risks, vulnerabilities to help us address them
• A report is generated to show vulnerabilities as high risk, medium risk, low risk and false positive
• Please find the reference to ZAP : here