v3.14: Deployment Instructions - OpenWIS/openwis GitHub Wiki

This page contains the instructions necessary for upgrading from OpenWIS v3.13 to OpenWIS v3.14. A significant part of this upgrade is the uplifting to the newer version of the middle-ware required by OpenWIS. As such it is very important to read all instructions carefully before proceeding.

At a minimum, the upgrade instructions should be written assuming the "reference stack", the reference deployment architecture of OpenWIS. However, if you know of, or are using, an alternative deployment architecture (e.g. running the staging post in JBoss instead of Tomcat) please consider including instructions for uplifting that as well.

Also, if you wish to write your instructions in a separate wiki page, please feel free to do so and add a link to it in the relevant section.

This section contains the instructions to uplift the middleware for each component of OpenWIS. The instructions are broken up into major sub-systems of the OpenWIS...

The Middleware target version scripts can be located under the repo openwis-deploy/middleware (See the README for more information). For example middleware/java_target.sh is the script for updating java to the target version of OpenJDK7

These instructions must be performed for each subsystem listed below. Note: Follow all instructions listed here before continuing.

TODO: Instructions for uplifting the version of RedHat

The Redhat target installation for the uplift was RHEL 6.5 (which was provided by Redhat via the AWS Market Place).

Once RHEL is installed, run the following commands to update yum followed by installation of dos2unix and the httpd server

yum update
yum install -y dos2unix
yum install -y httpd

The Version of Java agreed at the TC was **OpenJDK7_065 ** for all subsystems except for the portal subsystems which used OpenJDK7_019 due to some unresolved bugs in Openam and compatability issues with later versions of Java 7 patches

The following script removes previous java versions and installs the target version OpenJDK7_065 and then sets the path and updated profile

yum erase -y *java*
yum -y install java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el6_5.x86_64
sed -i 's/export PATH.*/JAVA_HOME=\/usr\/lib\/jvm\/java-1.7.0-openjdk-1.7.0.65.x86_64\/jre\nexport JAVA_HOME\nPATH=$JAVA_HOME\/bin:$PATH\n&/' /etc/profile

Note: Follow all instructions listed in the "For All Subsystems" section before continuing.

Note: adopted from RN-MigrationOpenAM-01-v01.0.doc

The following components need to be re-installed as described in the Installation Guide:

• OpenDJ
• OpenAM
• IDPDiscovery
• OpenWIS Security Web Services

To migrate data from OpenDS to OpenDJ:

1. Use export as ldif in OpenDS and import ldif in OpenDJ.
2. Don't import entry ou=people,dc=opensso,dc=java,dc=net (but import sub-entries)
3. Don't import entry ou=groups,dc=opensso,dc=java,dc=net (but import sub-entries)
4. Don't import entry ou=opensso adminusers,dc=opensso,dc=java,dc=net and sub-entries

Note: Follow all instructions listed in the "For All Subsystems" section before continuing.* Note: The database scripts are now under source control in openwis/openwis-database (See the README in this folder for installation details)

In general, the database is uplifted in the following sequence of steps:

• Install PostgreSQL 9.2 and PostGIS 2.1.5 (A script for installing these exists in openwis-deploy/middleware/portgres_target.sh)
• Drop citext extension from existing OpenWIS database
• Dump existing databases
• Configure and start the new PostgreSQL 9.2 server
• Restore the databases
• Re-install citext extension on the OpenWIS database

The above logic is detailed in following 8 steps to complete the uplift. A few assumptions for the process are as follows:

• The operating system is RedHat Enterprise 6

• OpenWIS is the only PostGIS enabled database.

• OpenWIS is the only database with citext extension

• All commands should be run with an account with sudo privilege to root, openwis and postgres. Alternatively you can use the actual account that is sudo'ed to run a command without the leading sudo part.

• Account postgres can start and stop postgresql service and has full permissions for the data directory.

• Once installed Java and Solr both need to be installed on the Database box (see openwis-deploy/DB4 for the scripted uplift on AWS)

Run following command to stop Solr if it is not already stopped.

sudo -u openwis ~openwis/stop_openwis_tomcat.sh

It is strongly recommended to also stop other OpenWIS components:

• Portals
• Data and management services
• Staging post

This backs up the whole working directory, not just a database dump. Assuming the working directory is /data/datadb, run following commands:

sudo -u postgres service postgresql-8.4 stop
sudo -u postgres tar zcf /data/datadb-8.4.tgz /data/datadb
sudo -u postgres service postgresql-8.4 start

PostgreSQL 9.2 server and related softwares are installed from their official repository.

# Install the repo for RedHat Enterprise 6
wget http://yum.postgresql.org/9.2/redhat/rhel-6-x86_64/pgdg-redhat92-9.2-8.noarch.rpm
# Install the softwares
sudo yum -y install pgdg-redhat92-9.2-8.noarch.rpm
sudo yum -y install postgresql92 postgresql92-server postgresql92-contrib postgis2_92-2.1.5
sudo yum -y update proj.x86_64

# Add the new library path globally
sudo bash -c 'cat > /etc/ld.so.conf.d/postgresql-9.2-libs.conf << .
/usr/pgsql-9.2/lib/
.'

# Disable the 8.4 lib path
sudo mv /etc/ld.so.conf.d/postgresql-8.4-libs.conf /etc/ld.so.conf.d/postgresql-8.4-libs.conf.bak

# Active the new lib path
sudo ldconfig -v

The citext extension needs to be dropped first from the existing database before the dump/restore process. This is necessary mainly because some table indices and constraints are not restored correctly for citext columns.

psql -U openwis -d OpenWIS -c 'alter table metadata alter column uuid type varchar(255);'
psql -U openwis -d OpenWIS -c 'alter table deletedmetadata alter column uuid type varchar(255);'
psql -U openwis -d OpenWIS -c 'alter table openwis_product_metadata alter column urn type varchar(255);'
psql -U postgres -d OpenWIS -f /usr/pgsql-8.4/share/contrib/uninstall_citext.sql

PostGIS enabled databases must be restored using the script provided along with the PostGIS package. This requires the OpenWIS database to be dumped separately from all other databases, which implies the OpenWIS database should be excluded from the pg_dumpall output. This is achieved by the following filter script in Python:

import sys
ignore = False
for line in sys.stdin:
    if line.startswith('\\connect "OpenWIS"'):
        ignore = True
    elif line.startswith('\\connect '):
        ignore = False
    if not ignore:
        sys.stdout.write(line)

Simply create above filter script with copy and paste. Then the databases can be dumped as follows:

pg_dumpall -U postgres | python PATH_TO_FILTER_SCRIPT > ALL.dump
pg_dump -U postgres -Fc -b -v -f OpenWIS.dump OpenWIS

NOTE: If there are other PostGIS enabled databases, they must be treated in the same way as the OpenWIS database.

Stop PostgreSQL 8.4 server. Configure and start the 9.2 server.

# Stop postgreSQL 8.4 server
sudo -u postgres service postgresql-8.4 stop

# Rename the old working directory
sudo -u postgres mv /data/datadb /data/datadb-8.4

# Create new working directory for PostgreSQL 9.2
sudo -u postgres mkdir /data/datadb

# Initilize the new database working directory
sudo -u postgres /usr/pgsql-9.2/bin/initdb -D /data/datadb

# Copy the settings
sudo -u postgres cp /data/datadb-8.4/pg_hba.conf /data/datadb/.

# PostgreSQL v9.1+ changes default value of standard_conforming_strings to on. Set it to off explicitly to match v8.4
sudo -u postgres bash -c "cat /data/datadb-8.4/postgresql.conf | sed -e 's/#\(standard_conforming_strings = off\)/\1/' > /data/datadb/postgresql.conf"
sudo cp /etc/sysconfig/pgsql/postgresql-8.4 /etc/sysconfig/pgsql/postgresql-9.2

# Start postgresql 9.2 server
sudo -u postgres service postgresql-9.2 start

All databases except OpenWIS are restored first. This also creates an empty OpenWIS database. PostGIS is required to be installed onto the empty OpenWIS database before the database itself is restored.

# Restore all databases except OpenWIS
psql -U postgres -f ALL.dump >& ALL.restore

# Install PostGIS 2 on the empty OpenWIS database
sudo -u postgres psql -d OpenWIS -f /usr/pgsql-9.2/share/contrib/postgis-2.1/postgis.sql >& postgis.install.log
sudo -u postgres psql -d OpenWIS -f /usr/pgsql-9.2/share/contrib/postgis-2.1/postgis_comments.sql >& postgis_comments.install.log
sudo -u postgres psql -d OpenWIS -f /usr/pgsql-9.2/share/contrib/postgis-2.1/spatial_ref_sys.sql >& spatial_ref_sys.install.log

# Install legacy PostGIS functions to support Solr
sudo -u postgres psql -d OpenWIS -f /usr/pgsql-9.2/share/contrib/postgis-2.1/legacy.sql >& legacy.install.log

# Restore the OpenWIS database (Using PostGIS specific restore script)
/usr/pgsql-9.2/share/contrib/postgis-2.1/postgis_restore.pl OpenWIS.dump | psql -U postgres OpenWIS >& OpenWIS.restore

# Install citext extension
psql -U postgres -d OpenWIS -c "CREATE EXTENSION citext";

# Change the columns back to citext 
psql -U openwis -d OpenWIS -c 'alter table metadata alter column uuid type citext;'
psql -U openwis -d OpenWIS -c 'alter table deletedmetadata alter column uuid type citext;'
psql -U openwis -d OpenWIS -c 'alter table openwis_product_metadata alter column urn type citext;'

Note: Follow all instructions listed in the "For All Subsystems" section before continuing.

Note: openwis-deploy/db_4/db_server_solr_install.sh scripts the installation for the AWS Stack

Instructions to uplift to the latest version of Solr (if required) and Tomcat. Solr is embedded in openpis-portal-solr war file which runs in Tomcat)

1) Create directory Structures for Solr Data if these do not already exist.

mkdir /var/opt/openwis
mkdir /var/opt/openwis/solr
mkdir /var/opt/openwis/solr/data

2) Change permissions of this folder to the user which tomcat is installed as

chown -R tomcat:tomcat /var/opt/openwis

3) Make webapp directory and unpack

mkdir <TOMCAT_HOME>/webapps/openwis-portal-solr

4) Unzip the war file into this folder above (i.e. openwis-portal-solr)

unzip openwis-portal-solr.war -d <TOMCAT_HOME>/webapps/openwis-portal-solr

5) Finally restart Tomcat

Note: Follow all instructions listed in the "For All Subsystems" section before continuing.

NOTE: instructions on how to install OpenJDK 7_065 assumed to be written in the "For All Subsystems" section.

NOTE 2: configuration and scripts are currently stored in a separate ZIP file. This may change in the future.

As the openwis user:

1. Download JBoss AS 7.1.1.Final from download.jboss.org and unzip it in the home directory:

    $ cd
    $ wget 'http://download.jboss.org/jbossas/7.1/jboss-as-7.1.1.Final/jboss-as-7.1.1.Final.tar.gz'
    $ tar -xvzf jboss-as-7.1.1.Final.tar.gz
   

2. The JBOSS_HOME environment variable will need to be changed to the new JBoss AS 7.1 in .bashrc:

    export JBOSS_HOME="$HOME/jboss-as-7.1.1.Final"
   

3. Unzip the openwis-dataservice-config-files.zip and copy the new start and stop scripts into the home directory of OpenWIS.

    $ unzip openwis-dataservice-config-files.zip
    $ cp openwis-dataservice-config/*.sh ~/.
   

4. [Only required for OpenWIS 3.14.6 and earlier] Deploy the openwis-dataservice-config-module into JBoss.

    $ unzip -d $JBOSS_HOME/modules openwis-dataservice-config-module.zip
   

5. Create a conf directory within the OpenWIS home directory. This will house the new configuration files used for OpenWIS.

    $ mkdir ~/conf
   

6. To simplify the migration of the openwis-dataservice-jndi-service.xml file, a migration script has been written (requires Python 2.6+, not Python 3). To perform the migration, run the script with the original path to openwis-dataservice-jndi-service.xml and copy the generated properties file into the newly created conf directory.

    $ cd openwis-dataservice-config/3.14_migration
    $ ./migrate_as5_config.py ~/EnterprisePlatform-5.1.0/jboss-eap-5.1/jboss-as/server/production/deploy/openwis-dataservice-jndi-service.xml
    $ cp *.properties ~/conf/.
   

7. In the same directory is another script used to generate a CLI batch file which will initialize the JBoss AS 5.1 environment for OpenWIS. Open the script in a text editor and change any values of the PROPERTIES dictionary to suite your environment:

    $ vi gen_setup_cli_script.py
    .. change any property values of the PROPERTIES hash ..
   

8. Run the script redirecting all output to a file in the home directory:

    $ ./gen_setup_cli_script.py > ~/setup.cli
   

9. Open $JBOSS_HOME/bin/standalone.conf in a text editor and add the following line:

    JAVA_OPTS="$JAVA_OPTS -Duser.timezone=UTC"
   

10. If JBoss logs are being written to another location, create a symbolic link from $JBOSS_HOME/standalone/log to the new logs directory:

    $ rmdir $JBOSS_HOME/standalone/log
    $ ln -s /location/of/log $JBOSS_HOME/standalone/log
    

11. Start JBoss. The rest of the instructions will need to be performed while JBoss is running.

    $ cd
    $ ./start_openwis_jboss.sh
    

1. Deploy the PostgreSQL JDBC driver used in JBoss AS 5.1 using the CLI:

    $ cd $JBOSS_HOME/bin
    $ ./jboss-cli.sh -c --command="deploy $HOME/EnterprisePlatform-5.1.0/jboss-eap-5.1/jboss-as/server/production/lib/postgresql-8.4-702.jdbc4.jar"
   

2. Run the setup script

    $ ./jboss-cli.sh -c --file=$HOME/setup.cli
   

3. Deploy the new EAR and WAR files

    $ ./jboss-cli.sh -c --command="deploy openwis-management-service.ear"
    $ ./jboss-cli.sh -c --command="deploy openwis-dataservice.ear"
   

Note: Follow all instructions listed in the "For All Subsystems" section before continuing.

These Instructions describe how to uplift in 2 ways

• to the latest version of Tomcat (if deployed in Tomcat) or
• to the latest version of JBoss AS 7 (if deployed alongside the data/management services).

1) Install pre-requisite middleware (RHEL 6.5, OpenJDK 7_065 and Tomcat 7_0.59 (see notes for installing target os, java and tomcat)

2) Sett up folder structure

mkdir /var/opt/openwis
mkdir /var/opt/openwis/stagingPost
chown -R openwis:openwis /var/opt/openwis

3) Unzip Staging post war as openwis user

su -l -c "unzip $1/stagingPost.war -d /var/opt/openwis/stagingPost" openwis

4) Amend server.xml of tomcat to include path if in different location (default is /stagingpost)

sed -i 's/</Host>/ <Context docBase="/var/opt/openwis/stagingPost" path="/stagingPost" reloadable="false"/>\n </Host>/' <TOMCAT_HOME>/conf/server.xml

5) Restart Tomcat

NOTE to uplifters of the Staging Post: I did this to test the uplifted data services. Feel free to use it or replace it as you see fit. -lmika

To deploy the staging post in the same JBoss AS 7.1 instance as the data services:

1. Cd into $JBOSS_HOME/standalone/deployments

2. Create a new link to the staging post data directory with the name stagingPost.war.

    ln -s /data/openwis/stagingPost stagingPost.war
   

3. Touch the file stagingPost.war.dodeploy.

    touch stagingPost.war.dodeploy
   

If the deployment goes smoothly, this file will be renamed to stagingPost.deployed. Do not delete this file: doing so will undeploy the staging post.

Note: Follow all instructions listed in the "For All Subsystems" section before continuing.

TODO: Instructions to uplift to the latest version of Tomcat

The security web services are now deployed in Tomcat. The WSDL URLs need to be update as followed:

http://<host>:<port>/openwis-securityservice/service/<SERVICE>?wsdl

For example:

http://<host>:8080/openwis-securityservice/service/UserManagementService?wsdl

The fedlet configuration step needs to be redone.

The SOAP endpoints of the data and management services have changed as part of the uplift from JBoss AS 5.1 to JBoss AS 7.1. The user portal will need to be reconfigured to make use of these new SOAP endpoints.

The following table shows the service path and new path of each of the data and management services.

Data services (key = openwis.metadataportal.dataservice.service name.wsdl):

Managaement Services (key = openwis.management.service name.wsdl):

To make the necessary changes:

1. Open "openwis-metadataportal.properties" in a text editor.

2. For each entry in the "Data Services" table:

   1. Look for the property referencing the Service Name. For example, the service name "subscriptionservice" will have the key openwis.metadataportal.dataservice.subscriptionservice.wsdl.

   2. Replace the path (i.e. the part after the hostname and port) component of the URL with the New URL Path. For example, for the "subcriptionservice", replace /openwis-dataservice-openwis-dataservice-server-ejb/SubscriptionService?wsdl with openwis.metadataportal.dataservice.subscriptionservice.wsdl. The final value should look something like this:

   openwis.metadataportal.dataservice.subscriptionservice.wsdl=
   http://dataservices:8180/openwis.metadataportal.dataservice.subscriptionservice.wsdl
   

3. Likewise, for each entry in the "Management Services" table:

   1. Look for the property referencing the Service Name. For example, the service name "controlservice" will have the key openwis.management.controlservice.wsdl.

   2. Replace the path (i.e. the part after the hostname and port) component of the URL with the New URL Path. For example, for the "controlservice", replace /openwis-management-service-openwis-management-service-ejb/ControlService?wsdl with /openwis-management-service-ejb/ControlService/ControlService?wsdl. The final value should look something like this:

      openwis.management.controlservice.wsdl=
      http://dataservices:8180/openwis-management-service-ejb/ControlService/ControlService?wsdl
      

   3. The key openwis.management.monitoringservice.wsdl is no longer being used, and can be removed.

Note: Follow all instructions listed in the "For All Subsystems" section before continuing.

TODO: Instructions to uplift to the latest version of Tomcat

(See "Update To Security Services" for User Portal)

(See "Fedlet" for User Portal)

(See "Update To Data Service" for User Portal)