OpenWIS Troubleshooting - OpenWIS/openwis GitHub Wiki

#OpenWIS Troubleshooting

Purpose and structure

This page is for the benefit of developers / integrators or support users.

Page Structure

Currently there are 3 lightwwight repositories which contribute to the OpenWIS software as well as the automatic deployment and testing

• OpenWIS Code (See https://github.com/OpenWIS/openwis)

  • See the Openwis Wiki and issues sections

• OpenWIS Integration Issues (See https://github.com/OpenWIS/openwis-deploy)

  • See the Openwis Deploy Wiki and issues sections

• OpenWIS Test Issues (See https://github.com/OpenWIS/openwis-automated-tests)

  • See the Openwis Test Wiki and issues sections

##Viewing information / issues

Each repository can be accessed via the main OpenWIS page as well as other projects within the OpenWIS association.

General Integration Issues

Some Issues for convenience which occasionally need attention are also documented here below for supporting integration:

Cloudbees / AWS connection issues

• Cloudbees IP Change - See https://github.com/OpenWIS/openwis-deploy/issues/6

For other issues with Cloudbees, their support available from their site should be used to create a new request https://cloudbees.zendesk.com/hc/en-us

Common Exceptions and Solution

###Synchronization errors

####Issue There is a known issue that if multiple schedules run at the same time an issue can occur whereby several of the jobs con't succeed (write dependencies check issues are reported in the logs). This is due to the way the transaction is implemented and is a known issue from 3.13 onwards.

####Solution / Guidance Schedules should be set up to run not the same time (Setting up schedules to run at least 1 minute apart solves the issue)

Issue

The following exception is displayed while running the ssoAdminTool to configure OpenAM.

com.sun.identity.security.AMSecurityPropertiesException: AdminTokenAction:  FATAL ERROR: Cannot obtain Application SSO token.
Check AMConfig.properties for the following properties
        com.sun.identity.agents.app.username
        com.iplanet.am.service.password

####Solution / Guidance

This is generally because the **ssoAdminTool **cannot connect to the servce using the public Server URL you define to communicate with OpenAM

First From Test run this command from the ssoAdminTools/bin directory to check the server being used

./ssoadm list-servers -u amAdmin -f passwd

should return <YOUR_PUBLIC_SERVER_HOST_NAME>/openam (e.g. http://wisadmin.openwis.io/openam)

1. Ensure you can see out to this url from the openam box (e.g wget http://wisadmin.openwis.io/openam) and is resolve in your /etc/hosts file

###Connections refused or Connection Pool Drops unexpectedly

####Issue

OpenWIS User portal appears to return 0 records from the database (e.g. When retrieving disseminations options for a da ta request)

On further examination on the database logs, connections are being left open and not closed gracefully resulting int firewall closing them prematurely

####Solution / Guidance

By default the tcp_keepalive_time is 7200 on RHEL 6.5 and above

The solution is to decrease the tcp_keep_alive time to a period less that a firewall or other network timeout setting which allow the connection to remain. 10 minutes have been trialled with a production load

To do this the following needs to be run on the VM hosting the Database:

echo 600 > /proc/sys/net/ipv4/tcp_keepalive_time

Note: The OpenWIS application should employ appropriate connection handling so that they don't remain open which lead to these issues and mitigation.