Trouble Shooting the Library Website

There are three main reasons the library site will go down

1. Apache Errors: See Apache Debugging
2. Postgres Errors: See MariaDB Debugging
3. Solr: See Solr Debugging (note this is unlikely to bring the entire site down)

Viewing the Load Balancer can also help to diagnose some issues.

Apache Debugging

Please log into the library servers library-prod1, library-prod3, and library-prod4 to check the apache server

ssh pulsys@library-prod1
ps -ef |grep apache

If there are more than ten apache processes the disk has likely dropped. Reboot the server

sudo /sbin/reboot

MariaDB Debugging

Otherwise check the connection to mariadb

ssh [email protected]
sudo su root
 mysqladmin status

That should respond quickly and not have a ton of slow queries. For example:

> mysqladmin status
Uptime: 8295460  Threads: 11  Questions: 1519382769  Slow queries: 0  Opens: 5741634  Flush tables: 1  Open tables: 2000  Queries per second avg: 183.158

If you do not get a quick response or things look weird reboot the server

sudo /sbin/reboot

Solr Debugging

Otherwise check the connection to solr

cd <local pul_solr directory>
 bundle exec cap library-prod solr:console

If the graphs to the right are full, or there are lots of errors in the log. Stop the solr service (It may take a minute or two to stop)

sudo service solr stop
sudo service solr start