Troubleshooting Scholar@UC - uclibs/ucrate GitHub Wiki

When troubleshooting problems with Scholar@UC, it's best to start by looking at the Scholar application logs. The logs will often tell you if the problem is related to Fedora, Solr, Redis, etc.

Scholar application

Runs on each web server

  • libschpwl1 & libschpwl2 (scholar.uc.edu)
  • libschqwl1 & libschqwl2 (scholar-qa.uc.edu)
  • libschdwl1 (scholar-dev.uc.edu)

Restart the app

  1. sudo su - scholar
  2. touch /srv/apps/curate_uc/tmp/restart.txt

Data

Configuration

  • /srv/apps/curate_uc/config

Logs

  • /srv/apps/curate_uc/log (this is a symlink on both load balanced servers to /mnt/common/scholar-logs/)
  • Turn off detailed errors in config/environments/production.rb
config.consider_all_requests_local = false

Apache

Web server

Runs on each web server

  • libschpwl1 & libschpwl2 (scholar.uc.edu)
  • libschqwl1 & libschqwl2 (scholar-qa.uc.edu)
  • libschdwl1 (scholar-dev.uc.edu)

Start

  • sudo systemctl start httpd

Stop

  • sudo systemctl stop httpd

Check if it's running

  • sudo systemctl status httpd

Ports

  • 80
  • 443

Configuration

  • /etc/httpd/
  • /etc/httpd/conf.d/25-scholar-qa.uc.edu.conf
  • /etc/httpd/conf.d/25-ssl-scholar-qa.uc.edu.conf

Logs

  • /var/log/httpd/

Dependencies

  • (none)

Fedora

Object store for content in Scholar@UC.

Runs on these servers

  • libschpfl1 (scholar.uc.edu)
  • libschqfl1 (scholar-qa.uc.edu)
  • libschdal1 (scholar-dev.uc.edu)

Start

  • sudo su - tomcat
  • /mnt/hydra/fcrepo/etc/fcrepo_start.sh

Stop

  • sudo su - tomcat
  • /opt/tomcat/bin/shutdown.sh

Check if it's running

  • ps -ef | grep java | grep fcrepo

Ports

  • 8080

Data

  • /mnt/hydra/fcrepo/data (Isilon)
  • MySQL database

Configuration

  • /mnt/hydra/fcrepo/etc/fcrepo_config.sh
  • /mnt/hydra/fcrepo/etc/fcrepo_variables.sh

Logs

  • /mnt/hydra/fcrepo/logs (Isilon)
  • /opt/tomcat/logs

Dependencies

  • Tomcat
  • Java
  • MySQL

Web Interface

  • Tunnel to the Fedora server:
    • Ex. ssh vbbssh.ad.uc.edu -L 8080:libschqfl1:8080
  • In browser visit http://localhost:8080/fedora

Fits

Provides file characterization

Runs on each web server

  • libschpwl1 & libschpwl2 (scholar.uc.edu)
  • libschqwl1 & libschqwl2 (scholar-qa.uc.edu)
  • libschdwl1 (scholar-dev.uc.edu)

Start

  • (does not stay resident in memory)

Stop

  • (does not stay resident in memory)

Check if it's running

  • (does not stay resident in memory)

Ports

  • (none)

Configuration

  • (none)

Logs

  • (none)

Dependencies

  • Java

MySQL (Fedora)

Required by the Fedora application

Runs on these servers

  • libitdb (scholar.uc.edu)
  • libitdbtest (scholar-qa.uc.edu)
  • libitdbtest (scholar-dev.uc.edu)

Start

  • sudo systemctl start mysql.service

Stop

  • sudo systemctl stop mysql.service

Check if it's running

  • sudo systemctl status mysql.service

Ports

  • 3306

Data

  • /mnt/hydra/fcrepo/mysql

Configuration

  • /etc/my.cnf

Logs

  • /var/log/mysqld.log

Dependencies

  • (none)

MySQL (Scholar app)

Required by the Scholar@UC Rails application

Runs on these servers

  • libitdb (scholar.uc.edu)
  • libitdbtest (scholar-qa.uc.edu)
  • libitdbtest (scholar-dev.uc.edu)

Passenger

Application server

Runs on each web server

  • libschpwl1 & libschpwl2 (scholar.uc.edu)
  • libschqwl1 & libschqwl2 (scholar-qa.uc.edu)
  • libschdwl1 (scholar-dev.uc.edu)

Start

  • Starts with Apache

Stop

  • Stops with Apache

Check if it's running

  • ps aux | grep 'Passenger' | grep -v grep

Ports

  • Whatever ports Apache uses

Configuration

  • /etc/httpd/
  • /etc/httpd/conf.d/passenger.conf
  • /etc/httpd/conf.d/25-scholar-qa.uc.edu.conf
  • /etc/httpd/conf.d/25-ssl-scholar-qa.uc.edu.conf

Logs

  • /var/log/httpd/

Dependencies

  • Apache

Redis

In-memory data store. Used by Scholar to manage background worker queues.

Runs on each web server

  • libschpwl1 & libschpwl2 (scholar.uc.edu)
  • libschqwl1 & libschqwl2 (scholar-qa.uc.edu)
  • libschdwl1 (scholar-dev.uc.edu)

Start

  • /usr/local/bin/redis-server --daemonize yes

Stop

  • sudo kill `ps aux | grep 'redis-server' | grep -v grep | awk '{print $2}'`

Check if it's running

  • ps aux | grep 'redis-server' | grep -v grep

Ports

  • 6379

Configuration

  • (none)

Logs

  • (none)

Dependencies

  • (none)

Shibboleth

Provides authentication via UC's Shibboleth

Runs on each web server

  • libschpwl1 & libschpwl2 (scholar.uc.edu)
  • libschqwl1 & libschqwl2 (scholar-qa.uc.edu)
  • libschdwl1 (scholar-dev.uc.edu)

Start

  • sudo systemctl start shibd

Stop

  • sudo systemctl stop shibd

Check if it's running

  • sudo systemctl status shibd

Ports

  • (none)

Configuration

  • /etc/shibboleth/
  • /etc/shibboleth/shibboleth2.xml
  • /etc/shibboleth/attribute-map.xml
  • /etc/httpd/conf.d/shib.conf
  • /etc/httpd/conf.d/25-scholar-qa.uc.edu.conf
  • /etc/httpd/conf.d/25-ssl-scholar-qa.uc.edu.conf

Logs

  • /var/logs/shibboleth/
  • /var/logs/shibboleth-www/

Dependencies

  • (none)

Sidekiq

Processes background jobs like attaching files to works, characterization, and creating derivatives.

Runs on each web server

  • libschpwl1 & libschpwl2 (scholar.uc.edu)
  • libschqwl1 & libschqwl2 (scholar-qa.uc.edu)
  • libschdwl1 (scholar-dev.uc.edu)

Start

  1. sudo su - scholar
  2. cd /srv/apps/curate_uc
  3. script/restart_sidekiq.sh production (use development on scholar-dev)

Stop

  • sudo su - scholar
  • cd /srv/apps/curate_uc
  • script/kill_sidekiq.sh

Check if it's running

  • ps aux | grep sidekiq | grep busy | grep -v grep

Ports

  • (none)

Configuration

  • /srv/apps/curate_uc/config/sidekiq.yml
  • /srv/apps/curate_uc/script/restart_sidekiq.sh (last line)

Logs

  • /srv/apps/curate_uc/log/sidekiq.log

Dependencies

  • Redis
  • Fits
  • Imagemagick
  • ffmpeg
  • LibreOffice

Web Interface

  1. Tunnel to the web server
    • Ex. ssh vbbssh.ad.uc.edu -L 8000:libschqwl1:80
  2. Visit http://localhost:8000
  3. Log in as a Scholar admin user
  4. Visit http://localhost:8000/sidekiq

Solr

Provides search indexing

Runs on these servers

  • libschpsl1 (scholar.uc.edu)
  • libschqsl1 (scholar-qa.uc.edu)
  • libschdal1 (scholar-dev.uc.edu)

Start

  • sudo service solr start

Stop

  • sudo service solr stop

Check if it's running

  • ps -ef | grep java | grep solr

Ports

  • 8983

Data

  • /var/solr/data

Configuration

  • /etc/default/solr.in.sh
  • /var/solr/data/collection1/conf

Logs

  • /var/solr/logs

Dependencies

  • Java

Web Interface

  • Tunnel to the Fedora server:
    • Ex. ssh vbbssh.ad.uc.edu -L 8983:libschqsl1:8983
  • In browser visit http://localhost:8983