Deugging, log locations - dmwm/WMCore GitHub Wiki
Once test is set up you can monitor through wmstats for the progress of jobs. If something goes long, check the logs for each component or application
Log files:
-
for reqmgr2, workqueue, reqmon (wmstats) : log files are located under "/data/srv/logs/", Each thread has its own log file and gets rotated everyday, Old logs are zipped.
bash-4.1$ ls /data/srv/logs/reqmgr2/ auxCacheUpdateTasks-20181210.log heartbeatMonitor-20181211.log reqmgr2-20181210.log auxCacheUpdateTasks-20181211.log heartbeatMonitor-20181212.log reqmgr2-20181211.log auxCacheUpdateTasks-20181212.log old-logs-20181202-0007.zip reqmgr2-20181212.log couchCleanup-20181205.log parentageFixTask-20181210.log statusChangeTasks-20181210.log couchCleanup-20181212.log parentageFixTask-20181211.log statusChangeTasks-20181211.log heartbeatMonitor-20181210.log parentageFixTask-20181212.log statusChangeTasks-20181212.log
-
For WMAgent: Each component has "ComponentLog" log file under /data/srv/wmagent/current/install/wmagent
> /data/srv/wmagent/current/install/wmagent/JobAccountant $ ls ComponentLog Daemon.xml.BAK.10-39-2018 Daemon.xml.BAK.29-39-2018
Logging level can be controlled by changing log level for each component in config file: /data/srv/wmagent/current/config/wmagent/config.py - component has to be restarted ( to apply the new log level. (INFO, DEBUG)
config.WorkQueueManager.logLevel = 'INFO'