WMStats Overview - dmwm/WMCore GitHub Wiki
WMStats Overview
- Monitor current status of request and collective job. (snap shot of the current status)
- Works as a debugging tool by providing a hierarchical view on request (starts from request to job level to track down the problem)
- Provide search tool for finding specific requests. (REST API, Web GUI)
- Provide error, warning information about the request and infrastructure (WMAgent components)
DB backend (CouchDB)
- wmstats (t0_wmstats): contains job summary for given workflow (including jobs per site, task and wmagent) - data is replicated from wmagents. It is transitional data which updated every given cycle.
- wmstats_logdb: contains errors for each request and wmagent (components/thread) not in terms of individual jobs. (i.e. a request is not advanced to next state due to the missing configuation, etc)
Design
WMStats uses MVVM (Model view view-model) architectural pattern. Codes connecting to DB (couchdb) and retrieving data is organized under Models. Then those data is converted to the data structures used in wmstats DataStruct Codes for displaying retrieved data is resides in Views Connection between data retrieval between Model and View is through by creating the custom events WMStats.Globals.js and EventListener. EventListener is not directly attached to Views but attached to ViewModels. ViewModels is abstract layer of the Views. So when data is retrieved, it will fire custom events and change the ViewModels and which eventually display the necessary view.
For the user interactions, Controller connects between user interaction (typing, mouse click to ViewModel).
The separation between Views and ViewModels is to support easier switch to different views with the same events. (i.e. same data and user interaction can trigger table view or graph (not yet implemented).
Implementation
WMStats collects the data from different sources (ReqMgr2, GlobalWorkQueue - pull, WMAgent -push) and provide the snapshot of job summary by requests) All the active data (not archived status requests) is pulled down to the cache periodically (every 10 min) in server side. And API provide the access to cached data not directly to backend DB. WMStats GUI interfaces is created using JavaScript, JQuery. It is client heavy operation (pull down all the data at once). Then data is refreshed every 5 min in the background. (Currently there is memory leak on client side)
Deployment
WMStats contains duplicate code in minified version. Only Minified js files (which contains all the js files) are deployed but not the original versions. This minified version is create by running the script combineMinifyWMStats.py. (Don't manually change those miified files)
Eventually this script need to be applied in deployment script so it won't be necessary to run this script manually.
Future development
WMStats is designed to handle ~ 5000 active requests. The limitation comes from the data size retrieved to frontend all at once. WMStats has frontend heavy design which gives more flexibility to retrieve statistics without overhead of network communications. However, it will require more memories on client side (web browser). This need to be changed if number of active requests are increased over the limit. To do that server need to send only necessary summary information and provide pagination for all other details.