Rotating New Relic and PM2 log files

Once the application is deployed to the staging, prod-test or production servers, new relic is enabled for application monitoring and PM2 is used to manage TEA instances. These two tools produce log files that are not automatically configurable to truncate themselves when they grow too large. Therefore, there is a need for us to use logrotate to manage the log files.

Background information

New Relic stores its log file in /home/tea/tea/app/newrelic_agent.log. On the other hand, PM2 stores its log files in the .pm2/logs sub-directory of the user's home directory. If PM2 was invoked by root, the log files are at /root/.pm2/logs/*.log or if you used the tea user, it would be at /home/tea/.pm2/logs/*.log

Rotating the log files

First, create a logrotate configuration file

1.. As root create a new file at /etc/logrotate.d/

        $ sudo su
        # vi /etc/logrotate.d/tea

2.. Add the following configuration. This configuration will rotate logs in all 3 locations by size of ~ 5MB, keeping 3 older versions. It will not throw an error if the files are missing, and will not create a new file if the current log file is empty.

/root/.pm2/logs/*.log /home/tea/.pm2/logs/*.log /home/tea/tea/app/newrelic_agent.log /home/cloudmall/server/tomcat/logs/catalina.out {
	size 5M
	rotate 3
	missingok
	notifempty
	sharedscripts
	copytruncate
}

3.. Now we need to set a cron job to run logrotate. Open the crontab file.

        # vi /etc/crontab

4.. Using vi, append the following schedule to the bottom of the crontab file. This schedules logrotate to run every 2 minutes.

        */2 * * * * root /etc/cron.d/tea

5.. Next we create the cron job file.

        # touch /var/log/logrotate.log
        # cp /etc/cron.daily/logrotate /etc/cron.d
        # mv /etc/cron.d/logrotate /etc/cron.d/tea
        # vi /etc/cron.d/tea

Then edit the /etc/cron.d/tea file as follows:

#!/bin/sh

/usr/sbin/logrotate -v /etc/logrotate.d/tea 2>&1 | /usr/bin/tee -a /var/log/logrotate.log >/dev/null
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

6.. (Optional) Next we reload cron to allow it to load the new schedule.

        # service crond restart

Note:

The 5MB log file size and cron job schedule of every 2 minutes is good for initial testing. Once you have verified that the logrotation is working, adjust the file size and schedule as needed. For example, set the log file size to 50MB or 100MB then schedule the job to run every 10 minutes.