minio mc Need help on minio trace log automated collection - allanrogerr/public GitHub Wiki

Issue

net/issues/7208 The proposed workaround with logrotate is flawed as described by https://serverfault.com/questions/221337/logrotate-successful-original-file-goes-back-to-original-size:

PR

https://github.com/minio/operator/pull/1403

When logrotate runs, it truncates the original file but "even though you truncate the file, the process writing to the file will continue writing at whichever offset it were at last. So what's happening is that logrotate truncates the file, size is zero, process writes to the file again, continuing at the offset it left off, and you now have a file with NULL-bytes up to the point where you truncated it plus the new entries written to the log."

Therefore the log "go(es) back to the same size it originally was".

Workaround - Setup

  1. Assuming minio is already installed, start minio in background
export GOROOT=/usr/local/go 
export PATH=$GOROOT/bin:$PATH 
alias minio=$HOME/go/bin/minio
nohup $HOME/go/bin/minio server --address :9000 --console-address :9090 $HOME/data &
  1. Set alias
mc alias set myminio http://localhost:9000 minioadmin minioadmin

Workaround - Logrotate - Fail

  1. Create trace file
sudo touch /var/log/minio.trace
sudo chown ubuntu /var/log/minio.trace
sudo chgrp ubuntu /var/log/minio.trace
sudo chmod 644 /var/log/minio.trace
  1. Setup logrotate
sudo touch /etc/logrotate.d/minio-client-trace
sudo vi /etc/logrotate.d/minio-client-trace

Add text

compress
/var/log/minio.trace {
        daily
        dateext
        copytruncate
}
  1. Restart logrotate
sudo service logrotate restart
sudo service logrotate.timer restart
sudo systemctl status logrotate
sudo systemctl status logrotate.timer
  1. Start trace in background. Ensure sufficient permissions.
nohup mc admin trace -a myminio &> /var/log/minio.trace &
  1. Test logrotate since it only runs once a day by default
sudo logrotate -f /etc/logrotate.d/minio-client-trace
  1. Observe large initial log not truncated
ls -ltr /var/log/minio*
image

Workaround - Apache rotatelogs - Pass

  1. Install apache
sudo apt-get install apache2 apache2-utils
  1. Run mc trace
nohup mc admin trace -a myminio | sudo /usr/bin/rotatelogs /var/log/minio.trace 1K &
  1. Observe logs being created every ~1K. No large logs are untruncatable
image

Solution - WIP

Summary

Added 3 new flags to support trace log files

  --rotate-trace-logs-size value       maximum size of a rotated trace log file (see UNITS)                                  
  --rotate-trace-logs-frequency value  frequency, in terms of seconds, of trace log file rotations                           
  --rotate-trace-logs-directory value  directory into which trace log files are placed
image
  1. Clone and make
git clone https://github.com/allanrogerr/mc/tree/add-rotate-trace
cd mc
make all
  1. Start a kind cluster and login to the console Port forward to service to access mc, and to console to create logs
kubectl port-forward svc/storage-lite-hl -n tenant-lite 9000
kubectl port-forward service/console -n minio-operator 9090

  1. Click around console to create admin trace logs

  2. Demo

a. User must specify trace log type is trying to log the trace to a specific directory

➜  mc git:(add-rotate-trace) ✗ sudo ./mc admin trace --rotate-trace-logs-pattern "/tmp/" minios3 --insecure
mc: <ERROR> You cannot specify flag --rotate-trace-logs-pattern if neither --rotate-trace-logs-size nor --rotate-trace-logs-frequency flags are specified as well.

b. User cannot specify rotate based on size as and frequency at the same time

➜  mc git:(add-rotate-trace) ✗ sudo ./mc admin trace --rotate-trace-logs-size 1kb --rotate-trace-logs-frequency 5 minios3 --insecure
mc: <ERROR> You cannot specify both --rotate-trace-logs-size and --rotate-trace-logs-frequency flags at the same time.

Examples

c. This command creates and logs to a new file every 2kb, in the working directory, with default naming. gzip then compresses the file, and removes the original.

sudo ./mc admin trace --rotate-trace-logs-size 2kb --rotate-trace-logs-compress gzip minios3 --insecure

d. This command creates and logs to a new file every 2kb, in the "/tmp/" directory, with default naming. gzip then compresses the file, and removes the original.

sudo ./mc admin trace --rotate-trace-logs-size 2kb --rotate-trace-logs-compress gzip --rotate-trace-logs-pattern "/tmp/" minios3 --insecure

e. This command creates and logs to a new file every 5kb, in the working directory, with default naming. zstd then compresses the file, and removes the original.

sudo ./mc admin trace --rotate-trace-logs-size 5kb --rotate-trace-logs-compress zstd minios3 --insecure

f. This command creates and logs to a new file every 5kb, in the "/tmp/" directory, with default naming. zstd then compresses the file, and removes the original.

sudo ./mc admin trace --rotate-trace-logs-size 5kb --rotate-trace-logs-compress zstd --rotate-trace-logs-pattern "/tmp/" minios3 --insecure

g. This command creates and logs to a new file every 10kb, in time striped directories directory, with time striped naming.

sudo ./mc admin trace --rotate-trace-logs-size 10kb --rotate-trace-logs-pattern "/tmp/minio/[2006-01-02]/minio-log-[15-04-05].log" minios3 --insecure

h. This command creates and logs to a new file every 10kb, in time striped directories directory, with time striped naming. gzip then compresses the file, and removes the original.

sudo ./mc admin trace --rotate-trace-logs-size 10kb --rotate-trace-logs-compress gzip --rotate-trace-logs-pattern "/tmp/minio/[2006-01-02]/minio-log-[15-04-05].log" minios3 --insecure

i. This command creates and logs to a new file every 10kb, in time striped directories, with time striped naming. zstd then compresses the file, and removes the original.

sudo ./mc admin trace --rotate-trace-logs-size 10kb --rotate-trace-logs-compress zstd --rotate-trace-logs-pattern "/tmp/minio/[2006-01-02]/minio-log-[15-04-05].log" minios3 --insecure

j. This command creates and logs to a new file every 1s, in the "/tmp/ directory, with time striped naming. gzip then compresses the file, and removes the original.

sudo ./mc admin trace --rotate-trace-logs-frequency 1 --rotate-trace-logs-compress gzip --rotate-trace-logs-pattern "/tmp/" minios3 --insecure

k. This command creates and logs to a new file every 5s, in the "/tmp/ directory, with time striped naming. zstd then compresses the file, and removes the original.

sudo ./mc admin trace --rotate-trace-logs-frequency 5 --rotate-trace-logs-compress zstd --rotate-trace-logs-pattern "/tmp/" minios3 --insecure

l. This command creates and logs to a new file every 10s, in time striped directories, with time striped naming.

sudo ./mc admin trace --rotate-trace-logs-frequency 10 --rotate-trace-logs-pattern "/tmp/minio/[2006-01-02]/minio-log-[15-04-05].log"  minios3 --insecure

m. This command creates and logs to a new file every 30s, in time striped directories, with time striped naming.

sudo ./mc admin trace --rotate-trace-logs-frequency 30 --rotate-trace-logs-compress gzip --rotate-trace-logs-pattern "/tmp/minio/[2006-01-02]/minio-log-[15-04-05].log" minios3 --insecure

n. This command creates and logs to a new file every 60s, in time striped directories, with time striped naming. zstd then compresses the file, and removes the original.

sudo ./mc admin trace --rotate-trace-logs-frequency 60 --rotate-trace-logs-compress zstd --rotate-trace-logs-pattern "/tmp/minio/[2006-01-02]/minio-log-[15-04-05].log" minios3 --insecure
⚠️ **GitHub.com Fallback** ⚠️