Ticket #343: Investigate Possible Security Breach – Service Outage on Multiple Servers - SupaHotBall/OE2-Group-D GitHub Wiki

Task

One of our Operations team members reported a possible security attack on a subset of servers, as some critical services went down unexpectedly. Please investigate the situation, identify the cause, and resolve all related issues. Ensure the incident and resolution are documented thoroughly, including:

  • ✅ Any commands, tools, or scripts used for detection and/or resolution
  • ✅ Steps to mitigate current issues and prevent future reoccurrence
  • ✅ Communication trail with stakeholders

Documentation should follow the standard incident response format as detailed below.

  • Trigger/Root Cause(s): Describe the underlying cause of the issue (e.g., malicious IP activity, vulnerability exploit, resource exhaustion, misconfiguration).

  • Impact: List impacted systems/services and how operations/users were affected.

  • Detection: Describe how the issue was identified (e.g., monitoring alerts, manual reports, log analysis).

  • Resolution Steps: Provide a step-by-step description of actions taken to contain, investigate, and resolve the issue.

  • Correspondence with Affected Parties: Include communication with the reporting Operations team member and the manager

  • Before and After Evidence: Screenshots, logs, or command outputs showing system state before and after remediation.

Steps Taken

Identify the issue. From nagios ui we can see that the db-server is down, indicating a connectivity issue with MariaDB. In order to investigate this, we should log into the db server and run sudo systemctl status mariadb and sudo journalctl -u mariadb.service --since "4 hours ago" | grep -Ei "failed|error" to check the logs

image

[group-d@Db ~]$ sudo systemctl status mysql
[sudo] password for group-d:
Sorry, try again.
[sudo] password for group-d:
× mariadb.service - MariaDB 10.6.21 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2025-05-25 16:28:57 UTC; 3h 47min ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 1514288 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SU>
    Process: 1514289 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/S>
    Process: 1514291 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=/usr/bin/galera_recove>
    Process: 1514368 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited,>
   Main PID: 1514368 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 115ms

May 25 16:28:56 db-d systemd[1]: Starting MariaDB 10.6.21 database server...
May 25 16:28:57 db-d mariadbd[1514368]: 2025-05-25 16:28:57 0 [Warning] Can't create test file '/var/lib/mysql/db-d.low>
May 25 16:28:57 db-d mariadbd[1514368]: [92B blob data]
May 25 16:28:57 db-d mariadbd[1514368]: 2025-05-25 16:28:57 0 [ERROR] Aborting
May 25 16:28:57 db-d systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
May 25 16:28:57 db-d systemd[1]: mariadb.service: Failed with result 'exit-code'.
May 25 16:28:57 db-d systemd[1]: Failed to start MariaDB 10.6.21 database server.
lines 1-20/20 (END)

image

We can see from the logs that it is failing to start the mariadb service due to a file permission error

[Warning] Can't create test file '/var/lib/mysql/db-d.lower-test'
[ERROR] Aborting

Fix Attempt 1: Ensure correct ownership of file system permissions

sudo chown -R mysql:mysql /var/lib/mysql

And then check with sudo ls -ld /var/lib/mysql

The file system has the correct permissions so we can sudo systemctl restart mariadb and check if it runs

Nothing changes and the restart of mariadb still fails with the error:

[group-d@Db ~]$ sudo systemctl restart mariadb
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xeu mariadb.service" for details.

We check sudo journalctl -xeu mariadb.service for more details

[group-d@Db ~]$ sudo journalctl -xeu mariadb.service
Journal file /var/log/journal/8f502ed8541244db8ceb559eef1d08d6/user-1002.journal is truncated, ignoring file.
░░ Subject: A start job for unit mariadb.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit mariadb.service has begun execution.
░░
░░ The job identifier is 1415454.
May 25 20:18:02 db-d systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit mariadb.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
May 25 20:18:02 db-d systemd[1]: mariadb.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit mariadb.service has entered the 'failed' state with result 'exit-code'.
May 25 20:18:02 db-d systemd[1]: Failed to start MariaDB 10.6.21 database server.
░░ Subject: A start job for unit mariadb.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit mariadb.service has finished with a failure.
░░
░░ The job identifier is 1415454 and the job result is failed.
[group-d@Db ~]$ sudo ls -ld /var/lib/mysql
drwxr-xr-x 6 mysql mysql 4096 May 25 20:18 /var/lib/mysql
[group-d@Db ~]$ mount | grep '/var/lib/mysql'
[group-d@Db ~]$ sudo systemctl restart mariadb
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xeu mariadb.service" for details.

This doesn't identify a specific problem but returns generic errors which aren't very helpful. However while checking how much free space the disk has a problem can be observed on the db-server.

[group-d@Db ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G   29G     0 100% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           783M   78M  705M  10% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
efivarfs        128M  9.8K  128M   1% /sys/firmware/efi/efivars
/dev/sda15      105M  6.1M   99M   6% /boot/efi
/dev/sdb1       7.8G   28K  7.4G   1% /mnt
tmpfs           392M  4.0K  392M   1% /run/user/1001
[group-d@Db ~]$ df -i
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/root      3870720 123935 3746785    4% /
tmpfs           500738      1  500737    1% /dev/shm
tmpfs           819200    758  818442    1% /run
tmpfs           500738      3  500735    1% /run/lock
efivarfs             0      0       0     - /sys/firmware/efi/efivars
/dev/sda15           0      0       0     - /boot/efi
/dev/sdb1       524288     12  524276    1% /mnt
tmpfs           100147     26  100121    1% /run/user/1001
[group-d@Db ~]$ sudo mysqld --user=mysql
[group-d@Db ~]$

This shows that the space on /dev/root is being 100% used, which could be a reason for why the mariadb service is failing to start. The same problem can be observed on the apps-d server

image

Upon further investigation, a malicious script was found in the /usr/local/bin/tmp/ directory which was running code which would fill up the disk space until it was full. This script was also found in the db-d server

image

In order to resolve this issue, the diskbomb directory which the script creates was removed with sudo rm -r diskbomb and the simdiskattack.service was disabled. After the fixes, disk space went from 100% used back down to 15%

image

image

The same process was followed to free up the disk space on db-d

[group-d@Db ~]$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  3.9G   26G  14% /

The services were then restarted. Apache2 on apps-d and MariaDB on db-d. Apache2 server remained active and up while MariaDB remained active for less than minute before failing again.

Check the error log for MariaDB `sudo cat /var/log/mysql/error.log``

The output was:

Version: '10.6.21-MariaDB-0ubuntu0.22.04.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Ubuntu 22.04
2025-04-08 20:19:32 31 [Warning] Access denied for user 'manager'@'mgmt-d'
2025-04-08 20:19:50 33 [Warning] Access denied for user 'root'@'mgmt-d' (using password: NO)
2025-04-08 20:20:02 34 [Warning] Access denied for user 'root'@'mgmt-d' (using password: NO)
2025-04-08 20:22:28 35 [Warning] Access denied for user 'root'@'mgmt-d' (using password: YES)
2025-04-08 20:22:54 36 [Warning] Access denied for user 'root'@'mgmt-d' (using password: NO)
2025-05-04 20:28:53 7522 [Warning] Aborted connection 7522 to db: 'unconnected' user: 'unauthenticated' host: 'apps-d' (This connection closed normally without authentication)
2025-05-04 20:30:55 7525 [Warning] Access denied for user 'owncloud'@'apps-d' (using password: YES)
2025-05-12  5:41:44 13699 [Warning] Access denied for user 'root'@'mgmt-d' (using password: NO)
2025-05-21  6:30:33 22046 [Warning] Aborted connection 22046 to db: 'owncloud' user: 'owncloud' host: 'apps-d' (Got an error reading communication packets)
2025-05-21  6:30:35 22055 [Warning] Aborted connection 22055 to db: 'owncloud' user: 'owncloud' host: 'apps-d' (Got an error reading communication packets)
2025-05-25 14:36:14 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
2025-05-25 14:36:14 0 [Note] InnoDB: FTS optimize thread exiting.
2025-05-25 14:36:15 0 [Note] InnoDB: Starting shutdown...
2025-05-25 14:36:15 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2025-05-25 14:36:15 0 [ERROR] InnoDB: The error means mariadbd does not have the access rights to the directory.
2025-05-25 14:36:15 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2025-05-25 14:36:15 0 [ERROR] InnoDB: Cannot open '/var/lib/mysql/ib_buffer_pool.incomplete' for writing: Permission denied
2025-05-25 14:36:15 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2025-05-25 14:36:15 0 [ERROR] InnoDB: The error means mariadbd does not have the access rights to the directory.
2025-05-25 14:36:15 0 [Note] InnoDB: Shutdown completed; log sequence number 3106543; transaction id 15694
2025-05-25 14:36:15 0 [Note] /usr/sbin/mariadbd: Shutdown complete
2025-05-25 20:17:11 0 [Note] Starting MariaDB 10.6.22-MariaDB-0ubuntu0.22.04.1 source revision 19644f6821d59ecca0f9b1f44fadb3b887061965 server_uid kgMdB4jyW3pktK6oQyqxQ/oJ+D4= as process 1561532
2025-05-25 20:17:11 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
2025-05-25 20:17:11 0 [ERROR] mariadbd: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
2025-05-25 20:17:11 0 [ERROR] mariadbd: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2025-05-25 20:17:11 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2025-05-25 20:17:11 0 [ERROR] InnoDB: The data file './ibdata1' must be writable
2025-05-25 20:17:11 0 [ERROR] InnoDB: The data file './ibdata1' must be writable
2025-05-25 20:17:11 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2025-05-25 20:17:11 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-05-25 20:17:11 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
2025-05-25 20:17:11 0 [ERROR] Failed to initialize plugins.
2025-05-25 20:17:11 0 [ERROR] Aborting
2025-05-25 20:18:02 0 [No2025-05-25 20:23:48 0 [Note] Starting MariaDB 10.6.22-MariaDB-0ubuntu0.22.04.1 source revision 19644f6821d59ecca0f9b1f44fadb3b887061965 server_uid kgMdB4jyW3pktK6oQyqxQ/oJ+D4= as process 1563368
2025-05-25 20:23:48 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
2025-05-25 20:23:48 0 [ERROR] mariadbd: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
2025-05-25 20:23:48 0 [ERROR] mariadbd: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2025-05-25 20:23:48 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2025-05-25 20:23:48 0 [ERROR] InnoDB: The data file './ibdata1' must be writable
2025-05-25 20:23:48 0 [ERROR] InnoDB: The data file './ibdata1' must be writable
2025-05-25 20:23:48 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2025-05-25 20:23:48 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-05-25 20:23:48 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
2025-05-25 20:23:48 0 [ERROR] Failed to initialize plugins.
2025-05-25 20:23:48 0 [ERROR] Aborting
2025-05-25 20:24:20 0 [Note] Starting MariaDB 10.6.22-MariaDB-0ubuntu0.22.04.1 source revision 19644f6821d59ecca0f9b1f44fadb3b887061965 server_uid kgMdB4jyW3pktK6oQyqxQ/oJ+D4= as process 1563470
2025-05-25 20:24:20 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
2025-05-25 20:24:20 0 [ERROR] mysqld: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
2025-05-25 20:24:20 0 [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2025-05-25 20:24:20 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2025-05-25 20:24:20 0 [ERROR] InnoDB: The data file './ibdata1' must be writable
2025-05-25 20:24:20 0 [ERROR] InnoDB: The data file './ibdata1' must be writable
2025-05-25 20:24:20 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2025-05-25 20:24:20 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-05-25 20:24:20 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
2025-05-25 20:24:20 0 [ERROR] Failed to initialize plugins.
2025-05-25 20:24:20 0 [ERROR] Aborting
[group-d@Db ~]$

These logs show that the permissions are still failing in /var/lib/mysql despite the permissions being changed earlier. This is strange but permissions will be changed again to ensure that the critical files have correct permissions on them

[group-d@Db ~]$ sudo chown -R mysql:mysql /var/lib/mysql
[group-d@Db ~]$ sudo chmod -R 755 /var/lib/mysql
[group-d@Db ~]$ ls -ld /var/lib/mysql
drwxr-xr-x 6 mysql mysql 4096 May 25 20:25 /var/lib/mysql
[group-d@Db ~]$ ls -l /var/lib/mysql | head -n 10
total 189284
-rwxr-xr-x 1 mysql mysql    827392 May 25 14:36 aria_log.00000001
-rwxr-xr-x 1 mysql mysql        52 May 25 14:36 aria_log_control
-rwxr-xr-x 1 mysql mysql     16384 May 20 00:21 ddl_recovery.log
-rwxr-xr-x 1 mysql mysql         0 Apr  2 06:00 debian-10.6.flag
-rwxr-xr-x 1 mysql mysql       910 Apr  8 20:18 ib_buffer_pool
-rwxr-xr-x 1 mysql mysql 100663296 May 25 14:36 ib_logfile0
-rwxr-xr-x 1 mysql mysql  79691776 May 25 14:36 ibdata1
-rwxr-xr-x 1 mysql mysql  12582912 Apr  8 20:18 ibtmp1
-rwxr-xr-x 1 mysql mysql         0 Mar 11 19:59 multi-master.info
[group-d@Db ~]$

The file ownership and permissions now look correct, and all the files mentioned in the error logs are now owned by mysql:mysql with the correct permissions. So restarting the mariadb service will be attempted again

[group-d@Db ~]$ sudo systemctl status mariadb
× mariadb.service - MariaDB 10.6.21 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2025-05-25 20:26:19 UTC; 57s ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 1564434 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SU>
    Process: 1564435 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/S>
    Process: 1564437 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=/usr/bin/galera_recove>
    Process: 1564514 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited,>
   Main PID: 1564514 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 119ms

May 25 20:26:19 db-d systemd[1]: Starting MariaDB 10.6.21 database server...
May 25 20:26:19 db-d mariadbd[1564514]: 2025-05-25 20:26:19 0 [Warning] Can't create test file '/var/lib/mysql/db-d.low>
May 25 20:26:19 db-d mariadbd[1564514]: [92B blob data]
May 25 20:26:19 db-d mariadbd[1564514]: 2025-05-25 20:26:19 0 [ERROR] Aborting
May 25 20:26:19 db-d systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
May 25 20:26:19 db-d systemd[1]: mariadb.service: Failed with result 'exit-code'.
May 25 20:26:19 db-d systemd[1]: Failed to start MariaDB 10.6.21 database server.
[group-d@Db ~]$ sudo tail -n 50 /var/log/mysql/error.log
2025-05-25 20:25:03 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-05-25 20:25:03 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
2025-05-25 20:25:03 0 [ERROR] Failed to initialize plugins.
2025-05-25 20:25:03 0 [ERROR] Aborting
2025-05-25 20:25:07 0 [Note] Starting MariaDB 10.6.22-MariaDB-0ubuntu0.22.04.1 source revision 19644f6821d59ecca0f9b1f44fadb3b887061965 server_uid kgMdB4jyW3pktK6oQyqxQ/oJ+D4= as process 1564017
2025-05-25 20:25:07 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
2025-05-25 20:25:07 0 [ERROR] mariadbd: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
2025-05-25 20:25:07 0 [ERROR] mariadbd: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2025-05-25 20:25:07 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2025-05-25 20:25:07 0 [ERROR] InnoDB: The data file './ibdata1' must be writable
2025-05-25 20:25:07 0 [ERROR] InnoDB: The data file './ibdata1' must be writable
2025-05-25 20:25:07 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2025-05-25 20:25:07 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-05-25 20:25:07 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
2025-05-25 20:25:07 0 [ERROR] Failed to initialize plugins.
2025-05-25 20:25:07 0 [ERROR] Aborting
2025-05-25 20:25:48 0 [Note] Starting MariaDB 10.6.22-MariaDB-0ubuntu0.22.04.1 source revision 19644f6821d59ecca0f9b1f44fadb3b887061965 server_uid kgMdB4jyW3pktK6oQyqxQ/oJ+D4= as process 1564250
2025-05-25 20:25:48 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
2025-05-25 20:25:48 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2025-05-25 20:25:48 0 [Note] InnoDB: Using transactional memory
2025-05-25 20:25:48 0 [Note] InnoDB: Number of pools: 1
2025-05-25 20:25:48 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2025-05-25 20:25:48 0 [Note] InnoDB: Using liburing
2025-05-25 20:25:48 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2025-05-25 20:25:48 0 [Note] InnoDB: Completed initialization of buffer pool
2025-05-25 20:25:48 0 [Note] InnoDB: 128 rollback segments are active.
2025-05-25 20:25:48 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2025-05-25 20:25:48 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2025-05-25 20:25:48 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2025-05-25 20:25:48 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2025-05-25 20:25:48 0 [Note] InnoDB: 10.6.22 started; log sequence number 3106543; transaction id 15606
2025-05-25 20:25:48 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-05-25 20:25:48 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2025-05-25 20:25:48 0 [Note] InnoDB: Buffer pool(s) load completed at 250525 20:25:48
2025-05-25 20:25:48 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2025-05-25 20:25:48 0 [Note] Server socket created on IP: '0.0.0.0'.
2025-05-25 20:25:48 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.6.22-MariaDB-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Ubuntu 22.04
2025-05-25 20:26:15 31 [Warning] Access denied for user 'root'@'localhost'
2025-05-25 20:26:18 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
2025-05-25 20:26:18 0 [Note] InnoDB: FTS optimize thread exiting.
2025-05-25 20:26:19 0 [Note] InnoDB: Starting shutdown...
2025-05-25 20:26:19 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2025-05-25 20:26:19 0 [ERROR] InnoDB: The error means mariadbd does not have the access rights to the directory.
2025-05-25 20:26:19 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2025-05-25 20:26:19 0 [ERROR] InnoDB: Cannot open '/var/lib/mysql/ib_buffer_pool.incomplete' for writing: Permission denied
2025-05-25 20:26:19 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2025-05-25 20:26:19 0 [ERROR] InnoDB: The error means mariadbd does not have the access rights to the directory.
2025-05-25 20:26:19 0 [Note] InnoDB: Shutdown completed; log sequence number 3106555; transaction id 15607
2025-05-25 20:26:19 0 [Note]

The MariaDB service has failed again, the error logs showing the exact same permission writing issues. At this point it is pretty clear that there is another malicious script or service running because the file permissions have changed despite them being changed manually. To confirm this, we can check the file permissions for /var/lib/mysql

[group-d@Db ~]$ ls -ld /var/lib/mysql
d--------- 7 mysql mysql 4096 May 25 21:05 /var/lib/mysql
[group-d@Db ~]$

It has returned to a state where no one has permissions for the file so there must be a script which is rewriting the permissions every time even though the permissions have already been changed manually.

The sim_mariadb_attack.sh script is found in the usr/local/bin/tmp directory - the same directory where the diskbomb script was discovered. The script is removed

image

Then the simmariadbattack service is located in /etc/systemd/system and disabled and removed too

image

After that, the daemon needs to be reloaded because the mariadbattack service was part of the system files

sudo systemctl daemon-reload

Restart the MariaDB service

sudo systemctl restart mariadb

Check the status

sudo systemctl status mariadb

image

The server has been up since the fix so MariaDB is up and running now. We can also connect to our OwnCloud web ui and login as normal

After Fix (Nagios UI):

image

Before Fix (Apache):

image

After Fix (Apache):

image

Before Fix (MariaDB):

image

After Fix (MariaDB):

image

Before Fix (OwnCloud):

image

After Fix (OwnCloud):

image

Challenges

  • Misleading File Permissions: Although chown and chmod were applied correctly to /var/lib/mysql, MariaDB still failed due to permissions silently reverting. This initially suggested a deeper system issue or persistent misconfiguration

  • Undetected Malicious Services: The sim_mariadb_attack and sim_disk_attack required thorough manual inspection of /usr/local/bin/tmp/ and /etc/systemd/system/ to identify and remove

  • False Sense of Recovery: MariaDB would sometimes start and run for up to a minute before crashing again. This created the illusion that the issue was resolved, leading to delays in identifying root causes

  • Monitoring Blind Spots: Nagios showed the database and web services as UP even when OwnCloud was unreachable. This led to a false positive state until manual checks revealed the truth. This gap delayed incident detection

External Resources

N/A

Ticket Reference

https://rt.dataraster.com/Ticket/Display.html?id=343