Unexpected Downtime on Critical Services — Suspected Security Attack

Trigger/Root Cause(s)

During investigation, two malicious scripts were found on the database server (db-a). These were deployed as custom systemdservices:

• /etc/systemd/system/simdiskattack.service

• /etc/systemd/system/simmariadbattack.service

The associated scripts were located in:

• /usr/local/bin/tmp/sim_disk_attack.sh

• /usr/local/bin/tmp/sim_mariadb_attack.sh

Contents of the scripts:

• sim_disk_attack.sh: This script continuously created random 10MB binary files in /var/tmp/diskbomb/ using dd and /dev/urandom, rapidly consuming disk space.

• sim_mariadb_attack.sh: This script repeatedly checked if MariaDB was active, then set the directory permissions of /var/lib/mysql to 000, effectively breaking the database. It attempted to restart MariaDB afterward, entering a failure loop.

---

Impact

• Affected System: db-a

• Impacted Services: MariaDB, disk space availability

• Effect: MariaDB was inaccessible due to permission change on /var/lib/mysql, and disk space was rapidly filled by the disk attack script, leading to potential service instability and monitoring failures.

Detection

• Notification was received via the team's monitoring Slack channel, indicating critical services were down.

• Service failure was then confirmed through the Nagios Web UI.

• Investigated disk space usage:

sudo du -h --max-depth=1 /var

• Analyzed timestamp and metadata of /var/tmp/diskbomb directory:

stat /var/tmp/diskbomb/

• Verified diskbomb-related commands using:

sudo grep -r "/var/tmp/diskbomb" /etc /root /home /usr/local /opt 2>/dev/null

• Confirmed .service configuration pointing to the malicious script:

sudo grep -r "/usr/local/bin/tmp/sim_disk_attack.sh" /etc/systemd/system /lib/systemd/system /usr/lib/systemd/system 2>/dev/null

• While detecting and tracing the disk attack, the sim_mariadb_attack.sh and corresponding systemd service were also discovered in the same /usr/local/bin/tmp directory, confirming parallel impact on MariaDB functionality.

• Reviewed the scripts:

---

Resolution Steps

• Investigated and confirmed the malicious systemd services:

sudo systemctl stop simdiskattack.service
sudo systemctl status simdiskattack.service

• Disabled and removed the services:

• Reloaded the systemd daemon:

sudo systemctl daemon-reload

• Removed the script and generated diskbomb content:

• Diskspace error was removed confirmed by the WEB UI:

• Followed similar steps for mitigating the mariadb service attack:

• Restored correct MariaDB permissions:

sudo chown -R mysql:mysql /var/lib/mysql
sudo find /var/lib/mysql -type d -exec chmod 750 {} \;
sudo find /var/lib/mysql -type f -exec chmod 640 {} \;

• Restarted MariaDB and verified access restored:

sudo systemctl restart mariadb.service

---

• Correspondence with Affected Parties:

  • Notified team member of discovery and actions taken
  • Shared progress updates on Teams during incident

---

Sub-Tickets

• Linked Sub-Ticket: apps-a server outage, assigned to separate team member

🔗 https://github.com/GriffinKat/group-a/wiki/Sub%E2%80%90Ticket-ID-%23345:-Investigate-Possible-Security-Breach-%E2%80%93-Service-Outage-on-DB%E2%80%90A-server

---

Time Taken

• 3.5 hours (Investigation, resolution, communication)

---

Lessons Learned

• Systemd can be used as a persistence mechanism for attacks or simulations

• Script-based privilege actions must be clearly communicated or tracked

Tools / Action Items / Commands used

systemctl, chmod, chown, rm, bash_history, find, grep, stat, du

• Suggested actions:

  • Audit who has permission to deploy systemd services

  • Implement logging or alerting on new .service file creation

  • Confirm whether this was an internal test and improve change tracking

---

Ticket Reference - https://rt.dataraster.com/Ticket/Display.html?id=340

---