Ticket ID #340 Investigate Possible Security Breach – Service Outage on Multiple Servers - GriffinKat/group-a GitHub Wiki

Unexpected Downtime on Critical Services — Suspected Security Attack

Trigger/Root Cause(s)

During investigation, two malicious scripts were found on the database server (db-a). These were deployed as custom systemdservices:

  • /etc/systemd/system/simdiskattack.service

  • /etc/systemd/system/simmariadbattack.service

The associated scripts were located in:

  • /usr/local/bin/tmp/sim_disk_attack.sh

  • /usr/local/bin/tmp/sim_mariadb_attack.sh

Contents of the scripts:

  • sim_disk_attack.sh: This script continuously created random 10MB binary files in /var/tmp/diskbomb/ using dd and /dev/urandom, rapidly consuming disk space.

  • sim_mariadb_attack.sh: This script repeatedly checked if MariaDB was active, then set the directory permissions of /var/lib/mysql to 000, effectively breaking the database. It attempted to restart MariaDB afterward, entering a failure loop.

image


Impact

  • Affected System: db-a

  • Impacted Services: MariaDB, disk space availability

  • Effect: MariaDB was inaccessible due to permission change on /var/lib/mysql, and disk space was rapidly filled by the disk attack script, leading to potential service instability and monitoring failures.

Detection

  • Notification was received via the team's monitoring Slack channel, indicating critical services were down.

image

  • Service failure was then confirmed through the Nagios Web UI.

image

  • Investigated disk space usage:
sudo du -h --max-depth=1 /var

image

  • Analyzed timestamp and metadata of /var/tmp/diskbomb directory:
stat /var/tmp/diskbomb/

image

  • Verified diskbomb-related commands using:
sudo grep -r "/var/tmp/diskbomb" /etc /root /home /usr/local /opt 2>/dev/null

image

  • Confirmed .service configuration pointing to the malicious script:
sudo grep -r "/usr/local/bin/tmp/sim_disk_attack.sh" /etc/systemd/system /lib/systemd/system /usr/lib/systemd/system 2>/dev/null

image

  • While detecting and tracing the disk attack, the sim_mariadb_attack.sh and corresponding systemd service were also discovered in the same /usr/local/bin/tmp directory, confirming parallel impact on MariaDB functionality.

  • Reviewed the scripts:

image

image


Resolution Steps

  • Investigated and confirmed the malicious systemd services:
sudo systemctl stop simdiskattack.service
sudo systemctl status simdiskattack.service

image

  • Disabled and removed the services:

image

image

  • Reloaded the systemd daemon:
sudo systemctl daemon-reload

image

  • Removed the script and generated diskbomb content:

image

image

  • Diskspace error was removed confirmed by the WEB UI:

image

  • Followed similar steps for mitigating the mariadb service attack:

image

image

image

  • Restored correct MariaDB permissions:
sudo chown -R mysql:mysql /var/lib/mysql
sudo find /var/lib/mysql -type d -exec chmod 750 {} \;
sudo find /var/lib/mysql -type f -exec chmod 640 {} \;
  • Restarted MariaDB and verified access restored:
sudo systemctl restart mariadb.service

image

image


  • Correspondence with Affected Parties:

    • Notified team member of discovery and actions taken
    • Shared progress updates on Teams during incident

image

image

image

image


Sub-Tickets

  • Linked Sub-Ticket: apps-a server outage, assigned to separate team member

🔗 https://github.com/GriffinKat/group-a/wiki/Sub%E2%80%90Ticket-ID-%23345:-Investigate-Possible-Security-Breach-%E2%80%93-Service-Outage-on-DB%E2%80%90A-server


Time Taken

  • 3.5 hours (Investigation, resolution, communication)

Lessons Learned

  • Systemd can be used as a persistence mechanism for attacks or simulations

  • Script-based privilege actions must be clearly communicated or tracked

Tools / Action Items / Commands used

systemctl, chmod, chown, rm, bash_history, find, grep, stat, du

  • Suggested actions:

    • Audit who has permission to deploy systemd services

    • Implement logging or alerting on new .service file creation

    • Confirm whether this was an internal test and improve change tracking


Ticket Reference - https://rt.dataraster.com/Ticket/Display.html?id=340