Ticket ID #340 Investigate Possible Security Breach – Service Outage on Multiple Servers - GriffinKat/group-a GitHub Wiki
Unexpected Downtime on Critical Services — Suspected Security Attack
Trigger/Root Cause(s)
During investigation, two malicious scripts were found on the database server (db-a). These were deployed as custom systemd
services:
-
/etc/systemd/system/simdiskattack.service
-
/etc/systemd/system/simmariadbattack.service
The associated scripts were located in:
-
/usr/local/bin/tmp/sim_disk_attack.sh
-
/usr/local/bin/tmp/sim_mariadb_attack.sh
Contents of the scripts:
-
sim_disk_attack.sh
: This script continuously created random 10MB binary files in /var/tmp/diskbomb/ using dd and /dev/urandom, rapidly consuming disk space. -
sim_mariadb_attack.sh
: This script repeatedly checked if MariaDB was active, then set the directory permissions of /var/lib/mysql to 000, effectively breaking the database. It attempted to restart MariaDB afterward, entering a failure loop.
Impact
-
Affected System:
db-a
-
Impacted Services: MariaDB, disk space availability
-
Effect: MariaDB was inaccessible due to permission change on /var/lib/mysql, and disk space was rapidly filled by the disk attack script, leading to potential service instability and monitoring failures.
Detection
- Notification was received via the team's monitoring Slack channel, indicating critical services were down.
- Service failure was then confirmed through the Nagios Web UI.
- Investigated disk space usage:
sudo du -h --max-depth=1 /var
- Analyzed timestamp and metadata of
/var/tmp/diskbomb
directory:
stat /var/tmp/diskbomb/
- Verified diskbomb-related commands using:
sudo grep -r "/var/tmp/diskbomb" /etc /root /home /usr/local /opt 2>/dev/null
- Confirmed .service configuration pointing to the malicious script:
sudo grep -r "/usr/local/bin/tmp/sim_disk_attack.sh" /etc/systemd/system /lib/systemd/system /usr/lib/systemd/system 2>/dev/null
-
While detecting and tracing the disk attack, the
sim_mariadb_attack.sh
and corresponding systemd service were also discovered in the same/usr/local/bin/tmp
directory, confirming parallel impact on MariaDB functionality. -
Reviewed the scripts:
Resolution Steps
- Investigated and confirmed the malicious
systemd
services:
sudo systemctl stop simdiskattack.service
sudo systemctl status simdiskattack.service
- Disabled and removed the services:
- Reloaded the systemd daemon:
sudo systemctl daemon-reload
- Removed the script and generated diskbomb content:
- Diskspace error was removed confirmed by the WEB UI:
- Followed similar steps for mitigating the mariadb service attack:
- Restored correct MariaDB permissions:
sudo chown -R mysql:mysql /var/lib/mysql
sudo find /var/lib/mysql -type d -exec chmod 750 {} \;
sudo find /var/lib/mysql -type f -exec chmod 640 {} \;
- Restarted MariaDB and verified access restored:
sudo systemctl restart mariadb.service
-
Correspondence with Affected Parties:
- Notified team member of discovery and actions taken
- Shared progress updates on Teams during incident
Sub-Tickets
- Linked Sub-Ticket:
apps-a
server outage, assigned to separate team member
Time Taken
- 3.5 hours (Investigation, resolution, communication)
Lessons Learned
-
Systemd can be used as a persistence mechanism for attacks or simulations
-
Script-based privilege actions must be clearly communicated or tracked
Tools / Action Items / Commands used
systemctl
, chmod
, chown
, rm
, bash_history
, find
, grep
, stat
, du
-
Suggested actions:
-
Audit who has permission to deploy systemd services
-
Implement logging or alerting on new .service file creation
-
Confirm whether this was an internal test and improve change tracking
-