Server Maintenance - ionathanch/ionathanch.github.io GitHub Wiki

General

Task	Command
Enable available site (NginX)	`sudo ln -s /etc/nginx/sites-available/<CONFIG> /etc/nginx/sites-enabled/<CONFIG>`
Reload/restart NginX	`sudo systemctl reload/restart nginx`
List SSL certificates	`sudo certbot certificates`
Append domain to SSL certificate	`sudo certbot --nginx --expand -d <OLD DOMAINS>,<NEW DOMAINS>`
Renew SSL certificate	`sudo certbot renew`
Copy file via SSH	`scp <FILE PATH> <USER>@<REMOTE>:<DEST PATH>`
Copy files, preserving all	`rsync -avP`
Shrink journal logs	`sudo journalctl --vacuum-size=1G`

See more Certbot commands and NginX templates.

Static IP address

Find the logical name of the network device:

$ sudo lshw -class network -short
H/W path   Device   Class    Description
===============================================================
/0/100/19  enp0s25  network  82578DC Gigabit Network Connection

Enable network:

$ sudo ip link set up enp0s25  # bring up device
$ sudo dhclient enp0s25        # enable DHCP

Create configuration /etc/netplan/config.yaml with the following:

network:
  version: 2
  ethernets:
    enp0s25:
      dhcp4: false
      addresses: [192.168.0.69/24]  # static IP with subnet prefix length /24
      gateway4: 192.168.0.1         # IP address of router
      nameservers:
        addresses: [84.200.69.80, 84.200.70.40]  # DNS nameservers

Apply configuration with sudo netplan apply.

UFW (Uncomplicated Firewall)

$ sudo ufw app list
Available applications:
  OpenSSH
  ...
$ sudo ufw allow "OpenSSH"   # add listed application
$ sudo ufw allow <PORT>/tcp  # add TCP <PORT>
$ sudo ufw allow <PORT>/udp  # add UDP <PORT>
$ sudo ufw reload            # enable new rules
$ sudo ufw status
Status: active
To     Action    From
---    ------    ----
...    ...       ...

SSH from a new machine

[ Local ] Generate SSH key (default RSA 3072 bits): ssh-keygen
[ Remote ] Temporarily enable password login in /etc/ssh/sshd_config: PasswordAuthentication yes
[ Remote ] Restart SSH server: sudo systemctl restart ssh
[ Local ] Copy public SSH key: ssh-copy-id <USER>@<REMOTE>
[ Remote ] Disable password login: PasswordAuthentication no

Resizing a filesystem

$ lsblk                     # find mounted file system
NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
...
sdb    XX:XX  0   XXG  0 disk /mnt/YYY
$ umount /dev/sdb           # unmount file system
$ e2fsck -f /dev/sdb        # check file system
e2fsck 1.45.5 (07-Jan-2020)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdb: XX/XX files (XX% non-contiguous), XX/XX blocks
$ resize2fs /dev/sdb        # resize file system
resize2fs 1.45.5 (07-Jan-2020)
Resizing the filesystem on /dev/sdb to XX (4k) blocks.
The filesystem on /dev/sdb is now XX (4k) blocks long.
$ lsblk /dev/sdb            # verify new size of file system
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb   XX:XX   0  XXG  0 disk 
$ mount /dev/sdb /mnt/YY    # remount file system

Hetzner storage box access

SSH, SCP, rsync, SFTP

To access via SSH key, copy public key to <username>@<username>.your-storagebox.de:.ssh/authorized_keys.

For convenience, set the username and port for this host in ~/.ssh/config:

HOST <username>.your-storagebox.de
  USER <username>
  PORT 23

The following commands omit the port and username. Note that directory names with spaces need to be quoted and have the spaces escaped.

# Using SCP to copy files (recursive, port 23)
$ scp -r <file> <username>.your-storagebox.de:<destination>
# Using rsync over SSH to copy files (archive, verbose, human-readable, progress & partial)
# Useful flags: --dry-run, --exclude, --delete, --stats
$ rsync -avhP --timeout=5 <directory> <username>.your-storagebox.de:<destination>
# Logging into SFTP over SSH
$ sftp <username>.your-storagebox.de

See Access with SSH/rsync/BorgBackup for more details.

rclone and SFTP

To access via SSH key, copy public key to .ssh/authorized_keys. Configuration for rclone in .config/rclone/rclone.conf:

[<name>]
type = sftp
host = <username>.your-storagebox.de
user = <username>
port = 23
key_file = ~/.ssh/id_ed25519

Mount using rclone mount <name> <mountpoint>. See SFTP for more details.

CIFS

To mount via CIFS, install cifs-utils first.

$ id -u <local username> && id -g <local username> # get UID and GID
$ sudo mkdir /mnt/<mountpoint>
$ sudo mount.cifs -o seal,username=<username>,password=<password>,uid=<UID>,gid=<GID> //<username>.your-storagebox.de/backup /mnt/<mountpoint>

Alternatively, add below to /etc/fstab and mount with sudo mount /mnt/<mountpoint>.

//<username>.your-storagebox.de/backup /mnt/<mountpoint> cifs iocharset=utf8,rw,seal,username=<username>,password=<password>,uid=<UID>,gid=<GID>,file_mode=0660,dir_mode=0770 0 0

See Access with SAMBA/CIFS for more details.

Docker

Task	Command
Login to Docker container shell	`docker exec -it <CONTAINER NAME> sh`
Copy files to Docker container	`docker cp <HOST PATH> <CONTAINER NAME>:<CONTAINER PATH>`
Force recreate container	`docker-compose up --force-recreate -d`
Remove unused image	`docker rmi <IMAGE>`

See Docker CLI docs for more commands and this Docker Compose template.

Installation

$ sudo apt install docker.io docker-compose
$ sudo gpasswd -a <USER> docker  # add user to the Docker group to use without sudo
$ newgrp docker                  # log in to Docker group

Updating images

For Gitea, remember to update configs, templates, and locales if needed. TTRSS does not need manual updating.

$ docker pull <IMAGE>
$ docker-compose up -d
$ docker image prune

Borg

Initialization

$ sudo borg init --encryption keyfile <REPO PATH>  # keyfile stored in `/root/.config/borg`

Backups

Create the following cron script at /etc/cron.hourly/borg-backup (sudo chmod +x <SCRIPT> to make it executable):

#!/bin/sh
borg create --exclude-from <EXCLUDES PATH> <REPO PATH>::$(date +'%FT%T') <SRC PATH>

Alternatively, add the following to crontab (note the escaped %!):

@hourly borg create --exclude-from <EXCLUDES PATH> <REPO PATH>::$(date +'\%FT\%T') <SRC PATH>

Add -svp for stats, verbose output, and incremental progress. Use sudo borg list <REPO PATH> to check backups. See here for exclusion patterns and here for backup name substitutions.

Pruning

Create the following cron script at /etc/cron.daily/borg-prune (again, sudo chmod +x <SCRIPT>):

#!/bin/sh
# keeps last 24 hourly backups, last 7 daily backups, last 2 weekly backups
borg prune -H 24 -d 7 -w 2 <REPO PATH>

Alternatively, add the following to crontab:

@daily borg prune -H 24 -d 7 -w 2 <REPO PATH>

Restoring

Use borg mount to access files and rsync -avP to copy them.

Nextcloud

Task	Action
HTTPS for reverse proxy	Add `'overwriteprotocol' \=> 'https'` in `config/config.php`
Run OCC command	`docker exec -u www-data nextcloud php occ <COMMAND>`
Add background cron jobs	`sudo crontab -u root -e`, add `/5 * * * docker exec -u www-data nextcloud php cron.php`

Manually removing file locks, bruteforce attempts, and file caches

$ sudo su
$ sqlite3 data/owncloud.db
sqlite> DELETE FROM oc_file_locks WHERE lock=-1;
sqlite> DELETE FROM oc_bruteforce_attempts WHERE ip ="<IP ADDRESS>";
sqlite> SELECT fileid FROM oc_filecache WHERE storage=2 AND path LIKE '%<FILENAME>%';
sqlite> DELETE FROM oc_filecache WHERE fileid=<FILEID>;
sqlite> .quit
$ exit

Gitea

See docs on customizations and configurations for details.

Modification	Location
Configurations	`$GITEA/config/app.ini` (see example)
Templates	`$GITEA/templates/*/.tmpl` (see templates)
Localizations	`$GITEA/options/locale/local_*.ini` (see locales)
CSS styling	`$GITEA/public/css/*.css` (see LESS for source)

TTRSS

Data	Location
Configs	`/var/www/ttrss/config.php` (in Docker container)
Feeds	`https://rss.ionathan.ch/opml.php?op=publish&key=<SECRET KEY>`
Prefs	`Preferences → Feeds → Export OPML` (checking `Include tt-rss settings`)

Mastodon

The commands below are executed as sudo su mastodon in its home directory ~/live. Use RAILS_ENV=production bundle exec rails --tasks to show commands.

Updating

Ruby may need to be upgraded via rbenv
Node may need to be updated in /etc/apt/sources.list.d/nodesource.list and upgraded via apt

$ git pull && git merge master origin/main
$ bundle install && yarn install --immutable
$ RAILS_ENV=production bundle exec rails assets:precompile
$ SKIP_POST_DEPLOYMENT_MIGRATIONS=true RAILS_ENV=production bundle exec rails db:migrate
$ systemctl restart mastodon-web mastodon-sidekiq mastodon-streaming
$ RAILS_ENV=production bundle exec rails db:migrate
$ RAILS_ENV=production bin/tootctl cache clear # optional

Modifying locales and favicons

Locales are located in config/locales and app/javascript/mastodon/locales; favicons are located in public/favicon.ico and app/javascript/icons/favicon-{16x16,32x32,64x64}.png. I think running the below will update app/javascript/ for you.

$ RAILS_ENV=production bundle exec rails assets:clobber
$ RAILS_ENV=production bundle exec rails assets:precompile
$ systemctl reload mastodon-web