shellsheet - feliyur/exercises GitHub Wiki

Shellsheet

Location of config/permissions on Ubuntu /etc/sudoers, /etc/groups
Show storage devices and UUID lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,UUID
Execute monitoring command every n seconds watch -n <sec> <command> <command args>
Compute directory size, show filesystem info. du, df, ncdu
Get / change hostname hostname to get, sudo hostnamectl set-hostname to set (or sudo edit /etc/hostname)
Get ip address ifconfig -a old style. New: ip address or ip a.
Who is logged on who -H, who -u, w, users, ps au the latter shows processes for each user.
Processes info ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head
Copy symlink cp -R --preserve=links
Clean up system logs sudo journalctl --vacuum-size=100M
Find and delete empty directories find /path/to/directory -type d -empty -delete
Find and filter files find /path/to/directory -type [f,d] -name 'supports wildcards' [-maxdepth <N> -mindepth <M>]
More complex find usage example with exec find /path/to/directory -type d -exec chmod 770 {} \; -type f -exec chmod 660 {} \;. Mind the \; which ends the command, and {} which is placeholder of current item.
List contents recursively tree [-L <max depth>]
Set a timeout for a blocking command timeout <n_seconds> <command and arguments> e.g. timeout 100 ssh <host> will close the ssh connection after 100 seconds.
Format a partition to ext4 sudo mkfs.ext4 /dev/nvme0n1
Set execute permissions only to subdirectories, but not to files chmod -R rwX <target dir>
get and control file access lists getfacl / setfacl
List groups user is part of id -a, groups
List path and permissions to target namei -l <target>

Navigate Shell History

To search back in shell command history use Ctrl+r (this is called "reverse-i-search"). Can use Ctrl+s to search forward, but possibly need to remap an overshadowing shortcut for this key combination ("stop terminal sequence"), adding stty stop ^J to .bashrc (see here).

!<command prefix> executes last command starting with prefix. history shows command history.

Edit fstab

Safe fstab line: UUID=<device UUID> <mountpoint> none <options> 0 0 Mountpoint must exist. Can check UUID with lsblk

Worth to almost always use nofail to allow the system to boot even if the drive cannot be mounted. E.g. replace <options> with defaults,nofail.

For network-facing systems, worth to use safe options nosuid,nodev.

Mount ntfs filesystem:

UUID=F4E2C7FAE2C7BF5C /media/Windows ntfs defaults,nofail,nls=utf8,umask=000,dmask=027,fmask=137,uid=1000,gid=1000,windows_names 0 0

Can use lsblk -o NAME,MOUNTPOINT,SIZE,FSTYPE,UUID command to list drives and found out UUID of an unmounted drive.

User Management

Create new user sudo adduser [username]
Add user to group sudo usermod -a -G groupName userName

To hide a user from login screen see here:

sudo vim /var/lib/AccountsService/users/[username]

Add / edit the following two lines:

[User]
SystemAccount=true

SSH

ssh <server> -C '<command>' Run command remotely and disconnect.
ssh <server> -L<local-port>:localhost:<remote-port> maps local port to remote
ssh <server> -R<local-port>:localhost:<remote-port> maps remote port to local (reverse tunnel)
ssh <params> -N Do not execute a remote command (useful for port forwarding).

Running ssh in the background (e.g. port-forwarding).

Installing: sudo apt-get install openssh-server

SSH Config

Generate ssh key ssh-keygen -t ed25519 -C "[email protected]"
Or ssh-keygen -t rsa -b 4096 -C "[email protected]"
Add local ssh key to remote authorized_keys ssh-copy-id <username@remote>. On Windows: type <key .pub file> | ssh <user@remote> "cat >> .ssh/authorized_keys"

~/.ssh/config file format:

Host <my_alias>
HostName <ip>
User <remote username>
To connect through proxy, do
ProxyCommand ssh yurif@csm nc %h %p
or
ProxyJump pi

To execute a remote command on connect (e.g., `cd` to some directory)
RequestTTY yes
RemoteCommand cd /home/katya/yuri; exec /bin/bash

To avoid / reduce connection drops (especially sftp in Nautilus), on the client side, add:
ServerAliveInterval 180
ServerAliveCountMax 2
To apply to all connections add this under Host *, like this:
Host *
    ServerAliveInterval 180
    ServerAliveCountMax 2

.ssh/config file permissions need to be 644.

Deal with connection drops

ssh connection parameters can be configured in /etc/ssh/sshd_config (globally) and in the $HOME/.ssh/config (per host). In particular, look at {Server,Client}Alive intervals which control how a connection is maintained alive, see above.

Change hostname

Taken from here.

sudo vim /etc/hostname
sudo vim /etc/hosts
sudo reboot

Scan network

The nmap utility scans for open ports, e.g. sudo nmap -p 22 192.168.1.0/24 scans for machines listening on port 22 (ssh).

Application shortcuts

.desktop files located at /usr/share/applications. Can link also to remote folders through sftp.

To create a desktop shortuct, need to create a .desktop file at ~/Desktop.

Remote File Transfer

rsync --partial --progress myfile username@server:/path/to/destination

Transfer file. --partial flag allows to resume an interrupted transfer. --remove-source-files and --delete deletes source after transfer. After --remove-source-files directory tree is left.

Flag Effect
--partial Allows to resume an interrupted transfer
-rP Shorthand for --recursive --partial --progress
-L Copy symlinks
--remove-source-files Remove source can be completed using find <dir> -type d -empty -delete
--delete
scp

TODO: Description

Managing Processes

Ctrl + z Puts current process in the background
jobs Lists current session jobs.
fg <id> Brings process <id> (from jobs output) to foreground.
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head Processes info

Useful Programs

fail2ban Service to block repeated accesses to computer/server.
imagemagick Set of cli tools for manipulation of images. In particular, convert converts formats and transforms, display opens image in a display window.
diffuse Graphical comparison allowing manual alignment
pdftk Set of cli tools for dealing with pdfs
xclip Allows to redirect programs output to clipboard
jq, yq command-line json / yaml parsing respectively
curl, wget download files. Syntax: wget url -O filename, curl url -o filename.
Curl writes to stdout by default.
tmux, screen Terminal multiplexers. Allow to run programs persistently without being connected. tmux cheatsheet, screen cheatsheet
htop, glances Display system resources / status. glances can be installed as a python package. Glances reference.
finger command-line utility showing users currently connected to the host
iotop Display I/O system resources (glances does this as well). iotop -oPa
timeshift Does automatic system backups. Can be configured via shell through /etc/timeshift.json.
eog "Eye of GNOME", displays images.
shuf Shuffles lines of input. Alternatively, can use perl: cat file1 file2 | perl -MList::Util=shuffle -wne 'print shuffle <>;', taken from here.
lazygit Command-line git gui. See git cheatsheet for keybindings.
nl Adds numbering to input lines

Archive Files

unzip file.zip -d destination_folder
unzip -t test.zip test if an archive is valid
unzip -l file.zip list contents
unzip file.zip -x a_particular_file exclude file
zip -r file.zip list of files and directories Compresses recursively.
tar -xvf Extracts (verbose) -C or --directory destination dir
tar -czvf file.tar.gz file-or-dir [-T list-of-files.txt] Compresses (-c) using gzip to file (-f). Recurses into directories by default (prevent with --no-recursion). -h Follows symlinks.
tar chf - RADIal/ | pv -p -s ${SIZE}k | gzip > $FS_AID_HOME/RADIal.tar.gz Compress with progress bar
7z x file Extracts. Need to sudo apt-get install p7zip-full to get 7z.

ncdu keyboard shortcuts

c, C Show file count, sort by file count
s Sort by size

PDF Files

pdftk myfile.pdf cat 1-3 5 7-end output outfile.pdf Output portions of pdf

Convert PDF into (true) black-and-white (good for sending as fax):

gs -sDEVICE=bmpgray -dNOPAUSE -dBATCH -r300x300 -sOutputFile=./Payement-confirmation.pdf ./Payement-confirmation.pdf

gs -o ./Payement-confirmation-bw.pdf -sDEVICE=pdfwrite -c "/osetrgbcolor {/setrgbcolor} bind def /setrgbcolor {pop [0 0 0] osetrgbcolor} def" -f ./Payement-confirmation-b\&w.pdf

Convert PDF to png:

convert -density 700 input.pdf -quality 90 output.png

Manually install gpg keys

curl -sSL
'http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xBBEBDCB318AD50EC6865090613B00F1FD2C19886'
| sudo apt-key add -

Parameter expansion

Documentation here.

${parameter#pattern} Deletes shortest (## for longest) match of pattern from beginning of parameter
${parameter%pattern} Deletes shortest (%% for longest) match of pattern from end of parameter
${parameter/pattern/string} Replaces the first match of pattern with string. // for all matches, # and % for matches at beginning and end respectively
${!variable} Refer to the variable who's name is stored in variable (indirection)

crontab

Crons are stored at /var/spool/cron/crontabs.

[sudo] crontab -l lists crons. sudo lists root's crons.
[sudo] crontab -e edit. sudo edits root's crons.
@reboot command Run command on startup. Probably should add as root.
01 04 1 1 1 <command path> The five time-and-date fields are as follows:
minute (0-59), hour (0-23, 0 = midnight), day (1-31),
month (1-12), weekday (0-6, 0 = Sunday).
Further line format options and docs.
Further documentation

To schedule a command to run once can use the at command (might need to apt-get install it). For example

echo 'mycommand' | at 16:00   # at now + 3 hours

or

at 16:00 -f some_script.sh

References 1, 2.

Special keys with AltGr Int'l keyboard

Problem: AltGr special keys not function Ubuntu 18.04. Solution: Keyboard shortcuts ==> Restore keyboard shortcuts (Super+Escape default).

prefix = right alt

Key Shortcut
é prefix+', e
è prefix+`, e
á, à idem.
ç, Ç prefix+, (hold prefix) c / shift+c
ï " + i
ĉ ^+c

Also see here

Bash command completion

Example

conda-workon() {
        module load conda; conda activate "$1"
}

_conda-workon_completions()
{
        if [ "${#COMP_WORDS[@]}" != "2" ]; then
        return
        fi

        envs_options="`ls $HOME/.conda/envs`"
        COMPREPLY=($(compgen -W "${envs_options}" "${COMP_WORDS[1]}"))
}

complete -F '_conda-workon_completions' 'conda-workon'

To enable for all users, add the completion script into /etc/bash_completion.d/

See here

Libraries installations

sudo apt install libopencv-dev 
sudo apt install libpcl-dev

Midnight Commander

sudo apt-get install mc

Can choose Lynx-like motion in Options (F9) -> Panel options,

mc -x start with mouse
Ctrl+o Back to shell
Tab Switch pane
Shift+F6 Renane file in place
%d/%f, %D/%F Refers to left dir/file and right dir/file respectively
Alt+I cd the other pane to the same directory as the current pane.
Alt+A Copy the cwd of the current pane to the command line.
Ctrl+R Refresh file list in pane.
Esc+Tab Bash auto-completion.
F9 ==> Right ==> sftp link ssh connection. Can use ~/ssh/config hostnames
F9 ==> Options ==> Panel ==> Lynx-like motion Left/right arrow keys exit / enter directories. To naje default set navigate_with_arrows=true in $HOME/.config/mc/ini.

More information about macros etc. here

And here

File Access Control Lists (ACLs)

A separate access permissions mechanism for file systems via access lists, complementary to permissions flags.

Docs here: https://linux.die.net/man/1/setfacl

chmod g+rwxs
setfacl -m d:g::rwx

GPUs

nvidia-smi List gpu usage and processes. Good to combine with watch -n for continuous status.
nvidia-smi -L List gpu devices. Other useful flags.
nvitop / nvtop Top command for nvidia gpus. Can install using python pip
sudo fuser -v /dev/nvidia* List all processes taking up the gpu. Without sudo will list just processes for the current user.

Ubuntu tweaks

LTS 22.04 show trash on desktop instead of dock

gsettings set org.gnome.shell.extensions.ding show-trash true
gsettings set org.gnome.shell.extensions.dash-to-dock show-trash false
⚠️ **GitHub.com Fallback** ⚠️