Linux - ttulka/technologies GitHub Wiki

BASH Shell

  • Ctrl-a to the beginning of the command line.

  • Ctrl-e to the end of the command line.

  • Ctrl-d delete a character the cursor is currently upon.

  • Ctrl-f forwards one char at the time.

  • Ctrl-b backwards one char at the time.

  • Alt-f forwards one word at the time.

  • Alt-b backwards one word at the time.

  • Alt-u uppercases one word at the current position.

  • Alt-l lowercases one word at the current position.

  • Ctrl-k cuts the line forwards from the current position.

  • Ctrl-u cuts the line backwards from the current position.

  • Ctrl-l clears the screen (same as clear).

  • Ctrl-r shows the history (start typing to search).

  • Commands starting with comma don't go to history.

  • "$A" double quotes dereference variables, '$A' single quotes go literal.

  • !! last command (sudo !! runs last cmd as su).

  • !$ last command argument (mkdir mydir and cd !$)

  • ? env variable contains the return value of the last executed command.

  • echo $$ prints PID of the current shell

$ ls not-exists
ls: cannot access 'not-exists': No such file or directory
$ echo $?
2
$ echo $?
0
  • `cmd`, $(cmd) runs command:
echo `ls`
echo $($(echo ls))
  • nohup no hang up. Cmd continues execution even when terminal is closed (nohup cp ~/bigfile .).
  • ls file.{jpg,png} expands to ls file.jpg file.png.
  • ls f{1..3} expands to ls f1 f2 f3.

vim

ls -l | vim - edits console output in vim

  • i, a, R switches into insert mode (insert, append, replace).
  • : switches into last line mode (from command mode).
    • w writes changs into file.
    • q quits editor.
    • ! forces action.
      • q! forces to quit (changes will be lost).
    • e <file> edits file.
  • Esc switches into command mode.
    • [count]operation{motion}
      • 5dw or d5w deletes five words.
      • 2d3w deletes three words two times (deletes six words).
    • h, j, k, l moves cursor left, down, up, right.
    • w next word, W next spaced word (ignores punctuation), 3w next three words etc.
    • b back one word, B back spaced word (ignores punctuation), 3b back three words etc.
    • e end of current word, 3e end of third word etc.
    • 0 beginning of line, 3 beginning of third line etc.
    • $ end of line.
    • G end of file.
    • gg beginning of file.
    • Crtl f page down
    • Crtl b page up
    • x deletes char, X deletes char before cursor, 3x deletes three chars etc.
    • dd deletes line, 3dd deletes three lines etc.
    • dw deletes word, 3dw deletes three words etc.
    • d0 deletes from current to beginning of line.
    • d$, D deletes from current to end of line.
    • dG deletes from current to end of file.
    • dgg deletes from current to beginning of file.
    • :123 moves to line 123, :$ moves to end of file.
    • u undoes last command.
    • Ctrl-r redoes the last undo.
    • . repeats last command.
      • dw.. deletes word three times.
    • /<text> searches forwards, ?<text> backwards, /<text>\c case-insensitive.
      • n searches next.
      • N searches previous.
    • :s/old/new/ replaces fist occurence only on current line.
      • :s/old/new/g replaces all occurences on current line.
      • :1,100s/old/new/g replaces all in lines from 1 to 100.
      • :1,$s/old/new/g replaces all in lines from 1 to end.
      • :%s/old/new/gc replaces all in whole file with confirm.
      • :s#/bin#/usr/bin#g using different separator.
    • yy copies line.
    • yw copies word.
    • d cuts.
    • y copies selected.
    • p pastes in cursor position, 3p pastes three times etc.
    • P pastes bevor cursor, 3P pastes three times etc.
    • guu lowercases line.
    • gUw uppercases word.
    • J joins two lines, 3J joins three line etc.
    • z Enter calibries cursor to the top.
    • Ctrl g shows info about current position in file.

Visual mode

  • v characterwise visual mode.
    • higlights one char at a time (together with moves).
    • o oposites the direction.
  • V linewise visual mode.
    • hightlights whole lines.
  • Ctrl-v blockwise visual mode.
    • hightlights vertical blocks.
  • :'<,'> ranges to selection
    • :'<,'>s/old/new/g replaces in selection
    • :'<,'>center centers selection
    • :'<,'>right alings selection to right
    • :'<,'>left alings selection to left

Buffers

To work with multiple file:

  • vim file1.txt file2.txt opens two files to edit.
  • :e file2.txt open file to edit.
  • :ls shows opened.
  • :b2, :buffer 2 edits second.
  • :buffer file2.txt edits by filename.
  • :bn edits next.
  • :bf edits first.
  • :bf edits last.
  • :bd removes actual buffer, :bd1, :bd file.txt, :1,3bd, :%bd.

Registers

  • :reg shows nice vim registers.
  • "2p pastes from the second register.

Macros

  1. qa starts recording macro "a".
  2. type commands like 0llx to delete third char of line.
  3. again q stops recording.
  4. @a applies macro "a".
  • :reg a shows macro "a".

Tabs

  • :tabedit <file> edits file in a new tab.
  • :tabs shows tabs.
  • gt, gT moves tabs forwards, resp. backwards.
  • :tabclose closes tab.

Settings

  • :set shows current settings.
  • :set <setting> sets the settings.
    • :set nu shows line numbers.
  • :set <settins>! or :set no<setting> reverts the setting.
    • :set nu! hides line numbers.
  • :set <setting>? shows setting status.

Useful settings:

  • nu line numbers.
  • si inherites cursor start by new line.
  • ic ignore-case for all searches and replaces.
  • hls highlighted search results.
  • list show hidden characters (tabs, new lines)
  • expandtab spaces instead of tabs.
  • showcmd command completion hints.
  • wrap wraps text.
  • bg=light, bg=dark background theme.

Default settings in ~/.vimrc

set encoding=utf-8

" Show line numbers (set nu)
set number
" Show cursor position info
set ruler

" Speed up scrolling in Vim
set ttyfast

" Converting tabs to spaces
:set expandtab
:set tabstop=4
:set shiftwidth=4
:retab

" Highlight matching search patterns
set hlsearch
" Enable incremental search
set incsearch

" Fixes common backspace problems
set backspace=indent,eol,start

" Show command completion hints
set showcmd

" Inherites cursor start by new line.
set ai
" Smart starting by new line.
set si

" Unsaved buffers as hidden - enables to switch without saving
set hidden

" Show hidden characters (tabs, new lines)
" set list

Input & Output

In Linux all I/O are files. Files descriptors are numbers (0, 1, 2, etc.) - Index node (inode).

  • stdin: 0

  • stdout: 1

  • stderr: 2

  • ls -i file displays the file descriptor number for the file.

  • > redirects stdout to a file and overrides (same as 1>).

  • >> redirects stdout to a file and appends.

  • < redirects input from a file to a command.

$ echo Hello > hello.txt
$ echo Hello2 >> hello.txt
$ sort < ls -l
  • & used with redirection to signal what a file descriptor is used.
  • 2>&1 combinates strerr with stdout.
  • 2>file redirects stderr to a file.
$ ls here not-here 1> out.txt 2> err.txt
  • >/dev/null ignores output.
$ ls here not-here 2>/dev/null
here 
$ ls here not-here >/dev/null 2>&1 
# reverse sorts a file content, output into a file:
$ sort -r < file1.txt > file1-reverse-order.txt
  • tee copies stdin to file(s) and also to stdout.
# copies output of ls command to out1 and out2 files:
$ ls -la | tee out1 out2
  • xargs builds and execute command lines from stdin.
# does not work because echo accepts only arguments:
$ ls | echo
# builds arguments from output of ls:
$ ls | xargs echo

Write content from the command line:

cat > file.txt <<EOF
Some content
of the file
EOF

Files

  • man hier describes the filesystem hierarchy.

Finding Files

  • find [path..] expression recursively finds files in path that match expression.
$ find . -name MyFile
$ find . -iname myfile  # ignores case
$ find . -name My*

# files starting with My with size 10 MB or greater and performs ls upon them:
$ find / -name My* -size +10MB -ls

Removing Files

  • rm file removes a file.
  • rm -r dir removes a directory and its content recursively.
  • rm -f file forces removal.
  • find . -type f -name '*.txt~' -delete removes by expression
  • find . -type d -name '.del' | xargs rm -rf removes all directories .del

Copying and Moving Files

  • cp src dest copies a file.

  • cp -r srcdir destdir copies a directory recursively.

  • mv src dest moves or renames a file.

  • scp src dest copies a file securely over network.

Archiving Files

  • tar c|x|t f tarfile [pattern] tars a file.
# create a tar archive:
$ tar cf archive.tar ./archive
# extract the content from a tar file:
$ tar xf archive.tar 
# tar and compress a file:
$ tar -czvf archive.tar.gz ./archive --exclude=*.mp4
# extract a compressed file to a directory:
$ tar -xzvf archive.tar.gz -C /tmp

Searching a file

  • grep pattern file displays lines matching pattern.
    • -i ignores case.
    • -c counts the occurrences.
    • -v inverting match, lines that don't match.
grep "final class" -r src/ --include=*.java

Comparing Files

  • diff file1 file2 compares two files.
    • -w ignores white spaces, -i case-insensitive, --color colorized.
  • sdiff file1 file2 side-by-side comparison.
  • vimdiff file1 file2 highlights differences in vim.

Links to Files

  • ln orig hardlink creates a hard-link.
    • Basically a different name for the same file (same inode, same size and content).
    • Hard-link are not effected by deleting the original file.
    • Not allowed for directories.
  • ln -s orig softlink creates a soft-link (symbolic link).

File stats

Display file or file system status.

  • stat -c%s file.txt file size in bytes
  • stat -c%U file.txt file owner username

Processes

  • ps

    • -e, -A everything, all processes.
    • -f full.
    • -u username.
    • -p pid displays info about the process with PID.

  • pstree PID prints a process tree.

  • process & starts the process in background.

  • Ctrl-c kills the foreground process .

  • Ctrl-z suspends the foreground process.

  • bg [%num] backgrounds a suspended process.

  • fg [%num] foregrounds a background process.

  • jobs [%num] lists jobs.

  • kill PID kills a process PID.

    • -15, -TERM default: SIGTERM, termination.
    • -9, -SIGKILL KILLSIG.

Users and Groups

  • id [username] shows real and effective user and group IDs.
  • who shows who is logged.
  • whoami prints effective userid.
  • w shows who is logged and what is doing.
  • groups [username] shows groups the [current] user is in.
  • chown -R root:root . changes this directory and subdirectories to root:root.

Permissions

rwx bin oct
--- 000 0
--x 001 1
-w- 010 2
-wx 011 3
r-- 100 4
r-x 101 5
rw- 110 6
rwx 111 7

Networking

  • ip tool for routing, network devices, tunnels, and interfaces (successor of ifconfig)
    • ip a addresses
    • ip a show dev em1 details for em1
    • ip l interfaces
    • ip l show dev em1 details for em1
    • ip r routing table
    • ip n ARP neighbor objects
  • ss sockets utility (successor of netstat)
    • ss -lt displays all listening TCP sockets
    • ss -a dst 127.0.0.1 displays all sockets to specific IP
    • ss -a state established '( dport = :https or sport = :https )'
    • ss -a '( dport = :ssh or sport = :ssh )'
  • tcpdump dumps traffic on a network
    • tcpdump -nnSX port 443 only HTTPS
    • tcpdump -i eth0 by interface
    • tcpdump host 1.1.1.1 by host
    • tcpdump src 1.1.1.1 resp. tcpdump dst 1.0.0.1 by source resp. destination
    • tcpdump net 1.2.3.0/24 by network
    • tcpdump tcp by protocol
    • tcpdump -c 1 -X icmp hex output
  • dig tool to gather DNS information
    • dig google.com
    • dig google.com +trace
    • dig -x 172.217.14.238 reverse DNS lookup for IP
  • traceroute route packets trace to network host
  • hostname displays hostname
  • nmap port scanning (illegal on someone else's networks)
    • nmap -sp 192.168.1.1/24 scans subnet
    • nmap scanme.nmap.org scans a host
  • hey sends some load to a web application
  • tc traffic control, shows / manipulates traffic control settings (https://github.com/tum-lkn/tcgui)

Command Tips

  • cd to the home dir.
  • cd - to the previous location.
  • mkdir -p a/b/c create dirs for the whole path.
  • find . -name '*.log' -size +50k -exec du -h {} \; executes du -h for each found file.
  • which java prints the whole path to the java command.
  • history shows executed commands history.
  • !! executes the last commands one more.
  • ^A to the beginning of the command line.
  • ^E to the end of the command line.
  • ^L cleans the screen.
  • ^C ends the command.
  • ^D ends typing (exit).
  • diff file1 file2 --side-by-side --color compares two files.
  • touch file1 creates a new file, update the timestamp if exists.
  • watch ps executes ps every second.
  • less file1 trails the file
  • tail -n 1 file1 prints the last one line from the file.
  • head -n 1 file1 prints the first one line from the file.
  • echo test{1..3} prints test1, test2 and test3.
  • touch test{1,3} touches test1 and test3.
  • screen starts a new terminal to run commands in detached mode (^D).
  • top system performance.
  • pdiof vim prints PID for the process.
  • 2>$1 redirects the stderr to stdout.
  • strace -p 123 traces the process (prints input).
  • stress -cpu 2 -io 1 -vm 1 -vm-bytes 128M -timeout 10s -verbose stress CPU.
  • cal -3 shows three months, previous, current and upcoming.
  • df -h shows disk usages in human-readable form.
  • du -h \home shows disk usage for directories in human-readable form.
  • du -sch ./* size of current directory.
  • echo LinuX | tr '[:upper:]' '[:lower:]' prints "linux", translating chars from stdin.
  • lsof lists all open files.
  • lsof -t -u tomas lists processes for user.
  • lsof -i TCP:22 lists open connection on protocol and port.
  • sudo netstat -tulpn | grep LISTEN lists all open ports.
  • function try { until $@; do sleep 2; done; } used like try <command> tries again until succeeds.
  • xxd bin.file shows content of file in hexadecimal format

sed

Sed (stream editor) is as a line oriented filter.

  • sed s/day/night/ substitutes string once in each line.
    • sed 's/[^ ]*/(&)/g' substitutes every word putting it into parenthesis.
  • sed 's/[a-z]*/(&)/' substitutes found string into parenthesis.
    • echo "123 abc" | sed -r 's/[0-9]+/& &/' prints 123 123 abc
  • \1 is the first remembered pattern, and the \2 is the second, up to nine.
    • echo abc123xyz | sed 's/\([a-z]*\).*/\1/' prints abc
    • echo "abc xyz 123" | sed 's/\([a-z]*\) \([a-z]*\)/\2 \1/' prints xyz abc 123
  • -n no printing, turns on silent mode.
    • p flag will cause the modified line to be printed.
      • sed -n 's/pattern/&/p' finds and prints only lines with pattern.
      • sed -n '/pattern/I p' finds and prints only lines with pattern case-insensitive.
      • sed -n '/pattern/ !p' prints only lines which do NOT match pattern.
  • -e before each command allows to run multiple commands.
    • sed -e 's/a/A/' -e 's/b/B/'
  • -f sedscript runs sed commands from file.

Escaping slashes

If the pattern contains a slash - say substituing /usr/local/bin to /common/bin - you can escape it:

  • sed 's/\/usr\/local\/bin/\/common\/bin/'

Or you can pick up any other delimiter (defined after s, there are always three):

  • sed 's_/usr/local/bin_/common/bin_'
  • sed 's:/usr/local/bin:/common/bin:'
  • sed 's|/usr/local/bin|/common/bin|'

Restrictions

  • sed '3 s/A/a/' substitutes thrid line only.
  • sed '/^#/ s/A/a/' substitutes lines starting with # only.
  • sed '1,100 s/A/a/' substitutes from first line to hundredth only.
  • sed '100,$ s/A/a/' substitutes from hundredth line to last only.

awk

Awk is a programming language for working on files.

  • awk '{print $3 "\t" $4}' input.txt prints third and fourth column separated with tab.
  • awk 'length($0) > 50' input.txt prints lines with more than 50 chars.
  • awk -F: '{print $1}' /etc/passwd prints first column by : separator.
  • echo "Hello Tom" | awk '{$2="Adam"; print $0}' prints Hello Adam.

BASH Scripting

#!/bin/bash
# this is a comment

echo "Total Arguments Count:" $#
echo "All Arguments values:" $@
echo "First->"  $1
echo "Second->" $2

rank=captain

if [ "$rank" = colonel ]
then
  echo Hannibal Smith
elif [ "$rank" = captain ]
then
  echo Howling Mad Murdock
elif [ "$rank" = lieutenant ]
then
  echo Templeton Peck
else
  echo B.A. Baracus
fi

case $rank in
    colonel) echo Hannibal Smith;;
    captain) echo Howling Mad Murdock;;
    lieutenant) echo Templeton Peck;;
    sergeant) echo B.A. Baracus;;
    *) echo OOPS;;
esac

counter=0

while [ $counter -lt 10 ]
do
  echo $counter
  counter=`expr $counter + 1`
done

until [ $lines -eq 10000 ]
do
    lines=`wc -l dates | awk '{print $1}'`
    sleep 5
done

for myval in Abel Bertha Charlie Delta Easy Fox Gumbo Henry India
do
  echo $myval Company
done
$ chmod +x script.sh
$ ./script.sh

Works on the command line as well:

# echos files from ls output:
$ for myfile in $(ls); do echo $myfile; done
# echos numbers from 1 to 5:
$ for i in {1..5}; do echo $i; done

Extras

  • echo "Hello world" | cowsay prints a cow saying the text.

Traffic control (tc) - package iproute2

  • tc qdisc add dev eth0 root netem delay 100ms 20ms - delay
  • tc qdisc add dev eth0 root netem loss 0.1% - packet loss
  • tc qdisc add dev eth0 root netem duplicate 1% - packet duplication
  • tc qdisc add dev eth0 root netem corrupt 0.1% - packet corruption

Spoof your MAC address

ifconfig eth0 down
ifconfig eth0 hw ether 00:11:22:33:44:55
ifconfig eth0 up

Assign new IP from DHCP DHCP server assigns IP addresses to all the systems on the subnet and keeps log files of which IP address is allocated to which machine at any one time. This makes it a great resource for forensic analysts to trace hackers with after an attack.

dhclient eth0

Kernel

Kernel starts with /sbin/init as PID 1, subsequently reads /etc/inittab.

  • lscpu show CPU architecture

Kernel Files

  • /boot/vmlinux - old, no compression,
  • /boot/vmlinuz - compressed with zlib.
  • uname -r prints the running version of Kernel.
  • /boot/initrd.img or /boot/initramfs - initial RAM disk file.

Services

Service scripts are located in /etc/init.d.

  • service --status-all shows services.
  • systemctl system service units.

References

⚠️ **GitHub.com Fallback** ⚠️