LinuxFoundation - dejanu/linux GitHub Wiki

The Boot Process

Power On -> BIOS SW(stored on a ROM chip on the motherboard) initializes the HW -> BootLoader(stored on the first sector of the Hard Disk aka MBR or in the EFI/UEFI partition) which loads the kernel image and initial RAM disk into memory.

The boot loader loads the Kernel and an initial RAM-based filesystem (initramfs)
the kernel run /sbin/init and becomes the initial process

Linux Filesystem Tree Layout

one large logical filesystem which can contain one or many distinct filesystems mounted at various points
Filesystem Hierarchy Standard : shareable (can shared between hosts)

	shareable	unshareable
static	/usr /opt	/etc /boot
-----------	------------------	--------------------
variable	/var/mail	/var/run /var/lock

Main directories:
- /bin : contains binary executable files (cat, kill, ps) . Command binaries which are deemed non essential enough are in /usr/bin.
- /lib: contains libraries needed to execute the binaries in /bin or /sbin . Also kernel module are in /lib/modules/
- /boot: essential files for booting the system (vmlinuz compressed linux kernel and initrd initial RAM filesystem which is mounted before the real root filesystem becomes available, config to configure kernel compilation)
- /dev: device nodes aka device files (byte-stream or block I/O devices). Network devices (eth1 , eth2) do not have device nodes
- /usr: can be thought of as a secondary hierarchy, need not reside in the same partition as the root directory
- /etc: contains machine-local configuration files/scripts (/etc/systemd contains config scripts for starting, stopping system services using systemd ; also /etc/init.d which contains scripts for System V initialisation)
- /var: logs in /var/log and cron jobs, mail file in /var/spool, lock files in /var/lock or cat /var/log/yum.log
- /proc: is the mount point for a pseudo-filesystem, where all information resides only in memory, not on disk. The entires in /proc are called virtual files with zero bytes in size.

# List files opened by a process
lsof -p PID 
# equivalent more or less with 
ls -l /proc/PID/fd

Processes

Process = executing program and associated resources (open files, signal handlers) and has various states: running, sleeping
Every process has a PID , PPID(parent PID), pgid(process group ID)
init usually is the first process run on a system, and is the ancestor of all subsequent processes running on the system (except for thos with [] arounf their name , they are initiated by the kernel)
orphaned processes (parent process dies before child) are adopted by init (thus the ppgid is set to 1) or (in distributions which use systemd) the ppgid is set to 2 (adopted by kernel thread known as kthreadd)
zombie process (defunct) is a process which terminates before his parent and released almost all the resources and remained only to convey (communicate) his exit status
processes are controlled by scheduling
process context = snapshot of process by trapping the state of the CPU registers
process permissions: programs marked with s (execute bit on) aka setuid programs run with the user-id of the user who owns the program versus program non-setuid which run with the permissions of the user who starts the program. (setuid program owned by root can be a security issue)
when process is launched: it runs with the effective user-id and group-id of the user who started it, and with the corresponding privileges. This behaviour can be modified by using special permissions

#setuid programs (e.g.: owned by root)
/etc/passwd
/etc/shadow

# setuid bit is represented by an s in place of x
ls -l /bin/passwd
-rwsr-xr-x. 1 root root 27768 Feb 11  2017 /bin/passwd

The setuid bit has no effect on directories. setuid or setgid bits are set, but the executable bit is not
When a process is started it has its own isolated space and it uses system calls to indirectly access the HW (the HW is managed by the kernel)

ULIMIT

there are 2 kinds of limits: HARD (set by the user) and SOFT (cannot exceed the hard limit).
usage: ulimit [options] [limit]

# ulimit command that reports/sets/resets a number of resource limits (file-size writing limits) associated with processes running under a shell

# list current limits
ulimit -a

# check HARD limit
ulimit -H -n

# check SOFT limit
ulimit -S -n

# no of leak-ed file descriptors supported by OS
ulimit -n 

# increase no of file descriptors (soft resources)
ulimit -n 1600

NICE

nice -n 5 command [args] = niceness value can range from -20 (the highest priority) to +19 (the lowest priority) aka set priorities
nice execute a process with modified priority and renice change the priority of an already running process

# start bash with nice value of 10
nice -n 10 bash

# change the value of bash session to 15
renice -n 15 -p (ppid)

# increase niceness by 5 units
renice +5 <PID>

process states : running -> sleeping(waiting) -> stopped(suspended Ctrl+Z sends SIGSTOP)-> zombie

# list all process on the system PID PPID PRI(priority) NI(nice value)
ps -elf

Signals (Inter_Process Communication)

Are used to emit notifications for processes to take action in response to unpredictable events
Processes cannot send signals to another process directly, this is done by the kernel via sys-calls.
kill -l list signals , basic syntax kill <SIGNAL> <PID> (default values is 15 SIGTERM)
pkill -u libby foobar kills process foobar of libby user
pkill -HUP rsyslogd (HUP has different behaviour for daemons , make rsyslog do re-read config file0
killall -9 bash kill all process with a given name

Process monitoring

process monitoring tools: top-process activity dynamically updated , ps-detailed information about processes , uptime-how long the system is running and average load, mpstat-multiprocessor usage, iostat-CPU utilisation and I/O statistics, sar-display and collect info about system activity, strace-information about sys calls a process makes
ps is the workhorse for displaying characteristics and it reads information from /proc
ps possibilities
1. UNIX-style: options which must be preceded by -
2. BSD-style: options, which must not be preceded by -
3. GNU-style long option, each of which must be preceded by --

###  BSD-style: display all processes owned by you and Lift the BSD-style "must have a tty" restriction
## USER  PID %CPU %MEM    VSZ   RSS TTY  STAT START   TIME COMMAND
ps aux 
ps auxf # will show the ancestry tree

# specified user-defined format for the ps output: [parent PID, PID, percentage memory usage
ps -o ppid,pid,etime,command,pmem,pcpu
ps e -o ppid,pid,etime,command,pmem,pcpu

### UNIX-style: display all processes (background ones to) using BSD long format with full format listing
## UID  PID  PPID  F %CPU PRI NI   SZ    RSS WCHAN     S     ADDR TTY    TIME CMD     STIME
ps -elf


# add columns besides the default values (PPID is shown in the default ps -elf)
ps -ef -O etime,pmem,command


# visual description of the process ancestry
pstree -aAp <PID>

top = interactive use 1 (each CPPU shown separately) , i (only interactive processes are shown), l (show load)

Package Management Systems

rpm = Redhat Package Manager (RHEL, Centos, Fedora) + SUSE (OpenSuse)
dpkg = Debian Package Manager (Debian, Ubuntu, Mint)
A give package may contain (executable files, data files, installation scripts, config files) https://www.digitalocean.com/community/tutorials/package-management-basics-apt-yum-dnf-pkg.
package types: Binary (must be compiled for each type/architecture of machine). Source (rrpmbuild --rebuild -rb p7zip-16.02-16.el8.src.rpm).
low-level utilities (rpm, dpkg) and high-level utilities (yum,dnf,apt,apt-get)
Every distribution has one or more package repositories where system utilities go to obtain software and to update with new versions, there are also external repositories like EPEL (Extra Packages for Enterprise Linux) - external package repository

4.1 RPM (RedHat Package Manager)
- rpm in itself doesn't retrieve packages over the network ,if not provided URI rpm installs only from the local machine
- types of packages: binary (<name>-<version>-<release>.<distro>.<architecture>.rpm) and source (<name>-<version>-<release>.<distro>.src.rpm) also packages with noarch.rpm extension don't depend on CPU architecture
- /var/lib/rpm = default directory which holds the RPM database (in form of Berkeley DB Hash Files).

# error: rpmdb: damaged header #929 retrieved -- skipping.  
rm -f /var/lib/rpm/__db* # remove lock files
rpm --rebuilddb

# alternative db directory (not /var/lib/rpm)
rpm --dbpath

# repair/rebuild DB
rpm --rebuilddb

# list installed packages 
# yum list installed
rpm -qa

# check file to which package it's belonging to
rpm -qf </path/to/file>

# information about the package (e.g if it is Relocatable)
rpm -qip <package.rpm> 

# list all files from the package aka $ls -lF $(rpm -ql </path/package>)
rpm -qil <package.rpm>

# verify if the files from the package are consistent with RPM database
rpm -Va </path/to/package> # no output means the package is ok

# install package
rpm -ivh <package.rpm>

# upgrade package (also remove the old package)
rpm -Uvh <package.rpm>

# freshening packages in current dir (when download patches and what to upgrade packages already installed)
rpm -Fvh *.rpm

# uninstall a package using --test flag before
rpm -e --test <package> (not path to package)

4.2 YUM (high-level package manager):

resolves dependencies automatically it also caches information to speed up performance
repos configuration: /etc/yum.repos.d
yum configuration: /etc/yum.conf
OS patches and update: /var/log/yum.log
toggle a particular repo in /etc/yum.repos.d change value from enabled to 0 or 1
dnf = next generation replacement for yum

# clear repo cache
yum clean all

# search for package
yum search <package>
yum list

# install package e.g yum install ngnix
yum install <package> 

# install from .rpm package
yum localinstall <package.rpm>

# check if updates are needed on local system
yum update
yum list updates
yum check-updates

# update package
yum update <package>

# list installed / available packages
yum list installed [installed | updates | available ]

# info about package $rpm -qip package
yum info <package>


# check file to which package it's belonging to $rpm -fq file
yum provides </path/to/file>
yum provides "/logrotate.conf"

# all packages with bash reference/installed and available/package info/dependencies
yum search bash
yum list bash
yum info bash
yum deplist bash

# show available repositories
yum repolist

# all packages that contain bash in their name
$ yum search bash

# installed and available bash packages 
$ sudo yum list bash

# package info
$ sudo yum info bash

# dependencies for package
$ sudo yum deplist bash

adding a new yum repo: vim /etc/yum.repos.d/webmin.repo


[Webmin]
name=Webmin Distribution Neutral
baseurl=http://download.webmin.com/download/yum
mirrorlist=http://download.webmin.com/download/yum/
mirrorlist
enabled=1
gpgcheck=0

You can (but should not) also turn off integrity checking with the gpgcheck variable.

I/O monitoring and I/O scheduling

system I/O bound when the CPU is in IDLE waiting for I/O to complete or the network is waiting to clear buffers
Network, RAM, CPU can wait for the I/O to complete
iostat = monitoring I/O devices activity iostat -k [KB] iostat -m [MB] iostat -xk [extended] (utilisation % close to 100 => system I/O bound)
iotop = display table of current I/O usage dynamically
ionice = set I/O scheduling and priority for a given process ionice [-c class] [-n priority] [-p pid] [command [args]]
I/O raporting bonnie and fs_mark
system performance depends on optimising I/O scheduling strategy
block layer [I/O scheduler - interface] low-level physical devices. I/O scheduler prioritises requests from Virtual Memory and Virtual File System to block-devices

LinuxFoundation - dejanu/linux GitHub Wiki

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️