11. advanced and devops - mishraxharshit/harshitxmishra.github.io GitHub Wiki
Phase 11 — Advanced and DevOps
Previous: [Phase 10 — Security](Phase-10-Security) | Back to: [Home](Home)
11.1 Containers with Docker
A container packages a program and all its dependencies into a single unit that runs consistently anywhere. Unlike a virtual machine, containers share the host kernel and are much lighter.
# Install Docker
sudo apt install docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER # add yourself to docker group
# Log out and back in for group change to take effect
# Core Docker workflow
docker pull nginx # download an image
docker images # list downloaded images
docker run nginx # run a container (foreground)
docker run -d nginx # run detached (background)
docker run -d -p 8080:80 nginx # map host port 8080 to container port 80
docker run -d --name webserver nginx # give it a name
# Container management
docker ps # running containers
docker ps -a # all containers including stopped
docker stop webserver
docker start webserver
docker rm webserver # delete container
docker rmi nginx # delete image
# Get a shell inside a running container
docker exec -it webserver bash
# View container logs
docker logs webserver
docker logs -f webserver # follow live
# Build your own image from a Dockerfile
docker build -t myapp:1.0 .
docker run -d -p 8080:80 myapp:1.0
A minimal Dockerfile:
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y python3 && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY app.py .
EXPOSE 8000
CMD ["python3", "app.py"]
11.2 Kernel Modules
The kernel is modular: drivers and features are compiled as modules and loaded only when needed.
# List all currently loaded modules
lsmod
# Get info about a module
modinfo ext4
modinfo usbhid
# Load a module
sudo modprobe bluetooth
# Remove a module
sudo modprobe -r bluetooth
# List modules that will load at boot
cat /etc/modules
# Add a module to load at boot
echo "bluetooth" | sudo tee -a /etc/modules
# Block a module from loading (useful for disabling hardware or security)
echo "blacklist btusb" | sudo tee -a /etc/modprobe.d/blacklist.conf
11.3 Performance Tuning
# CPU frequency scaling
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
# Options: performance, powersave, ondemand, conservative
sudo cpupower frequency-set -g performance # set all CPUs to performance
# Memory: transparent huge pages
cat /sys/kernel/mm/transparent_hugepage/enabled
# [always] madvise never
sudo echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
# Network: increase socket buffer sizes for high-throughput servers
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 134217728"
# Disk I/O scheduler
cat /sys/block/sda/queue/scheduler
# Options: mq-deadline, kyber, bfq, none
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler # for NVMe SSDs
11.4 Logical Volume Manager (LVM)
LVM adds a layer of abstraction over physical storage, making it easy to resize volumes without downtime.
Physical disks → Physical Volumes (PV) → Volume Group (VG) → Logical Volumes (LV) → Filesystem
# Create a physical volume on a disk or partition
sudo pvcreate /dev/sdb
# Create a volume group
sudo vgcreate myvg /dev/sdb
# Create logical volumes
sudo lvcreate -L 20G -n data myvg # 20 GB named "data"
sudo lvcreate -L 5G -n logs myvg # 5 GB named "logs"
# Format and mount
sudo mkfs.ext4 /dev/myvg/data
sudo mount /dev/myvg/data /mnt/data
# Extend a logical volume (add space)
sudo lvextend -L +10G /dev/myvg/data # add 10 GB
sudo resize2fs /dev/myvg/data # resize filesystem to use new space
# View LVM info
sudo pvs # physical volumes
sudo vgs # volume groups
sudo lvs # logical volumes
11.5 Environment Variables and Shell Configuration
# View all environment variables
env
printenv
# View a specific variable
echo $PATH
echo $HOME
echo $USER
# Set a variable for the current session
export MY_VAR="hello"
# Set a variable permanently for your user
nano ~/.bashrc # runs for every new interactive bash shell
nano ~/.profile # runs once on login
# Add to PATH permanently
echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc # reload without logging out
# Common variables
# PATH — directories to search for commands
# HOME — your home directory
# USER — your username
# SHELL — your default shell
# EDITOR — default text editor
# LANG — system language and locale
# TERM — terminal type
11.6 Virtualization with KVM
KVM (Kernel-based Virtual Machine) turns Linux into a hypervisor, running full virtual machines.
# Check if your CPU supports virtualisation
grep -E 'vmx|svm' /proc/cpuinfo
# Install KVM and management tools
sudo apt install qemu-kvm libvirt-daemon-system virtinst virt-viewer
# Add user to the libvirt group
sudo usermod -aG libvirt $USER
# Create a virtual machine
sudo virt-install \
--name ubuntu-vm \
--ram 2048 \
--disk path=/var/lib/libvirt/images/ubuntu.qcow2,size=20 \
--vcpus 2 \
--os-variant ubuntu22.04 \
--cdrom /path/to/ubuntu-22.04.iso \
--graphics vnc
# Manage VMs
virsh list --all # list all VMs
virsh start ubuntu-vm
virsh shutdown ubuntu-vm
virsh destroy ubuntu-vm # force off
virsh snapshot-create-as ubuntu-vm snap1 # take a snapshot
Phase 11 Exercises
Exercise 1: Install Docker. Pull the hello-world image and run it. Then pull the nginx image, run it on port 8080, and visit http://localhost:8080 in a browser.
Exercise 2: Use lsmod to list loaded kernel modules. Use modinfo to learn about one module. Try loading and unloading it with modprobe.
Exercise 3: Add a directory ~/bin to your PATH by editing ~/.bashrc. Create a script in ~/bin/hello that prints a message. Reload your shell and run hello from any directory.
Previous: [Phase 10 — Security](Phase-10-Security) | Back to: [Home](Home)
Troubleshooting Guide
Back to: [Home](Home)
When a Command Is Not Found
# Check if the command exists
which python3
type python3
# If not found, install the package
sudo apt search python3
sudo apt install python3
# Or it may be installed but not in your PATH
find /usr -name "python3" 2>/dev/null
# Then add that directory to PATH
When Permission Is Denied
# Check the file's permissions
ls -la file.txt
# Check who you are and what groups you are in
id
# Does the file owner or group match you?
# If not, you need sudo or to change permissions
# Check directory permissions (you need x on all directories in the path)
ls -la /path/to/
# Common fix: your user needs to be in a group
sudo usermod -aG groupname $USER
# Then log out and back in
When a Service Will Not Start
# Check the status and recent error messages
sudo systemctl status servicename
# See the last 50 log lines for the service
sudo journalctl -u servicename -n 50
# Check the service's configuration file for syntax errors
# (depends on the service — nginx has:)
sudo nginx -t
# apache has:
sudo apache2ctl configtest
# Try starting manually to see errors directly
sudo /usr/sbin/nginx -g "daemon off;"
When Disk Is Full
# Confirm disk is full
df -h
# Find what is using space
du -sh /var/log/* # check logs first
du -sh /home/* # check home directories
du -sh /tmp/*
# Clear old logs
sudo journalctl --vacuum-size=100M
# Find large files
find / -type f -size +100M 2>/dev/null | sort -k5 -rn
When Cannot Connect via SSH
# From the server console (not SSH):
# Is sshd running?
sudo systemctl status ssh
# Is it listening?
sudo ss -tlnp | grep :22
# Does the firewall allow it?
sudo ufw status
# Check the auth log for errors
sudo tail -50 /var/log/auth.log
# Is the user's .ssh directory and authorized_keys set up correctly?
ls -la ~/.ssh/
# ~/.ssh must be 700
# ~/.ssh/authorized_keys must be 600
When System Is Slow
# Check what is using CPU
top -b -n 1 | head -20
# Check what is using memory
free -h
ps aux --sort=-%mem | head -10
# Check disk I/O
iostat 1 5
sudo iotop -o -b -n 5
# Check for zombie processes
ps aux | grep Z
# Check system load
uptime
# Load average > number of CPU cores means system is overloaded
nproc # see how many CPUs you have
Back to: [Home](Home)