boot process FIXME (clean up and see FIXMEs) - alex-aleyan/linux_wiki GitHub Wiki

kernel boot messages:
/var/log/dmesg
dmesg | less

kernel run messages:
/var/log/

lspci #lists PCI buses on your computer and devices connected to them
  lspci -vvv
lsusb #lists information about the computer’s USB hubs along with any USB devices connected to the computer’s USB ports
lscpu #lists information about your processor

removable hardware (udevd daemon, /dev, /media, lib/modules Hardware Abstraction Layer HAL):
/dev/* : list of devices
/media/*: where the devices' filesystems are automatically mounted
lib/modules: where the KERNEL Modules are installed.
lsmod
modinfo sunrpc#list info about available modules
 modinfo /lib/modules/2.6.32-642.el6.x86_64/kernel/net/sunrpc/sunrpc.ko
 modinfo -d snd-seq-oss
modprobe MODULENAME #used to load a module
rmmod MODULENAME
  modprobe -r MODULENAME #removes module and the dependent modules that are not being used by other modules


udevadm test /block/sda
lspci | grep Mella
/sbin/modprobe knem > /dev/null 2>&1
dmesg # Kernel Ring Buffer

udevadm control --log-priority=info
/etc/udev/udev.conf # do "cat udev.conf" to see what it can be set to

References:

  • How linux Works: Ch4 page 81, ch5 and ch6.
  • Linux Bible: Ch21 - Boot-Up troubleshooting.

Boot process ()

  1. Starting HARDWARE from BIOS.
  2. The machine’s BIOS or boot firmware loads and runs a boot loader.
    1. Finding the location of the BOOT LOADER (GRUB) in the BOOT RECORD and starting the BOOT LOADER.
    2. The boot loader finds the kernel image on disk, loads it into memory, and starts it.
  3. Starting KERNEL and INITIAL RAM DISK for the selected OS.
    1. The kernel initializes the devices and its drivers.
    2. The kernel mounts the root filesystem.
  4. Starting the INIT Process with a process ID of 1 (upstart init daemon - this is the start of user space).
    1. The INIT sets the rest of the system processes in motion.
    2. Starting INITIALIZATION SCRIPTS and SERVICES for the current runlevel.
    3. At some point, INIT starts a process allowing you to log in (usually at the end or near the end of the boot).

1. BIOS (Basic Input Output System):

  1. Is a small program stored on the motherboard.
  2. Initializes hardware and hands off control of the boot process to the BOOT LOADER.
  3. BIOS is used to:
    1. Setup Utility.
    2. Change Boot Order (useful for Rescue Mode or Fresh Install to boot from CD).
    3. BIOS provides info about:
      • type of system.
      • its BIOS version.
      • its processors.
      • its memory slots and types.
      • whether it is 32-bit or 64-bit.
      • which devices are in each slot.
      • many details about the types of devices attached to the system.
  4. BIOS allows you to
    • enable/disable hardware.
    • change device settings.
    • configure NIC PXE Boot Settings to boot from server found on the network.
    • configure Virtualization Settings to run the system as a Virtual Host (CPU must include Intel Virtual Technology or AMD Secure Virtual Machine (SVM) support. It).
    • and many other features found on a processor and the motherboard.

2. GRUB Boot Loader:

  1. BIOS finds the master boot record on the first hard disk and begins loading that boot loader in stages.
    1. BOOT LOADER tries to run the KERNEL and the content of the initial RAM DISK (which contains DRIVES and SOFTWARE needed to boot your hardware).
    2. Boot Loader may fail when:
      1. Unable to locate Active Partition:
        1. if the boot loader is on the storage medium, the partition is usually marked as bootable.
        2. If sure the boot loader is on a disk, use fdisk to make the partition bootable.
      2. Selected boot device not available:
        1. Master Boot Record is either deleted from the hard drive, or the content of the Hard Drive is expected to be loaded from another boot loader (boot CD).
        2. Try booting the system from other media.
        3. If the master Boot Record indeed was erased, try booting rescue media to attempt to recover the contents of the disk.
        4. Try using Disk Forensics to find the data on the disk.
        5. If the Master Boot Record was overwritten due to another operating system being installed on another disk partition, try reinstalling the Master Boot Record from the Rescue Mode.
      3. Text-Based GRUB prompt appears:
        1. means that the master boot record portion of GRUB was found, but GRUB failed to find the next stage of the boot process and a menu of operating systems to load.
        2. This happens when the BIOS detects the disks in the wrong order and looks for the grub.conf file on the wrong partition. This can be corrected by listing the contents of the grub.conf and entering root, kernel and initrd lines manually. To list the content of the file, type cat (hd0,0)/grub/grub.conf ; if it doesn’t work, try (hd0,1)/grub/grub.conf ; if it doesn’t work, try (hd1,0)/grub/grub.conf and so on. Once the lines representing the grub.conf file are found, manually type root, kernel and initrd lines for the entry you want replacing the location of the hard drive you found on the root line and type boot. Once the system boots, manually fix the Boot Loader files (see Ch9).
    3. BOOT LOADER can be interrupted:
      1. when the BIOS finds the BOOT LOADER in the MASTER BOOT RECORD of the disk (fist 1k bytes?), and the BOOT LOADER finds the GRUB Configuration Files on that disk, the BOOT LOADER starts to count down for 3-5 s.
    4. Interrupt the boot loader to:
      1. start in a different runlevel:
        1. Highlight the OS you want and hit e.
        2. Highlight the KERNEL and hit e.
        3. Add the new runlevel to the end of the line (0: Halt; 1:single user; 2: not used; 3: full multi-user command line; 4:not used; 5:full multi-user graphical mode; 6:Reboot; also the default run level is listedn in /etc/inittab).
      2. select a different KERNEL:
        1. when a new KERNEL is installed using yum, old one is not removed. After selecting OS and hitting e; just use an arrow key to select a different KERNEL to boot.
      3. select a different OS.
      4. change boot options to add KERNEL arguments to:
        1. disable hardware temporarily (adding nousb to disable USB ports).
        2. bypass the init process and go straight to a shell (init 1; single user).

3. Starting the KERNEL:

  1. Once the Red Hat Enterprise Linux screen with slow-spinning icon appears, hot ESC key to see the messages regarding the boor process.

  2. Here the KERNEL tries to load the DRIVERS and MODULES needed to use the HARDWARE. Look for HARDWARE FAILURE messages.

  3. The messages shown during this step are copied to the KERNEL RING BUFFER which stores the KERNEL messages (boot messages and running service’ messages once booted):

    dmesg > ~/kernel_msg.txt
less /tmp/kernel_msg.txt

  4. Once KERNEL is done detecting HARDWARE, and loading DRIVERS; it passes off control to INIT PROCESS.

4. Init Process:

  • The Init Process consists of the /sbin/init running through the next sequence:

    • /etc/rc.d/rc.sysinit sets the environment path, starts swap, checks the file systems, and executes all other steps required for system initialization.
    • /etc/inittab #describes how the system should be set up in each SysV init runlevel, and sets the default runlevel and dictates that /sbin/update should be run whenever it starts a given runlevel
    • The /sbin/init sets the source function library: /etc/rc.d/init.d/functions
    • /etc/rc.d/rc3.d/ # init program runs the scripts in the /etc/rc.d/rc5.d/ directory to determine which processes from /etc/rc.d/init.d/functions to start and stop

  • Once the KERNEL hands off the control of the BOOT PROCESS to the INIT PROCESS, the INIT PROCESS checks /etc/initab file for directions on how to boot.

    • The /etc/initab file contains information on Default RUNLEVEL:

      [root@xes_blade02 ~]# view /etc/inittab
# inittab is only used by upstart for the default runlevel.
#
# ADDING OTHER CONFIGURATION HERE WILL HAVE NO EFFECT ON YOUR SYSTEM.
#
# System initialization is started by /etc/init/rcS.conf
#
# Individual runlevels are started by /etc/init/rc.conf
#
# Ctrl-Alt-Delete is handled by /etc/init/control-alt-delete.conf
#
# Terminal gettys are handled by /etc/init/tty.conf and /etc/init/serial.conf,
# with configuration in /etc/sysconfig/init.
#
# For information on how to write upstart event handlers, or how
# upstart works, see init(5), init(8), and initctl(8).
#
# Default runlevel. The runlevels used are:
#   0 - halt (Do NOT set initdefault to this)
#   1 - Single user mode
#   2 - Multiuser, without NFS (The same as 3, if you do not have networking)
#   3 - Full multiuser mode
#   4 - unused
#   5 - X11
#   6 - reboot (Do NOT set initdefault to this)
#
id:3:initdefault:

  • FIXME: Say something here

    • The /etc/initab file points to the files in the /etc/init to perform tasks as remapping keystrokes, start virtual console, identify the location of the script for initializing basic services on the /etc/rc.sysinit
    • The 2 common problems with the init are in rc.sysinit file and the runlevel scripts.

  • Troubleshooting rc.sysinit

    • When INIT runs rc.sysinit, the rc.sysinit sets system’s hostname, /proc and /sys filesystems, sets up SELinux, sets KERNEL PARAMETERS, and other tasks.

    • Setting up storage on the system is one of the most critical functions of rc.sysinit.

    • Typical rc.sysinit failures:

    • Local Mounts Fail: if entry from /etc/fstab fails to mount, the boot process ends but runlevel services never get called (test your fstab when modifying its content). As a result, you have root shell with only root filesystems mounted. To fix this problem, remount the root filesystem, correct the fstab file, mount the fstab entries to make sure they work and reboot:

      mount –o remount,rw /
vim /etc/fstab
mount –a
reboot

  • FIXME: Say something here

    • Hostname not set:

      • The rc.sysinit uses the value of “HOSTNAME=” from /etc/sysconfig/network, to set the system hostname.
      • If missing, the name localhost is used as a default value.
      • The hostname value can be acquired from DHCP server.
      • Check through the processing of rc.sysinit to determine the problem.

    • Cannot decrypt filesystem:

      • the rc.sysinit uses the content of the /etc/crypttab file to decrypt encrypted filesystems.
      • If the file becomes corrupted and the backup cannot be found, you are out of luck.

    • Troubleshooting runlevel processes:

      • Set the level of activity on the system.
      • Use view /etc/inittab file to see the runlevel specs.
      • In /etc/rc?.d/ directory each file is a pointer to a script in /etc/init.d/
      • Use runlevel command to see the current runlevel.
      • Can be changed by modifying /etc/inittab content or by using init command (telinit –verbose 3).
      • runlevel 0 (/etc/rc0.d): is the Shutdown level:
        • processes are stopped and computer is powered down.
      • runlevel 1 (/etc/rc1.d): Single-User level
        • only boot processes are run (processes required to boot the pc, mount filesystems, and make the system available from the console).
        • Networking processes are not started.
        • Authentication is bypassed and root is a default user in this level.
        • Using keyword single (or init –t1 S) instead of 1 (init 1) bypasses the start of scripts in /etc/rc1.d
      • runlevel 2 (/etc/rc2.d): multiuser runlevel:
        • Same as 3 but without networking.
        • Rarely used today.
      • Runlevel 3 (/etc/rc3.d): Multiuser with networking level:
        • used to boot the system when graphical environment is not used.
        • Graphical desktop can be launched after boot time by the user.
      • Runlevel 4 (/etc/rc4.d): typically not used and the same as runlevel 3.
      • Runlevel 5 (/etc/rc5.d): Multiuser with networking and graphical interface.
      • Runlevel 6 (/etc/rc6.d): reboot.