linux_notes - RicoJia/notes GitHub Wiki

========================================================================

Philosophy

========================================================================

  1. small programs, each prgram is very good at tis job.

  2. (input -> process -> output)

  3. Good ways to learn

    • customization is a good way to learn
    • sometimes documentation is bad. you need to read the source code.

========================================================================

Basics

========================================================================

  1. Operating System: designed in 1956
    • MS is poppular because it's pre-installed before coming out of factory. Also, the eco-system is big: 1. most repair shops deal with Windows 2. So many software developed on Windows 3
    • Linux is small and software is installed in pre-designated places, so during reboot, /temp will be cleaned. Windows is huge and software is installed anywhere.
    • Linux has strict access control, root is not freqeuntly used. That's why it's hard to make Linux Virus.
  2. HIstory: invented in 1991, there are 60%+ server, Android phones, IBM cloud, tablet, Apache webserver.
    • Unix Clone. Unix: multi-user multitasking operating system developed by Bell Lab. Written in C.
      • Multi-user: multiple users can run programs at the same time.
      • System portability: moving from one brand of computer to another, with minimum change of code. Also, when upgrading the system, the user do not need to input data again.
      • Program -> Kernel (System calls) -> Hardware
      • Proprietary: Expensive, hard to fix.
  • What is HiLinux? Small OS for streaming vid?
  1. Kernel: flour dough of a cake. Kernel can be customized for different flavours. Ubuntu is the biggest (since it's like MacOS), then Debian. Cloud server (Redhat). Linux commands are all the same!! the kernel is the same as well. Like DOS in windows.
    • What is Linux: small programs that do one thing well.
      • A program being executed is a processes
      • Can be created and destroyed
      • Created with an environment:
    • User ID and Group ID. What is a group ID?
    • Working Directory
    • Open Files
      • Variables,
      • Resource limits: max file size, max amount of memory
      • Signal Action Setting?
      • Can create other processes
      • Can communicate with other processes.
    • kill process: kill -9 {PID} will always kill process
      • pid = 1 for init/ shutdown process
      • OOM (out of memory killer) killer
      • process can ignore some signals, send SIGKILL you might not be able to kill it.
      • This is how you kill a known node: kill -9 $(pidof discovery_server)
  • to restart deja-dup without backup-now greyed-out - pkill, seems to kill a process using regex
    • pkill deja-dup
    • usually, use kill -9 ($pidof processname)
      • pkill shelf_manager may kill shelf_manager_test.yaml
      • pkill, why not even worked when we changed our name?
        • by default, it works with partial names
    • repeat a command every N seconds: watch -n N command

  1. linux has terminal, tty, console. tty is teletypewriter that emulates a teletype writer

    • Each virtual console (tty)can be switched using: ctrl + alt + f1, f2, f3, f4. and you can switch by alt + left or right.

    • terminal (teletypewriter) takes keyboarw inpuq -> shell -> terminal output.

      • tty was "teletypewriter", it's like instant messaging in 1800s.
      • On linux there's "Pseudo TTY, like you're writing to a another teletypewriter." There's also virtual TTY (hardware that emulates DEC teletypewriter)
      • `terminal emulates physical devices. To access virtual TTY:
        • ctrl + alt +f1 to login page
        • ctrl+ alt+f2 is your current interface.
        • ctrl+alt+f3 is tty3, ctralt+f4 is tty4. (two different virtual counsels) on a tty, you alt <-,
        • If there's freeze in graphical UI, try logging into virtual TTY, then 
          top
kill

    • Search in terminal:

      • ctrl + r. searches command line through history in reverse order.
      • You type part of the command, and up and down like Matlab to auto complete. but you need to Restart bash to apply the changes, or bind -f ~/.inputrc. Equivalently, it's like loading these rules using bind "\e[B":history-search-forward

    • C-r is cumbersome and prefer to use the arrow keys to search through history based on what I've started typing. in ~/.inputrc add the following lines:

      "\e[A":history-search-backward
  "\e[B":history-search-forward
  Restart bash to apply the changes.
    • Assignment. backing up your work .inputrc tells shell how to interpret inputs, such as key strokes.
    • Tmux Specific
      1. ctrl +b +%(horizontal), "(vertical), arrow (navigation), x (kill)

  2. environment variables

    • when a shell session starts, a process will gather these variables.
      1. environment variables will be inherited from child processes (like gedit from the current session), shell variables do not.
    • env, printenv. they can be accessed by processes. printenv PATH
    • go 
         export nombre="Rico" # so this sets an environment variable (export marks this line for this process and the sub processes, then you can see the variable in printenv VAR).
      # with spaces inside: KEY="value with spaces"
      # multiple values: KEY=value1:value2:...
   printenv nombre. #or you go echo $home ($HOME is global environment variable)
    • export VAR=something
    • unset env variable: clear an environment variable

  3. reminder: var=1 sets a shell variable, which can be become a env variable by export var. Only env variables can eb inherited, and only by child process, not by other shell sessions.

  4. Shell: an interface, Common ones are sh (Bourne Shell, created by Steve Bourne), Bash(Bourne Again Shell) less common ones are Tcsh (TC shell), Zsh (Z Shell, bash compatible, with extra oh-my-zsh commands). They run on the same kernel. All these shell languages run on different shell interpreters, so they are interpreted languages

    • bash commands are small programs that resides in /usr/local/bin, etc.
      1. Bash commands are searched in folders specified in $PATH
      2. you can write small bash scripts, and execute them. 
            export PATH=$PATH:/place/with/the/file
    • bash is the default for Mac. Use echo $SHELL to check whether you're using bash.
      1. dash - basic, POSIX compliant shell, invoked with /bin/sh. Used for maximum portability. Not a good shell to use interactively.
      2. GUI is running on shell.
      3. Each Linux distro may have different shell, different functionalities but similar.
    • Terminal: A program that runs on shell. An interface between humans and the machine. The commands we put in are called Command-Line-Interface (CMI). Another one is GUI
    • interactive(execute a command u put in) vs non-interactive(execute a script)
    • Login(SSh) vs Non Log-In (terminal you'd have to open manually)
    • configuration files: non-login needs ~/.bashrc, login needs ~/.bash_profile. Mac users uses zsh: ~/.zshrc
    • For ROS, most supported shells are dash, bash and zsh

  5. max number of threads on linux: cat /proc/sys/kernel/threads-max

  6. Process

    • ctrl + z: put process in sleep, and it's still in memory. ctrl + c kills foreground processes ctrl+c, to kill a processes, quit. it sends a signal to kernel.
      • top, then hit 1 can see how many cores. then technically you can get n*100% usage
    • kill a process
      1. check a list of processes that contain a specific name: ps aux | grep something
      2. do either bash kill pid killall process_name
      3. ps a | grep something may show grep as well
      • one way to get around this is to grep [s]omething, so grep won't show the exact match

Linux Distro

  • Linux Structure

  1. Application User Interface (AUI)

    • Word processors, compilers
    • Shell: Bash, TCsh, Zsh. check cat /etc/shells

  2. API: Language libraries, System Call Interface (But this is still Application Space)

  3. Kernel Space: Memory Management (RAM, Disk), Process Management, File Management (these are core utilities from the GNU project).

    • Device Driver
    • BIOS (Basic Input/Output System): First software to run, it does booting: detecting hardware, confugure it and test it. Then, other parts the OS will start.. However, a new firmware interface is UEFI (Unified Extensible Firmware Interface), which might replace BIOS with
      1. has a GUI, WIFI network connection
      2. Can boot on a disk larger than 2 TB
      3. Backward, forward compatible.

  4. Hardware

    • A distribution is how linux installs and updates its softwares. But as a result, each distro has different sets of softwares:

  5. Before 1993, users usually needed to compile an application source code into binary levels, then install them into Linux. We want to just pull them down.

  6. Then we had Slackware, which still requires compilation, but versioning and location searching was made easier. (Source based distributions)

  7. Then, we have Debian as an alternative, which installs pre-compiled software from the internet. (Debian Distributions)

    • dpkg (.deb) focuses on Stable packges, even if older
      • sudo dpkg --install [.deb File]
    • Ubuntu makes money from Ubuntu Server Support.
    • Kubuntu, Lubuntu uses the same Ubuntu software stack. But they have different interfaces.

  8. 除了Debian, 还有Red Hat 系？？ (RPM, Red Hat Package Manager).

    • RedHat -> Fedora, CentOS. Redhat diverted from Desktop to Servers, Fedora becomes testbed for Red Hat Software. After Quality Assurance, software will go to Fedora Linux. RedHat sells "subscription" so ppl can get access to the Red Hat community.
    • CentOS: RedHat only owns the logo of the software on Red Hat public servers. CentOS installs all software on Red Hat Public Servers. Binary Compatible. This is community support, so organizations might wanna pay RedHat instead.
    • Suse

  9. Other Distros

    1. redhat: you can run multiple virtual servers on top of the red hat server (virtualization, means runs one computer as a software on another computer). It has "openstack"
    2. Ubuntu: developed in 2003, has Ubuntu desktop and ubuntu server.
    3. Fedora: NASA: fedora servers. GUI is fast. Also has desktop and server
    4. Kali Linux: for Hacker. Hacking tools. Accessing to system.
    5. Suse: 1994. IBM Watson OS. IBM is a huge fan of it.

========================================================================

File System

========================================================================

  1. Basic structure of UBUNTU: This is called the tree program.

    • home file systems, private for each user
    • root file system
    • /usr file system, sharable read-only programs and data
      • /usr/bin: files to execute, stores standalone utilites, not shell built-in, and comes with man page.
      • /usr/lib: shared libraries for programs
    • /var: "variable data files", logs.
    • /dev: hardware devices
    • /etc: Important files for system config, like IP address
    • /opt: add on software
    • /sbin: system administrator's binary files, used only by root.
      • like userdel, so you can have a guess where things are already
    • /bin: non-system administrator's binary files for commands, like ls
    • /lib: shared libraries, kernel modules
    • /mnt: mount points
    • /media: removable media devices
    • /run: run time temporary data, used to be stored in var
    • /proc: enables processes and kernels to interact
    • /snap: Ubuntu snap: packages programs and all their dependencies together

  2. File types

    • Ordinary Disk files
    • Directory files (files that contain other files)
    • Special Files, for extra devices.
    • files are for reading & writing to. /dev/null is "dumpster" to which redirection will cause the datastream (stderr, stdin...) will clear
    • how to check if a disk is full: df

  3. File names

    • Special characters, such as *, &, /, should be avoided.
    • * is wild card, ? is one whatever char.
    • Having trailing / will copy the content of the folder over. No / is the directory itself
    • so no / in source, / in target

  4. Permission

    • ls -l is to see file permissions
    • drwxr-xrwx d means directory, else it will be _. r is read, w is write,x is to execute. user|group|other (other ppl on the computer can read the file. ). For a group, r means to see what's in the folder, w means to move files around. x means entering that directory
    • 2 ways to change file permission:
      • chomd u+x, or # - is to remove, u g o are user, group, others.
        Or chmod u=x, o=wx
      • chmod 755 (each number is binary representation of rwx combination.), is common to set python script. chmod 664 is to set python script back to its original permissions.

  5. Play around users and groups

    • Add/delete users sudo useradd USERNAME
      • if you see $ when you login, that's dash shell. Check the default shell interface.
    • change password: 
      su
password USER
    • Switch between users: 
      getent passwd #to see all users
getent groups   #see all groups
su #switch user
su USER
    • Add a group 
      sudo addgroup lol
    • make a user in sudo group: 
      su 
usermod -aG sudo USER
#**restart the User**
#verify it's in sudo group: 
groups USER
sudo ls /root
    • change owner if you get readonly issues: 
      sudo su    # 1. chown needs sudo priviledge, so User needs to be in sudo admin group.   
sudo su - USER_NAME is to log in to a different account
chown (-R for folder) $USER:$USER folder_or_file 
chgrp (-R for folder) $USER:$USER folder_or_file
    • make an account expired or recover it 
       sudo chage -E 0 ACCOUNT  #make the account expired
 sudo chage -E -1 ACCOUNT #recover the account

 #passwd method does not lock an account for non-interactive shells like SSH
 sudo passwd -l ACCOUNT
 sudo passwd -u ACCOUNT    
 
 #change interactive shell to a "fake one", but this only works for interactive shells. Non-interactive shells 
 # (even SSH has it), still works. 
 sudo usermod -s FAKESHELL ACCOUNT

  6. package management

    • everything inside ubuntu repo is very well tested. eveyrthing else is not.

    • Debian uses APT, and APT has too many commands.

    • apt-get: apt is one level above apt-get.Repositories are servers for software downloads.

      • /etc/apt/sources.list.d lists servers to search from.
      • /etc is configuration file
      • apt-get remove keeps the configuration files. If you don't want them, do apt-get purge.

    • apt-cache

      • apt-cache search something: lists all softwares installed.
      • you can also use aptitude to see status of those related files as well. 
        aptitude search vlc
#c means config files, p means not existing on system, i means installed.

    • apt is newer than apt-get, came out in Ubuntu 16.04. Debian uses APT (Advanced Packaging Tool) to manage packages, apt-get was one tool for interacting with APT. apt only has the widely used features from apt-get and apt-cache

      • apt will enable default features, such as progress bar, number of pkgs installed.
      • have a few non apt-get commands:
        • apt search = apt-cache search
        • apt list
      • apt install -y means yes to all questions. -q means quiet
      • apt clean removes /var/cache/apt/archives/
      • apt update #updates list of available packages and versions.

    • install using Deb

      sudo apt install path_to_deb_file

    • 2 commands are required to remove a package completely:

      apt-get --purge remove libjpeg-dev
apt-get clean && apt-get autoremove

  7. nohup

    1. Use: "no hang up". Processes usually terminates upon terminating the terminal (which generates SIGHUP). But in SSH, if you drop connection, or quitting the terminal, you can use this to keep running a process. nohup ./binary
    2. nohup will not let the process to receive SIGHUP
      1. nohup: once started, stdin is not available to users. And output will be directed to a file nohup.out
    4. a & at the end will print the PID

  8. open file explorer from commandline: nautilus --browser file_exchange_port/Fun_Projects

  9. min file size is 4k. That's one block

Key files

  • bashrc
    1. Alias is your own shortcut
        vim ~/.bashrc
    alias foxit='/home/luigi/Foxit/FoxitReader'

========================================================================

Hardware & Communication

========================================================================

USB

  1. check if something has been mounted

File Descriptors

  • File descriptors vs file handles
    • File descriptor is a non-negative integer given by Linux to an open file, as an index.
    • File handle is in high level programming languages, which has write(), read(), close(), etc. Internally, it uses file descriptors, e.g., file.fileno() gives you that numebr
  • memory mapped vs non memory mapped
    • if you have 1G file, but 500MB memory, mmap allows you to access memory like an array. But deeper down, it manages what to load / unload from the memory (address space) & disk. Note: loading is on a "page" level.
    • lsof mem means "memory mapped files". usually shared libraries
    • txt is program text, including code and data
  • REG: there are DIR, unix files that I see.
  • Linux
  • [] and [[ ]]: [] is called test. It doesn't allow && and ||. But [[ ]] is only specific to bash.
  • /proc/PID is a window to a running process. It has fd (all open files, status: status info about the process, environ: )
  • > vs >>: >> appends to a file.
  • grep -v: select files do NOT match the file. grep -c X: count how many has X.
  • find {DIR} shows files matching your criteria in DIR
  • hard link vs soft links: hardlink points to file data, soft link point to datapath
    • use ls -l to see them. 
      123456 -rw-r--r-- 2 user group 0 Jun 17 12:34 file1
      • 2 is the number of hardlinks
  • xargs -0 means inputs are terminated by null cahr, instead of white space. Usually used with find X -print0
  • readlink, read the softlink provided.
  • awk -F'[:]' '{print $1}' in gawk, print the first field is 1 (field before separateors), not 0.
  • htop: gives PID?
  • $(date) gives date like: Wed 14 Jun 2023 01:52:09 PM CDT

udev rules

  1. Turn on the udevadm monitor

    1. udev means userspace /dev. You can see idVendor, idProduct by lsusb
    2. Udev rule?
    SUBSYSTEMS=="usb", ACTION=="add", ATTRS{idVendor}=="06c2", ATTRS{idProduct}=="00[3-a][0-f]", MODE ="666", SYMLINK="something%n"

     - once you copy the udev rules to /etc/udev/rules.d, your usb device will be recognized. (as anything)
 - ```%n``` is a macro for the index from the kernel.
    1. reload rules sudo udevadm control --reload-rules
    2. see: https://opensource.com/article/18/11/udev

  2. see attr... of a device from udev rules: udevadm info /dev/video2

  3. see output of a video ffplay /dev/video0

  • list of devices: v4l2-ctl --list-devices

SSH

  1. Configure SSH

    ssh-keygen -t rsa #generate ssh keypair, you can choose where to store it, and if a password is required on your loacl machine. called rsa
ssh [email protected] mkdir -p .ssh
cat .ssh/id_rsa.pub | ssh [email protected] 'cat >> .ssh/authorized_keys'	#copy things over
ssh [email protected] "chmod 700 .ssh; chmod 640 .ssh/authorized_keys"
    • check existing users on this system cut –d: –f1 /etc/passwd
    • add an existing user as sudo: $ usermod -aG sudo User_name (you still need password once!)
    • if you do not want to put in password for anything (but you still need the sudo keyword)
      1. edit /etc/sudoer: you have to use sudo visudo
      2. edit %sudo ALL=(ALL:ALL) NOPASSWD:ALL

  2. Misc SSH Commands

    • ssh-copy-id SERVER_NAME copy the keypair over to the remote machine
    • You can execute any bash command over ssh: ssh SERVER_NAME COMMAND
      • Execute two commands together ssh SERVER_NAME `COMMAND1; COMMAND2' You need '' and ;
      • if you don't have '', then you exeucte the second command on the local machine ssh SERVER_NAME COMMAND1; COMMAND2
      • sudo ssh SERVER_NAME sudo COMMAND1. And you need sudo password for that.
    • ping the SSH if you don't get anything back from ping, it might be the firewall blocking you.

  3. see last login ip address: ${SSH_CLIENT}

    • check port: grep -i port /etc/ssh/sshd_config
    • specify port: rsync -e 'ssh -p PORT_NUM' -av...
    sudo -i
passwd

  4. SSH

    • Find server IP address: hostname -I

    • Start SSH on server

      sudo systemctl enable ssh
sudo systemctl start ssh		
# if not restarting, do
sudo systemctl restart ssh

    • Check server port sudo grep Port /etc/ssh/sshd_config

    • Connect local machine to server port ssh NAME@IP -p PORT

    • share ssh keypair (optional)

      • share key pair, credentials

  5. rsync

    • you don't need ssh
    • pay attention to the dir you write to. use ls -l | grep home to see permission rsync -av --delete arm_install/ student@turtlebotX:/home/student/instal
    • work_interface is the folder, work_interface/ is everything in there.

  6. sshfs

    1. download using wget: wget https://github.com/libfuse/sshfs/releases/download/sshfs-3.7.2/sshfs-3.7.2.tar.xz

    2. mount the system

      • sshfs -p 1337 [email protected]:/REMOTE_DIR /LOCAL_MOUNT_POINT
      • Note: the mount point on your system is to be temporarily "overwritten" by the remote dir, after unmounting, the mount point will go back to where it was.
      • may see "mountpoint not empty if sshfs is not installed!

    3. unmount: fusermount -u /home/ricojia/gym-delta-robot-trampoline

  7. Downloading from remote server:

    • on local machine, do ```scp -p <port_number> LOCAL_FILE_PATH remote_server@remote_ip:/REMOTE_FILE_PATH ``
    • ip a will tell you the port and address of your vpn: inet 10.212.134.206/32 scope global vpn

  8. Jump though the jumhost (in this case 10.101.4.14) and get onto destination (in this case 10.1.6.2) in a single shot

    1. ssh, (requires password twice). This one does X11 forwarding too. 
      ssh -tt -YC -o ServerAliveInterval=60 [email protected] "bash -c 'ssh  -YC -o ServerAliveInterval=60 [email protected]'"
    2. Scp stuff back to local machine passing through a jumphost (10.101.4.14) from remote host (10.1.6.2) in a single shot (requires password twice) 
      scp -r -oProxyCommand="ssh -W %h:%p [email protected]" [email protected]:/home/aradmin/production/spoofer_exp ./
    3. Scp stuff from local machine out to a store host. 
      scp -r -oProxyCommand="ssh -W %h:%p [email protected]" some_directory [email protected]:/home/aradmin/
    4. You might have one-way traffic for ssh: there's a proxy you have to jump through. The proxy forwards you through the port to the robot, but you are not exposed to the robot. So the robot can't see you.
      • So to scp, you have to do this on your local machine: scp moxi@moxi2:/home/moxi/moxi_dev/test.wav .

Misc

  1. Binary Compatible: you can swap it out with another binary without noticing any changes.

    • I use rip for safe deletion.

  2. input keyboard may collide with the terminator shortcuts, if you have installed another keyboard c-s-e. just delete the emoji

      • ctrl + k delete line after; ctrl+w: delete previous word. ctrl+del deletes the word after

  3. username must be less than 8 char, because a lot of commands like ps displays"<"8 char and '+'. This is historical

  4. head -c32: the first 32 bytes of data,basically the first 32 char of a Hex

  5. help

      man ls:    
         #/someword is to search, 
         j,k is down and up. 
         man cd doesn't exist because cd is not a program. but you can try help. 
         man was invented before www. so there's no manual pages that have no hyperlink
         man 2 mkdir vs man mkdir are the same. 
     info: emac style keys. 
         info for the "new tools"? it's better than man

  6. services

    • restart forticlient 
      sudo systemctl restart forticlient-scheduler.service
    • systemctl cat runtime_cloud_proxy.service
    • journalctl 
      journalctl -fu service --since "24 hour ago | grep -i restart"
journalctl --since "2015-06-26 23:15:00"
    • ln?
      • hardlink:
        • has the same actual content, even tho the orignal file has been moved/removed. But it's still a reference, not the content
          • ln src, dst
        • has the same permission, inode numbers
        • Doesn't work with dirs
        • hardlink must work under the same file system
      • Softlink:
        • it's pointed to the path of the content. So if the original file has been moved, it wouldn't work
        • ln -s src, dst
        • Can work under a different file system
    • Create a service dummy_echo.service:
      1. create script
        • Note: add shebang (first line), make sure filepath is right.
        • doesn't have to be in /usr/local/bin
      2. In /etc/systemd/system, create dummy_echo.service
        • Service file Doesn't need to be executable
        • when you change the service file, need to sudo systemctl daemon-reload (there'll be a reminder)
      3. Start service: sudo systemctl restart dummy_echo.sh
      4. check output: sudo journalctl -f -u dummy_echo. -f means realtime
        • tail -f log: print to file thru watching
    • File permision.
      • chmod XX5, will make the file not visible to you, if you're not owner, group (i.e., the file is root), because 5 means no write priviledge.

  7. record screen: c-s-alt-r

  8. switch on link sink:

    • pacmd list-sinks to see the index of audio sink
    • pacmd set-default-sinks index to switch on the audio sink

Common Errors & Small Adjustments

  1. Dummy output in sound
pulseaudio -k && sudo alsa force-reload
  1. Change Desktop
gsettings get org.gnome.desktop.background picture-options #see what we have for the picture options
gsettings set org.gnome.desktop.background picture-options 'scaled'

========================================================================

Operating System

========================================================================

  1. KVM (kernel virtual machine)

    • It's a type of hypervisor (virtual machine monitoring, VMM)
      1. a real computer has hardware (cpu, disk drive). VM (virtual machine) also has them, but only in code.
      2. So, we need hypervisor to host multiple VM.
        1. type 1, native baremetal: hardware -> VMM -> guest OS. Fast
        2. type 2 hosted: hardware -> host OS -> app... VM -> guest OS
    • KVM is built into Linux. which is actually below linux. So it's type 1 baremetal

  2. systemd process?

    • cockpit - replace byobu,
    • gRPC server, - anjana
    • cron jobs James

  3. X Server unnamed

  4. Keyboard:

    • if alt and windows keys are swapped: FN+P swaps Mac Layout to PC ( hold for 3 seconds) FN + O swaps PC Layout to Mac ( hold for 3 seconds)

GPU

  1. See Process: nvidia-smi If multiple GPUs are not available, use another one.

Networking

  1. DNS: domain name system. www.google.com -> 192.168.1.2
  2. DHCP Dynamic Host Configuration Protocol
    • In small networks, IP addresses can be assigned manually. But in larger ones, you need DHCP.

Rico's wiki

Home | Copyright © 2021 ricojia

⚠️ **GitHub.com Fallback** ⚠️