Process Thread Memory & Libraries kernel userspace - MarekBykowski/readme GitHub Wiki
A synthesis of how a process/thread is represented, how fork and
pthread_create differ, how libraries and files are wired, and the kernel
structs behind it all. Companion to the mb_labs; each section notes the lab
that exercises it.
PROCESS WORLD (MM) FILESYSTEM WORLD (VFS)
per-process (threads share one set) nobody's -- shared by all
+-----------------------------+
| task_struct |
| thread / scheduling |
+--------------+--------------+
| mm
v
+-----------------------------+
| mm_struct |
| address space |
+--------------+--------------+
| mmap (the VMA list)
v
+-----------------------------+ vm_file +-----------------------------+
| vm_area_struct |------------->| struct file |
| VMA (+ vm_file) | | an open-file instance |
+-----------------------------+ +--------------+--------------+
| f_inode / f_mapping
v
+-----------------------------+
| inode |
| metadata + address_space |
| (the page cache) |
+-----------------------------+
-
Left = the MM (process) world —
task_struct -> mm_struct -> vm_area_struct. Per-process: each process has its ownmmand VMAs (threads of a process share one set; separate processes each have their own). -
Right = the VFS (filesystem) world —
struct file -> inode -> page cache. Nobody's / shared: oneinode(and its page cache) is shared by everyone who maps or opens that file. -
vm_fileis the bridge: a file-backed VMA points into the VFS world, so the region's pages come from that file's page cache — which is how a shared.soends up as one physical copy behind many processes' private VMAs. Anonymous VMAs (heap/stack) havevm_file == NULLand stay entirely in the MM world (no VFS side).
Reading /proc/<pid>/maps walks the left column (mm->mmap) and, for each VMA,
follows vm_file into the right column to print the file's dev/inode + path.
fork() pthread_create()
| glibc | glibc
v v
clone(SIGCHLD) clone(CLONE_VM | CLONE_FILES | CLONE_FS |
(no sharing flags) CLONE_SIGHAND | CLONE_THREAD |
| CLONE_SETTLS | ...)
v v
kernel: copy_process() -> new task_struct; per resource: SHARE or COPY
copy_process()
/* simplified from kernel/fork.c */
static int copy_mm(unsigned long clone_flags, struct task_struct *tsk)
{
if (clone_flags & CLONE_VM) {
mm = oldmm; /* SHARE: same mm_struct, bump refcount (pthread) */
atomic_inc(&mm->mm_users);
} else {
mm = dup_mm(tsk, oldmm); /* COPY: new mm_struct, COW page tables (fork) */
}
tsk->mm = mm;
}
Rule in copy_process(): flag set (1) -> share the pointer; flag clear (0) -> copy.
| resource (task_struct field) | flag | fork (=0) | pthread (=1) |
|---|---|---|---|
address space mm
|
CLONE_VM | copy (new mm, COW) | share (same mm) |
fd table files
|
CLONE_FILES | copy | share |
cwd/root fs
|
CLONE_FS | copy | share |
signal handlers sighand
|
CLONE_SIGHAND | copy | share |
thread group / tgid
|
CLONE_THREAD | new tgid | same tgid |
| TLS (TPIDR_EL0) | CLONE_SETTLS | inherited | set per-thread |
- fork: new task_struct, own mm (copied, COW), new tgid -> a PROCESS.
- pthread: new task_struct, shared mm (same pointer), same tgid -> a THREAD.
- The kernel schedules both identically as tasks; everything else falls out of the
mmdecision.
Identity: getpid() == kernel tgid (process); gettid() == kernel pid (thread).
The word "pid" flips meaning between layers. (Lab: procthread_lab.)
task_struct -- ONE per thread (the schedulable entity)
|- state, flags(PF_KTHREAD), pid, tgid, comm[16], real_parent, group_leader
|- stack ------> kernel stack
|- mm ---------> mm_struct (address space; NULL for kthreads)
|- active_mm --> installed mm (kthreads borrow one)
|- files ------> files_struct (fd table)
|
(mm) mm_struct -- ONE per process (shared by its threads)
|- mmap ------> vm_area_struct (head of the VMA list)
|- mm_rb -----> rbtree of the VMAs (fast find_vma)
|- pgd -------> page tables (-> TTBR0)
|- mm_users --> # threads sharing it
|- map_count, mmap_sem, start_code/start_stack ...
|
(mmap) vm_area_struct -- ONE per memory region
|- vm_start, vm_end the VA range [start,end)
|- vm_flags VM_READ/WRITE/EXEC/SHARED
|- vm_page_prot -> PTE permission bits
|- vm_file backing struct file (NULL = anonymous)
|- f_path {mnt,dentry} -> pathname -> /proc maps
|- f_op -> file_operations (.read/.write/.mmap)
|- f_inode ----------> struct inode
|- i_ino -> /proc maps "inode"
|- s_dev -> /proc maps "dev"
|- i_mapping -> address_space (PAGE CACHE)
|- vm_pgoff offset into vm_file
|- vm_ops .fault/.open/.close handlers
|- vm_next/prev, vm_rb, vm_mm
-
task_struct = the thread: identity + pointers.
mmNULL for kthreads. -
mm_struct = the address space: page tables (
pgd) + the list of VMAs (mmap). Shared by threads (CLONE_VM), copied by fork. -
vm_area_struct (VMA) = one contiguous region with uniform perms/backing.
A
.so= several VMAs (text r-xp shared, data rw-p COW). -
files_struct = the fd table (array of
struct file *). -
struct file = one open-file instance (offset, flags,
f_op); your char devices'.read/.write/.mmapare reached through it.
Labs: pid/tgid/comm/mm -> procthread_lab,current_lab,findtask_lab;
mm->mmap/VMAs -> vma_lab; pgd/page tables -> ptwalk_lab; vm_ops/.mmap ->
mmap_lab; struct file/file_operations -> procfs/sysfs/mmap/procthread labs.
static .a
|
dynamic .so
|
|
|---|---|---|
| linked | build time (copied into binary) | runtime (ld.so mmaps it) |
| binary size | larger | smaller |
| memory across processes | own copy each | ONE shared RO-text copy |
| fix the lib | relink everything | drop-in, all benefit |
A .so's read-only text is mapped shared across every process using it -> one
physical copy via the page cache. Writable data is COW (per-process).
| userspace | kernel analog |
|---|---|
static .a in the binary |
built-in (obj-y -> vmlinux) |
dynamic .so via ld.so |
module .ko via insmod (linked vs kernel symtab) |
| libc: printf/malloc | kernel: printk/kmalloc (EXPORT_SYMBOL) |
| ldd | modinfo/depmod |
A .ko resolves undefined symbols against the kernel symbol table (only
EXPORT_SYMBOLed ones) + loaded modules; CRC (modversions) guards the ABI.
"Unknown symbol" = the needed export isn't present (e.g. IRQ_SIM before built in).
Bridge: the vDSO (linux-vdso.so.1) is a kernel-provided .so mapped into
every process for fast syscalls.
task_struct has no "library" concept. ld.so mmap()s the .so, creating
file-backed VMAs in the process's mm:
Process A Process B
mm_A (own VMA list) mm_B (own VMA list)
libc VMA_A vm_start=0x7fAA.. libc VMA_B vm_start=0x7fBB.. (ASLR: may differ)
vm_file --\ vm_file --\
\----------- struct file / inode: /lib/libc.so ----------/
|
v
page cache: libc text pages (ONE physical copy)
/ \
A page tables B page tables (map to SAME phys pages)
- The VMA is per-process (own descriptor, own virtual address;
.somust be PIC so any load address works). The file + physical pages are shared. - Execution is demand paging: PC jumps into the text VMA -> page fault ->
find_vmasees a file-backed VMA -> read the page fromvm_filevia the page cache -> shared across all mappers. (Lab:ptwalk_labwalks these page tables.) - Symbols/relocations (PLT/GOT) are entirely userspace
ld.sowork; the kernel only ever sees file-backed memory mappings.
Threads vs processes here: threads share the mm -> the same VMA list and same
virtual addresses (one mapping). Processes have separate mms -> separate VMAs
that merely point their vm_file at the same file. Both share the .so's
physical pages; only threads share the mapping itself.
/proc/<pid>/maps is the human-readable dump of that process's VMA list — the
kernel walks mm->mmap and prints one line per vm_area_struct. This is the
same data vma_lab printed from inside the kernel. Real excerpt (cat /proc/self/maps):
address perms offset dev inode pathname
558c2143b000-558c2143f000 r-xp 00002000 00:3e 6760083 /usr/bin/cat <- app text
558c21442000-558c21443000 rw-p 00008000 00:3e 6760083 /usr/bin/cat <- app data
558c33d12000-558c33d33000 rw-p 00000000 00:00 0 [heap] <- anonymous
7fb40e5b5000-7fb40e5dd000 r--p 00000000 00:3e 7795433 .../libc.so.6 <- libc header
7fb40e5dd000-7fb40e772000 r-xp 00028000 00:3e 7795433 .../libc.so.6 <- libc TEXT (shared RO)
7fb40e772000-7fb40e7ca000 r--p 001bd000 00:3e 7795433 .../libc.so.6 <- libc rodata
7fb40e7ca000-7fb40e7cb000 ---p 00215000 00:3e 7795433 .../libc.so.6 <- guard (no access)
7fb40e7cf000-7fb40e7d1000 rw-p 00219000 00:3e 7795433 .../libc.so.6 <- libc data (COW)
Each column maps straight to a vm_area_struct field:
| maps column | vm_area_struct field | meaning |
|---|---|---|
start-end |
vm_start / vm_end
|
the VA range |
perms (rwx) |
vm_flags |
VM_READ/WRITE/EXEC |
perms 4th char p/s
|
vm_flags MAP_PRIVATE/SHARED |
p = COW private, s = shared |
offset |
vm_pgoff |
offset into the backing file |
dev + inode
|
vm_file's inode |
which file (0 = anonymous) |
pathname |
vm_file's path |
the file, or [heap]/[stack]
|
What this shows about the .so:
-
One library = several VMAs — libc appears as ~5 lines (header
r--p, TEXTr-xp, rodatar--p, a---pguard, datarw-p). Same inode (7795433) = the same file split by segment permissions. - The
r-xptext line is the shared, read-only, executable mapping — one physical copy across every process (same inode, page cache). Therw-pdata line is COW/private. -
inode 0/ no path = anonymous ([heap],[stack], or barerw-p) —vm_file == NULL. -
Compare two processes:
grep libc /proc/PID1/maps /proc/PID2/maps— same inode, possibly different start addresses (ASLR), proving per-process VMA + shared file.
Related views:
-
cat /proc/<pid>/smaps— per-VMA detail:Rss,Pss(proportional set size — splits shared pages across sharers),Shared_CleanvsPrivate_Dirty; this is where you see the.sotext counted as shared. -
pmap -x <pid>— the same as a table with sizes. - The kernel prints each line in
fs/proc/task_mmu.c:show_map_vma()by walkingmm->mmap;vma_labdid the equivalent walk directly.
userspace: fd = open("/dev/x"); write(fd, ...)
| (fd = index)
v
task_struct.files -> files_struct (the fd TABLE: array of struct file*)
fd[3] -> struct file (one open: offset, flags, f_op)
f_op -> file_operations (.open/.read/.write/.mmap)
|
v
kernel: YOUR handler runs (procthread_lab pt_write, mmap_lab .mmap)
struct file -> inode (the file) -> dentry (its name) [VFS]
-
fd is just an index into the process's
files_struct. - The fd table is the
filesrow of the fork/pthread table: threads share onefiles_struct; fork copies it (but entries point at the samestruct file, so inherited fds share the offset). - Every char-device lab plugs into this VFS path via the
file_operationsit registers.
Symptom -> tool:
| symptom | reach for |
|---|---|
| hang / deadlock |
gdb -> thread apply all bt, or gstack <pid>
|
| data race / corruption | ThreadSanitizer (-fsanitize=thread) or Valgrind Helgrind |
| crash / bad pointer | AddressSanitizer (-fsanitize=address) or Valgrind Memcheck, then gdb core
|
| which thread is blocked/slow |
perf sched, off-CPU (bpftrace), /proc/<pid>/task/<tid>/wchan
|
| what a process is doing |
strace -f (follows forks+threads), perf trace
|
| lock contention |
perf lock, mutrace, Valgrind DRD |
gdb — the workhorse
- threads:
info threads,thread N,thread apply all bt(every thread's stack — the #1 deadlock tool),break f:line thread N,set scheduler-locking on. - processes:
set follow-fork-mode child,set detach-on-fork off,info inferiors/inferior N,catch fork,attach <pid>.
Bug detectors (rebuild with a sanitizer; fast + precise)
-
TSan
-fsanitize=thread— data races (unsynchronised shared access). -
ASan
-fsanitize=address— overflows, use-after-free. -
Valgrind
--tool=helgrind(races + ABBA lock-order, no rebuild, slow),--tool=drd(races + contention), default = Memcheck (memory).
Live introspection (no rebuild)
-
gstack <pid>/pstack <pid>— one-shot dump of all thread stacks. -
ps -eLf,top -H,pstree -tp(threads in{braces}). -
/proc/<pid>/task/<tid>/— per-thread;wchan= what it's blocked in;statushasThreads:.
Tracing / profiling: strace -f (per-thread syscalls), perf record -g
(CPU), perf sched latency (scheduling delay), off-CPU with bpftrace.
Post-mortem: ulimit -c unlimited; gdb ./prog core -> thread apply all bt.
Learn two first: gdb's thread apply all bt (most hangs, minutes) and
ThreadSanitizer (the races behind "impossible" bugs). A TSan report is an
unprotected atomics_lab-style access; a deadlock in thread apply all bt is
the userspace lockdep_lab/ABBA cycle.
A process is a task_struct whose mm points to an mm_struct (page
tables + a list of vm_area_struct regions). fork copies the mm (own
address space, new tgid -> process); pthread_create shares it (same mm,
same tgid -> thread). A shared library is not referenced as a "library" —
ld.so mmaps it into file-backed VMAs (vm_file -> the .so), so each
process has its own mapping but they all fault onto the same page-cache pages.
Files reach kernel code through the VFS: an fd indexes files_struct -> a
struct file -> your file_operations. Userspace uses static (.a, copied
in) or dynamic (.so, shared) libs; the kernel uses built-in (obj-y) or
modules (.ko, linked against exported symbols) — no libc. The kernel
schedules threads and processes identically as tasks; every difference you
observe flows from whether the mm (and tgid) is shared or copied.