• Normally, this is at the end of the help guides. However, it's at the start of this project to emphasize some important points
  • First, this will be a challenging project. It will produce a lot of frustration and you will spend a lot of time debugging. However, you have a lot of resources at your disposal, such as TA hours, this handout, Edstem, and your fellow classmates!
  • Second, this looks like an intimidating project but you can do it! As long as you have the drive to finish this project, it'll get done
  • Lastly, it's important to start early. However, even if you're unable to keep up with the Weenix deadlines, you can still get this project done. It will just require a faster pace
• As always, make sure to read through all of the documentation and resources we provide you. Even if you're strapped for time, this is a marathon -- not a sprint

• VM Handout
• VM Help Slides
• Debugging Handout
• Files and Memory Wiki

• Virtual Memory is a memory management technique that provides each process with its own "virtual" address space. Each virtual address points to somewhere in main memory where the data is stored.
  • Pros
    • Memory isolation: processes cannot access each other's memory
    • Shared libraries: multiple processes can have virtual addresses pointing to shared code (e.g. printf), reducing memory usage
    • Giant address spaces: virtual memory allows a process to address more memory than physically possible
    • Less fragmentation: a process's data does not have to be contiguous in physical memory, allowing for more efficient usage of space
  • Cons
    • Performance: address translation takes multiple memory references (this is why we have the tlb)
    • Performance 2: pagefaults require a trap to the kernel, as well as possibly reading from disk
• In Weenix, we manipulate the virtual address space by using a data structure called a Virtual Memory map (vmmap_t), and we use data structures called memory objects (mobj_t) to represent things stored in this address space such as files and devices
• In this project, some of the things you will be implementing include (but not limited to): memory objects, shadow objects, the heap, pagefaults, and anonymous objects. Your kernel will be able to start managing user address spaces, run user-level code, and service system calls
• You will not have to implement page tables or the creation of executables. These are handled in elf.c and pagetable.c. However, it is likely that you will end up GDBing through these files at some point, so we recommend taking a look at these files

• In VFS, memory objects essentially represented vnodes (files, directories, devices), but in VM you will see that they can represent even more
  • A mobj can be one of four types: vnode (MOBJ_VNODE), shadow (MOBJ_SHADOW), anonymous (MOBJ_ANON), or file system (MOBJ_FS)
• Remember that a mobj is a collection of pframes (stored in kernel memory) to represent something (such as a file that you're modifying, a process's stack, the file system, etc.)
• Important note: if you use mobj_ref(), make sure the mobj_t you're calling it on is locked. Otherwise, you will encounter some strange behavior (changing DBG flags will cause non-deterministic behavior)

• Processes and Virtual Memory are closely intertwined. Processes are each assigned a section of virtual memory where they store their stack, heap, read/write data, and read-only code data
• When working on Procs, you might have noticed that Procs had fields p_brk, p_start_brk, and p_vmmap
  • p_brk and p_start_brk are how we represent the heap. p_start_brk denotes the address where the bottom of the heap is, and p_brk denotes the upper bound of the heap
  • p_vmmap is the mapping of virtual memory associated with the processes' virtual address space

• A vmmap_t is how we represent a process's virtual address space. It is essentially a list of vmarea_ts

• A vmarea_t is how we represent chunks of a process's virtual address space. For example, say we wanted to reserve a chunk of an address space for a file. We would use a vmarea_t to tell the process that this chunk of the address space is occupied
• A vmarea_t has a range from [vma_start, vma_end)
  • A vmarea_t could be [3,5) -- pages 3 and 4 (doesn't include 5!). The next vmarea_t can then be [5, 7)
  • Draw this out to get a better sense of what it looks like!
• A vmarea_t also has the following fields:
  • vma_off - the offset from the beginning of a mobj_t that a vmarea_t represents. We need this field so that we can mmap() sections of a file into processes' address spaces. For example, say pages 20-25 of the file. We need to keep track of this offset, which, in this case, would be 20 pages
  • vma_prot - the permissions that a process has with this vmarea_t. For example, a process may have read and write access to a file mapped to this area, or only read access because the area represents a process's code segment. See the macros section below for specific permissions
  • vma_flags - the type of mapping that this vmarea_t represents. See the macros section below for specific flags
  • vma_vmmap - the vmmap_t that the vmarea_t belongs to
  • vma_obj - the underlying mobj_t of the vmarea_t (each vmarea_t is "backed" by some type of mobj -- the data it's representing)
  • vma_plink - the link that this vmarea_t has on its vmmap_t's list

• Threads are related to virtual memory in that they store metadata such as the kernel stack and register state (particularly the instruction pointer, stack pointer, and frame pointer), which will be very important when you implement do_fork()

• Permissions (vma_prot) -- defined in mman.h
  • PROT_NONE - no access
  • PROT_READ - read access
  • PROT_WRITE - write access
  • PROT_EXEC - execution access
• Mapping flags (vma_flags) -- defined in mman.h
  • MAP_SHARED - vmarea_t is shared between multiple processes
  • MAP_PRIVATE - vmarea_t is not shared with anyone
  • MAP_ANON - the underlying mobj_t is an anonymous object
  • MAP_FIXED - map the file at an address and remove/shrink any existing vmareas that would overlap
• Directions (used in vmmap.c and mmap.c) -- defined in vmmap.h
  • VMMAP_DIR_HILO - search through a vmmap from high addresses to low addresses
  • VMMAP_DIR_LOHI - search through a vmmap from low addresses to high addresses

• The following macros will be important to refer to for pagefault.c and are defined in pagefault.h:
  • FAULT_WRITE - a write operation was attempted at vaddr
  • FAULT_EXEC - an execute operation was attempted at vaddr

• ADDR_TO_PN(vaddr) - takes in a virtual address and returns the virtual page number of the page containing it
• PN_TO_ADDR(pn) - converts a virtual page number to the virtual address
• PAGE_OFFSET(addr) - given the virtual address within a page, returns the page offset (relative to the starting address of the page)
  • Similar to S5_DATA_OFFSET from S5FS -- probably helpful for reading/writing
• PAGE_ALIGN_UP(addr) - given a virtual address of a page, returns the aligned address (an address like 4096 that's already aligned would not change)
• PAGE_ALIGN_DOWN(addr) - given a virtual address of a page, returns the aligned down address
• PAGE_ALIGNED(addr) - returns 1 if the virtual address is page aligned
• USER_MEM_LOW - userland lowest address
• USER_MEM_HIGH - userland highest address

The following are some of the functions you will be completing in vmmap.c:

// Add a new vmarea_t to a vmmap_t
void vmmap_insert(vmmap_t *map, vmarea_t *new_vma);

// Find the vmarea_t containing vfn (page number)
vmarea_t *vmmap_lookup(vmmap_t *map, size_t vfn);

// Find any contiguous range of available addresses (based on length npages). Remember to take advantage of the USER_MEM_HIGH and USER_MEM_LOW macros as well as the list iterator macros
ssize_t vmmap_find_range(vmmap_t *map, size_t npages, int dir);

// Copy a vmmap_t. Used in proc_create() for do_fork! When using mobj_ref() don't forget to lock the mobj that you're ref'ing!
vmmap_t *vmmap_clone(vmmap_t *map);

// Remove a range of virtual addresses or pages from a vmmap_t. This has lots of edge cases!
long vmmap_remove(vmmap_t *map, size_t lopage, size_t npages);

// Reads starting from a virtual memory address (in userland) into a buffer (in kernel space)
long vmmap_read(vmmap_t *map, const void *vaddr, void *buf, size_t count);

// Writes to a virtual memory address (in userland) from a buffer (in kernel space)
long vmmap_write(vmmap_t *map, void *vaddr, const void *buf, size_t count);

A general recommendation for these functions is to draw it out. Visualizing it should make writing these functions easier to do

• There are macros related to this section and can be found in the "Macros" section of the handout

• handle_pagefault() is a function call that, well, handles pagefaults. More specifically, a handle_pagefault() is the protocol that occurs when a program tries to access an area of memory that is not currently located on the system RAM. It might be easier to think of handle_pagefault() as a function that takes in a virtual address and associates it with a physical one
  • Another way to think about it is that the hardware cannot find a physical address mapping for a given virtual address (pagetable translation)
    • For example, user process Chris tries to access virtual address 0x1234... but that address isn't in his pagetable. This causes an interrupt that leads to handle_pagefault(). The job of this function is to add the mapping to Chris's pagetable (or explode if there's an error)
• When handle_pagefault() gets an address, it first checks to see if the address corresponds to a valid page. If it doesn't, then it segfaults.
• handle_pagefault() then checks to see if the page can be accessed in the manner specified by the flags
  • If the address was being written to, but the process doesn't have write privilege at the address then there should be a segfault. To check for valid privileges, refer to the vmarea's permission flags, which are given in the "Macros" section in this handout
• After it's confirmed that the address corresponds to a valid page and has valid permissions, handle_pagefault() then attempts to get a valid pageframe (pframe_t) corresponding to the address. Note that the address of this page is a physical address.
• In order to understand why this is all necessary it's important to discuss the tradeoffs of storing mappings from virtual addresses to physical addresses in two different data structures (which Weenix does)
  • Weenix stores these mappings in vmmap_t and in a set of page directories and pagetables -- the hardware only uses page directories and pagetables
  • When mmap is used, the page directories/tables and vmmap_t become out of sync because we only update the vmmap_t
  • The pagefault handler helps us bring the two data structures back in sync
  • When you return from the pagefault handler, the hardware will attempt to access the memory again (which can lead to infinite loops if your handler is incorrect)
  • Finally, make sure that the TLB cache gets flushed whenever you change the pagetables

• Represent memory regions that are not backed by anything
• Used primarily for the stack and heap
• These functions should be fairly straightforward -- anon_fill_pframe() should fill the frame with zeros

• These system calls are sys_write(), sys_read(), and sys_getdents(). Take a look at the other system calls that are already written for you. It will give you an idea of what these functions may look like. These system calls will use the do_x functions that you wrote during VFS!
• For sys_read() and sys_write() you will have to allocate a buffer using page_alloc_n() which takes in npages. You will have to figure out what npages should be based on the number of bytes
• Use the ERROR_OUT and ERROR_OUT_RET macros when necessary. Also, if you run into an error don't forget to free the buffer you allocated for before you return
• sys_getdents() is trickier than sys_read() and sys_write() but if you need help to see what this function is supposed to do, looking at do_getdent() will be helpful

• These functions check that a virtual address range has certain permissions
• It will be helpful to use vmmap_lookup() to get the specific vmarea corresponding to the vaddr. Don't forget that you have addr_perm() when you're writing range_perm()!

• These system calls are do_mmap(), do_munmap(), do_brk(), and do_fork() (which will get its own section)
  • do_mmap() is for inserting a new memory mapping. It's essentially the "front-end" for vmmap_map(). You'll do extensive error checking, then call vmmap_map(), which may call the underlying mobj's vn_ops for mmap
    • Make sure you complete s5fs_mmap() and zero_mmap(), which may be called through vmmap_map()
  • do_munmap() is for removing memory mappings. The "front-end" for vmmap_remove()
  • do_brk() is for manipulating the program's heap boundary. This may involve using vmmap_lookup(), vmmap_remove(), and vmmap_map()

• do_fork() is a function call that duplicates the current process. We'll call the new process the child process and the old process the parent process.
• At a high level, do_fork() might not seem that difficult, but there are lots of intricacies that happen when we clone a process. One of the key difficulties of do_fork() lies in what happens to all of the vmarea_ts of our parent process. Should we copy them or use the ones that are associated with the parent process? Take a look at kthread_clone() and vmmap_clone().
• What do we need to do in order to duplicate a process? Well, we need to create the child process, duplicate the kthread from the parent process, and set up the stack, registers, and heap of the child process. Let's break it down section by section:
• Creating the child process is relatively painless: what function creates a new process? Make sure to error check accordingly.
  • Duplicating kthreads requires that you dive into the kthread.c file and fill out kthread_clone()
    • Once you've cloned the kthread, don't forget about any other fields for this new kthread that may need to still be set
  • We need to set up three registers: rax (return value), rip (instruction pointer), and rsp (stack pointer).
  • For rax, just set it to 0.
  • For rip, remember that the instruction pointer points to the address of the current instruction we wish to execute. Check out the userland_entry() function in exec.c
  • For rsp, take a look at fork_setup_stack() in fork.c
  • Double check that you've set the rip, rsp, and pml4 for the kthread's kt_ctx
• Make sure to unmap the pagetable of the parent process and flush its TLB.
• Finally, make the child process and corresponding thread runnable using sched_make_runnable(), and return the pid of the child process.
• Remember to error check all functions that can fail, and to cleanup in the case of failure.

• There's a special struct that you should be aware of -- mobj_shadow_t
  • This helps us implement copy-on-write in Weenix. For more information, check out the "FAQs" section of this handout
  • mobj - the memory object itself (containing the page frames that the process modifies)
  • shadowed - the memory object that is being shadowed (can be of type shadow or not)
    • Use this to traverse the shadow chain
  • bottom_mobj - the reference to the memory object at the bottom of the shadow chain (not of type shadow)
• Shadow objects are used primarily for private mappings and fork()
  • Changes made to a particular page in a privately mapped region should not be visible to another process accessing that same region
  • For example, the stack is privately mapped
• shadow_get_pframe()
  • Reads: traverse the shadow object chain and get the page from the first object that contains that page number
    • Will get the page from the bottom_mobj if not found in any of the shadow objects in the chain
  • Writes: use mobj_default_get_pframe() which trickles down to shadow_fill_pframe()
    • Will initialize a pageframe in the “top level” shadow object before calling shadow_fill_pframe()
• shadow_fill_pframe()
  • Fill the contents of a pframe by traversing the shadow object chain
    • And then once found, the content of the page frame is copied into the specified page frame
• shadow_collapse()
  • Collapses the shadow chain. Basically, you'll traverse down the shadow chain using the shadowed field. If the part of the chain that you're at is no longer a MOBJ_SHADOW that means you've reached the underlying mobj, bottom_mobj
  • If the refcount of the current node in the chain is at 1, you should "collapse it" which means you'll migrate the pframes upwards
  • Let's imagine you have a pointer at A and a pointer at B. If B has a refcount of 1, then you should add B's pframes that A doesn't already have to A and then make the pointer to B change to a pointer to C. Therefore, instead of A->B->C, it's now A->C. That means you'll have to deal with reference counts. If you don't have to destroy B, your pointer to A changes to a pointer to B, and your pointer to C changes to a pointer to D (or whatever is being shadowed by C). It may be that C is the last part of the chain (thus it's the bottom_mobj)
  • Locking and refcounts are going to be difficult to manage in this function, but as long as you understand it conceptually and draw it out, it shouldn't be too difficult!

• Pagetables (and pagetable translation)
  • However, you will be using functions that manipulate pagetables. These functions add mappings to the pagetables and flush the TLB (pagetable cache) whenever pagetables are changed
• Virtual memory mapping (the memory mapping for files and processes will be done for you but it will be using your code to do so -- you won't have to manually map out any virtual memory)
• The majority of system calls (though there are some for you to fill in)
• Execution of userland programs (you will be testing the programs in user/usr) -- if your code doesn't work though you won't be able to enter userland correctly!

• Operations on vmmaps/vmareas
  • Use vmtest.c and the autograder tests to confirm that this works correctly
• mobj_t operations for Anonymous Objects and Shadow Objects
• System calls -- do_mmap(), do_munmap(), do_brk(), do_fork(), sys_read(), sys_write(), sys_getdents()
• Access checks (checking if virtual address has certain permissions)
• Handling a pagefault
• Various loose-ends updating and completion (functions in a few different files)

You will be directly modifying the following files:

• access.c
  • Functions for checking permissions at a vaddr or range of vaddrs
• anon.c
  • Implement Anonymous Objects
• brk.c
  • Implement the brk syscall (do_brk), which modifies the heap's ending address
• fork.c
  • Implement the fork syscall (do_fork())
• mmap.c
  • Implement the mmap syscall, which adds a mapping to a process's address space (vmmap) and munmap syscall (unmaps)
• pagefault.c
  • Handle a user pagefault by updating the pagetable with the vmmap
• shadow.c
  • Implement Shadow Objects
• vmmap.c
  • Functions dealing with vmmap and vmarea manipulation
• kthread.c
  • Update kthread_create() for VM (if necessary)
  • Implement kthread_clone()
• proc.c
  • Update proc_create() for VM
    • p_pml4, p_vmmap and p_vmmap->vmm_proc
  • Update proc_cleanup() for VM
    • Clean-up the p_vmmap
• s5fs.c
  • Implement s5fs_mmap()
• memdevs.c
  • Implement zero_mmap()
• vnode_specials.c
  • Implement chardev_file_mmap(), chardev_file_fill_pframe(), chardev_file_flush_pframe() It will be helpful to refer to the following files:
• vmmap.h
  • Struct definitions for vmmap_t and vmarea_t
• syscall.h
  • Struct definitions for read_args_t, write_args_t and getdents_args_t
• exec.c
  • See how userland actually gets entered in code
• pagetable.c
  • See how pagetables are implemented

There will be several stages of testing throughout this project. You can think of three checkpoints:

• Getting vmmaps/vmareas working correctly
• Trying to enter userland
• Running the userland shell

1. Getting vmmaps/vmareas working correctly
   • A first good goal is to get your vmmap/vmarea code correct
   • Use vmtest.c and the vmmap tests
     • You can run the vmmap tests locally in your vmtest.c file. You'd use it the same way you've done in previous projects with kmain.c modifications
   • The vmmap tests require that you have implemented the other parts of the project too -- not just your vmmap code, but this is a good place to start testing
2. Trying to enter userland
   • Now that your autograder tests are passing, you're probably trying to enter userland
   • The next step here is to test segfault and hello
   • In your initproc_run() you want to use kernel_execve to run segfault and hello. You can put the code below between the VFS initialization and the kshells being created
   • Finally, to debug this you want to use GDB: new-userland hello or new-userland segfault depending on which one you're debugging
     • Put a breakpoint on do_exit()
       • If segfault is working correctly then the address causing the pagefault is 0x0 and the previous frame in the backtrace should be handle_pagefault().
       • If hello is working correctly then you should see "Hello, World!" being printed on your kshell
   • Once you have segfault and hello working, in order to try entering userland you should use kernel_execve("/sbin/init", argvec, envvec)

char *argv[2] = {"hello", NULL}; 
char *envp[1] = {NULL}; 
kernel_execve("/usr/bin/hello", argv, envp);

char *argv[2] = {"segfault", NULL}; 
char *envp[1] = {NULL}; 
kernel_execve("/usr/bin/segfault", argv, envp);

char *argv[2] = {"init", NULL}; 
char *envp[1] = {NULL}; 
kernel_execve("/sbin/init", argvec, envvec)

3. If you've made it this far that means you've successfully entered userland!
   • There are a series of tests that you can run in your user shell which are on the VM handout
   • The commands parallel and repeat are helpful for testing purposes
   • Try using different shells concurrently
   • Ultimately, you should be able to pass these tests with DYNAMIC enabled
     • Some S5FS tests will fail if DYNAMIC is disabled
   • Come up with creative ways to break your OS

• As always, the debugging handout is super helpful for debugging techniques. It will be particularly helpful to look at debugging pagefaults and userland
  • new-userland <test> will be something you refer to a lot
• Use KASSERTs, GDB, and DBG print statements

These are FAQs compiled by an OS HTA and some questions from Files and Memory Wiki

• Do not make your page numbers uint32_t, instead make them size_t
• Where should I set the vmmap's vmm_proc field?
  • You can initialize it in proc_create() after calling vmmap_clone()!
• What should ret in do_mmap() be set to?
  • You should set ret to be the address of the start of the vmarea. (i.e. PN_TO_ADRR(new_vma->vma_start) rather than the address of the vmarea struct.)
• I’m struggling a bit with the logic of vmmap_find_range(). How do I find the page ranges?
  • Think about using the list iterate macro to iterate through the vmareas in the vmmap.
    • Also, when looking at the lower and upper page boundaries, make sure to convert USER_MEM_LOW and USER_MEM_HIGH to be in terms of page numbers!
• In do_mmap() we are supposed to convert the len into a recognizable pagenum: is it sufficient to just pass in ADDR_TO_PN(len) into our functions?
  • ADDR_TO_PN simply right shifts 12 bits. You will need to figure out what the starting page is, what the ending page is, and then how many pages are spanned
• Considering the case where a vmmap is empty (has no vmareas yet), where should vmmap_find_range() return when given VMMAP_DIR_HILO? How do we determine the highest page number?
  • USER_MEM_HIGH is a constant giving the highest user-space virtual address. So if nothing has been allocated yet, that's where allocations would start for the VMMAP_DIR_HILO case.
• How does MAP_FIXED work (do_mmap() and vmmap_map()?
  • You start the mapping from the lopage if MAP_FIXED is set. Rather than looking for any free space that is large enough to put your mapping into, you would clear out an area starting at lopage for the mapping. Therefore, anything overlapping with that area should shrink or be removed
• What does it mean for a file to be mapped in the address space?
  • In order to do this, the user process will make a call to mmap, which is a system call. This will eventually propagate to call to vmmap_map().
  • mmap will return a pointer to the start of this region, thus when a process accesses an address in this region, they will be accessing (i.e. reading and writing to) the file directly that is mapped to this region. In other words, we do not have to make read/write system calls in order to read or write to the file.
  • The visibility of the changes made to the file and whether they will be flushed / discarded depends on the flags passed in to mmap. If a file is mapped privately, then the changes will not be visible to other processes, which means that if two processes map the same file, their changes would be specific and private to the process, and will not be flushed to the underlying file stored on disk.

char* mapped_region = mmap(....some file....); // mapped some file starting from page 2 (zero indexing)
*mapped_region = 'a'; // this would modify the first byte of page 2 of the file 

• When are the vmmap operations being called? When are vmmap_read and vmmap_write being called?
  • These operations are being used when data needs to be copied from/to user space. Doing so this way (i.e. accessing the memory through the corresponding memory objects for each vmarea) would avoid causing pagefaults for userland addresses when in the kernel. This is why in system calls, you must used copy_from_user and copy_to_user.

• In handle_pagefault, how do I specify the the pframe in mobj_get_pframe?
  • The page you are looking up is within a vm area, and there is some mobj that backs that vm area. So, find the number of pages from the start of the mobj to the vma, and the number of pages from the start of the vma to the page you're looking for.

• In general for syscalls, why do we need to use copy_from_user() rather than directly accessing the arguments from user space?
  • copy_from_user() has additional checks to verify that the address being accessed is in fact valid (i.e. has the correct permissions and is in user space)
  • Another reason that copy_from_user() / copy_to_user() are used is so that pagefaults do not occur when accessing user addresses, and vmmap_write() and vmmap_read() will ensure that those addresses have been mapped
• For syscalls, why do we need to use page_alloc_n() rather than just allocating a local buffer on the stack?
  • The reason that we don't want to allocate a local buffer on the stack is because a user could be requesting a very large amount of bytes and we don't want to potentially run out of space on the stack
  • As a side note, the reason we use page_alloc_n() rather than kmalloc(), is that with kmalloc(), it's used to allocate space for kernel data structures of a fixed size and there's a size limit to how much can be allocated (and there shouldn't be limits to how much a user could request to read)
• I've been trying to run /sbin/init, but I'm unable to see my 'three terminals' as mentioned in the handout.
  • Since this involved processes terminating early, this could be an issue with do_waitpid() — setting a breakpoint to that might help
  • If the issue doesn’t seem to be with do_waitpid(), you might want to check whether the processes for the three terminals are accessing an invalid memory location which doesn’t contain any vma causing them to do_exit()
    • Make sure to unmap the current proc's pagetable entries, not the the new proc's pagetable entries
    • Make sure you are increasing the refcounts correctly in shadow_create()
    • Make sure you specify forwrite correctly when mobj_get_pframe() is called from handle_pagefault() — it should be set if cause & FAULT_WRITE is true
  • Setting breakpoints on do_brk() and do_mmap() may also help
  • We would also recommend enabling syscall debugging statements in debug.c

• Why do we not set new_thread->kt_ctx.c_rbp in do_fork?
  • The new thread starts up in the first statement of userland_entry() and never returns from userland_entry(). Thus rbp will be established there, with no need to restore its previous value. The registers (from user mode) of the thread that called fork (they're referred to via the regs argument of do_fork()) are copied to the new thread's kernel stack (these registers include rbp) and restored just prior to entering user mode
• Does a forked child process share the same file descriptors with its parent process?
  • If process A opens file X, and forks to produce process B, and then A writes to file X, f_pos increases for B as well
  • When fork() is called, B receives a copy of A's file descriptors, which refer to the same file. So when A modifies the file, B will see those changes, and vice versa

• I'm not completely sure about the locking in shadow_fill_pframe()
  • Remember that you should start iterating/the search from o->shadowed
  • o is locked on entry to the function
  • Make sure that you lock the memory object that you are passing in to mobj_find/get_pframe. When you call mobj_find/get_pframe on the shadow object that you are searching, if the page frame is found, it will return the page frame locked. Make sure to unlock it once you are done copying data from it!
• I'm trying to start shadow_create() and am slightly confused about the terminology used here. When we say "create a shadow object" do we mean a mobj_t that is of type shadow, or do we mean a mobj_shadow_t ? I'm slightly confused as to what the distinction is and what exactly we need to be doing in this function.
  • All mobj_t structs of type MOBJ_SHADOW live inside their own mobj_shadow_t struct, which is why, when given a mobj_t of type SHADOW, you can use the MOBJ_TO_SO macro (defined in shadow.c) to get the corresponding mobj_shadow_t (and when you have a mobj_shadow_t, you can access its mobj_t as the field named mobj).
  • So in shadow_create(), you should use slab allocators to allocate sizeof(mobj_shadow_t) many bytes. Then, within that new mobj_shadow_t, the field named mobj is a new mobj_t, you want to be sure mobj_init() that
• What is the difference between a mobj_t and shadow_mobj_t?
  • mobj_t type is the same type you've worked with in VFS and S5FS - it's for any kind of memory object, and might represent data relevant to a file, a stack, a shadow object (for copy-on-write pages), or whatever else.
  • A mobj_shadow_t has extra fields that are relevant only to shadow objects: the shadowed field and the bottom_mobj field, which are critical for implementing the functions in shadow.c. It would be a waste of space to include those fields for the mobj_t instances that are not shadow objects, so Weenix's solution is to use the mobj_shadow_t to provide those extra fields as needed.

• How do the addresses that the memory objects use relate to userland addresses? Are the addresses that the kernel access virtual addresses? For instance, is pf->pf_addr a virtual or physical address?
  • pf_addr is a virtual address. This is because the processor is unaware of whether the addresses that it's operating on are virtual or the actual physical address. Weenix uses a memory management unit (MMU) and has enabled paging, which means that all addresses that the processor accesses are translated behind the scenes using the TLB (and pagetables). As you might notice, we never get a page fault for addresses in the kernel, and this is because all kernel addresses are already mapped into the page tables initially (and have corresponding physical addresses in RAM).
  • After handling the page fault, you effectively have two different virtual addresses that correspond to the same physical address. The two virtual addresses are the address of pf_addr and the user accessed virtual address. This is so that the kernel has a convenient means of referring to all its memory, but is also able to have some of memory be mapped by user processes.
• How are virtual page numbers related to page frames?
  • Virtual page numbers are virtual addresses that are bit shifted to the right by 12. Each virtual page number corresponds to a region of 4096 addresses. This page number is relative to the beginning of the address space.
  • Page frame numbers, depending on what the backing store is, can be thought of as an "offset" into the memory object. This page frame number is relative to the beginning of the mapped object.

• I'm failing the assertion regarding having an empty list of mutexes in check_curthr_cancelled()
  • Double check your implementation of shadow_fill_pframe()
  • Make sure that you are unlocking newly created shadow objects in vmmap_clone()
  • If vmmap_map() calls zero_mmap() it will create an anonymous object, which will be locked on return — make sure you unlock this anonymous object at some point (i.e., special-casing it in vmmap_map())
• What functions and what files should I have implemented before running usr/bin/segfualt and usr/bin/hello?
  • pagefault.c
  • access.c
  • vmmap.c
  • syscall.c
  • anon.c
  • If you have code that are handling shadow objects and privately mapped regions, you can comment it out until you start running sbin/init.
• How do I verify that usr/bin/segfault and usr/bin/hello are functioning as expected?
  • See "Testing" section of this handout
• usr/bin/segfault and usr/bin/hello are working as expected, but I'm getting refcount issues:
  • Make sure that you have modified your proc_cleanup() code to call vmmap_destroy() before the call to initproc_finish()
  • And that you have commented out your code that creates the kshells in initproc_run()
• How do I run a userland program using kernel_execve()?
  • See "Testing" section of this handout
• The tests for S5FS fail once VM is enabled
  • This is to be expected! This is because the statically linked binaries take up too much space on the disk (this should change once DYNAMIC is enabled)

• Double check that VM = 1 is set in Config.mk and run make clean all
• Read. Read through all of the documentation we give you. It will save you a lot of time if you spend some time reading through things and understanding how S5FS works. You won't get a 100% understanding on your first read, but it's good to have some baseline understanding of what you're implementing
• This is a big project, so we've provided a roadmap you can use:
  • Start with vmmap.c. Make all mappings shared for now (or comment out code that uses shadow functions)
  • Implement the pagefault handler in pagefault.c
  • Implement anonymous objects in anon.c
  • Fill out the system calls in syscall.c
  • Implement memory checks in access.c
  • Test your implementation so far with the provided vmmap tests
  • At this point, try running segfault. If that works, run hello. We do not recommend moving on if hello does not work.
  • Fill out brk.c and mmap.c
  • Implement fork.c and shadow objects in shadow.c
  • Fill out the functions from other projects that VM relies on
  • Try getting into userland (sbin/init). This is your next big hurdle
  • Once you've successfully entered userland, try the userland tests in your user shell. Once you are confident in your implementation of Weenix, set DYNAMIC = 1 in your Makefile and run the tests again. You want to make sure the tests work even with Dynamic on
  • If you've completed everything -- congratulations! You've finished Weenix!
• You're in the last stretch before completing OS. It has been a long semester, but you can do it! Make sure you finish strong. We're here to support you. Good luck :)