CSCI4150-File-System-Server

RequirementRequirement

This project is intended to integrate many aspects of OS design and implementation, from scheduling, to synchronization, to memory management, and file systems. You are to implement this in the xv6 OS (I’ll provide a repo link via Piazza). You will implement microkernel services on top of xv6 which is a monolithic OS! This is a large and hard project and is scoped around three team members. If you choose a smaller team, you’re still responsible for the entire project, so I suggest you form your team soon. The most important thing you can do to ensure you finish on time is to ​start early. Like now. To the keyboard!

Final goal: OS Service to do File-system Checkpoint-Restore

The final goal of the project is to implement a File-System Server (FSS) – implemented as a normal process – that communicates with the other processes in the system (called “clients), receiving requests for file system services (e.g. similar to open, read, write, close, mkdir, unlink), and it will make those calls on behalf of the requesting process. However, it will also be smart enough to enable the “undo” of any changes made to the file system by those processes. It will do this by saving the contents of files and directories to a “checkpoint/“ subdirectory when modifications are made. To restore this checkpoint, a simple program can simply copy everything incheckpoint/into the normal file system. The communication with the FSS is through shared memory regions coordinated with mutexes and condition variables (which xv6 does not already provide).

Specification and Implementation Plan

There are three main modules to this project.

1. The FSS which uses the normal kernel API to provide access to the file-system, and also to perform the operations to save files/directories when they are modified for checkpointing. This requires zero kernel programming.

2. The shared memory support between the FSS, and the processes that are using the FSS’s checkpointing support. This shared memory region is used to pass both the request being made (i.e. which type of call is it, read, open, etc…), and the corresponding data that is required for that call (the path and flags for open, the buffer and its length for read, etc…). This will require kernel hacking to make a shared memory region between the client and the FSS.

3. The synchronization code necessary to provide synchronization on that shared buffer. This will include both mutexes, and the ability to block waiting for an event (i.e. a new client request) – a function normally provided by condition variables. This will require kernel hacking to implement mutexes for user-level, and to add logic for condition variables. The FSS process should always execute with high priority as it is providing a service to other processes in the system.

There are varying levels of support for each of these three components, and you should be able to develop each relatively independently, so if one person does not meet their commitment, the others should be able to still make progress. Level 1 is intended to be very basic functionality. Leveling up from there strengthens your project. Only a project at project level 5 will receive full credit. Each level assumes the functionality from the previous levels (i.e. you can’t get level 2 without achieving level 1).

A note on group work and collaboration: I designed these modules to be independent, but that does not mean that they are of equal difficulty. Do not plan on simply assigning one to each team member, and when one of the modules is complete assume that team member is “done”. After a module is done, you must help the other members on the other modules. Each team member must stay up-to-date with each other teammate. You should use github to coordinate your code modifications. In fact, I highly​ suggest that once each of the implementations get to Level 1, that you start integrating them together. Once integrated, you can keep leveling up in each module.

Module #1: File-System Server

The FSS uses the normal kernel API and is a normal process. It uses IPC to talk to clients. Those clients make requests to it to access the file system on their behalf. This is not very useful on its own, but the FSS is smart because it provides a checkpoint and restore functionality. This means that when the FSS starts, it begins recording all files and directories that are modified by clients. It will record these files/directories in the /checkpoint/ directory. So for example, if /hw.txt exists, and you wrote a program to write to that file (via IPC with the FSS), then the FSS would copy the /hw.txt file to /checkpoint/hw.txt, and then perform the write on the client’s behalf. If you write a program to remove a directory, /foo/, then the FSS would add /checkpoint/foo/ before removing the directory for the client. A separate program called restore can be executed to restore all files and directories in the /checkpoint/ directory into their original location. This will undo all edits made in the mean-time. You do not have to track files and directories that are created and remove them upon restore. This module requires no kernel programming.

You’ll first need to understand how the client and the FSS communicate via IPC. Each of the file-system system calls need a corresponding FSS operation, prepended with fss_. The client will call these operations instead of the normal system calls. Thus, you will need to provide your implementations for fss_read, fss_write, fss_open, fss_close, fss_mkdir, fss_unlink. Note that open and mkdir are used to create files and directories, respectively, and unlink isused to remove files and directories. These functions will be used to pass through IPC to the FSS, which function is being performed (e.g. you could pass it as a string), and the arguments being passed to the function. For example, you could define a structure:

struct fss_request {
char operation[7];
int arg;
char data[1024];
};

Where the operation field contains the operation (“mkdir”, “close”, etc…), arg contains any numerical argument passed to the function, and data contains any actual data that is meant to be passed. Once you fill out this structure, you can pass it over IPC to the FSS, and it can read the structure and convert the request into its own logic for making real file-system system calls to the kernel. It is cleaner to use an enumerated type for the operation.

A difficult part of this arrangement is that the FSS must return the return value from the kernel back to the client, and in the case of read, we must return the data that was read. So you’ll likely want a fss_response structure that holds that response as well (e.g. an couple of ints, and a array for the data). It can send this structure via IPC back to the client that made the request

Module #2: Shared Memory

Shared memory is often used for IPC between processes. It is more efficient than message passing (e.g. pipes) as you can often have fewer memcpy operations (e.g. zero or one vs. two). This module will add shared memory to your project. Xv6 does not support shared memory, so this will require understanding the xv6 mechanisms for mapping memory into processes, and how that interacts with calls to sbrk (extending the amount of memory accessible by a process), and fork/exec/exit.

Module #3: Synchronization and Scheduling

Module 2 provides shared memory between processes. Now that we have shared memory, we must synchronize access to it! We want to provide a means for passing requests to the FSS, and for synchronizing the return value being passed to the client. Xv6 does not provide any memory sharing between user-level processes, so you’re job is to implement a set of mutex system calls, and to enable processes to block waiting for requests/responses, you’ll also provide condition variable system calls.

Overall Project

The levels for the overall project are:

• Level 0: Level 1 in one module.
• Level 1: Level 1 at least two modules.
• Level 2: Level 1 in three modules, and level 2 in at least two. Two of the modules must be integrated together.
• Level 3: All three modules must be integrated together.
• Level 4: Highest level - 1 in each module.
• Level 5: Highest level in all modules.