2012-12-14

The initial design inspiration for GotoLinux file hierarchy is the basic layout of GoboLinux. So we will start there. The intent of this document is to review the design to ensure that it fully fits our design goals and the needs of the times, and modify as necessary. A good example of the later, modern system are more likely to have SSDs to serve up non-volatile files quickly, while still relying on an HDD for long term volatile storage. Systems like this need to be taken into account.

The current GoboLinux file hierarchy is as follows:

  /Depot/
  /Files/
    Codecs/
    Documentation/
    Fonts/
    Plugins/
    Compile/
  /Mount
  /Programs
  /System/
    Links/
      Environment/
      Executables/
      Headers/
      Libraries/
      Manuals/
        info
        man{1-9}/
      Shared/
      Tasks/
    Settings/
      BootScripts/
    Variable/
      tmp/
    Kernel/
      Boot/
      Devices/
      Modules/
      Objects/
      Status/
  /Users

I point out that there seems to be no equivalent to the FHS srv/ folder. And I wonder where that should exactly fit into the this design. For now I will simply assume an additional /Serve directory.

For use with an SSD, it seems most prudent to divide the hierarchy up between rarely volatile, volatile and always volatile. The later can be further divided into persistent and temporary. Mapping the GoboLinux layout to these divisions we get partition tables as follows:

  RAM (tmpfs)
    1 /System/Variable/tmp

  SSD
    1 /Kernel/Boot
    2 /

  HDD
    1 /Users, /Serve and /Depot
    2 /System/Variable
    3 swap

Right off that bat we have an issue with primary HHD partition, since three different top-level directories are all vying for the same mount point. It would be inelegant to create a separate fixed size partitions for each of these. Clearly we need to devise a new top-level directory under which the rest of these reside. Something like:

  /Data/
    Users/
    Serve/
    Depot/

You may have also wondered about the partition for /Kernel/Boot. While not necessary, I have often heard that it is a wise precaution to have these files on their own partition. Which brings us to another point. Notice that three partitions are mounted sub-directories: /Kernel/Boot, /System/Variable and /System/Variable/tmp. Perhaps that's an indication that these locations should actually be "topside". Might we instead move them up to make /Boot and /Variable? Mounting sub-directories is perfectly doable though, so it isn't necessary go this far. But it's worth some consideration.

One detraction I have with this layout is that /Files straddles the fence between volatile and non-volatile. Clearly the contents are generally associated to the contents of Programs and they won't often change. But multiple programs may share these resources which is why they are separated. So most of the contents are non-volatile, yet Compile can be rather volatile as compilation of programs often creates many temporary files. So Compile, at least, should be moved, but we might instead opt to move the entire Files tree to Data.

The intention of the Media directory is to hold media files such as movies and music --files that are generally not personal but rather purchased content. It is a nice division to have in the organization of ones files but there are at least two problems with it. First, the name "media" isn't clear enough as it does not immediately convey the intent of its use. Second, it may be too arbitrary a division since there is no way to enforce the type of files the location contains. On the first count, a more suitable name might be Content or Library. On the second, we might just remove this directory altogether and just have Files.

The original intention of the Depot directory is to act as hold for unspecified common files. It seemed liked a good idea at first for every user to have such a location for unspecified personal files, but in retrospect it is probably not necessary. A user can have a sub-directory in Files to serve this purpose just as easily, as there is no real need to have a designated location for such files. Common files can instead reside in Files/Share. However we might wish to consider where to keep specific shared files such as fonts. Is Files/Share/Fonts the appropriate place? Or should there be a distinction between the location of such specific files and unspecified files? It seems this is the reason for previously discussed Media directory after all. Where GoboLinux has Files and Depot, GotoLinux has Media and Files --albeit we will likely rename Media in the final layout.

Now lets take a step back and reconsider the standard file hierarchy system (FHS) afresh by asking a few questions and making some derivative observations.

1. Are programs really different than any other types of file(s)? Generally we think of programs being installed at the system level under /usr and shared by all users. Whereas most other files tend to be kept in home, belonging to a particular user. But anyone who has used a Linux system extensively and at a deep level knows that programs can and do get installed under a user's home directory. These programs, like other home files are essentially private to the user.

2. Why are most media files private anyway? If you think about it, many files we keep in out home directories may just as well be shared with the rest of the users of the system. We don't often consider this because most home computers tend to have a single dominate user, or multiple users just share a single account. This occurs more often on corporate servers, but the in these case special methods of file sharing are generally employed via the network, such as FTP or SCM systems. Until recently these approaches even lacked standards with regard to the FHS, until srv/ was finally introduced.

Taking these idea into consideration, we might reorganize our file hierarchy accordingly:

  /System/
    Kernel/
    Mount/
    Settings/
    Variable/
    Temporary/
  /Share/
    Desktop/
    Files/
    Media/
    Programs/
    Settings/
    Variable/
    Temporary/
  /Users/
    foo/
      Desktop/
      Files/
      Media/
      Programs/
      Settings/
      Variable/
      Temporary/

In this layout, Share is available to everyone in the system, and has the same layout as each user's directory. Media and Files are separate for entertainment content, such a movies and music, as opposed to personal files. Programs, then, no longer reside in the root, but part of shared content, and users can have personal programs just as easily.

A noticeable problem with this layout is that it prevents all the Variable and Temporary directories from mounting to a single mount point. one solution to this is to invert the design. Each sub-directory (e.g. Files, Programs, etc.), could be moved to the root and each would have the same sub-directory structure of Share and Users. That is interesting because it largely reflects the root layout of the current FHS (less the esoteric names):

  usr  -> Programs
  etc  -> Settings
  home -> Files
  var  -> Variable
  tmp  -> Temporary

Not that these are precisely the same, but they do demonstrate a similar structure.

Another advantage is that it prevents end-users from "mucking-up" their home directories and not keeping to this structure.

We may be on to something here. In the next post we will consider this layout in more detail to see if it meets the project's goals.

I am not one to usually support opinionated designs. Too often the hit walls and then become hobbled. But in the case of the a file hierarchy system, there obviously has to be some order. Otherwise no program or user could reliably interact with one another.

Per the last blog post on rethinking the file hierarchy, we have come somewhat full circle and arrived at the following layout:

  /Depot/
  /Desktop/
  /Files/
  /Media/
  /Programs/
  /Settings/
  /System/
  /Variable/
  /Temporary/

And each of these, with the exception of System, has the same substructure of Share and Users. This is a great and innovative start. Let's now consider some additional aspects of what a good file hierarchy must entail.

One important aspect of file hierarchy design that this doesn't take into account is partitioning. It is important to be able to partition a system optimally, especially today when dual storage SSD+HDD style systems are becoming common. But rather then try to arbitrarily predetermine partition groupings of these root directories, we can offer a configurable solution by adding an optional path call Partitions. With Partitions a systems administrator can partition the root paths as needed.

  /Partitions/
    SSD/
      Programs/
      System/
    HDD/
      Depot/
      Desktop/
      Files/
      Media/
      Settings/
      Variable/
    RAM/
      Temporary/

Then the root directories a simply symbolic links to the physical locations:

  /Depot     ->  /Partitions/HDD/Depot
  /Desktop   ->  /Partitions/HDD/Desktop
  /Files     ->  /Partitions/HDD/Files
  /Media     ->  /Partitions/HDD/Media
  /Programs  ->  /Partitions/SSD/Programs
  /Settings  ->  /Partitions/HDD/Settings
  /System    ->  /Partitions/SSD/System
  /Variable  ->  /Partitions/HDD/Variable
  /Temporary ->  /Partitions/RAM/Temporary

Under each of our root directories is a standard division between Share and Users. The first is shared by all, the later contains a directory for each individual user. But what if need something in between? What if a certain group of users need to share files, of which other users should not be privy? For this we will introduce a new section called Groups. e.g.

  /Files/
    Share/
    Users/
    Groups/      

Groups, like Users contains arbitrarily named paths, each representing some "walled-off" set of files. But these files do not belong to a specific user, but are intended to be shared by specific sets of users. A good use case for this are company files.

One variance with the original GoboLinux hierarchy that our design seems to be at odds with is the location of Variable at root instead of under System. I think the missing consideration here is that Variable can hold both user saved data, such as game scores, and more ephemeral system related files such as logs. There are very different things and it is tempting to separate them. Indeed where under Variable will these system logs reside since currently its sub-directories are Share and Users. When it comes to things like system logs, it makes some sense for them to belong the administrators files. So under Users/root might be a good choice. But that may be too restrictive.

We might instead use a group location like Groups/admin, then any user who belongs to the admin group can access these system variables. Another possibility is to add a System location along side Share, Users and Groups, which gives a good specific location for variable system files to reside. Of course, this System location wouldn't make as much sense for locations like Desktop or Media, but it could be useful under Programs as a place for the Kernel image and other system start-up programs. This would be a clear place for Systems/Boot to link, for instance. I am not sure which is best. Or perhaps both approaches have a place. Something for further consideration.

We haven't yet mentioned what is probably the biggest draw back of this design --the fact that all the different directories are spread out and not under a single users directory. This means it's a pain to navigate for the average user because they'd have to move all the way to the root directory and back down again to their own directory when switching between different primary locations. For example, when navigating from Files to Media.

We have a solution for this problem however in the form of Desktop shortcuts. In fact that is really the whole point of the Desktop --to provide shortcuts to all the things the user would be interested. Right off the bat we know what some of those are so by default a users Desktop folder would look like this:

  \Desktop\
    Users\
      bob\
        Depot     -> \Depot\Users\bob\
        Files     -> \Files\Users\bob\
        Media     -> \Media\Users\bob\
        Settings  -> \Settings\Users\bob\

Links for the other locations (Programs, Variable and Temporary) could also be added, but they would probably only be of interest to power users.

Thus far we in our design progression, Programs has a layout of:

  /Programs/
    Groups/
    Share/
    Users/

But what is further down? If we were to follow GoboLinux's design we would find <Program>/<Version> entries below these. But I want to take it a bit further to prevent any files from being scattered elsewhere that really belong somewhere under programs.

One of the latest ideas GoboLinux was developing before development froze-up was System/Aliens. Many programming languages now have there own packaging system to facilitate ease of use. It is important that these system can be properly supported, so we too will introduce such a location, but under Programs instead of System.

In addition, some programs control installation of their own plug-ins. To facilitate these we also need to add a location called Plugins.

So Programs might actually look like this, noting that the Standard directory is the location of packages managed by the operating system itself.

    Programs/
      Groups/
        bobinc/
          Alien/
            LuaRocks/
            RubyGems/
          Standard/
          Plugins/ 
      Share/
        Alien/
          LuaRocks/
          RubyGems/
        Standard/
        Plugins/ 
      Users/
        bob/
          Alien/
            LuaRocks/
            RubyGems/
          Standard/
          Plugins/ 

However, this layout looks inconsistent with the layout of our root directory structure. If we were following the idea of types higher-up and groups and users at the bottom, then our layout would be inverted.

    Programs/
      Aliens/
        LuaRocks/
          Groups/
          Share/
          Users/
        RubyGems/
          Groups/
          Share/
          Users/
      Standard/
        Groups/
        Share/
        Users/  
      Plugins/ 
        Groups/
        Share/
        Users/

At this point I am not sure which is the best approach, or if some hybrid of the two makes sense. I will point out that Aliens could be removed and just have it's sub-directories at the same level as Standard and Plugins. We will also have to consider how other programs will handle this layout. Can, for example, the Chrome web browser be configured to handle these per-user locations?

We will have to revisit this question later to finalize the design. Perhaps further investigation in other areas will shine some light on this.

Another distinction made in our current file hierarchy is to divide users and groups into separate sub-directories. This is advantageous on account that is clearly divides users from groups, however it is a distinction that might not be strictly necessary. It probably not a good idea to name a group and a user the same regardless, and in a sense a group could be thought of as the administrators user account for that group. So instead of the sub-hierarchy with Share, Users and Groups, we would simply have the user and group directories together and Share would simply become another group to which all users automatically belong.

Files/
  Share/
  joe/
  pete/
  ACMEinc/

This would work just as well and save us a level of indirection. But on the downside, a modern Linux system tends to have many groups that are only relevant to system needs. Consider the list on a typical Ubuntu system:

daemon bin sys sync games man lp mail news uucp proxy www-data
backup list irc gnats nobody libuuid syslog klog hplip avahi-autoipd
gdm pulse saned messagebus polkituser avahi haldaemon chipcard
speech-dispatcher couchdb kernoops usbmux festival rtkit gmom
kdm lightdm colord whoopsie

When taking these into consideration the most organizational approach would be to create a separate sub-directory, e.g. Controls, which would house directories of this special type of group.

Files/
  Share/
  Users/
  Groups/
  Controls/

Of course most of these do not really require any directories. The home directories assigned to them are either non-existent or point to a logging directory within var. So this respect it may not be of any significant value to make such a strict division in any case.

If we are able to identify which directories are which, for example, by sorting order or color coding in a command line listing then it is really not such a bother. In that case the simplicity of a single layer outweighs the benefits of strict divisions.