Gateway mediated networking model with NFS-Ganesha

Synopsis

NFS-Ganesha (in sequel: Ganesha) is a userspace NFSv4 server, with a modular architecture with respect to storage backends. Ganesha would act as a mediator between Manila tenants and the storage pool, to achieve multitenancy, implementing the gateway mediated networking model¹.

The work will piggyback on generic driver infrastructure. We will utilize the networking model of the generic driver² and use service VM-s to host Ganesha instances.

Therefore the easiest to describe the target in terms of differences from the generic driver. The basic difference is that instead of kernel nfsd or CIFS we'll have Ganesha; the others are just the ones implied by this.

Feature-wise it means two things:

• NFSv4 support
• different backends; potentially any storage backend can be used that's available for Ganesha as an FSAL³.

We intend to make the Ganesha driver itself modular; that is, it shall consist of

• the Ganesha driver core which implements Ganesha management in general;
• FSAL driver classes that hook in a particular FSAL to the driver, according to configuration.

We initially plan to implement two FSAL drivers:

• GLUSTER (in sequel: Gluster, I'm using all caps here to match the FSAL name);
• VFS.

Gluster will be the focus of our effort to provide a practically usable storage backend; VFS will be given mostly as an acid test and demonstration of the modular internal structure. Contributions of other FSAL drivers are welcome.

Technical overview

The features of Ganesha (version 2.1.0) that we rely on are the following:

• configurability in Ganesha's own syntax (referred to as Ganesha config);
• adding or removing exports at runtime via dbus (reference).

Regarding these, let us make the following observations:

• Ganesha config: this data can easily be represented as a nested associative array (dict-of-dicts, semantically equivalent to containers that can be expressed in array-free JSON); in Ganesha parlance, dicts will be called blocks.
• Dynamic exports: upon adding an export, what gets passed to the Ganesha service to add is also represented via Ganesha config; we will refer to it as the export block.
• FSAL configuration: an export block has a sub-block that collects the parameters of the FSAL backing the export; we will refer to it as the FSAL block.
• Patching: nested associative arrays offer themselves to structural patching. Given two such objects, one regarded as the "base", other as the "patch" or "overlay", a modified version of the base can be obtained by recursively adding / overwriting keys of base with values provided by the patch.

The operation logic is as follows:

• We provide export block templates for each supported FSAL driver. These specify some preset values, and provide placeholders for parameters that are specified either at startup time (ie. from Manila config) or at runtime. The FSAL name will be a preset value in the included FSAL block and that will allow us to dispatch to the FSAL driver from the export block template.
• The administrator picks the export block template that she finds most suitable for the planned deployment, and places to Manila config space (/etc/manila/). The templates we provide shall serve as sane defaults but the administrator is free to customize them.
• allow_access is mapped to adding an export to ganesha. This happens by creating an overlay dict that includes the specific parameters of the export to be made, patching the export block template with it and adding the resultant export block to Ganesha with dbus.
• parametrizing the export can be divided to two parts: specifying parameters in and out out the FSAL sub-block. Specifying parameters out of the FSAL block follows a common logic and is managed by the Ganesha driver core; specifying the parameters of the FSAL block is FSAL specific and is managed by the FSAL driver. Actually, that's the main thing an FSAL driver has to do: implement the creation of the FSAL block overlay that is being applied to the FSAL block of the export block template upon allow_access.

Further remarks:

• deny_access, of course, is handled by a dbus call to remove the export that was added on behalf of the given access rule.
• The dbus actions that are made on behalf of allow_access/deny_access are affecting only the runtime state of the Ganesha service. We have to take care of persisting the export configuratons. Hereby we just point this out but not getting into details.
• create_share/delete_share is also FSAL specific. Our plan is to have a mount of the nfsv4 root at the disposal of Manila (the so-called service mount), and the FSAL drivers can make changes to the service mount to facilitate creation/deletion. (That said, it can be the case that some FSAL backend manipulation is not possible to perform via filesystem changes in the nfsv4 root, and the FSAL driver needs other mechanisms; the service mount, however, will be offered a generic interface to the FSAL backend.)
• Within the lifecycle of the Ganesha share, the exports added to Ganesha have to have strictly incremental export ids (a 16 bit unsigned value that Ganeshsa uses to address exports) in order to maintain clean mount state on the client (tenant) ends. The Ganesha driver core has to implement stricly monotonous assignment of export ids.

References

• Gateway mediated approach with NFS-Ganesha

Footnotes

^¹ Gateway mediated approach, general description (partially out of date)

^² A sketch of the network layout of the generic driver

^³ FSAL is Ganesha's term for a modular driver component, stands for "File System Abstraction Layer". Here is the list of current ones: Fsalsupport