PNFS - Enstore-org/enstore GitHub Wiki
Definition
PNFS is a service that provides namespace functionality for remote storage systems. It does not store file contents, but provides access to file metadata in a POSIX-like directory tree.
Each PNFS node (directory or file) has a unique PNFSID. Metadata is associated with directories using tags (more information below), and with files, in Enstore, using layers.
Tags
Tags are key:value metadata pairs associated with directories. They can be viewed using POSIX interface utilities, such as in this example showing the file_family tag:
(Values have been modified)
$ cat '/pnfs/fs/usr/user/exp/run/.(tag)(file_family)'
user-exp-run1$
- Note there is no new line in the tag value, this is an Enstore-specific convention.
Alternatively, the Enstore cli can be used to show all tags:
enstore sfs —tags will show all tags associated with that directory.
(Values have been modified)
[]$ enstore sfs --tags /pnfs/fs/usr/user/exp/run
.(tag)(file_family) = user-exp-run1
.(tag)(file_family_width) = 1
.(tag)(file_family_wrapper) = cern
.(tag)(library) = CD-LTO8G2
.(tag)(OSMTemplate) = StoreName sql
.(tag)(sGroup) = chimera
.(tag)(storage_group) = user
-rw-rw-r-- 12 root root 15 Jan 20 2011 /pnfs/fs/usr/user/exp/run/.(tag)(file_family)
-rw-rw-r-- 5008 43682 8938 1 Feb 11 2003 /pnfs/fs/usr/user/exp/run/.(tag)(file_family_width)
-rw-r--r-- 4587 root root 4 Feb 11 2003 /pnfs/fs/usr/user/exp/run/.(tag)(file_family_wrapper)
-rw-r--r-- 5263 root root 9 May 11 2015 /pnfs/fs/usr/user/exp/run/.(tag)(library)
-rw-r--r-- 146050989 root root 14 Feb 22 2012 /pnfs/fs/usr/user/exp/run/.(tag)(OSMTemplate)
-rw-r--r-- 146050989 root root 8 Feb 22 2012 /pnfs/fs/usr/user/exp/run/.(tag)(sGroup)
-rw-r--r-- 5337 root root 4 Feb 11 2003 /pnfs/fs/usr/user/exp/run/.(tag)(storage_group)
Note, PNFS does not know the meaning of these tags, they are interpreted by the application which utilizes PNFS (in this case, encp).
Layers
PNFS tracks file-specific metadata using a pattern called layers. For historical reasons, files can have up to 7 layers, although Enstore only uses layers 1 (bfid) and 4 (all metadata), whereas dCache uses layer 2.
Layers can also be read using POSIX utilities:
(Values have been modified)
$ cat '/pnfs/fs/usr/user/exp/run/.(use)(1)(run1_raw_data_c005.tar.bz2)' # layer 1 (bfid)
CDMS123451234500002
Or by using the Enstore CLI:
(Values have been modified)
[enstore@enstore03 ~]$ enstore sfs --cat /pnfs/fs/usr/user/exp/run/run1_raw_data_c005.tar.bz2 1
CDMS123451234500002
[enstore@enstore03 ~]$ enstore sfs --cat /pnfs/fs/usr/user/exp/run/run1_raw_data_c005.tar.bz2 2
2,0,0,0.0,0.0
:h=yes;c=1:1234abcd;l=20012345100;
[enstore@enstore03 ~]$ enstore sfs --cat /pnfs/fs/usr/user/exp/run/run1_raw_data_c005.tar.bz2 4
FM1111L8
0000_000000000_0000001
20012345100
user-exp-run1
/pnfs/fs/usr/user/exp/run/run1_raw_data_c005.tar.bz2
00001234567890ABCDEF1234567890ABCDEF
123451234500002
emv11223:/dev/rmt/123abc:12345ABCDE
1111222
Layer four contains several pieces of positional metadata, mostly unique to Enstore. It can be pretty-printed using hte Enstore cli:
(Values have been modified)
$ enstore sfs --xref /pnfs/fs/usr/user/exp/run/run1_raw_data_c005.tar.bz2
volume: FM1111L8
location_cookie: 0000_000000000_0000001
size: 20012345100
file_family: user-exp-run1
original_name: /pnfs/fs/usr/user/exp/run/run1_raw_data_c005.tar.bz2
map_file:
pnfsid_file: 00001234567890ABCDEF1234567890ABCDEF
pnfsid_map:
bfid: 123451234500002
origdrive: emv11223:/dev/rmt/123abc:12345ABCDE
crc: 1111222
Writing layers cannot simply be done with POSIX utilities, and is best done with a library like the included namespace library, written in Python.
Note this metadata is also stored in the enstoredb, and is typically not used by Enstore, although Enstore and related utilities sometimes check to ensure consistency between the data sources.