access_AccessFS - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki
The Access File System
The ACCESS system installed at NCI makes use of a common directory structure to hold scripts, programs, libraries and ancillary data related to the ACCESS models. This shared system reduces duplication of stored files & allows for collaboration between the different organisations comprising ACCESS.
File system quotas
To ensure limited storage space is used effectively NCI enforces file system quotas for all users & projects. These quotas are generally flexible, if you ask mailto:[email protected] the helpdesk for more space with a decent reason for it you should be able to get it.
The shared file system
The shared file system is administered by members of BoM, CSIRO & the CoE. To request programs or libraries be installed please email mailto:[email protected] the ACCESS helpdesk.
File Systems
/projects/access (~access)
The access administration home directory, used to hold scripts and data for running the UM & other models. It is backed up.
/g/data/access
This is also mounted on accessdev so is used for files that are useful on both systems like model prebuilds. Not backed up so only files that can be regenerated or re-downloaded should be stored here.
Useful paths
~access/modules
~access/data
Used to store larger files, e.g. ancillary data sets. Historically this was on a different filesystem but there's no no real distinction between it and the main ~access directory.
~access/bin
Helpful scripts for working with the UM
~access/umdir
Holds settings files and small executables for different UM versions.
~access/data/ancil
Ancillary data for different UM configurations (mainly older versions)
~access/umdir/ancil/atmos
New location for ancillary files. Many Met Office suites expect these to be in $UMDIR/ancil/atmos.
File access control lists
File access control lists allow finer grained control than the normal unix user, group and other permissions. They're used here to ensure that all access users (members of the access group at NCI) can use files in ~access and that the several members of the access.admin group can maintain the directories.
As an example for a particular file
% ls -l ~access/data/ancil/access_v2/qrparm.mask
-rw-rw-r--+ 1 saw562 access.admin 245760 Apr 20 2011 /projects/access/data/ancil/access_v2/qrparm.mask
% getfacl ~access/data/ancil/access_v2/qrparm.mask getfacl: Removing leading '/' from absolute path names
# file: projects/access/data/ancil/access_v2/qrparm.mask
# owner: saw562
# group: access.admin
user::rw-
group::rwx #effective:rw-
group:access:r-x #effective:r--
group:access.admin:rwx #effective:rw-
mask::rw-
other::r--
Default FACL settings for directories should mean that all files created in ~access have read/write permission for the access.admin group and read permission for the access group.
If you have problems with file permissions send a message to access_help.
FACLs and /short/PROJECT and /home
The /short/$PROJECT directories normally have read permission only for project members which can make wider collaboration difficult. The CSIRO p66 and BOM dp9 projects have used FACLs so that all access members can see the top level directories. Individual users then have the option of making their /short/PROJECT/USER directories more open.
Note that this doesn't affect permissions of directories that have more restrictive file permissions like $HOME/.ssh. CHECK HOW THIS WORKS.
User file system
To ensure that simulations can be easily run by different users we encourage the use of a few standard directories for model runs and outputs. Users have access to two main file systems, /home & /short.
File Systems
/home/xxx/$USER
This is the home directory of a user. It is backed up, & by default has a file system quota of 2 GB.
/short/$PROJECT/$USER
This is the user's scratch space. It is not backed up, and files left untouched for some time may be deleted, with the assumption that they can be recreated by the user. Each project has a quota for /short space, shared by all the users of that project. For example, if you log in using project w35 then any files you create under /short will use w35's quota.
Useful paths
/home/xxx/$USER/um_outdir
Output from UM run scripts, error messages & resource usage statistics will appear in these files
/home/xxx/$USER/umui_runs
UM run scripts, the umui copies scripts here before submitting them to the queue
/short/$PROJECT/$USER/UM_ROUTDIR/$USER
UM build & run directories, output from UM runs is located here
Porting Considerations
It may be valuable to have a more structured approach to installing shared programs and libraries on the new machine - e.g. CAP is currently at ~access/ancil.vn7.9, ~access/cap, ~access/umdir/vn7.7/ancil.
To date logins to ~access have been somewhat ad-hoc, with little accountability
The difference between ~access and /data/projects/access is historical - both are on the same filesystem & backed up now
Little use of the module facility for switching between versions
Will the directory structure be compatible with vayu - keeping a fair bit of cruft - or will it be restructured - requiring (possibly automated) porting for each model.
Possibilities
- Use sudo to get to the access account, rather than ssh
- Use access control lists to limit the need for ~access
- Only have a single base access directory rather than two
- Mirror NCI's /apps, /apps/Modules structure under access
- Have a specific location for ancillary data for each model
One option
/projects/access
apps
gcom
4.0
lib
include
4.2
lib
include
cylc
2.4.3
bin
git
bin
modules
gcom
4.0
4.2
cylc
2.4
git
ancil
um
access-1.3-cmip
access-1.3-amip
access-1.3-amip-N48
cable
access-1.3-cmip
some-cable-configuration
All directories viewable by users, each directory under apps has a documented owner(s) with write permissions who maintain that program/library. A limited number (~4?) of super users can sudo into the access account and edit anything in case of emergencies.
Data under ancil is separated by model (um,cable,mom), each with a set of administrators. Further separation should be by model configuration, each folder should correspond to a standard run, with the run owners also having write access. There may be benefit to having different subdirectories for different resolutions - note it will be hard to change once directories are set up and in use by simulations.
It would be nice if we could keep track of metadata for ancil files somehow - who created it, how, what jobs use it.
Should the global module system be used? Would need to consult with NCI to see if this is possible
Use Cases
Three types of users - ACCESS admin, Module admin & general users
Install a new module
A user wants a new piece of software/library installed for use by a subset of ACCESS users
- Nominate a person/people to be the Module admin & handle installation, testing and maintenance
- Send a request via helpdesk for a new module, providing description, contact, original source (provide a template to fill out?)
- ACCESS admin creates an apps subdirectory & sets permissions & creates module subdirectory & module file
- Module admin populates apps subdirectory & tests
- Successful installation announced to target users