*** These changes must be run on armis2-build.arc-ts.umich.edu ***

*** Any changes made must be pushed to the rest of the cluster using: sudo /usr/arcts/systems/scripts/ansibleSync.sh ***

Currently, these requests are coming through ServiceNow (labeled as "ARC-TS Login Request).

*** Do NOT create Armis 2 user logins unless the user has permission to reside in a Slurm account on Armis 2. ***

• If they need to user Precision Health contact Jeremy Hallum (jhallum) or Steve Wolodkin (wolodkin) on Slack to determine if the user has permission. 

  • The user must make a request stating that they are looking to use Armis2 to analyze precision health data.
  • Steven or Jeremy must verify that the user has completed the security verifications necessary to use Armis2 (that is, they are actually using the Data Direct application to get data for use on the cluster. Once Steven or Jeremy has giving approval, the user needs to be added to precisionhealth_project and added to the precision-health unix group.
  • Questions on this policy can go to [email protected] (mailto:[email protected]) or [email protected] (mailto:[email protected])

• If they do not have prior permission to be added to an account you can send them this note:

  • *Thank you for your interest in using Armis2. Unfortunately at this time we are unable to create a user login account for you. Before any user is able to have a user login on Armis2 they must have permission from an account owner to be added to that account. These accounts are different from user login accounts and can be recognized by the the naming convention of a PI name followed by digits, such as researcher1 or researcher99. **Once you have gotten permission to be added to such an account we can create your user login. Please also be aware the in order to use Armis2 you must fully complete all HIPAA training.  *

1. Run this command to create a login for a specified user on the operating system:

• sudo /usr/arcts/systems/scripts/addLinuxUser.sh <user>

*** You can comma delimitate users. See below ***

• sudo /usr/arcts/systems/scripts/addLinuxUser.sh <user1>,<user2>,<user3>

2. Run ansible sync to push the changes to /etc/passwd and /etc/group out to all the compute nodes:

• sudo /usr/arcts/systems/scripts/ansibleSync.sh

3) Add the user to the MCommunity group hpc-users-armis2

*** This will create a placeholder for a user that does not have a user login on Armis 2 ***

1. Run this command to add a specified user to a specified Slurm Account:

• sudo /usr/arcts/systems/scripts/addSlurmUser.sh -u <user> -a <slurm_account>

*** Note: Make sure the user's default Slurm Account is NOT the one you're removing them from ***

****** Change user's default Slurm account: ***sudo /opt/slurm/bin/sacctmgr modify user <user> set DefaultAccount=<slurm_account> where cluster=armis2 ***

1. Run this command to remove a specified user from a specified Slurm Account:

• sudo /usr/arcts/systems/scripts/delSlurmUser.sh -u <user> -a <slurm_account>

*** If the above command for some reason fails, the user can be removed with 'sacctmgr' ***

*** Cluster must be defined, otherwise the user will be removed from the Slurm Account but not in the database (not entirely removed). Systems will need to run this ***

• sudo /opt/slurm/bin/sacctmgr remove user <user> cluster=armis2 account=<slurm_account>

2. Remove the user from the scratch directory for the account you took them out of

• sudo /usr/bin/gpasswd -d $u <account>_root

Run sudo /usr/arcts/systems/scripts/ansibleSync.sh

1. Specific Slurm Account:

• sudo /usr/arcts/systems/scripts/addScratch.sh -a <slurm_account>

2. Specific user under specific Slurm Account

• sudo /usr/arcts/systems/scripts/addScratch.sh -a <slurm_account> -u <user>

3. Specific user

• sudo /usr/arcts/systems/scripts/addScratch.sh -u <user> *** NOT WORKING YET ***

*** The current process for creating a Slurm Account on Armis 2 (subject to change as this is going to be automated in the future) ***

1. Create the MCommunity group for the Slurm Account

• Go to mcommunity.umich.edu
• Go to "My Groups" and select "Create a group"
• Here is the required group setup:
  • Group Name: arcts-<slurm_account>-admins
  • Group E-Mail: arcts-<slurm_account>-admins
  • Description: leave blank
  • Joining the group: Owners must add members
  • Members list is viewable by: Anyone
  • Messages can be sent to the group by: Only Members
  • Add Owners: hpc-systems,arcts-helpdesk and 
    • One of these if the account is under the corresponding unit:
      1. LSA: lsait-ars-hpc-provisioning
      2. CoE: coe-arcts-hpc-admin
      3. Med: HITS-RAAC-HPC-Support
  • Add the PI and any requested admins as members
    • (Armis2 ONLY) If the account existed in Armis, bring over the members from the existing MCommunity group (likely "accountname-armis")

2. Run this command to Create the Slurm Account:

• sudo /usr/arcts/systems/scripts/addSlurmAccount.sh -a <slurm_account> -o <organization> (if the organization name has paces in it the name must be enclosed in quotes - e.g.,"public health")
  • You can find a list of organizations at the bottom of this section

3. Change the description for the Slurm Account to show the name of the admin MCommunity group:

• sudo /opt/slurm/bin/sacctmgr modify account name=<slurm_account> set description=arcts-<slurm_account>-admins (Help Desk no longer has this power)
• You can also add any limits at this step with the following flags:
  • -c = cpu
  • -g = gpu
  • -b = billing
  • -m = memory
  • -j = running job limit
  • -w = wallclock limit

4. Add the user list to the Slurm Account:

*** This will create a placeholder for any user that does not have a user login on Armis 2 ***

• sudo /usr/arcts/systems/scripts/addSlurmUser.sh -u <user> -a <slurm_account>

5. Create the Scratch directories for the Slurm Account:

• sudo /usr/arcts/systems/scripts/addScratch.sh -a <slurm_account>
• sudo /usr/arcts/systems/scripts/addScratch.sh -a <slurm_account> -u <user>
• sudo /usr/arcts/systems/scripts/addScratch.sh -u <user> *** NOT WORKING YET ***

6. Add the account's "blob" into the SlurmOps Database from gl-build.arc-ts.umich.edu

• Navigate to: /nfs/turbo/arcts-hpc-support/arcts-support/GL (or A2)
• Create the blob file: nano $account_name
• Populate the file with the blob

  • If the spending limit on the account is a one-time limit you will need to change "Spending Limit: " to None

  • You need to manually change the date format from MM/DD/YYYY to YYYY-MM-DD

  • You will need to manually change the School/College to meet the format of the Organizations in Slurm. All organization names are separated by spaces:

    architecture urban planning arcts art design business dearborn dentistry education engineering environment sustainability information inst social research its kinesiology law school life sciences institute literature science arts matthaei botanical gardens medicine music theatre dance nursing office research pharmacy public health public policy rackham school root social work transportation research inst um dearborn um flint umor

• Insert the blob into the SlurmOps DB: sudo python3 /usr/arcts/systems/scripts/addSlurmOps.py "$blob"

The process for creating an account for a class is the same as described above. A few things to keep in mind:

1. When naming a class account the format must be <department><course number><section number><semester>_class
   1. For example - eecs558s007w21_class
2. Each student receives $60.91 to use for computing. The best way to assign this is to set a template user:

   1. sudo ./addSlurmUser.sh -a <class account name> -u template_user -b 60.91

   2. Set the template user first. Doing this will ensure that all users added after will inherit the limits from the template user

   3. If the PI for the class wishes to have other limits set such as a core limit or a gpu limit you can add them to the template user using the appropriate flags:

-b                Slurm user billing limit (in > dollars)

-c                Slurm user cpu limit

-g                Slurm user gpu limit

-j                Slurm user running job limit

-m                Slurm user memory limit

-C                Slurm user per-job cpu limit

-M                Slurm user per-job mem limit

-G                Slurm user per-job gpu limit

The process for creating a UMRCP account is the same as any other account with a few differences:

• The UMRCP account must end in a0, otherwise the script will error out. 
• You must specify the-roption to indicate the account is a UMRCP account.
• You must also specify the-Hoption with the amount of CPU hours the account will get.

The script will convert the CPU hours into a billing limit and set it appropriately onto the account. Example:

$ sudo /usr/arcts/systems/scripts/addSlurmAccount.sh -a cgbriggs0 -o arcts -r -H 80000
Adding UMRCP Account cgbriggs0 to cgbriggs_root with limits ( GrpTRESMins=billing=12022222224 ) on greatlakes
/opt/slurm/bin/sacctmgr -i add account cgbriggs0 organization='arcts' parent=cgbriggs_root cluster=greatlakes Description='UMRCP'  GrpTRESMins=billing=12022222224
 Adding Account(s)
  cgbriggs0
 Settings
  Description     = umrcp
  Organization    = arcts
 Associations
  A = cgbriggs0  C = greatlakes
 Settings
  GrpTRESMins   = billing=12022222224
  Parent        = cgbriggs_root

$ sacctmgr show account cgbriggs0
   Account                Descr                  Org
---------- -------------------- --------------------
 cgbriggs0                umrcp                arcts

$ sacctmgr list assoc account=cgbriggs0 format=cluster,account,grptresmins -P
Cluster|Account|GrpTRESMins
greatlakes|cgbriggs0|billing=12022222224

UMRCP accounts are allotted an annual amount of 80,000 core hours. This amount may be allocated 100% to Great Lakes, 100% to Armis2, or some other ration between the two. Pay close attention to the hours requested for the account you are creating. 

• The process is the same as creating one for Great Lakes. All RCP accounts must end in '99':
  • sudo /usr/arcts/systems/scripts/addSlurmAccount.sh -a <account99> -o medicine -b 1115.40
    • *The $1115.40 is the RCP limit for either Great Lakes or Armis2. Please check the comments section of the request form for additional details. *

*** Not able to do yet. Entry here for future documentation ***

*** Unix groups created for MCommunity groups must use the same GID (Group ID) ***

1. Use GID from MCommunity group to create the local unix group with the following command:

• sudo /usr/sbin/groupadd -g <GID> <group_name>

1. Add users to local unix groups with the following command:

• sudo /usr/sbin/usermod -aG <group_name> <user>

1. Remove users from local unix groups with the following command:

• sudo /usr/sbin/gpasswd -d <user> <group_name>