globus - raeker/ARC-Wiki-Test GitHub Wiki
Installation of v4 is straight forward and ansiblized. Below is the basic process. Globus is encouraging new installs of GCS be v5 instead of v4
Multiple I/O nodes
-Use the same /etc/globus-connect-server.conf file on all servers
-First install on the server running the -id component, then all others
-
Install Globus Connect on all servers
-
Edit the .conf file on the id server as if it were the only one. Set the [MyProxy] server to hostname of the
server that will be the -id component
- Copy the following files/dirs to all servers
- /etc/globus-connect-server.conf
- /var/lib/globus-connect-server/grid-security/certificates/
-
Run globus-connect-server-setup on the first server
-
Run globus-connect-server-setup on remaining servers
With UM being a licensed "Provider" we can enable sharing on Server endpoints as well as users to enable sharing on their personal connect endpoints.
ARC-TS manages both aspects. Instructions below for various processes:
Bringing existing non-managed endpoints under management to enable sharing as well as HA feature. These are for the rest of campus as installation above on ARCTS systems automatically enrolls the endpoint.
- Endpoint Admin must enable sharing in the globus-connect-server.conf file and rerun globus-connect-setup
- Endpoint Admin emails us with endpoint name or uid
- We then navigate to apps.globus.org then find endpoint via search all endpoints (note that the endpoint must be made public in order to search for it).
- Set the "Managed Endpoint" to yes
Installation of v5 has been changing since it release. Currently ARC-TS runs v5.4 and instructions are below. Once the proceedure has been finalized process might be ansiblized.
Do not attempt to install and configure a GCS v5 endpoint without first readying and understanding all of the Globus Documentation at at https://docs.globus.org/globus-connect-server/v5.4/
Below is process with corrected commands and steps. Note that each step generates a UID which is required in many other steps in the process. Keep a record as they are generated as you will need them again.
- Install the software
CentOS/RedHat 7
$ sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
$ sudo yum install http://downloads.globus.org/globus-connect-server/stable/installers/repo/rpm/globus-repo-latest.noarch.rpm
These steps appear to no longer be needed $ sudo yum-config-manager --enable Globus-Connect-Server-5-Stable $ sudo yum-config-manager --enable Globus-Toolkit-6-Stable
$ sudo yum install globus-connect-server54
2) Create the v5 endpoint definition at https://developers.globus.org
-
Log into the Globus Developers Console, developers.globus.org.
-
Click Register a new Globus Connect Server v5.
-
If you dot note have a project to contain this server endpoint click Add another project and fill out the form. This project will be used to track your Globus Connect Server registrations. You can have other projects to organize endpoint if you wish.
-
Use the Add… menu in the Project Header to add other appropriate users in your organization as administrators of the project. Adding other administrators helps your organization avoid losing administrative control should any one administrator leave your organization.
-
Use the Add… menu in the project header to Add a new Globus Connect Server and fill out the form. The display name will be used to identify this endpoint to users when they access it for the first time. Use the same name here that you plan to use in later steps so your users will have a consistent experience.
-
Click Generate a New Client Secret and fill out the form.
-
Save the Client ID and Client Secret values. You will use them soon when creating your Globus Connect Server v5 endpoint.
For globus-turbo2 GCS V5 endpoint definition
ID: db3d45e9-7937-4372-a54b-4e84e0342bcc
Secret: qg3CmavNsTdTmCtEZfzdIoTyjXR8UpMBjt4kPGr6q8A=
- Setup the endpoint and nodes
You should have the public IP addresses for the server ready for the node setup step.
Step 1 will create a file called deployment-key.json which will be needed in others steps so be sure it is indeed created and safe after step one before proceeding to step 2. Note that id below is an example and you should use id generated above. You will also be prompted for the secret.
-
globus-connect-server endpoint setup "ARCTS Turbo Sensitive" --organization "University of Michigan" --client-id "db3d45e9-7937-4372-a54b-4e84e0342bcc" --owner [email protected]
-
globus-connect-server node setup --client-id "db3d45e9-7937-4372-a54b-4e84e0342bcc" --ip-address 141.211.212.166 --deployment-key "/root/deployment-key.json" --incoming-port-range 50000 53000
-
To enable subscription features on this endpoint, have the Globus subscription manager for your organization assign a subscription to this endpoint at
after you've set up at least one node
-
At this point you need to login to globus as a subscription manager to proceed with next steps "globus-connect-server login localhost" which gives a url to go get a key. Authenticate there and the key will be sent internally to the server.
- Configure the Gateways and collections within each
Setup the default paths for users. Here do not allow access to $HOME but allow to start at the root of the Gateway in a file called path-restrictions.json
{
"DATA_TYPE": "path_restrictions#1.0.0",
"none": ["$HOME"],
"read_write": ["/"]
}
In this case we are setting to High Assurance for sensitive data
globus-connect-server storage-gateway create posix "ARCTS Sensitive Data" \
--high-assurance --domain umich.edu
--authentication-timeout-mins $((60 * 24 * 7))
--user-deny root --restrict-paths file:path-restrictions.json
Storage Gateway ID: 247eb4ec-ad44-49dc-8095-8aad99772710
Create Collections
sharing-restrictions.json
{ "DATA_TYPE": "path_restrictions#1.0.0", "read": ["/"] }
globus-connect-server collection create \
247eb4ec-ad44-49dc-8095-8aad99772710 \
/ "Sensitive Turbo Volume Collection" \
--organization 'University of Michigan' \
--contact-email [email protected] \
--info-link https://arc-ts.umich.edu \
--description "Collection of ARC-TS Turbo volumes holding sensitive data" \
--keywords ARC-TS,ARCTS,Turbo \
--allow-guest-collections \
--sharing-restrict-paths file:sharing_restrictions.json \
--enable-https
Collection ID: d99385c2-872e-428d-9fc4-9f5a28397b10
The Box Connector is actually a Box application so we need to do the following as per documentation at https://docs.globus.org/premium-storage-connectors/v5.4/box/
Please be sure to use use same version here in install GCS, in this case v5.4
- Create a Box application as per settings indicated in the documentation. Save all ID and credentials you need these in next steps
- Copy Client ID under the OAuth2.0 Credentials section.
- Under the Add and Managr Public Keys section choose "Generate a Public/Private Keypair"
- You will be prompted to save .json file with this pair. You need this when creating the Box Storage Gateway
- This Box application now needs access to the UMich Enterprise Account. They may not understand what to do so be explicit below
- submit a ticket to UMich Box Enterprise Admins
- Tell them you need to Authorize a New App under the Custom Applications section
- provide the Client ID as that is what the admins will need to use when prompted for the API key from the "Create a Box Application" step
If you ever need to make changes to your application settings in step 1 you need to repeat step 2.
Put the "_81663_dfiidzp0_config.json ile in the root home directory for the next step to create the Gateway. Note the prefix will be different for each application so use what was downloaded.
Create a file to define default path and access permission of the Box Gateway
box-restrictions.json
{
"DATA_TYPE": "path_restrictions#1.0.0",
"read_write": ["/"]
}
globus-connect-server storage-gateway create box
--domain umich.edu \
--user-allow raeker \
--box-settings file:81663_dfiidzp0_config.json \
--high-assurance \
--authentication-timeout-mins $((60 * 24 * 7)) \
--restrict-paths file:box-restrictions.json \
"Umich Box"
Note the --user-allow raeker option. If used will only let raeker access the collection below. If left out any umich.edu user will be able to access the Box connector. Open up only when ready.
Storage Gateway ID: e33f9037-1a2c-4bdc-ac75-6726bfa631b6
globus-connect-server collection create \
e33f9037-1a2c-4bdc-ac75-6726bfa631b6 \
/ "UMich Box Collection" \
--organization 'University of Michigan' \
--contact-email [email protected] \
--info-link https://arc-ts.umich.edu \
--description "Collection of ARC-TS Turbo volumes holding sensitive
data" \
--keywords ARC-TS,ARCTS,Box,UMich
Collection ID: d4c63293-1e4e-4062-be2a-1ae86a50bbb6
The Box Connector is now ready to use.