To simplify the cut & paste of commands we have listed all used variables here along with their default values. Setting up your local environment once is easier than altering every line of code showed at this FAQ.

The default values for the following commands are (you may copy & paste this, altering them as required):

truststore_path=/usr/local/tomcat/conf/esg-truststore.ts
store_pass=changeit
port=443
server=pcmdi3.llnl.gov

Also assure the keytool app is in the current path (it's part of the JDK).

In reality none. But for the ESG security there is.

The keystore holds the private key which will be used for signing outgoing SSL trafic (the server certificate). This ensures the client the traffic will come always from the contacted server and will be encrypted. In order for the client to assure the server is really the one he tried to contact, the server certificate must be trusted. In the X509 context a certificate is really a chain of certificate. A certificate is said to be trusted if there is a trusted certificate (any AFAIK) in this chain. So the client _ must _ have already a certificate from this chain available locally. Normally this happens with the CAs, whose self-signed certificates are pre- installed in browsers, distros or even by the JVM. The typical file name in ESG is keystore-tomcat .

The trustore , on the contrary, holds certificates for client authentication. These are used to guarantee the server that clients are those _ trusted _ . While relying on MyProxy , the trustore must contain at least one certificate from the MyProxy certificate chain. Normally this is a common CA or the server generated SimpleCA. This way the server knows the client is to be trusted. The typical file name in ESG is jssecacerts .

See: [ http://java.sun.com/j2se/1.4.2/docs/guide/security/jsse/JSSERefGuide.ht ml#Stores ](http://java.sun.com/j2se/1.4.2/docs/guide/security/jsse/JSSERefGui de.html#Stores)

A monster one-liner for checking the SSL is trusted is:

test $(keytool -keystore $truststore_path -storepass $store_pass -list | grep "$(echo | openssl s_client -connect $server:$port 2>&1 | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -noout -fingerprint -md5 | sed -n 's/^[^=]*=//p')" | wc -l) == 1 && echo "Already trusted" || echo "Not Trusted"

The result will either be Already trusted or Not Trusted in which case the certificate should be added to the node truststore.

Another one-line Monster for adding the certificate to the tomcat truststore:

echo | openssl s_client -connect $server:$port 2>&1 | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'| keytool -keystore $truststore_path -storepass $store_pass -importcert -noprompt -alias $server

The result will either be certificate added or an error message. In case the certificate is not in the truststore but you still get an alias already in use error message, you should just change the alias parameter to something else (e.g. -alias ${server}-1 ). The alias is only used to avoid referring to the certificate by its real name (aka. the fingerprint).

After altering the truststore (or the keystore for what it matters) you MUST always restart tomcat.

NOTE: If the node was installed using the esg-node script, it is simple to add a host's certificate to the local truststore.  Simply use the --register flag.  
Ex %> esg-node --register <hostname of machine you wish to have cert of>

see the esgf installer site

simply by creating a new keystore and deleting the private key in it.

#create one with a self-signed cert
keytool -genkey -keyalg RSA -alias "selfsigned" -keystore empty.ks -storepass "$store_pass" -validity 360
#delete the cert
keytool -delete -alias "selfsigned" -keystore empty.ks -storepass "$store_pass"

This is a little complicated since neither keytool nor openssl tools can achieve this. For this you'll need the extkeytool from the Shibboleth project.

1. You will have to export the key to der (pkcs8) format:

   openssl pkcs8 -in hostkey.pem -topk8 -nocrypt -outform DER -out hostkey.pkcs8

2. Then if you are exporting a certificate chain, concatenate every certificate from the one whose private key we've just transformed to the root self-signed CA:

   cat hostcert.pem CA1.pem CA2.pem ... > cert.bundle

3. Create an empty keystore as mentioned in How do I create an empty keystore?

4. Use the extkeytool to import the certificate chain and key into the trustore.

   extkeytool -importkey -keystore empty.ks -alias tomcat -storepass $store_pass -keyfile hostkey.pkcs8 -certfile cert.bundle -provider org.bouncycastle.jce.provider.BouncyCastleProvider

5. Verify everything is in place

   keytool -v -list -keystore empty.ks

OR

If you used the esg-node installation script to setup your node you can import certificates, update certificates, create keystores, etc. The above steps are all done using the _ --install-keypair _ option. See the installer page for more details on the esg-node script and it's options.

_ --install-keypair _ - takes as input private key and public cert files and installs them. [ ]

%> esg-node --install-keypair private_key.pem public_cert.pem 

The user is them prompted for they certificate chain all the way to the root.  The user may supply a cert bundle, the individual certs and / or the URL(s) to said certs or cert chain. Also the various certificates are updated and the truststore is maintained and certs verified and displayed. The extkeytool suite is downloaded and installed and used for operations (described above).  

"Pretty much, use the esg-node installer to perform your key maintenance tasks, among other things. ;-) We've got you covered." -gavin

#if this is a keystore you'll have to change the private key pass also
keytool -keypasswd -keystore keystore.ks -storepass  $store_pass -alias tomcat -keypass  $store_pass -new <new_pass>

# with this command you can change the password of the store itself.
keytool -storepasswd -keystore keystore.ks -storepass $store_pass -new <new_pass>

The password of the keystore and that from the private key saved in it MUST be the same (see this )

keytool -export -alias <cert_alias> -keystore <keystore_file> -file exportcert.pem 

The output format from the above command is binary.... To get text output, do the following:

keytool -export -alias <cert_alias> -keystore <keystore_file> -rfc -file exportcert.pem

If you don't know the alias of the certificate, just list everything and look for it:

keytool -list -v -keystore <keystore_path>

(note: <keystore_path> maybe your keystore or truststore file. It refers to anything file is in java keystore format.

There are two steps to perform:

1. Activate Tomcat general logging : This depends on the logging procedure selected; and the default one depends on the tomcat version. Assuming Tomcat 6.x, the default logging framework is java.util.logging (you can change it to be log4j, but this would require tampering with the library path of the tomcat installation, and therefore is not addressed here. See links for more info). The easiest way to control this is to write a logging properties file, e.g.

   logging properties for apache

   .handlers = java.util.logging.ConsoleHandler

   java.util.logging.ConsoleHandler.level = FINEST java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

   org.apache.catalina.level=FINEST org.apache.catalina.loader.level=INFO

and tell tomcat where this is located by passing to the JVM the argument java.util.logging.config.file , e.g.

java ... -Djava.util.logging.config.file="<path_to_log_props>" ...

Links: tomcat 6.0 , (older tomcat 5.5.)

2. Activate SSL debugging : in order to tell tomcat that it should also debug connections a special VM argument is also required javax.net.debug for debugging JSSE SSL. The values are one of the following:

   all turn on all debugging ssl turn on ssl debugging

   The following can be used with ssl: record enable per-record tracing handshake print each handshake message keygen print key generation data session print session activity defaultctx print default SSL initialization sslctx print SSLContext tracing sessioncache print session cache tracing keymanager print key manager tracing trustmanager print trust manager tracing pluggability print pluggability tracing

       handshake debugging can be widened with:
       data         hex dump of each handshake message
       verbose      verbose handshake message printing
   
       record debugging can be widened with:
       plaintext    hex dump of record plaintext
       packet       print raw SSL/TLS packets
   

If you set it to help the above message will be displayed and the application quited.

Links: [ debugging SSL Java SE 6 ](http://download.oracle.com/docs/cd/E17409_0 1/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html#Debug)

• For Example if you are using eclipse alter the tomcat server start script (probably in Run->Run Configurations...->"Tomcat vX.X Server"->Arguments ) and add the following VM arguments:

  -Djava.util.logging.config.file="<absolute_path_to_log_props>" -Djavax.net.debug=ssl,handshake,verbose ...

append to closed appender named [xxxx].

There's a problem with the logger. Use -Dlog4j.debug while starting tomcat to see what's happening. The probable cause is that there are several loggers defined with the very same name. This might even be defined across different property files which might be inside jar files!! (I had that problem with the eske.jar library)

To find the files within the jars you can go to $CATALINA_HOME/webapps and execute this:

find . -type f -name '*.jar' | xargs egrep '[^/]log4j\.(xml|properties)'

Then you'll have to expand the jars, delete the files and then recreate the jars.

There's another problem caused when an appender with the same name as that from the parent appender is added to a child category. So change this:

[   1] <root>
[   2]     <priority value="ERROR" />
[   3]     <appender-ref ref="my_log" />
[   4] </root>
[   5] <logger name="some.class" additivity="false">
[   6]     <level value="DEBUG" />
[   7]     <appender-ref ref="my_log" />
[   8] </logger>

into this:

[   1] <root>
[   2]     <priority value="ERROR" />
[   3]     <appender-ref ref="my_log" />
[   4] </root>
[   5] <logger name="some.class">
[   6]     <level value="DEBUG" />
[   7] </logger>

some.class is already referencing my_log , so there's no need to do it twice. additivity="false" is apparently forbidding some.class _ descendants _ to write to the previous appenders, some.class itself is not being affected by this. (not clear to me, see this

trusted" mean?

A full error being thrown from a datanode (via Thredds) may look like the below:

SEVERE: Servlet.service() for servlet FileServer threw exception
javax.net.ssl.SSLPeerUnverifiedException: Target is not trusted
        at esg.security.utils.ssl.CertUtils.retrieveCertificates(CertUtils.java:131)
        at esg.orp.app.AuthenticationFilter.retrieveORPCert(AuthenticationFilter.java:153)
        at esg.orp.app.AuthenticationFilter.attemptValidation(AuthenticationFilter.java:117)
        at esg.orp.app.AccessControlFilterTemplate.doFilter(AccessControlFilterTemplate.java:62)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:662)

I was able to determine that this coincides with the following lines logged in the threddsServlet.log file:

WARN  - esg.security.utils.ssl.CertUtils - java.security.cert.CertificateException: No name matching HOSTNAME found
ERROR - esg.orp.app.AuthenticationFilter - The server at https://HOSTNAME/OpenidRelyingParty/home.htm is not trusted.

This means one of several things. First check that the Datanode trusts itself, by making sure that it has an entry for itself in the esg-truststore.ts file. You can do this by running:

esg-node --register localhost:443

Even if it says it's already present, it usually doesn't hurt to have the duplicate in your own truststore, but others may argue against it for size restrictions. Running the command to completion is also beneficial because it makes sure that your system has a sync'd esg-truststore.ts and jssecacerts file. Syncing these helps to be certain that if there's an error in your configuration on which truststore to use, even if it gets confused and uses the system wide file, it will match the esg required truststore.

If the issue persists, the next thing to check is that the certificate on your datanode has a valid CN parameter. No matter how you generated the certificate, it is REQUIRED that the CN parameter exactly matches your datanode's fully qualified domain name. For example, if your hostname is foo.bar.com, the certificate must report that the CN=foo.bar.com. To verify this, run the following command to output information about your host certificate:

$JAVA_HOME/bin/keytool -list -alias MY-HOST-ALIAS -v -keystore esg-truststore.ts

NOTE: MY-HOST-ALIAS is often your hostname, but it may differ. If you're unsure how to determine which alias you should be looking at, you should run the "esg-node --register" command listed above and it will tell you what alias name the certificate was added as.

If the CN does NOT match your fully qualified hostname, you will see the error above, even if the certificate is present in the truststore.

If this is the case, you should regenerate your node's certificate with the proper CN.

This is a complicated operation, and there are various pieces of information on how to do this scattered around this FAQ and others. Below is a comprehensive set of steps to do this manually from scratch. There may exist other tools to help, but this is presented as a standalone procedure.

Whatever you do, do NOT confuse Java's keytool program with the GNU keytool program. They have similar syntax and will cause many headaches if you mix and match commands. If you need to, substitute every keytool command shown below with "$JAVA_HOME/bin/keytool".

Also, make sure that your system has OpenSSL installed.

openssl req -out NEW-CERT.csr -new -newkey rsa:2048 -nodes -keyout NEW-KEY.key

You will be prompted for information about your machine, but the most important part is that when you are prompted for the Common Name, which looks like this,

Common Name (eg, YOUR name) []:

you MUST enter your machine's fully qualified hostname. This step also generates a new private key file. While you don't use it right now, "keep it secret, keep it safe".

NOTE : If you know which CA you plan to send this certificate to for signing, be sure to check with them on what their signing policy is. There may be specific values required that you should enter above (other than the Common Name we pointed out).

This can either be done via e-mail, scp, etc. Each CA has their own process, so contact your CA. The CA basically takes your certificate request and outputs a signed certificate that you will use.

If you are running your own CA, you can use the Globus Simple CA commands to sign a CSR as follows:

$GLOBUS_LOCATION/bin/grid-ca-sign -in NEW-CERT.csr -out NEW-CERT-signed.pem

The output file that needs to be shipped back is "NEW-CERT-signed.pem".

A few notes about signing certificates with your CA: First is that each CA has a specific signing policy that enforces which subjects can be used in the certificate being signed. Make sure that the certificate generated in Step 1 is compliant with the target CA.

Also, you will be prompted for the CA password when you issue this command to sign a certificate. If you're running Globus Simple CA and do not remember the CA password, it is available in your MyProxy configuration file _ /etc/myproxy-server.config _ .

Whether you get it from your CA via email, scp, or simply copy it from another directory, somehow locate the signed certificate. At this point, you have a signed certificate and a private key (generated from Step 1 above).

Once you get it back from the CA, it will be in PEM format (a particular kind of Certificate format). We need to convert the signed certificate to DER format so that it can be used with Java. You can convert both the signed certificate and the key in two separate steps as follows:

# First convert the signed certificate (note the extension difference of the output file)
openssl x509 -outform der -in NEW-CERT-signed.pem -out NEW-CERT-signed.der

# Then convert the private key
openssl pkcs8 -topk8 -nocrypt -in NEW-KEY.key -inform PEM -out NEW-KEY.der -outform DER

The next step is to create a new keystore and load it up with this new certificate/key pair. If you do not have ImportKey already on your system, you can download it from here:

http://www.agentbob.info/agentbob/79-AB.html

Download and compile the code as follows:

$ wget http://www.agentbob.info/agentbob/80/version/default/part/AttachmentData/data/ImportKey.java
$ javac ImportKey.java

Now run the tool with the following command, using your DER cert and key as input:

java ImportKey NEW-KEY.der NEW-CERT-signed.der

This creates a new keystore located at ${HOME}/keystore. ImportKey and the default alias is "importkey" and the default keystore password is "importkey".

For ESG, we'll move this file to a more well known location and change the password on it. There are actually two passwords to change: 1) The password of the keystore itself, and 2) The password of the key inside the keystore. Right now, they're both "importkey" and we'll change them both below, but note that they must also match, regardless of what new password we use. We also want to rename the default alias it was created under, as some tools require it to be another name. All of those steps are shown below with comments:

# Change directory to where you'd like the keystore to reside (using /usr/local/tomcat/conf as an example)
cd /usr/local/tomcat/conf

# Move the keystore here
mv ~/keystore.ImportKey esg-keystore

# Change the keystore alias (remember, the default password is "importkey")
# ESG requires this alias to be named "tomcat", so that's what we change it to here
keytool -changealias -alias importkey -destalias tomcat -keystore esg-keystore

# Now let's change the password of the key stored inside the keystore
keytool -keypasswd -alias tomcat -keypass importkey -new NEWPASSWORD -keystore esg-keystore

# Now let's change the password of the keystore itself, which is separate from the password
# of the encrypted key stored inside, but MUST match that value for ESG tools to work
keytool -storepasswd -new NEWPASSWORD  -keystore esg-keystore

# Finally, verify that the configuration is correct by listing the contents using the new password
keytool -list -keystore esg-keystore

Now that we have a new certificate generated and a shiny new keystore for our node, we need to import the certificate into our truststore. In ESG, a datanode or P2P node needs to trust itself for proper ORP ( OpenIdRelyingParty ) interactions with TDS (Thredds) and/or other local Java web services. Importing the certificate to our truststore is easy:

keytool -import -alias HOSTNAME -file NEW-CERT-signed.der -keystore esg-truststore.ts

Substitute in the HOSTNAME field above the fully qualified domain name of your server. It should match the CN field we carefully added in Step 1 above.

The esg-truststore.ts file referenced should be the official one available from the ESG Certificate page here:

http://esgf.org/esg-certs/index.html

Direct link to truststore:

https://rainbow.llnl.gov/dist/certs/esg-truststore.ts

Tomcat needs pointers to both your new keystore and your new truststore. Since we just created new entries that it should be using at runtime, we need to edit this configuration. Add or edit the following to the Connector section that runs on port 443:

truststoreFile="/usr/local/tomcat/conf/esg-truststore.ts" truststorePass="changeit"
keyAlias="tomcat"
keystoreFile="/usr/local/tomcat/conf/esg-keystore" keystorePass="NEWPASSWORD"

Note that the default ESG truststore password is "changeit". If you download the latest from the official page, this is already set, so do not modify that value in the configuration.

The keystore password should match the NEWPASSWORD you used in Step 6 above.

At this point, the new certificate and key generation is complete. Your Tomcat server should now start properly using the new certificate.

script)

The esg-node script has been a tool used throughout the setup of the ESGF Node. The script has become a valuable tool for performing various tasks associated with installation and maintenance of the node. The following describes using the script to perform the tasks described in the previous entry in a regimented way. For info on the installer (esg-node script) goto http://esgf.org/esgf-installer-site/

Please be sure to copy into a safe location the tomcat truststore, keystore, the exisiting pem files you may have, and the hostcert.pem and hostkey.pem files (located in /etc/grid-security) somewhere else safe; just as a precaution. These are the affected files so it is good to have a backup... I'll automate that too when I get a chance.

These steps may also be performed using the esg-node script.

(step 1-2)
To create a new ssl key and csr issue the following command.
%> esg-node --generate-ssl-key-and-csr

This results in the following files created in your tomcat's conf directory:
hostkey.pem
<you-host-machine-fqdn>-esg-node.csr

Then follow step 3 + 4 (above)
When you get the signed csr back put it in tomcat's conf directory as well and do the following:

(steps 5-8)
%> esg-node --install-keypair

Without any args the script will look in tomcat's conf directory for the files in needs.
Namely it is looking specifically for the hostkey.pem and the signed csr with the same name as the .csr file but with the .pem extension (<you-host-machine-fqdn>-esg-node.pem)

You will be prompted for your CA's certificate chain file(s).
For your convenience we have the VeriSign chain here: http://rainbow.llnl.gov/dist/etc/verisign_chain.cer
Enter this url in directly. You may enter in your own chain file(s) and/or URLs to them.

Do step 9 manually (when I get a chance I will automate that too ;-)

to https://pcmdi3.llnl.gov)

The following line retrieves the certificate from the server:

echo | openssl s_client -connect $server:$port 2>&1 | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'

It can be redirected to a file:

echo | openssl s_client -connect $server:$port 2>&1 | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > serv_cert.pem

Or directly displayed if piped to openssl x509 -text :

echo | openssl s_client -connect $server:$port 2>&1 | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -text

Like mentioned above, but you would also probably want to name it .0 so use this command:

#get the certificate in a proper form
(cert=$(echo | openssl s_client -connect $server:$port | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'); echo "$cert" > $(echo "$cert" | openssl x509 -noout -hash).0)

This is only required for exporting the server SSL, normally you just need the CA signing the server SSL (this is the same one in case of self-signed certificates for testing purposes)

Just point your browser to your openID. There are three outcomes I'm aware of:

Result

Meaning

You get an xrds(xml) file

Everything is fine, that's your openId which tells the requesting server where it should go for verifying your id.

HTTP Status 404 - /esgcet/myopenid/myname

If the path gets displayed, the servlet couldn't be found. Check it's started (in the logs or the default one is found here: /esgcet/openid/provider.htm). This could be the /esgcet component is not required for you or it's missing, check the url is right.

HTTP Status 404 \-

In this case the path is not being displayed, this means the servlet was found, but the servlet itself returned a 404 (in this case, the user is not found). Check the Gateway DB to see it's there.

That's a Datanode specific issue, check it here: Data Node FAQ

Security in a global environment is very closely tied to the time constrain. Be sure your server (Datanode and/or Gateway) runs an NTP service and that it's synchronized. You may check it by issuing the following command

ntpstat

It will report the maximum possible error, which should be less than a second.

A wrong set locale or changes in DST might cause problems and reject a by all other means valid certificate. Just something to consider.

For other security issues which could be derived from either the Gateway or the Data-node refer to the Gateway FAQ (especially for publication problems) or the ESGF Node FAQ (especially for gridFTP problems).