This page describes how to enable Knox for Active Directory. As a default, Knox is enabled for OpenLDAP demo provided with HDP which is enough for demonstration but not recommended for production.
There are two types of authentication in Knox which excludes themselves. The default is AD/LDAP authentication based on Shiro framework and the second one is SPNEGO.

HDP 2.6.5
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/configuring_authentication_knox.html

HDP 3.1
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/configuring-proxy-knox/content/setting_up_ldap_authentication.html

Shiro AD/LDAP requires user/password specification because while accessing the service, the client should firstly authenticated in Knox and secondly authenticate in service requested. SPNEGO provides SSO authentication mechanism, after passing the Knox gateway, the user credentials are automatically passed on to the service.
From a practical point of view, if Shiro AD/LDAP is active, the command line curl command should be accompanied by -u user:password parameter. Having SPNEGO enabled passwordless connection is possible with Kerberos ticket and *curl --negotiate -u :" parameters.

Make sure that Knox host can access AD directory LDAP/LDAPS tree. It is described here https://github.com/stanislawbartkowski/wikis/wiki/HDP-2.6.5-and-Active-Directory
In Active Directory, create or identify a separate container for Knox users. Also, create a separate user with search permission able to scan the content of the container. Make sure that the user can bind and read the container.

ldapsearch -b "cn=centos,dc=fyre,dc=net" -W -D CN=hadoopsearch,CN=centos,DC=fyre,DC=net -H ldap://verse1.fyre.net

Enable secure LDAP connection.

ldapsearch -b "cn=centos,dc=fyre,dc=net" -W -D CN=hadoopsearch,CN=centos,DC=fyre,DC=net -H ldaps://verse1.fyre.net

Collect all necessary data:

https://cwiki.apache.org/confluence/display/KNOX/Using+Apache+Knox+with+ActiveDirectory
Let's start with "Sample 3". This configuration allows AD user in CN=centos,DC=fyre,DC=net container to authenticate using over npn-secure LDAP connection. Prepare provider configuration.

 <provider>
      <role>authentication</role>
      <name>ShiroProvider</name>
      <enabled>true</enabled>
      <param name="main.ldapRealm" value="org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm"/>
      <param name="main.ldapContextFactory" value="org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory"/>
      <param name="main.ldapRealm.contextFactory"  value="$ldapContextFactory"/>

      <param name="main.ldapRealm.contextFactory.url"  value="ldap://verse1.fyre.net"/>
      <param name="main.ldapRealm.contextFactory.systemUsername"  value="CN=hadoopsearch,CN=centos,DC=fyre,DC=net"/>
      <param name="main.ldapRealm.contextFactory.systemPassword" value="*******"/>

      <param name="main.ldapRealm.searchBase" value="CN=centos,DC=fyre,DC=net"/>
      <param name="main.ldapRealm.userSearchAttributeName" value="sAMAccountName"/>
      <param name="main.ldapRealm.userObjectClass" value="person"/>

      <param name="urls./**" value="authcBasic"/>
    </provider>

OpenLDAP example

<provider>
      <role>authentication</role>
      <name>ShiroProvider</name>
      <enabled>true</enabled>
      <param name="main.ldapRealm" value="org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm"/>
      <param name="main.ldapContextFactory" value="org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory"/>
      <param name="main.ldapRealm.contextFactory"  value="$ldapContextFactory"/>

      <param name="main.ldapRealm.contextFactory.url"  value="ldap://ldap.sb.com"/>
      <param name="main.ldapRealm.contextFactory.systemUsername"  value="cn=proxy,dc=sb,dc=com"/>
      <param name="main.ldapRealm.contextFactory.systemPassword" value="*****"/>

      <param name="main.ldapRealm.searchBase" value="dc=sb,dc=com"/>
      <param name="main.ldapRealm.userSearchAttributeName" value="uid"/>
      <param name="main.ldapRealm.userObjectClass" value="posixAccount"/>

      <param name="urls./**" value="authcBasic"/>
    </provider>

Prepare, copy and paste the configuration into Ambari -> Knox -> Advanced topology. Replace existing provider definition valid for Demo OpenLDAP.
Restart Knox.

Test the configuration using command line.

knoxcli.sh system-user-auth-test --cluster default

System LDAP Bind successful.

knoxcli.sh user-auth-test --u user1 --cluster default

Password: 
LDAP authentication successful!

Run knoxcli.sh command with --d parameters. It produces more verbose output.

knoxcli.sh system-user-auth-test --cluster default --d

Also, browse through Knox diagnostic log file.

ll -ltr /var/log/knox/

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/setting_up_ldap_authentication.html
Import AD CA into JVM keystore. Important: this command should be executed on the host where Knox is installed, not the Ambari server host.

keytool -import -trustcacerts --file /root/ad.cert -keystore /var/lib/knox/data-2.6.5.1050-37/security/keystores/gateway.jks

Replace JVM --keystore parameter with file path appropriate to your environment.
In Knox configuration panel replace ldap with ldaps and restart Knox service.

      <param name="main.ldapRealm.contextFactory.url"  value="ldaps://verse1.fyre.net"/>

Test Knox again.

knoxcli.sh system-user-auth-test --cluster default
knoxcli.sh user-auth-test --u user1 --cluster default

So far, only user authentication is enabled. Next step is to allow authorization, assign a user to the groups. Before that, several additional information should be collected.

Prepare, copy and paste configuration parameter to enable group membership. The parameters should be added to <provider> definition in "Advanced topology" tab.

<param name ="main.ldapGroupContextFactory" value="org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory" />
<param name="main.ldapRealm.memberAttributeValueTemplate" value="CN={0},CN=centos,DC=fyre,DC=net"/>
<param name="main.ldapRealm.authorizationEnabled" value="true"/>

<param name="main.ldapRealm.userSearchBase" value="CN=centos,DC=fyre,DC=net"/>
<param name="main.ldapRealm.groupSearchBase" value="CN=centos,DC=fyre,DC=net"/>
<param name="main.ldapRealm.groupObjectClass" value="group"/>
<param name="main.ldapRealm.groupIdAttribute" value="sAMAccountName"/>
<param name="main.ldapRealm.memberAttribute" value="member"/>

Example for OpenLDAP.

<param name ="main.ldapGroupContextFactory" value="org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory" />
<param name="main.ldapRealm.memberAttributeValueTemplate" value="cn={0},ou=users,dc=sb,dc=com"/>
<param name="main.ldapRealm.authorizationEnabled" value="true"/>

<param name="main.ldapRealm.userSearchBase" value="dc=sb,dc=com"/>
<param name="main.ldapRealm.groupSearchBase" value="dc=sb,dc=com"/>
<param name="main.ldapRealm.groupObjectClass" value="posixGroup"/>
<param name="main.ldapRealm.groupIdAttribute" value="cn"/>
<param name="main.ldapRealm.memberAttribute" value="memberUid"/>

Several explanations.

main.ldapRealm.memberAttributeValueTemplate can be deduced from AD account properties. DN for user1 account is CN=User1,CN=centos,DC=fyre,DC=net. So the template to extract the user name is CN={0},CN=centos,DC=fyre,DC=net

main.ldapRealm.memberAttribute is the LDAP attribute in AD group property where group members are stored.

• Additional remark, group membership for OpenLDAP. The memberUid attribute of the group should contain values of uid attribute of the group. Below is an example of ldapusers group containing perf, user1 and user2 users.

Save the configuration and restart Knox. Run the test (mark --g option).

knoxcli.sh user-auth-test --u user1 --cluster default --g

Password: 
LDAP authentication successful!
user1 is a member of: centosgroup

No one is happy seeing a password as plain text. The solution is to encrypt the password in keystore.
https://community.hortonworks.com/content/supportkb/151104/how-to-encrypt-the-ldap-password-in-a-knox-topolog.html
Lesson learned: Password encryption is not recognized by knoxli.sh command line. After that, any attempt to test the user authentication ends up with the error message as below. There is no way to bypass it. Nevertheless, user/password authentication in any Knox enabled services is working whatsoever.

Unable to set property 'contextFactory.systemPassword' with value [S{ALIAS=ldapbinddnpassword}] on object of type org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.  If 'S{ALIAS=ldapbinddnpassword}' is a reference to another (previously defined) object, prefix it with '$' to indicate that the referenced object should be used as the actual value.  For example, $S{ALIAS=ldapbinddnpassword}

The first thing to do is to import the password into Knox secure store. The password alias here is ldapbinddnpassword

knoxcli.sh create-alias ldapbinddnpassword --cluster default --value xxxxxxxx

ldapbinddnpassword has been successfully created.

Verify that alias is enlisted in keystore.

knoxcli.sh list-alias --cluster default

Listing aliases for default
ldapbinddnpassword
encryptquerystring

2 items.

In Knox configuration panel, replace plain password with the alias and restart Knox

<param>
        <name>main.ldapRealm.contextFactory.systemPassword</name>
        <value>${ALIAS=ldapbinddnpassword}</value>
</param>

Access to the services can be restricted in Knox. As a default, all users and hosts can access service. "Can access" here means only not to be rejected by Knox itself. Beyond Knox, the service is still guarded by security means appropriate to the service requested. Knox is only the first line of defence. The access to Knox service can be restricted by modifying Authorization Provider section.
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/setting_up_authorization_provider.html
As a default, the provider is defined as:

  <provider>
    <role>authorization</role>
    <name>AclsAuthz</name>
    <enabled>true</enabled>
  </provider>

Default configuration does not impose any restriction. To limit the access, a more specific parameter should be added.

   <param>
       <name>$service_Name.acl</name>
      <value>$cluster_users;$groups_field;IP_field</value>
   </param>

For WebHDFS the $service_Name.acl should be webhdfs.acl. The value format is user;group;hostname or IP, it specifies the list of users, groups and/or hosts allowed to request WebHDFS service through Knox. It is only yes/no permission, Authorized or Not Authorized. Example:

  <provider>
    <role>authorization</role>
    <name>AclsAuthz</name>
    <enabled>true</enabled>

   <param>
     <name>webhdfs.acl</name>
     <value>*;dataadmin,datascience;*</value>
   </param>

  </provider>

Only users belonging to dataadmin or datascience groups can request the access to WebHDFS via Knox proxy. All other requestors are rejected at the gate.

Verify HDFS Custom core-site.xml

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/setting_up_spnego_authentication.html
On Knox node, extract user principal from spnego Kerberos keytab.

klist -kt /etc/security/keytabs/spnego.service.keytab

Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   0 04.02.2019 13:44:39 HTTP/[email protected]
   0 04.02.2019 13:44:39 HTTP/[email protected]
   0 04.02.2019 13:44:39 HTTP/[email protected]
   0 04.02.2019 13:44:39 HTTP/[email protected]
   0 04.02.2019 13:44:39 HTTP/[email protected]

Prepare SPNEGO provider configuration. As the value of hadoop.auth.config.kerberos.principal, copy and paste the principal found in spnego.service.keytab

<provider>
  <role>authentication</role>
  <name>HadoopAuth</name>
  <enabled>true</enabled>
  <param>
    <name>config.prefix</name>
    <value>hadoop.auth.config</value>
  </param>
  <param>
    <name>hadoop.auth.config.signature.secret</name>
    <value>knox-signature-secret</value>
  </param>
  <param>
    <name>hadoop.auth.config.type</name>
    <value>kerberos</value>
  </param>
  <param>
    <name>hadoop.auth.config.simple.anonymous.allowed</name>
    <value>false</value>
  </param>
  <param>
    <name>hadoop.auth.config.token.validity</name>
    <value>1800</value>
  </param>
  <param>
    <name>hadoop.auth.config.cookie.domain</name>
    <value>novalocal</value>
  </param>
  <param>
    <name>hadoop.auth.config.cookie.path</name>
    <value>gateway/default</value>
  </param>
  <param>
    <name>hadoop.auth.config.kerberos.principal</name>
    <value>HTTP/[email protected]</value>
  </param>
  <param>
    <name>hadoop.auth.config.kerberos.keytab</name>
    <value>/etc/security/keytabs/spnego.service.keytab</value>
  </param>
  <param>
    <name>hadoop.auth.config.kerberos.name.rules</name>
    <value>DEFAULT</value>
  </param>
</provider>

In Knox -> Config -> Advance topology, copy and paste the provider replacing existing ShiroProvider. Restart Knox.

Assuming user1 can access HDFS, run Knox->WebHDFS command.

kinit user1
curl -ik --negotiate -u : -X GET "https://a1.fyre.ibm.com:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS"

HTTP/1.1 401 Authentication required
Date: Mon, 11 Feb 2019 16:12:08 GMT
WWW-Authenticate: Negotiate
............
{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16887,"group":"hdfs","length":0,"modificationTime":1549312341745,"owner":"hdfs","pathSuffix":"amshbase","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16392,"group":"hadoop","length":0,"modificationTime":1549369210165,"owner":"yarn","pathSuffix":"app-logs","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
.....

If case of NameNode configured for HA, additional setting is required.
https://community.cloudera.com/t5/Community-Articles/How-to-configure-a-Knox-topology-for-namenode-HA/ta-p/244543
HA provider for WebHDFS should be added to default topology, gateway section.

<provider>
  <role>ha</role>
  <name>HaProvider</name>
  <enabled>true</enabled>
  <param>
    <name>WEBHDFS</name>
         <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value>

  </param>
</provider>

Assuming CentOs or RedHat hosts integrated and registered in Active Directory, the option is to enable PAM Knox authentication. PAM authentication compromises local Linux and AD users or any authentication provider configured for PAM.
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/configuring-proxy-knox/content/setting_up_pam_authentication.html
To enable PAM authentication, specify Shiro PAM provider in Knox -> Config -> Advance topology.

<provider> 
   <role>authentication</role> 
   <name>ShiroProvider</name> 
   <enabled>true</enabled> 
   <param> 
        <name>sessionTimeout</name> 
        <value>30</value>
    </param>                                              
    <param>
        <name>main.pamRealm</name> 
        <value>org.apache.hadoop.gateway.shirorealm.KnoxPamRealm</value>
    </param> 
    <param>                                                    
       <name>main.pamRealm.service</name> 
       <value>login</value> </param>
    <param>                                                    
       <name>urls./**</name> 
       <value>authcBasic</value> 
   </param>
</provider>

Knox UI is controlled by Advanced knoxsso-topology panel. As a default, it is configured to use Demo OpenLDAP and is not impacted by "Advanced topology* settings

In order to switch Knox UI authentication to an external AD/LDAP, modify two parameters leaving all other untouched.

In the case of AD, it is also possible to authenticate using AD credentials directly.

Advanced gateway-site

Command line knoxcli.sh is only partially enabled for PAM authentication.

• knoxcli.sh user-auth-test --u <user> --cluster default : works
• knoxcli.sh user-auth-test --u <user> --cluster default --g : group membership, fails
• knoxcli.sh system-user-auth-test --cluster default : does not work