How to Kerberize the CentOS/RedHat host and enable for LDAP/LDAPS authorization? It is described in many places but the information is dispersed. So I decided to prepare and bring a simple set of steps to achieve a goal, from scratch. The goal is obvious and straightforward.

• Define the users, groups and the group membership in LDAP.
• Enable for LDAPS (secure LDAP)
• Authenticate in Kerberos.

The target host is KVM CentOS machine and two docker containers: Kerberos and LDAP. The docker containers can be installed on the KVM host or on a separate host.

http://isoredirect.centos.org/centos/6/isos/x86_64/
Download the minimal installation of CentOS 7 system. Assume that the hostname of the machine is centos7/centos7.sb.com Immediately upgrade and install several useful tools.

yum update
yum install mlocate telnet ntp

Kerberos tickets are time-limited, so proper timing is essential.

systemctl enable ntpd
systemctl start ntpd
ntpstat

synchronised to NTP server (91.233.70.230) at stratum 2 
   time correct to within 198 ms
   polling server every 64 s

There are several options available: reuse existing Kerberos, install Kerberos on the same or separate host or use Kerberos as Docker container. During my test, I was using the Docker-Kerberos version: https://github.com/stanislawbartkowski/docker-kerberos. Follow the instruction provided.

Create the container, the realm name is CENTOS.COM.REALM. Assume that Kerberos hostname is kerberos.sb.com.

docker run -d --name kerberos -p 749:749 -p 88:88 -e REALM=CENTOS.COM.REALM ubuntu-kerberos

Install Kerberos client software on CentOS machine

yum install krb5-workstation

Configure

vi /etc/krb5.conf

[libdefaults]
............
 default_realm = CENTOS.COM.REALM
# dns_canonicalize_hostname = false

............
[realms]
CENTOS.COM.REALM = {
  kdc = kerberos.sb.com
  admin_server = kerberos.sb.com
  default_domain = sb.com
 }

Verify connection:

kadmin -p admin/admin (default password: admin)

(Important: in order to launch kadmin from remote machine, the Kerberos realms should be configured as default on a this machine, otherwise kadmin will hange after entering the password)

Authenticating as principal admin/admin with password.
Password for admin/[email protected]: 

kadmin: listprincs

K/[email protected]
admin/[email protected]
kadmin/[email protected]
kadmin/[email protected]
kadmin/[email protected]
kiprop/[email protected]
krbtgt/[email protected]

Add new principal

kadmin: addprinc guest

WARNING: no policy specified for [email protected]; defaulting to no policy
Enter password for principal "[email protected]": 
Re-enter password for principal "[email protected]": 
Principal "[email protected]" created.

Authenticate as newly created guest and get a ticket.

kinit guest
klist

Ticket cache: KEYRING:persistent:0:0
Default principal: [email protected]

Valid starting       Expires              Service principal
10.01.2019 22:35:37  11.01.2019 22:35:37  krbtgt/[email protected]
	renew until 10.01.2019 22:35:37

More details: https://www.certdepot.net/rhel7-configure-system-authenticate-using-kerberos/

yum install pam_krb5
authconfig --enablekrb5 --update
systemctl reload sshd

Review the sshd_config file. Make sure that the following parameters are enabled.

vi /etc/ssh/sshd_config

..........
# GSSAPI options
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
..........

Add host credentials to the Kerberos (replace centos7.sb.com with the hostname of the Kerberized machine)

kadmin -p admin/admin
addprinc -randkey host/[email protected]
addprinc -randkey host/[email protected]

The host credentials should be added to the /etc/krb5.keytab store on the host machine. But there is one tip to remember. The kvno number for the credentials should be equal to one. So before exporting credentials to the keytab file, purgekeys command should be launched to zero the kvno number.
Remark: if default_domain parameter is set in /etc/krb5.conf then host/centos principal (without domain name) can be ignored.

purgekeys -all host/[email protected]
purgekeys -all host/[email protected]

Then run ktadd command. Because every execution of ktadd increases the kvno number, the kvno number at this moment is one as desired. The next time it would be 2.
If the kadmin is executed on the CentOS machine, the default output is /etc/krb5.keytab as needed. If the command is executed on a separate machine then the output should be transported to the CentOS host machine.

ktadd host/[email protected]
ktadd host/[email protected]

Entry for principal host/[email protected] with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/[email protected] with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
kadmin:  ktadd host/[email protected]
Entry for principal host/[email protected] with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/[email protected] with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.

Pay attention to knvo number equals to one.

Add user guest (without password)

adduser guest

Login as user guest

ssh guest@centos7 (use password defined while the principal guest was created in Kerberos)

guest@centos7's password: 
[guest@centos7 ~]$ 

It is a recommended method of authentication. Obtain a Kerberos ticket on the client machine and log in to the CentOS host without passing a password across the network.

Important: the change should be done on the client side.
Verify the ssh_config file and make sure the following parameters are enabled. Parameter GSSAPIDelegateCredentials causes that the Kerberos ticket is transported from local host to the receiving end and it is not necessary to authenticate again to obtain a Kerberos ticket.

vi /etc/ssh/ssh_config

Host *
        GSSAPIAuthentication yes
        GSSAPIDelegateCredentials yes

Obtain Kerberos ticket on a separate machine.

kinit guest

Login as a guest user to the CentOS host. The guest should be logged without bothering about the password.

ssh guest@centos7

Last login: Fri Jan 11 00:16:55 2019 from kerberos
[guest@centos7 ~]$ 

It does not always run smoothly. Very often after enabling passwordless Kerberos authentication, the server ignores our effort and stubbornly is asking us for a password. One method is to run ssh command with -vv parameter.

ssh guest@centos7 -vv

The ssh command is outputting a lot of information and we can look for a clue. Next weapon is to run sshd daemon manually on the CentOS machine with -d (debug) parameter and analyze the output.

systemctl stop sshd
/usr/sbin/sshd -d -d -d
tail -f /var/log/secure

Sometimes helps recreating guest /etc/krb5.keytab file.

rm /etc/krb5.keytab
kadmin -p admin/admin
purgekeys ...
ktadd ...

During my test, I used the Docker version: https://github.com/osixia/docker-openldap. Base dn is "centos7.com". For the purpose of the test, use the auto-generated certificates. For a more serious purpose, more reliable certificates should be deployed, follow the remarks on the web page.

docker run -d -p 389:389 -p 636:636 --hostname ldap.sb.com --env LDAP_DOMAIN="centos7.com" --name ldap --detach osixia/openldap

Important: the hostname --hostname ldap.sb.com should match the hostname used in LDAP authorization. Otherwise, LDAPS (secure LDAP) will not work. The admin dn is "cn=admin,dc=centos7,dc=com" and password is admin. The base db is "dc=centos7,dc=com"

yum -y install openldap openldap-clients nss-pam-ldapd pam_ldap nscd autofs rpcbind nfs-utils

ldapsearch -x -H ldap://ldap.sb.com -b dc=centos7,dc=com -D "cn=admin,dc=centos7,dc=com" -w admin

# extended LDIF
#
# LDAPv3
# base <dc=centos7,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# centos7.com
dn: dc=centos7,dc=com
objectClass: top
objectClass: dcObject
objectClass: organization
o: Example Inc.
dc: centos7

# admin, centos7.com
dn: cn=admin,dc=centos7,dc=com
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: admin
description: LDAP administrator
userPassword:: e1NTSEF9VnZsU3dYOTZ3ZmZSU1RWZWhFUG91dlcrU05EVlQrbHg=

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2

It is a good practice to have a read-only user for scanning LDAP tree and do not use admin for this purpose. Also, anonymous binding is not recommended for security reason.
Security issue: the example sets proxy user password as secret. Do not use any defaults, in a production environment apply a strong password according to best practices.

# proxy, sb.my.com
dn: cn=proxy,dc=centos7,dc=com
cn: proxy
objectClass: organizationalRole
objectClass: simpleSecurityObject
objectClass: top
userPassword: secret

# users, sb.my.com
dn: ou=users,dc=centos7,dc=com
cn: users
objectClass: organizationalRole
objectClass: top
ou: users

# groups, sb.my.com
dn: ou=groups,dc=centos7,dc=com
cn: groups
objectClass: organizationalRole
objectClass: top
ou: groups

# ldapusers, groups, sb.my.com
dn: cn=ldapusers,ou=groups,dc=centos7,dc=com
cn: ldapusers
gidNumber: 2000
objectClass: posixGroup
objectClass: top

# user1, users, sb.my.com
dn: cn=user1,ou=users,dc=centos7,dc=com
cn: user1
gidNumber: 2000
homeDirectory: /home/user1
objectClass: account
objectClass: posixAccount
objectClass: shadowAccount
objectClass: top
uid: user1
uidNumber: 2000
loginShell: /bin/bash

# user2, users, sb.my.com
dn: cn=user2,ou=users,dc=centos7,dc=com
cn: user2
gidNumber: 2000
homeDirectory: /home/user2
objectClass: account
objectClass: posixAccount
objectClass: shadowAccount
objectClass: top
uid: user2
uidNumber: 2001

The schema above contains user cn=proxy,dc=centos7,dc=com (password: secret), and two Linux accounts. For Linux accounts the password is not provided here because the users are going to authenticate in Kerberos.

• cn=proxy,dc=centos7,dc=com (readonly binding)
• ou=users,dc=centos7,dc=com (subtree to keep Linux users)
• • cn=ldapusers,ou=groups,dc=centos7,dc=com (ldapusers - Linux group)
• ou=groups,dc=centos7,dc=com (subtree to keep Linux groups)
• • cn=user1,ou=users,dc=centos7,dc=com (user1 - Linux user belonging to ldapuser group)
• • cn=user2,ou=users,dc=centos7,dc=com (user2 - Linux user)
    

Add schema to LDAP repository

ldapadd -x -D "cn=admin,dc=centos7,dc=com" -w admin -H ldap://ldap.sb.com -f schema.ldif

adding new entry "cn=proxy,dc=centos7,dc=com"

adding new entry "ou=users,dc=centos7,dc=com"

adding new entry "ou=groups,dc=centos7,dc=com"

adding new entry "cn=ldapusers,ou=groups,dc=centos7,dc=com"

adding new entry "cn=user1,ou=users,dc=centos7,dc=com"

adding new entry "cn=user2,ou=users,dc=centos7,dc=com"

Prepare LDIF file to grant cn=proxy,dc=centos7,dc=com

dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {1}to dn.subtree="dc=centos7,dc=com" by dn="cn=proxy,dc=centos7,dc=com" read by dn="cn=proxy,dc=centos7,dc=com" search

The search order is {1}. Configuration administrator is cn=admin,cn=config, default password is config.

ldapmodify -H ldap://ldap.sb.com -w config -D cn=admin,cn=config -f acl.ldif

modifying entry "olcDatabase={1}mdb,cn=config"

Test whether proxy user can read LDAP tree.

ldapsearch -x -H ldap://ldap.sb.com -b "dc=centos7,dc=com" -D "cn=proxy,dc=centos7,dc=com" -w secret

The command should output the whole LDAP subtree.

authconfig --enableldap --ldapserver=ldap://ldap.sb.com:389/ --ldapbasedn="dc=centos7,dc=com" --enablemkhomedir --enablecache --disablefingerprint --kickstart

Manually modify /etc/nslcd.conf and add proxy user. Also uncomment /bin/bash shell for LDAP users.

........
# The distinguished name to bind to the server with.
# Optional: default is to bind anonymously.
binddn cn=proxy,dc=centos7,dc=com
# The credentials to bind with.
# Optional: default is no credentials.
# Note that if you set a bindpw you should check the permissions of this file.
bindpw secret
........
map    passwd loginShell    "/bin/bash"
...

Verify vi /etc/nsswitch.conf

....
passwd:     files sss ldap
shadow:     files sss ldap
group:      files sss ldap
....

Execute

systemctl enable autofs
systemctl restart autofs
systemctl enable nslcd
systemctl restart nslcd
systemctl restart nslcd

(Wait several minutes).

getent group

............
rpcuser:x:29:
nfsnobody:x:65534:
ldap:x:55:
ldapusers:*:2000:

getent passwd

rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin
nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
nslcd:x:65:55:LDAP Client User:/:/sbin/nologin
user1:x:2000:2000:user1:/home/user1:/bin/bash
user2:x:2001:2000:user2:/home/user2:/bin/bash

Add pricipals for user1 and user2 in Kerberos and try to login as user1. Make sure that home directory is created and login shell is bash, not default sh.

kinit user1
ssh user1@centos7

Creating directory '/home/user1'.
[user1@centos7 ~]$ 

id

uid=2000(user1) gid=2000(ldapusers) grupy=2000(ldapusers) kontekst=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

The LDAP users and groups are cached to increase performance. To make changes visible at once, it is necessary to clear caches.

for f in /var/db/nscd/*; do nscd -i $(basename $f); done

Another method is to launch nslcd daemon manually in debug mode and try to make out something from the output.

systemctl stop nslcd
nslcd -d

To improve the performance, Linux is keeping group membership in a local cache. The side effect could be that adding a new group for the existing user, may not be visible in Linux immediately. The enforce the visibility, it is enough to restart nscd deamom.

systemctl restart nscd
id <user>

openssl s_client -showcerts -connect ldap.sb.com:636

...............
SSL handshake has read 1982 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 384 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES256-GCM-SHA384
    Session-ID: 50A83445402E0286A96CD15D1C89BB173488154D19D93FD4AB0064FAD9A1D673
    Session-ID-ctx: 
    Master-Key: 63FB7347EF6C7D8340D161DD5C263BA2A2F17AAC31D45CB2C1C3C3277010D8F6E4507F815AFDDF9B44FC92BBBF3B17FD
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1547246216
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---

The certificates should be copied from LDAP server. Because certificates are plain text files, can be copied by simply copy and paste. In the case of docker container, it is possible to get into docker using a command:

docker exec -it ldap bash

Then change to /container/service/slapd/assets/certs directory and copy and paste three files: ca.crt, ldap.crt and ldap.key. The files should be placed in /etc/openldap/cacerts directory on CentOS server. The ldap.key should be not readable by the outside world, so the permissions should be set Linux 600. Also the ownership should be restricted to nslcd user.

chown nslcd:ldap /etc/openldap/cacerts/*

/etc/openldap/cacerts
[root@centos7 cacerts]# ll
[root@centos7 etc]# ll /etc/openldap/cacerts/
razem 12
-rw-r--r--. 1 nslcd ldap 1043 01-12 00:56 ca.crt
-rw-r--r--. 1 nslcd ldap 1095 01-12 01:43 ldap.crt
-rw-------. 1 nslcd ldap  288 01-12 01:22 ldap.key
[root@centos7 cacerts]# 

Add ca.crt to /etc/openldap/certs NSS database.

certutil -d /etc/openldap/certs/ -A -n "LDAP CA" -t CT,, -a -i /etc/openldap/cacerts/ca.crt

Check that certificate is present in the database.

certutil -L -d /etc/openldap/certs/

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

LDAP CA                                                      CT,, 

Verify that parameters TLS_CACERTDIR points to /etc/openldap/certs directory.

cat /etc/openldap/ldap.conf

.....
TLS_CACERTDIR /etc/openldap/certs

In the root home directory create a file:

vi .ldaprc

TLS_CERT /etc/openldap/cacerts/ldap.crt
TLS_KEY /etc/openldap/cacerts/ldap.key

Run ldapsearch against secure port.

ldapsearch -H ldaps://ldap.sb.com -b dc=centos7,dc=com -D "cn=admin,dc=centos7,dc=com" -w admin

The command should output the same result when using non-secure port ldap.
If something is wrong, do not pay attention to the message below, it is as expected.

TLS certificate verification: Error, self signed certificate in certificate chain

Look for other errors.

Docker LDAP

docker run -d -p 389:389 -p 636:636 --hostname ldap.sb.com --env LDAP_TLS_VERIFY_CLIENT=try --env LDAP_DOMAIN="centos7.com" --name ldap --detach osixia/openldap

For some reason, ldapsearch ignores TLS_CACERT parameter in /etc/openldap/ldap.con configuration file. Use TLS_CACERT parameter and add server certificates to the TLS_CACERT database using certutil tool.

authconfig --enableldaptls --update

Update configuration file manually.

vi /etc/nslcd.conf

Important: comment out ssl start_tls. It is in conflict with ldaps in uri parameter.

........
uri ldaps://ldap.sb.com
........
# Client certificate and key
# Use these, if your server requires client authentication.
tls_cert  /etc/openldap/cacerts/ldap.crt
tls_key  /etc/openldap/cacerts/ldap.key
.....
#ssl start_tls
tls_cacertdir /etc/openldap/certs

systemctl restart nslcd

getent group
getent passwd

The output should report ldapusers group and user1 and user2 users.

nslcd running as a systemctl service can be denied access to /tmp directory and the users and groups are not extracted from LDAP server. In /var/log/audit/audit.log file look for entries like:

type=AVC msg=audit(1547120540.771:482): avc:  denied  { write } for  pid=14179 comm="nslcd" name="tmp" dev="dm-0" ino=16777288 scontext=system_u:system_r:nslcd_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir permissive=0
type=SYSCALL msg=audit(1547120540.771:482): arch=c000003e syscall=83 success=no exit=-13 a0=7fe4a8036240 a1=1c0 a2=ffffffffffffff60 a3=313833 items=0 ppid=1 pid=14179 auid=4294967295 uid=65 gid=55 euid=65 suid=65 fsuid=65 egid=55 sgid=55 fsgid=55 tty=(none) ses=4294967295 comm="nslcd" exe="/usr/sbin/nslcd" subj=system_u:system_r:nslcd_t:s0 key=(null)
type=PROCTITLE msg=audit(1547120540.771:482): proctitle="/usr/sbin/nslcd"

To test it, relax SELinux temporarily:

setenforce 0
systemctl restart nslcd
getent group

If LDAP groups are returned wrongly, it means that nslcd is confined by SELinux. Of course, disabling SELinux for good is not an option because we want to harden, not to soften, security. The solution is extending security policy for nslcd_t service.
Install additional packages to deal with SELinux

yum install setools-console policycoreutils-python

Extend policy for nslcd service and rerun the test.

cat /var/log/audit/audit.log | audit2allow -a -M tmp
semodule -i tmp.pp
setenforce 1
systemctl restart nslcd
getent group
getent passwd

Using FreeIPA is a weight off someone's shoulders. It automizes most of the task including LDAPS. Only some usefull information.

• Install FreeIPA https://computingforgeeks.com/install-freeipa-server-centos-7/
• Open necessary ports.

firewall-cmd --permanent --add-port=88/tcp
firewall-cmd --permanent --add-port=389/tcp
firewall-cmd --permanent --add-port=443/tcp

• Client machine https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/identity_management_guide/linux-manual
• GUI: https://{IPA host name}/ipa/ui/#/e/user/search
• Run ipa-client-install
• To enable home directory creation:

authconfig --enablemkhomedir --update
systemctl restart sshd