Introduction

This article is an attempt to provide the background information, tools and mechanisms to spot and correct Public Key Infrastructure-related issues for those who are setting-up Centrify Multi-factor Authentication or trying to enroll Identity Broker clients.

Background

Public Key Infrastructure is at the heart of how many Internet and corporate infrastructure services are secured today and Centrify has always provided ways to make PKI simpler for organizations.  A big example is adcert, this tool provides support for enterprise trust and auto-enrollment for Microsoft Certification Authority for UNIX, Linux and Mac. 

After the introduction of Identity Service and Privilege Service and the advent of high-profile data breaches and industry guidance like PCI 3.2 Data Security Standard (Requirement 8, Sections 3, 10, 11 & 12), many organizations are rushing to implement Multi-factor Authentication.   Another big milestone is the popularity of hybrid clouds;  Centrify has introduced a new capability called Identity Broker, this new Linux Agent allows organizations to "enroll" in Centrify Privilege Service and to "bridge" multi-source enterprise directories like Active Directory, LDAP, Google Directory and Centrify directory.

All these scenarios make use of Public Key Infrastructure to establish the assurance that clients are talking to the right entity (non-repudiation) and encryption in transit is enforced.  An important point to understand about every Centrify SaaS or on premises tenant is that it has an internal certification authority that is used for multiple uses including encryption, non-repudiation, mobile management and authentication.

PKI Trust and Multi-factor Authentication

With Centrify Identity Service and Privilege Service, it's possible for current users of Centrify Express or Centrify Standard Edition to implement MFA in a very quick and effective way; supporting both modern and legacy-based (RADIUS) solutions.  In the platform's 16.10 release, Centrify proactively deprecated Integrated Windows Authentication (SPNEGO) over HTTP to exclusively use HTTPS.  

Before release 16.10, Centrify announced the deprecation of IWA over HTTP

The implication for users is that any interaction that used IWA (SPNEGO) required PKI trust in the authentication framework for MFA negotiation.

This means that framework after 16.10 looks like this:

As you can see from the framework above, there are 3 ways you can make sure the PKI requirements are satisfied:

• Enterprise Trust:  This is the preferred method. Ideally, an organization has a properly-implemented PKI trust capability. Unfortunately, this is relatively obscure especially in the mid-market.  A great benefit to organizations using Microsoft PKI is that Centrify DirectControl agents will take care of the enterprise trust automatically by bundling the Root and Intermediate CAs into the proper UNIX or Linux bundle.
• Public Trust:  This was a bit expensive a while back, but the easiest way to make sure that PKI trust works out of the box is to use certificates issued by a public vendor like Verisign or GoDaddy.
• Tenant Trust:  Each Centrify tenant will automatically create IWA certificates for all the Centrify connectors in the deployment.  This means that customers can either manually, with DevOps or with Microsoft GPOs can set up a trust chain.  Make sure you download the IWA Trust CA Certificate.  For more information on these steps, use this article.
  Use the "Download your IWA root CA certificate" option.

How to determine if your UNIX/Linux system is ready for MFA

Centrify provides a tool (adcdiag) that will allow users to spot issues with the MFA configuration.   For example, in a Centrified system with an AD environment with Microsoft PKI, the root CA certificate is automatically downloaded to the /var/centrify/net/certs folder and appended to the bundle corresponding to the platform.  

Here's a sample output from a Centrified system with Enterprise Trust:

This output is favorable because DirectControl (adclient) is making sure a lot of the moving parts are in place including making sure that any root or intermediate CA certificates are in the trust chain.  The reason why this "just works" is because a few seconds after joining AD, and if the system is allowed to certificate auto-enrollment, the client will make sure all the proper certs are provisioned to the system and the CA bundle is updated.  This makes this process work like plug and play. 

In cases when there is no trust, then the ca-bundle has to be updated with the IWA trust certificate from the tenant.  When you run the adcdiag, several checks will fail including this one:

 If you inspect the file referenced by adcdiag, there will be the following information in this section: 

"Error setting certificate verify locations" and this will point to the CA bundle for the platform (e.g. /etc/pki/tls/certs/ca-bundle.crt).  There are several ways to solve this issue:

• Enterprise:  Appending the root CA certificate in PEM format to the CA bundle file
• Public:  Making sure the CA bundle is up-to date
• Tenant:  Appending the IWA root ca in PEM format to the CA bundle file.

Fixing MFA CA Trust issues in UNIX/Linux platforms

You'll need to know:

1. How to get the certificate in question.
2. The encoding of the certificate you're receiving.
3. The location of the bundle for the operating system in question.
4. For large production deployments, you'll need to use a viable distribution method.

Scenario

adcdiag failed in a CentOS 6 system.  The issue is with the /etc/pki/tls/certs/ca-bundle.crt.  I am working with a SaaS instance of Identity Service.

Locate the IWA Cert and Determine the Encoding

1. In Admin Portal > Settings > Network > Centrify Connectors > Click the connector > Click the IWA Service tab (left)  and click "Download your IWA root CA certificate"
   
2. Locate the file and try to open it with a text editor.  If the text reads "--- begin certificate"  you are dealing with a usable certificate.
3. Save the file and transfer it to your target system (e.g. IWACert.crt)

Append the certificate to the CA bundle

1. In the target system, concatenate the contents of the certificate file to the platform CA bundle.  E.g.
   

   $ sudo cat /home/user/IWACert.crt >> /etc/pki/tls/certs/ca-bundle.crt

   Note:  there are OS utilities like "update-ca-trust" that perform this step the correct way.  This is for illustration only.
2. Re-run adcdiag and verify the results.

Enterprise Environments

As you can see, the steps above won't scale in a large environment.  This is why the preferred method is to have enterprise trust in place.  Other ways to distribute certificate settings include scripts, DevOps solutions like Chef or Puppet and in Microsoft PKI scenarios, you can use Group Policy.

How to determine if your Windows system is ready for MFA

Windows systems may be easier to work with when it comes to Enterprise Trust but you have to be skillful to troubleshoot as well.  Centrify software may receive MFA settings in different ways:

• Centrify zone:  Systems participating in a Centrify zone enjoy the benefits of advanced access control and Privilege Elevation.  In addition, zones can be tied to a specific Centrify platform instance (cloud or on premises) and users may receive roles (access) or privilege elevation elements (apps, desktops) that require multi-factor authentication.  In this case, there are two toolsets that can be used:  the diagnostics tool and the Access Manager's Identity Platform connectivity test tool.
• Automatically, from Group Policy or registry entries:  Systems that are not participating in zones, can receive settings automatically (if there's only one instance), from group policies or registry settings.

Windows Tools

• Diagnostics Tool:
  With the introduction of Centrify Infrastructure Services 2017.2, a new diagnostics tool has been included with the Centrify Agent for Windows™ 3.4.2.
  
  This tool performs the following checks:
  • Agent service connectivity:  verifies that the Centrify client software is running.
  • MFA configuration check: verifies that MFA is enabled.
  • Identity Service validation:  verifies that the IWA root certificate from the platform is trusted.
  • Connectivity check:  verifies that the client can reach the identity platform (also checks proxy settings).
  • MFA role and permissions check:  verifies that the system is authorized via CIP role and that the role has the "Computer Login and Privilege Elevation" permission.
  • Connector connectivity check: checks connectivity with the existing Centrify connectors.

• Centrify Access Manager - Identity Platform Connection Tool

  This Microsoft management console provides the capability to perform an end-to-end testing in scenarios where DirectAuthorize Roles are being used for MFA.  You'll need to be at least on version 2016.
  

  This option is under right-click Direct Manage Access Manager > Test Centrify Identity Platform Connection.

• Advanced - Diagnostics and Centrify Logger Service

  The Agent Configuration applet provides a "troubleshooting" tab that enables advanced capabilities like Diagnostics or Log inspection.

   

  The diagnostics function has been enhanced to help identify or troubleshoot issues with Identity Platform, this functionality is available if you are running version 3.4 (suite 2017) and above,:
  

  The Centrify logger service is installed with the Centrify Agent for Windows(tm) since 2017.1 (3.4.1), otherwise it has to be installed manually.
  

Built-in Windows Tools

• Certificates MMC snap-in:  This allows you to review all certificate store.
  Note that you have to be a local administrator to view the computer certificate store and that Centrify will add certificates in the local store of systems running the Connector.
  Make sure you review the Enterprise Trust certs in that scenario.
• Certutil:  This is one of the most powerful tools around "certutil -viewstore root" will display the trusted root CAs.

Identifying Issues

The Centrify Agent for Windows will provide you visual feedback when there's a PKI-related issue (see below) but internally it's checking the Certificates directory under \ProgramData\Centrify\DirectAuthorize for the binary blob that represents the tenant's certificate.

 In this case, the same solution applies, but in this case, we're placing the certificate in the trust store for Windows.

Like we discussed before, in large Enterprises, ideally Enterprise or Public trust is set up with automation tools or Group Policy.

Microsoft provides a great article on the topic:  https://technet.microsoft.com/en-us/library/cc772491(v=ws.11).aspx

 Bottom-line:  When attempting to configure MFA, don't forget this checklist:

• Is there a PKI trust between the system and the Centrify service?
• Can the system authenticate via Kerberos?  (is it joined to the domain natively (Windows) or via Centrify (UNIX/Linux))
• Is the machine added to a Centrify role that allows for Computer Login?
• Are all the ports required for communication cleared  (8443 or custom)?

PKI Trust and the Centrify Agent for Linux (cclient)

Identity Broker is Centrify's newest capability that allows for multi-directory authentication in private or public clouds.

IB also requires trust for operations like enrollment. 

Identifying issues with cdebug

Depending on the state of the Linux system (if the ca-bundle is non-existent, modified or outdated) the enrollment operation will fail.  Let's look at a failed enrollment log using two PuTTY windows.

Window 1:  /usr/share/centrifycc/bin/cdebug on
Window 1:  tail /var/centrify/centrifycc.log -f

Window 2:  cenroll -t tenant.my.centrify.com -c [code] -F all -l Identity-Broker-Users
Window 2:  Failed to initialize connection to Centrify identity platform: Failed to connect to Centrify identity platform

Window 1: 
Dec 17 18:16:07 engcen6 cenroll[9201]: DEBUG  Failed 
to post HTTP request: Post https://tenant.my.centrify.com/health/ping: 
x509: certificate signed by unknown authority

 This can be further verified with the cURL command:

$ curl https://tenant.my.centrify.com
$ curl: (77) Problem with the SSL CA cert (path? access rights?)

Remediation

In this particular case, my tenant is on-premises Privilege Service, so I  can follow the instructions on this KB:

KB-7973: How to configure Linux machine trusted certificate chain to perform enrollment for Centrify Agent

The steps are very similar to the ones outlined above.  The strategy depends on the use case Enterprise, Public or Tenant trust is being used.

When trying to enroll, the output is very different:

Verbose: Platform detected: centos_6_6_standard
Verbose: Trying to connect to Centrify identity platform [https://tenant.my.centrify.com/] without a proxy...
Enrolling in Centrify identity platform https://bootcamp.my.centrify.com/ using registration code...

Starting Centrify agent...
Centrify agent started.
Verbose: Trying to connect to Centrify identity platform [https://tenant.my.centrify.com/] without a proxy...

Feature enabled: Application-to-Application Password Management
Feature enabled: Centrify Agent Authentication

Verbose: Restarting Centrify agent after enabled features...

You have successfully enrolled in Centrify identity platform: https://tenant.my.centrify.com/

You may need to restart other services that rely upon PAM and NSS or simply
reboot the computer for proper operations. Failure to do so may result in
login problems for cloud users.

Constant Improvement

At Centrify capabilities change to provide ease of use and supportability.  We hope this article can help you anticipate issues with your testing or setup.  Ultimately, at the enterprise level, PKI is a vital capability that has to be taken seriously and designed to balance the people-process-technology triad.