As some of you know, Privilege Service (now part of Infrastructure Services) was designed as an Infrastructure-as-a-Service (IaaS) solution, and after it's relase and due to customer demand, we released a "customer-managed" version designed to be deployed and operated in an on-premises setting.
The goal of the lab is set-up the building-blocks to test the high availability capabilities of Privilege Service by leveraging Windows Server Failover Clustering (WSFC).
The design of this lab is based on the following test scenarios:
- Administrative Failover (upgrades, patches, etc).
- System, Network or Storage failure.
- Connector Failover Strategy.
- Disaster Recovery (backup/restore, orphaned or replicated database).
Summarized Windows Failover Clustering requirements
Detailed requirements here: https://technet.microsoft.com/en-us/library/jj612869(v=ws.11).aspx
- Active Directory - required for management and role computer objects, delegation and name resolution (DNS).
- Redundant Shared Storage - WSFC orchestrates the storage switching between active and passive nodes.
- Redundant Network Components - this is a requirement of any modern clustering technology (including WSFC) to provide the assurance of communications via multiple paths, backup network cards, etc.
- At least 3 Windows Server (2012R2) with Windows Failover Clustering and connected to the Storage and Network layers for the CPS Server and at least 2 to act as Centrify Connectors. This will provide the assurance that even when the Windows servers are being updated (or Centrify's software) there is at least a 2-node cluster in service.
- A properly set-up Windows Cluster, passes all the key validation tests during cluster setup.
Note: If this will be a test lab, and data availability is not important, you can use iSCSI virtual disks and virtual network components. Using such a configuration in a production environment does not constitute an effective availability control.
Centrify Privilege Service requirements
- Have a planned hostname and URL for the service (e.g. safe.example.com).
- A static IP address for the corresponding role in Windows Failover Clustering.
- Obtain an x.509 SSL certificate from a public or enterprise CA with the planned name for the service.
(If planning to use Smart Cards, make sure the service name and the servicename +zso is added in the DNS name of the x.509 certificate. E.g. vault.centrify.vms, vaultszo.centrify.vms). Make sure you have the password if the cert is protected.
- Privilege Service software - minimum version 17.7.161 with PostgreSQL back-end.
- Windows Cluster Nodes (ideally 3, and minimally 2)
These are Windows Server 2012 R2 with Quad Cores and 16GB of RAM. Plan for 50-100GB for the data.
- At least 2 Windows Servers to be used as Centrify Connectors (to maintain high-availability).
These are current 64-bit Windows Systems with Dual Core and 8GB of RAM
- A Windows administrator (that can add/remove systems from the domain and local admin rights).
- You should have or be ready to install a Windows Failover Cluster (DNS name and IP address decided) (e.g. cpsha/172.16.200.20)
What you should be familiar with
- Understanding of TCP/IP and name resolution, plus the ability to create DNS records and verify name resolution.
- Understanding of Active Directory administration and privileges to create computer objects.
- Administrative rights and familiarity with Microsoft Management Consoles like the Failover Cluster Manager tool.
- Understanding of network attached storage (e.g. iSCSI or FibreChannel) partitions, volumes, logical disks.
- Understanding of Public Key Infrastructure concepts.
- Is the cluster ready or does it need to be created, are the name and IP address secured?
- What will be the name of the service? (e.g. vault.example.com).
- Have you secured a static IP address for the service name?
- What will be the shared logical disk path for the Centrify Privilege Service database? (e.g. E:\cps-db).
- What will be the procedure to handle the configuration file?
This file contains configuration, encryption and recovery data (without it, no new nodes can be added, and recovery from backup or orphaned database is impossible).
- Is there a test plan to verify the highly available or recovery test cases? (see subsequent post).
- Are the ports required for CPS communication and Centrify Connectors clearly understood?
E.g. HTTP and HTTPS (TCP 80/443), PostgreSQL (TCP/UDP 5432), DirectConnect (TCP 30001).
Sample Lab Planning Notes
Block Diagram of Customer-Managed Privilege Service
- Verify Pre-Requisites.
- First Node: Installation and Verification.
- Installing Additional Nodes.
- Configuring Windows Failover Clustering for Centrify Privilege Service.
- Testing Administrative Failover.
These steps are performed to make sure the Windows Failover Clustering components are ready for Centrify Privilege Service.
- Verify Communications with the cluster
- Verify cluster Active Directory object (e.g. verify that the computer object in AD is alive an well).
- Verify cluster DNS name resolution (e.g. ping the cluster name from all systems).
- Verify CPS port communications (e.g verify the Windows firewall and check for CPS communications).
- Open Windows Failover Clustering and connect to your cluster
- Verify Validation Report of the cluster (Right-click Cluster > View Validation Report).
What are you looking for? Any warnings or errors in the report. For example, the lab I'm using has a few flaws:
Node member3.centrify.vms is reachable from Node member2.centrify.vms by only one
pair of network interfaces. It is possible that this network path is a single point
of failure for communication within the cluster. Please verify that this single path
is highly available, or consider adding additional networks to the cluster.
There you have it. I have only one NIC card, this is a single-point of failure. In a truly highly-available scenario, I'd probably have a pair of NIC cards dedicated to communication and/or storage and a heartbeat interface (just to talk to the cluster), this multiplies the risk per each node added.
- Verify Storage Layer and Disk Layout
- Make sure you understand the disk layout (e.g. logical letter and database and backups location); e.g. E:\cps-db and E:\backups.
- Verify that all cluster members can own the shared storage. (Pause each member and verify the storage switch).
First Node - Installation
Set all nodes but the first node to maintenance mode and secure the storage folder.
- Log on to your First node as a Windows Administrator (e.g. node-1).
- Open Failover Cluster Manager and connect to your cluster.
- Navigate to Cluster > Nodes and right click node-2, select Pause > Drain Roles.
- Repeat for all except for the current node (e.g. node-3, node-4 and so on).
- Verify that the shared disk designated for the CPS database is mounted in the current system.
E.g. open Windows explorer and navigate to the designated drive (e.g. E:\).
- Optional: If you haven't done so, create a folder for the CPS database (e.g. E:\cps-db).
Set name resolution of the service to the first node.
This step is required because during installation the setup the program will need to get a token from the web service, and it thas to be resolvable by name. This step will be undone before the additional nodes are added.
- Open your DNS Management tool (e.g. DNS Manager).
- Create a new CNAME record in your DNS zone. The record should be called as the service name (e.g. vault). The CNAME should point to your node-1 system.
- Verify by pinging the service name (E.g. vault.centrify.vms) from all the systems.
PS C:\> ping vault
Pinging node-1.centrify.vms [192.168.81.21] with 32 bytes of data:
Reply from 192.168.81.21: bytes=32 time
Install Privilege Service (GUI installation)
- Run the Privilege Service installation program, this will start the setup wizard.
- Welcome Page > Press Next.
- EULA Page > Check the box and press Next.
- License Information > Type the case sensitive Company Name and Key, press Next.
- Feature Selection page > Select "Clustered Primary" and press Next.
- Destination Folder page > Select the preferred location (typically not the shared drive, this is NOT the database path).
- Ready to Install page > Press Install.
- Completed page> Press Finish (answer YES to the UAC prompt).
Install Privilege Service (PowerShell)
This part consists in answering these questions:
Centrify Identity Service setup - Host Type: Primary
Starting clustered host setup: Thu, 28 Sep 2017 06:24:56 GMT
1. What username should the initial administrator account be created with?
(default: email@example.com): Sample answer: firstname.lastname@example.org
Don't select a suffix that conflicts internally, and retain this name because
it's the default sysadmin.
2. Initial administrator email address? (default: email@example.com):
Sample answer: firstname.lastname@example.org
This address will only be relevant once the system is SMTP-enabled, feel free to
use your valid business e-mail address.
3. Initial administrator password?: ********
4. Verify administrator password?: ********
Sample answer: Use a strong password or paraphrase, plan to change after setup.
5. FQDN to be used for this service? (default: node-1.example.com):
Sample answer: safe.example.com
This is the service name. When planning, make sure you understand the needs for
the service because once installed, it will respond just to that URL over HTTPS.
6. Would you like to provide a custom host certificate, if not, one will be
generated for you? [Y/N] (default: Y):
The best answer is Y, and to provide a valid x.509 SSL certificate. Although the
auto-generated cert, may be OK for testing, use a real cert for production.
7. Does your certificate require a password? [Y/N] (default: Y): N
Sample Answer: N (if it does not require it), answer Y and provide if needed.
8. Please Select folder for the service databases
This is when you browse for the database location. This has to be in a folder
on the cluster shared storage.
Setup / recovery file creation
Setup will now create a zip file that contains important information needed
to configure other cluster servers or to restore the database onto new systems
DO NOT LOSE IT. DO NOT MAKE UNSECURE COPIES
in particular it contains non recoverable encryption keys. Centrify will not be able
to recover backups without this file
NOTE: The database backup tools do NOT backup it up
Now browse for the location of this file. Secure accordingly because without it you
can't add any new nodes, restore from backup or orphaned database.
When CPS is ready, it's time to verify that it's set.
Internet services successfully stopped
Stopping Postgres Service
Starting Postgres Service
Internet services successfully started
Finished! Your system is now ready to be used via: https://vault.centrify.vms
Finishing web setup ...
Setup standalone/database host completed: Thu, 28 Sep 2017 06:41:21 GMT
First Node - Verification
- Browse to your service address (e.g. https://vault.example.com).
Note that if you did not get a good certificate (e.g. self-signed), your browser may require an exception.
- Attempt to log in with the credentials from the previous step (e.g. email@example.com).
- Navigate between the user and admin portals.
- Optionally, you can upload the configuration file (clconf.zip) as a secret.
- In the admin portal, go to Infrastructure > Secrets.
- Press Add File and Name it "CPS config file" and browse to the config file location.
- Press Save.
- Exit Privilege Service.
Adding Additional Nodes
- Sign-in to your next node (e.g. node-2) with administrative credentials.
- Run Privilege Service GUI Setup and this time, select Secondary Node and advance until PowerShell Configuration.
- During PowerShell, you'll get asked the location of the clconfig.zip file created with the first node.
Centrify Identity Service setup - Host Type: Secondary
Starting clustered host setup: Thu, 28 Sep 2017 05:04:15 GMT
Stopping existing services
Internet services successfully stopped
Stopping existing services
Internet services successfully stopped
Initializing configuration for secondary server
Ensuring VC redist is installed
Please Select location of cluster config zip file
This file was created in the 'config' directory of the primary server
Trusting CA for this Machine
Setup of secondary completed: Thu, 28 Sep 2017 07:35:58 GMT
- Repeat the process (1-3) in all the additional nodes.
- When completed with all nodes, copy the configuration file to a secure thumb drive, make copies and distribute securely and delete the configuration file from the primary node.
Configure Windows Server Failover Cluster for Privilege Service
Get the system, cluster and DNS ready
- Log in to your primary server (the one that is currently running).
- Stop the Centrify Identity Service Database Service and the Web Server.
Use the services applet, or PowerShell (Admin):
- Open Windows Failover Cluster and resume the nodes in maintenance (note that if you choose to fail roles back, this may shift the disks).
- Use your DNS Management tool to erase the CNAME record that was created in the previous section. The reason for this is that the new service (a WSFC Role) will have a DNS name controlled by Windows Clustering.
- Make sure that the service's DNS record vault is not resolvable by any of the systems. If required, flush the DNS cache using the ipconfig /flushdns command.
c:> ping vault
Ping request could not find the host vault. Please check the name and try again.
Installing Privilege Service as a Windows Failover Cluster Role
- Verify that the Centrify Identity Platform Database is stopped in the last active node.
- Open Failover Cluster Manager and connect to the cluster.
- Verify that all cluster nodes where CPS is installed are in the cluster.
- Right Click Roles and Select "Configure Role"; this starts the role configuration wizard.
- Before you begin page > press Next.
- Select Role page > select Generic Script.
- Generic Script info page > Type or paste the path below, then press Next.
C:\Program Files\Centrify\Centrify Identity Platform\scripts\iis_pgsql_cluster.vbs
- Client Access Point page > In Name, type the shortname for the service (e.g. vault) and the IP address for the service.
If you get an error stating that the network name is in use, this means that you have not cleaned-up DNS or the system cache.
- Storage page > Check the box next-to the logical disk (this will be the same logical letter and drive that has the folder with the CPS database information).
We have been consistently using the E:\ drive in this example. Note that there are other ways to do this setup, one includes installing the cluster AFTER CPS installation, in that case, the "Cluster validation tests" can potentially change the logical lettering of the drive. Make sure this is kept consistent.
- Summary page > press Finish.
- Now it's time to inspect the role, and verify what is the active node. Click on Roles and refresh. You should see the service name and the current owner node.
- Use your browser to connect to the service again. When you sign-in, go to the upper-right menu under the user name.
Testing an Administrative Failover
- On Windows failover cluster, connect to the cluster.
- Right-click the newly-created role (with the service name, e.g. vault) and elect Move > Best Available Node.
This step will move the cluster ownership to another node. Note the name.
- You can monitor the status of the service in the remote node (e.g. member4.centrify.vms)
PS C:\> Get-Service cisdb-pgsql -ComputerName member4
Status Name DisplayName
------ ---- -----------
Running cisdb-pgsql Centrify Identity Service Database
PS C:\> Get-Service w3svc -ComputerName member4
Status Name DisplayName
------ ---- -----------
Running w3svc World Wide Web Publishing Service
- Once you have confirmed the transfer was succesful, refresh your browser and confirm the ownership. Keep testing until all nodes are validated.
At this point, after verifying administrative node change to all systems, we can add our connectors.
Adding Centrify Connectors
In this version, the Centrify Connector service cannot be used inside any of the cluster nodes. To install a connector:
- Log on to a Windows 2012R2 (and up) 64 bit system.
- Open your browser and navigate to the service address (e.g. https://vault.example.com/manage?iwa=false)
- Go to Settings > Network and press Add Centrify Connector.
For more detail on Centrify Connector setup, review this article:
- Once your connectors are installed, you can authenticate AD users and perform CPS services like Discovery or Import.
What's next: Configuration?
The next article on this series covers failover testing, however, for some of the testing to be complete, you must configure the system to be functional (populate sytstems, etc); configuring CPS is not in the scope of this article, however, here are some configuration items (in checklist form)
- Verify that AD users can authenticate
- Configure SMTP, Google Maps API or Twilio Service settings.
- Configure Policies for User Access.
- Configure System and Account Security Settings.
- Configure Resource Subnet Mapping.
- Create Roles and assign administrative roles (e.g. Privilege Service User, Privilege Service Administrator).
- Configure and run a Network Discovery.
- Import systems, accounts via CSV import.
- Onboard Databases and Secrets.
- Set up Workflow.
- Configure SSH Gateway Settings.
Privilege Service On Premises - High Availablility - Where to next?