Here is a quick primer on the security risks associated stale Window Service Account Passwords and how Centrify’s new "multiplex account” feature helps… Towards the end is an embedded YouTube video demo showing the feature in action. This post is layed out across the topics shown below:
Windows services and scheduled tasks represents a significant enterprise risk. These accounts are often configured with elevated permissions. They have stale passwords over concerns that rotating their passwords could result in service outages, and passwords for these accounts are often reused through the environment. Furthermore, there are thousands of such accounts in the enterprise when you consider that: 1) there are 100’s - 1000’s of machines; 2) each of which may have several Windows services or scheduled tasks; and 3) each of these services in turn is configured with a domain or local account.
Now, keep in mind that Windows service and scheduled task account password management require special handling. Through some clever engineering by Centrify’s R&D team we introduced the "multiplexed account” feature in CPS to manage the complexity and concerns of service and scheduled task account password management (e.g. concerns over service outages). At first glance, you might find the multiplex accounts feature a bit abstract. However, given the scale and scope of the problem this feature solves, it’s worth taking a moment to understand. For me, the concept clicked by thinking through how not to solve the problem. I hope this helps…
How not to solve the problem… Example scenario
Let’s say we want to schedule periodic password rotation of a domain account used by SQL Server Agent service (e.g. we’ll call it SSAS@somedomain.com). This account username/password is shared across hundreds of Windows servers in our example environment.
Without multiplexing: What could go wrong if we tried rotating firstname.lastname@example.org’s password across all hundreds of end points? Let’s look at the process…
- CPS communicates with the nearest available Domain Controller (e.g. DC-1.somedomain.com) and changes SSAS@somedomain.com’s password;
- CPS communicates with the SQLServer-XX hosts and updates their respective SQL Server Agent service properties with this same password and attempts to restart the services;
- Many of the SQL Server Agent service instances on fail to restartdue to authentication errors with SSAS@somedomain.com.
Reasons for the restart failures:
- One reason is that domain replication is not instantaneous. Perhaps SQLServer-XX hosts spread across different sites and therefore rely on different domain controllers. When CPS updates the domain account's password via a domain controller in one site, it may take several minutes for that password change to propagate to domain controllers in other sites. Therefore, due to replication latency, we may have service authentication failures because some of these service instances attempt to authenticate with the new service password against domain controllers which still expects the old service account password.
- Another possibility is that updating the service’s properties with the new password across all its instances is not immediate. If we have a hundred Windows Servers all running the SQL Server Agent service with the same domain username/password, it will take several minutes to update all service instances with the new domain password. Therefore, it’s possible to have authentication failures because a service instance attempts to restart using its old domain password while its nearest domain controller expects the service account’s new password.
The Multiplexed Account Approach
This approach alone isn't a silver bullet. Rather this is one several tactics that should be used in concert with security practices. For example, Centrify recommends customers implement least privilege for accounts used to run Windows services and scheduled tasks.
Multiplexing provides a clever solution the service account rotation design challenges mentioned above. In short, multiplexed accounts provides a “time buffer”. With multiplexing, we define two accounts objects through Centrify Privilege Service which correspond to two Active Directory domain accounts. We refer to these as “sub-accounts”. These sub-accounts are two new domain accounts with the same rights/permissions as the original service account. The original account used by the Windows service is no longer needed. Going forward, the two sub-accounts take turns operating the Windows service. While one domain account is active - configured as the account for the Windows service - CPS is free to update the other sub-account in AD and across all end points where account is used. The multiplex account approach ensures that all the computers where the managed application account is used are synchronized before the password is rotated. If your password rotation interval is 90 days, for example, the application might run for 45 days using the subaccount1 managed password, then switch to using the identical subaccount2 managed password. When it’s time to switch, a new password is generated and all the computers with an application running under the subaccount2 managed password pick up the new subaccount1 managed password.
The two image below provides a basic checklist for configuring the CPS multiplexed account feature. The first image presents the inital one-time tasks.
This next image presents on-going tasks as new servics or scheduled tasks are discovered.
The YouTube link shows the feature in action.