Tips for finding Knowledge Articles

  • - Enter just a few key words related to your question or problem
  • - Add Key words to refine your search as necessary
  • - Do not use punctuation
  • - Search is not case sensitive
  • - Avoid non-descriptive filler words like "how", "the", "what", etc.
  • - If you do not find what you are looking for the first time,reduce the number of key words you enter and try searching again.
  • - Minimum supported Internet Explorer version is IE9
Home  >
article

KB-9436: SSH login slowness when there are large number of entries in /etc/security/limits.conf

Centrify DirectControl ,  

14 November,17 at 11:35 AM

Problem:

SSH terminal will take a long time to startup for user to login if there are large numbers of entries in the /etc/security/limits.conf file.

Answer:

SSH terminal login might take a long time to startup for user to complete their login, even if the debug log shows that PAM authentication and open session process were completed within second as follow:

Jan 19 11:39:52 sinsx0370004 adclient[11754]: DEBUG <fd:24 sshd(43577)> Received PAM_AUTHTOK: PAM_SUCCESS(0)
Jan 19 11:39:52 sinsx0370004 adclient[11754]: DEBUG <fd:27 PAMVerifyPassword > daemon.ipcclient2 executing request 'PAMVerifyPassword' in thread 140707904091904
...
Jan 19 11:39:52 sinsx0370004 adclient[11754]: DEBUG <fd:27 PAMVerifyPassword > daemon.ipcclient2 Stored credentials for user 'l858436', uid 109327878
Jan 19 11:39:52 sinsx0370004 adclient[11754]: DEBUG <fd:27 PAMVerifyPassword > daemon.ipcclient2 request 'PAMVerifyPassword' complete
Jan 19 11:39:52 sinsx0370004 adclient[11754]: DEBUG <fd:24 sshd(36350)> -> pam_sm_open_session
Jan 19 11:39:52 sinsx0370004 adclient[11754]: DEBUG <fd:24 sshd(36350)> Open session for user 'l858436': directory already exists.
Jan 19 11:39:52 sinsx0370004 adclient[11754]: DEBUG <fd:24 sshd(36350)> <- pam_sm_open_session, result=PAM_SUCCESS(0)

However, the logs is also showing thousands of getpwnam_centrifydc_r request and get_groups_centrifydc_r results as follow:

22651:Jan 19 11:39:49 sinsx0370004 adclient[11754]: DEBUG <fd:24 sshd(36350)> -> getpwnam_centrifydc_r  user="l858436"
22676:Jan 19 11:39:49 sinsx0370004 adclient[11754]: DEBUG <fd:24 sshd(36350)> <- getpwnam_centrifydc_r, result=NSS_SUCCESS(1)
...
172538:Jan 19 11:42:33 sinsx0370004 adclient[11754]: DEBUG <fd:24 sshd(36350)> -> getpwnam_centrifydc_r  user="l858436"
172586:Jan 19 11:42:33 sinsx0370004 adclient[11754]: DEBUG <fd:24 sshd(36350)> <- getpwnam_centrifydc_r, result=NSS_SUCCESS(1)
172590:Jan 19 11:42:33 sinsx0370004 adclient[11754]: DEBUG <fd:24 sshd(36350)> -> pam_sm_close_session
172633:Jan 19 11:42:33 sinsx0370004 adclient[11754]: DEBUG <fd:24 sshd(36350)> <- pam_sm_close_session, result=PAM_SUCCESS(0)

At the end, we have found out that there are more >2500 groups entries at the /etc/security/limits.conf:

grep -v ^# /etc/security/limits.conf | wc -l
2545


Resulting to a problem from the module pam_limits.so. As there are 2000+ groups in the limit file, this causes 2000+ of getgroups call. The module probably just loop through the groups and keep asking for which group the user belongs to and compare that with the each groups in the list. To verify if that is the cause of the slowness, you can test by editing /etc/pam.d/system-auth file and comment out the following line by adding the "#" before it:
 
session required pam_limits.so
 
Restart adclient with command "service centrifydc restart" and try the SSH login again.
After the test, revert the change to system-auth file and restart adclient once again.
 
If the above workaround does resolve login slowness, please consider reducing the number of entries in /etc/security/limits.conf to improve the performance.

Still have questions? Click here to log a technical support case, or collaborate with your peers in Centrify's Online Community.