Applies to: All versions of Centrify DB2 for applications
Question:
DB2 10.1 won't start when centrifydc_db2gsskrb5 is enabled
Its AIX 6.1 running IBM DB2 10.1.
$ lslpp -l |grep Cent
CentrifyDC.core 5.1.2.378 COMMITTED Centrify DirectControl Agent
CentrifyDC.db2 4.4.4.573 COMMITTED Centrify DB2 Plug-ins
Centrify's setupdb2.sh was run for the instance and everything went fine until db2start was executed.
db2start gives this error message:
$ db2start
SQL1365N db2start or db2stop failed in processing the plugin "centrifydc_db2gsskrb5". Reason code = "10".
04/25/2014 10:18:39 0 0 SQL1365N db2start or db2stop failed in processing the plugin "". Reason code = "".
SQL1032N No start database manager command was issued. SQLSTATE=57019
In db2diag.log the error is described here:
2014-04-25-08.44.08.011509-240 I9759A1735 LEVEL: Error
PID : 23396474 TID : 258 PROC : db2sysc 0
INSTANCE: udbnvp NODE : 000
HOSTNAME: hostname.yourcompany.com
EDUID : 258 EDUNAME: db2sysc 0
FUNCTION: DB2 Common, OSSe, OSSHLibrary::load, probe:70
MESSAGE : ECF=0x90000076=-1879048074=ECF_LIB_CANNOT_LOAD
Cannot load the specified library
DATA #1 : String, 409 bytes
Could not load module /usr/share/centrifydc/kerberos/lib64/libgssapi_krb5.a(shr.o.2.2).
Dependent module /usr/lib/libcrypto.a(libcrypto.so.0.9.8) could not be loaded.
The module has an invalid magic number.
Could not load module /insthome/udbnvp/sqllib/security64/plugin/server/centrifydc_db2gsskrb5.so.
Dependent module /usr/share/centrifydc/kerberos/lib64/libgssapi_krb5.a(shr.o.2.2) could not be loaded.
CALLSTCK: (Static functions may not be resolved correctly, as they are resolved
to the nearest symbol)
[0] 0x090000000C24E224 oss_log__FP9OSSLogFacUiN32UlN26iPPc + 0x1C4
[1] 0x090000000C24E024 ossLog + 0xC4
[2] 0x090000000C2510C0 load__11OSSHLibraryFPCcUl + 0x520
[3] 0x09000000143254A0 secLoadPluginGeneric__FP19SEC_PLUGIN_HANDLE_TPc + 0x12C
[4] 0x09000000176B1184 secLoadServerGSSPlugin__FP19SEC_PLUGIN_HANDLE_TPcT2PFiP
v_i + 0x74
[5] 0x0900000014322998 sqlexLoadAllPluginsServer__FP5sqlca + 0x12C4
[6] 0x09000000141B56B4 @88@sqloSystemControllerMain__FCUiPFv_iPFi_vPPvCPi + 0x
1348
[7] 0x0900000014351268 sqloRunInstance + 0x314
[8] 0x0000000100001B58 DB2main + 0xB58
[9] 0x090000001445FA48 sqloEDUMainEntry__FPcUi + 0x90
[10] 0x090000001445D5E4 sqloEDUEntry + 0x600
[11] 0x0900000000785D30 _pthread_body + 0xF0
[12] 0xFFFFFFFFFFFFFFFC ?unknown + 0xFFFFFFFF
Also:
2014-04-25-08.44.08.012216-240 I12936A1605 LEVEL: Error
PID : 23396474 TID : 258 PROC : db2sysc 0
INSTANCE: udbnvp NODE : 000
HOSTNAME: hostname.yourcompany.com
EDUID : 258 EDUNAME: db2sysc 0
FUNCTION: DB2 Common, OSSe, OSSHLibrary::load, probe:90
MESSAGE : ECF=0x90000076=-1879048074=ECF_LIB_CANNOT_LOAD
Cannot load the specified library
DATA #1 : String, 279 bytes
Could not load module /insthome/udbnvp/sqllib/security64/plugin/server/centrifydc_db2gsskrb5.so(shr_64.o).
File /insthome/udbnvp/sqllib/security64/plugin/server/centrifydc_db2gssk
rb5.so is not an archive or the file could not be read properly.
System error: Exec format error
CALLSTCK: (Static functions may not be resolved correctly, as they are resolved
to the nearest symbol)
[0] 0x090000000C24E224 oss_log__FP9OSSLogFacUiN32UlN26iPPc + 0x1C4
[1] 0x090000000C24E024 ossLog + 0xC4
[2] 0x090000000C251034 load__11OSSHLibraryFPCcUl + 0x494
[3] 0x09000000143254A0 secLoadPluginGeneric__FP19SEC_PLUGIN_HANDLE_TPc + 0x12C
[4] 0x09000000176B1184 secLoadServerGSSPlugin__FP19SEC_PLUGIN_HANDLE_TPcT2PFiP
v_i + 0x74
[5] 0x0900000014322998 sqlexLoadAllPluginsServer__FP5sqlca + 0x12C4
[6] 0x09000000141B56B4 @88@sqloSystemControllerMain__FCUiPFv_iPFi_vPPvCPi + 0x
1348
[7] 0x0900000014351268 sqloRunInstance + 0x314
[8] 0x0000000100001B58 DB2main + 0xB58
[9] 0x090000001445FA48 sqloEDUMainEntry__FPcUi + 0x90
[10] 0x090000001445D5E4 sqloEDUEntry + 0x600
[11] 0x0900000000785D30 _pthread_body + 0xF0
[12] 0xFFFFFFFFFFFFFFFC ?unknown + 0xFFFFFFFF
$ echo $LIBPATH
/insthome/udbnvp/sqllib/lib64:/usr/lib:/lib
All the Kerberos command line tools failed to run:
$ klist
Could not load program klist:
Symbol resolution failed for /usr/share/centrifydc/kerberos/lib/libkrb5support.a(shr.o.0.0) because:
Symbol memcpy (number 16) is not exported from dependent
module /usr/lib/libcrypto.a(libcrypto.so.0.9.8).
System error: Error 0
Examine .loader section symbols with the 'dump -Tv' command.
After fixing LIBPATH, its possible to run the Kerberos command-line tools:
$ echo $LIBPATH
/usr/share/centrifydc/lib64:/usr/share/centrifydc/kerberos/lib64:
$ klist
klist: No credentials cache found (ticket cache FILE:/var/krb5/security/creds/krb5cc_223)
The following messages are seen in db2diag.log:
Could not load module /usr/share/centrifydc/kerberos/lib64/libgssapi_krb5.a(shr.o.2.2).
Dependent module /usr/lib/libcrypto.a(libcrypto.so.0.9.8) could not be loaded.
The module has an invalid magic number.
Answer:
Replace /usr/lib/libcrypto.a with /usr/share/centrifydc/lib64/libcrypto.a
# rm -f libcrypto.a ; ln -s /usr/share/centrifydc/lib64/libcrypto.a .
CentrifyDC should be restarted.
db2start can be issued as the instance owner after checking LIBPATH:
$ echo $LIBPATH
/usr/share/centrifydc/lib64:/usr/share/centrifydc/kerberos/lib64:/insthome/udbnvp/sqllib/lib64:/usr/lib:/lib
$ klist
Ticket cache: FILE:/var/krb5/security/creds/krb5cc_223
Default principal: udbnvp/hostname.yourcompany.com@yourcompany.COM
[... etc ... ]
$ db2stop ; rm -f ./sqllib/db2dump/db2diag.log ; db2start
05/02/2014 09:08:46 0 0 SQL1032N No start database manager command was issued.
SQL1032N No start database manager command was issued. SQLSTATE=57019
05/02/2014 09:08:47 0 0 SQL1063N DB2START processing was successful.
SQL1063N DB2START processing was successful.
KB-11921: Dependent module could not be loaded