Netbackup: getsockopt SO_ERROR returned: 146 on RHEL (status code 57)

This is typical error on RHEL, the symptom is you'll getting status code 57 (client connection refused) from the activity monitor.

bash-2.03# /usr/openv/netbackup/bin/admincmd/bptestbpcd -debug -verbose -client linuxclient2.com
21:46:49.201 [14785] <2> bptestbpcd: VERBOSE = 0
21:46:49.233 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092
21:46:49.235 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092
21:46:50.235 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092
21:46:50.236 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092
21:46:52.236 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092
21:46:52.239 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092

Now access client linuxclient2.com and verify the xinetd process, usually when you getting socket error on new built server is because of this:

[root@linuxclient2 ~]# ps -ef|grep inetd

root     15700 15669  0 21:46 pts/0    00:00:00 grep inetd

[root@linuxclient2 ~]#  /etc/init.d/xinetd status

xinetd is stopped

[root@linuxclient2 ~]#  /etc/init.d/xinetd start

Starting xinetd:                                           [  OK  ]

[root@linuxclient2 ~]# ps -ef|grep inetd

root     15918     1  0 21:48 ?        00:00:00 xinetd -stayalive -pidfile /var/run/xinetd.pid

root     15921 15669  0 21:48 pts/0    00:00:00 grep inetd

[root@linuxclient2 tmp]# /etc/init.d/xinetd status

xinetd (pid  2965) is running...

Now go back to Master Server and run bptestbpcd again:

bash-2.03# /usr/openv/netbackup/bin/admincmd/bptestbpcd -debug -verbose -client linuxclient2.com

21:49:08.513 [15768] <2> bptestbpcd: VERBOSE = 0
21:49:08.557 [15768] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
21:49:08.557 [15768] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpcd
21:49:08.600 [15768] <2> logconnections: BPCD CONNECT FROM 143.193.71.13.63012 TO 10.239.104.19.13724
21:49:08.603 [15768] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.182: msg: VNETD CONNECT FROM 143.193.71.13.63013 TO 10.239.104.19.13724 fd = 6
21:49:08.622 [15768] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.535: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
21:49:08.666 [15768] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.552: ipc_string: /tmp/vnet-18988301536148664037000000000-RzfjSb
1 1 1
143.193.71.13:63012 -> 10.239.104.19:13724
143.193.71.13:63013 -> 10.239.104.19:13724
21:49:08.866 [15768] <2> bpcr_get_peername_rqst: Server peername length = 17
21:49:08.966 [15768] <2> bpcr_get_hostname_rqst: Server hostname length = 25
21:49:09.066 [15768] <2> bpcr_get_clientname_rqst: Server client name length = 25
21:49:09.166 [15768] <2> bpcr_get_version_rqst: bpcd version: 06540000
21:49:09.266 [15768] <2> bpcr_get_platform_rqst: Server client platform length = 14
21:49:09.366 [15768] <2> bpcr_get_version_rqst: bpcd version: 06540000
21:49:09.467 [15768] <2> bpcr_patch_version_rqst: theRest == > <
21:49:09.468 [15768] <2> bpcr_get_version_rqst: bpcd version: 06540000
21:49:09.566 [15768] <2> bpcr_patch_version_rqst: theRest == > <
21:49:09.567 [15768] <2> bpcr_get_version_rqst: bpcd version: 06540000
PEER_NAME = masterserver.com
HOST_NAME = linuxclient2.com
CLIENT_NAME = linuxclient2.com
VERSION = 0x06540000
PLATFORM = linuxR_x86_2.6
PATCH_VERSION = 6.5.6.0
SERVER_PATCH_VERSION = -1.-1.-1.-1
MASTER_SERVER = masterserver.com
EMM_SERVER = masterserver.com
21:49:09.673 [15768] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.182: msg: VNETD CONNECT FROM 143.193.71.13.63028 TO 10.239.104.19.13724 fd = 8
21:49:09.695 [15768] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.535: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
21:49:09.739 [15768] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.552: ipc_string: /tmp/vnet-18989301536149737033000000000-VQwExj
143.193.71.13:63028 -> 10.239.104.19:13724
<2>bptestbpcd: EXIT status = 0
21:49:09.745 [15768] <2> bptestbpcd: EXIT status = 0

Backup should now completed, from the NBU activity monitor the jobid is 6770640:

bash-2.03# /usr/openv/netbackup/bin/admincmd/bperror -U -jobid 6770640       3/30/2011 22:00:32 mediaserver linuxclient2.com  CLIENT       linuxclient2.com  POLICY Unix_Mon  SCHED       Weekly_full  EXIT STATUS 0 (the requested operation was       successfully completed)