Netbackup: getsockopt SO_ERROR returned: 146 on RHEL (status code 57)

This is typical error on RHEL, the symptom is you'll getting status code 57 (client connection refused) from the activity monitor.

bash-2.03# /usr/openv/netbackup/bin/admincmd/bptestbpcd -debug -verbose -client linuxclient2.com
21:46:49.201 [14785] <2> bptestbpcd: VERBOSE = 0
21:46:49.233 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092
21:46:49.235 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092
21:46:50.235 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092
21:46:50.236 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092
21:46:52.236 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092
21:46:52.239 [14785] <2> vnet_async_connect: vnet_vnetd.c.4237: getsockopt SO_ERROR returned: 146 0x00000092

Now access client linuxclient2.com and verify the xinetd process, usually when you getting socket error on new built server is because of this:

[root@linuxclient2 ~]# ps -ef|grep inetd

root     15700 15669  0 21:46 pts/0    00:00:00 grep inetd

[root@linuxclient2 ~]#  /etc/init.d/xinetd status

xinetd is stopped

[root@linuxclient2 ~]#  /etc/init.d/xinetd start

Starting xinetd:                                           [  OK  ]

[root@linuxclient2 ~]# ps -ef|grep inetd

root     15918     1  0 21:48 ?        00:00:00 xinetd -stayalive -pidfile /var/run/xinetd.pid

root     15921 15669  0 21:48 pts/0    00:00:00 grep inetd

[root@linuxclient2 tmp]# /etc/init.d/xinetd status

xinetd (pid  2965) is running...

Now go back to Master Server and run bptestbpcd again:

bash-2.03# /usr/openv/netbackup/bin/admincmd/bptestbpcd -debug -verbose -client linuxclient2.com

21:49:08.513 [15768] <2> bptestbpcd: VERBOSE = 0
21:49:08.557 [15768] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
21:49:08.557 [15768] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2062: service: bpcd
21:49:08.600 [15768] <2> logconnections: BPCD CONNECT FROM 143.193.71.13.63012 TO 10.239.104.19.13724
21:49:08.603 [15768] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.182: msg: VNETD CONNECT FROM 143.193.71.13.63013 TO 10.239.104.19.13724 fd = 6
21:49:08.622 [15768] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.535: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
21:49:08.666 [15768] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.552: ipc_string: /tmp/vnet-18988301536148664037000000000-RzfjSb
1 1 1
143.193.71.13:63012 -> 10.239.104.19:13724
143.193.71.13:63013 -> 10.239.104.19:13724
21:49:08.866 [15768] <2> bpcr_get_peername_rqst: Server peername length = 17
21:49:08.966 [15768] <2> bpcr_get_hostname_rqst: Server hostname length = 25
21:49:09.066 [15768] <2> bpcr_get_clientname_rqst: Server client name length = 25
21:49:09.166 [15768] <2> bpcr_get_version_rqst: bpcd version: 06540000
21:49:09.266 [15768] <2> bpcr_get_platform_rqst: Server client platform length = 14
21:49:09.366 [15768] <2> bpcr_get_version_rqst: bpcd version: 06540000
21:49:09.467 [15768] <2> bpcr_patch_version_rqst: theRest == > <
21:49:09.468 [15768] <2> bpcr_get_version_rqst: bpcd version: 06540000
21:49:09.566 [15768] <2> bpcr_patch_version_rqst: theRest == > <
21:49:09.567 [15768] <2> bpcr_get_version_rqst: bpcd version: 06540000
PEER_NAME = masterserver.com
HOST_NAME = linuxclient2.com
CLIENT_NAME = linuxclient2.com
VERSION = 0x06540000
PLATFORM = linuxR_x86_2.6
PATCH_VERSION = 6.5.6.0
SERVER_PATCH_VERSION = -1.-1.-1.-1
MASTER_SERVER = masterserver.com
EMM_SERVER = masterserver.com
21:49:09.673 [15768] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.182: msg: VNETD CONNECT FROM 143.193.71.13.63028 TO 10.239.104.19.13724 fd = 8
21:49:09.695 [15768] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.535: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
21:49:09.739 [15768] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.552: ipc_string: /tmp/vnet-18989301536149737033000000000-VQwExj
143.193.71.13:63028 -> 10.239.104.19:13724
<2>bptestbpcd: EXIT status = 0
21:49:09.745 [15768] <2> bptestbpcd: EXIT status = 0

Backup should now completed, from the NBU activity monitor the jobid is 6770640:

bash-2.03# /usr/openv/netbackup/bin/admincmd/bperror -U -jobid 6770640       3/30/2011 22:00:32 mediaserver linuxclient2.com  CLIENT       linuxclient2.com  POLICY Unix_Mon  SCHED       Weekly_full  EXIT STATUS 0 (the requested operation was       successfully completed)

Netbackup nbrbutil command

The nbrbutil command is useful when working with Resource Broker related issues.  The command can be found in the following locations:

Windows:  \NetBackup\bin\admincmdUNIX:  /usr/openv/netbackup/bin/admincmd
Switches are as follows:

Switch	Function
-dump	Dump all nbrb allocation and request list
-dumptables [-f filename]	Have nbrb log its internal state
-release [GUID]	Release the allocation with the given identifier
-cancel [GUID]	Cancel the allocation request with the given identifier
-resetAll	Reset all nbrb allocations, requests, and persisted state
-resetMediaServer [mediaserver]	Reset all nbrb EMM/MDS allocations related to LTID on the mediaserver
-listOrphanedMedia	List media reserved in EMM but with no corresponding allocation in RB
-listOrphanedDrives	List drives reserved in EMM but with no corresponding allocation in RB
-listOrphanedStus	List storage units reserved in EMM but with no corresponding allocation in RB
-releaseOrphanedMedia [name]	Release media reserved in EMM but with no corresponding allocation in RB
-releaseOrphanedDrive [name]	Release drives reserved in EMM but with no corresponding allocation in RB
-releaseOrphanedStu [name]	Release storage units reserved in EMM but with no corresponding allocation in RB
-releaseMDS [ID]	Release EMM/MDS allocations by MDS allocation ID
-releaseAllocHolds	Release allocation holds due to alloction errors for drives/media
-suspend	Suspend nbrb processing
-resume	Resume nbrb processing
-releaseDrive	Release all allocations for the named drive.
-releaseMedia	Release all allocations for the named media ID.
-listOrphanedPipes	List orphaned Fibre transport pipes.
-releaseOrphanedPipes	Release orphaned fibre transport pipes.

/apps/media/inquira/resources /resources

How to analyze Netbackup Unified Logs

For legacy Netbackup logs which is located under /usr/openv/netbackup/logs , below numbers can indicate to you the severity when troubleshooting. Usually look at number 16 for error or 32 for fatal error which means you need to focus and google that error to look for answer:

Severity:

1 = Unknown

2 = Debug

4 = Info

8 = Warning

16 = Error

32 = Critical

Example:

bash-3.00# pwd

/usr/openv/netbackup/logs/dbclient

bash-3.00# grep -i "<16>" log.032211

01:01:47.985 [19945] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/19945.0.1300767393

01:01:47.985 [19945] <16> VxBSAEndTxn: ERR - bsa_close() failed.

01:01:47.986 [19945] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:

02:27:59.607 [944] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/944.0.1300772039

The following directories can be created for administrative command logging.

• /usr/openv/netbackup/logs/admin - Log used as a central logging repository for all the commands located in the /usr/openv/netbackup/bin/admincmd directory.

• /usr/openv/netbackup/logs/bp - Log for the curses-based bp interface program (client-side).

• /usr/openv/netbackup/logs/bpadm - Log for the curses-based bpadm interface program (server-side).

• /usr/openv/netbackup/logs/bparchive - High-level logging of the bparchive command.

• /usr/openv/netbackup/logs/bpbackup - High-level logging of the bpbackup command.

• /usr/openv/netbackup/logs/bpdbjobs - High-level logging of the backup and restore job reporter command.

• /usr/openv/netbackup/logs/bplist - High-level logging of the detailed image list reporting command.

• /usr/openv/netbackup/logs/bpmount - High-level logging for the drive-listing command, used for ALL_LOCAL_DRIVES on the client.

• /usr/openv/netbackup/logs/bprestore - High-level logging of the restore command.

Logs for NetBackup daemons and other automatically-run commands:

• /usr/openv/netbackup/logs/bpbkar - Logs for the backup and archive daemon on the client (backup and archive operations).

• /usr/openv/netbackup/logs/bpbrm - Logs for the backup and restore manager on the media server.

• /usr/openv/netbackup/logs/bpbrmds - Logs for the backup and restore manager on the media server (for disk staging operations).

• /usr/openv/netbackup/logs/bpcd - Logs for the communications daemon on media servers and clients.

• /usr/openv/netbackup/logs/bpcoord - (For NetBackup 5.x) Logs for the bpcoord daemon started by bpsynth on the master server.

• /usr/openv/netbackup/logs/bpdbm - Logs for the database manager daemon on the master server.

• /usr/openv/netbackup/logs/bpdm - Logs for the disk manager daemon on the media server (disk storage units only).

• /usr/openv/netbackup/logs/bpjava-msvc - Java GUI authentication service on the master server.

• /usr/openv/netbackup/logs/bpjava-susvc - (For NetBackup 4.5 and 5.x) Java GUI login service on the master server.

• /usr/openv/netbackup/logs/bpjava-usvc - Java GUI "user" service on the master server.

• /usr/openv/netbackup/logs/bprd - Logs for the request daemon on the master server.

• /usr/openv/netbackup/logs/bpsched - Logs for the scheduler daemon on the master server.

• /usr/openv/netbackup/logs/bpsynth - (For NetBackup 5.x) Logs for the bpsynth process for synthetic backups on the master server.

• /usr/openv/netbackup/logs/bptm - Logs for the tape manager daemon on the media server (media manager storage units only).

• /usr/openv/netbackup/logs/mtfrd - (For NetBackup 5.x) Logs for phase 2 import of Backup Exec media on the master server.

• /usr/openv/netbackup/logs/tar - Logs for the restore daemon on the client (restore operations).

• /usr/openv/netbackup/logs/user_ops - Contains directories and files used by Java and database agents on servers and clients.

• /usr/openv/netbackup/logs/bphdb - Logs for the program that starts scripts to backup the database on servers and clients.

• /usr/openv/netbackup/logs/dbclient - log for the DB Extension clients

Daemons that use Unified Logging
A listing of processes and originators that use Unified logging include, but are not limited to:

• NetBackup Policy Execution Manager (nbpem)
• NetBackup Job Manager (nbjm)
• NetBackup Resource Broker (nbrb)
• NetBackup Enterprise Media Manager (nbemm)
• NetBackup Service Layer (nbsl)
• NetBackup Generic Job (nbgenjob)
• NetBackup Notification Service (nbnos)
• NetBackup for NDMP and ndmpagent
• NetBackup avrd and robotic processes
• NetBackup Operations Monitor (NOMTRS, NOMClient and NOMServer)
• Bare Metal Restore daemons (bmrd, bmbbd, bmrc, bmrs, bmrconfig, bmrsavecfg, bmrsrtadm, bmrprep, bmrsetupmaster, bmrsetupboot, bmrpkg, bmrcreatepkg, bmrcreatefloppy.exe, bmrrst.exe and bmrmap.exe)
• VERITAS Private Branch Exchange (VxPBX)

Daemons that continue to use Legacy logging
A listing of daemons that use Legacy logging include, but are not limited to:

• NetBackup Master Server daemons (bprd, bpdbm, bpjobd, bpdbjobs, admin)
• NetBackup Media Server daemons (bpbrm, bptm, bpdm)
• NetBackup Robotic daemons (vmd, avrd, ltid and robotic daemons)
• NetBackup Client daemons (bpcd, bpbkar, tar)

NetBackup Domain Network Analyzer (NBDNA)

Interesting topic Highlight from the Symantec NetBackup Customer Forum in September 2010.

Topic: NetBackup Domain Network Analyzer (NBDNA)

This is a utility that analyzes the NetBackup domain and its configuration for network issues, performance, and behavior in regards to hostname lookup and connectivity between NetBackup hosts and their role within the NetBackup domain.



NetBackup Domain Network Analyzer (NBDNA) from Peter Elliman on Vimeo.

Storage Networking Flow

Dell™ Solutions Overview Guide for Microsoft®Hyper-V™

I was looking for this one to configure Hyper-V on one of the PowerEdge systems.  I have a Vmware background before so i guess this wont be that hard.  Microsoft® Hyper-V™

Console Cable for your iPad and iPhone

So just can bring your iPad to console in your server, dont need a laptop anymore :)

7 Highly Defective People / Manager

How to find World Wide Name (WWN) for a fibre channel HBA on Windows Server

How to find World Wide Name (WWN) for a fibre channel HBA

You can uses one of the following ways to view WWNs:

* HBAnyware utility (Emulex HBA only)

* SANsurfer utility (Qlogic HBA only)

* SAN switch’s name server output

If none of the above mentioned tools are accessible you can try Fibre Channel Information Tool utility FCINFO.EXE available for download from Microsoft Download web site. It works on Microsoft Windows Server 2003 and Microsoft Windows 2000 systems and comes in a form of MSI package for x86, IA64 or x64 platforms. However, as stated in the Release Notes the Fibre Channel Information Tool does not have to be installed to work correctly. Three core files must be copied in a common folder if the Fibre Channel Information Tool is not installed using the MSI installer: FCINFO.EXE, HBAAPI.DLL, HBATAPI.DLL.

Fibre Channel Information Tool utility is also included in the Microsoft Product Support Reporting Tool and other vendors' reporting tools, such as EMC Grab Utilities (EMCReports), Hewlett Packard Services Reporting Tool (HPS Reports), etc. If you ever run one of those tools on your server, you would have FCINFO.EXE installed.

Running the Fibre Channel Information Tool at a command line with no arguments will give basic information (including WWN) about the installed host bus adapters (HBAs):

C:\>fcinfo

There are 2 adapters:

com.qlogic-QLA2300/2310-0: PortWWN: 21:00:00:e0:8b:08:95:df \\.\Scsi2:

com.emulex-LP9002-1: PortWWN: 10:00:00:00:c9:30:d0:17 \\.\Scsi3:

If you type FCINFO /? or FCINFO /?? you will get a long list of commands, many of which are very task specific and not necessary for information gathering. 

Source: WINTELGUY

How to add virtual ip to the non global zone

How to add the virtual ip to the non global zone by using the command on Solaris 10

#ifconfig hme0 addif ipaddress up zone zonename