FT pipes are in use

In a SAN Client environment, sometime we will see this errors while the jobs seems running but actually its queuing waiting for the FT Pipes to be opened.  But, often the FT pipes are available but some of them has been orphaned due to CORBA exception during cleanup. And if its reached time out you will get error code 83 on netbackup job.

In the Job Details, this will indicates and show you the FT Pipes are in use:

Sep 23, 2011 7:02:09 AM - awaiting resource stug-dpad_nbx_01. Waiting for resources.
          Reason: FT pipes are in use, Media server: nbu004.net2.com,
          Robot Type(Number): NONE(N/A), Media ID: N/A, Drive Name: N/A,
          Volume Pool: NetBackup, Storage Unit: stu-dpad_nbu004_01, Drive Scan Host: N/A,
          Disk Pool: dpad_nbu004_01, Disk Volume: /opt/openv/dpad_nbu004_01_06

Now, we list the orphaned pipes:

# /usr/openv/netbackup/bin/admincmd/nbrbutil -listOrphanedPipes

Pipe ID         Pipe State              Media Server
==============================================================================
3               PIPE_ORPHANED           nbu002.net2.com
10              PIPE_ORPHANED           nbu005.net2.com
12              PIPE_ORPHANED           nbu005.net2.com
13              PIPE_ORPHANED           nbu005.net2.com
14              PIPE_ORPHANED           nbu005.net2.com
15              PIPE_ORPHANED           nbu005.net2.com
16              PIPE_ORPHANED           nbu005.net2.com
4               PIPE_ORPHANED           nbu006.net2.com
5               PIPE_ORPHANED           nbu006.net2.com
6               PIPE_ORPHANED           nbu006.net2.com
7               PIPE_ORPHANED           nbu006.net2.com
8               PIPE_ORPHANED           nbu006.net2.com
9               PIPE_ORPHANED           nbu006.net2.com
10              PIPE_ORPHANED           nbu006.net2.com
11              PIPE_ORPHANED           nbu006.net2.com
12              PIPE_ORPHANED           nbu006.net2.com
13              PIPE_ORPHANED           nbu006.net2.com
15              PIPE_ORPHANED           nbu006.net2.com
16              PIPE_ORPHANED           nbu006.net2.com
16              PIPE_ORPHANED           nbu008.net2.com
1               PIPE_ORPHANED           nbu009.net2.com
3               PIPE_ORPHANED           nbu009.net2.com
4               PIPE_ORPHANED           nbu009.net2.com
5               PIPE_ORPHANED           nbu009.net2.com
6               PIPE_ORPHANED           nbu009.net2.com
7               PIPE_ORPHANED           nbu009.net2.com
8               PIPE_ORPHANED           nbu009.net2.com
9               PIPE_ORPHANED           nbu009.net2.com
14              PIPE_ORPHANED           nbu009.net2.com
15              PIPE_ORPHANED           nbu009.net2.com
16              PIPE_ORPHANED           nbu009.net2.com
8               PIPE_ORPHANED           nbu002.net2.com
12              PIPE_ORPHANED           nbu002.net2.com
1               PIPE_CLIENT_ORPHANED    nbu003.net2.com
2               PIPE_CLIENT_ORPHANED    nbu003.net2.com
3               PIPE_CLIENT_ORPHANED    nbu003.net2.com
4               PIPE_CLIENT_ORPHANED    nbu003.net2.com
5               PIPE_CLIENT_ORPHANED    nbu003.net2.com
6               PIPE_CLIENT_ORPHANED    nbu003.net2.com
7               PIPE_CLIENT_ORPHANED    nbu003.net2.com
8               PIPE_CLIENT_ORPHANED    nbu003.net2.com
9               PIPE_CLIENT_ORPHANED    nbu003.net2.com
10              PIPE_CLIENT_ORPHANED    nbu003.net2.com
11              PIPE_CLIENT_ORPHANED    nbu003.net2.com
12              PIPE_CLIENT_ORPHANED    nbu003.net2.com
13              PIPE_CLIENT_ORPHANED    nbu003.net2.com
14              PIPE_CLIENT_ORPHANED    nbu003.net2.com
15              PIPE_CLIENT_ORPHANED    nbu003.net2.com
16              PIPE_CLIENT_ORPHANED    nbu003.net2.com
1               PIPE_CLIENT_ORPHANED    nbu004.net2.com
2               PIPE_CLIENT_ORPHANED    nbu004.net2.com
3               PIPE_CLIENT_ORPHANED    nbu004.net2.com
4               PIPE_CLIENT_ORPHANED    nbu004.net2.com
6               PIPE_CLIENT_ORPHANED    nbu004.net2.com
7               PIPE_CLIENT_ORPHANED    nbu004.net2.com
8               PIPE_CLIENT_ORPHANED    nbu004.net2.com
9               PIPE_CLIENT_ORPHANED    nbu004.net2.com
10              PIPE_CLIENT_ORPHANED    nbu004.net2.com
11              PIPE_CLIENT_ORPHANED    nbu004.net2.com
12              PIPE_CLIENT_ORPHANED    nbu004.net2.com
13              PIPE_CLIENT_ORPHANED    nbu004.net2.com
14              PIPE_CLIENT_ORPHANED    nbu004.net2.com
15              PIPE_CLIENT_ORPHANED    nbu004.net2.com
16              PIPE_CLIENT_ORPHANED    nbu004.net2.com
1               PIPE_CLIENT_ORPHANED    nbu005.net2.com
2               PIPE_CLIENT_ORPHANED    nbu005.net2.com
3               PIPE_CLIENT_ORPHANED    nbu005.net2.com
4               PIPE_CLIENT_ORPHANED    nbu005.net2.com
5               PIPE_CLIENT_ORPHANED    nbu005.net2.com
6               PIPE_CLIENT_ORPHANED    nbu005.net2.com
7               PIPE_CLIENT_ORPHANED    nbu005.net2.com
8               PIPE_CLIENT_ORPHANED    nbu005.net2.com
9               PIPE_CLIENT_ORPHANED    nbu005.net2.com
11              PIPE_CLIENT_ORPHANED    nbu005.net2.com
2               PIPE_CLIENT_ORPHANED    nbu006.net2.com
1               PIPE_CLIENT_ORPHANED    nbu006.net2.com
14              PIPE_CLIENT_ORPHANED    nbu006.net2.com
1               PIPE_CLIENT_ORPHANED    nbu007.net2.com
2               PIPE_CLIENT_ORPHANED    nbu007.net2.com
3               PIPE_CLIENT_ORPHANED    nbu007.net2.com
4               PIPE_CLIENT_ORPHANED    nbu007.net2.com
5               PIPE_CLIENT_ORPHANED    nbu007.net2.com
6               PIPE_CLIENT_ORPHANED    nbu007.net2.com
7               PIPE_CLIENT_ORPHANED    nbu007.net2.com
8               PIPE_CLIENT_ORPHANED    nbu007.net2.com
9               PIPE_CLIENT_ORPHANED    nbu007.net2.com
10              PIPE_CLIENT_ORPHANED    nbu007.net2.com
11              PIPE_CLIENT_ORPHANED    nbu007.net2.com
12              PIPE_CLIENT_ORPHANED    nbu007.net2.com
13              PIPE_CLIENT_ORPHANED    nbu007.net2.com
14              PIPE_CLIENT_ORPHANED    nbu007.net2.com
15              PIPE_CLIENT_ORPHANED    nbu007.net2.com
16              PIPE_CLIENT_ORPHANED    nbu007.net2.com
1               PIPE_CLIENT_ORPHANED    nbu008.net2.com
2               PIPE_CLIENT_ORPHANED    nbu008.net2.com
3               PIPE_CLIENT_ORPHANED    nbu008.net2.com
4               PIPE_CLIENT_ORPHANED    nbu008.net2.com
5               PIPE_CLIENT_ORPHANED    nbu008.net2.com
6               PIPE_CLIENT_ORPHANED    nbu008.net2.com
7               PIPE_CLIENT_ORPHANED    nbu008.net2.com
8               PIPE_CLIENT_ORPHANED    nbu008.net2.com
9               PIPE_CLIENT_ORPHANED    nbu008.net2.com
10              PIPE_CLIENT_ORPHANED    nbu008.net2.com
11              PIPE_CLIENT_ORPHANED    nbu008.net2.com
12              PIPE_CLIENT_ORPHANED    nbu008.net2.com
13              PIPE_CLIENT_ORPHANED    nbu008.net2.com
14              PIPE_CLIENT_ORPHANED    nbu008.net2.com
15              PIPE_CLIENT_ORPHANED    nbu008.net2.com
6               PIPE_CLIENT_ORPHANED    nbu002.net2.com
10              PIPE_CLIENT_ORPHANED    nbu002.net2.com
14              PIPE_CLIENT_ORPHANED    nbu002.net2.com
16              PIPE_CLIENT_ORPHANED    nbu002.net2.com

As you can see, i have a lot of orphaned PIPES to be cleared.  Run below command to release the orphaned pipes:

# /usr/openv/netbackup/bin/admincmd/nbrbutil  -releaseOrphanedPipes

 List back if there's anymore orphaned pipes , it should be gone by now:

# /usr/openv/netbackup/bin/admincmd/nbrbutil -listOrphanedPipes