[Veritas-bu] Backups slow to a crawl

Jeff McCombs jeffm at nicusa.com
Wed Mar 23 12:32:27 CST 2005


Yeah, I originally thought that this might be a network problem myself.
However I have checked the network settings on the Sun systems and the Cisco
switches in-between. I'm even forcing a 100FDX on the switch and system just
to be safe (auto negotiation never works, regardless of what the vendors
say)

Seems that this is a MPX thing. I did some further testing and backing up
systems without multiplexing enabled, and the problem goes away. The rmt/1
device stops with the 100% busy and 0 kw/s, client full backups drop back
down into the 15 minute range...




On 3/23/05 10:56 AM, "Jorgensen, Bill" <Bill_Jorgensen at csgsystems.com>
wrote:

> Jeff:
> 
> A few things to consider (assuming a Sun sever as the NBU master):
> 
> 1.) Are you aware of anything that has changed on your NBU server?
> 2.) Are you aware of anything that has changed with your network?
> (Providing you are doing Ethernet-based backups. If not, what about the
> SAN?)
> 3.) Are you aware of any changes to the policies?
> 
> If no to the above try the following:
> 
> 1.) Find out what Veritas recommends for your environment for these two
> variables:
> NUMBER_DATA_BUFFERS
> SIZE_DATA_BUFFERS
> These are found in /usr/openv/netbackup/db/config. They may not give
> them to you if you open a ticket with the solution center (Professional
> Services). Ask around if they do not.
> 
> 2.) Check the network driver settings for a few things. This depends on
> the network type you are using. 100Mb-switched, 10Mb-switched, etc.
> 
> root[prod-backup:/]# ndd -get /dev/qfe adv_autoneg_cap
> 1
> root[prod-backup:/]# ndd -get /dev/qfe adv_100hdx_cap
> 1
> root[prod-backup:/]# ndd -get /dev/qfe adv_100fdx_cap
> 1
> What the output above is stating is that the qfe driver is set at 100
> half and full duplex, and autonegotiate. Once you know how the network
> driver is configured go to your network guys and ask them to see how the
> port on the switch is configured (unless you are the network guy). If
> the port is NOT set to 100-full or autonegotiate have them set it
> accordingly.
> 
> 3.) Reseat the RJ-45 connectors for the physical connections.
> 
> These are some things that have bit us in the past.
> 
> Good luck,
> 
> Bill
> 
> --------------------------------------------------------
>      Bill Jorgensen
>      CSG Systems, Inc.
>      (w) 303.200.3282
>      (p) 303.947.9733
> --------------------------------------------------------
>      UNIX... Spoken with hushed and
>      reverent tones.
> --------------------------------------------------------
> 
> -----Original Message-----
> From: veritas-bu-admin at mailman.eng.auburn.edu
> [mailto:veritas-bu-admin at mailman.eng.auburn.edu] On Behalf Of Jeff
> McCombs
> Sent: Wednesday, March 23, 2005 6:51 AM
> To: veritas-bu at mailman.eng.auburn.edu
> Subject: [Veritas-bu] Backups slow to a crawl
> 
> Gurus,
> 
>     NB 5.0 MP4, single combination media/master server, Solaris 9.
> Overland
> Neo 2000 26-slot 2 drive DLT.
> 
>     I'm noticing that for some reason or another, all of my client
> backups
> have slowed to a _crawl_. A _cumulative_ (!) backup of local disk on a
> Sun
> V100 is taking somewhere on the order of 2 hours at this point, and with
> over 40 systems, I'm blowing past my window  consistently.
> 
>     I'm not quite sure what's going on here, but as I sit and watch the
> output from 'iostat', I'm noticing that rmt/1 (the 2nd drive in the Neo)
> is
> fluxuating between 100% busy, with kw/s at close to zero, and busy @
> 1-15%
> and kw/s up into the 1000's.
> 
>     rmt/0 seems to be fine, kw/s sits consistently up in the 1.8-2K
> range,
> while busy is anywhere from 2% - 30% on average. My other disks aren't
> working hard, CPU isn't loaded and I've got plenty of memory.
> 
>     The policy I'm using allows for multiple datastreams, no limits on
> jobs,
> and most schedules allow for an MPX of 2. I'm backing up
> ALL_LOCAL_DRIVES on
> all clients, and I'm not using any NEW_STREAM directives. I'm not seeing
> any
> errors on the media either.
> 
>     Can anyone shed some light on what might be happening here? Am I
> looking
> at a drive that might be having some problems, or am I barking up the
> wrong
> tree, and it's something else entirely?
> 
>     A small sample of iostat output covering the affected devices is
> below.
> 
> sample (extra disks removed from putput);
> root at backup(pts/1):~# iostat -nx 1 100
>                     extended device statistics
>     r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
>     0.0    4.1    0.0  252.2  0.0  0.0    0.0    5.9   0   2 rmt/0
>     0.0    4.6    0.0  278.4  0.0  0.1    0.0   27.3   0  12 rmt/1
> 
>     0.0    4.1    0.0  252.3  0.0  0.0    0.0    5.9   0   2 rmt/0
>     0.0    4.6    0.0  278.4  0.0  0.1    0.0   27.3   0  12 rmt/1
> 
>     0.0   33.0    0.0 2076.4  0.0  0.2    0.0    5.8   0  19 rmt/0
>     0.0    2.0    0.0  125.8  0.0  1.0    0.0  490.0   0  98 rmt/1
> 
>     0.0   38.0    0.0 2394.0  0.0  0.2    0.0    5.4   0  21 rmt/0
>     0.0    8.0    0.0  504.0  0.0  1.0    0.0  124.9   0 100 rmt/1
> 
>     0.0   27.0    0.0 1701.1  0.0  0.2    0.0    6.5   0  17 rmt/0
>     0.0    2.0    0.0  126.0  0.0  1.0    0.0  499.9   0 100 rmt/1
> 
>     0.0   33.0    0.0 2078.9  0.0  0.2    0.0    5.3   0  18 rmt/0
>     0.0    0.0    0.0    0.0  0.0  1.0    0.0    0.0   0 100 rmt/1
> 
>     0.0   16.0    0.0 1008.0  0.0  0.1    0.0    6.2   0  10 rmt/0
>     0.0   13.0    0.0  819.0  0.0  0.6    0.0   48.4   0  63 rmt/1
> 
>     0.0   40.0    0.0 2520.1  0.0  0.2    0.0    5.9   0  24 rmt/0
>     0.0    0.0    0.0    0.0  0.0  1.0    0.0    0.0   0 100 rmt/1
> 
>     0.0   33.0    0.0 2078.9  0.0  0.2    0.0    5.3   0  18 rmt/0
>     0.0   10.0    0.0  630.0  0.0  1.0    0.0   99.9   0 100 rmt/1
> 

-- 
Jeff McCombs                 |                                    NIC, Inc
Systems Administrator        |                       http://www.nicusa.com
jeffm at nicusa.com             |                                NASDAQ: EGOV
Phone: (703) 909-3277        |        "NIC - the People Behind eGovernment"
--
    "My favorite thing about the internet, is that you get to go into
     the private world of real creeps without having to smell them."
            - Penn Jillett.





More information about the Veritas-bu mailing list