<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:v =
"urn:schemas-microsoft-com:vml" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:w =
"urn:schemas-microsoft-com:office:word" xmlns:st1 =
"urn:schemas-microsoft-com:office:smarttags"><HEAD><TITLE>Some info on my experiences with 10GbE</TITLE>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.3790.4064" name=GENERATOR><!--[if !mso]>
<STYLE>v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
</STYLE>
<![endif]--><o:SmartTagType name="place"
namespaceuri="urn:schemas-microsoft-com:office:smarttags"></o:SmartTagType><o:SmartTagType
name="PersonName"
namespaceuri="urn:schemas-microsoft-com:office:smarttags"></o:SmartTagType><!--[if !mso]>
<STYLE>st1\:* {
BEHAVIOR: url(#default#ieooui)
}
</STYLE>
<![endif]-->
<STYLE>@font-face {
font-family: MS Mincho;
}
@font-face {
font-family: Palatino Linotype;
}
@font-face {
font-family: Tahoma;
}
@font-face {
font-family: @MS Mincho;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline
}
P {
FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: "Times New Roman"; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto
}
SPAN.EmailStyle18 {
COLOR: navy; FONT-FAMILY: Arial; mso-style-type: personal-reply
}
DIV.Section1 {
page: Section1
}
</STYLE>
</HEAD>
<BODY lang=EN-US vLink=purple link=blue>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff size=2>I'd be be glad to share...</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff size=2>The OS is sol10 11/06, and I'm running the recommended
patch cluster that was available on 9/12 - kernel patch is
125100-10.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff size=2>For tunables, I've tested quite a few different
permutations of settings for tcp, but I didn't find a whole lot to be gained
from this. Peformance seemed to be best, as long as I was using a TCP
congestion window of 512k or 1024k (sol10 default max is 1024k). In the
end I basically bumped up the max buffer and window sizes to 10MB, enabled
window scaling, and bumped up the connection queues:</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff size=2>tcp_conn_req_max_q
8192<BR>tcp_conn_req_max_q0
8192<BR>tcp_max_buf
10485760<BR>tcp_cwnd_max
10485760<BR>tcp_recv_hiwat
65536<BR>tcp_xmit_hiwat
65536<BR></FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff size=2>The tunables that made a noticable difference regarding
performance are:</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff
size=2>ddi_msix_alloc_limit 8</FONT></SPAN></DIV>
<DIV><SPAN class=591482716-18102007><FONT face=Arial><FONT color=#0000ff
size=2>tcp_squeue_wput
1<BR></FONT></FONT></SPAN><FONT face=Arial color=#0000ff
size=2>ip_soft_rings_cnt
64<BR>ip_squeue_fanout 1</FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2>nxge0
accept_jumbo 1</FONT></DIV>
<DIV><SPAN class=591482716-18102007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=591482716-18102007><FONT face=Arial color=#0000ff size=2>only
one cpu/thread per core is interruptable (set using: psradm -i 1-3 5-7
9-11 13-15)</FONT></SPAN></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV><SPAN class=591482716-18102007><FONT face=Arial color=#0000ff size=2>You
can find Sun's recommended settings for these cards here: <A
href="http://www.solarisinternals.com/wiki/index.php/Networks">http://www.solarisinternals.com/wiki/index.php/Networks</A></FONT></SPAN></DIV>
<DIV><SPAN class=591482716-18102007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=591482716-18102007></SPAN><SPAN class=591482716-18102007><FONT
face=Arial color=#0000ff size=2>
<DIV><FONT size=+0><FONT face=Arial color=#0000ff size=2><SPAN
class=591482716-18102007></SPAN></FONT></FONT> </DIV>
<DIV><FONT size=+0><FONT face=Arial color=#0000ff size=2><SPAN
class=591482716-18102007>Also, the iperf commands that have provided the highest
throughput are:</SPAN></FONT></FONT></DIV>
<DIV><FONT size=+0><FONT face=Arial color=#0000ff size=2><SPAN
class=591482716-18102007></SPAN></FONT></FONT> </DIV>
<DIV><FONT size=+0><FONT face=Arial color=#0000ff size=2><SPAN
class=591482716-18102007>Server: iperf -s -f m -w 512K -l
512K</SPAN></FONT></FONT></DIV>
<DIV><FONT size=+0><FONT face=Arial color=#0000ff size=2><SPAN
class=591482716-18102007>Client: iperf -c <server> -f m -w 512K
-l 512K -t 600 -P <numstreams>
</SPAN></FONT></FONT></DIV></FONT></SPAN></DIV>
<DIV><SPAN class=591482716-18102007></SPAN><SPAN class=591482716-18102007><FONT
face=Arial color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=591482716-18102007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=591482716-18102007><FONT face=Arial color=#0000ff size=2>"Is
rss enabled?" Not sure what you're asking here...</FONT></SPAN></DIV>
<DIV><SPAN class=591482716-18102007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=591482716-18102007></SPAN><SPAN
class=591482716-18102007></SPAN><FONT><FONT face=Arial color=#0000ff
size=2><SPAN class=591482716-18102007></SPAN></FONT></FONT><FONT><FONT
face=Arial color=#0000ff size=2><SPAN
class=591482716-18102007></SPAN></FONT></FONT><FONT><FONT face=Arial
color=#0000ff size=2><SPAN
class=591482716-18102007></SPAN></FONT></FONT><FONT><FONT face=Arial
color=#0000ff size=2><SPAN
class=591482716-18102007></SPAN></FONT></FONT><FONT><FONT face=Arial
color=#0000ff size=2><SPAN
class=591482716-18102007></SPAN></FONT></FONT><FONT><FONT face=Arial
color=#0000ff size=2><SPAN
class=591482716-18102007></SPAN></FONT></FONT> </DIV>
<DIV><FONT><FONT face=Arial color=#0000ff size=2><SPAN
class=591482716-18102007>-devon</SPAN></FONT></FONT></DIV>
<DIV dir=ltr align=left><SPAN class=591482716-18102007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV><FONT face=Arial color=#0000ff
size=2></FONT><FONT face=Arial color=#0000ff size=2></FONT><FONT face=Arial
color=#0000ff size=2></FONT><FONT face=Arial color=#0000ff size=2></FONT><FONT
face=Arial color=#0000ff size=2></FONT><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Curtis Preston
[mailto:cpreston@glasshouse.com] <BR><B>Sent:</B> Thursday, October 18, 2007
1:07 AM<BR><B>To:</B> Peters, Devon C;
VERITAS-BU@mailman.eng.auburn.edu<BR><B>Subject:</B> RE: [Veritas-bu] Some info
on my experiences with 10GbE<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV class=Section1>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">7500 MB/s! That’s
the most impressive numbers I’ve ever seen by FAR. I may have to take back
my “10 GbE is a Lie!” blog post, and I’d be happy to do
so.<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Can you share things
besides the T2000? For example, <o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">what OS and patch
levels are you running?<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Any IP
patches?<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Any IP-specific
patches?<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">What ndd settings are
you using?<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Is rss
enabled?<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">“Input, I need
Input!”<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<DIV>
<P class=MsoNormal><FONT face="Palatino Linotype" color=navy size=3><SPAN
style="FONT-SIZE: 12pt; COLOR: navy; FONT-FAMILY: 'Palatino Linotype'">---<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face="Palatino Linotype" color=navy size=3><SPAN
style="FONT-SIZE: 12pt; COLOR: navy; FONT-FAMILY: 'Palatino Linotype'">W.
<st1:PersonName w:st="on">Curtis Preston</st1:PersonName></SPAN></FONT><FONT
color=navy><SPAN style="COLOR: navy"><o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face="Palatino Linotype" color=navy size=3><SPAN
style="FONT-SIZE: 12pt; COLOR: navy; FONT-FAMILY: 'Palatino Linotype'">Backup
Blog @ <A
href="http://www.backupcentral.com">www.backupcentral.com</A></SPAN></FONT><FONT
color=navy><SPAN style="COLOR: navy"><o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face="Palatino Linotype" color=navy size=3><SPAN
style="FONT-SIZE: 12pt; COLOR: navy; FONT-FAMILY: 'Palatino Linotype'">VP Data
Protection, GlassHouse Technologies</SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<DIV class=MsoNormal style="TEXT-ALIGN: center" align=center><FONT
face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt">
<HR tabIndex=-1 align=center width="100%" SIZE=2>
</SPAN></FONT></DIV>
<P class=MsoNormal><B><FONT face=Tahoma size=2><SPAN
style="FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">From:</SPAN></FONT></B><FONT
face=Tahoma size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">
veritas-bu-bounces@mailman.eng.auburn.edu
[mailto:veritas-bu-bounces@mailman.eng.auburn.edu] <B><SPAN
style="FONT-WEIGHT: bold">On Behalf Of </SPAN></B>Peters, Devon C<BR><B><SPAN
style="FONT-WEIGHT: bold">Sent:</SPAN></B> Wednesday, October 17, 2007 12:12
PM<BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B>
VERITAS-BU@mailman.eng.auburn.edu<BR><B><SPAN
style="FONT-WEIGHT: bold">Subject:</SPAN></B> [Veritas-bu] Some info on my
experiences with 10GbE</SPAN></FONT><o:p></o:p></P></DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"><o:p> </o:p></SPAN></FONT></P>
<P><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Since I've seen a little bit of talk
about 10GbE on here in the past I figured I'd share some of my
experiences...</SPAN></FONT> <o:p></o:p></P>
<P><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">I've recently been testing some of
Sun's dual-port 10GbE NICs on some small T2000's (1Ghz, 4-core). I'm only
using a single port on each card, and the servers are currently directly
connected to each other (waiting for my network team to get switches and fibre
in place).</SPAN></FONT><o:p></o:p></P>
<P><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">So
far, I've been able to drive throughput between these two systems to about
7500Mbit/sec using iperf. When the throughput gets this high, all the
cores/threads on the receiving T2000 become saturated and TCP retransmits start
climbing, but both systems remain quite responsive. Since these are only
4-core T2000's, I would guess that the 6 or 8-core T2000's (especially with
1.2Ghz or 1.4Ghz processors) should be capable of more throughput, possibly near
line speed.</SPAN></FONT><o:p></o:p></P>
<P><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">The
down side achieving this high of throughput is that it requires lots of data
streams. When transmitting with a single data stream, the most throughput
I've gotten is about 1500Mbit/sec. I only got up to 7500Mbit/s when using
64 data streams… Also, the biggest gains seem to be in the jump from 1 to
8 data streams; with 8 streams I was able to get throughput up to
6500Mbit/sec.</SPAN></FONT><o:p></o:p></P>
<P><FONT face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Our
goal for 10GbE, is to be able to restore data from tape at a speed of at least
2400Mbit/sec (300MB/sec). We have large daily backups (3-4TB) that we
would like to be able to restore (not backup) in a reasonable amount of
time. These restores are used to refresh our test and development
environments with current data. The actual backups are done with array
based snapshots (HDS ShadowCopy), which then get mounted and backed up by a
dedicated media server (6-core T2000). We're currently getting about
650MB/sec of throughput with the backups (9 streams on 3 LTO3 tape drives -
MPX=3 and it's very compressible data).</SPAN></FONT><o:p></o:p></P>
<P><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Going off my iperf results, the
restoring this data using 9 streams should get us well over 2400Mbit/sec.
But - we haven't installed the cards on our media servers yet, so I have yet to
see what the actual performanee of netbackup and LTO3 over 10GbE is. I'm
hopeful it'll be close to the iperf results, but if it doesn't meet the goal
then we'll be looking at other options.</SPAN></FONT><o:p></o:p></P>
<P><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">--</SPAN></FONT> <BR><st1:place
w:st="on"><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Devon</SPAN></FONT></st1:place><FONT
face=Arial size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">
Peters</SPAN></FONT> <o:p></o:p></P></DIV></BODY></HTML>