SOC limit (was Re: vxva vs. ODS (was Re: RSM 2000 list) )

[ft@angels.att.com]

On Sun, 1 Jun 1997, Doug Hughes wrote:
> Note for Sun people: It would probably be a good idea to make a note
> about not supporting non SSA disks for RAID-5 and RAID-0 to the web page
> without a purchase of the full VxVM release.  He does have a legitimate
> complaint about not being warned before hand (until the product was
> in hand - which then he could have found out in Chapter1)

I have a tangentially related issue regarding proper advertising, which is
about two years old - but I wonder if anyone else was bitten by this.

In the early days of our use of the SSA100 series, we had few enough (only
about 60) that the distribution of arrays to hosts did not require
"doubling up" arrays on SOCHAs (e.g. we only used one port on each host
card).  When we moved to a much larger configuration, however, we became
slot limited on the server side, which forced us to utilize both FC ports
on the SOCHAs if we wanted to meet our throughput goals and maintain a
reasonable processing to i/o ratio, dictated by our application needs.

To this point, we had worked fairly hard to understand the performance
characteristics of the arrays, and were able to sustain 18.5 to 20 MB/s
from each array for large sequential reads, which dominate this particular
application.  Our primary goal was sustaining the highest possible
sequential scan rate of our filesystems.

Given that the SOCHA was touted as having "two 25 MB/s full duplex" fibre
channel modules; we naively thought that, given our empirical ability to
sustain roughly 19 MB/s with one SSA112; we would be able to sustain twice
that with two arrays on a single SOCHA.  In fact, the "SPARCstorage Array
Architecture" whitepaper from 1996 says:

"Sun developed a Serial Optical Channel (SOC) ASIC that can service two
full-duplex, SBus to Fibre Channel conversions at 50 MB/second."

This is true in some respects, but is missing one vital detail which
governs real life.  It was not long after we started testing that we
discovered that the combined traffic of two SSA112's on a single SOCHA was
limited to about 27 MB/s. When testing each port alone, we got our typical
19 MB/s; but when sustaining read traffic to both arrays, it dropped to
about 13.5 MB/s per array (at least the balance was fair).

Upon inquiry, we were told that this was indeed a limitation of the SOC
ASIC, which was only able to queue 255 pending i/o requests for both
ports.  Given the typical SSA svc_t under our conditions (for 128k reads),
this translated to about 214.5 IOPS or 26.8 MB/s combined for both ports.
I dredged all the product information I could find and NEVER saw a
reference to this in any of the whitepapers or product literature.  Is
anyone else out there aware of this limit being publicised anywhere that I
should have seen it?

Obviously, there was no way to overcome this limit (which effectively
limited the filesystem throughput that the enterprise servers could
sustain), and we had to live with the degraded rate.  It was not all that
bad, but I must say that I was miffed that this was not in black and white
before we made our purchase decision.  Our anticipation of being able to
sustain about 40 MB/s to each SBus slot was one of the factors that led us
to choose FC over USCSI; but we had been duped.  The lesson learned was
indeed to test before deciding.

I was very glad to see this very fact outlined clearly in Brian Wong's
recent book.  In fact, he says: "In light of these realities, the second
port on each FC host adapter is primarily useful for archival storage
connectivity."  However, this was a bit late for us.

In any case; I think I would warn Sun to be more forthcoming with limits
such as this (or at least the technical data required to forecast them).
Even our SE's were not aware of this practical limit until after we
verified the results through testing, and they subsequently dug into the
engineering groups for confirmation.  Had they published more technical
details on the SOC ASIC (including the 255 command queue length) I could
have worked this out myself.

Live and learn.

Fred

--
Fred True				"My name is Ozymandias, King of Kings:
AT&T World Class Database Development	 Look on my works, ye Mighty,
ft@angels.att.com			 and despair!"
						-P. B. Shelley