Bandwidth with SAS expanders and RAID controllers

itemprop="text">

Having a RAID controller
that has eight internal SATA3 lanes, you can get 6 Gb/s on all eight drives. What if I
connect a 24 port SAS expander to an eight port RAID controller, do I still get max
throughput of 8 x 6 Gb/s, or am I able to get 24 x 6 Gb/s, assuming the expander is
rated for 6 Gb/s on all ports?

Of course the
PCIe bandwidth is going to limit it, as well as the RAID controller but is this right
theoritically speaking? PCIe 2.0 x8 has bandwidth of 4000 MB/s and PCIe 3.0 x8 has 7880
MB/s.

As an example, I was thinking of buying
LSI MegaRAID 9271-8i for my home server. It has eight internal SATA 6 Gb/s lanes. With
that one I am able to connect eight hard drives and they can work on their limits in
terms of transfer rates because there is one 6 Gb/s lane for each drive available. But
in the future the storage capacity might be too low. I thought I could just add a SAS
expander, like Intel RES2SV240. It is a 24 port expander rated for 6 Gb/s per port. So
do I get the full potential out of the expander, to have 6 Gb/s connection for all the
possible 24 drives? If so, could I buy the 9271-4i (has only four internal SATA ports)
and the Intel SAS expander to be able to connect up to 24 hard drives and have them work
at their full speed?

itemprop="text">

class="normal">Answer

Yes and
no...

Think about this: Your disks will not
perform at 6Gbps (unless they're SSDs). So some level of oversubscription is okay when
you go to using a SAS expander.

A
more common scenario is the use of an external JBOD storage enclosure. Those usually
have 1 or 2 x 4-lane SAS connectors linking them to the main server. Let's assume 4 x
6Gbps, so 24Gbps total bandwidth. Things are definitely oversubscribed there, as you may
have 24 disks linked at 6Gbps... but recall that most disks won't be able to achieve
more than 1.5 or 2Gbps in practice, so that level of oversubscription is
okay.

Remember, 6Gbps is just a link speed. You
will not be able to achieve that through an expander to all connected disks, because the
expander has upstream connections to a RAID controller. The RAID controller is the
limiting factor here.

See:
href="https://serverfault.com/questions/506583/do-sas-expanders-work-transparently-with-sas-controllers">Do
SAS expanders work transparently with SAS controllers?
/> href="https://serverfault.com/questions/510442/how-exactly-does-a-sas-sff-8087-breakout-cable-work-raid-connection-questions/510445#510445">How
exactly does a SAS SFF-8087 breakout cable work? + RAID/connection
questions