Take Disk Storage to the Next Level with SAS

Over the next several years, the serial-attached SCSI (SAS) interconnect is expected to replace Fibre Channel (FC) as the high-performance, primary storage drive interconnect of choice in enterprise storage systems.

NetApp started shipping SAS drives as an option in the FAS2000 family a few years ago and recently announced its first SAS-based storage enclosure: the DS4243. This new disk shelf offers the density, flexibility, and reliability necessary for the enterprise with frame array class resiliency.

Tech OnTap asked Chris Lueth of Technical Marketing to sit down with Doug Coatney of the Core Systems’ Storage Engineering team to find out more. We’ve broken their Q&A session into three broad topics:

• SAS technology versus Fibre Channel
• New disk shelf technology
• Out-of-band management capabilities

SAS Technology

Chris Lueth: Why is NetApp transitioning storage infrastructures from FC to SAS?

Doug Coatney: As most Tech OnTap readers are probably aware, the storage industry is already transitioning from Fibre Channel to SAS storage to fill high-performance storage needs. In 2008, almost twice as many SAS disks shipped as FC disks. By 2013, SAS shipments are forecast to be almost 60% of the HDD market, while FC shipments are expected to be concluded altogether.

Figure 1) Actual HDD shipments (2008) and forecast shipments (2013). SAS essentially cannibalizes the market for both Fibre Channel and parallel SCSI disk, while ATA remains relatively stable.

Chris: Does SAS offer technology advantages that are driving this transition?

Doug: Yes, there are three major factors that favor SAS:

• Point-to-point isolation
• Increased bandwidth
• Greatly improved connectivity

These capabilities offer significant advantages for anyone deploying SAS.

Figure 2) FC versus SAS topology.

Point-to-point. SAS uses a point-to point approach with nonblocking switches (expanders) similar to an Ethernet switch. These expanders provide complete isolation for each attached device, thereby avoiding many of the problems that can affect resiliency in loop-based topologies like Fibre Channel-Arbitrated Loop (FC-AL, the protocol used to connect to FC disks). SAS expanders provide path switching, arbitration and central management, and they also serve as traffic cops to make sure that all devices share bandwidth fairly.

Bandwidth. Current FC disk bandwidth is 4Gb/s per port. We have quad port PCIe adapters, so the aggregate FC-AL bandwidth per PCIe slot is 16Gb/s.

The NetApp® DS4243 product supports 3Gb/s SAS links. Looking at the same PCIe slot, a SAS quad adapter has four ports with 12Gb/s aggregate bandwidth each, for a total of 48Gb/s.

A storage-controller-to-disk-subsystem connection is made through a SAS “wide port.” The standard wide port is a set of four SAS lanes working together to provide maximum throughput. With SAS bandwidth of 3Gb/s, grouping four ports together yields a maximum bandwidth of 12Gb/s. This provides a wide pipe for storage systems to move data to and from disk subsystems. These lanes provide greater bandwidth from storage controllers to disk enclosures, plus bandwidth balancing, path redundancy, and improved error recovery, making it possible to safely scale a SAS connection to large numbers of disk drives.

Connectivity. With FC-AL, the theoretical maximum number of drives supported on a loop is 126.

For SAS, the number of drives that can be connected to a single SAS port is limited primarily by performance considerations. In a SAS domain, the use of expanders yields up to 16,384 uniquely addressed devices. A “fan-out” expander can connect to 128 edge expanders, each capable of connecting to 128 disk drives: 128 x 128 = 16,384. (These are theoretical maximums. See the following section for numbers of drives supported with the DS4243.)

Chris: What’s the difference between an FC disk and a SAS disk?

Doug: The main difference between a SAS disk and an equivalent FC disk is in the interface electronics. SAS disk drives do not introduce new technology risks because they are mechanically equivalent to existing FC disk drives, use the same SCSI command set, and have the same performance and reliability characteristics. SAS disk drives simply use a different serial communication protocol.

The NetApp DS4243 Disk Shelf

Figure 3) Front and rear views of the DS4243.

Chris: Can you give our readers an overview of the features of the new NetApp disk shelf?

Doug: The key thing that users should understand about the NetApp DS4243 disk drive expansion shelf is that it was designed by NetApp and optimized for use with NetApp storage systems. The new shelf uses 3Gb/s SAS; SAS wide ports (four lanes of 3Gb/s each) are used to connect the storage controller to the shelf. Up to 24 disk drives (all SAS or all SATA) are supported in a 4U-high rack-mount chassis, improving density by 22% over the previous generation disk subsystem. The shelf uses SAS technology (between the disk shelf and controller as well as within the shelf) and features new out-of-band management functionality, called alternate control path (ACP—more about this in the final section).

Power for the shelf is fully redundant: four power supplies are required for a disk shelf populated with SAS drives, two supplies are required for a disk shelf with SATA. The shelf also contains four 1U bays for electronics modules. Two are occupied by SAS I/O modules (IOM3). The remaining two bays are not currently used.

The mechanical and electrical design is based on the industry-standard Storage Bridge Bay (SBB) specification. SBB offers flexibility and standardization of enclosure interfaces that can help you streamline technology transitions. For example, current DS4243 components can be reused in future SAS-based storage technology.

Chris: What is a SAS “stack”?

Doug: NetApp uses the term “stack” to refer to a collection of correctly wired and interconnected SAS shelves and adapters. Up to 10 shelves are supported per stack for a total of 240 drives, except in the FAS2040 and FAS2050, which support only 4 shelves per stack. SAS and SATA drives can be mixed in the same stack but not in the same shelf.

Chris: How are SATA disk drives accommodated in the new shelf?

Doug: Another potential advantage of SAS over FC is that SAS provides native support for SATA drives through serial tunneling. Any SAS disk subsystem can easily support both SAS and SATA. However, serial tunneling does not support multiple paths to SATA disks, making high-availability (HA) configurations impossible.

To get around this limitation, the SATA drive carriers used in the DS4243 shelves contain a SATA-to-SAS bridge. This means that the native SAS protocol (with multipath capability) is used between the storage controller and every disk, regardless of type.

Chris: What do I need to take advantage of the new disk shelf?

Doug: All you need is a NetApp storage system with an available PCI Express (PCIe) slot, a SAS adapter, the correct version of Data ONTAP®, and the disk shelf hardware and cabling.

The DS4243 is supported on all currently shipping NetApp FAS, V-Series, and SA storage systems except the FAS 2020. If you want to install the new shelf on an existing storage system, you’ll need at least one available PCIe slot (to accommodate a SAS HBA), or you can use the embedded SAS port on a FAS2040. You’ll also need Data ONTAP 7.3.2 or later. Your existing disk drives from the DS14 family of products and the FAS2000 line cannot be used in the DS4243 because the drive carrier configuration is different.

Chris: Are enhanced resiliency features built into the new disk shelf?

Doug: The DS4243 is designed to provide greater resiliency, fault isolation, and recovery than any previous NetApp disk subsystem. In addition to the SAS capabilities already mentioned, it features redundant components and busses, nonvolatile event logs, and nondisruptive firmware updates.
All supported configurations provide either dual-path connections or multipath HA (MPHA) except for the FAS2040, which has limited SAS ports. In addition, the DS4243 features out-of-band management capabilities that provide frame array class resiliency, the first of its kind in a modular storage enclosure. The features complement the in-band resiliency capabilities of the SAS protocol. (See the following section for details.)

Out-of-Band Management with ACP

Chris: Can you talk more about the out-of-band management capabilities that NetApp added to the DS4243?

Doug: For out-of-band management on the DS4243, NetApp has created its Alternate Control Path (ACP) technology. ACP gives you a back door into your disk shelves. It is completely separate from the SAS data path and provides new options for nondisruptive recovery of shelf modules, including the ability to reset or power cycle an individual I/O module (IOM) or an entire domain (that is, all IOMs on the A domain). We designed in the ability to power cycle the entire shelf as well. Each IOM contains an ACP and each shelf contains two IOMs. The new ACP technology enhances the ability of Data ONTAP to automatically reset a misbehaving component in order to return it to a fully operational mode without disruption.

Although NetApp highly recommends it, ACP is not required. Because ACP is completely separate from the data path, the data path continues to function when ACP is not connected or not operational.

Chris: What type of network does ACP use?

Doug: ACP uses a dedicated Ethernet network. Storage shelves are connected by using a daisy-chain topology with a flyback connection for added resilience. IP addresses are automatically assigned by Data ONTAP. (The default ranges can be changed.)

Chris: What are the advantages of configuring ACP?

Doug: ACP enhances data availability by giving a storage controller the ability to reset a storage channel without having to communicate over that channel. If a channel is down or misbehaving, a quick reset can bring it back online without external intervention. ACP allows a storage system to recover from faults that might otherwise require it to reboot, and that’s a big advantage. Other vendors provide this level of capability only in their highest end tier 1 storage, while NetApp enables this feature regardless of the storage platform or tier.

Conclusion

With the release of the DS4243 disk shelf, NetApp has taken another step toward simplifying your storage environment. With the DS4243, you need only one type of disk shelf to accommodate all tiers of storage. You can choose performance-optimized SAS disks or capacity-optimized SATA (or both), and the industry-standard Storage Bridge Bay–based architecture provides flexibility for future expansion.

Doug Coatney Principal Storage Software Engineer NetApp Doug has 23 years of experience in the storage industry, the last 10 of which have been at NetApp. He has played key roles in the development of every disk subsystem NetApp has released during that time. NetApp holds over a dozen patents based on Doug’s work.

Chris Lueth Technical Marketing Engineer NetApp Chris has over 17 years of industry experience. Since joining NetApp more than 6 years ago, he has gained incredible technical breadth, including work on NearStore® deployments, RAID-DP®, SnapLock®, entry, midrange, and high-end platforms, and storage resiliency. He was previously a chip design engineer and worked on the first multiprocessor motherboard chipset before switching to UNIX® system administration and then to storage.