DBAs wanting to create a 10g Real Applications Cluster face many configurationdecisions. One of the more potentially confusing decisions involves the choice offilesystems. Gone are the days when DBAs simply had to choose between "raw" and"cooked". DBAs setting up a 10g RAC can still choose raw devices, but they alsohave several filesystem options, and these options vary considerably from platformto platform. Further, some storage options cannot be used for all the files in theRAC setup. This article gives an overview of the RAC storage options available.RAC Review----------Let's begin by reviewing the structure of a Real Applications Cluster. Physically,a RAC consists of several nodes (servers), connected to each other by a privateinterconnect. The database files are kept on a shared storage subsystem, wherethey're accessible to all nodes. And each node has a public network connection.In terms of software and configuration, the RAC has three basic components:cluster software and/or Cluster Ready Services, database software, and a method ofmanaging the shared storage subsystem.The cluster software can be vendor-supplied or Oracle-supplied, depending onplatform. Cluster Ready Services, or CRS, is a new feature in 10g. Where vendorclusterware is used, CRS interacts with the vendor clusterware to coordinatecluster membership information; without vendor clusterware, CRS, which is alsoknown as Oracle OSD Clusterware, provides complete cluster management.The database software is Oracle 10g with the RAC option, of course.Finally, the shared storage subsystem can be managed by one of the followingoptions: raw devices; Automatic Storage Management (ASM); Vendor-supplied clusterfile system (CFS), Oracle Cluster File System (OCFS), or vendor-supplied logicalvolume manager (LVM); or Networked File System (NFS) on a certified NetworkAttached Storage (NAS) device.Storage OptionsLet me clarify the foregoing alphabet soup with a table:--------------------------------------------------------Table 1. Storage options for the shared storage subsystemAbbrevStorage OptionRawRaw devices, no filesystemASMAutomatic Storage ManagementCFSCluster File SystemOCFSOracle Cluster File SystemLVMLogical Volume ManagerNFSNetwork File System (must be on certified NAS device)Before I delve into each of these storage options, a word about file types. Aregular single-instance database has three basic types of files: database softwareand dump files; datafiles, spfile, control files and log files, often referred toas "database files"; and it may have recovery files, if using RMAN. A RAC databasehas an additional type of file referred to as "CRS files". These consist of theOracle Cluster Registry (OCR) and the voting disk.Not all of these files have to be on the shared storage subsystem. The databasefiles and CRS files must be accessible to all instances, so must be on the sharedstorage subsystem. The database software can be on the shared subsystem and sharedbetween nodes; or each node can have its own ORACLE_HOME. The flash recovery areamust be shared by all instances, if used.Some storage options can't handle all of these file types. To take an obviousexample, the database software and dump files can't be stored on raw devices. This
isn't important for the dump files, but it does mean that choosing raw devicesprecludes having a shared ORACLE_HOME on the shared storage device.And to further complicate the picture, no OS platform is certified for all of theshared storage options. For example, only Linux and SPARC Solaris are supportedwith NFS, and the NFS must be on a certified NAS device. The following tablespells out which platforms and file types can use each storage option.Table 2. Platforms and file types able to use each storage optionStorage optionPlatformsFile types supportedFile types not supportedRawAll platformsDatabase, CRSSoftware/Dump files, RecoveryASMAll platformsDatabase, RecoveryCRS, Software/DumpCertified Vendor CFSAIX, HP Tru64 UNIX, SPARC SolarisAllNoneLVMHP-UX, HP Tru64 UNIX, SPARC SolarisAllNoneOCFSWindows, LinuxDatabase, CRS, RecoverySoftware/Dump filesNFSLinux, SPARC SolarisAllNone(Note: Mike Ault and Madhu Tumma have summarized the storage choices by platformin more detail in this excerpt from their recent book, Oracle 10g Grid Computingwith RAC, which I used as one source for this table.)Now that we have an idea of where we can use these storage options, let's examineeach option in a little more detail. We'll tackle them in order of Oracle'srecommendation, starting with Oracle's least preferred, raw devices, and finishingup with Oracle's top recommendation, ASM.Raw devicesRaw devices need little explanation. As with single-instance Oracle, eachtablespace requires a partition. You will also need to store your software anddump files elsewhere.Pros: You won't need to install any vendor or Oracle-supplied clusterware oradditional drivers.Cons: You won't be able to have a shared oracle home, and if you want to configurea flash recovery area, you'll need to choose another option for it. Manageablilityis an issue. Further, raw devices are a terrible choice if you expect to resize oradd tablespaces frequently, as this involves resizing or adding a partition.NFSNFS also requires little explanation. It must be used with a certified NAS device;Oracle has certified a number of NAS filers with its products, including productsfrom EMC, HP, NetApp and others. NFS on NAS can be a cost-effective alternative toa SAN for Linux and Solaris, especially if no SAN hardware is already installed.Pros: Ease of use and relatively low cost.Cons: Not suitable for all deployments. Analysts recommend SANs over NAS forlarge-scale transaction-intensive applications, although there's disagreement onhow big is too big for NAS.Vendor CFS and LVMsIf you're considering a vendor CFS or LVM, you'll need to check the 10g RealApplication Clusters Installation Guide for your platform and the Certify pages onMetaLink. A discussion of all the certified cluster file systems is beyond thescope of this article. Pros and cons depend on the specific solution, but somegeneral observations can be made:Pros: You can store all types of files associated with the instance on the CFS /logical volumes.Cons: Depends on CFS / LVM. And you won't be enjoying the manageability advantage
of ASM.oOCFSOCFS is the Oracle-supplied CFS for Linux and Windows. This is the only CFS thatcan be used with these platforms. The current version of OCFS was designedspecifically to store RAC files, and is not a full-featured CFS. You can storedatabase, CRS and recovery files on it, but it doesn't fully support genericfilesystem operations. Thus, for example, you cannot install a shared ORACLE_HOMEon an OCFS device.The next version of OCFS, OCFS2, is currently out in beta version and will supportgeneric filesystem operations, including a shared ORACLE_HOME.Pros: Provides a CFS option for Linux and Windows.Cons: Cannot store regular filesystem files such as Oracle software. Easier tomanage than raw devices, but not as manageable as NFS or ASM.ASMOracle recommends ASM for 10g RAC deployments, although CRS files cannot be storedon ASM. In fact, RAC installations using Oracle Database Standard Edition must useASM.AASM is a little bit like a logical volume manager and provides many of thebenefits of LVMs. But it also provides benefits LVMs don't: file-levelstriping/mirroring, and ease of manageability. Instead of running LVM software,you run an ASM instance, a new type of "instance" that largely consists ofprocesses and memory and stores its information in the ASM disks it's managing.Pros: File-level striping and mirroring; ease of manageability through Oraclesyntax and OEM.Cons: ASM files can only be managed through an Oracle application such as RMAN.This can be a weakness if you prefer third-party backup software or simple backupscripts. Cannot store CRS files or database software.ConclusionWe've seen that there's an array of storage options for the shared storage devicein your RAC. These options depend on your platform, and many options don't storeall types of database files, meaning they have to be used in conjunction withanother option. For example, a DBA wanting to use ASM to store database filesmight take a 12-disk SAN, create 11 ASM disks for the database files and flashrecovery area, leave the 12th disk raw and store CRS files on it, and maintainseparate ORACLE_HOMEs on the non-shared disks on each node.Table 3: Sample Disk Configuration w/ ASMIn shared storage subsytem:SAN Disk#In each node:Internal Disk#12345678910111212ASM disksDB files; flash recoveryRAWCRSOS filesORAHOMEWhen weighing the shared storage device options for your platform, start with theOracle Real Application Clusters Installation and Configuration Guide for yourplatform, available from OTN. Section III has a platform-specific discussion ofstorage options. Be sure to check the certification matrices on MetaLink as well.
of 3