| Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision |
| abstract:caviness:filesystems:lustre [2020-05-29 12:21] – frey | abstract:caviness:filesystems:lustre [2020-05-29 13:13] – frey |
|---|
| With four disks being used in parallel (example (b) above), the block writing overlaps and takes just 8 cycles to complete. | With four disks being used in parallel (example (b) above), the block writing overlaps and takes just 8 cycles to complete. |
| |
| Parallel use of multiple disks is the key behind many higher-performance disk technologies. RAID (Redundant Array of Independent Disks) level 6 uses three or more disks to improve i/o performance while retaining //parity// copies of data((The two parity copies in RAID-6 imply that given //N// 2 TB disks, only //N-2// actually store data. E.g. a three disk RAID-6 volume has a capacity of 2 TB.)). Should one or two of the constituent disks fail, the missing data can be reconstructed using the parity copies. It is RAID-6 that forms the basic building block of the Lustre filesystem on our clusters. | Parallel use of multiple disks is the key behind many higher-performance disk technologies. Caviness makes extensive use of //ZFS storage pools//, which allow multiple physical disks to be used as a single unit with parallel i/o benefits. //Parity data// is constructed when data is written to the pool such that the loss of a hard disk can be sustained: when a disk fails a new disk is substituted and the parity data yields the missing data on that disk. ZFS offers different levels of data redundancy, from simple mirroring of data on two disks to triple-parity that can tolerate the failure of three disks. ZFS double-parity (raid-z2) pools form the basis for the Lustre file system in Caviness. |
| |
| ===== A Storage Node ===== | ===== A Storage Node ===== |
| |
| For example, the Mills cluster contains five //storage appliances// that each contain many hard disks. For example, ''storage1'' contains 36 SATA hard disks (2 TB each) arranged as six 8 TB RAID-6 units: | The Caviness cluster contains multiple //Object Storage Targets// (OSTs) in each rack that each contain many hard disks. For example, ''ost0'' contains 11 SATA hard disks (8 TB each) managed as a ZFS storage pool (with one and an SSD acting as a read cache for improved performance: |
| |
| {{ :abstract:caviness:filesystems:caviness-lustre-oss_ost.png?400 |Example image of Caviness Lustre OSS/OST. }} | {{ :abstract:caviness:filesystems:caviness-lustre-oss_ost.png?400 |Example image of Caviness Lustre OSS/OST. }} |
| |
| Each of the six OST (Object Storage Target) units can survive the concurrent failure of one or two hard disks at the expense of storage space: the raw capacity of ''storage1'' is 72 TB, but the data resilience afforded by RAID-6 costs a full third of that capacity (leaving 48 TB). | Each OST can tolerate the concurrent failure of one or two hard disks at the expense of storage space: the raw capacity of ''storage1'' is 72 TB, but the data resilience afforded by RAID-6 costs a full third of that capacity (leaving 48 TB). |
| |
| The storage appliances are limited in their capabilities: they only function to move blocks of data to and from the disks they contain. In an HPC cluster the storage is shared between many compute nodes. Nodes funnel their i/o requests to the shared storage system by way of the cluster's private network. A dedicated machine called an OSS (Object Storage Server) acts as the middleman between the cluster nodes and the OSTs: | The storage appliances are limited in their capabilities: they only function to move blocks of data to and from the disks they contain. In an HPC cluster the storage is shared between many compute nodes. Nodes funnel their i/o requests to the shared storage system by way of the cluster's private network. A dedicated machine called an OSS (Object Storage Server) acts as the middleman between the cluster nodes and the OSTs: |