| Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision |
| abstract:farber:filesystems:lustre [2020-03-06 11:17] – [All About Lustre] mkyle | abstract:farber:filesystems:lustre [2020-03-06 11:22] – [The Lustre Filesystem] mkyle |
|---|
| ===== A Storage Node ===== | ===== A Storage Node ===== |
| |
| The Mills cluster contains five //storage appliances// that each contain many hard disks. For example, ''storage1'' contains 36 SATA hard disks (2 TB each) arranged as six 8 TB RAID-6 units: | For example, Mills cluster contains five //storage appliances// that each contain many hard disks. For example, ''storage1'' contains 36 SATA hard disks (2 TB each) arranged as six 8 TB RAID-6 units: |
| |
| {{ osts.png |The Mills storage1 appliance.}} | {{ osts.png |The Mills storage1 appliance.}} |
| {{ oss-osts.png |Cluster nodes send i/o requests to an OSS, which services a set of OSTs.}} | {{ oss-osts.png |Cluster nodes send i/o requests to an OSS, which services a set of OSTs.}} |
| |
| In Mills, each OSS is primarily responsible for one storage appliance's OSTs. As illustrated above, ''OST0000'' through ''OST0005'' are serviced primarily by ''OSS1''. If ''OSS1'' were to fail compute nodes would no longer be able to interact with those OSTs. This situation is tempered by having each OSS act as a //failover// OSS for a secondary set of OSTs. If ''OSS1'' fails, then ''OSS2'' will take control of ''OST0000'' through ''OST0005'' in addition to its own ''OST0006'' through ''OST000B''. When ''OSS1'' is repaired, it can retake control of its OSTs from its partner. | Each OSS is primarily responsible for one storage appliance's OSTs. As illustrated above, ''OST0000'' through ''OST0005'' are serviced primarily by ''OSS1''. If ''OSS1'' were to fail compute nodes would no longer be able to interact with those OSTs. This situation is tempered by having each OSS act as a //failover// OSS for a secondary set of OSTs. If ''OSS1'' fails, then ''OSS2'' will take control of ''OST0000'' through ''OST0005'' in addition to its own ''OST0006'' through ''OST000B''. When ''OSS1'' is repaired, it can retake control of its OSTs from its partner. |
| |
| <note important>The action of an OSS's taking over for its failover partner is not immediate. Usually anywhere from 5 to 10 minutes will pass before the partner OSS has fully assumed control over the OSTs.</note> | <note important>The action of an OSS's taking over for its failover partner is not immediate. Usually anywhere from 5 to 10 minutes will pass before the partner OSS has fully assumed control over the OSTs.</note> |
| * File system capacity is not limited by hard disk size | * File system capacity is not limited by hard disk size |
| |
| The capacity of a Lustre filesystem is the sum of its constituent OSTs, so a Lustre filesystem's capacity can be grown by the addition of OSTs (and possibly OSSs to serve them). For example, should the 172 TB Lustre filesystem on Mills begin to approach its capacity, additional capacity could be added with zero downtime by buying and installing another OSS pair. | The capacity of a Lustre filesystem is the sum of its constituent OSTs, so a Lustre filesystem's capacity can be grown by the addition of OSTs (and possibly OSSs to serve them). For example, should the 172 TB Lustre filesystem begins to reach its capacity, additional capacity could be added with zero downtime by buying and installing another OSS pair. |
| |
| <note important>Creating extremely large filesystems has one drawback: traversing the filesystem takes so much time that it becomes impossible to create off-site backups for further data resilience. For this reason Lustre filesystems are most often treated as volatile/scratch storage.</note> | <note important>Creating extremely large filesystems has one drawback: traversing the filesystem takes so much time that it becomes impossible to create off-site backups for further data resilience. For this reason Lustre filesystems are most often treated as volatile/scratch storage.</note> |