| Both sides previous revision Previous revision Next revision | Previous revision |
| abstract:caviness:filesystems:lustre [2020-05-29 13:36] – frey | abstract:caviness:filesystems:lustre [2024-09-25 14:33] (current) – bdeng |
|---|
| |
| The Caviness cluster contains multiple //Object Storage Targets// (OSTs) in each rack that each contain many hard disks. For example, ''ost0'' contains 10 SATA hard disks (8 TB each, 1 hot spare) managed as a ZFS storage pool with an SSD acting as a read cache for improved performance: | The Caviness cluster contains multiple //Object Storage Targets// (OSTs) in each rack that each contain many hard disks. For example, ''ost0'' contains 10 SATA hard disks (8 TB each, 1 hot spare) managed as a ZFS storage pool with an SSD acting as a read cache for improved performance: |
| |
| {{ :abstract:caviness:filesystems:caviness-lustre-oss_ost.png?400 |Example image of Caviness Lustre OSS/OST. }} | {{ :abstract:caviness:filesystems:caviness-lustre-oss_ost.png?400 |Example image of Caviness Lustre OSS/OST. }} |
| |
| For large files or files that are internally organized as "records((A //record// consists of a fixed-size sequence of bytes; the //i//-th record exists at an easily calculated offset within the file.))" i/o performance can be further improved by //striping// the file across multiple OSTs. Striping divides a file into a set of sequential, fixed-size chunks. The stripes are distributed round-robin to //N// unique Lustre objects -- and thus on //N// unique OSTs. For example, consider a 13 MiB file: | For large files or files that are internally organized as "records((A //record// consists of a fixed-size sequence of bytes; the //i//-th record exists at an easily calculated offset within the file.))" i/o performance can be further improved by //striping// the file across multiple OSTs. Striping divides a file into a set of sequential, fixed-size chunks. The stripes are distributed round-robin to //N// unique Lustre objects -- and thus on //N// unique OSTs. For example, consider a 13 MiB file: |
| |
| {{ lustre-striping.png?500 |Lustre striping.}} | {{ :abstract:caviness:filesystems:caviness-lustre-striping.png?500 |Lustre striping.}} |
| |
| Without striping, all 13 MiB of the file resides in a single object on ''OST0001'' (see (a) above). All i/o with respect to this file is handled by ''OSS1''; appending 5 MiB to the file will grow the object to 18 MiB. | Without striping, all 13 MiB of the file resides in a single object on ''ost0'' (see (a) above). All i/o with respect to this file is handled by ''oss0''; appending 5 MiB to the file will grow the object to 18 MiB. |
| |
| With a stripe count of three and size of 4 MiB, the Lustre filesystem pre-allocates three objects on unique OSTs on behalf of the file (see (b) above). The file is split into sequential segments of 4 MiB -- a stripe -- and the stripes are written round-robin to the objects allocated to the file. In this case, appending 5 MiB to the file will see stripe 3 extended to a full 4 MiB and a new stripe of 2 MiB added to the object on ''OST0007''. For large files and record-style files, striping introduces another level of parallelism that can dramatically increase the performance of programs that access them. | With a stripe count of three and size of 4 MiB, the Lustre filesystem pre-allocates three objects on unique OSTs on behalf of the file (see (b) above). The file is split into sequential segments of 4 MiB -- a stripe -- and the stripes are written round-robin to the objects allocated to the file. In this case, appending 5 MiB to the file will see stripe 3 extended to a full 4 MiB and a new stripe of 2 MiB added to the object on ''ost1''. For large files and record-style files, striping introduces another level of parallelism that can dramatically increase the performance of programs that access them. |
| |
| <note tip>File striping is established when the file is created. Use the ''lfs setstripe'' command to pre-allocate the objects for a striped file: ''lfs setstripe -c 4 -s 8m my_new_file.nc'' would create the file ''my_new_file.nc'' containing zero bytes with a stripe size (-s) of 8 MiB and striped across four objects (-c).</note> | <note tip>File striping is established when the file is created. Use the ''lfs setstripe'' command to pre-allocate the objects for a striped file: ''lfs setstripe -c 4 -S 8m my_new_file.nc'' would create the file ''my_new_file.nc'' containing zero bytes with a stripe size (-s) of 8 MiB and striped across four objects (-c).</note> |
| |
| <note tip>Once a file has been created its striping cannot be changed. However, creating a new file with ''lfs setstripe'' and copying the contents of the old file into it effectively changes the data's striping pattern.</note> | <note tip>Once a file has been created its striping cannot be changed. However, creating a new file with ''lfs setstripe'' and copying the contents of the old file into it effectively changes the data's striping pattern.</note> |
| |