Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision | |||
technical:generic:caviness-lustre-rebalance [2021-02-23 16:54] – frey | technical:generic:caviness-lustre-rebalance [2021-02-23 17:13] (current) – [Testing] frey | ||
---|---|---|---|
Line 70: | Line 70: | ||
===== Testing ===== | ===== Testing ===== | ||
- | All aspects of this workflow were tested using VirtualBox on a Mac laptop. | + | All aspects of this workflow were tested using VirtualBox on a Mac laptop. |
+ | |||
+ | The four VMs each had a virtual NIC configured in a named internal network ('' | ||
+ | <code bash> | ||
+ | [mds0 ~]$ modprobe lnet | ||
+ | [mds0 ~]$ lnetctl net configure --all | ||
+ | </ | ||
+ | |||
+ | The following VDIs were created: | ||
* 50 GB - mgt | * 50 GB - mgt | ||
* 250 GB - mdt0, mdt1 | * 250 GB - mdt0, mdt1 | ||
Line 76: | Line 84: | ||
The mgt and mdt0 VDIs were attached to mds0 and formatted: | The mgt and mdt0 VDIs were attached to mds0 and formatted: | ||
<code bash> | <code bash> | ||
- | $ mkfs.lustre --mgs --reformat \ | + | [mds0 ~]$ mkfs.lustre --mgs --reformat \ |
--servicenode=mds0@tcp --mgsnode=mds1@tcp \ | --servicenode=mds0@tcp --mgsnode=mds1@tcp \ | ||
--backfstype=ldiskfs \ | --backfstype=ldiskfs \ | ||
/dev/sdb | /dev/sdb | ||
- | + | [mds0 ~]$ mkfs.lustre --mdt --reformat \ | |
- | $ mkfs.lustre --mdt --reformat \ | + | |
--mgsnode=mds0@tcp --mgsnode=mds1@tcp \ | --mgsnode=mds0@tcp --mgsnode=mds1@tcp \ | ||
--servicenode=mds0@tcp --mgsnode=mds1@tcp \ | --servicenode=mds0@tcp --mgsnode=mds1@tcp \ | ||
--backfstype=ldiskfs --fsname=demo \ | --backfstype=ldiskfs --fsname=demo \ | ||
- | /dev/sdc1 | + | /dev/sdc |
</ | </ | ||
The ost0 VDI was attached to oss0 and formatted: | The ost0 VDI was attached to oss0 and formatted: | ||
<code bash> | <code bash> | ||
- | $ mkfs.lustre --ost --reformat --index=0 \ | + | [oss0 ~]$ mkfs.lustre --ost --reformat --index=0 \ |
--mgsnode=mds0@tcp --mgsnode=mds1@tcp \ | --mgsnode=mds0@tcp --mgsnode=mds1@tcp \ | ||
--servicenode=oss0@tcp --mgsnode=oss1@tcp \ | --servicenode=oss0@tcp --mgsnode=oss1@tcp \ | ||
Line 95: | Line 102: | ||
/dev/sdb | /dev/sdb | ||
</ | </ | ||
+ | The mgt and mdt0 were brought online: | ||
+ | <code bash> | ||
+ | [mds0 ~]$ mkdir -p /lustre/mgt / | ||
+ | [mds0 ~]$ mount -t lustre /dev/sdb /lustre/mgt | ||
+ | [mds0 ~]$ mount -t lustre /dev/sdc / | ||
+ | </ | ||
+ | Finally, ost0 was brought online: | ||
+ | <code bash> | ||
+ | [oss0 ~]$ mkdir -p / | ||
+ | [oss0 ~]$ mount -t lustre /dev/sdb / | ||
+ | </ | ||
+ | |||
+ | ==== Client Setup ==== | ||
+ | |||
+ | Another VM was created with the same version of CentOS 7 and the Lustre 2.10.3 client modules. | ||
+ | |||
+ | The " | ||
+ | <code bash> | ||
+ | [client ~]$ mkdir /demo | ||
+ | [client ~]$ mount -t lustre mdt0@tcp: | ||
+ | </ | ||
+ | |||
+ | At this point, some tests were performed in order to fill the metadata to approximately 70% of capacity. | ||
+ | |||
+ | ==== Addition of MDT ==== | ||
+ | |||
+ | The new MDT was formatted and brought online: | ||
+ | <code bash> | ||
+ | [mds1 ~]$ mkfs.lustre --mdt --reformat --index=1 \ | ||
+ | --mgsnode=mds0@tcp --mgsnode=mds1@tcp \ | ||
+ | --servicenode=mds1@tcp --mgsnode=mds0@tcp \ | ||
+ | --backfstype=ldiskfs --fsname=demo \ | ||
+ | /dev/sdb | ||
+ | [mds1 ~]$ mkdir -p /lustre/mgt / | ||
+ | [mds1 ~]$ mount -t lustre /dev/sdb / | ||
+ | </ | ||
+ | |||
+ | After a few moments, the client VM received the updated file system configuration and had mounted the new MDT. MDT usage and capacity changed accordingly. | ||
+ | |||
+ | Further testing was performed to confirm that | ||
+ | * by default all metadata additions were against mdt0 | ||
+ | * creating a new directory with metadata striping over mdt0 and mdt1 initially allowed a balanced creation of new files across both MDTs | ||
+ | * once mdt0 was filled to capacity, creation of new files whose name hashed and mapped to mdt0 failed; names that hashed and mapped to mdt1 succeeded | ||
+ | |||
+ | ==== Addition of OST ==== | ||
+ | |||
+ | The new OST was formatted and brought online: | ||
+ | <code bash> | ||
+ | [mds1 ~]$ mkfs.lustre --ost --reformat --index=1 \ | ||
+ | --mgsnode=mds0@tcp --mgsnode=mds1@tcp \ | ||
+ | --servicenode=oss1@tcp --mgsnode=oss0@tcp \ | ||
+ | --backfstype=ldiskfs --fsname=demo \ | ||
+ | /dev/sdb | ||
+ | [oss1 ~]$ mkdir -p / | ||
+ | [oss1 ~]$ mount -t lustre /dev/sdb / | ||
+ | </ | ||
+ | |||
+ | After a few moments, the client VM received the updated file system configuration and had mounted the new OST. OST usage and capacity changed accordingly. | ||
+ |