///!SITE UNDER CONSTRUCTION!///

A Quick Lustre Installation and Configuration Guide

This guide is intended for RHEL based systems. The latest Lustre compatible version, as of writing this, is version 8.7 so that is what will be used. This guide assumes a fairly standard installation of the operating system. It also assumes a user familiar with RHEL systems and Linux in general. It will not cover every step, nor will it cover every possible configuration. The official Lustre documentation should be consulted for more advanced usage. I will not be taking any questions.

Sections:

Basic Concepts:

Lustre Software

Lustre is a filesystem/clustering/networking tool for creating highly configurable storage setups. It has features like clustering, active/passive and active/active failure modes, network wide file striping, adaptive rerouting, Infiniband and RDMA support, ZFS, and more. The Lustre website has more detailed information about features and capabilities.

Lustre is a kernel module for Linux and as a result, it does require specific versions of the kernel alongside specific distributions in order to work. Check here for more information as to which systems are supported.

Server Types

Lustre divides the duties of the filesystem into several distinct groups which allows different servers to specialize for each of these duties. The three types are as follows:

Each function can be performed by one or more servers, and multiple servers can perform the same function. This allows for features like failover and data striping. A single server can also perform multiple functions. Having a combined MGS/MDS setup is not uncommon as the MGS requirements are low.

LNet

LNet is a Lustre specific communication protocol used for pretty much everything. It allows you to set which physical adapters go to which peers, specify optimal or preferred routes, as well as bond multiple connections. There's also more stuff it can do, but there's a LOT to go over.

Prerequisites:

As of right now, RHEL version 8.7 is the latest supported operating system.

The following software are dependencies of Lustre and must be installed:

The high availability, extras, and powertools repositories are also required.

yum-config-manager --enable ha powertools extras

Install epel-release now that the needed repositories have been enabled.

Installation:

Gathering the Software

Make sure to use the correct version of Lustre for your version of RHEL. You MUST use the kernel provided by Lustre in order to have it work.

Go here to download the latest RPMs for the Lustre server packages:

https://downloads.whamcloud.com/public/lustre/latest-release/el8.7/server/RPMS/x86_64/

You'll also need the e2fsprogs packages located here:

https://downloads.whamcloud.com/public/e2fsprogs/latest/el8/RPMS/x86_64/

Packages marked debugsource or tests are not required. These packages are fairly large at about 900MB and likely should not be used on production servers. Additionally, if you are not using ZFS, do not download any packages with zfs in the name, nor should you download lustre-all-dkms. Otherwise, you'll need to grab them in order to use ZFS.

Kernel Setup

In the directory that you've placed all of the packages use localinstall to install the new kernel. This may take a bit of time.

yum localinstall kernel* -y

Now that the new kernel has been installed, reboot and confirm that the Lustre kernel is loaded. For example:

uname -r

Should return somethine like this:

4.18.0-425.3.1.el8_lustre.x86_64

If not, you may need to tweak which kernel is automatically selected.

Misc Libraries and ZFS

Run the following commands to install the rest of the software. Note the order, it takes into account some dependency needs. This may take a bit of time as modules will need to compile. If you aren't using ZFS, simply leave out libzpool*, zfs*, libzfs*, and replace lustre-all-dkms* with lustre-ldiskfs-dkms*.

yum localinstall kmod-lustre-2.* -y
yum localinstall e2fsprogs-* libcom* libss* -y
yum localinstall zfs* libuutil* libzpool* libnvpair* libzfs* -y
yum localinstall lustre-all-dkms* lustre-debuginfo* -y
yum lcoalinstall lustre-2* lustre-osd-* lustre-resource-agents-* -y

Host Configuration

Lustre requires a unique host ID be set on the platform. Run the following two commands to generate one:

hid=`[ -f /etc/hostid ] && od -An -tx /etc/hostid|sed 's/ //g'`
[ "$hid" = `hostid` ] || genhostid

Load the newly installed modules to confirm that they are working:

modprobe zfs
modprobe lustre
modprobe lnet

If all three of those worked, place them into /etc/modules-load.d/lustre.conf. The file likely will not exist, so you'll need to make it. This will ensure that they load on boot. ksocklnd isn't a part of Lustre as far as I know, but it is needed for lnet to load. The contents of the file should look like this:

ksocklnd
zfs
lustre
lnet

Make sure to enable the Lustre services with systemd.

systemctl enable lustre
systemctl enable lnet

The software installation should be complete now. Reboot the system to veryify that the modules load automatically and that no errors occur. Check dmesg and the status of the services for errors. If you see -ENOENT from LNetError in dmesg, that's okay for now. This will be resolved when the disks and network are set up.

Disk Setup:

Creating the MDT

To create an MDT for a server:

mkfs.lustre --mgs --servicenode 10.0.0.50@tcp1 --backfstype=zfs z_mdt/p

This tells Lustre that the MGS is 10.0.0.50 and that it will be found at 10.0.0.50 on the tcp1 connection. The filesystem in use is ZFS, hence the --backfstype=zfs. The zpool selected is z_mdt and the partition to use is p.

Network Setup: