3. Installation - OpenMPDK/SMDK GitHub Wiki

3.1 Prerequisite

Step 1: Clone source code from GitHub

You can access SMDK source code on https://github.com/OpenMPDK/SMDK.

$ cd /path/to/clone
$ git clone https://github.com/OpenMPDK/SMDK.git
Directory name Description
lib A directory with SMDK kernel, allocator, and other OSSs.
script A directory with script to download package dependencies.
src/test A directory with source code and scripts for various test cases.
src/app A directory with application running scripts for IMDB (Redis, Memcached) and ML/AI (BERT, NASNet), etc.

Step 2: Install prerequisite

You can easily download and install the packages required to build and run SMDK kernel, allocator, application, and TCs through def_pkg_install.sh script, which is available for Ubuntu distro.

Note: the script is based on Ubuntu 22.04, depending on the version of Ubuntu package repo or POSIX distro, you may change the package name properly.

$ cd /path/to/SMDK/script/
$ sudo ./dep_pkg_install.sh

3.2 Kernel

Step 1: Build the kernel

You need to build SMDK kernel by referring the kernel config file. The default name of the kernel config file for the SMDK kernel is config-linux-6.9-smdk.

$ cd /path/to/SMDK/lib/
$ ls config-linux-6.9-smdk
config-linux-6.9-smdk

You can change the name tag by modifying the EXTRAVERSION field of the linux-6.9-smdk/Makefile. Edit the Makefile and check the version, patchlevel, sublevel, and extraversion. If those are not accurate, make command would fail. Depending on the system status, it may take some time.

Run $ sudo ./build_lib.sh kernel to build the kernel. It takes a few minutes to complete the kernel compilation. When the message below is shown, the kernel build is done and it is ready to update the kernel.

$ cd /path/to/SMDK/lib/
$ sudo ./build_lib.sh kernel
 
[build linux-6.9-smdk]
 …
 …
 …
[build linux-6.9-smdk]..success

If kernel build fails, run $ sudo ./build_lib.sh clean_kernel, and then build again.

Step 2: Install & update the kernel

Run the following commands to install and update the kernel.

$ cd /path/to/SMDK/lib/linux-6.9-smdk
$ sudo make INSTALL_MOD_STRIP=1 modules_install -j 8
$ sudo make install -j 8
$ sudo update-grub

If the update is successful, reboot the system and select the newly installed kernel from the boot loader menu. The figures below show the GRUB menu at Ubuntu 22.04 and how to select the kernel.

image

image

If SMDK kernel booting fails, it may be because the kernel config used in build is not compatible with your system.

When the kernel booting is done, you can check the CXL device is properly registered and recognized as the Movable Zone by running the command $ cat /proc/buddyinfo.

# cat /proc/buddyinfo
Node 0, zone      DMA      0      0      0      0      0      0      0      0      1      1      3
Node 0, zone    DMA32      7      4      4      4      4      5      3      5      4      4    436
Node 0, zone   Normal      2    852    478    142     71     31      6      3      1      1  13885
Node 1, zone  Movable      0      0      0      0      0      0      0      0      0      0  32768
Node 2, zone  Movable      0      0      0      0      0      0      0      0      0      0  32768
Node 3, zone  Movable      0      0      0      0      0      0      0      0      0      0  32768

How CXL memory is shown in /proc/buddyinfo depends on how you group the CXL devices. Please refer to Memory Partition section. If CXL devices are not detected on your system after the kernel update, check if the necessary CXL driver is loaded properly through the following command.

$ lsmod | grep cxl
cxl_pci                28672  0
cxl_mem                12288  0
cxl_port               16384  0
cxl_pmem               24576  0
cxl_acpi               24576  0
cxl_pmu                32768  0
cxl_core              258048  10 cxl_pmem,cxl_port,kmem,cxl_mem,cxl_pci,dax_cxl,cxl_acpi,cxl_pmu

If the driver is not loaded, run the following commands to insert the modules needed. Also, you may check the Kernel Guide section to check SRAT/e820 information.

$ sudo insmod /lib/modules/{kernel_name}/kernel/drivers/cxl/core/cxl_core.ko
$ sudo insmod /lib/modules/{kernel_name}/kernel/drivers/cxl/cxl_acpi.ko
$ sudo insmod /lib/modules/{kernel_name}/kernel/drivers/cxl/cxl_mem.ko
$ sudo insmod /lib/modules/{kernel_name}/kernel/drivers/cxl/cxl_pci.ko
$ sudo insmod /lib/modules/{kernel_name}/kernel/drivers/perf/cxl_pmu.ko

From v2.0, SMDK supports CMM-DC(a.k.a CXL PNM) device using OneMCC PNMLibrary (OneMCC).

SMDK supports PNM fsim(functional simulator) and Samsung CMM-DC device integration.

Note: If you want to integrate Samsung CMM-DC device, follow the below guides.

  1. Add modprobe.blacklist=dax_hmem to the kernel command line. During system booting, you can add kernel commands by pressing 'e' on the kernel selection grub screen. Please refer to the example of the boot screen above.
  2. In case of real CMM-DC device, hw is installed with device-dax automatically.

You can install fsim/HW using the scripts below. (e.g. IMDB fsim)

$ cd /path/to/SMDK/lib/pnm
$ ./setup_imdb_fsim -i

You can also uninstall fsim/HW using the scripts below. (e.g. DLRM HW)

$ cd /path/to/SMDK/lib/pnm
$ ./setup_dlrm_hw -u

3.3 Allocator Library

After the above steps are completed, build PNM library first as shown below. You need to build this before building other SMDK allocator libraries.

$ cd /path/to/SMDK/lib
$ ./build_lib.sh libpnm

If libpnm is ready, build the SMDK allocator libraries using the commands below.

$ cd /path/to/SMDK/lib
$ ./build_lib.sh libsmdk

When the build process is complete, five library files will be created in the directory /path/to/SMDK/lib/smdk_allocator/lib; libcxlmalloc.a, libcxlmalloc.so, libsmalloc.a, libsmalloc.so and libpnm.so. libcxlmalloc.a and libcxlmalloc.so are so-called compatible API library that enables the SMDK compatible path, whereas libsmalloc.a and libsmalloc.so are so-called optimization API library that works for SMDK optimization path. Please note that your application cannot be built/executed by linking these two libraries (libcxlmalloc and libsmalloc) at the same time. In addition, the py_smdk package for supporting Python3 language binding of the optimization path can be generated at /path/to/SMDK/lib/smdk_allocator/opt_api/py_smdk_pkg using the commands below.

$ cd /path/to/SMDK/lib
$ ./build_lib.sh py_smdk

3.4 Intelligent Tiering Engine

Intelligent Tiering Engine provides means to tier DDR/CXL memory across userspace/kernelspace. Until v1.4, it has offered ways to tier on priority and capacity perspective. From v1.5, it further offers so-called adaptive interleaving feature to lead to DDR/CXL bandwidth aggregation in a software way. In order to use the adaptive interleaving feature, below components are needed

  • monitor : the userspace daemon that monitors live bandwidth status of a system, interacting with kernel driver to maintain the bandwidth map of numa-nodes.
  • kernel : the kernel driver that listens to an on/offline event of a numa-node and notify it to the userspace monitor.
  • allocator : the smdk library that (de)allocate DDR or CXL DRAM. It determines a memory type being allocated, interacting with the monitor module.
  • pmu plugin : the plugin to access cpu pmu(performance monitoring unit) HW to efficiently measure live memory bandwidth. This is architecture dependent.

Note: You must have built the SMDK kernel beforehand and booted it. For kernel build and installation guide, please refer to Kernel section.

$ cd /path/to/SMDK/lib
$ ./build_lib.sh tierd

After a successful build, a daemon binary to create a bandwidth map of each NUMA node, a kernel module to detect hot adds and removals of nodes, and a performance monitoring unit (PMU) library and binary to measure the real-time bandwidth of the system are created; tierd, libpcm.so(Intel)/AMDuProfPcm(AMD).

You can run the following script to see if adaptive interleaving feature is available on your system. Note that in order to run this script, you need to build the SMDK Allocator and change the MLC_PATH in tierd.conf to match your repository path.

$ cd /path/to/SMDK/lib/tierd
# Modify MLC_PATH and AMD_UPROFPCM_PATH(in case of AMD architecture)
$ vi tierd.conf
$ sudo ./check_tierd.sh

TC 1 starts...
  RUN   /path/to/SMDK/src/test/tierd/run_tierd_allocator_test.sh     : PASSED!

TC 2 starts...
  RUN   /path/to/SMDK/src/test/tierd/run_tierd_daemon_test.sh        : PASSED!

TC 3 starts...
  RUN   /path/to/SMDK/src/test/tierd/run_tierd_driver_test.sh        : PASSED!

TC 4 starts...
  RUN   /path/to/SMDK/src/test/tierd/run_tierd_plugin_test.sh        : PASSED!


Total 4 TCs executed:  4  PASSED, 0  FAILED

Note: If running the script fails, check the error log in /path/to/SMDK/lib/tierd/TC_result/. If insmod of the Kernel module fails, make sure you have built and booted the SMDK Kernel. Please refer to Kernel for more details.

3.5 CXL-CLI

Note: From SMDK v1.5, the monitor command of the CXL-CLI is disabled by default. To use the command, please follow below 2 procedures.

  1. Install libtracefs and libtraceevent packages prior to enable the command. But some OS distros (e.g. older than Ubuntu 22.04 or fedora 33, etc.) would fail to find the packages from their repositories. In these cases, please manually build them directly using their source codes.
  2. Change the 'value' of 'libtracefs' to 'enabled' in /path/to/SMDK/lib/cxl_cli/meson_options.txt, and build CXL-CLI again.

CXL-CLI is a tool to interface with a CXL device and SMDK functions, e.g., getting CXL devices lists, CXL node partitioning(grouping), etc. You can build CXL-CLI tool in the following way.

$ cd /path/to/SMDK/lib
$ ./build_lib.sh cxl_cli   #The deliverables are created under "/path/to/SMDK/lib/cxl_cli/build/cxl" by default.

Please note that CXL-CLI tool is being developed and released to be compatible with the latest version of the SMDK kernel. If you have updated the SMDK kernel, please update the CLI tool as well, and vice versa.

Using CXL-CLI, you can also check the status of CXL node partitioning of your system in the following way.

$ ./cxl list -V --list_dev
[
   {
      "id":"cxl0",
      "start_address":"0x2080000000",
      "size":"0x2000000000",
      "node_id":"1",
      "socket_id":"0",
      "state":"online",
      "memdev":
      {
         "memdev_id":"0",
         "pci_bus_addr":"0000:16:00.0",
         "pci_cur_link_speed":"32.0 GT/s PCIe",
         "pci_cur_link_width":"8",
      }
   }
   {
      "id":"cxl1",
      ......
   }
   {
      "id":"cxl2",
      ......
   }
]

CXL-CLI provides a way to check the registration status of CXL devices and configure the CXL device nodes. Basically, by using CXL-CLI you can communicate with CXL devices through the command interfaces defined by the CXL specification. Please refer to CXL-CLI for more details.

3.6 Applications

If you have completed all of the steps above, you are all set to use SMDK. You can link the built SMDK allocator libraries to your applications to take advantage of their functionalities. For more specific instructions on how to do this, please refer to Benchmark and Applications. In addition, SMDK provides sources of some applications that can be used with SMDK. They can be also built through /path/to/SMDK/lib/build_lib.sh script like below.

$ cd /path/to/SMDK/lib
$ ./build_lib.sh {redis|memcached|memtier|qemu|numactl|voltdb|bm}

Note: To build VoltDB v11.3.0 source code in SMDK release, Python 2.X and Java 8 is required. The build would fail on higher version of Python and/or Java.

You can also use the libraries in your container virtualization environment. We provide scripts for creating Docker images for applications such as Redis, Memcached, STREAM and ML/AI applications (BERT, NASNet), as well as simple application execution scripts for using SMDK allocator libraries in those Docker containers. However, please note that these scripts may not work properly on your system and environment. In particular, in case of image creation scripts (Dockerfile and container_create.sh), you would need to modify some contents inside them depending on your system. For more details, please refer to the files in /path/to/SMDK/src/container, /path/to/SMDK/src/app/bert and /path/to/SMDK/src/app/nasnet . Also please refer to Containerization section of Experiment Results chapter to find out how we enabled and utilized SMDK in container environments.