CXL disaggregated memory

/ Publications / By

The Compute Express Link (CXL) is an open, industry-standard high-speed interconnect designed to maintain cache coherency between host processors (CPUs) and attached devices, such as GPUs, specialized AI accelerators, and expansion memory systems. CXL was initially developed by Intel and was launched in March 2019, together with the CXL Consortium, a group of leading technology companies, including Alibaba, Cisco, Dell EMC, Google, HPE, Huawei, Meta, Intel, and Microsoft. CXL is built on top of the physical layer of PCI Express (PCIe), the standard link found inside most computers. Its purpose is to address memory capacity bottlenecks and communication latency in data-intensive cloud and high-performance computing environments. CXL enables host processors and accelerator hardware to share memory dynamically and coherently, thus facilitating memory pooling and disaggregation in data centers.

Figure 1: CXL disaggregated memory deployment in a multi-host environment.

Protecting the shared memory pool
The HIGHER project focuses on protecting these CXL-based disaggregated memory pool resources. It will add extra safeguards on the “memory side” to ensure that ingress memory transactions are legitimate. The threat model analysed in the project is built around adversaries that can manipulate packets transmitted between a host and memory pool using techniques, such as Fault-Injection Attacks (FIAs).

How the system is set up
The figure above shows the deployment setup in a multi-host environment. The Host Platform Module (HPM) uses hypervisors that manage virtual machines (VMs) running user applications. The Fabric Manager is responsible for assigning memory pool resources to VMs that ask for additional memory. Each HPM will be built using a Rhea2 ARM CPU, whereas the memory safeguards will be implemented in an Altera Agilex 7 I-series FPGA that integrates a hardware CXL IP, which exposes a type-3 CXL device. The software stack will comprise:

  1. The host CPU operating system (Linux) with the latest kernel that supports CXL 2.0.
  2. An implementation of a subset of the Fabric Manager (FM) API, used to manage the CXL memory pool.
  3. A hypervisor that will manage VMs running applications.

How a memory request works
Interfacing between these layers will be as follows:

  1. Applications running on VMs may request additional memory from the hypervisor
  2. The hypervisor l informs the FM how much additional memory is needed.
  3. The FM will then send “bind” commands to the CXL memory Pool Manager (CPM), which enables the required segments across the memory pool populated in the FPGA card.
  4. The CPM then signals the HPM root port, so the latter can (i) identify the newly added memory segment, and (ii) assign it a Host Physical Address (HPA) space.
  5. From this point on, the VMs running on the HPM can use the additional memory that was allocated.

HIGHER considers safeguards at both inter- and intra- host levels. Inter-host safeguards will ensure that each host can only access memory space ranges that were allocated to it. Intra-host safeguards will protect against processes / VMs running on the same host that try to access memory regions outside of their allocated space. Internally, the system first translates CXL packets into AXI4 packets. These packets will then be forwarded to the memory safeguard logic, which checks each request. Legitimate requests are forwarded to the memory controller, which gives them access to memory pool resources, while illegal requests are rejected.