Michio Honda, University of Edinburgh, United Kingdom – October 2022

Datacenter transport widely used today is in-kernel TLS (kTLS) over Data Center TCP (DCTCP), because of the accessibility to many applications and ease of adoption for operators. kTLS was initially introduced for storage servers that issue sendfile() syscalls, but end-to-end encryption has been increasingly required as the default even in datacenters; for example, Facebook announced the use of kTLS over DCTCP in 2016. Although clean-slate transport protocols that achieve lower latency and higher throughput, including pHost, NDP and Homa, have been proposed, they are not used in production so far, perhaps because they are implemented using kernel-bypass networking such as Intel DPDK, which requires additional NIC(s) and largely disrupts existing operation based on familiar Linux tools (e.g., iproute2 and ss) and orchestration tools (e.g., Ansible, Chef and various eBPF-based management tools like Cilium). However, Homa, which is one of the state-of-the-art receiver-driven transport protocols, was implemented as a Linux kernel module in 2021, which runs alongside other transport protocols like TCP and SCTP without requiring kernel-bypass networking [1]. Homa offers message-based APIs based on the familiar iovec structure, and provides a one-to-many-style socket like SCTP so that the application does not have to monitor or scan as many file descriptors as the remote endpoints. Homa achieves considerably lower latency than TCP; it achieves 20us of 1.4kB message RTT in the testbed where TCP exhibits 30us [2]. Although Homa has a significant potential to innovate datacenter transport, encryption is still missing. The lack of transport-level encryption has two problems. First, the application needs to encrypt its messages by itself. Second, more importantly, application-level encryption hinders hardware inline TLS offloading, which is available in some NICs (e.g., Mellanox ConnectX-6 DX and Chelsio T6). This is because a Homa packet, which can contain multiple application messages, resembles a TCP packet to utilize TCP segmentation offload (TSO), and TLS offloading happens together with TSO, meaning that the kernel needs to create unencrypted packets. This problem can be avoided if the NIC understands the Homa packet format explicitly rather than treating it as a TCP segment, but we do not believe this could happen. To address the problems with using TLS over Homa, we propose HomaLS (Homa Layer Security), which integrates kTLS-based encryption with Homa so that the datacenter applications can utilize Homa and TLS hardware offloading opportunistically and transparently, in addition to obviating cryptographic operations in the applications. Our early-stage HomaLS prototype (Hardware TLS offload is not supported yet. So far we do not need to modify the kernel core) indicates the viability of our approach - it achieves 26–30% shorter RTT than kTLS over TCP for 16-1380 byte messages [2]. However, we still have to address a number of research challenges. First, we must efficiently maintain the mapping between the TLS-level sequence number and “TCP”-level one for each of 5 tuple, because hardware TLS offloading is stateful, unlike TSO. Second, we must thoroughly evaluate the protocol design based on the TLS offloading behavior in the NIC, because the TLS offload engine might interfere with the “TCP options” field where some Homa headers are embedded, like middleboxes in the Internet do. In case of that, we need to redesign the Homa packet format. Finally, we must examine the applicability of HomaLS to existing and new applications. In addition to existing applications such as gRPC, we plan to examine user-space (e.g., object storage) and the in-kernel storage applications (e.g., NFS), because those applications must handle both small and large messages and latency is of importance. We believe HomaLS improves throughput and latency of those storage applications that currently use TLS over DCTCP