Transparent Huge Pages (THP) in Linux

Introduction

Transparent Huge Pages (THP) is a Linux kernel feature that optimizes memory management by using large pages (typically 2MB instead of the standard 4KB). In Linux, memory is managed in fixed-size blocks called pages, usually 4KB in size. The operating system uses paging to map virtual addresses to physical memory efficiently. By using larger pages, THP reduces the overhead of managing many small pages, improving performance for memory-intensive applications. THP operates transparently, meaning it works automatically without requiring explicit configuration from the user or application.

What is THP?

Transparent Huge Pages (THP) is a memory management feature that allows the Linux kernel to use large pages for memory allocations. A large page (commonly 2MB) aggregates many small pages (typically 4KB) into a single large page. This feature is designed to work in conjunction with the Translation Lookaside Buffer (TLB), a critical component of the CPU’s memory management unit.

The TLB is a hardware cache that stores recent translations of virtual memory addresses to physical memory addresses. It acts as a quick lookup table, significantly speeding up memory access times. However, the TLB can only hold a limited number of entries.

By reducing the number of pages the kernel has to manage, THP decreases the frequency of TLB misses. A TLB miss occurs when the CPU needs to access memory, but the translation isn’t in the TLB, requiring a time-consuming page table walk. With larger pages, more memory can be covered by fewer TLB entries, leading to fewer TLB misses. This, in turn, minimizes page table overhead and improves overall memory access efficiency, particularly for memory-intensive applications.

How THP Works

Page Allocation

• Normally, memory is allocated in 4KB chunks (pages). However, for certain workloads, such as databases or scientific applications, using huge pages (2MB or more) can improve performance by reducing the number of pages the system has to manage.

Automatic Management

• With THP, the kernel transparently handles the promotion of regular 4KB pages to huge pages and demotes them back when necessary.
• Applications do not need to be modified to take advantage of THP, as the kernel automatically decides when to use huge pages.

TLB Optimization

• The TLB is a cache used by the CPU to speed up the translation of virtual addresses to physical memory addresses. Using huge pages reduces the number of TLB entries required because each huge page covers a larger portion of memory.
• This reduction in TLB misses leads to better CPU performance, especially for memory-intensive applications.

Modes of THP

You can manage these modes via /sys/kernel/mm/transparent_hugepage/. We talk about this in the configuration section.

Performance Benefits of THP

1. Improved Memory Access: With fewer pages to manage, there is less overhead associated with TLB misses, which speeds up memory access for large memory allocations.
2. Reduced Page Table Overhead: Fewer pages mean less memory used for storing page tables, improving efficiency in memory management.
3. Reduced Fragmentation: THP helps reduce internal fragmentation by grouping multiple small pages into larger, contiguous blocks.

Challenges of THP

1. Defragmentation Overhead: For large contiguous memory blocks to be allocated as huge pages, the kernel must ensure that enough contiguous free memory is available. This may require memory defragmentation, which can introduce overhead and affect performance.
2. Latency: When the kernel promotes or demotes pages, there might be some latency involved, particularly in high-performance, low-latency applications.
3. Application Compatibility: Some applications, especially those that rely on very fine-grained control over memory, might not benefit from THP and could experience performance degradation.

THP and Splunk

While THP improves performance for many applications, it can negatively impact applications like Splunk, which rely on frequent short-lived processes.(Splunk’s official documentation).

Performance Issues

1. Short-Lived Process Overhead: THP aggressively merges memory pages, which is inefficient for Splunk searches that are typically short-lived.
2. Memory Allocation Conflicts: THP interferes with jemalloc, preventing efficient memory release back to the OS, leading to increased memory usage.
3. I/O Performance Regression: Swapping huge pages introduces significant I/O overhead, reducing Splunk’s responsiveness. (Swapping is the process of moving data between RAM and disk when memory is full.)

Impact on Splunk Performance

• 30% decrease in indexing and search performance when THP is enabled.
• 30% increase in query latency, slowing down searches.
• Both indexing and search tiers are affected, though Forwarders experience less impact.

Splunk’s Recommendation

Splunk recommends disabling THP on all Linux systems running Splunk Enterprise.

Configuring THP

Note: Please make sure you validate the confirguration in a test environment before applying it to production systems.

1. Check THP Status:

   • To check the current THP status, run:

     cat /sys/kernel/mm/transparent_hugepage/enabled
     

   • The output will show one of the following:
     • [always] madvise never → THP is always enabled.
     • always [madvise] never → THP is enabled only for applications that request it.
     • always madvise [never] → THP is disabled.

2. Enable or Disable THP:

   • Enable THP:

     echo always > /sys/kernel/mm/transparent_hugepage/enabled
     

   • Disable THP (recommended for Splunk):

     echo never > /sys/kernel/mm/transparent_hugepage/enabled
     

3. Make THP Configuration Persistent:

   • Add the following to /etc/rc.local (or a systemd service for modern distributions):

     echo never > /sys/kernel/mm/transparent_hugepage/enabled
     echo never > /sys/kernel/mm/transparent_hugepage/defrag
     

   • On RHEL/CentOS: Create a systemd service in /etc/systemd/system/disable-thp.service:

     [Unit]
     Description=Disable Transparent Huge Pages
     After=sysinit.target local-fs.target
     
     [Service]
     Type=oneshot
     ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'
     ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/defrag'
     RemainAfterExit=true
     
     [Install]
     WantedBy=multi-user.target
     

     Then enable it:

     systemctl daemon-reload
     systemctl enable disable-thp
     systemctl start disable-thp
     

This ensures THP remains disabled even after a reboot.

Conclusion

Transparent Huge Pages (THP) can boost performance in memory-intensive applications by using larger pages to reduce TLB misses. However, it also comes with challenges, such as defragmentation overhead and potential compatibility issues. For applications like Splunk, disabling THP is recommended to avoid performance problems and ensure optimal performance.