This article provides an explanation on Transparent Huge Pages (THP) and why enabling them could be detrimental to system performance of contrail-config and contrail-analytics nodes.

The standard virtual memory page size on most of the common Linux distributions is set to 4KB by default. The HugePages feature enables the Linux kernel to manage large pages of memory in addition to the standard 4KB. A typical HugePage size on x86_64 based system is 2MB and can be configured to go up to 1GB.

HugePages are known to enhance the system performance by significantly reducing the number of Page table entries in the memory Page Table. The details on how HugePages boost the system performance is beyond the scope of this article. Refer to the following web page for more information on HugePages: https://kerneltalks.com/services/what-is-huge-pages-in-linux/

For an application to make use of HugePages, it has to contain an explicit instruction to do so. It’s not always practical to change applications in this way and may require significant changes at system administration level. Hence, the Transparent HugePages (THP) were introduced. If enabled, it can potentially allocate HugePages for applications without them actually “knowing” it; hence the transparency. The expectation is that this will improve application performance.

The allocation of Tranparent HugePages (THPs) is a somewhat complicated process. While traditional HugePages are reserved in virtual memory, THPs are not. Many applications assume the page size on Linux is 4096 and request memory allocation based on that assumption. However, if THPs are enabled at the OS level, the kernel will try to make space in the memory for a 2MB HugePage even if the request is for a standard 4KB page. The allocation process can potentially involve a number of kernel processes which may include kswapd, defrag, kcompactd and khugepaged.

All these tasks carry out certain actions to make enough room in the virtual memory for a THP allocation and can have a detrimental effect on the performance of CPU to the extent that all the other CPU intensive applications have to wait significant amount of time before all the precious CPU resources are released by the THP allocations tasks.

It has been observed that contrail cassandra process running inside contrail-config and contrail-analytics-db nodes, as well as contrail Kafka process running inside contrail-analytics nodes can trigger frequent page requests to kernel; and if THP is enabled at the OS level this can result in THP associated tasks to hog the CPU resulting in software-lockups for other processes running on the system.

Juniper does not recommend using THP on contrail-config and contrail-analytics nodes for the very same reason.
 

To disable THP at the OS level on debian based systems:

• $echo never > /sys/kernel/mm/transparent_hugepage/enabled

• THP can also be enabled/disabled at the system startup by adding the following parameter to system grub configuration file "/etc/default/grub" , but requires a system restart.

  • GRUB_CMDLINE_LINUX_DEFAULT="transparent_hugepage=never"​

To validate if THPs are being used in the system:

• $ grep Anon /proc/meminfo
AnonPages:           30121124 kB
AnonHugePages:  17479680 kB

To find out which processes are using what amount of THPs, use the following bash script:

• grep -e AnonHugePages /proc/*/smaps | awk '{ if($2>4) print $0} ' | awk -F "/" '{print $0; system("ps -fp" $3)}'
  /proc/13025/smaps:AnonHugePages:      4096 kB
  UID        PID    PPID  C STIME TTY          TIME                          CMD
  contrail 13025  1502   0 Sep19 ?              00:12:49 /usr/bin/python /usr/bin/contrail-device-manager --conf_file
   /etc/contrail/contrail-device-manager.conf --conf_file /etc/contrail/contrail-keystone-auth.conf --conf_file /etc/contrail
  /contrail-database.conf
   
  /proc/13025/smaps:AnonHugePages:      2048 kB
  UID        PID     PPID  C STIME TTY          TIME                          CMD
  contrail 13025  1502   0   Sep19 ?              00:12:49 /usr/bin/python /usr/bin/contrail-device-manager --conf_file
   /etc/contrail/contrail-device-manager.conf --conf_file /etc/contrail/contrail-keystone-auth.conf --conf_file /etc/contrail
  /contrail-database.conf