Huge pages

The huge page feature in OpenStack provides important performance
improvements for applications that are highly memory IO-bound.

Pages, the TLB and huge pages

Pages

Physical memory is segmented into a series of contiguous regions
called pages. Each page contains a number of bytes, referred to as the
page size. The system retrieves memory by accessing entire pages, rather
than byte by byte.

Translation Lookaside Buffer (TLB)

A TLB is used to map the virtual addresses of pages to the physical
addresses in actual memory. The TLB is a cache and is not limitless,
storing only the most recent or frequently accessed pages. During normal
operation, processes will sometimes attempt to retrieve pages that are
not stored in the cache. This is known as a TLB miss and results in a
delay as the processor iterates through the pages themselves to find the
missing address mapping.

Huge Pages

The standard page size in x86 systems is 4 kB. This is optimal for
general purpose computing but larger page sizes – 2 MB and 1 GB – are
also available. These larger page sizes are known as huge pages. Huge
pages result in less efficient memory usage as a process will not
generally use all memory available in each page. However, use of huge
pages will result in fewer overall pages and a reduced risk of TLB
misses. For processes that have significant memory requirements or are
memory intensive, the benefits of huge pages frequently outweigh the
drawbacks.

Persistent Huge Pages

On Linux hosts, persistent huge pages are huge pages that are
reserved upfront. The HugeTLB provides for the mechanism for this
upfront configuration of huge pages. The HugeTLB allows for the
allocation of varying quantities of different huge page sizes.
Allocation can be made at boot time or run time. Refer to the Linux
hugetlbfs guide for more information.

Transparent Huge Pages (THP)

On Linux hosts, transparent huge pages are huge pages that are
automatically provisioned based on process requests. Transparent huge
pages are provisioned on a best effort basis, attempting to provision 2
MB huge pages if available but falling back to 4 kB small pages if not.
However, no upfront configuration is necessary. Refer to the Linux
THP guide for more information.

Enabling huge pages on the
host

Persistent huge pages are required owing to their guaranteed
availability. However, persistent huge pages are not enabled by default
in most environments. The steps for enabling huge pages differ from
platform to platform and only the steps for Linux hosts are described
here. On Linux hosts, the number of persistent huge pages on the host
can be queried by checking /proc/meminfo:

$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

In this instance, there are 0 persistent huge pages
(HugePages_Total) and 0 transparent huge pages
(AnonHugePages) allocated. Huge pages can be allocated at
boot time or run time. Huge pages require a contiguous area of memory –
memory that gets increasingly fragmented the long a host is running.
Identifying contiguous areas of memory is an issue for all huge page
sizes, but it is particularly problematic for larger huge page sizes
such as 1 GB huge pages. Allocating huge pages at boot time will ensure
the correct number of huge pages is always available, while allocating
them at run time can fail if memory has become too fragmented.

To allocate huge pages at run time, the kernel boot parameters must
be extended to include some huge page-specific parameters. This can be
achieved by modifying /etc/default/grub and appending the
hugepagesz, hugepages, and
transparent_hugepages=never arguments to
GRUB_CMDLINE_LINUX. To allocate, for example, 2048
persistent 2 MB huge pages at boot time, run:

# echo 'GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX hugepagesz=2M hugepages=2048 transparent_hugepage=never"' > /etc/default/grub
$ grep GRUB_CMDLINE_LINUX /etc/default/grub
GRUB_CMDLINE_LINUX="..."
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX hugepagesz=2M hugepages=2048 transparent_hugepage=never"

Reboot the host, then validate that huge pages are now available:

$ grep "Huge" /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:    2048
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

There are now 2048 2 MB huge pages totalling 4 GB of huge pages.
These huge pages must be mounted. On most platforms, this happens
automatically. To verify that the huge pages are mounted, run:

# mount | grep huge
hugetlbfs on /dev/hugepages type hugetlbfs (rw)

In this instance, the huge pages are mounted at
/dev/hugepages. This mount point varies from platform to
platform. If the above command did not return anything, the hugepages
must be mounted manually. To mount the huge pages at
/dev/hugepages, run:

# mkdir -p /dev/hugepages
# mount -t hugetlbfs hugetlbfs /dev/hugepages

There are many more ways to configure huge pages, including
allocating huge pages at run time, specifying varying allocations for
different huge page sizes, or allocating huge pages from memory
affinitized to different NUMA nodes. For more information on configuring
huge pages on Linux hosts, refer to the Linux
hugetlbfs guide.

Customizing
instance huge pages allocations

By default, an instance does not use huge pages for its underlying
memory. However, huge pages can bring important or required performance
improvements for some workloads. Huge pages must be requested explicitly
through the use of flavor extra specs or image metadata. To request an
instance use huge pages, you can use the hw:mem_page_size flavor extra spec:

$ openstack flavor set m1.large --property hw:mem_page_size=large

Different platforms offer different huge page sizes. For example:
x86-based platforms offer 2 MB and 1 GB huge page sizes. Specific huge
page sizes can be also be requested, with or without a unit suffix. The
unit suffix must be one of: Kb(it), Kib(it), Mb(it), Mib(it), Gb(it),
Gib(it), Tb(it), Tib(it), KB, KiB, MB, MiB, GB, GiB, TB, TiB. Where a
unit suffix is not provided, Kilobytes are assumed. To request an
instance to use 2 MB huge pages, run one of:

$ openstack flavor set m1.large --property hw:mem_page_size=2MB

$ openstack flavor set m1.large --property hw:mem_page_size=2048

Enabling huge pages for an instance can have negative consequences
for other instances by consuming limited huge pages resources. To
explicitly request an instance use small pages, run:

$ openstack flavor set m1.large --property hw:mem_page_size=small

Finally, to leave the decision of huge or small pages to the compute
driver, run:

$ openstack flavor set m1.large --property hw:mem_page_size=any

For more information about the syntax for
hw:mem_page_size, refer to the documentation <hw:mem_page_size>.

Applications are frequently packaged as images. For applications that
require the IO performance improvements that huge pages provides,
configure image metadata to ensure instances always request the specific
page size regardless of flavor. To configure an image to use 1 GB huge
pages, run:

$ openstack image set [IMAGE_ID]  --property hw_mem_page_size=1GB

If the flavor specifies a numerical page size or a page size of
“small” the image is not allowed to specify a page size and if it does
an exception will be raised. If the flavor specifies a page size of
any or large then any page size specified in
the image will be used. By setting a small page size in the
flavor, administrators can prevent users requesting huge pages in
flavors and impacting resource utilization. To configure this page size,
run:

$ openstack flavor set m1.large --property hw:mem_page_size=small

For more information about image metadata, refer to the Image
metadata guide.