For more information: VMware blog: SAP HANA with Intel Optane Persistent Memory on VMware vSphere and VMware vSphere 7.0 U2 and vSphere HA for SAP HANA with DRAM and Intel® Optane™ PMem in App-Direct Mode
Pre-requisites and general SAP support limitations for Intel Optane PMem.
- Review information published in SAP Note 2913410 - SAP HANA on VMware vSphere with Persistent Memory.
- Only server platforms which are SAP HANA supported and certified and listed on VMware Compatibility Guide with “Persistent Memory” feature are supported.
- The CPU and memory prerequisites of SAP HANA on vSphere must be met as a prerequisite. For details refer to SAP Note 2393917 – SAP HANA on VMware vSphere 6.7 in production and SAP Note 2937606 – SAP HANA on VMware vSphere 7.0 in production.
- General questions and boundary conditions about SAP HANA and persistent memory are addressed in SAP Note 2700084, 2786237 and VMware KB 78094.
- SAP HANA non-prod support for Memory Mode (2954515- SAP HANA Persistent Memory - memory mode)
What is supported?
SAP has granted support for SAP HANA 2 SPS 4 (or later) on vSphere versions 6.7 (beginning with version 6.7 EP14) and vSphere 7.0 (beginning with version 7.0 P01) for 2 and 4 socket servers based on 2nd Gen Intel Xeon Scalable processors (formerly code named Cascade Lake). The maximum DRAM plus Optane PMem host memory configurations can be up to 15 TB (current memory limit when DRAM with DCPMM gets combined) and must follow the hardware vendors Optane PMem configuration guidelines.
The maximum VM size with vSphere 6.7 and vSphere 7.0 is limited to 256 vCPUs and 6 TB of memory. This results in SAP HANA VM sizes of 6 TB maximum for OLTP and 3 TB VM sizes for OLAP workloads (Class L). vSphere 7.0 supports OLAP workloads up to 6 TB (Class M).
Supported DRAM:PMem ratios are 2:1, 1:1, 1:2 and 1:4. Please refer to SAP note 2700084 for further details, use cases, and assistance in determining whether Optane PMem is applicable at all for your specific SAP HANA workload.
Supported Optane PMem Module sizes are 128, 256 and 512 GB. Supported maximal host memory DRAM-PMem configurations are listed below. Up to two SAP HANA VMs are supported per CPU socket, up to a 4-socket large server systems can get used:
Sizing of SAP HANA Optane PMem VMs:
The sizing of Optane PMem-enabled SAP HANA VMs is similar to bare metal SAP HANA systems with the limitation of a maximum size of 6 TB (mix of DRAM and Optane PMem) per VM. OLAP Class L workload sizigs are limited to 3 TB, Class M sizings support up to 6 TB total memory.
We recommend that an SAP HANA VM use the same DRAM:PMem ratio as the physical host/server DRAM:PMem ratio. However, if you have a growth plan, you might consider a larger physical memory configuration, and upgrade the VMs and SAP HANA over the lifetime.
For example, you have a 1:4 PMem ratio host, configured with 15 TB of total RAM (3 TB DRAM and 12 TB Optane PMem). An optimized resource scenario is to create four SAP HANA VMs on this server, each with 3.75 TB RAM (0.75 TB DRAM + 3 TB Optane PMem). If you are creating 6 TB VMs on this same 15 TB host, then only two SAP HANA VMs can be created, which is a non-optimized resource configuration as you can only leverage 12 of the installed 15 TB memory. In this case, a 1:1 DRAM:PMem configuration, with in total 12 TB (6 TB DRAM + 6 TB Optane PMem) represents a resource-optimized configuration. Future vSphere versions will support VM sizes >6 TB and will be able to leverage higher memory configurations!
See below examples for more details. The last example shows a configuration with 4 SAP HANA VMs with a 1:2 DRAM to PMem ratio.
4-Socket Host Configuration:
- Four 2nd Gen Intel Xeon Platinum processors, 24 x 128GB DRAM + 24 x 512GB Optane PMem = 15 TB total host memory with a 1:4 DRAM to PMem RATIO
- 2 x 6 TB SAP HANA VM with 1.5 TB DRAM and 4.5 TB Optane PMem RAM, with a 1:3 DRAM to PMem RATIO
- DRAM:PMem Ratio may not be suited for SAP HANA workload
- HW configuration does not fit and will lead to unusable PMem (RATIO mismatch)
4-Socket Host Configuration:
- Four 2nd Gen Intel Xeon Platinum processors, 24 x 256GB DRAM + 24 x 256GB Optane PMem = 12 TB total host memory with a 1:1 DRAM to PMem RATIO
- 2 x 6 TB SAP HANA VM with 3 TB DRAM and 3 TB Optane PMem RAM, with a 1:1 DRAM to PMem RATIO
- Higher memory costs due to DRAM module prices
4-Socket Host Configuration:
- Four 2nd Gen Intel Xeon Platinum processors, 24 x 128GB DRAM + 24 x 256GB Optane PMem = 9 TB total host memory
- 4 x VM with 0.75 TB DRAM and 1.5 TB Optane PMem RAM, total RAM per SAP HANA VM 2.25 TB with a 1:2 DRAM to PMem RATIO
- SAP HANA Sizing to verify if Optane PMem Ratio is applicable and if CPU resources are enough!
Note: WBS sizings are supported and allow OLAP workloads with Class M CPU requirements to leverage up to 6 TB of total memory (DRAM+PMem).
Which vSphere versions are required for SAP HANA with Optane PMEM?
vSphere Persistent MemoryTM requires vSphere Enterprise Plus. For details please review following document (page 6).
Which VMware vSphere 6.7 and 7.0 features are supported?
The following advanced vSphere features are supported with Optane PMem including VMware HA, vMotion, VMware DRS.
Using SAP HANA on VMware vSphere allows HANA users to leverage the flexibility of vSphere capabilities such as vMotion, which allow workloads to be migrated between vSphere hosts on Intel Xeon platforms without first having to be shut down. In addition, VMware’s Dynamic Resource Scheduler (DRS) works with a cluster of ESXi hosts to provide resource management capabilities like load balancing and virtual machine (VM) placement to ensure a balanced environment for VM workloads.
VMware HA is by now supported for SAP HANA VM Optane PMem use cases. Check out following VMware blog for details: VMware vSphere 7.0 U2 and vSphere HA for SAP HANA with DRAM and Intel® Optane™ PMem in App-Direct Mode
Following table shows a summary of the supported features and what needs to get considered when using these features:
vSphere HA support for SAP HANA PMem enabled VMs:
VMware HA was initially not supported for PMem enabled VMs before the vSphere 7.0 U2 release. Now VMware HA can support the failover and restart of PMem enabled VMs. Requirement is that the applications using PMem maintain data persistence not only on PMem, but also on shared disks.
SAP HANA is one of the applications that provide data persistence on disk. Because of this, VMware HA can use these data on the shared disks to initiate a failover of SAP HANA PMem enabled VMs to another PMem host. VMware HA will automatically re-create the VMs NVDIMM configuration but is not in control over post VM failover OS / application specific configuration steps, like the required re-creation of the SAP HANA DAX device configuration. This must get done manually or via a script, which is not provided by VMware nor SAP. Review Configuration Guide: Intel® OptaneTM DC Persistent Memory and SAP HANA® Platform Configuration for details on how to configure PMem for SAP HANA.
Below picture illustrates the failover of a PMem enabled SAP HANA VM via vSphere 7.0 U2 VMware HA and highlights that the PMem NVDIMM configuration gets automatically re-created as part of the VM failover process. Once the DAX device got configured inside the OS, SAP HANA can be started and will automatically load the data from disk to the new PMem regions assigned to this VM.
After a successful failover of this PMem enabled VM, a garbage collector process will identify failed over VMs and will free up the PMem resources previously used by this VM on the initial host. On the host this VM now runs, the PMem will be blocked and reserved for the live time of this VM (as long it gets not migrated or deleted from the host).
Below is a link to an example script for the automatic re-creation of the DAX device configuration on the OS level. The script has been developed by the Intel SAP Solution Engineering team and the Intel VMware Center of Excellence.
This script must be executed after the failover and re-start of the VM, prior to the restart of the SAP HANA Database. It is advised to automatically run this script as part of the OS start procedure, like as a custom service.
The script can be used as a template to create your own script that fits your unique environment.
Note: This script is not maintained nor supported by VMware, SAP or Intel, any usage of this script is upon your own responsibility.
More details and how to configure vSphere HA for PMem enabled SAP HANA VMs are published in this blog.
SAP HANA System Replication with Optane PMem enabled SAP HANA VMs:
SAP HANA System replication can get leveraged to protect, just like with physical deployed SAP HANA systems, an SAP HANA DRAM only or Optane PMem enabled VM.
Following example highlights how persistent memory can help to lowering the cost by leveraging a Optane PMem host as the replication target for two SAP HANA DRAM VMs. The failover of the DNS name and IP address gets handled by a cluster solution like Linux Pacemaker. The SAP HANA data gets replicated via HANA System Replication.
After a failover of a production SAP HANA VM it may be required to free up CPU resources on the failover host. This can get done by either shutdown the other disaster recovery (DR) VM or by moving it via vMotion to another DR host. Once the defective host got repaired and is back online, the SAP HANA production instance should get moved back and SAP HANA System Replication should resume its work.
This is just an example how SAP HSR and Optane PMem can get used in combination to lower the TCO of an DR environment. This example shows a host with a DRAM to Optane PMem ratio of 1:1, which is in this case the best choice, since it will allow the operation of the example DRAM-only SAP HANA VMs without further sizing.
SAP HANA VM Configuration details:
To configure an SAP HANA Optane PMem-enabled VM for optimal performance it is necessary to align the VM configuration to the underlying HW and especially NUMA configuration.
VMware KB 78094 provides information on how to configure the NVDIMMs (VMware’s representation of Optane PMem) correctly and align the NVDIMMs to the physical NUMA architecture of the physical server.
By default, Optane PMem allocation in vmkernel for VM NVDIMMs does not consider NUMA. This can result in the VM running on a certain NUMA node and Optane PMem allocated from a different NUMA node. This will cause NVDIMMs access in the VM to be remote, resulting in poor performance. To solve this, you must add the following settings to a VM configuration using vCenter:
Example for a 4-socket wide VM:
Parameter sched.pmem.prealloc=TRUE is an optional parameter equivalent to eager zero thick provisioning of VMDKs and improves initial writes to Optane PMem.
Beside these parameters, you may also configure the CPU NUMA node affinity or CPU affinities (pinning), as described in the SAP HANA best practices parameter guidelines listed here. For details on CPU affinities please review the 2nd page of this list.
Can I use vSAN / SAP HANA HCI with PMEM?
VMware vSAN does not have support for “App-Direct” mode (source) as cache or as a capacity tier device of vSAN. However, vSAN will work with vSphere hosts equipped with Intel Optane PMem in App-Direct mode and SAP HANA VM’s can leverage PMEN according to SAP Note 2913410. Please note that especially the VMware HA restriction (as described in note 2913410) applies and needs to get considered.
Also, vSAN will work with Intel Optane PMem 100 series in Memory Mode as long as such deployments are pre-approved by VMware as mentioned above.
Can I use VMware vSphere virtualised SAP HANA VMs with Memory Mode?
Yes, SAP has extended non-prod support for PMem Memory Mode and Mixed Mode. VMware provides no Mixed Mode PMem support, but provides full support for PMem Memory Mode, as outlined in VMware KB 67645 and SAP support note 2954515.
Please follow following configuration guidelnes and limitations:
vSphere is supporting the following maximum host capacity of Intel® Optane™ PMem 100 series in memory mode (MM) with SAP HANA starting with vSphere 6.7 U3 EP14 and 7.0 release as follows:
Up to 6TB for 2-socket and 12TB for 4-socket of Intel® Optane™ PMem 100 series in memory mode. For specific limits for your server platform, please consult VMware Compatibility Guide (VCG).
Supported vSphere Versions:
- 6.7 U3 EP14 or later
- vSphere 7.0 or later
Maximum VM capacity of Intel® Optane™ PMem 100 series enabled in vSphere:
- Up to 6TB of virtual RAM backed-up by Intel® Optane™ PMem 100 series in memory mode, see notes 2937606, 2913410.
VMware will support customers’ SAP HANA 2.0 non-prod deployments of Intel Optane PMem 100 series in Memory Mode when following conditions are met.
Customers are running the server platform with Balanced Profile BIOS setting from server OEMs. For more information, please contact your server OEM vendor.
Customers use one of the following 2-socket* server platform memory configurations:
- 1.5TB Intel Optane PMem + 384GB DRAM (12x128GB PMem + 12x32GB DRAM)
- 3TB Intel Optane PMem + 768GB DRAM (12x256GB PMem +12x64GB DRAM)
- 6TB Intel Optane PMem + 1.5TB DRAM (12x512GB PMem + 12x128GB DRAM)
OR customers use one of the following 4-socket* server platform memory configurations:
- 3TB Intel Optane PMem + 768GB DRAM (24x128GB PMem + 24x32GB DRAM) – Large
- 6TB Intel Optane PMem + 1.5TB DRAM (24x256GB PMem +24x64GB DRAM) – Very Large
- 12TB Intel Optane PMem + 3TB DRAM (24x512GB PMem + 24x128GB DRAM) – Very Very Large
Note: Mixed mode is not supported and if any of the above-stated conditions are not met, contact your VMware representative for approval to get support for your deployment. For all platform configurations, VMware highly recommends customers ensure the Active Memory as described in Understanding the Memory Active and Memory Usage indicators on the Performance tab (1002604) of the host steady-state at no time exceeds 50% of the available DRAM in the host, for their specific workload use-cases.
- VMware blog: SAP HANA with Intel Optane Persistent Memory on VMware vSphere
- VMware blog: SAP NetWeaver and SAP HANA with VMware vSphere 7.0
- VMware blog: VMware vSphere 7.0 U2 and vSphere HA for SAP HANA with DRAM and Intel® Optane™ PMem in App-Direct Mode
- ESXi build numbers: https://kb.vmware.com/s/article/2143832
- VMware “Persistent Memory” supported servers: VMware Compatibility Guide
- VMware KB 78094, NVDIMM configuration parameters
- vSphere Support for Intel's Optane Persistent Memory (PMEM) (KB 67645)
- SAP HANA best practices parameter list
- VMware KB 2145179, using CPU affinities
- VMware SAP HANA with Optane PMem Best Practices Document (document to be published at a later time)
- SAP on VMware – central webpage
- VMware vSphere 6.7 PMem documentation
- VMware vSphere 7.0 PMem documentation
- Configuration Guide: Intel® OptaneTM Persistent Memory and SAP HANA® Platform Configuration
- SAP Note 2700084 – FAQ: SAP HANA Persistent Memory
- SAP Note 2393917 – SAP HANA on VMware vSphere 6.7 in production.
- SAP Note 2937606 – SAP HANA on VMware vSphere 7.0 in production.
- SAP Note 2913410 – vSphere 6.7 and 7.0 SAP HANA PMem Support
- SAP support note 2954515 - SAP HANA Persistent Memory - memory mode, mixed mode
- SAP Note 2786237 – Sizing SAP HANA with Persistent Memory
- SAP Note 2813454– Recommendations for Persistent Memory Configuration with BW/4HANA
- SAP HANA is The First Major DBMS Optimized for Intel Optane Persistent Memory, link
- Innovation Squared with Intel and SAP: Persistent Memory Technology and SAP HANA
- SAP HANA Administration Guide for SAP HANA Platform – Optane PMem
- IDC White Paper – SAP HANA and Optane PMem
- Fujitsu white paper – Performance Test Report of Intel® Optane™ Persistent Memory on PRIMEFLEX for SAP HANA