For more information: VMware blog: SAP HANA with Intel Optane Persistent Memory on VMware vSphere
Pre-requisites and general SAP support limitations for Intel Optane PMem.
- Review information published in SAP Note 2913410 - SAP HANA on VMware vSphere with Persistent Memory.
- Only server platforms which are SAP HANA supported and certified and listed on VMware Compatibility Guide with “Persistent Memory” feature are supported.
- The CPU and memory prerequisites of SAP HANA on vSphere must be met as a prerequisite. For details refer to SAP Note 2393917 – SAP HANA on VMware vSphere 6.7 in production and SAP Note 2937606 – SAP HANA on VMware vSphere 7.0 in production.
- General questions and boundary conditions about SAP HANA and persistent memory are addressed in SAP Note 2700084, 2786237 and VMware KB 78094.
- SAP HANA non-prod support for Memory Mode (2954515- SAP HANA Persistent Memory - memory mode)
What is supported?
SAP has granted support for SAP HANA 2 SPS 4 (or later) on vSphere versions 6.7 (beginning with version 6.7 EP14) and vSphere 7.0 (beginning with version 7.0 P01) for 2 and 4 socket servers based on 2nd Gen Intel Xeon Scalable processors (formerly code named Cascade Lake). The maximum DRAM plus Optane PMem host memory configurations can be up to 15 TB (current memory limit when DRAM with DCPMM gets combined) and must follow the hardware vendors Optane PMem configuration guidelines.
The maximum VM size with vSphere 6.7 and vSphere 7.0 is limited to 256 vCPUs and 6 TB of memory. This results in SAP HANA VM sizes of 6 TB maximum for OLTP and 3 TB VM sizes for OLAP workloads (Class L). vSphere 7.0 supports OLAP workloads up to 6 TB (Class M).
Supported DRAM:PMem ratios are 2:1, 1:1, 1:2 and 1:4. Please refer to SAP note 2700084 for further details, use cases, and assistance in determining whether Optane PMem is applicable at all for your specific SAP HANA workload.
Supported Optane PMem Module sizes are 128, 256 and 512 GB. Supported maximal host memory DRAM-PMem configurations are listed below. Up to two SAP HANA VMs are supported per CPU socket, up to a 4-socket large server systems can get used:
Sizing of SAP HANA Optane PMem VMs:
The sizing of Optane PMem-enabled SAP HANA VMs is similar to bare metal SAP HANA systems with the limitation of a maximum size of 6 TB (mix of DRAM and Optane PMem) per VM. OLAP Class L workload sizigs are limited to 3 TB, Class M sizings support up to 6 TB total memory.
We recommend that an SAP HANA VM use the same DRAM:PMem ratio as the physical host/server DRAM:PMem ratio. However, if you have a growth plan, you might consider a larger physical memory configuration, and upgrade the VMs and SAP HANA over the lifetime.
For example, you have a 1:4 PMem ratio host, configured with 15 TB of total RAM (3 TB DRAM and 12 TB Optane PMem). An optimized resource scenario is to create four SAP HANA VMs on this server, each with 3.75 TB RAM (0.75 TB DRAM + 3 TB Optane PMem). If you are creating 6 TB VMs on this same 15 TB host, then only two SAP HANA VMs can be created, which is a non-optimized resource configuration as you can only leverage 12 of the installed 15 TB memory. In this case, a 1:1 DRAM:PMem configuration, with in total 12 TB (6 TB DRAM + 6 TB Optane PMem) represents a resource-optimized configuration. Future vSphere versions will support VM sizes >6 TB and will be able to leverage higher memory configurations!
See below examples for more details. The last example shows a configuration with 4 SAP HANA VMs with a 1:2 DRAM to PMem ratio.
4-Socket Host Configuration:
- Four 2nd Gen Intel Xeon Platinum processors, 24 x 128GB DRAM + 24 x 512GB Optane PMem = 15 TB total host memory with a 1:4 DRAM to PMem RATIO
- 2 x 6 TB SAP HANA VM with 1.5 TB DRAM and 4.5 TB Optane PMem RAM, with a 1:3 DRAM to PMem RATIO
- DRAM:PMem Ratio may not be suited for SAP HANA workload
- HW configuration does not fit and will lead to unusable PMem (RATIO mismatch)
4-Socket Host Configuration:
- Four 2nd Gen Intel Xeon Platinum processors, 24 x 256GB DRAM + 24 x 256GB Optane PMem = 12 TB total host memory with a 1:1 DRAM to PMem RATIO
- 2 x 6 TB SAP HANA VM with 3 TB DRAM and 3 TB Optane PMem RAM, with a 1:1 DRAM to PMem RATIO
- Higher memory costs due to DRAM module prices
4-Socket Host Configuration:
- Four 2nd Gen Intel Xeon Platinum processors, 24 x 128GB DRAM + 24 x 256GB Optane PMem = 9 TB total host memory
- 4 x VM with 0.75 TB DRAM and 1.5 TB Optane PMem RAM, total RAM per SAP HANA VM 2.25 TB with a 1:2 DRAM to PMem RATIO
- SAP HANA Sizing to verify if Optane PMem Ratio is applicable and if CPU resources are enough!
Note: WBS sizings are supported and allow OLAP workloads with Class M CPU requirements to leverage up to 6 TB of total memory (DRAM+PMem).
Which vSphere versions are required for SAP HANA with Optane PMEM?
vSphere Persistent MemoryTM requires vSphere Enterprise Plus. For details please review following document (page 6).
Which VMware vSphere 6.7 and 7.0 features are supported?
The following advanced vSphere features are supported with Optane PMem including VMware vMotion, VMware DRS.
Using SAP HANA on VMware vSphere allows HANA users to leverage the flexibility of vSphere capabilities such as vMotion, which allow workloads to be migrated between vSphere hosts on Intel Xeon platforms without first having to be shut down. In addition, VMware’s Dynamic Resource Scheduler (DRS) works with a cluster of ESXi hosts to provide resource management capabilities like load balancing and virtual machine (VM) placement to ensure a balanced environment for VM workloads.
VMware HA is yet not supported for SAP HANA VM Optane PMem use cases. A later vSphere 7.x version will support VMware HA for SAP HANA Optane PMem-enabled VM’s.
However, vSphere can be configured to restart SAP HANA Optane PMem VMs automatically on the same host in case of a VM failure. To configure this, – enable vSphere HA on the cluster and enable VM monitoring. Please note, if you enable vSphere HA on the cluster, it will not restart the SAP HANA VM with Optane PMem on a different host, if local host fails. For SAP HANA DRAM only VMs, vSphere HA can get used without restrictions or limitations.
A word on VMware HA, VMware vSphere HA allows automated restarts of VMs, when there is a server outage, or an operating system failure detected in the vSphere environment. VMware HA requires a file system accessible from all hosts on the vSphere cluster. Optane PMem in App Direct mode is not a shared resource, therefore, VM failover to another host in the vSphere cluster is not possible and is, as of today, not supported with Optane PMem-enabled VMs.
SAP HANA System Replication with Intel Optane PMem enabled SAP HANA VMs:
Depending on your SAP HANA SLAs, customers can define a strategy to achieve them. All SAP HANA build-in capabilities, like SAP HANA System Replication, are supported with Intel Optane PMem and can be leveraged instead of VMWare HA to be used to help recover from catastrophic host failures for HANA VMs.
Following table shows a summary of the supported features and what needs to get considered when using these features:
SAP HANA System Replication with Optane PMem enabled SAP HANA VMs:
SAP HANA System replication can get leveraged to protect, just like with physical deployed SAP HANA systems, an SAP HANA DRAM only or Optane PMem enabled VM.
Following example highlights how persistent memory can help to lowering the cost by leveraging a Optane PMem host as the replication target for two SAP HANA DRAM VMs. The failover of the DNS name and IP address gets handled by a cluster solution like Linux Pacemaker. The SAP HANA data gets replicated via HANA System Replication.
After a failover of a production SAP HANA VM it may be required to free up CPU resources on the failover host. This can get done by either shutdown the other disaster recovery (DR) VM or by moving it via vMotion to another DR host. Once the defective host got repaired and is back online, the SAP HANA production instance should get moved back and SAP HANA System Replication should resume its work.
This is just an example how SAP HSR and Optane PMem can get used in combination to lower the TCO of an DR environment. This example shows a host with a DRAM to Optane PMem ratio of 1:1, which is in this case the best choice, since it will allow the operation of the example DRAM-only SAP HANA VMs without further sizing.
SAP HANA VM Configuration details:
To configure an SAP HANA Optane PMem-enabled VM for optimal performance it is necessary to align the VM configuration to the underlying HW and especially NUMA configuration.
VMware KB 78094 provides information on how to configure the NVDIMMs (VMware’s representation of Optane PMem) correctly and align the NVDIMMs to the physical NUMA architecture of the physical server.
By default, Optane PMem allocation in vmkernel for VM NVDIMMs does not consider NUMA. This can result in the VM running on a certain NUMA node and Optane PMem allocated from a different NUMA node. This will cause NVDIMMs access in the VM to be remote, resulting in poor performance. To solve this, you must add the following settings to a VM configuration using vCenter:
Example for a 4-socket wide VM:
Parameter sched.pmem.prealloc=TRUE is an optional parameter equivalent to eager zero thick provisioning of VMDKs and improves initial writes to Optane PMem.
Beside these parameters, you may also configure the CPU NUMA node affinity or CPU affinities (pinning), as described in the SAP HANA best practices parameter guidelines listed here. For details on CPU affinities please review the 2nd page of this list.
Can I use vSAN / SAP HANA HCI with PMEM?
VMware vSAN does not have support for “App-Direct” mode (source) as cache or as a capacity tier device of vSAN. However, vSAN will work with vSphere hosts equipped with Intel Optane PMem in App-Direct mode and SAP HANA VM’s can leverage PMEN according to SAP Note 2913410. Please note that especially the VMware HA restriction (as described in note 2913410) applies and needs to get considered.
Also, vSAN will work with Intel Optane PMem 100 series in Memory Mode as long as such deployments are pre-approved by VMware as mentioned above.
Can I use VMware vSphere virtualised SAP HANA VMs with Memory Mode?
Yes, SAP has extended non-prod support for PMem Memory Mode and Mixed Mode. VMware provides no Mixed Mode PMem support, but provides full support for PMem Memory Mode, as outlined in VMware KB 67645 and SAP support note 2954515.
Please follow following configuration guidelnes and limitations:
vSphere is supporting the following maximum host capacity of Intel® Optane™ PMem 100 series in memory mode (MM) with SAP HANA starting with vSphere 6.7 U3 EP14 and 7.0 release as follows:
Up to 6TB for 2-socket and 12TB for 4-socket of Intel® Optane™ PMem 100 series in memory mode. For specific limits for your server platform, please consult VMware Compatibility Guide (VCG).
Supported vSphere Versions:
- 6.7 U3 EP14 or later
- vSphere 7.0 or later
Maximum VM capacity of Intel® Optane™ PMem 100 series enabled in vSphere:
- Up to 6TB of virtual RAM backed-up by Intel® Optane™ PMem 100 series in memory mode, see notes 2937606, 2913410.
VMware will support customers’ SAP HANA 2.0 non-prod deployments of Intel Optane PMem 100 series in Memory Mode when following conditions are met.
Customers are running the server platform with Balanced Profile BIOS setting from server OEMs. For more information, please contact your server OEM vendor.
Customers use one of the following 2-socket* server platform memory configurations:
- 1.5TB Intel Optane PMem + 384GB DRAM (12x128GB PMem + 12x32GB DRAM)
- 3TB Intel Optane PMem + 768GB DRAM (12x256GB PMem +12x64GB DRAM)
- 6TB Intel Optane PMem + 1.5TB DRAM (12x512GB PMem + 12x128GB DRAM)
OR customers use one of the following 4-socket* server platform memory configurations:
- 3TB Intel Optane PMem + 768GB DRAM (24x128GB PMem + 24x32GB DRAM) – Large
- 6TB Intel Optane PMem + 1.5TB DRAM (24x256GB PMem +24x64GB DRAM) – Very Large
- 12TB Intel Optane PMem + 3TB DRAM (24x512GB PMem + 24x128GB DRAM) – Very Very Large
Note: Mixed mode is not supported and if any of the above-stated conditions are not met, contact your VMware representative for approval to get support for your deployment. For all platform configurations, VMware highly recommends customers ensure the Active Memory as described in Understanding the Memory Active and Memory Usage indicators on the Performance tab (1002604) of the host steady-state at no time exceeds 50% of the available DRAM in the host, for their specific workload use-cases.
- VMware blog: SAP HANA with Intel Optane Persistent Memory on VMware vSphere
- VMware blog: SAP NetWeaver and SAP HANA with VMware vSphere 7.0
- ESXi build numbers: https://kb.vmware.com/s/article/2143832
- VMware “Persistent Memory” supported servers: VMware Compatibility Guide
- VMware KB 78094, NVDIMM configuration parameters
- vSphere Support for Intel's Optane Persistent Memory (PMEM) (KB 67645)
- SAP HANA best practices parameter list
- VMware KB 2145179, using CPU affinities
- VMware SAP HANA with Optane PMem Best Practices Document (document to be published at a later time)
- SAP on VMware – central webpage
- VMware vSphere 6.7 PMem documentation
- VMware vSphere 7.0 PMem documentation
- Configuration Guide: Intel® OptaneTM Persistent Memory and SAP HANA® Platform Configuration
- SAP Note 2700084 – FAQ: SAP HANA Persistent Memory
- SAP Note 2393917 – SAP HANA on VMware vSphere 6.7 in production.
- SAP Note 2937606 – SAP HANA on VMware vSphere 7.0 in production.
- SAP Note 2913410 – vSphere 6.7 and 7.0 SAP HANA PMem Support
- SAP support note 2954515 - SAP HANA Persistent Memory - memory mode, mixed mode
- SAP Note 2786237 – Sizing SAP HANA with Persistent Memory
- SAP Note 2813454– Recommendations for Persistent Memory Configuration with BW/4HANA
- SAP HANA is The First Major DBMS Optimized for Intel Optane Persistent Memory, link
- Innovation Squared with Intel and SAP: Persistent Memory Technology and SAP HANA
- SAP HANA Administration Guide for SAP HANA Platform – Optane PMem
- IDC White Paper – SAP HANA and Optane PMem
- Fujitsu white paper – Performance Test Report of Intel® Optane™ Persistent Memory on PRIMEFLEX for SAP HANA