NVMe Non Volatile Memory Express AIX IBM Power System

NVMe Non Volatile Memory Express

NVMe is the latest growing technology based upon the Flash Storage. Flash Storage as you can see day to day examples like USB (Pen Drive) is a data storage device that uses flash memory with a Universal Serial Bus Interface its very light weight, smaller in size and has capability of maximum storage.

IBM in its Power System version Power9 Introduces NVMe adapter that can be used to connect flash storage to accelerate the computing power of IBM Power System. The NVMe adapter  has a built in flash storage device that can be directly connected to Logical Partition (LPAR) and Virtual Input Output Server (VIOS).

The main purpose of introducing the NVMe Flash Storage to Power System was to caching the information to facilitate faster processing. In the normal processing the data will be moved from CPU to Storage Hard Disk as the capacity of RAM is not much capable to store GB's of code. With the introduction of Flash Caching the data processing will be transacted between CPU and the Flash Cache Disk this leading to faster processing speed, replacing slower Hard Disk Drives.

Below are the Requirement for NVMe on Power System

Hardware : Power9 and above
HMC Version : HMC V910 and above
VIOS Version : VIOS 2260 and above

There are two ways of using the NVMe Storage

  1. Assign directly to Logical Partition (Dedicated Mode) 
  2. Assign it to Virtual I/O Server VIOS and share the cache among the Power System LPAR or Use it for SSP Caching. (Sharing Mode) 

Dedicated Mode

Based on the requirement the NVMe Flash Storage can be assigned directly to the AIX Logical Partition. Upon assigning the Flash Storage to the AIX run it will display a new disk as hdisk with some number upon checking the properties using command it will display as below.

$ cache_mgt device list -l
$ lsattr -El hdisk5 | grep unique_id
unique_id       12345IBM     FlashSystem-8892277096-0000-0160-000012345FlashSystem-12345IBMfcp

To enable the caching between Flash Cache and Hard Disk Drive of AIX partition we need to assign the Flash Storage disk to the rootvg disk of Logical Partition (LPAR). Below command will assign the flash disk(hdisk5) to the Hard Disk Drive (hdisk0)

cache_mgt partition assign ‑P hdisk5 ‑t hdisk0

Suppose there are multiple rootvg disks or you have a requirement to cache multiple disks then its still permissible to enable the caching on multiple disks either its roovg or non rootvg disks can be assigned to single Flash Storage Disk.

cache_mgt partition assign ‑P hdisk5 ‑t hdisk0
cache_mgt partition assign ‑P hdisk5 ‑t hdisk1
cache_mgt partition assign ‑P hdisk5 ‑t hdisk2
cache_mgt partition assign ‑P hdisk5 ‑t hdisk4

Sharing Mode

Assigning the Flash Cache NVMe adapter to Virtual I/O Server for the purpose of sharing the cache disk among multiple Logical Paritions (LPAR's) or Caching the SSP will be termed as Sharing Mode.

Creating Cache Pool and Assigning to Partition

First list out the flash disk(devices) present in the vios and then create a cache pool using the cache_mgt command line

$ cache_mgt device list -l
$ cache_mgt pool create -d hdisk5 -p newpool
Pool newpool created with device hdisk5

After creating the cache pool create cache partition from the pool and assign it to vscsi device of Logical Partition.

$ cache_mgt partition create -p newpool -s 100M -P cachepart01
Partition cachepart01 created in pool newpool
$ cache_mgt cachepart01 assign -v vhost1
Partition cachepart01 assigned to vSCSI Host Adapter vhost1

This will complete the assignment to the Logical Partition, once done run Config Manager command (cfgmgr) and then the new disk will be listed. the assign the cache disk(cachepart01)  to rootg disk and start caching(hdisk0) process.

$ cache_mgt partition assign ‑P cachepart01 ‑t hdisk0
Partition cachepart01 assigned to target hdisk0

Caching the SSP 

If there is a Flash Cache disk present in VIOS and also Share Storage Pool(SSP) cluster is configured on the VIOS then SSP data can be cached for faster processing. This can be done with the help of command sspcache.

sspcache is the command that will automatically cache of the given size from either normal flash disk or it will partition the flash cache pool if cache pool already exists in the VIOS. Below is the usage of the sspcache comma to create and list the cache

$ sspcache -enable -sp -size 100M
Action processed successfully.

Note: This is an asynchronous operation.
      Please use `sspcache -status` to verify that this operation has succeeded
      on all nodes in the cluster at a later time.

      (5-10 minutes is the maximum expected delay.)
$ sspcache -status

Pool Information:
   Pool Name: pool
   Caching Status: ENABLED
   Cache Size: 100MB
$ cache_mgt partition list -l | grep -i ssp

With this SSP caching is enabled if required we can resize the SSP caching or disable the caching from SSP.

Limitations related to NVMe Flash Storage.

  • NVMe Flash Storage can be assigned to Single Partition or VIOS 
    • Assigning Single Huge NVMe Flash Storage to Dedicated LPAR may be loss of computing power as the disk may get under utilized
    • Assigning Flash Storage in Sharing Mode to VIOS will lead to de-allocation of Flash Storage when VIOS Update / downtime. In this scenario the LPAR will run in its normal speed as there is no Flash Storage getting from VIOS.
  • NVMe has high cost and low life cycle, with low life cycle this may lead to data loss and some times flash storage stop working

Post a Comment