NVMe Cache Management in AIX using cache_mgt and sspcache command

NVMe Storage Management in AIX

Non Volatile Memory express RAM is a technology to use the Flash Storage as the intermediate between the CPU and the Hard Disk drive to process large number of code/ programs at faster rate. We have explained the introduction part of NVMe storage usage in AIX here, in this post we will be covering the cache management in more detailed view.


NVMe Cache Management in AIX using cache_mgt and sspcache command


Caching roovg disk of VIOS
NVMe technology in not only limited to Logical Partition caching but also it enables Virtual I/O Server rootvg data to be cached. The rootvg data of VIOS can also be cached as by given below procedure. To start over list out the NVMe disks present in the VIOS and get the details of the disks to find the flash cache disk use cache_mgt command as below  

$ cache_mgt device list -l
hdisk10
hdisk11
Once we find the Flash Cache disk then create a cache pool, cache pool is having multiple flash cache disks combined together forming large storage. Below sample commands shows creation, listing, extending and removing the flash cache pool named mycp. Where hdisk10 and hdisk11 are part of cache pool mycp (mycp is any user defined name as here opted for just an illustration).
$
$ cache_mgt pool create -d hdisk10 -p mycp
Pool mycp created with devices hdisk10.
$
$ cache_mgt pool list -l
mycp,hdisk10
$
$ cache_mgt pool extend -p mycp -d hdisk11 -f
Pool mycp extended with devices hdisk11.
$
$ cache_mgt pool remove -p mycp -f
Pool mycp removed.

Below example will list the cache partition present in the vios then it will create a new chunk of cache partition from cache pool(mypart01 from mycp pool)) then we try to remove the cache partition. Then we try to create the cache partition again and then list the cache partition.

$ cache_mgt partition list -l
$
$ cache_mgt partition create -p mycp -s 10M -P mypart01
Partition mypart01 created in pool mycp.
$
$ cache_mgt partition remove -P mypart01
Partition mypart01 removed.
$
$ cache_mgt partition create -p mycp -s 10M -P mypart01
Partition mypart01 created in pool mycp.
$
$ cache_mgt partition list -l
mypart01,128M,mycp
$
Next we try to extend the cache partition by adding extra 10MB of storage, then again try to list the cache partition present in vios and assign the cache partition to rootvg disk of vios (hdisk0) then try to unassigned the cache partition disk.

$ cache_mgt partition extend -P mypart01 -s 20M
Partition mypart01 extended by size 20M.
$
$ cache_mgt partition list -l
mypart01,256M,mycp
$
$ cache_mgt partition assign -P mypart01 -t hdisk0
Partition mypart01 assigned to target hdisk0.
$
$ cache_mgt partition unassign -t hdisk0
Partition mypart01 unassigned from target hdisk0.
$

There is another option to set the flag for migration and its applicable to the flash cache disk assignment to LPAR when ever the flag is set to yes the migration process will look for the cache disk on the target setup. To set the the migration flag to yes or no we need to follow as given below.

$
$ cache_mgt partition assign -P mypart01 -t hdisk0
Partition mypart01 assigned to target hdisk0.
$ cache_mgt mig get -r -P mypart01
hdisk0,no
$
$ cache_mgt mig set -r yes -P mypart01
Flag 'required' for partition mypart01 set to 'yes'.
$ cache_mgt mig get -r -P mypart01
hdisk0,yes
$

The above examples are for managing the cache disks on the VIOS.


SSP Caching

SSP Cache Management is all about the caching the data related to VIOS cluster so that the cluster procesing is faster. To eanbel the SSP Caching we need to have the cluster getting created first else it will not work. After the cluster gets created assign the Cache disk to SSP using below command.

sspcache is the command that creates the cache disk by prompting the existing Flash Disks or Cache Pool to enable the ssp caching provide the size of flash cache disk size that needed to be used for ssp caching (it is generally taken as MB size)

$ sspcache -enable -sp -size 1000M
    Action processed successfully.
    Note: This is an asynchronous operation.
        Please use `sspcache -status` to verify that this operation has succeeded
        on all nodes in the cluster at a later time.
        (5-10 minutes is the maximum expected delay.)

Once we enable the ssp caching it will take 5 to 10 minutes to enable the caching and we can check the status using below command. Below command display viosv2 where the cache disk is assinged and viosv1/v3/v4 rest three vios are disabled as there is no flash cache disk present on these three vios.

$ sspcache -status
Pool Information:
   Pool Name: pool
   Caching Status: ENABLED
   Cache Size: 1000MB

Node Information:
   Node ‘viosv1' is unable to cache 'pool' because it has no cache pool.

   Node Name: viosv2
   Node ID: ABCD123
   Cached Pool: pool
   Caching LV: cmpart0
   Caching LV Size: 1024MB

   Node ‘viosv3' is unable to cache 'pool' because it has no cache pool.

   Node ‘viosv4’ is unable to cache 'pool' because it has no cache pool.

Statistics argument will display the cache partition name, cache size, Population Percent(percentage of cache that is full), Hit Rate (cache usage time instead of shared storage pool), Cache Target type(storage pool or virtual disk) and SSP Object Target ID.

$ sspcache -statistics
Caching Statistics for Local Node viosv2:
    Cache Name: cmpart0
    Size: 1000 MB
    Target Info:
        Population Percent: 0%
        Hit Rate: 0
        Target Type: SSP Pool
        Target ID: ABCD1234

SSP cache resize will resize the SSP cache disk.

$ sspcache -resize -sp -size 2000M
Action processed successfully.
Note: This is an asynchronous operation.
    Please use `sspcache -status` to verify that this operation has succeeded
    on all nodes in the cluster at a later time.
    (5-10 minutes is the maximum expected delay.)

SSP disable argument will disable the shared storage pool caching.

$ sspcache -disable -sp
Action processed successfully.

Note: This is an asynchronous operation.
      Please use `sspcache -status` to verify that this operation has succeeded
      on all nodes in the cluster at a later time.
      (5-10 minutes is the maximum expected delay.)

With these commands we try to manage the cache disks that are assigned to VIOS disk or to SSP pool.


Caching roovg disk of Logical Partition
With NVMe technology we can virtualize the Flash Storage and asign it to Logical Partition. Its similat to vSCSI disk assignment. Create the NVMe cache partition similar to the above VIOS assignment and assign to the target device. In case of VIOS its the Physical Target device hdisk, in case of Logical Partition assignment its Virtual SCSI Server Adapter.

$ cache_mgt device list -l
hdisk10
hdisk11

Once we find the Flash Cache disk then create a cache pool, cache pool is having multiple flash cache disks combined together forming large storage. Below sample commands shows creation, listing, extending and removing the flash cache pool named mycp. Where hdisk10 and hdisk11 are part of cache pool mycp (mycp is any user defined name as here opted for just an illustration).
$
$ cache_mgt pool create -d hdisk10 -p mycp
Pool mycp created with devices hdisk10.
$
$ cache_mgt pool list -l
mycp,hdisk10
$
$ cache_mgt pool extend -p mycp -d hdisk11 -f
Pool mycp extended with devices hdisk11.
$
$ cache_mgt pool remove -p mycp -f
Pool mycp removed.

Below example will list the cache partition present in the vios then it will create a new chunk of cache partition from cache pool(mypart01 from mycp pool)) then we try to remove the cache partition. Then we try to create the cache partition again and then list the cache partition.

$ cache_mgt partition list -l
$
$ cache_mgt partition create -p mycp -s 10M -P mypart01
Partition mypart01 created in pool mycp.
$
$ cache_mgt partition remove -P mypart01
Partition mypart01 removed.
$
$ cache_mgt partition create -p mycp -s 10M -P mypart01
Partition mypart01 created in pool mycp.
$
$ cache_mgt partition list -l
mypart01,128M,mycp
$
Next we try to extend the cache partition by adding extra 10MB of storage, then again try to list the cache partition present in vios and assign the cache partition to vhost24 Virtual SCSI Adapter of Target LPAR (Get vhost24 by running lsmap -all command) then try to unassigned the cache partition disk.

$ cache_mgt partition extend -P mypart01 -s 20M
Partition mypart01 extended by size 20M.
$
$ cache_mgt partition list -l
mypart01,256M,mycp
$
$ cache_mgt partition assign -P mypart01 -v vhost24
Partition mypart01 assigned to target vhost24.
$
$ cache_mgt partition unassign -v vhost24
Partition mypart01 unassigned from target vhost24.
$

The above examples are for managing the cache disks on the VIOS targeted to LPAR. On the partition run cfgmgr to re configure the device and then perform cache_mgt device list -l. We should be able to list the cache device. To assign newly configured cache partition follow the first procedure of assigning the cache partition to target physical disk as mentioned in VIOS Section.


Post a Comment

0 Comments