Live Kernel Update - AIX

Live Kernel Update LKU using HMC

Live Kernel Update (LKU) is a process of installing the kernel interim fixes (ifix), technology levels (TL) and service packs (SP) live without rebooting the AIX Operating System. IBM Introduced LKU technology in AIX 7.2 and later version, for LKU to be successful the existing software should be committed in AIX OS database.


live kernel update aix


Commit the existing software run smit install >> Software Maintenance and Utilities >> Commit Applied Software Updates (Remove Saved Files) or simply run "installp -c all"

# installp -c all
+-----------------------------------------------------------------------------+
                        Pre-commit Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...

WARNINGS
--------
  Problems described in this section are not likely to be the source of any
  immediate or serious failures, but further actions may be necessary or
  desired.

  Nothing to Commit
  -----------------
  There is nothing in the APPLIED state that needs to be committed.

  << End of Warning Section >>


Step 1: Copying the ifix, TL and SP

All the ifixes, TLs and SPs should be copied to directory /usr/sys/inst.images this is default directory LKU picks the install-able software. If the ifixes, TLs and SPs are present in another directory we need provide directory as input argument to geinstall "geninstall -d /path/to/directory" with required validation "-kp" or starting LKU option "-k" respectively to the process that we are running.

Step 2: Inputs to LKU

Add the inputs required by LKU in file lvupdate.data. Generally add alternative rootvg disk details and HMC details in this file, if any other info is required by software packages we can add here when update process is running and if the input is required to install software then LKU will fail.

# cp /var/adm/ras/liveupdate/lvupdate.template /var/adm/ras/liveupdate/lvupdate.data
# vi /var/adm/ras/liveupdate/lvupdate.data
disks:
        nhdisk  = hdisk18
        mhdisk  = hdisk19
        tohdisk =
        tshdisk =
hmc:
        lpar_id  =
        management_console = hmc.fully.qualified.domain.name.com
        user = hscroot 

Step 3: Exchange HMC Keys

Run hmcauth command to key exchange between AIX partiton and HMC hmc.fully.qualified.domain.name.com using hscroot user

# hmcauth
Enter HMC URI: hmc.fully.qualified.domain.name.com 
Enter HMC user name: hscroot
Enter HMC password: 
#

Step 4: LKU Validation

LKU validation is the process of validating the ifixes, service packages software are LKU capable, LKU Validation is also refereed as preview installation. Perform validation using below command.

# geninstall -kp
*******************************************************************************
Live Update PREVIEW:  Live Update operation will not actually occur.
*******************************************************************************

+-----------------------------------------------------------------------------+
                    Pre-Live Update Verification...
+-----------------------------------------------------------------------------+
Verifying environment...done
Verifying /var/adm/ras/liveupdate/lvupdate.data file...done
Computing the estimated time for the live update operation...done
Results...

EXECUTION INFORMATION
---------------------
  LPAR: partition01.fully.qualified.domain.name
  HMC: 9.x.x.x
  user: hscroot

  Blackout time(in seconds): 50
  Total operation time(in seconds): 2130

  << End of Information Section >>

+-----------------------------------------------------------------------------+
                    Live Update Requirement Verification...
+-----------------------------------------------------------------------------+

INFORMATION
-----------
INFO: Any system dumps present in the current dump logical volumes will not be available after live update is complete.

  << End of Information Section >>

+-----------------------------------------------------------------------------+
                    Live Update Preview Summary...
+-----------------------------------------------------------------------------+
The live update preview succeeded.

*******************************************************************************
End of Live Update PREVIEW:  No Live Update operation has actually occurred.
*******************************************************************************

Step 5: Live Kernel Update


geninstall (General Install) command will start the live kernel update process, this process is non interruptable process and it requires telnet session or should be run in background (nohup geninstall -k &). This process cannot be run under vterm and ssh because it will close the vterm / ssh connection during the middle of the process. When LKU process is going on no other installation should take place else it may interrupt the LKU.

# geninstall -k
+-----------------------------------------------------------------------------+
                    Pre-Live Update Verification...
+-----------------------------------------------------------------------------+
Verifying environment...done
Verifying /var/adm/ras/liveupdate/lvupdate.data file...done
Computing the estimated time for the live update operation...done
Results...
EXECUTION INFORMATION
---------------------
  LPAR: partition01.fully.qualified.domain.name
  HMC: 9.x.x.x
  user: hscroot
  Blackout time(in seconds): 12
  Total operation time(in seconds): 5130
  << End of Information Section >>
+-----------------------------------------------------------------------------+
                    Live Update Requirement Verification...
+-----------------------------------------------------------------------------+
INFORMATION
-----------
INFO: Any system dumps present in the current dump logical volumes will not be available after live update is complete.
  << End of Information Section >>
+-----------------------------------------------------------------------------+
                    Live Update Preview Summary...
+-----------------------------------------------------------------------------+
The live update preview succeeded.
Non-interruptable live update operation begins in 10 seconds.

Broadcast message from root@partition01.fully.qualified.domain.name
Broadcast message from root@ (pts/0) at 04:12:55 ...
Live AIX update in progress.

Initializing live update on original LPAR.
Validating original LPAR environment.
Beginning live update operation on original LPAR.
Requesting resources required for live update.
................................................................................................................................
Notifying applications of impending live update.
............
Creating rootvg for boot of surrogate.
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Starting the surrogate LPAR.
........................................................................................................................................................................................................................................................................................
Creating mirror of original LPAR's rootvg.
........................................................................................................................................................................................................................................................................................................................................
Moving workload to surrogate LPAR.
............................
        Blackout Time started.

        Blackout Time end.
Workload is running on surrogate LPAR.
................................................................................................................................................................................................................................................................................................................................................................................................................................
Shutting down the Original LPAR.
................................................................................................................
The live update operation succeeded.

Broadcast message from root@partition01.fully.qualified.domain.name
Broadcast message from root@ (pts/0) at 07:01:33 ...
Live AIX update completed.
# echo $?
0
#

Step 6: Debugging LKU Success / Failure logs


LKU process will generate log fine in the path /var/adm/ras/liveupdate/logs/lvupdlog we can open another session and track the update process. Upon failures we can check if any failures or the errors are present as below.

tail -f /var/adm/ras/liveupdate/logs/lvupdlog
cat /var/adm/ras/liveupdate/logs/lvupdlog | grep -i fail
cat /var/adm/ras/liveupdate/logs/lvupdlog | grep -i error


errpt entries will also display the event reports

# errpt | grep Live
12295E0B   0528070120 I S LVUPDATE       Live AIX update completed successfully
9A74C7AB   0528040920 I S LVUPDATE       Live AIX update started


When update is started 9A74C7AB event is generated and when update is completed 12295E0B gets generated.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Live Kernel Update LKU using PowerVC  

In above article we had seen the process of Live Update using Hardware Management Console HMC. But in this article we will be looking into the process of LKU using PowerVC method.
 
The method of performing the Live Kernel Update is same but with slight difference that will be explaining it in the below steps.

1. LKU with Migration

So consider the case where the user needs to perform LKU on manage system and there is very minimal amount of resource such as CPU and Memory on the Manage System. The Live Kernel Update will fail at the initial preview or verification stage itself. 
 
So in order LKU to work in this case there is an option of LKU with Migration option where user can migrate the partition to another manage system having abundant hardware resource and then perform the Live Kernel Update and then Migrate the Logical Partition back to its original manage system. With this option there should be an extra manage system / temporary manage system should be provided that acts as middlemen.

2. LKU with out Migration

LKU with out Migration requires the manage system that the Logical Partition is running should have resources (CPU/Memory) in adundance so that the requirement to run and complete the Live Kernel Update process can be completed using its own hardware resources. This can be considered similar as HMC running LKU process where instead of HMC we are Using PowerVC console.

Step 1 : Commit the packages, Copy ifixes and Prepare Template

Commit the AIX software templates run the command "installp -c all", Copy the downloaded ifixes, TL's, SP's into the directory "/usr/sys/inst.images". Prepare the Live Update template data by copying the template

# cp /var/adm/ras/liveupdate/lvupdate.template /var/adm/ras/liveupdate/lvupdate.data

Step 2: Set the Template 
 
Edit the file add the below lines "/var/adm/ras/liveupdate/lvupdate.data", where management_consoel is the fully qualified domain name, user is the root user of PowerVC, force_migration set to yes or no depending weather you require the Migration to take place forcefully and destination is the name of manage system connected to PowerVC where Migration of LPAR takes place temporary.
 
general:
        kext_check = yes

pvc:
        management_console = powervc.fully.qualified.domain.name.com
        user = root
        force_migration = no
        destination = cec1


Step 3: Authenticate with PowerVC
 
Authenticate with PowerVC using the pvcauth AIX command line with -a as powervc.fully.qualified.domian.name.com and -u is the PowerVC username and -p is the PowerVC password

# pvcauth -u root -p ****** -a powervc.fully.qualified.domain.name.com
 
 
Step 4: Run Preview 
 
Run the live update preview / LKU verification process using geninstall -kp command 
 
# geninstall -kp

*******************************************************************************
Live Update PREVIEW:  Live Update operation will not actually occur.
*******************************************************************************

+-----------------------------------------------------------------------------+
                    Pre-Live Update Verification...
+-----------------------------------------------------------------------------+
Verifying environment...done
Verifying /var/adm/ras/liveupdate/lvupdate.data file...done
Computing the estimated time for the live update operation...done
Results...

EXECUTION INFORMATION
---------------------
  LPAR: partition01.fully.qualified.domain.name
  PowerVC: 9.x.x.x
  user: root

  Blackout time(in seconds): 10
  Total operation time(in seconds): 1492

  << End of Information Section >>

+-----------------------------------------------------------------------------+
                    Live Update Requirement Verification...
+-----------------------------------------------------------------------------+

INFORMATION
-----------
INFO: Any system dumps present in the current dump logical volumes will not be available after live update is complete.
INFO: Temporary migration server: cec1.

  << End of Information Section >>

+-----------------------------------------------------------------------------+
                    Live Update Preview Summary...
+-----------------------------------------------------------------------------+
The live update preview succeeded.

*******************************************************************************
End of Live Update PREVIEW:  No Live Update operation has actually occurred.
*******************************************************************************
#

 
Step 5: Run Live Update Process
 
One the Live Update Preview is successful start the live kernel update process using geninstall -k option similar command as mentioned above. 

#
# nohup geninstall -k >> update.log &
# tail -f update.log

+-----------------------------------------------------------------------------+
                    Pre-Live Update Verification...
+-----------------------------------------------------------------------------+
Verifying environment...done
Verifying /var/adm/ras/liveupdate/lvupdate.data file...done
Computing the estimated time for the live update operation...done
Results...

EXECUTION INFORMATION
---------------------
  LPAR: partition01.fully.qualified.domain.name
  PowerVC: 9.x.x.x
  user: root

  Blackout time(in seconds): 10
  Total operation time(in seconds): 1913

  << End of Information Section >>

+-----------------------------------------------------------------------------+
                    Live Update Requirement Verification...
+-----------------------------------------------------------------------------+

INFORMATION
-----------
INFO: Any system dumps present in the current dump logical volumes will not be available after live update is complete.
INFO: Temporary migration server: cec1.

  << End of Information Section >>

+-----------------------------------------------------------------------------+
                    Live Update Preview Summary...
+-----------------------------------------------------------------------------+
The live update preview succeeded.

Non-interruptable live update operation begins in 10 seconds.


Broadcast message from rot@partition01 (pts/1) at 10:00:47 ...

Live AIX update in progress.


Initializing live update on original LPAR.

Validating original LPAR environment.

Beginning live update operation on original LPAR.
............................................................................................................................................................................................
Migrating to temporary destination.
................................................................................
Requesting resources required for live update.
....................................................................................................
Notifying applications of impending live update.

Creating rootvg for boot of surrogate.
........................................................................................................................................................................................................................................................
Starting the surrogate LPAR.
................................................................................................................................................................................................................................................................................
Creating mirror of original LPAR's rootvg.
............................................................................................................................................................................................................................................................................................
Moving workload to surrogate LPAR.
............
        Blackout Time started.

        Blackout Time end.

Workload is running on surrogate LPAR.
................................................................................................................................................................................................................................
Shutting down the Original LPAR.
................................................................................................
Migrating back to initial host.
........................................................................................................................................................

Broadcast message from root@partition01 (pts/1) at 11:12:22 ...

Live AIX update completed.

....
The live update operation succeeded.

#

Step 6: Verify the Completion of LKU with PowerVC Option 
 
To Verify the completion of LKU with PowerVC option check errpt we will be seeing the logs of LKU started and completed as well as the migration started and completed.


#
# errpt
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
12295E0B   0316111221 I S LVUPDATE       Live AIX update completed successfully
A5E6DB96   0316110721 I S pmig           Client Partition Migration Completed
08917DC6   0316110721 I S pmig           Client Partition Migration Started
A5E6DB96   0316101121 I S pmig           Client Partition Migration Completed
08917DC6   0316101021 I S pmig           Client Partition Migration Started
9A74C7AB   0316095721 I S LVUPDATE       Live AIX update started
#
#














AIX Kernel Update with Reboot

Here we will be explaining the option to update the SP, TL and ifixes packages with reboot option. This requires only two simple steps as we have the reboot option and the process is not live. We will be explaining the update process as below

Step 1: Copy the file sets and Install

Download the file sets of TL or SP or ifixes into the directory /usr/sys/inst.images and install the file sets using inutoc AIX command

# inutoc /usr/sys/inst.images 
 
inutoc command creates .toc file that will be later used by installp command for package installation. Optionally you can also run bffcreate -c -d /usr/sys/inst.images for creating the installed packages in backup file format and then run inutoc command again on same directory inutoc /usr/sys/inst.images 
 
 
Step 2: Update the packages

To install and update the packages on AIX Logical Partition run smit update_all command and reboot the lpar
 
 
# smit update_all
# reboot
 
Will update the AIX with latest patches/ TL's/  SP's and ifixes