Patching ZFS based SPARC system with Live Upgrade

Patching ZFS based SPARC 

system with Live Upgrade

Solaris – Patching with Live Upgrade, ZFS makes it so much easier

Solaris Live Upgrade is a superb tool that lets your operating system create an alternate boot environments. Live Upgrade is a simple way to update or patch systems and minimize downtime and mitigate risks often associated with patching efforts.  An admin can patch the system quickly without any system interruption and this is done by patching the alternate boot environment which the system will boot from on the next reboot after having been activated. Live Upgrade creates a copy of the active boot environment, and that copy is given a name. That copy becomes the alternate BE or boot environment. Because there are multiple BE’s or boot environments, the true beauty of Live Upgrade shows through. If problems occur with the newly created or patches BE, the original BE could be used as the backup plant boot image. So reverting back to a previous BE is the back-out plan for almost all Live Upgrade installations. Historically with UFS for even (I dread those days) with SVM, lucreate command was much more complicated as you had software raid. ZFS with snapshots and pools makes it so easy, it’s astounding. At the OBP or boot prom level, it’s mostly the same. At the ok promg, a boot -L will list the BE’s assuming the correct boot disk is mapped properly.

Live Upgrade and patching

Patching a Solaris 10 ZFS based system is done the same way you would path any basic Solaris system. You should be able to patch the Solaris 10 ZFS based system with Live upgrade successfully, and with no outages. The patches are downloaded and unzipped in a temporary location (not on /tmp)    Assumptions are that you have a valid and working rpool with zfs volumes. Lets look at our existing BE’s and the active boot environment is Nov2012.

# lustatus
Boot Environment           Is       Active Active    Can    Copy      
Name                       Complete Now    On Reboot Delete Status    
————————– ——– —— ——— —— ———-
Nov2012                    yes      yes    yes       no     -         
Oct2012                    yes      no     no        yes    -      
I need a new BE for next month, December. I normally have 2 BE’s and reotate and lurename them. But for this blog article, I will crate a new one.
# lucreate -n Dec2012
Analyzing system configuration.
Updating boot environment description database on all BEs.
Updating system configuration files.
Creating configuration for boot environment <Dec2012>.
Source boot environment is <Nov2012>.
Creating file systems on boot environment <Dec2012>.
Populating file systems on boot environment <Dec2012>.
Analyzing zones.
Duplicating ZFS datasets from PBE to ABE.
Creating snapshot for <rpool/ROOT/s10s_u10wos_17b> on <rpool/ROOT/s10s_u10wos_17b@Dec2012>.
Creating clone for <rpool/ROOT/s10s_u10wos_17b@Dec2012> on <rpool/ROOT/Dec2012>.
Mounting ABE <Dec2012>.
Generating file list.
Finalizing ABE.
Fixing zonepaths in ABE.
Unmounting ABE <Dec2012>.
Fixing properties on ZFS datasets in ABE.
Reverting state of zones in PBE <Nov2012>.
Making boot environment <Dec2012> bootable.
Population of boot environment <Dec2012> successful.
Creation of boot environment <Dec2012> successful.

Pretty slick. The lucreate in conjunction with ZFS created the rpool/ROOT/s10s_u10wos_17b@Dec2012 snapshot which was then cloned to rpool/ROOT/Dec2012. The rpool/ROOT/Dec2012 clone is what you will see at the OBP when you do a boot -L . Lets look at our BE’s status

# lutatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
————————– ——– —— ——— —— ———-
Nov2012                    yes      yes    yes       no     -
Oct2012                    yes      no     no        yes    -
Dec2012                    yes      no     no        yes    -

Lets patch the new Dec2012 BE.  Assumption here is that we have downloaded the latest recommended patch cluster from Sun or Oracle site. (depends who you have alliegience with). Lets patch the BE while the system is running, and doing whatever the system is supposed to do. Lets say it’s a DNS/NTP/Jumpstart server? Don’t know. Could be anything. I’ve downloaded the patch cluster, unziped it in /var/tmp

# uname -a
SunOS tweetybird 5.10 Generic_147440-12 sun4v sparc sun4v
# cd /var/tmp
# luupgrade -n Dec2012 -s /var/tmp/10_Recommended/patches -t `cat patch_order`
Validating the contents of the media </var/tmp/10_Recommended/patches>.
The media contains 364 software patches that can be added.
Mounting the BE <Dec2012>.
Adding patches to the BE <Dec2012>.
Validating patches …
Loading patches install installed on the system…
Done!
Loading patches requested to install.
…
Unmounting the BE <Dec2012>

The patch add to the BE <Dec2012> completed.

# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
————————– ——– —— ——— —— ———-
Nov2012                    yes      yes    yes       no     -
Oct2012                    yes      no     no        yes    -
Dec2012                    yes      no     no        yes    -
# luactivate Dec2012
# lutatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
————————– ——– —— ——— —— ———-
Nov2012                    yes      no     yes       no     -
Oct2012                    yes      no     no        yes    -
Dec2012                    yes      yes    no        yes    -

Lets reboot and makes sure the prober BE comes up. Use must use either init or shutdown, do not use halt or fastboot.

# init 6

After the server reboots, the Dec2012 should automatically be booted with the newly implemented patch bundle. So the Dec2012 is the new active BE. Lets check the kernel patch level:

# uname -a

SunOS tweetybird 5.10 Generic_147440-26 sun4v sparc sun4v

Looks good. With ZFS, Live Upgrade it’s so simple now. Heck, Live Upgrade workds wonders when you have a UFS based root volume and you dearly want to migrate over to a ZFS root volume. You will need the ZFS capable kernel to start. Create a pool called rpool using slices not the whole disk, then lucreate it to the rpool, activate it and then reboot and you are booting off of a new ZFS based Solaris system. There are a few tricks about creating the proper type of rpool. Maybe another blog entry on this. But Live Upgrade is a great tool for migrating UFS systems to ZFS. Again, with a slick backout option.

Disaster – You need an easy backout plan

Thanksfully having multiple BE’s you can choose to backout simply by choosing one of the previously installed BE’s. If the system boots up without trouble but applications are failing, simply luactivate the original BE and reboot. If the system fails to boot (yikes, this is rare), the from the boot prom, list the BE’s and choose the BE to boot from.

ok boot -L
.
.
.
Boot device: /pci@1f,0/pci@1/scsi@8/disk@0,0 File and args: -L
zfs-file-system
Loading: /platformsun4v/bootlst
    1.Nov2012
    2 Octv2012
    3 Dec2012
Select environment to boot: [ 1 - 3 ]: 1
to boot the selected entry, invoke:
boot [<root-device] -Z rpool/ROOT/Nov2012

and off you go. In special cases, when you have to backout and boot from the original BE and it fails, you will need to boot in fail safe mode and mount the current BE root slice and import the rootpool. Instructions are as follows:

ok boot –f  Failsafe

Now mount the current BE root slice to /mnt.

# zpool import rootpool
# zfs inherit -r mountpoint rootpool/ROOT/Dec2012
# zfs set mountpoint=/mnt  rootpool/ROOT/Dec2012
# zfs mount rootpool/ROOT/Dec2012

Here we are activating the previously (known good) BE

# /mnt/sbin/luactivate

If this works, you are golden , now reboot with init 6.

# init 6

Please Note Live Upgrade and LDom’s Require an Extra Step A quick note about Live Upgrade, ZFS and LDom’s. Preserving the Logical Domains Constraints database file when using the Oracle Solaris 10 Live Upgrade feature requires some hand holding. This is a special situation. If you are using Live Upgrade on a Control Domain, you need to enter the following line to the bottom of the /etc/lu/synclist file. As in append this line.

# echo “/var/opt/SUNWldm/ldom-db.xml     OVERWRITE” >> /etc/lu/synclist

This line is important, as it forces the database to be copied automatically from the active boot environment to the new boot environment when you switch boot environments. Otherwise, as you may well guess, you loose your LDom configuration.