Viktor Balogh's HP-UX blog

Bug: Installing a Netbackup Media server in a HP-UX SRP-enabled environment

Viktor Balogh — Thu, 15 Dec 2011 14:50:27 +0000

This is a bug I found during the installation of a Netbackup Media server. The target system was an HP-UX 11iv3 which has several SRP containers integrated into ServiceGuard. As I started the installation script on the global container, it failed with an swinstall error stating that it couldn’t set the SRP containers into maintenance mode. You should know that we plan to use the the media server from the global zone, and the client will be installed also only to the global zone.

That’s because two new options were added to the swinstall command to support the Secure Resource Partition functionality. The first one is the global_srp option, by which you can decide if the installation/removal should affect the global system. The other option is the local_srp_list which contains space separated system container names enclosed in quotes.

The behavior is as follows:

If neither global_srp nor local_srp_list options are specified, then the install, remove, or configure will occur on the system and all system containers.
If the global_srp=true and local_srp_list options are not specified, then the install, remove, or configure will occur only on the system.
If the global_srp=true and local_srp_list options are set, then the install, remove, configure will occur on the system and the listed system containers.
If the local_srp_list option is set and the global_srp option is not specified, then the install, remove, configure will occur only in the listed system containers.

The Netbackup Media server installation script untars a package, and calls a script within it which does swinstall among several things. So you just need to untar that package, and find the following script: usr/openv/netbackup/client/HP-UX-IA64/HP-UX11.31/installpbx

As we want the media server installed only into the global zone, search for the swinstall procedure (line 1035) in the installpbx script and change this one:

swinstall -x "mount_all_filesystems=false" \

to this:

swinstall -x "mount_all_filesystems=false" -x global_srp=true \

After that, re-package the files with tar, and the install script works now without any errors, the media server will be installed after the next-next-finish procedure.

Update: Here is some related information for installing Netbackup client within the SRP.

Configure RAID mode on a P410i

Viktor Balogh — Thu, 08 Dec 2011 11:06:57 +0000

This post is the successor of the previous one, which was written on how to change your P410i controller to RAID mode. Setting the controller to RAID mode is just half the job, you need to define the RAID level and assign physical disks to it. For this you can use the drvcfg -s command at the EFI shell to get into the P410i BIOS.

To determine the drv use the drivers command from EFI shell prompt.
Find the SAS Host Adapter and make a note of the Driver ID from the left column. That is the drv.
To determine the ctr use the drvcfg command from the efi shell prompt. Look for the SAS Host Adapter drv number and the corresponding number is the ctr:
Now type in drvcfg -s and the Bios or ORCA menu should appear.
To create a mirror, choose Create Logical Drive
Select the 2 physical disks that will be part of the mirror. Use the tab key to choose the Raid configuration. Press Enter . One should see the following menu:
Press F8 or esc 8 to save the configuration.

Source: HP Business Support Center

Changing a P410i from HBA mode to RAID mode

Viktor Balogh — Wed, 30 Nov 2011 10:16:36 +0000

So, you’ve just unboxed your freshly came Integrity server, which has a P410/P410i SAS controller. (P410i is an onboard integrated version of the same chipset.) To a P410i controller one can attach a maximum of two SAS drives, and the controller has two operation modes: either it can function as a plain SAS controller without any other intelligence, or we can have the capability of a RAID controller if we set it to RAID mode. Usually when you have to deal with brand new servers, the controller is set to SAS mode, but it is a good practice to install a system onto a hardware-based RAID 1 volume. In order to change the mode you should follow these steps:

Update the firmware of the disk controller.
Change the controller to RAID mode

1. Update the firmware of the disk controller.

Besides it is a best practice to update the firmware of every component to the newest recommended level, there are some known issues which you can avoid. By downloading the firmware, you will get two files: saupdate.efi and a sandman_xxxxxx.bin. The former is the binary which you can run from EFI shell, the latter is the firmware itself which will be uploaded to ROM. Put these two files onto a FAT32 USB stick and at system boot type map -r at the EFI shell.

Now change the pwd to the USB stick with the fs0: command and start the firmware update with:

fs0:> saupdate update p410i sandman_xxxxxx.bin

2. Change the P410i from HBA mode to RAID mode

You can query the current operation mode of your P410i controller like this:

fs0:\> saupdate get_mode p410i

The controller at 0:2:0:0 is in HBA mode

The controller at 0:42:0:0 is in HBA mode

The controller at 0:82:0:0 is in HBA mode

The controller at 0:C2:0:0 is in HBA mode

fs0:\>

For changing to RAID mode you should know that this will destroy the data on the disks, but as I have a new system it is no problem.

fs0:\> saupdate set_mode p410i raid

Warning:Changing controller mode could result in DATA LOSS!!!
Key in 'YES' to continue or 'NO' to quit:yes

Changing mode of the controller at 0:2:0:0 to RAID
Resetting and reinitializing controller, it may take several minutes
Controller mode change successful

Changing mode of the controller at 0:42:0:0 to RAID
Resetting and reinitializing controller, it may take several minutes
Controller mode change successful

Changing mode of the controller at 0:82:0:0 to RAID
Resetting and reinitializing controller, it may take several minutes
Controller mode change successful

Changing mode of the controller at 0:C2:0:0 to RAID
Resetting and reinitializing controller, it may take several minutes
Controller mode change successful
fs0:\>

At the end of the procedure you should type reconnect -r at the EFI shell. Now that our disk controller operates in RAID mode, in the next post I’ll show you how to setup the RAID array.

Checking the status of multipathing in 11iv3

Viktor Balogh — Thu, 24 Nov 2011 12:46:50 +0000

A common task in a sysadmin’s life is to assist to the changes which are affecting the SAN network. At smaller organizations it could be that besides the system administrator role you also need to manage the storage side. So here’s a post which might help you if you need to check the status of the LUN paths. For checking the availability of the LUNs, you can use the ioscan command like this:

# ioscan -P health -C disk
Class     I  H/W Path  health
===============================
disk      1  64000/0xfa00/0x1   online
disk      4  64000/0xfa00/0x6   online
disk      7  64000/0xfa00/0x7   online
disk     10  64000/0xfa00/0x8   online
disk     13  64000/0xfa00/0x9   online
disk     16  64000/0xfa00/0xa   online
disk     19  64000/0xfa00/0xb   online
...

In the above output you can see these health conditions:

online
offline
limited
disabled

Online means that all the paths are usable. However this does not necessary mean that all the paths are active. In case you are using active/passive multipathing only one path is active, the others are in stanby state. Offline means that the specific LUN is not accessible on either of its paths. If you see limited health, then then the LUN is not on all the paths are accessible. Disabled means that the LUN has been suspended as a result of some sort of error.

If you see a limited health in the above output, you might want to know qith which path you have problems. It’s easy to list the health status for each paths:

# ioscan -P health -C lunpath
Class     I  H/W Path  health
===============================
lunpath   0  0/0/0/2/0/0/0.0x0.0x0                                online
lunpath   1  0/0/0/2/0/0/0.0x0.0x4000000000000000                 online
lunpath   2  0/0/0/5/0/0/0.0x50014380113737e8.0x0                 online
lunpath   4  0/0/0/5/0/0/0.0x50014380113737e8.0x4001000000000000  online
lunpath   6  0/0/0/5/0/0/0.0x50014380113737e8.0x4002000000000000  standby
lunpath   8  0/0/0/5/0/0/0.0x50014380113737e8.0x4003000000000000  online
lunpath  10  0/0/0/5/0/0/0.0x50014380113737e8.0x4004000000000000  online
lunpath  13  0/0/0/5/0/0/0.0x50014380113737e8.0x4005000000000000  standby
lunpath  15  0/0/0/5/0/0/0.0x50014380113737e8.0x4006000000000000  online
lunpath  16  0/0/0/5/0/0/0.0x50014380113737e8.0x4007000000000000  standby
lunpath  18  0/0/0/5/0/0/0.0x50014380113737e8.0x4008000000000000  online
lunpath  20  0/0/0/5/0/0/0.0x50014380113737e8.0x4009000000000000  standby
lunpath  23  0/0/0/5/0/0/0.0x50014380113737e8.0x400a000000000000  online
lunpath  24  0/0/0/5/0/0/0.0x50014380113737e8.0x400b000000000000  standby
...

The health column in the above output may have the following values:

online
offline
unusable
disable
standby

Online is self-explanatory, just as offline. Unsable means that the lunpath authentication has failed. You get a disable status if a lunpath has been suspended because of an error. And finally you’ll see a standby status for all the LUNs which are accessible through an active/passive policy.

Note that the above checks are only valid if you are using the new mass storage stack which was introduced in 11iv3. If you are using legacy DSFs or have an older HP-UX version then you either use some software for multipathing (like HP Secure Path, EMC PowerPath or whichever SW comes with your storage), or multipathing is done by your volume manager (like LVM or VxVM) or you don’t use multipathing at all. In these cases you should have to figure out the method to check the paths (or you can search on this blog maybe I already wrote a post about that method, like this EMC cheatsheet.)

Recommended BladeSystem Firmware Installation Order

Viktor Balogh — Wed, 28 Sep 2011 12:15:25 +0000

For HP c-class BladeSystem Enclosures if there is an OS installed to the blade, update the firmware in the following recommended order:

Update the firmware of the server blades
Update the firmware of the Onboard Administrators
Now reboot the systems
Update any offline-only firmwares like Emulex or QLogic adapters
Update the firmware of the Virtual Connects using VCSU

For Blade servers without OS, the following order is recommended:

Update the Onboard Administrator firmware
Update the firmware of the Virtual Connect modules with VCSU
Update the firmware of any server-specific components

source: hp.com

Oracle Grid Control on HP-UX: swap shortage

Viktor Balogh — Wed, 21 Sep 2011 12:12:35 +0000

If you are using Oracle Grid Control on HP-UX you may have problems with swap usage. The values reported by Grid Control could be much higher as the values reported by the system trough swapinfo. Maybe it just didn’t turn out until you got a warning something like

“Swap Utilization is 100%, crossed warning (80) or critical (95) threshold.”

That’s how it looked like on our system in Enterprise Manager:

There is a so-called pseudo swap feature in HP-UX, by which the system uses RAM as swap space in order to be able to run more processes. Unfortunately Oracle Grid Control doesn’t know about this feature and thus it counts the free swap space incorrectly. The pseudo swap feature is on by default, but one cannot be sure so here are some ways to determine if it’s on. Either you can check the kernel parameter swapmem_on, or take a look at the swapinfo output: if it contains a line starting with “memory” then it’s on:

# swapinfo -ta
          Kb      Kb      Kb     PCT    START/     Kb
TYPE      AVAIL    USED    FREE    USED    LIMIT RESERVE  PRI  NAME
dev      196608   16616  179992      8%        0       -    1  /dev/vg00/lvol2
reserve       -   69260  -69260
memory    66588   38740   27848     58%
total    263196  124616  138580     47%        -       0    -
#

If you are using Oracle Grid Control and the reported free swap percentage is more than in the swapinfo output, you should set the following property to have it properly counted:

NMUPM_USE_PSEUDO_MEM=TRUE

It’s in the file $AGENT_HOME/sysman/config/emd.properties

Exporting SQL*Plus output to HTML

Viktor Balogh — Thu, 25 Aug 2011 13:52:35 +0000

Nowadays I deal with Oracle databases, and I found something cool which might be of interest to you. So here is a quick way to capture the SQL*Plus output to a html file:

SQL> set markup HTML on
SQL> spool index.html
SQL> -- do something
SQL> spool off
SQL> set markup HTML off

This captures the whole output in HTML format besides STDOUT into the given index.html file.

Exporting putty settings

Viktor Balogh — Fri, 19 Aug 2011 13:46:10 +0000

Here is a quick howto on how to import your saved hosts in putty from one machine to another. Putty stores its settings in the windows registry. With the following one-liner typed into the “Start menu” > “Run” you can save your settings as a .reg file.

regedit /e "%userprofile%\desktop\putty.reg" HKEY_CURRENT_USER\Software\Simontatham

After this the .reg file has been created on your desktop, you should copy it to the destination host and double-click it to import.

Glance adviser scripts

Viktor Balogh — Fri, 08 Jul 2011 08:19:52 +0000

The first most important thing in proactive performance tuning is to establish a baseline. This can be done by capturing performance data at normal operation. If you are familiar with the interactive glance, and want to have the output in that familiar format, you can automate data collecting with adviser scripts and have the output in an ASCII file. This is how to get started with adviser scripts:

You have a sample adviser syntax file here: /var/opt/perf/adviser.syntax

Here is a sample of mine which captures physical I/O data:

# cat glance.syntax
PRINT "---------------------", gbl_stattime, "--------------------"
DISK LOOP
PRINT bydsk_devname, " write rate: ", bydsk_phys_write_rate, " read rate: ", bydsk_phys_read_rate
#

You can run glance with this adviser script in non-interactive mode like this (with redirection you can capture the output into a file):

# glance -adviser_only -iterations 2 -syntax ./glance.syntax

Welcome to GlancePlus

---------------------10:09:47--------------------
0/1/1/0.0.0                              write rate:         2.8 read rate:         0.0
0/1/1/1.8.0                              write rate:         2.8 read rate:         0.0
0/1/1/0.1.0                              write rate:         0.0 read rate:         0.0
0/1/1/1.10.0                             write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.2.64.0.0.0.0                 write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.1.64.0.0.0.2                 write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.2.64.0.0.0.2                 write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.1.64.0.0.0.0                 write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.1.64.0.0.0.1                 write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.2.64.0.0.0.1                 write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.1.64.0.0.0.0                 write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.2.0.0.0.0.0                  write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.1.64.0.0.0.2                 write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.2.0.0.0.0.2                  write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.1.64.0.0.0.1                 write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.2.0.0.0.0.1                  write rate:         0.0 read rate:         0.0
---------------------10:09:52--------------------
0/1/1/0.0.0                              write rate:         4.4 read rate:         0.0
0/1/1/1.8.0                              write rate:         3.2 read rate:         0.0
0/1/1/0.1.0                              write rate:         0.0 read rate:         0.0
0/1/1/1.10.0                             write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.2.64.0.0.0.0                 write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.1.64.0.0.0.2                 write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.2.64.0.0.0.2                 write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.1.64.0.0.0.0                 write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.1.64.0.0.0.1                 write rate:         0.0 read rate:         0.0
0/3/1/0/4/0.2.64.0.0.0.1                 write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.1.64.0.0.0.0                 write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.2.0.0.0.0.0                  write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.1.64.0.0.0.2                 write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.2.0.0.0.0.2                  write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.1.64.0.0.0.1                 write rate:         0.0 read rate:         0.0
0/4/1/0/4/0.2.0.0.0.0.1                  write rate:         0.0 read rate:         0.0

#

For more info read the following:

http://h20247.www2.hp.com/enterprise/downloads/Glance.pdf

and the on-system pdf document: /opt/perf/paperdocs/gp/C/adviser.pdf

Physical vs. Logical I/O

Viktor Balogh — Tue, 05 Jul 2011 08:46:34 +0000

If you do performance tuning or troubleshooting with glance probably you have observed the terms logical I/O and physical I/O. This post is to clarify these shortly:

Physical I/O: the traffic toward the storage subsystem.

Logical I/O: the requests which come from running processes and threads trough system calls. If the requested data is not to be found in memory, it generates a physical I/O and the system fetches it from the disks.

Note that some tools are reporting either logical or physical I/O, and they simply flag it as “I/O” so you might not have the idea about what kind of I/O it reports. Built-in application performance monitoring tools are reporting the logical I/O (e.g. statspack, awr) while system tools like sar or iostat are reporting the physical I/O. To be sure use glance, it is capable of showing both physical and logical I/O if you select a process or thread.