Logging TSM Restores

If your wondering who is restoring files, and what they’re restoring, you can capture that information to a custom log file. It’s a pain to get the same kind of data out of the activity log, but you can dump it to a “formatted” text file. You can send it to a user defined script or program too. The offending codes we’re looking at are ANR0503 and ANR0504.

First, we need some options added to our dsmserv.opt file:

REPORTRETRIEVE YES
FILETEXTEXIT YES /var/log/tsm_res.log APPEND

The REPORTRETRIEVE option tells the server to log restores. You could stop with that, and be content to do searches of the activity log manually via the client. This will increase the volume of activity in the log, maybe significantly depending on how many restores you do. The FILETEXTEXIT command writes the specified lines to a file. And, we want it to startup automatically on reboot, write to the /var/log/tsm_res.log file, and APPEND new lines to that file. Once you’ve made these changes, bounce the TSM instance.

Now we have to go into the TSM admin client and enable the events to the exit with:

ENABLE EVENTS FILETEXT ANR0503,ANR0504

You can check the status of the receiver with “Q ENABLED FILETEXT”. You could also add “NODENAME=xxxxxx” to enable this for just certain nodes, which is pretty cool if you need to watch just a handfull of nodes.

There are a few more options for the above commands, check your Admin command reference or the TSM Infocenter website for other options.

The output in the file looks a lot like the TSM activity log, with a few fields added, like the date and the TSM server name.

Mac Terminal Tweaks

I spend most of my day in the terminal doing one thing or another. While the default setting work pretty well on the Mac, it’s lacking while connecting to a remote system. Here are some of the tweaks I’ve made to make the terminal app more useful for me.

Not all systems know how to handle a terminal type of xterm-color. I did find a terminal definition for xterm-color and successfully applied it to AIX, but I’d rather not do that to all my systems. So, you can change the terminal type to xterm by default. Under Terminal Preferences -> Settings -> Advanced, change the “Declare terminal as” to xterm. You can also select dtterm (not as well supported on AIX, and not color) or several flavors of vtterm.

For some reason, Apple thinks the page-up and page-down keys should scroll the buffer back. I’d rather actually use page-up and page-down in my terminal than just using it to scroll back. You can adjust the key codes under Terminal Preferences -> Settings -> Keyboard. Then, edit the keys and set them as:

page down: \033[6~
page up: \033[5~

You can also try:
Home: \001 or \033[7~
End: \005 or \033[4~

It also bugs me that the terminal doesn’t exit when I logout. I then end up going to the mouse and closing the window or hitting Command-W. You can make the terminal close, only if the program it’s running exits with a 0 return code. Go to Terminal Preferences -> Settings -> Shell, and change “When the shell exits” to “Close if the shell exited cleanly”.

I’d like it if the terminal name changed to the system I’m connected to, but I haven’t found how to do that yet. I have made the terminal window bar a little more useful. Under Terminal Preferences -> Settings -> Window, I have “Active process name”, “Dimensions”, and “Command key” checked. That gives me a window bar something like: Terminal – ssh – 80×24 – CMD4

I don’t care for the block cursor either. It tends to hide the character under the cursor in vi. So, I changed the cursor to a pipe. Under Terminal Preferences -> Settings -> Text, change the Cursor radio button to “| vertical bar”. You can also change it to an underscore.

While you’re in here, check out the different window themes. I’m using Homebrew now. It’s a nice green on black theme with just a little transparency. You can change your default new terminal session theme under Terminal Preferences -> Startup if you find one you like better.

DB2 for Mac install fails

What is it with IBM and products that fail to install?!?

I downloaded the beta of DB2 for Mac (version 9.5.2). The install went fine, until creating the actual instance. At which point it fails miserably. Firstly, it needs xcode installed, so signup for a free Apple Developer account and get that installed. After some digging, I found that the developer at IBM linked a bunch of executables to library files under /Users/rchase and /wsdb. Here is a quick script to resolve this by creating links in the right places:

mkdir /Users/rchase
mkdir /Users/rchase/build
mkdir /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1
mkdir /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/   
mkdir /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib

ln -s /opt/IBM/db2/V9.5/lib64/libdb2locale.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/
ln -s /opt/IBM/db2/V9.5/lib64/libdb2.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/
ln -s /opt/IBM/db2/V9.5/lib64/libdb2dascmn.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/
ln -s /opt/IBM/db2/V9.5/lib64/libdb2osse.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/
ln -s /opt/IBM/db2/V9.5/lib64/libdb2osse_db2.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/
ln -s /opt/IBM/db2/V9.5/lib64/libdb2install.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/
ln -s /opt/IBM/db2/V9.5/lib64/libdb2g11n.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/
ln -s /opt/IBM/db2/V9.5/lib64/libdb2genreg.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/
ln -s /opt/IBM/db2/V9.5/lib64/libdb2trcapi.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/
ln -s /opt/IBM/db2/V9.5/lib64/libdb2dasapi.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/
ln -s /opt/IBM/db2/V9.5/lib64/libdb2dasgcf.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/
ln -s /opt/IBM/db2/V9.5/lib64/libdb2sec.dylib /Users/rchase/build/rchase_db2_v95fp2_darwinport_darwin64_s081205_S1/INST/lib/

mkdir /wsdb
mkdir /wsdb/db2_v95fp2_darwinport
mkdir /wsdb/db2_v95fp2_darwinport/darwin64
mkdir /wsdb/db2_v95fp2_darwinport/darwin64/s081205
mkdir /wsdb/db2_v95fp2_darwinport/darwin64/s081205/INST
mkdir /wsdb/db2_v95fp2_darwinport/darwin64/s081205/INST/lib

ln -s  /opt/IBM/db2/V9.5/lib64/libdb2install.dylib /wsdb/db2_v95fp2_darwinport/darwin64/s081205/INST/lib/
ln -s  /opt/IBM/db2/V9.5/lib64/libdb2genreg.dylib /wsdb/db2_v95fp2_darwinport/darwin64/s081205/INST/lib/
ln -s  /opt/IBM/db2/V9.5/lib64/libdb2osse.dylib /wsdb/db2_v95fp2_darwinport/darwin64/s081205/INST/lib/
ln -s  /opt/IBM/db2/V9.5/lib64/libdb2trcapi.dylib /wsdb/db2_v95fp2_darwinport/darwin64/s081205/INST/lib/

After doing this, the programs don’t complain about missing libraries anymore…. I’m still unable to get db2icrt to create the instance, but I’m much closer than when I started. I would recommend doing the install, but make sure to select the option to not create any instances. Then, create the links to the missing files, and create the instance manually.

IBM LMT Agent Fails To Install

I recently had an AIX system where the IBM License Management Tool failed to install. The failure message I got is:

CODAG076E Writing the agent signature file to the agent installation folder has failed. Check the agent installation folder and ensure that: 
-You have write access to the folder. 
-Files are unlocked. 
-There is sufficient space on the disk.

Followed by:

CODAG011E - The command cannot be executed.

Of course, installp takes care of everything by removing the evidence. So, the usual install logs get wiped, if they were ever created. I started by checking the likely suspects, /var and /tmp. Both had plenty of free space and both had good permissions. I double-checked my response file, it looked good and has worked on other systems.

So, I turned to filesystem free space. All of my filesytems have plenty of free space, as a percentage of size. And, there aren’t any filesystem full messages in the errpt. But, /opt and / both didn’t have much free space in terms of actual bytes. After adding just a little space to both of those, the install went through normally. And, on another system adding some free space to /var resolved the same error.

Finding the HACMP configuration instance #

It’s possible for the HACMP configuration between two different nodes to be out of sync.  Or, you may want to push a config from one node to another.  We had one admin make changes to a down node, then try to sync the cluster.  To clean it up, we had to figure out which node had the latest config. If you want to see which configuration instance number each of your HACMP nodes is using, you can run:

lssrc -l -s topsvcs | grep Instance

or:

odmget HACMPtopsvcs

First chicken tractor completed!

I’ve decided to get into backyard chickens.  A friend from work has gotten WAY into raising his own chickens, and I thought it sounded like a cool way to get slightly more “green” and self-sufficient.  The chickens can eat lots of household kitchen waste, and provide healthy and tasty organic eggs.

I went with plans from Catawba Coops.  The plans cost $20 and you get the plans, a booklet about convincing your neighbors and local government that chickens are good, and a Google SketchUp file of the coop in various stages of completion.

The plans have  a fairly detailed cutlist, but I should have read the whole thing before starting, there are a few options mentioned in the text that is not apparent looking at the parts or cut list.  A table saw would have been nice.  But all you really need is a miter saw, jigsaw (might be able to get away without it), drill, hammer, and a tape measure.

The plans dependence on 2×2 lumber is kind of a problem for me.  No matter how careful you are, you’ll split some boards, and you can only really get 1 screw into most of the joints.  I redid parts of it with 2×4 lumber which didn’t add much weight, but it feels more sturdy.  I still ended up pre-drilling and recessing every screw, a drill with a variable clutch is a real help.  I also made some modifications.  Since I was on a deadline, I made one of the side doors non-removable, and made one of the end doors fence instead of solid lumber.  And, I’ll end up having to clamp the side door that is removable until the lumber dries, as it’s starting to warp a little.

The biggest failing of the plans for me is that the text doesn’t give you a lot of detail, but between the SketchUp file, the plans, and some creative deduction it’s not too difficult.

If I had to do it over again, there are several tweaks I might consider, but all-in-all it’s worked out pretty well so far.  And, it really does look awesome.  It’s something that no neighbor can complain that it doesn’t look good.  Next year after the wood is seasoned, I’ll stain it to match my house or deck.

Catawba Coop chicken tractor

Server SHOW commands

These are various server SHow commands I’ve picked up from various sources. This is definitely not an exhaustive list, and these commands are added and dropped from TSM based on the version, these commands are not supported and may not work for you.

SHow ASQUEUED:
If you have a client session or process stuck, it may be waiting for a drive. You can use this command to see if there are sessions queued waiting for mount points.

SHow ASVOL:
If you are having problems with sessions or processes queued, or waiting for tape volumes, then this command will display the in-memory list of assigned volumes.

SHow CONFIG:
This is one of the long running commands that produce lots of output. It actually issues a set of QUERY and SHOW commands then uses these to build a quite comprehensive picture of the state of the server. This command is worth running as a diagnostic data gathering exercise, to be analysed when free time permits.

SHow CSVars:
Shows client schedule variables and statistics.

SHow DEDUPDeleteinfo:
Shows Deduplication thread information and current worker thread chunk statistics.

SHow DEDUPThread:
Shows information and statistics on Deduplication threads.

SHow DEVCLass:
Is also useful if you have problems with drives. It displays the status of each device class, including the status of allocated drives.

SHow LIBINVentory:
Shows current state of the library inventory for the library specified.

SHOW LIBINVentory [ LibraryName}

SHow LIBRARY:
Displays the current state of the specified library and all of its drives.

SHOW LIBRary [ libraryName ]

SHow LOCK:
Displays lock holders and waiters.

SHow LVM:
Shows logical volume manager info: server disk volumes.

SHow LVMCOPYTABLE:
Shows the LVM database and recovery log copy table status.

SHow LVMVOLS:
Shows database and recovery log volume usage.

SHow MEMTREnd:
This command will report the memory used by the server in megabytes, recorded at hourly intervals for the last 50 hours (this is a constant in the server code and is not user-configurable). The reported usage represents the amount of memory internal server routines request from the pseudo-kernel memory routines. It does NOT represent the total amount of memory the server is using. Nevertheless, it is still useful in determining the server’s memory usage trend.

SHow MP:
Useful for determining which volume is in-use by a given mount point and other attributes for the assigned mount points.

SHow PENDing:
Shows pending administrative and client schedules.

SHow RESQueue:
Displays the resource queue. The resource queue is used to monitor common resources on the server. Typically, this is useful for displaying information about transactions, locks, and other resources used by a storage agent on the database server that it is configured to use.

Show SLOTS:
This command displays the current state of the specified library’s slot information.

SHow SLOTS libraryName

SHow SSPool:
Displays information storage pools.

SHow SSSession:
Shows information about sessions using volumes in storagepools.

SHow TIME:
Shows the current server date and time.

SHow TXN:
Shows Transaction manager statistics.

SHow TXNTable
Displays information about transactions in-use on the server. This information is useful for diagnosing hangs or other transaction related failures while the transaction is still open on the server.

SHow VOLINUSE:
Displays information on a specified volume in the server’s in-use list. This command will display additional information that may be helpful including whether the volume is currently pending removal from the in-use list.

SHow VOLINUSE VolumeName [ REMOVE=YES ]

NPIV N-Port changes w/ AIX

I was at a meeting with other storage admin’s where they talked about never using NPIV with AIX servers because AIX can’t handle it if the N-Port changes due to a N-Port failover in AG mode. I’ve never seen that. In testing our AIX boxes handled the failover without any problems. Part of the reason may be because I’ve enabled Dynamic Tracking and Fast I/O Failover on these fibre adapters. Dynamic Tracking allows for N-Port ID changes, and Fast I/O Failover makes the failure of a Fibre adapter happen faster, which can be good if you are using a multi-path driver. It’s a simple change, but requires either a reboot or bringing down the adapter for the changes. Here’s the command to make the changes in the ODM, which will be applied the next time you reboot:

chdev  -l fscsi1 -a dyntrk='yes' -a fc_err_recov='fast_fail' '-P'

The other, and better, option is to build F-Port trunks to front your AG switch, which preserves the PID in case of a ISL failure, but that requires a trunking license on your switches.

Kimballisms

I work with Tom (gasbag) Kimball, who is probably the most demented person I’ve met. He’s also a lacto-ovo vegetarian, but he’s in fantastic shape. He routinely runs triathlons in the Northern Michigan and occasionally, Florida. Here is a list of workplace sayings that I’ve termed Kimballisms.

Check TSM Client Levels

In order to check what level your clients are, your can run a simple select:

select node_name, -
cast (client_version as char)||'.'||-
cast (client_release as char)||'.'||-
cast (client_level as char)||'.'||-
cast (client_sublevel as char (2))-
CLIENT_LVL from nodes

Sometimes when you upgrade the server, you need a minimum level of client. Like the 6.1 server requires all clients to be at 5.5 or higher. To just see what clients are below a certain level, you can run:

select node_name, -
cast (client_version as char)||'.'||-
cast (client_release as char)||'.'||-
cast (client_level as char)||'.'||-
cast (client_sublevel as char (2))-
CLIENT_LVL from nodes where -
cast (client_version as char)||'.'||-
cast (client_release as char)||'.'||-
cast (client_level as char)||'.'||-
cast (client_sublevel as char (2))-
< '5.5.0.0'