Home Patrick

Installing TSM Client On Ubuntu

IBM distributes the TSM clients in RPM format, but not in deb format for Debian based Linux distributions. The re are some TSM clients in deb format out there, but the latest I could find is version 5.4. I found that alien converts the RPM files decently enough.

First, install alien:

sudo apt-get install alien

Then, get all the RPMs you want to install in the same directory and run:


alien -k *.rpm


There is a problem with the install scripts that they didn't create a link on my systems:


ln -s /opt/tivoli/tsm/client/lang/EN_US/dsmclientV3.cat /opt/tivoli/tsm/client/ba/bin/EN_US/dsmclientV3.cat


After that, the client should work just as if installed on a Redhat based system.

Removing stuborn VGs

I had a junior admin delete LUNs from the SAN before removing the volume group associated with it. The volume group was still in the ODM, but the PVs were gone and not coming back. Exporting the VG didn’t work:

# exportvg datavg
0516-772 exportvg: Unable to export volume group datavg

The VG is in the ODM:

 # odmget -q name=datavg CuDv

CuDv:
        name = "datavg"
...

The way to remove it from the ODM is:

 # odmdelete -o CuDv -q name=datavg

AIX – Remove failed MPIO paths

Here is a quick and dirty script to remove failed MPIO paths. You can end up with failed paths if you make some SAN connection changes.

for disk in `lsdev -Cc disk | grep 2107 | awk '{ print $1 }'`
do
        for path in `lspath -l $disk -F "status connection" | grep Failed | awk '{ print $2 }'`
        do
                echo $disk
                rmpath -l $disk -w $path -d
        done
done

secldapclntd Failure w/ HACMP

After upgrading to AIX 5.3 TL9 SP4, we found that secldapclntd will go into a death loop during a HACMP failover. It consumes more and more CPU until the system doesn’t have any capacity left, and stops the HACMP failover. Killing secldapclntd will let HACMP continue.

We didn’t see this behavior w/ AIX 5.3 TL8 SP3. IBM has identified a couple of issues that are probably coming together to cause our problem, but they won’t be fixed in TL9… ever. IBM’s work-around is to setup a pre and post-event script to stop secldapclntd before the IP takeover (and release) and restart it afterward. In testing, this works pretty well, and it only takes a few seconds to stop and start secldapclntd.

Here’s the workaround by IBM:

  1. Create a script “/usr/local/cluster/start-ldap.sh” and copy it to every node in the cluster

    2. #!/usr/bin/ksh
    echo “STOP LDAP CLIENT Daemon”
    /usr/sbin/start-secldapclntd
    exit 0

  2. Create a script “/usr/local/cluster/stop-ldap.sh” and copy it to every node in the cluster

    3. #!/usr/bin/ksh
    echo “STOP LDAP CLIENT Daemon”
    /usr/sbin/stop-secldapclntd
    exit 0

  3. Create a pre-event

    4. smitty hacmp
    Extended Configuration
    Extended Event Configuration
    Configure Pre/Post-Event Commands
    Add a Custom Cluster Event

    Type or select values in entry fields.
    Press Enter AFTER making all desired changes.
    [Entry Fields]

    * Cluster Event Name [] << I use the name "ldapClientStart" * Cluster Event Description [] << Start ldap client daemon * Cluster Event Script Filename [] << /usr/local/cluster/start-ldap.sh

  4. Create a post-event

    5. smitty hacmp
    Extended Configuration
    Extended Event Configuration
    Configure Pre/Post-Event Commands
    Add a Custom Cluster Event

    Type or select values in entry fields.
    Press Enter AFTER making all desired changes.
    [Entry Fields]

    * Cluster Event Name [] << I use the name "ldapClientStop" * Cluster Event Description [] << Stop ldap client daemon * Cluster Event Script Filename [] << /usr/local/cluster/stop-ldap.sh

  5. Update the acquire_service_addr event

    6. smitty hacmp
    Extended Configuration
    Extended Event Configuration
    Change/Show Pre-Defined HACMP Events
    (select –> acquire_service_addr )
    Change/Show Cluster Events

    Type or select values in entry fields.
    Press Enter AFTER making all desired changes.
    [Entry Fields]

    Event Name acquire_service_addr
    Pre-event Command [ldapClientStop] <<< the defined name "ldapClientStop" Post-event Command [ldapClientStart] <<< the defined name "ldapClientStart"

  6. Update the release_service_addr event

    7. smitty hacmp
    Extended Configuration
    Extended Event Configuration
    Change/Show Pre-Defined HACMP Events
    (select –> release_service_addr )
    Change/Show Cluster Events

    Type or select values in entry fields.
    Press Enter AFTER making all desired changes.
    [Entry Fields]

    Event Name release_service_addr
    Pre-event Command [ldapClientStop] <<< the defined name "ldapClientStop" Post-event Command [ldapClientStart] <<< the defined name "ldapClientStart"

  7. Synchronize the cluster configuration, and you’re done

Load balance algorithm w/ AIX and XIV

IBM only supports a queue depth of 1 when attaching to XIV with the default algorithm of round_robin. Usually round_robin or load_balance is the best choice, but since IBM is only supporting a queue depth of 1 at this time, there is a performance penalty for asynchronous I/Os. This looks to have been fixed in 5.3.10 (APAR IZ42730) and 6.1 (APAR IZ43146), but is still broken (probably never to be fixed) in earlier releases.

So, IBM recommendation is to split up your storage needs into a number of LUNs matching the number of paths to your XIV, use the fail_over algorithm with a larger queue depth, and assign a different path the highest priority for each LUN. This is kind of a poor man’s load balancing. It’s not that bad, other than having to look at 4 or more hdisks for every LUN, and having to figure out what path to give the highest priority for each one!

IBM doesn’t really see this as a problem, but it’s a huge pain to do correctly in an enterprise.

So, how do we start? First, figure out what hdisk you’re talking about, then run:

lspath -H -l hdiskx -F "status name parent path_id connection"
status  name   parent path_id connection

Enabled hdiskx fscsi0 0       50050763061302fb,4010401200000000
Enabled hdiskx fscsi0 1       50050763060302fb,4010401200000000
Enabled hdiskx fscsi1 2       50050763060802fb,4010401200000000
Enabled hdiskx fscsi1 3       50050763061802fb,4010401200000000

We need the parent device and the connection bit (WWN,LUN#) to specify just a single path. Then run:

lspath -AHE -l hdiskx -p fscsi0 -w "50050763061302fb,4010401200000000"
attribute value              description  user_settable

scsi_id   0x20400            SCSI ID      False
node_name 0x5005076306ffc2fb FC Node Name False
priority  1                  Priority     True

That shows you the priority of this path. You can see it’s still the default of 1. You can check the other paths too.

The goal is to spread out the load between all the available paths. To do this, we must create 4 LUNs. If we need 4GB, we need 4 1GB LUNs. Then we can give each one a different primary path. So, in this example, we should run:

chpath -l hdiskx -p fscsi0 -w 50050763061302fb,4010401200000000 -a priority=1
chpath -l hdiskx -p fscsi0 -w 50050763060302fb,4010401200000000 -a priority=2
chpath -l hdiskx -p fscsi1 -w 50050763060802fb,4010401200000000 -a priority=3
chpath -l hdiskx -p fscsi1 -w 50050763061802fb,4010401200000000 -a priority=4

The first command isn’t really necessary, but I was on a roll. Now, we have to change the algorithm for the hdisk and set the queue depth:

chdev -l  hdiskx -a algorithm=fail_over -a queue_depth=32

Make sure to stager the next one so that path 1 gets a priority of 1, 2 gets 2… and 0 gets a priority of 4. Rinse and repeat until you have 4 LUNs each with a different primary path.

Now wasn’t that easy. Oh, and when you add more disks, be sure to keep them distributed as evenly as possible.

IBM XIV Thin Provisioning

Thin provisioning on an IBM XIV is pretty hot, but there are some gotchas. Thin provisioning lets you actually allocate more space in LUNs to your hosts than you have in physical storage. So, if you have a lot of filesystems or volume groups that have a lot of free space, that’s cool. Where on other storage systems you would burn the whole space allocated by the LUNs, you’re only allocating (physically) as much as you’re really using. It’s easy to burn yourself, so you have to monitor your free space in the XIV “Storage Pools”. When a Storage Pool fills up, the volumes go to Read-Only mode until you resize the pool.

Now, here’s a catch. You can define the LUN to any size, but when the first I/O hits, the system allocates 17GB to the LUN regardless of size. So, you define a 8GB LUN (for giggles) on a thinly provisioned Storage Pool. When the first I/O hits (like you actually turned on the host, or ran cfgmgr, or did a scan for new hardware) 17GB will be reserved out of the Storage Pool. And, it burns this free space in 17GB increments. 18GB burns 34GB free space, ect…

Now, the system tries to keep you from doing stupid things like this. If you specify the size of the LUN in GB, it will automagically round up to the nearest 17GB chunk. But, if you specify in 512 byte blocks (because we all think in 512 byte blocks, right?), the LUN will appear to the host as the exact size specified. And, it still burns the 17GB free space.

So, at a minimum the actual physical space you need for a thinly provisioned Storage Pool is:
17GB X [# of LUNs]

Determining how many BB credits you need

At 4Gb, a full packet is about 1 km long, and at 2Gb a full packet is about 2km long!  Yes, at any given time 2k of your data is spread from the port to 1km down the cable (as the light travels).  Each packet burns 1 buffer, no matter the size of the packet.  The buffer credit isn’t released until the packet gets to the receiving switch and the sending switch receives an acknowledgment.  So, at 4Gb, you need 2 buffer credit for each km of distance.  1 to fill the 1km pipe to the receiving switch, and 1 waiting for the acknowledgment.

So, for a 10 km fibre path, you need 20 buffer credits, assuming that every packet is a full packet.  Even if every data packet is full, there is some inter-switch communication that doesn’t fill a packet.  That means that the packets are shorter and take less time to transmit, meaning there are more packets in the pipe.  So, the 2 packets per km number is the MINIMUM number you’ll need. And, we’re not counting the time it takes to process the frames on the switches, which is probably pretty minimal.

Also, be aware that on a 48k, there are 1024 buffers per ASIC, out of which 8 buffers per port are reserved. So, if you put a bunch of long distance ISLs all on one ASIC, you might run in into problems because there aren’t enough buffers to go around. If you have a GoldenEye based switch, you only get 700 buffer credits per ASIC, out of which 8 buffers per port are reserved. Use portbuffershow to see your buffer usage per ASIC.

Example config.cerner file

Here is a sample config.cerner file for Cerner 2007 and newer with WebSphere 6. Their documentation is given in and image PDF, so you can’t copy the text out and typing this by hand is prone to errors. Replace DOMAIN, USER, and PASSWORD with the correct info.

# config.cerner
#
# last update: 1-23-2007
#

System {
	Enterprise {
		PrimaryDomain = "DOMAIN"
		Domain {
			cert {
				Principal {
					Authority {
						User = "USER"
						Password = "PASSWORD"
					}
				}
			}
		}
	}
	Coding {
		System {
			Instance {
				DOMAIN {
					Source = "EISService"
				}
			}
		}
	}
	Preferences {
		System {
			Instance {
				DOMAIN {
					Source = "Millennium"
				}
			}
		}
	}
	Application {
		MillenniumMobile {
			Authentication {
				ContainerHook {
					Host = "localhost"
					Context = "MillenniumMobile"
					Protocol = "http"
					Port = "9080"
				}
			}
		}
	}
}