IBM only supports a queue depth of 1 when attaching to XIV with the default algorithm of round_robin. Usually round_robin or load_balance is the best choice, but since IBM is only supporting a queue depth of 1 at this time, there is a performance penalty for asynchronous I/Os. This looks to have been fixed in 5.3.10 (APAR IZ42730) and 6.1 (APAR IZ43146), but is still broken (probably never to be fixed) in earlier releases.

So, IBM recommendation is to split up your storage needs into a number of LUNs matching the number of paths to your XIV, use the fail_over algorithm with a larger queue depth, and assign a different path the highest priority for each LUN. This is kind of a poor man’s load balancing. It’s not that bad, other than having to look at 4 or more hdisks for every LUN, and having to figure out what path to give the highest priority for each one!

IBM doesn’t really see this as a problem, but it’s a huge pain to do correctly in an enterprise.

So, how do we start? First, figure out what hdisk you’re talking about, then run:

lspath -H -l hdiskx -F "status name parent path_id connection"
status  name   parent path_id connection

Enabled hdiskx fscsi0 0       50050763061302fb,4010401200000000
Enabled hdiskx fscsi0 1       50050763060302fb,4010401200000000
Enabled hdiskx fscsi1 2       50050763060802fb,4010401200000000
Enabled hdiskx fscsi1 3       50050763061802fb,4010401200000000

We need the parent device and the connection bit (WWN,LUN#) to specify just a single path. Then run:

lspath -AHE -l hdiskx -p fscsi0 -w "50050763061302fb,4010401200000000"
attribute value              description  user_settable

scsi_id   0x20400            SCSI ID      False
node_name 0x5005076306ffc2fb FC Node Name False
priority  1                  Priority     True

That shows you the priority of this path. You can see it’s still the default of 1. You can check the other paths too.

The goal is to spread out the load between all the available paths. To do this, we must create 4 LUNs. If we need 4GB, we need 4 1GB LUNs. Then we can give each one a different primary path. So, in this example, we should run:

chpath -l hdiskx -p fscsi0 -w 50050763061302fb,4010401200000000 -a priority=1
chpath -l hdiskx -p fscsi0 -w 50050763060302fb,4010401200000000 -a priority=2
chpath -l hdiskx -p fscsi1 -w 50050763060802fb,4010401200000000 -a priority=3
chpath -l hdiskx -p fscsi1 -w 50050763061802fb,4010401200000000 -a priority=4

The first command isn’t really necessary, but I was on a roll. Now, we have to change the algorithm for the hdisk and set the queue depth:

chdev -l  hdiskx -a algorithm=fail_over -a queue_depth=32

Make sure to stager the next one so that path 1 gets a priority of 1, 2 gets 2… and 0 gets a priority of 4. Rinse and repeat until you have 4 LUNs each with a different primary path.

Now wasn’t that easy. Oh, and when you add more disks, be sure to keep them distributed as evenly as possible.

IBM only supports a queue depth of 1 when attaching to XIV with the default algorithm of round_robin. Usually round_robin or load_balance is the best choice, but since IBM is only supporting a queue depth of 1 at this time, there is a performance penalty for asynchronous I\/Os. This looks to have been fixed in 5.3.10 (APAR IZ42730) and 6.1 (APAR IZ43146), but is still broken (probably never to be fixed) in earlier releases.<\/p>\n

So, IBM recommendation is to split up your storage needs into a number of LUNs matching the number of paths to your XIV, use the fail_over algorithm with a larger queue depth, and assign a different path the highest priority for each LUN. This is kind of a poor man’s load balancing. It’s not that bad, other than having to look at 4 or more hdisks for every LUN, and having to figure out what path to give the highest priority for each one! <\/p>\n

IBM doesn’t really see this as a problem, but it’s a huge pain to do correctly in an enterprise. <\/p>\n

So, how do we start? First, figure out what hdisk you’re talking about, then run:<\/p>\n

lspath -H -l hdiskx<\/em> -F \"status name parent path_id connection\"\nstatus  name   parent path_id connection\n\nEnabled hdiskx fscsi0 0       50050763061302fb,4010401200000000\nEnabled hdiskx fscsi0 1       50050763060302fb,4010401200000000\nEnabled hdiskx fscsi1 2       50050763060802fb,4010401200000000\nEnabled hdiskx fscsi1 3       50050763061802fb,4010401200000000<\/code><\/pre>\n
We need the parent device and the connection bit (WWN,LUN#) to specify just a single path.  Then run:<\/p>\n
lspath -AHE -l hdiskx<\/em> -p fscsi0<\/em> -w \"50050763061302fb,4010401200000000<\/em>\"\nattribute value              description  user_settable\n\nscsi_id   0x20400            SCSI ID      False\nnode_name 0x5005076306ffc2fb FC Node Name False\npriority  1                  Priority     True<\/code><\/pre>\n
That shows you the priority of this path.  You can see it’s still the default of 1. You can check the other paths too.<\/p>\n
The goal is to spread out the load between all the available paths. To do this, we must create 4 LUNs.  If we need 4GB, we need 4 1GB LUNs.  Then we can give each one a different primary path. So, in this example, we should run:<\/p>\n
chpath -l hdiskx<\/em> -p fscsi0<\/em> -w 50050763061302fb,4010401200000000<\/em> -a priority=1\nchpath -l hdiskx<\/em> -p fscsi0<\/em> -w 50050763060302fb,4010401200000000<\/em> -a priority=2\nchpath -l hdiskx<\/em> -p fscsi1<\/em> -w 50050763060802fb,4010401200000000<\/em> -a priority=3\nchpath -l hdiskx<\/em> -p fscsi1<\/em> -w 50050763061802fb,4010401200000000<\/em> -a priority=4<\/code><\/pre>\n
The first command isn’t really necessary, but I was on a roll. Now, we have to change the algorithm for the hdisk and set the queue depth:<\/p>\n
chdev -l  hdiskx<\/em> -a algorithm=fail_over -a queue_depth=32<\/code><\/pre>\n
Make sure to stager the next one so that path 1 gets a priority of 1, 2 gets 2… and 0 gets a priority of 4.  Rinse and repeat until you have 4 LUNs each with a different primary path.<\/p>\n
Now wasn’t that easy.  Oh, and when you add more disks, be sure to keep them distributed as evenly as possible.<\/p>\n","protected":false},"excerpt":{"rendered":null,"protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,7,1],"tags":[],"class_list":["post-394","post","type-post","status-publish","format-standard","hentry","category-aix-notes","category-san-notes","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/rootuser.ninja\/index.php\/wp-json\/wp\/v2\/posts\/394","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rootuser.ninja\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rootuser.ninja\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rootuser.ninja\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rootuser.ninja\/index.php\/wp-json\/wp\/v2\/comments?post=394"}],"version-history":[{"count":0,"href":"https:\/\/rootuser.ninja\/index.php\/wp-json\/wp\/v2\/posts\/394\/revisions"}],"wp:attachment":[{"href":"https:\/\/rootuser.ninja\/index.php\/wp-json\/wp\/v2\/media?parent=394"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rootuser.ninja\/index.php\/wp-json\/wp\/v2\/categories?post=394"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rootuser.ninja\/index.php\/wp-json\/wp\/v2\/tags?post=394"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}