UPDATED

Last year I budgeted for 3 P740C model’s to replace 3 P6 550 models that were getting long in the tooth. Because of the long lead time in our budget process and the continued downward pressure from IBM on their pricing, I was able to purchase 4 P7+ 740D models.  That is a big win for us.

After implementing new 7042-CR7 model HMCs (which I recommend everyone upgrade to) and powering on our first box, I noticed that the latest HMC code reports the server has a Power7 and not a Power7+.  The Power7 chip has been out for nearly a year, and the HMC has been through several updates since then, so why does it not show Power7+ the way it did for Power6+?  Here’s what the screen looks like:

So, what does the LPAR say when it’s powered on?  Everywhere I look, it’s Power7.  Here’s what the system thinks the CPU is:

nim # lsattr -El proc0
frequency   4228000000     Processor Speed       False
smt_enabled true           Processor SMT enabled False
smt_threads 4              Processor SMT threads False
state       enable         Processor state       False
type        PowerPC_POWER7 Processor type        False

And prtconf:

nim # prtconf 
System Model: IBM,8205-E6D
Machine Serial Number: 
Processor Type: PowerPC_POWER7
Processor Implementation Mode: POWER 7
Processor Version: PV_7_Compat

I do have a Power7 server running in Power6+ compatibility mode, here’s the output of prtconf on that server:

# prtconf

System Model: IBM,8202-E4B
Machine Serial Number: 10418BP
Processor Type: PowerPC_POWER7
Processor Implementation Mode: POWER 6
Processor Version: PV_6_Compat

So, maybe the OS commands aren’t aware of the CPU compatibility mode.  This is the latest firmware and the latest AIX 7.1 level.  I’m also running the latest HMC code, and I’ve confirmed the same behavior in the latest VIOS level (2.2.2.2).

Of course, the question was asked, did we really get what we paid for? So, I called my IBM Business Partner and asked their Technical sales team to dig into this.  The box does have Power7+ processors, so it’s wasn’t mis-ordered and it WAS built correctly in the factory. They reached out to some other customers running a new P7+ 770, and they’ve confirmed the same behavior there, so I assume this is the same across the product line.

Then I had a bit of luck. As part of this upgrade, I’m testing AME on our non-production servers.  The amepat tool, shows the correct processor mode:

nim # amepat

Command Invoked                : amepat

Date/Time of invocation        : Fri Sep 27 11:53:38 EDT 2013
Total Monitored time           : NA
Total Samples Collected        : NA

System Configuration:
---------------------
Partition Name                 : nim
Processor Implementation Mode  : POWER7+ Mode
Number Of Logical CPUs         : 4
Processor Entitled Capacity    : 0.10
Processor Max. Capacity        : 1.00
True Memory                    : 4.00 GB
SMT Threads                    : 4
Shared Processor Mode          : Enabled-Uncapped
Active Memory Sharing          : Disabled
Active Memory Expansion        : Enabled
Target Expanded Memory Size    : 8.00 GB
Target Memory Expansion factor : 2.00

There we see the expected Power7+ mode.  This command works and reports the processor correctly on systems without AME enabled, so it can be used on any LPAR to show the correct processor type for Power7+ systems.  Here is the output on our Power7 LPAR running in Power6+ mode:

# amepat

Command Invoked : amepat

Date/Time of invocation : Wed Oct 2 12:41:43 EDT 2013
Total Monitored time : NA
Total Samples Collected : NA

System Configuration:
---------------------
Partition Name : tsm1
Processor Implementation Mode : POWER6

So, amepat doesn’t report Power6+ for Power7 systems running in Power6+ mode.

Our IBM client team is looking into this issue, and I expect the relevant commands will be enhanced in a future service pack and HMC level.  But, in the mean time, we can prove that what we ordered is what was delivered.

UPDATE :

IBM’s answer:

Historically IBM has not included the “+” on any of our products (ie Power 5+, Power6 or Power7+).  You can open a PMR and request a Design Change Request (DCR) to have the “+” added for Power7 servers.

That is an interesting answer to me.  We never purchased any Power6+ servers, so I can’t comment on what the OS commands, lsattr and the like, may or may not report. But, the HMC most definitely did report a separate compatibility mode for Power6+. My only thought is that the Power7+ CPU didn’t introduce a new operational mode, which is a little surprising to me because of some of the work done in this chip.

I recently put up a test Splunk server to act as a central syslog repository, one of the issues in our security audits. There are some “open” projects to do this, but Splunk has a lot of features and is “pretty” compared to some of the open alternatives. Getting data from our Linux hosts was a snap, but data from our AIX hosts has a few minor annoyances. Fortunately, we were able to overcome them.

The syslogd shipped with AIX only supports UDP. rsyslog supports TCP, but hasn’t been ported to AIX. Another option is syslog-ng, for which there are open source and commercial versions compiled for AIX. But, after installing all the dependent RPMs for the open source version, it would only segfault with no indication of the problem. So, to support syslog via UDP, on the Splunk server you have to enable a UDP source. That’s easily accomplished by going to Manager -> Data Inputs -> UDP -> New, enter 514 for the port, set sourcetype to “From List”, and source type of “syslog”. Check “More settings” and select DNS for “Set host” and click Save.

Once that is done, add a line to /etc/syslog.conf on the source node to send the data you want Splunk to record to the Splunk server. If your splunk server is named “splunk” it would look something like this:

*.info        @splunk

One of the problems with AIX’s implementation of syslog is it’s format. Here’s what Splunk records:

3/26/13 12:32:07.000 PM	Mar 26 12:32:07 HOSTNAME Mar 26 12:32:07 Message forwarded from HOSTNAME: sshd[21168310]: Accepted publickey for root from xxx.xxx.xxx.xxx port 39508 ssh2 host=HOSTNAME   Options|  sourcetype=syslog   Options|  source=udp:514   Options|  process=HOSTNAME

The AIX implementation of syslog by default adds “Message forwarded from HOSTNAME:”. That’s a little annoying to look at, but worse is that Splunk uses the hostname of the source as the process name, so you lose the ability to search on the process field. You can turn this off on the source with:

stopsrc -s syslogd
chssys -s syslogd -a "-n"
startsrc -s syslogd

As usual, I’m late to the party. I was at the Power Systems Technical University in San Antonio several years ago (an awesome venue), and there was a session on the new AIXPert feature of AIX 6.1 (later back-ported to 5.3). At the time I though it was clunky and wasn’t too excited about it.

It’s really just a bunch of pre-packaged shell scripts, that are defined in an XML file you really need to manage manually (the horror). You run the master aixpert command and specify what XML file you want. You can go with AIX Defaults, Low, Medium, High, and SOX (if you’re into that kind of thing). When you run aixpert, it applies whatever settings are associated with the level you selected. Here is where I usually yawn. It’s just not that exciting. Everyone pretty much does that with a script or text file or some corporate documentation (probably written by someone without a clue, E&Y I’m looking at you). Yes, there are a couple of ways to get to a GUI, but it’s really more manageable for me with the commandline.

The bit that gets tedious is that no one definition is going to fit any one shop. So you have to export the standard definitions to a custom XML file and open it up and hack it by hand. Systems like XML, I don’t really are to read it all day long. But, it’s not that difficult, here’s an example:

<AIXPertEntry name="cust_udp_recvspace" function="udp_recvspace">
    <AIXPertRuleType type="LLS"/>
    <AIXPertDescription>Network option udp_recvspace: Set network option udp_recvspace's value to 655360</AIXPertDescription>
    <AIXPertPrereqList>bos.rte.security,bos.rte.shell,bos.rte.ILS,bos.net.tcp.client,bos.rte.commands,bos.rte.date</AIXPertPrereqList>
    <AIXPertCommand>/etc/security/aixpert/bin/ntwkopts</AIXPertCommand>
    <AIXPertArgs>udp_recvspace=655360 s cust_udp_recvspace</AIXPertArgs>
    <AIXPertGroup>Tune network options</AIXPertGroup>
 </AIXPertEntry>

Nothing too fancy there. It’s mostly fluff, the PrereqList, Command, and Args options are the important ones, the rest of more for the user than anything else. When you apply that XML, the system runs the shell script which sets the appropriate network option. It’s all fairly simple.

The cool part is that most of the things set with aixpert can also be checked with aixpert. When you apply an XML file, aixpert saves the rules that applied to /etc/security/aixpert/core/appliedaixpert.xml. If I modify that setting and run “aixpert -c”, aixpert parses the appliedaixpert.xml file and checks things out. This is what I get:

# aixpert -c
do_action(): rule(cust_udp_recvspace_CA6BE6C2) : failed.
Processedrules=66       Passedrules=65  Failedrules=1   Level=AllRules
        Input file=/etc/security/aixpert/core/appliedaixpert.xml

To set the world right again, you just re-apply your XML file. I found a minor issue here. I’ve had to remove the /etc/security/aixpert/core/appliedaixpert.xml file before setting a new one. You can get the same rule in there repeatedly, why IBM doesn’t offer a commandline switch to do that I don’t know.

Another cool thing, you can Undo the changes applied by the built-in aixpert rules. When aixpert applies a setting, it writes an undo rule to /etc/security/aixpert/core/undo.xml. Then, running “aixpert -u” will undo what you’ve already done. I would probably purge that once in a while too so that you can recover to a known good state.

So, just wrap a dirt-simple cron script around it to notify when something goes wrong… Something like this:

#!/usr/bin/perl

$EMAIL_ADDRESS = "user\@domain.net";
@TEMP = `hostname`;
$HOSTNAME = $TEMP[0];
chomp($HOSTNAME);
$MAIL_FROM_ADDR="aixpert@$HOSTNAME";

@OUTPUT = `/usr/sbin/aixpert -c 2>&1`;
@REPORT = grep(/^Processedrules.*/, @OUTPUT);

$REPORT[0] =~ s/\s+/=/g;
($null, $PROCESSED ,$null, $PASSED, $null, $FAILED, $null, $LEVEL) = split(/=/, $REPORT[0]);

if ( $FAILED > 0 ) {
        open (MAIL, "| /usr/sbin/sendmail -t ");
        select (MAIL);

        print "Mime-Version: 1.0\n";
        print "Content-type: text/html; charset=\"iso-8859-1\"\n";
        print "From: $mail_from_addr\n";
        print "To: $EMAIL_ADDRESS <$EMAIL_ADDRESS>\n";
        print "Subject: Aixpert $HOSTNAME: $FAILED test FAILED\n";
        print "<html><head></head><body>\n";
        print "<pre>\n";

        print "@OUTPUT";

        print MAIL "</pre>\n";        print MAIL "</body></html>\n";
        $~ = "STDOUT";
        close (MAIL);

        exit 1;
}
exit 0;

That should keep the auditors happy. There are enough basic security settings in the default XML files that with a little tweaking you can hit all or very nearly all of your security audit queries.

That’s all well and good, as far as that goes, but what really made me like aixpert is that it gives you a very simple framework to apply your own settings, and make sure those settings are correct. If you distribute an XML file and a few scripts around your enterprise, you can ensure that those settings are standardized across hosts too.

Here’s a simple script to make sure the attributes of your vscsi devices are correct:

#!/bin/perl

@ADAPTERS = `lsdev -c adapter -Sa | grep -E "^vscsi" | awk '{ print \$1 }'`;
$REPORT = $ENV{'AIXPERT_CHECK_REPORT'};

%ATTRIBUTES = ("vscsi_err_recov", "fast_fail",
                "vscsi_path_to", 30 );

if ( $REPORT == 1 ) {
        for (@ADAPTERS) {
                chomp($_);
                $ADAPT = $_;
                @TEMP = `lsattr -El $ADAPT | awk '{ print \$1, \$2}'`;
                for (@TEMP) {
                        ($ATTR, $VALUE) = split(/\s+/, $_);

                        if ( $VALUE != $ATTRIBUTES{$ATTR} && $ATTRIBUTES{$ATTR} ) {
                                print "$ADAPT attribute $ATTR is $VALUE, should be $ATTRIBUTES{$ATTR}\n";
                                $FAIL++;
                        }
                }
        }
        if ( $FAIL ) {
                exit 1;
        }
}else {
        for (@ADAPTERS) {
                chomp($_);
                $ADAPT = $_;
                @TEMP = `lsattr -El $ADAPT | awk '{ print \$1, \$2}'`;
                for (@TEMP) {
                        ($ATTR, $VALUE) = split(/\s+/, $_);

                        if ( $VALUE != $ATTRIBUTES{$ATTR} && $ATTRIBUTES{$ATTR} ) {
                                system("/usr/sbin/chdev -l $ADAPT -a $ATTR=$ATTRIBUTES{$ATTR} 2>&1 > /dev/null");
                        }
                }
        }
}

Using a script like this as a template, you can check and correct the value of any number of system attributes. Adding it to aixpert is a breeze, just add a stanza to your XML file:

<AIXPertEntry name="cust_vscsi_config" function="vscsi_config">
    <AIXPertRuleType type="MLS"/>
    <AIXPertDescription>Resets attributes of the vscsi devices</AIXPertDescription>
    <AIXPertPrereqList></AIXPertPrereqList>
    <AIXPertCommand>/etc/security/aixpert/custom/vscsi_config.pl</AIXPertCommand>
    <AIXPertArgs>GENERIC</AIXPertArgs>
    <AIXPertGroup>Custom Rules</AIXPertGroup>
</AIXPertEntry>

Once you decide to get into it, aixpert is a pretty nice little tool. There’s a great little movie created by Nigel Griffiths at the DeveloperWorks website to get you started too!

But, I still don’t care for System Director. 🙂