I recently put up a test Splunk server to act as a central syslog repository, one of the issues in our security audits. There are some “open” projects to do this, but Splunk has a lot of features and is “pretty” compared to some of the open alternatives. Getting data from our Linux hosts was a snap, but data from our AIX hosts has a few minor annoyances. Fortunately, we were able to overcome them.
The syslogd shipped with AIX only supports UDP. rsyslog supports TCP, but hasn’t been ported to AIX. Another option is syslog-ng, for which there are open source and commercial versions compiled for AIX. But, after installing all the dependent RPMs for the open source version, it would only segfault with no indication of the problem. So, to support syslog via UDP, on the Splunk server you have to enable a UDP source. That’s easily accomplished by going to Manager -> Data Inputs -> UDP -> New, enter 514 for the port, set sourcetype to “From List”, and source type of “syslog”. Check “More settings” and select DNS for “Set host” and click Save.
Once that is done, add a line to /etc/syslog.conf on the source node to send the data you want Splunk to record to the Splunk server. If your splunk server is named “splunk” it would look something like this:
*.info @splunk
One of the problems with AIX’s implementation of syslog is it’s format. Here’s what Splunk records:
3/26/13 12:32:07.000 PM Mar 26 12:32:07 HOSTNAME Mar 26 12:32:07 Message forwarded from HOSTNAME: sshd[21168310]: Accepted publickey for root from xxx.xxx.xxx.xxx port 39508 ssh2 host=HOSTNAME Options| sourcetype=syslog Options| source=udp:514 Options| process=HOSTNAME
The AIX implementation of syslog by default adds “Message forwarded from HOSTNAME:”. That’s a little annoying to look at, but worse is that Splunk uses the hostname of the source as the process name, so you lose the ability to search on the process field. You can turn this off on the source with:
stopsrc -s syslogd chssys -s syslogd -a "-n" startsrc -s syslogd