Splunk for Statistics using log4j SyslogAppender (Part 1)

Splunk is a great tool that helps organize, manage, and search IT data. Although Splunk's primary marketing focus seems to be on making logs searchable, it is also readily usable to analyze and report on usage since tracking usage is basically the process of summarizing log data.

Configuring Splunk for website visitor tracking is certainly more work than using AWStats or Google Analytics, but if you have non-standard sources of log data, Splunk provides unparalleled flexibility. For example, I recently wanted to aggregate usage statistics from custom log4j log data from an application running on a large number of Tomcat servers. Since I was already using log4j to handle writing the logs to local storage on each server, I simply needed to add a SyslogAppender to my Tomcat log4j.properties, as follows:

log4j.appender.statssyslog=org.apache.log4j.net.SyslogAppender
log4j.appender.statssyslog.SyslogHost=YourSplunkHostIP
log4j.appender.statssyslog.layout=org.apache.log4j.PatternLayout
log4j.appender.statssyslog.layout.ConversionPattern="%d{ISO8601}",%m
log4j.appender.statssyslog.Facility=USER

After configuring Splunk to receive syslog data on UDP port 514, I started to see the data appear in Splunk. Stay tuned for Part 2, which will provide details about configuring Splunk's Summary Indexing and reporting on our gathered statistics.

3 thoughts on “Splunk for Statistics using log4j SyslogAppender (Part 1)

  1. Christina Noren

    Glad to see this post on using Splunk for statistics. In fact Splunk's search language provides a rich syntax for doing all kinds of statistics and with our new 4.0 release there's fairly advanced charting, dashboarding and report building functionality.

    I think you hit an important point on custom log formats being one big reason to use Splunk rather than traditional web stats tools. Splunk's flexibility is absolutely key when dealing with nonstandard data.

    However, in addition to flexibility in terms of getting data in, flexibility of analytics is equally important – awstats etc. is great if I need a count of visitors, top paths, etc. for high level vistor tracking. However, if I need to support a business decision with the answer to a very specific question, canned reports don't cut it.

    We use Splunk for our own splunk.com visitor tracking with standard scheduled statistics delivered to us daily. But we also do a lot of ad hoc statistics and analysis on the same web logs to answer time-critical questions.

    btw, we plan to post a sanitized version of the web stats app we built internally for others to download soon – which should make the setup time to get standard stats a lot faster. Look for updates on our blogs.

  2. admin Post author

    @Christina Thanks for the input! Yes, Splunk provides a far superior experience with custom log formats, non-standard queries of log data (or on-the-fly statistics compilation), etc. I'm looking forward to upgrading to Splunk 4.0 as soon as the free license becomes available.

  3. steve

    i modified remote server log4j.properties with your code & installed splunk on my system.
    but i was unable to fetch the logs from a remote server.

    i installed splunk with syslog-tcp connection.

    what other configuration do i need to change. and which tcp port is suitable for this.

    Kindly revert.. its very urgent

Leave a Reply

Your email address will not be published. Required fields are marked *