How it works

LogicMonitor lets you monitor log files generated by your OS or applications such as MySQL, Tomcat, and so on. For example, you can monitor MySQL slow query log so an alert will be triggered when a slow query is logged in the log file.

The figure below shows the architecture.

Log file monitoring enhances our syslog monitoring facility. (In some cases, you can configure your log files to be sent via Syslog to the LogicMonitor collector, which then negates the need to install the LogWatcher component locally on the sever.)

A LogicMonitor logwatcher is required to be installed on Linux/Windows hosts to monitor log files.  By editing its configuration file you control which log files to monitor, what pattern to look for, and what kind of syslog messages to trigger. The logwatcher checks every line appended to the monitored log files. If the new line matches a configured pattern, a syslog message will be sent to LogicMonitor collector.

Upon receiving the syslog message, the collector reports it to LogicMonitor server that uses our advanced alerting facility so users can get notification via Emails/SMS, generate reports, etc.

How to configure

This section uses a concrete example to show you how to monitor log files.

We have a Linux host dev1.logicmonitor.com that runs a Tomcat web server. We want to monitor Tomcat's log file catalina.out so an alert will be sent every time Tomcat is shutdown for whatever reason (e.g. the JVM crashes or someone kills Tomcat).

Step 1. Create a syslog eventsource

Goto Settings -> Datasources, Eventsources, and Batch jobs -> New -> Eventsource to open the eventsource edit form.

The screenshot below creates an eventsource "Dev1 Tomcat shutdown". Its type is syslog. It has a filter "application equal logwatcher" telling the collector syslog messages for this eventsource will come from logwatcher. Any other type of syslog messages are ignored (by this event source).

Step 2. Install logwatcher on dev1

The logwatcher is a standalone java application (we will release native Linux/Windows version in the future). It runs as a Linux daemon or Windows service.

You can download the Linux logwatcher from here and Windows logwatcher from here.

The screenshot below shows how to install a Linux logwatcher.

The only thing you need to type in is the host on which the collector is running, to where the events are to be reported.

After the installation, logwatcher will be installed under /usr/local/logicmonitor/logwatcher, for Linux and C:\program files(x86)\logicmonitor\logwatcher\ for Windows.

Step 3. Configure logwatcher

For the Linux logwatcher:

Use your favorite text editor to open /usr/local/logicmonitor/logwatcher/config/logwatcher.xml.

For the Windows Logwatcher:

Use your favorite text editor to open /program files(x86)/logicmonitor/logwatcher/config/logwatcher.xml.

Our Tomcat process writes log messages to /usr/local/tomcat/logs/catalina.out. Every time Tomcat is stopped, a log message "INFO: Stopping Coyote HTTP/1.1 on http-443" will be written into catalina.out.

A configuration file to monitor this event in this file would look like this:

What you need to do is to add an <eventsource> element into logwatcher.xml.

<name> = The eventsource name in LogicMonitor.
<file> = The path where the file is located
<fileset> = The path where the file is located in regular expression format.  This support dynamic logfile names (for example if you logfile changes names by date).
<pattern> = A regular expression. If a substring of the new line matches it, a syslog message will be triggered.
<host> = Hostname where the logfile originated from (if for example you have a logfile repository which pulls in logfiles from multiple hosts).
<date> = Regular expression of date format in logfile (this will match any time stamps in the logfile based on the regular expression).

Logwatcher.xml format

logwatcher.xml is wrapped in a <configuration> element.

Within <configuration> element, there could be one <hostname> element and multiple <eventsource> elements.

<hostname>

This tells logwatcher what host name should be used for the syslog message host field.

Why is this element needed? Imagine we install logwatcher on host FOO, but host FOO also monitors host BAR's log files via NFS.  We want logwatcher to tell the collector there is an event happening on host BAR rather than host FOO.

<eventsource>

logwatcher.xml can contain multiple eventsource element, each of which defines an eventsource.

<eventsource> element contains a <name> element and multiple <match> elements.

<name>

The value of <name> element is the eventsource name you define within the LogicMonitor portal.

<match>

A <match> element defines a match that triggers a syslog message.  With a <match> element, you can specify:

• a <file> element - that tells logwatcher which log file to monitor
• a <fileset> element - allows logwatcher to watch a file with a dynamic name (ex a file that changes with the date) using a regular expression
• one or more <pattern> elements - that tells logwatcher what kind of log message will trigger an alert. If a log line matches more than one pattern element, multiple events will be generated.
• one or more <exclude> elements.  If the log line matches any of the exclude subelements, it will be ignored and not trigger an event.
• a <severity> element defines what level alert to trigger when the pattern is matched.

A <eventsource> element can contains multiple <match> elements. For example, you have a server running Tomcat and MySQL. You can define an eventsource "All critical application events" to monitor their log files. So, the following eventsource can be defined:

<eventsource>
     <name>All critical application events<name>
          <match>
          <file>usr/local/tomcat/logs/catalina.out</file>
          <pattern>Stopping .* 443</pattern>
          <severity>Warn</severity>
     </match>
     <match>
          <fileset>/usr/local/tomcat/logs/catalina\d\d\d\d-\d\d-\d\d.log</fileset>
          <pattern>Stopping .*</pattern>
          <severity>Critical</severity>
     </match>
</eventsource>