blog

How To Identify Incoming Data As A Custom Source Type

Wed, 12 Sep 2012 12:59:53 +0100

Overview

Honeycomb Lexicon comes with a large set of knowledge of popular data types and sources.

For data from sources that are unique to your organization or other unrecognized sources, Lexicon still stores and indexes the data, marking it as sourcetype ‘generic’.

This How To article describes the steps to mark this data to any sourcetype you like.

For example, if you have data in log files generated from a custom Web Server Application, the steps below will show you how to tag this data as a ‘webapp’ source type.

Outline

Honeycomb Lexicon has a huge range of options and features to tag, filter, parse and assign incoming data, however, you don’t have to utilize them all to get up and running quickly.

These steps talk about marking incoming data to a given sourcetype. To keep things simple, advanced and optional features aren't covered here, but can be found elsewhere in the knowledgebase and in documentation etc.

What You Need
To perform the steps you will need:

A running installation of Lexicon server
Administrator access to the Lexicon server’s installation
A text editor
Some custom data (either log files or syslog feed)
About 5-10 minutes of time to perform the steps

Procedure

The steps below use an example. Here are the example parameters - please adjust these to suit your environment:

The custom data is in a log file called webapp_XX.log (where ‘XX’ is some number)
The desired sourcetype tag name is ‘webapp’
The desired sourcetype print name is ‘My Web Application’
The installation is located on the server at C:\Honeycomb

To tag incoming data, we need to edit the ‘C:\Honeycomb\lexicon\cfg\inputs.conf’ file:

This file contains configuration for inputs into the local Lexicon server. It also contains lots of useful documentation on options, syntax etc.

1 Create a foldermonitor (or syslogserver) entry

Find the section for configuring folder monitors:
      Search for: ‘File System folder monitors’

You will see a number of entries beginning with ‘foldermonitor’. Here, we will add another entry for the incoming webapp logs.

Add a 'foldermonitor' line that holds the path to where the log files reside. This can be a local path (if, for example, you batch copy log files to the server), or a remote UNC path to another machine.

For example:
            foldermonitor=D:\MyLocalWebappLogFiles\*.log
                                    or
            foldermonitor=\\mywebserver\c$\webserver\logs\*.log
Note:
If you specify a remote UNC path, the Lexicon server must have permissions to read the share on the remote machine. You can do this by setting the user account for the Honeycomb Lexicon service in the Windows Service Control Manager (services.msc) to a user that has the correct permissions.
Note:
If you wish to tag incoming syslog data rather than log files, create a ‘syslogserver’ entry instead of a ‘foldermonitor’ entry. Note that if you use a syslog port that has other incoming data, your ‘match’ (see below) regular expression entry will need to uniquely identify the webapp data. To keep things simple, it can be useful to configure your syslog client to use a port that no other traffic is using - e.g. something other than udp:514, for example: ‘syslogserver=udp:5556’

2 Create a tag entry

Add a 'match' line that describes the tag you wish to use:
tagname:match=<regular expression>

For example:
webapp:match=(.+)

This entry denotes the sourcetype tag name 'webapp', and will match on any incoming data for assigned monitors (see next step).

3 Link the tag entry to the foldermonitor (or syslogserver) entry

Add a 'parser_assign' line that assigns the incoming monitor entry to the tag entry:
parser_assign=<monitor entry>|parser|<tagname>

Note:
It is important that the <monitor entry> value is specified exactly as it is described in the foldermonitor entry. Use Copy and Paste to ensure the text is precisely the same.

For example:
parser_assign=\\mywebserver\c$\webserver\logs\*.log|parser|webapp

4 Restart the Lexicon service

You can restart the service using Windows Service Control Manager (services.msc), or by issuing these commands in a console prompt window:
C:\sc stop lexiconsvc
C:\sc start lexiconsvc

5 Add a print tag to the client(s)

This optional step maps the sourcetype name (‘webapp’) to a display name used in the GUI, Reports etc.
On each client you wish to update:
      Create the folder: ‘C:\Honeycomb\mesh\apps\webapp’
      Create a subfolder: ‘C:\Honeycomb\mesh\apps\webapp\cfg’
      Create a new text file: ‘C:\Honeycomb\mesh\apps\webapp\cfg\printtags.conf’
      Edit the ‘C:\Honeycomb\mesh\apps\webapp\cfg\printtags.conf’ file.
      Add this line to the file and save:
            webapp=My Web Application

This entry will allow GUI tables, charts and reports etc. to replace occurrences of ‘webapp’ with ‘My Web Application, which is a little more user-friendly.
Note:
For some earlier versions of Lexicon, there is a separate and additional entry in order to add the print tag for reports. Here are the steps:
      Create a new text file: ‘C:\Honeycomb\mesh\apps\webapp\cfg\fieldmap.dat’
      Edit the ‘C:\Honeycomb\mesh\apps\webapp\cfg\fieldmap.dat’ file.
      Add this line to the file and save:
            webapp=My Web Application;2;false;0.7;160;LEFT
This entry gives report tables additional information about where to place entries within a report table.

With the above steps completed, lexicon will now identify and tag your matching incoming data as a 'webapp' source type. You can of course, perform these steps for as many types as you need.

Happy indexing!

Palo Alto Next Generation App!

Mon, 16 Apr 2012 16:27:05 +0100

Honeycomb Palo Alto Next Generation App!

Honeycomb's "Next Generation App!" adds enormous value to your Palo Alto Firewall. as with any firewall they are not designed with the intention of storing the information for long periods of time, this is where Honeycomb's Palo Alto LexApp becomes massively useful. Holding vast amount s of generated Log data this including that of Firewall, for given periods of time is what we do BEST!

Honeycomb has established a LexApp which delivers to its users “Palo Alto expert-in-a-box”, providing more wealth to users by adding next generation visibility to YOUR network data along with added correlation between application layer, and low level protocol analysis.
Honeycombs reporting and searching capabilities have extended that of Palo’s in-built tools to make the mining through historical and real-time events seamless. Pre-built with the honeycomb solution are set of

Palo Alto scheduled reports including but not limited to:

Palo Alto Traffic Weekly
Palo Alto Threat weekly
Palo Alto Access weekly

With the options of customizing scheduled reports to a time frame that suits your business.

With Honeycomb Lexicons “no-nonsense” approach to searching allows for quick searches of that extremely rich Palo Alto data at your fingertips. Lexicon Enterprise comes pre-built with a selection of Palo Alto Dashboards:

Palo Alto Traffic
Palo Alto Config
Palo Alto System
Palo Alto Threat

Honeycomb Lexicon Pre-Installation Guidelines

Thu, 15 Mar 2012 18:21:23 +0000

Overview

This post outlines some useful guidelines and best practices prior to installing the Honeycomb Lexicon^® and mesh^® services.

Windows Event Logs

The recommended settings for all in-scope Windows servers’ Event Logs should be set as follows:

Application Log 32768KB Overwrite events as needed

Security Log 196608KB Overwrite events as needed

System Log 32768KB Overwrite events as needed

Other Logs Default settings

Note that no pre-v6 Windows machines (e.g. Server 2003, XP etc.) should have Event Log sizes with a combined total exceeding 300MB. The above settings are optimal for log collection and performance. Whilst changing these settings is a server configuration change, it has no adverse impact on system performance, as long as there is at least 256MB of system disk drive space to house the Event Log database.

Note that for Windows Server 2008/Win7, the 300MB limit is not relevant, as the new Crimson event log mechanism in these Windows products is completely redesigned. The above settings, however, should still be applied to Event Logs on monitored Windows v6+ products (Server 2008, Windows 7) to ensure there is sufficient capacity.

Using Active Directory Group Policy is the most efficient way to set Event Log sizes.

Disk Storage

Collecting data from numerous machines for centralized/distributed storage can involve a significant disk storage requirement.

Here are some points that are worth considering when determining the data collection and disk requirements:

Windows Event Logs
- Monitor the Security Event Log on your Domain Controller(s), and note the number of records and time range stored (using the Event Log sizing below). This will give a rough indication of the amount of data that will be collected from Windows Event Logs across the network (DCs are a good choice, because they tend to be the ‘chattiest’).

Note that Windows 2008/7 generate much more verbose Event Log events than in previous versions. This places additional storage requirements on attached Lexicon^® storage. If you run a 2008+ Active Directory Domain, there are some mechanisms available to help bring down the size of generated events.

File Integrity Monitoring
- It is a worthwhile exercise to target File Integrity Monitoring to include only those files/areas that you really want/need to monitor/capture. It’s easy to say ‘monitor everything’, and the system will faithfully gather all data – however, note this will significantly increase the resources needed to store such data. mesh^® File Integrity Monitor Policies can be tailored to each individual server if needed, to ensure efficient collection and storage.
Network device syslog monitoring
- Network devices such as firewalls, VPNs, switches etc. should have their syslog data feed filtered to emit only the data that is required to be monitored. Network devices, left unfiltered can generate very large amounts of data.

Agentless Remote Event Log Monitoring

Ideally, a separate machine should be used for the Agentless Windows Event Log collection. Any reasonable specification machine will suffice for this purpose.

If Real-Time mode File Integrity Monitoring is targeted for the same machines as Windows Event Log Monitoring, some, most or all Windows Event Log monitoring can be done locally on each server, which would reduce or preclude the need for agentless monitoring. This can reduce the usage of WMI resources on your Windows systems, and also allows for detailed filtering of Wdinows Event Log traffic.

Reporting

If many/large reports are configured/run, it can be a good idea to configure these reports to run on separate hardware. If reports are configured that search across and/or produce very large data sets, this can require significant resources from a machine.

LexReporter can be distributed across multiple machines. Refer tot he LexReporter documentation included with Lexicon^®.

Distributed Options

Honeycomb Lexicon^® has very flexible options for distributing data collection, storage, search and reporting across multiple devices, thus allowing these operations to occur ‘closer’ to where it is generated and/or needed, whilst minimizing hardware requirements.

Data Backup

Lexicon^® includes both and automated and on-demand snapshot capability that can be used for backing up data. Alternatively, a replication server can be setup, with backups taken from the replica.

Network Devices

Devices such as firewalls, web filters, VPNs etc. can easily be configured to send their syslog data directly to Lexicon^®.

It’s worth spending a bit of time configuring these devices to send only the data required, so as not to fill up disk storage with unnecessary events.

Task Checklist

The following is a handy list of task items that can be performed prior to installation, which can prove helpful:

Decide which Windows server Event Logs need to be monitored, noting their physical location
Decide which NTFS volumes need to be monitored
Of the required NTFS volumes to be monitored, decide which data areas should be monitored for File Integrity Monitoring, the type of monitoring (Real-Time, Scanner, Access Monitoring), and which should be excluded (note that monitoring file access opens generates the most amount of traffic)
Decide which, if any, Microsoft Exchange servers should have Message Tracking monitored. Ensure any required servers have Message Tracking enabled

Note: When enabling Message Tracking on Exchange servers for the first time, test the systems thoroughly to ensure the server hardware can handle the extra load placed on it by Message Tracking – particularly at peak times

Decide which network devices are to be monitored
Configure all monitored network devices to only send the data required via syslog
Decide which custom applications/log files need to be monitored, noting their location(s), names etc.
Gather a list of email recipients to be used for reporting/alerting etc.
Decide on disk share location(s) for generated reports
Decide which events/types/searches are needed for alerting, including the type of action to take (email recipients, scripts, programs etc.)
Decide what access control mechanisms are required for your site – e.g. who has access to the mesh^®/Lexicon^® consoles, etc.
Decide how long you wish to hold ‘live’ data within Lexicon^® storage, and provision disk storage accordingly
Provision the required hardware for running Lexicon^® server(s), including installation of the operating system and any required updates/patches/roles/accounts. In an Active Directory environment, the Lexicon^® server should be a member of the Active Directory domain for which it will primarily monitor
Have ready, your organization’s backup strategy to capture automated snapshots to copy to offline/long term storage. It is a good idea to take regular backups of the configured snapshot folder, and to remove live index areas from your backup's configuration (these will have many in-use/locked files, and won't produce a reliable backup - the snapshots folder is used for this purpose)
Make sure you exempt Honeycomb Lexicon^® index folders from anti-virus/anti-malware etc. configurations, as these can have an adverse impact on performance (the folders to exempt will look something like: D:/Honeycomb/lexicon and all subfolders)
Create a dedicated (AD) Admin user account for use by mesh^® services to access remote resources (e.g. for Agentless Event Log monitoring). Typically, this is a Domain Admin account with an extremely strong password
The software installation of mesh^® and Lexicon^®will require a Domain Admin account to perform the installation
For segmented LANs, DMZs etc., ensure any required firewall rules/ports are enabled to allow mesh^®/Lexicon^® traffic to flow (e.g. syslog, WMI, DCOM traffic) mesh^® uses rmi:tcp:8238 by default for its communications. Lexicon^® uses http:8983 by default for search communications. The most common syslog port is udp:514 (but you can and often do configure multiple ports if there are many syslog devices)

Make the Most of Your Palo Alto Firewall Data

Mon, 05 Mar 2012 19:00:17 +0000

Intro

This post walks through the steps to integrate Palo Alto firewall data into Honeycomb Lexicon. Palo Alto next-generation firewalls provide a vast wealth of protection and visibility right through the network stack. Integrating Palo Alto firewall data into Honeycomb Lexicon leverages this data, and allows you to easily visualize data patterns, as well as correlate its data with the rest of your network.

Setup

Note: These instruction pertain to PAN-OS 4.x. If you are running PAN-OS 3.x, these instructions still apply, just some of the names/placements are a slightly different.

Configuring Palo Alto firewalls to forward data feeds into Lexicon is quite straightforward. Here are the steps:

1 Login to your Palo Alto firewall console using an administrative user account

2 Now we create a syslog object that will tell the system where to send data:

Click the Device tab

In the left-side tree, navigate to ‘Server Profiles -> Syslog’

At the bottom of the screen, click ‘Add’

In the ‘Syslog Server Profile’ screen:

Give the Profile object a name

Click ‘Add’

Enter a name for the syslog server (this can be any name, it doesn’t have to be a DNS or related name)

Enter a server name – this does need to be an IP address or DNS name. As this is a firewall that typically resides on the perimeter, it is prudent to use an IP address here

You can keep the default port of 514, or change it to another value. By default, Lexicon listens for incoming syslog traffic on UDP port 514, so if you change it, you will need to ensure a corresponding value is entered into the Lexicon server’s cfg/inputs.conf file

Leave the Facility default value of LOG_USER unchanged

Press OK

You should now see your new Syslog Server Profile entry in the list

3 Now that we have a Syslog Server Profile object, we need to assign it:

In the left-side tree, navigate to ‘Log Settings -> System’

You will see 5 entries in this view, one for each severity level of system messages

For each severity level:

Click on the ‘Severity’ value (e.g. ‘High’)

In the Syslog drop-down, select the newly-created Syslog Server Profile

Press OK

Now select ‘Log Settings -> Config’

In the ‘Log Settings – Config’ box, click on the right-side button to edit the settings

In the Syslog drop-down, select the newly-created Syslog Server Profile

Now select ‘Log Settings -> HIP Match

In the ‘Log Settings – HIP Match’ box, click on the right-side button to edit the settings

In the Syslog drop-down, select the newly-created Syslog Server Profile

To include Palo Alto's Traffic and Threat logs, follow these steps:

In the left-side tree, navigate to 'Objects -> Log Forwarding'

Click 'Add' at the bottom of the screen

Give the Log Forwarding Profile a name

For each Traffic and Threat Settings entry you wish to include, click on the relevant 'Syslog' column entry, and add the newly-created Syslog Server Profile

Press OK

4 The objects are now created and assigned, so all that’s left to do is to commit the changes:

Press the ‘Commit’ button in the top right

Click OK to continue

After a few moments, the changes will be applied, and you will begin to receive data from your Palo Alto firewall.

Gathering VMWare Host Data

Mon, 05 Mar 2012 18:58:46 +0000

Intro

Today's post talks about the configuration and setup for sending and receiving VMWare ESX host logs and system events into Honeycomb Lexicon.

As VMWare host systems become ever more pervasive across business network estates and data centres, it becomes increasingly essential to ensure the host is running smoothly. Awareness of problems and system issues is crucial for these mission-critical systems.

Honeycomb Lexicon includes a VMWare LexApp application that contains the Knowledge Sets, Dashboards, searches, reports etc. to monitor the health of your ESX servers. All that is required is to get VMWare servers forwarding data into Lexicon.

Important Lexicon Setup Information

VMWare ESX syslog messages use a standard Linux-style format:

<pri>month day time service: message

This format contains no provision to uniquely identify the message as coming from an ESX server (as opposed to, say, a generic Linux log file which uses the same format). As such, Lexicon (or any syslog server for that matter) has no intrinsic way of determing the event source type as VMWare.

The good news is that Lexicon has a number of configuration options that do allow proper sourcetype detection for generic/standardized message formats. These options are contained within the Lexicon server LexApp's sourcetypes.conf file installed on the Lexicon server.

To correctly identify VMWare syslog messages as a VMWare type, follow these simple setps:

1 On the Lexicon server that will receive VMWare events, open this file in a text editor:

<Honeycomb install folder>/lexicon/apps/vmware/cfg/sourcetypes.conf

2 Find the line that starts like this:

vmware:syslogvalidators= (typically about halfway down the file)

For the VMWare LexApp, the default value for this property is:

*:udp:1515 (this means all incoming traffic on udp port 1515)

3 Edit this value to match the VMWare ESX server(s) syslog configuration. Multiple servers are entered as a comma-delimited list. For example:

vmware:syslogvalidators=192.168.15.12:udp:514,10.0.1.14:udp:515,10.0.1.19:udp:515

It's important to be careful about using the wildcard (*) value for generic/standard syslog messages, as this means ALL incoming traffic on the given port will be interpreted as VMWare data. This is why the LexApp's default settings are not set to the standard 514 port - otherwise many other incoming events would be mis-categorized.

Also, as a general rule, using the IP address here tends to work better than a DNS name, as often the ESX servers don't have access to the same DNS resolution as the Lexicon server(s) receiving the syslog messages.

4 Once you've made the changes to reflect your VMWare configuration, save the file, restart the Lexicon service, and your system is ready to go!

Additional Note: There is also a corresponding option in sourcetypes.conf for file validation, in case you are importing log files directly into Lexicon. The VMWare LexApp sourcetypes.conf file has an example of how to use this option.

VMWare Syslog Setup

This section steps through the procedure for enabling syslog messages to be sent by your ESX server(s). ESX/ESXi hosts run a syslogd service that can forward messages from VMKernel and other ESX system components.

Here are the steps:

1 Modify the /etc/syslog.conf file to capture and forward events:

Open the /etc/syslog.conf file using a text editor

Add the following line to the end of the file:

*.*   @<IPAddress of your Lexicon server>

For example:

*.*   @192.168.15.10

Important: Note the white space between *.* and the @ symbol is a TAB, not spaces.

Save the file and exit your text editor

2 Restart the syslogd service with this command:

service syslog restart

3 It’s important now to check the ESX firewall is configured to allow outbound syslog traffic, so run this command to check the firewall’s configuration:

esxcfg-firewall -q|grep syslog

If the firewall is configured to allow syslog, you should see output similar to this:

syslog              : port 514 udp.out

If you don’t see any output, outbound syslog traffic is blocked, and needs to be enabled, so run this command to enable outbound syslog traffic:

esxcfg-firewall -o 514,udp,out,syslog && esxcfg-firewall –l

You can run the firewall check command above again to ensure the firewall is configured correctly.

Your ESX host is now configured to send its logs and system messages to your Lexicon server.

Honeycomb Lexicon, by default, monitors incoming traffic on UDP port 514, so VMWare data will be automatically indexed as it arrives from your ESX hosts.

You can check incoming VMWare traffic in Lexicon by Launching the Lexicon Browser (meshreporter.exe), and loading the VMWare Dashboard.