Netsweeper Log Parser

Overview: Netsweeper Log Parser

This document contains an overview to syslog, syslogparser and nslogparserctl providing flexible log file parsing similar to radius account log style of parsing. This service may need to be enabled in Managed Service in WebAdmin Settings to be viewable in the Services page.

Permissions for NS Log Parser

The permission for configuring this service is the same permission used for the Radius service.

Syslog Parser Overview

The Syslog Parser can parse syslog messages for Client sign-on and sign-off from the Network. This can be used to parse DHCP Option 82 or other log messages that include the subscriber User-Name, IP-Address and other information. This service can made viewable or hidden in the WebAdmin by selecting it in 'Managed Services' in 'WebAdmin Settings'.

nslogparserctl Service Overview

The nslogparserctl script will start and stop the syslog parser which can be used for flexible log file parsing.

Usage: /etc/init.d/nslogparserctl

Argument	Description
nslogparserctl status	Checks status
nslogparserctl start	Starts the service
nslogparserctl stop	Stops the service
nslogparserctl restart	Restarts the service

Log Parser Example 1

The following can be used to generate a generic entry into the log file, the description of this particular example, issued by a DHCP server, would be:

logger -t dhcp "IP address 162.217.215.10 MAC 78:a2:a0:e0:cc:0c is at 000525673 eth3:4"

logger -t dhcp "IP address {IP_ADDRESS} MAC {HARDWARE MAC} is at {USER-NAME} eth3:4"

Log Parser Example 2

The following can be used to change how we parse a generic log file line.

$SYSLOGPARSER_REGEXP = "/IP address ([^ ]*) MAC ([^ ]*) is at ([^ ]*)/";

$SYSLOGPARSER_REGEXP_MATCH = array(

'Acct-Status-Type' => 'Start',

1 => 'Framed-IP-Address',

2 => 'MAC',

3 => 'User-Name'

This is the default settings which will parse a line in the following format:

Nov 27 10:22:19 localhost dhcp: IP address 162.217.215.10 MAC 78:a2:a0:0e:cc:0c is at 000525673 eth3:4

Parsed output would be:

'Acct-Status-Type' => 'Start',

1 => ‘162.217.215.10',

2 => ‘78:a2:a0:0e:cc:0c’,

3 => '000525673'

In other implementations your log lines may differ and contain information that is different from the example above, the basic requirement would be that you end up with a parsed (Username, IP and MAC).

You can override the files or add multiple files to be processed by the syslog parser. The parser will move the file to a file_name.nsw, process it, once processed the file will be deleted, and the process will repeat.

$SYSLOGPARSER_FILES = array("/usr/local/netsweeper/logs/syslog.log");

rsyslogd Service

The rsyslog service is responsible for all local system logging. (Typically /var/log)

The rsyslog service can also be configured to handle additional logging streams including:

Remote machine logging
Remote service logging
Reroute exiting logging
Filter local / remote logging

Additional rsyslog Configuration Examples

Parse incoming TCP syslog connection for DHCP messages

This example parses incoming TCP syslog connection for DHCP messages and save to /usr/local/netsweeper/logs directory.

# Load Modules

module(load="imtcp")

module(load="imuxsock")

# rsyslog Templates

# rsyslog Input Modules

input(type="imtcp"

port="514")

# rsyslog RuleSets

ruleset(name="DHCPLogs") {

}

# Default RuleSet

if $programname == "DHCP" then {

action(type="omfile"

File="/usr/local/netsweeper/syslog.log")

}

Parse incoming UDP syslog connection for DHCP messages

This example parses incoming UDP syslog connection for DHCP messages and save to /usr/local/netsweeper/logs directory

# Load Modules

module(load="imudp")

module(load="imuxsock")

# rsyslog Templates

# rsyslog Input Modules

input(type="imudp"

port="514")

# rsyslog RuleSets

ruleset(name="DHCPLogs") {

}

# Default RuleSet

if $programname == "DHCP" then {

action(type="omfile"

File="/usr/local/netsweeper/syslog.log")

}

Parse incoming UDP syslog connections based on originating IP and separate logs

This example parses incoming UDP syslog connections based on the originating IP addresses and separates logs into different files in the /usr/local/netsweeper/logs directory

# Load Modules

module(load="imudp")

module(load="imuxsock")

# rsyslog Templates

# rsyslog Input Modules

input(type="imudp"

port="514")

# rsyslog RuleSets

ruleset(name="DHCPLogs") {

}

# Default RuleSet

if $fromhost-ip startswith '192.168.100.100' then /usr/local/netsweeper/syslog1.log

if $fromhost-ip startswith '192.168.100.102' then /usr/local/netsweeper/syslog2.log

Reference: Syslogparser.php Configuration File

This section is a reference to the Syslogparser.php configuration file.

This file can be found here:

vim /usr/local/netsweeper/webadmin/config/syslogparser.php

Configuration	Description
$ENABLE_SYSLOGPARSER_ LOGGING= false;	Set this to ‘true’ if you want to debug Syslogparser and require logs from the parser.
$SYSLOGPARSER_ LOGFILE_LEVEL=1;	The logfile level to log: - 1 INFO -2 STATS - 3 ERROR
$SYSLOGPARSER_ LOGFILE="syslogparser.log";	The logfile, will be /usr/local/netsweeper/logs/ It will be rotated only once to the filename. We do not keep more history then one single log file.
$SYSLOGPARSER_ LOGFILE_SIZE = 1048576 * 5;	The size of the log file before we rotate. We only keep 2 logs default 5 MB each, total 10MB
$PROCESS_INTERIM = false;	Process Interim updates the same way as we process SYSLOG Start packets
$GROUPNAME_LOOKUP_ FROM_USERNAME= true;	The USERNAME_ATTRIBUTE is used to lookup a client from the WebAdmin. The group this client belongs to is then used. If you have multiple clients the client with no IP Address or Subnet is first used. Otherwise, the first returned client is used. This DOES NOT CURRENTLY SUPPORT database partitioning and other advanced features. If this is set to true, we do not use the GROUPNAME_ATTRIBUTE UNLESS we cannot lookup the client via the WebAdmin Database. If no client exists, we will try to use the GROUPNAME_ATTRIBUTE.
$GROUPNAME_LOOKUP_FROM_ USERNAME_ALL_CLIENTS= false;	Lookup Group Name from Username using all clients from the database. The option $GROUPNAME_LOOKUP_FROM_USERNAME must be true. This will not only look for a Client with the Username matching the SYSLOG Accounting packet, but will also look for Clients with an IP Address and subnet with a matching username. This is recommended to be disabled. Feature added in 3.1.0 release. Enabling this option has a performance impact when Filtered and Non-Filtered subscriber SYSLOG Traffic is sent to the Netsweeper system and the $GROUPNAME_LOOKUP_ FROM_USERNAME option is enabled.
$GROUPNAME_LOOKUP_ FROM_USERNAME_ FALLTHROUGH= false	If we fail to lookup the groupname from the username in the Clients window, fail through to groupname attribute.
$GROUPNAME_ ATTRIBUTE = “Class”;	Name of the attribute to use to determine the group to place the client into. This attribute MUST exist for the IP Address Assignment and it MUST exist if we are going to create the group. By default, we use the Class attribute for the group name. See also: PARTITION_ATTRIBUTE
$GROUPNAME_ATTRIBUTE_ VALID_REGEXP=false;	The array or \| segmented regular expressions for each GROUPNAME_ATTRIBUTE for validation Examples: Must be numeric User-Name $GROUPNAME_ATTRIBUTE = "User-Name"; $GROUPNAME_ATTRIBUTE_VALID_REGEXP = "/^[0-9]+/"; Examples: Must be alpha User-Name, otherwise look for a Numeric Calling-Station-Id $GROUPNAME_ATTRIBUTE[0]="User-Name"; $GROUPNAME_ATTRIBUTE_VALID_REGEXP[0] = "/^[A-Za-z]+$/"; $GROUPNAME_ATTRIBUTE[1]="Calling-Station-Id"; $GROUPNAME_ATTRIBUTE_VALID_REGEXP[1] = "/^[0-9]+$/";
$USERNAME_ATTRIBUTE_ VALID_REGEXP = false;	The Attribute we get the username for the Client we assign to the group from
$ACCOUNTNAME_ ATTRIBUTE = "User-Name";	The Attribute we get the Account Name from
$ACCOUNTNAME_ATTRIBUTE_ VALID_REGEXP = false;	The array or \| segmented regular expressions for each ACCOUNTNAME_ATTRIBUTE for validation.
$IPADDRESS_ATTRIBUTE = "Framed-IP-Address";	The attribute we get the IPv4 Address from
$IP6ADDRESS_ATTRIBUTE = "Framed-IPv6-Prefix";	The attribute we get the IPv6 Address from: Framed-IPv6-Address, Login-IPv6-Host, or Framed-IPv6-Prefix.
$IP6PREFIX_ATTRIBUTE = "Framed-IPv6-Prefix";	The attribute we get the IPv6 Prefix from. The setting is used to check if the IP6 address is from IP6PREFIX_ATTRIBUTE. It is not used when the INTERFACEID_ATTRIBUTE is false. The INTERFACEID_ATTRIBUTE will be merged into the IP6ADDRESS_ATTRIBUTE only if the IPv6 address is from IP6PREFIX_ATTRIBUTE
$INTERFACEID_ATTRIBUTE = false;	The attribute we get the IPv6 interface identifier: Framed-Interface-Id. This is an optional attribute to merge with the Framed-IPv6-Prefix for a full IPv6 address of an interface. Set the attribute false to disable merging of the interface id with the ipv6 prefix.
$IP6ADDRESS_ATTRIBUTE_ VALID_REGEXP = false;	The array or \| segmented regular expressions for each IP6ADDRESS_ATTRIBUTE for validation.
$INTERFACEID_ATTRIBUTE_ VALID_REGEXP = false;	The array or \| segmented regular expressions for each INTERFACEID_ATTRIBUTE for validation.
$IPADDRESS_ATTRIBUTE_ VALID_REGEXP = false;	The array or \| segmented regular expressions for each IPADDRESS_ATTRIBUTE for validation.
$PARTITION_ATTRIBUTE = "User-Name";	Partition Attribute is the attribute which must exist for us to process the Accounting message. This attribute must simply exist, and possibly have a value.
$PARTITION_ATTRIBUTE_ VALID_REGEXP = false;	The array or \| segmented regular expressions for each PARITION_ATTRIBUTE for validation.
$PARTITION_VALUE = false;	The optional value of the Partition Attribute. This can be set to false, in which case we do not check the value. Otherwise, the value must be the specific value for the IP Address assignment to occur.
$PARTITION_CLEAN_ START = true;	Cleanup users that do not have a filtering group on SYSLOG Accounting Start, Interim. This will remove the Framed-IP-Address from the database if we cannot lookup the associated filtering group. This can happen if the PARTITION_ATTRIBUTE does not exist, or we cannot find the Client in the Clients window or if we cannot find the group.
$PARTITION_CLEAN_STOP = true;	Cleanup users that do not have a filtering group on SYSLOG Accounting Stop. This will remove the Framed-IP-Address from the database if we cannot lookup the associated filtering group. This can happen if the PARTITION_ATTRIBUTE does not exist, or we cannot find the Client in the Clients window or if we cannot find the group.
$CREATE_ACCOUNT=false;	Do we want to create a WebAdmin Account?
$CREATE_ACCOUNT_ PASSWORD="foobarbaz";	The password of the account.
$CREATE_ACCOUNT_ ORGANIZATION = "syslogparser";	The organization the user has, in Accounts window.
$CREATE_ACCOUNT_ CLASSIFICATION = "user";	The Classification of the user in the Accounts window.
$CREATE_ACCOUNT_ASSIGN_ TO_GROUP=false;	Assign account to group based on CLASS attribute.
$CREATE_GROUP=false;	Create the group if it does not exist.
$CLIENT_API_DELETE_ MODE=false;	If we delete/add the client or simple update.
$QUARANTINE_ATTRIBUTE = false;	The attribute used to check if the client is quarantined. If set to false, the quarantine section will not run.
$QUARANTINE_ATTRIBUTE_ VALID_REGEXP = false;	The array or \| segmented regular expressions for each GROUPNAME_ATTRIBUTE for validation.
$QUARANTINE_ATTRIBUTE_ VALUE = "quarantine";	Value to check if the client should use its own group, or the quarantine allow group.
$QUARANTINE_ATTRIBUTE_ CHECK_DATABASE = false;	Used to see if the client should be checked in the database, if there is no quarantine information in the syslog log entry.