LogMod5

Introduction to LogMod5

Netsweeper logging makes use of the Netsweeper LogMod 5.0 logging framework, which is a graph-based framework that provides a high level of flexibility.

This document describes some very simple configurations that can be used to solve a variety of common logging requirements.  In general, these settings can be used as is or as a reference to build even more complex logging configurations to meet specific requirements.

LogMod 5.0 was introduced into the Netsweeper product in version 2.6.29.14 and later.

During normal operation, this is how the process flows:

1. Log entries are queued up as they are generated.

2. This queue is periodically emptied and the entries in the queue enter the LogMod5 graph as a packet (bundle) of entries.

3. The packet starts at the lm5_entry node and moves through the graph via connections.

4. The packet reaches an end point and is processed in a variety of ways.

5. For example:

   • Written to nslogger_request.log log files

   • Sent to a remote logger

   • Stored temporarily to be processed later

This diagram explains in a little more detail:

The LogMod 5 Graph

Nodes in a LogMod5 graph are specified and arranged inside the Policy Server Settings (nsd.conf) which is accessed through System Tools > System Configuration > Policy Server Settings in the WebAdmin.  Specifying which nodes you want and how to connect and configure them is covered later.

The default configuration is a simple configuration that logs all entries to the local disk.  By default, we will use a maximum of 5 gigabytes of disk space utilizing many log files that are required until this maximum is reached.

Graph Entry Points

Packets enter the graph and flow 'downwards' from starting points in the graph.  In the examples above, the entry point is the node labeled lm5_entry.  There is only one other kind of entry node that you can use, it is called lm5_remote_entry and it is explained later.

Graph Middle Nodes

Entry Point Packets usually flow through 'Middle Nodes'.  Middle nodes usually have one entry connection from a node above, and one or more exit connections below.  Inside the node, the packet usually undergoes some processing.  Packets may be:

• routed – the node decides which exit connection (channel) to send the packet towards and then sends the packet to the node below connected on that channel

• copied – the node may accept packets from below and copy the packet to some or all of the nodes connected to its downstream channels

• split – some nodes might take the packet and break it into pieces and send some pieces to one channel and some pieces to another channel.  It might even be programmed to simply discard some pieces of the packet

• stored – some nodes might behave as temporary endpoints, storing a packet for a short time and then sending it downstream later.  (An example of this is the disk_queue node and the lm5_timer node, described later).

Graph End Nodes

Packets usually flow downwards through the graph without stopping until they reach an end node.  End nodes have one (or more) entry channels and no exit channels.  End nodes usually do things like:

• transmit packets over the network

• write packets to disk in the form of log entries

• discard entries, simply throwing them away

• count entries or collect other statistics

Examples of end nodes, described below, include the disk_writer, the tcpclient, and the devnull nodes.

Graph Examples

Here are a few examples of LogMod5 graphs:

Example 1

In this first graph, packets begin in the lm5_entry node, pass through the disk_queue node, and then are processed by the tcpclient #1 node.

Example 2

In the second graph, packets flow through an lm5_mirror node which makes a copy of the packet and sends a copy to disk_writer and a copy to tcpclient #1

Example 3

In the last graph, packets are first sent to tcpclient #1 but are sent to tcpclient #2 if there is any kind of error (for example if the remote logger is unavailable).  If tcpclient #2 is also down or if there is some kind of error, then packets will be sent to tcpclient #3.  If all three tcpclients are unavailable, then entries will not be logged anywhere.  It is important to note that if failure occurs back to the lm5_entry node errors will be logged into the policy server’s error log file.  We will discuss how to create a configuration which will fail open on failover later in this document.

 

Failure Modes

The LogMod5 framework has the ability to handle temporary (or permanent) errors or interruptions in service.  Understanding failure behavior is essential to properly configuring a LogMod5 system.

Types of Failure Handling

There are three kinds of failures that the LogMod5 system is designed to handle:

1. Packet failure

2. Packet partial failure

3. Node failure

Each of these is explained in detail in the following sections.

Packet Failure

As a packet is traveling downwards through the graph, any node along the way can decide to return either FAILURE or SUCCESS for the packet, or it can send the packet down one of its downstream channels and let some node below decide whether to return FAILURE or SUCCESS.  When a result (of either FAILURE or SUCCESS) is decided, every node above that point will get a chance (in reverse order) to react to that result.  A result of SUCCESS usually does not trigger any further action but, for some nodes, a result of FAILURE triggers special behavior.

The lm5_failover node is a good example on understanding how this works.  The lm5_failover node, responds to packet FAILURE by sending the packet to its next downstream channel and trying again.  The lm5_failover node responds to packet SUCCESS by not forwarding the packet to another channel, but simply returning the verdict of SUCCESS upstream.  In this way, the lm5_failover node does exactly what you would expect: it sends all packets to a primary channel and routes packets to a backup channel (or channels) if/when there is a problem with any of those packets.

The details of what FAILURE actually means, why it occurs, and how it is handled is a matter of node semantics and will be different for every node.  Refer to the 'Node Failure' section to learn how each node in a graph will handle FAILURE.

Packet Partial Failure

Partial failure, as described in the table of terminology, basically means that packet has returned FAILURE but in reality it was only a partial failure because some portion of the packet was processed successfully.  How it was processed will be different from node to node, but in the case of LogMod5 request logging, it usually means that some of the log entries are successfully saved to disk and some of the entries are not (yet) successfully saved anywhere.

Partial failure is handled the same as normal packet failure and there is usually no way to really tell the difference between a partial or full failure.  But, it is important to understand, that it is a capability of the LogMod5 system, that for many nodes it is handled as a special case, and it can sometimes cause surprises in behavior.

If a LogMod5 configuration is producing duplication or loss of entries, then it is usually a problem with partial failure and the graph should be considered carefully from the perspective of how partial failures will be handled.

Node Failure

Node failure is usually not as serious as it sounds.  Node failure typically indicates a temporary condition has come up that prevents the node from handling any further packets.  For example, the tcpclient node can enter the failed state in several different ways:

1. It has not yet been able to make a connection with the remote server.

2. The connection was lost (and the node is trying to re-establish it).

3. The connection is ‘clogged’ because the tcpclient has sent more data than the remote system can handle or more than the network connection can handle.  The failure state will go away when the ‘clog’ is cleared and the remote system is ready to process more data.

These are examples of node failure in the tcpclient node.  Nearly all of the LogMod5 nodes will have some kind of failure mode.  The disk writer node might report a failure when its disk is full or if its files cannot be written for some reason.

Node failure is quite different from packet failure.  It is possible and common for node failure to happen without any packet failure and sometimes an individual packet failure happens without node failure.  Refer to node descriptions to learn how each of these conditions can occur in each type of node.

End point nodes are not the only kinds of nodes that might go into a ‘failed state’.  Nodes in the middle of the graph can also enter the failed state.  For example, the lm5_failover node will enter the failed state if all of its connected downstream node instances are reporting failure.  A disk queue node will report failure if its downstream node is failed and if its internal disk is full or has reached its configured size limit.

Node failure is logmod5’s primary means of ensuring that log entries flow smoothly, that no data is lost, and that there is no system hanging (causing difficulty shutting down or temporary slowdowns in service).

A Node Failure Analogy

If you think of a graph as a vertical system of pipes and joints, and you think of the flow packets downward through the pipes as the flow of water, you can think of node failure as simply a closing of a valve above a node.  When a node is not able to process information (no matter how long or short this state lasts), the node 'closes the valve' and this prevents water flowing in.  With the valve closed, nodes immediately above that must make a decision to divert the water to a different downstream channel, to simply become a dead end, to turn off their own upstream valve, or in the case of some middle nodes (such as the disk_queue node) to become like a 'water tank' storing up packets until either the tank is full or the valve below is opened again.

Failover Graph Examples

Failover between three remote loggers

In this graph an lm5_failover node reroutes packet flow to the next working tcpclient, if/when a remote logger goes offline.

1. All remote loggers online: Log entries flow from lm5-entry through failover to tcpclient #1.

2. Remote Logger #1 goes offline: tcpclient #1 closes its 'inlet valve; failover diverts flow to tcpclient #2

3. Remote Logger #2 goes offline: tcpclient #2 turns off its 'inlet valve'; failover diverts flow to tcpclient #3; packets continue to flow and are now handled by tcpclient #3

 

In the above graph it is important to note that if all three tcpclients go offline, the lm5_failover node will also go into a failover state.  This will report back to the lm5_entry node a failure case.  This will start to log error messages into the policy server error log file whenever a group of log entries is processed and discarded.  To avoid having the policy server error log fill up or by ignoring failures of this sort, it is possible to add a devnull node in the failover three.  The devnull node will never fail and simply consume all entries.

Failover To Devnull Node To Avoid Error Logging

Tcpclient Node with disk_queue “temporary storage”

In this graph a disk_queue node acts as a “tank” and stores packets whenever the tcpclient node turns off the valve.

Two remote loggers with timer

In this interesting example, a timer node is used to schedule usage of a remote server.  If we have a production situation where there are two remote logging systems, one of which is to be used during the hours of 9am – 5pm and the other to be used during all other hours of the day, we can use a timer node to make it all happen.

In this example, notice that disk_queue nodes could be added to immediately above the tcpclients in order to ensure that log entries go to the intended destinations at the indented times.  Without any disk_queue  nodes, you will still send entries to remote system #2 in the case of an outage to remote system #1 during the 9-5 window when entries are intended to go to remote system #1.

Two remote loggers on Schedule with Disk Queues before TCPclient

 

Also, what would happen if you put a disk queue above the timer instead of below it?  In that case you would actually store all log entries during off hours and then transmit them all during the 9-5 window.  Failover to tcpclient #2 would only happen when the disk queue failed, for example, because local disk space was entirely consumed.

Transfer of Logs to One System, if disk queue errors failover to another remote system

 

Configuring LogMod5 in the Policy Server

This section describes the relevant settings in Policy Server configuration, nsd.conf file, and how to get set up using LogMod5.  By default, when you upgrade to the 2.6.29.14 release you will be already using the LogMod5 framework.  Check the bottom of the configuration file for the specific information in regard to the LodMod5 graph configured on your system.

Configure Logger

The logger module is a ‘legacy’ system predating logmod5.  The logger module is used directly by all the policy server request threads that are servicing individual requests for policy rulings, remote logging requests, etc.  When a request thread has a new log entry to record, it sends it to the logger module.  The logger module places it in a queue of log entries.  Once the log entry queue reaches a certain size or once a certain amount of time has passed (a fraction of a second), the queue is processed by sending it to a processing framework.

Prior to logmod5, the processing frameworks available were:

• local – a module which writes log entries to the local disk as nslogger_request*.log files

• remote – a module which transmits log entries to a configured remote system

With LogMod5 included in the policy server, a new framework is added.  The old frameworks are still available however.  There are now two new frameworks available (both essentially pointing to the same thing):

• LogMod5 – a module which routes log entries into a LogMod5 packet and gives this packet to the lm5_entry node

• logmod5_remote – a module which routes log entries into a LogMod5 packet and gives this packet to the lm5_entry_remote node

All logger nsd.conf settings are as they were before.  This includes any setting that begins with the word 'logger_' but which does not begin with the word logger_local or logger_remote.  (All the settings for logger_local and logger_remote will no longer apply).

 

Although you can list multiple handlers here (separated by commas) you should not use both LogMod5 and local at the same time.  This is because they both register to the same SNMP OIDs and will conflict, causing your SNMP to not work properly.

Configure LogMod5

Once the LogMod5 framework is enabled by specifying LogMod5 as a processing framework (as described above), you must configure LogMod5.  Configuring LogMod5 is done a little differently from most other settings in the policy server.  Configuring LogMod5 is done in three sections:

1. Instantiate nodes – by specifying node types followed by node names, you instantiate named instances of all the nodes you need in your graph(s).  If you require multiple instances of a node, simply specify the node type again on another line in nsd.conf and give the second instance a different node name.  Node names must be unique across the entire nsd.conf file.

2. Make connections – Connections are made in the graph one connection at a time.  Each graph connection is defined on a new line in the nsd.conf.  Graph connections are defined with ‘logger_logmod5_connect’ followed by the node name of the upper node, followed by the name of the lower node.  The order of the lines in nsd.conf will set the order of the channels for each node, and this generally determines the channel priority.  So, for connections where priority matters (such as with failover, where you have a first priority channel, followed by a second, and a third, etc), be sure to specify your highest priority connection first in the nsd.conf.

3. Provide node parameters – most (but not all nodes) will require some parameters for configuration.  In the case of the tcpclient node, this includes at least the remote host name and the remote port number.  Parameters are provided in this format: node_parameter_name value.

Look at an example configuration:

lm5_entry entry_point

lm5_disk_queue tmp_que

lm5_tcpclient remote_connection

 

logger_logmod5_connect entry_point tmp_que

logger_logmod5_connect tmp_que remote_connection

 

tmp_que->file_path /tmp/

tmp_que->file_prefix nsd_temporary_quefile

tmp_que->max_disk_size 10G

tmp_que->max_disk_size_thresh 100K

tmp_que->num_files 10

 

remote_connection->remote_logger_host 192.168.4.64

remote_connection->remote_logger_port 3000

With the setting below, the local logger will send log entries of 2 types to the logger:

• Old logger (3.0.8.0 and below) only supports 255 categories. 0 (zero) indicates limited categories supported by the remote logger.

• New logger (3.0.9.0 and up) that support unlimited categories. 1 indicates unlimited categories supported by the remote logger.

remote_connection->unlimited_cats 0

Example: Three Nodes are Instantiated

In this example, three nodes are instantiated (a lm5_entry node, a lm5_disk_queue node, and an lm5_tcpclient node) and these nodes are given meaningful and unique names.  Below this the two connections are made, forming a straight “tree” with entry point connected to queue connected to TCP below that.  Finally, each node receives its configuration parameters by specifying the name of the node, the parameter, and its value.

Creating Multiple Graphs in the same nsd.conf

As described above, in the section about the Logger, it is possible to have separate graphs for the remote and the local entries.  Creating a separate graph is done by instantiating all the nodes you need for both graphs, specifying connections (for both graphs), and then configuring all the nodes.  In other words, there’s nothing special about it.  Except, you have nodes that have no connection to each other.  As a result, you will have two independent graphs.

LogMod5 Quick Start Examples

You can copy or adapt one of these example configurations in order to get up and running easily.

Quick Start - Example 1

In this first example, we send all log entries for the "default" group to one log file and all others to a different log file:

lm5_entry entry_point

lm5_group_filter dflt_filter

lm5_diskwriter dflt_writer

lm5_diskwriter other_writer

logger_logmod5_connect entry_point dflt_filter

logger_logmod5_connect dflt_filter dflt_writer

logger_logmod5_connect dflt_filter other_writer

dflt_filter->filter default

dflt_writer->max_file_size 1G

dflt_writer->max_file_count 5

other_writer->max_file_size 1G

other_writer->max_file_count 5

Quick Start - Example 2

The second example will log all entries to a remote server. If there is an interruption in the network connection, log entries will be saved temporarily on the local hard drive and copied over automatically when the network connection is restored.

lm5_entry entry_point

lm5_disk_queue tmp_que

lm5_tcpclient remote_connection

logger_logmod5_connect entry_point tmp_que

logger_logmod5_connect tmp_que remote_connection

tmp_que->file_path /tmp/

tmp_que->file_prefix nsd_temporary_quefile

tmp_que->max_disk_size 10G

tmp_que->max_disk_size_thresh 100K

tmp_que->num_files 10

remote_connection->remote_logger_host 192.168.100.99

remote_connection->remote_logger_port 3000

Quick Start - Example 3

In the third example, we introduce a timer node to the queue and TCP configuration. The timer node can be used to effectively make the TCP node appear to be offline for certain periods (causing everything to be queued to disk during that time). This is useful if you would like to use nighttime bandwidth to transmit log entries (from the previous day) and not transmit any log entries during the day (just save them to be sent later). Nighttime will be 8pm -> 6am every day. Also, note that the timer will not interfere with the tmp_que's ability to handle real network outages during nighttime transmission. If the TCP connection goes offline then the disk queue will still queue up entries until the TCP connection is restored.

lm5_entry entry_point

lm5_disk_queue tmp_que

lm5_timer night_timer

lm5_tcpclient remote_connection

logger_logmod5_connect entry_point tmp_que

logger_logmod5_connect tmp_que night_timer

logger_logmod5_connect night_timer remote_connection

tmp_que->file_path /tmp/

tmp_que->file_prefix nsd_temporary_quefile

tmp_que->max_disk_size 1G

tmp_que->max_disk_size_thresh 100K

tmp_que->num_files 10

night_timer->schedule 20:00 06:00

remote_connection->remote_logger_host 192.168.100.99

remote_connection->remote_logger_port 3000

Quick Start - Example 4

In the fourth example, we split logging entries up using a hash. To do this you can either use one of the hashing nodes (or any combination of hashing nodes) or you can do, as in this example, and use the macro node which makes it very simple to configure any n-way hash to the disk. In this example, we make a 32-way hash by URL. This will create 32 different log files.

lm5_entry entry_point

lm5_hash_macro hash_mac

hash_mac->hash_type url

hash_mac->hash_size 32

logger_logmod5_connect entry_point hash_mac

Quick Start - Example 5

In the last example, we load balance (round robin) log entries between three different remote servers (note, if one connection fails then entries will automatically divert to the next available connection):

lm5_entry entry_point

lm5_lb_rr balancer

lm5_tcpclient remote1

lm5_tcpclient remote2

lm5_tcpclient remote3

logger_logmod5_connect entry_point balancer

logger_logmod5_connect balancer remote1

logger_logmod5_connect balancer remote2

logger_logmod5_connect balancer remote3

remote1->remote_logger_host 192.168.100.100

remote1->remote_logger_port 3000

remote2->remote_logger_host 192.168.100.101

remote2->remote_logger_port 3000

remote3->remote_logger_host 192.168.100.102

remote3->remote_logger_port 3000

Default 2.6.29.14 Configuration with No Remote Logging

The default configuration is a very simple configuration that logs all log entries to the local disk.  By default we will use a maximum of 5 gigabytes of disk space utilizing many log files that are required until this maximum is reached.

The above configuration is represented in the configuration file with the following policy server configuration specifications.

 

lm5_entry g1_entry

lm5_failover g1_entryfailover

logger_logmod5_connect g1_entry g1_entryfailover

 

lm5_diskwriter g1_disk

logger_logmod5_connect g1_entryfailover g1_disk

 

lm5_devnull g1_entrynull

logger_logmod5_connect g1_entryfailover g1_entrynull

 

All modules in this case will be using the default configuration settings.

 

If you wish to specify new settings for the amount of disk space the g1_disk can use, you would need to add the following configuration values:

g1_disk->max_disk_usage_size 100 gbytes

 

For a complete specification of each node type and the configuration settings, please refer to the “Appendix: Node Definitions” section at the end of this document.

Logging configurations should be configured once at deployment time. Log files might be orphaned in the event that configuration changes are made to a production system. Whenever modifying the logging configuration, you must be certain that old log files are renamed or manually removed.

There will be log files orphaned if the disk writer node name is modified, or the log file type is modified.

Please consult with your Netsweeper Systems Engineer, prior to altering logging configurations for production systems.

Default Remote Logging Upgrade

During an upgrade to 2.6.29.14, if you had a remote logging server configured, the following configuration will be used once you upgrade to the new release of the Netsweeper product.  You should verify after upgrade that the following configuration is present for all policy servers logging to remote servers.  It is important to note there are two graphs created.  One graph to send logs to the remote server and another that will save logs sent from a remote server locally.  By default previous releases did not distinguish between locally generated policy requests and requests from remote servers.  With LogMod5 you cannot store remote log requests to different files.

Simple Example LogMod5 Graphs

 

The following configuration will be created in the policy server configuration, nsd.conf file, when upgrading to the 2.6.29.14 release.

 

lm5_entry_remote rg1_entry

lm5_failover rg1_entryfailover

logger_logmod5_connect rg1_entry rg1_entryfailover

lm5_diskwriter rg1_disk

logger_logmod5_connect rg1_entryfailover rg1_disk

lm5_devnull rg1_entrynull

logger_logmod5_connect rg1_entryfailover rg1_entrynull

 

lm5_entry g1_entry

lm5_failover g1_entryfailover

logger_logmod5_connect g1_entry g1_entryfailover

 

lm5_mirror g1_mirror

logger_logmod5_connect g1_entryfailover g1_mirror

 

lm5_diskwriter g1_disk

logger_logmod5_connect g1_mirror g1_disk

 

lm5_tcpclient g1_tcp_1

logger_logmod5_connect g1_mirror g1_tcp_1

 

lm5_devnull g1_entrynull

logger_logmod5_connect g1_entryfailover g1_entrynull

 

g1_tcp_1->remote_logger_port 3000

g1_tcp_1->remote_logger_host 192.168.1.123

 

Multiple Remote Servers

The above configuration can be modified for more remote servers by adding more tcpclient nodes after the mirror node.  For example, the following could be added to the above configuration to add another tcpclient node that logs entries to another remote server.

lm5_tcpclient g1_tcp_2

logger_logmod5_connect g1_mirror g1_tcp_2

 

g1_tcp_2->remote_logger_port 3000

g1_tcp_2->remote_logger_host 192.168.2.222

 

The following graph would represent the complete above configuration for the lm5_entry.  You can see that the g1_tcp_2 has been added to the mirror node.

Removing the Local Disk Logging on Policy Server installations

In the above examples, all data was logged locally on each policy server instance and also mirrored to a remote logging server.  For performance reasons you may wish to remove the local logging aspect since these logs are only used for support, troubleshooting and diagnostic purposes.

Only remote logging to one server:

 

The configuration for the above example is rather simple since only the disk entry is removed.  For the lm5_entry graph, the following configuration could be used.  Please be advised this example does not include the logger_remote_handlers and the logmod5_remote_entry.

 

lm5_entry g1_entry

lm5_failover g1_entryfailover

logger_logmod5_connect g1_entry g1_entryfailover

 

lm5_tcpclient g1_tcp_1

logger_logmod5_connect g1_entryfailover g1_tcp_1

 

lm5_devnull g1_entrynull

logger_logmod5_connect g1_entryfailover g1_entrynull

 

g1_tcp_1->remote_logger_port 3000

g1_tcp_1->remote_logger_host 192.168.1.123

Guaranteed Remote Logging Configurations

One of the main advantages and features of the LogMod5 framework is the ability to guarantee log file delivery.  This is completed by adding the diskqueue node in the appropriate place in the graph.  Generally, this node should be used before a tcpclient node.

Guaranteed log file delivery to remote server:

 

The following configuration will build the complete configuration illustrated above.  This is very similar to the remote logging example with the addition of the g1_que_tcp_1 node which will queue failed log entries sent to the tcpclient node g1_tcp_1.

 

lm5_entry_remote rg1_entry

lm5_failover rg1_entryfailover

logger_logmod5_connect rg1_entry rg1_entryfailover

lm5_diskwriter rg1_disk

logger_logmod5_connect rg1_entryfailover rg1_disk

lm5_devnull rg1_entrynull

logger_logmod5_connect rg1_entryfailover rg1_entrynull

 

 

lm5_entry g1_entry

lm5_failover g1_entryfailover

logger_logmod5_connect g1_entry g1_entryfailover

 

lm5_mirror g1_mirror

logger_logmod5_connect g1_entryfailover g1_mirror

 

lm5_diskwriter g1_disk

logger_logmod5_connect g1_mirror g1_disk

 

lm5_disk_queue g1_que_tcp_1

logger_logmod5_connect g1_mirror g1_que_tcp_1

 

lm5_tcpclient g1_tcp_1

logger_logmod5_connect g1_que_tcp_1 g1_tcp_1

 

lm5_devnull g1_entrynull

logger_logmod5_connect g1_entryfailover g1_entrynull

 

g1_que_tcp_1->file_path /usr/local/netsweeper/logs

g1_que_tcp_1->file_prefix disk_queue_g1_que_tcp_1

g1_que_tcp_1->max_disk_size 10G

g1_que_tcp_1->max_disk_size_thresh 100K

g1_que_tcp_1->num_files 10

 

 

g1_tcp_1->remote_logger_port 3000

g1_tcp_1->remote_logger_host 192.168.1.123

 

It is recommended to always specify the file sizes for all node elements when possible.  This will make sure after an upgrade the configuration will remain the same.  Sometimes default values will change and may cause issues after an upgrade.  The example above sets the disk queue node to use 10 gigabytes of disk space in 10 files.

If there is a system failure that consumes all this space, the disk_queue node will fail.  This will proceed to make the mirror node fail, and the log entries will be sent to devnull, and no errors will be reported in the policy server error log.  This may not be the desired effect, and you may wish for errors to be logged.  If this is the case, you can remote the failover node and the devnull node to create a simpler graph.

Guaranteed log file delivery to remote server:

 

The following configuration will build the complete configuration illustrated above.  This is very similar to the remote logging example with the addition of the g1_que_tcp_1 node which will queue failed log entries sent to the tcpclient node g1_tcp_1.

 

lm5_entry_remote rg1_entry

lm5_diskwriter rg1_disk

logger_logmod5_connect rg1_entry rg1_disk

 

lm5_entry g1_entry

 

lm5_mirror g1_mirror

logger_logmod5_connect g1_entry g1_mirror

 

lm5_diskwriter g1_disk

logger_logmod5_connect g1_mirror g1_disk

 

lm5_disk_queue g1_que_tcp_1

logger_logmod5_connect g1_mirror g1_que_tcp_1

 

lm5_tcpclient g1_tcp_1

logger_logmod5_connect g1_que_tcp_1 g1_tcp_1

 

g1_que_tcp_1->file_path /usr/local/netsweeper/logs

g1_que_tcp_1->file_prefix disk_queue_g1_que_tcp_1

g1_que_tcp_1->max_disk_size 10G

g1_que_tcp_1->max_disk_size_thresh 100K

g1_que_tcp_1->num_files 10

 

 

g1_tcp_1->remote_logger_port 3000

g1_tcp_1->remote_logger_host 192.168.1.123

The above configuration will make sure that even if the disk queue node fails, errors will be reported in the policy server error log indicating a critical problem has occurred.

Other Configurations

There are many possible configurations with the LogMod5 framework introduced in the 2.6.29.14 release.  The above logging configurations will provide customers with the usable features available in the 2.6.29.14 release when combined with the other services like the WebAdmin interface and Reporter services.  Moving forward with Reporter performance and Reporter clustering the following configurations will become more apparent for some customers.

Reporting performance:

• Hashing of Log Files by User

• Hashing of Log Files by IP Address

• Log Filtering

Reporter clustering and high availability:

• Failover configuration

• Logging to multiple remote servers

These configurations will be documented and once the reporter and other services can take advantage of these new features.

Log File Rotation

The settings below can be used for log file rotation.  When they are changed, the log file format of logs will change.   Old log files will not be rotated or written to.  It is important when these settings are changed to manually remove old log files or manage log file rotation.

• log_peeraddress_enabled true

• log_identifier_enabled true

• logfile_type

Enabling Charting

To enable Charting, you have to modify the current LogMod5 configuration in the nsd.conf.  These lines need to be added. You will need to also comment out or modify the existing LogMod5 configuration.

lm5_entry g1_entry

lm5_snmp_cats snmp

lm5_snmp_topdomains snmp1

lm5_snmp_groupcats snmp2

lm5_snmp_groupreqs snmp3

lm5_snmp_clientcats snmp4

lm5_snmp_clientreqs snmp5

lm5_snmp_grouptopdomains snmp6

 

lm5_diskwriter g1_disk

 

logger_logmod5_connect g1_entry snmp

logger_logmod5_connect snmp snmp1

logger_logmod5_connect snmp1 snmp2

logger_logmod5_connect snmp2 snmp3

logger_logmod5_connect snmp3 snmp4

logger_logmod5_connect snmp4 snmp5

logger_logmod5_connect snmp5 snmp6

logger_logmod5_connect snmp6 g1_disk

 

After clicking Submit to save the changes, Restart the Policy Server.

Logging to a Syslog Node

You can set up LogMod5 to use syslog for request logging by using the lm5_syslog.  It is used as an exit node to send all log records to the syslog daemon.

# | logmod5 Syslog node     |

# Declare as:

#   lm5_syslog <NAME>

# This node sends all log records to the syslog daemon. Here are the configurable settings for the disk queue

#   NAME->logger_fields field1 field2 field3 ...

#     This allows to specify the log record fields that should be added to the logged record

#    This setting overrides the global logger_fields setting for this node.

#    For available field names see global logger_fields setting description.

#   Default (if neither NAME->logger_fields nor global logger_fields is specified):

#    NAME->logger_fields url ip user group denied category

#   NAME->syslog_ident string

#   This sets the syslog ident argument. It is the word that is prepended to every message in the syslog file.

#   Default:

#    none

#    it means that syslog uses the program name "nsd" instead

#   NAME->syslog_facility string

#       This sets the syslog facility argument. It provides syslog the information about message source.

#       It lets the syslog configuration file specify what file these messages should be written to.

#       The facility can be: LOG_DAEMON, LOG_LOCAL0 - LOG_LOCAL7 or LOG_USER.

#   Default:

#    NAME->syslog_facility LOG_USER

#   NAME->syslog_level string

#       This sets the syslog level argument. This determines the importance of the message and extends the facility argument for more flexibility of destination file selection. The syslog node uses the same level (importance) for all request log messages.

#    The level can be: LOG_ALERT, LOG_CRIT, LOG_ERR, LOG_WARNING, LOG_NOTICE, LOG_INFO or LOG_DEBUG.

#   Default:

#    NAME->syslog_facility LOG_INFO

Netsweeper Request Logger Framework Options

These settings are found in Policy Server Settings.

Request Log Record Fields

This setting allows you to specify request log records parts (record fields) that should be written to log files or sent to a remote logger server. This global setting specifies the fields set for the remote logger, text format and extended binary log files. But disk log writes can have their own settings for this field set. The date/time field is the only field that is always included into any log record because it is needed for the Reporter system.

Available Fields:

Option	Description
url	The request URL
client	The Client name
group	The Group name
policy	The Policy name
ip	The client workstation IP address
dst	The request destination server IP address
denied	The denied flag (0 if the request is allowed, 1 if request is denied)
category	Two fields: the whole list of categories that were assigned to the request the denied category if the request was denied
interceptor	The request interceptor IP address
server_id	The policy server identification
method	The request HTTP method
agent	The HTTP request User-Agent header line
referrer	The HTTP request Referrer header line
type	The Policy Event Type
data	The Policy Event Data
allcategories	Only the whole list of categories that were assigned to the request (not recommended, use the 'category' field instead)
deniedcategories	Only the denied category if the request was denied, all assigned categories otherwise (not recommended, use the 'category' field instead)

Synonyms

Field Name	Synonyms
url
client	user
group	(no synonyms)
policy	(no synonyms)
ip	clientip, client_ip
dst	dstip, dst_ip, destinationip, destination_ip
denied	deniedflag, denied_flag
category	cat, categories, cats
interceptor	peer, peerip, peer_ip, interceptorip, interceptor_ip
server_id	serverid
method	httpmethod, http_method
agent	useragent, user_agent
referrer	Referrer
type	event_type, event
data	(no synonyms)
allcategories	allcat, allcats, allcategory
deniedcategories	deniedcat, deniedcats, deniedcategory

Default:

This setting doesn't have a default value but if it is not specified each logger part uses own defaults:

• Remote logging sends ALL available fields

• Text format log writer: url ip user group denied category

• Extended binary log writer: user group category denied url ip dst

Format: logger_fields field1 field2 field3 ...

Disable the Request Logging Framework

This setting allows you to disable the entire request logging framework. This will disable both local and remote logging of policy requests.

Default: logger_disabled false

Format: logger_disabled [ true | false ]

Logger Thread CPU Settings

The following settings can be set for the logging thread

logger_affinity [ ALL  | cpuno ] [ cpuno ] ...

• Sets which CPU this thread can run on.  Starts at index of 1 and goes up to the number of CPU's on the system.  ALL for any CPU to be used.

logger_priority

• The priority of this thread, -19 highest and 20 lowest 0 is the default.

• logger_scheduler [ SCHED_RR | SCHED_FIFO | SCHED_BATCH | SCHED_OTHER ]

• Change the Scheduler used for the logger thread.

See above for more information.

Logger Thread Queue Check Interval

This is the amount of time in milliseconds we wait before we check the queue.  If the soft limit is hit, the logger thread will be forced to check the queue.  This should be left at 100 msecs or possibly be lowered on extremely high performance 16 core plus machines.

Default: logger_queue_check_interval_msec 100 msecs

Format: logger_queue_check_interval_msec milliseconds

Logger Identifier

This is the identifier for policy server generated log entries.  When sending #     to a remote system or locally logging to extended binary log files, we can store this logger identifier.

Default: logger_identifier $HOSTNAME

Format: logger_identifier string

Request Log Entry Fail Mode

If a log entry fails to be processed, we can optionally serve a deny page and deny the request.  If local logging is configured and the log entry cannot be written to disk, this failure mode will be enforced.  If remote logging is configured with out of band logging, the log entry must fail to send to the remote server and fail to be added to the out of band log file before the log entry fails.

Default: logger_failure denypage

Format: logger_failure {denypage|shutdown}

Logger Queue Soft Limit

This is the soft limit of the logger queue.  When adding log entries to the queue we signal the logger processing thread once this soft limit is reached.  This allows us to process log entries in batches to increase performance.

Default: logger_queue_soft_limit 4000

Format: logger_queue_soft_limit log entry count

Logger Queue Hard Limit

This is the hard limit of the logger queue.  If we reach this limit we start to delay policy server processing.

Default: logger_queue_hard_limit 10000

Format: logger_queue_hard_limit log entry count

Logger Queue Hard Limit Discard Log Entry

When the hard limit is hit, but default the policy service will slow down and wait for the queue to be processed.   However, on high performance systems you may wish to discard log entries.   When discarding log entries you can either serve a deny page or shutdown the policy service.

Default: logger_queue_hard_limit_discard false

Format: logger_queue_hard_limit_discard [true|false]

LogMod 5 Terminology

The following terminology is used in most LogMod5 documentation:

Term	Description
Graph OR graph instance	A collection of nodes and node connections, forming a complete “tree”, makes a ‘graph instance’.  It is possible to have multiple ‘graph instances’ running at the same time in the nsd.
Node OR node type	A contained ‘package’ of behaviour that can be placed in a graph.  Nodes are ‘instantiated’ into a graph.  Example ‘node types’ include: lm5_entry, lm5_failover, tcpclient, and many others (listed below).
Instance OR node instance	When a node is inserted into a graph becomes a ‘node instance’ in the graph.  Nodes can be instantiated multiple times in the same graph.  For example, the tcpclient node is instantiated multiple times in the example above, allowing packets to failover from one tcpclient node instance (configured to connect to some remote server) to another tcpclient instance (usually configured to connect to some completely different remote server).
Connection	The route followed by packets traveling between two nodes is usually called a ‘connection’.  Connections are always unidirectional, and should be thought of as flowing ‘downwards’.
Channel	Nodes will refer to their incoming connection(s), connections upwards, as ‘upstream channels’ and will refer to their outgoing connection(s) as ‘downstream channels’.  Channels are sometimes referred to by their number and, in those cases, numbering begins at #0, for example, a node with two downstream connections will have a ‘downstream channel #0’ and a ‘downstream channel #1’.
Packet	Information sent through a graph is always sent in a unit called a ‘packet’.  In LogMod5 logging, packets are always bundles of one or more request log entries.  In future releases, packets may contain other kinds of information as well
Signal	When working with the internals of LogMod5 nodes, you may hear about ‘signals’.  A ‘signal’ is sent by nodes to their immediate upstream and downstream neighbours to convey some simple information about that node’s status.  Signals are not described in this document, but the defined list of signals currently is the following four: START signal, STOP signal, DATA signal, and DONE signal.
Partial processing	Since some packets (most packets) will contain multiple log entries, it is possible for a packet to be ‘partially processed’.  This means that some of a packet’s entries have been successfully processed (sent to a remote server, for example) and some were not (because there was an error after the first few entries were sent).  Partial processing allows other nodes to takeover processing of a packet’s log entries by only handling the entries that have not been taken care of already.  This system ensures that all entries get processed once, with no duplication of entries and no loss of entries.

Appendix 1: Node Definitions

This section gives detailed description of each node and provides important details on fault handling, configuration requirements and parameters, and all other details necessary for using each type of node.  It is critical that any person configuring a LogMod5 graph, read the relevant parts of this section for the graph they are trying to configure.

Knowing the detailed node definitions is the last piece of the puzzle before being fully qualified to configure and use logmod5.

For a description of each of these options, refer to standard nsd.conf documentation.  These settings have the same meaning as they do in the logger_local module except that each setting applies only to this instance of the node.  This means you can have different nodes each logging a different file format, or with different max file sizes (for example).

Disk Writer NodeName

Declare as:

lm5_diskwriter <NAME>

 

This node writes log entries to the disk.  This is basically a copy of the standard log writing facilities and is configured in a very similar way.  Settings must be provided preceded by a node name, e.g. “NAME->disabled yes”.  Set it up with the following:

NODENAME->disabled

NODENAME->max_disk_usage_size

NODENAME->max_file_size

NODENAME->max_file_count

NODENAME->max_file_time_range

NODENAME->max_file_age

NODENAME->flush_interval_sec

NODENAME->logfile_type

NODENAME->log_peeraddress_enabled

NODENAME->log_identifier_enabled

NODENAME->max_time_gap

NODENAME->ip_type

 

Logging configurations should be configured once at deployment time. Log files might be orphaned in the event that configuration changes are made to a production system. Whenever modifying the logging configuration, you must be certain that old log files are renamed or manually removed.

There will be log files orphaned if the disk writer node name is modified, or the log file type is modified.

Please consult with your Netsweeper Systems Engineer, prior to altering logging configurations for production systems.

NAME->disabled {true|false}

    This will disable the entire disk writer.

Default: NAME->disabled false

NAME->max_disk_usage_size SIZE

This sets the maximum amount of space the logger is allowed to use in total. This setting will override the file count setting if it is smaller than the     file_count * file_size combination. This allows a hard limit of total disk usage the logger is allowed to use. The size can also contain unit information, such as megabytes (1000 * 1000 bytes),Kilobytes (1024 bytes), etc. For example, the default value of 5368709120 is equivalent to 5 gigabytes.

It is important to remember this sets the size of the log files *.log and does not include the size of the index files, *.ndx and *.rep. Generally, an index file is around 10% of the log file size. So if you set this to 1gb your disk usage may be 1.1gb withthe included index file.

It is recommended you never fully allocate your storage, you should always have more then 10% free space. A example for a Netsweeper system maybe the following storage allocations:

• 20% reserved for Logs and Log files (/usr/local/netsweeper/logs)

• 20% reserved for generated Report output (/usr/local/netsweeper/reports)

• 40% reserved for temporary report files (/usr/local/netsweeper/reports)

Default:

NAME->max_disk_usage_size 5368709120

NAME->max_file_size SIZE

This sets the size of a single log file. We recommend the default setting of 1gbit is used.  This allows for the log files to be copied to any system without running into size limitation and other issues.

Default:

NAME->max_file_size 1Gb

NAME->max_file_count NUMBER

This sets the maximum number of local log files we are allowed to have.  This setting can be set to -1 or 0 to indicate that an unlimited number of files are allowed as long as the max_file_size and max_disk_usage_size are properly maintained.

Default:

NAME->max_file_count 0

NAME->max_file_time_range SECOND

This sets the maximum data/time range (difference between the latest records and the first record timestamps) in a single file. Zero value means unlimited range.

Default:

NAME->max_file_time_range 0

NAME->max_file_age SECOND

This sets the maximum age (the time since the latest record timestamp) of a file. If it is exceeded the server removes the file at log files rotation. Zero value means unlimited age.

Default:

NAME->max_file_age 0

NAME->flush_interval_sec SECOND

This is the number of seconds we flush the local logs to the disk. By default this is set to 1 second, however, this can be increased on a heavily utilized system to possibly increase performance.

Default:

NAME->flush_interval_sec 1sec

NAME->logfile_type NUMBER

This configured the format of the log file we wish to store for local logs.

This can be the following formats

1. Binary Mode

2. Text Mode

3. Extended Binary Mode

Default:

NAME->logfile_type 3

NAME->logger_fields field1 field2 field3 ...

This allows you to specify the log record fields that should be written into the local Extended Binary or Text log file. It cannot be used with old Binary log file that has fixed list of fields. The date/time field is mandatory and is always written into the file. All other fields can be added or removed from the log record.     This setting overrides the global logger_fields setting for the current writer.

For available field names see global logger_fields setting description.

Default (if neither NAME->logger_fields nor global logger_fields is specified):

Text format log:

    NAME->logger_fields url ip user group denied category

Extended Binary log:

    NAME->logger_fields user group category denied url ip dst

NAME->log_peeraddress_enabled {true|false} (Obsolete)

This allows the peer connection address for a policy request to be logged into the local Extended Binary Log file. The setting is obsolete, use logger_fields instead

Default:

NAME->log_peeraddress_enabled false

NAME->log_identifier_enabled {true|false} (Obsolete)

This will allow logging the logger_identifier into the Local Log file. The logger identifier indicates which policy server processed the request. This is useful for debugging purposes to find out which policy server processed the request. This can also be used for generating load balancing reports to see what policy server is processing the most data. The setting is obsolete, use logger_fields instead

Default:

NAME->log_identifier_enabled false

NAME->max_time_gap seconds

If the policy server finds request log files with a size that is less than the max_file_size limit, the policy server can append new log records to these request log files. It can cause a problem for log file management if a request is appended to a file that has older records.

This setting limits the "age" of the file that can be appended to.  If the time interval between the file's latest log record and the current time is larger than the "max time gap" the policy server ignores this file and searches for another request log file.

Default: NAME->max_time_gap 24h

NAME->ip_type

This setting makes sense only for logfile_type 3 (Extended Binary Mode) and allows you to choose IP address format.

The value can be:

1. 1 - 4 bytes that is enough for IPv4 addresses but cannot keep IPv6

2. 2 - 16 bytes, it can keep IPv6 addresses and IPv6 as embedded into IPv6. Use it if you filter mostly IPv6 traffic.

3. 3 - variable size (5 or 17 bytes) where the first byte keeps the address type and the rest is the IP address itself. This is the best type for mixed IPv4 and IPv6 traffic.

Default:

NAME->ip_type 3

This node will produce log files with names containing the node name, so that multiple nodes can all log into the same directory (usually the /usr/local/netsweeper/logs/ directory).

This node requires >= 1 upstream neighbors and must be an endpoint (can't have downstream neighbors)

Disk Queue

Declare as:

lm5_disk_queue <NAME>

 

This node is used to temporarily preserve log entries on disk whenever its downstream node goes temporarily offline. This is useful if placed upstream from a TCP node which can occasionally go offline for a short time during network issues or momentary server outages. Other kinds of node may go offline as well, and this node can connect to them as well. For example, it may make sense in some situations to place one of these nodes upstream from a failover node. Failover nodes will go temporarily offline whenever all of their downstream neighbors are offline. The mirror node can go offline in a similar manner.

Here are the configurable settings for the disk queue:

NAME->file_path

This sets the location where the disk queue file(s) will be written. This setting is required.

NAME->file_prefix

This sets the name of the disk queue file(s) on disk. Additional characters may be appended to this. This setting is required.

NAME->disk_write_buffer

This sets the size of the memory buffer where data is buffered before being written to disk.

Default:

NAME->disk_write_buffer 32M

NAME->disk_read_buffer

This sets the size of the memory buffer where data is buffered after being read from disk.

Default:

NAME->disk_read_buffer 32M

NAME->max_disk_size

This sets the maximum possible size on disk for the file(s) on disk of this node.

Default:

NAME->max_disk_size 1T

NAME->max_disk_size_thresh

This sets the point short of a full disk where this node will stop accepting more entries. The value should be equal or larger than disk_write_buffer. This ensure that the disk_queue can inform upstream

of the pending size limit and still accept a full disk_write_buffer which has already sent on its way.

Default:

NAME->max_disk_size_thresh 100K

NAME->num_files

This sets the number of files on disk this queue will use. These file(s) will all share the max_disk_size setting (if max_disk_size is set to 1T and num_files is set to 5, then each file will be no larger than 0.2T). A number between 2 and 10 is usually sufficient.

Default:

NAME->num_files 2

This node requires >= upstream neighbors and can have only one direct connection downstream (i.e. must have exactly one downstream neighbor).

TCP Client

Declare as:

lm5_tcpclient <NAME>

 

The TCP client node transmits log entries to a remote server.  This node takes the log entries it receives from upstream and transmits them to the configured port and remote host.  This node carefully handles situations where:

• Server is temporarily unreachable

• Connection is congested

• Connection hangs (we send more data but it never gets acknowledged)

• Server disconnects our connection

• And other TCP situations

Most of the time, these circumstances are handled by the node entering node failure mode and allowing upstream nodes to react appropriately (usually either queuing up log entries or failing over to a different node).  This node will continuously attempt to make its connection, and will attempt to reconnect automatically any time the connection is lost.

Object	Type	Default	Description
block_on_full_buffer	Boolean	No	If the transmit buffer becomes full then it means either the logging server is currently overloaded or there is network congestion.  If you set this to block then the logging system will wait for the connection to clear up.  If you set this to not block, then the connection will go temporarily offline and entries will need to be queued or failed-over or something upstream.
block_timeout_msec	Time		If non-zero, then blocking mode will actually have a timeout point at which the send will unblock and will attempt to retransmit
block_timeout_attempts	Number		If a send block times out then a new send is attempted.  This setting controls how many attempts will be made before the packet is marked as a packet failure.
reconnect_delay_sec	Time		Number of seconds pause between attempts to connect to a remote server that is unavailable
use_keepalive	Boolean		Send TCP keepalives to keep connection alive and monitor connection health even if no traffic is flowing
keepalive_count	Number		Disconnect automatically if this number of keepalive packets are sent without ever receiving an acknowledgement
keepalive_idle	Time		Number of seconds the connection must be idle before keepalives will start to be sent
peer_stall_timeout	TIme		A peer will be counted as ‘stalled’ if it fails to read any data from our TCP stack’s send buffer (the one in the operating system stack) after this many seconds has elapsed
peer_stall_count	Number		the number of times a peer can be detected as stalled before it is automatically disconnected and a new connection attempt will be made (in the meantime, the node will be marked as node failure)
send_buffer_len	Size	5M	The number of bytes to allocate for sending log entries.  A larger value will tend to make performance a little better.
remote_logger_port	Number	None	The port on which to connect to the remote server.  This entry is required and has no default value.
remote_logger_host	String	None	The IP (or hostname) on which to connect to the remote server.  This entry is required and has no default value.

This complex node has many chances for entering a failure state.

Failure Type	Description
Packet Failures	Packet failure will generally occur if there was some reason packet could not be sent.  For example, if there is persistent congestion.
Node Failures	Node failure on the tcp node happens whenever the tcp node believes it can no longer work effectively.  This is usually either when it detects congestion or when the connection is lost or unavailable.

This node requires >= 1 upstream neighbors and must be an endpoint (having downstream neighbors will raise an error).

Option	Minimum	Maximum
Legal number of upstream neighbors	1	∞
Legal number of downstream neighbors	0	0

Failover

Declare as:

lm5_failover <NAME>

 

The failover node requires no configuration.  When placed in the graph it is used to route log entries to the highest priority, available, downstream neighbor.  Priority is determined by the order in which downstream neighbors are added - the highest priority neighbor is the one added first.  If the highest priority neighbor ever becomes unavailable, then the next highest priority will be used.  If the higher priority neighbor comes back online, then the failover will switch back again.

The failover also automatically handles temporary failures.  If there is a single log entry that fails to be processed for some reason, the failover node automatically makes another attempt at the next highest priority downstream neighbor for that single packet.

This node requires no configuration.

Failure Type	Description
Packet Failures	Packet failure is reported whenever all downstream channels are either unavailable or report packet failure on the packet.
Node Failures	Failover will enter failure mode when all downstream nodes are non-functional.

 

The failover node can have multiple upstream neighbors and requires at least one downstream neighbor (two if you want the node to make any sense).

	Minimum	Maximum
Legal number of upstream neighbors	1	∞
Legal number of downstream neighbors	1	∞

DevNULL node

Declare as:

lm5_devnull <NAME>

 

The devnull node can be used wherever an endpoint is desired which will not actually write any logs or transmit any network packets.  This will rarely be useful.

This node requires no configuration.

Failure Type	Description
Packet Failures	This node has no packet failure mode, this node will always report SUCCESS.
Node Failures	This node has no node failure mode, this node will always report as operational.

 

This node requires >= 1 upstream neighbors and is an endpoint (cannot have downstream neighbors).

	Minimum	Maximum
Legal number of upstream neighbors	1	∞
Legal number of downstream neighbors	-	0

 

Mirror node

Declare as:

lm5_mirror <NAME>

 

The mirror node copies log entries to multiple downstream neighbors.  This node is useful for making multiple copies of log entries (possibly on two different remote servers, or even on two different logs files on the same computer).

If a downstream neighbor reports as offline, then it will not get a copy of log entries for the period that it is offline.

The mirror node has only one configuration option:

	Type	Description
failure_mode	String	Three possible settings: LM5_MIRROR_ERR_STOP - use this setting to cause the mirror to stop mirroring the log entry as soon as an error occurs (additional channels will not get a copy).  report an error upstream LM5_MIRROR_ERR_FAIL - use this setting to cause mirror to continue mirroring if there's an error.  report an error upstream at the end LM5_MIRROR_ERR_SUCCESS - this setting causes mirror to keep sending on error and does not report the error upstream as long as we had at least one success

 

Packet Failures	Packet failure depends on the failure mode setting
Node Failures	Node failure happens if all downstream channels become offline.

 

This node requires >= 1 upstream and >= 1 downstream nodes.

	Minimum	Maximum
Legal number of upstream neighbors	1	∞
Legal number of downstream neighbors	1	∞

loadbalance (by size)

Declare as:

lm5_lb_size <NAME>

 

The bysize loadbalancer takes incoming packets and sends them to one of the downstream channels.  It keeps track of the approximate size of each entry passed downstream and tries to keep all downstream nodes equal in terms of amount of work they have successfully processed.

If one downstream node reports an error on one packet (packet failure) then the packet is sent to the next node in line.

This node has no configurable options.

Packet Failures	Packet failures will be reported if a packet arrives which cannot be processed by any of the downstream nodes.
Node Failures	This node will enter failure mode as soon as any of its downstream nodes is offline

 

This node can have more than one upstream connection.  It expects to have at least one downstream node.

	Minimum	Maximum
Legal number of upstream neighbors	1	∞
Legal number of downstream neighbors	1	∞

loadbalance (roundrobin)

Declare as:

lm5_lb_rr <NAME>

 

The roundrobin loadbalancer takes incoming packets and sends them to one of the downstream channels.  Each time a new packet comes in it is routed to the next downstream channel in line.  Packets are distributed roundrobin to each channel.

Downstream packet failures are handled by simply routing the packet to the next channel in line.

NOTE: packets containing bundles of log entries may vary GREATLY in size.  Some packets may contain only one entry while others contain hundreds.  RR distribution of packets does not guarantee RR distribution of individual log entries.  The roundrobin distribution loadbalancer is provided for special and probably unusual circumstances.  The balancer by-size will generally be the loadbalancer you want to use.

This node has no configurable options.

Packet Failures	Packet failure occurs if all downstream channels also report packet failure on a packet.
Node Failures	Node enters failure mode if any one downstream node enters failure mode.

 

This node requires at least one upstream neighbor and at least one downstream neighbor.

	Minimum	Maximum
Legal number of upstream neighbors	1	∞
Legal number of downstream neighbors	1	∞

Timer node

Declare as:

lm5_timer <NAME>

 

Use this node to schedule events in the LogMod5 graph.  The timer node does this by reporting node failure upstream any time it is outside of its scheduled window of operation.  This effectively prevents any packets from reaching all the nodes downstream from the timer node.

Using this node, you can, for example, cause log entries to be transmitted to a remote server late at night when bandwidth is available but not queued and saved for later during the day time.  To achieve this, you would position a disk queue node directly above the timer and the tcp transmitter below the timer.  In this way, when the timer was ‘off’ and reporting node failure, the disk queue above will queue packets.  When the timer becomes ‘on’ again, stops reporting node failure, the disk queue will release all the stored packets (in order).

This node has one configurable option:

	Type	Description
schedule	String	This is the schedule string, specifying start time and end time.  For example: 10:00 16:00 Will go online from 10am to 4pm. 00:01 06:00 Transfer logs between midnight and 6AM 18:00 04:00 Transfer logs between 6:00PM and 4:00AM This argument is required.  There is no default setting.

 

Packet Failures	This node will report packet failure if the downstream node reports failure.  It will also report failure if packets are sent to it when it is in node failure mode.
Node Failures	This node will report node failure if 1) it is outside its schedule window of operation or 2) if its downstream node reports node failure

The timer node can have multiple upstream neighbors but must have exactly one downstream neighbor.

	Minimum	Maximum
Legal number of upstream neighbors	1	∞
Legal number of downstream neighbors	1	1

Group Hash node

Declare as:

lm5_group_hash <NAME>

 

This node diverts log entries to a different downstream neighbor, depending on the contents of the "group" field in the log entry.  Because the function is a hash, it is not easy to predict ahead of time which downstream neighbor will get log entries with a particular group, but it is guaranteed that all occurrences of that particular group will be found in the same downstream neighbor. 

This hashing enables optimizations in reporting, making generation of certain group-related reports much faster.  Specifically, it allows the optimization of reports which group by the group field or which search for a specific group.

Packet Failures	Packet failure is reported if the downstream channel chosen by the hash reports packet failure.
Node Failures	This node reports node failure if any one of its downstream nodes reports node failure.

This node must have at least one upstream neighbor.  It must also have at least one downstream neighbor.  IMPORTANT: the number of downstream neighbors must be a power of 2 (eg: 1,2,4,8,16,32, etc.)  This requirement is in place for efficient and fast hashing across the downstream nodes.

	Minimum	Maximum
Legal number of upstream neighbors	1	∞
Legal number of downstream neighbors	1	∞

IP Hash node

Declare as:

lm5_ip_hash <NAME>

 

Refer to the description for the lm5_group_hash.  This node hashes by client IP, and is otherwise identical to the group hash node.

Username Hash node

Declare as:

lm5_username_hash <NAME>

 

Refer to the description for the lm5_group_hash.  This node hashes by client username, and is otherwise identical to the group hash node.

Denied Hash node

Declare as:

lm5_denied_hash <NAME>

 

Refer to the description for the lm5_group_hash.  This node hashes by whether or not the request was allowed or denied, and is otherwise identical to the group hash node.

The denied hash requires exactly two downstream neighbors.

logmod5 Hash nodes

Such nodes divert log entries to a different downstream neighbor, depending on the contents of the particular field in the log entry.  Because the function is a hash, it is not easy to predict which downstream neighbor will get log entries with a particular field value, BUT it is guaranteed that all occurrences of the same field value will be found in the same downstream neighbor.  This hashing enables optimizations in reporting, making generation of certain reports much faster.

All such nodes require >=1 upstream neighbors and >=1 downstream neighbors.

IMPORTANT: the number of downstream neighbors must be a power of 2

(e.g.: 2,4,8,16,32, etc.)

Since the logmod5 configuration with such node requires specification of multiple downstream nodes, it is not recommended to use them directly ( at least for simple cases. The logmod5 framework provides a "Hash Combo" node (see below) that is a macro automatically expanded to the specified hash node, required number of disk writer nodes and all needed links.

Group Hash Node

lm5_group_hash <NAME>

This node hashes by the policy group.

IP Hash node: lm5_ip_hash <NAME>

This node hashes by client IP.

Username Hash node

lm5_username_hash <NAME>

This node hashes by client username.

Denied Hash node:

lm5_denied_hash <NAME>

This node hashes by whether or not the request was allowed or denied.

Since this field has only two possible values the hash size greater than 2 doesn't make sense.

URL Hash node: lm5_url_hash <NAME>

This node hashes by requested URL.

Policy Event Type node:

lm5_type_hash <NAME>

This node hashes by Policy Event Type. Since this field has a few fixed values the hash size much greater than the number of known types (see the policy_event setting description) doesn't make sense.

logmod5 Filter Nodes

Such node diverts log entries based on whether they match or don't match a configured filter.  If the node has one downstream neighbor, then it is assumed that that downstream neighbor should receive all entries matching the filter (entries not matching the filter are simply thrown away).

If the node has two downstream neighbors then matching entries are sent to the higher priority (first neighbor) and the non-matching entries are sent to the other neighbor.

The filter node can have >= 1 upstream neighbors and == 1 or == 2 downstream neighbors.

This node requires one configuration item: NAME->filter <string>

The string the node will filter by.

Group filter node: lm5_group_filter <NAME>

This node filters by the Policy Group.

The filter for this node must be a group name.

IP Filter Node

lm5_ip_filter <NAME>

This node filters by client IP.

The filter text for this node must be an IPv4 address in dot notation.

Username filter node: lm5_username_filter <NAME>

This node filters by the client username.

The filter for this node must be a client username.

Denied Filter Node

lm5_denied_filter <NAME>

This node filters by whether or not the request was allowed or denied.

The filter string for this node should be one of two words: either "allowed" or "denied".

URL Filter Node:

lm5_url_filter <NAME>

This node filters by requested URL.

The filter string should be the beginning part of a URL.  The URL will be compared only up to the end of what is provided to the filter and all remaining chars in the log entry are ignored.  For example, the filter: "http://www.google.com/" will match the entry "http://www.google.com/q=search"

Policy Event Type Filter node:

lm5_type_filter <NAME>

This node filters by the Policy Event Type.

The filter for this node must be a Policy Event type name.

Category Filter node:

lm5_category_filter <NAME>

This node filters by Category. The filter for this node must be a Category number set, and must also have a filter set operation type that indicates how the category filter is to be applied.

Category filter format:

NAME->filter <string>

<string> is a comma-delimited list of category numbers

NAME->filter_set_operation <operation>

<operation> options for the filter_set_operation include:

set_match       - set has at least one match in the filter set (intersect)

set_no_match    - set does not intersect the filter set

set_exact_match - set is identical to the filter set

set_subset      - set is a subset of the filter set

set_superset    - set contains all of the filter set (but may contain more)

Hash Combo node

Declare as:

lm5_hash_macro <NAME>

 

This node combines the functionality of all the hash nodes and a configurable number of disk writer nodes.  This makes it easy to declare that you want to hash log entries into several log files.  If you don’t use the 'hash macro' node then you will have to individually declare every disk node.  For a hash of 32, 64, or 128 buckets (for instance) this will require a very long nsd.conf configuration file.  The hash macro makes this much easier by allowing very large hash configurations to be declared in just a few lines.

This node (which is actually several nodes all in one) will create several log files and distribute log entries into log files based on a hash function.

This node requires two configuration items:

	Type	Description
hash_type	String	Enter the type of hash desired, this should be one of: group ip denied url username type
hash_size	Number	How many different 'directions' or 'buckets' should log entries be hashed into?  This will determine how many different log files will be produced.  This number MUST be a power of 2 (2,4,8,16,32,64,128, etc).

Optional Nodes Independent of the Hash Type

Also, the node can have optional settings independent of the hash type:

NAME->mask_size NUMBER

If the hash_type is "ip", this setting allows you to calculate hash using only the network part (set with this mask) of the IP address. It means that requests from all workstations within this network will be logged into the same partial file.  By default the whole IP address is used, i.e. mask size is 32 bits for IPv4 and 128 bits for IPv6.

NAME->substring_offset NUMBER

NAME->substring_length NUMBER

If the hash type is "group" or "username", these setting allow you to calculate hash using only substring of the complete group or user name. It allows you to log requests whose group or user name have common parts (e.g. school1-students, school1-teachers) in the same file.

The offset can be a negative number. In this case it is counted from the end of the string. If offset is not specified it is 0. If the length is not specified or equal zero it means the substring from the offset to the end of the string. If both settings are not specified or equal to zero, the whole name is used for hash.

NAME->substring_offset NUMBER

NAME->substring_length NUMBER

If the hash type is 'group' or "username", these settings allow to calculate hash using only substring of the complete group or user name. It allows you to log request whose group or user name have common part (e.g. school1-students, school1-teachers) into the same file.  The offset can be a negative number. In this case it is counted from the end of the string.  If offset is not specified it is 0. If the length is not specified or equal zero it means the substring from the offset to the end of the string. If both settings are not specified or equal zero, the whole name is used for hash.

NAME->diskwriter->

It allows you to specify disk writer settings for all disk writer nodes created by the macro node.  For the complete list of settings see lm5_diskwriter node settings.

(Obsolete settings are not supported here.)

Packet Failures	This node has no packet failure mode, this node will always report SUCCESS.
Node Failures	This node has no node failure mode, this node will always report as operational.

 

The hash combo node must have exactly one upstream neighbor and cannot have any downstream neighbors.

	Minimum	Maximum
Legal number of upstream neighbors	1	1
Legal number of downstream neighbors	0	0

 

Logmod5 MultiWriter Combo node

Declare as:  lm5_multiwriter_macro <NAME>

Group Segmented Logging

Group Segmented Logging allows you to segment your log files by Group or by a specific tag.  This makes it easier to manage and search through request log archives for specific logs.

multiwriter_macro Node

The LogMod5 'multiwriter_macro' node that creates a log writer for each group.

Declare as: lm5_multiwriter_macro <NAME>

 

This kind of nodes is similar hash nodes but instead of numbered buckets it creates a separate disk writer for each value of the specified field. E.g. the Group Mutiwriter reads the list of all policy groups at policy server start and creates a separate disk writer node for each group. Then each group requests are written into the designated file.

This node can use a "log file tag" instead of the group. This tag can be set in the group properties and allows to log requests for multiple groups shared the same tag to the same file. If the log file tag is used for file selection but a group doesn't have the tag specified, its request have the default tag "-".

Also this kind of nodes can be used for Policy Event Types.

 

NAME->logger_field {group|tag|type}

It specifies if the group name or the log file tag or the Policy Event Type should be used for file selection.

 

NAME->diskwriter->...

It allows you to specify disk writer settings for all disk writer nodes created by the macro node.

For the complete list of settings see lm5_diskwriter node settings.

   (Obsolete settings are not supported here.)

 

The multiwriter combo node must have exactly one upstream neighbor and cannot have any downstream neighbors.

 

logmod5 Default Configuration

lm5_entry entry

lm5_diskwriter disk

logger_logmod5_connect entry disk

 

See the Logger Use Cases.