loggertest - Netsweeper Logger Test Utility

The loggertest utility composes and sends the network message for requests to the remote logging system. It can be used for logging system testing, test log files preparation for reporter testing and as an interface for other scripts and programs that need to save their own logs into Request Logs.

Usage: loggertest [-s <server>] [-i <clientip>] [-u <url>] [-n <username>] [-g <groupname>] [-c category]

Arguments:

Option

Description

-s <server>

The IP of the server

-p <protocol>

The version of logger protocol (1, 2 - default)

-q

Silent Mode

-i, --ip <clientip>

The IP of the client

-w, --dest-ip <destip>

The IP of the request destination

-h, --inter-ip <interip>

The IP of the interceptor host

-t <timestamp>

The timestamp (Unix time); if is not specified the test does not send the timestamp at all

-n <username>

The user name

-g <groupname>

The group name

-x, --policy <name>

The policy name

-z, --tag <logfiletag>

The log file tag

-a, --all-cats <category>

All categories code (can be a comma separated list)

-c, --deny-cats <category>

Denied categories code (can be a comma separated list)

-d

Set 'denied flag'

-u <url>

The URL.

-v, --server-id <serverid>

The logging server ID

-m, --http-method <number>

The HTTP method code

-A, --user-agent <name>

The User-Agent header string

-y, --referrer <url>

The Referrer header string

-T, --type <type>

The policy event type (name or number)

-D, --data <data>

The policy event data

-M, --module <name>

The module name

-W, --workstation <name>

The workstation name

-S, --screenshot <data>

The screenshot

-l

Loop the request

-f <file>

Perform a request on line terminated urls in the specified file. The file of '-' will read input from STDIN

-j

Parse the input in JSON format.

-r <fields>

The format for the line read from the input file.

-R, --rate <number>

The number of requests sent per second (for send from file or in loop)

-o, --offset <datetime>

The shifted start time for records read from a file

The single record composed from command line arguments can be repeated in the loop.

Log Record Fields

Most of command line arguments are log record field specification:

Log record field

Command line option

The file record field or JSON field name (see below)

Date and Time (Unix timestamp)

-t <timestamp>

t

Request URL

-u <url>

u

Client computer IP address

-i <address> or --ip <address>

i

Destination server IP address

-w <address> or --dest-ip <address>

w

Client name

-n <name>

n

Group name

-g <name>

g

Policy name

-x <name> or --policy <name>

x

Denied category list

-c <categories> or --deny-cats <categories>

c

All categories list

-a <categories> or --all-cats <categories>

a

Denied flag

-d (if it should be set) or nothing

d

Interceptor IP address

-h <address> or --inter-ip <address>

h

Policy server ID

-v <id> or --server-id <id>

v

Log file Tag

-z <name> or --tag <name>

z

HTTP request method

-m <number> or --http-method <number>

m

HTTP request User-Agent header

-A <name> or --user-agent <name>

A

HTTP request Referrer header

-y <name> or --referrer <url>

y

Policy event Type

-T <name or number> or --type <name or number>

T

Policy event Data

-D <data> or --data <data>

D

External module name

-M <name> or --module <name>

M

Client workstation name

-W <name> or --workstation <name>

W

Screenshot

-S <encoded image> or --screenshot <encoded image>

S

Input Data Reading Options

Description

Command line option

Input file

-f <name>

Read from the standard input

-f -

Input file format (log fields list)

-r <fields>

JSON input

-j

The input format "fields" is a solid string of letters where each letter defines a log record field in the input file (see last column in the table for log record fields). e.g. if the file line includes URL, client IP address and denied flag, like:

https://ibm.com 10.1.1.123 0
https://sex.com 10.1.1.123 1

and you are going to use all of them, the input file format should be '-r uid'. But you can also use only the first column or a few columns: '-r u' or '-r ui' (but there is no way to skip unneeded columns).

If the loggertest reads data from a file or standard input and the format is not specified, it is supposed that only first column makes sense and it is the URL, i.e. the default format is '-r u'.

The loggertest also recognizes input in the JSON format, like:

{ 'u' : 'https://ibm.com', 'i' : '10.1.1.123', 'd' : 0 }

The field names here should be the same one-letter field names as in the record format. But since the input line includes both names and values you do not need to specify the input record format but have to set the -j option.

And the record format can be used with the JSON input as a filter, i.e. you can specify only subsets of fields that you want to send for logging. In this case the order of fields does not matter and does not need to match the JSON fields order. e.g. for the example above the "fields filter" can be '-r iu' or '-r ui' or '-r du' and so on.

Log Sending Options

Description

Command line option

Remote logging server IP address

-s <address>

Logging protocol version (default: 2)

-p <number>

Silent mode

-q

Loop record sending

-l

Request rate

-R <number> or --rate <number>

Shift records timestamp

-o <date and time> or --offset <date and time>

Two first options are clear without detailed descriptions. The default and recommended protocol version is 2 because only this protocol allows the sending of all mentioned log record fields.

The silent mode means that the loggertest does not print a summary about each sent message.

The loop option allows you to repeat endlessly one single message to simulate logged traffic. It does not make sense for data read from a file.

The request rate allows you to slow down sending records read from a file or the single record repeated in the loop. The -R or --rate options expects the "request per second" number that internally is recalculated to the delay between request that provides the specified rate.

The number range is from 1 to 1,000,000,000 but actual upper limit is set by the hardware. If the range is not specified or the number is too big, the loggertest sends request as fast as possible (with no delays).

The Shift timestamp option can be useful if you need replay some data that already has timestamps.

If the input data doesn't have the timestamp or you use only command line arguments and don't specify the date/time, the loggertest sends the message without this filed and the logger server adds own local time timestamp when it receives the log record. But if each input record has the date/time, the loggertest and the remote server keep it and write it to the log file. In the second case the time intervals between records are preserved and if it was real traffic records for a hour, day or more the log file written by the logger server will cover the same time range. But in the first case timestamps and time intervals between records are not preserved and the traffic for a day can be written in a few seconds.

Sometimes it is needed to log existent records covering the same time range (hour, day, etc.) but as if they were made right now or at some chosen time. If you use them with original timestamps they will be written covering the same date/time range but with the same old date/time, if you don't use original timestamps the result will have new timestamps but compressed in the short time range.

The Shift timestamp option allows you to specify new start date/time for replayed records and solves the problem: new records keep intervals and cover the same date/time range but timestamps are recalculated and look like they are made at the specified time. The Shift records timestamp argument is the date and/or time for the first replayed records, other timestamps are calculated, respectively. If input records do not have timestamps (of you do not use them) this option does not make sense.

The date and/or time specification for the "Shift records timestamp" option

The option argument can be the word now or date and/or time specified in the format:

HH:MM                         (0-24 hours and minutes)

HH:MM:SS                   (0-24 hours, minutes and seconds)

mm/dd/yy                     (month, day and two digits of the year)

mm/dd/yy HH:MM

mm/dd/yy HH:MM:SS

mm/dd/yyyy                    (month, day and year)

mm/dd/yyyy HH:MM

mm/dd/yyyy HH:MM:SS

yyyy-mm-dd                    (yaer, month and day)

yyyy-mm-dd HH:MM

yyyy-mm-dd HH:MM:SS

If the argument includes both date and time they should be surrounded with quotes, e.g. "2020-01-01 12:00"

If the argument includes only time, the date is today

If the argument includes only date, the time is the current time of day the date and time can be any in the past or future