Netsweeper Categorization Overview
Category Name Server Processing
Product License Key Validation
This document describes the Netsweeper Category Name Server (CNS) communication: as a means of connecting with and using Netsweeper’s data center categorization engines to categorize web sites. It provides information about the Netsweeper CNS Network and a description of the CNS protocol, code samples, installation, and configuration instructions for a private CNS Server.
This information is intended for those using Netsweeper’s categorization engines to power their own application. For most, merely using the sample code will be enough to get started.
This section describes how Netsweeper categorizes Internet content and uses CNS in our own products. It introduces the architecture and the function of the Categorization Engines, the Master Database, the Master URL List, and the roles of the Category Name Server (CNS) and Master Category Name Server (MCNS).
The Netsweeper data center houses servers that categorize web sites on demand. A robust data management system supports the Categorization Engines to store and serve categorizations for URLs that have already been categorized. The management system is connected to end users through a network of servers (CNS Network) which consolidate categorization requests from end user servers and processes.
Requests for a categorization of a URL are made to a Category Name Server (CNS). The CNS categorizes the request if it can; otherwise it forwards the request to the Master CNS (MCNS). The MCNS categorizes the URL if possible; otherwise the URL is sent to the Categorization Engines. The Categorization Engines categorize the URL and write the result in to the MCNS Master database.
Many applications/appliances can feed into each CNS server and multiple CNS servers feed into the MCNS (the data center).
There are two types of Category Name Servers: a public CNS (which is a CNS that is hosted by Netsweeper and available to multiple customers) and a private CNS (which is a CNS that is dedicated to an individual customer). Both CNS deployments operate in the same manner, taking in categorization requests and processing them.
With a private CNS, you have the option of configuring and tailoring your CNS to manage traffic in a way that benefits your specific needs, for example, increasing cache size, forcing alternate categorizations on certain URLs, and traffic reporting. A public CNS uses a generic configuration with no customer-specific alterations or reporting.
When the CNS receives a categorization request, there are several processing steps which can be simplified to the following:
· Product license key validation
· URL parsing
· Category URL List lookup
· Master URL List lookup
· Local cache lookup
· MCNS request
There is a validation of the requesting service to ensure it is using a valid product license key. In the private CNS server, this validation can be disabled and customer servers, communication with the private CNS server can use any strings as a key. But these server’s keys should not be the same as the license key of the CNS server itself. This restriction prevents short loops in CNS Network configuration and the CNS server denies requests if they use its own license key.
Technically this is not a 'step', however, as part of the processing, the CNS applies the Netsweeper URL parsing rules before it categorizes a URL. You should understand these parsing rules and apply them, to some degree, before the URL is forwarded to the CNS.
The Category URL List is a list that is available only on private CNS deployments. This is the list you can edit in your private CNS and it allows you to force an alternate categorization on a URL. If an URL is found in the Category URL List, the CNS server stops the lookup and returns result to the requesting server.
This optional list provides a simple method to ignore the categorization that Netsweeper assigns to a site and to use the one of your choosing instead. In CNS deployments, this list is not widely used, but it provides an option for those situations where this is necessary.
The Master URL List is a specialized list of URLs that have been 'mastered' in that the categorization is forced by Netsweeper and is not dynamic or Categorization Engine based. It is derived from the Master CNS database but includes only most popular sites (usually it serves about 90% of requests).
The purpose of this list is twofold: first, for URLs (domains) that are very heavily used and considered static, it provides a method to minimize categorization traffic and second, it allows Netsweeper a rapid method to override a Categorization Engine categorization which appears to be incorrect.
In the first example, there are certain web sites (domains) that can be assigned to a specific category, without debate. For example, microsoft.com and everything under that domain is categorized as Technology; sex.com and everything under that domain is Pornography. In these situations, the URL can be categorized very rapidly using the Master URL List.
In the second example, there will be times that the Categorization Engines assign a category to a URL that is wrong. In these cases, the Master URL List provides Netsweeper's category review staff with a method to 'correct' the categorization while efforts are made to teach the Categorization Engines what is special about that URL and to recognize it and similar sites correctly in the future.
The Master URL List is updated on CNS machines several times a day.
Each CNS has a local cache of URLs and associated categorizations that have been provided by the upstream CNS server. The cache reflects the nature of the traffic of the applications and appliances feeding that CNS. A private CNS that is categorizing for an educational environment will reflect where the students and teachers are often going; a private CNS that is categorizing for a business clientele will reflect where the business users are going.
If the CNS is not able to categorize the URL, it passes the request on to the MCNS.
The Master CNS processes categorization requests in a manner similar to the CNS:
· Product license key validation (for CNS)
· URL parsing
· Master URL List
· Local cache
· Master Database
· Category Engine request
The MCNS hosts the source of the Master URL List and is always up to date. The Master Database holds all recently and historically categorized URLs. If the URL cannot be categorized by the MCNS, it passes the request to the Category Engine queue.
The Category Engines retrieve the URL and processes it. The Category Engine then writes the result and a Time To Live (TTL) for the categorization into the MCNS Master Database.
A request from an application through to the MCNS and back typically takes 500 ms. When a CNS makes a request to the MCNS, it will wait about 1 second for a response (configurable). If there is no response, a Network Timeout category is assigned.
If the MCNS is unable to categorize the request, it responds to the CNS with the category of 'New URL' and passes the URL to the Categorization Engine queue.
Within about 1 second, the Categorization Engines process the URL and writes the result to the Master Database. Once categorized, the ruling is available to all incoming requests.
Each categorization is assigned a TTL (Time To Live). The purpose of the TTL is to signal when the categorization has become 'stale' and the URL should be revisited by the Categorization Engines.
The expiry of the TTL does not itself trigger the Categorization Engines, rather it is the next request for categorization of that URL after the TTL has expired that triggers the engines. This avoids the engines refreshing categorizations for URLs that are not being requested.
When a URL is received, it is parsed based on a variety of rules. The CNS can process any URL that is compliant with RFC1738. But it applies a few of its own rules to the URL format.
The complete URL format is:
scheme://username:password@host:port/path?query#fragment
· The CNS server ignores the ‘username:password@’ and ‘#fragment’ parts if they are present. It is recommended to remove them before sending a URL to the CNS.
· The CNS server can process the ‘?query’ but it is recommended to strip it because it can include security sensitive user information or it can make unwanted changes in the HTTP server content (e.g. send a message or change user preferences) if the Categorization Engine tries to retrieve this URL.
· The minimal URL should include the ‘scheme’ and at least one segment of the host name:
scheme://host/
· Otherwise it will be categorized as Malformed URL and not processed.
· The URL host should not have empty segments, e.g. http://company..com or http://.company.com or http:/// . All such URLs will be categorized as Malformed URL without further processing.
· It is also recommended to omit the port part if it is standard: 80 for http and 443 for https.
The CNS Protocol is used to communicate between your application and a public or private CNS server. (See the following chapter for more information on setting up and configuring a private CNS Server.) The CNS Protocol defines how to request a categorization and what results to expect.
This document assumes that you are reasonably experienced in socket programming. We provide sample code to show you how to connect to the CNS, how to send a request and receive a response, and an example on how to parse the response data.
The Netsweeper Categorization Engines are the back-end system that drives Netsweeper's Internet monitoring and filtering solution. Unlike most filtering products, Netsweeper does not depend on daily or weekly list updates. Instead, it provides categorization of new URLs on demand. The Categorization Service enables a Netsweeper Policy Server, or any CNS client implementation, to request a category assignment for a URL.
Netsweeper does not 'crawl' the Internet categorizing sites. Rather it categorizes sites where people are going, when the first person (globally) goes to the site.
Sites need only to be categorized once. Once categorized, the category for the URL is served from the Master Database on the MCNS. All categorizations have a time to live assigned so that 'old' categorizations can be updated as necessary.
Your request to the CNS Network consists of the protocol version (from 101 to 104), your product license key and the URL you wish categorized. The 104 version protocol optionally includes the destination server IP address, the HTTP request referrer URL and the HTTP method.
Since the 103 version request and response include the unique ID number, the CNS server doesn’t use the request ID but returns it with the response. It helps your application identify the response.
The response from the CNS will be one or more categories, the TTL, and a response code.
Your product license key is tied to the external IP address you are making requests from.
Your application may communicate with the CNS through multiple simultaneous connections. Do not exceed 60 simultaneous connections per product license key.
It is important to note that not all requests should be sent to the CNS network as not all requests can be categorized by the Categorization Engines. Requests to private IP addresses, private domain names, or files with certain file extensions cannot be analyzed by the categorization engines. To improve performance, you should not forward these requests to the CNS.
If a request has a file extension that cannot be categorized, the CNS will return either the category 'No Text' or 'Image'—depending on the file extension. The file extension list is not customer modifiable.
If a request is made to a private (not publicly routable) IP network, as defined by the Internet Assigned Numbers Authority (IANA), the CNS assigns the category of Intranet Server. The following are private IP addresses that you should not send to the CNS network.
· 192.168.0.0 – 192.168.255.255
· 10.0.0.0 – 10.255.255.255
· 172.16.0.0 – 172.31.255.255
· 169.254.0.0 – 169.254.255.255
· 127.0.0.0 – 127.255.255.255
If the web server requested does not have a public top level domain (TLD) name, the CNS assigns the categories of ‘Intranet Server’. This categorization is based on the list of TLDs published by the IANA. Valid TLDs are listed at this site:
http://data.iana.org/TLD/tlds-alpha-by-domain.txt
For example, if your internal network has the domain '.school001', a request for
http://schedule.school001 would be categorized as 243 (Intranet Server).
Protocol Version |
First Release |
Comments |
101 |
before 2.6.x |
Obsolete, because it doesn’t support category code > 255 |
102 |
before 2.6.x |
Obsolete, because it doesn’t support category code > 255 |
103 |
3.1.0 |
|
104 |
4.0.3 |
Extends 103 with optional data fields |
All protocol versions have the same result codes:
· 0 – Processing Error
· 1 – Success
· 2 – No error but the Category is not assigned yet
3 – Invalid License Key
· 6 – Success, Master List URL. It is the same as 1 but also indicates that the URL is permanently categorized in the Netsweeper Database. Such URL usually has 1 year TTL.
Size (bytes) |
Data |
Comments |
3 |
Version |
It is 1, 0, 1 for this protocol. |
16 |
License Key |
The key string. If it is shorter than 16 bytes it is padded with 0. |
variable |
URL |
Zero byte terminated string. |
Size (bytes) |
Data |
Comments |
3 |
Version |
It is 1, 0, 1 for this protocol. |
255 |
Categories |
In this obsolete version category codes had range 0 – 255. Therefore, each byte could contain a single category. |
2 |
TTL |
The 16-bits number of seconds for Time To Live. |
1 |
Result |
See above for result codes. |
This is like 101 but the TTL size is increased and TTL became negative for responses that don’t have category defined and should not be cached in the CNS client.
Size (bytes) |
Data |
Comments |
3 |
Version |
It is 1, 0, 2 for this protocol. |
16 |
License Key |
The key string. If it is shorter than 16 bytes it is padded with 0. |
variable |
URL |
Zero byte terminated string. |
Size (bytes) |
Data |
Comments |
3 |
Version |
It is 1, 0, 2 for this protocol. |
255 |
Categories |
In this obsolete version category codes had range 0 – 255. Therefore, each byte could contain a single category. |
2 |
TTL |
The 32-bits number of seconds for Time To Live |
1 |
Result |
See above for result codes |
This protocol has significant changes since 102:
· The response has the License Key for check.
· The License Key is variable length newline (“\n”) terminated string.
· The request has a unique ID. The CNS server does not use it but returns with response for response identification.
· The URL is newline (instead of zero byte) terminated.
· The Categories is the text string of comma separated numbers. There is no range restriction anymore.
· The whole request or response is terminated with an empty line (a single newline byte).
· The response doesn’t include TTL and category field in the case of error.
Size (bytes) |
Data |
Comments |
3 |
Version |
It is 1, 0, 3 for this protocol. Note that these bytes are not ASCII encoded digits but bytes with values 1, 0, 3. |
variable |
License Key |
A text string terminated with the newline code. |
4 |
Request ID |
It can be any value (text or binary) in 4 bytes. The CNS server does not use it but returns with the response. |
variable |
URL |
The HTTP request URL terminated with the newline code. |
1 |
Terminator |
The whole request is terminated with an empty line, i.e. a single newline code. |
Size (bytes) |
Data |
Comments |
3 |
Version |
It is 1, 0, 3 for this protocol. |
variable |
License Key |
This is the same key that was sent in the request. It is a text string terminated with the newline code. |
4 |
Request ID |
Any value in 4 bytes. It is the same ID that was sent with the request. |
1 |
Result |
See above for result codes. |
4 |
TTL |
The 32-bit number of seconds for Time To Live. |
variable |
Categories |
The ASCII encoded string of comma separated numbers, it is terminated with the newline code. |
1 |
Terminator |
The whole response is terminated with an empty line, i.e. a single newline code. |
This is the version 103 protocol extended with three request optional fields. The request can include any one or all of them in any order.
Size (bytes) |
Data |
Comments |
3 |
Version |
It is 1, 0, 4 for this protocol. Note that these bytes are not ASCII encoded digits but bytes with values 1, 0, 3. |
variable |
License Key |
A text string terminated with the newline code. |
4 |
Request ID |
It can be any value (text or binary) in 4 bytes. The CNS server does not use it but returns with the response. |
variable |
URL |
The HTTP request URL terminated with the newline code. |
Optional fields |
||
1 |
Terminator |
The whole request is terminated with an empty line, i.e. a single newline code. |
Size (bytes) |
Data |
Comments |
3 |
Version |
It is 1, 0, 4 for this protocol. |
variable |
License Key |
This is the same key that was sent in the request. It is a text string terminated with the newline code. |
4 |
Request ID |
Any value in 4 bytes. It is the same ID that was sent with the request. |
1 |
Result |
See above for result codes. |
4 |
TTL |
The 32-bit number of seconds for Time To Live. |
variable |
Categories |
The ASCII encoded string of comma separated numbers, it is terminated with the newline code. |
1 |
Terminator |
The whole response is terminated with an empty line, i.e. a single newline code. |
All optional fields begin from one or two “field code” bytes, have variable length and end with terminating newline code (“\n”).
Field Code |
Data |
Comments |
D4 or D6 |
Destination address |
This is the destination server address. The number defines IPv4 or IPv6 address family. The address format here is the hex-binary string in the network byte order. |
R |
Referrer |
The HTTP request Referrer header. |
M |
HTTP Method |
This is a one byte ASCII char for the HTTP request method: 1 – GET, 2 - POST, 3 - PUT, 4 - HEAD, etc. |
In versions 103 and 104 for cases involving processing error (the result code 0), the response does not have the TTL and Categories fields and the Result code is immediately followed with the newline code. This newline is not the whole request termination but it is supposed to be the Category string termination.
In cases where the Category can not be assigned immediately (result code 2), the TTL field presents in the response but has negative value to avoid result caching. The Categories string includes the New URL category (254) and can include Protocol and Country categories (e.g. 303 – Hypertext Transfer Protocol, 2040 – Canada).
All examples are a simple command line for CNS clients for the CNS protocol version 104. They have the CNS server address and the License Key hardcoded in the program and read stdin in a loop for URLs. Empty input breaks the loop.
$license = "12345v5";
$server = "localhost";
$port = "3436";
# Open the CNS connection
use IO::Socket;
my $sock = new IO::Socket::INET (
PeerAddr => $server,
PeerPort => $port,
Proto => "tcp",);
die "Could not create socket: $!\n" unless $sock;
$id = 32;
while ( $url = <STDIN> ) {
chomp($url);
if ($url =~ /^$/) last;
# Compose and send the request
$message =
pack("ccc", 1, 0, 4). $license ."\n".
pack("L", $id). $url . "\n".
"\n";
$sock->send($message);
# Read and parse the response
$sock->recv($resp, 8192);
@ver = unpack("ccc", $resp);
$nl = index($resp, "\n");
$rlicense = substr($resp, 0, $nl);
$resp = substr($resp, $nl + 1);
($rid, $code) = unpack("Lc", $resp);
if ($code != 0) {
($rid, $code, $ttl, $resp) = unpack("Lcla*", $resp);
} else {
$ttl = 0;
($rid, $code, $resp) = unpack("Lca*", $resp);
}
$nl = index($resp, "\n");
$category = substr($resp, 0, $nl);
# Parse the result code
if ($code == 0) { $res = "Error";}
elsif ($code == 1) { $res = "Good";}
elsif ($code == 2) { $res = "No Category";}
elsif ($code == 3) { $res = "Invalid License Key";}
elsif ($code == 6) { $res = "Good, Masterlist";}
else { $res = "Unknown";}
print "Version: ".$ver[0].$ver[1].$ver[2]."\n";
print "License: ".$rlicense."\n";
print "Id: ".$rid."\n";
print "Result: ".$res."\n";
print "TTL: ".$ttl." seconds\n";
print "Category: ".$category."\n";
print "\n";
$id++;
}
close($sock);
<?
$license = "12345v100";
$server = "localhost";
$port = 3436;
// Open the CNS connection
$errno = 0;
$errstr = "";
$sock = fsockopen($server, $port, $errno, $errstr);
if ($sock === false) {
echo "Can't connect to the $server : $errstr\n";
exit;
}
$id = 32;
while ( $url = (fgets(STDIN)) ) {
$url = trim($url);
if (empty($url)) break;
// Compose and send the request
$message =
pack("ccc", 1, 0, 4). $license ."\n".
pack("L", $id). $url . "\n".
"\n";
fwrite($sock, $message);
// Read and parse the response
$state = 0;
for (;;) {
$str = fgets($sock);
if ($state == 0) {
// The first response line
$data1 = unpack("c3ver/a*license", $str);
$state++;
} else if ($state == 1) {
// The second response line
$data2 = unpack("Lid/Ccode", $str);
if ($data2["code"] != 0) {
$data2 =
npack("Lid/Ccode/lttl/a*category", $str);
} else {
$data2 =
unpack("Lid/Ccode/a*category", $str);
$data2["ttl"] = "";
}
$state++;
} else {
// it should be the terminating empty string: "\n"
break;
}
}
// Parse the result code
switch($data2["code"]) {
case 0: $res = "Error"; break;
case 1: $res = "Good"; break;
case 2: $res = "No Category"; break;
case 3: $res = "Invalid License Key"; break;
case 6: $res = "Good, Masterlist"; break;
default: $res = "Unknown";
}
$ver=($data1["ver1"]+'0').($data1["ver2"]+'0').($data1["ver3"]+'0');
echo "Version: ".$ver."\n";
echo "License: ".trim($data1["license"]) ."\n";
echo "Id: ".$data2["id"] ."\n";
echo "Result: ".$res ."\n";
echo "TTL: ".$data2["ttl"] ." seconds\n";
echo "Category: ".trim($data2["category"]) ."\n";
echo "\n";
$id++;
}
fclose($sock);
?>
license = "12345v5"
server = "localhost"
port = "3436"
-- Open the CNS connection
socket = require("socket")
sock = socket.tcp()
res, err = sock:connect(server, port)
if sock == nil then
print ("Can't connect to the " .. server .. ": " .. err)
end
id = 32
while true do
url = io.read()
url = url:gsub("(.-)\n$", "%1")
if url == "" then
break
end
-- Compose and send the request
message = string.char(1, 0, 4) .. license .. "\n"
.. string.char(id&255,(id>>8)&255,(id>>16)&255,id>>24)
.. url .. "\n\n"
sock:send(message)
-- Read the response
verstr = sock:receive(3)
rlicense = sock:receive()
idstr = sock:receive(4)
code = sock:receive(1)
if code:byte() ~= 0 then
ttlstr = sock:receive(4)
else
ttlstr = ""
end
category = sock:receive()
nl = sock:receive()
-- parse the response
ver = verstr:byte(1) .. verstr:byte(2) .. verstr:byte(3)
rid = idstr:byte(1)
+(idstr:byte(2) << 8)
+(idstr:byte(3) << 16)
+(idstr:byte(4) << 24)
ttl = ttlstr:byte(1)
+(ttlstr:byte(2) << 8)
+(ttlstr:byte(3) << 16)
+(ttlstr:byte(4) << 24)
if code:byte() == 0 then res = "Error"
elseif code:byte() == 1 then res = "Good"
elseif code:byte() == 2 then res = "No Category"
elseif code:byte() == 3 then res = "Invalid License Key"
elseif code:byte() == 6 then res = "Good, Masterlist"
else res = "Unknown"
end
print ("Version: ".. ver)
print ("License ".. rlicense )
print ("Id: ".. rid )
print ("Result: ".. res )
print ("TTL: ".. ttl )
print ("Category: ".. category )
print ()
id = id + 1
end
sock:close()
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
char *license = "12345v5";
char *server = "localhost";
char* port = "3436";
char buffer[8192];
char url[1024];
int id;
char *src, *dst;
char ver[4];
char *rlicense;
char *category;
int rid;
int rescode;
char *result;
unsigned int ttl;
int sock;
struct addrinfo hints = {
.ai_family = AF_INET,
.ai_socktype = SOCK_STREAM,
.ai_protocol = IPPROTO_TCP
};
struct addrinfo *addrinfo;
int main() {
// Open the CNS connection
sock = socket(AF_INET, SOCK_STREAM, 0);
if (sock < 0) {
printf("Can't create socket\n");
exit(-1);
}
if (getaddrinfo(server, port, &hints, &addrinfo) != 0) {
printf("Can't resolve server name %s\n", server);
exit(-1);
}
if (connect(sock, addrinfo->ai_addr, addrinfo->ai_addrlen) != 0) {
printf("Can't connect to server %s\n", server);
exit(-1);
}
id = 32;
while ( !feof(stdin) ) {
fgets(url, 1024, stdin);
src = url; while (*src && *src != '\n') src++; *src++ = '\0';
if (*url == '\0') break;
// Compose and send the request
dst = buffer;
*dst++ = 1;
*dst++ = 0;
*dst++ = 4;
strcpy(dst, license); dst += strlen(license); *dst++ = '\n';
memcpy(dst, &id, 4); dst += 4;
strcpy(dst, url); dst += strlen(url); *dst++ = '\n';
*dst++ = '\n';
send(sock, buffer, dst - buffer, 0);
// Read the response
recv(sock, buffer, 8192, 0);
src = buffer;
ver[0] = *src++ + '0';
ver[1] = *src++ + '0';
ver[2] = *src++ + '0';
ver[3] = '\0';
rlicense = src;
while (*src && *src != '\n') src++; *src++ = '\0';
memcpy(&rid, src, 4); src += 4;
rescode = *src++;
if (rescode != 0) {
memcpy(&ttl, src, 4); src += 4;
category = src;
while (*src && *src != '\n') src++; *src++ = '\0';
} else {
ttl = 0;
category = "";
}
// Parse the response code
switch (rescode) {
case 0: result = "Error"; break;
case 1: result = "Good"; break;
case 2: result = "No Category"; break;
case 3: result = "Invalid License Key"; break;
case 6: result = "Good, Masterlist"; break;
default: result = "Unknown"; break;
}
printf ("Version: %s\n", ver);
printf ("License: %s\n", rlicense);
printf ("Id: %d\n", rid);
printf ("Result: %s\n", result);
printf ("TTL: %d seconds\n", ttl);
printf ("Category: %s\n", category);
printf ("\n");
id++;
}
close(sock);
return 0;
}