Netsweeper Overview

Filtering and Deployment Type Comparison

The following table shows a comparison between the different types of Netsweeper filtering.  Please see relevant documentation for more information.

Filter Type	NSProxy	NS DNS	NS Capture Module	Client Filter
URL Filtering	Y	N	Y	Y
Full SSL Decryption	Y	N	N	Y
Off Network Filtering	Y	Y	N	Y
IP Based Policies	Y	Y	Y	N
Username based Policies	Y	N	Y	Y
Cloud Based Option	Y	Y	Y	Y
On-site Option	Y	Y	Y	Y
Categorization based on URL	Y	N	Y	Y
Filtering Override	Y	N	Y	Y
SafeSearch	Y	N	Y	Y
YouTube SafeSearch	Y	N	Y	Y

Features	Port Mirror	NSProxy	DNS	Client Filter
HTTPS Filtering Level	Hostname	URL with Selective SSL Decryption	Domain Name	URL
Safe Search	-	Yes	Yes	Yes
Policy Level	Per IP address	Per user	Per IP	Per user
Granular Social Media Controls	-	Yes	-	Yes
Cloud Hosted	-	Yes	Yes	Yes
Authentication	RADIUS, DHCP Syslog, AD	RADIUS, DHCP Syslog, AD, LDAP	RADIUS, DHCP Syslog, AD, LDAP	AD, LDAP
Installation on device	Not required	Certificate on device for SSL Decryption	Not required	Client app, browser, browser extension
Capacity per Policy Server (16 core CPU, 16 GB RAM)	150,000 web requests/second	10 Gbps, 6 Gbps SSL Decryption	10,000 lookup/second	150,000 policy lookup/second
Virtualization	Yes, but need dedicated access to NIC (VMWare)	Yes	Yes	Yes

Port Mirror Deployment

Port Mirroring is an 'Out of band' deployment, designed for content filtering in large networks. It requires a copy of outbound web traffic from a network tap, switch, router, or DPI. It is fast, effective, no latency, no network risk web filtering.  One Intel-based server 16 core CPU, 16GB RAM, 10GE Intel NIC can filter 150,000 web requests per second.

NSProxy Deployment

• Proxy based deployment, explicit or transparent proxy deployment
• Selective inspection and decryption at the edge gateway allows trusted SSL traffic through and decrypts per your policies for optimal security and throughput
• Advanced social media SSL protection allows access to safe parts of social media pages while restricting others
• Search keyword and embedded URL filtering features of Netsweeper software can be enabled for SSL/TLS secure websites like Google, Yahoo, and Bing.
• Decryption for individual workstations and or devices operating on Windows, Mac or Chrome
• Authentication behind NAT with HTTP/S cookies injection
• A single Intel-based server with 16 core CPU, 16GB RAM, 10GE Intel NIC can filter 10 Gbps of traffic & decrypt 6 Gbps of traffic
• Load balanced to multiple Policy Servers for scaling and redundancy

DNS Deployment

• Specify Netsweeper as DNS Server for filtered users
• Filtering is at the hostname level when the DNS lookup is made
• Per user filtering policy is enforced based on unique IP—requires RADIUS or DHCP logs integration
• Enforce Safe Search on Google and Restricted Mode on YouTube
• Traffic is logged at the hostname level and reports can be generated
• Ideal for managed networks e.g. schools & enterprise where users are restricted from changing network settings

Netsweeper Client Filter

Netsweeper Client Filters follows Policy set in the Netsweeper Policy Server.

• Apple iOS Safe Browser
• Android Safe Browser
• Chromebook Browser Extension
• Chrome Browser Extension
• Windows Filtering
• Apple OSX Filtering
• Filter the full URL for HTTPS, selective SSL Decryption for Windows & Apple OSX
• Enforce safe results on search engines
• Custom Branding
• The filtering policy management, logs & reports are done on the Netsweeper Manager in the Netsweeper Policy Server

Additional Features

Netsweeper Categories

Netsweeper comes with a complete set of categories available for your filtering needs.  Categorization updates are downloaded and applied automatically by the system.  If a web URL is categorized incorrectly, categorization updates are handled by our support team, and updates are then pushed out via a list update, which happens within 6–12 hours of opening the support ticket.

For the full list of Netsweeper Categories and their descriptions, see Netsweeper Category Definitions.

Netsweeper can apply policies based on the Geo IP location of the destination server.

Custom categories, allowed or denied lists can be created and maintained inside the Netsweeper Web Based Administration system.  Individual users can be assigned access to specific lists that can be managed.

Deny Pages

Netsweeper can display a custom deny page which is edited in the WebAdmin using a WYSIWYG editor or return 401, 403, 404, 450, 500 and 503 HTTP Response code to the web browser.

Allow and Deny Lists

Netsweeper allows the creation of individual allowed/denied lists as well as shared lists that can be used to monitor, allow or deny specific data such as URL, Scheme, Path, Query strings, File Extension and Regular Expressions.  Any of these can be defined to manage specific content.

Netsweeper has created a URL Lookup function within the Web Administration interface that allows the IT administrator to test conditions to determine if the content is allowed or denied.

Reporting

The Netsweeper Reporter subsystem enables you to make different kinds of reports for those requests. It can gather requests that match some criteria (e.g. requests issued from a specific IP address or requests to some web site), sort them, calculate statistics, and present the results as tables or charts.

Logging

The Netsweeper logging framework is based on a high performance, custom developed, logging framework named LogMod5, it is built on a graph-based framework which allows the user to construct custom logging designs with a high level of flexibility, one of which is the ability to export logs to SYSLOG.

Updating the URL Database

Netsweeper provides a single URL database that is cloud based which is downloaded in real-time as users are accessing content on the Internet.  Administrators may enter allowed and denied URL entries on a user, group or global basis.  Additionally, administrators can re-classify a URL if they do not agree with the cloud based classification.

Netsweeper developed its own URL Classification system to scale with the growth of the Internet.  When a user makes a request to access a URL that is not in the local Netsweeper server, the request is sent to the cloud based scan and classification system which in seconds will classify the URL and then share that classification across the globe.   Netsweeper is a crowdsourced URL classification system that classifies URLs as users are accessing them.

SNMP Capabilities

The Netsweeper product suite has the capability to integrate with any SNMP based Management system.

Ability to Integrate with Various Browsers and Operating Systems

Netsweeper's network-based deployments, such as NSProxy (transparent or explicit proxy), Port Mirror with the Capture Module, and DNS filtering, filter traffic at the network level. Because nothing need be installed on the devices being filtered, these deployments are compatible with all web browsers, mobile apps, mobile devices, and operating systems, including Google Workspace for Education. Where filtering needs to run on the device itself, Netsweeper also offers client-based deployments, such as Client Filter and the Workstation Agent. Netsweeper uses standard RFC-compliant methods to integrate with any network and works alongside any other RFC-compliant device or application.

Obtaining an End User's Identity

Netsweeper integrates with an existing directory system (Windows Active Directory, Apple Open Directory, Google Directory, Azure Directory, or Microsoft Entra ID using Microsoft Graph Directory) or with RADIUS Accounting to assign users to the correct filtering Policy based on their Group or attribute. Because Policies are assigned by IP address, the user's IP address must not be behind NAT. The exceptions are NSProxy, which can identify users behind NAT through HTTP/S cookie injection, and our thin client-based deployments, such as Client Filter or our Workstation Agent.

Ability to Differentiate by Group, Network or Device

Netsweeper filtering policies are applied based on IP addresses or subnets.  These IP addresses or subnets can be grouped into policies.  Policies include allowed or denied URLs, Categories including application protocol signature. Netsweeper can assign unknown or un-authenticated users into a ‘catchall’ policy with its own restrictions via the users or device IP address.

Filtering policies are defined based on the IP address of the User or device.  These IP addresses can be assigned into a group or common policy.  Policies can include specific URLs to allow or deny, categories and custom deny pages.  All policies are applied to the user or device IP address.  When the user authenticates against a supported system, the username is logged along with the request.  Netsweeper uses a seamless and agent-less integration with popular directory services and provides complete user identification, enabling simple, application-based policy definition per user or group.

APIs to Integrate with Netsweeper Database

Netsweeper provides APIs to allow customers to integrate with the Netsweeper database via a simple to use Web API.  There are two specific versions of the API that are available:

• Direct URL
• SOAP API (WSDL)

About Firewalls and Interfaces

Netsweeper is not a ‘firewall’ but rather a Packet Inspection System that is able to detect a variety of protocols using its unique outbound inspection system.  Policies are defined based on ‘category’ which means that application protocols are ‘categories’ that can be turned off and on a per IP, per subnet, per group or global basis.

Netsweeper utilizes Radius Accounting packets which are generated by the Wireless Access Controller to identify when a user has authenticated.  No additional software or agents are required.

All Internet traffic is IP based therefore BYOD devices that don’t already authenticate against a captive portal or directory service are filtered based on the device’s IP address. No additional software is required to be installed on the BYOD device since the filtering takes place using the devices IP address if the user does not authenticate using an existing captive portal.  If the user authenticates against a captive portal, the captive portal can be queried or the portal can notify the Netsweeper that a particular user is authenticated.

High-Level Components

The Netsweeper Policy Server is a term often used to describe the collection of all Netsweeper services that reside on a Netsweeper server.  It is important to note that this term can sometimes be confusing, because on the Netsweeper Policy Server resides many different services, including a 'Policy Service'.

The diagram below shows the Netsweeper Policy Server, and the services that reside on it.

Policy Service

The Policy Service, for which the Netsweeper Server is often named, is the heart of the Netsweeper filtering platform.  Its primary function is to make policy decisions and return these decisions to the interceptor service, such as the Capture Module or Enterprise Filter.  It also sends transaction information to the logging service for logging purposes.

Some important concepts regarding the Policy Service:

• The Policy Service loads the policy database into memory, through the WebAdmin service.  It does not connect directly to the policy (MySQL) database.  This is important to keep in mind when creating deployments with many policy servers/interceptors.
• While the Policy Service is configured by default to receive policy requests from the Capture Module, it can also receive requests from the Netsweeper Client Filter, Proxy Caches, and many other interceptor devices/services.
• The Policy Service is also known as the NSD (Netsweeper Daemon.)

Category Name Server (CNS)

At the heart of the Netsweeper solution are the Netsweeper data centers.  The Categorization engines scan the words and images from all new URLs and assign them to content categories.  Netsweeper’s millions of users worldwide generate more than 240 million web requests every hour, and the new URIs from among those are evaluated by our cloud-based categorization and added continuously to our Master Categorization Name Servers which in turn provide those categorizations to our customers.

Deployments

Multi-server Deployments

In a typical, high-demand network, multiple Policy Servers and Capture Modules can accommodate a high volume of outgoing internet traffic and provide failover support.  Inbound traffic does not travel through the Capture Module. A Layer 4 (Transport Layer) switch manages load balancing by routing or forwarding URL requests to available Policy Servers and Capture Modules.  In addition, Reporters assume all logging and reporting functions to preserve processing power on the Policy Servers and Capture Modules. 

For more information, see Multiple Server Setup and Installation Guide.

Remote Loggers and Reporters

The logging and reporting architecture allows for multiple reporting servers to work together to generate reports.  All servers share the same reporter database but have their own local set of request logs and save the report instances locally.

For more information, see Multiple Reporter Overview.

Network Connectivity

• Netsweeper requires certain network resources.
• Proper firewalling is important to meet most security policies.

For the full list of ports, protocols, and destinations, see Inbound and Outbound Ports.

All Servers

• DNS
• CNS
• Mail Server
• Netsweeper Updates

WebAdmin and the Policy Server

• Remote Admin (WebAdmin to Policy Servers and Loggers)
• Remote Logging (Policy Servers to Logger)

WebAdmin Database

• Database access

Agile Methodology

Netsweeper development uses an Agile software development process called 'Scrum'. Scrum is an iterative development cycle where requirements and solutions evolve through collaboration with cross-functional teams.  It allows development to focus on delivering the highest business value in the shortest time by rapid and repeated inspection of the actual working software (every two weeks to one month). In this process, the business sets the priorities and the teams self-organize to determine the best way to deliver the highest priority features. Requirements are captured as items in a ‘product back-log”. Items from this product back-log are promoted to the Sprint.  The development Sprint progresses through a series of short daily meetings called ‘scrums’ to monitor progress and identify obstacles.