Introduction CNS Categorization
Netsweeper Category Name Server (CNS) provides unique functionality for dynamically categorizing content.
This dynamic categorization of the Netsweeper CNS system allows Netsweeper to provide local cache entries which are tailored to the traffic in your network while still providing the access to over 5 billion categorized sites and the ability to categorize content seen for the first time in near real-time.
Netsweeper systems filter approximately 3 million requests a second for information from all across the world in multiple languages, multiple sectors and for multiple areas of information. Of this nearly 10 million requests per day are for brand new URLs which have never been seen before. With over 12 years in operations, our database of current URLs is over 3 billion items giving Netsweeper an unrivalled view of demographically accurate web history and usage.
This capability provides a true solution for the Web 2.0 content we see every day on common sites and allows you to provide a solution for this every changing content.
Early ‘surfing’ was a simple text based content system. Over the years with the advances in technologies and availability, and also the ever increasing demands of internet users, we have seen these simple text searches and responses now becoming ever increasing complex content displayed on our screens. From simple request and response interactions where the user initiated all the communications the web has become a bidirectional communications media, with all the inherent security issues.
The generation today that has grown up with web pages that have more and more capabilities to display pictures, sounds, streaming media, interactive content, have pushed the content into what we commonly refer to as Web 2.0. While the functionality behind the web is always increasing, the term Web 2.0 is not referring to a new revolutionary technology; it is referring to a trend in web usage to enhance creativity, information sharing and collaboration among the users
The main driver behind Web 2.0 is primarily the end user. Expectations of the content they view, the method of viewing it and the delivery of that content is driving the use of the web. The expectation of using the web to request and view information is out dated. Users now expect complexity of content and an interaction with that content
Web 2.0 has also brought us dynamic websites, a term used to describe a website that aggregates information from multiple sources and displays the result to a single request. This new advance has made the creation of advanced websites very easy and the results very pleasing to the end user.
With this change in the content delivered over HTTP, Netsweeper is continuously improving its approach of categorization to work with the increased size of content, scanning more varieties of content including JavaScript and running of these JavaScript elements to investigate the content delivered. This and many other new categorization technologies allow us to increase the functionality of the system to provide near real time categorization in this Web 2.0 environment.
The steady growth of Web 2.0 technologies and web driven applications has caused a significant increase in content to scan by our Categorization system, for example, the content delivered less than two years ago from http://www.cnn.com came from static HTML from a server hosted by CNN servers. Today this same URL will result in a series of JavaScripts which assemble dynamic content from over 55 sources, many external servers. Providing a categorization service which analyses this content in near real time is a feature which differentiates Netsweeper and is an area of constant improvement from our team.
Below is the flow diagram of how Netsweeper implementations utilize the CNS system for URI lookups to fulfill policy requests.
This diagram shows the full categorization process for every URI requested by a client. It must be remembered that this process, when viewed in the structure of a Web 2.0 page, or ‘mash-up’ of content will see each individual request generated by the single URI requested by the end user, and then each URI requested by the page also. This allows the Netsweeper solution to monitor and categorize every aspect of the content delivered to the end user, and not just rely upon the categorization of the URI requested or, typically, the Top Level Domain of that URI.
A ‘New URL’ means that the current users of Netsweeper (we have over 140 million end users using Netsweeper every single day) have not accessed these URIs before and the URIs had to go through this categorization process (as explained in the flow diagram above).
In addition, users and administrators are able to request a human review of URLs—either to add a URL to a category, remove a URL from a category, or add a URL to multiple categories. All sites reviewed manually are immediately updated in the Categorization Database and are available to the Master Category Name Server. These sites/updates are also downloaded nightly to the Category Name Server and Policy Server caches.
The mastered database is a database of humanly reviewed and re-classified sites that is checked every 5 minutes for any changes, and the changes are downloaded automatically.
When comparing Netsweeper to other filtering systems currently in the market, Netsweeper has the largest URI database of any of them containing over 3 billion unique URIs, each with its own Time-to-Live value so that if any requested URI has an expired TTL the system will re-download and re-classify the URI again, ensuring the database is updated.
Another way of thinking about this process is that unlike other systems the Netsweeper CNS is not a static database that’s downloaded on a scheduled basis, the Netsweeper system will categorize requests that are relevant to your users rather than our content teams guessing, offline, where users are going to go. This is an important distinction between Netsweeper and others, and while sometimes it may take up to a minute depending on latencies and download times to categorize a URI, a comparable response of ‘No Categorization’ from another vendor may take days to update in the database.
The entire Netsweeper categorization process—from initial outgoing Internet request for a URL never seen by the system before (worldwide) to Categorization Engine categorization and storage in the database—takes as little as one second and at most about a minute, depending on the global location of the network user and the speed of connection to the requested URL web server.
This provides an accurate and near real time solution for the ever changing content we see in today’s Web 2.0 world.