Techniques for sharing network security event information

[0001] This application is a continuation of U.S. patent application Ser. No. 14/615,228, filed on Feb. 5, 2015 on behalf of first-named inventor Andreas Seip Haugsnes for “TECHNIQUES FOR SHARING NETWORK SECURITY EVENT INFORMATION.” U.S. patent application Ser. No. 14/615,228 in turn claims priority to U.S. Provisional Patent Application No. 62/046,431, filed on Sep. 5, 2014 on behalf of first-named inventor Andreas Seip Haugsnes for “TECHNIQUES FOR SHARING NETWORK SECURITY EVENT INFORMATION,” and is also a continuation in-part of each of U.S. Utility patent application Ser. No. 13/556,524 (“Scalable Network Security Detection And Prevention Platform,” filed Jul. 24, 2012) and Ser. No. 14/536,386 (“Scalable Network Security With Fast Response Protocol,” filed on Nov. 11, 2014); the latter is a continuation of U.S. Utility patent application Ser. No. 13/556,553, filed Jul. 24, 2012 and issued on Dec. 16, 2014 as U.S. Pat. No. 8,914,406), each having a first-named inventor of Andreas Haugsnes. The aforementioned U.S. Utility patent applications also each claim priority to U.S. Provisional Application No. 61/593,853, filed Feb. 1, 2012 on behalf of inventor Andreas Haugsnes, and entitled “SCALABLE NETWORK SECURITY DETECTION AND PREVENTION PLATFORM.” Each of the foregoing prior applications is hereby incorporated by reference.

[0002] Private networks are at risk to directed attacks that attempt to overwhelm services, discover passwords and other valuable information, and otherwise misuse private network resources. The difficulty in detecting and mitigating these attacks is especially challenging when one considers the ever increasing use of the remote work place, and other cross-business “trusted” network connections that make it difficult to maintain a defined and pervasive “firewall” at network boundaries.

[0003] Various techniques have evolved in part to cope with these challenges. An enterprise having a private network can select from an ever increasing number of disparate products offered by different software vendors. While generally useful for their intended purposes, these systems require specialized training by dedicated personal for proper interpretation, deployment and maintenance; these systems are usually also incrementally added based on dynamic need, creating a hodge-podge of different systems, rather than leading to an efficient architecture that takes a client's current (and possibly dynamic) needs into account. Furthermore, one vendor's systems are typically incompatible with those of another vendor, leading to suboptimal solutions where a client must typically select a single vendor by balancing both advantages and disadvantages of the products of one vendor against another. Managed services have also arisen where the enterprise can contract a third party company to provide network management services, where the third party company remotely interfaces with the enterprise's security equipment or has personnel stationed on the enterprise's site to manage that security equipment; while generally beneficial to companies wishing to outsource IT security management, these solutions do little in and of themselves to address the vendor integration and custom architecture problems described above. In addition, there exists little in the way of mechanisms for one network security provider to share resources with another. That is, there are few mechanisms that enable network security providers to share experience or data regarding risks (including sources of questionable or offensive data, viruses, programs and sources of directed attacks); it is noted that many enterprises or other providers are reluctant to share such experience or data, because the act of sharing may expose vulnerabilities, opportunities or other consequences for that provider's network(s).

[0032] The invention defined by the enumerated claims may be better understood by referring to the following detailed description, which should be read in conjunction with the accompanying drawings. This description of one or more particular embodiments, set out below to enable one to build and use various implementations of the invention or inventions set forth by the claims, is not intended to limit the enumerated claims, but to exemplify their application.

[0033] The description set out below exemplifies (i) systems for providing network security (e.g., hosted by an enterprise or hosted for the enterprise by a third party), (ii) methods for providing network security (e.g., for one's own network or for a third party network), (iii) an information sharing repository and service for sharing information regarding security events, and (iv) other forms of devices and apparatuses that implement these things (e.g., instructions stored on machine readable media, such as software performing the same or similar functions as these methods or systems, or which allows a software user to build or implement similar methods or systems).

[0034] One embodiment provides techniques for communication between network security machines and products, potentially across dissimilar security products or network providers. Information regarding security events is processed to create templates or data records in a manner representing a common communication framework (CCF), e.g., using application program interfaces (APIs) that interact with vendor-specific products. In a variation of this embodiment, a caching protocol can be used which guarantees immediate return of “best available information” for certain types of queries (or automated response) with enforced response time, or that alternatively provides an asynchronous response (e.g., an automated response as soon as information is available) depending on type of query (or other trigger).

[0035] In another embodiment, a framework is established for information storage and for sharing of events for purposes of improving security. This framework can optionally use the CCF or caching protocol just described (or a different scheme). Information is received and stored for different security events (e.g., threats detected from specific files, specific locations or sources on the Internet and threats that are otherwise of concern from a security standpoint) and is maintained in a repository that can be queried by one or more network security providers. In one contemplated implementation, this framework is established and managed by a security services provider as a subscription- or other fee-based service for its clients. Such a security services provider can structure its services to provide cross-client alerts or reporting of common threats (or potential threats), all without revealing sensitive client information or vulnerabilities.

[0036] Still further, this disclosure provides a framework for building a custom security network and/or hosting such a network for third parties. Such network can be designed using a virtual machine (VM) architecture that permits cross-vendor or cross-product interaction, notwithstanding native dissimilarities or incompatibilities between products or their data operands or formats. Using a CCF, as introduced earlier, permits scripting of VM interaction and automation of processing and results by upstream and downstream products. If desired, this embodiment can be hosted as a service for third parties, permitting those parties (e.g., companies) to dynamically build a network security solution as needed, scaling services as needs permit, by adding and deleting services as necessary (e.g., as additional VM instantiations).

[0037] These various options can be combined in a number of ways. For example, a hosting service (e.g., which provides for dynamic customization of security services for individual clients using VM instantiation as described) can use the mentioned CCF and the mentioned communications protocol to assist with effective interaction between machines; such a service can also employ the mentioned repository for these same clients (or an overlapping or different set of clients) as an optional service. Many such permutations and combinations exist.

[0038] Additional detail will be provided below regarding each of these embodiments. Generally speaking, this disclosure is organized as follows. First, an information service and/or repository will be described that permits sharing of information between clients and/or network administrators. Second, a virtual security infrastructure will be described that permits building of a custom security solution, potentially involving different security products from different vendors. A communication infrastructure and protocol will be described in this context that permits these different products to “speak” with one another, and that permits scripted and/or automated responses and actions on a cross-product or cross-platform basis. As part of this description, a protocol will be described that permits vendors of different security products to both enforce real-time response processing as well as support asynchronous processing of more complex queries; this protocol facilitates the mentioned cross-product or cross-platform interaction. Finally, a number of business models will be discussed.

[0039] FIG. 1A shows an information sharing scheme 101 that uses an information repository to store events relating to network security. Generally speaking, four entities are illustrated in FIG. 1A for purposes of illustration including a first network provider 103a (i.e., an operator of “NET1”), a second network provider 103b (i.e., an operator of “NET2”), a source of a threat 105, and a party or organization 103c (e.g., a manager) that hosts the information repository; the information repository itself is designated using 109, representing a centralized database in this embodiment. Note that while the scheme of FIG. 1A involves a centralized database, it is also possible to have distributed mechanisms (e.g., using information stored directly within networks NET1 and NET2 for example), with a crawler or other mechanism used to acquire information and cache information at a query origin point or midstream network node as appropriate regarding queried network events. That is, the relatively centralized mechanism depicted in FIG. 1A is used to introduce basic concepts only and is exemplary.

[0040] As used below, an “event” is any type of information or data record that one network (or network administrator) might wish to log or to share with another network (or network administrator). Detect “threat” incidents are one such type of “event” that would be recorded in the ordinary course of network management. Typically, a “threat” is something that a network administrator might potentially wish to monitor, restrict or block; for example, a site, file, individual having inappropriate content (e.g., infringing or offensive content), or serving as a source of spam, viruses, or directed attacks would typically represent a threat. Note that any source or destination of traffic can be designated a potential threat, e.g., as used herein, a threat can encompass a circumstance where a relatively benign site is evaluated to determine threat level (e.g., and is assigned a threat level of “none” or “low,” for example).

[0041] In FIG. 1A, it should be assumed that the potential threat being evaluated has a particular internet protocol address (“IP1”) and that this particular address IP1 has been evaluated by network NET1 as potentially being a source of improper access attempts, SPAM, viruses, porn, or other activity that should be monitored and/or blocked. For example, security systems within NET1 can evaluate address IP1 as part of routine monitoring of a source or destination of traffic across a network 107a (e.g., a wide area network or “WAN” such as the Internet). It should be assumed that network NET1 wishes to share its evaluation with other networks or parties such that such other networks or parties can evaluate and potentially block access to or from site IP1 even though access to or communication with such other networks or parties may not yet exist. Accordingly, network IT1 transmits traffic 115 to a machine or repository, in this case managed by party or organization 103c (i.e., the manager of the information repository), for purposes of logging one or more event. Notes that there can be many event records generated that pertain to site IP1 and that an event can be generated in response to a security alert (e.g., detection of an inbound attack), or in response to any other desired evaluation. In some implementations, this manager 103c can be a machine or set of machines within a common umbrella network (or under common management with) network IT1, e.g., with a common enterprise, such as the government or a large company. Even in this situation, the event logging function may be transmitted in part over a public network and that the event logging information may reveal sensitive information such as a vulnerability of network IT1, a source of loss (e.g., representing potential loss of personally identifiable information), or the location of sensitive sites or systems; thus, in some implementations, client 103a (e.g., the entity associated with network NET1) sanitizes and encrypts such information before transmission 115 (e.g., for transmission over a public network). Sanitization is advantageous if logging of events is to be maintained by a third party (e.g., a third party security services provider) and/or if logged event information is to be shared with third parties (e.g., business partners, specific third parties, with clients of a third party security services provider or with the public at large). Encryption and related advantages will be discussed further below.

[0042] The information repository 109 logs reported events in a manner that is searchable. Logged fields for each event may include nearly any type of information, e.g., without limitation, any fields discussed below in connection with FIGS. 6A and 6B and, if desired, the information repository can be designed to store many different event types (e.g., mixed types having different fields or template information, optionally sorted by event type or class). Each event is stored as a discrete record in the repository 109 (or potentially multiple repositories), and is seen in this embodiment to include a threat event type, an IP address and a hash or digital signature associated with the corresponding event based on one or more keys unique to each different source of event reporting. In particular, it was earlier mentioned that an event can be generated by network provider 103a (NET1) that evaluates the source of the threat 105 as a “threat” and that a particular record 113 is generated and submitted to the information repository 109 to represent this evaluation as a specific event. Numeral 113a refers to the information logged for this specific event record, and numeral 113b represents the digital signature applied to (or the hash generated from) this event record using a unique symmetric key assigned to party 103a (i.e., and associated with network NET1). Note once again that the digital signature for each logged event will be dependent on the information on each threat record, as well as the key or keys used by the party that signed/provided that threat record; the information event record 113 in this case is also seen to contain information regarding the perceived source of the threat, i.e., source address (Src:=IP1). In the depicted embodiment, one or more shared keys are provided to each client or network (e.g., parties 103a and 103b) by the party or organization managing the information repository, known only to these two parties; this symmetric key architecture is used to provide anonymity and capabilities for source-specified data filtering, as will be explained below. As mentioned above, encryption and signing are optional, representing a design choice for the specific embodiment, and alternate embodiments can use other forms of security (e.g., PKI, two-factor authentication) or none at all.

[0043] It should be assumed that a second network provider (e.g., operator of “NET2”) now encounters the network address “IP1” either as part of incoming or outgoing traffic via WAN 107b and wishes to perform a reputation check. In some implementations, this second WAN 107b also represents the Internet and can therefore form a part of WAN 107a. Accordingly, the second network “NET2” sends a query to the manager 103c to learn whether there is any available reputation information associated with network address “IP1.” Note that as mentioned above, in other embodiments, this query can be run in a distributed manner, e.g., forwarded to multiple nodes (e.g., as a asynchronous query request). This information is received and processed by query logic 119, and is run against the information repository 109. In this case, the query logic 119 will detect a “hit” because the argument “IP1” will match specific record 113 of the information repository (i.e., that record containing IP address field 113c matching “IP1”). Note that it was previously mentioned that event record 113 is stored in a manner that is sanitized. In such a situation, there may be no information directly identifying the source of the particular threat record 113 as party 103a. However, the digital signature or hash stored for this record [H{NET1}] in this embodiment will correspond to only one shared key assigned by the manager of the information repository 103c. The manager therefore computes a hash using each known key until the manager finds a match between a stored hash and the newly-computed hash, which in turn identifies a particular key. The manager then uses a key/contact database (129) to access a set of permissions specified by network “NET1.” For example, these policies can specify which fields may be shared, a permitted list of recipients, and so forth. Note that it is possible for NET1 to be assigned many such keys, each associated with different policies, and potentially used for different event types. These policies and related information can be stored in database 131 (e.g., as part of a relational database including the information repository, the key/contact database, and other database information as necessary or appropriate) and retrieved as necessary. The retrieved permissions, in turn, are used to specify whether and how information regarding specific event record 113 can be shared with third parties. Assuming that network “NET1” allows its reported information to be shared with network “NET2,” the manager of the information repository can responds with a reply to the query from party 103b with information providing reputation information for IP address “IP1,” as represented by numeral 123 (all as consistent with permissions previously granted by provider 103a, as anonymously represented by the particular symmetric key used to sign the particular security event).

[0044] A few points should be noted about the scheme of FIG. 1A. First, a communication protocol for fast searching and response can optionally be implemented to ensure predictability in answering queries such as the query from party 103b. For example, a “cache-style” memory management process such as discussed below can be used to ensure that party 103b's query is answered immediately with whatever cached data is available (e.g. based on a synchronous response and associated command). In a distributed logging architecture (such as referred to earlier) this protocol is used to guarantee a timely response to party 103b. If threat data is stored elsewhere (e.g., on the network of manager 103c, party 103a's network or in another location), the use of an asynchronous command relayed to other networks (or network machines) ensures that the data will be asynchronously retrieved and cached by manager 103c in case another query is received for the subject-information in the future (this information can at that point be immediately provided). Note that other communication schemes are possible, and the referenced cache-style protocol represents only one methodology for handling queries. Second, while a symmetric key sharing scheme is referenced above for purposes of optional encryption in-transit and anonymous correlation of a reporting source with a set of permissions, some designers may instead choose to store information without anonymity or using other mechanisms for source identification and permission correlation. Finally, as noted earlier, “events” may include things other than “threats” and may refer to reputation of things other than IP addresses; for example, the same techniques can be employed for file name reputation (e.g., virus detection), evaluation of email sender domain names, identifying individual end users and many other types of arguments.

[0045] Note also that FIG. 1A shows that the manager 103c also employs optional sanitization 117, customer management logic 125, and a chat or communication service 133. In one embodiment, events can be reported using an automated web interface of the information repository 109; in such case, especially if third party network providers 103a and 103b lack sanitization protocols, manager 103c can elect to itself sanitize reported events or to convert reported data to a predetermined communication format (e.g., CCF) for storage in the manager's information repository 109. Alternatively, unsanitized information can be stored and then optionally sanitized as a part of any query processing or reporting of query results. In an embodiment which supports a set of source-specified permissions (e.g., stored in database 131), optional chat, email, or other social networking services can also be employed to share information between cooperating networks (e.g., “NET1” and “NET2”) or to otherwise provide for communication between these cooperating networks. For example, if desired, an administrator of network NET1 can choose to provide contact, telephone and/or email information as part of a response to queries from other networks, with such information automatically forwarded to each querying network (e.g., to NET2, for optional, direct contact between NET1 and NET2 without involving manager 103c). Many other possibilities also exist, e.g., the administrator of NET1 can be notified that another network has reviewed event data from NET1 and can at that time elect to have such contact information offered to the administrator of NET2. If the parties have a pre-established relationship (e.g., as “friends”), the event reporting source (“NET1”) can choose as part of its permissions to provide chat-status (either in connection with a query or event, or on a general basis). Such a chat service can optionally be linked to events or to the information repository 109, to permit network administrators (e.g., of networks “NET1” and “NET2” to share and discuss information regarding threats in real time or on an asynchronous basis). If also desired, this service can also be used to generate alerts, for example, to notify the party reporting an event that another party has generated a related event (i.e., an event with a common field). Such alerts can then be sent to a client requesting this feature, such as a network administrator for network provider 103a. Clearly, many such optional features and implementations exist.

[0046] FIG. 1A also provides detail regarding methods, services and business models. For example, manager 103c can host the information repository 109 and related services on a subscription basis, providing keys and information-sharing services for fee. Each client is managed via customer management logic 125, with one or more keys assigned to each client in connection with a subscription. If desired, each key can be provided on a temporary or one-time use basis.

[0047] Also, depending on the level of correlation established in response to a query (or a response), further actions can be taken. For example, in one embodiment an event is detected at a first network. Responsive to this event, a query or a correlation is performed against information possessed be a second network (e.g., the query can be performed by the manager 103c as a standalone function, or in a manner integrated with the second network 103b). Additional actions can then be taken depending on the result of this correlation; for example, if the initial correlation yields a match, indicating a possible security risk, this correlation can be deemed to warrant further searching (for example, of a local database or one or more third party networks), conditioned on this match. In one embodiment, each of these three networks can belong to different parties, with the threat processing performed as a centralized or distributed service. In yet another embodiment, a service bureau can be offered (e.g., by manager 103c) which collects security feeds from multiple sources, that correlates these, and that initiates other queries or performs additional (local or remote searching, e.g., of stored incident records) depending on perceived risk of the event (e.g., threat). Clearly, many such permutations are possible.

[0048] FIG. 1B shows more detail associated with a possible method, service or business model, generally designated using reference numeral 151. In particular, individual events are reported (automatically or selectively) as per numeral 152. These events can be transmitted over a private or public network for receipt by a gateway 153. The gateway confirms authenticity, i.e., that the event corresponds to a valid client, and also handles receipt and transmission errors as appropriate. For embodiments where event reporting is encrypted, the gateway 153 also provides decryption services. In one embodiment, as referenced by numeral 155, the reported events are received in the form of predefined templates using specific fields, e.g., in a CCF, for example, as represented by the non-limiting examples discussed below in connection with FIGS. 6A and 6B. The event reporting can also be made interactive, e.g., using an event reporting interface of the manager 103c, with each template being created by the information repository. In another embodiment, each client has client software (e.g., a dedicated application, a client module or temporary code, e.g., JavaScript) that creates the template format and uses a VPN or secure transmission scheme to transmit the event record to the information repository (see, e.g., FIG. 1C, discussed below). As depicted by dashed-line blocks 157, 159 and 161, the information repository optionally sanitizes, stores and generates time stamp information for each record as appropriate, for example, in mass storage, server memory, on magnetic tape, or in another manner. These general functions complete an offline process via which each event is reported and logged. Note that in a dissociated model, these logging functions can each be provided by multiple parties, and that query functions can be provided by these same parties or a different party or service.

[0049] As referenced by dashed-line 163, a query process is performed at a later point in time for newly-arriving queries (163). Each query can be received by a gateway 167 which, for example, services a number of sources of queries, such as third parties, different network providers or administrators, and so forth. As represented by dashed-line block 169, in one embodiment, query access is limited to specific individuals or clients and is a non-public service through which sensitive information might be transmitted; accordingly, in this embodiment, inbound queries are screened to verify requestor authority and/or to perform decryption. If a query is valid, as represented by numeral 171, the query is executed, returning a result 173. This result can provide a null response if the query is unsuccessful and can also return multiple “hits” dependent on design. Per numerals 175 and 177, a set of client permissions can be looked up (e.g., using the optional “client key” as an anonymous permissions' index if desired), and retrieved results sanitized or filtered (e.g., according to client permissions or according to policy of the manager of the information repository). A query response message 179 can then be generated and transmitted as desired, for example, using the “cache-style” response scheme mentioned previously or another scheme. Finally, a chat, email or alert service can also be established, optionally as part of the query process, per numeral 181.

[0050] FIG. 1C is used to provide detail regarding an optional client-side module or code used to interact with an information sharing or reporting service (such as with manager 103c from FIG. 1A); this detail is generally designated using numeral 191 in FIG. 1C. Note that the term “client” is used in reference to FIG. 1C not just to identify a private subscription or other third party service, but rather, to denote actions of any party in sending event information to an information repository or source of a query. As mentioned, this code can be part of a client module or software application previously distributed to the client, or can be implemented as code that is downloaded and executed at the time of execution (e.g., as JavaScript). Generally speaking, such code consists of instructions stored on non-transitory machine-readable media that, when executed, causes a client machine (e.g., server, computer, portable device, etc.) to act in a specific manner. More specifically, an information sharing service (and associated interface to exchange event information) can be provided to as either a part of a client security network or as a third party hosted security service 192. More information on such a service, including the optional use of VMs 198, will be provided below. As referenced earlier, an architecture is advantageously employed that permits cross-product or cross-platform communication (e.g., between products of different vendors) in a manner normalized for communication and in a manner that permits automated or scripted response and interaction. For example, each VM can be instantiated as part of a scalable security service as will be described further below, using an architecture and communications format that is conducive to standardized or normalized event reporting or logging. The VMs can optionally be part of a client network, a third-party hosted security network that provides the information sharing service, or yet another network (e.g., independent of the information sharing/reporting service). Per numeral 193, the client module or code optionally uses sanitization processes and common communication format (CCF) to facilitate anonymous reporting of data and logging of events in an efficient, searchable manner. Events can be reported to a third party information sharing/reporting service, or instead be logged in a client database(s) 194, for example, optionally performed as a server process. As depicted by numeral 195, the client can then choose to share information via its gateways with a third party sharing/reporting service or to any particular query source or other network destination (e.g., in a distributed environment). The gateway 195, can be configured to use security processes 195, which for example, encrypt reported data, or otherwise restrict reporting to an information sharing service or in accordance with any pertinent policies 196. Finally, on a solicited, unsolicited, immediate or batched basis, the client can report events or query responses as appropriate, e.g., to a third party query source or logging service, per numeral 197.

[0051] An information sharing service/repository as described provides many advantages to network security administrators and to providers of network security. First, it is possible for network administrators to share information with other organizations and query a much larger set of pooled data, to better evaluate potential threats, and to communicate directly or indirectly. This pooling of information can be across entities or organizations, but not that this is not strictly necessary for all embodiments, e.g., it is possible to combine data from multiple domains or networks within a single organization; for example, a modern organization's network infrastructure might include many subnetworks for different sites, different countries, different classes of employees or contractors, and different network types (e.g., VPNs). The mentioned architecture facilitates pooling of information from multiple sources irrespective of organizational boundaries. A common communications format (CCF) simplifies these processes and, using the teachings provided below, it becomes possible to automatically leverage such pooled data (i.e., events) to recommend or generate automated responses; such automation can also optionally be shared by network security administrators seeking to better mutual counter threats. Once again, organizations which might otherwise refrain from participating in such an information sharing scheme can benefit from the sanitization processes discussed above, with details of reported events being made both (a) anonymous, and (b) selectively deliverable according to source preferences.

[0052] As should be apparent, therefore, these processes significantly enhance the capabilities of network security in quickly and proactively responding to threats and improving the general level of network security.

[0053] In the sections that follow, detail will be provided regarding optional implementation of (a) a hosted, scalable security service or infrastructure; (b) a common communications format (CCF), useful for normalized interaction on a cross-product or cross-platform basis in such an infrastructure (as well as for event reporting); and (c) a cache-style communications protocol useful for enforcing communications and expected timing and response of communications in such a system (and for ordered query processing, for example, to a repository of event information, as discussed above).

[0054] In a typical application, a security network can be designed to perform intrusion monitoring and/or detection using distributed threat analytics, and responsive to results of intrusion analysis, to automatically take one or more actions using one or more security products run as virtual machines, though a CCF or translation layer (e.g., implemented using one or more API's). The virtualization infrastructure permits services to be added and subtracted dynamically, including new security software products from commercial vendors, potentially even in situations where those products are natively incompatible with one another; this environment also facilitates easy scaling of selected services. Scripting of automatic responses (for example) permits generation of an infrastructure where, notwithstanding which security software products are selected or deselected, reactions can be automatically invoked as reactions to detected threats (that is, these reactions can be designed for example even before services are selected, as part of the general infrastructure). To provide several examples, a threat detected by the intrusion monitoring or detection service (IDS) can be converted to a template or object with specific fields or attributes, and these fields or attributes can be used with a command or communication infrastructure to invoke macros or scripts that then import the rules, threats or other data into virtual machines (VMs) which have been selected for use on behalf of a specific private network. A template created in this manner can also be used for event reporting or logging, as introduced above. As should be apparent, each security product effectively relies on translation (either native, or effectuated with an API) to the common communication format, with standard query formats and scripting as appropriate. This architecture facilitates an environment where additional rules and policies can be automatically created and dynamically applied, providing a new degree of scalability and flexibility in network security management. Perhaps otherwise stated, normalizing inputs and outputs of various security products in a virtual environment facilitates a framework where services can be dynamically added as desired, and where new security policies and rules and interactions between these products can be automatically and/or flexibly created and applied.

[0055] Note that these techniques permit dynamic interaction between VMs in a manner that can improve reliability. Thus, using queries for information between systems, it is possible for an individual security product (i.e., VM in this implementation) to automatically rank possible threats and apply processing dependent upon the reputation associated with each individual threat, all prior to reporting an event or otherwise responding. As mentioned above, one embodiment provides for a cache management scheme/multi-tier response structure where queries associated with relatively sophisticated analysis can be processed with or without enforced response time requirements. By leveraging standardized deployments, these techniques empower existing security products to perform real-time lookups prior to making a decision of the outcome, or alternatively, to elect for more reasoned analysis, or to use both of these methods. These techniques also enable significant reduction in the processing and logging of superfluous data, i.e., by enabling security products to emphasize only confirmed threats and/or results of processing.

[0056] Prior to proceeding with the description, a few general terms used by this disclosure will be introduced. The terms “intrusion detection system” (IDS) and “intrusion monitoring” are generally used interchangeably, that is, to refer to a system that detects threats by monitoring wide area traffic, although not necessarily being able to block or otherwise remedy those threats; generally speaking, however, intrusion detection or “IDS” will be used to refer to such monitoring services. The “Internet” is used as a moniker for any wide area network (or “WAN”), i.e., security for world wide web (www) applications hosted by private network is a principle application of the techniques provided by this disclosure, but it should be understood that the teachings herein can be applied to any wide area network, including potentially applications such as Bluetooth, phone networks, semi-private and private networks and most other forms of wide area network that present unknown security risks—the term “Internet” is used both to refer to the world wide web as well as potentially any of these forms of WAN connected thereto or that form a part thereof. The term “cloud” refers to an unknown part of this WAN, with unknown or unspecific destinations and sources of traffic. A “private network” is any type of network or sub-network (subnet) where it is desired to provide security, and can include for example, a home network, the network of an enterprise or company, a governmental or institutional network (e.g., a university network) and so forth; the term “client network” should be understood to primary refer to a private network that is paying a third party for hosted security services, i.e., the “client” is the one paying for or otherwise receiving the benefit of security services to protect its network, and the third party possesses or manages security for the network of the client using techniques provided herein—the devices (e.g., computers, servers and routers) of the client represents the “client network” and the devices of the third party used to provide security represent the “security services network” or “hosted security network.” Note that the “client network” can also encompass other third party networks, e.g., the networks of trusted business partners, customers of that client, contractors, semi-trusted networks of others (e.g., a foreign subsidiary of the client), and potentially other third parties. The term “real-time,” as introduced above, refers to timing that is sufficiently fast as to enable in-line processing a packet or data in transit (e.g., as part of a normal processing pipeline or buffering scheme for traffic in general), and may consist of an interval or seconds or a much shorter response time—in some specifically contemplated implementations, this is effectively implemented as a specific, maximum response time defined by specification, e.g., 1 millisecond. Finally, as alluded to earlier, “traffic” can include any form of messaging or communication over a “WAN,” including packets, emails, instant messaging, secure transactions, web site accesses, telecommunications, media transfer, and many other forms of communication, whether encrypted or not.

[0057] FIG. 2A provides an illustration of a configuration 201 for a hosted security service 211, namely, one that services a diverse set of clients with client-specific resources and services. More particularly, this figure is used to help illustrate a service bureau method, that is, where security services are provided for multiple remote client networks for a fee. In this implementation, it should be assumed that n clients each have respective networks 203, 205, 207 for which the clients wish to procure hosted security services. Once again, a cloud graphic 209 is used to represent a WAN such as the Internet. Inbound traffic intended for distribution within the respective client networks 203, 205, 207 is directed to respective proxy sites within the hosted security network. The hosted security service 211 in this case represents a network controlled by the service bureau provider, and thus not only receives traffic from the cloud (as represented by path 213) but is also separated from each of the client networks by the cloud as well, as represented by three respective paths 233, 235 and 237. With security services performed remotely from each of the client networks 203, 205 and 207, it is desired generally to maintain a level of confidence in the post-security processed traffic and, to this effect, each of the paths 233, 235, 237 between the service bureau network 211 and the respective client networks are typically secure paths, for example, a secure tunnel formed as a virtual private network or otherwise using IPSec, SSL or similar processes. Note that these paths 233, 235, 237 all travel across a WAN (e.g., through the Internet or other applicable network) but are shown separately to reflect separate encryption for each client. As with the embodiments discussed above and below, a number of implementation variations exist, e.g., each client network 203, 205, 207 can forward all of its outbound Internet traffic through the service bureau network 211 for relay to ultimate destinations via path 213, or each client network can also simply communicate outbound Internet traffic to the cloud directly, as represented by paths 239, 241, 243, or redirect inbound traffic to the service bureau network 211.

[0058] As part of the service bureau network 211, each respective client network 203, 205, 207 is given its own dedicated proxy site or sites, labeled 215, 217 and 219 in FIG. 2A. That is, at least one and potentially any number of sites are given to each respective client for direction of inbound traffic for security processing. Thus, for example, if a hypothetical company “ABCZ, Inc.” was to route its traffic through hypothetical service bureau network at “servicebureau.com,” ABCZ, Inc. would be given one or more exclusive proxy site addresses within servicebureau.com such as ABCZ.servicebureau.com or ABCZ.US.servicebureau.com, ABCZ.EP.servicebureau.com, and so forth. Clearly, these examples are not meant to be limiting, and any naming methodology can be used. The use of one or more dedicated sites for each client network 203, 205 or 207 permits a unique set of rules and services (e.g., VMs) to be applied to all outbound traffic traveling through that site in accordance with security network configuration for the specific client, e.g., any specified IDS or other security service can be automatically applied to all received traffic, with unique services, policies, rules and other customizations, separately selected and applied for each client network. This client-specific customization (e.g., by respective administrators) is represented by respective control boxes 221, 223 and 225. While it is optional that outbound traffic from the client network to the cloud also be processed in this manner, it is contemplated that in a typical implementation, a specific client network will choose to have inbound and outbound traffic both processed by the security services network, but in different manners customized to the client through services selection and scripting. Again, an administrator for the specific client network can, using the VM infrastructure introduced earlier, build any desired network protocol using different and dynamically selectable security products, defining an entire security network infrastructure literally within minutes. For outbound web access Internet traffic, a X-Forwarded-For (XFF) process is typically used, for outbound email traffic, a “d=” header based technique is typically used, and for other network traffic, analogous processes are used so as to ensure return path specification to the pertinent proxy site. As depicted by respective sets of forwarding configurations (e.g., circuitry/machines configured to forward inbound traffic to the respective client network 203, 205, 207 using appropriate routing and encryption), once the customized services and processing represented by boxes 221, 223 and 225 have been applied, with any filtering, blocking or other processing applied as appropriate, that remaining inbound traffic which is suitable for forwarding is conveyed to the respective client network.

[0059] Each client network 203, 205, 207 is optionally provided with a respective secure web interface 245, 247, 249 for dynamically changing services (e.g., tearing down, reconfiguring, adding to or otherwise modifying configurations, including selection and deselection of services and setting automatic bandwidth maximums and minimums). Such a secure web portal is established by well-known web application processes, and typically relies on 2-factor or better security to authenticate an authorized client administrator or administrators acting on behalf of each client network. To provide an example, a particular client could elect to authorize up to a predetermined number of VMs to provide a specific service (e.g., if each VM services up to a maximum number of flows or packets, a client administrator could effectively pre-authorize use of “up to three” machines, with additional bandwidth expansion restricted without further client administrator authorization).

[0060] Reflecting on the structure depicted by FIG. 2A, a VM-based architecture and CCF usage permits a single entity (e.g., a service bureau) to provide for-fee or other services on an independent basis for each of multiple client networks (that is, for multiple private networks). A client database (not shown in FIG. 2A) stores amongst other information client security network configuration that allows the hosted security architecture to be uniquely customized for each client using what are effectively fungible resources built from the VM architecture, with translation and normalized/standardized communication between components. Each client can dynamically select and remove services and reconfigure its security functions in real-time, as desired, retrieve metadata, run vulnerability assessment tools, and generate reports. In addition to this dynamic capability, translation and interaction as has previously been described is automatically applied to each system, so as to apply automatically generate communications and apply reactive functions depending on the VM services selected by the pertinent client.

[0061] Note that the architecture described above can be implemented for one client only, e.g., as a hosted service, but also as a configurable security service within a single client's network. Using a CCF, this architecture is consistent with event reporting in a manner normalized across platforms in a manner amendable to optional information sharing, as introduced earlier. In a hosted environment which services multiple clients, information sharing can occur within a single network, e.g., at a common site which services multiple clients, with information being shared between those clients (or to a different client pool). In a private configuration (e.g., an enterprise network), the VM-based scalable security network principles introduced above can be employed with or without event logging. If logging is performed, events may be stored, reported to a third party, dynamically used to answer select third party or network queries, or none of the above.

[0062] FIG. 2B is an illustrative diagram used to further discuss these principles. FIG. 2B shows a security method 251 and related embodiments. In particular, a private network 252 is seen at the left as including a number of computers 253 that seek access to web pages or otherwise to communicate (i.e., transmit or receive information) out-of-network; these machines may represent web sites or other applications of the private network that are to be remotely accessed by third parties. Such web pages and other out-of-network communications are graphically depicted by a cloud icon 254, to denote that their sources and/or destinations could be anywhere on the Internet or other form of a public wide area network (WAN). As is typical for private networks, both inbound and outbound communications are routed through an edge device 255, which serves as a border device (or collection of devices) for the private network 252 and which may also optionally provide security functions such as a network firewall and other services. For large networks (e.g., company networks), an administrator or network security team would conventionally manage security, for example, using software at the edge device 255 or at other network devices, on an ad hoc basis. To cite some examples, it would be conventional to provide some level of antivirus screening, source blocking, spam filtering, and perhaps content filtering (e.g., to screen offensive, copy-protected or otherwise inappropriate material), all typically implemented by custom, reactive programming by the administrator or security personnel. Note that although depicted as one “device,” the edge device 255 represents potentially many machines and tools that cooperate together to provide data processing and forwarding including, for example, routers, load balancers, forwarders, exchange servers, and many other types of devices.

[0063] Automated management can be provided as a supplement or replacement to the manual and custom services indicated above, in part, by using a security architecture represented by dashed-line box 256 in FIG. 2A. Note that this depicted service may optionally be implemented outside of the private network 252, that is, at a remote network, with Internet traffic inbound to the private network being instead directed to a proxy site 257 and only thereafter being farmed out within the private network, for example as represented by paths 258 and 259. In embodiments where this automated management is performed within the private network 252, it should be assumed that the functions within box 256 are integrated with the edge device 255 (again, which may consist of any number of network machines and associated components and tools) or otherwise within the private network.

[0064] More specifically, inbound Internet traffic is received at a proxy site 257 and is then processed by services implemented as one or more virtual machines (VMs). A virtual machine in this context should be viewed as software running on non-specific hardware platform, using an emulated instruction set as necessary or appropriate. Generally speaking, in the system of FIG. 2A, each virtual machine might consist of a security software product of any vendor, with emulation software sufficient to run the security software product on nonspecific hardware. The security software products can include an intrusion monitoring system, also referred to sometimes as an intrusion detection system (IDS), a firewall system (FWS), an intrusion prevention system (IPS), a security event manager service (SEMS), an antivirus service (AVS), a vulnerability assessment tool (VAT) or any other form of security software product (e.g., for email or any other form of network traffic). Generally speaking, an IDS monitors traffic patterns using distributed threat analytics, meaning that packets or messages representing diverse communication (different sessions, users, or destinations) are analyzed to detect patterns representing a distributed network threat. If the IDS detects such a threat, for example, concentrated requests for access to a web page of the private network 252 from a single source or small group of sources (representing a possible denial of service attack), the IDS typically generates an alarm, alert or other response to indicate this fact. As part of an integration scheme mentioned below, for example, in one hypothetical implementation, an IDS can review local, fast access memory (e.g., a cache implemented in random access memory “RAM”) for security threats matching one or more identifiers for machines or applications (e.g., IP addresses, such as an IP address range passed as part of a query, a MAC address, URL, etc.). If the IDS has a record matching a specified identifier with a specific threat indication (e.g., threat level=“high”), the IDS can send a response to a FWS that causes the FWS to block the associated traffic, in real-time. Typically, the FWS might issue a query for a syn packet that commences an Internet transaction. If the IDS is not aware of such a threat, the FWS receives no response to its query (or alternatively, a null response), which permits the Internet session to proceed (again, in real time). If network delays, the need to access mass storage, or the need for further processing cause a valid threat to be not detected, an asynchronous process run (in addition to the non-response or null response) can cause the cache to be updated (e.g., without further response to the FWS), such that the IDS cache is “ready” should a similar query be received in the future (whether from the FWS or any other VM). Clearly, other implementation options also exist.

[0065] Note that an IDS typically does not itself take direct mitigation action responsive to such an alarm, e.g., this would conventionally be left up to the manual actions of a human network administrator. A FWS typically is positioned at or near the edge device and typically simply either blocks or allows individual messages according to pre-established rules or policies; for example, a FWS might reject traffic originating from a specific sender IP address. Also note that an IPS typically provides a more sophisticated form of filtering, for example, by routing or otherwise filtering traffic matching specific patterns; an IPS can take a number of different actions depending on rule and policy, for example, quarantining messages (e.g., emails with attachments), querying the source, routing traffic in a specific manner and so forth. A SEMS is typically a customized framework for coordinating multiple specific systems, and typically also manages alerts and archiving for the human administrator, based on relatively slow correlation across large amounts of historical data. An AVS typically filters individual messages for viruses or other malware. Finally, a VAT typically tests a specified system (e.g., a client system) to assess vulnerability to attacks, for example, by testing for system misconfiguration, application misconfiguration, application loading and response times, port errors and other forms of vulnerabilities. Note that many such products are available today from various manufacturers including Microsoft, Cisco, Juniper Networks, Symantec, and many other companies.

[0066] In the embodiment depicted in FIG. 2A, it should be assumed that software products from these various manufacturers are run as virtual machines (VMs) with inputs and outputs translated to a common communication format (CCF) with common message types (mentioned earlier) used to trigger interaction between those machines. Thus, for example, a Cisco software IDS product can be used to trigger a VAT port scan by a software product from a different vendor; this interaction might be formatted as a request for a new configuration with a passed argument (in the form of the CCF and translated data). Again referring to the example integration scheme discussed above, a VAT port scan typically might not be relied upon for real-time response and, thus, a response to the configuration message might be automatically generated at the time the port scan is completed, i.e., with no pending, active security operation stalled awaiting its completion. Note also that the use of plural products of the same type is also expressly contemplated. That is to say, the architecture presented in FIG. 2A permits for example two IDS systems to be used in parallel, each providing respective results. A client administrator can evaluate competing software products (to select one for implementation on a more-long-term basis), or alternatively, can use these products for more sophisticated behavior (e.g., a voting process where two products of the same type are used in parallel to obtain greater sensitivity in detecting attacks and assessing associated threat levels, or to provide for a suite of varied responses depending on the product detecting the threat). Again, the use of a VM platform with translation and broadcasting of events facilitates nearly any type of desired behavior. Note that in addition to or in lieu of automatically invoking a predetermined message type (request for information, notification, request for new configuration), other forms of triggers or scripting may also be used depending on embodiment.

[0067] Thus, inbound Internet traffic received at proxy site 257 (that is, Internet traffic traveling in the direction of the private network) is passed to one or more virtual machines for processing. FIG. 2A indicates that first virtual machine (VM) 261 is configured to provide IDS services; notably, this VM is depicted in dashed lines to indicate that in this embodiment, an IDS is optionally used to screen all traffic received at the site, prior to normalization of that traffic. Results of using this VM are then normalized by a translation layer 262 and passed to one or more VMs, seen here to include hardware 263 having one or different software products 1-n (numbered 268 in FIG. 2A). Note that a VM can include plural applications (e.g., 269, 270) running on a single piece of hardware, a single application (e.g., 271) running on a single piece of hardware, or a single application implemented as multiple virtual machines and running on several pieces of hardware (e.g., 270). Again, each application can be a software product from a different vendor or manufacturer or the same or different type of software product relative to other applications; in FIG. 2A, VM 269 is exemplified as an IPS, VM 270 is exemplified as a FWS, and VM 271 is exemplified as a SEMS. In the system of FIG. 2A, the translation layer 262 normalizes the inputs and outputs of each VM, in a manner so as to enable a common set of messages representing function calls, scripts, macros, load balancing or other programming to be applied; the results of these functions is one or more messages broadcast to multiple VMs or transmitted to a single VM, for example, using a network protocol (e.g., TCPIP). This translated exchange with each VM is identified by reference numeral 264 in FIG. 2A. Note that as indicated by box 265, each exchange involves a common communication format (CCF); reaction (including the generation of alerts, reports and other rules or configurations) is indicated by reference numeral 266. Note also that in alternate embodiments, the IDS (and indeed, any number of IDS products) can be implemented as one or more post-normalization VMs, e.g., an IDS can optionally be treated as any other software product, generically represented by numeral 268. Once any required processing has been completed (e.g., processing by the IDS 261 only, and preconfigured or triggered processing by other VMs 269, 270 and/or 271 through the translation layer 262), forwarding circuitry 272 is then used to forward the processed inbound Internet traffic on for distribution through the private network, for example, to various machines (253).

[0068] One benefit also provided by the architecture illustrated in FIG. 2A is the ability to easily scale hardware and selected services. To illustrate this point, expansion arrow 267 indicates that the security services can be scaled by adding additional VMs on additional hardware 263 as necessary. In one embodiment, this scaling is automatic; in other embodiments, scaling can be conditioned on administrator approval (e.g., client fee approval), or automatically performed within predefined parameters and requiring approval only once specified milestones are reached. Thus, to illustrate this point, if it is assumed that a specific software product (e.g., a Cisco IPS) can effectively manage up to a predetermined packet rate or number of users, as flow dynamically changes, additional machines can be transparently rolled into service as new configuration requests to handle temporary or long term bandwidth issues. Load balancing can be performed using dedicated machines, although in FIG. 2A, it is illustrated as being managed by scripting or other reactive functions, e.g., when flow exceeds an VM maximum, additional VMs are automatically established, with the proxy site 257 or an associated router or IPS automatically configured to reroute messaging to the pertinent VM. Notably, in one embodiment discussed further below, this type of approach facilitates service bureau methods where hosted security can be performed on behalf of a client, and where the client is charged fees on a micropayments basis, e.g., the client pays for each piece of hardware or VM only to the extent (duration and/or flow) consumed. When traffic levels fall below minimums, surplus VMs can be phased out. In an environment where specific, predetermined message types are used, as referenced earlier, this scaling can be automatically provided by new configuration requests, which are checked against any pre-arranged client limits or parameters, and then automatically invoked as new VMs. In a hosted environment, a client administrator can use a secure web portal or other interface to remotely and dynamically change management parameters as needed, for example, by dynamically authorizing additional services and/or VMs (e.g., above pre-configured client limits or parameters, if any). A TCP based memory access scheme (e.g., consistent with the integration and memory management examples introduced in this disclosure) facilitations this process, that is, is completely consistent with this scalability.

[0069] As mentioned above, Internet traffic inbound to the private network is routed through the proxy site 257 and is forwarded by forwarding circuitry 272. A number of mechanisms exist to ensure that inbound Internet traffic is forwarded by the Internet (e.g., “by the cloud”) in this manner. For example, the client can change its public domain name service (DNS) entries to list the proxy site. Alternatively and/or in addition, the outbound Internet traffic coming from the private network 103 can be processed so as to add headers or other fields that will cause any return traffic to go through the proxy site. To provide two examples of this, outbound Internet traffic can be routed to the proxy site 257 and/or other equipment represented by box 256 and a X-Forwarded-For (XFF) process can then be used to direct this traffic outward to the cloud, effectively routing it via paths 259 and 258. A return path is thus inserted into this communication so as to ensure that replies are directed to the proxy site 257. An advantage of this technique is that the private network 252 may choose to have screening employed on outbound traffic, e.g., antivirus screening and/or screening for inappropriate material or access to inappropriate sites, for example, implemented as additional VMs. It is also possible for the edge device 255 of the private network to perform this or a similar insertion and instead direct its outbound traffic via dashed-line route 260, with a return path inserted so as to direct reply communications to the proxy site 257. In the case of email messaging, domain keys (DK) and/or domain keys identified mail (DKIM) message headers may be modified, for example, to insert a value for the proxy site 257 into the “d=” or “i=” headers as appropriate. Analogous techniques exist for other forms of network communications. Again, a specific implementation can be used depending on whether the processes presented in FIG. 2A are employed as part of a single enterprise's security efforts, or are performed on a service bureau basis for multiple clients. Examples of each of these implementations are provided further below, and it is possible to have mixed implementations, e.g., in a service bureau environment, one client's private network can direct its outbound traffic directly to the cloud, while another client's private network may choose to route all outbound traffic through the service bureau network.

[0070] A number of advantages should be apparent from FIG. 2A. First, the use of a virtual machine architecture, and a normalization and translation process, permits ready and dynamic scaling of software security products and dynamic selection and deselection of those products. A software product may be selected and implemented as a virtual machine in a desired position or posture with other network security services, literally in minutes, with any existing configuration automatically updated to mesh with the newly selected product. The use of a virtual platform permits integration of software products generally without substantial need to modify or otherwise specifically configure the private network 252. A client represented by that private network 252 can thus be assured of having up-to-date security software products, and ability to dynamically switch between products and scale, without having to necessarily pay annual or similar license fees with each release, and without need to pay substantial fees before product performance can be evaluated. Second, while virtual platforms for software are known, the use of translation and, in particular, the use of a common communication format (CCF) using predetermined message types and/or other structure for communication exchange irrespective of security service permits different products with different capabilities (and from different vendors) to be used dynamically and interchangeably. More specifically, an intrusion detection system (IDS) can be employed to detect problems, with scripting then used to automatically take preemptive or curative actions using other products, not just specific products, but to classes of products having compatibility types that permit different products to address the same problem in different ways. This interoperability between machines is facilitated using techniques for normalizing data and invoking certain predefined, standard messages, effectively as function calls between machines. Further below, the use of templates and other approaches, and associated integration techniques, will be presented to exemplify implementation of these features and related capabilities.

[0071] FIG. 3A is a block diagram used to explain a security method 301 in which different security services (e.g., different types of services, and different vendors' products) are implemented as virtual machines, and are used with an intrusion monitoring or detection service to provide scalability and adjustable reaction capability. This Figure presents one hypothetical configuration of multiple VMs. Note that as with the embodiment discussed just above, this method permits dynamic selection and deselection of security services for a private network, implemented as respective virtual machines, as indicated by reference numeral 302. Per numeral 303, the selection amongst different selectable services can be optionally performed by a client administrator, for example, via a secure web application; per numeral 304, an integration schema provides fast reactive intra-VM response, for example, using the multi-tier real-time/asynchronous processing and querying techniques introduced earlier.

[0072] With services being selected and appropriate VMs being instantiated, traffic is first received at a proxy site, as indicated by reference numeral 305. The traffic is then processed by or otherwise routed to an IDS system 306, which performs distributed threat analytics on the inbound traffic; traffic is also routed as preconfigured for the private network, e.g., to other VMs for processing as appropriate (see, e.g., FIG. 3D, below). In addition to this preconfigured processing, detected threats or other results from the IDS can be used to trigger conditional functions, for example, scripted commands to another of the VMs, such as for example, to invoke a new rule or policy. To provide one example, relating to conditional “extra” AVS scrutiny, if the IDS system were to detect traffic patterns implying a heightened virus or malware threat, the results of the IDS indicating this fact could be scripted to cause an IPS to route a portion of the inbound traffic to a specific AVS VM. This scripting could be a implemented as a native function of an IPS product, or on a custom-basis by an administrator for the particular security network. It might be the case that a remote client network already has antivirus capabilities, but in this example, the distributed threat analytics would be used to invoke added AVS scrutiny through a standardized interface based on triggers generated as a result of traffic pattern analysis. Per reference numeral 307, results of the IDS processing are translated to a common communication format (CCF), e.g., a template or database record associating a specific IP address (source or destination) or other identifier for machine or application with a particular vulnerability or threat level; in one embodiment, these results include a variety of data from the IDS and, in a second embodiment, only alerts from the IDS or responses to queries are translated as events and logged or exchanged between machines using CCF. Per the information sharing techniques described earlier, threat assessments can be stored locally (and accessed pursuant to an inter-machine or inter-network query) and can optionally be reported to a third party service (e.g., an information repository such as discussed in connection with FIGS. 1A-1C, above). Note that as will be discussed below, in many implementations, metadata (e.g., a binary record of traffic) can be produced continuously and logged, either the IDS, by scripting, or by a separate VM. Should a threat be detected or other need for action, predetermined message types (i.e., a CCF) can be used to invoke the pertinent VM and associated security service, per numeral 308, relying on these logs (i.e., on the archived metadata) as necessary. Elaborating on these points, a reply to a query can provide a uniform resource indicator (e.g., a “URI” or “URL”) which provides a network access to metadata contained within the log, or another source of non-cached data associated with data carried by a query response; also, reputation can be enhanced by permitting the IDS to query other systems as to whether or not identified metadata represents a threat—thus, short of a detected threat, if supported by the specific IDS, the IDS can issue a “request” for information on a broadcast basis to other services, with arguments being passed using the CCF and optionally, the memory management scheme and other integration techniques introduced below. Alternatively, if a threat is detected, this information can also be normalized and broadcast to other services using the CCF; that is, using a network communication protocol (e.g. TCPIP), a message can be sent out to all other VMs broadcasting an event, with each recipient service having functionality to modify policies or take corrective actions responsive to the detected threat, and potentially to retrieve additional, non-cached data. Note that the combination of predetermined message types and the use of common network transmission protocols facilitate interaction between different platforms and security products. In addition, the IDS can also issue a request for a new configuration (e.g., a new VM instance corresponding to an existing or new service, within pre-established client authorization limits). This functionality can either be supported natively by the particular security software vendor as an engrained aspect of its products, or it can be effectuated via an API that sits on top of the IDS (or other VM).

[0073] Again, per process 302, each service operating in the desired client network can be arranged in advance as virtual machines, for example, by a client administrator, and can be running in the background or made active by a trigger from the IDS. Depending on the selected service(s) from a pool of available services, the aforementioned message types (or alternatively, a particular pool of scripts and triggers) can be made automatically available to the client based on a capability type which matches a capability type of services selected for use on behalf of the specific private network. These functions can be modified by the system as threats emerge or as software updates or scripting updates otherwise occur. Further examples and detail will be provided further below. As finally denoted by process block 308, subject to the IDS and any pertinent security actions (i.e., functions provided by the additional, selected security service(s) beyond the IDS), traffic is then forwarded on to the private network (e.g., a client network).

[0074] FIG. 3B is a block diagram used to explain a hosted security method 311, where a client uses a secure web interface 315 to dynamically select (add) and deselect machines and services as virtual machines. More particularly, FIG. 3B is used to explain how services can be hosted for a client network on a managed service or a service bureau basis. Arrow 313 represents access to the secure web interface 315 by a client administrator. The administrator is provided with dynamic capability to select various security software services, which as mentioned, can be different commercial software products running on a virtualized platform. The administrator can also specify bandwidth limits used to provide automatic VM scalability up to a predetermined limit; if the system needs to exceed these limits, the administrator can be alerted, for example, via the secure web interface 315 or via an email alerts or other alter system (generally represented by arrows 325). As the administrator effects changes in services, and/or rules, policies and scripting as desired, these changes are recorded in a client database 317 which represents a record defining the security service configuration for the associated private network, and which also provides a billing record (in the event the security services are provided on a micropayments or “pay-as-you-go” basis). As seen with reference to functional boxes 319, 321 and 323, the implementation stored in the client database is used in defining archiving and logging parameters, alerts, and invoicing. Depending on the network configuration defined by the client record, individual machines (hardware) can be managed (327), with machines being added or removed from use based on the client database record (329) and with machines being used in selected geographies, for example, geographically distributed data centers (331). In this regard, it is noted that one optional automated function (described further below) permits reactive routing of Internet traffic meeting certain parameters to specific machines and/or specific geographies; thus, for example, if a particular network is experiencing a high volume of directed attacks originating from “geography A”, as detected by the IDS, it is possible to automatically and dynamically reroute all such traffic to a dummy server located in “geography A,” or otherwise to quarantine such traffic through appropriate routing. This capability is enhanced if choices exist to use machine (hardware) resources (327) at different data centers (331).

[0075] As indicated, the client database specifies the security services configuration for each specific client, and in this regard, represents a selection amongst multiple, individually selective software products having different capabilities and generally representing different vendors' products (333). Any selected products are each automatically implemented as one or more virtual machines, per reference numeral 335, and can be run in the background or activated as needed, for example, based on IDS requests for new configurations and appropriate translation (337), with data exchange using a memory management scheme 338 (e.g., based on the “cache” techniques introduced earlier). As noted by dashed-line block 339, the selectable services can generally include AV, IDS, IPS, FWS, SEMS and other types of security products (that is, security products having similar or other capability types).

[0076] This architecture is further illustrated in FIG. 3C, which shows a method 351 where multiple clients each use respective services and policy suites, and where each client can independently add and deselect virtual machines via respective secure web interfaces. As denoted by boxes 353 and 355, a service bureau can host security for multiple clients through its own network, providing a virtual machine (VM) infrastructure for these clients to pick and choose security software products from different manufacturers or vendors. Translation and a common communication format (CCF) are used to permit these products to communicate with each other in their virtualized environment, and to report events. The use of a limited set of predetermined message types are also used to trigger actions responsive to the detected threats, with these events and other arguments passed between VMs using the CCF. Subject to any desired processing (e.g., baseline configuration) and any triggered actions responsive to a detected threat, traffic which has been suitably processed or filtered is then forwarded on to the particular client network. These actions are respectively represented by function blocks 357, 359, 360 and 361. As noted above, each client can have a security web service application made available to its proxied administrators, with these administrators dynamically selecting services and building a network configuration as appropriate (363, 365).

[0077] FIG. 3D provides a hypothetical example of one baseline configuration 367 which could be instantiated by a client. In particular, a graph indicates that a client has built a virtual network consisting in sequence of an antivirus service (AVS) 369, a firewall service (FWS) 371, an intrusion detection service (IDS) 373, an intrusion prevention service (IPS) 375 and a VPN forwarding structure 377. The particular client also has elected to have a web cache application (WCA) 379 and a specific vulnerability assessment tool (VAT) 381 conditionally invoked as potential threats are identified, with specific attacks redirected to a dummy machine 383. Each system is built as a virtual machine (VM). In FIG. 3C, round shapes represent automatic communications applied by the system, to invoke or otherwise control conditional actions of others of the VMs in response to detected threats. As should be apparent, this configuration can be specific to the particular client, with other clients electing to have other configurations and fewer or greater resources. Note that the interaction invoked is dependent on the services selected for the particular client, with additional rules and configurations automatically applied on an evolving basis, i.e., as threats are detected. The client can also use its secure web interface to electively add, delete or modify the policies de facto applied by any specific machine 369-383, with configurations, policies and associated metadata for traffic received by the client stored in, or otherwise identified by, the client database. Bandwidth parameters elected by the client are not separately depicted in FIG. 3D.

[0078] Note that it was earlier mentioned that a number of different configurations can be used to implement principles discussed above. Some of the contemplated configurations are introduced by examples presented in FIGS. 4A through 4E; other configurations will also be apparent to those having skill in the art.

[0079] More particularly, FIG. 4A shows one configuration 401 for a hosted security service 411, where a private network (represented by edge device 403) directs outbound traffic to a wide area network (“WAN,” represented by cloud 407), but where traffic from the cloud (including return traffic) is directed to the hosted security service 411. The traffic can come from various sources including messaging, web accesses from the private network to the cloud or vice-versa, FTP transactions and other forms of one or two way exchanges. As introduced above, private network on its end (per box 413) can change its public DNS records to point to a proxy site or sites (not shown in FIG. 4A) within the hosted network 411, and otherwise syncs up inbound Internet traffic with a return destination identifying the proxy sites or sites; the correspondence between the private network's actions and the particular proxy site or sites is represented by path 417. In this particular case, outbound traffic represented by path 405 is forwarded by the edge device to the cloud, as represented by path 409 with any appropriate return path information inserted as appropriate, such that response traffic proceeds via path 419 to the hosted network 411 and the appropriate proxy site therein. As noted by box 415, virtual machines (VMs) have previously been configured to supply baseline processing as well as interact with one another via standardized communications to respond to detected threats. Subject to processing of this traffic by the hosted service, traffic is then forwarded over the Internet to the private network via secure VPN, along route 421. The secure VPN in this case is represented by reference numeral 423. Thus, FIG. 4A identifies a configuration where inbound Internet traffic to the private network is processed by a hosted security service 411, but where the private network on its end manages communications directly with the cloud so that the inbound Internet traffic takes this route. This implementation is advantageous for private networks wishing to process their own outbound traffic, particular where the private network itself wishes to screen outbound traffic, or does not want a third party (such as hosted security service 411) to have visibility into certain information (e.g., personal information) outbound from the private network.

[0080] FIG. 4B shows another configuration 425 for another hosted security service 429; in this figure, numerals repeated from FIG. 4A should be assumed to represent like elements. A private network (represented by edge device 427) routes all outbound traffic and receives inbound traffic through hosted security service 429; that is, unlike the embodiment of FIG. 4A, the private network simply routes all outbound Internet traffic out over the Internet to the hosted security service 429 over path 433. As was the case before, the private network also updates its public DNS entries to point inbound web accesses to the proxy site (not shown) within the network of the hosted security service 429. On its end, the hosted security service uses an XFF protocol or other protocol where it effectively specifies a return path (439) on outgoing traffic from the private network and, optionally, the private network can implement VMs (441) that perform outbound traffic processing (e.g., AVS, content screening, blocking of specific destinations and so forth, as has previously been described). The outbound traffic is then routed over path 437 to the cloud 407. Inbound traffic (whether return traffic or otherwise) is processed in the manner indicated above, that is, is received at the proxy site, processed by VMs 441, and forwarded post-security screening over a secure path (path 433 or another path) and then is routed within the private network, as indicated by numeral 431. The secure VPN for at least traffic inbound to the private network from hosted security service 439 is denoted by reference numeral 433. Reflecting on the configuration represented by FIG. 4B, this implementation is advantageous for private networks wishing to outsource security for both inbound and outbound traffic, e.g., the private network can elect to have the hosted security service 429 perform all conventional forms of network security for all external traffic whatsoever.

[0081] FIG. 4C shows yet another configuration 443 for a hosted security service 449; unlike the foregoing examples, the private network (represented by edge device 445) itself directly handles inbound and outbound traffic with the cloud (via path 451), but itself transparently redirects inbound traffic (and optionally outbound traffic) to the hosted security service 449 for security processing via path 453, and receives return, security-processed traffic from the hosted security service via path 455. This implementation has the advantage that no changes to the private network's DNS names are required, but it does require extra routing and potentially additional infrastructure on the part of the private network. The edge device 445 receives outbound traffic from its network intended for the WAN via path 447 and also distributes screened, inbound traffic along this same path. At least for inbound traffic from the WAN, the edge device 445 redirects that traffic as indicated by box 459. Once again, the private network ensures that its redirect destination corresponds (461) to the address of the desired proxy site (not shown in FIG. 4C). At the hosted security network 459, the redirected traffic is processed by VMs 457. Once screening is completed, filtered traffic is returned to the edge device 445 for routing within the private network. Note that a secure protocol (e.g., VPN, SSL, etc.), denoted by numeral 463, is used at least for “backend” configuration of the security network via path 461 and forwarding of screened traffic back to the edge device 445. Note also that, as contrasted with the implementations mentioned earlier, the scheme represented by FIG. 4C is amenable to completely transparent use of the hosted security service 449, e.g., although separated from the private network by the Internet, the private network can be perceived as both source and destination of the WAN traffic as appropriate, removing or avoiding if desired any externally identifiable indicia that the hosted security service was used. This scheme also presents the advantage that a private network can elect to process some, but not all, of its traffic using the hosted security service. For example, a private network might choose to have external web accesses to its website routed through hosted security service 449, but may choose to internally process employee emails using self-managed antivirus and malware scanning prior to delivering emails to an exchange server. Many variations of a mixed security outsourcing model are possible using the presented frameworks.

[0082] FIG. 4D shows still another configuration 465 for a security service, this time entirely structured within a private network, where communication path 471 represents a connection to the public Internet or another WAN. Note that especially in this case, the principles articulated above can be implemented in software entirely within the enterprise (e.g., within the private network), in the form of a commercially-distributed software package or in the form of a custom-built solution. As before, the cloud is represented by numeral 407, with inbound and outbound traffic traveling between an edge device 469 and internal systems 473 within the private network via paths 475 and 477. However, instead of sending traffic outside of the private network for screening, the edge device 469 sends the traffic to internally-hosted VMs 479. These VMs perform client-configured processing as described before, with interaction produced by the VMs (in the form of responses 481, provided over control paths 483 and 485) being used to control the internal systems 473, the edge device 469, or other processes. This operation is like the operations described earlier, but in this case, communications, and data are more easily customized to interact with equipment internal to private network (e.g., that might lie in a separate network from any service bureau); for example, translation and triggers can be interfaced with any conventional private network device, such as for example, routers, secure web apps, load balancers and other equipment internal to the private network. This operation is also possible with a hosted security service (e.g., per FIGS. 4A-C, above), but many private networks may elect to limit control over their own internal systems as a matter of preference. Note that as was the case before, the communications or other scripting used in association with the VMs can include the new creation or configurations of rules or policies with arguments passed using a standardized interface such as with predetermined broadcast message types and data being passed in a common communication format, using a template- or object-based approach for events and data (such as discussed below) or the memory management techniques discussed herein. Note that the framework for providing this infrastructure, including commercial security products to be implemented as VMs can be provided with underlying code which sets up the virtualization environment for different software apps (e.g., as instructions stored on machine-readable media 486), or can be separately installed via web download or activation, per path 487. It is also possible for the platform for hosting the VMs to be built partially or entirely from scratch within the private network.

[0083] FIG. 4E shows still another configuration 489 for a hosted security service 429, namely, one in which a VPN concentrator 497 consolidates Internet traffic from multiple private network edge devices 427 and 491. This implementation provides a number of benefits for a client wishing to provide remote access to trusted third parties, such as business partners and contractors, each represented as separate private networks, but which are to appear to the outside world as having a common affiliation with the client. For example, edge device 427 might represent the perimeter of a large enterprise (e.g., a large company, or the government), while edge device 491 might represent the perimeter of its contractor; under these circumstances, it may be desired that the large enterprise dictate security requirements for the contractor's sponsored or affiliate web traffic. The implementation is depicted as similar to the implementation seen in FIG. 4B; once again, like numerals represent the same elements as introduced earlier.

[0084] In a typical implementation, outgoing traffic from each private network is sent out to the cloud 407 by the hosted security network via path 437, using VMs 441 with channeled DNS and return path treatment (represented by numeral 439). A VPN concentrator 497 is used to support multiple encrypted tunnels for each respective stream of traffic, and can also be used (e.g., in conjunction with other VMs) to route traffic in between the two private networks (e.g., with no processing, or with specific processing relative to traffic involving the cloud 407). Common VPN concentrators (such as from Cisco and other manufacturers) support multiple encryption protocols, and thus, secure communications (represented by numeral 435) can be handled very differently with each private network. This implementation provides a number of advantages for the trusted access model, i.e., different security models can be specified for “direct” trusted access to the client network versus for inbound traffic from the cloud, with each security model built, configured or reconfigured using services dynamically selected by client's proxied administrator.

[0085] In the discussion below, FIGS. 5A-9 are used to describe implementation options. Following this discussion, this disclosure will provide some example communications and responses (i.e., examples of automated responses to distributed threat analytics facilitated by the teachings of this disclosure, exemplified by FIGS. 10A-10E); finally, this disclosure will present some commercial application examples (i.e., with reference to FIG. 11).

[0086] A. Design of an Exemplary Hosted Service.

[0087] FIG. 5A shows a typical client implementation 501 using a hosted security service. In a typical implementation, an administrator proxied by the client builds a default configuration by establishing VM services (as core services, 503) using a client portal 505. That is, the client can log into this portal 505 using a secure web interface to initially select a suite of hardware or software resources, and then can dynamically adjust those resources as conditions warrant, per function block 507. The client typically proxies multiple network administrators for this purpose, e.g., each with at least some authority to define network security parameters. This same group or a different group can also be provided with log access, metadata access and report functions, per block 509. The generated reports can be tailored to different level of abstractions for differed management levels or other reporting purposes, or can provide drill downs for specific VM performance, vulnerabilities, detected threats, load management, or other system parameters. Alerts (511) representing threats in progress (or threats recently resolved) can be retrieved by administrators via the portal or alternatively can be pushed to the administrators as email alerts (per arrow 513). As noted, configuration changes made via the portal are recorded as part of the client database and are optionally sent to a billing module 515 (e.g., for the case where software is being licensed to the client on a subscription or consumption basis).

[0088] Inbound traffic is received at a proxy site associated with a specific client, and default processing is performed on this message, such as for example metadata archiving and basic antivirus and malware scanning, per function blocks 517 and 519. A suite of services selected by the client administrator is then invoked depending on the client specified configuration, for example, as stored as a record in association with the client per block 521; this set of services can include a string of services such as was diagrammatically indicated in FIG. 3C, with each selected service once again being implemented as a virtual machine. Traffic received at the proxy site is routed to VMs in the specified sequence (523) with load monitoring and balancing within any pre-authorized bandwidth limits established by the client record being automatically performed (per numeral 525). As mentioned, this load balancing can be effectuated by scripting or in other well-known manner.

[0089] Numeral 526 depicts the performance of distributed threat analytics. Typically, this is performed by pattern processing the metadata captured and logged earlier in the process, using a correlation engine 528. Any associated results, including any detected threats, are exported, with a translation and normalization (530) being used to convert the results, especially detected threats, to a common data format. The translation can take a raw input, for example, data representing an event, and convert it into a table or other structured format, such as an XML description with several fields respectively for an identifier, port address and threat level. For example, data can be generated in XML form and used either in this manner or converted to a format used by all VMs (with either native support for the selected format or APIs as appropriate to provide translations for each machine. To exemplify, a particular VM (e.g., an VAT VM) might generate data having the form:

	<xml>
	<IPAddr>123.123.123.123</IPAddr>
	<Port>80</port>
	<threat>high</threat>
	</xml>.

This event could be used as-is if supported, or could be converted (e.g., by API) to a string such as

• • [0090] {“dst_ipv4”:123.123.123.123,“dst_port”:80, “threat”:high}.
    Now that an IDS VM, for example, can see what the “dst_ipv4” and associated “threat” fields are, it can also look them up by query. Scripting (e.g. macros, per block 531) associated with a VM for example can then issue queries and process results to generate triggers 529, which essentially enforcing certain predetermined commands or rules which invoke other VMs. As denoted by block 531, the trigger could, for example, be used to invoke a vulnerability assessment tool using arguments passed by the common communication format. Note that as contrasted with a SEM which might be used to generate an alert or other action, the CCF can be used with plural VMs, i.e., the CCF can be used to broadcast messages (e.g., as a notification) to multiple VMs, with native security product code (i.e., the software product itself) or an API providing reactive capabilities; perhaps otherwise stated, one threat can be used to trigger multiple VMs (with the same or different capabilities), all through preprogrammed (VM) capabilities or through scripting or configuration via the secure web interface. As indicated by FIG. 5A, in one implementation, events are represented a specific template, created and logged in a database; for example, Mongo DB or another structured framework could be used to provide such a system. In one example, if a threat is correlated with a specific identifier (e.g., source or destination IP address, MAC address, etc.), then upon a determination expressed in the template that the threat is “high,” the CCF automatically provides a broadcast message that can be “understood” by a FWS (if implemented); real-time processing can be facilitated using the “cache” based VM response scheme introduced above and discussed below in connection with FIGS. 7A-8D. Threat processing in general is collectively illustrated by numerals 531, 533 and 534 in FIG. 5A, with optional logging and/or external reporting of detected events (e.g., threats) indicated by numeral 534. Note that this reporting is indicated in dashed-lines to represent that logging can be automatic within the system with threats or other events either (a) automatically (i.e., unsolicitedly) reported externally (e.g., to an information sharing service or repository of a hosted security provider or a third party, (b) periodically pulled by such an external service or repository, (c) forwarded to the client administrator (e.g., alerts can be used for this), (d) not shared, but recorded for tracking purposes, or (e) handled in some other manner. Importantly, load balancers (535) are applied at each given tier; should a particular service be flooded with traffic, then load monitoring and balancing is applied specific to that service to invoke new VMs as appropriate to maintain throughput (e.g., up to pre-authorized bandwidth limits, per numeral 537). Finally, any remaining traffic which has been processed and determined to not present a threat is forward by forwarding circuitry, per numeral 539, onto the client network.

[0091] While FIG. 5A shows one exemplary configuration, a client using the architecture described earlier can build any desired VM infrastructure, with or without an IDS as may be desired by the particular client. In this light, it would be helpful to further discuss the event data structure as well as how coding is used to facilitate communication between different VMs, particularly those that implement software from different vendors.

[0092] FIG. 5B shows an exemplary layer model 551 to illustrate different levels of code for implementation a security service that permits dynamic selection and deselection of VM services; in particular, respective layers include machine configuration 553, application software implementation 555, the use of objects and templates to provide an event data structure 557, and scripting and web applications which effectuate a normalized communication infrastructure 559. Thus, an entity wishing to build a hosted network infrastructure codes the individual machines (hardware) to serve as a virtual platform for the different, supported software products, and codes the virtualization infrastructure, as represented by reference numeral 553. Each software product either natively supports (or supports through the use of one or more suitable APIs) a memory management scheme 554 which provides support for real-time queries as well as more sophisticated analysis as indicated by box 554, in one embodiment, this memory management scheme can be implemented as a TCP based form of direct access, for example using Memcached. Layer 555 represents software applications on this infrastructure, in a manner where instances can be created as virtual machines as desired. At a third layer, template generation and data mining provide a normalized event data structure, for example, by converting all events and raw data into a standardized XML description such as introduced just above. Finally, web application infrastructure and associated APIs and scripting are implemented as a fourth layer, as depicted by box 559, with a standardized interface (i.e., cross VM communication format) being indicated by numeral 560; in one embodiment, this infrastructure relies on code and scripting written in Ruby and Ruby On Rails to implement functions of invoicing, VM configuration, secure web portals, load balancing, automated policy generation responsive to detected threats, and commands between VMs. As mentioned earlier, these communications capabilities can also be retrofitted into existing commercial security software products.

[0093] As alluded to earlier, standardized communications and passing of events back and forth can be effected in a number of ways. FIG. 6A is used to illustrate (601) a template-based format for a security system. The template-based format provides a partial means of establishing the various VMs, on a customer-specific basis, for exchanging information representing threats between VMs and otherwise recording events as records, and for commanding reactions to detected threats by specific VMs. As exemplified above, this structure can also be implemented by using an XML translation scheme. To provide a specific example, the “threat template,” generally designated by reference numeral 603, can be used to provide normalized, translated data in this implementation, identifying specific IDS results determined by the IDS to represent threats. Note that this data format can be used in a number of ways, e.g., it is possible as mentioned above to compare the IDS products of two different competitors to evaluate those products and to effectively provide a reputation service. The template generated by one of these products is substantially identical to the template generated by the other, indicating the template or service producing the threat, the source address and port, the targeted customer, and whether the threat level is gauged at low, medium or high. Note that APIs built in advance as part of the infrastructure and, more specifically, the translation layer, perform any conversions/translation to get to these results. Thus, for example, an IDS from one particular vendor may not natively represent threat levels in this manner, but the translation layer converts data output by the IDS into this type of assessment, to normalize that data and enable the comparison or sophisticated behavior just mentioned. This was exemplified earlier. If two IDS systems evaluate the same traffic from the same source to be threats, one determined to be a high threat, the other determined to be low for example, this information could be considered in selecting a long-term IDS product. Alternatively, different sets of policies/scripting could be invoked depending on which vendor's product (i.e., which “Template” in this example) identified the threat as high, and so forth; that is, a high threat level from one IDS product can invoke a first reaction (e.g., reply requests for information, or other form of reaction), while a high threat detected by another IDS product can electively be used to trigger a different response (e.g., from a VM with different capability).

[0094] In FIG. 6A, the “services” template identifies a software application, the “machines” template indicates a specific piece of hardware assigned one or more virtual machines and the “templates” template identifies a single instance of a service running on a machine as a single virtual machine. For example, a particular “service” would reflect a specific vendor IDS product (intrusion detection or monitoring product, vulnerability assessment tool, or another type of product). As implied by the template's attributes, each created template identifies current throughput, active status, and whether the specific software instance has a capability to detect specific threats (e.g., malware, denial of service attack, etc), assess specific vulnerability (e.g., port scan or software misconfiguration), block ports and so forth, as represented by an alphanumeric code for the “Capability” attribute. To exemplify usage, if a threat template produced from a specific source (e.g., “template” template) has a specific threshold (e.g., threat >=“medium”), then another template (aka an instance of a VM in this example) or a service with a specifically desired capability type can be activated and passed the threat as an argument as a reactive measure; to further elaborate this point, a firewall service (FWS) can step-in in response to a broadcast notification identifying a specific threat, in a manner that all “listening” FWS VMs (or a specific FWS) add the source identified in the provided threat argument to its list of blocked sources.

[0095] During configuration, a client administrator adds “services” (which are generally associated with one or more templates) and authorizes minimum and maximum numbers of machines, and as implied by the “site” attribute, can authorize these on a specific site (network) basis. Notably, the “site” and “data center” templates are used to manage equipment at geographically dispersed locations; introducing an example that will be used further below, a hypothetical service might have data centers in the US, in Europe, in Japan (Tokyo) and in Singapore. If attacks against a client are determined to arise from a specific geography, the threat can be used to automatically route traffic from the specific geography to a specific data center, to concentrate and isolate the attack, and prevent it from affecting services (with clean traffic routed to other data centers so as to not be impacted by the attack). A client can have machines at any number of data centers supported by the security services provider, as selected and elected by the client (e.g., for fee), and can install different networks across data centers using the “sites” template. As implied by the name, the “Customer ID” template is linked to a specific customer configuration and is used to identify permissions and perform authentication, and for billing and other purposes.

[0096] The foregoing represents one preferred implementation model, but there are other ways that can be used to implement a virtualization architecture as described above. For example, an object-based programming approach can also be used, with both threats and services defined as objects. When a threat object is created, it can be passed in course to a service object for processing and can also be used to, depending on attributes of the object, call or invoke a service (e.g., for specific types of threats). Note that an object-based approach is also represented by FIG. 6A, i.e., the design of objects as threats or services, each object having attributes used for scripting and event translation, is conceptually analogous to templates design and usage. As will be shown below, a relational database can also be used to build a system, although as implied, it is believed the templates-model presented above presents the framework best suited to web application development at the present time.

[0097] FIG. 6B provides an illustrative diagram 621 showing how a relational database can be implemented in an alternative architecture to provide similar capabilities. In this example a database can be established to log all threats as they occur. This database establishes a de facto data structure that can be used with scripting to invoke specific functions or services. For example, as indicated by numeral 621, the database identifying threats can have a field for originating VM capability, threat severity level, source, originating VM associated client, a link to associated metadata, and other information. If a VM receives a broadcast notification of a threat record from a specific VM, or from a VM matching a specific alphanumeric capability (e.g., from an IDS), another VM can be automatically invoked and passed the specific threat or other event record as an argument. Note that the relational database model can also be used to manage credentials for various clients, generate alerts, identify the client infrastructure, sort metadata, identify VMs, and store policies for specific VMs (e.g., in a manner that can be called up an easily edited by a client administrator). For example, as indicated by an alerts database 625, different thresholds can be used to generate different types of alerts, for example, using different communication mechanisms, provided to different client personnel, or in another manner.

[0098] For both FIGS. 6A and 6B, not all possible templates or databases are shown, and it is within the capability of a software designer with experience in these fields to build a translation and normalization model using a template, object or relational database model (e.g., using a model-view-controller architecture) and to rely on a standardized interface for passing communications and arguments back and forth between machines, and generating associated reactions. As shown below, a standardized interface permits security software product vendors to adapt their products to make use of capabilities provided by other machines, or to employ APIs to facilitate data exchange and translation. The use of a memory management architecture that provides for enforced real-time queries as well as more long term (e.g., sophisticated) processing also leads to enhanced system capabilities. Note also that in connection with the information sharing principles discussed earlier, nearly any piece information can be reported as an event, i.e., events are not limited to threat reporting, and one skilled in the art can apply the information sharing principles discussed above to other types of events (e.g., anything represented by FIG. 6A or 6B), or potentially outside the field of information security.

[0099] FIG. 7A provides an example of a hypothetical configuration 701, in this case, consisting of three physical machines 703, 705 and 707 and four VMs 709, 711, 713 and 715. FIG. 7 is used to introduce come communication protocols that help ensure meaningful responses in either a VM-based system or a query-based system. Note that as was introduced earlier and as is represented as running on the third physical machine 707, more than one VM (e.g., 713/715) can be employed on any physical machine. As before, each physical machine can be a computer (e.g., a server) running emulation software to permit VM instantiation irrespective of physical machine characteristics. Each VM can also be any type of security software product, such as a IDS, IPS, VAT, SEM, AVS, FWS, and so forth, as was similarly introduced earlier. Also, either or both of the VM application software and/or emulation software can be presented as instructions stored on computer- or machine readable storage (i.e., a physical form of storage media, such as an electromagnetic or optical disk, electromagnetic, state change, resistive, or flash memory, or another physical device that provides storage accessible to a computer).

[0100] The first VM 709 is indicated as having dedicated random access memory (“RAM”) space 717, the second VM 711 is illustrated as having both RAM 717 and a local disk 723, and the third and fourth VMs 713/715 are depicted as sharing both RAM 721 and disk storage 725. The RAM 717, 719 and 721 represents any form of “quick” or “fast” “access memory,” that is, memory local to the physical machine and providing relatively fast access times; while depicted in FIG. 7A as comprising RAM (e.g., dynamic RAM), other forms of memory, volatile or non-volatile, can also be relied upon for this purpose. The fast access memory is to be distinguished from relatively slow memory that does not necessarily provide real-time response capability, such as a mass storage device (represents as disk 723 or 725). This latter memory can be physically part of, or attached to the respective machine 705 or 707, or positioned at another location within the network. As represented by 708, each physical machine interacts with other physical machines using a network access path, that is, via a wide area network, where communication is in the form of packet-based transmissions between VMs or between a VM and any cache or “slow” memory. Each physical machine has a communication interface (e.g., a network transport or TCP manager dedicated for this purpose), not seen in FIG. 7A.

[0101] To further illustrate interaction between these machines, the manufacturer of each security software product may specify required amounts of RAM or other fast storage, disk storage capacity, or other parameters needed for effective application execution and interaction with other VMs. Reservation or configuration of this memory can be established at VM instantiation, e.g., at run-time. Additionally, any APIs needed to interface an associated software application with the network environment (including other VMs) may also contribute to the required RAM and other memory capacity reservation requirements.

[0102] FIG. 7B provides more detail regarding a cache-style management scheme that can be employed by a VM. This scheme is represented in FIG. 7B by reference numeral 727. In particular, it should be assumed that a VM in question (731) is to receive message queries, notifications or configurations from other VMs, represented by box 729. As an example, in a distributed information sharing system, VM 731 could represent an IPS or other system of a first network provider (e.g., provider of NET1 from FIG. 1A), and it might be that another VM (e.g., the IDS of a second network provider such as from “NET2” from FIG. 1A) seeks information regarding a specific IP address. An inbound message (e.g., a query) may be structured in a manner that requires a real-time response, or alternatively, it may not require such a response; in contemplated applications, the nature of the response will be indicated by the command or query type, and generally for a threat query, a real-time response will be requested. Note that each inbound message uses a common communication format (CCF) such as was identified earlier, and parses data stored in this format (as applicable) such as for example a look up to determine whether any VM is aware of a “high” threat level associated with IP Address (identifier) 123.123.123.123. Each inbound query is forwarded from a network transport layer 733 to a memory manager 735. The memory manager first determines whether a given inbound communication requires a response within a predetermined period of time (e.g. whether it requires a “real-time” response). This determination can performed a number of different ways, dependent on desired implementation. For example, in one embodiment, each inbound query can include a command or field that specifies a response time (e.g., that includes a specific time parameter, or that presents a bit or field alerting the memory manager that a response is to be provided within a defined interval). In other embodiments, this determination can be based on the source of the message (e.g., VM type), on a specific command included with the query (e.g., where “command A” specifies an operation and “command ‘A-1’ specifies the same operation but mandates a fast response), inferred based on circumstance, or in some other manner. In the schema used in one, specifically contemplated implementation, each inbound query either specifies fast response (e.g., <1 millisecond or “1 ms”) or it does not specify this fast response, and the memory manager reacts differently depending on this specification; if there is a need for a fast response, the memory manager checks an index 737 and local cache 739 for data representing a response, and immediately forwards any data that it has. This forwarding occurs via TCP-based network exchange via box 733. If no matching data is found, no data is provided back to the source of the message (i.e., either a null response is provided within the predetermined time frame of 1 ms, or no response is provided and the original source of the query assumes the response is null). In the event of such a “cache miss,” the VM 731 can perform more sophisticated processing that is not expected to provide a real-time response, either performing the processing itself, by forwarding a request for processing to another VM or by retrieving data from slower storage (e.g., mass storage), all as represented by box 743. As denoted by box 741 and the term “asynchronous,” this operation can be performed using a network transport process, and results in the memory manager 735 updating its cache (and potentially its index) responsive to this query, but generally not automatically sending response data (e.g., after the predetermined response interval) to the original source 729 of the inbound message.

[0103] If an inbound query is determined to not require a fast response, the operation can be quite different—for example, the memory manager 735 is not required to send a cache-based response, but instead can await completion of processing responsive to the query (e.g., potentially involving relay of a configuration or query, or an entirely different query) to another VM 743; only once this processing is complete and/or a response received from VM/mass storage 743 is substantive data responsive to the original, inbound query returned to VM 729. Note that slight variations in this methodology can be applied, dependent on embodiment. For example, in one implementation, if no fast response is required, the memory manager 735 can indeed fill the response with data from cache and then send a second response with updated data as soon as the “asynchronous” operation is complete. As denoted by the term “synchronous,” the memory manager can be made interrogate its index 737 and local cache 739 immediately in response to the inbound query, with an immediate, automatic response if a match is found. By contrast, as denoted by the term “asynchronous,” for some operations, the VM 731 is primarily concerned with updating its cache and may or may not be required to provide a (further) response to an original source of the inbound query.

[0104] Reflecting on the schema introduced by FIG. 7B, several advantages and benefits should be apparent. First, by providing cache-style management, with a synchronous/asynchronous response structure, both enforced real-time processing and more sophisticated query processing can be supported. If it is for example assumed that the first VM is a FWS and that VM 731 is an IPS, the FWS can issue a query to check whether a new syn packet represents a threat. This query can have multiple fields, such as a field for sender IP address and a custom-configured or vendor-configured question, such as whether a threat level=“high” (another field) is associated with the provided IP address; using another typical example, the IP address can represent a destination IP address or other identifier (e.g., associated with inbound or outbound traffic), and the threat level field can be configured to query whether the IPS is aware of any vulnerability associated with the destination, or whether the destination represents (for example) a dangerous website. The query can pass a template between machines, with some or all fields of the passed template populated and used to scan for a match. A query can, depending on configuration, itself expressly demand immediate response, or this can be understood from context by the recipient (e.g., based on the fact the original source is of a “type=FWS”). The IPS 731 searches its index 737 and cache 739 and (hypothetically) determines that the provided IP address indeed does have a threat level associated with it of “high”—with this data being stored in fast access memory, the VM 731 is capable of immediately sending this data to the FWS in the form of a notification response, and the FWS (receiving this information within the specified interval) then can block the session represented by the new packet. This type of processing can be applied to any form of Internet traffic, including email exchange for example. Changing the hypothetical, if the IPS 731 did not have a match in local memory, the FWS would either receive no response or a null response, and would presume (based on no response within the specified time) that no threat existed; the IPS 731 optionally could then to asynchronously investigate the query, for example, using a query to another VM (743, for example a VAT, enquiring via separate query whether the provided IP address was associated with any issues). The VM 743 could perform sophisticated processing (such as by maintaining a count of such queries, or performing processing or other analysis of metadata associated with the original query). If VM 743 then determined responsive to such a query that a threat was associated with the provided IP address, it could then provide an “asynchronous” response to VM 731 which then would update the cache 739 and index 737 as appropriate. Note in this example that the query from IPS 731 to VM 743 represents a type that does not require fast processing, and so VM 743 is not required to pass a response back to IPS 731 until its processing is complete. Thus, as should be apparent, a multi-tier response structure (e.g., involving two or more response periods such as described) facilitates interaction between security products using a common communication format (CCF). Similarly, from the vantage point of any particular security software product vendor, integration is facilitated by an understanding of a command structure that permits leveraging a framework for response types based on need; this facilitates a powerful infrastructure, with fast response timing relatively insensitive to the number of VMs implemented or the number of physical machines employed; this greatly enhances scalability.

[0105] FIG. 7C provides a flow diagram 745 associated with the memory management scheme just introduced. In particular, the need for fast response can first be determined, as represented by decision function 749. If such a quick response is needed (y) and a match is found in cache per decision block 753 (y/y), the data is retrieved from cache and sent back to the source of the query, per block 755. The process can then optionally continue on to also asynchronously update the cache, per numeral 757; as mentioned, this update can be performed by the VM in question (depending on its function) or can be filled from remote storage or in response to a further query to another VM. If the inbound query requires a fast response, but the cache does not contain data responsive to the query (y/n), the VM in question can perform retrieval and response outside of the predetermined time period, updating the cache as data is obtained, per function blocks 759 and 757. The withholding of the “fast” response within the specified time period permits the original source to presume that there is no threat, i.e., to equate no response with no threat, in real-time. In contemplated embodiments, no response need ever be provided to the original inbound query even once cache is updated (e.g., an FWS may have moved on to other real-time actions and may have no current need for this data). At the same time, facilitated by the cache update, the VM in question is ready to respond, with data on hand and adapted for fast, cache-style access, should another, similar query be prospectively received (whether from the same source of the original query, or another VM). Alternatively, if the inbound query does not need an immediate response (n), data can be retrieved and forwarded to the original query source once it is obtained, per function box 751 (e.g., when available, irrespective of any predetermined time period defined for fast response).

[0106] As should be apparent to those having skill in computer programming, FIG. 7C illustrates a two-tier timing structure for responding to queries, but three, four or greater tier architecture can also be used, each tier providing processing for messages having different needs. The two-tier example presented in FIG. 7C is illustrative only.

[0107] FIG. 7D helps further illustrate these principles. As with the embodiment introduced by FIG. 7A, three physical machines 771, 773 and 775 are illustrated, supporting four virtual machines 763, 765, 767 and 769. Each physical machine also has local fast access memory 779, 773 and 775, respectively. In this example, however, a central, mass storage device 777 is used as a centralized, TCP-accessible network asset; note that for some implementations, the fast access memory could similarly be configured as a centralized network asset, if accessible quickly enough via the network to meet specification.

[0108] FIG. 7D highlights communication flow originating from one VM, 763. That is, a first arrow 789 represents a network connection between this VM and the second physical machine 773, a second arrow 787 represents a network connection between this VM and the third physical machine 775, and a third arrow 791 represents a network connection between this VM and the mass storage 777. It should be understood that each other VM (765, 767, 769) generates similar communications to other machines (or VMs) and storage, but arrows representing these connections are omitted to simplify the figure. The communication flow of these other VMs should be assumed to be identical to VM 763. More specifically, each VM can send a message (query/request, notification or configuration) to any other machine, either intended for a VM, or in the form of a direct memory access to the other machine or to the mass storage 777.

[0109] FIG. 7E provides another illustrative diagram 793, used to explain how a model-view-controller architecture can be employed to generate communication between machines and otherwise provide reactive capability to detected intrusion or other events. This figure is used to build on the relational DB model introduced with respect to FIGS. 6A-6B and to further explain how individual record (or template or object) attributes and fields can be automatically invoked in a reactive manner based on a CCF, e.g., in response to detected threats. Notably, the left side of FIG. 7E provides an example object (“Example Object CEDF”) with exemplary fields listed within that object (“severity,” “source,” “originating VM Capability,” “Time/Date,” “Originating VM,” “Client ID” and “Indicator/link”), the middle of the figure provides some examples of threat levels and VM capabilities that can be used for scripting on a client/specific basis, and the right side of the figure identifies possible scripting (as denoted by the right-side header “macros/scripting”). In a web application embodiment, a templates structure is used and the scripting is advantageously implemented using an open source framework suited for web application development and associated inter-application function calls, such as Ruby and Ruby On Rails. More specifically, as seen in the middle of FIG. 7E, the detected threat level can be used by notification or configuration messages to invoke specific actions, based on an isolated threat record, or based on a voting or other sophisticated behavior (e.g., a rule invoked if two IDS systems separately call out the same the threat). Attention is drawn to box 795, which identifies one framework for identifying device capability (e.g., based on VM type) and for providing a response framework that meets real-time requirements (or other response timing specifications). A VM type field for the source of a query can be used in a number of ways; for example, if the VM which identified the threat (Originating VM) has an “IDS” capability, this argument and the identified threat level can be scripted to invoke configuration of a VAT VM (if the client has elected to use this service) to automatically retrieve linked metadata to identify the destination that the threat was addressed to and run a specific vulnerability assessment scan of the identified destination, e.g., a server configuration scan. The invoked VAT can then be used to automatically provide data back to the client identifying whether or not vulnerability exists, all without human intervention. The type field can also implicitly convey required response time, i.e., in some embodiments, each VM can be configured to “understand” that if a query source is a FWS (for example) that immediate or other specification-defined response time is required (e.g., less than 1 ms, as mentioned); alternatively, response time can also be specified by a query itself (e.g., using a specific response time field, a dedicated command type, or in some other manner). Note that unlike a SEM product which might be custom-programmed to interlink two sources, the virtualization architecture set forth herein permits any number of products to act in parallel including products of different vendors, and to be dynamically varied—in addition to a VAT for example, scripting for a threat from an “IDS” VM can be evolved automatically or manually updated to additionally invoke a FWS, as has previously been mentioned, to block further traffic from the identified source. As alluded to previously, this capability is not simply dependent upon SEM processing of historical data, i.e., message exchange between machines (e.g., a CCF such as request, notification and configuration messages) can be used to add reputations to possibly threats, to rank threats that align with detected vulnerabilities, or take other actions without awaiting extensive correlation of past data. Note that the use of these functions also requires less data logging; that is, a SEM traditionally might log vast amounts of data; by permitting function calls between different machines, and correlation of SEM patterns with detected vulnerabilities or other VM products, the amount of data processed can be filtered and restricted to events or data meeting specified thresholds, all on a dynamic, reactive basis. The performance of logging (“logging engine”, generation of reactive triggers, alerts to clients or to an operator of a security services network) is represented at the right-hand side of FIG. 7E.

[0110] It has previously been mentioned that a common communication format (CCF) implements a standardized communication protocol that includes predefined message types that can be understood by each VM in the security network. For example, in one embodiment, by filtering detected events and responsively generating one of a request for information (request), notification (of detected events) or (request for new) configuration, each machine can be made to report events, to query other “listening” VMs (or specific VMs) for confirmation of detected events or threat levels, and to invoke rules, new machines or other reactions to detected events. As mentioned earlier, this information can be shared or extended across networks. This functionality as mentioned can be natively supported by different commercial security products or otherwise implemented via an API that sits atop such security products, and it permits different vendors products to speak to one another, to pass arguments (e.g., data) back and forth, and to provide honed results based on threat reputation assessment by dialog with other products. In addition, it can be leveraged to drastically reduce the amount of data processed by the security network, by improving capability of specific VMs to filter data and report and log just those events meeting certain thresholds.

[0111] Note that the right hand side of FIG. 7E shows a memory management column; as just indicated, in some embodiments, a common communication framework can provide a defined mechanism for different VMs to interact with each other and also to interact with memory. For example, as represented by boxes 797 and 799, a memory management scheme can be employed where each individual VM or the system itself maintains a local cache in fast access memory. If a VM itself needs access to data in a local cache, this of course is near-instantaneous, but as noted by box 799, TCP-based interaction can also be provided for each other security product VM in the security network. As described above, a selectively-forced response time can be provided for certain command types needing immediate access to information (e.g., for commands from a firewall system, for example); if no data exists responsive to a query (e.g., a query as to whether a new syn packet associated with a particular IP address or IP address range presents a threat), no response to the query is returned, effectively causing the FWS (in this example) to equate no response with a negative response and allow admission of the particular session. On the other hand, as previously introduced, an asynchronous process is then also triggered to further resolve the issue (e.g., transparently to the FWS), such that, if the particular IP address or IP address range does present an issue (e.g., previously determined vulnerability for destination IP or a high security threat), future requests from the FWS will find updated data in cache sufficient for immediate response (i.e., a future session would be blocked by the FWS). In this manner, a standardized integration and interaction architecture can accommodate a need for real-time response, and also accommodate more complex network security queries that take time to investigate.

[0112] FIG. 8A provides an illustrative diagram 801 used to illustrate integration from a security software product vendor's perspective. The depicted scheme permits vendors to designate “public” fields, available for standardized interaction with VMs from other vendors, and “private” fields that are unique to the vendor (but may be shared across that vendor's different products). A dashed-line box 803 in FIG. 8A represents a virtual machine (VM). Relative to the principles introduced in the discussion of FIGS. 7A-7E, the vendor defines operands in the vendor's application software 805 to either represent private fields (807) or public fields (809). Private fields are typically proprietary to the vendor, and are maintained in memory (but not necessarily as part of the “cache” used to serve data to other machines); that is, typically private fields are stored only in private memory or in local disk storage (represented by box 811). Public fields (809), in turn, are those that are published and made accessible to other vendors for use in connection with their security products, with interface either being established natively by the vendor's own code (i.e., code that is par to of the security product) or via an API (written either by the vendor or by an entity providing the VM infrastructure). A function of the API for example, might be to equate vendor specific codes with the “string based” format indicated earlier and its specific fields (or similarly, to access a database using standardized protocols). Public fields (809) typically can be directly accessed from the network (e.g., as DMA), and can either be stored in cache 813 or “slow” memory (e.g., as part of an asynchronous process 815). Note that these same techniques can be used in connection with an information sharing repository, discussed in connection with FIGS. 1A-1C. For example, a particular entity can in one embodiment of such a repository designate certain fields as externally sharable or non-sharable; using multiple (e.g. symmetric) encryption keys, this information can made to differ for different events (e.g., different event types).

[0113] FIG. 8B presents a diagram 821 used to example another embodiment of a memory management scheme slightly more complex than the one introduced above. A VM 823 and/or its support infrastructure can feature modules for TCP management 827 (e.g., WAN communication management), application software 825, and memory management 833; a module in this context should be construed to refer to function-specific code, hardware, or code running on dedicated hardware, as appropriate. Message exchange 831 relies on the TCP management module, with messages being sent externally to other VMs or network-accessible storage 829, as appropriate, and being internally routed to the application software 825 or the memory manager 833, as appropriate. In this embodiment, the fast access memory is organized into three different databases or partitions, 835, 837 and 839. A first database, represented by index 835, stores what is effectively a network map, and contains templates or pointers to templates identifying available machine configurations (e.g., available VM resources, whether currently activate or otherwise). The index permits the VM 823 to selectively request configurations that invoke other VMs (e.g., for specific types of processing); similarly, the index typically identifies capabilities of these other machines, such as VM type. The index permits the VM 823, when it receives a query, to send further queries to (or request configuration of) other VMs and/or the disk storage 829, as appropriate. A first level “L1” cache 837 provides a database containing complete responses for queries that require fast access; for example, using the FWS example provided just above, if VM 823 represents an IPS and a query response represents a Boolean decision that “yes (there is a high level threat associated with the source's request),” this type of response would typically be provided directly from the first level L1 cache 837. Note that memory space associated with this cache space may be limited depending on physical hardware and the presence of other VMs running on that hardware; the memory manager 833 can rely upon a cache eviction protocol to purge stale cache contents based on frequency of use, time since last use, type of data, any combination of these things, and potentially other factors. The second level “L2” cache 839 is typically used to store response data as well as a referral for non-cached data; the “L2” cache may store data that is stale and occasionally needs updating, either based on functions of the application software 825, or based on queries to other VMs or disk storage, represented by numerals 831 and 829. For example, if VM 823 instantiates a vulnerability assessment tool, it may be that it possesses “fresh data” in L2 cache which is immediately served up to the requestor; alternatively, if a predetermined period of time has passed, the VM 823 may refuse to pass data and instead update “L2” cache contents, e.g., by query to another machine. To provide still another example, entries or records in the “L2” cache may contain pointers (e.g., locators such as URIs) to sources of non-cached or data, such as archives, metadata, or other types of data, and it may pass these pointers back to the original source of the query, such that the original source can then directly access that data. The L2 cache can also have limited capacity (e.g., dictated by available RAM of the physical machine running the pertinent VM) and can be managed using cache eviction strategies, as introduced above.

[0114] As should be apparent, this design is exemplary only, and many other designs will occur to those skilled in the art for effecting efficient interaction between different VM products.

[0115] FIG. 8C provides an illustrative diagram 841 which helps to further illustrate processing in one hypothetical configuration. Inbound web traffic, for example, received at a hosted security proxy site 845, is forwarded to a virtual firewall system (FWS) 847. The FWS broadcasts a TCP-based query to the security services network, per communication arrow 849, enquiring whether a source or destination IP address associated with the new traffic represents a known threat. This query is picked up by one or more IDS VMs 851, which interrogate their respective L1/L2 caches as appropriate; if it is assumed that an immediate response is required (as in this example), the L1 cache is examined. In alternate embodiments, the L2 cache or both the L1/L2 caches are examined. If one of the IDS VMs (851) is successful at matching the query to one or more IP addresses in its fast access memory (i.e., in its L1 and/or L2 caches in this example), and identifies a threat level (e.g., threat >low), records an event as appropriate, and sends an immediate response (as designated by response arrow 853) to the FWS VM 847, which then blocks the new traffic. The IDS VM in question can further relay this information (i.e., as denoted by arrow 855) to another VM, for example, a vulnerability assessment tool (VAT VM 857) either in response to the query or on an unsolicited basis (e.g., the VAT may keep a count of hostile accesses for different IP addresses or address ranges). A configuration request can also automatically be generated and sent for specific, new services, e.g., for a port scan responsive to unusual traffic patterns. Note that if more than one IDS VM is used, each VM can be assigned to a dedicated FWS (or other specific systems, or otherwise partitioned such that the identically-functioned VMs do not overlap in responding to a common request). If the IDS VM cache 861 resulted in a miss, the IDS VM 851 would not respond (or provide a null response within the predetermined time frame, e.g., 1 ms), thus permitting FWS VM 847 to forward (permit) the new traffic, which would be send to a TCP manager 851 and thereafter forwarded to the intended destination, per numeral 823. Assuming this traffic did represent a threat which the IDS VM was initially unaware of, the VAT VM 857 might identify data representing this threat from its L1/L2 caches, and it might respond via communication 859 to the IDS VM 851, which then would update its cache(s) as appropriate. If necessary or appropriate, the returned data could contain operands (e.g., threat=high, IP address=“w.x.y.z:80”) or a URI pointing to sources of non-cached data, with the URI being stored in L1/L2 cache 861. As noted either, this scheme provides a framework that permits relatively slow, asynchronous processing to be managed as part of a common communication framework that supports fast response processing.

[0116] FIG. 8D provides a block diagram 861 used to explain one method for implementing multi-level cache management techniques in a VM security network, and once again shows a two-level cache management technique. More specifically, a memory manager determines whether a match exists in cache, as represented by decision block 863; L1 cache, L2 cache, or both may be searched, with any located data being served from these locations, as represented by functional boxes 865 and 867. If a fast response is determined necessary and no data exists in the L1 cache, the original query is ignored or sent a null response (as denoted by “NR” no response designation in box 865). Alternatively, whether or not a fast response is needed, the L2 cache can be searched and an asynchronous update process initiated per function box 867. If there is no match at all in cache, an asynchronous update process can be initiated, as represented by functional block 869.

[0117] FIG. 9 is used to explain different allocations of responsibility, depending on implementation, and to explain some differing business models. A general architecture is represented by numeral 901, with the architecture divided into three horizontal partitions (separated by dividing lines 903 and 1105), to indicate different locations where various software components or responsibility can be implemented. First, a leftmost partition represents software and/or modules that can be installed on a client network in some implementations, generally represented by reference numeral 906. A middle partition represents functions that would typically be installed on the security services network. Note that as indicated earlier, for some embodiments, the functions of these two partitions can optionally be combined and implemented in one and the same network (e.g., the network of a large company or entity which provides its own security). Core functions (907) typically include the virtualization architecture discussed above. Finally, the rightmost partition identifies software and functions that can optionally reside outside of the service provider network, e.g., either in the hands of the service provider or a trusted business partner to the security service, i.e., that provides select services collateral to those of the security services provider.

[0118] As for the client network, once again, configuration is left in the hands of a client administrator; the client network ideally supports infrastructure for establishing a secure tunnel with the security service for configuration, management and backend actions as appropriate. This functionality is represented generally by reference numeral 909. In implementations that do not involve a web interface, the client network can be provided with a client module or specialized software that allows it to perform remote configuration and interaction; this software and its installation and control over a machine (i.e., a computer) is symbolized by icons 915 and 917 in FIG. 9. Note that if the virtualization architecture is to be distributed as a commercial software product, with or without software security products of various vendors, then this software product is also part of the machine readable media (denoted by disc icon 917, although the machine readable media may be any form of memory, e.g., DRAM in a server). As mentioned, in one model, the virtualization infrastructure can be distributed as commercial software, with a software user then electively downloading individual security applications as desired. The client network also supports any desired routing or configuration of its infrastructure (e.g., to route outbound Internet traffic to the security services network, per the examples of FIGS. 4A and 4B, to change DNS entries for inbound traffic, and so forth), and for traffic encryption and authentication. Some or all of these capabilities can be provided to the private network in question, as mentioned, as a software product 917. As denoted by ellipses 918, client responsibilities identified by the left partition is typically replicated at least partially for each client or private network supported by the core functions 907.

[0119] The middle partition of FIG. 9 represents functions that can be associated with the security services network. The associated responsibilities include providing the core functions (including VM support, event reporting and logging, etc.) 907, infrastructure for routing and communication with each private network receiving security services 919, and encryption and authentication (including VPN tunneling) services 921. In addition, the security services network can also include modules to handle payment for third party software licenses and other costs, and to invoice its clients receiving security services, per numerals 923 and 927. For example, it was previously mentioned that one contemplated business model supports a micropayments form of licensing, where client networks are invoiced on a “pay-as-you go” basis, i.e., just for those third party software products that are used and just for the duration or volume of use. To provide one example, depending on the license fee structure utilized, a client installing and operating a specific IPS “for one day” can be invoiced for 1 day's use only, or on another abbreviated basis; the virtualization model is such that the client need not extensively configure its own internal systems to meet the requirements associated with the specific “IPS,” and equally important, if the client decides to instead use another product, the client can elect to deactivate the associated VM without incurring annual license subscriptions or copy purchase costs; similarly, should the particular software vendor distribute a newer version replacement software, the security services provider can update its offering in a manner where the client is billed based on use, and thus, pays little if any for the upgrade. Fees can also be charged for external queries, for example, should a network administrator wish to leverage an external information repository. Naturally, may fee or subscription models can also be used, e.g., a client network can be charged, without limitation, a flat fee, a monthly fee, an annual fee, a per service flat, monthly or annual fee, or on some other basis. The software that implements and supports these various functions, and the machine or machines that the software controls, are collectively represented using icons 929 and 930.

[0120] Note that the network of the security service provider communicates with different entities; (a) multiple clients, represented by the left partition, each one of which has one or more of the responsibilities represented by numerals 906, 911, 913 and 915, and (b) one or more third party entities such as services providers to the service bureau, represented by the right partition, such as software licensors and regulatory authorities; these entities are represented by numerals 931, 932 and 935.

[0121] Elaborating on structure that can be outsourced or run as a complementary service to the security services network, the outsourcing of data center management provides a first example, 931. If a third party provides this support, the license payments function of the service provider network includes payment functions as appropriate to compensate the data center providers, e.g., these payments if desired can be tracked and invoiced to the account of the client network they are used to support. Similarly, if per-use or per-client software fees are paid to individual software vendors, then license payments (and support as necessary) are provided to (and from) these entities, as represented by numeral 932; note that a trusted link can be established for this purpose, e.g., to provide visibility and messaging access to support personnel. Other subscriptions and the optional outsourcing of client billing (e.g., for security services received) is represented by numeral 933; again, such entities can be provided with trusted access, such as incremental billing information for each private network receiving security services. Also, an information repository or sharing service may also be outsourced, or provided by another vendor. Generally speaking, any number of service providers to the security services network can be meshed with the security services network as represented by continuation arrow 941. If necessary, the security services provider network can also provide these third parties with access into the security services provider network, for audit, accountability or other purposes.

[0122] A special case is presented by boxes 925 and 935, labeled “regulatory.” For some fields of endeavor, particularly when dealing with health and safety, national security and financial administration, information possessed by the client network can be tightly regulated by governmental authority. Therefore, if warranted by the implementation and the specific business model, a third party entity (e.g., a regulatory agency, such as an agency of the government) can be provided with partial or complete visibility into the security and security networks of specific clients. For example, newer generations of laws provide for tight control over patient privacy and access to health records; depending on client business and the WAN traffic at issue, it may be desired under some circumstances to provide regulatory authorities with trusted, direct access to the private network, as well as to metadata, security configurations, records of client administrator services selection and other dynamic actions, and to other data, for accountability, national security, and other purposes.

[0123] Note that software modules to support any of the above interactions can be provided to these latter entities, as depicted by software and machine readable media icons 937 and 939; generally speaking, these modules support authentication and “trusted partner” connectively, to enable or facilitate electronic interactions with the security services provider, as described above.

[0124] It should be appreciated that by providing a flexible virtualization architecture that can support network security software products in any desired configuration, this disclosure provides specific examples of system, methods, services and software that greatly enhance network security. For example, this disclosure enables network security to be customized on a dynamic, individualized basis for each of multiple clients, e.g., via a common service bureau. These techniques can be extended to any private network, or hosted security for any private network, as well as commercial software products.

[0125] What has been described is a set of methods, software and systems design to implement network security services with a flexible architecture, and to provide an infrastructure for sharing of normalized information representing common events (e.g., threats) between networks, in a manner that preserves anonymity and yet facilities communication and interaction. Other applications will readily occur to those who design electronics or software in view of the teachings provided above, or in view of the invention defined by the claims set forth below. For example, while description above has focused on a template-based (or object-based or relational database) implementation for virtualization and translation, other techniques are also possible. Furthermore, while many of the examples presented above are rooted in intrusion monitoring and/or detection and response, other applications also exist.

[0126] Accordingly, the foregoing discussion is intended to be illustrative only; other designs, uses, alternatives, modifications and improvements are also possible which are nonetheless within the scope of the present disclosure, which is limited and defined only by the following claims and equivalents thereto.