Physical infrastructure management system

[0001] This application is a continuation of U.S. patent application Ser. No. 13/666,430, filed Nov. 1, 2012, which will issue as U.S. Pat. No. 8,719,205 on May 6, 2014; which is a continuation of U.S. patent application Ser. No. 12/640,160, filed Dec. 17, 2009, which claims priority to U.S. Provisional Application No. 61/139,975, filed Dec. 22, 2008, the subject matter of which is hereby incorporated by reference in its entirety.

[0002] The present application relates to network management systems, and more particularly to a system for acquiring and providing physical location information and other information related to physical layer infrastructure, such as environmental information, about network elements.

[0003] Communication networks are growing in number and complexity. Data centers that house network elements have likewise grown increasingly large and complex. With this growing complexity, it can be difficult for network administrators to maintain accurate records regarding the physical locations of assets within a communication network. In many cases, the physical location of network assets is tracked manually, with significant opportunities for errors or omissions.

[0004] This can lead to many problems for network administrators and technicians. It is possible for an administrator or technician to know where a fault is within a network based on the logical network topology, but not to know exactly where in a particular building the fault is located. More particularly, it is possible in some network management systems to know that a server connected to a particular switch port is experiencing a fault, but if the physical location records are not accurate, it may take a significant amount of work to actually determine the physical location of the faulty server.

[0005] The lack of information regarding physical location of assets can also cause problems with network growth planning. For example, a network administrator or technician may have little idea of how much free space, connectivity, thermal capacity, power capacity, or functioning physical layer is available in cabinets in a data center without undertaking a significant effort in manually mapping out the location of network assets and related physical layer infrastructure elements on a cabinet-by-cabinet basis.

[0006] Another challenge encountered in expansion planning is a lack of knowledge about the physical environment where expansion is desirable or necessary. For example, an administrator may know that rack space is available for installation of a new group of servers. But the administrator may have little to no information about the impact that adding a server group will have on the power consumption, heat generation, structural cabling, power capacity, weight, security, or grounding in a cabinet or area of a data center. In some cases, this could necessitate an engineering review that will add expense and delay to network expansion projects. In addition, the lack of important information inhibits automation since the physical representation of a network may not be reconciled with the logical connections within the network.

[0007] It is desirable to have a system that addresses the problems listed above by automatically acquiring and tracking physical location information regarding network assets and by providing physical layer infrastructure data and other environmental information regarding network installations.

[0015] The present application provides a system that uses intelligent patch panels with enhanced patch cords and software to enable the discovery and updating of physical location information for network elements, as well as other information associated with a container or other boundary relevant to network design and operations information. The present invention uses hardware and methodologies discussed in U.S. Provisional Patent Application Ser. No. 61/113,868, entitled “Intelligent Patching System,” and filed on Nov. 12, 2008, which is incorporated herein by reference in its entirety.

[0016] Patch panels can be incorporated into a network in two different general deployment styles. In the first style, called “interconnect,” a patch panel is connected directly to network switching over horizontal cabling, with the horizontal cabling generally attached to the back panel of the patch panel. In an interconnect deployment, other network equipment is connected to the front of the patch panel for communication with the network switches. In the second style, called “cross connect,” two patch panels are used between a switch and other network equipment that connects to the switch. Both styles of deployment have advantages. An interconnect deployment saves money and space because fewer patch panels are needed, but a cross connect deployment makes it easier to implement changes in connectivity in a network.

[0017] A variety of specialized patch cords may be used in connection with the present invention, depending on the particular application. Patch cords may be considered as falling into two general embodiments: a “nine wire” patch cord style, which has one additional conductor in addition to the traditional four-pair Ethernet patch cord, and a “ten wire” patch cord style, which has two additional conductors. While the terms “nine wire” and “ten wire” will be used herein, it is to be understood that these terms are used to refer to patch cords having either one or two additional conductors used for patch cord management functions. For example, a “nine wire” patch cord could be replaced with a fiber optic patch cord having one conductor used for patch cord management functions, in addition to the optical fiber.

[0018] According to certain embodiments and applications of the present invention, different specific types of patch cord may be used. In interconnect deployments, nine-wire and ten-wire patch cords that integrate hardware for allowing detection of insertion or removal of their remote ends (the ends not attached to the patch panel) may be used. In cross-connect deployments, intelligent patch panels can be used in conjunction with a nine-wire patch cord style that does not necessarily need to incorporate hardware that allows insertion or removal detection at the remote end of the patch cord. This is because two intelligent patch panels can be used to communicate connectivity information to one another over ninth wires of patch cords.

[0019] Turning to FIG. 1, the present invention uses intelligent patch panels installed in cabinets, which may in turn be housed in data centers, data closets, or other locations. Two intelligent patch panels 10 and 10′ are illustrated in FIG. 1, as installed in two cabinets 12 and 12′. The intelligent patch panels 10 and 10′ have front panel ports 20, and each front panel port 20 is connected to a corresponding rear connector of the patch panel (such as a punchdown connector or a modular connector). The rear connectors, in turn, are connected to other parts of the network (illustrated as structured cabling cloud 14) by horizontal cabling 16. Management ports 17 of the intelligent patch panels 10 (which may be Ethernet ports) are connected via management connections 19 to the network cloud 14, and can transmit data to and receive data from a physical infrastructure management (PIM) server 15 through the structured cabling cloud 14. Although the management ports 17 are illustrated in FIGS. 1-3 as being on the front panels of intelligent patch panels for clarity, it is to be understood that the management ports 17 may be provided on the rear of the intelligent patch panels. The PIM server 15 is illustrated as a laptop in FIG. 1, but it is to be understood that the PIM server runs PIM software and may be implemented on a variety of different hardware platforms, using a variety of interfaces. It is preferred for the PIM server to have a graphical user interface.

[0020] Ten-wire patch cords 18 connect patch ports 20 of the intelligent patch panels 10 to network hardware installed in the cabinets 12, including switches 22 and servers 24. For example, in FIG. 1, a ten-wire patch cord 18a connects patch port 20a to switch port 21a. In the embodiment of FIG. 1, all of the network equipment within a cabinet is connected via ten-wire patch cords 18 to patch ports 20 located on the front of the intelligent patch panel 10. Using this connection scheme, if the physical location of an intelligent patch panel 10 is known, including specifically the location of the cabinet 12, then the location of all of the network equipment known to be connected to the corresponding intelligent patch panel 10 is also known and associated with a container, such as a cabinet. Each intelligent patch panel 10 or 10′ is provided with physical location information, defining exactly where it is located. Each intelligent patch panel 10 or 10′ is further capable of reporting this physical location information to the PIM server 15.

[0021] According to one embodiment, the patch cords 18 are provided with insertion detection switches at the plugs of the patch cords that are plugged into the network elements such as the switches and servers (i.e., the plugs on the ends of the patch cord that are not connected to the intelligent patch panels 10). According to another embodiment, insertion or removal detection is accomplished by a ground detection scheme as explained in U.S. Provisional Patent Application Ser. No. 61/113,868.

[0022] FIG. 2 shows additional hardware that can be used in one embodiment of the present invention. FIG. 2 contains essentially the same hardware and connections shown in FIG. 1, with the addition of power outlet units (POU's) 26. POU management ports 27 of the POU's 26 can be connected with patch cords 28 via daisy chaining to a management port 17 of the intelligent patch panel 10 installed at the corresponding rack of the POU's 26. The POU's 26 contain power outlets that are used to power network equipment such as switches 22 and servers 24 and allow information about power consumption at the cabinet 12 to be collected by the intelligent patch panel 10. The connection of POU's 26 to an intelligent patch panel 10 allows power and environmental data to be associated with a defined space (e.g., a cabinet). Aspects of power outlet units are disclosed in U.S. Provisional Application Ser. No. 61/040,870, filed on Mar. 31, 2008 and entitled, “Power Outlet Unit,” which is incorporated herein by reference in its entirety.

[0023] FIG. 3 shows additional hardware that can be incorporated into one embodiment of the present invention. In the embodiment of FIG. 3, various types of sensors are connected to sensing and control ports 30 provided on the POU's 26. Sensors for use with the present invention can include relative humidity and temperature sensors 32. An additional sensor 34 is also illustrated. Examples of sensors that may be used with the present invention include sensors for temperature, relative humidity, or cabinet door position (including a reed switch with a magnet or an equivalent mechanism), and sensors for detecting the quality of grounding connections in the cabinet 12. Other examples of sensors that can be used include those for detecting power quality, bit error rate, and horizontal cabling quality.

[0024] Environmental data collected by sensors can provide critical information to data center managers for decision-making related to physical server deployments. The environmental information can also provide useful criteria to virtualization software for its selection of the optimum hardware (including physical infrastructure and server hardware) resources to host virtual machines. Sensor data logs can also provide a record of evidence that compels hardware vendors to honor their warranty repair obligations. Software can turn data such as temperature and current data into easy-to-use and relevant information such as information about thermal capacity, power capacity, and grounding quality, for example.

[0025] Actuators 36 can also be connected to the sensing and control ports 30. Examples of actuators that can be used with the present invention include cabinet door locks for security and optical signal beacons (mounted, for example, atop each cabinet) to assist in locating targeted physical assets within large data centers.

[0026] Systems and methods of the present invention allow for physical location information regarding network assets connected to intelligent patch panels to be provided to a network management system or PIM software. In addition, the physical locations of network assets later connected to an intelligent patch panel (following initial installation of the intelligent patch panel) will be known because the physical locations of each intelligent patch panel in a network are known by software on the PIM server 15, and perhaps other network management software as may be used in specific network deployments.

[0027] The right server cabinet 12′ in FIG. 1 connects to the left switch cabinet 12 through the data center's horizontal structured cabling, depicted as cloud 14. The horizontal structured cabling between the cabinets 12 and 12′ should connect only directly from the rear of the switch cabinet intelligent patch panel to the rear of the server cabinet patch panel. While this system has been illustrated with only one switch cabinet 12 and only one server cabinet 12′, a plurality of switches may undergo distribution within several cabinets, and so may a plurality of servers. The location information methodology of the present invention will still function under these circumstances.

[0028] A method of associating physical location data and automated horizontal link detection with network assets will now be described. Returning to FIG. 1, it shows a first cabinet 12 in which switches 22 are installed and a second cabinet 12′ in which servers 24 are installed. The process of provisioning the intelligent patch panel 10 in the first cabinet 12 will be described first. The term “provisioning” as used herein refers to the process of providing information to an intelligent patch panel to allow the intelligent patch panel to gain knowledge of the equipment that is connected to it. The term also refers to the process by which the PIM software forms logical connections between an intelligent patch panel and the equipment that is connected to it. This can be done by using a predetermined procedure for serially connecting ports of an intelligent patch panel, as well as equipment within a cabinet, to a provisioning port 23 provided on the intelligent patch panel.

[0029] Using a PC to access the intelligent patch panel's management port, a data center technician assigns a unique identifier into each intelligent patch panel's nonvolatile memory to represent:

[0030] (a) Identification labeling of the cabinet (cabinet number);

[0031] (b) Location on the physical floor plan of the data center;

[0032] (c) Cabinet type (switch or server cabinet); and

[0033] (d) Network elements associated with the intelligent patch panel.

[0034] If more than one intelligent patch panel resides in a given cabinet, they all receive the same cabinet number and type identifier.

[0035] Following the installation of switches 22 and the intelligent patch panel 10 in the first cabinet, the provisioning port 23 is used with the ten-wire patch cords 18 to enable the intelligent patch panel 10 to report to the PIM server 15 which switch ports 21 are connected to which panel ports 20. This can be accomplished using a patch cord installation procedure as described in U.S. Provisional Patent Application Ser. No. 61/113,868, with information about switch port connections being collected when the switch plug of a patch cord is plugged into a switch port and the panel plug of the patch cord is connected to the provisioning port of the intelligent patch panel.

[0036] Additional exemplary, useful information that can be collected by or provided to the PIM Software includes:

[0037] (a) switch MAC address;

[0038] (b) switch IP address;

[0039] (c) switch chassis id or serial number;

[0040] (d) switch port identification, MAC address, or IP address;

[0041] (e) assigned host name;

[0042] (f) 3D visualization information in real time or real time;

[0043] (g) uptime;

[0044] (h) model and manufacturer;

[0045] (i) service level agreement information;

[0046] (j) mean time to recovery (MTTR); and

[0047] (k) mean time between failures (MTBF).

[0048] Next, the process of provisioning the intelligent patch panel 12′ will be described. According to one embodiment of the present invention, a particular provisioning method should be used when installing intelligent patch panels and other network assets in a cabinet to enable physical location information to be collected and managed by a PIM server or other management system. These steps occur after installation of the main distribution area cabinets and switches, horizontal cabling, and server cabinets, along with their intelligent patch panels:

• • [0049] a) Plug a laptop into a management port 17 located on the rear of the intelligent patch panel 10′. Assign to the intelligent patch panel's nonvolatile memory the cabinet ID (identification number) and type (server cabinet patch panel), and the physical location of the cabinet 12′.
  • [0050] b) Insert a nine-wire patch cord 18 into the server cabinet's intelligent patch panel provisioning port 23 (shown in FIG. 1). This signals the intelligent patch panel 10 to start initial mapping of the horizontal cables 16 (provisioning).
  • [0051] c) Insert the other end of the 9-wire patch cord 18 into the cabinet's first front patch panel port 20 (the cabling from the rear of this port already connects through the horizontal cabling to the targeted external switch port). The provisioning port uses CDP (Cisco Discovery Protocol) or LLDP (Link Layer Discovery Protocol) to acquire and memorize information uniquely indicative of the specific external switch and switch port (port MAC address, or switch MAC address and port ID). The intelligent patch panel 10′ commits this information to memory (and signals completion to the user).
  • [0052] d) With one end of the 9-wire patch cord 18 still inserted into the provisioning port 23, disconnect the other end from patch panel port 1, and insert it into port 2. The provisioning port 23 again acquires information that identifies the next switch port and commits the information to memory. Repeat for all of intelligent panel ports 20 in the server cabinet 12′, in other words, for all of the horizontal cables terminated in the server cabinet.
  • [0053] e) The PIM Software (resident on the PIM server 15 shown as a laptop connected to the structured cabling cloud 14) collects this information from the intelligent patch panel 10 along with its cabinet number, cabinet type, and location. The PIM software may collect this information after initial discovery of the intelligent patch panel 10, and during subsequent periodic polling of the intelligent patch panel 10.
  • [0054] f) The PIM Software resident in the PIM server 15 then:
    • [0055] looks for all instances of each identified switch port in its database (it should find only two);
    • [0056] associates each individual server cabinet's intelligent patch panel port with its matching switch cabinet intelligent patch panel port; and
    • [0057] logically joins each of these port pairs as connected to each other with a horizontal cable.

[0058] Through repetition of this method at all of the intelligent patch panel ports in all of the server cabinets in a network, the PIM software maps end-to-end connectivity of all of the copper horizontal cabling in the data center. The PIM software then knows: (1) the spatial location and logical port identification of each terminus of each horizontal cable; (2) the physical and logical location of switches and their cabinets; and (3) the physical and logical location of server cabinets and the associated intelligent patch panels. At this point in the process, the PIM software does not have information about specific server identifications or locations.

[0059] Next, the servers are patched in a method that enables the specific physical locations of servers to be known. These steps occur after the provisioning of the server cabinets in conjunction with steps (a) through (f) above. As discussed above, two different patch cord styles may be used in connection with provisioning the ports of the intelligent patch panel 10′.

• • [0060] a) Insert the blade-end of a ten-wire patch cord 18 into the targeted intelligent patch panel port 20 contained within a server cabinet 12′. The intelligent patch panel 10′ detects and memorizes the insertion into this port (PIM software has already associated this port with a switch port in, for example, a distribution area, at the other end of the horizontal cable).
  • [0061] b) Insert the switch end (“far” end) of that same ten-wire patch cord 18 into the targeted server port 25.
  • [0062] c) The intelligent patch panel port 20 detects insertion of the intelligent patch cord 18 into the server port and signals the event of this patched connection up to the PIM Software.
  • [0063] d) The PIM Software periodically queries the identified switch about its specific port known to connect at the other end of the horizontal cable. This periodic query allows elapsed time between initial patching of servers within their cabinets and the eventual onset of Ethernet communications from/to those servers.
  • [0064] e) The switch immediately or eventually yields the MAC address of the server connected to its identified port.
  • [0065] f) The PIM software designates that server MAC address as inventory resident within the server cabinet 10′.

[0066] PIM software utilizes SNMP (Simple Network Management Protocol) to enable messaging between network devices and a central management console or server. Management Information Blocks (MIBs) contain statistic counters, port status, and other information about these network devices. A management console issues GET and SET commands to operate on particular MIB variables for network equipment nodes. The various network device agents, such as those contained in distributed intelligent patch panels modules, can issue TRAP messages to the PIM server to report events such as patch disconnects and other alerts. Third-party SNMP-based network management systems, such as HP OpenView, may integrate or cooperate with Panduit PIM software if the extended MIB variable structures are shared.

[0067] PIM Software discovers all of the devices within the network. According to one embodiment, certain limitations exist with respect to the availability of switch information: the switches must have SNMP activated, and their address table information must generally be made available to the PIM software. With these permissions, the PIM software can discover and aggregate:

[0068] a) Switch MAC address;

[0069] b) Switch IP address;

[0070] c) Switch chassis id or serial number;

[0071] d) Switch port identification, MAC address, or IP address;

[0072] e) Assigned host name;

[0073] f) Model and manufacturer; and

[0074] g) The IP and MAC addresses of the active devices connected to each switch port.

[0075] Again, data center personnel assigned a unique identifier to the intelligent patch panel in the field to represent its cabinet. Since the intelligent patch panels 10 and 10′ (and therefore the PIM software) have memorized the intelligent patch panels' lists of connected switch ports, and since these switches have made their port-connection-to-server data available to the PIM software, then the PIM software can map and designate each of these connected switches and servers as physically resident within their respective cabinets. The PIM software can also map the horizontal cabling connectivity within the data center.

[0076] FIG. 4 illustrates the known connections and inferred connections in one embodiment of the present invention. When the intelligent patch panel 10 is installed, provisioned, and connected to a switch 22, the switch ID is known and associated with a particular port of the intelligent patch panel 10. This is indicated by arrow 40 in FIG. 4. Further, when the intelligent patch panel 10′ is installed and provisioned, the PIM software is able to complete its knowledge of the link between the switch 22 and the particular port of the intelligent patch panel 10′. This is shown by arrow 42. An inferred knowledge of the horizontal cabling connection between the rear panel of the first intelligent patch panel 10 and the rear panel of the intelligent patch panel 10′ is also gained at this time. This inferred knowledge is shown by arrow 43 in FIG. 4. Finally, when the server ID query is accomplished (by obtaining the server ID from the corresponding switch port to which it is connected), the PIM software learns the server ID for each server port that is connected to each front port of the second intelligent patch panel 10′. This is indicated by arrow 44 in FIG. 4.

[0077] FIG. 5 illustrates data known by the PIM software following provisioning steps as outlined above. The left data set 46 shows information known by the PIM software based on information provided to the software by a first intelligent patch panel 10 in a switch cabinet 12. The right data set 48 shows information known by the PIM software based on information provided to the software by a second patch panel 10′ in a server cabinet 12′. The highlighted data strings, 50 and 52, show data that are linked by the PIM software based on the software learning that a particular horizontal cable joins switch port number 13 through first intelligent patch panel port number 7, to second intelligent patch panel port 21 to a particular server port. Note that in FIG. 5, the server ID is still to be learned from the switch by the PIM software.

[0078] Knowing the physical locations of the intelligent patch panels of the present invention, it is possible for the PIM software to go beyond a typical topological map 54 of network assets, as shown in FIG. 6 to provide a physical location map 56 of network assets as shown in FIG. 7. Using the techniques and hardware of the present invention, it is possible to illustrate to a network administrator or user which specific cabinet (cabinets 12a-e are illustrated in FIG. 7) particular network assets are located in. By knowing the size of the network cabinets at issue, the PIM software is also able to inform an administrator about how much free space 58 is available for additional hardware at specific cabinets. The automated knowledge of free space 58 can be combined with the knowledge of environmental and power usage information to allow an administrator to intelligently plan for network expansions without the need for a direct physical inspection of the expansion area. Systems and methods of the present invention can also be used to assist in the creation of work orders by providing location information and other information to technicians.

[0079] The ability of a physical infrastructure management system as disclosed herein to automatically reconcile itself (i.e., to confirm that all connections have been implemented as intended and flag any improper implementations) enables automated decisions to be made, such as the virtual movement of an application from one server to another, without the need for a manual verification or reconciliation step.

[0080] In the above description, it is to be understood that references to “cabinets” refer to one embodiment of the present invention. It is also possible for the physical location information and other benefits of the present invention to be provided with reference to racks or rooms or other relevant physical locations (i.e., “containers”) that house intelligent patch panels.