Apparatus and method for downloading core file in a network device

[0002] This invention relates to a method and apparatus for debugging faults occurring in a router or other network device and more particularly to compressing core file and storing the compressed core file into an internal flash memory.

[0003] Network servers and other types of network devices often experience unrecoverable faults. One example of an unrecoverable fault occurs when a routine writes an invalid address value into core memory. When a process tries to access the illegal address value, a fault occurs. For example, a process may request a memory address for a status register used for conducting a direct memory access (DMA) operation. If the memory address is invalid, a fatal error occurs when the process attempts to access the memory address, which causes the router to reset.

[0004] Viewing core files is vital to resolving fatal fault errors. A core file is essentially a copy of DRAM which contains the program, program pointers, program variables, etc. The core file provides a snap-shot of the router at the time the fault occurred. DRAM is used to meet performance requirements of the system and since the contents of the DRAM are destroyed after a reset operation, the core file must be downloaded to another storage device. Routers can be equipped with some flash memory. However, due to the cost of flash memory, the flash memory is not large enough to hold all DRAM contents. Thus, the core file must be downloaded to an external server connected to the router through a local area network (LAN). The core file can then be analyzed by an engineer from a computer or workstation to identify the source of the fault.

[0005] The problem with copying a core file to an external device is that the fault condition causing the router to shutdown may be caused by a process that must be operational in order to download the core file. For example, the fault may be caused by a software error with a network protocol or LAN media drivers. If these network interface processes are not operational, the core file cannot be successfully downloaded to an external network device. Thus, in the past, a special image had to be created in order to investigate the fault. The special image is produced by modifying operating code to print out specific identified information before the fault occurs. Generating special images to locate faults requires a large amount of trial and error which is extremely time consuming. Alternatively, the router is taken out of production so that the current content of the main memory can be analyzed with a ROM monitor.

[0006] Accordingly, a need remains for a faster more reliable way to save core file after a fault condition occurs in a network device.

[0007] A network device, such as a router or switch, downloads a core file into a local flash memory. In order to increase storage capacity, the core file is compressed before being dumped into the local flash memory. The flash memory is local and internal to the network device. Because network interface elements do not have to be functional for a successful core download, the core download is faster and more reliable than existing download techniques.

[0008] In one embodiment, the network device comprises a router having a CPU for controlling packet processing operations. DRAM is used for a main memory and its contents constitutes the core file. Network interface elements are coupled between the CPU and different external networks. The network interface elements process and route the packets received from the external networks. The core file is downloaded from the main memory to local flash memory independently of these network interface elements.

[0009] During the shutdown routine, interrupts are disabled for any processing elements, such as the network interface elements, that are not needed to perform the core download. Thus, the CPU is not interrupted by routines that could generate additional fault conditions. Because these processing elements are disabled, the DRAM contents cannot be modified by other processes that might be operating after the fault condition. Thus, the core file will more accurately represent a snapshot of the system at the time the fault condition occurred.

[0010] In one embodiment of the invention, the CPU downloads the core file to the same local flash memory used for storing the router operating routine and the router shutdown routine. Router platforms may contain more than one flash memory device and different flash memory configurations. The network device can also be configured by a user to download all or part of the core file into one or more of the different flash memory devices used in the specific platform.

[0011] In order to increase download capacity, each byte of the core file is compressed using a standard compression routine. The compressed core file is written into a temporary buffer in main memory. Once the temporary buffer is full, the contents of the buffer are downloaded into the local flash memory.

[0012] The router is coupled to a network server through a LAN. The router is reset after completing the core download. The server uses a file transfer operation to access the router and read the core file from local flash memory. The core file is then analyzed to determine the state of the router when the shutdown event occurred.

[0013] The foregoing and other objects, features and advantages of the invention will become more readily apparent from the following detailed description of a preferred embodiment of the invention which proceeds with reference to the accompanying drawings.

[0014] FIG. 1 is a block diagram of a network device according to one embodiment of the invention.

[0015] FIG. 2 is a detailed diagram of processing elements in the network device shown in FIG. 1.

[0016] FIG. 3 is flow diagram showing how the network device operates according to the invention.

[0017] Referring to FIG. 1, a network device 12 is shown in dashed boxes 40 and 41 and is coupled to a LAN 25. A network device is defined as any system that processes data or communicates through a network. In one embodiment of the invention, the network device 12 comprises a router that processes and transfers network packets to and from different external devices on different networks or buses. A server 26 is coupled to the router 12 through the LAN 25. The router 12 includes a CPU 14 coupled through an internal bus 13 and a system arbiter 16 to a main memory 18. The main memory 18 comprises a Dynamic Random Access Memory (DRAM). Multiple memory devices are coupled to bus 13 and include a flash/Read Only Memory (ROM) 20 used for router bootup, an Electrically Erasable Read Only Memory (EEROM) 21 used for configuring the router 12, and flash memories 22 and 24 used for storing router routines. A PCMCIA card 42 connects the router 12 to PCMCIA compatible devices (not shown).

[0018] Multiple network interface elements are shown in dashed box 40 and are used to connect the router 12 to different networks. In the example shown in FIG. 1, the network interface elements 40 include a packet memory arbiter 46 that arbitrates access to a packet memory 44 between an Ethernet or token ring controller 50 and a serial bus controller 54. A LAN media interface 52 is coupled between LAN 25 and controller 50. Serial interfaces 32 and 34 are coupled between serial lines (not shown) and controller 54.

[0019] Three slots 59 are connected to data bus connections 55, a Direct Memory Access (DMA) bus 57 and a Time Division Multiplex (TDM) bus 39. Telephone line interface cards and modem cards (not shown) are coupled to the slots 59. Calls received by the telephone line interface card are coupled to the modems through the TDM bus 39 or sent over the DMA bus 57 to the packet memory arbiter 46. A console 28 accesses the internal bus 13 through a DUART 27. Other devices access the processing elements in router 12 through an auxiliary port 30 also coupled to the DUART 27. The network interface elements 40, CPU 14 and internal memory devices are referred to generally as processing elements.

[0020] The general operation of the processing elements described in FIG. 1 are known to those skilled in the art and are therefore not described in further detail. One router using the architecture shown in FIG. 1 is the Model No. 5200 router manufactured by Cisco Systems, Incorporated, 170 West Tasman Drive, San Jose, Calif.

[0021] Referring to FIG. 2, the CPU 14 includes an interrupt handler 60 that receives interrupt requests from the different processing elements in the router 12. The interrupt handler 60 jumps to different routines that service the interrupt requests made by the different processing elements. Interrupt handlers are well known and are, therefore, not described in further detail. The main memory 18 stores the information that constitutes the core file for the router 12. Core file 61 includes the values of stack pointers, routine variables, the last operating instruction, values set by the last operating instruction, status register addresses, program counters and any other data stored in the main memory 18.

[0022] The local flash memory 22 stores a system image that includes operating routines 62, a shutdown routine 64, a compression routine 66 and a flash core copy routine 67. The operating routines 62 include bootup routines, routing protocols, device drivers, configuration routines, etc. and any other routines used by the router to process data. The CPU 14 starts the shutdown routine 64 after detecting a shutdown event. The shutdown routine 64 uses the flash core copy routine 67 to download the contents of main memory 18 to local flash memory 22. The flash core copy routine 67 also calls the compression routine 66 that compresses the contents of main memory 18 before being downloaded to local flash memory 22. The boot flash memory 24 contains a boot program 65 used by the router 12 to boot the operating routine 62 after a reset. The flash core copy routine 67 can alternatively copy part of the core file 61 into a portion of the boot flash memory 24 (core file #2).

[0023] During initial configuration of the router 12, space is preallocated in main memory 18 for a temporary buffer and memory required for compression routines. If space in main memory 18 is allocated to other processes, the CPU 14 might not be able to successfully allocate space in main memory 18 for the temporary buffer when a shutdown event occurs. By preallocating space in main memory 18, the flash core copy routine 67 is assured of having sufficient space for compressing and downloading core file 61.

[0024] Because volatile DRAM is used for the main memory 18, the contents of the main memory 18 are lost any time the router 12 is reset. Shutdown events causing a reset occur for any one of a variety of software or hardware faults. For example, a shutdown event occurs when a process loads an invalid address into main memory 18. When another process tries to use the invalid address, a bus error occurs causing the interrupt handler 60 to call the shutdown routine 64.

[0025] If the shutdown routine attempts to download the core file 61 to server 26 (FIG. 1) via a FTP command, the network interface routine used to conduct the FTP operation may be the same routine causing the fault. The CPU 14 would then be unable to successfully download the core file 61 to server 26.

[0026] The flash core copy routine 67 according to the present invention solves this problem by downloading the core file 61 to non-volatile local flash memory 22. Thus, the contents of the core file 61 will not be destroyed when the router 12 is reset. Because the flash core copy routine 67 downloads the core file to local memory, operational status of network interface routines and devices will not affect the core file download process.

[0027] The flash core copy routine 67 disables interrupts for all processing elements in the router 12, other than those processing elements used for downloading the core file 61 into local flash memory 22. For example, the CPU 14 has multiple levels of interrupt priority. When a shutdown event occurs, the CPU 14 is brought up to a higher interrupt level ignoring interrupts at lower levels. Disabling interrupts keeps the CPU 14 from having to service requests generated by interface elements 40 while downloading the core file 61 into local flash memory 22.

[0028] Because other interrupts are disabled, the shutdown routine is not disrupted by the interface elements 40 or other processes. If not disabled, the data in main memory 18 could continue to be modified by the interface elements 40 after the shutdown event. By disabling all unnecessary processing elements, the core file provides a more accurate snapshot of the system at the time the system crash occurred.

[0029] FIG. 3 describes how the flash core copy routine 67 downloads the contents of main memory 18 into local flash memory 22 according to the invention. The CPU 14 in step 70 runs a standard boot routine 65 in ROM/FLASH memory 20 that boots an operating routine. Step 72 runs the operating routine. After an instruction is completed in the operating routine, the CPU 14 checks for interrupts from any one of the processing elements in the router 12. If an interrupt request is detected, the CPU 14 services the interrupt then continues running the operating routine in step 72.

[0030] If a fatal error occurs in decision step 74, the CPU 14 first stores the address location of the operating routine on a program stack pointer. The address pointer for the shutdown routine 64 is read by the CPU 14. In step 76, the shutdown routine 64 calls the flash core copy routine 67 which disables the interrupts for the network interface elements 40 and any other processing elements that are not needed to download the contents from main memory 18 to local flash memory 22.

[0031] The flash core copy routine 67 reads 1 byte from the DRAM 18 in step 78. Step 80 uses the compression routine 66 to compress and store the compressed byte from DRAM 18 into the temporary buffer in main memory 18. If the temporary buffer is full in decision block 82, the compressed data in the temporary buffer is downloaded into the local flash memory 22 in step 84. Any standard compression routine, such as compression routines using a standard hash algorithm, can be used to compress the core file. One hash based compression routine is explained in U.S. Pat. No. 4,558,302 to Welch.

[0032] After the temporary buffer is downloaded into flash memory in step 84, or if the temporary buffer is not full in decision step 82, decision step 86 determines whether there are any more bytes in the main memory DRAM 18. If all bytes of the main memory 18 have been compressed, any remaining compressed data in the temporary buffer is downloaded into local flash memory in decision step 88. If there is more data in main memory 18, decision step 86 reads the next byte in step 78.

[0033] The compressed core file 61 can be loaded into the same local flash memory 22 that stores the operating routine 62 and the shutdown routine 64. Part or all of the compressed core file can also be stored in boot flash memory 24. If there is insufficient space in local flash memories 22 and 24, the flash core copy routine 67 stores as many 4K blocks of compressed core file 61 as possible. The remainder of the core file 61 is then downloaded word by word until there is no more space available in the local flash memories. After the compressed core file 61 is downloaded into local flash memory 22, and possibly flash memory 24, the router 12 is reset in step 90.

[0034] Usually after the router 12 is reset, the previous fault condition causing the shutdown no longer exists. The compressed core file 61 in local flash memory 22 can then be transferred over LAN 25 using an internet protocol command initiated from the server 26 or router 12. However, if the network command fails, the compressed core file 61 in local flash memory 22 can be accessed through the console 28 or other devices coupled to auxiliary port 30.

[0035] Having described and illustrated the principles of the invention in a preferred embodiment thereof, it should be apparent that the invention can be modified in arrangement and detail without departing from such principles. I claim all modifications and variation coming within the spirit and scope of the following claims.