Low-level programming language plugin to augment high-level programming language setup of an SDN switch

[0001] This application claims priority under 35 U.S.C. § 119 from U.S. Provisional Application No. 62/171,230, entitled “Low-Level Programming Language Plugin to Augment High-Level Programming Language Setup of an SDN Switch,” filed on Jun. 4, 2015; the subject matter of which is incorporated herein by reference.

[0002] The described embodiments relate generally to programmable SDN switches.

[0003] An SDN node is generally configured at configuration time, and then at run-time packet traffic is received into the SDN node via one of a number of input ports. The SDN node performs parsing of the packet, to identify various fields. The SDN node then matches against the fields to determine whether the packet matches a particular flow entry. From the matching flow entry, the SDN node then determines what action or actions to take. For example, the action may be to output the packet from a particular output port of the SDN node. Configuration at configuration time generally involves specifying how incoming packets will be parsed, how matching will be performed, and a set of actions that can be taken if a match is detected. For example, the OpenFlow 1.4.0 standard (as set forth by the Open Networking Foundation in “OpenFlow Switch Specification”, version 1.4.0, Oct. 14, 2013, available from www.opennetworking.org) provides a predefined high-level structure for defining data path handling of a packet. Using OpenFlow commands, one or more match tables are set up in the SDN node, and the arrangement of those match tables is specified. In addition, OpenFlow commands are used to specify what the different possible actions are that might be indicated if a match occurs. Using other OpenFlow commands, the match tables can be populated with flow entries. In addition, at run-time once the SDN node is operating and is handling packet traffic, OpenFlow commands can also be communicated to the SDN node, thereby causing the SDN node to insert new flow entries into its match tables, or to delete flow entries from its match tables. The overall setup, and ways that the flow tables can be arranged, is specified by the OpenFlow standard.

[0004] The P4 programming language (“Programming Protocol-Independent Packet Processors” domain-specific language, as set forth by the P4 Language Consortium, at www.P4.org) is usable to provide an alternative to OpenFlow in that a user can use P4 language programming code to define, in a more flexible way, the data plane structure to be employed by the SDN node. For example, P4 is usable to set up a program-defined run-time data path, rather than just a table-based run-time data path as is used in OpenFlow. How the parsing of incoming packets is performed, and how and what the match tables are, and what the actions are and do is expressed in the P4 language. For example, P4 can be used to specify that a packet that matches in a particular way will be traffic-managed in a particular way and will be output from a particular output port of the SDN node. If P4 is employed, then the P4 code that sets up the data plane structure is compiled into native code. The native code is code that can be executed by the processor of the SDN node. The native code is loaded into the SDN node, and the processor then executes the code so as to realize the desired data plane packet handling structure.

[0005] In a first novel aspect, a first amount of high-level programming language code and a second amount of a low-level programming language code is compiled thereby obtaining a first section of native code and a second section of native code. The first amount of high-level programming language code at least in part defines how an SDN switch performs a matching in a first condition. The second amount of low-level programming language code at least in part defines how the SDN switch performs matching in a second condition. The first section of native code is loaded into the SDN switch such that a first processor of the SDN switch can execute the first section of native code. The second section of native code is loaded into the SDN switch such that a second processor of the SDN switch can execute the second section of native code.

[0006] In one example, the SDN switch receives and processes packets such that first packets are matched in the SDN switch in accordance with a first matching specification defined by first native code that results from compiling the first amount of high-level programming language code. The SDN switch receives and processes packets such that first packets are matched in the SDN switch in accordance with a second matching specification defined by second native code that results from compiling the first amount of high-level programming language code.

[0007] In another example, a first part of the first native code that results from compiling the first amount of high-level programming language code is a part of the first section of native code that is loaded in the SDN switch such that a first processor of the SDN switch can execute the first section of native code. A second part of the first native code that results from compiling the first amount of high-level programming language code is a part of the second section of native code that is loaded in the SDN switch such that a second processor of the SDN switch can execute the second section of native code.

[0008] In yet another example, the high-level programming language code is P4 programming language code and the low-level programming language code is C programming language code.

[0009] In another example, the compiling occurs in the SDN switch. Alternatively, in another example the compiling does not occur in the SDN switch.

[0010] In another example, the first processor is an x86 host processor that executes a first instruction set and the second processor is a processor of a Network Flow Processor (NFP) that executes a second instruction set.

[0011] In a further example, the matching specified by the second amount of low-level programming language code cannot be specified using the high-level programming language.

[0012] In another example, the first section of native code is combined with the second section of native code into a single file. In one scenario the file is an Executable and Linkable Format (ELF) file.

[0013] In a second novel aspect, a first amount of high-level programming language code and a second amount of a low-level programming language code is compiled thereby obtaining a first section of native code and a second section of native code. The first amount of high-level programming language code at least in part defines how an SDN switch performs a first action, and wherein the second amount of low-level programming language code at least in part defines how the SDN switch performs a second action. The first section of native code is loaded into the SDN switch such that a first processor of the SDN switch can execute the first section of native code. The second section of native code is loaded into the SDN switch such that a second processor of the SDN switch can execute the second section of native code.

[0014] In one example, the first processor is an x86 host processor that executes a first instruction set, and wherein the second processor is a processor of a Network Flow Processor (NFP) that executes a second instruction set.

[0015] In another example, a third section of native code is obtained by the compiling of the first amount of high-level programming language code and the third section of native code is used during the compiling of the second amount of low-level programming language code.

[0016] Other embodiments and advantages are described in the detailed description below. This summary does not purport to define the invention. The invention is defined by the claims. Other structures and methods are described.

[0023] Reference will now be made in detail to background examples and some embodiments of the invention, examples of which are illustrated in the accompanying drawings.

[0024] FIG. 1 is a diagram of a system (in this example, a data center system) 1 that includes a network operator's work station 2, a network controller 3, a gateway node 4, a plurality of spine nodes 5-7, a plurality of leaf nodes 8-11, and several racks 12-15 of server nodes. Reference numerals 16-20 identify some of the server nodes. The nodes (or switches) are network switch appliances that are interconnected and that can intercommunicate packets between them. Each receives packets and output packets. In one example, a packet received from the internet 21 is received by the gateway node 4. The gateway node 4 adds a tag or otherwise encapsulates the packet so that the packet can be forwarded between the various nodes of the system using the tag or encapsulating information. In this way, the packet may be forwarded from the gateway node, through a spine node, to the leaf node, and to a destination server node. The server node may then examine the packet, and then output the packet, or another a packet. In one example, the packet in modified form passes from the server node, back through various other nodes, and back to the gateway node. The gateway node determines that the packet is to be output to the internet, so the gateway node strips off the tag, and outputs the packet to the internet. Each node in this example is a Software Defined Networking (SDN) switch appliance within a data center 22. The SDN node is configured and controlled by the central network controller 3. In one example, the OpenFlow protocol is used by the network controller 3 to set up and provision flow entries into the OpenFlow flow tables in the various nodes. In another example, the operation of the nodes is not setup and controlled using OpenFlow, but rather how the nodes perform parsing, matching and actions is setup and specified in another way. Regardless of the data plane parsing, matching and actions of the nodes are setup, the network controller 3 at run-time can modify and add and delete match specifications and action specifications as the node operates.

[0025] FIG. 2 is a more detailed diagram of one of the nodes of FIG. 1 in accordance with one novel aspect. In this particular example, the node is one of the servers, server 16. The node 16 includes a host computer motherboard 23, and an expansion Network Interface Card (NIC) card 24. The host computer and the NIC intercommunicate using a PCIe bus 25. The host computer 23 includes a first processor P1 that is of a first type. In the present example, processor P1 is an x86-architecture processor. The NIC card 24 includes a Network Flow Processor (NFP) integrated circuit 26. The NFP 26 in turn includes multiple processors of a second type. In the present example, processor P2 and processor P3 are MicroEngine (ME) architecture processors. The processors of the first and second sets differ in multiple ways. One way they differ is that they execute native instructions of different instruction sets. During run-time, packets are received onto the node 16 as indicated by arrow 27. The NIC performs parsing on the incoming packets, and attempts to match in accordance with matching specifications. If a packet is determined to be match, then the NFP 26 performs an associated defined action or actions. In one example, if a match is detected, then the flow entry identified by the match specifies an indicator of an action. The indicated action is to output the packet from a specific output port of the NFP. Arrow 28 in the diagram illustrates the output packet traffic from SDN node 16. Although not illustrated, the NIC 24 typically has many different input ports via which packets are received, has many different output ports out of which packets can be transmitted. Under some conditions, for example if a packet is received but a match is not detected, then the NFP may forward the packet across PCIe bus 25 to the host computer. The processor P1 of the host computer then executes code and analyzes the packet. As a result of execution of this code, the processor P1 determines an action to take. The action may, for example, be to return the packet to the NFP and to instruct the NFP to output the packet in a particular way from a particular output port.

[0026] In accordance with one novel method, a program 30 includes a first amount of programming code 31 and a second amount of programming code 32. The first amount of programming code 31 is written in a high-level programming language. The first amount of code defines first matching requirements and first actions for the SDN node. In the present example, the high-level programming language is the P4 programming language. The second amount of programming code 32 is written in a low-level programming language. The second amount of programming code defines second matching requirements and/or second actions for the node. In the present example, the low-level programming language is the C programming language. In one example, these two amounts 31 and 32 of code are written by the network operator using work station 2.

[0027] Next, the two amounts 31 and 32 of code are “translated” 33 into native code. Translation, as the term is used here, means compiling and linking as necessary so that an appropriate section of native code is generated for each of the processors P1, P2 and P3. The code for each processor is in the native code that is executable by that processor. Together the various processors, each executing its own section of native code, carry out of the overall data plane handling of packet traffic, as set up by the network operator. In the illustrated example, code section C1 is native code for execution by the P1 processor, the code section C2 is native code for execution by the P2 processor, and the code section C3 is native code for execution by the P3 processor.

[0028] Next, each of the sections C1, C2 and C3 of native code is loaded 34 into the appropriation processor. C1 is loaded into the host computer so that it can be executed by the host CPU P1. C2 is loaded in the NFP of the NIC, so that the MicroEngine processor P2 can execute that code. C3 is loaded in the NFP of the NIC, so that the MicroEngine processor P3 can execute that code. This setting up and configuring of the SDN node 16 to have a desired data plane is performed at configuration time of the node. After configuration time, the SDN node 16 is then operated at run-time. During the run-time the SDN node 16 handles packet traffic using the setup data plane.

[0029] FIG. 3 is a diagram of the structure of the overall program 30. The notation “H/L” in a block denotes that the function performed by that particular block is defined by the first amount of code 31 that is written in the high-level programming language. The notation “L/L” in a block denotes that the function performed by that particular block is defined by the second amount of code 32 that is written in the low-level programming language. A first parsing operation 36 is defined and specified by the high-level language code. For example, the P4 language can be used to specify that after all MAC headers, that a next IP header may be detected, and if it is detected it would be of a certain number of bytes, and would have a particular format. If that specific type of IP header is detected by the H/L parse operation 36, then that header should be supplied to the match function. In addition, a second parsing operation 37 is defined and specified by the low-level programming language code. For example, the C programming language can be used to specify that after all MAC headers, that a next user-defined header may be detected, and if it is it would be of a certain number of bytes, and would have a particular format. If that special user-defined header is detected by the L/L parse operation, then that special user-defined header should be supplied to the match function.

[0030] A first match operation 38 is defined and specified by the high-level programming language code. For example, the P4 language can be used to specify an arrangement of match tables, and to specify which values as output by the parse operation are to be matched against which match values, and in which order the matching is to occur. In addition, a second match operation 39 is defined and specified by the low-level programming language code. For example, the C programming language can be used to specify an analysis or test to be performed on various portions of the packet as output by the parser operations, where the result of the analysis or test is then supplied to the action function. As compared to the P4 programming language, the C programming language is a lower-level, and more powerful, language that can operate on any arbitrary part of a parsed field or payload, including parts of the data of the payload, and that can make decisions based on stored state, and that can implement multi-level decision trees, and that can utilize bit-wise Boolean, arithmetic operations, and complex logic tests at the bit level. In one example, if either the H/L match operation 38 detects a match or if the L/L match operation detects a match 39, then an indication of the detected match is supplied to the action operation. The software of the program 30 is flexible and can be written such that in some cases of execution of the resulting native code only matching as defined by the high-level programming language is performed, whereas in other cases of execution the result of high-level programming language defined matching in turn causes low-level programming language defined matching to be initiated. Alternatively, the software of the program 30 can be written so that each of the high-level programming language defined matching block 38 and the low-level programming language defined matching block 39 output a true or false indications. Only if both output true (indications that both blocks 38 and 39 returned a “match” detection) is the associated action condition (as defined by one or the other of blocks 40 and 41) initiated.

[0031] A first action operation 40 is defined and specified by the high-level language code. For example, the high-level language can setup and specify that if a match in a particular match table occurs, then a predefined action to output the packet from a particular NIC output port is the action. A second action operation 41 is defined and specified by the amount of low-level programming code. The C programming language may, for example, be used to set up flexible and powerful bit-wise complex modifications of various parts of the incoming packet in order to generate an output packet, where the type of packet is not a conventional packet and where the modification to be done is not done in the handling of packets of any conventional protocol. The second action of action operation 41 may be an action to perform further actions and manipulations in addition to actions and modifications that are indicated to be done by the high-level programming language defined action of operation block 40. The second action operation may also perform actions and manipulations that are performed in conditions in which no actions are indicated to be taken by the high-level operation block 40.

[0032] A first QOS/TM operation 42 is defined and specified by the high-level programming language code. This code defines egress processing, for example including egress modification and QoS (Quality Of Service) and traffic management handling. A second egress operation 43 and 44 is defined and specified by the low-level code. In the present example, if the egress packet meets certain match criteria as determined by operation 53, then a specific action is taken where the action as defined by operation 44 is egress handling of the packet so as to fine tune traffic management.

[0033] In one example, the handling of packets specified by the high-level language code implement a data plane as would be found in an OpenFlow switch where only packets of conventional protocols are handled by the NFP on the NIC, and packets having unconventional or unrecognized formats must be forwarded from the NIC to the host CPU. The amount of low-level programming language code, however, is a type of plugin or patch that allows the SDN 16 node to recognize and handle special packets that employ a fully user-defined protocol employing customized headers. The user-defined protocol may be a layer 2 protocol, or a layer 3 protocol, or a layer 4 protocol. Switch operation in accordance with the special user-defined protocol can require fine-granularity bit-wise manipulation and the use of complex logic decision trees, and logical decision making employing conditional logic equations and expressions, and may use stored state information, where these manipulations and decisions are not possible using only the conventional high-level programming language P4 to setup and configure the SDN node 16. In one example, the network operator of the data center uses the high-level programming language P4 to setup and define and configure the nodes of the data center 22 to perform a custom type of tunneling, where specials headers are employed in this tunneling scheme that are not understood by nefarious entities who attempt to listen in or snoop on communications within the data center without the network operator's authorization. The low-level programming language code is used to set up and configure the nodes to handle details of the protocol, for example to detect and to block attacks in special circumstances, and to then take special actions. The native code compiled from low-level programming language code, when triggered on the node, can operate on parts of the data payload and can cause complex bit-wise actions and logical tests to be performed in a way that cannot be specified by the high-level programming language.

[0034] In a second embodiment, the amount of high-level programming language code 31 is not provided to setup the first matching requirements and first actions, but the SDN node 16 is OpenFlow compliant and implements an OpenFlow data path flow table scheme for defining the arrangement of flow tables and how matching is done. According to OpenFlow, certain header fields are to be matched against certain flow tables, where the flow tables have flow entries to be supplied at run-time. The amount of low-level programming language code 32 is, however provided. This amount of low-level programming language code 32 may, for example, define a special specific action. As a result of translation and loading into the node, an action table has a flow entry, where the action value of the flow entry is associated with an amount of native code. The native code is code that was generated by compiling the amount of low-level programming code. In one example, this native code is loaded into the NFP or the NIC. At run time a packet is received onto the NIC. The NFP performs parsing and matching as set up according to OpenFlow rules, but the packet is found to match the flow entry whose action indicates the amount of native code. As a result, one of the processors P2 or P3 executes the amount of native code. Due to the use of the low-level programming language, the action can be an action that cannot be performed by the SDN node if the SDN node is simply operated as a conventional OpenFlow switch. The action may be a complex fine granularity action that the hardware egress circuitry of the NFP is not predesigned to perform. Using the low-level programming language plugin aspect and feature of the present invention, however, a complex egress operation can be specified in the low-level programming language so that it can then be performed in software (due to execution of the resulting native code by a processor on the NFP).

[0035] A compilation process to create a combined native code program is illustrated in FIG. 4. The combined native code program is composed of program 1 and program 2, where program 1 is expressed in a high level language (HLL), for example P4, and where program 2 is expressed in a low level language (LLL), for example C. Steps 52-62 are performed to create the combined native code program for a target processor. In one example, the target processor is processor P2 shown in FIG. 2. In another example, the target processor is processor P3 shown in FIG. 2.

[0036] Program 1 may invoke a certain part of program 2 at a certain stage of the execution of program 1, for example in the case of program 1 expressing the behavior of a network switch, program 2 may be invoked to perform matching (classification) of network packets, modification of network packets, e.g. field modification or encapsulation/decapsulation (actions). In these cases, execution of program 1 will continue after the relevant part of program 2 has executed.

[0037] Another option is for program 1 to complete processing of a data item (for example a packet in the case of a network switch), with the data item being subsequently passed to program 2 for processing. As this is tantamount to the portion of program 1 executing after program 1 being empty, it is not explicitly covered here.

[0038] The following description covers compiling the programs to native code for processors P2 and P3, where P2 is a core on a multi-core processor chip, and P3 is another core on the same multi-core chip. The following description further covers where P2 and P3 are the same type of processor (for example microengines—MEs—on a Network Flow Processor), as depicted in FIG. 2.

[0039] The case where the two processors are cores located on different chips, or are different types of processors (for example where they are actually P1 and P2 in FIG. 2), while not explicitly covered by this description, follows analogously.

[0040] In all cases, the processors need to be able to exchange data being processed (e.g. packets) by sending the data back and forth over a communication channel and/or by depositing the data in memory accessible to both of the processors.

[0041] The result of compilation is in the case being described a single file combining the result of compilation for both processors.

[0042] Where the processors are located in different chips, the final combining (linking) step may be omitted, resulting in two different files to be executed on the two processors.

[0043] In step 52, Program 1 in HLL native code is compiled to program 1 in HLL intermediate representation (IR), where program 1 in HLL IR expresses the semantics (behavior) of the program by using a standardized syntax, for example a text based syntax like JSON, YAML or XML, or where the IR expresses the semantics of the program using data structures in the memory of the compiler, for example an abstract syntax tree. The program 1 in HLL IR does not represent the original textual representation of the program 1 (i.e. its syntax) as-is; for example, comments are removed and alternative representations of the same required behavior (semantics) may be transformed into a single representation. It does however contain sufficient information to unambiguously and accurately reflect the desired behavior that was expressed in the original program.

[0044] In step 54, program 1 in HLL IR is compiled to P2 native code. The compiler first performs lexical analysis of the input program 1 in HLL IR (stream of text), whereupon the resulting set of tokens is processed by a parser, yielding an abstract syntax tree. This can be emitted to the IR as-is, or optimizations which simplify and speed up subsequent execution of the program without affecting the semantics of the program can be invoked at this stage. Constructing a suitable compiler is simplified using commercial or open source lexical analysis and parsing toolkits. A ready-made compiler (obtained from a commercial vendor or open source project) may of course alternatively be obtained. For the case of the HLL being the P4 language, such a compiler is available at the www.P4.org web site, from which you can download a P4 compiler and load it on to a computer system, such as the network operator's work station 2 shown in FIG. 2. The P4 compiler can then be executed on the computer system.

[0045] After completion of step 54, program 1 in HLL IR is now compiled and transformed to program 1 in native code for P2, either by directly transforming the program 1 HLL IR to native binary code, or by transforming it to a lower level text based language (for example assembler or C) and subsequently invoking a compiler/assembler with the ability to take as input the lower level language to produce the native binary code.

[0046] In step 58, Program 2 in LLL native code is compiled to program 2 in LLL intermediate representation (IR), where program 2 in LLL IR expresses the semantics (behavior) of the program by using a standardized syntax, for example a text based syntax like JSON, YAML or XML, or where the IR expresses the semantics of the program using data structures in the memory of the compiler, for example an abstract syntax tree. The program 2 in LLL IR does not represent the original textual representation of the program 2 (i.e. its syntax) as-is; for example, comments are removed and alternative representations of the same required behavior (semantics) may be transformed into a single representation. It does however contain sufficient information to unambiguously and accurately reflect the desired behavior that was expressed in the original program.

[0047] In one embodiment, program 2 in LLL is compiled along with compiled program 1 code created during step 54. In this embodiment, the program 2 in LLL IR utilizes information gathered during the compiling of program 1 in HLL. For example, program 1 may determine that packets being parsed are ethernet packets and for each ethernet packet determine an ethernet source address, ethernet destination address and an ethernet type. In the event that program 2 requires ethernet source address, ethernet destination address or ethernet type information, program 2 can attain the ethernet information determined during the compiling process of program 1, thereby avoiding the need to repeat the determination of the same ethernet information during the compiling of program 2.

[0048] In step 60, program 2 in LLL IR is compiled to P3 native code. The compiler first performs lexical analysis of the input program 2 in LLL IR (stream of text), whereupon the resulting set of tokens is processed by a parser, yielding an abstract syntax tree. This can be emitted to the IR as-is, or optimizations which simplify and speed up subsequent execution of the program without affecting the semantics of the program can be invoked at this stage. Constructing a suitable compiler is simplified using commercial or open source lexical analysis and parsing toolkits. A ready-made compiler (obtained from a commercial vendor or open source project) may of course alternatively be obtained. For the case of the LLL being the C code, such a compiler is readily available.

[0049] After completion of step 60, program 2 in LLL IR is now compiled and transformed to program 2 in native code for P3, either by directly transforming the program 2 LLL IR to native binary code, or by transforming it to a lower level text based language (for example assembler or C) and subsequently invoking a compiler/assembler with the ability to take as input the lower level language to produce the native binary code.

[0050] In step 62, program 1 in P2 native code is combined with program 2 in P3 native code. The combined native code program is then programmed into processors P2 and P3. In one example, the combined file includes the code for the P2 processor is stored in one section of the combined file and the code for the P3 processor stored in another section of the combined file. Each section within the combined file identifies the process to which the code in the section belongs. This is may be identified by storing the applicable processor number in each section of the combined file. This enables the loader to write the code image from the appropriate section of the combined file into the correct processor's code storage. A combined file may be an Executable and Linkable Format (ELF).

[0051] Program 1 in P2 native code needs to be directly executable on P2. For the case of the HLL being the P4 language, the program will for example amongst others express packet parsing and matching operations, as well as actions that need to be performed when packets match configured rules. Actions may include forwarding or dropping (discarding) packets, or may require packet modification, for example changing header fields or encapsulation/decapsulation.

[0052] For P4 HLL, packet parsing is transformed to native code by emitting native code instructions that read packet data from the memory where packets are stored (on-chip or off-chip memory), and examining the data to decode the outermost header into packet field variables (typically stored in on-chip memory). Depending on the values of the fields that identify which higher layer protocol is embedded in the outer header, a contained header may also be read and decoded into individual fields, with the process being repeated until all fields to which the P4 program refers having been decoded.

[0053] Additionally for P4 HLL, packet matching is transformed to native code by generating matching code customized to the specific fields being matched (e.g. the code may vary according to the field width and the number of fields) as well as the desired matching semantics (e.g. an exact match may be transformed to a hash table lookup whereas a longest prefix match may be implemented using a trie match algorithm). The generated code may invoke existing libraries or macros that are independent of the programs being compiled. In this case, the native code is composed of a mixture of emitted code and references to the existing code. In the case of more complicated matching, covering multiple fields to be matched against a set of rules in priority order, with the option of wildcards in each field, a more complex matching algorithm such as an algorithmic TCAM implementation or the well known DCFL algorithm may be employed. These more complex matching algorithms are typically implemented in such libraries.

[0054] Further for P4 HLL, actions are transformed to native code by generating equivalent operations expressed in the native instruction set, for example a set field action will be transformed into an action that directly rewrites the field in the memory associated with the packet, or it may update the extracted field value and cause the field to be subsequently written back to the packet to modify the packet, using a flag (typically called a dirty flag) to indicate that this is required.

[0055] Moreover for P4 HLL, quality of service is implemented by the P4 program having the ability to specify the queue/scheduler/shaper/meter to be employed, together with appropriate parameters. The actual QoS operations are implemented in existing native code libraries or by hardware components.

[0056] FIG. 5 illustrates an invocation of a Low Level Language (LLL) program by a High Level Language (HLL) program. Program 1 in the HLL IR may contain an invocation of functionality expressed in program 2 in LLL. In this case, the compilation of operations prior to the invocation proceeds as described above regarding FIG. 4.

[0057] The compilation of the invocation of program 2 by program 1 entails making the data being processed by the program (e.g. a packet, in the case of a network switch) available to the processor executing program 2, together with any additional data (e.g. state) required for executing the program.

[0058] In the present embodiment, processors P2 and P3 share memory, but are not the same core. Some of the data (e.g. the packet data and some of the associated metadata) may be exchanged by writing it to memory shared by both processors (either on-chip memory like CTM/IMEM/EMEM or off-chip memory like DRAM in the case of the Network Flow Processor), with other data (e.g. further associated metadata, as well as the notification that invocation is required) being transported over a queue (in on-chip or off-chip memory) to/from the second processor. The emitted native code will invoke the required primitives or library functions to effect this operation. This code is referred to as the “State Import/Export” code 71 and 72 in FIG. 5. For more information regarding a Network Flow Processor including processors with off-chip memories: U.S. patent application Ser. No. 13/399,888, entitled “Island-Based Network Flow Processor Integrated Circuit”, filed on Feb. 17, 2012, now issued as U.S. Pat. No. 9,237,095 (the entire subject matter of which is incorporated herein by reference).

[0059] Where the processors do not share memory, all the required data needs to be exported into a message which is transported over a queue (implemented for example via a bus like PCI Express) between the processors. The emitted native code will invoke the required primitives or library functions to effect this operation.

[0060] In the case of the two processors being the same core on the same chip, the emitted native code can directly cause program 2 to be invoked from program 1, by issuing a function or subroutine call, or by emitting program 2's native code in line. In this case, the queues 73 and 74 depicted in FIG. 5 are not required.

[0061] Data conversion may be required to enable program 2 to access the data being operated on as well as access the associated state. This data conversion may for example involve data type conversion (e.g. conversion between different widths of integers or conversion between integers and strings) and data representation conversion (e.g. endianness conversion). The data conversion is performed by processor P2 or processor P3, or by a combination of the processors, with an intermediate data format being employed.

[0062] Moving the data between memories and other memories (e.g. local memory and shared memory) as well as between memories and queues is effected by generating appropriate native code combined with invocations of library functions that perform these tasks.

[0063] The code that needs to be executed on processor P3 to perform these data conversion and movement operations is emitted as LLL (e.g. C) code by the compilation process. This code is referred to as the “compiled program 1 code to be used during the compiling of program 2” in FIG. 4.

[0064] For each header field parsed by the P4 code in Program 1, this code would for example contain a C accessor function/macro, or a structure type declaration by which the fields in a header can be accessed. Packet metadata (e.g. the ingress/egress port or the timestamp) would be similarly represented and accessed.

[0065] The results of matching (which entries matched and which actions they are associated with) as well as QoS related information (e.g. the queue and forwarding class to be employed for a given packet) are similarly represented.

[0066] Once the LLL code has completed executing for a given item of data being processed (e.g. packet), information is transformed in the other direction, to enable the HLL program to continue processing. The result of matching operations performed by the LLL code may for example be made available to the HLL program as metadata which can be matched (along with metada previously computed by the HLL code and information contained in the data being processed itself, e.g. packet fields) in order to further influence the matching process. Alternatively the LLL code can directly supply to the HLL code the actions to be executed or the QoS queues/parameters to be used.

[0067] In this way, the LLL (e.g. C) code can interface to and integrate with native code emitted by the HLL (e.g. P4) compiler.

[0068] The compilation of the LLL (e.g. C) entails the compilation of Program 2, supplied by the author of Program 2 in a set of text files, combined with the compilation of the LLL Code to Interface to Program 1.

[0069] Compilation of the LLL (e.g. C) to native code is not described in detail here, as this proceeds analogously to compilation of HLL code as described above. Furthermore this process is well known to any programmer skilled in the art of compiler construction, and a multitude of existing compilers (commercial and open source, for example GCC and LLVM CLANG) are readily available. The result of the compilation of LLL code to native code is native binary code executable on processor P3.

[0070] In one embodiment, emitted native binary code for processor P2 and processor P3 is linked together along with any required dependencies (e.g. initialization code or libraries containing functions to which the code refers), resulting in a single complete executable (binary) file containing all the native code.

[0071] Such linking entails combining the binary code resulting from the aforementioned compilation processes with binary code obtained from libraries and standard initialization code known to the linker. The combining process concatenates the native code fragments (e.g. functions) into one code image for each processor, adjusting the code as needed to ensure correct execution, e.g. the actual address at which a function is placed in the code image needs to be used where functions are invoked, and the actual address where each data structure is positioned is similarly embedded in the emitted code image.

[0072] Linking native code fragments (e.g. objects or portions of libraries) together is a process well known to those skilled in the art of constructing toolchains, and existing commercially available or open source linkers can be employed, e.g. the GCC open source project supplies a linker. For more details, refer to the documentation accompanying such linkers.

[0073] The result of this processing is a single file containing the code images to be executed on processor P2 and processor P3 which in this description are cores on a single chip (e.g. microengines on a single NFP). Should the target processors actually be located on different chips, multiple files may be produced.

[0074] The compilation process described above is typically performed on the software developer's workstation. It may also be performed on the host server, i.e. processor P1 in FIG. 2.

[0075] The executable (binary) code resulting from compilation is then moved to a non-volatile form of memory (e.g. magnetic hard disk drive or solid state disk) attached to the host (server) in which the processors are located, i.e. in a file accessible to processor P1 in FIG. 2.

[0076] The executable code is loaded into the code store of processors P2 and P3 by writing each instruction into the appropriate location in said code store. This is performed by issuing write operations over the PCIe bus linking processors P1 and P2/P3, or by packing a number of instructions into a control message which is transported over the PCIe bus from processor P1 to an agent running on one of the cores in the same chip as processors P2/P3, e.g. one of the microengines or an ARM core. The agent unpacks the instructions and proceeds to write them into the code store.

[0077] Once the executable code has been loaded, execution commences by processor P1 or the agent running on the same chip as processors P2/P3 writing to a control register to enable processors P2 and P3 to start executing.

[0078] This application claims priority under 35 U.S.C. §119 from U.S. Provisional Application No. 62/171,230, entitled “Low-Level Programming Language Plugin to Augment High-Level Programming Language Setup of an SDN Switch,” filed on Jun. 4, 2015; the subject matter of which is incorporated herein by reference.

[0079] Although certain specific embodiments are described above for instructional purposes, the teachings of this patent document have general applicability and are not limited to the specific embodiments described above. Accordingly, various modifications, adaptations, and combinations of various features of the described embodiments can be practiced without departing from the scope of the invention as set forth in the claims.