Apparatus, Method, System and System Method for Distributed Routing in a Multipoint Communication System

This invention relates, in general, to data communications and data communications systems and devices and, more specifically, to an apparatus, method and system for distributed routing in a multipoint communication system.

With the advent of multimedia communications, data transmission has become increasingly complex. For example, multimedia communications applications such as real time transmission of digitally encoded video, voice, and other forms of data, may require new forms and systems for data communication and data transmission. One such new communication system is the CableComm.TM. system currently being developed by Motorola, Inc. In the CableComm.TM. system, a hybrid optical fiber and coaxial cable is utilized to provide substantial bandwidth over existing cable lines to secondary stations such as individual, subscriber access units connected to a personal computer, workstation, other data terminal equipment ("DTE"), for example, in households having new or preexisting cable television capability. These coaxial cables are further connected to fiber optical cables to a central location having centralized, primary (or "head end") controllers or stations having receiving and transmitting capability. Such primary equipment may be connected to any variety of networks or other information sources, from the Internet, various on line services, telephone networks, to video/movie subscriber service. With the CableComm.TM. system, digital data may be transmitted both in the downstream direction, from the primary station or controller (connected to a network) to the secondary station of an individual user (subscriber access unit), and in the upstream direction, from the secondary station to the primary station (and to a network). As discussed in greater detail below, the topology of such a communication system may be described as having a primary node (such as one formed by a primary station), connected via an intermediate network to many secondary nodes (such as those formed by secondary stations), which are then connected to corresponding terminals (forming a secondary network).

In the CableComm.TM. system, downstream data is currently intended to be transmitted using 64 quadrature amplitude modulation ("QAM") at a rate of 30 m bps (megabits per second), at 5 m symbols/second utilizing 6 bits/symbol, over channels having 6 MHz bandwidth in the frequency spectrum of 50-750 MHz. Anticipating asymmetrical requirements with large amounts of data tending to be transmitted in the downstream direction rather than the upstream direction, less capacity is provided for upstream data transmission, using .pi./4 differential quadrature phase shift keying (.pi./4-DQPSK) modulation in the frequency band from 5-42 MHz with a symbol rate of 384 k symbols/sec with 2 bits/symbol. In addition, the communication system is designed to have a multipoint configuration, i.e . . . Many end users (secondary stations, also referred to as subscriber access units) transmitting upstream to a primary station, with one or more primary stations transmitting downstream to the secondary stations. The communication system is also designed for asynchronous transmission, with users transmitting and receiving packets of encoded data, such as video or text files. In addition, it is also highly likely that transmission may be bursty, with various users receiving or transmitting data at indeterminate intervals over selected channels in response to polling, contention, or other protocols from the primary station, rather than transmitting a more continuous and synchronous stream of information over a dedicated or circuit switched connection.

Routing of packets (or frames) of information in such a multipoint communication system, having secondary stations which are geographically distributed from each other (i.e . . . , Physically situated in a variety of different locations), presents special difficulties. In the prior art, routing is typically performed within a non-geographically distributed area, such as within a local area network ("LAN"). In addition, in the prior art, routing between various locations has required the assignment of an internetwork address to every device within the system, resulting in excessive use of a limited resource, as there are an inherently limited number of such four byte internetworking addresses. In the prior art, moreover, the implementation of routing functions at a secondary node has tended to require complicated and expensive equipment, severely limiting the expansion of such systems to consumers. Accordingly, a need has remained to provide such routing of information between a terminal (such as a personal computer or other DTE) and a primary node, where many of the terminals are geographically distributed, without excessive utilization of internetwork addresses, and implementable utilizing comparatively less expensive secondary node equipment, such as secondary stations.

FIG. 1 is a block diagram illustrating a communication system in accordance with the present invention.

FIG. 2 is a block diagram illustrating a primary station apparatus in accordance with the present invention.

FIG. 3 is a block diagram illustrating a secondary station apparatus in accordance with the present invention.

FIG. 4 is a block diagram illustrating the prior art assignment of MAC layer and network layer addresses in a two-network internetwork connected by a router.

FIG. 5 is a diagram illustrating a generalized depiction of a packet format of internetwork packets transmitted on a MAC network of an internetwork.

FIG. 6 is a block diagram illustrating a prior art centralized topology consisting of a primary node connected to one or more secondary nodes via an intermediate network.

FIG. 7 is a diagram illustrating the prior art address contents of the MAC layer header and network layer headers of packets forwarded in a centralized topology using prior art internetwork routing techniques.

FIG. 8 is a block diagram of a communication system, having logical networks and internetwork addresses in accordance with the present invention.

FIG. 9 is a diagram illustrating the contents of an internetwork packet forwarded utilizing the apparatus and method of the present invention.

FIG. 10 is a flow diagram of the method of the present invention for transmission of data packets to a terminal.

FIG. 11 is a flow diagram of the method of the present invention for transmission of data packets from a terminal.

FIG. 12 is a flow diagram illustrating the system method of the present invention for distributed internetwork routing of information in a communication system.

As mentioned above, a need has remained to provide for distributed routing in emerging multipoint communications systems, such as the CableComm.TM. system, without excessive utilization of internetwork addresses, and implementable utilizing comparatively low cost equipment. In accordance with the present invention, such distributed routing is provided, utilizing a minimal or optimal number of internetwork addresses, and utilizing secondary stations which may be implemented at comparatively low cost, with primary network routing functionality implemented in a primary station. FIG. 1 is a block diagram illustrating a communication system 100 in accordance with the present invention, such as a multipoint communication system. As illustrated in FIG. 1, a primary station 101, also referred to as a primary transceiver unit 101, is coupled to a plurality of secondary stations 110.sub.a through 110.sub.n, via communication media 115 and 116. In the preferred embodiment, communication media 115 and 116 are hybrid optical fiber and coaxial cable. In other embodiments, the communication media may be coaxial cable, fiber optic cable, twisted pair wires, and so on, and may also include air, atmosphere or space for wireless and satellite communication. The primary station 101 is also coupled to a network 105, which may include networks such as the Internet, on line services, telephone and cable networks, and other communication systems. The secondary stations 110.sub.a through 110.sub.n are illustrated in FIG. 1 as connected to the primary station 101 on two segments or branches of a communication medium, such as communication media 115 and 116. Equivalently, the secondary stations 110.sub.a through 110.sub.n may be connected to more than one primary station, and may be connected to a primary station (such as primary station 101) utilizing more or fewer branches, segments or sections of any communication medium.

Continuing to refer to FIG. 1, in the preferred embodiment, the communication medium, such as communication media 115 and 116, has or supports a plurality of communication channels. For ease of reference, the communication channels in which a primary station, such as the primary station 101, transmits information, signals, or other data to a secondary station, such as secondary station 110.sub.n, are referred to as downstream channels or downstream communication channels. Also for ease of reference, the communication channels in which a secondary station, such as secondary station 110.sub.n, transmits information, signals, or other data to a primary station, such as primary station 101, are referred to as upstream channels or upstream communication channels. These various upstream and downstream channels may, of course, be the same physical channel or may be separate physical channels, for example, through time division multiplexing or frequency division multiplexing. These various channels may also be logically divided in other ways, in addition to upstream and downstream directions. As mentioned above, in the preferred embodiment of the CableComm.TM. system, the communication medium is hybrid fiber coaxial cable, with downstream channels in the frequency spectrum of 50-750 MHz, and with upstream channels in the frequency band from 5-42 MHz.

FIG. 2 is a block diagram illustrating a primary station 101 in accordance with the present invention. The primary station 101 (also referred to as a primary transceiver) is coupled to a communication medium 114 for upstream and downstream communication to one or more secondary stations (not illustrated), and is coupleable to a network, such as the Internet, through a network interface 119. The primary station includes a processor arrangement 120 which is connected to a plurality of channel interfaces, channel interface 125.sub.a through channel interface 125.sub.n, for communication over the communication medium 114. The processor arrangement 120 includes a master controller 121 having or connected to memory 122, and one or more additional processors 130.sub.a1 through 130.sub.n2 and corresponding associated memories 131.sub.a1 through 131.sub.n2. In the preferred embodiment, the master controller 121 is a Motorola M68040 processor, and the memory 122 is 16 MB RAM. The master controller 121 performs a variety of higher level functions in the preferred embodiment, such as the spectrum management of the present invention, plus other functions such as routing, management of secondary stations, and communication protocol management (such as SNMP management). The master controller 121 is connected to a plurality of other processors, collectively referred to as processors 130 and separately illustrated as processor 130.sub.a1, processor 130.sub.a2, through processor 130.sub.n1 and processor 130.sub.n2. Each of these processors, processor 130.sub.a1, processor 130.sub.a2, through processor 130.sub.n1 and processor 130.sub.n2, is also coupled to or contains corresponding memory circuits, memory 131.sub.a1, memory 131.sub.a2, through memory 131.sub.n1 and memory 131.sub.n2. In the preferred embodiment, each of these processors 130 are also Motorola M68040 processors, while the corresponding memory circuits, memory 131.sub.a1 through memory 131.sub.n2, are 4 MB RAM. In the preferred embodiment, the processors 130 perform such functions related to upstream and downstream data protocols, such as sending a poll message or an acknowledgment message downstream. Each of these processors 130.sub.a1 through 130.sub.n2 of the processor arrangement 120 are connected to corresponding receivers and transmitters of the channel interfaces, channel interface 125.sub.a through channel interface 125.sub.n (collectively referred to as channel interfaces 125), namely, receiver 135.sub.a through receiver 135.sub.n (collectively referred to as receivers 135) and transmitter 136.sub.a through transmitter 136.sub.n (collectively referred to as transmitters 136). In the preferred embodiment, depending upon the functions implemented, each of the receivers 135.sub.a through 135.sub.n may include a Motorola M68302 processor, a Motorola 56000 series digital signal processor, a ZIF SYN integrated circuit, and an LSI Logic L64714 (Reed-Solomon decoder), for demodulation and for decoding forward error correction and cyclic redundancy checks. In the preferred embodiment, also depending upon the functions implemented, each of the transmitters 136.sub.a through 136.sub.n may include a Motorola M68302 processor, a Motorola 56000 series digital signal processor, a ZIF SYN integrated circuit, and an LSI Logic L64711 (Reed-Solomon encoder), for modulation and for coding for forward error correction and cyclic redundancy checks. As a consequence, as used herein, the channel interfaces 125 may be considered to perform the functions of data and other signal reception and transmission, regardless of the specific hardware implementations and additional functions which may or may riot be implemented. The various memories illustrated, such as memory 122 or 131.sub.a1, may also be embodied or contained within their corresponding processors, such as master controller 121 or processor 130.sub.a1. The functions of these various components with respect to the present invention are explained in greater detail below with reference to FIGS. 8-12.

FIG. 3 is a block diagram illustrating a secondary station 110.sub.n in accordance with the present invention. The secondary station 110.sub.n includes a processor 150, with the processor 150 having or coupled to a memory 155. In the preferred embodiment, the processor 150 is a Motorola M68302 processor (also known as an integrated multiprotocol processor), and the memory 155 is 256 k ram. The processor 150 is coupled to an interface 170, such as an ethernet port or an RS232 interface, for connection to a computer, a workstation, or other data terminal equipment. The processor 150 is also coupled to a channel interface 160 for communication over the communication medium 114. The channel interface 160, in the preferred embodiment, depending upon the functions implemented, includes a Motorola M68HC11 integrated circuit, a ZIF SYN integrated circuit, a Broadcom BCM3100 QAMlink integrated circuit, a Motorola TxMod integrated circuit, and LSI Logic L64711 and L64714 integrated circuits, and performs such functions as forward error correction encoding and decoding, QAM demodulation (for downstream reception), .pi./4-DQPSK modulation (for upstream transmission), transmit level and frequency adjustment, for data and other signal reception and transmission. As a consequence, as used herein, the channel interface 160 may be considered to perform the functions of data and other signal reception and transmission, regardless of the specific hardware implementations and additional functions which may or may not be implemented. The memory illustrated as memory 155 may also be embodied or contained within the corresponding processor 150. The additional functions of these components of the secondary station 110.sub.n with respect to the invention are also described in greater detail below with reference to FIGS. 8-12.

In the preferred apparatus embodiments illustrated in FIG. 2, the distributed routing method discussed below with reference to FIGS. 8 through 12 may be programmed and stored, as a set of program instructions for subsequent execution, in the processor arrangement 120, and more particularly, in the master controller 121 and its associated memory 122, for a primary station, such as primary station 101 illustrated in FIG. 2. In the preferred apparatus embodiments illustrated in FIG. 2, the distributed routing method discussed below with reference to FIGS. 8 through 12 may be programmed and stored, as a set of program instructions for subsequent execution, in the processor 150, and its associated memory 155, for a secondary station, such as secondary station 110.sub.n illustrated in FIG. 3.

FIG. 4 is a block diagram of two networks connected via a router, to illustrate the prior art operation of internetwork routing. In FIG. 4, two media (or medium) access control ("MAC") layer networks 200 are connected by an internetworking router 201. Connected to each corresponding MAC network 200 is a terminal (or host) 202 and a terminal (or host) 203, such as personal computers or workstations. Each of the MAC networks 200 may be a broadcast network such as a local area network ("LAN"), or a non-broadcast circuit-oriented network such as ATM (asynchronous transfer mode), frame relay, or x.25. Each node of a MAC network 200 is assigned a MAC layer address. In most MAC network implementations, all packets transmitted on the MAC network have a header which contains a destination MAC address 210 and a source MAC address 211, as illustrated in FIG. 5, discussed below.

An internetworking node connecting two MAC networks may provide communication between them by repeating, bridging, or routing. Repeating is done at the physical layer (layer 1), and refers to the operation of simply retransmitting everything received on one MAC network onto the other(s). Repeating is impractical in a communication system such as communication system 100 due to its wasteful use of available bandwidth and due to distance and/or timing constraints of existing layer 2 protocols.

Bridging is implemented at the data link layer (layer 2), and refers to the operation of recognizing which MAC network contains which MAC addresses, and forwarding to an attached MAC network only those packets destined for MAC addresses on that network. If a packet is received to a broadcast (all-MACs) address, a bridge copies it to all attached MAC networks. Bridging is generally considered impractical to operate on a large, geographically dispersed scale, and has largely been supplanted by routing in modern networking practice.

Routing is implemented at the network layer (layer 3). Routing depends on assigning an address (or number) to each MAC network, referred to herein as an internetwork network address. Each host or terminal on a MAC network, such as terminals 202 and 203, is also assigned an address or number specific to that host, referred to herein as an internetwork host address. The internetwork host address is usually different than the MAC address, but some routing protocols (such as IPX) use the MAC address as the internetwork host address. In general, hosts on different networks may have the same internetwork host address, so that the combination of both internetwork network address and internetwork host address is needed to uniquely address a host or terminal on an internetwork. This combination of an internetwork network address and internetwork host address is referred to as an internetwork address.

FIG. 5 is a diagram illustrating a generalized depiction of a packet format of internetwork packets transmitted on a MAC network of an internetwork. Routed internetwork packets, as transmitted on a MAC of an internetwork, are illustrated in FIG. 5, as packet 209. As a MAC layer packet, an internetwork packet 209 has a MAC layer which includes a destination MAC address 210 and a source MAC address 211. The MAC (or an immediately adjacent logical link control or LLC) layer header also includes a protocol field 212 to select the protocol of the packet 209. The protocol field 212 contains a protocol designation, such as a protocol number, which indicates which of several possible network layer protocols is used to encode the next part of the packet. For example, the common ethernet LAN uses a two-byte type field to indicate the protocol of the network layer; the value hex 0800 is used in the protocol field to indicate that the Internet protocol (IP) is used for the network layer portion of the packet. The network layer of the packet 209 contains the destination internetwork address 213 and the source internetwork address 214. Each of these internetwork addresses 213 and 214 are instances of a two-part internet address consisting of an internetwork network address and an internetwork host address.

A router, in the prior art, forwards internetwork packets of the form in FIG. 5 as follows: if the destination internetwork address 213 is to an internetwork network address of a directly attached MAC network, it determines the MAC address of the particular host addressed in the destination internetwork address 213, and sends a MAC frame to the final destination host that contains the network layer of the original incoming packet. Different routing protocols use different mechanisms to determine the MAC address. A common mechanism is the address resolution protocol (ARP) used as part of the common TCP/IP internetworking protocol in widespread use. If the destination network address 213 is not to a directly connected internetwork network address, then the router must consult its routing table. The routing table of a router associates known internetwork network addresses and the "next hop" router. The next hop router is another router, attached to (at least) one of the MAC networks attached to the router with the packet to be forwarded, that is capable of forwarding the packet closer to its final destination. Routers typically exchange information among themselves as to which networks are reachable and at what cost.

Common routing protocols in use today include the IP component of TCP/IP, IPX, Decnet, Appletalk, and the OSI IS--IS protocol. These protocols all have different mechanisms for assigning network and host addresses, for determining the MAC addresses of locally attached hosts, and for exchanging network reachability information. Indeed, the principal reason for dividing internetwork addresses into two parts--a network part and a host part--is to permit routers to exchange only network number information. With millions of potential internet hosts, any protocol which required the periodic exchange of all host numbers is generally considered impractical. All of these protocols rely on the concept of two-part internetwork addresses, and packets composed of separate MAC layer and network layer headers, as depicted in FIG. 5.

FIG. 6 is a block diagram illustrating a prior art centralized topology consisting of a primary node connected to one or more secondary nodes via an intermediate network. A number of terminal nodes (or hosts) 220, 221, 222, 223, and 224 are connected to MAC networks 230, 231, and 232, which are referred to as secondary MAC networks as each MAC network 230, 231, and 232, is connected to a corresponding secondary node 240, 241 and 242. The secondary nodes are connected to an intermediate MAC network 255. Without loss of generality to other forms of MAC networks, the intermediate network will be considered for concreteness as a non-broadcast circuit-switched network, i.e., considered to be implemented as a series of real or virtual circuits connecting the secondary nodes 240, 241 and 242 to a single primary node 250. In this topology, all communication between the terminals 220, 221, 222, 223, and 224 and other networks must go through the primary node 250.

FIG. 6 also illustrates the prior art assignment of internetwork addresses to this centralized topology. Each of the MAC networks 230, 231, and 232 is correspondingly assigned a separate internetwork network address (or internetwork address) IN1, IN2, and IN3. The intermediate network 255 is also assigned an internetwork address, INC. Each terminal is assigned an internetwork host address, which together with the internetwork network address assigned to the secondary MAC network connecting it with a secondary node, forms the terminal's internetwork address. Furthermore, as an addressable node on two different networks (the secondary network and the intermediate network), the secondary nodes are assigned host addresses on each of these networks. For example, secondary node S1 is assigned host address IS1T on the IN1 network 260, and is assigned host address IS1C on the INC network 255. The primary node 250 is a router with a routing table that associates the network address of each secondary MAC with the next-hop network host address on the intermediate network 255 of the secondary MAC network. For example, the primary maintains an association of network number IN1 with IS1C.

FIG. 7 is a diagram illustrating the prior art address contents of the MAC layer header and network layer headers of packets forwarded in a centralized topology using prior art internetwork routing techniques. FIG. 7 illustrates, as an example, the addressing in the MAC layer and network layer of internetwork packets forwarded from some host on an other network to terminal T3 on the secondary MAC network 231. The other host (or the closest router between the other host and the primary node) sends to the primary node 250 a packet 280 with a MAC destination of the MAC address of the primary node 250 on the other network (MPO). The source-MAC address of packet 280 is the other host's MAC address, MHO. The network layer contains a destination internetwork address of the internetwork address for terminal T3, IT3. The source network layer address is the network address of the packet originator, the other host, IH. If the other host is connected to the primary node 250 by a series of routers, the network layer of packet would remain unchanged, but the MAC layer would have as its source address the MAC address of the last router that forwarded the packet to the primary node 255. The primary node 250 examines the network portion of the destination internetwork address IT3, finding it to be IN2, and utilizing its routing table, determines that the proper next hop is to the secondary router S2, with internetwork address IS2C on the intermediate network 255. The primary node 250 then determines the MAC address of the secondary node 241 on the intermediate network, perhaps by looking up the MAC address in an ARP table (for ARP-based protocols such as IP), and forwards the packet 281 on the intermediate network 255. Packet 281 has a destination MAC address of MS2C, with a source MAC address of the primary node 250 MAC on the intermediate network, MPC, and with network layer unchanged. Secondary node 241 (S2) also operates as a complete router, examining the destination internetwork address IT3, extracts its network portion IN3, and notes that an attached secondary MAC network is assigned that network number IN3. Secondary node 241 (S2) then extracts the internetwork host portion of the destination internetwork address IT3, obtaining IHT3, finds the MAC address MT3 associated with IHT3 (perhaps via ARP), and sends packet 282 on the secondary MAC network 231 which it shares with terminal 222 (T3). Packet 282 has a MAC destination address of MT3, with a MAC source address of secondary node 241 (S2) on the secondary MAC network, MS2T. The network layer addresses are again unchanged.

The prior art routing methodology illustrated in FIGS. 6 and 7, as mentioned above, is problematic when applied to the IP routing protocol in a centralized network involving potentially thousands of geographically distributed secondary nodes. Such a network is envisioned when ATM or other metropolitan-area networks provide connections of millions of consumer homes to the Internet. The principal problem is the relative dearth of IP addresses, as the IP protocol uses only 32 bits to encode both the network portion and the host portion of the internetwork address. In addition, if consumers were to implement multiple-terminal MAC networks in their homes, then classic routing address assignment would call for the assignment of an IP subnetwork number to each consumer MAC network, i.e., every home. Such IP subnetworks require a minimum of four IP host addresses: one for the terminal, one for the secondary node (secondary MAC network side), and two for broadcast purposes (the all-zeros and all-ones host numbers). Furthermore, traditional IP numbering requires a separate IP network address for the intermediate network, and another IP host address for the interface of the secondary node on the intermediate network. Such proliferation of internetworking addresses is avoided utilizing the method and apparatus of the present invention.

Other than the apparatus and method of the present invention, other possible approaches to reduce the proliferation of internetworking protocol ("IP") addresses may include implementing a MAC layer bridge in the secondary node. Such an approach, however, would be highly complex and costly, and utilize excessive bandwidth by retransmission of all broadcasts. Another potential approach, such as a multi-access broadband LAN approach, also consumes excessive bandwidth from retransmission of everything received from a secondary node, restrictions on maximum distance and minimum frame length, and restrictions upon which secondary nodes may utilize the network at any given time (distributed multi-access control). Other possible approaches, such as utilization of a modem, requires that a single terminal be connected to a single secondary node (modem) utilizing a circuit switched, non-broadcast network. In accordance with the present invention, however, a method and apparatus is provided for distributing the functions of internetwork routing among primary and secondary nodes in a centralized system topology, such that internetwork network addresses are not required for secondary MAC networks, and no internetwork host addresses are required for the secondary nodes. The present invention minimizes intermediate network traffic to only that required for internetwork routing (as opposed to ongoing rebroadcasts of all traffic), and yet provides for extremely simple operation of the secondary nodes, which may be implemented at low cost.

In accordance with the present invention, a "distributed" router is implemented in which secondary stations, as secondary nodes, operate as if they were the MAC network interfaces of a single, geographically distributed, gigantic router. From the point of view of the connected terminals, they function as though they are connected to a single router. From the point of view of the other networks, they function as though a single router provides access to a set of networks that comprise all terminals connected to the secondary MAC networks. The apparatus and method of the present invention also avoids a restriction of traditional routers, which require separate internetwork addresses (net numbers) for each MAC interface.

In addition, in accordance with the present invention, calls for the assignment of "logical" or virtual internetwork network numbers (or addresses) are assigned to groups of terminals, independently of the physical networks connecting each such terminal, i.e., there is no required association between the secondary MAC networks and the logical network numbers. Indeed, the invention may support hundreds or thousands more secondary MACs than logical networks. This allows all of the IP host addresses associated with an IP network address to be assigned directly to existing terminals (whereas the assignment of logical network numbers to secondary MAC networks would result in many IP host addresses being assigned but unused or wasted on the "all-zeroes" and "all-ones" broadcast addresses).

As discussed in greater detail below, all terminals are assigned internetwork addresses in the logical networks. The primary node maintains an association between the terminal's logical internetwork address and the intermediate network MAC address (e.g., circuit number) of the secondary node connected to the secondary MAC network (to which the terminal is also connected). Advantageously, the primary node may store the MAC addresses of the secondary nodes and all terminals, allowing it to prepare even the MAC layer header of the packet forwarded from the secondary node to the terminal. This permits less complicated secondary station (node) implementation and operation, since it need merely strip off a secondary network encapsulation layer and forward the packet onto the secondary MAC network.

FIG. 8 is a block diagram of a communication system, having logical networks and internetwork addresses in accordance with the present invention. The communication system illustrated in FIG. 8 employes a similar centralized topology as that illustrated in FIG. 6, namely, a primary node connected via an intermediate network to a plurality of secondary nodes (having terminals connected via a secondary MAC network), but using the distributed routing method and apparatus of the present invention. Rather than assigning internetwork addresses (numbers) to each of the secondary MAC networks 320, 321 and 322, a set of logical network addresses or numbers, such as LN1 (logical network one) and LN2 (logical network two) are defined, and the terminals 300, 301, 302, 303 and 304 are arbitrarily assigned to one (or more) of the logical networks. In FIG. 8, terminals T1 (300), T2 (301), and T3 (302) are assigned to logical network LN1 and terminals T4 (303) and T5 are assigned to LN2. The terminals 300, 301, 302, 303 and 304 maintain their respective secondary MAC network addresses (MT1, MT2, etc.) on their respective secondary MAC networks. Terminal T3 (302), for instance, is defined to have secondary MAC network address MT3, and an internetwork address IT3. The internetwork address of the terminal T3 is considered to be the tuple (LN1, IHT3), i.e., a network part of logical network LN1, and a host part of internet host address IHT3.

Continuing to refer to FIG. 8, the secondary nodes 340.sub.a, 340.sub.b, and 340.sub.c (such as secondary stations 110) forward packets from the corresponding secondary MAC networks 320, 321 and 322 to the primary node 350 (such as a primary station 101), provided those packets have a protocol field which indicates that they are of the desired internetworking (or ancillary) protocols. For example, a secondary node connected to an ethernet secondary MAC would forward only ethernet packets with a type code of hex 0800 (for IP) and hex 0806 (for the ancillary arp protocol). The secondary nodes also should be capable of recognizing different encapsulations of protocols on the secondary MAC networks. For example, they must recognize both the Type encapsulation of ethernet as mentioned above and the Sub Network Access Protocol (SNAP) encapsulation of IP and ARP in ethernet, as described in the Internet Society's Request For Comments (RFC) 1042. Further, for an ethernet secondary MAC, the secondary node only needs to examine for forwarding those ethernet frames that contain the secondary node's ethernet MAC address MS2T, the ethernet broadcast address, and ethernet multicast addresses if the secondary node is part of a multicast group. The primary node recognizes internetwork protocol transmissions from the terminals and builds an association between a terminal sender's internetwork host address and the secondary node which forwarded the packet. It forwards internetwork packets from terminals to "other" internetwork hosts as a router typically operates.

Continuing to refer to FIG. 8, the operation of a primary node (such as primary station 101) is illustrated for the forwarding of packets from other internetwork hosts to a terminal. The particular example utilizes is an internetwork packet from a host 252 with internetwork host address (denoted IH) to terminal T3 (302). Corresponding packets are illustrated in FIG. 9. The other host (or a router adjacent to the primary node) 252 sends to the primary node 350 a packet (360 in FIG. 9) addressed to a MAC destination of the primary node's MAC address MPO on the other MAC network 251. The source MAC address is MHO, corresponding to the other host (or router) 252. The network layer of the packet 360 has a destination internetwork address to T3, denoted herein as IT3 and consisting of a logical network number (or address) LN1 and a host number IHT3. When the primary node 350 receives this packet 360, it determines, through, for example, a "forwarding" table of internetwork addresses, that IT3 is associated with secondary node S2 (340.sub.b). In the preferred embodiment, the forwarding table associates the following information with each known terminal internetwork address: (a) a secondary node MAC address on an intermediate MAC network; (b) a secondary node MAC address on a secondary MAC network; a terminal node MAC address on a secondary MAC network; and an encapsulation type of internetwork port as used by the terminal (or terminal node) on the secondary MAC network. All or some of the information in such a forwarding table may be learned by the primary node 350 or provideded by a communication system operator or service provider. In the preferred embodiment, this forwarding table entry would have been built based on an earlier Internetwork (or ARP) transmission by T3. If no association to the full internetwork address IT3 exists in the forwarding table of primary node 350, it may initiate an ARP broadcast to be sent on some or all secondary MAC networks requesting host IHT3 to identifiy itself. The primary node may have associations statically or dynamically configured which identify which secondary stations are eligible to connect to certain logical networks.

Using the information in the forwarding table, the primary node 350 prepares the MAC layer packet to be transmitted on the secondary MAC network 321, and encapsulates it in an intermediate network packet to secondary node S2 (340.sub.b), depicted as packet 365 in FIG. 9. The packet from the primary node 350 to S2 is sent as a MAC layer packet from the source MAC address of the primary node 350 on the intermediate network (MPC) to the destination MAC address of secondary S2 on the intermediate network (MS2C). In the preferred embodiment, the primary node's MAC address (MPC) is unnecessary and is omitted, because only one primary node communicates with a specific secondary node at a given time. Inside this intermediate MAC packet 365 resides the full secondary MAC packet 370 to be transmitted by S2 on the secondary MAC network 321. The secondary MAC packet 370 is addressed to the secondary MAC address of the terminal T3 (MT3) with a source MAC address of the secondary node's MAC address on the secondary network (MS2T). The encapsulation type in use by T3 is used to build the MAC layer header (e.g., whether ethernet Type or SNAP encapsulation is used for IP on ethernet). The secondary node S2 need only strip the intermediate MAC layer header from the packet 365 to form packet 370 and forward onto the secondary MAC network the already-prepared secondary MAC packet 370.

FIG. 9, in conjunction with FIG. 8, also illustrates a packet transmitted from terminal T3 (302), as the source, to be sent to the other host 252, as the destination. Packet 375, transmitted from terminal T3 (302) to the secondary node 304.sub.b contains the destination MAC address on the secondary MAC network of secondary node S2 (MS2T), the source MAC address on the secondary MAC network of the terminal T3 (MT3), the destination internetwork address of the other host (IH), and the source internetwork address of the terminal T3 (IT3). The secondary node deletes or omits the destination secondary node MAC address on the secondary MAC network from packet 375, and encapsulates deleted version to form packet 380, adding the destination MAC address of the primary node on the intermediate network (MPC), and the source MAC address of the secondary node on the intermediate network (MS2C). In the preferred embodiment, the destination secondary node MAC address on the secondary MAC network may be omitted by the secondary station, as this information is already known to the primary node, and transmission overhead may be reduced as a consequence of the omission. The primary node then forms packet 385, containing the destination MAC address of the other host 252 on the other network 251 (MHO), the primary node source MAC address on the other network (MPO), followed by the destination internetwork address of the other host (IH), and the source internetwork address of the terminal T3 (IT3).

The functions of the secondary nodes, in summary, are: (a) to forward to the primary node packets received from the secondary network (i.e., packets (with their data and other information) which contain the secondary's MAC address, the ethernet broadcast MAC address, or a multicast MAC address for a multicast group that this secondary station belongs to) only if they have internetwork protocol type or its ancillary ARP protocol type; and (b) to copy onto the secondary MAC network packets (with their data and other information) received from the intermediate network which contain secondary MAC packets. Alternatively, the secondary nodes may insert their own secondary MAC address in the source MAC address field of packets forwarded onto the secondary MAC network, and/or may ensure that packets meet requirements of the secondary MAC network (e.g., for an ethernet secondary MAC network, the adding of pad bytes to meet minimum frame size requirements may be done by the the primary node or the secondary node). The secondary nodes are not required to maintain any state information on the set of terminals attached to them, or the available logical networks, or indeed any status of other secondary nodes, permitting a much less complex and lower cost implementation than prior art bridge or router implementations.

FIG. 10 is a flow diagram of the method of the present invention for transmission of data packets to a terminal. Beginning with start step 400, a first data packet is received, having as its destination address an internetwork address of a first terminal, step 405. Next, a secondary MAC network packet if formed from the first data packet, step 410. The secondary MAC network packet is encapsulated in an intermediate network packet having, as a destination address, an intermediate network address of a secondary node corresponding to the terminal, step 415. The intermediate network packet is transmitted to the secondary node, step 420. The secondary MAC network packet (previously encapsulated) is retransmitted to the terminal, step 425, and the process may end, return step 430.

FIG. 11 is a flow diagram of the method of the present invention for transmission of data packets from a terminal. Beginning with start step 450, a first packet is transmitted to a secondary node, the first packet having an internetwork portion with a destination internetwork address and having a secondary MAC network portion with a destination address of a secondary node, step 455. Next, an intermediate network packet, having a destination address of a primary node on an intermediate network, is formed by encapsulating the first packet and omitting the destination address of the secondary node in the secondary MAC network portion. The intermediate network packet is then transmitted on the intermediate network to a primary node, step 465. Lastly, a second packet is transmitted having a destination MAC address of the destination host and a source internetwork address of the terminal, step 470, and the process may end, return step 475.

FIG. 12 is a flow diagram illustrating the system method in accordance with the present invention for distributed internetwork routing of information in a communication system. Beginning with start step 500, a first internetwork protocol address of a plurality of internetwork protocol addresses is assigned to a terminal of the plurality of terminals, step 505. Next or concurrently with step 505, a second internetwork protocol address of the plurality of internetwork protocol addresses is assigned to the primary node, step 510. Then, as illustrated FIGS. 9 and 10, information is transferred between the terminal and the primary node, via a secondary node of the plurality of secondary nodes, independently of any internetwork protocol address of the secondary node, step 515, and the process may end, return step 520. More specifically, step 515 may include transmitting a packet of a plurality of packets from a secondary media access control network to the intermediate media access control network based upon a secondary media access control network destination address in conjunction with a protocol number field. Step 515 may also include transmitting a packet of a plurality of packets from the intermediate media access control network to a secondary media access control network based upon a secondary media access control network destination address (or a secondary media access control network frame) encapsulated within an intermediate media access control network packet.

In summary, FIG. 2 in light of FIGS. 8-12 illustrates an apparatus for distributed internetwork routing of information in a communication system, the communication system having a topology in which a primary node is coupleable to a plurality of secondary nodes via an intermediate network, in which the plurality of secondary nodes are coupleable to a secondary media access control network, in which the plurality of secondary nodes are geographically distributed, and in which a plurality of terminals are coupleable to the plurality of secondary nodes. The apparatus then comprises: first, a network interface 119 coupleable to a first network for reception of a first packet and for transmission of a second packet, the first packet having as a destination address an internetwork address of a first terminal of the plurality of terminals; second, a channel interface 125 coupleable to the intermediate network for the reception and transmission of an intermediate network packet; and third, a processor arrangement 120 coupled to the channel interface, the processor arrangement responsive through a set of program instructions to form a first secondary media access control network packet from the first packet and to form a first intermediate network packet for transmission on the intermediate network via the channel interface, the first intermediate network packet having, as a destination address, an intermediate network address of a secondary node of the plurality of secondary nodes, the secondary node corresponding to the first terminal; the processor arrangement further responsive to form the second packet from a second intermediate network packet, the second packet having a destination media access control address of a destination host and a source internetwork address of a second terminal of the plurality of terminals.

Also in summary, FIG. 3 in light of FIGS. 8-12 illustrate an apparatus for distributed internetwork routing of information in a communication system, the communication system having a topology in which a primary node is coupleable to a plurality of secondary nodes via an intermediate network, in which the plurality of secondary nodes are coupleable to a secondary media access control network, in which the plurality of secondary nodes are geographically distributed, and in which a plurality of terminals are coupleable to the plurality of secondary nodes. The apparatus comprises: first, a channel interface 160 coupleable to the intermediate network for reception of a first intermediate network packet and for transmission of a second intermediate network packet; second, a second interface 170 coupleable to a secondary media access control network for transmission of a first secondary media access control packet to a first terminal of the plurality of terminals and for reception of a second secondary media access control packet from a second terminal of the plurality of terminals; and third, a processor arrangement 150 coupled to the first channel interface 160 and to the second interface 170, the processor arrangement responsive through a set of program instructions to form the first secondary media access control packet from the first intermediate network packet by deleting intermediate network information, the processor arrangement further responsive to form the second intermediate network packet from the second secondary media access control packet by encapsulating the second secondary media access control packet in the second intermediate network packet.

The apparatus, method, system and system methodology of the present invention provide numerous advantages. First, the primary routing functionality is implemented in primary nodes or stations, resulting in the ability to implement secondary stations at relatively low cost. Second, the use of internetwork addresses is optimized, resulting in avoiding the undue and excess multiplication of internetwork addresses of the prior art. Third, multiple terminals may be connected to the same secondary station, such as several personal computers within a consumer household. Fourth, transmission bandwidth is conserved through the deletion of unnecessary information transmitted over the intermediate network. Fifth, primary stations provide for secondary MAC network packets, resulting in simplified secondary station functionality.

From the foregoing, it will be observed that numerous variations and modifications may be effected without departing from the spirit and scope of the novel concept of the invention. It is to be understood that no limitation with respect to the specific methods and apparatus illustrated herein is intended or should be inferred. It is, of course, intended to cover by the appended claims all such modifications as fall within the scope of the claims. The invention is further defined by the following claims.