COMPUTING DEVICE WITH ETHERNET CONNECTIVITY FOR VIRTUAL MACHINES ON SEVERAL SYSTEMS ON A CHIP THAT ARE CONNECTED WITH POINT-TO-POINT DATA LINKS

[0001] The present invention is related to a computing device, in particular for automotive applications, with Ethernet connectivity for virtual machines on several systems on a chip that are connected with point-to-point data links. The invention is further related to a vehicle comprising such a computing device.

[0002] To provide Ethernet connectivity for virtual machines on several systems on a chip (SoC), there are two aspects that need to be considered. The first aspect is the provision of Ethernet connectivity to several SoCs, whereas the second aspect is the provision of Ethernet connectivity to virtual machines inside the SoCs. Existing solutions for these aspects have significant disadvantages for automotive applications, in particular with regard to costs, performance or non-compliance with mandatory automotive requirements.

[0003] With respect to the provision of Ethernet connectivity to several SoCs, several solutions are known.

[0004] According to one solution, all SoCs are provided with a dedicated network connection. Each SoC has a connection to a dedicated port of an Ethernet switch. The type of the port can be, e.g., a reduced gigabit media independent interface (RGMII), a peripheral component interconnect express (PCIe) interface, or a similar interface. However, as each SoC needs a separate connection to a dedicated port of an Ethernet switch, this solution is rather expensive and inefficient.

[0005] According to another solution, the SoCs are connected via PCIe and a PCIe switch with non-transparent bridge (NTB) ports. At least one SoC has an Ethernet network connection. On each SoC, an NTB-transport stack software is executed. The NTB-transport stack provides a connection to the other SoCs. A dedicated NTB-transport link to each SoC is used. The different links are connected via an Ethernet bridge in software, which distributes the traffic between the connected SoCs. At least on one SoC the Ethernet bridge has an additional port, which is connected to the Ethernet network. With this setup, each SoC can communicate with each other SoC and can, via the dedicated SoC with network connection, also communicate with the network. However, one or more additional Ethernet frame copies are necessary, which is handled by the NTB-transport stack, as data are not copied directly between virtual machines. In addition, the solution does not support full End-to-End Quality-of-Service requirements, e.g. blocking traffic from sources that exceed a specified bandwidth limit, or is more complex to implement, which causes additional CPU load. Furthermore, a spatial and temporal isolation between communication of virtual machines is not fully guaranteed. A further issue is that a PCIe switch with NTB ports is required. In the rather cost-sensitive automotive market, this might be a blocker.

[0006] According to another solution, the SoCs are connected via PCIe without a non-transparent bridge. However, there are currently no known implementations using this approach.

[0007] With respect to the provision of Ethernet connectivity to virtual machines inside the SoCs, several solutions are known.

[0008] According to one solution, a software-based virtualization of Ethernet connectivity inside the SoCs is used. To this end, the SoC Ethernet connection is used by a single virtual machine, which provides an Ethernet bridge in software. The other virtual machines can connect to these ports via an interprocess communication (IPC) mechanism, e.g. a shared memory. However, since the Ethernet connectivity virtualization inside the SoCs is done separately, this leads to additional Ethernet frame copies. This increases the CPU load on the SoCs and limits the data throughput.

[0009] According to another solution, a hardware-supported virtualization of Ethernet connectivity inside the SoCs is used. To this end, SoC Ethernet connection hardware is required that provides dedicated receive/transmit (Rx/Ix) queues and data processing mechanism, e.g. direct memory access (DMA) channels, for each virtual machine. For example, a PCIe network card with single-root input/output virtualization (SR-IOV) support may be used. However, this solution has the disadvantage that additional hardware is required.

[0010] It is an object of the present invention to provide an improved solution for providing Ethernet connectivity for virtual machines on several systems on a chip that are connected with point-to-point data links.

[0011] This object is achieved by a computing device according to claim 1. The dependent claims include advantageous further developments and improvements of the present principles as described below.

[0012] According to an aspect of the invention, a computing device comprises two or more systems on a chip, wherein one system on a chip is a root system on a chip and the other systems on a chip are end point systems on a chip that are connected to the root system on a chip with point-to-point data links, each system on a chip comprising one or more virtual machines, and wherein one system on a chip provides a connection to an Ethernet network. The virtual machines are connected via a virtual Ethernet link, and each system on a chip comprises an instance of a distributed virtual switch, which is configured to provide a virtualized access to the Ethernet network for the virtual machines of the respective system on a chip. The instances of the distributed virtual switch preferably are configured to provide a virtual Ethernet link to each virtual machine of the respective system on a chip.

[0013] According to the invention, a distributed virtual switch is used, i.e. a virtual switch distributed over the various SoCs. As the functionality is distributed among the instances of the distributed virtual switch, the processing load for network communication is balanced. The distributed virtual switch provides an optimized data path for Ethernet connectivity for each virtual machine in an environment with multiple SoCs, which are connected via a bus system, e.g., PCIe. Only one of the SoCs has a connection to an Ethernet switch. It is not necessary to provide an Ethernet connection to a dedicated port of the Ethernet switch for each SoC. Instead, the SoCs only need to be connected via a hardware connection. No PCIe switch is necessary for this hardware connection. As fewer ports are required, the hardware requirements on the Ethernet switch are reduced. The distributed virtual switch provides a generic Ethernet communication control and data path to all virtual machines on all SoCs, i.e., each virtual machine on each SoC has a generic Ethernet communication path irrespective of whether the peer is on the same SoC or on another SoC or on the network.

[0014] In an advantageous embodiment, the instances of the distributed virtual switch are software components that are executed in a privileged mode. In this way, it is ensured that the processor executing the respective software components may perform any operation allowed by its architecture. For example, the instances of the distributed virtual switch may be implemented as hypervisor extensions. Such a hypervisor is generally provided for managing and controlling the one or more virtual machines.

[0015] In an advantageous embodiment, the instance of the distributed virtual switch at the root system on a chip is configured to discover the instances of the distributed virtual switch of the related end point systems on a chip via the point-to-point data links and to establish a dedicated communication channel to each related instance of the distributed virtual switch of the end point systems on a chip. In this way, the need to copy the Ethernet frames is reduced to a minimum. For example, a unicast Ethernet frame from one virtual machine to a virtual machine on another SoC is only copied a single time. This reduces the CPU load at the SoCs and increases the data throughput.

[0016] In an advantageous embodiment, for each virtual Ethernet link the instances of the distributed virtual switch are configured to handle frame transmission requests to local virtual machines using data transfer. Preferably, hardware-accelerated data transfer is used, e.g. direct memory access.

[0017] In an advantageous embodiment, for each virtual Ethernet link the instance of the distributed virtual switch at the root system on a chip is configured to serve frame transmission requests to virtual machines on a target system on a chip by forwarding the request to the instance of the distributed virtual switch on the target system on a chip (SoC2-SoC3) and providing frame metadata including a data source address of the actual frame, such as a PCIe source address. This may include virtual to physical address translation. The distributed virtual switch provides an address translation between the guest physical address space of the virtual machine to the physical address followed to the PCIe address space used for PCIe DMA transactions on the SoC. In this way, no input-output memory management unit for guest physical address to physical address space translation is needed, which is typically not available for embedded devices or does not support translation for multiple guest physical address spaces.

[0018] In an advantageous embodiment, for each virtual Ethernet link the instances of the distributed virtual switch at the end point systems on a chip serve a frame transmission request to a remote virtual machine by forwarding the request to the instance of the distributed virtual switch at the root system on a chip and providing the frame metadata including the data source address of the actual frame, e.g., the PCIe source address. In this way, the instance of the distributed virtual switch at the root system on a chip is able to forward the request to the instance of the distributed virtual switch at the related target end point system on a chip.

[0019] In an advantageous embodiment, the instance of the distributed virtual switch at the root system on a chip handles a frame transmission request received from an end point system on a chip to a remote virtual machine by further forwarding the request to the instance of the distributed virtual switch at the related target end point system on a chip. In this way, communication between the various end point systems on a chip is enabled.

[0020] In an advantageous embodiment, for each virtual Ethernet link the instances of the distributed virtual switch fetch data targeted to this link on request from the instances of the distributed virtual switch at remote systems on a chip. In this way, it is ensured that the requested data is reliably provided to the link.

[0021] In an advantageous embodiment, the instance of the distributed virtual switch at the root system on a chip forwards fetch requests not targeting this instance of the distributed virtual switch to the instance of the distributed virtual switch at the related end point system on a chip. In this way, it is ensured that the fetch request arrives at the correct end point system on a chip.

[0022] In an advantageous embodiment, the instances of the distributed virtual switch are configured to provide a spatial isolation of the communication related to the virtual machines. For example, the distributed virtual switch as an independent component can ensure that the data to be received and transmitted by any virtual machine are write protected and read protected against any other virtual machine. This is an important aspect for an automotive grade network support for virtual machines on SoCs.

[0023] In an advantageous embodiment, the instances of the distributed virtual switch are configured to provide a temporal isolation between the virtual machines with regard to Ethernet communication. For example, the distributed virtual switch as an independent component can provide a temporal isolation between PCIe bus requests of virtual machines. Using a functionality of the hypervisor or an input-output memory management unit, only the distributed virtual switch gets access to the PCIe bus. The virtual machines do not have access to the PCIe bus at all. This mechanism prevents any virtual machine from intentionally or unintentionally overloading the PCIe bus. Furthermore, the distributed virtual switch as an independent component can provide a temporal isolation of communication of the virtual machines related to virtual functions. For example, the distributed virtual switch may limit the bandwidth or number of transmitted frames per virtual machine according to configured values.

[0024] In an advantageous embodiment, the instances of the distributed virtual switch are configured to scan outgoing and incoming Ethernet traffic from and to each virtual machine. The instances of the distributed virtual switch can then trigger defined actions. For example, the virtual switch may be configured to enforce further network separation, such as a VLAN (Virtual Local Area Network) for Ethernet. Furthermore, the virtual switch may be configured to block traffic from unauthorized sources or sources that exceed a bandwidth limit, to mirror traffic, or to generate traffic statistics on the level of the virtual machines.

[0025] In an advantageous embodiment, the instances of the distributed virtual switch are configured to scan ingress traffic and egress traffic and to perform plausibility checks. For example, the instances of the distributed virtual switch may check the match of a virtual machine to an SoC, the plausibility of MAC (Media-Access-Control) addresses, etc.

[0026] In an advantageous embodiment, the instance of the distributed virtual switch of the system on a chip providing the connection to the Ethernet network has exclusive access to an Ethernet network device.

[0027] In an advantageous embodiment, for each virtual Ethernet link the instances of the distributed virtual switch are configured to serve frame transmission request to the Ethernet network by forwarding the request to the instance of the distributed virtual switch of the system on a chip providing the connection to the Ethernet network. In this way, the distributed virtual switch on the target SoC is able to retrieve the next free Tx buffer from the Ethernet driver.

[0028] In an advantageous embodiment, the instance of the distributed virtual switch of the system on a chip providing the connection to the Ethernet network is configured to fetch data targeted to this Ethernet network from local virtual machines and from instances of the distributed virtual switch of remote systems on a chip. In this way, the frames to be transmitted are reliably provided to the Ethernet network.

[0029] In an advantageous embodiment, the instance of the distributed virtual switch of the system on a chip providing the connection to the Ethernet network is configured to serve received frames from the Ethernet network to local virtual machines using data transfer and to remote virtual machines by forwarding the frame metadata to the instance of the distributed virtual switch of the target system on a chip. In this way, an optimized communication from the Ethernet network to the virtual machines on the various SoCs is achieved.

[0030] Advantageously, a vehicle comprises a computing device according to the invention. The describes solution allows providing Ethernet connectivity for electronic control units (ECU) with several SoCs. This is gaining an increasing importance for automotive high performance computers (HPC), combined HPCs, which combine Interior/Network-HPC, advanced driver assistance systems and an infotainment HPC in one ECU, and applications in the field of advanced driver assistance systems in general. The automotive market is currently moving toward the usage of PCIe interfaces, especially for ECU internal communication. Using the described solution, the ECU costs can be significantly reduced.

[0031] Further features of the present invention will become apparent from the following description and the appended claims in conjunction with the figures.

[0036] The present description illustrates the principles of the present disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the disclosure.

[0037] All examples and conditional language recited herein are intended for educational purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.

[0038] Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0039] Thus, for example, it will be appreciated by those skilled in the art that the diagrams presented herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure.

[0040] The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, systems on a chip, microcontrollers, read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage.

[0041] Other hardware, conventional and/or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.

[0042] In the claims hereof, any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a combination of circuit elements that performs that function or software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function. The disclosure as defined by such claims resides in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. It is thus regarded that any means that can provide those functionalities are equivalent to those shown herein.

[0043] When Ethernet was introduced in the automotive industry, control units were usually connected via internal Ethernet controllers to Ethernet switches. With increasing performance requirements and tighter integration of several control units, high performance computers containing several independent virtual machines were introduced. In this case, virtual machine managers or hypervisors HV are used to partition several operating systems.

[0044] FIG. 1 schematically illustrates a known solution for providing Ethernet connectivity for virtual machines VM1.1-VM3.2 on several SoCs SoC1-SoC3 of a computing device CD. According to this solution, all SoCs SoC1-SoC3 are provided with a dedicated connection to an Ethernet network ETH. Each SoC SoC1-SoC3 has a connection to a dedicated port of an Ethernet switch SW. The type of the port can be, e.g., an RGMII interface or a PCIe interface.

[0045] FIG. 2 schematically illustrates a solution according to the invention for providing Ethernet connectivity for virtual machines VM1.1-VM3.2 on several SoCs SoC1-SoC3 of a computing device CD that are connected with point-to-point data links. As can be seen, a first SoC SoC1 is a root SoC, whereas the other SoCs SoC2-SoC3 are end point SoCs. The root SoC SoC1 has a hardware connection to each end point SoC SoC2-SoC3. This hardware connection supports remote direct memory accesses. A typical example for such a hardware connection is a PCIe connection for all SoCs SoC1-SoC3. In that case, the root SoC SoC1 is the PCIe root complex and the end point SoCs SoC2-SoC3 are PCIe end points. However, also other types of hardware connection can be used. The root SoC SoC1 has a connection to the Ethernet network ETH, which may be an automotive Ethernet network. This connection can be either over a separate PCIe channel or also via the via another interface, e.g. an RGMII interface. Each SoC SoC1-SoC3 has one or more operating systems, i.e. virtual machines VM1.1-VM3.2, which are created and run by a hypervisor HV.

[0046] According to the invention, a distributed virtual switch DVS is implemented on each SoC SoC1-SoC3. The distributed virtual switch DVS may be, for example, a software component running in privileged mode, e.g. as an extension of the hypervisor HV. The distributed virtual switch DVS provides an optimized Ethernet connectivity for each virtual machine VM1.1-VM3.2 on each SoC SoC1-SoC3 to other virtual machines VM1.1-VM3.2 on the same SoC SoC1-SoC3, to other virtual machines VM1.1-VM3.2 on different SoCs SoC1-SoC3, and to the Ethernet network ETH. For this purpose, the distributed virtual switch DVS at the root SoC SoC1 provides a network device NetDev1.1-NetDev1.2 to each virtual machine VM1.1-VM1.2 running on the root SoC SoC1. The thin dotted arrows between the network devices NetDev1.1-NetDev1.2 and the virtual machines VM1.1-VM1.2 indicate transmit and receive queue accesses. In addition, this distributed virtual switch DVS provides for each end point SoC SoC2-SoC3 one dedicated distributed virtual switch driver Drv2-Drv3, which is linked to the respective distributed virtual switch device Dev2-Dev3 of the end point SoCs c. The distributed virtual switches DVS at the end point SoCs SoC2-SoC3 provide a network device NetDev2.1-NetDev3.2 to each virtual machine VM2.1-VM3.2 running on the end point SoCs virtual machine VM1.1-VM1.2 running on the root SoC SoC1 and one distributed virtual switch device Dev2-Dev3, which is linked to the corresponding dedicated distributed virtual switch driver Drv2-Drv3 of the distributed virtual switch DVS at the root SoC SoC1.

[0047] The distributed virtual switch DVS at the root SoC SoC1 has a peer-to-peer communication with the distributed virtual switches DVS at the end point SoCs SoC2-SoC3. This means that each distributed virtual switch driver Drv2-Drv3 has a receive queue, which contains metadata of Ethernet frames, e.g. a destination MAC address, a VLAN tag, or a buffer address of an Ethernet frame transmitted by a virtual machine VM1.1-VM3.2. Accordingly, each distributed virtual switch device Dev2-Dev3 has a receive queue, which contains the metadata of the Ethernet frames. Each distributed virtual switch driver Drv2-Drv3 and each distributed virtual switch device Dev2-Dev3 can insert an entry in the receive queue of its linked peer on the other SoC SoC1-SoC3. The distributed virtual switch DVS of the root SoC SoC1 comprises a virtual PCIe switch, which virtualizes a physical switch in software. Furthermore, the distributed virtual switch DVS of the root SoC SoC1 has access to an Ethernet network device, e.g. an Ethernet switch, via an Ethernet driver EthDrv.

[0048] Advantageously, the distributed virtual switch DVS further has additional information with regard to each virtual machine VM1.1-VM3.2, e.g. an allowed bandwidth, an arbitration priority between the queues of a virtual machine VM1.1-VM3.2 and between several virtual machines VM1.1-VM3.2, a guest physical address mapping, and so on. Because of this information and the full control of the configuration of the network connection, e.g. the Ethernet switch, and the data and control path of each virtual network device, those devices can guarantee a spatial and temporal separation.

[0049] FIG. 3 schematically illustrates the transmission of a frame from a virtual machine VM2.1 on an end point SoC SoC2 to a virtual machine VM1.1 on a SoC SoC1 using the solution of FIG. 2. The source virtual machine VM2.1 transmits an Ethernet frame via a Tx queue access to the network device NetDev2.1, i.e. the source virtual machine VM2.1 puts the transmitted frame into the Tx queue. The distributed virtual switch DVS at the source virtual machine VM2.1 periodically checks the Tx queues of all available network devices NetDev2.1-NetDev2.2. It will thus detect the new available Tx frame and determines the target of the frame by reading address information of the frame, e.g. a destination MAC address or a VLAN tag. The distributed virtual switch DVS at the source virtual machine VM2.1 recognizes the root SoC SoC1 as the target of the frame based on a configured routing table. As a result, the distributed virtual switch DVS at the source virtual machine VM2.1 puts related metadata into the Rx queue of the linked distributed virtual switch driver Drv2 of the distributed virtual switch DVS at the target SoC SoC1. The related metadata includes address information of the frame, e.g. the destination MAC address or the VLAN tag, and a PCIe address of the Tx buffer with the transmitted frame. The metadata transfer, or more generally the control data access, is indicated by the thick dotted arrow. The insertion of the entry with metadata in the Rx queue of the target SoC SoC1 is then done via the distributed virtual switch device Dev2 at the source virtual machine VM2.1, which performs a PCIe write to the linked distributed virtual switch driver Drv2 at the target SoC SoC1.

[0050] The distributed virtual switch DVS at the target SoC SoC1 periodically checks if there is a new entry in the Rx queue of the local distributed virtual switch drivers Drv2-Drv3. The distributed virtual switch DVS will thus detect the new entry with the metadata. With the help of the routing information in the metadata and based on a configured routing table, the distributed virtual switch DVS at the target SoC SoC1 determines that the destination of this frame is a virtual machine VM1.1 on this SoC SoC1. The distributed virtual switch DVS at the target SoC SoC1 thus retrieves the next free Rx buffer from the network device NetDev1.1 of the destination virtual machine VM1.1. The distributed virtual switch DVS at the target SoC SoC1 now sets up a DMA copy of the frame from the Tx buffer on the SoC SoC2 at the source virtual machine VM2.1 to this Rx buffer of the destination virtual machine VM1.1. The DMA copy is executed via a PCIe link and is indicated by the thick solid arrow between the source virtual machine VM2.1 and the destination virtual machine VM1.1. After the DMA copy is finished, the distributed virtual switch DVS at the target SoC SoC1 informs the destination virtual machine VM1.1 that a new frame has been received and provides the filled Rx buffer back to the virtual machine VM1.1. Furthermore, it informs the distributed virtual switch DVS at the source virtual machine VM2.1 that the frame copy is finished, and that the Tx buffer can be released. The distributed virtual switch DVS at the source virtual machine VM2.1 recognizes this information. It informs the source virtual machine VM2.1 that the transmission is finished and returns the Tx buffer back to the virtual machine VM2.1.

[0051] FIG. 4 schematically illustrates the transmission of a frame from a virtual machine VM2.1 on an end point SoC SoC2 to a network ETH using the solution of FIG. 2. The source virtual machine VM2.1 transmits an Ethernet frame via a queue access to the network device NetDev2.1, i.e. the source virtual machine VM2.1 puts the transmitted frame into the Tx queue. The distributed virtual switch DVS at the source virtual machine VM2.1 periodically checks the Tx queues of all available network devices NetDev2.1-NetDev2.2. It will thus detect the new available Tx frame and determines the target of the frame by reading address information of the frame, e.g. a destination MAC address or a VLAN tag. The distributed virtual switch DVS at the source virtual machine VM2.1 recognizes the root SoC SoC1 as the target of the frame based on a configured routing table. As a result, the distributed virtual switch DVS at the source virtual machine VM2.1 puts related metadata into the Rx queue of the linked distributed virtual switch driver Drv2 of the distributed virtual switch DVS at the target SoC SoC1. The related metadata includes address information of the frame, e.g. the destination MAC address or the VLAN tag, and a PCIe address of the Tx buffer with the transmitted frame. The metadata transfer, or more generally the control data access, is indicated by the thick dotted arrow. The insertion of the entry with metadata in the Rx queue of the target SoC SoC1 is then done via the distributed virtual switch device Dev2 at the source virtual machine VM2.1, which performs a PCIe write to the linked distributed virtual switch driver Drv2 at the target SoC SoC1.

[0052] The distributed virtual switch DVS at the target SoC SoC1 periodically checks if there is a new entry in the Rx queue of the local distributed virtual switch drivers Drv2-Drv3. The distributed virtual switch DVS will thus detect the new entry with the metadata. With the help of the routing information in the metadata and based on a configured routing table, the distributed virtual switch DVS at the target SoC SoC1 determines that the destination of this frame is the network ETH. The distributed virtual switch DVS at the target SoC SoC1 thus retrieves the next free Tx buffer from the Ethernet driver EthDrv of the network device. The distributed virtual switch DVS at the target SoC SoC1 now sets up a DMA copy of the frame from the Tx buffer on the SoC SoC2 at the source virtual machine VM2.1 to this Tx buffer of the Ethernet driver EthDrv of the network device. The DMA copy is executed via a PCIe link and is indicated by the thick solid arrow between the source virtual machine VM2.1 and the network ETH. After the DMA copy is finished, the distributed virtual switch DVS at the target SoC SoC1 informs the network device that a new frame for transmission is available. Furthermore, it informs the distributed virtual switch DVS at the source virtual machine VM2.1 that the frame copy is finished, and that the Tx buffer can be released. The distributed virtual switch DVS at the source virtual machine VM2.1 recognizes this information. It informs the source virtual machine VM2.1 that the transmission is finished and returns the Tx buffer back to the virtual machine VM2.1.