[jailhouse article] look mum, no VM exits

Ramsauer R, Kiszka J, Lohmann D, et al. Look mum, no VM exits!(almost)[J]. arXiv preprint arXiv:1705.06932, 2017.

This paper proposes a new method based on Linux Of 、 Operating system independent partitions Hypervisor —— Jailhouse, It will Linux Combined with a strictly isolated system . The design goal of this article is to minimize the amount of code and reduce Hypervisor Yes GuestOS The interference of .Jailhouse Assign hardware directly to GuestOS, Thus, the complex hardware processing boot problem is changed from Hypervisor Move to the general operating system (GuestOS).Jailhouse Establish quarantine domain , And can directly access physical resources , No emulation or semi virtualization .Jailhouse It's not just reserved Linux Versatility of the operating system , It also takes advantage of its simplicity , It makes it easy to implement security critical and real-time critical workloads running in isolated domains .

1. Introduction

Manufacturers of safety critical and non critical products tend to split components at different critical levels into different hardware , For example, different physics CPU Perform tasks at different critical levels . In the traditional mixed critical environment , A single control task is bound with dedicated physical control hardware , A typical physical controller is a programmable logic controller （PLC）, One car can contain dozens to a hundred different controllers , Integrating these systems into a single hardware controller is an architectural trend [4], Because it can not only improve the maintainability of software , It can also reduce the cost of the overall hardware . The method of this paper uses CPU Virtualization extension technology creates an execution environment , Use static partition technology to divide hardware resources , It can migrate existing programs to strictly isolated execution domains .

Jailhouse By sending messages to the system and I/O Bus insertion “ Virtual barrier ”, Deal with symmetry more (SMP) The system is converted to asymmetric multiprocessing (AMP) System . From a hardware point of view , The system bus is still shared , however GuestOS（ Prisoner ） Only part of the physical hardware can be used , It's like being imprisoned in a cell .

Jailhouse Started by Linux The kernel module in the system is enabled .Jailhouse Control all hardware resources , Reassign them to... According to the system configuration Linux, And will Linux Promote to virtual machine state （ virtual machine ）.Jailhouse Of Hypervisor The core acts as a virtual machine monitor (VMM). This scheme is not applicable to traditional Hypervisor classification [8] —— It can be seen as Type-1 and Type-2 Hypervisor Mixing ： It's like bare metal Hypervisor Run on the original hardware , No underlying operating system , But if not Linux As an assistant to provide the function of initializing hardware ,Jailhouse Still can't run .Linux Used as a boot loader , But not for the management system . And others are designed to manage hardware resources and prohibit GuestOS The system directly accesses the real-time partition （ for example ,PikeOS [10]） Different ,Jailhouse Only direct hardware access is supported .Jailhouse No complex and time-consuming （ And a half ） virtualization [2] Solution to simulate device drivers and share physical hardware resources , It's like exokernel Methods [7], Because it only provides isolation （ Expand by leveraging virtualization ）, No scheduler or virtual CPU.

Main contents of this paper ：

• Jailhouse The architecture of , One that runs on multiple architectures has complete functions 、 Non scheduling 、 real time 、 Static partitioning and open source Hypervisor.
• An example of running a hybrid critical application .
• Delay virtual machine Hypervisor Advantages of activation .
• Nvidia Jetson TK1 Typical micro benchmark of interrupt system

2. Related Work

although Hypervisor It is usually optimized for high throughput and optimal performance in the desktop and enterprise areas , But virtualization solutions for real-time embedded systems are especially targeted at low latency 、 The deterministic calculation cycle and the ability to maintain real-time are insufficient , Many embedded virtual machines Hypervisor Adopt the established practice of classic virtualization ： Over virtualized hardware 、 Semi virtualization [2] Or device emulation , as well as GuestOS Dispatch .

Crespo et al. Introduced XtratuM [5] The embedded Hypervisor. Their approach focuses on the design constraints given by avionics guidelines and specifications . They focused on Xtratum Memory management on 、 Clock and timer management 、 Interrupt management 、 Super call interface with rich functions and its own scheduler .XtratuM Is a mature virtual machine Hypervisor.

PikeOS [10] Allow different GuestOS Or local tasks . For running guest operating systems ,PikeOS Use semi virtualization and hardware assisted virtualization , But direct I/O visit . For application scheduling ,PikeOS Time and priority driven scheduling , And use best effort scheduling for non critical tasks .

In order to achieve time and space isolation ,Hypervisor Not all virtualization extensions are always required . Pinto et al [16] By using ARM TrustZone Technology in a single CPU Upper and Linux Run the real-time operating system in parallel . Their approach uses fast interrupts only for real-time critical devices (FIQ) To maintain real-time functionality . And Convention IRQ comparison , These interrupts directly reach the real-time operating system and Hypervisor The safe world of execution . Normal interruption to the non safe world isolated from the safe world , This method only isolates the non safe world from the safe world . Besides ,TrustZone Method allows only two domains to be created .

Quest-V [12] yes Quest A progress of operating system , In several ways with Jailhouse be similar . It aims to minimize Hypervisor Active static hardware partitioning . And Quest-V comparison ,Jailhouse Just one. VMM, No device driver is implemented , This greatly reduces its code base .Quest-V Rely on semi virtualization scheme to guide Linux Kernel as GuestOS.

Compared with all these systems ,Jailhouse from Linux Start （ And use its ability to initialize most of the hardware ）, Then use delay （ Or delay ） virtual machine Hypervisor Activate [18] Partition the hardware to the already running Linux Next .

3. Static Hardware Partition

3.1 Jailhouse Philosophy

As shown in the figure , Activate Jailhouse Our job is to include Hypervisor (HV) Of Linux With the help of kernel module , At every CPU After execution HV After starting the code ,Linux Continue as J ailhouse Of Guest function （ One cell）, namely root cell.

Linux It is an operating system with complete hardware support ,Jailhouse Taking advantage of this, we use Linux.Jailhouse The atypical delayed activation process of has considerable practical advantages , That is, most hardware initialization completely sinks to Linux, also Jailhouse Can fully focus on managing virtualization extensions . And exo-kernel [7] The method is similar to ,Jailhouse It's a exohypervisor. Direct allocation of hardware devices allows Linux Continue as before . And other partition methods （ for example ,[12]） Different ,Jailhouse No specific device drivers are required .

Jailhouse Assume that physical hardware resources do not need to be in GuestOS To share . To create additional domains （ be called non-root cell）,Jailhouse from Linux in “ Release ” Hardware resources （ for example CPU、 Memory 、PCI or MMIO equipment ） And reassign them to the new domain . This includes physics CPU：cell The configuration file of contains at least one CPU And a certain amount of memory , These memories are made up of root cell Preloading .

Linux Make the selected CPU Go offline and call Hypervisor, Allocate resources by providing cell Configuration file to create a new cell. Other resources , Such as PCI equipment 、 Memory mapped device or I/O port , It can also be reassigned to new Guest(cell).Hypervisor Block access to these resources from any other domain .non-root cell Can be created dynamically 、 The destruction （ namely , Allocate resources back root cell） Or restart .

because Jailhouse Only remap and reallocate resources , The ideal design idea is that it doesn't need to set up and start everything except management tasks GuestOS Still active after , And only intercept in case of access conflict ：“No VM exists!”, however , Hardware （ not yet ） It is completely suitable for this method , So on current hardware , The following situations still need VMM Intervention ：

• Interrupt reinjection （ It depends on the architecture , Interruptions may not arrive directly Guest）
• Intercept non virtualizable hardware resources （ for example ,ARM General interrupt controller on (GIC) Part of ）
• Access platform details （ for example , Access control coprocessor CP15 or ARM Power state control interface on (PSCI)）
• Simulation of some instructions （ for example x86 Upper cpuid）

The following pitfalls are inevitable , And it does not contradict the concept of this article , Because they are only in jailbreak or cell Management occurs ：

• Access conflict （ Memory 、I/O port ）
• cell management （ for example , establish 、 start-up 、 Stop or destroy cell）

These interceptions introduce overhead and latency —— Of course , Because virtualization comes at a price [6]. In section four , This paper introduces an example of the evaluation of a basic micro benchmark , That is, the additional delay of interruption .

Even though GuestOS The resources between are strictly isolated ,Jailhouse Still allow cell Share physical pages . In addition to satisfaction cell Communication between , It also allows shared memory mapped I/O page , if necessary , This allows access to hardware resources from multiple domains . However , This concurrent access is not by Jailhouse Arbitrated , Need by GuestOS Properly solve .

The figure shows three cell Possible zoning system layout ：Linux root cell（ green ）、 An additional Linux non-root cell（ Blue ） And a minimalist real-time operating system （ Red ）.cell The communication between them is realized through shared memory area and signaling interface , This minimalist design is in Hypervisor No additional device driver logic is required in . Based on hardware support ,cell Communication between is through MessageSignaled Interrupts (MSI-X) Or traditional interrupts are based on virtual PCI The device realizes .GuestOS This device can be used to implement virtual Ethernet devices . In the absence of PCI On supported systems ,Jailhouse Simulate a general and simple PCI Host controller .

3.2 Support

The partitioning method allows operating systems or bare metal applications that have passed security certification to work with Linux Running on a parallel multi-core system . It is worth mentioning that , Even though Jailhouse Support four different CPU framework , Beyond the framework provided by many experimental or research systems , But its minimalist approach leads to only a few thousand lines of code in the core . This simplifies the certification process , But it allows developers to focus on important issues , Instead of spending time providing drivers that the system will not use in its life cycle . The simplicity of the core code is right Hypervisor A good foundation for formal verification , Similar to formal verification of related system software [11].

Jailhouse With your own inmate library, Allow simple demo applications to run . except Linux Several other operating systems can be used as Jailhouse GuestOS Use （x86 [3] Upper L4 Fiasco.OC、ARM Upper FreeRTOS、ARM64 Upper Erika Enterprise RTOS v3）. With very limited efforts, this paper successfully serves ARM Architecture transplantation RTEMS Real-time operating system .

3.3 Practicability

In order to prove that the method in this paper is particularly suitable for practical application , This paper designs a （ Mixing critical ） Multi rotor control system . The requirements for such platforms can be comparable to many common industrial equipment ： Flight stack is a key part of safety and real-time system with high reliability requirements , Be responsible for the balance and navigation of the aircraft . The sensor value must be sampled at a high data rate 、 It is processed and finally used to control the rotor . For safe and reliable tasks , The control loop must make a deterministic response . System crash may lead to real crash and serious consequences .

The flight stack is Jailhouse cell Run in , Not mission critical , For example, with the ground station WiFi Communication or camera tracking , Due to available Linux Software ecosystem , It can be easily implemented in non critical parts . Key hardware components , for example SPI、I2C or GPIO equipment , Assigned to key cell. The hardware platform of this paper has four cores Cortex-A15 ARMv7 CPU Of Nvidia Jetson TK1, Connect the accelerometer 、GPS、 Sensor board of compass and gyroscope . Two physics CPU Allocated to non critical parts , Two physics CPU Assigned to key parts .

Key domain execution has Preempt_RT The second refinement of the real-time kernel extension Linux operating system .Ardupilot Provide flight control , There is no need to modify except for board support . This means that existing applications can be deployed effortlessly in Jailhouse in , And it is suitable for real-time safety critical systems based on existing components .

4. EVALUATION

As mentioned earlier ,Jailhouse Our goal is to minimize VMM The activities of . Although this is theoretically possible , But there is VMM Additional delays will be introduced [6], But not VMM There will be no such delays .

In order to evaluate and determine the （ real time ） performance , Several environmental conditions must be considered . It is difficult to quantify the overhead of the hypervisor with a single scalar , It's not even possible . Due to the limited space of the paper , This article will introduce the measurement of interrupt delay in detail as an example , And describe other important measurements .

It should be noted that , Such benchmarks do not measure the overhead of the hypervisor , Instead, it measures the overhead of the hypervisor running on a specific hardware platform . For all that , These measurements can still lead to trends in the performance of the management program .

a) Super call ： A typical benchmark for hypervisors is the cost of super calls . stay Jailhouse Under the circumstances , There is no need to consider super calls , Because they are only used for unit management purposes , And will never appear in the hot path .

b) Shared system bus ： Different guests Access memory asynchronously . Although hunger does not occur on supported architectures , But memory or I/O The extensive use of buses may lead to Guests Significantly slower . Although the problem is SMP Applications are well known , But when executing multiple payloads designed for a single core platform asynchronously , Its impact must be evaluated .

c) Architecture dependent traps ： Due to architecture limitations ,Jailhouse Devices that are necessary to emulate the hardware platform and cannot be virtualized in the hardware （ for example , As ARM An interrupt distributor that is part of the general interrupt controller on the architecture ）. According to the use of these devices , The impact of management procedures must be analyzed .

d) Interrupt delay ：Jailhouse Support two versions ARM General interrupt controller ,GICv2 and GICv3 [13, 14]. Both implementations have the same architectural limitations ： Interruptions do not arrive directly Guest. They arrive at the management program , Then as a virtual IRQ Refill Guest. This will cause overhead in the management program , Because it must redirect interrupts to the appropriate Guest, Then switch the privilege level .

The automatic measurement setup in this paper includes a Nvidia Jetson TK1（ Tetranuclear Cortex-A15 @2.32GHz） As the target platform , And a for performing actual measurements Arduino Uno Hardware .

To measure this delay , This article will delay bare metal （ That is, there is no minimum delay of the management program ） Compare with the delay when there is a management program .Arduino It will trigger... On the target board regularly GPIO Pin , This leads to interruption . The only task of a non root unit is to switch to another GPIO Respond to the interruption as soon as possible . therefore , This paper implements a using Jailhouse Their own inmates library The simplicity of GuestOS. To minimize the code size of the response to make it as fast as possible , For switching GPIO The instructions of are written directly in the interrupt vector table by assembler . Measurements without a hypervisor represent the minimum delay that can be achieved on the selected hardware platform . The delay difference between the presence and absence of hypervisors measures when hypervisors and others Guest Delay introduced when asynchronously accessing the system bus .Uno Ensure with 62.5ns For accurate measurement . In order to verify the measurement results , In this paper, the delay measured manually by oscilloscope is used to verify the sample measurement value .

This paper repeats the measurement under several conditions （ for example , Put the load on other clients to measure the impact on the shared system bus ） The arithmetic mean, standard deviation and maximum delay are given . Each measurement runs for four hours , And 10Hz and 50Hz The interrupt frequency of is repeated , To determine the influence of measurement frequency . The results can be found in the table . The first two lines show the minimum interrupt delay measured without the presence of a hypervisor , The difference from other measurements indicates the overhead introduced by the management program .

The delay introduced by the hypervisor does not obviously depend on the interrupt frequency , It depends on the neighboring Guest Utilization ratio . This effect is caused by the shared system bus ： The hypervisor wants to access the memory needed for scheduling interrupts , Others Guest Access the same bus asynchronously .

On average, , The interrupt delay is about 810ns, The deviation is small . For all that , Outliers will still cause near 5µs Delay of . Compared with the cycle time of typical industrial communication bus system , Maximum latency is acceptable for many applications .

5. DISCUSSION

Jailhouse The simple design method of produces a manageable number of source code lines (SLOC). This is a crucial factor for formal verification from an academic perspective and system certification from an industrial perspective （ This article recognizes , except Linux Outside the kernel , A large number of software chains are also required （ for example UEFI firm-code 、 Boot loader, etc ） To guide the process , And to some extent, it needs to be considered in such certification ）.

Jailhouse A total of nearly... For four different architectures 30k SLOC. This includes the core of the management procedure 、 Sample code 、 Kernel drivers as well as user space tools and tool scripts . Most of the code is architecture independent . The total core code of public key management program shared across all architectures is less than 3.4k SLOC.x86 The architecture related code of is about 7.4k SLOC, And at the same time Intel and AMD, as well as ARM（ARMv7 and ARMv8） about 5.4k SLOC.

Many systematic studies start from scratch , And spent a lot of energy on re implementing the existing device drivers . however , Lack of device support is still a major obstacle to its usability . Quest-V More than half of the source code lines （≈140k SLOC Medium 70k SLOC） Is the device driver .XtratuM Have close to 27k Of SLOC, Than Quest-V More lightweight , And only the basic driver for debugging output is implemented . For all that ,Quest-V and XtratuM The public version of currently only supports x86 framework .

Jailhouse Deliberately not following classic virtualization methods , But its design usually does not eliminate the use of these technologies . This will be Jailhouse It is possible to use it as an experimental system platform , The platform allows attention to be focused on practical issues , Instead of re implementing the foundation from scratch .Jailhouse It's an investigation AMP Ideal platform for hardware and software behavior under workloads . Besides , It performs similar digital signal processing on the original hardware (DSP) The workload of provides a convenient and comfortable environment .

Modern multi-core systems have provided enough Physics CPU, Many actual embedded use cases do not need to be scheduled in the management program . in fact , Real time embedded management program [5] Many basic requirements of , For example, real-time scheduling strategy 、 Efficient context switching or deterministic hypervisor calls , Not even in cell Solve the problem in the configuration file . because Jailhouse No virtualization CPU、 Over virtualize hardware or partition scheduling , Therefore, there will be no expensive partition context switching or scheduling problems [23]. Super calls are for administrative purposes only , It is not used to adjust access to shared hardware .

According to the interrupt system and architecture , An interrupt may reach the hypervisor . On such a platform , towards Guest Re injecting interrupts is a common job of managing programs , It introduces unexpected extra interrupt delays . For those that support interrupt remapping 64 position x86 framework , The problem has been solved , And will be realized in the future GICv4 [14] canonical ARM Solved in the architecture , This is conducive to the ultimate goal of this article , namely no VM exists!.

However , Traps caused by hardware design are inevitable . In the current ARM Architecturally , Interrupt allocators must be virtualized .Varanasi and Heiser [22] Assuming that this does not cause performance problems . In the implementation process of this article , It is observed in this paper that Preempt_RT Real time patch Linux The kernel uses a lot of interrupt allocators , This leads to high activity in the management program . Such problems should be solved by proper hardware design , So that unmodified Guest.

6. CONOLUTION AND FUTURE WORK

Static partition manager technology is a promising method for embedded real-time virtualization , Because they minimize contact with GuestOS Interaction . Compared with semi virtualization technology , Assign hardware directly to GuestOS Allow unmodified applications to run , Without administrative program overhead . The minimalist hypervisor core simplifies authentication . By executing the standard operating system as a visitor , This article also minimizes the amount of work required to migrate existing traditional payload applications . By implementing a complex demonstration platform , This paper successfully demonstrates the practicality of hardware partitioning .

Although the standard virtualization extensions provided by current hardware seem to be sufficient to directly implement this article and many other methods , But the actual hardware has many limitations , These limitations may completely destroy the advantages and guarantees of the partition and virtualization based approach . The future work of this paper will solve the problems and focus on evaluating the performance of the management program .