[jailhouse article] jailhouse hypervisor

author ：Maxim Baryshnikov, Czech Technical University Prague

This article describes Jailhouse Hypervisor Main concepts and principles of .
This article implements a simple demo application （ Interact with high-precision event timers ）.
In this paper, a small operating system （L4 Fiasco） Migration to Jailhouse cell in .
This paper evaluates the shared memory system for Jailhouse The impact of real-time properties of running programs .benchmark The test results show that , Two interactive CPU, In the worst case , Because for L3 The share of , Resulting in memory access bandwidth reduction .

1. Jailhouse characteristic

Jailhouse By the chief developer Jan Kiszka be responsible for , Focus on safety related examples （ Industrial processes 、 Aerospace 、 Medicine, etc ）, Its application scenario is special .Jailhouse The main feature of ： It's not in GuestOS Share multi-core processor resources symmetrically between , Instead, divide different resources into GuestOS Independent use .

Vitaliy Sinitsyn This evaluation ,Jailhouse In existing Linux Set to enable asymmetric multiprocessing , And split the system into two parts called cell Independent partition of . Every cell Run one GuestOS And have a set of fully controlled allocation resources （CPU、 Memory area 、PCI equipment ）.Jailhouse My job is to manage cell And keep them isolated from each other .

Suppose this paper has a multi-core system , This article wants to use a physics CPU To handle hard real-time tasks , the other one core For user interaction （GUI/HMI）, Other physics CPU Used to collect sensor data . obviously ,GUI The application cannot affect the first physical CPU The job of . How to prevent from being disturbed , This is it. Jailhouse The job of .

Jailhouse Our goal is safety critical industrial applications . Such applications usually need to be certified by independent agencies according to many security standards , The goal of certification is to gain confidence that the system is reliable enough to safely perform its intended function . The safety standard divides the safety functions into several levels according to the required degree of reliability （ Security integrity level ）, More stringent requirements for more critical systems . Complex systems are difficult （ It's very expensive ） Or unable to authenticate . That's why Jailhouse Is a very simple Hypervisor, It only contains the isolation function to isolate GuestOS The function needed . None of them cell Can access devices that do not belong to them , because Hypervisor Will stop it . therefore , Real time applications are not affected by anything that happens in other partitions .

Jailhouse It is in user space through Jailhouse VMM Device drivers that issue super calls to manage .Jailhouse Not part of the kernel , It runs at the lowest level . The kernel module is only used to put Jailhouse The binary image of is loaded into the memory reserved by the kernel .

2. celll configuration

Every cell（root cell/non-root cell） Static configuration must be carried out before startup , This configuration determines cell Which hardware resources can be accessed .Jailhouse Use *.c File as configuration file , Among them, parameters must be assigned to special C Fields of structure , These fields are defined in hypervisor/include/jailhouse/cell-config.h in .

non-root cell The fields are shown below ,root cell The fields are shown in the following figure .

about root cell, The structure needs to contain （struct jailhouse_system）, instead of struct jailhouse_cell_desc, There is no difference between the two in the rest .

To configure non-root Only handwritten configuration files , and root cell, There are such tools , There are two steps ：

1、jailhouse config collect <name-of-arch.tar> Collect target platform information .

2、jailhouse config create -r <name-of-arch.tar> <name-of-conf.c> Need to install python-mako library . After creation , The configuration is compiled into the original binary ,Jailhouse Creating cell Will use .

3. Jailhouse enabling

Jailhouse Start up requires the following steps ：

1、 Install on the target platform Linux>=3.18

2、 Compile and install on the target platform Jailhouse.

3、 stay Linux Add the reservation option for the memory area in the kernel startup parameters （ Such as memmap=）, The value must be equal to .phys_start And in root cell The configuration of the .size Field values are the same .

4、 load jailhouse.ko The kernel module . This can create /dev/jailhouse equipment , The device can be accessed through tools in user space .Jailhouse User space program to /dev/jailhouse send out JAILHOUSE_ENABLE request , This request instructs the driver to call jailhouse_cmd_enable()(driver/main.c). In this function , The driver first performs some verification . It checks CPU Mark to determine the CPU Which virtualization technology to use , Then perform basic verification on the configured binary file . Then call request_firmware() Function in /lib/firmware Search folders for jailhouse-inter.bin perhaps jailhouse-amd.bin. The driver will step 2 The memory area reserved in is remapped to the kernel address space memory （ Use ioremap_page_range(…)）, Therefore, it can be accessed from user space Jailhouse, The driver copies the binary file at the beginning of this memory area , And copy behind it cell Configuration of . then , call jailhouse_cell_create() function .

5、 perform hypervisor Start command ：jailhouse enable <path/to/cell/conf.cell>.enable The final stage is CPU initialization .Jailhouse For each CPU call entry_hypervisor() Function to start （hypervisor/x86/entry.S Medium arch_entry）.Jailhouse Need to be cell and CPU Interface between , So it saves the state of the system , And then in CPU0 Set its environment during initialization . Include ： by Jailhouse and APIC Set paging 、 Create interrupt descriptor table 、 Create and configure virtual machine extensions (VME), It also sets UART Communication to write debugging information , So you can ttyS0 See the output information on . For all CPU, This process is consistent ： to update IDT and GDT, Reset CR3, Set up VMCS. Last Hypervisor send out VMLAUNCH Instructions . From the start ,Linux No longer run on bare metal , But in Jailhouse Of cell Run in .

4. Cell initialization and start process

When the user executes jailhouse cell create <path/to/conf.cell> when , It reads the configuration file into memory , And will JAILHOUSE_CELL_CREATE The command is sent to the driver , call jailhouse_cmd_cell_create()(driver/control.c) Copy the above configuration file from user space memory to kernel space , Also on cell desc Inspection . then , According to the configuration, it is guest obtain image, stay sysfs New cell Information about , And will ask for CPU from Linux root cell Allocated from , from Linux Intermediate unloading PCI equipment .Jailhouse simulation PCI The virtual driver removes it from Linux Delete in , Because the real pull-out cannot be performed . When the driver sends JAILHOUSE_HC_CELL_CREATE When super calling ,Jailhouse call hypervisor/control.c Medium cell_create(), Give orders , Let all new cell The processor of is paused , In addition to the current processor , It can prevent competition between them .

then , Start cell Initialization process .cell_init() Function use I/O The value of the port bitmap is filled cell The structure of the cpu_set Field , And call the process to save PCI、IOAPIC and IOMMU Wait for the location and handler of the memory mapping device . then , Check CPU After not being assigned , Jump to arch_cell_create()(hypervisor/arch/x86/control.c), Start "shrinking". The focus here is to follow the concept of one-to-one distribution ： If root cell There are initial cell Things needed , that Linux cell(root cell) Your access will be denied , And new cell Get it , under these circumstances ,Linux cell Will stop on the first visit . redistribute I/O port 、IOAPIC、IOMMU、PCI After waiting for resources , Configure the communication area （ This is a hypervisor And specific cell Can read and write each cell Shared memory area ）. It also contains information about PM Timer address information , The distribution of CPU Number , Contains the current cell The state of （ for example ,running、running/locked）.

Last ,cell Will be submitted to cell list, In the communication area cell The status will be set to JAILHOUSE_CELL_SHUT_DOWN, And for each cell Of CPU,Jailhouse issue arch_cpu_resume().

In the new cell In the implementation of inmate, Need to put inmate Move to cell The memory area of ：jailhouse cell load <name-of-cell> <inmate.bin> -a <off-in-guest>

be-all inmates Are treated as raw binaries . The size of this binary file must be less than or equal to the guest Memory area . The driver will JAILHOUSE_HC_CELL_SET_LOADABLE Send to hypervisor, And mark as loadable guest Area remapped to root cell Address space of . You can see the serial port information ：“Cell can be loaded”. The driver stores binary files at a given address .

Last , To start it , The user should call ：jailhouse cell start <name-of-cell>, This can lead to HyperCall —— JAILHOUSE_HC_CELL_START, From its point of view , It leads to Jailhouse Of cell_start() Execution from root cell To Guest Unmap all loadable areas .cell The state of becomes JAILHOUSE_CELL_RUNNING, And in cell Each CPU On the call arch_cpu_reset(). This will to cell Each of the CPU Send false startup interprocessor interrupt （SIPI）. The next #VMEXIT when ,Guest The instruction pointer will be set to 0xFFFF0, also inmate Start execution .

5. Inmate demos

Jailhouse Provides a small framework , It is easy to develop simple applications without operating system . The dependent library The header file is defined in inmate.h in , It includes memory allocation and remapping 、APIC and IOAPIC The initialization 、 Interrupt handler settings 、 And PCI Some interactions of devices are even basic SMP operation （smp_wait_for_all_cpus()、smp_start_cpu()）.

inmates Start code of （header.S for 64-bit、header-32.S for 32-bit） Such as Listing 2.4 Shown .Jailhouse in inmates The entry point of is 0xFFFF0, Here's a trick to jump to 16 Bit code part （ be used for GDT And protection mode flag settings ）.inmate Binary files are offset 0xF0000 Load into GuestOS in .

Listing 2.5 Link description file shown in （inmate.lds） Ensure the following .16 Bit startup The segment is bound at the beginning of the binary file （0x0000）.boot Segment follows startup After the paragraph , Fixed in 0xFFF0, So add the offset address , The correct entry address will be given ..text、.data、.rodata Some have their virtual memory addresses （VMA, function output file when , The virtual address of this part ）, This includes the load offset . But their load memory address （LMA, Load the address of this part ） No offset loaded .

therefore , stay 0xFFF0 Situated .boot Segment has only one instruction ：ljmp $0xf000, $start16, It makes instruction pointer At the physical address 0xf0000 Move up . after , When GDT And when the protection mode flag is set , It will jump to 32 A code , Set paging here , And finally inmate_main() Function entrance .

5.1 APIC demo

APIC- Advanced programmable interrupt controller . This is a typical inmate, Usually used to demonstrate Jailhouse function . It's for APIC The timer sets the interrupt and measures the actual time between events . besides , It also demonstrates the use of inter domain communication and manipulate the cell state.

The demo Configuration files such as Listing 2.8 Shown ,configs/apic-demo.c, Two memory areas are defined ： load inmate The memory area of （1MB）, The immediate area for communication （4KB）, The latter has an additional logo JAILHOUSE_MEM_CONMM_REGION, Give Way Jailhouse Know where to read / Write a message , It will be on the serial port 0 Print a log on .

Launching apic-demo cell:

jailhouse cell create /jailhouse/configs/apic-demo.cell

jailhouse cell load apic-demo /jailhouse/inmates/apic-demo.bin -a 0xf000

jailhouse cell start apic-demo

jailhouse cell shutdown apic-demo

jailhouse cell shutdonw apic-demo

demo After it starts running ,cell state Set to JAILHOUSE_CELL_RUNNING_LOCKED, This is by setting commo_region->cell_state Realized , Usually , It means Jailhouse unable shrink cell. after , Application calibration timestamp counter （inmates/lib/x86/timing.c）, And initialization APIC Timer . Then set the handler for the interrupt of the timer , therefore , Every time the next interrupt occurs , Will calculate jitter .“ Jitter is the expected time and time （ Delay ） Differences between , In terms of performance , Smaller means jailhouse The less you perceive ”. The program waits for messages in the communication area , If a shutdown request occurs , Then the program sends a message , Indicates that it cannot be closed now , If this request occurs a second time ,apic-demo Will interrupt the cycle . take apic-demo Of cell The status changes to JAILHOUSE_CELL_SHUT_DOWN, therefore Jailhouse Know that the shutdown process is going well .

5.2 HPET demo

The high-precision event timer realized by the author .

6. L4 Fiasco.OC launch

stay cell When using bare metal programs in , May help solve simple problems , But in most cases , When you have to implement something more complex （ Network protocol stack 、 Autopilot, etc ） when , Need to run the operating system , therefore , Migrate some operating systems to inmate It's necessary .

at present , Can be in non-root cell Start in Linux, also Documentation/non-root-linux.txt Describes how to do , The kernel must be modified and configured in a specific way , User space tools for booting the kernel must also be created . however , Facing real-time tasks ,Linux Is not competent .

choice Fiasco.OC Migration is to meet the real-time requirements .

6.1 Overview

Fiasco.OC It's Dresden University Fiasco The micro kernel based operating system developed by the team . It consists of L4 Microkernel and and L4 Runtime environment （L4Re） The relevant user level program consists of . The kernel itself is very simple . It also provides communication between processes （IPC）、 establish / Delete address space （ Mission ） And threads .

Fiasco The minimalism of ： The microkernel provides a total of 7 System calls , let me put it another way , The microkernel uses only 7 A system call rules the world .

The minimum configuration that can start the operating system must include Fiasco kernel 、 be called Sigma0 Of Root-Pager、Root-Task（Moe） And at least one user space application running on it .Sigma0 Provides memory for user space programs （ Remap 、 Distribution, etc ） Of API.Moe Run on top of the paging manager , It is the task that the kernel starts first . It provides more abstract interfaces for all other user space applications .

6.2 Fiasco The boot process

For transplantation Jailhouse,Fiasco The system configuration 、 Start process configuration 、 Kernel configuration .

7. Benchmarks

Evaluate memory systems for different cell The performance impact of software running in . Test when different cell Impact of simultaneous access to memory .

In an ordinary multi-core processor ,core share L3 cache , And other parts of the memory system are also shared . Now suppose , Every core Perform completely different procedures , therefore , Each program needs to access different memory areas , under these circumstances ,core Will compete with each other cache, And memory access time will increase . Besides , When one core Want to access memory while another core Also want to access memory , There is bound to be one core Need to wait , In this way core In a competitive situation , One of them core Memory intensive applications on will significantly slow down others core On the application . All these problems mentioned will have a negative impact on the real-time and security of the system .

A simple benchmark, The benchmark Can be in cell in 、 Run on bare metal . The purpose is to study the use of Jailhouse when , Whether memory access slows down .