LoongArch Processor Virtualization

The LoongArch instruction set is an independent RISC instruction set released by China's Loongson Zhongke Company in 2020, which includes five modules: the basic instruction set, binary translation extension (LBT), vector extension (LSX), advanced vector extension (LASX), and virtualization extension (LVZ).

This article will provide a brief introduction to the CPU virtualization design of the LoongArch instruction set, with related explanations from the currently publicly available KVM source code and code comments.

Introduction to LoongArch Registers

Conventions for General Registers Usage

NameAliasUsagePreserved across calls
$r0$zeroConstant 0(constant)
$r1$raReturn addressNo
$r2$tpThread pointer(not allocatable)
$r3$spStack pointerYes
$r4 - $r5$a0 - $a1Argument/return value registersNo
$r6 - $r11$a2 - $a7Argument registersNo
$r12 - $r20$t0 - $t8Temporary registersNo
$r21Reserved(not allocatable)
$r22$fp / $s9Frame pointer / static registerYes
$r23 - $r31$s0 - $s8Static registersYes

Conventions for Floating Point Registers Usage

NameAliasUsagePreserved across calls
$f0 - $f1$fa0 - $fa1Argument/return value registersNo
$f2 - $f7$fa2 - $fa7Argument registersNo
$f8 - $f23$ft0 - $ft15Temporary registersNo
$f24 - $f31$fs0 - $fs7Static registersYes

Temporary registers are also known as caller-saved registers. Static registers are also known as callee-saved registers.

CSR Registers

Control and Status Register (CSR) is a special class of registers in the LoongArch architecture used to control the processor's operational state. List of CSR registers (excluding new CSRs in the LVZ virtualization extension):

NumberNameNumberNameNumberName
0x0Current mode information CRMD0x1Exception prior mode information PRMD0x2Extension part enable EUEN
0x3Miscellaneous control MISC0x4Exception configuration ECFG0x5Exception status ESTAT
0x6Exception return address ERA0x7Error virtual address BADV0x8Error instruction BADI
0xcException entry address EENTRY0x10TLB index TLBIDX0x11TLB entry high TLBEHI
0x12TLB entry low 0 TLBELO00x13TLB entry low 1 TLBELO10x18Address space identifier ASID
0x19Low half address space global directory base PGDL0x1AHigh half address space global directory base PGDH0x1BGlobal directory base PGD
0x1CPage table traversal control low half PWCL0x1DPage table traversal control high half PWCH0x1ESTLB page size STLBPS
0x1FReduced virtual address configuration RVACFG0x20Processor number CPUID0x21Privilege resource configuration info 1 PRCFG1
0x22Privilege resource configuration info 2 PRCFG20x23Privilege resource configuration info 3 PRCFG30x30+n (0≤n≤15)Data save SAVEn
0x40Timer number TID0x41Timer configuration TCFG0x42Timer value TVAL
0x43Timer compensation CNTC0x44Timer interrupt clear TICLR0x60LLBit control LLBCTL
0x80Implementation related control 1 IMPCTL10x81Implementation related control 2 IMPCTL20x88TLB refill exception entry address TLBRENTRY
0x89TLB refill exception error virtual address TLBRBADV0x8ATLB refill exception return address TLBRERA0x8BTLB refill exception data save TLBRSAVE
0x8CTLB refill exception entry low 0 TLBRELO00x8DTLB refill exception entry low 1 TLBRELO10x8ETLB refill exception entry high TLBREHI
0x8FTLB refill exception prior mode information TLBRPRMD0x90Machine error control MERRCTL0x91Machine error information 1 MERRINFO1
0x92Machine error information 2 MERRINFO20x93Machine error exception entry address MERRENTRY0x94Machine error exception return address MERRERA
0x95Machine error exception data save MERRSAVE0x98Cache tag CTAG0x180+n (0≤n≤3)Direct mapping configuration window n DMWn
0x200+2n (0≤n≤31)Performance monitoring configuration n PMCFGn0x201+2n (0≤n≤31)Performance monitoring counter n PMCNTn0x300Load/store monitor point overall control MWPC
0x301Load/store monitor point overall status MWPS0x310+8n (0≤n≤7)Load/store monitor point n configuration 1 MWPnCFG10x311+8n (0≤n≤7)Load/store monitor point n configuration 2 MWPnCFG2
0x312+8n (0≤n≤7)Load/store monitor point n configuration 3 MWPnCFG30x313+8n (0≤n≤7)Load/store monitor point n configuration 4 MWPnCFG40x380Instruction fetch monitor point overall control FWPC
0x381Instruction fetch monitor point overall status FWPS0x390+8n (0≤n≤7)Instruction fetch monitor point n configuration 1 FWPnCFG10x391+8n (0≤n≤7)Instruction fetch monitor point n configuration 2 FWPnCFG2
0x392+8n (0≤n≤7)Instruction fetch monitor point n configuration 3 FWPnCFG30x393+8n (0≤n≤7)Instruction fetch monitor point n configuration 4 FWPnCFG40x500Debug register DBG
0x501Debug exception return address DERA0x502Debug data save DSAVE

For processors that have implemented the LVZ virtualization extension, there is an additional set of CSR registers for controlling virtualization.

NumberName
0x15Guest TLB control GTLBC
0x16TLBRD read Guest item TRGP
0x50Guest status GSTAT
0x51Guest control GCTL
0x52Guest interrupt control GINTC
0x53Guest counter compensation GCNTC

GCSR Register Group

In LoongArch processors that implement virtualization, there is an additional group of GCSR (Guest Control and Status Register) registers.

Process of Entering Guest Mode (from Linux KVM source code)

  1. switch_to_guest:
  2. Clear the CSR.ECFG.VS field (set to 0, i.e., all exceptions share one entry address)
  3. Read the guest eentry saved in the Hypervisor (guest OS interrupt vector address) -> GEENTRY
    1. Then write GEENTRY to CSR.EENTRY
  4. Read the guest era saved in the Hypervisor (guest OS exception return address) -> GPC
    1. Then write GPC to CSR.ERA
  5. Read the CSR.PGDL global page table address, save it in the Hypervisor
  6. Load the guest pgdl from the Hypervisor into CSR.PGDL
  7. Read CSR.GSTAT.GID and CSR.GTLBC.TGID, write to CSR.GTLBC
  8. Set CSR.PRMD.PIE to 1, turn on Hypervisor-level global interrupts
  9. Set CSR.GSTAT.PGM to 1, the purpose is to make the ertn instruction enter guest mode
  10. The Hypervisor restores the guest's general registers (GPRS) saved by itself to the hardware registers (restore the scene)
  11. Execute the ertn instruction, enter guest mode
codesubcodeAbbreviationIntroduction
22-GSPRGuest-sensitive privileged resource exception, triggered by cpucfg, idle, cacop instructions, and when the virtual machine accesses non-existent GCSR and IOCSR, forcing a trap into the Hypervisor for processing (such as software simulation)
23-HVCException triggered by the hvcl supercall instruction
240GCMGuest GCSR software modification exception
241GCHCGuest GCSR hardware modification exception

Process of Handling Exceptions Under Guest Mode (from Linux KVM source code)

  1. kvm_exc_entry:

  2. The Hypervisor first saves the guest's general registers (GPRS), protecting the scene.

  3. The Hypervisor saves CSR.ESTAT -> host ESTAT

  4. The Hypervisor saves CSR.ERA -> GPC

  5. The Hypervisor saves CSR.BADV -> host BADV, i.e., when an address error exception is triggered, the erroneous virtual address is recorded

  6. The Hypervisor saves CSR.BADI -> host BADI, this register is used to record the instruction code of the instruction that triggered the synchronous class exception, synchronous class exceptions refer to all exceptions except for interrupts (INT), guest CSR hardware modification exceptions (GCHC), and machine error exceptions (MERR).

  7. Read the host ECFG saved by the Hypervisor, write to CSR.ECFG (i.e., switch to the host's exception configuration)

  8. Read the host EENTRY saved by the Hypervisor, write to CSR.EENTRY

  9. Read the host PGD saved by the Hypervisor, write to CSR.PGDL (restore the host page table global directory base, low half space)

  10. Set CSR.GSTAT.PGM off

  11. Clear the GTLBC.TGID field

  12. Restore kvm per CPU registers

    1. The kvm assembly involves KVM_ARCH_HTP, KVM_ARCH_HSP, KVM_ARCH_HPERCPU
  13. Jump to KVM_ARCH_HANDLE_EXIT to handle the exception

  14. Determine if the return of the function just now is <=0

    1. If <=0, continue running the host
    2. Otherwise, continue running the guest, save percpu registers, as it may switch to a different CPU to continue running the guest. Save host percpu registers to CSR.KSAVE register
  15. Jump to switch_to_guest

vCPU Context Registers to be Saved

According to the LoongArch function call specification, if you need to manually switch the CPU function running context, the registers to be saved are as follows (excluding floating point registers): $s0-$s9, $sp, $ra

References

[1] Loongson Zhongke Technology Co., Ltd. Loongson Architecture ELF psABI Specification. Version 2.01.

[2] Loongson Zhongke Technology Co., Ltd. Loongson Architecture Reference Manual. Volume One: Basic Architecture.

[3] https://github.com/torvalds/linux/blob/master/arch/loongarch/kvm/switch.S.