| .. SPDX-License-Identifier: GPL-2.0 |
| |
| =========================== |
| Hypercall Op-codes (hcalls) |
| =========================== |
| |
| Overview |
| ========= |
| |
| Virtualization on 64-bit Power Book3S Platforms is based on the PAPR |
| specification [1]_ which describes the run-time environment for a guest |
| operating system and how it should interact with the hypervisor for |
| privileged operations. Currently there are two PAPR compliant hypervisors: |
| |
| - **IBM PowerVM (PHYP)**: IBM's proprietary hypervisor that supports AIX, |
| IBM-i and Linux as supported guests (termed as Logical Partitions |
| or LPARS). It supports the full PAPR specification. |
| |
| - **Qemu/KVM**: Supports PPC64 linux guests running on a PPC64 linux host. |
| Though it only implements a subset of PAPR specification called LoPAPR [2]_. |
| |
| On PPC64 arch a guest kernel running on top of a PAPR hypervisor is called |
| a *pSeries guest*. A pseries guest runs in a supervisor mode (HV=0) and must |
| issue hypercalls to the hypervisor whenever it needs to perform an action |
| that is hypervisor privileged [3]_ or for other services managed by the |
| hypervisor. |
| |
| Hence a Hypercall (hcall) is essentially a request by the pseries guest |
| asking hypervisor to perform a privileged operation on behalf of the guest. The |
| guest issues a with necessary input operands. The hypervisor after performing |
| the privilege operation returns a status code and output operands back to the |
| guest. |
| |
| HCALL ABI |
| ========= |
| The ABI specification for a hcall between a pseries guest and PAPR hypervisor |
| is covered in section 14.5.3 of ref [2]_. Switch to the Hypervisor context is |
| done via the instruction **HVCS** that expects the Opcode for hcall is set in *r3* |
| and any in-arguments for the hcall are provided in registers *r4-r12*. If values |
| have to be passed through a memory buffer, the data stored in that buffer should be |
| in Big-endian byte order. |
| |
| Once control returns back to the guest after hypervisor has serviced the |
| 'HVCS' instruction the return value of the hcall is available in *r3* and any |
| out values are returned in registers *r4-r12*. Again like in case of in-arguments, |
| any out values stored in a memory buffer will be in Big-endian byte order. |
| |
| Powerpc arch code provides convenient wrappers named **plpar_hcall_xxx** defined |
| in a arch specific header [4]_ to issue hcalls from the linux kernel |
| running as pseries guest. |
| |
| Register Conventions |
| ==================== |
| |
| Any hcall should follow same register convention as described in section 2.2.1.1 |
| of "64-Bit ELF V2 ABI Specification: Power Architecture"[5]_. Table below |
| summarizes these conventions: |
| |
| +----------+----------+-------------------------------------------+ |
| | Register |Volatile | Purpose | |
| | Range |(Y/N) | | |
| +==========+==========+===========================================+ |
| | r0 | Y | Optional-usage | |
| +----------+----------+-------------------------------------------+ |
| | r1 | N | Stack Pointer | |
| +----------+----------+-------------------------------------------+ |
| | r2 | N | TOC | |
| +----------+----------+-------------------------------------------+ |
| | r3 | Y | hcall opcode/return value | |
| +----------+----------+-------------------------------------------+ |
| | r4-r10 | Y | in and out values | |
| +----------+----------+-------------------------------------------+ |
| | r11 | Y | Optional-usage/Environmental pointer | |
| +----------+----------+-------------------------------------------+ |
| | r12 | Y | Optional-usage/Function entry address at | |
| | | | global entry point | |
| +----------+----------+-------------------------------------------+ |
| | r13 | N | Thread-Pointer | |
| +----------+----------+-------------------------------------------+ |
| | r14-r31 | N | Local Variables | |
| +----------+----------+-------------------------------------------+ |
| | LR | Y | Link Register | |
| +----------+----------+-------------------------------------------+ |
| | CTR | Y | Loop Counter | |
| +----------+----------+-------------------------------------------+ |
| | XER | Y | Fixed-point exception register. | |
| +----------+----------+-------------------------------------------+ |
| | CR0-1 | Y | Condition register fields. | |
| +----------+----------+-------------------------------------------+ |
| | CR2-4 | N | Condition register fields. | |
| +----------+----------+-------------------------------------------+ |
| | CR5-7 | Y | Condition register fields. | |
| +----------+----------+-------------------------------------------+ |
| | Others | N | | |
| +----------+----------+-------------------------------------------+ |
| |
| DRC & DRC Indexes |
| ================= |
| :: |
| |
| DR1 Guest |
| +--+ +------------+ +---------+ |
| | | <----> | | | User | |
| +--+ DRC1 | | DRC | Space | |
| | PAPR | Index +---------+ |
| DR2 | Hypervisor | | | |
| +--+ | | <-----> | Kernel | |
| | | <----> | | Hcall | | |
| +--+ DRC2 +------------+ +---------+ |
| |
| PAPR hypervisor terms shared hardware resources like PCI devices, NVDIMMs etc |
| available for use by LPARs as Dynamic Resource (DR). When a DR is allocated to |
| an LPAR, PHYP creates a data-structure called Dynamic Resource Connector (DRC) |
| to manage LPAR access. An LPAR refers to a DRC via an opaque 32-bit number |
| called DRC-Index. The DRC-index value is provided to the LPAR via device-tree |
| where its present as an attribute in the device tree node associated with the |
| DR. |
| |
| HCALL Return-values |
| =================== |
| |
| After servicing the hcall, hypervisor sets the return-value in *r3* indicating |
| success or failure of the hcall. In case of a failure an error code indicates |
| the cause for error. These codes are defined and documented in arch specific |
| header [4]_. |
| |
| In some cases a hcall can potentially take a long time and need to be issued |
| multiple times in order to be completely serviced. These hcalls will usually |
| accept an opaque value *continue-token* within there argument list and a |
| return value of *H_CONTINUE* indicates that hypervisor hasn't still finished |
| servicing the hcall yet. |
| |
| To make such hcalls the guest need to set *continue-token == 0* for the |
| initial call and use the hypervisor returned value of *continue-token* |
| for each subsequent hcall until hypervisor returns a non *H_CONTINUE* |
| return value. |
| |
| HCALL Op-codes |
| ============== |
| |
| Below is a partial list of HCALLs that are supported by PHYP. For the |
| corresponding opcode values please look into the arch specific header [4]_: |
| |
| **H_SCM_READ_METADATA** |
| |
| | Input: *drcIndex, offset, buffer-address, numBytesToRead* |
| | Out: *numBytesRead* |
| | Return Value: *H_Success, H_Parameter, H_P2, H_P3, H_Hardware* |
| |
| Given a DRC Index of an NVDIMM, read N-bytes from the metadata area |
| associated with it, at a specified offset and copy it to provided buffer. |
| The metadata area stores configuration information such as label information, |
| bad-blocks etc. The metadata area is located out-of-band of NVDIMM storage |
| area hence a separate access semantics is provided. |
| |
| **H_SCM_WRITE_METADATA** |
| |
| | Input: *drcIndex, offset, data, numBytesToWrite* |
| | Out: *None* |
| | Return Value: *H_Success, H_Parameter, H_P2, H_P4, H_Hardware* |
| |
| Given a DRC Index of an NVDIMM, write N-bytes to the metadata area |
| associated with it, at the specified offset and from the provided buffer. |
| |
| **H_SCM_BIND_MEM** |
| |
| | Input: *drcIndex, startingScmBlockIndex, numScmBlocksToBind,* |
| | *targetLogicalMemoryAddress, continue-token* |
| | Out: *continue-token, targetLogicalMemoryAddress, numScmBlocksToBound* |
| | Return Value: *H_Success, H_Parameter, H_P2, H_P3, H_P4, H_Overlap,* |
| | *H_Too_Big, H_P5, H_Busy* |
| |
| Given a DRC-Index of an NVDIMM, map a continuous SCM blocks range |
| *(startingScmBlockIndex, startingScmBlockIndex+numScmBlocksToBind)* to the guest |
| at *targetLogicalMemoryAddress* within guest physical address space. In |
| case *targetLogicalMemoryAddress == 0xFFFFFFFF_FFFFFFFF* then hypervisor |
| assigns a target address to the guest. The HCALL can fail if the Guest has |
| an active PTE entry to the SCM block being bound. |
| |
| **H_SCM_UNBIND_MEM** |
| | Input: drcIndex, startingScmLogicalMemoryAddress, numScmBlocksToUnbind |
| | Out: numScmBlocksUnbound |
| | Return Value: *H_Success, H_Parameter, H_P2, H_P3, H_In_Use, H_Overlap,* |
| | *H_Busy, H_LongBusyOrder1mSec, H_LongBusyOrder10mSec* |
| |
| Given a DRC-Index of an NVDimm, unmap *numScmBlocksToUnbind* SCM blocks starting |
| at *startingScmLogicalMemoryAddress* from guest physical address space. The |
| HCALL can fail if the Guest has an active PTE entry to the SCM block being |
| unbound. |
| |
| **H_SCM_QUERY_BLOCK_MEM_BINDING** |
| |
| | Input: *drcIndex, scmBlockIndex* |
| | Out: *Guest-Physical-Address* |
| | Return Value: *H_Success, H_Parameter, H_P2, H_NotFound* |
| |
| Given a DRC-Index and an SCM Block index return the guest physical address to |
| which the SCM block is mapped to. |
| |
| **H_SCM_QUERY_LOGICAL_MEM_BINDING** |
| |
| | Input: *Guest-Physical-Address* |
| | Out: *drcIndex, scmBlockIndex* |
| | Return Value: *H_Success, H_Parameter, H_P2, H_NotFound* |
| |
| Given a guest physical address return which DRC Index and SCM block is mapped |
| to that address. |
| |
| **H_SCM_UNBIND_ALL** |
| |
| | Input: *scmTargetScope, drcIndex* |
| | Out: *None* |
| | Return Value: *H_Success, H_Parameter, H_P2, H_P3, H_In_Use, H_Busy,* |
| | *H_LongBusyOrder1mSec, H_LongBusyOrder10mSec* |
| |
| Depending on the Target scope unmap all SCM blocks belonging to all NVDIMMs |
| or all SCM blocks belonging to a single NVDIMM identified by its drcIndex |
| from the LPAR memory. |
| |
| **H_SCM_HEALTH** |
| |
| | Input: drcIndex |
| | Out: *health-bitmap (r4), health-bit-valid-bitmap (r5)* |
| | Return Value: *H_Success, H_Parameter, H_Hardware* |
| |
| Given a DRC Index return the info on predictive failure and overall health of |
| the PMEM device. The asserted bits in the health-bitmap indicate one or more states |
| (described in table below) of the PMEM device and health-bit-valid-bitmap indicate |
| which bits in health-bitmap are valid. The bits are reported in |
| reverse bit ordering for example a value of 0xC400000000000000 |
| indicates bits 0, 1, and 5 are valid. |
| |
| Health Bitmap Flags: |
| |
| +------+-----------------------------------------------------------------------+ |
| | Bit | Definition | |
| +======+=======================================================================+ |
| | 00 | PMEM device is unable to persist memory contents. | |
| | | If the system is powered down, nothing will be saved. | |
| +------+-----------------------------------------------------------------------+ |
| | 01 | PMEM device failed to persist memory contents. Either contents were | |
| | | not saved successfully on power down or were not restored properly on | |
| | | power up. | |
| +------+-----------------------------------------------------------------------+ |
| | 02 | PMEM device contents are persisted from previous IPL. The data from | |
| | | the last boot were successfully restored. | |
| +------+-----------------------------------------------------------------------+ |
| | 03 | PMEM device contents are not persisted from previous IPL. There was no| |
| | | data to restore from the last boot. | |
| +------+-----------------------------------------------------------------------+ |
| | 04 | PMEM device memory life remaining is critically low | |
| +------+-----------------------------------------------------------------------+ |
| | 05 | PMEM device will be garded off next IPL due to failure | |
| +------+-----------------------------------------------------------------------+ |
| | 06 | PMEM device contents cannot persist due to current platform health | |
| | | status. A hardware failure may prevent data from being saved or | |
| | | restored. | |
| +------+-----------------------------------------------------------------------+ |
| | 07 | PMEM device is unable to persist memory contents in certain conditions| |
| +------+-----------------------------------------------------------------------+ |
| | 08 | PMEM device is encrypted | |
| +------+-----------------------------------------------------------------------+ |
| | 09 | PMEM device has successfully completed a requested erase or secure | |
| | | erase procedure. | |
| +------+-----------------------------------------------------------------------+ |
| |10:63 | Reserved / Unused | |
| +------+-----------------------------------------------------------------------+ |
| |
| **H_SCM_PERFORMANCE_STATS** |
| |
| | Input: drcIndex, resultBuffer Addr |
| | Out: None |
| | Return Value: *H_Success, H_Parameter, H_Unsupported, H_Hardware, H_Authority, H_Privilege* |
| |
| Given a DRC Index collect the performance statistics for NVDIMM and copy them |
| to the resultBuffer. |
| |
| **H_SCM_FLUSH** |
| |
| | Input: *drcIndex, continue-token* |
| | Out: *continue-token* |
| | Return Value: *H_SUCCESS, H_Parameter, H_P2, H_BUSY* |
| |
| Given a DRC Index Flush the data to backend NVDIMM device. |
| |
| The hcall returns H_BUSY when the flush takes longer time and the hcall needs |
| to be issued multiple times in order to be completely serviced. The |
| *continue-token* from the output to be passed in the argument list of |
| subsequent hcalls to the hypervisor until the hcall is completely serviced |
| at which point H_SUCCESS or other error is returned by the hypervisor. |
| |
| References |
| ========== |
| .. [1] "Power Architecture Platform Reference" |
| https://en.wikipedia.org/wiki/Power_Architecture_Platform_Reference |
| .. [2] "Linux on Power Architecture Platform Reference" |
| https://members.openpowerfoundation.org/document/dl/469 |
| .. [3] "Definitions and Notation" Book III-Section 14.5.3 |
| https://openpowerfoundation.org/?resource_lib=power-isa-version-3-0 |
| .. [4] arch/powerpc/include/asm/hvcall.h |
| .. [5] "64-Bit ELF V2 ABI Specification: Power Architecture" |
| https://openpowerfoundation.org/?resource_lib=64-bit-elf-v2-abi-specification-power-architecture |
