Documentation/misc-devices/ibmvmc.rst - linux - Git at Google

 .. SPDX-License-Identifier: GPL-2.0+

 ======================================================
 IBM Virtual Management Channel Kernel Driver (IBMVMC)
 ======================================================

 :Authors:
 	Dave Engebretsen <engebret@us.ibm.com>,
 	Adam Reznechek <adreznec@linux.vnet.ibm.com>,
 	Steven Royer <seroyer@linux.vnet.ibm.com>,
 	Bryant G. Ly <bryantly@linux.vnet.ibm.com>,

 Introduction
 ============

 Note: Knowledge of virtualization technology is required to understand
 this document.

 A good reference document would be:

 https://openpowerfoundation.org/wp-content/uploads/2016/05/LoPAPR_DRAFT_v11_24March2016_cmt1.pdf

 The Virtual Management Channel (VMC) is a logical device which provides an
 interface between the hypervisor and a management partition. This interface
 is like a message passing interface. This management partition is intended
 to provide an alternative to systems that use a Hardware Management
 Console (HMC) - based system management.

 The primary hardware management solution that is developed by IBM relies
 on an appliance server named the Hardware Management Console (HMC),
 packaged as an external tower or rack-mounted personal computer. In a
 Power Systems environment, a single HMC can manage multiple POWER
 processor-based systems.

 Management Application
 ----------------------

 In the management partition, a management application exists which enables
 a system administrator to configure the system’s partitioning
 characteristics via a command line interface (CLI) or Representational
 State Transfer Application (REST API's).

 The management application runs on a Linux logical partition on a
 POWER8 or newer processor-based server that is virtualized by PowerVM.
 System configuration, maintenance, and control functions which
 traditionally require an HMC can be implemented in the management
 application using a combination of HMC to hypervisor interfaces and
 existing operating system methods. This tool provides a subset of the
 functions implemented by the HMC and enables basic partition configuration.
 The set of HMC to hypervisor messages supported by the management
 application component are passed to the hypervisor over a VMC interface,
 which is defined below.

 The VMC enables the management partition to provide basic partitioning
 functions:

 - Logical Partitioning Configuration
 - Start, and stop actions for individual partitions
 - Display of partition status
 - Management of virtual Ethernet
 - Management of virtual Storage
 - Basic system management

 Virtual Management Channel (VMC)
 --------------------------------

 A logical device, called the Virtual Management Channel (VMC), is defined
 for communicating between the management application and the hypervisor. It
 basically creates the pipes that enable virtualization management
 software. This device is presented to a designated management partition as
 a virtual device.

 This communication device uses Command/Response Queue (CRQ) and the
 Remote Direct Memory Access (RDMA) interfaces. A three-way handshake is
 defined that must take place to establish that both the hypervisor and
 management partition sides of the channel are running prior to
 sending/receiving any of the protocol messages.

 This driver also utilizes Transport Event CRQs. CRQ messages are sent
 when the hypervisor detects one of the peer partitions has abnormally
 terminated, or one side has called H_FREE_CRQ to close their CRQ.
 Two new classes of CRQ messages are introduced for the VMC device. VMC
 Administrative messages are used for each partition using the VMC to
 communicate capabilities to their partner. HMC Interface messages are used
 for the actual flow of HMC messages between the management partition and
 the hypervisor. As most HMC messages far exceed the size of a CRQ buffer,
 a virtual DMA (RMDA) of the HMC message data is done prior to each HMC
 Interface CRQ message. Only the management partition drives RDMA
 operations; hypervisors never directly cause the movement of message data.


 Terminology
 -----------
 RDMA
         Remote Direct Memory Access is DMA transfer from the server to its
         client or from the server to its partner partition. DMA refers
         to both physical I/O to and from memory operations and to memory
         to memory move operations.
 CRQ
         Command/Response Queue a facility which is used to communicate
         between partner partitions. Transport events which are signaled
         from the hypervisor to partition are also reported in this queue.

 Example Management Partition VMC Driver Interface
 =================================================

 This section provides an example for the management application
 implementation where a device driver is used to interface to the VMC
 device. This driver consists of a new device, for example /dev/ibmvmc,
 which provides interfaces to open, close, read, write, and perform
 ioctl’s against the VMC device.

 VMC Interface Initialization
 ----------------------------

 The device driver is responsible for initializing the VMC when the driver
 is loaded. It first creates and initializes the CRQ. Next, an exchange of
 VMC capabilities is performed to indicate the code version and number of
 resources available in both the management partition and the hypervisor.
 Finally, the hypervisor requests that the management partition create an
 initial pool of VMC buffers, one buffer for each possible HMC connection,
 which will be used for management application  session initialization.
 Prior to completion of this initialization sequence, the device returns
 EBUSY to open() calls. EIO is returned for all open() failures.

 ::

         Management Partition		Hypervisor
                         CRQ INIT
         ---------------------------------------->
         	   CRQ INIT COMPLETE
         <----------------------------------------
         	      CAPABILITIES
         ---------------------------------------->
         	 CAPABILITIES RESPONSE
         <----------------------------------------
               ADD BUFFER (HMC IDX=0,1,..)         _
         <----------------------------------------  |
         	  ADD BUFFER RESPONSE              | - Perform # HMCs Iterations
         ----------------------------------------> -

 VMC Interface Open
 ------------------

 After the basic VMC channel has been initialized, an HMC session level
 connection can be established. The application layer performs an open() to
 the VMC device and executes an ioctl() against it, indicating the HMC ID
 (32 bytes of data) for this session. If the VMC device is in an invalid
 state, EIO will be returned for the ioctl(). The device driver creates a
 new HMC session value (ranging from 1 to 255) and HMC index value (starting
 at index 0 and ranging to 254) for this HMC ID. The driver then does an
 RDMA of the HMC ID to the hypervisor, and then sends an Interface Open
 message to the hypervisor to establish the session over the VMC. After the
 hypervisor receives this information, it sends Add Buffer messages to the
 management partition to seed an initial pool of buffers for the new HMC
 connection. Finally, the hypervisor sends an Interface Open Response
 message, to indicate that it is ready for normal runtime messaging. The
 following illustrates this VMC flow:

 ::

         Management Partition             Hypervisor
         	      RDMA HMC ID
         ---------------------------------------->
         	    Interface Open
         ---------------------------------------->
         	      Add Buffer                  _
         <----------------------------------------  |
         	  Add Buffer Response              | - Perform N Iterations
         ----------------------------------------> -
         	Interface Open Response
         <----------------------------------------

 VMC Interface Runtime
 ---------------------

 During normal runtime, the management application and the hypervisor
 exchange HMC messages via the Signal VMC message and RDMA operations. When
 sending data to the hypervisor, the management application performs a
 write() to the VMC device, and the driver RDMA’s the data to the hypervisor
 and then sends a Signal Message. If a write() is attempted before VMC
 device buffers have been made available by the hypervisor, or no buffers
 are currently available, EBUSY is returned in response to the write(). A
 write() will return EIO for all other errors, such as an invalid device
 state. When the hypervisor sends a message to the management, the data is
 put into a VMC buffer and an Signal Message is sent to the VMC driver in
 the management partition. The driver RDMA’s the buffer into the partition
 and passes the data up to the appropriate management application via a
 read() to the VMC device. The read() request blocks if there is no buffer
 available to read. The management application may use select() to wait for
 the VMC device to become ready with data to read.

 ::

         Management Partition             Hypervisor
         		MSG RDMA
         ---------------------------------------->
         		SIGNAL MSG
         ---------------------------------------->
         		SIGNAL MSG
         <----------------------------------------
         		MSG RDMA
         <----------------------------------------

 VMC Interface Close
 -------------------

 HMC session level connections are closed by the management partition when
 the application layer performs a close() against the device. This action
 results in an Interface Close message flowing to the hypervisor, which
 causes the session to be terminated. The device driver must free any
 storage allocated for buffers for this HMC connection.

 ::

         Management Partition             Hypervisor
         	     INTERFACE CLOSE
         ---------------------------------------->
                 INTERFACE CLOSE RESPONSE
         <----------------------------------------

 Additional Information
 ======================

 For more information on the documentation for CRQ Messages, VMC Messages,
 HMC interface Buffers, and signal messages please refer to the Linux on
 Power Architecture Platform Reference. Section F.
	.. SPDX-License-Identifier: GPL-2.0+

	======================================================
	IBM Virtual Management Channel Kernel Driver (IBMVMC)
	======================================================

	:Authors:
	Dave Engebretsen <engebret@us.ibm.com>,
	Adam Reznechek <adreznec@linux.vnet.ibm.com>,
	Steven Royer <seroyer@linux.vnet.ibm.com>,
	Bryant G. Ly <bryantly@linux.vnet.ibm.com>,

	Introduction
	============

	Note: Knowledge of virtualization technology is required to understand
	this document.

	A good reference document would be:

	https://openpowerfoundation.org/wp-content/uploads/2016/05/LoPAPR_DRAFT_v11_24March2016_cmt1.pdf

	The Virtual Management Channel (VMC) is a logical device which provides an
	interface between the hypervisor and a management partition. This interface
	is like a message passing interface. This management partition is intended
	to provide an alternative to systems that use a Hardware Management
	Console (HMC) - based system management.

	The primary hardware management solution that is developed by IBM relies
	on an appliance server named the Hardware Management Console (HMC),
	packaged as an external tower or rack-mounted personal computer. In a
	Power Systems environment, a single HMC can manage multiple POWER
	processor-based systems.

	Management Application
	----------------------

	In the management partition, a management application exists which enables
	a system administrator to configure the system’s partitioning
	characteristics via a command line interface (CLI) or Representational
	State Transfer Application (REST API's).

	The management application runs on a Linux logical partition on a
	POWER8 or newer processor-based server that is virtualized by PowerVM.
	System configuration, maintenance, and control functions which
	traditionally require an HMC can be implemented in the management
	application using a combination of HMC to hypervisor interfaces and
	existing operating system methods. This tool provides a subset of the
	functions implemented by the HMC and enables basic partition configuration.
	The set of HMC to hypervisor messages supported by the management
	application component are passed to the hypervisor over a VMC interface,
	which is defined below.

	The VMC enables the management partition to provide basic partitioning
	functions:

	- Logical Partitioning Configuration
	- Start, and stop actions for individual partitions
	- Display of partition status
	- Management of virtual Ethernet
	- Management of virtual Storage
	- Basic system management

	Virtual Management Channel (VMC)
	--------------------------------

	A logical device, called the Virtual Management Channel (VMC), is defined
	for communicating between the management application and the hypervisor. It
	basically creates the pipes that enable virtualization management
	software. This device is presented to a designated management partition as
	a virtual device.

	This communication device uses Command/Response Queue (CRQ) and the
	Remote Direct Memory Access (RDMA) interfaces. A three-way handshake is
	defined that must take place to establish that both the hypervisor and
	management partition sides of the channel are running prior to
	sending/receiving any of the protocol messages.

	This driver also utilizes Transport Event CRQs. CRQ messages are sent
	when the hypervisor detects one of the peer partitions has abnormally
	terminated, or one side has called H_FREE_CRQ to close their CRQ.
	Two new classes of CRQ messages are introduced for the VMC device. VMC
	Administrative messages are used for each partition using the VMC to
	communicate capabilities to their partner. HMC Interface messages are used
	for the actual flow of HMC messages between the management partition and
	the hypervisor. As most HMC messages far exceed the size of a CRQ buffer,
	a virtual DMA (RMDA) of the HMC message data is done prior to each HMC
	Interface CRQ message. Only the management partition drives RDMA
	operations; hypervisors never directly cause the movement of message data.


	Terminology
	-----------
	RDMA
	Remote Direct Memory Access is DMA transfer from the server to its
	client or from the server to its partner partition. DMA refers
	to both physical I/O to and from memory operations and to memory
	to memory move operations.
	CRQ
	Command/Response Queue a facility which is used to communicate
	between partner partitions. Transport events which are signaled
	from the hypervisor to partition are also reported in this queue.

	Example Management Partition VMC Driver Interface
	=================================================

	This section provides an example for the management application
	implementation where a device driver is used to interface to the VMC
	device. This driver consists of a new device, for example /dev/ibmvmc,
	which provides interfaces to open, close, read, write, and perform
	ioctl’s against the VMC device.

	VMC Interface Initialization
	----------------------------

	The device driver is responsible for initializing the VMC when the driver
	is loaded. It first creates and initializes the CRQ. Next, an exchange of
	VMC capabilities is performed to indicate the code version and number of
	resources available in both the management partition and the hypervisor.
	Finally, the hypervisor requests that the management partition create an
	initial pool of VMC buffers, one buffer for each possible HMC connection,
	which will be used for management application session initialization.
	Prior to completion of this initialization sequence, the device returns
	EBUSY to open() calls. EIO is returned for all open() failures.

	::

	Management Partition Hypervisor
	CRQ INIT
	---------------------------------------->
	CRQ INIT COMPLETE
	<----------------------------------------
	CAPABILITIES
	---------------------------------------->
	CAPABILITIES RESPONSE
	<----------------------------------------
	ADD BUFFER (HMC IDX=0,1,..) _
	<---------------------------------------- \|
	ADD BUFFER RESPONSE \| - Perform # HMCs Iterations
	----------------------------------------> -

	VMC Interface Open
	------------------

	After the basic VMC channel has been initialized, an HMC session level
	connection can be established. The application layer performs an open() to
	the VMC device and executes an ioctl() against it, indicating the HMC ID
	(32 bytes of data) for this session. If the VMC device is in an invalid
	state, EIO will be returned for the ioctl(). The device driver creates a
	new HMC session value (ranging from 1 to 255) and HMC index value (starting
	at index 0 and ranging to 254) for this HMC ID. The driver then does an
	RDMA of the HMC ID to the hypervisor, and then sends an Interface Open
	message to the hypervisor to establish the session over the VMC. After the
	hypervisor receives this information, it sends Add Buffer messages to the
	management partition to seed an initial pool of buffers for the new HMC
	connection. Finally, the hypervisor sends an Interface Open Response
	message, to indicate that it is ready for normal runtime messaging. The
	following illustrates this VMC flow:

	::

	Management Partition Hypervisor
	RDMA HMC ID
	---------------------------------------->
	Interface Open
	---------------------------------------->
	Add Buffer _
	<---------------------------------------- \|
	Add Buffer Response \| - Perform N Iterations
	----------------------------------------> -
	Interface Open Response
	<----------------------------------------

	VMC Interface Runtime
	---------------------

	During normal runtime, the management application and the hypervisor
	exchange HMC messages via the Signal VMC message and RDMA operations. When
	sending data to the hypervisor, the management application performs a
	write() to the VMC device, and the driver RDMA’s the data to the hypervisor
	and then sends a Signal Message. If a write() is attempted before VMC
	device buffers have been made available by the hypervisor, or no buffers
	are currently available, EBUSY is returned in response to the write(). A
	write() will return EIO for all other errors, such as an invalid device
	state. When the hypervisor sends a message to the management, the data is
	put into a VMC buffer and an Signal Message is sent to the VMC driver in
	the management partition. The driver RDMA’s the buffer into the partition
	and passes the data up to the appropriate management application via a
	read() to the VMC device. The read() request blocks if there is no buffer
	available to read. The management application may use select() to wait for
	the VMC device to become ready with data to read.

	::

	Management Partition Hypervisor
	MSG RDMA
	---------------------------------------->
	SIGNAL MSG
	---------------------------------------->
	SIGNAL MSG
	<----------------------------------------
	MSG RDMA
	<----------------------------------------

	VMC Interface Close
	-------------------

	HMC session level connections are closed by the management partition when
	the application layer performs a close() against the device. This action
	results in an Interface Close message flowing to the hypervisor, which
	causes the session to be terminated. The device driver must free any
	storage allocated for buffers for this HMC connection.

	::

	Management Partition Hypervisor
	INTERFACE CLOSE
	---------------------------------------->
	INTERFACE CLOSE RESPONSE
	<----------------------------------------

	Additional Information
	======================

	For more information on the documentation for CRQ Messages, VMC Messages,
	HMC interface Buffers, and signal messages please refer to the Linux on
	Power Architecture Platform Reference. Section F.