| .. SPDX-License-Identifier: GPL-2.0 |
| |
| ================= |
| PCI NTB Function |
| ================= |
| |
| :Author: Kishon Vijay Abraham I <kishon@ti.com> |
| |
| PCI Non-Transparent Bridges (NTB) allow two host systems to communicate |
| with each other by exposing each host as a device to the other host. |
| NTBs typically support the ability to generate interrupts on the remote |
| machine, expose memory ranges as BARs, and perform DMA. They also support |
| scratchpads, which are areas of memory within the NTB that are accessible |
| from both machines. |
| |
| PCI NTB Function allows two different systems (or hosts) to communicate |
| with each other by configuring the endpoint instances in such a way that |
| transactions from one system are routed to the other system. |
| |
| In the below diagram, PCI NTB function configures the SoC with multiple |
| PCI Endpoint (EP) instances in such a way that transactions from one EP |
| controller are routed to the other EP controller. Once PCI NTB function |
| configures the SoC with multiple EP instances, HOST1 and HOST2 can |
| communicate with each other using SoC as a bridge. |
| |
| .. code-block:: text |
| |
| +-------------+ +-------------+ |
| | | | | |
| | HOST1 | | HOST2 | |
| | | | | |
| +------^------+ +------^------+ |
| | | |
| | | |
| +---------|-------------------------------------------------|---------+ |
| | +------v------+ +------v------+ | |
| | | | | | | |
| | | EP | | EP | | |
| | | CONTROLLER1 | | CONTROLLER2 | | |
| | | <-----------------------------------> | | |
| | | | | | | |
| | | | | | | |
| | | | SoC With Multiple EP Instances | | | |
| | | | (Configured using NTB Function) | | | |
| | +-------------+ +-------------+ | |
| +---------------------------------------------------------------------+ |
| |
| Constructs used for Implementing NTB |
| ==================================== |
| |
| 1) Config Region |
| 2) Self Scratchpad Registers |
| 3) Peer Scratchpad Registers |
| 4) Doorbell (DB) Registers |
| 5) Memory Window (MW) |
| |
| |
| Config Region: |
| -------------- |
| |
| Config Region is a construct that is specific to NTB implemented using NTB |
| Endpoint Function Driver. The host and endpoint side NTB function driver will |
| exchange information with each other using this region. Config Region has |
| Control/Status Registers for configuring the Endpoint Controller. Host can |
| write into this region for configuring the outbound Address Translation Unit |
| (ATU) and to indicate the link status. Endpoint can indicate the status of |
| commands issued by host in this region. Endpoint can also indicate the |
| scratchpad offset and number of memory windows to the host using this region. |
| |
| The format of Config Region is given below. All the fields here are 32 bits. |
| |
| .. code-block:: text |
| |
| +------------------------+ |
| | COMMAND | |
| +------------------------+ |
| | ARGUMENT | |
| +------------------------+ |
| | STATUS | |
| +------------------------+ |
| | TOPOLOGY | |
| +------------------------+ |
| | ADDRESS (LOWER 32) | |
| +------------------------+ |
| | ADDRESS (UPPER 32) | |
| +------------------------+ |
| | SIZE | |
| +------------------------+ |
| | NO OF MEMORY WINDOW | |
| +------------------------+ |
| | MEMORY WINDOW1 OFFSET | |
| +------------------------+ |
| | SPAD OFFSET | |
| +------------------------+ |
| | SPAD COUNT | |
| +------------------------+ |
| | DB ENTRY SIZE | |
| +------------------------+ |
| | DB DATA | |
| +------------------------+ |
| | : | |
| +------------------------+ |
| | : | |
| +------------------------+ |
| | DB DATA | |
| +------------------------+ |
| |
| |
| COMMAND: |
| |
| NTB function supports three commands: |
| |
| CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before |
| invoking this command, the host should allocate and initialize |
| MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the |
| Endpoint). The endpoint on receiving this command will configure |
| the outbound ATU such that transactions to Doorbell BAR will be routed |
| to the MSI/MSI-X address programmed by the host. The ARGUMENT |
| register should be populated with number of DBs to configure (in the |
| lower 16 bits) and if MSI or MSI-X should be configured (BIT 16). |
| |
| CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The |
| host invokes this command after allocating a buffer that can be |
| accessed by remote host. The allocated address should be programmed |
| in the ADDRESS register (64 bit), the size should be programmed in |
| the SIZE register and the memory window index should be programmed |
| in the ARGUMENT register. The endpoint on receiving this command |
| will configure the outbound ATU such that transactions to MW BAR |
| are routed to the address provided by the host. |
| |
| CMD_LINK_UP (0x3): Command to indicate an NTB application is |
| bound to the EP device on the host side. Once the endpoint |
| receives this command from both the hosts, the endpoint will |
| raise a LINK_UP event to both the hosts to indicate the host |
| NTB applications can start communicating with each other. |
| |
| ARGUMENT: |
| |
| The value of this register is based on the commands issued in |
| command register. See COMMAND section for more information. |
| |
| TOPOLOGY: |
| |
| Set to NTB_TOPO_B2B_USD for Primary interface |
| Set to NTB_TOPO_B2B_DSD for Secondary interface |
| |
| ADDRESS/SIZE: |
| |
| Address and Size to be used while configuring the memory window. |
| See "CMD_CONFIGURE_MW" for more info. |
| |
| MEMORY WINDOW1 OFFSET: |
| |
| Memory Window 1 and Doorbell registers are packed together in the |
| same BAR. The initial portion of the region will have doorbell |
| registers and the latter portion of the region is for memory window 1. |
| This register will specify the offset of the memory window 1. |
| |
| NO OF MEMORY WINDOW: |
| |
| Specifies the number of memory windows supported by the NTB device. |
| |
| SPAD OFFSET: |
| |
| Self scratchpad region and config region are packed together in the |
| same BAR. The initial portion of the region will have config region |
| and the latter portion of the region is for self scratchpad. This |
| register will specify the offset of the self scratchpad registers. |
| |
| SPAD COUNT: |
| |
| Specifies the number of scratchpad registers supported by the NTB |
| device. |
| |
| DB ENTRY SIZE: |
| |
| Used to determine the offset within the DB BAR that should be written |
| in order to raise doorbell. EPF NTB can use either MSI or MSI-X to |
| ring doorbell (MSI-X support will be added later). MSI uses same |
| address for all the interrupts and MSI-X can provide different |
| addresses for different interrupts. The MSI/MSI-X address is provided |
| by the host and the address it gives is based on the MSI/MSI-X |
| implementation supported by the host. For instance, ARM platform |
| using GIC ITS will have the same MSI-X address for all the interrupts. |
| In order to support all the combinations and use the same mechanism |
| for both MSI and MSI-X, EPF NTB allocates a separate region in the |
| Outbound Address Space for each of the interrupts. This region will |
| be mapped to the MSI/MSI-X address provided by the host. If a host |
| provides the same address for all the interrupts, all the regions |
| will be translated to the same address. If a host provides different |
| addresses, the regions will be translated to different addresses. This |
| will ensure there is no difference while raising the doorbell. |
| |
| DB DATA: |
| |
| EPF NTB supports 32 interrupts, so there are 32 DB DATA registers. |
| This holds the MSI/MSI-X data that has to be written to MSI address |
| for raising doorbell interrupt. This will be populated by EPF NTB |
| while invoking CMD_CONFIGURE_DOORBELL. |
| |
| Scratchpad Registers: |
| --------------------- |
| |
| Each host has its own register space allocated in the memory of NTB endpoint |
| controller. They are both readable and writable from both sides of the bridge. |
| They are used by applications built over NTB and can be used to pass control |
| and status information between both sides of a device. |
| |
| Scratchpad registers has 2 parts |
| 1) Self Scratchpad: Host's own register space |
| 2) Peer Scratchpad: Remote host's register space. |
| |
| Doorbell Registers: |
| ------------------- |
| |
| Doorbell Registers are used by the hosts to interrupt each other. |
| |
| Memory Window: |
| -------------- |
| |
| Actual transfer of data between the two hosts will happen using the |
| memory window. |
| |
| Modeling Constructs: |
| ==================== |
| |
| There are 5 or more distinct regions (config, self scratchpad, peer |
| scratchpad, doorbell, one or more memory windows) to be modeled to achieve |
| NTB functionality. At least one memory window is required while more than |
| one is permitted. All these regions should be mapped to BARs for hosts to |
| access these regions. |
| |
| If one 32-bit BAR is allocated for each of these regions, the scheme would |
| look like this: |
| |
| ====== =============== |
| BAR NO CONSTRUCTS USED |
| ====== =============== |
| BAR0 Config Region |
| BAR1 Self Scratchpad |
| BAR2 Peer Scratchpad |
| BAR3 Doorbell |
| BAR4 Memory Window 1 |
| BAR5 Memory Window 2 |
| ====== =============== |
| |
| However if we allocate a separate BAR for each of the regions, there would not |
| be enough BARs for all the regions in a platform that supports only 64-bit |
| BARs. |
| |
| In order to be supported by most of the platforms, the regions should be |
| packed and mapped to BARs in a way that provides NTB functionality and |
| also makes sure the host doesn't access any region that it is not supposed |
| to. |
| |
| The following scheme is used in EPF NTB Function: |
| |
| ====== =============================== |
| BAR NO CONSTRUCTS USED |
| ====== =============================== |
| BAR0 Config Region + Self Scratchpad |
| BAR1 Peer Scratchpad |
| BAR2 Doorbell + Memory Window 1 |
| BAR3 Memory Window 2 |
| BAR4 Memory Window 3 |
| BAR5 Memory Window 4 |
| ====== =============================== |
| |
| With this scheme, for the basic NTB functionality 3 BARs should be sufficient. |
| |
| Modeling Config/Scratchpad Region: |
| ---------------------------------- |
| |
| .. code-block:: text |
| |
| +-----------------+------->+------------------+ +-----------------+ |
| | BAR0 | | CONFIG REGION | | BAR0 | |
| +-----------------+----+ +------------------+<-------+-----------------+ |
| | BAR1 | | |SCRATCHPAD REGION | | BAR1 | |
| +-----------------+ +-->+------------------+<-------+-----------------+ |
| | BAR2 | Local Memory | BAR2 | |
| +-----------------+ +-----------------+ |
| | BAR3 | | BAR3 | |
| +-----------------+ +-----------------+ |
| | BAR4 | | BAR4 | |
| +-----------------+ +-----------------+ |
| | BAR5 | | BAR5 | |
| +-----------------+ +-----------------+ |
| EP CONTROLLER 1 EP CONTROLLER 2 |
| |
| Above diagram shows Config region + Scratchpad region for HOST1 (connected to |
| EP controller 1) allocated in local memory. The HOST1 can access the config |
| region and scratchpad region (self scratchpad) using BAR0 of EP controller 1. |
| The peer host (HOST2 connected to EP controller 2) can also access this |
| scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This |
| diagram shows the case where Config region and Scratchpad regions are allocated |
| for HOST1, however the same is applicable for HOST2. |
| |
| Modeling Doorbell/Memory Window 1: |
| ---------------------------------- |
| |
| .. code-block:: text |
| |
| +-----------------+ +----->+----------------+-----------+-----------------+ |
| | BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 | |
| +-----------------+ | +----------------+ +-----------------+ |
| | BAR1 | | | Doorbell 2 +---------+ | | |
| +-----------------+----+ +----------------+ | | | |
| | BAR2 | | Doorbell 3 +-------+ | +-----------------+ |
| +-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 | |
| | BAR3 | | | Doorbell 4 +-----+ | +-----------------+ |
| +-----------------+ | |----------------+ | | | | |
| | BAR4 | | | | | | +-----------------+ |
| +-----------------+ | | MW1 +---+ | +-->+ MSI-X ADDRESS 3|| |
| | BAR5 | | | | | | +-----------------+ |
| +-----------------+ +----->-----------------+ | | | | |
| EP CONTROLLER 1 | | | | +-----------------+ |
| | | | +---->+ MSI-X ADDRESS 4 | |
| +----------------+ | +-----------------+ |
| EP CONTROLLER 2 | | | |
| (OB SPACE) | | | |
| +-------> MW1 | |
| | | |
| | | |
| +-----------------+ |
| | | |
| | | |
| | | |
| | | |
| | | |
| +-----------------+ |
| PCI Address Space |
| (Managed by HOST2) |
| |
| Above diagram shows how the doorbell and memory window 1 is mapped so that |
| HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access |
| buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and |
| memory window 1 regions are allocated in EP controller 2 outbound (OB) address |
| space. Allocating and configuring BARs for doorbell and memory window1 |
| is done during the initialization phase of NTB endpoint function driver. |
| Mapping from EP controller 2 OB space to PCI address space is done when HOST2 |
| sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL. |
| |
| Modeling Optional Memory Windows: |
| --------------------------------- |
| |
| This is modeled the same was as MW1 but each of the additional memory windows |
| is mapped to separate BARs. |
