blob: 0595098a74d9db1d09d960574ebdc5ad7318b7c4 [file] [log] [blame]
Mike Rapoportaa9f34e2018-03-21 21:22:22 +02001=====================================
Jérôme Glissebffc33e2017-09-08 16:11:19 -07002Heterogeneous Memory Management (HMM)
Mike Rapoportaa9f34e2018-03-21 21:22:22 +02003=====================================
Jérôme Glissebffc33e2017-09-08 16:11:19 -07004
Jérôme Glissee8eddfd2018-04-10 16:29:16 -07005Provide infrastructure and helpers to integrate non-conventional memory (device
6memory like GPU on board memory) into regular kernel path, with the cornerstone
7of this being specialized struct page for such memory (see sections 5 to 7 of
8this document).
Jérôme Glissebffc33e2017-09-08 16:11:19 -07009
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070010HMM also provides optional helpers for SVM (Share Virtual Memory), i.e.,
Ralph Campbell2076e5c2019-05-06 16:29:38 -070011allowing a device to transparently access program addresses coherently with
Jonathan Corbet24844fd2018-04-16 14:25:08 -060012the CPU meaning that any valid pointer on the CPU is also a valid pointer
13for the device. This is becoming mandatory to simplify the use of advanced
14heterogeneous computing where GPU, DSP, or FPGA are used to perform various
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070015computations on behalf of a process.
Ralph Campbell76ea4702018-04-10 16:28:11 -070016
17This document is divided as follows: in the first section I expose the problems
18related to using device specific memory allocators. In the second section, I
19expose the hardware limitations that are inherent to many platforms. The third
20section gives an overview of the HMM design. The fourth section explains how
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070021CPU page-table mirroring works and the purpose of HMM in this context. The
Ralph Campbell76ea4702018-04-10 16:28:11 -070022fifth section deals with how device memory is represented inside the kernel.
Ralph Campbell2076e5c2019-05-06 16:29:38 -070023Finally, the last section presents a new migration helper that allows
24leveraging the device DMA engine.
Jérôme Glissebffc33e2017-09-08 16:11:19 -070025
Mike Rapoportaa9f34e2018-03-21 21:22:22 +020026.. contents:: :local:
Jérôme Glissebffc33e2017-09-08 16:11:19 -070027
Jonathan Corbet24844fd2018-04-16 14:25:08 -060028Problems of using a device specific memory allocator
29====================================================
Jérôme Glissebffc33e2017-09-08 16:11:19 -070030
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070031Devices with a large amount of on board memory (several gigabytes) like GPUs
Ralph Campbell76ea4702018-04-10 16:28:11 -070032have historically managed their memory through dedicated driver specific APIs.
33This creates a disconnect between memory allocated and managed by a device
34driver and regular application memory (private anonymous, shared memory, or
35regular file backed memory). From here on I will refer to this aspect as split
36address space. I use shared address space to refer to the opposite situation:
37i.e., one in which any application memory region can be used by a device
38transparently.
Jérôme Glissebffc33e2017-09-08 16:11:19 -070039
Ralph Campbell2076e5c2019-05-06 16:29:38 -070040Split address space happens because devices can only access memory allocated
41through a device specific API. This implies that all memory objects in a program
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070042are not equal from the device point of view which complicates large programs
43that rely on a wide set of libraries.
Jérôme Glissebffc33e2017-09-08 16:11:19 -070044
Ralph Campbell2076e5c2019-05-06 16:29:38 -070045Concretely, this means that code that wants to leverage devices like GPUs needs
46to copy objects between generically allocated memory (malloc, mmap private, mmap
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070047share) and memory allocated through the device driver API (this still ends up
48with an mmap but of the device file).
Jérôme Glissebffc33e2017-09-08 16:11:19 -070049
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070050For flat data sets (array, grid, image, ...) this isn't too hard to achieve but
Ralph Campbell2076e5c2019-05-06 16:29:38 -070051for complex data sets (list, tree, ...) it's hard to get right. Duplicating a
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070052complex data set needs to re-map all the pointer relations between each of its
Ralph Campbell2076e5c2019-05-06 16:29:38 -070053elements. This is error prone and programs get harder to debug because of the
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070054duplicate data set and addresses.
Jérôme Glissebffc33e2017-09-08 16:11:19 -070055
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070056Split address space also means that libraries cannot transparently use data
Ralph Campbell76ea4702018-04-10 16:28:11 -070057they are getting from the core program or another library and thus each library
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070058might have to duplicate its input data set using the device specific memory
Ralph Campbell76ea4702018-04-10 16:28:11 -070059allocator. Large projects suffer from this and waste resources because of the
60various memory copies.
Jérôme Glissebffc33e2017-09-08 16:11:19 -070061
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070062Duplicating each library API to accept as input or output memory allocated by
Jérôme Glissebffc33e2017-09-08 16:11:19 -070063each device specific allocator is not a viable option. It would lead to a
Ralph Campbell76ea4702018-04-10 16:28:11 -070064combinatorial explosion in the library entry points.
Jérôme Glissebffc33e2017-09-08 16:11:19 -070065
Ralph Campbell76ea4702018-04-10 16:28:11 -070066Finally, with the advance of high level language constructs (in C++ but in
67other languages too) it is now possible for the compiler to leverage GPUs and
68other devices without programmer knowledge. Some compiler identified patterns
69are only do-able with a shared address space. It is also more reasonable to use
70a shared address space for all other patterns.
Jérôme Glissebffc33e2017-09-08 16:11:19 -070071
72
Jonathan Corbet24844fd2018-04-16 14:25:08 -060073I/O bus, device memory characteristics
74======================================
Jérôme Glissebffc33e2017-09-08 16:11:19 -070075
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070076I/O buses cripple shared address spaces due to a few limitations. Most I/O
77buses only allow basic memory access from device to main memory; even cache
Ralph Campbell2076e5c2019-05-06 16:29:38 -070078coherency is often optional. Access to device memory from a CPU is even more
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070079limited. More often than not, it is not cache coherent.
Jérôme Glissebffc33e2017-09-08 16:11:19 -070080
Ralph Campbell76ea4702018-04-10 16:28:11 -070081If we only consider the PCIE bus, then a device can access main memory (often
82through an IOMMU) and be cache coherent with the CPUs. However, it only allows
Ralph Campbell2076e5c2019-05-06 16:29:38 -070083a limited set of atomic operations from the device on main memory. This is worse
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070084in the other direction: the CPU can only access a limited range of the device
85memory and cannot perform atomic operations on it. Thus device memory cannot
Ralph Campbell76ea4702018-04-10 16:28:11 -070086be considered the same as regular memory from the kernel point of view.
Jérôme Glissebffc33e2017-09-08 16:11:19 -070087
88Another crippling factor is the limited bandwidth (~32GBytes/s with PCIE 4.0
Ralph Campbell76ea4702018-04-10 16:28:11 -070089and 16 lanes). This is 33 times less than the fastest GPU memory (1 TBytes/s).
90The final limitation is latency. Access to main memory from the device has an
91order of magnitude higher latency than when the device accesses its own memory.
Jérôme Glissebffc33e2017-09-08 16:11:19 -070092
Ralph Campbell76ea4702018-04-10 16:28:11 -070093Some platforms are developing new I/O buses or additions/modifications to PCIE
Ralph Campbell2076e5c2019-05-06 16:29:38 -070094to address some of these limitations (OpenCAPI, CCIX). They mainly allow
95two-way cache coherency between CPU and device and allow all atomic operations the
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070096architecture supports. Sadly, not all platforms are following this trend and
Ralph Campbell76ea4702018-04-10 16:28:11 -070097some major architectures are left without hardware solutions to these problems.
Jérôme Glissebffc33e2017-09-08 16:11:19 -070098
Jérôme Glissee8eddfd2018-04-10 16:29:16 -070099So for shared address space to make sense, not only must we allow devices to
100access any memory but we must also permit any memory to be migrated to device
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700101memory while the device is using it (blocking CPU access while it happens).
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700102
103
Jonathan Corbet24844fd2018-04-16 14:25:08 -0600104Shared address space and migration
105==================================
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700106
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700107HMM intends to provide two main features. The first one is to share the address
Ralph Campbell76ea4702018-04-10 16:28:11 -0700108space by duplicating the CPU page table in the device page table so the same
109address points to the same physical memory for any valid main memory address in
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700110the process address space.
111
Ralph Campbell76ea4702018-04-10 16:28:11 -0700112To achieve this, HMM offers a set of helpers to populate the device page table
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700113while keeping track of CPU page table updates. Device page table updates are
Ralph Campbell76ea4702018-04-10 16:28:11 -0700114not as easy as CPU page table updates. To update the device page table, you must
115allocate a buffer (or use a pool of pre-allocated buffers) and write GPU
116specific commands in it to perform the update (unmap, cache invalidations, and
Jérôme Glissee8eddfd2018-04-10 16:29:16 -0700117flush, ...). This cannot be done through common code for all devices. Hence
Ralph Campbell76ea4702018-04-10 16:28:11 -0700118why HMM provides helpers to factor out everything that can be while leaving the
119hardware specific details to the device driver.
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700120
Jérôme Glissee8eddfd2018-04-10 16:29:16 -0700121The second mechanism HMM provides is a new kind of ZONE_DEVICE memory that
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700122allows allocating a struct page for each page of device memory. Those pages
Jérôme Glissee8eddfd2018-04-10 16:29:16 -0700123are special because the CPU cannot map them. However, they allow migrating
Ralph Campbell76ea4702018-04-10 16:28:11 -0700124main memory to device memory using existing migration mechanisms and everything
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700125looks like a page that is swapped out to disk from the CPU point of view. Using a
126struct page gives the easiest and cleanest integration with existing mm
127mechanisms. Here again, HMM only provides helpers, first to hotplug new ZONE_DEVICE
Ralph Campbell76ea4702018-04-10 16:28:11 -0700128memory for the device memory and second to perform migration. Policy decisions
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700129of what and when to migrate is left to the device driver.
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700130
Ralph Campbell76ea4702018-04-10 16:28:11 -0700131Note that any CPU access to a device page triggers a page fault and a migration
132back to main memory. For example, when a page backing a given CPU address A is
133migrated from a main memory page to a device page, then any CPU access to
134address A triggers a page fault and initiates a migration back to main memory.
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700135
Ralph Campbell76ea4702018-04-10 16:28:11 -0700136With these two features, HMM not only allows a device to mirror process address
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700137space and keeps both CPU and device page tables synchronized, but also
138leverages device memory by migrating the part of the data set that is actively being
Ralph Campbell76ea4702018-04-10 16:28:11 -0700139used by the device.
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700140
141
Mike Rapoportaa9f34e2018-03-21 21:22:22 +0200142Address space mirroring implementation and API
143==============================================
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700144
Ralph Campbell76ea4702018-04-10 16:28:11 -0700145Address space mirroring's main objective is to allow duplication of a range of
146CPU page table into a device page table; HMM helps keep both synchronized. A
Jérôme Glissee8eddfd2018-04-10 16:29:16 -0700147device driver that wants to mirror a process address space must start with the
Jason Gunthorpea22dd502019-11-12 16:22:30 -0400148registration of a mmu_interval_notifier::
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700149
Jason Gunthorpe5292e242020-01-14 11:29:52 -0400150 int mmu_interval_notifier_insert(struct mmu_interval_notifier *interval_sub,
151 struct mm_struct *mm, unsigned long start,
152 unsigned long length,
153 const struct mmu_interval_notifier_ops *ops);
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700154
Jason Gunthorpe5292e242020-01-14 11:29:52 -0400155During the ops->invalidate() callback the device driver must perform the
156update action to the range (mark range read only, or fully unmap, etc.). The
157device must complete the update before the driver callback returns.
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700158
Ralph Campbell76ea4702018-04-10 16:28:11 -0700159When the device driver wants to populate a range of virtual addresses, it can
Christoph Hellwigd45d464b2019-07-25 17:56:47 -0700160use::
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700161
Jason Gunthorpebe957c82020-05-01 15:20:45 -0300162 int hmm_range_fault(struct hmm_range *range);
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700163
Jason Gunthorpe6bfef2f2020-03-27 17:00:16 -0300164It will trigger a page fault on missing or read-only entries if write access is
165requested (see below). Page faults use the generic mm page fault code path just
Marco Pagani090a7f12023-08-25 15:35:46 +0200166like a CPU page fault. The usage pattern is::
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700167
168 int driver_populate_range(...)
169 {
170 struct hmm_range range;
171 ...
Jérôme Glisse25f23a02019-05-13 17:19:55 -0700172
Jason Gunthorpe5292e242020-01-14 11:29:52 -0400173 range.notifier = &interval_sub;
Jérôme Glisse25f23a02019-05-13 17:19:55 -0700174 range.start = ...;
175 range.end = ...;
Jason Gunthorpe2733ea12020-05-01 15:20:48 -0300176 range.hmm_pfns = ...;
Jérôme Glissea3e0d412019-05-13 17:20:01 -0700177
Jason Gunthorpe5292e242020-01-14 11:29:52 -0400178 if (!mmget_not_zero(interval_sub->notifier.mm))
Jason Gunthorpea22dd502019-11-12 16:22:30 -0400179 return -EFAULT;
Jérôme Glisse25f23a02019-05-13 17:19:55 -0700180
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700181 again:
Jason Gunthorpe5292e242020-01-14 11:29:52 -0400182 range.notifier_seq = mmu_interval_read_begin(&interval_sub);
Michel Lespinasse3e4e28c2020-06-08 21:33:51 -0700183 mmap_read_lock(mm);
Jason Gunthorpe6bfef2f2020-03-27 17:00:16 -0300184 ret = hmm_range_fault(&range);
Jérôme Glisse25f23a02019-05-13 17:19:55 -0700185 if (ret) {
Michel Lespinasse3e4e28c2020-06-08 21:33:51 -0700186 mmap_read_unlock(mm);
Jason Gunthorpea22dd502019-11-12 16:22:30 -0400187 if (ret == -EBUSY)
188 goto again;
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700189 return ret;
Jérôme Glisse25f23a02019-05-13 17:19:55 -0700190 }
Michel Lespinasse3e4e28c2020-06-08 21:33:51 -0700191 mmap_read_unlock(mm);
Jason Gunthorpea22dd502019-11-12 16:22:30 -0400192
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700193 take_lock(driver->update);
Jason Gunthorpea22dd502019-11-12 16:22:30 -0400194 if (mmu_interval_read_retry(&ni, range.notifier_seq) {
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700195 release_lock(driver->update);
196 goto again;
197 }
198
Jason Gunthorpea22dd502019-11-12 16:22:30 -0400199 /* Use pfns array content to update device page table,
200 * under the update lock */
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700201
202 release_lock(driver->update);
203 return 0;
204 }
205
Ralph Campbell76ea4702018-04-10 16:28:11 -0700206The driver->update lock is the same lock that the driver takes inside its
Jason Gunthorpea22dd502019-11-12 16:22:30 -0400207invalidate() callback. That lock must be held before calling
208mmu_interval_read_retry() to avoid any race with a concurrent CPU page table
209update.
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700210
Jérôme Glisse023a0192019-05-13 17:20:05 -0700211Leverage default_flags and pfn_flags_mask
212=========================================
213
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700214The hmm_range struct has 2 fields, default_flags and pfn_flags_mask, that specify
215fault or snapshot policy for the whole range instead of having to set them
216for each entry in the pfns array.
Jérôme Glisse023a0192019-05-13 17:20:05 -0700217
Jason Gunthorpe2733ea12020-05-01 15:20:48 -0300218For instance if the device driver wants pages for a range with at least read
219permission, it sets::
Jérôme Glisse023a0192019-05-13 17:20:05 -0700220
Jason Gunthorpe2733ea12020-05-01 15:20:48 -0300221 range->default_flags = HMM_PFN_REQ_FAULT;
Jérôme Glisse023a0192019-05-13 17:20:05 -0700222 range->pfn_flags_mask = 0;
223
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700224and calls hmm_range_fault() as described above. This will fill fault all pages
Jérôme Glisse023a0192019-05-13 17:20:05 -0700225in the range with at least read permission.
226
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700227Now let's say the driver wants to do the same except for one page in the range for
228which it wants to have write permission. Now driver set::
Randy Dunlap91173c62019-05-31 22:29:57 -0700229
Jason Gunthorpe2733ea12020-05-01 15:20:48 -0300230 range->default_flags = HMM_PFN_REQ_FAULT;
231 range->pfn_flags_mask = HMM_PFN_REQ_WRITE;
232 range->pfns[index_of_write] = HMM_PFN_REQ_WRITE;
Jérôme Glisse023a0192019-05-13 17:20:05 -0700233
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700234With this, HMM will fault in all pages with at least read (i.e., valid) and for the
Jérôme Glisse023a0192019-05-13 17:20:05 -0700235address == range->start + (index_of_write << PAGE_SHIFT) it will fault with
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700236write permission i.e., if the CPU pte does not have write permission set then HMM
Jérôme Glisse023a0192019-05-13 17:20:05 -0700237will call handle_mm_fault().
238
Jason Gunthorpe2733ea12020-05-01 15:20:48 -0300239After hmm_range_fault completes the flag bits are set to the current state of
240the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be set if the page is
241writable.
Jérôme Glisse023a0192019-05-13 17:20:05 -0700242
243
Mike Rapoportaa9f34e2018-03-21 21:22:22 +0200244Represent and manage device memory from core kernel point of view
245=================================================================
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700246
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700247Several different designs were tried to support device memory. The first one
248used a device specific data structure to keep information about migrated memory
249and HMM hooked itself in various places of mm code to handle any access to
Ralph Campbell76ea4702018-04-10 16:28:11 -0700250addresses that were backed by device memory. It turns out that this ended up
251replicating most of the fields of struct page and also needed many kernel code
252paths to be updated to understand this new kind of memory.
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700253
Ralph Campbell76ea4702018-04-10 16:28:11 -0700254Most kernel code paths never try to access the memory behind a page
255but only care about struct page contents. Because of this, HMM switched to
256directly using struct page for device memory which left most kernel code paths
257unaware of the difference. We only need to make sure that no one ever tries to
258map those pages from the CPU side.
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700259
Jonathan Corbet24844fd2018-04-16 14:25:08 -0600260Migration to and from device memory
261===================================
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700262
Ralph Campbellf7ebd9e2020-09-09 14:29:56 -0700263Because the CPU cannot access device memory directly, the device driver must
264use hardware DMA or device specific load/store instructions to migrate data.
265The migrate_vma_setup(), migrate_vma_pages(), and migrate_vma_finalize()
266functions are designed to make drivers easier to write and to centralize common
267code across drivers.
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700268
Ralph Campbellf7ebd9e2020-09-09 14:29:56 -0700269Before migrating pages to device private memory, special device private
270``struct page`` need to be created. These will be used as special "swap"
271page table entries so that a CPU process will fault if it tries to access
272a page that has been migrated to device private memory.
273
274These can be allocated and freed with::
275
276 struct resource *res;
277 struct dev_pagemap pagemap;
278
279 res = request_free_mem_region(&iomem_resource, /* number of bytes */,
280 "name of driver resource");
281 pagemap.type = MEMORY_DEVICE_PRIVATE;
282 pagemap.range.start = res->start;
283 pagemap.range.end = res->end;
284 pagemap.nr_range = 1;
285 pagemap.ops = &device_devmem_ops;
286 memremap_pages(&pagemap, numa_node_id());
287
288 memunmap_pages(&pagemap);
289 release_mem_region(pagemap.range.start, range_len(&pagemap.range));
290
291There are also devm_request_free_mem_region(), devm_memremap_pages(),
292devm_memunmap_pages(), and devm_release_mem_region() when the resources can
293be tied to a ``struct device``.
294
295The overall migration steps are similar to migrating NUMA pages within system
Mike Rapoport (IBM)ee865882023-02-01 11:41:55 +0200296memory (see Documentation/mm/page_migration.rst) but the steps are split
Ralph Campbellf7ebd9e2020-09-09 14:29:56 -0700297between device driver specific code and shared common code:
298
2991. ``mmap_read_lock()``
300
301 The device driver has to pass a ``struct vm_area_struct`` to
302 migrate_vma_setup() so the mmap_read_lock() or mmap_write_lock() needs to
303 be held for the duration of the migration.
304
3052. ``migrate_vma_setup(struct migrate_vma *args)``
306
307 The device driver initializes the ``struct migrate_vma`` fields and passes
308 the pointer to migrate_vma_setup(). The ``args->flags`` field is used to
309 filter which source pages should be migrated. For example, setting
310 ``MIGRATE_VMA_SELECT_SYSTEM`` will only migrate system memory and
311 ``MIGRATE_VMA_SELECT_DEVICE_PRIVATE`` will only migrate pages residing in
312 device private memory. If the latter flag is set, the ``args->pgmap_owner``
313 field is used to identify device private pages owned by the driver. This
314 avoids trying to migrate device private pages residing in other devices.
315 Currently only anonymous private VMA ranges can be migrated to or from
316 system memory and device private memory.
317
318 One of the first steps migrate_vma_setup() does is to invalidate other
319 device's MMUs with the ``mmu_notifier_invalidate_range_start(()`` and
320 ``mmu_notifier_invalidate_range_end()`` calls around the page table
321 walks to fill in the ``args->src`` array with PFNs to be migrated.
322 The ``invalidate_range_start()`` callback is passed a
323 ``struct mmu_notifier_range`` with the ``event`` field set to
Alistair Popple6b49bf62021-06-30 18:54:19 -0700324 ``MMU_NOTIFY_MIGRATE`` and the ``owner`` field set to
Ralph Campbellf7ebd9e2020-09-09 14:29:56 -0700325 the ``args->pgmap_owner`` field passed to migrate_vma_setup(). This is
326 allows the device driver to skip the invalidation callback and only
327 invalidate device private MMU mappings that are actually migrating.
328 This is explained more in the next section.
329
330 While walking the page tables, a ``pte_none()`` or ``is_zero_pfn()``
331 entry results in a valid "zero" PFN stored in the ``args->src`` array.
332 This lets the driver allocate device private memory and clear it instead
333 of copying a page of zeros. Valid PTE entries to system memory or
334 device private struct pages will be locked with ``lock_page()``, isolated
335 from the LRU (if system memory since device private pages are not on
336 the LRU), unmapped from the process, and a special migration PTE is
337 inserted in place of the original PTE.
338 migrate_vma_setup() also clears the ``args->dst`` array.
339
3403. The device driver allocates destination pages and copies source pages to
341 destination pages.
342
343 The driver checks each ``src`` entry to see if the ``MIGRATE_PFN_MIGRATE``
344 bit is set and skips entries that are not migrating. The device driver
345 can also choose to skip migrating a page by not filling in the ``dst``
346 array for that page.
347
348 The driver then allocates either a device private struct page or a
349 system memory page, locks the page with ``lock_page()``, and fills in the
350 ``dst`` array entry with::
351
Alistair Poppleab092432021-11-10 20:32:40 -0800352 dst[i] = migrate_pfn(page_to_pfn(dpage));
Ralph Campbellf7ebd9e2020-09-09 14:29:56 -0700353
354 Now that the driver knows that this page is being migrated, it can
355 invalidate device private MMU mappings and copy device private memory
356 to system memory or another device private page. The core Linux kernel
357 handles CPU page table invalidations so the device driver only has to
358 invalidate its own MMU mappings.
359
360 The driver can use ``migrate_pfn_to_page(src[i])`` to get the
361 ``struct page`` of the source and either copy the source page to the
362 destination or clear the destination device private memory if the pointer
363 is ``NULL`` meaning the source page was not populated in system memory.
364
3654. ``migrate_vma_pages()``
366
367 This step is where the migration is actually "committed".
368
369 If the source page was a ``pte_none()`` or ``is_zero_pfn()`` page, this
370 is where the newly allocated page is inserted into the CPU's page table.
371 This can fail if a CPU thread faults on the same page. However, the page
372 table is locked and only one of the new pages will be inserted.
373 The device driver will see that the ``MIGRATE_PFN_MIGRATE`` bit is cleared
374 if it loses the race.
375
376 If the source page was locked, isolated, etc. the source ``struct page``
377 information is now copied to destination ``struct page`` finalizing the
378 migration on the CPU side.
379
3805. Device driver updates device MMU page tables for pages still migrating,
381 rolling back pages not migrating.
382
383 If the ``src`` entry still has ``MIGRATE_PFN_MIGRATE`` bit set, the device
384 driver can update the device MMU and set the write enable bit if the
385 ``MIGRATE_PFN_WRITE`` bit is set.
386
3876. ``migrate_vma_finalize()``
388
389 This step replaces the special migration page table entry with the new
390 page's page table entry and releases the reference to the source and
391 destination ``struct page``.
392
3937. ``mmap_read_unlock()``
394
395 The lock can now be released.
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700396
Alistair Poppleb756a3b2021-06-30 18:54:25 -0700397Exclusive access memory
398=======================
399
400Some devices have features such as atomic PTE bits that can be used to implement
401atomic access to system memory. To support atomic operations to a shared virtual
402memory page such a device needs access to that page which is exclusive of any
403userspace access from the CPU. The ``make_device_exclusive_range()`` function
404can be used to make a memory range inaccessible from userspace.
405
406This replaces all mappings for pages in the given range with special swap
407entries. Any attempt to access the swap entry results in a fault which is
408resovled by replacing the entry with the original mapping. A driver gets
409notified that the mapping has been changed by MMU notifiers, after which point
410it will no longer have exclusive access to the page. Exclusive access is
Bjorn Helgaasd56b6992023-08-14 16:28:22 -0500411guaranteed to last until the driver drops the page lock and page reference, at
Alistair Poppleb756a3b2021-06-30 18:54:25 -0700412which point any CPU faults on the page may proceed as described.
413
Mike Rapoportaa9f34e2018-03-21 21:22:22 +0200414Memory cgroup (memcg) and rss accounting
415========================================
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700416
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700417For now, device memory is accounted as any regular page in rss counters (either
Ralph Campbell76ea4702018-04-10 16:28:11 -0700418anonymous if device page is used for anonymous, file if device page is used for
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700419file backed page, or shmem if device page is used for shared memory). This is a
Ralph Campbell76ea4702018-04-10 16:28:11 -0700420deliberate choice to keep existing applications, that might start using device
421memory without knowing about it, running unimpacted.
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700422
Jérôme Glissee8eddfd2018-04-10 16:29:16 -0700423A drawback is that the OOM killer might kill an application using a lot of
Ralph Campbell76ea4702018-04-10 16:28:11 -0700424device memory and not a lot of regular system memory and thus not freeing much
425system memory. We want to gather more real world experience on how applications
426and system react under memory pressure in the presence of device memory before
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700427deciding to account device memory differently.
428
429
Ralph Campbell76ea4702018-04-10 16:28:11 -0700430Same decision was made for memory cgroup. Device memory pages are accounted
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700431against same memory cgroup a regular page would be accounted to. This does
432simplify migration to and from device memory. This also means that migration
Jérôme Glissee8eddfd2018-04-10 16:29:16 -0700433back from device memory to regular memory cannot fail because it would
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700434go above memory cgroup limit. We might revisit this choice latter on once we
Ralph Campbell76ea4702018-04-10 16:28:11 -0700435get more experience in how device memory is used and its impact on memory
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700436resource control.
437
438
Ralph Campbell2076e5c2019-05-06 16:29:38 -0700439Note that device memory can never be pinned by a device driver nor through GUP
Jérôme Glissebffc33e2017-09-08 16:11:19 -0700440and thus such memory is always free upon process exit. Or when last reference
Ralph Campbell76ea4702018-04-10 16:28:11 -0700441is dropped in case of shared memory or file backed memory.