Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 1 | Cache and TLB Flushing |
| 2 | Under Linux |
| 3 | |
| 4 | David S. Miller <davem@redhat.com> |
| 5 | |
| 6 | This document describes the cache/tlb flushing interfaces called |
| 7 | by the Linux VM subsystem. It enumerates over each interface, |
Francis Galiegue | a33f322 | 2010-04-23 00:08:02 +0200 | [diff] [blame] | 8 | describes its intended purpose, and what side effect is expected |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 9 | after the interface is invoked. |
| 10 | |
| 11 | The side effects described below are stated for a uniprocessor |
| 12 | implementation, and what is to happen on that single processor. The |
| 13 | SMP cases are a simple extension, in that you just extend the |
| 14 | definition such that the side effect for a particular interface occurs |
| 15 | on all processors in the system. Don't let this scare you into |
| 16 | thinking SMP cache/tlb flushing must be so inefficient, this is in |
| 17 | fact an area where many optimizations are possible. For example, |
| 18 | if it can be proven that a user address space has never executed |
KOSAKI Motohiro | de03c72 | 2011-05-24 17:12:15 -0700 | [diff] [blame] | 19 | on a cpu (see mm_cpumask()), one need not perform a flush |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 20 | for this address space on that cpu. |
| 21 | |
| 22 | First, the TLB flushing interfaces, since they are the simplest. The |
| 23 | "TLB" is abstracted under Linux as something the cpu uses to cache |
| 24 | virtual-->physical address translations obtained from the software |
| 25 | page tables. Meaning that if the software page tables change, it is |
| 26 | possible for stale translations to exist in this "TLB" cache. |
| 27 | Therefore when software page table changes occur, the kernel will |
| 28 | invoke one of the following flush methods _after_ the page table |
| 29 | changes occur: |
| 30 | |
| 31 | 1) void flush_tlb_all(void) |
| 32 | |
| 33 | The most severe flush of all. After this interface runs, |
| 34 | any previous page table modification whatsoever will be |
| 35 | visible to the cpu. |
| 36 | |
| 37 | This is usually invoked when the kernel page tables are |
| 38 | changed, since such translations are "global" in nature. |
| 39 | |
| 40 | 2) void flush_tlb_mm(struct mm_struct *mm) |
| 41 | |
| 42 | This interface flushes an entire user address space from |
| 43 | the TLB. After running, this interface must make sure that |
| 44 | any previous page table modifications for the address space |
| 45 | 'mm' will be visible to the cpu. That is, after running, |
| 46 | there will be no entries in the TLB for 'mm'. |
| 47 | |
| 48 | This interface is used to handle whole address space |
| 49 | page table operations such as what happens during |
| 50 | fork, and exec. |
| 51 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 52 | 3) void flush_tlb_range(struct vm_area_struct *vma, |
| 53 | unsigned long start, unsigned long end) |
| 54 | |
| 55 | Here we are flushing a specific range of (user) virtual |
| 56 | address translations from the TLB. After running, this |
| 57 | interface must make sure that any previous page table |
| 58 | modifications for the address space 'vma->vm_mm' in the range |
| 59 | 'start' to 'end-1' will be visible to the cpu. That is, after |
Paul Gortmaker | 548a195 | 2013-08-12 19:55:49 -0400 | [diff] [blame] | 60 | running, there will be no entries in the TLB for 'mm' for |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 61 | virtual addresses in the range 'start' to 'end-1'. |
| 62 | |
| 63 | The "vma" is the backing store being used for the region. |
| 64 | Primarily, this is used for munmap() type operations. |
| 65 | |
| 66 | The interface is provided in hopes that the port can find |
| 67 | a suitably efficient method for removing multiple page |
| 68 | sized translations from the TLB, instead of having the kernel |
| 69 | call flush_tlb_page (see below) for each entry which may be |
| 70 | modified. |
| 71 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 72 | 4) void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) |
| 73 | |
| 74 | This time we need to remove the PAGE_SIZE sized translation |
| 75 | from the TLB. The 'vma' is the backing structure used by |
| 76 | Linux to keep track of mmap'd regions for a process, the |
| 77 | address space is available via vma->vm_mm. Also, one may |
| 78 | test (vma->vm_flags & VM_EXEC) to see if this region is |
| 79 | executable (and thus could be in the 'instruction TLB' in |
| 80 | split-tlb type setups). |
| 81 | |
| 82 | After running, this interface must make sure that any previous |
| 83 | page table modification for address space 'vma->vm_mm' for |
| 84 | user virtual address 'addr' will be visible to the cpu. That |
| 85 | is, after running, there will be no entries in the TLB for |
| 86 | 'vma->vm_mm' for virtual address 'addr'. |
| 87 | |
| 88 | This is used primarily during fault processing. |
| 89 | |
Benjamin Herrenschmidt | 1c7037d | 2007-10-18 23:39:14 -0700 | [diff] [blame] | 90 | 5) void update_mmu_cache(struct vm_area_struct *vma, |
Russell King | 4b3073e | 2009-12-18 16:40:18 +0000 | [diff] [blame] | 91 | unsigned long address, pte_t *ptep) |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 92 | |
| 93 | At the end of every page fault, this routine is invoked to |
| 94 | tell the architecture specific code that a translation |
Russell King | 4b3073e | 2009-12-18 16:40:18 +0000 | [diff] [blame] | 95 | now exists at virtual address "address" for address space |
| 96 | "vma->vm_mm", in the software page tables. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 97 | |
| 98 | A port may use this information in any way it so chooses. |
| 99 | For example, it could use this event to pre-load TLB |
| 100 | translations for software managed TLB configurations. |
| 101 | The sparc64 port currently does this. |
| 102 | |
Benjamin Herrenschmidt | 1c7037d | 2007-10-18 23:39:14 -0700 | [diff] [blame] | 103 | 6) void tlb_migrate_finish(struct mm_struct *mm) |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 104 | |
| 105 | This interface is called at the end of an explicit |
| 106 | process migration. This interface provides a hook |
| 107 | to allow a platform to update TLB or context-specific |
| 108 | information for the address space. |
| 109 | |
| 110 | The ia64 sn2 platform is one example of a platform |
| 111 | that uses this interface. |
| 112 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 113 | Next, we have the cache flushing interfaces. In general, when Linux |
| 114 | is changing an existing virtual-->physical mapping to a new value, |
| 115 | the sequence will be in one of the following forms: |
| 116 | |
| 117 | 1) flush_cache_mm(mm); |
| 118 | change_all_page_tables_of(mm); |
| 119 | flush_tlb_mm(mm); |
| 120 | |
| 121 | 2) flush_cache_range(vma, start, end); |
| 122 | change_range_of_page_tables(mm, start, end); |
| 123 | flush_tlb_range(vma, start, end); |
| 124 | |
| 125 | 3) flush_cache_page(vma, addr, pfn); |
| 126 | set_pte(pte_pointer, new_pte_val); |
| 127 | flush_tlb_page(vma, addr); |
| 128 | |
| 129 | The cache level flush will always be first, because this allows |
| 130 | us to properly handle systems whose caches are strict and require |
| 131 | a virtual-->physical translation to exist for a virtual address |
| 132 | when that virtual address is flushed from the cache. The HyperSparc |
| 133 | cpu is one such cpu with this attribute. |
| 134 | |
| 135 | The cache flushing routines below need only deal with cache flushing |
| 136 | to the extent that it is necessary for a particular cpu. Mostly, |
| 137 | these routines must be implemented for cpus which have virtually |
| 138 | indexed caches which must be flushed when virtual-->physical |
| 139 | translations are changed or removed. So, for example, the physically |
| 140 | indexed physically tagged caches of IA32 processors have no need to |
| 141 | implement these interfaces since the caches are fully synchronized |
| 142 | and have no dependency on translation information. |
| 143 | |
| 144 | Here are the routines, one by one: |
| 145 | |
| 146 | 1) void flush_cache_mm(struct mm_struct *mm) |
| 147 | |
| 148 | This interface flushes an entire user address space from |
| 149 | the caches. That is, after running, there will be no cache |
| 150 | lines associated with 'mm'. |
| 151 | |
| 152 | This interface is used to handle whole address space |
Ralf Baechle | ec8c0446 | 2006-12-12 17:14:57 +0000 | [diff] [blame] | 153 | page table operations such as what happens during exit and exec. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 154 | |
Ralf Baechle | ec8c0446 | 2006-12-12 17:14:57 +0000 | [diff] [blame] | 155 | 2) void flush_cache_dup_mm(struct mm_struct *mm) |
| 156 | |
| 157 | This interface flushes an entire user address space from |
| 158 | the caches. That is, after running, there will be no cache |
| 159 | lines associated with 'mm'. |
| 160 | |
| 161 | This interface is used to handle whole address space |
| 162 | page table operations such as what happens during fork. |
| 163 | |
| 164 | This option is separate from flush_cache_mm to allow some |
| 165 | optimizations for VIPT caches. |
| 166 | |
| 167 | 3) void flush_cache_range(struct vm_area_struct *vma, |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 168 | unsigned long start, unsigned long end) |
| 169 | |
| 170 | Here we are flushing a specific range of (user) virtual |
| 171 | addresses from the cache. After running, there will be no |
| 172 | entries in the cache for 'vma->vm_mm' for virtual addresses in |
| 173 | the range 'start' to 'end-1'. |
| 174 | |
| 175 | The "vma" is the backing store being used for the region. |
| 176 | Primarily, this is used for munmap() type operations. |
| 177 | |
| 178 | The interface is provided in hopes that the port can find |
| 179 | a suitably efficient method for removing multiple page |
| 180 | sized regions from the cache, instead of having the kernel |
| 181 | call flush_cache_page (see below) for each entry which may be |
| 182 | modified. |
| 183 | |
Ralf Baechle | ec8c0446 | 2006-12-12 17:14:57 +0000 | [diff] [blame] | 184 | 4) void flush_cache_page(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn) |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 185 | |
| 186 | This time we need to remove a PAGE_SIZE sized range |
| 187 | from the cache. The 'vma' is the backing structure used by |
| 188 | Linux to keep track of mmap'd regions for a process, the |
| 189 | address space is available via vma->vm_mm. Also, one may |
| 190 | test (vma->vm_flags & VM_EXEC) to see if this region is |
| 191 | executable (and thus could be in the 'instruction cache' in |
| 192 | "Harvard" type cache layouts). |
| 193 | |
| 194 | The 'pfn' indicates the physical page frame (shift this value |
| 195 | left by PAGE_SHIFT to get the physical address) that 'addr' |
| 196 | translates to. It is this mapping which should be removed from |
| 197 | the cache. |
| 198 | |
| 199 | After running, there will be no entries in the cache for |
| 200 | 'vma->vm_mm' for virtual address 'addr' which translates |
| 201 | to 'pfn'. |
| 202 | |
| 203 | This is used primarily during fault processing. |
| 204 | |
Ralf Baechle | ec8c0446 | 2006-12-12 17:14:57 +0000 | [diff] [blame] | 205 | 5) void flush_cache_kmaps(void) |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 206 | |
| 207 | This routine need only be implemented if the platform utilizes |
| 208 | highmem. It will be called right before all of the kmaps |
| 209 | are invalidated. |
| 210 | |
| 211 | After running, there will be no entries in the cache for |
| 212 | the kernel virtual address range PKMAP_ADDR(0) to |
| 213 | PKMAP_ADDR(LAST_PKMAP). |
| 214 | |
| 215 | This routing should be implemented in asm/highmem.h |
| 216 | |
Ralf Baechle | ec8c0446 | 2006-12-12 17:14:57 +0000 | [diff] [blame] | 217 | 6) void flush_cache_vmap(unsigned long start, unsigned long end) |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 218 | void flush_cache_vunmap(unsigned long start, unsigned long end) |
| 219 | |
| 220 | Here in these two interfaces we are flushing a specific range |
| 221 | of (kernel) virtual addresses from the cache. After running, |
| 222 | there will be no entries in the cache for the kernel address |
| 223 | space for virtual addresses in the range 'start' to 'end-1'. |
| 224 | |
| 225 | The first of these two routines is invoked after map_vm_area() |
| 226 | has installed the page table entries. The second is invoked |
Benjamin Herrenschmidt | c19c03f | 2007-06-04 15:15:35 +1000 | [diff] [blame] | 227 | before unmap_kernel_range() deletes the page table entries. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 228 | |
| 229 | There exists another whole class of cpu cache issues which currently |
| 230 | require a whole different set of interfaces to handle properly. |
| 231 | The biggest problem is that of virtual aliasing in the data cache |
| 232 | of a processor. |
| 233 | |
Francis Galiegue | a33f322 | 2010-04-23 00:08:02 +0200 | [diff] [blame] | 234 | Is your port susceptible to virtual aliasing in its D-cache? |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 235 | Well, if your D-cache is virtually indexed, is larger in size than |
| 236 | PAGE_SIZE, and does not prevent multiple cache lines for the same |
| 237 | physical address from existing at once, you have this problem. |
| 238 | |
| 239 | If your D-cache has this problem, first define asm/shmparam.h SHMLBA |
| 240 | properly, it should essentially be the size of your virtually |
| 241 | addressed D-cache (or if the size is variable, the largest possible |
| 242 | size). This setting will force the SYSv IPC layer to only allow user |
| 243 | processes to mmap shared memory at address which are a multiple of |
| 244 | this value. |
| 245 | |
| 246 | NOTE: This does not fix shared mmaps, check out the sparc64 port for |
| 247 | one way to solve this (in particular SPARC_FLAG_MMAPSHARED). |
| 248 | |
| 249 | Next, you have to solve the D-cache aliasing issue for all |
| 250 | other cases. Please keep in mind that fact that, for a given page |
| 251 | mapped into some user address space, there is always at least one more |
Francis Galiegue | a33f322 | 2010-04-23 00:08:02 +0200 | [diff] [blame] | 252 | mapping, that of the kernel in its linear mapping starting at |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 253 | PAGE_OFFSET. So immediately, once the first user maps a given |
| 254 | physical page into its address space, by implication the D-cache |
| 255 | aliasing problem has the potential to exist since the kernel already |
| 256 | maps this page at its virtual address. |
| 257 | |
| 258 | void copy_user_page(void *to, void *from, unsigned long addr, struct page *page) |
| 259 | void clear_user_page(void *to, unsigned long addr, struct page *page) |
| 260 | |
| 261 | These two routines store data in user anonymous or COW |
| 262 | pages. It allows a port to efficiently avoid D-cache alias |
| 263 | issues between userspace and the kernel. |
| 264 | |
| 265 | For example, a port may temporarily map 'from' and 'to' to |
| 266 | kernel virtual addresses during the copy. The virtual address |
| 267 | for these two pages is chosen in such a way that the kernel |
| 268 | load/store instructions happen to virtual addresses which are |
| 269 | of the same "color" as the user mapping of the page. Sparc64 |
| 270 | for example, uses this technique. |
| 271 | |
| 272 | The 'addr' parameter tells the virtual address where the |
| 273 | user will ultimately have this page mapped, and the 'page' |
| 274 | parameter gives a pointer to the struct page of the target. |
| 275 | |
| 276 | If D-cache aliasing is not an issue, these two routines may |
| 277 | simply call memcpy/memset directly and do nothing more. |
| 278 | |
| 279 | void flush_dcache_page(struct page *page) |
| 280 | |
| 281 | Any time the kernel writes to a page cache page, _OR_ |
| 282 | the kernel is about to read from a page cache page and |
| 283 | user space shared/writable mappings of this page potentially |
| 284 | exist, this routine is called. |
| 285 | |
| 286 | NOTE: This routine need only be called for page cache pages |
| 287 | which can potentially ever be mapped into the address |
| 288 | space of a user process. So for example, VFS layer code |
| 289 | handling vfs symlinks in the page cache need not call |
| 290 | this interface at all. |
| 291 | |
| 292 | The phrase "kernel writes to a page cache page" means, |
| 293 | specifically, that the kernel executes store instructions |
| 294 | that dirty data in that page at the page->virtual mapping |
| 295 | of that page. It is important to flush here to handle |
| 296 | D-cache aliasing, to make sure these kernel stores are |
| 297 | visible to user space mappings of that page. |
| 298 | |
| 299 | The corollary case is just as important, if there are users |
| 300 | which have shared+writable mappings of this file, we must make |
| 301 | sure that kernel reads of these pages will see the most recent |
| 302 | stores done by the user. |
| 303 | |
| 304 | If D-cache aliasing is not an issue, this routine may |
| 305 | simply be defined as a nop on that architecture. |
| 306 | |
| 307 | There is a bit set aside in page->flags (PG_arch_1) as |
| 308 | "architecture private". The kernel guarantees that, |
| 309 | for pagecache pages, it will clear this bit when such |
| 310 | a page first enters the pagecache. |
| 311 | |
| 312 | This allows these interfaces to be implemented much more |
| 313 | efficiently. It allows one to "defer" (perhaps indefinitely) |
| 314 | the actual flush if there are currently no user processes |
| 315 | mapping this page. See sparc64's flush_dcache_page and |
| 316 | update_mmu_cache implementations for an example of how to go |
| 317 | about doing this. |
| 318 | |
| 319 | The idea is, first at flush_dcache_page() time, if |
Kirill A. Shutemov | 27ba064 | 2015-02-10 14:09:59 -0800 | [diff] [blame] | 320 | page->mapping->i_mmap is an empty tree, just mark the architecture |
| 321 | private page flag bit. Later, in update_mmu_cache(), a check is |
| 322 | made of this flag bit, and if set the flush is done and the flag |
| 323 | bit is cleared. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 324 | |
| 325 | IMPORTANT NOTE: It is often important, if you defer the flush, |
| 326 | that the actual flush occurs on the same CPU |
| 327 | as did the cpu stores into the page to make it |
| 328 | dirty. Again, see sparc64 for examples of how |
| 329 | to deal with this. |
| 330 | |
| 331 | void copy_to_user_page(struct vm_area_struct *vma, struct page *page, |
| 332 | unsigned long user_vaddr, |
| 333 | void *dst, void *src, int len) |
| 334 | void copy_from_user_page(struct vm_area_struct *vma, struct page *page, |
| 335 | unsigned long user_vaddr, |
| 336 | void *dst, void *src, int len) |
| 337 | When the kernel needs to copy arbitrary data in and out |
| 338 | of arbitrary user pages (f.e. for ptrace()) it will use |
| 339 | these two routines. |
| 340 | |
| 341 | Any necessary cache flushing or other coherency operations |
| 342 | that need to occur should happen here. If the processor's |
| 343 | instruction cache does not snoop cpu stores, it is very |
| 344 | likely that you will need to flush the instruction cache |
| 345 | for copy_to_user_page(). |
| 346 | |
Russell King | a6f36be | 2006-12-30 22:24:19 +0000 | [diff] [blame] | 347 | void flush_anon_page(struct vm_area_struct *vma, struct page *page, |
| 348 | unsigned long vmaddr) |
James Bottomley | 03beb07 | 2006-03-26 01:36:57 -0800 | [diff] [blame] | 349 | When the kernel needs to access the contents of an anonymous |
| 350 | page, it calls this function (currently only |
| 351 | get_user_pages()). Note: flush_dcache_page() deliberately |
| 352 | doesn't work for an anonymous page. The default |
| 353 | implementation is a nop (and should remain so for all coherent |
| 354 | architectures). For incoherent architectures, it should flush |
Russell King | a6f36be | 2006-12-30 22:24:19 +0000 | [diff] [blame] | 355 | the cache of the page at vmaddr. |
James Bottomley | 03beb07 | 2006-03-26 01:36:57 -0800 | [diff] [blame] | 356 | |
James Bottomley | 5a3a5a9 | 2006-03-26 01:36:59 -0800 | [diff] [blame] | 357 | void flush_kernel_dcache_page(struct page *page) |
| 358 | When the kernel needs to modify a user page is has obtained |
| 359 | with kmap, it calls this function after all modifications are |
| 360 | complete (but before kunmapping it) to bring the underlying |
| 361 | page up to date. It is assumed here that the user has no |
| 362 | incoherent cached copies (i.e. the original page was obtained |
| 363 | from a mechanism like get_user_pages()). The default |
| 364 | implementation is a nop and should remain so on all coherent |
| 365 | architectures. On incoherent architectures, this should flush |
| 366 | the kernel cache for page (using page_address(page)). |
| 367 | |
| 368 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 369 | void flush_icache_range(unsigned long start, unsigned long end) |
| 370 | When the kernel stores into addresses that it will execute |
| 371 | out of (eg when loading modules), this function is called. |
| 372 | |
| 373 | If the icache does not snoop stores then this routine will need |
| 374 | to flush it. |
| 375 | |
| 376 | void flush_icache_page(struct vm_area_struct *vma, struct page *page) |
| 377 | All the functionality of flush_icache_page can be implemented in |
Paul Gortmaker | 548a195 | 2013-08-12 19:55:49 -0400 | [diff] [blame] | 378 | flush_dcache_page and update_mmu_cache. In the future, the hope |
| 379 | is to remove this interface completely. |
James Bottomley | 9df5f741 | 2010-01-25 11:42:20 -0600 | [diff] [blame] | 380 | |
| 381 | The final category of APIs is for I/O to deliberately aliased address |
| 382 | ranges inside the kernel. Such aliases are set up by use of the |
| 383 | vmap/vmalloc API. Since kernel I/O goes via physical pages, the I/O |
| 384 | subsystem assumes that the user mapping and kernel offset mapping are |
| 385 | the only aliases. This isn't true for vmap aliases, so anything in |
| 386 | the kernel trying to do I/O to vmap areas must manually manage |
| 387 | coherency. It must do this by flushing the vmap range before doing |
| 388 | I/O and invalidating it after the I/O returns. |
| 389 | |
| 390 | void flush_kernel_vmap_range(void *vaddr, int size) |
| 391 | flushes the kernel cache for a given virtual address range in |
| 392 | the vmap area. This is to make sure that any data the kernel |
| 393 | modified in the vmap range is made visible to the physical |
| 394 | page. The design is to make this area safe to perform I/O on. |
| 395 | Note that this API does *not* also flush the offset map alias |
| 396 | of the area. |
| 397 | |
| 398 | void invalidate_kernel_vmap_range(void *vaddr, int size) invalidates |
| 399 | the cache for a given virtual address range in the vmap area |
| 400 | which prevents the processor from making the cache stale by |
| 401 | speculatively reading data while the I/O was occurring to the |
| 402 | physical pages. This is only necessary for data reads into the |
| 403 | vmap area. |