)]}'
{
  "commit": "de53fd7aedb100f03e5d2231cfce0e4993282425",
  "tree": "c847e25dc4377d0f206bd48c30fb3b96a572646a",
  "parents": [
    "139d025cda1da5484e7287b35c019fe1dcf9b650"
  ],
  "author": {
    "name": "Dave Chiluk",
    "email": "chiluk+linux@indeed.com",
    "time": "Tue Jul 23 11:44:26 2019 -0500"
  },
  "committer": {
    "name": "Peter Zijlstra",
    "email": "peterz@infradead.org",
    "time": "Thu Aug 08 09:09:30 2019 +0200"
  },
  "message": "sched/fair: Fix low cpu usage with high throttling by removing expiration of cpu-local slices\n\nIt has been observed, that highly-threaded, non-cpu-bound applications\nrunning under cpu.cfs_quota_us constraints can hit a high percentage of\nperiods throttled while simultaneously not consuming the allocated\namount of quota. This use case is typical of user-interactive non-cpu\nbound applications, such as those running in kubernetes or mesos when\nrun on multiple cpu cores.\n\nThis has been root caused to cpu-local run queue being allocated per cpu\nbandwidth slices, and then not fully using that slice within the period.\nAt which point the slice and quota expires. This expiration of unused\nslice results in applications not being able to utilize the quota for\nwhich they are allocated.\n\nThe non-expiration of per-cpu slices was recently fixed by\n\u0027commit 512ac999d275 (\"sched/fair: Fix bandwidth timer clock drift\ncondition\")\u0027. Prior to that it appears that this had been broken since\nat least \u0027commit 51f2176d74ac (\"sched/fair: Fix unlocked reads of some\ncfs_b-\u003equota/period\")\u0027 which was introduced in v3.16-rc1 in 2014. That\nadded the following conditional which resulted in slices never being\nexpired.\n\nif (cfs_rq-\u003eruntime_expires !\u003d cfs_b-\u003eruntime_expires) {\n\t/* extend local deadline, drift is bounded above by 2 ticks */\n\tcfs_rq-\u003eruntime_expires +\u003d TICK_NSEC;\n\nBecause this was broken for nearly 5 years, and has recently been fixed\nand is now being noticed by many users running kubernetes\n(https://github.com/kubernetes/kubernetes/issues/67577) it is my opinion\nthat the mechanisms around expiring runtime should be removed\naltogether.\n\nThis allows quota already allocated to per-cpu run-queues to live longer\nthan the period boundary. This allows threads on runqueues that do not\nuse much CPU to continue to use their remaining slice over a longer\nperiod of time than cpu.cfs_period_us. However, this helps prevent the\nabove condition of hitting throttling while also not fully utilizing\nyour cpu quota.\n\nThis theoretically allows a machine to use slightly more than its\nallotted quota in some periods. This overflow would be bounded by the\nremaining quota left on each per-cpu runqueueu. This is typically no\nmore than min_cfs_rq_runtime\u003d1ms per cpu. For CPU bound tasks this will\nchange nothing, as they should theoretically fully utilize all of their\nquota in each period. For user-interactive tasks as described above this\nprovides a much better user/application experience as their cpu\nutilization will more closely match the amount they requested when they\nhit throttling. This means that cpu limits no longer strictly apply per\nperiod for non-cpu bound applications, but that they are still accurate\nover longer timeframes.\n\nThis greatly improves performance of high-thread-count, non-cpu bound\napplications with low cfs_quota_us allocation on high-core-count\nmachines. In the case of an artificial testcase (10ms/100ms of quota on\n80 CPU machine), this commit resulted in almost 30x performance\nimprovement, while still maintaining correct cpu quota restrictions.\nThat testcase is available at https://github.com/indeedeng/fibtest.\n\nFixes: 512ac999d275 (\"sched/fair: Fix bandwidth timer clock drift condition\")\nSigned-off-by: Dave Chiluk \u003cchiluk+linux@indeed.com\u003e\nSigned-off-by: Peter Zijlstra (Intel) \u003cpeterz@infradead.org\u003e\nReviewed-by: Phil Auld \u003cpauld@redhat.com\u003e\nReviewed-by: Ben Segall \u003cbsegall@google.com\u003e\nCc: Ingo Molnar \u003cmingo@redhat.com\u003e\nCc: John Hammond \u003cjhammond@indeed.com\u003e\nCc: Jonathan Corbet \u003ccorbet@lwn.net\u003e\nCc: Kyle Anderson \u003ckwa@yelp.com\u003e\nCc: Gabriel Munos \u003cgmunoz@netflix.com\u003e\nCc: Peter Oskolkov \u003cposk@posk.io\u003e\nCc: Cong Wang \u003cxiyou.wangcong@gmail.com\u003e\nCc: Brendan Gregg \u003cbgregg@netflix.com\u003e\nLink: https://lkml.kernel.org/r/1563900266-19734-2-git-send-email-chiluk+linux@indeed.com\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "3a9064219656cc97a2bc5881ed67e9fdb2ea15b2",
      "old_mode": 33188,
      "old_path": "Documentation/scheduler/sched-bwc.rst",
      "new_id": "9801d6b284b1ecfdd51b7e1ab5ab5f9bc6da3b40",
      "new_mode": 33188,
      "new_path": "Documentation/scheduler/sched-bwc.rst"
    },
    {
      "type": "modify",
      "old_id": "fb75c0bea80f2552d2e39292b0d8c6fa4c9de2e3",
      "old_mode": 33188,
      "old_path": "kernel/sched/fair.c",
      "new_id": "7d8043fc8317000eb777028e19b0818d8de77c5a",
      "new_mode": 33188,
      "new_path": "kernel/sched/fair.c"
    },
    {
      "type": "modify",
      "old_id": "7583faddba335a438da60617c11650fcb6293a67",
      "old_mode": 33188,
      "old_path": "kernel/sched/sched.h",
      "new_id": "ea48aa5daeee8fb283c03c9c94f8d0d390a32471",
      "new_mode": 33188,
      "new_path": "kernel/sched/sched.h"
    }
  ]
}