| blob | a5eb7d19a65d241650e26a2b4e5a303437b46878 |
1 CFQ (Complete Fairness Queueing)
2 ===============================
4 The main aim of CFQ scheduler is to provide a fair allocation of the disk
5 I/O bandwidth for all the processes which requests an I/O operation.
7 CFQ maintains the per process queue for the processes which request I/O
8 operation(syncronous requests). In case of asynchronous requests, all the
9 requests from all the processes are batched together according to their
10 process's I/O priority.
12 CFQ ioscheduler tunables
13 ========================
15 slice_idle
16 ----------
17 This specifies how long CFQ should idle for next request on certain cfq queues
18 (for sequential workloads) and service trees (for random workloads) before
19 queue is expired and CFQ selects next queue to dispatch from.
21 By default slice_idle is a non-zero value. That means by default we idle on
22 queues/service trees. This can be very helpful on highly seeky media like
23 single spindle SATA/SAS disks where we can cut down on overall number of
24 seeks and see improved throughput.
26 Setting slice_idle to 0 will remove all the idling on queues/service tree
27 level and one should see an overall improved throughput on faster storage
28 devices like multiple SATA/SAS disks in hardware RAID configuration. The down
29 side is that isolation provided from WRITES also goes down and notion of
30 IO priority becomes weaker.
32 So depending on storage and workload, it might be useful to set slice_idle=0.
33 In general I think for SATA/SAS disks and software RAID of SATA/SAS disks
34 keeping slice_idle enabled should be useful. For any configurations where
35 there are multiple spindles behind single LUN (Host based hardware RAID
36 controller or for storage arrays), setting slice_idle=0 might end up in better
37 throughput and acceptable latencies.
39 back_seek_max
40 -------------
41 This specifies, given in Kbytes, the maximum "distance" for backward seeking.
42 The distance is the amount of space from the current head location to the
43 sectors that are backward in terms of distance.
45 This parameter allows the scheduler to anticipate requests in the "backward"
46 direction and consider them as being the "next" if they are within this
47 distance from the current head location.
49 back_seek_penalty
50 -----------------
51 This parameter is used to compute the cost of backward seeking. If the
52 backward distance of request is just 1/back_seek_penalty from a "front"
53 request, then the seeking cost of two requests is considered equivalent.
55 So scheduler will not bias toward one or the other request (otherwise scheduler
56 will bias toward front request). Default value of back_seek_penalty is 2.
58 fifo_expire_async
59 -----------------
60 This parameter is used to set the timeout of asynchronous requests. Default
61 value of this is 248ms.
63 fifo_expire_sync
64 ----------------
65 This parameter is used to set the timeout of synchronous requests. Default
66 value of this is 124ms. In case to favor synchronous requests over asynchronous
67 one, this value should be decreased relative to fifo_expire_async.
69 slice_async
70 -----------
71 This parameter is same as of slice_sync but for asynchronous queue. The
72 default value is 40ms.
74 slice_async_rq
75 --------------
76 This parameter is used to limit the dispatching of asynchronous request to
77 device request queue in queue's slice time. The maximum number of request that
78 are allowed to be dispatched also depends upon the io priority. Default value
79 for this is 2.
81 slice_sync
82 ----------
83 When a queue is selected for execution, the queues IO requests are only
84 executed for a certain amount of time(time_slice) before switching to another
85 queue. This parameter is used to calculate the time slice of synchronous
86 queue.
88 time_slice is computed using the below equation:-
89 time_slice = slice_sync + (slice_sync/5 * (4 - prio)). To increase the
90 time_slice of synchronous queue, increase the value of slice_sync. Default
91 value is 100ms.
93 quantum
94 -------
95 This specifies the number of request dispatched to the device queue. In a
96 queue's time slice, a request will not be dispatched if the number of request
97 in the device exceeds this parameter. This parameter is used for synchronous
98 request.
100 In case of storage with several disk, this setting can limit the parallel
101 processing of request. Therefore, increasing the value can imporve the
102 performace although this can cause the latency of some I/O to increase due
103 to more number of requests.
105 CFQ Group scheduling
106 ====================
108 CFQ supports blkio cgroup and has "blkio." prefixed files in each
109 blkio cgroup directory. It is weight-based and there are four knobs
110 for configuration - weight[_device] and leaf_weight[_device].
111 Internal cgroup nodes (the ones with children) can also have tasks in
112 them, so the former two configure how much proportion the cgroup as a
113 whole is entitled to at its parent's level while the latter two
114 configure how much proportion the tasks in the cgroup have compared to
115 its direct children.
117 Another way to think about it is assuming that each internal node has
118 an implicit leaf child node which hosts all the tasks whose weight is
119 configured by leaf_weight[_device]. Let's assume a blkio hierarchy
120 composed of five cgroups - root, A, B, AA and AB - with the following
121 weights where the names represent the hierarchy.
123 weight leaf_weight
124 root : 125 125
125 A : 500 750
126 B : 250 500
127 AA : 500 500
128 AB : 1000 500
130 root never has a parent making its weight is meaningless. For backward
131 compatibility, weight is always kept in sync with leaf_weight. B, AA
132 and AB have no child and thus its tasks have no children cgroup to
133 compete with. They always get 100% of what the cgroup won at the
134 parent level. Considering only the weights which matter, the hierarchy
135 looks like the following.
137 root
138 / | \
139 A B leaf
140 500 250 125
141 / | \
142 AA AB leaf
143 500 1000 750
145 If all cgroups have active IOs and competing with each other, disk
146 time will be distributed like the following.
148 Distribution below root. The total active weight at this level is
149 A:500 + B:250 + C:125 = 875.
151 root-leaf : 125 / 875 =~ 14%
152 A : 500 / 875 =~ 57%
153 B(-leaf) : 250 / 875 =~ 28%
155 A has children and further distributes its 57% among the children and
156 the implicit leaf node. The total active weight at this level is
157 AA:500 + AB:1000 + A-leaf:750 = 2250.
159 A-leaf : ( 750 / 2250) * A =~ 19%
160 AA(-leaf) : ( 500 / 2250) * A =~ 12%
161 AB(-leaf) : (1000 / 2250) * A =~ 25%
163 CFQ IOPS Mode for group scheduling
164 ===================================
165 Basic CFQ design is to provide priority based time slices. Higher priority
166 process gets bigger time slice and lower priority process gets smaller time
167 slice. Measuring time becomes harder if storage is fast and supports NCQ and
168 it would be better to dispatch multiple requests from multiple cfq queues in
169 request queue at a time. In such scenario, it is not possible to measure time
170 consumed by single queue accurately.
172 What is possible though is to measure number of requests dispatched from a
173 single queue and also allow dispatch from multiple cfq queue at the same time.
174 This effectively becomes the fairness in terms of IOPS (IO operations per
175 second).
177 If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches
178 to IOPS mode and starts providing fairness in terms of number of requests
179 dispatched. Note that this mode switching takes effect only for group
180 scheduling. For non-cgroup users nothing should change.
182 CFQ IO scheduler Idling Theory
183 ===============================
184 Idling on a queue is primarily about waiting for the next request to come
185 on same queue after completion of a request. In this process CFQ will not
186 dispatch requests from other cfq queues even if requests are pending there.
188 The rationale behind idling is that it can cut down on number of seeks
189 on rotational media. For example, if a process is doing dependent
190 sequential reads (next read will come on only after completion of previous
191 one), then not dispatching request from other queue should help as we
192 did not move the disk head and kept on dispatching sequential IO from
193 one queue.
195 CFQ has following service trees and various queues are put on these trees.
197 sync-idle sync-noidle async
199 All cfq queues doing synchronous sequential IO go on to sync-idle tree.
200 On this tree we idle on each queue individually.
202 All synchronous non-sequential queues go on sync-noidle tree. Also any
203 request which are marked with REQ_NOIDLE go on this service tree. On this
204 tree we do not idle on individual queues instead idle on the whole group
205 of queues or the tree. So if there are 4 queues waiting for IO to dispatch
206 we will idle only once last queue has dispatched the IO and there is
207 no more IO on this service tree.
209 All async writes go on async service tree. There is no idling on async
210 queues.
212 CFQ has some optimizations for SSDs and if it detects a non-rotational
213 media which can support higher queue depth (multiple requests at in
214 flight at a time), then it cuts down on idling of individual queues and
215 all the queues move to sync-noidle tree and only tree idle remains. This
216 tree idling provides isolation with buffered write queues on async tree.
218 FAQ
219 ===
220 Q1. Why to idle at all on queues marked with REQ_NOIDLE.
222 A1. We only do tree idle (all queues on sync-noidle tree) on queues marked
223 with REQ_NOIDLE. This helps in providing isolation with all the sync-idle
224 queues. Otherwise in presence of many sequential readers, other
225 synchronous IO might not get fair share of disk.
227 For example, if there are 10 sequential readers doing IO and they get
228 100ms each. If a REQ_NOIDLE request comes in, it will be scheduled
229 roughly after 1 second. If after completion of REQ_NOIDLE request we
230 do not idle, and after a couple of milli seconds a another REQ_NOIDLE
231 request comes in, again it will be scheduled after 1second. Repeat it
232 and notice how a workload can lose its disk share and suffer due to
233 multiple sequential readers.
235 fsync can generate dependent IO where bunch of data is written in the
236 context of fsync, and later some journaling data is written. Journaling
237 data comes in only after fsync has finished its IO (atleast for ext4
238 that seemed to be the case). Now if one decides not to idle on fsync
239 thread due to REQ_NOIDLE, then next journaling write will not get
240 scheduled for another second. A process doing small fsync, will suffer
241 badly in presence of multiple sequential readers.
243 Hence doing tree idling on threads using REQ_NOIDLE flag on requests
244 provides isolation from multiple sequential readers and at the same
245 time we do not idle on individual threads.
247 Q2. When to specify REQ_NOIDLE
248 A2. I would think whenever one is doing synchronous write and not expecting
249 more writes to be dispatched from same context soon, should be able
250 to specify REQ_NOIDLE on writes and that probably should work well for
251 most of the cases.