4 * The contents of this file are subject to the terms of the
5 * Common Development and Distribution License (the "License").
6 * You may not use this file except in compliance with the License.
8 * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
9 * or http://www.opensolaris.org/os/licensing.
10 * See the License for the specific language governing permissions
11 * and limitations under the License.
13 * When distributing Covered Code, include this CDDL HEADER in each
14 * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
15 * If applicable, add the following below this CDDL HEADER, with the
16 * fields enclosed by brackets "[]" replaced with your own identifying
17 * information: Portions Copyright [yyyy] [name of copyright owner]
22 * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
23 * Copyright (c) 2019, Joyent, Inc.
24 * Copyright (c) 2011, 2018 by Delphix. All rights reserved.
25 * Copyright (c) 2014 by Saso Kiselkov. All rights reserved.
26 * Copyright 2017 Nexenta Systems, Inc. All rights reserved.
27 * Copyright (c) 2011, 2019, Delphix. All rights reserved.
28 * Copyright (c) 2020, George Amanakis. All rights reserved.
29 * Copyright (c) 2020, The FreeBSD Foundation [1]
30 * Copyright 2024 Bill Sommerfeld <sommerfeld@hamachi.org>
32 * [1] Portions of this software were developed by Allan Jude
33 * under sponsorship from the FreeBSD Foundation.
37 * DVA-based Adjustable Replacement Cache
39 * While much of the theory of operation used here is
40 * based on the self-tuning, low overhead replacement cache
41 * presented by Megiddo and Modha at FAST 2003, there are some
42 * significant differences:
44 * 1. The Megiddo and Modha model assumes any page is evictable.
45 * Pages in its cache cannot be "locked" into memory. This makes
46 * the eviction algorithm simple: evict the last page in the list.
47 * This also make the performance characteristics easy to reason
48 * about. Our cache is not so simple. At any given moment, some
49 * subset of the blocks in the cache are un-evictable because we
50 * have handed out a reference to them. Blocks are only evictable
51 * when there are no external references active. This makes
52 * eviction far more problematic: we choose to evict the evictable
53 * blocks that are the "lowest" in the list.
55 * There are times when it is not possible to evict the requested
56 * space. In these circumstances we are unable to adjust the cache
57 * size. To prevent the cache growing unbounded at these times we
58 * implement a "cache throttle" that slows the flow of new data
59 * into the cache until we can make space available.
61 * 2. The Megiddo and Modha model assumes a fixed cache size.
62 * Pages are evicted when the cache is full and there is a cache
63 * miss. Our model has a variable sized cache. It grows with
64 * high use, but also tries to react to memory pressure from the
65 * operating system: decreasing its size when system memory is
68 * 3. The Megiddo and Modha model assumes a fixed page size. All
69 * elements of the cache are therefore exactly the same size. So
70 * when adjusting the cache size following a cache miss, its simply
71 * a matter of choosing a single page to evict. In our model, we
72 * have variable sized cache blocks (rangeing from 512 bytes to
73 * 128K bytes). We therefore choose a set of blocks to evict to make
74 * space for a cache miss that approximates as closely as possible
75 * the space used by the new block.
77 * See also: "ARC: A Self-Tuning, Low Overhead Replacement Cache"
78 * by N. Megiddo & D. Modha, FAST 2003
84 * A new reference to a cache buffer can be obtained in two
85 * ways: 1) via a hash table lookup using the DVA as a key,
86 * or 2) via one of the ARC lists. The arc_read() interface
87 * uses method 1, while the internal ARC algorithms for
88 * adjusting the cache use method 2. We therefore provide two
89 * types of locks: 1) the hash table lock array, and 2) the
92 * Buffers do not have their own mutexes, rather they rely on the
93 * hash table mutexes for the bulk of their protection (i.e. most
94 * fields in the arc_buf_hdr_t are protected by these mutexes).
96 * buf_hash_find() returns the appropriate mutex (held) when it
97 * locates the requested buffer in the hash table. It returns
98 * NULL for the mutex if the buffer was not in the table.
100 * buf_hash_remove() expects the appropriate hash mutex to be
101 * already held before it is invoked.
103 * Each ARC state also has a mutex which is used to protect the
104 * buffer list associated with the state. When attempting to
105 * obtain a hash table lock while holding an ARC list lock you
106 * must use: mutex_tryenter() to avoid deadlock. Also note that
107 * the active state mutex must be held before the ghost state mutex.
109 * Note that the majority of the performance stats are manipulated
110 * with atomic operations.
112 * The L2ARC uses the l2ad_mtx on each vdev for the following:
114 * - L2ARC buflist creation
115 * - L2ARC buflist eviction
116 * - L2ARC write completion, which walks L2ARC buflists
117 * - ARC header destruction, as it removes from L2ARC buflists
118 * - ARC header release, as it removes from L2ARC buflists
124 * Every block that is in the ARC is tracked by an arc_buf_hdr_t structure.
125 * This structure can point either to a block that is still in the cache or to
126 * one that is only accessible in an L2 ARC device, or it can provide
127 * information about a block that was recently evicted. If a block is
128 * only accessible in the L2ARC, then the arc_buf_hdr_t only has enough
129 * information to retrieve it from the L2ARC device. This information is
130 * stored in the l2arc_buf_hdr_t sub-structure of the arc_buf_hdr_t. A block
131 * that is in this state cannot access the data directly.
133 * Blocks that are actively being referenced or have not been evicted
134 * are cached in the L1ARC. The L1ARC (l1arc_buf_hdr_t) is a structure within
135 * the arc_buf_hdr_t that will point to the data block in memory. A block can
136 * only be read by a consumer if it has an l1arc_buf_hdr_t. The L1ARC
137 * caches data in two ways -- in a list of ARC buffers (arc_buf_t) and
138 * also in the arc_buf_hdr_t's private physical data block pointer (b_pabd).
140 * The L1ARC's data pointer may or may not be uncompressed. The ARC has the
141 * ability to store the physical data (b_pabd) associated with the DVA of the
142 * arc_buf_hdr_t. Since the b_pabd is a copy of the on-disk physical block,
143 * it will match its on-disk compression characteristics. This behavior can be
144 * disabled by setting 'zfs_compressed_arc_enabled' to B_FALSE. When the
145 * compressed ARC functionality is disabled, the b_pabd will point to an
146 * uncompressed version of the on-disk data.
148 * Data in the L1ARC is not accessed by consumers of the ARC directly. Each
149 * arc_buf_hdr_t can have multiple ARC buffers (arc_buf_t) which reference it.
150 * Each ARC buffer (arc_buf_t) is being actively accessed by a specific ARC
151 * consumer. The ARC will provide references to this data and will keep it
152 * cached until it is no longer in use. The ARC caches only the L1ARC's physical
153 * data block and will evict any arc_buf_t that is no longer referenced. The
154 * amount of memory consumed by the arc_buf_ts' data buffers can be seen via the
155 * "overhead_size" kstat.
157 * Depending on the consumer, an arc_buf_t can be requested in uncompressed or
158 * compressed form. The typical case is that consumers will want uncompressed
159 * data, and when that happens a new data buffer is allocated where the data is
160 * decompressed for them to use. Currently the only consumer who wants
161 * compressed arc_buf_t's is "zfs send", when it streams data exactly as it
162 * exists on disk. When this happens, the arc_buf_t's data buffer is shared
163 * with the arc_buf_hdr_t.
165 * Here is a diagram showing an arc_buf_hdr_t referenced by two arc_buf_t's. The
166 * first one is owned by a compressed send consumer (and therefore references
167 * the same compressed data buffer as the arc_buf_hdr_t) and the second could be
168 * used by any other consumer (and has its own uncompressed copy of the data
183 * | b_buf +------------>+-----------+ arc_buf_t
184 * | b_pabd +-+ |b_next +---->+-----------+
185 * +-----------+ | |-----------| |b_next +-->NULL
186 * | |b_comp = T | +-----------+
187 * | |b_data +-+ |b_comp = F |
188 * | +-----------+ | |b_data +-+
189 * +->+------+ | +-----------+ |
191 * data | |<--------------+ | uncompressed
192 * +------+ compressed, | data
193 * shared +-->+------+
198 * When a consumer reads a block, the ARC must first look to see if the
199 * arc_buf_hdr_t is cached. If the hdr is cached then the ARC allocates a new
200 * arc_buf_t and either copies uncompressed data into a new data buffer from an
201 * existing uncompressed arc_buf_t, decompresses the hdr's b_pabd buffer into a
202 * new data buffer, or shares the hdr's b_pabd buffer, depending on whether the
203 * hdr is compressed and the desired compression characteristics of the
204 * arc_buf_t consumer. If the arc_buf_t ends up sharing data with the
205 * arc_buf_hdr_t and both of them are uncompressed then the arc_buf_t must be
206 * the last buffer in the hdr's b_buf list, however a shared compressed buf can
207 * be anywhere in the hdr's list.
209 * The diagram below shows an example of an uncompressed ARC hdr that is
210 * sharing its data with an arc_buf_t (note that the shared uncompressed buf is
211 * the last element in the buf list):
223 * | | arc_buf_t (shared)
224 * | b_buf +------------>+---------+ arc_buf_t
225 * | | |b_next +---->+---------+
226 * | b_pabd +-+ |---------| |b_next +-->NULL
227 * +-----------+ | | | +---------+
229 * | +---------+ | |b_data +-+
230 * +->+------+ | +---------+ |
232 * uncompressed | | | |
235 * | uncompressed | | |
238 * +---------------------------------+
240 * Writing to the ARC requires that the ARC first discard the hdr's b_pabd
241 * since the physical block is about to be rewritten. The new data contents
242 * will be contained in the arc_buf_t. As the I/O pipeline performs the write,
243 * it may compress the data before writing it to disk. The ARC will be called
244 * with the transformed data and will bcopy the transformed on-disk block into
245 * a newly allocated b_pabd. Writes are always done into buffers which have
246 * either been loaned (and hence are new and don't have other readers) or
247 * buffers which have been released (and hence have their own hdr, if there
248 * were originally other readers of the buf's original hdr). This ensures that
249 * the ARC only needs to update a single buf and its hdr after a write occurs.
251 * When the L2ARC is in use, it will also take advantage of the b_pabd. The
252 * L2ARC will always write the contents of b_pabd to the L2ARC. This means
253 * that when compressed ARC is enabled that the L2ARC blocks are identical
254 * to the on-disk block in the main data pool. This provides a significant
255 * advantage since the ARC can leverage the bp's checksum when reading from the
256 * L2ARC to determine if the contents are valid. However, if the compressed
257 * ARC is disabled, then the L2ARC's block must be transformed to look
258 * like the physical block in the main data pool before comparing the
259 * checksum and determining its validity.
261 * The L1ARC has a slightly different system for storing encrypted data.
262 * Raw (encrypted + possibly compressed) data has a few subtle differences from
263 * data that is just compressed. The biggest difference is that it is not
264 * possible to decrypt encrypted data (or visa versa) if the keys aren't loaded.
265 * The other difference is that encryption cannot be treated as a suggestion.
266 * If a caller would prefer compressed data, but they actually wind up with
267 * uncompressed data the worst thing that could happen is there might be a
268 * performance hit. If the caller requests encrypted data, however, we must be
269 * sure they actually get it or else secret information could be leaked. Raw
270 * data is stored in hdr->b_crypt_hdr.b_rabd. An encrypted header, therefore,
271 * may have both an encrypted version and a decrypted version of its data at
272 * once. When a caller needs a raw arc_buf_t, it is allocated and the data is
273 * copied out of this header. To avoid complications with b_pabd, raw buffers
279 #include <sys/spa_impl.h>
280 #include <sys/zio_compress.h>
281 #include <sys/zio_checksum.h>
282 #include <sys/zfs_context.h>
284 #include <sys/refcount.h>
285 #include <sys/vdev.h>
286 #include <sys/vdev_impl.h>
287 #include <sys/dsl_pool.h>
288 #include <sys/zio_checksum.h>
289 #include <sys/multilist.h>
292 #include <sys/fm/fs/zfs.h>
294 #include <sys/vmsystm.h>
296 #include <sys/fs/swapnode.h>
297 #include <sys/dnlc.h>
299 #include <sys/callb.h>
300 #include <sys/kstat.h>
301 #include <sys/zthr.h>
302 #include <zfs_fletcher.h>
303 #include <sys/arc_impl.h>
304 #include <sys/aggsum.h>
305 #include <sys/cityhash.h>
306 #include <sys/param.h>
309 /* set with ZFS_DEBUG=watch, to enable watchpoints on frozen buffers */
310 boolean_t arc_watch
= B_FALSE
;
315 * This thread's job is to keep enough free memory in the system, by
316 * calling arc_kmem_reap_now() plus arc_shrink(), which improves
317 * arc_available_memory().
319 static zthr_t
*arc_reap_zthr
;
322 * This thread's job is to keep arc_size under arc_c, by calling
323 * arc_adjust(), which improves arc_is_overflowing().
325 static zthr_t
*arc_adjust_zthr
;
327 static kmutex_t arc_adjust_lock
;
328 static kcondvar_t arc_adjust_waiters_cv
;
329 static boolean_t arc_adjust_needed
= B_FALSE
;
331 uint_t arc_reduce_dnlc_percent
= 3;
334 * The number of headers to evict in arc_evict_state_impl() before
335 * dropping the sublist lock and evicting from another sublist. A lower
336 * value means we're more likely to evict the "correct" header (i.e. the
337 * oldest header in the arc state), but comes with higher overhead
338 * (i.e. more invocations of arc_evict_state_impl()).
340 int zfs_arc_evict_batch_limit
= 10;
342 /* number of seconds before growing cache again */
343 int arc_grow_retry
= 60;
346 * Minimum time between calls to arc_kmem_reap_soon(). Note that this will
347 * be converted to ticks, so with the default hz=100, a setting of 15 ms
348 * will actually wait 2 ticks, or 20ms.
350 int arc_kmem_cache_reap_retry_ms
= 1000;
352 /* shift of arc_c for calculating overflow limit in arc_get_data_impl */
353 int zfs_arc_overflow_shift
= 8;
355 /* shift of arc_c for calculating both min and max arc_p */
356 int arc_p_min_shift
= 4;
358 /* log2(fraction of arc to reclaim) */
359 int arc_shrink_shift
= 7;
362 * log2(fraction of ARC which must be free to allow growing).
363 * I.e. If there is less than arc_c >> arc_no_grow_shift free memory,
364 * when reading a new block into the ARC, we will evict an equal-sized block
367 * This must be less than arc_shrink_shift, so that when we shrink the ARC,
368 * we will still not allow it to grow.
370 int arc_no_grow_shift
= 5;
374 * minimum lifespan of a prefetch block in clock ticks
375 * (initialized in arc_init())
377 static int zfs_arc_min_prefetch_ms
= 1;
378 static int zfs_arc_min_prescient_prefetch_ms
= 6;
381 * If this percent of memory is free, don't throttle.
383 int arc_lotsfree_percent
= 10;
385 static boolean_t arc_initialized
;
388 * The arc has filled available memory and has now warmed up.
390 static boolean_t arc_warm
;
393 * log2 fraction of the zio arena to keep free.
395 int arc_zio_arena_free_shift
= 2;
398 * These tunables are for performance analysis.
400 uint64_t zfs_arc_max
;
401 uint64_t zfs_arc_min
;
402 uint64_t zfs_arc_meta_limit
= 0;
403 uint64_t zfs_arc_meta_min
= 0;
404 int zfs_arc_grow_retry
= 0;
405 int zfs_arc_shrink_shift
= 0;
406 int zfs_arc_p_min_shift
= 0;
407 int zfs_arc_average_blocksize
= 8 * 1024; /* 8KB */
410 * ARC dirty data constraints for arc_tempreserve_space() throttle
412 uint_t zfs_arc_dirty_limit_percent
= 50; /* total dirty data limit */
413 uint_t zfs_arc_anon_limit_percent
= 25; /* anon block dirty limit */
414 uint_t zfs_arc_pool_dirty_percent
= 20; /* each pool's anon allowance */
416 boolean_t zfs_compressed_arc_enabled
= B_TRUE
;
419 static arc_state_t ARC_anon
;
420 static arc_state_t ARC_mru
;
421 static arc_state_t ARC_mru_ghost
;
422 static arc_state_t ARC_mfu
;
423 static arc_state_t ARC_mfu_ghost
;
424 static arc_state_t ARC_l2c_only
;
426 arc_stats_t arc_stats
= {
427 { "hits", KSTAT_DATA_UINT64
},
428 { "misses", KSTAT_DATA_UINT64
},
429 { "demand_data_hits", KSTAT_DATA_UINT64
},
430 { "demand_data_misses", KSTAT_DATA_UINT64
},
431 { "demand_metadata_hits", KSTAT_DATA_UINT64
},
432 { "demand_metadata_misses", KSTAT_DATA_UINT64
},
433 { "prefetch_data_hits", KSTAT_DATA_UINT64
},
434 { "prefetch_data_misses", KSTAT_DATA_UINT64
},
435 { "prefetch_metadata_hits", KSTAT_DATA_UINT64
},
436 { "prefetch_metadata_misses", KSTAT_DATA_UINT64
},
437 { "mru_hits", KSTAT_DATA_UINT64
},
438 { "mru_ghost_hits", KSTAT_DATA_UINT64
},
439 { "mfu_hits", KSTAT_DATA_UINT64
},
440 { "mfu_ghost_hits", KSTAT_DATA_UINT64
},
441 { "deleted", KSTAT_DATA_UINT64
},
442 { "mutex_miss", KSTAT_DATA_UINT64
},
443 { "access_skip", KSTAT_DATA_UINT64
},
444 { "evict_skip", KSTAT_DATA_UINT64
},
445 { "evict_not_enough", KSTAT_DATA_UINT64
},
446 { "evict_l2_cached", KSTAT_DATA_UINT64
},
447 { "evict_l2_eligible", KSTAT_DATA_UINT64
},
448 { "evict_l2_eligible_mfu", KSTAT_DATA_UINT64
},
449 { "evict_l2_eligible_mru", KSTAT_DATA_UINT64
},
450 { "evict_l2_ineligible", KSTAT_DATA_UINT64
},
451 { "evict_l2_skip", KSTAT_DATA_UINT64
},
452 { "hash_elements", KSTAT_DATA_UINT64
},
453 { "hash_elements_max", KSTAT_DATA_UINT64
},
454 { "hash_collisions", KSTAT_DATA_UINT64
},
455 { "hash_chains", KSTAT_DATA_UINT64
},
456 { "hash_chain_max", KSTAT_DATA_UINT64
},
457 { "p", KSTAT_DATA_UINT64
},
458 { "c", KSTAT_DATA_UINT64
},
459 { "c_min", KSTAT_DATA_UINT64
},
460 { "c_max", KSTAT_DATA_UINT64
},
461 { "size", KSTAT_DATA_UINT64
},
462 { "compressed_size", KSTAT_DATA_UINT64
},
463 { "uncompressed_size", KSTAT_DATA_UINT64
},
464 { "overhead_size", KSTAT_DATA_UINT64
},
465 { "hdr_size", KSTAT_DATA_UINT64
},
466 { "data_size", KSTAT_DATA_UINT64
},
467 { "metadata_size", KSTAT_DATA_UINT64
},
468 { "other_size", KSTAT_DATA_UINT64
},
469 { "anon_size", KSTAT_DATA_UINT64
},
470 { "anon_evictable_data", KSTAT_DATA_UINT64
},
471 { "anon_evictable_metadata", KSTAT_DATA_UINT64
},
472 { "mru_size", KSTAT_DATA_UINT64
},
473 { "mru_evictable_data", KSTAT_DATA_UINT64
},
474 { "mru_evictable_metadata", KSTAT_DATA_UINT64
},
475 { "mru_ghost_size", KSTAT_DATA_UINT64
},
476 { "mru_ghost_evictable_data", KSTAT_DATA_UINT64
},
477 { "mru_ghost_evictable_metadata", KSTAT_DATA_UINT64
},
478 { "mfu_size", KSTAT_DATA_UINT64
},
479 { "mfu_evictable_data", KSTAT_DATA_UINT64
},
480 { "mfu_evictable_metadata", KSTAT_DATA_UINT64
},
481 { "mfu_ghost_size", KSTAT_DATA_UINT64
},
482 { "mfu_ghost_evictable_data", KSTAT_DATA_UINT64
},
483 { "mfu_ghost_evictable_metadata", KSTAT_DATA_UINT64
},
484 { "l2_hits", KSTAT_DATA_UINT64
},
485 { "l2_misses", KSTAT_DATA_UINT64
},
486 { "l2_prefetch_asize", KSTAT_DATA_UINT64
},
487 { "l2_mru_asize", KSTAT_DATA_UINT64
},
488 { "l2_mfu_asize", KSTAT_DATA_UINT64
},
489 { "l2_bufc_data_asize", KSTAT_DATA_UINT64
},
490 { "l2_bufc_metadata_asize", KSTAT_DATA_UINT64
},
491 { "l2_feeds", KSTAT_DATA_UINT64
},
492 { "l2_rw_clash", KSTAT_DATA_UINT64
},
493 { "l2_read_bytes", KSTAT_DATA_UINT64
},
494 { "l2_write_bytes", KSTAT_DATA_UINT64
},
495 { "l2_writes_sent", KSTAT_DATA_UINT64
},
496 { "l2_writes_done", KSTAT_DATA_UINT64
},
497 { "l2_writes_error", KSTAT_DATA_UINT64
},
498 { "l2_writes_lock_retry", KSTAT_DATA_UINT64
},
499 { "l2_evict_lock_retry", KSTAT_DATA_UINT64
},
500 { "l2_evict_reading", KSTAT_DATA_UINT64
},
501 { "l2_evict_l1cached", KSTAT_DATA_UINT64
},
502 { "l2_free_on_write", KSTAT_DATA_UINT64
},
503 { "l2_abort_lowmem", KSTAT_DATA_UINT64
},
504 { "l2_cksum_bad", KSTAT_DATA_UINT64
},
505 { "l2_io_error", KSTAT_DATA_UINT64
},
506 { "l2_size", KSTAT_DATA_UINT64
},
507 { "l2_asize", KSTAT_DATA_UINT64
},
508 { "l2_hdr_size", KSTAT_DATA_UINT64
},
509 { "l2_log_blk_writes", KSTAT_DATA_UINT64
},
510 { "l2_log_blk_avg_asize", KSTAT_DATA_UINT64
},
511 { "l2_log_blk_asize", KSTAT_DATA_UINT64
},
512 { "l2_log_blk_count", KSTAT_DATA_UINT64
},
513 { "l2_data_to_meta_ratio", KSTAT_DATA_UINT64
},
514 { "l2_rebuild_success", KSTAT_DATA_UINT64
},
515 { "l2_rebuild_unsupported", KSTAT_DATA_UINT64
},
516 { "l2_rebuild_io_errors", KSTAT_DATA_UINT64
},
517 { "l2_rebuild_dh_errors", KSTAT_DATA_UINT64
},
518 { "l2_rebuild_cksum_lb_errors", KSTAT_DATA_UINT64
},
519 { "l2_rebuild_lowmem", KSTAT_DATA_UINT64
},
520 { "l2_rebuild_size", KSTAT_DATA_UINT64
},
521 { "l2_rebuild_asize", KSTAT_DATA_UINT64
},
522 { "l2_rebuild_bufs", KSTAT_DATA_UINT64
},
523 { "l2_rebuild_bufs_precached", KSTAT_DATA_UINT64
},
524 { "l2_rebuild_log_blks", KSTAT_DATA_UINT64
},
525 { "memory_throttle_count", KSTAT_DATA_UINT64
},
526 { "arc_meta_used", KSTAT_DATA_UINT64
},
527 { "arc_meta_limit", KSTAT_DATA_UINT64
},
528 { "arc_meta_max", KSTAT_DATA_UINT64
},
529 { "arc_meta_min", KSTAT_DATA_UINT64
},
530 { "async_upgrade_sync", KSTAT_DATA_UINT64
},
531 { "demand_hit_predictive_prefetch", KSTAT_DATA_UINT64
},
532 { "demand_hit_prescient_prefetch", KSTAT_DATA_UINT64
},
535 #define ARCSTAT_MAX(stat, val) { \
537 while ((val) > (m = arc_stats.stat.value.ui64) && \
538 (m != atomic_cas_64(&arc_stats.stat.value.ui64, m, (val)))) \
542 #define ARCSTAT_MAXSTAT(stat) \
543 ARCSTAT_MAX(stat##_max, arc_stats.stat.value.ui64)
546 * We define a macro to allow ARC hits/misses to be easily broken down by
547 * two separate conditions, giving a total of four different subtypes for
548 * each of hits and misses (so eight statistics total).
550 #define ARCSTAT_CONDSTAT(cond1, stat1, notstat1, cond2, stat2, notstat2, stat) \
553 ARCSTAT_BUMP(arcstat_##stat1##_##stat2##_##stat); \
555 ARCSTAT_BUMP(arcstat_##stat1##_##notstat2##_##stat); \
559 ARCSTAT_BUMP(arcstat_##notstat1##_##stat2##_##stat); \
561 ARCSTAT_BUMP(arcstat_##notstat1##_##notstat2##_##stat);\
566 * This macro allows us to use kstats as floating averages. Each time we
567 * update this kstat, we first factor it and the update value by
568 * ARCSTAT_AVG_FACTOR to shrink the new value's contribution to the overall
569 * average. This macro assumes that integer loads and stores are atomic, but
570 * is not safe for multiple writers updating the kstat in parallel (only the
571 * last writer's update will remain).
573 #define ARCSTAT_F_AVG_FACTOR 3
574 #define ARCSTAT_F_AVG(stat, value) \
576 uint64_t x = ARCSTAT(stat); \
577 x = x - x / ARCSTAT_F_AVG_FACTOR + \
578 (value) / ARCSTAT_F_AVG_FACTOR; \
584 static arc_state_t
*arc_anon
;
585 static arc_state_t
*arc_mru
;
586 static arc_state_t
*arc_mru_ghost
;
587 static arc_state_t
*arc_mfu
;
588 static arc_state_t
*arc_mfu_ghost
;
589 static arc_state_t
*arc_l2c_only
;
592 * There are also some ARC variables that we want to export, but that are
593 * updated so often that having the canonical representation be the statistic
594 * variable causes a performance bottleneck. We want to use aggsum_t's for these
595 * instead, but still be able to export the kstat in the same way as before.
596 * The solution is to always use the aggsum version, except in the kstat update
600 aggsum_t arc_meta_used
;
601 aggsum_t astat_data_size
;
602 aggsum_t astat_metadata_size
;
603 aggsum_t astat_hdr_size
;
604 aggsum_t astat_other_size
;
605 aggsum_t astat_l2_hdr_size
;
607 static int arc_no_grow
; /* Don't try to grow cache size */
608 static hrtime_t arc_growtime
;
609 static uint64_t arc_tempreserve
;
610 static uint64_t arc_loaned_bytes
;
612 #define GHOST_STATE(state) \
613 ((state) == arc_mru_ghost || (state) == arc_mfu_ghost || \
614 (state) == arc_l2c_only)
616 #define HDR_IN_HASH_TABLE(hdr) ((hdr)->b_flags & ARC_FLAG_IN_HASH_TABLE)
617 #define HDR_IO_IN_PROGRESS(hdr) ((hdr)->b_flags & ARC_FLAG_IO_IN_PROGRESS)
618 #define HDR_IO_ERROR(hdr) ((hdr)->b_flags & ARC_FLAG_IO_ERROR)
619 #define HDR_PREFETCH(hdr) ((hdr)->b_flags & ARC_FLAG_PREFETCH)
620 #define HDR_PRESCIENT_PREFETCH(hdr) \
621 ((hdr)->b_flags & ARC_FLAG_PRESCIENT_PREFETCH)
622 #define HDR_COMPRESSION_ENABLED(hdr) \
623 ((hdr)->b_flags & ARC_FLAG_COMPRESSED_ARC)
625 #define HDR_L2CACHE(hdr) ((hdr)->b_flags & ARC_FLAG_L2CACHE)
626 #define HDR_L2_READING(hdr) \
627 (((hdr)->b_flags & ARC_FLAG_IO_IN_PROGRESS) && \
628 ((hdr)->b_flags & ARC_FLAG_HAS_L2HDR))
629 #define HDR_L2_WRITING(hdr) ((hdr)->b_flags & ARC_FLAG_L2_WRITING)
630 #define HDR_L2_EVICTED(hdr) ((hdr)->b_flags & ARC_FLAG_L2_EVICTED)
631 #define HDR_L2_WRITE_HEAD(hdr) ((hdr)->b_flags & ARC_FLAG_L2_WRITE_HEAD)
632 #define HDR_PROTECTED(hdr) ((hdr)->b_flags & ARC_FLAG_PROTECTED)
633 #define HDR_NOAUTH(hdr) ((hdr)->b_flags & ARC_FLAG_NOAUTH)
634 #define HDR_SHARED_DATA(hdr) ((hdr)->b_flags & ARC_FLAG_SHARED_DATA)
636 #define HDR_ISTYPE_METADATA(hdr) \
637 ((hdr)->b_flags & ARC_FLAG_BUFC_METADATA)
638 #define HDR_ISTYPE_DATA(hdr) (!HDR_ISTYPE_METADATA(hdr))
640 #define HDR_HAS_L1HDR(hdr) ((hdr)->b_flags & ARC_FLAG_HAS_L1HDR)
641 #define HDR_HAS_L2HDR(hdr) ((hdr)->b_flags & ARC_FLAG_HAS_L2HDR)
642 #define HDR_HAS_RABD(hdr) \
643 (HDR_HAS_L1HDR(hdr) && HDR_PROTECTED(hdr) && \
644 (hdr)->b_crypt_hdr.b_rabd != NULL)
645 #define HDR_ENCRYPTED(hdr) \
646 (HDR_PROTECTED(hdr) && DMU_OT_IS_ENCRYPTED((hdr)->b_crypt_hdr.b_ot))
647 #define HDR_AUTHENTICATED(hdr) \
648 (HDR_PROTECTED(hdr) && !DMU_OT_IS_ENCRYPTED((hdr)->b_crypt_hdr.b_ot))
650 /* For storing compression mode in b_flags */
651 #define HDR_COMPRESS_OFFSET (highbit64(ARC_FLAG_COMPRESS_0) - 1)
653 #define HDR_GET_COMPRESS(hdr) ((enum zio_compress)BF32_GET((hdr)->b_flags, \
654 HDR_COMPRESS_OFFSET, SPA_COMPRESSBITS))
655 #define HDR_SET_COMPRESS(hdr, cmp) BF32_SET((hdr)->b_flags, \
656 HDR_COMPRESS_OFFSET, SPA_COMPRESSBITS, (cmp));
658 #define ARC_BUF_LAST(buf) ((buf)->b_next == NULL)
659 #define ARC_BUF_SHARED(buf) ((buf)->b_flags & ARC_BUF_FLAG_SHARED)
660 #define ARC_BUF_COMPRESSED(buf) ((buf)->b_flags & ARC_BUF_FLAG_COMPRESSED)
661 #define ARC_BUF_ENCRYPTED(buf) ((buf)->b_flags & ARC_BUF_FLAG_ENCRYPTED)
667 #define HDR_FULL_CRYPT_SIZE ((int64_t)sizeof (arc_buf_hdr_t))
668 #define HDR_FULL_SIZE ((int64_t)offsetof(arc_buf_hdr_t, b_crypt_hdr))
669 #define HDR_L2ONLY_SIZE ((int64_t)offsetof(arc_buf_hdr_t, b_l1hdr))
672 * Hash table routines
675 #define HT_LOCK_PAD 64
680 unsigned char pad
[(HT_LOCK_PAD
- sizeof (kmutex_t
))];
684 #define BUF_LOCKS 256
685 typedef struct buf_hash_table
{
687 arc_buf_hdr_t
**ht_table
;
688 struct ht_lock ht_locks
[BUF_LOCKS
];
691 static buf_hash_table_t buf_hash_table
;
693 #define BUF_HASH_INDEX(spa, dva, birth) \
694 (buf_hash(spa, dva, birth) & buf_hash_table.ht_mask)
695 #define BUF_HASH_LOCK_NTRY(idx) (buf_hash_table.ht_locks[idx & (BUF_LOCKS-1)])
696 #define BUF_HASH_LOCK(idx) (&(BUF_HASH_LOCK_NTRY(idx).ht_lock))
697 #define HDR_LOCK(hdr) \
698 (BUF_HASH_LOCK(BUF_HASH_INDEX(hdr->b_spa, &hdr->b_dva, hdr->b_birth)))
700 uint64_t zfs_crc64_table
[256];
706 #define L2ARC_WRITE_SIZE (8 * 1024 * 1024) /* initial write max */
707 #define L2ARC_HEADROOM 2 /* num of writes */
709 * If we discover during ARC scan any buffers to be compressed, we boost
710 * our headroom for the next scanning cycle by this percentage multiple.
712 #define L2ARC_HEADROOM_BOOST 200
713 #define L2ARC_FEED_SECS 1 /* caching interval secs */
714 #define L2ARC_FEED_MIN_MS 200 /* min caching interval ms */
717 * We can feed L2ARC from two states of ARC buffers, mru and mfu,
718 * and each of the state has two types: data and metadata.
720 #define L2ARC_FEED_TYPES 4
723 #define l2arc_writes_sent ARCSTAT(arcstat_l2_writes_sent)
724 #define l2arc_writes_done ARCSTAT(arcstat_l2_writes_done)
726 /* L2ARC Performance Tunables */
727 uint64_t l2arc_write_max
= L2ARC_WRITE_SIZE
; /* default max write size */
728 uint64_t l2arc_write_boost
= L2ARC_WRITE_SIZE
; /* extra write during warmup */
729 uint64_t l2arc_headroom
= L2ARC_HEADROOM
; /* number of dev writes */
730 uint64_t l2arc_headroom_boost
= L2ARC_HEADROOM_BOOST
;
731 uint64_t l2arc_feed_secs
= L2ARC_FEED_SECS
; /* interval seconds */
732 uint64_t l2arc_feed_min_ms
= L2ARC_FEED_MIN_MS
; /* min interval milliseconds */
733 boolean_t l2arc_noprefetch
= B_TRUE
; /* don't cache prefetch bufs */
734 boolean_t l2arc_feed_again
= B_TRUE
; /* turbo warmup */
735 boolean_t l2arc_norw
= B_TRUE
; /* no reads during writes */
736 int l2arc_meta_percent
= 33; /* limit on headers size */
741 static list_t L2ARC_dev_list
; /* device list */
742 static list_t
*l2arc_dev_list
; /* device list pointer */
743 static kmutex_t l2arc_dev_mtx
; /* device list mutex */
744 static l2arc_dev_t
*l2arc_dev_last
; /* last device used */
745 static list_t L2ARC_free_on_write
; /* free after write buf list */
746 static list_t
*l2arc_free_on_write
; /* free after write list ptr */
747 static kmutex_t l2arc_free_on_write_mtx
; /* mutex for list */
748 static uint64_t l2arc_ndev
; /* number of devices */
750 typedef struct l2arc_read_callback
{
751 arc_buf_hdr_t
*l2rcb_hdr
; /* read header */
752 blkptr_t l2rcb_bp
; /* original blkptr */
753 zbookmark_phys_t l2rcb_zb
; /* original bookmark */
754 int l2rcb_flags
; /* original flags */
755 abd_t
*l2rcb_abd
; /* temporary buffer */
756 } l2arc_read_callback_t
;
758 typedef struct l2arc_data_free
{
759 /* protected by l2arc_free_on_write_mtx */
762 arc_buf_contents_t l2df_type
;
763 list_node_t l2df_list_node
;
766 static kmutex_t l2arc_feed_thr_lock
;
767 static kcondvar_t l2arc_feed_thr_cv
;
768 static uint8_t l2arc_thread_exit
;
770 static kmutex_t l2arc_rebuild_thr_lock
;
771 static kcondvar_t l2arc_rebuild_thr_cv
;
773 enum arc_hdr_alloc_flags
{
774 ARC_HDR_ALLOC_RDATA
= 0x1,
775 ARC_HDR_DO_ADAPT
= 0x2,
779 static abd_t
*arc_get_data_abd(arc_buf_hdr_t
*, uint64_t, void *, boolean_t
);
780 typedef enum arc_fill_flags
{
781 ARC_FILL_LOCKED
= 1 << 0, /* hdr lock is held */
782 ARC_FILL_COMPRESSED
= 1 << 1, /* fill with compressed data */
783 ARC_FILL_ENCRYPTED
= 1 << 2, /* fill with encrypted data */
784 ARC_FILL_NOAUTH
= 1 << 3, /* don't attempt to authenticate */
785 ARC_FILL_IN_PLACE
= 1 << 4 /* fill in place (special case) */
788 static void *arc_get_data_buf(arc_buf_hdr_t
*, uint64_t, void *);
789 static void arc_get_data_impl(arc_buf_hdr_t
*, uint64_t, void *, boolean_t
);
790 static void arc_free_data_abd(arc_buf_hdr_t
*, abd_t
*, uint64_t, void *);
791 static void arc_free_data_buf(arc_buf_hdr_t
*, void *, uint64_t, void *);
792 static void arc_free_data_impl(arc_buf_hdr_t
*hdr
, uint64_t size
, void *tag
);
793 static void arc_hdr_free_pabd(arc_buf_hdr_t
*, boolean_t
);
794 static void arc_hdr_alloc_pabd(arc_buf_hdr_t
*, int);
795 static void arc_access(arc_buf_hdr_t
*, kmutex_t
*);
796 static boolean_t
arc_is_overflowing();
797 static void arc_buf_watch(arc_buf_t
*);
798 static l2arc_dev_t
*l2arc_vdev_get(vdev_t
*vd
);
800 static arc_buf_contents_t
arc_buf_type(arc_buf_hdr_t
*);
801 static uint32_t arc_bufc_to_flags(arc_buf_contents_t
);
802 static inline void arc_hdr_set_flags(arc_buf_hdr_t
*hdr
, arc_flags_t flags
);
803 static inline void arc_hdr_clear_flags(arc_buf_hdr_t
*hdr
, arc_flags_t flags
);
805 static boolean_t
l2arc_write_eligible(uint64_t, arc_buf_hdr_t
*);
806 static void l2arc_read_done(zio_t
*);
807 static void l2arc_do_free_on_write(void);
808 static void l2arc_hdr_arcstats_update(arc_buf_hdr_t
*hdr
, boolean_t incr
,
809 boolean_t state_only
);
811 #define l2arc_hdr_arcstats_increment(hdr) \
812 l2arc_hdr_arcstats_update((hdr), B_TRUE, B_FALSE)
813 #define l2arc_hdr_arcstats_decrement(hdr) \
814 l2arc_hdr_arcstats_update((hdr), B_FALSE, B_FALSE)
815 #define l2arc_hdr_arcstats_increment_state(hdr) \
816 l2arc_hdr_arcstats_update((hdr), B_TRUE, B_TRUE)
817 #define l2arc_hdr_arcstats_decrement_state(hdr) \
818 l2arc_hdr_arcstats_update((hdr), B_FALSE, B_TRUE)
821 * The arc_all_memory function is a ZoL enhancement that lives in their OSL
822 * code. In user-space code, which is used primarily for testing, we return
823 * half of all memory.
829 return (ptob(physmem
));
831 return ((sysconf(_SC_PAGESIZE
) * sysconf(_SC_PHYS_PAGES
)) / 2);
836 * We use Cityhash for this. It's fast, and has good hash properties without
837 * requiring any large static buffers.
840 buf_hash(uint64_t spa
, const dva_t
*dva
, uint64_t birth
)
842 return (cityhash4(spa
, dva
->dva_word
[0], dva
->dva_word
[1], birth
));
845 #define HDR_EMPTY(hdr) \
846 ((hdr)->b_dva.dva_word[0] == 0 && \
847 (hdr)->b_dva.dva_word[1] == 0)
849 #define HDR_EMPTY_OR_LOCKED(hdr) \
850 (HDR_EMPTY(hdr) || MUTEX_HELD(HDR_LOCK(hdr)))
852 #define HDR_EQUAL(spa, dva, birth, hdr) \
853 ((hdr)->b_dva.dva_word[0] == (dva)->dva_word[0]) && \
854 ((hdr)->b_dva.dva_word[1] == (dva)->dva_word[1]) && \
855 ((hdr)->b_birth == birth) && ((hdr)->b_spa == spa)
858 buf_discard_identity(arc_buf_hdr_t
*hdr
)
860 hdr
->b_dva
.dva_word
[0] = 0;
861 hdr
->b_dva
.dva_word
[1] = 0;
865 static arc_buf_hdr_t
*
866 buf_hash_find(uint64_t spa
, const blkptr_t
*bp
, kmutex_t
**lockp
)
868 const dva_t
*dva
= BP_IDENTITY(bp
);
869 uint64_t birth
= BP_PHYSICAL_BIRTH(bp
);
870 uint64_t idx
= BUF_HASH_INDEX(spa
, dva
, birth
);
871 kmutex_t
*hash_lock
= BUF_HASH_LOCK(idx
);
874 mutex_enter(hash_lock
);
875 for (hdr
= buf_hash_table
.ht_table
[idx
]; hdr
!= NULL
;
876 hdr
= hdr
->b_hash_next
) {
877 if (HDR_EQUAL(spa
, dva
, birth
, hdr
)) {
882 mutex_exit(hash_lock
);
888 * Insert an entry into the hash table. If there is already an element
889 * equal to elem in the hash table, then the already existing element
890 * will be returned and the new element will not be inserted.
891 * Otherwise returns NULL.
892 * If lockp == NULL, the caller is assumed to already hold the hash lock.
894 static arc_buf_hdr_t
*
895 buf_hash_insert(arc_buf_hdr_t
*hdr
, kmutex_t
**lockp
)
897 uint64_t idx
= BUF_HASH_INDEX(hdr
->b_spa
, &hdr
->b_dva
, hdr
->b_birth
);
898 kmutex_t
*hash_lock
= BUF_HASH_LOCK(idx
);
902 ASSERT(!DVA_IS_EMPTY(&hdr
->b_dva
));
903 ASSERT(hdr
->b_birth
!= 0);
904 ASSERT(!HDR_IN_HASH_TABLE(hdr
));
908 mutex_enter(hash_lock
);
910 ASSERT(MUTEX_HELD(hash_lock
));
913 for (fhdr
= buf_hash_table
.ht_table
[idx
], i
= 0; fhdr
!= NULL
;
914 fhdr
= fhdr
->b_hash_next
, i
++) {
915 if (HDR_EQUAL(hdr
->b_spa
, &hdr
->b_dva
, hdr
->b_birth
, fhdr
))
919 hdr
->b_hash_next
= buf_hash_table
.ht_table
[idx
];
920 buf_hash_table
.ht_table
[idx
] = hdr
;
921 arc_hdr_set_flags(hdr
, ARC_FLAG_IN_HASH_TABLE
);
923 /* collect some hash table performance data */
925 ARCSTAT_BUMP(arcstat_hash_collisions
);
927 ARCSTAT_BUMP(arcstat_hash_chains
);
929 ARCSTAT_MAX(arcstat_hash_chain_max
, i
);
932 ARCSTAT_BUMP(arcstat_hash_elements
);
933 ARCSTAT_MAXSTAT(arcstat_hash_elements
);
939 buf_hash_remove(arc_buf_hdr_t
*hdr
)
941 arc_buf_hdr_t
*fhdr
, **hdrp
;
942 uint64_t idx
= BUF_HASH_INDEX(hdr
->b_spa
, &hdr
->b_dva
, hdr
->b_birth
);
944 ASSERT(MUTEX_HELD(BUF_HASH_LOCK(idx
)));
945 ASSERT(HDR_IN_HASH_TABLE(hdr
));
947 hdrp
= &buf_hash_table
.ht_table
[idx
];
948 while ((fhdr
= *hdrp
) != hdr
) {
949 ASSERT3P(fhdr
, !=, NULL
);
950 hdrp
= &fhdr
->b_hash_next
;
952 *hdrp
= hdr
->b_hash_next
;
953 hdr
->b_hash_next
= NULL
;
954 arc_hdr_clear_flags(hdr
, ARC_FLAG_IN_HASH_TABLE
);
956 /* collect some hash table performance data */
957 ARCSTAT_BUMPDOWN(arcstat_hash_elements
);
959 if (buf_hash_table
.ht_table
[idx
] &&
960 buf_hash_table
.ht_table
[idx
]->b_hash_next
== NULL
)
961 ARCSTAT_BUMPDOWN(arcstat_hash_chains
);
965 * l2arc_mfuonly : A ZFS module parameter that controls whether only MFU
966 * metadata and data are cached from ARC into L2ARC.
968 int l2arc_mfuonly
= 0;
971 * Global data structures and functions for the buf kmem cache.
974 static kmem_cache_t
*hdr_full_cache
;
975 static kmem_cache_t
*hdr_full_crypt_cache
;
976 static kmem_cache_t
*hdr_l2only_cache
;
977 static kmem_cache_t
*buf_cache
;
984 kmem_free(buf_hash_table
.ht_table
,
985 (buf_hash_table
.ht_mask
+ 1) * sizeof (void *));
986 for (i
= 0; i
< BUF_LOCKS
; i
++)
987 mutex_destroy(&buf_hash_table
.ht_locks
[i
].ht_lock
);
988 kmem_cache_destroy(hdr_full_cache
);
989 kmem_cache_destroy(hdr_full_crypt_cache
);
990 kmem_cache_destroy(hdr_l2only_cache
);
991 kmem_cache_destroy(buf_cache
);
995 * Constructor callback - called when the cache is empty
996 * and a new buf is requested.
1000 hdr_full_cons(void *vbuf
, void *unused
, int kmflag
)
1002 arc_buf_hdr_t
*hdr
= vbuf
;
1004 bzero(hdr
, HDR_FULL_SIZE
);
1005 hdr
->b_l1hdr
.b_byteswap
= DMU_BSWAP_NUMFUNCS
;
1006 cv_init(&hdr
->b_l1hdr
.b_cv
, NULL
, CV_DEFAULT
, NULL
);
1007 zfs_refcount_create(&hdr
->b_l1hdr
.b_refcnt
);
1008 mutex_init(&hdr
->b_l1hdr
.b_freeze_lock
, NULL
, MUTEX_DEFAULT
, NULL
);
1009 multilist_link_init(&hdr
->b_l1hdr
.b_arc_node
);
1010 arc_space_consume(HDR_FULL_SIZE
, ARC_SPACE_HDRS
);
1017 hdr_full_crypt_cons(void *vbuf
, void *unused
, int kmflag
)
1019 arc_buf_hdr_t
*hdr
= vbuf
;
1021 (void) hdr_full_cons(vbuf
, unused
, kmflag
);
1022 bzero(&hdr
->b_crypt_hdr
, sizeof (hdr
->b_crypt_hdr
));
1023 arc_space_consume(sizeof (hdr
->b_crypt_hdr
), ARC_SPACE_HDRS
);
1030 hdr_l2only_cons(void *vbuf
, void *unused
, int kmflag
)
1032 arc_buf_hdr_t
*hdr
= vbuf
;
1034 bzero(hdr
, HDR_L2ONLY_SIZE
);
1035 arc_space_consume(HDR_L2ONLY_SIZE
, ARC_SPACE_L2HDRS
);
1042 buf_cons(void *vbuf
, void *unused
, int kmflag
)
1044 arc_buf_t
*buf
= vbuf
;
1046 bzero(buf
, sizeof (arc_buf_t
));
1047 mutex_init(&buf
->b_evict_lock
, NULL
, MUTEX_DEFAULT
, NULL
);
1048 arc_space_consume(sizeof (arc_buf_t
), ARC_SPACE_HDRS
);
1054 * Destructor callback - called when a cached buf is
1055 * no longer required.
1059 hdr_full_dest(void *vbuf
, void *unused
)
1061 arc_buf_hdr_t
*hdr
= vbuf
;
1063 ASSERT(HDR_EMPTY(hdr
));
1064 cv_destroy(&hdr
->b_l1hdr
.b_cv
);
1065 zfs_refcount_destroy(&hdr
->b_l1hdr
.b_refcnt
);
1066 mutex_destroy(&hdr
->b_l1hdr
.b_freeze_lock
);
1067 ASSERT(!multilist_link_active(&hdr
->b_l1hdr
.b_arc_node
));
1068 arc_space_return(HDR_FULL_SIZE
, ARC_SPACE_HDRS
);
1073 hdr_full_crypt_dest(void *vbuf
, void *unused
)
1075 arc_buf_hdr_t
*hdr
= vbuf
;
1077 hdr_full_dest(hdr
, unused
);
1078 arc_space_return(sizeof (hdr
->b_crypt_hdr
), ARC_SPACE_HDRS
);
1083 hdr_l2only_dest(void *vbuf
, void *unused
)
1085 arc_buf_hdr_t
*hdr
= vbuf
;
1087 ASSERT(HDR_EMPTY(hdr
));
1088 arc_space_return(HDR_L2ONLY_SIZE
, ARC_SPACE_L2HDRS
);
1093 buf_dest(void *vbuf
, void *unused
)
1095 arc_buf_t
*buf
= vbuf
;
1097 mutex_destroy(&buf
->b_evict_lock
);
1098 arc_space_return(sizeof (arc_buf_t
), ARC_SPACE_HDRS
);
1102 * Reclaim callback -- invoked when memory is low.
1106 hdr_recl(void *unused
)
1108 dprintf("hdr_recl called\n");
1110 * umem calls the reclaim func when we destroy the buf cache,
1111 * which is after we do arc_fini().
1113 if (arc_initialized
)
1114 zthr_wakeup(arc_reap_zthr
);
1121 uint64_t hsize
= 1ULL << 12;
1125 * The hash table is big enough to fill all of physical memory
1126 * with an average block size of zfs_arc_average_blocksize (default 8K).
1127 * By default, the table will take up
1128 * totalmem * sizeof(void*) / 8K (1MB per GB with 8-byte pointers).
1130 while (hsize
* zfs_arc_average_blocksize
< physmem
* PAGESIZE
)
1133 buf_hash_table
.ht_mask
= hsize
- 1;
1134 buf_hash_table
.ht_table
=
1135 kmem_zalloc(hsize
* sizeof (void*), KM_NOSLEEP
);
1136 if (buf_hash_table
.ht_table
== NULL
) {
1137 ASSERT(hsize
> (1ULL << 8));
1142 hdr_full_cache
= kmem_cache_create("arc_buf_hdr_t_full", HDR_FULL_SIZE
,
1143 0, hdr_full_cons
, hdr_full_dest
, hdr_recl
, NULL
, NULL
, 0);
1144 hdr_full_crypt_cache
= kmem_cache_create("arc_buf_hdr_t_full_crypt",
1145 HDR_FULL_CRYPT_SIZE
, 0, hdr_full_crypt_cons
, hdr_full_crypt_dest
,
1146 hdr_recl
, NULL
, NULL
, 0);
1147 hdr_l2only_cache
= kmem_cache_create("arc_buf_hdr_t_l2only",
1148 HDR_L2ONLY_SIZE
, 0, hdr_l2only_cons
, hdr_l2only_dest
, hdr_recl
,
1150 buf_cache
= kmem_cache_create("arc_buf_t", sizeof (arc_buf_t
),
1151 0, buf_cons
, buf_dest
, NULL
, NULL
, NULL
, 0);
1153 for (i
= 0; i
< 256; i
++)
1154 for (ct
= zfs_crc64_table
+ i
, *ct
= i
, j
= 8; j
> 0; j
--)
1155 *ct
= (*ct
>> 1) ^ (-(*ct
& 1) & ZFS_CRC64_POLY
);
1157 for (i
= 0; i
< BUF_LOCKS
; i
++) {
1158 mutex_init(&buf_hash_table
.ht_locks
[i
].ht_lock
,
1159 NULL
, MUTEX_DEFAULT
, NULL
);
1164 * This is the size that the buf occupies in memory. If the buf is compressed,
1165 * it will correspond to the compressed size. You should use this method of
1166 * getting the buf size unless you explicitly need the logical size.
1169 arc_buf_size(arc_buf_t
*buf
)
1171 return (ARC_BUF_COMPRESSED(buf
) ?
1172 HDR_GET_PSIZE(buf
->b_hdr
) : HDR_GET_LSIZE(buf
->b_hdr
));
1176 arc_buf_lsize(arc_buf_t
*buf
)
1178 return (HDR_GET_LSIZE(buf
->b_hdr
));
1182 * This function will return B_TRUE if the buffer is encrypted in memory.
1183 * This buffer can be decrypted by calling arc_untransform().
1186 arc_is_encrypted(arc_buf_t
*buf
)
1188 return (ARC_BUF_ENCRYPTED(buf
) != 0);
1192 * Returns B_TRUE if the buffer represents data that has not had its MAC
1196 arc_is_unauthenticated(arc_buf_t
*buf
)
1198 return (HDR_NOAUTH(buf
->b_hdr
) != 0);
1202 arc_get_raw_params(arc_buf_t
*buf
, boolean_t
*byteorder
, uint8_t *salt
,
1203 uint8_t *iv
, uint8_t *mac
)
1205 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
1207 ASSERT(HDR_PROTECTED(hdr
));
1209 bcopy(hdr
->b_crypt_hdr
.b_salt
, salt
, ZIO_DATA_SALT_LEN
);
1210 bcopy(hdr
->b_crypt_hdr
.b_iv
, iv
, ZIO_DATA_IV_LEN
);
1211 bcopy(hdr
->b_crypt_hdr
.b_mac
, mac
, ZIO_DATA_MAC_LEN
);
1212 *byteorder
= (hdr
->b_l1hdr
.b_byteswap
== DMU_BSWAP_NUMFUNCS
) ?
1214 ZFS_HOST_BYTEORDER
: !ZFS_HOST_BYTEORDER
;
1218 * Indicates how this buffer is compressed in memory. If it is not compressed
1219 * the value will be ZIO_COMPRESS_OFF. It can be made normally readable with
1220 * arc_untransform() as long as it is also unencrypted.
1223 arc_get_compression(arc_buf_t
*buf
)
1225 return (ARC_BUF_COMPRESSED(buf
) ?
1226 HDR_GET_COMPRESS(buf
->b_hdr
) : ZIO_COMPRESS_OFF
);
1229 #define ARC_MINTIME (hz>>4) /* 62 ms */
1232 * Return the compression algorithm used to store this data in the ARC. If ARC
1233 * compression is enabled or this is an encrypted block, this will be the same
1234 * as what's used to store it on-disk. Otherwise, this will be ZIO_COMPRESS_OFF.
1236 static inline enum zio_compress
1237 arc_hdr_get_compress(arc_buf_hdr_t
*hdr
)
1239 return (HDR_COMPRESSION_ENABLED(hdr
) ?
1240 HDR_GET_COMPRESS(hdr
) : ZIO_COMPRESS_OFF
);
1243 static inline boolean_t
1244 arc_buf_is_shared(arc_buf_t
*buf
)
1246 boolean_t shared
= (buf
->b_data
!= NULL
&&
1247 buf
->b_hdr
->b_l1hdr
.b_pabd
!= NULL
&&
1248 abd_is_linear(buf
->b_hdr
->b_l1hdr
.b_pabd
) &&
1249 buf
->b_data
== abd_to_buf(buf
->b_hdr
->b_l1hdr
.b_pabd
));
1250 IMPLY(shared
, HDR_SHARED_DATA(buf
->b_hdr
));
1251 IMPLY(shared
, ARC_BUF_SHARED(buf
));
1252 IMPLY(shared
, ARC_BUF_COMPRESSED(buf
) || ARC_BUF_LAST(buf
));
1255 * It would be nice to assert arc_can_share() too, but the "hdr isn't
1256 * already being shared" requirement prevents us from doing that.
1263 * Free the checksum associated with this header. If there is no checksum, this
1267 arc_cksum_free(arc_buf_hdr_t
*hdr
)
1269 ASSERT(HDR_HAS_L1HDR(hdr
));
1271 mutex_enter(&hdr
->b_l1hdr
.b_freeze_lock
);
1272 if (hdr
->b_l1hdr
.b_freeze_cksum
!= NULL
) {
1273 kmem_free(hdr
->b_l1hdr
.b_freeze_cksum
, sizeof (zio_cksum_t
));
1274 hdr
->b_l1hdr
.b_freeze_cksum
= NULL
;
1276 mutex_exit(&hdr
->b_l1hdr
.b_freeze_lock
);
1280 * Return true iff at least one of the bufs on hdr is not compressed.
1281 * Encrypted buffers count as compressed.
1284 arc_hdr_has_uncompressed_buf(arc_buf_hdr_t
*hdr
)
1286 ASSERT(hdr
->b_l1hdr
.b_state
== arc_anon
|| HDR_EMPTY_OR_LOCKED(hdr
));
1288 for (arc_buf_t
*b
= hdr
->b_l1hdr
.b_buf
; b
!= NULL
; b
= b
->b_next
) {
1289 if (!ARC_BUF_COMPRESSED(b
)) {
1297 * If we've turned on the ZFS_DEBUG_MODIFY flag, verify that the buf's data
1298 * matches the checksum that is stored in the hdr. If there is no checksum,
1299 * or if the buf is compressed, this is a no-op.
1302 arc_cksum_verify(arc_buf_t
*buf
)
1304 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
1307 if (!(zfs_flags
& ZFS_DEBUG_MODIFY
))
1310 if (ARC_BUF_COMPRESSED(buf
))
1313 ASSERT(HDR_HAS_L1HDR(hdr
));
1315 mutex_enter(&hdr
->b_l1hdr
.b_freeze_lock
);
1317 if (hdr
->b_l1hdr
.b_freeze_cksum
== NULL
|| HDR_IO_ERROR(hdr
)) {
1318 mutex_exit(&hdr
->b_l1hdr
.b_freeze_lock
);
1322 fletcher_2_native(buf
->b_data
, arc_buf_size(buf
), NULL
, &zc
);
1323 if (!ZIO_CHECKSUM_EQUAL(*hdr
->b_l1hdr
.b_freeze_cksum
, zc
))
1324 panic("buffer modified while frozen!");
1325 mutex_exit(&hdr
->b_l1hdr
.b_freeze_lock
);
1329 * This function makes the assumption that data stored in the L2ARC
1330 * will be transformed exactly as it is in the main pool. Because of
1331 * this we can verify the checksum against the reading process's bp.
1334 arc_cksum_is_equal(arc_buf_hdr_t
*hdr
, zio_t
*zio
)
1336 ASSERT(!BP_IS_EMBEDDED(zio
->io_bp
));
1337 VERIFY3U(BP_GET_PSIZE(zio
->io_bp
), ==, HDR_GET_PSIZE(hdr
));
1340 * Block pointers always store the checksum for the logical data.
1341 * If the block pointer has the gang bit set, then the checksum
1342 * it represents is for the reconstituted data and not for an
1343 * individual gang member. The zio pipeline, however, must be able to
1344 * determine the checksum of each of the gang constituents so it
1345 * treats the checksum comparison differently than what we need
1346 * for l2arc blocks. This prevents us from using the
1347 * zio_checksum_error() interface directly. Instead we must call the
1348 * zio_checksum_error_impl() so that we can ensure the checksum is
1349 * generated using the correct checksum algorithm and accounts for the
1350 * logical I/O size and not just a gang fragment.
1352 return (zio_checksum_error_impl(zio
->io_spa
, zio
->io_bp
,
1353 BP_GET_CHECKSUM(zio
->io_bp
), zio
->io_abd
, zio
->io_size
,
1354 zio
->io_offset
, NULL
) == 0);
1358 * Given a buf full of data, if ZFS_DEBUG_MODIFY is enabled this computes a
1359 * checksum and attaches it to the buf's hdr so that we can ensure that the buf
1360 * isn't modified later on. If buf is compressed or there is already a checksum
1361 * on the hdr, this is a no-op (we only checksum uncompressed bufs).
1364 arc_cksum_compute(arc_buf_t
*buf
)
1366 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
1368 if (!(zfs_flags
& ZFS_DEBUG_MODIFY
))
1371 ASSERT(HDR_HAS_L1HDR(hdr
));
1373 mutex_enter(&buf
->b_hdr
->b_l1hdr
.b_freeze_lock
);
1374 if (hdr
->b_l1hdr
.b_freeze_cksum
!= NULL
|| ARC_BUF_COMPRESSED(buf
)) {
1375 mutex_exit(&hdr
->b_l1hdr
.b_freeze_lock
);
1379 ASSERT(!ARC_BUF_ENCRYPTED(buf
));
1380 ASSERT(!ARC_BUF_COMPRESSED(buf
));
1381 hdr
->b_l1hdr
.b_freeze_cksum
= kmem_alloc(sizeof (zio_cksum_t
),
1383 fletcher_2_native(buf
->b_data
, arc_buf_size(buf
), NULL
,
1384 hdr
->b_l1hdr
.b_freeze_cksum
);
1385 mutex_exit(&hdr
->b_l1hdr
.b_freeze_lock
);
1390 typedef struct procctl
{
1398 arc_buf_unwatch(arc_buf_t
*buf
)
1405 ctl
.prwatch
.pr_vaddr
= (uintptr_t)buf
->b_data
;
1406 ctl
.prwatch
.pr_size
= 0;
1407 ctl
.prwatch
.pr_wflags
= 0;
1408 result
= write(arc_procfd
, &ctl
, sizeof (ctl
));
1409 ASSERT3U(result
, ==, sizeof (ctl
));
1416 arc_buf_watch(arc_buf_t
*buf
)
1423 ctl
.prwatch
.pr_vaddr
= (uintptr_t)buf
->b_data
;
1424 ctl
.prwatch
.pr_size
= arc_buf_size(buf
);
1425 ctl
.prwatch
.pr_wflags
= WA_WRITE
;
1426 result
= write(arc_procfd
, &ctl
, sizeof (ctl
));
1427 ASSERT3U(result
, ==, sizeof (ctl
));
1432 static arc_buf_contents_t
1433 arc_buf_type(arc_buf_hdr_t
*hdr
)
1435 arc_buf_contents_t type
;
1436 if (HDR_ISTYPE_METADATA(hdr
)) {
1437 type
= ARC_BUFC_METADATA
;
1439 type
= ARC_BUFC_DATA
;
1441 VERIFY3U(hdr
->b_type
, ==, type
);
1446 arc_is_metadata(arc_buf_t
*buf
)
1448 return (HDR_ISTYPE_METADATA(buf
->b_hdr
) != 0);
1452 arc_bufc_to_flags(arc_buf_contents_t type
)
1456 /* metadata field is 0 if buffer contains normal data */
1458 case ARC_BUFC_METADATA
:
1459 return (ARC_FLAG_BUFC_METADATA
);
1463 panic("undefined ARC buffer type!");
1464 return ((uint32_t)-1);
1468 arc_buf_thaw(arc_buf_t
*buf
)
1470 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
1472 ASSERT3P(hdr
->b_l1hdr
.b_state
, ==, arc_anon
);
1473 ASSERT(!HDR_IO_IN_PROGRESS(hdr
));
1475 arc_cksum_verify(buf
);
1478 * Compressed buffers do not manipulate the b_freeze_cksum.
1480 if (ARC_BUF_COMPRESSED(buf
))
1483 ASSERT(HDR_HAS_L1HDR(hdr
));
1484 arc_cksum_free(hdr
);
1486 mutex_enter(&hdr
->b_l1hdr
.b_freeze_lock
);
1488 if (zfs_flags
& ZFS_DEBUG_MODIFY
) {
1489 if (hdr
->b_l1hdr
.b_thawed
!= NULL
)
1490 kmem_free(hdr
->b_l1hdr
.b_thawed
, 1);
1491 hdr
->b_l1hdr
.b_thawed
= kmem_alloc(1, KM_SLEEP
);
1495 mutex_exit(&hdr
->b_l1hdr
.b_freeze_lock
);
1497 arc_buf_unwatch(buf
);
1501 arc_buf_freeze(arc_buf_t
*buf
)
1503 if (!(zfs_flags
& ZFS_DEBUG_MODIFY
))
1506 if (ARC_BUF_COMPRESSED(buf
))
1509 ASSERT(HDR_HAS_L1HDR(buf
->b_hdr
));
1510 arc_cksum_compute(buf
);
1514 * The arc_buf_hdr_t's b_flags should never be modified directly. Instead,
1515 * the following functions should be used to ensure that the flags are
1516 * updated in a thread-safe way. When manipulating the flags either
1517 * the hash_lock must be held or the hdr must be undiscoverable. This
1518 * ensures that we're not racing with any other threads when updating
1522 arc_hdr_set_flags(arc_buf_hdr_t
*hdr
, arc_flags_t flags
)
1524 ASSERT(HDR_EMPTY_OR_LOCKED(hdr
));
1525 hdr
->b_flags
|= flags
;
1529 arc_hdr_clear_flags(arc_buf_hdr_t
*hdr
, arc_flags_t flags
)
1531 ASSERT(HDR_EMPTY_OR_LOCKED(hdr
));
1532 hdr
->b_flags
&= ~flags
;
1536 * Setting the compression bits in the arc_buf_hdr_t's b_flags is
1537 * done in a special way since we have to clear and set bits
1538 * at the same time. Consumers that wish to set the compression bits
1539 * must use this function to ensure that the flags are updated in
1540 * thread-safe manner.
1543 arc_hdr_set_compress(arc_buf_hdr_t
*hdr
, enum zio_compress cmp
)
1545 ASSERT(HDR_EMPTY_OR_LOCKED(hdr
));
1548 * Holes and embedded blocks will always have a psize = 0 so
1549 * we ignore the compression of the blkptr and set the
1550 * arc_buf_hdr_t's compression to ZIO_COMPRESS_OFF.
1551 * Holes and embedded blocks remain anonymous so we don't
1552 * want to uncompress them. Mark them as uncompressed.
1554 if (!zfs_compressed_arc_enabled
|| HDR_GET_PSIZE(hdr
) == 0) {
1555 arc_hdr_clear_flags(hdr
, ARC_FLAG_COMPRESSED_ARC
);
1556 ASSERT(!HDR_COMPRESSION_ENABLED(hdr
));
1558 arc_hdr_set_flags(hdr
, ARC_FLAG_COMPRESSED_ARC
);
1559 ASSERT(HDR_COMPRESSION_ENABLED(hdr
));
1562 HDR_SET_COMPRESS(hdr
, cmp
);
1563 ASSERT3U(HDR_GET_COMPRESS(hdr
), ==, cmp
);
1567 * Looks for another buf on the same hdr which has the data decompressed, copies
1568 * from it, and returns true. If no such buf exists, returns false.
1571 arc_buf_try_copy_decompressed_data(arc_buf_t
*buf
)
1573 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
1574 boolean_t copied
= B_FALSE
;
1576 ASSERT(HDR_HAS_L1HDR(hdr
));
1577 ASSERT3P(buf
->b_data
, !=, NULL
);
1578 ASSERT(!ARC_BUF_COMPRESSED(buf
));
1580 for (arc_buf_t
*from
= hdr
->b_l1hdr
.b_buf
; from
!= NULL
;
1581 from
= from
->b_next
) {
1582 /* can't use our own data buffer */
1587 if (!ARC_BUF_COMPRESSED(from
)) {
1588 bcopy(from
->b_data
, buf
->b_data
, arc_buf_size(buf
));
1595 * Note: With encryption support, the following assertion is no longer
1596 * necessarily valid. If we receive two back to back raw snapshots
1597 * (send -w), the second receive can use a hdr with a cksum already
1598 * calculated. This happens via:
1599 * dmu_recv_stream() -> receive_read_record() -> arc_loan_raw_buf()
1600 * The rsend/send_mixed_raw test case exercises this code path.
1602 * There were no decompressed bufs, so there should not be a
1603 * checksum on the hdr either.
1604 * EQUIV(!copied, hdr->b_l1hdr.b_freeze_cksum == NULL);
1611 * Return the size of the block, b_pabd, that is stored in the arc_buf_hdr_t.
1614 arc_hdr_size(arc_buf_hdr_t
*hdr
)
1618 if (arc_hdr_get_compress(hdr
) != ZIO_COMPRESS_OFF
&&
1619 HDR_GET_PSIZE(hdr
) > 0) {
1620 size
= HDR_GET_PSIZE(hdr
);
1622 ASSERT3U(HDR_GET_LSIZE(hdr
), !=, 0);
1623 size
= HDR_GET_LSIZE(hdr
);
1629 arc_hdr_authenticate(arc_buf_hdr_t
*hdr
, spa_t
*spa
, uint64_t dsobj
)
1633 uint64_t lsize
= HDR_GET_LSIZE(hdr
);
1634 uint64_t psize
= HDR_GET_PSIZE(hdr
);
1635 void *tmpbuf
= NULL
;
1636 abd_t
*abd
= hdr
->b_l1hdr
.b_pabd
;
1638 ASSERT(HDR_EMPTY_OR_LOCKED(hdr
));
1639 ASSERT(HDR_AUTHENTICATED(hdr
));
1640 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, !=, NULL
);
1643 * The MAC is calculated on the compressed data that is stored on disk.
1644 * However, if compressed arc is disabled we will only have the
1645 * decompressed data available to us now. Compress it into a temporary
1646 * abd so we can verify the MAC. The performance overhead of this will
1647 * be relatively low, since most objects in an encrypted objset will
1648 * be encrypted (instead of authenticated) anyway.
1650 if (HDR_GET_COMPRESS(hdr
) != ZIO_COMPRESS_OFF
&&
1651 !HDR_COMPRESSION_ENABLED(hdr
)) {
1652 tmpbuf
= zio_buf_alloc(lsize
);
1653 abd
= abd_get_from_buf(tmpbuf
, lsize
);
1654 abd_take_ownership_of_buf(abd
, B_TRUE
);
1656 csize
= zio_compress_data(HDR_GET_COMPRESS(hdr
),
1657 hdr
->b_l1hdr
.b_pabd
, tmpbuf
, lsize
);
1658 ASSERT3U(csize
, <=, psize
);
1659 abd_zero_off(abd
, csize
, psize
- csize
);
1663 * Authentication is best effort. We authenticate whenever the key is
1664 * available. If we succeed we clear ARC_FLAG_NOAUTH.
1666 if (hdr
->b_crypt_hdr
.b_ot
== DMU_OT_OBJSET
) {
1667 ASSERT3U(HDR_GET_COMPRESS(hdr
), ==, ZIO_COMPRESS_OFF
);
1668 ASSERT3U(lsize
, ==, psize
);
1669 ret
= spa_do_crypt_objset_mac_abd(B_FALSE
, spa
, dsobj
, abd
,
1670 psize
, hdr
->b_l1hdr
.b_byteswap
!= DMU_BSWAP_NUMFUNCS
);
1672 ret
= spa_do_crypt_mac_abd(B_FALSE
, spa
, dsobj
, abd
, psize
,
1673 hdr
->b_crypt_hdr
.b_mac
);
1677 arc_hdr_clear_flags(hdr
, ARC_FLAG_NOAUTH
);
1678 else if (ret
!= ENOENT
)
1694 * This function will take a header that only has raw encrypted data in
1695 * b_crypt_hdr.b_rabd and decrypt it into a new buffer which is stored in
1696 * b_l1hdr.b_pabd. If designated in the header flags, this function will
1697 * also decompress the data.
1700 arc_hdr_decrypt(arc_buf_hdr_t
*hdr
, spa_t
*spa
, const zbookmark_phys_t
*zb
)
1705 boolean_t no_crypt
= B_FALSE
;
1706 boolean_t bswap
= (hdr
->b_l1hdr
.b_byteswap
!= DMU_BSWAP_NUMFUNCS
);
1708 ASSERT(HDR_EMPTY_OR_LOCKED(hdr
));
1709 ASSERT(HDR_ENCRYPTED(hdr
));
1711 arc_hdr_alloc_pabd(hdr
, ARC_HDR_DO_ADAPT
);
1713 ret
= spa_do_crypt_abd(B_FALSE
, spa
, zb
, hdr
->b_crypt_hdr
.b_ot
,
1714 B_FALSE
, bswap
, hdr
->b_crypt_hdr
.b_salt
, hdr
->b_crypt_hdr
.b_iv
,
1715 hdr
->b_crypt_hdr
.b_mac
, HDR_GET_PSIZE(hdr
), hdr
->b_l1hdr
.b_pabd
,
1716 hdr
->b_crypt_hdr
.b_rabd
, &no_crypt
);
1721 abd_copy(hdr
->b_l1hdr
.b_pabd
, hdr
->b_crypt_hdr
.b_rabd
,
1722 HDR_GET_PSIZE(hdr
));
1726 * If this header has disabled arc compression but the b_pabd is
1727 * compressed after decrypting it, we need to decompress the newly
1730 if (HDR_GET_COMPRESS(hdr
) != ZIO_COMPRESS_OFF
&&
1731 !HDR_COMPRESSION_ENABLED(hdr
)) {
1733 * We want to make sure that we are correctly honoring the
1734 * zfs_abd_scatter_enabled setting, so we allocate an abd here
1735 * and then loan a buffer from it, rather than allocating a
1736 * linear buffer and wrapping it in an abd later.
1738 cabd
= arc_get_data_abd(hdr
, arc_hdr_size(hdr
), hdr
, B_TRUE
);
1739 tmp
= abd_borrow_buf(cabd
, arc_hdr_size(hdr
));
1741 ret
= zio_decompress_data(HDR_GET_COMPRESS(hdr
),
1742 hdr
->b_l1hdr
.b_pabd
, tmp
, HDR_GET_PSIZE(hdr
),
1743 HDR_GET_LSIZE(hdr
));
1745 abd_return_buf(cabd
, tmp
, arc_hdr_size(hdr
));
1749 abd_return_buf_copy(cabd
, tmp
, arc_hdr_size(hdr
));
1750 arc_free_data_abd(hdr
, hdr
->b_l1hdr
.b_pabd
,
1751 arc_hdr_size(hdr
), hdr
);
1752 hdr
->b_l1hdr
.b_pabd
= cabd
;
1758 arc_hdr_free_pabd(hdr
, B_FALSE
);
1760 arc_free_data_buf(hdr
, cabd
, arc_hdr_size(hdr
), hdr
);
1766 * This function is called during arc_buf_fill() to prepare the header's
1767 * abd plaintext pointer for use. This involves authenticated protected
1768 * data and decrypting encrypted data into the plaintext abd.
1771 arc_fill_hdr_crypt(arc_buf_hdr_t
*hdr
, kmutex_t
*hash_lock
, spa_t
*spa
,
1772 const zbookmark_phys_t
*zb
, boolean_t noauth
)
1776 ASSERT(HDR_PROTECTED(hdr
));
1778 if (hash_lock
!= NULL
)
1779 mutex_enter(hash_lock
);
1781 if (HDR_NOAUTH(hdr
) && !noauth
) {
1783 * The caller requested authenticated data but our data has
1784 * not been authenticated yet. Verify the MAC now if we can.
1786 ret
= arc_hdr_authenticate(hdr
, spa
, zb
->zb_objset
);
1789 } else if (HDR_HAS_RABD(hdr
) && hdr
->b_l1hdr
.b_pabd
== NULL
) {
1791 * If we only have the encrypted version of the data, but the
1792 * unencrypted version was requested we take this opportunity
1793 * to store the decrypted version in the header for future use.
1795 ret
= arc_hdr_decrypt(hdr
, spa
, zb
);
1800 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, !=, NULL
);
1802 if (hash_lock
!= NULL
)
1803 mutex_exit(hash_lock
);
1808 if (hash_lock
!= NULL
)
1809 mutex_exit(hash_lock
);
1815 * This function is used by the dbuf code to decrypt bonus buffers in place.
1816 * The dbuf code itself doesn't have any locking for decrypting a shared dnode
1817 * block, so we use the hash lock here to protect against concurrent calls to
1822 arc_buf_untransform_in_place(arc_buf_t
*buf
, kmutex_t
*hash_lock
)
1824 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
1826 ASSERT(HDR_ENCRYPTED(hdr
));
1827 ASSERT3U(hdr
->b_crypt_hdr
.b_ot
, ==, DMU_OT_DNODE
);
1828 ASSERT(HDR_EMPTY_OR_LOCKED(hdr
));
1829 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, !=, NULL
);
1831 zio_crypt_copy_dnode_bonus(hdr
->b_l1hdr
.b_pabd
, buf
->b_data
,
1833 buf
->b_flags
&= ~ARC_BUF_FLAG_ENCRYPTED
;
1834 buf
->b_flags
&= ~ARC_BUF_FLAG_COMPRESSED
;
1835 hdr
->b_crypt_hdr
.b_ebufcnt
-= 1;
1839 * Given a buf that has a data buffer attached to it, this function will
1840 * efficiently fill the buf with data of the specified compression setting from
1841 * the hdr and update the hdr's b_freeze_cksum if necessary. If the buf and hdr
1842 * are already sharing a data buf, no copy is performed.
1844 * If the buf is marked as compressed but uncompressed data was requested, this
1845 * will allocate a new data buffer for the buf, remove that flag, and fill the
1846 * buf with uncompressed data. You can't request a compressed buf on a hdr with
1847 * uncompressed data, and (since we haven't added support for it yet) if you
1848 * want compressed data your buf must already be marked as compressed and have
1849 * the correct-sized data buffer.
1852 arc_buf_fill(arc_buf_t
*buf
, spa_t
*spa
, const zbookmark_phys_t
*zb
,
1853 arc_fill_flags_t flags
)
1856 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
1857 boolean_t hdr_compressed
=
1858 (arc_hdr_get_compress(hdr
) != ZIO_COMPRESS_OFF
);
1859 boolean_t compressed
= (flags
& ARC_FILL_COMPRESSED
) != 0;
1860 boolean_t encrypted
= (flags
& ARC_FILL_ENCRYPTED
) != 0;
1861 dmu_object_byteswap_t bswap
= hdr
->b_l1hdr
.b_byteswap
;
1862 kmutex_t
*hash_lock
= (flags
& ARC_FILL_LOCKED
) ? NULL
: HDR_LOCK(hdr
);
1864 ASSERT3P(buf
->b_data
, !=, NULL
);
1865 IMPLY(compressed
, hdr_compressed
|| ARC_BUF_ENCRYPTED(buf
));
1866 IMPLY(compressed
, ARC_BUF_COMPRESSED(buf
));
1867 IMPLY(encrypted
, HDR_ENCRYPTED(hdr
));
1868 IMPLY(encrypted
, ARC_BUF_ENCRYPTED(buf
));
1869 IMPLY(encrypted
, ARC_BUF_COMPRESSED(buf
));
1870 IMPLY(encrypted
, !ARC_BUF_SHARED(buf
));
1873 * If the caller wanted encrypted data we just need to copy it from
1874 * b_rabd and potentially byteswap it. We won't be able to do any
1875 * further transforms on it.
1878 ASSERT(HDR_HAS_RABD(hdr
));
1879 abd_copy_to_buf(buf
->b_data
, hdr
->b_crypt_hdr
.b_rabd
,
1880 HDR_GET_PSIZE(hdr
));
1885 * Adjust encrypted and authenticated headers to accomodate
1886 * the request if needed. Dnode blocks (ARC_FILL_IN_PLACE) are
1887 * allowed to fail decryption due to keys not being loaded
1888 * without being marked as an IO error.
1890 if (HDR_PROTECTED(hdr
)) {
1891 error
= arc_fill_hdr_crypt(hdr
, hash_lock
, spa
,
1892 zb
, !!(flags
& ARC_FILL_NOAUTH
));
1893 if (error
== EACCES
&& (flags
& ARC_FILL_IN_PLACE
) != 0) {
1895 } else if (error
!= 0) {
1896 if (hash_lock
!= NULL
)
1897 mutex_enter(hash_lock
);
1898 arc_hdr_set_flags(hdr
, ARC_FLAG_IO_ERROR
);
1899 if (hash_lock
!= NULL
)
1900 mutex_exit(hash_lock
);
1906 * There is a special case here for dnode blocks which are
1907 * decrypting their bonus buffers. These blocks may request to
1908 * be decrypted in-place. This is necessary because there may
1909 * be many dnodes pointing into this buffer and there is
1910 * currently no method to synchronize replacing the backing
1911 * b_data buffer and updating all of the pointers. Here we use
1912 * the hash lock to ensure there are no races. If the need
1913 * arises for other types to be decrypted in-place, they must
1914 * add handling here as well.
1916 if ((flags
& ARC_FILL_IN_PLACE
) != 0) {
1917 ASSERT(!hdr_compressed
);
1918 ASSERT(!compressed
);
1921 if (HDR_ENCRYPTED(hdr
) && ARC_BUF_ENCRYPTED(buf
)) {
1922 ASSERT3U(hdr
->b_crypt_hdr
.b_ot
, ==, DMU_OT_DNODE
);
1924 if (hash_lock
!= NULL
)
1925 mutex_enter(hash_lock
);
1926 arc_buf_untransform_in_place(buf
, hash_lock
);
1927 if (hash_lock
!= NULL
)
1928 mutex_exit(hash_lock
);
1930 /* Compute the hdr's checksum if necessary */
1931 arc_cksum_compute(buf
);
1937 if (hdr_compressed
== compressed
) {
1938 if (!arc_buf_is_shared(buf
)) {
1939 abd_copy_to_buf(buf
->b_data
, hdr
->b_l1hdr
.b_pabd
,
1943 ASSERT(hdr_compressed
);
1944 ASSERT(!compressed
);
1945 ASSERT3U(HDR_GET_LSIZE(hdr
), !=, HDR_GET_PSIZE(hdr
));
1948 * If the buf is sharing its data with the hdr, unlink it and
1949 * allocate a new data buffer for the buf.
1951 if (arc_buf_is_shared(buf
)) {
1952 ASSERT(ARC_BUF_COMPRESSED(buf
));
1954 /* We need to give the buf its own b_data */
1955 buf
->b_flags
&= ~ARC_BUF_FLAG_SHARED
;
1957 arc_get_data_buf(hdr
, HDR_GET_LSIZE(hdr
), buf
);
1958 arc_hdr_clear_flags(hdr
, ARC_FLAG_SHARED_DATA
);
1960 /* Previously overhead was 0; just add new overhead */
1961 ARCSTAT_INCR(arcstat_overhead_size
, HDR_GET_LSIZE(hdr
));
1962 } else if (ARC_BUF_COMPRESSED(buf
)) {
1963 /* We need to reallocate the buf's b_data */
1964 arc_free_data_buf(hdr
, buf
->b_data
, HDR_GET_PSIZE(hdr
),
1967 arc_get_data_buf(hdr
, HDR_GET_LSIZE(hdr
), buf
);
1969 /* We increased the size of b_data; update overhead */
1970 ARCSTAT_INCR(arcstat_overhead_size
,
1971 HDR_GET_LSIZE(hdr
) - HDR_GET_PSIZE(hdr
));
1975 * Regardless of the buf's previous compression settings, it
1976 * should not be compressed at the end of this function.
1978 buf
->b_flags
&= ~ARC_BUF_FLAG_COMPRESSED
;
1981 * Try copying the data from another buf which already has a
1982 * decompressed version. If that's not possible, it's time to
1983 * bite the bullet and decompress the data from the hdr.
1985 if (arc_buf_try_copy_decompressed_data(buf
)) {
1986 /* Skip byteswapping and checksumming (already done) */
1987 ASSERT3P(hdr
->b_l1hdr
.b_freeze_cksum
, !=, NULL
);
1990 error
= zio_decompress_data(HDR_GET_COMPRESS(hdr
),
1991 hdr
->b_l1hdr
.b_pabd
, buf
->b_data
,
1992 HDR_GET_PSIZE(hdr
), HDR_GET_LSIZE(hdr
));
1995 * Absent hardware errors or software bugs, this should
1996 * be impossible, but log it anyway so we can debug it.
2000 "hdr %p, compress %d, psize %d, lsize %d",
2001 hdr
, arc_hdr_get_compress(hdr
),
2002 HDR_GET_PSIZE(hdr
), HDR_GET_LSIZE(hdr
));
2003 if (hash_lock
!= NULL
)
2004 mutex_enter(hash_lock
);
2005 arc_hdr_set_flags(hdr
, ARC_FLAG_IO_ERROR
);
2006 if (hash_lock
!= NULL
)
2007 mutex_exit(hash_lock
);
2008 return (SET_ERROR(EIO
));
2014 /* Byteswap the buf's data if necessary */
2015 if (bswap
!= DMU_BSWAP_NUMFUNCS
) {
2016 ASSERT(!HDR_SHARED_DATA(hdr
));
2017 ASSERT3U(bswap
, <, DMU_BSWAP_NUMFUNCS
);
2018 dmu_ot_byteswap
[bswap
].ob_func(buf
->b_data
, HDR_GET_LSIZE(hdr
));
2021 /* Compute the hdr's checksum if necessary */
2022 arc_cksum_compute(buf
);
2028 * If this function is being called to decrypt an encrypted buffer or verify an
2029 * authenticated one, the key must be loaded and a mapping must be made
2030 * available in the keystore via spa_keystore_create_mapping() or one of its
2034 arc_untransform(arc_buf_t
*buf
, spa_t
*spa
, const zbookmark_phys_t
*zb
,
2038 arc_fill_flags_t flags
= 0;
2041 flags
|= ARC_FILL_IN_PLACE
;
2043 ret
= arc_buf_fill(buf
, spa
, zb
, flags
);
2044 if (ret
== ECKSUM
) {
2046 * Convert authentication and decryption errors to EIO
2047 * (and generate an ereport) before leaving the ARC.
2049 ret
= SET_ERROR(EIO
);
2050 spa_log_error(spa
, zb
);
2051 (void) zfs_ereport_post(FM_EREPORT_ZFS_AUTHENTICATION
,
2052 spa
, NULL
, zb
, NULL
, 0, 0);
2059 * Increment the amount of evictable space in the arc_state_t's refcount.
2060 * We account for the space used by the hdr and the arc buf individually
2061 * so that we can add and remove them from the refcount individually.
2064 arc_evictable_space_increment(arc_buf_hdr_t
*hdr
, arc_state_t
*state
)
2066 arc_buf_contents_t type
= arc_buf_type(hdr
);
2068 ASSERT(HDR_HAS_L1HDR(hdr
));
2070 if (GHOST_STATE(state
)) {
2071 ASSERT0(hdr
->b_l1hdr
.b_bufcnt
);
2072 ASSERT3P(hdr
->b_l1hdr
.b_buf
, ==, NULL
);
2073 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, ==, NULL
);
2074 ASSERT(!HDR_HAS_RABD(hdr
));
2075 (void) zfs_refcount_add_many(&state
->arcs_esize
[type
],
2076 HDR_GET_LSIZE(hdr
), hdr
);
2080 ASSERT(!GHOST_STATE(state
));
2081 if (hdr
->b_l1hdr
.b_pabd
!= NULL
) {
2082 (void) zfs_refcount_add_many(&state
->arcs_esize
[type
],
2083 arc_hdr_size(hdr
), hdr
);
2085 if (HDR_HAS_RABD(hdr
)) {
2086 (void) zfs_refcount_add_many(&state
->arcs_esize
[type
],
2087 HDR_GET_PSIZE(hdr
), hdr
);
2089 for (arc_buf_t
*buf
= hdr
->b_l1hdr
.b_buf
; buf
!= NULL
;
2090 buf
= buf
->b_next
) {
2091 if (arc_buf_is_shared(buf
))
2093 (void) zfs_refcount_add_many(&state
->arcs_esize
[type
],
2094 arc_buf_size(buf
), buf
);
2099 * Decrement the amount of evictable space in the arc_state_t's refcount.
2100 * We account for the space used by the hdr and the arc buf individually
2101 * so that we can add and remove them from the refcount individually.
2104 arc_evictable_space_decrement(arc_buf_hdr_t
*hdr
, arc_state_t
*state
)
2106 arc_buf_contents_t type
= arc_buf_type(hdr
);
2108 ASSERT(HDR_HAS_L1HDR(hdr
));
2110 if (GHOST_STATE(state
)) {
2111 ASSERT0(hdr
->b_l1hdr
.b_bufcnt
);
2112 ASSERT3P(hdr
->b_l1hdr
.b_buf
, ==, NULL
);
2113 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, ==, NULL
);
2114 ASSERT(!HDR_HAS_RABD(hdr
));
2115 (void) zfs_refcount_remove_many(&state
->arcs_esize
[type
],
2116 HDR_GET_LSIZE(hdr
), hdr
);
2120 ASSERT(!GHOST_STATE(state
));
2121 if (hdr
->b_l1hdr
.b_pabd
!= NULL
) {
2122 (void) zfs_refcount_remove_many(&state
->arcs_esize
[type
],
2123 arc_hdr_size(hdr
), hdr
);
2125 if (HDR_HAS_RABD(hdr
)) {
2126 (void) zfs_refcount_remove_many(&state
->arcs_esize
[type
],
2127 HDR_GET_PSIZE(hdr
), hdr
);
2129 for (arc_buf_t
*buf
= hdr
->b_l1hdr
.b_buf
; buf
!= NULL
;
2130 buf
= buf
->b_next
) {
2131 if (arc_buf_is_shared(buf
))
2133 (void) zfs_refcount_remove_many(&state
->arcs_esize
[type
],
2134 arc_buf_size(buf
), buf
);
2139 * Add a reference to this hdr indicating that someone is actively
2140 * referencing that memory. When the refcount transitions from 0 to 1,
2141 * we remove it from the respective arc_state_t list to indicate that
2142 * it is not evictable.
2145 add_reference(arc_buf_hdr_t
*hdr
, void *tag
)
2147 ASSERT(HDR_HAS_L1HDR(hdr
));
2148 if (!HDR_EMPTY(hdr
) && !MUTEX_HELD(HDR_LOCK(hdr
))) {
2149 ASSERT(hdr
->b_l1hdr
.b_state
== arc_anon
);
2150 ASSERT(zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
));
2151 ASSERT3P(hdr
->b_l1hdr
.b_buf
, ==, NULL
);
2154 arc_state_t
*state
= hdr
->b_l1hdr
.b_state
;
2156 if ((zfs_refcount_add(&hdr
->b_l1hdr
.b_refcnt
, tag
) == 1) &&
2157 (state
!= arc_anon
)) {
2158 /* We don't use the L2-only state list. */
2159 if (state
!= arc_l2c_only
) {
2160 multilist_remove(state
->arcs_list
[arc_buf_type(hdr
)],
2162 arc_evictable_space_decrement(hdr
, state
);
2164 /* remove the prefetch flag if we get a reference */
2165 if (HDR_HAS_L2HDR(hdr
))
2166 l2arc_hdr_arcstats_decrement_state(hdr
);
2167 arc_hdr_clear_flags(hdr
, ARC_FLAG_PREFETCH
);
2168 if (HDR_HAS_L2HDR(hdr
))
2169 l2arc_hdr_arcstats_increment_state(hdr
);
2174 * Remove a reference from this hdr. When the reference transitions from
2175 * 1 to 0 and we're not anonymous, then we add this hdr to the arc_state_t's
2176 * list making it eligible for eviction.
2179 remove_reference(arc_buf_hdr_t
*hdr
, kmutex_t
*hash_lock
, void *tag
)
2182 arc_state_t
*state
= hdr
->b_l1hdr
.b_state
;
2184 ASSERT(HDR_HAS_L1HDR(hdr
));
2185 ASSERT(state
== arc_anon
|| MUTEX_HELD(hash_lock
));
2186 ASSERT(!GHOST_STATE(state
));
2189 * arc_l2c_only counts as a ghost state so we don't need to explicitly
2190 * check to prevent usage of the arc_l2c_only list.
2192 if (((cnt
= zfs_refcount_remove(&hdr
->b_l1hdr
.b_refcnt
, tag
)) == 0) &&
2193 (state
!= arc_anon
)) {
2194 multilist_insert(state
->arcs_list
[arc_buf_type(hdr
)], hdr
);
2195 ASSERT3U(hdr
->b_l1hdr
.b_bufcnt
, >, 0);
2196 arc_evictable_space_increment(hdr
, state
);
2202 * Move the supplied buffer to the indicated state. The hash lock
2203 * for the buffer must be held by the caller.
2206 arc_change_state(arc_state_t
*new_state
, arc_buf_hdr_t
*hdr
,
2207 kmutex_t
*hash_lock
)
2209 arc_state_t
*old_state
;
2212 boolean_t update_old
, update_new
;
2213 arc_buf_contents_t buftype
= arc_buf_type(hdr
);
2216 * We almost always have an L1 hdr here, since we call arc_hdr_realloc()
2217 * in arc_read() when bringing a buffer out of the L2ARC. However, the
2218 * L1 hdr doesn't always exist when we change state to arc_anon before
2219 * destroying a header, in which case reallocating to add the L1 hdr is
2222 if (HDR_HAS_L1HDR(hdr
)) {
2223 old_state
= hdr
->b_l1hdr
.b_state
;
2224 refcnt
= zfs_refcount_count(&hdr
->b_l1hdr
.b_refcnt
);
2225 bufcnt
= hdr
->b_l1hdr
.b_bufcnt
;
2227 update_old
= (bufcnt
> 0 || hdr
->b_l1hdr
.b_pabd
!= NULL
||
2230 old_state
= arc_l2c_only
;
2233 update_old
= B_FALSE
;
2235 update_new
= update_old
;
2237 ASSERT(MUTEX_HELD(hash_lock
));
2238 ASSERT3P(new_state
, !=, old_state
);
2239 ASSERT(!GHOST_STATE(new_state
) || bufcnt
== 0);
2240 ASSERT(old_state
!= arc_anon
|| bufcnt
<= 1);
2243 * If this buffer is evictable, transfer it from the
2244 * old state list to the new state list.
2247 if (old_state
!= arc_anon
&& old_state
!= arc_l2c_only
) {
2248 ASSERT(HDR_HAS_L1HDR(hdr
));
2249 multilist_remove(old_state
->arcs_list
[buftype
], hdr
);
2251 if (GHOST_STATE(old_state
)) {
2253 ASSERT3P(hdr
->b_l1hdr
.b_buf
, ==, NULL
);
2254 update_old
= B_TRUE
;
2256 arc_evictable_space_decrement(hdr
, old_state
);
2258 if (new_state
!= arc_anon
&& new_state
!= arc_l2c_only
) {
2261 * An L1 header always exists here, since if we're
2262 * moving to some L1-cached state (i.e. not l2c_only or
2263 * anonymous), we realloc the header to add an L1hdr
2266 ASSERT(HDR_HAS_L1HDR(hdr
));
2267 multilist_insert(new_state
->arcs_list
[buftype
], hdr
);
2269 if (GHOST_STATE(new_state
)) {
2271 ASSERT3P(hdr
->b_l1hdr
.b_buf
, ==, NULL
);
2272 update_new
= B_TRUE
;
2274 arc_evictable_space_increment(hdr
, new_state
);
2278 ASSERT(!HDR_EMPTY(hdr
));
2279 if (new_state
== arc_anon
&& HDR_IN_HASH_TABLE(hdr
))
2280 buf_hash_remove(hdr
);
2282 /* adjust state sizes (ignore arc_l2c_only) */
2284 if (update_new
&& new_state
!= arc_l2c_only
) {
2285 ASSERT(HDR_HAS_L1HDR(hdr
));
2286 if (GHOST_STATE(new_state
)) {
2290 * When moving a header to a ghost state, we first
2291 * remove all arc buffers. Thus, we'll have a
2292 * bufcnt of zero, and no arc buffer to use for
2293 * the reference. As a result, we use the arc
2294 * header pointer for the reference.
2296 (void) zfs_refcount_add_many(&new_state
->arcs_size
,
2297 HDR_GET_LSIZE(hdr
), hdr
);
2298 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, ==, NULL
);
2299 ASSERT(!HDR_HAS_RABD(hdr
));
2301 uint32_t buffers
= 0;
2304 * Each individual buffer holds a unique reference,
2305 * thus we must remove each of these references one
2308 for (arc_buf_t
*buf
= hdr
->b_l1hdr
.b_buf
; buf
!= NULL
;
2309 buf
= buf
->b_next
) {
2310 ASSERT3U(bufcnt
, !=, 0);
2314 * When the arc_buf_t is sharing the data
2315 * block with the hdr, the owner of the
2316 * reference belongs to the hdr. Only
2317 * add to the refcount if the arc_buf_t is
2320 if (arc_buf_is_shared(buf
))
2323 (void) zfs_refcount_add_many(
2324 &new_state
->arcs_size
,
2325 arc_buf_size(buf
), buf
);
2327 ASSERT3U(bufcnt
, ==, buffers
);
2329 if (hdr
->b_l1hdr
.b_pabd
!= NULL
) {
2330 (void) zfs_refcount_add_many(
2331 &new_state
->arcs_size
,
2332 arc_hdr_size(hdr
), hdr
);
2335 if (HDR_HAS_RABD(hdr
)) {
2336 (void) zfs_refcount_add_many(
2337 &new_state
->arcs_size
,
2338 HDR_GET_PSIZE(hdr
), hdr
);
2343 if (update_old
&& old_state
!= arc_l2c_only
) {
2344 ASSERT(HDR_HAS_L1HDR(hdr
));
2345 if (GHOST_STATE(old_state
)) {
2347 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, ==, NULL
);
2348 ASSERT(!HDR_HAS_RABD(hdr
));
2351 * When moving a header off of a ghost state,
2352 * the header will not contain any arc buffers.
2353 * We use the arc header pointer for the reference
2354 * which is exactly what we did when we put the
2355 * header on the ghost state.
2358 (void) zfs_refcount_remove_many(&old_state
->arcs_size
,
2359 HDR_GET_LSIZE(hdr
), hdr
);
2361 uint32_t buffers
= 0;
2364 * Each individual buffer holds a unique reference,
2365 * thus we must remove each of these references one
2368 for (arc_buf_t
*buf
= hdr
->b_l1hdr
.b_buf
; buf
!= NULL
;
2369 buf
= buf
->b_next
) {
2370 ASSERT3U(bufcnt
, !=, 0);
2374 * When the arc_buf_t is sharing the data
2375 * block with the hdr, the owner of the
2376 * reference belongs to the hdr. Only
2377 * add to the refcount if the arc_buf_t is
2380 if (arc_buf_is_shared(buf
))
2383 (void) zfs_refcount_remove_many(
2384 &old_state
->arcs_size
, arc_buf_size(buf
),
2387 ASSERT3U(bufcnt
, ==, buffers
);
2388 ASSERT(hdr
->b_l1hdr
.b_pabd
!= NULL
||
2391 if (hdr
->b_l1hdr
.b_pabd
!= NULL
) {
2392 (void) zfs_refcount_remove_many(
2393 &old_state
->arcs_size
, arc_hdr_size(hdr
),
2397 if (HDR_HAS_RABD(hdr
)) {
2398 (void) zfs_refcount_remove_many(
2399 &old_state
->arcs_size
, HDR_GET_PSIZE(hdr
),
2405 if (HDR_HAS_L1HDR(hdr
)) {
2406 hdr
->b_l1hdr
.b_state
= new_state
;
2408 if (HDR_HAS_L2HDR(hdr
) && new_state
!= arc_l2c_only
) {
2409 l2arc_hdr_arcstats_decrement_state(hdr
);
2410 hdr
->b_l2hdr
.b_arcs_state
= new_state
->arcs_state
;
2411 l2arc_hdr_arcstats_increment_state(hdr
);
2416 * L2 headers should never be on the L2 state list since they don't
2417 * have L1 headers allocated.
2419 ASSERT(multilist_is_empty(arc_l2c_only
->arcs_list
[ARC_BUFC_DATA
]) &&
2420 multilist_is_empty(arc_l2c_only
->arcs_list
[ARC_BUFC_METADATA
]));
2424 arc_space_consume(uint64_t space
, arc_space_type_t type
)
2426 ASSERT(type
>= 0 && type
< ARC_SPACE_NUMTYPES
);
2429 case ARC_SPACE_DATA
:
2430 aggsum_add(&astat_data_size
, space
);
2432 case ARC_SPACE_META
:
2433 aggsum_add(&astat_metadata_size
, space
);
2435 case ARC_SPACE_OTHER
:
2436 aggsum_add(&astat_other_size
, space
);
2438 case ARC_SPACE_HDRS
:
2439 aggsum_add(&astat_hdr_size
, space
);
2441 case ARC_SPACE_L2HDRS
:
2442 aggsum_add(&astat_l2_hdr_size
, space
);
2446 if (type
!= ARC_SPACE_DATA
)
2447 aggsum_add(&arc_meta_used
, space
);
2449 aggsum_add(&arc_size
, space
);
2453 arc_space_return(uint64_t space
, arc_space_type_t type
)
2455 ASSERT(type
>= 0 && type
< ARC_SPACE_NUMTYPES
);
2458 case ARC_SPACE_DATA
:
2459 aggsum_add(&astat_data_size
, -space
);
2461 case ARC_SPACE_META
:
2462 aggsum_add(&astat_metadata_size
, -space
);
2464 case ARC_SPACE_OTHER
:
2465 aggsum_add(&astat_other_size
, -space
);
2467 case ARC_SPACE_HDRS
:
2468 aggsum_add(&astat_hdr_size
, -space
);
2470 case ARC_SPACE_L2HDRS
:
2471 aggsum_add(&astat_l2_hdr_size
, -space
);
2475 if (type
!= ARC_SPACE_DATA
) {
2476 ASSERT(aggsum_compare(&arc_meta_used
, space
) >= 0);
2478 * We use the upper bound here rather than the precise value
2479 * because the arc_meta_max value doesn't need to be
2480 * precise. It's only consumed by humans via arcstats.
2482 if (arc_meta_max
< aggsum_upper_bound(&arc_meta_used
))
2483 arc_meta_max
= aggsum_upper_bound(&arc_meta_used
);
2484 aggsum_add(&arc_meta_used
, -space
);
2487 ASSERT(aggsum_compare(&arc_size
, space
) >= 0);
2488 aggsum_add(&arc_size
, -space
);
2492 * Given a hdr and a buf, returns whether that buf can share its b_data buffer
2493 * with the hdr's b_pabd.
2496 arc_can_share(arc_buf_hdr_t
*hdr
, arc_buf_t
*buf
)
2499 * The criteria for sharing a hdr's data are:
2500 * 1. the buffer is not encrypted
2501 * 2. the hdr's compression matches the buf's compression
2502 * 3. the hdr doesn't need to be byteswapped
2503 * 4. the hdr isn't already being shared
2504 * 5. the buf is either compressed or it is the last buf in the hdr list
2506 * Criterion #5 maintains the invariant that shared uncompressed
2507 * bufs must be the final buf in the hdr's b_buf list. Reading this, you
2508 * might ask, "if a compressed buf is allocated first, won't that be the
2509 * last thing in the list?", but in that case it's impossible to create
2510 * a shared uncompressed buf anyway (because the hdr must be compressed
2511 * to have the compressed buf). You might also think that #3 is
2512 * sufficient to make this guarantee, however it's possible
2513 * (specifically in the rare L2ARC write race mentioned in
2514 * arc_buf_alloc_impl()) there will be an existing uncompressed buf that
2515 * is sharable, but wasn't at the time of its allocation. Rather than
2516 * allow a new shared uncompressed buf to be created and then shuffle
2517 * the list around to make it the last element, this simply disallows
2518 * sharing if the new buf isn't the first to be added.
2520 ASSERT3P(buf
->b_hdr
, ==, hdr
);
2521 boolean_t hdr_compressed
= arc_hdr_get_compress(hdr
) !=
2523 boolean_t buf_compressed
= ARC_BUF_COMPRESSED(buf
) != 0;
2524 return (!ARC_BUF_ENCRYPTED(buf
) &&
2525 buf_compressed
== hdr_compressed
&&
2526 hdr
->b_l1hdr
.b_byteswap
== DMU_BSWAP_NUMFUNCS
&&
2527 !HDR_SHARED_DATA(hdr
) &&
2528 (ARC_BUF_LAST(buf
) || ARC_BUF_COMPRESSED(buf
)));
2532 * Allocate a buf for this hdr. If you care about the data that's in the hdr,
2533 * or if you want a compressed buffer, pass those flags in. Returns 0 if the
2534 * copy was made successfully, or an error code otherwise.
2537 arc_buf_alloc_impl(arc_buf_hdr_t
*hdr
, spa_t
*spa
, const zbookmark_phys_t
*zb
,
2538 void *tag
, boolean_t encrypted
, boolean_t compressed
, boolean_t noauth
,
2539 boolean_t fill
, arc_buf_t
**ret
)
2542 arc_fill_flags_t flags
= ARC_FILL_LOCKED
;
2544 ASSERT(HDR_HAS_L1HDR(hdr
));
2545 ASSERT3U(HDR_GET_LSIZE(hdr
), >, 0);
2546 VERIFY(hdr
->b_type
== ARC_BUFC_DATA
||
2547 hdr
->b_type
== ARC_BUFC_METADATA
);
2548 ASSERT3P(ret
, !=, NULL
);
2549 ASSERT3P(*ret
, ==, NULL
);
2550 IMPLY(encrypted
, compressed
);
2552 buf
= *ret
= kmem_cache_alloc(buf_cache
, KM_PUSHPAGE
);
2555 buf
->b_next
= hdr
->b_l1hdr
.b_buf
;
2558 add_reference(hdr
, tag
);
2561 * We're about to change the hdr's b_flags. We must either
2562 * hold the hash_lock or be undiscoverable.
2564 ASSERT(HDR_EMPTY_OR_LOCKED(hdr
));
2567 * Only honor requests for compressed bufs if the hdr is actually
2568 * compressed. This must be overriden if the buffer is encrypted since
2569 * encrypted buffers cannot be decompressed.
2572 buf
->b_flags
|= ARC_BUF_FLAG_COMPRESSED
;
2573 buf
->b_flags
|= ARC_BUF_FLAG_ENCRYPTED
;
2574 flags
|= ARC_FILL_COMPRESSED
| ARC_FILL_ENCRYPTED
;
2575 } else if (compressed
&&
2576 arc_hdr_get_compress(hdr
) != ZIO_COMPRESS_OFF
) {
2577 buf
->b_flags
|= ARC_BUF_FLAG_COMPRESSED
;
2578 flags
|= ARC_FILL_COMPRESSED
;
2583 flags
|= ARC_FILL_NOAUTH
;
2587 * If the hdr's data can be shared then we share the data buffer and
2588 * set the appropriate bit in the hdr's b_flags to indicate the hdr is
2589 * allocate a new buffer to store the buf's data.
2591 * There are two additional restrictions here because we're sharing
2592 * hdr -> buf instead of the usual buf -> hdr. First, the hdr can't be
2593 * actively involved in an L2ARC write, because if this buf is used by
2594 * an arc_write() then the hdr's data buffer will be released when the
2595 * write completes, even though the L2ARC write might still be using it.
2596 * Second, the hdr's ABD must be linear so that the buf's user doesn't
2597 * need to be ABD-aware.
2599 boolean_t can_share
= arc_can_share(hdr
, buf
) && !HDR_L2_WRITING(hdr
) &&
2600 hdr
->b_l1hdr
.b_pabd
!= NULL
&& abd_is_linear(hdr
->b_l1hdr
.b_pabd
);
2602 /* Set up b_data and sharing */
2604 buf
->b_data
= abd_to_buf(hdr
->b_l1hdr
.b_pabd
);
2605 buf
->b_flags
|= ARC_BUF_FLAG_SHARED
;
2606 arc_hdr_set_flags(hdr
, ARC_FLAG_SHARED_DATA
);
2609 arc_get_data_buf(hdr
, arc_buf_size(buf
), buf
);
2610 ARCSTAT_INCR(arcstat_overhead_size
, arc_buf_size(buf
));
2612 VERIFY3P(buf
->b_data
, !=, NULL
);
2614 hdr
->b_l1hdr
.b_buf
= buf
;
2615 hdr
->b_l1hdr
.b_bufcnt
+= 1;
2617 hdr
->b_crypt_hdr
.b_ebufcnt
+= 1;
2620 * If the user wants the data from the hdr, we need to either copy or
2621 * decompress the data.
2624 ASSERT3P(zb
, !=, NULL
);
2625 return (arc_buf_fill(buf
, spa
, zb
, flags
));
2631 static char *arc_onloan_tag
= "onloan";
2634 arc_loaned_bytes_update(int64_t delta
)
2636 atomic_add_64(&arc_loaned_bytes
, delta
);
2638 /* assert that it did not wrap around */
2639 ASSERT3S(atomic_add_64_nv(&arc_loaned_bytes
, 0), >=, 0);
2643 * Loan out an anonymous arc buffer. Loaned buffers are not counted as in
2644 * flight data by arc_tempreserve_space() until they are "returned". Loaned
2645 * buffers must be returned to the arc before they can be used by the DMU or
2649 arc_loan_buf(spa_t
*spa
, boolean_t is_metadata
, int size
)
2651 arc_buf_t
*buf
= arc_alloc_buf(spa
, arc_onloan_tag
,
2652 is_metadata
? ARC_BUFC_METADATA
: ARC_BUFC_DATA
, size
);
2654 arc_loaned_bytes_update(arc_buf_size(buf
));
2660 arc_loan_compressed_buf(spa_t
*spa
, uint64_t psize
, uint64_t lsize
,
2661 enum zio_compress compression_type
)
2663 arc_buf_t
*buf
= arc_alloc_compressed_buf(spa
, arc_onloan_tag
,
2664 psize
, lsize
, compression_type
);
2666 arc_loaned_bytes_update(arc_buf_size(buf
));
2672 arc_loan_raw_buf(spa_t
*spa
, uint64_t dsobj
, boolean_t byteorder
,
2673 const uint8_t *salt
, const uint8_t *iv
, const uint8_t *mac
,
2674 dmu_object_type_t ot
, uint64_t psize
, uint64_t lsize
,
2675 enum zio_compress compression_type
)
2677 arc_buf_t
*buf
= arc_alloc_raw_buf(spa
, arc_onloan_tag
, dsobj
,
2678 byteorder
, salt
, iv
, mac
, ot
, psize
, lsize
, compression_type
);
2680 atomic_add_64(&arc_loaned_bytes
, psize
);
2685 * Performance tuning of L2ARC persistence:
2687 * l2arc_rebuild_enabled : A ZFS module parameter that controls whether adding
2688 * an L2ARC device (either at pool import or later) will attempt
2689 * to rebuild L2ARC buffer contents.
2690 * l2arc_rebuild_blocks_min_l2size : A ZFS module parameter that controls
2691 * whether log blocks are written to the L2ARC device. If the L2ARC
2692 * device is less than 1GB, the amount of data l2arc_evict()
2693 * evicts is significant compared to the amount of restored L2ARC
2694 * data. In this case do not write log blocks in L2ARC in order
2695 * not to waste space.
2697 int l2arc_rebuild_enabled
= B_TRUE
;
2698 unsigned long l2arc_rebuild_blocks_min_l2size
= 1024 * 1024 * 1024;
2700 /* L2ARC persistence rebuild control routines. */
2701 void l2arc_rebuild_vdev(vdev_t
*vd
, boolean_t reopen
);
2702 static void l2arc_dev_rebuild_start(l2arc_dev_t
*dev
);
2703 static int l2arc_rebuild(l2arc_dev_t
*dev
);
2705 /* L2ARC persistence read I/O routines. */
2706 static int l2arc_dev_hdr_read(l2arc_dev_t
*dev
);
2707 static int l2arc_log_blk_read(l2arc_dev_t
*dev
,
2708 const l2arc_log_blkptr_t
*this_lp
, const l2arc_log_blkptr_t
*next_lp
,
2709 l2arc_log_blk_phys_t
*this_lb
, l2arc_log_blk_phys_t
*next_lb
,
2710 zio_t
*this_io
, zio_t
**next_io
);
2711 static zio_t
*l2arc_log_blk_fetch(vdev_t
*vd
,
2712 const l2arc_log_blkptr_t
*lp
, l2arc_log_blk_phys_t
*lb
);
2713 static void l2arc_log_blk_fetch_abort(zio_t
*zio
);
2715 /* L2ARC persistence block restoration routines. */
2716 static void l2arc_log_blk_restore(l2arc_dev_t
*dev
,
2717 const l2arc_log_blk_phys_t
*lb
, uint64_t lb_asize
);
2718 static void l2arc_hdr_restore(const l2arc_log_ent_phys_t
*le
,
2721 /* L2ARC persistence write I/O routines. */
2722 static void l2arc_dev_hdr_update(l2arc_dev_t
*dev
);
2723 static void l2arc_log_blk_commit(l2arc_dev_t
*dev
, zio_t
*pio
,
2724 l2arc_write_callback_t
*cb
);
2726 /* L2ARC persistence auxilliary routines. */
2727 boolean_t
l2arc_log_blkptr_valid(l2arc_dev_t
*dev
,
2728 const l2arc_log_blkptr_t
*lbp
);
2729 static boolean_t
l2arc_log_blk_insert(l2arc_dev_t
*dev
,
2730 const arc_buf_hdr_t
*ab
);
2731 boolean_t
l2arc_range_check_overlap(uint64_t bottom
,
2732 uint64_t top
, uint64_t check
);
2733 static void l2arc_blk_fetch_done(zio_t
*zio
);
2734 static inline uint64_t
2735 l2arc_log_blk_overhead(uint64_t write_sz
, l2arc_dev_t
*dev
);
2738 * Return a loaned arc buffer to the arc.
2741 arc_return_buf(arc_buf_t
*buf
, void *tag
)
2743 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
2745 ASSERT3P(buf
->b_data
, !=, NULL
);
2746 ASSERT(HDR_HAS_L1HDR(hdr
));
2747 (void) zfs_refcount_add(&hdr
->b_l1hdr
.b_refcnt
, tag
);
2748 (void) zfs_refcount_remove(&hdr
->b_l1hdr
.b_refcnt
, arc_onloan_tag
);
2750 arc_loaned_bytes_update(-arc_buf_size(buf
));
2753 /* Detach an arc_buf from a dbuf (tag) */
2755 arc_loan_inuse_buf(arc_buf_t
*buf
, void *tag
)
2757 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
2759 ASSERT3P(buf
->b_data
, !=, NULL
);
2760 ASSERT(HDR_HAS_L1HDR(hdr
));
2761 (void) zfs_refcount_add(&hdr
->b_l1hdr
.b_refcnt
, arc_onloan_tag
);
2762 (void) zfs_refcount_remove(&hdr
->b_l1hdr
.b_refcnt
, tag
);
2764 arc_loaned_bytes_update(arc_buf_size(buf
));
2768 l2arc_free_abd_on_write(abd_t
*abd
, size_t size
, arc_buf_contents_t type
)
2770 l2arc_data_free_t
*df
= kmem_alloc(sizeof (*df
), KM_SLEEP
);
2773 df
->l2df_size
= size
;
2774 df
->l2df_type
= type
;
2775 mutex_enter(&l2arc_free_on_write_mtx
);
2776 list_insert_head(l2arc_free_on_write
, df
);
2777 mutex_exit(&l2arc_free_on_write_mtx
);
2781 arc_hdr_free_on_write(arc_buf_hdr_t
*hdr
, boolean_t free_rdata
)
2783 arc_state_t
*state
= hdr
->b_l1hdr
.b_state
;
2784 arc_buf_contents_t type
= arc_buf_type(hdr
);
2785 uint64_t size
= (free_rdata
) ? HDR_GET_PSIZE(hdr
) : arc_hdr_size(hdr
);
2787 /* protected by hash lock, if in the hash table */
2788 if (multilist_link_active(&hdr
->b_l1hdr
.b_arc_node
)) {
2789 ASSERT(zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
));
2790 ASSERT(state
!= arc_anon
&& state
!= arc_l2c_only
);
2792 (void) zfs_refcount_remove_many(&state
->arcs_esize
[type
],
2795 (void) zfs_refcount_remove_many(&state
->arcs_size
, size
, hdr
);
2796 if (type
== ARC_BUFC_METADATA
) {
2797 arc_space_return(size
, ARC_SPACE_META
);
2799 ASSERT(type
== ARC_BUFC_DATA
);
2800 arc_space_return(size
, ARC_SPACE_DATA
);
2804 l2arc_free_abd_on_write(hdr
->b_crypt_hdr
.b_rabd
, size
, type
);
2806 l2arc_free_abd_on_write(hdr
->b_l1hdr
.b_pabd
, size
, type
);
2811 * Share the arc_buf_t's data with the hdr. Whenever we are sharing the
2812 * data buffer, we transfer the refcount ownership to the hdr and update
2813 * the appropriate kstats.
2816 arc_share_buf(arc_buf_hdr_t
*hdr
, arc_buf_t
*buf
)
2819 arc_state_t
*state
= hdr
->b_l1hdr
.b_state
;
2821 ASSERT(arc_can_share(hdr
, buf
));
2822 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, ==, NULL
);
2823 ASSERT(!ARC_BUF_ENCRYPTED(buf
));
2824 ASSERT(HDR_EMPTY_OR_LOCKED(hdr
));
2827 * Start sharing the data buffer. We transfer the
2828 * refcount ownership to the hdr since it always owns
2829 * the refcount whenever an arc_buf_t is shared.
2831 zfs_refcount_transfer_ownership_many(&hdr
->b_l1hdr
.b_state
->arcs_size
,
2832 arc_hdr_size(hdr
), buf
, hdr
);
2833 hdr
->b_l1hdr
.b_pabd
= abd_get_from_buf(buf
->b_data
, arc_buf_size(buf
));
2834 abd_take_ownership_of_buf(hdr
->b_l1hdr
.b_pabd
,
2835 HDR_ISTYPE_METADATA(hdr
));
2836 arc_hdr_set_flags(hdr
, ARC_FLAG_SHARED_DATA
);
2837 buf
->b_flags
|= ARC_BUF_FLAG_SHARED
;
2840 * Since we've transferred ownership to the hdr we need
2841 * to increment its compressed and uncompressed kstats and
2842 * decrement the overhead size.
2844 ARCSTAT_INCR(arcstat_compressed_size
, arc_hdr_size(hdr
));
2845 ARCSTAT_INCR(arcstat_uncompressed_size
, HDR_GET_LSIZE(hdr
));
2846 ARCSTAT_INCR(arcstat_overhead_size
, -arc_buf_size(buf
));
2850 arc_unshare_buf(arc_buf_hdr_t
*hdr
, arc_buf_t
*buf
)
2853 arc_state_t
*state
= hdr
->b_l1hdr
.b_state
;
2855 ASSERT(arc_buf_is_shared(buf
));
2856 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, !=, NULL
);
2857 ASSERT(HDR_EMPTY_OR_LOCKED(hdr
));
2860 * We are no longer sharing this buffer so we need
2861 * to transfer its ownership to the rightful owner.
2863 zfs_refcount_transfer_ownership_many(&hdr
->b_l1hdr
.b_state
->arcs_size
,
2864 arc_hdr_size(hdr
), hdr
, buf
);
2865 arc_hdr_clear_flags(hdr
, ARC_FLAG_SHARED_DATA
);
2866 abd_release_ownership_of_buf(hdr
->b_l1hdr
.b_pabd
);
2867 abd_put(hdr
->b_l1hdr
.b_pabd
);
2868 hdr
->b_l1hdr
.b_pabd
= NULL
;
2869 buf
->b_flags
&= ~ARC_BUF_FLAG_SHARED
;
2872 * Since the buffer is no longer shared between
2873 * the arc buf and the hdr, count it as overhead.
2875 ARCSTAT_INCR(arcstat_compressed_size
, -arc_hdr_size(hdr
));
2876 ARCSTAT_INCR(arcstat_uncompressed_size
, -HDR_GET_LSIZE(hdr
));
2877 ARCSTAT_INCR(arcstat_overhead_size
, arc_buf_size(buf
));
2881 * Remove an arc_buf_t from the hdr's buf list and return the last
2882 * arc_buf_t on the list. If no buffers remain on the list then return
2886 arc_buf_remove(arc_buf_hdr_t
*hdr
, arc_buf_t
*buf
)
2888 arc_buf_t
**bufp
= &hdr
->b_l1hdr
.b_buf
;
2889 arc_buf_t
*lastbuf
= NULL
;
2891 ASSERT(HDR_HAS_L1HDR(hdr
));
2892 ASSERT(HDR_EMPTY_OR_LOCKED(hdr
));
2895 * Remove the buf from the hdr list and locate the last
2896 * remaining buffer on the list.
2898 while (*bufp
!= NULL
) {
2900 *bufp
= buf
->b_next
;
2903 * If we've removed a buffer in the middle of
2904 * the list then update the lastbuf and update
2907 if (*bufp
!= NULL
) {
2909 bufp
= &(*bufp
)->b_next
;
2913 ASSERT3P(lastbuf
, !=, buf
);
2914 IMPLY(hdr
->b_l1hdr
.b_bufcnt
> 0, lastbuf
!= NULL
);
2915 IMPLY(hdr
->b_l1hdr
.b_bufcnt
> 0, hdr
->b_l1hdr
.b_buf
!= NULL
);
2916 IMPLY(lastbuf
!= NULL
, ARC_BUF_LAST(lastbuf
));
2922 * Free up buf->b_data and pull the arc_buf_t off of the the arc_buf_hdr_t's
2926 arc_buf_destroy_impl(arc_buf_t
*buf
)
2928 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
2931 * Free up the data associated with the buf but only if we're not
2932 * sharing this with the hdr. If we are sharing it with the hdr, the
2933 * hdr is responsible for doing the free.
2935 if (buf
->b_data
!= NULL
) {
2937 * We're about to change the hdr's b_flags. We must either
2938 * hold the hash_lock or be undiscoverable.
2940 ASSERT(HDR_EMPTY_OR_LOCKED(hdr
));
2942 arc_cksum_verify(buf
);
2943 arc_buf_unwatch(buf
);
2945 if (arc_buf_is_shared(buf
)) {
2946 arc_hdr_clear_flags(hdr
, ARC_FLAG_SHARED_DATA
);
2948 uint64_t size
= arc_buf_size(buf
);
2949 arc_free_data_buf(hdr
, buf
->b_data
, size
, buf
);
2950 ARCSTAT_INCR(arcstat_overhead_size
, -size
);
2954 ASSERT(hdr
->b_l1hdr
.b_bufcnt
> 0);
2955 hdr
->b_l1hdr
.b_bufcnt
-= 1;
2957 if (ARC_BUF_ENCRYPTED(buf
)) {
2958 hdr
->b_crypt_hdr
.b_ebufcnt
-= 1;
2961 * If we have no more encrypted buffers and we've
2962 * already gotten a copy of the decrypted data we can
2963 * free b_rabd to save some space.
2965 if (hdr
->b_crypt_hdr
.b_ebufcnt
== 0 &&
2966 HDR_HAS_RABD(hdr
) && hdr
->b_l1hdr
.b_pabd
!= NULL
&&
2967 !HDR_IO_IN_PROGRESS(hdr
)) {
2968 arc_hdr_free_pabd(hdr
, B_TRUE
);
2973 arc_buf_t
*lastbuf
= arc_buf_remove(hdr
, buf
);
2975 if (ARC_BUF_SHARED(buf
) && !ARC_BUF_COMPRESSED(buf
)) {
2977 * If the current arc_buf_t is sharing its data buffer with the
2978 * hdr, then reassign the hdr's b_pabd to share it with the new
2979 * buffer at the end of the list. The shared buffer is always
2980 * the last one on the hdr's buffer list.
2982 * There is an equivalent case for compressed bufs, but since
2983 * they aren't guaranteed to be the last buf in the list and
2984 * that is an exceedingly rare case, we just allow that space be
2985 * wasted temporarily. We must also be careful not to share
2986 * encrypted buffers, since they cannot be shared.
2988 if (lastbuf
!= NULL
&& !ARC_BUF_ENCRYPTED(lastbuf
)) {
2989 /* Only one buf can be shared at once */
2990 VERIFY(!arc_buf_is_shared(lastbuf
));
2991 /* hdr is uncompressed so can't have compressed buf */
2992 VERIFY(!ARC_BUF_COMPRESSED(lastbuf
));
2994 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, !=, NULL
);
2995 arc_hdr_free_pabd(hdr
, B_FALSE
);
2998 * We must setup a new shared block between the
2999 * last buffer and the hdr. The data would have
3000 * been allocated by the arc buf so we need to transfer
3001 * ownership to the hdr since it's now being shared.
3003 arc_share_buf(hdr
, lastbuf
);
3005 } else if (HDR_SHARED_DATA(hdr
)) {
3007 * Uncompressed shared buffers are always at the end
3008 * of the list. Compressed buffers don't have the
3009 * same requirements. This makes it hard to
3010 * simply assert that the lastbuf is shared so
3011 * we rely on the hdr's compression flags to determine
3012 * if we have a compressed, shared buffer.
3014 ASSERT3P(lastbuf
, !=, NULL
);
3015 ASSERT(arc_buf_is_shared(lastbuf
) ||
3016 arc_hdr_get_compress(hdr
) != ZIO_COMPRESS_OFF
);
3020 * Free the checksum if we're removing the last uncompressed buf from
3023 if (!arc_hdr_has_uncompressed_buf(hdr
)) {
3024 arc_cksum_free(hdr
);
3027 /* clean up the buf */
3029 kmem_cache_free(buf_cache
, buf
);
3033 arc_hdr_alloc_pabd(arc_buf_hdr_t
*hdr
, int alloc_flags
)
3036 boolean_t alloc_rdata
= ((alloc_flags
& ARC_HDR_ALLOC_RDATA
) != 0);
3037 boolean_t do_adapt
= ((alloc_flags
& ARC_HDR_DO_ADAPT
) != 0);
3039 ASSERT3U(HDR_GET_LSIZE(hdr
), >, 0);
3040 ASSERT(HDR_HAS_L1HDR(hdr
));
3041 ASSERT(!HDR_SHARED_DATA(hdr
) || alloc_rdata
);
3042 IMPLY(alloc_rdata
, HDR_PROTECTED(hdr
));
3045 size
= HDR_GET_PSIZE(hdr
);
3046 ASSERT3P(hdr
->b_crypt_hdr
.b_rabd
, ==, NULL
);
3047 hdr
->b_crypt_hdr
.b_rabd
= arc_get_data_abd(hdr
, size
, hdr
,
3049 ASSERT3P(hdr
->b_crypt_hdr
.b_rabd
, !=, NULL
);
3051 size
= arc_hdr_size(hdr
);
3052 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, ==, NULL
);
3053 hdr
->b_l1hdr
.b_pabd
= arc_get_data_abd(hdr
, size
, hdr
,
3055 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, !=, NULL
);
3058 ARCSTAT_INCR(arcstat_compressed_size
, size
);
3059 ARCSTAT_INCR(arcstat_uncompressed_size
, HDR_GET_LSIZE(hdr
));
3063 arc_hdr_free_pabd(arc_buf_hdr_t
*hdr
, boolean_t free_rdata
)
3065 uint64_t size
= (free_rdata
) ? HDR_GET_PSIZE(hdr
) : arc_hdr_size(hdr
);
3067 ASSERT(HDR_HAS_L1HDR(hdr
));
3068 ASSERT(hdr
->b_l1hdr
.b_pabd
!= NULL
|| HDR_HAS_RABD(hdr
));
3069 IMPLY(free_rdata
, HDR_HAS_RABD(hdr
));
3073 * If the hdr is currently being written to the l2arc then
3074 * we defer freeing the data by adding it to the l2arc_free_on_write
3075 * list. The l2arc will free the data once it's finished
3076 * writing it to the l2arc device.
3078 if (HDR_L2_WRITING(hdr
)) {
3079 arc_hdr_free_on_write(hdr
, free_rdata
);
3080 ARCSTAT_BUMP(arcstat_l2_free_on_write
);
3081 } else if (free_rdata
) {
3082 arc_free_data_abd(hdr
, hdr
->b_crypt_hdr
.b_rabd
, size
, hdr
);
3084 arc_free_data_abd(hdr
, hdr
->b_l1hdr
.b_pabd
,
3089 hdr
->b_crypt_hdr
.b_rabd
= NULL
;
3091 hdr
->b_l1hdr
.b_pabd
= NULL
;
3094 if (hdr
->b_l1hdr
.b_pabd
== NULL
&& !HDR_HAS_RABD(hdr
))
3095 hdr
->b_l1hdr
.b_byteswap
= DMU_BSWAP_NUMFUNCS
;
3097 ARCSTAT_INCR(arcstat_compressed_size
, -size
);
3098 ARCSTAT_INCR(arcstat_uncompressed_size
, -HDR_GET_LSIZE(hdr
));
3101 static arc_buf_hdr_t
*
3102 arc_hdr_alloc(uint64_t spa
, int32_t psize
, int32_t lsize
,
3103 boolean_t
protected, enum zio_compress compression_type
,
3104 arc_buf_contents_t type
, boolean_t alloc_rdata
)
3107 int flags
= ARC_HDR_DO_ADAPT
;
3109 VERIFY(type
== ARC_BUFC_DATA
|| type
== ARC_BUFC_METADATA
);
3111 hdr
= kmem_cache_alloc(hdr_full_crypt_cache
, KM_PUSHPAGE
);
3113 hdr
= kmem_cache_alloc(hdr_full_cache
, KM_PUSHPAGE
);
3115 flags
|= alloc_rdata
? ARC_HDR_ALLOC_RDATA
: 0;
3116 ASSERT(HDR_EMPTY(hdr
));
3117 ASSERT3P(hdr
->b_l1hdr
.b_freeze_cksum
, ==, NULL
);
3118 ASSERT3P(hdr
->b_l1hdr
.b_thawed
, ==, NULL
);
3119 HDR_SET_PSIZE(hdr
, psize
);
3120 HDR_SET_LSIZE(hdr
, lsize
);
3124 arc_hdr_set_flags(hdr
, arc_bufc_to_flags(type
) | ARC_FLAG_HAS_L1HDR
);
3125 arc_hdr_set_compress(hdr
, compression_type
);
3127 arc_hdr_set_flags(hdr
, ARC_FLAG_PROTECTED
);
3129 hdr
->b_l1hdr
.b_state
= arc_anon
;
3130 hdr
->b_l1hdr
.b_arc_access
= 0;
3131 hdr
->b_l1hdr
.b_bufcnt
= 0;
3132 hdr
->b_l1hdr
.b_buf
= NULL
;
3135 * Allocate the hdr's buffer. This will contain either
3136 * the compressed or uncompressed data depending on the block
3137 * it references and compressed arc enablement.
3139 arc_hdr_alloc_pabd(hdr
, flags
);
3140 ASSERT(zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
));
3146 * Transition between the two allocation states for the arc_buf_hdr struct.
3147 * The arc_buf_hdr struct can be allocated with (hdr_full_cache) or without
3148 * (hdr_l2only_cache) the fields necessary for the L1 cache - the smaller
3149 * version is used when a cache buffer is only in the L2ARC in order to reduce
3152 static arc_buf_hdr_t
*
3153 arc_hdr_realloc(arc_buf_hdr_t
*hdr
, kmem_cache_t
*old
, kmem_cache_t
*new)
3155 ASSERT(HDR_HAS_L2HDR(hdr
));
3157 arc_buf_hdr_t
*nhdr
;
3158 l2arc_dev_t
*dev
= hdr
->b_l2hdr
.b_dev
;
3160 ASSERT((old
== hdr_full_cache
&& new == hdr_l2only_cache
) ||
3161 (old
== hdr_l2only_cache
&& new == hdr_full_cache
));
3164 * if the caller wanted a new full header and the header is to be
3165 * encrypted we will actually allocate the header from the full crypt
3166 * cache instead. The same applies to freeing from the old cache.
3168 if (HDR_PROTECTED(hdr
) && new == hdr_full_cache
)
3169 new = hdr_full_crypt_cache
;
3170 if (HDR_PROTECTED(hdr
) && old
== hdr_full_cache
)
3171 old
= hdr_full_crypt_cache
;
3173 nhdr
= kmem_cache_alloc(new, KM_PUSHPAGE
);
3175 ASSERT(MUTEX_HELD(HDR_LOCK(hdr
)));
3176 buf_hash_remove(hdr
);
3178 bcopy(hdr
, nhdr
, HDR_L2ONLY_SIZE
);
3180 if (new == hdr_full_cache
|| new == hdr_full_crypt_cache
) {
3181 arc_hdr_set_flags(nhdr
, ARC_FLAG_HAS_L1HDR
);
3183 * arc_access and arc_change_state need to be aware that a
3184 * header has just come out of L2ARC, so we set its state to
3185 * l2c_only even though it's about to change.
3187 nhdr
->b_l1hdr
.b_state
= arc_l2c_only
;
3189 /* Verify previous threads set to NULL before freeing */
3190 ASSERT3P(nhdr
->b_l1hdr
.b_pabd
, ==, NULL
);
3191 ASSERT(!HDR_HAS_RABD(hdr
));
3193 ASSERT3P(hdr
->b_l1hdr
.b_buf
, ==, NULL
);
3194 ASSERT0(hdr
->b_l1hdr
.b_bufcnt
);
3195 ASSERT3P(hdr
->b_l1hdr
.b_freeze_cksum
, ==, NULL
);
3198 * If we've reached here, We must have been called from
3199 * arc_evict_hdr(), as such we should have already been
3200 * removed from any ghost list we were previously on
3201 * (which protects us from racing with arc_evict_state),
3202 * thus no locking is needed during this check.
3204 ASSERT(!multilist_link_active(&hdr
->b_l1hdr
.b_arc_node
));
3207 * A buffer must not be moved into the arc_l2c_only
3208 * state if it's not finished being written out to the
3209 * l2arc device. Otherwise, the b_l1hdr.b_pabd field
3210 * might try to be accessed, even though it was removed.
3212 VERIFY(!HDR_L2_WRITING(hdr
));
3213 VERIFY3P(hdr
->b_l1hdr
.b_pabd
, ==, NULL
);
3214 ASSERT(!HDR_HAS_RABD(hdr
));
3217 if (hdr
->b_l1hdr
.b_thawed
!= NULL
) {
3218 kmem_free(hdr
->b_l1hdr
.b_thawed
, 1);
3219 hdr
->b_l1hdr
.b_thawed
= NULL
;
3223 arc_hdr_clear_flags(nhdr
, ARC_FLAG_HAS_L1HDR
);
3226 * The header has been reallocated so we need to re-insert it into any
3229 (void) buf_hash_insert(nhdr
, NULL
);
3231 ASSERT(list_link_active(&hdr
->b_l2hdr
.b_l2node
));
3233 mutex_enter(&dev
->l2ad_mtx
);
3236 * We must place the realloc'ed header back into the list at
3237 * the same spot. Otherwise, if it's placed earlier in the list,
3238 * l2arc_write_buffers() could find it during the function's
3239 * write phase, and try to write it out to the l2arc.
3241 list_insert_after(&dev
->l2ad_buflist
, hdr
, nhdr
);
3242 list_remove(&dev
->l2ad_buflist
, hdr
);
3244 mutex_exit(&dev
->l2ad_mtx
);
3247 * Since we're using the pointer address as the tag when
3248 * incrementing and decrementing the l2ad_alloc refcount, we
3249 * must remove the old pointer (that we're about to destroy) and
3250 * add the new pointer to the refcount. Otherwise we'd remove
3251 * the wrong pointer address when calling arc_hdr_destroy() later.
3254 (void) zfs_refcount_remove_many(&dev
->l2ad_alloc
, arc_hdr_size(hdr
),
3256 (void) zfs_refcount_add_many(&dev
->l2ad_alloc
, arc_hdr_size(nhdr
),
3259 buf_discard_identity(hdr
);
3260 kmem_cache_free(old
, hdr
);
3266 * This function allows an L1 header to be reallocated as a crypt
3267 * header and vice versa. If we are going to a crypt header, the
3268 * new fields will be zeroed out.
3270 static arc_buf_hdr_t
*
3271 arc_hdr_realloc_crypt(arc_buf_hdr_t
*hdr
, boolean_t need_crypt
)
3273 arc_buf_hdr_t
*nhdr
;
3275 kmem_cache_t
*ncache
, *ocache
;
3278 * This function requires that hdr is in the arc_anon state.
3279 * Therefore it won't have any L2ARC data for us to worry about
3282 ASSERT(HDR_HAS_L1HDR(hdr
));
3283 ASSERT(!HDR_HAS_L2HDR(hdr
));
3284 ASSERT3U(!!HDR_PROTECTED(hdr
), !=, need_crypt
);
3285 ASSERT3P(hdr
->b_l1hdr
.b_state
, ==, arc_anon
);
3286 ASSERT(!multilist_link_active(&hdr
->b_l1hdr
.b_arc_node
));
3287 ASSERT(!list_link_active(&hdr
->b_l2hdr
.b_l2node
));
3288 ASSERT3P(hdr
->b_hash_next
, ==, NULL
);
3291 ncache
= hdr_full_crypt_cache
;
3292 ocache
= hdr_full_cache
;
3294 ncache
= hdr_full_cache
;
3295 ocache
= hdr_full_crypt_cache
;
3298 nhdr
= kmem_cache_alloc(ncache
, KM_PUSHPAGE
);
3301 * Copy all members that aren't locks or condvars to the new header.
3302 * No lists are pointing to us (as we asserted above), so we don't
3303 * need to worry about the list nodes.
3305 nhdr
->b_dva
= hdr
->b_dva
;
3306 nhdr
->b_birth
= hdr
->b_birth
;
3307 nhdr
->b_type
= hdr
->b_type
;
3308 nhdr
->b_flags
= hdr
->b_flags
;
3309 nhdr
->b_psize
= hdr
->b_psize
;
3310 nhdr
->b_lsize
= hdr
->b_lsize
;
3311 nhdr
->b_spa
= hdr
->b_spa
;
3312 nhdr
->b_l2hdr
.b_dev
= hdr
->b_l2hdr
.b_dev
;
3313 nhdr
->b_l2hdr
.b_daddr
= hdr
->b_l2hdr
.b_daddr
;
3314 nhdr
->b_l1hdr
.b_freeze_cksum
= hdr
->b_l1hdr
.b_freeze_cksum
;
3315 nhdr
->b_l1hdr
.b_bufcnt
= hdr
->b_l1hdr
.b_bufcnt
;
3316 nhdr
->b_l1hdr
.b_byteswap
= hdr
->b_l1hdr
.b_byteswap
;
3317 nhdr
->b_l1hdr
.b_state
= hdr
->b_l1hdr
.b_state
;
3318 nhdr
->b_l1hdr
.b_arc_access
= hdr
->b_l1hdr
.b_arc_access
;
3319 nhdr
->b_l1hdr
.b_acb
= hdr
->b_l1hdr
.b_acb
;
3320 nhdr
->b_l1hdr
.b_pabd
= hdr
->b_l1hdr
.b_pabd
;
3322 if (hdr
->b_l1hdr
.b_thawed
!= NULL
) {
3323 nhdr
->b_l1hdr
.b_thawed
= hdr
->b_l1hdr
.b_thawed
;
3324 hdr
->b_l1hdr
.b_thawed
= NULL
;
3329 * This refcount_add() exists only to ensure that the individual
3330 * arc buffers always point to a header that is referenced, avoiding
3331 * a small race condition that could trigger ASSERTs.
3333 (void) zfs_refcount_add(&nhdr
->b_l1hdr
.b_refcnt
, FTAG
);
3334 nhdr
->b_l1hdr
.b_buf
= hdr
->b_l1hdr
.b_buf
;
3335 for (buf
= nhdr
->b_l1hdr
.b_buf
; buf
!= NULL
; buf
= buf
->b_next
) {
3336 mutex_enter(&buf
->b_evict_lock
);
3338 mutex_exit(&buf
->b_evict_lock
);
3340 zfs_refcount_transfer(&nhdr
->b_l1hdr
.b_refcnt
, &hdr
->b_l1hdr
.b_refcnt
);
3341 (void) zfs_refcount_remove(&nhdr
->b_l1hdr
.b_refcnt
, FTAG
);
3342 ASSERT0(zfs_refcount_count(&hdr
->b_l1hdr
.b_refcnt
));
3345 * We have already asserted that we are not on any ghost lists, and we
3346 * are never called to switch from a crypt to non-crypt header
3347 * with a non-NULL rabd (this is asserted below).
3348 * This leaves the hdr's b_pabd buffer to deal with.
3350 if (hdr
->b_l1hdr
.b_pabd
!= NULL
) {
3351 zfs_refcount_transfer_ownership_many(
3352 &hdr
->b_l1hdr
.b_state
->arcs_size
, arc_hdr_size(hdr
),
3357 arc_hdr_set_flags(nhdr
, ARC_FLAG_PROTECTED
);
3359 arc_hdr_clear_flags(nhdr
, ARC_FLAG_PROTECTED
);
3362 /* unset all members of the original hdr */
3363 bzero(&hdr
->b_dva
, sizeof (dva_t
));
3365 hdr
->b_type
= ARC_BUFC_INVALID
;
3370 hdr
->b_l2hdr
.b_dev
= NULL
;
3371 hdr
->b_l2hdr
.b_daddr
= 0;
3372 hdr
->b_l1hdr
.b_freeze_cksum
= NULL
;
3373 hdr
->b_l1hdr
.b_buf
= NULL
;
3374 hdr
->b_l1hdr
.b_bufcnt
= 0;
3375 hdr
->b_l1hdr
.b_byteswap
= 0;
3376 hdr
->b_l1hdr
.b_state
= NULL
;
3377 hdr
->b_l1hdr
.b_arc_access
= 0;
3378 hdr
->b_l1hdr
.b_acb
= NULL
;
3379 hdr
->b_l1hdr
.b_pabd
= NULL
;
3381 if (ocache
== hdr_full_crypt_cache
) {
3382 ASSERT(!HDR_HAS_RABD(hdr
));
3383 hdr
->b_crypt_hdr
.b_ot
= DMU_OT_NONE
;
3384 hdr
->b_crypt_hdr
.b_ebufcnt
= 0;
3385 hdr
->b_crypt_hdr
.b_dsobj
= 0;
3386 bzero(hdr
->b_crypt_hdr
.b_salt
, ZIO_DATA_SALT_LEN
);
3387 bzero(hdr
->b_crypt_hdr
.b_iv
, ZIO_DATA_IV_LEN
);
3388 bzero(hdr
->b_crypt_hdr
.b_mac
, ZIO_DATA_MAC_LEN
);
3391 buf_discard_identity(hdr
);
3392 kmem_cache_free(ocache
, hdr
);
3398 * This function is used by the send / receive code to convert a newly
3399 * allocated arc_buf_t to one that is suitable for a raw encrypted write. It
3400 * is also used to allow the root objset block to be uupdated without altering
3401 * its embedded MACs. Both block types will always be uncompressed so we do not
3402 * have to worry about compression type or psize.
3405 arc_convert_to_raw(arc_buf_t
*buf
, uint64_t dsobj
, boolean_t byteorder
,
3406 dmu_object_type_t ot
, const uint8_t *salt
, const uint8_t *iv
,
3409 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
3411 ASSERT(ot
== DMU_OT_DNODE
|| ot
== DMU_OT_OBJSET
);
3412 ASSERT(HDR_HAS_L1HDR(hdr
));
3413 ASSERT3P(hdr
->b_l1hdr
.b_state
, ==, arc_anon
);
3415 buf
->b_flags
|= (ARC_BUF_FLAG_COMPRESSED
| ARC_BUF_FLAG_ENCRYPTED
);
3416 if (!HDR_PROTECTED(hdr
))
3417 hdr
= arc_hdr_realloc_crypt(hdr
, B_TRUE
);
3418 hdr
->b_crypt_hdr
.b_dsobj
= dsobj
;
3419 hdr
->b_crypt_hdr
.b_ot
= ot
;
3420 hdr
->b_l1hdr
.b_byteswap
= (byteorder
== ZFS_HOST_BYTEORDER
) ?
3421 DMU_BSWAP_NUMFUNCS
: DMU_OT_BYTESWAP(ot
);
3422 if (!arc_hdr_has_uncompressed_buf(hdr
))
3423 arc_cksum_free(hdr
);
3426 bcopy(salt
, hdr
->b_crypt_hdr
.b_salt
, ZIO_DATA_SALT_LEN
);
3428 bcopy(iv
, hdr
->b_crypt_hdr
.b_iv
, ZIO_DATA_IV_LEN
);
3430 bcopy(mac
, hdr
->b_crypt_hdr
.b_mac
, ZIO_DATA_MAC_LEN
);
3434 * Allocate a new arc_buf_hdr_t and arc_buf_t and return the buf to the caller.
3435 * The buf is returned thawed since we expect the consumer to modify it.
3438 arc_alloc_buf(spa_t
*spa
, void *tag
, arc_buf_contents_t type
, int32_t size
)
3440 arc_buf_hdr_t
*hdr
= arc_hdr_alloc(spa_load_guid(spa
), size
, size
,
3441 B_FALSE
, ZIO_COMPRESS_OFF
, type
, B_FALSE
);
3443 arc_buf_t
*buf
= NULL
;
3444 VERIFY0(arc_buf_alloc_impl(hdr
, spa
, NULL
, tag
, B_FALSE
, B_FALSE
,
3445 B_FALSE
, B_FALSE
, &buf
));
3452 * Allocates an ARC buf header that's in an evicted & L2-cached state.
3453 * This is used during l2arc reconstruction to make empty ARC buffers
3454 * which circumvent the regular disk->arc->l2arc path and instead come
3455 * into being in the reverse order, i.e. l2arc->arc.
3458 arc_buf_alloc_l2only(size_t size
, arc_buf_contents_t type
, l2arc_dev_t
*dev
,
3459 dva_t dva
, uint64_t daddr
, int32_t psize
, uint64_t birth
,
3460 enum zio_compress compress
, boolean_t
protected,
3461 boolean_t prefetch
, arc_state_type_t arcs_state
)
3466 hdr
= kmem_cache_alloc(hdr_l2only_cache
, KM_SLEEP
);
3467 hdr
->b_birth
= birth
;
3470 arc_hdr_set_flags(hdr
, arc_bufc_to_flags(type
) | ARC_FLAG_HAS_L2HDR
);
3471 HDR_SET_LSIZE(hdr
, size
);
3472 HDR_SET_PSIZE(hdr
, psize
);
3473 arc_hdr_set_compress(hdr
, compress
);
3475 arc_hdr_set_flags(hdr
, ARC_FLAG_PROTECTED
);
3477 arc_hdr_set_flags(hdr
, ARC_FLAG_PREFETCH
);
3478 hdr
->b_spa
= spa_load_guid(dev
->l2ad_vdev
->vdev_spa
);
3482 hdr
->b_l2hdr
.b_dev
= dev
;
3483 hdr
->b_l2hdr
.b_daddr
= daddr
;
3484 hdr
->b_l2hdr
.b_arcs_state
= arcs_state
;
3490 * Allocate a compressed buf in the same manner as arc_alloc_buf. Don't use this
3491 * for bufs containing metadata.
3494 arc_alloc_compressed_buf(spa_t
*spa
, void *tag
, uint64_t psize
, uint64_t lsize
,
3495 enum zio_compress compression_type
)
3497 ASSERT3U(lsize
, >, 0);
3498 ASSERT3U(lsize
, >=, psize
);
3499 ASSERT3U(compression_type
, >, ZIO_COMPRESS_OFF
);
3500 ASSERT3U(compression_type
, <, ZIO_COMPRESS_FUNCTIONS
);
3502 arc_buf_hdr_t
*hdr
= arc_hdr_alloc(spa_load_guid(spa
), psize
, lsize
,
3503 B_FALSE
, compression_type
, ARC_BUFC_DATA
, B_FALSE
);
3505 arc_buf_t
*buf
= NULL
;
3506 VERIFY0(arc_buf_alloc_impl(hdr
, spa
, NULL
, tag
, B_FALSE
,
3507 B_TRUE
, B_FALSE
, B_FALSE
, &buf
));
3509 ASSERT3P(hdr
->b_l1hdr
.b_freeze_cksum
, ==, NULL
);
3511 if (!arc_buf_is_shared(buf
)) {
3513 * To ensure that the hdr has the correct data in it if we call
3514 * arc_untransform() on this buf before it's been written to
3515 * disk, it's easiest if we just set up sharing between the
3518 ASSERT(!abd_is_linear(hdr
->b_l1hdr
.b_pabd
));
3519 arc_hdr_free_pabd(hdr
, B_FALSE
);
3520 arc_share_buf(hdr
, buf
);
3527 arc_alloc_raw_buf(spa_t
*spa
, void *tag
, uint64_t dsobj
, boolean_t byteorder
,
3528 const uint8_t *salt
, const uint8_t *iv
, const uint8_t *mac
,
3529 dmu_object_type_t ot
, uint64_t psize
, uint64_t lsize
,
3530 enum zio_compress compression_type
)
3534 arc_buf_contents_t type
= DMU_OT_IS_METADATA(ot
) ?
3535 ARC_BUFC_METADATA
: ARC_BUFC_DATA
;
3537 ASSERT3U(lsize
, >, 0);
3538 ASSERT3U(lsize
, >=, psize
);
3539 ASSERT3U(compression_type
, >=, ZIO_COMPRESS_OFF
);
3540 ASSERT3U(compression_type
, <, ZIO_COMPRESS_FUNCTIONS
);
3542 hdr
= arc_hdr_alloc(spa_load_guid(spa
), psize
, lsize
, B_TRUE
,
3543 compression_type
, type
, B_TRUE
);
3545 hdr
->b_crypt_hdr
.b_dsobj
= dsobj
;
3546 hdr
->b_crypt_hdr
.b_ot
= ot
;
3547 hdr
->b_l1hdr
.b_byteswap
= (byteorder
== ZFS_HOST_BYTEORDER
) ?
3548 DMU_BSWAP_NUMFUNCS
: DMU_OT_BYTESWAP(ot
);
3549 bcopy(salt
, hdr
->b_crypt_hdr
.b_salt
, ZIO_DATA_SALT_LEN
);
3550 bcopy(iv
, hdr
->b_crypt_hdr
.b_iv
, ZIO_DATA_IV_LEN
);
3551 bcopy(mac
, hdr
->b_crypt_hdr
.b_mac
, ZIO_DATA_MAC_LEN
);
3554 * This buffer will be considered encrypted even if the ot is not an
3555 * encrypted type. It will become authenticated instead in
3556 * arc_write_ready().
3559 VERIFY0(arc_buf_alloc_impl(hdr
, spa
, NULL
, tag
, B_TRUE
, B_TRUE
,
3560 B_FALSE
, B_FALSE
, &buf
));
3562 ASSERT3P(hdr
->b_l1hdr
.b_freeze_cksum
, ==, NULL
);
3568 l2arc_hdr_arcstats_update(arc_buf_hdr_t
*hdr
, boolean_t incr
,
3569 boolean_t state_only
)
3571 l2arc_buf_hdr_t
*l2hdr
= &hdr
->b_l2hdr
;
3572 l2arc_dev_t
*dev
= l2hdr
->b_dev
;
3573 uint64_t lsize
= HDR_GET_LSIZE(hdr
);
3574 uint64_t psize
= HDR_GET_PSIZE(hdr
);
3575 uint64_t asize
= vdev_psize_to_asize(dev
->l2ad_vdev
, psize
);
3576 arc_buf_contents_t type
= hdr
->b_type
;
3591 /* If the buffer is a prefetch, count it as such. */
3592 if (HDR_PREFETCH(hdr
)) {
3593 ARCSTAT_INCR(arcstat_l2_prefetch_asize
, asize_s
);
3596 * We use the value stored in the L2 header upon initial
3597 * caching in L2ARC. This value will be updated in case
3598 * an MRU/MRU_ghost buffer transitions to MFU but the L2ARC
3599 * metadata (log entry) cannot currently be updated. Having
3600 * the ARC state in the L2 header solves the problem of a
3601 * possibly absent L1 header (apparent in buffers restored
3602 * from persistent L2ARC).
3604 switch (hdr
->b_l2hdr
.b_arcs_state
) {
3605 case ARC_STATE_MRU_GHOST
:
3607 ARCSTAT_INCR(arcstat_l2_mru_asize
, asize_s
);
3609 case ARC_STATE_MFU_GHOST
:
3611 ARCSTAT_INCR(arcstat_l2_mfu_asize
, asize_s
);
3621 ARCSTAT_INCR(arcstat_l2_psize
, psize_s
);
3622 ARCSTAT_INCR(arcstat_l2_lsize
, lsize_s
);
3626 ARCSTAT_INCR(arcstat_l2_bufc_data_asize
, asize_s
);
3628 case ARC_BUFC_METADATA
:
3629 ARCSTAT_INCR(arcstat_l2_bufc_metadata_asize
, asize_s
);
3638 arc_hdr_l2hdr_destroy(arc_buf_hdr_t
*hdr
)
3640 l2arc_buf_hdr_t
*l2hdr
= &hdr
->b_l2hdr
;
3641 l2arc_dev_t
*dev
= l2hdr
->b_dev
;
3642 uint64_t psize
= HDR_GET_PSIZE(hdr
);
3643 uint64_t asize
= vdev_psize_to_asize(dev
->l2ad_vdev
, psize
);
3645 ASSERT(MUTEX_HELD(&dev
->l2ad_mtx
));
3646 ASSERT(HDR_HAS_L2HDR(hdr
));
3648 list_remove(&dev
->l2ad_buflist
, hdr
);
3650 l2arc_hdr_arcstats_decrement(hdr
);
3651 vdev_space_update(dev
->l2ad_vdev
, -asize
, 0, 0);
3653 (void) zfs_refcount_remove_many(&dev
->l2ad_alloc
, arc_hdr_size(hdr
),
3655 arc_hdr_clear_flags(hdr
, ARC_FLAG_HAS_L2HDR
);
3659 arc_hdr_destroy(arc_buf_hdr_t
*hdr
)
3661 if (HDR_HAS_L1HDR(hdr
)) {
3662 ASSERT(hdr
->b_l1hdr
.b_buf
== NULL
||
3663 hdr
->b_l1hdr
.b_bufcnt
> 0);
3664 ASSERT(zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
));
3665 ASSERT3P(hdr
->b_l1hdr
.b_state
, ==, arc_anon
);
3667 ASSERT(!HDR_IO_IN_PROGRESS(hdr
));
3668 ASSERT(!HDR_IN_HASH_TABLE(hdr
));
3670 if (HDR_HAS_L2HDR(hdr
)) {
3671 l2arc_dev_t
*dev
= hdr
->b_l2hdr
.b_dev
;
3672 boolean_t buflist_held
= MUTEX_HELD(&dev
->l2ad_mtx
);
3675 mutex_enter(&dev
->l2ad_mtx
);
3678 * Even though we checked this conditional above, we
3679 * need to check this again now that we have the
3680 * l2ad_mtx. This is because we could be racing with
3681 * another thread calling l2arc_evict() which might have
3682 * destroyed this header's L2 portion as we were waiting
3683 * to acquire the l2ad_mtx. If that happens, we don't
3684 * want to re-destroy the header's L2 portion.
3686 if (HDR_HAS_L2HDR(hdr
))
3687 arc_hdr_l2hdr_destroy(hdr
);
3690 mutex_exit(&dev
->l2ad_mtx
);
3694 * The header's identity can only be safely discarded once it is no
3695 * longer discoverable. This requires removing it from the hash table
3696 * and the l2arc header list. After this point the hash lock can not
3697 * be used to protect the header.
3699 if (!HDR_EMPTY(hdr
))
3700 buf_discard_identity(hdr
);
3702 if (HDR_HAS_L1HDR(hdr
)) {
3703 arc_cksum_free(hdr
);
3705 while (hdr
->b_l1hdr
.b_buf
!= NULL
)
3706 arc_buf_destroy_impl(hdr
->b_l1hdr
.b_buf
);
3709 if (hdr
->b_l1hdr
.b_thawed
!= NULL
) {
3710 kmem_free(hdr
->b_l1hdr
.b_thawed
, 1);
3711 hdr
->b_l1hdr
.b_thawed
= NULL
;
3715 if (hdr
->b_l1hdr
.b_pabd
!= NULL
)
3716 arc_hdr_free_pabd(hdr
, B_FALSE
);
3718 if (HDR_HAS_RABD(hdr
))
3719 arc_hdr_free_pabd(hdr
, B_TRUE
);
3722 ASSERT3P(hdr
->b_hash_next
, ==, NULL
);
3723 if (HDR_HAS_L1HDR(hdr
)) {
3724 ASSERT(!multilist_link_active(&hdr
->b_l1hdr
.b_arc_node
));
3725 ASSERT3P(hdr
->b_l1hdr
.b_acb
, ==, NULL
);
3727 if (!HDR_PROTECTED(hdr
)) {
3728 kmem_cache_free(hdr_full_cache
, hdr
);
3730 kmem_cache_free(hdr_full_crypt_cache
, hdr
);
3733 kmem_cache_free(hdr_l2only_cache
, hdr
);
3738 arc_buf_destroy(arc_buf_t
*buf
, void* tag
)
3740 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
3742 if (hdr
->b_l1hdr
.b_state
== arc_anon
) {
3743 ASSERT3U(hdr
->b_l1hdr
.b_bufcnt
, ==, 1);
3744 ASSERT(!HDR_IO_IN_PROGRESS(hdr
));
3745 VERIFY0(remove_reference(hdr
, NULL
, tag
));
3746 arc_hdr_destroy(hdr
);
3750 kmutex_t
*hash_lock
= HDR_LOCK(hdr
);
3751 mutex_enter(hash_lock
);
3753 ASSERT3P(hdr
, ==, buf
->b_hdr
);
3754 ASSERT(hdr
->b_l1hdr
.b_bufcnt
> 0);
3755 ASSERT3P(hash_lock
, ==, HDR_LOCK(hdr
));
3756 ASSERT3P(hdr
->b_l1hdr
.b_state
, !=, arc_anon
);
3757 ASSERT3P(buf
->b_data
, !=, NULL
);
3759 (void) remove_reference(hdr
, hash_lock
, tag
);
3760 arc_buf_destroy_impl(buf
);
3761 mutex_exit(hash_lock
);
3765 * Evict the arc_buf_hdr that is provided as a parameter. The resultant
3766 * state of the header is dependent on its state prior to entering this
3767 * function. The following transitions are possible:
3769 * - arc_mru -> arc_mru_ghost
3770 * - arc_mfu -> arc_mfu_ghost
3771 * - arc_mru_ghost -> arc_l2c_only
3772 * - arc_mru_ghost -> deleted
3773 * - arc_mfu_ghost -> arc_l2c_only
3774 * - arc_mfu_ghost -> deleted
3777 arc_evict_hdr(arc_buf_hdr_t
*hdr
, kmutex_t
*hash_lock
)
3779 arc_state_t
*evicted_state
, *state
;
3780 int64_t bytes_evicted
= 0;
3781 int min_lifetime
= HDR_PRESCIENT_PREFETCH(hdr
) ?
3782 zfs_arc_min_prescient_prefetch_ms
: zfs_arc_min_prefetch_ms
;
3784 ASSERT(MUTEX_HELD(hash_lock
));
3785 ASSERT(HDR_HAS_L1HDR(hdr
));
3787 state
= hdr
->b_l1hdr
.b_state
;
3788 if (GHOST_STATE(state
)) {
3789 ASSERT(!HDR_IO_IN_PROGRESS(hdr
));
3790 ASSERT3P(hdr
->b_l1hdr
.b_buf
, ==, NULL
);
3793 * l2arc_write_buffers() relies on a header's L1 portion
3794 * (i.e. its b_pabd field) during its write phase.
3795 * Thus, we cannot push a header onto the arc_l2c_only
3796 * state (removing its L1 piece) until the header is
3797 * done being written to the l2arc.
3799 if (HDR_HAS_L2HDR(hdr
) && HDR_L2_WRITING(hdr
)) {
3800 ARCSTAT_BUMP(arcstat_evict_l2_skip
);
3801 return (bytes_evicted
);
3804 ARCSTAT_BUMP(arcstat_deleted
);
3805 bytes_evicted
+= HDR_GET_LSIZE(hdr
);
3807 DTRACE_PROBE1(arc__delete
, arc_buf_hdr_t
*, hdr
);
3809 if (HDR_HAS_L2HDR(hdr
)) {
3810 ASSERT(hdr
->b_l1hdr
.b_pabd
== NULL
);
3811 ASSERT(!HDR_HAS_RABD(hdr
));
3813 * This buffer is cached on the 2nd Level ARC;
3814 * don't destroy the header.
3816 arc_change_state(arc_l2c_only
, hdr
, hash_lock
);
3818 * dropping from L1+L2 cached to L2-only,
3819 * realloc to remove the L1 header.
3821 hdr
= arc_hdr_realloc(hdr
, hdr_full_cache
,
3824 arc_change_state(arc_anon
, hdr
, hash_lock
);
3825 arc_hdr_destroy(hdr
);
3827 return (bytes_evicted
);
3830 ASSERT(state
== arc_mru
|| state
== arc_mfu
);
3831 evicted_state
= (state
== arc_mru
) ? arc_mru_ghost
: arc_mfu_ghost
;
3833 /* prefetch buffers have a minimum lifespan */
3834 if (HDR_IO_IN_PROGRESS(hdr
) ||
3835 ((hdr
->b_flags
& (ARC_FLAG_PREFETCH
| ARC_FLAG_INDIRECT
)) &&
3836 ddi_get_lbolt() - hdr
->b_l1hdr
.b_arc_access
< min_lifetime
* hz
)) {
3837 ARCSTAT_BUMP(arcstat_evict_skip
);
3838 return (bytes_evicted
);
3841 ASSERT0(zfs_refcount_count(&hdr
->b_l1hdr
.b_refcnt
));
3842 while (hdr
->b_l1hdr
.b_buf
) {
3843 arc_buf_t
*buf
= hdr
->b_l1hdr
.b_buf
;
3844 if (!mutex_tryenter(&buf
->b_evict_lock
)) {
3845 ARCSTAT_BUMP(arcstat_mutex_miss
);
3848 if (buf
->b_data
!= NULL
)
3849 bytes_evicted
+= HDR_GET_LSIZE(hdr
);
3850 mutex_exit(&buf
->b_evict_lock
);
3851 arc_buf_destroy_impl(buf
);
3854 if (HDR_HAS_L2HDR(hdr
)) {
3855 ARCSTAT_INCR(arcstat_evict_l2_cached
, HDR_GET_LSIZE(hdr
));
3857 if (l2arc_write_eligible(hdr
->b_spa
, hdr
)) {
3858 ARCSTAT_INCR(arcstat_evict_l2_eligible
,
3859 HDR_GET_LSIZE(hdr
));
3861 switch (state
->arcs_state
) {
3864 arcstat_evict_l2_eligible_mru
,
3865 HDR_GET_LSIZE(hdr
));
3869 arcstat_evict_l2_eligible_mfu
,
3870 HDR_GET_LSIZE(hdr
));
3876 ARCSTAT_INCR(arcstat_evict_l2_ineligible
,
3877 HDR_GET_LSIZE(hdr
));
3881 if (hdr
->b_l1hdr
.b_bufcnt
== 0) {
3882 arc_cksum_free(hdr
);
3884 bytes_evicted
+= arc_hdr_size(hdr
);
3887 * If this hdr is being evicted and has a compressed
3888 * buffer then we discard it here before we change states.
3889 * This ensures that the accounting is updated correctly
3890 * in arc_free_data_impl().
3892 if (hdr
->b_l1hdr
.b_pabd
!= NULL
)
3893 arc_hdr_free_pabd(hdr
, B_FALSE
);
3895 if (HDR_HAS_RABD(hdr
))
3896 arc_hdr_free_pabd(hdr
, B_TRUE
);
3898 arc_change_state(evicted_state
, hdr
, hash_lock
);
3899 ASSERT(HDR_IN_HASH_TABLE(hdr
));
3900 arc_hdr_set_flags(hdr
, ARC_FLAG_IN_HASH_TABLE
);
3901 DTRACE_PROBE1(arc__evict
, arc_buf_hdr_t
*, hdr
);
3904 return (bytes_evicted
);
3908 arc_evict_state_impl(multilist_t
*ml
, int idx
, arc_buf_hdr_t
*marker
,
3909 uint64_t spa
, int64_t bytes
)
3911 multilist_sublist_t
*mls
;
3912 uint64_t bytes_evicted
= 0;
3914 kmutex_t
*hash_lock
;
3915 int evict_count
= 0;
3917 ASSERT3P(marker
, !=, NULL
);
3918 IMPLY(bytes
< 0, bytes
== ARC_EVICT_ALL
);
3920 mls
= multilist_sublist_lock(ml
, idx
);
3922 for (hdr
= multilist_sublist_prev(mls
, marker
); hdr
!= NULL
;
3923 hdr
= multilist_sublist_prev(mls
, marker
)) {
3924 if ((bytes
!= ARC_EVICT_ALL
&& bytes_evicted
>= bytes
) ||
3925 (evict_count
>= zfs_arc_evict_batch_limit
))
3929 * To keep our iteration location, move the marker
3930 * forward. Since we're not holding hdr's hash lock, we
3931 * must be very careful and not remove 'hdr' from the
3932 * sublist. Otherwise, other consumers might mistake the
3933 * 'hdr' as not being on a sublist when they call the
3934 * multilist_link_active() function (they all rely on
3935 * the hash lock protecting concurrent insertions and
3936 * removals). multilist_sublist_move_forward() was
3937 * specifically implemented to ensure this is the case
3938 * (only 'marker' will be removed and re-inserted).
3940 multilist_sublist_move_forward(mls
, marker
);
3943 * The only case where the b_spa field should ever be
3944 * zero, is the marker headers inserted by
3945 * arc_evict_state(). It's possible for multiple threads
3946 * to be calling arc_evict_state() concurrently (e.g.
3947 * dsl_pool_close() and zio_inject_fault()), so we must
3948 * skip any markers we see from these other threads.
3950 if (hdr
->b_spa
== 0)
3953 /* we're only interested in evicting buffers of a certain spa */
3954 if (spa
!= 0 && hdr
->b_spa
!= spa
) {
3955 ARCSTAT_BUMP(arcstat_evict_skip
);
3959 hash_lock
= HDR_LOCK(hdr
);
3962 * We aren't calling this function from any code path
3963 * that would already be holding a hash lock, so we're
3964 * asserting on this assumption to be defensive in case
3965 * this ever changes. Without this check, it would be
3966 * possible to incorrectly increment arcstat_mutex_miss
3967 * below (e.g. if the code changed such that we called
3968 * this function with a hash lock held).
3970 ASSERT(!MUTEX_HELD(hash_lock
));
3972 if (mutex_tryenter(hash_lock
)) {
3973 uint64_t evicted
= arc_evict_hdr(hdr
, hash_lock
);
3974 mutex_exit(hash_lock
);
3976 bytes_evicted
+= evicted
;
3979 * If evicted is zero, arc_evict_hdr() must have
3980 * decided to skip this header, don't increment
3981 * evict_count in this case.
3987 * If arc_size isn't overflowing, signal any
3988 * threads that might happen to be waiting.
3990 * For each header evicted, we wake up a single
3991 * thread. If we used cv_broadcast, we could
3992 * wake up "too many" threads causing arc_size
3993 * to significantly overflow arc_c; since
3994 * arc_get_data_impl() doesn't check for overflow
3995 * when it's woken up (it doesn't because it's
3996 * possible for the ARC to be overflowing while
3997 * full of un-evictable buffers, and the
3998 * function should proceed in this case).
4000 * If threads are left sleeping, due to not
4001 * using cv_broadcast here, they will be woken
4002 * up via cv_broadcast in arc_adjust_cb() just
4003 * before arc_adjust_zthr sleeps.
4005 mutex_enter(&arc_adjust_lock
);
4006 if (!arc_is_overflowing())
4007 cv_signal(&arc_adjust_waiters_cv
);
4008 mutex_exit(&arc_adjust_lock
);
4010 ARCSTAT_BUMP(arcstat_mutex_miss
);
4014 multilist_sublist_unlock(mls
);
4016 return (bytes_evicted
);
4020 * Evict buffers from the given arc state, until we've removed the
4021 * specified number of bytes. Move the removed buffers to the
4022 * appropriate evict state.
4024 * This function makes a "best effort". It skips over any buffers
4025 * it can't get a hash_lock on, and so, may not catch all candidates.
4026 * It may also return without evicting as much space as requested.
4028 * If bytes is specified using the special value ARC_EVICT_ALL, this
4029 * will evict all available (i.e. unlocked and evictable) buffers from
4030 * the given arc state; which is used by arc_flush().
4033 arc_evict_state(arc_state_t
*state
, uint64_t spa
, int64_t bytes
,
4034 arc_buf_contents_t type
)
4036 uint64_t total_evicted
= 0;
4037 multilist_t
*ml
= state
->arcs_list
[type
];
4039 arc_buf_hdr_t
**markers
;
4041 IMPLY(bytes
< 0, bytes
== ARC_EVICT_ALL
);
4043 num_sublists
= multilist_get_num_sublists(ml
);
4046 * If we've tried to evict from each sublist, made some
4047 * progress, but still have not hit the target number of bytes
4048 * to evict, we want to keep trying. The markers allow us to
4049 * pick up where we left off for each individual sublist, rather
4050 * than starting from the tail each time.
4052 markers
= kmem_zalloc(sizeof (*markers
) * num_sublists
, KM_SLEEP
);
4053 for (int i
= 0; i
< num_sublists
; i
++) {
4054 markers
[i
] = kmem_cache_alloc(hdr_full_cache
, KM_SLEEP
);
4057 * A b_spa of 0 is used to indicate that this header is
4058 * a marker. This fact is used in arc_adjust_type() and
4059 * arc_evict_state_impl().
4061 markers
[i
]->b_spa
= 0;
4063 multilist_sublist_t
*mls
= multilist_sublist_lock(ml
, i
);
4064 multilist_sublist_insert_tail(mls
, markers
[i
]);
4065 multilist_sublist_unlock(mls
);
4069 * While we haven't hit our target number of bytes to evict, or
4070 * we're evicting all available buffers.
4072 while (total_evicted
< bytes
|| bytes
== ARC_EVICT_ALL
) {
4074 * Start eviction using a randomly selected sublist,
4075 * this is to try and evenly balance eviction across all
4076 * sublists. Always starting at the same sublist
4077 * (e.g. index 0) would cause evictions to favor certain
4078 * sublists over others.
4080 int sublist_idx
= multilist_get_random_index(ml
);
4081 uint64_t scan_evicted
= 0;
4083 for (int i
= 0; i
< num_sublists
; i
++) {
4084 uint64_t bytes_remaining
;
4085 uint64_t bytes_evicted
;
4087 if (bytes
== ARC_EVICT_ALL
)
4088 bytes_remaining
= ARC_EVICT_ALL
;
4089 else if (total_evicted
< bytes
)
4090 bytes_remaining
= bytes
- total_evicted
;
4094 bytes_evicted
= arc_evict_state_impl(ml
, sublist_idx
,
4095 markers
[sublist_idx
], spa
, bytes_remaining
);
4097 scan_evicted
+= bytes_evicted
;
4098 total_evicted
+= bytes_evicted
;
4100 /* we've reached the end, wrap to the beginning */
4101 if (++sublist_idx
>= num_sublists
)
4106 * If we didn't evict anything during this scan, we have
4107 * no reason to believe we'll evict more during another
4108 * scan, so break the loop.
4110 if (scan_evicted
== 0) {
4111 /* This isn't possible, let's make that obvious */
4112 ASSERT3S(bytes
, !=, 0);
4115 * When bytes is ARC_EVICT_ALL, the only way to
4116 * break the loop is when scan_evicted is zero.
4117 * In that case, we actually have evicted enough,
4118 * so we don't want to increment the kstat.
4120 if (bytes
!= ARC_EVICT_ALL
) {
4121 ASSERT3S(total_evicted
, <, bytes
);
4122 ARCSTAT_BUMP(arcstat_evict_not_enough
);
4129 for (int i
= 0; i
< num_sublists
; i
++) {
4130 multilist_sublist_t
*mls
= multilist_sublist_lock(ml
, i
);
4131 multilist_sublist_remove(mls
, markers
[i
]);
4132 multilist_sublist_unlock(mls
);
4134 kmem_cache_free(hdr_full_cache
, markers
[i
]);
4136 kmem_free(markers
, sizeof (*markers
) * num_sublists
);
4138 return (total_evicted
);
4142 * Flush all "evictable" data of the given type from the arc state
4143 * specified. This will not evict any "active" buffers (i.e. referenced).
4145 * When 'retry' is set to B_FALSE, the function will make a single pass
4146 * over the state and evict any buffers that it can. Since it doesn't
4147 * continually retry the eviction, it might end up leaving some buffers
4148 * in the ARC due to lock misses.
4150 * When 'retry' is set to B_TRUE, the function will continually retry the
4151 * eviction until *all* evictable buffers have been removed from the
4152 * state. As a result, if concurrent insertions into the state are
4153 * allowed (e.g. if the ARC isn't shutting down), this function might
4154 * wind up in an infinite loop, continually trying to evict buffers.
4157 arc_flush_state(arc_state_t
*state
, uint64_t spa
, arc_buf_contents_t type
,
4160 uint64_t evicted
= 0;
4162 while (zfs_refcount_count(&state
->arcs_esize
[type
]) != 0) {
4163 evicted
+= arc_evict_state(state
, spa
, ARC_EVICT_ALL
, type
);
4173 * Evict the specified number of bytes from the state specified,
4174 * restricting eviction to the spa and type given. This function
4175 * prevents us from trying to evict more from a state's list than
4176 * is "evictable", and to skip evicting altogether when passed a
4177 * negative value for "bytes". In contrast, arc_evict_state() will
4178 * evict everything it can, when passed a negative value for "bytes".
4181 arc_adjust_impl(arc_state_t
*state
, uint64_t spa
, int64_t bytes
,
4182 arc_buf_contents_t type
)
4186 if (bytes
> 0 && zfs_refcount_count(&state
->arcs_esize
[type
]) > 0) {
4187 delta
= MIN(zfs_refcount_count(&state
->arcs_esize
[type
]),
4189 return (arc_evict_state(state
, spa
, delta
, type
));
4196 * Evict metadata buffers from the cache, such that arc_meta_used is
4197 * capped by the arc_meta_limit tunable.
4200 arc_adjust_meta(uint64_t meta_used
)
4202 uint64_t total_evicted
= 0;
4206 * If we're over the meta limit, we want to evict enough
4207 * metadata to get back under the meta limit. We don't want to
4208 * evict so much that we drop the MRU below arc_p, though. If
4209 * we're over the meta limit more than we're over arc_p, we
4210 * evict some from the MRU here, and some from the MFU below.
4212 target
= MIN((int64_t)(meta_used
- arc_meta_limit
),
4213 (int64_t)(zfs_refcount_count(&arc_anon
->arcs_size
) +
4214 zfs_refcount_count(&arc_mru
->arcs_size
) - arc_p
));
4216 total_evicted
+= arc_adjust_impl(arc_mru
, 0, target
, ARC_BUFC_METADATA
);
4219 * Similar to the above, we want to evict enough bytes to get us
4220 * below the meta limit, but not so much as to drop us below the
4221 * space allotted to the MFU (which is defined as arc_c - arc_p).
4223 target
= MIN((int64_t)(meta_used
- arc_meta_limit
),
4224 (int64_t)(zfs_refcount_count(&arc_mfu
->arcs_size
) -
4227 total_evicted
+= arc_adjust_impl(arc_mfu
, 0, target
, ARC_BUFC_METADATA
);
4229 return (total_evicted
);
4233 * Return the type of the oldest buffer in the given arc state
4235 * This function will select a random sublist of type ARC_BUFC_DATA and
4236 * a random sublist of type ARC_BUFC_METADATA. The tail of each sublist
4237 * is compared, and the type which contains the "older" buffer will be
4240 static arc_buf_contents_t
4241 arc_adjust_type(arc_state_t
*state
)
4243 multilist_t
*data_ml
= state
->arcs_list
[ARC_BUFC_DATA
];
4244 multilist_t
*meta_ml
= state
->arcs_list
[ARC_BUFC_METADATA
];
4245 int data_idx
= multilist_get_random_index(data_ml
);
4246 int meta_idx
= multilist_get_random_index(meta_ml
);
4247 multilist_sublist_t
*data_mls
;
4248 multilist_sublist_t
*meta_mls
;
4249 arc_buf_contents_t type
;
4250 arc_buf_hdr_t
*data_hdr
;
4251 arc_buf_hdr_t
*meta_hdr
;
4254 * We keep the sublist lock until we're finished, to prevent
4255 * the headers from being destroyed via arc_evict_state().
4257 data_mls
= multilist_sublist_lock(data_ml
, data_idx
);
4258 meta_mls
= multilist_sublist_lock(meta_ml
, meta_idx
);
4261 * These two loops are to ensure we skip any markers that
4262 * might be at the tail of the lists due to arc_evict_state().
4265 for (data_hdr
= multilist_sublist_tail(data_mls
); data_hdr
!= NULL
;
4266 data_hdr
= multilist_sublist_prev(data_mls
, data_hdr
)) {
4267 if (data_hdr
->b_spa
!= 0)
4271 for (meta_hdr
= multilist_sublist_tail(meta_mls
); meta_hdr
!= NULL
;
4272 meta_hdr
= multilist_sublist_prev(meta_mls
, meta_hdr
)) {
4273 if (meta_hdr
->b_spa
!= 0)
4277 if (data_hdr
== NULL
&& meta_hdr
== NULL
) {
4278 type
= ARC_BUFC_DATA
;
4279 } else if (data_hdr
== NULL
) {
4280 ASSERT3P(meta_hdr
, !=, NULL
);
4281 type
= ARC_BUFC_METADATA
;
4282 } else if (meta_hdr
== NULL
) {
4283 ASSERT3P(data_hdr
, !=, NULL
);
4284 type
= ARC_BUFC_DATA
;
4286 ASSERT3P(data_hdr
, !=, NULL
);
4287 ASSERT3P(meta_hdr
, !=, NULL
);
4289 /* The headers can't be on the sublist without an L1 header */
4290 ASSERT(HDR_HAS_L1HDR(data_hdr
));
4291 ASSERT(HDR_HAS_L1HDR(meta_hdr
));
4293 if (data_hdr
->b_l1hdr
.b_arc_access
<
4294 meta_hdr
->b_l1hdr
.b_arc_access
) {
4295 type
= ARC_BUFC_DATA
;
4297 type
= ARC_BUFC_METADATA
;
4301 multilist_sublist_unlock(meta_mls
);
4302 multilist_sublist_unlock(data_mls
);
4308 * Evict buffers from the cache, such that arc_size is capped by arc_c.
4313 uint64_t total_evicted
= 0;
4316 uint64_t asize
= aggsum_value(&arc_size
);
4317 uint64_t ameta
= aggsum_value(&arc_meta_used
);
4320 * If we're over arc_meta_limit, we want to correct that before
4321 * potentially evicting data buffers below.
4323 total_evicted
+= arc_adjust_meta(ameta
);
4328 * If we're over the target cache size, we want to evict enough
4329 * from the list to get back to our target size. We don't want
4330 * to evict too much from the MRU, such that it drops below
4331 * arc_p. So, if we're over our target cache size more than
4332 * the MRU is over arc_p, we'll evict enough to get back to
4333 * arc_p here, and then evict more from the MFU below.
4335 target
= MIN((int64_t)(asize
- arc_c
),
4336 (int64_t)(zfs_refcount_count(&arc_anon
->arcs_size
) +
4337 zfs_refcount_count(&arc_mru
->arcs_size
) + ameta
- arc_p
));
4340 * If we're below arc_meta_min, always prefer to evict data.
4341 * Otherwise, try to satisfy the requested number of bytes to
4342 * evict from the type which contains older buffers; in an
4343 * effort to keep newer buffers in the cache regardless of their
4344 * type. If we cannot satisfy the number of bytes from this
4345 * type, spill over into the next type.
4347 if (arc_adjust_type(arc_mru
) == ARC_BUFC_METADATA
&&
4348 ameta
> arc_meta_min
) {
4349 bytes
= arc_adjust_impl(arc_mru
, 0, target
, ARC_BUFC_METADATA
);
4350 total_evicted
+= bytes
;
4353 * If we couldn't evict our target number of bytes from
4354 * metadata, we try to get the rest from data.
4359 arc_adjust_impl(arc_mru
, 0, target
, ARC_BUFC_DATA
);
4361 bytes
= arc_adjust_impl(arc_mru
, 0, target
, ARC_BUFC_DATA
);
4362 total_evicted
+= bytes
;
4365 * If we couldn't evict our target number of bytes from
4366 * data, we try to get the rest from metadata.
4371 arc_adjust_impl(arc_mru
, 0, target
, ARC_BUFC_METADATA
);
4377 * Now that we've tried to evict enough from the MRU to get its
4378 * size back to arc_p, if we're still above the target cache
4379 * size, we evict the rest from the MFU.
4381 target
= asize
- arc_c
;
4383 if (arc_adjust_type(arc_mfu
) == ARC_BUFC_METADATA
&&
4384 ameta
> arc_meta_min
) {
4385 bytes
= arc_adjust_impl(arc_mfu
, 0, target
, ARC_BUFC_METADATA
);
4386 total_evicted
+= bytes
;
4389 * If we couldn't evict our target number of bytes from
4390 * metadata, we try to get the rest from data.
4395 arc_adjust_impl(arc_mfu
, 0, target
, ARC_BUFC_DATA
);
4397 bytes
= arc_adjust_impl(arc_mfu
, 0, target
, ARC_BUFC_DATA
);
4398 total_evicted
+= bytes
;
4401 * If we couldn't evict our target number of bytes from
4402 * data, we try to get the rest from data.
4407 arc_adjust_impl(arc_mfu
, 0, target
, ARC_BUFC_METADATA
);
4411 * Adjust ghost lists
4413 * In addition to the above, the ARC also defines target values
4414 * for the ghost lists. The sum of the mru list and mru ghost
4415 * list should never exceed the target size of the cache, and
4416 * the sum of the mru list, mfu list, mru ghost list, and mfu
4417 * ghost list should never exceed twice the target size of the
4418 * cache. The following logic enforces these limits on the ghost
4419 * caches, and evicts from them as needed.
4421 target
= zfs_refcount_count(&arc_mru
->arcs_size
) +
4422 zfs_refcount_count(&arc_mru_ghost
->arcs_size
) - arc_c
;
4424 bytes
= arc_adjust_impl(arc_mru_ghost
, 0, target
, ARC_BUFC_DATA
);
4425 total_evicted
+= bytes
;
4430 arc_adjust_impl(arc_mru_ghost
, 0, target
, ARC_BUFC_METADATA
);
4433 * We assume the sum of the mru list and mfu list is less than
4434 * or equal to arc_c (we enforced this above), which means we
4435 * can use the simpler of the two equations below:
4437 * mru + mfu + mru ghost + mfu ghost <= 2 * arc_c
4438 * mru ghost + mfu ghost <= arc_c
4440 target
= zfs_refcount_count(&arc_mru_ghost
->arcs_size
) +
4441 zfs_refcount_count(&arc_mfu_ghost
->arcs_size
) - arc_c
;
4443 bytes
= arc_adjust_impl(arc_mfu_ghost
, 0, target
, ARC_BUFC_DATA
);
4444 total_evicted
+= bytes
;
4449 arc_adjust_impl(arc_mfu_ghost
, 0, target
, ARC_BUFC_METADATA
);
4451 return (total_evicted
);
4455 arc_flush(spa_t
*spa
, boolean_t retry
)
4460 * If retry is B_TRUE, a spa must not be specified since we have
4461 * no good way to determine if all of a spa's buffers have been
4462 * evicted from an arc state.
4464 ASSERT(!retry
|| spa
== 0);
4467 guid
= spa_load_guid(spa
);
4469 (void) arc_flush_state(arc_mru
, guid
, ARC_BUFC_DATA
, retry
);
4470 (void) arc_flush_state(arc_mru
, guid
, ARC_BUFC_METADATA
, retry
);
4472 (void) arc_flush_state(arc_mfu
, guid
, ARC_BUFC_DATA
, retry
);
4473 (void) arc_flush_state(arc_mfu
, guid
, ARC_BUFC_METADATA
, retry
);
4475 (void) arc_flush_state(arc_mru_ghost
, guid
, ARC_BUFC_DATA
, retry
);
4476 (void) arc_flush_state(arc_mru_ghost
, guid
, ARC_BUFC_METADATA
, retry
);
4478 (void) arc_flush_state(arc_mfu_ghost
, guid
, ARC_BUFC_DATA
, retry
);
4479 (void) arc_flush_state(arc_mfu_ghost
, guid
, ARC_BUFC_METADATA
, retry
);
4483 arc_reduce_target_size(int64_t to_free
)
4485 uint64_t asize
= aggsum_value(&arc_size
);
4486 if (arc_c
> arc_c_min
) {
4488 if (arc_c
> arc_c_min
+ to_free
)
4489 atomic_add_64(&arc_c
, -to_free
);
4493 atomic_add_64(&arc_p
, -(arc_p
>> arc_shrink_shift
));
4495 arc_c
= MAX(asize
, arc_c_min
);
4497 arc_p
= (arc_c
>> 1);
4498 ASSERT(arc_c
>= arc_c_min
);
4499 ASSERT((int64_t)arc_p
>= 0);
4502 if (asize
> arc_c
) {
4503 /* See comment in arc_adjust_cb_check() on why lock+flag */
4504 mutex_enter(&arc_adjust_lock
);
4505 arc_adjust_needed
= B_TRUE
;
4506 mutex_exit(&arc_adjust_lock
);
4507 zthr_wakeup(arc_adjust_zthr
);
4511 typedef enum free_memory_reason_t
{
4516 FMR_PAGES_PP_MAXIMUM
,
4519 } free_memory_reason_t
;
4521 int64_t last_free_memory
;
4522 free_memory_reason_t last_free_reason
;
4525 * Additional reserve of pages for pp_reserve.
4527 int64_t arc_pages_pp_reserve
= 64;
4530 * Additional reserve of pages for swapfs.
4532 int64_t arc_swapfs_reserve
= 64;
4535 * Return the amount of memory that can be consumed before reclaim will be
4536 * needed. Positive if there is sufficient free memory, negative indicates
4537 * the amount of memory that needs to be freed up.
4540 arc_available_memory(void)
4542 int64_t lowest
= INT64_MAX
;
4544 free_memory_reason_t r
= FMR_UNKNOWN
;
4548 n
= PAGESIZE
* (-needfree
);
4556 * check that we're out of range of the pageout scanner. It starts to
4557 * schedule paging if freemem is less than lotsfree and needfree.
4558 * lotsfree is the high-water mark for pageout, and needfree is the
4559 * number of needed free pages. We add extra pages here to make sure
4560 * the scanner doesn't start up while we're freeing memory.
4562 n
= PAGESIZE
* (freemem
- lotsfree
- needfree
- desfree
);
4569 * check to make sure that swapfs has enough space so that anon
4570 * reservations can still succeed. anon_resvmem() checks that the
4571 * availrmem is greater than swapfs_minfree, and the number of reserved
4572 * swap pages. We also add a bit of extra here just to prevent
4573 * circumstances from getting really dire.
4575 n
= PAGESIZE
* (availrmem
- swapfs_minfree
- swapfs_reserve
-
4576 desfree
- arc_swapfs_reserve
);
4579 r
= FMR_SWAPFS_MINFREE
;
4584 * Check that we have enough availrmem that memory locking (e.g., via
4585 * mlock(3C) or memcntl(2)) can still succeed. (pages_pp_maximum
4586 * stores the number of pages that cannot be locked; when availrmem
4587 * drops below pages_pp_maximum, page locking mechanisms such as
4588 * page_pp_lock() will fail.)
4590 n
= PAGESIZE
* (availrmem
- pages_pp_maximum
-
4591 arc_pages_pp_reserve
);
4594 r
= FMR_PAGES_PP_MAXIMUM
;
4599 * If zio data pages are being allocated out of a separate heap segment,
4600 * then enforce that the size of available vmem for this arena remains
4601 * above about 1/4th (1/(2^arc_zio_arena_free_shift)) free.
4603 * Note that reducing the arc_zio_arena_free_shift keeps more virtual
4604 * memory (in the zio_arena) free, which can avoid memory
4605 * fragmentation issues.
4607 if (zio_arena
!= NULL
) {
4608 n
= (int64_t)vmem_size(zio_arena
, VMEM_FREE
) -
4609 (vmem_size(zio_arena
, VMEM_ALLOC
) >>
4610 arc_zio_arena_free_shift
);
4617 /* Every 100 calls, free a small amount */
4618 if (spa_get_random(100) == 0)
4622 last_free_memory
= lowest
;
4623 last_free_reason
= r
;
4630 * Determine if the system is under memory pressure and is asking
4631 * to reclaim memory. A return value of B_TRUE indicates that the system
4632 * is under memory pressure and that the arc should adjust accordingly.
4635 arc_reclaim_needed(void)
4637 return (arc_available_memory() < 0);
4641 arc_kmem_reap_soon(void)
4644 kmem_cache_t
*prev_cache
= NULL
;
4645 kmem_cache_t
*prev_data_cache
= NULL
;
4646 extern kmem_cache_t
*zio_buf_cache
[];
4647 extern kmem_cache_t
*zio_data_buf_cache
[];
4648 extern kmem_cache_t
*zfs_btree_leaf_cache
;
4649 extern kmem_cache_t
*abd_chunk_cache
;
4652 if (aggsum_compare(&arc_meta_used
, arc_meta_limit
) >= 0) {
4654 * We are exceeding our meta-data cache limit.
4655 * Purge some DNLC entries to release holds on meta-data.
4657 dnlc_reduce_cache((void *)(uintptr_t)arc_reduce_dnlc_percent
);
4661 for (i
= 0; i
< SPA_MAXBLOCKSIZE
>> SPA_MINBLOCKSHIFT
; i
++) {
4662 if (zio_buf_cache
[i
] != prev_cache
) {
4663 prev_cache
= zio_buf_cache
[i
];
4664 kmem_cache_reap_soon(zio_buf_cache
[i
]);
4666 if (zio_data_buf_cache
[i
] != prev_data_cache
) {
4667 prev_data_cache
= zio_data_buf_cache
[i
];
4668 kmem_cache_reap_soon(zio_data_buf_cache
[i
]);
4671 kmem_cache_reap_soon(abd_chunk_cache
);
4672 kmem_cache_reap_soon(buf_cache
);
4673 kmem_cache_reap_soon(hdr_full_cache
);
4674 kmem_cache_reap_soon(hdr_l2only_cache
);
4675 kmem_cache_reap_soon(zfs_btree_leaf_cache
);
4677 if (zio_arena
!= NULL
) {
4679 * Ask the vmem arena to reclaim unused memory from its
4682 vmem_qcache_reap(zio_arena
);
4688 arc_adjust_cb_check(void *arg
, zthr_t
*zthr
)
4691 * This is necessary in order for the mdb ::arc dcmd to
4692 * show up to date information. Since the ::arc command
4693 * does not call the kstat's update function, without
4694 * this call, the command may show stale stats for the
4695 * anon, mru, mru_ghost, mfu, and mfu_ghost lists. Even
4696 * with this change, the data might be up to 1 second
4697 * out of date(the arc_adjust_zthr has a maximum sleep
4698 * time of 1 second); but that should suffice. The
4699 * arc_state_t structures can be queried directly if more
4700 * accurate information is needed.
4702 if (arc_ksp
!= NULL
)
4703 arc_ksp
->ks_update(arc_ksp
, KSTAT_READ
);
4706 * We have to rely on arc_get_data_impl() to tell us when to adjust,
4707 * rather than checking if we are overflowing here, so that we are
4708 * sure to not leave arc_get_data_impl() waiting on
4709 * arc_adjust_waiters_cv. If we have become "not overflowing" since
4710 * arc_get_data_impl() checked, we need to wake it up. We could
4711 * broadcast the CV here, but arc_get_data_impl() may have not yet
4712 * gone to sleep. We would need to use a mutex to ensure that this
4713 * function doesn't broadcast until arc_get_data_impl() has gone to
4714 * sleep (e.g. the arc_adjust_lock). However, the lock ordering of
4715 * such a lock would necessarily be incorrect with respect to the
4716 * zthr_lock, which is held before this function is called, and is
4717 * held by arc_get_data_impl() when it calls zthr_wakeup().
4719 return (arc_adjust_needed
);
4723 * Keep arc_size under arc_c by running arc_adjust which evicts data
4728 arc_adjust_cb(void *arg
, zthr_t
*zthr
)
4730 uint64_t evicted
= 0;
4732 /* Evict from cache */
4733 evicted
= arc_adjust();
4736 * If evicted is zero, we couldn't evict anything
4737 * via arc_adjust(). This could be due to hash lock
4738 * collisions, but more likely due to the majority of
4739 * arc buffers being unevictable. Therefore, even if
4740 * arc_size is above arc_c, another pass is unlikely to
4741 * be helpful and could potentially cause us to enter an
4742 * infinite loop. Additionally, zthr_iscancelled() is
4743 * checked here so that if the arc is shutting down, the
4744 * broadcast will wake any remaining arc adjust waiters.
4746 mutex_enter(&arc_adjust_lock
);
4747 arc_adjust_needed
= !zthr_iscancelled(arc_adjust_zthr
) &&
4748 evicted
> 0 && aggsum_compare(&arc_size
, arc_c
) > 0;
4749 if (!arc_adjust_needed
) {
4751 * We're either no longer overflowing, or we
4752 * can't evict anything more, so we should wake
4755 cv_broadcast(&arc_adjust_waiters_cv
);
4757 mutex_exit(&arc_adjust_lock
);
4762 arc_reap_cb_check(void *arg
, zthr_t
*zthr
)
4764 int64_t free_memory
= arc_available_memory();
4767 * If a kmem reap is already active, don't schedule more. We must
4768 * check for this because kmem_cache_reap_soon() won't actually
4769 * block on the cache being reaped (this is to prevent callers from
4770 * becoming implicitly blocked by a system-wide kmem reap -- which,
4771 * on a system with many, many full magazines, can take minutes).
4773 if (!kmem_cache_reap_active() &&
4775 arc_no_grow
= B_TRUE
;
4778 * Wait at least zfs_grow_retry (default 60) seconds
4779 * before considering growing.
4781 arc_growtime
= gethrtime() + SEC2NSEC(arc_grow_retry
);
4783 } else if (free_memory
< arc_c
>> arc_no_grow_shift
) {
4784 arc_no_grow
= B_TRUE
;
4785 } else if (gethrtime() >= arc_growtime
) {
4786 arc_no_grow
= B_FALSE
;
4793 * Keep enough free memory in the system by reaping the ARC's kmem
4794 * caches. To cause more slabs to be reapable, we may reduce the
4795 * target size of the cache (arc_c), causing the arc_adjust_cb()
4796 * to free more buffers.
4800 arc_reap_cb(void *arg
, zthr_t
*zthr
)
4802 int64_t free_memory
;
4805 * Kick off asynchronous kmem_reap()'s of all our caches.
4807 arc_kmem_reap_soon();
4810 * Wait at least arc_kmem_cache_reap_retry_ms between
4811 * arc_kmem_reap_soon() calls. Without this check it is possible to
4812 * end up in a situation where we spend lots of time reaping
4813 * caches, while we're near arc_c_min. Waiting here also gives the
4814 * subsequent free memory check a chance of finding that the
4815 * asynchronous reap has already freed enough memory, and we don't
4816 * need to call arc_reduce_target_size().
4818 delay((hz
* arc_kmem_cache_reap_retry_ms
+ 999) / 1000);
4821 * Reduce the target size as needed to maintain the amount of free
4822 * memory in the system at a fraction of the arc_size (1/128th by
4823 * default). If oversubscribed (free_memory < 0) then reduce the
4824 * target arc_size by the deficit amount plus the fractional
4825 * amount. If free memory is positive but less then the fractional
4826 * amount, reduce by what is needed to hit the fractional amount.
4828 free_memory
= arc_available_memory();
4831 (arc_c
>> arc_shrink_shift
) - free_memory
;
4834 to_free
= MAX(to_free
, ptob(needfree
));
4836 arc_reduce_target_size(to_free
);
4841 * Adapt arc info given the number of bytes we are trying to add and
4842 * the state that we are coming from. This function is only called
4843 * when we are adding new content to the cache.
4846 arc_adapt(int bytes
, arc_state_t
*state
)
4849 uint64_t arc_p_min
= (arc_c
>> arc_p_min_shift
);
4850 int64_t mrug_size
= zfs_refcount_count(&arc_mru_ghost
->arcs_size
);
4851 int64_t mfug_size
= zfs_refcount_count(&arc_mfu_ghost
->arcs_size
);
4855 * Adapt the target size of the MRU list:
4856 * - if we just hit in the MRU ghost list, then increase
4857 * the target size of the MRU list.
4858 * - if we just hit in the MFU ghost list, then increase
4859 * the target size of the MFU list by decreasing the
4860 * target size of the MRU list.
4862 if (state
== arc_mru_ghost
) {
4863 mult
= (mrug_size
>= mfug_size
) ? 1 : (mfug_size
/ mrug_size
);
4864 mult
= MIN(mult
, 10); /* avoid wild arc_p adjustment */
4866 arc_p
= MIN(arc_c
- arc_p_min
, arc_p
+ bytes
* mult
);
4867 } else if (state
== arc_mfu_ghost
) {
4870 mult
= (mfug_size
>= mrug_size
) ? 1 : (mrug_size
/ mfug_size
);
4871 mult
= MIN(mult
, 10);
4873 delta
= MIN(bytes
* mult
, arc_p
);
4874 arc_p
= MAX(arc_p_min
, arc_p
- delta
);
4876 ASSERT((int64_t)arc_p
>= 0);
4879 * Wake reap thread if we do not have any available memory
4881 if (arc_reclaim_needed()) {
4882 zthr_wakeup(arc_reap_zthr
);
4890 if (arc_c
>= arc_c_max
)
4894 * If we're within (2 * maxblocksize) bytes of the target
4895 * cache size, increment the target cache size
4897 if (aggsum_compare(&arc_size
, arc_c
- (2ULL << SPA_MAXBLOCKSHIFT
)) >
4899 atomic_add_64(&arc_c
, (int64_t)bytes
);
4900 if (arc_c
> arc_c_max
)
4902 else if (state
== arc_anon
)
4903 atomic_add_64(&arc_p
, (int64_t)bytes
);
4907 ASSERT((int64_t)arc_p
>= 0);
4911 * Check if arc_size has grown past our upper threshold, determined by
4912 * zfs_arc_overflow_shift.
4915 arc_is_overflowing(void)
4917 /* Always allow at least one block of overflow */
4918 uint64_t overflow
= MAX(SPA_MAXBLOCKSIZE
,
4919 arc_c
>> zfs_arc_overflow_shift
);
4922 * We just compare the lower bound here for performance reasons. Our
4923 * primary goals are to make sure that the arc never grows without
4924 * bound, and that it can reach its maximum size. This check
4925 * accomplishes both goals. The maximum amount we could run over by is
4926 * 2 * aggsum_borrow_multiplier * NUM_CPUS * the average size of a block
4927 * in the ARC. In practice, that's in the tens of MB, which is low
4928 * enough to be safe.
4930 return (aggsum_lower_bound(&arc_size
) >= arc_c
+ overflow
);
4934 arc_get_data_abd(arc_buf_hdr_t
*hdr
, uint64_t size
, void *tag
,
4937 arc_buf_contents_t type
= arc_buf_type(hdr
);
4939 arc_get_data_impl(hdr
, size
, tag
, do_adapt
);
4940 if (type
== ARC_BUFC_METADATA
) {
4941 return (abd_alloc(size
, B_TRUE
));
4943 ASSERT(type
== ARC_BUFC_DATA
);
4944 return (abd_alloc(size
, B_FALSE
));
4949 arc_get_data_buf(arc_buf_hdr_t
*hdr
, uint64_t size
, void *tag
)
4951 arc_buf_contents_t type
= arc_buf_type(hdr
);
4953 arc_get_data_impl(hdr
, size
, tag
, B_TRUE
);
4954 if (type
== ARC_BUFC_METADATA
) {
4955 return (zio_buf_alloc(size
));
4957 ASSERT(type
== ARC_BUFC_DATA
);
4958 return (zio_data_buf_alloc(size
));
4963 * Allocate a block and return it to the caller. If we are hitting the
4964 * hard limit for the cache size, we must sleep, waiting for the eviction
4965 * thread to catch up. If we're past the target size but below the hard
4966 * limit, we'll only signal the reclaim thread and continue on.
4969 arc_get_data_impl(arc_buf_hdr_t
*hdr
, uint64_t size
, void *tag
,
4972 arc_state_t
*state
= hdr
->b_l1hdr
.b_state
;
4973 arc_buf_contents_t type
= arc_buf_type(hdr
);
4976 arc_adapt(size
, state
);
4979 * If arc_size is currently overflowing, and has grown past our
4980 * upper limit, we must be adding data faster than the evict
4981 * thread can evict. Thus, to ensure we don't compound the
4982 * problem by adding more data and forcing arc_size to grow even
4983 * further past its target size, we halt and wait for the
4984 * eviction thread to catch up.
4986 * It's also possible that the reclaim thread is unable to evict
4987 * enough buffers to get arc_size below the overflow limit (e.g.
4988 * due to buffers being un-evictable, or hash lock collisions).
4989 * In this case, we want to proceed regardless if we're
4990 * overflowing; thus we don't use a while loop here.
4992 if (arc_is_overflowing()) {
4993 mutex_enter(&arc_adjust_lock
);
4996 * Now that we've acquired the lock, we may no longer be
4997 * over the overflow limit, lets check.
4999 * We're ignoring the case of spurious wake ups. If that
5000 * were to happen, it'd let this thread consume an ARC
5001 * buffer before it should have (i.e. before we're under
5002 * the overflow limit and were signalled by the reclaim
5003 * thread). As long as that is a rare occurrence, it
5004 * shouldn't cause any harm.
5006 if (arc_is_overflowing()) {
5007 arc_adjust_needed
= B_TRUE
;
5008 zthr_wakeup(arc_adjust_zthr
);
5009 (void) cv_wait(&arc_adjust_waiters_cv
,
5012 mutex_exit(&arc_adjust_lock
);
5015 VERIFY3U(hdr
->b_type
, ==, type
);
5016 if (type
== ARC_BUFC_METADATA
) {
5017 arc_space_consume(size
, ARC_SPACE_META
);
5019 arc_space_consume(size
, ARC_SPACE_DATA
);
5023 * Update the state size. Note that ghost states have a
5024 * "ghost size" and so don't need to be updated.
5026 if (!GHOST_STATE(state
)) {
5028 (void) zfs_refcount_add_many(&state
->arcs_size
, size
, tag
);
5031 * If this is reached via arc_read, the link is
5032 * protected by the hash lock. If reached via
5033 * arc_buf_alloc, the header should not be accessed by
5034 * any other thread. And, if reached via arc_read_done,
5035 * the hash lock will protect it if it's found in the
5036 * hash table; otherwise no other thread should be
5037 * trying to [add|remove]_reference it.
5039 if (multilist_link_active(&hdr
->b_l1hdr
.b_arc_node
)) {
5040 ASSERT(zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
));
5041 (void) zfs_refcount_add_many(&state
->arcs_esize
[type
],
5046 * If we are growing the cache, and we are adding anonymous
5047 * data, and we have outgrown arc_p, update arc_p
5049 if (aggsum_compare(&arc_size
, arc_c
) < 0 &&
5050 hdr
->b_l1hdr
.b_state
== arc_anon
&&
5051 (zfs_refcount_count(&arc_anon
->arcs_size
) +
5052 zfs_refcount_count(&arc_mru
->arcs_size
) > arc_p
))
5053 arc_p
= MIN(arc_c
, arc_p
+ size
);
5058 arc_free_data_abd(arc_buf_hdr_t
*hdr
, abd_t
*abd
, uint64_t size
, void *tag
)
5060 arc_free_data_impl(hdr
, size
, tag
);
5065 arc_free_data_buf(arc_buf_hdr_t
*hdr
, void *buf
, uint64_t size
, void *tag
)
5067 arc_buf_contents_t type
= arc_buf_type(hdr
);
5069 arc_free_data_impl(hdr
, size
, tag
);
5070 if (type
== ARC_BUFC_METADATA
) {
5071 zio_buf_free(buf
, size
);
5073 ASSERT(type
== ARC_BUFC_DATA
);
5074 zio_data_buf_free(buf
, size
);
5079 * Free the arc data buffer.
5082 arc_free_data_impl(arc_buf_hdr_t
*hdr
, uint64_t size
, void *tag
)
5084 arc_state_t
*state
= hdr
->b_l1hdr
.b_state
;
5085 arc_buf_contents_t type
= arc_buf_type(hdr
);
5087 /* protected by hash lock, if in the hash table */
5088 if (multilist_link_active(&hdr
->b_l1hdr
.b_arc_node
)) {
5089 ASSERT(zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
));
5090 ASSERT(state
!= arc_anon
&& state
!= arc_l2c_only
);
5092 (void) zfs_refcount_remove_many(&state
->arcs_esize
[type
],
5095 (void) zfs_refcount_remove_many(&state
->arcs_size
, size
, tag
);
5097 VERIFY3U(hdr
->b_type
, ==, type
);
5098 if (type
== ARC_BUFC_METADATA
) {
5099 arc_space_return(size
, ARC_SPACE_META
);
5101 ASSERT(type
== ARC_BUFC_DATA
);
5102 arc_space_return(size
, ARC_SPACE_DATA
);
5107 * This routine is called whenever a buffer is accessed.
5108 * NOTE: the hash lock is dropped in this function.
5111 arc_access(arc_buf_hdr_t
*hdr
, kmutex_t
*hash_lock
)
5115 ASSERT(MUTEX_HELD(hash_lock
));
5116 ASSERT(HDR_HAS_L1HDR(hdr
));
5118 if (hdr
->b_l1hdr
.b_state
== arc_anon
) {
5120 * This buffer is not in the cache, and does not
5121 * appear in our "ghost" list. Add the new buffer
5125 ASSERT0(hdr
->b_l1hdr
.b_arc_access
);
5126 hdr
->b_l1hdr
.b_arc_access
= ddi_get_lbolt();
5127 DTRACE_PROBE1(new_state__mru
, arc_buf_hdr_t
*, hdr
);
5128 arc_change_state(arc_mru
, hdr
, hash_lock
);
5130 } else if (hdr
->b_l1hdr
.b_state
== arc_mru
) {
5131 now
= ddi_get_lbolt();
5134 * If this buffer is here because of a prefetch, then either:
5135 * - clear the flag if this is a "referencing" read
5136 * (any subsequent access will bump this into the MFU state).
5138 * - move the buffer to the head of the list if this is
5139 * another prefetch (to make it less likely to be evicted).
5141 if (HDR_PREFETCH(hdr
) || HDR_PRESCIENT_PREFETCH(hdr
)) {
5142 if (zfs_refcount_count(&hdr
->b_l1hdr
.b_refcnt
) == 0) {
5143 /* link protected by hash lock */
5144 ASSERT(multilist_link_active(
5145 &hdr
->b_l1hdr
.b_arc_node
));
5147 if (HDR_HAS_L2HDR(hdr
))
5148 l2arc_hdr_arcstats_decrement_state(hdr
);
5149 arc_hdr_clear_flags(hdr
,
5151 ARC_FLAG_PRESCIENT_PREFETCH
);
5152 ARCSTAT_BUMP(arcstat_mru_hits
);
5153 if (HDR_HAS_L2HDR(hdr
))
5154 l2arc_hdr_arcstats_increment_state(hdr
);
5156 hdr
->b_l1hdr
.b_arc_access
= now
;
5161 * This buffer has been "accessed" only once so far,
5162 * but it is still in the cache. Move it to the MFU
5165 if (now
> hdr
->b_l1hdr
.b_arc_access
+ ARC_MINTIME
) {
5167 * More than 125ms have passed since we
5168 * instantiated this buffer. Move it to the
5169 * most frequently used state.
5171 hdr
->b_l1hdr
.b_arc_access
= now
;
5172 DTRACE_PROBE1(new_state__mfu
, arc_buf_hdr_t
*, hdr
);
5173 arc_change_state(arc_mfu
, hdr
, hash_lock
);
5175 ARCSTAT_BUMP(arcstat_mru_hits
);
5176 } else if (hdr
->b_l1hdr
.b_state
== arc_mru_ghost
) {
5177 arc_state_t
*new_state
;
5179 * This buffer has been "accessed" recently, but
5180 * was evicted from the cache. Move it to the
5183 if (HDR_PREFETCH(hdr
) || HDR_PRESCIENT_PREFETCH(hdr
)) {
5184 new_state
= arc_mru
;
5185 if (zfs_refcount_count(&hdr
->b_l1hdr
.b_refcnt
) > 0) {
5186 if (HDR_HAS_L2HDR(hdr
))
5187 l2arc_hdr_arcstats_decrement_state(hdr
);
5188 arc_hdr_clear_flags(hdr
,
5190 ARC_FLAG_PRESCIENT_PREFETCH
);
5191 if (HDR_HAS_L2HDR(hdr
))
5192 l2arc_hdr_arcstats_increment_state(hdr
);
5194 DTRACE_PROBE1(new_state__mru
, arc_buf_hdr_t
*, hdr
);
5196 new_state
= arc_mfu
;
5197 DTRACE_PROBE1(new_state__mfu
, arc_buf_hdr_t
*, hdr
);
5200 hdr
->b_l1hdr
.b_arc_access
= ddi_get_lbolt();
5201 arc_change_state(new_state
, hdr
, hash_lock
);
5203 ARCSTAT_BUMP(arcstat_mru_ghost_hits
);
5204 } else if (hdr
->b_l1hdr
.b_state
== arc_mfu
) {
5206 * This buffer has been accessed more than once and is
5207 * still in the cache. Keep it in the MFU state.
5209 * NOTE: an add_reference() that occurred when we did
5210 * the arc_read() will have kicked this off the list.
5211 * If it was a prefetch, we will explicitly move it to
5212 * the head of the list now.
5214 ARCSTAT_BUMP(arcstat_mfu_hits
);
5215 hdr
->b_l1hdr
.b_arc_access
= ddi_get_lbolt();
5216 } else if (hdr
->b_l1hdr
.b_state
== arc_mfu_ghost
) {
5217 arc_state_t
*new_state
= arc_mfu
;
5219 * This buffer has been accessed more than once but has
5220 * been evicted from the cache. Move it back to the
5224 if (HDR_PREFETCH(hdr
) || HDR_PRESCIENT_PREFETCH(hdr
)) {
5226 * This is a prefetch access...
5227 * move this block back to the MRU state.
5229 new_state
= arc_mru
;
5232 hdr
->b_l1hdr
.b_arc_access
= ddi_get_lbolt();
5233 DTRACE_PROBE1(new_state__mfu
, arc_buf_hdr_t
*, hdr
);
5234 arc_change_state(new_state
, hdr
, hash_lock
);
5236 ARCSTAT_BUMP(arcstat_mfu_ghost_hits
);
5237 } else if (hdr
->b_l1hdr
.b_state
== arc_l2c_only
) {
5239 * This buffer is on the 2nd Level ARC.
5242 hdr
->b_l1hdr
.b_arc_access
= ddi_get_lbolt();
5243 DTRACE_PROBE1(new_state__mfu
, arc_buf_hdr_t
*, hdr
);
5244 arc_change_state(arc_mfu
, hdr
, hash_lock
);
5246 ASSERT(!"invalid arc state");
5251 * This routine is called by dbuf_hold() to update the arc_access() state
5252 * which otherwise would be skipped for entries in the dbuf cache.
5255 arc_buf_access(arc_buf_t
*buf
)
5257 mutex_enter(&buf
->b_evict_lock
);
5258 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
5261 * Avoid taking the hash_lock when possible as an optimization.
5262 * The header must be checked again under the hash_lock in order
5263 * to handle the case where it is concurrently being released.
5265 if (hdr
->b_l1hdr
.b_state
== arc_anon
|| HDR_EMPTY(hdr
)) {
5266 mutex_exit(&buf
->b_evict_lock
);
5270 kmutex_t
*hash_lock
= HDR_LOCK(hdr
);
5271 mutex_enter(hash_lock
);
5273 if (hdr
->b_l1hdr
.b_state
== arc_anon
|| HDR_EMPTY(hdr
)) {
5274 mutex_exit(hash_lock
);
5275 mutex_exit(&buf
->b_evict_lock
);
5276 ARCSTAT_BUMP(arcstat_access_skip
);
5280 mutex_exit(&buf
->b_evict_lock
);
5282 ASSERT(hdr
->b_l1hdr
.b_state
== arc_mru
||
5283 hdr
->b_l1hdr
.b_state
== arc_mfu
);
5285 DTRACE_PROBE1(arc__hit
, arc_buf_hdr_t
*, hdr
);
5286 arc_access(hdr
, hash_lock
);
5287 mutex_exit(hash_lock
);
5289 ARCSTAT_BUMP(arcstat_hits
);
5290 ARCSTAT_CONDSTAT(!HDR_PREFETCH(hdr
),
5291 demand
, prefetch
, !HDR_ISTYPE_METADATA(hdr
), data
, metadata
, hits
);
5294 /* a generic arc_read_done_func_t which you can use */
5297 arc_bcopy_func(zio_t
*zio
, const zbookmark_phys_t
*zb
, const blkptr_t
*bp
,
5298 arc_buf_t
*buf
, void *arg
)
5303 bcopy(buf
->b_data
, arg
, arc_buf_size(buf
));
5304 arc_buf_destroy(buf
, arg
);
5307 /* a generic arc_read_done_func_t */
5309 arc_getbuf_func(zio_t
*zio
, const zbookmark_phys_t
*zb
, const blkptr_t
*bp
,
5310 arc_buf_t
*buf
, void *arg
)
5312 arc_buf_t
**bufp
= arg
;
5315 ASSERT(zio
== NULL
|| zio
->io_error
!= 0);
5318 ASSERT(zio
== NULL
|| zio
->io_error
== 0);
5320 ASSERT(buf
->b_data
!= NULL
);
5325 arc_hdr_verify(arc_buf_hdr_t
*hdr
, const blkptr_t
*bp
)
5327 if (BP_IS_HOLE(bp
) || BP_IS_EMBEDDED(bp
)) {
5328 ASSERT3U(HDR_GET_PSIZE(hdr
), ==, 0);
5329 ASSERT3U(arc_hdr_get_compress(hdr
), ==, ZIO_COMPRESS_OFF
);
5331 if (HDR_COMPRESSION_ENABLED(hdr
)) {
5332 ASSERT3U(arc_hdr_get_compress(hdr
), ==,
5333 BP_GET_COMPRESS(bp
));
5335 ASSERT3U(HDR_GET_LSIZE(hdr
), ==, BP_GET_LSIZE(bp
));
5336 ASSERT3U(HDR_GET_PSIZE(hdr
), ==, BP_GET_PSIZE(bp
));
5337 ASSERT3U(!!HDR_PROTECTED(hdr
), ==, BP_IS_PROTECTED(bp
));
5342 * XXX this should be changed to return an error, and callers
5343 * re-read from disk on failure (on nondebug bits).
5346 arc_hdr_verify_checksum(spa_t
*spa
, arc_buf_hdr_t
*hdr
, const blkptr_t
*bp
)
5348 arc_hdr_verify(hdr
, bp
);
5349 if (BP_IS_HOLE(bp
) || BP_IS_EMBEDDED(bp
))
5353 if (BP_IS_ENCRYPTED(bp
)) {
5354 if (HDR_HAS_RABD(hdr
)) {
5355 abd
= hdr
->b_crypt_hdr
.b_rabd
;
5357 } else if (HDR_COMPRESSION_ENABLED(hdr
)) {
5358 abd
= hdr
->b_l1hdr
.b_pabd
;
5362 * The offset is only used for labels, which are not
5363 * cached in the ARC, so it doesn't matter what we
5364 * pass for the offset parameter.
5366 int psize
= HDR_GET_PSIZE(hdr
);
5367 err
= zio_checksum_error_impl(spa
, bp
,
5368 BP_GET_CHECKSUM(bp
), abd
, psize
, 0, NULL
);
5371 * Use abd_copy_to_buf() rather than
5372 * abd_borrow_buf_copy() so that we are sure to
5373 * include the buf in crash dumps.
5375 void *buf
= kmem_alloc(psize
, KM_SLEEP
);
5376 abd_copy_to_buf(buf
, abd
, psize
);
5377 panic("checksum of cached data doesn't match BP "
5378 "err=%u hdr=%p bp=%p abd=%p buf=%p",
5379 err
, (void *)hdr
, (void *)bp
, (void *)abd
, buf
);
5385 arc_read_done(zio_t
*zio
)
5387 blkptr_t
*bp
= zio
->io_bp
;
5388 arc_buf_hdr_t
*hdr
= zio
->io_private
;
5389 kmutex_t
*hash_lock
= NULL
;
5390 arc_callback_t
*callback_list
;
5391 arc_callback_t
*acb
;
5392 boolean_t freeable
= B_FALSE
;
5395 * The hdr was inserted into hash-table and removed from lists
5396 * prior to starting I/O. We should find this header, since
5397 * it's in the hash table, and it should be legit since it's
5398 * not possible to evict it during the I/O. The only possible
5399 * reason for it not to be found is if we were freed during the
5402 if (HDR_IN_HASH_TABLE(hdr
)) {
5403 ASSERT3U(hdr
->b_birth
, ==, BP_PHYSICAL_BIRTH(zio
->io_bp
));
5404 ASSERT3U(hdr
->b_dva
.dva_word
[0], ==,
5405 BP_IDENTITY(zio
->io_bp
)->dva_word
[0]);
5406 ASSERT3U(hdr
->b_dva
.dva_word
[1], ==,
5407 BP_IDENTITY(zio
->io_bp
)->dva_word
[1]);
5409 arc_buf_hdr_t
*found
= buf_hash_find(hdr
->b_spa
, zio
->io_bp
,
5412 ASSERT((found
== hdr
&&
5413 DVA_EQUAL(&hdr
->b_dva
, BP_IDENTITY(zio
->io_bp
))) ||
5414 (found
== hdr
&& HDR_L2_READING(hdr
)));
5415 ASSERT3P(hash_lock
, !=, NULL
);
5418 if (BP_IS_PROTECTED(bp
)) {
5419 hdr
->b_crypt_hdr
.b_ot
= BP_GET_TYPE(bp
);
5420 hdr
->b_crypt_hdr
.b_dsobj
= zio
->io_bookmark
.zb_objset
;
5421 zio_crypt_decode_params_bp(bp
, hdr
->b_crypt_hdr
.b_salt
,
5422 hdr
->b_crypt_hdr
.b_iv
);
5424 if (BP_GET_TYPE(bp
) == DMU_OT_INTENT_LOG
) {
5427 tmpbuf
= abd_borrow_buf_copy(zio
->io_abd
,
5428 sizeof (zil_chain_t
));
5429 zio_crypt_decode_mac_zil(tmpbuf
,
5430 hdr
->b_crypt_hdr
.b_mac
);
5431 abd_return_buf(zio
->io_abd
, tmpbuf
,
5432 sizeof (zil_chain_t
));
5434 zio_crypt_decode_mac_bp(bp
, hdr
->b_crypt_hdr
.b_mac
);
5438 if (zio
->io_error
== 0) {
5439 /* byteswap if necessary */
5440 if (BP_SHOULD_BYTESWAP(zio
->io_bp
)) {
5441 if (BP_GET_LEVEL(zio
->io_bp
) > 0) {
5442 hdr
->b_l1hdr
.b_byteswap
= DMU_BSWAP_UINT64
;
5444 hdr
->b_l1hdr
.b_byteswap
=
5445 DMU_OT_BYTESWAP(BP_GET_TYPE(zio
->io_bp
));
5448 hdr
->b_l1hdr
.b_byteswap
= DMU_BSWAP_NUMFUNCS
;
5452 arc_hdr_clear_flags(hdr
, ARC_FLAG_L2_EVICTED
);
5454 callback_list
= hdr
->b_l1hdr
.b_acb
;
5455 ASSERT3P(callback_list
, !=, NULL
);
5457 if (hash_lock
&& zio
->io_error
== 0 &&
5458 hdr
->b_l1hdr
.b_state
== arc_anon
) {
5460 * Only call arc_access on anonymous buffers. This is because
5461 * if we've issued an I/O for an evicted buffer, we've already
5462 * called arc_access (to prevent any simultaneous readers from
5463 * getting confused).
5465 arc_access(hdr
, hash_lock
);
5469 * If a read request has a callback (i.e. acb_done is not NULL), then we
5470 * make a buf containing the data according to the parameters which were
5471 * passed in. The implementation of arc_buf_alloc_impl() ensures that we
5472 * aren't needlessly decompressing the data multiple times.
5474 int callback_cnt
= 0;
5475 for (acb
= callback_list
; acb
!= NULL
; acb
= acb
->acb_next
) {
5481 if (zio
->io_error
!= 0)
5484 int error
= arc_buf_alloc_impl(hdr
, zio
->io_spa
,
5485 &acb
->acb_zb
, acb
->acb_private
, acb
->acb_encrypted
,
5486 acb
->acb_compressed
, acb
->acb_noauth
, B_TRUE
,
5490 * Assert non-speculative zios didn't fail because an
5491 * encryption key wasn't loaded
5493 ASSERT((zio
->io_flags
& ZIO_FLAG_SPECULATIVE
) ||
5497 * If we failed to decrypt, report an error now (as the zio
5498 * layer would have done if it had done the transforms).
5500 if (error
== ECKSUM
) {
5501 ASSERT(BP_IS_PROTECTED(bp
));
5502 error
= SET_ERROR(EIO
);
5503 if ((zio
->io_flags
& ZIO_FLAG_SPECULATIVE
) == 0) {
5504 spa_log_error(zio
->io_spa
, &acb
->acb_zb
);
5505 (void) zfs_ereport_post(
5506 FM_EREPORT_ZFS_AUTHENTICATION
,
5507 zio
->io_spa
, NULL
, &acb
->acb_zb
, zio
, 0, 0);
5513 * Decompression failed. Set io_error
5514 * so that when we call acb_done (below),
5515 * we will indicate that the read failed.
5516 * Note that in the unusual case where one
5517 * callback is compressed and another
5518 * uncompressed, we will mark all of them
5519 * as failed, even though the uncompressed
5520 * one can't actually fail. In this case,
5521 * the hdr will not be anonymous, because
5522 * if there are multiple callbacks, it's
5523 * because multiple threads found the same
5524 * arc buf in the hash table.
5526 zio
->io_error
= error
;
5531 * If there are multiple callbacks, we must have the hash lock,
5532 * because the only way for multiple threads to find this hdr is
5533 * in the hash table. This ensures that if there are multiple
5534 * callbacks, the hdr is not anonymous. If it were anonymous,
5535 * we couldn't use arc_buf_destroy() in the error case below.
5537 ASSERT(callback_cnt
< 2 || hash_lock
!= NULL
);
5539 hdr
->b_l1hdr
.b_acb
= NULL
;
5540 arc_hdr_clear_flags(hdr
, ARC_FLAG_IO_IN_PROGRESS
);
5541 if (callback_cnt
== 0)
5542 ASSERT(hdr
->b_l1hdr
.b_pabd
!= NULL
|| HDR_HAS_RABD(hdr
));
5544 ASSERT(zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
) ||
5545 callback_list
!= NULL
);
5547 if (zio
->io_error
== 0) {
5548 arc_hdr_verify(hdr
, zio
->io_bp
);
5550 arc_hdr_set_flags(hdr
, ARC_FLAG_IO_ERROR
);
5551 if (hdr
->b_l1hdr
.b_state
!= arc_anon
)
5552 arc_change_state(arc_anon
, hdr
, hash_lock
);
5553 if (HDR_IN_HASH_TABLE(hdr
))
5554 buf_hash_remove(hdr
);
5555 freeable
= zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
);
5559 * Broadcast before we drop the hash_lock to avoid the possibility
5560 * that the hdr (and hence the cv) might be freed before we get to
5561 * the cv_broadcast().
5563 cv_broadcast(&hdr
->b_l1hdr
.b_cv
);
5565 if (hash_lock
!= NULL
) {
5566 mutex_exit(hash_lock
);
5569 * This block was freed while we waited for the read to
5570 * complete. It has been removed from the hash table and
5571 * moved to the anonymous state (so that it won't show up
5574 ASSERT3P(hdr
->b_l1hdr
.b_state
, ==, arc_anon
);
5575 freeable
= zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
);
5578 /* execute each callback and free its structure */
5579 while ((acb
= callback_list
) != NULL
) {
5581 if (acb
->acb_done
!= NULL
) {
5582 if (zio
->io_error
!= 0 && acb
->acb_buf
!= NULL
) {
5584 * If arc_buf_alloc_impl() fails during
5585 * decompression, the buf will still be
5586 * allocated, and needs to be freed here.
5588 arc_buf_destroy(acb
->acb_buf
, acb
->acb_private
);
5589 acb
->acb_buf
= NULL
;
5591 acb
->acb_done(zio
, &zio
->io_bookmark
, zio
->io_bp
,
5592 acb
->acb_buf
, acb
->acb_private
);
5595 if (acb
->acb_zio_dummy
!= NULL
) {
5596 acb
->acb_zio_dummy
->io_error
= zio
->io_error
;
5597 zio_nowait(acb
->acb_zio_dummy
);
5600 callback_list
= acb
->acb_next
;
5601 kmem_free(acb
, sizeof (arc_callback_t
));
5605 arc_hdr_destroy(hdr
);
5609 * "Read" the block at the specified DVA (in bp) via the
5610 * cache. If the block is found in the cache, invoke the provided
5611 * callback immediately and return. Note that the `zio' parameter
5612 * in the callback will be NULL in this case, since no IO was
5613 * required. If the block is not in the cache pass the read request
5614 * on to the spa with a substitute callback function, so that the
5615 * requested block will be added to the cache.
5617 * If a read request arrives for a block that has a read in-progress,
5618 * either wait for the in-progress read to complete (and return the
5619 * results); or, if this is a read with a "done" func, add a record
5620 * to the read to invoke the "done" func when the read completes,
5621 * and return; or just return.
5623 * arc_read_done() will invoke all the requested "done" functions
5624 * for readers of this block.
5627 arc_read(zio_t
*pio
, spa_t
*spa
, const blkptr_t
*bp
, arc_read_done_func_t
*done
,
5628 void *private, zio_priority_t priority
, int zio_flags
,
5629 arc_flags_t
*arc_flags
, const zbookmark_phys_t
*zb
)
5631 arc_buf_hdr_t
*hdr
= NULL
;
5632 kmutex_t
*hash_lock
= NULL
;
5634 uint64_t guid
= spa_load_guid(spa
);
5635 boolean_t compressed_read
= (zio_flags
& ZIO_FLAG_RAW_COMPRESS
) != 0;
5636 boolean_t encrypted_read
= BP_IS_ENCRYPTED(bp
) &&
5637 (zio_flags
& ZIO_FLAG_RAW_ENCRYPT
) != 0;
5638 boolean_t noauth_read
= BP_IS_AUTHENTICATED(bp
) &&
5639 (zio_flags
& ZIO_FLAG_RAW_ENCRYPT
) != 0;
5642 ASSERT(!BP_IS_EMBEDDED(bp
) ||
5643 BPE_GET_ETYPE(bp
) == BP_EMBEDDED_TYPE_DATA
);
5646 if (!BP_IS_EMBEDDED(bp
)) {
5648 * Embedded BP's have no DVA and require no I/O to "read".
5649 * Create an anonymous arc buf to back it.
5651 hdr
= buf_hash_find(guid
, bp
, &hash_lock
);
5655 * Determine if we have an L1 cache hit or a cache miss. For simplicity
5656 * we maintain encrypted data seperately from compressed / uncompressed
5657 * data. If the user is requesting raw encrypted data and we don't have
5658 * that in the header we will read from disk to guarantee that we can
5659 * get it even if the encryption keys aren't loaded.
5661 if (hdr
!= NULL
&& HDR_HAS_L1HDR(hdr
) && (HDR_HAS_RABD(hdr
) ||
5662 (hdr
->b_l1hdr
.b_pabd
!= NULL
&& !encrypted_read
))) {
5663 arc_buf_t
*buf
= NULL
;
5664 *arc_flags
|= ARC_FLAG_CACHED
;
5666 if (HDR_IO_IN_PROGRESS(hdr
)) {
5667 zio_t
*head_zio
= hdr
->b_l1hdr
.b_acb
->acb_zio_head
;
5669 ASSERT3P(head_zio
, !=, NULL
);
5670 if ((hdr
->b_flags
& ARC_FLAG_PRIO_ASYNC_READ
) &&
5671 priority
== ZIO_PRIORITY_SYNC_READ
) {
5673 * This is a sync read that needs to wait for
5674 * an in-flight async read. Request that the
5675 * zio have its priority upgraded.
5677 zio_change_priority(head_zio
, priority
);
5678 DTRACE_PROBE1(arc__async__upgrade__sync
,
5679 arc_buf_hdr_t
*, hdr
);
5680 ARCSTAT_BUMP(arcstat_async_upgrade_sync
);
5682 if (hdr
->b_flags
& ARC_FLAG_PREDICTIVE_PREFETCH
) {
5683 arc_hdr_clear_flags(hdr
,
5684 ARC_FLAG_PREDICTIVE_PREFETCH
);
5687 if (*arc_flags
& ARC_FLAG_WAIT
) {
5688 cv_wait(&hdr
->b_l1hdr
.b_cv
, hash_lock
);
5689 mutex_exit(hash_lock
);
5692 ASSERT(*arc_flags
& ARC_FLAG_NOWAIT
);
5695 arc_callback_t
*acb
= NULL
;
5697 acb
= kmem_zalloc(sizeof (arc_callback_t
),
5699 acb
->acb_done
= done
;
5700 acb
->acb_private
= private;
5701 acb
->acb_compressed
= compressed_read
;
5702 acb
->acb_encrypted
= encrypted_read
;
5703 acb
->acb_noauth
= noauth_read
;
5706 acb
->acb_zio_dummy
= zio_null(pio
,
5707 spa
, NULL
, NULL
, NULL
, zio_flags
);
5709 ASSERT3P(acb
->acb_done
, !=, NULL
);
5710 acb
->acb_zio_head
= head_zio
;
5711 acb
->acb_next
= hdr
->b_l1hdr
.b_acb
;
5712 hdr
->b_l1hdr
.b_acb
= acb
;
5713 mutex_exit(hash_lock
);
5716 mutex_exit(hash_lock
);
5720 ASSERT(hdr
->b_l1hdr
.b_state
== arc_mru
||
5721 hdr
->b_l1hdr
.b_state
== arc_mfu
);
5724 if (hdr
->b_flags
& ARC_FLAG_PREDICTIVE_PREFETCH
) {
5726 * This is a demand read which does not have to
5727 * wait for i/o because we did a predictive
5728 * prefetch i/o for it, which has completed.
5731 arc__demand__hit__predictive__prefetch
,
5732 arc_buf_hdr_t
*, hdr
);
5734 arcstat_demand_hit_predictive_prefetch
);
5735 arc_hdr_clear_flags(hdr
,
5736 ARC_FLAG_PREDICTIVE_PREFETCH
);
5739 if (hdr
->b_flags
& ARC_FLAG_PRESCIENT_PREFETCH
) {
5741 arcstat_demand_hit_prescient_prefetch
);
5742 arc_hdr_clear_flags(hdr
,
5743 ARC_FLAG_PRESCIENT_PREFETCH
);
5746 ASSERT(!BP_IS_EMBEDDED(bp
) || !BP_IS_HOLE(bp
));
5748 arc_hdr_verify_checksum(spa
, hdr
, bp
);
5750 /* Get a buf with the desired data in it. */
5751 rc
= arc_buf_alloc_impl(hdr
, spa
, zb
, private,
5752 encrypted_read
, compressed_read
, noauth_read
,
5756 * Convert authentication and decryption errors
5757 * to EIO (and generate an ereport if needed)
5758 * before leaving the ARC.
5760 rc
= SET_ERROR(EIO
);
5761 if ((zio_flags
& ZIO_FLAG_SPECULATIVE
) == 0) {
5762 spa_log_error(spa
, zb
);
5763 (void) zfs_ereport_post(
5764 FM_EREPORT_ZFS_AUTHENTICATION
,
5765 spa
, NULL
, zb
, NULL
, 0, 0);
5769 (void) remove_reference(hdr
, hash_lock
,
5771 arc_buf_destroy_impl(buf
);
5774 /* assert any errors weren't due to unloaded keys */
5775 ASSERT((zio_flags
& ZIO_FLAG_SPECULATIVE
) ||
5777 } else if (*arc_flags
& ARC_FLAG_PREFETCH
&&
5778 zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
)) {
5779 if (HDR_HAS_L2HDR(hdr
))
5780 l2arc_hdr_arcstats_decrement_state(hdr
);
5781 arc_hdr_set_flags(hdr
, ARC_FLAG_PREFETCH
);
5782 if (HDR_HAS_L2HDR(hdr
))
5783 l2arc_hdr_arcstats_increment_state(hdr
);
5785 DTRACE_PROBE1(arc__hit
, arc_buf_hdr_t
*, hdr
);
5786 arc_access(hdr
, hash_lock
);
5787 if (*arc_flags
& ARC_FLAG_PRESCIENT_PREFETCH
)
5788 arc_hdr_set_flags(hdr
, ARC_FLAG_PRESCIENT_PREFETCH
);
5789 if (*arc_flags
& ARC_FLAG_L2CACHE
)
5790 arc_hdr_set_flags(hdr
, ARC_FLAG_L2CACHE
);
5791 mutex_exit(hash_lock
);
5792 ARCSTAT_BUMP(arcstat_hits
);
5793 ARCSTAT_CONDSTAT(!HDR_PREFETCH(hdr
),
5794 demand
, prefetch
, !HDR_ISTYPE_METADATA(hdr
),
5795 data
, metadata
, hits
);
5798 done(NULL
, zb
, bp
, buf
, private);
5800 uint64_t lsize
= BP_GET_LSIZE(bp
);
5801 uint64_t psize
= BP_GET_PSIZE(bp
);
5802 arc_callback_t
*acb
;
5805 boolean_t devw
= B_FALSE
;
5808 int alloc_flags
= encrypted_read
? ARC_HDR_ALLOC_RDATA
: 0;
5811 /* this block is not in the cache */
5812 arc_buf_hdr_t
*exists
= NULL
;
5813 arc_buf_contents_t type
= BP_GET_BUFC_TYPE(bp
);
5814 hdr
= arc_hdr_alloc(spa_load_guid(spa
), psize
, lsize
,
5815 BP_IS_PROTECTED(bp
), BP_GET_COMPRESS(bp
), type
,
5818 if (!BP_IS_EMBEDDED(bp
)) {
5819 hdr
->b_dva
= *BP_IDENTITY(bp
);
5820 hdr
->b_birth
= BP_PHYSICAL_BIRTH(bp
);
5821 exists
= buf_hash_insert(hdr
, &hash_lock
);
5823 if (exists
!= NULL
) {
5824 /* somebody beat us to the hash insert */
5825 mutex_exit(hash_lock
);
5826 buf_discard_identity(hdr
);
5827 arc_hdr_destroy(hdr
);
5828 goto top
; /* restart the IO request */
5832 * This block is in the ghost cache or encrypted data
5833 * was requested and we didn't have it. If it was
5834 * L2-only (and thus didn't have an L1 hdr),
5835 * we realloc the header to add an L1 hdr.
5837 if (!HDR_HAS_L1HDR(hdr
)) {
5838 hdr
= arc_hdr_realloc(hdr
, hdr_l2only_cache
,
5842 if (GHOST_STATE(hdr
->b_l1hdr
.b_state
)) {
5843 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, ==, NULL
);
5844 ASSERT(!HDR_HAS_RABD(hdr
));
5845 ASSERT(!HDR_IO_IN_PROGRESS(hdr
));
5846 ASSERT0(zfs_refcount_count(
5847 &hdr
->b_l1hdr
.b_refcnt
));
5848 ASSERT3P(hdr
->b_l1hdr
.b_buf
, ==, NULL
);
5849 ASSERT3P(hdr
->b_l1hdr
.b_freeze_cksum
, ==, NULL
);
5850 } else if (HDR_IO_IN_PROGRESS(hdr
)) {
5852 * If this header already had an IO in progress
5853 * and we are performing another IO to fetch
5854 * encrypted data we must wait until the first
5855 * IO completes so as not to confuse
5856 * arc_read_done(). This should be very rare
5857 * and so the performance impact shouldn't
5860 cv_wait(&hdr
->b_l1hdr
.b_cv
, hash_lock
);
5861 mutex_exit(hash_lock
);
5866 * This is a delicate dance that we play here.
5867 * This hdr might be in the ghost list so we access
5868 * it to move it out of the ghost list before we
5869 * initiate the read. If it's a prefetch then
5870 * it won't have a callback so we'll remove the
5871 * reference that arc_buf_alloc_impl() created. We
5872 * do this after we've called arc_access() to
5873 * avoid hitting an assert in remove_reference().
5875 arc_adapt(arc_hdr_size(hdr
), hdr
->b_l1hdr
.b_state
);
5876 arc_access(hdr
, hash_lock
);
5877 arc_hdr_alloc_pabd(hdr
, alloc_flags
);
5880 if (encrypted_read
) {
5881 ASSERT(HDR_HAS_RABD(hdr
));
5882 size
= HDR_GET_PSIZE(hdr
);
5883 hdr_abd
= hdr
->b_crypt_hdr
.b_rabd
;
5884 zio_flags
|= ZIO_FLAG_RAW
;
5886 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, !=, NULL
);
5887 size
= arc_hdr_size(hdr
);
5888 hdr_abd
= hdr
->b_l1hdr
.b_pabd
;
5890 if (arc_hdr_get_compress(hdr
) != ZIO_COMPRESS_OFF
) {
5891 zio_flags
|= ZIO_FLAG_RAW_COMPRESS
;
5895 * For authenticated bp's, we do not ask the ZIO layer
5896 * to authenticate them since this will cause the entire
5897 * IO to fail if the key isn't loaded. Instead, we
5898 * defer authentication until arc_buf_fill(), which will
5899 * verify the data when the key is available.
5901 if (BP_IS_AUTHENTICATED(bp
))
5902 zio_flags
|= ZIO_FLAG_RAW_ENCRYPT
;
5905 if (*arc_flags
& ARC_FLAG_PREFETCH
&&
5906 zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
)) {
5907 if (HDR_HAS_L2HDR(hdr
))
5908 l2arc_hdr_arcstats_decrement_state(hdr
);
5909 arc_hdr_set_flags(hdr
, ARC_FLAG_PREFETCH
);
5910 if (HDR_HAS_L2HDR(hdr
))
5911 l2arc_hdr_arcstats_increment_state(hdr
);
5913 if (*arc_flags
& ARC_FLAG_PRESCIENT_PREFETCH
)
5914 arc_hdr_set_flags(hdr
, ARC_FLAG_PRESCIENT_PREFETCH
);
5916 if (*arc_flags
& ARC_FLAG_L2CACHE
)
5917 arc_hdr_set_flags(hdr
, ARC_FLAG_L2CACHE
);
5918 if (BP_IS_AUTHENTICATED(bp
))
5919 arc_hdr_set_flags(hdr
, ARC_FLAG_NOAUTH
);
5920 if (BP_GET_LEVEL(bp
) > 0)
5921 arc_hdr_set_flags(hdr
, ARC_FLAG_INDIRECT
);
5922 if (*arc_flags
& ARC_FLAG_PREDICTIVE_PREFETCH
)
5923 arc_hdr_set_flags(hdr
, ARC_FLAG_PREDICTIVE_PREFETCH
);
5924 ASSERT(!GHOST_STATE(hdr
->b_l1hdr
.b_state
));
5926 acb
= kmem_zalloc(sizeof (arc_callback_t
), KM_SLEEP
);
5927 acb
->acb_done
= done
;
5928 acb
->acb_private
= private;
5929 acb
->acb_compressed
= compressed_read
;
5930 acb
->acb_encrypted
= encrypted_read
;
5931 acb
->acb_noauth
= noauth_read
;
5934 ASSERT3P(hdr
->b_l1hdr
.b_acb
, ==, NULL
);
5935 hdr
->b_l1hdr
.b_acb
= acb
;
5936 arc_hdr_set_flags(hdr
, ARC_FLAG_IO_IN_PROGRESS
);
5938 if (HDR_HAS_L2HDR(hdr
) &&
5939 (vd
= hdr
->b_l2hdr
.b_dev
->l2ad_vdev
) != NULL
) {
5940 devw
= hdr
->b_l2hdr
.b_dev
->l2ad_writing
;
5941 addr
= hdr
->b_l2hdr
.b_daddr
;
5943 * Lock out L2ARC device removal.
5945 if (vdev_is_dead(vd
) ||
5946 !spa_config_tryenter(spa
, SCL_L2ARC
, vd
, RW_READER
))
5951 * We count both async reads and scrub IOs as asynchronous so
5952 * that both can be upgraded in the event of a cache hit while
5953 * the read IO is still in-flight.
5955 if (priority
== ZIO_PRIORITY_ASYNC_READ
||
5956 priority
== ZIO_PRIORITY_SCRUB
)
5957 arc_hdr_set_flags(hdr
, ARC_FLAG_PRIO_ASYNC_READ
);
5959 arc_hdr_clear_flags(hdr
, ARC_FLAG_PRIO_ASYNC_READ
);
5962 * At this point, we have a level 1 cache miss. Try again in
5963 * L2ARC if possible.
5965 ASSERT3U(HDR_GET_LSIZE(hdr
), ==, lsize
);
5967 DTRACE_PROBE4(arc__miss
, arc_buf_hdr_t
*, hdr
, blkptr_t
*, bp
,
5968 uint64_t, lsize
, zbookmark_phys_t
*, zb
);
5969 ARCSTAT_BUMP(arcstat_misses
);
5970 ARCSTAT_CONDSTAT(!HDR_PREFETCH(hdr
),
5971 demand
, prefetch
, !HDR_ISTYPE_METADATA(hdr
),
5972 data
, metadata
, misses
);
5974 if (vd
!= NULL
&& l2arc_ndev
!= 0 && !(l2arc_norw
&& devw
)) {
5976 * Read from the L2ARC if the following are true:
5977 * 1. The L2ARC vdev was previously cached.
5978 * 2. This buffer still has L2ARC metadata.
5979 * 3. This buffer isn't currently writing to the L2ARC.
5980 * 4. The L2ARC entry wasn't evicted, which may
5981 * also have invalidated the vdev.
5982 * 5. This isn't prefetch or l2arc_noprefetch is 0.
5984 if (HDR_HAS_L2HDR(hdr
) &&
5985 !HDR_L2_WRITING(hdr
) && !HDR_L2_EVICTED(hdr
) &&
5986 !(l2arc_noprefetch
&& HDR_PREFETCH(hdr
))) {
5987 l2arc_read_callback_t
*cb
;
5991 DTRACE_PROBE1(l2arc__hit
, arc_buf_hdr_t
*, hdr
);
5992 ARCSTAT_BUMP(arcstat_l2_hits
);
5994 cb
= kmem_zalloc(sizeof (l2arc_read_callback_t
),
5996 cb
->l2rcb_hdr
= hdr
;
5999 cb
->l2rcb_flags
= zio_flags
;
6002 * When Compressed ARC is disabled, but the
6003 * L2ARC block is compressed, arc_hdr_size()
6004 * will have returned LSIZE rather than PSIZE.
6006 if (HDR_GET_COMPRESS(hdr
) != ZIO_COMPRESS_OFF
&&
6007 !HDR_COMPRESSION_ENABLED(hdr
) &&
6008 HDR_GET_PSIZE(hdr
) != 0) {
6009 size
= HDR_GET_PSIZE(hdr
);
6012 asize
= vdev_psize_to_asize(vd
, size
);
6013 if (asize
!= size
) {
6014 abd
= abd_alloc_for_io(asize
,
6015 HDR_ISTYPE_METADATA(hdr
));
6016 cb
->l2rcb_abd
= abd
;
6021 ASSERT(addr
>= VDEV_LABEL_START_SIZE
&&
6022 addr
+ asize
<= vd
->vdev_psize
-
6023 VDEV_LABEL_END_SIZE
);
6026 * l2arc read. The SCL_L2ARC lock will be
6027 * released by l2arc_read_done().
6028 * Issue a null zio if the underlying buffer
6029 * was squashed to zero size by compression.
6031 ASSERT3U(arc_hdr_get_compress(hdr
), !=,
6032 ZIO_COMPRESS_EMPTY
);
6033 rzio
= zio_read_phys(pio
, vd
, addr
,
6036 l2arc_read_done
, cb
, priority
,
6037 zio_flags
| ZIO_FLAG_DONT_CACHE
|
6039 ZIO_FLAG_DONT_PROPAGATE
|
6040 ZIO_FLAG_DONT_RETRY
, B_FALSE
);
6041 acb
->acb_zio_head
= rzio
;
6043 if (hash_lock
!= NULL
)
6044 mutex_exit(hash_lock
);
6046 DTRACE_PROBE2(l2arc__read
, vdev_t
*, vd
,
6048 ARCSTAT_INCR(arcstat_l2_read_bytes
,
6049 HDR_GET_PSIZE(hdr
));
6051 if (*arc_flags
& ARC_FLAG_NOWAIT
) {
6056 ASSERT(*arc_flags
& ARC_FLAG_WAIT
);
6057 if (zio_wait(rzio
) == 0)
6060 /* l2arc read error; goto zio_read() */
6061 if (hash_lock
!= NULL
)
6062 mutex_enter(hash_lock
);
6064 DTRACE_PROBE1(l2arc__miss
,
6065 arc_buf_hdr_t
*, hdr
);
6066 ARCSTAT_BUMP(arcstat_l2_misses
);
6067 if (HDR_L2_WRITING(hdr
))
6068 ARCSTAT_BUMP(arcstat_l2_rw_clash
);
6069 spa_config_exit(spa
, SCL_L2ARC
, vd
);
6073 spa_config_exit(spa
, SCL_L2ARC
, vd
);
6074 if (l2arc_ndev
!= 0) {
6075 DTRACE_PROBE1(l2arc__miss
,
6076 arc_buf_hdr_t
*, hdr
);
6077 ARCSTAT_BUMP(arcstat_l2_misses
);
6081 rzio
= zio_read(pio
, spa
, bp
, hdr_abd
, size
,
6082 arc_read_done
, hdr
, priority
, zio_flags
, zb
);
6083 acb
->acb_zio_head
= rzio
;
6085 if (hash_lock
!= NULL
)
6086 mutex_exit(hash_lock
);
6088 if (*arc_flags
& ARC_FLAG_WAIT
)
6089 return (zio_wait(rzio
));
6091 ASSERT(*arc_flags
& ARC_FLAG_NOWAIT
);
6098 * Notify the arc that a block was freed, and thus will never be used again.
6101 arc_freed(spa_t
*spa
, const blkptr_t
*bp
)
6104 kmutex_t
*hash_lock
;
6105 uint64_t guid
= spa_load_guid(spa
);
6107 ASSERT(!BP_IS_EMBEDDED(bp
));
6109 hdr
= buf_hash_find(guid
, bp
, &hash_lock
);
6114 * We might be trying to free a block that is still doing I/O
6115 * (i.e. prefetch) or has a reference (i.e. a dedup-ed,
6116 * dmu_sync-ed block). If this block is being prefetched, then it
6117 * would still have the ARC_FLAG_IO_IN_PROGRESS flag set on the hdr
6118 * until the I/O completes. A block may also have a reference if it is
6119 * part of a dedup-ed, dmu_synced write. The dmu_sync() function would
6120 * have written the new block to its final resting place on disk but
6121 * without the dedup flag set. This would have left the hdr in the MRU
6122 * state and discoverable. When the txg finally syncs it detects that
6123 * the block was overridden in open context and issues an override I/O.
6124 * Since this is a dedup block, the override I/O will determine if the
6125 * block is already in the DDT. If so, then it will replace the io_bp
6126 * with the bp from the DDT and allow the I/O to finish. When the I/O
6127 * reaches the done callback, dbuf_write_override_done, it will
6128 * check to see if the io_bp and io_bp_override are identical.
6129 * If they are not, then it indicates that the bp was replaced with
6130 * the bp in the DDT and the override bp is freed. This allows
6131 * us to arrive here with a reference on a block that is being
6132 * freed. So if we have an I/O in progress, or a reference to
6133 * this hdr, then we don't destroy the hdr.
6135 if (!HDR_HAS_L1HDR(hdr
) || (!HDR_IO_IN_PROGRESS(hdr
) &&
6136 zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
))) {
6137 arc_change_state(arc_anon
, hdr
, hash_lock
);
6138 arc_hdr_destroy(hdr
);
6139 mutex_exit(hash_lock
);
6141 mutex_exit(hash_lock
);
6147 * Release this buffer from the cache, making it an anonymous buffer. This
6148 * must be done after a read and prior to modifying the buffer contents.
6149 * If the buffer has more than one reference, we must make
6150 * a new hdr for the buffer.
6153 arc_release(arc_buf_t
*buf
, void *tag
)
6156 * It would be nice to assert that if its DMU metadata (level >
6157 * 0 || it's the dnode file), then it must be syncing context.
6158 * But we don't know that information at this level.
6161 mutex_enter(&buf
->b_evict_lock
);
6163 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
6165 ASSERT(HDR_HAS_L1HDR(hdr
));
6168 * We don't grab the hash lock prior to this check, because if
6169 * the buffer's header is in the arc_anon state, it won't be
6170 * linked into the hash table.
6172 if (hdr
->b_l1hdr
.b_state
== arc_anon
) {
6173 mutex_exit(&buf
->b_evict_lock
);
6175 * If we are called from dmu_convert_mdn_block_to_raw(),
6176 * a write might be in progress. This is OK because
6177 * the caller won't change the content of this buffer,
6178 * only the flags (via arc_convert_to_raw()).
6180 /* ASSERT(!HDR_IO_IN_PROGRESS(hdr)); */
6181 ASSERT(!HDR_IN_HASH_TABLE(hdr
));
6182 ASSERT(!HDR_HAS_L2HDR(hdr
));
6183 ASSERT(HDR_EMPTY(hdr
));
6185 ASSERT3U(hdr
->b_l1hdr
.b_bufcnt
, ==, 1);
6186 ASSERT3S(zfs_refcount_count(&hdr
->b_l1hdr
.b_refcnt
), ==, 1);
6187 ASSERT(!list_link_active(&hdr
->b_l1hdr
.b_arc_node
));
6189 hdr
->b_l1hdr
.b_arc_access
= 0;
6192 * If the buf is being overridden then it may already
6193 * have a hdr that is not empty.
6195 buf_discard_identity(hdr
);
6201 kmutex_t
*hash_lock
= HDR_LOCK(hdr
);
6202 mutex_enter(hash_lock
);
6205 * Wait for any other IO for this hdr, as additional
6206 * buf(s) could be about to appear, in which case
6207 * we would not want to transition hdr to arc_anon.
6209 while (HDR_IO_IN_PROGRESS(hdr
)) {
6210 DTRACE_PROBE1(arc_release__io
, arc_buf_hdr_t
*, hdr
);
6211 cv_wait(&hdr
->b_l1hdr
.b_cv
, hash_lock
);
6215 * This assignment is only valid as long as the hash_lock is
6216 * held, we must be careful not to reference state or the
6217 * b_state field after dropping the lock.
6219 arc_state_t
*state
= hdr
->b_l1hdr
.b_state
;
6220 ASSERT3P(hash_lock
, ==, HDR_LOCK(hdr
));
6221 ASSERT3P(state
, !=, arc_anon
);
6223 /* this buffer is not on any list */
6224 ASSERT3S(zfs_refcount_count(&hdr
->b_l1hdr
.b_refcnt
), >, 0);
6226 if (HDR_HAS_L2HDR(hdr
)) {
6227 mutex_enter(&hdr
->b_l2hdr
.b_dev
->l2ad_mtx
);
6230 * We have to recheck this conditional again now that
6231 * we're holding the l2ad_mtx to prevent a race with
6232 * another thread which might be concurrently calling
6233 * l2arc_evict(). In that case, l2arc_evict() might have
6234 * destroyed the header's L2 portion as we were waiting
6235 * to acquire the l2ad_mtx.
6237 if (HDR_HAS_L2HDR(hdr
))
6238 arc_hdr_l2hdr_destroy(hdr
);
6240 mutex_exit(&hdr
->b_l2hdr
.b_dev
->l2ad_mtx
);
6244 * Do we have more than one buf?
6246 if (hdr
->b_l1hdr
.b_bufcnt
> 1) {
6247 arc_buf_hdr_t
*nhdr
;
6248 uint64_t spa
= hdr
->b_spa
;
6249 uint64_t psize
= HDR_GET_PSIZE(hdr
);
6250 uint64_t lsize
= HDR_GET_LSIZE(hdr
);
6251 boolean_t
protected = HDR_PROTECTED(hdr
);
6252 enum zio_compress compress
= arc_hdr_get_compress(hdr
);
6253 arc_buf_contents_t type
= arc_buf_type(hdr
);
6254 VERIFY3U(hdr
->b_type
, ==, type
);
6256 ASSERT(hdr
->b_l1hdr
.b_buf
!= buf
|| buf
->b_next
!= NULL
);
6257 (void) remove_reference(hdr
, hash_lock
, tag
);
6259 if (arc_buf_is_shared(buf
) && !ARC_BUF_COMPRESSED(buf
)) {
6260 ASSERT3P(hdr
->b_l1hdr
.b_buf
, !=, buf
);
6261 ASSERT(ARC_BUF_LAST(buf
));
6265 * Pull the data off of this hdr and attach it to
6266 * a new anonymous hdr. Also find the last buffer
6267 * in the hdr's buffer list.
6269 arc_buf_t
*lastbuf
= arc_buf_remove(hdr
, buf
);
6270 ASSERT3P(lastbuf
, !=, NULL
);
6273 * If the current arc_buf_t and the hdr are sharing their data
6274 * buffer, then we must stop sharing that block.
6276 if (arc_buf_is_shared(buf
)) {
6277 ASSERT3P(hdr
->b_l1hdr
.b_buf
, !=, buf
);
6278 VERIFY(!arc_buf_is_shared(lastbuf
));
6281 * First, sever the block sharing relationship between
6282 * buf and the arc_buf_hdr_t.
6284 arc_unshare_buf(hdr
, buf
);
6287 * Now we need to recreate the hdr's b_pabd. Since we
6288 * have lastbuf handy, we try to share with it, but if
6289 * we can't then we allocate a new b_pabd and copy the
6290 * data from buf into it.
6292 if (arc_can_share(hdr
, lastbuf
)) {
6293 arc_share_buf(hdr
, lastbuf
);
6295 arc_hdr_alloc_pabd(hdr
, ARC_HDR_DO_ADAPT
);
6296 abd_copy_from_buf(hdr
->b_l1hdr
.b_pabd
,
6297 buf
->b_data
, psize
);
6299 VERIFY3P(lastbuf
->b_data
, !=, NULL
);
6300 } else if (HDR_SHARED_DATA(hdr
)) {
6302 * Uncompressed shared buffers are always at the end
6303 * of the list. Compressed buffers don't have the
6304 * same requirements. This makes it hard to
6305 * simply assert that the lastbuf is shared so
6306 * we rely on the hdr's compression flags to determine
6307 * if we have a compressed, shared buffer.
6309 ASSERT(arc_buf_is_shared(lastbuf
) ||
6310 arc_hdr_get_compress(hdr
) != ZIO_COMPRESS_OFF
);
6311 ASSERT(!ARC_BUF_SHARED(buf
));
6313 ASSERT(hdr
->b_l1hdr
.b_pabd
!= NULL
|| HDR_HAS_RABD(hdr
));
6314 ASSERT3P(state
, !=, arc_l2c_only
);
6316 (void) zfs_refcount_remove_many(&state
->arcs_size
,
6317 arc_buf_size(buf
), buf
);
6319 if (zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
)) {
6320 ASSERT3P(state
, !=, arc_l2c_only
);
6321 (void) zfs_refcount_remove_many(
6322 &state
->arcs_esize
[type
],
6323 arc_buf_size(buf
), buf
);
6326 hdr
->b_l1hdr
.b_bufcnt
-= 1;
6327 if (ARC_BUF_ENCRYPTED(buf
))
6328 hdr
->b_crypt_hdr
.b_ebufcnt
-= 1;
6330 arc_cksum_verify(buf
);
6331 arc_buf_unwatch(buf
);
6333 /* if this is the last uncompressed buf free the checksum */
6334 if (!arc_hdr_has_uncompressed_buf(hdr
))
6335 arc_cksum_free(hdr
);
6337 mutex_exit(hash_lock
);
6340 * Allocate a new hdr. The new hdr will contain a b_pabd
6341 * buffer which will be freed in arc_write().
6343 nhdr
= arc_hdr_alloc(spa
, psize
, lsize
, protected,
6344 compress
, type
, HDR_HAS_RABD(hdr
));
6345 ASSERT3P(nhdr
->b_l1hdr
.b_buf
, ==, NULL
);
6346 ASSERT0(nhdr
->b_l1hdr
.b_bufcnt
);
6347 ASSERT0(zfs_refcount_count(&nhdr
->b_l1hdr
.b_refcnt
));
6348 VERIFY3U(nhdr
->b_type
, ==, type
);
6349 ASSERT(!HDR_SHARED_DATA(nhdr
));
6351 nhdr
->b_l1hdr
.b_buf
= buf
;
6352 nhdr
->b_l1hdr
.b_bufcnt
= 1;
6353 if (ARC_BUF_ENCRYPTED(buf
))
6354 nhdr
->b_crypt_hdr
.b_ebufcnt
= 1;
6355 (void) zfs_refcount_add(&nhdr
->b_l1hdr
.b_refcnt
, tag
);
6358 mutex_exit(&buf
->b_evict_lock
);
6359 (void) zfs_refcount_add_many(&arc_anon
->arcs_size
,
6360 arc_buf_size(buf
), buf
);
6362 mutex_exit(&buf
->b_evict_lock
);
6363 ASSERT(zfs_refcount_count(&hdr
->b_l1hdr
.b_refcnt
) == 1);
6364 /* protected by hash lock, or hdr is on arc_anon */
6365 ASSERT(!multilist_link_active(&hdr
->b_l1hdr
.b_arc_node
));
6366 ASSERT(!HDR_IO_IN_PROGRESS(hdr
));
6367 arc_change_state(arc_anon
, hdr
, hash_lock
);
6368 hdr
->b_l1hdr
.b_arc_access
= 0;
6370 mutex_exit(hash_lock
);
6371 buf_discard_identity(hdr
);
6377 arc_released(arc_buf_t
*buf
)
6381 mutex_enter(&buf
->b_evict_lock
);
6382 released
= (buf
->b_data
!= NULL
&&
6383 buf
->b_hdr
->b_l1hdr
.b_state
== arc_anon
);
6384 mutex_exit(&buf
->b_evict_lock
);
6390 arc_referenced(arc_buf_t
*buf
)
6394 mutex_enter(&buf
->b_evict_lock
);
6395 referenced
= (zfs_refcount_count(&buf
->b_hdr
->b_l1hdr
.b_refcnt
));
6396 mutex_exit(&buf
->b_evict_lock
);
6397 return (referenced
);
6402 arc_write_ready(zio_t
*zio
)
6404 arc_write_callback_t
*callback
= zio
->io_private
;
6405 arc_buf_t
*buf
= callback
->awcb_buf
;
6406 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
6407 blkptr_t
*bp
= zio
->io_bp
;
6408 uint64_t psize
= BP_IS_HOLE(bp
) ? 0 : BP_GET_PSIZE(bp
);
6410 ASSERT(HDR_HAS_L1HDR(hdr
));
6411 ASSERT(!zfs_refcount_is_zero(&buf
->b_hdr
->b_l1hdr
.b_refcnt
));
6412 ASSERT(hdr
->b_l1hdr
.b_bufcnt
> 0);
6415 * If we're reexecuting this zio because the pool suspended, then
6416 * cleanup any state that was previously set the first time the
6417 * callback was invoked.
6419 if (zio
->io_flags
& ZIO_FLAG_REEXECUTED
) {
6420 arc_cksum_free(hdr
);
6421 arc_buf_unwatch(buf
);
6422 if (hdr
->b_l1hdr
.b_pabd
!= NULL
) {
6423 if (arc_buf_is_shared(buf
)) {
6424 arc_unshare_buf(hdr
, buf
);
6426 arc_hdr_free_pabd(hdr
, B_FALSE
);
6430 if (HDR_HAS_RABD(hdr
))
6431 arc_hdr_free_pabd(hdr
, B_TRUE
);
6433 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, ==, NULL
);
6434 ASSERT(!HDR_HAS_RABD(hdr
));
6435 ASSERT(!HDR_SHARED_DATA(hdr
));
6436 ASSERT(!arc_buf_is_shared(buf
));
6438 callback
->awcb_ready(zio
, buf
, callback
->awcb_private
);
6440 if (HDR_IO_IN_PROGRESS(hdr
))
6441 ASSERT(zio
->io_flags
& ZIO_FLAG_REEXECUTED
);
6443 arc_hdr_set_flags(hdr
, ARC_FLAG_IO_IN_PROGRESS
);
6445 if (BP_IS_PROTECTED(bp
) != !!HDR_PROTECTED(hdr
))
6446 hdr
= arc_hdr_realloc_crypt(hdr
, BP_IS_PROTECTED(bp
));
6448 if (BP_IS_PROTECTED(bp
)) {
6449 /* ZIL blocks are written through zio_rewrite */
6450 ASSERT3U(BP_GET_TYPE(bp
), !=, DMU_OT_INTENT_LOG
);
6451 ASSERT(HDR_PROTECTED(hdr
));
6453 if (BP_SHOULD_BYTESWAP(bp
)) {
6454 if (BP_GET_LEVEL(bp
) > 0) {
6455 hdr
->b_l1hdr
.b_byteswap
= DMU_BSWAP_UINT64
;
6457 hdr
->b_l1hdr
.b_byteswap
=
6458 DMU_OT_BYTESWAP(BP_GET_TYPE(bp
));
6461 hdr
->b_l1hdr
.b_byteswap
= DMU_BSWAP_NUMFUNCS
;
6464 hdr
->b_crypt_hdr
.b_ot
= BP_GET_TYPE(bp
);
6465 hdr
->b_crypt_hdr
.b_dsobj
= zio
->io_bookmark
.zb_objset
;
6466 zio_crypt_decode_params_bp(bp
, hdr
->b_crypt_hdr
.b_salt
,
6467 hdr
->b_crypt_hdr
.b_iv
);
6468 zio_crypt_decode_mac_bp(bp
, hdr
->b_crypt_hdr
.b_mac
);
6472 * If this block was written for raw encryption but the zio layer
6473 * ended up only authenticating it, adjust the buffer flags now.
6475 if (BP_IS_AUTHENTICATED(bp
) && ARC_BUF_ENCRYPTED(buf
)) {
6476 arc_hdr_set_flags(hdr
, ARC_FLAG_NOAUTH
);
6477 buf
->b_flags
&= ~ARC_BUF_FLAG_ENCRYPTED
;
6478 if (BP_GET_COMPRESS(bp
) == ZIO_COMPRESS_OFF
)
6479 buf
->b_flags
&= ~ARC_BUF_FLAG_COMPRESSED
;
6480 } else if (BP_IS_HOLE(bp
) && ARC_BUF_ENCRYPTED(buf
)) {
6481 buf
->b_flags
&= ~ARC_BUF_FLAG_ENCRYPTED
;
6482 buf
->b_flags
&= ~ARC_BUF_FLAG_COMPRESSED
;
6485 /* this must be done after the buffer flags are adjusted */
6486 arc_cksum_compute(buf
);
6488 enum zio_compress compress
;
6489 if (BP_IS_HOLE(bp
) || BP_IS_EMBEDDED(bp
)) {
6490 compress
= ZIO_COMPRESS_OFF
;
6492 ASSERT3U(HDR_GET_LSIZE(hdr
), ==, BP_GET_LSIZE(bp
));
6493 compress
= BP_GET_COMPRESS(bp
);
6495 HDR_SET_PSIZE(hdr
, psize
);
6496 arc_hdr_set_compress(hdr
, compress
);
6498 if (zio
->io_error
!= 0 || psize
== 0)
6502 * Fill the hdr with data. If the buffer is encrypted we have no choice
6503 * but to copy the data into b_rabd. If the hdr is compressed, the data
6504 * we want is available from the zio, otherwise we can take it from
6507 * We might be able to share the buf's data with the hdr here. However,
6508 * doing so would cause the ARC to be full of linear ABDs if we write a
6509 * lot of shareable data. As a compromise, we check whether scattered
6510 * ABDs are allowed, and assume that if they are then the user wants
6511 * the ARC to be primarily filled with them regardless of the data being
6512 * written. Therefore, if they're allowed then we allocate one and copy
6513 * the data into it; otherwise, we share the data directly if we can.
6515 if (ARC_BUF_ENCRYPTED(buf
)) {
6516 ASSERT3U(psize
, >, 0);
6517 ASSERT(ARC_BUF_COMPRESSED(buf
));
6518 arc_hdr_alloc_pabd(hdr
, ARC_HDR_DO_ADAPT
|ARC_HDR_ALLOC_RDATA
);
6519 abd_copy(hdr
->b_crypt_hdr
.b_rabd
, zio
->io_abd
, psize
);
6520 } else if (zfs_abd_scatter_enabled
|| !arc_can_share(hdr
, buf
)) {
6522 * Ideally, we would always copy the io_abd into b_pabd, but the
6523 * user may have disabled compressed ARC, thus we must check the
6524 * hdr's compression setting rather than the io_bp's.
6526 if (BP_IS_ENCRYPTED(bp
)) {
6527 ASSERT3U(psize
, >, 0);
6528 arc_hdr_alloc_pabd(hdr
,
6529 ARC_HDR_DO_ADAPT
|ARC_HDR_ALLOC_RDATA
);
6530 abd_copy(hdr
->b_crypt_hdr
.b_rabd
, zio
->io_abd
, psize
);
6531 } else if (arc_hdr_get_compress(hdr
) != ZIO_COMPRESS_OFF
&&
6532 !ARC_BUF_COMPRESSED(buf
)) {
6533 ASSERT3U(psize
, >, 0);
6534 arc_hdr_alloc_pabd(hdr
, ARC_HDR_DO_ADAPT
);
6535 abd_copy(hdr
->b_l1hdr
.b_pabd
, zio
->io_abd
, psize
);
6537 ASSERT3U(zio
->io_orig_size
, ==, arc_hdr_size(hdr
));
6538 arc_hdr_alloc_pabd(hdr
, ARC_HDR_DO_ADAPT
);
6539 abd_copy_from_buf(hdr
->b_l1hdr
.b_pabd
, buf
->b_data
,
6543 ASSERT3P(buf
->b_data
, ==, abd_to_buf(zio
->io_orig_abd
));
6544 ASSERT3U(zio
->io_orig_size
, ==, arc_buf_size(buf
));
6545 ASSERT3U(hdr
->b_l1hdr
.b_bufcnt
, ==, 1);
6546 arc_share_buf(hdr
, buf
);
6550 arc_hdr_verify(hdr
, bp
);
6554 arc_write_children_ready(zio_t
*zio
)
6556 arc_write_callback_t
*callback
= zio
->io_private
;
6557 arc_buf_t
*buf
= callback
->awcb_buf
;
6559 callback
->awcb_children_ready(zio
, buf
, callback
->awcb_private
);
6563 * The SPA calls this callback for each physical write that happens on behalf
6564 * of a logical write. See the comment in dbuf_write_physdone() for details.
6567 arc_write_physdone(zio_t
*zio
)
6569 arc_write_callback_t
*cb
= zio
->io_private
;
6570 if (cb
->awcb_physdone
!= NULL
)
6571 cb
->awcb_physdone(zio
, cb
->awcb_buf
, cb
->awcb_private
);
6575 arc_write_done(zio_t
*zio
)
6577 arc_write_callback_t
*callback
= zio
->io_private
;
6578 arc_buf_t
*buf
= callback
->awcb_buf
;
6579 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
6581 ASSERT3P(hdr
->b_l1hdr
.b_acb
, ==, NULL
);
6583 if (zio
->io_error
== 0) {
6584 arc_hdr_verify(hdr
, zio
->io_bp
);
6586 if (BP_IS_HOLE(zio
->io_bp
) || BP_IS_EMBEDDED(zio
->io_bp
)) {
6587 buf_discard_identity(hdr
);
6589 hdr
->b_dva
= *BP_IDENTITY(zio
->io_bp
);
6590 hdr
->b_birth
= BP_PHYSICAL_BIRTH(zio
->io_bp
);
6593 ASSERT(HDR_EMPTY(hdr
));
6597 * If the block to be written was all-zero or compressed enough to be
6598 * embedded in the BP, no write was performed so there will be no
6599 * dva/birth/checksum. The buffer must therefore remain anonymous
6602 if (!HDR_EMPTY(hdr
)) {
6603 arc_buf_hdr_t
*exists
;
6604 kmutex_t
*hash_lock
;
6606 ASSERT3U(zio
->io_error
, ==, 0);
6608 arc_cksum_verify(buf
);
6610 exists
= buf_hash_insert(hdr
, &hash_lock
);
6611 if (exists
!= NULL
) {
6613 * This can only happen if we overwrite for
6614 * sync-to-convergence, because we remove
6615 * buffers from the hash table when we arc_free().
6617 if (zio
->io_flags
& ZIO_FLAG_IO_REWRITE
) {
6618 if (!BP_EQUAL(&zio
->io_bp_orig
, zio
->io_bp
))
6619 panic("bad overwrite, hdr=%p exists=%p",
6620 (void *)hdr
, (void *)exists
);
6621 ASSERT(zfs_refcount_is_zero(
6622 &exists
->b_l1hdr
.b_refcnt
));
6623 arc_change_state(arc_anon
, exists
, hash_lock
);
6624 arc_hdr_destroy(exists
);
6625 mutex_exit(hash_lock
);
6626 exists
= buf_hash_insert(hdr
, &hash_lock
);
6627 ASSERT3P(exists
, ==, NULL
);
6628 } else if (zio
->io_flags
& ZIO_FLAG_NOPWRITE
) {
6630 ASSERT(zio
->io_prop
.zp_nopwrite
);
6631 if (!BP_EQUAL(&zio
->io_bp_orig
, zio
->io_bp
))
6632 panic("bad nopwrite, hdr=%p exists=%p",
6633 (void *)hdr
, (void *)exists
);
6636 ASSERT(hdr
->b_l1hdr
.b_bufcnt
== 1);
6637 ASSERT(hdr
->b_l1hdr
.b_state
== arc_anon
);
6638 ASSERT(BP_GET_DEDUP(zio
->io_bp
));
6639 ASSERT(BP_GET_LEVEL(zio
->io_bp
) == 0);
6642 arc_hdr_clear_flags(hdr
, ARC_FLAG_IO_IN_PROGRESS
);
6643 /* if it's not anon, we are doing a scrub */
6644 if (exists
== NULL
&& hdr
->b_l1hdr
.b_state
== arc_anon
)
6645 arc_access(hdr
, hash_lock
);
6646 mutex_exit(hash_lock
);
6648 arc_hdr_clear_flags(hdr
, ARC_FLAG_IO_IN_PROGRESS
);
6651 ASSERT(!zfs_refcount_is_zero(&hdr
->b_l1hdr
.b_refcnt
));
6652 callback
->awcb_done(zio
, buf
, callback
->awcb_private
);
6654 abd_put(zio
->io_abd
);
6655 kmem_free(callback
, sizeof (arc_write_callback_t
));
6659 arc_write(zio_t
*pio
, spa_t
*spa
, uint64_t txg
, blkptr_t
*bp
, arc_buf_t
*buf
,
6660 boolean_t l2arc
, const zio_prop_t
*zp
, arc_write_done_func_t
*ready
,
6661 arc_write_done_func_t
*children_ready
, arc_write_done_func_t
*physdone
,
6662 arc_write_done_func_t
*done
, void *private, zio_priority_t priority
,
6663 int zio_flags
, const zbookmark_phys_t
*zb
)
6665 arc_buf_hdr_t
*hdr
= buf
->b_hdr
;
6666 arc_write_callback_t
*callback
;
6668 zio_prop_t localprop
= *zp
;
6670 ASSERT3P(ready
, !=, NULL
);
6671 ASSERT3P(done
, !=, NULL
);
6672 ASSERT(!HDR_IO_ERROR(hdr
));
6673 ASSERT(!HDR_IO_IN_PROGRESS(hdr
));
6674 ASSERT3P(hdr
->b_l1hdr
.b_acb
, ==, NULL
);
6675 ASSERT3U(hdr
->b_l1hdr
.b_bufcnt
, >, 0);
6677 arc_hdr_set_flags(hdr
, ARC_FLAG_L2CACHE
);
6679 if (ARC_BUF_ENCRYPTED(buf
)) {
6680 ASSERT(ARC_BUF_COMPRESSED(buf
));
6681 localprop
.zp_encrypt
= B_TRUE
;
6682 localprop
.zp_compress
= HDR_GET_COMPRESS(hdr
);
6684 localprop
.zp_byteorder
=
6685 (hdr
->b_l1hdr
.b_byteswap
== DMU_BSWAP_NUMFUNCS
) ?
6686 ZFS_HOST_BYTEORDER
: !ZFS_HOST_BYTEORDER
;
6687 bcopy(hdr
->b_crypt_hdr
.b_salt
, localprop
.zp_salt
,
6689 bcopy(hdr
->b_crypt_hdr
.b_iv
, localprop
.zp_iv
,
6691 bcopy(hdr
->b_crypt_hdr
.b_mac
, localprop
.zp_mac
,
6693 if (DMU_OT_IS_ENCRYPTED(localprop
.zp_type
)) {
6694 localprop
.zp_nopwrite
= B_FALSE
;
6695 localprop
.zp_copies
=
6696 MIN(localprop
.zp_copies
, SPA_DVAS_PER_BP
- 1);
6698 zio_flags
|= ZIO_FLAG_RAW
;
6699 } else if (ARC_BUF_COMPRESSED(buf
)) {
6700 ASSERT3U(HDR_GET_LSIZE(hdr
), !=, arc_buf_size(buf
));
6701 localprop
.zp_compress
= HDR_GET_COMPRESS(hdr
);
6702 zio_flags
|= ZIO_FLAG_RAW_COMPRESS
;
6705 callback
= kmem_zalloc(sizeof (arc_write_callback_t
), KM_SLEEP
);
6706 callback
->awcb_ready
= ready
;
6707 callback
->awcb_children_ready
= children_ready
;
6708 callback
->awcb_physdone
= physdone
;
6709 callback
->awcb_done
= done
;
6710 callback
->awcb_private
= private;
6711 callback
->awcb_buf
= buf
;
6714 * The hdr's b_pabd is now stale, free it now. A new data block
6715 * will be allocated when the zio pipeline calls arc_write_ready().
6717 if (hdr
->b_l1hdr
.b_pabd
!= NULL
) {
6719 * If the buf is currently sharing the data block with
6720 * the hdr then we need to break that relationship here.
6721 * The hdr will remain with a NULL data pointer and the
6722 * buf will take sole ownership of the block.
6724 if (arc_buf_is_shared(buf
)) {
6725 arc_unshare_buf(hdr
, buf
);
6727 arc_hdr_free_pabd(hdr
, B_FALSE
);
6729 VERIFY3P(buf
->b_data
, !=, NULL
);
6732 if (HDR_HAS_RABD(hdr
))
6733 arc_hdr_free_pabd(hdr
, B_TRUE
);
6735 if (!(zio_flags
& ZIO_FLAG_RAW
))
6736 arc_hdr_set_compress(hdr
, ZIO_COMPRESS_OFF
);
6738 ASSERT(!arc_buf_is_shared(buf
));
6739 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, ==, NULL
);
6741 zio
= zio_write(pio
, spa
, txg
, bp
,
6742 abd_get_from_buf(buf
->b_data
, HDR_GET_LSIZE(hdr
)),
6743 HDR_GET_LSIZE(hdr
), arc_buf_size(buf
), &localprop
, arc_write_ready
,
6744 (children_ready
!= NULL
) ? arc_write_children_ready
: NULL
,
6745 arc_write_physdone
, arc_write_done
, callback
,
6746 priority
, zio_flags
, zb
);
6752 arc_memory_throttle(spa_t
*spa
, uint64_t reserve
, uint64_t txg
)
6755 uint64_t available_memory
= ptob(freemem
);
6758 if (freemem
> physmem
* arc_lotsfree_percent
/ 100)
6761 if (txg
> spa
->spa_lowmem_last_txg
) {
6762 spa
->spa_lowmem_last_txg
= txg
;
6763 spa
->spa_lowmem_page_load
= 0;
6766 * If we are in pageout, we know that memory is already tight,
6767 * the arc is already going to be evicting, so we just want to
6768 * continue to let page writes occur as quickly as possible.
6770 if (curproc
== proc_pageout
) {
6771 if (spa
->spa_lowmem_page_load
>
6772 MAX(ptob(minfree
), available_memory
) / 4)
6773 return (SET_ERROR(ERESTART
));
6774 /* Note: reserve is inflated, so we deflate */
6775 atomic_add_64(&spa
->spa_lowmem_page_load
, reserve
/ 8);
6777 } else if (spa
->spa_lowmem_page_load
> 0 && arc_reclaim_needed()) {
6778 /* memory is low, delay before restarting */
6779 ARCSTAT_INCR(arcstat_memory_throttle_count
, 1);
6780 return (SET_ERROR(EAGAIN
));
6782 spa
->spa_lowmem_page_load
= 0;
6783 #endif /* _KERNEL */
6788 * In more extreme cases, return B_TRUE if system memory is tight enough
6789 * that ZFS should defer work requiring new allocations.
6792 arc_memory_is_low(void)
6795 if (freemem
< minfree
+ needfree
)
6797 #endif /* _KERNEL */
6802 arc_tempreserve_clear(uint64_t reserve
)
6804 atomic_add_64(&arc_tempreserve
, -reserve
);
6805 ASSERT((int64_t)arc_tempreserve
>= 0);
6809 arc_tempreserve_space(spa_t
*spa
, uint64_t reserve
, uint64_t txg
)
6814 if (reserve
> arc_c
/4 && !arc_no_grow
)
6815 arc_c
= MIN(arc_c_max
, reserve
* 4);
6816 if (reserve
> arc_c
)
6817 return (SET_ERROR(ENOMEM
));
6820 * Don't count loaned bufs as in flight dirty data to prevent long
6821 * network delays from blocking transactions that are ready to be
6822 * assigned to a txg.
6825 /* assert that it has not wrapped around */
6826 ASSERT3S(atomic_add_64_nv(&arc_loaned_bytes
, 0), >=, 0);
6828 anon_size
= MAX((int64_t)(zfs_refcount_count(&arc_anon
->arcs_size
) -
6829 arc_loaned_bytes
), 0);
6832 * Writes will, almost always, require additional memory allocations
6833 * in order to compress/encrypt/etc the data. We therefore need to
6834 * make sure that there is sufficient available memory for this.
6836 error
= arc_memory_throttle(spa
, reserve
, txg
);
6841 * Throttle writes when the amount of dirty data in the cache
6842 * gets too large. We try to keep the cache less than half full
6843 * of dirty blocks so that our sync times don't grow too large.
6845 * In the case of one pool being built on another pool, we want
6846 * to make sure we don't end up throttling the lower (backing)
6847 * pool when the upper pool is the majority contributor to dirty
6848 * data. To insure we make forward progress during throttling, we
6849 * also check the current pool's net dirty data and only throttle
6850 * if it exceeds zfs_arc_pool_dirty_percent of the anonymous dirty
6851 * data in the cache.
6853 * Note: if two requests come in concurrently, we might let them
6854 * both succeed, when one of them should fail. Not a huge deal.
6856 uint64_t total_dirty
= reserve
+ arc_tempreserve
+ anon_size
;
6857 uint64_t spa_dirty_anon
= spa_dirty_data(spa
);
6859 if (total_dirty
> arc_c
* zfs_arc_dirty_limit_percent
/ 100 &&
6860 anon_size
> arc_c
* zfs_arc_anon_limit_percent
/ 100 &&
6861 spa_dirty_anon
> anon_size
* zfs_arc_pool_dirty_percent
/ 100) {
6862 uint64_t meta_esize
=
6864 &arc_anon
->arcs_esize
[ARC_BUFC_METADATA
]);
6865 uint64_t data_esize
=
6866 zfs_refcount_count(&arc_anon
->arcs_esize
[ARC_BUFC_DATA
]);
6867 dprintf("failing, arc_tempreserve=%lluK anon_meta=%lluK "
6868 "anon_data=%lluK tempreserve=%lluK arc_c=%lluK\n",
6869 arc_tempreserve
>> 10, meta_esize
>> 10,
6870 data_esize
>> 10, reserve
>> 10, arc_c
>> 10);
6871 return (SET_ERROR(ERESTART
));
6873 atomic_add_64(&arc_tempreserve
, reserve
);
6878 arc_kstat_update_state(arc_state_t
*state
, kstat_named_t
*size
,
6879 kstat_named_t
*evict_data
, kstat_named_t
*evict_metadata
)
6881 size
->value
.ui64
= zfs_refcount_count(&state
->arcs_size
);
6882 evict_data
->value
.ui64
=
6883 zfs_refcount_count(&state
->arcs_esize
[ARC_BUFC_DATA
]);
6884 evict_metadata
->value
.ui64
=
6885 zfs_refcount_count(&state
->arcs_esize
[ARC_BUFC_METADATA
]);
6889 arc_kstat_update(kstat_t
*ksp
, int rw
)
6891 arc_stats_t
*as
= ksp
->ks_data
;
6893 if (rw
== KSTAT_WRITE
) {
6896 arc_kstat_update_state(arc_anon
,
6897 &as
->arcstat_anon_size
,
6898 &as
->arcstat_anon_evictable_data
,
6899 &as
->arcstat_anon_evictable_metadata
);
6900 arc_kstat_update_state(arc_mru
,
6901 &as
->arcstat_mru_size
,
6902 &as
->arcstat_mru_evictable_data
,
6903 &as
->arcstat_mru_evictable_metadata
);
6904 arc_kstat_update_state(arc_mru_ghost
,
6905 &as
->arcstat_mru_ghost_size
,
6906 &as
->arcstat_mru_ghost_evictable_data
,
6907 &as
->arcstat_mru_ghost_evictable_metadata
);
6908 arc_kstat_update_state(arc_mfu
,
6909 &as
->arcstat_mfu_size
,
6910 &as
->arcstat_mfu_evictable_data
,
6911 &as
->arcstat_mfu_evictable_metadata
);
6912 arc_kstat_update_state(arc_mfu_ghost
,
6913 &as
->arcstat_mfu_ghost_size
,
6914 &as
->arcstat_mfu_ghost_evictable_data
,
6915 &as
->arcstat_mfu_ghost_evictable_metadata
);
6917 ARCSTAT(arcstat_size
) = aggsum_value(&arc_size
);
6918 ARCSTAT(arcstat_meta_used
) = aggsum_value(&arc_meta_used
);
6919 ARCSTAT(arcstat_data_size
) = aggsum_value(&astat_data_size
);
6920 ARCSTAT(arcstat_metadata_size
) =
6921 aggsum_value(&astat_metadata_size
);
6922 ARCSTAT(arcstat_hdr_size
) = aggsum_value(&astat_hdr_size
);
6923 ARCSTAT(arcstat_other_size
) = aggsum_value(&astat_other_size
);
6924 ARCSTAT(arcstat_l2_hdr_size
) = aggsum_value(&astat_l2_hdr_size
);
6931 * This function *must* return indices evenly distributed between all
6932 * sublists of the multilist. This is needed due to how the ARC eviction
6933 * code is laid out; arc_evict_state() assumes ARC buffers are evenly
6934 * distributed between all sublists and uses this assumption when
6935 * deciding which sublist to evict from and how much to evict from it.
6938 arc_state_multilist_index_func(multilist_t
*ml
, void *obj
)
6940 arc_buf_hdr_t
*hdr
= obj
;
6943 * We rely on b_dva to generate evenly distributed index
6944 * numbers using buf_hash below. So, as an added precaution,
6945 * let's make sure we never add empty buffers to the arc lists.
6947 ASSERT(!HDR_EMPTY(hdr
));
6950 * The assumption here, is the hash value for a given
6951 * arc_buf_hdr_t will remain constant throughout its lifetime
6952 * (i.e. its b_spa, b_dva, and b_birth fields don't change).
6953 * Thus, we don't need to store the header's sublist index
6954 * on insertion, as this index can be recalculated on removal.
6956 * Also, the low order bits of the hash value are thought to be
6957 * distributed evenly. Otherwise, in the case that the multilist
6958 * has a power of two number of sublists, each sublists' usage
6959 * would not be evenly distributed.
6961 return (buf_hash(hdr
->b_spa
, &hdr
->b_dva
, hdr
->b_birth
) %
6962 multilist_get_num_sublists(ml
));
6966 arc_state_init(void)
6968 arc_anon
= &ARC_anon
;
6970 arc_mru_ghost
= &ARC_mru_ghost
;
6972 arc_mfu_ghost
= &ARC_mfu_ghost
;
6973 arc_l2c_only
= &ARC_l2c_only
;
6975 arc_mru
->arcs_list
[ARC_BUFC_METADATA
] =
6976 multilist_create(sizeof (arc_buf_hdr_t
),
6977 offsetof(arc_buf_hdr_t
, b_l1hdr
.b_arc_node
),
6978 arc_state_multilist_index_func
);
6979 arc_mru
->arcs_list
[ARC_BUFC_DATA
] =
6980 multilist_create(sizeof (arc_buf_hdr_t
),
6981 offsetof(arc_buf_hdr_t
, b_l1hdr
.b_arc_node
),
6982 arc_state_multilist_index_func
);
6983 arc_mru_ghost
->arcs_list
[ARC_BUFC_METADATA
] =
6984 multilist_create(sizeof (arc_buf_hdr_t
),
6985 offsetof(arc_buf_hdr_t
, b_l1hdr
.b_arc_node
),
6986 arc_state_multilist_index_func
);
6987 arc_mru_ghost
->arcs_list
[ARC_BUFC_DATA
] =
6988 multilist_create(sizeof (arc_buf_hdr_t
),
6989 offsetof(arc_buf_hdr_t
, b_l1hdr
.b_arc_node
),
6990 arc_state_multilist_index_func
);
6991 arc_mfu
->arcs_list
[ARC_BUFC_METADATA
] =
6992 multilist_create(sizeof (arc_buf_hdr_t
),
6993 offsetof(arc_buf_hdr_t
, b_l1hdr
.b_arc_node
),
6994 arc_state_multilist_index_func
);
6995 arc_mfu
->arcs_list
[ARC_BUFC_DATA
] =
6996 multilist_create(sizeof (arc_buf_hdr_t
),
6997 offsetof(arc_buf_hdr_t
, b_l1hdr
.b_arc_node
),
6998 arc_state_multilist_index_func
);
6999 arc_mfu_ghost
->arcs_list
[ARC_BUFC_METADATA
] =
7000 multilist_create(sizeof (arc_buf_hdr_t
),
7001 offsetof(arc_buf_hdr_t
, b_l1hdr
.b_arc_node
),
7002 arc_state_multilist_index_func
);
7003 arc_mfu_ghost
->arcs_list
[ARC_BUFC_DATA
] =
7004 multilist_create(sizeof (arc_buf_hdr_t
),
7005 offsetof(arc_buf_hdr_t
, b_l1hdr
.b_arc_node
),
7006 arc_state_multilist_index_func
);
7007 arc_l2c_only
->arcs_list
[ARC_BUFC_METADATA
] =
7008 multilist_create(sizeof (arc_buf_hdr_t
),
7009 offsetof(arc_buf_hdr_t
, b_l1hdr
.b_arc_node
),
7010 arc_state_multilist_index_func
);
7011 arc_l2c_only
->arcs_list
[ARC_BUFC_DATA
] =
7012 multilist_create(sizeof (arc_buf_hdr_t
),
7013 offsetof(arc_buf_hdr_t
, b_l1hdr
.b_arc_node
),
7014 arc_state_multilist_index_func
);
7016 zfs_refcount_create(&arc_anon
->arcs_esize
[ARC_BUFC_METADATA
]);
7017 zfs_refcount_create(&arc_anon
->arcs_esize
[ARC_BUFC_DATA
]);
7018 zfs_refcount_create(&arc_mru
->arcs_esize
[ARC_BUFC_METADATA
]);
7019 zfs_refcount_create(&arc_mru
->arcs_esize
[ARC_BUFC_DATA
]);
7020 zfs_refcount_create(&arc_mru_ghost
->arcs_esize
[ARC_BUFC_METADATA
]);
7021 zfs_refcount_create(&arc_mru_ghost
->arcs_esize
[ARC_BUFC_DATA
]);
7022 zfs_refcount_create(&arc_mfu
->arcs_esize
[ARC_BUFC_METADATA
]);
7023 zfs_refcount_create(&arc_mfu
->arcs_esize
[ARC_BUFC_DATA
]);
7024 zfs_refcount_create(&arc_mfu_ghost
->arcs_esize
[ARC_BUFC_METADATA
]);
7025 zfs_refcount_create(&arc_mfu_ghost
->arcs_esize
[ARC_BUFC_DATA
]);
7026 zfs_refcount_create(&arc_l2c_only
->arcs_esize
[ARC_BUFC_METADATA
]);
7027 zfs_refcount_create(&arc_l2c_only
->arcs_esize
[ARC_BUFC_DATA
]);
7029 zfs_refcount_create(&arc_anon
->arcs_size
);
7030 zfs_refcount_create(&arc_mru
->arcs_size
);
7031 zfs_refcount_create(&arc_mru_ghost
->arcs_size
);
7032 zfs_refcount_create(&arc_mfu
->arcs_size
);
7033 zfs_refcount_create(&arc_mfu_ghost
->arcs_size
);
7034 zfs_refcount_create(&arc_l2c_only
->arcs_size
);
7036 aggsum_init(&arc_meta_used
, 0);
7037 aggsum_init(&arc_size
, 0);
7038 aggsum_init(&astat_data_size
, 0);
7039 aggsum_init(&astat_metadata_size
, 0);
7040 aggsum_init(&astat_hdr_size
, 0);
7041 aggsum_init(&astat_other_size
, 0);
7042 aggsum_init(&astat_l2_hdr_size
, 0);
7044 arc_anon
->arcs_state
= ARC_STATE_ANON
;
7045 arc_mru
->arcs_state
= ARC_STATE_MRU
;
7046 arc_mru_ghost
->arcs_state
= ARC_STATE_MRU_GHOST
;
7047 arc_mfu
->arcs_state
= ARC_STATE_MFU
;
7048 arc_mfu_ghost
->arcs_state
= ARC_STATE_MFU_GHOST
;
7049 arc_l2c_only
->arcs_state
= ARC_STATE_L2C_ONLY
;
7053 arc_state_fini(void)
7055 zfs_refcount_destroy(&arc_anon
->arcs_esize
[ARC_BUFC_METADATA
]);
7056 zfs_refcount_destroy(&arc_anon
->arcs_esize
[ARC_BUFC_DATA
]);
7057 zfs_refcount_destroy(&arc_mru
->arcs_esize
[ARC_BUFC_METADATA
]);
7058 zfs_refcount_destroy(&arc_mru
->arcs_esize
[ARC_BUFC_DATA
]);
7059 zfs_refcount_destroy(&arc_mru_ghost
->arcs_esize
[ARC_BUFC_METADATA
]);
7060 zfs_refcount_destroy(&arc_mru_ghost
->arcs_esize
[ARC_BUFC_DATA
]);
7061 zfs_refcount_destroy(&arc_mfu
->arcs_esize
[ARC_BUFC_METADATA
]);
7062 zfs_refcount_destroy(&arc_mfu
->arcs_esize
[ARC_BUFC_DATA
]);
7063 zfs_refcount_destroy(&arc_mfu_ghost
->arcs_esize
[ARC_BUFC_METADATA
]);
7064 zfs_refcount_destroy(&arc_mfu_ghost
->arcs_esize
[ARC_BUFC_DATA
]);
7065 zfs_refcount_destroy(&arc_l2c_only
->arcs_esize
[ARC_BUFC_METADATA
]);
7066 zfs_refcount_destroy(&arc_l2c_only
->arcs_esize
[ARC_BUFC_DATA
]);
7068 zfs_refcount_destroy(&arc_anon
->arcs_size
);
7069 zfs_refcount_destroy(&arc_mru
->arcs_size
);
7070 zfs_refcount_destroy(&arc_mru_ghost
->arcs_size
);
7071 zfs_refcount_destroy(&arc_mfu
->arcs_size
);
7072 zfs_refcount_destroy(&arc_mfu_ghost
->arcs_size
);
7073 zfs_refcount_destroy(&arc_l2c_only
->arcs_size
);
7075 multilist_destroy(arc_mru
->arcs_list
[ARC_BUFC_METADATA
]);
7076 multilist_destroy(arc_mru_ghost
->arcs_list
[ARC_BUFC_METADATA
]);
7077 multilist_destroy(arc_mfu
->arcs_list
[ARC_BUFC_METADATA
]);
7078 multilist_destroy(arc_mfu_ghost
->arcs_list
[ARC_BUFC_METADATA
]);
7079 multilist_destroy(arc_mru
->arcs_list
[ARC_BUFC_DATA
]);
7080 multilist_destroy(arc_mru_ghost
->arcs_list
[ARC_BUFC_DATA
]);
7081 multilist_destroy(arc_mfu
->arcs_list
[ARC_BUFC_DATA
]);
7082 multilist_destroy(arc_mfu_ghost
->arcs_list
[ARC_BUFC_DATA
]);
7083 multilist_destroy(arc_l2c_only
->arcs_list
[ARC_BUFC_METADATA
]);
7084 multilist_destroy(arc_l2c_only
->arcs_list
[ARC_BUFC_DATA
]);
7086 aggsum_fini(&arc_meta_used
);
7087 aggsum_fini(&arc_size
);
7088 aggsum_fini(&astat_data_size
);
7089 aggsum_fini(&astat_metadata_size
);
7090 aggsum_fini(&astat_hdr_size
);
7091 aggsum_fini(&astat_other_size
);
7092 aggsum_fini(&astat_l2_hdr_size
);
7106 * allmem is "all memory that we could possibly use".
7109 uint64_t allmem
= ptob(physmem
- swapfs_minfree
);
7111 uint64_t allmem
= (physmem
* PAGESIZE
) / 2;
7113 mutex_init(&arc_adjust_lock
, NULL
, MUTEX_DEFAULT
, NULL
);
7114 cv_init(&arc_adjust_waiters_cv
, NULL
, CV_DEFAULT
, NULL
);
7117 * Set the minimum cache size to 1/64 of all memory, with a hard
7120 arc_c_min
= MAX(allmem
/ 64, 64 << 20);
7122 * In a system with a lot of physical memory this will still result in
7123 * an ARC size floor that is quite large in absolute terms. Cap the
7124 * growth of the value at 1GB.
7126 arc_c_min
= MIN(arc_c_min
, 1 << 30);
7128 /* set max to 3/4 of all memory, or all but 1GB, whichever is more */
7129 if (allmem
>= 1 << 30)
7130 arc_c_max
= allmem
- (1 << 30);
7132 arc_c_max
= arc_c_min
;
7133 arc_c_max
= MAX(allmem
* 3 / 4, arc_c_max
);
7136 * In userland, there's only the memory pressure that we artificially
7137 * create (see arc_available_memory()). Don't let arc_c get too
7138 * small, because it can cause transactions to be larger than
7139 * arc_c, causing arc_tempreserve_space() to fail.
7142 arc_c_min
= arc_c_max
/ 2;
7146 * Allow the tunables to override our calculations if they are
7147 * reasonable (ie. over 64MB)
7149 if (zfs_arc_max
> 64 << 20 && zfs_arc_max
< allmem
) {
7150 arc_c_max
= zfs_arc_max
;
7151 arc_c_min
= MIN(arc_c_min
, arc_c_max
);
7153 if (zfs_arc_min
> 64 << 20 && zfs_arc_min
<= arc_c_max
)
7154 arc_c_min
= zfs_arc_min
;
7157 arc_p
= (arc_c
>> 1);
7159 /* limit meta-data to 1/4 of the arc capacity */
7160 arc_meta_limit
= arc_c_max
/ 4;
7164 * Metadata is stored in the kernel's heap. Don't let us
7165 * use more than half the heap for the ARC.
7167 arc_meta_limit
= MIN(arc_meta_limit
,
7168 vmem_size(heap_arena
, VMEM_ALLOC
| VMEM_FREE
) / 2);
7171 /* Allow the tunable to override if it is reasonable */
7172 if (zfs_arc_meta_limit
> 0 && zfs_arc_meta_limit
<= arc_c_max
)
7173 arc_meta_limit
= zfs_arc_meta_limit
;
7175 if (zfs_arc_meta_min
> 0) {
7176 arc_meta_min
= zfs_arc_meta_min
;
7178 arc_meta_min
= arc_c_min
/ 2;
7181 if (zfs_arc_grow_retry
> 0)
7182 arc_grow_retry
= zfs_arc_grow_retry
;
7184 if (zfs_arc_shrink_shift
> 0)
7185 arc_shrink_shift
= zfs_arc_shrink_shift
;
7188 * Ensure that arc_no_grow_shift is less than arc_shrink_shift.
7190 if (arc_no_grow_shift
>= arc_shrink_shift
)
7191 arc_no_grow_shift
= arc_shrink_shift
- 1;
7193 if (zfs_arc_p_min_shift
> 0)
7194 arc_p_min_shift
= zfs_arc_p_min_shift
;
7196 /* if kmem_flags are set, lets try to use less memory */
7197 if (kmem_debugging())
7199 if (arc_c
< arc_c_min
)
7205 * The arc must be "uninitialized", so that hdr_recl() (which is
7206 * registered by buf_init()) will not access arc_reap_zthr before
7209 ASSERT(!arc_initialized
);
7212 arc_ksp
= kstat_create("zfs", 0, "arcstats", "misc", KSTAT_TYPE_NAMED
,
7213 sizeof (arc_stats
) / sizeof (kstat_named_t
), KSTAT_FLAG_VIRTUAL
);
7215 if (arc_ksp
!= NULL
) {
7216 arc_ksp
->ks_data
= &arc_stats
;
7217 arc_ksp
->ks_update
= arc_kstat_update
;
7218 kstat_install(arc_ksp
);
7221 arc_adjust_zthr
= zthr_create(arc_adjust_cb_check
,
7222 arc_adjust_cb
, NULL
);
7223 arc_reap_zthr
= zthr_create_timer(arc_reap_cb_check
,
7224 arc_reap_cb
, NULL
, SEC2NSEC(1));
7226 arc_initialized
= B_TRUE
;
7230 * Calculate maximum amount of dirty data per pool.
7232 * If it has been set by /etc/system, take that.
7233 * Otherwise, use a percentage of physical memory defined by
7234 * zfs_dirty_data_max_percent (default 10%) with a cap at
7235 * zfs_dirty_data_max_max (default 4GB).
7237 if (zfs_dirty_data_max
== 0) {
7238 zfs_dirty_data_max
= physmem
* PAGESIZE
*
7239 zfs_dirty_data_max_percent
/ 100;
7240 zfs_dirty_data_max
= MIN(zfs_dirty_data_max
,
7241 zfs_dirty_data_max_max
);
7248 /* Use B_TRUE to ensure *all* buffers are evicted */
7249 arc_flush(NULL
, B_TRUE
);
7251 arc_initialized
= B_FALSE
;
7253 if (arc_ksp
!= NULL
) {
7254 kstat_delete(arc_ksp
);
7258 (void) zthr_cancel(arc_adjust_zthr
);
7259 zthr_destroy(arc_adjust_zthr
);
7261 (void) zthr_cancel(arc_reap_zthr
);
7262 zthr_destroy(arc_reap_zthr
);
7264 mutex_destroy(&arc_adjust_lock
);
7265 cv_destroy(&arc_adjust_waiters_cv
);
7268 * buf_fini() must proceed arc_state_fini() because buf_fin() may
7269 * trigger the release of kmem magazines, which can callback to
7270 * arc_space_return() which accesses aggsums freed in act_state_fini().
7275 ASSERT0(arc_loaned_bytes
);
7281 * The level 2 ARC (L2ARC) is a cache layer in-between main memory and disk.
7282 * It uses dedicated storage devices to hold cached data, which are populated
7283 * using large infrequent writes. The main role of this cache is to boost
7284 * the performance of random read workloads. The intended L2ARC devices
7285 * include short-stroked disks, solid state disks, and other media with
7286 * substantially faster read latency than disk.
7288 * +-----------------------+
7290 * +-----------------------+
7293 * l2arc_feed_thread() arc_read()
7297 * +---------------+ |
7299 * +---------------+ |
7304 * +-------+ +-------+
7306 * | cache | | cache |
7307 * +-------+ +-------+
7308 * +=========+ .-----.
7309 * : L2ARC : |-_____-|
7310 * : devices : | Disks |
7311 * +=========+ `-_____-'
7313 * Read requests are satisfied from the following sources, in order:
7316 * 2) vdev cache of L2ARC devices
7318 * 4) vdev cache of disks
7321 * Some L2ARC device types exhibit extremely slow write performance.
7322 * To accommodate for this there are some significant differences between
7323 * the L2ARC and traditional cache design:
7325 * 1. There is no eviction path from the ARC to the L2ARC. Evictions from
7326 * the ARC behave as usual, freeing buffers and placing headers on ghost
7327 * lists. The ARC does not send buffers to the L2ARC during eviction as
7328 * this would add inflated write latencies for all ARC memory pressure.
7330 * 2. The L2ARC attempts to cache data from the ARC before it is evicted.
7331 * It does this by periodically scanning buffers from the eviction-end of
7332 * the MFU and MRU ARC lists, copying them to the L2ARC devices if they are
7333 * not already there. It scans until a headroom of buffers is satisfied,
7334 * which itself is a buffer for ARC eviction. If a compressible buffer is
7335 * found during scanning and selected for writing to an L2ARC device, we
7336 * temporarily boost scanning headroom during the next scan cycle to make
7337 * sure we adapt to compression effects (which might significantly reduce
7338 * the data volume we write to L2ARC). The thread that does this is
7339 * l2arc_feed_thread(), illustrated below; example sizes are included to
7340 * provide a better sense of ratio than this diagram:
7343 * +---------------------+----------+
7344 * ARC_mfu |:::::#:::::::::::::::|o#o###o###|-->. # already on L2ARC
7345 * +---------------------+----------+ | o L2ARC eligible
7346 * ARC_mru |:#:::::::::::::::::::|#o#ooo####|-->| : ARC buffer
7347 * +---------------------+----------+ |
7348 * 15.9 Gbytes ^ 32 Mbytes |
7350 * l2arc_feed_thread()
7352 * l2arc write hand <--[oooo]--'
7356 * +==============================+
7357 * L2ARC dev |####|#|###|###| |####| ... |
7358 * +==============================+
7361 * 3. If an ARC buffer is copied to the L2ARC but then hit instead of
7362 * evicted, then the L2ARC has cached a buffer much sooner than it probably
7363 * needed to, potentially wasting L2ARC device bandwidth and storage. It is
7364 * safe to say that this is an uncommon case, since buffers at the end of
7365 * the ARC lists have moved there due to inactivity.
7367 * 4. If the ARC evicts faster than the L2ARC can maintain a headroom,
7368 * then the L2ARC simply misses copying some buffers. This serves as a
7369 * pressure valve to prevent heavy read workloads from both stalling the ARC
7370 * with waits and clogging the L2ARC with writes. This also helps prevent
7371 * the potential for the L2ARC to churn if it attempts to cache content too
7372 * quickly, such as during backups of the entire pool.
7374 * 5. After system boot and before the ARC has filled main memory, there are
7375 * no evictions from the ARC and so the tails of the ARC_mfu and ARC_mru
7376 * lists can remain mostly static. Instead of searching from tail of these
7377 * lists as pictured, the l2arc_feed_thread() will search from the list heads
7378 * for eligible buffers, greatly increasing its chance of finding them.
7380 * The L2ARC device write speed is also boosted during this time so that
7381 * the L2ARC warms up faster. Since there have been no ARC evictions yet,
7382 * there are no L2ARC reads, and no fear of degrading read performance
7383 * through increased writes.
7385 * 6. Writes to the L2ARC devices are grouped and sent in-sequence, so that
7386 * the vdev queue can aggregate them into larger and fewer writes. Each
7387 * device is written to in a rotor fashion, sweeping writes through
7388 * available space then repeating.
7390 * 7. The L2ARC does not store dirty content. It never needs to flush
7391 * write buffers back to disk based storage.
7393 * 8. If an ARC buffer is written (and dirtied) which also exists in the
7394 * L2ARC, the now stale L2ARC buffer is immediately dropped.
7396 * The performance of the L2ARC can be tweaked by a number of tunables, which
7397 * may be necessary for different workloads:
7399 * l2arc_write_max max write bytes per interval
7400 * l2arc_write_boost extra write bytes during device warmup
7401 * l2arc_noprefetch skip caching prefetched buffers
7402 * l2arc_headroom number of max device writes to precache
7403 * l2arc_headroom_boost when we find compressed buffers during ARC
7404 * scanning, we multiply headroom by this
7405 * percentage factor for the next scan cycle,
7406 * since more compressed buffers are likely to
7408 * l2arc_feed_secs seconds between L2ARC writing
7410 * Tunables may be removed or added as future performance improvements are
7411 * integrated, and also may become zpool properties.
7413 * There are three key functions that control how the L2ARC warms up:
7415 * l2arc_write_eligible() check if a buffer is eligible to cache
7416 * l2arc_write_size() calculate how much to write
7417 * l2arc_write_interval() calculate sleep delay between writes
7419 * These three functions determine what to write, how much, and how quickly
7422 * L2ARC persistence:
7424 * When writing buffers to L2ARC, we periodically add some metadata to
7425 * make sure we can pick them up after reboot, thus dramatically reducing
7426 * the impact that any downtime has on the performance of storage systems
7427 * with large caches.
7429 * The implementation works fairly simply by integrating the following two
7432 * *) When writing to the L2ARC, we occasionally write a "l2arc log block",
7433 * which is an additional piece of metadata which describes what's been
7434 * written. This allows us to rebuild the arc_buf_hdr_t structures of the
7435 * main ARC buffers. There are 2 linked-lists of log blocks headed by
7436 * dh_start_lbps[2]. We alternate which chain we append to, so they are
7437 * time-wise and offset-wise interleaved, but that is an optimization rather
7438 * than for correctness. The log block also includes a pointer to the
7439 * previous block in its chain.
7441 * *) We reserve SPA_MINBLOCKSIZE of space at the start of each L2ARC device
7442 * for our header bookkeeping purposes. This contains a device header,
7443 * which contains our top-level reference structures. We update it each
7444 * time we write a new log block, so that we're able to locate it in the
7445 * L2ARC device. If this write results in an inconsistent device header
7446 * (e.g. due to power failure), we detect this by verifying the header's
7447 * checksum and simply fail to reconstruct the L2ARC after reboot.
7449 * Implementation diagram:
7451 * +=== L2ARC device (not to scale) ======================================+
7452 * | ___two newest log block pointers__.__________ |
7453 * | / \dh_start_lbps[1] |
7454 * | / \ \dh_start_lbps[0]|
7456 * ||L2 dev|....|lb |bufs |lb |bufs |lb |bufs |lb |bufs |lb |---(empty)---|
7457 * || hdr| ^ /^ /^ / / |
7458 * |+------+ ...--\-------/ \-----/--\------/ / |
7459 * | \--------------/ \--------------/ |
7460 * +======================================================================+
7462 * As can be seen on the diagram, rather than using a simple linked list,
7463 * we use a pair of linked lists with alternating elements. This is a
7464 * performance enhancement due to the fact that we only find out the
7465 * address of the next log block access once the current block has been
7466 * completely read in. Obviously, this hurts performance, because we'd be
7467 * keeping the device's I/O queue at only a 1 operation deep, thus
7468 * incurring a large amount of I/O round-trip latency. Having two lists
7469 * allows us to fetch two log blocks ahead of where we are currently
7470 * rebuilding L2ARC buffers.
7472 * On-device data structures:
7474 * L2ARC device header: l2arc_dev_hdr_phys_t
7475 * L2ARC log block: l2arc_log_blk_phys_t
7477 * L2ARC reconstruction:
7479 * When writing data, we simply write in the standard rotary fashion,
7480 * evicting buffers as we go and simply writing new data over them (writing
7481 * a new log block every now and then). This obviously means that once we
7482 * loop around the end of the device, we will start cutting into an already
7483 * committed log block (and its referenced data buffers), like so:
7485 * current write head__ __old tail
7488 * <--|bufs |lb |bufs |lb | |bufs |lb |bufs |lb |-->
7489 * ^ ^^^^^^^^^___________________________________
7491 * <<nextwrite>> may overwrite this blk and/or its bufs --'
7493 * When importing the pool, we detect this situation and use it to stop
7494 * our scanning process (see l2arc_rebuild).
7496 * There is one significant caveat to consider when rebuilding ARC contents
7497 * from an L2ARC device: what about invalidated buffers? Given the above
7498 * construction, we cannot update blocks which we've already written to amend
7499 * them to remove buffers which were invalidated. Thus, during reconstruction,
7500 * we might be populating the cache with buffers for data that's not on the
7501 * main pool anymore, or may have been overwritten!
7503 * As it turns out, this isn't a problem. Every arc_read request includes
7504 * both the DVA and, crucially, the birth TXG of the BP the caller is
7505 * looking for. So even if the cache were populated by completely rotten
7506 * blocks for data that had been long deleted and/or overwritten, we'll
7507 * never actually return bad data from the cache, since the DVA with the
7508 * birth TXG uniquely identify a block in space and time - once created,
7509 * a block is immutable on disk. The worst thing we have done is wasted
7510 * some time and memory at l2arc rebuild to reconstruct outdated ARC
7511 * entries that will get dropped from the l2arc as it is being updated
7514 * L2ARC buffers that have been evicted by l2arc_evict() ahead of the write
7515 * hand are not restored. This is done by saving the offset (in bytes)
7516 * l2arc_evict() has evicted to in the L2ARC device header and taking it
7517 * into account when restoring buffers.
7521 l2arc_write_eligible(uint64_t spa_guid
, arc_buf_hdr_t
*hdr
)
7524 * A buffer is *not* eligible for the L2ARC if it:
7525 * 1. belongs to a different spa.
7526 * 2. is already cached on the L2ARC.
7527 * 3. has an I/O in progress (it may be an incomplete read).
7528 * 4. is flagged not eligible (zfs property).
7529 * 5. is a prefetch and l2arc_noprefetch is set.
7531 if (hdr
->b_spa
!= spa_guid
|| HDR_HAS_L2HDR(hdr
) ||
7532 HDR_IO_IN_PROGRESS(hdr
) || !HDR_L2CACHE(hdr
) ||
7533 (l2arc_noprefetch
&& HDR_PREFETCH(hdr
)))
7540 l2arc_write_size(l2arc_dev_t
*dev
)
7542 uint64_t size
, dev_size
;
7545 * Make sure our globals have meaningful values in case the user
7548 size
= l2arc_write_max
;
7550 cmn_err(CE_NOTE
, "Bad value for l2arc_write_max, value must "
7551 "be greater than zero, resetting it to the default (%d)",
7553 size
= l2arc_write_max
= L2ARC_WRITE_SIZE
;
7556 if (arc_warm
== B_FALSE
)
7557 size
+= l2arc_write_boost
;
7560 * Make sure the write size does not exceed the size of the cache
7561 * device. This is important in l2arc_evict(), otherwise infinite
7562 * iteration can occur.
7564 dev_size
= dev
->l2ad_end
- dev
->l2ad_start
;
7565 if ((size
+ l2arc_log_blk_overhead(size
, dev
)) >= dev_size
) {
7566 cmn_err(CE_NOTE
, "l2arc_write_max or l2arc_write_boost "
7567 "plus the overhead of log blocks (persistent L2ARC, "
7568 "%" PRIu64
" bytes) exceeds the size of the cache device "
7569 "(guid %" PRIu64
"), resetting them to the default (%d)",
7570 l2arc_log_blk_overhead(size
, dev
),
7571 dev
->l2ad_vdev
->vdev_guid
, L2ARC_WRITE_SIZE
);
7572 size
= l2arc_write_max
= l2arc_write_boost
= L2ARC_WRITE_SIZE
;
7574 if (arc_warm
== B_FALSE
)
7575 size
+= l2arc_write_boost
;
7583 l2arc_write_interval(clock_t began
, uint64_t wanted
, uint64_t wrote
)
7585 clock_t interval
, next
, now
;
7588 * If the ARC lists are busy, increase our write rate; if the
7589 * lists are stale, idle back. This is achieved by checking
7590 * how much we previously wrote - if it was more than half of
7591 * what we wanted, schedule the next write much sooner.
7593 if (l2arc_feed_again
&& wrote
> (wanted
/ 2))
7594 interval
= (hz
* l2arc_feed_min_ms
) / 1000;
7596 interval
= hz
* l2arc_feed_secs
;
7598 now
= ddi_get_lbolt();
7599 next
= MAX(now
, MIN(now
+ interval
, began
+ interval
));
7605 * Cycle through L2ARC devices. This is how L2ARC load balances.
7606 * If a device is returned, this also returns holding the spa config lock.
7608 static l2arc_dev_t
*
7609 l2arc_dev_get_next(void)
7611 l2arc_dev_t
*first
, *next
= NULL
;
7614 * Lock out the removal of spas (spa_namespace_lock), then removal
7615 * of cache devices (l2arc_dev_mtx). Once a device has been selected,
7616 * both locks will be dropped and a spa config lock held instead.
7618 mutex_enter(&spa_namespace_lock
);
7619 mutex_enter(&l2arc_dev_mtx
);
7621 /* if there are no vdevs, there is nothing to do */
7622 if (l2arc_ndev
== 0)
7626 next
= l2arc_dev_last
;
7628 /* loop around the list looking for a non-faulted vdev */
7630 next
= list_head(l2arc_dev_list
);
7632 next
= list_next(l2arc_dev_list
, next
);
7634 next
= list_head(l2arc_dev_list
);
7637 /* if we have come back to the start, bail out */
7640 else if (next
== first
)
7643 } while (vdev_is_dead(next
->l2ad_vdev
) || next
->l2ad_rebuild
);
7645 /* if we were unable to find any usable vdevs, return NULL */
7646 if (vdev_is_dead(next
->l2ad_vdev
) || next
->l2ad_rebuild
)
7649 l2arc_dev_last
= next
;
7652 mutex_exit(&l2arc_dev_mtx
);
7655 * Grab the config lock to prevent the 'next' device from being
7656 * removed while we are writing to it.
7659 spa_config_enter(next
->l2ad_spa
, SCL_L2ARC
, next
, RW_READER
);
7660 mutex_exit(&spa_namespace_lock
);
7666 * Free buffers that were tagged for destruction.
7669 l2arc_do_free_on_write()
7672 l2arc_data_free_t
*df
, *df_prev
;
7674 mutex_enter(&l2arc_free_on_write_mtx
);
7675 buflist
= l2arc_free_on_write
;
7677 for (df
= list_tail(buflist
); df
; df
= df_prev
) {
7678 df_prev
= list_prev(buflist
, df
);
7679 ASSERT3P(df
->l2df_abd
, !=, NULL
);
7680 abd_free(df
->l2df_abd
);
7681 list_remove(buflist
, df
);
7682 kmem_free(df
, sizeof (l2arc_data_free_t
));
7685 mutex_exit(&l2arc_free_on_write_mtx
);
7689 * A write to a cache device has completed. Update all headers to allow
7690 * reads from these buffers to begin.
7693 l2arc_write_done(zio_t
*zio
)
7695 l2arc_write_callback_t
*cb
;
7696 l2arc_lb_abd_buf_t
*abd_buf
;
7697 l2arc_lb_ptr_buf_t
*lb_ptr_buf
;
7699 l2arc_dev_hdr_phys_t
*l2dhdr
;
7701 arc_buf_hdr_t
*head
, *hdr
, *hdr_prev
;
7702 kmutex_t
*hash_lock
;
7703 int64_t bytes_dropped
= 0;
7705 cb
= zio
->io_private
;
7706 ASSERT3P(cb
, !=, NULL
);
7707 dev
= cb
->l2wcb_dev
;
7708 l2dhdr
= dev
->l2ad_dev_hdr
;
7709 ASSERT3P(dev
, !=, NULL
);
7710 head
= cb
->l2wcb_head
;
7711 ASSERT3P(head
, !=, NULL
);
7712 buflist
= &dev
->l2ad_buflist
;
7713 ASSERT3P(buflist
, !=, NULL
);
7714 DTRACE_PROBE2(l2arc__iodone
, zio_t
*, zio
,
7715 l2arc_write_callback_t
*, cb
);
7718 * All writes completed, or an error was hit.
7721 mutex_enter(&dev
->l2ad_mtx
);
7722 for (hdr
= list_prev(buflist
, head
); hdr
; hdr
= hdr_prev
) {
7723 hdr_prev
= list_prev(buflist
, hdr
);
7725 hash_lock
= HDR_LOCK(hdr
);
7728 * We cannot use mutex_enter or else we can deadlock
7729 * with l2arc_write_buffers (due to swapping the order
7730 * the hash lock and l2ad_mtx are taken).
7732 if (!mutex_tryenter(hash_lock
)) {
7734 * Missed the hash lock. We must retry so we
7735 * don't leave the ARC_FLAG_L2_WRITING bit set.
7737 ARCSTAT_BUMP(arcstat_l2_writes_lock_retry
);
7740 * We don't want to rescan the headers we've
7741 * already marked as having been written out, so
7742 * we reinsert the head node so we can pick up
7743 * where we left off.
7745 list_remove(buflist
, head
);
7746 list_insert_after(buflist
, hdr
, head
);
7748 mutex_exit(&dev
->l2ad_mtx
);
7751 * We wait for the hash lock to become available
7752 * to try and prevent busy waiting, and increase
7753 * the chance we'll be able to acquire the lock
7754 * the next time around.
7756 mutex_enter(hash_lock
);
7757 mutex_exit(hash_lock
);
7762 * We could not have been moved into the arc_l2c_only
7763 * state while in-flight due to our ARC_FLAG_L2_WRITING
7764 * bit being set. Let's just ensure that's being enforced.
7766 ASSERT(HDR_HAS_L1HDR(hdr
));
7768 if (zio
->io_error
!= 0) {
7770 * Error - drop L2ARC entry.
7772 list_remove(buflist
, hdr
);
7773 arc_hdr_clear_flags(hdr
, ARC_FLAG_HAS_L2HDR
);
7775 uint64_t psize
= HDR_GET_PSIZE(hdr
);
7776 l2arc_hdr_arcstats_decrement(hdr
);
7779 vdev_psize_to_asize(dev
->l2ad_vdev
, psize
);
7780 (void) zfs_refcount_remove_many(&dev
->l2ad_alloc
,
7781 arc_hdr_size(hdr
), hdr
);
7785 * Allow ARC to begin reads and ghost list evictions to
7788 arc_hdr_clear_flags(hdr
, ARC_FLAG_L2_WRITING
);
7790 mutex_exit(hash_lock
);
7794 * Free the allocated abd buffers for writing the log blocks.
7795 * If the zio failed reclaim the allocated space and remove the
7796 * pointers to these log blocks from the log block pointer list
7797 * of the L2ARC device.
7799 while ((abd_buf
= list_remove_tail(&cb
->l2wcb_abd_list
)) != NULL
) {
7800 abd_free(abd_buf
->abd
);
7801 zio_buf_free(abd_buf
, sizeof (*abd_buf
));
7802 if (zio
->io_error
!= 0) {
7803 lb_ptr_buf
= list_remove_head(&dev
->l2ad_lbptr_list
);
7805 * L2BLK_GET_PSIZE returns aligned size for log
7809 L2BLK_GET_PSIZE((lb_ptr_buf
->lb_ptr
)->lbp_prop
);
7810 bytes_dropped
+= asize
;
7811 ARCSTAT_INCR(arcstat_l2_log_blk_asize
, -asize
);
7812 ARCSTAT_BUMPDOWN(arcstat_l2_log_blk_count
);
7813 zfs_refcount_remove_many(&dev
->l2ad_lb_asize
, asize
,
7815 zfs_refcount_remove(&dev
->l2ad_lb_count
, lb_ptr_buf
);
7816 kmem_free(lb_ptr_buf
->lb_ptr
,
7817 sizeof (l2arc_log_blkptr_t
));
7818 kmem_free(lb_ptr_buf
, sizeof (l2arc_lb_ptr_buf_t
));
7821 list_destroy(&cb
->l2wcb_abd_list
);
7823 if (zio
->io_error
!= 0) {
7824 ARCSTAT_BUMP(arcstat_l2_writes_error
);
7827 * Restore the lbps array in the header to its previous state.
7828 * If the list of log block pointers is empty, zero out the
7829 * log block pointers in the device header.
7831 lb_ptr_buf
= list_head(&dev
->l2ad_lbptr_list
);
7832 for (int i
= 0; i
< 2; i
++) {
7833 if (lb_ptr_buf
== NULL
) {
7835 * If the list is empty zero out the device
7836 * header. Otherwise zero out the second log
7837 * block pointer in the header.
7840 bzero(l2dhdr
, dev
->l2ad_dev_hdr_asize
);
7842 bzero(&l2dhdr
->dh_start_lbps
[i
],
7843 sizeof (l2arc_log_blkptr_t
));
7847 bcopy(lb_ptr_buf
->lb_ptr
, &l2dhdr
->dh_start_lbps
[i
],
7848 sizeof (l2arc_log_blkptr_t
));
7849 lb_ptr_buf
= list_next(&dev
->l2ad_lbptr_list
,
7854 atomic_inc_64(&l2arc_writes_done
);
7855 list_remove(buflist
, head
);
7856 ASSERT(!HDR_HAS_L1HDR(head
));
7857 kmem_cache_free(hdr_l2only_cache
, head
);
7858 mutex_exit(&dev
->l2ad_mtx
);
7860 ASSERT(dev
->l2ad_vdev
!= NULL
);
7861 vdev_space_update(dev
->l2ad_vdev
, -bytes_dropped
, 0, 0);
7863 l2arc_do_free_on_write();
7865 kmem_free(cb
, sizeof (l2arc_write_callback_t
));
7869 l2arc_untransform(zio_t
*zio
, l2arc_read_callback_t
*cb
)
7872 spa_t
*spa
= zio
->io_spa
;
7873 arc_buf_hdr_t
*hdr
= cb
->l2rcb_hdr
;
7874 blkptr_t
*bp
= zio
->io_bp
;
7875 uint8_t salt
[ZIO_DATA_SALT_LEN
];
7876 uint8_t iv
[ZIO_DATA_IV_LEN
];
7877 uint8_t mac
[ZIO_DATA_MAC_LEN
];
7878 boolean_t no_crypt
= B_FALSE
;
7881 * ZIL data is never be written to the L2ARC, so we don't need
7882 * special handling for its unique MAC storage.
7884 ASSERT3U(BP_GET_TYPE(bp
), !=, DMU_OT_INTENT_LOG
);
7885 ASSERT(MUTEX_HELD(HDR_LOCK(hdr
)));
7886 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, !=, NULL
);
7889 * If the data was encrypted, decrypt it now. Note that
7890 * we must check the bp here and not the hdr, since the
7891 * hdr does not have its encryption parameters updated
7892 * until arc_read_done().
7894 if (BP_IS_ENCRYPTED(bp
)) {
7895 abd_t
*eabd
= arc_get_data_abd(hdr
, arc_hdr_size(hdr
), hdr
,
7898 zio_crypt_decode_params_bp(bp
, salt
, iv
);
7899 zio_crypt_decode_mac_bp(bp
, mac
);
7901 ret
= spa_do_crypt_abd(B_FALSE
, spa
, &cb
->l2rcb_zb
,
7902 BP_GET_TYPE(bp
), BP_GET_DEDUP(bp
), BP_SHOULD_BYTESWAP(bp
),
7903 salt
, iv
, mac
, HDR_GET_PSIZE(hdr
), eabd
,
7904 hdr
->b_l1hdr
.b_pabd
, &no_crypt
);
7906 arc_free_data_abd(hdr
, eabd
, arc_hdr_size(hdr
), hdr
);
7911 * If we actually performed decryption, replace b_pabd
7912 * with the decrypted data. Otherwise we can just throw
7913 * our decryption buffer away.
7916 arc_free_data_abd(hdr
, hdr
->b_l1hdr
.b_pabd
,
7917 arc_hdr_size(hdr
), hdr
);
7918 hdr
->b_l1hdr
.b_pabd
= eabd
;
7921 arc_free_data_abd(hdr
, eabd
, arc_hdr_size(hdr
), hdr
);
7926 * If the L2ARC block was compressed, but ARC compression
7927 * is disabled we decompress the data into a new buffer and
7928 * replace the existing data.
7930 if (HDR_GET_COMPRESS(hdr
) != ZIO_COMPRESS_OFF
&&
7931 !HDR_COMPRESSION_ENABLED(hdr
)) {
7932 abd_t
*cabd
= arc_get_data_abd(hdr
, arc_hdr_size(hdr
), hdr
,
7934 void *tmp
= abd_borrow_buf(cabd
, arc_hdr_size(hdr
));
7936 ret
= zio_decompress_data(HDR_GET_COMPRESS(hdr
),
7937 hdr
->b_l1hdr
.b_pabd
, tmp
, HDR_GET_PSIZE(hdr
),
7938 HDR_GET_LSIZE(hdr
));
7940 abd_return_buf_copy(cabd
, tmp
, arc_hdr_size(hdr
));
7941 arc_free_data_abd(hdr
, cabd
, arc_hdr_size(hdr
), hdr
);
7945 abd_return_buf_copy(cabd
, tmp
, arc_hdr_size(hdr
));
7946 arc_free_data_abd(hdr
, hdr
->b_l1hdr
.b_pabd
,
7947 arc_hdr_size(hdr
), hdr
);
7948 hdr
->b_l1hdr
.b_pabd
= cabd
;
7950 zio
->io_size
= HDR_GET_LSIZE(hdr
);
7961 * A read to a cache device completed. Validate buffer contents before
7962 * handing over to the regular ARC routines.
7965 l2arc_read_done(zio_t
*zio
)
7968 l2arc_read_callback_t
*cb
= zio
->io_private
;
7970 kmutex_t
*hash_lock
;
7971 boolean_t valid_cksum
;
7972 boolean_t using_rdata
= (BP_IS_ENCRYPTED(&cb
->l2rcb_bp
) &&
7973 (cb
->l2rcb_flags
& ZIO_FLAG_RAW_ENCRYPT
));
7975 ASSERT3P(zio
->io_vd
, !=, NULL
);
7976 ASSERT(zio
->io_flags
& ZIO_FLAG_DONT_PROPAGATE
);
7978 spa_config_exit(zio
->io_spa
, SCL_L2ARC
, zio
->io_vd
);
7980 ASSERT3P(cb
, !=, NULL
);
7981 hdr
= cb
->l2rcb_hdr
;
7982 ASSERT3P(hdr
, !=, NULL
);
7984 hash_lock
= HDR_LOCK(hdr
);
7985 mutex_enter(hash_lock
);
7986 ASSERT3P(hash_lock
, ==, HDR_LOCK(hdr
));
7989 * If the data was read into a temporary buffer,
7990 * move it and free the buffer.
7992 if (cb
->l2rcb_abd
!= NULL
) {
7993 ASSERT3U(arc_hdr_size(hdr
), <, zio
->io_size
);
7994 if (zio
->io_error
== 0) {
7996 abd_copy(hdr
->b_crypt_hdr
.b_rabd
,
7997 cb
->l2rcb_abd
, arc_hdr_size(hdr
));
7999 abd_copy(hdr
->b_l1hdr
.b_pabd
,
8000 cb
->l2rcb_abd
, arc_hdr_size(hdr
));
8005 * The following must be done regardless of whether
8006 * there was an error:
8007 * - free the temporary buffer
8008 * - point zio to the real ARC buffer
8009 * - set zio size accordingly
8010 * These are required because zio is either re-used for
8011 * an I/O of the block in the case of the error
8012 * or the zio is passed to arc_read_done() and it
8015 abd_free(cb
->l2rcb_abd
);
8016 zio
->io_size
= zio
->io_orig_size
= arc_hdr_size(hdr
);
8019 ASSERT(HDR_HAS_RABD(hdr
));
8020 zio
->io_abd
= zio
->io_orig_abd
=
8021 hdr
->b_crypt_hdr
.b_rabd
;
8023 ASSERT3P(hdr
->b_l1hdr
.b_pabd
, !=, NULL
);
8024 zio
->io_abd
= zio
->io_orig_abd
= hdr
->b_l1hdr
.b_pabd
;
8028 ASSERT3P(zio
->io_abd
, !=, NULL
);
8031 * Check this survived the L2ARC journey.
8033 ASSERT(zio
->io_abd
== hdr
->b_l1hdr
.b_pabd
||
8034 (HDR_HAS_RABD(hdr
) && zio
->io_abd
== hdr
->b_crypt_hdr
.b_rabd
));
8035 zio
->io_bp_copy
= cb
->l2rcb_bp
; /* XXX fix in L2ARC 2.0 */
8036 zio
->io_bp
= &zio
->io_bp_copy
; /* XXX fix in L2ARC 2.0 */
8038 valid_cksum
= arc_cksum_is_equal(hdr
, zio
);
8041 * b_rabd will always match the data as it exists on disk if it is
8042 * being used. Therefore if we are reading into b_rabd we do not
8043 * attempt to untransform the data.
8045 if (valid_cksum
&& !using_rdata
)
8046 tfm_error
= l2arc_untransform(zio
, cb
);
8048 if (valid_cksum
&& tfm_error
== 0 && zio
->io_error
== 0 &&
8049 !HDR_L2_EVICTED(hdr
)) {
8050 mutex_exit(hash_lock
);
8051 zio
->io_private
= hdr
;
8055 * Buffer didn't survive caching. Increment stats and
8056 * reissue to the original storage device.
8058 if (zio
->io_error
!= 0) {
8059 ARCSTAT_BUMP(arcstat_l2_io_error
);
8061 zio
->io_error
= SET_ERROR(EIO
);
8063 if (!valid_cksum
|| tfm_error
!= 0)
8064 ARCSTAT_BUMP(arcstat_l2_cksum_bad
);
8067 * If there's no waiter, issue an async i/o to the primary
8068 * storage now. If there *is* a waiter, the caller must
8069 * issue the i/o in a context where it's OK to block.
8071 if (zio
->io_waiter
== NULL
) {
8072 zio_t
*pio
= zio_unique_parent(zio
);
8073 void *abd
= (using_rdata
) ?
8074 hdr
->b_crypt_hdr
.b_rabd
: hdr
->b_l1hdr
.b_pabd
;
8076 ASSERT(!pio
|| pio
->io_child_type
== ZIO_CHILD_LOGICAL
);
8078 zio
= zio_read(pio
, zio
->io_spa
, zio
->io_bp
,
8079 abd
, zio
->io_size
, arc_read_done
,
8080 hdr
, zio
->io_priority
, cb
->l2rcb_flags
,
8084 * Original ZIO will be freed, so we need to update
8085 * ARC header with the new ZIO pointer to be used
8086 * by zio_change_priority() in arc_read().
8088 for (struct arc_callback
*acb
= hdr
->b_l1hdr
.b_acb
;
8089 acb
!= NULL
; acb
= acb
->acb_next
)
8090 acb
->acb_zio_head
= zio
;
8092 mutex_exit(hash_lock
);
8095 mutex_exit(hash_lock
);
8099 kmem_free(cb
, sizeof (l2arc_read_callback_t
));
8103 * This is the list priority from which the L2ARC will search for pages to
8104 * cache. This is used within loops (0..3) to cycle through lists in the
8105 * desired order. This order can have a significant effect on cache
8108 * Currently the metadata lists are hit first, MFU then MRU, followed by
8109 * the data lists. This function returns a locked list, and also returns
8112 static multilist_sublist_t
*
8113 l2arc_sublist_lock(int list_num
)
8115 multilist_t
*ml
= NULL
;
8118 ASSERT(list_num
>= 0 && list_num
< L2ARC_FEED_TYPES
);
8122 ml
= arc_mfu
->arcs_list
[ARC_BUFC_METADATA
];
8125 ml
= arc_mru
->arcs_list
[ARC_BUFC_METADATA
];
8128 ml
= arc_mfu
->arcs_list
[ARC_BUFC_DATA
];
8131 ml
= arc_mru
->arcs_list
[ARC_BUFC_DATA
];
8138 * Return a randomly-selected sublist. This is acceptable
8139 * because the caller feeds only a little bit of data for each
8140 * call (8MB). Subsequent calls will result in different
8141 * sublists being selected.
8143 idx
= multilist_get_random_index(ml
);
8144 return (multilist_sublist_lock(ml
, idx
));
8148 * Calculates the maximum overhead of L2ARC metadata log blocks for a given
8149 * L2ARC write size. l2arc_evict and l2arc_write_size need to include this
8150 * overhead in processing to make sure there is enough headroom available
8151 * when writing buffers.
8153 static inline uint64_t
8154 l2arc_log_blk_overhead(uint64_t write_sz
, l2arc_dev_t
*dev
)
8156 if (dev
->l2ad_log_entries
== 0) {
8159 uint64_t log_entries
= write_sz
>> SPA_MINBLOCKSHIFT
;
8161 uint64_t log_blocks
= (log_entries
+
8162 dev
->l2ad_log_entries
- 1) /
8163 dev
->l2ad_log_entries
;
8165 return (vdev_psize_to_asize(dev
->l2ad_vdev
,
8166 sizeof (l2arc_log_blk_phys_t
)) * log_blocks
);
8171 * Evict buffers from the device write hand to the distance specified in
8172 * bytes. This distance may span populated buffers, it may span nothing.
8173 * This is clearing a region on the L2ARC device ready for writing.
8174 * If the 'all' boolean is set, every buffer is evicted.
8177 l2arc_evict(l2arc_dev_t
*dev
, uint64_t distance
, boolean_t all
)
8180 arc_buf_hdr_t
*hdr
, *hdr_prev
;
8181 kmutex_t
*hash_lock
;
8183 l2arc_lb_ptr_buf_t
*lb_ptr_buf
, *lb_ptr_buf_prev
;
8186 buflist
= &dev
->l2ad_buflist
;
8189 * We need to add in the worst case scenario of log block overhead.
8191 distance
+= l2arc_log_blk_overhead(distance
, dev
);
8195 if (dev
->l2ad_hand
>= (dev
->l2ad_end
- distance
)) {
8197 * When there is no space to accommodate upcoming writes,
8198 * evict to the end. Then bump the write and evict hands
8199 * to the start and iterate. This iteration does not
8200 * happen indefinitely as we make sure in
8201 * l2arc_write_size() that when the write hand is reset,
8202 * the write size does not exceed the end of the device.
8205 taddr
= dev
->l2ad_end
;
8207 taddr
= dev
->l2ad_hand
+ distance
;
8209 DTRACE_PROBE4(l2arc__evict
, l2arc_dev_t
*, dev
, list_t
*, buflist
,
8210 uint64_t, taddr
, boolean_t
, all
);
8213 * This check has to be placed after deciding whether to iterate
8216 if (!all
&& dev
->l2ad_first
) {
8218 * This is the first sweep through the device. There is
8225 * When rebuilding L2ARC we retrieve the evict hand from the header of
8226 * the device. Of note, l2arc_evict() does not actually delete buffers
8227 * from the cache device, but keeping track of the evict hand will be
8228 * useful when TRIM is implemented.
8230 dev
->l2ad_evict
= MAX(dev
->l2ad_evict
, taddr
);
8233 mutex_enter(&dev
->l2ad_mtx
);
8235 * We have to account for evicted log blocks. Run vdev_space_update()
8236 * on log blocks whose offset (in bytes) is before the evicted offset
8237 * (in bytes) by searching in the list of pointers to log blocks
8238 * present in the L2ARC device.
8240 for (lb_ptr_buf
= list_tail(&dev
->l2ad_lbptr_list
); lb_ptr_buf
;
8241 lb_ptr_buf
= lb_ptr_buf_prev
) {
8243 lb_ptr_buf_prev
= list_prev(&dev
->l2ad_lbptr_list
, lb_ptr_buf
);
8245 /* L2BLK_GET_PSIZE returns aligned size for log blocks */
8246 uint64_t asize
= L2BLK_GET_PSIZE(
8247 (lb_ptr_buf
->lb_ptr
)->lbp_prop
);
8250 * We don't worry about log blocks left behind (ie
8251 * lbp_payload_start < l2ad_hand) because l2arc_write_buffers()
8252 * will never write more than l2arc_evict() evicts.
8254 if (!all
&& l2arc_log_blkptr_valid(dev
, lb_ptr_buf
->lb_ptr
)) {
8257 vdev_space_update(dev
->l2ad_vdev
, -asize
, 0, 0);
8258 ARCSTAT_INCR(arcstat_l2_log_blk_asize
, -asize
);
8259 ARCSTAT_BUMPDOWN(arcstat_l2_log_blk_count
);
8260 zfs_refcount_remove_many(&dev
->l2ad_lb_asize
, asize
,
8262 zfs_refcount_remove(&dev
->l2ad_lb_count
, lb_ptr_buf
);
8263 list_remove(&dev
->l2ad_lbptr_list
, lb_ptr_buf
);
8264 kmem_free(lb_ptr_buf
->lb_ptr
,
8265 sizeof (l2arc_log_blkptr_t
));
8266 kmem_free(lb_ptr_buf
, sizeof (l2arc_lb_ptr_buf_t
));
8270 for (hdr
= list_tail(buflist
); hdr
; hdr
= hdr_prev
) {
8271 hdr_prev
= list_prev(buflist
, hdr
);
8273 ASSERT(!HDR_EMPTY(hdr
));
8274 hash_lock
= HDR_LOCK(hdr
);
8277 * We cannot use mutex_enter or else we can deadlock
8278 * with l2arc_write_buffers (due to swapping the order
8279 * the hash lock and l2ad_mtx are taken).
8281 if (!mutex_tryenter(hash_lock
)) {
8283 * Missed the hash lock. Retry.
8285 ARCSTAT_BUMP(arcstat_l2_evict_lock_retry
);
8286 mutex_exit(&dev
->l2ad_mtx
);
8287 mutex_enter(hash_lock
);
8288 mutex_exit(hash_lock
);
8293 * A header can't be on this list if it doesn't have L2 header.
8295 ASSERT(HDR_HAS_L2HDR(hdr
));
8297 /* Ensure this header has finished being written. */
8298 ASSERT(!HDR_L2_WRITING(hdr
));
8299 ASSERT(!HDR_L2_WRITE_HEAD(hdr
));
8301 if (!all
&& (hdr
->b_l2hdr
.b_daddr
>= dev
->l2ad_evict
||
8302 hdr
->b_l2hdr
.b_daddr
< dev
->l2ad_hand
)) {
8304 * We've evicted to the target address,
8305 * or the end of the device.
8307 mutex_exit(hash_lock
);
8311 if (!HDR_HAS_L1HDR(hdr
)) {
8312 ASSERT(!HDR_L2_READING(hdr
));
8314 * This doesn't exist in the ARC. Destroy.
8315 * arc_hdr_destroy() will call list_remove()
8316 * and decrement arcstat_l2_lsize.
8318 arc_change_state(arc_anon
, hdr
, hash_lock
);
8319 arc_hdr_destroy(hdr
);
8321 ASSERT(hdr
->b_l1hdr
.b_state
!= arc_l2c_only
);
8322 ARCSTAT_BUMP(arcstat_l2_evict_l1cached
);
8324 * Invalidate issued or about to be issued
8325 * reads, since we may be about to write
8326 * over this location.
8328 if (HDR_L2_READING(hdr
)) {
8329 ARCSTAT_BUMP(arcstat_l2_evict_reading
);
8330 arc_hdr_set_flags(hdr
, ARC_FLAG_L2_EVICTED
);
8333 arc_hdr_l2hdr_destroy(hdr
);
8335 mutex_exit(hash_lock
);
8337 mutex_exit(&dev
->l2ad_mtx
);
8341 * We need to check if we evict all buffers, otherwise we may iterate
8344 if (!all
&& rerun
) {
8346 * Bump device hand to the device start if it is approaching the
8347 * end. l2arc_evict() has already evicted ahead for this case.
8349 dev
->l2ad_hand
= dev
->l2ad_start
;
8350 dev
->l2ad_evict
= dev
->l2ad_start
;
8351 dev
->l2ad_first
= B_FALSE
;
8355 ASSERT3U(dev
->l2ad_hand
+ distance
, <, dev
->l2ad_end
);
8356 if (!dev
->l2ad_first
)
8357 ASSERT3U(dev
->l2ad_hand
, <, dev
->l2ad_evict
);
8361 * Handle any abd transforms that might be required for writing to the L2ARC.
8362 * If successful, this function will always return an abd with the data
8363 * transformed as it is on disk in a new abd of asize bytes.
8366 l2arc_apply_transforms(spa_t
*spa
, arc_buf_hdr_t
*hdr
, uint64_t asize
,
8371 abd_t
*cabd
= NULL
, *eabd
= NULL
, *to_write
= hdr
->b_l1hdr
.b_pabd
;
8372 enum zio_compress compress
= HDR_GET_COMPRESS(hdr
);
8373 uint64_t psize
= HDR_GET_PSIZE(hdr
);
8374 uint64_t size
= arc_hdr_size(hdr
);
8375 boolean_t ismd
= HDR_ISTYPE_METADATA(hdr
);
8376 boolean_t bswap
= (hdr
->b_l1hdr
.b_byteswap
!= DMU_BSWAP_NUMFUNCS
);
8377 dsl_crypto_key_t
*dck
= NULL
;
8378 uint8_t mac
[ZIO_DATA_MAC_LEN
] = { 0 };
8379 boolean_t no_crypt
= B_FALSE
;
8381 ASSERT((HDR_GET_COMPRESS(hdr
) != ZIO_COMPRESS_OFF
&&
8382 !HDR_COMPRESSION_ENABLED(hdr
)) ||
8383 HDR_ENCRYPTED(hdr
) || HDR_SHARED_DATA(hdr
) || psize
!= asize
);
8384 ASSERT3U(psize
, <=, asize
);
8387 * If this data simply needs its own buffer, we simply allocate it
8388 * and copy the data. This may be done to eliminate a dependency on a
8389 * shared buffer or to reallocate the buffer to match asize.
8391 if (HDR_HAS_RABD(hdr
) && asize
!= psize
) {
8392 ASSERT3U(asize
, >=, psize
);
8393 to_write
= abd_alloc_for_io(asize
, ismd
);
8394 abd_copy(to_write
, hdr
->b_crypt_hdr
.b_rabd
, psize
);
8396 abd_zero_off(to_write
, psize
, asize
- psize
);
8400 if ((compress
== ZIO_COMPRESS_OFF
|| HDR_COMPRESSION_ENABLED(hdr
)) &&
8401 !HDR_ENCRYPTED(hdr
)) {
8402 ASSERT3U(size
, ==, psize
);
8403 to_write
= abd_alloc_for_io(asize
, ismd
);
8404 abd_copy(to_write
, hdr
->b_l1hdr
.b_pabd
, size
);
8406 abd_zero_off(to_write
, size
, asize
- size
);
8410 if (compress
!= ZIO_COMPRESS_OFF
&& !HDR_COMPRESSION_ENABLED(hdr
)) {
8411 cabd
= abd_alloc_for_io(asize
, ismd
);
8412 tmp
= abd_borrow_buf(cabd
, asize
);
8414 psize
= zio_compress_data(compress
, to_write
, tmp
, size
);
8415 ASSERT3U(psize
, <=, HDR_GET_PSIZE(hdr
));
8417 bzero((char *)tmp
+ psize
, asize
- psize
);
8418 psize
= HDR_GET_PSIZE(hdr
);
8419 abd_return_buf_copy(cabd
, tmp
, asize
);
8423 if (HDR_ENCRYPTED(hdr
)) {
8424 eabd
= abd_alloc_for_io(asize
, ismd
);
8427 * If the dataset was disowned before the buffer
8428 * made it to this point, the key to re-encrypt
8429 * it won't be available. In this case we simply
8430 * won't write the buffer to the L2ARC.
8432 ret
= spa_keystore_lookup_key(spa
, hdr
->b_crypt_hdr
.b_dsobj
,
8437 ret
= zio_do_crypt_abd(B_TRUE
, &dck
->dck_key
,
8438 hdr
->b_crypt_hdr
.b_ot
, bswap
, hdr
->b_crypt_hdr
.b_salt
,
8439 hdr
->b_crypt_hdr
.b_iv
, mac
, psize
, to_write
, eabd
,
8445 abd_copy(eabd
, to_write
, psize
);
8448 abd_zero_off(eabd
, psize
, asize
- psize
);
8450 /* assert that the MAC we got here matches the one we saved */
8451 ASSERT0(bcmp(mac
, hdr
->b_crypt_hdr
.b_mac
, ZIO_DATA_MAC_LEN
));
8452 spa_keystore_dsl_key_rele(spa
, dck
, FTAG
);
8454 if (to_write
== cabd
)
8461 ASSERT3P(to_write
, !=, hdr
->b_l1hdr
.b_pabd
);
8462 *abd_out
= to_write
;
8467 spa_keystore_dsl_key_rele(spa
, dck
, FTAG
);
8478 l2arc_blk_fetch_done(zio_t
*zio
)
8480 l2arc_read_callback_t
*cb
;
8482 cb
= zio
->io_private
;
8483 if (cb
->l2rcb_abd
!= NULL
)
8484 abd_put(cb
->l2rcb_abd
);
8485 kmem_free(cb
, sizeof (l2arc_read_callback_t
));
8489 * Find and write ARC buffers to the L2ARC device.
8491 * An ARC_FLAG_L2_WRITING flag is set so that the L2ARC buffers are not valid
8492 * for reading until they have completed writing.
8493 * The headroom_boost is an in-out parameter used to maintain headroom boost
8494 * state between calls to this function.
8496 * Returns the number of bytes actually written (which may be smaller than
8497 * the delta by which the device hand has changed due to alignment and the
8498 * writing of log blocks).
8501 l2arc_write_buffers(spa_t
*spa
, l2arc_dev_t
*dev
, uint64_t target_sz
)
8503 arc_buf_hdr_t
*hdr
, *hdr_prev
, *head
;
8504 uint64_t write_asize
, write_psize
, write_lsize
, headroom
;
8506 l2arc_write_callback_t
*cb
= NULL
;
8508 uint64_t guid
= spa_load_guid(spa
);
8509 l2arc_dev_hdr_phys_t
*l2dhdr
= dev
->l2ad_dev_hdr
;
8511 ASSERT3P(dev
->l2ad_vdev
, !=, NULL
);
8514 write_lsize
= write_asize
= write_psize
= 0;
8516 head
= kmem_cache_alloc(hdr_l2only_cache
, KM_PUSHPAGE
);
8517 arc_hdr_set_flags(head
, ARC_FLAG_L2_WRITE_HEAD
| ARC_FLAG_HAS_L2HDR
);
8520 * Copy buffers for L2ARC writing.
8522 for (int try = 0; try < L2ARC_FEED_TYPES
; try++) {
8524 * If try == 1 or 3, we cache MRU metadata and data
8527 if (l2arc_mfuonly
) {
8528 if (try == 1 || try == 3)
8532 multilist_sublist_t
*mls
= l2arc_sublist_lock(try);
8533 uint64_t passed_sz
= 0;
8535 VERIFY3P(mls
, !=, NULL
);
8538 * L2ARC fast warmup.
8540 * Until the ARC is warm and starts to evict, read from the
8541 * head of the ARC lists rather than the tail.
8543 if (arc_warm
== B_FALSE
)
8544 hdr
= multilist_sublist_head(mls
);
8546 hdr
= multilist_sublist_tail(mls
);
8548 headroom
= target_sz
* l2arc_headroom
;
8549 if (zfs_compressed_arc_enabled
)
8550 headroom
= (headroom
* l2arc_headroom_boost
) / 100;
8552 for (; hdr
; hdr
= hdr_prev
) {
8553 kmutex_t
*hash_lock
;
8554 abd_t
*to_write
= NULL
;
8556 if (arc_warm
== B_FALSE
)
8557 hdr_prev
= multilist_sublist_next(mls
, hdr
);
8559 hdr_prev
= multilist_sublist_prev(mls
, hdr
);
8561 hash_lock
= HDR_LOCK(hdr
);
8562 if (!mutex_tryenter(hash_lock
)) {
8564 * Skip this buffer rather than waiting.
8569 passed_sz
+= HDR_GET_LSIZE(hdr
);
8570 if (l2arc_headroom
!= 0 && passed_sz
> headroom
) {
8574 mutex_exit(hash_lock
);
8578 if (!l2arc_write_eligible(guid
, hdr
)) {
8579 mutex_exit(hash_lock
);
8583 ASSERT(HDR_HAS_L1HDR(hdr
));
8585 ASSERT3U(HDR_GET_PSIZE(hdr
), >, 0);
8586 ASSERT3U(arc_hdr_size(hdr
), >, 0);
8587 ASSERT(hdr
->b_l1hdr
.b_pabd
!= NULL
||
8589 uint64_t psize
= HDR_GET_PSIZE(hdr
);
8590 uint64_t asize
= vdev_psize_to_asize(dev
->l2ad_vdev
,
8593 if ((write_asize
+ asize
) > target_sz
) {
8595 mutex_exit(hash_lock
);
8600 * We rely on the L1 portion of the header below, so
8601 * it's invalid for this header to have been evicted out
8602 * of the ghost cache, prior to being written out. The
8603 * ARC_FLAG_L2_WRITING bit ensures this won't happen.
8605 arc_hdr_set_flags(hdr
, ARC_FLAG_L2_WRITING
);
8608 * If this header has b_rabd, we can use this since it
8609 * must always match the data exactly as it exists on
8610 * disk. Otherwise, the L2ARC can normally use the
8611 * hdr's data, but if we're sharing data between the
8612 * hdr and one of its bufs, L2ARC needs its own copy of
8613 * the data so that the ZIO below can't race with the
8614 * buf consumer. To ensure that this copy will be
8615 * available for the lifetime of the ZIO and be cleaned
8616 * up afterwards, we add it to the l2arc_free_on_write
8617 * queue. If we need to apply any transforms to the
8618 * data (compression, encryption) we will also need the
8621 if (HDR_HAS_RABD(hdr
) && psize
== asize
) {
8622 to_write
= hdr
->b_crypt_hdr
.b_rabd
;
8623 } else if ((HDR_COMPRESSION_ENABLED(hdr
) ||
8624 HDR_GET_COMPRESS(hdr
) == ZIO_COMPRESS_OFF
) &&
8625 !HDR_ENCRYPTED(hdr
) && !HDR_SHARED_DATA(hdr
) &&
8627 to_write
= hdr
->b_l1hdr
.b_pabd
;
8630 arc_buf_contents_t type
= arc_buf_type(hdr
);
8632 ret
= l2arc_apply_transforms(spa
, hdr
, asize
,
8635 arc_hdr_clear_flags(hdr
,
8636 ARC_FLAG_L2_WRITING
);
8637 mutex_exit(hash_lock
);
8641 l2arc_free_abd_on_write(to_write
, asize
, type
);
8646 * Insert a dummy header on the buflist so
8647 * l2arc_write_done() can find where the
8648 * write buffers begin without searching.
8650 mutex_enter(&dev
->l2ad_mtx
);
8651 list_insert_head(&dev
->l2ad_buflist
, head
);
8652 mutex_exit(&dev
->l2ad_mtx
);
8655 sizeof (l2arc_write_callback_t
), KM_SLEEP
);
8656 cb
->l2wcb_dev
= dev
;
8657 cb
->l2wcb_head
= head
;
8659 * Create a list to save allocated abd buffers
8660 * for l2arc_log_blk_commit().
8662 list_create(&cb
->l2wcb_abd_list
,
8663 sizeof (l2arc_lb_abd_buf_t
),
8664 offsetof(l2arc_lb_abd_buf_t
, node
));
8665 pio
= zio_root(spa
, l2arc_write_done
, cb
,
8669 hdr
->b_l2hdr
.b_dev
= dev
;
8670 hdr
->b_l2hdr
.b_daddr
= dev
->l2ad_hand
;
8671 hdr
->b_l2hdr
.b_arcs_state
=
8672 hdr
->b_l1hdr
.b_state
->arcs_state
;
8673 arc_hdr_set_flags(hdr
,
8674 ARC_FLAG_L2_WRITING
| ARC_FLAG_HAS_L2HDR
);
8676 mutex_enter(&dev
->l2ad_mtx
);
8677 list_insert_head(&dev
->l2ad_buflist
, hdr
);
8678 mutex_exit(&dev
->l2ad_mtx
);
8680 (void) zfs_refcount_add_many(&dev
->l2ad_alloc
,
8681 arc_hdr_size(hdr
), hdr
);
8683 wzio
= zio_write_phys(pio
, dev
->l2ad_vdev
,
8684 hdr
->b_l2hdr
.b_daddr
, asize
, to_write
,
8685 ZIO_CHECKSUM_OFF
, NULL
, hdr
,
8686 ZIO_PRIORITY_ASYNC_WRITE
,
8687 ZIO_FLAG_CANFAIL
, B_FALSE
);
8689 write_lsize
+= HDR_GET_LSIZE(hdr
);
8690 DTRACE_PROBE2(l2arc__write
, vdev_t
*, dev
->l2ad_vdev
,
8693 write_psize
+= psize
;
8694 write_asize
+= asize
;
8695 dev
->l2ad_hand
+= asize
;
8696 l2arc_hdr_arcstats_increment(hdr
);
8697 vdev_space_update(dev
->l2ad_vdev
, asize
, 0, 0);
8699 mutex_exit(hash_lock
);
8702 * Append buf info to current log and commit if full.
8703 * arcstat_l2_{size,asize} kstats are updated
8706 if (l2arc_log_blk_insert(dev
, hdr
))
8707 l2arc_log_blk_commit(dev
, pio
, cb
);
8709 (void) zio_nowait(wzio
);
8712 multilist_sublist_unlock(mls
);
8718 /* No buffers selected for writing? */
8720 ASSERT0(write_lsize
);
8721 ASSERT(!HDR_HAS_L1HDR(head
));
8722 kmem_cache_free(hdr_l2only_cache
, head
);
8725 * Although we did not write any buffers l2ad_evict may
8728 if (dev
->l2ad_evict
!= l2dhdr
->dh_evict
)
8729 l2arc_dev_hdr_update(dev
);
8734 if (!dev
->l2ad_first
)
8735 ASSERT3U(dev
->l2ad_hand
, <=, dev
->l2ad_evict
);
8737 ASSERT3U(write_asize
, <=, target_sz
);
8738 ARCSTAT_BUMP(arcstat_l2_writes_sent
);
8739 ARCSTAT_INCR(arcstat_l2_write_bytes
, write_psize
);
8741 dev
->l2ad_writing
= B_TRUE
;
8742 (void) zio_wait(pio
);
8743 dev
->l2ad_writing
= B_FALSE
;
8746 * Update the device header after the zio completes as
8747 * l2arc_write_done() may have updated the memory holding the log block
8748 * pointers in the device header.
8750 l2arc_dev_hdr_update(dev
);
8752 return (write_asize
);
8756 l2arc_hdr_limit_reached(void)
8758 int64_t s
= aggsum_upper_bound(&astat_l2_hdr_size
);
8760 return (arc_reclaim_needed() || (s
> arc_meta_limit
* 3 / 4) ||
8761 (s
> (arc_warm
? arc_c
: arc_c_max
) * l2arc_meta_percent
/ 100));
8765 * This thread feeds the L2ARC at regular intervals. This is the beating
8766 * heart of the L2ARC.
8770 l2arc_feed_thread(void *unused
)
8775 uint64_t size
, wrote
;
8776 clock_t begin
, next
= ddi_get_lbolt();
8778 CALLB_CPR_INIT(&cpr
, &l2arc_feed_thr_lock
, callb_generic_cpr
, FTAG
);
8780 mutex_enter(&l2arc_feed_thr_lock
);
8782 while (l2arc_thread_exit
== 0) {
8783 CALLB_CPR_SAFE_BEGIN(&cpr
);
8784 (void) cv_timedwait(&l2arc_feed_thr_cv
, &l2arc_feed_thr_lock
,
8786 CALLB_CPR_SAFE_END(&cpr
, &l2arc_feed_thr_lock
);
8787 next
= ddi_get_lbolt() + hz
;
8790 * Quick check for L2ARC devices.
8792 mutex_enter(&l2arc_dev_mtx
);
8793 if (l2arc_ndev
== 0) {
8794 mutex_exit(&l2arc_dev_mtx
);
8797 mutex_exit(&l2arc_dev_mtx
);
8798 begin
= ddi_get_lbolt();
8801 * This selects the next l2arc device to write to, and in
8802 * doing so the next spa to feed from: dev->l2ad_spa. This
8803 * will return NULL if there are now no l2arc devices or if
8804 * they are all faulted.
8806 * If a device is returned, its spa's config lock is also
8807 * held to prevent device removal. l2arc_dev_get_next()
8808 * will grab and release l2arc_dev_mtx.
8810 if ((dev
= l2arc_dev_get_next()) == NULL
)
8813 spa
= dev
->l2ad_spa
;
8814 ASSERT3P(spa
, !=, NULL
);
8817 * If the pool is read-only then force the feed thread to
8818 * sleep a little longer.
8820 if (!spa_writeable(spa
)) {
8821 next
= ddi_get_lbolt() + 5 * l2arc_feed_secs
* hz
;
8822 spa_config_exit(spa
, SCL_L2ARC
, dev
);
8827 * Avoid contributing to memory pressure.
8829 if (l2arc_hdr_limit_reached()) {
8830 ARCSTAT_BUMP(arcstat_l2_abort_lowmem
);
8831 spa_config_exit(spa
, SCL_L2ARC
, dev
);
8835 ARCSTAT_BUMP(arcstat_l2_feeds
);
8837 size
= l2arc_write_size(dev
);
8840 * Evict L2ARC buffers that will be overwritten.
8842 l2arc_evict(dev
, size
, B_FALSE
);
8845 * Write ARC buffers.
8847 wrote
= l2arc_write_buffers(spa
, dev
, size
);
8850 * Calculate interval between writes.
8852 next
= l2arc_write_interval(begin
, size
, wrote
);
8853 spa_config_exit(spa
, SCL_L2ARC
, dev
);
8856 l2arc_thread_exit
= 0;
8857 cv_broadcast(&l2arc_feed_thr_cv
);
8858 CALLB_CPR_EXIT(&cpr
); /* drops l2arc_feed_thr_lock */
8863 l2arc_vdev_present(vdev_t
*vd
)
8865 return (l2arc_vdev_get(vd
) != NULL
);
8869 * Returns the l2arc_dev_t associated with a particular vdev_t or NULL if
8870 * the vdev_t isn't an L2ARC device.
8872 static l2arc_dev_t
*
8873 l2arc_vdev_get(vdev_t
*vd
)
8877 mutex_enter(&l2arc_dev_mtx
);
8878 for (dev
= list_head(l2arc_dev_list
); dev
!= NULL
;
8879 dev
= list_next(l2arc_dev_list
, dev
)) {
8880 if (dev
->l2ad_vdev
== vd
)
8883 mutex_exit(&l2arc_dev_mtx
);
8889 * Add a vdev for use by the L2ARC. By this point the spa has already
8890 * validated the vdev and opened it.
8893 l2arc_add_vdev(spa_t
*spa
, vdev_t
*vd
)
8895 l2arc_dev_t
*adddev
;
8896 uint64_t l2dhdr_asize
;
8898 ASSERT(!l2arc_vdev_present(vd
));
8901 * Create a new l2arc device entry.
8903 adddev
= kmem_zalloc(sizeof (l2arc_dev_t
), KM_SLEEP
);
8904 adddev
->l2ad_spa
= spa
;
8905 adddev
->l2ad_vdev
= vd
;
8906 /* leave extra size for an l2arc device header */
8907 l2dhdr_asize
= adddev
->l2ad_dev_hdr_asize
=
8908 MAX(sizeof (*adddev
->l2ad_dev_hdr
), 1 << vd
->vdev_ashift
);
8909 adddev
->l2ad_start
= VDEV_LABEL_START_SIZE
+ l2dhdr_asize
;
8910 adddev
->l2ad_end
= VDEV_LABEL_START_SIZE
+ vdev_get_min_asize(vd
);
8911 ASSERT3U(adddev
->l2ad_start
, <, adddev
->l2ad_end
);
8912 adddev
->l2ad_hand
= adddev
->l2ad_start
;
8913 adddev
->l2ad_evict
= adddev
->l2ad_start
;
8914 adddev
->l2ad_first
= B_TRUE
;
8915 adddev
->l2ad_writing
= B_FALSE
;
8916 adddev
->l2ad_dev_hdr
= kmem_zalloc(l2dhdr_asize
, KM_SLEEP
);
8918 mutex_init(&adddev
->l2ad_mtx
, NULL
, MUTEX_DEFAULT
, NULL
);
8920 * This is a list of all ARC buffers that are still valid on the
8923 list_create(&adddev
->l2ad_buflist
, sizeof (arc_buf_hdr_t
),
8924 offsetof(arc_buf_hdr_t
, b_l2hdr
.b_l2node
));
8927 * This is a list of pointers to log blocks that are still present
8930 list_create(&adddev
->l2ad_lbptr_list
, sizeof (l2arc_lb_ptr_buf_t
),
8931 offsetof(l2arc_lb_ptr_buf_t
, node
));
8933 vdev_space_update(vd
, 0, 0, adddev
->l2ad_end
- adddev
->l2ad_hand
);
8934 zfs_refcount_create(&adddev
->l2ad_alloc
);
8935 zfs_refcount_create(&adddev
->l2ad_lb_asize
);
8936 zfs_refcount_create(&adddev
->l2ad_lb_count
);
8939 * Add device to global list
8941 mutex_enter(&l2arc_dev_mtx
);
8942 list_insert_head(l2arc_dev_list
, adddev
);
8943 atomic_inc_64(&l2arc_ndev
);
8944 mutex_exit(&l2arc_dev_mtx
);
8947 * Decide if vdev is eligible for L2ARC rebuild
8949 l2arc_rebuild_vdev(adddev
->l2ad_vdev
, B_FALSE
);
8953 l2arc_rebuild_vdev(vdev_t
*vd
, boolean_t reopen
)
8955 l2arc_dev_t
*dev
= NULL
;
8956 l2arc_dev_hdr_phys_t
*l2dhdr
;
8957 uint64_t l2dhdr_asize
;
8960 dev
= l2arc_vdev_get(vd
);
8961 ASSERT3P(dev
, !=, NULL
);
8962 spa
= dev
->l2ad_spa
;
8963 l2dhdr
= dev
->l2ad_dev_hdr
;
8964 l2dhdr_asize
= dev
->l2ad_dev_hdr_asize
;
8967 * The L2ARC has to hold at least the payload of one log block for
8968 * them to be restored (persistent L2ARC). The payload of a log block
8969 * depends on the amount of its log entries. We always write log blocks
8970 * with 1022 entries. How many of them are committed or restored depends
8971 * on the size of the L2ARC device. Thus the maximum payload of
8972 * one log block is 1022 * SPA_MAXBLOCKSIZE = 16GB. If the L2ARC device
8973 * is less than that, we reduce the amount of committed and restored
8974 * log entries per block so as to enable persistence.
8976 if (dev
->l2ad_end
< l2arc_rebuild_blocks_min_l2size
) {
8977 dev
->l2ad_log_entries
= 0;
8979 dev
->l2ad_log_entries
= MIN((dev
->l2ad_end
-
8980 dev
->l2ad_start
) >> SPA_MAXBLOCKSHIFT
,
8981 L2ARC_LOG_BLK_MAX_ENTRIES
);
8985 * Read the device header, if an error is returned do not rebuild L2ARC.
8987 if (l2arc_dev_hdr_read(dev
) == 0 && dev
->l2ad_log_entries
> 0) {
8989 * If we are onlining a cache device (vdev_reopen) that was
8990 * still present (l2arc_vdev_present()) and rebuild is enabled,
8991 * we should evict all ARC buffers and pointers to log blocks
8992 * and reclaim their space before restoring its contents to
8996 if (!l2arc_rebuild_enabled
) {
8999 l2arc_evict(dev
, 0, B_TRUE
);
9000 /* start a new log block */
9001 dev
->l2ad_log_ent_idx
= 0;
9002 dev
->l2ad_log_blk_payload_asize
= 0;
9003 dev
->l2ad_log_blk_payload_start
= 0;
9007 * Just mark the device as pending for a rebuild. We won't
9008 * be starting a rebuild in line here as it would block pool
9009 * import. Instead spa_load_impl will hand that off to an
9010 * async task which will call l2arc_spa_rebuild_start.
9012 dev
->l2ad_rebuild
= B_TRUE
;
9013 } else if (spa_writeable(spa
)) {
9015 * In this case create a new header. We zero out the memory
9016 * holding the header to reset dh_start_lbps.
9018 bzero(l2dhdr
, l2dhdr_asize
);
9019 l2arc_dev_hdr_update(dev
);
9024 * Remove a vdev from the L2ARC.
9027 l2arc_remove_vdev(vdev_t
*vd
)
9029 l2arc_dev_t
*remdev
= NULL
;
9032 * Find the device by vdev
9034 remdev
= l2arc_vdev_get(vd
);
9035 ASSERT3P(remdev
, !=, NULL
);
9038 * Cancel any ongoing or scheduled rebuild.
9040 mutex_enter(&l2arc_rebuild_thr_lock
);
9041 if (remdev
->l2ad_rebuild_began
== B_TRUE
) {
9042 remdev
->l2ad_rebuild_cancel
= B_TRUE
;
9043 while (remdev
->l2ad_rebuild
== B_TRUE
)
9044 cv_wait(&l2arc_rebuild_thr_cv
, &l2arc_rebuild_thr_lock
);
9046 mutex_exit(&l2arc_rebuild_thr_lock
);
9049 * Remove device from global list
9051 mutex_enter(&l2arc_dev_mtx
);
9052 list_remove(l2arc_dev_list
, remdev
);
9053 l2arc_dev_last
= NULL
; /* may have been invalidated */
9054 atomic_dec_64(&l2arc_ndev
);
9055 mutex_exit(&l2arc_dev_mtx
);
9058 * Clear all buflists and ARC references. L2ARC device flush.
9060 l2arc_evict(remdev
, 0, B_TRUE
);
9061 list_destroy(&remdev
->l2ad_buflist
);
9062 ASSERT(list_is_empty(&remdev
->l2ad_lbptr_list
));
9063 list_destroy(&remdev
->l2ad_lbptr_list
);
9064 mutex_destroy(&remdev
->l2ad_mtx
);
9065 zfs_refcount_destroy(&remdev
->l2ad_alloc
);
9066 zfs_refcount_destroy(&remdev
->l2ad_lb_asize
);
9067 zfs_refcount_destroy(&remdev
->l2ad_lb_count
);
9068 kmem_free(remdev
->l2ad_dev_hdr
, remdev
->l2ad_dev_hdr_asize
);
9069 kmem_free(remdev
, sizeof (l2arc_dev_t
));
9075 l2arc_thread_exit
= 0;
9077 l2arc_writes_sent
= 0;
9078 l2arc_writes_done
= 0;
9080 mutex_init(&l2arc_feed_thr_lock
, NULL
, MUTEX_DEFAULT
, NULL
);
9081 cv_init(&l2arc_feed_thr_cv
, NULL
, CV_DEFAULT
, NULL
);
9082 mutex_init(&l2arc_rebuild_thr_lock
, NULL
, MUTEX_DEFAULT
, NULL
);
9083 cv_init(&l2arc_rebuild_thr_cv
, NULL
, CV_DEFAULT
, NULL
);
9084 mutex_init(&l2arc_dev_mtx
, NULL
, MUTEX_DEFAULT
, NULL
);
9085 mutex_init(&l2arc_free_on_write_mtx
, NULL
, MUTEX_DEFAULT
, NULL
);
9087 l2arc_dev_list
= &L2ARC_dev_list
;
9088 l2arc_free_on_write
= &L2ARC_free_on_write
;
9089 list_create(l2arc_dev_list
, sizeof (l2arc_dev_t
),
9090 offsetof(l2arc_dev_t
, l2ad_node
));
9091 list_create(l2arc_free_on_write
, sizeof (l2arc_data_free_t
),
9092 offsetof(l2arc_data_free_t
, l2df_list_node
));
9099 * This is called from dmu_fini(), which is called from spa_fini();
9100 * Because of this, we can assume that all l2arc devices have
9101 * already been removed when the pools themselves were removed.
9104 l2arc_do_free_on_write();
9106 mutex_destroy(&l2arc_feed_thr_lock
);
9107 cv_destroy(&l2arc_feed_thr_cv
);
9108 mutex_destroy(&l2arc_rebuild_thr_lock
);
9109 cv_destroy(&l2arc_rebuild_thr_cv
);
9110 mutex_destroy(&l2arc_dev_mtx
);
9111 mutex_destroy(&l2arc_free_on_write_mtx
);
9113 list_destroy(l2arc_dev_list
);
9114 list_destroy(l2arc_free_on_write
);
9120 if (!(spa_mode_global
& FWRITE
))
9123 (void) thread_create(NULL
, 0, l2arc_feed_thread
, NULL
, 0, &p0
,
9124 TS_RUN
, minclsyspri
);
9130 if (!(spa_mode_global
& FWRITE
))
9133 mutex_enter(&l2arc_feed_thr_lock
);
9134 cv_signal(&l2arc_feed_thr_cv
); /* kick thread out of startup */
9135 l2arc_thread_exit
= 1;
9136 while (l2arc_thread_exit
!= 0)
9137 cv_wait(&l2arc_feed_thr_cv
, &l2arc_feed_thr_lock
);
9138 mutex_exit(&l2arc_feed_thr_lock
);
9142 * Punches out rebuild threads for the L2ARC devices in a spa. This should
9143 * be called after pool import from the spa async thread, since starting
9144 * these threads directly from spa_import() will make them part of the
9145 * "zpool import" context and delay process exit (and thus pool import).
9148 l2arc_spa_rebuild_start(spa_t
*spa
)
9150 ASSERT(MUTEX_HELD(&spa_namespace_lock
));
9153 * Locate the spa's l2arc devices and kick off rebuild threads.
9155 for (int i
= 0; i
< spa
->spa_l2cache
.sav_count
; i
++) {
9157 l2arc_vdev_get(spa
->spa_l2cache
.sav_vdevs
[i
]);
9159 /* Don't attempt a rebuild if the vdev is UNAVAIL */
9162 mutex_enter(&l2arc_rebuild_thr_lock
);
9163 if (dev
->l2ad_rebuild
&& !dev
->l2ad_rebuild_cancel
) {
9164 dev
->l2ad_rebuild_began
= B_TRUE
;
9165 (void) thread_create(NULL
, 0,
9166 (void (*)(void *))l2arc_dev_rebuild_start
,
9167 dev
, 0, &p0
, TS_RUN
, minclsyspri
);
9169 mutex_exit(&l2arc_rebuild_thr_lock
);
9174 * Main entry point for L2ARC rebuilding.
9177 l2arc_dev_rebuild_start(l2arc_dev_t
*dev
)
9179 VERIFY(!dev
->l2ad_rebuild_cancel
);
9180 VERIFY(dev
->l2ad_rebuild
);
9181 (void) l2arc_rebuild(dev
);
9182 mutex_enter(&l2arc_rebuild_thr_lock
);
9183 dev
->l2ad_rebuild_began
= B_FALSE
;
9184 dev
->l2ad_rebuild
= B_FALSE
;
9185 mutex_exit(&l2arc_rebuild_thr_lock
);
9191 * This function implements the actual L2ARC metadata rebuild. It:
9192 * starts reading the log block chain and restores each block's contents
9193 * to memory (reconstructing arc_buf_hdr_t's).
9195 * Operation stops under any of the following conditions:
9197 * 1) We reach the end of the log block chain.
9198 * 2) We encounter *any* error condition (cksum errors, io errors)
9201 l2arc_rebuild(l2arc_dev_t
*dev
)
9203 vdev_t
*vd
= dev
->l2ad_vdev
;
9204 spa_t
*spa
= vd
->vdev_spa
;
9206 l2arc_dev_hdr_phys_t
*l2dhdr
= dev
->l2ad_dev_hdr
;
9207 l2arc_log_blk_phys_t
*this_lb
, *next_lb
;
9208 zio_t
*this_io
= NULL
, *next_io
= NULL
;
9209 l2arc_log_blkptr_t lbps
[2];
9210 l2arc_lb_ptr_buf_t
*lb_ptr_buf
;
9211 boolean_t lock_held
;
9213 this_lb
= kmem_zalloc(sizeof (*this_lb
), KM_SLEEP
);
9214 next_lb
= kmem_zalloc(sizeof (*next_lb
), KM_SLEEP
);
9217 * We prevent device removal while issuing reads to the device,
9218 * then during the rebuilding phases we drop this lock again so
9219 * that a spa_unload or device remove can be initiated - this is
9220 * safe, because the spa will signal us to stop before removing
9221 * our device and wait for us to stop.
9223 spa_config_enter(spa
, SCL_L2ARC
, vd
, RW_READER
);
9227 * Retrieve the persistent L2ARC device state.
9228 * L2BLK_GET_PSIZE returns aligned size for log blocks.
9230 dev
->l2ad_evict
= MAX(l2dhdr
->dh_evict
, dev
->l2ad_start
);
9231 dev
->l2ad_hand
= MAX(l2dhdr
->dh_start_lbps
[0].lbp_daddr
+
9232 L2BLK_GET_PSIZE((&l2dhdr
->dh_start_lbps
[0])->lbp_prop
),
9234 dev
->l2ad_first
= !!(l2dhdr
->dh_flags
& L2ARC_DEV_HDR_EVICT_FIRST
);
9237 * In case the zfs module parameter l2arc_rebuild_enabled is false
9238 * we do not start the rebuild process.
9240 if (!l2arc_rebuild_enabled
)
9243 /* Prepare the rebuild process */
9244 bcopy(l2dhdr
->dh_start_lbps
, lbps
, sizeof (lbps
));
9246 /* Start the rebuild process */
9248 if (!l2arc_log_blkptr_valid(dev
, &lbps
[0]))
9251 if ((err
= l2arc_log_blk_read(dev
, &lbps
[0], &lbps
[1],
9252 this_lb
, next_lb
, this_io
, &next_io
)) != 0)
9256 * Our memory pressure valve. If the system is running low
9257 * on memory, rather than swamping memory with new ARC buf
9258 * hdrs, we opt not to rebuild the L2ARC. At this point,
9259 * however, we have already set up our L2ARC dev to chain in
9260 * new metadata log blocks, so the user may choose to offline/
9261 * online the L2ARC dev at a later time (or re-import the pool)
9262 * to reconstruct it (when there's less memory pressure).
9264 if (l2arc_hdr_limit_reached()) {
9265 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_lowmem
);
9266 cmn_err(CE_NOTE
, "System running low on memory, "
9267 "aborting L2ARC rebuild.");
9268 err
= SET_ERROR(ENOMEM
);
9272 spa_config_exit(spa
, SCL_L2ARC
, vd
);
9273 lock_held
= B_FALSE
;
9276 * Now that we know that the next_lb checks out alright, we
9277 * can start reconstruction from this log block.
9278 * L2BLK_GET_PSIZE returns aligned size for log blocks.
9280 uint64_t asize
= L2BLK_GET_PSIZE((&lbps
[0])->lbp_prop
);
9281 l2arc_log_blk_restore(dev
, this_lb
, asize
);
9284 * log block restored, include its pointer in the list of
9285 * pointers to log blocks present in the L2ARC device.
9287 lb_ptr_buf
= kmem_zalloc(sizeof (l2arc_lb_ptr_buf_t
), KM_SLEEP
);
9288 lb_ptr_buf
->lb_ptr
= kmem_zalloc(sizeof (l2arc_log_blkptr_t
),
9290 bcopy(&lbps
[0], lb_ptr_buf
->lb_ptr
,
9291 sizeof (l2arc_log_blkptr_t
));
9292 mutex_enter(&dev
->l2ad_mtx
);
9293 list_insert_tail(&dev
->l2ad_lbptr_list
, lb_ptr_buf
);
9294 ARCSTAT_INCR(arcstat_l2_log_blk_asize
, asize
);
9295 ARCSTAT_BUMP(arcstat_l2_log_blk_count
);
9296 zfs_refcount_add_many(&dev
->l2ad_lb_asize
, asize
, lb_ptr_buf
);
9297 zfs_refcount_add(&dev
->l2ad_lb_count
, lb_ptr_buf
);
9298 mutex_exit(&dev
->l2ad_mtx
);
9299 vdev_space_update(vd
, asize
, 0, 0);
9303 * Protection against loops of log blocks:
9305 * l2ad_hand l2ad_evict
9307 * l2ad_start |=======================================| l2ad_end
9308 * -----|||----|||---|||----|||
9310 * ---|||---|||----|||---|||
9313 * In this situation the pointer of log block (4) passes
9314 * l2arc_log_blkptr_valid() but the log block should not be
9315 * restored as it is overwritten by the payload of log block
9316 * (0). Only log blocks (0)-(3) should be restored. We check
9317 * whether l2ad_evict lies in between the payload starting
9318 * offset of the next log block (lbps[1].lbp_payload_start)
9319 * and the payload starting offset of the present log block
9320 * (lbps[0].lbp_payload_start). If true and this isn't the
9321 * first pass, we are looping from the beginning and we should
9325 if (l2arc_range_check_overlap(lbps
[1].lbp_payload_start
,
9326 lbps
[0].lbp_payload_start
, dev
->l2ad_evict
) &&
9331 mutex_enter(&l2arc_rebuild_thr_lock
);
9332 if (dev
->l2ad_rebuild_cancel
) {
9333 dev
->l2ad_rebuild
= B_FALSE
;
9334 cv_signal(&l2arc_rebuild_thr_cv
);
9335 mutex_exit(&l2arc_rebuild_thr_lock
);
9336 err
= SET_ERROR(ECANCELED
);
9339 mutex_exit(&l2arc_rebuild_thr_lock
);
9340 if (spa_config_tryenter(spa
, SCL_L2ARC
, vd
,
9346 * L2ARC config lock held by somebody in writer,
9347 * possibly due to them trying to remove us. They'll
9348 * likely to want us to shut down, so after a little
9349 * delay, we check l2ad_rebuild_cancel and retry
9356 * Continue with the next log block.
9359 lbps
[1] = this_lb
->lb_prev_lbp
;
9360 PTR_SWAP(this_lb
, next_lb
);
9365 if (this_io
!= NULL
)
9366 l2arc_log_blk_fetch_abort(this_io
);
9368 if (next_io
!= NULL
)
9369 l2arc_log_blk_fetch_abort(next_io
);
9370 kmem_free(this_lb
, sizeof (*this_lb
));
9371 kmem_free(next_lb
, sizeof (*next_lb
));
9373 if (!l2arc_rebuild_enabled
) {
9374 spa_history_log_internal(spa
, "L2ARC rebuild", NULL
,
9376 } else if (err
== 0 && zfs_refcount_count(&dev
->l2ad_lb_count
) > 0) {
9377 ARCSTAT_BUMP(arcstat_l2_rebuild_success
);
9378 spa_history_log_internal(spa
, "L2ARC rebuild", NULL
,
9379 "successful, restored %llu blocks",
9380 (u_longlong_t
)zfs_refcount_count(&dev
->l2ad_lb_count
));
9381 } else if (err
== 0 && zfs_refcount_count(&dev
->l2ad_lb_count
) == 0) {
9383 * No error but also nothing restored, meaning the lbps array
9384 * in the device header points to invalid/non-present log
9385 * blocks. Reset the header.
9387 spa_history_log_internal(spa
, "L2ARC rebuild", NULL
,
9388 "no valid log blocks");
9389 bzero(l2dhdr
, dev
->l2ad_dev_hdr_asize
);
9390 l2arc_dev_hdr_update(dev
);
9391 } else if (err
== ECANCELED
) {
9393 * In case the rebuild was canceled do not log to spa history
9394 * log as the pool may be in the process of being removed.
9396 zfs_dbgmsg("L2ARC rebuild aborted, restored %llu blocks",
9397 zfs_refcount_count(&dev
->l2ad_lb_count
));
9398 } else if (err
!= 0) {
9399 spa_history_log_internal(spa
, "L2ARC rebuild", NULL
,
9400 "aborted, restored %llu blocks",
9401 (u_longlong_t
)zfs_refcount_count(&dev
->l2ad_lb_count
));
9405 spa_config_exit(spa
, SCL_L2ARC
, vd
);
9411 * Attempts to read the device header on the provided L2ARC device and writes
9412 * it to `hdr'. On success, this function returns 0, otherwise the appropriate
9413 * error code is returned.
9416 l2arc_dev_hdr_read(l2arc_dev_t
*dev
)
9420 l2arc_dev_hdr_phys_t
*l2dhdr
= dev
->l2ad_dev_hdr
;
9421 const uint64_t l2dhdr_asize
= dev
->l2ad_dev_hdr_asize
;
9424 guid
= spa_guid(dev
->l2ad_vdev
->vdev_spa
);
9426 abd
= abd_get_from_buf(l2dhdr
, l2dhdr_asize
);
9428 err
= zio_wait(zio_read_phys(NULL
, dev
->l2ad_vdev
,
9429 VDEV_LABEL_START_SIZE
, l2dhdr_asize
, abd
,
9430 ZIO_CHECKSUM_LABEL
, NULL
, NULL
, ZIO_PRIORITY_SYNC_READ
,
9431 ZIO_FLAG_DONT_CACHE
| ZIO_FLAG_CANFAIL
|
9432 ZIO_FLAG_DONT_PROPAGATE
| ZIO_FLAG_DONT_RETRY
|
9433 ZIO_FLAG_SPECULATIVE
, B_FALSE
));
9438 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_dh_errors
);
9439 zfs_dbgmsg("L2ARC IO error (%d) while reading device header, "
9440 "vdev guid: %llu", err
, dev
->l2ad_vdev
->vdev_guid
);
9444 if (l2dhdr
->dh_magic
== BSWAP_64(L2ARC_DEV_HDR_MAGIC
))
9445 byteswap_uint64_array(l2dhdr
, sizeof (*l2dhdr
));
9447 if (l2dhdr
->dh_magic
!= L2ARC_DEV_HDR_MAGIC
||
9448 l2dhdr
->dh_spa_guid
!= guid
||
9449 l2dhdr
->dh_vdev_guid
!= dev
->l2ad_vdev
->vdev_guid
||
9450 l2dhdr
->dh_version
!= L2ARC_PERSISTENT_VERSION
||
9451 l2dhdr
->dh_log_entries
!= dev
->l2ad_log_entries
||
9452 l2dhdr
->dh_end
!= dev
->l2ad_end
||
9453 !l2arc_range_check_overlap(dev
->l2ad_start
, dev
->l2ad_end
,
9454 l2dhdr
->dh_evict
)) {
9456 * Attempt to rebuild a device containing no actual dev hdr
9457 * or containing a header from some other pool or from another
9458 * version of persistent L2ARC.
9460 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_unsupported
);
9461 return (SET_ERROR(ENOTSUP
));
9468 * Reads L2ARC log blocks from storage and validates their contents.
9470 * This function implements a simple fetcher to make sure that while
9471 * we're processing one buffer the L2ARC is already fetching the next
9474 * The arguments this_lp and next_lp point to the current and next log block
9475 * address in the block chain. Similarly, this_lb and next_lb hold the
9476 * l2arc_log_blk_phys_t's of the current and next L2ARC blk.
9478 * The `this_io' and `next_io' arguments are used for block fetching.
9479 * When issuing the first blk IO during rebuild, you should pass NULL for
9480 * `this_io'. This function will then issue a sync IO to read the block and
9481 * also issue an async IO to fetch the next block in the block chain. The
9482 * fetched IO is returned in `next_io'. On subsequent calls to this
9483 * function, pass the value returned in `next_io' from the previous call
9484 * as `this_io' and a fresh `next_io' pointer to hold the next fetch IO.
9485 * Prior to the call, you should initialize your `next_io' pointer to be
9486 * NULL. If no fetch IO was issued, the pointer is left set at NULL.
9488 * On success, this function returns 0, otherwise it returns an appropriate
9489 * error code. On error the fetching IO is aborted and cleared before
9490 * returning from this function. Therefore, if we return `success', the
9491 * caller can assume that we have taken care of cleanup of fetch IOs.
9494 l2arc_log_blk_read(l2arc_dev_t
*dev
,
9495 const l2arc_log_blkptr_t
*this_lbp
, const l2arc_log_blkptr_t
*next_lbp
,
9496 l2arc_log_blk_phys_t
*this_lb
, l2arc_log_blk_phys_t
*next_lb
,
9497 zio_t
*this_io
, zio_t
**next_io
)
9504 ASSERT(this_lbp
!= NULL
&& next_lbp
!= NULL
);
9505 ASSERT(this_lb
!= NULL
&& next_lb
!= NULL
);
9506 ASSERT(next_io
!= NULL
&& *next_io
== NULL
);
9507 ASSERT(l2arc_log_blkptr_valid(dev
, this_lbp
));
9510 * Check to see if we have issued the IO for this log block in a
9511 * previous run. If not, this is the first call, so issue it now.
9513 if (this_io
== NULL
) {
9514 this_io
= l2arc_log_blk_fetch(dev
->l2ad_vdev
, this_lbp
,
9519 * Peek to see if we can start issuing the next IO immediately.
9521 if (l2arc_log_blkptr_valid(dev
, next_lbp
)) {
9523 * Start issuing IO for the next log block early - this
9524 * should help keep the L2ARC device busy while we
9525 * decompress and restore this log block.
9527 *next_io
= l2arc_log_blk_fetch(dev
->l2ad_vdev
, next_lbp
,
9531 /* Wait for the IO to read this log block to complete */
9532 if ((err
= zio_wait(this_io
)) != 0) {
9533 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_io_errors
);
9534 zfs_dbgmsg("L2ARC IO error (%d) while reading log block, "
9535 "offset: %llu, vdev guid: %llu", err
, this_lbp
->lbp_daddr
,
9536 dev
->l2ad_vdev
->vdev_guid
);
9541 * Make sure the buffer checks out.
9542 * L2BLK_GET_PSIZE returns aligned size for log blocks.
9544 asize
= L2BLK_GET_PSIZE((this_lbp
)->lbp_prop
);
9545 fletcher_4_native(this_lb
, asize
, NULL
, &cksum
);
9546 if (!ZIO_CHECKSUM_EQUAL(cksum
, this_lbp
->lbp_cksum
)) {
9547 ARCSTAT_BUMP(arcstat_l2_rebuild_abort_cksum_lb_errors
);
9548 zfs_dbgmsg("L2ARC log block cksum failed, offset: %llu, "
9549 "vdev guid: %llu, l2ad_hand: %llu, l2ad_evict: %llu",
9550 this_lbp
->lbp_daddr
, dev
->l2ad_vdev
->vdev_guid
,
9551 dev
->l2ad_hand
, dev
->l2ad_evict
);
9552 err
= SET_ERROR(ECKSUM
);
9556 /* Now we can take our time decoding this buffer */
9557 switch (L2BLK_GET_COMPRESS((this_lbp
)->lbp_prop
)) {
9558 case ZIO_COMPRESS_OFF
:
9560 case ZIO_COMPRESS_LZ4
:
9561 abd
= abd_alloc_for_io(asize
, B_TRUE
);
9562 abd_copy_from_buf_off(abd
, this_lb
, 0, asize
);
9563 if ((err
= zio_decompress_data(
9564 L2BLK_GET_COMPRESS((this_lbp
)->lbp_prop
),
9565 abd
, this_lb
, asize
, sizeof (*this_lb
))) != 0) {
9566 err
= SET_ERROR(EINVAL
);
9571 err
= SET_ERROR(EINVAL
);
9574 if (this_lb
->lb_magic
== BSWAP_64(L2ARC_LOG_BLK_MAGIC
))
9575 byteswap_uint64_array(this_lb
, sizeof (*this_lb
));
9576 if (this_lb
->lb_magic
!= L2ARC_LOG_BLK_MAGIC
) {
9577 err
= SET_ERROR(EINVAL
);
9581 /* Abort an in-flight fetch I/O in case of error */
9582 if (err
!= 0 && *next_io
!= NULL
) {
9583 l2arc_log_blk_fetch_abort(*next_io
);
9592 * Restores the payload of a log block to ARC. This creates empty ARC hdr
9593 * entries which only contain an l2arc hdr, essentially restoring the
9594 * buffers to their L2ARC evicted state. This function also updates space
9595 * usage on the L2ARC vdev to make sure it tracks restored buffers.
9598 l2arc_log_blk_restore(l2arc_dev_t
*dev
, const l2arc_log_blk_phys_t
*lb
,
9601 uint64_t size
= 0, asize
= 0;
9602 uint64_t log_entries
= dev
->l2ad_log_entries
;
9605 * Usually arc_adapt() is called only for data, not headers, but
9606 * since we may allocate significant amount of memory here, let ARC
9609 arc_adapt(log_entries
* HDR_L2ONLY_SIZE
, arc_l2c_only
);
9611 for (int i
= log_entries
- 1; i
>= 0; i
--) {
9613 * Restore goes in the reverse temporal direction to preserve
9614 * correct temporal ordering of buffers in the l2ad_buflist.
9615 * l2arc_hdr_restore also does a list_insert_tail instead of
9616 * list_insert_head on the l2ad_buflist:
9618 * LIST l2ad_buflist LIST
9619 * HEAD <------ (time) ------ TAIL
9620 * direction +-----+-----+-----+-----+-----+ direction
9621 * of l2arc <== | buf | buf | buf | buf | buf | ===> of rebuild
9622 * fill +-----+-----+-----+-----+-----+
9626 * l2arc_feed_thread l2arc_rebuild
9627 * will place new bufs here restores bufs here
9629 * During l2arc_rebuild() the device is not used by
9630 * l2arc_feed_thread() as dev->l2ad_rebuild is set to true.
9632 size
+= L2BLK_GET_LSIZE((&lb
->lb_entries
[i
])->le_prop
);
9633 asize
+= vdev_psize_to_asize(dev
->l2ad_vdev
,
9634 L2BLK_GET_PSIZE((&lb
->lb_entries
[i
])->le_prop
));
9635 l2arc_hdr_restore(&lb
->lb_entries
[i
], dev
);
9639 * Record rebuild stats:
9640 * size Logical size of restored buffers in the L2ARC
9641 * asize Aligned size of restored buffers in the L2ARC
9643 ARCSTAT_INCR(arcstat_l2_rebuild_size
, size
);
9644 ARCSTAT_INCR(arcstat_l2_rebuild_asize
, asize
);
9645 ARCSTAT_INCR(arcstat_l2_rebuild_bufs
, log_entries
);
9646 ARCSTAT_F_AVG(arcstat_l2_log_blk_avg_asize
, lb_asize
);
9647 ARCSTAT_F_AVG(arcstat_l2_data_to_meta_ratio
, asize
/ lb_asize
);
9648 ARCSTAT_BUMP(arcstat_l2_rebuild_log_blks
);
9652 * Restores a single ARC buf hdr from a log entry. The ARC buffer is put
9653 * into a state indicating that it has been evicted to L2ARC.
9656 l2arc_hdr_restore(const l2arc_log_ent_phys_t
*le
, l2arc_dev_t
*dev
)
9658 arc_buf_hdr_t
*hdr
, *exists
;
9659 kmutex_t
*hash_lock
;
9660 arc_buf_contents_t type
= L2BLK_GET_TYPE((le
)->le_prop
);
9664 * Do all the allocation before grabbing any locks, this lets us
9665 * sleep if memory is full and we don't have to deal with failed
9668 hdr
= arc_buf_alloc_l2only(L2BLK_GET_LSIZE((le
)->le_prop
), type
,
9669 dev
, le
->le_dva
, le
->le_daddr
,
9670 L2BLK_GET_PSIZE((le
)->le_prop
), le
->le_birth
,
9671 L2BLK_GET_COMPRESS((le
)->le_prop
),
9672 L2BLK_GET_PROTECTED((le
)->le_prop
),
9673 L2BLK_GET_PREFETCH((le
)->le_prop
),
9674 L2BLK_GET_STATE((le
)->le_prop
));
9675 asize
= vdev_psize_to_asize(dev
->l2ad_vdev
,
9676 L2BLK_GET_PSIZE((le
)->le_prop
));
9679 * vdev_space_update() has to be called before arc_hdr_destroy() to
9680 * avoid underflow since the latter also calls vdev_space_update().
9682 l2arc_hdr_arcstats_increment(hdr
);
9683 vdev_space_update(dev
->l2ad_vdev
, asize
, 0, 0);
9685 mutex_enter(&dev
->l2ad_mtx
);
9686 list_insert_tail(&dev
->l2ad_buflist
, hdr
);
9687 (void) zfs_refcount_add_many(&dev
->l2ad_alloc
, arc_hdr_size(hdr
), hdr
);
9688 mutex_exit(&dev
->l2ad_mtx
);
9690 exists
= buf_hash_insert(hdr
, &hash_lock
);
9692 /* Buffer was already cached, no need to restore it. */
9693 arc_hdr_destroy(hdr
);
9695 * If the buffer is already cached, check whether it has
9696 * L2ARC metadata. If not, enter them and update the flag.
9697 * This is important is case of onlining a cache device, since
9698 * we previously evicted all L2ARC metadata from ARC.
9700 if (!HDR_HAS_L2HDR(exists
)) {
9701 arc_hdr_set_flags(exists
, ARC_FLAG_HAS_L2HDR
);
9702 exists
->b_l2hdr
.b_dev
= dev
;
9703 exists
->b_l2hdr
.b_daddr
= le
->le_daddr
;
9704 exists
->b_l2hdr
.b_arcs_state
=
9705 L2BLK_GET_STATE((le
)->le_prop
);
9706 mutex_enter(&dev
->l2ad_mtx
);
9707 list_insert_tail(&dev
->l2ad_buflist
, exists
);
9708 (void) zfs_refcount_add_many(&dev
->l2ad_alloc
,
9709 arc_hdr_size(exists
), exists
);
9710 mutex_exit(&dev
->l2ad_mtx
);
9711 l2arc_hdr_arcstats_increment(exists
);
9712 vdev_space_update(dev
->l2ad_vdev
, asize
, 0, 0);
9714 ARCSTAT_BUMP(arcstat_l2_rebuild_bufs_precached
);
9717 mutex_exit(hash_lock
);
9721 * Starts an asynchronous read IO to read a log block. This is used in log
9722 * block reconstruction to start reading the next block before we are done
9723 * decoding and reconstructing the current block, to keep the l2arc device
9724 * nice and hot with read IO to process.
9725 * The returned zio will contain newly allocated memory buffers for the IO
9726 * data which should then be freed by the caller once the zio is no longer
9727 * needed (i.e. due to it having completed). If you wish to abort this
9728 * zio, you should do so using l2arc_log_blk_fetch_abort, which takes
9729 * care of disposing of the allocated buffers correctly.
9732 l2arc_log_blk_fetch(vdev_t
*vd
, const l2arc_log_blkptr_t
*lbp
,
9733 l2arc_log_blk_phys_t
*lb
)
9737 l2arc_read_callback_t
*cb
;
9739 /* L2BLK_GET_PSIZE returns aligned size for log blocks */
9740 asize
= L2BLK_GET_PSIZE((lbp
)->lbp_prop
);
9741 ASSERT(asize
<= sizeof (l2arc_log_blk_phys_t
));
9743 cb
= kmem_zalloc(sizeof (l2arc_read_callback_t
), KM_SLEEP
);
9744 cb
->l2rcb_abd
= abd_get_from_buf(lb
, asize
);
9745 pio
= zio_root(vd
->vdev_spa
, l2arc_blk_fetch_done
, cb
,
9746 ZIO_FLAG_DONT_CACHE
| ZIO_FLAG_CANFAIL
| ZIO_FLAG_DONT_PROPAGATE
|
9747 ZIO_FLAG_DONT_RETRY
);
9748 (void) zio_nowait(zio_read_phys(pio
, vd
, lbp
->lbp_daddr
, asize
,
9749 cb
->l2rcb_abd
, ZIO_CHECKSUM_OFF
, NULL
, NULL
,
9750 ZIO_PRIORITY_ASYNC_READ
, ZIO_FLAG_DONT_CACHE
| ZIO_FLAG_CANFAIL
|
9751 ZIO_FLAG_DONT_PROPAGATE
| ZIO_FLAG_DONT_RETRY
, B_FALSE
));
9757 * Aborts a zio returned from l2arc_log_blk_fetch and frees the data
9758 * buffers allocated for it.
9761 l2arc_log_blk_fetch_abort(zio_t
*zio
)
9763 (void) zio_wait(zio
);
9767 * Creates a zio to update the device header on an l2arc device.
9770 l2arc_dev_hdr_update(l2arc_dev_t
*dev
)
9772 l2arc_dev_hdr_phys_t
*l2dhdr
= dev
->l2ad_dev_hdr
;
9773 const uint64_t l2dhdr_asize
= dev
->l2ad_dev_hdr_asize
;
9777 VERIFY(spa_config_held(dev
->l2ad_spa
, SCL_STATE_ALL
, RW_READER
));
9779 l2dhdr
->dh_magic
= L2ARC_DEV_HDR_MAGIC
;
9780 l2dhdr
->dh_version
= L2ARC_PERSISTENT_VERSION
;
9781 l2dhdr
->dh_spa_guid
= spa_guid(dev
->l2ad_vdev
->vdev_spa
);
9782 l2dhdr
->dh_vdev_guid
= dev
->l2ad_vdev
->vdev_guid
;
9783 l2dhdr
->dh_log_entries
= dev
->l2ad_log_entries
;
9784 l2dhdr
->dh_evict
= dev
->l2ad_evict
;
9785 l2dhdr
->dh_start
= dev
->l2ad_start
;
9786 l2dhdr
->dh_end
= dev
->l2ad_end
;
9787 l2dhdr
->dh_lb_asize
= zfs_refcount_count(&dev
->l2ad_lb_asize
);
9788 l2dhdr
->dh_lb_count
= zfs_refcount_count(&dev
->l2ad_lb_count
);
9789 l2dhdr
->dh_flags
= 0;
9790 if (dev
->l2ad_first
)
9791 l2dhdr
->dh_flags
|= L2ARC_DEV_HDR_EVICT_FIRST
;
9793 abd
= abd_get_from_buf(l2dhdr
, l2dhdr_asize
);
9795 err
= zio_wait(zio_write_phys(NULL
, dev
->l2ad_vdev
,
9796 VDEV_LABEL_START_SIZE
, l2dhdr_asize
, abd
, ZIO_CHECKSUM_LABEL
, NULL
,
9797 NULL
, ZIO_PRIORITY_ASYNC_WRITE
, ZIO_FLAG_CANFAIL
, B_FALSE
));
9802 zfs_dbgmsg("L2ARC IO error (%d) while writing device header, "
9803 "vdev guid: %llu", err
, dev
->l2ad_vdev
->vdev_guid
);
9808 * Commits a log block to the L2ARC device. This routine is invoked from
9809 * l2arc_write_buffers when the log block fills up.
9810 * This function allocates some memory to temporarily hold the serialized
9811 * buffer to be written. This is then released in l2arc_write_done.
9814 l2arc_log_blk_commit(l2arc_dev_t
*dev
, zio_t
*pio
, l2arc_write_callback_t
*cb
)
9816 l2arc_log_blk_phys_t
*lb
= &dev
->l2ad_log_blk
;
9817 l2arc_dev_hdr_phys_t
*l2dhdr
= dev
->l2ad_dev_hdr
;
9818 uint64_t psize
, asize
;
9820 l2arc_lb_abd_buf_t
*abd_buf
;
9822 l2arc_lb_ptr_buf_t
*lb_ptr_buf
;
9824 VERIFY3S(dev
->l2ad_log_ent_idx
, ==, dev
->l2ad_log_entries
);
9826 tmpbuf
= zio_buf_alloc(sizeof (*lb
));
9827 abd_buf
= zio_buf_alloc(sizeof (*abd_buf
));
9828 abd_buf
->abd
= abd_get_from_buf(lb
, sizeof (*lb
));
9829 lb_ptr_buf
= kmem_zalloc(sizeof (l2arc_lb_ptr_buf_t
), KM_SLEEP
);
9830 lb_ptr_buf
->lb_ptr
= kmem_zalloc(sizeof (l2arc_log_blkptr_t
), KM_SLEEP
);
9832 /* link the buffer into the block chain */
9833 lb
->lb_prev_lbp
= l2dhdr
->dh_start_lbps
[1];
9834 lb
->lb_magic
= L2ARC_LOG_BLK_MAGIC
;
9837 * l2arc_log_blk_commit() may be called multiple times during a single
9838 * l2arc_write_buffers() call. Save the allocated abd buffers in a list
9839 * so we can free them in l2arc_write_done() later on.
9841 list_insert_tail(&cb
->l2wcb_abd_list
, abd_buf
);
9843 /* try to compress the buffer */
9844 psize
= zio_compress_data(ZIO_COMPRESS_LZ4
,
9845 abd_buf
->abd
, tmpbuf
, sizeof (*lb
));
9847 /* a log block is never entirely zero */
9849 asize
= vdev_psize_to_asize(dev
->l2ad_vdev
, psize
);
9850 ASSERT(asize
<= sizeof (*lb
));
9853 * Update the start log block pointer in the device header to point
9854 * to the log block we're about to write.
9856 l2dhdr
->dh_start_lbps
[1] = l2dhdr
->dh_start_lbps
[0];
9857 l2dhdr
->dh_start_lbps
[0].lbp_daddr
= dev
->l2ad_hand
;
9858 l2dhdr
->dh_start_lbps
[0].lbp_payload_asize
=
9859 dev
->l2ad_log_blk_payload_asize
;
9860 l2dhdr
->dh_start_lbps
[0].lbp_payload_start
=
9861 dev
->l2ad_log_blk_payload_start
;
9864 (&l2dhdr
->dh_start_lbps
[0])->lbp_prop
, sizeof (*lb
));
9866 (&l2dhdr
->dh_start_lbps
[0])->lbp_prop
, asize
);
9868 (&l2dhdr
->dh_start_lbps
[0])->lbp_prop
,
9869 ZIO_CHECKSUM_FLETCHER_4
);
9870 if (asize
< sizeof (*lb
)) {
9871 /* compression succeeded */
9872 bzero(tmpbuf
+ psize
, asize
- psize
);
9874 (&l2dhdr
->dh_start_lbps
[0])->lbp_prop
,
9877 /* compression failed */
9878 bcopy(lb
, tmpbuf
, sizeof (*lb
));
9880 (&l2dhdr
->dh_start_lbps
[0])->lbp_prop
,
9884 /* checksum what we're about to write */
9885 fletcher_4_native(tmpbuf
, asize
, NULL
,
9886 &l2dhdr
->dh_start_lbps
[0].lbp_cksum
);
9888 abd_put(abd_buf
->abd
);
9890 /* perform the write itself */
9891 abd_buf
->abd
= abd_get_from_buf(tmpbuf
, sizeof (*lb
));
9892 abd_take_ownership_of_buf(abd_buf
->abd
, B_TRUE
);
9893 wzio
= zio_write_phys(pio
, dev
->l2ad_vdev
, dev
->l2ad_hand
,
9894 asize
, abd_buf
->abd
, ZIO_CHECKSUM_OFF
, NULL
, NULL
,
9895 ZIO_PRIORITY_ASYNC_WRITE
, ZIO_FLAG_CANFAIL
, B_FALSE
);
9896 DTRACE_PROBE2(l2arc__write
, vdev_t
*, dev
->l2ad_vdev
, zio_t
*, wzio
);
9897 (void) zio_nowait(wzio
);
9899 dev
->l2ad_hand
+= asize
;
9901 * Include the committed log block's pointer in the list of pointers
9902 * to log blocks present in the L2ARC device.
9904 bcopy(&l2dhdr
->dh_start_lbps
[0], lb_ptr_buf
->lb_ptr
,
9905 sizeof (l2arc_log_blkptr_t
));
9906 mutex_enter(&dev
->l2ad_mtx
);
9907 list_insert_head(&dev
->l2ad_lbptr_list
, lb_ptr_buf
);
9908 ARCSTAT_INCR(arcstat_l2_log_blk_asize
, asize
);
9909 ARCSTAT_BUMP(arcstat_l2_log_blk_count
);
9910 zfs_refcount_add_many(&dev
->l2ad_lb_asize
, asize
, lb_ptr_buf
);
9911 zfs_refcount_add(&dev
->l2ad_lb_count
, lb_ptr_buf
);
9912 mutex_exit(&dev
->l2ad_mtx
);
9913 vdev_space_update(dev
->l2ad_vdev
, asize
, 0, 0);
9915 /* bump the kstats */
9916 ARCSTAT_INCR(arcstat_l2_write_bytes
, asize
);
9917 ARCSTAT_BUMP(arcstat_l2_log_blk_writes
);
9918 ARCSTAT_F_AVG(arcstat_l2_log_blk_avg_asize
, asize
);
9919 ARCSTAT_F_AVG(arcstat_l2_data_to_meta_ratio
,
9920 dev
->l2ad_log_blk_payload_asize
/ asize
);
9922 /* start a new log block */
9923 dev
->l2ad_log_ent_idx
= 0;
9924 dev
->l2ad_log_blk_payload_asize
= 0;
9925 dev
->l2ad_log_blk_payload_start
= 0;
9929 * Validates an L2ARC log block address to make sure that it can be read
9930 * from the provided L2ARC device.
9933 l2arc_log_blkptr_valid(l2arc_dev_t
*dev
, const l2arc_log_blkptr_t
*lbp
)
9935 /* L2BLK_GET_PSIZE returns aligned size for log blocks */
9936 uint64_t asize
= L2BLK_GET_PSIZE((lbp
)->lbp_prop
);
9937 uint64_t end
= lbp
->lbp_daddr
+ asize
- 1;
9938 uint64_t start
= lbp
->lbp_payload_start
;
9939 boolean_t evicted
= B_FALSE
;
9943 * A log block is valid if all of the following conditions are true:
9944 * - it fits entirely (including its payload) between l2ad_start and
9946 * - it has a valid size
9947 * - neither the log block itself nor part of its payload was evicted
9950 * l2ad_hand l2ad_evict
9955 * l2ad_start ============================================ l2ad_end
9956 * --------------------------||||
9963 l2arc_range_check_overlap(start
, end
, dev
->l2ad_hand
) ||
9964 l2arc_range_check_overlap(start
, end
, dev
->l2ad_evict
) ||
9965 l2arc_range_check_overlap(dev
->l2ad_hand
, dev
->l2ad_evict
, start
) ||
9966 l2arc_range_check_overlap(dev
->l2ad_hand
, dev
->l2ad_evict
, end
);
9968 return (start
>= dev
->l2ad_start
&& end
<= dev
->l2ad_end
&&
9969 asize
> 0 && asize
<= sizeof (l2arc_log_blk_phys_t
) &&
9970 (!evicted
|| dev
->l2ad_first
));
9974 * Inserts ARC buffer header `hdr' into the current L2ARC log block on
9975 * the device. The buffer being inserted must be present in L2ARC.
9976 * Returns B_TRUE if the L2ARC log block is full and needs to be committed
9977 * to L2ARC, or B_FALSE if it still has room for more ARC buffers.
9980 l2arc_log_blk_insert(l2arc_dev_t
*dev
, const arc_buf_hdr_t
*hdr
)
9982 l2arc_log_blk_phys_t
*lb
= &dev
->l2ad_log_blk
;
9983 l2arc_log_ent_phys_t
*le
;
9985 if (dev
->l2ad_log_entries
== 0)
9988 int index
= dev
->l2ad_log_ent_idx
++;
9990 ASSERT3S(index
, <, dev
->l2ad_log_entries
);
9991 ASSERT(HDR_HAS_L2HDR(hdr
));
9993 le
= &lb
->lb_entries
[index
];
9994 bzero(le
, sizeof (*le
));
9995 le
->le_dva
= hdr
->b_dva
;
9996 le
->le_birth
= hdr
->b_birth
;
9997 le
->le_daddr
= hdr
->b_l2hdr
.b_daddr
;
9999 dev
->l2ad_log_blk_payload_start
= le
->le_daddr
;
10000 L2BLK_SET_LSIZE((le
)->le_prop
, HDR_GET_LSIZE(hdr
));
10001 L2BLK_SET_PSIZE((le
)->le_prop
, HDR_GET_PSIZE(hdr
));
10002 L2BLK_SET_COMPRESS((le
)->le_prop
, HDR_GET_COMPRESS(hdr
));
10003 L2BLK_SET_TYPE((le
)->le_prop
, hdr
->b_type
);
10004 L2BLK_SET_PROTECTED((le
)->le_prop
, !!(HDR_PROTECTED(hdr
)));
10005 L2BLK_SET_PREFETCH((le
)->le_prop
, !!(HDR_PREFETCH(hdr
)));
10006 L2BLK_SET_STATE((le
)->le_prop
, hdr
->b_l1hdr
.b_state
->arcs_state
);
10008 dev
->l2ad_log_blk_payload_asize
+= vdev_psize_to_asize(dev
->l2ad_vdev
,
10009 HDR_GET_PSIZE(hdr
));
10011 return (dev
->l2ad_log_ent_idx
== dev
->l2ad_log_entries
);
10015 * Checks whether a given L2ARC device address sits in a time-sequential
10016 * range. The trick here is that the L2ARC is a rotary buffer, so we can't
10017 * just do a range comparison, we need to handle the situation in which the
10018 * range wraps around the end of the L2ARC device. Arguments:
10019 * bottom -- Lower end of the range to check (written to earlier).
10020 * top -- Upper end of the range to check (written to later).
10021 * check -- The address for which we want to determine if it sits in
10022 * between the top and bottom.
10024 * The 3-way conditional below represents the following cases:
10026 * bottom < top : Sequentially ordered case:
10027 * <check>--------+-------------------+
10028 * | (overlap here?) |
10030 * |---------------<bottom>============<top>--------------|
10032 * bottom > top: Looped-around case:
10033 * <check>--------+------------------+
10034 * | (overlap here?) |
10036 * |===============<top>---------------<bottom>===========|
10039 * +---------------+---------<check>
10041 * top == bottom : Just a single address comparison.
10044 l2arc_range_check_overlap(uint64_t bottom
, uint64_t top
, uint64_t check
)
10047 return (bottom
<= check
&& check
<= top
);
10048 else if (bottom
> top
)
10049 return (check
<= top
|| bottom
<= check
);
10051 return (check
== top
);