4 * The contents of this file are subject to the terms of the
5 * Common Development and Distribution License (the "License").
6 * You may not use this file except in compliance with the License.
8 * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
9 * or http://www.opensolaris.org/os/licensing.
10 * See the License for the specific language governing permissions
11 * and limitations under the License.
13 * When distributing Covered Code, include this CDDL HEADER in each
14 * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
15 * If applicable, add the following below this CDDL HEADER, with the
16 * fields enclosed by brackets "[]" replaced with your own identifying
17 * information: Portions Copyright [yyyy] [name of copyright owner]
22 * Copyright 2010 Sun Microsystems, Inc. All rights reserved.
23 * Use is subject to license terms.
24 * Copyright (c) 2016 by Delphix. All rights reserved.
28 * Page Retire - Big Theory Statement.
30 * This file handles removing sections of faulty memory from use when the
31 * user land FMA Diagnosis Engine requests that a page be removed or when
32 * a CE or UE is detected by the hardware.
34 * In the bad old days, the kernel side of Page Retire did a lot of the work
35 * on its own. Now, with the DE keeping track of errors, the kernel side is
36 * rather simple minded on most platforms.
38 * Errors are all reflected to the DE, and after digesting the error and
39 * looking at all previously reported errors, the DE decides what should
40 * be done about the current error. If the DE wants a particular page to
41 * be retired, then the kernel page retire code is invoked via an ioctl.
42 * On non-FMA platforms, the ue_drain and ce_drain paths ends up calling
43 * page retire to handle the error. Since page retire is just a simple
44 * mechanism it doesn't need to differentiate between the different callers.
46 * The p_toxic field in the page_t is used to indicate which errors have
47 * occurred and what action has been taken on a given page. Because errors are
48 * reported without regard to the locked state of a page, no locks are used
49 * to SET the error bits in p_toxic. However, in order to clear the error
50 * bits, the page_t must be held exclusively locked.
52 * When page_retire() is called, it must be able to acquire locks, sleep, etc.
53 * It must not be called from high-level interrupt context.
55 * Depending on how the requested page is being used at the time of the retire
56 * request (and on the availability of sufficient system resources), the page
57 * may be retired immediately, or just marked for retirement later. For
58 * example, locked pages are marked, while free pages are retired. Multiple
59 * requests may be made to retire the same page, although there is no need
60 * to: once the p_toxic flags are set, the page will be retired as soon as it
61 * can be exclusively locked.
63 * The retire mechanism is driven centrally out of page_unlock(). To expedite
64 * the retirement of pages, further requests for SE_SHARED locks are denied
65 * as long as a page retirement is pending. In addition, as long as pages are
66 * pending retirement a background thread runs periodically trying to retire
67 * those pages. Pages which could not be retired while the system is running
68 * are scrubbed prior to rebooting to avoid latent errors on the next boot.
70 * UE pages without persistent errors are scrubbed and returned to service.
71 * Recidivist pages, as well as FMA-directed requests for retirement, result
72 * in the page being taken out of service. Once the decision is made to take
73 * a page out of service, the page is cleared, hashed onto the retired_pages
74 * vnode, marked as retired, and it is unlocked. No other requesters (except
75 * for unretire) are allowed to lock retired pages.
77 * The public routines return (sadly) 0 if they worked and a non-zero error
78 * value if something went wrong. This is done for the ioctl side of the
79 * world to allow errors to be reflected all the way out to user land. The
80 * non-zero values are explained in comments atop each function.
86 * 1. Trying to retire non-relocatable kvp pages may result in a
87 * quagmire. This is because seg_kmem() no longer keeps its pages locked,
88 * and calls page_lookup() in the free path; since kvp pages are modified
89 * and don't have a usable backing store, page_retire() can't do anything
90 * with them, and we'll keep denying the lock to seg_kmem_free() in a
91 * vicious cycle. To prevent that, we don't deny locks to kvp pages, and
92 * hence only try to retire a page from page_unlock() in the free path.
93 * Since most kernel pages are indefinitely held anyway, and don't
94 * participate in I/O, this is of little consequence.
96 * 2. Low memory situations will be interesting. If we don't have
97 * enough memory for page_relocate() to succeed, we won't be able to
98 * retire dirty pages; nobody will be able to push them out to disk
99 * either, since we aggressively deny the page lock. We could change
100 * fsflush so it can recognize this situation, grab the lock, and push
101 * the page out, where we'll catch it in the free path and retire it.
103 * 3. Beware of places that have code like this in them:
105 * if (! page_tryupgrade(pp)) {
107 * while (! page_lock(pp, SE_EXCL, NULL, P_RECLAIM)) {
113 * The problem is that pp can change identity right after the
114 * page_unlock() call. In particular, page_retire() can step in
115 * there, change pp's identity, and hash pp onto the retired_vnode.
117 * Of course, other functions besides page_retire() can have the
118 * same effect. A kmem reader can waltz by, set up a mapping to the
119 * page, and then unlock the page. Page_free() will then go castors
120 * up. So if anybody is doing this, it's already a bug.
122 * 4. mdboot()'s call into page_retire_mdboot() should probably be
123 * moved lower. Where the call is made now, we can get into trouble
124 * by scrubbing a kernel page that is then accessed later.
127 #include <sys/types.h>
128 #include <sys/param.h>
129 #include <sys/systm.h>
130 #include <sys/mman.h>
131 #include <sys/vnode.h>
132 #include <sys/vfs_opreg.h>
133 #include <sys/cmn_err.h>
134 #include <sys/ksynch.h>
135 #include <sys/thread.h>
136 #include <sys/disp.h>
137 #include <sys/ontrap.h>
138 #include <sys/vmsystm.h>
139 #include <sys/mem_config.h>
140 #include <sys/atomic.h>
141 #include <sys/callb.h>
142 #include <sys/kobj.h>
144 #include <vm/vm_dep.h>
147 #include <vm/seg_kmem.h>
150 * vnode for all pages which are retired from the VM system;
152 vnode_t
*retired_pages
;
154 static int page_retire_pp_finish(page_t
*, void *, uint_t
);
157 * Make a list of all of the pages that have been marked for retirement
158 * but are not yet retired. At system shutdown, we will scrub all of the
159 * pages in the list in case there are outstanding UEs. Then, we
160 * cross-check this list against the number of pages that are yet to be
161 * retired, and if we find inconsistencies, we scan every page_t in the
162 * whole system looking for any pages that need to be scrubbed for UEs.
163 * The background thread also uses this queue to determine which pages
164 * it should keep trying to retire.
167 #define PR_PENDING_QMAX 32
169 #define PR_PENDING_QMAX 256
171 page_t
*pr_pending_q
[PR_PENDING_QMAX
];
175 * Page retire global kstats
177 struct page_retire_kstat
{
178 kstat_named_t pr_retired
;
179 kstat_named_t pr_requested
;
180 kstat_named_t pr_requested_free
;
181 kstat_named_t pr_enqueue_fail
;
182 kstat_named_t pr_dequeue_fail
;
183 kstat_named_t pr_pending
;
184 kstat_named_t pr_pending_kas
;
185 kstat_named_t pr_failed
;
186 kstat_named_t pr_failed_kernel
;
187 kstat_named_t pr_limit
;
188 kstat_named_t pr_limit_exceeded
;
189 kstat_named_t pr_fma
;
190 kstat_named_t pr_mce
;
192 kstat_named_t pr_ue_cleared_retire
;
193 kstat_named_t pr_ue_cleared_free
;
194 kstat_named_t pr_ue_persistent
;
195 kstat_named_t pr_unretired
;
198 static struct page_retire_kstat page_retire_kstat
= {
199 { "pages_retired", KSTAT_DATA_UINT64
},
200 { "pages_retire_request", KSTAT_DATA_UINT64
},
201 { "pages_retire_request_free", KSTAT_DATA_UINT64
},
202 { "pages_notenqueued", KSTAT_DATA_UINT64
},
203 { "pages_notdequeued", KSTAT_DATA_UINT64
},
204 { "pages_pending", KSTAT_DATA_UINT64
},
205 { "pages_pending_kas", KSTAT_DATA_UINT64
},
206 { "pages_deferred", KSTAT_DATA_UINT64
},
207 { "pages_deferred_kernel", KSTAT_DATA_UINT64
},
208 { "pages_limit", KSTAT_DATA_UINT64
},
209 { "pages_limit_exceeded", KSTAT_DATA_UINT64
},
210 { "pages_fma", KSTAT_DATA_UINT64
},
211 { "pages_multiple_ce", KSTAT_DATA_UINT64
},
212 { "pages_ue", KSTAT_DATA_UINT64
},
213 { "pages_ue_cleared_retired", KSTAT_DATA_UINT64
},
214 { "pages_ue_cleared_freed", KSTAT_DATA_UINT64
},
215 { "pages_ue_persistent", KSTAT_DATA_UINT64
},
216 { "pages_unretired", KSTAT_DATA_UINT64
},
219 static kstat_t
*page_retire_ksp
= NULL
;
221 #define PR_INCR_KSTAT(stat) \
222 atomic_inc_64(&(page_retire_kstat.stat.value.ui64))
223 #define PR_DECR_KSTAT(stat) \
224 atomic_dec_64(&(page_retire_kstat.stat.value.ui64))
226 #define PR_KSTAT_RETIRED_CE (page_retire_kstat.pr_mce.value.ui64)
227 #define PR_KSTAT_RETIRED_FMA (page_retire_kstat.pr_fma.value.ui64)
228 #define PR_KSTAT_RETIRED_NOTUE (PR_KSTAT_RETIRED_CE + PR_KSTAT_RETIRED_FMA)
229 #define PR_KSTAT_PENDING (page_retire_kstat.pr_pending.value.ui64)
230 #define PR_KSTAT_PENDING_KAS (page_retire_kstat.pr_pending_kas.value.ui64)
231 #define PR_KSTAT_EQFAIL (page_retire_kstat.pr_enqueue_fail.value.ui64)
232 #define PR_KSTAT_DQFAIL (page_retire_kstat.pr_dequeue_fail.value.ui64)
235 * page retire kstats to list all retired pages
237 static int pr_list_kstat_update(kstat_t
*ksp
, int rw
);
238 static int pr_list_kstat_snapshot(kstat_t
*ksp
, void *buf
, int rw
);
239 kmutex_t pr_list_kstat_mutex
;
242 * Limit the number of multiple CE page retires.
243 * The default is 0.1% of physmem, or 1 in 1000 pages. This is set in
244 * basis points, where 100 basis points equals one percent.
247 uint64_t max_pages_retired_bps
= MCE_BPT
;
248 #define PAGE_RETIRE_LIMIT ((physmem * max_pages_retired_bps) / 10000)
251 * Control over the verbosity of page retirement.
253 * When set to zero (the default), no messages will be printed.
254 * When set to one, summary messages will be printed.
255 * When set > one, all messages will be printed.
257 * A value of one will trigger detailed messages for retirement operations,
258 * and is intended as a platform tunable for processors where FMA's DE does
259 * not run (e.g., spitfire). Values > one are intended for debugging only.
261 int page_retire_messages
= 0;
264 * Control whether or not we return scrubbed UE pages to service.
265 * By default we do not since FMA wants to run its diagnostics first
266 * and then ask us to unretire the page if it passes. Non-FMA platforms
267 * may set this to zero so we will only retire recidivist pages. It should
268 * not be changed by the user.
270 int page_retire_first_ue
= 1;
273 * Master enable for page retire. This prevents a CE or UE early in boot
274 * from trying to retire a page before page_retire_init() has finished
275 * setting things up. This is internal only and is not a tunable!
277 static int pr_enable
= 0;
279 static void (*memscrub_notify_func
)(uint64_t);
282 struct page_retire_debug
{
302 int prd_uenotscrubbed
;
314 int prd_checkmiss_pend
;
315 int prd_checkmiss_noerr
;
322 int prd_nofreedemote
;
327 #define PR_DEBUG(foo) ((pr_debug.foo)++)
330 * A type histogram. We record the incidence of the various toxic
331 * flag combinations along with the interesting page attributes. The
332 * goal is to get as many combinations as we can while driving all
333 * pr_debug values nonzero (indicating we've exercised all possible
334 * code paths across all possible page types). Not all combinations
335 * will make sense -- e.g. PRT_MOD|PRT_KERNEL.
337 * pr_type offset bit encoding (when examining with a debugger):
348 #define PRT_NAMED 0x01
349 #define PRT_KERNEL 0x02
350 #define PRT_FREE 0x04
352 #define PRT_FMA 0x00 /* yes, this is not a mistake */
357 int pr_types
[PRT_ALL
+1];
359 #define PR_TYPES(pp) { \
362 whichtype |= PRT_NAMED; \
364 whichtype |= PRT_KERNEL; \
366 whichtype |= PRT_FREE; \
368 whichtype |= PRT_MOD; \
369 if (pp->p_toxic & PR_UE) \
370 whichtype |= PRT_UE; \
371 if (pp->p_toxic & PR_MCE) \
372 whichtype |= PRT_MCE; \
373 pr_types[whichtype]++; \
383 #define MTBF(v, f) (((++(v)) & (f)) != (f))
387 #define PR_DEBUG(foo) /* nothing */
388 #define PR_TYPES(foo) /* nothing */
389 #define MTBF(v, f) (1)
394 * page_retire_done() - completion processing
396 * Used by the page_retire code for common completion processing.
397 * It keeps track of how many times a given result has happened,
398 * and writes out an occasional message.
400 * May be called with a NULL pp (PRD_INVALID_PA case).
402 #define PRD_INVALID_KEY -1
403 #define PRD_SUCCESS 0
404 #define PRD_PENDING 1
406 #define PRD_DUPLICATE 3
407 #define PRD_INVALID_PA 4
409 #define PRD_UE_SCRUBBED 6
410 #define PRD_UNR_SUCCESS 7
411 #define PRD_UNR_CANTLOCK 8
412 #define PRD_UNR_NOT 9
414 typedef struct page_retire_op
{
415 int pr_key
; /* one of the PRD_* defines from above */
416 int pr_count
; /* How many times this has happened */
417 int pr_retval
; /* return value */
418 int pr_msglvl
; /* message level - when to print */
419 char *pr_message
; /* Cryptic message for field service */
422 static page_retire_op_t page_retire_ops
[] = {
423 /* key count retval msglvl message */
424 {PRD_SUCCESS
, 0, 0, 1,
425 "Page 0x%08x.%08x removed from service"},
426 {PRD_PENDING
, 0, EAGAIN
, 2,
427 "Page 0x%08x.%08x will be retired on free"},
428 {PRD_FAILED
, 0, EAGAIN
, 0, NULL
},
429 {PRD_DUPLICATE
, 0, EIO
, 2,
430 "Page 0x%08x.%08x already retired or pending"},
431 {PRD_INVALID_PA
, 0, EINVAL
, 2,
432 "PA 0x%08x.%08x is not a relocatable page"},
434 "Page 0x%08x.%08x not retired due to limit exceeded"},
435 {PRD_UE_SCRUBBED
, 0, 0, 1,
436 "Previously reported error on page 0x%08x.%08x cleared"},
437 {PRD_UNR_SUCCESS
, 0, 0, 1,
438 "Page 0x%08x.%08x returned to service"},
439 {PRD_UNR_CANTLOCK
, 0, EAGAIN
, 2,
440 "Page 0x%08x.%08x could not be unretired"},
441 {PRD_UNR_NOT
, 0, EIO
, 2,
442 "Page 0x%08x.%08x is not retired"},
443 {PRD_INVALID_KEY
, 0, 0, 0, NULL
} /* MUST BE LAST! */
447 * print a message if page_retire_messages is true.
449 #define PR_MESSAGE(debuglvl, msglvl, msg, pa) \
451 uint64_t p = (uint64_t)pa; \
452 if (page_retire_messages >= msglvl && msg != NULL) { \
453 cmn_err(debuglvl, msg, \
454 (uint32_t)(p >> 32), (uint32_t)p); \
459 * Note that multiple bits may be set in a single settoxic operation.
460 * May be called without the page locked.
463 page_settoxic(page_t
*pp
, uchar_t bits
)
465 atomic_or_8(&pp
->p_toxic
, bits
);
469 * Note that multiple bits may cleared in a single clrtoxic operation.
470 * Must be called with the page exclusively locked to prevent races which
471 * may attempt to retire a page without any toxic bits set.
472 * Note that the PR_CAPTURE bit can be cleared without the exclusive lock
473 * being held as there is a separate mutex which protects that bit.
476 page_clrtoxic(page_t
*pp
, uchar_t bits
)
478 ASSERT((bits
& PR_CAPTURE
) || PAGE_EXCL(pp
));
479 atomic_and_8(&pp
->p_toxic
, ~bits
);
483 * Prints any page retire messages to the user, and decides what
484 * error code is appropriate for the condition reported.
487 page_retire_done(page_t
*pp
, int code
)
489 page_retire_op_t
*prop
;
494 pa
= mmu_ptob((uint64_t)pp
->p_pagenum
);
498 for (i
= 0; page_retire_ops
[i
].pr_key
!= PRD_INVALID_KEY
; i
++) {
499 if (page_retire_ops
[i
].pr_key
== code
) {
500 prop
= &page_retire_ops
[i
];
506 if (page_retire_ops
[i
].pr_key
== PRD_INVALID_KEY
) {
507 cmn_err(CE_PANIC
, "page_retire_done: Invalid opcode %d", code
);
511 ASSERT(prop
->pr_key
== code
);
515 PR_MESSAGE(CE_NOTE
, prop
->pr_msglvl
, prop
->pr_message
, pa
);
517 page_settoxic(pp
, PR_MSG
);
520 return (prop
->pr_retval
);
524 * Act like page_destroy(), but instead of freeing the page, hash it onto
525 * the retired_pages vnode, and mark it retired.
527 * For fun, we try to scrub the page until it's squeaky clean.
528 * availrmem is adjusted here.
531 page_retire_destroy(page_t
*pp
)
533 uoff_t off
= (uoff_t
)((uintptr_t)pp
);
535 ASSERT(PAGE_EXCL(pp
));
536 ASSERT(!PP_ISFREE(pp
));
537 ASSERT(pp
->p_szc
== 0);
538 ASSERT(!hat_page_is_mapped(pp
));
539 ASSERT(!pp
->p_vnode
);
541 page_clr_all_props(pp
);
542 pagescrub(pp
, 0, MMU_PAGESIZE
);
546 if (page_hashin(pp
, retired_pages
, off
, false) == 0) {
547 cmn_err(CE_PANIC
, "retired page %p hashin failed", (void *)pp
);
550 page_settoxic(pp
, PR_RETIRED
);
551 PR_INCR_KSTAT(pr_retired
);
553 if (pp
->p_toxic
& PR_FMA
) {
554 PR_INCR_KSTAT(pr_fma
);
555 } else if (pp
->p_toxic
& PR_UE
) {
556 PR_INCR_KSTAT(pr_ue
);
558 PR_INCR_KSTAT(pr_mce
);
561 mutex_enter(&freemem_lock
);
563 mutex_exit(&freemem_lock
);
569 * Check whether the number of pages which have been retired already exceeds
570 * the maximum allowable percentage of memory which may be retired.
572 * Returns 1 if the limit has been exceeded.
575 page_retire_limit(void)
577 if (PR_KSTAT_RETIRED_NOTUE
>= (uint64_t)PAGE_RETIRE_LIMIT
) {
578 PR_INCR_KSTAT(pr_limit_exceeded
);
585 #define MSG_DM "Data Mismatch occurred at PA 0x%08x.%08x" \
586 "[ 0x%x != 0x%x ] while attempting to clear previously " \
587 "reported error; page removed from service"
589 #define MSG_UE "Uncorrectable Error occurred at PA 0x%08x.%08x while " \
590 "attempting to clear previously reported error; page removed " \
594 * Attempt to clear a UE from a page.
595 * Returns 1 if the error has been successfully cleared.
598 page_clear_transient_ue(page_t
*pp
)
603 uint32_t pa_hi
, pa_lo
;
608 ASSERT(PAGE_EXCL(pp
));
609 ASSERT(PP_PR_REQ(pp
));
610 ASSERT(pp
->p_szc
== 0);
611 ASSERT(!hat_page_is_mapped(pp
));
614 * Clear the page and attempt to clear the UE. If we trap
615 * on the next access to the page, we know the UE has recurred.
617 pagescrub(pp
, 0, PAGESIZE
);
620 * Map the page and write a bunch of bit patterns to compare
621 * what we wrote with what we read back. This isn't a perfect
622 * test but it should be good enough to catch most of the
623 * recurring UEs. If this fails to catch a recurrent UE, we'll
624 * retire the page the next time we see a UE on the page.
626 kaddr
= ppmapin(pp
, PROT_READ
|PROT_WRITE
, (caddr_t
)-1);
628 pa
= ptob((uint64_t)page_pptonum(pp
));
629 pa_hi
= (uint32_t)(pa
>> 32);
630 pa_lo
= (uint32_t)pa
;
633 * Disable preemption to prevent the off chance that
634 * we migrate while in the middle of running through
635 * the bit pattern and run on a different processor
636 * than what we started on.
641 * Fill the page with each (0x00 - 0xFF] bit pattern, flushing
642 * the cache in between reading and writing. We do this under
643 * on_trap() protection to avoid recursion.
645 if (on_trap(&otd
, OT_DATA_EC
)) {
646 PR_MESSAGE(CE_WARN
, 1, MSG_UE
, pa
);
649 for (wb
= 0xff; wb
> 0; wb
--) {
650 for (i
= 0; i
< PAGESIZE
; i
++) {
654 sync_data_memory(kaddr
, PAGESIZE
);
656 for (i
= 0; i
< PAGESIZE
; i
++) {
660 * We had a mismatch without a trap.
661 * Uh-oh. Something is really wrong
664 if (page_retire_messages
) {
665 cmn_err(CE_WARN
, MSG_DM
,
666 pa_hi
, pa_lo
, rb
, wb
);
669 goto out
; /* double break */
679 return (errors
? 0 : 1);
683 * Try to clear a page_t with a single UE. If the UE was transient, it is
684 * returned to service, and we return 1. Otherwise we return 0 meaning
685 * that further processing is required to retire the page.
688 page_retire_transient_ue(page_t
*pp
)
690 ASSERT(PAGE_EXCL(pp
));
691 ASSERT(!hat_page_is_mapped(pp
));
694 * If this page is a repeat offender, retire it under the
695 * "two strikes and you're out" rule. The caller is responsible
696 * for scrubbing the page to try to clear the error.
698 if (pp
->p_toxic
& PR_UE_SCRUBBED
) {
699 PR_INCR_KSTAT(pr_ue_persistent
);
703 if (page_clear_transient_ue(pp
)) {
705 * We set the PR_SCRUBBED_UE bit; if we ever see this
706 * page again, we will retire it, no questions asked.
708 page_settoxic(pp
, PR_UE_SCRUBBED
);
710 if (page_retire_first_ue
) {
711 PR_INCR_KSTAT(pr_ue_cleared_retire
);
714 PR_INCR_KSTAT(pr_ue_cleared_free
);
716 page_clrtoxic(pp
, PR_UE
| PR_MCE
| PR_MSG
);
718 VN_DISPOSE(pp
, B_FREE
, 1, kcred
);
723 PR_INCR_KSTAT(pr_ue_persistent
);
728 * Update the statistics dynamically when our kstat is read.
731 page_retire_kstat_update(kstat_t
*ksp
, int rw
)
733 struct page_retire_kstat
*pr
;
741 pr
= (struct page_retire_kstat
*)ksp
->ks_data
;
742 ASSERT(pr
== &page_retire_kstat
);
743 pr
->pr_limit
.value
.ui64
= PAGE_RETIRE_LIMIT
;
756 pr_list_kstat_update(kstat_t
*ksp
, int rw
)
761 if (rw
== KSTAT_WRITE
)
764 mutex_enter(page_vnode_mutex(retired_pages
));
765 /* Needs to be under a lock so that for loop will work right */
766 if (!vn_has_cached_data(retired_pages
)) {
767 mutex_exit(page_vnode_mutex(retired_pages
));
769 ksp
->ks_data_size
= 0;
774 for (pp
= vnode_get_next(retired_pages
, vnode_get_head(retired_pages
));
775 pp
!= NULL
; pp
= vnode_get_next(retired_pages
, pp
)) {
778 mutex_exit(page_vnode_mutex(retired_pages
));
780 ksp
->ks_ndata
= count
;
781 ksp
->ks_data_size
= count
* 2 * sizeof (uint64_t);
787 * all spans will be pagesize and no coalescing will be done with the
791 pr_list_kstat_snapshot(kstat_t
*ksp
, void *buf
, int rw
)
799 if (rw
== KSTAT_WRITE
)
802 ksp
->ks_snaptime
= gethrtime();
804 kspmem
= (struct memunit
*)buf
;
806 mutex_enter(page_vnode_mutex(retired_pages
));
807 pp
= vnode_get_head(retired_pages
);
808 if (((caddr_t
)kspmem
>= (caddr_t
)buf
+ ksp
->ks_data_size
) ||
810 mutex_exit(page_vnode_mutex(retired_pages
));
813 kspmem
->address
= ptob(pp
->p_pagenum
);
814 kspmem
->size
= PAGESIZE
;
816 for (pp
= vnode_get_next(retired_pages
, pp
); pp
!= NULL
;
817 pp
= vnode_get_next(retired_pages
, pp
), kspmem
++) {
818 if ((caddr_t
)kspmem
>= (caddr_t
)buf
+ ksp
->ks_data_size
)
820 kspmem
->address
= ptob(pp
->p_pagenum
);
821 kspmem
->size
= PAGESIZE
;
823 mutex_exit(page_vnode_mutex(retired_pages
));
829 * page_retire_pend_count -- helper function for page_capture_thread,
830 * returns the number of pages pending retirement.
833 page_retire_pend_count(void)
835 return (PR_KSTAT_PENDING
);
839 page_retire_pend_kas_count(void)
841 return (PR_KSTAT_PENDING_KAS
);
845 page_retire_incr_pend_count(void *datap
)
847 PR_INCR_KSTAT(pr_pending
);
849 if ((datap
== &kvp
) || (datap
== &zvp
)) {
850 PR_INCR_KSTAT(pr_pending_kas
);
855 page_retire_decr_pend_count(void *datap
)
857 PR_DECR_KSTAT(pr_pending
);
859 if ((datap
== &kvp
) || (datap
== &zvp
)) {
860 PR_DECR_KSTAT(pr_pending_kas
);
865 * Initialize the page retire mechanism:
867 * - Establish the correctable error retire limit.
868 * - Initialize locks.
869 * - Build the retired_pages vnode.
870 * - Set up the kstats.
871 * - Fire off the background thread.
872 * - Tell page_retire() it's OK to start retiring pages.
875 page_retire_init(void)
877 const fs_operation_def_t retired_vnodeops_template
[] = {
880 struct vnodeops
*vops
;
883 const uint_t page_retire_ndata
=
884 sizeof (page_retire_kstat
) / sizeof (kstat_named_t
);
886 ASSERT(page_retire_ksp
== NULL
);
888 if (max_pages_retired_bps
<= 0) {
889 max_pages_retired_bps
= MCE_BPT
;
892 mutex_init(&pr_q_mutex
, NULL
, MUTEX_DEFAULT
, NULL
);
894 retired_pages
= vn_alloc(KM_SLEEP
);
895 if (vn_make_ops("retired_pages", retired_vnodeops_template
, &vops
)) {
897 "page_retired_init: can't make retired vnodeops");
899 vn_setops(retired_pages
, vops
);
901 if ((page_retire_ksp
= kstat_create("unix", 0, "page_retire",
902 "misc", KSTAT_TYPE_NAMED
, page_retire_ndata
,
903 KSTAT_FLAG_VIRTUAL
)) == NULL
) {
904 cmn_err(CE_WARN
, "kstat_create for page_retire failed");
906 page_retire_ksp
->ks_data
= (void *)&page_retire_kstat
;
907 page_retire_ksp
->ks_update
= page_retire_kstat_update
;
908 kstat_install(page_retire_ksp
);
911 mutex_init(&pr_list_kstat_mutex
, NULL
, MUTEX_DEFAULT
, NULL
);
912 ksp
= kstat_create("unix", 0, "page_retire_list", "misc",
913 KSTAT_TYPE_RAW
, 0, KSTAT_FLAG_VAR_SIZE
| KSTAT_FLAG_VIRTUAL
);
915 ksp
->ks_update
= pr_list_kstat_update
;
916 ksp
->ks_snapshot
= pr_list_kstat_snapshot
;
917 ksp
->ks_lock
= &pr_list_kstat_mutex
;
921 memscrub_notify_func
=
922 (void(*)(uint64_t))kobj_getsymvalue("memscrub_notify", 0);
924 page_capture_register_callback(PC_RETIRE
, -1, page_retire_pp_finish
);
929 * page_retire_hunt() callback for the retire thread.
932 page_retire_thread_cb(page_t
*pp
)
935 if (!PP_ISKAS(pp
) && page_trylock(pp
, SE_EXCL
)) {
936 PR_DEBUG(prd_tclocked
);
942 * Callback used by page_trycapture() to finish off retiring a page.
943 * The page has already been cleaned and we've been given sole access to
945 * Always returns 0 to indicate that callback succeded as the callback never
946 * fails to finish retiring the given page.
950 page_retire_pp_finish(page_t
*pp
, void *notused
, uint_t flags
)
954 ASSERT(PAGE_EXCL(pp
));
955 ASSERT(pp
->p_iolock_state
== 0);
956 ASSERT(pp
->p_szc
== 0);
961 * The problem page is locked, demoted, unmapped, not free,
962 * hashed out, and not COW or mlocked (whew!).
964 * Now we select our ammunition, take it around back, and shoot it.
968 if (page_retire_transient_ue(pp
)) {
969 PR_DEBUG(prd_uescrubbed
);
970 (void) page_retire_done(pp
, PRD_UE_SCRUBBED
);
972 PR_DEBUG(prd_uenotscrubbed
);
973 page_retire_destroy(pp
);
974 (void) page_retire_done(pp
, PRD_SUCCESS
);
977 } else if (toxic
& PR_FMA
) {
979 page_retire_destroy(pp
);
980 (void) page_retire_done(pp
, PRD_SUCCESS
);
982 } else if (toxic
& PR_MCE
) {
984 page_retire_destroy(pp
);
985 (void) page_retire_done(pp
, PRD_SUCCESS
);
990 * When page_retire_first_ue is set to zero and a UE occurs which is
991 * transient, it's possible that we clear some flags set by a second
992 * UE error on the page which occurs while the first is currently being
993 * handled and thus we need to handle the case where none of the above
994 * are set. In this instance, PR_UE_SCRUBBED should be set and thus
995 * we should execute the UE code above.
997 if (toxic
& PR_UE_SCRUBBED
) {
1002 * It's impossible to get here.
1004 panic("bad toxic flags 0x%x in page_retire_pp_finish\n", toxic
);
1009 * page_retire() - the front door in to retire a page.
1011 * Ideally, page_retire() would instantly retire the requested page.
1012 * Unfortunately, some pages are locked or otherwise tied up and cannot be
1013 * retired right away. We use the page capture logic to deal with this
1014 * situation as it will continuously try to retire the page in the background
1015 * if the first attempt fails. Success is determined by looking to see whether
1016 * the page has been retired after the page_trycapture() attempt.
1021 * - EINVAL when the PA is whacko,
1022 * - EIO if the page is already retired or already pending retirement, or
1023 * - EAGAIN if the page could not be _immediately_ retired but is pending.
1026 page_retire(uint64_t pa
, uchar_t reason
)
1030 ASSERT(reason
& PR_REASONS
); /* there must be a reason */
1031 ASSERT(!(reason
& ~PR_REASONS
)); /* but no other bits */
1033 pp
= page_numtopp_nolock(mmu_btop(pa
));
1035 PR_MESSAGE(CE_WARN
, 1, "Cannot schedule clearing of error on"
1036 " page 0x%08x.%08x; page is not relocatable memory", pa
);
1037 return (page_retire_done(pp
, PRD_INVALID_PA
));
1039 if (PP_RETIRED(pp
)) {
1041 return (page_retire_done(pp
, PRD_DUPLICATE
));
1044 if (memscrub_notify_func
!= NULL
) {
1045 (void) memscrub_notify_func(pa
);
1048 if ((reason
& PR_UE
) && !PP_TOXIC(pp
)) {
1049 PR_MESSAGE(CE_NOTE
, 1, "Scheduling clearing of error on"
1050 " page 0x%08x.%08x", pa
);
1051 } else if (PP_PR_REQ(pp
)) {
1053 return (page_retire_done(pp
, PRD_DUPLICATE
));
1055 PR_MESSAGE(CE_NOTE
, 1, "Scheduling removal of"
1056 " page 0x%08x.%08x", pa
);
1059 /* Avoid setting toxic bits in the first place */
1060 if ((reason
& (PR_FMA
| PR_MCE
)) && !(reason
& PR_UE
) &&
1061 page_retire_limit()) {
1062 return (page_retire_done(pp
, PRD_LIMIT
));
1065 if (MTBF(pr_calls
, pr_mtbf
)) {
1066 page_settoxic(pp
, reason
);
1067 if (page_trycapture(pp
, 0, CAPTURE_RETIRE
, pp
->p_vnode
) == 0) {
1068 PR_DEBUG(prd_prlocked
);
1070 PR_DEBUG(prd_prnotlocked
);
1073 PR_DEBUG(prd_prnotlocked
);
1076 if (PP_RETIRED(pp
)) {
1077 PR_DEBUG(prd_prretired
);
1081 PR_INCR_KSTAT(pr_failed
);
1083 if (pp
->p_toxic
& PR_MSG
) {
1084 return (page_retire_done(pp
, PRD_FAILED
));
1086 return (page_retire_done(pp
, PRD_PENDING
));
1092 * Take a retired page off the retired-pages vnode and clear the toxic flags.
1093 * If "free" is nonzero, lock it and put it back on the freelist. If "free"
1094 * is zero, the caller already holds SE_EXCL lock so we simply unretire it
1095 * and don't do anything else with it.
1097 * Any unretire messages are printed from this routine.
1099 * Returns 0 if page pp was unretired; else an error code.
1102 * PR_UNR_FREE - lock the page, clear the toxic flags and free it
1104 * PR_UNR_TEMP - lock the page, unretire it, leave the toxic
1105 * bits set as is and return it to the caller.
1106 * PR_UNR_CLEAN - page is SE_EXCL locked, unretire it, clear the
1107 * toxic flags and return it to caller as is.
1110 page_unretire_pp(page_t
*pp
, int flags
)
1113 * To be retired, a page has to be hashed onto the retired_pages vnode
1114 * and have PR_RETIRED set in p_toxic.
1116 if (flags
== PR_UNR_CLEAN
||
1117 page_try_reclaim_lock(pp
, SE_EXCL
, SE_RETIRED
)) {
1118 ASSERT(PAGE_EXCL(pp
));
1119 PR_DEBUG(prd_ulocked
);
1120 if (!PP_RETIRED(pp
)) {
1121 PR_DEBUG(prd_unotretired
);
1123 return (page_retire_done(pp
, PRD_UNR_NOT
));
1126 PR_MESSAGE(CE_NOTE
, 1, "unretiring retired"
1127 " page 0x%08x.%08x", mmu_ptob((uint64_t)pp
->p_pagenum
));
1128 if (pp
->p_toxic
& PR_FMA
) {
1129 PR_DECR_KSTAT(pr_fma
);
1130 } else if (pp
->p_toxic
& PR_UE
) {
1131 PR_DECR_KSTAT(pr_ue
);
1133 PR_DECR_KSTAT(pr_mce
);
1136 if (flags
== PR_UNR_TEMP
)
1137 page_clrtoxic(pp
, PR_RETIRED
);
1139 page_clrtoxic(pp
, PR_TOXICFLAGS
);
1141 if (flags
== PR_UNR_FREE
) {
1142 PR_DEBUG(prd_udestroy
);
1143 page_destroy(pp
, 0);
1145 PR_DEBUG(prd_uhashout
);
1146 page_hashout(pp
, false);
1149 mutex_enter(&freemem_lock
);
1151 mutex_exit(&freemem_lock
);
1153 PR_DEBUG(prd_uunretired
);
1154 PR_DECR_KSTAT(pr_retired
);
1155 PR_INCR_KSTAT(pr_unretired
);
1156 return (page_retire_done(pp
, PRD_UNR_SUCCESS
));
1158 PR_DEBUG(prd_unotlocked
);
1159 return (page_retire_done(pp
, PRD_UNR_CANTLOCK
));
1163 * Return a page to service by moving it from the retired_pages vnode
1164 * onto the freelist.
1166 * Called from mmioctl_page_retire() on behalf of the FMA DE.
1170 * - 0 if the page is unretired,
1171 * - EAGAIN if the pp can not be locked,
1172 * - EINVAL if the PA is whacko, and
1173 * - EIO if the pp is not retired.
1176 page_unretire(uint64_t pa
)
1180 pp
= page_numtopp_nolock(mmu_btop(pa
));
1182 return (page_retire_done(pp
, PRD_INVALID_PA
));
1185 return (page_unretire_pp(pp
, PR_UNR_FREE
));
1189 * Test a page to see if it is retired. If errors is non-NULL, the toxic
1190 * bits of the page are returned. Returns 0 on success, error code on failure.
1193 page_retire_check_pp(page_t
*pp
, uint64_t *errors
)
1197 if (PP_RETIRED(pp
)) {
1198 PR_DEBUG(prd_checkhit
);
1200 } else if (PP_PR_REQ(pp
)) {
1201 PR_DEBUG(prd_checkmiss_pend
);
1204 PR_DEBUG(prd_checkmiss_noerr
);
1209 * We have magically arranged the bit values returned to fmd(1M)
1210 * to line up with the FMA, MCE, and UE bits of the page_t.
1213 uint64_t toxic
= (uint64_t)(pp
->p_toxic
& PR_ERRMASK
);
1214 if (toxic
& PR_UE_SCRUBBED
) {
1215 toxic
&= ~PR_UE_SCRUBBED
;
1225 * Test to see if the page_t for a given PA is retired, and return the
1226 * hardware errors we have seen on the page if requested.
1228 * Called from mmioctl_page_retire on behalf of the FMA DE.
1232 * - 0 if the page is retired,
1233 * - EIO if the page is not retired and has no errors,
1234 * - EAGAIN if the page is not retired but is pending; and
1235 * - EINVAL if the PA is whacko.
1238 page_retire_check(uint64_t pa
, uint64_t *errors
)
1246 pp
= page_numtopp_nolock(mmu_btop(pa
));
1248 return (page_retire_done(pp
, PRD_INVALID_PA
));
1251 return (page_retire_check_pp(pp
, errors
));
1255 * Page retire self-test. For now, it always returns 0.
1258 page_retire_test(void)
1260 page_t
*first
, *pp
, *cpp
, *cpp2
, *lpp
;
1263 * Tests the corner case where a large page can't be retired
1264 * because one of the constituent pages is locked. We mark
1265 * one page to be retired and try to retire it, and mark the
1266 * other page to be retired but don't try to retire it, so
1267 * that page_unlock() in the failure path will recurse and try
1268 * to retire THAT page. This is the worst possible situation
1269 * we can get ourselves into.
1272 pp
= first
= page_first();
1274 if (pp
->p_szc
&& PP_PAGEROOT(pp
) == pp
) {
1276 lpp
= PP_ISFREE(pp
)? pp
: pp
+ 2;
1278 if (!page_trylock(lpp
, pp
== lpp
? SE_EXCL
: SE_SHARED
))
1280 if (!page_trylock(cpp
, SE_EXCL
)) {
1286 (void) page_retire(ptob(cpp
->p_pagenum
), PR_FMA
);
1290 (void) page_retire(ptob(cpp
->p_pagenum
), PR_FMA
);
1291 (void) page_retire(ptob(cpp2
->p_pagenum
), PR_FMA
);
1293 } while ((pp
= page_next(pp
)) != first
);