kernel - Improve physio performance (2)
* Increase the cap on pbuf_mem buffers from 256 to 512. 256
wasn't enough to max-out three NVMe devices.
* Add 25% hysteresis to the pbuf_{mem,kva,raw}_count counters
to reduce unnecessary tsleep()s and wakeup()s (and thus
unnecessary IPIs) when the pbuf pool is exhausted.
Add a tiny bit of hysteresis for the localized *pfreecnt
as subsystems tend to use smaller values (e.g. pageout
code).
* In physio tests throughput with 3 x NVMe + 4 x SATA SSDs
increases to 6.5 GBytes/sec and max IOPS @ 4K increases
to 1.05M IOPS (yes, that's million). (random read
from urandom-filled partition using 32KB and 4KB blocks,
with high user process concurrency).