cheneygc: Fix heap overflow detection for unsigned os_vm_size_t

cheneygc: Fix heap overflow detection for unsigned os_vm_size_t

  * Instead of checking to see if the overall size of the trigger
area (unused part of a dynamic space) is negative, check first to
see if it would BE negative if the operation were to be performed,
allowing the high-water detection to occur even if os_vm_size_t is
unsigned (thus, not having any negative values).

  * As a practical upshot, this means that we get a sane error
message when we run out of heap space, instead of either silently
doing something stupidly wrong or having a rather opaque message
about mprotect() failing and various advice that applies to gencgc
but not cheneygc.