Make speculative accept() faster for the common case