From ef4330c23bb47b97a859dbdbae1c784fd2ca402f Mon Sep 17 00:00:00 2001 From: Peter Maydell Date: Fri, 22 Jul 2016 13:41:52 +0100 Subject: [PATCH] linux-user: Handle brk() attempts with very large sizes In do_brk(), we were inadvertently truncating the size of a requested brk() from the guest by putting it into an 'int' variable. This meant that we would incorrectly report success back to the guest rather than a failed allocation, typically resulting in the guest then segfaulting. Use abi_ulong instead. This fixes a crash in the '31370.cc' test in the gcc libstdc++ test suite (the test case starts by trying to allocate a very large size and reduces the size until the allocation succeeds). Signed-off-by: Peter Maydell Signed-off-by: Riku Voipio --- linux-user/syscall.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/linux-user/syscall.c b/linux-user/syscall.c index df6f2a9d0f..833f853200 100644 --- a/linux-user/syscall.c +++ b/linux-user/syscall.c @@ -839,7 +839,7 @@ void target_set_brk(abi_ulong new_brk) abi_long do_brk(abi_ulong new_brk) { abi_long mapped_addr; - int new_alloc_size; + abi_ulong new_alloc_size; DEBUGF_BRK("do_brk(" TARGET_ABI_FMT_lx ") -> ", new_brk); -- 2.11.4.GIT