linux-fedora/oom-fix-integer-overflow-of-points.patch

   1
   2 Delivered-To: jwboyer@gmail.com
   3 Received: by 10.220.45.11 with SMTP id c11cs62970vcf;
   4         Mon, 31 Oct 2011 08:56:49 -0700 (PDT)
   5 Received: by 10.101.15.19 with SMTP id s19mr2706064ani.103.1320076596057;
   6         Mon, 31 Oct 2011 08:56:36 -0700 (PDT)
   7 Return-Path: <linux-kernel-owner@vger.kernel.org>
   8 Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67])
   9         by mx.google.com with ESMTP id x8si7676575ani.27.2011.10.31.08.56.32;
  10         Mon, 31 Oct 2011 08:56:36 -0700 (PDT)
  11 Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
  12 Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mail=linux-kernel-owner@vger.kernel.org
  13 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
  14         id S934545Ab1JaP4X (ORCPT <rfc822;mel.lkml@gmail.com> + 99 others);
  15         Mon, 31 Oct 2011 11:56:23 -0400
  16 Received: from mx1.redhat.com ([209.132.183.28]:23653 "EHLO mx1.redhat.com"
  17         rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
  18         id S934538Ab1JaP4X (ORCPT <rfc822;linux-kernel@vger.kernel.org>);
  19         Mon, 31 Oct 2011 11:56:23 -0400
  20 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
  21         by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p9VFuHOO027543
  22         (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
  23         Mon, 31 Oct 2011 11:56:18 -0400
  24 Received: from dhcp-26-164.brq.redhat.com (dhcp-26-164.brq.redhat.com [10.34.26.164])
  25         by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id p9VFuEK3018476;
  26         Mon, 31 Oct 2011 11:56:15 -0400
  27 From:   Frantisek Hrbata <fhrbata@redhat.com>
  28 To:     rientjes@google.com
  29 Cc:     linux-mm@kvack.org, linux-kernel@vger.kernel.org,
  30         akpm@linux-foundation.org, kosaki.motohiro@jp.fujitsu.com,
  31         oleg@redhat.com, minchan.kim@gmail.com, stable@kernel.org,
  32         eteo@redhat.com, pmatouse@redhat.com
  33 Subject: [PATCH v2] oom: fix integer overflow of points in oom_badness
  34 Date:   Mon, 31 Oct 2011 16:56:09 +0100
  35 Message-Id: <1320076569-23872-1-git-send-email-fhrbata@redhat.com>
  36 In-Reply-To: <1320048865-13175-1-git-send-email-fhrbata@redhat.com>
  37 References: <1320048865-13175-1-git-send-email-fhrbata@redhat.com>
  38 X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
  39 Sender: linux-kernel-owner@vger.kernel.org
  40 Precedence: bulk
  41 List-ID: <linux-kernel.vger.kernel.org>
  42 X-Mailing-List: linux-kernel@vger.kernel.org
  43
  44 An integer overflow will happen on 64bit archs if task's sum of rss, swapents
  45 and nr_ptes exceeds (2^31)/1000 value. This was introduced by commit
  46
  47 f755a04 oom: use pte pages in OOM score
  48
  49 where the oom score computation was divided into several steps and it's no
  50 longer computed as one expression in unsigned long(rss, swapents, nr_pte are
  51 unsigned long), where the result value assigned to points(int) is in
  52 range(1..1000). So there could be an int overflow while computing
  53
  54 176          points *= 1000;
  55
  56 and points may have negative value. Meaning the oom score for a mem hog task
  57 will be one.
  58
  59 196          if (points <= 0)
  60 197                  return 1;
  61
  62 For example:
  63 [ 3366]     0  3366 35390480 24303939   5       0             0 oom01
  64 Out of memory: Kill process 3366 (oom01) score 1 or sacrifice child
  65
  66 Here the oom1 process consumes more than 24303939(rss)*4096~=92GB physical
  67 memory, but it's oom score is one.
  68
  69 In this situation the mem hog task is skipped and oom killer kills another and
  70 most probably innocent task with oom score greater than one.
  71
  72 The points variable should be of type long instead of int to prevent the int
  73 overflow.
  74
  75 Signed-off-by: Frantisek Hrbata <fhrbata@redhat.com>
  76 ---
  77  mm/oom_kill.c |    2 +-
  78  1 files changed, 1 insertions(+), 1 deletions(-)
  79
  80 diff --git a/mm/oom_kill.c b/mm/oom_kill.c
  81 index 626303b..e9a1785 100644
  82 --- a/mm/oom_kill.c
  83 +++ b/mm/oom_kill.c
  84 @@ -162,7 +162,7 @@ static bool oom_unkillable_task(struct task_struct *p,
  85  unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *mem,
  86                       const nodemask_t *nodemask, unsigned long totalpages)
  87  {
  88 -       int points;
  89 +       long points;
  90
  91         if (oom_unkillable_task(p, mem, nodemask))
  92                 return 0;
  93 --
  94 1.7.6.4
  95
  96 --
  97 To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
  98 the body of a message to majordomo@vger.kernel.org
  99 More majordomo info at  http://vger.kernel.org/majordomo-info.html
 100 Please read the FAQ at  http://www.tux.org/lkml/