Support fused multiply-adds in fully-masked reductions
commite3798ed9f885f93206156529d61dd08a721b5f96
authorrsandifo <rsandifo@138bc75d-0d04-0410-961f-82ee72b054a4>
Thu, 12 Jul 2018 13:01:48 +0000 (12 13:01 +0000)
committerrsandifo <rsandifo@138bc75d-0d04-0410-961f-82ee72b054a4>
Thu, 12 Jul 2018 13:01:48 +0000 (12 13:01 +0000)
tree87c7aa8363d38fe0e4022d33f8a25f14b6dc8ff8
parent6682fc023eac9c73fbe4d025463f648df876c158
Support fused multiply-adds in fully-masked reductions

This patch adds support for fusing a conditional add or subtract
with a multiplication, so that we can use fused multiply-add and
multiply-subtract operations for fully-masked reductions.  E.g.
for SVE we vectorise:

  double res = 0.0;
  for (int i = 0; i < n; ++i)
    res += x[i] * y[i];

using a fully-masked loop in which the loop body has the form:

  res_1 = PHI<0(preheader), res_2(latch)>;
  avec = .MASK_LOAD (loop_mask, a)
  bvec = .MASK_LOAD (loop_mask, b)
  prod = avec * bvec;
  res_2 = .COND_ADD (loop_mask, res_1, prod, res_1);

where the last statement does the equivalent of:

  res_2 = loop_mask ? res_1 + prod : res_1;

(operating elementwise).  The point of the patch is to convert the last
two statements into:

  res_s = .COND_FMA (loop_mask, avec, bvec, res_1, res_1);

which is equivalent to:

  res_2 = loop_mask ? fma (avec, bvec, res_1) : res_1;

(again operating elementwise).

2018-07-12  Richard Sandiford  <richard.sandiford@linaro.org>
    Alan Hayward  <alan.hayward@arm.com>
    David Sherwood  <david.sherwood@arm.com>

gcc/
* internal-fn.h (can_interpret_as_conditional_op_p): Declare.
* internal-fn.c (can_interpret_as_conditional_op_p): New function.
* tree-ssa-math-opts.c (convert_mult_to_fma_1): Handle conditional
plus and minus and convert them into IFN_COND_FMA-based sequences.
(convert_mult_to_fma): Handle conditional plus and minus.

gcc/testsuite/
* gcc.dg/vect/vect-fma-2.c: New test.
* gcc.target/aarch64/sve/reduc_4.c: Likewise.
* gcc.target/aarch64/sve/reduc_6.c: Likewise.
* gcc.target/aarch64/sve/reduc_7.c: Likewise.

git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk@262588 138bc75d-0d04-0410-961f-82ee72b054a4
gcc/ChangeLog
gcc/internal-fn.c
gcc/internal-fn.h
gcc/testsuite/ChangeLog
gcc/testsuite/gcc.dg/vect/vect-fma-2.c [new file with mode: 0644]
gcc/testsuite/gcc.target/aarch64/sve/reduc_4.c [new file with mode: 0644]
gcc/testsuite/gcc.target/aarch64/sve/reduc_6.c [new file with mode: 0644]
gcc/testsuite/gcc.target/aarch64/sve/reduc_7.c [new file with mode: 0644]
gcc/tree-ssa-math-opts.c