Fix profile update after peeled epilogues
commit838237aeeba578fc2cf42bfd3ecb9d9a4fb7a2b4
authorJan Hubicka <jh@suse.cz>
Sun, 6 Aug 2023 19:23:31 +0000 (6 21:23 +0200)
committerJan Hubicka <jh@suse.cz>
Sun, 6 Aug 2023 19:23:31 +0000 (6 21:23 +0200)
tree2539c5a2f161c15c5d4f109dcdcdef65d0c70374
parent3802297528685aa16c4f43bb1f0cedbdbf22923d
Fix profile update after peeled epilogues

Epilogue peeling expects the scalar loop to have same number of executions as
the vector loop which is true at the beggining of vectorization. However if the
epilogues are vectorized, this is no longer the case.  In this situation the
loop preheader is replaced by new guard code with correct profile, however
loop body is left unscaled.  This leads to loop that exists more often then
it is entered.

This patch add slogic to scale the frequencies down and also to fix profile
of original preheader where necesary.

Bootstrapped/regtested x86_64-linux, comitted.

gcc/ChangeLog:

* tree-vect-loop-manip.cc (vect_do_peeling): Fix profile update of peeled epilogues.

gcc/testsuite/ChangeLog:

* gcc.dg/vect/vect-bitfield-read-1.c: Check profile consistency.
* gcc.dg/vect/vect-bitfield-read-2.c: Check profile consistency.
* gcc.dg/vect/vect-bitfield-read-3.c: Check profile consistency.
* gcc.dg/vect/vect-bitfield-read-4.c: Check profile consistency.
* gcc.dg/vect/vect-bitfield-read-5.c: Check profile consistency.
* gcc.dg/vect/vect-bitfield-read-6.c: Check profile consistency.
* gcc.dg/vect/vect-bitfield-read-7.c: Check profile consistency.
* gcc.dg/vect/vect-bitfield-write-1.c: Check profile consistency.
* gcc.dg/vect/vect-bitfield-write-2.c: Check profile consistency.
* gcc.dg/vect/vect-bitfield-write-3.c: Check profile consistency.
* gcc.dg/vect/vect-bitfield-write-4.c: Check profile consistency.
* gcc.dg/vect/vect-bitfield-write-5.c: Check profile consistency.
* gcc.dg/vect/vect-epilogues-2.c: Check profile consistency.
* gcc.dg/vect/vect-epilogues.c: Check profile consistency.
* gcc.dg/vect/vect-mask-store-move-1.c: Check profile consistency.
16 files changed:
gcc/testsuite/gcc.dg/vect/vect-bitfield-read-1.c
gcc/testsuite/gcc.dg/vect/vect-bitfield-read-2.c
gcc/testsuite/gcc.dg/vect/vect-bitfield-read-3.c
gcc/testsuite/gcc.dg/vect/vect-bitfield-read-4.c
gcc/testsuite/gcc.dg/vect/vect-bitfield-read-5.c
gcc/testsuite/gcc.dg/vect/vect-bitfield-read-6.c
gcc/testsuite/gcc.dg/vect/vect-bitfield-read-7.c
gcc/testsuite/gcc.dg/vect/vect-bitfield-write-1.c
gcc/testsuite/gcc.dg/vect/vect-bitfield-write-2.c
gcc/testsuite/gcc.dg/vect/vect-bitfield-write-3.c
gcc/testsuite/gcc.dg/vect/vect-bitfield-write-4.c
gcc/testsuite/gcc.dg/vect/vect-bitfield-write-5.c
gcc/testsuite/gcc.dg/vect/vect-epilogues-2.c
gcc/testsuite/gcc.dg/vect/vect-epilogues.c
gcc/testsuite/gcc.dg/vect/vect-mask-store-move-1.c
gcc/tree-vect-loop-manip.cc