xfail experimental/memory_resource/resource_adaptor.cc on 32-bit Solaris/x86 (PR...
[official-gcc.git] / gcc / haifa-sched.c
blob4a899b56173dd7d67db084ed86d24ad865ccadcd
1 /* Instruction scheduling pass.
2 Copyright (C) 1992-2018 Free Software Foundation, Inc.
3 Contributed by Michael Tiemann (tiemann@cygnus.com) Enhanced by,
4 and currently maintained by, Jim Wilson (wilson@cygnus.com)
6 This file is part of GCC.
8 GCC is free software; you can redistribute it and/or modify it under
9 the terms of the GNU General Public License as published by the Free
10 Software Foundation; either version 3, or (at your option) any later
11 version.
13 GCC is distributed in the hope that it will be useful, but WITHOUT ANY
14 WARRANTY; without even the implied warranty of MERCHANTABILITY or
15 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
16 for more details.
18 You should have received a copy of the GNU General Public License
19 along with GCC; see the file COPYING3. If not see
20 <http://www.gnu.org/licenses/>. */
22 /* Instruction scheduling pass. This file, along with sched-deps.c,
23 contains the generic parts. The actual entry point for
24 the normal instruction scheduling pass is found in sched-rgn.c.
26 We compute insn priorities based on data dependencies. Flow
27 analysis only creates a fraction of the data-dependencies we must
28 observe: namely, only those dependencies which the combiner can be
29 expected to use. For this pass, we must therefore create the
30 remaining dependencies we need to observe: register dependencies,
31 memory dependencies, dependencies to keep function calls in order,
32 and the dependence between a conditional branch and the setting of
33 condition codes are all dealt with here.
35 The scheduler first traverses the data flow graph, starting with
36 the last instruction, and proceeding to the first, assigning values
37 to insn_priority as it goes. This sorts the instructions
38 topologically by data dependence.
40 Once priorities have been established, we order the insns using
41 list scheduling. This works as follows: starting with a list of
42 all the ready insns, and sorted according to priority number, we
43 schedule the insn from the end of the list by placing its
44 predecessors in the list according to their priority order. We
45 consider this insn scheduled by setting the pointer to the "end" of
46 the list to point to the previous insn. When an insn has no
47 predecessors, we either queue it until sufficient time has elapsed
48 or add it to the ready list. As the instructions are scheduled or
49 when stalls are introduced, the queue advances and dumps insns into
50 the ready list. When all insns down to the lowest priority have
51 been scheduled, the critical path of the basic block has been made
52 as short as possible. The remaining insns are then scheduled in
53 remaining slots.
55 The following list shows the order in which we want to break ties
56 among insns in the ready list:
58 1. choose insn with the longest path to end of bb, ties
59 broken by
60 2. choose insn with least contribution to register pressure,
61 ties broken by
62 3. prefer in-block upon interblock motion, ties broken by
63 4. prefer useful upon speculative motion, ties broken by
64 5. choose insn with largest control flow probability, ties
65 broken by
66 6. choose insn with the least dependences upon the previously
67 scheduled insn, or finally
68 7 choose the insn which has the most insns dependent on it.
69 8. choose insn with lowest UID.
71 Memory references complicate matters. Only if we can be certain
72 that memory references are not part of the data dependency graph
73 (via true, anti, or output dependence), can we move operations past
74 memory references. To first approximation, reads can be done
75 independently, while writes introduce dependencies. Better
76 approximations will yield fewer dependencies.
78 Before reload, an extended analysis of interblock data dependences
79 is required for interblock scheduling. This is performed in
80 compute_block_dependences ().
82 Dependencies set up by memory references are treated in exactly the
83 same way as other dependencies, by using insn backward dependences
84 INSN_BACK_DEPS. INSN_BACK_DEPS are translated into forward dependences
85 INSN_FORW_DEPS for the purpose of forward list scheduling.
87 Having optimized the critical path, we may have also unduly
88 extended the lifetimes of some registers. If an operation requires
89 that constants be loaded into registers, it is certainly desirable
90 to load those constants as early as necessary, but no earlier.
91 I.e., it will not do to load up a bunch of registers at the
92 beginning of a basic block only to use them at the end, if they
93 could be loaded later, since this may result in excessive register
94 utilization.
96 Note that since branches are never in basic blocks, but only end
97 basic blocks, this pass will not move branches. But that is ok,
98 since we can use GNU's delayed branch scheduling pass to take care
99 of this case.
101 Also note that no further optimizations based on algebraic
102 identities are performed, so this pass would be a good one to
103 perform instruction splitting, such as breaking up a multiply
104 instruction into shifts and adds where that is profitable.
106 Given the memory aliasing analysis that this pass should perform,
107 it should be possible to remove redundant stores to memory, and to
108 load values from registers instead of hitting memory.
110 Before reload, speculative insns are moved only if a 'proof' exists
111 that no exception will be caused by this, and if no live registers
112 exist that inhibit the motion (live registers constraints are not
113 represented by data dependence edges).
115 This pass must update information that subsequent passes expect to
116 be correct. Namely: reg_n_refs, reg_n_sets, reg_n_deaths,
117 reg_n_calls_crossed, and reg_live_length. Also, BB_HEAD, BB_END.
119 The information in the line number notes is carefully retained by
120 this pass. Notes that refer to the starting and ending of
121 exception regions are also carefully retained by this pass. All
122 other NOTE insns are grouped in their same relative order at the
123 beginning of basic blocks and regions that have been scheduled. */
125 #include "config.h"
126 #include "system.h"
127 #include "coretypes.h"
128 #include "backend.h"
129 #include "target.h"
130 #include "rtl.h"
131 #include "cfghooks.h"
132 #include "df.h"
133 #include "memmodel.h"
134 #include "tm_p.h"
135 #include "insn-config.h"
136 #include "regs.h"
137 #include "ira.h"
138 #include "recog.h"
139 #include "insn-attr.h"
140 #include "cfgrtl.h"
141 #include "cfgbuild.h"
142 #include "sched-int.h"
143 #include "common/common-target.h"
144 #include "params.h"
145 #include "dbgcnt.h"
146 #include "cfgloop.h"
147 #include "dumpfile.h"
148 #include "print-rtl.h"
150 #ifdef INSN_SCHEDULING
152 /* True if we do register pressure relief through live-range
153 shrinkage. */
154 static bool live_range_shrinkage_p;
156 /* Switch on live range shrinkage. */
157 void
158 initialize_live_range_shrinkage (void)
160 live_range_shrinkage_p = true;
163 /* Switch off live range shrinkage. */
164 void
165 finish_live_range_shrinkage (void)
167 live_range_shrinkage_p = false;
170 /* issue_rate is the number of insns that can be scheduled in the same
171 machine cycle. It can be defined in the config/mach/mach.h file,
172 otherwise we set it to 1. */
174 int issue_rate;
176 /* This can be set to true by a backend if the scheduler should not
177 enable a DCE pass. */
178 bool sched_no_dce;
180 /* The current initiation interval used when modulo scheduling. */
181 static int modulo_ii;
183 /* The maximum number of stages we are prepared to handle. */
184 static int modulo_max_stages;
186 /* The number of insns that exist in each iteration of the loop. We use this
187 to detect when we've scheduled all insns from the first iteration. */
188 static int modulo_n_insns;
190 /* The current count of insns in the first iteration of the loop that have
191 already been scheduled. */
192 static int modulo_insns_scheduled;
194 /* The maximum uid of insns from the first iteration of the loop. */
195 static int modulo_iter0_max_uid;
197 /* The number of times we should attempt to backtrack when modulo scheduling.
198 Decreased each time we have to backtrack. */
199 static int modulo_backtracks_left;
201 /* The stage in which the last insn from the original loop was
202 scheduled. */
203 static int modulo_last_stage;
205 /* sched-verbose controls the amount of debugging output the
206 scheduler prints. It is controlled by -fsched-verbose=N:
207 N=0: no debugging output.
208 N=1: default value.
209 N=2: bb's probabilities, detailed ready list info, unit/insn info.
210 N=3: rtl at abort point, control-flow, regions info.
211 N=5: dependences info. */
212 int sched_verbose = 0;
214 /* Debugging file. All printouts are sent to dump. */
215 FILE *sched_dump = 0;
217 /* This is a placeholder for the scheduler parameters common
218 to all schedulers. */
219 struct common_sched_info_def *common_sched_info;
221 #define INSN_TICK(INSN) (HID (INSN)->tick)
222 #define INSN_EXACT_TICK(INSN) (HID (INSN)->exact_tick)
223 #define INSN_TICK_ESTIMATE(INSN) (HID (INSN)->tick_estimate)
224 #define INTER_TICK(INSN) (HID (INSN)->inter_tick)
225 #define FEEDS_BACKTRACK_INSN(INSN) (HID (INSN)->feeds_backtrack_insn)
226 #define SHADOW_P(INSN) (HID (INSN)->shadow_p)
227 #define MUST_RECOMPUTE_SPEC_P(INSN) (HID (INSN)->must_recompute_spec)
228 /* Cached cost of the instruction. Use insn_sched_cost to get cost of the
229 insn. -1 here means that the field is not initialized. */
230 #define INSN_COST(INSN) (HID (INSN)->cost)
232 /* If INSN_TICK of an instruction is equal to INVALID_TICK,
233 then it should be recalculated from scratch. */
234 #define INVALID_TICK (-(max_insn_queue_index + 1))
235 /* The minimal value of the INSN_TICK of an instruction. */
236 #define MIN_TICK (-max_insn_queue_index)
238 /* Original order of insns in the ready list.
239 Used to keep order of normal insns while separating DEBUG_INSNs. */
240 #define INSN_RFS_DEBUG_ORIG_ORDER(INSN) (HID (INSN)->rfs_debug_orig_order)
242 /* The deciding reason for INSN's place in the ready list. */
243 #define INSN_LAST_RFS_WIN(INSN) (HID (INSN)->last_rfs_win)
245 /* List of important notes we must keep around. This is a pointer to the
246 last element in the list. */
247 rtx_insn *note_list;
249 static struct spec_info_def spec_info_var;
250 /* Description of the speculative part of the scheduling.
251 If NULL - no speculation. */
252 spec_info_t spec_info = NULL;
254 /* True, if recovery block was added during scheduling of current block.
255 Used to determine, if we need to fix INSN_TICKs. */
256 static bool haifa_recovery_bb_recently_added_p;
258 /* True, if recovery block was added during this scheduling pass.
259 Used to determine if we should have empty memory pools of dependencies
260 after finishing current region. */
261 bool haifa_recovery_bb_ever_added_p;
263 /* Counters of different types of speculative instructions. */
264 static int nr_begin_data, nr_be_in_data, nr_begin_control, nr_be_in_control;
266 /* Array used in {unlink, restore}_bb_notes. */
267 static rtx_insn **bb_header = 0;
269 /* Basic block after which recovery blocks will be created. */
270 static basic_block before_recovery;
272 /* Basic block just before the EXIT_BLOCK and after recovery, if we have
273 created it. */
274 basic_block after_recovery;
276 /* FALSE if we add bb to another region, so we don't need to initialize it. */
277 bool adding_bb_to_current_region_p = true;
279 /* Queues, etc. */
281 /* An instruction is ready to be scheduled when all insns preceding it
282 have already been scheduled. It is important to ensure that all
283 insns which use its result will not be executed until its result
284 has been computed. An insn is maintained in one of four structures:
286 (P) the "Pending" set of insns which cannot be scheduled until
287 their dependencies have been satisfied.
288 (Q) the "Queued" set of insns that can be scheduled when sufficient
289 time has passed.
290 (R) the "Ready" list of unscheduled, uncommitted insns.
291 (S) the "Scheduled" list of insns.
293 Initially, all insns are either "Pending" or "Ready" depending on
294 whether their dependencies are satisfied.
296 Insns move from the "Ready" list to the "Scheduled" list as they
297 are committed to the schedule. As this occurs, the insns in the
298 "Pending" list have their dependencies satisfied and move to either
299 the "Ready" list or the "Queued" set depending on whether
300 sufficient time has passed to make them ready. As time passes,
301 insns move from the "Queued" set to the "Ready" list.
303 The "Pending" list (P) are the insns in the INSN_FORW_DEPS of the
304 unscheduled insns, i.e., those that are ready, queued, and pending.
305 The "Queued" set (Q) is implemented by the variable `insn_queue'.
306 The "Ready" list (R) is implemented by the variables `ready' and
307 `n_ready'.
308 The "Scheduled" list (S) is the new insn chain built by this pass.
310 The transition (R->S) is implemented in the scheduling loop in
311 `schedule_block' when the best insn to schedule is chosen.
312 The transitions (P->R and P->Q) are implemented in `schedule_insn' as
313 insns move from the ready list to the scheduled list.
314 The transition (Q->R) is implemented in 'queue_to_insn' as time
315 passes or stalls are introduced. */
317 /* Implement a circular buffer to delay instructions until sufficient
318 time has passed. For the new pipeline description interface,
319 MAX_INSN_QUEUE_INDEX is a power of two minus one which is not less
320 than maximal time of instruction execution computed by genattr.c on
321 the base maximal time of functional unit reservations and getting a
322 result. This is the longest time an insn may be queued. */
324 static rtx_insn_list **insn_queue;
325 static int q_ptr = 0;
326 static int q_size = 0;
327 #define NEXT_Q(X) (((X)+1) & max_insn_queue_index)
328 #define NEXT_Q_AFTER(X, C) (((X)+C) & max_insn_queue_index)
330 #define QUEUE_SCHEDULED (-3)
331 #define QUEUE_NOWHERE (-2)
332 #define QUEUE_READY (-1)
333 /* QUEUE_SCHEDULED - INSN is scheduled.
334 QUEUE_NOWHERE - INSN isn't scheduled yet and is neither in
335 queue or ready list.
336 QUEUE_READY - INSN is in ready list.
337 N >= 0 - INSN queued for X [where NEXT_Q_AFTER (q_ptr, X) == N] cycles. */
339 #define QUEUE_INDEX(INSN) (HID (INSN)->queue_index)
341 /* The following variable value refers for all current and future
342 reservations of the processor units. */
343 state_t curr_state;
345 /* The following variable value is size of memory representing all
346 current and future reservations of the processor units. */
347 size_t dfa_state_size;
349 /* The following array is used to find the best insn from ready when
350 the automaton pipeline interface is used. */
351 signed char *ready_try = NULL;
353 /* The ready list. */
354 struct ready_list ready = {NULL, 0, 0, 0, 0};
356 /* The pointer to the ready list (to be removed). */
357 static struct ready_list *readyp = &ready;
359 /* Scheduling clock. */
360 static int clock_var;
362 /* Clock at which the previous instruction was issued. */
363 static int last_clock_var;
365 /* Set to true if, when queuing a shadow insn, we discover that it would be
366 scheduled too late. */
367 static bool must_backtrack;
369 /* The following variable value is number of essential insns issued on
370 the current cycle. An insn is essential one if it changes the
371 processors state. */
372 int cycle_issued_insns;
374 /* This records the actual schedule. It is built up during the main phase
375 of schedule_block, and afterwards used to reorder the insns in the RTL. */
376 static vec<rtx_insn *> scheduled_insns;
378 static int may_trap_exp (const_rtx, int);
380 /* Nonzero iff the address is comprised from at most 1 register. */
381 #define CONST_BASED_ADDRESS_P(x) \
382 (REG_P (x) \
383 || ((GET_CODE (x) == PLUS || GET_CODE (x) == MINUS \
384 || (GET_CODE (x) == LO_SUM)) \
385 && (CONSTANT_P (XEXP (x, 0)) \
386 || CONSTANT_P (XEXP (x, 1)))))
388 /* Returns a class that insn with GET_DEST(insn)=x may belong to,
389 as found by analyzing insn's expression. */
392 static int haifa_luid_for_non_insn (rtx x);
394 /* Haifa version of sched_info hooks common to all headers. */
395 const struct common_sched_info_def haifa_common_sched_info =
397 NULL, /* fix_recovery_cfg */
398 NULL, /* add_block */
399 NULL, /* estimate_number_of_insns */
400 haifa_luid_for_non_insn, /* luid_for_non_insn */
401 SCHED_PASS_UNKNOWN /* sched_pass_id */
404 /* Mapping from instruction UID to its Logical UID. */
405 vec<int> sched_luids;
407 /* Next LUID to assign to an instruction. */
408 int sched_max_luid = 1;
410 /* Haifa Instruction Data. */
411 vec<haifa_insn_data_def> h_i_d;
413 void (* sched_init_only_bb) (basic_block, basic_block);
415 /* Split block function. Different schedulers might use different functions
416 to handle their internal data consistent. */
417 basic_block (* sched_split_block) (basic_block, rtx);
419 /* Create empty basic block after the specified block. */
420 basic_block (* sched_create_empty_bb) (basic_block);
422 /* Return the number of cycles until INSN is expected to be ready.
423 Return zero if it already is. */
424 static int
425 insn_delay (rtx_insn *insn)
427 return MAX (INSN_TICK (insn) - clock_var, 0);
430 static int
431 may_trap_exp (const_rtx x, int is_store)
433 enum rtx_code code;
435 if (x == 0)
436 return TRAP_FREE;
437 code = GET_CODE (x);
438 if (is_store)
440 if (code == MEM && may_trap_p (x))
441 return TRAP_RISKY;
442 else
443 return TRAP_FREE;
445 if (code == MEM)
447 /* The insn uses memory: a volatile load. */
448 if (MEM_VOLATILE_P (x))
449 return IRISKY;
450 /* An exception-free load. */
451 if (!may_trap_p (x))
452 return IFREE;
453 /* A load with 1 base register, to be further checked. */
454 if (CONST_BASED_ADDRESS_P (XEXP (x, 0)))
455 return PFREE_CANDIDATE;
456 /* No info on the load, to be further checked. */
457 return PRISKY_CANDIDATE;
459 else
461 const char *fmt;
462 int i, insn_class = TRAP_FREE;
464 /* Neither store nor load, check if it may cause a trap. */
465 if (may_trap_p (x))
466 return TRAP_RISKY;
467 /* Recursive step: walk the insn... */
468 fmt = GET_RTX_FORMAT (code);
469 for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
471 if (fmt[i] == 'e')
473 int tmp_class = may_trap_exp (XEXP (x, i), is_store);
474 insn_class = WORST_CLASS (insn_class, tmp_class);
476 else if (fmt[i] == 'E')
478 int j;
479 for (j = 0; j < XVECLEN (x, i); j++)
481 int tmp_class = may_trap_exp (XVECEXP (x, i, j), is_store);
482 insn_class = WORST_CLASS (insn_class, tmp_class);
483 if (insn_class == TRAP_RISKY || insn_class == IRISKY)
484 break;
487 if (insn_class == TRAP_RISKY || insn_class == IRISKY)
488 break;
490 return insn_class;
494 /* Classifies rtx X of an insn for the purpose of verifying that X can be
495 executed speculatively (and consequently the insn can be moved
496 speculatively), by examining X, returning:
497 TRAP_RISKY: store, or risky non-load insn (e.g. division by variable).
498 TRAP_FREE: non-load insn.
499 IFREE: load from a globally safe location.
500 IRISKY: volatile load.
501 PFREE_CANDIDATE, PRISKY_CANDIDATE: load that need to be checked for
502 being either PFREE or PRISKY. */
504 static int
505 haifa_classify_rtx (const_rtx x)
507 int tmp_class = TRAP_FREE;
508 int insn_class = TRAP_FREE;
509 enum rtx_code code;
511 if (GET_CODE (x) == PARALLEL)
513 int i, len = XVECLEN (x, 0);
515 for (i = len - 1; i >= 0; i--)
517 tmp_class = haifa_classify_rtx (XVECEXP (x, 0, i));
518 insn_class = WORST_CLASS (insn_class, tmp_class);
519 if (insn_class == TRAP_RISKY || insn_class == IRISKY)
520 break;
523 else
525 code = GET_CODE (x);
526 switch (code)
528 case CLOBBER:
529 /* Test if it is a 'store'. */
530 tmp_class = may_trap_exp (XEXP (x, 0), 1);
531 break;
532 case SET:
533 /* Test if it is a store. */
534 tmp_class = may_trap_exp (SET_DEST (x), 1);
535 if (tmp_class == TRAP_RISKY)
536 break;
537 /* Test if it is a load. */
538 tmp_class =
539 WORST_CLASS (tmp_class,
540 may_trap_exp (SET_SRC (x), 0));
541 break;
542 case COND_EXEC:
543 tmp_class = haifa_classify_rtx (COND_EXEC_CODE (x));
544 if (tmp_class == TRAP_RISKY)
545 break;
546 tmp_class = WORST_CLASS (tmp_class,
547 may_trap_exp (COND_EXEC_TEST (x), 0));
548 break;
549 case TRAP_IF:
550 tmp_class = TRAP_RISKY;
551 break;
552 default:;
554 insn_class = tmp_class;
557 return insn_class;
561 haifa_classify_insn (const_rtx insn)
563 return haifa_classify_rtx (PATTERN (insn));
566 /* After the scheduler initialization function has been called, this function
567 can be called to enable modulo scheduling. II is the initiation interval
568 we should use, it affects the delays for delay_pairs that were recorded as
569 separated by a given number of stages.
571 MAX_STAGES provides us with a limit
572 after which we give up scheduling; the caller must have unrolled at least
573 as many copies of the loop body and recorded delay_pairs for them.
575 INSNS is the number of real (non-debug) insns in one iteration of
576 the loop. MAX_UID can be used to test whether an insn belongs to
577 the first iteration of the loop; all of them have a uid lower than
578 MAX_UID. */
579 void
580 set_modulo_params (int ii, int max_stages, int insns, int max_uid)
582 modulo_ii = ii;
583 modulo_max_stages = max_stages;
584 modulo_n_insns = insns;
585 modulo_iter0_max_uid = max_uid;
586 modulo_backtracks_left = PARAM_VALUE (PARAM_MAX_MODULO_BACKTRACK_ATTEMPTS);
589 /* A structure to record a pair of insns where the first one is a real
590 insn that has delay slots, and the second is its delayed shadow.
591 I1 is scheduled normally and will emit an assembly instruction,
592 while I2 describes the side effect that takes place at the
593 transition between cycles CYCLES and (CYCLES + 1) after I1. */
594 struct delay_pair
596 struct delay_pair *next_same_i1;
597 rtx_insn *i1, *i2;
598 int cycles;
599 /* When doing modulo scheduling, we a delay_pair can also be used to
600 show that I1 and I2 are the same insn in a different stage. If that
601 is the case, STAGES will be nonzero. */
602 int stages;
605 /* Helpers for delay hashing. */
607 struct delay_i1_hasher : nofree_ptr_hash <delay_pair>
609 typedef void *compare_type;
610 static inline hashval_t hash (const delay_pair *);
611 static inline bool equal (const delay_pair *, const void *);
614 /* Returns a hash value for X, based on hashing just I1. */
616 inline hashval_t
617 delay_i1_hasher::hash (const delay_pair *x)
619 return htab_hash_pointer (x->i1);
622 /* Return true if I1 of pair X is the same as that of pair Y. */
624 inline bool
625 delay_i1_hasher::equal (const delay_pair *x, const void *y)
627 return x->i1 == y;
630 struct delay_i2_hasher : free_ptr_hash <delay_pair>
632 typedef void *compare_type;
633 static inline hashval_t hash (const delay_pair *);
634 static inline bool equal (const delay_pair *, const void *);
637 /* Returns a hash value for X, based on hashing just I2. */
639 inline hashval_t
640 delay_i2_hasher::hash (const delay_pair *x)
642 return htab_hash_pointer (x->i2);
645 /* Return true if I2 of pair X is the same as that of pair Y. */
647 inline bool
648 delay_i2_hasher::equal (const delay_pair *x, const void *y)
650 return x->i2 == y;
653 /* Two hash tables to record delay_pairs, one indexed by I1 and the other
654 indexed by I2. */
655 static hash_table<delay_i1_hasher> *delay_htab;
656 static hash_table<delay_i2_hasher> *delay_htab_i2;
658 /* Called through htab_traverse. Walk the hashtable using I2 as
659 index, and delete all elements involving an UID higher than
660 that pointed to by *DATA. */
662 haifa_htab_i2_traverse (delay_pair **slot, int *data)
664 int maxuid = *data;
665 struct delay_pair *p = *slot;
666 if (INSN_UID (p->i2) >= maxuid || INSN_UID (p->i1) >= maxuid)
668 delay_htab_i2->clear_slot (slot);
670 return 1;
673 /* Called through htab_traverse. Walk the hashtable using I2 as
674 index, and delete all elements involving an UID higher than
675 that pointed to by *DATA. */
677 haifa_htab_i1_traverse (delay_pair **pslot, int *data)
679 int maxuid = *data;
680 struct delay_pair *p, *first, **pprev;
682 if (INSN_UID ((*pslot)->i1) >= maxuid)
684 delay_htab->clear_slot (pslot);
685 return 1;
687 pprev = &first;
688 for (p = *pslot; p; p = p->next_same_i1)
690 if (INSN_UID (p->i2) < maxuid)
692 *pprev = p;
693 pprev = &p->next_same_i1;
696 *pprev = NULL;
697 if (first == NULL)
698 delay_htab->clear_slot (pslot);
699 else
700 *pslot = first;
701 return 1;
704 /* Discard all delay pairs which involve an insn with an UID higher
705 than MAX_UID. */
706 void
707 discard_delay_pairs_above (int max_uid)
709 delay_htab->traverse <int *, haifa_htab_i1_traverse> (&max_uid);
710 delay_htab_i2->traverse <int *, haifa_htab_i2_traverse> (&max_uid);
713 /* This function can be called by a port just before it starts the final
714 scheduling pass. It records the fact that an instruction with delay
715 slots has been split into two insns, I1 and I2. The first one will be
716 scheduled normally and initiates the operation. The second one is a
717 shadow which must follow a specific number of cycles after I1; its only
718 purpose is to show the side effect that occurs at that cycle in the RTL.
719 If a JUMP_INSN or a CALL_INSN has been split, I1 should be a normal INSN,
720 while I2 retains the original insn type.
722 There are two ways in which the number of cycles can be specified,
723 involving the CYCLES and STAGES arguments to this function. If STAGES
724 is zero, we just use the value of CYCLES. Otherwise, STAGES is a factor
725 which is multiplied by MODULO_II to give the number of cycles. This is
726 only useful if the caller also calls set_modulo_params to enable modulo
727 scheduling. */
729 void
730 record_delay_slot_pair (rtx_insn *i1, rtx_insn *i2, int cycles, int stages)
732 struct delay_pair *p = XNEW (struct delay_pair);
733 struct delay_pair **slot;
735 p->i1 = i1;
736 p->i2 = i2;
737 p->cycles = cycles;
738 p->stages = stages;
740 if (!delay_htab)
742 delay_htab = new hash_table<delay_i1_hasher> (10);
743 delay_htab_i2 = new hash_table<delay_i2_hasher> (10);
745 slot = delay_htab->find_slot_with_hash (i1, htab_hash_pointer (i1), INSERT);
746 p->next_same_i1 = *slot;
747 *slot = p;
748 slot = delay_htab_i2->find_slot (p, INSERT);
749 *slot = p;
752 /* Examine the delay pair hashtable to see if INSN is a shadow for another,
753 and return the other insn if so. Return NULL otherwise. */
754 rtx_insn *
755 real_insn_for_shadow (rtx_insn *insn)
757 struct delay_pair *pair;
759 if (!delay_htab)
760 return NULL;
762 pair = delay_htab_i2->find_with_hash (insn, htab_hash_pointer (insn));
763 if (!pair || pair->stages > 0)
764 return NULL;
765 return pair->i1;
768 /* For a pair P of insns, return the fixed distance in cycles from the first
769 insn after which the second must be scheduled. */
770 static int
771 pair_delay (struct delay_pair *p)
773 if (p->stages == 0)
774 return p->cycles;
775 else
776 return p->stages * modulo_ii;
779 /* Given an insn INSN, add a dependence on its delayed shadow if it
780 has one. Also try to find situations where shadows depend on each other
781 and add dependencies to the real insns to limit the amount of backtracking
782 needed. */
783 void
784 add_delay_dependencies (rtx_insn *insn)
786 struct delay_pair *pair;
787 sd_iterator_def sd_it;
788 dep_t dep;
790 if (!delay_htab)
791 return;
793 pair = delay_htab_i2->find_with_hash (insn, htab_hash_pointer (insn));
794 if (!pair)
795 return;
796 add_dependence (insn, pair->i1, REG_DEP_ANTI);
797 if (pair->stages)
798 return;
800 FOR_EACH_DEP (pair->i2, SD_LIST_BACK, sd_it, dep)
802 rtx_insn *pro = DEP_PRO (dep);
803 struct delay_pair *other_pair
804 = delay_htab_i2->find_with_hash (pro, htab_hash_pointer (pro));
805 if (!other_pair || other_pair->stages)
806 continue;
807 if (pair_delay (other_pair) >= pair_delay (pair))
809 if (sched_verbose >= 4)
811 fprintf (sched_dump, ";;\tadding dependence %d <- %d\n",
812 INSN_UID (other_pair->i1),
813 INSN_UID (pair->i1));
814 fprintf (sched_dump, ";;\tpair1 %d <- %d, cost %d\n",
815 INSN_UID (pair->i1),
816 INSN_UID (pair->i2),
817 pair_delay (pair));
818 fprintf (sched_dump, ";;\tpair2 %d <- %d, cost %d\n",
819 INSN_UID (other_pair->i1),
820 INSN_UID (other_pair->i2),
821 pair_delay (other_pair));
823 add_dependence (pair->i1, other_pair->i1, REG_DEP_ANTI);
828 /* Forward declarations. */
830 static int priority (rtx_insn *);
831 static int autopref_rank_for_schedule (const rtx_insn *, const rtx_insn *);
832 static int rank_for_schedule (const void *, const void *);
833 static void swap_sort (rtx_insn **, int);
834 static void queue_insn (rtx_insn *, int, const char *);
835 static int schedule_insn (rtx_insn *);
836 static void adjust_priority (rtx_insn *);
837 static void advance_one_cycle (void);
838 static void extend_h_i_d (void);
841 /* Notes handling mechanism:
842 =========================
843 Generally, NOTES are saved before scheduling and restored after scheduling.
844 The scheduler distinguishes between two types of notes:
846 (1) LOOP_BEGIN, LOOP_END, SETJMP, EHREGION_BEG, EHREGION_END notes:
847 Before scheduling a region, a pointer to the note is added to the insn
848 that follows or precedes it. (This happens as part of the data dependence
849 computation). After scheduling an insn, the pointer contained in it is
850 used for regenerating the corresponding note (in reemit_notes).
852 (2) All other notes (e.g. INSN_DELETED): Before scheduling a block,
853 these notes are put in a list (in rm_other_notes() and
854 unlink_other_notes ()). After scheduling the block, these notes are
855 inserted at the beginning of the block (in schedule_block()). */
857 static void ready_add (struct ready_list *, rtx_insn *, bool);
858 static rtx_insn *ready_remove_first (struct ready_list *);
859 static rtx_insn *ready_remove_first_dispatch (struct ready_list *ready);
861 static void queue_to_ready (struct ready_list *);
862 static int early_queue_to_ready (state_t, struct ready_list *);
864 /* The following functions are used to implement multi-pass scheduling
865 on the first cycle. */
866 static rtx_insn *ready_remove (struct ready_list *, int);
867 static void ready_remove_insn (rtx_insn *);
869 static void fix_inter_tick (rtx_insn *, rtx_insn *);
870 static int fix_tick_ready (rtx_insn *);
871 static void change_queue_index (rtx_insn *, int);
873 /* The following functions are used to implement scheduling of data/control
874 speculative instructions. */
876 static void extend_h_i_d (void);
877 static void init_h_i_d (rtx_insn *);
878 static int haifa_speculate_insn (rtx_insn *, ds_t, rtx *);
879 static void generate_recovery_code (rtx_insn *);
880 static void process_insn_forw_deps_be_in_spec (rtx_insn *, rtx_insn *, ds_t);
881 static void begin_speculative_block (rtx_insn *);
882 static void add_to_speculative_block (rtx_insn *);
883 static void init_before_recovery (basic_block *);
884 static void create_check_block_twin (rtx_insn *, bool);
885 static void fix_recovery_deps (basic_block);
886 static bool haifa_change_pattern (rtx_insn *, rtx);
887 static void dump_new_block_header (int, basic_block, rtx_insn *, rtx_insn *);
888 static void restore_bb_notes (basic_block);
889 static void fix_jump_move (rtx_insn *);
890 static void move_block_after_check (rtx_insn *);
891 static void move_succs (vec<edge, va_gc> **, basic_block);
892 static void sched_remove_insn (rtx_insn *);
893 static void clear_priorities (rtx_insn *, rtx_vec_t *);
894 static void calc_priorities (rtx_vec_t);
895 static void add_jump_dependencies (rtx_insn *, rtx_insn *);
897 #endif /* INSN_SCHEDULING */
899 /* Point to state used for the current scheduling pass. */
900 struct haifa_sched_info *current_sched_info;
902 #ifndef INSN_SCHEDULING
903 void
904 schedule_insns (void)
907 #else
909 /* Do register pressure sensitive insn scheduling if the flag is set
910 up. */
911 enum sched_pressure_algorithm sched_pressure;
913 /* Map regno -> its pressure class. The map defined only when
914 SCHED_PRESSURE != SCHED_PRESSURE_NONE. */
915 enum reg_class *sched_regno_pressure_class;
917 /* The current register pressure. Only elements corresponding pressure
918 classes are defined. */
919 static int curr_reg_pressure[N_REG_CLASSES];
921 /* Saved value of the previous array. */
922 static int saved_reg_pressure[N_REG_CLASSES];
924 /* Register living at given scheduling point. */
925 static bitmap curr_reg_live;
927 /* Saved value of the previous array. */
928 static bitmap saved_reg_live;
930 /* Registers mentioned in the current region. */
931 static bitmap region_ref_regs;
933 /* Temporary bitmap used for SCHED_PRESSURE_MODEL. */
934 static bitmap tmp_bitmap;
936 /* Effective number of available registers of a given class (see comment
937 in sched_pressure_start_bb). */
938 static int sched_class_regs_num[N_REG_CLASSES];
939 /* Number of call_saved_regs and fixed_regs. Helpers for calculating of
940 sched_class_regs_num. */
941 static int call_saved_regs_num[N_REG_CLASSES];
942 static int fixed_regs_num[N_REG_CLASSES];
944 /* Initiate register pressure relative info for scheduling the current
945 region. Currently it is only clearing register mentioned in the
946 current region. */
947 void
948 sched_init_region_reg_pressure_info (void)
950 bitmap_clear (region_ref_regs);
953 /* PRESSURE[CL] describes the pressure on register class CL. Update it
954 for the birth (if BIRTH_P) or death (if !BIRTH_P) of register REGNO.
955 LIVE tracks the set of live registers; if it is null, assume that
956 every birth or death is genuine. */
957 static inline void
958 mark_regno_birth_or_death (bitmap live, int *pressure, int regno, bool birth_p)
960 enum reg_class pressure_class;
962 pressure_class = sched_regno_pressure_class[regno];
963 if (regno >= FIRST_PSEUDO_REGISTER)
965 if (pressure_class != NO_REGS)
967 if (birth_p)
969 if (!live || bitmap_set_bit (live, regno))
970 pressure[pressure_class]
971 += (ira_reg_class_max_nregs
972 [pressure_class][PSEUDO_REGNO_MODE (regno)]);
974 else
976 if (!live || bitmap_clear_bit (live, regno))
977 pressure[pressure_class]
978 -= (ira_reg_class_max_nregs
979 [pressure_class][PSEUDO_REGNO_MODE (regno)]);
983 else if (pressure_class != NO_REGS
984 && ! TEST_HARD_REG_BIT (ira_no_alloc_regs, regno))
986 if (birth_p)
988 if (!live || bitmap_set_bit (live, regno))
989 pressure[pressure_class]++;
991 else
993 if (!live || bitmap_clear_bit (live, regno))
994 pressure[pressure_class]--;
999 /* Initiate current register pressure related info from living
1000 registers given by LIVE. */
1001 static void
1002 initiate_reg_pressure_info (bitmap live)
1004 int i;
1005 unsigned int j;
1006 bitmap_iterator bi;
1008 for (i = 0; i < ira_pressure_classes_num; i++)
1009 curr_reg_pressure[ira_pressure_classes[i]] = 0;
1010 bitmap_clear (curr_reg_live);
1011 EXECUTE_IF_SET_IN_BITMAP (live, 0, j, bi)
1012 if (sched_pressure == SCHED_PRESSURE_MODEL
1013 || current_nr_blocks == 1
1014 || bitmap_bit_p (region_ref_regs, j))
1015 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure, j, true);
1018 /* Mark registers in X as mentioned in the current region. */
1019 static void
1020 setup_ref_regs (rtx x)
1022 int i, j;
1023 const RTX_CODE code = GET_CODE (x);
1024 const char *fmt;
1026 if (REG_P (x))
1028 bitmap_set_range (region_ref_regs, REGNO (x), REG_NREGS (x));
1029 return;
1031 fmt = GET_RTX_FORMAT (code);
1032 for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
1033 if (fmt[i] == 'e')
1034 setup_ref_regs (XEXP (x, i));
1035 else if (fmt[i] == 'E')
1037 for (j = 0; j < XVECLEN (x, i); j++)
1038 setup_ref_regs (XVECEXP (x, i, j));
1042 /* Initiate current register pressure related info at the start of
1043 basic block BB. */
1044 static void
1045 initiate_bb_reg_pressure_info (basic_block bb)
1047 unsigned int i ATTRIBUTE_UNUSED;
1048 rtx_insn *insn;
1050 if (current_nr_blocks > 1)
1051 FOR_BB_INSNS (bb, insn)
1052 if (NONDEBUG_INSN_P (insn))
1053 setup_ref_regs (PATTERN (insn));
1054 initiate_reg_pressure_info (df_get_live_in (bb));
1055 if (bb_has_eh_pred (bb))
1056 for (i = 0; ; ++i)
1058 unsigned int regno = EH_RETURN_DATA_REGNO (i);
1060 if (regno == INVALID_REGNUM)
1061 break;
1062 if (! bitmap_bit_p (df_get_live_in (bb), regno))
1063 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure,
1064 regno, true);
1068 /* Save current register pressure related info. */
1069 static void
1070 save_reg_pressure (void)
1072 int i;
1074 for (i = 0; i < ira_pressure_classes_num; i++)
1075 saved_reg_pressure[ira_pressure_classes[i]]
1076 = curr_reg_pressure[ira_pressure_classes[i]];
1077 bitmap_copy (saved_reg_live, curr_reg_live);
1080 /* Restore saved register pressure related info. */
1081 static void
1082 restore_reg_pressure (void)
1084 int i;
1086 for (i = 0; i < ira_pressure_classes_num; i++)
1087 curr_reg_pressure[ira_pressure_classes[i]]
1088 = saved_reg_pressure[ira_pressure_classes[i]];
1089 bitmap_copy (curr_reg_live, saved_reg_live);
1092 /* Return TRUE if the register is dying after its USE. */
1093 static bool
1094 dying_use_p (struct reg_use_data *use)
1096 struct reg_use_data *next;
1098 for (next = use->next_regno_use; next != use; next = next->next_regno_use)
1099 if (NONDEBUG_INSN_P (next->insn)
1100 && QUEUE_INDEX (next->insn) != QUEUE_SCHEDULED)
1101 return false;
1102 return true;
1105 /* Print info about the current register pressure and its excess for
1106 each pressure class. */
1107 static void
1108 print_curr_reg_pressure (void)
1110 int i;
1111 enum reg_class cl;
1113 fprintf (sched_dump, ";;\t");
1114 for (i = 0; i < ira_pressure_classes_num; i++)
1116 cl = ira_pressure_classes[i];
1117 gcc_assert (curr_reg_pressure[cl] >= 0);
1118 fprintf (sched_dump, " %s:%d(%d)", reg_class_names[cl],
1119 curr_reg_pressure[cl],
1120 curr_reg_pressure[cl] - sched_class_regs_num[cl]);
1122 fprintf (sched_dump, "\n");
1125 /* Determine if INSN has a condition that is clobbered if a register
1126 in SET_REGS is modified. */
1127 static bool
1128 cond_clobbered_p (rtx_insn *insn, HARD_REG_SET set_regs)
1130 rtx pat = PATTERN (insn);
1131 gcc_assert (GET_CODE (pat) == COND_EXEC);
1132 if (TEST_HARD_REG_BIT (set_regs, REGNO (XEXP (COND_EXEC_TEST (pat), 0))))
1134 sd_iterator_def sd_it;
1135 dep_t dep;
1136 haifa_change_pattern (insn, ORIG_PAT (insn));
1137 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep)
1138 DEP_STATUS (dep) &= ~DEP_CANCELLED;
1139 TODO_SPEC (insn) = HARD_DEP;
1140 if (sched_verbose >= 2)
1141 fprintf (sched_dump,
1142 ";;\t\tdequeue insn %s because of clobbered condition\n",
1143 (*current_sched_info->print_insn) (insn, 0));
1144 return true;
1147 return false;
1150 /* This function should be called after modifying the pattern of INSN,
1151 to update scheduler data structures as needed. */
1152 static void
1153 update_insn_after_change (rtx_insn *insn)
1155 sd_iterator_def sd_it;
1156 dep_t dep;
1158 dfa_clear_single_insn_cache (insn);
1160 sd_it = sd_iterator_start (insn,
1161 SD_LIST_FORW | SD_LIST_BACK | SD_LIST_RES_BACK);
1162 while (sd_iterator_cond (&sd_it, &dep))
1164 DEP_COST (dep) = UNKNOWN_DEP_COST;
1165 sd_iterator_next (&sd_it);
1168 /* Invalidate INSN_COST, so it'll be recalculated. */
1169 INSN_COST (insn) = -1;
1170 /* Invalidate INSN_TICK, so it'll be recalculated. */
1171 INSN_TICK (insn) = INVALID_TICK;
1173 /* Invalidate autoprefetch data entry. */
1174 INSN_AUTOPREF_MULTIPASS_DATA (insn)[0].status
1175 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED;
1176 INSN_AUTOPREF_MULTIPASS_DATA (insn)[1].status
1177 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED;
1181 /* Two VECs, one to hold dependencies for which pattern replacements
1182 need to be applied or restored at the start of the next cycle, and
1183 another to hold an integer that is either one, to apply the
1184 corresponding replacement, or zero to restore it. */
1185 static vec<dep_t> next_cycle_replace_deps;
1186 static vec<int> next_cycle_apply;
1188 static void apply_replacement (dep_t, bool);
1189 static void restore_pattern (dep_t, bool);
1191 /* Look at the remaining dependencies for insn NEXT, and compute and return
1192 the TODO_SPEC value we should use for it. This is called after one of
1193 NEXT's dependencies has been resolved.
1194 We also perform pattern replacements for predication, and for broken
1195 replacement dependencies. The latter is only done if FOR_BACKTRACK is
1196 false. */
1198 static ds_t
1199 recompute_todo_spec (rtx_insn *next, bool for_backtrack)
1201 ds_t new_ds;
1202 sd_iterator_def sd_it;
1203 dep_t dep, modify_dep = NULL;
1204 int n_spec = 0;
1205 int n_control = 0;
1206 int n_replace = 0;
1207 bool first_p = true;
1209 if (sd_lists_empty_p (next, SD_LIST_BACK))
1210 /* NEXT has all its dependencies resolved. */
1211 return 0;
1213 if (!sd_lists_empty_p (next, SD_LIST_HARD_BACK))
1214 return HARD_DEP;
1216 /* If NEXT is intended to sit adjacent to this instruction, we don't
1217 want to try to break any dependencies. Treat it as a HARD_DEP. */
1218 if (SCHED_GROUP_P (next))
1219 return HARD_DEP;
1221 /* Now we've got NEXT with speculative deps only.
1222 1. Look at the deps to see what we have to do.
1223 2. Check if we can do 'todo'. */
1224 new_ds = 0;
1226 FOR_EACH_DEP (next, SD_LIST_BACK, sd_it, dep)
1228 rtx_insn *pro = DEP_PRO (dep);
1229 ds_t ds = DEP_STATUS (dep) & SPECULATIVE;
1231 if (DEBUG_INSN_P (pro) && !DEBUG_INSN_P (next))
1232 continue;
1234 if (ds)
1236 n_spec++;
1237 if (first_p)
1239 first_p = false;
1241 new_ds = ds;
1243 else
1244 new_ds = ds_merge (new_ds, ds);
1246 else if (DEP_TYPE (dep) == REG_DEP_CONTROL)
1248 if (QUEUE_INDEX (pro) != QUEUE_SCHEDULED)
1250 n_control++;
1251 modify_dep = dep;
1253 DEP_STATUS (dep) &= ~DEP_CANCELLED;
1255 else if (DEP_REPLACE (dep) != NULL)
1257 if (QUEUE_INDEX (pro) != QUEUE_SCHEDULED)
1259 n_replace++;
1260 modify_dep = dep;
1262 DEP_STATUS (dep) &= ~DEP_CANCELLED;
1266 if (n_replace > 0 && n_control == 0 && n_spec == 0)
1268 if (!dbg_cnt (sched_breakdep))
1269 return HARD_DEP;
1270 FOR_EACH_DEP (next, SD_LIST_BACK, sd_it, dep)
1272 struct dep_replacement *desc = DEP_REPLACE (dep);
1273 if (desc != NULL)
1275 if (desc->insn == next && !for_backtrack)
1277 gcc_assert (n_replace == 1);
1278 apply_replacement (dep, true);
1280 DEP_STATUS (dep) |= DEP_CANCELLED;
1283 return 0;
1286 else if (n_control == 1 && n_replace == 0 && n_spec == 0)
1288 rtx_insn *pro, *other;
1289 rtx new_pat;
1290 rtx cond = NULL_RTX;
1291 bool success;
1292 rtx_insn *prev = NULL;
1293 int i;
1294 unsigned regno;
1296 if ((current_sched_info->flags & DO_PREDICATION) == 0
1297 || (ORIG_PAT (next) != NULL_RTX
1298 && PREDICATED_PAT (next) == NULL_RTX))
1299 return HARD_DEP;
1301 pro = DEP_PRO (modify_dep);
1302 other = real_insn_for_shadow (pro);
1303 if (other != NULL_RTX)
1304 pro = other;
1306 cond = sched_get_reverse_condition_uncached (pro);
1307 regno = REGNO (XEXP (cond, 0));
1309 /* Find the last scheduled insn that modifies the condition register.
1310 We can stop looking once we find the insn we depend on through the
1311 REG_DEP_CONTROL; if the condition register isn't modified after it,
1312 we know that it still has the right value. */
1313 if (QUEUE_INDEX (pro) == QUEUE_SCHEDULED)
1314 FOR_EACH_VEC_ELT_REVERSE (scheduled_insns, i, prev)
1316 HARD_REG_SET t;
1318 find_all_hard_reg_sets (prev, &t, true);
1319 if (TEST_HARD_REG_BIT (t, regno))
1320 return HARD_DEP;
1321 if (prev == pro)
1322 break;
1324 if (ORIG_PAT (next) == NULL_RTX)
1326 ORIG_PAT (next) = PATTERN (next);
1328 new_pat = gen_rtx_COND_EXEC (VOIDmode, cond, PATTERN (next));
1329 success = haifa_change_pattern (next, new_pat);
1330 if (!success)
1331 return HARD_DEP;
1332 PREDICATED_PAT (next) = new_pat;
1334 else if (PATTERN (next) != PREDICATED_PAT (next))
1336 bool success = haifa_change_pattern (next,
1337 PREDICATED_PAT (next));
1338 gcc_assert (success);
1340 DEP_STATUS (modify_dep) |= DEP_CANCELLED;
1341 return DEP_CONTROL;
1344 if (PREDICATED_PAT (next) != NULL_RTX)
1346 int tick = INSN_TICK (next);
1347 bool success = haifa_change_pattern (next,
1348 ORIG_PAT (next));
1349 INSN_TICK (next) = tick;
1350 gcc_assert (success);
1353 /* We can't handle the case where there are both speculative and control
1354 dependencies, so we return HARD_DEP in such a case. Also fail if
1355 we have speculative dependencies with not enough points, or more than
1356 one control dependency. */
1357 if ((n_spec > 0 && (n_control > 0 || n_replace > 0))
1358 || (n_spec > 0
1359 /* Too few points? */
1360 && ds_weak (new_ds) < spec_info->data_weakness_cutoff)
1361 || n_control > 0
1362 || n_replace > 0)
1363 return HARD_DEP;
1365 return new_ds;
1368 /* Pointer to the last instruction scheduled. */
1369 static rtx_insn *last_scheduled_insn;
1371 /* Pointer to the last nondebug instruction scheduled within the
1372 block, or the prev_head of the scheduling block. Used by
1373 rank_for_schedule, so that insns independent of the last scheduled
1374 insn will be preferred over dependent instructions. */
1375 static rtx_insn *last_nondebug_scheduled_insn;
1377 /* Pointer that iterates through the list of unscheduled insns if we
1378 have a dbg_cnt enabled. It always points at an insn prior to the
1379 first unscheduled one. */
1380 static rtx_insn *nonscheduled_insns_begin;
1382 /* Compute cost of executing INSN.
1383 This is the number of cycles between instruction issue and
1384 instruction results. */
1386 insn_sched_cost (rtx_insn *insn)
1388 int cost;
1390 if (sched_fusion)
1391 return 0;
1393 if (sel_sched_p ())
1395 if (recog_memoized (insn) < 0)
1396 return 0;
1398 cost = insn_default_latency (insn);
1399 if (cost < 0)
1400 cost = 0;
1402 return cost;
1405 cost = INSN_COST (insn);
1407 if (cost < 0)
1409 /* A USE insn, or something else we don't need to
1410 understand. We can't pass these directly to
1411 result_ready_cost or insn_default_latency because it will
1412 trigger a fatal error for unrecognizable insns. */
1413 if (recog_memoized (insn) < 0)
1415 INSN_COST (insn) = 0;
1416 return 0;
1418 else
1420 cost = insn_default_latency (insn);
1421 if (cost < 0)
1422 cost = 0;
1424 INSN_COST (insn) = cost;
1428 return cost;
1431 /* Compute cost of dependence LINK.
1432 This is the number of cycles between instruction issue and
1433 instruction results.
1434 ??? We also use this function to call recog_memoized on all insns. */
1436 dep_cost_1 (dep_t link, dw_t dw)
1438 rtx_insn *insn = DEP_PRO (link);
1439 rtx_insn *used = DEP_CON (link);
1440 int cost;
1442 if (DEP_COST (link) != UNKNOWN_DEP_COST)
1443 return DEP_COST (link);
1445 if (delay_htab)
1447 struct delay_pair *delay_entry;
1448 delay_entry
1449 = delay_htab_i2->find_with_hash (used, htab_hash_pointer (used));
1450 if (delay_entry)
1452 if (delay_entry->i1 == insn)
1454 DEP_COST (link) = pair_delay (delay_entry);
1455 return DEP_COST (link);
1460 /* A USE insn should never require the value used to be computed.
1461 This allows the computation of a function's result and parameter
1462 values to overlap the return and call. We don't care about the
1463 dependence cost when only decreasing register pressure. */
1464 if (recog_memoized (used) < 0)
1466 cost = 0;
1467 recog_memoized (insn);
1469 else
1471 enum reg_note dep_type = DEP_TYPE (link);
1473 cost = insn_sched_cost (insn);
1475 if (INSN_CODE (insn) >= 0)
1477 if (dep_type == REG_DEP_ANTI)
1478 cost = 0;
1479 else if (dep_type == REG_DEP_OUTPUT)
1481 cost = (insn_default_latency (insn)
1482 - insn_default_latency (used));
1483 if (cost <= 0)
1484 cost = 1;
1486 else if (bypass_p (insn))
1487 cost = insn_latency (insn, used);
1491 if (targetm.sched.adjust_cost)
1492 cost = targetm.sched.adjust_cost (used, (int) dep_type, insn, cost,
1493 dw);
1495 if (cost < 0)
1496 cost = 0;
1499 DEP_COST (link) = cost;
1500 return cost;
1503 /* Compute cost of dependence LINK.
1504 This is the number of cycles between instruction issue and
1505 instruction results. */
1507 dep_cost (dep_t link)
1509 return dep_cost_1 (link, 0);
1512 /* Use this sel-sched.c friendly function in reorder2 instead of increasing
1513 INSN_PRIORITY explicitly. */
1514 void
1515 increase_insn_priority (rtx_insn *insn, int amount)
1517 if (!sel_sched_p ())
1519 /* We're dealing with haifa-sched.c INSN_PRIORITY. */
1520 if (INSN_PRIORITY_KNOWN (insn))
1521 INSN_PRIORITY (insn) += amount;
1523 else
1525 /* In sel-sched.c INSN_PRIORITY is not kept up to date.
1526 Use EXPR_PRIORITY instead. */
1527 sel_add_to_insn_priority (insn, amount);
1531 /* Return 'true' if DEP should be included in priority calculations. */
1532 static bool
1533 contributes_to_priority_p (dep_t dep)
1535 if (DEBUG_INSN_P (DEP_CON (dep))
1536 || DEBUG_INSN_P (DEP_PRO (dep)))
1537 return false;
1539 /* Critical path is meaningful in block boundaries only. */
1540 if (!current_sched_info->contributes_to_priority (DEP_CON (dep),
1541 DEP_PRO (dep)))
1542 return false;
1544 if (DEP_REPLACE (dep) != NULL)
1545 return false;
1547 /* If flag COUNT_SPEC_IN_CRITICAL_PATH is set,
1548 then speculative instructions will less likely be
1549 scheduled. That is because the priority of
1550 their producers will increase, and, thus, the
1551 producers will more likely be scheduled, thus,
1552 resolving the dependence. */
1553 if (sched_deps_info->generate_spec_deps
1554 && !(spec_info->flags & COUNT_SPEC_IN_CRITICAL_PATH)
1555 && (DEP_STATUS (dep) & SPECULATIVE))
1556 return false;
1558 return true;
1561 /* Compute the number of nondebug deps in list LIST for INSN. */
1563 static int
1564 dep_list_size (rtx_insn *insn, sd_list_types_def list)
1566 sd_iterator_def sd_it;
1567 dep_t dep;
1568 int dbgcount = 0, nodbgcount = 0;
1570 if (!MAY_HAVE_DEBUG_INSNS)
1571 return sd_lists_size (insn, list);
1573 FOR_EACH_DEP (insn, list, sd_it, dep)
1575 if (DEBUG_INSN_P (DEP_CON (dep)))
1576 dbgcount++;
1577 else if (!DEBUG_INSN_P (DEP_PRO (dep)))
1578 nodbgcount++;
1581 gcc_assert (dbgcount + nodbgcount == sd_lists_size (insn, list));
1583 return nodbgcount;
1586 bool sched_fusion;
1588 /* Compute the priority number for INSN. */
1589 static int
1590 priority (rtx_insn *insn)
1592 if (! INSN_P (insn))
1593 return 0;
1595 /* We should not be interested in priority of an already scheduled insn. */
1596 gcc_assert (QUEUE_INDEX (insn) != QUEUE_SCHEDULED);
1598 if (!INSN_PRIORITY_KNOWN (insn))
1600 int this_priority = -1;
1602 if (sched_fusion)
1604 int this_fusion_priority;
1606 targetm.sched.fusion_priority (insn, FUSION_MAX_PRIORITY,
1607 &this_fusion_priority, &this_priority);
1608 INSN_FUSION_PRIORITY (insn) = this_fusion_priority;
1610 else if (dep_list_size (insn, SD_LIST_FORW) == 0)
1611 /* ??? We should set INSN_PRIORITY to insn_sched_cost when and insn
1612 has some forward deps but all of them are ignored by
1613 contributes_to_priority hook. At the moment we set priority of
1614 such insn to 0. */
1615 this_priority = insn_sched_cost (insn);
1616 else
1618 rtx_insn *prev_first, *twin;
1619 basic_block rec;
1621 /* For recovery check instructions we calculate priority slightly
1622 different than that of normal instructions. Instead of walking
1623 through INSN_FORW_DEPS (check) list, we walk through
1624 INSN_FORW_DEPS list of each instruction in the corresponding
1625 recovery block. */
1627 /* Selective scheduling does not define RECOVERY_BLOCK macro. */
1628 rec = sel_sched_p () ? NULL : RECOVERY_BLOCK (insn);
1629 if (!rec || rec == EXIT_BLOCK_PTR_FOR_FN (cfun))
1631 prev_first = PREV_INSN (insn);
1632 twin = insn;
1634 else
1636 prev_first = NEXT_INSN (BB_HEAD (rec));
1637 twin = PREV_INSN (BB_END (rec));
1642 sd_iterator_def sd_it;
1643 dep_t dep;
1645 FOR_EACH_DEP (twin, SD_LIST_FORW, sd_it, dep)
1647 rtx_insn *next;
1648 int next_priority;
1650 next = DEP_CON (dep);
1652 if (BLOCK_FOR_INSN (next) != rec)
1654 int cost;
1656 if (!contributes_to_priority_p (dep))
1657 continue;
1659 if (twin == insn)
1660 cost = dep_cost (dep);
1661 else
1663 struct _dep _dep1, *dep1 = &_dep1;
1665 init_dep (dep1, insn, next, REG_DEP_ANTI);
1667 cost = dep_cost (dep1);
1670 next_priority = cost + priority (next);
1672 if (next_priority > this_priority)
1673 this_priority = next_priority;
1677 twin = PREV_INSN (twin);
1679 while (twin != prev_first);
1682 if (this_priority < 0)
1684 gcc_assert (this_priority == -1);
1686 this_priority = insn_sched_cost (insn);
1689 INSN_PRIORITY (insn) = this_priority;
1690 INSN_PRIORITY_STATUS (insn) = 1;
1693 return INSN_PRIORITY (insn);
1696 /* Macros and functions for keeping the priority queue sorted, and
1697 dealing with queuing and dequeuing of instructions. */
1699 /* For each pressure class CL, set DEATH[CL] to the number of registers
1700 in that class that die in INSN. */
1702 static void
1703 calculate_reg_deaths (rtx_insn *insn, int *death)
1705 int i;
1706 struct reg_use_data *use;
1708 for (i = 0; i < ira_pressure_classes_num; i++)
1709 death[ira_pressure_classes[i]] = 0;
1710 for (use = INSN_REG_USE_LIST (insn); use != NULL; use = use->next_insn_use)
1711 if (dying_use_p (use))
1712 mark_regno_birth_or_death (0, death, use->regno, true);
1715 /* Setup info about the current register pressure impact of scheduling
1716 INSN at the current scheduling point. */
1717 static void
1718 setup_insn_reg_pressure_info (rtx_insn *insn)
1720 int i, change, before, after, hard_regno;
1721 int excess_cost_change;
1722 machine_mode mode;
1723 enum reg_class cl;
1724 struct reg_pressure_data *pressure_info;
1725 int *max_reg_pressure;
1726 static int death[N_REG_CLASSES];
1728 gcc_checking_assert (!DEBUG_INSN_P (insn));
1730 excess_cost_change = 0;
1731 calculate_reg_deaths (insn, death);
1732 pressure_info = INSN_REG_PRESSURE (insn);
1733 max_reg_pressure = INSN_MAX_REG_PRESSURE (insn);
1734 gcc_assert (pressure_info != NULL && max_reg_pressure != NULL);
1735 for (i = 0; i < ira_pressure_classes_num; i++)
1737 cl = ira_pressure_classes[i];
1738 gcc_assert (curr_reg_pressure[cl] >= 0);
1739 change = (int) pressure_info[i].set_increase - death[cl];
1740 before = MAX (0, max_reg_pressure[i] - sched_class_regs_num[cl]);
1741 after = MAX (0, max_reg_pressure[i] + change
1742 - sched_class_regs_num[cl]);
1743 hard_regno = ira_class_hard_regs[cl][0];
1744 gcc_assert (hard_regno >= 0);
1745 mode = reg_raw_mode[hard_regno];
1746 excess_cost_change += ((after - before)
1747 * (ira_memory_move_cost[mode][cl][0]
1748 + ira_memory_move_cost[mode][cl][1]));
1750 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insn) = excess_cost_change;
1753 /* This is the first page of code related to SCHED_PRESSURE_MODEL.
1754 It tries to make the scheduler take register pressure into account
1755 without introducing too many unnecessary stalls. It hooks into the
1756 main scheduling algorithm at several points:
1758 - Before scheduling starts, model_start_schedule constructs a
1759 "model schedule" for the current block. This model schedule is
1760 chosen solely to keep register pressure down. It does not take the
1761 target's pipeline or the original instruction order into account,
1762 except as a tie-breaker. It also doesn't work to a particular
1763 pressure limit.
1765 This model schedule gives us an idea of what pressure can be
1766 achieved for the block and gives us an example of a schedule that
1767 keeps to that pressure. It also makes the final schedule less
1768 dependent on the original instruction order. This is important
1769 because the original order can either be "wide" (many values live
1770 at once, such as in user-scheduled code) or "narrow" (few values
1771 live at once, such as after loop unrolling, where several
1772 iterations are executed sequentially).
1774 We do not apply this model schedule to the rtx stream. We simply
1775 record it in model_schedule. We also compute the maximum pressure,
1776 MP, that was seen during this schedule.
1778 - Instructions are added to the ready queue even if they require
1779 a stall. The length of the stall is instead computed as:
1781 MAX (INSN_TICK (INSN) - clock_var, 0)
1783 (= insn_delay). This allows rank_for_schedule to choose between
1784 introducing a deliberate stall or increasing pressure.
1786 - Before sorting the ready queue, model_set_excess_costs assigns
1787 a pressure-based cost to each ready instruction in the queue.
1788 This is the instruction's INSN_REG_PRESSURE_EXCESS_COST_CHANGE
1789 (ECC for short) and is effectively measured in cycles.
1791 - rank_for_schedule ranks instructions based on:
1793 ECC (insn) + insn_delay (insn)
1795 then as:
1797 insn_delay (insn)
1799 So, for example, an instruction X1 with an ECC of 1 that can issue
1800 now will win over an instruction X0 with an ECC of zero that would
1801 introduce a stall of one cycle. However, an instruction X2 with an
1802 ECC of 2 that can issue now will lose to both X0 and X1.
1804 - When an instruction is scheduled, model_recompute updates the model
1805 schedule with the new pressures (some of which might now exceed the
1806 original maximum pressure MP). model_update_limit_points then searches
1807 for the new point of maximum pressure, if not already known. */
1809 /* Used to separate high-verbosity debug information for SCHED_PRESSURE_MODEL
1810 from surrounding debug information. */
1811 #define MODEL_BAR \
1812 ";;\t\t+------------------------------------------------------\n"
1814 /* Information about the pressure on a particular register class at a
1815 particular point of the model schedule. */
1816 struct model_pressure_data {
1817 /* The pressure at this point of the model schedule, or -1 if the
1818 point is associated with an instruction that has already been
1819 scheduled. */
1820 int ref_pressure;
1822 /* The maximum pressure during or after this point of the model schedule. */
1823 int max_pressure;
1826 /* Per-instruction information that is used while building the model
1827 schedule. Here, "schedule" refers to the model schedule rather
1828 than the main schedule. */
1829 struct model_insn_info {
1830 /* The instruction itself. */
1831 rtx_insn *insn;
1833 /* If this instruction is in model_worklist, these fields link to the
1834 previous (higher-priority) and next (lower-priority) instructions
1835 in the list. */
1836 struct model_insn_info *prev;
1837 struct model_insn_info *next;
1839 /* While constructing the schedule, QUEUE_INDEX describes whether an
1840 instruction has already been added to the schedule (QUEUE_SCHEDULED),
1841 is in model_worklist (QUEUE_READY), or neither (QUEUE_NOWHERE).
1842 old_queue records the value that QUEUE_INDEX had before scheduling
1843 started, so that we can restore it once the schedule is complete. */
1844 int old_queue;
1846 /* The relative importance of an unscheduled instruction. Higher
1847 values indicate greater importance. */
1848 unsigned int model_priority;
1850 /* The length of the longest path of satisfied true dependencies
1851 that leads to this instruction. */
1852 unsigned int depth;
1854 /* The length of the longest path of dependencies of any kind
1855 that leads from this instruction. */
1856 unsigned int alap;
1858 /* The number of predecessor nodes that must still be scheduled. */
1859 int unscheduled_preds;
1862 /* Information about the pressure limit for a particular register class.
1863 This structure is used when applying a model schedule to the main
1864 schedule. */
1865 struct model_pressure_limit {
1866 /* The maximum register pressure seen in the original model schedule. */
1867 int orig_pressure;
1869 /* The maximum register pressure seen in the current model schedule
1870 (which excludes instructions that have already been scheduled). */
1871 int pressure;
1873 /* The point of the current model schedule at which PRESSURE is first
1874 reached. It is set to -1 if the value needs to be recomputed. */
1875 int point;
1878 /* Describes a particular way of measuring register pressure. */
1879 struct model_pressure_group {
1880 /* Index PCI describes the maximum pressure on ira_pressure_classes[PCI]. */
1881 struct model_pressure_limit limits[N_REG_CLASSES];
1883 /* Index (POINT * ira_num_pressure_classes + PCI) describes the pressure
1884 on register class ira_pressure_classes[PCI] at point POINT of the
1885 current model schedule. A POINT of model_num_insns describes the
1886 pressure at the end of the schedule. */
1887 struct model_pressure_data *model;
1890 /* Index POINT gives the instruction at point POINT of the model schedule.
1891 This array doesn't change during main scheduling. */
1892 static vec<rtx_insn *> model_schedule;
1894 /* The list of instructions in the model worklist, sorted in order of
1895 decreasing priority. */
1896 static struct model_insn_info *model_worklist;
1898 /* Index I describes the instruction with INSN_LUID I. */
1899 static struct model_insn_info *model_insns;
1901 /* The number of instructions in the model schedule. */
1902 static int model_num_insns;
1904 /* The index of the first instruction in model_schedule that hasn't yet been
1905 added to the main schedule, or model_num_insns if all of them have. */
1906 static int model_curr_point;
1908 /* Describes the pressure before each instruction in the model schedule. */
1909 static struct model_pressure_group model_before_pressure;
1911 /* The first unused model_priority value (as used in model_insn_info). */
1912 static unsigned int model_next_priority;
1915 /* The model_pressure_data for ira_pressure_classes[PCI] in GROUP
1916 at point POINT of the model schedule. */
1917 #define MODEL_PRESSURE_DATA(GROUP, POINT, PCI) \
1918 (&(GROUP)->model[(POINT) * ira_pressure_classes_num + (PCI)])
1920 /* The maximum pressure on ira_pressure_classes[PCI] in GROUP at or
1921 after point POINT of the model schedule. */
1922 #define MODEL_MAX_PRESSURE(GROUP, POINT, PCI) \
1923 (MODEL_PRESSURE_DATA (GROUP, POINT, PCI)->max_pressure)
1925 /* The pressure on ira_pressure_classes[PCI] in GROUP at point POINT
1926 of the model schedule. */
1927 #define MODEL_REF_PRESSURE(GROUP, POINT, PCI) \
1928 (MODEL_PRESSURE_DATA (GROUP, POINT, PCI)->ref_pressure)
1930 /* Information about INSN that is used when creating the model schedule. */
1931 #define MODEL_INSN_INFO(INSN) \
1932 (&model_insns[INSN_LUID (INSN)])
1934 /* The instruction at point POINT of the model schedule. */
1935 #define MODEL_INSN(POINT) \
1936 (model_schedule[POINT])
1939 /* Return INSN's index in the model schedule, or model_num_insns if it
1940 doesn't belong to that schedule. */
1942 static int
1943 model_index (rtx_insn *insn)
1945 if (INSN_MODEL_INDEX (insn) == 0)
1946 return model_num_insns;
1947 return INSN_MODEL_INDEX (insn) - 1;
1950 /* Make sure that GROUP->limits is up-to-date for the current point
1951 of the model schedule. */
1953 static void
1954 model_update_limit_points_in_group (struct model_pressure_group *group)
1956 int pci, max_pressure, point;
1958 for (pci = 0; pci < ira_pressure_classes_num; pci++)
1960 /* We may have passed the final point at which the pressure in
1961 group->limits[pci].pressure was reached. Update the limit if so. */
1962 max_pressure = MODEL_MAX_PRESSURE (group, model_curr_point, pci);
1963 group->limits[pci].pressure = max_pressure;
1965 /* Find the point at which MAX_PRESSURE is first reached. We need
1966 to search in three cases:
1968 - We've already moved past the previous pressure point.
1969 In this case we search forward from model_curr_point.
1971 - We scheduled the previous point of maximum pressure ahead of
1972 its position in the model schedule, but doing so didn't bring
1973 the pressure point earlier. In this case we search forward
1974 from that previous pressure point.
1976 - Scheduling an instruction early caused the maximum pressure
1977 to decrease. In this case we will have set the pressure
1978 point to -1, and we search forward from model_curr_point. */
1979 point = MAX (group->limits[pci].point, model_curr_point);
1980 while (point < model_num_insns
1981 && MODEL_REF_PRESSURE (group, point, pci) < max_pressure)
1982 point++;
1983 group->limits[pci].point = point;
1985 gcc_assert (MODEL_REF_PRESSURE (group, point, pci) == max_pressure);
1986 gcc_assert (MODEL_MAX_PRESSURE (group, point, pci) == max_pressure);
1990 /* Make sure that all register-pressure limits are up-to-date for the
1991 current position in the model schedule. */
1993 static void
1994 model_update_limit_points (void)
1996 model_update_limit_points_in_group (&model_before_pressure);
1999 /* Return the model_index of the last unscheduled use in chain USE
2000 outside of USE's instruction. Return -1 if there are no other uses,
2001 or model_num_insns if the register is live at the end of the block. */
2003 static int
2004 model_last_use_except (struct reg_use_data *use)
2006 struct reg_use_data *next;
2007 int last, index;
2009 last = -1;
2010 for (next = use->next_regno_use; next != use; next = next->next_regno_use)
2011 if (NONDEBUG_INSN_P (next->insn)
2012 && QUEUE_INDEX (next->insn) != QUEUE_SCHEDULED)
2014 index = model_index (next->insn);
2015 if (index == model_num_insns)
2016 return model_num_insns;
2017 if (last < index)
2018 last = index;
2020 return last;
2023 /* An instruction with model_index POINT has just been scheduled, and it
2024 adds DELTA to the pressure on ira_pressure_classes[PCI] after POINT - 1.
2025 Update MODEL_REF_PRESSURE (GROUP, POINT, PCI) and
2026 MODEL_MAX_PRESSURE (GROUP, POINT, PCI) accordingly. */
2028 static void
2029 model_start_update_pressure (struct model_pressure_group *group,
2030 int point, int pci, int delta)
2032 int next_max_pressure;
2034 if (point == model_num_insns)
2036 /* The instruction wasn't part of the model schedule; it was moved
2037 from a different block. Update the pressure for the end of
2038 the model schedule. */
2039 MODEL_REF_PRESSURE (group, point, pci) += delta;
2040 MODEL_MAX_PRESSURE (group, point, pci) += delta;
2042 else
2044 /* Record that this instruction has been scheduled. Nothing now
2045 changes between POINT and POINT + 1, so get the maximum pressure
2046 from the latter. If the maximum pressure decreases, the new
2047 pressure point may be before POINT. */
2048 MODEL_REF_PRESSURE (group, point, pci) = -1;
2049 next_max_pressure = MODEL_MAX_PRESSURE (group, point + 1, pci);
2050 if (MODEL_MAX_PRESSURE (group, point, pci) > next_max_pressure)
2052 MODEL_MAX_PRESSURE (group, point, pci) = next_max_pressure;
2053 if (group->limits[pci].point == point)
2054 group->limits[pci].point = -1;
2059 /* Record that scheduling a later instruction has changed the pressure
2060 at point POINT of the model schedule by DELTA (which might be 0).
2061 Update GROUP accordingly. Return nonzero if these changes might
2062 trigger changes to previous points as well. */
2064 static int
2065 model_update_pressure (struct model_pressure_group *group,
2066 int point, int pci, int delta)
2068 int ref_pressure, max_pressure, next_max_pressure;
2070 /* If POINT hasn't yet been scheduled, update its pressure. */
2071 ref_pressure = MODEL_REF_PRESSURE (group, point, pci);
2072 if (ref_pressure >= 0 && delta != 0)
2074 ref_pressure += delta;
2075 MODEL_REF_PRESSURE (group, point, pci) = ref_pressure;
2077 /* Check whether the maximum pressure in the overall schedule
2078 has increased. (This means that the MODEL_MAX_PRESSURE of
2079 every point <= POINT will need to increase too; see below.) */
2080 if (group->limits[pci].pressure < ref_pressure)
2081 group->limits[pci].pressure = ref_pressure;
2083 /* If we are at maximum pressure, and the maximum pressure
2084 point was previously unknown or later than POINT,
2085 bring it forward. */
2086 if (group->limits[pci].pressure == ref_pressure
2087 && !IN_RANGE (group->limits[pci].point, 0, point))
2088 group->limits[pci].point = point;
2090 /* If POINT used to be the point of maximum pressure, but isn't
2091 any longer, we need to recalculate it using a forward walk. */
2092 if (group->limits[pci].pressure > ref_pressure
2093 && group->limits[pci].point == point)
2094 group->limits[pci].point = -1;
2097 /* Update the maximum pressure at POINT. Changes here might also
2098 affect the maximum pressure at POINT - 1. */
2099 next_max_pressure = MODEL_MAX_PRESSURE (group, point + 1, pci);
2100 max_pressure = MAX (ref_pressure, next_max_pressure);
2101 if (MODEL_MAX_PRESSURE (group, point, pci) != max_pressure)
2103 MODEL_MAX_PRESSURE (group, point, pci) = max_pressure;
2104 return 1;
2106 return 0;
2109 /* INSN has just been scheduled. Update the model schedule accordingly. */
2111 static void
2112 model_recompute (rtx_insn *insn)
2114 struct {
2115 int last_use;
2116 int regno;
2117 } uses[FIRST_PSEUDO_REGISTER + MAX_RECOG_OPERANDS];
2118 struct reg_use_data *use;
2119 struct reg_pressure_data *reg_pressure;
2120 int delta[N_REG_CLASSES];
2121 int pci, point, mix, new_last, cl, ref_pressure, queue;
2122 unsigned int i, num_uses, num_pending_births;
2123 bool print_p;
2125 /* The destinations of INSN were previously live from POINT onwards, but are
2126 now live from model_curr_point onwards. Set up DELTA accordingly. */
2127 point = model_index (insn);
2128 reg_pressure = INSN_REG_PRESSURE (insn);
2129 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2131 cl = ira_pressure_classes[pci];
2132 delta[cl] = reg_pressure[pci].set_increase;
2135 /* Record which registers previously died at POINT, but which now die
2136 before POINT. Adjust DELTA so that it represents the effect of
2137 this change after POINT - 1. Set NUM_PENDING_BIRTHS to the number of
2138 registers that will be born in the range [model_curr_point, POINT). */
2139 num_uses = 0;
2140 num_pending_births = 0;
2141 bitmap_clear (tmp_bitmap);
2142 for (use = INSN_REG_USE_LIST (insn); use != NULL; use = use->next_insn_use)
2144 new_last = model_last_use_except (use);
2145 if (new_last < point && bitmap_set_bit (tmp_bitmap, use->regno))
2147 gcc_assert (num_uses < ARRAY_SIZE (uses));
2148 uses[num_uses].last_use = new_last;
2149 uses[num_uses].regno = use->regno;
2150 /* This register is no longer live after POINT - 1. */
2151 mark_regno_birth_or_death (NULL, delta, use->regno, false);
2152 num_uses++;
2153 if (new_last >= 0)
2154 num_pending_births++;
2158 /* Update the MODEL_REF_PRESSURE and MODEL_MAX_PRESSURE for POINT.
2159 Also set each group pressure limit for POINT. */
2160 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2162 cl = ira_pressure_classes[pci];
2163 model_start_update_pressure (&model_before_pressure,
2164 point, pci, delta[cl]);
2167 /* Walk the model schedule backwards, starting immediately before POINT. */
2168 print_p = false;
2169 if (point != model_curr_point)
2172 point--;
2173 insn = MODEL_INSN (point);
2174 queue = QUEUE_INDEX (insn);
2176 if (queue != QUEUE_SCHEDULED)
2178 /* DELTA describes the effect of the move on the register pressure
2179 after POINT. Make it describe the effect on the pressure
2180 before POINT. */
2181 i = 0;
2182 while (i < num_uses)
2184 if (uses[i].last_use == point)
2186 /* This register is now live again. */
2187 mark_regno_birth_or_death (NULL, delta,
2188 uses[i].regno, true);
2190 /* Remove this use from the array. */
2191 uses[i] = uses[num_uses - 1];
2192 num_uses--;
2193 num_pending_births--;
2195 else
2196 i++;
2199 if (sched_verbose >= 5)
2201 if (!print_p)
2203 fprintf (sched_dump, MODEL_BAR);
2204 fprintf (sched_dump, ";;\t\t| New pressure for model"
2205 " schedule\n");
2206 fprintf (sched_dump, MODEL_BAR);
2207 print_p = true;
2210 fprintf (sched_dump, ";;\t\t| %3d %4d %-30s ",
2211 point, INSN_UID (insn),
2212 str_pattern_slim (PATTERN (insn)));
2213 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2215 cl = ira_pressure_classes[pci];
2216 ref_pressure = MODEL_REF_PRESSURE (&model_before_pressure,
2217 point, pci);
2218 fprintf (sched_dump, " %s:[%d->%d]",
2219 reg_class_names[ira_pressure_classes[pci]],
2220 ref_pressure, ref_pressure + delta[cl]);
2222 fprintf (sched_dump, "\n");
2226 /* Adjust the pressure at POINT. Set MIX to nonzero if POINT - 1
2227 might have changed as well. */
2228 mix = num_pending_births;
2229 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2231 cl = ira_pressure_classes[pci];
2232 mix |= delta[cl];
2233 mix |= model_update_pressure (&model_before_pressure,
2234 point, pci, delta[cl]);
2237 while (mix && point > model_curr_point);
2239 if (print_p)
2240 fprintf (sched_dump, MODEL_BAR);
2243 /* After DEP, which was cancelled, has been resolved for insn NEXT,
2244 check whether the insn's pattern needs restoring. */
2245 static bool
2246 must_restore_pattern_p (rtx_insn *next, dep_t dep)
2248 if (QUEUE_INDEX (next) == QUEUE_SCHEDULED)
2249 return false;
2251 if (DEP_TYPE (dep) == REG_DEP_CONTROL)
2253 gcc_assert (ORIG_PAT (next) != NULL_RTX);
2254 gcc_assert (next == DEP_CON (dep));
2256 else
2258 struct dep_replacement *desc = DEP_REPLACE (dep);
2259 if (desc->insn != next)
2261 gcc_assert (*desc->loc == desc->orig);
2262 return false;
2265 return true;
2268 /* model_spill_cost (CL, P, P') returns the cost of increasing the
2269 pressure on CL from P to P'. We use this to calculate a "base ECC",
2270 baseECC (CL, X), for each pressure class CL and each instruction X.
2271 Supposing X changes the pressure on CL from P to P', and that the
2272 maximum pressure on CL in the current model schedule is MP', then:
2274 * if X occurs before or at the next point of maximum pressure in
2275 the model schedule and P' > MP', then:
2277 baseECC (CL, X) = model_spill_cost (CL, MP, P')
2279 The idea is that the pressure after scheduling a fixed set of
2280 instructions -- in this case, the set up to and including the
2281 next maximum pressure point -- is going to be the same regardless
2282 of the order; we simply want to keep the intermediate pressure
2283 under control. Thus X has a cost of zero unless scheduling it
2284 now would exceed MP'.
2286 If all increases in the set are by the same amount, no zero-cost
2287 instruction will ever cause the pressure to exceed MP'. However,
2288 if X is instead moved past an instruction X' with pressure in the
2289 range (MP' - (P' - P), MP'), the pressure at X' will increase
2290 beyond MP'. Since baseECC is very much a heuristic anyway,
2291 it doesn't seem worth the overhead of tracking cases like these.
2293 The cost of exceeding MP' is always based on the original maximum
2294 pressure MP. This is so that going 2 registers over the original
2295 limit has the same cost regardless of whether it comes from two
2296 separate +1 deltas or from a single +2 delta.
2298 * if X occurs after the next point of maximum pressure in the model
2299 schedule and P' > P, then:
2301 baseECC (CL, X) = model_spill_cost (CL, MP, MP' + (P' - P))
2303 That is, if we move X forward across a point of maximum pressure,
2304 and if X increases the pressure by P' - P, then we conservatively
2305 assume that scheduling X next would increase the maximum pressure
2306 by P' - P. Again, the cost of doing this is based on the original
2307 maximum pressure MP, for the same reason as above.
2309 * if P' < P, P > MP, and X occurs at or after the next point of
2310 maximum pressure, then:
2312 baseECC (CL, X) = -model_spill_cost (CL, MAX (MP, P'), P)
2314 That is, if we have already exceeded the original maximum pressure MP,
2315 and if X might reduce the maximum pressure again -- or at least push
2316 it further back, and thus allow more scheduling freedom -- it is given
2317 a negative cost to reflect the improvement.
2319 * otherwise,
2321 baseECC (CL, X) = 0
2323 In this case, X is not expected to affect the maximum pressure MP',
2324 so it has zero cost.
2326 We then create a combined value baseECC (X) that is the sum of
2327 baseECC (CL, X) for each pressure class CL.
2329 baseECC (X) could itself be used as the ECC value described above.
2330 However, this is often too conservative, in the sense that it
2331 tends to make high-priority instructions that increase pressure
2332 wait too long in cases where introducing a spill would be better.
2333 For this reason the final ECC is a priority-adjusted form of
2334 baseECC (X). Specifically, we calculate:
2336 P (X) = INSN_PRIORITY (X) - insn_delay (X) - baseECC (X)
2337 baseP = MAX { P (X) | baseECC (X) <= 0 }
2339 Then:
2341 ECC (X) = MAX (MIN (baseP - P (X), baseECC (X)), 0)
2343 Thus an instruction's effect on pressure is ignored if it has a high
2344 enough priority relative to the ones that don't increase pressure.
2345 Negative values of baseECC (X) do not increase the priority of X
2346 itself, but they do make it harder for other instructions to
2347 increase the pressure further.
2349 This pressure cost is deliberately timid. The intention has been
2350 to choose a heuristic that rarely interferes with the normal list
2351 scheduler in cases where that scheduler would produce good code.
2352 We simply want to curb some of its worst excesses. */
2354 /* Return the cost of increasing the pressure in class CL from FROM to TO.
2356 Here we use the very simplistic cost model that every register above
2357 sched_class_regs_num[CL] has a spill cost of 1. We could use other
2358 measures instead, such as one based on MEMORY_MOVE_COST. However:
2360 (1) In order for an instruction to be scheduled, the higher cost
2361 would need to be justified in a single saving of that many stalls.
2362 This is overly pessimistic, because the benefit of spilling is
2363 often to avoid a sequence of several short stalls rather than
2364 a single long one.
2366 (2) The cost is still arbitrary. Because we are not allocating
2367 registers during scheduling, we have no way of knowing for
2368 sure how many memory accesses will be required by each spill,
2369 where the spills will be placed within the block, or even
2370 which block(s) will contain the spills.
2372 So a higher cost than 1 is often too conservative in practice,
2373 forcing blocks to contain unnecessary stalls instead of spill code.
2374 The simple cost below seems to be the best compromise. It reduces
2375 the interference with the normal list scheduler, which helps make
2376 it more suitable for a default-on option. */
2378 static int
2379 model_spill_cost (int cl, int from, int to)
2381 from = MAX (from, sched_class_regs_num[cl]);
2382 return MAX (to, from) - from;
2385 /* Return baseECC (ira_pressure_classes[PCI], POINT), given that
2386 P = curr_reg_pressure[ira_pressure_classes[PCI]] and that
2387 P' = P + DELTA. */
2389 static int
2390 model_excess_group_cost (struct model_pressure_group *group,
2391 int point, int pci, int delta)
2393 int pressure, cl;
2395 cl = ira_pressure_classes[pci];
2396 if (delta < 0 && point >= group->limits[pci].point)
2398 pressure = MAX (group->limits[pci].orig_pressure,
2399 curr_reg_pressure[cl] + delta);
2400 return -model_spill_cost (cl, pressure, curr_reg_pressure[cl]);
2403 if (delta > 0)
2405 if (point > group->limits[pci].point)
2406 pressure = group->limits[pci].pressure + delta;
2407 else
2408 pressure = curr_reg_pressure[cl] + delta;
2410 if (pressure > group->limits[pci].pressure)
2411 return model_spill_cost (cl, group->limits[pci].orig_pressure,
2412 pressure);
2415 return 0;
2418 /* Return baseECC (MODEL_INSN (INSN)). Dump the costs to sched_dump
2419 if PRINT_P. */
2421 static int
2422 model_excess_cost (rtx_insn *insn, bool print_p)
2424 int point, pci, cl, cost, this_cost, delta;
2425 struct reg_pressure_data *insn_reg_pressure;
2426 int insn_death[N_REG_CLASSES];
2428 calculate_reg_deaths (insn, insn_death);
2429 point = model_index (insn);
2430 insn_reg_pressure = INSN_REG_PRESSURE (insn);
2431 cost = 0;
2433 if (print_p)
2434 fprintf (sched_dump, ";;\t\t| %3d %4d | %4d %+3d |", point,
2435 INSN_UID (insn), INSN_PRIORITY (insn), insn_delay (insn));
2437 /* Sum up the individual costs for each register class. */
2438 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2440 cl = ira_pressure_classes[pci];
2441 delta = insn_reg_pressure[pci].set_increase - insn_death[cl];
2442 this_cost = model_excess_group_cost (&model_before_pressure,
2443 point, pci, delta);
2444 cost += this_cost;
2445 if (print_p)
2446 fprintf (sched_dump, " %s:[%d base cost %d]",
2447 reg_class_names[cl], delta, this_cost);
2450 if (print_p)
2451 fprintf (sched_dump, "\n");
2453 return cost;
2456 /* Dump the next points of maximum pressure for GROUP. */
2458 static void
2459 model_dump_pressure_points (struct model_pressure_group *group)
2461 int pci, cl;
2463 fprintf (sched_dump, ";;\t\t| pressure points");
2464 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2466 cl = ira_pressure_classes[pci];
2467 fprintf (sched_dump, " %s:[%d->%d at ", reg_class_names[cl],
2468 curr_reg_pressure[cl], group->limits[pci].pressure);
2469 if (group->limits[pci].point < model_num_insns)
2470 fprintf (sched_dump, "%d:%d]", group->limits[pci].point,
2471 INSN_UID (MODEL_INSN (group->limits[pci].point)));
2472 else
2473 fprintf (sched_dump, "end]");
2475 fprintf (sched_dump, "\n");
2478 /* Set INSN_REG_PRESSURE_EXCESS_COST_CHANGE for INSNS[0...COUNT-1]. */
2480 static void
2481 model_set_excess_costs (rtx_insn **insns, int count)
2483 int i, cost, priority_base, priority;
2484 bool print_p;
2486 /* Record the baseECC value for each instruction in the model schedule,
2487 except that negative costs are converted to zero ones now rather than
2488 later. Do not assign a cost to debug instructions, since they must
2489 not change code-generation decisions. Experiments suggest we also
2490 get better results by not assigning a cost to instructions from
2491 a different block.
2493 Set PRIORITY_BASE to baseP in the block comment above. This is the
2494 maximum priority of the "cheap" instructions, which should always
2495 include the next model instruction. */
2496 priority_base = 0;
2497 print_p = false;
2498 for (i = 0; i < count; i++)
2499 if (INSN_MODEL_INDEX (insns[i]))
2501 if (sched_verbose >= 6 && !print_p)
2503 fprintf (sched_dump, MODEL_BAR);
2504 fprintf (sched_dump, ";;\t\t| Pressure costs for ready queue\n");
2505 model_dump_pressure_points (&model_before_pressure);
2506 fprintf (sched_dump, MODEL_BAR);
2507 print_p = true;
2509 cost = model_excess_cost (insns[i], print_p);
2510 if (cost <= 0)
2512 priority = INSN_PRIORITY (insns[i]) - insn_delay (insns[i]) - cost;
2513 priority_base = MAX (priority_base, priority);
2514 cost = 0;
2516 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insns[i]) = cost;
2518 if (print_p)
2519 fprintf (sched_dump, MODEL_BAR);
2521 /* Use MAX (baseECC, 0) and baseP to calculcate ECC for each
2522 instruction. */
2523 for (i = 0; i < count; i++)
2525 cost = INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insns[i]);
2526 priority = INSN_PRIORITY (insns[i]) - insn_delay (insns[i]);
2527 if (cost > 0 && priority > priority_base)
2529 cost += priority_base - priority;
2530 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insns[i]) = MAX (cost, 0);
2536 /* Enum of rank_for_schedule heuristic decisions. */
2537 enum rfs_decision {
2538 RFS_LIVE_RANGE_SHRINK1, RFS_LIVE_RANGE_SHRINK2,
2539 RFS_SCHED_GROUP, RFS_PRESSURE_DELAY, RFS_PRESSURE_TICK,
2540 RFS_FEEDS_BACKTRACK_INSN, RFS_PRIORITY, RFS_SPECULATION,
2541 RFS_SCHED_RANK, RFS_LAST_INSN, RFS_PRESSURE_INDEX,
2542 RFS_DEP_COUNT, RFS_TIE, RFS_FUSION, RFS_N };
2544 /* Corresponding strings for print outs. */
2545 static const char *rfs_str[RFS_N] = {
2546 "RFS_LIVE_RANGE_SHRINK1", "RFS_LIVE_RANGE_SHRINK2",
2547 "RFS_SCHED_GROUP", "RFS_PRESSURE_DELAY", "RFS_PRESSURE_TICK",
2548 "RFS_FEEDS_BACKTRACK_INSN", "RFS_PRIORITY", "RFS_SPECULATION",
2549 "RFS_SCHED_RANK", "RFS_LAST_INSN", "RFS_PRESSURE_INDEX",
2550 "RFS_DEP_COUNT", "RFS_TIE", "RFS_FUSION" };
2552 /* Statistical breakdown of rank_for_schedule decisions. */
2553 struct rank_for_schedule_stats_t { unsigned stats[RFS_N]; };
2554 static rank_for_schedule_stats_t rank_for_schedule_stats;
2556 /* Return the result of comparing insns TMP and TMP2 and update
2557 Rank_For_Schedule statistics. */
2558 static int
2559 rfs_result (enum rfs_decision decision, int result, rtx tmp, rtx tmp2)
2561 ++rank_for_schedule_stats.stats[decision];
2562 if (result < 0)
2563 INSN_LAST_RFS_WIN (tmp) = decision;
2564 else if (result > 0)
2565 INSN_LAST_RFS_WIN (tmp2) = decision;
2566 else
2567 gcc_unreachable ();
2568 return result;
2571 /* Sorting predicate to move DEBUG_INSNs to the top of ready list, while
2572 keeping normal insns in original order. */
2574 static int
2575 rank_for_schedule_debug (const void *x, const void *y)
2577 rtx_insn *tmp = *(rtx_insn * const *) y;
2578 rtx_insn *tmp2 = *(rtx_insn * const *) x;
2580 /* Schedule debug insns as early as possible. */
2581 if (DEBUG_INSN_P (tmp) && !DEBUG_INSN_P (tmp2))
2582 return -1;
2583 else if (!DEBUG_INSN_P (tmp) && DEBUG_INSN_P (tmp2))
2584 return 1;
2585 else if (DEBUG_INSN_P (tmp) && DEBUG_INSN_P (tmp2))
2586 return INSN_LUID (tmp) - INSN_LUID (tmp2);
2587 else
2588 return INSN_RFS_DEBUG_ORIG_ORDER (tmp2) - INSN_RFS_DEBUG_ORIG_ORDER (tmp);
2591 /* Returns a positive value if x is preferred; returns a negative value if
2592 y is preferred. Should never return 0, since that will make the sort
2593 unstable. */
2595 static int
2596 rank_for_schedule (const void *x, const void *y)
2598 rtx_insn *tmp = *(rtx_insn * const *) y;
2599 rtx_insn *tmp2 = *(rtx_insn * const *) x;
2600 int tmp_class, tmp2_class;
2601 int val, priority_val, info_val, diff;
2603 if (live_range_shrinkage_p)
2605 /* Don't use SCHED_PRESSURE_MODEL -- it results in much worse
2606 code. */
2607 gcc_assert (sched_pressure == SCHED_PRESSURE_WEIGHTED);
2608 if ((INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp) < 0
2609 || INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp2) < 0)
2610 && (diff = (INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp)
2611 - INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp2))) != 0)
2612 return rfs_result (RFS_LIVE_RANGE_SHRINK1, diff, tmp, tmp2);
2613 /* Sort by INSN_LUID (original insn order), so that we make the
2614 sort stable. This minimizes instruction movement, thus
2615 minimizing sched's effect on debugging and cross-jumping. */
2616 return rfs_result (RFS_LIVE_RANGE_SHRINK2,
2617 INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2);
2620 /* The insn in a schedule group should be issued the first. */
2621 if (flag_sched_group_heuristic &&
2622 SCHED_GROUP_P (tmp) != SCHED_GROUP_P (tmp2))
2623 return rfs_result (RFS_SCHED_GROUP, SCHED_GROUP_P (tmp2) ? 1 : -1,
2624 tmp, tmp2);
2626 /* Make sure that priority of TMP and TMP2 are initialized. */
2627 gcc_assert (INSN_PRIORITY_KNOWN (tmp) && INSN_PRIORITY_KNOWN (tmp2));
2629 if (sched_fusion)
2631 /* The instruction that has the same fusion priority as the last
2632 instruction is the instruction we picked next. If that is not
2633 the case, we sort ready list firstly by fusion priority, then
2634 by priority, and at last by INSN_LUID. */
2635 int a = INSN_FUSION_PRIORITY (tmp);
2636 int b = INSN_FUSION_PRIORITY (tmp2);
2637 int last = -1;
2639 if (last_nondebug_scheduled_insn
2640 && !NOTE_P (last_nondebug_scheduled_insn)
2641 && BLOCK_FOR_INSN (tmp)
2642 == BLOCK_FOR_INSN (last_nondebug_scheduled_insn))
2643 last = INSN_FUSION_PRIORITY (last_nondebug_scheduled_insn);
2645 if (a != last && b != last)
2647 if (a == b)
2649 a = INSN_PRIORITY (tmp);
2650 b = INSN_PRIORITY (tmp2);
2652 if (a != b)
2653 return rfs_result (RFS_FUSION, b - a, tmp, tmp2);
2654 else
2655 return rfs_result (RFS_FUSION,
2656 INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2);
2658 else if (a == b)
2660 gcc_assert (last_nondebug_scheduled_insn
2661 && !NOTE_P (last_nondebug_scheduled_insn));
2662 last = INSN_PRIORITY (last_nondebug_scheduled_insn);
2664 a = abs (INSN_PRIORITY (tmp) - last);
2665 b = abs (INSN_PRIORITY (tmp2) - last);
2666 if (a != b)
2667 return rfs_result (RFS_FUSION, a - b, tmp, tmp2);
2668 else
2669 return rfs_result (RFS_FUSION,
2670 INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2);
2672 else if (a == last)
2673 return rfs_result (RFS_FUSION, -1, tmp, tmp2);
2674 else
2675 return rfs_result (RFS_FUSION, 1, tmp, tmp2);
2678 if (sched_pressure != SCHED_PRESSURE_NONE)
2680 /* Prefer insn whose scheduling results in the smallest register
2681 pressure excess. */
2682 if ((diff = (INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp)
2683 + insn_delay (tmp)
2684 - INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp2)
2685 - insn_delay (tmp2))))
2686 return rfs_result (RFS_PRESSURE_DELAY, diff, tmp, tmp2);
2689 if (sched_pressure != SCHED_PRESSURE_NONE
2690 && (INSN_TICK (tmp2) > clock_var || INSN_TICK (tmp) > clock_var)
2691 && INSN_TICK (tmp2) != INSN_TICK (tmp))
2693 diff = INSN_TICK (tmp) - INSN_TICK (tmp2);
2694 return rfs_result (RFS_PRESSURE_TICK, diff, tmp, tmp2);
2697 /* If we are doing backtracking in this schedule, prefer insns that
2698 have forward dependencies with negative cost against an insn that
2699 was already scheduled. */
2700 if (current_sched_info->flags & DO_BACKTRACKING)
2702 priority_val = FEEDS_BACKTRACK_INSN (tmp2) - FEEDS_BACKTRACK_INSN (tmp);
2703 if (priority_val)
2704 return rfs_result (RFS_FEEDS_BACKTRACK_INSN, priority_val, tmp, tmp2);
2707 /* Prefer insn with higher priority. */
2708 priority_val = INSN_PRIORITY (tmp2) - INSN_PRIORITY (tmp);
2710 if (flag_sched_critical_path_heuristic && priority_val)
2711 return rfs_result (RFS_PRIORITY, priority_val, tmp, tmp2);
2713 if (PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) >= 0)
2715 int autopref = autopref_rank_for_schedule (tmp, tmp2);
2716 if (autopref != 0)
2717 return autopref;
2720 /* Prefer speculative insn with greater dependencies weakness. */
2721 if (flag_sched_spec_insn_heuristic && spec_info)
2723 ds_t ds1, ds2;
2724 dw_t dw1, dw2;
2725 int dw;
2727 ds1 = TODO_SPEC (tmp) & SPECULATIVE;
2728 if (ds1)
2729 dw1 = ds_weak (ds1);
2730 else
2731 dw1 = NO_DEP_WEAK;
2733 ds2 = TODO_SPEC (tmp2) & SPECULATIVE;
2734 if (ds2)
2735 dw2 = ds_weak (ds2);
2736 else
2737 dw2 = NO_DEP_WEAK;
2739 dw = dw2 - dw1;
2740 if (dw > (NO_DEP_WEAK / 8) || dw < -(NO_DEP_WEAK / 8))
2741 return rfs_result (RFS_SPECULATION, dw, tmp, tmp2);
2744 info_val = (*current_sched_info->rank) (tmp, tmp2);
2745 if (flag_sched_rank_heuristic && info_val)
2746 return rfs_result (RFS_SCHED_RANK, info_val, tmp, tmp2);
2748 /* Compare insns based on their relation to the last scheduled
2749 non-debug insn. */
2750 if (flag_sched_last_insn_heuristic && last_nondebug_scheduled_insn)
2752 dep_t dep1;
2753 dep_t dep2;
2754 rtx_insn *last = last_nondebug_scheduled_insn;
2756 /* Classify the instructions into three classes:
2757 1) Data dependent on last schedule insn.
2758 2) Anti/Output dependent on last scheduled insn.
2759 3) Independent of last scheduled insn, or has latency of one.
2760 Choose the insn from the highest numbered class if different. */
2761 dep1 = sd_find_dep_between (last, tmp, true);
2763 if (dep1 == NULL || dep_cost (dep1) == 1)
2764 tmp_class = 3;
2765 else if (/* Data dependence. */
2766 DEP_TYPE (dep1) == REG_DEP_TRUE)
2767 tmp_class = 1;
2768 else
2769 tmp_class = 2;
2771 dep2 = sd_find_dep_between (last, tmp2, true);
2773 if (dep2 == NULL || dep_cost (dep2) == 1)
2774 tmp2_class = 3;
2775 else if (/* Data dependence. */
2776 DEP_TYPE (dep2) == REG_DEP_TRUE)
2777 tmp2_class = 1;
2778 else
2779 tmp2_class = 2;
2781 if ((val = tmp2_class - tmp_class))
2782 return rfs_result (RFS_LAST_INSN, val, tmp, tmp2);
2785 /* Prefer instructions that occur earlier in the model schedule. */
2786 if (sched_pressure == SCHED_PRESSURE_MODEL)
2788 diff = model_index (tmp) - model_index (tmp2);
2789 if (diff != 0)
2790 return rfs_result (RFS_PRESSURE_INDEX, diff, tmp, tmp2);
2793 /* Prefer the insn which has more later insns that depend on it.
2794 This gives the scheduler more freedom when scheduling later
2795 instructions at the expense of added register pressure. */
2797 val = (dep_list_size (tmp2, SD_LIST_FORW)
2798 - dep_list_size (tmp, SD_LIST_FORW));
2800 if (flag_sched_dep_count_heuristic && val != 0)
2801 return rfs_result (RFS_DEP_COUNT, val, tmp, tmp2);
2803 /* If insns are equally good, sort by INSN_LUID (original insn order),
2804 so that we make the sort stable. This minimizes instruction movement,
2805 thus minimizing sched's effect on debugging and cross-jumping. */
2806 return rfs_result (RFS_TIE, INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2);
2809 /* Resort the array A in which only element at index N may be out of order. */
2811 HAIFA_INLINE static void
2812 swap_sort (rtx_insn **a, int n)
2814 rtx_insn *insn = a[n - 1];
2815 int i = n - 2;
2817 while (i >= 0 && rank_for_schedule (a + i, &insn) >= 0)
2819 a[i + 1] = a[i];
2820 i -= 1;
2822 a[i + 1] = insn;
2825 /* Add INSN to the insn queue so that it can be executed at least
2826 N_CYCLES after the currently executing insn. Preserve insns
2827 chain for debugging purposes. REASON will be printed in debugging
2828 output. */
2830 HAIFA_INLINE static void
2831 queue_insn (rtx_insn *insn, int n_cycles, const char *reason)
2833 int next_q = NEXT_Q_AFTER (q_ptr, n_cycles);
2834 rtx_insn_list *link = alloc_INSN_LIST (insn, insn_queue[next_q]);
2835 int new_tick;
2837 gcc_assert (n_cycles <= max_insn_queue_index);
2838 gcc_assert (!DEBUG_INSN_P (insn));
2840 insn_queue[next_q] = link;
2841 q_size += 1;
2843 if (sched_verbose >= 2)
2845 fprintf (sched_dump, ";;\t\tReady-->Q: insn %s: ",
2846 (*current_sched_info->print_insn) (insn, 0));
2848 fprintf (sched_dump, "queued for %d cycles (%s).\n", n_cycles, reason);
2851 QUEUE_INDEX (insn) = next_q;
2853 if (current_sched_info->flags & DO_BACKTRACKING)
2855 new_tick = clock_var + n_cycles;
2856 if (INSN_TICK (insn) == INVALID_TICK || INSN_TICK (insn) < new_tick)
2857 INSN_TICK (insn) = new_tick;
2859 if (INSN_EXACT_TICK (insn) != INVALID_TICK
2860 && INSN_EXACT_TICK (insn) < clock_var + n_cycles)
2862 must_backtrack = true;
2863 if (sched_verbose >= 2)
2864 fprintf (sched_dump, ";;\t\tcausing a backtrack.\n");
2869 /* Remove INSN from queue. */
2870 static void
2871 queue_remove (rtx_insn *insn)
2873 gcc_assert (QUEUE_INDEX (insn) >= 0);
2874 remove_free_INSN_LIST_elem (insn, &insn_queue[QUEUE_INDEX (insn)]);
2875 q_size--;
2876 QUEUE_INDEX (insn) = QUEUE_NOWHERE;
2879 /* Return a pointer to the bottom of the ready list, i.e. the insn
2880 with the lowest priority. */
2882 rtx_insn **
2883 ready_lastpos (struct ready_list *ready)
2885 gcc_assert (ready->n_ready >= 1);
2886 return ready->vec + ready->first - ready->n_ready + 1;
2889 /* Add an element INSN to the ready list so that it ends up with the
2890 lowest/highest priority depending on FIRST_P. */
2892 HAIFA_INLINE static void
2893 ready_add (struct ready_list *ready, rtx_insn *insn, bool first_p)
2895 if (!first_p)
2897 if (ready->first == ready->n_ready)
2899 memmove (ready->vec + ready->veclen - ready->n_ready,
2900 ready_lastpos (ready),
2901 ready->n_ready * sizeof (rtx));
2902 ready->first = ready->veclen - 1;
2904 ready->vec[ready->first - ready->n_ready] = insn;
2906 else
2908 if (ready->first == ready->veclen - 1)
2910 if (ready->n_ready)
2911 /* ready_lastpos() fails when called with (ready->n_ready == 0). */
2912 memmove (ready->vec + ready->veclen - ready->n_ready - 1,
2913 ready_lastpos (ready),
2914 ready->n_ready * sizeof (rtx));
2915 ready->first = ready->veclen - 2;
2917 ready->vec[++(ready->first)] = insn;
2920 ready->n_ready++;
2921 if (DEBUG_INSN_P (insn))
2922 ready->n_debug++;
2924 gcc_assert (QUEUE_INDEX (insn) != QUEUE_READY);
2925 QUEUE_INDEX (insn) = QUEUE_READY;
2927 if (INSN_EXACT_TICK (insn) != INVALID_TICK
2928 && INSN_EXACT_TICK (insn) < clock_var)
2930 must_backtrack = true;
2934 /* Remove the element with the highest priority from the ready list and
2935 return it. */
2937 HAIFA_INLINE static rtx_insn *
2938 ready_remove_first (struct ready_list *ready)
2940 rtx_insn *t;
2942 gcc_assert (ready->n_ready);
2943 t = ready->vec[ready->first--];
2944 ready->n_ready--;
2945 if (DEBUG_INSN_P (t))
2946 ready->n_debug--;
2947 /* If the queue becomes empty, reset it. */
2948 if (ready->n_ready == 0)
2949 ready->first = ready->veclen - 1;
2951 gcc_assert (QUEUE_INDEX (t) == QUEUE_READY);
2952 QUEUE_INDEX (t) = QUEUE_NOWHERE;
2954 return t;
2957 /* The following code implements multi-pass scheduling for the first
2958 cycle. In other words, we will try to choose ready insn which
2959 permits to start maximum number of insns on the same cycle. */
2961 /* Return a pointer to the element INDEX from the ready. INDEX for
2962 insn with the highest priority is 0, and the lowest priority has
2963 N_READY - 1. */
2965 rtx_insn *
2966 ready_element (struct ready_list *ready, int index)
2968 gcc_assert (ready->n_ready && index < ready->n_ready);
2970 return ready->vec[ready->first - index];
2973 /* Remove the element INDEX from the ready list and return it. INDEX
2974 for insn with the highest priority is 0, and the lowest priority
2975 has N_READY - 1. */
2977 HAIFA_INLINE static rtx_insn *
2978 ready_remove (struct ready_list *ready, int index)
2980 rtx_insn *t;
2981 int i;
2983 if (index == 0)
2984 return ready_remove_first (ready);
2985 gcc_assert (ready->n_ready && index < ready->n_ready);
2986 t = ready->vec[ready->first - index];
2987 ready->n_ready--;
2988 if (DEBUG_INSN_P (t))
2989 ready->n_debug--;
2990 for (i = index; i < ready->n_ready; i++)
2991 ready->vec[ready->first - i] = ready->vec[ready->first - i - 1];
2992 QUEUE_INDEX (t) = QUEUE_NOWHERE;
2993 return t;
2996 /* Remove INSN from the ready list. */
2997 static void
2998 ready_remove_insn (rtx_insn *insn)
3000 int i;
3002 for (i = 0; i < readyp->n_ready; i++)
3003 if (ready_element (readyp, i) == insn)
3005 ready_remove (readyp, i);
3006 return;
3008 gcc_unreachable ();
3011 /* Calculate difference of two statistics set WAS and NOW.
3012 Result returned in WAS. */
3013 static void
3014 rank_for_schedule_stats_diff (rank_for_schedule_stats_t *was,
3015 const rank_for_schedule_stats_t *now)
3017 for (int i = 0; i < RFS_N; ++i)
3018 was->stats[i] = now->stats[i] - was->stats[i];
3021 /* Print rank_for_schedule statistics. */
3022 static void
3023 print_rank_for_schedule_stats (const char *prefix,
3024 const rank_for_schedule_stats_t *stats,
3025 struct ready_list *ready)
3027 for (int i = 0; i < RFS_N; ++i)
3028 if (stats->stats[i])
3030 fprintf (sched_dump, "%s%20s: %u", prefix, rfs_str[i], stats->stats[i]);
3032 if (ready != NULL)
3033 /* Print out insns that won due to RFS_<I>. */
3035 rtx_insn **p = ready_lastpos (ready);
3037 fprintf (sched_dump, ":");
3038 /* Start with 1 since least-priority insn didn't have any wins. */
3039 for (int j = 1; j < ready->n_ready; ++j)
3040 if (INSN_LAST_RFS_WIN (p[j]) == i)
3041 fprintf (sched_dump, " %s",
3042 (*current_sched_info->print_insn) (p[j], 0));
3044 fprintf (sched_dump, "\n");
3048 /* Separate DEBUG_INSNS from normal insns. DEBUG_INSNs go to the end
3049 of array. */
3050 static void
3051 ready_sort_debug (struct ready_list *ready)
3053 int i;
3054 rtx_insn **first = ready_lastpos (ready);
3056 for (i = 0; i < ready->n_ready; ++i)
3057 if (!DEBUG_INSN_P (first[i]))
3058 INSN_RFS_DEBUG_ORIG_ORDER (first[i]) = i;
3060 qsort (first, ready->n_ready, sizeof (rtx), rank_for_schedule_debug);
3063 /* Sort non-debug insns in the ready list READY by ascending priority.
3064 Assumes that all debug insns are separated from the real insns. */
3065 static void
3066 ready_sort_real (struct ready_list *ready)
3068 int i;
3069 rtx_insn **first = ready_lastpos (ready);
3070 int n_ready_real = ready->n_ready - ready->n_debug;
3072 if (sched_pressure == SCHED_PRESSURE_WEIGHTED)
3073 for (i = 0; i < n_ready_real; ++i)
3074 setup_insn_reg_pressure_info (first[i]);
3075 else if (sched_pressure == SCHED_PRESSURE_MODEL
3076 && model_curr_point < model_num_insns)
3077 model_set_excess_costs (first, n_ready_real);
3079 rank_for_schedule_stats_t stats1;
3080 if (sched_verbose >= 4)
3081 stats1 = rank_for_schedule_stats;
3083 if (n_ready_real == 2)
3084 swap_sort (first, n_ready_real);
3085 else if (n_ready_real > 2)
3086 qsort (first, n_ready_real, sizeof (rtx), rank_for_schedule);
3088 if (sched_verbose >= 4)
3090 rank_for_schedule_stats_diff (&stats1, &rank_for_schedule_stats);
3091 print_rank_for_schedule_stats (";;\t\t", &stats1, ready);
3095 /* Sort the ready list READY by ascending priority. */
3096 static void
3097 ready_sort (struct ready_list *ready)
3099 if (ready->n_debug > 0)
3100 ready_sort_debug (ready);
3101 else
3102 ready_sort_real (ready);
3105 /* PREV is an insn that is ready to execute. Adjust its priority if that
3106 will help shorten or lengthen register lifetimes as appropriate. Also
3107 provide a hook for the target to tweak itself. */
3109 HAIFA_INLINE static void
3110 adjust_priority (rtx_insn *prev)
3112 /* ??? There used to be code here to try and estimate how an insn
3113 affected register lifetimes, but it did it by looking at REG_DEAD
3114 notes, which we removed in schedule_region. Nor did it try to
3115 take into account register pressure or anything useful like that.
3117 Revisit when we have a machine model to work with and not before. */
3119 if (targetm.sched.adjust_priority)
3120 INSN_PRIORITY (prev) =
3121 targetm.sched.adjust_priority (prev, INSN_PRIORITY (prev));
3124 /* Advance DFA state STATE on one cycle. */
3125 void
3126 advance_state (state_t state)
3128 if (targetm.sched.dfa_pre_advance_cycle)
3129 targetm.sched.dfa_pre_advance_cycle ();
3131 if (targetm.sched.dfa_pre_cycle_insn)
3132 state_transition (state,
3133 targetm.sched.dfa_pre_cycle_insn ());
3135 state_transition (state, NULL);
3137 if (targetm.sched.dfa_post_cycle_insn)
3138 state_transition (state,
3139 targetm.sched.dfa_post_cycle_insn ());
3141 if (targetm.sched.dfa_post_advance_cycle)
3142 targetm.sched.dfa_post_advance_cycle ();
3145 /* Advance time on one cycle. */
3146 HAIFA_INLINE static void
3147 advance_one_cycle (void)
3149 advance_state (curr_state);
3150 if (sched_verbose >= 4)
3151 fprintf (sched_dump, ";;\tAdvance the current state.\n");
3154 /* Update register pressure after scheduling INSN. */
3155 static void
3156 update_register_pressure (rtx_insn *insn)
3158 struct reg_use_data *use;
3159 struct reg_set_data *set;
3161 gcc_checking_assert (!DEBUG_INSN_P (insn));
3163 for (use = INSN_REG_USE_LIST (insn); use != NULL; use = use->next_insn_use)
3164 if (dying_use_p (use))
3165 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure,
3166 use->regno, false);
3167 for (set = INSN_REG_SET_LIST (insn); set != NULL; set = set->next_insn_set)
3168 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure,
3169 set->regno, true);
3172 /* Set up or update (if UPDATE_P) max register pressure (see its
3173 meaning in sched-int.h::_haifa_insn_data) for all current BB insns
3174 after insn AFTER. */
3175 static void
3176 setup_insn_max_reg_pressure (rtx_insn *after, bool update_p)
3178 int i, p;
3179 bool eq_p;
3180 rtx_insn *insn;
3181 static int max_reg_pressure[N_REG_CLASSES];
3183 save_reg_pressure ();
3184 for (i = 0; i < ira_pressure_classes_num; i++)
3185 max_reg_pressure[ira_pressure_classes[i]]
3186 = curr_reg_pressure[ira_pressure_classes[i]];
3187 for (insn = NEXT_INSN (after);
3188 insn != NULL_RTX && ! BARRIER_P (insn)
3189 && BLOCK_FOR_INSN (insn) == BLOCK_FOR_INSN (after);
3190 insn = NEXT_INSN (insn))
3191 if (NONDEBUG_INSN_P (insn))
3193 eq_p = true;
3194 for (i = 0; i < ira_pressure_classes_num; i++)
3196 p = max_reg_pressure[ira_pressure_classes[i]];
3197 if (INSN_MAX_REG_PRESSURE (insn)[i] != p)
3199 eq_p = false;
3200 INSN_MAX_REG_PRESSURE (insn)[i]
3201 = max_reg_pressure[ira_pressure_classes[i]];
3204 if (update_p && eq_p)
3205 break;
3206 update_register_pressure (insn);
3207 for (i = 0; i < ira_pressure_classes_num; i++)
3208 if (max_reg_pressure[ira_pressure_classes[i]]
3209 < curr_reg_pressure[ira_pressure_classes[i]])
3210 max_reg_pressure[ira_pressure_classes[i]]
3211 = curr_reg_pressure[ira_pressure_classes[i]];
3213 restore_reg_pressure ();
3216 /* Update the current register pressure after scheduling INSN. Update
3217 also max register pressure for unscheduled insns of the current
3218 BB. */
3219 static void
3220 update_reg_and_insn_max_reg_pressure (rtx_insn *insn)
3222 int i;
3223 int before[N_REG_CLASSES];
3225 for (i = 0; i < ira_pressure_classes_num; i++)
3226 before[i] = curr_reg_pressure[ira_pressure_classes[i]];
3227 update_register_pressure (insn);
3228 for (i = 0; i < ira_pressure_classes_num; i++)
3229 if (curr_reg_pressure[ira_pressure_classes[i]] != before[i])
3230 break;
3231 if (i < ira_pressure_classes_num)
3232 setup_insn_max_reg_pressure (insn, true);
3235 /* Set up register pressure at the beginning of basic block BB whose
3236 insns starting after insn AFTER. Set up also max register pressure
3237 for all insns of the basic block. */
3238 void
3239 sched_setup_bb_reg_pressure_info (basic_block bb, rtx_insn *after)
3241 gcc_assert (sched_pressure == SCHED_PRESSURE_WEIGHTED);
3242 initiate_bb_reg_pressure_info (bb);
3243 setup_insn_max_reg_pressure (after, false);
3246 /* If doing predication while scheduling, verify whether INSN, which
3247 has just been scheduled, clobbers the conditions of any
3248 instructions that must be predicated in order to break their
3249 dependencies. If so, remove them from the queues so that they will
3250 only be scheduled once their control dependency is resolved. */
3252 static void
3253 check_clobbered_conditions (rtx_insn *insn)
3255 HARD_REG_SET t;
3256 int i;
3258 if ((current_sched_info->flags & DO_PREDICATION) == 0)
3259 return;
3261 find_all_hard_reg_sets (insn, &t, true);
3263 restart:
3264 for (i = 0; i < ready.n_ready; i++)
3266 rtx_insn *x = ready_element (&ready, i);
3267 if (TODO_SPEC (x) == DEP_CONTROL && cond_clobbered_p (x, t))
3269 ready_remove_insn (x);
3270 goto restart;
3273 for (i = 0; i <= max_insn_queue_index; i++)
3275 rtx_insn_list *link;
3276 int q = NEXT_Q_AFTER (q_ptr, i);
3278 restart_queue:
3279 for (link = insn_queue[q]; link; link = link->next ())
3281 rtx_insn *x = link->insn ();
3282 if (TODO_SPEC (x) == DEP_CONTROL && cond_clobbered_p (x, t))
3284 queue_remove (x);
3285 goto restart_queue;
3291 /* Return (in order):
3293 - positive if INSN adversely affects the pressure on one
3294 register class
3296 - negative if INSN reduces the pressure on one register class
3298 - 0 if INSN doesn't affect the pressure on any register class. */
3300 static int
3301 model_classify_pressure (struct model_insn_info *insn)
3303 struct reg_pressure_data *reg_pressure;
3304 int death[N_REG_CLASSES];
3305 int pci, cl, sum;
3307 calculate_reg_deaths (insn->insn, death);
3308 reg_pressure = INSN_REG_PRESSURE (insn->insn);
3309 sum = 0;
3310 for (pci = 0; pci < ira_pressure_classes_num; pci++)
3312 cl = ira_pressure_classes[pci];
3313 if (death[cl] < reg_pressure[pci].set_increase)
3314 return 1;
3315 sum += reg_pressure[pci].set_increase - death[cl];
3317 return sum;
3320 /* Return true if INSN1 should come before INSN2 in the model schedule. */
3322 static int
3323 model_order_p (struct model_insn_info *insn1, struct model_insn_info *insn2)
3325 unsigned int height1, height2;
3326 unsigned int priority1, priority2;
3328 /* Prefer instructions with a higher model priority. */
3329 if (insn1->model_priority != insn2->model_priority)
3330 return insn1->model_priority > insn2->model_priority;
3332 /* Combine the length of the longest path of satisfied true dependencies
3333 that leads to each instruction (depth) with the length of the longest
3334 path of any dependencies that leads from the instruction (alap).
3335 Prefer instructions with the greatest combined length. If the combined
3336 lengths are equal, prefer instructions with the greatest depth.
3338 The idea is that, if we have a set S of "equal" instructions that each
3339 have ALAP value X, and we pick one such instruction I, any true-dependent
3340 successors of I that have ALAP value X - 1 should be preferred over S.
3341 This encourages the schedule to be "narrow" rather than "wide".
3342 However, if I is a low-priority instruction that we decided to
3343 schedule because of its model_classify_pressure, and if there
3344 is a set of higher-priority instructions T, the aforementioned
3345 successors of I should not have the edge over T. */
3346 height1 = insn1->depth + insn1->alap;
3347 height2 = insn2->depth + insn2->alap;
3348 if (height1 != height2)
3349 return height1 > height2;
3350 if (insn1->depth != insn2->depth)
3351 return insn1->depth > insn2->depth;
3353 /* We have no real preference between INSN1 an INSN2 as far as attempts
3354 to reduce pressure go. Prefer instructions with higher priorities. */
3355 priority1 = INSN_PRIORITY (insn1->insn);
3356 priority2 = INSN_PRIORITY (insn2->insn);
3357 if (priority1 != priority2)
3358 return priority1 > priority2;
3360 /* Use the original rtl sequence as a tie-breaker. */
3361 return insn1 < insn2;
3364 /* Add INSN to the model worklist immediately after PREV. Add it to the
3365 beginning of the list if PREV is null. */
3367 static void
3368 model_add_to_worklist_at (struct model_insn_info *insn,
3369 struct model_insn_info *prev)
3371 gcc_assert (QUEUE_INDEX (insn->insn) == QUEUE_NOWHERE);
3372 QUEUE_INDEX (insn->insn) = QUEUE_READY;
3374 insn->prev = prev;
3375 if (prev)
3377 insn->next = prev->next;
3378 prev->next = insn;
3380 else
3382 insn->next = model_worklist;
3383 model_worklist = insn;
3385 if (insn->next)
3386 insn->next->prev = insn;
3389 /* Remove INSN from the model worklist. */
3391 static void
3392 model_remove_from_worklist (struct model_insn_info *insn)
3394 gcc_assert (QUEUE_INDEX (insn->insn) == QUEUE_READY);
3395 QUEUE_INDEX (insn->insn) = QUEUE_NOWHERE;
3397 if (insn->prev)
3398 insn->prev->next = insn->next;
3399 else
3400 model_worklist = insn->next;
3401 if (insn->next)
3402 insn->next->prev = insn->prev;
3405 /* Add INSN to the model worklist. Start looking for a suitable position
3406 between neighbors PREV and NEXT, testing at most MAX_SCHED_READY_INSNS
3407 insns either side. A null PREV indicates the beginning of the list and
3408 a null NEXT indicates the end. */
3410 static void
3411 model_add_to_worklist (struct model_insn_info *insn,
3412 struct model_insn_info *prev,
3413 struct model_insn_info *next)
3415 int count;
3417 count = MAX_SCHED_READY_INSNS;
3418 if (count > 0 && prev && model_order_p (insn, prev))
3421 count--;
3422 prev = prev->prev;
3424 while (count > 0 && prev && model_order_p (insn, prev));
3425 else
3426 while (count > 0 && next && model_order_p (next, insn))
3428 count--;
3429 prev = next;
3430 next = next->next;
3432 model_add_to_worklist_at (insn, prev);
3435 /* INSN may now have a higher priority (in the model_order_p sense)
3436 than before. Move it up the worklist if necessary. */
3438 static void
3439 model_promote_insn (struct model_insn_info *insn)
3441 struct model_insn_info *prev;
3442 int count;
3444 prev = insn->prev;
3445 count = MAX_SCHED_READY_INSNS;
3446 while (count > 0 && prev && model_order_p (insn, prev))
3448 count--;
3449 prev = prev->prev;
3451 if (prev != insn->prev)
3453 model_remove_from_worklist (insn);
3454 model_add_to_worklist_at (insn, prev);
3458 /* Add INSN to the end of the model schedule. */
3460 static void
3461 model_add_to_schedule (rtx_insn *insn)
3463 unsigned int point;
3465 gcc_assert (QUEUE_INDEX (insn) == QUEUE_NOWHERE);
3466 QUEUE_INDEX (insn) = QUEUE_SCHEDULED;
3468 point = model_schedule.length ();
3469 model_schedule.quick_push (insn);
3470 INSN_MODEL_INDEX (insn) = point + 1;
3473 /* Analyze the instructions that are to be scheduled, setting up
3474 MODEL_INSN_INFO (...) and model_num_insns accordingly. Add ready
3475 instructions to model_worklist. */
3477 static void
3478 model_analyze_insns (void)
3480 rtx_insn *start, *end, *iter;
3481 sd_iterator_def sd_it;
3482 dep_t dep;
3483 struct model_insn_info *insn, *con;
3485 model_num_insns = 0;
3486 start = PREV_INSN (current_sched_info->next_tail);
3487 end = current_sched_info->prev_head;
3488 for (iter = start; iter != end; iter = PREV_INSN (iter))
3489 if (NONDEBUG_INSN_P (iter))
3491 insn = MODEL_INSN_INFO (iter);
3492 insn->insn = iter;
3493 FOR_EACH_DEP (iter, SD_LIST_FORW, sd_it, dep)
3495 con = MODEL_INSN_INFO (DEP_CON (dep));
3496 if (con->insn && insn->alap < con->alap + 1)
3497 insn->alap = con->alap + 1;
3500 insn->old_queue = QUEUE_INDEX (iter);
3501 QUEUE_INDEX (iter) = QUEUE_NOWHERE;
3503 insn->unscheduled_preds = dep_list_size (iter, SD_LIST_HARD_BACK);
3504 if (insn->unscheduled_preds == 0)
3505 model_add_to_worklist (insn, NULL, model_worklist);
3507 model_num_insns++;
3511 /* The global state describes the register pressure at the start of the
3512 model schedule. Initialize GROUP accordingly. */
3514 static void
3515 model_init_pressure_group (struct model_pressure_group *group)
3517 int pci, cl;
3519 for (pci = 0; pci < ira_pressure_classes_num; pci++)
3521 cl = ira_pressure_classes[pci];
3522 group->limits[pci].pressure = curr_reg_pressure[cl];
3523 group->limits[pci].point = 0;
3525 /* Use index model_num_insns to record the state after the last
3526 instruction in the model schedule. */
3527 group->model = XNEWVEC (struct model_pressure_data,
3528 (model_num_insns + 1) * ira_pressure_classes_num);
3531 /* Record that MODEL_REF_PRESSURE (GROUP, POINT, PCI) is PRESSURE.
3532 Update the maximum pressure for the whole schedule. */
3534 static void
3535 model_record_pressure (struct model_pressure_group *group,
3536 int point, int pci, int pressure)
3538 MODEL_REF_PRESSURE (group, point, pci) = pressure;
3539 if (group->limits[pci].pressure < pressure)
3541 group->limits[pci].pressure = pressure;
3542 group->limits[pci].point = point;
3546 /* INSN has just been added to the end of the model schedule. Record its
3547 register-pressure information. */
3549 static void
3550 model_record_pressures (struct model_insn_info *insn)
3552 struct reg_pressure_data *reg_pressure;
3553 int point, pci, cl, delta;
3554 int death[N_REG_CLASSES];
3556 point = model_index (insn->insn);
3557 if (sched_verbose >= 2)
3559 if (point == 0)
3561 fprintf (sched_dump, "\n;;\tModel schedule:\n;;\n");
3562 fprintf (sched_dump, ";;\t| idx insn | mpri hght dpth prio |\n");
3564 fprintf (sched_dump, ";;\t| %3d %4d | %4d %4d %4d %4d | %-30s ",
3565 point, INSN_UID (insn->insn), insn->model_priority,
3566 insn->depth + insn->alap, insn->depth,
3567 INSN_PRIORITY (insn->insn),
3568 str_pattern_slim (PATTERN (insn->insn)));
3570 calculate_reg_deaths (insn->insn, death);
3571 reg_pressure = INSN_REG_PRESSURE (insn->insn);
3572 for (pci = 0; pci < ira_pressure_classes_num; pci++)
3574 cl = ira_pressure_classes[pci];
3575 delta = reg_pressure[pci].set_increase - death[cl];
3576 if (sched_verbose >= 2)
3577 fprintf (sched_dump, " %s:[%d,%+d]", reg_class_names[cl],
3578 curr_reg_pressure[cl], delta);
3579 model_record_pressure (&model_before_pressure, point, pci,
3580 curr_reg_pressure[cl]);
3582 if (sched_verbose >= 2)
3583 fprintf (sched_dump, "\n");
3586 /* All instructions have been added to the model schedule. Record the
3587 final register pressure in GROUP and set up all MODEL_MAX_PRESSUREs. */
3589 static void
3590 model_record_final_pressures (struct model_pressure_group *group)
3592 int point, pci, max_pressure, ref_pressure, cl;
3594 for (pci = 0; pci < ira_pressure_classes_num; pci++)
3596 /* Record the final pressure for this class. */
3597 cl = ira_pressure_classes[pci];
3598 point = model_num_insns;
3599 ref_pressure = curr_reg_pressure[cl];
3600 model_record_pressure (group, point, pci, ref_pressure);
3602 /* Record the original maximum pressure. */
3603 group->limits[pci].orig_pressure = group->limits[pci].pressure;
3605 /* Update the MODEL_MAX_PRESSURE for every point of the schedule. */
3606 max_pressure = ref_pressure;
3607 MODEL_MAX_PRESSURE (group, point, pci) = max_pressure;
3608 while (point > 0)
3610 point--;
3611 ref_pressure = MODEL_REF_PRESSURE (group, point, pci);
3612 max_pressure = MAX (max_pressure, ref_pressure);
3613 MODEL_MAX_PRESSURE (group, point, pci) = max_pressure;
3618 /* Update all successors of INSN, given that INSN has just been scheduled. */
3620 static void
3621 model_add_successors_to_worklist (struct model_insn_info *insn)
3623 sd_iterator_def sd_it;
3624 struct model_insn_info *con;
3625 dep_t dep;
3627 FOR_EACH_DEP (insn->insn, SD_LIST_FORW, sd_it, dep)
3629 con = MODEL_INSN_INFO (DEP_CON (dep));
3630 /* Ignore debug instructions, and instructions from other blocks. */
3631 if (con->insn)
3633 con->unscheduled_preds--;
3635 /* Update the depth field of each true-dependent successor.
3636 Increasing the depth gives them a higher priority than
3637 before. */
3638 if (DEP_TYPE (dep) == REG_DEP_TRUE && con->depth < insn->depth + 1)
3640 con->depth = insn->depth + 1;
3641 if (QUEUE_INDEX (con->insn) == QUEUE_READY)
3642 model_promote_insn (con);
3645 /* If this is a true dependency, or if there are no remaining
3646 dependencies for CON (meaning that CON only had non-true
3647 dependencies), make sure that CON is on the worklist.
3648 We don't bother otherwise because it would tend to fill the
3649 worklist with a lot of low-priority instructions that are not
3650 yet ready to issue. */
3651 if ((con->depth > 0 || con->unscheduled_preds == 0)
3652 && QUEUE_INDEX (con->insn) == QUEUE_NOWHERE)
3653 model_add_to_worklist (con, insn, insn->next);
3658 /* Give INSN a higher priority than any current instruction, then give
3659 unscheduled predecessors of INSN a higher priority still. If any of
3660 those predecessors are not on the model worklist, do the same for its
3661 predecessors, and so on. */
3663 static void
3664 model_promote_predecessors (struct model_insn_info *insn)
3666 struct model_insn_info *pro, *first;
3667 sd_iterator_def sd_it;
3668 dep_t dep;
3670 if (sched_verbose >= 7)
3671 fprintf (sched_dump, ";;\t+--- priority of %d = %d, priority of",
3672 INSN_UID (insn->insn), model_next_priority);
3673 insn->model_priority = model_next_priority++;
3674 model_remove_from_worklist (insn);
3675 model_add_to_worklist_at (insn, NULL);
3677 first = NULL;
3678 for (;;)
3680 FOR_EACH_DEP (insn->insn, SD_LIST_HARD_BACK, sd_it, dep)
3682 pro = MODEL_INSN_INFO (DEP_PRO (dep));
3683 /* The first test is to ignore debug instructions, and instructions
3684 from other blocks. */
3685 if (pro->insn
3686 && pro->model_priority != model_next_priority
3687 && QUEUE_INDEX (pro->insn) != QUEUE_SCHEDULED)
3689 pro->model_priority = model_next_priority;
3690 if (sched_verbose >= 7)
3691 fprintf (sched_dump, " %d", INSN_UID (pro->insn));
3692 if (QUEUE_INDEX (pro->insn) == QUEUE_READY)
3694 /* PRO is already in the worklist, but it now has
3695 a higher priority than before. Move it at the
3696 appropriate place. */
3697 model_remove_from_worklist (pro);
3698 model_add_to_worklist (pro, NULL, model_worklist);
3700 else
3702 /* PRO isn't in the worklist. Recursively process
3703 its predecessors until we find one that is. */
3704 pro->next = first;
3705 first = pro;
3709 if (!first)
3710 break;
3711 insn = first;
3712 first = insn->next;
3714 if (sched_verbose >= 7)
3715 fprintf (sched_dump, " = %d\n", model_next_priority);
3716 model_next_priority++;
3719 /* Pick one instruction from model_worklist and process it. */
3721 static void
3722 model_choose_insn (void)
3724 struct model_insn_info *insn, *fallback;
3725 int count;
3727 if (sched_verbose >= 7)
3729 fprintf (sched_dump, ";;\t+--- worklist:\n");
3730 insn = model_worklist;
3731 count = MAX_SCHED_READY_INSNS;
3732 while (count > 0 && insn)
3734 fprintf (sched_dump, ";;\t+--- %d [%d, %d, %d, %d]\n",
3735 INSN_UID (insn->insn), insn->model_priority,
3736 insn->depth + insn->alap, insn->depth,
3737 INSN_PRIORITY (insn->insn));
3738 count--;
3739 insn = insn->next;
3743 /* Look for a ready instruction whose model_classify_priority is zero
3744 or negative, picking the highest-priority one. Adding such an
3745 instruction to the schedule now should do no harm, and may actually
3746 do some good.
3748 Failing that, see whether there is an instruction with the highest
3749 extant model_priority that is not yet ready, but which would reduce
3750 pressure if it became ready. This is designed to catch cases like:
3752 (set (mem (reg R1)) (reg R2))
3754 where the instruction is the last remaining use of R1 and where the
3755 value of R2 is not yet available (or vice versa). The death of R1
3756 means that this instruction already reduces pressure. It is of
3757 course possible that the computation of R2 involves other registers
3758 that are hard to kill, but such cases are rare enough for this
3759 heuristic to be a win in general.
3761 Failing that, just pick the highest-priority instruction in the
3762 worklist. */
3763 count = MAX_SCHED_READY_INSNS;
3764 insn = model_worklist;
3765 fallback = 0;
3766 for (;;)
3768 if (count == 0 || !insn)
3770 insn = fallback ? fallback : model_worklist;
3771 break;
3773 if (insn->unscheduled_preds)
3775 if (model_worklist->model_priority == insn->model_priority
3776 && !fallback
3777 && model_classify_pressure (insn) < 0)
3778 fallback = insn;
3780 else
3782 if (model_classify_pressure (insn) <= 0)
3783 break;
3785 count--;
3786 insn = insn->next;
3789 if (sched_verbose >= 7 && insn != model_worklist)
3791 if (insn->unscheduled_preds)
3792 fprintf (sched_dump, ";;\t+--- promoting insn %d, with dependencies\n",
3793 INSN_UID (insn->insn));
3794 else
3795 fprintf (sched_dump, ";;\t+--- promoting insn %d, which is ready\n",
3796 INSN_UID (insn->insn));
3798 if (insn->unscheduled_preds)
3799 /* INSN isn't yet ready to issue. Give all its predecessors the
3800 highest priority. */
3801 model_promote_predecessors (insn);
3802 else
3804 /* INSN is ready. Add it to the end of model_schedule and
3805 process its successors. */
3806 model_add_successors_to_worklist (insn);
3807 model_remove_from_worklist (insn);
3808 model_add_to_schedule (insn->insn);
3809 model_record_pressures (insn);
3810 update_register_pressure (insn->insn);
3814 /* Restore all QUEUE_INDEXs to the values that they had before
3815 model_start_schedule was called. */
3817 static void
3818 model_reset_queue_indices (void)
3820 unsigned int i;
3821 rtx_insn *insn;
3823 FOR_EACH_VEC_ELT (model_schedule, i, insn)
3824 QUEUE_INDEX (insn) = MODEL_INSN_INFO (insn)->old_queue;
3827 /* We have calculated the model schedule and spill costs. Print a summary
3828 to sched_dump. */
3830 static void
3831 model_dump_pressure_summary (void)
3833 int pci, cl;
3835 fprintf (sched_dump, ";; Pressure summary:");
3836 for (pci = 0; pci < ira_pressure_classes_num; pci++)
3838 cl = ira_pressure_classes[pci];
3839 fprintf (sched_dump, " %s:%d", reg_class_names[cl],
3840 model_before_pressure.limits[pci].pressure);
3842 fprintf (sched_dump, "\n\n");
3845 /* Initialize the SCHED_PRESSURE_MODEL information for the current
3846 scheduling region. */
3848 static void
3849 model_start_schedule (basic_block bb)
3851 model_next_priority = 1;
3852 model_schedule.create (sched_max_luid);
3853 model_insns = XCNEWVEC (struct model_insn_info, sched_max_luid);
3855 gcc_assert (bb == BLOCK_FOR_INSN (NEXT_INSN (current_sched_info->prev_head)));
3856 initiate_reg_pressure_info (df_get_live_in (bb));
3858 model_analyze_insns ();
3859 model_init_pressure_group (&model_before_pressure);
3860 while (model_worklist)
3861 model_choose_insn ();
3862 gcc_assert (model_num_insns == (int) model_schedule.length ());
3863 if (sched_verbose >= 2)
3864 fprintf (sched_dump, "\n");
3866 model_record_final_pressures (&model_before_pressure);
3867 model_reset_queue_indices ();
3869 XDELETEVEC (model_insns);
3871 model_curr_point = 0;
3872 initiate_reg_pressure_info (df_get_live_in (bb));
3873 if (sched_verbose >= 1)
3874 model_dump_pressure_summary ();
3877 /* Free the information associated with GROUP. */
3879 static void
3880 model_finalize_pressure_group (struct model_pressure_group *group)
3882 XDELETEVEC (group->model);
3885 /* Free the information created by model_start_schedule. */
3887 static void
3888 model_end_schedule (void)
3890 model_finalize_pressure_group (&model_before_pressure);
3891 model_schedule.release ();
3894 /* Prepare reg pressure scheduling for basic block BB. */
3895 static void
3896 sched_pressure_start_bb (basic_block bb)
3898 /* Set the number of available registers for each class taking into account
3899 relative probability of current basic block versus function prologue and
3900 epilogue.
3901 * If the basic block executes much more often than the prologue/epilogue
3902 (e.g., inside a hot loop), then cost of spill in the prologue is close to
3903 nil, so the effective number of available registers is
3904 (ira_class_hard_regs_num[cl] - fixed_regs_num[cl] - 0).
3905 * If the basic block executes as often as the prologue/epilogue,
3906 then spill in the block is as costly as in the prologue, so the effective
3907 number of available registers is
3908 (ira_class_hard_regs_num[cl] - fixed_regs_num[cl]
3909 - call_saved_regs_num[cl]).
3910 Note that all-else-equal, we prefer to spill in the prologue, since that
3911 allows "extra" registers for other basic blocks of the function.
3912 * If the basic block is on the cold path of the function and executes
3913 rarely, then we should always prefer to spill in the block, rather than
3914 in the prologue/epilogue. The effective number of available register is
3915 (ira_class_hard_regs_num[cl] - fixed_regs_num[cl]
3916 - call_saved_regs_num[cl]). */
3918 int i;
3919 int entry_freq = ENTRY_BLOCK_PTR_FOR_FN (cfun)->count.to_frequency (cfun);
3920 int bb_freq = bb->count.to_frequency (cfun);
3922 if (bb_freq == 0)
3924 if (entry_freq == 0)
3925 entry_freq = bb_freq = 1;
3927 if (bb_freq < entry_freq)
3928 bb_freq = entry_freq;
3930 for (i = 0; i < ira_pressure_classes_num; ++i)
3932 enum reg_class cl = ira_pressure_classes[i];
3933 sched_class_regs_num[cl] = ira_class_hard_regs_num[cl]
3934 - fixed_regs_num[cl];
3935 sched_class_regs_num[cl]
3936 -= (call_saved_regs_num[cl] * entry_freq) / bb_freq;
3940 if (sched_pressure == SCHED_PRESSURE_MODEL)
3941 model_start_schedule (bb);
3944 /* A structure that holds local state for the loop in schedule_block. */
3945 struct sched_block_state
3947 /* True if no real insns have been scheduled in the current cycle. */
3948 bool first_cycle_insn_p;
3949 /* True if a shadow insn has been scheduled in the current cycle, which
3950 means that no more normal insns can be issued. */
3951 bool shadows_only_p;
3952 /* True if we're winding down a modulo schedule, which means that we only
3953 issue insns with INSN_EXACT_TICK set. */
3954 bool modulo_epilogue;
3955 /* Initialized with the machine's issue rate every cycle, and updated
3956 by calls to the variable_issue hook. */
3957 int can_issue_more;
3960 /* INSN is the "currently executing insn". Launch each insn which was
3961 waiting on INSN. READY is the ready list which contains the insns
3962 that are ready to fire. CLOCK is the current cycle. The function
3963 returns necessary cycle advance after issuing the insn (it is not
3964 zero for insns in a schedule group). */
3966 static int
3967 schedule_insn (rtx_insn *insn)
3969 sd_iterator_def sd_it;
3970 dep_t dep;
3971 int i;
3972 int advance = 0;
3974 if (sched_verbose >= 1)
3976 struct reg_pressure_data *pressure_info;
3977 fprintf (sched_dump, ";;\t%3i--> %s %-40s:",
3978 clock_var, (*current_sched_info->print_insn) (insn, 1),
3979 str_pattern_slim (PATTERN (insn)));
3981 if (recog_memoized (insn) < 0)
3982 fprintf (sched_dump, "nothing");
3983 else
3984 print_reservation (sched_dump, insn);
3985 pressure_info = INSN_REG_PRESSURE (insn);
3986 if (pressure_info != NULL)
3988 fputc (':', sched_dump);
3989 for (i = 0; i < ira_pressure_classes_num; i++)
3990 fprintf (sched_dump, "%s%s%+d(%d)",
3991 scheduled_insns.length () > 1
3992 && INSN_LUID (insn)
3993 < INSN_LUID (scheduled_insns[scheduled_insns.length () - 2]) ? "@" : "",
3994 reg_class_names[ira_pressure_classes[i]],
3995 pressure_info[i].set_increase, pressure_info[i].change);
3997 if (sched_pressure == SCHED_PRESSURE_MODEL
3998 && model_curr_point < model_num_insns
3999 && model_index (insn) == model_curr_point)
4000 fprintf (sched_dump, ":model %d", model_curr_point);
4001 fputc ('\n', sched_dump);
4004 if (sched_pressure == SCHED_PRESSURE_WEIGHTED && !DEBUG_INSN_P (insn))
4005 update_reg_and_insn_max_reg_pressure (insn);
4007 /* Scheduling instruction should have all its dependencies resolved and
4008 should have been removed from the ready list. */
4009 gcc_assert (sd_lists_empty_p (insn, SD_LIST_HARD_BACK));
4011 /* Reset debug insns invalidated by moving this insn. */
4012 if (MAY_HAVE_DEBUG_BIND_INSNS && !DEBUG_INSN_P (insn))
4013 for (sd_it = sd_iterator_start (insn, SD_LIST_BACK);
4014 sd_iterator_cond (&sd_it, &dep);)
4016 rtx_insn *dbg = DEP_PRO (dep);
4017 struct reg_use_data *use, *next;
4019 if (DEP_STATUS (dep) & DEP_CANCELLED)
4021 sd_iterator_next (&sd_it);
4022 continue;
4025 gcc_assert (DEBUG_BIND_INSN_P (dbg));
4027 if (sched_verbose >= 6)
4028 fprintf (sched_dump, ";;\t\tresetting: debug insn %d\n",
4029 INSN_UID (dbg));
4031 /* ??? Rather than resetting the debug insn, we might be able
4032 to emit a debug temp before the just-scheduled insn, but
4033 this would involve checking that the expression at the
4034 point of the debug insn is equivalent to the expression
4035 before the just-scheduled insn. They might not be: the
4036 expression in the debug insn may depend on other insns not
4037 yet scheduled that set MEMs, REGs or even other debug
4038 insns. It's not clear that attempting to preserve debug
4039 information in these cases is worth the effort, given how
4040 uncommon these resets are and the likelihood that the debug
4041 temps introduced won't survive the schedule change. */
4042 INSN_VAR_LOCATION_LOC (dbg) = gen_rtx_UNKNOWN_VAR_LOC ();
4043 df_insn_rescan (dbg);
4045 /* Unknown location doesn't use any registers. */
4046 for (use = INSN_REG_USE_LIST (dbg); use != NULL; use = next)
4048 struct reg_use_data *prev = use;
4050 /* Remove use from the cyclic next_regno_use chain first. */
4051 while (prev->next_regno_use != use)
4052 prev = prev->next_regno_use;
4053 prev->next_regno_use = use->next_regno_use;
4054 next = use->next_insn_use;
4055 free (use);
4057 INSN_REG_USE_LIST (dbg) = NULL;
4059 /* We delete rather than resolve these deps, otherwise we
4060 crash in sched_free_deps(), because forward deps are
4061 expected to be released before backward deps. */
4062 sd_delete_dep (sd_it);
4065 gcc_assert (QUEUE_INDEX (insn) == QUEUE_NOWHERE);
4066 QUEUE_INDEX (insn) = QUEUE_SCHEDULED;
4068 if (sched_pressure == SCHED_PRESSURE_MODEL
4069 && model_curr_point < model_num_insns
4070 && NONDEBUG_INSN_P (insn))
4072 if (model_index (insn) == model_curr_point)
4074 model_curr_point++;
4075 while (model_curr_point < model_num_insns
4076 && (QUEUE_INDEX (MODEL_INSN (model_curr_point))
4077 == QUEUE_SCHEDULED));
4078 else
4079 model_recompute (insn);
4080 model_update_limit_points ();
4081 update_register_pressure (insn);
4082 if (sched_verbose >= 2)
4083 print_curr_reg_pressure ();
4086 gcc_assert (INSN_TICK (insn) >= MIN_TICK);
4087 if (INSN_TICK (insn) > clock_var)
4088 /* INSN has been prematurely moved from the queue to the ready list.
4089 This is possible only if following flags are set. */
4090 gcc_assert (flag_sched_stalled_insns || sched_fusion);
4092 /* ??? Probably, if INSN is scheduled prematurely, we should leave
4093 INSN_TICK untouched. This is a machine-dependent issue, actually. */
4094 INSN_TICK (insn) = clock_var;
4096 check_clobbered_conditions (insn);
4098 /* Update dependent instructions. First, see if by scheduling this insn
4099 now we broke a dependence in a way that requires us to change another
4100 insn. */
4101 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
4102 sd_iterator_cond (&sd_it, &dep); sd_iterator_next (&sd_it))
4104 struct dep_replacement *desc = DEP_REPLACE (dep);
4105 rtx_insn *pro = DEP_PRO (dep);
4106 if (QUEUE_INDEX (pro) != QUEUE_SCHEDULED
4107 && desc != NULL && desc->insn == pro)
4108 apply_replacement (dep, false);
4111 /* Go through and resolve forward dependencies. */
4112 for (sd_it = sd_iterator_start (insn, SD_LIST_FORW);
4113 sd_iterator_cond (&sd_it, &dep);)
4115 rtx_insn *next = DEP_CON (dep);
4116 bool cancelled = (DEP_STATUS (dep) & DEP_CANCELLED) != 0;
4118 /* Resolve the dependence between INSN and NEXT.
4119 sd_resolve_dep () moves current dep to another list thus
4120 advancing the iterator. */
4121 sd_resolve_dep (sd_it);
4123 if (cancelled)
4125 if (must_restore_pattern_p (next, dep))
4126 restore_pattern (dep, false);
4127 continue;
4130 /* Don't bother trying to mark next as ready if insn is a debug
4131 insn. If insn is the last hard dependency, it will have
4132 already been discounted. */
4133 if (DEBUG_INSN_P (insn) && !DEBUG_INSN_P (next))
4134 continue;
4136 if (!IS_SPECULATION_BRANCHY_CHECK_P (insn))
4138 int effective_cost;
4140 effective_cost = try_ready (next);
4142 if (effective_cost >= 0
4143 && SCHED_GROUP_P (next)
4144 && advance < effective_cost)
4145 advance = effective_cost;
4147 else
4148 /* Check always has only one forward dependence (to the first insn in
4149 the recovery block), therefore, this will be executed only once. */
4151 gcc_assert (sd_lists_empty_p (insn, SD_LIST_FORW));
4152 fix_recovery_deps (RECOVERY_BLOCK (insn));
4156 /* Annotate the instruction with issue information -- TImode
4157 indicates that the instruction is expected not to be able
4158 to issue on the same cycle as the previous insn. A machine
4159 may use this information to decide how the instruction should
4160 be aligned. */
4161 if (issue_rate > 1
4162 && GET_CODE (PATTERN (insn)) != USE
4163 && GET_CODE (PATTERN (insn)) != CLOBBER
4164 && !DEBUG_INSN_P (insn))
4166 if (reload_completed)
4167 PUT_MODE (insn, clock_var > last_clock_var ? TImode : VOIDmode);
4168 last_clock_var = clock_var;
4171 if (nonscheduled_insns_begin != NULL_RTX)
4172 /* Indicate to debug counters that INSN is scheduled. */
4173 nonscheduled_insns_begin = insn;
4175 return advance;
4178 /* Functions for handling of notes. */
4180 /* Add note list that ends on FROM_END to the end of TO_ENDP. */
4181 void
4182 concat_note_lists (rtx_insn *from_end, rtx_insn **to_endp)
4184 rtx_insn *from_start;
4186 /* It's easy when have nothing to concat. */
4187 if (from_end == NULL)
4188 return;
4190 /* It's also easy when destination is empty. */
4191 if (*to_endp == NULL)
4193 *to_endp = from_end;
4194 return;
4197 from_start = from_end;
4198 while (PREV_INSN (from_start) != NULL)
4199 from_start = PREV_INSN (from_start);
4201 SET_PREV_INSN (from_start) = *to_endp;
4202 SET_NEXT_INSN (*to_endp) = from_start;
4203 *to_endp = from_end;
4206 /* Delete notes between HEAD and TAIL and put them in the chain
4207 of notes ended by NOTE_LIST. */
4208 void
4209 remove_notes (rtx_insn *head, rtx_insn *tail)
4211 rtx_insn *next_tail, *insn, *next;
4213 note_list = 0;
4214 if (head == tail && !INSN_P (head))
4215 return;
4217 next_tail = NEXT_INSN (tail);
4218 for (insn = head; insn != next_tail; insn = next)
4220 next = NEXT_INSN (insn);
4221 if (!NOTE_P (insn))
4222 continue;
4224 switch (NOTE_KIND (insn))
4226 case NOTE_INSN_BASIC_BLOCK:
4227 continue;
4229 case NOTE_INSN_EPILOGUE_BEG:
4230 if (insn != tail)
4232 remove_insn (insn);
4233 add_reg_note (next, REG_SAVE_NOTE,
4234 GEN_INT (NOTE_INSN_EPILOGUE_BEG));
4235 break;
4237 /* FALLTHRU */
4239 default:
4240 remove_insn (insn);
4242 /* Add the note to list that ends at NOTE_LIST. */
4243 SET_PREV_INSN (insn) = note_list;
4244 SET_NEXT_INSN (insn) = NULL_RTX;
4245 if (note_list)
4246 SET_NEXT_INSN (note_list) = insn;
4247 note_list = insn;
4248 break;
4251 gcc_assert ((sel_sched_p () || insn != tail) && insn != head);
4255 /* A structure to record enough data to allow us to backtrack the scheduler to
4256 a previous state. */
4257 struct haifa_saved_data
4259 /* Next entry on the list. */
4260 struct haifa_saved_data *next;
4262 /* Backtracking is associated with scheduling insns that have delay slots.
4263 DELAY_PAIR points to the structure that contains the insns involved, and
4264 the number of cycles between them. */
4265 struct delay_pair *delay_pair;
4267 /* Data used by the frontend (e.g. sched-ebb or sched-rgn). */
4268 void *fe_saved_data;
4269 /* Data used by the backend. */
4270 void *be_saved_data;
4272 /* Copies of global state. */
4273 int clock_var, last_clock_var;
4274 struct ready_list ready;
4275 state_t curr_state;
4277 rtx_insn *last_scheduled_insn;
4278 rtx_insn *last_nondebug_scheduled_insn;
4279 rtx_insn *nonscheduled_insns_begin;
4280 int cycle_issued_insns;
4282 /* Copies of state used in the inner loop of schedule_block. */
4283 struct sched_block_state sched_block;
4285 /* We don't need to save q_ptr, as its value is arbitrary and we can set it
4286 to 0 when restoring. */
4287 int q_size;
4288 rtx_insn_list **insn_queue;
4290 /* Describe pattern replacements that occurred since this backtrack point
4291 was queued. */
4292 vec<dep_t> replacement_deps;
4293 vec<int> replace_apply;
4295 /* A copy of the next-cycle replacement vectors at the time of the backtrack
4296 point. */
4297 vec<dep_t> next_cycle_deps;
4298 vec<int> next_cycle_apply;
4301 /* A record, in reverse order, of all scheduled insns which have delay slots
4302 and may require backtracking. */
4303 static struct haifa_saved_data *backtrack_queue;
4305 /* For every dependency of INSN, set the FEEDS_BACKTRACK_INSN bit according
4306 to SET_P. */
4307 static void
4308 mark_backtrack_feeds (rtx_insn *insn, int set_p)
4310 sd_iterator_def sd_it;
4311 dep_t dep;
4312 FOR_EACH_DEP (insn, SD_LIST_HARD_BACK, sd_it, dep)
4314 FEEDS_BACKTRACK_INSN (DEP_PRO (dep)) = set_p;
4318 /* Save the current scheduler state so that we can backtrack to it
4319 later if necessary. PAIR gives the insns that make it necessary to
4320 save this point. SCHED_BLOCK is the local state of schedule_block
4321 that need to be saved. */
4322 static void
4323 save_backtrack_point (struct delay_pair *pair,
4324 struct sched_block_state sched_block)
4326 int i;
4327 struct haifa_saved_data *save = XNEW (struct haifa_saved_data);
4329 save->curr_state = xmalloc (dfa_state_size);
4330 memcpy (save->curr_state, curr_state, dfa_state_size);
4332 save->ready.first = ready.first;
4333 save->ready.n_ready = ready.n_ready;
4334 save->ready.n_debug = ready.n_debug;
4335 save->ready.veclen = ready.veclen;
4336 save->ready.vec = XNEWVEC (rtx_insn *, ready.veclen);
4337 memcpy (save->ready.vec, ready.vec, ready.veclen * sizeof (rtx));
4339 save->insn_queue = XNEWVEC (rtx_insn_list *, max_insn_queue_index + 1);
4340 save->q_size = q_size;
4341 for (i = 0; i <= max_insn_queue_index; i++)
4343 int q = NEXT_Q_AFTER (q_ptr, i);
4344 save->insn_queue[i] = copy_INSN_LIST (insn_queue[q]);
4347 save->clock_var = clock_var;
4348 save->last_clock_var = last_clock_var;
4349 save->cycle_issued_insns = cycle_issued_insns;
4350 save->last_scheduled_insn = last_scheduled_insn;
4351 save->last_nondebug_scheduled_insn = last_nondebug_scheduled_insn;
4352 save->nonscheduled_insns_begin = nonscheduled_insns_begin;
4354 save->sched_block = sched_block;
4356 save->replacement_deps.create (0);
4357 save->replace_apply.create (0);
4358 save->next_cycle_deps = next_cycle_replace_deps.copy ();
4359 save->next_cycle_apply = next_cycle_apply.copy ();
4361 if (current_sched_info->save_state)
4362 save->fe_saved_data = (*current_sched_info->save_state) ();
4364 if (targetm.sched.alloc_sched_context)
4366 save->be_saved_data = targetm.sched.alloc_sched_context ();
4367 targetm.sched.init_sched_context (save->be_saved_data, false);
4369 else
4370 save->be_saved_data = NULL;
4372 save->delay_pair = pair;
4374 save->next = backtrack_queue;
4375 backtrack_queue = save;
4377 while (pair)
4379 mark_backtrack_feeds (pair->i2, 1);
4380 INSN_TICK (pair->i2) = INVALID_TICK;
4381 INSN_EXACT_TICK (pair->i2) = clock_var + pair_delay (pair);
4382 SHADOW_P (pair->i2) = pair->stages == 0;
4383 pair = pair->next_same_i1;
4387 /* Walk the ready list and all queues. If any insns have unresolved backwards
4388 dependencies, these must be cancelled deps, broken by predication. Set or
4389 clear (depending on SET) the DEP_CANCELLED bit in DEP_STATUS. */
4391 static void
4392 toggle_cancelled_flags (bool set)
4394 int i;
4395 sd_iterator_def sd_it;
4396 dep_t dep;
4398 if (ready.n_ready > 0)
4400 rtx_insn **first = ready_lastpos (&ready);
4401 for (i = 0; i < ready.n_ready; i++)
4402 FOR_EACH_DEP (first[i], SD_LIST_BACK, sd_it, dep)
4403 if (!DEBUG_INSN_P (DEP_PRO (dep)))
4405 if (set)
4406 DEP_STATUS (dep) |= DEP_CANCELLED;
4407 else
4408 DEP_STATUS (dep) &= ~DEP_CANCELLED;
4411 for (i = 0; i <= max_insn_queue_index; i++)
4413 int q = NEXT_Q_AFTER (q_ptr, i);
4414 rtx_insn_list *link;
4415 for (link = insn_queue[q]; link; link = link->next ())
4417 rtx_insn *insn = link->insn ();
4418 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep)
4419 if (!DEBUG_INSN_P (DEP_PRO (dep)))
4421 if (set)
4422 DEP_STATUS (dep) |= DEP_CANCELLED;
4423 else
4424 DEP_STATUS (dep) &= ~DEP_CANCELLED;
4430 /* Undo the replacements that have occurred after backtrack point SAVE
4431 was placed. */
4432 static void
4433 undo_replacements_for_backtrack (struct haifa_saved_data *save)
4435 while (!save->replacement_deps.is_empty ())
4437 dep_t dep = save->replacement_deps.pop ();
4438 int apply_p = save->replace_apply.pop ();
4440 if (apply_p)
4441 restore_pattern (dep, true);
4442 else
4443 apply_replacement (dep, true);
4445 save->replacement_deps.release ();
4446 save->replace_apply.release ();
4449 /* Pop entries from the SCHEDULED_INSNS vector up to and including INSN.
4450 Restore their dependencies to an unresolved state, and mark them as
4451 queued nowhere. */
4453 static void
4454 unschedule_insns_until (rtx_insn *insn)
4456 auto_vec<rtx_insn *> recompute_vec;
4458 /* Make two passes over the insns to be unscheduled. First, we clear out
4459 dependencies and other trivial bookkeeping. */
4460 for (;;)
4462 rtx_insn *last;
4463 sd_iterator_def sd_it;
4464 dep_t dep;
4466 last = scheduled_insns.pop ();
4468 /* This will be changed by restore_backtrack_point if the insn is in
4469 any queue. */
4470 QUEUE_INDEX (last) = QUEUE_NOWHERE;
4471 if (last != insn)
4472 INSN_TICK (last) = INVALID_TICK;
4474 if (modulo_ii > 0 && INSN_UID (last) < modulo_iter0_max_uid)
4475 modulo_insns_scheduled--;
4477 for (sd_it = sd_iterator_start (last, SD_LIST_RES_FORW);
4478 sd_iterator_cond (&sd_it, &dep);)
4480 rtx_insn *con = DEP_CON (dep);
4481 sd_unresolve_dep (sd_it);
4482 if (!MUST_RECOMPUTE_SPEC_P (con))
4484 MUST_RECOMPUTE_SPEC_P (con) = 1;
4485 recompute_vec.safe_push (con);
4489 if (last == insn)
4490 break;
4493 /* A second pass, to update ready and speculation status for insns
4494 depending on the unscheduled ones. The first pass must have
4495 popped the scheduled_insns vector up to the point where we
4496 restart scheduling, as recompute_todo_spec requires it to be
4497 up-to-date. */
4498 while (!recompute_vec.is_empty ())
4500 rtx_insn *con;
4502 con = recompute_vec.pop ();
4503 MUST_RECOMPUTE_SPEC_P (con) = 0;
4504 if (!sd_lists_empty_p (con, SD_LIST_HARD_BACK))
4506 TODO_SPEC (con) = HARD_DEP;
4507 INSN_TICK (con) = INVALID_TICK;
4508 if (PREDICATED_PAT (con) != NULL_RTX)
4509 haifa_change_pattern (con, ORIG_PAT (con));
4511 else if (QUEUE_INDEX (con) != QUEUE_SCHEDULED)
4512 TODO_SPEC (con) = recompute_todo_spec (con, true);
4516 /* Restore scheduler state from the topmost entry on the backtracking queue.
4517 PSCHED_BLOCK_P points to the local data of schedule_block that we must
4518 overwrite with the saved data.
4519 The caller must already have called unschedule_insns_until. */
4521 static void
4522 restore_last_backtrack_point (struct sched_block_state *psched_block)
4524 int i;
4525 struct haifa_saved_data *save = backtrack_queue;
4527 backtrack_queue = save->next;
4529 if (current_sched_info->restore_state)
4530 (*current_sched_info->restore_state) (save->fe_saved_data);
4532 if (targetm.sched.alloc_sched_context)
4534 targetm.sched.set_sched_context (save->be_saved_data);
4535 targetm.sched.free_sched_context (save->be_saved_data);
4538 /* Do this first since it clobbers INSN_TICK of the involved
4539 instructions. */
4540 undo_replacements_for_backtrack (save);
4542 /* Clear the QUEUE_INDEX of everything in the ready list or one
4543 of the queues. */
4544 if (ready.n_ready > 0)
4546 rtx_insn **first = ready_lastpos (&ready);
4547 for (i = 0; i < ready.n_ready; i++)
4549 rtx_insn *insn = first[i];
4550 QUEUE_INDEX (insn) = QUEUE_NOWHERE;
4551 INSN_TICK (insn) = INVALID_TICK;
4554 for (i = 0; i <= max_insn_queue_index; i++)
4556 int q = NEXT_Q_AFTER (q_ptr, i);
4558 for (rtx_insn_list *link = insn_queue[q]; link; link = link->next ())
4560 rtx_insn *x = link->insn ();
4561 QUEUE_INDEX (x) = QUEUE_NOWHERE;
4562 INSN_TICK (x) = INVALID_TICK;
4564 free_INSN_LIST_list (&insn_queue[q]);
4567 free (ready.vec);
4568 ready = save->ready;
4570 if (ready.n_ready > 0)
4572 rtx_insn **first = ready_lastpos (&ready);
4573 for (i = 0; i < ready.n_ready; i++)
4575 rtx_insn *insn = first[i];
4576 QUEUE_INDEX (insn) = QUEUE_READY;
4577 TODO_SPEC (insn) = recompute_todo_spec (insn, true);
4578 INSN_TICK (insn) = save->clock_var;
4582 q_ptr = 0;
4583 q_size = save->q_size;
4584 for (i = 0; i <= max_insn_queue_index; i++)
4586 int q = NEXT_Q_AFTER (q_ptr, i);
4588 insn_queue[q] = save->insn_queue[q];
4590 for (rtx_insn_list *link = insn_queue[q]; link; link = link->next ())
4592 rtx_insn *x = link->insn ();
4593 QUEUE_INDEX (x) = i;
4594 TODO_SPEC (x) = recompute_todo_spec (x, true);
4595 INSN_TICK (x) = save->clock_var + i;
4598 free (save->insn_queue);
4600 toggle_cancelled_flags (true);
4602 clock_var = save->clock_var;
4603 last_clock_var = save->last_clock_var;
4604 cycle_issued_insns = save->cycle_issued_insns;
4605 last_scheduled_insn = save->last_scheduled_insn;
4606 last_nondebug_scheduled_insn = save->last_nondebug_scheduled_insn;
4607 nonscheduled_insns_begin = save->nonscheduled_insns_begin;
4609 *psched_block = save->sched_block;
4611 memcpy (curr_state, save->curr_state, dfa_state_size);
4612 free (save->curr_state);
4614 mark_backtrack_feeds (save->delay_pair->i2, 0);
4616 gcc_assert (next_cycle_replace_deps.is_empty ());
4617 next_cycle_replace_deps = save->next_cycle_deps.copy ();
4618 next_cycle_apply = save->next_cycle_apply.copy ();
4620 free (save);
4622 for (save = backtrack_queue; save; save = save->next)
4624 mark_backtrack_feeds (save->delay_pair->i2, 1);
4628 /* Discard all data associated with the topmost entry in the backtrack
4629 queue. If RESET_TICK is false, we just want to free the data. If true,
4630 we are doing this because we discovered a reason to backtrack. In the
4631 latter case, also reset the INSN_TICK for the shadow insn. */
4632 static void
4633 free_topmost_backtrack_point (bool reset_tick)
4635 struct haifa_saved_data *save = backtrack_queue;
4636 int i;
4638 backtrack_queue = save->next;
4640 if (reset_tick)
4642 struct delay_pair *pair = save->delay_pair;
4643 while (pair)
4645 INSN_TICK (pair->i2) = INVALID_TICK;
4646 INSN_EXACT_TICK (pair->i2) = INVALID_TICK;
4647 pair = pair->next_same_i1;
4649 undo_replacements_for_backtrack (save);
4651 else
4653 save->replacement_deps.release ();
4654 save->replace_apply.release ();
4657 if (targetm.sched.free_sched_context)
4658 targetm.sched.free_sched_context (save->be_saved_data);
4659 if (current_sched_info->restore_state)
4660 free (save->fe_saved_data);
4661 for (i = 0; i <= max_insn_queue_index; i++)
4662 free_INSN_LIST_list (&save->insn_queue[i]);
4663 free (save->insn_queue);
4664 free (save->curr_state);
4665 free (save->ready.vec);
4666 free (save);
4669 /* Free the entire backtrack queue. */
4670 static void
4671 free_backtrack_queue (void)
4673 while (backtrack_queue)
4674 free_topmost_backtrack_point (false);
4677 /* Apply a replacement described by DESC. If IMMEDIATELY is false, we
4678 may have to postpone the replacement until the start of the next cycle,
4679 at which point we will be called again with IMMEDIATELY true. This is
4680 only done for machines which have instruction packets with explicit
4681 parallelism however. */
4682 static void
4683 apply_replacement (dep_t dep, bool immediately)
4685 struct dep_replacement *desc = DEP_REPLACE (dep);
4686 if (!immediately && targetm.sched.exposed_pipeline && reload_completed)
4688 next_cycle_replace_deps.safe_push (dep);
4689 next_cycle_apply.safe_push (1);
4691 else
4693 bool success;
4695 if (QUEUE_INDEX (desc->insn) == QUEUE_SCHEDULED)
4696 return;
4698 if (sched_verbose >= 5)
4699 fprintf (sched_dump, "applying replacement for insn %d\n",
4700 INSN_UID (desc->insn));
4702 success = validate_change (desc->insn, desc->loc, desc->newval, 0);
4703 gcc_assert (success);
4705 update_insn_after_change (desc->insn);
4706 if ((TODO_SPEC (desc->insn) & (HARD_DEP | DEP_POSTPONED)) == 0)
4707 fix_tick_ready (desc->insn);
4709 if (backtrack_queue != NULL)
4711 backtrack_queue->replacement_deps.safe_push (dep);
4712 backtrack_queue->replace_apply.safe_push (1);
4717 /* We have determined that a pattern involved in DEP must be restored.
4718 If IMMEDIATELY is false, we may have to postpone the replacement
4719 until the start of the next cycle, at which point we will be called
4720 again with IMMEDIATELY true. */
4721 static void
4722 restore_pattern (dep_t dep, bool immediately)
4724 rtx_insn *next = DEP_CON (dep);
4725 int tick = INSN_TICK (next);
4727 /* If we already scheduled the insn, the modified version is
4728 correct. */
4729 if (QUEUE_INDEX (next) == QUEUE_SCHEDULED)
4730 return;
4732 if (!immediately && targetm.sched.exposed_pipeline && reload_completed)
4734 next_cycle_replace_deps.safe_push (dep);
4735 next_cycle_apply.safe_push (0);
4736 return;
4740 if (DEP_TYPE (dep) == REG_DEP_CONTROL)
4742 if (sched_verbose >= 5)
4743 fprintf (sched_dump, "restoring pattern for insn %d\n",
4744 INSN_UID (next));
4745 haifa_change_pattern (next, ORIG_PAT (next));
4747 else
4749 struct dep_replacement *desc = DEP_REPLACE (dep);
4750 bool success;
4752 if (sched_verbose >= 5)
4753 fprintf (sched_dump, "restoring pattern for insn %d\n",
4754 INSN_UID (desc->insn));
4755 tick = INSN_TICK (desc->insn);
4757 success = validate_change (desc->insn, desc->loc, desc->orig, 0);
4758 gcc_assert (success);
4759 update_insn_after_change (desc->insn);
4760 if (backtrack_queue != NULL)
4762 backtrack_queue->replacement_deps.safe_push (dep);
4763 backtrack_queue->replace_apply.safe_push (0);
4766 INSN_TICK (next) = tick;
4767 if (TODO_SPEC (next) == DEP_POSTPONED)
4768 return;
4770 if (sd_lists_empty_p (next, SD_LIST_BACK))
4771 TODO_SPEC (next) = 0;
4772 else if (!sd_lists_empty_p (next, SD_LIST_HARD_BACK))
4773 TODO_SPEC (next) = HARD_DEP;
4776 /* Perform pattern replacements that were queued up until the next
4777 cycle. */
4778 static void
4779 perform_replacements_new_cycle (void)
4781 int i;
4782 dep_t dep;
4783 FOR_EACH_VEC_ELT (next_cycle_replace_deps, i, dep)
4785 int apply_p = next_cycle_apply[i];
4786 if (apply_p)
4787 apply_replacement (dep, true);
4788 else
4789 restore_pattern (dep, true);
4791 next_cycle_replace_deps.truncate (0);
4792 next_cycle_apply.truncate (0);
4795 /* Compute INSN_TICK_ESTIMATE for INSN. PROCESSED is a bitmap of
4796 instructions we've previously encountered, a set bit prevents
4797 recursion. BUDGET is a limit on how far ahead we look, it is
4798 reduced on recursive calls. Return true if we produced a good
4799 estimate, or false if we exceeded the budget. */
4800 static bool
4801 estimate_insn_tick (bitmap processed, rtx_insn *insn, int budget)
4803 sd_iterator_def sd_it;
4804 dep_t dep;
4805 int earliest = INSN_TICK (insn);
4807 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep)
4809 rtx_insn *pro = DEP_PRO (dep);
4810 int t;
4812 if (DEP_STATUS (dep) & DEP_CANCELLED)
4813 continue;
4815 if (QUEUE_INDEX (pro) == QUEUE_SCHEDULED)
4816 gcc_assert (INSN_TICK (pro) + dep_cost (dep) <= INSN_TICK (insn));
4817 else
4819 int cost = dep_cost (dep);
4820 if (cost >= budget)
4821 return false;
4822 if (!bitmap_bit_p (processed, INSN_LUID (pro)))
4824 if (!estimate_insn_tick (processed, pro, budget - cost))
4825 return false;
4827 gcc_assert (INSN_TICK_ESTIMATE (pro) != INVALID_TICK);
4828 t = INSN_TICK_ESTIMATE (pro) + cost;
4829 if (earliest == INVALID_TICK || t > earliest)
4830 earliest = t;
4833 bitmap_set_bit (processed, INSN_LUID (insn));
4834 INSN_TICK_ESTIMATE (insn) = earliest;
4835 return true;
4838 /* Examine the pair of insns in P, and estimate (optimistically, assuming
4839 infinite resources) the cycle in which the delayed shadow can be issued.
4840 Return the number of cycles that must pass before the real insn can be
4841 issued in order to meet this constraint. */
4842 static int
4843 estimate_shadow_tick (struct delay_pair *p)
4845 auto_bitmap processed;
4846 int t;
4847 bool cutoff;
4849 cutoff = !estimate_insn_tick (processed, p->i2,
4850 max_insn_queue_index + pair_delay (p));
4851 if (cutoff)
4852 return max_insn_queue_index;
4853 t = INSN_TICK_ESTIMATE (p->i2) - (clock_var + pair_delay (p) + 1);
4854 if (t > 0)
4855 return t;
4856 return 0;
4859 /* If INSN has no unresolved backwards dependencies, add it to the schedule and
4860 recursively resolve all its forward dependencies. */
4861 static void
4862 resolve_dependencies (rtx_insn *insn)
4864 sd_iterator_def sd_it;
4865 dep_t dep;
4867 /* Don't use sd_lists_empty_p; it ignores debug insns. */
4868 if (DEPS_LIST_FIRST (INSN_HARD_BACK_DEPS (insn)) != NULL
4869 || DEPS_LIST_FIRST (INSN_SPEC_BACK_DEPS (insn)) != NULL)
4870 return;
4872 if (sched_verbose >= 4)
4873 fprintf (sched_dump, ";;\tquickly resolving %d\n", INSN_UID (insn));
4875 if (QUEUE_INDEX (insn) >= 0)
4876 queue_remove (insn);
4878 scheduled_insns.safe_push (insn);
4880 /* Update dependent instructions. */
4881 for (sd_it = sd_iterator_start (insn, SD_LIST_FORW);
4882 sd_iterator_cond (&sd_it, &dep);)
4884 rtx_insn *next = DEP_CON (dep);
4886 if (sched_verbose >= 4)
4887 fprintf (sched_dump, ";;\t\tdep %d against %d\n", INSN_UID (insn),
4888 INSN_UID (next));
4890 /* Resolve the dependence between INSN and NEXT.
4891 sd_resolve_dep () moves current dep to another list thus
4892 advancing the iterator. */
4893 sd_resolve_dep (sd_it);
4895 if (!IS_SPECULATION_BRANCHY_CHECK_P (insn))
4897 resolve_dependencies (next);
4899 else
4900 /* Check always has only one forward dependence (to the first insn in
4901 the recovery block), therefore, this will be executed only once. */
4903 gcc_assert (sd_lists_empty_p (insn, SD_LIST_FORW));
4909 /* Return the head and tail pointers of ebb starting at BEG and ending
4910 at END. */
4911 void
4912 get_ebb_head_tail (basic_block beg, basic_block end,
4913 rtx_insn **headp, rtx_insn **tailp)
4915 rtx_insn *beg_head = BB_HEAD (beg);
4916 rtx_insn * beg_tail = BB_END (beg);
4917 rtx_insn * end_head = BB_HEAD (end);
4918 rtx_insn * end_tail = BB_END (end);
4920 /* Don't include any notes or labels at the beginning of the BEG
4921 basic block, or notes at the end of the END basic blocks. */
4923 if (LABEL_P (beg_head))
4924 beg_head = NEXT_INSN (beg_head);
4926 while (beg_head != beg_tail)
4927 if (NOTE_P (beg_head))
4928 beg_head = NEXT_INSN (beg_head);
4929 else if (DEBUG_INSN_P (beg_head))
4931 rtx_insn * note, *next;
4933 for (note = NEXT_INSN (beg_head);
4934 note != beg_tail;
4935 note = next)
4937 next = NEXT_INSN (note);
4938 if (NOTE_P (note))
4940 if (sched_verbose >= 9)
4941 fprintf (sched_dump, "reorder %i\n", INSN_UID (note));
4943 reorder_insns_nobb (note, note, PREV_INSN (beg_head));
4945 if (BLOCK_FOR_INSN (note) != beg)
4946 df_insn_change_bb (note, beg);
4948 else if (!DEBUG_INSN_P (note))
4949 break;
4952 break;
4954 else
4955 break;
4957 *headp = beg_head;
4959 if (beg == end)
4960 end_head = beg_head;
4961 else if (LABEL_P (end_head))
4962 end_head = NEXT_INSN (end_head);
4964 while (end_head != end_tail)
4965 if (NOTE_P (end_tail))
4966 end_tail = PREV_INSN (end_tail);
4967 else if (DEBUG_INSN_P (end_tail))
4969 rtx_insn * note, *prev;
4971 for (note = PREV_INSN (end_tail);
4972 note != end_head;
4973 note = prev)
4975 prev = PREV_INSN (note);
4976 if (NOTE_P (note))
4978 if (sched_verbose >= 9)
4979 fprintf (sched_dump, "reorder %i\n", INSN_UID (note));
4981 reorder_insns_nobb (note, note, end_tail);
4983 if (end_tail == BB_END (end))
4984 BB_END (end) = note;
4986 if (BLOCK_FOR_INSN (note) != end)
4987 df_insn_change_bb (note, end);
4989 else if (!DEBUG_INSN_P (note))
4990 break;
4993 break;
4995 else
4996 break;
4998 *tailp = end_tail;
5001 /* Return nonzero if there are no real insns in the range [ HEAD, TAIL ]. */
5004 no_real_insns_p (const rtx_insn *head, const rtx_insn *tail)
5006 while (head != NEXT_INSN (tail))
5008 if (!NOTE_P (head) && !LABEL_P (head))
5009 return 0;
5010 head = NEXT_INSN (head);
5012 return 1;
5015 /* Restore-other-notes: NOTE_LIST is the end of a chain of notes
5016 previously found among the insns. Insert them just before HEAD. */
5017 rtx_insn *
5018 restore_other_notes (rtx_insn *head, basic_block head_bb)
5020 if (note_list != 0)
5022 rtx_insn *note_head = note_list;
5024 if (head)
5025 head_bb = BLOCK_FOR_INSN (head);
5026 else
5027 head = NEXT_INSN (bb_note (head_bb));
5029 while (PREV_INSN (note_head))
5031 set_block_for_insn (note_head, head_bb);
5032 note_head = PREV_INSN (note_head);
5034 /* In the above cycle we've missed this note. */
5035 set_block_for_insn (note_head, head_bb);
5037 SET_PREV_INSN (note_head) = PREV_INSN (head);
5038 SET_NEXT_INSN (PREV_INSN (head)) = note_head;
5039 SET_PREV_INSN (head) = note_list;
5040 SET_NEXT_INSN (note_list) = head;
5042 if (BLOCK_FOR_INSN (head) != head_bb)
5043 BB_END (head_bb) = note_list;
5045 head = note_head;
5048 return head;
5051 /* When we know we are going to discard the schedule due to a failed attempt
5052 at modulo scheduling, undo all replacements. */
5053 static void
5054 undo_all_replacements (void)
5056 rtx_insn *insn;
5057 int i;
5059 FOR_EACH_VEC_ELT (scheduled_insns, i, insn)
5061 sd_iterator_def sd_it;
5062 dep_t dep;
5064 /* See if we must undo a replacement. */
5065 for (sd_it = sd_iterator_start (insn, SD_LIST_RES_FORW);
5066 sd_iterator_cond (&sd_it, &dep); sd_iterator_next (&sd_it))
5068 struct dep_replacement *desc = DEP_REPLACE (dep);
5069 if (desc != NULL)
5070 validate_change (desc->insn, desc->loc, desc->orig, 0);
5075 /* Return first non-scheduled insn in the current scheduling block.
5076 This is mostly used for debug-counter purposes. */
5077 static rtx_insn *
5078 first_nonscheduled_insn (void)
5080 rtx_insn *insn = (nonscheduled_insns_begin != NULL_RTX
5081 ? nonscheduled_insns_begin
5082 : current_sched_info->prev_head);
5086 insn = next_nonnote_nondebug_insn (insn);
5088 while (QUEUE_INDEX (insn) == QUEUE_SCHEDULED);
5090 return insn;
5093 /* Move insns that became ready to fire from queue to ready list. */
5095 static void
5096 queue_to_ready (struct ready_list *ready)
5098 rtx_insn *insn;
5099 rtx_insn_list *link;
5100 rtx_insn *skip_insn;
5102 q_ptr = NEXT_Q (q_ptr);
5104 if (dbg_cnt (sched_insn) == false)
5105 /* If debug counter is activated do not requeue the first
5106 nonscheduled insn. */
5107 skip_insn = first_nonscheduled_insn ();
5108 else
5109 skip_insn = NULL;
5111 /* Add all pending insns that can be scheduled without stalls to the
5112 ready list. */
5113 for (link = insn_queue[q_ptr]; link; link = link->next ())
5115 insn = link->insn ();
5116 q_size -= 1;
5118 if (sched_verbose >= 2)
5119 fprintf (sched_dump, ";;\t\tQ-->Ready: insn %s: ",
5120 (*current_sched_info->print_insn) (insn, 0));
5122 /* If the ready list is full, delay the insn for 1 cycle.
5123 See the comment in schedule_block for the rationale. */
5124 if (!reload_completed
5125 && (ready->n_ready - ready->n_debug > MAX_SCHED_READY_INSNS
5126 || (sched_pressure == SCHED_PRESSURE_MODEL
5127 /* Limit pressure recalculations to MAX_SCHED_READY_INSNS
5128 instructions too. */
5129 && model_index (insn) > (model_curr_point
5130 + MAX_SCHED_READY_INSNS)))
5131 && !(sched_pressure == SCHED_PRESSURE_MODEL
5132 && model_curr_point < model_num_insns
5133 /* Always allow the next model instruction to issue. */
5134 && model_index (insn) == model_curr_point)
5135 && !SCHED_GROUP_P (insn)
5136 && insn != skip_insn)
5138 if (sched_verbose >= 2)
5139 fprintf (sched_dump, "keeping in queue, ready full\n");
5140 queue_insn (insn, 1, "ready full");
5142 else
5144 ready_add (ready, insn, false);
5145 if (sched_verbose >= 2)
5146 fprintf (sched_dump, "moving to ready without stalls\n");
5149 free_INSN_LIST_list (&insn_queue[q_ptr]);
5151 /* If there are no ready insns, stall until one is ready and add all
5152 of the pending insns at that point to the ready list. */
5153 if (ready->n_ready == 0)
5155 int stalls;
5157 for (stalls = 1; stalls <= max_insn_queue_index; stalls++)
5159 if ((link = insn_queue[NEXT_Q_AFTER (q_ptr, stalls)]))
5161 for (; link; link = link->next ())
5163 insn = link->insn ();
5164 q_size -= 1;
5166 if (sched_verbose >= 2)
5167 fprintf (sched_dump, ";;\t\tQ-->Ready: insn %s: ",
5168 (*current_sched_info->print_insn) (insn, 0));
5170 ready_add (ready, insn, false);
5171 if (sched_verbose >= 2)
5172 fprintf (sched_dump, "moving to ready with %d stalls\n", stalls);
5174 free_INSN_LIST_list (&insn_queue[NEXT_Q_AFTER (q_ptr, stalls)]);
5176 advance_one_cycle ();
5178 break;
5181 advance_one_cycle ();
5184 q_ptr = NEXT_Q_AFTER (q_ptr, stalls);
5185 clock_var += stalls;
5186 if (sched_verbose >= 2)
5187 fprintf (sched_dump, ";;\tAdvancing clock by %d cycle[s] to %d\n",
5188 stalls, clock_var);
5192 /* Used by early_queue_to_ready. Determines whether it is "ok" to
5193 prematurely move INSN from the queue to the ready list. Currently,
5194 if a target defines the hook 'is_costly_dependence', this function
5195 uses the hook to check whether there exist any dependences which are
5196 considered costly by the target, between INSN and other insns that
5197 have already been scheduled. Dependences are checked up to Y cycles
5198 back, with default Y=1; The flag -fsched-stalled-insns-dep=Y allows
5199 controlling this value.
5200 (Other considerations could be taken into account instead (or in
5201 addition) depending on user flags and target hooks. */
5203 static bool
5204 ok_for_early_queue_removal (rtx_insn *insn)
5206 if (targetm.sched.is_costly_dependence)
5208 int n_cycles;
5209 int i = scheduled_insns.length ();
5210 for (n_cycles = flag_sched_stalled_insns_dep; n_cycles; n_cycles--)
5212 while (i-- > 0)
5214 int cost;
5216 rtx_insn *prev_insn = scheduled_insns[i];
5218 if (!NOTE_P (prev_insn))
5220 dep_t dep;
5222 dep = sd_find_dep_between (prev_insn, insn, true);
5224 if (dep != NULL)
5226 cost = dep_cost (dep);
5228 if (targetm.sched.is_costly_dependence (dep, cost,
5229 flag_sched_stalled_insns_dep - n_cycles))
5230 return false;
5234 if (GET_MODE (prev_insn) == TImode) /* end of dispatch group */
5235 break;
5238 if (i == 0)
5239 break;
5243 return true;
5247 /* Remove insns from the queue, before they become "ready" with respect
5248 to FU latency considerations. */
5250 static int
5251 early_queue_to_ready (state_t state, struct ready_list *ready)
5253 rtx_insn *insn;
5254 rtx_insn_list *link;
5255 rtx_insn_list *next_link;
5256 rtx_insn_list *prev_link;
5257 bool move_to_ready;
5258 int cost;
5259 state_t temp_state = alloca (dfa_state_size);
5260 int stalls;
5261 int insns_removed = 0;
5264 Flag '-fsched-stalled-insns=X' determines the aggressiveness of this
5265 function:
5267 X == 0: There is no limit on how many queued insns can be removed
5268 prematurely. (flag_sched_stalled_insns = -1).
5270 X >= 1: Only X queued insns can be removed prematurely in each
5271 invocation. (flag_sched_stalled_insns = X).
5273 Otherwise: Early queue removal is disabled.
5274 (flag_sched_stalled_insns = 0)
5277 if (! flag_sched_stalled_insns)
5278 return 0;
5280 for (stalls = 0; stalls <= max_insn_queue_index; stalls++)
5282 if ((link = insn_queue[NEXT_Q_AFTER (q_ptr, stalls)]))
5284 if (sched_verbose > 6)
5285 fprintf (sched_dump, ";; look at index %d + %d\n", q_ptr, stalls);
5287 prev_link = 0;
5288 while (link)
5290 next_link = link->next ();
5291 insn = link->insn ();
5292 if (insn && sched_verbose > 6)
5293 print_rtl_single (sched_dump, insn);
5295 memcpy (temp_state, state, dfa_state_size);
5296 if (recog_memoized (insn) < 0)
5297 /* non-negative to indicate that it's not ready
5298 to avoid infinite Q->R->Q->R... */
5299 cost = 0;
5300 else
5301 cost = state_transition (temp_state, insn);
5303 if (sched_verbose >= 6)
5304 fprintf (sched_dump, "transition cost = %d\n", cost);
5306 move_to_ready = false;
5307 if (cost < 0)
5309 move_to_ready = ok_for_early_queue_removal (insn);
5310 if (move_to_ready == true)
5312 /* move from Q to R */
5313 q_size -= 1;
5314 ready_add (ready, insn, false);
5316 if (prev_link)
5317 XEXP (prev_link, 1) = next_link;
5318 else
5319 insn_queue[NEXT_Q_AFTER (q_ptr, stalls)] = next_link;
5321 free_INSN_LIST_node (link);
5323 if (sched_verbose >= 2)
5324 fprintf (sched_dump, ";;\t\tEarly Q-->Ready: insn %s\n",
5325 (*current_sched_info->print_insn) (insn, 0));
5327 insns_removed++;
5328 if (insns_removed == flag_sched_stalled_insns)
5329 /* Remove no more than flag_sched_stalled_insns insns
5330 from Q at a time. */
5331 return insns_removed;
5335 if (move_to_ready == false)
5336 prev_link = link;
5338 link = next_link;
5339 } /* while link */
5340 } /* if link */
5342 } /* for stalls.. */
5344 return insns_removed;
5348 /* Print the ready list for debugging purposes.
5349 If READY_TRY is non-zero then only print insns that max_issue
5350 will consider. */
5351 static void
5352 debug_ready_list_1 (struct ready_list *ready, signed char *ready_try)
5354 rtx_insn **p;
5355 int i;
5357 if (ready->n_ready == 0)
5359 fprintf (sched_dump, "\n");
5360 return;
5363 p = ready_lastpos (ready);
5364 for (i = 0; i < ready->n_ready; i++)
5366 if (ready_try != NULL && ready_try[ready->n_ready - i - 1])
5367 continue;
5369 fprintf (sched_dump, " %s:%d",
5370 (*current_sched_info->print_insn) (p[i], 0),
5371 INSN_LUID (p[i]));
5372 if (sched_pressure != SCHED_PRESSURE_NONE)
5373 fprintf (sched_dump, "(cost=%d",
5374 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (p[i]));
5375 fprintf (sched_dump, ":prio=%d", INSN_PRIORITY (p[i]));
5376 if (INSN_TICK (p[i]) > clock_var)
5377 fprintf (sched_dump, ":delay=%d", INSN_TICK (p[i]) - clock_var);
5378 if (sched_pressure == SCHED_PRESSURE_MODEL)
5379 fprintf (sched_dump, ":idx=%d",
5380 model_index (p[i]));
5381 if (sched_pressure != SCHED_PRESSURE_NONE)
5382 fprintf (sched_dump, ")");
5384 fprintf (sched_dump, "\n");
5387 /* Print the ready list. Callable from debugger. */
5388 static void
5389 debug_ready_list (struct ready_list *ready)
5391 debug_ready_list_1 (ready, NULL);
5394 /* Search INSN for REG_SAVE_NOTE notes and convert them back into insn
5395 NOTEs. This is used for NOTE_INSN_EPILOGUE_BEG, so that sched-ebb
5396 replaces the epilogue note in the correct basic block. */
5397 void
5398 reemit_notes (rtx_insn *insn)
5400 rtx note;
5401 rtx_insn *last = insn;
5403 for (note = REG_NOTES (insn); note; note = XEXP (note, 1))
5405 if (REG_NOTE_KIND (note) == REG_SAVE_NOTE)
5407 enum insn_note note_type = (enum insn_note) INTVAL (XEXP (note, 0));
5409 last = emit_note_before (note_type, last);
5410 remove_note (insn, note);
5415 /* Move INSN. Reemit notes if needed. Update CFG, if needed. */
5416 static void
5417 move_insn (rtx_insn *insn, rtx_insn *last, rtx nt)
5419 if (PREV_INSN (insn) != last)
5421 basic_block bb;
5422 rtx_insn *note;
5423 int jump_p = 0;
5425 bb = BLOCK_FOR_INSN (insn);
5427 /* BB_HEAD is either LABEL or NOTE. */
5428 gcc_assert (BB_HEAD (bb) != insn);
5430 if (BB_END (bb) == insn)
5431 /* If this is last instruction in BB, move end marker one
5432 instruction up. */
5434 /* Jumps are always placed at the end of basic block. */
5435 jump_p = control_flow_insn_p (insn);
5437 gcc_assert (!jump_p
5438 || ((common_sched_info->sched_pass_id == SCHED_RGN_PASS)
5439 && IS_SPECULATION_BRANCHY_CHECK_P (insn))
5440 || (common_sched_info->sched_pass_id
5441 == SCHED_EBB_PASS));
5443 gcc_assert (BLOCK_FOR_INSN (PREV_INSN (insn)) == bb);
5445 BB_END (bb) = PREV_INSN (insn);
5448 gcc_assert (BB_END (bb) != last);
5450 if (jump_p)
5451 /* We move the block note along with jump. */
5453 gcc_assert (nt);
5455 note = NEXT_INSN (insn);
5456 while (NOTE_NOT_BB_P (note) && note != nt)
5457 note = NEXT_INSN (note);
5459 if (note != nt
5460 && (LABEL_P (note)
5461 || BARRIER_P (note)))
5462 note = NEXT_INSN (note);
5464 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note));
5466 else
5467 note = insn;
5469 SET_NEXT_INSN (PREV_INSN (insn)) = NEXT_INSN (note);
5470 SET_PREV_INSN (NEXT_INSN (note)) = PREV_INSN (insn);
5472 SET_NEXT_INSN (note) = NEXT_INSN (last);
5473 SET_PREV_INSN (NEXT_INSN (last)) = note;
5475 SET_NEXT_INSN (last) = insn;
5476 SET_PREV_INSN (insn) = last;
5478 bb = BLOCK_FOR_INSN (last);
5480 if (jump_p)
5482 fix_jump_move (insn);
5484 if (BLOCK_FOR_INSN (insn) != bb)
5485 move_block_after_check (insn);
5487 gcc_assert (BB_END (bb) == last);
5490 df_insn_change_bb (insn, bb);
5492 /* Update BB_END, if needed. */
5493 if (BB_END (bb) == last)
5494 BB_END (bb) = insn;
5497 SCHED_GROUP_P (insn) = 0;
5500 /* Return true if scheduling INSN will finish current clock cycle. */
5501 static bool
5502 insn_finishes_cycle_p (rtx_insn *insn)
5504 if (SCHED_GROUP_P (insn))
5505 /* After issuing INSN, rest of the sched_group will be forced to issue
5506 in order. Don't make any plans for the rest of cycle. */
5507 return true;
5509 /* Finishing the block will, apparently, finish the cycle. */
5510 if (current_sched_info->insn_finishes_block_p
5511 && current_sched_info->insn_finishes_block_p (insn))
5512 return true;
5514 return false;
5517 /* Helper for autopref_multipass_init. Given a SET in PAT and whether
5518 we're expecting a memory WRITE or not, check that the insn is relevant to
5519 the autoprefetcher modelling code. Return true iff that is the case.
5520 If it is relevant, record the base register of the memory op in BASE and
5521 the offset in OFFSET. */
5523 static bool
5524 analyze_set_insn_for_autopref (rtx pat, bool write, rtx *base, int *offset)
5526 if (GET_CODE (pat) != SET)
5527 return false;
5529 rtx mem = write ? SET_DEST (pat) : SET_SRC (pat);
5530 if (!MEM_P (mem))
5531 return false;
5533 struct address_info info;
5534 decompose_mem_address (&info, mem);
5536 /* TODO: Currently only (base+const) addressing is supported. */
5537 if (info.base == NULL || !REG_P (*info.base)
5538 || (info.disp != NULL && !CONST_INT_P (*info.disp)))
5539 return false;
5541 *base = *info.base;
5542 *offset = info.disp ? INTVAL (*info.disp) : 0;
5543 return true;
5546 /* Functions to model cache auto-prefetcher.
5548 Some of the CPUs have cache auto-prefetcher, which /seems/ to initiate
5549 memory prefetches if it sees instructions with consequitive memory accesses
5550 in the instruction stream. Details of such hardware units are not published,
5551 so we can only guess what exactly is going on there.
5552 In the scheduler, we model abstract auto-prefetcher. If there are memory
5553 insns in the ready list (or the queue) that have same memory base, but
5554 different offsets, then we delay the insns with larger offsets until insns
5555 with smaller offsets get scheduled. If PARAM_SCHED_AUTOPREF_QUEUE_DEPTH
5556 is "1", then we look at the ready list; if it is N>1, then we also look
5557 through N-1 queue entries.
5558 If the param is N>=0, then rank_for_schedule will consider auto-prefetching
5559 among its heuristics.
5560 Param value of "-1" disables modelling of the auto-prefetcher. */
5562 /* Initialize autoprefetcher model data for INSN. */
5563 static void
5564 autopref_multipass_init (const rtx_insn *insn, int write)
5566 autopref_multipass_data_t data = &INSN_AUTOPREF_MULTIPASS_DATA (insn)[write];
5568 gcc_assert (data->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED);
5569 data->base = NULL_RTX;
5570 data->offset = 0;
5571 /* Set insn entry initialized, but not relevant for auto-prefetcher. */
5572 data->status = AUTOPREF_MULTIPASS_DATA_IRRELEVANT;
5574 rtx pat = PATTERN (insn);
5576 /* We have a multi-set insn like a load-multiple or store-multiple.
5577 We care about these as long as all the memory ops inside the PARALLEL
5578 have the same base register. We care about the minimum and maximum
5579 offsets from that base but don't check for the order of those offsets
5580 within the PARALLEL insn itself. */
5581 if (GET_CODE (pat) == PARALLEL)
5583 int n_elems = XVECLEN (pat, 0);
5585 int i, offset;
5586 rtx base, prev_base = NULL_RTX;
5587 int min_offset = INT_MAX;
5589 for (i = 0; i < n_elems; i++)
5591 rtx set = XVECEXP (pat, 0, i);
5592 if (GET_CODE (set) != SET)
5593 return;
5595 if (!analyze_set_insn_for_autopref (set, write, &base, &offset))
5596 return;
5598 /* Ensure that all memory operations in the PARALLEL use the same
5599 base register. */
5600 if (i > 0 && REGNO (base) != REGNO (prev_base))
5601 return;
5602 prev_base = base;
5603 min_offset = MIN (min_offset, offset);
5606 /* If we reached here then we have a valid PARALLEL of multiple memory ops
5607 with prev_base as the base and min_offset containing the offset. */
5608 gcc_assert (prev_base);
5609 data->base = prev_base;
5610 data->offset = min_offset;
5611 data->status = AUTOPREF_MULTIPASS_DATA_NORMAL;
5612 return;
5615 /* Otherwise this is a single set memory operation. */
5616 rtx set = single_set (insn);
5617 if (set == NULL_RTX)
5618 return;
5620 if (!analyze_set_insn_for_autopref (set, write, &data->base,
5621 &data->offset))
5622 return;
5624 /* This insn is relevant for the auto-prefetcher.
5625 The base and offset fields will have been filled in the
5626 analyze_set_insn_for_autopref call above. */
5627 data->status = AUTOPREF_MULTIPASS_DATA_NORMAL;
5630 /* Helper function for rank_for_schedule sorting. */
5631 static int
5632 autopref_rank_for_schedule (const rtx_insn *insn1, const rtx_insn *insn2)
5634 int r = 0;
5635 for (int write = 0; write < 2 && !r; ++write)
5637 autopref_multipass_data_t data1
5638 = &INSN_AUTOPREF_MULTIPASS_DATA (insn1)[write];
5639 autopref_multipass_data_t data2
5640 = &INSN_AUTOPREF_MULTIPASS_DATA (insn2)[write];
5642 if (data1->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED)
5643 autopref_multipass_init (insn1, write);
5645 if (data2->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED)
5646 autopref_multipass_init (insn2, write);
5648 int irrel1 = data1->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT;
5649 int irrel2 = data2->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT;
5651 if (!irrel1 && !irrel2)
5652 r = data1->offset - data2->offset;
5653 else
5654 r = irrel2 - irrel1;
5657 return r;
5660 /* True if header of debug dump was printed. */
5661 static bool autopref_multipass_dfa_lookahead_guard_started_dump_p;
5663 /* Helper for autopref_multipass_dfa_lookahead_guard.
5664 Return "1" if INSN1 should be delayed in favor of INSN2. */
5665 static int
5666 autopref_multipass_dfa_lookahead_guard_1 (const rtx_insn *insn1,
5667 const rtx_insn *insn2, int write)
5669 autopref_multipass_data_t data1
5670 = &INSN_AUTOPREF_MULTIPASS_DATA (insn1)[write];
5671 autopref_multipass_data_t data2
5672 = &INSN_AUTOPREF_MULTIPASS_DATA (insn2)[write];
5674 if (data2->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED)
5675 autopref_multipass_init (insn2, write);
5676 if (data2->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT)
5677 return 0;
5679 if (rtx_equal_p (data1->base, data2->base)
5680 && data1->offset > data2->offset)
5682 if (sched_verbose >= 2)
5684 if (!autopref_multipass_dfa_lookahead_guard_started_dump_p)
5686 fprintf (sched_dump,
5687 ";;\t\tnot trying in max_issue due to autoprefetch "
5688 "model: ");
5689 autopref_multipass_dfa_lookahead_guard_started_dump_p = true;
5692 fprintf (sched_dump, " %d(%d)", INSN_UID (insn1), INSN_UID (insn2));
5695 return 1;
5698 return 0;
5701 /* General note:
5703 We could have also hooked autoprefetcher model into
5704 first_cycle_multipass_backtrack / first_cycle_multipass_issue hooks
5705 to enable intelligent selection of "[r1+0]=r2; [r1+4]=r3" on the same cycle
5706 (e.g., once "[r1+0]=r2" is issued in max_issue(), "[r1+4]=r3" gets
5707 unblocked). We don't bother about this yet because target of interest
5708 (ARM Cortex-A15) can issue only 1 memory operation per cycle. */
5710 /* Implementation of first_cycle_multipass_dfa_lookahead_guard hook.
5711 Return "1" if INSN1 should not be considered in max_issue due to
5712 auto-prefetcher considerations. */
5714 autopref_multipass_dfa_lookahead_guard (rtx_insn *insn1, int ready_index)
5716 int r = 0;
5718 /* Exit early if the param forbids this or if we're not entering here through
5719 normal haifa scheduling. This can happen if selective scheduling is
5720 explicitly enabled. */
5721 if (!insn_queue || PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) <= 0)
5722 return 0;
5724 if (sched_verbose >= 2 && ready_index == 0)
5725 autopref_multipass_dfa_lookahead_guard_started_dump_p = false;
5727 for (int write = 0; write < 2; ++write)
5729 autopref_multipass_data_t data1
5730 = &INSN_AUTOPREF_MULTIPASS_DATA (insn1)[write];
5732 if (data1->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED)
5733 autopref_multipass_init (insn1, write);
5734 if (data1->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT)
5735 continue;
5737 if (ready_index == 0
5738 && data1->status == AUTOPREF_MULTIPASS_DATA_DONT_DELAY)
5739 /* We allow only a single delay on priviledged instructions.
5740 Doing otherwise would cause infinite loop. */
5742 if (sched_verbose >= 2)
5744 if (!autopref_multipass_dfa_lookahead_guard_started_dump_p)
5746 fprintf (sched_dump,
5747 ";;\t\tnot trying in max_issue due to autoprefetch "
5748 "model: ");
5749 autopref_multipass_dfa_lookahead_guard_started_dump_p = true;
5752 fprintf (sched_dump, " *%d*", INSN_UID (insn1));
5754 continue;
5757 for (int i2 = 0; i2 < ready.n_ready; ++i2)
5759 rtx_insn *insn2 = get_ready_element (i2);
5760 if (insn1 == insn2)
5761 continue;
5762 r = autopref_multipass_dfa_lookahead_guard_1 (insn1, insn2, write);
5763 if (r)
5765 if (ready_index == 0)
5767 r = -1;
5768 data1->status = AUTOPREF_MULTIPASS_DATA_DONT_DELAY;
5770 goto finish;
5774 if (PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) == 1)
5775 continue;
5777 /* Everything from the current queue slot should have been moved to
5778 the ready list. */
5779 gcc_assert (insn_queue[NEXT_Q_AFTER (q_ptr, 0)] == NULL_RTX);
5781 int n_stalls = PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) - 1;
5782 if (n_stalls > max_insn_queue_index)
5783 n_stalls = max_insn_queue_index;
5785 for (int stalls = 1; stalls <= n_stalls; ++stalls)
5787 for (rtx_insn_list *link = insn_queue[NEXT_Q_AFTER (q_ptr, stalls)];
5788 link != NULL_RTX;
5789 link = link->next ())
5791 rtx_insn *insn2 = link->insn ();
5792 r = autopref_multipass_dfa_lookahead_guard_1 (insn1, insn2,
5793 write);
5794 if (r)
5796 /* Queue INSN1 until INSN2 can issue. */
5797 r = -stalls;
5798 if (ready_index == 0)
5799 data1->status = AUTOPREF_MULTIPASS_DATA_DONT_DELAY;
5800 goto finish;
5806 finish:
5807 if (sched_verbose >= 2
5808 && autopref_multipass_dfa_lookahead_guard_started_dump_p
5809 && (ready_index == ready.n_ready - 1 || r < 0))
5810 /* This does not /always/ trigger. We don't output EOL if the last
5811 insn is not recognized (INSN_CODE < 0) and lookahead_guard is not
5812 called. We can live with this. */
5813 fprintf (sched_dump, "\n");
5815 return r;
5818 /* Define type for target data used in multipass scheduling. */
5819 #ifndef TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DATA_T
5820 # define TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DATA_T int
5821 #endif
5822 typedef TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DATA_T first_cycle_multipass_data_t;
5824 /* The following structure describe an entry of the stack of choices. */
5825 struct choice_entry
5827 /* Ordinal number of the issued insn in the ready queue. */
5828 int index;
5829 /* The number of the rest insns whose issues we should try. */
5830 int rest;
5831 /* The number of issued essential insns. */
5832 int n;
5833 /* State after issuing the insn. */
5834 state_t state;
5835 /* Target-specific data. */
5836 first_cycle_multipass_data_t target_data;
5839 /* The following array is used to implement a stack of choices used in
5840 function max_issue. */
5841 static struct choice_entry *choice_stack;
5843 /* This holds the value of the target dfa_lookahead hook. */
5844 int dfa_lookahead;
5846 /* The following variable value is maximal number of tries of issuing
5847 insns for the first cycle multipass insn scheduling. We define
5848 this value as constant*(DFA_LOOKAHEAD**ISSUE_RATE). We would not
5849 need this constraint if all real insns (with non-negative codes)
5850 had reservations because in this case the algorithm complexity is
5851 O(DFA_LOOKAHEAD**ISSUE_RATE). Unfortunately, the dfa descriptions
5852 might be incomplete and such insn might occur. For such
5853 descriptions, the complexity of algorithm (without the constraint)
5854 could achieve DFA_LOOKAHEAD ** N , where N is the queue length. */
5855 static int max_lookahead_tries;
5857 /* The following function returns maximal (or close to maximal) number
5858 of insns which can be issued on the same cycle and one of which
5859 insns is insns with the best rank (the first insn in READY). To
5860 make this function tries different samples of ready insns. READY
5861 is current queue `ready'. Global array READY_TRY reflects what
5862 insns are already issued in this try. The function stops immediately,
5863 if it reached the such a solution, that all instruction can be issued.
5864 INDEX will contain index of the best insn in READY. The following
5865 function is used only for first cycle multipass scheduling.
5867 PRIVILEGED_N >= 0
5869 This function expects recognized insns only. All USEs,
5870 CLOBBERs, etc must be filtered elsewhere. */
5872 max_issue (struct ready_list *ready, int privileged_n, state_t state,
5873 bool first_cycle_insn_p, int *index)
5875 int n, i, all, n_ready, best, delay, tries_num;
5876 int more_issue;
5877 struct choice_entry *top;
5878 rtx_insn *insn;
5880 if (sched_fusion)
5881 return 0;
5883 n_ready = ready->n_ready;
5884 gcc_assert (dfa_lookahead >= 1 && privileged_n >= 0
5885 && privileged_n <= n_ready);
5887 /* Init MAX_LOOKAHEAD_TRIES. */
5888 if (max_lookahead_tries == 0)
5890 max_lookahead_tries = 100;
5891 for (i = 0; i < issue_rate; i++)
5892 max_lookahead_tries *= dfa_lookahead;
5895 /* Init max_points. */
5896 more_issue = issue_rate - cycle_issued_insns;
5897 gcc_assert (more_issue >= 0);
5899 /* The number of the issued insns in the best solution. */
5900 best = 0;
5902 top = choice_stack;
5904 /* Set initial state of the search. */
5905 memcpy (top->state, state, dfa_state_size);
5906 top->rest = dfa_lookahead;
5907 top->n = 0;
5908 if (targetm.sched.first_cycle_multipass_begin)
5909 targetm.sched.first_cycle_multipass_begin (&top->target_data,
5910 ready_try, n_ready,
5911 first_cycle_insn_p);
5913 /* Count the number of the insns to search among. */
5914 for (all = i = 0; i < n_ready; i++)
5915 if (!ready_try [i])
5916 all++;
5918 if (sched_verbose >= 2)
5920 fprintf (sched_dump, ";;\t\tmax_issue among %d insns:", all);
5921 debug_ready_list_1 (ready, ready_try);
5924 /* I is the index of the insn to try next. */
5925 i = 0;
5926 tries_num = 0;
5927 for (;;)
5929 if (/* If we've reached a dead end or searched enough of what we have
5930 been asked... */
5931 top->rest == 0
5932 /* or have nothing else to try... */
5933 || i >= n_ready
5934 /* or should not issue more. */
5935 || top->n >= more_issue)
5937 /* ??? (... || i == n_ready). */
5938 gcc_assert (i <= n_ready);
5940 /* We should not issue more than issue_rate instructions. */
5941 gcc_assert (top->n <= more_issue);
5943 if (top == choice_stack)
5944 break;
5946 if (best < top - choice_stack)
5948 if (privileged_n)
5950 n = privileged_n;
5951 /* Try to find issued privileged insn. */
5952 while (n && !ready_try[--n])
5956 if (/* If all insns are equally good... */
5957 privileged_n == 0
5958 /* Or a privileged insn will be issued. */
5959 || ready_try[n])
5960 /* Then we have a solution. */
5962 best = top - choice_stack;
5963 /* This is the index of the insn issued first in this
5964 solution. */
5965 *index = choice_stack [1].index;
5966 if (top->n == more_issue || best == all)
5967 break;
5971 /* Set ready-list index to point to the last insn
5972 ('i++' below will advance it to the next insn). */
5973 i = top->index;
5975 /* Backtrack. */
5976 ready_try [i] = 0;
5978 if (targetm.sched.first_cycle_multipass_backtrack)
5979 targetm.sched.first_cycle_multipass_backtrack (&top->target_data,
5980 ready_try, n_ready);
5982 top--;
5983 memcpy (state, top->state, dfa_state_size);
5985 else if (!ready_try [i])
5987 tries_num++;
5988 if (tries_num > max_lookahead_tries)
5989 break;
5990 insn = ready_element (ready, i);
5991 delay = state_transition (state, insn);
5992 if (delay < 0)
5994 if (state_dead_lock_p (state)
5995 || insn_finishes_cycle_p (insn))
5996 /* We won't issue any more instructions in the next
5997 choice_state. */
5998 top->rest = 0;
5999 else
6000 top->rest--;
6002 n = top->n;
6003 if (memcmp (top->state, state, dfa_state_size) != 0)
6004 n++;
6006 /* Advance to the next choice_entry. */
6007 top++;
6008 /* Initialize it. */
6009 top->rest = dfa_lookahead;
6010 top->index = i;
6011 top->n = n;
6012 memcpy (top->state, state, dfa_state_size);
6013 ready_try [i] = 1;
6015 if (targetm.sched.first_cycle_multipass_issue)
6016 targetm.sched.first_cycle_multipass_issue (&top->target_data,
6017 ready_try, n_ready,
6018 insn,
6019 &((top - 1)
6020 ->target_data));
6022 i = -1;
6026 /* Increase ready-list index. */
6027 i++;
6030 if (targetm.sched.first_cycle_multipass_end)
6031 targetm.sched.first_cycle_multipass_end (best != 0
6032 ? &choice_stack[1].target_data
6033 : NULL);
6035 /* Restore the original state of the DFA. */
6036 memcpy (state, choice_stack->state, dfa_state_size);
6038 return best;
6041 /* The following function chooses insn from READY and modifies
6042 READY. The following function is used only for first
6043 cycle multipass scheduling.
6044 Return:
6045 -1 if cycle should be advanced,
6046 0 if INSN_PTR is set to point to the desirable insn,
6047 1 if choose_ready () should be restarted without advancing the cycle. */
6048 static int
6049 choose_ready (struct ready_list *ready, bool first_cycle_insn_p,
6050 rtx_insn **insn_ptr)
6052 if (dbg_cnt (sched_insn) == false)
6054 if (nonscheduled_insns_begin == NULL_RTX)
6055 nonscheduled_insns_begin = current_sched_info->prev_head;
6057 rtx_insn *insn = first_nonscheduled_insn ();
6059 if (QUEUE_INDEX (insn) == QUEUE_READY)
6060 /* INSN is in the ready_list. */
6062 ready_remove_insn (insn);
6063 *insn_ptr = insn;
6064 return 0;
6067 /* INSN is in the queue. Advance cycle to move it to the ready list. */
6068 gcc_assert (QUEUE_INDEX (insn) >= 0);
6069 return -1;
6072 if (dfa_lookahead <= 0 || SCHED_GROUP_P (ready_element (ready, 0))
6073 || DEBUG_INSN_P (ready_element (ready, 0)))
6075 if (targetm.sched.dispatch (NULL, IS_DISPATCH_ON))
6076 *insn_ptr = ready_remove_first_dispatch (ready);
6077 else
6078 *insn_ptr = ready_remove_first (ready);
6080 return 0;
6082 else
6084 /* Try to choose the best insn. */
6085 int index = 0, i;
6086 rtx_insn *insn;
6088 insn = ready_element (ready, 0);
6089 if (INSN_CODE (insn) < 0)
6091 *insn_ptr = ready_remove_first (ready);
6092 return 0;
6095 /* Filter the search space. */
6096 for (i = 0; i < ready->n_ready; i++)
6098 ready_try[i] = 0;
6100 insn = ready_element (ready, i);
6102 /* If this insn is recognizable we should have already
6103 recognized it earlier.
6104 ??? Not very clear where this is supposed to be done.
6105 See dep_cost_1. */
6106 gcc_checking_assert (INSN_CODE (insn) >= 0
6107 || recog_memoized (insn) < 0);
6108 if (INSN_CODE (insn) < 0)
6110 /* Non-recognized insns at position 0 are handled above. */
6111 gcc_assert (i > 0);
6112 ready_try[i] = 1;
6113 continue;
6116 if (targetm.sched.first_cycle_multipass_dfa_lookahead_guard)
6118 ready_try[i]
6119 = (targetm.sched.first_cycle_multipass_dfa_lookahead_guard
6120 (insn, i));
6122 if (ready_try[i] < 0)
6123 /* Queue instruction for several cycles.
6124 We need to restart choose_ready as we have changed
6125 the ready list. */
6127 change_queue_index (insn, -ready_try[i]);
6128 return 1;
6131 /* Make sure that we didn't end up with 0'th insn filtered out.
6132 Don't be tempted to make life easier for backends and just
6133 requeue 0'th insn if (ready_try[0] == 0) and restart
6134 choose_ready. Backends should be very considerate about
6135 requeueing instructions -- especially the highest priority
6136 one at position 0. */
6137 gcc_assert (ready_try[i] == 0 || i > 0);
6138 if (ready_try[i])
6139 continue;
6142 gcc_assert (ready_try[i] == 0);
6143 /* INSN made it through the scrutiny of filters! */
6146 if (max_issue (ready, 1, curr_state, first_cycle_insn_p, &index) == 0)
6148 *insn_ptr = ready_remove_first (ready);
6149 if (sched_verbose >= 4)
6150 fprintf (sched_dump, ";;\t\tChosen insn (but can't issue) : %s \n",
6151 (*current_sched_info->print_insn) (*insn_ptr, 0));
6152 return 0;
6154 else
6156 if (sched_verbose >= 4)
6157 fprintf (sched_dump, ";;\t\tChosen insn : %s\n",
6158 (*current_sched_info->print_insn)
6159 (ready_element (ready, index), 0));
6161 *insn_ptr = ready_remove (ready, index);
6162 return 0;
6167 /* This function is called when we have successfully scheduled a
6168 block. It uses the schedule stored in the scheduled_insns vector
6169 to rearrange the RTL. PREV_HEAD is used as the anchor to which we
6170 append the scheduled insns; TAIL is the insn after the scheduled
6171 block. TARGET_BB is the argument passed to schedule_block. */
6173 static void
6174 commit_schedule (rtx_insn *prev_head, rtx_insn *tail, basic_block *target_bb)
6176 unsigned int i;
6177 rtx_insn *insn;
6179 last_scheduled_insn = prev_head;
6180 for (i = 0;
6181 scheduled_insns.iterate (i, &insn);
6182 i++)
6184 if (control_flow_insn_p (last_scheduled_insn)
6185 || current_sched_info->advance_target_bb (*target_bb, insn))
6187 *target_bb = current_sched_info->advance_target_bb (*target_bb, 0);
6189 if (sched_verbose)
6191 rtx_insn *x;
6193 x = next_real_insn (last_scheduled_insn);
6194 gcc_assert (x);
6195 dump_new_block_header (1, *target_bb, x, tail);
6198 last_scheduled_insn = bb_note (*target_bb);
6201 if (current_sched_info->begin_move_insn)
6202 (*current_sched_info->begin_move_insn) (insn, last_scheduled_insn);
6203 move_insn (insn, last_scheduled_insn,
6204 current_sched_info->next_tail);
6205 if (!DEBUG_INSN_P (insn))
6206 reemit_notes (insn);
6207 last_scheduled_insn = insn;
6210 scheduled_insns.truncate (0);
6213 /* Examine all insns on the ready list and queue those which can't be
6214 issued in this cycle. TEMP_STATE is temporary scheduler state we
6215 can use as scratch space. If FIRST_CYCLE_INSN_P is true, no insns
6216 have been issued for the current cycle, which means it is valid to
6217 issue an asm statement.
6219 If SHADOWS_ONLY_P is true, we eliminate all real insns and only
6220 leave those for which SHADOW_P is true. If MODULO_EPILOGUE is true,
6221 we only leave insns which have an INSN_EXACT_TICK. */
6223 static void
6224 prune_ready_list (state_t temp_state, bool first_cycle_insn_p,
6225 bool shadows_only_p, bool modulo_epilogue_p)
6227 int i, pass;
6228 bool sched_group_found = false;
6229 int min_cost_group = 0;
6231 if (sched_fusion)
6232 return;
6234 for (i = 0; i < ready.n_ready; i++)
6236 rtx_insn *insn = ready_element (&ready, i);
6237 if (SCHED_GROUP_P (insn))
6239 sched_group_found = true;
6240 break;
6244 /* Make two passes if there's a SCHED_GROUP_P insn; make sure to handle
6245 such an insn first and note its cost. If at least one SCHED_GROUP_P insn
6246 gets queued, then all other insns get queued for one cycle later. */
6247 for (pass = sched_group_found ? 0 : 1; pass < 2; )
6249 int n = ready.n_ready;
6250 for (i = 0; i < n; i++)
6252 rtx_insn *insn = ready_element (&ready, i);
6253 int cost = 0;
6254 const char *reason = "resource conflict";
6256 if (DEBUG_INSN_P (insn))
6257 continue;
6259 if (sched_group_found && !SCHED_GROUP_P (insn)
6260 && ((pass == 0) || (min_cost_group >= 1)))
6262 if (pass == 0)
6263 continue;
6264 cost = min_cost_group;
6265 reason = "not in sched group";
6267 else if (modulo_epilogue_p
6268 && INSN_EXACT_TICK (insn) == INVALID_TICK)
6270 cost = max_insn_queue_index;
6271 reason = "not an epilogue insn";
6273 else if (shadows_only_p && !SHADOW_P (insn))
6275 cost = 1;
6276 reason = "not a shadow";
6278 else if (recog_memoized (insn) < 0)
6280 if (!first_cycle_insn_p
6281 && (GET_CODE (PATTERN (insn)) == ASM_INPUT
6282 || asm_noperands (PATTERN (insn)) >= 0))
6283 cost = 1;
6284 reason = "asm";
6286 else if (sched_pressure != SCHED_PRESSURE_NONE)
6288 if (sched_pressure == SCHED_PRESSURE_MODEL
6289 && INSN_TICK (insn) <= clock_var)
6291 memcpy (temp_state, curr_state, dfa_state_size);
6292 if (state_transition (temp_state, insn) >= 0)
6293 INSN_TICK (insn) = clock_var + 1;
6295 cost = 0;
6297 else
6299 int delay_cost = 0;
6301 if (delay_htab)
6303 struct delay_pair *delay_entry;
6304 delay_entry
6305 = delay_htab->find_with_hash (insn,
6306 htab_hash_pointer (insn));
6307 while (delay_entry && delay_cost == 0)
6309 delay_cost = estimate_shadow_tick (delay_entry);
6310 if (delay_cost > max_insn_queue_index)
6311 delay_cost = max_insn_queue_index;
6312 delay_entry = delay_entry->next_same_i1;
6316 memcpy (temp_state, curr_state, dfa_state_size);
6317 cost = state_transition (temp_state, insn);
6318 if (cost < 0)
6319 cost = 0;
6320 else if (cost == 0)
6321 cost = 1;
6322 if (cost < delay_cost)
6324 cost = delay_cost;
6325 reason = "shadow tick";
6328 if (cost >= 1)
6330 if (SCHED_GROUP_P (insn) && cost > min_cost_group)
6331 min_cost_group = cost;
6332 ready_remove (&ready, i);
6333 /* Normally we'd want to queue INSN for COST cycles. However,
6334 if SCHED_GROUP_P is set, then we must ensure that nothing
6335 else comes between INSN and its predecessor. If there is
6336 some other insn ready to fire on the next cycle, then that
6337 invariant would be broken.
6339 So when SCHED_GROUP_P is set, just queue this insn for a
6340 single cycle. */
6341 queue_insn (insn, SCHED_GROUP_P (insn) ? 1 : cost, reason);
6342 if (i + 1 < n)
6343 break;
6346 if (i == n)
6347 pass++;
6351 /* Called when we detect that the schedule is impossible. We examine the
6352 backtrack queue to find the earliest insn that caused this condition. */
6354 static struct haifa_saved_data *
6355 verify_shadows (void)
6357 struct haifa_saved_data *save, *earliest_fail = NULL;
6358 for (save = backtrack_queue; save; save = save->next)
6360 int t;
6361 struct delay_pair *pair = save->delay_pair;
6362 rtx_insn *i1 = pair->i1;
6364 for (; pair; pair = pair->next_same_i1)
6366 rtx_insn *i2 = pair->i2;
6368 if (QUEUE_INDEX (i2) == QUEUE_SCHEDULED)
6369 continue;
6371 t = INSN_TICK (i1) + pair_delay (pair);
6372 if (t < clock_var)
6374 if (sched_verbose >= 2)
6375 fprintf (sched_dump,
6376 ";;\t\tfailed delay requirements for %d/%d (%d->%d)"
6377 ", not ready\n",
6378 INSN_UID (pair->i1), INSN_UID (pair->i2),
6379 INSN_TICK (pair->i1), INSN_EXACT_TICK (pair->i2));
6380 earliest_fail = save;
6381 break;
6383 if (QUEUE_INDEX (i2) >= 0)
6385 int queued_for = INSN_TICK (i2);
6387 if (t < queued_for)
6389 if (sched_verbose >= 2)
6390 fprintf (sched_dump,
6391 ";;\t\tfailed delay requirements for %d/%d"
6392 " (%d->%d), queued too late\n",
6393 INSN_UID (pair->i1), INSN_UID (pair->i2),
6394 INSN_TICK (pair->i1), INSN_EXACT_TICK (pair->i2));
6395 earliest_fail = save;
6396 break;
6402 return earliest_fail;
6405 /* Print instructions together with useful scheduling information between
6406 HEAD and TAIL (inclusive). */
6407 static void
6408 dump_insn_stream (rtx_insn *head, rtx_insn *tail)
6410 fprintf (sched_dump, ";;\t| insn | prio |\n");
6412 rtx_insn *next_tail = NEXT_INSN (tail);
6413 for (rtx_insn *insn = head; insn != next_tail; insn = NEXT_INSN (insn))
6415 int priority = NOTE_P (insn) ? 0 : INSN_PRIORITY (insn);
6416 const char *pattern = (NOTE_P (insn)
6417 ? "note"
6418 : str_pattern_slim (PATTERN (insn)));
6420 fprintf (sched_dump, ";;\t| %4d | %4d | %-30s ",
6421 INSN_UID (insn), priority, pattern);
6423 if (sched_verbose >= 4)
6425 if (NOTE_P (insn) || LABEL_P (insn) || recog_memoized (insn) < 0)
6426 fprintf (sched_dump, "nothing");
6427 else
6428 print_reservation (sched_dump, insn);
6430 fprintf (sched_dump, "\n");
6434 /* Use forward list scheduling to rearrange insns of block pointed to by
6435 TARGET_BB, possibly bringing insns from subsequent blocks in the same
6436 region. */
6438 bool
6439 schedule_block (basic_block *target_bb, state_t init_state)
6441 int i;
6442 bool success = modulo_ii == 0;
6443 struct sched_block_state ls;
6444 state_t temp_state = NULL; /* It is used for multipass scheduling. */
6445 int sort_p, advance, start_clock_var;
6447 /* Head/tail info for this block. */
6448 rtx_insn *prev_head = current_sched_info->prev_head;
6449 rtx_insn *next_tail = current_sched_info->next_tail;
6450 rtx_insn *head = NEXT_INSN (prev_head);
6451 rtx_insn *tail = PREV_INSN (next_tail);
6453 if ((current_sched_info->flags & DONT_BREAK_DEPENDENCIES) == 0
6454 && sched_pressure != SCHED_PRESSURE_MODEL && !sched_fusion)
6455 find_modifiable_mems (head, tail);
6457 /* We used to have code to avoid getting parameters moved from hard
6458 argument registers into pseudos.
6460 However, it was removed when it proved to be of marginal benefit
6461 and caused problems because schedule_block and compute_forward_dependences
6462 had different notions of what the "head" insn was. */
6464 gcc_assert (head != tail || INSN_P (head));
6466 haifa_recovery_bb_recently_added_p = false;
6468 backtrack_queue = NULL;
6470 /* Debug info. */
6471 if (sched_verbose)
6473 dump_new_block_header (0, *target_bb, head, tail);
6475 if (sched_verbose >= 2)
6477 dump_insn_stream (head, tail);
6478 memset (&rank_for_schedule_stats, 0,
6479 sizeof (rank_for_schedule_stats));
6483 if (init_state == NULL)
6484 state_reset (curr_state);
6485 else
6486 memcpy (curr_state, init_state, dfa_state_size);
6488 /* Clear the ready list. */
6489 ready.first = ready.veclen - 1;
6490 ready.n_ready = 0;
6491 ready.n_debug = 0;
6493 /* It is used for first cycle multipass scheduling. */
6494 temp_state = alloca (dfa_state_size);
6496 if (targetm.sched.init)
6497 targetm.sched.init (sched_dump, sched_verbose, ready.veclen);
6499 /* We start inserting insns after PREV_HEAD. */
6500 last_scheduled_insn = prev_head;
6501 last_nondebug_scheduled_insn = NULL;
6502 nonscheduled_insns_begin = NULL;
6504 gcc_assert ((NOTE_P (last_scheduled_insn)
6505 || DEBUG_INSN_P (last_scheduled_insn))
6506 && BLOCK_FOR_INSN (last_scheduled_insn) == *target_bb);
6508 /* Initialize INSN_QUEUE. Q_SIZE is the total number of insns in the
6509 queue. */
6510 q_ptr = 0;
6511 q_size = 0;
6513 insn_queue = XALLOCAVEC (rtx_insn_list *, max_insn_queue_index + 1);
6514 memset (insn_queue, 0, (max_insn_queue_index + 1) * sizeof (rtx));
6516 /* Start just before the beginning of time. */
6517 clock_var = -1;
6519 /* We need queue and ready lists and clock_var be initialized
6520 in try_ready () (which is called through init_ready_list ()). */
6521 (*current_sched_info->init_ready_list) ();
6523 if (sched_pressure)
6524 sched_pressure_start_bb (*target_bb);
6526 /* The algorithm is O(n^2) in the number of ready insns at any given
6527 time in the worst case. Before reload we are more likely to have
6528 big lists so truncate them to a reasonable size. */
6529 if (!reload_completed
6530 && ready.n_ready - ready.n_debug > MAX_SCHED_READY_INSNS)
6532 ready_sort_debug (&ready);
6533 ready_sort_real (&ready);
6535 /* Find first free-standing insn past MAX_SCHED_READY_INSNS.
6536 If there are debug insns, we know they're first. */
6537 for (i = MAX_SCHED_READY_INSNS + ready.n_debug; i < ready.n_ready; i++)
6538 if (!SCHED_GROUP_P (ready_element (&ready, i)))
6539 break;
6541 if (sched_verbose >= 2)
6543 fprintf (sched_dump,
6544 ";;\t\tReady list on entry: %d insns: ", ready.n_ready);
6545 debug_ready_list (&ready);
6546 fprintf (sched_dump,
6547 ";;\t\t before reload => truncated to %d insns\n", i);
6550 /* Delay all insns past it for 1 cycle. If debug counter is
6551 activated make an exception for the insn right after
6552 nonscheduled_insns_begin. */
6554 rtx_insn *skip_insn;
6556 if (dbg_cnt (sched_insn) == false)
6557 skip_insn = first_nonscheduled_insn ();
6558 else
6559 skip_insn = NULL;
6561 while (i < ready.n_ready)
6563 rtx_insn *insn;
6565 insn = ready_remove (&ready, i);
6567 if (insn != skip_insn)
6568 queue_insn (insn, 1, "list truncated");
6570 if (skip_insn)
6571 ready_add (&ready, skip_insn, true);
6575 /* Now we can restore basic block notes and maintain precise cfg. */
6576 restore_bb_notes (*target_bb);
6578 last_clock_var = -1;
6580 advance = 0;
6582 gcc_assert (scheduled_insns.length () == 0);
6583 sort_p = TRUE;
6584 must_backtrack = false;
6585 modulo_insns_scheduled = 0;
6587 ls.modulo_epilogue = false;
6588 ls.first_cycle_insn_p = true;
6590 /* Loop until all the insns in BB are scheduled. */
6591 while ((*current_sched_info->schedule_more_p) ())
6593 perform_replacements_new_cycle ();
6596 start_clock_var = clock_var;
6598 clock_var++;
6600 advance_one_cycle ();
6602 /* Add to the ready list all pending insns that can be issued now.
6603 If there are no ready insns, increment clock until one
6604 is ready and add all pending insns at that point to the ready
6605 list. */
6606 queue_to_ready (&ready);
6608 gcc_assert (ready.n_ready);
6610 if (sched_verbose >= 2)
6612 fprintf (sched_dump, ";;\t\tReady list after queue_to_ready:");
6613 debug_ready_list (&ready);
6615 advance -= clock_var - start_clock_var;
6617 while (advance > 0);
6619 if (ls.modulo_epilogue)
6621 int stage = clock_var / modulo_ii;
6622 if (stage > modulo_last_stage * 2 + 2)
6624 if (sched_verbose >= 2)
6625 fprintf (sched_dump,
6626 ";;\t\tmodulo scheduled succeeded at II %d\n",
6627 modulo_ii);
6628 success = true;
6629 goto end_schedule;
6632 else if (modulo_ii > 0)
6634 int stage = clock_var / modulo_ii;
6635 if (stage > modulo_max_stages)
6637 if (sched_verbose >= 2)
6638 fprintf (sched_dump,
6639 ";;\t\tfailing schedule due to excessive stages\n");
6640 goto end_schedule;
6642 if (modulo_n_insns == modulo_insns_scheduled
6643 && stage > modulo_last_stage)
6645 if (sched_verbose >= 2)
6646 fprintf (sched_dump,
6647 ";;\t\tfound kernel after %d stages, II %d\n",
6648 stage, modulo_ii);
6649 ls.modulo_epilogue = true;
6653 prune_ready_list (temp_state, true, false, ls.modulo_epilogue);
6654 if (ready.n_ready == 0)
6655 continue;
6656 if (must_backtrack)
6657 goto do_backtrack;
6659 ls.shadows_only_p = false;
6660 cycle_issued_insns = 0;
6661 ls.can_issue_more = issue_rate;
6662 for (;;)
6664 rtx_insn *insn;
6665 int cost;
6666 bool asm_p;
6668 if (sort_p && ready.n_ready > 0)
6670 /* Sort the ready list based on priority. This must be
6671 done every iteration through the loop, as schedule_insn
6672 may have readied additional insns that will not be
6673 sorted correctly. */
6674 ready_sort (&ready);
6676 if (sched_verbose >= 2)
6678 fprintf (sched_dump,
6679 ";;\t\tReady list after ready_sort: ");
6680 debug_ready_list (&ready);
6684 /* We don't want md sched reorder to even see debug isns, so put
6685 them out right away. */
6686 if (ready.n_ready && DEBUG_INSN_P (ready_element (&ready, 0))
6687 && (*current_sched_info->schedule_more_p) ())
6689 while (ready.n_ready && DEBUG_INSN_P (ready_element (&ready, 0)))
6691 rtx_insn *insn = ready_remove_first (&ready);
6692 gcc_assert (DEBUG_INSN_P (insn));
6693 (*current_sched_info->begin_schedule_ready) (insn);
6694 scheduled_insns.safe_push (insn);
6695 last_scheduled_insn = insn;
6696 advance = schedule_insn (insn);
6697 gcc_assert (advance == 0);
6698 if (ready.n_ready > 0)
6699 ready_sort (&ready);
6703 if (ls.first_cycle_insn_p && !ready.n_ready)
6704 break;
6706 resume_after_backtrack:
6707 /* Allow the target to reorder the list, typically for
6708 better instruction bundling. */
6709 if (sort_p
6710 && (ready.n_ready == 0
6711 || !SCHED_GROUP_P (ready_element (&ready, 0))))
6713 if (ls.first_cycle_insn_p && targetm.sched.reorder)
6714 ls.can_issue_more
6715 = targetm.sched.reorder (sched_dump, sched_verbose,
6716 ready_lastpos (&ready),
6717 &ready.n_ready, clock_var);
6718 else if (!ls.first_cycle_insn_p && targetm.sched.reorder2)
6719 ls.can_issue_more
6720 = targetm.sched.reorder2 (sched_dump, sched_verbose,
6721 ready.n_ready
6722 ? ready_lastpos (&ready) : NULL,
6723 &ready.n_ready, clock_var);
6726 restart_choose_ready:
6727 if (sched_verbose >= 2)
6729 fprintf (sched_dump, ";;\tReady list (t = %3d): ",
6730 clock_var);
6731 debug_ready_list (&ready);
6732 if (sched_pressure == SCHED_PRESSURE_WEIGHTED)
6733 print_curr_reg_pressure ();
6736 if (ready.n_ready == 0
6737 && ls.can_issue_more
6738 && reload_completed)
6740 /* Allow scheduling insns directly from the queue in case
6741 there's nothing better to do (ready list is empty) but
6742 there are still vacant dispatch slots in the current cycle. */
6743 if (sched_verbose >= 6)
6744 fprintf (sched_dump,";;\t\tSecond chance\n");
6745 memcpy (temp_state, curr_state, dfa_state_size);
6746 if (early_queue_to_ready (temp_state, &ready))
6747 ready_sort (&ready);
6750 if (ready.n_ready == 0
6751 || !ls.can_issue_more
6752 || state_dead_lock_p (curr_state)
6753 || !(*current_sched_info->schedule_more_p) ())
6754 break;
6756 /* Select and remove the insn from the ready list. */
6757 if (sort_p)
6759 int res;
6761 insn = NULL;
6762 res = choose_ready (&ready, ls.first_cycle_insn_p, &insn);
6764 if (res < 0)
6765 /* Finish cycle. */
6766 break;
6767 if (res > 0)
6768 goto restart_choose_ready;
6770 gcc_assert (insn != NULL_RTX);
6772 else
6773 insn = ready_remove_first (&ready);
6775 if (sched_pressure != SCHED_PRESSURE_NONE
6776 && INSN_TICK (insn) > clock_var)
6778 ready_add (&ready, insn, true);
6779 advance = 1;
6780 break;
6783 if (targetm.sched.dfa_new_cycle
6784 && targetm.sched.dfa_new_cycle (sched_dump, sched_verbose,
6785 insn, last_clock_var,
6786 clock_var, &sort_p))
6787 /* SORT_P is used by the target to override sorting
6788 of the ready list. This is needed when the target
6789 has modified its internal structures expecting that
6790 the insn will be issued next. As we need the insn
6791 to have the highest priority (so it will be returned by
6792 the ready_remove_first call above), we invoke
6793 ready_add (&ready, insn, true).
6794 But, still, there is one issue: INSN can be later
6795 discarded by scheduler's front end through
6796 current_sched_info->can_schedule_ready_p, hence, won't
6797 be issued next. */
6799 ready_add (&ready, insn, true);
6800 break;
6803 sort_p = TRUE;
6805 if (current_sched_info->can_schedule_ready_p
6806 && ! (*current_sched_info->can_schedule_ready_p) (insn))
6807 /* We normally get here only if we don't want to move
6808 insn from the split block. */
6810 TODO_SPEC (insn) = DEP_POSTPONED;
6811 goto restart_choose_ready;
6814 if (delay_htab)
6816 /* If this insn is the first part of a delay-slot pair, record a
6817 backtrack point. */
6818 struct delay_pair *delay_entry;
6819 delay_entry
6820 = delay_htab->find_with_hash (insn, htab_hash_pointer (insn));
6821 if (delay_entry)
6823 save_backtrack_point (delay_entry, ls);
6824 if (sched_verbose >= 2)
6825 fprintf (sched_dump, ";;\t\tsaving backtrack point\n");
6829 /* DECISION is made. */
6831 if (modulo_ii > 0 && INSN_UID (insn) < modulo_iter0_max_uid)
6833 modulo_insns_scheduled++;
6834 modulo_last_stage = clock_var / modulo_ii;
6836 if (TODO_SPEC (insn) & SPECULATIVE)
6837 generate_recovery_code (insn);
6839 if (targetm.sched.dispatch (NULL, IS_DISPATCH_ON))
6840 targetm.sched.dispatch_do (insn, ADD_TO_DISPATCH_WINDOW);
6842 /* Update counters, etc in the scheduler's front end. */
6843 (*current_sched_info->begin_schedule_ready) (insn);
6844 scheduled_insns.safe_push (insn);
6845 gcc_assert (NONDEBUG_INSN_P (insn));
6846 last_nondebug_scheduled_insn = last_scheduled_insn = insn;
6848 if (recog_memoized (insn) >= 0)
6850 memcpy (temp_state, curr_state, dfa_state_size);
6851 cost = state_transition (curr_state, insn);
6852 if (sched_pressure != SCHED_PRESSURE_WEIGHTED && !sched_fusion)
6853 gcc_assert (cost < 0);
6854 if (memcmp (temp_state, curr_state, dfa_state_size) != 0)
6855 cycle_issued_insns++;
6856 asm_p = false;
6858 else
6859 asm_p = (GET_CODE (PATTERN (insn)) == ASM_INPUT
6860 || asm_noperands (PATTERN (insn)) >= 0);
6862 if (targetm.sched.variable_issue)
6863 ls.can_issue_more =
6864 targetm.sched.variable_issue (sched_dump, sched_verbose,
6865 insn, ls.can_issue_more);
6866 /* A naked CLOBBER or USE generates no instruction, so do
6867 not count them against the issue rate. */
6868 else if (GET_CODE (PATTERN (insn)) != USE
6869 && GET_CODE (PATTERN (insn)) != CLOBBER)
6870 ls.can_issue_more--;
6871 advance = schedule_insn (insn);
6873 if (SHADOW_P (insn))
6874 ls.shadows_only_p = true;
6876 /* After issuing an asm insn we should start a new cycle. */
6877 if (advance == 0 && asm_p)
6878 advance = 1;
6880 if (must_backtrack)
6881 break;
6883 if (advance != 0)
6884 break;
6886 ls.first_cycle_insn_p = false;
6887 if (ready.n_ready > 0)
6888 prune_ready_list (temp_state, false, ls.shadows_only_p,
6889 ls.modulo_epilogue);
6892 do_backtrack:
6893 if (!must_backtrack)
6894 for (i = 0; i < ready.n_ready; i++)
6896 rtx_insn *insn = ready_element (&ready, i);
6897 if (INSN_EXACT_TICK (insn) == clock_var)
6899 must_backtrack = true;
6900 clock_var++;
6901 break;
6904 if (must_backtrack && modulo_ii > 0)
6906 if (modulo_backtracks_left == 0)
6907 goto end_schedule;
6908 modulo_backtracks_left--;
6910 while (must_backtrack)
6912 struct haifa_saved_data *failed;
6913 rtx_insn *failed_insn;
6915 must_backtrack = false;
6916 failed = verify_shadows ();
6917 gcc_assert (failed);
6919 failed_insn = failed->delay_pair->i1;
6920 /* Clear these queues. */
6921 perform_replacements_new_cycle ();
6922 toggle_cancelled_flags (false);
6923 unschedule_insns_until (failed_insn);
6924 while (failed != backtrack_queue)
6925 free_topmost_backtrack_point (true);
6926 restore_last_backtrack_point (&ls);
6927 if (sched_verbose >= 2)
6928 fprintf (sched_dump, ";;\t\trewind to cycle %d\n", clock_var);
6929 /* Delay by at least a cycle. This could cause additional
6930 backtracking. */
6931 queue_insn (failed_insn, 1, "backtracked");
6932 advance = 0;
6933 if (must_backtrack)
6934 continue;
6935 if (ready.n_ready > 0)
6936 goto resume_after_backtrack;
6937 else
6939 if (clock_var == 0 && ls.first_cycle_insn_p)
6940 goto end_schedule;
6941 advance = 1;
6942 break;
6945 ls.first_cycle_insn_p = true;
6947 if (ls.modulo_epilogue)
6948 success = true;
6949 end_schedule:
6950 if (!ls.first_cycle_insn_p || advance)
6951 advance_one_cycle ();
6952 perform_replacements_new_cycle ();
6953 if (modulo_ii > 0)
6955 /* Once again, debug insn suckiness: they can be on the ready list
6956 even if they have unresolved dependencies. To make our view
6957 of the world consistent, remove such "ready" insns. */
6958 restart_debug_insn_loop:
6959 for (i = ready.n_ready - 1; i >= 0; i--)
6961 rtx_insn *x;
6963 x = ready_element (&ready, i);
6964 if (DEPS_LIST_FIRST (INSN_HARD_BACK_DEPS (x)) != NULL
6965 || DEPS_LIST_FIRST (INSN_SPEC_BACK_DEPS (x)) != NULL)
6967 ready_remove (&ready, i);
6968 goto restart_debug_insn_loop;
6971 for (i = ready.n_ready - 1; i >= 0; i--)
6973 rtx_insn *x;
6975 x = ready_element (&ready, i);
6976 resolve_dependencies (x);
6978 for (i = 0; i <= max_insn_queue_index; i++)
6980 rtx_insn_list *link;
6981 while ((link = insn_queue[i]) != NULL)
6983 rtx_insn *x = link->insn ();
6984 insn_queue[i] = link->next ();
6985 QUEUE_INDEX (x) = QUEUE_NOWHERE;
6986 free_INSN_LIST_node (link);
6987 resolve_dependencies (x);
6992 if (!success)
6993 undo_all_replacements ();
6995 /* Debug info. */
6996 if (sched_verbose)
6998 fprintf (sched_dump, ";;\tReady list (final): ");
6999 debug_ready_list (&ready);
7002 if (modulo_ii == 0 && current_sched_info->queue_must_finish_empty)
7003 /* Sanity check -- queue must be empty now. Meaningless if region has
7004 multiple bbs. */
7005 gcc_assert (!q_size && !ready.n_ready && !ready.n_debug);
7006 else if (modulo_ii == 0)
7008 /* We must maintain QUEUE_INDEX between blocks in region. */
7009 for (i = ready.n_ready - 1; i >= 0; i--)
7011 rtx_insn *x;
7013 x = ready_element (&ready, i);
7014 QUEUE_INDEX (x) = QUEUE_NOWHERE;
7015 TODO_SPEC (x) = HARD_DEP;
7018 if (q_size)
7019 for (i = 0; i <= max_insn_queue_index; i++)
7021 rtx_insn_list *link;
7022 for (link = insn_queue[i]; link; link = link->next ())
7024 rtx_insn *x;
7026 x = link->insn ();
7027 QUEUE_INDEX (x) = QUEUE_NOWHERE;
7028 TODO_SPEC (x) = HARD_DEP;
7030 free_INSN_LIST_list (&insn_queue[i]);
7034 if (sched_pressure == SCHED_PRESSURE_MODEL)
7035 model_end_schedule ();
7037 if (success)
7039 commit_schedule (prev_head, tail, target_bb);
7040 if (sched_verbose)
7041 fprintf (sched_dump, ";; total time = %d\n", clock_var);
7043 else
7044 last_scheduled_insn = tail;
7046 scheduled_insns.truncate (0);
7048 if (!current_sched_info->queue_must_finish_empty
7049 || haifa_recovery_bb_recently_added_p)
7051 /* INSN_TICK (minimum clock tick at which the insn becomes
7052 ready) may be not correct for the insn in the subsequent
7053 blocks of the region. We should use a correct value of
7054 `clock_var' or modify INSN_TICK. It is better to keep
7055 clock_var value equal to 0 at the start of a basic block.
7056 Therefore we modify INSN_TICK here. */
7057 fix_inter_tick (NEXT_INSN (prev_head), last_scheduled_insn);
7060 if (targetm.sched.finish)
7062 targetm.sched.finish (sched_dump, sched_verbose);
7063 /* Target might have added some instructions to the scheduled block
7064 in its md_finish () hook. These new insns don't have any data
7065 initialized and to identify them we extend h_i_d so that they'll
7066 get zero luids. */
7067 sched_extend_luids ();
7070 /* Update head/tail boundaries. */
7071 head = NEXT_INSN (prev_head);
7072 tail = last_scheduled_insn;
7074 if (sched_verbose)
7076 fprintf (sched_dump, ";; new head = %d\n;; new tail = %d\n",
7077 INSN_UID (head), INSN_UID (tail));
7079 if (sched_verbose >= 2)
7081 dump_insn_stream (head, tail);
7082 print_rank_for_schedule_stats (";; TOTAL ", &rank_for_schedule_stats,
7083 NULL);
7086 fprintf (sched_dump, "\n");
7089 head = restore_other_notes (head, NULL);
7091 current_sched_info->head = head;
7092 current_sched_info->tail = tail;
7094 free_backtrack_queue ();
7096 return success;
7099 /* Set_priorities: compute priority of each insn in the block. */
7102 set_priorities (rtx_insn *head, rtx_insn *tail)
7104 rtx_insn *insn;
7105 int n_insn;
7106 int sched_max_insns_priority =
7107 current_sched_info->sched_max_insns_priority;
7108 rtx_insn *prev_head;
7110 if (head == tail && ! INSN_P (head))
7111 gcc_unreachable ();
7113 n_insn = 0;
7115 prev_head = PREV_INSN (head);
7116 for (insn = tail; insn != prev_head; insn = PREV_INSN (insn))
7118 if (!INSN_P (insn))
7119 continue;
7121 n_insn++;
7122 (void) priority (insn);
7124 gcc_assert (INSN_PRIORITY_KNOWN (insn));
7126 sched_max_insns_priority = MAX (sched_max_insns_priority,
7127 INSN_PRIORITY (insn));
7130 current_sched_info->sched_max_insns_priority = sched_max_insns_priority;
7132 return n_insn;
7135 /* Set sched_dump and sched_verbose for the desired debugging output. */
7136 void
7137 setup_sched_dump (void)
7139 sched_verbose = sched_verbose_param;
7140 sched_dump = dump_file;
7141 if (!dump_file)
7142 sched_verbose = 0;
7145 /* Allocate data for register pressure sensitive scheduling. */
7146 static void
7147 alloc_global_sched_pressure_data (void)
7149 if (sched_pressure != SCHED_PRESSURE_NONE)
7151 int i, max_regno = max_reg_num ();
7153 if (sched_dump != NULL)
7154 /* We need info about pseudos for rtl dumps about pseudo
7155 classes and costs. */
7156 regstat_init_n_sets_and_refs ();
7157 ira_set_pseudo_classes (true, sched_verbose ? sched_dump : NULL);
7158 sched_regno_pressure_class
7159 = (enum reg_class *) xmalloc (max_regno * sizeof (enum reg_class));
7160 for (i = 0; i < max_regno; i++)
7161 sched_regno_pressure_class[i]
7162 = (i < FIRST_PSEUDO_REGISTER
7163 ? ira_pressure_class_translate[REGNO_REG_CLASS (i)]
7164 : ira_pressure_class_translate[reg_allocno_class (i)]);
7165 curr_reg_live = BITMAP_ALLOC (NULL);
7166 if (sched_pressure == SCHED_PRESSURE_WEIGHTED)
7168 saved_reg_live = BITMAP_ALLOC (NULL);
7169 region_ref_regs = BITMAP_ALLOC (NULL);
7171 if (sched_pressure == SCHED_PRESSURE_MODEL)
7172 tmp_bitmap = BITMAP_ALLOC (NULL);
7174 /* Calculate number of CALL_SAVED_REGS and FIXED_REGS in register classes
7175 that we calculate register pressure for. */
7176 for (int c = 0; c < ira_pressure_classes_num; ++c)
7178 enum reg_class cl = ira_pressure_classes[c];
7180 call_saved_regs_num[cl] = 0;
7181 fixed_regs_num[cl] = 0;
7183 for (int i = 0; i < ira_class_hard_regs_num[cl]; ++i)
7184 if (!call_used_regs[ira_class_hard_regs[cl][i]])
7185 ++call_saved_regs_num[cl];
7186 else if (fixed_regs[ira_class_hard_regs[cl][i]])
7187 ++fixed_regs_num[cl];
7192 /* Free data for register pressure sensitive scheduling. Also called
7193 from schedule_region when stopping sched-pressure early. */
7194 void
7195 free_global_sched_pressure_data (void)
7197 if (sched_pressure != SCHED_PRESSURE_NONE)
7199 if (regstat_n_sets_and_refs != NULL)
7200 regstat_free_n_sets_and_refs ();
7201 if (sched_pressure == SCHED_PRESSURE_WEIGHTED)
7203 BITMAP_FREE (region_ref_regs);
7204 BITMAP_FREE (saved_reg_live);
7206 if (sched_pressure == SCHED_PRESSURE_MODEL)
7207 BITMAP_FREE (tmp_bitmap);
7208 BITMAP_FREE (curr_reg_live);
7209 free (sched_regno_pressure_class);
7213 /* Initialize some global state for the scheduler. This function works
7214 with the common data shared between all the schedulers. It is called
7215 from the scheduler specific initialization routine. */
7217 void
7218 sched_init (void)
7220 /* Disable speculative loads in their presence if cc0 defined. */
7221 if (HAVE_cc0)
7222 flag_schedule_speculative_load = 0;
7224 if (targetm.sched.dispatch (NULL, IS_DISPATCH_ON))
7225 targetm.sched.dispatch_do (NULL, DISPATCH_INIT);
7227 if (live_range_shrinkage_p)
7228 sched_pressure = SCHED_PRESSURE_WEIGHTED;
7229 else if (flag_sched_pressure
7230 && !reload_completed
7231 && common_sched_info->sched_pass_id == SCHED_RGN_PASS)
7232 sched_pressure = ((enum sched_pressure_algorithm)
7233 PARAM_VALUE (PARAM_SCHED_PRESSURE_ALGORITHM));
7234 else
7235 sched_pressure = SCHED_PRESSURE_NONE;
7237 if (sched_pressure != SCHED_PRESSURE_NONE)
7238 ira_setup_eliminable_regset ();
7240 /* Initialize SPEC_INFO. */
7241 if (targetm.sched.set_sched_flags)
7243 spec_info = &spec_info_var;
7244 targetm.sched.set_sched_flags (spec_info);
7246 if (spec_info->mask != 0)
7248 spec_info->data_weakness_cutoff =
7249 (PARAM_VALUE (PARAM_SCHED_SPEC_PROB_CUTOFF) * MAX_DEP_WEAK) / 100;
7250 spec_info->control_weakness_cutoff =
7251 (PARAM_VALUE (PARAM_SCHED_SPEC_PROB_CUTOFF)
7252 * REG_BR_PROB_BASE) / 100;
7254 else
7255 /* So we won't read anything accidentally. */
7256 spec_info = NULL;
7259 else
7260 /* So we won't read anything accidentally. */
7261 spec_info = 0;
7263 /* Initialize issue_rate. */
7264 if (targetm.sched.issue_rate)
7265 issue_rate = targetm.sched.issue_rate ();
7266 else
7267 issue_rate = 1;
7269 if (targetm.sched.first_cycle_multipass_dfa_lookahead
7270 /* Don't use max_issue with reg_pressure scheduling. Multipass
7271 scheduling and reg_pressure scheduling undo each other's decisions. */
7272 && sched_pressure == SCHED_PRESSURE_NONE)
7273 dfa_lookahead = targetm.sched.first_cycle_multipass_dfa_lookahead ();
7274 else
7275 dfa_lookahead = 0;
7277 /* Set to "0" so that we recalculate. */
7278 max_lookahead_tries = 0;
7280 if (targetm.sched.init_dfa_pre_cycle_insn)
7281 targetm.sched.init_dfa_pre_cycle_insn ();
7283 if (targetm.sched.init_dfa_post_cycle_insn)
7284 targetm.sched.init_dfa_post_cycle_insn ();
7286 dfa_start ();
7287 dfa_state_size = state_size ();
7289 init_alias_analysis ();
7291 if (!sched_no_dce)
7292 df_set_flags (DF_LR_RUN_DCE);
7293 df_note_add_problem ();
7295 /* More problems needed for interloop dep calculation in SMS. */
7296 if (common_sched_info->sched_pass_id == SCHED_SMS_PASS)
7298 df_rd_add_problem ();
7299 df_chain_add_problem (DF_DU_CHAIN + DF_UD_CHAIN);
7302 df_analyze ();
7304 /* Do not run DCE after reload, as this can kill nops inserted
7305 by bundling. */
7306 if (reload_completed)
7307 df_clear_flags (DF_LR_RUN_DCE);
7309 regstat_compute_calls_crossed ();
7311 if (targetm.sched.init_global)
7312 targetm.sched.init_global (sched_dump, sched_verbose, get_max_uid () + 1);
7314 alloc_global_sched_pressure_data ();
7316 curr_state = xmalloc (dfa_state_size);
7319 static void haifa_init_only_bb (basic_block, basic_block);
7321 /* Initialize data structures specific to the Haifa scheduler. */
7322 void
7323 haifa_sched_init (void)
7325 setup_sched_dump ();
7326 sched_init ();
7328 scheduled_insns.create (0);
7330 if (spec_info != NULL)
7332 sched_deps_info->use_deps_list = 1;
7333 sched_deps_info->generate_spec_deps = 1;
7336 /* Initialize luids, dependency caches, target and h_i_d for the
7337 whole function. */
7339 sched_init_bbs ();
7341 auto_vec<basic_block> bbs (n_basic_blocks_for_fn (cfun));
7342 basic_block bb;
7343 FOR_EACH_BB_FN (bb, cfun)
7344 bbs.quick_push (bb);
7345 sched_init_luids (bbs);
7346 sched_deps_init (true);
7347 sched_extend_target ();
7348 haifa_init_h_i_d (bbs);
7351 sched_init_only_bb = haifa_init_only_bb;
7352 sched_split_block = sched_split_block_1;
7353 sched_create_empty_bb = sched_create_empty_bb_1;
7354 haifa_recovery_bb_ever_added_p = false;
7356 nr_begin_data = nr_begin_control = nr_be_in_data = nr_be_in_control = 0;
7357 before_recovery = 0;
7358 after_recovery = 0;
7360 modulo_ii = 0;
7363 /* Finish work with the data specific to the Haifa scheduler. */
7364 void
7365 haifa_sched_finish (void)
7367 sched_create_empty_bb = NULL;
7368 sched_split_block = NULL;
7369 sched_init_only_bb = NULL;
7371 if (spec_info && spec_info->dump)
7373 char c = reload_completed ? 'a' : 'b';
7375 fprintf (spec_info->dump,
7376 ";; %s:\n", current_function_name ());
7378 fprintf (spec_info->dump,
7379 ";; Procedure %cr-begin-data-spec motions == %d\n",
7380 c, nr_begin_data);
7381 fprintf (spec_info->dump,
7382 ";; Procedure %cr-be-in-data-spec motions == %d\n",
7383 c, nr_be_in_data);
7384 fprintf (spec_info->dump,
7385 ";; Procedure %cr-begin-control-spec motions == %d\n",
7386 c, nr_begin_control);
7387 fprintf (spec_info->dump,
7388 ";; Procedure %cr-be-in-control-spec motions == %d\n",
7389 c, nr_be_in_control);
7392 scheduled_insns.release ();
7394 /* Finalize h_i_d, dependency caches, and luids for the whole
7395 function. Target will be finalized in md_global_finish (). */
7396 sched_deps_finish ();
7397 sched_finish_luids ();
7398 current_sched_info = NULL;
7399 insn_queue = NULL;
7400 sched_finish ();
7403 /* Free global data used during insn scheduling. This function works with
7404 the common data shared between the schedulers. */
7406 void
7407 sched_finish (void)
7409 haifa_finish_h_i_d ();
7410 free_global_sched_pressure_data ();
7411 free (curr_state);
7413 if (targetm.sched.finish_global)
7414 targetm.sched.finish_global (sched_dump, sched_verbose);
7416 end_alias_analysis ();
7418 regstat_free_calls_crossed ();
7420 dfa_finish ();
7423 /* Free all delay_pair structures that were recorded. */
7424 void
7425 free_delay_pairs (void)
7427 if (delay_htab)
7429 delay_htab->empty ();
7430 delay_htab_i2->empty ();
7434 /* Fix INSN_TICKs of the instructions in the current block as well as
7435 INSN_TICKs of their dependents.
7436 HEAD and TAIL are the begin and the end of the current scheduled block. */
7437 static void
7438 fix_inter_tick (rtx_insn *head, rtx_insn *tail)
7440 /* Set of instructions with corrected INSN_TICK. */
7441 auto_bitmap processed;
7442 /* ??? It is doubtful if we should assume that cycle advance happens on
7443 basic block boundaries. Basically insns that are unconditionally ready
7444 on the start of the block are more preferable then those which have
7445 a one cycle dependency over insn from the previous block. */
7446 int next_clock = clock_var + 1;
7448 /* Iterates over scheduled instructions and fix their INSN_TICKs and
7449 INSN_TICKs of dependent instructions, so that INSN_TICKs are consistent
7450 across different blocks. */
7451 for (tail = NEXT_INSN (tail); head != tail; head = NEXT_INSN (head))
7453 if (INSN_P (head))
7455 int tick;
7456 sd_iterator_def sd_it;
7457 dep_t dep;
7459 tick = INSN_TICK (head);
7460 gcc_assert (tick >= MIN_TICK);
7462 /* Fix INSN_TICK of instruction from just scheduled block. */
7463 if (bitmap_set_bit (processed, INSN_LUID (head)))
7465 tick -= next_clock;
7467 if (tick < MIN_TICK)
7468 tick = MIN_TICK;
7470 INSN_TICK (head) = tick;
7473 if (DEBUG_INSN_P (head))
7474 continue;
7476 FOR_EACH_DEP (head, SD_LIST_RES_FORW, sd_it, dep)
7478 rtx_insn *next;
7480 next = DEP_CON (dep);
7481 tick = INSN_TICK (next);
7483 if (tick != INVALID_TICK
7484 /* If NEXT has its INSN_TICK calculated, fix it.
7485 If not - it will be properly calculated from
7486 scratch later in fix_tick_ready. */
7487 && bitmap_set_bit (processed, INSN_LUID (next)))
7489 tick -= next_clock;
7491 if (tick < MIN_TICK)
7492 tick = MIN_TICK;
7494 if (tick > INTER_TICK (next))
7495 INTER_TICK (next) = tick;
7496 else
7497 tick = INTER_TICK (next);
7499 INSN_TICK (next) = tick;
7506 /* Check if NEXT is ready to be added to the ready or queue list.
7507 If "yes", add it to the proper list.
7508 Returns:
7509 -1 - is not ready yet,
7510 0 - added to the ready list,
7511 0 < N - queued for N cycles. */
7513 try_ready (rtx_insn *next)
7515 ds_t old_ts, new_ts;
7517 old_ts = TODO_SPEC (next);
7519 gcc_assert (!(old_ts & ~(SPECULATIVE | HARD_DEP | DEP_CONTROL | DEP_POSTPONED))
7520 && (old_ts == HARD_DEP
7521 || old_ts == DEP_POSTPONED
7522 || (old_ts & SPECULATIVE)
7523 || old_ts == DEP_CONTROL));
7525 new_ts = recompute_todo_spec (next, false);
7527 if (new_ts & (HARD_DEP | DEP_POSTPONED))
7528 gcc_assert (new_ts == old_ts
7529 && QUEUE_INDEX (next) == QUEUE_NOWHERE);
7530 else if (current_sched_info->new_ready)
7531 new_ts = current_sched_info->new_ready (next, new_ts);
7533 /* * if !(old_ts & SPECULATIVE) (e.g. HARD_DEP or 0), then insn might
7534 have its original pattern or changed (speculative) one. This is due
7535 to changing ebb in region scheduling.
7536 * But if (old_ts & SPECULATIVE), then we are pretty sure that insn
7537 has speculative pattern.
7539 We can't assert (!(new_ts & HARD_DEP) || new_ts == old_ts) here because
7540 control-speculative NEXT could have been discarded by sched-rgn.c
7541 (the same case as when discarded by can_schedule_ready_p ()). */
7543 if ((new_ts & SPECULATIVE)
7544 /* If (old_ts == new_ts), then (old_ts & SPECULATIVE) and we don't
7545 need to change anything. */
7546 && new_ts != old_ts)
7548 int res;
7549 rtx new_pat;
7551 gcc_assert ((new_ts & SPECULATIVE) && !(new_ts & ~SPECULATIVE));
7553 res = haifa_speculate_insn (next, new_ts, &new_pat);
7555 switch (res)
7557 case -1:
7558 /* It would be nice to change DEP_STATUS of all dependences,
7559 which have ((DEP_STATUS & SPECULATIVE) == new_ts) to HARD_DEP,
7560 so we won't reanalyze anything. */
7561 new_ts = HARD_DEP;
7562 break;
7564 case 0:
7565 /* We follow the rule, that every speculative insn
7566 has non-null ORIG_PAT. */
7567 if (!ORIG_PAT (next))
7568 ORIG_PAT (next) = PATTERN (next);
7569 break;
7571 case 1:
7572 if (!ORIG_PAT (next))
7573 /* If we gonna to overwrite the original pattern of insn,
7574 save it. */
7575 ORIG_PAT (next) = PATTERN (next);
7577 res = haifa_change_pattern (next, new_pat);
7578 gcc_assert (res);
7579 break;
7581 default:
7582 gcc_unreachable ();
7586 /* We need to restore pattern only if (new_ts == 0), because otherwise it is
7587 either correct (new_ts & SPECULATIVE),
7588 or we simply don't care (new_ts & HARD_DEP). */
7590 gcc_assert (!ORIG_PAT (next)
7591 || !IS_SPECULATION_BRANCHY_CHECK_P (next));
7593 TODO_SPEC (next) = new_ts;
7595 if (new_ts & (HARD_DEP | DEP_POSTPONED))
7597 /* We can't assert (QUEUE_INDEX (next) == QUEUE_NOWHERE) here because
7598 control-speculative NEXT could have been discarded by sched-rgn.c
7599 (the same case as when discarded by can_schedule_ready_p ()). */
7600 /*gcc_assert (QUEUE_INDEX (next) == QUEUE_NOWHERE);*/
7602 change_queue_index (next, QUEUE_NOWHERE);
7604 return -1;
7606 else if (!(new_ts & BEGIN_SPEC)
7607 && ORIG_PAT (next) && PREDICATED_PAT (next) == NULL_RTX
7608 && !IS_SPECULATION_CHECK_P (next))
7609 /* We should change pattern of every previously speculative
7610 instruction - and we determine if NEXT was speculative by using
7611 ORIG_PAT field. Except one case - speculation checks have ORIG_PAT
7612 pat too, so skip them. */
7614 bool success = haifa_change_pattern (next, ORIG_PAT (next));
7615 gcc_assert (success);
7616 ORIG_PAT (next) = 0;
7619 if (sched_verbose >= 2)
7621 fprintf (sched_dump, ";;\t\tdependencies resolved: insn %s",
7622 (*current_sched_info->print_insn) (next, 0));
7624 if (spec_info && spec_info->dump)
7626 if (new_ts & BEGIN_DATA)
7627 fprintf (spec_info->dump, "; data-spec;");
7628 if (new_ts & BEGIN_CONTROL)
7629 fprintf (spec_info->dump, "; control-spec;");
7630 if (new_ts & BE_IN_CONTROL)
7631 fprintf (spec_info->dump, "; in-control-spec;");
7633 if (TODO_SPEC (next) & DEP_CONTROL)
7634 fprintf (sched_dump, " predicated");
7635 fprintf (sched_dump, "\n");
7638 adjust_priority (next);
7640 return fix_tick_ready (next);
7643 /* Calculate INSN_TICK of NEXT and add it to either ready or queue list. */
7644 static int
7645 fix_tick_ready (rtx_insn *next)
7647 int tick, delay;
7649 if (!DEBUG_INSN_P (next) && !sd_lists_empty_p (next, SD_LIST_RES_BACK))
7651 int full_p;
7652 sd_iterator_def sd_it;
7653 dep_t dep;
7655 tick = INSN_TICK (next);
7656 /* if tick is not equal to INVALID_TICK, then update
7657 INSN_TICK of NEXT with the most recent resolved dependence
7658 cost. Otherwise, recalculate from scratch. */
7659 full_p = (tick == INVALID_TICK);
7661 FOR_EACH_DEP (next, SD_LIST_RES_BACK, sd_it, dep)
7663 rtx_insn *pro = DEP_PRO (dep);
7664 int tick1;
7666 gcc_assert (INSN_TICK (pro) >= MIN_TICK);
7668 tick1 = INSN_TICK (pro) + dep_cost (dep);
7669 if (tick1 > tick)
7670 tick = tick1;
7672 if (!full_p)
7673 break;
7676 else
7677 tick = -1;
7679 INSN_TICK (next) = tick;
7681 delay = tick - clock_var;
7682 if (delay <= 0 || sched_pressure != SCHED_PRESSURE_NONE || sched_fusion)
7683 delay = QUEUE_READY;
7685 change_queue_index (next, delay);
7687 return delay;
7690 /* Move NEXT to the proper queue list with (DELAY >= 1),
7691 or add it to the ready list (DELAY == QUEUE_READY),
7692 or remove it from ready and queue lists at all (DELAY == QUEUE_NOWHERE). */
7693 static void
7694 change_queue_index (rtx_insn *next, int delay)
7696 int i = QUEUE_INDEX (next);
7698 gcc_assert (QUEUE_NOWHERE <= delay && delay <= max_insn_queue_index
7699 && delay != 0);
7700 gcc_assert (i != QUEUE_SCHEDULED);
7702 if ((delay > 0 && NEXT_Q_AFTER (q_ptr, delay) == i)
7703 || (delay < 0 && delay == i))
7704 /* We have nothing to do. */
7705 return;
7707 /* Remove NEXT from wherever it is now. */
7708 if (i == QUEUE_READY)
7709 ready_remove_insn (next);
7710 else if (i >= 0)
7711 queue_remove (next);
7713 /* Add it to the proper place. */
7714 if (delay == QUEUE_READY)
7715 ready_add (readyp, next, false);
7716 else if (delay >= 1)
7717 queue_insn (next, delay, "change queue index");
7719 if (sched_verbose >= 2)
7721 fprintf (sched_dump, ";;\t\ttick updated: insn %s",
7722 (*current_sched_info->print_insn) (next, 0));
7724 if (delay == QUEUE_READY)
7725 fprintf (sched_dump, " into ready\n");
7726 else if (delay >= 1)
7727 fprintf (sched_dump, " into queue with cost=%d\n", delay);
7728 else
7729 fprintf (sched_dump, " removed from ready or queue lists\n");
7733 static int sched_ready_n_insns = -1;
7735 /* Initialize per region data structures. */
7736 void
7737 sched_extend_ready_list (int new_sched_ready_n_insns)
7739 int i;
7741 if (sched_ready_n_insns == -1)
7742 /* At the first call we need to initialize one more choice_stack
7743 entry. */
7745 i = 0;
7746 sched_ready_n_insns = 0;
7747 scheduled_insns.reserve (new_sched_ready_n_insns);
7749 else
7750 i = sched_ready_n_insns + 1;
7752 ready.veclen = new_sched_ready_n_insns + issue_rate;
7753 ready.vec = XRESIZEVEC (rtx_insn *, ready.vec, ready.veclen);
7755 gcc_assert (new_sched_ready_n_insns >= sched_ready_n_insns);
7757 ready_try = (signed char *) xrecalloc (ready_try, new_sched_ready_n_insns,
7758 sched_ready_n_insns,
7759 sizeof (*ready_try));
7761 /* We allocate +1 element to save initial state in the choice_stack[0]
7762 entry. */
7763 choice_stack = XRESIZEVEC (struct choice_entry, choice_stack,
7764 new_sched_ready_n_insns + 1);
7766 for (; i <= new_sched_ready_n_insns; i++)
7768 choice_stack[i].state = xmalloc (dfa_state_size);
7770 if (targetm.sched.first_cycle_multipass_init)
7771 targetm.sched.first_cycle_multipass_init (&(choice_stack[i]
7772 .target_data));
7775 sched_ready_n_insns = new_sched_ready_n_insns;
7778 /* Free per region data structures. */
7779 void
7780 sched_finish_ready_list (void)
7782 int i;
7784 free (ready.vec);
7785 ready.vec = NULL;
7786 ready.veclen = 0;
7788 free (ready_try);
7789 ready_try = NULL;
7791 for (i = 0; i <= sched_ready_n_insns; i++)
7793 if (targetm.sched.first_cycle_multipass_fini)
7794 targetm.sched.first_cycle_multipass_fini (&(choice_stack[i]
7795 .target_data));
7797 free (choice_stack [i].state);
7799 free (choice_stack);
7800 choice_stack = NULL;
7802 sched_ready_n_insns = -1;
7805 static int
7806 haifa_luid_for_non_insn (rtx x)
7808 gcc_assert (NOTE_P (x) || LABEL_P (x));
7810 return 0;
7813 /* Generates recovery code for INSN. */
7814 static void
7815 generate_recovery_code (rtx_insn *insn)
7817 if (TODO_SPEC (insn) & BEGIN_SPEC)
7818 begin_speculative_block (insn);
7820 /* Here we have insn with no dependencies to
7821 instructions other then CHECK_SPEC ones. */
7823 if (TODO_SPEC (insn) & BE_IN_SPEC)
7824 add_to_speculative_block (insn);
7827 /* Helper function.
7828 Tries to add speculative dependencies of type FS between instructions
7829 in deps_list L and TWIN. */
7830 static void
7831 process_insn_forw_deps_be_in_spec (rtx_insn *insn, rtx_insn *twin, ds_t fs)
7833 sd_iterator_def sd_it;
7834 dep_t dep;
7836 FOR_EACH_DEP (insn, SD_LIST_FORW, sd_it, dep)
7838 ds_t ds;
7839 rtx_insn *consumer;
7841 consumer = DEP_CON (dep);
7843 ds = DEP_STATUS (dep);
7845 if (/* If we want to create speculative dep. */
7847 /* And we can do that because this is a true dep. */
7848 && (ds & DEP_TYPES) == DEP_TRUE)
7850 gcc_assert (!(ds & BE_IN_SPEC));
7852 if (/* If this dep can be overcome with 'begin speculation'. */
7853 ds & BEGIN_SPEC)
7854 /* Then we have a choice: keep the dep 'begin speculative'
7855 or transform it into 'be in speculative'. */
7857 if (/* In try_ready we assert that if insn once became ready
7858 it can be removed from the ready (or queue) list only
7859 due to backend decision. Hence we can't let the
7860 probability of the speculative dep to decrease. */
7861 ds_weak (ds) <= ds_weak (fs))
7863 ds_t new_ds;
7865 new_ds = (ds & ~BEGIN_SPEC) | fs;
7867 if (/* consumer can 'be in speculative'. */
7868 sched_insn_is_legitimate_for_speculation_p (consumer,
7869 new_ds))
7870 /* Transform it to be in speculative. */
7871 ds = new_ds;
7874 else
7875 /* Mark the dep as 'be in speculative'. */
7876 ds |= fs;
7880 dep_def _new_dep, *new_dep = &_new_dep;
7882 init_dep_1 (new_dep, twin, consumer, DEP_TYPE (dep), ds);
7883 sd_add_dep (new_dep, false);
7888 /* Generates recovery code for BEGIN speculative INSN. */
7889 static void
7890 begin_speculative_block (rtx_insn *insn)
7892 if (TODO_SPEC (insn) & BEGIN_DATA)
7893 nr_begin_data++;
7894 if (TODO_SPEC (insn) & BEGIN_CONTROL)
7895 nr_begin_control++;
7897 create_check_block_twin (insn, false);
7899 TODO_SPEC (insn) &= ~BEGIN_SPEC;
7902 static void haifa_init_insn (rtx_insn *);
7904 /* Generates recovery code for BE_IN speculative INSN. */
7905 static void
7906 add_to_speculative_block (rtx_insn *insn)
7908 ds_t ts;
7909 sd_iterator_def sd_it;
7910 dep_t dep;
7911 auto_vec<rtx_insn *, 10> twins;
7913 ts = TODO_SPEC (insn);
7914 gcc_assert (!(ts & ~BE_IN_SPEC));
7916 if (ts & BE_IN_DATA)
7917 nr_be_in_data++;
7918 if (ts & BE_IN_CONTROL)
7919 nr_be_in_control++;
7921 TODO_SPEC (insn) &= ~BE_IN_SPEC;
7922 gcc_assert (!TODO_SPEC (insn));
7924 DONE_SPEC (insn) |= ts;
7926 /* First we convert all simple checks to branchy. */
7927 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
7928 sd_iterator_cond (&sd_it, &dep);)
7930 rtx_insn *check = DEP_PRO (dep);
7932 if (IS_SPECULATION_SIMPLE_CHECK_P (check))
7934 create_check_block_twin (check, true);
7936 /* Restart search. */
7937 sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
7939 else
7940 /* Continue search. */
7941 sd_iterator_next (&sd_it);
7944 auto_vec<rtx_insn *> priorities_roots;
7945 clear_priorities (insn, &priorities_roots);
7947 while (1)
7949 rtx_insn *check, *twin;
7950 basic_block rec;
7952 /* Get the first backward dependency of INSN. */
7953 sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
7954 if (!sd_iterator_cond (&sd_it, &dep))
7955 /* INSN has no backward dependencies left. */
7956 break;
7958 gcc_assert ((DEP_STATUS (dep) & BEGIN_SPEC) == 0
7959 && (DEP_STATUS (dep) & BE_IN_SPEC) != 0
7960 && (DEP_STATUS (dep) & DEP_TYPES) == DEP_TRUE);
7962 check = DEP_PRO (dep);
7964 gcc_assert (!IS_SPECULATION_CHECK_P (check) && !ORIG_PAT (check)
7965 && QUEUE_INDEX (check) == QUEUE_NOWHERE);
7967 rec = BLOCK_FOR_INSN (check);
7969 twin = emit_insn_before (copy_insn (PATTERN (insn)), BB_END (rec));
7970 haifa_init_insn (twin);
7972 sd_copy_back_deps (twin, insn, true);
7974 if (sched_verbose && spec_info->dump)
7975 /* INSN_BB (insn) isn't determined for twin insns yet.
7976 So we can't use current_sched_info->print_insn. */
7977 fprintf (spec_info->dump, ";;\t\tGenerated twin insn : %d/rec%d\n",
7978 INSN_UID (twin), rec->index);
7980 twins.safe_push (twin);
7982 /* Add dependences between TWIN and all appropriate
7983 instructions from REC. */
7984 FOR_EACH_DEP (insn, SD_LIST_SPEC_BACK, sd_it, dep)
7986 rtx_insn *pro = DEP_PRO (dep);
7988 gcc_assert (DEP_TYPE (dep) == REG_DEP_TRUE);
7990 /* INSN might have dependencies from the instructions from
7991 several recovery blocks. At this iteration we process those
7992 producers that reside in REC. */
7993 if (BLOCK_FOR_INSN (pro) == rec)
7995 dep_def _new_dep, *new_dep = &_new_dep;
7997 init_dep (new_dep, pro, twin, REG_DEP_TRUE);
7998 sd_add_dep (new_dep, false);
8002 process_insn_forw_deps_be_in_spec (insn, twin, ts);
8004 /* Remove all dependencies between INSN and insns in REC. */
8005 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
8006 sd_iterator_cond (&sd_it, &dep);)
8008 rtx_insn *pro = DEP_PRO (dep);
8010 if (BLOCK_FOR_INSN (pro) == rec)
8011 sd_delete_dep (sd_it);
8012 else
8013 sd_iterator_next (&sd_it);
8017 /* We couldn't have added the dependencies between INSN and TWINS earlier
8018 because that would make TWINS appear in the INSN_BACK_DEPS (INSN). */
8019 unsigned int i;
8020 rtx_insn *twin;
8021 FOR_EACH_VEC_ELT_REVERSE (twins, i, twin)
8023 dep_def _new_dep, *new_dep = &_new_dep;
8025 init_dep (new_dep, insn, twin, REG_DEP_OUTPUT);
8026 sd_add_dep (new_dep, false);
8029 calc_priorities (priorities_roots);
8032 /* Extends and fills with zeros (only the new part) array pointed to by P. */
8033 void *
8034 xrecalloc (void *p, size_t new_nmemb, size_t old_nmemb, size_t size)
8036 gcc_assert (new_nmemb >= old_nmemb);
8037 p = XRESIZEVAR (void, p, new_nmemb * size);
8038 memset (((char *) p) + old_nmemb * size, 0, (new_nmemb - old_nmemb) * size);
8039 return p;
8042 /* Helper function.
8043 Find fallthru edge from PRED. */
8044 edge
8045 find_fallthru_edge_from (basic_block pred)
8047 edge e;
8048 basic_block succ;
8050 succ = pred->next_bb;
8051 gcc_assert (succ->prev_bb == pred);
8053 if (EDGE_COUNT (pred->succs) <= EDGE_COUNT (succ->preds))
8055 e = find_fallthru_edge (pred->succs);
8057 if (e)
8059 gcc_assert (e->dest == succ);
8060 return e;
8063 else
8065 e = find_fallthru_edge (succ->preds);
8067 if (e)
8069 gcc_assert (e->src == pred);
8070 return e;
8074 return NULL;
8077 /* Extend per basic block data structures. */
8078 static void
8079 sched_extend_bb (void)
8081 /* The following is done to keep current_sched_info->next_tail non null. */
8082 rtx_insn *end = BB_END (EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb);
8083 rtx_insn *insn = DEBUG_INSN_P (end) ? prev_nondebug_insn (end) : end;
8084 if (NEXT_INSN (end) == 0
8085 || (!NOTE_P (insn)
8086 && !LABEL_P (insn)
8087 /* Don't emit a NOTE if it would end up before a BARRIER. */
8088 && !BARRIER_P (next_nondebug_insn (end))))
8090 rtx_note *note = emit_note_after (NOTE_INSN_DELETED, end);
8091 /* Make note appear outside BB. */
8092 set_block_for_insn (note, NULL);
8093 BB_END (EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb) = end;
8097 /* Init per basic block data structures. */
8098 void
8099 sched_init_bbs (void)
8101 sched_extend_bb ();
8104 /* Initialize BEFORE_RECOVERY variable. */
8105 static void
8106 init_before_recovery (basic_block *before_recovery_ptr)
8108 basic_block last;
8109 edge e;
8111 last = EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb;
8112 e = find_fallthru_edge_from (last);
8114 if (e)
8116 /* We create two basic blocks:
8117 1. Single instruction block is inserted right after E->SRC
8118 and has jump to
8119 2. Empty block right before EXIT_BLOCK.
8120 Between these two blocks recovery blocks will be emitted. */
8122 basic_block single, empty;
8124 /* If the fallthrough edge to exit we've found is from the block we've
8125 created before, don't do anything more. */
8126 if (last == after_recovery)
8127 return;
8129 adding_bb_to_current_region_p = false;
8131 single = sched_create_empty_bb (last);
8132 empty = sched_create_empty_bb (single);
8134 /* Add new blocks to the root loop. */
8135 if (current_loops != NULL)
8137 add_bb_to_loop (single, (*current_loops->larray)[0]);
8138 add_bb_to_loop (empty, (*current_loops->larray)[0]);
8141 single->count = last->count;
8142 empty->count = last->count;
8143 BB_COPY_PARTITION (single, last);
8144 BB_COPY_PARTITION (empty, last);
8146 redirect_edge_succ (e, single);
8147 make_single_succ_edge (single, empty, 0);
8148 make_single_succ_edge (empty, EXIT_BLOCK_PTR_FOR_FN (cfun),
8149 EDGE_FALLTHRU);
8151 rtx_code_label *label = block_label (empty);
8152 rtx_jump_insn *x = emit_jump_insn_after (targetm.gen_jump (label),
8153 BB_END (single));
8154 JUMP_LABEL (x) = label;
8155 LABEL_NUSES (label)++;
8156 haifa_init_insn (x);
8158 emit_barrier_after (x);
8160 sched_init_only_bb (empty, NULL);
8161 sched_init_only_bb (single, NULL);
8162 sched_extend_bb ();
8164 adding_bb_to_current_region_p = true;
8165 before_recovery = single;
8166 after_recovery = empty;
8168 if (before_recovery_ptr)
8169 *before_recovery_ptr = before_recovery;
8171 if (sched_verbose >= 2 && spec_info->dump)
8172 fprintf (spec_info->dump,
8173 ";;\t\tFixed fallthru to EXIT : %d->>%d->%d->>EXIT\n",
8174 last->index, single->index, empty->index);
8176 else
8177 before_recovery = last;
8180 /* Returns new recovery block. */
8181 basic_block
8182 sched_create_recovery_block (basic_block *before_recovery_ptr)
8184 rtx_insn *barrier;
8185 basic_block rec;
8187 haifa_recovery_bb_recently_added_p = true;
8188 haifa_recovery_bb_ever_added_p = true;
8190 init_before_recovery (before_recovery_ptr);
8192 barrier = get_last_bb_insn (before_recovery);
8193 gcc_assert (BARRIER_P (barrier));
8195 rtx_insn *label = emit_label_after (gen_label_rtx (), barrier);
8197 rec = create_basic_block (label, label, before_recovery);
8199 /* A recovery block always ends with an unconditional jump. */
8200 emit_barrier_after (BB_END (rec));
8202 if (BB_PARTITION (before_recovery) != BB_UNPARTITIONED)
8203 BB_SET_PARTITION (rec, BB_COLD_PARTITION);
8205 if (sched_verbose && spec_info->dump)
8206 fprintf (spec_info->dump, ";;\t\tGenerated recovery block rec%d\n",
8207 rec->index);
8209 return rec;
8212 /* Create edges: FIRST_BB -> REC; FIRST_BB -> SECOND_BB; REC -> SECOND_BB
8213 and emit necessary jumps. */
8214 void
8215 sched_create_recovery_edges (basic_block first_bb, basic_block rec,
8216 basic_block second_bb)
8218 int edge_flags;
8220 /* This is fixing of incoming edge. */
8221 /* ??? Which other flags should be specified? */
8222 if (BB_PARTITION (first_bb) != BB_PARTITION (rec))
8223 /* Partition type is the same, if it is "unpartitioned". */
8224 edge_flags = EDGE_CROSSING;
8225 else
8226 edge_flags = 0;
8228 edge e2 = single_succ_edge (first_bb);
8229 edge e = make_edge (first_bb, rec, edge_flags);
8231 /* TODO: The actual probability can be determined and is computed as
8232 'todo_spec' variable in create_check_block_twin and
8233 in sel-sched.c `check_ds' in create_speculation_check. */
8234 e->probability = profile_probability::very_unlikely ();
8235 rec->count = e->count ();
8236 e2->probability = e->probability.invert ();
8238 rtx_code_label *label = block_label (second_bb);
8239 rtx_jump_insn *jump = emit_jump_insn_after (targetm.gen_jump (label),
8240 BB_END (rec));
8241 JUMP_LABEL (jump) = label;
8242 LABEL_NUSES (label)++;
8244 if (BB_PARTITION (second_bb) != BB_PARTITION (rec))
8245 /* Partition type is the same, if it is "unpartitioned". */
8247 /* Rewritten from cfgrtl.c. */
8248 if (crtl->has_bb_partition && targetm_common.have_named_sections)
8250 /* We don't need the same note for the check because
8251 any_condjump_p (check) == true. */
8252 CROSSING_JUMP_P (jump) = 1;
8254 edge_flags = EDGE_CROSSING;
8256 else
8257 edge_flags = 0;
8259 make_single_succ_edge (rec, second_bb, edge_flags);
8260 if (dom_info_available_p (CDI_DOMINATORS))
8261 set_immediate_dominator (CDI_DOMINATORS, rec, first_bb);
8264 /* This function creates recovery code for INSN. If MUTATE_P is nonzero,
8265 INSN is a simple check, that should be converted to branchy one. */
8266 static void
8267 create_check_block_twin (rtx_insn *insn, bool mutate_p)
8269 basic_block rec;
8270 rtx_insn *label, *check, *twin;
8271 rtx check_pat;
8272 ds_t fs;
8273 sd_iterator_def sd_it;
8274 dep_t dep;
8275 dep_def _new_dep, *new_dep = &_new_dep;
8276 ds_t todo_spec;
8278 gcc_assert (ORIG_PAT (insn) != NULL_RTX);
8280 if (!mutate_p)
8281 todo_spec = TODO_SPEC (insn);
8282 else
8284 gcc_assert (IS_SPECULATION_SIMPLE_CHECK_P (insn)
8285 && (TODO_SPEC (insn) & SPECULATIVE) == 0);
8287 todo_spec = CHECK_SPEC (insn);
8290 todo_spec &= SPECULATIVE;
8292 /* Create recovery block. */
8293 if (mutate_p || targetm.sched.needs_block_p (todo_spec))
8295 rec = sched_create_recovery_block (NULL);
8296 label = BB_HEAD (rec);
8298 else
8300 rec = EXIT_BLOCK_PTR_FOR_FN (cfun);
8301 label = NULL;
8304 /* Emit CHECK. */
8305 check_pat = targetm.sched.gen_spec_check (insn, label, todo_spec);
8307 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun))
8309 /* To have mem_reg alive at the beginning of second_bb,
8310 we emit check BEFORE insn, so insn after splitting
8311 insn will be at the beginning of second_bb, which will
8312 provide us with the correct life information. */
8313 check = emit_jump_insn_before (check_pat, insn);
8314 JUMP_LABEL (check) = label;
8315 LABEL_NUSES (label)++;
8317 else
8318 check = emit_insn_before (check_pat, insn);
8320 /* Extend data structures. */
8321 haifa_init_insn (check);
8323 /* CHECK is being added to current region. Extend ready list. */
8324 gcc_assert (sched_ready_n_insns != -1);
8325 sched_extend_ready_list (sched_ready_n_insns + 1);
8327 if (current_sched_info->add_remove_insn)
8328 current_sched_info->add_remove_insn (insn, 0);
8330 RECOVERY_BLOCK (check) = rec;
8332 if (sched_verbose && spec_info->dump)
8333 fprintf (spec_info->dump, ";;\t\tGenerated check insn : %s\n",
8334 (*current_sched_info->print_insn) (check, 0));
8336 gcc_assert (ORIG_PAT (insn));
8338 /* Initialize TWIN (twin is a duplicate of original instruction
8339 in the recovery block). */
8340 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun))
8342 sd_iterator_def sd_it;
8343 dep_t dep;
8345 FOR_EACH_DEP (insn, SD_LIST_RES_BACK, sd_it, dep)
8346 if ((DEP_STATUS (dep) & DEP_OUTPUT) != 0)
8348 struct _dep _dep2, *dep2 = &_dep2;
8350 init_dep (dep2, DEP_PRO (dep), check, REG_DEP_TRUE);
8352 sd_add_dep (dep2, true);
8355 twin = emit_insn_after (ORIG_PAT (insn), BB_END (rec));
8356 haifa_init_insn (twin);
8358 if (sched_verbose && spec_info->dump)
8359 /* INSN_BB (insn) isn't determined for twin insns yet.
8360 So we can't use current_sched_info->print_insn. */
8361 fprintf (spec_info->dump, ";;\t\tGenerated twin insn : %d/rec%d\n",
8362 INSN_UID (twin), rec->index);
8364 else
8366 ORIG_PAT (check) = ORIG_PAT (insn);
8367 HAS_INTERNAL_DEP (check) = 1;
8368 twin = check;
8369 /* ??? We probably should change all OUTPUT dependencies to
8370 (TRUE | OUTPUT). */
8373 /* Copy all resolved back dependencies of INSN to TWIN. This will
8374 provide correct value for INSN_TICK (TWIN). */
8375 sd_copy_back_deps (twin, insn, true);
8377 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun))
8378 /* In case of branchy check, fix CFG. */
8380 basic_block first_bb, second_bb;
8381 rtx_insn *jump;
8383 first_bb = BLOCK_FOR_INSN (check);
8384 second_bb = sched_split_block (first_bb, check);
8386 sched_create_recovery_edges (first_bb, rec, second_bb);
8388 sched_init_only_bb (second_bb, first_bb);
8389 sched_init_only_bb (rec, EXIT_BLOCK_PTR_FOR_FN (cfun));
8391 jump = BB_END (rec);
8392 haifa_init_insn (jump);
8395 /* Move backward dependences from INSN to CHECK and
8396 move forward dependences from INSN to TWIN. */
8398 /* First, create dependencies between INSN's producers and CHECK & TWIN. */
8399 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep)
8401 rtx_insn *pro = DEP_PRO (dep);
8402 ds_t ds;
8404 /* If BEGIN_DATA: [insn ~~TRUE~~> producer]:
8405 check --TRUE--> producer ??? or ANTI ???
8406 twin --TRUE--> producer
8407 twin --ANTI--> check
8409 If BEGIN_CONTROL: [insn ~~ANTI~~> producer]:
8410 check --ANTI--> producer
8411 twin --ANTI--> producer
8412 twin --ANTI--> check
8414 If BE_IN_SPEC: [insn ~~TRUE~~> producer]:
8415 check ~~TRUE~~> producer
8416 twin ~~TRUE~~> producer
8417 twin --ANTI--> check */
8419 ds = DEP_STATUS (dep);
8421 if (ds & BEGIN_SPEC)
8423 gcc_assert (!mutate_p);
8424 ds &= ~BEGIN_SPEC;
8427 init_dep_1 (new_dep, pro, check, DEP_TYPE (dep), ds);
8428 sd_add_dep (new_dep, false);
8430 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun))
8432 DEP_CON (new_dep) = twin;
8433 sd_add_dep (new_dep, false);
8437 /* Second, remove backward dependencies of INSN. */
8438 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
8439 sd_iterator_cond (&sd_it, &dep);)
8441 if ((DEP_STATUS (dep) & BEGIN_SPEC)
8442 || mutate_p)
8443 /* We can delete this dep because we overcome it with
8444 BEGIN_SPECULATION. */
8445 sd_delete_dep (sd_it);
8446 else
8447 sd_iterator_next (&sd_it);
8450 /* Future Speculations. Determine what BE_IN speculations will be like. */
8451 fs = 0;
8453 /* Fields (DONE_SPEC (x) & BEGIN_SPEC) and CHECK_SPEC (x) are set only
8454 here. */
8456 gcc_assert (!DONE_SPEC (insn));
8458 if (!mutate_p)
8460 ds_t ts = TODO_SPEC (insn);
8462 DONE_SPEC (insn) = ts & BEGIN_SPEC;
8463 CHECK_SPEC (check) = ts & BEGIN_SPEC;
8465 /* Luckiness of future speculations solely depends upon initial
8466 BEGIN speculation. */
8467 if (ts & BEGIN_DATA)
8468 fs = set_dep_weak (fs, BE_IN_DATA, get_dep_weak (ts, BEGIN_DATA));
8469 if (ts & BEGIN_CONTROL)
8470 fs = set_dep_weak (fs, BE_IN_CONTROL,
8471 get_dep_weak (ts, BEGIN_CONTROL));
8473 else
8474 CHECK_SPEC (check) = CHECK_SPEC (insn);
8476 /* Future speculations: call the helper. */
8477 process_insn_forw_deps_be_in_spec (insn, twin, fs);
8479 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun))
8481 /* Which types of dependencies should we use here is,
8482 generally, machine-dependent question... But, for now,
8483 it is not. */
8485 if (!mutate_p)
8487 init_dep (new_dep, insn, check, REG_DEP_TRUE);
8488 sd_add_dep (new_dep, false);
8490 init_dep (new_dep, insn, twin, REG_DEP_OUTPUT);
8491 sd_add_dep (new_dep, false);
8493 else
8495 if (spec_info->dump)
8496 fprintf (spec_info->dump, ";;\t\tRemoved simple check : %s\n",
8497 (*current_sched_info->print_insn) (insn, 0));
8499 /* Remove all dependencies of the INSN. */
8501 sd_it = sd_iterator_start (insn, (SD_LIST_FORW
8502 | SD_LIST_BACK
8503 | SD_LIST_RES_BACK));
8504 while (sd_iterator_cond (&sd_it, &dep))
8505 sd_delete_dep (sd_it);
8508 /* If former check (INSN) already was moved to the ready (or queue)
8509 list, add new check (CHECK) there too. */
8510 if (QUEUE_INDEX (insn) != QUEUE_NOWHERE)
8511 try_ready (check);
8513 /* Remove old check from instruction stream and free its
8514 data. */
8515 sched_remove_insn (insn);
8518 init_dep (new_dep, check, twin, REG_DEP_ANTI);
8519 sd_add_dep (new_dep, false);
8521 else
8523 init_dep_1 (new_dep, insn, check, REG_DEP_TRUE, DEP_TRUE | DEP_OUTPUT);
8524 sd_add_dep (new_dep, false);
8527 if (!mutate_p)
8528 /* Fix priorities. If MUTATE_P is nonzero, this is not necessary,
8529 because it'll be done later in add_to_speculative_block. */
8531 auto_vec<rtx_insn *> priorities_roots;
8533 clear_priorities (twin, &priorities_roots);
8534 calc_priorities (priorities_roots);
8538 /* Removes dependency between instructions in the recovery block REC
8539 and usual region instructions. It keeps inner dependences so it
8540 won't be necessary to recompute them. */
8541 static void
8542 fix_recovery_deps (basic_block rec)
8544 rtx_insn *note, *insn, *jump;
8545 auto_vec<rtx_insn *, 10> ready_list;
8546 auto_bitmap in_ready;
8548 /* NOTE - a basic block note. */
8549 note = NEXT_INSN (BB_HEAD (rec));
8550 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note));
8551 insn = BB_END (rec);
8552 gcc_assert (JUMP_P (insn));
8553 insn = PREV_INSN (insn);
8557 sd_iterator_def sd_it;
8558 dep_t dep;
8560 for (sd_it = sd_iterator_start (insn, SD_LIST_FORW);
8561 sd_iterator_cond (&sd_it, &dep);)
8563 rtx_insn *consumer = DEP_CON (dep);
8565 if (BLOCK_FOR_INSN (consumer) != rec)
8567 sd_delete_dep (sd_it);
8569 if (bitmap_set_bit (in_ready, INSN_LUID (consumer)))
8570 ready_list.safe_push (consumer);
8572 else
8574 gcc_assert ((DEP_STATUS (dep) & DEP_TYPES) == DEP_TRUE);
8576 sd_iterator_next (&sd_it);
8580 insn = PREV_INSN (insn);
8582 while (insn != note);
8584 /* Try to add instructions to the ready or queue list. */
8585 unsigned int i;
8586 rtx_insn *temp;
8587 FOR_EACH_VEC_ELT_REVERSE (ready_list, i, temp)
8588 try_ready (temp);
8590 /* Fixing jump's dependences. */
8591 insn = BB_HEAD (rec);
8592 jump = BB_END (rec);
8594 gcc_assert (LABEL_P (insn));
8595 insn = NEXT_INSN (insn);
8597 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (insn));
8598 add_jump_dependencies (insn, jump);
8601 /* Change pattern of INSN to NEW_PAT. Invalidate cached haifa
8602 instruction data. */
8603 static bool
8604 haifa_change_pattern (rtx_insn *insn, rtx new_pat)
8606 int t;
8608 t = validate_change (insn, &PATTERN (insn), new_pat, 0);
8609 if (!t)
8610 return false;
8612 update_insn_after_change (insn);
8613 return true;
8616 /* -1 - can't speculate,
8617 0 - for speculation with REQUEST mode it is OK to use
8618 current instruction pattern,
8619 1 - need to change pattern for *NEW_PAT to be speculative. */
8621 sched_speculate_insn (rtx_insn *insn, ds_t request, rtx *new_pat)
8623 gcc_assert (current_sched_info->flags & DO_SPECULATION
8624 && (request & SPECULATIVE)
8625 && sched_insn_is_legitimate_for_speculation_p (insn, request));
8627 if ((request & spec_info->mask) != request)
8628 return -1;
8630 if (request & BE_IN_SPEC
8631 && !(request & BEGIN_SPEC))
8632 return 0;
8634 return targetm.sched.speculate_insn (insn, request, new_pat);
8637 static int
8638 haifa_speculate_insn (rtx_insn *insn, ds_t request, rtx *new_pat)
8640 gcc_assert (sched_deps_info->generate_spec_deps
8641 && !IS_SPECULATION_CHECK_P (insn));
8643 if (HAS_INTERNAL_DEP (insn)
8644 || SCHED_GROUP_P (insn))
8645 return -1;
8647 return sched_speculate_insn (insn, request, new_pat);
8650 /* Print some information about block BB, which starts with HEAD and
8651 ends with TAIL, before scheduling it.
8652 I is zero, if scheduler is about to start with the fresh ebb. */
8653 static void
8654 dump_new_block_header (int i, basic_block bb, rtx_insn *head, rtx_insn *tail)
8656 if (!i)
8657 fprintf (sched_dump,
8658 ";; ======================================================\n");
8659 else
8660 fprintf (sched_dump,
8661 ";; =====================ADVANCING TO=====================\n");
8662 fprintf (sched_dump,
8663 ";; -- basic block %d from %d to %d -- %s reload\n",
8664 bb->index, INSN_UID (head), INSN_UID (tail),
8665 (reload_completed ? "after" : "before"));
8666 fprintf (sched_dump,
8667 ";; ======================================================\n");
8668 fprintf (sched_dump, "\n");
8671 /* Unlink basic block notes and labels and saves them, so they
8672 can be easily restored. We unlink basic block notes in EBB to
8673 provide back-compatibility with the previous code, as target backends
8674 assume, that there'll be only instructions between
8675 current_sched_info->{head and tail}. We restore these notes as soon
8676 as we can.
8677 FIRST (LAST) is the first (last) basic block in the ebb.
8678 NB: In usual case (FIRST == LAST) nothing is really done. */
8679 void
8680 unlink_bb_notes (basic_block first, basic_block last)
8682 /* We DON'T unlink basic block notes of the first block in the ebb. */
8683 if (first == last)
8684 return;
8686 bb_header = XNEWVEC (rtx_insn *, last_basic_block_for_fn (cfun));
8688 /* Make a sentinel. */
8689 if (last->next_bb != EXIT_BLOCK_PTR_FOR_FN (cfun))
8690 bb_header[last->next_bb->index] = 0;
8692 first = first->next_bb;
8695 rtx_insn *prev, *label, *note, *next;
8697 label = BB_HEAD (last);
8698 if (LABEL_P (label))
8699 note = NEXT_INSN (label);
8700 else
8701 note = label;
8702 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note));
8704 prev = PREV_INSN (label);
8705 next = NEXT_INSN (note);
8706 gcc_assert (prev && next);
8708 SET_NEXT_INSN (prev) = next;
8709 SET_PREV_INSN (next) = prev;
8711 bb_header[last->index] = label;
8713 if (last == first)
8714 break;
8716 last = last->prev_bb;
8718 while (1);
8721 /* Restore basic block notes.
8722 FIRST is the first basic block in the ebb. */
8723 static void
8724 restore_bb_notes (basic_block first)
8726 if (!bb_header)
8727 return;
8729 /* We DON'T unlink basic block notes of the first block in the ebb. */
8730 first = first->next_bb;
8731 /* Remember: FIRST is actually a second basic block in the ebb. */
8733 while (first != EXIT_BLOCK_PTR_FOR_FN (cfun)
8734 && bb_header[first->index])
8736 rtx_insn *prev, *label, *note, *next;
8738 label = bb_header[first->index];
8739 prev = PREV_INSN (label);
8740 next = NEXT_INSN (prev);
8742 if (LABEL_P (label))
8743 note = NEXT_INSN (label);
8744 else
8745 note = label;
8746 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note));
8748 bb_header[first->index] = 0;
8750 SET_NEXT_INSN (prev) = label;
8751 SET_NEXT_INSN (note) = next;
8752 SET_PREV_INSN (next) = note;
8754 first = first->next_bb;
8757 free (bb_header);
8758 bb_header = 0;
8761 /* Helper function.
8762 Fix CFG after both in- and inter-block movement of
8763 control_flow_insn_p JUMP. */
8764 static void
8765 fix_jump_move (rtx_insn *jump)
8767 basic_block bb, jump_bb, jump_bb_next;
8769 bb = BLOCK_FOR_INSN (PREV_INSN (jump));
8770 jump_bb = BLOCK_FOR_INSN (jump);
8771 jump_bb_next = jump_bb->next_bb;
8773 gcc_assert (common_sched_info->sched_pass_id == SCHED_EBB_PASS
8774 || IS_SPECULATION_BRANCHY_CHECK_P (jump));
8776 if (!NOTE_INSN_BASIC_BLOCK_P (BB_END (jump_bb_next)))
8777 /* if jump_bb_next is not empty. */
8778 BB_END (jump_bb) = BB_END (jump_bb_next);
8780 if (BB_END (bb) != PREV_INSN (jump))
8781 /* Then there are instruction after jump that should be placed
8782 to jump_bb_next. */
8783 BB_END (jump_bb_next) = BB_END (bb);
8784 else
8785 /* Otherwise jump_bb_next is empty. */
8786 BB_END (jump_bb_next) = NEXT_INSN (BB_HEAD (jump_bb_next));
8788 /* To make assertion in move_insn happy. */
8789 BB_END (bb) = PREV_INSN (jump);
8791 update_bb_for_insn (jump_bb_next);
8794 /* Fix CFG after interblock movement of control_flow_insn_p JUMP. */
8795 static void
8796 move_block_after_check (rtx_insn *jump)
8798 basic_block bb, jump_bb, jump_bb_next;
8799 vec<edge, va_gc> *t;
8801 bb = BLOCK_FOR_INSN (PREV_INSN (jump));
8802 jump_bb = BLOCK_FOR_INSN (jump);
8803 jump_bb_next = jump_bb->next_bb;
8805 update_bb_for_insn (jump_bb);
8807 gcc_assert (IS_SPECULATION_CHECK_P (jump)
8808 || IS_SPECULATION_CHECK_P (BB_END (jump_bb_next)));
8810 unlink_block (jump_bb_next);
8811 link_block (jump_bb_next, bb);
8813 t = bb->succs;
8814 bb->succs = 0;
8815 move_succs (&(jump_bb->succs), bb);
8816 move_succs (&(jump_bb_next->succs), jump_bb);
8817 move_succs (&t, jump_bb_next);
8819 df_mark_solutions_dirty ();
8821 common_sched_info->fix_recovery_cfg
8822 (bb->index, jump_bb->index, jump_bb_next->index);
8825 /* Helper function for move_block_after_check.
8826 This functions attaches edge vector pointed to by SUCCSP to
8827 block TO. */
8828 static void
8829 move_succs (vec<edge, va_gc> **succsp, basic_block to)
8831 edge e;
8832 edge_iterator ei;
8834 gcc_assert (to->succs == 0);
8836 to->succs = *succsp;
8838 FOR_EACH_EDGE (e, ei, to->succs)
8839 e->src = to;
8841 *succsp = 0;
8844 /* Remove INSN from the instruction stream.
8845 INSN should have any dependencies. */
8846 static void
8847 sched_remove_insn (rtx_insn *insn)
8849 sd_finish_insn (insn);
8851 change_queue_index (insn, QUEUE_NOWHERE);
8852 current_sched_info->add_remove_insn (insn, 1);
8853 delete_insn (insn);
8856 /* Clear priorities of all instructions, that are forward dependent on INSN.
8857 Store in vector pointed to by ROOTS_PTR insns on which priority () should
8858 be invoked to initialize all cleared priorities. */
8859 static void
8860 clear_priorities (rtx_insn *insn, rtx_vec_t *roots_ptr)
8862 sd_iterator_def sd_it;
8863 dep_t dep;
8864 bool insn_is_root_p = true;
8866 gcc_assert (QUEUE_INDEX (insn) != QUEUE_SCHEDULED);
8868 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep)
8870 rtx_insn *pro = DEP_PRO (dep);
8872 if (INSN_PRIORITY_STATUS (pro) >= 0
8873 && QUEUE_INDEX (insn) != QUEUE_SCHEDULED)
8875 /* If DEP doesn't contribute to priority then INSN itself should
8876 be added to priority roots. */
8877 if (contributes_to_priority_p (dep))
8878 insn_is_root_p = false;
8880 INSN_PRIORITY_STATUS (pro) = -1;
8881 clear_priorities (pro, roots_ptr);
8885 if (insn_is_root_p)
8886 roots_ptr->safe_push (insn);
8889 /* Recompute priorities of instructions, whose priorities might have been
8890 changed. ROOTS is a vector of instructions whose priority computation will
8891 trigger initialization of all cleared priorities. */
8892 static void
8893 calc_priorities (rtx_vec_t roots)
8895 int i;
8896 rtx_insn *insn;
8898 FOR_EACH_VEC_ELT (roots, i, insn)
8899 priority (insn);
8903 /* Add dependences between JUMP and other instructions in the recovery
8904 block. INSN is the first insn the recovery block. */
8905 static void
8906 add_jump_dependencies (rtx_insn *insn, rtx_insn *jump)
8910 insn = NEXT_INSN (insn);
8911 if (insn == jump)
8912 break;
8914 if (dep_list_size (insn, SD_LIST_FORW) == 0)
8916 dep_def _new_dep, *new_dep = &_new_dep;
8918 init_dep (new_dep, insn, jump, REG_DEP_ANTI);
8919 sd_add_dep (new_dep, false);
8922 while (1);
8924 gcc_assert (!sd_lists_empty_p (jump, SD_LIST_BACK));
8927 /* Extend data structures for logical insn UID. */
8928 void
8929 sched_extend_luids (void)
8931 int new_luids_max_uid = get_max_uid () + 1;
8933 sched_luids.safe_grow_cleared (new_luids_max_uid);
8936 /* Initialize LUID for INSN. */
8937 void
8938 sched_init_insn_luid (rtx_insn *insn)
8940 int i = INSN_P (insn) ? 1 : common_sched_info->luid_for_non_insn (insn);
8941 int luid;
8943 if (i >= 0)
8945 luid = sched_max_luid;
8946 sched_max_luid += i;
8948 else
8949 luid = -1;
8951 SET_INSN_LUID (insn, luid);
8954 /* Initialize luids for BBS.
8955 The hook common_sched_info->luid_for_non_insn () is used to determine
8956 if notes, labels, etc. need luids. */
8957 void
8958 sched_init_luids (bb_vec_t bbs)
8960 int i;
8961 basic_block bb;
8963 sched_extend_luids ();
8964 FOR_EACH_VEC_ELT (bbs, i, bb)
8966 rtx_insn *insn;
8968 FOR_BB_INSNS (bb, insn)
8969 sched_init_insn_luid (insn);
8973 /* Free LUIDs. */
8974 void
8975 sched_finish_luids (void)
8977 sched_luids.release ();
8978 sched_max_luid = 1;
8981 /* Return logical uid of INSN. Helpful while debugging. */
8983 insn_luid (rtx_insn *insn)
8985 return INSN_LUID (insn);
8988 /* Extend per insn data in the target. */
8989 void
8990 sched_extend_target (void)
8992 if (targetm.sched.h_i_d_extended)
8993 targetm.sched.h_i_d_extended ();
8996 /* Extend global scheduler structures (those, that live across calls to
8997 schedule_block) to include information about just emitted INSN. */
8998 static void
8999 extend_h_i_d (void)
9001 int reserve = (get_max_uid () + 1 - h_i_d.length ());
9002 if (reserve > 0
9003 && ! h_i_d.space (reserve))
9005 h_i_d.safe_grow_cleared (3 * get_max_uid () / 2);
9006 sched_extend_target ();
9010 /* Initialize h_i_d entry of the INSN with default values.
9011 Values, that are not explicitly initialized here, hold zero. */
9012 static void
9013 init_h_i_d (rtx_insn *insn)
9015 if (INSN_LUID (insn) > 0)
9017 INSN_COST (insn) = -1;
9018 QUEUE_INDEX (insn) = QUEUE_NOWHERE;
9019 INSN_TICK (insn) = INVALID_TICK;
9020 INSN_EXACT_TICK (insn) = INVALID_TICK;
9021 INTER_TICK (insn) = INVALID_TICK;
9022 TODO_SPEC (insn) = HARD_DEP;
9023 INSN_AUTOPREF_MULTIPASS_DATA (insn)[0].status
9024 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED;
9025 INSN_AUTOPREF_MULTIPASS_DATA (insn)[1].status
9026 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED;
9030 /* Initialize haifa_insn_data for BBS. */
9031 void
9032 haifa_init_h_i_d (bb_vec_t bbs)
9034 int i;
9035 basic_block bb;
9037 extend_h_i_d ();
9038 FOR_EACH_VEC_ELT (bbs, i, bb)
9040 rtx_insn *insn;
9042 FOR_BB_INSNS (bb, insn)
9043 init_h_i_d (insn);
9047 /* Finalize haifa_insn_data. */
9048 void
9049 haifa_finish_h_i_d (void)
9051 int i;
9052 haifa_insn_data_t data;
9053 reg_use_data *use, *next_use;
9054 reg_set_data *set, *next_set;
9056 FOR_EACH_VEC_ELT (h_i_d, i, data)
9058 free (data->max_reg_pressure);
9059 free (data->reg_pressure);
9060 for (use = data->reg_use_list; use != NULL; use = next_use)
9062 next_use = use->next_insn_use;
9063 free (use);
9065 for (set = data->reg_set_list; set != NULL; set = next_set)
9067 next_set = set->next_insn_set;
9068 free (set);
9072 h_i_d.release ();
9075 /* Init data for the new insn INSN. */
9076 static void
9077 haifa_init_insn (rtx_insn *insn)
9079 gcc_assert (insn != NULL);
9081 sched_extend_luids ();
9082 sched_init_insn_luid (insn);
9083 sched_extend_target ();
9084 sched_deps_init (false);
9085 extend_h_i_d ();
9086 init_h_i_d (insn);
9088 if (adding_bb_to_current_region_p)
9090 sd_init_insn (insn);
9092 /* Extend dependency caches by one element. */
9093 extend_dependency_caches (1, false);
9095 if (sched_pressure != SCHED_PRESSURE_NONE)
9096 init_insn_reg_pressure_info (insn);
9099 /* Init data for the new basic block BB which comes after AFTER. */
9100 static void
9101 haifa_init_only_bb (basic_block bb, basic_block after)
9103 gcc_assert (bb != NULL);
9105 sched_init_bbs ();
9107 if (common_sched_info->add_block)
9108 /* This changes only data structures of the front-end. */
9109 common_sched_info->add_block (bb, after);
9112 /* A generic version of sched_split_block (). */
9113 basic_block
9114 sched_split_block_1 (basic_block first_bb, rtx after)
9116 edge e;
9118 e = split_block (first_bb, after);
9119 gcc_assert (e->src == first_bb);
9121 /* sched_split_block emits note if *check == BB_END. Probably it
9122 is better to rip that note off. */
9124 return e->dest;
9127 /* A generic version of sched_create_empty_bb (). */
9128 basic_block
9129 sched_create_empty_bb_1 (basic_block after)
9131 return create_empty_bb (after);
9134 /* Insert PAT as an INSN into the schedule and update the necessary data
9135 structures to account for it. */
9136 rtx_insn *
9137 sched_emit_insn (rtx pat)
9139 rtx_insn *insn = emit_insn_before (pat, first_nonscheduled_insn ());
9140 haifa_init_insn (insn);
9142 if (current_sched_info->add_remove_insn)
9143 current_sched_info->add_remove_insn (insn, 0);
9145 (*current_sched_info->begin_schedule_ready) (insn);
9146 scheduled_insns.safe_push (insn);
9148 last_scheduled_insn = insn;
9149 return insn;
9152 /* This function returns a candidate satisfying dispatch constraints from
9153 the ready list. */
9155 static rtx_insn *
9156 ready_remove_first_dispatch (struct ready_list *ready)
9158 int i;
9159 rtx_insn *insn = ready_element (ready, 0);
9161 if (ready->n_ready == 1
9162 || !INSN_P (insn)
9163 || INSN_CODE (insn) < 0
9164 || !active_insn_p (insn)
9165 || targetm.sched.dispatch (insn, FITS_DISPATCH_WINDOW))
9166 return ready_remove_first (ready);
9168 for (i = 1; i < ready->n_ready; i++)
9170 insn = ready_element (ready, i);
9172 if (!INSN_P (insn)
9173 || INSN_CODE (insn) < 0
9174 || !active_insn_p (insn))
9175 continue;
9177 if (targetm.sched.dispatch (insn, FITS_DISPATCH_WINDOW))
9179 /* Return ith element of ready. */
9180 insn = ready_remove (ready, i);
9181 return insn;
9185 if (targetm.sched.dispatch (NULL, DISPATCH_VIOLATION))
9186 return ready_remove_first (ready);
9188 for (i = 1; i < ready->n_ready; i++)
9190 insn = ready_element (ready, i);
9192 if (!INSN_P (insn)
9193 || INSN_CODE (insn) < 0
9194 || !active_insn_p (insn))
9195 continue;
9197 /* Return i-th element of ready. */
9198 if (targetm.sched.dispatch (insn, IS_CMP))
9199 return ready_remove (ready, i);
9202 return ready_remove_first (ready);
9205 /* Get number of ready insn in the ready list. */
9208 number_in_ready (void)
9210 return ready.n_ready;
9213 /* Get number of ready's in the ready list. */
9215 rtx_insn *
9216 get_ready_element (int i)
9218 return ready_element (&ready, i);
9221 #endif /* INSN_SCHEDULING */