Daily bump.
[official-gcc.git] / gcc / haifa-sched.c
blob07de1bbbb2391954928d017889ef0aa679aa8d62
1 /* Instruction scheduling pass.
2 Copyright (C) 1992-2017 Free Software Foundation, Inc.
3 Contributed by Michael Tiemann (tiemann@cygnus.com) Enhanced by,
4 and currently maintained by, Jim Wilson (wilson@cygnus.com)
6 This file is part of GCC.
8 GCC is free software; you can redistribute it and/or modify it under
9 the terms of the GNU General Public License as published by the Free
10 Software Foundation; either version 3, or (at your option) any later
11 version.
13 GCC is distributed in the hope that it will be useful, but WITHOUT ANY
14 WARRANTY; without even the implied warranty of MERCHANTABILITY or
15 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
16 for more details.
18 You should have received a copy of the GNU General Public License
19 along with GCC; see the file COPYING3. If not see
20 <http://www.gnu.org/licenses/>. */
22 /* Instruction scheduling pass. This file, along with sched-deps.c,
23 contains the generic parts. The actual entry point for
24 the normal instruction scheduling pass is found in sched-rgn.c.
26 We compute insn priorities based on data dependencies. Flow
27 analysis only creates a fraction of the data-dependencies we must
28 observe: namely, only those dependencies which the combiner can be
29 expected to use. For this pass, we must therefore create the
30 remaining dependencies we need to observe: register dependencies,
31 memory dependencies, dependencies to keep function calls in order,
32 and the dependence between a conditional branch and the setting of
33 condition codes are all dealt with here.
35 The scheduler first traverses the data flow graph, starting with
36 the last instruction, and proceeding to the first, assigning values
37 to insn_priority as it goes. This sorts the instructions
38 topologically by data dependence.
40 Once priorities have been established, we order the insns using
41 list scheduling. This works as follows: starting with a list of
42 all the ready insns, and sorted according to priority number, we
43 schedule the insn from the end of the list by placing its
44 predecessors in the list according to their priority order. We
45 consider this insn scheduled by setting the pointer to the "end" of
46 the list to point to the previous insn. When an insn has no
47 predecessors, we either queue it until sufficient time has elapsed
48 or add it to the ready list. As the instructions are scheduled or
49 when stalls are introduced, the queue advances and dumps insns into
50 the ready list. When all insns down to the lowest priority have
51 been scheduled, the critical path of the basic block has been made
52 as short as possible. The remaining insns are then scheduled in
53 remaining slots.
55 The following list shows the order in which we want to break ties
56 among insns in the ready list:
58 1. choose insn with the longest path to end of bb, ties
59 broken by
60 2. choose insn with least contribution to register pressure,
61 ties broken by
62 3. prefer in-block upon interblock motion, ties broken by
63 4. prefer useful upon speculative motion, ties broken by
64 5. choose insn with largest control flow probability, ties
65 broken by
66 6. choose insn with the least dependences upon the previously
67 scheduled insn, or finally
68 7 choose the insn which has the most insns dependent on it.
69 8. choose insn with lowest UID.
71 Memory references complicate matters. Only if we can be certain
72 that memory references are not part of the data dependency graph
73 (via true, anti, or output dependence), can we move operations past
74 memory references. To first approximation, reads can be done
75 independently, while writes introduce dependencies. Better
76 approximations will yield fewer dependencies.
78 Before reload, an extended analysis of interblock data dependences
79 is required for interblock scheduling. This is performed in
80 compute_block_dependences ().
82 Dependencies set up by memory references are treated in exactly the
83 same way as other dependencies, by using insn backward dependences
84 INSN_BACK_DEPS. INSN_BACK_DEPS are translated into forward dependences
85 INSN_FORW_DEPS for the purpose of forward list scheduling.
87 Having optimized the critical path, we may have also unduly
88 extended the lifetimes of some registers. If an operation requires
89 that constants be loaded into registers, it is certainly desirable
90 to load those constants as early as necessary, but no earlier.
91 I.e., it will not do to load up a bunch of registers at the
92 beginning of a basic block only to use them at the end, if they
93 could be loaded later, since this may result in excessive register
94 utilization.
96 Note that since branches are never in basic blocks, but only end
97 basic blocks, this pass will not move branches. But that is ok,
98 since we can use GNU's delayed branch scheduling pass to take care
99 of this case.
101 Also note that no further optimizations based on algebraic
102 identities are performed, so this pass would be a good one to
103 perform instruction splitting, such as breaking up a multiply
104 instruction into shifts and adds where that is profitable.
106 Given the memory aliasing analysis that this pass should perform,
107 it should be possible to remove redundant stores to memory, and to
108 load values from registers instead of hitting memory.
110 Before reload, speculative insns are moved only if a 'proof' exists
111 that no exception will be caused by this, and if no live registers
112 exist that inhibit the motion (live registers constraints are not
113 represented by data dependence edges).
115 This pass must update information that subsequent passes expect to
116 be correct. Namely: reg_n_refs, reg_n_sets, reg_n_deaths,
117 reg_n_calls_crossed, and reg_live_length. Also, BB_HEAD, BB_END.
119 The information in the line number notes is carefully retained by
120 this pass. Notes that refer to the starting and ending of
121 exception regions are also carefully retained by this pass. All
122 other NOTE insns are grouped in their same relative order at the
123 beginning of basic blocks and regions that have been scheduled. */
125 #include "config.h"
126 #include "system.h"
127 #include "coretypes.h"
128 #include "backend.h"
129 #include "target.h"
130 #include "rtl.h"
131 #include "cfghooks.h"
132 #include "df.h"
133 #include "memmodel.h"
134 #include "tm_p.h"
135 #include "insn-config.h"
136 #include "regs.h"
137 #include "ira.h"
138 #include "recog.h"
139 #include "insn-attr.h"
140 #include "cfgrtl.h"
141 #include "cfgbuild.h"
142 #include "sched-int.h"
143 #include "common/common-target.h"
144 #include "params.h"
145 #include "dbgcnt.h"
146 #include "cfgloop.h"
147 #include "dumpfile.h"
148 #include "print-rtl.h"
150 #ifdef INSN_SCHEDULING
152 /* True if we do register pressure relief through live-range
153 shrinkage. */
154 static bool live_range_shrinkage_p;
156 /* Switch on live range shrinkage. */
157 void
158 initialize_live_range_shrinkage (void)
160 live_range_shrinkage_p = true;
163 /* Switch off live range shrinkage. */
164 void
165 finish_live_range_shrinkage (void)
167 live_range_shrinkage_p = false;
170 /* issue_rate is the number of insns that can be scheduled in the same
171 machine cycle. It can be defined in the config/mach/mach.h file,
172 otherwise we set it to 1. */
174 int issue_rate;
176 /* This can be set to true by a backend if the scheduler should not
177 enable a DCE pass. */
178 bool sched_no_dce;
180 /* The current initiation interval used when modulo scheduling. */
181 static int modulo_ii;
183 /* The maximum number of stages we are prepared to handle. */
184 static int modulo_max_stages;
186 /* The number of insns that exist in each iteration of the loop. We use this
187 to detect when we've scheduled all insns from the first iteration. */
188 static int modulo_n_insns;
190 /* The current count of insns in the first iteration of the loop that have
191 already been scheduled. */
192 static int modulo_insns_scheduled;
194 /* The maximum uid of insns from the first iteration of the loop. */
195 static int modulo_iter0_max_uid;
197 /* The number of times we should attempt to backtrack when modulo scheduling.
198 Decreased each time we have to backtrack. */
199 static int modulo_backtracks_left;
201 /* The stage in which the last insn from the original loop was
202 scheduled. */
203 static int modulo_last_stage;
205 /* sched-verbose controls the amount of debugging output the
206 scheduler prints. It is controlled by -fsched-verbose=N:
207 N=0: no debugging output.
208 N=1: default value.
209 N=2: bb's probabilities, detailed ready list info, unit/insn info.
210 N=3: rtl at abort point, control-flow, regions info.
211 N=5: dependences info. */
212 int sched_verbose = 0;
214 /* Debugging file. All printouts are sent to dump. */
215 FILE *sched_dump = 0;
217 /* This is a placeholder for the scheduler parameters common
218 to all schedulers. */
219 struct common_sched_info_def *common_sched_info;
221 #define INSN_TICK(INSN) (HID (INSN)->tick)
222 #define INSN_EXACT_TICK(INSN) (HID (INSN)->exact_tick)
223 #define INSN_TICK_ESTIMATE(INSN) (HID (INSN)->tick_estimate)
224 #define INTER_TICK(INSN) (HID (INSN)->inter_tick)
225 #define FEEDS_BACKTRACK_INSN(INSN) (HID (INSN)->feeds_backtrack_insn)
226 #define SHADOW_P(INSN) (HID (INSN)->shadow_p)
227 #define MUST_RECOMPUTE_SPEC_P(INSN) (HID (INSN)->must_recompute_spec)
228 /* Cached cost of the instruction. Use insn_cost to get cost of the
229 insn. -1 here means that the field is not initialized. */
230 #define INSN_COST(INSN) (HID (INSN)->cost)
232 /* If INSN_TICK of an instruction is equal to INVALID_TICK,
233 then it should be recalculated from scratch. */
234 #define INVALID_TICK (-(max_insn_queue_index + 1))
235 /* The minimal value of the INSN_TICK of an instruction. */
236 #define MIN_TICK (-max_insn_queue_index)
238 /* Original order of insns in the ready list.
239 Used to keep order of normal insns while separating DEBUG_INSNs. */
240 #define INSN_RFS_DEBUG_ORIG_ORDER(INSN) (HID (INSN)->rfs_debug_orig_order)
242 /* The deciding reason for INSN's place in the ready list. */
243 #define INSN_LAST_RFS_WIN(INSN) (HID (INSN)->last_rfs_win)
245 /* List of important notes we must keep around. This is a pointer to the
246 last element in the list. */
247 rtx_insn *note_list;
249 static struct spec_info_def spec_info_var;
250 /* Description of the speculative part of the scheduling.
251 If NULL - no speculation. */
252 spec_info_t spec_info = NULL;
254 /* True, if recovery block was added during scheduling of current block.
255 Used to determine, if we need to fix INSN_TICKs. */
256 static bool haifa_recovery_bb_recently_added_p;
258 /* True, if recovery block was added during this scheduling pass.
259 Used to determine if we should have empty memory pools of dependencies
260 after finishing current region. */
261 bool haifa_recovery_bb_ever_added_p;
263 /* Counters of different types of speculative instructions. */
264 static int nr_begin_data, nr_be_in_data, nr_begin_control, nr_be_in_control;
266 /* Array used in {unlink, restore}_bb_notes. */
267 static rtx_insn **bb_header = 0;
269 /* Basic block after which recovery blocks will be created. */
270 static basic_block before_recovery;
272 /* Basic block just before the EXIT_BLOCK and after recovery, if we have
273 created it. */
274 basic_block after_recovery;
276 /* FALSE if we add bb to another region, so we don't need to initialize it. */
277 bool adding_bb_to_current_region_p = true;
279 /* Queues, etc. */
281 /* An instruction is ready to be scheduled when all insns preceding it
282 have already been scheduled. It is important to ensure that all
283 insns which use its result will not be executed until its result
284 has been computed. An insn is maintained in one of four structures:
286 (P) the "Pending" set of insns which cannot be scheduled until
287 their dependencies have been satisfied.
288 (Q) the "Queued" set of insns that can be scheduled when sufficient
289 time has passed.
290 (R) the "Ready" list of unscheduled, uncommitted insns.
291 (S) the "Scheduled" list of insns.
293 Initially, all insns are either "Pending" or "Ready" depending on
294 whether their dependencies are satisfied.
296 Insns move from the "Ready" list to the "Scheduled" list as they
297 are committed to the schedule. As this occurs, the insns in the
298 "Pending" list have their dependencies satisfied and move to either
299 the "Ready" list or the "Queued" set depending on whether
300 sufficient time has passed to make them ready. As time passes,
301 insns move from the "Queued" set to the "Ready" list.
303 The "Pending" list (P) are the insns in the INSN_FORW_DEPS of the
304 unscheduled insns, i.e., those that are ready, queued, and pending.
305 The "Queued" set (Q) is implemented by the variable `insn_queue'.
306 The "Ready" list (R) is implemented by the variables `ready' and
307 `n_ready'.
308 The "Scheduled" list (S) is the new insn chain built by this pass.
310 The transition (R->S) is implemented in the scheduling loop in
311 `schedule_block' when the best insn to schedule is chosen.
312 The transitions (P->R and P->Q) are implemented in `schedule_insn' as
313 insns move from the ready list to the scheduled list.
314 The transition (Q->R) is implemented in 'queue_to_insn' as time
315 passes or stalls are introduced. */
317 /* Implement a circular buffer to delay instructions until sufficient
318 time has passed. For the new pipeline description interface,
319 MAX_INSN_QUEUE_INDEX is a power of two minus one which is not less
320 than maximal time of instruction execution computed by genattr.c on
321 the base maximal time of functional unit reservations and getting a
322 result. This is the longest time an insn may be queued. */
324 static rtx_insn_list **insn_queue;
325 static int q_ptr = 0;
326 static int q_size = 0;
327 #define NEXT_Q(X) (((X)+1) & max_insn_queue_index)
328 #define NEXT_Q_AFTER(X, C) (((X)+C) & max_insn_queue_index)
330 #define QUEUE_SCHEDULED (-3)
331 #define QUEUE_NOWHERE (-2)
332 #define QUEUE_READY (-1)
333 /* QUEUE_SCHEDULED - INSN is scheduled.
334 QUEUE_NOWHERE - INSN isn't scheduled yet and is neither in
335 queue or ready list.
336 QUEUE_READY - INSN is in ready list.
337 N >= 0 - INSN queued for X [where NEXT_Q_AFTER (q_ptr, X) == N] cycles. */
339 #define QUEUE_INDEX(INSN) (HID (INSN)->queue_index)
341 /* The following variable value refers for all current and future
342 reservations of the processor units. */
343 state_t curr_state;
345 /* The following variable value is size of memory representing all
346 current and future reservations of the processor units. */
347 size_t dfa_state_size;
349 /* The following array is used to find the best insn from ready when
350 the automaton pipeline interface is used. */
351 signed char *ready_try = NULL;
353 /* The ready list. */
354 struct ready_list ready = {NULL, 0, 0, 0, 0};
356 /* The pointer to the ready list (to be removed). */
357 static struct ready_list *readyp = &ready;
359 /* Scheduling clock. */
360 static int clock_var;
362 /* Clock at which the previous instruction was issued. */
363 static int last_clock_var;
365 /* Set to true if, when queuing a shadow insn, we discover that it would be
366 scheduled too late. */
367 static bool must_backtrack;
369 /* The following variable value is number of essential insns issued on
370 the current cycle. An insn is essential one if it changes the
371 processors state. */
372 int cycle_issued_insns;
374 /* This records the actual schedule. It is built up during the main phase
375 of schedule_block, and afterwards used to reorder the insns in the RTL. */
376 static vec<rtx_insn *> scheduled_insns;
378 static int may_trap_exp (const_rtx, int);
380 /* Nonzero iff the address is comprised from at most 1 register. */
381 #define CONST_BASED_ADDRESS_P(x) \
382 (REG_P (x) \
383 || ((GET_CODE (x) == PLUS || GET_CODE (x) == MINUS \
384 || (GET_CODE (x) == LO_SUM)) \
385 && (CONSTANT_P (XEXP (x, 0)) \
386 || CONSTANT_P (XEXP (x, 1)))))
388 /* Returns a class that insn with GET_DEST(insn)=x may belong to,
389 as found by analyzing insn's expression. */
392 static int haifa_luid_for_non_insn (rtx x);
394 /* Haifa version of sched_info hooks common to all headers. */
395 const struct common_sched_info_def haifa_common_sched_info =
397 NULL, /* fix_recovery_cfg */
398 NULL, /* add_block */
399 NULL, /* estimate_number_of_insns */
400 haifa_luid_for_non_insn, /* luid_for_non_insn */
401 SCHED_PASS_UNKNOWN /* sched_pass_id */
404 /* Mapping from instruction UID to its Logical UID. */
405 vec<int> sched_luids;
407 /* Next LUID to assign to an instruction. */
408 int sched_max_luid = 1;
410 /* Haifa Instruction Data. */
411 vec<haifa_insn_data_def> h_i_d;
413 void (* sched_init_only_bb) (basic_block, basic_block);
415 /* Split block function. Different schedulers might use different functions
416 to handle their internal data consistent. */
417 basic_block (* sched_split_block) (basic_block, rtx);
419 /* Create empty basic block after the specified block. */
420 basic_block (* sched_create_empty_bb) (basic_block);
422 /* Return the number of cycles until INSN is expected to be ready.
423 Return zero if it already is. */
424 static int
425 insn_delay (rtx_insn *insn)
427 return MAX (INSN_TICK (insn) - clock_var, 0);
430 static int
431 may_trap_exp (const_rtx x, int is_store)
433 enum rtx_code code;
435 if (x == 0)
436 return TRAP_FREE;
437 code = GET_CODE (x);
438 if (is_store)
440 if (code == MEM && may_trap_p (x))
441 return TRAP_RISKY;
442 else
443 return TRAP_FREE;
445 if (code == MEM)
447 /* The insn uses memory: a volatile load. */
448 if (MEM_VOLATILE_P (x))
449 return IRISKY;
450 /* An exception-free load. */
451 if (!may_trap_p (x))
452 return IFREE;
453 /* A load with 1 base register, to be further checked. */
454 if (CONST_BASED_ADDRESS_P (XEXP (x, 0)))
455 return PFREE_CANDIDATE;
456 /* No info on the load, to be further checked. */
457 return PRISKY_CANDIDATE;
459 else
461 const char *fmt;
462 int i, insn_class = TRAP_FREE;
464 /* Neither store nor load, check if it may cause a trap. */
465 if (may_trap_p (x))
466 return TRAP_RISKY;
467 /* Recursive step: walk the insn... */
468 fmt = GET_RTX_FORMAT (code);
469 for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
471 if (fmt[i] == 'e')
473 int tmp_class = may_trap_exp (XEXP (x, i), is_store);
474 insn_class = WORST_CLASS (insn_class, tmp_class);
476 else if (fmt[i] == 'E')
478 int j;
479 for (j = 0; j < XVECLEN (x, i); j++)
481 int tmp_class = may_trap_exp (XVECEXP (x, i, j), is_store);
482 insn_class = WORST_CLASS (insn_class, tmp_class);
483 if (insn_class == TRAP_RISKY || insn_class == IRISKY)
484 break;
487 if (insn_class == TRAP_RISKY || insn_class == IRISKY)
488 break;
490 return insn_class;
494 /* Classifies rtx X of an insn for the purpose of verifying that X can be
495 executed speculatively (and consequently the insn can be moved
496 speculatively), by examining X, returning:
497 TRAP_RISKY: store, or risky non-load insn (e.g. division by variable).
498 TRAP_FREE: non-load insn.
499 IFREE: load from a globally safe location.
500 IRISKY: volatile load.
501 PFREE_CANDIDATE, PRISKY_CANDIDATE: load that need to be checked for
502 being either PFREE or PRISKY. */
504 static int
505 haifa_classify_rtx (const_rtx x)
507 int tmp_class = TRAP_FREE;
508 int insn_class = TRAP_FREE;
509 enum rtx_code code;
511 if (GET_CODE (x) == PARALLEL)
513 int i, len = XVECLEN (x, 0);
515 for (i = len - 1; i >= 0; i--)
517 tmp_class = haifa_classify_rtx (XVECEXP (x, 0, i));
518 insn_class = WORST_CLASS (insn_class, tmp_class);
519 if (insn_class == TRAP_RISKY || insn_class == IRISKY)
520 break;
523 else
525 code = GET_CODE (x);
526 switch (code)
528 case CLOBBER:
529 /* Test if it is a 'store'. */
530 tmp_class = may_trap_exp (XEXP (x, 0), 1);
531 break;
532 case SET:
533 /* Test if it is a store. */
534 tmp_class = may_trap_exp (SET_DEST (x), 1);
535 if (tmp_class == TRAP_RISKY)
536 break;
537 /* Test if it is a load. */
538 tmp_class =
539 WORST_CLASS (tmp_class,
540 may_trap_exp (SET_SRC (x), 0));
541 break;
542 case COND_EXEC:
543 tmp_class = haifa_classify_rtx (COND_EXEC_CODE (x));
544 if (tmp_class == TRAP_RISKY)
545 break;
546 tmp_class = WORST_CLASS (tmp_class,
547 may_trap_exp (COND_EXEC_TEST (x), 0));
548 break;
549 case TRAP_IF:
550 tmp_class = TRAP_RISKY;
551 break;
552 default:;
554 insn_class = tmp_class;
557 return insn_class;
561 haifa_classify_insn (const_rtx insn)
563 return haifa_classify_rtx (PATTERN (insn));
566 /* After the scheduler initialization function has been called, this function
567 can be called to enable modulo scheduling. II is the initiation interval
568 we should use, it affects the delays for delay_pairs that were recorded as
569 separated by a given number of stages.
571 MAX_STAGES provides us with a limit
572 after which we give up scheduling; the caller must have unrolled at least
573 as many copies of the loop body and recorded delay_pairs for them.
575 INSNS is the number of real (non-debug) insns in one iteration of
576 the loop. MAX_UID can be used to test whether an insn belongs to
577 the first iteration of the loop; all of them have a uid lower than
578 MAX_UID. */
579 void
580 set_modulo_params (int ii, int max_stages, int insns, int max_uid)
582 modulo_ii = ii;
583 modulo_max_stages = max_stages;
584 modulo_n_insns = insns;
585 modulo_iter0_max_uid = max_uid;
586 modulo_backtracks_left = PARAM_VALUE (PARAM_MAX_MODULO_BACKTRACK_ATTEMPTS);
589 /* A structure to record a pair of insns where the first one is a real
590 insn that has delay slots, and the second is its delayed shadow.
591 I1 is scheduled normally and will emit an assembly instruction,
592 while I2 describes the side effect that takes place at the
593 transition between cycles CYCLES and (CYCLES + 1) after I1. */
594 struct delay_pair
596 struct delay_pair *next_same_i1;
597 rtx_insn *i1, *i2;
598 int cycles;
599 /* When doing modulo scheduling, we a delay_pair can also be used to
600 show that I1 and I2 are the same insn in a different stage. If that
601 is the case, STAGES will be nonzero. */
602 int stages;
605 /* Helpers for delay hashing. */
607 struct delay_i1_hasher : nofree_ptr_hash <delay_pair>
609 typedef void *compare_type;
610 static inline hashval_t hash (const delay_pair *);
611 static inline bool equal (const delay_pair *, const void *);
614 /* Returns a hash value for X, based on hashing just I1. */
616 inline hashval_t
617 delay_i1_hasher::hash (const delay_pair *x)
619 return htab_hash_pointer (x->i1);
622 /* Return true if I1 of pair X is the same as that of pair Y. */
624 inline bool
625 delay_i1_hasher::equal (const delay_pair *x, const void *y)
627 return x->i1 == y;
630 struct delay_i2_hasher : free_ptr_hash <delay_pair>
632 typedef void *compare_type;
633 static inline hashval_t hash (const delay_pair *);
634 static inline bool equal (const delay_pair *, const void *);
637 /* Returns a hash value for X, based on hashing just I2. */
639 inline hashval_t
640 delay_i2_hasher::hash (const delay_pair *x)
642 return htab_hash_pointer (x->i2);
645 /* Return true if I2 of pair X is the same as that of pair Y. */
647 inline bool
648 delay_i2_hasher::equal (const delay_pair *x, const void *y)
650 return x->i2 == y;
653 /* Two hash tables to record delay_pairs, one indexed by I1 and the other
654 indexed by I2. */
655 static hash_table<delay_i1_hasher> *delay_htab;
656 static hash_table<delay_i2_hasher> *delay_htab_i2;
658 /* Called through htab_traverse. Walk the hashtable using I2 as
659 index, and delete all elements involving an UID higher than
660 that pointed to by *DATA. */
662 haifa_htab_i2_traverse (delay_pair **slot, int *data)
664 int maxuid = *data;
665 struct delay_pair *p = *slot;
666 if (INSN_UID (p->i2) >= maxuid || INSN_UID (p->i1) >= maxuid)
668 delay_htab_i2->clear_slot (slot);
670 return 1;
673 /* Called through htab_traverse. Walk the hashtable using I2 as
674 index, and delete all elements involving an UID higher than
675 that pointed to by *DATA. */
677 haifa_htab_i1_traverse (delay_pair **pslot, int *data)
679 int maxuid = *data;
680 struct delay_pair *p, *first, **pprev;
682 if (INSN_UID ((*pslot)->i1) >= maxuid)
684 delay_htab->clear_slot (pslot);
685 return 1;
687 pprev = &first;
688 for (p = *pslot; p; p = p->next_same_i1)
690 if (INSN_UID (p->i2) < maxuid)
692 *pprev = p;
693 pprev = &p->next_same_i1;
696 *pprev = NULL;
697 if (first == NULL)
698 delay_htab->clear_slot (pslot);
699 else
700 *pslot = first;
701 return 1;
704 /* Discard all delay pairs which involve an insn with an UID higher
705 than MAX_UID. */
706 void
707 discard_delay_pairs_above (int max_uid)
709 delay_htab->traverse <int *, haifa_htab_i1_traverse> (&max_uid);
710 delay_htab_i2->traverse <int *, haifa_htab_i2_traverse> (&max_uid);
713 /* This function can be called by a port just before it starts the final
714 scheduling pass. It records the fact that an instruction with delay
715 slots has been split into two insns, I1 and I2. The first one will be
716 scheduled normally and initiates the operation. The second one is a
717 shadow which must follow a specific number of cycles after I1; its only
718 purpose is to show the side effect that occurs at that cycle in the RTL.
719 If a JUMP_INSN or a CALL_INSN has been split, I1 should be a normal INSN,
720 while I2 retains the original insn type.
722 There are two ways in which the number of cycles can be specified,
723 involving the CYCLES and STAGES arguments to this function. If STAGES
724 is zero, we just use the value of CYCLES. Otherwise, STAGES is a factor
725 which is multiplied by MODULO_II to give the number of cycles. This is
726 only useful if the caller also calls set_modulo_params to enable modulo
727 scheduling. */
729 void
730 record_delay_slot_pair (rtx_insn *i1, rtx_insn *i2, int cycles, int stages)
732 struct delay_pair *p = XNEW (struct delay_pair);
733 struct delay_pair **slot;
735 p->i1 = i1;
736 p->i2 = i2;
737 p->cycles = cycles;
738 p->stages = stages;
740 if (!delay_htab)
742 delay_htab = new hash_table<delay_i1_hasher> (10);
743 delay_htab_i2 = new hash_table<delay_i2_hasher> (10);
745 slot = delay_htab->find_slot_with_hash (i1, htab_hash_pointer (i1), INSERT);
746 p->next_same_i1 = *slot;
747 *slot = p;
748 slot = delay_htab_i2->find_slot (p, INSERT);
749 *slot = p;
752 /* Examine the delay pair hashtable to see if INSN is a shadow for another,
753 and return the other insn if so. Return NULL otherwise. */
754 rtx_insn *
755 real_insn_for_shadow (rtx_insn *insn)
757 struct delay_pair *pair;
759 if (!delay_htab)
760 return NULL;
762 pair = delay_htab_i2->find_with_hash (insn, htab_hash_pointer (insn));
763 if (!pair || pair->stages > 0)
764 return NULL;
765 return pair->i1;
768 /* For a pair P of insns, return the fixed distance in cycles from the first
769 insn after which the second must be scheduled. */
770 static int
771 pair_delay (struct delay_pair *p)
773 if (p->stages == 0)
774 return p->cycles;
775 else
776 return p->stages * modulo_ii;
779 /* Given an insn INSN, add a dependence on its delayed shadow if it
780 has one. Also try to find situations where shadows depend on each other
781 and add dependencies to the real insns to limit the amount of backtracking
782 needed. */
783 void
784 add_delay_dependencies (rtx_insn *insn)
786 struct delay_pair *pair;
787 sd_iterator_def sd_it;
788 dep_t dep;
790 if (!delay_htab)
791 return;
793 pair = delay_htab_i2->find_with_hash (insn, htab_hash_pointer (insn));
794 if (!pair)
795 return;
796 add_dependence (insn, pair->i1, REG_DEP_ANTI);
797 if (pair->stages)
798 return;
800 FOR_EACH_DEP (pair->i2, SD_LIST_BACK, sd_it, dep)
802 rtx_insn *pro = DEP_PRO (dep);
803 struct delay_pair *other_pair
804 = delay_htab_i2->find_with_hash (pro, htab_hash_pointer (pro));
805 if (!other_pair || other_pair->stages)
806 continue;
807 if (pair_delay (other_pair) >= pair_delay (pair))
809 if (sched_verbose >= 4)
811 fprintf (sched_dump, ";;\tadding dependence %d <- %d\n",
812 INSN_UID (other_pair->i1),
813 INSN_UID (pair->i1));
814 fprintf (sched_dump, ";;\tpair1 %d <- %d, cost %d\n",
815 INSN_UID (pair->i1),
816 INSN_UID (pair->i2),
817 pair_delay (pair));
818 fprintf (sched_dump, ";;\tpair2 %d <- %d, cost %d\n",
819 INSN_UID (other_pair->i1),
820 INSN_UID (other_pair->i2),
821 pair_delay (other_pair));
823 add_dependence (pair->i1, other_pair->i1, REG_DEP_ANTI);
828 /* Forward declarations. */
830 static int priority (rtx_insn *);
831 static int autopref_rank_for_schedule (const rtx_insn *, const rtx_insn *);
832 static int rank_for_schedule (const void *, const void *);
833 static void swap_sort (rtx_insn **, int);
834 static void queue_insn (rtx_insn *, int, const char *);
835 static int schedule_insn (rtx_insn *);
836 static void adjust_priority (rtx_insn *);
837 static void advance_one_cycle (void);
838 static void extend_h_i_d (void);
841 /* Notes handling mechanism:
842 =========================
843 Generally, NOTES are saved before scheduling and restored after scheduling.
844 The scheduler distinguishes between two types of notes:
846 (1) LOOP_BEGIN, LOOP_END, SETJMP, EHREGION_BEG, EHREGION_END notes:
847 Before scheduling a region, a pointer to the note is added to the insn
848 that follows or precedes it. (This happens as part of the data dependence
849 computation). After scheduling an insn, the pointer contained in it is
850 used for regenerating the corresponding note (in reemit_notes).
852 (2) All other notes (e.g. INSN_DELETED): Before scheduling a block,
853 these notes are put in a list (in rm_other_notes() and
854 unlink_other_notes ()). After scheduling the block, these notes are
855 inserted at the beginning of the block (in schedule_block()). */
857 static void ready_add (struct ready_list *, rtx_insn *, bool);
858 static rtx_insn *ready_remove_first (struct ready_list *);
859 static rtx_insn *ready_remove_first_dispatch (struct ready_list *ready);
861 static void queue_to_ready (struct ready_list *);
862 static int early_queue_to_ready (state_t, struct ready_list *);
864 /* The following functions are used to implement multi-pass scheduling
865 on the first cycle. */
866 static rtx_insn *ready_remove (struct ready_list *, int);
867 static void ready_remove_insn (rtx_insn *);
869 static void fix_inter_tick (rtx_insn *, rtx_insn *);
870 static int fix_tick_ready (rtx_insn *);
871 static void change_queue_index (rtx_insn *, int);
873 /* The following functions are used to implement scheduling of data/control
874 speculative instructions. */
876 static void extend_h_i_d (void);
877 static void init_h_i_d (rtx_insn *);
878 static int haifa_speculate_insn (rtx_insn *, ds_t, rtx *);
879 static void generate_recovery_code (rtx_insn *);
880 static void process_insn_forw_deps_be_in_spec (rtx_insn *, rtx_insn *, ds_t);
881 static void begin_speculative_block (rtx_insn *);
882 static void add_to_speculative_block (rtx_insn *);
883 static void init_before_recovery (basic_block *);
884 static void create_check_block_twin (rtx_insn *, bool);
885 static void fix_recovery_deps (basic_block);
886 static bool haifa_change_pattern (rtx_insn *, rtx);
887 static void dump_new_block_header (int, basic_block, rtx_insn *, rtx_insn *);
888 static void restore_bb_notes (basic_block);
889 static void fix_jump_move (rtx_insn *);
890 static void move_block_after_check (rtx_insn *);
891 static void move_succs (vec<edge, va_gc> **, basic_block);
892 static void sched_remove_insn (rtx_insn *);
893 static void clear_priorities (rtx_insn *, rtx_vec_t *);
894 static void calc_priorities (rtx_vec_t);
895 static void add_jump_dependencies (rtx_insn *, rtx_insn *);
897 #endif /* INSN_SCHEDULING */
899 /* Point to state used for the current scheduling pass. */
900 struct haifa_sched_info *current_sched_info;
902 #ifndef INSN_SCHEDULING
903 void
904 schedule_insns (void)
907 #else
909 /* Do register pressure sensitive insn scheduling if the flag is set
910 up. */
911 enum sched_pressure_algorithm sched_pressure;
913 /* Map regno -> its pressure class. The map defined only when
914 SCHED_PRESSURE != SCHED_PRESSURE_NONE. */
915 enum reg_class *sched_regno_pressure_class;
917 /* The current register pressure. Only elements corresponding pressure
918 classes are defined. */
919 static int curr_reg_pressure[N_REG_CLASSES];
921 /* Saved value of the previous array. */
922 static int saved_reg_pressure[N_REG_CLASSES];
924 /* Register living at given scheduling point. */
925 static bitmap curr_reg_live;
927 /* Saved value of the previous array. */
928 static bitmap saved_reg_live;
930 /* Registers mentioned in the current region. */
931 static bitmap region_ref_regs;
933 /* Effective number of available registers of a given class (see comment
934 in sched_pressure_start_bb). */
935 static int sched_class_regs_num[N_REG_CLASSES];
936 /* Number of call_saved_regs and fixed_regs. Helpers for calculating of
937 sched_class_regs_num. */
938 static int call_saved_regs_num[N_REG_CLASSES];
939 static int fixed_regs_num[N_REG_CLASSES];
941 /* Initiate register pressure relative info for scheduling the current
942 region. Currently it is only clearing register mentioned in the
943 current region. */
944 void
945 sched_init_region_reg_pressure_info (void)
947 bitmap_clear (region_ref_regs);
950 /* PRESSURE[CL] describes the pressure on register class CL. Update it
951 for the birth (if BIRTH_P) or death (if !BIRTH_P) of register REGNO.
952 LIVE tracks the set of live registers; if it is null, assume that
953 every birth or death is genuine. */
954 static inline void
955 mark_regno_birth_or_death (bitmap live, int *pressure, int regno, bool birth_p)
957 enum reg_class pressure_class;
959 pressure_class = sched_regno_pressure_class[regno];
960 if (regno >= FIRST_PSEUDO_REGISTER)
962 if (pressure_class != NO_REGS)
964 if (birth_p)
966 if (!live || bitmap_set_bit (live, regno))
967 pressure[pressure_class]
968 += (ira_reg_class_max_nregs
969 [pressure_class][PSEUDO_REGNO_MODE (regno)]);
971 else
973 if (!live || bitmap_clear_bit (live, regno))
974 pressure[pressure_class]
975 -= (ira_reg_class_max_nregs
976 [pressure_class][PSEUDO_REGNO_MODE (regno)]);
980 else if (pressure_class != NO_REGS
981 && ! TEST_HARD_REG_BIT (ira_no_alloc_regs, regno))
983 if (birth_p)
985 if (!live || bitmap_set_bit (live, regno))
986 pressure[pressure_class]++;
988 else
990 if (!live || bitmap_clear_bit (live, regno))
991 pressure[pressure_class]--;
996 /* Initiate current register pressure related info from living
997 registers given by LIVE. */
998 static void
999 initiate_reg_pressure_info (bitmap live)
1001 int i;
1002 unsigned int j;
1003 bitmap_iterator bi;
1005 for (i = 0; i < ira_pressure_classes_num; i++)
1006 curr_reg_pressure[ira_pressure_classes[i]] = 0;
1007 bitmap_clear (curr_reg_live);
1008 EXECUTE_IF_SET_IN_BITMAP (live, 0, j, bi)
1009 if (sched_pressure == SCHED_PRESSURE_MODEL
1010 || current_nr_blocks == 1
1011 || bitmap_bit_p (region_ref_regs, j))
1012 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure, j, true);
1015 /* Mark registers in X as mentioned in the current region. */
1016 static void
1017 setup_ref_regs (rtx x)
1019 int i, j;
1020 const RTX_CODE code = GET_CODE (x);
1021 const char *fmt;
1023 if (REG_P (x))
1025 bitmap_set_range (region_ref_regs, REGNO (x), REG_NREGS (x));
1026 return;
1028 fmt = GET_RTX_FORMAT (code);
1029 for (i = GET_RTX_LENGTH (code) - 1; i >= 0; i--)
1030 if (fmt[i] == 'e')
1031 setup_ref_regs (XEXP (x, i));
1032 else if (fmt[i] == 'E')
1034 for (j = 0; j < XVECLEN (x, i); j++)
1035 setup_ref_regs (XVECEXP (x, i, j));
1039 /* Initiate current register pressure related info at the start of
1040 basic block BB. */
1041 static void
1042 initiate_bb_reg_pressure_info (basic_block bb)
1044 unsigned int i ATTRIBUTE_UNUSED;
1045 rtx_insn *insn;
1047 if (current_nr_blocks > 1)
1048 FOR_BB_INSNS (bb, insn)
1049 if (NONDEBUG_INSN_P (insn))
1050 setup_ref_regs (PATTERN (insn));
1051 initiate_reg_pressure_info (df_get_live_in (bb));
1052 if (bb_has_eh_pred (bb))
1053 for (i = 0; ; ++i)
1055 unsigned int regno = EH_RETURN_DATA_REGNO (i);
1057 if (regno == INVALID_REGNUM)
1058 break;
1059 if (! bitmap_bit_p (df_get_live_in (bb), regno))
1060 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure,
1061 regno, true);
1065 /* Save current register pressure related info. */
1066 static void
1067 save_reg_pressure (void)
1069 int i;
1071 for (i = 0; i < ira_pressure_classes_num; i++)
1072 saved_reg_pressure[ira_pressure_classes[i]]
1073 = curr_reg_pressure[ira_pressure_classes[i]];
1074 bitmap_copy (saved_reg_live, curr_reg_live);
1077 /* Restore saved register pressure related info. */
1078 static void
1079 restore_reg_pressure (void)
1081 int i;
1083 for (i = 0; i < ira_pressure_classes_num; i++)
1084 curr_reg_pressure[ira_pressure_classes[i]]
1085 = saved_reg_pressure[ira_pressure_classes[i]];
1086 bitmap_copy (curr_reg_live, saved_reg_live);
1089 /* Return TRUE if the register is dying after its USE. */
1090 static bool
1091 dying_use_p (struct reg_use_data *use)
1093 struct reg_use_data *next;
1095 for (next = use->next_regno_use; next != use; next = next->next_regno_use)
1096 if (NONDEBUG_INSN_P (next->insn)
1097 && QUEUE_INDEX (next->insn) != QUEUE_SCHEDULED)
1098 return false;
1099 return true;
1102 /* Print info about the current register pressure and its excess for
1103 each pressure class. */
1104 static void
1105 print_curr_reg_pressure (void)
1107 int i;
1108 enum reg_class cl;
1110 fprintf (sched_dump, ";;\t");
1111 for (i = 0; i < ira_pressure_classes_num; i++)
1113 cl = ira_pressure_classes[i];
1114 gcc_assert (curr_reg_pressure[cl] >= 0);
1115 fprintf (sched_dump, " %s:%d(%d)", reg_class_names[cl],
1116 curr_reg_pressure[cl],
1117 curr_reg_pressure[cl] - sched_class_regs_num[cl]);
1119 fprintf (sched_dump, "\n");
1122 /* Determine if INSN has a condition that is clobbered if a register
1123 in SET_REGS is modified. */
1124 static bool
1125 cond_clobbered_p (rtx_insn *insn, HARD_REG_SET set_regs)
1127 rtx pat = PATTERN (insn);
1128 gcc_assert (GET_CODE (pat) == COND_EXEC);
1129 if (TEST_HARD_REG_BIT (set_regs, REGNO (XEXP (COND_EXEC_TEST (pat), 0))))
1131 sd_iterator_def sd_it;
1132 dep_t dep;
1133 haifa_change_pattern (insn, ORIG_PAT (insn));
1134 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep)
1135 DEP_STATUS (dep) &= ~DEP_CANCELLED;
1136 TODO_SPEC (insn) = HARD_DEP;
1137 if (sched_verbose >= 2)
1138 fprintf (sched_dump,
1139 ";;\t\tdequeue insn %s because of clobbered condition\n",
1140 (*current_sched_info->print_insn) (insn, 0));
1141 return true;
1144 return false;
1147 /* This function should be called after modifying the pattern of INSN,
1148 to update scheduler data structures as needed. */
1149 static void
1150 update_insn_after_change (rtx_insn *insn)
1152 sd_iterator_def sd_it;
1153 dep_t dep;
1155 dfa_clear_single_insn_cache (insn);
1157 sd_it = sd_iterator_start (insn,
1158 SD_LIST_FORW | SD_LIST_BACK | SD_LIST_RES_BACK);
1159 while (sd_iterator_cond (&sd_it, &dep))
1161 DEP_COST (dep) = UNKNOWN_DEP_COST;
1162 sd_iterator_next (&sd_it);
1165 /* Invalidate INSN_COST, so it'll be recalculated. */
1166 INSN_COST (insn) = -1;
1167 /* Invalidate INSN_TICK, so it'll be recalculated. */
1168 INSN_TICK (insn) = INVALID_TICK;
1170 /* Invalidate autoprefetch data entry. */
1171 INSN_AUTOPREF_MULTIPASS_DATA (insn)[0].status
1172 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED;
1173 INSN_AUTOPREF_MULTIPASS_DATA (insn)[1].status
1174 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED;
1178 /* Two VECs, one to hold dependencies for which pattern replacements
1179 need to be applied or restored at the start of the next cycle, and
1180 another to hold an integer that is either one, to apply the
1181 corresponding replacement, or zero to restore it. */
1182 static vec<dep_t> next_cycle_replace_deps;
1183 static vec<int> next_cycle_apply;
1185 static void apply_replacement (dep_t, bool);
1186 static void restore_pattern (dep_t, bool);
1188 /* Look at the remaining dependencies for insn NEXT, and compute and return
1189 the TODO_SPEC value we should use for it. This is called after one of
1190 NEXT's dependencies has been resolved.
1191 We also perform pattern replacements for predication, and for broken
1192 replacement dependencies. The latter is only done if FOR_BACKTRACK is
1193 false. */
1195 static ds_t
1196 recompute_todo_spec (rtx_insn *next, bool for_backtrack)
1198 ds_t new_ds;
1199 sd_iterator_def sd_it;
1200 dep_t dep, modify_dep = NULL;
1201 int n_spec = 0;
1202 int n_control = 0;
1203 int n_replace = 0;
1204 bool first_p = true;
1206 if (sd_lists_empty_p (next, SD_LIST_BACK))
1207 /* NEXT has all its dependencies resolved. */
1208 return 0;
1210 if (!sd_lists_empty_p (next, SD_LIST_HARD_BACK))
1211 return HARD_DEP;
1213 /* If NEXT is intended to sit adjacent to this instruction, we don't
1214 want to try to break any dependencies. Treat it as a HARD_DEP. */
1215 if (SCHED_GROUP_P (next))
1216 return HARD_DEP;
1218 /* Now we've got NEXT with speculative deps only.
1219 1. Look at the deps to see what we have to do.
1220 2. Check if we can do 'todo'. */
1221 new_ds = 0;
1223 FOR_EACH_DEP (next, SD_LIST_BACK, sd_it, dep)
1225 rtx_insn *pro = DEP_PRO (dep);
1226 ds_t ds = DEP_STATUS (dep) & SPECULATIVE;
1228 if (DEBUG_INSN_P (pro) && !DEBUG_INSN_P (next))
1229 continue;
1231 if (ds)
1233 n_spec++;
1234 if (first_p)
1236 first_p = false;
1238 new_ds = ds;
1240 else
1241 new_ds = ds_merge (new_ds, ds);
1243 else if (DEP_TYPE (dep) == REG_DEP_CONTROL)
1245 if (QUEUE_INDEX (pro) != QUEUE_SCHEDULED)
1247 n_control++;
1248 modify_dep = dep;
1250 DEP_STATUS (dep) &= ~DEP_CANCELLED;
1252 else if (DEP_REPLACE (dep) != NULL)
1254 if (QUEUE_INDEX (pro) != QUEUE_SCHEDULED)
1256 n_replace++;
1257 modify_dep = dep;
1259 DEP_STATUS (dep) &= ~DEP_CANCELLED;
1263 if (n_replace > 0 && n_control == 0 && n_spec == 0)
1265 if (!dbg_cnt (sched_breakdep))
1266 return HARD_DEP;
1267 FOR_EACH_DEP (next, SD_LIST_BACK, sd_it, dep)
1269 struct dep_replacement *desc = DEP_REPLACE (dep);
1270 if (desc != NULL)
1272 if (desc->insn == next && !for_backtrack)
1274 gcc_assert (n_replace == 1);
1275 apply_replacement (dep, true);
1277 DEP_STATUS (dep) |= DEP_CANCELLED;
1280 return 0;
1283 else if (n_control == 1 && n_replace == 0 && n_spec == 0)
1285 rtx_insn *pro, *other;
1286 rtx new_pat;
1287 rtx cond = NULL_RTX;
1288 bool success;
1289 rtx_insn *prev = NULL;
1290 int i;
1291 unsigned regno;
1293 if ((current_sched_info->flags & DO_PREDICATION) == 0
1294 || (ORIG_PAT (next) != NULL_RTX
1295 && PREDICATED_PAT (next) == NULL_RTX))
1296 return HARD_DEP;
1298 pro = DEP_PRO (modify_dep);
1299 other = real_insn_for_shadow (pro);
1300 if (other != NULL_RTX)
1301 pro = other;
1303 cond = sched_get_reverse_condition_uncached (pro);
1304 regno = REGNO (XEXP (cond, 0));
1306 /* Find the last scheduled insn that modifies the condition register.
1307 We can stop looking once we find the insn we depend on through the
1308 REG_DEP_CONTROL; if the condition register isn't modified after it,
1309 we know that it still has the right value. */
1310 if (QUEUE_INDEX (pro) == QUEUE_SCHEDULED)
1311 FOR_EACH_VEC_ELT_REVERSE (scheduled_insns, i, prev)
1313 HARD_REG_SET t;
1315 find_all_hard_reg_sets (prev, &t, true);
1316 if (TEST_HARD_REG_BIT (t, regno))
1317 return HARD_DEP;
1318 if (prev == pro)
1319 break;
1321 if (ORIG_PAT (next) == NULL_RTX)
1323 ORIG_PAT (next) = PATTERN (next);
1325 new_pat = gen_rtx_COND_EXEC (VOIDmode, cond, PATTERN (next));
1326 success = haifa_change_pattern (next, new_pat);
1327 if (!success)
1328 return HARD_DEP;
1329 PREDICATED_PAT (next) = new_pat;
1331 else if (PATTERN (next) != PREDICATED_PAT (next))
1333 bool success = haifa_change_pattern (next,
1334 PREDICATED_PAT (next));
1335 gcc_assert (success);
1337 DEP_STATUS (modify_dep) |= DEP_CANCELLED;
1338 return DEP_CONTROL;
1341 if (PREDICATED_PAT (next) != NULL_RTX)
1343 int tick = INSN_TICK (next);
1344 bool success = haifa_change_pattern (next,
1345 ORIG_PAT (next));
1346 INSN_TICK (next) = tick;
1347 gcc_assert (success);
1350 /* We can't handle the case where there are both speculative and control
1351 dependencies, so we return HARD_DEP in such a case. Also fail if
1352 we have speculative dependencies with not enough points, or more than
1353 one control dependency. */
1354 if ((n_spec > 0 && (n_control > 0 || n_replace > 0))
1355 || (n_spec > 0
1356 /* Too few points? */
1357 && ds_weak (new_ds) < spec_info->data_weakness_cutoff)
1358 || n_control > 0
1359 || n_replace > 0)
1360 return HARD_DEP;
1362 return new_ds;
1365 /* Pointer to the last instruction scheduled. */
1366 static rtx_insn *last_scheduled_insn;
1368 /* Pointer to the last nondebug instruction scheduled within the
1369 block, or the prev_head of the scheduling block. Used by
1370 rank_for_schedule, so that insns independent of the last scheduled
1371 insn will be preferred over dependent instructions. */
1372 static rtx_insn *last_nondebug_scheduled_insn;
1374 /* Pointer that iterates through the list of unscheduled insns if we
1375 have a dbg_cnt enabled. It always points at an insn prior to the
1376 first unscheduled one. */
1377 static rtx_insn *nonscheduled_insns_begin;
1379 /* Compute cost of executing INSN.
1380 This is the number of cycles between instruction issue and
1381 instruction results. */
1383 insn_cost (rtx_insn *insn)
1385 int cost;
1387 if (sched_fusion)
1388 return 0;
1390 if (sel_sched_p ())
1392 if (recog_memoized (insn) < 0)
1393 return 0;
1395 cost = insn_default_latency (insn);
1396 if (cost < 0)
1397 cost = 0;
1399 return cost;
1402 cost = INSN_COST (insn);
1404 if (cost < 0)
1406 /* A USE insn, or something else we don't need to
1407 understand. We can't pass these directly to
1408 result_ready_cost or insn_default_latency because it will
1409 trigger a fatal error for unrecognizable insns. */
1410 if (recog_memoized (insn) < 0)
1412 INSN_COST (insn) = 0;
1413 return 0;
1415 else
1417 cost = insn_default_latency (insn);
1418 if (cost < 0)
1419 cost = 0;
1421 INSN_COST (insn) = cost;
1425 return cost;
1428 /* Compute cost of dependence LINK.
1429 This is the number of cycles between instruction issue and
1430 instruction results.
1431 ??? We also use this function to call recog_memoized on all insns. */
1433 dep_cost_1 (dep_t link, dw_t dw)
1435 rtx_insn *insn = DEP_PRO (link);
1436 rtx_insn *used = DEP_CON (link);
1437 int cost;
1439 if (DEP_COST (link) != UNKNOWN_DEP_COST)
1440 return DEP_COST (link);
1442 if (delay_htab)
1444 struct delay_pair *delay_entry;
1445 delay_entry
1446 = delay_htab_i2->find_with_hash (used, htab_hash_pointer (used));
1447 if (delay_entry)
1449 if (delay_entry->i1 == insn)
1451 DEP_COST (link) = pair_delay (delay_entry);
1452 return DEP_COST (link);
1457 /* A USE insn should never require the value used to be computed.
1458 This allows the computation of a function's result and parameter
1459 values to overlap the return and call. We don't care about the
1460 dependence cost when only decreasing register pressure. */
1461 if (recog_memoized (used) < 0)
1463 cost = 0;
1464 recog_memoized (insn);
1466 else
1468 enum reg_note dep_type = DEP_TYPE (link);
1470 cost = insn_cost (insn);
1472 if (INSN_CODE (insn) >= 0)
1474 if (dep_type == REG_DEP_ANTI)
1475 cost = 0;
1476 else if (dep_type == REG_DEP_OUTPUT)
1478 cost = (insn_default_latency (insn)
1479 - insn_default_latency (used));
1480 if (cost <= 0)
1481 cost = 1;
1483 else if (bypass_p (insn))
1484 cost = insn_latency (insn, used);
1488 if (targetm.sched.adjust_cost)
1489 cost = targetm.sched.adjust_cost (used, (int) dep_type, insn, cost,
1490 dw);
1492 if (cost < 0)
1493 cost = 0;
1496 DEP_COST (link) = cost;
1497 return cost;
1500 /* Compute cost of dependence LINK.
1501 This is the number of cycles between instruction issue and
1502 instruction results. */
1504 dep_cost (dep_t link)
1506 return dep_cost_1 (link, 0);
1509 /* Use this sel-sched.c friendly function in reorder2 instead of increasing
1510 INSN_PRIORITY explicitly. */
1511 void
1512 increase_insn_priority (rtx_insn *insn, int amount)
1514 if (!sel_sched_p ())
1516 /* We're dealing with haifa-sched.c INSN_PRIORITY. */
1517 if (INSN_PRIORITY_KNOWN (insn))
1518 INSN_PRIORITY (insn) += amount;
1520 else
1522 /* In sel-sched.c INSN_PRIORITY is not kept up to date.
1523 Use EXPR_PRIORITY instead. */
1524 sel_add_to_insn_priority (insn, amount);
1528 /* Return 'true' if DEP should be included in priority calculations. */
1529 static bool
1530 contributes_to_priority_p (dep_t dep)
1532 if (DEBUG_INSN_P (DEP_CON (dep))
1533 || DEBUG_INSN_P (DEP_PRO (dep)))
1534 return false;
1536 /* Critical path is meaningful in block boundaries only. */
1537 if (!current_sched_info->contributes_to_priority (DEP_CON (dep),
1538 DEP_PRO (dep)))
1539 return false;
1541 if (DEP_REPLACE (dep) != NULL)
1542 return false;
1544 /* If flag COUNT_SPEC_IN_CRITICAL_PATH is set,
1545 then speculative instructions will less likely be
1546 scheduled. That is because the priority of
1547 their producers will increase, and, thus, the
1548 producers will more likely be scheduled, thus,
1549 resolving the dependence. */
1550 if (sched_deps_info->generate_spec_deps
1551 && !(spec_info->flags & COUNT_SPEC_IN_CRITICAL_PATH)
1552 && (DEP_STATUS (dep) & SPECULATIVE))
1553 return false;
1555 return true;
1558 /* Compute the number of nondebug deps in list LIST for INSN. */
1560 static int
1561 dep_list_size (rtx_insn *insn, sd_list_types_def list)
1563 sd_iterator_def sd_it;
1564 dep_t dep;
1565 int dbgcount = 0, nodbgcount = 0;
1567 if (!MAY_HAVE_DEBUG_INSNS)
1568 return sd_lists_size (insn, list);
1570 FOR_EACH_DEP (insn, list, sd_it, dep)
1572 if (DEBUG_INSN_P (DEP_CON (dep)))
1573 dbgcount++;
1574 else if (!DEBUG_INSN_P (DEP_PRO (dep)))
1575 nodbgcount++;
1578 gcc_assert (dbgcount + nodbgcount == sd_lists_size (insn, list));
1580 return nodbgcount;
1583 bool sched_fusion;
1585 /* Compute the priority number for INSN. */
1586 static int
1587 priority (rtx_insn *insn)
1589 if (! INSN_P (insn))
1590 return 0;
1592 /* We should not be interested in priority of an already scheduled insn. */
1593 gcc_assert (QUEUE_INDEX (insn) != QUEUE_SCHEDULED);
1595 if (!INSN_PRIORITY_KNOWN (insn))
1597 int this_priority = -1;
1599 if (sched_fusion)
1601 int this_fusion_priority;
1603 targetm.sched.fusion_priority (insn, FUSION_MAX_PRIORITY,
1604 &this_fusion_priority, &this_priority);
1605 INSN_FUSION_PRIORITY (insn) = this_fusion_priority;
1607 else if (dep_list_size (insn, SD_LIST_FORW) == 0)
1608 /* ??? We should set INSN_PRIORITY to insn_cost when and insn has
1609 some forward deps but all of them are ignored by
1610 contributes_to_priority hook. At the moment we set priority of
1611 such insn to 0. */
1612 this_priority = insn_cost (insn);
1613 else
1615 rtx_insn *prev_first, *twin;
1616 basic_block rec;
1618 /* For recovery check instructions we calculate priority slightly
1619 different than that of normal instructions. Instead of walking
1620 through INSN_FORW_DEPS (check) list, we walk through
1621 INSN_FORW_DEPS list of each instruction in the corresponding
1622 recovery block. */
1624 /* Selective scheduling does not define RECOVERY_BLOCK macro. */
1625 rec = sel_sched_p () ? NULL : RECOVERY_BLOCK (insn);
1626 if (!rec || rec == EXIT_BLOCK_PTR_FOR_FN (cfun))
1628 prev_first = PREV_INSN (insn);
1629 twin = insn;
1631 else
1633 prev_first = NEXT_INSN (BB_HEAD (rec));
1634 twin = PREV_INSN (BB_END (rec));
1639 sd_iterator_def sd_it;
1640 dep_t dep;
1642 FOR_EACH_DEP (twin, SD_LIST_FORW, sd_it, dep)
1644 rtx_insn *next;
1645 int next_priority;
1647 next = DEP_CON (dep);
1649 if (BLOCK_FOR_INSN (next) != rec)
1651 int cost;
1653 if (!contributes_to_priority_p (dep))
1654 continue;
1656 if (twin == insn)
1657 cost = dep_cost (dep);
1658 else
1660 struct _dep _dep1, *dep1 = &_dep1;
1662 init_dep (dep1, insn, next, REG_DEP_ANTI);
1664 cost = dep_cost (dep1);
1667 next_priority = cost + priority (next);
1669 if (next_priority > this_priority)
1670 this_priority = next_priority;
1674 twin = PREV_INSN (twin);
1676 while (twin != prev_first);
1679 if (this_priority < 0)
1681 gcc_assert (this_priority == -1);
1683 this_priority = insn_cost (insn);
1686 INSN_PRIORITY (insn) = this_priority;
1687 INSN_PRIORITY_STATUS (insn) = 1;
1690 return INSN_PRIORITY (insn);
1693 /* Macros and functions for keeping the priority queue sorted, and
1694 dealing with queuing and dequeuing of instructions. */
1696 /* For each pressure class CL, set DEATH[CL] to the number of registers
1697 in that class that die in INSN. */
1699 static void
1700 calculate_reg_deaths (rtx_insn *insn, int *death)
1702 int i;
1703 struct reg_use_data *use;
1705 for (i = 0; i < ira_pressure_classes_num; i++)
1706 death[ira_pressure_classes[i]] = 0;
1707 for (use = INSN_REG_USE_LIST (insn); use != NULL; use = use->next_insn_use)
1708 if (dying_use_p (use))
1709 mark_regno_birth_or_death (0, death, use->regno, true);
1712 /* Setup info about the current register pressure impact of scheduling
1713 INSN at the current scheduling point. */
1714 static void
1715 setup_insn_reg_pressure_info (rtx_insn *insn)
1717 int i, change, before, after, hard_regno;
1718 int excess_cost_change;
1719 machine_mode mode;
1720 enum reg_class cl;
1721 struct reg_pressure_data *pressure_info;
1722 int *max_reg_pressure;
1723 static int death[N_REG_CLASSES];
1725 gcc_checking_assert (!DEBUG_INSN_P (insn));
1727 excess_cost_change = 0;
1728 calculate_reg_deaths (insn, death);
1729 pressure_info = INSN_REG_PRESSURE (insn);
1730 max_reg_pressure = INSN_MAX_REG_PRESSURE (insn);
1731 gcc_assert (pressure_info != NULL && max_reg_pressure != NULL);
1732 for (i = 0; i < ira_pressure_classes_num; i++)
1734 cl = ira_pressure_classes[i];
1735 gcc_assert (curr_reg_pressure[cl] >= 0);
1736 change = (int) pressure_info[i].set_increase - death[cl];
1737 before = MAX (0, max_reg_pressure[i] - sched_class_regs_num[cl]);
1738 after = MAX (0, max_reg_pressure[i] + change
1739 - sched_class_regs_num[cl]);
1740 hard_regno = ira_class_hard_regs[cl][0];
1741 gcc_assert (hard_regno >= 0);
1742 mode = reg_raw_mode[hard_regno];
1743 excess_cost_change += ((after - before)
1744 * (ira_memory_move_cost[mode][cl][0]
1745 + ira_memory_move_cost[mode][cl][1]));
1747 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insn) = excess_cost_change;
1750 /* This is the first page of code related to SCHED_PRESSURE_MODEL.
1751 It tries to make the scheduler take register pressure into account
1752 without introducing too many unnecessary stalls. It hooks into the
1753 main scheduling algorithm at several points:
1755 - Before scheduling starts, model_start_schedule constructs a
1756 "model schedule" for the current block. This model schedule is
1757 chosen solely to keep register pressure down. It does not take the
1758 target's pipeline or the original instruction order into account,
1759 except as a tie-breaker. It also doesn't work to a particular
1760 pressure limit.
1762 This model schedule gives us an idea of what pressure can be
1763 achieved for the block and gives us an example of a schedule that
1764 keeps to that pressure. It also makes the final schedule less
1765 dependent on the original instruction order. This is important
1766 because the original order can either be "wide" (many values live
1767 at once, such as in user-scheduled code) or "narrow" (few values
1768 live at once, such as after loop unrolling, where several
1769 iterations are executed sequentially).
1771 We do not apply this model schedule to the rtx stream. We simply
1772 record it in model_schedule. We also compute the maximum pressure,
1773 MP, that was seen during this schedule.
1775 - Instructions are added to the ready queue even if they require
1776 a stall. The length of the stall is instead computed as:
1778 MAX (INSN_TICK (INSN) - clock_var, 0)
1780 (= insn_delay). This allows rank_for_schedule to choose between
1781 introducing a deliberate stall or increasing pressure.
1783 - Before sorting the ready queue, model_set_excess_costs assigns
1784 a pressure-based cost to each ready instruction in the queue.
1785 This is the instruction's INSN_REG_PRESSURE_EXCESS_COST_CHANGE
1786 (ECC for short) and is effectively measured in cycles.
1788 - rank_for_schedule ranks instructions based on:
1790 ECC (insn) + insn_delay (insn)
1792 then as:
1794 insn_delay (insn)
1796 So, for example, an instruction X1 with an ECC of 1 that can issue
1797 now will win over an instruction X0 with an ECC of zero that would
1798 introduce a stall of one cycle. However, an instruction X2 with an
1799 ECC of 2 that can issue now will lose to both X0 and X1.
1801 - When an instruction is scheduled, model_recompute updates the model
1802 schedule with the new pressures (some of which might now exceed the
1803 original maximum pressure MP). model_update_limit_points then searches
1804 for the new point of maximum pressure, if not already known. */
1806 /* Used to separate high-verbosity debug information for SCHED_PRESSURE_MODEL
1807 from surrounding debug information. */
1808 #define MODEL_BAR \
1809 ";;\t\t+------------------------------------------------------\n"
1811 /* Information about the pressure on a particular register class at a
1812 particular point of the model schedule. */
1813 struct model_pressure_data {
1814 /* The pressure at this point of the model schedule, or -1 if the
1815 point is associated with an instruction that has already been
1816 scheduled. */
1817 int ref_pressure;
1819 /* The maximum pressure during or after this point of the model schedule. */
1820 int max_pressure;
1823 /* Per-instruction information that is used while building the model
1824 schedule. Here, "schedule" refers to the model schedule rather
1825 than the main schedule. */
1826 struct model_insn_info {
1827 /* The instruction itself. */
1828 rtx_insn *insn;
1830 /* If this instruction is in model_worklist, these fields link to the
1831 previous (higher-priority) and next (lower-priority) instructions
1832 in the list. */
1833 struct model_insn_info *prev;
1834 struct model_insn_info *next;
1836 /* While constructing the schedule, QUEUE_INDEX describes whether an
1837 instruction has already been added to the schedule (QUEUE_SCHEDULED),
1838 is in model_worklist (QUEUE_READY), or neither (QUEUE_NOWHERE).
1839 old_queue records the value that QUEUE_INDEX had before scheduling
1840 started, so that we can restore it once the schedule is complete. */
1841 int old_queue;
1843 /* The relative importance of an unscheduled instruction. Higher
1844 values indicate greater importance. */
1845 unsigned int model_priority;
1847 /* The length of the longest path of satisfied true dependencies
1848 that leads to this instruction. */
1849 unsigned int depth;
1851 /* The length of the longest path of dependencies of any kind
1852 that leads from this instruction. */
1853 unsigned int alap;
1855 /* The number of predecessor nodes that must still be scheduled. */
1856 int unscheduled_preds;
1859 /* Information about the pressure limit for a particular register class.
1860 This structure is used when applying a model schedule to the main
1861 schedule. */
1862 struct model_pressure_limit {
1863 /* The maximum register pressure seen in the original model schedule. */
1864 int orig_pressure;
1866 /* The maximum register pressure seen in the current model schedule
1867 (which excludes instructions that have already been scheduled). */
1868 int pressure;
1870 /* The point of the current model schedule at which PRESSURE is first
1871 reached. It is set to -1 if the value needs to be recomputed. */
1872 int point;
1875 /* Describes a particular way of measuring register pressure. */
1876 struct model_pressure_group {
1877 /* Index PCI describes the maximum pressure on ira_pressure_classes[PCI]. */
1878 struct model_pressure_limit limits[N_REG_CLASSES];
1880 /* Index (POINT * ira_num_pressure_classes + PCI) describes the pressure
1881 on register class ira_pressure_classes[PCI] at point POINT of the
1882 current model schedule. A POINT of model_num_insns describes the
1883 pressure at the end of the schedule. */
1884 struct model_pressure_data *model;
1887 /* Index POINT gives the instruction at point POINT of the model schedule.
1888 This array doesn't change during main scheduling. */
1889 static vec<rtx_insn *> model_schedule;
1891 /* The list of instructions in the model worklist, sorted in order of
1892 decreasing priority. */
1893 static struct model_insn_info *model_worklist;
1895 /* Index I describes the instruction with INSN_LUID I. */
1896 static struct model_insn_info *model_insns;
1898 /* The number of instructions in the model schedule. */
1899 static int model_num_insns;
1901 /* The index of the first instruction in model_schedule that hasn't yet been
1902 added to the main schedule, or model_num_insns if all of them have. */
1903 static int model_curr_point;
1905 /* Describes the pressure before each instruction in the model schedule. */
1906 static struct model_pressure_group model_before_pressure;
1908 /* The first unused model_priority value (as used in model_insn_info). */
1909 static unsigned int model_next_priority;
1912 /* The model_pressure_data for ira_pressure_classes[PCI] in GROUP
1913 at point POINT of the model schedule. */
1914 #define MODEL_PRESSURE_DATA(GROUP, POINT, PCI) \
1915 (&(GROUP)->model[(POINT) * ira_pressure_classes_num + (PCI)])
1917 /* The maximum pressure on ira_pressure_classes[PCI] in GROUP at or
1918 after point POINT of the model schedule. */
1919 #define MODEL_MAX_PRESSURE(GROUP, POINT, PCI) \
1920 (MODEL_PRESSURE_DATA (GROUP, POINT, PCI)->max_pressure)
1922 /* The pressure on ira_pressure_classes[PCI] in GROUP at point POINT
1923 of the model schedule. */
1924 #define MODEL_REF_PRESSURE(GROUP, POINT, PCI) \
1925 (MODEL_PRESSURE_DATA (GROUP, POINT, PCI)->ref_pressure)
1927 /* Information about INSN that is used when creating the model schedule. */
1928 #define MODEL_INSN_INFO(INSN) \
1929 (&model_insns[INSN_LUID (INSN)])
1931 /* The instruction at point POINT of the model schedule. */
1932 #define MODEL_INSN(POINT) \
1933 (model_schedule[POINT])
1936 /* Return INSN's index in the model schedule, or model_num_insns if it
1937 doesn't belong to that schedule. */
1939 static int
1940 model_index (rtx_insn *insn)
1942 if (INSN_MODEL_INDEX (insn) == 0)
1943 return model_num_insns;
1944 return INSN_MODEL_INDEX (insn) - 1;
1947 /* Make sure that GROUP->limits is up-to-date for the current point
1948 of the model schedule. */
1950 static void
1951 model_update_limit_points_in_group (struct model_pressure_group *group)
1953 int pci, max_pressure, point;
1955 for (pci = 0; pci < ira_pressure_classes_num; pci++)
1957 /* We may have passed the final point at which the pressure in
1958 group->limits[pci].pressure was reached. Update the limit if so. */
1959 max_pressure = MODEL_MAX_PRESSURE (group, model_curr_point, pci);
1960 group->limits[pci].pressure = max_pressure;
1962 /* Find the point at which MAX_PRESSURE is first reached. We need
1963 to search in three cases:
1965 - We've already moved past the previous pressure point.
1966 In this case we search forward from model_curr_point.
1968 - We scheduled the previous point of maximum pressure ahead of
1969 its position in the model schedule, but doing so didn't bring
1970 the pressure point earlier. In this case we search forward
1971 from that previous pressure point.
1973 - Scheduling an instruction early caused the maximum pressure
1974 to decrease. In this case we will have set the pressure
1975 point to -1, and we search forward from model_curr_point. */
1976 point = MAX (group->limits[pci].point, model_curr_point);
1977 while (point < model_num_insns
1978 && MODEL_REF_PRESSURE (group, point, pci) < max_pressure)
1979 point++;
1980 group->limits[pci].point = point;
1982 gcc_assert (MODEL_REF_PRESSURE (group, point, pci) == max_pressure);
1983 gcc_assert (MODEL_MAX_PRESSURE (group, point, pci) == max_pressure);
1987 /* Make sure that all register-pressure limits are up-to-date for the
1988 current position in the model schedule. */
1990 static void
1991 model_update_limit_points (void)
1993 model_update_limit_points_in_group (&model_before_pressure);
1996 /* Return the model_index of the last unscheduled use in chain USE
1997 outside of USE's instruction. Return -1 if there are no other uses,
1998 or model_num_insns if the register is live at the end of the block. */
2000 static int
2001 model_last_use_except (struct reg_use_data *use)
2003 struct reg_use_data *next;
2004 int last, index;
2006 last = -1;
2007 for (next = use->next_regno_use; next != use; next = next->next_regno_use)
2008 if (NONDEBUG_INSN_P (next->insn)
2009 && QUEUE_INDEX (next->insn) != QUEUE_SCHEDULED)
2011 index = model_index (next->insn);
2012 if (index == model_num_insns)
2013 return model_num_insns;
2014 if (last < index)
2015 last = index;
2017 return last;
2020 /* An instruction with model_index POINT has just been scheduled, and it
2021 adds DELTA to the pressure on ira_pressure_classes[PCI] after POINT - 1.
2022 Update MODEL_REF_PRESSURE (GROUP, POINT, PCI) and
2023 MODEL_MAX_PRESSURE (GROUP, POINT, PCI) accordingly. */
2025 static void
2026 model_start_update_pressure (struct model_pressure_group *group,
2027 int point, int pci, int delta)
2029 int next_max_pressure;
2031 if (point == model_num_insns)
2033 /* The instruction wasn't part of the model schedule; it was moved
2034 from a different block. Update the pressure for the end of
2035 the model schedule. */
2036 MODEL_REF_PRESSURE (group, point, pci) += delta;
2037 MODEL_MAX_PRESSURE (group, point, pci) += delta;
2039 else
2041 /* Record that this instruction has been scheduled. Nothing now
2042 changes between POINT and POINT + 1, so get the maximum pressure
2043 from the latter. If the maximum pressure decreases, the new
2044 pressure point may be before POINT. */
2045 MODEL_REF_PRESSURE (group, point, pci) = -1;
2046 next_max_pressure = MODEL_MAX_PRESSURE (group, point + 1, pci);
2047 if (MODEL_MAX_PRESSURE (group, point, pci) > next_max_pressure)
2049 MODEL_MAX_PRESSURE (group, point, pci) = next_max_pressure;
2050 if (group->limits[pci].point == point)
2051 group->limits[pci].point = -1;
2056 /* Record that scheduling a later instruction has changed the pressure
2057 at point POINT of the model schedule by DELTA (which might be 0).
2058 Update GROUP accordingly. Return nonzero if these changes might
2059 trigger changes to previous points as well. */
2061 static int
2062 model_update_pressure (struct model_pressure_group *group,
2063 int point, int pci, int delta)
2065 int ref_pressure, max_pressure, next_max_pressure;
2067 /* If POINT hasn't yet been scheduled, update its pressure. */
2068 ref_pressure = MODEL_REF_PRESSURE (group, point, pci);
2069 if (ref_pressure >= 0 && delta != 0)
2071 ref_pressure += delta;
2072 MODEL_REF_PRESSURE (group, point, pci) = ref_pressure;
2074 /* Check whether the maximum pressure in the overall schedule
2075 has increased. (This means that the MODEL_MAX_PRESSURE of
2076 every point <= POINT will need to increase too; see below.) */
2077 if (group->limits[pci].pressure < ref_pressure)
2078 group->limits[pci].pressure = ref_pressure;
2080 /* If we are at maximum pressure, and the maximum pressure
2081 point was previously unknown or later than POINT,
2082 bring it forward. */
2083 if (group->limits[pci].pressure == ref_pressure
2084 && !IN_RANGE (group->limits[pci].point, 0, point))
2085 group->limits[pci].point = point;
2087 /* If POINT used to be the point of maximum pressure, but isn't
2088 any longer, we need to recalculate it using a forward walk. */
2089 if (group->limits[pci].pressure > ref_pressure
2090 && group->limits[pci].point == point)
2091 group->limits[pci].point = -1;
2094 /* Update the maximum pressure at POINT. Changes here might also
2095 affect the maximum pressure at POINT - 1. */
2096 next_max_pressure = MODEL_MAX_PRESSURE (group, point + 1, pci);
2097 max_pressure = MAX (ref_pressure, next_max_pressure);
2098 if (MODEL_MAX_PRESSURE (group, point, pci) != max_pressure)
2100 MODEL_MAX_PRESSURE (group, point, pci) = max_pressure;
2101 return 1;
2103 return 0;
2106 /* INSN has just been scheduled. Update the model schedule accordingly. */
2108 static void
2109 model_recompute (rtx_insn *insn)
2111 struct {
2112 int last_use;
2113 int regno;
2114 } uses[FIRST_PSEUDO_REGISTER + MAX_RECOG_OPERANDS];
2115 struct reg_use_data *use;
2116 struct reg_pressure_data *reg_pressure;
2117 int delta[N_REG_CLASSES];
2118 int pci, point, mix, new_last, cl, ref_pressure, queue;
2119 unsigned int i, num_uses, num_pending_births;
2120 bool print_p;
2122 /* The destinations of INSN were previously live from POINT onwards, but are
2123 now live from model_curr_point onwards. Set up DELTA accordingly. */
2124 point = model_index (insn);
2125 reg_pressure = INSN_REG_PRESSURE (insn);
2126 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2128 cl = ira_pressure_classes[pci];
2129 delta[cl] = reg_pressure[pci].set_increase;
2132 /* Record which registers previously died at POINT, but which now die
2133 before POINT. Adjust DELTA so that it represents the effect of
2134 this change after POINT - 1. Set NUM_PENDING_BIRTHS to the number of
2135 registers that will be born in the range [model_curr_point, POINT). */
2136 num_uses = 0;
2137 num_pending_births = 0;
2138 for (use = INSN_REG_USE_LIST (insn); use != NULL; use = use->next_insn_use)
2140 new_last = model_last_use_except (use);
2141 if (new_last < point)
2143 gcc_assert (num_uses < ARRAY_SIZE (uses));
2144 uses[num_uses].last_use = new_last;
2145 uses[num_uses].regno = use->regno;
2146 /* This register is no longer live after POINT - 1. */
2147 mark_regno_birth_or_death (NULL, delta, use->regno, false);
2148 num_uses++;
2149 if (new_last >= 0)
2150 num_pending_births++;
2154 /* Update the MODEL_REF_PRESSURE and MODEL_MAX_PRESSURE for POINT.
2155 Also set each group pressure limit for POINT. */
2156 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2158 cl = ira_pressure_classes[pci];
2159 model_start_update_pressure (&model_before_pressure,
2160 point, pci, delta[cl]);
2163 /* Walk the model schedule backwards, starting immediately before POINT. */
2164 print_p = false;
2165 if (point != model_curr_point)
2168 point--;
2169 insn = MODEL_INSN (point);
2170 queue = QUEUE_INDEX (insn);
2172 if (queue != QUEUE_SCHEDULED)
2174 /* DELTA describes the effect of the move on the register pressure
2175 after POINT. Make it describe the effect on the pressure
2176 before POINT. */
2177 i = 0;
2178 while (i < num_uses)
2180 if (uses[i].last_use == point)
2182 /* This register is now live again. */
2183 mark_regno_birth_or_death (NULL, delta,
2184 uses[i].regno, true);
2186 /* Remove this use from the array. */
2187 uses[i] = uses[num_uses - 1];
2188 num_uses--;
2189 num_pending_births--;
2191 else
2192 i++;
2195 if (sched_verbose >= 5)
2197 if (!print_p)
2199 fprintf (sched_dump, MODEL_BAR);
2200 fprintf (sched_dump, ";;\t\t| New pressure for model"
2201 " schedule\n");
2202 fprintf (sched_dump, MODEL_BAR);
2203 print_p = true;
2206 fprintf (sched_dump, ";;\t\t| %3d %4d %-30s ",
2207 point, INSN_UID (insn),
2208 str_pattern_slim (PATTERN (insn)));
2209 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2211 cl = ira_pressure_classes[pci];
2212 ref_pressure = MODEL_REF_PRESSURE (&model_before_pressure,
2213 point, pci);
2214 fprintf (sched_dump, " %s:[%d->%d]",
2215 reg_class_names[ira_pressure_classes[pci]],
2216 ref_pressure, ref_pressure + delta[cl]);
2218 fprintf (sched_dump, "\n");
2222 /* Adjust the pressure at POINT. Set MIX to nonzero if POINT - 1
2223 might have changed as well. */
2224 mix = num_pending_births;
2225 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2227 cl = ira_pressure_classes[pci];
2228 mix |= delta[cl];
2229 mix |= model_update_pressure (&model_before_pressure,
2230 point, pci, delta[cl]);
2233 while (mix && point > model_curr_point);
2235 if (print_p)
2236 fprintf (sched_dump, MODEL_BAR);
2239 /* After DEP, which was cancelled, has been resolved for insn NEXT,
2240 check whether the insn's pattern needs restoring. */
2241 static bool
2242 must_restore_pattern_p (rtx_insn *next, dep_t dep)
2244 if (QUEUE_INDEX (next) == QUEUE_SCHEDULED)
2245 return false;
2247 if (DEP_TYPE (dep) == REG_DEP_CONTROL)
2249 gcc_assert (ORIG_PAT (next) != NULL_RTX);
2250 gcc_assert (next == DEP_CON (dep));
2252 else
2254 struct dep_replacement *desc = DEP_REPLACE (dep);
2255 if (desc->insn != next)
2257 gcc_assert (*desc->loc == desc->orig);
2258 return false;
2261 return true;
2264 /* model_spill_cost (CL, P, P') returns the cost of increasing the
2265 pressure on CL from P to P'. We use this to calculate a "base ECC",
2266 baseECC (CL, X), for each pressure class CL and each instruction X.
2267 Supposing X changes the pressure on CL from P to P', and that the
2268 maximum pressure on CL in the current model schedule is MP', then:
2270 * if X occurs before or at the next point of maximum pressure in
2271 the model schedule and P' > MP', then:
2273 baseECC (CL, X) = model_spill_cost (CL, MP, P')
2275 The idea is that the pressure after scheduling a fixed set of
2276 instructions -- in this case, the set up to and including the
2277 next maximum pressure point -- is going to be the same regardless
2278 of the order; we simply want to keep the intermediate pressure
2279 under control. Thus X has a cost of zero unless scheduling it
2280 now would exceed MP'.
2282 If all increases in the set are by the same amount, no zero-cost
2283 instruction will ever cause the pressure to exceed MP'. However,
2284 if X is instead moved past an instruction X' with pressure in the
2285 range (MP' - (P' - P), MP'), the pressure at X' will increase
2286 beyond MP'. Since baseECC is very much a heuristic anyway,
2287 it doesn't seem worth the overhead of tracking cases like these.
2289 The cost of exceeding MP' is always based on the original maximum
2290 pressure MP. This is so that going 2 registers over the original
2291 limit has the same cost regardless of whether it comes from two
2292 separate +1 deltas or from a single +2 delta.
2294 * if X occurs after the next point of maximum pressure in the model
2295 schedule and P' > P, then:
2297 baseECC (CL, X) = model_spill_cost (CL, MP, MP' + (P' - P))
2299 That is, if we move X forward across a point of maximum pressure,
2300 and if X increases the pressure by P' - P, then we conservatively
2301 assume that scheduling X next would increase the maximum pressure
2302 by P' - P. Again, the cost of doing this is based on the original
2303 maximum pressure MP, for the same reason as above.
2305 * if P' < P, P > MP, and X occurs at or after the next point of
2306 maximum pressure, then:
2308 baseECC (CL, X) = -model_spill_cost (CL, MAX (MP, P'), P)
2310 That is, if we have already exceeded the original maximum pressure MP,
2311 and if X might reduce the maximum pressure again -- or at least push
2312 it further back, and thus allow more scheduling freedom -- it is given
2313 a negative cost to reflect the improvement.
2315 * otherwise,
2317 baseECC (CL, X) = 0
2319 In this case, X is not expected to affect the maximum pressure MP',
2320 so it has zero cost.
2322 We then create a combined value baseECC (X) that is the sum of
2323 baseECC (CL, X) for each pressure class CL.
2325 baseECC (X) could itself be used as the ECC value described above.
2326 However, this is often too conservative, in the sense that it
2327 tends to make high-priority instructions that increase pressure
2328 wait too long in cases where introducing a spill would be better.
2329 For this reason the final ECC is a priority-adjusted form of
2330 baseECC (X). Specifically, we calculate:
2332 P (X) = INSN_PRIORITY (X) - insn_delay (X) - baseECC (X)
2333 baseP = MAX { P (X) | baseECC (X) <= 0 }
2335 Then:
2337 ECC (X) = MAX (MIN (baseP - P (X), baseECC (X)), 0)
2339 Thus an instruction's effect on pressure is ignored if it has a high
2340 enough priority relative to the ones that don't increase pressure.
2341 Negative values of baseECC (X) do not increase the priority of X
2342 itself, but they do make it harder for other instructions to
2343 increase the pressure further.
2345 This pressure cost is deliberately timid. The intention has been
2346 to choose a heuristic that rarely interferes with the normal list
2347 scheduler in cases where that scheduler would produce good code.
2348 We simply want to curb some of its worst excesses. */
2350 /* Return the cost of increasing the pressure in class CL from FROM to TO.
2352 Here we use the very simplistic cost model that every register above
2353 sched_class_regs_num[CL] has a spill cost of 1. We could use other
2354 measures instead, such as one based on MEMORY_MOVE_COST. However:
2356 (1) In order for an instruction to be scheduled, the higher cost
2357 would need to be justified in a single saving of that many stalls.
2358 This is overly pessimistic, because the benefit of spilling is
2359 often to avoid a sequence of several short stalls rather than
2360 a single long one.
2362 (2) The cost is still arbitrary. Because we are not allocating
2363 registers during scheduling, we have no way of knowing for
2364 sure how many memory accesses will be required by each spill,
2365 where the spills will be placed within the block, or even
2366 which block(s) will contain the spills.
2368 So a higher cost than 1 is often too conservative in practice,
2369 forcing blocks to contain unnecessary stalls instead of spill code.
2370 The simple cost below seems to be the best compromise. It reduces
2371 the interference with the normal list scheduler, which helps make
2372 it more suitable for a default-on option. */
2374 static int
2375 model_spill_cost (int cl, int from, int to)
2377 from = MAX (from, sched_class_regs_num[cl]);
2378 return MAX (to, from) - from;
2381 /* Return baseECC (ira_pressure_classes[PCI], POINT), given that
2382 P = curr_reg_pressure[ira_pressure_classes[PCI]] and that
2383 P' = P + DELTA. */
2385 static int
2386 model_excess_group_cost (struct model_pressure_group *group,
2387 int point, int pci, int delta)
2389 int pressure, cl;
2391 cl = ira_pressure_classes[pci];
2392 if (delta < 0 && point >= group->limits[pci].point)
2394 pressure = MAX (group->limits[pci].orig_pressure,
2395 curr_reg_pressure[cl] + delta);
2396 return -model_spill_cost (cl, pressure, curr_reg_pressure[cl]);
2399 if (delta > 0)
2401 if (point > group->limits[pci].point)
2402 pressure = group->limits[pci].pressure + delta;
2403 else
2404 pressure = curr_reg_pressure[cl] + delta;
2406 if (pressure > group->limits[pci].pressure)
2407 return model_spill_cost (cl, group->limits[pci].orig_pressure,
2408 pressure);
2411 return 0;
2414 /* Return baseECC (MODEL_INSN (INSN)). Dump the costs to sched_dump
2415 if PRINT_P. */
2417 static int
2418 model_excess_cost (rtx_insn *insn, bool print_p)
2420 int point, pci, cl, cost, this_cost, delta;
2421 struct reg_pressure_data *insn_reg_pressure;
2422 int insn_death[N_REG_CLASSES];
2424 calculate_reg_deaths (insn, insn_death);
2425 point = model_index (insn);
2426 insn_reg_pressure = INSN_REG_PRESSURE (insn);
2427 cost = 0;
2429 if (print_p)
2430 fprintf (sched_dump, ";;\t\t| %3d %4d | %4d %+3d |", point,
2431 INSN_UID (insn), INSN_PRIORITY (insn), insn_delay (insn));
2433 /* Sum up the individual costs for each register class. */
2434 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2436 cl = ira_pressure_classes[pci];
2437 delta = insn_reg_pressure[pci].set_increase - insn_death[cl];
2438 this_cost = model_excess_group_cost (&model_before_pressure,
2439 point, pci, delta);
2440 cost += this_cost;
2441 if (print_p)
2442 fprintf (sched_dump, " %s:[%d base cost %d]",
2443 reg_class_names[cl], delta, this_cost);
2446 if (print_p)
2447 fprintf (sched_dump, "\n");
2449 return cost;
2452 /* Dump the next points of maximum pressure for GROUP. */
2454 static void
2455 model_dump_pressure_points (struct model_pressure_group *group)
2457 int pci, cl;
2459 fprintf (sched_dump, ";;\t\t| pressure points");
2460 for (pci = 0; pci < ira_pressure_classes_num; pci++)
2462 cl = ira_pressure_classes[pci];
2463 fprintf (sched_dump, " %s:[%d->%d at ", reg_class_names[cl],
2464 curr_reg_pressure[cl], group->limits[pci].pressure);
2465 if (group->limits[pci].point < model_num_insns)
2466 fprintf (sched_dump, "%d:%d]", group->limits[pci].point,
2467 INSN_UID (MODEL_INSN (group->limits[pci].point)));
2468 else
2469 fprintf (sched_dump, "end]");
2471 fprintf (sched_dump, "\n");
2474 /* Set INSN_REG_PRESSURE_EXCESS_COST_CHANGE for INSNS[0...COUNT-1]. */
2476 static void
2477 model_set_excess_costs (rtx_insn **insns, int count)
2479 int i, cost, priority_base, priority;
2480 bool print_p;
2482 /* Record the baseECC value for each instruction in the model schedule,
2483 except that negative costs are converted to zero ones now rather than
2484 later. Do not assign a cost to debug instructions, since they must
2485 not change code-generation decisions. Experiments suggest we also
2486 get better results by not assigning a cost to instructions from
2487 a different block.
2489 Set PRIORITY_BASE to baseP in the block comment above. This is the
2490 maximum priority of the "cheap" instructions, which should always
2491 include the next model instruction. */
2492 priority_base = 0;
2493 print_p = false;
2494 for (i = 0; i < count; i++)
2495 if (INSN_MODEL_INDEX (insns[i]))
2497 if (sched_verbose >= 6 && !print_p)
2499 fprintf (sched_dump, MODEL_BAR);
2500 fprintf (sched_dump, ";;\t\t| Pressure costs for ready queue\n");
2501 model_dump_pressure_points (&model_before_pressure);
2502 fprintf (sched_dump, MODEL_BAR);
2503 print_p = true;
2505 cost = model_excess_cost (insns[i], print_p);
2506 if (cost <= 0)
2508 priority = INSN_PRIORITY (insns[i]) - insn_delay (insns[i]) - cost;
2509 priority_base = MAX (priority_base, priority);
2510 cost = 0;
2512 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insns[i]) = cost;
2514 if (print_p)
2515 fprintf (sched_dump, MODEL_BAR);
2517 /* Use MAX (baseECC, 0) and baseP to calculcate ECC for each
2518 instruction. */
2519 for (i = 0; i < count; i++)
2521 cost = INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insns[i]);
2522 priority = INSN_PRIORITY (insns[i]) - insn_delay (insns[i]);
2523 if (cost > 0 && priority > priority_base)
2525 cost += priority_base - priority;
2526 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (insns[i]) = MAX (cost, 0);
2532 /* Enum of rank_for_schedule heuristic decisions. */
2533 enum rfs_decision {
2534 RFS_LIVE_RANGE_SHRINK1, RFS_LIVE_RANGE_SHRINK2,
2535 RFS_SCHED_GROUP, RFS_PRESSURE_DELAY, RFS_PRESSURE_TICK,
2536 RFS_FEEDS_BACKTRACK_INSN, RFS_PRIORITY, RFS_SPECULATION,
2537 RFS_SCHED_RANK, RFS_LAST_INSN, RFS_PRESSURE_INDEX,
2538 RFS_DEP_COUNT, RFS_TIE, RFS_FUSION, RFS_N };
2540 /* Corresponding strings for print outs. */
2541 static const char *rfs_str[RFS_N] = {
2542 "RFS_LIVE_RANGE_SHRINK1", "RFS_LIVE_RANGE_SHRINK2",
2543 "RFS_SCHED_GROUP", "RFS_PRESSURE_DELAY", "RFS_PRESSURE_TICK",
2544 "RFS_FEEDS_BACKTRACK_INSN", "RFS_PRIORITY", "RFS_SPECULATION",
2545 "RFS_SCHED_RANK", "RFS_LAST_INSN", "RFS_PRESSURE_INDEX",
2546 "RFS_DEP_COUNT", "RFS_TIE", "RFS_FUSION" };
2548 /* Statistical breakdown of rank_for_schedule decisions. */
2549 struct rank_for_schedule_stats_t { unsigned stats[RFS_N]; };
2550 static rank_for_schedule_stats_t rank_for_schedule_stats;
2552 /* Return the result of comparing insns TMP and TMP2 and update
2553 Rank_For_Schedule statistics. */
2554 static int
2555 rfs_result (enum rfs_decision decision, int result, rtx tmp, rtx tmp2)
2557 ++rank_for_schedule_stats.stats[decision];
2558 if (result < 0)
2559 INSN_LAST_RFS_WIN (tmp) = decision;
2560 else if (result > 0)
2561 INSN_LAST_RFS_WIN (tmp2) = decision;
2562 else
2563 gcc_unreachable ();
2564 return result;
2567 /* Sorting predicate to move DEBUG_INSNs to the top of ready list, while
2568 keeping normal insns in original order. */
2570 static int
2571 rank_for_schedule_debug (const void *x, const void *y)
2573 rtx_insn *tmp = *(rtx_insn * const *) y;
2574 rtx_insn *tmp2 = *(rtx_insn * const *) x;
2576 /* Schedule debug insns as early as possible. */
2577 if (DEBUG_INSN_P (tmp) && !DEBUG_INSN_P (tmp2))
2578 return -1;
2579 else if (!DEBUG_INSN_P (tmp) && DEBUG_INSN_P (tmp2))
2580 return 1;
2581 else if (DEBUG_INSN_P (tmp) && DEBUG_INSN_P (tmp2))
2582 return INSN_LUID (tmp) - INSN_LUID (tmp2);
2583 else
2584 return INSN_RFS_DEBUG_ORIG_ORDER (tmp2) - INSN_RFS_DEBUG_ORIG_ORDER (tmp);
2587 /* Returns a positive value if x is preferred; returns a negative value if
2588 y is preferred. Should never return 0, since that will make the sort
2589 unstable. */
2591 static int
2592 rank_for_schedule (const void *x, const void *y)
2594 rtx_insn *tmp = *(rtx_insn * const *) y;
2595 rtx_insn *tmp2 = *(rtx_insn * const *) x;
2596 int tmp_class, tmp2_class;
2597 int val, priority_val, info_val, diff;
2599 if (live_range_shrinkage_p)
2601 /* Don't use SCHED_PRESSURE_MODEL -- it results in much worse
2602 code. */
2603 gcc_assert (sched_pressure == SCHED_PRESSURE_WEIGHTED);
2604 if ((INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp) < 0
2605 || INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp2) < 0)
2606 && (diff = (INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp)
2607 - INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp2))) != 0)
2608 return rfs_result (RFS_LIVE_RANGE_SHRINK1, diff, tmp, tmp2);
2609 /* Sort by INSN_LUID (original insn order), so that we make the
2610 sort stable. This minimizes instruction movement, thus
2611 minimizing sched's effect on debugging and cross-jumping. */
2612 return rfs_result (RFS_LIVE_RANGE_SHRINK2,
2613 INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2);
2616 /* The insn in a schedule group should be issued the first. */
2617 if (flag_sched_group_heuristic &&
2618 SCHED_GROUP_P (tmp) != SCHED_GROUP_P (tmp2))
2619 return rfs_result (RFS_SCHED_GROUP, SCHED_GROUP_P (tmp2) ? 1 : -1,
2620 tmp, tmp2);
2622 /* Make sure that priority of TMP and TMP2 are initialized. */
2623 gcc_assert (INSN_PRIORITY_KNOWN (tmp) && INSN_PRIORITY_KNOWN (tmp2));
2625 if (sched_fusion)
2627 /* The instruction that has the same fusion priority as the last
2628 instruction is the instruction we picked next. If that is not
2629 the case, we sort ready list firstly by fusion priority, then
2630 by priority, and at last by INSN_LUID. */
2631 int a = INSN_FUSION_PRIORITY (tmp);
2632 int b = INSN_FUSION_PRIORITY (tmp2);
2633 int last = -1;
2635 if (last_nondebug_scheduled_insn
2636 && !NOTE_P (last_nondebug_scheduled_insn)
2637 && BLOCK_FOR_INSN (tmp)
2638 == BLOCK_FOR_INSN (last_nondebug_scheduled_insn))
2639 last = INSN_FUSION_PRIORITY (last_nondebug_scheduled_insn);
2641 if (a != last && b != last)
2643 if (a == b)
2645 a = INSN_PRIORITY (tmp);
2646 b = INSN_PRIORITY (tmp2);
2648 if (a != b)
2649 return rfs_result (RFS_FUSION, b - a, tmp, tmp2);
2650 else
2651 return rfs_result (RFS_FUSION,
2652 INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2);
2654 else if (a == b)
2656 gcc_assert (last_nondebug_scheduled_insn
2657 && !NOTE_P (last_nondebug_scheduled_insn));
2658 last = INSN_PRIORITY (last_nondebug_scheduled_insn);
2660 a = abs (INSN_PRIORITY (tmp) - last);
2661 b = abs (INSN_PRIORITY (tmp2) - last);
2662 if (a != b)
2663 return rfs_result (RFS_FUSION, a - b, tmp, tmp2);
2664 else
2665 return rfs_result (RFS_FUSION,
2666 INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2);
2668 else if (a == last)
2669 return rfs_result (RFS_FUSION, -1, tmp, tmp2);
2670 else
2671 return rfs_result (RFS_FUSION, 1, tmp, tmp2);
2674 if (sched_pressure != SCHED_PRESSURE_NONE)
2676 /* Prefer insn whose scheduling results in the smallest register
2677 pressure excess. */
2678 if ((diff = (INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp)
2679 + insn_delay (tmp)
2680 - INSN_REG_PRESSURE_EXCESS_COST_CHANGE (tmp2)
2681 - insn_delay (tmp2))))
2682 return rfs_result (RFS_PRESSURE_DELAY, diff, tmp, tmp2);
2685 if (sched_pressure != SCHED_PRESSURE_NONE
2686 && (INSN_TICK (tmp2) > clock_var || INSN_TICK (tmp) > clock_var)
2687 && INSN_TICK (tmp2) != INSN_TICK (tmp))
2689 diff = INSN_TICK (tmp) - INSN_TICK (tmp2);
2690 return rfs_result (RFS_PRESSURE_TICK, diff, tmp, tmp2);
2693 /* If we are doing backtracking in this schedule, prefer insns that
2694 have forward dependencies with negative cost against an insn that
2695 was already scheduled. */
2696 if (current_sched_info->flags & DO_BACKTRACKING)
2698 priority_val = FEEDS_BACKTRACK_INSN (tmp2) - FEEDS_BACKTRACK_INSN (tmp);
2699 if (priority_val)
2700 return rfs_result (RFS_FEEDS_BACKTRACK_INSN, priority_val, tmp, tmp2);
2703 /* Prefer insn with higher priority. */
2704 priority_val = INSN_PRIORITY (tmp2) - INSN_PRIORITY (tmp);
2706 if (flag_sched_critical_path_heuristic && priority_val)
2707 return rfs_result (RFS_PRIORITY, priority_val, tmp, tmp2);
2709 if (PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) >= 0)
2711 int autopref = autopref_rank_for_schedule (tmp, tmp2);
2712 if (autopref != 0)
2713 return autopref;
2716 /* Prefer speculative insn with greater dependencies weakness. */
2717 if (flag_sched_spec_insn_heuristic && spec_info)
2719 ds_t ds1, ds2;
2720 dw_t dw1, dw2;
2721 int dw;
2723 ds1 = TODO_SPEC (tmp) & SPECULATIVE;
2724 if (ds1)
2725 dw1 = ds_weak (ds1);
2726 else
2727 dw1 = NO_DEP_WEAK;
2729 ds2 = TODO_SPEC (tmp2) & SPECULATIVE;
2730 if (ds2)
2731 dw2 = ds_weak (ds2);
2732 else
2733 dw2 = NO_DEP_WEAK;
2735 dw = dw2 - dw1;
2736 if (dw > (NO_DEP_WEAK / 8) || dw < -(NO_DEP_WEAK / 8))
2737 return rfs_result (RFS_SPECULATION, dw, tmp, tmp2);
2740 info_val = (*current_sched_info->rank) (tmp, tmp2);
2741 if (flag_sched_rank_heuristic && info_val)
2742 return rfs_result (RFS_SCHED_RANK, info_val, tmp, tmp2);
2744 /* Compare insns based on their relation to the last scheduled
2745 non-debug insn. */
2746 if (flag_sched_last_insn_heuristic && last_nondebug_scheduled_insn)
2748 dep_t dep1;
2749 dep_t dep2;
2750 rtx_insn *last = last_nondebug_scheduled_insn;
2752 /* Classify the instructions into three classes:
2753 1) Data dependent on last schedule insn.
2754 2) Anti/Output dependent on last scheduled insn.
2755 3) Independent of last scheduled insn, or has latency of one.
2756 Choose the insn from the highest numbered class if different. */
2757 dep1 = sd_find_dep_between (last, tmp, true);
2759 if (dep1 == NULL || dep_cost (dep1) == 1)
2760 tmp_class = 3;
2761 else if (/* Data dependence. */
2762 DEP_TYPE (dep1) == REG_DEP_TRUE)
2763 tmp_class = 1;
2764 else
2765 tmp_class = 2;
2767 dep2 = sd_find_dep_between (last, tmp2, true);
2769 if (dep2 == NULL || dep_cost (dep2) == 1)
2770 tmp2_class = 3;
2771 else if (/* Data dependence. */
2772 DEP_TYPE (dep2) == REG_DEP_TRUE)
2773 tmp2_class = 1;
2774 else
2775 tmp2_class = 2;
2777 if ((val = tmp2_class - tmp_class))
2778 return rfs_result (RFS_LAST_INSN, val, tmp, tmp2);
2781 /* Prefer instructions that occur earlier in the model schedule. */
2782 if (sched_pressure == SCHED_PRESSURE_MODEL
2783 && INSN_BB (tmp) == target_bb && INSN_BB (tmp2) == target_bb)
2785 diff = model_index (tmp) - model_index (tmp2);
2786 gcc_assert (diff != 0);
2787 return rfs_result (RFS_PRESSURE_INDEX, diff, tmp, tmp2);
2790 /* Prefer the insn which has more later insns that depend on it.
2791 This gives the scheduler more freedom when scheduling later
2792 instructions at the expense of added register pressure. */
2794 val = (dep_list_size (tmp2, SD_LIST_FORW)
2795 - dep_list_size (tmp, SD_LIST_FORW));
2797 if (flag_sched_dep_count_heuristic && val != 0)
2798 return rfs_result (RFS_DEP_COUNT, val, tmp, tmp2);
2800 /* If insns are equally good, sort by INSN_LUID (original insn order),
2801 so that we make the sort stable. This minimizes instruction movement,
2802 thus minimizing sched's effect on debugging and cross-jumping. */
2803 return rfs_result (RFS_TIE, INSN_LUID (tmp) - INSN_LUID (tmp2), tmp, tmp2);
2806 /* Resort the array A in which only element at index N may be out of order. */
2808 HAIFA_INLINE static void
2809 swap_sort (rtx_insn **a, int n)
2811 rtx_insn *insn = a[n - 1];
2812 int i = n - 2;
2814 while (i >= 0 && rank_for_schedule (a + i, &insn) >= 0)
2816 a[i + 1] = a[i];
2817 i -= 1;
2819 a[i + 1] = insn;
2822 /* Add INSN to the insn queue so that it can be executed at least
2823 N_CYCLES after the currently executing insn. Preserve insns
2824 chain for debugging purposes. REASON will be printed in debugging
2825 output. */
2827 HAIFA_INLINE static void
2828 queue_insn (rtx_insn *insn, int n_cycles, const char *reason)
2830 int next_q = NEXT_Q_AFTER (q_ptr, n_cycles);
2831 rtx_insn_list *link = alloc_INSN_LIST (insn, insn_queue[next_q]);
2832 int new_tick;
2834 gcc_assert (n_cycles <= max_insn_queue_index);
2835 gcc_assert (!DEBUG_INSN_P (insn));
2837 insn_queue[next_q] = link;
2838 q_size += 1;
2840 if (sched_verbose >= 2)
2842 fprintf (sched_dump, ";;\t\tReady-->Q: insn %s: ",
2843 (*current_sched_info->print_insn) (insn, 0));
2845 fprintf (sched_dump, "queued for %d cycles (%s).\n", n_cycles, reason);
2848 QUEUE_INDEX (insn) = next_q;
2850 if (current_sched_info->flags & DO_BACKTRACKING)
2852 new_tick = clock_var + n_cycles;
2853 if (INSN_TICK (insn) == INVALID_TICK || INSN_TICK (insn) < new_tick)
2854 INSN_TICK (insn) = new_tick;
2856 if (INSN_EXACT_TICK (insn) != INVALID_TICK
2857 && INSN_EXACT_TICK (insn) < clock_var + n_cycles)
2859 must_backtrack = true;
2860 if (sched_verbose >= 2)
2861 fprintf (sched_dump, ";;\t\tcausing a backtrack.\n");
2866 /* Remove INSN from queue. */
2867 static void
2868 queue_remove (rtx_insn *insn)
2870 gcc_assert (QUEUE_INDEX (insn) >= 0);
2871 remove_free_INSN_LIST_elem (insn, &insn_queue[QUEUE_INDEX (insn)]);
2872 q_size--;
2873 QUEUE_INDEX (insn) = QUEUE_NOWHERE;
2876 /* Return a pointer to the bottom of the ready list, i.e. the insn
2877 with the lowest priority. */
2879 rtx_insn **
2880 ready_lastpos (struct ready_list *ready)
2882 gcc_assert (ready->n_ready >= 1);
2883 return ready->vec + ready->first - ready->n_ready + 1;
2886 /* Add an element INSN to the ready list so that it ends up with the
2887 lowest/highest priority depending on FIRST_P. */
2889 HAIFA_INLINE static void
2890 ready_add (struct ready_list *ready, rtx_insn *insn, bool first_p)
2892 if (!first_p)
2894 if (ready->first == ready->n_ready)
2896 memmove (ready->vec + ready->veclen - ready->n_ready,
2897 ready_lastpos (ready),
2898 ready->n_ready * sizeof (rtx));
2899 ready->first = ready->veclen - 1;
2901 ready->vec[ready->first - ready->n_ready] = insn;
2903 else
2905 if (ready->first == ready->veclen - 1)
2907 if (ready->n_ready)
2908 /* ready_lastpos() fails when called with (ready->n_ready == 0). */
2909 memmove (ready->vec + ready->veclen - ready->n_ready - 1,
2910 ready_lastpos (ready),
2911 ready->n_ready * sizeof (rtx));
2912 ready->first = ready->veclen - 2;
2914 ready->vec[++(ready->first)] = insn;
2917 ready->n_ready++;
2918 if (DEBUG_INSN_P (insn))
2919 ready->n_debug++;
2921 gcc_assert (QUEUE_INDEX (insn) != QUEUE_READY);
2922 QUEUE_INDEX (insn) = QUEUE_READY;
2924 if (INSN_EXACT_TICK (insn) != INVALID_TICK
2925 && INSN_EXACT_TICK (insn) < clock_var)
2927 must_backtrack = true;
2931 /* Remove the element with the highest priority from the ready list and
2932 return it. */
2934 HAIFA_INLINE static rtx_insn *
2935 ready_remove_first (struct ready_list *ready)
2937 rtx_insn *t;
2939 gcc_assert (ready->n_ready);
2940 t = ready->vec[ready->first--];
2941 ready->n_ready--;
2942 if (DEBUG_INSN_P (t))
2943 ready->n_debug--;
2944 /* If the queue becomes empty, reset it. */
2945 if (ready->n_ready == 0)
2946 ready->first = ready->veclen - 1;
2948 gcc_assert (QUEUE_INDEX (t) == QUEUE_READY);
2949 QUEUE_INDEX (t) = QUEUE_NOWHERE;
2951 return t;
2954 /* The following code implements multi-pass scheduling for the first
2955 cycle. In other words, we will try to choose ready insn which
2956 permits to start maximum number of insns on the same cycle. */
2958 /* Return a pointer to the element INDEX from the ready. INDEX for
2959 insn with the highest priority is 0, and the lowest priority has
2960 N_READY - 1. */
2962 rtx_insn *
2963 ready_element (struct ready_list *ready, int index)
2965 gcc_assert (ready->n_ready && index < ready->n_ready);
2967 return ready->vec[ready->first - index];
2970 /* Remove the element INDEX from the ready list and return it. INDEX
2971 for insn with the highest priority is 0, and the lowest priority
2972 has N_READY - 1. */
2974 HAIFA_INLINE static rtx_insn *
2975 ready_remove (struct ready_list *ready, int index)
2977 rtx_insn *t;
2978 int i;
2980 if (index == 0)
2981 return ready_remove_first (ready);
2982 gcc_assert (ready->n_ready && index < ready->n_ready);
2983 t = ready->vec[ready->first - index];
2984 ready->n_ready--;
2985 if (DEBUG_INSN_P (t))
2986 ready->n_debug--;
2987 for (i = index; i < ready->n_ready; i++)
2988 ready->vec[ready->first - i] = ready->vec[ready->first - i - 1];
2989 QUEUE_INDEX (t) = QUEUE_NOWHERE;
2990 return t;
2993 /* Remove INSN from the ready list. */
2994 static void
2995 ready_remove_insn (rtx_insn *insn)
2997 int i;
2999 for (i = 0; i < readyp->n_ready; i++)
3000 if (ready_element (readyp, i) == insn)
3002 ready_remove (readyp, i);
3003 return;
3005 gcc_unreachable ();
3008 /* Calculate difference of two statistics set WAS and NOW.
3009 Result returned in WAS. */
3010 static void
3011 rank_for_schedule_stats_diff (rank_for_schedule_stats_t *was,
3012 const rank_for_schedule_stats_t *now)
3014 for (int i = 0; i < RFS_N; ++i)
3015 was->stats[i] = now->stats[i] - was->stats[i];
3018 /* Print rank_for_schedule statistics. */
3019 static void
3020 print_rank_for_schedule_stats (const char *prefix,
3021 const rank_for_schedule_stats_t *stats,
3022 struct ready_list *ready)
3024 for (int i = 0; i < RFS_N; ++i)
3025 if (stats->stats[i])
3027 fprintf (sched_dump, "%s%20s: %u", prefix, rfs_str[i], stats->stats[i]);
3029 if (ready != NULL)
3030 /* Print out insns that won due to RFS_<I>. */
3032 rtx_insn **p = ready_lastpos (ready);
3034 fprintf (sched_dump, ":");
3035 /* Start with 1 since least-priority insn didn't have any wins. */
3036 for (int j = 1; j < ready->n_ready; ++j)
3037 if (INSN_LAST_RFS_WIN (p[j]) == i)
3038 fprintf (sched_dump, " %s",
3039 (*current_sched_info->print_insn) (p[j], 0));
3041 fprintf (sched_dump, "\n");
3045 /* Separate DEBUG_INSNS from normal insns. DEBUG_INSNs go to the end
3046 of array. */
3047 static void
3048 ready_sort_debug (struct ready_list *ready)
3050 int i;
3051 rtx_insn **first = ready_lastpos (ready);
3053 for (i = 0; i < ready->n_ready; ++i)
3054 if (!DEBUG_INSN_P (first[i]))
3055 INSN_RFS_DEBUG_ORIG_ORDER (first[i]) = i;
3057 qsort (first, ready->n_ready, sizeof (rtx), rank_for_schedule_debug);
3060 /* Sort non-debug insns in the ready list READY by ascending priority.
3061 Assumes that all debug insns are separated from the real insns. */
3062 static void
3063 ready_sort_real (struct ready_list *ready)
3065 int i;
3066 rtx_insn **first = ready_lastpos (ready);
3067 int n_ready_real = ready->n_ready - ready->n_debug;
3069 if (sched_pressure == SCHED_PRESSURE_WEIGHTED)
3070 for (i = 0; i < n_ready_real; ++i)
3071 setup_insn_reg_pressure_info (first[i]);
3072 else if (sched_pressure == SCHED_PRESSURE_MODEL
3073 && model_curr_point < model_num_insns)
3074 model_set_excess_costs (first, n_ready_real);
3076 rank_for_schedule_stats_t stats1;
3077 if (sched_verbose >= 4)
3078 stats1 = rank_for_schedule_stats;
3080 if (n_ready_real == 2)
3081 swap_sort (first, n_ready_real);
3082 else if (n_ready_real > 2)
3083 qsort (first, n_ready_real, sizeof (rtx), rank_for_schedule);
3085 if (sched_verbose >= 4)
3087 rank_for_schedule_stats_diff (&stats1, &rank_for_schedule_stats);
3088 print_rank_for_schedule_stats (";;\t\t", &stats1, ready);
3092 /* Sort the ready list READY by ascending priority. */
3093 static void
3094 ready_sort (struct ready_list *ready)
3096 if (ready->n_debug > 0)
3097 ready_sort_debug (ready);
3098 else
3099 ready_sort_real (ready);
3102 /* PREV is an insn that is ready to execute. Adjust its priority if that
3103 will help shorten or lengthen register lifetimes as appropriate. Also
3104 provide a hook for the target to tweak itself. */
3106 HAIFA_INLINE static void
3107 adjust_priority (rtx_insn *prev)
3109 /* ??? There used to be code here to try and estimate how an insn
3110 affected register lifetimes, but it did it by looking at REG_DEAD
3111 notes, which we removed in schedule_region. Nor did it try to
3112 take into account register pressure or anything useful like that.
3114 Revisit when we have a machine model to work with and not before. */
3116 if (targetm.sched.adjust_priority)
3117 INSN_PRIORITY (prev) =
3118 targetm.sched.adjust_priority (prev, INSN_PRIORITY (prev));
3121 /* Advance DFA state STATE on one cycle. */
3122 void
3123 advance_state (state_t state)
3125 if (targetm.sched.dfa_pre_advance_cycle)
3126 targetm.sched.dfa_pre_advance_cycle ();
3128 if (targetm.sched.dfa_pre_cycle_insn)
3129 state_transition (state,
3130 targetm.sched.dfa_pre_cycle_insn ());
3132 state_transition (state, NULL);
3134 if (targetm.sched.dfa_post_cycle_insn)
3135 state_transition (state,
3136 targetm.sched.dfa_post_cycle_insn ());
3138 if (targetm.sched.dfa_post_advance_cycle)
3139 targetm.sched.dfa_post_advance_cycle ();
3142 /* Advance time on one cycle. */
3143 HAIFA_INLINE static void
3144 advance_one_cycle (void)
3146 advance_state (curr_state);
3147 if (sched_verbose >= 4)
3148 fprintf (sched_dump, ";;\tAdvance the current state.\n");
3151 /* Update register pressure after scheduling INSN. */
3152 static void
3153 update_register_pressure (rtx_insn *insn)
3155 struct reg_use_data *use;
3156 struct reg_set_data *set;
3158 gcc_checking_assert (!DEBUG_INSN_P (insn));
3160 for (use = INSN_REG_USE_LIST (insn); use != NULL; use = use->next_insn_use)
3161 if (dying_use_p (use))
3162 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure,
3163 use->regno, false);
3164 for (set = INSN_REG_SET_LIST (insn); set != NULL; set = set->next_insn_set)
3165 mark_regno_birth_or_death (curr_reg_live, curr_reg_pressure,
3166 set->regno, true);
3169 /* Set up or update (if UPDATE_P) max register pressure (see its
3170 meaning in sched-int.h::_haifa_insn_data) for all current BB insns
3171 after insn AFTER. */
3172 static void
3173 setup_insn_max_reg_pressure (rtx_insn *after, bool update_p)
3175 int i, p;
3176 bool eq_p;
3177 rtx_insn *insn;
3178 static int max_reg_pressure[N_REG_CLASSES];
3180 save_reg_pressure ();
3181 for (i = 0; i < ira_pressure_classes_num; i++)
3182 max_reg_pressure[ira_pressure_classes[i]]
3183 = curr_reg_pressure[ira_pressure_classes[i]];
3184 for (insn = NEXT_INSN (after);
3185 insn != NULL_RTX && ! BARRIER_P (insn)
3186 && BLOCK_FOR_INSN (insn) == BLOCK_FOR_INSN (after);
3187 insn = NEXT_INSN (insn))
3188 if (NONDEBUG_INSN_P (insn))
3190 eq_p = true;
3191 for (i = 0; i < ira_pressure_classes_num; i++)
3193 p = max_reg_pressure[ira_pressure_classes[i]];
3194 if (INSN_MAX_REG_PRESSURE (insn)[i] != p)
3196 eq_p = false;
3197 INSN_MAX_REG_PRESSURE (insn)[i]
3198 = max_reg_pressure[ira_pressure_classes[i]];
3201 if (update_p && eq_p)
3202 break;
3203 update_register_pressure (insn);
3204 for (i = 0; i < ira_pressure_classes_num; i++)
3205 if (max_reg_pressure[ira_pressure_classes[i]]
3206 < curr_reg_pressure[ira_pressure_classes[i]])
3207 max_reg_pressure[ira_pressure_classes[i]]
3208 = curr_reg_pressure[ira_pressure_classes[i]];
3210 restore_reg_pressure ();
3213 /* Update the current register pressure after scheduling INSN. Update
3214 also max register pressure for unscheduled insns of the current
3215 BB. */
3216 static void
3217 update_reg_and_insn_max_reg_pressure (rtx_insn *insn)
3219 int i;
3220 int before[N_REG_CLASSES];
3222 for (i = 0; i < ira_pressure_classes_num; i++)
3223 before[i] = curr_reg_pressure[ira_pressure_classes[i]];
3224 update_register_pressure (insn);
3225 for (i = 0; i < ira_pressure_classes_num; i++)
3226 if (curr_reg_pressure[ira_pressure_classes[i]] != before[i])
3227 break;
3228 if (i < ira_pressure_classes_num)
3229 setup_insn_max_reg_pressure (insn, true);
3232 /* Set up register pressure at the beginning of basic block BB whose
3233 insns starting after insn AFTER. Set up also max register pressure
3234 for all insns of the basic block. */
3235 void
3236 sched_setup_bb_reg_pressure_info (basic_block bb, rtx_insn *after)
3238 gcc_assert (sched_pressure == SCHED_PRESSURE_WEIGHTED);
3239 initiate_bb_reg_pressure_info (bb);
3240 setup_insn_max_reg_pressure (after, false);
3243 /* If doing predication while scheduling, verify whether INSN, which
3244 has just been scheduled, clobbers the conditions of any
3245 instructions that must be predicated in order to break their
3246 dependencies. If so, remove them from the queues so that they will
3247 only be scheduled once their control dependency is resolved. */
3249 static void
3250 check_clobbered_conditions (rtx_insn *insn)
3252 HARD_REG_SET t;
3253 int i;
3255 if ((current_sched_info->flags & DO_PREDICATION) == 0)
3256 return;
3258 find_all_hard_reg_sets (insn, &t, true);
3260 restart:
3261 for (i = 0; i < ready.n_ready; i++)
3263 rtx_insn *x = ready_element (&ready, i);
3264 if (TODO_SPEC (x) == DEP_CONTROL && cond_clobbered_p (x, t))
3266 ready_remove_insn (x);
3267 goto restart;
3270 for (i = 0; i <= max_insn_queue_index; i++)
3272 rtx_insn_list *link;
3273 int q = NEXT_Q_AFTER (q_ptr, i);
3275 restart_queue:
3276 for (link = insn_queue[q]; link; link = link->next ())
3278 rtx_insn *x = link->insn ();
3279 if (TODO_SPEC (x) == DEP_CONTROL && cond_clobbered_p (x, t))
3281 queue_remove (x);
3282 goto restart_queue;
3288 /* Return (in order):
3290 - positive if INSN adversely affects the pressure on one
3291 register class
3293 - negative if INSN reduces the pressure on one register class
3295 - 0 if INSN doesn't affect the pressure on any register class. */
3297 static int
3298 model_classify_pressure (struct model_insn_info *insn)
3300 struct reg_pressure_data *reg_pressure;
3301 int death[N_REG_CLASSES];
3302 int pci, cl, sum;
3304 calculate_reg_deaths (insn->insn, death);
3305 reg_pressure = INSN_REG_PRESSURE (insn->insn);
3306 sum = 0;
3307 for (pci = 0; pci < ira_pressure_classes_num; pci++)
3309 cl = ira_pressure_classes[pci];
3310 if (death[cl] < reg_pressure[pci].set_increase)
3311 return 1;
3312 sum += reg_pressure[pci].set_increase - death[cl];
3314 return sum;
3317 /* Return true if INSN1 should come before INSN2 in the model schedule. */
3319 static int
3320 model_order_p (struct model_insn_info *insn1, struct model_insn_info *insn2)
3322 unsigned int height1, height2;
3323 unsigned int priority1, priority2;
3325 /* Prefer instructions with a higher model priority. */
3326 if (insn1->model_priority != insn2->model_priority)
3327 return insn1->model_priority > insn2->model_priority;
3329 /* Combine the length of the longest path of satisfied true dependencies
3330 that leads to each instruction (depth) with the length of the longest
3331 path of any dependencies that leads from the instruction (alap).
3332 Prefer instructions with the greatest combined length. If the combined
3333 lengths are equal, prefer instructions with the greatest depth.
3335 The idea is that, if we have a set S of "equal" instructions that each
3336 have ALAP value X, and we pick one such instruction I, any true-dependent
3337 successors of I that have ALAP value X - 1 should be preferred over S.
3338 This encourages the schedule to be "narrow" rather than "wide".
3339 However, if I is a low-priority instruction that we decided to
3340 schedule because of its model_classify_pressure, and if there
3341 is a set of higher-priority instructions T, the aforementioned
3342 successors of I should not have the edge over T. */
3343 height1 = insn1->depth + insn1->alap;
3344 height2 = insn2->depth + insn2->alap;
3345 if (height1 != height2)
3346 return height1 > height2;
3347 if (insn1->depth != insn2->depth)
3348 return insn1->depth > insn2->depth;
3350 /* We have no real preference between INSN1 an INSN2 as far as attempts
3351 to reduce pressure go. Prefer instructions with higher priorities. */
3352 priority1 = INSN_PRIORITY (insn1->insn);
3353 priority2 = INSN_PRIORITY (insn2->insn);
3354 if (priority1 != priority2)
3355 return priority1 > priority2;
3357 /* Use the original rtl sequence as a tie-breaker. */
3358 return insn1 < insn2;
3361 /* Add INSN to the model worklist immediately after PREV. Add it to the
3362 beginning of the list if PREV is null. */
3364 static void
3365 model_add_to_worklist_at (struct model_insn_info *insn,
3366 struct model_insn_info *prev)
3368 gcc_assert (QUEUE_INDEX (insn->insn) == QUEUE_NOWHERE);
3369 QUEUE_INDEX (insn->insn) = QUEUE_READY;
3371 insn->prev = prev;
3372 if (prev)
3374 insn->next = prev->next;
3375 prev->next = insn;
3377 else
3379 insn->next = model_worklist;
3380 model_worklist = insn;
3382 if (insn->next)
3383 insn->next->prev = insn;
3386 /* Remove INSN from the model worklist. */
3388 static void
3389 model_remove_from_worklist (struct model_insn_info *insn)
3391 gcc_assert (QUEUE_INDEX (insn->insn) == QUEUE_READY);
3392 QUEUE_INDEX (insn->insn) = QUEUE_NOWHERE;
3394 if (insn->prev)
3395 insn->prev->next = insn->next;
3396 else
3397 model_worklist = insn->next;
3398 if (insn->next)
3399 insn->next->prev = insn->prev;
3402 /* Add INSN to the model worklist. Start looking for a suitable position
3403 between neighbors PREV and NEXT, testing at most MAX_SCHED_READY_INSNS
3404 insns either side. A null PREV indicates the beginning of the list and
3405 a null NEXT indicates the end. */
3407 static void
3408 model_add_to_worklist (struct model_insn_info *insn,
3409 struct model_insn_info *prev,
3410 struct model_insn_info *next)
3412 int count;
3414 count = MAX_SCHED_READY_INSNS;
3415 if (count > 0 && prev && model_order_p (insn, prev))
3418 count--;
3419 prev = prev->prev;
3421 while (count > 0 && prev && model_order_p (insn, prev));
3422 else
3423 while (count > 0 && next && model_order_p (next, insn))
3425 count--;
3426 prev = next;
3427 next = next->next;
3429 model_add_to_worklist_at (insn, prev);
3432 /* INSN may now have a higher priority (in the model_order_p sense)
3433 than before. Move it up the worklist if necessary. */
3435 static void
3436 model_promote_insn (struct model_insn_info *insn)
3438 struct model_insn_info *prev;
3439 int count;
3441 prev = insn->prev;
3442 count = MAX_SCHED_READY_INSNS;
3443 while (count > 0 && prev && model_order_p (insn, prev))
3445 count--;
3446 prev = prev->prev;
3448 if (prev != insn->prev)
3450 model_remove_from_worklist (insn);
3451 model_add_to_worklist_at (insn, prev);
3455 /* Add INSN to the end of the model schedule. */
3457 static void
3458 model_add_to_schedule (rtx_insn *insn)
3460 unsigned int point;
3462 gcc_assert (QUEUE_INDEX (insn) == QUEUE_NOWHERE);
3463 QUEUE_INDEX (insn) = QUEUE_SCHEDULED;
3465 point = model_schedule.length ();
3466 model_schedule.quick_push (insn);
3467 INSN_MODEL_INDEX (insn) = point + 1;
3470 /* Analyze the instructions that are to be scheduled, setting up
3471 MODEL_INSN_INFO (...) and model_num_insns accordingly. Add ready
3472 instructions to model_worklist. */
3474 static void
3475 model_analyze_insns (void)
3477 rtx_insn *start, *end, *iter;
3478 sd_iterator_def sd_it;
3479 dep_t dep;
3480 struct model_insn_info *insn, *con;
3482 model_num_insns = 0;
3483 start = PREV_INSN (current_sched_info->next_tail);
3484 end = current_sched_info->prev_head;
3485 for (iter = start; iter != end; iter = PREV_INSN (iter))
3486 if (NONDEBUG_INSN_P (iter))
3488 insn = MODEL_INSN_INFO (iter);
3489 insn->insn = iter;
3490 FOR_EACH_DEP (iter, SD_LIST_FORW, sd_it, dep)
3492 con = MODEL_INSN_INFO (DEP_CON (dep));
3493 if (con->insn && insn->alap < con->alap + 1)
3494 insn->alap = con->alap + 1;
3497 insn->old_queue = QUEUE_INDEX (iter);
3498 QUEUE_INDEX (iter) = QUEUE_NOWHERE;
3500 insn->unscheduled_preds = dep_list_size (iter, SD_LIST_HARD_BACK);
3501 if (insn->unscheduled_preds == 0)
3502 model_add_to_worklist (insn, NULL, model_worklist);
3504 model_num_insns++;
3508 /* The global state describes the register pressure at the start of the
3509 model schedule. Initialize GROUP accordingly. */
3511 static void
3512 model_init_pressure_group (struct model_pressure_group *group)
3514 int pci, cl;
3516 for (pci = 0; pci < ira_pressure_classes_num; pci++)
3518 cl = ira_pressure_classes[pci];
3519 group->limits[pci].pressure = curr_reg_pressure[cl];
3520 group->limits[pci].point = 0;
3522 /* Use index model_num_insns to record the state after the last
3523 instruction in the model schedule. */
3524 group->model = XNEWVEC (struct model_pressure_data,
3525 (model_num_insns + 1) * ira_pressure_classes_num);
3528 /* Record that MODEL_REF_PRESSURE (GROUP, POINT, PCI) is PRESSURE.
3529 Update the maximum pressure for the whole schedule. */
3531 static void
3532 model_record_pressure (struct model_pressure_group *group,
3533 int point, int pci, int pressure)
3535 MODEL_REF_PRESSURE (group, point, pci) = pressure;
3536 if (group->limits[pci].pressure < pressure)
3538 group->limits[pci].pressure = pressure;
3539 group->limits[pci].point = point;
3543 /* INSN has just been added to the end of the model schedule. Record its
3544 register-pressure information. */
3546 static void
3547 model_record_pressures (struct model_insn_info *insn)
3549 struct reg_pressure_data *reg_pressure;
3550 int point, pci, cl, delta;
3551 int death[N_REG_CLASSES];
3553 point = model_index (insn->insn);
3554 if (sched_verbose >= 2)
3556 if (point == 0)
3558 fprintf (sched_dump, "\n;;\tModel schedule:\n;;\n");
3559 fprintf (sched_dump, ";;\t| idx insn | mpri hght dpth prio |\n");
3561 fprintf (sched_dump, ";;\t| %3d %4d | %4d %4d %4d %4d | %-30s ",
3562 point, INSN_UID (insn->insn), insn->model_priority,
3563 insn->depth + insn->alap, insn->depth,
3564 INSN_PRIORITY (insn->insn),
3565 str_pattern_slim (PATTERN (insn->insn)));
3567 calculate_reg_deaths (insn->insn, death);
3568 reg_pressure = INSN_REG_PRESSURE (insn->insn);
3569 for (pci = 0; pci < ira_pressure_classes_num; pci++)
3571 cl = ira_pressure_classes[pci];
3572 delta = reg_pressure[pci].set_increase - death[cl];
3573 if (sched_verbose >= 2)
3574 fprintf (sched_dump, " %s:[%d,%+d]", reg_class_names[cl],
3575 curr_reg_pressure[cl], delta);
3576 model_record_pressure (&model_before_pressure, point, pci,
3577 curr_reg_pressure[cl]);
3579 if (sched_verbose >= 2)
3580 fprintf (sched_dump, "\n");
3583 /* All instructions have been added to the model schedule. Record the
3584 final register pressure in GROUP and set up all MODEL_MAX_PRESSUREs. */
3586 static void
3587 model_record_final_pressures (struct model_pressure_group *group)
3589 int point, pci, max_pressure, ref_pressure, cl;
3591 for (pci = 0; pci < ira_pressure_classes_num; pci++)
3593 /* Record the final pressure for this class. */
3594 cl = ira_pressure_classes[pci];
3595 point = model_num_insns;
3596 ref_pressure = curr_reg_pressure[cl];
3597 model_record_pressure (group, point, pci, ref_pressure);
3599 /* Record the original maximum pressure. */
3600 group->limits[pci].orig_pressure = group->limits[pci].pressure;
3602 /* Update the MODEL_MAX_PRESSURE for every point of the schedule. */
3603 max_pressure = ref_pressure;
3604 MODEL_MAX_PRESSURE (group, point, pci) = max_pressure;
3605 while (point > 0)
3607 point--;
3608 ref_pressure = MODEL_REF_PRESSURE (group, point, pci);
3609 max_pressure = MAX (max_pressure, ref_pressure);
3610 MODEL_MAX_PRESSURE (group, point, pci) = max_pressure;
3615 /* Update all successors of INSN, given that INSN has just been scheduled. */
3617 static void
3618 model_add_successors_to_worklist (struct model_insn_info *insn)
3620 sd_iterator_def sd_it;
3621 struct model_insn_info *con;
3622 dep_t dep;
3624 FOR_EACH_DEP (insn->insn, SD_LIST_FORW, sd_it, dep)
3626 con = MODEL_INSN_INFO (DEP_CON (dep));
3627 /* Ignore debug instructions, and instructions from other blocks. */
3628 if (con->insn)
3630 con->unscheduled_preds--;
3632 /* Update the depth field of each true-dependent successor.
3633 Increasing the depth gives them a higher priority than
3634 before. */
3635 if (DEP_TYPE (dep) == REG_DEP_TRUE && con->depth < insn->depth + 1)
3637 con->depth = insn->depth + 1;
3638 if (QUEUE_INDEX (con->insn) == QUEUE_READY)
3639 model_promote_insn (con);
3642 /* If this is a true dependency, or if there are no remaining
3643 dependencies for CON (meaning that CON only had non-true
3644 dependencies), make sure that CON is on the worklist.
3645 We don't bother otherwise because it would tend to fill the
3646 worklist with a lot of low-priority instructions that are not
3647 yet ready to issue. */
3648 if ((con->depth > 0 || con->unscheduled_preds == 0)
3649 && QUEUE_INDEX (con->insn) == QUEUE_NOWHERE)
3650 model_add_to_worklist (con, insn, insn->next);
3655 /* Give INSN a higher priority than any current instruction, then give
3656 unscheduled predecessors of INSN a higher priority still. If any of
3657 those predecessors are not on the model worklist, do the same for its
3658 predecessors, and so on. */
3660 static void
3661 model_promote_predecessors (struct model_insn_info *insn)
3663 struct model_insn_info *pro, *first;
3664 sd_iterator_def sd_it;
3665 dep_t dep;
3667 if (sched_verbose >= 7)
3668 fprintf (sched_dump, ";;\t+--- priority of %d = %d, priority of",
3669 INSN_UID (insn->insn), model_next_priority);
3670 insn->model_priority = model_next_priority++;
3671 model_remove_from_worklist (insn);
3672 model_add_to_worklist_at (insn, NULL);
3674 first = NULL;
3675 for (;;)
3677 FOR_EACH_DEP (insn->insn, SD_LIST_HARD_BACK, sd_it, dep)
3679 pro = MODEL_INSN_INFO (DEP_PRO (dep));
3680 /* The first test is to ignore debug instructions, and instructions
3681 from other blocks. */
3682 if (pro->insn
3683 && pro->model_priority != model_next_priority
3684 && QUEUE_INDEX (pro->insn) != QUEUE_SCHEDULED)
3686 pro->model_priority = model_next_priority;
3687 if (sched_verbose >= 7)
3688 fprintf (sched_dump, " %d", INSN_UID (pro->insn));
3689 if (QUEUE_INDEX (pro->insn) == QUEUE_READY)
3691 /* PRO is already in the worklist, but it now has
3692 a higher priority than before. Move it at the
3693 appropriate place. */
3694 model_remove_from_worklist (pro);
3695 model_add_to_worklist (pro, NULL, model_worklist);
3697 else
3699 /* PRO isn't in the worklist. Recursively process
3700 its predecessors until we find one that is. */
3701 pro->next = first;
3702 first = pro;
3706 if (!first)
3707 break;
3708 insn = first;
3709 first = insn->next;
3711 if (sched_verbose >= 7)
3712 fprintf (sched_dump, " = %d\n", model_next_priority);
3713 model_next_priority++;
3716 /* Pick one instruction from model_worklist and process it. */
3718 static void
3719 model_choose_insn (void)
3721 struct model_insn_info *insn, *fallback;
3722 int count;
3724 if (sched_verbose >= 7)
3726 fprintf (sched_dump, ";;\t+--- worklist:\n");
3727 insn = model_worklist;
3728 count = MAX_SCHED_READY_INSNS;
3729 while (count > 0 && insn)
3731 fprintf (sched_dump, ";;\t+--- %d [%d, %d, %d, %d]\n",
3732 INSN_UID (insn->insn), insn->model_priority,
3733 insn->depth + insn->alap, insn->depth,
3734 INSN_PRIORITY (insn->insn));
3735 count--;
3736 insn = insn->next;
3740 /* Look for a ready instruction whose model_classify_priority is zero
3741 or negative, picking the highest-priority one. Adding such an
3742 instruction to the schedule now should do no harm, and may actually
3743 do some good.
3745 Failing that, see whether there is an instruction with the highest
3746 extant model_priority that is not yet ready, but which would reduce
3747 pressure if it became ready. This is designed to catch cases like:
3749 (set (mem (reg R1)) (reg R2))
3751 where the instruction is the last remaining use of R1 and where the
3752 value of R2 is not yet available (or vice versa). The death of R1
3753 means that this instruction already reduces pressure. It is of
3754 course possible that the computation of R2 involves other registers
3755 that are hard to kill, but such cases are rare enough for this
3756 heuristic to be a win in general.
3758 Failing that, just pick the highest-priority instruction in the
3759 worklist. */
3760 count = MAX_SCHED_READY_INSNS;
3761 insn = model_worklist;
3762 fallback = 0;
3763 for (;;)
3765 if (count == 0 || !insn)
3767 insn = fallback ? fallback : model_worklist;
3768 break;
3770 if (insn->unscheduled_preds)
3772 if (model_worklist->model_priority == insn->model_priority
3773 && !fallback
3774 && model_classify_pressure (insn) < 0)
3775 fallback = insn;
3777 else
3779 if (model_classify_pressure (insn) <= 0)
3780 break;
3782 count--;
3783 insn = insn->next;
3786 if (sched_verbose >= 7 && insn != model_worklist)
3788 if (insn->unscheduled_preds)
3789 fprintf (sched_dump, ";;\t+--- promoting insn %d, with dependencies\n",
3790 INSN_UID (insn->insn));
3791 else
3792 fprintf (sched_dump, ";;\t+--- promoting insn %d, which is ready\n",
3793 INSN_UID (insn->insn));
3795 if (insn->unscheduled_preds)
3796 /* INSN isn't yet ready to issue. Give all its predecessors the
3797 highest priority. */
3798 model_promote_predecessors (insn);
3799 else
3801 /* INSN is ready. Add it to the end of model_schedule and
3802 process its successors. */
3803 model_add_successors_to_worklist (insn);
3804 model_remove_from_worklist (insn);
3805 model_add_to_schedule (insn->insn);
3806 model_record_pressures (insn);
3807 update_register_pressure (insn->insn);
3811 /* Restore all QUEUE_INDEXs to the values that they had before
3812 model_start_schedule was called. */
3814 static void
3815 model_reset_queue_indices (void)
3817 unsigned int i;
3818 rtx_insn *insn;
3820 FOR_EACH_VEC_ELT (model_schedule, i, insn)
3821 QUEUE_INDEX (insn) = MODEL_INSN_INFO (insn)->old_queue;
3824 /* We have calculated the model schedule and spill costs. Print a summary
3825 to sched_dump. */
3827 static void
3828 model_dump_pressure_summary (void)
3830 int pci, cl;
3832 fprintf (sched_dump, ";; Pressure summary:");
3833 for (pci = 0; pci < ira_pressure_classes_num; pci++)
3835 cl = ira_pressure_classes[pci];
3836 fprintf (sched_dump, " %s:%d", reg_class_names[cl],
3837 model_before_pressure.limits[pci].pressure);
3839 fprintf (sched_dump, "\n\n");
3842 /* Initialize the SCHED_PRESSURE_MODEL information for the current
3843 scheduling region. */
3845 static void
3846 model_start_schedule (basic_block bb)
3848 model_next_priority = 1;
3849 model_schedule.create (sched_max_luid);
3850 model_insns = XCNEWVEC (struct model_insn_info, sched_max_luid);
3852 gcc_assert (bb == BLOCK_FOR_INSN (NEXT_INSN (current_sched_info->prev_head)));
3853 initiate_reg_pressure_info (df_get_live_in (bb));
3855 model_analyze_insns ();
3856 model_init_pressure_group (&model_before_pressure);
3857 while (model_worklist)
3858 model_choose_insn ();
3859 gcc_assert (model_num_insns == (int) model_schedule.length ());
3860 if (sched_verbose >= 2)
3861 fprintf (sched_dump, "\n");
3863 model_record_final_pressures (&model_before_pressure);
3864 model_reset_queue_indices ();
3866 XDELETEVEC (model_insns);
3868 model_curr_point = 0;
3869 initiate_reg_pressure_info (df_get_live_in (bb));
3870 if (sched_verbose >= 1)
3871 model_dump_pressure_summary ();
3874 /* Free the information associated with GROUP. */
3876 static void
3877 model_finalize_pressure_group (struct model_pressure_group *group)
3879 XDELETEVEC (group->model);
3882 /* Free the information created by model_start_schedule. */
3884 static void
3885 model_end_schedule (void)
3887 model_finalize_pressure_group (&model_before_pressure);
3888 model_schedule.release ();
3891 /* Prepare reg pressure scheduling for basic block BB. */
3892 static void
3893 sched_pressure_start_bb (basic_block bb)
3895 /* Set the number of available registers for each class taking into account
3896 relative probability of current basic block versus function prologue and
3897 epilogue.
3898 * If the basic block executes much more often than the prologue/epilogue
3899 (e.g., inside a hot loop), then cost of spill in the prologue is close to
3900 nil, so the effective number of available registers is
3901 (ira_class_hard_regs_num[cl] - fixed_regs_num[cl] - 0).
3902 * If the basic block executes as often as the prologue/epilogue,
3903 then spill in the block is as costly as in the prologue, so the effective
3904 number of available registers is
3905 (ira_class_hard_regs_num[cl] - fixed_regs_num[cl]
3906 - call_saved_regs_num[cl]).
3907 Note that all-else-equal, we prefer to spill in the prologue, since that
3908 allows "extra" registers for other basic blocks of the function.
3909 * If the basic block is on the cold path of the function and executes
3910 rarely, then we should always prefer to spill in the block, rather than
3911 in the prologue/epilogue. The effective number of available register is
3912 (ira_class_hard_regs_num[cl] - fixed_regs_num[cl]
3913 - call_saved_regs_num[cl]). */
3915 int i;
3916 int entry_freq = ENTRY_BLOCK_PTR_FOR_FN (cfun)->frequency;
3917 int bb_freq = bb->frequency;
3919 if (bb_freq == 0)
3921 if (entry_freq == 0)
3922 entry_freq = bb_freq = 1;
3924 if (bb_freq < entry_freq)
3925 bb_freq = entry_freq;
3927 for (i = 0; i < ira_pressure_classes_num; ++i)
3929 enum reg_class cl = ira_pressure_classes[i];
3930 sched_class_regs_num[cl] = ira_class_hard_regs_num[cl]
3931 - fixed_regs_num[cl];
3932 sched_class_regs_num[cl]
3933 -= (call_saved_regs_num[cl] * entry_freq) / bb_freq;
3937 if (sched_pressure == SCHED_PRESSURE_MODEL)
3938 model_start_schedule (bb);
3941 /* A structure that holds local state for the loop in schedule_block. */
3942 struct sched_block_state
3944 /* True if no real insns have been scheduled in the current cycle. */
3945 bool first_cycle_insn_p;
3946 /* True if a shadow insn has been scheduled in the current cycle, which
3947 means that no more normal insns can be issued. */
3948 bool shadows_only_p;
3949 /* True if we're winding down a modulo schedule, which means that we only
3950 issue insns with INSN_EXACT_TICK set. */
3951 bool modulo_epilogue;
3952 /* Initialized with the machine's issue rate every cycle, and updated
3953 by calls to the variable_issue hook. */
3954 int can_issue_more;
3957 /* INSN is the "currently executing insn". Launch each insn which was
3958 waiting on INSN. READY is the ready list which contains the insns
3959 that are ready to fire. CLOCK is the current cycle. The function
3960 returns necessary cycle advance after issuing the insn (it is not
3961 zero for insns in a schedule group). */
3963 static int
3964 schedule_insn (rtx_insn *insn)
3966 sd_iterator_def sd_it;
3967 dep_t dep;
3968 int i;
3969 int advance = 0;
3971 if (sched_verbose >= 1)
3973 struct reg_pressure_data *pressure_info;
3974 fprintf (sched_dump, ";;\t%3i--> %s %-40s:",
3975 clock_var, (*current_sched_info->print_insn) (insn, 1),
3976 str_pattern_slim (PATTERN (insn)));
3978 if (recog_memoized (insn) < 0)
3979 fprintf (sched_dump, "nothing");
3980 else
3981 print_reservation (sched_dump, insn);
3982 pressure_info = INSN_REG_PRESSURE (insn);
3983 if (pressure_info != NULL)
3985 fputc (':', sched_dump);
3986 for (i = 0; i < ira_pressure_classes_num; i++)
3987 fprintf (sched_dump, "%s%s%+d(%d)",
3988 scheduled_insns.length () > 1
3989 && INSN_LUID (insn)
3990 < INSN_LUID (scheduled_insns[scheduled_insns.length () - 2]) ? "@" : "",
3991 reg_class_names[ira_pressure_classes[i]],
3992 pressure_info[i].set_increase, pressure_info[i].change);
3994 if (sched_pressure == SCHED_PRESSURE_MODEL
3995 && model_curr_point < model_num_insns
3996 && model_index (insn) == model_curr_point)
3997 fprintf (sched_dump, ":model %d", model_curr_point);
3998 fputc ('\n', sched_dump);
4001 if (sched_pressure == SCHED_PRESSURE_WEIGHTED && !DEBUG_INSN_P (insn))
4002 update_reg_and_insn_max_reg_pressure (insn);
4004 /* Scheduling instruction should have all its dependencies resolved and
4005 should have been removed from the ready list. */
4006 gcc_assert (sd_lists_empty_p (insn, SD_LIST_HARD_BACK));
4008 /* Reset debug insns invalidated by moving this insn. */
4009 if (MAY_HAVE_DEBUG_INSNS && !DEBUG_INSN_P (insn))
4010 for (sd_it = sd_iterator_start (insn, SD_LIST_BACK);
4011 sd_iterator_cond (&sd_it, &dep);)
4013 rtx_insn *dbg = DEP_PRO (dep);
4014 struct reg_use_data *use, *next;
4016 if (DEP_STATUS (dep) & DEP_CANCELLED)
4018 sd_iterator_next (&sd_it);
4019 continue;
4022 gcc_assert (DEBUG_INSN_P (dbg));
4024 if (sched_verbose >= 6)
4025 fprintf (sched_dump, ";;\t\tresetting: debug insn %d\n",
4026 INSN_UID (dbg));
4028 /* ??? Rather than resetting the debug insn, we might be able
4029 to emit a debug temp before the just-scheduled insn, but
4030 this would involve checking that the expression at the
4031 point of the debug insn is equivalent to the expression
4032 before the just-scheduled insn. They might not be: the
4033 expression in the debug insn may depend on other insns not
4034 yet scheduled that set MEMs, REGs or even other debug
4035 insns. It's not clear that attempting to preserve debug
4036 information in these cases is worth the effort, given how
4037 uncommon these resets are and the likelihood that the debug
4038 temps introduced won't survive the schedule change. */
4039 INSN_VAR_LOCATION_LOC (dbg) = gen_rtx_UNKNOWN_VAR_LOC ();
4040 df_insn_rescan (dbg);
4042 /* Unknown location doesn't use any registers. */
4043 for (use = INSN_REG_USE_LIST (dbg); use != NULL; use = next)
4045 struct reg_use_data *prev = use;
4047 /* Remove use from the cyclic next_regno_use chain first. */
4048 while (prev->next_regno_use != use)
4049 prev = prev->next_regno_use;
4050 prev->next_regno_use = use->next_regno_use;
4051 next = use->next_insn_use;
4052 free (use);
4054 INSN_REG_USE_LIST (dbg) = NULL;
4056 /* We delete rather than resolve these deps, otherwise we
4057 crash in sched_free_deps(), because forward deps are
4058 expected to be released before backward deps. */
4059 sd_delete_dep (sd_it);
4062 gcc_assert (QUEUE_INDEX (insn) == QUEUE_NOWHERE);
4063 QUEUE_INDEX (insn) = QUEUE_SCHEDULED;
4065 if (sched_pressure == SCHED_PRESSURE_MODEL
4066 && model_curr_point < model_num_insns
4067 && NONDEBUG_INSN_P (insn))
4069 if (model_index (insn) == model_curr_point)
4071 model_curr_point++;
4072 while (model_curr_point < model_num_insns
4073 && (QUEUE_INDEX (MODEL_INSN (model_curr_point))
4074 == QUEUE_SCHEDULED));
4075 else
4076 model_recompute (insn);
4077 model_update_limit_points ();
4078 update_register_pressure (insn);
4079 if (sched_verbose >= 2)
4080 print_curr_reg_pressure ();
4083 gcc_assert (INSN_TICK (insn) >= MIN_TICK);
4084 if (INSN_TICK (insn) > clock_var)
4085 /* INSN has been prematurely moved from the queue to the ready list.
4086 This is possible only if following flags are set. */
4087 gcc_assert (flag_sched_stalled_insns || sched_fusion);
4089 /* ??? Probably, if INSN is scheduled prematurely, we should leave
4090 INSN_TICK untouched. This is a machine-dependent issue, actually. */
4091 INSN_TICK (insn) = clock_var;
4093 check_clobbered_conditions (insn);
4095 /* Update dependent instructions. First, see if by scheduling this insn
4096 now we broke a dependence in a way that requires us to change another
4097 insn. */
4098 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
4099 sd_iterator_cond (&sd_it, &dep); sd_iterator_next (&sd_it))
4101 struct dep_replacement *desc = DEP_REPLACE (dep);
4102 rtx_insn *pro = DEP_PRO (dep);
4103 if (QUEUE_INDEX (pro) != QUEUE_SCHEDULED
4104 && desc != NULL && desc->insn == pro)
4105 apply_replacement (dep, false);
4108 /* Go through and resolve forward dependencies. */
4109 for (sd_it = sd_iterator_start (insn, SD_LIST_FORW);
4110 sd_iterator_cond (&sd_it, &dep);)
4112 rtx_insn *next = DEP_CON (dep);
4113 bool cancelled = (DEP_STATUS (dep) & DEP_CANCELLED) != 0;
4115 /* Resolve the dependence between INSN and NEXT.
4116 sd_resolve_dep () moves current dep to another list thus
4117 advancing the iterator. */
4118 sd_resolve_dep (sd_it);
4120 if (cancelled)
4122 if (must_restore_pattern_p (next, dep))
4123 restore_pattern (dep, false);
4124 continue;
4127 /* Don't bother trying to mark next as ready if insn is a debug
4128 insn. If insn is the last hard dependency, it will have
4129 already been discounted. */
4130 if (DEBUG_INSN_P (insn) && !DEBUG_INSN_P (next))
4131 continue;
4133 if (!IS_SPECULATION_BRANCHY_CHECK_P (insn))
4135 int effective_cost;
4137 effective_cost = try_ready (next);
4139 if (effective_cost >= 0
4140 && SCHED_GROUP_P (next)
4141 && advance < effective_cost)
4142 advance = effective_cost;
4144 else
4145 /* Check always has only one forward dependence (to the first insn in
4146 the recovery block), therefore, this will be executed only once. */
4148 gcc_assert (sd_lists_empty_p (insn, SD_LIST_FORW));
4149 fix_recovery_deps (RECOVERY_BLOCK (insn));
4153 /* Annotate the instruction with issue information -- TImode
4154 indicates that the instruction is expected not to be able
4155 to issue on the same cycle as the previous insn. A machine
4156 may use this information to decide how the instruction should
4157 be aligned. */
4158 if (issue_rate > 1
4159 && GET_CODE (PATTERN (insn)) != USE
4160 && GET_CODE (PATTERN (insn)) != CLOBBER
4161 && !DEBUG_INSN_P (insn))
4163 if (reload_completed)
4164 PUT_MODE (insn, clock_var > last_clock_var ? TImode : VOIDmode);
4165 last_clock_var = clock_var;
4168 if (nonscheduled_insns_begin != NULL_RTX)
4169 /* Indicate to debug counters that INSN is scheduled. */
4170 nonscheduled_insns_begin = insn;
4172 return advance;
4175 /* Functions for handling of notes. */
4177 /* Add note list that ends on FROM_END to the end of TO_ENDP. */
4178 void
4179 concat_note_lists (rtx_insn *from_end, rtx_insn **to_endp)
4181 rtx_insn *from_start;
4183 /* It's easy when have nothing to concat. */
4184 if (from_end == NULL)
4185 return;
4187 /* It's also easy when destination is empty. */
4188 if (*to_endp == NULL)
4190 *to_endp = from_end;
4191 return;
4194 from_start = from_end;
4195 while (PREV_INSN (from_start) != NULL)
4196 from_start = PREV_INSN (from_start);
4198 SET_PREV_INSN (from_start) = *to_endp;
4199 SET_NEXT_INSN (*to_endp) = from_start;
4200 *to_endp = from_end;
4203 /* Delete notes between HEAD and TAIL and put them in the chain
4204 of notes ended by NOTE_LIST. */
4205 void
4206 remove_notes (rtx_insn *head, rtx_insn *tail)
4208 rtx_insn *next_tail, *insn, *next;
4210 note_list = 0;
4211 if (head == tail && !INSN_P (head))
4212 return;
4214 next_tail = NEXT_INSN (tail);
4215 for (insn = head; insn != next_tail; insn = next)
4217 next = NEXT_INSN (insn);
4218 if (!NOTE_P (insn))
4219 continue;
4221 switch (NOTE_KIND (insn))
4223 case NOTE_INSN_BASIC_BLOCK:
4224 continue;
4226 case NOTE_INSN_EPILOGUE_BEG:
4227 if (insn != tail)
4229 remove_insn (insn);
4230 add_reg_note (next, REG_SAVE_NOTE,
4231 GEN_INT (NOTE_INSN_EPILOGUE_BEG));
4232 break;
4234 /* FALLTHRU */
4236 default:
4237 remove_insn (insn);
4239 /* Add the note to list that ends at NOTE_LIST. */
4240 SET_PREV_INSN (insn) = note_list;
4241 SET_NEXT_INSN (insn) = NULL_RTX;
4242 if (note_list)
4243 SET_NEXT_INSN (note_list) = insn;
4244 note_list = insn;
4245 break;
4248 gcc_assert ((sel_sched_p () || insn != tail) && insn != head);
4252 /* A structure to record enough data to allow us to backtrack the scheduler to
4253 a previous state. */
4254 struct haifa_saved_data
4256 /* Next entry on the list. */
4257 struct haifa_saved_data *next;
4259 /* Backtracking is associated with scheduling insns that have delay slots.
4260 DELAY_PAIR points to the structure that contains the insns involved, and
4261 the number of cycles between them. */
4262 struct delay_pair *delay_pair;
4264 /* Data used by the frontend (e.g. sched-ebb or sched-rgn). */
4265 void *fe_saved_data;
4266 /* Data used by the backend. */
4267 void *be_saved_data;
4269 /* Copies of global state. */
4270 int clock_var, last_clock_var;
4271 struct ready_list ready;
4272 state_t curr_state;
4274 rtx_insn *last_scheduled_insn;
4275 rtx_insn *last_nondebug_scheduled_insn;
4276 rtx_insn *nonscheduled_insns_begin;
4277 int cycle_issued_insns;
4279 /* Copies of state used in the inner loop of schedule_block. */
4280 struct sched_block_state sched_block;
4282 /* We don't need to save q_ptr, as its value is arbitrary and we can set it
4283 to 0 when restoring. */
4284 int q_size;
4285 rtx_insn_list **insn_queue;
4287 /* Describe pattern replacements that occurred since this backtrack point
4288 was queued. */
4289 vec<dep_t> replacement_deps;
4290 vec<int> replace_apply;
4292 /* A copy of the next-cycle replacement vectors at the time of the backtrack
4293 point. */
4294 vec<dep_t> next_cycle_deps;
4295 vec<int> next_cycle_apply;
4298 /* A record, in reverse order, of all scheduled insns which have delay slots
4299 and may require backtracking. */
4300 static struct haifa_saved_data *backtrack_queue;
4302 /* For every dependency of INSN, set the FEEDS_BACKTRACK_INSN bit according
4303 to SET_P. */
4304 static void
4305 mark_backtrack_feeds (rtx_insn *insn, int set_p)
4307 sd_iterator_def sd_it;
4308 dep_t dep;
4309 FOR_EACH_DEP (insn, SD_LIST_HARD_BACK, sd_it, dep)
4311 FEEDS_BACKTRACK_INSN (DEP_PRO (dep)) = set_p;
4315 /* Save the current scheduler state so that we can backtrack to it
4316 later if necessary. PAIR gives the insns that make it necessary to
4317 save this point. SCHED_BLOCK is the local state of schedule_block
4318 that need to be saved. */
4319 static void
4320 save_backtrack_point (struct delay_pair *pair,
4321 struct sched_block_state sched_block)
4323 int i;
4324 struct haifa_saved_data *save = XNEW (struct haifa_saved_data);
4326 save->curr_state = xmalloc (dfa_state_size);
4327 memcpy (save->curr_state, curr_state, dfa_state_size);
4329 save->ready.first = ready.first;
4330 save->ready.n_ready = ready.n_ready;
4331 save->ready.n_debug = ready.n_debug;
4332 save->ready.veclen = ready.veclen;
4333 save->ready.vec = XNEWVEC (rtx_insn *, ready.veclen);
4334 memcpy (save->ready.vec, ready.vec, ready.veclen * sizeof (rtx));
4336 save->insn_queue = XNEWVEC (rtx_insn_list *, max_insn_queue_index + 1);
4337 save->q_size = q_size;
4338 for (i = 0; i <= max_insn_queue_index; i++)
4340 int q = NEXT_Q_AFTER (q_ptr, i);
4341 save->insn_queue[i] = copy_INSN_LIST (insn_queue[q]);
4344 save->clock_var = clock_var;
4345 save->last_clock_var = last_clock_var;
4346 save->cycle_issued_insns = cycle_issued_insns;
4347 save->last_scheduled_insn = last_scheduled_insn;
4348 save->last_nondebug_scheduled_insn = last_nondebug_scheduled_insn;
4349 save->nonscheduled_insns_begin = nonscheduled_insns_begin;
4351 save->sched_block = sched_block;
4353 save->replacement_deps.create (0);
4354 save->replace_apply.create (0);
4355 save->next_cycle_deps = next_cycle_replace_deps.copy ();
4356 save->next_cycle_apply = next_cycle_apply.copy ();
4358 if (current_sched_info->save_state)
4359 save->fe_saved_data = (*current_sched_info->save_state) ();
4361 if (targetm.sched.alloc_sched_context)
4363 save->be_saved_data = targetm.sched.alloc_sched_context ();
4364 targetm.sched.init_sched_context (save->be_saved_data, false);
4366 else
4367 save->be_saved_data = NULL;
4369 save->delay_pair = pair;
4371 save->next = backtrack_queue;
4372 backtrack_queue = save;
4374 while (pair)
4376 mark_backtrack_feeds (pair->i2, 1);
4377 INSN_TICK (pair->i2) = INVALID_TICK;
4378 INSN_EXACT_TICK (pair->i2) = clock_var + pair_delay (pair);
4379 SHADOW_P (pair->i2) = pair->stages == 0;
4380 pair = pair->next_same_i1;
4384 /* Walk the ready list and all queues. If any insns have unresolved backwards
4385 dependencies, these must be cancelled deps, broken by predication. Set or
4386 clear (depending on SET) the DEP_CANCELLED bit in DEP_STATUS. */
4388 static void
4389 toggle_cancelled_flags (bool set)
4391 int i;
4392 sd_iterator_def sd_it;
4393 dep_t dep;
4395 if (ready.n_ready > 0)
4397 rtx_insn **first = ready_lastpos (&ready);
4398 for (i = 0; i < ready.n_ready; i++)
4399 FOR_EACH_DEP (first[i], SD_LIST_BACK, sd_it, dep)
4400 if (!DEBUG_INSN_P (DEP_PRO (dep)))
4402 if (set)
4403 DEP_STATUS (dep) |= DEP_CANCELLED;
4404 else
4405 DEP_STATUS (dep) &= ~DEP_CANCELLED;
4408 for (i = 0; i <= max_insn_queue_index; i++)
4410 int q = NEXT_Q_AFTER (q_ptr, i);
4411 rtx_insn_list *link;
4412 for (link = insn_queue[q]; link; link = link->next ())
4414 rtx_insn *insn = link->insn ();
4415 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep)
4416 if (!DEBUG_INSN_P (DEP_PRO (dep)))
4418 if (set)
4419 DEP_STATUS (dep) |= DEP_CANCELLED;
4420 else
4421 DEP_STATUS (dep) &= ~DEP_CANCELLED;
4427 /* Undo the replacements that have occurred after backtrack point SAVE
4428 was placed. */
4429 static void
4430 undo_replacements_for_backtrack (struct haifa_saved_data *save)
4432 while (!save->replacement_deps.is_empty ())
4434 dep_t dep = save->replacement_deps.pop ();
4435 int apply_p = save->replace_apply.pop ();
4437 if (apply_p)
4438 restore_pattern (dep, true);
4439 else
4440 apply_replacement (dep, true);
4442 save->replacement_deps.release ();
4443 save->replace_apply.release ();
4446 /* Pop entries from the SCHEDULED_INSNS vector up to and including INSN.
4447 Restore their dependencies to an unresolved state, and mark them as
4448 queued nowhere. */
4450 static void
4451 unschedule_insns_until (rtx_insn *insn)
4453 auto_vec<rtx_insn *> recompute_vec;
4455 /* Make two passes over the insns to be unscheduled. First, we clear out
4456 dependencies and other trivial bookkeeping. */
4457 for (;;)
4459 rtx_insn *last;
4460 sd_iterator_def sd_it;
4461 dep_t dep;
4463 last = scheduled_insns.pop ();
4465 /* This will be changed by restore_backtrack_point if the insn is in
4466 any queue. */
4467 QUEUE_INDEX (last) = QUEUE_NOWHERE;
4468 if (last != insn)
4469 INSN_TICK (last) = INVALID_TICK;
4471 if (modulo_ii > 0 && INSN_UID (last) < modulo_iter0_max_uid)
4472 modulo_insns_scheduled--;
4474 for (sd_it = sd_iterator_start (last, SD_LIST_RES_FORW);
4475 sd_iterator_cond (&sd_it, &dep);)
4477 rtx_insn *con = DEP_CON (dep);
4478 sd_unresolve_dep (sd_it);
4479 if (!MUST_RECOMPUTE_SPEC_P (con))
4481 MUST_RECOMPUTE_SPEC_P (con) = 1;
4482 recompute_vec.safe_push (con);
4486 if (last == insn)
4487 break;
4490 /* A second pass, to update ready and speculation status for insns
4491 depending on the unscheduled ones. The first pass must have
4492 popped the scheduled_insns vector up to the point where we
4493 restart scheduling, as recompute_todo_spec requires it to be
4494 up-to-date. */
4495 while (!recompute_vec.is_empty ())
4497 rtx_insn *con;
4499 con = recompute_vec.pop ();
4500 MUST_RECOMPUTE_SPEC_P (con) = 0;
4501 if (!sd_lists_empty_p (con, SD_LIST_HARD_BACK))
4503 TODO_SPEC (con) = HARD_DEP;
4504 INSN_TICK (con) = INVALID_TICK;
4505 if (PREDICATED_PAT (con) != NULL_RTX)
4506 haifa_change_pattern (con, ORIG_PAT (con));
4508 else if (QUEUE_INDEX (con) != QUEUE_SCHEDULED)
4509 TODO_SPEC (con) = recompute_todo_spec (con, true);
4513 /* Restore scheduler state from the topmost entry on the backtracking queue.
4514 PSCHED_BLOCK_P points to the local data of schedule_block that we must
4515 overwrite with the saved data.
4516 The caller must already have called unschedule_insns_until. */
4518 static void
4519 restore_last_backtrack_point (struct sched_block_state *psched_block)
4521 int i;
4522 struct haifa_saved_data *save = backtrack_queue;
4524 backtrack_queue = save->next;
4526 if (current_sched_info->restore_state)
4527 (*current_sched_info->restore_state) (save->fe_saved_data);
4529 if (targetm.sched.alloc_sched_context)
4531 targetm.sched.set_sched_context (save->be_saved_data);
4532 targetm.sched.free_sched_context (save->be_saved_data);
4535 /* Do this first since it clobbers INSN_TICK of the involved
4536 instructions. */
4537 undo_replacements_for_backtrack (save);
4539 /* Clear the QUEUE_INDEX of everything in the ready list or one
4540 of the queues. */
4541 if (ready.n_ready > 0)
4543 rtx_insn **first = ready_lastpos (&ready);
4544 for (i = 0; i < ready.n_ready; i++)
4546 rtx_insn *insn = first[i];
4547 QUEUE_INDEX (insn) = QUEUE_NOWHERE;
4548 INSN_TICK (insn) = INVALID_TICK;
4551 for (i = 0; i <= max_insn_queue_index; i++)
4553 int q = NEXT_Q_AFTER (q_ptr, i);
4555 for (rtx_insn_list *link = insn_queue[q]; link; link = link->next ())
4557 rtx_insn *x = link->insn ();
4558 QUEUE_INDEX (x) = QUEUE_NOWHERE;
4559 INSN_TICK (x) = INVALID_TICK;
4561 free_INSN_LIST_list (&insn_queue[q]);
4564 free (ready.vec);
4565 ready = save->ready;
4567 if (ready.n_ready > 0)
4569 rtx_insn **first = ready_lastpos (&ready);
4570 for (i = 0; i < ready.n_ready; i++)
4572 rtx_insn *insn = first[i];
4573 QUEUE_INDEX (insn) = QUEUE_READY;
4574 TODO_SPEC (insn) = recompute_todo_spec (insn, true);
4575 INSN_TICK (insn) = save->clock_var;
4579 q_ptr = 0;
4580 q_size = save->q_size;
4581 for (i = 0; i <= max_insn_queue_index; i++)
4583 int q = NEXT_Q_AFTER (q_ptr, i);
4585 insn_queue[q] = save->insn_queue[q];
4587 for (rtx_insn_list *link = insn_queue[q]; link; link = link->next ())
4589 rtx_insn *x = link->insn ();
4590 QUEUE_INDEX (x) = i;
4591 TODO_SPEC (x) = recompute_todo_spec (x, true);
4592 INSN_TICK (x) = save->clock_var + i;
4595 free (save->insn_queue);
4597 toggle_cancelled_flags (true);
4599 clock_var = save->clock_var;
4600 last_clock_var = save->last_clock_var;
4601 cycle_issued_insns = save->cycle_issued_insns;
4602 last_scheduled_insn = save->last_scheduled_insn;
4603 last_nondebug_scheduled_insn = save->last_nondebug_scheduled_insn;
4604 nonscheduled_insns_begin = save->nonscheduled_insns_begin;
4606 *psched_block = save->sched_block;
4608 memcpy (curr_state, save->curr_state, dfa_state_size);
4609 free (save->curr_state);
4611 mark_backtrack_feeds (save->delay_pair->i2, 0);
4613 gcc_assert (next_cycle_replace_deps.is_empty ());
4614 next_cycle_replace_deps = save->next_cycle_deps.copy ();
4615 next_cycle_apply = save->next_cycle_apply.copy ();
4617 free (save);
4619 for (save = backtrack_queue; save; save = save->next)
4621 mark_backtrack_feeds (save->delay_pair->i2, 1);
4625 /* Discard all data associated with the topmost entry in the backtrack
4626 queue. If RESET_TICK is false, we just want to free the data. If true,
4627 we are doing this because we discovered a reason to backtrack. In the
4628 latter case, also reset the INSN_TICK for the shadow insn. */
4629 static void
4630 free_topmost_backtrack_point (bool reset_tick)
4632 struct haifa_saved_data *save = backtrack_queue;
4633 int i;
4635 backtrack_queue = save->next;
4637 if (reset_tick)
4639 struct delay_pair *pair = save->delay_pair;
4640 while (pair)
4642 INSN_TICK (pair->i2) = INVALID_TICK;
4643 INSN_EXACT_TICK (pair->i2) = INVALID_TICK;
4644 pair = pair->next_same_i1;
4646 undo_replacements_for_backtrack (save);
4648 else
4650 save->replacement_deps.release ();
4651 save->replace_apply.release ();
4654 if (targetm.sched.free_sched_context)
4655 targetm.sched.free_sched_context (save->be_saved_data);
4656 if (current_sched_info->restore_state)
4657 free (save->fe_saved_data);
4658 for (i = 0; i <= max_insn_queue_index; i++)
4659 free_INSN_LIST_list (&save->insn_queue[i]);
4660 free (save->insn_queue);
4661 free (save->curr_state);
4662 free (save->ready.vec);
4663 free (save);
4666 /* Free the entire backtrack queue. */
4667 static void
4668 free_backtrack_queue (void)
4670 while (backtrack_queue)
4671 free_topmost_backtrack_point (false);
4674 /* Apply a replacement described by DESC. If IMMEDIATELY is false, we
4675 may have to postpone the replacement until the start of the next cycle,
4676 at which point we will be called again with IMMEDIATELY true. This is
4677 only done for machines which have instruction packets with explicit
4678 parallelism however. */
4679 static void
4680 apply_replacement (dep_t dep, bool immediately)
4682 struct dep_replacement *desc = DEP_REPLACE (dep);
4683 if (!immediately && targetm.sched.exposed_pipeline && reload_completed)
4685 next_cycle_replace_deps.safe_push (dep);
4686 next_cycle_apply.safe_push (1);
4688 else
4690 bool success;
4692 if (QUEUE_INDEX (desc->insn) == QUEUE_SCHEDULED)
4693 return;
4695 if (sched_verbose >= 5)
4696 fprintf (sched_dump, "applying replacement for insn %d\n",
4697 INSN_UID (desc->insn));
4699 success = validate_change (desc->insn, desc->loc, desc->newval, 0);
4700 gcc_assert (success);
4702 update_insn_after_change (desc->insn);
4703 if ((TODO_SPEC (desc->insn) & (HARD_DEP | DEP_POSTPONED)) == 0)
4704 fix_tick_ready (desc->insn);
4706 if (backtrack_queue != NULL)
4708 backtrack_queue->replacement_deps.safe_push (dep);
4709 backtrack_queue->replace_apply.safe_push (1);
4714 /* We have determined that a pattern involved in DEP must be restored.
4715 If IMMEDIATELY is false, we may have to postpone the replacement
4716 until the start of the next cycle, at which point we will be called
4717 again with IMMEDIATELY true. */
4718 static void
4719 restore_pattern (dep_t dep, bool immediately)
4721 rtx_insn *next = DEP_CON (dep);
4722 int tick = INSN_TICK (next);
4724 /* If we already scheduled the insn, the modified version is
4725 correct. */
4726 if (QUEUE_INDEX (next) == QUEUE_SCHEDULED)
4727 return;
4729 if (!immediately && targetm.sched.exposed_pipeline && reload_completed)
4731 next_cycle_replace_deps.safe_push (dep);
4732 next_cycle_apply.safe_push (0);
4733 return;
4737 if (DEP_TYPE (dep) == REG_DEP_CONTROL)
4739 if (sched_verbose >= 5)
4740 fprintf (sched_dump, "restoring pattern for insn %d\n",
4741 INSN_UID (next));
4742 haifa_change_pattern (next, ORIG_PAT (next));
4744 else
4746 struct dep_replacement *desc = DEP_REPLACE (dep);
4747 bool success;
4749 if (sched_verbose >= 5)
4750 fprintf (sched_dump, "restoring pattern for insn %d\n",
4751 INSN_UID (desc->insn));
4752 tick = INSN_TICK (desc->insn);
4754 success = validate_change (desc->insn, desc->loc, desc->orig, 0);
4755 gcc_assert (success);
4756 update_insn_after_change (desc->insn);
4757 if (backtrack_queue != NULL)
4759 backtrack_queue->replacement_deps.safe_push (dep);
4760 backtrack_queue->replace_apply.safe_push (0);
4763 INSN_TICK (next) = tick;
4764 if (TODO_SPEC (next) == DEP_POSTPONED)
4765 return;
4767 if (sd_lists_empty_p (next, SD_LIST_BACK))
4768 TODO_SPEC (next) = 0;
4769 else if (!sd_lists_empty_p (next, SD_LIST_HARD_BACK))
4770 TODO_SPEC (next) = HARD_DEP;
4773 /* Perform pattern replacements that were queued up until the next
4774 cycle. */
4775 static void
4776 perform_replacements_new_cycle (void)
4778 int i;
4779 dep_t dep;
4780 FOR_EACH_VEC_ELT (next_cycle_replace_deps, i, dep)
4782 int apply_p = next_cycle_apply[i];
4783 if (apply_p)
4784 apply_replacement (dep, true);
4785 else
4786 restore_pattern (dep, true);
4788 next_cycle_replace_deps.truncate (0);
4789 next_cycle_apply.truncate (0);
4792 /* Compute INSN_TICK_ESTIMATE for INSN. PROCESSED is a bitmap of
4793 instructions we've previously encountered, a set bit prevents
4794 recursion. BUDGET is a limit on how far ahead we look, it is
4795 reduced on recursive calls. Return true if we produced a good
4796 estimate, or false if we exceeded the budget. */
4797 static bool
4798 estimate_insn_tick (bitmap processed, rtx_insn *insn, int budget)
4800 sd_iterator_def sd_it;
4801 dep_t dep;
4802 int earliest = INSN_TICK (insn);
4804 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep)
4806 rtx_insn *pro = DEP_PRO (dep);
4807 int t;
4809 if (DEP_STATUS (dep) & DEP_CANCELLED)
4810 continue;
4812 if (QUEUE_INDEX (pro) == QUEUE_SCHEDULED)
4813 gcc_assert (INSN_TICK (pro) + dep_cost (dep) <= INSN_TICK (insn));
4814 else
4816 int cost = dep_cost (dep);
4817 if (cost >= budget)
4818 return false;
4819 if (!bitmap_bit_p (processed, INSN_LUID (pro)))
4821 if (!estimate_insn_tick (processed, pro, budget - cost))
4822 return false;
4824 gcc_assert (INSN_TICK_ESTIMATE (pro) != INVALID_TICK);
4825 t = INSN_TICK_ESTIMATE (pro) + cost;
4826 if (earliest == INVALID_TICK || t > earliest)
4827 earliest = t;
4830 bitmap_set_bit (processed, INSN_LUID (insn));
4831 INSN_TICK_ESTIMATE (insn) = earliest;
4832 return true;
4835 /* Examine the pair of insns in P, and estimate (optimistically, assuming
4836 infinite resources) the cycle in which the delayed shadow can be issued.
4837 Return the number of cycles that must pass before the real insn can be
4838 issued in order to meet this constraint. */
4839 static int
4840 estimate_shadow_tick (struct delay_pair *p)
4842 bitmap_head processed;
4843 int t;
4844 bool cutoff;
4845 bitmap_initialize (&processed, 0);
4847 cutoff = !estimate_insn_tick (&processed, p->i2,
4848 max_insn_queue_index + pair_delay (p));
4849 bitmap_clear (&processed);
4850 if (cutoff)
4851 return max_insn_queue_index;
4852 t = INSN_TICK_ESTIMATE (p->i2) - (clock_var + pair_delay (p) + 1);
4853 if (t > 0)
4854 return t;
4855 return 0;
4858 /* If INSN has no unresolved backwards dependencies, add it to the schedule and
4859 recursively resolve all its forward dependencies. */
4860 static void
4861 resolve_dependencies (rtx_insn *insn)
4863 sd_iterator_def sd_it;
4864 dep_t dep;
4866 /* Don't use sd_lists_empty_p; it ignores debug insns. */
4867 if (DEPS_LIST_FIRST (INSN_HARD_BACK_DEPS (insn)) != NULL
4868 || DEPS_LIST_FIRST (INSN_SPEC_BACK_DEPS (insn)) != NULL)
4869 return;
4871 if (sched_verbose >= 4)
4872 fprintf (sched_dump, ";;\tquickly resolving %d\n", INSN_UID (insn));
4874 if (QUEUE_INDEX (insn) >= 0)
4875 queue_remove (insn);
4877 scheduled_insns.safe_push (insn);
4879 /* Update dependent instructions. */
4880 for (sd_it = sd_iterator_start (insn, SD_LIST_FORW);
4881 sd_iterator_cond (&sd_it, &dep);)
4883 rtx_insn *next = DEP_CON (dep);
4885 if (sched_verbose >= 4)
4886 fprintf (sched_dump, ";;\t\tdep %d against %d\n", INSN_UID (insn),
4887 INSN_UID (next));
4889 /* Resolve the dependence between INSN and NEXT.
4890 sd_resolve_dep () moves current dep to another list thus
4891 advancing the iterator. */
4892 sd_resolve_dep (sd_it);
4894 if (!IS_SPECULATION_BRANCHY_CHECK_P (insn))
4896 resolve_dependencies (next);
4898 else
4899 /* Check always has only one forward dependence (to the first insn in
4900 the recovery block), therefore, this will be executed only once. */
4902 gcc_assert (sd_lists_empty_p (insn, SD_LIST_FORW));
4908 /* Return the head and tail pointers of ebb starting at BEG and ending
4909 at END. */
4910 void
4911 get_ebb_head_tail (basic_block beg, basic_block end,
4912 rtx_insn **headp, rtx_insn **tailp)
4914 rtx_insn *beg_head = BB_HEAD (beg);
4915 rtx_insn * beg_tail = BB_END (beg);
4916 rtx_insn * end_head = BB_HEAD (end);
4917 rtx_insn * end_tail = BB_END (end);
4919 /* Don't include any notes or labels at the beginning of the BEG
4920 basic block, or notes at the end of the END basic blocks. */
4922 if (LABEL_P (beg_head))
4923 beg_head = NEXT_INSN (beg_head);
4925 while (beg_head != beg_tail)
4926 if (NOTE_P (beg_head))
4927 beg_head = NEXT_INSN (beg_head);
4928 else if (DEBUG_INSN_P (beg_head))
4930 rtx_insn * note, *next;
4932 for (note = NEXT_INSN (beg_head);
4933 note != beg_tail;
4934 note = next)
4936 next = NEXT_INSN (note);
4937 if (NOTE_P (note))
4939 if (sched_verbose >= 9)
4940 fprintf (sched_dump, "reorder %i\n", INSN_UID (note));
4942 reorder_insns_nobb (note, note, PREV_INSN (beg_head));
4944 if (BLOCK_FOR_INSN (note) != beg)
4945 df_insn_change_bb (note, beg);
4947 else if (!DEBUG_INSN_P (note))
4948 break;
4951 break;
4953 else
4954 break;
4956 *headp = beg_head;
4958 if (beg == end)
4959 end_head = beg_head;
4960 else if (LABEL_P (end_head))
4961 end_head = NEXT_INSN (end_head);
4963 while (end_head != end_tail)
4964 if (NOTE_P (end_tail))
4965 end_tail = PREV_INSN (end_tail);
4966 else if (DEBUG_INSN_P (end_tail))
4968 rtx_insn * note, *prev;
4970 for (note = PREV_INSN (end_tail);
4971 note != end_head;
4972 note = prev)
4974 prev = PREV_INSN (note);
4975 if (NOTE_P (note))
4977 if (sched_verbose >= 9)
4978 fprintf (sched_dump, "reorder %i\n", INSN_UID (note));
4980 reorder_insns_nobb (note, note, end_tail);
4982 if (end_tail == BB_END (end))
4983 BB_END (end) = note;
4985 if (BLOCK_FOR_INSN (note) != end)
4986 df_insn_change_bb (note, end);
4988 else if (!DEBUG_INSN_P (note))
4989 break;
4992 break;
4994 else
4995 break;
4997 *tailp = end_tail;
5000 /* Return nonzero if there are no real insns in the range [ HEAD, TAIL ]. */
5003 no_real_insns_p (const rtx_insn *head, const rtx_insn *tail)
5005 while (head != NEXT_INSN (tail))
5007 if (!NOTE_P (head) && !LABEL_P (head))
5008 return 0;
5009 head = NEXT_INSN (head);
5011 return 1;
5014 /* Restore-other-notes: NOTE_LIST is the end of a chain of notes
5015 previously found among the insns. Insert them just before HEAD. */
5016 rtx_insn *
5017 restore_other_notes (rtx_insn *head, basic_block head_bb)
5019 if (note_list != 0)
5021 rtx_insn *note_head = note_list;
5023 if (head)
5024 head_bb = BLOCK_FOR_INSN (head);
5025 else
5026 head = NEXT_INSN (bb_note (head_bb));
5028 while (PREV_INSN (note_head))
5030 set_block_for_insn (note_head, head_bb);
5031 note_head = PREV_INSN (note_head);
5033 /* In the above cycle we've missed this note. */
5034 set_block_for_insn (note_head, head_bb);
5036 SET_PREV_INSN (note_head) = PREV_INSN (head);
5037 SET_NEXT_INSN (PREV_INSN (head)) = note_head;
5038 SET_PREV_INSN (head) = note_list;
5039 SET_NEXT_INSN (note_list) = head;
5041 if (BLOCK_FOR_INSN (head) != head_bb)
5042 BB_END (head_bb) = note_list;
5044 head = note_head;
5047 return head;
5050 /* When we know we are going to discard the schedule due to a failed attempt
5051 at modulo scheduling, undo all replacements. */
5052 static void
5053 undo_all_replacements (void)
5055 rtx_insn *insn;
5056 int i;
5058 FOR_EACH_VEC_ELT (scheduled_insns, i, insn)
5060 sd_iterator_def sd_it;
5061 dep_t dep;
5063 /* See if we must undo a replacement. */
5064 for (sd_it = sd_iterator_start (insn, SD_LIST_RES_FORW);
5065 sd_iterator_cond (&sd_it, &dep); sd_iterator_next (&sd_it))
5067 struct dep_replacement *desc = DEP_REPLACE (dep);
5068 if (desc != NULL)
5069 validate_change (desc->insn, desc->loc, desc->orig, 0);
5074 /* Return first non-scheduled insn in the current scheduling block.
5075 This is mostly used for debug-counter purposes. */
5076 static rtx_insn *
5077 first_nonscheduled_insn (void)
5079 rtx_insn *insn = (nonscheduled_insns_begin != NULL_RTX
5080 ? nonscheduled_insns_begin
5081 : current_sched_info->prev_head);
5085 insn = next_nonnote_nondebug_insn (insn);
5087 while (QUEUE_INDEX (insn) == QUEUE_SCHEDULED);
5089 return insn;
5092 /* Move insns that became ready to fire from queue to ready list. */
5094 static void
5095 queue_to_ready (struct ready_list *ready)
5097 rtx_insn *insn;
5098 rtx_insn_list *link;
5099 rtx_insn *skip_insn;
5101 q_ptr = NEXT_Q (q_ptr);
5103 if (dbg_cnt (sched_insn) == false)
5104 /* If debug counter is activated do not requeue the first
5105 nonscheduled insn. */
5106 skip_insn = first_nonscheduled_insn ();
5107 else
5108 skip_insn = NULL;
5110 /* Add all pending insns that can be scheduled without stalls to the
5111 ready list. */
5112 for (link = insn_queue[q_ptr]; link; link = link->next ())
5114 insn = link->insn ();
5115 q_size -= 1;
5117 if (sched_verbose >= 2)
5118 fprintf (sched_dump, ";;\t\tQ-->Ready: insn %s: ",
5119 (*current_sched_info->print_insn) (insn, 0));
5121 /* If the ready list is full, delay the insn for 1 cycle.
5122 See the comment in schedule_block for the rationale. */
5123 if (!reload_completed
5124 && (ready->n_ready - ready->n_debug > MAX_SCHED_READY_INSNS
5125 || (sched_pressure == SCHED_PRESSURE_MODEL
5126 /* Limit pressure recalculations to MAX_SCHED_READY_INSNS
5127 instructions too. */
5128 && model_index (insn) > (model_curr_point
5129 + MAX_SCHED_READY_INSNS)))
5130 && !(sched_pressure == SCHED_PRESSURE_MODEL
5131 && model_curr_point < model_num_insns
5132 /* Always allow the next model instruction to issue. */
5133 && model_index (insn) == model_curr_point)
5134 && !SCHED_GROUP_P (insn)
5135 && insn != skip_insn)
5137 if (sched_verbose >= 2)
5138 fprintf (sched_dump, "keeping in queue, ready full\n");
5139 queue_insn (insn, 1, "ready full");
5141 else
5143 ready_add (ready, insn, false);
5144 if (sched_verbose >= 2)
5145 fprintf (sched_dump, "moving to ready without stalls\n");
5148 free_INSN_LIST_list (&insn_queue[q_ptr]);
5150 /* If there are no ready insns, stall until one is ready and add all
5151 of the pending insns at that point to the ready list. */
5152 if (ready->n_ready == 0)
5154 int stalls;
5156 for (stalls = 1; stalls <= max_insn_queue_index; stalls++)
5158 if ((link = insn_queue[NEXT_Q_AFTER (q_ptr, stalls)]))
5160 for (; link; link = link->next ())
5162 insn = link->insn ();
5163 q_size -= 1;
5165 if (sched_verbose >= 2)
5166 fprintf (sched_dump, ";;\t\tQ-->Ready: insn %s: ",
5167 (*current_sched_info->print_insn) (insn, 0));
5169 ready_add (ready, insn, false);
5170 if (sched_verbose >= 2)
5171 fprintf (sched_dump, "moving to ready with %d stalls\n", stalls);
5173 free_INSN_LIST_list (&insn_queue[NEXT_Q_AFTER (q_ptr, stalls)]);
5175 advance_one_cycle ();
5177 break;
5180 advance_one_cycle ();
5183 q_ptr = NEXT_Q_AFTER (q_ptr, stalls);
5184 clock_var += stalls;
5185 if (sched_verbose >= 2)
5186 fprintf (sched_dump, ";;\tAdvancing clock by %d cycle[s] to %d\n",
5187 stalls, clock_var);
5191 /* Used by early_queue_to_ready. Determines whether it is "ok" to
5192 prematurely move INSN from the queue to the ready list. Currently,
5193 if a target defines the hook 'is_costly_dependence', this function
5194 uses the hook to check whether there exist any dependences which are
5195 considered costly by the target, between INSN and other insns that
5196 have already been scheduled. Dependences are checked up to Y cycles
5197 back, with default Y=1; The flag -fsched-stalled-insns-dep=Y allows
5198 controlling this value.
5199 (Other considerations could be taken into account instead (or in
5200 addition) depending on user flags and target hooks. */
5202 static bool
5203 ok_for_early_queue_removal (rtx_insn *insn)
5205 if (targetm.sched.is_costly_dependence)
5207 int n_cycles;
5208 int i = scheduled_insns.length ();
5209 for (n_cycles = flag_sched_stalled_insns_dep; n_cycles; n_cycles--)
5211 while (i-- > 0)
5213 int cost;
5215 rtx_insn *prev_insn = scheduled_insns[i];
5217 if (!NOTE_P (prev_insn))
5219 dep_t dep;
5221 dep = sd_find_dep_between (prev_insn, insn, true);
5223 if (dep != NULL)
5225 cost = dep_cost (dep);
5227 if (targetm.sched.is_costly_dependence (dep, cost,
5228 flag_sched_stalled_insns_dep - n_cycles))
5229 return false;
5233 if (GET_MODE (prev_insn) == TImode) /* end of dispatch group */
5234 break;
5237 if (i == 0)
5238 break;
5242 return true;
5246 /* Remove insns from the queue, before they become "ready" with respect
5247 to FU latency considerations. */
5249 static int
5250 early_queue_to_ready (state_t state, struct ready_list *ready)
5252 rtx_insn *insn;
5253 rtx_insn_list *link;
5254 rtx_insn_list *next_link;
5255 rtx_insn_list *prev_link;
5256 bool move_to_ready;
5257 int cost;
5258 state_t temp_state = alloca (dfa_state_size);
5259 int stalls;
5260 int insns_removed = 0;
5263 Flag '-fsched-stalled-insns=X' determines the aggressiveness of this
5264 function:
5266 X == 0: There is no limit on how many queued insns can be removed
5267 prematurely. (flag_sched_stalled_insns = -1).
5269 X >= 1: Only X queued insns can be removed prematurely in each
5270 invocation. (flag_sched_stalled_insns = X).
5272 Otherwise: Early queue removal is disabled.
5273 (flag_sched_stalled_insns = 0)
5276 if (! flag_sched_stalled_insns)
5277 return 0;
5279 for (stalls = 0; stalls <= max_insn_queue_index; stalls++)
5281 if ((link = insn_queue[NEXT_Q_AFTER (q_ptr, stalls)]))
5283 if (sched_verbose > 6)
5284 fprintf (sched_dump, ";; look at index %d + %d\n", q_ptr, stalls);
5286 prev_link = 0;
5287 while (link)
5289 next_link = link->next ();
5290 insn = link->insn ();
5291 if (insn && sched_verbose > 6)
5292 print_rtl_single (sched_dump, insn);
5294 memcpy (temp_state, state, dfa_state_size);
5295 if (recog_memoized (insn) < 0)
5296 /* non-negative to indicate that it's not ready
5297 to avoid infinite Q->R->Q->R... */
5298 cost = 0;
5299 else
5300 cost = state_transition (temp_state, insn);
5302 if (sched_verbose >= 6)
5303 fprintf (sched_dump, "transition cost = %d\n", cost);
5305 move_to_ready = false;
5306 if (cost < 0)
5308 move_to_ready = ok_for_early_queue_removal (insn);
5309 if (move_to_ready == true)
5311 /* move from Q to R */
5312 q_size -= 1;
5313 ready_add (ready, insn, false);
5315 if (prev_link)
5316 XEXP (prev_link, 1) = next_link;
5317 else
5318 insn_queue[NEXT_Q_AFTER (q_ptr, stalls)] = next_link;
5320 free_INSN_LIST_node (link);
5322 if (sched_verbose >= 2)
5323 fprintf (sched_dump, ";;\t\tEarly Q-->Ready: insn %s\n",
5324 (*current_sched_info->print_insn) (insn, 0));
5326 insns_removed++;
5327 if (insns_removed == flag_sched_stalled_insns)
5328 /* Remove no more than flag_sched_stalled_insns insns
5329 from Q at a time. */
5330 return insns_removed;
5334 if (move_to_ready == false)
5335 prev_link = link;
5337 link = next_link;
5338 } /* while link */
5339 } /* if link */
5341 } /* for stalls.. */
5343 return insns_removed;
5347 /* Print the ready list for debugging purposes.
5348 If READY_TRY is non-zero then only print insns that max_issue
5349 will consider. */
5350 static void
5351 debug_ready_list_1 (struct ready_list *ready, signed char *ready_try)
5353 rtx_insn **p;
5354 int i;
5356 if (ready->n_ready == 0)
5358 fprintf (sched_dump, "\n");
5359 return;
5362 p = ready_lastpos (ready);
5363 for (i = 0; i < ready->n_ready; i++)
5365 if (ready_try != NULL && ready_try[ready->n_ready - i - 1])
5366 continue;
5368 fprintf (sched_dump, " %s:%d",
5369 (*current_sched_info->print_insn) (p[i], 0),
5370 INSN_LUID (p[i]));
5371 if (sched_pressure != SCHED_PRESSURE_NONE)
5372 fprintf (sched_dump, "(cost=%d",
5373 INSN_REG_PRESSURE_EXCESS_COST_CHANGE (p[i]));
5374 fprintf (sched_dump, ":prio=%d", INSN_PRIORITY (p[i]));
5375 if (INSN_TICK (p[i]) > clock_var)
5376 fprintf (sched_dump, ":delay=%d", INSN_TICK (p[i]) - clock_var);
5377 if (sched_pressure == SCHED_PRESSURE_MODEL)
5378 fprintf (sched_dump, ":idx=%d",
5379 model_index (p[i]));
5380 if (sched_pressure != SCHED_PRESSURE_NONE)
5381 fprintf (sched_dump, ")");
5383 fprintf (sched_dump, "\n");
5386 /* Print the ready list. Callable from debugger. */
5387 static void
5388 debug_ready_list (struct ready_list *ready)
5390 debug_ready_list_1 (ready, NULL);
5393 /* Search INSN for REG_SAVE_NOTE notes and convert them back into insn
5394 NOTEs. This is used for NOTE_INSN_EPILOGUE_BEG, so that sched-ebb
5395 replaces the epilogue note in the correct basic block. */
5396 void
5397 reemit_notes (rtx_insn *insn)
5399 rtx note;
5400 rtx_insn *last = insn;
5402 for (note = REG_NOTES (insn); note; note = XEXP (note, 1))
5404 if (REG_NOTE_KIND (note) == REG_SAVE_NOTE)
5406 enum insn_note note_type = (enum insn_note) INTVAL (XEXP (note, 0));
5408 last = emit_note_before (note_type, last);
5409 remove_note (insn, note);
5414 /* Move INSN. Reemit notes if needed. Update CFG, if needed. */
5415 static void
5416 move_insn (rtx_insn *insn, rtx_insn *last, rtx nt)
5418 if (PREV_INSN (insn) != last)
5420 basic_block bb;
5421 rtx_insn *note;
5422 int jump_p = 0;
5424 bb = BLOCK_FOR_INSN (insn);
5426 /* BB_HEAD is either LABEL or NOTE. */
5427 gcc_assert (BB_HEAD (bb) != insn);
5429 if (BB_END (bb) == insn)
5430 /* If this is last instruction in BB, move end marker one
5431 instruction up. */
5433 /* Jumps are always placed at the end of basic block. */
5434 jump_p = control_flow_insn_p (insn);
5436 gcc_assert (!jump_p
5437 || ((common_sched_info->sched_pass_id == SCHED_RGN_PASS)
5438 && IS_SPECULATION_BRANCHY_CHECK_P (insn))
5439 || (common_sched_info->sched_pass_id
5440 == SCHED_EBB_PASS));
5442 gcc_assert (BLOCK_FOR_INSN (PREV_INSN (insn)) == bb);
5444 BB_END (bb) = PREV_INSN (insn);
5447 gcc_assert (BB_END (bb) != last);
5449 if (jump_p)
5450 /* We move the block note along with jump. */
5452 gcc_assert (nt);
5454 note = NEXT_INSN (insn);
5455 while (NOTE_NOT_BB_P (note) && note != nt)
5456 note = NEXT_INSN (note);
5458 if (note != nt
5459 && (LABEL_P (note)
5460 || BARRIER_P (note)))
5461 note = NEXT_INSN (note);
5463 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note));
5465 else
5466 note = insn;
5468 SET_NEXT_INSN (PREV_INSN (insn)) = NEXT_INSN (note);
5469 SET_PREV_INSN (NEXT_INSN (note)) = PREV_INSN (insn);
5471 SET_NEXT_INSN (note) = NEXT_INSN (last);
5472 SET_PREV_INSN (NEXT_INSN (last)) = note;
5474 SET_NEXT_INSN (last) = insn;
5475 SET_PREV_INSN (insn) = last;
5477 bb = BLOCK_FOR_INSN (last);
5479 if (jump_p)
5481 fix_jump_move (insn);
5483 if (BLOCK_FOR_INSN (insn) != bb)
5484 move_block_after_check (insn);
5486 gcc_assert (BB_END (bb) == last);
5489 df_insn_change_bb (insn, bb);
5491 /* Update BB_END, if needed. */
5492 if (BB_END (bb) == last)
5493 BB_END (bb) = insn;
5496 SCHED_GROUP_P (insn) = 0;
5499 /* Return true if scheduling INSN will finish current clock cycle. */
5500 static bool
5501 insn_finishes_cycle_p (rtx_insn *insn)
5503 if (SCHED_GROUP_P (insn))
5504 /* After issuing INSN, rest of the sched_group will be forced to issue
5505 in order. Don't make any plans for the rest of cycle. */
5506 return true;
5508 /* Finishing the block will, apparently, finish the cycle. */
5509 if (current_sched_info->insn_finishes_block_p
5510 && current_sched_info->insn_finishes_block_p (insn))
5511 return true;
5513 return false;
5516 /* Helper for autopref_multipass_init. Given a SET in PAT and whether
5517 we're expecting a memory WRITE or not, check that the insn is relevant to
5518 the autoprefetcher modelling code. Return true iff that is the case.
5519 If it is relevant, record the base register of the memory op in BASE and
5520 the offset in OFFSET. */
5522 static bool
5523 analyze_set_insn_for_autopref (rtx pat, bool write, rtx *base, int *offset)
5525 if (GET_CODE (pat) != SET)
5526 return false;
5528 rtx mem = write ? SET_DEST (pat) : SET_SRC (pat);
5529 if (!MEM_P (mem))
5530 return false;
5532 struct address_info info;
5533 decompose_mem_address (&info, mem);
5535 /* TODO: Currently only (base+const) addressing is supported. */
5536 if (info.base == NULL || !REG_P (*info.base)
5537 || (info.disp != NULL && !CONST_INT_P (*info.disp)))
5538 return false;
5540 *base = *info.base;
5541 *offset = info.disp ? INTVAL (*info.disp) : 0;
5542 return true;
5545 /* Functions to model cache auto-prefetcher.
5547 Some of the CPUs have cache auto-prefetcher, which /seems/ to initiate
5548 memory prefetches if it sees instructions with consequitive memory accesses
5549 in the instruction stream. Details of such hardware units are not published,
5550 so we can only guess what exactly is going on there.
5551 In the scheduler, we model abstract auto-prefetcher. If there are memory
5552 insns in the ready list (or the queue) that have same memory base, but
5553 different offsets, then we delay the insns with larger offsets until insns
5554 with smaller offsets get scheduled. If PARAM_SCHED_AUTOPREF_QUEUE_DEPTH
5555 is "1", then we look at the ready list; if it is N>1, then we also look
5556 through N-1 queue entries.
5557 If the param is N>=0, then rank_for_schedule will consider auto-prefetching
5558 among its heuristics.
5559 Param value of "-1" disables modelling of the auto-prefetcher. */
5561 /* Initialize autoprefetcher model data for INSN. */
5562 static void
5563 autopref_multipass_init (const rtx_insn *insn, int write)
5565 autopref_multipass_data_t data = &INSN_AUTOPREF_MULTIPASS_DATA (insn)[write];
5567 gcc_assert (data->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED);
5568 data->base = NULL_RTX;
5569 data->min_offset = 0;
5570 data->max_offset = 0;
5571 data->multi_mem_insn_p = false;
5572 /* Set insn entry initialized, but not relevant for auto-prefetcher. */
5573 data->status = AUTOPREF_MULTIPASS_DATA_IRRELEVANT;
5575 rtx pat = PATTERN (insn);
5577 /* We have a multi-set insn like a load-multiple or store-multiple.
5578 We care about these as long as all the memory ops inside the PARALLEL
5579 have the same base register. We care about the minimum and maximum
5580 offsets from that base but don't check for the order of those offsets
5581 within the PARALLEL insn itself. */
5582 if (GET_CODE (pat) == PARALLEL)
5584 int n_elems = XVECLEN (pat, 0);
5586 int i = 0;
5587 rtx prev_base = NULL_RTX;
5588 int min_offset = 0;
5589 int max_offset = 0;
5591 for (i = 0; i < n_elems; i++)
5593 rtx set = XVECEXP (pat, 0, i);
5594 if (GET_CODE (set) != SET)
5595 return;
5597 rtx base = NULL_RTX;
5598 int offset = 0;
5599 if (!analyze_set_insn_for_autopref (set, write, &base, &offset))
5600 return;
5602 if (i == 0)
5604 prev_base = base;
5605 min_offset = offset;
5606 max_offset = offset;
5608 /* Ensure that all memory operations in the PARALLEL use the same
5609 base register. */
5610 else if (REGNO (base) != REGNO (prev_base))
5611 return;
5612 else
5614 min_offset = MIN (min_offset, offset);
5615 max_offset = MAX (max_offset, offset);
5619 /* If we reached here then we have a valid PARALLEL of multiple memory
5620 ops with prev_base as the base and min_offset and max_offset
5621 containing the offsets range. */
5622 gcc_assert (prev_base);
5623 data->base = prev_base;
5624 data->min_offset = min_offset;
5625 data->max_offset = max_offset;
5626 data->multi_mem_insn_p = true;
5627 data->status = AUTOPREF_MULTIPASS_DATA_NORMAL;
5629 return;
5632 /* Otherwise this is a single set memory operation. */
5633 rtx set = single_set (insn);
5634 if (set == NULL_RTX)
5635 return;
5637 if (!analyze_set_insn_for_autopref (set, write, &data->base,
5638 &data->min_offset))
5639 return;
5641 /* This insn is relevant for the auto-prefetcher.
5642 The base and offset fields will have been filled in the
5643 analyze_set_insn_for_autopref call above. */
5644 data->status = AUTOPREF_MULTIPASS_DATA_NORMAL;
5648 /* Helper for autopref_rank_for_schedule. Given the data of two
5649 insns relevant to the auto-prefetcher modelling code DATA1 and DATA2
5650 return their comparison result. Return 0 if there is no sensible
5651 ranking order for the two insns. */
5653 static int
5654 autopref_rank_data (autopref_multipass_data_t data1,
5655 autopref_multipass_data_t data2)
5657 /* Simple case when both insns are simple single memory ops. */
5658 if (!data1->multi_mem_insn_p && !data2->multi_mem_insn_p)
5659 return data1->min_offset - data2->min_offset;
5661 /* Two load/store multiple insns. Return 0 if the offset ranges
5662 overlap and the difference between the minimum offsets otherwise. */
5663 else if (data1->multi_mem_insn_p && data2->multi_mem_insn_p)
5665 int min1 = data1->min_offset;
5666 int max1 = data1->max_offset;
5667 int min2 = data2->min_offset;
5668 int max2 = data2->max_offset;
5670 if (max1 < min2 || min1 > max2)
5671 return min1 - min2;
5672 else
5673 return 0;
5676 /* The other two cases is a pair of a load/store multiple and
5677 a simple memory op. Return 0 if the single op's offset is within the
5678 range of the multi-op insn and the difference between the single offset
5679 and the minimum offset of the multi-set insn otherwise. */
5680 else if (data1->multi_mem_insn_p && !data2->multi_mem_insn_p)
5682 int max1 = data1->max_offset;
5683 int min1 = data1->min_offset;
5685 if (data2->min_offset >= min1
5686 && data2->min_offset <= max1)
5687 return 0;
5688 else
5689 return min1 - data2->min_offset;
5691 else
5693 int max2 = data2->max_offset;
5694 int min2 = data2->min_offset;
5696 if (data1->min_offset >= min2
5697 && data1->min_offset <= max2)
5698 return 0;
5699 else
5700 return data1->min_offset - min2;
5704 /* Helper function for rank_for_schedule sorting. */
5705 static int
5706 autopref_rank_for_schedule (const rtx_insn *insn1, const rtx_insn *insn2)
5708 for (int write = 0; write < 2; ++write)
5710 autopref_multipass_data_t data1
5711 = &INSN_AUTOPREF_MULTIPASS_DATA (insn1)[write];
5712 autopref_multipass_data_t data2
5713 = &INSN_AUTOPREF_MULTIPASS_DATA (insn2)[write];
5715 if (data1->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED)
5716 autopref_multipass_init (insn1, write);
5717 if (data1->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT)
5718 continue;
5720 if (data2->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED)
5721 autopref_multipass_init (insn2, write);
5722 if (data2->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT)
5723 continue;
5725 if (!rtx_equal_p (data1->base, data2->base))
5726 continue;
5728 return autopref_rank_data (data1, data2);
5731 return 0;
5734 /* True if header of debug dump was printed. */
5735 static bool autopref_multipass_dfa_lookahead_guard_started_dump_p;
5737 /* Helper for autopref_multipass_dfa_lookahead_guard.
5738 Return "1" if INSN1 should be delayed in favor of INSN2. */
5739 static int
5740 autopref_multipass_dfa_lookahead_guard_1 (const rtx_insn *insn1,
5741 const rtx_insn *insn2, int write)
5743 autopref_multipass_data_t data1
5744 = &INSN_AUTOPREF_MULTIPASS_DATA (insn1)[write];
5745 autopref_multipass_data_t data2
5746 = &INSN_AUTOPREF_MULTIPASS_DATA (insn2)[write];
5748 if (data2->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED)
5749 autopref_multipass_init (insn2, write);
5750 if (data2->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT)
5751 return 0;
5753 if (rtx_equal_p (data1->base, data2->base)
5754 && autopref_rank_data (data1, data2) > 0)
5756 if (sched_verbose >= 2)
5758 if (!autopref_multipass_dfa_lookahead_guard_started_dump_p)
5760 fprintf (sched_dump,
5761 ";;\t\tnot trying in max_issue due to autoprefetch "
5762 "model: ");
5763 autopref_multipass_dfa_lookahead_guard_started_dump_p = true;
5766 fprintf (sched_dump, " %d(%d)", INSN_UID (insn1), INSN_UID (insn2));
5769 return 1;
5772 return 0;
5775 /* General note:
5777 We could have also hooked autoprefetcher model into
5778 first_cycle_multipass_backtrack / first_cycle_multipass_issue hooks
5779 to enable intelligent selection of "[r1+0]=r2; [r1+4]=r3" on the same cycle
5780 (e.g., once "[r1+0]=r2" is issued in max_issue(), "[r1+4]=r3" gets
5781 unblocked). We don't bother about this yet because target of interest
5782 (ARM Cortex-A15) can issue only 1 memory operation per cycle. */
5784 /* Implementation of first_cycle_multipass_dfa_lookahead_guard hook.
5785 Return "1" if INSN1 should not be considered in max_issue due to
5786 auto-prefetcher considerations. */
5788 autopref_multipass_dfa_lookahead_guard (rtx_insn *insn1, int ready_index)
5790 int r = 0;
5792 /* Exit early if the param forbids this or if we're not entering here through
5793 normal haifa scheduling. This can happen if selective scheduling is
5794 explicitly enabled. */
5795 if (!insn_queue || PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) <= 0)
5796 return 0;
5798 if (sched_verbose >= 2 && ready_index == 0)
5799 autopref_multipass_dfa_lookahead_guard_started_dump_p = false;
5801 for (int write = 0; write < 2; ++write)
5803 autopref_multipass_data_t data1
5804 = &INSN_AUTOPREF_MULTIPASS_DATA (insn1)[write];
5806 if (data1->status == AUTOPREF_MULTIPASS_DATA_UNINITIALIZED)
5807 autopref_multipass_init (insn1, write);
5808 if (data1->status == AUTOPREF_MULTIPASS_DATA_IRRELEVANT)
5809 continue;
5811 if (ready_index == 0
5812 && data1->status == AUTOPREF_MULTIPASS_DATA_DONT_DELAY)
5813 /* We allow only a single delay on priviledged instructions.
5814 Doing otherwise would cause infinite loop. */
5816 if (sched_verbose >= 2)
5818 if (!autopref_multipass_dfa_lookahead_guard_started_dump_p)
5820 fprintf (sched_dump,
5821 ";;\t\tnot trying in max_issue due to autoprefetch "
5822 "model: ");
5823 autopref_multipass_dfa_lookahead_guard_started_dump_p = true;
5826 fprintf (sched_dump, " *%d*", INSN_UID (insn1));
5828 continue;
5831 for (int i2 = 0; i2 < ready.n_ready; ++i2)
5833 rtx_insn *insn2 = get_ready_element (i2);
5834 if (insn1 == insn2)
5835 continue;
5836 r = autopref_multipass_dfa_lookahead_guard_1 (insn1, insn2, write);
5837 if (r)
5839 if (ready_index == 0)
5841 r = -1;
5842 data1->status = AUTOPREF_MULTIPASS_DATA_DONT_DELAY;
5844 goto finish;
5848 if (PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) == 1)
5849 continue;
5851 /* Everything from the current queue slot should have been moved to
5852 the ready list. */
5853 gcc_assert (insn_queue[NEXT_Q_AFTER (q_ptr, 0)] == NULL_RTX);
5855 int n_stalls = PARAM_VALUE (PARAM_SCHED_AUTOPREF_QUEUE_DEPTH) - 1;
5856 if (n_stalls > max_insn_queue_index)
5857 n_stalls = max_insn_queue_index;
5859 for (int stalls = 1; stalls <= n_stalls; ++stalls)
5861 for (rtx_insn_list *link = insn_queue[NEXT_Q_AFTER (q_ptr, stalls)];
5862 link != NULL_RTX;
5863 link = link->next ())
5865 rtx_insn *insn2 = link->insn ();
5866 r = autopref_multipass_dfa_lookahead_guard_1 (insn1, insn2,
5867 write);
5868 if (r)
5870 /* Queue INSN1 until INSN2 can issue. */
5871 r = -stalls;
5872 if (ready_index == 0)
5873 data1->status = AUTOPREF_MULTIPASS_DATA_DONT_DELAY;
5874 goto finish;
5880 finish:
5881 if (sched_verbose >= 2
5882 && autopref_multipass_dfa_lookahead_guard_started_dump_p
5883 && (ready_index == ready.n_ready - 1 || r < 0))
5884 /* This does not /always/ trigger. We don't output EOL if the last
5885 insn is not recognized (INSN_CODE < 0) and lookahead_guard is not
5886 called. We can live with this. */
5887 fprintf (sched_dump, "\n");
5889 return r;
5892 /* Define type for target data used in multipass scheduling. */
5893 #ifndef TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DATA_T
5894 # define TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DATA_T int
5895 #endif
5896 typedef TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DATA_T first_cycle_multipass_data_t;
5898 /* The following structure describe an entry of the stack of choices. */
5899 struct choice_entry
5901 /* Ordinal number of the issued insn in the ready queue. */
5902 int index;
5903 /* The number of the rest insns whose issues we should try. */
5904 int rest;
5905 /* The number of issued essential insns. */
5906 int n;
5907 /* State after issuing the insn. */
5908 state_t state;
5909 /* Target-specific data. */
5910 first_cycle_multipass_data_t target_data;
5913 /* The following array is used to implement a stack of choices used in
5914 function max_issue. */
5915 static struct choice_entry *choice_stack;
5917 /* This holds the value of the target dfa_lookahead hook. */
5918 int dfa_lookahead;
5920 /* The following variable value is maximal number of tries of issuing
5921 insns for the first cycle multipass insn scheduling. We define
5922 this value as constant*(DFA_LOOKAHEAD**ISSUE_RATE). We would not
5923 need this constraint if all real insns (with non-negative codes)
5924 had reservations because in this case the algorithm complexity is
5925 O(DFA_LOOKAHEAD**ISSUE_RATE). Unfortunately, the dfa descriptions
5926 might be incomplete and such insn might occur. For such
5927 descriptions, the complexity of algorithm (without the constraint)
5928 could achieve DFA_LOOKAHEAD ** N , where N is the queue length. */
5929 static int max_lookahead_tries;
5931 /* The following function returns maximal (or close to maximal) number
5932 of insns which can be issued on the same cycle and one of which
5933 insns is insns with the best rank (the first insn in READY). To
5934 make this function tries different samples of ready insns. READY
5935 is current queue `ready'. Global array READY_TRY reflects what
5936 insns are already issued in this try. The function stops immediately,
5937 if it reached the such a solution, that all instruction can be issued.
5938 INDEX will contain index of the best insn in READY. The following
5939 function is used only for first cycle multipass scheduling.
5941 PRIVILEGED_N >= 0
5943 This function expects recognized insns only. All USEs,
5944 CLOBBERs, etc must be filtered elsewhere. */
5946 max_issue (struct ready_list *ready, int privileged_n, state_t state,
5947 bool first_cycle_insn_p, int *index)
5949 int n, i, all, n_ready, best, delay, tries_num;
5950 int more_issue;
5951 struct choice_entry *top;
5952 rtx_insn *insn;
5954 if (sched_fusion)
5955 return 0;
5957 n_ready = ready->n_ready;
5958 gcc_assert (dfa_lookahead >= 1 && privileged_n >= 0
5959 && privileged_n <= n_ready);
5961 /* Init MAX_LOOKAHEAD_TRIES. */
5962 if (max_lookahead_tries == 0)
5964 max_lookahead_tries = 100;
5965 for (i = 0; i < issue_rate; i++)
5966 max_lookahead_tries *= dfa_lookahead;
5969 /* Init max_points. */
5970 more_issue = issue_rate - cycle_issued_insns;
5971 gcc_assert (more_issue >= 0);
5973 /* The number of the issued insns in the best solution. */
5974 best = 0;
5976 top = choice_stack;
5978 /* Set initial state of the search. */
5979 memcpy (top->state, state, dfa_state_size);
5980 top->rest = dfa_lookahead;
5981 top->n = 0;
5982 if (targetm.sched.first_cycle_multipass_begin)
5983 targetm.sched.first_cycle_multipass_begin (&top->target_data,
5984 ready_try, n_ready,
5985 first_cycle_insn_p);
5987 /* Count the number of the insns to search among. */
5988 for (all = i = 0; i < n_ready; i++)
5989 if (!ready_try [i])
5990 all++;
5992 if (sched_verbose >= 2)
5994 fprintf (sched_dump, ";;\t\tmax_issue among %d insns:", all);
5995 debug_ready_list_1 (ready, ready_try);
5998 /* I is the index of the insn to try next. */
5999 i = 0;
6000 tries_num = 0;
6001 for (;;)
6003 if (/* If we've reached a dead end or searched enough of what we have
6004 been asked... */
6005 top->rest == 0
6006 /* or have nothing else to try... */
6007 || i >= n_ready
6008 /* or should not issue more. */
6009 || top->n >= more_issue)
6011 /* ??? (... || i == n_ready). */
6012 gcc_assert (i <= n_ready);
6014 /* We should not issue more than issue_rate instructions. */
6015 gcc_assert (top->n <= more_issue);
6017 if (top == choice_stack)
6018 break;
6020 if (best < top - choice_stack)
6022 if (privileged_n)
6024 n = privileged_n;
6025 /* Try to find issued privileged insn. */
6026 while (n && !ready_try[--n])
6030 if (/* If all insns are equally good... */
6031 privileged_n == 0
6032 /* Or a privileged insn will be issued. */
6033 || ready_try[n])
6034 /* Then we have a solution. */
6036 best = top - choice_stack;
6037 /* This is the index of the insn issued first in this
6038 solution. */
6039 *index = choice_stack [1].index;
6040 if (top->n == more_issue || best == all)
6041 break;
6045 /* Set ready-list index to point to the last insn
6046 ('i++' below will advance it to the next insn). */
6047 i = top->index;
6049 /* Backtrack. */
6050 ready_try [i] = 0;
6052 if (targetm.sched.first_cycle_multipass_backtrack)
6053 targetm.sched.first_cycle_multipass_backtrack (&top->target_data,
6054 ready_try, n_ready);
6056 top--;
6057 memcpy (state, top->state, dfa_state_size);
6059 else if (!ready_try [i])
6061 tries_num++;
6062 if (tries_num > max_lookahead_tries)
6063 break;
6064 insn = ready_element (ready, i);
6065 delay = state_transition (state, insn);
6066 if (delay < 0)
6068 if (state_dead_lock_p (state)
6069 || insn_finishes_cycle_p (insn))
6070 /* We won't issue any more instructions in the next
6071 choice_state. */
6072 top->rest = 0;
6073 else
6074 top->rest--;
6076 n = top->n;
6077 if (memcmp (top->state, state, dfa_state_size) != 0)
6078 n++;
6080 /* Advance to the next choice_entry. */
6081 top++;
6082 /* Initialize it. */
6083 top->rest = dfa_lookahead;
6084 top->index = i;
6085 top->n = n;
6086 memcpy (top->state, state, dfa_state_size);
6087 ready_try [i] = 1;
6089 if (targetm.sched.first_cycle_multipass_issue)
6090 targetm.sched.first_cycle_multipass_issue (&top->target_data,
6091 ready_try, n_ready,
6092 insn,
6093 &((top - 1)
6094 ->target_data));
6096 i = -1;
6100 /* Increase ready-list index. */
6101 i++;
6104 if (targetm.sched.first_cycle_multipass_end)
6105 targetm.sched.first_cycle_multipass_end (best != 0
6106 ? &choice_stack[1].target_data
6107 : NULL);
6109 /* Restore the original state of the DFA. */
6110 memcpy (state, choice_stack->state, dfa_state_size);
6112 return best;
6115 /* The following function chooses insn from READY and modifies
6116 READY. The following function is used only for first
6117 cycle multipass scheduling.
6118 Return:
6119 -1 if cycle should be advanced,
6120 0 if INSN_PTR is set to point to the desirable insn,
6121 1 if choose_ready () should be restarted without advancing the cycle. */
6122 static int
6123 choose_ready (struct ready_list *ready, bool first_cycle_insn_p,
6124 rtx_insn **insn_ptr)
6126 if (dbg_cnt (sched_insn) == false)
6128 if (nonscheduled_insns_begin == NULL_RTX)
6129 nonscheduled_insns_begin = current_sched_info->prev_head;
6131 rtx_insn *insn = first_nonscheduled_insn ();
6133 if (QUEUE_INDEX (insn) == QUEUE_READY)
6134 /* INSN is in the ready_list. */
6136 ready_remove_insn (insn);
6137 *insn_ptr = insn;
6138 return 0;
6141 /* INSN is in the queue. Advance cycle to move it to the ready list. */
6142 gcc_assert (QUEUE_INDEX (insn) >= 0);
6143 return -1;
6146 if (dfa_lookahead <= 0 || SCHED_GROUP_P (ready_element (ready, 0))
6147 || DEBUG_INSN_P (ready_element (ready, 0)))
6149 if (targetm.sched.dispatch (NULL, IS_DISPATCH_ON))
6150 *insn_ptr = ready_remove_first_dispatch (ready);
6151 else
6152 *insn_ptr = ready_remove_first (ready);
6154 return 0;
6156 else
6158 /* Try to choose the best insn. */
6159 int index = 0, i;
6160 rtx_insn *insn;
6162 insn = ready_element (ready, 0);
6163 if (INSN_CODE (insn) < 0)
6165 *insn_ptr = ready_remove_first (ready);
6166 return 0;
6169 /* Filter the search space. */
6170 for (i = 0; i < ready->n_ready; i++)
6172 ready_try[i] = 0;
6174 insn = ready_element (ready, i);
6176 /* If this insn is recognizable we should have already
6177 recognized it earlier.
6178 ??? Not very clear where this is supposed to be done.
6179 See dep_cost_1. */
6180 gcc_checking_assert (INSN_CODE (insn) >= 0
6181 || recog_memoized (insn) < 0);
6182 if (INSN_CODE (insn) < 0)
6184 /* Non-recognized insns at position 0 are handled above. */
6185 gcc_assert (i > 0);
6186 ready_try[i] = 1;
6187 continue;
6190 if (targetm.sched.first_cycle_multipass_dfa_lookahead_guard)
6192 ready_try[i]
6193 = (targetm.sched.first_cycle_multipass_dfa_lookahead_guard
6194 (insn, i));
6196 if (ready_try[i] < 0)
6197 /* Queue instruction for several cycles.
6198 We need to restart choose_ready as we have changed
6199 the ready list. */
6201 change_queue_index (insn, -ready_try[i]);
6202 return 1;
6205 /* Make sure that we didn't end up with 0'th insn filtered out.
6206 Don't be tempted to make life easier for backends and just
6207 requeue 0'th insn if (ready_try[0] == 0) and restart
6208 choose_ready. Backends should be very considerate about
6209 requeueing instructions -- especially the highest priority
6210 one at position 0. */
6211 gcc_assert (ready_try[i] == 0 || i > 0);
6212 if (ready_try[i])
6213 continue;
6216 gcc_assert (ready_try[i] == 0);
6217 /* INSN made it through the scrutiny of filters! */
6220 if (max_issue (ready, 1, curr_state, first_cycle_insn_p, &index) == 0)
6222 *insn_ptr = ready_remove_first (ready);
6223 if (sched_verbose >= 4)
6224 fprintf (sched_dump, ";;\t\tChosen insn (but can't issue) : %s \n",
6225 (*current_sched_info->print_insn) (*insn_ptr, 0));
6226 return 0;
6228 else
6230 if (sched_verbose >= 4)
6231 fprintf (sched_dump, ";;\t\tChosen insn : %s\n",
6232 (*current_sched_info->print_insn)
6233 (ready_element (ready, index), 0));
6235 *insn_ptr = ready_remove (ready, index);
6236 return 0;
6241 /* This function is called when we have successfully scheduled a
6242 block. It uses the schedule stored in the scheduled_insns vector
6243 to rearrange the RTL. PREV_HEAD is used as the anchor to which we
6244 append the scheduled insns; TAIL is the insn after the scheduled
6245 block. TARGET_BB is the argument passed to schedule_block. */
6247 static void
6248 commit_schedule (rtx_insn *prev_head, rtx_insn *tail, basic_block *target_bb)
6250 unsigned int i;
6251 rtx_insn *insn;
6253 last_scheduled_insn = prev_head;
6254 for (i = 0;
6255 scheduled_insns.iterate (i, &insn);
6256 i++)
6258 if (control_flow_insn_p (last_scheduled_insn)
6259 || current_sched_info->advance_target_bb (*target_bb, insn))
6261 *target_bb = current_sched_info->advance_target_bb (*target_bb, 0);
6263 if (sched_verbose)
6265 rtx_insn *x;
6267 x = next_real_insn (last_scheduled_insn);
6268 gcc_assert (x);
6269 dump_new_block_header (1, *target_bb, x, tail);
6272 last_scheduled_insn = bb_note (*target_bb);
6275 if (current_sched_info->begin_move_insn)
6276 (*current_sched_info->begin_move_insn) (insn, last_scheduled_insn);
6277 move_insn (insn, last_scheduled_insn,
6278 current_sched_info->next_tail);
6279 if (!DEBUG_INSN_P (insn))
6280 reemit_notes (insn);
6281 last_scheduled_insn = insn;
6284 scheduled_insns.truncate (0);
6287 /* Examine all insns on the ready list and queue those which can't be
6288 issued in this cycle. TEMP_STATE is temporary scheduler state we
6289 can use as scratch space. If FIRST_CYCLE_INSN_P is true, no insns
6290 have been issued for the current cycle, which means it is valid to
6291 issue an asm statement.
6293 If SHADOWS_ONLY_P is true, we eliminate all real insns and only
6294 leave those for which SHADOW_P is true. If MODULO_EPILOGUE is true,
6295 we only leave insns which have an INSN_EXACT_TICK. */
6297 static void
6298 prune_ready_list (state_t temp_state, bool first_cycle_insn_p,
6299 bool shadows_only_p, bool modulo_epilogue_p)
6301 int i, pass;
6302 bool sched_group_found = false;
6303 int min_cost_group = 1;
6305 if (sched_fusion)
6306 return;
6308 for (i = 0; i < ready.n_ready; i++)
6310 rtx_insn *insn = ready_element (&ready, i);
6311 if (SCHED_GROUP_P (insn))
6313 sched_group_found = true;
6314 break;
6318 /* Make two passes if there's a SCHED_GROUP_P insn; make sure to handle
6319 such an insn first and note its cost, then schedule all other insns
6320 for one cycle later. */
6321 for (pass = sched_group_found ? 0 : 1; pass < 2; )
6323 int n = ready.n_ready;
6324 for (i = 0; i < n; i++)
6326 rtx_insn *insn = ready_element (&ready, i);
6327 int cost = 0;
6328 const char *reason = "resource conflict";
6330 if (DEBUG_INSN_P (insn))
6331 continue;
6333 if (sched_group_found && !SCHED_GROUP_P (insn))
6335 if (pass == 0)
6336 continue;
6337 cost = min_cost_group;
6338 reason = "not in sched group";
6340 else if (modulo_epilogue_p
6341 && INSN_EXACT_TICK (insn) == INVALID_TICK)
6343 cost = max_insn_queue_index;
6344 reason = "not an epilogue insn";
6346 else if (shadows_only_p && !SHADOW_P (insn))
6348 cost = 1;
6349 reason = "not a shadow";
6351 else if (recog_memoized (insn) < 0)
6353 if (!first_cycle_insn_p
6354 && (GET_CODE (PATTERN (insn)) == ASM_INPUT
6355 || asm_noperands (PATTERN (insn)) >= 0))
6356 cost = 1;
6357 reason = "asm";
6359 else if (sched_pressure != SCHED_PRESSURE_NONE)
6361 if (sched_pressure == SCHED_PRESSURE_MODEL
6362 && INSN_TICK (insn) <= clock_var)
6364 memcpy (temp_state, curr_state, dfa_state_size);
6365 if (state_transition (temp_state, insn) >= 0)
6366 INSN_TICK (insn) = clock_var + 1;
6368 cost = 0;
6370 else
6372 int delay_cost = 0;
6374 if (delay_htab)
6376 struct delay_pair *delay_entry;
6377 delay_entry
6378 = delay_htab->find_with_hash (insn,
6379 htab_hash_pointer (insn));
6380 while (delay_entry && delay_cost == 0)
6382 delay_cost = estimate_shadow_tick (delay_entry);
6383 if (delay_cost > max_insn_queue_index)
6384 delay_cost = max_insn_queue_index;
6385 delay_entry = delay_entry->next_same_i1;
6389 memcpy (temp_state, curr_state, dfa_state_size);
6390 cost = state_transition (temp_state, insn);
6391 if (cost < 0)
6392 cost = 0;
6393 else if (cost == 0)
6394 cost = 1;
6395 if (cost < delay_cost)
6397 cost = delay_cost;
6398 reason = "shadow tick";
6401 if (cost >= 1)
6403 if (SCHED_GROUP_P (insn) && cost > min_cost_group)
6404 min_cost_group = cost;
6405 ready_remove (&ready, i);
6406 /* Normally we'd want to queue INSN for COST cycles. However,
6407 if SCHED_GROUP_P is set, then we must ensure that nothing
6408 else comes between INSN and its predecessor. If there is
6409 some other insn ready to fire on the next cycle, then that
6410 invariant would be broken.
6412 So when SCHED_GROUP_P is set, just queue this insn for a
6413 single cycle. */
6414 queue_insn (insn, SCHED_GROUP_P (insn) ? 1 : cost, reason);
6415 if (i + 1 < n)
6416 break;
6419 if (i == n)
6420 pass++;
6424 /* Called when we detect that the schedule is impossible. We examine the
6425 backtrack queue to find the earliest insn that caused this condition. */
6427 static struct haifa_saved_data *
6428 verify_shadows (void)
6430 struct haifa_saved_data *save, *earliest_fail = NULL;
6431 for (save = backtrack_queue; save; save = save->next)
6433 int t;
6434 struct delay_pair *pair = save->delay_pair;
6435 rtx_insn *i1 = pair->i1;
6437 for (; pair; pair = pair->next_same_i1)
6439 rtx_insn *i2 = pair->i2;
6441 if (QUEUE_INDEX (i2) == QUEUE_SCHEDULED)
6442 continue;
6444 t = INSN_TICK (i1) + pair_delay (pair);
6445 if (t < clock_var)
6447 if (sched_verbose >= 2)
6448 fprintf (sched_dump,
6449 ";;\t\tfailed delay requirements for %d/%d (%d->%d)"
6450 ", not ready\n",
6451 INSN_UID (pair->i1), INSN_UID (pair->i2),
6452 INSN_TICK (pair->i1), INSN_EXACT_TICK (pair->i2));
6453 earliest_fail = save;
6454 break;
6456 if (QUEUE_INDEX (i2) >= 0)
6458 int queued_for = INSN_TICK (i2);
6460 if (t < queued_for)
6462 if (sched_verbose >= 2)
6463 fprintf (sched_dump,
6464 ";;\t\tfailed delay requirements for %d/%d"
6465 " (%d->%d), queued too late\n",
6466 INSN_UID (pair->i1), INSN_UID (pair->i2),
6467 INSN_TICK (pair->i1), INSN_EXACT_TICK (pair->i2));
6468 earliest_fail = save;
6469 break;
6475 return earliest_fail;
6478 /* Print instructions together with useful scheduling information between
6479 HEAD and TAIL (inclusive). */
6480 static void
6481 dump_insn_stream (rtx_insn *head, rtx_insn *tail)
6483 fprintf (sched_dump, ";;\t| insn | prio |\n");
6485 rtx_insn *next_tail = NEXT_INSN (tail);
6486 for (rtx_insn *insn = head; insn != next_tail; insn = NEXT_INSN (insn))
6488 int priority = NOTE_P (insn) ? 0 : INSN_PRIORITY (insn);
6489 const char *pattern = (NOTE_P (insn)
6490 ? "note"
6491 : str_pattern_slim (PATTERN (insn)));
6493 fprintf (sched_dump, ";;\t| %4d | %4d | %-30s ",
6494 INSN_UID (insn), priority, pattern);
6496 if (sched_verbose >= 4)
6498 if (NOTE_P (insn) || LABEL_P (insn) || recog_memoized (insn) < 0)
6499 fprintf (sched_dump, "nothing");
6500 else
6501 print_reservation (sched_dump, insn);
6503 fprintf (sched_dump, "\n");
6507 /* Use forward list scheduling to rearrange insns of block pointed to by
6508 TARGET_BB, possibly bringing insns from subsequent blocks in the same
6509 region. */
6511 bool
6512 schedule_block (basic_block *target_bb, state_t init_state)
6514 int i;
6515 bool success = modulo_ii == 0;
6516 struct sched_block_state ls;
6517 state_t temp_state = NULL; /* It is used for multipass scheduling. */
6518 int sort_p, advance, start_clock_var;
6520 /* Head/tail info for this block. */
6521 rtx_insn *prev_head = current_sched_info->prev_head;
6522 rtx_insn *next_tail = current_sched_info->next_tail;
6523 rtx_insn *head = NEXT_INSN (prev_head);
6524 rtx_insn *tail = PREV_INSN (next_tail);
6526 if ((current_sched_info->flags & DONT_BREAK_DEPENDENCIES) == 0
6527 && sched_pressure != SCHED_PRESSURE_MODEL && !sched_fusion)
6528 find_modifiable_mems (head, tail);
6530 /* We used to have code to avoid getting parameters moved from hard
6531 argument registers into pseudos.
6533 However, it was removed when it proved to be of marginal benefit
6534 and caused problems because schedule_block and compute_forward_dependences
6535 had different notions of what the "head" insn was. */
6537 gcc_assert (head != tail || INSN_P (head));
6539 haifa_recovery_bb_recently_added_p = false;
6541 backtrack_queue = NULL;
6543 /* Debug info. */
6544 if (sched_verbose)
6546 dump_new_block_header (0, *target_bb, head, tail);
6548 if (sched_verbose >= 2)
6550 dump_insn_stream (head, tail);
6551 memset (&rank_for_schedule_stats, 0,
6552 sizeof (rank_for_schedule_stats));
6556 if (init_state == NULL)
6557 state_reset (curr_state);
6558 else
6559 memcpy (curr_state, init_state, dfa_state_size);
6561 /* Clear the ready list. */
6562 ready.first = ready.veclen - 1;
6563 ready.n_ready = 0;
6564 ready.n_debug = 0;
6566 /* It is used for first cycle multipass scheduling. */
6567 temp_state = alloca (dfa_state_size);
6569 if (targetm.sched.init)
6570 targetm.sched.init (sched_dump, sched_verbose, ready.veclen);
6572 /* We start inserting insns after PREV_HEAD. */
6573 last_scheduled_insn = prev_head;
6574 last_nondebug_scheduled_insn = NULL;
6575 nonscheduled_insns_begin = NULL;
6577 gcc_assert ((NOTE_P (last_scheduled_insn)
6578 || DEBUG_INSN_P (last_scheduled_insn))
6579 && BLOCK_FOR_INSN (last_scheduled_insn) == *target_bb);
6581 /* Initialize INSN_QUEUE. Q_SIZE is the total number of insns in the
6582 queue. */
6583 q_ptr = 0;
6584 q_size = 0;
6586 insn_queue = XALLOCAVEC (rtx_insn_list *, max_insn_queue_index + 1);
6587 memset (insn_queue, 0, (max_insn_queue_index + 1) * sizeof (rtx));
6589 /* Start just before the beginning of time. */
6590 clock_var = -1;
6592 /* We need queue and ready lists and clock_var be initialized
6593 in try_ready () (which is called through init_ready_list ()). */
6594 (*current_sched_info->init_ready_list) ();
6596 if (sched_pressure)
6597 sched_pressure_start_bb (*target_bb);
6599 /* The algorithm is O(n^2) in the number of ready insns at any given
6600 time in the worst case. Before reload we are more likely to have
6601 big lists so truncate them to a reasonable size. */
6602 if (!reload_completed
6603 && ready.n_ready - ready.n_debug > MAX_SCHED_READY_INSNS)
6605 ready_sort_debug (&ready);
6606 ready_sort_real (&ready);
6608 /* Find first free-standing insn past MAX_SCHED_READY_INSNS.
6609 If there are debug insns, we know they're first. */
6610 for (i = MAX_SCHED_READY_INSNS + ready.n_debug; i < ready.n_ready; i++)
6611 if (!SCHED_GROUP_P (ready_element (&ready, i)))
6612 break;
6614 if (sched_verbose >= 2)
6616 fprintf (sched_dump,
6617 ";;\t\tReady list on entry: %d insns: ", ready.n_ready);
6618 debug_ready_list (&ready);
6619 fprintf (sched_dump,
6620 ";;\t\t before reload => truncated to %d insns\n", i);
6623 /* Delay all insns past it for 1 cycle. If debug counter is
6624 activated make an exception for the insn right after
6625 nonscheduled_insns_begin. */
6627 rtx_insn *skip_insn;
6629 if (dbg_cnt (sched_insn) == false)
6630 skip_insn = first_nonscheduled_insn ();
6631 else
6632 skip_insn = NULL;
6634 while (i < ready.n_ready)
6636 rtx_insn *insn;
6638 insn = ready_remove (&ready, i);
6640 if (insn != skip_insn)
6641 queue_insn (insn, 1, "list truncated");
6643 if (skip_insn)
6644 ready_add (&ready, skip_insn, true);
6648 /* Now we can restore basic block notes and maintain precise cfg. */
6649 restore_bb_notes (*target_bb);
6651 last_clock_var = -1;
6653 advance = 0;
6655 gcc_assert (scheduled_insns.length () == 0);
6656 sort_p = TRUE;
6657 must_backtrack = false;
6658 modulo_insns_scheduled = 0;
6660 ls.modulo_epilogue = false;
6661 ls.first_cycle_insn_p = true;
6663 /* Loop until all the insns in BB are scheduled. */
6664 while ((*current_sched_info->schedule_more_p) ())
6666 perform_replacements_new_cycle ();
6669 start_clock_var = clock_var;
6671 clock_var++;
6673 advance_one_cycle ();
6675 /* Add to the ready list all pending insns that can be issued now.
6676 If there are no ready insns, increment clock until one
6677 is ready and add all pending insns at that point to the ready
6678 list. */
6679 queue_to_ready (&ready);
6681 gcc_assert (ready.n_ready);
6683 if (sched_verbose >= 2)
6685 fprintf (sched_dump, ";;\t\tReady list after queue_to_ready:");
6686 debug_ready_list (&ready);
6688 advance -= clock_var - start_clock_var;
6690 while (advance > 0);
6692 if (ls.modulo_epilogue)
6694 int stage = clock_var / modulo_ii;
6695 if (stage > modulo_last_stage * 2 + 2)
6697 if (sched_verbose >= 2)
6698 fprintf (sched_dump,
6699 ";;\t\tmodulo scheduled succeeded at II %d\n",
6700 modulo_ii);
6701 success = true;
6702 goto end_schedule;
6705 else if (modulo_ii > 0)
6707 int stage = clock_var / modulo_ii;
6708 if (stage > modulo_max_stages)
6710 if (sched_verbose >= 2)
6711 fprintf (sched_dump,
6712 ";;\t\tfailing schedule due to excessive stages\n");
6713 goto end_schedule;
6715 if (modulo_n_insns == modulo_insns_scheduled
6716 && stage > modulo_last_stage)
6718 if (sched_verbose >= 2)
6719 fprintf (sched_dump,
6720 ";;\t\tfound kernel after %d stages, II %d\n",
6721 stage, modulo_ii);
6722 ls.modulo_epilogue = true;
6726 prune_ready_list (temp_state, true, false, ls.modulo_epilogue);
6727 if (ready.n_ready == 0)
6728 continue;
6729 if (must_backtrack)
6730 goto do_backtrack;
6732 ls.shadows_only_p = false;
6733 cycle_issued_insns = 0;
6734 ls.can_issue_more = issue_rate;
6735 for (;;)
6737 rtx_insn *insn;
6738 int cost;
6739 bool asm_p;
6741 if (sort_p && ready.n_ready > 0)
6743 /* Sort the ready list based on priority. This must be
6744 done every iteration through the loop, as schedule_insn
6745 may have readied additional insns that will not be
6746 sorted correctly. */
6747 ready_sort (&ready);
6749 if (sched_verbose >= 2)
6751 fprintf (sched_dump,
6752 ";;\t\tReady list after ready_sort: ");
6753 debug_ready_list (&ready);
6757 /* We don't want md sched reorder to even see debug isns, so put
6758 them out right away. */
6759 if (ready.n_ready && DEBUG_INSN_P (ready_element (&ready, 0))
6760 && (*current_sched_info->schedule_more_p) ())
6762 while (ready.n_ready && DEBUG_INSN_P (ready_element (&ready, 0)))
6764 rtx_insn *insn = ready_remove_first (&ready);
6765 gcc_assert (DEBUG_INSN_P (insn));
6766 (*current_sched_info->begin_schedule_ready) (insn);
6767 scheduled_insns.safe_push (insn);
6768 last_scheduled_insn = insn;
6769 advance = schedule_insn (insn);
6770 gcc_assert (advance == 0);
6771 if (ready.n_ready > 0)
6772 ready_sort (&ready);
6776 if (ls.first_cycle_insn_p && !ready.n_ready)
6777 break;
6779 resume_after_backtrack:
6780 /* Allow the target to reorder the list, typically for
6781 better instruction bundling. */
6782 if (sort_p
6783 && (ready.n_ready == 0
6784 || !SCHED_GROUP_P (ready_element (&ready, 0))))
6786 if (ls.first_cycle_insn_p && targetm.sched.reorder)
6787 ls.can_issue_more
6788 = targetm.sched.reorder (sched_dump, sched_verbose,
6789 ready_lastpos (&ready),
6790 &ready.n_ready, clock_var);
6791 else if (!ls.first_cycle_insn_p && targetm.sched.reorder2)
6792 ls.can_issue_more
6793 = targetm.sched.reorder2 (sched_dump, sched_verbose,
6794 ready.n_ready
6795 ? ready_lastpos (&ready) : NULL,
6796 &ready.n_ready, clock_var);
6799 restart_choose_ready:
6800 if (sched_verbose >= 2)
6802 fprintf (sched_dump, ";;\tReady list (t = %3d): ",
6803 clock_var);
6804 debug_ready_list (&ready);
6805 if (sched_pressure == SCHED_PRESSURE_WEIGHTED)
6806 print_curr_reg_pressure ();
6809 if (ready.n_ready == 0
6810 && ls.can_issue_more
6811 && reload_completed)
6813 /* Allow scheduling insns directly from the queue in case
6814 there's nothing better to do (ready list is empty) but
6815 there are still vacant dispatch slots in the current cycle. */
6816 if (sched_verbose >= 6)
6817 fprintf (sched_dump,";;\t\tSecond chance\n");
6818 memcpy (temp_state, curr_state, dfa_state_size);
6819 if (early_queue_to_ready (temp_state, &ready))
6820 ready_sort (&ready);
6823 if (ready.n_ready == 0
6824 || !ls.can_issue_more
6825 || state_dead_lock_p (curr_state)
6826 || !(*current_sched_info->schedule_more_p) ())
6827 break;
6829 /* Select and remove the insn from the ready list. */
6830 if (sort_p)
6832 int res;
6834 insn = NULL;
6835 res = choose_ready (&ready, ls.first_cycle_insn_p, &insn);
6837 if (res < 0)
6838 /* Finish cycle. */
6839 break;
6840 if (res > 0)
6841 goto restart_choose_ready;
6843 gcc_assert (insn != NULL_RTX);
6845 else
6846 insn = ready_remove_first (&ready);
6848 if (sched_pressure != SCHED_PRESSURE_NONE
6849 && INSN_TICK (insn) > clock_var)
6851 ready_add (&ready, insn, true);
6852 advance = 1;
6853 break;
6856 if (targetm.sched.dfa_new_cycle
6857 && targetm.sched.dfa_new_cycle (sched_dump, sched_verbose,
6858 insn, last_clock_var,
6859 clock_var, &sort_p))
6860 /* SORT_P is used by the target to override sorting
6861 of the ready list. This is needed when the target
6862 has modified its internal structures expecting that
6863 the insn will be issued next. As we need the insn
6864 to have the highest priority (so it will be returned by
6865 the ready_remove_first call above), we invoke
6866 ready_add (&ready, insn, true).
6867 But, still, there is one issue: INSN can be later
6868 discarded by scheduler's front end through
6869 current_sched_info->can_schedule_ready_p, hence, won't
6870 be issued next. */
6872 ready_add (&ready, insn, true);
6873 break;
6876 sort_p = TRUE;
6878 if (current_sched_info->can_schedule_ready_p
6879 && ! (*current_sched_info->can_schedule_ready_p) (insn))
6880 /* We normally get here only if we don't want to move
6881 insn from the split block. */
6883 TODO_SPEC (insn) = DEP_POSTPONED;
6884 goto restart_choose_ready;
6887 if (delay_htab)
6889 /* If this insn is the first part of a delay-slot pair, record a
6890 backtrack point. */
6891 struct delay_pair *delay_entry;
6892 delay_entry
6893 = delay_htab->find_with_hash (insn, htab_hash_pointer (insn));
6894 if (delay_entry)
6896 save_backtrack_point (delay_entry, ls);
6897 if (sched_verbose >= 2)
6898 fprintf (sched_dump, ";;\t\tsaving backtrack point\n");
6902 /* DECISION is made. */
6904 if (modulo_ii > 0 && INSN_UID (insn) < modulo_iter0_max_uid)
6906 modulo_insns_scheduled++;
6907 modulo_last_stage = clock_var / modulo_ii;
6909 if (TODO_SPEC (insn) & SPECULATIVE)
6910 generate_recovery_code (insn);
6912 if (targetm.sched.dispatch (NULL, IS_DISPATCH_ON))
6913 targetm.sched.dispatch_do (insn, ADD_TO_DISPATCH_WINDOW);
6915 /* Update counters, etc in the scheduler's front end. */
6916 (*current_sched_info->begin_schedule_ready) (insn);
6917 scheduled_insns.safe_push (insn);
6918 gcc_assert (NONDEBUG_INSN_P (insn));
6919 last_nondebug_scheduled_insn = last_scheduled_insn = insn;
6921 if (recog_memoized (insn) >= 0)
6923 memcpy (temp_state, curr_state, dfa_state_size);
6924 cost = state_transition (curr_state, insn);
6925 if (sched_pressure != SCHED_PRESSURE_WEIGHTED && !sched_fusion)
6926 gcc_assert (cost < 0);
6927 if (memcmp (temp_state, curr_state, dfa_state_size) != 0)
6928 cycle_issued_insns++;
6929 asm_p = false;
6931 else
6932 asm_p = (GET_CODE (PATTERN (insn)) == ASM_INPUT
6933 || asm_noperands (PATTERN (insn)) >= 0);
6935 if (targetm.sched.variable_issue)
6936 ls.can_issue_more =
6937 targetm.sched.variable_issue (sched_dump, sched_verbose,
6938 insn, ls.can_issue_more);
6939 /* A naked CLOBBER or USE generates no instruction, so do
6940 not count them against the issue rate. */
6941 else if (GET_CODE (PATTERN (insn)) != USE
6942 && GET_CODE (PATTERN (insn)) != CLOBBER)
6943 ls.can_issue_more--;
6944 advance = schedule_insn (insn);
6946 if (SHADOW_P (insn))
6947 ls.shadows_only_p = true;
6949 /* After issuing an asm insn we should start a new cycle. */
6950 if (advance == 0 && asm_p)
6951 advance = 1;
6953 if (must_backtrack)
6954 break;
6956 if (advance != 0)
6957 break;
6959 ls.first_cycle_insn_p = false;
6960 if (ready.n_ready > 0)
6961 prune_ready_list (temp_state, false, ls.shadows_only_p,
6962 ls.modulo_epilogue);
6965 do_backtrack:
6966 if (!must_backtrack)
6967 for (i = 0; i < ready.n_ready; i++)
6969 rtx_insn *insn = ready_element (&ready, i);
6970 if (INSN_EXACT_TICK (insn) == clock_var)
6972 must_backtrack = true;
6973 clock_var++;
6974 break;
6977 if (must_backtrack && modulo_ii > 0)
6979 if (modulo_backtracks_left == 0)
6980 goto end_schedule;
6981 modulo_backtracks_left--;
6983 while (must_backtrack)
6985 struct haifa_saved_data *failed;
6986 rtx_insn *failed_insn;
6988 must_backtrack = false;
6989 failed = verify_shadows ();
6990 gcc_assert (failed);
6992 failed_insn = failed->delay_pair->i1;
6993 /* Clear these queues. */
6994 perform_replacements_new_cycle ();
6995 toggle_cancelled_flags (false);
6996 unschedule_insns_until (failed_insn);
6997 while (failed != backtrack_queue)
6998 free_topmost_backtrack_point (true);
6999 restore_last_backtrack_point (&ls);
7000 if (sched_verbose >= 2)
7001 fprintf (sched_dump, ";;\t\trewind to cycle %d\n", clock_var);
7002 /* Delay by at least a cycle. This could cause additional
7003 backtracking. */
7004 queue_insn (failed_insn, 1, "backtracked");
7005 advance = 0;
7006 if (must_backtrack)
7007 continue;
7008 if (ready.n_ready > 0)
7009 goto resume_after_backtrack;
7010 else
7012 if (clock_var == 0 && ls.first_cycle_insn_p)
7013 goto end_schedule;
7014 advance = 1;
7015 break;
7018 ls.first_cycle_insn_p = true;
7020 if (ls.modulo_epilogue)
7021 success = true;
7022 end_schedule:
7023 if (!ls.first_cycle_insn_p || advance)
7024 advance_one_cycle ();
7025 perform_replacements_new_cycle ();
7026 if (modulo_ii > 0)
7028 /* Once again, debug insn suckiness: they can be on the ready list
7029 even if they have unresolved dependencies. To make our view
7030 of the world consistent, remove such "ready" insns. */
7031 restart_debug_insn_loop:
7032 for (i = ready.n_ready - 1; i >= 0; i--)
7034 rtx_insn *x;
7036 x = ready_element (&ready, i);
7037 if (DEPS_LIST_FIRST (INSN_HARD_BACK_DEPS (x)) != NULL
7038 || DEPS_LIST_FIRST (INSN_SPEC_BACK_DEPS (x)) != NULL)
7040 ready_remove (&ready, i);
7041 goto restart_debug_insn_loop;
7044 for (i = ready.n_ready - 1; i >= 0; i--)
7046 rtx_insn *x;
7048 x = ready_element (&ready, i);
7049 resolve_dependencies (x);
7051 for (i = 0; i <= max_insn_queue_index; i++)
7053 rtx_insn_list *link;
7054 while ((link = insn_queue[i]) != NULL)
7056 rtx_insn *x = link->insn ();
7057 insn_queue[i] = link->next ();
7058 QUEUE_INDEX (x) = QUEUE_NOWHERE;
7059 free_INSN_LIST_node (link);
7060 resolve_dependencies (x);
7065 if (!success)
7066 undo_all_replacements ();
7068 /* Debug info. */
7069 if (sched_verbose)
7071 fprintf (sched_dump, ";;\tReady list (final): ");
7072 debug_ready_list (&ready);
7075 if (modulo_ii == 0 && current_sched_info->queue_must_finish_empty)
7076 /* Sanity check -- queue must be empty now. Meaningless if region has
7077 multiple bbs. */
7078 gcc_assert (!q_size && !ready.n_ready && !ready.n_debug);
7079 else if (modulo_ii == 0)
7081 /* We must maintain QUEUE_INDEX between blocks in region. */
7082 for (i = ready.n_ready - 1; i >= 0; i--)
7084 rtx_insn *x;
7086 x = ready_element (&ready, i);
7087 QUEUE_INDEX (x) = QUEUE_NOWHERE;
7088 TODO_SPEC (x) = HARD_DEP;
7091 if (q_size)
7092 for (i = 0; i <= max_insn_queue_index; i++)
7094 rtx_insn_list *link;
7095 for (link = insn_queue[i]; link; link = link->next ())
7097 rtx_insn *x;
7099 x = link->insn ();
7100 QUEUE_INDEX (x) = QUEUE_NOWHERE;
7101 TODO_SPEC (x) = HARD_DEP;
7103 free_INSN_LIST_list (&insn_queue[i]);
7107 if (sched_pressure == SCHED_PRESSURE_MODEL)
7108 model_end_schedule ();
7110 if (success)
7112 commit_schedule (prev_head, tail, target_bb);
7113 if (sched_verbose)
7114 fprintf (sched_dump, ";; total time = %d\n", clock_var);
7116 else
7117 last_scheduled_insn = tail;
7119 scheduled_insns.truncate (0);
7121 if (!current_sched_info->queue_must_finish_empty
7122 || haifa_recovery_bb_recently_added_p)
7124 /* INSN_TICK (minimum clock tick at which the insn becomes
7125 ready) may be not correct for the insn in the subsequent
7126 blocks of the region. We should use a correct value of
7127 `clock_var' or modify INSN_TICK. It is better to keep
7128 clock_var value equal to 0 at the start of a basic block.
7129 Therefore we modify INSN_TICK here. */
7130 fix_inter_tick (NEXT_INSN (prev_head), last_scheduled_insn);
7133 if (targetm.sched.finish)
7135 targetm.sched.finish (sched_dump, sched_verbose);
7136 /* Target might have added some instructions to the scheduled block
7137 in its md_finish () hook. These new insns don't have any data
7138 initialized and to identify them we extend h_i_d so that they'll
7139 get zero luids. */
7140 sched_extend_luids ();
7143 /* Update head/tail boundaries. */
7144 head = NEXT_INSN (prev_head);
7145 tail = last_scheduled_insn;
7147 if (sched_verbose)
7149 fprintf (sched_dump, ";; new head = %d\n;; new tail = %d\n",
7150 INSN_UID (head), INSN_UID (tail));
7152 if (sched_verbose >= 2)
7154 dump_insn_stream (head, tail);
7155 print_rank_for_schedule_stats (";; TOTAL ", &rank_for_schedule_stats,
7156 NULL);
7159 fprintf (sched_dump, "\n");
7162 head = restore_other_notes (head, NULL);
7164 current_sched_info->head = head;
7165 current_sched_info->tail = tail;
7167 free_backtrack_queue ();
7169 return success;
7172 /* Set_priorities: compute priority of each insn in the block. */
7175 set_priorities (rtx_insn *head, rtx_insn *tail)
7177 rtx_insn *insn;
7178 int n_insn;
7179 int sched_max_insns_priority =
7180 current_sched_info->sched_max_insns_priority;
7181 rtx_insn *prev_head;
7183 if (head == tail && ! INSN_P (head))
7184 gcc_unreachable ();
7186 n_insn = 0;
7188 prev_head = PREV_INSN (head);
7189 for (insn = tail; insn != prev_head; insn = PREV_INSN (insn))
7191 if (!INSN_P (insn))
7192 continue;
7194 n_insn++;
7195 (void) priority (insn);
7197 gcc_assert (INSN_PRIORITY_KNOWN (insn));
7199 sched_max_insns_priority = MAX (sched_max_insns_priority,
7200 INSN_PRIORITY (insn));
7203 current_sched_info->sched_max_insns_priority = sched_max_insns_priority;
7205 return n_insn;
7208 /* Set sched_dump and sched_verbose for the desired debugging output. */
7209 void
7210 setup_sched_dump (void)
7212 sched_verbose = sched_verbose_param;
7213 sched_dump = dump_file;
7214 if (!dump_file)
7215 sched_verbose = 0;
7218 /* Allocate data for register pressure sensitive scheduling. */
7219 static void
7220 alloc_global_sched_pressure_data (void)
7222 if (sched_pressure != SCHED_PRESSURE_NONE)
7224 int i, max_regno = max_reg_num ();
7226 if (sched_dump != NULL)
7227 /* We need info about pseudos for rtl dumps about pseudo
7228 classes and costs. */
7229 regstat_init_n_sets_and_refs ();
7230 ira_set_pseudo_classes (true, sched_verbose ? sched_dump : NULL);
7231 sched_regno_pressure_class
7232 = (enum reg_class *) xmalloc (max_regno * sizeof (enum reg_class));
7233 for (i = 0; i < max_regno; i++)
7234 sched_regno_pressure_class[i]
7235 = (i < FIRST_PSEUDO_REGISTER
7236 ? ira_pressure_class_translate[REGNO_REG_CLASS (i)]
7237 : ira_pressure_class_translate[reg_allocno_class (i)]);
7238 curr_reg_live = BITMAP_ALLOC (NULL);
7239 if (sched_pressure == SCHED_PRESSURE_WEIGHTED)
7241 saved_reg_live = BITMAP_ALLOC (NULL);
7242 region_ref_regs = BITMAP_ALLOC (NULL);
7245 /* Calculate number of CALL_SAVED_REGS and FIXED_REGS in register classes
7246 that we calculate register pressure for. */
7247 for (int c = 0; c < ira_pressure_classes_num; ++c)
7249 enum reg_class cl = ira_pressure_classes[c];
7251 call_saved_regs_num[cl] = 0;
7252 fixed_regs_num[cl] = 0;
7254 for (int i = 0; i < ira_class_hard_regs_num[cl]; ++i)
7255 if (!call_used_regs[ira_class_hard_regs[cl][i]])
7256 ++call_saved_regs_num[cl];
7257 else if (fixed_regs[ira_class_hard_regs[cl][i]])
7258 ++fixed_regs_num[cl];
7263 /* Free data for register pressure sensitive scheduling. Also called
7264 from schedule_region when stopping sched-pressure early. */
7265 void
7266 free_global_sched_pressure_data (void)
7268 if (sched_pressure != SCHED_PRESSURE_NONE)
7270 if (regstat_n_sets_and_refs != NULL)
7271 regstat_free_n_sets_and_refs ();
7272 if (sched_pressure == SCHED_PRESSURE_WEIGHTED)
7274 BITMAP_FREE (region_ref_regs);
7275 BITMAP_FREE (saved_reg_live);
7277 BITMAP_FREE (curr_reg_live);
7278 free (sched_regno_pressure_class);
7282 /* Initialize some global state for the scheduler. This function works
7283 with the common data shared between all the schedulers. It is called
7284 from the scheduler specific initialization routine. */
7286 void
7287 sched_init (void)
7289 /* Disable speculative loads in their presence if cc0 defined. */
7290 if (HAVE_cc0)
7291 flag_schedule_speculative_load = 0;
7293 if (targetm.sched.dispatch (NULL, IS_DISPATCH_ON))
7294 targetm.sched.dispatch_do (NULL, DISPATCH_INIT);
7296 if (live_range_shrinkage_p)
7297 sched_pressure = SCHED_PRESSURE_WEIGHTED;
7298 else if (flag_sched_pressure
7299 && !reload_completed
7300 && common_sched_info->sched_pass_id == SCHED_RGN_PASS)
7301 sched_pressure = ((enum sched_pressure_algorithm)
7302 PARAM_VALUE (PARAM_SCHED_PRESSURE_ALGORITHM));
7303 else
7304 sched_pressure = SCHED_PRESSURE_NONE;
7306 if (sched_pressure != SCHED_PRESSURE_NONE)
7307 ira_setup_eliminable_regset ();
7309 /* Initialize SPEC_INFO. */
7310 if (targetm.sched.set_sched_flags)
7312 spec_info = &spec_info_var;
7313 targetm.sched.set_sched_flags (spec_info);
7315 if (spec_info->mask != 0)
7317 spec_info->data_weakness_cutoff =
7318 (PARAM_VALUE (PARAM_SCHED_SPEC_PROB_CUTOFF) * MAX_DEP_WEAK) / 100;
7319 spec_info->control_weakness_cutoff =
7320 (PARAM_VALUE (PARAM_SCHED_SPEC_PROB_CUTOFF)
7321 * REG_BR_PROB_BASE) / 100;
7323 else
7324 /* So we won't read anything accidentally. */
7325 spec_info = NULL;
7328 else
7329 /* So we won't read anything accidentally. */
7330 spec_info = 0;
7332 /* Initialize issue_rate. */
7333 if (targetm.sched.issue_rate)
7334 issue_rate = targetm.sched.issue_rate ();
7335 else
7336 issue_rate = 1;
7338 if (targetm.sched.first_cycle_multipass_dfa_lookahead
7339 /* Don't use max_issue with reg_pressure scheduling. Multipass
7340 scheduling and reg_pressure scheduling undo each other's decisions. */
7341 && sched_pressure == SCHED_PRESSURE_NONE)
7342 dfa_lookahead = targetm.sched.first_cycle_multipass_dfa_lookahead ();
7343 else
7344 dfa_lookahead = 0;
7346 /* Set to "0" so that we recalculate. */
7347 max_lookahead_tries = 0;
7349 if (targetm.sched.init_dfa_pre_cycle_insn)
7350 targetm.sched.init_dfa_pre_cycle_insn ();
7352 if (targetm.sched.init_dfa_post_cycle_insn)
7353 targetm.sched.init_dfa_post_cycle_insn ();
7355 dfa_start ();
7356 dfa_state_size = state_size ();
7358 init_alias_analysis ();
7360 if (!sched_no_dce)
7361 df_set_flags (DF_LR_RUN_DCE);
7362 df_note_add_problem ();
7364 /* More problems needed for interloop dep calculation in SMS. */
7365 if (common_sched_info->sched_pass_id == SCHED_SMS_PASS)
7367 df_rd_add_problem ();
7368 df_chain_add_problem (DF_DU_CHAIN + DF_UD_CHAIN);
7371 df_analyze ();
7373 /* Do not run DCE after reload, as this can kill nops inserted
7374 by bundling. */
7375 if (reload_completed)
7376 df_clear_flags (DF_LR_RUN_DCE);
7378 regstat_compute_calls_crossed ();
7380 if (targetm.sched.init_global)
7381 targetm.sched.init_global (sched_dump, sched_verbose, get_max_uid () + 1);
7383 alloc_global_sched_pressure_data ();
7385 curr_state = xmalloc (dfa_state_size);
7388 static void haifa_init_only_bb (basic_block, basic_block);
7390 /* Initialize data structures specific to the Haifa scheduler. */
7391 void
7392 haifa_sched_init (void)
7394 setup_sched_dump ();
7395 sched_init ();
7397 scheduled_insns.create (0);
7399 if (spec_info != NULL)
7401 sched_deps_info->use_deps_list = 1;
7402 sched_deps_info->generate_spec_deps = 1;
7405 /* Initialize luids, dependency caches, target and h_i_d for the
7406 whole function. */
7408 sched_init_bbs ();
7410 auto_vec<basic_block> bbs (n_basic_blocks_for_fn (cfun));
7411 basic_block bb;
7412 FOR_EACH_BB_FN (bb, cfun)
7413 bbs.quick_push (bb);
7414 sched_init_luids (bbs);
7415 sched_deps_init (true);
7416 sched_extend_target ();
7417 haifa_init_h_i_d (bbs);
7420 sched_init_only_bb = haifa_init_only_bb;
7421 sched_split_block = sched_split_block_1;
7422 sched_create_empty_bb = sched_create_empty_bb_1;
7423 haifa_recovery_bb_ever_added_p = false;
7425 nr_begin_data = nr_begin_control = nr_be_in_data = nr_be_in_control = 0;
7426 before_recovery = 0;
7427 after_recovery = 0;
7429 modulo_ii = 0;
7432 /* Finish work with the data specific to the Haifa scheduler. */
7433 void
7434 haifa_sched_finish (void)
7436 sched_create_empty_bb = NULL;
7437 sched_split_block = NULL;
7438 sched_init_only_bb = NULL;
7440 if (spec_info && spec_info->dump)
7442 char c = reload_completed ? 'a' : 'b';
7444 fprintf (spec_info->dump,
7445 ";; %s:\n", current_function_name ());
7447 fprintf (spec_info->dump,
7448 ";; Procedure %cr-begin-data-spec motions == %d\n",
7449 c, nr_begin_data);
7450 fprintf (spec_info->dump,
7451 ";; Procedure %cr-be-in-data-spec motions == %d\n",
7452 c, nr_be_in_data);
7453 fprintf (spec_info->dump,
7454 ";; Procedure %cr-begin-control-spec motions == %d\n",
7455 c, nr_begin_control);
7456 fprintf (spec_info->dump,
7457 ";; Procedure %cr-be-in-control-spec motions == %d\n",
7458 c, nr_be_in_control);
7461 scheduled_insns.release ();
7463 /* Finalize h_i_d, dependency caches, and luids for the whole
7464 function. Target will be finalized in md_global_finish (). */
7465 sched_deps_finish ();
7466 sched_finish_luids ();
7467 current_sched_info = NULL;
7468 insn_queue = NULL;
7469 sched_finish ();
7472 /* Free global data used during insn scheduling. This function works with
7473 the common data shared between the schedulers. */
7475 void
7476 sched_finish (void)
7478 haifa_finish_h_i_d ();
7479 free_global_sched_pressure_data ();
7480 free (curr_state);
7482 if (targetm.sched.finish_global)
7483 targetm.sched.finish_global (sched_dump, sched_verbose);
7485 end_alias_analysis ();
7487 regstat_free_calls_crossed ();
7489 dfa_finish ();
7492 /* Free all delay_pair structures that were recorded. */
7493 void
7494 free_delay_pairs (void)
7496 if (delay_htab)
7498 delay_htab->empty ();
7499 delay_htab_i2->empty ();
7503 /* Fix INSN_TICKs of the instructions in the current block as well as
7504 INSN_TICKs of their dependents.
7505 HEAD and TAIL are the begin and the end of the current scheduled block. */
7506 static void
7507 fix_inter_tick (rtx_insn *head, rtx_insn *tail)
7509 /* Set of instructions with corrected INSN_TICK. */
7510 bitmap_head processed;
7511 /* ??? It is doubtful if we should assume that cycle advance happens on
7512 basic block boundaries. Basically insns that are unconditionally ready
7513 on the start of the block are more preferable then those which have
7514 a one cycle dependency over insn from the previous block. */
7515 int next_clock = clock_var + 1;
7517 bitmap_initialize (&processed, 0);
7519 /* Iterates over scheduled instructions and fix their INSN_TICKs and
7520 INSN_TICKs of dependent instructions, so that INSN_TICKs are consistent
7521 across different blocks. */
7522 for (tail = NEXT_INSN (tail); head != tail; head = NEXT_INSN (head))
7524 if (INSN_P (head))
7526 int tick;
7527 sd_iterator_def sd_it;
7528 dep_t dep;
7530 tick = INSN_TICK (head);
7531 gcc_assert (tick >= MIN_TICK);
7533 /* Fix INSN_TICK of instruction from just scheduled block. */
7534 if (bitmap_set_bit (&processed, INSN_LUID (head)))
7536 tick -= next_clock;
7538 if (tick < MIN_TICK)
7539 tick = MIN_TICK;
7541 INSN_TICK (head) = tick;
7544 if (DEBUG_INSN_P (head))
7545 continue;
7547 FOR_EACH_DEP (head, SD_LIST_RES_FORW, sd_it, dep)
7549 rtx_insn *next;
7551 next = DEP_CON (dep);
7552 tick = INSN_TICK (next);
7554 if (tick != INVALID_TICK
7555 /* If NEXT has its INSN_TICK calculated, fix it.
7556 If not - it will be properly calculated from
7557 scratch later in fix_tick_ready. */
7558 && bitmap_set_bit (&processed, INSN_LUID (next)))
7560 tick -= next_clock;
7562 if (tick < MIN_TICK)
7563 tick = MIN_TICK;
7565 if (tick > INTER_TICK (next))
7566 INTER_TICK (next) = tick;
7567 else
7568 tick = INTER_TICK (next);
7570 INSN_TICK (next) = tick;
7575 bitmap_clear (&processed);
7578 /* Check if NEXT is ready to be added to the ready or queue list.
7579 If "yes", add it to the proper list.
7580 Returns:
7581 -1 - is not ready yet,
7582 0 - added to the ready list,
7583 0 < N - queued for N cycles. */
7585 try_ready (rtx_insn *next)
7587 ds_t old_ts, new_ts;
7589 old_ts = TODO_SPEC (next);
7591 gcc_assert (!(old_ts & ~(SPECULATIVE | HARD_DEP | DEP_CONTROL | DEP_POSTPONED))
7592 && (old_ts == HARD_DEP
7593 || old_ts == DEP_POSTPONED
7594 || (old_ts & SPECULATIVE)
7595 || old_ts == DEP_CONTROL));
7597 new_ts = recompute_todo_spec (next, false);
7599 if (new_ts & (HARD_DEP | DEP_POSTPONED))
7600 gcc_assert (new_ts == old_ts
7601 && QUEUE_INDEX (next) == QUEUE_NOWHERE);
7602 else if (current_sched_info->new_ready)
7603 new_ts = current_sched_info->new_ready (next, new_ts);
7605 /* * if !(old_ts & SPECULATIVE) (e.g. HARD_DEP or 0), then insn might
7606 have its original pattern or changed (speculative) one. This is due
7607 to changing ebb in region scheduling.
7608 * But if (old_ts & SPECULATIVE), then we are pretty sure that insn
7609 has speculative pattern.
7611 We can't assert (!(new_ts & HARD_DEP) || new_ts == old_ts) here because
7612 control-speculative NEXT could have been discarded by sched-rgn.c
7613 (the same case as when discarded by can_schedule_ready_p ()). */
7615 if ((new_ts & SPECULATIVE)
7616 /* If (old_ts == new_ts), then (old_ts & SPECULATIVE) and we don't
7617 need to change anything. */
7618 && new_ts != old_ts)
7620 int res;
7621 rtx new_pat;
7623 gcc_assert ((new_ts & SPECULATIVE) && !(new_ts & ~SPECULATIVE));
7625 res = haifa_speculate_insn (next, new_ts, &new_pat);
7627 switch (res)
7629 case -1:
7630 /* It would be nice to change DEP_STATUS of all dependences,
7631 which have ((DEP_STATUS & SPECULATIVE) == new_ts) to HARD_DEP,
7632 so we won't reanalyze anything. */
7633 new_ts = HARD_DEP;
7634 break;
7636 case 0:
7637 /* We follow the rule, that every speculative insn
7638 has non-null ORIG_PAT. */
7639 if (!ORIG_PAT (next))
7640 ORIG_PAT (next) = PATTERN (next);
7641 break;
7643 case 1:
7644 if (!ORIG_PAT (next))
7645 /* If we gonna to overwrite the original pattern of insn,
7646 save it. */
7647 ORIG_PAT (next) = PATTERN (next);
7649 res = haifa_change_pattern (next, new_pat);
7650 gcc_assert (res);
7651 break;
7653 default:
7654 gcc_unreachable ();
7658 /* We need to restore pattern only if (new_ts == 0), because otherwise it is
7659 either correct (new_ts & SPECULATIVE),
7660 or we simply don't care (new_ts & HARD_DEP). */
7662 gcc_assert (!ORIG_PAT (next)
7663 || !IS_SPECULATION_BRANCHY_CHECK_P (next));
7665 TODO_SPEC (next) = new_ts;
7667 if (new_ts & (HARD_DEP | DEP_POSTPONED))
7669 /* We can't assert (QUEUE_INDEX (next) == QUEUE_NOWHERE) here because
7670 control-speculative NEXT could have been discarded by sched-rgn.c
7671 (the same case as when discarded by can_schedule_ready_p ()). */
7672 /*gcc_assert (QUEUE_INDEX (next) == QUEUE_NOWHERE);*/
7674 change_queue_index (next, QUEUE_NOWHERE);
7676 return -1;
7678 else if (!(new_ts & BEGIN_SPEC)
7679 && ORIG_PAT (next) && PREDICATED_PAT (next) == NULL_RTX
7680 && !IS_SPECULATION_CHECK_P (next))
7681 /* We should change pattern of every previously speculative
7682 instruction - and we determine if NEXT was speculative by using
7683 ORIG_PAT field. Except one case - speculation checks have ORIG_PAT
7684 pat too, so skip them. */
7686 bool success = haifa_change_pattern (next, ORIG_PAT (next));
7687 gcc_assert (success);
7688 ORIG_PAT (next) = 0;
7691 if (sched_verbose >= 2)
7693 fprintf (sched_dump, ";;\t\tdependencies resolved: insn %s",
7694 (*current_sched_info->print_insn) (next, 0));
7696 if (spec_info && spec_info->dump)
7698 if (new_ts & BEGIN_DATA)
7699 fprintf (spec_info->dump, "; data-spec;");
7700 if (new_ts & BEGIN_CONTROL)
7701 fprintf (spec_info->dump, "; control-spec;");
7702 if (new_ts & BE_IN_CONTROL)
7703 fprintf (spec_info->dump, "; in-control-spec;");
7705 if (TODO_SPEC (next) & DEP_CONTROL)
7706 fprintf (sched_dump, " predicated");
7707 fprintf (sched_dump, "\n");
7710 adjust_priority (next);
7712 return fix_tick_ready (next);
7715 /* Calculate INSN_TICK of NEXT and add it to either ready or queue list. */
7716 static int
7717 fix_tick_ready (rtx_insn *next)
7719 int tick, delay;
7721 if (!DEBUG_INSN_P (next) && !sd_lists_empty_p (next, SD_LIST_RES_BACK))
7723 int full_p;
7724 sd_iterator_def sd_it;
7725 dep_t dep;
7727 tick = INSN_TICK (next);
7728 /* if tick is not equal to INVALID_TICK, then update
7729 INSN_TICK of NEXT with the most recent resolved dependence
7730 cost. Otherwise, recalculate from scratch. */
7731 full_p = (tick == INVALID_TICK);
7733 FOR_EACH_DEP (next, SD_LIST_RES_BACK, sd_it, dep)
7735 rtx_insn *pro = DEP_PRO (dep);
7736 int tick1;
7738 gcc_assert (INSN_TICK (pro) >= MIN_TICK);
7740 tick1 = INSN_TICK (pro) + dep_cost (dep);
7741 if (tick1 > tick)
7742 tick = tick1;
7744 if (!full_p)
7745 break;
7748 else
7749 tick = -1;
7751 INSN_TICK (next) = tick;
7753 delay = tick - clock_var;
7754 if (delay <= 0 || sched_pressure != SCHED_PRESSURE_NONE || sched_fusion)
7755 delay = QUEUE_READY;
7757 change_queue_index (next, delay);
7759 return delay;
7762 /* Move NEXT to the proper queue list with (DELAY >= 1),
7763 or add it to the ready list (DELAY == QUEUE_READY),
7764 or remove it from ready and queue lists at all (DELAY == QUEUE_NOWHERE). */
7765 static void
7766 change_queue_index (rtx_insn *next, int delay)
7768 int i = QUEUE_INDEX (next);
7770 gcc_assert (QUEUE_NOWHERE <= delay && delay <= max_insn_queue_index
7771 && delay != 0);
7772 gcc_assert (i != QUEUE_SCHEDULED);
7774 if ((delay > 0 && NEXT_Q_AFTER (q_ptr, delay) == i)
7775 || (delay < 0 && delay == i))
7776 /* We have nothing to do. */
7777 return;
7779 /* Remove NEXT from wherever it is now. */
7780 if (i == QUEUE_READY)
7781 ready_remove_insn (next);
7782 else if (i >= 0)
7783 queue_remove (next);
7785 /* Add it to the proper place. */
7786 if (delay == QUEUE_READY)
7787 ready_add (readyp, next, false);
7788 else if (delay >= 1)
7789 queue_insn (next, delay, "change queue index");
7791 if (sched_verbose >= 2)
7793 fprintf (sched_dump, ";;\t\ttick updated: insn %s",
7794 (*current_sched_info->print_insn) (next, 0));
7796 if (delay == QUEUE_READY)
7797 fprintf (sched_dump, " into ready\n");
7798 else if (delay >= 1)
7799 fprintf (sched_dump, " into queue with cost=%d\n", delay);
7800 else
7801 fprintf (sched_dump, " removed from ready or queue lists\n");
7805 static int sched_ready_n_insns = -1;
7807 /* Initialize per region data structures. */
7808 void
7809 sched_extend_ready_list (int new_sched_ready_n_insns)
7811 int i;
7813 if (sched_ready_n_insns == -1)
7814 /* At the first call we need to initialize one more choice_stack
7815 entry. */
7817 i = 0;
7818 sched_ready_n_insns = 0;
7819 scheduled_insns.reserve (new_sched_ready_n_insns);
7821 else
7822 i = sched_ready_n_insns + 1;
7824 ready.veclen = new_sched_ready_n_insns + issue_rate;
7825 ready.vec = XRESIZEVEC (rtx_insn *, ready.vec, ready.veclen);
7827 gcc_assert (new_sched_ready_n_insns >= sched_ready_n_insns);
7829 ready_try = (signed char *) xrecalloc (ready_try, new_sched_ready_n_insns,
7830 sched_ready_n_insns,
7831 sizeof (*ready_try));
7833 /* We allocate +1 element to save initial state in the choice_stack[0]
7834 entry. */
7835 choice_stack = XRESIZEVEC (struct choice_entry, choice_stack,
7836 new_sched_ready_n_insns + 1);
7838 for (; i <= new_sched_ready_n_insns; i++)
7840 choice_stack[i].state = xmalloc (dfa_state_size);
7842 if (targetm.sched.first_cycle_multipass_init)
7843 targetm.sched.first_cycle_multipass_init (&(choice_stack[i]
7844 .target_data));
7847 sched_ready_n_insns = new_sched_ready_n_insns;
7850 /* Free per region data structures. */
7851 void
7852 sched_finish_ready_list (void)
7854 int i;
7856 free (ready.vec);
7857 ready.vec = NULL;
7858 ready.veclen = 0;
7860 free (ready_try);
7861 ready_try = NULL;
7863 for (i = 0; i <= sched_ready_n_insns; i++)
7865 if (targetm.sched.first_cycle_multipass_fini)
7866 targetm.sched.first_cycle_multipass_fini (&(choice_stack[i]
7867 .target_data));
7869 free (choice_stack [i].state);
7871 free (choice_stack);
7872 choice_stack = NULL;
7874 sched_ready_n_insns = -1;
7877 static int
7878 haifa_luid_for_non_insn (rtx x)
7880 gcc_assert (NOTE_P (x) || LABEL_P (x));
7882 return 0;
7885 /* Generates recovery code for INSN. */
7886 static void
7887 generate_recovery_code (rtx_insn *insn)
7889 if (TODO_SPEC (insn) & BEGIN_SPEC)
7890 begin_speculative_block (insn);
7892 /* Here we have insn with no dependencies to
7893 instructions other then CHECK_SPEC ones. */
7895 if (TODO_SPEC (insn) & BE_IN_SPEC)
7896 add_to_speculative_block (insn);
7899 /* Helper function.
7900 Tries to add speculative dependencies of type FS between instructions
7901 in deps_list L and TWIN. */
7902 static void
7903 process_insn_forw_deps_be_in_spec (rtx_insn *insn, rtx_insn *twin, ds_t fs)
7905 sd_iterator_def sd_it;
7906 dep_t dep;
7908 FOR_EACH_DEP (insn, SD_LIST_FORW, sd_it, dep)
7910 ds_t ds;
7911 rtx_insn *consumer;
7913 consumer = DEP_CON (dep);
7915 ds = DEP_STATUS (dep);
7917 if (/* If we want to create speculative dep. */
7919 /* And we can do that because this is a true dep. */
7920 && (ds & DEP_TYPES) == DEP_TRUE)
7922 gcc_assert (!(ds & BE_IN_SPEC));
7924 if (/* If this dep can be overcome with 'begin speculation'. */
7925 ds & BEGIN_SPEC)
7926 /* Then we have a choice: keep the dep 'begin speculative'
7927 or transform it into 'be in speculative'. */
7929 if (/* In try_ready we assert that if insn once became ready
7930 it can be removed from the ready (or queue) list only
7931 due to backend decision. Hence we can't let the
7932 probability of the speculative dep to decrease. */
7933 ds_weak (ds) <= ds_weak (fs))
7935 ds_t new_ds;
7937 new_ds = (ds & ~BEGIN_SPEC) | fs;
7939 if (/* consumer can 'be in speculative'. */
7940 sched_insn_is_legitimate_for_speculation_p (consumer,
7941 new_ds))
7942 /* Transform it to be in speculative. */
7943 ds = new_ds;
7946 else
7947 /* Mark the dep as 'be in speculative'. */
7948 ds |= fs;
7952 dep_def _new_dep, *new_dep = &_new_dep;
7954 init_dep_1 (new_dep, twin, consumer, DEP_TYPE (dep), ds);
7955 sd_add_dep (new_dep, false);
7960 /* Generates recovery code for BEGIN speculative INSN. */
7961 static void
7962 begin_speculative_block (rtx_insn *insn)
7964 if (TODO_SPEC (insn) & BEGIN_DATA)
7965 nr_begin_data++;
7966 if (TODO_SPEC (insn) & BEGIN_CONTROL)
7967 nr_begin_control++;
7969 create_check_block_twin (insn, false);
7971 TODO_SPEC (insn) &= ~BEGIN_SPEC;
7974 static void haifa_init_insn (rtx_insn *);
7976 /* Generates recovery code for BE_IN speculative INSN. */
7977 static void
7978 add_to_speculative_block (rtx_insn *insn)
7980 ds_t ts;
7981 sd_iterator_def sd_it;
7982 dep_t dep;
7983 auto_vec<rtx_insn *, 10> twins;
7985 ts = TODO_SPEC (insn);
7986 gcc_assert (!(ts & ~BE_IN_SPEC));
7988 if (ts & BE_IN_DATA)
7989 nr_be_in_data++;
7990 if (ts & BE_IN_CONTROL)
7991 nr_be_in_control++;
7993 TODO_SPEC (insn) &= ~BE_IN_SPEC;
7994 gcc_assert (!TODO_SPEC (insn));
7996 DONE_SPEC (insn) |= ts;
7998 /* First we convert all simple checks to branchy. */
7999 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
8000 sd_iterator_cond (&sd_it, &dep);)
8002 rtx_insn *check = DEP_PRO (dep);
8004 if (IS_SPECULATION_SIMPLE_CHECK_P (check))
8006 create_check_block_twin (check, true);
8008 /* Restart search. */
8009 sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
8011 else
8012 /* Continue search. */
8013 sd_iterator_next (&sd_it);
8016 auto_vec<rtx_insn *> priorities_roots;
8017 clear_priorities (insn, &priorities_roots);
8019 while (1)
8021 rtx_insn *check, *twin;
8022 basic_block rec;
8024 /* Get the first backward dependency of INSN. */
8025 sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
8026 if (!sd_iterator_cond (&sd_it, &dep))
8027 /* INSN has no backward dependencies left. */
8028 break;
8030 gcc_assert ((DEP_STATUS (dep) & BEGIN_SPEC) == 0
8031 && (DEP_STATUS (dep) & BE_IN_SPEC) != 0
8032 && (DEP_STATUS (dep) & DEP_TYPES) == DEP_TRUE);
8034 check = DEP_PRO (dep);
8036 gcc_assert (!IS_SPECULATION_CHECK_P (check) && !ORIG_PAT (check)
8037 && QUEUE_INDEX (check) == QUEUE_NOWHERE);
8039 rec = BLOCK_FOR_INSN (check);
8041 twin = emit_insn_before (copy_insn (PATTERN (insn)), BB_END (rec));
8042 haifa_init_insn (twin);
8044 sd_copy_back_deps (twin, insn, true);
8046 if (sched_verbose && spec_info->dump)
8047 /* INSN_BB (insn) isn't determined for twin insns yet.
8048 So we can't use current_sched_info->print_insn. */
8049 fprintf (spec_info->dump, ";;\t\tGenerated twin insn : %d/rec%d\n",
8050 INSN_UID (twin), rec->index);
8052 twins.safe_push (twin);
8054 /* Add dependences between TWIN and all appropriate
8055 instructions from REC. */
8056 FOR_EACH_DEP (insn, SD_LIST_SPEC_BACK, sd_it, dep)
8058 rtx_insn *pro = DEP_PRO (dep);
8060 gcc_assert (DEP_TYPE (dep) == REG_DEP_TRUE);
8062 /* INSN might have dependencies from the instructions from
8063 several recovery blocks. At this iteration we process those
8064 producers that reside in REC. */
8065 if (BLOCK_FOR_INSN (pro) == rec)
8067 dep_def _new_dep, *new_dep = &_new_dep;
8069 init_dep (new_dep, pro, twin, REG_DEP_TRUE);
8070 sd_add_dep (new_dep, false);
8074 process_insn_forw_deps_be_in_spec (insn, twin, ts);
8076 /* Remove all dependencies between INSN and insns in REC. */
8077 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
8078 sd_iterator_cond (&sd_it, &dep);)
8080 rtx_insn *pro = DEP_PRO (dep);
8082 if (BLOCK_FOR_INSN (pro) == rec)
8083 sd_delete_dep (sd_it);
8084 else
8085 sd_iterator_next (&sd_it);
8089 /* We couldn't have added the dependencies between INSN and TWINS earlier
8090 because that would make TWINS appear in the INSN_BACK_DEPS (INSN). */
8091 unsigned int i;
8092 rtx_insn *twin;
8093 FOR_EACH_VEC_ELT_REVERSE (twins, i, twin)
8095 dep_def _new_dep, *new_dep = &_new_dep;
8097 init_dep (new_dep, insn, twin, REG_DEP_OUTPUT);
8098 sd_add_dep (new_dep, false);
8101 calc_priorities (priorities_roots);
8104 /* Extends and fills with zeros (only the new part) array pointed to by P. */
8105 void *
8106 xrecalloc (void *p, size_t new_nmemb, size_t old_nmemb, size_t size)
8108 gcc_assert (new_nmemb >= old_nmemb);
8109 p = XRESIZEVAR (void, p, new_nmemb * size);
8110 memset (((char *) p) + old_nmemb * size, 0, (new_nmemb - old_nmemb) * size);
8111 return p;
8114 /* Helper function.
8115 Find fallthru edge from PRED. */
8116 edge
8117 find_fallthru_edge_from (basic_block pred)
8119 edge e;
8120 basic_block succ;
8122 succ = pred->next_bb;
8123 gcc_assert (succ->prev_bb == pred);
8125 if (EDGE_COUNT (pred->succs) <= EDGE_COUNT (succ->preds))
8127 e = find_fallthru_edge (pred->succs);
8129 if (e)
8131 gcc_assert (e->dest == succ);
8132 return e;
8135 else
8137 e = find_fallthru_edge (succ->preds);
8139 if (e)
8141 gcc_assert (e->src == pred);
8142 return e;
8146 return NULL;
8149 /* Extend per basic block data structures. */
8150 static void
8151 sched_extend_bb (void)
8153 /* The following is done to keep current_sched_info->next_tail non null. */
8154 rtx_insn *end = BB_END (EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb);
8155 rtx_insn *insn = DEBUG_INSN_P (end) ? prev_nondebug_insn (end) : end;
8156 if (NEXT_INSN (end) == 0
8157 || (!NOTE_P (insn)
8158 && !LABEL_P (insn)
8159 /* Don't emit a NOTE if it would end up before a BARRIER. */
8160 && !BARRIER_P (NEXT_INSN (end))))
8162 rtx_note *note = emit_note_after (NOTE_INSN_DELETED, end);
8163 /* Make note appear outside BB. */
8164 set_block_for_insn (note, NULL);
8165 BB_END (EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb) = end;
8169 /* Init per basic block data structures. */
8170 void
8171 sched_init_bbs (void)
8173 sched_extend_bb ();
8176 /* Initialize BEFORE_RECOVERY variable. */
8177 static void
8178 init_before_recovery (basic_block *before_recovery_ptr)
8180 basic_block last;
8181 edge e;
8183 last = EXIT_BLOCK_PTR_FOR_FN (cfun)->prev_bb;
8184 e = find_fallthru_edge_from (last);
8186 if (e)
8188 /* We create two basic blocks:
8189 1. Single instruction block is inserted right after E->SRC
8190 and has jump to
8191 2. Empty block right before EXIT_BLOCK.
8192 Between these two blocks recovery blocks will be emitted. */
8194 basic_block single, empty;
8196 /* If the fallthrough edge to exit we've found is from the block we've
8197 created before, don't do anything more. */
8198 if (last == after_recovery)
8199 return;
8201 adding_bb_to_current_region_p = false;
8203 single = sched_create_empty_bb (last);
8204 empty = sched_create_empty_bb (single);
8206 /* Add new blocks to the root loop. */
8207 if (current_loops != NULL)
8209 add_bb_to_loop (single, (*current_loops->larray)[0]);
8210 add_bb_to_loop (empty, (*current_loops->larray)[0]);
8213 single->count = last->count;
8214 empty->count = last->count;
8215 single->frequency = last->frequency;
8216 empty->frequency = last->frequency;
8217 BB_COPY_PARTITION (single, last);
8218 BB_COPY_PARTITION (empty, last);
8220 redirect_edge_succ (e, single);
8221 make_single_succ_edge (single, empty, 0);
8222 make_single_succ_edge (empty, EXIT_BLOCK_PTR_FOR_FN (cfun),
8223 EDGE_FALLTHRU);
8225 rtx_code_label *label = block_label (empty);
8226 rtx_jump_insn *x = emit_jump_insn_after (targetm.gen_jump (label),
8227 BB_END (single));
8228 JUMP_LABEL (x) = label;
8229 LABEL_NUSES (label)++;
8230 haifa_init_insn (x);
8232 emit_barrier_after (x);
8234 sched_init_only_bb (empty, NULL);
8235 sched_init_only_bb (single, NULL);
8236 sched_extend_bb ();
8238 adding_bb_to_current_region_p = true;
8239 before_recovery = single;
8240 after_recovery = empty;
8242 if (before_recovery_ptr)
8243 *before_recovery_ptr = before_recovery;
8245 if (sched_verbose >= 2 && spec_info->dump)
8246 fprintf (spec_info->dump,
8247 ";;\t\tFixed fallthru to EXIT : %d->>%d->%d->>EXIT\n",
8248 last->index, single->index, empty->index);
8250 else
8251 before_recovery = last;
8254 /* Returns new recovery block. */
8255 basic_block
8256 sched_create_recovery_block (basic_block *before_recovery_ptr)
8258 rtx_insn *barrier;
8259 basic_block rec;
8261 haifa_recovery_bb_recently_added_p = true;
8262 haifa_recovery_bb_ever_added_p = true;
8264 init_before_recovery (before_recovery_ptr);
8266 barrier = get_last_bb_insn (before_recovery);
8267 gcc_assert (BARRIER_P (barrier));
8269 rtx_insn *label = emit_label_after (gen_label_rtx (), barrier);
8271 rec = create_basic_block (label, label, before_recovery);
8273 /* A recovery block always ends with an unconditional jump. */
8274 emit_barrier_after (BB_END (rec));
8276 if (BB_PARTITION (before_recovery) != BB_UNPARTITIONED)
8277 BB_SET_PARTITION (rec, BB_COLD_PARTITION);
8279 if (sched_verbose && spec_info->dump)
8280 fprintf (spec_info->dump, ";;\t\tGenerated recovery block rec%d\n",
8281 rec->index);
8283 return rec;
8286 /* Create edges: FIRST_BB -> REC; FIRST_BB -> SECOND_BB; REC -> SECOND_BB
8287 and emit necessary jumps. */
8288 void
8289 sched_create_recovery_edges (basic_block first_bb, basic_block rec,
8290 basic_block second_bb)
8292 int edge_flags;
8294 /* This is fixing of incoming edge. */
8295 /* ??? Which other flags should be specified? */
8296 if (BB_PARTITION (first_bb) != BB_PARTITION (rec))
8297 /* Partition type is the same, if it is "unpartitioned". */
8298 edge_flags = EDGE_CROSSING;
8299 else
8300 edge_flags = 0;
8302 make_edge (first_bb, rec, edge_flags);
8303 rtx_code_label *label = block_label (second_bb);
8304 rtx_jump_insn *jump = emit_jump_insn_after (targetm.gen_jump (label),
8305 BB_END (rec));
8306 JUMP_LABEL (jump) = label;
8307 LABEL_NUSES (label)++;
8309 if (BB_PARTITION (second_bb) != BB_PARTITION (rec))
8310 /* Partition type is the same, if it is "unpartitioned". */
8312 /* Rewritten from cfgrtl.c. */
8313 if (flag_reorder_blocks_and_partition
8314 && targetm_common.have_named_sections)
8316 /* We don't need the same note for the check because
8317 any_condjump_p (check) == true. */
8318 CROSSING_JUMP_P (jump) = 1;
8320 edge_flags = EDGE_CROSSING;
8322 else
8323 edge_flags = 0;
8325 make_single_succ_edge (rec, second_bb, edge_flags);
8326 if (dom_info_available_p (CDI_DOMINATORS))
8327 set_immediate_dominator (CDI_DOMINATORS, rec, first_bb);
8330 /* This function creates recovery code for INSN. If MUTATE_P is nonzero,
8331 INSN is a simple check, that should be converted to branchy one. */
8332 static void
8333 create_check_block_twin (rtx_insn *insn, bool mutate_p)
8335 basic_block rec;
8336 rtx_insn *label, *check, *twin;
8337 rtx check_pat;
8338 ds_t fs;
8339 sd_iterator_def sd_it;
8340 dep_t dep;
8341 dep_def _new_dep, *new_dep = &_new_dep;
8342 ds_t todo_spec;
8344 gcc_assert (ORIG_PAT (insn) != NULL_RTX);
8346 if (!mutate_p)
8347 todo_spec = TODO_SPEC (insn);
8348 else
8350 gcc_assert (IS_SPECULATION_SIMPLE_CHECK_P (insn)
8351 && (TODO_SPEC (insn) & SPECULATIVE) == 0);
8353 todo_spec = CHECK_SPEC (insn);
8356 todo_spec &= SPECULATIVE;
8358 /* Create recovery block. */
8359 if (mutate_p || targetm.sched.needs_block_p (todo_spec))
8361 rec = sched_create_recovery_block (NULL);
8362 label = BB_HEAD (rec);
8364 else
8366 rec = EXIT_BLOCK_PTR_FOR_FN (cfun);
8367 label = NULL;
8370 /* Emit CHECK. */
8371 check_pat = targetm.sched.gen_spec_check (insn, label, todo_spec);
8373 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun))
8375 /* To have mem_reg alive at the beginning of second_bb,
8376 we emit check BEFORE insn, so insn after splitting
8377 insn will be at the beginning of second_bb, which will
8378 provide us with the correct life information. */
8379 check = emit_jump_insn_before (check_pat, insn);
8380 JUMP_LABEL (check) = label;
8381 LABEL_NUSES (label)++;
8383 else
8384 check = emit_insn_before (check_pat, insn);
8386 /* Extend data structures. */
8387 haifa_init_insn (check);
8389 /* CHECK is being added to current region. Extend ready list. */
8390 gcc_assert (sched_ready_n_insns != -1);
8391 sched_extend_ready_list (sched_ready_n_insns + 1);
8393 if (current_sched_info->add_remove_insn)
8394 current_sched_info->add_remove_insn (insn, 0);
8396 RECOVERY_BLOCK (check) = rec;
8398 if (sched_verbose && spec_info->dump)
8399 fprintf (spec_info->dump, ";;\t\tGenerated check insn : %s\n",
8400 (*current_sched_info->print_insn) (check, 0));
8402 gcc_assert (ORIG_PAT (insn));
8404 /* Initialize TWIN (twin is a duplicate of original instruction
8405 in the recovery block). */
8406 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun))
8408 sd_iterator_def sd_it;
8409 dep_t dep;
8411 FOR_EACH_DEP (insn, SD_LIST_RES_BACK, sd_it, dep)
8412 if ((DEP_STATUS (dep) & DEP_OUTPUT) != 0)
8414 struct _dep _dep2, *dep2 = &_dep2;
8416 init_dep (dep2, DEP_PRO (dep), check, REG_DEP_TRUE);
8418 sd_add_dep (dep2, true);
8421 twin = emit_insn_after (ORIG_PAT (insn), BB_END (rec));
8422 haifa_init_insn (twin);
8424 if (sched_verbose && spec_info->dump)
8425 /* INSN_BB (insn) isn't determined for twin insns yet.
8426 So we can't use current_sched_info->print_insn. */
8427 fprintf (spec_info->dump, ";;\t\tGenerated twin insn : %d/rec%d\n",
8428 INSN_UID (twin), rec->index);
8430 else
8432 ORIG_PAT (check) = ORIG_PAT (insn);
8433 HAS_INTERNAL_DEP (check) = 1;
8434 twin = check;
8435 /* ??? We probably should change all OUTPUT dependencies to
8436 (TRUE | OUTPUT). */
8439 /* Copy all resolved back dependencies of INSN to TWIN. This will
8440 provide correct value for INSN_TICK (TWIN). */
8441 sd_copy_back_deps (twin, insn, true);
8443 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun))
8444 /* In case of branchy check, fix CFG. */
8446 basic_block first_bb, second_bb;
8447 rtx_insn *jump;
8449 first_bb = BLOCK_FOR_INSN (check);
8450 second_bb = sched_split_block (first_bb, check);
8452 sched_create_recovery_edges (first_bb, rec, second_bb);
8454 sched_init_only_bb (second_bb, first_bb);
8455 sched_init_only_bb (rec, EXIT_BLOCK_PTR_FOR_FN (cfun));
8457 jump = BB_END (rec);
8458 haifa_init_insn (jump);
8461 /* Move backward dependences from INSN to CHECK and
8462 move forward dependences from INSN to TWIN. */
8464 /* First, create dependencies between INSN's producers and CHECK & TWIN. */
8465 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep)
8467 rtx_insn *pro = DEP_PRO (dep);
8468 ds_t ds;
8470 /* If BEGIN_DATA: [insn ~~TRUE~~> producer]:
8471 check --TRUE--> producer ??? or ANTI ???
8472 twin --TRUE--> producer
8473 twin --ANTI--> check
8475 If BEGIN_CONTROL: [insn ~~ANTI~~> producer]:
8476 check --ANTI--> producer
8477 twin --ANTI--> producer
8478 twin --ANTI--> check
8480 If BE_IN_SPEC: [insn ~~TRUE~~> producer]:
8481 check ~~TRUE~~> producer
8482 twin ~~TRUE~~> producer
8483 twin --ANTI--> check */
8485 ds = DEP_STATUS (dep);
8487 if (ds & BEGIN_SPEC)
8489 gcc_assert (!mutate_p);
8490 ds &= ~BEGIN_SPEC;
8493 init_dep_1 (new_dep, pro, check, DEP_TYPE (dep), ds);
8494 sd_add_dep (new_dep, false);
8496 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun))
8498 DEP_CON (new_dep) = twin;
8499 sd_add_dep (new_dep, false);
8503 /* Second, remove backward dependencies of INSN. */
8504 for (sd_it = sd_iterator_start (insn, SD_LIST_SPEC_BACK);
8505 sd_iterator_cond (&sd_it, &dep);)
8507 if ((DEP_STATUS (dep) & BEGIN_SPEC)
8508 || mutate_p)
8509 /* We can delete this dep because we overcome it with
8510 BEGIN_SPECULATION. */
8511 sd_delete_dep (sd_it);
8512 else
8513 sd_iterator_next (&sd_it);
8516 /* Future Speculations. Determine what BE_IN speculations will be like. */
8517 fs = 0;
8519 /* Fields (DONE_SPEC (x) & BEGIN_SPEC) and CHECK_SPEC (x) are set only
8520 here. */
8522 gcc_assert (!DONE_SPEC (insn));
8524 if (!mutate_p)
8526 ds_t ts = TODO_SPEC (insn);
8528 DONE_SPEC (insn) = ts & BEGIN_SPEC;
8529 CHECK_SPEC (check) = ts & BEGIN_SPEC;
8531 /* Luckiness of future speculations solely depends upon initial
8532 BEGIN speculation. */
8533 if (ts & BEGIN_DATA)
8534 fs = set_dep_weak (fs, BE_IN_DATA, get_dep_weak (ts, BEGIN_DATA));
8535 if (ts & BEGIN_CONTROL)
8536 fs = set_dep_weak (fs, BE_IN_CONTROL,
8537 get_dep_weak (ts, BEGIN_CONTROL));
8539 else
8540 CHECK_SPEC (check) = CHECK_SPEC (insn);
8542 /* Future speculations: call the helper. */
8543 process_insn_forw_deps_be_in_spec (insn, twin, fs);
8545 if (rec != EXIT_BLOCK_PTR_FOR_FN (cfun))
8547 /* Which types of dependencies should we use here is,
8548 generally, machine-dependent question... But, for now,
8549 it is not. */
8551 if (!mutate_p)
8553 init_dep (new_dep, insn, check, REG_DEP_TRUE);
8554 sd_add_dep (new_dep, false);
8556 init_dep (new_dep, insn, twin, REG_DEP_OUTPUT);
8557 sd_add_dep (new_dep, false);
8559 else
8561 if (spec_info->dump)
8562 fprintf (spec_info->dump, ";;\t\tRemoved simple check : %s\n",
8563 (*current_sched_info->print_insn) (insn, 0));
8565 /* Remove all dependencies of the INSN. */
8567 sd_it = sd_iterator_start (insn, (SD_LIST_FORW
8568 | SD_LIST_BACK
8569 | SD_LIST_RES_BACK));
8570 while (sd_iterator_cond (&sd_it, &dep))
8571 sd_delete_dep (sd_it);
8574 /* If former check (INSN) already was moved to the ready (or queue)
8575 list, add new check (CHECK) there too. */
8576 if (QUEUE_INDEX (insn) != QUEUE_NOWHERE)
8577 try_ready (check);
8579 /* Remove old check from instruction stream and free its
8580 data. */
8581 sched_remove_insn (insn);
8584 init_dep (new_dep, check, twin, REG_DEP_ANTI);
8585 sd_add_dep (new_dep, false);
8587 else
8589 init_dep_1 (new_dep, insn, check, REG_DEP_TRUE, DEP_TRUE | DEP_OUTPUT);
8590 sd_add_dep (new_dep, false);
8593 if (!mutate_p)
8594 /* Fix priorities. If MUTATE_P is nonzero, this is not necessary,
8595 because it'll be done later in add_to_speculative_block. */
8597 auto_vec<rtx_insn *> priorities_roots;
8599 clear_priorities (twin, &priorities_roots);
8600 calc_priorities (priorities_roots);
8604 /* Removes dependency between instructions in the recovery block REC
8605 and usual region instructions. It keeps inner dependences so it
8606 won't be necessary to recompute them. */
8607 static void
8608 fix_recovery_deps (basic_block rec)
8610 rtx_insn *note, *insn, *jump;
8611 auto_vec<rtx_insn *, 10> ready_list;
8612 bitmap_head in_ready;
8614 bitmap_initialize (&in_ready, 0);
8616 /* NOTE - a basic block note. */
8617 note = NEXT_INSN (BB_HEAD (rec));
8618 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note));
8619 insn = BB_END (rec);
8620 gcc_assert (JUMP_P (insn));
8621 insn = PREV_INSN (insn);
8625 sd_iterator_def sd_it;
8626 dep_t dep;
8628 for (sd_it = sd_iterator_start (insn, SD_LIST_FORW);
8629 sd_iterator_cond (&sd_it, &dep);)
8631 rtx_insn *consumer = DEP_CON (dep);
8633 if (BLOCK_FOR_INSN (consumer) != rec)
8635 sd_delete_dep (sd_it);
8637 if (bitmap_set_bit (&in_ready, INSN_LUID (consumer)))
8638 ready_list.safe_push (consumer);
8640 else
8642 gcc_assert ((DEP_STATUS (dep) & DEP_TYPES) == DEP_TRUE);
8644 sd_iterator_next (&sd_it);
8648 insn = PREV_INSN (insn);
8650 while (insn != note);
8652 bitmap_clear (&in_ready);
8654 /* Try to add instructions to the ready or queue list. */
8655 unsigned int i;
8656 rtx_insn *temp;
8657 FOR_EACH_VEC_ELT_REVERSE (ready_list, i, temp)
8658 try_ready (temp);
8660 /* Fixing jump's dependences. */
8661 insn = BB_HEAD (rec);
8662 jump = BB_END (rec);
8664 gcc_assert (LABEL_P (insn));
8665 insn = NEXT_INSN (insn);
8667 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (insn));
8668 add_jump_dependencies (insn, jump);
8671 /* Change pattern of INSN to NEW_PAT. Invalidate cached haifa
8672 instruction data. */
8673 static bool
8674 haifa_change_pattern (rtx_insn *insn, rtx new_pat)
8676 int t;
8678 t = validate_change (insn, &PATTERN (insn), new_pat, 0);
8679 if (!t)
8680 return false;
8682 update_insn_after_change (insn);
8683 return true;
8686 /* -1 - can't speculate,
8687 0 - for speculation with REQUEST mode it is OK to use
8688 current instruction pattern,
8689 1 - need to change pattern for *NEW_PAT to be speculative. */
8691 sched_speculate_insn (rtx_insn *insn, ds_t request, rtx *new_pat)
8693 gcc_assert (current_sched_info->flags & DO_SPECULATION
8694 && (request & SPECULATIVE)
8695 && sched_insn_is_legitimate_for_speculation_p (insn, request));
8697 if ((request & spec_info->mask) != request)
8698 return -1;
8700 if (request & BE_IN_SPEC
8701 && !(request & BEGIN_SPEC))
8702 return 0;
8704 return targetm.sched.speculate_insn (insn, request, new_pat);
8707 static int
8708 haifa_speculate_insn (rtx_insn *insn, ds_t request, rtx *new_pat)
8710 gcc_assert (sched_deps_info->generate_spec_deps
8711 && !IS_SPECULATION_CHECK_P (insn));
8713 if (HAS_INTERNAL_DEP (insn)
8714 || SCHED_GROUP_P (insn))
8715 return -1;
8717 return sched_speculate_insn (insn, request, new_pat);
8720 /* Print some information about block BB, which starts with HEAD and
8721 ends with TAIL, before scheduling it.
8722 I is zero, if scheduler is about to start with the fresh ebb. */
8723 static void
8724 dump_new_block_header (int i, basic_block bb, rtx_insn *head, rtx_insn *tail)
8726 if (!i)
8727 fprintf (sched_dump,
8728 ";; ======================================================\n");
8729 else
8730 fprintf (sched_dump,
8731 ";; =====================ADVANCING TO=====================\n");
8732 fprintf (sched_dump,
8733 ";; -- basic block %d from %d to %d -- %s reload\n",
8734 bb->index, INSN_UID (head), INSN_UID (tail),
8735 (reload_completed ? "after" : "before"));
8736 fprintf (sched_dump,
8737 ";; ======================================================\n");
8738 fprintf (sched_dump, "\n");
8741 /* Unlink basic block notes and labels and saves them, so they
8742 can be easily restored. We unlink basic block notes in EBB to
8743 provide back-compatibility with the previous code, as target backends
8744 assume, that there'll be only instructions between
8745 current_sched_info->{head and tail}. We restore these notes as soon
8746 as we can.
8747 FIRST (LAST) is the first (last) basic block in the ebb.
8748 NB: In usual case (FIRST == LAST) nothing is really done. */
8749 void
8750 unlink_bb_notes (basic_block first, basic_block last)
8752 /* We DON'T unlink basic block notes of the first block in the ebb. */
8753 if (first == last)
8754 return;
8756 bb_header = XNEWVEC (rtx_insn *, last_basic_block_for_fn (cfun));
8758 /* Make a sentinel. */
8759 if (last->next_bb != EXIT_BLOCK_PTR_FOR_FN (cfun))
8760 bb_header[last->next_bb->index] = 0;
8762 first = first->next_bb;
8765 rtx_insn *prev, *label, *note, *next;
8767 label = BB_HEAD (last);
8768 if (LABEL_P (label))
8769 note = NEXT_INSN (label);
8770 else
8771 note = label;
8772 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note));
8774 prev = PREV_INSN (label);
8775 next = NEXT_INSN (note);
8776 gcc_assert (prev && next);
8778 SET_NEXT_INSN (prev) = next;
8779 SET_PREV_INSN (next) = prev;
8781 bb_header[last->index] = label;
8783 if (last == first)
8784 break;
8786 last = last->prev_bb;
8788 while (1);
8791 /* Restore basic block notes.
8792 FIRST is the first basic block in the ebb. */
8793 static void
8794 restore_bb_notes (basic_block first)
8796 if (!bb_header)
8797 return;
8799 /* We DON'T unlink basic block notes of the first block in the ebb. */
8800 first = first->next_bb;
8801 /* Remember: FIRST is actually a second basic block in the ebb. */
8803 while (first != EXIT_BLOCK_PTR_FOR_FN (cfun)
8804 && bb_header[first->index])
8806 rtx_insn *prev, *label, *note, *next;
8808 label = bb_header[first->index];
8809 prev = PREV_INSN (label);
8810 next = NEXT_INSN (prev);
8812 if (LABEL_P (label))
8813 note = NEXT_INSN (label);
8814 else
8815 note = label;
8816 gcc_assert (NOTE_INSN_BASIC_BLOCK_P (note));
8818 bb_header[first->index] = 0;
8820 SET_NEXT_INSN (prev) = label;
8821 SET_NEXT_INSN (note) = next;
8822 SET_PREV_INSN (next) = note;
8824 first = first->next_bb;
8827 free (bb_header);
8828 bb_header = 0;
8831 /* Helper function.
8832 Fix CFG after both in- and inter-block movement of
8833 control_flow_insn_p JUMP. */
8834 static void
8835 fix_jump_move (rtx_insn *jump)
8837 basic_block bb, jump_bb, jump_bb_next;
8839 bb = BLOCK_FOR_INSN (PREV_INSN (jump));
8840 jump_bb = BLOCK_FOR_INSN (jump);
8841 jump_bb_next = jump_bb->next_bb;
8843 gcc_assert (common_sched_info->sched_pass_id == SCHED_EBB_PASS
8844 || IS_SPECULATION_BRANCHY_CHECK_P (jump));
8846 if (!NOTE_INSN_BASIC_BLOCK_P (BB_END (jump_bb_next)))
8847 /* if jump_bb_next is not empty. */
8848 BB_END (jump_bb) = BB_END (jump_bb_next);
8850 if (BB_END (bb) != PREV_INSN (jump))
8851 /* Then there are instruction after jump that should be placed
8852 to jump_bb_next. */
8853 BB_END (jump_bb_next) = BB_END (bb);
8854 else
8855 /* Otherwise jump_bb_next is empty. */
8856 BB_END (jump_bb_next) = NEXT_INSN (BB_HEAD (jump_bb_next));
8858 /* To make assertion in move_insn happy. */
8859 BB_END (bb) = PREV_INSN (jump);
8861 update_bb_for_insn (jump_bb_next);
8864 /* Fix CFG after interblock movement of control_flow_insn_p JUMP. */
8865 static void
8866 move_block_after_check (rtx_insn *jump)
8868 basic_block bb, jump_bb, jump_bb_next;
8869 vec<edge, va_gc> *t;
8871 bb = BLOCK_FOR_INSN (PREV_INSN (jump));
8872 jump_bb = BLOCK_FOR_INSN (jump);
8873 jump_bb_next = jump_bb->next_bb;
8875 update_bb_for_insn (jump_bb);
8877 gcc_assert (IS_SPECULATION_CHECK_P (jump)
8878 || IS_SPECULATION_CHECK_P (BB_END (jump_bb_next)));
8880 unlink_block (jump_bb_next);
8881 link_block (jump_bb_next, bb);
8883 t = bb->succs;
8884 bb->succs = 0;
8885 move_succs (&(jump_bb->succs), bb);
8886 move_succs (&(jump_bb_next->succs), jump_bb);
8887 move_succs (&t, jump_bb_next);
8889 df_mark_solutions_dirty ();
8891 common_sched_info->fix_recovery_cfg
8892 (bb->index, jump_bb->index, jump_bb_next->index);
8895 /* Helper function for move_block_after_check.
8896 This functions attaches edge vector pointed to by SUCCSP to
8897 block TO. */
8898 static void
8899 move_succs (vec<edge, va_gc> **succsp, basic_block to)
8901 edge e;
8902 edge_iterator ei;
8904 gcc_assert (to->succs == 0);
8906 to->succs = *succsp;
8908 FOR_EACH_EDGE (e, ei, to->succs)
8909 e->src = to;
8911 *succsp = 0;
8914 /* Remove INSN from the instruction stream.
8915 INSN should have any dependencies. */
8916 static void
8917 sched_remove_insn (rtx_insn *insn)
8919 sd_finish_insn (insn);
8921 change_queue_index (insn, QUEUE_NOWHERE);
8922 current_sched_info->add_remove_insn (insn, 1);
8923 delete_insn (insn);
8926 /* Clear priorities of all instructions, that are forward dependent on INSN.
8927 Store in vector pointed to by ROOTS_PTR insns on which priority () should
8928 be invoked to initialize all cleared priorities. */
8929 static void
8930 clear_priorities (rtx_insn *insn, rtx_vec_t *roots_ptr)
8932 sd_iterator_def sd_it;
8933 dep_t dep;
8934 bool insn_is_root_p = true;
8936 gcc_assert (QUEUE_INDEX (insn) != QUEUE_SCHEDULED);
8938 FOR_EACH_DEP (insn, SD_LIST_BACK, sd_it, dep)
8940 rtx_insn *pro = DEP_PRO (dep);
8942 if (INSN_PRIORITY_STATUS (pro) >= 0
8943 && QUEUE_INDEX (insn) != QUEUE_SCHEDULED)
8945 /* If DEP doesn't contribute to priority then INSN itself should
8946 be added to priority roots. */
8947 if (contributes_to_priority_p (dep))
8948 insn_is_root_p = false;
8950 INSN_PRIORITY_STATUS (pro) = -1;
8951 clear_priorities (pro, roots_ptr);
8955 if (insn_is_root_p)
8956 roots_ptr->safe_push (insn);
8959 /* Recompute priorities of instructions, whose priorities might have been
8960 changed. ROOTS is a vector of instructions whose priority computation will
8961 trigger initialization of all cleared priorities. */
8962 static void
8963 calc_priorities (rtx_vec_t roots)
8965 int i;
8966 rtx_insn *insn;
8968 FOR_EACH_VEC_ELT (roots, i, insn)
8969 priority (insn);
8973 /* Add dependences between JUMP and other instructions in the recovery
8974 block. INSN is the first insn the recovery block. */
8975 static void
8976 add_jump_dependencies (rtx_insn *insn, rtx_insn *jump)
8980 insn = NEXT_INSN (insn);
8981 if (insn == jump)
8982 break;
8984 if (dep_list_size (insn, SD_LIST_FORW) == 0)
8986 dep_def _new_dep, *new_dep = &_new_dep;
8988 init_dep (new_dep, insn, jump, REG_DEP_ANTI);
8989 sd_add_dep (new_dep, false);
8992 while (1);
8994 gcc_assert (!sd_lists_empty_p (jump, SD_LIST_BACK));
8997 /* Extend data structures for logical insn UID. */
8998 void
8999 sched_extend_luids (void)
9001 int new_luids_max_uid = get_max_uid () + 1;
9003 sched_luids.safe_grow_cleared (new_luids_max_uid);
9006 /* Initialize LUID for INSN. */
9007 void
9008 sched_init_insn_luid (rtx_insn *insn)
9010 int i = INSN_P (insn) ? 1 : common_sched_info->luid_for_non_insn (insn);
9011 int luid;
9013 if (i >= 0)
9015 luid = sched_max_luid;
9016 sched_max_luid += i;
9018 else
9019 luid = -1;
9021 SET_INSN_LUID (insn, luid);
9024 /* Initialize luids for BBS.
9025 The hook common_sched_info->luid_for_non_insn () is used to determine
9026 if notes, labels, etc. need luids. */
9027 void
9028 sched_init_luids (bb_vec_t bbs)
9030 int i;
9031 basic_block bb;
9033 sched_extend_luids ();
9034 FOR_EACH_VEC_ELT (bbs, i, bb)
9036 rtx_insn *insn;
9038 FOR_BB_INSNS (bb, insn)
9039 sched_init_insn_luid (insn);
9043 /* Free LUIDs. */
9044 void
9045 sched_finish_luids (void)
9047 sched_luids.release ();
9048 sched_max_luid = 1;
9051 /* Return logical uid of INSN. Helpful while debugging. */
9053 insn_luid (rtx_insn *insn)
9055 return INSN_LUID (insn);
9058 /* Extend per insn data in the target. */
9059 void
9060 sched_extend_target (void)
9062 if (targetm.sched.h_i_d_extended)
9063 targetm.sched.h_i_d_extended ();
9066 /* Extend global scheduler structures (those, that live across calls to
9067 schedule_block) to include information about just emitted INSN. */
9068 static void
9069 extend_h_i_d (void)
9071 int reserve = (get_max_uid () + 1 - h_i_d.length ());
9072 if (reserve > 0
9073 && ! h_i_d.space (reserve))
9075 h_i_d.safe_grow_cleared (3 * get_max_uid () / 2);
9076 sched_extend_target ();
9080 /* Initialize h_i_d entry of the INSN with default values.
9081 Values, that are not explicitly initialized here, hold zero. */
9082 static void
9083 init_h_i_d (rtx_insn *insn)
9085 if (INSN_LUID (insn) > 0)
9087 INSN_COST (insn) = -1;
9088 QUEUE_INDEX (insn) = QUEUE_NOWHERE;
9089 INSN_TICK (insn) = INVALID_TICK;
9090 INSN_EXACT_TICK (insn) = INVALID_TICK;
9091 INTER_TICK (insn) = INVALID_TICK;
9092 TODO_SPEC (insn) = HARD_DEP;
9093 INSN_AUTOPREF_MULTIPASS_DATA (insn)[0].status
9094 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED;
9095 INSN_AUTOPREF_MULTIPASS_DATA (insn)[1].status
9096 = AUTOPREF_MULTIPASS_DATA_UNINITIALIZED;
9100 /* Initialize haifa_insn_data for BBS. */
9101 void
9102 haifa_init_h_i_d (bb_vec_t bbs)
9104 int i;
9105 basic_block bb;
9107 extend_h_i_d ();
9108 FOR_EACH_VEC_ELT (bbs, i, bb)
9110 rtx_insn *insn;
9112 FOR_BB_INSNS (bb, insn)
9113 init_h_i_d (insn);
9117 /* Finalize haifa_insn_data. */
9118 void
9119 haifa_finish_h_i_d (void)
9121 int i;
9122 haifa_insn_data_t data;
9123 reg_use_data *use, *next_use;
9124 reg_set_data *set, *next_set;
9126 FOR_EACH_VEC_ELT (h_i_d, i, data)
9128 free (data->max_reg_pressure);
9129 free (data->reg_pressure);
9130 for (use = data->reg_use_list; use != NULL; use = next_use)
9132 next_use = use->next_insn_use;
9133 free (use);
9135 for (set = data->reg_set_list; set != NULL; set = next_set)
9137 next_set = set->next_insn_set;
9138 free (set);
9142 h_i_d.release ();
9145 /* Init data for the new insn INSN. */
9146 static void
9147 haifa_init_insn (rtx_insn *insn)
9149 gcc_assert (insn != NULL);
9151 sched_extend_luids ();
9152 sched_init_insn_luid (insn);
9153 sched_extend_target ();
9154 sched_deps_init (false);
9155 extend_h_i_d ();
9156 init_h_i_d (insn);
9158 if (adding_bb_to_current_region_p)
9160 sd_init_insn (insn);
9162 /* Extend dependency caches by one element. */
9163 extend_dependency_caches (1, false);
9165 if (sched_pressure != SCHED_PRESSURE_NONE)
9166 init_insn_reg_pressure_info (insn);
9169 /* Init data for the new basic block BB which comes after AFTER. */
9170 static void
9171 haifa_init_only_bb (basic_block bb, basic_block after)
9173 gcc_assert (bb != NULL);
9175 sched_init_bbs ();
9177 if (common_sched_info->add_block)
9178 /* This changes only data structures of the front-end. */
9179 common_sched_info->add_block (bb, after);
9182 /* A generic version of sched_split_block (). */
9183 basic_block
9184 sched_split_block_1 (basic_block first_bb, rtx after)
9186 edge e;
9188 e = split_block (first_bb, after);
9189 gcc_assert (e->src == first_bb);
9191 /* sched_split_block emits note if *check == BB_END. Probably it
9192 is better to rip that note off. */
9194 return e->dest;
9197 /* A generic version of sched_create_empty_bb (). */
9198 basic_block
9199 sched_create_empty_bb_1 (basic_block after)
9201 return create_empty_bb (after);
9204 /* Insert PAT as an INSN into the schedule and update the necessary data
9205 structures to account for it. */
9206 rtx_insn *
9207 sched_emit_insn (rtx pat)
9209 rtx_insn *insn = emit_insn_before (pat, first_nonscheduled_insn ());
9210 haifa_init_insn (insn);
9212 if (current_sched_info->add_remove_insn)
9213 current_sched_info->add_remove_insn (insn, 0);
9215 (*current_sched_info->begin_schedule_ready) (insn);
9216 scheduled_insns.safe_push (insn);
9218 last_scheduled_insn = insn;
9219 return insn;
9222 /* This function returns a candidate satisfying dispatch constraints from
9223 the ready list. */
9225 static rtx_insn *
9226 ready_remove_first_dispatch (struct ready_list *ready)
9228 int i;
9229 rtx_insn *insn = ready_element (ready, 0);
9231 if (ready->n_ready == 1
9232 || !INSN_P (insn)
9233 || INSN_CODE (insn) < 0
9234 || !active_insn_p (insn)
9235 || targetm.sched.dispatch (insn, FITS_DISPATCH_WINDOW))
9236 return ready_remove_first (ready);
9238 for (i = 1; i < ready->n_ready; i++)
9240 insn = ready_element (ready, i);
9242 if (!INSN_P (insn)
9243 || INSN_CODE (insn) < 0
9244 || !active_insn_p (insn))
9245 continue;
9247 if (targetm.sched.dispatch (insn, FITS_DISPATCH_WINDOW))
9249 /* Return ith element of ready. */
9250 insn = ready_remove (ready, i);
9251 return insn;
9255 if (targetm.sched.dispatch (NULL, DISPATCH_VIOLATION))
9256 return ready_remove_first (ready);
9258 for (i = 1; i < ready->n_ready; i++)
9260 insn = ready_element (ready, i);
9262 if (!INSN_P (insn)
9263 || INSN_CODE (insn) < 0
9264 || !active_insn_p (insn))
9265 continue;
9267 /* Return i-th element of ready. */
9268 if (targetm.sched.dispatch (insn, IS_CMP))
9269 return ready_remove (ready, i);
9272 return ready_remove_first (ready);
9275 /* Get number of ready insn in the ready list. */
9278 number_in_ready (void)
9280 return ready.n_ready;
9283 /* Get number of ready's in the ready list. */
9285 rtx_insn *
9286 get_ready_element (int i)
9288 return ready_element (&ready, i);
9291 #endif /* INSN_SCHEDULING */