benchtest: script to compare two benchmarks
commit0cd2828695cc328aa1b48379436d15c39d433076
authorSiddhesh Poyarekar <siddhesh@redhat.com>
Mon, 1 Jun 2015 17:44:11 +0000 (1 23:14 +0530)
committerSiddhesh Poyarekar <siddhesh@redhat.com>
Mon, 1 Jun 2015 17:44:11 +0000 (1 23:14 +0530)
treec7f6fc5aab240ffdb8c7f321056ea9185c3ec154
parent0994b9b6f685d460ee72170824a5393b592dc3c5
benchtest: script to compare two benchmarks

This script is a sample implementation that uses import_bench to
construct two benchmark objects and compare them.  If detailed timing
information is available (when one does `make DETAILED=1 bench`), it
writes out graphs for all functions it benchmarks and prints
significant differences in timings of the two benchmark runs.  If
detailed timing information is not available, it points out
significant differences in aggregate times.

Call this script as follows:

  compare_bench.py schema_file.json bench1.out bench2.out

Alternatively, if one wants to set a different threshold for warnings
(default is a 10% difference):

  compare_bench.py schema_file.json bench1.out bench2.out 25

The threshold in the example above is 25%.  schema_file.json is the
JSON schema (which is $srcdir/benchtests/scripts/benchout.schema.json
for the benchmark output file) and bench1.out and bench2.out are the
two benchmark output files to compare.

The key functionality here is the compress_timings function which
groups together points that are close together into a single point
that is the mean of all its representative points.  Any point in such
a group is at most 1.5x the smallest point in that group.  The
detailed derivation is a comment in the function.

* benchtests/scripts/compare_bench.py: New file.
* benchtests/scripts/import_bench.py (mean): New function.
(split_list): Likewise.
(do_for_all_timings): Likewise.
(compress_timings): Likewise.
ChangeLog
benchtests/scripts/compare_bench.py [new file with mode: 0755]
benchtests/scripts/import_bench.py