Bug
1830741 - Add custom comparators to mach-try-perf. r=perftest-reviewers,kshampur
This patch adds the ability to run custom comparisons through "comparators". The code is modified to create a BasePerfComparator that provides the existing default behaviour. A new method, and command-line arguments are added to accept either a path to a custom comparator, or a name of a builtin comparator.
To add new builtin comparators, they must be added to the perfcomparators.py file with an `@comparator` decorator. It must also subclass the BasePerfComparator. With this, a BenchmarkComparator is added that lets us pass custom options to raptor through the PERF_FLAGS environment variable (see --extra-args). To do this, some code needed to be moved around such as where the try_config is created. This lets us reset the configuration between the base, and new revision if needed.
The BenchmarkComparator lets us pass PR links as arguments, or the actual benchmark repo, revision, and branch. The PR link gets parsed into the required info using Github API requests. Note that links to direct commits do not work at the moment as the API doesn't provide the branch information for those kinds of requests.
Depends on
D177030
Differential Revision: https://phabricator.services.mozilla.com/
D177031