• This PR adds the basic functionality to specify a list of custom conda channels using a new key in the JSON config file; the associated feature request issue is #405
• This actually looks fairly clean to me (and works for me), but there are probably some things that the core devs will know that I haven't thought of
• I suspect the way the channels are added may need slight modification -- installing packages is done in a single call to conda for performance reasons, and we may wish to do the same for adding channels (I currently loop over the input channels as you can see)
• need for tests? -- make sure this works when adding a single vs. multiple custom channels? mock the call to those channels? how much of a pain would this be to do? Maybe there's something simpler that can be done test-wise.

I run into a weird issue which I cannot track down (using asv v0.2.1), so I thought I'd ask here if it is a known or common problem.

If I run a simple test using asv dev with a parameter that can take three different values, I get the following output:

asv/> asv dev -b dipole_time
· Discovering benchmarks
· Running 1 total benchmarks (1 commits * 1 environments * 1 benchmarks)
[  0.00%] ·· Building for existing-py_home_dtr_anaconda3_bin_python
[  0.00%] ·· Benchmarking existing-py_home_dtr_anaconda3_bin_python
[100.00%] ··· Running model.Dipole.time_dipole_time                                    ok
[100.00%] ···· 
               ====== ==========
                loop            
               ------ ----------
                None   139.11ms 
                freq   322.62ms 
                off    139.09ms 
               ====== ==========

These times agree with the times I get if I run the tests simply with %timeit in an IPython console.

Now if I run asv run instead of asv dev,

asv/> asv run -b time_dipole_time
· Fetching recent changes
· Creating environments
· Discovering benchmarks
·· Uninstalling from conda-py3.6
·· Building for conda-py3.6
·· Installing into conda-py3.6.
· Running 1 total benchmarks (1 commits * 1 environments * 1 benchmarks)
[  0.00%] · For empymod commit hash b804477a:
[  0.00%] ·· Building for conda-py3.6.
[  0.00%] ·· Benchmarking conda-py3.6
[100.00%] ··· Running model.Dipole.time_dipole_time                          655.54ms;...

the result stored in b804477a-conda-py3.6.json is:

[snip]
    "results": {
        "model.Dipole.time_dipole_time": {
            "params": [
                [
                    "None",
                    "'freq'",
                    "'off'"
                ]
            ],
            "result": [
                0.6555442148,
                0.3023681289999999,
                0.6842122744999999
            ]
        }
[snip]

The times vastly differ between the asv dev and the asv run results. Any idea why that could be?

This is just a very experimental idea for a 'compare' command that can be used to compare two specific revisions, e.g:

$ asv compare 7810d6d7 19aa5743
· Fetching recent changes.

All benchmarks:

  before     after    ratio
  98.30ms   68.19ms      0.69  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_center
 139.58ms  108.97ms      0.78  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_exact
  98.94ms   67.96ms      0.69  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_subpixel_01
 105.35ms   73.63ms      0.70  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_subpixel_05
 116.30ms   86.23ms      0.74  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_subpixel_10
  48.15ms   51.00ms      1.06  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_center
  68.18ms   69.50ms      1.02  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_exact
  48.59ms   51.24ms      1.05  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_subpixel_01
  51.13ms   53.71ms      1.05  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_subpixel_05
  56.26ms   58.65ms      1.04  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_subpixel_10
    4.71s  231.49ms      0.05  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_center
    1.70s     1.67s      0.98  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_exact
    4.72s  242.42ms      0.05  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_subpixel_01
   failed  454.41ms       n/a  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_subpixel_05
   failed     1.13s       n/a  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_subpixel_10
...

Benchmarks are color-coded as green if they are faster by a given threshold, and red if they are slower. The threshold can be adjusted with the --threshold option.

The output can be split between benchmarks that got better, didn't change, or got worse, with the --split option:

$ asv compare 7810d6d7 19aa5743 --split -t 1.5
· Fetching recent changes..

Benchmarks that have improved:

  before     after    ratio
    4.71s  231.49ms      0.05  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_center
    4.72s  242.42ms      0.05  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_elli_ann_subpixel_01
   failed  454.41ms       n/a
[snip]
   failed  159.65ms       n/a  benchmarks.AperturePhotometry.time_small_data_multiple_small_apertures_elli_subpixel_05
   failed  215.41ms       n/a  benchmarks.AperturePhotometry.time_small_data_multiple_small_apertures_elli_subpixel_10

Benchmarks that have stayed the same:

  before     after    ratio
  98.30ms   68.19ms      0.69  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_center
 139.58ms  108.97ms      0.78  benchmarks.AperturePhotometry.time_big_data_error_multiple_big_apertures_circ_ann_exact
  98.94ms   67.96ms      0.69
[snip]
   1.55ms    1.52ms      0.98  benchmarks.AperturePhotometry.time_small_data_single_small_aperture_elli_subpixel_05
   1.72ms    1.58ms      0.92  benchmarks.AperturePhotometry.time_small_data_single_small_aperture_elli_subpixel_10

Benchmarks that have got worse:

  before     after    ratio
   2.30ms    3.71ms      1.61  benchmarks.AperturePhotometry.time_big_data_single_big_aperture_elli_center
   2.31ms    3.71ms      1.61  benchmarks.AperturePhotometry.time_big_data_single_big_aperture_elli_subpixel_01

I implemented this because I needed it for a specific set of benchmarks where I'm trying to compare the impact of different optimization strategies, but of course you can reject this or make suggestions for significant changes, as needed.

Commits date are not necessary monotonic along the commit log, especially if working with rebase workflow (instead of merge), this generate false positive or missed regression detection and unwanted peaks in the web UI.

This patch introduce a revision number as a drop-in replacement of commit date. This revision number is an incremental number representing the number of ancestors commits.

In the web UI the distance between commits is the number of commits between them. Commits can still be displayed by date using the date selector.

Closes #390

Record more benchmark samples, and compute, display, and use statistics based on them.

• Change default settings to record more (but shorter) samples. Changes meaning of goal_time Makes determination of goal_time more accurate; however, no big changes to methodology --- there's room to improve here.
• Do warmup before benchmark --- this seems to matter also on CPython (although possibly not due to CPython itself but some OS/CPU effects)
• Display spread of measurements (median +/- half of interquartile range)
• Estimate confidence interval and use that to decide what is not significant in asv compare.
• Optionally, save samples to the json files.
• Switch to gzipped files.

The statistical confidence estimates are a somewhat tricky point, because timing samples usually have strong autocorrelation (multimodality, stepwise changes in location, etc.), which makes simple stuff often misleading. There's some alleviation for this currently there in that it tries to regress the timing sample time series looking for steps, and adds those to the CI. Not rigorous, but probably better than nothing.

The problem is that there's low-frequency noise in the measurement, so measuring for a couple of second does not give a good idea of the full distribution.

todo:

• [x] more tests, e.g. printing and formatting is mostly untested
• [x] do we really need to gzip?
• [x] update documentation
• [x] tuning for pypy

When running on a repo with e.g. 10,000 commits, this allows one to see the general trends building up slowly rather than doing each commit sequentially.

Implement parameterization of benchmarks.

Plotting can deal with numerical and categorical parameters, user can select in the UI parameter combinations to show and what x-axis to use.

Addresses gh-36.

Sample output here: https://pv.github.io/scipy-bench/

Some sample code here: https://github.com/pv/scipy-bench/tree/param/benchmarks/

TODO:

• [x] ensure asv compare and other cmdline commands deal with parameterized results
• [x] produce text output (tables) showing results, either always or add a command line switch
• [x] for many benchmark parameters, the selection buttons become too small; switch to vertical view (currently used for commit selection) when many choices
• [x] check javascript performance for large numbers of commits
• [x] it probably loads the data set many times from the server, could be refactored a bit if so
• [x] more careful type check for params and param_names on python side, the javascript code does currently not tolerate garbage input here (or, the javascript code should make these checks)
• [x] better unit testing
• [x] also number and repeats may need to be parameter-dependent?
• [x] should parameterized benchmarks be considered as separate benchmarks, or as benchmarks with multiple return values as currently? Is the parameterization API elegant and suitable?
• [x] should the API be such that first implement benchmarks with multiple return values, and then build parameterized benchmarks on top of that?

TODO for later (separate PR):

• what would be good summary plot? now it just averages over parameters (should these switch to geom means?)

When running asv continuous to compare commits A and B, all the benchmarks from A run followed by all the benchmarks from B, e.g. "A.foo, A.bar, A.baz, B.foo, B.bar, B.baz". Would it be feasible to instead run "A.foo, B.foo, A.bar, B.bar, A.baz, B.baz"?

(In fact because each benchmark is run multiple times, Ideally I would like the staggering to be even finer-grained.)

The thought here is that background processes make noise auto-correlated, so the running comparisons back-to-back may give more informative ratios. (Based on amateur speculation)

I am trying to use asv to visualize the performance difference between two commits. I ran the benchmarks fine (it is these benchmarks, except I changed the repo url to my local repo).

But when I asv publish and asv preview, none of the graphs load. In the console, I have a lot of errors like

:8080/graphs/summary/cse.TimeCSE.time_cse.json?_=1501786104493 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/dsolve.TimeDsolve01.time_dsolve.json?_=1501786104494 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/integrate.TimeIntegration01.time_doit.json?_=1501786104495 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/lambdify.TimeLambdifyCreation.time_lambdify_create.json?_=1501786104497 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_functions.json?_=1501786104499 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/integrate.TimeIntegration01.time_doit_meijerg.json?_=1501786104496 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/lambdify.TimeLambdifyEvaluate.time_lambdify_evaluate.json?_=1501786104498 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_symbols.json?_=1501786104500 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/large_exprs.TimeLargeExpressionOperations.time_cse.json?_=1501786104503 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/large_exprs.TimeLargeExpressionOperations.time_count_ops.json?_=1501786104502 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/large_exprs.TimeLargeExpressionOperations.peakmem_subs.json?_=1501786104501 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/large_exprs.TimeLargeExpressionOperations.time_free_symbols.json?_=1501786104504 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_functions.json?_=1501786104505 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_symbols.json?_=1501786104506 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/logic.LogicSuite.time_dpll.json?_=1501786104509 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/logic.LogicSuite.time_dpll2.json?_=1501786104510 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/large_exprs.TimeLargeExpressionOperations.time_manual_jacobian_wrt_functions.json?_=1501786104507 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/large_exprs.TimeLargeExpressionOperations.time_subs.json?_=1501786104508 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/logic.LogicSuite.time_load_file.json?_=1501786104511 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/solve.TimeMatrixArithmetic.time_dense_multiply.json?_=1501786104515 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/solve.TimeMatrixOperations.time_det.json?_=1501786104516 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/physics.mechanics.kane.KanesMethodMassSpringDamper.time_kanesmethod_mass_spring_damper.json?_=1501786104512 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/solve.TimeMatrixOperations.time_det_berkowitz.json?_=1501786104518 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/solve.TimeMatrixArithmetic.time_dense_add.json?_=1501786104514 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/physics.mechanics.lagrange.LagrangesMethodMassSpringDamper.time_lagrangesmethod_mass_spring_damper.json?_=1501786104513 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/solve.TimeMatrixOperations.time_det_bareiss.json?_=1501786104517 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/solve.TimeMatrixOperations.time_rank.json?_=1501786104519 Failed to load resource: the server responded with a status of 404 (File not found)
:8080/graphs/summary/solve.TimeMatrixOperations.time_rref.json?_=1501786104520 Failed to load resource: the server responded with a status of 404 (File not found)

I've tried clearing the html directory and republishing and it doesn't help.

This is with asv 0.2. I also tried using the git version, but that failed:

$PYTHONPATH=~/Documents/asv python -m asv publish
Traceback (most recent call last):
  File "/Users/aaronmeurer/anaconda3/lib/python3.5/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Users/aaronmeurer/anaconda3/lib/python3.5/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/Users/aaronmeurer/anaconda3/lib/python3.5/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/Users/aaronmeurer/Documents/asv/asv/__init__.py", line 17, in <module>
    from ._version import __version__, __githash__, __release__
ImportError: No module named 'asv._version'

Add a second summary page, showing a list of benchmarks (instead of a grid).

The list shows benchmarks only for one choice of environment parameters (machine, cpu, packages, ...) at once.

Also show how the benchmark performance has evolved "recently", some ideas borrowed from http://speed.pypy.org/changes/

Some overlap with the Regressions display, with the exception that regressions shows regressions across all branches, and does not show improved results.

Demo: https://pv.github.io/numpy-bench/#summarylist https://pv.github.io/scipy-bench/#summarylist

• [x] what information should it actually show?
• [x] what does "recently" mean? 1 month, 10 revisions, ...?
• [x] what to do with benchmarks with no results?
• [x] the param selector should deal sensibly if there are no results for some param combination
• [x] tests, etc. polish
• [x] a selector for the time/revision range that you want to consider (e.g. show the worst regression ever vs. the last one vs. benchmark-specific configuration in the config file)
• [x] maybe add expandable view to show all regressions?
• [x] the popup graphs

pandas got a grant to work in our benchmarks, which use asv, and could be improved in different ways. We'd benefit from some work on asv itself, and we've got resources to help improve it.

Seems like asv is abandoned. I contacted @pv and @mdboom to discuss how we can move forward with asv, and coordinate with the resources we've got, but no answer after a while. If we don't hear back soon from them, and we don't find a better option, we're planning to fork the project. If the maintainers of the original asv have the time to come back to it after we fork, we'll coordinate and see if we can merge our changes back to this repo, or what's best. But with the current situation, seems like the only option, other than leave asv abandoned, use the current version forever, or move to a different project.

Opening this issue for visibility, so other users, developers and stakeholders of asv are aware, and can provide their feedback.

Currently, running asv publish when results are present (say through a HASHFILE) based on tag commit hashes, the tags cannot be found and so the graphs are essentially empty:

asv publish
[ 11.11%] · Loading machine info
[ 22.22%] · Getting params, commits, tags and branches
[ 33.33%] · Loading results.
[ 33.33%] ·· Couldn't find 4f0a3eb8 in branches (main)
[ 33.33%] ·· Couldn't find ef0ec786 in branches (main).
[ 33.33%] ·· Couldn't find 7ce41185 in branches (main).
[ 33.33%] ·· Couldn't find f6dddcb2 in branches (main)
[ 33.33%] ·· Couldn't find 08772f91 in branches (main).
[ 33.33%] ·· Couldn't find 21cacafb in branches (main)
[ 33.33%] ·· Couldn't find 5726e6ce in branches (main).
[ 33.33%] ·· Couldn't find 8cec8201 in branches (main)
[ 33.33%] ·· Couldn't find f8021557 in branches (main).
[ 33.33%] ·· Couldn't find 4adc87df in branches (main)
[ 33.33%] ·· Couldn't find 6377d884 in branches (main).
[ 33.33%] ·· Couldn't find e47cbb69 in branches (main).
[ 33.33%] ·· Couldn't find 54c52f13 in branches (main)
[ 33.33%] ·· Couldn't find 7d4349e3 in branches (main).
[ 33.33%] ·· Couldn't find 1f82da74 in branches (main)
[ 33.33%] ·· Couldn't find 5c598ed6 in branches (main).
[ 33.33%] ·· Couldn't find de82cd94 in branches (main)
[ 33.33%] ·· Couldn't find 754e59d5 in branches (main)
[ 44.44%] · Detecting steps.
[ 55.56%] · Generating graphs
[ 66.67%] · Generating output for SummaryGrid
[ 77.78%] · Generating output for SummaryList
[ 88.89%] · Generating output for Regressions.
[100.00%] · Writing index

We are trying to use asv to monitor a few different parameters in our benchmarks. We would like it to just "run" a few times.

can we return a tuple in track_ methods? How would we specify the units?

In https://github.com/pandas-dev/pandas/actions/runs/3515019336/jobs/5892108533 I got

[ 33.27%] ··· plotting.FramePlotting.time_frame_plot                  1/9 failed
[ 33.27%] ··· ========= =========
                 kind            
              --------- ---------
                 line    472±0ms 
                 bar     503±0ms 
                 area    580±0ms 
                 barh    501±0ms 
                 hist    508±0ms 
                 kde     947±0ms 
                 pie     478±0ms 
               scatter   383±0ms 
                hexbin    failed 
              ========= =========

with no further info

I had to clone asv, insert a breakpoint, and do

print(result.stderr)

to figure out what had caused the failure

It'd be useful if this could be shown in the report too

ASV tracks performance regression and improvements per machine and python versions. However, tracking performance on multiple configurations/optimizations options for python builds can only be done as separate machines configurations. Though, comparing machines against each other is only visible per benchmark view.

This PR will provide a comparison list for machines against each other. It can, also, exports the comparison list to markdown format (example).