Mercurial > repos > public > sbplib_julia

comparison benchmark/benchmark_utils.jl @ 1320:6ae2ec4cef5a tooling/benchmarks

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Rename functions for comparing benchmarks. Refactor their code. Allow comparing the current working directory
author Jonatan Werpers <jonatan@werpers.com>
date Fri, 28 Apr 2023 08:53:39 +0200
parents cb4d57ce2ed1
children 42738616422e
comparison
equal deleted inserted replaced
1319:cb4d57ce2ed1 1320:6ae2ec4cef5a
8 const sbplib_root = splitpath(pathof(Sbplib))[1:end-2] |> joinpath 8 const sbplib_root = splitpath(pathof(Sbplib))[1:end-2] |> joinpath
9 const results_dir = mkpath(joinpath(sbplib_root, "benchmark/results")) 9 const results_dir = mkpath(joinpath(sbplib_root, "benchmark/results"))
10 const template_path = joinpath(sbplib_root, "benchmark/result.tmpl") 10 const template_path = joinpath(sbplib_root, "benchmark/result.tmpl")
11 11
12 """ 12 """
13 main(args...; kwargs...) 13 mainmain(;rev=nothing, target=nothing, baseline=nothing , kwargs...)
14 14
15 Calls `run_benchmark(args...; kwargs...)` and writes the results as an HTML file in `benchmark/results`. 15 Calls `run_benchmark(args...; kwargs...)` and writes the results as an HTML
16 See [`run_benchmark`](@ref) for possible arguments. 16 file in `benchmark/results`.
17 """ 17
18 function main(args...; kwargs...) 18 * If `rev` is set, the benchmarks are run for the given mercurial revision.
19 r = run_benchmark(args...; kwargs...) 19 * If only `baseline` is set, the current working directory is compared with
20 the revision given in `baseline`.
21 * If both `target` and `baseline` is set those revision are compared.
22
23 For control over what happens to the benchmark result datastructure see the
24 different methods of [`run_benchmark`](@ref)
25 """
26 function main(;rev=nothing, target=nothing, baseline=nothing , kwargs...)
27 if !isnothing(rev)
28 r = run_benchmark(rev; kwargs...)
29 elseif !isnothing(baseline)
30 if isnothing(target)
31 r = compare_benchmarks(baseline; kwargs...)
32 else
33 r = compare_benchmarks(target, baseline; kwargs...)
34 end
35 else
36 # Neither rev, or baseline were set => Run on current working directory.
37 r = run_benchmark(;kwargs...)
38 end
39
20 file_path = write_result_html(r) 40 file_path = write_result_html(r)
21 open_in_default_browser(file_path) 41 open_in_default_browser(file_path)
22 end 42 end
23 43
24 44
25 """ 45 """
26 run_benchmark() 46 run_benchmark()
27 47
28 Runs the benchmark suite for the current working directory and returns a `PkgBenchmark.BenchmarkResult` 48 Run the benchmark suite for the current working directory and return a
49 `PkgBenchmark.BenchmarkResult`
29 """ 50 """
30 function run_benchmark(;kwargs...) 51 function run_benchmark(;kwargs...)
31 r = PkgBenchmark.benchmarkpkg(Sbplib; kwargs...) 52 r = PkgBenchmark.benchmarkpkg(Sbplib; kwargs...)
32 53
33 rev = hg_rev() # Should be changed to hg_id() when the html can handle it. 54 rev = hg_rev() # Should be changed to hg_id() when the html can handle it.
44 65
45 Returns a `PkgBenchmark.BenchmarkResult` 66 Returns a `PkgBenchmark.BenchmarkResult`
46 """ 67 """
47 function run_benchmark(rev; kwargs...) 68 function run_benchmark(rev; kwargs...)
48 return hg_at_revision(rev) do 69 return hg_at_revision(rev) do
49 run_benchmark(;kwargs...) 70 run_benchmark(; kwargs...)
50 end 71 end
51 end 72 end
52 73
53 """ 74 """
54 run_benchmark(target, baseline, f=minimum; judgekwargs=Dict()) 75 compare_benchmarks(target, baseline, f=minimum; judgekwargs=Dict())
55 76
56 Runs the benchmark at revisions `target` and `baseline` and compares them 77 Runs the benchmark at revisions `target` and `baseline` and compares them
57 using `PkgBenchmark.judge`. `f` is the function used to compare. `judgekwargs` 78 using `PkgBenchmark.judge`. `f` is the function used to compare. `judgekwargs`
58 are keyword arguments passed to `judge`. 79 are keyword arguments passed to `judge`.
59 80
60 `target` and `baseline` can be any identifier compatible with `hg update`. 81 `target` and `baseline` can be any identifier compatible with `hg update`.
61 82
62 Returns a `PkgBenchmark.BenchmarkJudgement` 83 Returns a `PkgBenchmark.BenchmarkJudgement`
63 """ 84 """
64 function run_benchmark(target, baseline, f=minimum; judgekwargs=Dict(), kwargs...) 85 function compare_benchmarks(target, baseline, f=minimum; judgekwargs=Dict(), kwargs...)
65 rev_before = hg_rev() 86 t = run_benchmark(target; kwargs...)
66 hg_update(target) 87 b = run_benchmark(baseline; kwargs...)
88
89 return PkgBenchmark.judge(t,b,f; judgekwargs...)
90 end
91
92 """
93 compare_benchmarks(baseline, ...)
94
95 Compare the results at the current working directory with the revision
96 specified in `baseline`.
97
98 Accepts the same arguments as the two revision version.
99 """
100 function compare_benchmark(baseline, f=minimum; judgekwargs=Dict(), kwargs...)
67 t = run_benchmark(;kwargs...) 101 t = run_benchmark(;kwargs...)
68 hg_update(baseline) 102 b = run_benchmark(baseline; kwargs...)
69 b = run_benchmark(;kwargs...)
70 hg_update(rev_before)
71 103
72 return PkgBenchmark.judge(t,b,f; judgekwargs...) 104 return PkgBenchmark.judge(t,b,f; judgekwargs...)
73 end 105 end
74 106
75 107