Tutorial
In this tutorial we illustrate the main uses of SolverBenchmark
.
First, let's create fake data. It is imperative that the data for each solver be stored in DataFrame
s, and the collection of different solver must be stored in a dictionary of Symbol
to DataFrame
.
In our examples we'll use the following data.
using DataFrames, Printf, Random
Random.seed!(0)
n = 10
names = [:alpha, :beta, :gamma]
stats = Dict(name => DataFrame(:id => 1:n,
:name => [@sprintf("prob%03d", i) for i = 1:n],
:status => map(x -> x < 0.75 ? :first_order : :failure, rand(n)),
:f => randn(n),
:t => 1e-3 .+ rand(n) * 1000,
:iter => rand(10:10:100, n),
:irrelevant => randn(n)) for name in names)
Dict{Symbol,DataFrames.DataFrame} with 3 entries:
:alpha => 10×7 DataFrame. Omitted printing of 1 columns…
:beta => 10×7 DataFrame. Omitted printing of 1 columns…
:gamma => 10×7 DataFrame. Omitted printing of 1 columns…
The data consists of a (fake) run of three solvers alpha
, beta
and gamma
. Each solver has a column id
, which is necessary for joining the solvers (names can be repeated), and columns name
, status
, f
, t
and iter
corresponding to problem results. There is also a column irrelevant
with extra information that will not be used to produce our benchmarks.
Here are the statistics of solver alpha
:
stats[:alpha]
id | name | status | f | t | iter | irrelevant | |
---|---|---|---|---|---|---|---|
Int64 | String | Symbol | Float64 | Float64 | Int64 | Float64 | |
1 | 1 | prob001 | failure | -0.688907 | 62.3686 | 70 | 0.891315 |
2 | 2 | prob002 | failure | -0.762804 | 353.13 | 10 | -2.29449 |
3 | 3 | prob003 | first_order | 0.397482 | 767.603 | 10 | -1.17303 |
4 | 4 | prob004 | first_order | 0.81163 | 43.142 | 80 | -0.879915 |
5 | 5 | prob005 | first_order | -0.346355 | 267.986 | 30 | -0.494043 |
6 | 6 | prob006 | first_order | -0.187573 | 66.8474 | 80 | 0.511959 |
7 | 7 | prob007 | first_order | -1.60726 | 156.638 | 60 | -0.758701 |
8 | 8 | prob008 | first_order | -2.48079 | 605.298 | 40 | -0.76796 |
9 | 9 | prob009 | first_order | 2.27623 | 135.746 | 40 | -0.0303032 |
10 | 10 | prob010 | failure | 0.219693 | 838.119 | 50 | -1.31924 |
Tables
The first thing we may want to do is produce a table for each solver. Notice that the solver result is already a DataFrame, so there are a few options available in other packages, as well as simply printing the DataFrame. Our concern here is two-fold: producing publication-ready LaTeX tables, and web-ready markdown tables.
The simplest use is pretty_stats(io, dataframe)
. By default, io
is stdout
:
using SolverBenchmark
pretty_stats(stats[:alpha])
┌────────┬─────────┬─────────────┬───────────┬───────────┬────────┬────────────┐
│ id │ name │ status │ f │ t │ iter │ irrelevant │
├────────┼─────────┼─────────────┼───────────┼───────────┼────────┼────────────┤
│ 1 │ prob001 │ failure │ -6.89e-01 │ 6.24e+01 │ 70 │ 8.91e-01 │
│ 2 │ prob002 │ failure │ -7.63e-01 │ 3.53e+02 │ 10 │ -2.29e+00 │
│ 3 │ prob003 │ first_order │ 3.97e-01 │ 7.68e+02 │ 10 │ -1.17e+00 │
│ 4 │ prob004 │ first_order │ 8.12e-01 │ 4.31e+01 │ 80 │ -8.80e-01 │
│ 5 │ prob005 │ first_order │ -3.46e-01 │ 2.68e+02 │ 30 │ -4.94e-01 │
│ 6 │ prob006 │ first_order │ -1.88e-01 │ 6.68e+01 │ 80 │ 5.12e-01 │
│ 7 │ prob007 │ first_order │ -1.61e+00 │ 1.57e+02 │ 60 │ -7.59e-01 │
│ 8 │ prob008 │ first_order │ -2.48e+00 │ 6.05e+02 │ 40 │ -7.68e-01 │
│ 9 │ prob009 │ first_order │ 2.28e+00 │ 1.36e+02 │ 40 │ -3.03e-02 │
│ 10 │ prob010 │ failure │ 2.20e-01 │ 8.38e+02 │ 50 │ -1.32e+00 │
└────────┴─────────┴─────────────┴───────────┴───────────┴────────┴────────────┘
Printing is LaTeX format is achieved with pretty_latex_stats
:
pretty_latex_stats(stats[:alpha])
\begin{longtable}{rrrrrrr}
\hline
id & name & status & f & t & iter & irrelevant \\\hline
\endhead
\hline
\multicolumn{7}{r}{{\bfseries Continued on next page}}\\
\hline
\endfoot
\endlastfoot
\( 1\) & prob001 & failure & \(-6.89\)e\(-01\) & \( 6.24\)e\(+01\) & \( 70\) & \( 8.91\)e\(-01\) \\
\( 2\) & prob002 & failure & \(-7.63\)e\(-01\) & \( 3.53\)e\(+02\) & \( 10\) & \(-2.29\)e\(+00\) \\
\( 3\) & prob003 & first\_order & \( 3.97\)e\(-01\) & \( 7.68\)e\(+02\) & \( 10\) & \(-1.17\)e\(+00\) \\
\( 4\) & prob004 & first\_order & \( 8.12\)e\(-01\) & \( 4.31\)e\(+01\) & \( 80\) & \(-8.80\)e\(-01\) \\
\( 5\) & prob005 & first\_order & \(-3.46\)e\(-01\) & \( 2.68\)e\(+02\) & \( 30\) & \(-4.94\)e\(-01\) \\
\( 6\) & prob006 & first\_order & \(-1.88\)e\(-01\) & \( 6.68\)e\(+01\) & \( 80\) & \( 5.12\)e\(-01\) \\
\( 7\) & prob007 & first\_order & \(-1.61\)e\(+00\) & \( 1.57\)e\(+02\) & \( 60\) & \(-7.59\)e\(-01\) \\
\( 8\) & prob008 & first\_order & \(-2.48\)e\(+00\) & \( 6.05\)e\(+02\) & \( 40\) & \(-7.68\)e\(-01\) \\
\( 9\) & prob009 & first\_order & \( 2.28\)e\(+00\) & \( 1.36\)e\(+02\) & \( 40\) & \(-3.03\)e\(-02\) \\
\( 10\) & prob010 & failure & \( 2.20\)e\(-01\) & \( 8.38\)e\(+02\) & \( 50\) & \(-1.32\)e\(+00\) \\\hline
\end{longtable}
Alternatively, you can print to a file.
open("alpha.tex", "w") do io
println(io, "\\documentclass[varwidth=20cm,crop=true]{standalone}")
println(io, "\\usepackage{longtable}")
println(io, "\\begin{document}")
pretty_latex_stats(io, stats[:alpha])
println(io, "\\end{document}")
end
run(`latexmk -quiet -pdf alpha.tex`)
run(`pdf2svg alpha.pdf alpha.svg`)
Process(`pdf2svg alpha.pdf alpha.svg`, ProcessExited(0))
If only a subset of columns should be printed, the DataFrame should be indexed accordingly:
df = stats[:alpha]
pretty_stats(df[!, [:name, :f, :t]])
┌─────────┬───────────┬───────────┐
│ name │ f │ t │
├─────────┼───────────┼───────────┤
│ prob001 │ -6.89e-01 │ 6.24e+01 │
│ prob002 │ -7.63e-01 │ 3.53e+02 │
│ prob003 │ 3.97e-01 │ 7.68e+02 │
│ prob004 │ 8.12e-01 │ 4.31e+01 │
│ prob005 │ -3.46e-01 │ 2.68e+02 │
│ prob006 │ -1.88e-01 │ 6.68e+01 │
│ prob007 │ -1.61e+00 │ 1.57e+02 │
│ prob008 │ -2.48e+00 │ 6.05e+02 │
│ prob009 │ 2.28e+00 │ 1.36e+02 │
│ prob010 │ 2.20e-01 │ 8.38e+02 │
└─────────┴───────────┴───────────┘
Markdown tables may be generated by supplying the PrettyTables tf
keyword argument to specify the table format:
pretty_stats(df[!, [:name, :f, :t]], tf=markdown)
| name | f | t |
|---------|-----------|-----------|
| prob001 | -6.89e-01 | 6.24e+01 |
| prob002 | -7.63e-01 | 3.53e+02 |
| prob003 | 3.97e-01 | 7.68e+02 |
| prob004 | 8.12e-01 | 4.31e+01 |
| prob005 | -3.46e-01 | 2.68e+02 |
| prob006 | -1.88e-01 | 6.68e+01 |
| prob007 | -1.61e+00 | 1.57e+02 |
| prob008 | -2.48e+00 | 6.05e+02 |
| prob009 | 2.28e+00 | 1.36e+02 |
| prob010 | 2.20e-01 | 8.38e+02 |
All values of tf
accepted by PrettyTables may be used in SolverBenchmark.
The fmt_override
option overrides the formatting of a specific column. The argument should be a dictionary of Symbol
to format strings, where the format string will be applied to each element of the column.
The hdr_override
changes the column headers.
fmt_override = Dict(:f => "%+10.3e",
:t => "%08.2f")
hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
pretty_stats(stdout,
df[!, [:name, :f, :t]],
col_formatters = fmt_override,
hdr_override = hdr_override)
┌─────────┬────────────┬──────────┐
│ Name │ f(x) │ Time │
├─────────┼────────────┼──────────┤
│ prob001 │ -6.889e-01 │ 00062.37 │
│ prob002 │ -7.628e-01 │ 00353.13 │
│ prob003 │ +3.975e-01 │ 00767.60 │
│ prob004 │ +8.116e-01 │ 00043.14 │
│ prob005 │ -3.464e-01 │ 00267.99 │
│ prob006 │ -1.876e-01 │ 00066.85 │
│ prob007 │ -1.607e+00 │ 00156.64 │
│ prob008 │ -2.481e+00 │ 00605.30 │
│ prob009 │ +2.276e+00 │ 00135.75 │
│ prob010 │ +2.197e-01 │ 00838.12 │
└─────────┴────────────┴──────────┘
While col_formatters
is for simple format strings, the PrettyTables API lets us define more elaborate formatters in the form of functions:
fmt_override = Dict(:f => "%+10.3e",
:t => "%08.2f")
hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
pretty_stats(df[!, [:name, :f, :t]],
col_formatters = fmt_override,
hdr_override = hdr_override,
formatters = (v, i, j) -> begin
if j == 3 # t is the 3rd column
vi = floor(Int, v)
minutes = div(vi, 60)
seconds = vi % 60
micros = round(Int, 1e6 * (v - vi))
@sprintf("%2dm %02ds %06dμs", minutes, seconds, micros)
else
v
end
end)
┌─────────┬────────────┬──────────────────┐
│ Name │ f(x) │ Time │
├─────────┼────────────┼──────────────────┤
│ prob001 │ -6.889e-01 │ 1m 02s 368558μs │
│ prob002 │ -7.628e-01 │ 5m 53s 129773μs │
│ prob003 │ +3.975e-01 │ 12m 47s 602896μs │
│ prob004 │ +8.116e-01 │ 0m 43s 142023μs │
│ prob005 │ -3.464e-01 │ 4m 27s 985634μs │
│ prob006 │ -1.876e-01 │ 1m 06s 847444μs │
│ prob007 │ -1.607e+00 │ 2m 36s 637637μs │
│ prob008 │ -2.481e+00 │ 10m 05s 297740μs │
│ prob009 │ +2.276e+00 │ 2m 15s 745559μs │
│ prob010 │ +2.197e-01 │ 13m 58s 118754μs │
└─────────┴────────────┴──────────────────┘
See the PrettyTables.jl documentation for more information.
When using LaTeX format, the output must be understood by LaTeX. By default, numerical data in the table is wrapped in inline math environments. But those math environments would interfere with our formatting of the time. Thus we must first disable them for the time
column using col_formatters
, and then apply the PrettyTables formatter as above:
fmt_override = Dict(:f => "%+10.3e",
:t => "%08.2f")
hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
open("alpha2.tex", "w") do io
println(io, "\\documentclass[varwidth=20cm,crop=true]{standalone}")
println(io, "\\usepackage{longtable}")
println(io, "\\begin{document}")
pretty_latex_stats(io,
df[!, [:name, :status, :f, :t, :iter]],
col_formatters = Dict(:t => "%f"), # disable default formatting of t
formatters = (v,i,j) -> begin
if j == 4
xi = floor(Int, v)
minutes = div(xi, 60)
seconds = xi % 60
micros = round(Int, 1e6 * (v - xi))
@sprintf("\\(%2d\\)m \\(%02d\\)s \\(%06d \\mu\\)s", minutes, seconds, micros)
else
v
end
end)
println(io, "\\end{document}")
end
Joining tables
In some occasions, instead of/in addition to showing individual results, we show a table with the result of multiple solvers.
df = join(stats, [:f, :t])
pretty_stats(stdout, df)
┌────────┬───────────┬───────────┬───────────┬───────────┬───────────┬───────────┐
│ id │ f_alpha │ t_alpha │ f_beta │ t_beta │ f_gamma │ t_gamma │
├────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┤
│ 1 │ -6.89e-01 │ 6.24e+01 │ -1.05e+00 │ 1.77e+02 │ 6.34e-02 │ 3.26e+01 │
│ 2 │ -7.63e-01 │ 3.53e+02 │ 8.16e-01 │ 8.00e+01 │ 1.19e-01 │ 6.86e+02 │
│ 3 │ 3.97e-01 │ 7.68e+02 │ 1.53e-01 │ 6.82e+02 │ 2.71e+00 │ 8.41e+02 │
│ 4 │ 8.12e-01 │ 4.31e+01 │ -3.30e-01 │ 9.29e+02 │ -6.91e-01 │ 1.90e+02 │
│ 5 │ -3.46e-01 │ 2.68e+02 │ 1.44e+00 │ 9.73e+02 │ -5.51e-02 │ 1.62e+02 │
│ 6 │ -1.88e-01 │ 6.68e+01 │ -4.43e-01 │ 6.52e+02 │ 4.23e-01 │ 8.97e+02 │
│ 7 │ -1.61e+00 │ 1.57e+02 │ 1.10e+00 │ 5.97e+02 │ -1.43e+00 │ 9.54e+01 │
│ 8 │ -2.48e+00 │ 6.05e+02 │ -2.51e-01 │ 4.79e+02 │ -4.50e-01 │ 7.77e+02 │
│ 9 │ 2.28e+00 │ 1.36e+02 │ 2.92e-01 │ 6.32e+01 │ -8.81e-01 │ 8.68e+02 │
│ 10 │ 2.20e-01 │ 8.38e+02 │ -3.47e+00 │ 4.71e+02 │ 1.08e+00 │ 8.38e+02 │
└────────┴───────────┴───────────┴───────────┴───────────┴───────────┴───────────┘
The column :id
is used as guide on where to join. In addition, we may have repeated columns between the solvers. We convery that information with argument invariant_cols
.
df = join(stats, [:f, :t], invariant_cols=[:name])
pretty_stats(stdout, df)
┌────────┬─────────┬───────────┬───────────┬───────────┬───────────┬───────────┬───────────┐
│ id │ name │ f_alpha │ t_alpha │ f_beta │ t_beta │ f_gamma │ t_gamma │
├────────┼─────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┤
│ 1 │ prob001 │ -6.89e-01 │ 6.24e+01 │ -1.05e+00 │ 1.77e+02 │ 6.34e-02 │ 3.26e+01 │
│ 2 │ prob002 │ -7.63e-01 │ 3.53e+02 │ 8.16e-01 │ 8.00e+01 │ 1.19e-01 │ 6.86e+02 │
│ 3 │ prob003 │ 3.97e-01 │ 7.68e+02 │ 1.53e-01 │ 6.82e+02 │ 2.71e+00 │ 8.41e+02 │
│ 4 │ prob004 │ 8.12e-01 │ 4.31e+01 │ -3.30e-01 │ 9.29e+02 │ -6.91e-01 │ 1.90e+02 │
│ 5 │ prob005 │ -3.46e-01 │ 2.68e+02 │ 1.44e+00 │ 9.73e+02 │ -5.51e-02 │ 1.62e+02 │
│ 6 │ prob006 │ -1.88e-01 │ 6.68e+01 │ -4.43e-01 │ 6.52e+02 │ 4.23e-01 │ 8.97e+02 │
│ 7 │ prob007 │ -1.61e+00 │ 1.57e+02 │ 1.10e+00 │ 5.97e+02 │ -1.43e+00 │ 9.54e+01 │
│ 8 │ prob008 │ -2.48e+00 │ 6.05e+02 │ -2.51e-01 │ 4.79e+02 │ -4.50e-01 │ 7.77e+02 │
│ 9 │ prob009 │ 2.28e+00 │ 1.36e+02 │ 2.92e-01 │ 6.32e+01 │ -8.81e-01 │ 8.68e+02 │
│ 10 │ prob010 │ 2.20e-01 │ 8.38e+02 │ -3.47e+00 │ 4.71e+02 │ 1.08e+00 │ 8.38e+02 │
└────────┴─────────┴───────────┴───────────┴───────────┴───────────┴───────────┴───────────┘
join
also accepts hdr_override
for changing the column name before appending _solver
.
hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
df = join(stats, [:f, :t], invariant_cols=[:name], hdr_override=hdr_override)
pretty_stats(stdout, df)
┌────────┬─────────┬────────────┬────────────┬───────────┬───────────┬────────────┬────────────┐
│ id │ name │ f(x)_alpha │ Time_alpha │ f(x)_beta │ Time_beta │ f(x)_gamma │ Time_gamma │
├────────┼─────────┼────────────┼────────────┼───────────┼───────────┼────────────┼────────────┤
│ 1 │ prob001 │ -6.89e-01 │ 6.24e+01 │ -1.05e+00 │ 1.77e+02 │ 6.34e-02 │ 3.26e+01 │
│ 2 │ prob002 │ -7.63e-01 │ 3.53e+02 │ 8.16e-01 │ 8.00e+01 │ 1.19e-01 │ 6.86e+02 │
│ 3 │ prob003 │ 3.97e-01 │ 7.68e+02 │ 1.53e-01 │ 6.82e+02 │ 2.71e+00 │ 8.41e+02 │
│ 4 │ prob004 │ 8.12e-01 │ 4.31e+01 │ -3.30e-01 │ 9.29e+02 │ -6.91e-01 │ 1.90e+02 │
│ 5 │ prob005 │ -3.46e-01 │ 2.68e+02 │ 1.44e+00 │ 9.73e+02 │ -5.51e-02 │ 1.62e+02 │
│ 6 │ prob006 │ -1.88e-01 │ 6.68e+01 │ -4.43e-01 │ 6.52e+02 │ 4.23e-01 │ 8.97e+02 │
│ 7 │ prob007 │ -1.61e+00 │ 1.57e+02 │ 1.10e+00 │ 5.97e+02 │ -1.43e+00 │ 9.54e+01 │
│ 8 │ prob008 │ -2.48e+00 │ 6.05e+02 │ -2.51e-01 │ 4.79e+02 │ -4.50e-01 │ 7.77e+02 │
│ 9 │ prob009 │ 2.28e+00 │ 1.36e+02 │ 2.92e-01 │ 6.32e+01 │ -8.81e-01 │ 8.68e+02 │
│ 10 │ prob010 │ 2.20e-01 │ 8.38e+02 │ -3.47e+00 │ 4.71e+02 │ 1.08e+00 │ 8.38e+02 │
└────────┴─────────┴────────────┴────────────┴───────────┴───────────┴────────────┴────────────┘
hdr_override = Dict(:name => "Name", :f => "\\(f(x)\\)", :t => "Time")
df = join(stats, [:f, :t], invariant_cols=[:name], hdr_override=hdr_override)
open("alpha3.tex", "w") do io
println(io, "\\documentclass[varwidth=20cm,crop=true]{standalone}")
println(io, "\\usepackage{longtable}")
println(io, "\\begin{document}")
pretty_latex_stats(io, df)
println(io, "\\end{document}")
end
Profiles
Performance profiles are a comparison tool developed by Dolan and Moré, 2002 that takes into account the relative performance of a solver and whether it has achieved convergence for each problem. SolverBenchmark.jl
uses BenchmarkProfiles.jl for generating performance profiles from the dictionary of DataFrame
s.
The basic usage is performance_profile(stats, cost)
, where cost
is a function applied to a DataFrame
and returning a vector.
using Plots
pyplot()
p = performance_profile(stats, df -> df.t)
Plots.svg(p, "profile1")
Notice that we used df -> df.t
which corresponds to the column :t
of the DataFrame
s. This does not take into account that the solvers have failed for a few problems (according to column :status). The next profile takes that into account.
cost(df) = (df.status .!= :success) * Inf + df.t
p = performance_profile(stats, cost)
Plots.svg(p, "profile2")
Profile wall
Another profile function is profile_solvers
, which creates a wall of performance profiles, accepting multiple costs and doing 1 vs 1 comparisons in addition to the traditional performance profile.
solved(df) = (df.status .== :success)
costs = [df -> .!solved(df) * Inf + df.t, df -> .!solved(df) * Inf + df.iter]
costnames = ["Time", "Iterations"]
p = profile_solvers(stats, costs, costnames)
Plots.svg(p, "profile3")