Skip to the content.

acm_mirum

This is the tempo_eval report for the ‘acm_mirum’ corpus.

Reports for other corpora may be found here.

Table of Contents

References for ‘acm_mirum’

References

1.0

Attribute Value
Corpus acm_mirum
Version 1.0
Curator Geoffroy Peeters
Annotator, bibtex Peeters2012
Annotator, ref_url http://recherche.ircam.fr/anasyn/peeters/pub/2012_ACMMIRUM/

2.0

Attribute Value
Corpus acm_mirum
Version 2.0
Curator Graham Percival
Annotator, bibtex Percival2014
Annotator, ref_url http://www.marsyas.info/tempo/

Basic Statistics

Reference Size Min Max Avg Stdev Sweet Oct. Start Sweet Oct. Coverage
1.0 1410 36.00 257.00 102.54 32.73 69.00 0.73
2.0 1410 37.00 257.00 102.72 32.59 69.00 0.73

Table 1: Basic statistics.

CSV JSON LATEX PICKLE

Smoothed Tempo Distribution

Figure 1: Percentage of values in tempo interval.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimates for ‘acm_mirum’

Estimators

boeck2015/tempodetector2016_default

Attribute Value
Corpus acm_mirum
Version 0.17.dev0
Annotation Tools TempoDetector.2016, madmom, https://github.com/CPJKU/madmom
Annotator, bibtex Boeck2015

boeck2019/multi_task

Attribute Value
Corpus acm_mirum
Version 0.0.1
Annotation Tools model=multi_task, https://github.com/superbock/ISMIR2019
Annotator, bibtex Boeck2019

boeck2019/multi_task_hjdb

Attribute Value
Corpus acm_mirum
Version 0.0.1
Annotation Tools model=multi_task_hjdb, https://github.com/superbock/ISMIR2019
Annotator, bibtex Boeck2019

boeck2020/dar

Attribute Value
Corpus acm_mirum
Version 0.0.1
Annotation Tools https://github.com/superbock/ISMIR2020
Annotator, bibtex Boeck2020

davies2009/mirex_qm_tempotracker

Attribute Value  
Corpus acm_mirum  
Version 1.0  
Annotation Tools QM Tempotracker, Sonic Annotator plugin. https://code.soundsoftware.ac.uk/projects/mirex2013/repository/show/audio_tempo_estimation/qm-tempotracker Note that the current macOS build of ‘qm-vamp-plugins’ was used.  
Annotator, bibtex Davies2009 Davies2007

echonest/version_3_2_1

Attribute Value
Corpus acm_mirum
Version 3.2.1
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools Echo Nest track analyzer v3.2.1
Annotator, bibtex Percival2014

gkiokas2012/default

Attribute Value
Corpus acm_mirum
Version 1.0
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools Gkiokas2012
Annotator, bibtex Gkiokas2012

klapuri2006/percival2014

Attribute Value
Corpus acm_mirum
Version 1.0
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools Klapuri 2006
Annotator, bibtex Klapuri2006

oliveira2010/ibt

Attribute Value
Corpus acm_mirum
Version 1.0
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools Oliveira 2010
Annotator, bibtex Oliveira2010

percival2014/stem

Attribute Value
Corpus acm_mirum
Version 1.0
Annotation Tools percival 2014, ‘tempo’ implementation from Marsyas, http://marsyas.info, git checkout tempo-stem
Annotator, bibtex Percival2014

scheirer1998/percival2014

Attribute Value
Corpus acm_mirum
Version 1.0
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools Scheirer 1998
Annotator, bibtex Scheirer1998

schreiber2014/default

Attribute Value
Corpus acm_mirum
Version 0.0.1
Annotation Tools schreiber 2014, http://www.tagtraum.com/tempo_estimation.html
Annotator, bibtex Schreiber2014

schreiber2017/ismir2017

Attribute Value
Corpus acm_mirum
Version 0.0.4
Annotation Tools schreiber 2017, model=ismir2017, http://www.tagtraum.com/tempo_estimation.html
Annotator, bibtex Schreiber2017

schreiber2017/mirex2017

Attribute Value
Corpus acm_mirum
Version 0.0.4
Annotation Tools schreiber 2017, model=mirex2017, http://www.tagtraum.com/tempo_estimation.html
Annotator, bibtex Schreiber2017

schreiber2018/cnn

Attribute Value
Corpus acm_mirum
Version 0.0.2
Data Source Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018.
Annotation Tools schreiber tempo-cnn (model=cnn), https://github.com/hendriks73/tempo-cnn

schreiber2018/fcn

Attribute Value
Corpus acm_mirum
Version 0.0.2
Data Source Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018.
Annotation Tools schreiber tempo-cnn (model=fcn), https://github.com/hendriks73/tempo-cnn

schreiber2018/ismir2018

Attribute Value
Corpus acm_mirum
Version 0.0.2
Data Source Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018.
Annotation Tools schreiber tempo-cnn (model=ismir2018), https://github.com/hendriks73/tempo-cnn

sun2021/default

Attribute Value
Corpus acm_mirum
Version 0.0.2
Data Source Xiaoheng Sun, Qiqi He, Yongwei Gao, Wei Li. Musical Tempo Estimation Using a Multi-scale Network. in Proc. of the 22nd Int. Society for Music Information Retrieval Conf., Online, 2021
Annotation Tools https://github.com/Qqi-HE/TempoEstimation_MGANet
Annotator, bibtex Sun2021

zplane/auftakt_v3

Attribute Value
Corpus acm_mirum
Version 3.0
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools zplane aufTAKT version 3.0, http://licensing.zplane.de/technology#auftakt
Annotator, bibtex Percival2014

Basic Statistics

Estimator Size Min Max Avg Stdev Sweet Oct. Start Sweet Oct. Coverage
boeck2015/tempodetector2016_default 1410 41.96 240.00 118.09 34.59 84.00 0.72
boeck2019/multi_task 1410 39.89 206.08 107.56 28.79 72.00 0.82
boeck2019/multi_task_hjdb 1410 39.64 204.80 108.48 29.73 72.00 0.79
boeck2020/dar 1410 48.02 237.16 109.24 32.30 73.00 0.76
davies2009/mirex_qm_tempotracker 1410 60.80 191.41 120.01 27.50 84.00 0.87
echonest/version_3_2_1 1410 45.91 197.99 108.64 30.82 72.00 0.76
gkiokas2012/default 1410 44.00 218.00 107.42 30.72 73.00 0.77
klapuri2006/percival2014 1410 65.01 164.06 110.33 23.89 76.00 0.93
oliveira2010/ibt 1410 80.00 167.00 118.42 23.87 81.00 1.00
percival2014/stem 1410 50.67 156.60 102.13 22.97 72.00 0.92
scheirer1998/percival2014 1312 61.35 181.82 106.51 31.06 77.00 0.72
schreiber2014/default 1410 56.34 160.08 99.91 22.74 68.00 0.90
schreiber2017/ismir2017 1410 40.72 203.92 105.72 27.80 72.00 0.83
schreiber2017/mirex2017 1410 28.86 193.36 104.59 29.61 72.00 0.79
schreiber2018/cnn 1410 41.00 204.00 111.56 31.17 74.00 0.77
schreiber2018/fcn 1410 40.00 214.00 109.67 33.17 72.00 0.75
schreiber2018/ismir2018 1410 56.00 204.00 110.56 28.18 74.00 0.85
sun2021/default 1410 47.00 240.00 110.64 32.64 73.00 0.75
zplane/auftakt_v3 1410 65.00 171.00 108.30 25.58 79.00 0.85

Table 2: Basic statistics.

CSV JSON LATEX PICKLE

Smoothed Tempo Distribution

Figure 2: Percentage of values in tempo interval.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy

Accuracy1 is defined as the percentage of correct estimates, allowing a 4% tolerance for individual BPM values.

Accuracy2 additionally permits estimates to be wrong by a factor of 2, 3, 1/2 or 1/3 (so-called octave errors).

See [Gouyon2006].

Note: When comparing accuracy values for different algorithms, keep in mind that an algorithm may have been trained on the test set or that the test set may have even been created using one of the tested algorithms.

Accuracy Results for 1.0

Estimator Accuracy1 Accuracy2
schreiber2017/mirex2017 0.8348 0.9078
boeck2020/dar 0.7844 0.9085
schreiber2018/cnn 0.7645 0.9071
schreiber2018/fcn 0.7631 0.9007
sun2021/default 0.7582 0.8851
schreiber2017/ismir2017 0.7511 0.8936
schreiber2018/ismir2018 0.7163 0.8972
boeck2019/multi_task 0.6936 0.9000
schreiber2014/default 0.6922 0.8794
boeck2015/tempodetector2016_default 0.6858 0.9000
boeck2019/multi_task_hjdb 0.6851 0.8972
echonest/version_3_2_1 0.6766 0.8539
percival2014/stem 0.6702 0.9000
gkiokas2012/default 0.6610 0.8972
zplane/auftakt_v3 0.6461 0.8660
klapuri2006/percival2014 0.6270 0.8865
davies2009/mirex_qm_tempotracker 0.6085 0.8645
oliveira2010/ibt 0.5794 0.8603
scheirer1998/percival2014 0.4957 0.7149

Table 3: Mean accuracy of estimates compared to version 1.0 with 4% tolerance ordered by Accuracy1.

CSV JSON LATEX PICKLE

Raw data Accuracy1: CSV JSON LATEX PICKLE

Raw data Accuracy2: CSV JSON LATEX PICKLE

Accuracy1 for 1.0

Figure 3: Mean Accuracy1 for estimates compared to version 1.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for 1.0

Figure 4: Mean Accuracy2 for estimates compared to version 1.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy Results for 2.0

Estimator Accuracy1 Accuracy2
schreiber2017/mirex2017 0.9092 0.9872
boeck2020/dar 0.8475 0.9908
schreiber2018/cnn 0.8291 0.9844
schreiber2018/fcn 0.8277 0.9794
sun2021/default 0.8206 0.9638
schreiber2017/ismir2017 0.8184 0.9730
schreiber2018/ismir2018 0.7809 0.9766
schreiber2014/default 0.7617 0.9603
boeck2019/multi_task 0.7574 0.9773
boeck2019/multi_task_hjdb 0.7489 0.9738
boeck2015/tempodetector2016_default 0.7404 0.9780
echonest/version_3_2_1 0.7390 0.9291
percival2014/stem 0.7369 0.9794
gkiokas2012/default 0.7270 0.9801
zplane/auftakt_v3 0.7021 0.9390
klapuri2006/percival2014 0.6879 0.9688
davies2009/mirex_qm_tempotracker 0.6603 0.9348
oliveira2010/ibt 0.6305 0.9312
scheirer1998/percival2014 0.5355 0.7723

Table 4: Mean accuracy of estimates compared to version 2.0 with 4% tolerance ordered by Accuracy1.

CSV JSON LATEX PICKLE

Raw data Accuracy1: CSV JSON LATEX PICKLE

Raw data Accuracy2: CSV JSON LATEX PICKLE

Accuracy1 for 2.0

Figure 5: Mean Accuracy1 for estimates compared to version 2.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for 2.0

Figure 6: Mean Accuracy2 for estimates compared to version 2.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Differing Items

For which items did a given estimator not estimate a correct value with respect to a given ground truth? Are there items which are either very difficult, not suitable for the task, or incorrectly annotated and therefore never estimated correctly, regardless which estimator is used?

Differing Items Accuracy1

Items with different tempo annotations (Accuracy1, 4% tolerance) in different versions:

1.0 compared with boeck2015/tempodetector2016_default (443 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10503810.clip’ ‘105233.clip’ ‘10563001.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10766976.clip’ ‘10809231.clip’ … CSV

1.0 compared with boeck2019/multi_task (432 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10503810.clip’ ‘10563001.clip’ ‘10726039.clip’ ‘10809231.clip’ ‘10875997.clip’ ‘10893272.clip’ … CSV

1.0 compared with boeck2019/multi_task_hjdb (444 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10503810.clip’ ‘10563001.clip’ ‘10726039.clip’ ‘10809231.clip’ ‘10875997.clip’ ‘10893272.clip’ … CSV

1.0 compared with boeck2020/dar (304 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10726039.clip’ ‘10809231.clip’ ‘10875997.clip’ ‘10893272.clip’ … CSV

1.0 compared with davies2009/mirex_qm_tempotracker (552 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10503810.clip’ ‘10563001.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10726076.clip’ ‘1074945.clip’ ‘10809231.clip’ … CSV

1.0 compared with echonest/version_3_2_1 (456 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104260.clip’ ‘10503810.clip’ ‘10563001.clip’ ‘10563039.clip’ ‘1074945.clip’ ‘10766976.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘10894070.clip’ … CSV

1.0 compared with gkiokas2012/default (478 differences): ‘10118334.clip’ ‘10258351.clip’ ‘10270809.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10503810.clip’ ‘10563001.clip’ ‘10563039.clip’ ‘1074945.clip’ ‘10809231.clip’ … CSV

1.0 compared with klapuri2006/percival2014 (526 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘105009.clip’ ‘10503810.clip’ ‘105233.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10726076.clip’ … CSV

1.0 compared with oliveira2010/ibt (593 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10726058.clip’ ‘10726076.clip’ ‘1074945.clip’ … CSV

1.0 compared with percival2014/stem (465 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10726076.clip’ ‘10809231.clip’ ‘10893272.clip’ … CSV

1.0 compared with scheirer1998/percival2014 (711 differences): ‘10118334.clip’ ‘10258351.clip’ ‘10270809.clip’ ‘10332517.clip’ ‘104186.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘105009.clip’ ‘10503810.clip’ ‘105233.clip’ … CSV

1.0 compared with schreiber2014/default (434 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘105009.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10726058.clip’ ‘10809231.clip’ … CSV

1.0 compared with schreiber2017/ismir2017 (351 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘10894286.clip’ ‘11173645.clip’ ‘11430821.clip’ … CSV

1.0 compared with schreiber2017/mirex2017 (233 differences): ‘10118334.clip’ ‘10258351.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ ‘12185327.clip’ … CSV

1.0 compared with schreiber2018/cnn (332 differences): ‘10118334.clip’ ‘10258351.clip’ ‘10389587.clip’ ‘104222.clip’ ‘104260.clip’ ‘10503810.clip’ ‘10563001.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10726058.clip’ ‘10726076.clip’ … CSV

1.0 compared with schreiber2018/fcn (334 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10503810.clip’ ‘10563001.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10726058.clip’ ‘10766976.clip’ ‘10809231.clip’ … CSV

1.0 compared with schreiber2018/ismir2018 (400 differences): ‘10118334.clip’ ‘10258351.clip’ ‘10389587.clip’ ‘104222.clip’ ‘104260.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10809231.clip’ ‘10875997.clip’ ‘10893272.clip’ … CSV

1.0 compared with sun2021/default (341 differences): ‘10118334.clip’ ‘10258351.clip’ ‘10389587.clip’ ‘104222.clip’ ‘104260.clip’ ‘10503810.clip’ ‘105233.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10726039.clip’ ‘10726076.clip’ … CSV

1.0 compared with zplane/auftakt_v3 (499 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10503810.clip’ ‘1071042.clip’ ‘10726032.clip’ ‘10726058.clip’ ‘10726076.clip’ ‘1074945.clip’ … CSV

2.0 compared with boeck2015/tempodetector2016_default (366 differences): ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘105233.clip’ ‘10563001.clip’ ‘1071042.clip’ ‘10766976.clip’ ‘10875997.clip’ ‘10894070.clip’ ‘10894286.clip’ ‘1094919.clip’ … CSV

2.0 compared with boeck2019/multi_task (342 differences): ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10563001.clip’ ‘10726039.clip’ ‘10875997.clip’ ‘10894070.clip’ ‘10894286.clip’ ‘11116238.clip’ ‘11173645.clip’ … CSV

2.0 compared with boeck2019/multi_task_hjdb (354 differences): ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10563001.clip’ ‘10726039.clip’ ‘10875997.clip’ ‘10894070.clip’ ‘10894286.clip’ ‘11116238.clip’ ‘11173645.clip’ … CSV

2.0 compared with boeck2020/dar (215 differences): ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘1071042.clip’ ‘10726039.clip’ ‘10875997.clip’ ‘10894070.clip’ ‘10894286.clip’ ‘11173645.clip’ ‘11401306.clip’ ‘11622.clip’ … CSV

2.0 compared with davies2009/mirex_qm_tempotracker (479 differences): ‘10258351.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10563001.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10726076.clip’ ‘1074945.clip’ ‘10894070.clip’ ‘10894286.clip’ ‘11030307.clip’ … CSV

2.0 compared with echonest/version_3_2_1 (368 differences): ‘10258351.clip’ ‘104260.clip’ ‘10563001.clip’ ‘1074945.clip’ ‘10766976.clip’ ‘10893272.clip’ ‘10894070.clip’ ‘10894286.clip’ ‘11173645.clip’ ‘11409728.clip’ ‘11430821.clip’ … CSV

2.0 compared with gkiokas2012/default (385 differences): ‘10258351.clip’ ‘10270809.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10563001.clip’ ‘1074945.clip’ ‘10894286.clip’ ‘11030307.clip’ ‘11173645.clip’ ‘11471649.clip’ … CSV

2.0 compared with klapuri2006/percival2014 (440 differences): ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘105009.clip’ ‘105233.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10894070.clip’ ‘10894286.clip’ ‘1108465.clip’ … CSV

2.0 compared with oliveira2010/ibt (521 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10726058.clip’ ‘10726076.clip’ ‘1074945.clip’ ‘10894070.clip’ … CSV

2.0 compared with percival2014/stem (371 differences): ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10563039.clip’ ‘1071042.clip’ ‘10894286.clip’ ‘11030307.clip’ ‘11116238.clip’ ‘11173645.clip’ ‘11554698.clip’ … CSV

2.0 compared with scheirer1998/percival2014 (655 differences): ‘10118334.clip’ ‘10258351.clip’ ‘10270809.clip’ ‘10332517.clip’ ‘104186.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘105009.clip’ ‘105233.clip’ ‘10563001.clip’ … CSV

2.0 compared with schreiber2014/default (336 differences): ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘105009.clip’ ‘1071042.clip’ ‘10726058.clip’ ‘10894286.clip’ ‘11030307.clip’ ‘11116238.clip’ ‘11173645.clip’ … CSV

2.0 compared with schreiber2017/ismir2017 (256 differences): ‘10258351.clip’ ‘104260.clip’ ‘10443543.clip’ ‘10894286.clip’ ‘11173645.clip’ ‘11612127.clip’ ‘11622.clip’ ‘1162915.clip’ ‘11812770.clip’ ‘12185327.clip’ ‘1245162.clip’ … CSV

2.0 compared with schreiber2017/mirex2017 (128 differences): ‘10258351.clip’ ‘11173645.clip’ ‘11612127.clip’ ‘12185327.clip’ ‘1245162.clip’ ‘12638738.clip’ ‘129570.clip’ ‘13041056.clip’ ‘13086293.clip’ ‘1358509.clip’ ‘1393863.clip’ … CSV

2.0 compared with schreiber2018/cnn (241 differences): ‘10258351.clip’ ‘10389587.clip’ ‘104222.clip’ ‘104260.clip’ ‘10563001.clip’ ‘1071042.clip’ ‘10726058.clip’ ‘10726076.clip’ ‘10875997.clip’ ‘10894070.clip’ ‘10894286.clip’ … CSV

2.0 compared with schreiber2018/fcn (243 differences): ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10563001.clip’ ‘1071042.clip’ ‘10726058.clip’ ‘10766976.clip’ ‘10875997.clip’ ‘10894286.clip’ ‘1094938.clip’ ‘11030307.clip’ … CSV

2.0 compared with schreiber2018/ismir2018 (309 differences): ‘10258351.clip’ ‘10389587.clip’ ‘104222.clip’ ‘104260.clip’ ‘1071042.clip’ ‘10875997.clip’ ‘10894070.clip’ ‘10894286.clip’ ‘11173645.clip’ ‘11612127.clip’ ‘11622.clip’ … CSV

2.0 compared with sun2021/default (253 differences): ‘10258351.clip’ ‘10389587.clip’ ‘104222.clip’ ‘104260.clip’ ‘105233.clip’ ‘1071042.clip’ ‘10726039.clip’ ‘10726076.clip’ ‘10875997.clip’ ‘10894070.clip’ ‘10894286.clip’ … CSV

2.0 compared with zplane/auftakt_v3 (420 differences): ‘10118334.clip’ ‘10258351.clip’ ‘104222.clip’ ‘104260.clip’ ‘10443543.clip’ ‘1071042.clip’ ‘10726032.clip’ ‘10726058.clip’ ‘10726076.clip’ ‘1074945.clip’ ‘10894286.clip’ … CSV

None of the estimators estimated the following 46 items ‘correctly’ using Accuracy1: ‘10258351.clip’ ‘11173645.clip’ ‘12185327.clip’ ‘1245162.clip’ ‘12638738.clip’ ‘129570.clip’ ‘1358509.clip’ ‘1393863.clip’ ‘1433718.clip’ ‘1492191.clip’ ‘14970371.clip’ … CSV

Differing Items Accuracy2

Items with different tempo annotations (Accuracy2, 4% tolerance) in different versions:

1.0 compared with boeck2015/tempodetector2016_default (141 differences): ‘10118334.clip’ ‘104260.clip’ ‘10503810.clip’ ‘105233.clip’ ‘10563039.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘1108465.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ … CSV

1.0 compared with boeck2019/multi_task (141 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ ‘1245162.clip’ ‘12643846.clip’ ‘12906077.clip’ … CSV

1.0 compared with boeck2019/multi_task_hjdb (145 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ ‘1245162.clip’ ‘12643846.clip’ ‘12906123.clip’ … CSV

1.0 compared with boeck2020/dar (129 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ ‘1245162.clip’ ‘1248986.clip’ … CSV

1.0 compared with davies2009/mirex_qm_tempotracker (191 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10726076.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘1108465.clip’ ‘11193878.clip’ ‘11430821.clip’ ‘11552767.clip’ ‘1173974.clip’ … CSV

1.0 compared with echonest/version_3_2_1 (206 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10766976.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘1173975.clip’ ‘11883189.clip’ … CSV

1.0 compared with gkiokas2012/default (145 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ ‘1245162.clip’ ‘12643846.clip’ … CSV

1.0 compared with klapuri2006/percival2014 (160 differences): ‘10118334.clip’ ‘105009.clip’ ‘10503810.clip’ ‘105233.clip’ ‘10563039.clip’ ‘10726076.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘1108465.clip’ ‘11430821.clip’ ‘1173974.clip’ … CSV

1.0 compared with oliveira2010/ibt (197 differences): ‘10118334.clip’ ‘104260.clip’ ‘10503810.clip’ ‘10726058.clip’ ‘10726076.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘1107612.clip’ ‘11095374.clip’ ‘11173645.clip’ ‘11430821.clip’ … CSV

1.0 compared with percival2014/stem (141 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10726076.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ ‘1245162.clip’ … CSV

1.0 compared with scheirer1998/percival2014 (402 differences): ‘10118334.clip’ ‘10270809.clip’ ‘104260.clip’ ‘105009.clip’ ‘10503810.clip’ ‘10563001.clip’ ‘10563039.clip’ ‘10726076.clip’ ‘1077416.clip’ ‘10809231.clip’ ‘10875997.clip’ … CSV

1.0 compared with schreiber2014/default (170 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10726058.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ ‘1245162.clip’ … CSV

1.0 compared with schreiber2017/ismir2017 (150 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ ‘1245162.clip’ ‘12643846.clip’ … CSV

1.0 compared with schreiber2017/mirex2017 (130 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ ‘1245162.clip’ ‘12643846.clip’ … CSV

1.0 compared with schreiber2018/cnn (131 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10726058.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ ‘1245162.clip’ … CSV

1.0 compared with schreiber2018/fcn (140 differences): ‘10118334.clip’ ‘104260.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10726058.clip’ ‘10766976.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘11612127.clip’ … CSV

1.0 compared with schreiber2018/ismir2018 (145 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10563039.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ ‘1245162.clip’ ‘12643846.clip’ … CSV

1.0 compared with sun2021/default (162 differences): ‘10118334.clip’ ‘104260.clip’ ‘10503810.clip’ ‘105233.clip’ ‘10563039.clip’ ‘10726076.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘1107612.clip’ ‘11173645.clip’ ‘11430821.clip’ … CSV

1.0 compared with zplane/auftakt_v3 (189 differences): ‘10118334.clip’ ‘10503810.clip’ ‘10726032.clip’ ‘10726058.clip’ ‘10726076.clip’ ‘10809231.clip’ ‘10893272.clip’ ‘11119266.clip’ ‘11430821.clip’ ‘1173974.clip’ ‘11883189.clip’ … CSV

2.0 compared with boeck2015/tempodetector2016_default (31 differences): ‘104260.clip’ ‘105233.clip’ ‘1108465.clip’ ‘11173645.clip’ ‘1358509.clip’ ‘14600040.clip’ ‘164628.clip’ ‘1827592.clip’ ‘208474.clip’ ‘2252893.clip’ ‘2347473.clip’ … CSV

2.0 compared with boeck2019/multi_task (32 differences): ‘11173645.clip’ ‘12906077.clip’ ‘1359069.clip’ ‘1435815.clip’ ‘14600040.clip’ ‘164628.clip’ ‘168497.clip’ ‘168499.clip’ ‘2373182.clip’ ‘2517897.clip’ ‘253654.clip’ … CSV

2.0 compared with boeck2019/multi_task_hjdb (37 differences): ‘11173645.clip’ ‘1245162.clip’ ‘1359069.clip’ ‘1435815.clip’ ‘14600040.clip’ ‘164628.clip’ ‘168497.clip’ ‘168499.clip’ ‘168502.clip’ ‘2373182.clip’ ‘2517897.clip’ … CSV

2.0 compared with boeck2020/dar (13 differences): ‘11173645.clip’ ‘1245162.clip’ ‘13036093.clip’ ‘164628.clip’ ‘168499.clip’ ‘250576.clip’ ‘3333901.clip’ ‘4326130.clip’ ‘458707.clip’ ‘6018930.clip’ ‘8231796.clip’ … CSV

2.0 compared with davies2009/mirex_qm_tempotracker (92 differences): ‘10726076.clip’ ‘1108465.clip’ ‘11193878.clip’ ‘11552767.clip’ ‘12185327.clip’ ‘13036093.clip’ ‘13376370.clip’ ‘1355192.clip’ ‘1359069.clip’ ‘143682.clip’ ‘15416738.clip’ … CSV

2.0 compared with echonest/version_3_2_1 (100 differences): ‘10766976.clip’ ‘11173645.clip’ ‘11430821.clip’ ‘1173975.clip’ ‘13036093.clip’ ‘13065657.clip’ ‘13167246.clip’ ‘13376370.clip’ ‘13561397.clip’ ‘1385105.clip’ ‘13851876.clip’ … CSV

2.0 compared with gkiokas2012/default (28 differences): ‘11173645.clip’ ‘13036093.clip’ ‘1393863.clip’ ‘15416738.clip’ ‘168499.clip’ ‘1827592.clip’ ‘2252893.clip’ ‘2284109.clip’ ‘2347473.clip’ ‘256907.clip’ ‘281623.clip’ … CSV

2.0 compared with klapuri2006/percival2014 (44 differences): ‘105009.clip’ ‘105233.clip’ ‘1108465.clip’ ‘129570.clip’ ‘13376370.clip’ ‘14180121.clip’ ‘14600040.clip’ ‘166285.clip’ ‘168497.clip’ ‘168499.clip’ ‘168502.clip’ … CSV

2.0 compared with oliveira2010/ibt (97 differences): ‘104260.clip’ ‘10563039.clip’ ‘10726058.clip’ ‘10726076.clip’ ‘1107612.clip’ ‘11095374.clip’ ‘11173645.clip’ ‘12185327.clip’ ‘12643846.clip’ ‘129570.clip’ ‘13036093.clip’ … CSV

2.0 compared with percival2014/stem (29 differences): ‘11173645.clip’ ‘1245162.clip’ ‘13851876.clip’ ‘14600040.clip’ ‘15416738.clip’ ‘164628.clip’ ‘168499.clip’ ‘1949678.clip’ ‘2232914.clip’ ‘2411992.clip’ ‘282894.clip’ … CSV

2.0 compared with scheirer1998/percival2014 (321 differences): ‘10118334.clip’ ‘10270809.clip’ ‘104260.clip’ ‘105009.clip’ ‘10563001.clip’ ‘10726076.clip’ ‘1077416.clip’ ‘10875997.clip’ ‘10893272.clip’ ‘1103942.clip’ ‘1108465.clip’ … CSV

2.0 compared with schreiber2014/default (56 differences): ‘10726058.clip’ ‘11173645.clip’ ‘11612127.clip’ ‘14180121.clip’ ‘1447613.clip’ ‘14600040.clip’ ‘14634350.clip’ ‘15952618.clip’ ‘164628.clip’ ‘168497.clip’ ‘168499.clip’ … CSV

2.0 compared with schreiber2017/ismir2017 (38 differences): ‘11173645.clip’ ‘11612127.clip’ ‘1245162.clip’ ‘14180121.clip’ ‘1447613.clip’ ‘14600040.clip’ ‘15952618.clip’ ‘164628.clip’ ‘1827592.clip’ ‘2214125.clip’ ‘2232914.clip’ … CSV

2.0 compared with schreiber2017/mirex2017 (18 differences): ‘11173645.clip’ ‘11612127.clip’ ‘1245162.clip’ ‘1447613.clip’ ‘14600040.clip’ ‘164628.clip’ ‘2232914.clip’ ‘2233308.clip’ ‘3188766.clip’ ‘3732728.clip’ ‘4326130.clip’ … CSV

2.0 compared with schreiber2018/cnn (22 differences): ‘10726058.clip’ ‘11173645.clip’ ‘11612127.clip’ ‘14600040.clip’ ‘15416738.clip’ ‘164628.clip’ ‘168499.clip’ ‘281623.clip’ ‘282894.clip’ ‘3110264.clip’ ‘3176577.clip’ … CSV

2.0 compared with schreiber2018/fcn (29 differences): ‘104260.clip’ ‘10726058.clip’ ‘10766976.clip’ ‘11173645.clip’ ‘14600040.clip’ ‘14634350.clip’ ‘15416738.clip’ ‘164628.clip’ ‘168499.clip’ ‘281623.clip’ ‘3030777.clip’ … CSV

2.0 compared with schreiber2018/ismir2018 (33 differences): ‘11173645.clip’ ‘11612127.clip’ ‘1245162.clip’ ‘12937673.clip’ ‘13851876.clip’ ‘14600040.clip’ ‘15416738.clip’ ‘168499.clip’ ‘1827592.clip’ ‘3073826.clip’ ‘3110264.clip’ … CSV

2.0 compared with sun2021/default (51 differences): ‘104260.clip’ ‘105233.clip’ ‘10726076.clip’ ‘1107612.clip’ ‘11173645.clip’ ‘12185327.clip’ ‘14180121.clip’ ‘15952618.clip’ ‘168499.clip’ ‘1827592.clip’ ‘2284109.clip’ … CSV

2.0 compared with zplane/auftakt_v3 (86 differences): ‘10726032.clip’ ‘10726058.clip’ ‘10726076.clip’ ‘11119266.clip’ ‘11612127.clip’ ‘1173974.clip’ ‘1245162.clip’ ‘12937673.clip’ ‘13036093.clip’ ‘13376370.clip’ ‘1385105.clip’ … CSV

All tracks were estimated ‘correctly’ by at least one system.

Significance of Differences

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.1783 0.5183 0.0000 0.0000 0.9574 0.3552 0.0001 0.0000 0.8385 0.0000 0.1410 0.0000 0.0000 0.0000 0.0000 0.0005 0.0000 0.0063
boeck2019/multi_task 0.1783 1.0000 0.1691 0.0000 0.0000 0.1649 0.0242 0.0000 0.0000 0.0725 0.0000 0.7416 0.0000 0.0000 0.0000 0.0000 0.0393 0.0000 0.0000
boeck2019/multi_task_hjdb 0.5183 0.1691 1.0000 0.0000 0.0000 0.4821 0.1093 0.0000 0.0000 0.3416 0.0000 0.2954 0.0000 0.0000 0.0000 0.0000 0.0051 0.0000 0.0002
boeck2020/dar 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0054 0.0000 0.0566 0.0501 0.0000 0.0019 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0077 0.0032 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0012
echonest/version_3_2_1 0.9574 0.1649 0.4821 0.0000 0.0000 1.0000 0.3904 0.0001 0.0000 0.9125 0.0000 0.0773 0.0000 0.0000 0.0000 0.0000 0.0006 0.0000 0.0055
gkiokas2012/default 0.3552 0.0242 0.1093 0.0000 0.0000 0.3904 1.0000 0.0028 0.0000 0.4306 0.0000 0.0057 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0577
klapuri2006/percival2014 0.0001 0.0000 0.0000 0.0000 0.0077 0.0001 0.0028 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2369
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0032 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.8385 0.0725 0.3416 0.0000 0.0000 0.9125 0.4306 0.0000 0.0000 1.0000 0.0000 0.0120 0.0000 0.0000 0.0000 0.0000 0.0001 0.0000 0.0035
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.1410 0.7416 0.2954 0.0000 0.0000 0.0773 0.0057 0.0000 0.0000 0.0120 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0871 0.0000 0.0000
schreiber2017/ismir2017 0.0000 0.0000 0.0000 0.0054 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.3282 0.4020 0.0003 0.8944 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0000 0.0000 0.0566 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.3282 0.0000 1.0000 0.9385 0.0000 0.3904 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0000 0.0501 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4020 0.0000 0.9385 1.0000 0.0000 0.5182 0.0000
schreiber2018/ismir2018 0.0005 0.0393 0.0051 0.0000 0.0000 0.0006 0.0000 0.0000 0.0000 0.0001 0.0000 0.0871 0.0003 0.0000 0.0000 0.0000 1.0000 0.0002 0.0000
sun2021/default 0.0000 0.0000 0.0000 0.0019 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.8944 0.0000 0.3904 0.5182 0.0002 1.0000 0.0000
zplane/auftakt_v3 0.0063 0.0000 0.0002 0.0000 0.0012 0.0055 0.0577 0.2369 0.0000 0.0035 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 5: McNemar p-values, using reference annotations 2.0 as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.5435 1.0000 0.0000 0.0000 0.5030 0.0680 0.0000 0.0000 0.2657 0.0000 0.6703 0.0000 0.0000 0.0000 0.0000 0.0057 0.0000 0.0031
boeck2019/multi_task 0.5435 1.0000 0.1550 0.0000 0.0000 0.1900 0.0124 0.0000 0.0000 0.0368 0.0000 0.9455 0.0000 0.0000 0.0000 0.0000 0.0401 0.0000 0.0000
boeck2019/multi_task_hjdb 1.0000 0.1550 1.0000 0.0000 0.0000 0.5411 0.0676 0.0000 0.0000 0.2295 0.0000 0.5692 0.0000 0.0000 0.0000 0.0000 0.0048 0.0000 0.0012
boeck2020/dar 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0009 0.0000 0.0350 0.0307 0.0000 0.0028 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0001 0.0727 0.0038 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0025
echonest/version_3_2_1 0.5030 0.1900 0.5411 0.0000 0.0000 1.0000 0.2360 0.0001 0.0000 0.6490 0.0000 0.2143 0.0000 0.0000 0.0000 0.0000 0.0007 0.0000 0.0172
gkiokas2012/default 0.0680 0.0124 0.0676 0.0000 0.0001 0.2360 1.0000 0.0068 0.0000 0.4541 0.0000 0.0084 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2410
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0727 0.0001 0.0068 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0899
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0038 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.2657 0.0368 0.2295 0.0000 0.0000 0.6490 0.4541 0.0000 0.0000 1.0000 0.0000 0.0210 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0361
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.6703 0.9455 0.5692 0.0000 0.0000 0.2143 0.0084 0.0000 0.0000 0.0210 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0234 0.0000 0.0000
schreiber2017/ismir2017 0.0000 0.0000 0.0000 0.0009 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.2020 0.2567 0.0007 0.5476 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0000 0.0000 0.0350 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2020 0.0000 1.0000 0.9362 0.0000 0.5309 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0000 0.0307 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2567 0.0000 0.9362 1.0000 0.0000 0.6556 0.0000
schreiber2018/ismir2018 0.0057 0.0401 0.0048 0.0000 0.0000 0.0007 0.0000 0.0000 0.0000 0.0000 0.0000 0.0234 0.0007 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000
sun2021/default 0.0000 0.0000 0.0000 0.0028 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5476 0.0000 0.5309 0.6556 0.0000 1.0000 0.0000
zplane/auftakt_v3 0.0031 0.0000 0.0012 0.0000 0.0025 0.0172 0.2410 0.0899 0.0000 0.0361 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 6: McNemar p-values, using reference annotations 1.0 as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 1.0000 0.4799 0.0039 0.0000 0.0000 0.7552 0.1175 0.0000 0.8804 0.0000 0.0026 0.3817 0.0470 0.2110 0.8830 0.8804 0.0105 0.0000
boeck2019/multi_task 1.0000 1.0000 0.2266 0.0005 0.0000 0.0000 0.6177 0.1550 0.0000 0.7359 0.0000 0.0012 0.4709 0.0243 0.1102 0.7428 1.0000 0.0127 0.0000
boeck2019/multi_task_hjdb 0.4799 0.2266 1.0000 0.0000 0.0000 0.0000 0.2327 0.4500 0.0000 0.2800 0.0000 0.0145 1.0000 0.0019 0.0201 0.2800 0.6718 0.0925 0.0000
boeck2020/dar 0.0039 0.0005 0.0000 1.0000 0.0000 0.0000 0.0081 0.0000 0.0000 0.0025 0.0000 0.0000 0.0000 0.3018 0.0784 0.0025 0.0005 0.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.5625 0.0000 0.0000 0.7139 0.0000 0.0000 0.0013 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002 0.6612
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.5625 1.0000 0.0000 0.0000 0.8467 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1980
gkiokas2012/default 0.7552 0.6177 0.2327 0.0081 0.0000 0.0000 1.0000 0.0365 0.0000 1.0000 0.0000 0.0000 0.1539 0.1325 0.3449 1.0000 0.4731 0.0004 0.0000
klapuri2006/percival2014 0.1175 0.1550 0.4500 0.0000 0.0000 0.0000 0.0365 1.0000 0.0000 0.0444 0.0000 0.1550 0.4966 0.0003 0.0009 0.0627 0.1690 0.4011 0.0000
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.7139 0.8467 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.3049
percival2014/stem 0.8804 0.7359 0.2800 0.0025 0.0000 0.0000 1.0000 0.0444 0.0000 1.0000 0.0000 0.0004 0.2221 0.0522 0.2478 1.0000 0.5966 0.0026 0.0000
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0026 0.0012 0.0145 0.0000 0.0013 0.0000 0.0000 0.1550 0.0000 0.0004 0.0000 1.0000 0.0009 0.0000 0.0000 0.0000 0.0014 0.5758 0.0002
schreiber2017/ismir2017 0.3817 0.4709 1.0000 0.0000 0.0000 0.0000 0.1539 0.4966 0.0000 0.2221 0.0000 0.0009 1.0000 0.0000 0.0090 0.1877 0.4996 0.0854 0.0000
schreiber2017/mirex2017 0.0470 0.0243 0.0019 0.3018 0.0000 0.0000 0.1325 0.0003 0.0000 0.0522 0.0000 0.0000 0.0000 1.0000 0.5235 0.0433 0.0107 0.0000 0.0000
schreiber2018/cnn 0.2110 0.1102 0.0201 0.0784 0.0000 0.0000 0.3449 0.0009 0.0000 0.2478 0.0000 0.0000 0.0090 0.5235 1.0000 0.1892 0.0614 0.0000 0.0000
schreiber2018/fcn 0.8830 0.7428 0.2800 0.0025 0.0000 0.0000 1.0000 0.0627 0.0000 1.0000 0.0000 0.0000 0.1877 0.0433 0.1892 1.0000 0.6076 0.0026 0.0000
schreiber2018/ismir2018 0.8804 1.0000 0.6718 0.0005 0.0000 0.0000 0.4731 0.1690 0.0000 0.5966 0.0000 0.0014 0.4996 0.0107 0.0614 0.6076 1.0000 0.0133 0.0000
sun2021/default 0.0105 0.0127 0.0925 0.0000 0.0002 0.0000 0.0004 0.4011 0.0000 0.0026 0.0000 0.5758 0.0854 0.0000 0.0000 0.0026 0.0133 1.0000 0.0001
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.6612 0.1980 0.0000 0.0000 0.3049 0.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000

Table 7: McNemar p-values, using reference annotations 2.0 as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.8918 0.6936 0.0884 0.0000 0.0000 0.6778 0.0327 0.0000 0.8918 0.0000 0.0008 0.2717 0.1263 0.2116 1.0000 0.6655 0.0154 0.0000
boeck2019/multi_task 0.8918 1.0000 0.3437 0.0501 0.0000 0.0000 0.6516 0.0248 0.0000 1.0000 0.0000 0.0001 0.2221 0.0614 0.1325 1.0000 0.6655 0.0170 0.0000
boeck2019/multi_task_hjdb 0.6936 0.3437 1.0000 0.0070 0.0000 0.0000 1.0000 0.0919 0.0000 0.6587 0.0000 0.0013 0.5515 0.0135 0.0385 0.5601 0.8918 0.0647 0.0000
boeck2020/dar 0.0884 0.0501 0.0070 1.0000 0.0000 0.0000 0.0090 0.0000 0.0000 0.0501 0.0000 0.0000 0.0008 1.0000 0.8555 0.0895 0.0139 0.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.2699 0.0000 0.0041 0.6634 0.0000 0.0000 0.0783 0.0002 0.0000 0.0000 0.0000 0.0000 0.0176 0.9312
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.2699 1.0000 0.0000 0.0000 0.4517 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1219
gkiokas2012/default 0.6778 0.6516 1.0000 0.0090 0.0000 0.0000 1.0000 0.0722 0.0000 0.6516 0.0000 0.0003 0.5515 0.0275 0.0288 0.5424 1.0000 0.0270 0.0000
klapuri2006/percival2014 0.0327 0.0248 0.0919 0.0000 0.0041 0.0000 0.0722 1.0000 0.0002 0.0163 0.0000 0.2530 0.2203 0.0000 0.0001 0.0225 0.0674 0.9020 0.0008
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.6634 0.4517 0.0000 0.0002 1.0000 0.0000 0.0000 0.0048 0.0000 0.0000 0.0000 0.0000 0.0000 0.0005 0.4655
percival2014/stem 0.8918 1.0000 0.6587 0.0501 0.0000 0.0000 0.6516 0.0163 0.0000 1.0000 0.0000 0.0001 0.2430 0.0708 0.1742 1.0000 0.6440 0.0125 0.0000
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0008 0.0001 0.0013 0.0000 0.0783 0.0001 0.0003 0.2530 0.0048 0.0001 0.0000 1.0000 0.0001 0.0000 0.0000 0.0000 0.0005 0.3662 0.0204
schreiber2017/ismir2017 0.2717 0.2221 0.5515 0.0008 0.0002 0.0000 0.5515 0.2203 0.0000 0.2430 0.0000 0.0001 1.0000 0.0000 0.0034 0.1641 0.5327 0.1550 0.0000
schreiber2017/mirex2017 0.1263 0.0614 0.0135 1.0000 0.0000 0.0000 0.0275 0.0000 0.0000 0.0708 0.0000 0.0000 0.0000 1.0000 1.0000 0.0987 0.0201 0.0001 0.0000
schreiber2018/cnn 0.2116 0.1325 0.0385 0.8555 0.0000 0.0000 0.0288 0.0001 0.0000 0.1742 0.0000 0.0000 0.0034 1.0000 1.0000 0.1221 0.0201 0.0000 0.0000
schreiber2018/fcn 1.0000 1.0000 0.5601 0.0895 0.0000 0.0000 0.5424 0.0225 0.0000 1.0000 0.0000 0.0000 0.1641 0.0987 0.1221 1.0000 0.5114 0.0054 0.0000
schreiber2018/ismir2018 0.6655 0.6655 0.8918 0.0139 0.0000 0.0000 1.0000 0.0674 0.0000 0.6440 0.0000 0.0005 0.5327 0.0201 0.0201 0.5114 1.0000 0.0270 0.0000
sun2021/default 0.0154 0.0170 0.0647 0.0000 0.0176 0.0000 0.0270 0.9020 0.0005 0.0125 0.0000 0.3662 0.1550 0.0001 0.0000 0.0054 0.0270 1.0000 0.0053
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.9312 0.1219 0.0000 0.0008 0.4655 0.0000 0.0000 0.0204 0.0000 0.0000 0.0000 0.0000 0.0000 0.0053 1.0000

Table 8: McNemar p-values, using reference annotations 1.0 as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Accuracy1 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean Accuracy1 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

Accuracy1 on Tempo-Subsets for 1.0

Figure 7: Mean Accuracy1 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on Tempo-Subsets for 2.0

Figure 8: Mean Accuracy1 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean Accuracy2 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

Accuracy2 on Tempo-Subsets for 1.0

Figure 9: Mean Accuracy2 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on Tempo-Subsets for 2.0

Figure 10: Mean Accuracy2 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy1 for Tempo

When fitting a generalized additive model (GAM) to Accuracy1-values and a ground truth, what Accuracy1 can we expect with confidence?

Estimated Accuracy1 for Tempo for 1.0

Predictions of GAMs trained on Accuracy1 for estimates for reference 1.0.

Figure 11: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy1 for Tempo for 2.0

Predictions of GAMs trained on Accuracy1 for estimates for reference 2.0.

Figure 12: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy2 for Tempo

When fitting a generalized additive model (GAM) to Accuracy2-values and a ground truth, what Accuracy2 can we expect with confidence?

Estimated Accuracy2 for Tempo for 1.0

Predictions of GAMs trained on Accuracy2 for estimates for reference 1.0.

Figure 13: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy2 for Tempo for 2.0

Predictions of GAMs trained on Accuracy2 for estimates for reference 2.0.

Figure 14: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 and OE2

OE1 is defined as octave error between an estimate E and a reference value R.This means that the most common errors—by a factor of 2 or ½—have the same magnitude, namely 1: OE2(E) = log2(E/R).

OE2 is the signed OE1 corresponding to the minimum absolute OE1 allowing the octaveerrors 2, 3, 1/2, and 1/3: OE2(E) = arg minx(|x|) with x ∈ {OE1(E), OE1(2E), OE1(3E), OE1(½E), OE1(⅓E)}

Mean OE1/OE2 Results for 1.0

Estimator OE1_MEAN OE1_STDEV OE2_MEAN OE2_STDEV
schreiber2017/mirex2017 0.0430 0.3048 -0.0044 0.0699
sun2021/default 0.1207 0.3878 -0.0172 0.0879
schreiber2018/cnn 0.1386 0.3923 -0.0046 0.0729
boeck2020/dar 0.1026 0.3999 -0.0031 0.0683
schreiber2018/fcn 0.1050 0.4075 -0.0064 0.0772
schreiber2017/ismir2017 0.0678 0.4191 -0.0091 0.0859
schreiber2018/ismir2018 0.1355 0.4572 -0.0088 0.0824
echonest/version_3_2_1 0.0985 0.4642 -0.0089 0.1044
schreiber2014/default -0.0013 0.4664 -0.0159 0.0956
boeck2019/multi_task 0.0907 0.4862 -0.0033 0.0793
boeck2019/multi_task_hjdb 0.0997 0.4925 -0.0025 0.0819
boeck2015/tempodetector2016_default 0.2135 0.4983 -0.0046 0.0765
percival2014/stem 0.0307 0.5103 -0.0040 0.0751
zplane/auftakt_v3 0.1116 0.5119 -0.0136 0.1059
gkiokas2012/default 0.0821 0.5199 -0.0076 0.0831
davies2009/mirex_qm_tempotracker 0.2614 0.5321 0.0221 0.0914
oliveira2010/ibt 0.2509 0.5421 -0.0066 0.0984
klapuri2006/percival2014 0.1444 0.5490 -0.0051 0.0839
scheirer1998/percival2014 0.0736 0.5693 0.0302 0.1589

Table 9: Mean OE1/OE2 for estimates compared to version 1.0 ordered by standard deviation.

CSV JSON LATEX PICKLE

Raw data OE1: CSV JSON LATEX PICKLE

Raw data OE2: CSV JSON LATEX PICKLE

OE1 distribution for 1.0

Figure 15: OE1 for estimates compared to version 1.0. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 distribution for 1.0

Figure 16: OE2 for estimates compared to version 1.0. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean OE1/OE2 Results for 2.0

Estimator OE1_MEAN OE1_STDEV OE2_MEAN OE2_STDEV
schreiber2017/mirex2017 0.0394 0.2958 -0.0044 0.0384
sun2021/default 0.1172 0.3845 -0.0187 0.0664
schreiber2018/cnn 0.1350 0.3886 -0.0046 0.0441
boeck2020/dar 0.0991 0.3962 -0.0041 0.0368
schreiber2018/fcn 0.1014 0.4024 -0.0064 0.0517
schreiber2017/ismir2017 0.0643 0.4141 -0.0091 0.0629
schreiber2018/ismir2018 0.1320 0.4518 -0.0084 0.0590
echonest/version_3_2_1 0.0950 0.4614 -0.0132 0.0875
schreiber2014/default -0.0048 0.4615 -0.0159 0.0760
boeck2019/multi_task 0.0872 0.4829 -0.0050 0.0561
boeck2019/multi_task_hjdb 0.0962 0.4887 -0.0056 0.0616
boeck2015/tempodetector2016_default 0.2100 0.4957 -0.0049 0.0487
zplane/auftakt_v3 0.1080 0.5032 -0.0183 0.0921
percival2014/stem 0.0271 0.5040 -0.0036 0.0510
gkiokas2012/default 0.0786 0.5136 -0.0074 0.0581
davies2009/mirex_qm_tempotracker 0.2578 0.5266 0.0229 0.0714
oliveira2010/ibt 0.2473 0.5368 -0.0074 0.0815
klapuri2006/percival2014 0.1409 0.5433 -0.0053 0.0580
scheirer1998/percival2014 0.0697 0.5681 0.0306 0.1516

Table 10: Mean OE1/OE2 for estimates compared to version 2.0 ordered by standard deviation.

CSV JSON LATEX PICKLE

Raw data OE1: CSV JSON LATEX PICKLE

Raw data OE2: CSV JSON LATEX PICKLE

OE1 distribution for 2.0

Figure 17: OE1 for estimates compared to version 2.0. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 distribution for 2.0

Figure 18: OE2 for estimates compared to version 2.0. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Significance of Differences

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0067 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.1105 0.2844 0.0000 0.5284 0.5252 0.0000 0.0000 0.0000 0.3366 0.0000 0.0314 0.0000 0.0000 0.2300 0.0001 0.0111 0.0805
boeck2019/multi_task_hjdb 0.0000 0.1105 1.0000 0.7938 0.0000 0.9264 0.1975 0.0003 0.0000 0.0000 0.0873 0.0000 0.0042 0.0000 0.0006 0.6574 0.0014 0.0752 0.3264
boeck2020/dar 0.0000 0.2844 0.7938 1.0000 0.0000 0.7256 0.1375 0.0018 0.0000 0.0000 0.0573 0.0000 0.0006 0.0000 0.0001 0.8179 0.0018 0.0322 0.4812
davies2009/mirex_qm_tempotracker 0.0001 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.2650 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0000 0.5284 0.9264 0.7256 0.0000 1.0000 0.2104 0.0004 0.0000 0.0000 0.0453 0.0000 0.0064 0.0000 0.0005 0.5907 0.0018 0.0603 0.3123
gkiokas2012/default 0.0000 0.5252 0.1975 0.1375 0.0000 0.2104 1.0000 0.0000 0.0000 0.0000 0.2579 0.0000 0.2650 0.0026 0.0000 0.0860 0.0000 0.0032 0.0163
klapuri2006/percival2014 0.0000 0.0000 0.0003 0.0018 0.0000 0.0004 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6424 0.0045 0.4080 0.0845 0.0025
oliveira2010/ibt 0.0067 0.0000 0.0000 0.0000 0.2650 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0062 0.0004 0.0011 0.3013 0.0000 0.0000 0.0000 0.0000 0.0000
scheirer1998/percival2014 0.0000 0.3366 0.0873 0.0573 0.0000 0.0453 0.2579 0.0000 0.0000 0.0062 1.0000 0.0000 0.9128 0.0463 0.0000 0.0324 0.0000 0.0017 0.0074
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/ismir2017 0.0000 0.0314 0.0042 0.0006 0.0000 0.0064 0.2650 0.0000 0.0000 0.0011 0.9128 0.0000 1.0000 0.0044 0.0000 0.0003 0.0000 0.0000 0.0002
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0026 0.0000 0.0000 0.3013 0.0463 0.0000 0.0044 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0000 0.0006 0.0001 0.0000 0.0005 0.0000 0.6424 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0004 0.7457 0.0526 0.0258
schreiber2018/fcn 0.0000 0.2300 0.6574 0.8179 0.0000 0.5907 0.0860 0.0045 0.0000 0.0000 0.0324 0.0000 0.0003 0.0000 0.0004 1.0000 0.0050 0.1092 0.6207
schreiber2018/ismir2018 0.0000 0.0001 0.0014 0.0018 0.0000 0.0018 0.0000 0.4080 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7457 0.0050 1.0000 0.1645 0.0341
sun2021/default 0.0000 0.0111 0.0752 0.0322 0.0000 0.0603 0.0032 0.0845 0.0000 0.0000 0.0017 0.0000 0.0000 0.0000 0.0526 0.1092 0.1645 1.0000 0.4676
zplane/auftakt_v3 0.0000 0.0805 0.3264 0.4812 0.0000 0.3123 0.0163 0.0025 0.0000 0.0000 0.0074 0.0000 0.0002 0.0000 0.0258 0.6207 0.0341 0.4676 1.0000

Table 11: Paired t-test p-values, using reference annotations 2.0 as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0067 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.1105 0.2844 0.0000 0.5284 0.5252 0.0000 0.0000 0.0000 0.3366 0.0000 0.0314 0.0000 0.0000 0.2300 0.0001 0.0111 0.0805
boeck2019/multi_task_hjdb 0.0000 0.1105 1.0000 0.7938 0.0000 0.9264 0.1975 0.0003 0.0000 0.0000 0.0873 0.0000 0.0042 0.0000 0.0006 0.6574 0.0014 0.0752 0.3264
boeck2020/dar 0.0000 0.2844 0.7938 1.0000 0.0000 0.7256 0.1375 0.0018 0.0000 0.0000 0.0573 0.0000 0.0006 0.0000 0.0001 0.8179 0.0018 0.0322 0.4812
davies2009/mirex_qm_tempotracker 0.0001 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.2650 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0000 0.5284 0.9264 0.7256 0.0000 1.0000 0.2104 0.0004 0.0000 0.0000 0.0453 0.0000 0.0064 0.0000 0.0005 0.5907 0.0018 0.0603 0.3123
gkiokas2012/default 0.0000 0.5252 0.1975 0.1375 0.0000 0.2104 1.0000 0.0000 0.0000 0.0000 0.2579 0.0000 0.2650 0.0026 0.0000 0.0860 0.0000 0.0032 0.0163
klapuri2006/percival2014 0.0000 0.0000 0.0003 0.0018 0.0000 0.0004 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6424 0.0045 0.4080 0.0845 0.0025
oliveira2010/ibt 0.0067 0.0000 0.0000 0.0000 0.2650 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0062 0.0004 0.0011 0.3013 0.0000 0.0000 0.0000 0.0000 0.0000
scheirer1998/percival2014 0.0000 0.3366 0.0873 0.0573 0.0000 0.0453 0.2579 0.0000 0.0000 0.0062 1.0000 0.0000 0.9128 0.0463 0.0000 0.0324 0.0000 0.0017 0.0074
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/ismir2017 0.0000 0.0314 0.0042 0.0006 0.0000 0.0064 0.2650 0.0000 0.0000 0.0011 0.9128 0.0000 1.0000 0.0044 0.0000 0.0003 0.0000 0.0000 0.0002
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0026 0.0000 0.0000 0.3013 0.0463 0.0000 0.0044 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0000 0.0006 0.0001 0.0000 0.0005 0.0000 0.6424 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0004 0.7457 0.0526 0.0258
schreiber2018/fcn 0.0000 0.2300 0.6574 0.8179 0.0000 0.5907 0.0860 0.0045 0.0000 0.0000 0.0324 0.0000 0.0003 0.0000 0.0004 1.0000 0.0050 0.1092 0.6207
schreiber2018/ismir2018 0.0000 0.0001 0.0014 0.0018 0.0000 0.0018 0.0000 0.4080 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7457 0.0050 1.0000 0.1645 0.0341
sun2021/default 0.0000 0.0111 0.0752 0.0322 0.0000 0.0603 0.0032 0.0845 0.0000 0.0000 0.0017 0.0000 0.0000 0.0000 0.0526 0.1092 0.1645 1.0000 0.4676
zplane/auftakt_v3 0.0000 0.0805 0.3264 0.4812 0.0000 0.3123 0.0163 0.0025 0.0000 0.0000 0.0074 0.0000 0.0002 0.0000 0.0258 0.6207 0.0341 0.4676 1.0000

Table 12: Paired t-test p-values, using reference annotations 1.0 as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.9315 0.6956 0.5957 0.0000 0.0006 0.1088 0.8152 0.2494 0.4725 0.0000 0.0000 0.0129 0.7336 0.8650 0.3635 0.0205 0.0000 0.0000
boeck2019/multi_task 0.9315 1.0000 0.5629 0.5151 0.0000 0.0007 0.1332 0.8874 0.3180 0.3772 0.0000 0.0000 0.0307 0.6790 0.7758 0.3699 0.0460 0.0000 0.0000
boeck2019/multi_task_hjdb 0.6956 0.5629 1.0000 0.3323 0.0000 0.0023 0.3173 0.8820 0.4520 0.2961 0.0000 0.0000 0.0695 0.4587 0.5392 0.6340 0.1217 0.0000 0.0000
boeck2020/dar 0.5957 0.5151 0.3323 1.0000 0.0000 0.0001 0.0315 0.4663 0.1493 0.7277 0.0000 0.0000 0.0026 0.7923 0.6600 0.0763 0.0022 0.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0848 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0006 0.0007 0.0023 0.0001 0.0000 1.0000 0.0142 0.0010 0.0400 0.0001 0.0000 0.2504 0.0863 0.0002 0.0004 0.0039 0.0496 0.0117 0.0604
gkiokas2012/default 0.1088 0.1332 0.3173 0.0315 0.0000 0.0142 1.0000 0.2318 0.9969 0.0134 0.0000 0.0000 0.3207 0.0473 0.0383 0.4655 0.5170 0.0000 0.0000
klapuri2006/percival2014 0.8152 0.8874 0.8820 0.4663 0.0000 0.0010 0.2318 1.0000 0.3003 0.3771 0.0000 0.0000 0.0423 0.6079 0.6827 0.5557 0.1001 0.0000 0.0000
oliveira2010/ibt 0.2494 0.3180 0.4520 0.1493 0.0000 0.0400 0.9969 0.3003 1.0000 0.1095 0.0000 0.0003 0.4727 0.1855 0.2286 0.6675 0.6843 0.0000 0.0001
percival2014/stem 0.4725 0.3772 0.2961 0.7277 0.0000 0.0001 0.0134 0.3771 0.1095 1.0000 0.0000 0.0000 0.0021 0.5542 0.5121 0.0729 0.0020 0.0000 0.0000
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0848 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.2504 0.0000 0.0000 0.0003 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0859 0.2549
schreiber2017/ismir2017 0.0129 0.0307 0.0695 0.0026 0.0000 0.0863 0.3207 0.0423 0.4727 0.0021 0.0000 0.0000 1.0000 0.0011 0.0076 0.1227 0.6926 0.0000 0.0001
schreiber2017/mirex2017 0.7336 0.6790 0.4587 0.7923 0.0000 0.0002 0.0473 0.6079 0.1855 0.5542 0.0000 0.0000 0.0011 1.0000 0.8536 0.1151 0.0064 0.0000 0.0000
schreiber2018/cnn 0.8650 0.7758 0.5392 0.6600 0.0000 0.0004 0.0383 0.6827 0.2286 0.5121 0.0000 0.0000 0.0076 0.8536 1.0000 0.0896 0.0112 0.0000 0.0000
schreiber2018/fcn 0.3635 0.3699 0.6340 0.0763 0.0000 0.0039 0.4655 0.5557 0.6675 0.0729 0.0000 0.0000 0.1227 0.1151 0.0896 1.0000 0.1878 0.0000 0.0000
schreiber2018/ismir2018 0.0205 0.0460 0.1217 0.0022 0.0000 0.0496 0.5170 0.1001 0.6843 0.0020 0.0000 0.0000 0.6926 0.0064 0.0112 0.1878 1.0000 0.0000 0.0001
sun2021/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0117 0.0000 0.0000 0.0000 0.0000 0.0000 0.0859 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.8738
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0000 0.0604 0.0000 0.0000 0.0001 0.0000 0.0000 0.2549 0.0001 0.0000 0.0000 0.0000 0.0001 0.8738 1.0000

Table 13: Paired t-test p-values, using reference annotations 2.0 as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.4684 0.2678 0.3061 0.0000 0.0955 0.0925 0.7844 0.4103 0.7647 0.0000 0.0000 0.0088 0.8997 0.9845 0.2850 0.0134 0.0000 0.0008
boeck2019/multi_task 0.4684 1.0000 0.4164 0.8994 0.0000 0.0245 0.0201 0.3981 0.2038 0.6498 0.0000 0.0000 0.0030 0.4887 0.3943 0.0562 0.0041 0.0000 0.0001
boeck2019/multi_task_hjdb 0.2678 0.4164 1.0000 0.6782 0.0000 0.0120 0.0132 0.2353 0.1099 0.3874 0.0000 0.0000 0.0015 0.2814 0.2079 0.0269 0.0018 0.0000 0.0000
boeck2020/dar 0.3061 0.8994 0.6782 1.0000 0.0000 0.0236 0.0102 0.2720 0.1669 0.4681 0.0000 0.0000 0.0003 0.2214 0.1960 0.0135 0.0003 0.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0808 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0955 0.0245 0.0120 0.0236 0.0000 1.0000 0.6050 0.1602 0.4190 0.0627 0.0000 0.0096 0.9496 0.0847 0.0942 0.3147 0.9695 0.0008 0.1218
gkiokas2012/default 0.0925 0.0201 0.0132 0.0102 0.0000 0.6050 1.0000 0.1797 0.6699 0.0363 0.0000 0.0000 0.4085 0.0688 0.0635 0.4786 0.3920 0.0000 0.0246
klapuri2006/percival2014 0.7844 0.3981 0.2353 0.2720 0.0000 0.1602 0.1797 1.0000 0.4953 0.6075 0.0000 0.0000 0.0541 0.7114 0.7863 0.5354 0.0617 0.0000 0.0013
oliveira2010/ibt 0.4103 0.2038 0.1099 0.1669 0.0000 0.4190 0.6699 0.4953 1.0000 0.3200 0.0000 0.0001 0.3091 0.3850 0.4419 0.9495 0.3551 0.0000 0.0104
percival2014/stem 0.7647 0.6498 0.3874 0.4681 0.0000 0.0627 0.0363 0.6075 0.3200 1.0000 0.0000 0.0000 0.0058 0.7999 0.6924 0.1094 0.0023 0.0000 0.0001
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0808 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0096 0.0000 0.0000 0.0001 0.0000 0.0000 1.0000 0.0001 0.0000 0.0000 0.0000 0.0002 0.4677 0.3381
schreiber2017/ismir2017 0.0088 0.0030 0.0015 0.0003 0.0000 0.9496 0.4085 0.0541 0.3091 0.0058 0.0000 0.0001 1.0000 0.0011 0.0076 0.1227 0.8864 0.0000 0.0667
schreiber2017/mirex2017 0.8997 0.4887 0.2814 0.2214 0.0000 0.0847 0.0688 0.7114 0.3850 0.7999 0.0000 0.0000 0.0011 1.0000 0.8536 0.1151 0.0077 0.0000 0.0004
schreiber2018/cnn 0.9845 0.3943 0.2079 0.1960 0.0000 0.0942 0.0635 0.7863 0.4419 0.6924 0.0000 0.0000 0.0076 0.8536 1.0000 0.0896 0.0079 0.0000 0.0005
schreiber2018/fcn 0.2850 0.0562 0.0269 0.0135 0.0000 0.3147 0.4786 0.5354 0.9495 0.1094 0.0000 0.0000 0.1227 0.1151 0.0896 1.0000 0.1333 0.0000 0.0052
schreiber2018/ismir2018 0.0134 0.0041 0.0018 0.0003 0.0000 0.9695 0.3920 0.0617 0.3551 0.0023 0.0000 0.0002 0.8864 0.0077 0.0079 0.1333 1.0000 0.0000 0.0696
sun2021/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0008 0.0000 0.0000 0.0000 0.0000 0.0000 0.4677 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.1541
zplane/auftakt_v3 0.0008 0.0001 0.0000 0.0000 0.0000 0.1218 0.0246 0.0013 0.0104 0.0001 0.0000 0.3381 0.0667 0.0004 0.0005 0.0052 0.0696 0.1541 1.0000

Table 14: Paired t-test p-values, using reference annotations 1.0 as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

OE1 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean OE1 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

OE1 on Tempo-Subsets for 1.0

Figure 19: Mean OE1 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on Tempo-Subsets for 2.0

Figure 20: Mean OE1 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean OE2 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

OE2 on Tempo-Subsets for 1.0

Figure 21: Mean OE2 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on Tempo-Subsets for 2.0

Figure 22: Mean OE2 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE1 for Tempo

When fitting a generalized additive model (GAM) to OE1-values and a ground truth, what OE1 can we expect with confidence?

Estimated OE1 for Tempo for 1.0

Predictions of GAMs trained on OE1 for estimates for reference 1.0.

Figure 23: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE1 for Tempo for 2.0

Predictions of GAMs trained on OE1 for estimates for reference 2.0.

Figure 24: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE2 for Tempo

When fitting a generalized additive model (GAM) to OE2-values and a ground truth, what OE2 can we expect with confidence?

Estimated OE2 for Tempo for 1.0

Predictions of GAMs trained on OE2 for estimates for reference 1.0.

Figure 25: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE2 for Tempo for 2.0

Predictions of GAMs trained on OE2 for estimates for reference 2.0.

Figure 26: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 and AOE2

AOE1 is defined as absolute octave error between an estimate and a reference value: AOE1(E) = |log2(E/R)|.

AOE2 is the minimum of AOE1 allowing the octave errors 2, 3, 1/2, and 1/3: AOE2(E) = min(AOE1(E), AOE1(2E), AOE1(3E), AOE1(½E), AOE1(⅓E)).

Mean AOE1/AOE2 Results for 1.0

Estimator AOE1_MEAN AOE1_STDEV AOE2_MEAN AOE2_STDEV
schreiber2017/mirex2017 0.1109 0.2871 0.0262 0.0650
boeck2020/dar 0.1758 0.3735 0.0264 0.0631
schreiber2018/cnn 0.1862 0.3721 0.0271 0.0679
sun2021/default 0.1870 0.3605 0.0379 0.0812
schreiber2018/fcn 0.1882 0.3763 0.0290 0.0718
schreiber2017/ismir2017 0.1949 0.3772 0.0322 0.0801
schreiber2018/ismir2018 0.2357 0.4145 0.0307 0.0770
schreiber2014/default 0.2415 0.3990 0.0366 0.0897
echonest/version_3_2_1 0.2461 0.4057 0.0427 0.0957
boeck2019/multi_task 0.2569 0.4227 0.0304 0.0733
boeck2019/multi_task_hjdb 0.2643 0.4273 0.0313 0.0757
percival2014/stem 0.2764 0.4300 0.0285 0.0697
boeck2015/tempodetector2016_default 0.2859 0.4606 0.0310 0.0701
gkiokas2012/default 0.2877 0.4407 0.0309 0.0775
zplane/auftakt_v3 0.2922 0.4349 0.0424 0.0980
klapuri2006/percival2014 0.3268 0.4642 0.0329 0.0773
davies2009/mirex_qm_tempotracker 0.3488 0.4793 0.0440 0.0831
scheirer1998/percival2014 0.3613 0.4461 0.0804 0.1403
oliveira2010/ibt 0.3671 0.4712 0.0424 0.0890

Table 15: Mean AOE1/AOE2 for estimates compared to version 1.0 ordered by mean.

CSV JSON LATEX PICKLE

Raw data AOE1: CSV JSON LATEX PICKLE

Raw data AOE2: CSV JSON LATEX PICKLE

AOE1 distribution for 1.0

Figure 27: AOE1 for estimates compared to version 1.0. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 distribution for 1.0

Figure 28: AOE2 for estimates compared to version 1.0. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean AOE1/AOE2 Results for 2.0

Estimator AOE1_MEAN AOE1_STDEV AOE2_MEAN AOE2_STDEV
schreiber2017/mirex2017 0.0957 0.2826 0.0128 0.0365
boeck2020/dar 0.1640 0.3740 0.0131 0.0347
schreiber2018/cnn 0.1727 0.3734 0.0136 0.0422
schreiber2018/fcn 0.1748 0.3764 0.0159 0.0497
sun2021/default 0.1750 0.3618 0.0252 0.0642
schreiber2017/ismir2017 0.1808 0.3780 0.0188 0.0607
schreiber2018/ismir2018 0.2221 0.4151 0.0175 0.0570
schreiber2014/default 0.2275 0.4015 0.0233 0.0740
echonest/version_3_2_1 0.2342 0.4087 0.0302 0.0832
boeck2019/multi_task 0.2450 0.4251 0.0176 0.0535
boeck2019/multi_task_hjdb 0.2523 0.4295 0.0191 0.0588
percival2014/stem 0.2623 0.4313 0.0158 0.0486
gkiokas2012/default 0.2745 0.4411 0.0170 0.0560
boeck2015/tempodetector2016_default 0.2746 0.4631 0.0176 0.0457
zplane/auftakt_v3 0.2801 0.4318 0.0311 0.0886
klapuri2006/percival2014 0.3141 0.4651 0.0188 0.0552
davies2009/mirex_qm_tempotracker 0.3384 0.4789 0.0325 0.0676
scheirer1998/percival2014 0.3552 0.4488 0.0708 0.1375
oliveira2010/ibt 0.3562 0.4717 0.0307 0.0759

Table 16: Mean AOE1/AOE2 for estimates compared to version 2.0 ordered by mean.

CSV JSON LATEX PICKLE

Raw data AOE1: CSV JSON LATEX PICKLE

Raw data AOE2: CSV JSON LATEX PICKLE

AOE1 distribution for 2.0

Figure 29: AOE1 for estimates compared to version 2.0. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 distribution for 2.0

Figure 30: AOE2 for estimates compared to version 2.0. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Significance of Differences

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0161 0.0688 0.0000 0.0000 0.0022 0.9949 0.0035 0.0000 0.3908 0.0000 0.0008 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6914
boeck2019/multi_task 0.0161 1.0000 0.1903 0.0000 0.0000 0.3717 0.0260 0.0000 0.0000 0.1168 0.0000 0.0980 0.0000 0.0000 0.0000 0.0000 0.0359 0.0000 0.0025
boeck2019/multi_task_hjdb 0.0688 0.1903 1.0000 0.0000 0.0000 0.1451 0.0941 0.0000 0.0000 0.3994 0.0000 0.0281 0.0000 0.0000 0.0000 0.0000 0.0063 0.0000 0.0188
boeck2020/dar 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0954 0.0000 0.3469 0.2855 0.0000 0.1860 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0127 0.0552 0.0000 0.3032 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0022 0.3717 0.1451 0.0000 0.0000 1.0000 0.0015 0.0000 0.0000 0.0222 0.0000 0.5738 0.0000 0.0000 0.0000 0.0000 0.2949 0.0000 0.0003
gkiokas2012/default 0.9949 0.0260 0.0941 0.0000 0.0000 0.0015 1.0000 0.0017 0.0000 0.2922 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6477
klapuri2006/percival2014 0.0035 0.0000 0.0000 0.0000 0.0127 0.0000 0.0017 1.0000 0.0000 0.0000 0.0018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0016
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0552 0.0000 0.0000 0.0000 1.0000 0.0000 0.8445 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.3908 0.1168 0.3994 0.0000 0.0000 0.0222 0.2922 0.0000 0.0000 1.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0002 0.0000 0.1000
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.3032 0.0000 0.0000 0.0018 0.8445 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0008 0.0980 0.0281 0.0000 0.0000 0.5738 0.0001 0.0000 0.0000 0.0001 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.6058 0.0000 0.0000
schreiber2017/ismir2017 0.0000 0.0000 0.0000 0.0954 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.4215 0.5511 0.0001 0.5775 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0000 0.0000 0.3469 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4215 0.0000 1.0000 0.8258 0.0000 0.7965 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0000 0.2855 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5511 0.0000 0.8258 1.0000 0.0000 0.9785 0.0000
schreiber2018/ismir2018 0.0000 0.0359 0.0063 0.0000 0.0000 0.2949 0.0000 0.0000 0.0000 0.0002 0.0000 0.6058 0.0001 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000
sun2021/default 0.0000 0.0000 0.0000 0.1860 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5775 0.0000 0.7965 0.9785 0.0000 1.0000 0.0000
zplane/auftakt_v3 0.6914 0.0025 0.0188 0.0000 0.0000 0.0003 0.6477 0.0016 0.0000 0.1000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 17: Paired t-test p-values, using reference annotations 2.0 as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0164 0.0738 0.0000 0.0000 0.0022 0.8944 0.0022 0.0000 0.5050 0.0000 0.0014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6397
boeck2019/multi_task 0.0164 1.0000 0.1764 0.0000 0.0000 0.3636 0.0179 0.0000 0.0000 0.0723 0.0000 0.1407 0.0000 0.0000 0.0000 0.0000 0.0497 0.0000 0.0019
boeck2019/multi_task_hjdb 0.0738 0.1764 1.0000 0.0000 0.0000 0.1351 0.0728 0.0000 0.0000 0.3027 0.0000 0.0410 0.0000 0.0000 0.0000 0.0000 0.0088 0.0000 0.0167
boeck2020/dar 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0555 0.0000 0.2548 0.2104 0.0000 0.1759 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0221 0.0469 0.0000 0.4824 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0022 0.3636 0.1351 0.0000 0.0000 1.0000 0.0008 0.0000 0.0000 0.0122 0.0000 0.6968 0.0000 0.0000 0.0000 0.0000 0.3647 0.0000 0.0002
gkiokas2012/default 0.8944 0.0179 0.0728 0.0000 0.0000 0.0008 1.0000 0.0017 0.0000 0.3238 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7065
klapuri2006/percival2014 0.0022 0.0000 0.0000 0.0000 0.0221 0.0000 0.0017 1.0000 0.0000 0.0000 0.0084 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0012
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0469 0.0000 0.0000 0.0000 1.0000 0.0000 0.5590 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.5050 0.0723 0.3027 0.0000 0.0000 0.0122 0.3238 0.0000 0.0000 1.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0002 0.0000 0.1391
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.4824 0.0000 0.0000 0.0084 0.5590 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0014 0.1407 0.0410 0.0000 0.0000 0.6968 0.0001 0.0000 0.0000 0.0001 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.5818 0.0000 0.0000
schreiber2017/ismir2017 0.0000 0.0000 0.0000 0.0555 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.3870 0.5129 0.0001 0.4431 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0000 0.0000 0.2548 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.3870 0.0000 1.0000 0.8220 0.0000 0.9280 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0000 0.2104 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5129 0.0000 0.8220 1.0000 0.0000 0.8928 0.0000
schreiber2018/ismir2018 0.0000 0.0497 0.0088 0.0000 0.0000 0.3647 0.0000 0.0000 0.0000 0.0002 0.0000 0.5818 0.0001 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000
sun2021/default 0.0000 0.0000 0.0000 0.1759 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4431 0.0000 0.9280 0.8928 0.0000 1.0000 0.0000
zplane/auftakt_v3 0.6397 0.0019 0.0167 0.0000 0.0000 0.0002 0.7065 0.0012 0.0000 0.1391 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 18: Paired t-test p-values, using reference annotations 1.0 as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.9937 0.4098 0.0007 0.0000 0.0000 0.6997 0.4706 0.0000 0.2819 0.0000 0.0039 0.4474 0.0003 0.0070 0.2751 0.9416 0.0000 0.0000
boeck2019/multi_task 0.9937 1.0000 0.1345 0.0007 0.0000 0.0000 0.6958 0.5093 0.0000 0.2413 0.0000 0.0025 0.5087 0.0009 0.0041 0.2364 0.9427 0.0000 0.0000
boeck2019/multi_task_hjdb 0.4098 0.1345 1.0000 0.0000 0.0000 0.0000 0.2482 0.9069 0.0000 0.0559 0.0000 0.0295 0.8942 0.0000 0.0004 0.0486 0.3775 0.0010 0.0000
boeck2020/dar 0.0007 0.0007 0.0000 1.0000 0.0000 0.0000 0.0076 0.0002 0.0000 0.0141 0.0000 0.0000 0.0002 0.7345 0.6108 0.0184 0.0011 0.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.3969 0.0000 0.0000 0.4393 0.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0018 0.6000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.3969 1.0000 0.0000 0.0000 0.8334 0.0000 0.0000 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0084 0.7325
gkiokas2012/default 0.6997 0.6958 0.2482 0.0076 0.0000 0.0000 1.0000 0.2831 0.0000 0.4208 0.0000 0.0003 0.2681 0.0050 0.0110 0.4177 0.7234 0.0000 0.0000
klapuri2006/percival2014 0.4706 0.5093 0.9069 0.0002 0.0000 0.0000 0.2831 1.0000 0.0000 0.0667 0.0000 0.0176 0.9914 0.0001 0.0007 0.0791 0.4415 0.0001 0.0000
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.4393 0.8334 0.0000 0.0000 1.0000 0.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0057 0.8778
percival2014/stem 0.2819 0.2413 0.0559 0.0141 0.0000 0.0000 0.4208 0.0667 0.0000 1.0000 0.0000 0.0001 0.0840 0.0140 0.0762 0.9914 0.2265 0.0000 0.0000
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0039 0.0025 0.0295 0.0000 0.0002 0.0004 0.0003 0.0176 0.0002 0.0001 0.0000 1.0000 0.0009 0.0000 0.0000 0.0000 0.0009 0.2286 0.0001
schreiber2017/ismir2017 0.4474 0.5087 0.8942 0.0002 0.0000 0.0000 0.2681 0.9914 0.0000 0.0840 0.0000 0.0009 1.0000 0.0000 0.0005 0.0555 0.3866 0.0000 0.0000
schreiber2017/mirex2017 0.0003 0.0009 0.0000 0.7345 0.0000 0.0000 0.0050 0.0001 0.0000 0.0140 0.0000 0.0000 0.0000 1.0000 0.4675 0.0120 0.0007 0.0000 0.0000
schreiber2018/cnn 0.0070 0.0041 0.0004 0.6108 0.0000 0.0000 0.0110 0.0007 0.0000 0.0762 0.0000 0.0000 0.0005 0.4675 1.0000 0.0254 0.0036 0.0000 0.0000
schreiber2018/fcn 0.2751 0.2364 0.0486 0.0184 0.0000 0.0000 0.4177 0.0791 0.0000 0.9914 0.0000 0.0000 0.0555 0.0120 0.0254 1.0000 0.2300 0.0000 0.0000
schreiber2018/ismir2018 0.9416 0.9427 0.3775 0.0011 0.0000 0.0000 0.7234 0.4415 0.0000 0.2265 0.0000 0.0009 0.3866 0.0007 0.0036 0.2300 1.0000 0.0000 0.0000
sun2021/default 0.0000 0.0000 0.0010 0.0000 0.0018 0.0084 0.0000 0.0001 0.0057 0.0000 0.0000 0.2286 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0065
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.6000 0.7325 0.0000 0.0000 0.8778 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0065 1.0000

Table 19: Paired t-test p-values, using reference annotations 2.0 as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.7217 0.8502 0.0005 0.0000 0.0000 0.9722 0.2553 0.0000 0.1165 0.0000 0.0036 0.4346 0.0003 0.0078 0.2086 0.8692 0.0001 0.0000
boeck2019/multi_task 0.7217 1.0000 0.3124 0.0015 0.0000 0.0000 0.7346 0.1657 0.0000 0.1754 0.0000 0.0007 0.3079 0.0029 0.0125 0.3208 0.8276 0.0000 0.0000
boeck2019/multi_task_hjdb 0.8502 0.3124 1.0000 0.0002 0.0000 0.0000 0.8277 0.3860 0.0000 0.0806 0.0000 0.0050 0.5966 0.0004 0.0032 0.1288 0.7470 0.0002 0.0000
boeck2020/dar 0.0005 0.0015 0.0002 1.0000 0.0000 0.0000 0.0020 0.0000 0.0000 0.0631 0.0000 0.0000 0.0001 0.8028 0.4661 0.0205 0.0011 0.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.6022 0.0000 0.0000 0.4540 0.0000 0.0000 0.0029 0.0000 0.0000 0.0000 0.0000 0.0000 0.0077 0.5309
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.6022 1.0000 0.0000 0.0000 0.8885 0.0000 0.0000 0.0013 0.0000 0.0000 0.0000 0.0000 0.0000 0.0094 0.8916
gkiokas2012/default 0.9722 0.7346 0.8277 0.0020 0.0000 0.0000 1.0000 0.2317 0.0000 0.0898 0.0000 0.0008 0.4282 0.0019 0.0052 0.1812 0.8990 0.0000 0.0000
klapuri2006/percival2014 0.2553 0.1657 0.3860 0.0000 0.0000 0.0000 0.2317 1.0000 0.0000 0.0053 0.0000 0.0511 0.6883 0.0000 0.0001 0.0198 0.2004 0.0026 0.0000
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.4540 0.8885 0.0000 0.0000 1.0000 0.0000 0.0000 0.0043 0.0000 0.0000 0.0000 0.0000 0.0000 0.0218 1.0000
percival2014/stem 0.1165 0.1754 0.0806 0.0631 0.0000 0.0000 0.0898 0.0053 0.0000 1.0000 0.0000 0.0000 0.0266 0.0608 0.2647 0.6959 0.0907 0.0000 0.0000
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0036 0.0007 0.0050 0.0000 0.0029 0.0013 0.0008 0.0511 0.0043 0.0000 0.0000 1.0000 0.0007 0.0000 0.0000 0.0000 0.0004 0.4260 0.0031
schreiber2017/ismir2017 0.4346 0.3079 0.5966 0.0001 0.0000 0.0000 0.4282 0.6883 0.0000 0.0266 0.0000 0.0007 1.0000 0.0000 0.0006 0.0335 0.3211 0.0003 0.0000
schreiber2017/mirex2017 0.0003 0.0029 0.0004 0.8028 0.0000 0.0000 0.0019 0.0000 0.0000 0.0608 0.0000 0.0000 0.0000 1.0000 0.4118 0.0190 0.0008 0.0000 0.0000
schreiber2018/cnn 0.0078 0.0125 0.0032 0.4661 0.0000 0.0000 0.0052 0.0001 0.0000 0.2647 0.0000 0.0000 0.0006 0.4118 1.0000 0.0605 0.0051 0.0000 0.0000
schreiber2018/fcn 0.2086 0.3208 0.1288 0.0205 0.0000 0.0000 0.1812 0.0198 0.0000 0.6959 0.0000 0.0000 0.0335 0.0190 0.0605 1.0000 0.1904 0.0000 0.0000
schreiber2018/ismir2018 0.8692 0.8276 0.7470 0.0011 0.0000 0.0000 0.8990 0.2004 0.0000 0.0907 0.0000 0.0004 0.3211 0.0008 0.0051 0.1904 1.0000 0.0000 0.0000
sun2021/default 0.0001 0.0000 0.0002 0.0000 0.0077 0.0094 0.0000 0.0026 0.0218 0.0000 0.0000 0.4260 0.0003 0.0000 0.0000 0.0000 0.0000 1.0000 0.0267
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.5309 0.8916 0.0000 0.0000 1.0000 0.0000 0.0000 0.0031 0.0000 0.0000 0.0000 0.0000 0.0000 0.0267 1.0000

Table 20: Paired t-test p-values, using reference annotations 1.0 as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

AOE1 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean AOE1 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

AOE1 on Tempo-Subsets for 1.0

Figure 31: Mean AOE1 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on Tempo-Subsets for 2.0

Figure 32: Mean AOE1 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean AOE2 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

AOE2 on Tempo-Subsets for 1.0

Figure 33: Mean AOE2 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on Tempo-Subsets for 2.0

Figure 34: Mean AOE2 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE1 for Tempo

When fitting a generalized additive model (GAM) to AOE1-values and a ground truth, what AOE1 can we expect with confidence?

Estimated AOE1 for Tempo for 1.0

Predictions of GAMs trained on AOE1 for estimates for reference 1.0.

Figure 35: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE1 for Tempo for 2.0

Predictions of GAMs trained on AOE1 for estimates for reference 2.0.

Figure 36: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE2 for Tempo

When fitting a generalized additive model (GAM) to AOE2-values and a ground truth, what AOE2 can we expect with confidence?

Estimated AOE2 for Tempo for 1.0

Predictions of GAMs trained on AOE2 for estimates for reference 1.0.

Figure 37: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE2 for Tempo for 2.0

Predictions of GAMs trained on AOE2 for estimates for reference 2.0.

Figure 38: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG


Generated by tempo_eval 0.1.1 on 2022-06-29 18:07. Size L.