Skip to the content.

ballroom

This is the tempo_eval report for the ‘ballroom’ corpus.

Reports for other corpora may be found here.

Table of Contents

References for ‘ballroom’

References

1.0

Attribute Value
Corpus ballroom
Version 1.0
Curator Simon Dixon
Data Source BallroomDancers.com, checked by human
Annotator, bibtex Gouyon2006
Annotator, ref_url http://mtg.upf.edu/ismir2004/contest/tempoContest/node5.html

2.0

Attribute Value
Corpus ballroom
Version 2.0
Curator Florian Krebs
Data Source manual annotation
Annotation Tools derived from beat annotations
Annotation Rules median of inter beat intervals
Annotator, bibtex Krebs2013
Annotator, ref_url https://github.com/CPJKU/BallroomAnnotations

2.0-no-dupes

Attribute Value
Corpus ballroom
Version 2.0-no-dupes
Curator Florian Krebs
Data Source manual annotation
Annotation Tools derived from beat annotations
Annotation Rules median of inter beat intervals, duplicate tracks removed (http://media.aau.dk/null_space_pursuits/2014/01/ballroom-dataset.html)
Annotator, bibtex Krebs2013
Annotator, ref_url https://github.com/CPJKU/BallroomAnnotations

3.0

Attribute Value
Corpus ballroom
Version 3.0
Curator Florian Krebs
Data Source manual annotation
Annotation Tools derived from beat annotations
Annotation Rules based on median of inter corresponding beat intervals
Annotator, bibtex Krebs2013
Annotator, ref_url https://github.com/CPJKU/BallroomAnnotations

3.0-no-dupes

Attribute Value
Corpus ballroom
Version 3.0-no-dupes
Curator Florian Krebs
Data Source manual annotation
Annotation Tools derived from beat annotations
Annotation Rules based on median of inter corresponding beat intervals, duplicate tracks removed (http://media.aau.dk/null_space_pursuits/2014/01/ballroom-dataset.html)
Annotator, bibtex Krebs2013
Annotator, ref_url https://github.com/CPJKU/BallroomAnnotations

4.0

Attribute Value
Corpus ballroom
Version 4.0
Curator Graham Percival
Data Source BallroomDancers.com, checked by human
Annotator, bibtex Percival2014
Annotator, ref_url http://www.marsyas.info/tempo/

Basic Statistics

Reference Size Min Max Avg Stdev Sweet Oct. Start Sweet Oct. Coverage
1.0 698 60.00 224.00 130.14 39.53 91.00 0.71
2.0 698 68.85 214.29 129.80 39.69 72.00 0.71
2.0-no-dupes 685 68.85 214.29 130.03 39.83 72.00 0.71
3.0 698 68.57 214.29 129.77 39.72 72.00 0.71
3.0-no-dupes 685 68.57 214.29 130.00 39.87 72.00 0.71
4.0 698 58.00 219.00 129.77 39.63 71.00 0.71

Table 1: Basic statistics.

CSV JSON LATEX PICKLE

Smoothed Tempo Distribution

Figure 1: Percentage of values in tempo interval.

CSV JSON LATEX PICKLE SVG PDF PNG

Tag Distribution for ‘tag_open’

Figure 2: Percentage of tracks tagged with tags from namespace ‘tag_open’. Annotations are from reference 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Beat-Based Tempo Variation

Figure 3: Fraction of the dataset with beat-annotated tracks with cvar < τ.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimates for ‘ballroom’

Estimators

boeck2015/tempodetector2016_default

Attribute Value
Corpus ballroom
Version 0.17.dev0
Annotation Tools TempoDetector.2016, madmom, https://github.com/CPJKU/madmom
Annotator, bibtex Boeck2015

boeck2019/multi_task

Attribute Value
Corpus ballroom
Version 0.0.1
Annotation Tools model=multi_task, https://github.com/superbock/ISMIR2019
Annotator, bibtex Boeck2019

boeck2019/multi_task_hjdb

Attribute Value
Corpus ballroom
Version 0.0.1
Annotation Tools model=multi_task_hjdb, https://github.com/superbock/ISMIR2019
Annotator, bibtex Boeck2019

boeck2020/dar

Attribute Value
Corpus ballroom
Version 0.0.1
Annotation Tools https://github.com/superbock/ISMIR2020
Annotator, bibtex Boeck2020

davies2009/mirex_qm_tempotracker

Attribute Value  
Corpus ballroom  
Version 1.0  
Annotation Tools QM Tempotracker, Sonic Annotator plugin. https://code.soundsoftware.ac.uk/projects/mirex2013/repository/show/audio_tempo_estimation/qm-tempotracker Note that the current macOS build of ‘qm-vamp-plugins’ was used.  
Annotator, bibtex Davies2009 Davies2007

echonest/version_3_2_1

Attribute Value
Corpus ballroom
Version 3.2.1
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools Echo Nest track analyzer v3.2.1
Annotator, bibtex Percival2014

gkiokas2012/default

Attribute Value
Corpus ballroom
Version 1.0
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools Gkiokas2012
Annotator, bibtex Gkiokas2012

klapuri2006/percival2014

Attribute Value
Corpus ballroom
Version 1.0
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools Klapuri 2006
Annotator, bibtex Klapuri2006

oliveira2010/ibt

Attribute Value
Corpus ballroom
Version 1.0
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools Oliveira 2010
Annotator, bibtex Oliveira2010

percival2014/stem

Attribute Value
Corpus ballroom
Version 1.0
Annotation Tools percival 2014, ‘tempo’ implementation from Marsyas, http://marsyas.info, git checkout tempo-stem
Annotator, bibtex Percival2014

scheirer1998/percival2014

Attribute Value
Corpus ballroom
Version 1.0
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools Scheirer 1998
Annotator, bibtex Scheirer1998

schreiber2014/default

Attribute Value
Corpus ballroom
Version 0.0.1
Annotation Tools schreiber 2014, http://www.tagtraum.com/tempo_estimation.html
Annotator, bibtex Schreiber2014

schreiber2017/ismir2017

Attribute Value
Corpus ballroom
Version 0.0.4
Annotation Tools schreiber 2017, model=ismir2017, http://www.tagtraum.com/tempo_estimation.html
Annotator, bibtex Schreiber2017

schreiber2017/mirex2017

Attribute Value
Corpus ballroom
Version 0.0.4
Annotation Tools schreiber 2017, model=mirex2017, http://www.tagtraum.com/tempo_estimation.html
Annotator, bibtex Schreiber2017

schreiber2018/cnn

Attribute Value
Corpus ballroom
Version 0.0.2
Data Source Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018.
Annotation Tools schreiber tempo-cnn (model=cnn), https://github.com/hendriks73/tempo-cnn

schreiber2018/fcn

Attribute Value
Corpus ballroom
Version 0.0.2
Data Source Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018.
Annotation Tools schreiber tempo-cnn (model=fcn), https://github.com/hendriks73/tempo-cnn

schreiber2018/ismir2018

Attribute Value
Corpus ballroom
Version 0.0.2
Data Source Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018.
Annotation Tools schreiber tempo-cnn (model=ismir2018), https://github.com/hendriks73/tempo-cnn

sun2021/default

Attribute Value
Corpus ballroom
Version 0.0.2
Data Source Xiaoheng Sun, Qiqi He, Yongwei Gao, Wei Li. Musical Tempo Estimation Using a Multi-scale Network. in Proc. of the 22nd Int. Society for Music Information Retrieval Conf., Online, 2021
Annotation Tools https://github.com/Qqi-HE/TempoEstimation_MGANet
Annotator, bibtex Sun2021

zplane/auftakt_v3

Attribute Value
Corpus ballroom
Version 3.0
Data Source Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014.
Annotation Tools zplane aufTAKT version 3.0, http://licensing.zplane.de/technology#auftakt
Annotator, bibtex Percival2014

Basic Statistics

Estimator Size Min Max Avg Stdev Sweet Oct. Start Sweet Oct. Coverage
boeck2015/tempodetector2016_default 698 57.69 214.29 117.51 31.52 74.00 0.83
boeck2019/multi_task 685 73.23 211.40 130.61 39.91 72.00 0.71
boeck2019/multi_task_hjdb 685 52.38 212.21 130.79 39.93 73.00 0.70
boeck2020/dar 685 73.20 214.25 129.90 39.69 72.00 0.71
davies2009/mirex_qm_tempotracker 698 60.09 206.72 120.29 28.62 84.00 0.87
echonest/version_3_2_1 697 43.20 207.60 104.37 31.47 67.00 0.75
gkiokas2012/default 698 40.00 207.00 98.23 22.77 67.00 0.86
klapuri2006/percival2014 698 59.75 169.44 106.18 16.64 76.00 0.99
oliveira2010/ibt 698 83.00 167.00 106.53 17.34 81.00 1.00
percival2014/stem 698 57.90 150.34 101.91 18.04 68.00 0.96
scheirer1998/percival2014 647 61.35 181.82 114.10 31.46 69.00 0.77
schreiber2014/default 698 55.83 142.73 101.77 17.17 69.00 0.96
schreiber2017/ismir2017 698 60.25 208.50 122.75 36.08 72.00 0.78
schreiber2017/mirex2017 698 42.27 211.97 126.02 38.12 72.00 0.74
schreiber2018/cnn 698 71.00 216.00 129.79 39.49 70.00 0.72
schreiber2018/fcn 698 62.00 232.00 128.65 39.00 74.00 0.73
schreiber2018/ismir2018 698 75.00 212.00 126.29 37.97 74.00 0.76
sun2021/default 698 60.00 215.00 129.33 39.54 73.00 0.72
zplane/auftakt_v3 698 65.90 166.50 105.01 17.50 74.00 0.98

Table 2: Basic statistics.

CSV JSON LATEX PICKLE

Smoothed Tempo Distribution

Figure 4: Percentage of values in tempo interval.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy

Accuracy1 is defined as the percentage of correct estimates, allowing a 4% tolerance for individual BPM values.

Accuracy2 additionally permits estimates to be wrong by a factor of 2, 3, 1/2 or 1/3 (so-called octave errors).

See [Gouyon2006].

Note: When comparing accuracy values for different algorithms, keep in mind that an algorithm may have been trained on the test set or that the test set may have even been created using one of the tested algorithms.

Accuracy Results for 1.0

Estimator Accuracy1 Accuracy2
sun2021/default 0.9441 0.9585
boeck2019/multi_task 0.9370 0.9499
boeck2020/dar 0.9370 0.9456
schreiber2018/cnn 0.9341 0.9570
boeck2019/multi_task_hjdb 0.9327 0.9470
schreiber2018/fcn 0.9155 0.9613
schreiber2018/ismir2018 0.9126 0.9599
schreiber2017/mirex2017 0.8825 0.9527
schreiber2017/ismir2017 0.8309 0.9484
boeck2015/tempodetector2016_default 0.8052 0.9542
davies2009/mirex_qm_tempotracker 0.6619 0.8968
schreiber2014/default 0.6433 0.9384
zplane/auftakt_v3 0.6418 0.9212
percival2014/stem 0.6275 0.9198
klapuri2006/percival2014 0.6232 0.9011
oliveira2010/ibt 0.6203 0.8782
gkiokas2012/default 0.6003 0.9441
echonest/version_3_2_1 0.5516 0.8410
scheirer1998/percival2014 0.5258 0.7364

Table 3: Mean accuracy of estimates compared to version 1.0 with 4% tolerance ordered by Accuracy1.

CSV JSON LATEX PICKLE

Raw data Accuracy1: CSV JSON LATEX PICKLE

Raw data Accuracy2: CSV JSON LATEX PICKLE

Accuracy1 for 1.0

Figure 5: Mean Accuracy1 for estimates compared to version 1.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for 1.0

Figure 6: Mean Accuracy2 for estimates compared to version 1.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy Results for 2.0

Estimator Accuracy1 Accuracy2
boeck2020/dar 0.9742 0.9814
sun2021/default 0.9742 0.9871
schreiber2018/cnn 0.9685 0.9928
boeck2019/multi_task 0.9670 0.9814
boeck2019/multi_task_hjdb 0.9656 0.9799
schreiber2018/ismir2018 0.9470 0.9943
schreiber2018/fcn 0.9456 0.9900
schreiber2017/mirex2017 0.9169 0.9885
schreiber2017/ismir2017 0.8682 0.9842
boeck2015/tempodetector2016_default 0.8453 1.0000
davies2009/mirex_qm_tempotracker 0.6762 0.9241
schreiber2014/default 0.6719 0.9728
zplane/auftakt_v3 0.6676 0.9513
percival2014/stem 0.6576 0.9556
klapuri2006/percival2014 0.6461 0.9284
oliveira2010/ibt 0.6447 0.9069
gkiokas2012/default 0.6304 0.9814
echonest/version_3_2_1 0.5745 0.8739
scheirer1998/percival2014 0.5387 0.7607

Table 4: Mean accuracy of estimates compared to version 2.0 with 4% tolerance ordered by Accuracy1.

CSV JSON LATEX PICKLE

Raw data Accuracy1: CSV JSON LATEX PICKLE

Raw data Accuracy2: CSV JSON LATEX PICKLE

Accuracy1 for 2.0

Figure 7: Mean Accuracy1 for estimates compared to version 2.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for 2.0

Figure 8: Mean Accuracy2 for estimates compared to version 2.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy Results for 2.0-no-dupes

Estimator Accuracy1 Accuracy2
boeck2020/dar 0.9927 1.0000
boeck2019/multi_task 0.9854 1.0000
boeck2019/multi_task_hjdb 0.9839 0.9985
sun2021/default 0.9752 0.9883
schreiber2018/cnn 0.9679 0.9927
schreiber2018/ismir2018 0.9460 0.9942
schreiber2018/fcn 0.9445 0.9898
schreiber2017/mirex2017 0.9153 0.9883
schreiber2017/ismir2017 0.8657 0.9839
boeck2015/tempodetector2016_default 0.8423 1.0000
davies2009/mirex_qm_tempotracker 0.6730 0.9226
schreiber2014/default 0.6672 0.9723
zplane/auftakt_v3 0.6642 0.9518
percival2014/stem 0.6540 0.9547
klapuri2006/percival2014 0.6423 0.9285
oliveira2010/ibt 0.6409 0.9051
gkiokas2012/default 0.6248 0.9810
echonest/version_3_2_1 0.5693 0.8730
scheirer1998/percival2014 0.5416 0.7635

Table 5: Mean accuracy of estimates compared to version 2.0-no-dupes with 4% tolerance ordered by Accuracy1.

CSV JSON LATEX PICKLE

Raw data Accuracy1: CSV JSON LATEX PICKLE

Raw data Accuracy2: CSV JSON LATEX PICKLE

Accuracy1 for 2.0-no-dupes

Figure 9: Mean Accuracy1 for estimates compared to version 2.0-no-dupes depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for 2.0-no-dupes

Figure 10: Mean Accuracy2 for estimates compared to version 2.0-no-dupes depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy Results for 3.0

Estimator Accuracy1 Accuracy2
sun2021/default 0.9771 0.9900
boeck2020/dar 0.9742 0.9814
schreiber2018/cnn 0.9699 0.9928
boeck2019/multi_task 0.9670 0.9814
boeck2019/multi_task_hjdb 0.9656 0.9799
schreiber2018/fcn 0.9470 0.9914
schreiber2018/ismir2018 0.9470 0.9943
schreiber2017/mirex2017 0.9169 0.9885
schreiber2017/ismir2017 0.8682 0.9842
boeck2015/tempodetector2016_default 0.8453 0.9986
davies2009/mirex_qm_tempotracker 0.6819 0.9284
schreiber2014/default 0.6719 0.9728
zplane/auftakt_v3 0.6691 0.9527
percival2014/stem 0.6547 0.9527
oliveira2010/ibt 0.6461 0.9097
klapuri2006/percival2014 0.6461 0.9284
gkiokas2012/default 0.6289 0.9799
echonest/version_3_2_1 0.5745 0.8739
scheirer1998/percival2014 0.5372 0.7579

Table 6: Mean accuracy of estimates compared to version 3.0 with 4% tolerance ordered by Accuracy1.

CSV JSON LATEX PICKLE

Raw data Accuracy1: CSV JSON LATEX PICKLE

Raw data Accuracy2: CSV JSON LATEX PICKLE

Accuracy1 for 3.0

Figure 11: Mean Accuracy1 for estimates compared to version 3.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for 3.0

Figure 12: Mean Accuracy2 for estimates compared to version 3.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy Results for 3.0-no-dupes

Estimator Accuracy1 Accuracy2
boeck2020/dar 0.9927 1.0000
boeck2019/multi_task 0.9854 1.0000
boeck2019/multi_task_hjdb 0.9839 0.9985
sun2021/default 0.9781 0.9912
schreiber2018/cnn 0.9693 0.9927
schreiber2018/fcn 0.9460 0.9912
schreiber2018/ismir2018 0.9460 0.9942
schreiber2017/mirex2017 0.9153 0.9883
schreiber2017/ismir2017 0.8657 0.9839
boeck2015/tempodetector2016_default 0.8423 0.9985
davies2009/mirex_qm_tempotracker 0.6788 0.9270
schreiber2014/default 0.6672 0.9723
zplane/auftakt_v3 0.6657 0.9533
percival2014/stem 0.6511 0.9518
oliveira2010/ibt 0.6423 0.9080
klapuri2006/percival2014 0.6423 0.9285
gkiokas2012/default 0.6234 0.9796
echonest/version_3_2_1 0.5693 0.8730
scheirer1998/percival2014 0.5401 0.7606

Table 7: Mean accuracy of estimates compared to version 3.0-no-dupes with 4% tolerance ordered by Accuracy1.

CSV JSON LATEX PICKLE

Raw data Accuracy1: CSV JSON LATEX PICKLE

Raw data Accuracy2: CSV JSON LATEX PICKLE

Accuracy1 for 3.0-no-dupes

Figure 13: Mean Accuracy1 for estimates compared to version 3.0-no-dupes depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for 3.0-no-dupes

Figure 14: Mean Accuracy2 for estimates compared to version 3.0-no-dupes depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy Results for 4.0

Estimator Accuracy1 Accuracy2
sun2021/default 0.9599 0.9814
boeck2020/dar 0.9585 0.9756
schreiber2018/cnn 0.9570 0.9871
boeck2019/multi_task 0.9542 0.9756
boeck2019/multi_task_hjdb 0.9513 0.9756
schreiber2018/fcn 0.9341 0.9871
schreiber2018/ismir2018 0.9327 0.9871
schreiber2017/mirex2017 0.9054 0.9842
schreiber2017/ismir2017 0.8539 0.9799
boeck2015/tempodetector2016_default 0.8324 0.9871
davies2009/mirex_qm_tempotracker 0.6762 0.9169
zplane/auftakt_v3 0.6691 0.9484
schreiber2014/default 0.6676 0.9670
percival2014/stem 0.6562 0.9499
klapuri2006/percival2014 0.6490 0.9284
oliveira2010/ibt 0.6433 0.9026
gkiokas2012/default 0.6318 0.9799
echonest/version_3_2_1 0.5702 0.8653
scheirer1998/percival2014 0.5372 0.7564

Table 8: Mean accuracy of estimates compared to version 4.0 with 4% tolerance ordered by Accuracy1.

CSV JSON LATEX PICKLE

Raw data Accuracy1: CSV JSON LATEX PICKLE

Raw data Accuracy2: CSV JSON LATEX PICKLE

Accuracy1 for 4.0

Figure 15: Mean Accuracy1 for estimates compared to version 4.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for 4.0

Figure 16: Mean Accuracy2 for estimates compared to version 4.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Differing Items

For which items did a given estimator not estimate a correct value with respect to a given ground truth? Are there items which are either very difficult, not suitable for the task, or incorrectly annotated and therefore never estimated correctly, regardless which estimator is used?

Differing Items Accuracy1

Items with different tempo annotations (Accuracy1, 4% tolerance) in different versions:

1.0 compared with boeck2015/tempodetector2016_default (136 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-13’ … CSV

1.0 compared with boeck2019/multi_task (44 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latin_Jam-13’ … CSV

1.0 compared with boeck2019/multi_task_hjdb (47 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latin_Jam-13’ … CSV

1.0 compared with boeck2020/dar (44 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Commitments-08’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ … CSV

1.0 compared with davies2009/mirex_qm_tempotracker (236 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ … CSV

1.0 compared with echonest/version_3_2_1 (313 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ … CSV

1.0 compared with gkiokas2012/default (279 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

1.0 compared with klapuri2006/percival2014 (263 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

1.0 compared with oliveira2010/ibt (265 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ … CSV

1.0 compared with percival2014/stem (260 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ … CSV

1.0 compared with scheirer1998/percival2014 (331 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ … CSV

1.0 compared with schreiber2014/default (249 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-09’ … CSV

1.0 compared with schreiber2017/ismir2017 (118 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ … CSV

1.0 compared with schreiber2017/mirex2017 (82 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-03’ … CSV

1.0 compared with schreiber2018/cnn (46 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ … CSV

1.0 compared with schreiber2018/fcn (59 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-11’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-Fire-13’ ‘Albums-GloriaEstefan_MiTierra-06’ … CSV

1.0 compared with schreiber2018/ismir2018 (61 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ ‘Albums-Fire-03’ … CSV

1.0 compared with sun2021/default (39 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Commitments-11’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-01’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ … CSV

1.0 compared with zplane/auftakt_v3 (250 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

2.0 compared with boeck2015/tempodetector2016_default (108 differences): ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ … CSV

2.0 compared with boeck2019/multi_task (23 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV

2.0 compared with boeck2019/multi_task_hjdb (24 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV

2.0 compared with boeck2020/dar (18 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Commitments-08’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV

2.0 compared with davies2009/mirex_qm_tempotracker (226 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-15’ … CSV

2.0 compared with echonest/version_3_2_1 (297 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

2.0 compared with gkiokas2012/default (258 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

2.0 compared with klapuri2006/percival2014 (247 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ … CSV

2.0 compared with oliveira2010/ibt (248 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

2.0 compared with percival2014/stem (239 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

2.0 compared with scheirer1998/percival2014 (322 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ … CSV

2.0 compared with schreiber2014/default (229 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-09’ ‘Albums-Ballroom_Magic-10’ … CSV

2.0 compared with schreiber2017/ismir2017 (92 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ … CSV

2.0 compared with schreiber2017/mirex2017 (58 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-07’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam-12’ … CSV

2.0 compared with schreiber2018/cnn (22 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Macumba-16’ ‘Media-103302’ ‘Media-103315’ ‘Media-103618’ ‘Media-103711’ ‘Media-103715’ ‘Media-103905’ … CSV

2.0 compared with schreiber2018/fcn (38 differences): ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam3-04’ ‘Albums-Latin_Jam3-05’ ‘Albums-Latin_Jam4-02’ ‘Albums-Macumba-16’ … CSV

2.0 compared with schreiber2018/ismir2018 (37 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ ‘Albums-Latin_Jam3-11’ ‘Albums-Latin_Jam3-12’ … CSV

2.0 compared with sun2021/default (18 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Secret_Garden-01’ ‘Media-100615’ ‘Media-103715’ ‘Media-103905’ ‘Media-104409’ ‘Media-104704’ ‘Media-104705’ ‘Media-105002’ ‘Media-105110’ … CSV

2.0 compared with zplane/auftakt_v3 (232 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

2.0-no-dupes compared with boeck2015/tempodetector2016_default (108 differences): ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ … CSV

2.0-no-dupes compared with boeck2019/multi_task (10 differences): ‘Albums-Chrisanne3-03’ ‘Albums-Mambo_Kings-10’ ‘Albums-Step_By_Step-16’ ‘Media-103606’ ‘Media-103614’ ‘Media-103715’ ‘Media-103905’ ‘Media-105207’ ‘Media-105403’ ‘Media-106009’ CSV

2.0-no-dupes compared with boeck2019/multi_task_hjdb (11 differences): ‘Albums-Chrisanne3-03’ ‘Albums-Mambo_Kings-10’ ‘Media-103606’ ‘Media-103715’ ‘Media-103905’ ‘Media-104418’ ‘Media-105207’ ‘Media-105302’ ‘Media-105403’ ‘Media-106009’ ‘Media-106118’ … CSV

2.0-no-dupes compared with boeck2020/dar (5 differences): ‘Albums-Commitments-08’ ‘Albums-Latin_Jam3-11’ ‘Media-103715’ ‘Media-104418’ ‘Media-105411’ CSV

2.0-no-dupes compared with davies2009/mirex_qm_tempotracker (224 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-15’ … CSV

2.0-no-dupes compared with echonest/version_3_2_1 (295 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

2.0-no-dupes compared with gkiokas2012/default (257 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

2.0-no-dupes compared with klapuri2006/percival2014 (245 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ … CSV

2.0-no-dupes compared with oliveira2010/ibt (246 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

2.0-no-dupes compared with percival2014/stem (237 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

2.0-no-dupes compared with scheirer1998/percival2014 (314 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ … CSV

2.0-no-dupes compared with schreiber2014/default (228 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-09’ ‘Albums-Ballroom_Magic-10’ … CSV

2.0-no-dupes compared with schreiber2017/ismir2017 (92 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ … CSV

2.0-no-dupes compared with schreiber2017/mirex2017 (58 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-07’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam-12’ … CSV

2.0-no-dupes compared with schreiber2018/cnn (22 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Macumba-16’ ‘Media-103302’ ‘Media-103315’ ‘Media-103618’ ‘Media-103711’ ‘Media-103715’ ‘Media-103905’ … CSV

2.0-no-dupes compared with schreiber2018/fcn (38 differences): ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam3-04’ ‘Albums-Latin_Jam3-05’ ‘Albums-Latin_Jam4-02’ ‘Albums-Macumba-16’ … CSV

2.0-no-dupes compared with schreiber2018/ismir2018 (37 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ ‘Albums-Latin_Jam3-11’ ‘Albums-Latin_Jam3-12’ … CSV

2.0-no-dupes compared with sun2021/default (17 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Secret_Garden-01’ ‘Media-100615’ ‘Media-103715’ ‘Media-103905’ ‘Media-104409’ ‘Media-104704’ ‘Media-105002’ ‘Media-105110’ ‘Media-105212’ … CSV

2.0-no-dupes compared with zplane/auftakt_v3 (230 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

3.0 compared with boeck2015/tempodetector2016_default (108 differences): ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ … CSV

3.0 compared with boeck2019/multi_task (23 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV

3.0 compared with boeck2019/multi_task_hjdb (24 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV

3.0 compared with boeck2020/dar (18 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Commitments-08’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV

3.0 compared with davies2009/mirex_qm_tempotracker (222 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-15’ … CSV

3.0 compared with echonest/version_3_2_1 (297 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

3.0 compared with gkiokas2012/default (259 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

3.0 compared with klapuri2006/percival2014 (247 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ … CSV

3.0 compared with oliveira2010/ibt (247 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

3.0 compared with percival2014/stem (241 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

3.0 compared with scheirer1998/percival2014 (323 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ … CSV

3.0 compared with schreiber2014/default (229 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-09’ ‘Albums-Ballroom_Magic-10’ … CSV

3.0 compared with schreiber2017/ismir2017 (92 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ … CSV

3.0 compared with schreiber2017/mirex2017 (58 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-07’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam-12’ … CSV

3.0 compared with schreiber2018/cnn (21 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Macumba-16’ ‘Media-103302’ ‘Media-103315’ ‘Media-103618’ ‘Media-103711’ ‘Media-103715’ ‘Media-103905’ … CSV

3.0 compared with schreiber2018/fcn (37 differences): ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam3-04’ ‘Albums-Latin_Jam3-05’ ‘Albums-Latin_Jam4-02’ ‘Albums-Macumba-16’ … CSV

3.0 compared with schreiber2018/ismir2018 (37 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ ‘Albums-Latin_Jam3-11’ ‘Albums-Latin_Jam3-12’ … CSV

3.0 compared with sun2021/default (16 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Secret_Garden-01’ ‘Media-100615’ ‘Media-103715’ ‘Media-103905’ ‘Media-104409’ ‘Media-104704’ ‘Media-104705’ ‘Media-105002’ ‘Media-105110’ … CSV

3.0 compared with zplane/auftakt_v3 (231 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

3.0-no-dupes compared with boeck2015/tempodetector2016_default (108 differences): ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ … CSV

3.0-no-dupes compared with boeck2019/multi_task (10 differences): ‘Albums-Chrisanne3-03’ ‘Albums-Mambo_Kings-10’ ‘Albums-Step_By_Step-16’ ‘Media-103606’ ‘Media-103614’ ‘Media-103715’ ‘Media-103905’ ‘Media-105207’ ‘Media-105403’ ‘Media-106009’ CSV

3.0-no-dupes compared with boeck2019/multi_task_hjdb (11 differences): ‘Albums-Chrisanne3-03’ ‘Albums-Mambo_Kings-10’ ‘Media-103606’ ‘Media-103715’ ‘Media-103905’ ‘Media-104418’ ‘Media-105207’ ‘Media-105302’ ‘Media-105403’ ‘Media-106009’ ‘Media-106118’ … CSV

3.0-no-dupes compared with boeck2020/dar (5 differences): ‘Albums-Commitments-08’ ‘Albums-Latin_Jam3-11’ ‘Media-103715’ ‘Media-104418’ ‘Media-105411’ CSV

3.0-no-dupes compared with davies2009/mirex_qm_tempotracker (220 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-15’ … CSV

3.0-no-dupes compared with echonest/version_3_2_1 (295 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

3.0-no-dupes compared with gkiokas2012/default (258 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

3.0-no-dupes compared with klapuri2006/percival2014 (245 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ … CSV

3.0-no-dupes compared with oliveira2010/ibt (245 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

3.0-no-dupes compared with percival2014/stem (239 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

3.0-no-dupes compared with scheirer1998/percival2014 (315 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ … CSV

3.0-no-dupes compared with schreiber2014/default (228 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-09’ ‘Albums-Ballroom_Magic-10’ … CSV

3.0-no-dupes compared with schreiber2017/ismir2017 (92 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ … CSV

3.0-no-dupes compared with schreiber2017/mirex2017 (58 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-07’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam-12’ … CSV

3.0-no-dupes compared with schreiber2018/cnn (21 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Macumba-16’ ‘Media-103302’ ‘Media-103315’ ‘Media-103618’ ‘Media-103711’ ‘Media-103715’ ‘Media-103905’ … CSV

3.0-no-dupes compared with schreiber2018/fcn (37 differences): ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam3-04’ ‘Albums-Latin_Jam3-05’ ‘Albums-Latin_Jam4-02’ ‘Albums-Macumba-16’ … CSV

3.0-no-dupes compared with schreiber2018/ismir2018 (37 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ ‘Albums-Latin_Jam3-11’ ‘Albums-Latin_Jam3-12’ … CSV

3.0-no-dupes compared with sun2021/default (15 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Secret_Garden-01’ ‘Media-100615’ ‘Media-103715’ ‘Media-103905’ ‘Media-104409’ ‘Media-104704’ ‘Media-105002’ ‘Media-105110’ ‘Media-105212’ … CSV

3.0-no-dupes compared with zplane/auftakt_v3 (229 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

4.0 compared with boeck2015/tempodetector2016_default (117 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ … CSV

4.0 compared with boeck2019/multi_task (32 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ … CSV

4.0 compared with boeck2019/multi_task_hjdb (34 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ … CSV

4.0 compared with boeck2020/dar (29 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Commitments-08’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ … CSV

4.0 compared with davies2009/mirex_qm_tempotracker (226 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ … CSV

4.0 compared with echonest/version_3_2_1 (300 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ … CSV

4.0 compared with gkiokas2012/default (257 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV

4.0 compared with klapuri2006/percival2014 (245 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

4.0 compared with oliveira2010/ibt (249 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ … CSV

4.0 compared with percival2014/stem (240 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ … CSV

4.0 compared with scheirer1998/percival2014 (323 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ … CSV

4.0 compared with schreiber2014/default (232 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ … CSV

4.0 compared with schreiber2017/ismir2017 (102 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ … CSV

4.0 compared with schreiber2017/mirex2017 (66 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-07’ … CSV

4.0 compared with schreiber2018/cnn (30 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Latino_Latino-03’ ‘Albums-Macumba-16’ ‘Albums-Secret_Garden-05’ ‘Albums-Step_By_Step-15’ ‘Albums-Step_By_Step-16’ … CSV

4.0 compared with schreiber2018/fcn (46 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-11’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam3-04’ ‘Albums-Latin_Jam3-05’ ‘Albums-Latin_Jam4-02’ … CSV

4.0 compared with schreiber2018/ismir2018 (47 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ … CSV

4.0 compared with sun2021/default (28 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Commitments-11’ ‘Albums-GloriaEstefan_MiTierra-01’ ‘Albums-Latino_Latino-03’ ‘Albums-Latino_Latino-06’ ‘Albums-Secret_Garden-01’ ‘Albums-Step_By_Step-15’ ‘Albums-Step_By_Step-16’ … CSV

4.0 compared with zplane/auftakt_v3 (231 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV

All tracks were estimated ‘correctly’ by at least one system.

Differing Items Accuracy2

Items with different tempo annotations (Accuracy2, 4% tolerance) in different versions:

1.0 compared with boeck2015/tempodetector2016_default (32 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-05’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Secret_Garden-01’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-05’ … CSV

1.0 compared with boeck2019/multi_task (35 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latin_Jam-13’ … CSV

1.0 compared with boeck2019/multi_task_hjdb (37 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latin_Jam-13’ … CSV

1.0 compared with boeck2020/dar (38 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latin_Jam-13’ … CSV

1.0 compared with davies2009/mirex_qm_tempotracker (72 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ … CSV

1.0 compared with echonest/version_3_2_1 (111 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Ballroom_Magic-18’ … CSV

1.0 compared with gkiokas2012/default (39 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-15’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ … CSV

1.0 compared with klapuri2006/percival2014 (69 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ … CSV

1.0 compared with oliveira2010/ibt (85 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ … CSV

1.0 compared with percival2014/stem (56 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ … CSV

1.0 compared with scheirer1998/percival2014 (184 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ … CSV

1.0 compared with schreiber2014/default (43 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-Fire-13’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ … CSV

1.0 compared with schreiber2017/ismir2017 (36 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-Fire-13’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ … CSV

1.0 compared with schreiber2017/mirex2017 (33 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-Fire-13’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Secret_Garden-01’ … CSV

1.0 compared with schreiber2018/cnn (30 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Secret_Garden-01’ ‘Albums-Secret_Garden-02’ … CSV

1.0 compared with schreiber2018/fcn (27 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-11’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-Fire-13’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Secret_Garden-01’ … CSV

1.0 compared with schreiber2018/ismir2018 (28 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Secret_Garden-01’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-05’ … CSV

1.0 compared with sun2021/default (29 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-11’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-01’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latino_Latino-06’ … CSV

1.0 compared with zplane/auftakt_v3 (55 differences): ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ … CSV

2.0 compared with boeck2015/tempodetector2016_default: No differences.

2.0 compared with boeck2019/multi_task (13 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV

2.0 compared with boeck2019/multi_task_hjdb (14 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV

2.0 compared with boeck2020/dar (13 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV

2.0 compared with davies2009/mirex_qm_tempotracker (53 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Latino_Latino-03’ … CSV

2.0 compared with echonest/version_3_2_1 (88 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-03’ … CSV

2.0 compared with gkiokas2012/default (13 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne3-15’ ‘Albums-Latino_Latino-03’ ‘Albums-Secret_Garden-02’ ‘Media-103710’ ‘Media-103905’ ‘Media-104711’ ‘Media-105002’ ‘Media-105007’ ‘Media-105111’ … CSV

2.0 compared with klapuri2006/percival2014 (50 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ … CSV

2.0 compared with oliveira2010/ibt (65 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ … CSV

2.0 compared with percival2014/stem (31 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100604’ ‘Media-100609’ … CSV

2.0 compared with scheirer1998/percival2014 (167 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ … CSV

2.0 compared with schreiber2014/default (19 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103308’ ‘Media-103315’ ‘Media-104302’ ‘Media-104608’ ‘Media-104703’ ‘Media-104811’ … CSV

2.0 compared with schreiber2017/ismir2017 (11 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-104303’ ‘Media-104608’ ‘Media-105004’ ‘Media-105420’ ‘Media-105701’ ‘Media-105702’ … CSV

2.0 compared with schreiber2017/mirex2017 (8 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103315’ ‘Media-104303’ ‘Media-104608’ ‘Media-105214’ ‘Media-105420’ CSV

2.0 compared with schreiber2018/cnn (5 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103315’ ‘Media-104905’ ‘Media-105302’ CSV

2.0 compared with schreiber2018/fcn (7 differences): ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103905’ ‘Media-105002’ ‘Media-105302’ CSV

2.0 compared with schreiber2018/ismir2018 (4 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103905’ ‘Media-105911’ CSV

2.0 compared with sun2021/default (9 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Secret_Garden-01’ ‘Media-103905’ ‘Media-104704’ ‘Media-104705’ ‘Media-105002’ ‘Media-105212’ ‘Media-105215’ ‘Media-105302’ CSV

2.0 compared with zplane/auftakt_v3 (34 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100603’ … CSV

2.0-no-dupes compared with boeck2015/tempodetector2016_default: No differences.

2.0-no-dupes compared with boeck2019/multi_task: No differences.

2.0-no-dupes compared with boeck2019/multi_task_hjdb (1 differences): ‘Media-105302’ CSV

2.0-no-dupes compared with boeck2020/dar: No differences.

2.0-no-dupes compared with davies2009/mirex_qm_tempotracker (53 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Latino_Latino-03’ … CSV

2.0-no-dupes compared with echonest/version_3_2_1 (87 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne1-07’ … CSV

2.0-no-dupes compared with gkiokas2012/default (13 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne3-15’ ‘Albums-Latino_Latino-03’ ‘Albums-Secret_Garden-02’ ‘Media-103710’ ‘Media-103905’ ‘Media-104711’ ‘Media-105002’ ‘Media-105007’ ‘Media-105111’ … CSV

2.0-no-dupes compared with klapuri2006/percival2014 (49 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ … CSV

2.0-no-dupes compared with oliveira2010/ibt (65 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ … CSV

2.0-no-dupes compared with percival2014/stem (31 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100604’ ‘Media-100609’ … CSV

2.0-no-dupes compared with scheirer1998/percival2014 (162 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-10’ … CSV

2.0-no-dupes compared with schreiber2014/default (19 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103308’ ‘Media-103315’ ‘Media-104302’ ‘Media-104608’ ‘Media-104703’ ‘Media-104811’ … CSV

2.0-no-dupes compared with schreiber2017/ismir2017 (11 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-104303’ ‘Media-104608’ ‘Media-105004’ ‘Media-105420’ ‘Media-105701’ ‘Media-105702’ … CSV

2.0-no-dupes compared with schreiber2017/mirex2017 (8 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103315’ ‘Media-104303’ ‘Media-104608’ ‘Media-105214’ ‘Media-105420’ CSV

2.0-no-dupes compared with schreiber2018/cnn (5 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103315’ ‘Media-104905’ ‘Media-105302’ CSV

2.0-no-dupes compared with schreiber2018/fcn (7 differences): ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103905’ ‘Media-105002’ ‘Media-105302’ CSV

2.0-no-dupes compared with schreiber2018/ismir2018 (4 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103905’ ‘Media-105911’ CSV

2.0-no-dupes compared with sun2021/default (8 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Secret_Garden-01’ ‘Media-103905’ ‘Media-104704’ ‘Media-105002’ ‘Media-105212’ ‘Media-105215’ ‘Media-105302’ CSV

2.0-no-dupes compared with zplane/auftakt_v3 (33 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100603’ … CSV

3.0 compared with boeck2015/tempodetector2016_default (1 differences): ‘Media-103905’ CSV

3.0 compared with boeck2019/multi_task (13 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV

3.0 compared with boeck2019/multi_task_hjdb (14 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV

3.0 compared with boeck2020/dar (13 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV

3.0 compared with davies2009/mirex_qm_tempotracker (50 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Latino_Latino-03’ … CSV

3.0 compared with echonest/version_3_2_1 (88 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-03’ … CSV

3.0 compared with gkiokas2012/default (14 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne3-15’ ‘Albums-Latino_Latino-03’ ‘Albums-Secret_Garden-02’ ‘Media-103710’ ‘Media-103905’ ‘Media-104711’ ‘Media-105002’ ‘Media-105007’ … CSV

3.0 compared with klapuri2006/percival2014 (50 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ … CSV

3.0 compared with oliveira2010/ibt (63 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ … CSV

3.0 compared with percival2014/stem (33 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100604’ ‘Media-100609’ … CSV

3.0 compared with scheirer1998/percival2014 (169 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ … CSV

3.0 compared with schreiber2014/default (19 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103308’ ‘Media-103315’ ‘Media-104302’ ‘Media-104608’ ‘Media-104703’ ‘Media-104811’ … CSV

3.0 compared with schreiber2017/ismir2017 (11 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-104303’ ‘Media-104608’ ‘Media-105004’ ‘Media-105420’ ‘Media-105701’ ‘Media-105702’ … CSV

3.0 compared with schreiber2017/mirex2017 (8 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103315’ ‘Media-104303’ ‘Media-104608’ ‘Media-105214’ ‘Media-105420’ CSV

3.0 compared with schreiber2018/cnn (5 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103315’ ‘Media-103905’ ‘Media-104905’ CSV

3.0 compared with schreiber2018/fcn (6 differences): ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103905’ ‘Media-105002’ CSV

3.0 compared with schreiber2018/ismir2018 (4 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103905’ ‘Media-105911’ CSV

3.0 compared with sun2021/default (7 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Secret_Garden-01’ ‘Media-103905’ ‘Media-104704’ ‘Media-104705’ ‘Media-105002’ ‘Media-105212’ CSV

3.0 compared with zplane/auftakt_v3 (33 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100603’ … CSV

3.0-no-dupes compared with boeck2015/tempodetector2016_default (1 differences): ‘Media-103905’ CSV

3.0-no-dupes compared with boeck2019/multi_task: No differences.

3.0-no-dupes compared with boeck2019/multi_task_hjdb (1 differences): ‘Media-105302’ CSV

3.0-no-dupes compared with boeck2020/dar: No differences.

3.0-no-dupes compared with davies2009/mirex_qm_tempotracker (50 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Latino_Latino-03’ … CSV

3.0-no-dupes compared with echonest/version_3_2_1 (87 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne1-07’ … CSV

3.0-no-dupes compared with gkiokas2012/default (14 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne3-15’ ‘Albums-Latino_Latino-03’ ‘Albums-Secret_Garden-02’ ‘Media-103710’ ‘Media-103905’ ‘Media-104711’ ‘Media-105002’ ‘Media-105007’ … CSV

3.0-no-dupes compared with klapuri2006/percival2014 (49 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ … CSV

3.0-no-dupes compared with oliveira2010/ibt (63 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ … CSV

3.0-no-dupes compared with percival2014/stem (33 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100604’ ‘Media-100609’ … CSV

3.0-no-dupes compared with scheirer1998/percival2014 (164 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-10’ … CSV

3.0-no-dupes compared with schreiber2014/default (19 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103308’ ‘Media-103315’ ‘Media-104302’ ‘Media-104608’ ‘Media-104703’ ‘Media-104811’ … CSV

3.0-no-dupes compared with schreiber2017/ismir2017 (11 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-104303’ ‘Media-104608’ ‘Media-105004’ ‘Media-105420’ ‘Media-105701’ ‘Media-105702’ … CSV

3.0-no-dupes compared with schreiber2017/mirex2017 (8 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103315’ ‘Media-104303’ ‘Media-104608’ ‘Media-105214’ ‘Media-105420’ CSV

3.0-no-dupes compared with schreiber2018/cnn (5 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103315’ ‘Media-103905’ ‘Media-104905’ CSV

3.0-no-dupes compared with schreiber2018/fcn (6 differences): ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103905’ ‘Media-105002’ CSV

3.0-no-dupes compared with schreiber2018/ismir2018 (4 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103905’ ‘Media-105911’ CSV

3.0-no-dupes compared with sun2021/default (6 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Secret_Garden-01’ ‘Media-103905’ ‘Media-104704’ ‘Media-105002’ ‘Media-105212’ CSV

3.0-no-dupes compared with zplane/auftakt_v3 (32 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100603’ … CSV

4.0 compared with boeck2015/tempodetector2016_default (9 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-05’ ‘Albums-Secret_Garden-05’ ‘Albums-StrictlyDancing_Tango-08’ ‘Media-103905’ ‘Media-104404’ ‘Media-104908’ ‘Media-105007’ ‘Media-105101’ CSV

4.0 compared with boeck2019/multi_task (17 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV

4.0 compared with boeck2019/multi_task_hjdb (17 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV

4.0 compared with boeck2020/dar (17 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV

4.0 compared with davies2009/mirex_qm_tempotracker (58 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ … CSV

4.0 compared with echonest/version_3_2_1 (94 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne1-01’ … CSV

4.0 compared with gkiokas2012/default (14 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne3-15’ ‘Albums-Latino_Latino-03’ ‘Albums-Secret_Garden-05’ ‘Media-103710’ ‘Media-103905’ ‘Media-103911’ ‘Media-104601’ ‘Media-104711’ … CSV

4.0 compared with klapuri2006/percival2014 (50 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ … CSV

4.0 compared with oliveira2010/ibt (68 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ … CSV

4.0 compared with percival2014/stem (35 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-05’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ … CSV

4.0 compared with scheirer1998/percival2014 (170 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ … CSV

4.0 compared with schreiber2014/default (23 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Secret_Garden-05’ ‘Media-103308’ ‘Media-103315’ ‘Media-104302’ ‘Media-104608’ … CSV

4.0 compared with schreiber2017/ismir2017 (14 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Secret_Garden-05’ ‘Media-104303’ ‘Media-104608’ ‘Media-105004’ ‘Media-105007’ … CSV

4.0 compared with schreiber2017/mirex2017 (11 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Secret_Garden-05’ ‘Media-103315’ ‘Media-104303’ ‘Media-104608’ ‘Media-105007’ ‘Media-105214’ ‘Media-105420’ … CSV

4.0 compared with schreiber2018/cnn (9 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-05’ ‘Media-103315’ ‘Media-103709’ ‘Media-103905’ ‘Media-104905’ ‘Media-105007’ CSV

4.0 compared with schreiber2018/fcn (9 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-11’ ‘Albums-Fire-13’ ‘Albums-Secret_Garden-05’ ‘Media-103905’ ‘Media-103911’ ‘Media-105007’ ‘Media-105211’ CSV

4.0 compared with schreiber2018/ismir2018 (9 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-05’ ‘Albums-StrictlyDancing_Tango-11’ ‘Media-103905’ ‘Media-103911’ ‘Media-105007’ ‘Media-105911’ CSV

4.0 compared with sun2021/default (13 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-11’ ‘Albums-GloriaEstefan_MiTierra-01’ ‘Albums-Latino_Latino-06’ ‘Albums-Secret_Garden-01’ ‘Media-103709’ ‘Media-103905’ ‘Media-103911’ ‘Media-104901’ ‘Media-105007’ … CSV

4.0 compared with zplane/auftakt_v3 (36 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-05’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ … CSV

All tracks were estimated ‘correctly’ by at least one system.

Significance of Differences

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1176 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.3750 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.8776 0.0534 0.0300 0.5224 0.0000
boeck2019/multi_task_hjdb 0.0000 0.3750 1.0000 0.6291 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000 0.1263 0.0869 0.2682 0.0000
boeck2020/dar 0.0000 1.0000 0.6291 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.8746 0.0534 0.0300 0.4996 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0009 0.0120 0.0045 0.0437 0.0000 0.3223 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2314
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0144 0.0001 0.0002 0.0000 0.2664 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0009 0.0144 1.0000 0.1058 0.1800 0.0295 0.0012 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0021
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0120 0.0001 0.1058 1.0000 0.8776 0.7838 0.0000 0.0925 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0596
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0045 0.0002 0.1800 0.8776 1.0000 0.6085 0.0000 0.0894 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0489
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0437 0.0000 0.0295 0.7838 0.6085 1.0000 0.0000 0.1925 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2203
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.2664 0.0012 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.3223 0.0000 0.0005 0.0925 0.0894 0.1925 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000
schreiber2017/ismir2017 0.1176 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0095 0.0170 0.0000 0.0000
schreiber2018/cnn 0.0000 0.8776 1.0000 0.8746 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0660 0.0237 0.2810 0.0000
schreiber2018/fcn 0.0000 0.0534 0.1263 0.0534 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0095 0.0660 1.0000 0.8776 0.0055 0.0000
schreiber2018/ismir2018 0.0000 0.0300 0.0869 0.0300 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0170 0.0237 0.8776 1.0000 0.0026 0.0000
sun2021/default 0.0000 0.5224 0.2682 0.4996 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2810 0.0055 0.0026 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.2314 0.0000 0.0021 0.0596 0.0489 0.2203 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 9: McNemar p-values, using reference annotations 1.0 as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1637 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 1.0000 0.2668 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.8746 0.0759 0.0704 0.2962 0.0000
boeck2019/multi_task_hjdb 0.0000 1.0000 1.0000 0.1460 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.7493 0.0919 0.0984 0.2153 0.0000
boeck2020/dar 0.0000 0.2668 0.1460 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7428 0.0110 0.0110 0.8506 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0059 0.0215 0.0180 0.1186 0.0000 0.6320 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4812
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0067 0.0001 0.0001 0.0000 0.1037 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0059 0.0067 1.0000 0.2410 0.2566 0.0356 0.0001 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0029
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0215 0.0001 0.2410 1.0000 1.0000 0.5044 0.0000 0.0385 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0226
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0180 0.0001 0.2566 1.0000 1.0000 0.5115 0.0000 0.0512 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0365
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.1186 0.0000 0.0356 0.5044 0.5115 1.0000 0.0000 0.1619 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2203
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.1037 0.0001 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.6320 0.0000 0.0005 0.0385 0.0512 0.1619 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.8955
schreiber2017/ismir2017 0.1637 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0001 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0170 0.0154 0.0000 0.0000
schreiber2018/cnn 0.0000 0.8746 0.7493 0.7428 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0195 0.0090 0.4421 0.0000
schreiber2018/fcn 0.0000 0.0759 0.0919 0.0110 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0170 0.0195 1.0000 1.0000 0.0025 0.0000
schreiber2018/ismir2018 0.0000 0.0704 0.0984 0.0110 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0154 0.0090 1.0000 1.0000 0.0019 0.0000
sun2021/default 0.0000 0.2962 0.2153 0.8506 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4421 0.0025 0.0019 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.4812 0.0000 0.0029 0.0226 0.0365 0.2203 0.0000 0.8955 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 10: McNemar p-values, using reference annotations 3.0 as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1637 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 1.0000 0.2668 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0522 0.0000 0.0000 0.3833 0.0000
boeck2019/multi_task_hjdb 0.0000 1.0000 1.0000 0.1460 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0755 0.0000 0.0000 0.5034 0.0000
boeck2020/dar 0.0000 0.2668 0.1460 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0015 0.0000 0.0000 0.0213 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0045 0.0215 0.0180 0.1158 0.0000 0.5752 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4812
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0081 0.0001 0.0001 0.0000 0.2083 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0045 0.0081 1.0000 0.1983 0.2134 0.0248 0.0003 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0019
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0215 0.0001 0.1983 1.0000 1.0000 0.4966 0.0000 0.0498 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0226
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0180 0.0001 0.2134 1.0000 1.0000 0.5044 0.0000 0.0647 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0365
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.1158 0.0000 0.0248 0.4966 0.5044 1.0000 0.0000 0.2000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2116
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.2083 0.0003 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.5752 0.0000 0.0005 0.0498 0.0647 0.2000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000
schreiber2017/ismir2017 0.1637 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0170 0.0154 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0522 0.0755 0.0015 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0195 0.0090 0.3269 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0170 0.0195 1.0000 1.0000 0.0013 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0154 0.0090 1.0000 1.0000 0.0009 0.0000
sun2021/default 0.0000 0.3833 0.5034 0.0213 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.3269 0.0013 0.0009 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.4812 0.0000 0.0019 0.0226 0.0365 0.2116 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 11: McNemar p-values, using reference annotations 3.0-no-dupes as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1637 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 1.0000 0.2668 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000 0.0581 0.0704 0.4996 0.0000
boeck2019/multi_task_hjdb 0.0000 1.0000 1.0000 0.1460 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.8714 0.0649 0.0984 0.3771 0.0000
boeck2020/dar 0.0000 0.2668 0.1460 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6271 0.0078 0.0110 1.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0174 0.0554 0.0357 0.2981 0.0000 0.8732 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6610
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0055 0.0001 0.0001 0.0000 0.1190 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0174 0.0055 1.0000 0.2891 0.3481 0.0248 0.0001 0.0008 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0058
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0554 0.0001 0.2891 1.0000 1.0000 0.3409 0.0000 0.0385 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0357
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0357 0.0001 0.3481 1.0000 1.0000 0.2976 0.0000 0.0402 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0365
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.2981 0.0000 0.0248 0.3409 0.2976 1.0000 0.0000 0.2451 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4188
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.1190 0.0001 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.8732 0.0000 0.0008 0.0385 0.0402 0.2451 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7946
schreiber2017/ismir2017 0.1637 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0001 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0245 0.0154 0.0000 0.0000
schreiber2018/cnn 0.0000 1.0000 0.8714 0.6271 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0195 0.0167 0.5716 0.0000
schreiber2018/fcn 0.0000 0.0581 0.0649 0.0078 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0245 0.0195 1.0000 1.0000 0.0045 0.0000
schreiber2018/ismir2018 0.0000 0.0704 0.0984 0.0110 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0154 0.0167 1.0000 1.0000 0.0066 0.0000
sun2021/default 0.0000 0.4996 0.3771 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5716 0.0045 0.0066 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.6610 0.0000 0.0058 0.0357 0.0365 0.4188 0.0000 0.7946 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 12: McNemar p-values, using reference annotations 2.0 as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1637 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 1.0000 0.2668 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0357 0.0000 0.0000 0.2100 0.0000
boeck2019/multi_task_hjdb 0.0000 1.0000 1.0000 0.1460 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0433 0.0000 0.0000 0.2632 0.0000
boeck2020/dar 0.0000 0.2668 0.1460 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0009 0.0000 0.0000 0.0075 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0138 0.0554 0.0357 0.2944 0.0000 0.8102 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6610
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0067 0.0001 0.0001 0.0000 0.2343 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0138 0.0067 1.0000 0.2410 0.2945 0.0169 0.0003 0.0008 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0039
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0554 0.0001 0.2410 1.0000 1.0000 0.3317 0.0000 0.0498 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0357
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0357 0.0001 0.2945 1.0000 1.0000 0.2892 0.0000 0.0512 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0365
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.2944 0.0000 0.0169 0.3317 0.2892 1.0000 0.0000 0.2976 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4101
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.2343 0.0003 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.8102 0.0000 0.0008 0.0498 0.0512 0.2976 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.8955
schreiber2017/ismir2017 0.1637 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0245 0.0154 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0357 0.0433 0.0009 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0195 0.0167 0.4421 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0245 0.0195 1.0000 1.0000 0.0025 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0154 0.0167 1.0000 1.0000 0.0037 0.0000
sun2021/default 0.0000 0.2100 0.2632 0.0075 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4421 0.0025 0.0037 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.6610 0.0000 0.0039 0.0357 0.0365 0.4101 0.0000 0.8955 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 13: McNemar p-values, using reference annotations 2.0-no-dupes as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2031 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.6250 0.5811 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.8776 0.0759 0.0581 0.6358 0.0000
boeck2019/multi_task_hjdb 0.0000 0.6250 1.0000 0.2266 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 0.6358 0.1189 0.1048 0.4177 0.0000
boeck2020/dar 0.0000 0.5811 0.2266 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0213 0.0175 1.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0195 0.0818 0.0286 0.2578 0.0000 0.6851 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7183
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0020 0.0000 0.0001 0.0000 0.1478 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0195 0.0020 1.0000 0.2410 0.4750 0.0498 0.0000 0.0035 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0064
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0818 0.0000 0.2410 1.0000 0.6358 0.5901 0.0000 0.1299 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0436
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0286 0.0001 0.4750 0.6358 1.0000 0.2976 0.0000 0.0647 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0175
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.2578 0.0000 0.0498 0.5901 0.2976 1.0000 0.0000 0.3581 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2624
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.1478 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.6851 0.0000 0.0035 0.1299 0.0647 0.3581 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000
schreiber2017/ismir2017 0.2031 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0001 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0245 0.0319 0.0000 0.0000
schreiber2018/cnn 0.0000 0.8776 0.6358 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0195 0.0076 0.8555 0.0000
schreiber2018/fcn 0.0000 0.0759 0.1189 0.0213 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0245 0.0195 1.0000 1.0000 0.0114 0.0000
schreiber2018/ismir2018 0.0000 0.0581 0.1048 0.0175 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0319 0.0076 1.0000 1.0000 0.0079 0.0000
sun2021/default 0.0000 0.6358 0.4177 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.8555 0.0114 0.0079 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.7183 0.0000 0.0064 0.0436 0.0175 0.2624 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 14: McNemar p-values, using reference annotations 4.0 as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.6776 0.4049 0.2632 0.0000 0.0000 0.1671 0.0000 0.0000 0.0001 0.0000 0.0433 0.4807 1.0000 0.8036 0.2668 0.4240 0.6636 0.0002
boeck2019/multi_task 0.6776 1.0000 0.5000 0.4531 0.0000 0.0000 0.5847 0.0000 0.0000 0.0031 0.0000 0.2295 1.0000 0.8388 0.4049 0.1153 0.2100 0.3269 0.0037
boeck2019/multi_task_hjdb 0.4049 0.5000 1.0000 1.0000 0.0001 0.0000 0.8555 0.0000 0.0000 0.0079 0.0000 0.3915 1.0000 0.5413 0.2100 0.0525 0.1078 0.1849 0.0096
boeck2020/dar 0.2632 0.4531 1.0000 1.0000 0.0001 0.0000 1.0000 0.0000 0.0000 0.0133 0.0000 0.5114 0.8506 0.4049 0.1338 0.0266 0.0525 0.1078 0.0137
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0001 0.0001 1.0000 0.0000 0.0000 0.8041 0.1237 0.0365 0.0000 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0363
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0034 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.1671 0.5847 0.8555 1.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0115 0.0000 0.6177 0.6900 0.2863 0.0931 0.0075 0.0127 0.0639 0.0166
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.8041 0.0000 0.0000 1.0000 0.0402 0.0854 0.0000 0.0003 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0436
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.1237 0.0034 0.0000 0.0402 1.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002
percival2014/stem 0.0001 0.0031 0.0079 0.0133 0.0365 0.0000 0.0115 0.0854 0.0002 1.0000 0.0000 0.0533 0.0017 0.0003 0.0000 0.0000 0.0000 0.0000 1.0000
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0433 0.2295 0.3915 0.5114 0.0004 0.0000 0.6177 0.0003 0.0000 0.0533 0.0000 1.0000 0.0923 0.0309 0.0106 0.0015 0.0059 0.0201 0.0961
schreiber2017/ismir2017 0.4807 1.0000 1.0000 0.8506 0.0000 0.0000 0.6900 0.0000 0.0000 0.0017 0.0000 0.0923 1.0000 0.4531 0.2379 0.0352 0.0768 0.2295 0.0034
schreiber2017/mirex2017 1.0000 0.8388 0.5413 0.4049 0.0000 0.0000 0.2863 0.0000 0.0000 0.0003 0.0000 0.0309 0.4531 1.0000 0.5488 0.1460 0.2668 0.5034 0.0005
schreiber2018/cnn 0.8036 0.4049 0.2100 0.1338 0.0000 0.0000 0.0931 0.0000 0.0000 0.0000 0.0000 0.0106 0.2379 0.5488 1.0000 0.5078 0.7539 1.0000 0.0000
schreiber2018/fcn 0.2668 0.1153 0.0525 0.0266 0.0000 0.0000 0.0075 0.0000 0.0000 0.0000 0.0000 0.0015 0.0352 0.1460 0.5078 1.0000 1.0000 0.7744 0.0000
schreiber2018/ismir2018 0.4240 0.2100 0.1078 0.0525 0.0000 0.0000 0.0127 0.0000 0.0000 0.0000 0.0000 0.0059 0.0768 0.2668 0.7539 1.0000 1.0000 1.0000 0.0000
sun2021/default 0.6636 0.3269 0.1849 0.1078 0.0000 0.0000 0.0639 0.0000 0.0000 0.0000 0.0000 0.0201 0.2295 0.5034 1.0000 0.7744 1.0000 1.0000 0.0001
zplane/auftakt_v3 0.0002 0.0037 0.0096 0.0137 0.0363 0.0000 0.0166 0.0436 0.0002 1.0000 0.0000 0.0961 0.0034 0.0005 0.0000 0.0000 0.0000 0.0001 1.0000

Table 15: McNemar p-values, using reference annotations 1.0 as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0018 0.0010 0.0018 0.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0063 0.0391 0.1250 0.0625 0.2500 0.0312 0.0000
boeck2019/multi_task 0.0018 1.0000 1.0000 1.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0045 0.0000 0.3771 0.8388 0.3833 0.0963 0.1671 0.0490 0.2379 0.0037
boeck2019/multi_task_hjdb 0.0010 1.0000 1.0000 1.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0066 0.0000 0.4869 0.6900 0.2863 0.0636 0.1153 0.0309 0.1671 0.0054
boeck2020/dar 0.0018 1.0000 1.0000 1.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0045 0.0000 0.3771 0.8388 0.3833 0.0963 0.1671 0.0490 0.2379 0.0037
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 1.0000 0.0984 0.0046 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0186
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0046 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0002 1.0000 1.0000 1.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0019 0.0000 0.4869 0.6900 0.2863 0.0490 0.0768 0.0213 0.1185 0.0019
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 1.0000 0.0919 0.0213 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0115
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0984 0.0046 0.0000 0.0919 1.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002
percival2014/stem 0.0000 0.0045 0.0066 0.0045 0.0046 0.0000 0.0019 0.0213 0.0001 1.0000 0.0000 0.0385 0.0003 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.3771 0.4869 0.3771 0.0000 0.0000 0.4869 0.0000 0.0000 0.0385 0.0000 1.0000 0.0215 0.0074 0.0013 0.0072 0.0007 0.0227 0.0385
schreiber2017/ismir2017 0.0063 0.8388 0.6900 0.8388 0.0000 0.0000 0.6900 0.0000 0.0000 0.0003 0.0000 0.0215 1.0000 0.4531 0.1460 0.2668 0.0654 0.4545 0.0005
schreiber2017/mirex2017 0.0391 0.3833 0.2863 0.3833 0.0000 0.0000 0.2863 0.0000 0.0000 0.0000 0.0000 0.0074 0.4531 1.0000 0.4531 0.7539 0.2891 1.0000 0.0001
schreiber2018/cnn 0.1250 0.0963 0.0636 0.0963 0.0000 0.0000 0.0490 0.0000 0.0000 0.0000 0.0000 0.0013 0.1460 0.4531 1.0000 1.0000 1.0000 0.7266 0.0000
schreiber2018/fcn 0.0625 0.1671 0.1153 0.1671 0.0000 0.0000 0.0768 0.0000 0.0000 0.0000 0.0000 0.0072 0.2668 0.7539 1.0000 1.0000 0.6875 1.0000 0.0000
schreiber2018/ismir2018 0.2500 0.0490 0.0309 0.0490 0.0000 0.0000 0.0213 0.0000 0.0000 0.0000 0.0000 0.0007 0.0654 0.2891 1.0000 0.6875 1.0000 0.4531 0.0000
sun2021/default 0.0312 0.2379 0.1671 0.2379 0.0000 0.0000 0.1185 0.0000 0.0000 0.0000 0.0000 0.0227 0.4545 1.0000 0.7266 1.0000 0.4531 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0037 0.0054 0.0037 0.0186 0.0000 0.0019 0.0115 0.0002 1.0000 0.0000 0.0385 0.0005 0.0001 0.0000 0.0000 0.0000 0.0000 1.0000

Table 16: McNemar p-values, using reference annotations 3.0 as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0063 0.0391 0.1250 0.0625 0.2500 0.0625 0.0000
boeck2019/multi_task 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0010 0.0078 0.0625 0.0312 0.1250 0.0312 0.0000
boeck2019/multi_task_hjdb 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0010 0.0000 0.0000 0.0000 0.0000 0.0000 0.0063 0.0391 0.2188 0.1250 0.3750 0.1250 0.0000
boeck2020/dar 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0010 0.0078 0.0625 0.0312 0.1250 0.0312 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 1.0000 0.0984 0.0046 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0114
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0063 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0002 0.0001 0.0010 0.0001 0.0000 0.0000 1.0000 0.0000 0.0000 0.0019 0.0000 0.4869 0.6900 0.2863 0.0490 0.0768 0.0213 0.0574 0.0029
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 1.0000 0.0649 0.0293 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0115
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0984 0.0063 0.0000 0.0649 1.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0046 0.0000 0.0019 0.0293 0.0001 1.0000 0.0000 0.0385 0.0003 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4869 0.0001 0.0000 0.0385 0.0000 1.0000 0.0215 0.0074 0.0013 0.0072 0.0007 0.0106 0.0533
schreiber2017/ismir2017 0.0063 0.0010 0.0063 0.0010 0.0000 0.0000 0.6900 0.0000 0.0000 0.0003 0.0000 0.0215 1.0000 0.4531 0.1460 0.2668 0.0654 0.3018 0.0008
schreiber2017/mirex2017 0.0391 0.0078 0.0391 0.0078 0.0000 0.0000 0.2863 0.0000 0.0000 0.0000 0.0000 0.0074 0.4531 1.0000 0.4531 0.7539 0.2891 0.7744 0.0001
schreiber2018/cnn 0.1250 0.0625 0.2188 0.0625 0.0000 0.0000 0.0490 0.0000 0.0000 0.0000 0.0000 0.0013 0.1460 0.4531 1.0000 1.0000 1.0000 1.0000 0.0000
schreiber2018/fcn 0.0625 0.0312 0.1250 0.0312 0.0000 0.0000 0.0768 0.0000 0.0000 0.0000 0.0000 0.0072 0.2668 0.7539 1.0000 1.0000 0.6875 1.0000 0.0000
schreiber2018/ismir2018 0.2500 0.1250 0.3750 0.1250 0.0000 0.0000 0.0213 0.0000 0.0000 0.0000 0.0000 0.0007 0.0654 0.2891 1.0000 0.6875 1.0000 0.6875 0.0000
sun2021/default 0.0625 0.0312 0.1250 0.0312 0.0000 0.0000 0.0574 0.0000 0.0000 0.0000 0.0000 0.0106 0.3018 0.7744 1.0000 1.0000 0.6875 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0114 0.0000 0.0029 0.0115 0.0001 1.0000 0.0000 0.0533 0.0008 0.0001 0.0000 0.0000 0.0000 0.0000 1.0000

Table 17: McNemar p-values, using reference annotations 3.0-no-dupes as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0002 0.0001 0.0002 0.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0010 0.0078 0.0625 0.0156 0.1250 0.0039 0.0000
boeck2019/multi_task 0.0002 1.0000 1.0000 1.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0096 0.0000 0.3771 0.8388 0.3833 0.0963 0.2632 0.0490 0.5034 0.0025
boeck2019/multi_task_hjdb 0.0001 1.0000 1.0000 1.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0137 0.0000 0.4869 0.6900 0.2863 0.0490 0.1671 0.0309 0.3593 0.0037
boeck2020/dar 0.0002 1.0000 1.0000 1.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0096 0.0000 0.3771 0.8388 0.3833 0.0963 0.2632 0.0490 0.5034 0.0025
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0001 0.0000 0.7877 0.1263 0.0003 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0110
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000 0.0000 0.0000 0.0086 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0002 1.0000 1.0000 1.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0029 0.0000 0.3771 0.8388 0.3833 0.0963 0.2101 0.0352 0.4545 0.0005
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.7877 0.0000 0.0000 1.0000 0.0534 0.0079 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0195
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.1263 0.0086 0.0000 0.0534 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001
percival2014/stem 0.0000 0.0096 0.0137 0.0096 0.0003 0.0000 0.0029 0.0079 0.0000 1.0000 0.0000 0.0730 0.0008 0.0001 0.0000 0.0000 0.0000 0.0001 0.7428
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.3771 0.4869 0.3771 0.0000 0.0000 0.3771 0.0000 0.0000 0.0730 0.0000 1.0000 0.0215 0.0074 0.0013 0.0169 0.0007 0.0755 0.0275
schreiber2017/ismir2017 0.0010 0.8388 0.6900 0.8388 0.0000 0.0000 0.8388 0.0000 0.0000 0.0008 0.0000 0.0215 1.0000 0.4531 0.1460 0.4240 0.0654 0.8145 0.0003
schreiber2017/mirex2017 0.0078 0.3833 0.2863 0.3833 0.0000 0.0000 0.3833 0.0000 0.0000 0.0001 0.0000 0.0074 0.4531 1.0000 0.4531 1.0000 0.2891 1.0000 0.0000
schreiber2018/cnn 0.0625 0.0963 0.0490 0.0963 0.0000 0.0000 0.0963 0.0000 0.0000 0.0000 0.0000 0.0013 0.1460 0.4531 1.0000 0.7266 1.0000 0.3437 0.0000
schreiber2018/fcn 0.0156 0.2632 0.1671 0.2632 0.0000 0.0000 0.2101 0.0000 0.0000 0.0000 0.0000 0.0169 0.4240 1.0000 0.7266 1.0000 0.4531 0.7539 0.0000
schreiber2018/ismir2018 0.1250 0.0490 0.0309 0.0490 0.0000 0.0000 0.0352 0.0000 0.0000 0.0000 0.0000 0.0007 0.0654 0.2891 1.0000 0.4531 1.0000 0.1797 0.0000
sun2021/default 0.0039 0.5034 0.3593 0.5034 0.0000 0.0000 0.4545 0.0000 0.0000 0.0001 0.0000 0.0755 0.8145 1.0000 0.3437 0.7539 0.1797 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0025 0.0037 0.0025 0.0110 0.0000 0.0005 0.0195 0.0001 0.7428 0.0000 0.0275 0.0003 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 18: McNemar p-values, using reference annotations 2.0 as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0010 0.0078 0.0625 0.0156 0.1250 0.0078 0.0000
boeck2019/multi_task 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0010 0.0078 0.0625 0.0156 0.1250 0.0078 0.0000
boeck2019/multi_task_hjdb 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0063 0.0391 0.1250 0.0312 0.3750 0.0156 0.0000
boeck2020/dar 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0010 0.0078 0.0625 0.0156 0.1250 0.0078 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0001 0.0000 0.6835 0.1263 0.0003 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0066
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000 0.0000 0.0000 0.0115 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0002 0.0002 0.0018 0.0002 0.0000 0.0000 1.0000 0.0000 0.0000 0.0029 0.0000 0.3771 0.8388 0.3833 0.0963 0.2101 0.0352 0.3018 0.0008
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.6835 0.0000 0.0000 1.0000 0.0365 0.0114 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0195
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.1263 0.0115 0.0000 0.0365 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0003 0.0000 0.0029 0.0114 0.0000 1.0000 0.0000 0.0730 0.0008 0.0001 0.0000 0.0000 0.0000 0.0000 0.8679
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.3771 0.0001 0.0000 0.0730 0.0000 1.0000 0.0215 0.0074 0.0013 0.0169 0.0007 0.0433 0.0385
schreiber2017/ismir2017 0.0010 0.0010 0.0063 0.0010 0.0000 0.0000 0.8388 0.0000 0.0000 0.0008 0.0000 0.0215 1.0000 0.4531 0.1460 0.4240 0.0654 0.6291 0.0005
schreiber2017/mirex2017 0.0078 0.0078 0.0391 0.0078 0.0000 0.0000 0.3833 0.0000 0.0000 0.0001 0.0000 0.0074 0.4531 1.0000 0.4531 1.0000 0.2891 1.0000 0.0001
schreiber2018/cnn 0.0625 0.0625 0.1250 0.0625 0.0000 0.0000 0.0963 0.0000 0.0000 0.0000 0.0000 0.0013 0.1460 0.4531 1.0000 0.7266 1.0000 0.5078 0.0000
schreiber2018/fcn 0.0156 0.0156 0.0312 0.0156 0.0000 0.0000 0.2101 0.0000 0.0000 0.0000 0.0000 0.0169 0.4240 1.0000 0.7266 1.0000 0.4531 1.0000 0.0000
schreiber2018/ismir2018 0.1250 0.1250 0.3750 0.1250 0.0000 0.0000 0.0352 0.0000 0.0000 0.0000 0.0000 0.0007 0.0654 0.2891 1.0000 0.4531 1.0000 0.2891 0.0000
sun2021/default 0.0078 0.0078 0.0156 0.0078 0.0000 0.0000 0.3018 0.0000 0.0000 0.0000 0.0000 0.0433 0.6291 1.0000 0.5078 1.0000 0.2891 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0066 0.0000 0.0008 0.0195 0.0001 0.8679 0.0000 0.0385 0.0005 0.0001 0.0000 0.0000 0.0000 0.0000 1.0000

Table 19: McNemar p-values, using reference annotations 2.0-no-dupes as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.1153 0.1153 0.0963 0.0000 0.0000 0.3018 0.0000 0.0000 0.0000 0.0000 0.0066 0.3323 0.7905 1.0000 1.0000 1.0000 0.4545 0.0000
boeck2019/multi_task 0.1153 1.0000 1.0000 1.0000 0.0000 0.0000 0.6900 0.0000 0.0000 0.0096 0.0000 0.3915 0.6900 0.2863 0.1153 0.1153 0.1153 0.5572 0.0054
boeck2019/multi_task_hjdb 0.1153 1.0000 1.0000 1.0000 0.0000 0.0000 0.6900 0.0000 0.0000 0.0096 0.0000 0.3915 0.6900 0.2863 0.1153 0.1153 0.1153 0.5572 0.0054
boeck2020/dar 0.0963 1.0000 1.0000 1.0000 0.0000 0.0000 0.6776 0.0000 0.0000 0.0096 0.0000 0.3915 0.6900 0.2863 0.0963 0.0963 0.0963 0.5413 0.0054
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0001 0.0000 0.3581 0.2370 0.0011 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0038
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000 0.0000 0.0000 0.0034 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.3018 0.6900 0.6900 0.6776 0.0000 0.0000 1.0000 0.0000 0.0000 0.0008 0.0000 0.1496 1.0000 0.6476 0.3018 0.2266 0.2668 1.0000 0.0005
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.3581 0.0000 0.0000 1.0000 0.0175 0.0489 0.0000 0.0003 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0436
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.2370 0.0034 0.0000 0.0175 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001
percival2014/stem 0.0000 0.0096 0.0096 0.0096 0.0011 0.0000 0.0008 0.0489 0.0000 1.0000 0.0000 0.0730 0.0005 0.0001 0.0000 0.0000 0.0000 0.0003 1.0000
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0066 0.3915 0.3915 0.3915 0.0000 0.0000 0.1496 0.0003 0.0000 0.0730 0.0000 1.0000 0.0117 0.0042 0.0013 0.0043 0.0043 0.0987 0.0596
schreiber2017/ismir2017 0.3323 0.6900 0.6900 0.6900 0.0000 0.0000 1.0000 0.0000 0.0000 0.0005 0.0000 0.0117 1.0000 0.4531 0.2668 0.2668 0.2668 1.0000 0.0005
schreiber2017/mirex2017 0.7905 0.2863 0.2863 0.2863 0.0000 0.0000 0.6476 0.0000 0.0000 0.0001 0.0000 0.0042 0.4531 1.0000 0.7266 0.7539 0.7539 0.8145 0.0001
schreiber2018/cnn 1.0000 0.1153 0.1153 0.0963 0.0000 0.0000 0.3018 0.0000 0.0000 0.0000 0.0000 0.0013 0.2668 0.7266 1.0000 1.0000 1.0000 0.3877 0.0000
schreiber2018/fcn 1.0000 0.1153 0.1153 0.0963 0.0000 0.0000 0.2266 0.0000 0.0000 0.0000 0.0000 0.0043 0.2668 0.7539 1.0000 1.0000 1.0000 0.3437 0.0000
schreiber2018/ismir2018 1.0000 0.1153 0.1153 0.0963 0.0000 0.0000 0.2668 0.0000 0.0000 0.0000 0.0000 0.0043 0.2668 0.7539 1.0000 1.0000 1.0000 0.3877 0.0000
sun2021/default 0.4545 0.5572 0.5572 0.5413 0.0000 0.0000 1.0000 0.0000 0.0000 0.0003 0.0000 0.0987 1.0000 0.8145 0.3877 0.3437 0.3877 1.0000 0.0004
zplane/auftakt_v3 0.0000 0.0054 0.0054 0.0054 0.0038 0.0000 0.0005 0.0436 0.0001 1.0000 0.0000 0.0596 0.0005 0.0001 0.0000 0.0000 0.0000 0.0004 1.0000

Table 20: McNemar p-values, using reference annotations 4.0 as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Accuracy1 on cvar-Subsets

How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?

Accuracy1 on cvar-Subsets for 1.0 based on cvar-Values from 1.0

Figure 17: Mean Accuracy1 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on cvar-Subsets for 2.0 based on cvar-Values from 1.0

Figure 18: Mean Accuracy1 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0

Figure 19: Mean Accuracy1 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on cvar-Subsets for 3.0 based on cvar-Values from 1.0

Figure 20: Mean Accuracy1 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0

Figure 21: Mean Accuracy1 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on cvar-Subsets for 4.0 based on cvar-Values from 1.0

Figure 22: Mean Accuracy1 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on cvar-Subsets

How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?

Accuracy2 on cvar-Subsets for 1.0 based on cvar-Values from 1.0

Figure 23: Mean Accuracy2 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on cvar-Subsets for 2.0 based on cvar-Values from 1.0

Figure 24: Mean Accuracy2 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0

Figure 25: Mean Accuracy2 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on cvar-Subsets for 3.0 based on cvar-Values from 1.0

Figure 26: Mean Accuracy2 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0

Figure 27: Mean Accuracy2 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on cvar-Subsets for 4.0 based on cvar-Values from 1.0

Figure 28: Mean Accuracy2 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean Accuracy1 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

Accuracy1 on Tempo-Subsets for 1.0

Figure 29: Mean Accuracy1 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on Tempo-Subsets for 2.0

Figure 30: Mean Accuracy1 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on Tempo-Subsets for 2.0-no-dupes

Figure 31: Mean Accuracy1 for estimates compared to version 2.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on Tempo-Subsets for 3.0

Figure 32: Mean Accuracy1 for estimates compared to version 3.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on Tempo-Subsets for 3.0-no-dupes

Figure 33: Mean Accuracy1 for estimates compared to version 3.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 on Tempo-Subsets for 4.0

Figure 34: Mean Accuracy1 for estimates compared to version 4.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean Accuracy2 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

Accuracy2 on Tempo-Subsets for 1.0

Figure 35: Mean Accuracy2 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on Tempo-Subsets for 2.0

Figure 36: Mean Accuracy2 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on Tempo-Subsets for 2.0-no-dupes

Figure 37: Mean Accuracy2 for estimates compared to version 2.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on Tempo-Subsets for 3.0

Figure 38: Mean Accuracy2 for estimates compared to version 3.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on Tempo-Subsets for 3.0-no-dupes

Figure 39: Mean Accuracy2 for estimates compared to version 3.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 on Tempo-Subsets for 4.0

Figure 40: Mean Accuracy2 for estimates compared to version 4.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy1 for Tempo

When fitting a generalized additive model (GAM) to Accuracy1-values and a ground truth, what Accuracy1 can we expect with confidence?

Estimated Accuracy1 for Tempo for 1.0

Predictions of GAMs trained on Accuracy1 for estimates for reference 1.0.

Figure 41: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy1 for Tempo for 2.0

Predictions of GAMs trained on Accuracy1 for estimates for reference 2.0.

Figure 42: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy1 for Tempo for 2.0-no-dupes

Predictions of GAMs trained on Accuracy1 for estimates for reference 2.0-no-dupes.

Figure 43: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy1 for Tempo for 3.0

Predictions of GAMs trained on Accuracy1 for estimates for reference 3.0.

Figure 44: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy1 for Tempo for 3.0-no-dupes

Predictions of GAMs trained on Accuracy1 for estimates for reference 3.0-no-dupes.

Figure 45: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy1 for Tempo for 4.0

Predictions of GAMs trained on Accuracy1 for estimates for reference 4.0.

Figure 46: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy2 for Tempo

When fitting a generalized additive model (GAM) to Accuracy2-values and a ground truth, what Accuracy2 can we expect with confidence?

Estimated Accuracy2 for Tempo for 1.0

Predictions of GAMs trained on Accuracy2 for estimates for reference 1.0.

Figure 47: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy2 for Tempo for 2.0

Predictions of GAMs trained on Accuracy2 for estimates for reference 2.0.

Figure 48: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy2 for Tempo for 2.0-no-dupes

Predictions of GAMs trained on Accuracy2 for estimates for reference 2.0-no-dupes.

Figure 49: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy2 for Tempo for 3.0

Predictions of GAMs trained on Accuracy2 for estimates for reference 3.0.

Figure 50: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy2 for Tempo for 3.0-no-dupes

Predictions of GAMs trained on Accuracy2 for estimates for reference 3.0-no-dupes.

Figure 51: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy2 for Tempo for 4.0

Predictions of GAMs trained on Accuracy2 for estimates for reference 4.0.

Figure 52: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

Accuracy1 for ‘tag_open’ Tags for 1.0

Figure 53: Mean Accuracy1 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 for ‘tag_open’ Tags for 2.0

Figure 54: Mean Accuracy1 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 for ‘tag_open’ Tags for 2.0-no-dupes

Figure 55: Mean Accuracy1 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 for ‘tag_open’ Tags for 3.0

Figure 56: Mean Accuracy1 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 for ‘tag_open’ Tags for 3.0-no-dupes

Figure 57: Mean Accuracy1 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy1 for ‘tag_open’ Tags for 4.0

Figure 58: Mean Accuracy1 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

Accuracy2 for ‘tag_open’ Tags for 1.0

Figure 59: Mean Accuracy2 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for ‘tag_open’ Tags for 2.0

Figure 60: Mean Accuracy2 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for ‘tag_open’ Tags for 2.0-no-dupes

Figure 61: Mean Accuracy2 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for ‘tag_open’ Tags for 3.0

Figure 62: Mean Accuracy2 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for ‘tag_open’ Tags for 3.0-no-dupes

Figure 63: Mean Accuracy2 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy2 for ‘tag_open’ Tags for 4.0

Figure 64: Mean Accuracy2 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 and OE2

OE1 is defined as octave error between an estimate E and a reference value R.This means that the most common errors—by a factor of 2 or ½—have the same magnitude, namely 1: OE2(E) = log2(E/R).

OE2 is the signed OE1 corresponding to the minimum absolute OE1 allowing the octaveerrors 2, 3, 1/2, and 1/3: OE2(E) = arg minx(|x|) with x ∈ {OE1(E), OE1(2E), OE1(3E), OE1(½E), OE1(⅓E)}

Mean OE1/OE2 Results for 1.0

Estimator OE1_MEAN OE1_STDEV OE2_MEAN OE2_STDEV
boeck2020/dar -0.0059 0.1163 -0.0029 0.0491
boeck2019/multi_task 0.0018 0.1407 -0.0026 0.0485
boeck2019/multi_task_hjdb 0.0037 0.1474 -0.0022 0.0497
sun2021/default -0.0096 0.1479 -0.0044 0.0532
schreiber2018/cnn -0.0037 0.1774 0.0006 0.0574
schreiber2018/ismir2018 -0.0403 0.2284 -0.0016 0.0579
schreiber2018/fcn -0.0156 0.2311 0.0001 0.0568
schreiber2017/mirex2017 -0.0456 0.2765 -0.0027 0.0668
schreiber2017/ismir2017 -0.0779 0.3494 0.0003 0.0716
boeck2015/tempodetector2016_default -0.1312 0.3865 -0.0032 0.0503
schreiber2014/default -0.3127 0.4610 0.0025 0.0840
zplane/auftakt_v3 -0.2660 0.4749 0.0042 0.0841
oliveira2010/ibt -0.2440 0.4841 0.0067 0.0931
klapuri2006/percival2014 -0.2478 0.4892 0.0152 0.1012
percival2014/stem -0.3133 0.4924 0.0001 0.0767
davies2009/mirex_qm_tempotracker -0.0879 0.5170 0.0241 0.0900
scheirer1998/percival2014 -0.1809 0.5199 0.0246 0.1546
echonest/version_3_2_1 -0.3190 0.5379 -0.0129 0.1318
gkiokas2012/default -0.3835 0.5661 -0.0014 0.0664

Table 21: Mean OE1/OE2 for estimates compared to version 1.0 ordered by standard deviation.

CSV JSON LATEX PICKLE

Raw data OE1: CSV JSON LATEX PICKLE

Raw data OE2: CSV JSON LATEX PICKLE

OE1 distribution for 1.0

Figure 65: OE1 for estimates compared to version 1.0. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 distribution for 1.0

Figure 66: OE2 for estimates compared to version 1.0. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean OE1/OE2 Results for 2.0

Estimator OE1_MEAN OE1_STDEV OE2_MEAN OE2_STDEV
boeck2020/dar -0.0011 0.0857 0.0004 0.0082
boeck2019/multi_task 0.0066 0.1206 0.0007 0.0084
boeck2019/multi_task_hjdb 0.0084 0.1229 0.0012 0.0151
sun2021/default -0.0050 0.1258 -0.0013 0.0223
schreiber2018/cnn 0.0008 0.1613 0.0037 0.0307
schreiber2018/fcn -0.0111 0.2170 0.0032 0.0268
schreiber2018/ismir2018 -0.0358 0.2173 0.0029 0.0315
schreiber2017/mirex2017 -0.0411 0.2706 0.0004 0.0450
schreiber2017/ismir2017 -0.0734 0.3414 0.0034 0.0527
boeck2015/tempodetector2016_default -0.1267 0.3846 -0.0002 0.0101
schreiber2014/default -0.3082 0.4613 0.0056 0.0686
zplane/auftakt_v3 -0.2615 0.4735 0.0050 0.0716
oliveira2010/ibt -0.2395 0.4860 0.0084 0.0855
klapuri2006/percival2014 -0.2433 0.4893 0.0154 0.0919
percival2014/stem -0.3088 0.4920 0.0032 0.0613
davies2009/mirex_qm_tempotracker -0.0834 0.5198 0.0257 0.0794
scheirer1998/percival2014 -0.1751 0.5206 0.0264 0.1524
echonest/version_3_2_1 -0.3145 0.5408 -0.0126 0.1225
gkiokas2012/default -0.3789 0.5666 0.0002 0.0439

Table 22: Mean OE1/OE2 for estimates compared to version 2.0 ordered by standard deviation.

CSV JSON LATEX PICKLE

Raw data OE1: CSV JSON LATEX PICKLE

Raw data OE2: CSV JSON LATEX PICKLE

OE1 distribution for 2.0

Figure 67: OE1 for estimates compared to version 2.0. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 distribution for 2.0

Figure 68: OE2 for estimates compared to version 2.0. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean OE1/OE2 Results for 2.0-no-dupes

Estimator OE1_MEAN OE1_STDEV OE2_MEAN OE2_STDEV
boeck2020/dar -0.0011 0.0857 0.0004 0.0082
boeck2019/multi_task 0.0066 0.1206 0.0007 0.0084
boeck2019/multi_task_hjdb 0.0084 0.1229 0.0012 0.0151
sun2021/default -0.0052 0.1269 -0.0014 0.0221
schreiber2018/cnn 0.0008 0.1628 0.0037 0.0310
schreiber2018/fcn -0.0114 0.2191 0.0032 0.0270
schreiber2018/ismir2018 -0.0365 0.2193 0.0029 0.0318
schreiber2017/mirex2017 -0.0419 0.2731 0.0005 0.0454
schreiber2017/ismir2017 -0.0747 0.3445 0.0035 0.0532
boeck2015/tempodetector2016_default -0.1291 0.3878 -0.0002 0.0101
schreiber2014/default -0.3125 0.4630 0.0057 0.0692
zplane/auftakt_v3 -0.2659 0.4746 0.0056 0.0711
oliveira2010/ibt -0.2438 0.4866 0.0088 0.0862
klapuri2006/percival2014 -0.2475 0.4906 0.0161 0.0923
percival2014/stem -0.3117 0.4936 0.0033 0.0619
scheirer1998/percival2014 -0.1745 0.5197 0.0254 0.1520
davies2009/mirex_qm_tempotracker -0.0854 0.5217 0.0258 0.0802
echonest/version_3_2_1 -0.3181 0.5437 -0.0135 0.1223
gkiokas2012/default -0.3847 0.5691 0.0002 0.0443

Table 23: Mean OE1/OE2 for estimates compared to version 2.0-no-dupes ordered by standard deviation.

CSV JSON LATEX PICKLE

Raw data OE1: CSV JSON LATEX PICKLE

Raw data OE2: CSV JSON LATEX PICKLE

OE1 distribution for 2.0-no-dupes

Figure 69: OE1 for estimates compared to version 2.0-no-dupes. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 distribution for 2.0-no-dupes

Figure 70: OE2 for estimates compared to version 2.0-no-dupes. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean OE1/OE2 Results for 3.0

Estimator OE1_MEAN OE1_STDEV OE2_MEAN OE2_STDEV
boeck2020/dar -0.0007 0.0855 0.0008 0.0057
boeck2019/multi_task 0.0070 0.1209 0.0011 0.0050
boeck2019/multi_task_hjdb 0.0089 0.1231 0.0016 0.0130
sun2021/default -0.0046 0.1255 -0.0009 0.0211
schreiber2018/cnn 0.0013 0.1616 0.0041 0.0299
schreiber2018/fcn -0.0107 0.2174 0.0037 0.0261
schreiber2018/ismir2018 -0.0353 0.2179 0.0019 0.0317
schreiber2017/mirex2017 -0.0407 0.2708 0.0009 0.0446
schreiber2017/ismir2017 -0.0729 0.3415 0.0038 0.0523
boeck2015/tempodetector2016_default -0.1263 0.3854 0.0003 0.0098
schreiber2014/default -0.3077 0.4619 0.0060 0.0682
zplane/auftakt_v3 -0.2610 0.4741 0.0055 0.0706
oliveira2010/ibt -0.2390 0.4865 0.0088 0.0851
klapuri2006/percival2014 -0.2428 0.4899 0.0159 0.0916
percival2014/stem -0.3084 0.4925 0.0036 0.0607
davies2009/mirex_qm_tempotracker -0.0830 0.5202 0.0262 0.0791
scheirer1998/percival2014 -0.1745 0.5208 0.0269 0.1521
echonest/version_3_2_1 -0.3140 0.5407 -0.0108 0.1230
gkiokas2012/default -0.3785 0.5673 0.0007 0.0430

Table 24: Mean OE1/OE2 for estimates compared to version 3.0 ordered by standard deviation.

CSV JSON LATEX PICKLE

Raw data OE1: CSV JSON LATEX PICKLE

Raw data OE2: CSV JSON LATEX PICKLE

OE1 distribution for 3.0

Figure 71: OE1 for estimates compared to version 3.0. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 distribution for 3.0

Figure 72: OE2 for estimates compared to version 3.0. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean OE1/OE2 Results for 3.0-no-dupes

Estimator OE1_MEAN OE1_STDEV OE2_MEAN OE2_STDEV
boeck2020/dar -0.0007 0.0855 0.0008 0.0057
boeck2019/multi_task 0.0070 0.1209 0.0011 0.0050
boeck2019/multi_task_hjdb 0.0089 0.1231 0.0016 0.0130
sun2021/default -0.0047 0.1267 -0.0010 0.0209
schreiber2018/cnn 0.0012 0.1631 0.0041 0.0302
schreiber2018/fcn -0.0109 0.2194 0.0037 0.0263
schreiber2018/ismir2018 -0.0361 0.2199 0.0019 0.0319
schreiber2017/mirex2017 -0.0415 0.2733 0.0009 0.0450
schreiber2017/ismir2017 -0.0743 0.3445 0.0039 0.0528
boeck2015/tempodetector2016_default -0.1287 0.3886 0.0003 0.0098
schreiber2014/default -0.3121 0.4637 0.0061 0.0688
zplane/auftakt_v3 -0.2655 0.4753 0.0061 0.0701
oliveira2010/ibt -0.2434 0.4872 0.0092 0.0858
klapuri2006/percival2014 -0.2471 0.4912 0.0166 0.0919
percival2014/stem -0.3113 0.4942 0.0037 0.0613
scheirer1998/percival2014 -0.1739 0.5199 0.0260 0.1517
davies2009/mirex_qm_tempotracker -0.0850 0.5221 0.0262 0.0799
echonest/version_3_2_1 -0.3177 0.5437 -0.0116 0.1228
gkiokas2012/default -0.3843 0.5699 0.0006 0.0434

Table 25: Mean OE1/OE2 for estimates compared to version 3.0-no-dupes ordered by standard deviation.

CSV JSON LATEX PICKLE

Raw data OE1: CSV JSON LATEX PICKLE

Raw data OE2: CSV JSON LATEX PICKLE

OE1 distribution for 3.0-no-dupes

Figure 73: OE1 for estimates compared to version 3.0-no-dupes. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 distribution for 3.0-no-dupes

Figure 74: OE2 for estimates compared to version 3.0-no-dupes. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean OE1/OE2 Results for 4.0

Estimator OE1_MEAN OE1_STDEV OE2_MEAN OE2_STDEV
boeck2020/dar -0.0005 0.1370 -0.0034 0.0277
boeck2019/multi_task 0.0072 0.1519 -0.0031 0.0275
boeck2019/multi_task_hjdb 0.0090 0.1632 -0.0026 0.0294
sun2021/default -0.0046 0.1646 -0.0052 0.0328
schreiber2018/cnn 0.0012 0.1855 -0.0002 0.0402
schreiber2018/ismir2018 -0.0354 0.2365 -0.0024 0.0405
schreiber2018/fcn -0.0107 0.2425 -0.0007 0.0373
schreiber2017/mirex2017 -0.0407 0.2863 -0.0035 0.0521
schreiber2017/ismir2017 -0.0730 0.3578 -0.0005 0.0587
boeck2015/tempodetector2016_default -0.1263 0.3889 -0.0040 0.0288
schreiber2014/default -0.3078 0.4631 0.0017 0.0729
zplane/auftakt_v3 -0.2610 0.4768 0.0020 0.0688
oliveira2010/ibt -0.2391 0.4851 0.0045 0.0834
klapuri2006/percival2014 -0.2429 0.4902 0.0130 0.0898
percival2014/stem -0.3084 0.4931 0.0022 0.0654
davies2009/mirex_qm_tempotracker -0.0830 0.5159 0.0247 0.0786
scheirer1998/percival2014 -0.1746 0.5214 0.0231 0.1514
echonest/version_3_2_1 -0.3141 0.5408 -0.0094 0.1257
gkiokas2012/default -0.3785 0.5694 -0.0022 0.0436

Table 26: Mean OE1/OE2 for estimates compared to version 4.0 ordered by standard deviation.

CSV JSON LATEX PICKLE

Raw data OE1: CSV JSON LATEX PICKLE

Raw data OE2: CSV JSON LATEX PICKLE

OE1 distribution for 4.0

Figure 75: OE1 for estimates compared to version 4.0. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 distribution for 4.0

Figure 76: OE2 for estimates compared to version 4.0. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Significance of Differences

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0080 0.0000 0.0000 0.0000 0.0000 0.0000 0.0125 0.0000 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.5376 0.1454 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4201 0.0429 0.0000 0.0666 0.0000
boeck2019/multi_task_hjdb 0.0000 0.5376 1.0000 0.0521 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2688 0.0184 0.0000 0.0267 0.0000
boeck2020/dar 0.0000 0.1454 0.0521 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002 0.7815 0.2267 0.0000 0.4173 0.0000
davies2009/mirex_qm_tempotracker 0.0080 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5806 0.0264 0.0000 0.0002 0.0127 0.0001 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0004 0.0001 0.0000 0.7582 0.0000 0.7238 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0018
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.0000 1.0000 0.5485 0.0000 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0074
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5485 1.0000 0.0000 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0009
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0000 0.7582 0.0000 0.0000 0.0000 1.0000 0.0000 0.9477 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
scheirer1998/percival2014 0.0125 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 0.0005 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.7238 0.0000 0.0000 0.0000 0.9477 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/ismir2017 0.0005 0.0000 0.0000 0.0000 0.5806 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0014 0.0000 0.0000 0.0038 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0002 0.0264 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0014 1.0000 0.0001 0.0108 0.6452 0.0008 0.0000
schreiber2018/cnn 0.0000 0.4201 0.2688 0.7815 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000 0.1677 0.0000 0.3931 0.0000
schreiber2018/fcn 0.0000 0.0429 0.0184 0.2267 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0108 0.1677 1.0000 0.0047 0.5052 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0127 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0038 0.6452 0.0000 0.0047 1.0000 0.0005 0.0000
sun2021/default 0.0000 0.0666 0.0267 0.4173 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0008 0.3931 0.5052 0.0005 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0000 0.0018 0.0000 0.0074 0.0009 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 27: Paired t-test p-values, using reference annotations 1.0 as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0080 0.0000 0.0000 0.0000 0.0000 0.0000 0.0125 0.0000 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.5376 0.1454 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4201 0.0429 0.0000 0.0666 0.0000
boeck2019/multi_task_hjdb 0.0000 0.5376 1.0000 0.0521 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2688 0.0184 0.0000 0.0267 0.0000
boeck2020/dar 0.0000 0.1454 0.0521 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002 0.7815 0.2267 0.0000 0.4173 0.0000
davies2009/mirex_qm_tempotracker 0.0080 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5806 0.0264 0.0000 0.0002 0.0127 0.0001 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0004 0.0001 0.0000 0.7582 0.0000 0.7238 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0018
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.0000 1.0000 0.5485 0.0000 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0074
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5485 1.0000 0.0000 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0009
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0000 0.7582 0.0000 0.0000 0.0000 1.0000 0.0000 0.9477 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
scheirer1998/percival2014 0.0125 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 0.0005 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.7238 0.0000 0.0000 0.0000 0.9477 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/ismir2017 0.0005 0.0000 0.0000 0.0000 0.5806 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0014 0.0000 0.0000 0.0038 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0002 0.0264 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0014 1.0000 0.0001 0.0108 0.6452 0.0008 0.0000
schreiber2018/cnn 0.0000 0.4201 0.2688 0.7815 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000 0.1677 0.0000 0.3931 0.0000
schreiber2018/fcn 0.0000 0.0429 0.0184 0.2267 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0108 0.1677 1.0000 0.0047 0.5052 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0127 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0038 0.6452 0.0000 0.0047 1.0000 0.0005 0.0000
sun2021/default 0.0000 0.0666 0.0267 0.4173 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0008 0.3931 0.5052 0.0005 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0000 0.0018 0.0000 0.0074 0.0009 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 28: Paired t-test p-values, using reference annotations 3.0 as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0080 0.0000 0.0000 0.0000 0.0000 0.0000 0.0190 0.0000 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.5376 0.1454 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4201 0.0429 0.0000 0.0666 0.0000
boeck2019/multi_task_hjdb 0.0000 0.5376 1.0000 0.0521 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2688 0.0184 0.0000 0.0267 0.0000
boeck2020/dar 0.0000 0.1454 0.0521 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002 0.7815 0.2267 0.0000 0.4173 0.0000
davies2009/mirex_qm_tempotracker 0.0080 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5620 0.0240 0.0000 0.0002 0.0116 0.0001 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0003 0.0001 0.0000 0.7311 0.0000 0.7607 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0025
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0003 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.0000 1.0000 0.5661 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0078
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5661 1.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0010
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0000 0.7311 0.0000 0.0000 0.0000 1.0000 0.0000 0.9337 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
scheirer1998/percival2014 0.0190 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002 0.0002 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.7607 0.0000 0.0000 0.0000 0.9337 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/ismir2017 0.0005 0.0000 0.0000 0.0000 0.5620 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0014 0.0000 0.0000 0.0038 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0002 0.0240 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0014 1.0000 0.0001 0.0110 0.6498 0.0008 0.0000
schreiber2018/cnn 0.0000 0.4201 0.2688 0.7815 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000 0.1677 0.0000 0.3934 0.0000
schreiber2018/fcn 0.0000 0.0429 0.0184 0.2267 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0110 0.1677 1.0000 0.0047 0.5048 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0116 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0038 0.6498 0.0000 0.0047 1.0000 0.0005 0.0000
sun2021/default 0.0000 0.0666 0.0267 0.4173 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0008 0.3934 0.5048 0.0005 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0000 0.0025 0.0000 0.0078 0.0010 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 29: Paired t-test p-values, using reference annotations 3.0-no-dupes as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0080 0.0000 0.0000 0.0000 0.0000 0.0000 0.0125 0.0000 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.5376 0.1454 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4201 0.0429 0.0000 0.0666 0.0000
boeck2019/multi_task_hjdb 0.0000 0.5376 1.0000 0.0521 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2688 0.0184 0.0000 0.0267 0.0000
boeck2020/dar 0.0000 0.1454 0.0521 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002 0.7815 0.2267 0.0000 0.4173 0.0000
davies2009/mirex_qm_tempotracker 0.0080 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5806 0.0264 0.0000 0.0002 0.0127 0.0001 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0004 0.0001 0.0000 0.7582 0.0000 0.7238 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0018
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.0000 1.0000 0.5485 0.0000 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0074
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5485 1.0000 0.0000 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0009
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0000 0.7582 0.0000 0.0000 0.0000 1.0000 0.0000 0.9477 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
scheirer1998/percival2014 0.0125 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 0.0005 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.7238 0.0000 0.0000 0.0000 0.9477 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/ismir2017 0.0005 0.0000 0.0000 0.0000 0.5806 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0014 0.0000 0.0000 0.0038 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0002 0.0264 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0014 1.0000 0.0001 0.0108 0.6452 0.0008 0.0000
schreiber2018/cnn 0.0000 0.4201 0.2688 0.7815 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000 0.1677 0.0000 0.3931 0.0000
schreiber2018/fcn 0.0000 0.0429 0.0184 0.2267 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0108 0.1677 1.0000 0.0047 0.5052 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0127 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0038 0.6452 0.0000 0.0047 1.0000 0.0005 0.0000
sun2021/default 0.0000 0.0666 0.0267 0.4173 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0008 0.3931 0.5052 0.0005 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0000 0.0018 0.0000 0.0074 0.0009 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 30: Paired t-test p-values, using reference annotations 2.0 as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0080 0.0000 0.0000 0.0000 0.0000 0.0000 0.0190 0.0000 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.5376 0.1454 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4201 0.0429 0.0000 0.0666 0.0000
boeck2019/multi_task_hjdb 0.0000 0.5376 1.0000 0.0521 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2688 0.0184 0.0000 0.0267 0.0000
boeck2020/dar 0.0000 0.1454 0.0521 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002 0.7815 0.2267 0.0000 0.4173 0.0000
davies2009/mirex_qm_tempotracker 0.0080 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5620 0.0240 0.0000 0.0002 0.0116 0.0001 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0003 0.0001 0.0000 0.7311 0.0000 0.7607 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0025
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0003 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.0000 1.0000 0.5661 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0078
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5661 1.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0010
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0000 0.7311 0.0000 0.0000 0.0000 1.0000 0.0000 0.9337 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
scheirer1998/percival2014 0.0190 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002 0.0002 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.7607 0.0000 0.0000 0.0000 0.9337 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/ismir2017 0.0005 0.0000 0.0000 0.0000 0.5620 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0014 0.0000 0.0000 0.0038 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0002 0.0240 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0014 1.0000 0.0001 0.0110 0.6498 0.0008 0.0000
schreiber2018/cnn 0.0000 0.4201 0.2688 0.7815 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000 0.1677 0.0000 0.3934 0.0000
schreiber2018/fcn 0.0000 0.0429 0.0184 0.2267 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0110 0.1677 1.0000 0.0047 0.5048 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0116 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0038 0.6498 0.0000 0.0047 1.0000 0.0005 0.0000
sun2021/default 0.0000 0.0666 0.0267 0.4173 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0008 0.3934 0.5048 0.0005 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0000 0.0025 0.0000 0.0078 0.0010 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 31: Paired t-test p-values, using reference annotations 2.0-no-dupes as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0080 0.0000 0.0000 0.0000 0.0000 0.0000 0.0125 0.0000 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.5376 0.1454 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4201 0.0429 0.0000 0.0666 0.0000
boeck2019/multi_task_hjdb 0.0000 0.5376 1.0000 0.0521 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2688 0.0184 0.0000 0.0267 0.0000
boeck2020/dar 0.0000 0.1454 0.0521 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002 0.7815 0.2267 0.0000 0.4173 0.0000
davies2009/mirex_qm_tempotracker 0.0080 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5806 0.0264 0.0000 0.0002 0.0127 0.0001 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0004 0.0001 0.0000 0.7582 0.0000 0.7238 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0018
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.0000 1.0000 0.5485 0.0000 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0074
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5485 1.0000 0.0000 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0009
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0000 0.7582 0.0000 0.0000 0.0000 1.0000 0.0000 0.9477 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
scheirer1998/percival2014 0.0125 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 0.0005 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.7238 0.0000 0.0000 0.0000 0.9477 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/ismir2017 0.0005 0.0000 0.0000 0.0000 0.5806 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0014 0.0000 0.0000 0.0038 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0002 0.0264 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0014 1.0000 0.0001 0.0108 0.6452 0.0008 0.0000
schreiber2018/cnn 0.0000 0.4201 0.2688 0.7815 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 1.0000 0.1677 0.0000 0.3931 0.0000
schreiber2018/fcn 0.0000 0.0429 0.0184 0.2267 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0108 0.1677 1.0000 0.0047 0.5052 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0127 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0038 0.6452 0.0000 0.0047 1.0000 0.0005 0.0000
sun2021/default 0.0000 0.0666 0.0267 0.4173 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0008 0.3931 0.5052 0.0005 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0000 0.0018 0.0000 0.0074 0.0009 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 32: Paired t-test p-values, using reference annotations 4.0 as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0191 0.0197 0.1158 0.0000 0.0409 0.3570 0.0000 0.0024 0.1554 0.0000 0.0288 0.0813 0.7460 0.0013 0.0011 0.2169 0.1789 0.0102
boeck2019/multi_task 0.0191 1.0000 0.3775 0.1249 0.0000 0.0184 0.6232 0.0000 0.0040 0.2820 0.0000 0.0577 0.1700 0.8820 0.0106 0.0138 0.5422 0.0070 0.0122
boeck2019/multi_task_hjdb 0.0197 0.3775 1.0000 0.1273 0.0000 0.0151 0.7943 0.0000 0.0061 0.3490 0.0000 0.0891 0.2631 0.6985 0.0386 0.0570 0.8194 0.0034 0.0165
boeck2020/dar 0.1158 0.1249 0.1273 1.0000 0.0000 0.0216 0.5115 0.0000 0.0029 0.2234 0.0000 0.0446 0.1290 0.9639 0.0040 0.0045 0.3985 0.0277 0.0091
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0415 0.0000 0.0000 0.8497 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0409 0.0184 0.0151 0.0216 0.0000 1.0000 0.0261 0.0000 0.0010 0.0161 0.0000 0.0054 0.0112 0.0373 0.0057 0.0068 0.0232 0.0809 0.0028
gkiokas2012/default 0.3570 0.6232 0.7943 0.5115 0.0000 0.0261 1.0000 0.0000 0.0202 0.5904 0.0000 0.2283 0.5372 0.6302 0.3704 0.4764 0.9306 0.1451 0.0912
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0415 0.0000 0.0000 1.0000 0.0290 0.0002 0.1870 0.0009 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0082
oliveira2010/ibt 0.0024 0.0040 0.0061 0.0029 0.0000 0.0010 0.0202 0.0290 1.0000 0.0960 0.0103 0.3110 0.0755 0.0050 0.0517 0.0455 0.0091 0.0005 0.5573
percival2014/stem 0.1554 0.2820 0.3490 0.2234 0.0000 0.0161 0.5904 0.0002 0.0960 1.0000 0.0001 0.4524 0.9451 0.2892 0.8397 0.9927 0.4584 0.0439 0.2257
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.8497 0.0000 0.0000 0.1870 0.0103 0.0001 1.0000 0.0009 0.0002 0.0000 0.0001 0.0001 0.0000 0.0000 0.0012
schreiber2014/default 0.0288 0.0577 0.0891 0.0446 0.0000 0.0054 0.2283 0.0009 0.3110 0.4524 0.0009 1.0000 0.2496 0.0262 0.4255 0.3397 0.1098 0.0068 0.6257
schreiber2017/ismir2017 0.0813 0.1700 0.2631 0.1290 0.0000 0.0112 0.5372 0.0000 0.0755 0.9451 0.0002 0.2496 1.0000 0.0587 0.8821 0.9283 0.3290 0.0149 0.2333
schreiber2017/mirex2017 0.7460 0.8820 0.6985 0.9639 0.0000 0.0373 0.6302 0.0000 0.0050 0.2892 0.0000 0.0262 0.0587 1.0000 0.0184 0.0729 0.5386 0.2865 0.0335
schreiber2018/cnn 0.0013 0.0106 0.0386 0.0040 0.0000 0.0057 0.3704 0.0000 0.0517 0.8397 0.0001 0.4255 0.8821 0.0184 1.0000 0.6915 0.0487 0.0000 0.2225
schreiber2018/fcn 0.0011 0.0138 0.0570 0.0045 0.0000 0.0068 0.4764 0.0000 0.0455 0.9927 0.0001 0.3397 0.9283 0.0729 0.6915 1.0000 0.2103 0.0003 0.1580
schreiber2018/ismir2018 0.2169 0.5422 0.8194 0.3985 0.0000 0.0232 0.9306 0.0000 0.0091 0.4584 0.0000 0.1098 0.3290 0.5386 0.0487 0.2103 1.0000 0.0071 0.0497
sun2021/default 0.1789 0.0070 0.0034 0.0277 0.0000 0.0809 0.1451 0.0000 0.0005 0.0439 0.0000 0.0068 0.0149 0.2865 0.0000 0.0003 0.0071 1.0000 0.0038
zplane/auftakt_v3 0.0102 0.0122 0.0165 0.0091 0.0000 0.0028 0.0912 0.0082 0.5573 0.2257 0.0012 0.6257 0.2333 0.0335 0.2225 0.1580 0.0497 0.0038 1.0000

Table 33: Paired t-test p-values, using reference annotations 1.0 as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0191 0.0197 0.1158 0.0000 0.0177 0.8188 0.0000 0.0087 0.1572 0.0000 0.0288 0.0813 0.7460 0.0013 0.0011 0.2169 0.1789 0.0558
boeck2019/multi_task 0.0191 1.0000 0.3775 0.1249 0.0000 0.0073 0.7633 0.0000 0.0143 0.2769 0.0000 0.0577 0.1700 0.8820 0.0106 0.0138 0.5422 0.0070 0.0671
boeck2019/multi_task_hjdb 0.0197 0.3775 1.0000 0.1273 0.0000 0.0059 0.5845 0.0000 0.0209 0.4038 0.0001 0.0891 0.2631 0.6985 0.0386 0.0570 0.8194 0.0034 0.0893
boeck2020/dar 0.1158 0.1249 0.1273 1.0000 0.0000 0.0086 0.9202 0.0000 0.0104 0.2220 0.0000 0.0446 0.1290 0.9639 0.0040 0.0045 0.3985 0.0277 0.0512
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0193 0.0000 0.0000 0.7757 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0177 0.0073 0.0059 0.0086 0.0000 1.0000 0.0222 0.0000 0.0010 0.0071 0.0000 0.0017 0.0047 0.0171 0.0021 0.0024 0.0098 0.0403 0.0047
gkiokas2012/default 0.8188 0.7633 0.5845 0.9202 0.0000 0.0222 1.0000 0.0001 0.0202 0.2826 0.0000 0.0810 0.2209 0.9386 0.0816 0.1082 0.5587 0.3663 0.1156
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0193 0.0000 0.0001 1.0000 0.0787 0.0026 0.1069 0.0089 0.0008 0.0001 0.0005 0.0004 0.0001 0.0000 0.0105
oliveira2010/ibt 0.0087 0.0143 0.0209 0.0104 0.0000 0.0010 0.0202 0.0787 1.0000 0.1865 0.0102 0.4990 0.1635 0.0163 0.1317 0.1108 0.0301 0.0022 0.4232
percival2014/stem 0.1572 0.2769 0.4038 0.2220 0.0000 0.0071 0.2826 0.0026 0.1865 1.0000 0.0002 0.4458 0.9448 0.2866 0.8289 0.9927 0.4570 0.0476 0.5945
scheirer1998/percival2014 0.0000 0.0000 0.0001 0.0000 0.7757 0.0000 0.0000 0.1069 0.0102 0.0002 1.0000 0.0014 0.0003 0.0000 0.0002 0.0001 0.0001 0.0000 0.0012
schreiber2014/default 0.0288 0.0577 0.0891 0.0446 0.0000 0.0017 0.0810 0.0089 0.4990 0.4458 0.0014 1.0000 0.2496 0.0262 0.4255 0.3397 0.1098 0.0068 0.8716
schreiber2017/ismir2017 0.0813 0.1700 0.2631 0.1290 0.0000 0.0047 0.2209 0.0008 0.1635 0.9448 0.0003 0.2496 1.0000 0.0587 0.8821 0.9283 0.3290 0.0149 0.5970
schreiber2017/mirex2017 0.7460 0.8820 0.6985 0.9639 0.0000 0.0171 0.9386 0.0001 0.0163 0.2866 0.0000 0.0262 0.0587 1.0000 0.0184 0.0729 0.5386 0.2865 0.1314
schreiber2018/cnn 0.0013 0.0106 0.0386 0.0040 0.0000 0.0021 0.0816 0.0005 0.1317 0.8289 0.0002 0.4255 0.8821 0.0184 1.0000 0.6915 0.0487 0.0000 0.6243
schreiber2018/fcn 0.0011 0.0138 0.0570 0.0045 0.0000 0.0024 0.1082 0.0004 0.1108 0.9927 0.0001 0.3397 0.9283 0.0729 0.6915 1.0000 0.2103 0.0003 0.4977
schreiber2018/ismir2018 0.2169 0.5422 0.8194 0.3985 0.0000 0.0098 0.5587 0.0001 0.0301 0.4570 0.0001 0.1098 0.3290 0.5386 0.0487 0.2103 1.0000 0.0071 0.1982
sun2021/default 0.1789 0.0070 0.0034 0.0277 0.0000 0.0403 0.3663 0.0000 0.0022 0.0476 0.0000 0.0068 0.0149 0.2865 0.0000 0.0003 0.0071 1.0000 0.0215
zplane/auftakt_v3 0.0558 0.0671 0.0893 0.0512 0.0000 0.0047 0.1156 0.0105 0.4232 0.5945 0.0012 0.8716 0.5970 0.1314 0.6243 0.4977 0.1982 0.0215 1.0000

Table 34: Paired t-test p-values, using reference annotations 3.0 as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0191 0.0197 0.1158 0.0000 0.0118 0.8171 0.0000 0.0069 0.1541 0.0000 0.0271 0.0762 0.7198 0.0013 0.0012 0.2205 0.1696 0.0320
boeck2019/multi_task 0.0191 1.0000 0.3775 0.1249 0.0000 0.0073 0.7633 0.0000 0.0143 0.2769 0.0000 0.0577 0.1700 0.8820 0.0106 0.0138 0.5422 0.0070 0.0671
boeck2019/multi_task_hjdb 0.0197 0.3775 1.0000 0.1273 0.0000 0.0059 0.5845 0.0000 0.0209 0.4038 0.0001 0.0891 0.2631 0.6985 0.0386 0.0570 0.8194 0.0034 0.0893
boeck2020/dar 0.1158 0.1249 0.1273 1.0000 0.0000 0.0086 0.9202 0.0000 0.0104 0.2220 0.0000 0.0446 0.1290 0.9639 0.0040 0.0045 0.3985 0.0277 0.0512
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0300 0.0000 0.0000 0.8935 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0118 0.0073 0.0059 0.0086 0.0000 1.0000 0.0156 0.0000 0.0006 0.0048 0.0000 0.0010 0.0030 0.0112 0.0013 0.0015 0.0066 0.0285 0.0023
gkiokas2012/default 0.8171 0.7633 0.5845 0.9202 0.0000 0.0156 1.0000 0.0001 0.0167 0.2792 0.0001 0.0776 0.2127 0.9196 0.0831 0.1102 0.5642 0.3614 0.0762
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0300 0.0000 0.0001 1.0000 0.0711 0.0018 0.1697 0.0065 0.0005 0.0000 0.0003 0.0002 0.0000 0.0000 0.0114
oliveira2010/ibt 0.0069 0.0143 0.0209 0.0104 0.0000 0.0006 0.0167 0.0711 1.0000 0.1680 0.0180 0.4709 0.1480 0.0138 0.1112 0.0935 0.0242 0.0016 0.4614
percival2014/stem 0.1541 0.2769 0.4038 0.2220 0.0000 0.0048 0.2792 0.0018 0.1680 1.0000 0.0004 0.4384 0.9344 0.2928 0.8425 0.9941 0.4471 0.0456 0.4916
scheirer1998/percival2014 0.0000 0.0000 0.0001 0.0000 0.8935 0.0000 0.0001 0.1697 0.0180 0.0004 1.0000 0.0027 0.0006 0.0001 0.0004 0.0003 0.0001 0.0000 0.0028
schreiber2014/default 0.0271 0.0577 0.0891 0.0446 0.0000 0.0010 0.0776 0.0065 0.4709 0.4384 0.0027 1.0000 0.2490 0.0261 0.4070 0.3241 0.1033 0.0062 0.9822
schreiber2017/ismir2017 0.0762 0.1700 0.2631 0.1290 0.0000 0.0030 0.2127 0.0005 0.1480 0.9344 0.0006 0.2490 1.0000 0.0587 0.9125 0.8967 0.3107 0.0132 0.4905
schreiber2017/mirex2017 0.7198 0.8820 0.6985 0.9639 0.0000 0.0112 0.9196 0.0000 0.0138 0.2928 0.0001 0.0261 0.0587 1.0000 0.0212 0.0807 0.5676 0.2661 0.0921
schreiber2018/cnn 0.0013 0.0106 0.0386 0.0040 0.0000 0.0013 0.0831 0.0003 0.1112 0.8425 0.0004 0.4070 0.9125 0.0212 1.0000 0.6915 0.0487 0.0000 0.4843
schreiber2018/fcn 0.0012 0.0138 0.0570 0.0045 0.0000 0.0015 0.1102 0.0002 0.0935 0.9941 0.0003 0.3241 0.8967 0.0807 0.6915 1.0000 0.2103 0.0002 0.3680
schreiber2018/ismir2018 0.2205 0.5422 0.8194 0.3985 0.0000 0.0066 0.5642 0.0000 0.0242 0.4471 0.0001 0.1033 0.3107 0.5676 0.0487 0.2103 1.0000 0.0068 0.1310
sun2021/default 0.1696 0.0070 0.0034 0.0277 0.0000 0.0285 0.3614 0.0000 0.0016 0.0456 0.0000 0.0062 0.0132 0.2661 0.0000 0.0002 0.0068 1.0000 0.0105
zplane/auftakt_v3 0.0320 0.0671 0.0893 0.0512 0.0000 0.0023 0.0762 0.0114 0.4614 0.4916 0.0028 0.9822 0.4905 0.0921 0.4843 0.3680 0.1310 0.0105 1.0000

Table 35: Paired t-test p-values, using reference annotations 3.0-no-dupes as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0191 0.0197 0.1158 0.0000 0.0073 0.8188 0.0000 0.0087 0.1572 0.0000 0.0288 0.0813 0.7460 0.0013 0.0011 0.0121 0.1789 0.0558
boeck2019/multi_task 0.0191 1.0000 0.3775 0.1249 0.0000 0.0028 0.7633 0.0000 0.0143 0.2769 0.0000 0.0577 0.1700 0.8820 0.0106 0.0138 0.0751 0.0070 0.0671
boeck2019/multi_task_hjdb 0.0197 0.3775 1.0000 0.1273 0.0000 0.0022 0.5845 0.0000 0.0209 0.4038 0.0001 0.0891 0.2631 0.6985 0.0386 0.0570 0.1855 0.0034 0.0893
boeck2020/dar 0.1158 0.1249 0.1273 1.0000 0.0000 0.0033 0.9202 0.0000 0.0104 0.2220 0.0000 0.0446 0.1290 0.9639 0.0040 0.0045 0.0312 0.0277 0.0512
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0193 0.0000 0.0000 0.7757 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0073 0.0028 0.0022 0.0033 0.0000 1.0000 0.0101 0.0000 0.0004 0.0031 0.0000 0.0006 0.0019 0.0073 0.0007 0.0009 0.0011 0.0188 0.0021
gkiokas2012/default 0.8188 0.7633 0.5845 0.9202 0.0000 0.0101 1.0000 0.0001 0.0202 0.2826 0.0000 0.0810 0.2209 0.9386 0.0816 0.1082 0.1649 0.3663 0.1156
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0193 0.0000 0.0001 1.0000 0.0787 0.0026 0.1069 0.0089 0.0008 0.0001 0.0005 0.0004 0.0003 0.0000 0.0105
oliveira2010/ibt 0.0087 0.0143 0.0209 0.0104 0.0000 0.0004 0.0202 0.0787 1.0000 0.1865 0.0102 0.4990 0.1635 0.0163 0.1317 0.1108 0.0893 0.0022 0.4232
percival2014/stem 0.1572 0.2769 0.4038 0.2220 0.0000 0.0031 0.2826 0.0026 0.1865 1.0000 0.0002 0.4458 0.9448 0.2866 0.8289 0.9927 0.8924 0.0476 0.5945
scheirer1998/percival2014 0.0000 0.0000 0.0001 0.0000 0.7757 0.0000 0.0000 0.1069 0.0102 0.0002 1.0000 0.0014 0.0003 0.0000 0.0002 0.0001 0.0001 0.0000 0.0012
schreiber2014/default 0.0288 0.0577 0.0891 0.0446 0.0000 0.0006 0.0810 0.0089 0.4990 0.4458 0.0014 1.0000 0.2496 0.0262 0.4255 0.3397 0.3000 0.0068 0.8716
schreiber2017/ismir2017 0.0813 0.1700 0.2631 0.1290 0.0000 0.0019 0.2209 0.0008 0.1635 0.9448 0.0003 0.2496 1.0000 0.0587 0.8821 0.9283 0.8043 0.0149 0.5970
schreiber2017/mirex2017 0.7460 0.8820 0.6985 0.9639 0.0000 0.0073 0.9386 0.0001 0.0163 0.2866 0.0000 0.0262 0.0587 1.0000 0.0184 0.0729 0.1496 0.2865 0.1314
schreiber2018/cnn 0.0013 0.0106 0.0386 0.0040 0.0000 0.0007 0.0816 0.0005 0.1317 0.8289 0.0002 0.4255 0.8821 0.0184 1.0000 0.6915 0.4350 0.0000 0.6243
schreiber2018/fcn 0.0011 0.0138 0.0570 0.0045 0.0000 0.0009 0.1082 0.0004 0.1108 0.9927 0.0001 0.3397 0.9283 0.0729 0.6915 1.0000 0.7731 0.0003 0.4977
schreiber2018/ismir2018 0.0121 0.0751 0.1855 0.0312 0.0000 0.0011 0.1649 0.0003 0.0893 0.8924 0.0001 0.3000 0.8043 0.1496 0.4350 0.7731 1.0000 0.0004 0.4565
sun2021/default 0.1789 0.0070 0.0034 0.0277 0.0000 0.0188 0.3663 0.0000 0.0022 0.0476 0.0000 0.0068 0.0149 0.2865 0.0000 0.0003 0.0004 1.0000 0.0215
zplane/auftakt_v3 0.0558 0.0671 0.0893 0.0512 0.0000 0.0021 0.1156 0.0105 0.4232 0.5945 0.0012 0.8716 0.5970 0.1314 0.6243 0.4977 0.4565 0.0215 1.0000

Table 36: Paired t-test p-values, using reference annotations 2.0 as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0191 0.0197 0.1158 0.0000 0.0047 0.8171 0.0000 0.0069 0.1541 0.0000 0.0271 0.0762 0.7198 0.0013 0.0012 0.0124 0.1696 0.0320
boeck2019/multi_task 0.0191 1.0000 0.3775 0.1249 0.0000 0.0028 0.7633 0.0000 0.0143 0.2769 0.0000 0.0577 0.1700 0.8820 0.0106 0.0138 0.0751 0.0070 0.0671
boeck2019/multi_task_hjdb 0.0197 0.3775 1.0000 0.1273 0.0000 0.0022 0.5845 0.0000 0.0209 0.4038 0.0001 0.0891 0.2631 0.6985 0.0386 0.0570 0.1855 0.0034 0.0893
boeck2020/dar 0.1158 0.1249 0.1273 1.0000 0.0000 0.0033 0.9202 0.0000 0.0104 0.2220 0.0000 0.0446 0.1290 0.9639 0.0040 0.0045 0.0312 0.0277 0.0512
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0300 0.0000 0.0000 0.8935 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0047 0.0028 0.0022 0.0033 0.0000 1.0000 0.0068 0.0000 0.0002 0.0020 0.0000 0.0004 0.0012 0.0046 0.0004 0.0005 0.0006 0.0127 0.0010
gkiokas2012/default 0.8171 0.7633 0.5845 0.9202 0.0000 0.0068 1.0000 0.0001 0.0167 0.2792 0.0001 0.0776 0.2127 0.9196 0.0831 0.1102 0.1677 0.3614 0.0762
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0300 0.0000 0.0001 1.0000 0.0711 0.0018 0.1697 0.0065 0.0005 0.0000 0.0003 0.0002 0.0002 0.0000 0.0114
oliveira2010/ibt 0.0069 0.0143 0.0209 0.0104 0.0000 0.0002 0.0167 0.0711 1.0000 0.1680 0.0180 0.4709 0.1480 0.0138 0.1112 0.0935 0.0748 0.0016 0.4614
percival2014/stem 0.1541 0.2769 0.4038 0.2220 0.0000 0.0020 0.2792 0.0018 0.1680 1.0000 0.0004 0.4384 0.9344 0.2928 0.8425 0.9941 0.8789 0.0456 0.4916
scheirer1998/percival2014 0.0000 0.0000 0.0001 0.0000 0.8935 0.0000 0.0001 0.1697 0.0180 0.0004 1.0000 0.0027 0.0006 0.0001 0.0004 0.0003 0.0002 0.0000 0.0028
schreiber2014/default 0.0271 0.0577 0.0891 0.0446 0.0000 0.0004 0.0776 0.0065 0.4709 0.4384 0.0027 1.0000 0.2490 0.0261 0.4070 0.3241 0.2863 0.0062 0.9822
schreiber2017/ismir2017 0.0762 0.1700 0.2631 0.1290 0.0000 0.0012 0.2127 0.0005 0.1480 0.9344 0.0006 0.2490 1.0000 0.0587 0.9125 0.8967 0.7757 0.0132 0.4905
schreiber2017/mirex2017 0.7198 0.8820 0.6985 0.9639 0.0000 0.0046 0.9196 0.0000 0.0138 0.2928 0.0001 0.0261 0.0587 1.0000 0.0212 0.0807 0.1621 0.2661 0.0921
schreiber2018/cnn 0.0013 0.0106 0.0386 0.0040 0.0000 0.0004 0.0831 0.0003 0.1112 0.8425 0.0004 0.4070 0.9125 0.0212 1.0000 0.6915 0.4350 0.0000 0.4843
schreiber2018/fcn 0.0012 0.0138 0.0570 0.0045 0.0000 0.0005 0.1102 0.0002 0.0935 0.9941 0.0003 0.3241 0.8967 0.0807 0.6915 1.0000 0.7731 0.0002 0.3680
schreiber2018/ismir2018 0.0124 0.0751 0.1855 0.0312 0.0000 0.0006 0.1677 0.0002 0.0748 0.8789 0.0002 0.2863 0.7757 0.1621 0.4350 0.7731 1.0000 0.0004 0.3417
sun2021/default 0.1696 0.0070 0.0034 0.0277 0.0000 0.0127 0.3614 0.0000 0.0016 0.0456 0.0000 0.0062 0.0132 0.2661 0.0000 0.0002 0.0004 1.0000 0.0105
zplane/auftakt_v3 0.0320 0.0671 0.0893 0.0512 0.0000 0.0010 0.0762 0.0114 0.4614 0.4916 0.0028 0.9822 0.4905 0.0921 0.4843 0.3680 0.3417 0.0105 1.0000

Table 37: Paired t-test p-values, using reference annotations 2.0-no-dupes as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0191 0.0197 0.1158 0.0000 0.2769 0.3251 0.0000 0.0087 0.0349 0.0000 0.0288 0.0813 0.7460 0.0013 0.0011 0.2169 0.1789 0.0231
boeck2019/multi_task 0.0191 1.0000 0.3775 0.1249 0.0000 0.1676 0.6080 0.0000 0.0143 0.0665 0.0000 0.0577 0.1700 0.8820 0.0106 0.0138 0.5422 0.0070 0.0279
boeck2019/multi_task_hjdb 0.0197 0.3775 1.0000 0.1273 0.0000 0.1452 0.7862 0.0000 0.0209 0.0817 0.0000 0.0891 0.2631 0.6985 0.0386 0.0570 0.8194 0.0034 0.0382
boeck2020/dar 0.1158 0.1249 0.1273 1.0000 0.0000 0.1873 0.4868 0.0000 0.0104 0.0510 0.0000 0.0446 0.1290 0.9639 0.0040 0.0045 0.3985 0.0277 0.0204
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0057 0.0000 0.0000 0.9621 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.2769 0.1676 0.1452 0.1873 0.0000 1.0000 0.1533 0.0005 0.0185 0.0316 0.0000 0.0521 0.0992 0.2474 0.0704 0.0811 0.1763 0.4121 0.0469
gkiokas2012/default 0.3251 0.6080 0.7862 0.4868 0.0000 0.1533 1.0000 0.0001 0.0514 0.1294 0.0000 0.2183 0.5250 0.6183 0.3486 0.4448 0.9284 0.1183 0.1571
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0057 0.0005 0.0001 1.0000 0.0290 0.0089 0.1286 0.0028 0.0002 0.0000 0.0001 0.0001 0.0000 0.0000 0.0065
oliveira2010/ibt 0.0087 0.0143 0.0209 0.0104 0.0000 0.0185 0.0514 0.0290 1.0000 0.5709 0.0063 0.4990 0.1635 0.0163 0.1317 0.1108 0.0301 0.0022 0.5446
percival2014/stem 0.0349 0.0665 0.0817 0.0510 0.0000 0.0316 0.1294 0.0089 0.5709 1.0000 0.0010 0.8922 0.4102 0.0744 0.4124 0.3284 0.1150 0.0093 0.9520
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.9621 0.0000 0.0000 0.1286 0.0063 0.0010 1.0000 0.0009 0.0002 0.0000 0.0001 0.0001 0.0000 0.0000 0.0012
schreiber2014/default 0.0288 0.0577 0.0891 0.0446 0.0000 0.0521 0.2183 0.0028 0.4990 0.8922 0.0009 1.0000 0.2496 0.0262 0.4255 0.3397 0.1098 0.0068 0.9299
schreiber2017/ismir2017 0.0813 0.1700 0.2631 0.1290 0.0000 0.0992 0.5250 0.0002 0.1635 0.4102 0.0002 0.2496 1.0000 0.0587 0.8821 0.9283 0.3290 0.0149 0.4170
schreiber2017/mirex2017 0.7460 0.8820 0.6985 0.9639 0.0000 0.2474 0.6183 0.0000 0.0163 0.0744 0.0000 0.0262 0.0587 1.0000 0.0184 0.0729 0.5386 0.2865 0.0699
schreiber2018/cnn 0.0013 0.0106 0.0386 0.0040 0.0000 0.0704 0.3486 0.0001 0.1317 0.4124 0.0001 0.4255 0.8821 0.0184 1.0000 0.6915 0.0487 0.0000 0.4186
schreiber2018/fcn 0.0011 0.0138 0.0570 0.0045 0.0000 0.0811 0.4448 0.0001 0.1108 0.3284 0.0001 0.3397 0.9283 0.0729 0.6915 1.0000 0.2103 0.0003 0.3101
schreiber2018/ismir2018 0.2169 0.5422 0.8194 0.3985 0.0000 0.1763 0.9284 0.0000 0.0301 0.1150 0.0000 0.1098 0.3290 0.5386 0.0487 0.2103 1.0000 0.0071 0.1047
sun2021/default 0.1789 0.0070 0.0034 0.0277 0.0000 0.4121 0.1183 0.0000 0.0022 0.0093 0.0000 0.0068 0.0149 0.2865 0.0000 0.0003 0.0071 1.0000 0.0080
zplane/auftakt_v3 0.0231 0.0279 0.0382 0.0204 0.0000 0.0469 0.1571 0.0065 0.5446 0.9520 0.0012 0.9299 0.4170 0.0699 0.4186 0.3101 0.1047 0.0080 1.0000

Table 38: Paired t-test p-values, using reference annotations 4.0 as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

OE1 on cvar-Subsets

How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?

OE1 on cvar-Subsets for 1.0 based on cvar-Values from 1.0

Figure 77: Mean OE1 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on cvar-Subsets for 2.0 based on cvar-Values from 1.0

Figure 78: Mean OE1 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0

Figure 79: Mean OE1 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on cvar-Subsets for 3.0 based on cvar-Values from 1.0

Figure 80: Mean OE1 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0

Figure 81: Mean OE1 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on cvar-Subsets for 4.0 based on cvar-Values from 1.0

Figure 82: Mean OE1 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on cvar-Subsets

How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?

OE2 on cvar-Subsets for 1.0 based on cvar-Values from 1.0

Figure 83: Mean OE2 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on cvar-Subsets for 2.0 based on cvar-Values from 1.0

Figure 84: Mean OE2 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0

Figure 85: Mean OE2 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on cvar-Subsets for 3.0 based on cvar-Values from 1.0

Figure 86: Mean OE2 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0

Figure 87: Mean OE2 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on cvar-Subsets for 4.0 based on cvar-Values from 1.0

Figure 88: Mean OE2 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean OE1 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

OE1 on Tempo-Subsets for 1.0

Figure 89: Mean OE1 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on Tempo-Subsets for 2.0

Figure 90: Mean OE1 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on Tempo-Subsets for 2.0-no-dupes

Figure 91: Mean OE1 for estimates compared to version 2.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on Tempo-Subsets for 3.0

Figure 92: Mean OE1 for estimates compared to version 3.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on Tempo-Subsets for 3.0-no-dupes

Figure 93: Mean OE1 for estimates compared to version 3.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 on Tempo-Subsets for 4.0

Figure 94: Mean OE1 for estimates compared to version 4.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean OE2 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

OE2 on Tempo-Subsets for 1.0

Figure 95: Mean OE2 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on Tempo-Subsets for 2.0

Figure 96: Mean OE2 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on Tempo-Subsets for 2.0-no-dupes

Figure 97: Mean OE2 for estimates compared to version 2.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on Tempo-Subsets for 3.0

Figure 98: Mean OE2 for estimates compared to version 3.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on Tempo-Subsets for 3.0-no-dupes

Figure 99: Mean OE2 for estimates compared to version 3.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE2 on Tempo-Subsets for 4.0

Figure 100: Mean OE2 for estimates compared to version 4.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE1 for Tempo

When fitting a generalized additive model (GAM) to OE1-values and a ground truth, what OE1 can we expect with confidence?

Estimated OE1 for Tempo for 1.0

Predictions of GAMs trained on OE1 for estimates for reference 1.0.

Figure 101: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE1 for Tempo for 2.0

Predictions of GAMs trained on OE1 for estimates for reference 2.0.

Figure 102: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE1 for Tempo for 2.0-no-dupes

Predictions of GAMs trained on OE1 for estimates for reference 2.0-no-dupes.

Figure 103: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE1 for Tempo for 3.0

Predictions of GAMs trained on OE1 for estimates for reference 3.0.

Figure 104: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE1 for Tempo for 3.0-no-dupes

Predictions of GAMs trained on OE1 for estimates for reference 3.0-no-dupes.

Figure 105: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE1 for Tempo for 4.0

Predictions of GAMs trained on OE1 for estimates for reference 4.0.

Figure 106: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE2 for Tempo

When fitting a generalized additive model (GAM) to OE2-values and a ground truth, what OE2 can we expect with confidence?

Estimated OE2 for Tempo for 1.0

Predictions of GAMs trained on OE2 for estimates for reference 1.0.

Figure 107: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE2 for Tempo for 2.0

Predictions of GAMs trained on OE2 for estimates for reference 2.0.

Figure 108: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE2 for Tempo for 2.0-no-dupes

Predictions of GAMs trained on OE2 for estimates for reference 2.0-no-dupes.

Figure 109: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE2 for Tempo for 3.0

Predictions of GAMs trained on OE2 for estimates for reference 3.0.

Figure 110: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE2 for Tempo for 3.0-no-dupes

Predictions of GAMs trained on OE2 for estimates for reference 3.0-no-dupes.

Figure 111: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE2 for Tempo for 4.0

Predictions of GAMs trained on OE2 for estimates for reference 4.0.

Figure 112: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

OE1 for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

OE1 for ‘tag_open’ Tags for 1.0

Figure 113: OE1 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE1 for ‘tag_open’ Tags for 2.0

Figure 114: OE1 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE1 for ‘tag_open’ Tags for 2.0-no-dupes

Figure 115: OE1 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE1 for ‘tag_open’ Tags for 3.0

Figure 116: OE1 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE1 for ‘tag_open’ Tags for 3.0-no-dupes

Figure 117: OE1 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE1 for ‘tag_open’ Tags for 4.0

Figure 118: OE1 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE2 for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

OE2 for ‘tag_open’ Tags for 1.0

Figure 119: OE2 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE2 for ‘tag_open’ Tags for 2.0

Figure 120: OE2 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE2 for ‘tag_open’ Tags for 2.0-no-dupes

Figure 121: OE2 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE2 for ‘tag_open’ Tags for 3.0

Figure 122: OE2 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE2 for ‘tag_open’ Tags for 3.0-no-dupes

Figure 123: OE2 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE2 for ‘tag_open’ Tags for 4.0

Figure 124: OE2 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE1 and AOE2

AOE1 is defined as absolute octave error between an estimate and a reference value: AOE1(E) = |log2(E/R)|.

AOE2 is the minimum of AOE1 allowing the octave errors 2, 3, 1/2, and 1/3: AOE2(E) = min(AOE1(E), AOE1(2E), AOE1(3E), AOE1(½E), AOE1(⅓E)).

Mean AOE1/AOE2 Results for 1.0

Estimator AOE1_MEAN AOE1_STDEV AOE2_MEAN AOE2_STDEV
boeck2020/dar 0.0256 0.1136 0.0151 0.0468
boeck2019/multi_task 0.0308 0.1373 0.0143 0.0464
sun2021/default 0.0321 0.1447 0.0152 0.0511
boeck2019/multi_task_hjdb 0.0331 0.1437 0.0147 0.0476
schreiber2018/cnn 0.0433 0.1720 0.0161 0.0551
schreiber2018/fcn 0.0647 0.2224 0.0155 0.0546
schreiber2018/ismir2018 0.0650 0.2227 0.0155 0.0558
schreiber2017/mirex2017 0.0921 0.2647 0.0195 0.0640
schreiber2017/ismir2017 0.1414 0.3289 0.0213 0.0684
boeck2015/tempodetector2016_default 0.1748 0.3689 0.0182 0.0470
davies2009/mirex_qm_tempotracker 0.3026 0.4284 0.0436 0.0823
zplane/auftakt_v3 0.3134 0.4450 0.0283 0.0793
oliveira2010/ibt 0.3213 0.4367 0.0417 0.0836
klapuri2006/percival2014 0.3238 0.4425 0.0380 0.0950
schreiber2014/default 0.3263 0.4515 0.0258 0.0800
percival2014/stem 0.3367 0.4767 0.0259 0.0722
scheirer1998/percival2014 0.3431 0.4305 0.0753 0.1372
echonest/version_3_2_1 0.3888 0.4899 0.0553 0.1204
gkiokas2012/default 0.4115 0.5460 0.0198 0.0634

Table 39: Mean AOE1/AOE2 for estimates compared to version 1.0 ordered by mean.

CSV JSON LATEX PICKLE

Raw data AOE1: CSV JSON LATEX PICKLE

Raw data AOE2: CSV JSON LATEX PICKLE

AOE1 distribution for 1.0

Figure 125: AOE1 for estimates compared to version 1.0. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 distribution for 1.0

Figure 126: AOE2 for estimates compared to version 1.0. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean AOE1/AOE2 Results for 2.0

Estimator AOE1_MEAN AOE1_STDEV AOE2_MEAN AOE2_STDEV
boeck2020/dar 0.0131 0.0847 0.0059 0.0058
boeck2019/multi_task 0.0203 0.1191 0.0059 0.0061
boeck2019/multi_task_hjdb 0.0211 0.1213 0.0063 0.0138
sun2021/default 0.0243 0.1235 0.0104 0.0197
schreiber2018/cnn 0.0344 0.1576 0.0096 0.0294
schreiber2018/fcn 0.0556 0.2101 0.0098 0.0252
schreiber2018/ismir2018 0.0567 0.2128 0.0094 0.0302
schreiber2017/mirex2017 0.0830 0.2609 0.0107 0.0437
schreiber2017/ismir2017 0.1296 0.3243 0.0124 0.0513
boeck2015/tempodetector2016_default 0.1628 0.3708 0.0055 0.0084
davies2009/mirex_qm_tempotracker 0.3010 0.4320 0.0380 0.0743
zplane/auftakt_v3 0.3064 0.4458 0.0207 0.0687
klapuri2006/percival2014 0.3171 0.4450 0.0301 0.0882
oliveira2010/ibt 0.3173 0.4391 0.0357 0.0781
schreiber2014/default 0.3186 0.4541 0.0171 0.0667
percival2014/stem 0.3294 0.4785 0.0176 0.0588
scheirer1998/percival2014 0.3399 0.4314 0.0712 0.1373
echonest/version_3_2_1 0.3839 0.4939 0.0471 0.1138
gkiokas2012/default 0.4057 0.5478 0.0126 0.0421

Table 40: Mean AOE1/AOE2 for estimates compared to version 2.0 ordered by mean.

CSV JSON LATEX PICKLE

Raw data AOE1: CSV JSON LATEX PICKLE

Raw data AOE2: CSV JSON LATEX PICKLE

AOE1 distribution for 2.0

Figure 127: AOE1 for estimates compared to version 2.0. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 distribution for 2.0

Figure 128: AOE2 for estimates compared to version 2.0. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean AOE1/AOE2 Results for 2.0-no-dupes

Estimator AOE1_MEAN AOE1_STDEV AOE2_MEAN AOE2_STDEV
boeck2020/dar 0.0131 0.0847 0.0059 0.0058
boeck2019/multi_task 0.0203 0.1191 0.0059 0.0061
boeck2019/multi_task_hjdb 0.0211 0.1213 0.0063 0.0138
sun2021/default 0.0244 0.1246 0.0103 0.0196
schreiber2018/cnn 0.0349 0.1591 0.0096 0.0297
schreiber2018/fcn 0.0565 0.2119 0.0098 0.0254
schreiber2018/ismir2018 0.0576 0.2147 0.0095 0.0305
schreiber2017/mirex2017 0.0845 0.2631 0.0108 0.0441
schreiber2017/ismir2017 0.1319 0.3269 0.0126 0.0518
boeck2015/tempodetector2016_default 0.1659 0.3736 0.0056 0.0085
davies2009/mirex_qm_tempotracker 0.3034 0.4329 0.0384 0.0750
zplane/auftakt_v3 0.3096 0.4473 0.0205 0.0683
oliveira2010/ibt 0.3203 0.4400 0.0362 0.0788
klapuri2006/percival2014 0.3204 0.4463 0.0302 0.0887
schreiber2014/default 0.3231 0.4557 0.0173 0.0673
percival2014/stem 0.3326 0.4799 0.0178 0.0593
scheirer1998/percival2014 0.3386 0.4311 0.0709 0.1368
echonest/version_3_2_1 0.3888 0.4956 0.0472 0.1137
gkiokas2012/default 0.4117 0.5499 0.0127 0.0425

Table 41: Mean AOE1/AOE2 for estimates compared to version 2.0-no-dupes ordered by mean.

CSV JSON LATEX PICKLE

Raw data AOE1: CSV JSON LATEX PICKLE

Raw data AOE2: CSV JSON LATEX PICKLE

AOE1 distribution for 2.0-no-dupes

Figure 129: AOE1 for estimates compared to version 2.0-no-dupes. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 distribution for 2.0-no-dupes

Figure 130: AOE2 for estimates compared to version 2.0-no-dupes. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean AOE1/AOE2 Results for 3.0

Estimator AOE1_MEAN AOE1_STDEV AOE2_MEAN AOE2_STDEV
boeck2020/dar 0.0109 0.0848 0.0037 0.0044
boeck2019/multi_task 0.0179 0.1198 0.0034 0.0039
boeck2019/multi_task_hjdb 0.0188 0.1220 0.0038 0.0125
sun2021/default 0.0225 0.1236 0.0086 0.0193
schreiber2018/cnn 0.0324 0.1583 0.0076 0.0292
schreiber2018/fcn 0.0537 0.2109 0.0077 0.0252
schreiber2018/ismir2018 0.0550 0.2138 0.0075 0.0308
schreiber2017/mirex2017 0.0803 0.2618 0.0078 0.0439
schreiber2017/ismir2017 0.1270 0.3252 0.0096 0.0516
boeck2015/tempodetector2016_default 0.1648 0.3705 0.0072 0.0066
davies2009/mirex_qm_tempotracker 0.3016 0.4320 0.0383 0.0740
zplane/auftakt_v3 0.3046 0.4473 0.0179 0.0685
oliveira2010/ibt 0.3161 0.4403 0.0339 0.0785
klapuri2006/percival2014 0.3165 0.4458 0.0286 0.0885
schreiber2014/default 0.3168 0.4557 0.0142 0.0669
percival2014/stem 0.3276 0.4799 0.0150 0.0589
scheirer1998/percival2014 0.3398 0.4315 0.0707 0.1373
echonest/version_3_2_1 0.3822 0.4949 0.0450 0.1149
gkiokas2012/default 0.4046 0.5490 0.0105 0.0417

Table 42: Mean AOE1/AOE2 for estimates compared to version 3.0 ordered by mean.

CSV JSON LATEX PICKLE

Raw data AOE1: CSV JSON LATEX PICKLE

Raw data AOE2: CSV JSON LATEX PICKLE

AOE1 distribution for 3.0

Figure 131: AOE1 for estimates compared to version 3.0. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 distribution for 3.0

Figure 132: AOE2 for estimates compared to version 3.0. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean AOE1/AOE2 Results for 3.0-no-dupes

Estimator AOE1_MEAN AOE1_STDEV AOE2_MEAN AOE2_STDEV
boeck2020/dar 0.0109 0.0848 0.0037 0.0044
boeck2019/multi_task 0.0179 0.1198 0.0034 0.0039
boeck2019/multi_task_hjdb 0.0188 0.1220 0.0038 0.0125
sun2021/default 0.0226 0.1247 0.0085 0.0192
schreiber2018/cnn 0.0330 0.1597 0.0076 0.0295
schreiber2018/fcn 0.0546 0.2128 0.0078 0.0254
schreiber2018/ismir2018 0.0559 0.2157 0.0076 0.0311
schreiber2017/mirex2017 0.0818 0.2640 0.0079 0.0443
schreiber2017/ismir2017 0.1294 0.3278 0.0097 0.0521
boeck2015/tempodetector2016_default 0.1678 0.3733 0.0072 0.0066
davies2009/mirex_qm_tempotracker 0.3040 0.4329 0.0386 0.0747
zplane/auftakt_v3 0.3079 0.4489 0.0177 0.0681
oliveira2010/ibt 0.3191 0.4413 0.0343 0.0792
klapuri2006/percival2014 0.3199 0.4472 0.0287 0.0889
schreiber2014/default 0.3213 0.4574 0.0144 0.0676
percival2014/stem 0.3308 0.4813 0.0153 0.0594
scheirer1998/percival2014 0.3385 0.4313 0.0704 0.1368
echonest/version_3_2_1 0.3871 0.4967 0.0451 0.1148
gkiokas2012/default 0.4107 0.5511 0.0106 0.0421

Table 43: Mean AOE1/AOE2 for estimates compared to version 3.0-no-dupes ordered by mean.

CSV JSON LATEX PICKLE

Raw data AOE1: CSV JSON LATEX PICKLE

Raw data AOE2: CSV JSON LATEX PICKLE

AOE1 distribution for 3.0-no-dupes

Figure 133: AOE1 for estimates compared to version 3.0-no-dupes. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 distribution for 3.0-no-dupes

Figure 134: AOE2 for estimates compared to version 3.0-no-dupes. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Mean AOE1/AOE2 Results for 4.0

Estimator AOE1_MEAN AOE1_STDEV AOE2_MEAN AOE2_STDEV
boeck2020/dar 0.0270 0.1343 0.0093 0.0263
boeck2019/multi_task 0.0305 0.1490 0.0085 0.0264
sun2021/default 0.0339 0.1612 0.0098 0.0318
boeck2019/multi_task_hjdb 0.0344 0.1598 0.0088 0.0281
schreiber2018/cnn 0.0425 0.1806 0.0102 0.0389
schreiber2018/ismir2018 0.0646 0.2303 0.0095 0.0395
schreiber2018/fcn 0.0659 0.2336 0.0095 0.0360
schreiber2017/mirex2017 0.0932 0.2737 0.0135 0.0505
schreiber2017/ismir2017 0.1426 0.3362 0.0152 0.0567
boeck2015/tempodetector2016_default 0.1712 0.3713 0.0121 0.0264
davies2009/mirex_qm_tempotracker 0.2975 0.4296 0.0382 0.0730
zplane/auftakt_v3 0.3084 0.4476 0.0214 0.0654
oliveira2010/ibt 0.3163 0.4386 0.0359 0.0755
klapuri2006/percival2014 0.3183 0.4449 0.0311 0.0852
schreiber2014/default 0.3212 0.4539 0.0197 0.0702
percival2014/stem 0.3309 0.4783 0.0202 0.0623
scheirer1998/percival2014 0.3402 0.4320 0.0712 0.1356
echonest/version_3_2_1 0.3852 0.4927 0.0496 0.1159
gkiokas2012/default 0.4061 0.5501 0.0125 0.0418

Table 44: Mean AOE1/AOE2 for estimates compared to version 4.0 ordered by mean.

CSV JSON LATEX PICKLE

Raw data AOE1: CSV JSON LATEX PICKLE

Raw data AOE2: CSV JSON LATEX PICKLE

AOE1 distribution for 4.0

Figure 135: AOE1 for estimates compared to version 4.0. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 distribution for 4.0

Figure 136: AOE2 for estimates compared to version 4.0. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Significance of Differences

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0268 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.4058 0.2964 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0611 0.0001 0.0000 0.8386 0.0000
boeck2019/multi_task_hjdb 0.0000 0.4058 1.0000 0.1247 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1183 0.0001 0.0002 0.8701 0.0000
boeck2020/dar 0.0000 0.2964 0.1247 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0057 0.0000 0.0000 0.1955 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.1215 0.1767 0.0445 0.0232 0.1422 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4642
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.1913 0.0001 0.0000 0.0016 0.0077 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.1913 1.0000 0.0000 0.0000 0.0000 0.0014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.1215 0.0001 0.0000 1.0000 0.6905 0.1382 0.2342 0.7689 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1114
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.1767 0.0000 0.0000 0.6905 1.0000 0.0849 0.1832 0.5806 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2140
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0445 0.0016 0.0000 0.1382 0.0849 1.0000 0.6524 0.2608 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0072
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0232 0.0077 0.0014 0.2342 0.1832 0.6524 1.0000 0.3117 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0873
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.1422 0.0001 0.0000 0.7689 0.5806 0.2608 0.3117 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1283
schreiber2017/ismir2017 0.0268 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0169 0.0159 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0611 0.1183 0.0057 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0116 0.0062 0.0907 0.0000
schreiber2018/fcn 0.0000 0.0001 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0169 0.0116 1.0000 0.9708 0.0003 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0159 0.0062 0.9708 1.0000 0.0001 0.0000
sun2021/default 0.0000 0.8386 0.8701 0.1955 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0907 0.0003 0.0001 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.4642 0.0000 0.0000 0.1114 0.2140 0.0072 0.0873 0.1283 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 45: Paired t-test p-values, using reference annotations 1.0 as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0136 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.7786 0.1816 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0346 0.0000 0.0000 0.4559 0.0000
boeck2019/multi_task_hjdb 0.0000 0.7786 1.0000 0.1088 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0391 0.0000 0.0000 0.5249 0.0000
boeck2020/dar 0.0000 0.1816 0.1088 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0009 0.0000 0.0000 0.0194 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.2803 0.2996 0.1297 0.0380 0.3557 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.8410
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.2013 0.0001 0.0001 0.0010 0.0148 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.2013 1.0000 0.0000 0.0000 0.0000 0.0023 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.2803 0.0001 0.0000 1.0000 0.9454 0.2071 0.1679 0.9757 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0734
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.2996 0.0001 0.0000 0.9454 1.0000 0.2025 0.1596 0.9396 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0765
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.1297 0.0010 0.0000 0.2071 0.2025 1.0000 0.4723 0.2491 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0081
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0380 0.0148 0.0023 0.1679 0.1596 0.4723 1.0000 0.1874 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0444
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.3557 0.0001 0.0000 0.9757 0.9396 0.2491 0.1874 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1566
schreiber2017/ismir2017 0.0136 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0220 0.0276 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0346 0.0391 0.0009 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0126 0.0053 0.1368 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0220 0.0126 1.0000 0.8801 0.0005 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0276 0.0053 0.8801 1.0000 0.0002 0.0000
sun2021/default 0.0000 0.4559 0.5249 0.0194 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1368 0.0005 0.0002 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.8410 0.0000 0.0000 0.0734 0.0765 0.0081 0.0444 0.1566 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 46: Paired t-test p-values, using reference annotations 3.0 as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0138 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.7786 0.1816 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0346 0.0000 0.0000 0.4559 0.0000
boeck2019/multi_task_hjdb 0.0000 0.7786 1.0000 0.1088 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0391 0.0000 0.0000 0.5249 0.0000
boeck2020/dar 0.0000 0.1816 0.1088 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0009 0.0000 0.0000 0.0194 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.2608 0.2910 0.1224 0.0576 0.3010 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7987
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.1864 0.0001 0.0000 0.0008 0.0078 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.1864 1.0000 0.0000 0.0000 0.0000 0.0010 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.2608 0.0001 0.0000 1.0000 0.9007 0.2099 0.2392 0.8699 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0782
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.2910 0.0000 0.0000 0.9007 1.0000 0.1895 0.2219 0.8093 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0911
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.1224 0.0008 0.0000 0.2099 0.1895 1.0000 0.5819 0.3122 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0083
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0576 0.0078 0.0010 0.2392 0.2219 0.5819 1.0000 0.2883 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0715
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.3010 0.0001 0.0000 0.8699 0.8093 0.3122 0.2883 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1248
schreiber2017/ismir2017 0.0138 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0218 0.0273 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0346 0.0391 0.0009 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0126 0.0053 0.1299 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0218 0.0126 1.0000 0.8801 0.0004 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0273 0.0053 0.8801 1.0000 0.0002 0.0000
sun2021/default 0.0000 0.4559 0.5249 0.0194 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1299 0.0004 0.0002 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.7987 0.0000 0.0000 0.0782 0.0911 0.0083 0.0715 0.1248 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 47: Paired t-test p-values, using reference annotations 3.0-no-dupes as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0298 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.7882 0.1684 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0400 0.0000 0.0000 0.5149 0.0000
boeck2019/multi_task_hjdb 0.0000 0.7882 1.0000 0.1007 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0447 0.0000 0.0000 0.5867 0.0000
boeck2020/dar 0.0000 0.1684 0.1007 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0010 0.0000 0.0000 0.0240 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.2437 0.2427 0.0977 0.0340 0.2843 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7189
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.2125 0.0001 0.0001 0.0010 0.0118 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.2125 1.0000 0.0000 0.0000 0.0000 0.0019 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.2437 0.0001 0.0000 1.0000 0.9722 0.1614 0.1787 0.8634 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1053
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.2427 0.0001 0.0000 0.9722 1.0000 0.1809 0.1808 0.8903 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0912
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0977 0.0010 0.0000 0.1614 0.1809 1.0000 0.5293 0.2486 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0079
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0340 0.0118 0.0019 0.1787 0.1808 0.5293 1.0000 0.2215 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0554
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.2843 0.0001 0.0000 0.8634 0.8903 0.2486 0.2215 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1539
schreiber2017/ismir2017 0.0298 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0183 0.0216 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0400 0.0447 0.0010 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0127 0.0058 0.1281 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0183 0.0127 1.0000 0.9033 0.0004 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0216 0.0058 0.9033 1.0000 0.0002 0.0000
sun2021/default 0.0000 0.5149 0.5867 0.0240 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1281 0.0004 0.0002 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.7189 0.0000 0.0000 0.1053 0.0912 0.0079 0.0554 0.1539 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 48: Paired t-test p-values, using reference annotations 2.0 as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0294 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.7882 0.1684 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0400 0.0000 0.0000 0.5149 0.0000
boeck2019/multi_task_hjdb 0.0000 0.7882 1.0000 0.1007 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0447 0.0000 0.0000 0.5867 0.0000
boeck2020/dar 0.0000 0.1684 0.1007 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0010 0.0000 0.0000 0.0240 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.2253 0.2347 0.0922 0.0519 0.2378 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.6802
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.1974 0.0001 0.0000 0.0008 0.0061 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.1974 1.0000 0.0000 0.0000 0.0000 0.0008 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.2253 0.0001 0.0000 1.0000 0.9822 0.1652 0.2534 0.7636 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1098
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.2347 0.0000 0.0000 0.9822 1.0000 0.1707 0.2489 0.7645 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1061
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0922 0.0008 0.0000 0.1652 0.1707 1.0000 0.6441 0.3125 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0081
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0519 0.0061 0.0008 0.2534 0.2489 0.6441 1.0000 0.3337 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0875
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.2378 0.0001 0.0000 0.7636 0.7645 0.3125 0.3337 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1221
schreiber2017/ismir2017 0.0294 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0181 0.0213 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0400 0.0447 0.0010 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0127 0.0058 0.1219 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0181 0.0127 1.0000 0.9033 0.0004 0.0000
schreiber2018/ismir2018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0213 0.0058 0.9033 1.0000 0.0002 0.0000
sun2021/default 0.0000 0.5149 0.5867 0.0240 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1219 0.0004 0.0002 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.6802 0.0000 0.0000 0.1098 0.1061 0.0081 0.0875 0.1221 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 49: Paired t-test p-values, using reference annotations 2.0-no-dupes as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0613 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
boeck2019/multi_task 0.0000 1.0000 0.2034 0.5058 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0739 0.0000 0.0001 0.5524 0.0000
boeck2019/multi_task_hjdb 0.0000 0.2034 1.0000 0.1311 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2013 0.0001 0.0004 0.9809 0.0000
boeck2020/dar 0.0000 0.5058 0.1311 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0154 0.0000 0.0000 0.1502 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.1312 0.1787 0.0508 0.0180 0.1469 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4625
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.2314 0.0001 0.0000 0.0010 0.0103 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.2314 1.0000 0.0000 0.0000 0.0000 0.0019 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.1312 0.0001 0.0000 1.0000 0.7431 0.1534 0.1930 0.7383 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1351
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.1787 0.0000 0.0000 0.7431 1.0000 0.1047 0.1561 0.5892 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2235
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0508 0.0010 0.0000 0.1534 0.1047 1.0000 0.5673 0.2988 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0099
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0180 0.0103 0.0019 0.1930 0.1561 0.5673 1.0000 0.2739 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0699
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.1469 0.0001 0.0000 0.7383 0.5892 0.2988 0.2739 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1354
schreiber2017/ismir2017 0.0613 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2017/mirex2017 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0189 0.0127 0.0000 0.0000
schreiber2018/cnn 0.0000 0.0739 0.2013 0.0154 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0064 0.0067 0.2018 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0189 0.0064 1.0000 0.8797 0.0004 0.0000
schreiber2018/ismir2018 0.0000 0.0001 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0127 0.0067 0.8797 1.0000 0.0005 0.0000
sun2021/default 0.0000 0.5524 0.9809 0.1502 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2018 0.0004 0.0005 1.0000 0.0000
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.4625 0.0000 0.0000 0.1351 0.2235 0.0099 0.0699 0.1354 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000

Table 50: Paired t-test p-values, using reference annotations 4.0 as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.3037 0.0000 0.0000 0.0006 0.0000 0.0022 0.1130 0.4266 0.0619 0.0051 0.0142 0.0002 0.0001
boeck2019/multi_task 0.0000 1.0000 0.1827 0.0001 0.0000 0.0000 0.0006 0.0000 0.0000 0.0000 0.0000 0.0000 0.0005 0.0026 0.1574 0.3086 0.4115 0.4810 0.0000
boeck2019/multi_task_hjdb 0.0000 0.1827 1.0000 0.3333 0.0000 0.0000 0.0018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0012 0.0066 0.3172 0.5733 0.6518 0.8816 0.0000
boeck2020/dar 0.0000 0.0001 0.3333 1.0000 0.0000 0.0000 0.0032 0.0000 0.0000 0.0000 0.0000 0.0000 0.0020 0.0114 0.4630 0.8165 0.8728 0.7501 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0081 0.0000 0.0900 0.5027 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0081 1.0000 0.0000 0.0001 0.0010 0.0000 0.0020 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.3037 0.0006 0.0018 0.0032 0.0000 0.0000 1.0000 0.0000 0.0000 0.0183 0.0000 0.0403 0.5495 0.9031 0.0505 0.0149 0.0168 0.0059 0.0020
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0900 0.0001 0.0000 1.0000 0.2410 0.0004 0.0000 0.0006 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0014
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.5027 0.0010 0.0000 0.2410 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.0006 0.0000 0.0000 0.0000 0.0000 0.0000 0.0183 0.0004 0.0000 1.0000 0.0000 0.9726 0.0680 0.0120 0.0000 0.0000 0.0000 0.0000 0.3900
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0020 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0022 0.0000 0.0000 0.0000 0.0000 0.0000 0.0403 0.0006 0.0000 0.9726 0.0000 1.0000 0.0125 0.0052 0.0000 0.0000 0.0000 0.0000 0.4085
schreiber2017/ismir2017 0.1130 0.0005 0.0012 0.0020 0.0000 0.0000 0.5495 0.0000 0.0000 0.0680 0.0000 0.0125 1.0000 0.2658 0.0065 0.0014 0.0026 0.0014 0.0164
schreiber2017/mirex2017 0.4266 0.0026 0.0066 0.0114 0.0000 0.0000 0.9031 0.0000 0.0000 0.0120 0.0000 0.0052 0.2658 1.0000 0.0112 0.0076 0.0138 0.0063 0.0025
schreiber2018/cnn 0.0619 0.1574 0.3172 0.4630 0.0000 0.0000 0.0505 0.0000 0.0000 0.0000 0.0000 0.0000 0.0065 0.0112 1.0000 0.5978 0.5061 0.3779 0.0000
schreiber2018/fcn 0.0051 0.3086 0.5733 0.8165 0.0000 0.0000 0.0149 0.0000 0.0000 0.0000 0.0000 0.0000 0.0014 0.0076 0.5978 1.0000 0.9673 0.7962 0.0000
schreiber2018/ismir2018 0.0142 0.4115 0.6518 0.8728 0.0000 0.0000 0.0168 0.0000 0.0000 0.0000 0.0000 0.0000 0.0026 0.0138 0.5061 0.9673 1.0000 0.7980 0.0000
sun2021/default 0.0002 0.4810 0.8816 0.7501 0.0000 0.0000 0.0059 0.0000 0.0000 0.0000 0.0000 0.0000 0.0014 0.0063 0.3779 0.7962 0.7980 1.0000 0.0000
zplane/auftakt_v3 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0020 0.0014 0.0000 0.3900 0.0000 0.4085 0.0164 0.0025 0.0000 0.0000 0.0000 0.0000 1.0000

Table 51: Paired t-test p-values, using reference annotations 1.0 as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0380 0.0000 0.0000 0.0004 0.0000 0.0059 0.2236 0.7170 0.7549 0.5960 0.8012 0.0546 0.0000
boeck2019/multi_task 0.0000 1.0000 0.3895 0.1018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0015 0.0074 0.0002 0.0000 0.0004 0.0000 0.0000
boeck2019/multi_task_hjdb 0.0000 0.3895 1.0000 0.8364 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.0035 0.0177 0.0013 0.0001 0.0027 0.0000 0.0000
boeck2020/dar 0.0000 0.1018 0.8364 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0022 0.0121 0.0004 0.0000 0.0007 0.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.1361 0.0000 0.0038 0.1321 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.1361 1.0000 0.0000 0.0002 0.0095 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0380 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0775 0.0000 0.2139 0.7198 0.2403 0.1220 0.1171 0.1023 0.2554 0.0072
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0038 0.0002 0.0000 1.0000 0.0979 0.0001 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0008
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.1321 0.0095 0.0000 0.0979 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0775 0.0001 0.0000 1.0000 0.0000 0.7791 0.0345 0.0042 0.0003 0.0009 0.0005 0.0026 0.3143
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0059 0.0000 0.0001 0.0000 0.0000 0.0000 0.2139 0.0001 0.0000 0.7791 0.0000 1.0000 0.0116 0.0043 0.0046 0.0069 0.0077 0.0248 0.2387
schreiber2017/ismir2017 0.2236 0.0015 0.0035 0.0022 0.0000 0.0000 0.7198 0.0000 0.0000 0.0345 0.0000 0.0116 1.0000 0.2470 0.2829 0.2963 0.2759 0.6072 0.0061
schreiber2017/mirex2017 0.7170 0.0074 0.0177 0.0121 0.0000 0.0000 0.2403 0.0000 0.0000 0.0042 0.0000 0.0043 0.2470 1.0000 0.8477 0.9407 0.8411 0.6148 0.0007
schreiber2018/cnn 0.7549 0.0002 0.0013 0.0004 0.0000 0.0000 0.1220 0.0000 0.0000 0.0003 0.0000 0.0046 0.2829 0.8477 1.0000 0.8901 0.9487 0.2666 0.0001
schreiber2018/fcn 0.5960 0.0000 0.0001 0.0000 0.0000 0.0000 0.1171 0.0000 0.0000 0.0009 0.0000 0.0069 0.2963 0.9407 0.8901 1.0000 0.8330 0.4119 0.0001
schreiber2018/ismir2018 0.8012 0.0004 0.0027 0.0007 0.0000 0.0000 0.1023 0.0000 0.0000 0.0005 0.0000 0.0077 0.2759 0.8411 0.9487 0.8330 1.0000 0.2429 0.0001
sun2021/default 0.0546 0.0000 0.0000 0.0000 0.0000 0.0000 0.2554 0.0000 0.0000 0.0026 0.0000 0.0248 0.6072 0.6148 0.2666 0.4119 0.2429 1.0000 0.0004
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0072 0.0008 0.0000 0.3143 0.0000 0.2387 0.0061 0.0007 0.0001 0.0001 0.0001 0.0004 1.0000

Table 52: Paired t-test p-values, using reference annotations 3.0 as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0360 0.0000 0.0000 0.0004 0.0000 0.0053 0.2073 0.6791 0.7265 0.5655 0.7729 0.0766 0.0001
boeck2019/multi_task 0.0000 1.0000 0.3895 0.1018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0015 0.0074 0.0002 0.0000 0.0004 0.0000 0.0000
boeck2019/multi_task_hjdb 0.0000 0.3895 1.0000 0.8364 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.0035 0.0177 0.0013 0.0001 0.0027 0.0000 0.0000
boeck2020/dar 0.0000 0.1018 0.8364 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0022 0.0121 0.0004 0.0000 0.0007 0.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.1514 0.0000 0.0036 0.1472 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.1514 1.0000 0.0000 0.0003 0.0121 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0360 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0748 0.0000 0.2077 0.7352 0.2495 0.1228 0.1180 0.1031 0.2137 0.0103
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0036 0.0003 0.0000 1.0000 0.0857 0.0001 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0007
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.1472 0.0121 0.0000 0.0857 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0748 0.0001 0.0000 1.0000 0.0000 0.7814 0.0348 0.0043 0.0003 0.0008 0.0005 0.0018 0.3939
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0053 0.0000 0.0001 0.0000 0.0000 0.0000 0.2077 0.0001 0.0000 0.7814 0.0000 1.0000 0.0116 0.0043 0.0044 0.0065 0.0073 0.0195 0.2996
schreiber2017/ismir2017 0.2073 0.0015 0.0035 0.0022 0.0000 0.0000 0.7352 0.0000 0.0000 0.0348 0.0000 0.0116 1.0000 0.2470 0.2725 0.2851 0.2659 0.5275 0.0089
schreiber2017/mirex2017 0.6791 0.0074 0.0177 0.0121 0.0000 0.0000 0.2495 0.0000 0.0000 0.0043 0.0000 0.0043 0.2470 1.0000 0.8223 0.9167 0.8196 0.7144 0.0011
schreiber2018/cnn 0.7265 0.0002 0.0013 0.0004 0.0000 0.0000 0.1228 0.0000 0.0000 0.0003 0.0000 0.0044 0.2725 0.8223 1.0000 0.8901 0.9487 0.3508 0.0002
schreiber2018/fcn 0.5655 0.0000 0.0001 0.0000 0.0000 0.0000 0.1180 0.0000 0.0000 0.0008 0.0000 0.0065 0.2851 0.9167 0.8901 1.0000 0.8331 0.5060 0.0001
schreiber2018/ismir2018 0.7729 0.0004 0.0027 0.0007 0.0000 0.0000 0.1031 0.0000 0.0000 0.0005 0.0000 0.0073 0.2659 0.8196 0.9487 0.8331 1.0000 0.3216 0.0001
sun2021/default 0.0766 0.0000 0.0000 0.0000 0.0000 0.0000 0.2137 0.0000 0.0000 0.0018 0.0000 0.0195 0.5275 0.7144 0.3508 0.5060 0.3216 1.0000 0.0006
zplane/auftakt_v3 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0103 0.0007 0.0000 0.3939 0.0000 0.2996 0.0089 0.0011 0.0002 0.0001 0.0001 0.0006 1.0000

Table 53: Paired t-test p-values, using reference annotations 3.0-no-dupes as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.3443 0.1935 0.4237 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0004 0.0021 0.0003 0.0000 0.0006 0.0000 0.0000
boeck2019/multi_task 0.3443 1.0000 0.4278 0.7818 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0008 0.0040 0.0011 0.0000 0.0016 0.0000 0.0000
boeck2019/multi_task_hjdb 0.1935 0.4278 1.0000 0.3703 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0021 0.0107 0.0056 0.0006 0.0094 0.0000 0.0000
boeck2020/dar 0.4237 0.7818 0.3703 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0007 0.0036 0.0009 0.0000 0.0018 0.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0413 0.0000 0.0184 0.4352 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0413 1.0000 0.0000 0.0001 0.0076 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0549 0.0000 0.1314 0.9387 0.3899 0.1146 0.1125 0.0866 0.1822 0.0035
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0184 0.0001 0.0000 1.0000 0.0779 0.0003 0.0000 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0033
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.4352 0.0076 0.0000 0.0779 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0549 0.0003 0.0000 1.0000 0.0000 0.8774 0.0450 0.0060 0.0001 0.0004 0.0002 0.0008 0.2739
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1314 0.0004 0.0000 0.8774 0.0000 1.0000 0.0101 0.0038 0.0013 0.0021 0.0019 0.0068 0.2521
schreiber2017/ismir2017 0.0004 0.0008 0.0021 0.0007 0.0000 0.0000 0.9387 0.0000 0.0000 0.0450 0.0000 0.0101 1.0000 0.2481 0.1332 0.1380 0.1119 0.2878 0.0062
schreiber2017/mirex2017 0.0021 0.0040 0.0107 0.0036 0.0000 0.0000 0.3899 0.0000 0.0000 0.0060 0.0000 0.0038 0.2481 1.0000 0.4263 0.5389 0.4298 0.8781 0.0008
schreiber2018/cnn 0.0003 0.0011 0.0056 0.0009 0.0000 0.0000 0.1146 0.0000 0.0000 0.0001 0.0000 0.0013 0.1332 0.4263 1.0000 0.8643 0.8745 0.3937 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0006 0.0000 0.0000 0.0000 0.1125 0.0000 0.0000 0.0004 0.0000 0.0021 0.1380 0.5389 0.8643 1.0000 0.7354 0.5595 0.0000
schreiber2018/ismir2018 0.0006 0.0016 0.0094 0.0018 0.0000 0.0000 0.0866 0.0000 0.0000 0.0002 0.0000 0.0019 0.1119 0.4298 0.8745 0.7354 1.0000 0.2817 0.0000
sun2021/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1822 0.0000 0.0000 0.0008 0.0000 0.0068 0.2878 0.8781 0.3937 0.5595 0.2817 1.0000 0.0001
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0035 0.0033 0.0000 0.2739 0.0000 0.2521 0.0062 0.0008 0.0000 0.0000 0.0000 0.0001 1.0000

Table 54: Paired t-test p-values, using reference annotations 2.0 as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.3443 0.1935 0.4237 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0005 0.0024 0.0004 0.0000 0.0009 0.0000 0.0000
boeck2019/multi_task 0.3443 1.0000 0.4278 0.7818 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0008 0.0040 0.0011 0.0000 0.0016 0.0000 0.0000
boeck2019/multi_task_hjdb 0.1935 0.4278 1.0000 0.3703 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0021 0.0107 0.0056 0.0006 0.0094 0.0000 0.0000
boeck2020/dar 0.4237 0.7818 0.3703 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0007 0.0036 0.0009 0.0000 0.0018 0.0000 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0481 0.0000 0.0175 0.4601 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0481 1.0000 0.0000 0.0001 0.0099 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0529 0.0000 0.1260 0.9594 0.4058 0.1166 0.1147 0.0883 0.1536 0.0050
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0175 0.0001 0.0000 1.0000 0.0692 0.0004 0.0000 0.0005 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0028
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.4601 0.0099 0.0000 0.0692 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0529 0.0004 0.0000 1.0000 0.0000 0.8835 0.0460 0.0061 0.0001 0.0004 0.0002 0.0005 0.3430
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1260 0.0005 0.0000 0.8835 0.0000 1.0000 0.0101 0.0038 0.0012 0.0020 0.0018 0.0052 0.3149
schreiber2017/ismir2017 0.0005 0.0008 0.0021 0.0007 0.0000 0.0000 0.9594 0.0000 0.0000 0.0460 0.0000 0.0101 1.0000 0.2481 0.1268 0.1310 0.1062 0.2386 0.0090
schreiber2017/mirex2017 0.0024 0.0040 0.0107 0.0036 0.0000 0.0000 0.4058 0.0000 0.0000 0.0061 0.0000 0.0038 0.2481 1.0000 0.4059 0.5172 0.4120 0.7731 0.0012
schreiber2018/cnn 0.0004 0.0011 0.0056 0.0009 0.0000 0.0000 0.1166 0.0000 0.0000 0.0001 0.0000 0.0012 0.1268 0.4059 1.0000 0.8643 0.8745 0.4912 0.0000
schreiber2018/fcn 0.0000 0.0000 0.0006 0.0000 0.0000 0.0000 0.1147 0.0000 0.0000 0.0004 0.0000 0.0020 0.1310 0.5172 0.8643 1.0000 0.7354 0.6627 0.0000
schreiber2018/ismir2018 0.0009 0.0016 0.0094 0.0018 0.0000 0.0000 0.0883 0.0000 0.0000 0.0002 0.0000 0.0018 0.1062 0.4120 0.8745 0.7354 1.0000 0.3666 0.0000
sun2021/default 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1536 0.0000 0.0000 0.0005 0.0000 0.0052 0.2386 0.7731 0.4912 0.6627 0.3666 1.0000 0.0001
zplane/auftakt_v3 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0050 0.0028 0.0000 0.3430 0.0000 0.3149 0.0090 0.0012 0.0000 0.0000 0.0000 0.0001 1.0000

Table 55: Paired t-test p-values, using reference annotations 2.0-no-dupes as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator boeck2015/tempodetector2016_default boeck2019/multi_task boeck2019/multi_task_hjdb boeck2020/dar davies2009/mirex_qm_tempotracker echonest/version_3_2_1 gkiokas2012/default klapuri2006/percival2014 oliveira2010/ibt percival2014/stem scheirer1998/percival2014 schreiber2014/default schreiber2017/ismir2017 schreiber2017/mirex2017 schreiber2018/cnn schreiber2018/fcn schreiber2018/ismir2018 sun2021/default zplane/auftakt_v3
boeck2015/tempodetector2016_default 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.7998 0.0000 0.0000 0.0002 0.0000 0.0021 0.1103 0.4185 0.0838 0.0056 0.0191 0.0028 0.0003
boeck2019/multi_task 0.0000 1.0000 0.2536 0.0000 0.0000 0.0000 0.0073 0.0000 0.0000 0.0000 0.0000 0.0000 0.0005 0.0027 0.1257 0.3043 0.3633 0.1251 0.0000
boeck2019/multi_task_hjdb 0.0000 0.2536 1.0000 0.2256 0.0000 0.0000 0.0158 0.0000 0.0000 0.0000 0.0000 0.0000 0.0011 0.0061 0.2450 0.5316 0.5589 0.3364 0.0000
boeck2020/dar 0.0000 0.0000 0.2256 1.0000 0.0000 0.0000 0.0297 0.0000 0.0000 0.0000 0.0000 0.0000 0.0022 0.0123 0.4102 0.8386 0.8278 0.6378 0.0000
davies2009/mirex_qm_tempotracker 0.0000 0.0000 0.0000 0.0000 1.0000 0.0106 0.0000 0.0324 0.4180 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
echonest/version_3_2_1 0.0000 0.0000 0.0000 0.0000 0.0106 1.0000 0.0000 0.0000 0.0013 0.0000 0.0006 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
gkiokas2012/default 0.7998 0.0073 0.0158 0.0297 0.0000 0.0000 1.0000 0.0000 0.0000 0.0023 0.0000 0.0126 0.2656 0.6635 0.2085 0.0757 0.0906 0.0917 0.0010
klapuri2006/percival2014 0.0000 0.0000 0.0000 0.0000 0.0324 0.0000 0.0000 1.0000 0.1346 0.0015 0.0000 0.0016 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0016
oliveira2010/ibt 0.0000 0.0000 0.0000 0.0000 0.4180 0.0013 0.0000 0.1346 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
percival2014/stem 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000 0.0023 0.0015 0.0000 1.0000 0.0000 0.8555 0.0442 0.0063 0.0000 0.0000 0.0000 0.0000 0.6790
scheirer1998/percival2014 0.0000 0.0000 0.0000 0.0000 0.0000 0.0006 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
schreiber2014/default 0.0021 0.0000 0.0000 0.0000 0.0000 0.0000 0.0126 0.0016 0.0000 0.8555 0.0000 1.0000 0.0111 0.0047 0.0000 0.0000 0.0000 0.0001 0.5759
schreiber2017/ismir2017 0.1103 0.0005 0.0011 0.0022 0.0000 0.0000 0.2656 0.0000 0.0000 0.0442 0.0000 0.0111 1.0000 0.2658 0.0081 0.0014 0.0031 0.0041 0.0345
schreiber2017/mirex2017 0.4185 0.0027 0.0061 0.0123 0.0000 0.0000 0.6635 0.0000 0.0000 0.0063 0.0000 0.0047 0.2658 1.0000 0.0149 0.0063 0.0141 0.0194 0.0064
schreiber2018/cnn 0.0838 0.1257 0.2450 0.4102 0.0000 0.0000 0.2085 0.0000 0.0000 0.0000 0.0000 0.0000 0.0081 0.0149 1.0000 0.5200 0.4809 0.6878 0.0000
schreiber2018/fcn 0.0056 0.3043 0.5316 0.8386 0.0000 0.0000 0.0757 0.0000 0.0000 0.0000 0.0000 0.0000 0.0014 0.0063 0.5200 1.0000 0.9652 0.7889 0.0000
schreiber2018/ismir2018 0.0191 0.3633 0.5589 0.8278 0.0000 0.0000 0.0906 0.0000 0.0000 0.0000 0.0000 0.0000 0.0031 0.0141 0.4809 0.9652 1.0000 0.7920 0.0000
sun2021/default 0.0028 0.1251 0.3364 0.6378 0.0000 0.0000 0.0917 0.0000 0.0000 0.0000 0.0000 0.0001 0.0041 0.0194 0.6878 0.7889 0.7920 1.0000 0.0000
zplane/auftakt_v3 0.0003 0.0000 0.0000 0.0000 0.0000 0.0000 0.0010 0.0016 0.0000 0.6790 0.0000 0.5759 0.0345 0.0064 0.0000 0.0000 0.0000 0.0000 1.0000

Table 56: Paired t-test p-values, using reference annotations 4.0 as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

AOE1 on cvar-Subsets

How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?

AOE1 on cvar-Subsets for 1.0 based on cvar-Values from 1.0

Figure 137: Mean AOE1 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on cvar-Subsets for 2.0 based on cvar-Values from 1.0

Figure 138: Mean AOE1 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0

Figure 139: Mean AOE1 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on cvar-Subsets for 3.0 based on cvar-Values from 1.0

Figure 140: Mean AOE1 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0

Figure 141: Mean AOE1 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on cvar-Subsets for 4.0 based on cvar-Values from 1.0

Figure 142: Mean AOE1 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on cvar-Subsets

How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?

AOE2 on cvar-Subsets for 1.0 based on cvar-Values from 1.0

Figure 143: Mean AOE2 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on cvar-Subsets for 2.0 based on cvar-Values from 1.0

Figure 144: Mean AOE2 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0

Figure 145: Mean AOE2 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on cvar-Subsets for 3.0 based on cvar-Values from 1.0

Figure 146: Mean AOE2 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0

Figure 147: Mean AOE2 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on cvar-Subsets for 4.0 based on cvar-Values from 1.0

Figure 148: Mean AOE2 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean AOE1 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

AOE1 on Tempo-Subsets for 1.0

Figure 149: Mean AOE1 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on Tempo-Subsets for 2.0

Figure 150: Mean AOE1 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on Tempo-Subsets for 2.0-no-dupes

Figure 151: Mean AOE1 for estimates compared to version 2.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on Tempo-Subsets for 3.0

Figure 152: Mean AOE1 for estimates compared to version 3.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on Tempo-Subsets for 3.0-no-dupes

Figure 153: Mean AOE1 for estimates compared to version 3.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 on Tempo-Subsets for 4.0

Figure 154: Mean AOE1 for estimates compared to version 4.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean AOE2 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

AOE2 on Tempo-Subsets for 1.0

Figure 155: Mean AOE2 for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on Tempo-Subsets for 2.0

Figure 156: Mean AOE2 for estimates compared to version 2.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on Tempo-Subsets for 2.0-no-dupes

Figure 157: Mean AOE2 for estimates compared to version 2.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on Tempo-Subsets for 3.0

Figure 158: Mean AOE2 for estimates compared to version 3.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on Tempo-Subsets for 3.0-no-dupes

Figure 159: Mean AOE2 for estimates compared to version 3.0-no-dupes for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE2 on Tempo-Subsets for 4.0

Figure 160: Mean AOE2 for estimates compared to version 4.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE1 for Tempo

When fitting a generalized additive model (GAM) to AOE1-values and a ground truth, what AOE1 can we expect with confidence?

Estimated AOE1 for Tempo for 1.0

Predictions of GAMs trained on AOE1 for estimates for reference 1.0.

Figure 161: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE1 for Tempo for 2.0

Predictions of GAMs trained on AOE1 for estimates for reference 2.0.

Figure 162: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE1 for Tempo for 2.0-no-dupes

Predictions of GAMs trained on AOE1 for estimates for reference 2.0-no-dupes.

Figure 163: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE1 for Tempo for 3.0

Predictions of GAMs trained on AOE1 for estimates for reference 3.0.

Figure 164: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE1 for Tempo for 3.0-no-dupes

Predictions of GAMs trained on AOE1 for estimates for reference 3.0-no-dupes.

Figure 165: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE1 for Tempo for 4.0

Predictions of GAMs trained on AOE1 for estimates for reference 4.0.

Figure 166: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE2 for Tempo

When fitting a generalized additive model (GAM) to AOE2-values and a ground truth, what AOE2 can we expect with confidence?

Estimated AOE2 for Tempo for 1.0

Predictions of GAMs trained on AOE2 for estimates for reference 1.0.

Figure 167: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE2 for Tempo for 2.0

Predictions of GAMs trained on AOE2 for estimates for reference 2.0.

Figure 168: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE2 for Tempo for 2.0-no-dupes

Predictions of GAMs trained on AOE2 for estimates for reference 2.0-no-dupes.

Figure 169: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE2 for Tempo for 3.0

Predictions of GAMs trained on AOE2 for estimates for reference 3.0.

Figure 170: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE2 for Tempo for 3.0-no-dupes

Predictions of GAMs trained on AOE2 for estimates for reference 3.0-no-dupes.

Figure 171: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE2 for Tempo for 4.0

Predictions of GAMs trained on AOE2 for estimates for reference 4.0.

Figure 172: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE1 for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

AOE1 for ‘tag_open’ Tags for 1.0

Figure 173: AOE1 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE1 for ‘tag_open’ Tags for 2.0

Figure 174: AOE1 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE1 for ‘tag_open’ Tags for 2.0-no-dupes

Figure 175: AOE1 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE1 for ‘tag_open’ Tags for 3.0

Figure 176: AOE1 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE1 for ‘tag_open’ Tags for 3.0-no-dupes

Figure 177: AOE1 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE1 for ‘tag_open’ Tags for 4.0

Figure 178: AOE1 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE2 for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

AOE2 for ‘tag_open’ Tags for 1.0

Figure 179: AOE2 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE2 for ‘tag_open’ Tags for 2.0

Figure 180: AOE2 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE2 for ‘tag_open’ Tags for 2.0-no-dupes

Figure 181: AOE2 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE2 for ‘tag_open’ Tags for 3.0

Figure 182: AOE2 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE2 for ‘tag_open’ Tags for 3.0-no-dupes

Figure 183: AOE2 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE2 for ‘tag_open’ Tags for 4.0

Figure 184: AOE2 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG


Generated by tempo_eval 0.1.1 on 2022-06-29 18:16. Size L.