ballroom
This is the tempo_eval report for the ‘ballroom’ corpus.
Reports for other corpora may be found here.
Table of Contents
- References for ‘ballroom’
- Estimates for ‘ballroom’
- Estimators
- boeck2015/tempodetector2016_default
- boeck2019/multi_task
- boeck2019/multi_task_hjdb
- boeck2020/dar
- davies2009/mirex_qm_tempotracker
- echonest/version_3_2_1
- gkiokas2012/default
- klapuri2006/percival2014
- oliveira2010/ibt
- percival2014/stem
- scheirer1998/percival2014
- schreiber2014/default
- schreiber2017/ismir2017
- schreiber2017/mirex2017
- schreiber2018/cnn
- schreiber2018/fcn
- schreiber2018/ismir2018
- sun2021/default
- zplane/auftakt_v3
- Basic Statistics
- Smoothed Tempo Distribution
- Accuracy
- Accuracy Results for 1.0
- Accuracy1 for 1.0
- Accuracy2 for 1.0
- Accuracy Results for 2.0
- Accuracy1 for 2.0
- Accuracy2 for 2.0
- Accuracy Results for 2.0-no-dupes
- Accuracy1 for 2.0-no-dupes
- Accuracy2 for 2.0-no-dupes
- Accuracy Results for 3.0
- Accuracy1 for 3.0
- Accuracy2 for 3.0
- Accuracy Results for 3.0-no-dupes
- Accuracy1 for 3.0-no-dupes
- Accuracy2 for 3.0-no-dupes
- Accuracy Results for 4.0
- Accuracy1 for 4.0
- Accuracy2 for 4.0
- Differing Items
- Significance of Differences
- Accuracy1 on cvar-Subsets
- Accuracy2 on cvar-Subsets
- Accuracy1 on Tempo-Subsets
- Accuracy2 on Tempo-Subsets
- Estimated Accuracy1 for Tempo
- Estimated Accuracy2 for Tempo
- Accuracy1 for ‘tag_open’ Tags
- Accuracy2 for ‘tag_open’ Tags
- OE1 and OE2
- Mean OE1/OE2 Results for 1.0
- OE1 distribution for 1.0
- OE2 distribution for 1.0
- Mean OE1/OE2 Results for 2.0
- OE1 distribution for 2.0
- OE2 distribution for 2.0
- Mean OE1/OE2 Results for 2.0-no-dupes
- OE1 distribution for 2.0-no-dupes
- OE2 distribution for 2.0-no-dupes
- Mean OE1/OE2 Results for 3.0
- OE1 distribution for 3.0
- OE2 distribution for 3.0
- Mean OE1/OE2 Results for 3.0-no-dupes
- OE1 distribution for 3.0-no-dupes
- OE2 distribution for 3.0-no-dupes
- Mean OE1/OE2 Results for 4.0
- OE1 distribution for 4.0
- OE2 distribution for 4.0
- Significance of Differences
- OE1 on cvar-Subsets
- OE2 on cvar-Subsets
- OE1 on Tempo-Subsets
- OE2 on Tempo-Subsets
- Estimated OE1 for Tempo
- Estimated OE2 for Tempo
- OE1 for ‘tag_open’ Tags
- OE2 for ‘tag_open’ Tags
- AOE1 and AOE2
- Mean AOE1/AOE2 Results for 1.0
- AOE1 distribution for 1.0
- AOE2 distribution for 1.0
- Mean AOE1/AOE2 Results for 2.0
- AOE1 distribution for 2.0
- AOE2 distribution for 2.0
- Mean AOE1/AOE2 Results for 2.0-no-dupes
- AOE1 distribution for 2.0-no-dupes
- AOE2 distribution for 2.0-no-dupes
- Mean AOE1/AOE2 Results for 3.0
- AOE1 distribution for 3.0
- AOE2 distribution for 3.0
- Mean AOE1/AOE2 Results for 3.0-no-dupes
- AOE1 distribution for 3.0-no-dupes
- AOE2 distribution for 3.0-no-dupes
- Mean AOE1/AOE2 Results for 4.0
- AOE1 distribution for 4.0
- AOE2 distribution for 4.0
- Significance of Differences
- AOE1 on cvar-Subsets
- AOE2 on cvar-Subsets
- AOE1 on Tempo-Subsets
- AOE2 on Tempo-Subsets
- Estimated AOE1 for Tempo
- Estimated AOE2 for Tempo
- AOE1 for ‘tag_open’ Tags
- AOE2 for ‘tag_open’ Tags
- Estimators
References for ‘ballroom’
References
1.0
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 1.0 |
| Curator | Simon Dixon |
| Data Source | BallroomDancers.com, checked by human |
| Annotator, bibtex | Gouyon2006 |
| Annotator, ref_url | http://mtg.upf.edu/ismir2004/contest/tempoContest/node5.html |
2.0
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 2.0 |
| Curator | Florian Krebs |
| Data Source | manual annotation |
| Annotation Tools | derived from beat annotations |
| Annotation Rules | median of inter beat intervals |
| Annotator, bibtex | Krebs2013 |
| Annotator, ref_url | https://github.com/CPJKU/BallroomAnnotations |
2.0-no-dupes
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 2.0-no-dupes |
| Curator | Florian Krebs |
| Data Source | manual annotation |
| Annotation Tools | derived from beat annotations |
| Annotation Rules | median of inter beat intervals, duplicate tracks removed (http://media.aau.dk/null_space_pursuits/2014/01/ballroom-dataset.html) |
| Annotator, bibtex | Krebs2013 |
| Annotator, ref_url | https://github.com/CPJKU/BallroomAnnotations |
3.0
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 3.0 |
| Curator | Florian Krebs |
| Data Source | manual annotation |
| Annotation Tools | derived from beat annotations |
| Annotation Rules | based on median of inter corresponding beat intervals |
| Annotator, bibtex | Krebs2013 |
| Annotator, ref_url | https://github.com/CPJKU/BallroomAnnotations |
3.0-no-dupes
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 3.0-no-dupes |
| Curator | Florian Krebs |
| Data Source | manual annotation |
| Annotation Tools | derived from beat annotations |
| Annotation Rules | based on median of inter corresponding beat intervals, duplicate tracks removed (http://media.aau.dk/null_space_pursuits/2014/01/ballroom-dataset.html) |
| Annotator, bibtex | Krebs2013 |
| Annotator, ref_url | https://github.com/CPJKU/BallroomAnnotations |
4.0
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 4.0 |
| Curator | Graham Percival |
| Data Source | BallroomDancers.com, checked by human |
| Annotator, bibtex | Percival2014 |
| Annotator, ref_url | http://www.marsyas.info/tempo/ |
Basic Statistics
| Reference | Size | Min | Max | Avg | Stdev | Sweet Oct. Start | Sweet Oct. Coverage |
|---|---|---|---|---|---|---|---|
| 1.0 | 698 | 60.00 | 224.00 | 130.14 | 39.53 | 91.00 | 0.71 |
| 2.0 | 698 | 68.85 | 214.29 | 129.80 | 39.69 | 72.00 | 0.71 |
| 2.0-no-dupes | 685 | 68.85 | 214.29 | 130.03 | 39.83 | 72.00 | 0.71 |
| 3.0 | 698 | 68.57 | 214.29 | 129.77 | 39.72 | 72.00 | 0.71 |
| 3.0-no-dupes | 685 | 68.57 | 214.29 | 130.00 | 39.87 | 72.00 | 0.71 |
| 4.0 | 698 | 58.00 | 219.00 | 129.77 | 39.63 | 71.00 | 0.71 |
Smoothed Tempo Distribution
Figure 1: Percentage of values in tempo interval.
CSV JSON LATEX PICKLE SVG PDF PNG
Tag Distribution for ‘tag_open’
Figure 2: Percentage of tracks tagged with tags from namespace ‘tag_open’. Annotations are from reference 1.0.
CSV JSON LATEX PICKLE SVG PDF PNG
Beat-Based Tempo Variation
Figure 3: Fraction of the dataset with beat-annotated tracks with cvar < τ.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimates for ‘ballroom’
Estimators
boeck2015/tempodetector2016_default
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 0.17.dev0 |
| Annotation Tools | TempoDetector.2016, madmom, https://github.com/CPJKU/madmom |
| Annotator, bibtex | Boeck2015 |
boeck2019/multi_task
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 0.0.1 |
| Annotation Tools | model=multi_task, https://github.com/superbock/ISMIR2019 |
| Annotator, bibtex | Boeck2019 |
boeck2019/multi_task_hjdb
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 0.0.1 |
| Annotation Tools | model=multi_task_hjdb, https://github.com/superbock/ISMIR2019 |
| Annotator, bibtex | Boeck2019 |
boeck2020/dar
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 0.0.1 |
| Annotation Tools | https://github.com/superbock/ISMIR2020 |
| Annotator, bibtex | Boeck2020 |
davies2009/mirex_qm_tempotracker
| Attribute | Value | |
|---|---|---|
| Corpus | ballroom | |
| Version | 1.0 | |
| Annotation Tools | QM Tempotracker, Sonic Annotator plugin. https://code.soundsoftware.ac.uk/projects/mirex2013/repository/show/audio_tempo_estimation/qm-tempotracker Note that the current macOS build of ‘qm-vamp-plugins’ was used. | |
| Annotator, bibtex | Davies2009 | Davies2007 |
echonest/version_3_2_1
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 3.2.1 |
| Data Source | Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014. |
| Annotation Tools | Echo Nest track analyzer v3.2.1 |
| Annotator, bibtex | Percival2014 |
gkiokas2012/default
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 1.0 |
| Data Source | Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014. |
| Annotation Tools | Gkiokas2012 |
| Annotator, bibtex | Gkiokas2012 |
klapuri2006/percival2014
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 1.0 |
| Data Source | Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014. |
| Annotation Tools | Klapuri 2006 |
| Annotator, bibtex | Klapuri2006 |
oliveira2010/ibt
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 1.0 |
| Data Source | Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014. |
| Annotation Tools | Oliveira 2010 |
| Annotator, bibtex | Oliveira2010 |
percival2014/stem
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 1.0 |
| Annotation Tools | percival 2014, ‘tempo’ implementation from Marsyas, http://marsyas.info, git checkout tempo-stem |
| Annotator, bibtex | Percival2014 |
scheirer1998/percival2014
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 1.0 |
| Data Source | Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014. |
| Annotation Tools | Scheirer 1998 |
| Annotator, bibtex | Scheirer1998 |
schreiber2014/default
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 0.0.1 |
| Annotation Tools | schreiber 2014, http://www.tagtraum.com/tempo_estimation.html |
| Annotator, bibtex | Schreiber2014 |
schreiber2017/ismir2017
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 0.0.4 |
| Annotation Tools | schreiber 2017, model=ismir2017, http://www.tagtraum.com/tempo_estimation.html |
| Annotator, bibtex | Schreiber2017 |
schreiber2017/mirex2017
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 0.0.4 |
| Annotation Tools | schreiber 2017, model=mirex2017, http://www.tagtraum.com/tempo_estimation.html |
| Annotator, bibtex | Schreiber2017 |
schreiber2018/cnn
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 0.0.2 |
| Data Source | Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018. |
| Annotation Tools | schreiber tempo-cnn (model=cnn), https://github.com/hendriks73/tempo-cnn |
schreiber2018/fcn
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 0.0.2 |
| Data Source | Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018. |
| Annotation Tools | schreiber tempo-cnn (model=fcn), https://github.com/hendriks73/tempo-cnn |
schreiber2018/ismir2018
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 0.0.2 |
| Data Source | Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018. |
| Annotation Tools | schreiber tempo-cnn (model=ismir2018), https://github.com/hendriks73/tempo-cnn |
sun2021/default
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 0.0.2 |
| Data Source | Xiaoheng Sun, Qiqi He, Yongwei Gao, Wei Li. Musical Tempo Estimation Using a Multi-scale Network. in Proc. of the 22nd Int. Society for Music Information Retrieval Conf., Online, 2021 |
| Annotation Tools | https://github.com/Qqi-HE/TempoEstimation_MGANet |
| Annotator, bibtex | Sun2021 |
zplane/auftakt_v3
| Attribute | Value |
|---|---|
| Corpus | ballroom |
| Version | 3.0 |
| Data Source | Graham Percival and George Tzanetakis. Streamlined tempo estimation based on autocorrelation and crosscorrelation with pulses. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1765–1776, 2014. |
| Annotation Tools | zplane aufTAKT version 3.0, http://licensing.zplane.de/technology#auftakt |
| Annotator, bibtex | Percival2014 |
Basic Statistics
| Estimator | Size | Min | Max | Avg | Stdev | Sweet Oct. Start | Sweet Oct. Coverage |
|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 698 | 57.69 | 214.29 | 117.51 | 31.52 | 74.00 | 0.83 |
| boeck2019/multi_task | 685 | 73.23 | 211.40 | 130.61 | 39.91 | 72.00 | 0.71 |
| boeck2019/multi_task_hjdb | 685 | 52.38 | 212.21 | 130.79 | 39.93 | 73.00 | 0.70 |
| boeck2020/dar | 685 | 73.20 | 214.25 | 129.90 | 39.69 | 72.00 | 0.71 |
| davies2009/mirex_qm_tempotracker | 698 | 60.09 | 206.72 | 120.29 | 28.62 | 84.00 | 0.87 |
| echonest/version_3_2_1 | 697 | 43.20 | 207.60 | 104.37 | 31.47 | 67.00 | 0.75 |
| gkiokas2012/default | 698 | 40.00 | 207.00 | 98.23 | 22.77 | 67.00 | 0.86 |
| klapuri2006/percival2014 | 698 | 59.75 | 169.44 | 106.18 | 16.64 | 76.00 | 0.99 |
| oliveira2010/ibt | 698 | 83.00 | 167.00 | 106.53 | 17.34 | 81.00 | 1.00 |
| percival2014/stem | 698 | 57.90 | 150.34 | 101.91 | 18.04 | 68.00 | 0.96 |
| scheirer1998/percival2014 | 647 | 61.35 | 181.82 | 114.10 | 31.46 | 69.00 | 0.77 |
| schreiber2014/default | 698 | 55.83 | 142.73 | 101.77 | 17.17 | 69.00 | 0.96 |
| schreiber2017/ismir2017 | 698 | 60.25 | 208.50 | 122.75 | 36.08 | 72.00 | 0.78 |
| schreiber2017/mirex2017 | 698 | 42.27 | 211.97 | 126.02 | 38.12 | 72.00 | 0.74 |
| schreiber2018/cnn | 698 | 71.00 | 216.00 | 129.79 | 39.49 | 70.00 | 0.72 |
| schreiber2018/fcn | 698 | 62.00 | 232.00 | 128.65 | 39.00 | 74.00 | 0.73 |
| schreiber2018/ismir2018 | 698 | 75.00 | 212.00 | 126.29 | 37.97 | 74.00 | 0.76 |
| sun2021/default | 698 | 60.00 | 215.00 | 129.33 | 39.54 | 73.00 | 0.72 |
| zplane/auftakt_v3 | 698 | 65.90 | 166.50 | 105.01 | 17.50 | 74.00 | 0.98 |
Smoothed Tempo Distribution
Figure 4: Percentage of values in tempo interval.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy
Accuracy1 is defined as the percentage of correct estimates, allowing a 4% tolerance for individual BPM values.
Accuracy2 additionally permits estimates to be wrong by a factor of 2, 3, 1/2 or 1/3 (so-called octave errors).
See [Gouyon2006].
Note: When comparing accuracy values for different algorithms, keep in mind that an algorithm may have been trained on the test set or that the test set may have even been created using one of the tested algorithms.
Accuracy Results for 1.0
| Estimator | Accuracy1 | Accuracy2 |
|---|---|---|
| sun2021/default | 0.9441 | 0.9585 |
| boeck2019/multi_task | 0.9370 | 0.9499 |
| boeck2020/dar | 0.9370 | 0.9456 |
| schreiber2018/cnn | 0.9341 | 0.9570 |
| boeck2019/multi_task_hjdb | 0.9327 | 0.9470 |
| schreiber2018/fcn | 0.9155 | 0.9613 |
| schreiber2018/ismir2018 | 0.9126 | 0.9599 |
| schreiber2017/mirex2017 | 0.8825 | 0.9527 |
| schreiber2017/ismir2017 | 0.8309 | 0.9484 |
| boeck2015/tempodetector2016_default | 0.8052 | 0.9542 |
| davies2009/mirex_qm_tempotracker | 0.6619 | 0.8968 |
| schreiber2014/default | 0.6433 | 0.9384 |
| zplane/auftakt_v3 | 0.6418 | 0.9212 |
| percival2014/stem | 0.6275 | 0.9198 |
| klapuri2006/percival2014 | 0.6232 | 0.9011 |
| oliveira2010/ibt | 0.6203 | 0.8782 |
| gkiokas2012/default | 0.6003 | 0.9441 |
| echonest/version_3_2_1 | 0.5516 | 0.8410 |
| scheirer1998/percival2014 | 0.5258 | 0.7364 |
Table 3: Mean accuracy of estimates compared to version 1.0 with 4% tolerance ordered by Accuracy1.
Raw data Accuracy1: CSV JSON LATEX PICKLE
Raw data Accuracy2: CSV JSON LATEX PICKLE
Accuracy1 for 1.0
Figure 5: Mean Accuracy1 for estimates compared to version 1.0 depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for 1.0
Figure 6: Mean Accuracy2 for estimates compared to version 1.0 depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy Results for 2.0
| Estimator | Accuracy1 | Accuracy2 |
|---|---|---|
| boeck2020/dar | 0.9742 | 0.9814 |
| sun2021/default | 0.9742 | 0.9871 |
| schreiber2018/cnn | 0.9685 | 0.9928 |
| boeck2019/multi_task | 0.9670 | 0.9814 |
| boeck2019/multi_task_hjdb | 0.9656 | 0.9799 |
| schreiber2018/ismir2018 | 0.9470 | 0.9943 |
| schreiber2018/fcn | 0.9456 | 0.9900 |
| schreiber2017/mirex2017 | 0.9169 | 0.9885 |
| schreiber2017/ismir2017 | 0.8682 | 0.9842 |
| boeck2015/tempodetector2016_default | 0.8453 | 1.0000 |
| davies2009/mirex_qm_tempotracker | 0.6762 | 0.9241 |
| schreiber2014/default | 0.6719 | 0.9728 |
| zplane/auftakt_v3 | 0.6676 | 0.9513 |
| percival2014/stem | 0.6576 | 0.9556 |
| klapuri2006/percival2014 | 0.6461 | 0.9284 |
| oliveira2010/ibt | 0.6447 | 0.9069 |
| gkiokas2012/default | 0.6304 | 0.9814 |
| echonest/version_3_2_1 | 0.5745 | 0.8739 |
| scheirer1998/percival2014 | 0.5387 | 0.7607 |
Table 4: Mean accuracy of estimates compared to version 2.0 with 4% tolerance ordered by Accuracy1.
Raw data Accuracy1: CSV JSON LATEX PICKLE
Raw data Accuracy2: CSV JSON LATEX PICKLE
Accuracy1 for 2.0
Figure 7: Mean Accuracy1 for estimates compared to version 2.0 depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for 2.0
Figure 8: Mean Accuracy2 for estimates compared to version 2.0 depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy Results for 2.0-no-dupes
| Estimator | Accuracy1 | Accuracy2 |
|---|---|---|
| boeck2020/dar | 0.9927 | 1.0000 |
| boeck2019/multi_task | 0.9854 | 1.0000 |
| boeck2019/multi_task_hjdb | 0.9839 | 0.9985 |
| sun2021/default | 0.9752 | 0.9883 |
| schreiber2018/cnn | 0.9679 | 0.9927 |
| schreiber2018/ismir2018 | 0.9460 | 0.9942 |
| schreiber2018/fcn | 0.9445 | 0.9898 |
| schreiber2017/mirex2017 | 0.9153 | 0.9883 |
| schreiber2017/ismir2017 | 0.8657 | 0.9839 |
| boeck2015/tempodetector2016_default | 0.8423 | 1.0000 |
| davies2009/mirex_qm_tempotracker | 0.6730 | 0.9226 |
| schreiber2014/default | 0.6672 | 0.9723 |
| zplane/auftakt_v3 | 0.6642 | 0.9518 |
| percival2014/stem | 0.6540 | 0.9547 |
| klapuri2006/percival2014 | 0.6423 | 0.9285 |
| oliveira2010/ibt | 0.6409 | 0.9051 |
| gkiokas2012/default | 0.6248 | 0.9810 |
| echonest/version_3_2_1 | 0.5693 | 0.8730 |
| scheirer1998/percival2014 | 0.5416 | 0.7635 |
Table 5: Mean accuracy of estimates compared to version 2.0-no-dupes with 4% tolerance ordered by Accuracy1.
Raw data Accuracy1: CSV JSON LATEX PICKLE
Raw data Accuracy2: CSV JSON LATEX PICKLE
Accuracy1 for 2.0-no-dupes
Figure 9: Mean Accuracy1 for estimates compared to version 2.0-no-dupes depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for 2.0-no-dupes
Figure 10: Mean Accuracy2 for estimates compared to version 2.0-no-dupes depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy Results for 3.0
| Estimator | Accuracy1 | Accuracy2 |
|---|---|---|
| sun2021/default | 0.9771 | 0.9900 |
| boeck2020/dar | 0.9742 | 0.9814 |
| schreiber2018/cnn | 0.9699 | 0.9928 |
| boeck2019/multi_task | 0.9670 | 0.9814 |
| boeck2019/multi_task_hjdb | 0.9656 | 0.9799 |
| schreiber2018/fcn | 0.9470 | 0.9914 |
| schreiber2018/ismir2018 | 0.9470 | 0.9943 |
| schreiber2017/mirex2017 | 0.9169 | 0.9885 |
| schreiber2017/ismir2017 | 0.8682 | 0.9842 |
| boeck2015/tempodetector2016_default | 0.8453 | 0.9986 |
| davies2009/mirex_qm_tempotracker | 0.6819 | 0.9284 |
| schreiber2014/default | 0.6719 | 0.9728 |
| zplane/auftakt_v3 | 0.6691 | 0.9527 |
| percival2014/stem | 0.6547 | 0.9527 |
| oliveira2010/ibt | 0.6461 | 0.9097 |
| klapuri2006/percival2014 | 0.6461 | 0.9284 |
| gkiokas2012/default | 0.6289 | 0.9799 |
| echonest/version_3_2_1 | 0.5745 | 0.8739 |
| scheirer1998/percival2014 | 0.5372 | 0.7579 |
Table 6: Mean accuracy of estimates compared to version 3.0 with 4% tolerance ordered by Accuracy1.
Raw data Accuracy1: CSV JSON LATEX PICKLE
Raw data Accuracy2: CSV JSON LATEX PICKLE
Accuracy1 for 3.0
Figure 11: Mean Accuracy1 for estimates compared to version 3.0 depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for 3.0
Figure 12: Mean Accuracy2 for estimates compared to version 3.0 depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy Results for 3.0-no-dupes
| Estimator | Accuracy1 | Accuracy2 |
|---|---|---|
| boeck2020/dar | 0.9927 | 1.0000 |
| boeck2019/multi_task | 0.9854 | 1.0000 |
| boeck2019/multi_task_hjdb | 0.9839 | 0.9985 |
| sun2021/default | 0.9781 | 0.9912 |
| schreiber2018/cnn | 0.9693 | 0.9927 |
| schreiber2018/fcn | 0.9460 | 0.9912 |
| schreiber2018/ismir2018 | 0.9460 | 0.9942 |
| schreiber2017/mirex2017 | 0.9153 | 0.9883 |
| schreiber2017/ismir2017 | 0.8657 | 0.9839 |
| boeck2015/tempodetector2016_default | 0.8423 | 0.9985 |
| davies2009/mirex_qm_tempotracker | 0.6788 | 0.9270 |
| schreiber2014/default | 0.6672 | 0.9723 |
| zplane/auftakt_v3 | 0.6657 | 0.9533 |
| percival2014/stem | 0.6511 | 0.9518 |
| oliveira2010/ibt | 0.6423 | 0.9080 |
| klapuri2006/percival2014 | 0.6423 | 0.9285 |
| gkiokas2012/default | 0.6234 | 0.9796 |
| echonest/version_3_2_1 | 0.5693 | 0.8730 |
| scheirer1998/percival2014 | 0.5401 | 0.7606 |
Table 7: Mean accuracy of estimates compared to version 3.0-no-dupes with 4% tolerance ordered by Accuracy1.
Raw data Accuracy1: CSV JSON LATEX PICKLE
Raw data Accuracy2: CSV JSON LATEX PICKLE
Accuracy1 for 3.0-no-dupes
Figure 13: Mean Accuracy1 for estimates compared to version 3.0-no-dupes depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for 3.0-no-dupes
Figure 14: Mean Accuracy2 for estimates compared to version 3.0-no-dupes depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy Results for 4.0
| Estimator | Accuracy1 | Accuracy2 |
|---|---|---|
| sun2021/default | 0.9599 | 0.9814 |
| boeck2020/dar | 0.9585 | 0.9756 |
| schreiber2018/cnn | 0.9570 | 0.9871 |
| boeck2019/multi_task | 0.9542 | 0.9756 |
| boeck2019/multi_task_hjdb | 0.9513 | 0.9756 |
| schreiber2018/fcn | 0.9341 | 0.9871 |
| schreiber2018/ismir2018 | 0.9327 | 0.9871 |
| schreiber2017/mirex2017 | 0.9054 | 0.9842 |
| schreiber2017/ismir2017 | 0.8539 | 0.9799 |
| boeck2015/tempodetector2016_default | 0.8324 | 0.9871 |
| davies2009/mirex_qm_tempotracker | 0.6762 | 0.9169 |
| zplane/auftakt_v3 | 0.6691 | 0.9484 |
| schreiber2014/default | 0.6676 | 0.9670 |
| percival2014/stem | 0.6562 | 0.9499 |
| klapuri2006/percival2014 | 0.6490 | 0.9284 |
| oliveira2010/ibt | 0.6433 | 0.9026 |
| gkiokas2012/default | 0.6318 | 0.9799 |
| echonest/version_3_2_1 | 0.5702 | 0.8653 |
| scheirer1998/percival2014 | 0.5372 | 0.7564 |
Table 8: Mean accuracy of estimates compared to version 4.0 with 4% tolerance ordered by Accuracy1.
Raw data Accuracy1: CSV JSON LATEX PICKLE
Raw data Accuracy2: CSV JSON LATEX PICKLE
Accuracy1 for 4.0
Figure 15: Mean Accuracy1 for estimates compared to version 4.0 depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for 4.0
Figure 16: Mean Accuracy2 for estimates compared to version 4.0 depending on tolerance.
CSV JSON LATEX PICKLE SVG PDF PNG
Differing Items
For which items did a given estimator not estimate a correct value with respect to a given ground truth? Are there items which are either very difficult, not suitable for the task, or incorrectly annotated and therefore never estimated correctly, regardless which estimator is used?
Differing Items Accuracy1
Items with different tempo annotations (Accuracy1, 4% tolerance) in different versions:
1.0 compared with boeck2015/tempodetector2016_default (136 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-13’ … CSV
1.0 compared with boeck2019/multi_task (44 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latin_Jam-13’ … CSV
1.0 compared with boeck2019/multi_task_hjdb (47 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latin_Jam-13’ … CSV
1.0 compared with boeck2020/dar (44 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Commitments-08’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ … CSV
1.0 compared with davies2009/mirex_qm_tempotracker (236 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ … CSV
1.0 compared with echonest/version_3_2_1 (313 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ … CSV
1.0 compared with gkiokas2012/default (279 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
1.0 compared with klapuri2006/percival2014 (263 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
1.0 compared with oliveira2010/ibt (265 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ … CSV
1.0 compared with percival2014/stem (260 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ … CSV
1.0 compared with scheirer1998/percival2014 (331 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ … CSV
1.0 compared with schreiber2014/default (249 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-09’ … CSV
1.0 compared with schreiber2017/ismir2017 (118 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ … CSV
1.0 compared with schreiber2017/mirex2017 (82 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-03’ … CSV
1.0 compared with schreiber2018/cnn (46 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ … CSV
1.0 compared with schreiber2018/fcn (59 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-11’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-Fire-13’ ‘Albums-GloriaEstefan_MiTierra-06’ … CSV
1.0 compared with schreiber2018/ismir2018 (61 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ ‘Albums-Fire-03’ … CSV
1.0 compared with sun2021/default (39 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Commitments-11’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-01’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ … CSV
1.0 compared with zplane/auftakt_v3 (250 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
2.0 compared with boeck2015/tempodetector2016_default (108 differences): ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ … CSV
2.0 compared with boeck2019/multi_task (23 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV
2.0 compared with boeck2019/multi_task_hjdb (24 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV
2.0 compared with boeck2020/dar (18 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Commitments-08’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV
2.0 compared with davies2009/mirex_qm_tempotracker (226 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-15’ … CSV
2.0 compared with echonest/version_3_2_1 (297 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
2.0 compared with gkiokas2012/default (258 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
2.0 compared with klapuri2006/percival2014 (247 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ … CSV
2.0 compared with oliveira2010/ibt (248 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
2.0 compared with percival2014/stem (239 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
2.0 compared with scheirer1998/percival2014 (322 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ … CSV
2.0 compared with schreiber2014/default (229 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-09’ ‘Albums-Ballroom_Magic-10’ … CSV
2.0 compared with schreiber2017/ismir2017 (92 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ … CSV
2.0 compared with schreiber2017/mirex2017 (58 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-07’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam-12’ … CSV
2.0 compared with schreiber2018/cnn (22 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Macumba-16’ ‘Media-103302’ ‘Media-103315’ ‘Media-103618’ ‘Media-103711’ ‘Media-103715’ ‘Media-103905’ … CSV
2.0 compared with schreiber2018/fcn (38 differences): ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam3-04’ ‘Albums-Latin_Jam3-05’ ‘Albums-Latin_Jam4-02’ ‘Albums-Macumba-16’ … CSV
2.0 compared with schreiber2018/ismir2018 (37 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ ‘Albums-Latin_Jam3-11’ ‘Albums-Latin_Jam3-12’ … CSV
2.0 compared with sun2021/default (18 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Secret_Garden-01’ ‘Media-100615’ ‘Media-103715’ ‘Media-103905’ ‘Media-104409’ ‘Media-104704’ ‘Media-104705’ ‘Media-105002’ ‘Media-105110’ … CSV
2.0 compared with zplane/auftakt_v3 (232 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
2.0-no-dupes compared with boeck2015/tempodetector2016_default (108 differences): ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ … CSV
2.0-no-dupes compared with boeck2019/multi_task (10 differences): ‘Albums-Chrisanne3-03’ ‘Albums-Mambo_Kings-10’ ‘Albums-Step_By_Step-16’ ‘Media-103606’ ‘Media-103614’ ‘Media-103715’ ‘Media-103905’ ‘Media-105207’ ‘Media-105403’ ‘Media-106009’ CSV
2.0-no-dupes compared with boeck2019/multi_task_hjdb (11 differences): ‘Albums-Chrisanne3-03’ ‘Albums-Mambo_Kings-10’ ‘Media-103606’ ‘Media-103715’ ‘Media-103905’ ‘Media-104418’ ‘Media-105207’ ‘Media-105302’ ‘Media-105403’ ‘Media-106009’ ‘Media-106118’ … CSV
2.0-no-dupes compared with boeck2020/dar (5 differences): ‘Albums-Commitments-08’ ‘Albums-Latin_Jam3-11’ ‘Media-103715’ ‘Media-104418’ ‘Media-105411’ CSV
2.0-no-dupes compared with davies2009/mirex_qm_tempotracker (224 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-15’ … CSV
2.0-no-dupes compared with echonest/version_3_2_1 (295 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
2.0-no-dupes compared with gkiokas2012/default (257 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
2.0-no-dupes compared with klapuri2006/percival2014 (245 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ … CSV
2.0-no-dupes compared with oliveira2010/ibt (246 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
2.0-no-dupes compared with percival2014/stem (237 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
2.0-no-dupes compared with scheirer1998/percival2014 (314 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ … CSV
2.0-no-dupes compared with schreiber2014/default (228 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-09’ ‘Albums-Ballroom_Magic-10’ … CSV
2.0-no-dupes compared with schreiber2017/ismir2017 (92 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ … CSV
2.0-no-dupes compared with schreiber2017/mirex2017 (58 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-07’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam-12’ … CSV
2.0-no-dupes compared with schreiber2018/cnn (22 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Macumba-16’ ‘Media-103302’ ‘Media-103315’ ‘Media-103618’ ‘Media-103711’ ‘Media-103715’ ‘Media-103905’ … CSV
2.0-no-dupes compared with schreiber2018/fcn (38 differences): ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam3-04’ ‘Albums-Latin_Jam3-05’ ‘Albums-Latin_Jam4-02’ ‘Albums-Macumba-16’ … CSV
2.0-no-dupes compared with schreiber2018/ismir2018 (37 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ ‘Albums-Latin_Jam3-11’ ‘Albums-Latin_Jam3-12’ … CSV
2.0-no-dupes compared with sun2021/default (17 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Secret_Garden-01’ ‘Media-100615’ ‘Media-103715’ ‘Media-103905’ ‘Media-104409’ ‘Media-104704’ ‘Media-105002’ ‘Media-105110’ ‘Media-105212’ … CSV
2.0-no-dupes compared with zplane/auftakt_v3 (230 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
3.0 compared with boeck2015/tempodetector2016_default (108 differences): ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ … CSV
3.0 compared with boeck2019/multi_task (23 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV
3.0 compared with boeck2019/multi_task_hjdb (24 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV
3.0 compared with boeck2020/dar (18 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Commitments-08’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV
3.0 compared with davies2009/mirex_qm_tempotracker (222 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-15’ … CSV
3.0 compared with echonest/version_3_2_1 (297 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
3.0 compared with gkiokas2012/default (259 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
3.0 compared with klapuri2006/percival2014 (247 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ … CSV
3.0 compared with oliveira2010/ibt (247 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
3.0 compared with percival2014/stem (241 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
3.0 compared with scheirer1998/percival2014 (323 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ … CSV
3.0 compared with schreiber2014/default (229 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-09’ ‘Albums-Ballroom_Magic-10’ … CSV
3.0 compared with schreiber2017/ismir2017 (92 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ … CSV
3.0 compared with schreiber2017/mirex2017 (58 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-07’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam-12’ … CSV
3.0 compared with schreiber2018/cnn (21 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Macumba-16’ ‘Media-103302’ ‘Media-103315’ ‘Media-103618’ ‘Media-103711’ ‘Media-103715’ ‘Media-103905’ … CSV
3.0 compared with schreiber2018/fcn (37 differences): ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam3-04’ ‘Albums-Latin_Jam3-05’ ‘Albums-Latin_Jam4-02’ ‘Albums-Macumba-16’ … CSV
3.0 compared with schreiber2018/ismir2018 (37 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ ‘Albums-Latin_Jam3-11’ ‘Albums-Latin_Jam3-12’ … CSV
3.0 compared with sun2021/default (16 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Secret_Garden-01’ ‘Media-100615’ ‘Media-103715’ ‘Media-103905’ ‘Media-104409’ ‘Media-104704’ ‘Media-104705’ ‘Media-105002’ ‘Media-105110’ … CSV
3.0 compared with zplane/auftakt_v3 (231 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
3.0-no-dupes compared with boeck2015/tempodetector2016_default (108 differences): ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ … CSV
3.0-no-dupes compared with boeck2019/multi_task (10 differences): ‘Albums-Chrisanne3-03’ ‘Albums-Mambo_Kings-10’ ‘Albums-Step_By_Step-16’ ‘Media-103606’ ‘Media-103614’ ‘Media-103715’ ‘Media-103905’ ‘Media-105207’ ‘Media-105403’ ‘Media-106009’ CSV
3.0-no-dupes compared with boeck2019/multi_task_hjdb (11 differences): ‘Albums-Chrisanne3-03’ ‘Albums-Mambo_Kings-10’ ‘Media-103606’ ‘Media-103715’ ‘Media-103905’ ‘Media-104418’ ‘Media-105207’ ‘Media-105302’ ‘Media-105403’ ‘Media-106009’ ‘Media-106118’ … CSV
3.0-no-dupes compared with boeck2020/dar (5 differences): ‘Albums-Commitments-08’ ‘Albums-Latin_Jam3-11’ ‘Media-103715’ ‘Media-104418’ ‘Media-105411’ CSV
3.0-no-dupes compared with davies2009/mirex_qm_tempotracker (220 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-15’ … CSV
3.0-no-dupes compared with echonest/version_3_2_1 (295 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
3.0-no-dupes compared with gkiokas2012/default (258 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
3.0-no-dupes compared with klapuri2006/percival2014 (245 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ … CSV
3.0-no-dupes compared with oliveira2010/ibt (245 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
3.0-no-dupes compared with percival2014/stem (239 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
3.0-no-dupes compared with scheirer1998/percival2014 (315 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ … CSV
3.0-no-dupes compared with schreiber2014/default (228 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-09’ ‘Albums-Ballroom_Magic-10’ … CSV
3.0-no-dupes compared with schreiber2017/ismir2017 (92 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ … CSV
3.0-no-dupes compared with schreiber2017/mirex2017 (58 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-07’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam-12’ … CSV
3.0-no-dupes compared with schreiber2018/cnn (21 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Macumba-16’ ‘Media-103302’ ‘Media-103315’ ‘Media-103618’ ‘Media-103711’ ‘Media-103715’ ‘Media-103905’ … CSV
3.0-no-dupes compared with schreiber2018/fcn (37 differences): ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam3-04’ ‘Albums-Latin_Jam3-05’ ‘Albums-Latin_Jam4-02’ ‘Albums-Macumba-16’ … CSV
3.0-no-dupes compared with schreiber2018/ismir2018 (37 differences): ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ ‘Albums-Latin_Jam3-11’ ‘Albums-Latin_Jam3-12’ … CSV
3.0-no-dupes compared with sun2021/default (15 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Secret_Garden-01’ ‘Media-100615’ ‘Media-103715’ ‘Media-103905’ ‘Media-104409’ ‘Media-104704’ ‘Media-105002’ ‘Media-105110’ ‘Media-105212’ … CSV
3.0-no-dupes compared with zplane/auftakt_v3 (229 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
4.0 compared with boeck2015/tempodetector2016_default (117 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-14’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-12’ … CSV
4.0 compared with boeck2019/multi_task (32 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ … CSV
4.0 compared with boeck2019/multi_task_hjdb (34 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ … CSV
4.0 compared with boeck2020/dar (29 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Commitments-08’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ … CSV
4.0 compared with davies2009/mirex_qm_tempotracker (226 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ … CSV
4.0 compared with echonest/version_3_2_1 (300 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ … CSV
4.0 compared with gkiokas2012/default (257 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-09’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ … CSV
4.0 compared with klapuri2006/percival2014 (245 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
4.0 compared with oliveira2010/ibt (249 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ … CSV
4.0 compared with percival2014/stem (240 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ … CSV
4.0 compared with scheirer1998/percival2014 (323 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-08’ ‘Albums-Ballroom_Classics4-11’ … CSV
4.0 compared with schreiber2014/default (232 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ ‘Albums-Ballroom_Magic-04’ … CSV
4.0 compared with schreiber2017/ismir2017 (102 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-16’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-13’ … CSV
4.0 compared with schreiber2017/mirex2017 (66 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-10’ ‘Albums-Fire-07’ … CSV
4.0 compared with schreiber2018/cnn (30 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Latino_Latino-03’ ‘Albums-Macumba-16’ ‘Albums-Secret_Garden-05’ ‘Albums-Step_By_Step-15’ ‘Albums-Step_By_Step-16’ … CSV
4.0 compared with schreiber2018/fcn (46 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Cafe_Paradiso-02’ ‘Albums-Chrisanne2-11’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-11’ ‘Albums-Fire-13’ ‘Albums-Latin_Jam3-04’ ‘Albums-Latin_Jam3-05’ ‘Albums-Latin_Jam4-02’ … CSV
4.0 compared with schreiber2018/ismir2018 (47 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-15’ ‘Albums-Ballroom_Magic-17’ ‘Albums-Cafe_Paradiso-15’ ‘Albums-Chrisanne2-07’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-08’ ‘Albums-Chrisanne3-09’ ‘Albums-Chrisanne3-15’ … CSV
4.0 compared with sun2021/default (28 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-08’ ‘Albums-Commitments-11’ ‘Albums-GloriaEstefan_MiTierra-01’ ‘Albums-Latino_Latino-03’ ‘Albums-Latino_Latino-06’ ‘Albums-Secret_Garden-01’ ‘Albums-Step_By_Step-15’ ‘Albums-Step_By_Step-16’ … CSV
4.0 compared with zplane/auftakt_v3 (231 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-12’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Classics4-14’ ‘Albums-Ballroom_Classics4-18’ ‘Albums-Ballroom_Classics4-19’ ‘Albums-Ballroom_Classics4-20’ … CSV
All tracks were estimated ‘correctly’ by at least one system.
Differing Items Accuracy2
Items with different tempo annotations (Accuracy2, 4% tolerance) in different versions:
1.0 compared with boeck2015/tempodetector2016_default (32 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-05’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Secret_Garden-01’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-05’ … CSV
1.0 compared with boeck2019/multi_task (35 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latin_Jam-13’ … CSV
1.0 compared with boeck2019/multi_task_hjdb (37 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latin_Jam-13’ … CSV
1.0 compared with boeck2020/dar (38 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Chrisanne3-03’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latin_Jam-13’ … CSV
1.0 compared with davies2009/mirex_qm_tempotracker (72 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ … CSV
1.0 compared with echonest/version_3_2_1 (111 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Ballroom_Magic-18’ … CSV
1.0 compared with gkiokas2012/default (39 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-15’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ … CSV
1.0 compared with klapuri2006/percival2014 (69 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ … CSV
1.0 compared with oliveira2010/ibt (85 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ … CSV
1.0 compared with percival2014/stem (56 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ … CSV
1.0 compared with scheirer1998/percival2014 (184 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ … CSV
1.0 compared with schreiber2014/default (43 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-Fire-13’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ … CSV
1.0 compared with schreiber2017/ismir2017 (36 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-Fire-13’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ … CSV
1.0 compared with schreiber2017/mirex2017 (33 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-Fire-13’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Secret_Garden-01’ … CSV
1.0 compared with schreiber2018/cnn (30 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Secret_Garden-01’ ‘Albums-Secret_Garden-02’ … CSV
1.0 compared with schreiber2018/fcn (27 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-11’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-Fire-13’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Secret_Garden-01’ … CSV
1.0 compared with schreiber2018/ismir2018 (28 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Secret_Garden-01’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-05’ … CSV
1.0 compared with sun2021/default (29 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-11’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-01’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ ‘Albums-GloriaEstefan_MiTierra-11’ ‘Albums-Latino_Latino-06’ … CSV
1.0 compared with zplane/auftakt_v3 (55 differences): ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-03’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-03’ ‘Albums-Fire-09’ ‘Albums-GloriaEstefan_MiTierra-06’ ‘Albums-GloriaEstefan_MiTierra-08’ … CSV
2.0 compared with boeck2015/tempodetector2016_default: No differences.
2.0 compared with boeck2019/multi_task (13 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV
2.0 compared with boeck2019/multi_task_hjdb (14 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV
2.0 compared with boeck2020/dar (13 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV
2.0 compared with davies2009/mirex_qm_tempotracker (53 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Latino_Latino-03’ … CSV
2.0 compared with echonest/version_3_2_1 (88 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-03’ … CSV
2.0 compared with gkiokas2012/default (13 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne3-15’ ‘Albums-Latino_Latino-03’ ‘Albums-Secret_Garden-02’ ‘Media-103710’ ‘Media-103905’ ‘Media-104711’ ‘Media-105002’ ‘Media-105007’ ‘Media-105111’ … CSV
2.0 compared with klapuri2006/percival2014 (50 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ … CSV
2.0 compared with oliveira2010/ibt (65 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ … CSV
2.0 compared with percival2014/stem (31 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100604’ ‘Media-100609’ … CSV
2.0 compared with scheirer1998/percival2014 (167 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ … CSV
2.0 compared with schreiber2014/default (19 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103308’ ‘Media-103315’ ‘Media-104302’ ‘Media-104608’ ‘Media-104703’ ‘Media-104811’ … CSV
2.0 compared with schreiber2017/ismir2017 (11 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-104303’ ‘Media-104608’ ‘Media-105004’ ‘Media-105420’ ‘Media-105701’ ‘Media-105702’ … CSV
2.0 compared with schreiber2017/mirex2017 (8 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103315’ ‘Media-104303’ ‘Media-104608’ ‘Media-105214’ ‘Media-105420’ CSV
2.0 compared with schreiber2018/cnn (5 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103315’ ‘Media-104905’ ‘Media-105302’ CSV
2.0 compared with schreiber2018/fcn (7 differences): ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103905’ ‘Media-105002’ ‘Media-105302’ CSV
2.0 compared with schreiber2018/ismir2018 (4 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103905’ ‘Media-105911’ CSV
2.0 compared with sun2021/default (9 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Secret_Garden-01’ ‘Media-103905’ ‘Media-104704’ ‘Media-104705’ ‘Media-105002’ ‘Media-105212’ ‘Media-105215’ ‘Media-105302’ CSV
2.0 compared with zplane/auftakt_v3 (34 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100603’ … CSV
2.0-no-dupes compared with boeck2015/tempodetector2016_default: No differences.
2.0-no-dupes compared with boeck2019/multi_task: No differences.
2.0-no-dupes compared with boeck2019/multi_task_hjdb (1 differences): ‘Media-105302’ CSV
2.0-no-dupes compared with boeck2020/dar: No differences.
2.0-no-dupes compared with davies2009/mirex_qm_tempotracker (53 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Latino_Latino-03’ … CSV
2.0-no-dupes compared with echonest/version_3_2_1 (87 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne1-07’ … CSV
2.0-no-dupes compared with gkiokas2012/default (13 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne3-15’ ‘Albums-Latino_Latino-03’ ‘Albums-Secret_Garden-02’ ‘Media-103710’ ‘Media-103905’ ‘Media-104711’ ‘Media-105002’ ‘Media-105007’ ‘Media-105111’ … CSV
2.0-no-dupes compared with klapuri2006/percival2014 (49 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ … CSV
2.0-no-dupes compared with oliveira2010/ibt (65 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ … CSV
2.0-no-dupes compared with percival2014/stem (31 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100604’ ‘Media-100609’ … CSV
2.0-no-dupes compared with scheirer1998/percival2014 (162 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-10’ … CSV
2.0-no-dupes compared with schreiber2014/default (19 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103308’ ‘Media-103315’ ‘Media-104302’ ‘Media-104608’ ‘Media-104703’ ‘Media-104811’ … CSV
2.0-no-dupes compared with schreiber2017/ismir2017 (11 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-104303’ ‘Media-104608’ ‘Media-105004’ ‘Media-105420’ ‘Media-105701’ ‘Media-105702’ … CSV
2.0-no-dupes compared with schreiber2017/mirex2017 (8 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103315’ ‘Media-104303’ ‘Media-104608’ ‘Media-105214’ ‘Media-105420’ CSV
2.0-no-dupes compared with schreiber2018/cnn (5 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103315’ ‘Media-104905’ ‘Media-105302’ CSV
2.0-no-dupes compared with schreiber2018/fcn (7 differences): ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103905’ ‘Media-105002’ ‘Media-105302’ CSV
2.0-no-dupes compared with schreiber2018/ismir2018 (4 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103905’ ‘Media-105911’ CSV
2.0-no-dupes compared with sun2021/default (8 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Secret_Garden-01’ ‘Media-103905’ ‘Media-104704’ ‘Media-105002’ ‘Media-105212’ ‘Media-105215’ ‘Media-105302’ CSV
2.0-no-dupes compared with zplane/auftakt_v3 (33 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100603’ … CSV
3.0 compared with boeck2015/tempodetector2016_default (1 differences): ‘Media-103905’ CSV
3.0 compared with boeck2019/multi_task (13 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV
3.0 compared with boeck2019/multi_task_hjdb (14 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV
3.0 compared with boeck2020/dar (13 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ ‘Media-103414’ … CSV
3.0 compared with davies2009/mirex_qm_tempotracker (50 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Latino_Latino-03’ … CSV
3.0 compared with echonest/version_3_2_1 (88 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-03’ … CSV
3.0 compared with gkiokas2012/default (14 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne3-15’ ‘Albums-Latino_Latino-03’ ‘Albums-Secret_Garden-02’ ‘Media-103710’ ‘Media-103905’ ‘Media-104711’ ‘Media-105002’ ‘Media-105007’ … CSV
3.0 compared with klapuri2006/percival2014 (50 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ … CSV
3.0 compared with oliveira2010/ibt (63 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ … CSV
3.0 compared with percival2014/stem (33 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100604’ ‘Media-100609’ … CSV
3.0 compared with scheirer1998/percival2014 (169 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ … CSV
3.0 compared with schreiber2014/default (19 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103308’ ‘Media-103315’ ‘Media-104302’ ‘Media-104608’ ‘Media-104703’ ‘Media-104811’ … CSV
3.0 compared with schreiber2017/ismir2017 (11 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-104303’ ‘Media-104608’ ‘Media-105004’ ‘Media-105420’ ‘Media-105701’ ‘Media-105702’ … CSV
3.0 compared with schreiber2017/mirex2017 (8 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103315’ ‘Media-104303’ ‘Media-104608’ ‘Media-105214’ ‘Media-105420’ CSV
3.0 compared with schreiber2018/cnn (5 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103315’ ‘Media-103905’ ‘Media-104905’ CSV
3.0 compared with schreiber2018/fcn (6 differences): ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103905’ ‘Media-105002’ CSV
3.0 compared with schreiber2018/ismir2018 (4 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103905’ ‘Media-105911’ CSV
3.0 compared with sun2021/default (7 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Secret_Garden-01’ ‘Media-103905’ ‘Media-104704’ ‘Media-104705’ ‘Media-105002’ ‘Media-105212’ CSV
3.0 compared with zplane/auftakt_v3 (33 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100603’ … CSV
3.0-no-dupes compared with boeck2015/tempodetector2016_default (1 differences): ‘Media-103905’ CSV
3.0-no-dupes compared with boeck2019/multi_task: No differences.
3.0-no-dupes compared with boeck2019/multi_task_hjdb (1 differences): ‘Media-105302’ CSV
3.0-no-dupes compared with boeck2020/dar: No differences.
3.0-no-dupes compared with davies2009/mirex_qm_tempotracker (50 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Latino_Latino-03’ … CSV
3.0-no-dupes compared with echonest/version_3_2_1 (87 differences): ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne1-07’ … CSV
3.0-no-dupes compared with gkiokas2012/default (14 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne3-15’ ‘Albums-Latino_Latino-03’ ‘Albums-Secret_Garden-02’ ‘Media-103710’ ‘Media-103905’ ‘Media-104711’ ‘Media-105002’ ‘Media-105007’ … CSV
3.0-no-dupes compared with klapuri2006/percival2014 (49 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ … CSV
3.0-no-dupes compared with oliveira2010/ibt (63 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-03’ … CSV
3.0-no-dupes compared with percival2014/stem (33 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100604’ ‘Media-100609’ … CSV
3.0-no-dupes compared with scheirer1998/percival2014 (164 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-10’ … CSV
3.0-no-dupes compared with schreiber2014/default (19 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103308’ ‘Media-103315’ ‘Media-104302’ ‘Media-104608’ ‘Media-104703’ ‘Media-104811’ … CSV
3.0-no-dupes compared with schreiber2017/ismir2017 (11 differences): ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-104303’ ‘Media-104608’ ‘Media-105004’ ‘Media-105420’ ‘Media-105701’ ‘Media-105702’ … CSV
3.0-no-dupes compared with schreiber2017/mirex2017 (8 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103315’ ‘Media-104303’ ‘Media-104608’ ‘Media-105214’ ‘Media-105420’ CSV
3.0-no-dupes compared with schreiber2018/cnn (5 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103315’ ‘Media-103905’ ‘Media-104905’ CSV
3.0-no-dupes compared with schreiber2018/fcn (6 differences): ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Media-103905’ ‘Media-105002’ CSV
3.0-no-dupes compared with schreiber2018/ismir2018 (4 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Media-103905’ ‘Media-105911’ CSV
3.0-no-dupes compared with sun2021/default (6 differences): ‘Albums-Chrisanne3-07’ ‘Albums-Secret_Garden-01’ ‘Media-103905’ ‘Media-104704’ ‘Media-105002’ ‘Media-105212’ CSV
3.0-no-dupes compared with zplane/auftakt_v3 (32 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ ‘Media-100603’ … CSV
4.0 compared with boeck2015/tempodetector2016_default (9 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-05’ ‘Albums-Secret_Garden-05’ ‘Albums-StrictlyDancing_Tango-08’ ‘Media-103905’ ‘Media-104404’ ‘Media-104908’ ‘Media-105007’ ‘Media-105101’ CSV
4.0 compared with boeck2019/multi_task (17 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV
4.0 compared with boeck2019/multi_task_hjdb (17 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV
4.0 compared with boeck2020/dar (17 differences): ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne2-12’ ‘Albums-Fire-09’ ‘Albums-Latin_Jam-13’ ‘Albums-Latin_Jam-14’ ‘Albums-Latin_Jam-15’ ‘Albums-Latin_Jam2-13’ ‘Albums-Latin_Jam2-14’ ‘Albums-Latin_Jam2-15’ … CSV
4.0 compared with davies2009/mirex_qm_tempotracker (58 differences): ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ ‘Albums-Chrisanne3-02’ ‘Albums-Chrisanne3-07’ … CSV
4.0 compared with echonest/version_3_2_1 (94 differences): ‘Albums-AnaBelen_Veneo-11’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Ballroom_Magic-07’ ‘Albums-Ballroom_Magic-18’ ‘Albums-Chrisanne1-01’ … CSV
4.0 compared with gkiokas2012/default (14 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne1-03’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne3-15’ ‘Albums-Latino_Latino-03’ ‘Albums-Secret_Garden-05’ ‘Media-103710’ ‘Media-103905’ ‘Media-103911’ ‘Media-104601’ ‘Media-104711’ … CSV
4.0 compared with klapuri2006/percival2014 (50 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-11’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-01’ ‘Albums-Chrisanne1-02’ ‘Albums-Chrisanne1-08’ ‘Albums-Chrisanne2-01’ ‘Albums-Chrisanne2-03’ … CSV
4.0 compared with oliveira2010/ibt (68 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-13’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Cafe_Paradiso-16’ ‘Albums-Chrisanne1-12’ ‘Albums-Chrisanne1-14’ … CSV
4.0 compared with percival2014/stem (35 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ ‘Albums-Chrisanne1-13’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-05’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ … CSV
4.0 compared with scheirer1998/percival2014 (170 differences): ‘Albums-AnaBelen_Veneo-03’ ‘Albums-AnaBelen_Veneo-15’ ‘Albums-Ballroom_Classics4-01’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Classics4-05’ ‘Albums-Ballroom_Classics4-07’ ‘Albums-Ballroom_Classics4-13’ ‘Albums-Ballroom_Magic-01’ ‘Albums-Ballroom_Magic-02’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Ballroom_Magic-05’ … CSV
4.0 compared with schreiber2014/default (23 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Secret_Garden-05’ ‘Media-103308’ ‘Media-103315’ ‘Media-104302’ ‘Media-104608’ … CSV
4.0 compared with schreiber2017/ismir2017 (14 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Secret_Garden-05’ ‘Media-104303’ ‘Media-104608’ ‘Media-105004’ ‘Media-105007’ … CSV
4.0 compared with schreiber2017/mirex2017 (11 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Fire-13’ ‘Albums-Secret_Garden-05’ ‘Media-103315’ ‘Media-104303’ ‘Media-104608’ ‘Media-105007’ ‘Media-105214’ ‘Media-105420’ … CSV
4.0 compared with schreiber2018/cnn (9 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-05’ ‘Media-103315’ ‘Media-103709’ ‘Media-103905’ ‘Media-104905’ ‘Media-105007’ CSV
4.0 compared with schreiber2018/fcn (9 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-09’ ‘Albums-Commitments-11’ ‘Albums-Fire-13’ ‘Albums-Secret_Garden-05’ ‘Media-103905’ ‘Media-103911’ ‘Media-105007’ ‘Media-105211’ CSV
4.0 compared with schreiber2018/ismir2018 (9 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-05’ ‘Albums-StrictlyDancing_Tango-11’ ‘Media-103905’ ‘Media-103911’ ‘Media-105007’ ‘Media-105911’ CSV
4.0 compared with sun2021/default (13 differences): ‘Albums-Ballroom_Classics4-03’ ‘Albums-Chrisanne3-07’ ‘Albums-Commitments-11’ ‘Albums-GloriaEstefan_MiTierra-01’ ‘Albums-Latino_Latino-06’ ‘Albums-Secret_Garden-01’ ‘Media-103709’ ‘Media-103905’ ‘Media-103911’ ‘Media-104901’ ‘Media-105007’ … CSV
4.0 compared with zplane/auftakt_v3 (36 differences): ‘Albums-AnaBelen_Veneo-02’ ‘Albums-Ballroom_Classics4-02’ ‘Albums-Ballroom_Classics4-03’ ‘Albums-Ballroom_Magic-04’ ‘Albums-Chrisanne1-14’ ‘Albums-Chrisanne3-01’ ‘Albums-Chrisanne3-09’ ‘Albums-Secret_Garden-02’ ‘Albums-Secret_Garden-05’ ‘Albums-Secret_Garden-06’ ‘Albums-Step_By_Step-04’ … CSV
All tracks were estimated ‘correctly’ by at least one system.
Significance of Differences
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1176 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.3750 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8776 | 0.0534 | 0.0300 | 0.5224 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.3750 | 1.0000 | 0.6291 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 | 0.1263 | 0.0869 | 0.2682 | 0.0000 |
| boeck2020/dar | 0.0000 | 1.0000 | 0.6291 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8746 | 0.0534 | 0.0300 | 0.4996 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0009 | 0.0120 | 0.0045 | 0.0437 | 0.0000 | 0.3223 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2314 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0144 | 0.0001 | 0.0002 | 0.0000 | 0.2664 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0009 | 0.0144 | 1.0000 | 0.1058 | 0.1800 | 0.0295 | 0.0012 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0021 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0120 | 0.0001 | 0.1058 | 1.0000 | 0.8776 | 0.7838 | 0.0000 | 0.0925 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0596 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0045 | 0.0002 | 0.1800 | 0.8776 | 1.0000 | 0.6085 | 0.0000 | 0.0894 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0489 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0437 | 0.0000 | 0.0295 | 0.7838 | 0.6085 | 1.0000 | 0.0000 | 0.1925 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2203 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2664 | 0.0012 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.3223 | 0.0000 | 0.0005 | 0.0925 | 0.0894 | 0.1925 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
| schreiber2017/ismir2017 | 0.1176 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0095 | 0.0170 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.8776 | 1.0000 | 0.8746 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0660 | 0.0237 | 0.2810 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0534 | 0.1263 | 0.0534 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0095 | 0.0660 | 1.0000 | 0.8776 | 0.0055 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0300 | 0.0869 | 0.0300 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0170 | 0.0237 | 0.8776 | 1.0000 | 0.0026 | 0.0000 |
| sun2021/default | 0.0000 | 0.5224 | 0.2682 | 0.4996 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2810 | 0.0055 | 0.0026 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2314 | 0.0000 | 0.0021 | 0.0596 | 0.0489 | 0.2203 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 9: McNemar p-values, using reference annotations 1.0 as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1637 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 1.0000 | 0.2668 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.8746 | 0.0759 | 0.0704 | 0.2962 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 1.0000 | 1.0000 | 0.1460 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.7493 | 0.0919 | 0.0984 | 0.2153 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.2668 | 0.1460 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7428 | 0.0110 | 0.0110 | 0.8506 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0059 | 0.0215 | 0.0180 | 0.1186 | 0.0000 | 0.6320 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4812 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0067 | 0.0001 | 0.0001 | 0.0000 | 0.1037 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0059 | 0.0067 | 1.0000 | 0.2410 | 0.2566 | 0.0356 | 0.0001 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0029 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0215 | 0.0001 | 0.2410 | 1.0000 | 1.0000 | 0.5044 | 0.0000 | 0.0385 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0226 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0180 | 0.0001 | 0.2566 | 1.0000 | 1.0000 | 0.5115 | 0.0000 | 0.0512 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0365 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1186 | 0.0000 | 0.0356 | 0.5044 | 0.5115 | 1.0000 | 0.0000 | 0.1619 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2203 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1037 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.6320 | 0.0000 | 0.0005 | 0.0385 | 0.0512 | 0.1619 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8955 |
| schreiber2017/ismir2017 | 0.1637 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0001 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0170 | 0.0154 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.8746 | 0.7493 | 0.7428 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0195 | 0.0090 | 0.4421 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0759 | 0.0919 | 0.0110 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0170 | 0.0195 | 1.0000 | 1.0000 | 0.0025 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0704 | 0.0984 | 0.0110 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0154 | 0.0090 | 1.0000 | 1.0000 | 0.0019 | 0.0000 |
| sun2021/default | 0.0000 | 0.2962 | 0.2153 | 0.8506 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4421 | 0.0025 | 0.0019 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4812 | 0.0000 | 0.0029 | 0.0226 | 0.0365 | 0.2203 | 0.0000 | 0.8955 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 10: McNemar p-values, using reference annotations 3.0 as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1637 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 1.0000 | 0.2668 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0522 | 0.0000 | 0.0000 | 0.3833 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 1.0000 | 1.0000 | 0.1460 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0755 | 0.0000 | 0.0000 | 0.5034 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.2668 | 0.1460 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0015 | 0.0000 | 0.0000 | 0.0213 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0045 | 0.0215 | 0.0180 | 0.1158 | 0.0000 | 0.5752 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4812 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0081 | 0.0001 | 0.0001 | 0.0000 | 0.2083 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0045 | 0.0081 | 1.0000 | 0.1983 | 0.2134 | 0.0248 | 0.0003 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0019 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0215 | 0.0001 | 0.1983 | 1.0000 | 1.0000 | 0.4966 | 0.0000 | 0.0498 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0226 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0180 | 0.0001 | 0.2134 | 1.0000 | 1.0000 | 0.5044 | 0.0000 | 0.0647 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0365 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1158 | 0.0000 | 0.0248 | 0.4966 | 0.5044 | 1.0000 | 0.0000 | 0.2000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2116 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2083 | 0.0003 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5752 | 0.0000 | 0.0005 | 0.0498 | 0.0647 | 0.2000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
| schreiber2017/ismir2017 | 0.1637 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0170 | 0.0154 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.0522 | 0.0755 | 0.0015 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0195 | 0.0090 | 0.3269 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0170 | 0.0195 | 1.0000 | 1.0000 | 0.0013 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0154 | 0.0090 | 1.0000 | 1.0000 | 0.0009 | 0.0000 |
| sun2021/default | 0.0000 | 0.3833 | 0.5034 | 0.0213 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.3269 | 0.0013 | 0.0009 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4812 | 0.0000 | 0.0019 | 0.0226 | 0.0365 | 0.2116 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 11: McNemar p-values, using reference annotations 3.0-no-dupes as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1637 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 1.0000 | 0.2668 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 | 0.0581 | 0.0704 | 0.4996 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 1.0000 | 1.0000 | 0.1460 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.8714 | 0.0649 | 0.0984 | 0.3771 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.2668 | 0.1460 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.6271 | 0.0078 | 0.0110 | 1.0000 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0174 | 0.0554 | 0.0357 | 0.2981 | 0.0000 | 0.8732 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.6610 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0055 | 0.0001 | 0.0001 | 0.0000 | 0.1190 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0174 | 0.0055 | 1.0000 | 0.2891 | 0.3481 | 0.0248 | 0.0001 | 0.0008 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0058 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0554 | 0.0001 | 0.2891 | 1.0000 | 1.0000 | 0.3409 | 0.0000 | 0.0385 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0357 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0357 | 0.0001 | 0.3481 | 1.0000 | 1.0000 | 0.2976 | 0.0000 | 0.0402 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0365 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2981 | 0.0000 | 0.0248 | 0.3409 | 0.2976 | 1.0000 | 0.0000 | 0.2451 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4188 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1190 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8732 | 0.0000 | 0.0008 | 0.0385 | 0.0402 | 0.2451 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7946 |
| schreiber2017/ismir2017 | 0.1637 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0001 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0245 | 0.0154 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 1.0000 | 0.8714 | 0.6271 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0195 | 0.0167 | 0.5716 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0581 | 0.0649 | 0.0078 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0245 | 0.0195 | 1.0000 | 1.0000 | 0.0045 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0704 | 0.0984 | 0.0110 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0154 | 0.0167 | 1.0000 | 1.0000 | 0.0066 | 0.0000 |
| sun2021/default | 0.0000 | 0.4996 | 0.3771 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5716 | 0.0045 | 0.0066 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.6610 | 0.0000 | 0.0058 | 0.0357 | 0.0365 | 0.4188 | 0.0000 | 0.7946 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 12: McNemar p-values, using reference annotations 2.0 as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1637 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 1.0000 | 0.2668 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0357 | 0.0000 | 0.0000 | 0.2100 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 1.0000 | 1.0000 | 0.1460 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0433 | 0.0000 | 0.0000 | 0.2632 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.2668 | 0.1460 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0009 | 0.0000 | 0.0000 | 0.0075 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0138 | 0.0554 | 0.0357 | 0.2944 | 0.0000 | 0.8102 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.6610 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0067 | 0.0001 | 0.0001 | 0.0000 | 0.2343 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0138 | 0.0067 | 1.0000 | 0.2410 | 0.2945 | 0.0169 | 0.0003 | 0.0008 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0039 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0554 | 0.0001 | 0.2410 | 1.0000 | 1.0000 | 0.3317 | 0.0000 | 0.0498 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0357 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0357 | 0.0001 | 0.2945 | 1.0000 | 1.0000 | 0.2892 | 0.0000 | 0.0512 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0365 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2944 | 0.0000 | 0.0169 | 0.3317 | 0.2892 | 1.0000 | 0.0000 | 0.2976 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4101 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2343 | 0.0003 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8102 | 0.0000 | 0.0008 | 0.0498 | 0.0512 | 0.2976 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8955 |
| schreiber2017/ismir2017 | 0.1637 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0245 | 0.0154 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.0357 | 0.0433 | 0.0009 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0195 | 0.0167 | 0.4421 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0245 | 0.0195 | 1.0000 | 1.0000 | 0.0025 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0154 | 0.0167 | 1.0000 | 1.0000 | 0.0037 | 0.0000 |
| sun2021/default | 0.0000 | 0.2100 | 0.2632 | 0.0075 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4421 | 0.0025 | 0.0037 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.6610 | 0.0000 | 0.0039 | 0.0357 | 0.0365 | 0.4101 | 0.0000 | 0.8955 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 13: McNemar p-values, using reference annotations 2.0-no-dupes as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2031 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.6250 | 0.5811 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.8776 | 0.0759 | 0.0581 | 0.6358 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.6250 | 1.0000 | 0.2266 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0004 | 0.6358 | 0.1189 | 0.1048 | 0.4177 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.5811 | 0.2266 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0213 | 0.0175 | 1.0000 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0195 | 0.0818 | 0.0286 | 0.2578 | 0.0000 | 0.6851 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7183 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0020 | 0.0000 | 0.0001 | 0.0000 | 0.1478 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0195 | 0.0020 | 1.0000 | 0.2410 | 0.4750 | 0.0498 | 0.0000 | 0.0035 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0064 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0818 | 0.0000 | 0.2410 | 1.0000 | 0.6358 | 0.5901 | 0.0000 | 0.1299 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0436 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0286 | 0.0001 | 0.4750 | 0.6358 | 1.0000 | 0.2976 | 0.0000 | 0.0647 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0175 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2578 | 0.0000 | 0.0498 | 0.5901 | 0.2976 | 1.0000 | 0.0000 | 0.3581 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2624 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1478 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.6851 | 0.0000 | 0.0035 | 0.1299 | 0.0647 | 0.3581 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
| schreiber2017/ismir2017 | 0.2031 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0001 | 0.0004 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0245 | 0.0319 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.8776 | 0.6358 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0195 | 0.0076 | 0.8555 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0759 | 0.1189 | 0.0213 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0245 | 0.0195 | 1.0000 | 1.0000 | 0.0114 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0581 | 0.1048 | 0.0175 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0319 | 0.0076 | 1.0000 | 1.0000 | 0.0079 | 0.0000 |
| sun2021/default | 0.0000 | 0.6358 | 0.4177 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8555 | 0.0114 | 0.0079 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7183 | 0.0000 | 0.0064 | 0.0436 | 0.0175 | 0.2624 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 14: McNemar p-values, using reference annotations 4.0 as groundtruth with Accuracy1 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.6776 | 0.4049 | 0.2632 | 0.0000 | 0.0000 | 0.1671 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0433 | 0.4807 | 1.0000 | 0.8036 | 0.2668 | 0.4240 | 0.6636 | 0.0002 |
| boeck2019/multi_task | 0.6776 | 1.0000 | 0.5000 | 0.4531 | 0.0000 | 0.0000 | 0.5847 | 0.0000 | 0.0000 | 0.0031 | 0.0000 | 0.2295 | 1.0000 | 0.8388 | 0.4049 | 0.1153 | 0.2100 | 0.3269 | 0.0037 |
| boeck2019/multi_task_hjdb | 0.4049 | 0.5000 | 1.0000 | 1.0000 | 0.0001 | 0.0000 | 0.8555 | 0.0000 | 0.0000 | 0.0079 | 0.0000 | 0.3915 | 1.0000 | 0.5413 | 0.2100 | 0.0525 | 0.1078 | 0.1849 | 0.0096 |
| boeck2020/dar | 0.2632 | 0.4531 | 1.0000 | 1.0000 | 0.0001 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0133 | 0.0000 | 0.5114 | 0.8506 | 0.4049 | 0.1338 | 0.0266 | 0.0525 | 0.1078 | 0.0137 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0001 | 0.0001 | 1.0000 | 0.0000 | 0.0000 | 0.8041 | 0.1237 | 0.0365 | 0.0000 | 0.0004 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0363 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0034 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.1671 | 0.5847 | 0.8555 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0115 | 0.0000 | 0.6177 | 0.6900 | 0.2863 | 0.0931 | 0.0075 | 0.0127 | 0.0639 | 0.0166 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8041 | 0.0000 | 0.0000 | 1.0000 | 0.0402 | 0.0854 | 0.0000 | 0.0003 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0436 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1237 | 0.0034 | 0.0000 | 0.0402 | 1.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0002 |
| percival2014/stem | 0.0001 | 0.0031 | 0.0079 | 0.0133 | 0.0365 | 0.0000 | 0.0115 | 0.0854 | 0.0002 | 1.0000 | 0.0000 | 0.0533 | 0.0017 | 0.0003 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0433 | 0.2295 | 0.3915 | 0.5114 | 0.0004 | 0.0000 | 0.6177 | 0.0003 | 0.0000 | 0.0533 | 0.0000 | 1.0000 | 0.0923 | 0.0309 | 0.0106 | 0.0015 | 0.0059 | 0.0201 | 0.0961 |
| schreiber2017/ismir2017 | 0.4807 | 1.0000 | 1.0000 | 0.8506 | 0.0000 | 0.0000 | 0.6900 | 0.0000 | 0.0000 | 0.0017 | 0.0000 | 0.0923 | 1.0000 | 0.4531 | 0.2379 | 0.0352 | 0.0768 | 0.2295 | 0.0034 |
| schreiber2017/mirex2017 | 1.0000 | 0.8388 | 0.5413 | 0.4049 | 0.0000 | 0.0000 | 0.2863 | 0.0000 | 0.0000 | 0.0003 | 0.0000 | 0.0309 | 0.4531 | 1.0000 | 0.5488 | 0.1460 | 0.2668 | 0.5034 | 0.0005 |
| schreiber2018/cnn | 0.8036 | 0.4049 | 0.2100 | 0.1338 | 0.0000 | 0.0000 | 0.0931 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0106 | 0.2379 | 0.5488 | 1.0000 | 0.5078 | 0.7539 | 1.0000 | 0.0000 |
| schreiber2018/fcn | 0.2668 | 0.1153 | 0.0525 | 0.0266 | 0.0000 | 0.0000 | 0.0075 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0015 | 0.0352 | 0.1460 | 0.5078 | 1.0000 | 1.0000 | 0.7744 | 0.0000 |
| schreiber2018/ismir2018 | 0.4240 | 0.2100 | 0.1078 | 0.0525 | 0.0000 | 0.0000 | 0.0127 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0059 | 0.0768 | 0.2668 | 0.7539 | 1.0000 | 1.0000 | 1.0000 | 0.0000 |
| sun2021/default | 0.6636 | 0.3269 | 0.1849 | 0.1078 | 0.0000 | 0.0000 | 0.0639 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0201 | 0.2295 | 0.5034 | 1.0000 | 0.7744 | 1.0000 | 1.0000 | 0.0001 |
| zplane/auftakt_v3 | 0.0002 | 0.0037 | 0.0096 | 0.0137 | 0.0363 | 0.0000 | 0.0166 | 0.0436 | 0.0002 | 1.0000 | 0.0000 | 0.0961 | 0.0034 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 |
Table 15: McNemar p-values, using reference annotations 1.0 as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0018 | 0.0010 | 0.0018 | 0.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0063 | 0.0391 | 0.1250 | 0.0625 | 0.2500 | 0.0312 | 0.0000 |
| boeck2019/multi_task | 0.0018 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0045 | 0.0000 | 0.3771 | 0.8388 | 0.3833 | 0.0963 | 0.1671 | 0.0490 | 0.2379 | 0.0037 |
| boeck2019/multi_task_hjdb | 0.0010 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0066 | 0.0000 | 0.4869 | 0.6900 | 0.2863 | 0.0636 | 0.1153 | 0.0309 | 0.1671 | 0.0054 |
| boeck2020/dar | 0.0018 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0045 | 0.0000 | 0.3771 | 0.8388 | 0.3833 | 0.0963 | 0.1671 | 0.0490 | 0.2379 | 0.0037 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0984 | 0.0046 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0186 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0046 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0002 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0019 | 0.0000 | 0.4869 | 0.6900 | 0.2863 | 0.0490 | 0.0768 | 0.0213 | 0.1185 | 0.0019 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0919 | 0.0213 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0115 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0984 | 0.0046 | 0.0000 | 0.0919 | 1.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0002 |
| percival2014/stem | 0.0000 | 0.0045 | 0.0066 | 0.0045 | 0.0046 | 0.0000 | 0.0019 | 0.0213 | 0.0001 | 1.0000 | 0.0000 | 0.0385 | 0.0003 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.3771 | 0.4869 | 0.3771 | 0.0000 | 0.0000 | 0.4869 | 0.0000 | 0.0000 | 0.0385 | 0.0000 | 1.0000 | 0.0215 | 0.0074 | 0.0013 | 0.0072 | 0.0007 | 0.0227 | 0.0385 |
| schreiber2017/ismir2017 | 0.0063 | 0.8388 | 0.6900 | 0.8388 | 0.0000 | 0.0000 | 0.6900 | 0.0000 | 0.0000 | 0.0003 | 0.0000 | 0.0215 | 1.0000 | 0.4531 | 0.1460 | 0.2668 | 0.0654 | 0.4545 | 0.0005 |
| schreiber2017/mirex2017 | 0.0391 | 0.3833 | 0.2863 | 0.3833 | 0.0000 | 0.0000 | 0.2863 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0074 | 0.4531 | 1.0000 | 0.4531 | 0.7539 | 0.2891 | 1.0000 | 0.0001 |
| schreiber2018/cnn | 0.1250 | 0.0963 | 0.0636 | 0.0963 | 0.0000 | 0.0000 | 0.0490 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0013 | 0.1460 | 0.4531 | 1.0000 | 1.0000 | 1.0000 | 0.7266 | 0.0000 |
| schreiber2018/fcn | 0.0625 | 0.1671 | 0.1153 | 0.1671 | 0.0000 | 0.0000 | 0.0768 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0072 | 0.2668 | 0.7539 | 1.0000 | 1.0000 | 0.6875 | 1.0000 | 0.0000 |
| schreiber2018/ismir2018 | 0.2500 | 0.0490 | 0.0309 | 0.0490 | 0.0000 | 0.0000 | 0.0213 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0007 | 0.0654 | 0.2891 | 1.0000 | 0.6875 | 1.0000 | 0.4531 | 0.0000 |
| sun2021/default | 0.0312 | 0.2379 | 0.1671 | 0.2379 | 0.0000 | 0.0000 | 0.1185 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0227 | 0.4545 | 1.0000 | 0.7266 | 1.0000 | 0.4531 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0037 | 0.0054 | 0.0037 | 0.0186 | 0.0000 | 0.0019 | 0.0115 | 0.0002 | 1.0000 | 0.0000 | 0.0385 | 0.0005 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 16: McNemar p-values, using reference annotations 3.0 as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0063 | 0.0391 | 0.1250 | 0.0625 | 0.2500 | 0.0625 | 0.0000 |
| boeck2019/multi_task | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 | 0.0078 | 0.0625 | 0.0312 | 0.1250 | 0.0312 | 0.0000 |
| boeck2019/multi_task_hjdb | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0010 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0063 | 0.0391 | 0.2188 | 0.1250 | 0.3750 | 0.1250 | 0.0000 |
| boeck2020/dar | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 | 0.0078 | 0.0625 | 0.0312 | 0.1250 | 0.0312 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0984 | 0.0046 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0114 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0063 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0002 | 0.0001 | 0.0010 | 0.0001 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0019 | 0.0000 | 0.4869 | 0.6900 | 0.2863 | 0.0490 | 0.0768 | 0.0213 | 0.0574 | 0.0029 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0649 | 0.0293 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0115 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0984 | 0.0063 | 0.0000 | 0.0649 | 1.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0046 | 0.0000 | 0.0019 | 0.0293 | 0.0001 | 1.0000 | 0.0000 | 0.0385 | 0.0003 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4869 | 0.0001 | 0.0000 | 0.0385 | 0.0000 | 1.0000 | 0.0215 | 0.0074 | 0.0013 | 0.0072 | 0.0007 | 0.0106 | 0.0533 |
| schreiber2017/ismir2017 | 0.0063 | 0.0010 | 0.0063 | 0.0010 | 0.0000 | 0.0000 | 0.6900 | 0.0000 | 0.0000 | 0.0003 | 0.0000 | 0.0215 | 1.0000 | 0.4531 | 0.1460 | 0.2668 | 0.0654 | 0.3018 | 0.0008 |
| schreiber2017/mirex2017 | 0.0391 | 0.0078 | 0.0391 | 0.0078 | 0.0000 | 0.0000 | 0.2863 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0074 | 0.4531 | 1.0000 | 0.4531 | 0.7539 | 0.2891 | 0.7744 | 0.0001 |
| schreiber2018/cnn | 0.1250 | 0.0625 | 0.2188 | 0.0625 | 0.0000 | 0.0000 | 0.0490 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0013 | 0.1460 | 0.4531 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0000 |
| schreiber2018/fcn | 0.0625 | 0.0312 | 0.1250 | 0.0312 | 0.0000 | 0.0000 | 0.0768 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0072 | 0.2668 | 0.7539 | 1.0000 | 1.0000 | 0.6875 | 1.0000 | 0.0000 |
| schreiber2018/ismir2018 | 0.2500 | 0.1250 | 0.3750 | 0.1250 | 0.0000 | 0.0000 | 0.0213 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0007 | 0.0654 | 0.2891 | 1.0000 | 0.6875 | 1.0000 | 0.6875 | 0.0000 |
| sun2021/default | 0.0625 | 0.0312 | 0.1250 | 0.0312 | 0.0000 | 0.0000 | 0.0574 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0106 | 0.3018 | 0.7744 | 1.0000 | 1.0000 | 0.6875 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0114 | 0.0000 | 0.0029 | 0.0115 | 0.0001 | 1.0000 | 0.0000 | 0.0533 | 0.0008 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 17: McNemar p-values, using reference annotations 3.0-no-dupes as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0002 | 0.0001 | 0.0002 | 0.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 | 0.0078 | 0.0625 | 0.0156 | 0.1250 | 0.0039 | 0.0000 |
| boeck2019/multi_task | 0.0002 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0096 | 0.0000 | 0.3771 | 0.8388 | 0.3833 | 0.0963 | 0.2632 | 0.0490 | 0.5034 | 0.0025 |
| boeck2019/multi_task_hjdb | 0.0001 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0137 | 0.0000 | 0.4869 | 0.6900 | 0.2863 | 0.0490 | 0.1671 | 0.0309 | 0.3593 | 0.0037 |
| boeck2020/dar | 0.0002 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0096 | 0.0000 | 0.3771 | 0.8388 | 0.3833 | 0.0963 | 0.2632 | 0.0490 | 0.5034 | 0.0025 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0001 | 0.0000 | 0.7877 | 0.1263 | 0.0003 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0110 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 | 0.0000 | 0.0000 | 0.0086 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0002 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0029 | 0.0000 | 0.3771 | 0.8388 | 0.3833 | 0.0963 | 0.2101 | 0.0352 | 0.4545 | 0.0005 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7877 | 0.0000 | 0.0000 | 1.0000 | 0.0534 | 0.0079 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0195 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1263 | 0.0086 | 0.0000 | 0.0534 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 |
| percival2014/stem | 0.0000 | 0.0096 | 0.0137 | 0.0096 | 0.0003 | 0.0000 | 0.0029 | 0.0079 | 0.0000 | 1.0000 | 0.0000 | 0.0730 | 0.0008 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.7428 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.3771 | 0.4869 | 0.3771 | 0.0000 | 0.0000 | 0.3771 | 0.0000 | 0.0000 | 0.0730 | 0.0000 | 1.0000 | 0.0215 | 0.0074 | 0.0013 | 0.0169 | 0.0007 | 0.0755 | 0.0275 |
| schreiber2017/ismir2017 | 0.0010 | 0.8388 | 0.6900 | 0.8388 | 0.0000 | 0.0000 | 0.8388 | 0.0000 | 0.0000 | 0.0008 | 0.0000 | 0.0215 | 1.0000 | 0.4531 | 0.1460 | 0.4240 | 0.0654 | 0.8145 | 0.0003 |
| schreiber2017/mirex2017 | 0.0078 | 0.3833 | 0.2863 | 0.3833 | 0.0000 | 0.0000 | 0.3833 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0074 | 0.4531 | 1.0000 | 0.4531 | 1.0000 | 0.2891 | 1.0000 | 0.0000 |
| schreiber2018/cnn | 0.0625 | 0.0963 | 0.0490 | 0.0963 | 0.0000 | 0.0000 | 0.0963 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0013 | 0.1460 | 0.4531 | 1.0000 | 0.7266 | 1.0000 | 0.3437 | 0.0000 |
| schreiber2018/fcn | 0.0156 | 0.2632 | 0.1671 | 0.2632 | 0.0000 | 0.0000 | 0.2101 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0169 | 0.4240 | 1.0000 | 0.7266 | 1.0000 | 0.4531 | 0.7539 | 0.0000 |
| schreiber2018/ismir2018 | 0.1250 | 0.0490 | 0.0309 | 0.0490 | 0.0000 | 0.0000 | 0.0352 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0007 | 0.0654 | 0.2891 | 1.0000 | 0.4531 | 1.0000 | 0.1797 | 0.0000 |
| sun2021/default | 0.0039 | 0.5034 | 0.3593 | 0.5034 | 0.0000 | 0.0000 | 0.4545 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0755 | 0.8145 | 1.0000 | 0.3437 | 0.7539 | 0.1797 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0025 | 0.0037 | 0.0025 | 0.0110 | 0.0000 | 0.0005 | 0.0195 | 0.0001 | 0.7428 | 0.0000 | 0.0275 | 0.0003 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 18: McNemar p-values, using reference annotations 2.0 as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 | 0.0078 | 0.0625 | 0.0156 | 0.1250 | 0.0078 | 0.0000 |
| boeck2019/multi_task | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 | 0.0078 | 0.0625 | 0.0156 | 0.1250 | 0.0078 | 0.0000 |
| boeck2019/multi_task_hjdb | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0063 | 0.0391 | 0.1250 | 0.0312 | 0.3750 | 0.0156 | 0.0000 |
| boeck2020/dar | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 | 0.0078 | 0.0625 | 0.0156 | 0.1250 | 0.0078 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0001 | 0.0000 | 0.6835 | 0.1263 | 0.0003 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0066 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 | 0.0000 | 0.0000 | 0.0115 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0002 | 0.0002 | 0.0018 | 0.0002 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0029 | 0.0000 | 0.3771 | 0.8388 | 0.3833 | 0.0963 | 0.2101 | 0.0352 | 0.3018 | 0.0008 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.6835 | 0.0000 | 0.0000 | 1.0000 | 0.0365 | 0.0114 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0195 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1263 | 0.0115 | 0.0000 | 0.0365 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0003 | 0.0000 | 0.0029 | 0.0114 | 0.0000 | 1.0000 | 0.0000 | 0.0730 | 0.0008 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8679 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.3771 | 0.0001 | 0.0000 | 0.0730 | 0.0000 | 1.0000 | 0.0215 | 0.0074 | 0.0013 | 0.0169 | 0.0007 | 0.0433 | 0.0385 |
| schreiber2017/ismir2017 | 0.0010 | 0.0010 | 0.0063 | 0.0010 | 0.0000 | 0.0000 | 0.8388 | 0.0000 | 0.0000 | 0.0008 | 0.0000 | 0.0215 | 1.0000 | 0.4531 | 0.1460 | 0.4240 | 0.0654 | 0.6291 | 0.0005 |
| schreiber2017/mirex2017 | 0.0078 | 0.0078 | 0.0391 | 0.0078 | 0.0000 | 0.0000 | 0.3833 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0074 | 0.4531 | 1.0000 | 0.4531 | 1.0000 | 0.2891 | 1.0000 | 0.0001 |
| schreiber2018/cnn | 0.0625 | 0.0625 | 0.1250 | 0.0625 | 0.0000 | 0.0000 | 0.0963 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0013 | 0.1460 | 0.4531 | 1.0000 | 0.7266 | 1.0000 | 0.5078 | 0.0000 |
| schreiber2018/fcn | 0.0156 | 0.0156 | 0.0312 | 0.0156 | 0.0000 | 0.0000 | 0.2101 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0169 | 0.4240 | 1.0000 | 0.7266 | 1.0000 | 0.4531 | 1.0000 | 0.0000 |
| schreiber2018/ismir2018 | 0.1250 | 0.1250 | 0.3750 | 0.1250 | 0.0000 | 0.0000 | 0.0352 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0007 | 0.0654 | 0.2891 | 1.0000 | 0.4531 | 1.0000 | 0.2891 | 0.0000 |
| sun2021/default | 0.0078 | 0.0078 | 0.0156 | 0.0078 | 0.0000 | 0.0000 | 0.3018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0433 | 0.6291 | 1.0000 | 0.5078 | 1.0000 | 0.2891 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0066 | 0.0000 | 0.0008 | 0.0195 | 0.0001 | 0.8679 | 0.0000 | 0.0385 | 0.0005 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 19: McNemar p-values, using reference annotations 2.0-no-dupes as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.1153 | 0.1153 | 0.0963 | 0.0000 | 0.0000 | 0.3018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0066 | 0.3323 | 0.7905 | 1.0000 | 1.0000 | 1.0000 | 0.4545 | 0.0000 |
| boeck2019/multi_task | 0.1153 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 0.6900 | 0.0000 | 0.0000 | 0.0096 | 0.0000 | 0.3915 | 0.6900 | 0.2863 | 0.1153 | 0.1153 | 0.1153 | 0.5572 | 0.0054 |
| boeck2019/multi_task_hjdb | 0.1153 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 0.6900 | 0.0000 | 0.0000 | 0.0096 | 0.0000 | 0.3915 | 0.6900 | 0.2863 | 0.1153 | 0.1153 | 0.1153 | 0.5572 | 0.0054 |
| boeck2020/dar | 0.0963 | 1.0000 | 1.0000 | 1.0000 | 0.0000 | 0.0000 | 0.6776 | 0.0000 | 0.0000 | 0.0096 | 0.0000 | 0.3915 | 0.6900 | 0.2863 | 0.0963 | 0.0963 | 0.0963 | 0.5413 | 0.0054 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0001 | 0.0000 | 0.3581 | 0.2370 | 0.0011 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0038 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 | 0.0000 | 0.0000 | 0.0034 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.3018 | 0.6900 | 0.6900 | 0.6776 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0008 | 0.0000 | 0.1496 | 1.0000 | 0.6476 | 0.3018 | 0.2266 | 0.2668 | 1.0000 | 0.0005 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.3581 | 0.0000 | 0.0000 | 1.0000 | 0.0175 | 0.0489 | 0.0000 | 0.0003 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0436 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2370 | 0.0034 | 0.0000 | 0.0175 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 |
| percival2014/stem | 0.0000 | 0.0096 | 0.0096 | 0.0096 | 0.0011 | 0.0000 | 0.0008 | 0.0489 | 0.0000 | 1.0000 | 0.0000 | 0.0730 | 0.0005 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0003 | 1.0000 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0066 | 0.3915 | 0.3915 | 0.3915 | 0.0000 | 0.0000 | 0.1496 | 0.0003 | 0.0000 | 0.0730 | 0.0000 | 1.0000 | 0.0117 | 0.0042 | 0.0013 | 0.0043 | 0.0043 | 0.0987 | 0.0596 |
| schreiber2017/ismir2017 | 0.3323 | 0.6900 | 0.6900 | 0.6900 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0005 | 0.0000 | 0.0117 | 1.0000 | 0.4531 | 0.2668 | 0.2668 | 0.2668 | 1.0000 | 0.0005 |
| schreiber2017/mirex2017 | 0.7905 | 0.2863 | 0.2863 | 0.2863 | 0.0000 | 0.0000 | 0.6476 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0042 | 0.4531 | 1.0000 | 0.7266 | 0.7539 | 0.7539 | 0.8145 | 0.0001 |
| schreiber2018/cnn | 1.0000 | 0.1153 | 0.1153 | 0.0963 | 0.0000 | 0.0000 | 0.3018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0013 | 0.2668 | 0.7266 | 1.0000 | 1.0000 | 1.0000 | 0.3877 | 0.0000 |
| schreiber2018/fcn | 1.0000 | 0.1153 | 0.1153 | 0.0963 | 0.0000 | 0.0000 | 0.2266 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0043 | 0.2668 | 0.7539 | 1.0000 | 1.0000 | 1.0000 | 0.3437 | 0.0000 |
| schreiber2018/ismir2018 | 1.0000 | 0.1153 | 0.1153 | 0.0963 | 0.0000 | 0.0000 | 0.2668 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0043 | 0.2668 | 0.7539 | 1.0000 | 1.0000 | 1.0000 | 0.3877 | 0.0000 |
| sun2021/default | 0.4545 | 0.5572 | 0.5572 | 0.5413 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0003 | 0.0000 | 0.0987 | 1.0000 | 0.8145 | 0.3877 | 0.3437 | 0.3877 | 1.0000 | 0.0004 |
| zplane/auftakt_v3 | 0.0000 | 0.0054 | 0.0054 | 0.0054 | 0.0038 | 0.0000 | 0.0005 | 0.0436 | 0.0001 | 1.0000 | 0.0000 | 0.0596 | 0.0005 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0004 | 1.0000 |
Table 20: McNemar p-values, using reference annotations 4.0 as groundtruth with Accuracy2 [Gouyon2006]. H0: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H0, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.
Accuracy1 on cvar-Subsets
How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?
Accuracy1 on cvar-Subsets for 1.0 based on cvar-Values from 1.0
Figure 17: Mean Accuracy1 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 on cvar-Subsets for 2.0 based on cvar-Values from 1.0
Figure 18: Mean Accuracy1 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0
Figure 19: Mean Accuracy1 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 on cvar-Subsets for 3.0 based on cvar-Values from 1.0
Figure 20: Mean Accuracy1 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0
Figure 21: Mean Accuracy1 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 on cvar-Subsets for 4.0 based on cvar-Values from 1.0
Figure 22: Mean Accuracy1 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on cvar-Subsets
How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?
Accuracy2 on cvar-Subsets for 1.0 based on cvar-Values from 1.0
Figure 23: Mean Accuracy2 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on cvar-Subsets for 2.0 based on cvar-Values from 1.0
Figure 24: Mean Accuracy2 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0
Figure 25: Mean Accuracy2 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on cvar-Subsets for 3.0 based on cvar-Values from 1.0
Figure 26: Mean Accuracy2 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0
Figure 27: Mean Accuracy2 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on cvar-Subsets for 4.0 based on cvar-Values from 1.0
Figure 28: Mean Accuracy2 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 on Tempo-Subsets
How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean Accuracy1 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.
Accuracy1 on Tempo-Subsets for 1.0
Figure 29: Mean Accuracy1 for estimates compared to version 1.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 on Tempo-Subsets for 2.0
Figure 30: Mean Accuracy1 for estimates compared to version 2.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 on Tempo-Subsets for 2.0-no-dupes
Figure 31: Mean Accuracy1 for estimates compared to version 2.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 on Tempo-Subsets for 3.0
Figure 32: Mean Accuracy1 for estimates compared to version 3.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 on Tempo-Subsets for 3.0-no-dupes
Figure 33: Mean Accuracy1 for estimates compared to version 3.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 on Tempo-Subsets for 4.0
Figure 34: Mean Accuracy1 for estimates compared to version 4.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on Tempo-Subsets
How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean Accuracy2 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.
Accuracy2 on Tempo-Subsets for 1.0
Figure 35: Mean Accuracy2 for estimates compared to version 1.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on Tempo-Subsets for 2.0
Figure 36: Mean Accuracy2 for estimates compared to version 2.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on Tempo-Subsets for 2.0-no-dupes
Figure 37: Mean Accuracy2 for estimates compared to version 2.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on Tempo-Subsets for 3.0
Figure 38: Mean Accuracy2 for estimates compared to version 3.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on Tempo-Subsets for 3.0-no-dupes
Figure 39: Mean Accuracy2 for estimates compared to version 3.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 on Tempo-Subsets for 4.0
Figure 40: Mean Accuracy2 for estimates compared to version 4.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy1 for Tempo
When fitting a generalized additive model (GAM) to Accuracy1-values and a ground truth, what Accuracy1 can we expect with confidence?
Estimated Accuracy1 for Tempo for 1.0
Predictions of GAMs trained on Accuracy1 for estimates for reference 1.0.
Figure 41: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy1 for Tempo for 2.0
Predictions of GAMs trained on Accuracy1 for estimates for reference 2.0.
Figure 42: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy1 for Tempo for 2.0-no-dupes
Predictions of GAMs trained on Accuracy1 for estimates for reference 2.0-no-dupes.
Figure 43: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy1 for Tempo for 3.0
Predictions of GAMs trained on Accuracy1 for estimates for reference 3.0.
Figure 44: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy1 for Tempo for 3.0-no-dupes
Predictions of GAMs trained on Accuracy1 for estimates for reference 3.0-no-dupes.
Figure 45: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy1 for Tempo for 4.0
Predictions of GAMs trained on Accuracy1 for estimates for reference 4.0.
Figure 46: Accuracy1 predictions of a generalized additive model (GAM) fit to Accuracy1 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy2 for Tempo
When fitting a generalized additive model (GAM) to Accuracy2-values and a ground truth, what Accuracy2 can we expect with confidence?
Estimated Accuracy2 for Tempo for 1.0
Predictions of GAMs trained on Accuracy2 for estimates for reference 1.0.
Figure 47: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy2 for Tempo for 2.0
Predictions of GAMs trained on Accuracy2 for estimates for reference 2.0.
Figure 48: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy2 for Tempo for 2.0-no-dupes
Predictions of GAMs trained on Accuracy2 for estimates for reference 2.0-no-dupes.
Figure 49: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy2 for Tempo for 3.0
Predictions of GAMs trained on Accuracy2 for estimates for reference 3.0.
Figure 50: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy2 for Tempo for 3.0-no-dupes
Predictions of GAMs trained on Accuracy2 for estimates for reference 3.0-no-dupes.
Figure 51: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated Accuracy2 for Tempo for 4.0
Predictions of GAMs trained on Accuracy2 for estimates for reference 4.0.
Figure 52: Accuracy2 predictions of a generalized additive model (GAM) fit to Accuracy2 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 for ‘tag_open’ Tags
How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.
Accuracy1 for ‘tag_open’ Tags for 1.0
Figure 53: Mean Accuracy1 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 for ‘tag_open’ Tags for 2.0
Figure 54: Mean Accuracy1 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 for ‘tag_open’ Tags for 2.0-no-dupes
Figure 55: Mean Accuracy1 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 for ‘tag_open’ Tags for 3.0
Figure 56: Mean Accuracy1 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 for ‘tag_open’ Tags for 3.0-no-dupes
Figure 57: Mean Accuracy1 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy1 for ‘tag_open’ Tags for 4.0
Figure 58: Mean Accuracy1 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for ‘tag_open’ Tags
How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.
Accuracy2 for ‘tag_open’ Tags for 1.0
Figure 59: Mean Accuracy2 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for ‘tag_open’ Tags for 2.0
Figure 60: Mean Accuracy2 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for ‘tag_open’ Tags for 2.0-no-dupes
Figure 61: Mean Accuracy2 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for ‘tag_open’ Tags for 3.0
Figure 62: Mean Accuracy2 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for ‘tag_open’ Tags for 3.0-no-dupes
Figure 63: Mean Accuracy2 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
Accuracy2 for ‘tag_open’ Tags for 4.0
Figure 64: Mean Accuracy2 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 and OE2
OE1 is defined as octave error between an estimate E and a reference value R.This means that the most common errors—by a factor of 2 or ½—have the same magnitude, namely 1: OE2(E) = log2(E/R).
OE2 is the signed OE1 corresponding to the minimum absolute OE1 allowing the octaveerrors 2, 3, 1/2, and 1/3: OE2(E) = arg minx(|x|) with x ∈ {OE1(E), OE1(2E), OE1(3E), OE1(½E), OE1(⅓E)}
Mean OE1/OE2 Results for 1.0
| Estimator | OE1_MEAN | OE1_STDEV | OE2_MEAN | OE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | -0.0059 | 0.1163 | -0.0029 | 0.0491 |
| boeck2019/multi_task | 0.0018 | 0.1407 | -0.0026 | 0.0485 |
| boeck2019/multi_task_hjdb | 0.0037 | 0.1474 | -0.0022 | 0.0497 |
| sun2021/default | -0.0096 | 0.1479 | -0.0044 | 0.0532 |
| schreiber2018/cnn | -0.0037 | 0.1774 | 0.0006 | 0.0574 |
| schreiber2018/ismir2018 | -0.0403 | 0.2284 | -0.0016 | 0.0579 |
| schreiber2018/fcn | -0.0156 | 0.2311 | 0.0001 | 0.0568 |
| schreiber2017/mirex2017 | -0.0456 | 0.2765 | -0.0027 | 0.0668 |
| schreiber2017/ismir2017 | -0.0779 | 0.3494 | 0.0003 | 0.0716 |
| boeck2015/tempodetector2016_default | -0.1312 | 0.3865 | -0.0032 | 0.0503 |
| schreiber2014/default | -0.3127 | 0.4610 | 0.0025 | 0.0840 |
| zplane/auftakt_v3 | -0.2660 | 0.4749 | 0.0042 | 0.0841 |
| oliveira2010/ibt | -0.2440 | 0.4841 | 0.0067 | 0.0931 |
| klapuri2006/percival2014 | -0.2478 | 0.4892 | 0.0152 | 0.1012 |
| percival2014/stem | -0.3133 | 0.4924 | 0.0001 | 0.0767 |
| davies2009/mirex_qm_tempotracker | -0.0879 | 0.5170 | 0.0241 | 0.0900 |
| scheirer1998/percival2014 | -0.1809 | 0.5199 | 0.0246 | 0.1546 |
| echonest/version_3_2_1 | -0.3190 | 0.5379 | -0.0129 | 0.1318 |
| gkiokas2012/default | -0.3835 | 0.5661 | -0.0014 | 0.0664 |
Table 21: Mean OE1/OE2 for estimates compared to version 1.0 ordered by standard deviation.
Raw data OE1: CSV JSON LATEX PICKLE
Raw data OE2: CSV JSON LATEX PICKLE
OE1 distribution for 1.0
Figure 65: OE1 for estimates compared to version 1.0. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 distribution for 1.0
Figure 66: OE2 for estimates compared to version 1.0. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Mean OE1/OE2 Results for 2.0
| Estimator | OE1_MEAN | OE1_STDEV | OE2_MEAN | OE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | -0.0011 | 0.0857 | 0.0004 | 0.0082 |
| boeck2019/multi_task | 0.0066 | 0.1206 | 0.0007 | 0.0084 |
| boeck2019/multi_task_hjdb | 0.0084 | 0.1229 | 0.0012 | 0.0151 |
| sun2021/default | -0.0050 | 0.1258 | -0.0013 | 0.0223 |
| schreiber2018/cnn | 0.0008 | 0.1613 | 0.0037 | 0.0307 |
| schreiber2018/fcn | -0.0111 | 0.2170 | 0.0032 | 0.0268 |
| schreiber2018/ismir2018 | -0.0358 | 0.2173 | 0.0029 | 0.0315 |
| schreiber2017/mirex2017 | -0.0411 | 0.2706 | 0.0004 | 0.0450 |
| schreiber2017/ismir2017 | -0.0734 | 0.3414 | 0.0034 | 0.0527 |
| boeck2015/tempodetector2016_default | -0.1267 | 0.3846 | -0.0002 | 0.0101 |
| schreiber2014/default | -0.3082 | 0.4613 | 0.0056 | 0.0686 |
| zplane/auftakt_v3 | -0.2615 | 0.4735 | 0.0050 | 0.0716 |
| oliveira2010/ibt | -0.2395 | 0.4860 | 0.0084 | 0.0855 |
| klapuri2006/percival2014 | -0.2433 | 0.4893 | 0.0154 | 0.0919 |
| percival2014/stem | -0.3088 | 0.4920 | 0.0032 | 0.0613 |
| davies2009/mirex_qm_tempotracker | -0.0834 | 0.5198 | 0.0257 | 0.0794 |
| scheirer1998/percival2014 | -0.1751 | 0.5206 | 0.0264 | 0.1524 |
| echonest/version_3_2_1 | -0.3145 | 0.5408 | -0.0126 | 0.1225 |
| gkiokas2012/default | -0.3789 | 0.5666 | 0.0002 | 0.0439 |
Table 22: Mean OE1/OE2 for estimates compared to version 2.0 ordered by standard deviation.
Raw data OE1: CSV JSON LATEX PICKLE
Raw data OE2: CSV JSON LATEX PICKLE
OE1 distribution for 2.0
Figure 67: OE1 for estimates compared to version 2.0. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 distribution for 2.0
Figure 68: OE2 for estimates compared to version 2.0. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Mean OE1/OE2 Results for 2.0-no-dupes
| Estimator | OE1_MEAN | OE1_STDEV | OE2_MEAN | OE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | -0.0011 | 0.0857 | 0.0004 | 0.0082 |
| boeck2019/multi_task | 0.0066 | 0.1206 | 0.0007 | 0.0084 |
| boeck2019/multi_task_hjdb | 0.0084 | 0.1229 | 0.0012 | 0.0151 |
| sun2021/default | -0.0052 | 0.1269 | -0.0014 | 0.0221 |
| schreiber2018/cnn | 0.0008 | 0.1628 | 0.0037 | 0.0310 |
| schreiber2018/fcn | -0.0114 | 0.2191 | 0.0032 | 0.0270 |
| schreiber2018/ismir2018 | -0.0365 | 0.2193 | 0.0029 | 0.0318 |
| schreiber2017/mirex2017 | -0.0419 | 0.2731 | 0.0005 | 0.0454 |
| schreiber2017/ismir2017 | -0.0747 | 0.3445 | 0.0035 | 0.0532 |
| boeck2015/tempodetector2016_default | -0.1291 | 0.3878 | -0.0002 | 0.0101 |
| schreiber2014/default | -0.3125 | 0.4630 | 0.0057 | 0.0692 |
| zplane/auftakt_v3 | -0.2659 | 0.4746 | 0.0056 | 0.0711 |
| oliveira2010/ibt | -0.2438 | 0.4866 | 0.0088 | 0.0862 |
| klapuri2006/percival2014 | -0.2475 | 0.4906 | 0.0161 | 0.0923 |
| percival2014/stem | -0.3117 | 0.4936 | 0.0033 | 0.0619 |
| scheirer1998/percival2014 | -0.1745 | 0.5197 | 0.0254 | 0.1520 |
| davies2009/mirex_qm_tempotracker | -0.0854 | 0.5217 | 0.0258 | 0.0802 |
| echonest/version_3_2_1 | -0.3181 | 0.5437 | -0.0135 | 0.1223 |
| gkiokas2012/default | -0.3847 | 0.5691 | 0.0002 | 0.0443 |
Table 23: Mean OE1/OE2 for estimates compared to version 2.0-no-dupes ordered by standard deviation.
Raw data OE1: CSV JSON LATEX PICKLE
Raw data OE2: CSV JSON LATEX PICKLE
OE1 distribution for 2.0-no-dupes
Figure 69: OE1 for estimates compared to version 2.0-no-dupes. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 distribution for 2.0-no-dupes
Figure 70: OE2 for estimates compared to version 2.0-no-dupes. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Mean OE1/OE2 Results for 3.0
| Estimator | OE1_MEAN | OE1_STDEV | OE2_MEAN | OE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | -0.0007 | 0.0855 | 0.0008 | 0.0057 |
| boeck2019/multi_task | 0.0070 | 0.1209 | 0.0011 | 0.0050 |
| boeck2019/multi_task_hjdb | 0.0089 | 0.1231 | 0.0016 | 0.0130 |
| sun2021/default | -0.0046 | 0.1255 | -0.0009 | 0.0211 |
| schreiber2018/cnn | 0.0013 | 0.1616 | 0.0041 | 0.0299 |
| schreiber2018/fcn | -0.0107 | 0.2174 | 0.0037 | 0.0261 |
| schreiber2018/ismir2018 | -0.0353 | 0.2179 | 0.0019 | 0.0317 |
| schreiber2017/mirex2017 | -0.0407 | 0.2708 | 0.0009 | 0.0446 |
| schreiber2017/ismir2017 | -0.0729 | 0.3415 | 0.0038 | 0.0523 |
| boeck2015/tempodetector2016_default | -0.1263 | 0.3854 | 0.0003 | 0.0098 |
| schreiber2014/default | -0.3077 | 0.4619 | 0.0060 | 0.0682 |
| zplane/auftakt_v3 | -0.2610 | 0.4741 | 0.0055 | 0.0706 |
| oliveira2010/ibt | -0.2390 | 0.4865 | 0.0088 | 0.0851 |
| klapuri2006/percival2014 | -0.2428 | 0.4899 | 0.0159 | 0.0916 |
| percival2014/stem | -0.3084 | 0.4925 | 0.0036 | 0.0607 |
| davies2009/mirex_qm_tempotracker | -0.0830 | 0.5202 | 0.0262 | 0.0791 |
| scheirer1998/percival2014 | -0.1745 | 0.5208 | 0.0269 | 0.1521 |
| echonest/version_3_2_1 | -0.3140 | 0.5407 | -0.0108 | 0.1230 |
| gkiokas2012/default | -0.3785 | 0.5673 | 0.0007 | 0.0430 |
Table 24: Mean OE1/OE2 for estimates compared to version 3.0 ordered by standard deviation.
Raw data OE1: CSV JSON LATEX PICKLE
Raw data OE2: CSV JSON LATEX PICKLE
OE1 distribution for 3.0
Figure 71: OE1 for estimates compared to version 3.0. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 distribution for 3.0
Figure 72: OE2 for estimates compared to version 3.0. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Mean OE1/OE2 Results for 3.0-no-dupes
| Estimator | OE1_MEAN | OE1_STDEV | OE2_MEAN | OE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | -0.0007 | 0.0855 | 0.0008 | 0.0057 |
| boeck2019/multi_task | 0.0070 | 0.1209 | 0.0011 | 0.0050 |
| boeck2019/multi_task_hjdb | 0.0089 | 0.1231 | 0.0016 | 0.0130 |
| sun2021/default | -0.0047 | 0.1267 | -0.0010 | 0.0209 |
| schreiber2018/cnn | 0.0012 | 0.1631 | 0.0041 | 0.0302 |
| schreiber2018/fcn | -0.0109 | 0.2194 | 0.0037 | 0.0263 |
| schreiber2018/ismir2018 | -0.0361 | 0.2199 | 0.0019 | 0.0319 |
| schreiber2017/mirex2017 | -0.0415 | 0.2733 | 0.0009 | 0.0450 |
| schreiber2017/ismir2017 | -0.0743 | 0.3445 | 0.0039 | 0.0528 |
| boeck2015/tempodetector2016_default | -0.1287 | 0.3886 | 0.0003 | 0.0098 |
| schreiber2014/default | -0.3121 | 0.4637 | 0.0061 | 0.0688 |
| zplane/auftakt_v3 | -0.2655 | 0.4753 | 0.0061 | 0.0701 |
| oliveira2010/ibt | -0.2434 | 0.4872 | 0.0092 | 0.0858 |
| klapuri2006/percival2014 | -0.2471 | 0.4912 | 0.0166 | 0.0919 |
| percival2014/stem | -0.3113 | 0.4942 | 0.0037 | 0.0613 |
| scheirer1998/percival2014 | -0.1739 | 0.5199 | 0.0260 | 0.1517 |
| davies2009/mirex_qm_tempotracker | -0.0850 | 0.5221 | 0.0262 | 0.0799 |
| echonest/version_3_2_1 | -0.3177 | 0.5437 | -0.0116 | 0.1228 |
| gkiokas2012/default | -0.3843 | 0.5699 | 0.0006 | 0.0434 |
Table 25: Mean OE1/OE2 for estimates compared to version 3.0-no-dupes ordered by standard deviation.
Raw data OE1: CSV JSON LATEX PICKLE
Raw data OE2: CSV JSON LATEX PICKLE
OE1 distribution for 3.0-no-dupes
Figure 73: OE1 for estimates compared to version 3.0-no-dupes. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 distribution for 3.0-no-dupes
Figure 74: OE2 for estimates compared to version 3.0-no-dupes. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Mean OE1/OE2 Results for 4.0
| Estimator | OE1_MEAN | OE1_STDEV | OE2_MEAN | OE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | -0.0005 | 0.1370 | -0.0034 | 0.0277 |
| boeck2019/multi_task | 0.0072 | 0.1519 | -0.0031 | 0.0275 |
| boeck2019/multi_task_hjdb | 0.0090 | 0.1632 | -0.0026 | 0.0294 |
| sun2021/default | -0.0046 | 0.1646 | -0.0052 | 0.0328 |
| schreiber2018/cnn | 0.0012 | 0.1855 | -0.0002 | 0.0402 |
| schreiber2018/ismir2018 | -0.0354 | 0.2365 | -0.0024 | 0.0405 |
| schreiber2018/fcn | -0.0107 | 0.2425 | -0.0007 | 0.0373 |
| schreiber2017/mirex2017 | -0.0407 | 0.2863 | -0.0035 | 0.0521 |
| schreiber2017/ismir2017 | -0.0730 | 0.3578 | -0.0005 | 0.0587 |
| boeck2015/tempodetector2016_default | -0.1263 | 0.3889 | -0.0040 | 0.0288 |
| schreiber2014/default | -0.3078 | 0.4631 | 0.0017 | 0.0729 |
| zplane/auftakt_v3 | -0.2610 | 0.4768 | 0.0020 | 0.0688 |
| oliveira2010/ibt | -0.2391 | 0.4851 | 0.0045 | 0.0834 |
| klapuri2006/percival2014 | -0.2429 | 0.4902 | 0.0130 | 0.0898 |
| percival2014/stem | -0.3084 | 0.4931 | 0.0022 | 0.0654 |
| davies2009/mirex_qm_tempotracker | -0.0830 | 0.5159 | 0.0247 | 0.0786 |
| scheirer1998/percival2014 | -0.1746 | 0.5214 | 0.0231 | 0.1514 |
| echonest/version_3_2_1 | -0.3141 | 0.5408 | -0.0094 | 0.1257 |
| gkiokas2012/default | -0.3785 | 0.5694 | -0.0022 | 0.0436 |
Table 26: Mean OE1/OE2 for estimates compared to version 4.0 ordered by standard deviation.
Raw data OE1: CSV JSON LATEX PICKLE
Raw data OE2: CSV JSON LATEX PICKLE
OE1 distribution for 4.0
Figure 75: OE1 for estimates compared to version 4.0. Shown are the mean OE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 distribution for 4.0
Figure 76: OE2 for estimates compared to version 4.0. Shown are the mean OE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Significance of Differences
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0125 | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.5376 | 0.1454 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4201 | 0.0429 | 0.0000 | 0.0666 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.5376 | 1.0000 | 0.0521 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2688 | 0.0184 | 0.0000 | 0.0267 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1454 | 0.0521 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.7815 | 0.2267 | 0.0000 | 0.4173 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5806 | 0.0264 | 0.0000 | 0.0002 | 0.0127 | 0.0001 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0004 | 0.0001 | 0.0000 | 0.7582 | 0.0000 | 0.7238 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0018 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0004 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 1.0000 | 0.5485 | 0.0000 | 0.0004 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0074 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5485 | 1.0000 | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0009 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7582 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.9477 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| scheirer1998/percival2014 | 0.0125 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0004 | 0.0005 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7238 | 0.0000 | 0.0000 | 0.0000 | 0.9477 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/ismir2017 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.5806 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0014 | 0.0000 | 0.0000 | 0.0038 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0264 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0014 | 1.0000 | 0.0001 | 0.0108 | 0.6452 | 0.0008 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.4201 | 0.2688 | 0.7815 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 | 0.1677 | 0.0000 | 0.3931 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0429 | 0.0184 | 0.2267 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0108 | 0.1677 | 1.0000 | 0.0047 | 0.5052 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0127 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0038 | 0.6452 | 0.0000 | 0.0047 | 1.0000 | 0.0005 | 0.0000 |
| sun2021/default | 0.0000 | 0.0666 | 0.0267 | 0.4173 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0008 | 0.3931 | 0.5052 | 0.0005 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0018 | 0.0000 | 0.0074 | 0.0009 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 27: Paired t-test p-values, using reference annotations 1.0 as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0125 | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.5376 | 0.1454 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4201 | 0.0429 | 0.0000 | 0.0666 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.5376 | 1.0000 | 0.0521 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2688 | 0.0184 | 0.0000 | 0.0267 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1454 | 0.0521 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.7815 | 0.2267 | 0.0000 | 0.4173 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5806 | 0.0264 | 0.0000 | 0.0002 | 0.0127 | 0.0001 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0004 | 0.0001 | 0.0000 | 0.7582 | 0.0000 | 0.7238 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0018 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0004 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 1.0000 | 0.5485 | 0.0000 | 0.0004 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0074 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5485 | 1.0000 | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0009 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7582 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.9477 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| scheirer1998/percival2014 | 0.0125 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0004 | 0.0005 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7238 | 0.0000 | 0.0000 | 0.0000 | 0.9477 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/ismir2017 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.5806 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0014 | 0.0000 | 0.0000 | 0.0038 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0264 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0014 | 1.0000 | 0.0001 | 0.0108 | 0.6452 | 0.0008 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.4201 | 0.2688 | 0.7815 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 | 0.1677 | 0.0000 | 0.3931 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0429 | 0.0184 | 0.2267 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0108 | 0.1677 | 1.0000 | 0.0047 | 0.5052 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0127 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0038 | 0.6452 | 0.0000 | 0.0047 | 1.0000 | 0.0005 | 0.0000 |
| sun2021/default | 0.0000 | 0.0666 | 0.0267 | 0.4173 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0008 | 0.3931 | 0.5052 | 0.0005 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0018 | 0.0000 | 0.0074 | 0.0009 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 28: Paired t-test p-values, using reference annotations 3.0 as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0190 | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.5376 | 0.1454 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4201 | 0.0429 | 0.0000 | 0.0666 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.5376 | 1.0000 | 0.0521 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2688 | 0.0184 | 0.0000 | 0.0267 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1454 | 0.0521 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.7815 | 0.2267 | 0.0000 | 0.4173 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5620 | 0.0240 | 0.0000 | 0.0002 | 0.0116 | 0.0001 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0003 | 0.0001 | 0.0000 | 0.7311 | 0.0000 | 0.7607 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0025 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0003 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 1.0000 | 0.5661 | 0.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0078 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5661 | 1.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7311 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.9337 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| scheirer1998/percival2014 | 0.0190 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0002 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7607 | 0.0000 | 0.0000 | 0.0000 | 0.9337 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/ismir2017 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.5620 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0014 | 0.0000 | 0.0000 | 0.0038 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0240 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0014 | 1.0000 | 0.0001 | 0.0110 | 0.6498 | 0.0008 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.4201 | 0.2688 | 0.7815 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 | 0.1677 | 0.0000 | 0.3934 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0429 | 0.0184 | 0.2267 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0110 | 0.1677 | 1.0000 | 0.0047 | 0.5048 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0116 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0038 | 0.6498 | 0.0000 | 0.0047 | 1.0000 | 0.0005 | 0.0000 |
| sun2021/default | 0.0000 | 0.0666 | 0.0267 | 0.4173 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0008 | 0.3934 | 0.5048 | 0.0005 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0025 | 0.0000 | 0.0078 | 0.0010 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 29: Paired t-test p-values, using reference annotations 3.0-no-dupes as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0125 | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.5376 | 0.1454 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4201 | 0.0429 | 0.0000 | 0.0666 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.5376 | 1.0000 | 0.0521 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2688 | 0.0184 | 0.0000 | 0.0267 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1454 | 0.0521 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.7815 | 0.2267 | 0.0000 | 0.4173 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5806 | 0.0264 | 0.0000 | 0.0002 | 0.0127 | 0.0001 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0004 | 0.0001 | 0.0000 | 0.7582 | 0.0000 | 0.7238 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0018 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0004 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 1.0000 | 0.5485 | 0.0000 | 0.0004 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0074 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5485 | 1.0000 | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0009 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7582 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.9477 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| scheirer1998/percival2014 | 0.0125 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0004 | 0.0005 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7238 | 0.0000 | 0.0000 | 0.0000 | 0.9477 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/ismir2017 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.5806 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0014 | 0.0000 | 0.0000 | 0.0038 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0264 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0014 | 1.0000 | 0.0001 | 0.0108 | 0.6452 | 0.0008 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.4201 | 0.2688 | 0.7815 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 | 0.1677 | 0.0000 | 0.3931 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0429 | 0.0184 | 0.2267 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0108 | 0.1677 | 1.0000 | 0.0047 | 0.5052 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0127 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0038 | 0.6452 | 0.0000 | 0.0047 | 1.0000 | 0.0005 | 0.0000 |
| sun2021/default | 0.0000 | 0.0666 | 0.0267 | 0.4173 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0008 | 0.3931 | 0.5052 | 0.0005 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0018 | 0.0000 | 0.0074 | 0.0009 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 30: Paired t-test p-values, using reference annotations 2.0 as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0190 | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.5376 | 0.1454 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4201 | 0.0429 | 0.0000 | 0.0666 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.5376 | 1.0000 | 0.0521 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2688 | 0.0184 | 0.0000 | 0.0267 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1454 | 0.0521 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.7815 | 0.2267 | 0.0000 | 0.4173 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5620 | 0.0240 | 0.0000 | 0.0002 | 0.0116 | 0.0001 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0003 | 0.0001 | 0.0000 | 0.7311 | 0.0000 | 0.7607 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0025 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0003 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 1.0000 | 0.5661 | 0.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0078 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5661 | 1.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7311 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.9337 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| scheirer1998/percival2014 | 0.0190 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0002 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7607 | 0.0000 | 0.0000 | 0.0000 | 0.9337 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/ismir2017 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.5620 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0014 | 0.0000 | 0.0000 | 0.0038 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0240 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0014 | 1.0000 | 0.0001 | 0.0110 | 0.6498 | 0.0008 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.4201 | 0.2688 | 0.7815 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 | 0.1677 | 0.0000 | 0.3934 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0429 | 0.0184 | 0.2267 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0110 | 0.1677 | 1.0000 | 0.0047 | 0.5048 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0116 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0038 | 0.6498 | 0.0000 | 0.0047 | 1.0000 | 0.0005 | 0.0000 |
| sun2021/default | 0.0000 | 0.0666 | 0.0267 | 0.4173 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0008 | 0.3934 | 0.5048 | 0.0005 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0025 | 0.0000 | 0.0078 | 0.0010 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 31: Paired t-test p-values, using reference annotations 2.0-no-dupes as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0125 | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.5376 | 0.1454 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4201 | 0.0429 | 0.0000 | 0.0666 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.5376 | 1.0000 | 0.0521 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2688 | 0.0184 | 0.0000 | 0.0267 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1454 | 0.0521 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.7815 | 0.2267 | 0.0000 | 0.4173 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0080 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5806 | 0.0264 | 0.0000 | 0.0002 | 0.0127 | 0.0001 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0004 | 0.0001 | 0.0000 | 0.7582 | 0.0000 | 0.7238 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0018 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0004 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 1.0000 | 0.5485 | 0.0000 | 0.0004 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0074 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5485 | 1.0000 | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0009 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7582 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.9477 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| scheirer1998/percival2014 | 0.0125 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0004 | 0.0005 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7238 | 0.0000 | 0.0000 | 0.0000 | 0.9477 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/ismir2017 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.5806 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0014 | 0.0000 | 0.0000 | 0.0038 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0002 | 0.0264 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0014 | 1.0000 | 0.0001 | 0.0108 | 0.6452 | 0.0008 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.4201 | 0.2688 | 0.7815 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 | 0.1677 | 0.0000 | 0.3931 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0429 | 0.0184 | 0.2267 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0108 | 0.1677 | 1.0000 | 0.0047 | 0.5052 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0127 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0038 | 0.6452 | 0.0000 | 0.0047 | 1.0000 | 0.0005 | 0.0000 |
| sun2021/default | 0.0000 | 0.0666 | 0.0267 | 0.4173 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0008 | 0.3931 | 0.5052 | 0.0005 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0018 | 0.0000 | 0.0074 | 0.0009 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 32: Paired t-test p-values, using reference annotations 4.0 as groundtruth with OE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0191 | 0.0197 | 0.1158 | 0.0000 | 0.0409 | 0.3570 | 0.0000 | 0.0024 | 0.1554 | 0.0000 | 0.0288 | 0.0813 | 0.7460 | 0.0013 | 0.0011 | 0.2169 | 0.1789 | 0.0102 |
| boeck2019/multi_task | 0.0191 | 1.0000 | 0.3775 | 0.1249 | 0.0000 | 0.0184 | 0.6232 | 0.0000 | 0.0040 | 0.2820 | 0.0000 | 0.0577 | 0.1700 | 0.8820 | 0.0106 | 0.0138 | 0.5422 | 0.0070 | 0.0122 |
| boeck2019/multi_task_hjdb | 0.0197 | 0.3775 | 1.0000 | 0.1273 | 0.0000 | 0.0151 | 0.7943 | 0.0000 | 0.0061 | 0.3490 | 0.0000 | 0.0891 | 0.2631 | 0.6985 | 0.0386 | 0.0570 | 0.8194 | 0.0034 | 0.0165 |
| boeck2020/dar | 0.1158 | 0.1249 | 0.1273 | 1.0000 | 0.0000 | 0.0216 | 0.5115 | 0.0000 | 0.0029 | 0.2234 | 0.0000 | 0.0446 | 0.1290 | 0.9639 | 0.0040 | 0.0045 | 0.3985 | 0.0277 | 0.0091 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0415 | 0.0000 | 0.0000 | 0.8497 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.0409 | 0.0184 | 0.0151 | 0.0216 | 0.0000 | 1.0000 | 0.0261 | 0.0000 | 0.0010 | 0.0161 | 0.0000 | 0.0054 | 0.0112 | 0.0373 | 0.0057 | 0.0068 | 0.0232 | 0.0809 | 0.0028 |
| gkiokas2012/default | 0.3570 | 0.6232 | 0.7943 | 0.5115 | 0.0000 | 0.0261 | 1.0000 | 0.0000 | 0.0202 | 0.5904 | 0.0000 | 0.2283 | 0.5372 | 0.6302 | 0.3704 | 0.4764 | 0.9306 | 0.1451 | 0.0912 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0415 | 0.0000 | 0.0000 | 1.0000 | 0.0290 | 0.0002 | 0.1870 | 0.0009 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0082 |
| oliveira2010/ibt | 0.0024 | 0.0040 | 0.0061 | 0.0029 | 0.0000 | 0.0010 | 0.0202 | 0.0290 | 1.0000 | 0.0960 | 0.0103 | 0.3110 | 0.0755 | 0.0050 | 0.0517 | 0.0455 | 0.0091 | 0.0005 | 0.5573 |
| percival2014/stem | 0.1554 | 0.2820 | 0.3490 | 0.2234 | 0.0000 | 0.0161 | 0.5904 | 0.0002 | 0.0960 | 1.0000 | 0.0001 | 0.4524 | 0.9451 | 0.2892 | 0.8397 | 0.9927 | 0.4584 | 0.0439 | 0.2257 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8497 | 0.0000 | 0.0000 | 0.1870 | 0.0103 | 0.0001 | 1.0000 | 0.0009 | 0.0002 | 0.0000 | 0.0001 | 0.0001 | 0.0000 | 0.0000 | 0.0012 |
| schreiber2014/default | 0.0288 | 0.0577 | 0.0891 | 0.0446 | 0.0000 | 0.0054 | 0.2283 | 0.0009 | 0.3110 | 0.4524 | 0.0009 | 1.0000 | 0.2496 | 0.0262 | 0.4255 | 0.3397 | 0.1098 | 0.0068 | 0.6257 |
| schreiber2017/ismir2017 | 0.0813 | 0.1700 | 0.2631 | 0.1290 | 0.0000 | 0.0112 | 0.5372 | 0.0000 | 0.0755 | 0.9451 | 0.0002 | 0.2496 | 1.0000 | 0.0587 | 0.8821 | 0.9283 | 0.3290 | 0.0149 | 0.2333 |
| schreiber2017/mirex2017 | 0.7460 | 0.8820 | 0.6985 | 0.9639 | 0.0000 | 0.0373 | 0.6302 | 0.0000 | 0.0050 | 0.2892 | 0.0000 | 0.0262 | 0.0587 | 1.0000 | 0.0184 | 0.0729 | 0.5386 | 0.2865 | 0.0335 |
| schreiber2018/cnn | 0.0013 | 0.0106 | 0.0386 | 0.0040 | 0.0000 | 0.0057 | 0.3704 | 0.0000 | 0.0517 | 0.8397 | 0.0001 | 0.4255 | 0.8821 | 0.0184 | 1.0000 | 0.6915 | 0.0487 | 0.0000 | 0.2225 |
| schreiber2018/fcn | 0.0011 | 0.0138 | 0.0570 | 0.0045 | 0.0000 | 0.0068 | 0.4764 | 0.0000 | 0.0455 | 0.9927 | 0.0001 | 0.3397 | 0.9283 | 0.0729 | 0.6915 | 1.0000 | 0.2103 | 0.0003 | 0.1580 |
| schreiber2018/ismir2018 | 0.2169 | 0.5422 | 0.8194 | 0.3985 | 0.0000 | 0.0232 | 0.9306 | 0.0000 | 0.0091 | 0.4584 | 0.0000 | 0.1098 | 0.3290 | 0.5386 | 0.0487 | 0.2103 | 1.0000 | 0.0071 | 0.0497 |
| sun2021/default | 0.1789 | 0.0070 | 0.0034 | 0.0277 | 0.0000 | 0.0809 | 0.1451 | 0.0000 | 0.0005 | 0.0439 | 0.0000 | 0.0068 | 0.0149 | 0.2865 | 0.0000 | 0.0003 | 0.0071 | 1.0000 | 0.0038 |
| zplane/auftakt_v3 | 0.0102 | 0.0122 | 0.0165 | 0.0091 | 0.0000 | 0.0028 | 0.0912 | 0.0082 | 0.5573 | 0.2257 | 0.0012 | 0.6257 | 0.2333 | 0.0335 | 0.2225 | 0.1580 | 0.0497 | 0.0038 | 1.0000 |
Table 33: Paired t-test p-values, using reference annotations 1.0 as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0191 | 0.0197 | 0.1158 | 0.0000 | 0.0177 | 0.8188 | 0.0000 | 0.0087 | 0.1572 | 0.0000 | 0.0288 | 0.0813 | 0.7460 | 0.0013 | 0.0011 | 0.2169 | 0.1789 | 0.0558 |
| boeck2019/multi_task | 0.0191 | 1.0000 | 0.3775 | 0.1249 | 0.0000 | 0.0073 | 0.7633 | 0.0000 | 0.0143 | 0.2769 | 0.0000 | 0.0577 | 0.1700 | 0.8820 | 0.0106 | 0.0138 | 0.5422 | 0.0070 | 0.0671 |
| boeck2019/multi_task_hjdb | 0.0197 | 0.3775 | 1.0000 | 0.1273 | 0.0000 | 0.0059 | 0.5845 | 0.0000 | 0.0209 | 0.4038 | 0.0001 | 0.0891 | 0.2631 | 0.6985 | 0.0386 | 0.0570 | 0.8194 | 0.0034 | 0.0893 |
| boeck2020/dar | 0.1158 | 0.1249 | 0.1273 | 1.0000 | 0.0000 | 0.0086 | 0.9202 | 0.0000 | 0.0104 | 0.2220 | 0.0000 | 0.0446 | 0.1290 | 0.9639 | 0.0040 | 0.0045 | 0.3985 | 0.0277 | 0.0512 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0193 | 0.0000 | 0.0000 | 0.7757 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.0177 | 0.0073 | 0.0059 | 0.0086 | 0.0000 | 1.0000 | 0.0222 | 0.0000 | 0.0010 | 0.0071 | 0.0000 | 0.0017 | 0.0047 | 0.0171 | 0.0021 | 0.0024 | 0.0098 | 0.0403 | 0.0047 |
| gkiokas2012/default | 0.8188 | 0.7633 | 0.5845 | 0.9202 | 0.0000 | 0.0222 | 1.0000 | 0.0001 | 0.0202 | 0.2826 | 0.0000 | 0.0810 | 0.2209 | 0.9386 | 0.0816 | 0.1082 | 0.5587 | 0.3663 | 0.1156 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0193 | 0.0000 | 0.0001 | 1.0000 | 0.0787 | 0.0026 | 0.1069 | 0.0089 | 0.0008 | 0.0001 | 0.0005 | 0.0004 | 0.0001 | 0.0000 | 0.0105 |
| oliveira2010/ibt | 0.0087 | 0.0143 | 0.0209 | 0.0104 | 0.0000 | 0.0010 | 0.0202 | 0.0787 | 1.0000 | 0.1865 | 0.0102 | 0.4990 | 0.1635 | 0.0163 | 0.1317 | 0.1108 | 0.0301 | 0.0022 | 0.4232 |
| percival2014/stem | 0.1572 | 0.2769 | 0.4038 | 0.2220 | 0.0000 | 0.0071 | 0.2826 | 0.0026 | 0.1865 | 1.0000 | 0.0002 | 0.4458 | 0.9448 | 0.2866 | 0.8289 | 0.9927 | 0.4570 | 0.0476 | 0.5945 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.7757 | 0.0000 | 0.0000 | 0.1069 | 0.0102 | 0.0002 | 1.0000 | 0.0014 | 0.0003 | 0.0000 | 0.0002 | 0.0001 | 0.0001 | 0.0000 | 0.0012 |
| schreiber2014/default | 0.0288 | 0.0577 | 0.0891 | 0.0446 | 0.0000 | 0.0017 | 0.0810 | 0.0089 | 0.4990 | 0.4458 | 0.0014 | 1.0000 | 0.2496 | 0.0262 | 0.4255 | 0.3397 | 0.1098 | 0.0068 | 0.8716 |
| schreiber2017/ismir2017 | 0.0813 | 0.1700 | 0.2631 | 0.1290 | 0.0000 | 0.0047 | 0.2209 | 0.0008 | 0.1635 | 0.9448 | 0.0003 | 0.2496 | 1.0000 | 0.0587 | 0.8821 | 0.9283 | 0.3290 | 0.0149 | 0.5970 |
| schreiber2017/mirex2017 | 0.7460 | 0.8820 | 0.6985 | 0.9639 | 0.0000 | 0.0171 | 0.9386 | 0.0001 | 0.0163 | 0.2866 | 0.0000 | 0.0262 | 0.0587 | 1.0000 | 0.0184 | 0.0729 | 0.5386 | 0.2865 | 0.1314 |
| schreiber2018/cnn | 0.0013 | 0.0106 | 0.0386 | 0.0040 | 0.0000 | 0.0021 | 0.0816 | 0.0005 | 0.1317 | 0.8289 | 0.0002 | 0.4255 | 0.8821 | 0.0184 | 1.0000 | 0.6915 | 0.0487 | 0.0000 | 0.6243 |
| schreiber2018/fcn | 0.0011 | 0.0138 | 0.0570 | 0.0045 | 0.0000 | 0.0024 | 0.1082 | 0.0004 | 0.1108 | 0.9927 | 0.0001 | 0.3397 | 0.9283 | 0.0729 | 0.6915 | 1.0000 | 0.2103 | 0.0003 | 0.4977 |
| schreiber2018/ismir2018 | 0.2169 | 0.5422 | 0.8194 | 0.3985 | 0.0000 | 0.0098 | 0.5587 | 0.0001 | 0.0301 | 0.4570 | 0.0001 | 0.1098 | 0.3290 | 0.5386 | 0.0487 | 0.2103 | 1.0000 | 0.0071 | 0.1982 |
| sun2021/default | 0.1789 | 0.0070 | 0.0034 | 0.0277 | 0.0000 | 0.0403 | 0.3663 | 0.0000 | 0.0022 | 0.0476 | 0.0000 | 0.0068 | 0.0149 | 0.2865 | 0.0000 | 0.0003 | 0.0071 | 1.0000 | 0.0215 |
| zplane/auftakt_v3 | 0.0558 | 0.0671 | 0.0893 | 0.0512 | 0.0000 | 0.0047 | 0.1156 | 0.0105 | 0.4232 | 0.5945 | 0.0012 | 0.8716 | 0.5970 | 0.1314 | 0.6243 | 0.4977 | 0.1982 | 0.0215 | 1.0000 |
Table 34: Paired t-test p-values, using reference annotations 3.0 as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0191 | 0.0197 | 0.1158 | 0.0000 | 0.0118 | 0.8171 | 0.0000 | 0.0069 | 0.1541 | 0.0000 | 0.0271 | 0.0762 | 0.7198 | 0.0013 | 0.0012 | 0.2205 | 0.1696 | 0.0320 |
| boeck2019/multi_task | 0.0191 | 1.0000 | 0.3775 | 0.1249 | 0.0000 | 0.0073 | 0.7633 | 0.0000 | 0.0143 | 0.2769 | 0.0000 | 0.0577 | 0.1700 | 0.8820 | 0.0106 | 0.0138 | 0.5422 | 0.0070 | 0.0671 |
| boeck2019/multi_task_hjdb | 0.0197 | 0.3775 | 1.0000 | 0.1273 | 0.0000 | 0.0059 | 0.5845 | 0.0000 | 0.0209 | 0.4038 | 0.0001 | 0.0891 | 0.2631 | 0.6985 | 0.0386 | 0.0570 | 0.8194 | 0.0034 | 0.0893 |
| boeck2020/dar | 0.1158 | 0.1249 | 0.1273 | 1.0000 | 0.0000 | 0.0086 | 0.9202 | 0.0000 | 0.0104 | 0.2220 | 0.0000 | 0.0446 | 0.1290 | 0.9639 | 0.0040 | 0.0045 | 0.3985 | 0.0277 | 0.0512 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0300 | 0.0000 | 0.0000 | 0.8935 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.0118 | 0.0073 | 0.0059 | 0.0086 | 0.0000 | 1.0000 | 0.0156 | 0.0000 | 0.0006 | 0.0048 | 0.0000 | 0.0010 | 0.0030 | 0.0112 | 0.0013 | 0.0015 | 0.0066 | 0.0285 | 0.0023 |
| gkiokas2012/default | 0.8171 | 0.7633 | 0.5845 | 0.9202 | 0.0000 | 0.0156 | 1.0000 | 0.0001 | 0.0167 | 0.2792 | 0.0001 | 0.0776 | 0.2127 | 0.9196 | 0.0831 | 0.1102 | 0.5642 | 0.3614 | 0.0762 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0300 | 0.0000 | 0.0001 | 1.0000 | 0.0711 | 0.0018 | 0.1697 | 0.0065 | 0.0005 | 0.0000 | 0.0003 | 0.0002 | 0.0000 | 0.0000 | 0.0114 |
| oliveira2010/ibt | 0.0069 | 0.0143 | 0.0209 | 0.0104 | 0.0000 | 0.0006 | 0.0167 | 0.0711 | 1.0000 | 0.1680 | 0.0180 | 0.4709 | 0.1480 | 0.0138 | 0.1112 | 0.0935 | 0.0242 | 0.0016 | 0.4614 |
| percival2014/stem | 0.1541 | 0.2769 | 0.4038 | 0.2220 | 0.0000 | 0.0048 | 0.2792 | 0.0018 | 0.1680 | 1.0000 | 0.0004 | 0.4384 | 0.9344 | 0.2928 | 0.8425 | 0.9941 | 0.4471 | 0.0456 | 0.4916 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.8935 | 0.0000 | 0.0001 | 0.1697 | 0.0180 | 0.0004 | 1.0000 | 0.0027 | 0.0006 | 0.0001 | 0.0004 | 0.0003 | 0.0001 | 0.0000 | 0.0028 |
| schreiber2014/default | 0.0271 | 0.0577 | 0.0891 | 0.0446 | 0.0000 | 0.0010 | 0.0776 | 0.0065 | 0.4709 | 0.4384 | 0.0027 | 1.0000 | 0.2490 | 0.0261 | 0.4070 | 0.3241 | 0.1033 | 0.0062 | 0.9822 |
| schreiber2017/ismir2017 | 0.0762 | 0.1700 | 0.2631 | 0.1290 | 0.0000 | 0.0030 | 0.2127 | 0.0005 | 0.1480 | 0.9344 | 0.0006 | 0.2490 | 1.0000 | 0.0587 | 0.9125 | 0.8967 | 0.3107 | 0.0132 | 0.4905 |
| schreiber2017/mirex2017 | 0.7198 | 0.8820 | 0.6985 | 0.9639 | 0.0000 | 0.0112 | 0.9196 | 0.0000 | 0.0138 | 0.2928 | 0.0001 | 0.0261 | 0.0587 | 1.0000 | 0.0212 | 0.0807 | 0.5676 | 0.2661 | 0.0921 |
| schreiber2018/cnn | 0.0013 | 0.0106 | 0.0386 | 0.0040 | 0.0000 | 0.0013 | 0.0831 | 0.0003 | 0.1112 | 0.8425 | 0.0004 | 0.4070 | 0.9125 | 0.0212 | 1.0000 | 0.6915 | 0.0487 | 0.0000 | 0.4843 |
| schreiber2018/fcn | 0.0012 | 0.0138 | 0.0570 | 0.0045 | 0.0000 | 0.0015 | 0.1102 | 0.0002 | 0.0935 | 0.9941 | 0.0003 | 0.3241 | 0.8967 | 0.0807 | 0.6915 | 1.0000 | 0.2103 | 0.0002 | 0.3680 |
| schreiber2018/ismir2018 | 0.2205 | 0.5422 | 0.8194 | 0.3985 | 0.0000 | 0.0066 | 0.5642 | 0.0000 | 0.0242 | 0.4471 | 0.0001 | 0.1033 | 0.3107 | 0.5676 | 0.0487 | 0.2103 | 1.0000 | 0.0068 | 0.1310 |
| sun2021/default | 0.1696 | 0.0070 | 0.0034 | 0.0277 | 0.0000 | 0.0285 | 0.3614 | 0.0000 | 0.0016 | 0.0456 | 0.0000 | 0.0062 | 0.0132 | 0.2661 | 0.0000 | 0.0002 | 0.0068 | 1.0000 | 0.0105 |
| zplane/auftakt_v3 | 0.0320 | 0.0671 | 0.0893 | 0.0512 | 0.0000 | 0.0023 | 0.0762 | 0.0114 | 0.4614 | 0.4916 | 0.0028 | 0.9822 | 0.4905 | 0.0921 | 0.4843 | 0.3680 | 0.1310 | 0.0105 | 1.0000 |
Table 35: Paired t-test p-values, using reference annotations 3.0-no-dupes as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0191 | 0.0197 | 0.1158 | 0.0000 | 0.0073 | 0.8188 | 0.0000 | 0.0087 | 0.1572 | 0.0000 | 0.0288 | 0.0813 | 0.7460 | 0.0013 | 0.0011 | 0.0121 | 0.1789 | 0.0558 |
| boeck2019/multi_task | 0.0191 | 1.0000 | 0.3775 | 0.1249 | 0.0000 | 0.0028 | 0.7633 | 0.0000 | 0.0143 | 0.2769 | 0.0000 | 0.0577 | 0.1700 | 0.8820 | 0.0106 | 0.0138 | 0.0751 | 0.0070 | 0.0671 |
| boeck2019/multi_task_hjdb | 0.0197 | 0.3775 | 1.0000 | 0.1273 | 0.0000 | 0.0022 | 0.5845 | 0.0000 | 0.0209 | 0.4038 | 0.0001 | 0.0891 | 0.2631 | 0.6985 | 0.0386 | 0.0570 | 0.1855 | 0.0034 | 0.0893 |
| boeck2020/dar | 0.1158 | 0.1249 | 0.1273 | 1.0000 | 0.0000 | 0.0033 | 0.9202 | 0.0000 | 0.0104 | 0.2220 | 0.0000 | 0.0446 | 0.1290 | 0.9639 | 0.0040 | 0.0045 | 0.0312 | 0.0277 | 0.0512 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0193 | 0.0000 | 0.0000 | 0.7757 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.0073 | 0.0028 | 0.0022 | 0.0033 | 0.0000 | 1.0000 | 0.0101 | 0.0000 | 0.0004 | 0.0031 | 0.0000 | 0.0006 | 0.0019 | 0.0073 | 0.0007 | 0.0009 | 0.0011 | 0.0188 | 0.0021 |
| gkiokas2012/default | 0.8188 | 0.7633 | 0.5845 | 0.9202 | 0.0000 | 0.0101 | 1.0000 | 0.0001 | 0.0202 | 0.2826 | 0.0000 | 0.0810 | 0.2209 | 0.9386 | 0.0816 | 0.1082 | 0.1649 | 0.3663 | 0.1156 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0193 | 0.0000 | 0.0001 | 1.0000 | 0.0787 | 0.0026 | 0.1069 | 0.0089 | 0.0008 | 0.0001 | 0.0005 | 0.0004 | 0.0003 | 0.0000 | 0.0105 |
| oliveira2010/ibt | 0.0087 | 0.0143 | 0.0209 | 0.0104 | 0.0000 | 0.0004 | 0.0202 | 0.0787 | 1.0000 | 0.1865 | 0.0102 | 0.4990 | 0.1635 | 0.0163 | 0.1317 | 0.1108 | 0.0893 | 0.0022 | 0.4232 |
| percival2014/stem | 0.1572 | 0.2769 | 0.4038 | 0.2220 | 0.0000 | 0.0031 | 0.2826 | 0.0026 | 0.1865 | 1.0000 | 0.0002 | 0.4458 | 0.9448 | 0.2866 | 0.8289 | 0.9927 | 0.8924 | 0.0476 | 0.5945 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.7757 | 0.0000 | 0.0000 | 0.1069 | 0.0102 | 0.0002 | 1.0000 | 0.0014 | 0.0003 | 0.0000 | 0.0002 | 0.0001 | 0.0001 | 0.0000 | 0.0012 |
| schreiber2014/default | 0.0288 | 0.0577 | 0.0891 | 0.0446 | 0.0000 | 0.0006 | 0.0810 | 0.0089 | 0.4990 | 0.4458 | 0.0014 | 1.0000 | 0.2496 | 0.0262 | 0.4255 | 0.3397 | 0.3000 | 0.0068 | 0.8716 |
| schreiber2017/ismir2017 | 0.0813 | 0.1700 | 0.2631 | 0.1290 | 0.0000 | 0.0019 | 0.2209 | 0.0008 | 0.1635 | 0.9448 | 0.0003 | 0.2496 | 1.0000 | 0.0587 | 0.8821 | 0.9283 | 0.8043 | 0.0149 | 0.5970 |
| schreiber2017/mirex2017 | 0.7460 | 0.8820 | 0.6985 | 0.9639 | 0.0000 | 0.0073 | 0.9386 | 0.0001 | 0.0163 | 0.2866 | 0.0000 | 0.0262 | 0.0587 | 1.0000 | 0.0184 | 0.0729 | 0.1496 | 0.2865 | 0.1314 |
| schreiber2018/cnn | 0.0013 | 0.0106 | 0.0386 | 0.0040 | 0.0000 | 0.0007 | 0.0816 | 0.0005 | 0.1317 | 0.8289 | 0.0002 | 0.4255 | 0.8821 | 0.0184 | 1.0000 | 0.6915 | 0.4350 | 0.0000 | 0.6243 |
| schreiber2018/fcn | 0.0011 | 0.0138 | 0.0570 | 0.0045 | 0.0000 | 0.0009 | 0.1082 | 0.0004 | 0.1108 | 0.9927 | 0.0001 | 0.3397 | 0.9283 | 0.0729 | 0.6915 | 1.0000 | 0.7731 | 0.0003 | 0.4977 |
| schreiber2018/ismir2018 | 0.0121 | 0.0751 | 0.1855 | 0.0312 | 0.0000 | 0.0011 | 0.1649 | 0.0003 | 0.0893 | 0.8924 | 0.0001 | 0.3000 | 0.8043 | 0.1496 | 0.4350 | 0.7731 | 1.0000 | 0.0004 | 0.4565 |
| sun2021/default | 0.1789 | 0.0070 | 0.0034 | 0.0277 | 0.0000 | 0.0188 | 0.3663 | 0.0000 | 0.0022 | 0.0476 | 0.0000 | 0.0068 | 0.0149 | 0.2865 | 0.0000 | 0.0003 | 0.0004 | 1.0000 | 0.0215 |
| zplane/auftakt_v3 | 0.0558 | 0.0671 | 0.0893 | 0.0512 | 0.0000 | 0.0021 | 0.1156 | 0.0105 | 0.4232 | 0.5945 | 0.0012 | 0.8716 | 0.5970 | 0.1314 | 0.6243 | 0.4977 | 0.4565 | 0.0215 | 1.0000 |
Table 36: Paired t-test p-values, using reference annotations 2.0 as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0191 | 0.0197 | 0.1158 | 0.0000 | 0.0047 | 0.8171 | 0.0000 | 0.0069 | 0.1541 | 0.0000 | 0.0271 | 0.0762 | 0.7198 | 0.0013 | 0.0012 | 0.0124 | 0.1696 | 0.0320 |
| boeck2019/multi_task | 0.0191 | 1.0000 | 0.3775 | 0.1249 | 0.0000 | 0.0028 | 0.7633 | 0.0000 | 0.0143 | 0.2769 | 0.0000 | 0.0577 | 0.1700 | 0.8820 | 0.0106 | 0.0138 | 0.0751 | 0.0070 | 0.0671 |
| boeck2019/multi_task_hjdb | 0.0197 | 0.3775 | 1.0000 | 0.1273 | 0.0000 | 0.0022 | 0.5845 | 0.0000 | 0.0209 | 0.4038 | 0.0001 | 0.0891 | 0.2631 | 0.6985 | 0.0386 | 0.0570 | 0.1855 | 0.0034 | 0.0893 |
| boeck2020/dar | 0.1158 | 0.1249 | 0.1273 | 1.0000 | 0.0000 | 0.0033 | 0.9202 | 0.0000 | 0.0104 | 0.2220 | 0.0000 | 0.0446 | 0.1290 | 0.9639 | 0.0040 | 0.0045 | 0.0312 | 0.0277 | 0.0512 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0300 | 0.0000 | 0.0000 | 0.8935 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.0047 | 0.0028 | 0.0022 | 0.0033 | 0.0000 | 1.0000 | 0.0068 | 0.0000 | 0.0002 | 0.0020 | 0.0000 | 0.0004 | 0.0012 | 0.0046 | 0.0004 | 0.0005 | 0.0006 | 0.0127 | 0.0010 |
| gkiokas2012/default | 0.8171 | 0.7633 | 0.5845 | 0.9202 | 0.0000 | 0.0068 | 1.0000 | 0.0001 | 0.0167 | 0.2792 | 0.0001 | 0.0776 | 0.2127 | 0.9196 | 0.0831 | 0.1102 | 0.1677 | 0.3614 | 0.0762 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0300 | 0.0000 | 0.0001 | 1.0000 | 0.0711 | 0.0018 | 0.1697 | 0.0065 | 0.0005 | 0.0000 | 0.0003 | 0.0002 | 0.0002 | 0.0000 | 0.0114 |
| oliveira2010/ibt | 0.0069 | 0.0143 | 0.0209 | 0.0104 | 0.0000 | 0.0002 | 0.0167 | 0.0711 | 1.0000 | 0.1680 | 0.0180 | 0.4709 | 0.1480 | 0.0138 | 0.1112 | 0.0935 | 0.0748 | 0.0016 | 0.4614 |
| percival2014/stem | 0.1541 | 0.2769 | 0.4038 | 0.2220 | 0.0000 | 0.0020 | 0.2792 | 0.0018 | 0.1680 | 1.0000 | 0.0004 | 0.4384 | 0.9344 | 0.2928 | 0.8425 | 0.9941 | 0.8789 | 0.0456 | 0.4916 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.8935 | 0.0000 | 0.0001 | 0.1697 | 0.0180 | 0.0004 | 1.0000 | 0.0027 | 0.0006 | 0.0001 | 0.0004 | 0.0003 | 0.0002 | 0.0000 | 0.0028 |
| schreiber2014/default | 0.0271 | 0.0577 | 0.0891 | 0.0446 | 0.0000 | 0.0004 | 0.0776 | 0.0065 | 0.4709 | 0.4384 | 0.0027 | 1.0000 | 0.2490 | 0.0261 | 0.4070 | 0.3241 | 0.2863 | 0.0062 | 0.9822 |
| schreiber2017/ismir2017 | 0.0762 | 0.1700 | 0.2631 | 0.1290 | 0.0000 | 0.0012 | 0.2127 | 0.0005 | 0.1480 | 0.9344 | 0.0006 | 0.2490 | 1.0000 | 0.0587 | 0.9125 | 0.8967 | 0.7757 | 0.0132 | 0.4905 |
| schreiber2017/mirex2017 | 0.7198 | 0.8820 | 0.6985 | 0.9639 | 0.0000 | 0.0046 | 0.9196 | 0.0000 | 0.0138 | 0.2928 | 0.0001 | 0.0261 | 0.0587 | 1.0000 | 0.0212 | 0.0807 | 0.1621 | 0.2661 | 0.0921 |
| schreiber2018/cnn | 0.0013 | 0.0106 | 0.0386 | 0.0040 | 0.0000 | 0.0004 | 0.0831 | 0.0003 | 0.1112 | 0.8425 | 0.0004 | 0.4070 | 0.9125 | 0.0212 | 1.0000 | 0.6915 | 0.4350 | 0.0000 | 0.4843 |
| schreiber2018/fcn | 0.0012 | 0.0138 | 0.0570 | 0.0045 | 0.0000 | 0.0005 | 0.1102 | 0.0002 | 0.0935 | 0.9941 | 0.0003 | 0.3241 | 0.8967 | 0.0807 | 0.6915 | 1.0000 | 0.7731 | 0.0002 | 0.3680 |
| schreiber2018/ismir2018 | 0.0124 | 0.0751 | 0.1855 | 0.0312 | 0.0000 | 0.0006 | 0.1677 | 0.0002 | 0.0748 | 0.8789 | 0.0002 | 0.2863 | 0.7757 | 0.1621 | 0.4350 | 0.7731 | 1.0000 | 0.0004 | 0.3417 |
| sun2021/default | 0.1696 | 0.0070 | 0.0034 | 0.0277 | 0.0000 | 0.0127 | 0.3614 | 0.0000 | 0.0016 | 0.0456 | 0.0000 | 0.0062 | 0.0132 | 0.2661 | 0.0000 | 0.0002 | 0.0004 | 1.0000 | 0.0105 |
| zplane/auftakt_v3 | 0.0320 | 0.0671 | 0.0893 | 0.0512 | 0.0000 | 0.0010 | 0.0762 | 0.0114 | 0.4614 | 0.4916 | 0.0028 | 0.9822 | 0.4905 | 0.0921 | 0.4843 | 0.3680 | 0.3417 | 0.0105 | 1.0000 |
Table 37: Paired t-test p-values, using reference annotations 2.0-no-dupes as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0191 | 0.0197 | 0.1158 | 0.0000 | 0.2769 | 0.3251 | 0.0000 | 0.0087 | 0.0349 | 0.0000 | 0.0288 | 0.0813 | 0.7460 | 0.0013 | 0.0011 | 0.2169 | 0.1789 | 0.0231 |
| boeck2019/multi_task | 0.0191 | 1.0000 | 0.3775 | 0.1249 | 0.0000 | 0.1676 | 0.6080 | 0.0000 | 0.0143 | 0.0665 | 0.0000 | 0.0577 | 0.1700 | 0.8820 | 0.0106 | 0.0138 | 0.5422 | 0.0070 | 0.0279 |
| boeck2019/multi_task_hjdb | 0.0197 | 0.3775 | 1.0000 | 0.1273 | 0.0000 | 0.1452 | 0.7862 | 0.0000 | 0.0209 | 0.0817 | 0.0000 | 0.0891 | 0.2631 | 0.6985 | 0.0386 | 0.0570 | 0.8194 | 0.0034 | 0.0382 |
| boeck2020/dar | 0.1158 | 0.1249 | 0.1273 | 1.0000 | 0.0000 | 0.1873 | 0.4868 | 0.0000 | 0.0104 | 0.0510 | 0.0000 | 0.0446 | 0.1290 | 0.9639 | 0.0040 | 0.0045 | 0.3985 | 0.0277 | 0.0204 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0057 | 0.0000 | 0.0000 | 0.9621 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.2769 | 0.1676 | 0.1452 | 0.1873 | 0.0000 | 1.0000 | 0.1533 | 0.0005 | 0.0185 | 0.0316 | 0.0000 | 0.0521 | 0.0992 | 0.2474 | 0.0704 | 0.0811 | 0.1763 | 0.4121 | 0.0469 |
| gkiokas2012/default | 0.3251 | 0.6080 | 0.7862 | 0.4868 | 0.0000 | 0.1533 | 1.0000 | 0.0001 | 0.0514 | 0.1294 | 0.0000 | 0.2183 | 0.5250 | 0.6183 | 0.3486 | 0.4448 | 0.9284 | 0.1183 | 0.1571 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0057 | 0.0005 | 0.0001 | 1.0000 | 0.0290 | 0.0089 | 0.1286 | 0.0028 | 0.0002 | 0.0000 | 0.0001 | 0.0001 | 0.0000 | 0.0000 | 0.0065 |
| oliveira2010/ibt | 0.0087 | 0.0143 | 0.0209 | 0.0104 | 0.0000 | 0.0185 | 0.0514 | 0.0290 | 1.0000 | 0.5709 | 0.0063 | 0.4990 | 0.1635 | 0.0163 | 0.1317 | 0.1108 | 0.0301 | 0.0022 | 0.5446 |
| percival2014/stem | 0.0349 | 0.0665 | 0.0817 | 0.0510 | 0.0000 | 0.0316 | 0.1294 | 0.0089 | 0.5709 | 1.0000 | 0.0010 | 0.8922 | 0.4102 | 0.0744 | 0.4124 | 0.3284 | 0.1150 | 0.0093 | 0.9520 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.9621 | 0.0000 | 0.0000 | 0.1286 | 0.0063 | 0.0010 | 1.0000 | 0.0009 | 0.0002 | 0.0000 | 0.0001 | 0.0001 | 0.0000 | 0.0000 | 0.0012 |
| schreiber2014/default | 0.0288 | 0.0577 | 0.0891 | 0.0446 | 0.0000 | 0.0521 | 0.2183 | 0.0028 | 0.4990 | 0.8922 | 0.0009 | 1.0000 | 0.2496 | 0.0262 | 0.4255 | 0.3397 | 0.1098 | 0.0068 | 0.9299 |
| schreiber2017/ismir2017 | 0.0813 | 0.1700 | 0.2631 | 0.1290 | 0.0000 | 0.0992 | 0.5250 | 0.0002 | 0.1635 | 0.4102 | 0.0002 | 0.2496 | 1.0000 | 0.0587 | 0.8821 | 0.9283 | 0.3290 | 0.0149 | 0.4170 |
| schreiber2017/mirex2017 | 0.7460 | 0.8820 | 0.6985 | 0.9639 | 0.0000 | 0.2474 | 0.6183 | 0.0000 | 0.0163 | 0.0744 | 0.0000 | 0.0262 | 0.0587 | 1.0000 | 0.0184 | 0.0729 | 0.5386 | 0.2865 | 0.0699 |
| schreiber2018/cnn | 0.0013 | 0.0106 | 0.0386 | 0.0040 | 0.0000 | 0.0704 | 0.3486 | 0.0001 | 0.1317 | 0.4124 | 0.0001 | 0.4255 | 0.8821 | 0.0184 | 1.0000 | 0.6915 | 0.0487 | 0.0000 | 0.4186 |
| schreiber2018/fcn | 0.0011 | 0.0138 | 0.0570 | 0.0045 | 0.0000 | 0.0811 | 0.4448 | 0.0001 | 0.1108 | 0.3284 | 0.0001 | 0.3397 | 0.9283 | 0.0729 | 0.6915 | 1.0000 | 0.2103 | 0.0003 | 0.3101 |
| schreiber2018/ismir2018 | 0.2169 | 0.5422 | 0.8194 | 0.3985 | 0.0000 | 0.1763 | 0.9284 | 0.0000 | 0.0301 | 0.1150 | 0.0000 | 0.1098 | 0.3290 | 0.5386 | 0.0487 | 0.2103 | 1.0000 | 0.0071 | 0.1047 |
| sun2021/default | 0.1789 | 0.0070 | 0.0034 | 0.0277 | 0.0000 | 0.4121 | 0.1183 | 0.0000 | 0.0022 | 0.0093 | 0.0000 | 0.0068 | 0.0149 | 0.2865 | 0.0000 | 0.0003 | 0.0071 | 1.0000 | 0.0080 |
| zplane/auftakt_v3 | 0.0231 | 0.0279 | 0.0382 | 0.0204 | 0.0000 | 0.0469 | 0.1571 | 0.0065 | 0.5446 | 0.9520 | 0.0012 | 0.9299 | 0.4170 | 0.0699 | 0.4186 | 0.3101 | 0.1047 | 0.0080 | 1.0000 |
Table 38: Paired t-test p-values, using reference annotations 4.0 as groundtruth with OE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
OE1 on cvar-Subsets
How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?
OE1 on cvar-Subsets for 1.0 based on cvar-Values from 1.0
Figure 77: Mean OE1 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 on cvar-Subsets for 2.0 based on cvar-Values from 1.0
Figure 78: Mean OE1 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0
Figure 79: Mean OE1 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 on cvar-Subsets for 3.0 based on cvar-Values from 1.0
Figure 80: Mean OE1 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0
Figure 81: Mean OE1 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 on cvar-Subsets for 4.0 based on cvar-Values from 1.0
Figure 82: Mean OE1 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on cvar-Subsets
How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?
OE2 on cvar-Subsets for 1.0 based on cvar-Values from 1.0
Figure 83: Mean OE2 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on cvar-Subsets for 2.0 based on cvar-Values from 1.0
Figure 84: Mean OE2 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0
Figure 85: Mean OE2 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on cvar-Subsets for 3.0 based on cvar-Values from 1.0
Figure 86: Mean OE2 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0
Figure 87: Mean OE2 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on cvar-Subsets for 4.0 based on cvar-Values from 1.0
Figure 88: Mean OE2 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 on Tempo-Subsets
How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean OE1 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.
OE1 on Tempo-Subsets for 1.0
Figure 89: Mean OE1 for estimates compared to version 1.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 on Tempo-Subsets for 2.0
Figure 90: Mean OE1 for estimates compared to version 2.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 on Tempo-Subsets for 2.0-no-dupes
Figure 91: Mean OE1 for estimates compared to version 2.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 on Tempo-Subsets for 3.0
Figure 92: Mean OE1 for estimates compared to version 3.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 on Tempo-Subsets for 3.0-no-dupes
Figure 93: Mean OE1 for estimates compared to version 3.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 on Tempo-Subsets for 4.0
Figure 94: Mean OE1 for estimates compared to version 4.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on Tempo-Subsets
How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean OE2 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.
OE2 on Tempo-Subsets for 1.0
Figure 95: Mean OE2 for estimates compared to version 1.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on Tempo-Subsets for 2.0
Figure 96: Mean OE2 for estimates compared to version 2.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on Tempo-Subsets for 2.0-no-dupes
Figure 97: Mean OE2 for estimates compared to version 2.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on Tempo-Subsets for 3.0
Figure 98: Mean OE2 for estimates compared to version 3.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on Tempo-Subsets for 3.0-no-dupes
Figure 99: Mean OE2 for estimates compared to version 3.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
OE2 on Tempo-Subsets for 4.0
Figure 100: Mean OE2 for estimates compared to version 4.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE1 for Tempo
When fitting a generalized additive model (GAM) to OE1-values and a ground truth, what OE1 can we expect with confidence?
Estimated OE1 for Tempo for 1.0
Predictions of GAMs trained on OE1 for estimates for reference 1.0.
Figure 101: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE1 for Tempo for 2.0
Predictions of GAMs trained on OE1 for estimates for reference 2.0.
Figure 102: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE1 for Tempo for 2.0-no-dupes
Predictions of GAMs trained on OE1 for estimates for reference 2.0-no-dupes.
Figure 103: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE1 for Tempo for 3.0
Predictions of GAMs trained on OE1 for estimates for reference 3.0.
Figure 104: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE1 for Tempo for 3.0-no-dupes
Predictions of GAMs trained on OE1 for estimates for reference 3.0-no-dupes.
Figure 105: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE1 for Tempo for 4.0
Predictions of GAMs trained on OE1 for estimates for reference 4.0.
Figure 106: OE1 predictions of a generalized additive model (GAM) fit to OE1 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE2 for Tempo
When fitting a generalized additive model (GAM) to OE2-values and a ground truth, what OE2 can we expect with confidence?
Estimated OE2 for Tempo for 1.0
Predictions of GAMs trained on OE2 for estimates for reference 1.0.
Figure 107: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE2 for Tempo for 2.0
Predictions of GAMs trained on OE2 for estimates for reference 2.0.
Figure 108: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE2 for Tempo for 2.0-no-dupes
Predictions of GAMs trained on OE2 for estimates for reference 2.0-no-dupes.
Figure 109: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE2 for Tempo for 3.0
Predictions of GAMs trained on OE2 for estimates for reference 3.0.
Figure 110: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE2 for Tempo for 3.0-no-dupes
Predictions of GAMs trained on OE2 for estimates for reference 3.0-no-dupes.
Figure 111: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated OE2 for Tempo for 4.0
Predictions of GAMs trained on OE2 for estimates for reference 4.0.
Figure 112: OE2 predictions of a generalized additive model (GAM) fit to OE2 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
OE1 for ‘tag_open’ Tags
How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.
OE1 for ‘tag_open’ Tags for 1.0
Figure 113: OE1 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.
OE1 for ‘tag_open’ Tags for 2.0
Figure 114: OE1 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.
OE1 for ‘tag_open’ Tags for 2.0-no-dupes
Figure 115: OE1 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.
OE1 for ‘tag_open’ Tags for 3.0
Figure 116: OE1 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.
OE1 for ‘tag_open’ Tags for 3.0-no-dupes
Figure 117: OE1 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.
OE1 for ‘tag_open’ Tags for 4.0
Figure 118: OE1 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.
OE2 for ‘tag_open’ Tags
How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.
OE2 for ‘tag_open’ Tags for 1.0
Figure 119: OE2 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.
OE2 for ‘tag_open’ Tags for 2.0
Figure 120: OE2 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.
OE2 for ‘tag_open’ Tags for 2.0-no-dupes
Figure 121: OE2 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.
OE2 for ‘tag_open’ Tags for 3.0
Figure 122: OE2 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.
OE2 for ‘tag_open’ Tags for 3.0-no-dupes
Figure 123: OE2 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.
OE2 for ‘tag_open’ Tags for 4.0
Figure 124: OE2 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.
AOE1 and AOE2
AOE1 is defined as absolute octave error between an estimate and a reference value: AOE1(E) = |log2(E/R)|.
AOE2 is the minimum of AOE1 allowing the octave errors 2, 3, 1/2, and 1/3: AOE2(E) = min(AOE1(E), AOE1(2E), AOE1(3E), AOE1(½E), AOE1(⅓E)).
Mean AOE1/AOE2 Results for 1.0
| Estimator | AOE1_MEAN | AOE1_STDEV | AOE2_MEAN | AOE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | 0.0256 | 0.1136 | 0.0151 | 0.0468 |
| boeck2019/multi_task | 0.0308 | 0.1373 | 0.0143 | 0.0464 |
| sun2021/default | 0.0321 | 0.1447 | 0.0152 | 0.0511 |
| boeck2019/multi_task_hjdb | 0.0331 | 0.1437 | 0.0147 | 0.0476 |
| schreiber2018/cnn | 0.0433 | 0.1720 | 0.0161 | 0.0551 |
| schreiber2018/fcn | 0.0647 | 0.2224 | 0.0155 | 0.0546 |
| schreiber2018/ismir2018 | 0.0650 | 0.2227 | 0.0155 | 0.0558 |
| schreiber2017/mirex2017 | 0.0921 | 0.2647 | 0.0195 | 0.0640 |
| schreiber2017/ismir2017 | 0.1414 | 0.3289 | 0.0213 | 0.0684 |
| boeck2015/tempodetector2016_default | 0.1748 | 0.3689 | 0.0182 | 0.0470 |
| davies2009/mirex_qm_tempotracker | 0.3026 | 0.4284 | 0.0436 | 0.0823 |
| zplane/auftakt_v3 | 0.3134 | 0.4450 | 0.0283 | 0.0793 |
| oliveira2010/ibt | 0.3213 | 0.4367 | 0.0417 | 0.0836 |
| klapuri2006/percival2014 | 0.3238 | 0.4425 | 0.0380 | 0.0950 |
| schreiber2014/default | 0.3263 | 0.4515 | 0.0258 | 0.0800 |
| percival2014/stem | 0.3367 | 0.4767 | 0.0259 | 0.0722 |
| scheirer1998/percival2014 | 0.3431 | 0.4305 | 0.0753 | 0.1372 |
| echonest/version_3_2_1 | 0.3888 | 0.4899 | 0.0553 | 0.1204 |
| gkiokas2012/default | 0.4115 | 0.5460 | 0.0198 | 0.0634 |
Table 39: Mean AOE1/AOE2 for estimates compared to version 1.0 ordered by mean.
Raw data AOE1: CSV JSON LATEX PICKLE
Raw data AOE2: CSV JSON LATEX PICKLE
AOE1 distribution for 1.0
Figure 125: AOE1 for estimates compared to version 1.0. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 distribution for 1.0
Figure 126: AOE2 for estimates compared to version 1.0. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Mean AOE1/AOE2 Results for 2.0
| Estimator | AOE1_MEAN | AOE1_STDEV | AOE2_MEAN | AOE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | 0.0131 | 0.0847 | 0.0059 | 0.0058 |
| boeck2019/multi_task | 0.0203 | 0.1191 | 0.0059 | 0.0061 |
| boeck2019/multi_task_hjdb | 0.0211 | 0.1213 | 0.0063 | 0.0138 |
| sun2021/default | 0.0243 | 0.1235 | 0.0104 | 0.0197 |
| schreiber2018/cnn | 0.0344 | 0.1576 | 0.0096 | 0.0294 |
| schreiber2018/fcn | 0.0556 | 0.2101 | 0.0098 | 0.0252 |
| schreiber2018/ismir2018 | 0.0567 | 0.2128 | 0.0094 | 0.0302 |
| schreiber2017/mirex2017 | 0.0830 | 0.2609 | 0.0107 | 0.0437 |
| schreiber2017/ismir2017 | 0.1296 | 0.3243 | 0.0124 | 0.0513 |
| boeck2015/tempodetector2016_default | 0.1628 | 0.3708 | 0.0055 | 0.0084 |
| davies2009/mirex_qm_tempotracker | 0.3010 | 0.4320 | 0.0380 | 0.0743 |
| zplane/auftakt_v3 | 0.3064 | 0.4458 | 0.0207 | 0.0687 |
| klapuri2006/percival2014 | 0.3171 | 0.4450 | 0.0301 | 0.0882 |
| oliveira2010/ibt | 0.3173 | 0.4391 | 0.0357 | 0.0781 |
| schreiber2014/default | 0.3186 | 0.4541 | 0.0171 | 0.0667 |
| percival2014/stem | 0.3294 | 0.4785 | 0.0176 | 0.0588 |
| scheirer1998/percival2014 | 0.3399 | 0.4314 | 0.0712 | 0.1373 |
| echonest/version_3_2_1 | 0.3839 | 0.4939 | 0.0471 | 0.1138 |
| gkiokas2012/default | 0.4057 | 0.5478 | 0.0126 | 0.0421 |
Table 40: Mean AOE1/AOE2 for estimates compared to version 2.0 ordered by mean.
Raw data AOE1: CSV JSON LATEX PICKLE
Raw data AOE2: CSV JSON LATEX PICKLE
AOE1 distribution for 2.0
Figure 127: AOE1 for estimates compared to version 2.0. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 distribution for 2.0
Figure 128: AOE2 for estimates compared to version 2.0. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Mean AOE1/AOE2 Results for 2.0-no-dupes
| Estimator | AOE1_MEAN | AOE1_STDEV | AOE2_MEAN | AOE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | 0.0131 | 0.0847 | 0.0059 | 0.0058 |
| boeck2019/multi_task | 0.0203 | 0.1191 | 0.0059 | 0.0061 |
| boeck2019/multi_task_hjdb | 0.0211 | 0.1213 | 0.0063 | 0.0138 |
| sun2021/default | 0.0244 | 0.1246 | 0.0103 | 0.0196 |
| schreiber2018/cnn | 0.0349 | 0.1591 | 0.0096 | 0.0297 |
| schreiber2018/fcn | 0.0565 | 0.2119 | 0.0098 | 0.0254 |
| schreiber2018/ismir2018 | 0.0576 | 0.2147 | 0.0095 | 0.0305 |
| schreiber2017/mirex2017 | 0.0845 | 0.2631 | 0.0108 | 0.0441 |
| schreiber2017/ismir2017 | 0.1319 | 0.3269 | 0.0126 | 0.0518 |
| boeck2015/tempodetector2016_default | 0.1659 | 0.3736 | 0.0056 | 0.0085 |
| davies2009/mirex_qm_tempotracker | 0.3034 | 0.4329 | 0.0384 | 0.0750 |
| zplane/auftakt_v3 | 0.3096 | 0.4473 | 0.0205 | 0.0683 |
| oliveira2010/ibt | 0.3203 | 0.4400 | 0.0362 | 0.0788 |
| klapuri2006/percival2014 | 0.3204 | 0.4463 | 0.0302 | 0.0887 |
| schreiber2014/default | 0.3231 | 0.4557 | 0.0173 | 0.0673 |
| percival2014/stem | 0.3326 | 0.4799 | 0.0178 | 0.0593 |
| scheirer1998/percival2014 | 0.3386 | 0.4311 | 0.0709 | 0.1368 |
| echonest/version_3_2_1 | 0.3888 | 0.4956 | 0.0472 | 0.1137 |
| gkiokas2012/default | 0.4117 | 0.5499 | 0.0127 | 0.0425 |
Table 41: Mean AOE1/AOE2 for estimates compared to version 2.0-no-dupes ordered by mean.
Raw data AOE1: CSV JSON LATEX PICKLE
Raw data AOE2: CSV JSON LATEX PICKLE
AOE1 distribution for 2.0-no-dupes
Figure 129: AOE1 for estimates compared to version 2.0-no-dupes. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 distribution for 2.0-no-dupes
Figure 130: AOE2 for estimates compared to version 2.0-no-dupes. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Mean AOE1/AOE2 Results for 3.0
| Estimator | AOE1_MEAN | AOE1_STDEV | AOE2_MEAN | AOE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | 0.0109 | 0.0848 | 0.0037 | 0.0044 |
| boeck2019/multi_task | 0.0179 | 0.1198 | 0.0034 | 0.0039 |
| boeck2019/multi_task_hjdb | 0.0188 | 0.1220 | 0.0038 | 0.0125 |
| sun2021/default | 0.0225 | 0.1236 | 0.0086 | 0.0193 |
| schreiber2018/cnn | 0.0324 | 0.1583 | 0.0076 | 0.0292 |
| schreiber2018/fcn | 0.0537 | 0.2109 | 0.0077 | 0.0252 |
| schreiber2018/ismir2018 | 0.0550 | 0.2138 | 0.0075 | 0.0308 |
| schreiber2017/mirex2017 | 0.0803 | 0.2618 | 0.0078 | 0.0439 |
| schreiber2017/ismir2017 | 0.1270 | 0.3252 | 0.0096 | 0.0516 |
| boeck2015/tempodetector2016_default | 0.1648 | 0.3705 | 0.0072 | 0.0066 |
| davies2009/mirex_qm_tempotracker | 0.3016 | 0.4320 | 0.0383 | 0.0740 |
| zplane/auftakt_v3 | 0.3046 | 0.4473 | 0.0179 | 0.0685 |
| oliveira2010/ibt | 0.3161 | 0.4403 | 0.0339 | 0.0785 |
| klapuri2006/percival2014 | 0.3165 | 0.4458 | 0.0286 | 0.0885 |
| schreiber2014/default | 0.3168 | 0.4557 | 0.0142 | 0.0669 |
| percival2014/stem | 0.3276 | 0.4799 | 0.0150 | 0.0589 |
| scheirer1998/percival2014 | 0.3398 | 0.4315 | 0.0707 | 0.1373 |
| echonest/version_3_2_1 | 0.3822 | 0.4949 | 0.0450 | 0.1149 |
| gkiokas2012/default | 0.4046 | 0.5490 | 0.0105 | 0.0417 |
Table 42: Mean AOE1/AOE2 for estimates compared to version 3.0 ordered by mean.
Raw data AOE1: CSV JSON LATEX PICKLE
Raw data AOE2: CSV JSON LATEX PICKLE
AOE1 distribution for 3.0
Figure 131: AOE1 for estimates compared to version 3.0. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 distribution for 3.0
Figure 132: AOE2 for estimates compared to version 3.0. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Mean AOE1/AOE2 Results for 3.0-no-dupes
| Estimator | AOE1_MEAN | AOE1_STDEV | AOE2_MEAN | AOE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | 0.0109 | 0.0848 | 0.0037 | 0.0044 |
| boeck2019/multi_task | 0.0179 | 0.1198 | 0.0034 | 0.0039 |
| boeck2019/multi_task_hjdb | 0.0188 | 0.1220 | 0.0038 | 0.0125 |
| sun2021/default | 0.0226 | 0.1247 | 0.0085 | 0.0192 |
| schreiber2018/cnn | 0.0330 | 0.1597 | 0.0076 | 0.0295 |
| schreiber2018/fcn | 0.0546 | 0.2128 | 0.0078 | 0.0254 |
| schreiber2018/ismir2018 | 0.0559 | 0.2157 | 0.0076 | 0.0311 |
| schreiber2017/mirex2017 | 0.0818 | 0.2640 | 0.0079 | 0.0443 |
| schreiber2017/ismir2017 | 0.1294 | 0.3278 | 0.0097 | 0.0521 |
| boeck2015/tempodetector2016_default | 0.1678 | 0.3733 | 0.0072 | 0.0066 |
| davies2009/mirex_qm_tempotracker | 0.3040 | 0.4329 | 0.0386 | 0.0747 |
| zplane/auftakt_v3 | 0.3079 | 0.4489 | 0.0177 | 0.0681 |
| oliveira2010/ibt | 0.3191 | 0.4413 | 0.0343 | 0.0792 |
| klapuri2006/percival2014 | 0.3199 | 0.4472 | 0.0287 | 0.0889 |
| schreiber2014/default | 0.3213 | 0.4574 | 0.0144 | 0.0676 |
| percival2014/stem | 0.3308 | 0.4813 | 0.0153 | 0.0594 |
| scheirer1998/percival2014 | 0.3385 | 0.4313 | 0.0704 | 0.1368 |
| echonest/version_3_2_1 | 0.3871 | 0.4967 | 0.0451 | 0.1148 |
| gkiokas2012/default | 0.4107 | 0.5511 | 0.0106 | 0.0421 |
Table 43: Mean AOE1/AOE2 for estimates compared to version 3.0-no-dupes ordered by mean.
Raw data AOE1: CSV JSON LATEX PICKLE
Raw data AOE2: CSV JSON LATEX PICKLE
AOE1 distribution for 3.0-no-dupes
Figure 133: AOE1 for estimates compared to version 3.0-no-dupes. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 distribution for 3.0-no-dupes
Figure 134: AOE2 for estimates compared to version 3.0-no-dupes. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Mean AOE1/AOE2 Results for 4.0
| Estimator | AOE1_MEAN | AOE1_STDEV | AOE2_MEAN | AOE2_STDEV |
|---|---|---|---|---|
| boeck2020/dar | 0.0270 | 0.1343 | 0.0093 | 0.0263 |
| boeck2019/multi_task | 0.0305 | 0.1490 | 0.0085 | 0.0264 |
| sun2021/default | 0.0339 | 0.1612 | 0.0098 | 0.0318 |
| boeck2019/multi_task_hjdb | 0.0344 | 0.1598 | 0.0088 | 0.0281 |
| schreiber2018/cnn | 0.0425 | 0.1806 | 0.0102 | 0.0389 |
| schreiber2018/ismir2018 | 0.0646 | 0.2303 | 0.0095 | 0.0395 |
| schreiber2018/fcn | 0.0659 | 0.2336 | 0.0095 | 0.0360 |
| schreiber2017/mirex2017 | 0.0932 | 0.2737 | 0.0135 | 0.0505 |
| schreiber2017/ismir2017 | 0.1426 | 0.3362 | 0.0152 | 0.0567 |
| boeck2015/tempodetector2016_default | 0.1712 | 0.3713 | 0.0121 | 0.0264 |
| davies2009/mirex_qm_tempotracker | 0.2975 | 0.4296 | 0.0382 | 0.0730 |
| zplane/auftakt_v3 | 0.3084 | 0.4476 | 0.0214 | 0.0654 |
| oliveira2010/ibt | 0.3163 | 0.4386 | 0.0359 | 0.0755 |
| klapuri2006/percival2014 | 0.3183 | 0.4449 | 0.0311 | 0.0852 |
| schreiber2014/default | 0.3212 | 0.4539 | 0.0197 | 0.0702 |
| percival2014/stem | 0.3309 | 0.4783 | 0.0202 | 0.0623 |
| scheirer1998/percival2014 | 0.3402 | 0.4320 | 0.0712 | 0.1356 |
| echonest/version_3_2_1 | 0.3852 | 0.4927 | 0.0496 | 0.1159 |
| gkiokas2012/default | 0.4061 | 0.5501 | 0.0125 | 0.0418 |
Table 44: Mean AOE1/AOE2 for estimates compared to version 4.0 ordered by mean.
Raw data AOE1: CSV JSON LATEX PICKLE
Raw data AOE2: CSV JSON LATEX PICKLE
AOE1 distribution for 4.0
Figure 135: AOE1 for estimates compared to version 4.0. Shown are the mean AOE1 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 distribution for 4.0
Figure 136: AOE2 for estimates compared to version 4.0. Shown are the mean AOE2 and an empirical distribution of the sample, using kernel density estimation (KDE).
CSV JSON LATEX PICKLE SVG PDF PNG
Significance of Differences
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0268 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.4058 | 0.2964 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0611 | 0.0001 | 0.0000 | 0.8386 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.4058 | 1.0000 | 0.1247 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1183 | 0.0001 | 0.0002 | 0.8701 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.2964 | 0.1247 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0057 | 0.0000 | 0.0000 | 0.1955 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.1215 | 0.1767 | 0.0445 | 0.0232 | 0.1422 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4642 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.1913 | 0.0001 | 0.0000 | 0.0016 | 0.0077 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1913 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1215 | 0.0001 | 0.0000 | 1.0000 | 0.6905 | 0.1382 | 0.2342 | 0.7689 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1114 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1767 | 0.0000 | 0.0000 | 0.6905 | 1.0000 | 0.0849 | 0.1832 | 0.5806 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2140 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0445 | 0.0016 | 0.0000 | 0.1382 | 0.0849 | 1.0000 | 0.6524 | 0.2608 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0072 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0232 | 0.0077 | 0.0014 | 0.2342 | 0.1832 | 0.6524 | 1.0000 | 0.3117 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0873 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1422 | 0.0001 | 0.0000 | 0.7689 | 0.5806 | 0.2608 | 0.3117 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1283 |
| schreiber2017/ismir2017 | 0.0268 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0169 | 0.0159 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.0611 | 0.1183 | 0.0057 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0116 | 0.0062 | 0.0907 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0001 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0169 | 0.0116 | 1.0000 | 0.9708 | 0.0003 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0159 | 0.0062 | 0.9708 | 1.0000 | 0.0001 | 0.0000 |
| sun2021/default | 0.0000 | 0.8386 | 0.8701 | 0.1955 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0907 | 0.0003 | 0.0001 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4642 | 0.0000 | 0.0000 | 0.1114 | 0.2140 | 0.0072 | 0.0873 | 0.1283 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 45: Paired t-test p-values, using reference annotations 1.0 as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0136 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.7786 | 0.1816 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0346 | 0.0000 | 0.0000 | 0.4559 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.7786 | 1.0000 | 0.1088 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0391 | 0.0000 | 0.0000 | 0.5249 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1816 | 0.1088 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0009 | 0.0000 | 0.0000 | 0.0194 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.2803 | 0.2996 | 0.1297 | 0.0380 | 0.3557 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8410 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.2013 | 0.0001 | 0.0001 | 0.0010 | 0.0148 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2013 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0023 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2803 | 0.0001 | 0.0000 | 1.0000 | 0.9454 | 0.2071 | 0.1679 | 0.9757 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0734 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2996 | 0.0001 | 0.0000 | 0.9454 | 1.0000 | 0.2025 | 0.1596 | 0.9396 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0765 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1297 | 0.0010 | 0.0000 | 0.2071 | 0.2025 | 1.0000 | 0.4723 | 0.2491 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0081 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0380 | 0.0148 | 0.0023 | 0.1679 | 0.1596 | 0.4723 | 1.0000 | 0.1874 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0444 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.3557 | 0.0001 | 0.0000 | 0.9757 | 0.9396 | 0.2491 | 0.1874 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1566 |
| schreiber2017/ismir2017 | 0.0136 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0220 | 0.0276 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.0346 | 0.0391 | 0.0009 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0126 | 0.0053 | 0.1368 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0220 | 0.0126 | 1.0000 | 0.8801 | 0.0005 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0276 | 0.0053 | 0.8801 | 1.0000 | 0.0002 | 0.0000 |
| sun2021/default | 0.0000 | 0.4559 | 0.5249 | 0.0194 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1368 | 0.0005 | 0.0002 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.8410 | 0.0000 | 0.0000 | 0.0734 | 0.0765 | 0.0081 | 0.0444 | 0.1566 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 46: Paired t-test p-values, using reference annotations 3.0 as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0138 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.7786 | 0.1816 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0346 | 0.0000 | 0.0000 | 0.4559 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.7786 | 1.0000 | 0.1088 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0391 | 0.0000 | 0.0000 | 0.5249 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1816 | 0.1088 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0009 | 0.0000 | 0.0000 | 0.0194 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.2608 | 0.2910 | 0.1224 | 0.0576 | 0.3010 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7987 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.1864 | 0.0001 | 0.0000 | 0.0008 | 0.0078 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1864 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2608 | 0.0001 | 0.0000 | 1.0000 | 0.9007 | 0.2099 | 0.2392 | 0.8699 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0782 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2910 | 0.0000 | 0.0000 | 0.9007 | 1.0000 | 0.1895 | 0.2219 | 0.8093 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0911 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1224 | 0.0008 | 0.0000 | 0.2099 | 0.1895 | 1.0000 | 0.5819 | 0.3122 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0083 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0576 | 0.0078 | 0.0010 | 0.2392 | 0.2219 | 0.5819 | 1.0000 | 0.2883 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0715 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.3010 | 0.0001 | 0.0000 | 0.8699 | 0.8093 | 0.3122 | 0.2883 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1248 |
| schreiber2017/ismir2017 | 0.0138 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0218 | 0.0273 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.0346 | 0.0391 | 0.0009 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0126 | 0.0053 | 0.1299 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0218 | 0.0126 | 1.0000 | 0.8801 | 0.0004 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0273 | 0.0053 | 0.8801 | 1.0000 | 0.0002 | 0.0000 |
| sun2021/default | 0.0000 | 0.4559 | 0.5249 | 0.0194 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1299 | 0.0004 | 0.0002 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7987 | 0.0000 | 0.0000 | 0.0782 | 0.0911 | 0.0083 | 0.0715 | 0.1248 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 47: Paired t-test p-values, using reference annotations 3.0-no-dupes as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0298 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.7882 | 0.1684 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0400 | 0.0000 | 0.0000 | 0.5149 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.7882 | 1.0000 | 0.1007 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0447 | 0.0000 | 0.0000 | 0.5867 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1684 | 0.1007 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 | 0.0000 | 0.0000 | 0.0240 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.2437 | 0.2427 | 0.0977 | 0.0340 | 0.2843 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7189 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.2125 | 0.0001 | 0.0001 | 0.0010 | 0.0118 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2125 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0019 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2437 | 0.0001 | 0.0000 | 1.0000 | 0.9722 | 0.1614 | 0.1787 | 0.8634 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1053 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2427 | 0.0001 | 0.0000 | 0.9722 | 1.0000 | 0.1809 | 0.1808 | 0.8903 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0912 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0977 | 0.0010 | 0.0000 | 0.1614 | 0.1809 | 1.0000 | 0.5293 | 0.2486 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0079 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0340 | 0.0118 | 0.0019 | 0.1787 | 0.1808 | 0.5293 | 1.0000 | 0.2215 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0554 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2843 | 0.0001 | 0.0000 | 0.8634 | 0.8903 | 0.2486 | 0.2215 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1539 |
| schreiber2017/ismir2017 | 0.0298 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0183 | 0.0216 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.0400 | 0.0447 | 0.0010 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0127 | 0.0058 | 0.1281 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0183 | 0.0127 | 1.0000 | 0.9033 | 0.0004 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0216 | 0.0058 | 0.9033 | 1.0000 | 0.0002 | 0.0000 |
| sun2021/default | 0.0000 | 0.5149 | 0.5867 | 0.0240 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1281 | 0.0004 | 0.0002 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7189 | 0.0000 | 0.0000 | 0.1053 | 0.0912 | 0.0079 | 0.0554 | 0.1539 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 48: Paired t-test p-values, using reference annotations 2.0 as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0294 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.7882 | 0.1684 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0400 | 0.0000 | 0.0000 | 0.5149 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.7882 | 1.0000 | 0.1007 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0447 | 0.0000 | 0.0000 | 0.5867 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1684 | 0.1007 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 | 0.0000 | 0.0000 | 0.0240 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.2253 | 0.2347 | 0.0922 | 0.0519 | 0.2378 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.6802 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.1974 | 0.0001 | 0.0000 | 0.0008 | 0.0061 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1974 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0008 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2253 | 0.0001 | 0.0000 | 1.0000 | 0.9822 | 0.1652 | 0.2534 | 0.7636 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1098 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2347 | 0.0000 | 0.0000 | 0.9822 | 1.0000 | 0.1707 | 0.2489 | 0.7645 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1061 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0922 | 0.0008 | 0.0000 | 0.1652 | 0.1707 | 1.0000 | 0.6441 | 0.3125 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0081 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0519 | 0.0061 | 0.0008 | 0.2534 | 0.2489 | 0.6441 | 1.0000 | 0.3337 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0875 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2378 | 0.0001 | 0.0000 | 0.7636 | 0.7645 | 0.3125 | 0.3337 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1221 |
| schreiber2017/ismir2017 | 0.0294 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0181 | 0.0213 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.0400 | 0.0447 | 0.0010 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0127 | 0.0058 | 0.1219 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0181 | 0.0127 | 1.0000 | 0.9033 | 0.0004 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0213 | 0.0058 | 0.9033 | 1.0000 | 0.0002 | 0.0000 |
| sun2021/default | 0.0000 | 0.5149 | 0.5867 | 0.0240 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1219 | 0.0004 | 0.0002 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.6802 | 0.0000 | 0.0000 | 0.1098 | 0.1061 | 0.0081 | 0.0875 | 0.1221 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 49: Paired t-test p-values, using reference annotations 2.0-no-dupes as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0613 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.2034 | 0.5058 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0739 | 0.0000 | 0.0001 | 0.5524 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.2034 | 1.0000 | 0.1311 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2013 | 0.0001 | 0.0004 | 0.9809 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.5058 | 0.1311 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0154 | 0.0000 | 0.0000 | 0.1502 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.1312 | 0.1787 | 0.0508 | 0.0180 | 0.1469 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4625 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.2314 | 0.0001 | 0.0000 | 0.0010 | 0.0103 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2314 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0019 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1312 | 0.0001 | 0.0000 | 1.0000 | 0.7431 | 0.1534 | 0.1930 | 0.7383 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1351 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1787 | 0.0000 | 0.0000 | 0.7431 | 1.0000 | 0.1047 | 0.1561 | 0.5892 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2235 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0508 | 0.0010 | 0.0000 | 0.1534 | 0.1047 | 1.0000 | 0.5673 | 0.2988 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0099 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0180 | 0.0103 | 0.0019 | 0.1930 | 0.1561 | 0.5673 | 1.0000 | 0.2739 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0699 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1469 | 0.0001 | 0.0000 | 0.7383 | 0.5892 | 0.2988 | 0.2739 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1354 |
| schreiber2017/ismir2017 | 0.0613 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2017/mirex2017 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0189 | 0.0127 | 0.0000 | 0.0000 |
| schreiber2018/cnn | 0.0000 | 0.0739 | 0.2013 | 0.0154 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0064 | 0.0067 | 0.2018 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0189 | 0.0064 | 1.0000 | 0.8797 | 0.0004 | 0.0000 |
| schreiber2018/ismir2018 | 0.0000 | 0.0001 | 0.0004 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0127 | 0.0067 | 0.8797 | 1.0000 | 0.0005 | 0.0000 |
| sun2021/default | 0.0000 | 0.5524 | 0.9809 | 0.1502 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2018 | 0.0004 | 0.0005 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4625 | 0.0000 | 0.0000 | 0.1351 | 0.2235 | 0.0099 | 0.0699 | 0.1354 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 50: Paired t-test p-values, using reference annotations 4.0 as groundtruth with AOE1. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.3037 | 0.0000 | 0.0000 | 0.0006 | 0.0000 | 0.0022 | 0.1130 | 0.4266 | 0.0619 | 0.0051 | 0.0142 | 0.0002 | 0.0001 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.1827 | 0.0001 | 0.0000 | 0.0000 | 0.0006 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0005 | 0.0026 | 0.1574 | 0.3086 | 0.4115 | 0.4810 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.1827 | 1.0000 | 0.3333 | 0.0000 | 0.0000 | 0.0018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0012 | 0.0066 | 0.3172 | 0.5733 | 0.6518 | 0.8816 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.0001 | 0.3333 | 1.0000 | 0.0000 | 0.0000 | 0.0032 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0020 | 0.0114 | 0.4630 | 0.8165 | 0.8728 | 0.7501 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0081 | 0.0000 | 0.0900 | 0.5027 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0081 | 1.0000 | 0.0000 | 0.0001 | 0.0010 | 0.0000 | 0.0020 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.3037 | 0.0006 | 0.0018 | 0.0032 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0183 | 0.0000 | 0.0403 | 0.5495 | 0.9031 | 0.0505 | 0.0149 | 0.0168 | 0.0059 | 0.0020 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0900 | 0.0001 | 0.0000 | 1.0000 | 0.2410 | 0.0004 | 0.0000 | 0.0006 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0014 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.5027 | 0.0010 | 0.0000 | 0.2410 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| percival2014/stem | 0.0006 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0183 | 0.0004 | 0.0000 | 1.0000 | 0.0000 | 0.9726 | 0.0680 | 0.0120 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.3900 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0020 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0022 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0403 | 0.0006 | 0.0000 | 0.9726 | 0.0000 | 1.0000 | 0.0125 | 0.0052 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4085 |
| schreiber2017/ismir2017 | 0.1130 | 0.0005 | 0.0012 | 0.0020 | 0.0000 | 0.0000 | 0.5495 | 0.0000 | 0.0000 | 0.0680 | 0.0000 | 0.0125 | 1.0000 | 0.2658 | 0.0065 | 0.0014 | 0.0026 | 0.0014 | 0.0164 |
| schreiber2017/mirex2017 | 0.4266 | 0.0026 | 0.0066 | 0.0114 | 0.0000 | 0.0000 | 0.9031 | 0.0000 | 0.0000 | 0.0120 | 0.0000 | 0.0052 | 0.2658 | 1.0000 | 0.0112 | 0.0076 | 0.0138 | 0.0063 | 0.0025 |
| schreiber2018/cnn | 0.0619 | 0.1574 | 0.3172 | 0.4630 | 0.0000 | 0.0000 | 0.0505 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0065 | 0.0112 | 1.0000 | 0.5978 | 0.5061 | 0.3779 | 0.0000 |
| schreiber2018/fcn | 0.0051 | 0.3086 | 0.5733 | 0.8165 | 0.0000 | 0.0000 | 0.0149 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0014 | 0.0076 | 0.5978 | 1.0000 | 0.9673 | 0.7962 | 0.0000 |
| schreiber2018/ismir2018 | 0.0142 | 0.4115 | 0.6518 | 0.8728 | 0.0000 | 0.0000 | 0.0168 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0026 | 0.0138 | 0.5061 | 0.9673 | 1.0000 | 0.7980 | 0.0000 |
| sun2021/default | 0.0002 | 0.4810 | 0.8816 | 0.7501 | 0.0000 | 0.0000 | 0.0059 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0014 | 0.0063 | 0.3779 | 0.7962 | 0.7980 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0020 | 0.0014 | 0.0000 | 0.3900 | 0.0000 | 0.4085 | 0.0164 | 0.0025 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 51: Paired t-test p-values, using reference annotations 1.0 as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0380 | 0.0000 | 0.0000 | 0.0004 | 0.0000 | 0.0059 | 0.2236 | 0.7170 | 0.7549 | 0.5960 | 0.8012 | 0.0546 | 0.0000 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.3895 | 0.1018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0015 | 0.0074 | 0.0002 | 0.0000 | 0.0004 | 0.0000 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.3895 | 1.0000 | 0.8364 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0035 | 0.0177 | 0.0013 | 0.0001 | 0.0027 | 0.0000 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1018 | 0.8364 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0022 | 0.0121 | 0.0004 | 0.0000 | 0.0007 | 0.0000 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.1361 | 0.0000 | 0.0038 | 0.1321 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1361 | 1.0000 | 0.0000 | 0.0002 | 0.0095 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0380 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0775 | 0.0000 | 0.2139 | 0.7198 | 0.2403 | 0.1220 | 0.1171 | 0.1023 | 0.2554 | 0.0072 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0038 | 0.0002 | 0.0000 | 1.0000 | 0.0979 | 0.0001 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0008 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1321 | 0.0095 | 0.0000 | 0.0979 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| percival2014/stem | 0.0004 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0775 | 0.0001 | 0.0000 | 1.0000 | 0.0000 | 0.7791 | 0.0345 | 0.0042 | 0.0003 | 0.0009 | 0.0005 | 0.0026 | 0.3143 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0059 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.2139 | 0.0001 | 0.0000 | 0.7791 | 0.0000 | 1.0000 | 0.0116 | 0.0043 | 0.0046 | 0.0069 | 0.0077 | 0.0248 | 0.2387 |
| schreiber2017/ismir2017 | 0.2236 | 0.0015 | 0.0035 | 0.0022 | 0.0000 | 0.0000 | 0.7198 | 0.0000 | 0.0000 | 0.0345 | 0.0000 | 0.0116 | 1.0000 | 0.2470 | 0.2829 | 0.2963 | 0.2759 | 0.6072 | 0.0061 |
| schreiber2017/mirex2017 | 0.7170 | 0.0074 | 0.0177 | 0.0121 | 0.0000 | 0.0000 | 0.2403 | 0.0000 | 0.0000 | 0.0042 | 0.0000 | 0.0043 | 0.2470 | 1.0000 | 0.8477 | 0.9407 | 0.8411 | 0.6148 | 0.0007 |
| schreiber2018/cnn | 0.7549 | 0.0002 | 0.0013 | 0.0004 | 0.0000 | 0.0000 | 0.1220 | 0.0000 | 0.0000 | 0.0003 | 0.0000 | 0.0046 | 0.2829 | 0.8477 | 1.0000 | 0.8901 | 0.9487 | 0.2666 | 0.0001 |
| schreiber2018/fcn | 0.5960 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.1171 | 0.0000 | 0.0000 | 0.0009 | 0.0000 | 0.0069 | 0.2963 | 0.9407 | 0.8901 | 1.0000 | 0.8330 | 0.4119 | 0.0001 |
| schreiber2018/ismir2018 | 0.8012 | 0.0004 | 0.0027 | 0.0007 | 0.0000 | 0.0000 | 0.1023 | 0.0000 | 0.0000 | 0.0005 | 0.0000 | 0.0077 | 0.2759 | 0.8411 | 0.9487 | 0.8330 | 1.0000 | 0.2429 | 0.0001 |
| sun2021/default | 0.0546 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2554 | 0.0000 | 0.0000 | 0.0026 | 0.0000 | 0.0248 | 0.6072 | 0.6148 | 0.2666 | 0.4119 | 0.2429 | 1.0000 | 0.0004 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0072 | 0.0008 | 0.0000 | 0.3143 | 0.0000 | 0.2387 | 0.0061 | 0.0007 | 0.0001 | 0.0001 | 0.0001 | 0.0004 | 1.0000 |
Table 52: Paired t-test p-values, using reference annotations 3.0 as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0360 | 0.0000 | 0.0000 | 0.0004 | 0.0000 | 0.0053 | 0.2073 | 0.6791 | 0.7265 | 0.5655 | 0.7729 | 0.0766 | 0.0001 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.3895 | 0.1018 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0015 | 0.0074 | 0.0002 | 0.0000 | 0.0004 | 0.0000 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.3895 | 1.0000 | 0.8364 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0035 | 0.0177 | 0.0013 | 0.0001 | 0.0027 | 0.0000 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.1018 | 0.8364 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0022 | 0.0121 | 0.0004 | 0.0000 | 0.0007 | 0.0000 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.1514 | 0.0000 | 0.0036 | 0.1472 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1514 | 1.0000 | 0.0000 | 0.0003 | 0.0121 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0360 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0748 | 0.0000 | 0.2077 | 0.7352 | 0.2495 | 0.1228 | 0.1180 | 0.1031 | 0.2137 | 0.0103 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0036 | 0.0003 | 0.0000 | 1.0000 | 0.0857 | 0.0001 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0007 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1472 | 0.0121 | 0.0000 | 0.0857 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| percival2014/stem | 0.0004 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0748 | 0.0001 | 0.0000 | 1.0000 | 0.0000 | 0.7814 | 0.0348 | 0.0043 | 0.0003 | 0.0008 | 0.0005 | 0.0018 | 0.3939 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0053 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.2077 | 0.0001 | 0.0000 | 0.7814 | 0.0000 | 1.0000 | 0.0116 | 0.0043 | 0.0044 | 0.0065 | 0.0073 | 0.0195 | 0.2996 |
| schreiber2017/ismir2017 | 0.2073 | 0.0015 | 0.0035 | 0.0022 | 0.0000 | 0.0000 | 0.7352 | 0.0000 | 0.0000 | 0.0348 | 0.0000 | 0.0116 | 1.0000 | 0.2470 | 0.2725 | 0.2851 | 0.2659 | 0.5275 | 0.0089 |
| schreiber2017/mirex2017 | 0.6791 | 0.0074 | 0.0177 | 0.0121 | 0.0000 | 0.0000 | 0.2495 | 0.0000 | 0.0000 | 0.0043 | 0.0000 | 0.0043 | 0.2470 | 1.0000 | 0.8223 | 0.9167 | 0.8196 | 0.7144 | 0.0011 |
| schreiber2018/cnn | 0.7265 | 0.0002 | 0.0013 | 0.0004 | 0.0000 | 0.0000 | 0.1228 | 0.0000 | 0.0000 | 0.0003 | 0.0000 | 0.0044 | 0.2725 | 0.8223 | 1.0000 | 0.8901 | 0.9487 | 0.3508 | 0.0002 |
| schreiber2018/fcn | 0.5655 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.1180 | 0.0000 | 0.0000 | 0.0008 | 0.0000 | 0.0065 | 0.2851 | 0.9167 | 0.8901 | 1.0000 | 0.8331 | 0.5060 | 0.0001 |
| schreiber2018/ismir2018 | 0.7729 | 0.0004 | 0.0027 | 0.0007 | 0.0000 | 0.0000 | 0.1031 | 0.0000 | 0.0000 | 0.0005 | 0.0000 | 0.0073 | 0.2659 | 0.8196 | 0.9487 | 0.8331 | 1.0000 | 0.3216 | 0.0001 |
| sun2021/default | 0.0766 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2137 | 0.0000 | 0.0000 | 0.0018 | 0.0000 | 0.0195 | 0.5275 | 0.7144 | 0.3508 | 0.5060 | 0.3216 | 1.0000 | 0.0006 |
| zplane/auftakt_v3 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0103 | 0.0007 | 0.0000 | 0.3939 | 0.0000 | 0.2996 | 0.0089 | 0.0011 | 0.0002 | 0.0001 | 0.0001 | 0.0006 | 1.0000 |
Table 53: Paired t-test p-values, using reference annotations 3.0-no-dupes as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.3443 | 0.1935 | 0.4237 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0004 | 0.0021 | 0.0003 | 0.0000 | 0.0006 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.3443 | 1.0000 | 0.4278 | 0.7818 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0008 | 0.0040 | 0.0011 | 0.0000 | 0.0016 | 0.0000 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.1935 | 0.4278 | 1.0000 | 0.3703 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0021 | 0.0107 | 0.0056 | 0.0006 | 0.0094 | 0.0000 | 0.0000 |
| boeck2020/dar | 0.4237 | 0.7818 | 0.3703 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0007 | 0.0036 | 0.0009 | 0.0000 | 0.0018 | 0.0000 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0413 | 0.0000 | 0.0184 | 0.4352 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0413 | 1.0000 | 0.0000 | 0.0001 | 0.0076 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0549 | 0.0000 | 0.1314 | 0.9387 | 0.3899 | 0.1146 | 0.1125 | 0.0866 | 0.1822 | 0.0035 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0184 | 0.0001 | 0.0000 | 1.0000 | 0.0779 | 0.0003 | 0.0000 | 0.0004 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0033 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4352 | 0.0076 | 0.0000 | 0.0779 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0549 | 0.0003 | 0.0000 | 1.0000 | 0.0000 | 0.8774 | 0.0450 | 0.0060 | 0.0001 | 0.0004 | 0.0002 | 0.0008 | 0.2739 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1314 | 0.0004 | 0.0000 | 0.8774 | 0.0000 | 1.0000 | 0.0101 | 0.0038 | 0.0013 | 0.0021 | 0.0019 | 0.0068 | 0.2521 |
| schreiber2017/ismir2017 | 0.0004 | 0.0008 | 0.0021 | 0.0007 | 0.0000 | 0.0000 | 0.9387 | 0.0000 | 0.0000 | 0.0450 | 0.0000 | 0.0101 | 1.0000 | 0.2481 | 0.1332 | 0.1380 | 0.1119 | 0.2878 | 0.0062 |
| schreiber2017/mirex2017 | 0.0021 | 0.0040 | 0.0107 | 0.0036 | 0.0000 | 0.0000 | 0.3899 | 0.0000 | 0.0000 | 0.0060 | 0.0000 | 0.0038 | 0.2481 | 1.0000 | 0.4263 | 0.5389 | 0.4298 | 0.8781 | 0.0008 |
| schreiber2018/cnn | 0.0003 | 0.0011 | 0.0056 | 0.0009 | 0.0000 | 0.0000 | 0.1146 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0013 | 0.1332 | 0.4263 | 1.0000 | 0.8643 | 0.8745 | 0.3937 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0000 | 0.0006 | 0.0000 | 0.0000 | 0.0000 | 0.1125 | 0.0000 | 0.0000 | 0.0004 | 0.0000 | 0.0021 | 0.1380 | 0.5389 | 0.8643 | 1.0000 | 0.7354 | 0.5595 | 0.0000 |
| schreiber2018/ismir2018 | 0.0006 | 0.0016 | 0.0094 | 0.0018 | 0.0000 | 0.0000 | 0.0866 | 0.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0019 | 0.1119 | 0.4298 | 0.8745 | 0.7354 | 1.0000 | 0.2817 | 0.0000 |
| sun2021/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1822 | 0.0000 | 0.0000 | 0.0008 | 0.0000 | 0.0068 | 0.2878 | 0.8781 | 0.3937 | 0.5595 | 0.2817 | 1.0000 | 0.0001 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0035 | 0.0033 | 0.0000 | 0.2739 | 0.0000 | 0.2521 | 0.0062 | 0.0008 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 |
Table 54: Paired t-test p-values, using reference annotations 2.0 as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.3443 | 0.1935 | 0.4237 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0005 | 0.0024 | 0.0004 | 0.0000 | 0.0009 | 0.0000 | 0.0000 |
| boeck2019/multi_task | 0.3443 | 1.0000 | 0.4278 | 0.7818 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0008 | 0.0040 | 0.0011 | 0.0000 | 0.0016 | 0.0000 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.1935 | 0.4278 | 1.0000 | 0.3703 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0021 | 0.0107 | 0.0056 | 0.0006 | 0.0094 | 0.0000 | 0.0000 |
| boeck2020/dar | 0.4237 | 0.7818 | 0.3703 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0007 | 0.0036 | 0.0009 | 0.0000 | 0.0018 | 0.0000 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0481 | 0.0000 | 0.0175 | 0.4601 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0481 | 1.0000 | 0.0000 | 0.0001 | 0.0099 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0529 | 0.0000 | 0.1260 | 0.9594 | 0.4058 | 0.1166 | 0.1147 | 0.0883 | 0.1536 | 0.0050 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0175 | 0.0001 | 0.0000 | 1.0000 | 0.0692 | 0.0004 | 0.0000 | 0.0005 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0028 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4601 | 0.0099 | 0.0000 | 0.0692 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| percival2014/stem | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0529 | 0.0004 | 0.0000 | 1.0000 | 0.0000 | 0.8835 | 0.0460 | 0.0061 | 0.0001 | 0.0004 | 0.0002 | 0.0005 | 0.3430 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1260 | 0.0005 | 0.0000 | 0.8835 | 0.0000 | 1.0000 | 0.0101 | 0.0038 | 0.0012 | 0.0020 | 0.0018 | 0.0052 | 0.3149 |
| schreiber2017/ismir2017 | 0.0005 | 0.0008 | 0.0021 | 0.0007 | 0.0000 | 0.0000 | 0.9594 | 0.0000 | 0.0000 | 0.0460 | 0.0000 | 0.0101 | 1.0000 | 0.2481 | 0.1268 | 0.1310 | 0.1062 | 0.2386 | 0.0090 |
| schreiber2017/mirex2017 | 0.0024 | 0.0040 | 0.0107 | 0.0036 | 0.0000 | 0.0000 | 0.4058 | 0.0000 | 0.0000 | 0.0061 | 0.0000 | 0.0038 | 0.2481 | 1.0000 | 0.4059 | 0.5172 | 0.4120 | 0.7731 | 0.0012 |
| schreiber2018/cnn | 0.0004 | 0.0011 | 0.0056 | 0.0009 | 0.0000 | 0.0000 | 0.1166 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0012 | 0.1268 | 0.4059 | 1.0000 | 0.8643 | 0.8745 | 0.4912 | 0.0000 |
| schreiber2018/fcn | 0.0000 | 0.0000 | 0.0006 | 0.0000 | 0.0000 | 0.0000 | 0.1147 | 0.0000 | 0.0000 | 0.0004 | 0.0000 | 0.0020 | 0.1310 | 0.5172 | 0.8643 | 1.0000 | 0.7354 | 0.6627 | 0.0000 |
| schreiber2018/ismir2018 | 0.0009 | 0.0016 | 0.0094 | 0.0018 | 0.0000 | 0.0000 | 0.0883 | 0.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0018 | 0.1062 | 0.4120 | 0.8745 | 0.7354 | 1.0000 | 0.3666 | 0.0000 |
| sun2021/default | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1536 | 0.0000 | 0.0000 | 0.0005 | 0.0000 | 0.0052 | 0.2386 | 0.7731 | 0.4912 | 0.6627 | 0.3666 | 1.0000 | 0.0001 |
| zplane/auftakt_v3 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0050 | 0.0028 | 0.0000 | 0.3430 | 0.0000 | 0.3149 | 0.0090 | 0.0012 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 1.0000 |
Table 55: Paired t-test p-values, using reference annotations 2.0-no-dupes as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
| Estimator | boeck2015/tempodetector2016_default | boeck2019/multi_task | boeck2019/multi_task_hjdb | boeck2020/dar | davies2009/mirex_qm_tempotracker | echonest/version_3_2_1 | gkiokas2012/default | klapuri2006/percival2014 | oliveira2010/ibt | percival2014/stem | scheirer1998/percival2014 | schreiber2014/default | schreiber2017/ismir2017 | schreiber2017/mirex2017 | schreiber2018/cnn | schreiber2018/fcn | schreiber2018/ismir2018 | sun2021/default | zplane/auftakt_v3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boeck2015/tempodetector2016_default | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.7998 | 0.0000 | 0.0000 | 0.0002 | 0.0000 | 0.0021 | 0.1103 | 0.4185 | 0.0838 | 0.0056 | 0.0191 | 0.0028 | 0.0003 |
| boeck2019/multi_task | 0.0000 | 1.0000 | 0.2536 | 0.0000 | 0.0000 | 0.0000 | 0.0073 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0005 | 0.0027 | 0.1257 | 0.3043 | 0.3633 | 0.1251 | 0.0000 |
| boeck2019/multi_task_hjdb | 0.0000 | 0.2536 | 1.0000 | 0.2256 | 0.0000 | 0.0000 | 0.0158 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0011 | 0.0061 | 0.2450 | 0.5316 | 0.5589 | 0.3364 | 0.0000 |
| boeck2020/dar | 0.0000 | 0.0000 | 0.2256 | 1.0000 | 0.0000 | 0.0000 | 0.0297 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0022 | 0.0123 | 0.4102 | 0.8386 | 0.8278 | 0.6378 | 0.0000 |
| davies2009/mirex_qm_tempotracker | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0106 | 0.0000 | 0.0324 | 0.4180 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| echonest/version_3_2_1 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0106 | 1.0000 | 0.0000 | 0.0000 | 0.0013 | 0.0000 | 0.0006 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| gkiokas2012/default | 0.7998 | 0.0073 | 0.0158 | 0.0297 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0023 | 0.0000 | 0.0126 | 0.2656 | 0.6635 | 0.2085 | 0.0757 | 0.0906 | 0.0917 | 0.0010 |
| klapuri2006/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0324 | 0.0000 | 0.0000 | 1.0000 | 0.1346 | 0.0015 | 0.0000 | 0.0016 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0016 |
| oliveira2010/ibt | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.4180 | 0.0013 | 0.0000 | 0.1346 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| percival2014/stem | 0.0002 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0023 | 0.0015 | 0.0000 | 1.0000 | 0.0000 | 0.8555 | 0.0442 | 0.0063 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.6790 |
| scheirer1998/percival2014 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0006 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| schreiber2014/default | 0.0021 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0126 | 0.0016 | 0.0000 | 0.8555 | 0.0000 | 1.0000 | 0.0111 | 0.0047 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.5759 |
| schreiber2017/ismir2017 | 0.1103 | 0.0005 | 0.0011 | 0.0022 | 0.0000 | 0.0000 | 0.2656 | 0.0000 | 0.0000 | 0.0442 | 0.0000 | 0.0111 | 1.0000 | 0.2658 | 0.0081 | 0.0014 | 0.0031 | 0.0041 | 0.0345 |
| schreiber2017/mirex2017 | 0.4185 | 0.0027 | 0.0061 | 0.0123 | 0.0000 | 0.0000 | 0.6635 | 0.0000 | 0.0000 | 0.0063 | 0.0000 | 0.0047 | 0.2658 | 1.0000 | 0.0149 | 0.0063 | 0.0141 | 0.0194 | 0.0064 |
| schreiber2018/cnn | 0.0838 | 0.1257 | 0.2450 | 0.4102 | 0.0000 | 0.0000 | 0.2085 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0081 | 0.0149 | 1.0000 | 0.5200 | 0.4809 | 0.6878 | 0.0000 |
| schreiber2018/fcn | 0.0056 | 0.3043 | 0.5316 | 0.8386 | 0.0000 | 0.0000 | 0.0757 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0014 | 0.0063 | 0.5200 | 1.0000 | 0.9652 | 0.7889 | 0.0000 |
| schreiber2018/ismir2018 | 0.0191 | 0.3633 | 0.5589 | 0.8278 | 0.0000 | 0.0000 | 0.0906 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0031 | 0.0141 | 0.4809 | 0.9652 | 1.0000 | 0.7920 | 0.0000 |
| sun2021/default | 0.0028 | 0.1251 | 0.3364 | 0.6378 | 0.0000 | 0.0000 | 0.0917 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0001 | 0.0041 | 0.0194 | 0.6878 | 0.7889 | 0.7920 | 1.0000 | 0.0000 |
| zplane/auftakt_v3 | 0.0003 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0010 | 0.0016 | 0.0000 | 0.6790 | 0.0000 | 0.5759 | 0.0345 | 0.0064 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 |
Table 56: Paired t-test p-values, using reference annotations 4.0 as groundtruth with AOE2. H0: the true mean difference between paired samples is zero. If p<=ɑ, reject H0, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.
AOE1 on cvar-Subsets
How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?
AOE1 on cvar-Subsets for 1.0 based on cvar-Values from 1.0
Figure 137: Mean AOE1 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 on cvar-Subsets for 2.0 based on cvar-Values from 1.0
Figure 138: Mean AOE1 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0
Figure 139: Mean AOE1 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 on cvar-Subsets for 3.0 based on cvar-Values from 1.0
Figure 140: Mean AOE1 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0
Figure 141: Mean AOE1 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 on cvar-Subsets for 4.0 based on cvar-Values from 1.0
Figure 142: Mean AOE1 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on cvar-Subsets
How well does an estimator perform, when only taking tracks into account that have a cvar-value of less than τ, i.e., have a more or less stable beat?
AOE2 on cvar-Subsets for 1.0 based on cvar-Values from 1.0
Figure 143: Mean AOE2 compared to version 1.0 for tracks with cvar < τ based on beat annotations from 1.0.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on cvar-Subsets for 2.0 based on cvar-Values from 1.0
Figure 144: Mean AOE2 compared to version 2.0 for tracks with cvar < τ based on beat annotations from 2.0.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on cvar-Subsets for 2.0-no-dupes based on cvar-Values from 1.0
Figure 145: Mean AOE2 compared to version 2.0-no-dupes for tracks with cvar < τ based on beat annotations from 2.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on cvar-Subsets for 3.0 based on cvar-Values from 1.0
Figure 146: Mean AOE2 compared to version 3.0 for tracks with cvar < τ based on beat annotations from 3.0.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on cvar-Subsets for 3.0-no-dupes based on cvar-Values from 1.0
Figure 147: Mean AOE2 compared to version 3.0-no-dupes for tracks with cvar < τ based on beat annotations from 3.0-no-dupes.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on cvar-Subsets for 4.0 based on cvar-Values from 1.0
Figure 148: Mean AOE2 compared to version 4.0 for tracks with cvar < τ based on beat annotations from 4.0.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 on Tempo-Subsets
How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean AOE1 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.
AOE1 on Tempo-Subsets for 1.0
Figure 149: Mean AOE1 for estimates compared to version 1.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 on Tempo-Subsets for 2.0
Figure 150: Mean AOE1 for estimates compared to version 2.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 on Tempo-Subsets for 2.0-no-dupes
Figure 151: Mean AOE1 for estimates compared to version 2.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 on Tempo-Subsets for 3.0
Figure 152: Mean AOE1 for estimates compared to version 3.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 on Tempo-Subsets for 3.0-no-dupes
Figure 153: Mean AOE1 for estimates compared to version 3.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 on Tempo-Subsets for 4.0
Figure 154: Mean AOE1 for estimates compared to version 4.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on Tempo-Subsets
How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean AOE2 for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.
AOE2 on Tempo-Subsets for 1.0
Figure 155: Mean AOE2 for estimates compared to version 1.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on Tempo-Subsets for 2.0
Figure 156: Mean AOE2 for estimates compared to version 2.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on Tempo-Subsets for 2.0-no-dupes
Figure 157: Mean AOE2 for estimates compared to version 2.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on Tempo-Subsets for 3.0
Figure 158: Mean AOE2 for estimates compared to version 3.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on Tempo-Subsets for 3.0-no-dupes
Figure 159: Mean AOE2 for estimates compared to version 3.0-no-dupes for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE2 on Tempo-Subsets for 4.0
Figure 160: Mean AOE2 for estimates compared to version 4.0 for tempo intervals around T.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE1 for Tempo
When fitting a generalized additive model (GAM) to AOE1-values and a ground truth, what AOE1 can we expect with confidence?
Estimated AOE1 for Tempo for 1.0
Predictions of GAMs trained on AOE1 for estimates for reference 1.0.
Figure 161: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE1 for Tempo for 2.0
Predictions of GAMs trained on AOE1 for estimates for reference 2.0.
Figure 162: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE1 for Tempo for 2.0-no-dupes
Predictions of GAMs trained on AOE1 for estimates for reference 2.0-no-dupes.
Figure 163: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE1 for Tempo for 3.0
Predictions of GAMs trained on AOE1 for estimates for reference 3.0.
Figure 164: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE1 for Tempo for 3.0-no-dupes
Predictions of GAMs trained on AOE1 for estimates for reference 3.0-no-dupes.
Figure 165: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE1 for Tempo for 4.0
Predictions of GAMs trained on AOE1 for estimates for reference 4.0.
Figure 166: AOE1 predictions of a generalized additive model (GAM) fit to AOE1 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE2 for Tempo
When fitting a generalized additive model (GAM) to AOE2-values and a ground truth, what AOE2 can we expect with confidence?
Estimated AOE2 for Tempo for 1.0
Predictions of GAMs trained on AOE2 for estimates for reference 1.0.
Figure 167: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 1.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE2 for Tempo for 2.0
Predictions of GAMs trained on AOE2 for estimates for reference 2.0.
Figure 168: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 2.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE2 for Tempo for 2.0-no-dupes
Predictions of GAMs trained on AOE2 for estimates for reference 2.0-no-dupes.
Figure 169: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 2.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE2 for Tempo for 3.0
Predictions of GAMs trained on AOE2 for estimates for reference 3.0.
Figure 170: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 3.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE2 for Tempo for 3.0-no-dupes
Predictions of GAMs trained on AOE2 for estimates for reference 3.0-no-dupes.
Figure 171: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 3.0-no-dupes. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
Estimated AOE2 for Tempo for 4.0
Predictions of GAMs trained on AOE2 for estimates for reference 4.0.
Figure 172: AOE2 predictions of a generalized additive model (GAM) fit to AOE2 results for 4.0. The 95% confidence interval around the prediction is shaded in gray.
CSV JSON LATEX PICKLE SVG PDF PNG
AOE1 for ‘tag_open’ Tags
How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.
AOE1 for ‘tag_open’ Tags for 1.0
Figure 173: AOE1 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.
AOE1 for ‘tag_open’ Tags for 2.0
Figure 174: AOE1 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.
AOE1 for ‘tag_open’ Tags for 2.0-no-dupes
Figure 175: AOE1 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.
AOE1 for ‘tag_open’ Tags for 3.0
Figure 176: AOE1 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.
AOE1 for ‘tag_open’ Tags for 3.0-no-dupes
Figure 177: AOE1 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.
AOE1 for ‘tag_open’ Tags for 4.0
Figure 178: AOE1 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.
AOE2 for ‘tag_open’ Tags
How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.
AOE2 for ‘tag_open’ Tags for 1.0
Figure 179: AOE2 of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.
AOE2 for ‘tag_open’ Tags for 2.0
Figure 180: AOE2 of estimates compared to version 2.0 depending on tag from namespace ‘tag_open’.
AOE2 for ‘tag_open’ Tags for 2.0-no-dupes
Figure 181: AOE2 of estimates compared to version 2.0-no-dupes depending on tag from namespace ‘tag_open’.
AOE2 for ‘tag_open’ Tags for 3.0
Figure 182: AOE2 of estimates compared to version 3.0 depending on tag from namespace ‘tag_open’.
AOE2 for ‘tag_open’ Tags for 3.0-no-dupes
Figure 183: AOE2 of estimates compared to version 3.0-no-dupes depending on tag from namespace ‘tag_open’.
AOE2 for ‘tag_open’ Tags for 4.0
Figure 184: AOE2 of estimates compared to version 4.0 depending on tag from namespace ‘tag_open’.
Generated by tempo_eval 0.1.1 on 2022-06-29 18:16. Size L.