rwc_mdb_g

This is the tempo_eval report for the ‘rwc_mdb_g’ corpus.

Reports for other corpora may be found here.

Table of Contents

• References for ‘rwc_mdb_g’
  • References
    • 1.0
  • Basic Statistics
  • Smoothed Tempo Distribution
  • Tag Distribution for ‘tag_open’
  • Beat-Based Tempo Variation
• Estimates for ‘rwc_mdb_g’
  • Estimators
    • boeck2015/tempodetector2016_default
    • davies2009/mirex_qm_tempotracker
    • percival2014/stem
    • schreiber2014/default
    • schreiber2017/ismir2017
    • schreiber2017/mirex2017
    • schreiber2018/cnn
    • schreiber2018/fcn
    • schreiber2018/ismir2018
  • Basic Statistics
  • Smoothed Tempo Distribution
  • Accuracy
    • Accuracy Results for 1.0
    • Accuracy₁ for 1.0
    • Accuracy₂ for 1.0
    • Differing Items
    • Significance of Differences
    • Accuracy₁ on c_var-Subsets
    • Accuracy₂ on c_var-Subsets
    • Accuracy₁ on Tempo-Subsets
    • Accuracy₂ on Tempo-Subsets
    • Estimated Accuracy₁ for Tempo
    • Estimated Accuracy₂ for Tempo
    • Accuracy₁ for ‘tag_open’ Tags
    • Accuracy₂ for ‘tag_open’ Tags
  • OE₁ and OE₂
    • Mean OE₁/OE₂ Results for 1.0
    • OE₁ distribution for 1.0
    • OE₂ distribution for 1.0
    • Significance of Differences
    • OE₁ on c_var-Subsets
    • OE₂ on c_var-Subsets
    • OE₁ on Tempo-Subsets
    • OE₂ on Tempo-Subsets
    • Estimated OE₁ for Tempo
    • Estimated OE₂ for Tempo
    • OE₁ for ‘tag_open’ Tags
    • OE₂ for ‘tag_open’ Tags
  • AOE₁ and AOE₂
    • Mean AOE₁/AOE₂ Results for 1.0
    • AOE₁ distribution for 1.0
    • AOE₂ distribution for 1.0
    • Significance of Differences
    • AOE₁ on c_var-Subsets
    • AOE₂ on c_var-Subsets
    • AOE₁ on Tempo-Subsets
    • AOE₂ on Tempo-Subsets
    • Estimated AOE₁ for Tempo
    • Estimated AOE₂ for Tempo
    • AOE₁ for ‘tag_open’ Tags
    • AOE₂ for ‘tag_open’ Tags

References for ‘rwc_mdb_g’

References

1.0

Attribute	Value
Corpus	rwc_mdb_g
Version	1.0
Curator	Masataka Goto
Data Source	manual annotation
Annotation Tools	derived from beat annotations
Annotation Rules	median of corresponding inter beat intervals
Annotator, name	Masataka Goto
Annotator, bibtex	Goto2006
Annotator, ref_url	https://staff.aist.go.jp/m.goto/RWC-MDB/AIST-Annotation/

Basic Statistics

Reference	Size	Min	Max	Avg	Stdev	Sweet Oct. Start	Sweet Oct. Coverage
1.0	102	30.15	235.81	103.57	34.17	73.00	0.75

Table 1: Basic statistics.

CSV JSON LATEX PICKLE

Smoothed Tempo Distribution

Figure 1: Percentage of values in tempo interval.

CSV JSON LATEX PICKLE SVG PDF PNG

Tag Distribution for ‘tag_open’

Figure 2: Percentage of tracks tagged with tags from namespace ‘tag_open’. Annotations are from reference 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Beat-Based Tempo Variation

Figure 3: Fraction of the dataset with beat-annotated tracks with c_var < τ.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimates for ‘rwc_mdb_g’

Estimators

boeck2015/tempodetector2016_default

Attribute	Value
Corpus	rwc_mdb_g
Version	0.17.dev0
Annotation Tools	TempoDetector.2016, madmom, https://github.com/CPJKU/madmom
Annotator, bibtex	Boeck2015

davies2009/mirex_qm_tempotracker

Attribute	Value
Corpus	rwc_mdb_g
Version	1.0
Annotation Tools	QM Tempotracker, Sonic Annotator plugin. https://code.soundsoftware.ac.uk/projects/mirex2013/repository/show/audio_tempo_estimation/qm-tempotracker Note that the current macOS build of ‘qm-vamp-plugins’ was used.
Annotator, bibtex	Davies2009	Davies2007

percival2014/stem

Attribute	Value
Corpus	rwc_mdb_g
Version	1.0
Annotation Tools	percival 2014, ‘tempo’ implementation from Marsyas, http://marsyas.info, git checkout tempo-stem
Annotator, bibtex	Percival2014

schreiber2014/default

Attribute	Value
Corpus	rwc_mdb_g
Version	0.0.1
Annotation Tools	schreiber 2014, http://www.tagtraum.com/tempo_estimation.html
Annotator, bibtex	Schreiber2014

schreiber2017/ismir2017

Attribute	Value
Corpus	rwc_mdb_g
Version	0.0.4
Annotation Tools	schreiber 2017, model=ismir2017, http://www.tagtraum.com/tempo_estimation.html
Annotator, bibtex	Schreiber2017

schreiber2017/mirex2017

Attribute	Value
Corpus	rwc_mdb_g
Version	0.0.4
Annotation Tools	schreiber 2017, model=mirex2017, http://www.tagtraum.com/tempo_estimation.html
Annotator, bibtex	Schreiber2017

schreiber2018/cnn

Attribute	Value
Corpus
Version	0.0.3
Data Source	Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018.
Annotation Tools	schreiber tempo-cnn (model=cnn), https://github.com/hendriks73/tempo-cnn

schreiber2018/fcn

Attribute	Value
Corpus
Version	0.0.3
Data Source	Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018.
Annotation Tools	schreiber tempo-cnn (model=fcn), https://github.com/hendriks73/tempo-cnn

schreiber2018/ismir2018

Attribute	Value
Corpus
Version	0.0.3
Data Source	Hendrik Schreiber, Meinard Müller. A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, Sept. 2018.
Annotation Tools	schreiber tempo-cnn (model=ismir2018), https://github.com/hendriks73/tempo-cnn

Basic Statistics

Estimator	Size	Min	Max	Avg	Stdev	Sweet Oct. Start	Sweet Oct. Coverage
boeck2015/tempodetector2016_default	102	41.10	214.29	98.98	31.83	70.00	0.74
davies2009/mirex_qm_tempotracker	102	65.42	198.77	126.49	24.86	90.00	0.92
percival2014/stem	102	64.60	147.66	102.56	22.32	70.00	0.94
schreiber2014/default	102	58.18	145.06	100.44	23.32	65.00	0.88
schreiber2017/ismir2017	102	40.53	203.77	104.51	26.90	73.00	0.87
schreiber2017/mirex2017	102	40.53	203.77	102.90	28.78	73.00	0.84
schreiber2018/cnn	102	50.00	216.00	108.29	25.92	73.00	0.93
schreiber2018/fcn	102	41.00	208.00	106.76	28.52	74.00	0.84
schreiber2018/ismir2018	102	55.00	204.00	106.61	25.89	73.00	0.88

Table 2: Basic statistics.

CSV JSON LATEX PICKLE

Smoothed Tempo Distribution

Figure 4: Percentage of values in tempo interval.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy

Accuracy₁ is defined as the percentage of correct estimates, allowing a 4% tolerance for individual BPM values.

Accuracy₂ additionally permits estimates to be wrong by a factor of 2, 3, 1/2 or 1/3 (so-called octave errors).

See [Gouyon2006].

Note: When comparing accuracy values for different algorithms, keep in mind that an algorithm may have been trained on the test set or that the test set may have even been created using one of the tested algorithms.

Accuracy Results for 1.0

Estimator	Accuracy1	Accuracy2
schreiber2018/fcn	0.7255	0.8431
schreiber2018/cnn	0.7059	0.8333
schreiber2017/ismir2017	0.6961	0.8137
schreiber2018/ismir2018	0.6863	0.8235
schreiber2017/mirex2017	0.6863	0.8137
boeck2015/tempodetector2016_default	0.6667	0.8824
percival2014/stem	0.6569	0.8431
schreiber2014/default	0.6471	0.7941
davies2009/mirex_qm_tempotracker	0.5882	0.7647

Table 3: Mean accuracy of estimates compared to version 1.0 with 4% tolerance ordered by Accuracy₁.

CSV JSON LATEX PICKLE

Raw data Accuracy₁: CSV JSON LATEX PICKLE

Raw data Accuracy₂: CSV JSON LATEX PICKLE

Accuracy₁ for 1.0

Figure 5: Mean Accuracy₁ for estimates compared to version 1.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy₂ for 1.0

Figure 6: Mean Accuracy₂ for estimates compared to version 1.0 depending on tolerance.

CSV JSON LATEX PICKLE SVG PDF PNG

Differing Items

For which items did a given estimator not estimate a correct value with respect to a given ground truth? Are there items which are either very difficult, not suitable for the task, or incorrectly annotated and therefore never estimated correctly, regardless which estimator is used?

Differing Items Accuracy₁

Items with different tempo annotations (Accuracy₁, 4% tolerance) in different versions:

1.0 compared with boeck2015/tempodetector2016_default (34 differences): ‘RM-G003’ ‘RM-G011’ ‘RM-G012’ ‘RM-G032’ ‘RM-G037’ ‘RM-G039’ ‘RM-G041’ ‘RM-G043’ ‘RM-G046’ ‘RM-G047’ ‘RM-G048’ … CSV

1.0 compared with davies2009/mirex_qm_tempotracker (42 differences): ‘RM-G002’ ‘RM-G005’ ‘RM-G006’ ‘RM-G032’ ‘RM-G033’ ‘RM-G037’ ‘RM-G038’ ‘RM-G044’ ‘RM-G045’ ‘RM-G046’ ‘RM-G047’ … CSV

1.0 compared with percival2014/stem (35 differences): ‘RM-G010’ ‘RM-G011’ ‘RM-G021’ ‘RM-G032’ ‘RM-G037’ ‘RM-G043’ ‘RM-G046’ ‘RM-G047’ ‘RM-G048’ ‘RM-G050’ ‘RM-G052’ … CSV

1.0 compared with schreiber2014/default (36 differences): ‘RM-G028’ ‘RM-G032’ ‘RM-G037’ ‘RM-G043’ ‘RM-G046’ ‘RM-G047’ ‘RM-G048’ ‘RM-G050’ ‘RM-G051’ ‘RM-G052’ ‘RM-G053’ … CSV

1.0 compared with schreiber2017/ismir2017 (31 differences): ‘RM-G029’ ‘RM-G032’ ‘RM-G037’ ‘RM-G043’ ‘RM-G045’ ‘RM-G046’ ‘RM-G047’ ‘RM-G050’ ‘RM-G052’ ‘RM-G053’ ‘RM-G057’ … CSV

1.0 compared with schreiber2017/mirex2017 (32 differences): ‘RM-G029’ ‘RM-G032’ ‘RM-G037’ ‘RM-G043’ ‘RM-G045’ ‘RM-G046’ ‘RM-G047’ ‘RM-G048’ ‘RM-G050’ ‘RM-G052’ ‘RM-G053’ … CSV

1.0 compared with schreiber2018/cnn (30 differences): ‘RM-G029’ ‘RM-G030’ ‘RM-G032’ ‘RM-G037’ ‘RM-G045’ ‘RM-G046’ ‘RM-G047’ ‘RM-G048’ ‘RM-G050’ ‘RM-G052’ ‘RM-G056’ … CSV

1.0 compared with schreiber2018/fcn (28 differences): ‘RM-G029’ ‘RM-G032’ ‘RM-G033’ ‘RM-G037’ ‘RM-G045’ ‘RM-G047’ ‘RM-G048’ ‘RM-G050’ ‘RM-G052’ ‘RM-G056’ ‘RM-G058_A’ … CSV

1.0 compared with schreiber2018/ismir2018 (32 differences): ‘RM-G029’ ‘RM-G032’ ‘RM-G033’ ‘RM-G037’ ‘RM-G045’ ‘RM-G046’ ‘RM-G047’ ‘RM-G048’ ‘RM-G050’ ‘RM-G052’ ‘RM-G056’ … CSV

None of the estimators estimated the following 14 items ‘correctly’ using Accuracy₁: ‘RM-G032’ ‘RM-G037’ ‘RM-G047’ ‘RM-G052’ ‘RM-G058_A’ ‘RM-G058_C’ ‘RM-G060’ ‘RM-G063’ ‘RM-G079’ ‘RM-G085’ ‘RM-G089’ … CSV

Differing Items Accuracy₂

Items with different tempo annotations (Accuracy₂, 4% tolerance) in different versions:

1.0 compared with boeck2015/tempodetector2016_default (12 differences): ‘RM-G058_A’ ‘RM-G058_C’ ‘RM-G060’ ‘RM-G061’ ‘RM-G063’ ‘RM-G079’ ‘RM-G085’ ‘RM-G089’ ‘RM-G093’ ‘RM-G095’ ‘RM-G097’ … CSV

1.0 compared with davies2009/mirex_qm_tempotracker (24 differences): ‘RM-G046’ ‘RM-G051’ ‘RM-G052’ ‘RM-G058_A’ ‘RM-G058_B’ ‘RM-G058_C’ ‘RM-G060’ ‘RM-G061’ ‘RM-G063’ ‘RM-G077’ ‘RM-G079’ … CSV

1.0 compared with percival2014/stem (16 differences): ‘RM-G056’ ‘RM-G058_A’ ‘RM-G058_C’ ‘RM-G060’ ‘RM-G063’ ‘RM-G077’ ‘RM-G079’ ‘RM-G085’ ‘RM-G086’ ‘RM-G087’ ‘RM-G088’ … CSV

1.0 compared with schreiber2014/default (21 differences): ‘RM-G028’ ‘RM-G050’ ‘RM-G051’ ‘RM-G053’ ‘RM-G057’ ‘RM-G058_A’ ‘RM-G058_C’ ‘RM-G067’ ‘RM-G068’ ‘RM-G075’ ‘RM-G079’ … CSV

1.0 compared with schreiber2017/ismir2017 (19 differences): ‘RM-G052’ ‘RM-G053’ ‘RM-G058_A’ ‘RM-G058_C’ ‘RM-G060’ ‘RM-G061’ ‘RM-G063’ ‘RM-G068’ ‘RM-G079’ ‘RM-G084’ ‘RM-G085’ … CSV

1.0 compared with schreiber2017/mirex2017 (19 differences): ‘RM-G052’ ‘RM-G053’ ‘RM-G058_A’ ‘RM-G058_C’ ‘RM-G060’ ‘RM-G061’ ‘RM-G063’ ‘RM-G068’ ‘RM-G079’ ‘RM-G084’ ‘RM-G085’ … CSV

1.0 compared with schreiber2018/cnn (17 differences): ‘RM-G056’ ‘RM-G058_A’ ‘RM-G058_C’ ‘RM-G060’ ‘RM-G061’ ‘RM-G063’ ‘RM-G068’ ‘RM-G079’ ‘RM-G085’ ‘RM-G086’ ‘RM-G087’ … CSV

1.0 compared with schreiber2018/fcn (16 differences): ‘RM-G050’ ‘RM-G056’ ‘RM-G058_A’ ‘RM-G058_C’ ‘RM-G061’ ‘RM-G063’ ‘RM-G079’ ‘RM-G084’ ‘RM-G085’ ‘RM-G086’ ‘RM-G087’ … CSV

1.0 compared with schreiber2018/ismir2018 (18 differences): ‘RM-G056’ ‘RM-G058_A’ ‘RM-G058_C’ ‘RM-G060’ ‘RM-G061’ ‘RM-G063’ ‘RM-G079’ ‘RM-G080’ ‘RM-G084’ ‘RM-G085’ ‘RM-G086’ … CSV

None of the estimators estimated the following 5 items ‘correctly’ using Accuracy₂: ‘RM-G058_A’ ‘RM-G058_C’ ‘RM-G079’ ‘RM-G095’ ‘RM-G097’ CSV

Significance of Differences

Estimator	boeck2015/tempodetector2016_default	davies2009/mirex_qm_tempotracker	percival2014/stem	schreiber2014/default	schreiber2017/ismir2017	schreiber2017/mirex2017	schreiber2018/cnn	schreiber2018/fcn	schreiber2018/ismir2018
boeck2015/tempodetector2016_default	1.0000	0.2295	1.0000	0.8238	0.6776	0.8318	0.5413	0.3269	0.8450
davies2009/mirex_qm_tempotracker	0.2295	1.0000	0.2649	0.3616	0.0433	0.0639	0.0357	0.0066	0.0755
percival2014/stem	1.0000	0.2649	1.0000	1.0000	0.5034	0.6291	0.3018	0.1671	0.6476
schreiber2014/default	0.8238	0.3616	1.0000	1.0000	0.3323	0.4545	0.2863	0.1338	0.4807
schreiber2017/ismir2017	0.6776	0.0433	0.5034	0.3323	1.0000	1.0000	1.0000	0.5488	1.0000
schreiber2017/mirex2017	0.8318	0.0639	0.6291	0.4545	1.0000	1.0000	0.7539	0.3437	1.0000
schreiber2018/cnn	0.5413	0.0357	0.3018	0.2863	1.0000	0.7539	1.0000	0.7266	0.7266
schreiber2018/fcn	0.3269	0.0066	0.1671	0.1338	0.5488	0.3437	0.7266	1.0000	0.2188
schreiber2018/ismir2018	0.8450	0.0755	0.6476	0.4807	1.0000	1.0000	0.7266	0.2188	1.0000

Table 4: McNemar p-values, using reference annotations 1.0 as groundtruth with Accuracy₁ [Gouyon2006]. H₀: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H₀, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator	boeck2015/tempodetector2016_default	davies2009/mirex_qm_tempotracker	percival2014/stem	schreiber2014/default	schreiber2017/ismir2017	schreiber2017/mirex2017	schreiber2018/cnn	schreiber2018/fcn	schreiber2018/ismir2018
boeck2015/tempodetector2016_default	1.0000	0.0042	0.2891	0.0352	0.0391	0.0391	0.1250	0.3437	0.0312
davies2009/mirex_qm_tempotracker	0.0042	1.0000	0.0574	0.6636	0.3018	0.3018	0.1185	0.0768	0.1796
percival2014/stem	0.2891	0.0574	1.0000	0.3593	0.5488	0.5488	1.0000	1.0000	0.6875
schreiber2014/default	0.0352	0.6636	0.3593	1.0000	0.8036	0.8036	0.4807	0.3323	0.6072
schreiber2017/ismir2017	0.0391	0.3018	0.5488	0.8036	1.0000	1.0000	0.6875	0.5078	1.0000
schreiber2017/mirex2017	0.0391	0.3018	0.5488	0.8036	1.0000	1.0000	0.6875	0.5078	1.0000
schreiber2018/cnn	0.1250	0.1185	1.0000	0.4807	0.6875	0.6875	1.0000	1.0000	1.0000
schreiber2018/fcn	0.3437	0.0768	1.0000	0.3323	0.5078	0.5078	1.0000	1.0000	0.6875
schreiber2018/ismir2018	0.0312	0.1796	0.6875	0.6072	1.0000	1.0000	1.0000	0.6875	1.0000

Table 5: McNemar p-values, using reference annotations 1.0 as groundtruth with Accuracy₂ [Gouyon2006]. H₀: both estimators disagree with the groundtruth to the same amount. If p<=ɑ, reject H₀, i.e. we have a significant difference in the disagreement with the groundtruth. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Accuracy₁ on c_var-Subsets

How well does an estimator perform, when only taking tracks into account that have a c_var-value of less than τ, i.e., have a more or less stable beat?

Accuracy₁ on c_var-Subsets for 1.0 based on c_var-Values from 1.0

Figure 7: Mean Accuracy₁ compared to version 1.0 for tracks with c_var < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy₂ on c_var-Subsets

How well does an estimator perform, when only taking tracks into account that have a c_var-value of less than τ, i.e., have a more or less stable beat?

Accuracy₂ on c_var-Subsets for 1.0 based on c_var-Values from 1.0

Figure 8: Mean Accuracy₂ compared to version 1.0 for tracks with c_var < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy₁ on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean Accuracy₁ for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

Accuracy₁ on Tempo-Subsets for 1.0

Figure 9: Mean Accuracy₁ for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy₂ on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean Accuracy₂ for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

Accuracy₂ on Tempo-Subsets for 1.0

Figure 10: Mean Accuracy₂ for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy₁ for Tempo

When fitting a generalized additive model (GAM) to Accuracy₁-values and a ground truth, what Accuracy₁ can we expect with confidence?

Estimated Accuracy₁ for Tempo for 1.0

Predictions of GAMs trained on Accuracy₁ for estimates for reference 1.0.

Figure 11: Accuracy₁ predictions of a generalized additive model (GAM) fit to Accuracy₁ results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated Accuracy₂ for Tempo

When fitting a generalized additive model (GAM) to Accuracy₂-values and a ground truth, what Accuracy₂ can we expect with confidence?

Estimated Accuracy₂ for Tempo for 1.0

Predictions of GAMs trained on Accuracy₂ for estimates for reference 1.0.

Figure 12: Accuracy₂ predictions of a generalized additive model (GAM) fit to Accuracy₂ results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy₁ for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

Accuracy₁ for ‘tag_open’ Tags for 1.0

Figure 13: Mean Accuracy₁ of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

Accuracy₂ for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

Accuracy₂ for ‘tag_open’ Tags for 1.0

Figure 14: Mean Accuracy₂ of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

CSV JSON LATEX PICKLE SVG PDF PNG

OE₁ and OE₂

OE₁ is defined as octave error between an estimate E and a reference value R.This means that the most common errors—by a factor of 2 or ½—have the same magnitude, namely 1: OE2(E) = log2(E/R).

OE₂ is the signed OE₁ corresponding to the minimum absolute OE₁ allowing the octaveerrors 2, 3, 1/2, and 1/3: OE2(E) = arg minx(|x|) with x ∈ {OE1(E), OE1(2E), OE1(3E), OE1(½E), OE1(⅓E)}

Mean OE₁/OE₂ Results for 1.0

Estimator	OE1_MEAN	OE1_STDEV	OE2_MEAN	OE2_STDEV
schreiber2018/fcn	0.0752	0.4131	0.0008	0.0814
schreiber2018/ismir2018	0.0842	0.4711	0.0098	0.1056
schreiber2014/default	-0.0001	0.4740	0.0040	0.1273
schreiber2017/ismir2017	0.0456	0.4949	0.0064	0.1231
schreiber2018/cnn	0.1101	0.5001	0.0064	0.1046
schreiber2017/mirex2017	0.0105	0.5231	0.0105	0.1229
davies2009/mirex_qm_tempotracker	0.3452	0.5231	0.0477	0.1317
percival2014/stem	0.0358	0.5328	0.0104	0.1080
boeck2015/tempodetector2016_default	-0.0604	0.5891	0.0042	0.0864

Table 6: Mean OE1/OE2 for estimates compared to version 1.0 ordered by standard deviation.

CSV JSON LATEX PICKLE

Raw data OE₁: CSV JSON LATEX PICKLE

Raw data OE₂: CSV JSON LATEX PICKLE

OE₁ distribution for 1.0

Figure 15: OE₁ for estimates compared to version 1.0. Shown are the mean OE₁ and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

OE₂ distribution for 1.0

Figure 16: OE₂ for estimates compared to version 1.0. Shown are the mean OE₂ and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Significance of Differences

Estimator	boeck2015/tempodetector2016_default	davies2009/mirex_qm_tempotracker	percival2014/stem	schreiber2014/default	schreiber2017/ismir2017	schreiber2017/mirex2017	schreiber2018/cnn	schreiber2018/fcn	schreiber2018/ismir2018
boeck2015/tempodetector2016_default	1.0000	0.0000	0.0595	0.1988	0.0481	0.1903	0.0004	0.0121	0.0033
davies2009/mirex_qm_tempotracker	0.0000	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
percival2014/stem	0.0595	0.0000	1.0000	0.3348	0.8249	0.5785	0.0446	0.3918	0.2488
schreiber2014/default	0.1988	0.0000	0.3348	1.0000	0.2919	0.8112	0.0034	0.0531	0.0173
schreiber2017/ismir2017	0.0481	0.0000	0.8249	0.2919	1.0000	0.1214	0.0899	0.4587	0.2787
schreiber2017/mirex2017	0.1903	0.0000	0.5785	0.8112	0.1214	1.0000	0.0132	0.1096	0.0495
schreiber2018/cnn	0.0004	0.0000	0.0446	0.0034	0.0899	0.0132	1.0000	0.3222	0.2773
schreiber2018/fcn	0.0121	0.0000	0.3918	0.0531	0.4587	0.1096	0.3222	1.0000	0.7272
schreiber2018/ismir2018	0.0033	0.0000	0.2488	0.0173	0.2787	0.0495	0.2773	0.7272	1.0000

Table 7: Paired t-test p-values, using reference annotations 1.0 as groundtruth with OE₁. H₀: the true mean difference between paired samples is zero. If p<=ɑ, reject H₀, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator	boeck2015/tempodetector2016_default	davies2009/mirex_qm_tempotracker	percival2014/stem	schreiber2014/default	schreiber2017/ismir2017	schreiber2017/mirex2017	schreiber2018/cnn	schreiber2018/fcn	schreiber2018/ismir2018
boeck2015/tempodetector2016_default	1.0000	0.0035	0.4700	0.9906	0.8261	0.5411	0.8412	0.7104	0.5678
davies2009/mirex_qm_tempotracker	0.0035	1.0000	0.0160	0.0229	0.0258	0.0396	0.0129	0.0016	0.0263
percival2014/stem	0.4700	0.0160	1.0000	0.7159	0.7689	0.9975	0.7057	0.2997	0.9558
schreiber2014/default	0.9906	0.0229	0.7159	1.0000	0.8764	0.6800	0.8712	0.8531	0.6970
schreiber2017/ismir2017	0.8261	0.0258	0.7689	0.8764	1.0000	0.3197	0.9967	0.6776	0.6919
schreiber2017/mirex2017	0.5411	0.0396	0.9975	0.6800	0.3197	1.0000	0.7659	0.4653	0.9353
schreiber2018/cnn	0.8412	0.0129	0.7057	0.8712	0.9967	0.7659	1.0000	0.6564	0.7872
schreiber2018/fcn	0.7104	0.0016	0.2997	0.8531	0.6776	0.4653	0.6564	1.0000	0.4167
schreiber2018/ismir2018	0.5678	0.0263	0.9558	0.6970	0.6919	0.9353	0.7872	0.4167	1.0000

Table 8: Paired t-test p-values, using reference annotations 1.0 as groundtruth with OE₂. H₀: the true mean difference between paired samples is zero. If p<=ɑ, reject H₀, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

OE₁ on c_var-Subsets

How well does an estimator perform, when only taking tracks into account that have a c_var-value of less than τ, i.e., have a more or less stable beat?

OE₁ on c_var-Subsets for 1.0 based on c_var-Values from 1.0

Figure 17: Mean OE₁ compared to version 1.0 for tracks with c_var < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

OE₂ on c_var-Subsets

How well does an estimator perform, when only taking tracks into account that have a c_var-value of less than τ, i.e., have a more or less stable beat?

OE₂ on c_var-Subsets for 1.0 based on c_var-Values from 1.0

Figure 18: Mean OE₂ compared to version 1.0 for tracks with c_var < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

OE₁ on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean OE₁ for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

OE₁ on Tempo-Subsets for 1.0

Figure 19: Mean OE₁ for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

OE₂ on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean OE₂ for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

OE₂ on Tempo-Subsets for 1.0

Figure 20: Mean OE₂ for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE₁ for Tempo

When fitting a generalized additive model (GAM) to OE₁-values and a ground truth, what OE₁ can we expect with confidence?

Estimated OE₁ for Tempo for 1.0

Predictions of GAMs trained on OE₁ for estimates for reference 1.0.

Figure 21: OE₁ predictions of a generalized additive model (GAM) fit to OE₁ results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated OE₂ for Tempo

When fitting a generalized additive model (GAM) to OE₂-values and a ground truth, what OE₂ can we expect with confidence?

Estimated OE₂ for Tempo for 1.0

Predictions of GAMs trained on OE₂ for estimates for reference 1.0.

Figure 22: OE₂ predictions of a generalized additive model (GAM) fit to OE₂ results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

OE₁ for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

OE₁ for ‘tag_open’ Tags for 1.0

Figure 23: OE₁ of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

OE₂ for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

OE₂ for ‘tag_open’ Tags for 1.0

Figure 24: OE₂ of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE₁ and AOE₂

AOE₁ is defined as absolute octave error between an estimate and a reference value: AOE1(E) = |log2(E/R)|.

AOE₂ is the minimum of AOE₁ allowing the octave errors 2, 3, 1/2, and 1/3: AOE2(E) = min(AOE1(E), AOE1(2E), AOE1(3E), AOE1(½E), AOE1(⅓E)).

Mean AOE₁/AOE₂ Results for 1.0

Estimator	AOE1_MEAN	AOE1_STDEV	AOE2_MEAN	AOE2_STDEV
schreiber2018/fcn	0.1925	0.3731	0.0349	0.0735
schreiber2018/ismir2018	0.2344	0.4172	0.0435	0.0967
schreiber2017/ismir2017	0.2368	0.4370	0.0504	0.1124
schreiber2018/cnn	0.2425	0.4510	0.0421	0.0959
schreiber2014/default	0.2524	0.4012	0.0583	0.1132
schreiber2017/mirex2017	0.2629	0.4524	0.0505	0.1126
percival2014/stem	0.2844	0.4520	0.0412	0.1003
boeck2015/tempodetector2016_default	0.3251	0.4950	0.0322	0.0803
davies2009/mirex_qm_tempotracker	0.3748	0.5023	0.0770	0.1170

Table 9: Mean AOE1/AOE2 for estimates compared to version 1.0 ordered by mean.

CSV JSON LATEX PICKLE

Raw data AOE₁: CSV JSON LATEX PICKLE

Raw data AOE₂: CSV JSON LATEX PICKLE

AOE₁ distribution for 1.0

Figure 25: AOE₁ for estimates compared to version 1.0. Shown are the mean AOE₁ and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

AOE₂ distribution for 1.0

Figure 26: AOE₂ for estimates compared to version 1.0. Shown are the mean AOE₂ and an empirical distribution of the sample, using kernel density estimation (KDE).

CSV JSON LATEX PICKLE SVG PDF PNG

Significance of Differences

Estimator	boeck2015/tempodetector2016_default	davies2009/mirex_qm_tempotracker	percival2014/stem	schreiber2014/default	schreiber2017/ismir2017	schreiber2017/mirex2017	schreiber2018/cnn	schreiber2018/fcn	schreiber2018/ismir2018
boeck2015/tempodetector2016_default	1.0000	0.3962	0.3668	0.0990	0.0873	0.2150	0.0840	0.0095	0.0597
davies2009/mirex_qm_tempotracker	0.3962	1.0000	0.0746	0.0202	0.0057	0.0244	0.0066	0.0001	0.0037
percival2014/stem	0.3668	0.0746	1.0000	0.3532	0.2590	0.6076	0.2533	0.0316	0.2058
schreiber2014/default	0.0990	0.0202	0.3532	1.0000	0.7043	0.7963	0.7935	0.0996	0.5974
schreiber2017/ismir2017	0.0873	0.0057	0.2590	0.7043	1.0000	0.1825	0.8793	0.2486	0.9464
schreiber2017/mirex2017	0.2150	0.0244	0.6076	0.7963	0.1825	1.0000	0.5888	0.0675	0.4276
schreiber2018/cnn	0.0840	0.0066	0.2533	0.7935	0.8793	0.5888	1.0000	0.1347	0.7319
schreiber2018/fcn	0.0095	0.0001	0.0316	0.0996	0.2486	0.0675	0.1347	1.0000	0.0830
schreiber2018/ismir2018	0.0597	0.0037	0.2058	0.5974	0.9464	0.4276	0.7319	0.0830	1.0000

Table 10: Paired t-test p-values, using reference annotations 1.0 as groundtruth with AOE₁. H₀: the true mean difference between paired samples is zero. If p<=ɑ, reject H₀, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

Estimator	boeck2015/tempodetector2016_default	davies2009/mirex_qm_tempotracker	percival2014/stem	schreiber2014/default	schreiber2017/ismir2017	schreiber2017/mirex2017	schreiber2018/cnn	schreiber2018/fcn	schreiber2018/ismir2018
boeck2015/tempodetector2016_default	1.0000	0.0000	0.2501	0.0165	0.0075	0.0074	0.2462	0.7279	0.0215
davies2009/mirex_qm_tempotracker	0.0000	1.0000	0.0006	0.1957	0.0259	0.0265	0.0021	0.0002	0.0032
percival2014/stem	0.2501	0.0006	1.0000	0.1890	0.3630	0.3590	0.9097	0.4176	0.7551
schreiber2014/default	0.0165	0.1957	0.1890	1.0000	0.4552	0.4613	0.1545	0.0267	0.1557
schreiber2017/ismir2017	0.0075	0.0259	0.3630	0.4552	1.0000	0.3197	0.2793	0.1038	0.3444
schreiber2017/mirex2017	0.0074	0.0265	0.3590	0.4613	0.3197	1.0000	0.2735	0.1022	0.3396
schreiber2018/cnn	0.2462	0.0021	0.9097	0.1545	0.2793	0.2735	1.0000	0.3162	0.8534
schreiber2018/fcn	0.7279	0.0002	0.4176	0.0267	0.1038	0.1022	0.3162	1.0000	0.2246
schreiber2018/ismir2018	0.0215	0.0032	0.7551	0.1557	0.3444	0.3396	0.8534	0.2246	1.0000

Table 11: Paired t-test p-values, using reference annotations 1.0 as groundtruth with AOE₂. H₀: the true mean difference between paired samples is zero. If p<=ɑ, reject H₀, i.e. we have a significant difference between estimates from the two algorithms. In the table, p-values<0.05 are set in bold.

CSV JSON LATEX PICKLE

AOE₁ on c_var-Subsets

How well does an estimator perform, when only taking tracks into account that have a c_var-value of less than τ, i.e., have a more or less stable beat?

AOE₁ on c_var-Subsets for 1.0 based on c_var-Values from 1.0

Figure 27: Mean AOE₁ compared to version 1.0 for tracks with c_var < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE₂ on c_var-Subsets

How well does an estimator perform, when only taking tracks into account that have a c_var-value of less than τ, i.e., have a more or less stable beat?

AOE₂ on c_var-Subsets for 1.0 based on c_var-Values from 1.0

Figure 28: Mean AOE₂ compared to version 1.0 for tracks with c_var < τ based on beat annotations from 1.0.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE₁ on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean AOE₁ for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

AOE₁ on Tempo-Subsets for 1.0

Figure 29: Mean AOE₁ for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE₂ on Tempo-Subsets

How well does an estimator perform, when only taking a subset of the reference annotations into account? The graphs show mean AOE₂ for reference subsets with tempi in [T-10,T+10] BPM. Note that the graphs do not show confidence intervals and that some values may be based on very few estimates.

AOE₂ on Tempo-Subsets for 1.0

Figure 30: Mean AOE₂ for estimates compared to version 1.0 for tempo intervals around T.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE₁ for Tempo

When fitting a generalized additive model (GAM) to AOE₁-values and a ground truth, what AOE₁ can we expect with confidence?

Estimated AOE₁ for Tempo for 1.0

Predictions of GAMs trained on AOE₁ for estimates for reference 1.0.

Figure 31: AOE₁ predictions of a generalized additive model (GAM) fit to AOE₁ results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

Estimated AOE₂ for Tempo

When fitting a generalized additive model (GAM) to AOE₂-values and a ground truth, what AOE₂ can we expect with confidence?

Estimated AOE₂ for Tempo for 1.0

Predictions of GAMs trained on AOE₂ for estimates for reference 1.0.

Figure 32: AOE₂ predictions of a generalized additive model (GAM) fit to AOE₂ results for 1.0. The 95% confidence interval around the prediction is shaded in gray.

CSV JSON LATEX PICKLE SVG PDF PNG

AOE₁ for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

AOE₁ for ‘tag_open’ Tags for 1.0

Figure 33: AOE₁ of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

AOE₂ for ‘tag_open’ Tags

How well does an estimator perform, when only taking tracks into account that are tagged with some kind of label? Note that some values may be based on very few estimates.

AOE₂ for ‘tag_open’ Tags for 1.0

Figure 34: AOE₂ of estimates compared to version 1.0 depending on tag from namespace ‘tag_open’.

SVG PDF PNG

---

Generated by tempo_eval 0.1.1 on 2022-06-29 18:52. Size L.