Automatic Musical Instrument Identification

Some recent papers on AMII

2011-05-30T07:10:00.000-07:00

Some AMII papers recently release :

Al-Taee, M.A. et al., 2009. Analysis and Pattern Recognition of Woodwind Musical Tones Applied to Query-by-Playing Melody Retrieval. World Congress on Engineering 2009 (Volume 1), 1, pp.118-123.
Driscoll, V.D. et al., 2009. Effects of Training on Recognition of Musical Instruments Presented through Cochlear Implant Simulations. Journal of the American Academy of Audiology, 20(1), pp.71-82. Available at: http://openurl.ingenta.com/content/xref?genre=article&issn=1050-0545&volume=20&issue=1&spage=71 [Accessed May 30, 2011].
Eldridge, M., Saltzman, E. & Lahav, A., 2010. Seeing what you hear: Visual feedback improves pitch recognition. European Journal of Cognitive Psychology, 22(7), pp.1078-1091. Available at: http://www.informaworld.com/openurl?genre=article&doi=10.1080/09541440903316136&magic=crossref||D404A21C5BB053405B1A640AFFD44AE3 [Accessed May 30, 2011].
Jiang, W. et al., 2010. Multiple Classifiers for Different Features in Timbre Estimation. In Z. W. Ras & L.-S. Tsay, eds. Advances in Intelligent Information Systems. Springer.
Morvidone, M., Sturm, B.L. & Daudet, L., 2010. Incorporating scale information with cepstral features: Experiments on musical instrument recognition. Pattern Recognition Letters, 31(12), pp.1489-1497. Available at: http://linkinghub.elsevier.com/retrieve/pii/S0167865509003778 [Accessed May 30, 2011].
Tjoa, S.K. & Liu, K.J.R., 2010. Musical Instrument Recognition Using Biologically Inspired Filtering OF Temporal Dictionary Atoms. In ISMIR 2010. pp. 435-440. Available at: http://www.cspl.umd.edu/sig/publications/Tjoa_ISMIR_201008.pdf.

More online music resources

2010-10-19T03:44:00.000-07:00

Naxos Music Library

Oxford Music Online

Pretty cool document organiser

2010-09-15T08:37:00.000-07:00

I stumbled upon Mendeley Desktop. Could it be a free replacement for Endnote?

The LibSVM forum is here

2010-08-27T07:29:00.000-07:00

Here it is! The LibSVM forum.

My thanks to Benoit Mathieu for helping me with YAAFE

2010-08-18T06:07:00.000-07:00

I must thank again Benoit Mathieu for helping me with the YAAFE installation. I was having a torrid time, spending almost 8 days installing UNIX (Ubuntu and then Fedora) and the various required extensions:

HDF5 http://www.hdfgroup.org/HDF5/>`_ version 1.8 or higher
libsndfile http://www.mega-nerd.com/libsndfile/>
libmpg123 http://www.mpg123.de/api/>
liblapack http://www.netlib.org/lapack/>
blitz++ http://www.oonumerics.org/blitz/>
argtable http://argtable.sourceforge.net/>

Users beware of the need to also install:

SWIG
python-devel
c (g++) compiler
Fortran (f90/f95) compiler
cmake

For users running Windows OS, you can install VMWare Player to host a virtual UNIX OS.

I can now say I'm quite efficient in the use of UNIX - about which I've been very impressed. I now have a standalone dedicated UNIX (Red Hat Fedora 13) system, which is a joy to work on.

Now to enjoy some feature extracting!

Presentation on use of SVM for Instrument ID

2010-07-28T07:52:00.001-07:00

Uwe Ligges and Sebastian Krey present their research on Instrument ID, "SVM based Classification of Instruments-Timbre Analysis" in these slides.

I'm hoping to soon get a copy of their publication "Krey, S. and Ligges, U. (2010): SVM based Instrument and Timbre Classification. In Locarek-Junge, H. and Weihs, C. (eds.). Classification as a Tool for Research, Springer-Verlag, Berlin. (in print)"

A very good (relatively up-to-date - 2007) overview of Classification in Music Research

2010-07-28T07:27:00.000-07:00

Link:

http://http//www.statistik.tu-dortmund.de/fileadmin/user_upload/Lehrstuehle/Datenanalyse/weihs07classification.pdf

The abstract:

Since some few years, classification in music research is a very broad and quickly growing field. Most important for adequate classi cation is the knowledge of adequate observable or deduced features on the basis of which meaningful groups or classes can be distinguished. Unsupervised classi cation additionally needs an adequate similarity or distance measure grouping is to be based upon. Evaluation of supervised learning is typically based on the error rates of the classi cation rules. In this paper we fi rst discuss typical problems and possible influential features derived from signal analysis, mental mechanisms or concepts, and compositional structure. Then, we present typical solutions of such tasks related to music research, namely for organization of music collections, transcription of music signals, cognitive psychology of music, and compositional structure analysis.

More AMII researchers

2010-07-28T07:05:00.000-07:00

Sebastian Krey

Dr. Uwe Ligges

Two papers available online are of interest:

Weihs, C., Reuter, C. and Ligges, U. (2004): "Register Classification by Timbre", Technical Report 71/2004. SFB 475, Department of Statistics, University of Dortmund, Germany.
Szepannek, G., Ligges, U., Luebke, K., Raabe, N. and Weihs, C. (2005): "Local Models in Register Classification by Timbre", Technical Report 47/2005. SFB 475, Department of Statistics, University of Dortmund, Germany.

Papers and posters of interest in this coming ISMIR

2010-07-28T04:20:00.000-07:00

ISMIR is around the corner - the program is up.

In relation to AMII - these look interesting:

"Musical Instrument Recognition using Biologically Inspired Filtering of Temporal Dictionary Atoms", Steven K. Tjoa and K.J. Ray Liu (My thanks to Steven for emailing me a copy of this today)
"YAAFE, an Easy to Use and Efficient Audio Feature Extraction Software", Benoit Mathieu, Slim Essid, Thomas Fillon, Jacques Prado and Gaël Richard

The YAAFE toolbox is available here with an extension module here. Of the extension features, the SpectralIrregularity seems the most interesting IMO.

The available features in YAAFE basic are:

AmplitudeModulation
AutoCorrelation
ComplexDomainOnsetDetection
Energy
Envelope
EnvelopeShapeStatistics
Frames
LPC
LSF
Loudness
MFCC
MagnitudeSpectrum
OBSI
OBSIR
PerceptualSharpness
PerceptualSpread
SpectralCrestFactorPerBand
SpectralDecrease
SpectralFlatness
SpectralFlatnessPerBand
SpectralFlux
SpectralRolloff
SpectralShapeStatistics
SpectralSlope
SpectralVariation
TemporalShapeStatistics
ZCR

Available feature transforms:

AutoCorrelationPeaksIntegrator
Cepstrum
Derivate
HistogramIntegrator
SlopeIntegrator
StatisticalIntegrator

AND TO THINK OF THE AMOUNT OF TIME I SPENT WRITING CODE TO EXTRACT THESE FEATURES!!!!

Another AMII researcher - Alexey Ozerov

2010-07-26T06:31:00.000-07:00

I stumbled upon Alexey Ozerov - musical instrument recognition being just one aspect of his research.

Two papers are of significant interest to my own research:

Ozerov, A., Essid, S. Tech Rep 2009 "Instrument recognition in polyphonic music based on NMF decomposition and SVM classification"
J.-L. Durrieu, A. Ozerov, C. Févotte, G. Richard and B. David, "Main instrument separation from stereophonic audio signals using a source/filter model", In EUSIPCO, 17th European Signal Processing Conference, Glasgow, Scotland, August 24-28, 2009.

SVMs demystified...

2010-07-19T16:37:00.000-07:00

I came across an excellent blog 'Onionesque Reality' by Shubhendu Trivedi. One article entitled "Demystifying Support Vector Machines for Beginners" does exactly what it says on the tin. Trivedi also blogs how SVMs can be used in Face Recognition and even attempts to answer why SVMs are so called.

Two books which are recommended caught my interest:

"Support Vector Machines and other Kernel Based Learning methods" by Nello Cristianini and John-Shawe Taylor.

"Learning with Kernels" by Bernhard Scholkopf and Alexander Smola. (Perfect book for beginners)

I've a lot of reading to do!

And before I forget, this lecture on SVM is meant to be awesome.

Another toolbox!

2010-07-19T16:34:00.000-07:00

SHOGUN - A large scale machine learning toolbox. Will I or wont I...?

Validating predictive models

2010-07-18T13:20:00.000-07:00

Will Dwinnell blogs about Validating Predictive Models with links:

Something different - Music movies

2010-07-18T03:38:00.000-07:00

A superb list of music movies is provided by the UC library (Berkeley). Would be well worth checking some of these out.

Ensemble classifiers win PAKDD 2010

2010-07-17T13:35:00.000-07:00

Taken from Data Mining and Predictive Analysis:

PAKDD-10 Data Mining Competition Winner: Ensembles Again!

The PAKDD-10 Data Mining Competition results are in, and ensembles occupied the top 4 positions, and I think the top 5. The winner used Stochastic Gradient Boosting and Random Forests in Statistica, second place a combination of logistic regression and Stochastic Gradient Boosting (and Salford Systems CART for some feature extraction). Interestingly to me, the 5th place finisher used WEKA, an open source software tool.

Evaluating classifier performance

2010-07-17T12:54:00.000-07:00

An article in the excellent blog by Will Dwinnel, discusses evaluating classifier performance using Confusion Matrices, ROC, Lift charts and AUROC. Links to external resources are provided:

AUC: a Statistically Consistent and more Discriminating Measure than Accuracy, by Charles X. Ling, Jin Huang and Harry Zhang
Evaluating Performance, from “ROC Graphs: Notes and Practical Considerations for Researchers, by T. Fawcett
The Use of the Area Under the ROC Curve in the Evaluation of Machine Learning Algorithms, by Andrew P. Bradley

In the blog entry "Model Performance Measurement", Matlab code is provided for various performance measurement routines:

'L-1' (mean absolute error)
'L-2' (mean squared error)
'L-4'
'L-16'
'L-Infinity'
'RMS' (root mean squared error)
'AUC' (requires tiedrank() from Statistics Toolbox)
'Bias'
'Conditional Entropy'
'Cross-Entropy'
'F-Measure'
'Informational Loss'
'MAPE'
'Median Squared Error'
'Worst 10%'
'Worst 20%'

Ensemble Classifiers: Toolboxes and links

2010-06-21T11:07:00.000-07:00

This link to Matlab Toolboxes is not new but still remains quite useful, especially as it is updated.

ENTOOL: Ensemble Classifiers for Statistical Learning (Toolbox for Matlab)

Classification Tool (Matlab)

Learning and Mining from Data

Lots of examples available

Feature Extraction, Foundations and Applications by Isabelle Guyon, Steve Gunn, Masoud Nikravesh, and Lofti Zadeh

DART-Europe E-theses Portal

2010-06-11T13:58:00.000-07:00

DART-Europe E-theses Portal is available for use to search online thesis repositories to subscribing universities and institutions.

The following theses are of interest:

Audio content processing for automatic music genre classification : descriptors, databases, and classifiers, Guaus, Enric.
Aspects in Analysis and Model-Based Sound Synthesis of Plucked String Instruments, Erkut, Cumhur
Correspondences between music and body movement, Haga, Egil
Design and realisation of an efficient content based music playlist generation system, Balkema, Jan Wietse
Feature extraction of musical content for automatic music transcription, Zhou, Ruohua
Machine annotation of traditional Irish dance music, Duggan, Bryan
Music complexity: a multi-faceted description of audio content, Streich, Sebastian
Probabilistic models for music, Paiement, Jean-François
Searching and Classifying non-textual information, Arentz, Will Archer
Structural analysis and segmentation of music signals, Ong, Bee Suan
The acoustics of the violin, Johnson, E
Tonal description of music audio signals, Gómez Gutiérrez, Emilia
Acoustic Event Detection and Classification, Temko, Andriy
Audio content processing for automatic music genre classification : descriptors, databases, and classifiers, Guaus, Enric

How to complete your PhD

2010-03-10T04:52:00.000-08:00

Some tips from Academic Productivity: How to complete your PhD.

Introduction to Matlab course from MIT

2010-03-10T04:48:00.000-08:00

An introduction to Matlab course has been added to MIT OpenCourseWare.

How not to write a PhD thesis

2010-02-03T05:04:00.001-08:00

A great article in the Times Higher Education on 'How not to write a PhD thesis' by Tara Brabazon (Professor of Media Studies, University of Brighton.)

Need help understanding statistics?

2010-01-07T06:28:00.000-08:00

I found a great video based tutorial on interpreting statistics by The Teaching Company. Information on the tutorial can be found here. It will only cost you close to 250 USDollars!

A listing of some new relevant publications

2009-12-15T11:47:00.000-08:00

A listing of some recent publications (well most are recent) relevant to my own research area:

Brown, J.C. and P. Smaragdis (2009).``Hidden Markov and Gaussian mixture models for automatic call classification" J.Acoust. Soc. Am. 125, EL221-224.

Reconnaissance des instruments dans la musique polyphonique par décomposition NMF et classification SVM, Alexey Ozerov, Slim Essid and Maurice Charbit

Incorporating prior knowledge on the digital media creation process into audio classifiers, M. Lardeur, S. Essid, G. Richard

The Significance of the Non-Harmonic “Noise” Versus the Harmonic Series for Musical Instrument Recognition, Livshin, A., Rodet, X., 2006. ISMIR 2006

Livshin, A., Rodet, X. 2009 - Purging Musical Instrument Sample Databases Using Automatic Musical Instrument Recognition Methods IEEE Transactions on Audio, Speech and Language Processing, Vol 17, no. 5. pp. 1046-1051, July 2009

Martins, L.G. (PhD Thesis) 2009 - Livshin, A., Rodet, X. 2009 - Purging Musical Instrument Sample Databases Using Automatic Musical Instrument Recognition Methods

Jiang, W., Zhang, X., Ras, Z.W., and Cohen, A. (2009). Multiple Classifiers for Different Features in Timbre Estimation, in Advances in Intelligent Information Systems (Eds. Ras, Z.W. and Tsay, L.), Studies in Computational Intelligence, Springer, will appear

Soo-Chang Pei, Nien-Teh Hsu 2009 - Instrumentation analysis and identification of polyphonic music using beat-synchronous feature integration and fuzzy clustering

ISMIR 2009, Session Title: Musical Instrument Recognition & Multipitch Detection

Fuhrmann, F., Haro, M., Herrera, P. (2009). Scalability, generality and temporal aspects in automatic recognition of predominant musical instruments in polyphonic music. Conference of the International Society for Music Information Retrieval (ISMIR)

Musical Instrument Recognition in Polyphonic Audio Using Source-Filter Model for Sound Separation, Toni Heittola, Anssi Klapuri and Tuomas Virtanen (ISMIR)

ISMIR 2009 Poster Session (PS3):

Automatic Identification of Instrument Classes in Polyphonic and Poly-Instrument Audio, Philippe Hamel, Sean Wood and Douglas Eck

Perceptions based on prior knowledge?

2009-11-17T16:01:00.000-08:00

I found a very interesting journal article on how our perceptions are based on prior experience. It ties in with the perception of sounds.

Some more papers on instrument ID

2009-09-04T07:28:00.000-07:00

Three papers my supervisor has brought to my attention:

"Instrumentation analysis and identification of polyphonic music using beat-synchronous feature integration and fuzzy clustering", by Soo-Chang Pei and Nien-Teh Hsu
"Polyphonic Musical Instrument Recognition Based on a Dynamic Model of the Spectral Envelope", J.J. Burred, A. Röbel and T. Sikora
"Musical Instrument Recognition in Polyphonic Audio Using Source-Filter Model for Sound Separation", T. Heittola, A. Klapuri, and T. Virtanen

IPEM and the psychology of electronic music

2009-08-27T08:29:00.000-07:00

I'm working with the IPEM Toolbox. A difficult task given that it's tied to Matlab versions 5.3.1 and 6.0. The source code is available and can be toyed with to try to make it compatible with later versions. A fix is to find the original versions, more as a time saving exercise! But as an aside, the concept of IPEM and the extraction of perceptual features is of interest to my own research. And in particular, being an electronic music fan, this article discusses the "Psychology of Electronic Music".

4475 separted notes

2009-08-12T11:34:00.000-07:00

I separated the notes from the majority of the IOWA musical instrument samples. Total notes = 4475. WOW! It wasn't easy! I'll be writing about how I did it in my next publication. Look out a few short lines describing about 9 day's of hard work!!

Another Instrument ID paper

2009-08-03T03:59:00.000-07:00

I came across another paper on musical instrument ID today:

Loughran,R., Walker,J., O'Neill, M., O'Farrell, M., 2008 - Musical Instrument Identification Using Principal Component Analysis and Multi-Layered Perceptrons.pdf

Loughran, R. uses the temporal envelope of the signal residual (post removal of the RMS temporal envelope) as a feature which I haven't come across before in the literature.

"Temporal and spectral envelopes. The temporal envelope was found by calculating the RMS energy envelope of each sound, which was then filtered using a 3rd order low pass Butterworth filter. This envelope was calculated over the length of each note and so includes temporal information on how the energy within the sound changes over time. Thus this envelope incorporates information regarding the attack time which has been shown to be of high importance to instrument classification [13]. The temporal envelope was then subtracted from the original sound to find the residual. The temporal residual envelope was calculated from the RMS of this residual.

A possible approach to data preparation

2009-07-31T08:48:00.000-07:00

As more of a note to self, there are some interesting aspects of the final year project, "Musical Instrument Detection" by Gautham J. Mysore and Gregory Sell and SongHui Chon, which I may consider:

The ensemble approach to classification:
"...we attempt the problem of identifying the instrumentation of a musical signal at any given time using several machine learning techniques(logistic regression, K-NN, SVM).We approached the problem as a series of separate binary classifications (as opposed to a multivariate problem) so that we could mix and match the best algorithm for each instrument to create the best overall classifier."
The mixture signals were created artificially:
"Then, to create one of the combinations above, a random signal for each instrument were all combined randomly. In this way, we created 52 total signals for each instrumental combination."

HMM toolboxes

2009-07-30T07:56:00.000-07:00

Some toolboxes and links:

Hidden Markov Model Toolbox for Matlab
Mendel HMM Toolbox for Matlab
H2M: A set of Matlab/Octave functions for the estimation of mixtures and hidden markov models

Modelling the temporal dynamics of timbre

2009-07-30T07:38:00.001-07:00

"Musical Instrument Timbres Classification with Spectral Features", Giulio Agostini, Maurizio Longari, Emanuele Pollastri (2001):

A considerable number of features is currently available in the literature, each one describing some aspects of audio content [22, 23]. In the digital domain, features are usually calculated from a window of samples, which is normally very short compared to the total duration of a tone. Thus, we must face the problem of summarizing their temporal evolution into a small set of values. Mean, standard deviation, skewness, and autocorrelation have been the preferred strategies for their simplicity, but more advanced methods like hidden Markov models could be employed, as illustrated in [21, 22]. By combining these time-spanning statistics with the known features, an impressive number of variables can be extracted from each sound. The researcher, though, has to carefully select them in order to both keep the time required for the extraction to a minimum and, more importantly, to prevent from incurring into the so-called curse of dimensionality.

Taken from "Computer Models for Musical Instrument Identification", Nicolas D. Chétry (pg.180):

When modelling timbre, our system and the ones encountered in the literature lose time consideration. In other words, the temporal organisation of the various acoustic events is not represented at the model level. This approach is understandable if one considers timbre as a global attribute of sound. However, we showed in our experiments that, for example, onset and steady-state segments of tones have different characteristics, so that they each contribute to a particular aspect of timbre. Instead of averaging them in one single model, one could think of independently and explicitly modelling them. Similar to speaker recognition, the incorporation of
Dynamic Time Warping (DTW) or Hidden Markov Models (HMM) can constitute a possible orientation for future research.

In a discussion with my supervisor, we noted the importance of considering the temporal dynamics of an instrument sound. As the uniqueness of the spectral envelope cannot be absolutely guaranteed across instruments, extra information about the signal temporal behaviour are often required in order to increase the systems performance. The main difficulty in extracting such temporal features resides in the fact that robust automated pre-processing techniques for onsets or transients detection are difficult to design, especially in the case of pitched musical sounds. As noted by Chetry,

"For this reason, a more general approach is preferred. It consists of appending the delta (speed) and delta-deltas (acceleration) coefficients to the feature vector in order to include information about its evolution with time."

Excellent review of statistical pattern recognition

2009-07-22T10:48:00.000-07:00

As more of a 'note to self', Jain provides an excellent overview (review) of Statistical Pattern Recognition in the paper: "Jain, A.K., Duin, P.W., Mao, J. 2000 - Statistical Pattern Recognition - A Review.pdf" This can be found online here.

Important note from PRTools 4.1 User Manual

2009-07-13T09:02:00.000-07:00

The performance of classification functions can be improved by the following methods:

A reject option in which the objects close to the decision boundary are not classified. They are rejected and might be classified by hand or by another classifier.
The selection or averaging of classifiers.
A multi-stage classifier for combining classification results of several other classifiers.

For all these methods it is profitable or necessary that a classifier yields some distance measure, confidence or posterior probability in addition to the hard, unambiguous assignment of labels.

PCA - tips and tricks

2009-07-10T10:46:00.000-07:00

PCA: tips & tricks

If n < (number of features K), remove directions without data: k <- n-1.
Usually we set the mean of data to a zero vector: x <- (x - μ).
PCA is sensitive to scaling, so it’s best to standardize: xi <- (xi - μi) / standdev
Images and signal classes are usually invariant to scaling; normalize each sample by removing its mean and dividing by its standard deviation.

Which k features are the best to base the decision on?

2009-07-10T09:08:00.000-07:00

A good discussion on feature selection based on ideas published in the paper, "T.M. Cover, The best two independent measurements are not the two best", can be found here. The introduction is as follows:

We have all had the experience of waiting for a friend in a crowded place. The friend is late so, we scan the crowd hoping to recognize the friend from far away. Lets say the friend has short, dark hair, they wear glasses, and often wear a favorite green sweater. Then, consciously or unconsciously, we use what we know about our friend, like the short hair and glasses, to try to pick them out of the crowd. We have all also had a case of mistaken identity. Perhaps a stranger is also wearing a green sweater, or has glasses and short, dark hair, and it is only upon glancing at another of the stranger's features that we notice the difference.

Important note on pattern recognition

2009-07-10T06:35:00.000-07:00

We know how to construct classifiers. Evaluation is crucial to find the best one.

Online statistics textbook

2009-07-07T06:44:00.001-07:00

This is an excellent online resource for learning statistical analysis: The Electronic Statistics Textbook.

Proper citation:

Proper citation: (Electronic Version): StatSoft, Inc. (2007). Electronic Statistics Textbook. Tulsa, OK: StatSoft. WEB: http://www.statsoft.com/textbook/stathome.html.
Proper citation: (Printed Version): Hill, T. & Lewicki, P. (2007). STATISTICS Methods and Applications. StatSoft, Tulsa, OK.

The Cepstrum PDA is flawed!

2009-07-07T04:44:00.000-07:00

I received a correspondence from Arturo Camachoa who wrote a paper showing that the Cepstrum PDA does not work on periodic signals. Details of the paper are as follows:

Comment on “Cepstrum pitch determination” [J. Acoust. Soc. Am. 41, 293–309 (1967)] (L)
Arturo Camachoa, Computational NeuroEngineering Laboratory, University of Florida, Gainesville, Florida 32611
[Received 16 May 2008; revised 25 August 2008; accepted 26 August 2008] "In a paper by A. Michael Noll [J. Acoust. Soc. Am. 41, 293–309 1967, the use of the cepstrum was proposed to determine the pitch of a signal. This paper shows that such a method does not work on periodic signals. © 2008 Acoustical Society of America. ][DOI: 10.1121/1.2988293]

Feature extraction online repository

2009-07-06T08:14:00.000-07:00

I was thinking to myself, "Researchers should pool their collective thought, not solely through the publication of papers and hence ideas, but also provide a platform on which to build (my interpretation being 'the Matlab source code!!?')"

For surely, the publication of universally accepted de facto code-standards would promote further extensions, and not burden/hamper researchers with the task of having to 'code it themselves'. A chore. A duplicative task - a non-advancement in the collective knowledge - not so?

And hence, by divine nature, an answer popped out of the google blue:

http://www.ifs.tuwien.ac.at/mir/

"the creation of large audio collections" - an online repository. Now that's what I call music.

Magic.

Joey.

Thematic papers

2009-07-06T06:53:00.000-07:00

Very relevant to my own work:

Wieczorkowska,A., Kolczynska, E. 2007 - "Quality of musical instrument sound identification for various levels of accompanying sounds"
Analysis and recognition of expression in musical gestures.

Another interesting Matlab toolbox

2009-07-04T13:18:00.000-07:00

PsySound3.

PsySound3 is software for the analysis of sound recordings using physical and psychoacoustical algorithms. It is an easy to use platform that does precise analysis using standard acoustical measurements, as well as implementations of psychoacoustical and musical models (such as loudness, sharpness, roughness, fluctuation strength, pitch, rhythm and running IACC).

I was just looking for pitch detection ideas but there are many 'Audio Analysers' to choose from.

Another toolbox to play with!!

Note segmentation

2009-07-04T13:11:00.001-07:00

I spent quite some time writing code for note segmentation based on onset-detection. I came across this interesting approach by Declan Murphy and Kristoffer Jensen. Don't think I have the time to implement the code myself - unfortunately!!

Pitch tracking

2009-07-04T13:01:00.000-07:00

Juan Pablo writes about 'Autocorrelation Pitch Tracking' in his ISMIR 2000 paper. Seems like the best option for my work. From my research, there are many other approaches, the most popular of which seem to be:

Maximum likelihood
The short-time average magnitude difference function (AMDF)
And many more...

An interesting explanation of two pitch tracking (or fundamental frequency estimation) approaches exists here.

p.s. And there's more:

Bozena Kostek provides a good discussion on Pitch Detection Algorithms (PDAs) in her paper "Musical Instrument Classification and DUET Analysis Employing Music Information Retrieval Techniques."
High accuracy and Octave error immune pitch detection algorithms

And again! A discussion on robust F0 estimation exists in the Auditory forum here.

AMII. New People. New papers.

2009-07-03T10:11:00.000-07:00

I uncovered some researchers in AMII. Both work for the Automatic Indexing of Audio project at The University of North Carolina, Charlotte:

The papers of interest from the 'Automatic Indexing of Audio Project' publications page are the following:

Great online pattern recognition course

2009-07-03T06:44:00.000-07:00

The "Advanced Statistical Pattern Recognition" is a very useful and informative course. It uses PRTools for Matlab software.

Matlab Pattern Recognition Toolboxes

2009-07-03T06:07:00.000-07:00

Just a note on some of the pattern recognition toolboxes I've come across. There are many, but these look like the most interesting and useful:

Listed on the TOOLDIAG site are a collection of resources. The pick of the bunch is the following, "The Pattern Recognition Group, TU Delft." Wow!

Very interesting final year project on instrument ID!

2009-07-01T09:33:00.000-07:00

I came across a very interesting project today by Davyd Madeley of The University of Western Australia. His final year project, "Automatic Computer Classification of Solo Musical Instruments" certainly captures my interest! Davyd interestingly documents the various stages of this project in his 'articles' section:

Of course, all these files are available on this page.

Some ideas from Holbrey's report

2009-06-30T04:04:00.000-07:00

I'm posting some interesting notes from Richard Holbrey's report on dimensionality reduction:

Imola K. Fodor, June 2002 - "A survey of dimension reduction techniques"
Hierarchical Clustering Explorer for Interactive Exploration of Multidimensional Data
Key data analysis functions available in the GeneSpring Platform software (Docs): kNN, Cros gene error model, Multiple testing corrections, One-way ANOVA, Two-way ANOVA, PCA, QT clustering
Clustering analysis
Clustering software
J.J. Verbeek (PhD thesis) 2004 - "Mixture Models for Clustering and Dimension Reduction"
Michael I. Jordan - "The Kernel trick" (Transforming the data can make it linearly separable)
Cross validation

Multi-dimensional data visualisation

2009-06-29T11:39:00.000-07:00

Considering my basic system to date is based on just 9 features, this still means the system is processing data of dimension 48. To be able to visualise this data would be of benefit for several reasons, including my own understanding of the feature data being extracted and for explanatory purposes to the eventual readers of my thesis. I came across an excellent report by Richard Holbrey of Leeds University who describes data dimension reduction in simplicity and offers good insight into available tools for data visualisation. It's well worth the read.

It's been awhile...

2009-06-17T06:30:00.000-07:00

It certainly been awhile since I last posted. I'm currently rebuilding my AMIIS. The AMIIS will be self contained, extensible and 'hopefully' quite impressive. I've the data processing working so far. I got kNNs to work on the data yesterday and trying to get GMMs working today. I'm interfacing with other toolboxes so hence the delay. I do, however, envision a restructuring of the system again to be more inline with the toolboxes I'm using. It all depends on how successful each toolbox fairs at classifying...???

GMM/Bayes toolbox

2009-05-17T16:17:00.000-07:00

I've been having some problems running my GMMs with new data I'm working with. Actually, the data is that which includes the fix I made to the delta-coefficients I referred to in previous posts. I came across this toolbox today which I will try to run tomorrow:

GMMBAYES - Bayesian Classifier and Gaussian Mixture Model ToolBox

This toolbox includes a fix which forces a matrix to be a valid covariance matrix - exactly the problem I've been trying to fix!!

File name: gmmb_covfixer.m
%
% covmatrix = GMMB_COVFIXER(matrix)
% Matrix is forced (complex conjugate) symmetric,
% positive definite and its diagonal real valued.
%

AES Poster presentation went well

2009-05-17T16:10:00.000-07:00

My trip to Munich for the AES 126th Convention went well. I presented my poster in the "Measurements session", which was on Sunday. My poster can be found here.

Three new papers on Instrument ID

2009-04-28T03:25:00.000-07:00

My supervisor Mikel Gainza presented at ICASSP 2009 in Tapei, Taiwan. Thanks Mikel for the heads up on three papers on musical instrument ID:

"Instrument Analysis and Identification of Polyphonic Music using Beat-Synchronous feature integration and fuzzy clustering", Soo-Chang Pei, Nien-Teh Hsu
"Polyphonic Musical Instrument Recognition Based on a Dynamic Model of the Spectral Envelope", Juan José Burred, Axel Röbel, Thomas Sikora
"Interpolating Hidden Markov Model and Its application to Automatic Instrument Recognition", Tuomas Virtanen, Toni Heittola

Very useful maths tutorial website

2009-04-23T08:01:00.000-07:00

In his own words, this maths tutorial site by Paul Dawkins offers a "complete set of free online (and downloadable) notes and/or tutorials for classes" he teaches at Lamar University.

Very useful indeed! Thank you Paul Dawkins.

And not to forget the Linear Algebra course...

2009-04-22T07:31:00.001-07:00

I forgot to mention the Linear Algebra course from MIT. Although it's long and somewhat challenging, the lecturer clearly illustrates some of the fundamentals of linear algebra. The notion of basis vectors has become a lot clearer to me and as the lectures progress, other areas become much more conceptualised. To be honest, these lectures were a chore at the beginning but as the lecturer paints the picture, I can visualise the notions involved. Very interesting and highly recommended. I just have to snigger at the fact that I've had to venture into another domain of mathematics and jot down my experiences in my 'travel' log. AMII is most certainly a journey of the intellect.

MITOpenCourseWare - some excellent material

2009-04-16T07:25:00.000-07:00

Below are some excellent online courses from the MIT Open Courseware. There are many courses available. I've selected a few here which are related to my own research:

The bible on pattern recognition

2009-04-08T16:24:00.000-07:00

I came across an excellent book on pattern recognition: Neural Networks for Pattern Recognition by Christopher M. Bishop. I've actually enjoyed reading this book thus far. While other books on the same topic have been overcomplicated with mathematics, this book simplifies the concepts with good wording alongside the required mathematics. An excellent reference in my opininion. For example, the 'Curse of dimensionality' is explained very well, much more so than any literature I've come across in my research.

MFCCs, delta and delta-delta coefficients exlained very nicely

2009-03-24T11:01:00.000-07:00

Jinjin Ye's Master of Science thesis, "Speech Recognition Using Time Domain Features from Phase Space Reconstructions" explains the calculation of the velocity and acceleration MFCC coefficients very simply and concisely. See section 2.1.2 'Common Features'.

Cepstral analysis, regression and the such in speech recognition

2009-03-24T10:07:00.000-07:00

On continuing on the magical roller coaster ride that is the discovery of the calculation of delta coefficients, I've come up with these papers which add more insight. These are referenced (not clearly though!) in the Manson paper, I previously mentioned:

My explorers hard hat is on. Flash light in hand. Compass at the ready. Into the dark caverns of calculus I immerse...

Use the Usenet!

2009-03-23T12:58:00.000-07:00

I should have thought of this a long time ago.

The usenet groups are a very useful source of information in general. This matlab group should prove very useful indeed.

Delta and Delta-delta coefficients - explained!

2009-03-23T12:13:00.000-07:00

So what are delta and delta-delta coefficients?

To those of us who have read the literature on AMII, it is noticable that the research papers on AMII who have incorporated the use of MFCCs, delta coefficients and delta-delta coefficients, have failed to include any explanation as to how to actually compute the delta and delta-delta coefficients. This was a hurdle I needed to overcome myself, and to my dismay, I've discovered that my own calculations of these coefficients has been erroneous. Information on the maths behind their calculations was not easy to come by. My rigorous searches were not in vain, however. The following paper offers some insight:

- Mason, J.S., Zhang, X. 1991 - Velocity and acceleration features in speaker recognition

From this paper:

The, word 'dynamic' is sometimes used synonymously with first order analysis (features from which are often given the prefix 'delta'), and it should be emphasised that here we adopt the more general usage, encompassing under the term dynamic anything that is 'non-static', ie velocity, acceleration and higher orders.

So what are 'Dynamic features':

Dynamic features can be derived from either temporal differencing or from regression analysis. In the first case the dynamic feature is derived by simply subtracting static features separated by a suitable time span, with an iterative process for higher orders. In the second case a polynomial fit is applied to the static series. Both involve a window moving along the time course, and it is the importance of choosing an appropriate window size which is demonstrated here.

So that explains the use of the window in the code which I used as my reference, this being the implementation provided by Dan Ellis in Rastamat.

Gaël Richard visits the ARG

2009-03-23T11:54:00.000-07:00

Prof. Gaël Richard of Télécom Paris Tech visited the Audio Research Group last Friday March 20^th. He gave a seminar on approaches to Automatic Musical Instrument Recognition in DIT Kevin Street, which had a reasonable attendance.

Considering Prof. Richard is an eminence in the area of audio research, it was a real honour for the ARG to have him visit us.

Again, my appreciation to Prof. Richard for answering my questions relating to AMII and for offering us all some insight into his own research into musical instrument recognition.

Good code repository

2009-03-19T06:04:00.000-07:00

Besides the usual popular DSP resources online, e.g. Mathworks and DSPRelated, I came across this very useful programmer's repository, Programmer's United Development Net (PUDN).

In particular the Speech-Voice recognition/combine section has some matlab source files which implement some of the useful features such as MFCCs and other coefficients.

ps. The English version of PUDN (Speech) is here.

A pot of gold

2009-03-16T10:45:00.000-07:00

I've discovered my own little pot of gold, just before St. Patrick's Day. Roger Jang has translated some Chinese books on Pattern Recognition into English and put them online. These look like excellent resources. Not to forget to mention the toolboxes and other resources on his website. From first glimpse, the interactive tutorials look very educational and may shed some light on areas I've had trouble understanding myself.

Multiclass classification using SVM

2009-03-12T10:44:00.000-07:00

I've been trying to get some learning algorithms implemented on my current data.

I started using SVM (light) and have just realised that the classifier works for just binary classification, i.e. max 2 classes! So, I needed to find a solution, and I didn't have to look too far (Well, at least I hope it provides the solution as I haven't implemented the classifier just yet.) SVM (Multiclass) by the same author of the light version, Thorsten Joachims is an implementation of the multi-class Support Vector Machine (SVM) described in:

On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines, Koby Crammer and Yoram Singer (Journal of Machine Learning Research), 2001.

SVM Multiclass can be found here.
SVM Light can be found here.
Koby Crammer's home site is here.

I crashed the AES submission website!!

2009-03-12T10:40:00.001-07:00

I finally submitted my AES 126 paper, but not without some last minute panic. When uploading my paper, the AES submission website crashed! I contacted the AES the following day and it turned out that my paper along with some Eastern European paper submissions was causing the same problem. I did all I could to assist, supplying code, software versions, logs etc. and the problem was resolved. Apparently, some 'illegal' character was causing the problem. That's about all I know. Anyway, my first paper has finally gone through and as the man says, it's like getting your first tattoo, you just want to get more...or something to that effect.

Pulling an all-nighter to get paper for AES completed

2009-02-26T21:27:00.000-08:00

Nearly finished the first draft of my paper for the AES 126th Convention in Munich. It's looking good. Nearly 8 pages in total - just within the limits I believe. Maybe my supervisors will trim it down, short back and sides, and leave me to brush up over the weekend. Deadline is this coming Tuesday March 3rd. My first paper...

Did Dudley Ryder have it right 294 years ago?

2009-02-26T04:03:00.000-08:00

I recently read an article about diary keeping and found the comment by GoloshtheFox on the same page, very interesting:

"Dudley Ryder, a prolific eighteenth-century London diarist, had the following to say on the matter of diaries 294 years ago:

‘It is certainly very useful to accustom ones self to write down ones thoughts. It both brings one to a good style and readiness of words and also helps very much to disgest our thoughts and bring them into a method and fastens them in the memory. - (27 August 1715)."

Not everybody is the same, but I can certainly see the pattern unfolding in blogs and social networking sites such as Facebook. They are in some way a keep-note on aspects of each of our lives. We post our pictures, post our notes, post our blogs, keep track of the activities of our friends etc.

These are a few thoughts I've tried to digest and liberate through the wonderful medium of the internet. Keeping this blog, has certainly helped me in my work. Now back to the grind :-)

The GUI has arrived!

2009-02-26T03:09:00.000-08:00

So, I've spent the past two days (and a bit) learning about Matlab GUI and developing a simple (but useful) GUI application for the onset detection and note separation. I'm pretty happy with the results, bearing in mind this is still the alpha version and it needs to be tried and tested.

Here are a few graphics:

Time plot for Double Bass E1B1

Post-processed onset plot for Double Bass E1B1 with cutoff boundaries for the extracted notes:

I still need to add parameter options for smoothing and threshold. These parameters are currently set to constant in the application.

Wow, onsets are very interesting

2009-02-24T03:05:00.000-08:00

My algorithm is working a treat so far. My previous post regarding the onset characteristics of the Double Bass was touching on an important attribute of music instruments. As I go along through the instrument families, the onset characteristics of the instruments are plain to see.

For example, taking the IOWA sample for the Viola:

Note the sharp onset for the Viola. Of course, the importance of onsets in musical instrument recognition has been well documented in the literature so this is no great discovery. However, it is very interesting when one sees the characteristics for themselves through their own work.

Some papers which have studied onset detection and musical instrument recognition:

Kitahara et al 2006 Instrogram - A New Musical Instrument Recognition Technique Without Using Onset Detection NOR F0 Estimation
Bello, J.P. et al. 2004 - On the use of phase and energy for musical onset detection in the complex domain
Bello, J.P. et al. 2005 - A Tutorial on Onset Detection in Music Signals
Dixon, S. 2006 - Onset Detection Revisited
Duxbury, C., Bello, J.P., Davies, M., Sandler, M. 2003 - A combined phase and amplitude based approach to onset detecion for audio segmentation
Duxbury, C., Bello, J.P., Davies, M., Sandler, M. 2003 - Complex Domain Onset Detection for Musical Signals
Lacoste, A., Douglas, E. 2006 - Machine Learning for Note Onset Detection
Lacoste, A., Eck, D. 2005 - A Supervised Classification Algorithm for Note Onset Detection
Leveau, P. ISMIR 2004 - Methodology and tools for the evaluation of automatic onset detection algorithms in music

Onset detection - it's tricky business

2009-02-23T11:23:00.000-08:00

I ran into some complications with extracting some notes. The instrument I was looking at was the Double Bass. The IOWA file can be found here.

Here's a demonstration of the problem encountered.

Taking a closer look at note 'G' :

The spurious peaks detected in the audio sample proved problematic for my algorithm. I suspect that this being an 'arco' sample, the problem could be due to vibrato?? Anyway, my work around involved adding more thresholds. I noticed that there tend to be short periods of non-peak activity in the actual notes themselves. This was causing some confusion. To overcome this I added a min-gap time threshold so these gaps would not interfere. It took a while! So to continue and no doubt, there will be other issues. I'm still to get started on the front end...!

Iowa notes separated successfully

2009-02-20T10:09:00.000-08:00

I've been working on some onset detection algorithms and trying to separate the IOWA music samples so I can build up a database of instrument notes. The code I use is based on an adaptation of Mikel Gainza's code...which in turn is based on the following papers:

Duxbury, C. et al. - Complex Domain Onset Detection for Musical Signals
Bello, J.P. et al. - On the use of phase and energy for musical onset detection in the complex domain

I got the algorithms working in MATLAB today. Here's a run through of a simple example:

Music sample: Tuba, file: Tuba.pp.C1B1.aiff

Processed using complex onset detection:

Time plot

Onsets detected

Smoothed onsets

Smoothed onsets with applied threshold

Detected peaks for note C1

Resultant note wav files:
All the resultant .wav files from this example can be found here.

I'm going to add a front end in Matlab so the various parameters can be manipulated to suit the wav file under investigation. This should prove a useful tool for many once it is completed.

Better than Real at Lightwave

2009-01-29T03:30:00.000-08:00

I went to Beau lotto's presentation "Better than Real" yesterday at the Lightwave Exhibition at the Science Gallery.

It was interesting...to a degree. In a nutshell, Dr. Lotto discussed how human's perceive colour, and how perception is dependent, on context and, our brain's historical experience making meaning out of present context. lottolab's latest work is based on the notion of sensory substitution - replacing one sense with another. The example provided is how blind people can perceive their world through their ears, i.e. hearing your visual world. Beau provides the example from nature of how Dolphin's use sonar for navigation.

A thought that came to mind was that I'd come across previous work in this area. The work at vOICe, "Vision technology for the totally blind" is a fascinating project and well worth a read. Also, the book "The Blind Watchmaker" by Richard Dawkins has some extremely interesting writings on the evolution of bats and 'echolocation'.

WOW - the wonders of modern tech

2009-01-27T10:44:00.000-08:00

Can somebody please tell me how I'm gonna identify this musical instrument?:
http://www.youtube.com/watch?v=kfrONZjakRY

The Ocarina! More details here:
http://www.dancemusicblog.com/software/first-iphone-musical-instrument-the-ocarina/#more-183

I do a bit of DJing myself. I came across this amazing new digital controller; the EKS otus. Looks simply amazing. Controls two decks in one, via a select switch. I've been looking through some forums, for instance, Native Instruments. Mixed responses from the Tractor user community. All in all though, on a personal level, I'd seriously consider purchasing one of these given the amount of storage required for my current setup and my interest in digital music. I can see myself veering down the road of the laptop, digital DJ and this offers a nice solution. Just have to get the finances together to purchase a new MAC!!

Came across a good blog on Dance music too: dancemusicblog.com

Some interesting Pattern Recognition toolboxes...

2009-01-27T09:03:00.000-08:00

Just a short note on some interesting Pattern Recognition toolboxes I've come across recently:

DD_Tools
"...data description toolbox wants to provide tools, classifiers and evaluation functions for the research of one-class classification (or data description)."

The toolbox is an extension of the PRTools toolbox.

PRSD Studio
I've applied for a trial licence with prsysdesign. This looks like a very interesting Matlab toolbox.

Paper accepted for 126 AES Convention!

2009-01-22T08:15:00.000-08:00

My paper entitled "Evaluating Ground Truth for ADRess as a Preprocess for Automatic Musical Instrument Identification" has been accepted.

Happy Days.

So I'm off to the 126th AES Convention in Munich second week in May.

Understanding the Fundamentals of Music

2009-01-21T07:14:00.000-08:00

I came across a fantastic course entitled "Understanding the Fundamentals of Music" by Robert Greenberg.

I've listened to the first 4 lectures. I must say that Robert Greenberg gets his ideas across very well. The focus has been on Western Orchestral music, and offers excellent insight into many of the instruments I've encountered on my own research.

On a personal note; an example provided by Greenberg in the discussion on 'Dynamics' surprised me greatly. Does Ludwig van Beethoven's Piano Sonata no. 8 in C Minor, op. 13 (Pathétique, 1798) not sound uncannily like Linda Ronstadt & James Ingram's, "Some Where Out There"? You can't believe how disappointed I was (am) by this blatant rip-off. I loved this song as a child.

A short list of some of the books which I 'hope' to read shortly:

The onset of discoveries

2009-01-08T07:47:00.000-08:00

I came across some excellent resources for onset detection - I will add to this list (from an offline compilation) in time but for now, this site has proved valuable:

http://old.lam.jussieu.fr/src/Membres/Leveau/SOL/SOL.htm

This in conjunction with the work of Mikel Gainza (within the ARG - thanks for the code) and Juan Pablo Bello (http://homepages.nyu.edu/~jb2843/Home.html), should prove a worthy springboard.

It's never too late...

2009-01-07T08:24:00.000-08:00

They say it's never too late. With 11 months remaining, today is a momentous day. For today, I plant my well seasoned PhD tree into the digital soil of inter-web land. I step back to admire the many branches which have grown, some bare, brown and rotten to the core, others with blossoming leaves, providing nourishment to crawling ideas (insects!) which have in-turn multiplied and evolved. All that remains, is for the roots to settle in their new found plot (blog!) as I hope they come to terms with a more stable foundation having been quite unsettled to-date with numerous homes (including an old notepad, which to my dismay took a run and jump and still remains 'somewhere out there', or lost in the oblivion).

So where am I in terms of my research?

Onset detection. Yes. Indeed. Why? I need to segment all the notes from the IOWA samples. So where's my horse? We must ride forth, and conquer without any further adieu, remembering to pause at energy bursts, consider their authenticity as an onset, validate, take note, and ride the sustain...

By the way: AMII = Automatic Musical Instrument Identification (in case you thought it was AM I Insane)