A listing of some new relevant publications

A listing of some recent publications (well most are recent) relevant to my own research area:

• Brown, J.C. and P. Smaragdis (2009).``Hidden Markov and Gaussian mixture models for automatic call classification" J.Acoust. Soc. Am. 125, EL221-224.


• Reconnaissance des instruments dans la musique polyphonique par décomposition NMF et classification SVM, Alexey Ozerov, Slim Essid and Maurice Charbit


• Incorporating prior knowledge on the digital media creation process into audio classifiers, M. Lardeur, S. Essid, G. Richard


• The Significance of the Non-Harmonic “Noise” Versus the Harmonic Series for Musical Instrument Recognition, Livshin, A., Rodet, X., 2006. ISMIR 2006


• Livshin, A., Rodet, X. 2009 - Purging Musical Instrument Sample Databases Using Automatic Musical Instrument Recognition Methods IEEE Transactions on Audio, Speech and Language Processing, Vol 17, no. 5. pp. 1046-1051, July 2009


• Martins, L.G. (PhD Thesis) 2009 - Livshin, A., Rodet, X. 2009 - Purging Musical Instrument Sample Databases Using Automatic Musical Instrument Recognition Methods


• Jiang, W., Zhang, X., Ras, Z.W., and Cohen, A. (2009). Multiple Classifiers for Different Features in Timbre Estimation, in Advances in Intelligent Information Systems (Eds. Ras, Z.W. and Tsay, L.), Studies in Computational Intelligence, Springer, will appear


• Soo-Chang Pei, Nien-Teh Hsu 2009 - Instrumentation analysis and identification of polyphonic music using beat-synchronous feature integration and fuzzy clustering

ISMIR 2009, Session Title: Musical Instrument Recognition & Multipitch Detection

• Fuhrmann, F., Haro, M., Herrera, P. (2009). Scalability, generality and temporal aspects in automatic recognition of predominant musical instruments in polyphonic music. Conference of the International Society for Music Information Retrieval (ISMIR)


• Musical Instrument Recognition in Polyphonic Audio Using Source-Filter Model for Sound Separation, Toni Heittola, Anssi Klapuri and Tuomas Virtanen (ISMIR)

ISMIR 2009 Poster Session (PS3):

• Automatic Identification of Instrument Classes in Polyphonic and Poly-Instrument Audio, Philippe Hamel, Sean Wood and Douglas Eck

Perceptions based on prior knowledge?

I found a very interesting journal article on how our perceptions are based on prior experience. It ties in with the perception of sounds.

Some more papers on instrument ID

Three papers my supervisor has brought to my attention:

• "Instrumentation analysis and identification of polyphonic music using beat-synchronous feature integration and fuzzy clustering", by Soo-Chang Pei and Nien-Teh Hsu
  
• "Polyphonic Musical Instrument Recognition Based on a Dynamic Model of the Spectral Envelope", J.J. Burred, A. Röbel and T. Sikora
• "Musical Instrument Recognition in Polyphonic Audio Using Source-Filter Model for Sound Separation", T. Heittola, A. Klapuri, and T. Virtanen

IPEM and the psychology of electronic music

I'm working with the IPEM Toolbox. A difficult task given that it's tied to Matlab versions 5.3.1 and 6.0. The source code is available and can be toyed with to try to make it compatible with later versions. A fix is to find the original versions, more as a time saving exercise! But as an aside, the concept of IPEM and the extraction of perceptual features is of interest to my own research. And in particular, being an electronic music fan, this article discusses the "Psychology of Electronic Music".

4475 separted notes

I separated the notes from the majority of the IOWA musical instrument samples. Total notes = 4475. WOW! It wasn't easy! I'll be writing about how I did it in my next publication. Look out a few short lines describing about 9 day's of hard work!!

Another Instrument ID paper

I came across another paper on musical instrument ID today:

Loughran,R., Walker,J., O'Neill, M., O'Farrell, M., 2008 - Musical Instrument Identification Using Principal Component Analysis and Multi-Layered Perceptrons.pdf

Loughran, R. uses the temporal envelope of the signal residual (post removal of the RMS temporal envelope) as a feature which I haven't come across before in the literature.

"Temporal and spectral envelopes. The temporal envelope was found by calculating the RMS energy envelope of each sound, which was then filtered using a 3rd order low pass Butterworth filter. This envelope was calculated over the length of each note and so includes temporal information on how the energy within the sound changes over time. Thus this envelope incorporates information regarding the attack time which has been shown to be of high importance to instrument classification [13]. The temporal envelope was then subtracted from the original sound to find the residual. The temporal residual envelope was calculated from the RMS of this residual.

A possible approach to data preparation

As more of a note to self, there are some interesting aspects of the final year project, "Musical Instrument Detection" by Gautham J. Mysore and Gregory Sell and SongHui Chon, which I may consider:

1. The ensemble approach to classification:
   "...we attempt the problem of identifying the instrumentation of a musical signal at any given time using several machine learning techniques(logistic regression, K-NN, SVM).We approached the problem as a series of separate binary classifications (as opposed to a multivariate problem) so that we could mix and match the best algorithm for each instrument to create the best overall classifier."
2. The mixture signals were created artificially:
   "Then, to create one of the combinations above, a random signal for each instrument were all combined randomly. In this way, we created 52 total signals for each instrumental combination."

HMM toolboxes

Some toolboxes and links:

Hidden Markov Model Toolbox for Matlab
Mendel HMM Toolbox for Matlab
H2M: A set of Matlab/Octave functions for the estimation of mixtures and hidden markov models

Modelling the temporal dynamics of timbre

"Musical Instrument Timbres Classification with Spectral Features", Giulio Agostini, Maurizio Longari, Emanuele Pollastri (2001):

A considerable number of features is currently available in the literature, each one describing some aspects of audio content [22, 23]. In the digital domain, features are usually calculated from a window of samples, which is normally very short compared to the total duration of a tone. Thus, we must face the problem of summarizing their temporal evolution into a small set of values. Mean, standard deviation, skewness, and autocorrelation have been the preferred strategies for their simplicity, but more advanced methods like hidden Markov models could be employed, as illustrated in [21, 22]. By combining these time-spanning statistics with the known features, an impressive number of variables can be extracted from each sound. The researcher, though, has to carefully select them in order to both keep the time required for the extraction to a minimum and, more importantly, to prevent from incurring into the so-called curse of dimensionality.

Taken from "Computer Models for Musical Instrument Identification", Nicolas D. Chétry (pg.180):

When modelling timbre, our system and the ones encountered in the literature lose time consideration. In other words, the temporal organisation of the various acoustic events is not represented at the model level. This approach is understandable if one considers timbre as a global attribute of sound. However, we showed in our experiments that, for example, onset and steady-state segments of tones have different characteristics, so that they each contribute to a particular aspect of timbre. Instead of averaging them in one single model, one could think of independently and explicitly modelling them. Similar to speaker recognition, the incorporation of
Dynamic Time Warping (DTW) or Hidden Markov Models (HMM) can constitute a possible orientation for future research.

In a discussion with my supervisor, we noted the importance of considering the temporal dynamics of an instrument sound. As the uniqueness of the spectral envelope cannot be absolutely guaranteed across instruments, extra information about the signal temporal behaviour are often required in order to increase the systems performance. The main difficulty in extracting such temporal features resides in the fact that robust automated pre-processing techniques for onsets or transients detection are difficult to design, especially in the case of pitched musical sounds. As noted by Chetry,

"For this reason, a more general approach is preferred. It consists of appending the delta (speed) and delta-deltas (acceleration) coefficients to the feature vector in order to include information about its evolution with time."

Excellent review of statistical pattern recognition

As more of a 'note to self', Jain provides an excellent overview (review) of Statistical Pattern Recognition in the paper: "Jain, A.K., Duin, P.W., Mao, J. 2000 - Statistical Pattern Recognition - A Review.pdf" This can be found online here.

Important note from PRTools 4.1 User Manual

The performance of classification functions can be improved by the following methods:

1. A reject option in which the objects close to the decision boundary are not classified. They are rejected and might be classified by hand or by another classifier.
2. The selection or averaging of classifiers.
3. A multi-stage classifier for combining classification results of several other classifiers.

For all these methods it is profitable or necessary that a classifier yields some distance measure, confidence or posterior probability in addition to the hard, unambiguous assignment of labels.

PCA - tips and tricks

PCA: tips & tricks

• If n < (number of features K), remove directions without data: k <- n-1.
• Usually we set the mean of data to a zero vector: x <- (x - μ).
• PCA is sensitive to scaling, so it’s best to standardize: xi <- (xi - μi) / standdev
• Images and signal classes are usually invariant to scaling; normalize each sample by removing its mean and dividing by its standard deviation.

Which k features are the best to base the decision on?

A good discussion on feature selection based on ideas published in the paper, "T.M. Cover, The best two independent measurements are not the two best", can be found here. The introduction is as follows:

We have all had the experience of waiting for a friend in a crowded place. The friend is late so, we scan the crowd hoping to recognize the friend from far away. Lets say the friend has short, dark hair, they wear glasses, and often wear a favorite green sweater. Then, consciously or unconsciously, we use what we know about our friend, like the short hair and glasses, to try to pick them out of the crowd. We have all also had a case of mistaken identity. Perhaps a stranger is also wearing a green sweater, or has glasses and short, dark hair, and it is only upon glancing at another of the stranger's features that we notice the difference.

Important note on pattern recognition

We know how to construct classifiers. Evaluation is crucial to find the best one.

Online statistics textbook

This is an excellent online resource for learning statistical analysis: The Electronic Statistics Textbook.

Proper citation:

Proper citation: (Electronic Version): StatSoft, Inc. (2007). Electronic Statistics Textbook. Tulsa, OK: StatSoft. WEB: http://www.statsoft.com/textbook/stathome.html.
Proper citation: (Printed Version): Hill, T. & Lewicki, P. (2007). STATISTICS Methods and Applications. StatSoft, Tulsa, OK.

The Cepstrum PDA is flawed!

I received a correspondence from Arturo Camachoa who wrote a paper showing that the Cepstrum PDA does not work on periodic signals. Details of the paper are as follows:

Comment on “Cepstrum pitch determination” [J. Acoust. Soc. Am. 41, 293–309 (1967)] (L)
Arturo Camachoa, Computational NeuroEngineering Laboratory, University of Florida, Gainesville, Florida 32611
[Received 16 May 2008; revised 25 August 2008; accepted 26 August 2008] "In a paper by A. Michael Noll [J. Acoust. Soc. Am. 41, 293–309
1967, the use of the cepstrum was proposed to determine the pitch of a signal. This paper shows that such a method does not work on periodic signals. © 2008 Acoustical Society of America. ][DOI: 10.1121/1.2988293]

Feature extraction online repository

I was thinking to myself, "Researchers should pool their collective thought, not solely through the publication of papers and hence ideas, but also provide a platform on which to build (my interpretation being 'the Matlab source code!!?')"

For surely, the publication of universally accepted de facto code-standards would promote further extensions, and not burden/hamper researchers with the task of having to 'code it themselves'. A chore. A duplicative task - a non-advancement in the collective knowledge - not so?

And hence, by divine nature, an answer popped out of the google blue:

http://www.ifs.tuwien.ac.at/mir/

"the creation of large audio collections" - an online repository. Now that's what I call music.

Magic.

Joey.

Thematic papers

Very relevant to my own work:

• Wieczorkowska,A., Kolczynska, E. 2007 - "Quality of musical instrument sound identification for various levels of accompanying sounds"
• Analysis and recognition of expression in musical gestures.
  

Another interesting Matlab toolbox

PsySound3.

PsySound3 is software for the analysis of sound recordings using physical and psychoacoustical algorithms. It is an easy to use platform that does precise analysis using standard acoustical measurements, as well as implementations of psychoacoustical and musical models (such as loudness, sharpness, roughness, fluctuation strength, pitch, rhythm and running IACC).

I was just looking for pitch detection ideas but there are many 'Audio Analysers' to choose from.

Another toolbox to play with!!

Note segmentation

I spent quite some time writing code for note segmentation based on onset-detection. I came across this interesting approach by Declan Murphy and Kristoffer Jensen. Don't think I have the time to implement the code myself - unfortunately!!

Pitch tracking

Juan Pablo writes about 'Autocorrelation Pitch Tracking' in his ISMIR 2000 paper. Seems like the best option for my work. From my research, there are many other approaches, the most popular of which seem to be:

• Maximum likelihood
• The short-time average magnitude difference function (AMDF)
• And many more...

An interesting explanation of two pitch tracking (or fundamental frequency estimation) approaches exists here.

p.s. And there's more:

• Bozena Kostek provides a good discussion on Pitch Detection Algorithms (PDAs) in her paper "Musical Instrument Classification and DUET Analysis Employing Music Information Retrieval Techniques."
• High accuracy and Octave error immune pitch detection algorithms

And again! A discussion on robust F0 estimation exists in the Auditory forum here.

AMII. New People. New papers.

I uncovered some researchers in AMII. Both work for the Automatic Indexing of Audio project at The University of North Carolina, Charlotte:

• Amanda Cohen Mostafavi
• Wenxin Jiang

The papers of interest from the 'Automatic Indexing of Audio Project' publications page are the following:

• Multiple classifiers for different features in timbre estimation
• Music Instrument Estimation in Polyphonic Sound Based on Short-Term Spectrum Match
• Cascade Classifiers for Hierarchical Decision Systems
• Training of Classifiers for the Recognition of a Musical Instrument Dominating in the Same-Pitch Mix
• Identification of Dominating Instrument in Mixes of Sounds of the Same Pitch
• Quality of Musical Instrument Sound Identification for Various Levels of Accompanying Sounds

Great online pattern recognition course

The "Advanced Statistical Pattern Recognition" is a very useful and informative course. It uses PRTools for Matlab software.

Matlab Pattern Recognition Toolboxes

Just a note on some of the pattern recognition toolboxes I've come across. There are many, but these look like the most interesting and useful:

• Statistical Pattern Recognition Toolbox for Matlab
• PRTools: The Matlab Toolbox for Pattern Recognition
• TOOLDIAG: Pattern recognition toolbox
• MatlabArsenal
• Data Clustering and Pattern Recognition Toolbox
  

Listed on the TOOLDIAG site are a collection of resources. The pick of the bunch is the following, "The Pattern Recognition Group, TU Delft." Wow!

Very interesting final year project on instrument ID!

I came across a very interesting project today by Davyd Madeley of The University of Western Australia. His final year project, "Automatic Computer Classification of Solo Musical Instruments" certainly captures my interest! Davyd interestingly documents the various stages of this project in his 'articles' section:

• Thesis Proposal
• Lightning talk
• Project presentation (Interim)
• Final year project seminar
• Final Year Project dissertation

Of course, all these files are available on this page.

Some ideas from Holbrey's report

I'm posting some interesting notes from Richard Holbrey's report on dimensionality reduction:

• Imola K. Fodor, June 2002 - "A survey of dimension reduction techniques"
• Hierarchical Clustering Explorer for Interactive Exploration of Multidimensional Data
• Key data analysis functions available in the GeneSpring Platform software (Docs): kNN, Cros gene error model, Multiple testing corrections, One-way ANOVA, Two-way ANOVA, PCA, QT clustering
  
• Clustering analysis
• Clustering software
• J.J. Verbeek (PhD thesis) 2004 - "Mixture Models for Clustering and Dimension Reduction"
• Michael I. Jordan - "The Kernel trick" (Transforming the data can make it linearly separable)
  
• Cross validation

Multi-dimensional data visualisation

Considering my basic system to date is based on just 9 features, this still means the system is processing data of dimension 48. To be able to visualise this data would be of benefit for several reasons, including my own understanding of the feature data being extracted and for explanatory purposes to the eventual readers of my thesis. I came across an excellent report by Richard Holbrey of Leeds University who describes data dimension reduction in simplicity and offers good insight into available tools for data visualisation. It's well worth the read.

It's been awhile...

It certainly been awhile since I last posted. I'm currently rebuilding my AMIIS. The AMIIS will be self contained, extensible and 'hopefully' quite impressive. I've the data processing working so far. I got kNNs to work on the data yesterday and trying to get GMMs working today. I'm interfacing with other toolboxes so hence the delay. I do, however, envision a restructuring of the system again to be more inline with the toolboxes I'm using. It all depends on how successful each toolbox fairs at classifying...???

GMM/Bayes toolbox

I've been having some problems running my GMMs with new data I'm working with. Actually, the data is that which includes the fix I made to the delta-coefficients I referred to in previous posts. I came across this toolbox today which I will try to run tomorrow:

GMMBAYES - Bayesian Classifier and Gaussian Mixture Model ToolBox

This toolbox includes a fix which forces a matrix to be a valid covariance matrix - exactly the problem I've been trying to fix!!

File name: gmmb_covfixer.m
%
% covmatrix = GMMB_COVFIXER(matrix)
% Matrix is forced (complex conjugate) symmetric,
% positive definite and its diagonal real valued.
%

AES Poster presentation went well

My trip to Munich for the AES 126th Convention went well. I presented my poster in the "Measurements session", which was on Sunday. My poster can be found here.

Three new papers on Instrument ID

My supervisor Mikel Gainza presented at ICASSP 2009 in Tapei, Taiwan. Thanks Mikel for the heads up on three papers on musical instrument ID:

• "Instrument Analysis and Identification of Polyphonic Music using Beat-Synchronous feature integration and fuzzy clustering", Soo-Chang Pei, Nien-Teh Hsu
• "Polyphonic Musical Instrument Recognition Based on a Dynamic Model of the Spectral Envelope", Juan José Burred, Axel Röbel, Thomas Sikora
• "Interpolating Hidden Markov Model and Its application to Automatic Instrument Recognition", Tuomas Virtanen, Toni Heittola

Very useful maths tutorial website

In his own words, this maths tutorial site by Paul Dawkins offers a "complete set of free online (and downloadable) notes and/or tutorials for classes" he teaches at Lamar University.

Very useful indeed! Thank you Paul Dawkins.

And not to forget the Linear Algebra course...

I forgot to mention the Linear Algebra course from MIT. Although it's long and somewhat challenging, the lecturer clearly illustrates some of the fundamentals of linear algebra. The notion of basis vectors has become a lot clearer to me and as the lectures progress, other areas become much more conceptualised. To be honest, these lectures were a chore at the beginning but as the lecturer paints the picture, I can visualise the notions involved. Very interesting and highly recommended. I just have to snigger at the fact that I've had to venture into another domain of mathematics and jot down my experiences in my 'travel' log. AMII is most certainly a journey of the intellect.

MITOpenCourseWare - some excellent material

Below are some excellent online courses from the MIT Open Courseware. There are many courses available. I've selected a few here which are related to my own research:

• Introduction to Java
• Machine Learning
• Discrete-Time Signal Processing
• Automatic Speech Recognition
• Acoustics of Speech and Hearing
• Brain Mechanisms for Hearing and Speech
• Neural Coding and Perception of Sound
• Pattern Recognition for Machine Learning
• Introduction to Neural Networks

The bible on pattern recognition

I came across an excellent book on pattern recognition: Neural Networks for Pattern Recognition by Christopher M. Bishop. I've actually enjoyed reading this book thus far. While other books on the same topic have been overcomplicated with mathematics, this book simplifies the concepts with good wording alongside the required mathematics. An excellent reference in my opininion. For example, the 'Curse of dimensionality' is explained very well, much more so than any literature I've come across in my research.

MFCCs, delta and delta-delta coefficients exlained very nicely

Jinjin Ye's Master of Science thesis, "Speech Recognition Using Time Domain Features from Phase Space Reconstructions" explains the calculation of the velocity and acceleration MFCC coefficients very simply and concisely. See section 2.1.2 'Common Features'.

Cepstral analysis, regression and the such in speech recognition

On continuing on the magical roller coaster ride that is the discovery of the calculation of delta coefficients, I've come up with these papers which add more insight. These are referenced (not clearly though!) in the Manson paper, I previously mentioned:

• Furui, S. 1981 - Cepstral analysis technique for automatic speaker verification
• Applebaum,T.H., Hanson,B.A. 1991 - Regression features for recognition of speech in quiet and in noise
• Soong, F.K, and Rosenberg, A.E. 1988 - On the use of instantaneous and transitional spectral information in speaker recognition

My explorers hard hat is on. Flash light in hand. Compass at the ready. Into the dark caverns of calculus I immerse...

Use the Usenet!

I should have thought of this a long time ago.

The usenet groups are a very useful source of information in general. This matlab group should prove very useful indeed.

Delta and Delta-delta coefficients - explained!

So what are delta and delta-delta coefficients?

To those of us who have read the literature on AMII, it is noticable that the research papers on AMII who have incorporated the use of MFCCs, delta coefficients and delta-delta coefficients, have failed to include any explanation as to how to actually compute the delta and delta-delta coefficients. This was a hurdle I needed to overcome myself, and to my dismay, I've discovered that my own calculations of these coefficients has been erroneous. Information on the maths behind their calculations was not easy to come by. My rigorous searches were not in vain, however. The following paper offers some insight:

- Mason, J.S., Zhang, X. 1991 - Velocity and acceleration features in speaker recognition

From this paper:

The, word 'dynamic' is sometimes used synonymously with first order analysis (features from which are often given the prefix 'delta'), and it should be emphasised that here we adopt the more general usage, encompassing under the term dynamic anything that is 'non-static', ie velocity, acceleration and higher orders.

So what are 'Dynamic features':

Dynamic features can be derived from either temporal differencing or from regression analysis. In the first case the dynamic feature is derived by simply subtracting static features separated by a suitable time span, with an iterative process for higher orders. In the second case a polynomial fit is applied to the static series. Both involve a window moving along the time course, and it is the importance of choosing an appropriate window size which is demonstrated here.

So that explains the use of the window in the code which I used as my reference, this being the implementation provided by Dan Ellis in Rastamat.

Gaël Richard visits the ARG

Prof. Gaël Richard of Télécom Paris Tech visited the Audio Research Group last Friday March 20^th. He gave a seminar on approaches to Automatic Musical Instrument Recognition in DIT Kevin Street, which had a reasonable attendance.

Considering Prof. Richard is an eminence in the area of audio research, it was a real honour for the ARG to have him visit us.

Again, my appreciation to Prof. Richard for answering my questions relating to AMII and for offering us all some insight into his own research into musical instrument recognition.

Good code repository

Besides the usual popular DSP resources online, e.g. Mathworks and DSPRelated, I came across this very useful programmer's repository, Programmer's United Development Net (PUDN).

In particular the Speech-Voice recognition/combine section has some matlab source files which implement some of the useful features such as MFCCs and other coefficients.

ps. The English version of PUDN (Speech) is here.

A pot of gold

I've discovered my own little pot of gold, just before St. Patrick's Day. Roger Jang has translated some Chinese books on Pattern Recognition into English and put them online. These look like excellent resources. Not to forget to mention the toolboxes and other resources on his website. From first glimpse, the interactive tutorials look very educational and may shed some light on areas I've had trouble understanding myself.

Multiclass classification using SVM

I've been trying to get some learning algorithms implemented on my current data.

I started using SVM (light) and have just realised that the classifier works for just binary classification, i.e. max 2 classes! So, I needed to find a solution, and I didn't have to look too far (Well, at least I hope it provides the solution as I haven't implemented the classifier just yet.) SVM (Multiclass) by the same author of the light version, Thorsten Joachims is an implementation of the multi-class Support Vector Machine (SVM) described in:

On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines, Koby Crammer and Yoram Singer (Journal of Machine Learning Research), 2001.

SVM Multiclass can be found here.
SVM Light can be found here.
Koby Crammer's home site is here.

I crashed the AES submission website!!

I finally submitted my AES 126 paper, but not without some last minute panic. When uploading my paper, the AES submission website crashed! I contacted the AES the following day and it turned out that my paper along with some Eastern European paper submissions was causing the same problem. I did all I could to assist, supplying code, software versions, logs etc. and the problem was resolved. Apparently, some 'illegal' character was causing the problem. That's about all I know. Anyway, my first paper has finally gone through and as the man says, it's like getting your first tattoo, you just want to get more...or something to that effect.

Pulling an all-nighter to get paper for AES completed

Nearly finished the first draft of my paper for the AES 126th Convention in Munich. It's looking good. Nearly 8 pages in total - just within the limits I believe. Maybe my supervisors will trim it down, short back and sides, and leave me to brush up over the weekend. Deadline is this coming Tuesday March 3rd. My first paper...

Did Dudley Ryder have it right 294 years ago?

I recently read an article about diary keeping and found the comment by GoloshtheFox on the same page, very interesting:

"Dudley Ryder, a prolific eighteenth-century London diarist, had the following to say on the matter of diaries 294 years ago:

‘It is certainly very useful to accustom ones self to write down ones thoughts. It both brings one to a good style and readiness of words and also helps very much to disgest our thoughts and bring them into a method and fastens them in the memory. - (27 August 1715)."

Not everybody is the same, but I can certainly see the pattern unfolding in blogs and social networking sites such as Facebook. They are in some way a keep-note on aspects of each of our lives. We post our pictures, post our notes, post our blogs, keep track of the activities of our friends etc.

These are a few thoughts I've tried to digest and liberate through the wonderful medium of the internet. Keeping this blog, has certainly helped me in my work. Now back to the grind :-)

The GUI has arrived!

So, I've spent the past two days (and a bit) learning about Matlab GUI and developing a simple (but useful) GUI application for the onset detection and note separation. I'm pretty happy with the results, bearing in mind this is still the alpha version and it needs to be tried and tested.

Here are a few graphics:

Time plot for Double Bass E1B1



Post-processed onset plot for Double Bass E1B1 with cutoff boundaries for the extracted notes:



I still need to add parameter options for smoothing and threshold. These parameters are currently set to constant in the application.

Wow, onsets are very interesting

My algorithm is working a treat so far. My previous post regarding the onset characteristics of the Double Bass was touching on an important attribute of music instruments. As I go along through the instrument families, the onset characteristics of the instruments are plain to see.

For example, taking the IOWA sample for the Viola:



Note the sharp onset for the Viola. Of course, the importance of onsets in musical instrument recognition has been well documented in the literature so this is no great discovery. However, it is very interesting when one sees the characteristics for themselves through their own work.

Some papers which have studied onset detection and musical instrument recognition:

• Kitahara et al 2006 Instrogram - A New Musical Instrument Recognition Technique Without Using Onset Detection NOR F0 Estimation
• Bello, J.P. et al. 2004 - On the use of phase and energy for musical onset detection in the complex domain
• Bello, J.P. et al. 2005 - A Tutorial on Onset Detection in Music Signals
• Dixon, S. 2006 - Onset Detection Revisited
• Duxbury, C., Bello, J.P., Davies, M., Sandler, M. 2003 - A combined phase and amplitude based approach to onset detecion for audio segmentation
• Duxbury, C., Bello, J.P., Davies, M., Sandler, M. 2003 - Complex Domain Onset Detection for Musical Signals
• Lacoste, A., Douglas, E. 2006 - Machine Learning for Note Onset Detection
• Lacoste, A., Eck, D. 2005 - A Supervised Classification Algorithm for Note Onset Detection
• Leveau, P. ISMIR 2004 - Methodology and tools for the evaluation of automatic onset detection algorithms in music

Onset detection - it's tricky business

I ran into some complications with extracting some notes. The instrument I was looking at was the Double Bass. The IOWA file can be found here.

Here's a demonstration of the problem encountered.











Taking a closer look at note 'G' :



The spurious peaks detected in the audio sample proved problematic for my algorithm. I suspect that this being an 'arco' sample, the problem could be due to vibrato?? Anyway, my work around involved adding more thresholds. I noticed that there tend to be short periods of non-peak activity in the actual notes themselves. This was causing some confusion. To overcome this I added a min-gap time threshold so these gaps would not interfere. It took a while! So to continue and no doubt, there will be other issues. I'm still to get started on the front end...!

Iowa notes separated successfully

I've been working on some onset detection algorithms and trying to separate the IOWA music samples so I can build up a database of instrument notes. The code I use is based on an adaptation of Mikel Gainza's code...which in turn is based on the following papers:

Duxbury, C. et al. - Complex Domain Onset Detection for Musical Signals
Bello, J.P. et al. - On the use of phase and energy for musical onset detection in the complex domain

I got the algorithms working in MATLAB today. Here's a run through of a simple example:

Music sample: Tuba, file: Tuba.pp.C1B1.aiff

Processed using complex onset detection:

Time plot





Onsets detected





Smoothed onsets





Smoothed onsets with applied threshold





Detected peaks for note C1





Resultant note wav files:
All the resultant .wav files from this example can be found here.

I'm going to add a front end in Matlab so the various parameters can be manipulated to suit the wav file under investigation. This should prove a useful tool for many once it is completed.

Better than Real at Lightwave

I went to Beau lotto's presentation "Better than Real" yesterday at the Lightwave Exhibition at the Science Gallery.

It was interesting...to a degree. In a nutshell, Dr. Lotto discussed how human's perceive colour, and how perception is dependent, on context and, our brain's historical experience making meaning out of present context. lottolab's latest work is based on the notion of sensory substitution - replacing one sense with another. The example provided is how blind people can perceive their world through their ears, i.e. hearing your visual world. Beau provides the example from nature of how Dolphin's use sonar for navigation.

A thought that came to mind was that I'd come across previous work in this area. The work at vOICe, "Vision technology for the totally blind" is a fascinating project and well worth a read. Also, the book "The Blind Watchmaker" by Richard Dawkins has some extremely interesting writings on the evolution of bats and 'echolocation'.

WOW - the wonders of modern tech

Can somebody please tell me how I'm gonna identify this musical instrument?:
http://www.youtube.com/watch?v=kfrONZjakRY

The Ocarina! More details here:
http://www.dancemusicblog.com/software/first-iphone-musical-instrument-the-ocarina/#more-183

I do a bit of DJing myself. I came across this amazing new digital controller; the EKS otus. Looks simply amazing. Controls two decks in one, via a select switch. I've been looking through some forums, for instance, Native Instruments. Mixed responses from the Tractor user community. All in all though, on a personal level, I'd seriously consider purchasing one of these given the amount of storage required for my current setup and my interest in digital music. I can see myself veering down the road of the laptop, digital DJ and this offers a nice solution. Just have to get the finances together to purchase a new MAC!!

Came across a good blog on Dance music too: dancemusicblog.com

Some interesting Pattern Recognition toolboxes...

Just a short note on some interesting Pattern Recognition toolboxes I've come across recently:

DD_Tools
"...data description toolbox wants to provide tools, classifiers and evaluation functions for the research of one-class classification (or data description)."

The toolbox is an extension of the PRTools toolbox.

PRSD Studio
I've applied for a trial licence with prsysdesign. This looks like a very interesting Matlab toolbox.

Paper accepted for 126 AES Convention!

My paper entitled "Evaluating Ground Truth for ADRess as a Preprocess for Automatic Musical Instrument Identification" has been accepted.

Happy Days.

So I'm off to the 126th AES Convention in Munich second week in May.

Understanding the Fundamentals of Music

I came across a fantastic course entitled "Understanding the Fundamentals of Music" by Robert Greenberg.

I've listened to the first 4 lectures. I must say that Robert Greenberg gets his ideas across very well. The focus has been on Western Orchestral music, and offers excellent insight into many of the instruments I've encountered on my own research.

On a personal note; an example provided by Greenberg in the discussion on 'Dynamics' surprised me greatly. Does Ludwig van Beethoven's Piano Sonata no. 8 in C Minor, op. 13 (Pathétique, 1798) not sound uncannily like Linda Ronstadt & James Ingram's, "Some Where Out There"? You can't believe how disappointed I was (am) by this blatant rip-off. I loved this song as a child.

A short list of some of the books which I 'hope' to read shortly:

• The Physics of Musical Instruments, by Neville H. Fletcher and Thomas D. Rossin.
• Fundamentals of Musical Acoustics, by Arthur H. Benade
• The Musician's Guide to Acoustics, by Murray Campbell and Clive Greated
• Science and Music, by Sir James Jeans
• The Math Behind the Music, by Leon Harkleroad
• The Acoustical Foundations of Music, by J Backus

The onset of discoveries

I came across some excellent resources for onset detection - I will add to this list (from an offline compilation) in time but for now, this site has proved valuable:

http://old.lam.jussieu.fr/src/Membres/Leveau/SOL/SOL.htm

This in conjunction with the work of Mikel Gainza (within the ARG - thanks for the code) and Juan Pablo Bello (http://homepages.nyu.edu/~jb2843/Home.html), should prove a worthy springboard.

It's never too late...

They say it's never too late. With 11 months remaining, today is a momentous day. For today, I plant my well seasoned PhD tree into the digital soil of inter-web land. I step back to admire the many branches which have grown, some bare, brown and rotten to the core, others with blossoming leaves, providing nourishment to crawling ideas (insects!) which have in-turn multiplied and evolved. All that remains, is for the roots to settle in their new found plot (blog!) as I hope they come to terms with a more stable foundation having been quite unsettled to-date with numerous homes (including an old notepad, which to my dismay took a run and jump and still remains 'somewhere out there', or lost in the oblivion).

So where am I in terms of my research?

Onset detection. Yes. Indeed. Why? I need to segment all the notes from the IOWA samples. So where's my horse? We must ride forth, and conquer without any further adieu, remembering to pause at energy bursts, consider their authenticity as an onset, validate, take note, and ride the sustain...

By the way: AMII = Automatic Musical Instrument Identification (in case you thought it was AM I Insane)