Blind Source Separation and Independent Component Analysis

About this page

This is not very much of tutorial value and you may need prior exposition to source separation ideas to figure out what I am talking about below. An introduction to the statistical principles at work in ICA/BSS is given in the tutorial paper listed below.

Tutorial paper

I have written a tutorial paper Blind signal separation: statistical principles, for the Proceedings of IEEE which explains the statistical principles behind source separation and ICA.

Source separation and equivariance

It is very rewarding to explore the consequences of this simple fact. It leads in particular to the notion of `serial updating' by following the relative gradient by which efficient adaptive algorithms can be derived. A summary of these ideas can be found in The invariant approach to source separation published in the proceedings of NOLTA'95 and in Performance and implementation of invariant source separation algorithms published in the proceedings of ISCAS'96.

Note: the idea of `relative gradient' has been independently introduced by Pr. Amari who defines a `natural gradient' (based on the Riemannian structure of the probability model) which, after some simplification is identical to our relative gradient (based on the group structure) in the case of ICA. David MacKay also arrived at a similar idea by what he calls a `covariant' approach. If you are really worried about the differences between natural gradient and relative gradient, have a look at this short paper about learning on manifolds from the proceedings of SSAP'98.

The notion of `estimating function' is helpful to unify many approaches to the source separation problem: maximum likelihood, infomax, contrast optimization, cumulant matching... This recent conference paper about Estimating equations for source separation, published in the proceedings of ICASSP '97 summarizes a good part of these ideas.

An efficient batch algorithm: JADE

For off-line ICA, we have developed with Antoine Souloumiac an algorithm based on the (joint) diagonalization of cumulant matrices. `Good' statistical performance is achieved by involving all the cumulants of order 2 and 4 while a fast optimization is obtained by the device of joint diagonalization.

See this page about joint diagonalization.

JADE has been successfully applied to the processing of real data sets, such as found in mobile telephony and in airport radar as well as to bio-medical signals (ECG, EEG, multi-electrode neural recordings).

What we think is the strongest point of JADE for applications of ICA is that it works off-the-shelf (no parameter tuning). Actually, we advocate using the code provided below as a plug-in replacement for PCA (whenever one is willing to investigate if such a replacement is appropriate). The weakest point of the current implementation is that the number of sources (but not of sensors) is limited in practice (by the available memory) to something like 40 or 50 depending on your computer.

• Papers. The Jade algorithm is described or discussed in these papers.
  • The first journal paper describing JADE is Blind beamforming for non-Gaussian signals. This is a reprint from IEE Proc-F., a journal with a strong focus on array processing. This paper includes some performance analysis (inclusing the noisy case) and comparisons to more classic approches to signal separation than ICA. Readers unfamiliar with beamforming and the processing of complex signals may prefer to read the paper published in Neural Computation.

  • The conference paper: On the performance of orthogonal algorithms (reprint from EUSIPCO'94) also adresses the issue of asymptotic performance analysis. It elaborates on the performance of algorithms which, like JADE and many others, are based on pre-whitening followed by contrast optimization. This is a rather technical paper.

  • The more recent paper High-order contrasts for independent component analysis, published in `Neural Computation' 11(1):157--192, 1999, gives another try at explaining why JADE and related approaches are statistically and comptutationally attractive. The paper considers only the noise-free real case, but includes more explanations about JADE as well as comparisons to gradient-based methods.

• Code. The JADE algorithm was originally developed to process complex signals, motivated by applications to digital communications. I now make available another implementation which is tuned to process more efficiently real-valued signals.
  • Implementation in Matlab/octave for the ICA of real-valued data.
    
    • Here is an implementation of JADE for real-valued data (latest release: version 1.8. May 2005, Re-licensed in 2013.).
    • You may want to test the code on this toy demo calling program.
    • This stand-alone program reads an ascii file of input signals, runs jade on it, and produces an ascii file of separated components. It can be called from the shell if octave (the free Matlab clone) is installed on your system. No need for programming at all, just read the header of the file...

  • Implementation in Matlab/octave for complex-valued signals.
    You can download the JADE algorithm for complex-valued signals as a Matlab function (new, faster, cleaner version: nov. 97, Re-licensed in 2013). To run a Matlab demo of it, you can use this calling program.

  • Implementation in python for real-valued signals.
    This is a port of the matlab/octave code to Python+NumPy by Gabriël J.L. Beckers. The first release is here. but you will find more details at the dedicated page of Gabriël.

  • Implementation in C for real-valued signals.
    You can download an implementation of JADE in C for real-valued signals.

  • Matlab MEX-files for real-valued signals.
    This is available from the same archive as above, i.e. here.

  • Implementation in R for real-valued signals.
    JADE has been ported to R by Klaus Nordhausen. The package can be found on CRAN (Comprehensive R archive network) .

  • An implementation in Julia by Marco Congedo on his github.

  • In Python several ports over the web. I found this one for instance.

  • Other implementations.
    Have you ported JADE to some other language? Please, let me know.

  • Joint diagonalization only
    Some code for joint diagonalization in matlab/octave.

Adaptive algorithms: relative gradient algorithms

• Papers.
  • Adaptive source separation with uniform performance. This is a brief four-page description published in the proceedings of EUSIPCO '94.
  • Equivariant adaptive source separation. This comprehensive version appeared in IEEE Tr. on S.P. (dec. 96) but, due to an editing mistake, the published version does not include figure 8 showing an application to real digital communication signals. This is corrected in this on-line version.

• Code. By the way, this algorithm works pretty well... Check it out! Get a Matlab demo of it. Beware: this is a demo with complex-valued signals, of the type encountered in digital communications.

Multi-dimensional independent component analysis.