Deep learning
Proud news flash!!! All three volumes of my Deep Belief Net trilogy have been purchased by a major international publisher. Thus, they are no longer available from CreateSpace. They are now available from Amazon and all major booksellers.
Deep belief nets are one of the most exciting recent developments in artificial intelligence. The structure of these elegant models is much closer to that of human brains than traditional neural networks; they have a ‘thought process’ that is capable of learning abstract concepts built from simpler primitives. A typical deep belief net can learn to recognize complex patterns by optimizing millions of parameters, yet this model can still be resistant to overfitting.
My three volume set, "Deep Belief Nets in C++ and CUDA C" presents the essential building blocks of several of the most common forms of deep belief nets. At each step the text provides intuitive motivation, a summary of the most important equations relevant to the topic, and concludes with highly commented source code for threaded computation on modern multiple-core CPUs as well as massive parallel processing on computers with CUDA-capable video display cards.
Volume I focuses on restricted Boltzmann machines as the primary building block of deep belief nets. Autoencoders are interesting and powerful alternatives to restricted Boltzmann machines.
Volume II presents the essential theory, equations, and computer code for a particular type of autoencoder which is extensible to the complex domain. This extension is useful for many signal and image processing applications which inherently feature complex-domain predictors.
Volume II also presents several algorithms for preprocessing time series and image data. These algorithms focus on the creation of complex-domain predictors that are suitable for input to a complex-domain autoencoder.
Finally, Volume II extends the restricted Boltzmann machine of Volume I by providing a method for embedding class information in the input layer of the top-level RBM. This facilitates generative display of samples from individual classes rather than from the entire data distribution. The ability to see the features that the model has learned for each class separately can be invaluable.
Volume III presents convolutional nets, a form of deep belief net that is especially adapted to classification of images.
Source code for all routines presented in the books, and the DEEP and CONVNET programs which implement these algorithms, are available for free download. See the links at the bottom of this page.
DEEP is a free program which has the ability to train, analyze, and test deep belief nets. Some notable features of Version 2 include:
--- Ability to read standard-format database files such as Excel CSV files as well as files produced by many major data analysis packages
--- Ability to read labeled image files in MNIST format and optionally apply a two-dimensional Fourier transform to the image data
--- Ability to read a time series file and compute trend and velocity of trend change for a moving window
--- Ability to read a time series file and compute full complex Fourier coefficients in a moving window
--- Ability to read a time series file and compute full complex Morlet wavelet coefficients in a moving window
--- Models can be numerically predictive or employ SoftMax classification
--- Contrastive divergence training of restricted Boltzmann machines (RBMs)
--- Option in the above to embed class labels for class-specific generative sampling
--- Real or complex-domain autoencoding and regression
--- Greedy training of stacks of RBMs and autoencoders
--- Conjugate gradient training of one or more 'supervised' layers that follow the RBM layer(s)
--- Optional supervised fine tuning of the complete network
--- If the user's computer does not contain a CUDA-capable video display card, all training is done with multithreading to keep all cores of the CPU busy. If a CUDA-capable card is present, its massive parallel processing capabilities are used.
--- If the input data is images, each hidden neuron's receptive field can be displayed in black-and-white or color for RBMs.
--- If the input data is images and RBMs are employed, generative samples can be taken and displayed, using either training cases or random hidden neuron activations as starting points for Gibbs sampling.
--- A test set can be processed to evaluate the performance of a trained model.
--- Cross validation can be performed with any number of folds, optional shuffling, and optional boundary guards.
CONVNET is a free program which reads MNIST and CIFAR-10 format image files and allows the developer to design, train, and test convolutional nets. It can also read a time-series file and convert the data path into a sequence of images. If the user's computer has a CUDA-capable device, it will be used for training.
DEEP and CONVNET are works in progress. New capabilities will likely be added on a regular basis. Stay tuned.
Last but not least, please understand that DEEP and CONVNET are experimental programs. They are provided free of charge to interested users for educational purposes only. In all likelihood these programs contain errors and omissions. If you use these programs for a purpose in which loss is possible, then you are fully responsible for any and all losses associated with this use. The developer disclaims all responsibility for losses which the user may incur.
To see and optionally order Volume I from Amazon.com, click here. For Volume II (removed). For Volume III (removed).
To see the Table of Contents of Volume I, click here. For Volume II, click here. For Volume III, click here.
To download the DEEP User's Manual (Chapter 6 of Volume II), click here.
To download the CONVNET User's Manual (Chapter 5 of Volume III), click here.
To download a zip file containing the DEEP program (DEEP.EXX must be renamed DEEP.EXE), User's Manual, and core source code, click here. The included CUDA runtime DLL must be placed in the same directory as DEEP.EXE.
To download a zip file containing the CONVNET program, User's Manual, and core source code, click here. The included CUDA runtime DLL must be placed in the same directory as CONVNET.EXE.
News update... I have just released CONVNET 2.0. It contains several enhancements to Version 1, but it is not as thoroughly tested. Therefore, I am not just replacing Version 1. I am making this a separate download. The four enhancements are:
1) It can now take advantage of multiple CUDA devices on your computer, although because I have only two devices on mine, I have not been able to test it with more than two.
2) Computation of the confusion matrix is now performed on the CUDA device(s), so it is much faster than Version 1, which used the host CPU for confusion computation.
3) Confusion computation can now be interrupted with the ESCape key.
4) If the number of weights for each output neuron does not exceed 101, singular value decomposition is used during simulated annealing to compute nearly optimal output weights for each trial set of hidden-layer weights. This provides tremendously better starting weight estimates and hence speeds convergence of the refinement phase.
Please click here if you wish to download a zip file containing the following items:
1) ConvNet2.EXE (Version 2 and its CUDA runtime DLL, the same DLL as for Version 1)
2) MOD_CUDA_V2.cu (The original Version 1 C source file, modified for multiple devices)
3) Supplement.PDF (Operating instructions for the enhancements to the program, as well as a discussion of the changes in MOD_CUDA_V2.cu needed to convert it from using a single device to using multiple devices)
For DEEP errata, click here.
These programs contain no embedded advertising or malware of any sort, nor do they report back to me any information about your computer. Because the downloaded Zip file contains an executable (.EXE), some anti-virus software may flag it as suspicious. I assure you, these programs are totally free of viruses et cetera. You can verify this yourself by downloading the file without opening it, and then having your anti-virus program scan the downloaded file.