Autoencoders

Autoencoders are neural network algorithms that provide as output the presented input after a process of compression. For autoencoders the hidden layer h is a compressed representation of the input x, which is obtained by an encode function h = f(x). This representation must be decompressed by a decode function, x' = g(h), in order to obtain a reconstruction of the input.

Here is a simple autoencoder representation where the red neurons are expected to be an approximation of the green neurons.

Although it may sound contradictory, it is not really desired that an autoencoder perfectly copies the input. Our main interest here is on allowing the hidden layer to learn relevant properties of the input and discarding the ones that are not so important. This is achieved by defining the hidden layer with lower dimension than the input.

The loss function of an autoencoder is represented by L(x, f(x)). If our function f(x)) is linear and we choose the mean square error as loss, then the autoencoder is learning the principal subspace of the input and it performs a similar job as PCA. Nonlinear encode/decode function are very powerful and depending on its capacity it might be complex enough to map the exact input into the output.

References

https://blog.keras.io/building-autoencoders-in-keras.html