I propose to place here (good) questions and my responses to them, with a view to making my introductory page more useful.
Well...yes and no. Each weight update moves each weight in such a way that it reduces the overall error. If all the moves were infinitesimally small, then that would ensure that the overall error decreased. But only infinitesimally slowly. Given the impatience of people (like, they don't want to wait 10,000,000 years...) they use larger size weight movements. If the weight surface (which is a mapping from the weight space -> the positive real numbers: the weight space is probably of very high dimension) is complicated, with deep ravines etc. then there is certainly the possiblity that weight movement will cause overshoot, and not decrease the weight. Momentum is often used to help with this (time precludes a detailed explanation of how & why: try googling momentum and backprop), but it can still be a problem. Alternatives include attempting to use steepest detcent (rather than simply descent in error space), but this usually involves inverting large matrices, and is certainly a non-local learning technique. In conclusion: yes, it might never learn even although the network is such that it could learn. Usually, one gets round this by (i) starting from a number of different points in weight space, and (ii) using a variety of learning rates and momentum rates (or even reducing them if it looks as though the error sum is oscillating.
In a word, no. The output of the neural network does not in general define a unique input. (Or to put it mathematically, the function embodied in the network is not (uniquely) invertable.) There's no reason why it should be - often we really do want different inputs to map to the same output!
I couldn't manage to find a proper recent introduction, but there is a rather old article called An Introduction to Fuzzy Logic and Neural Networks (from the 1993 International DevCon) on the web. There are books too (but you can look up Amazon as easily as I can!).
If you have any difficulties accessing this page, or you have any queries/suggestions arising from this page, please email: