Exploring the Magic of Neural Networks in Letter Prediction and Visualization | December 14 2025, 23:35

I am currently experimenting with training simple neural networks – primarily to automate the existing toolkit, and some things just seem like magic.

There is a database of 32,000 names. There is a neural network filled with random numbers. I start training, with only this list of names as input. The first layer of the neural network is embeddings, and I set the number of dimensions to 2 for easy visualization. And after 200,000 iterations of training, the system clearly separates vowels from consonants, and for some reason, places the letter “q” slightly apart from other consonants. It seems that this is because the letter ‘q’ almost exclusively predicts the letter ‘u’ (Queen, Quincy, Quentin).

It also very reliably separates vowels and consonants in Russian names. In Russian names, the letters b and l are somewhat away from the other consonants, as are the soft and hard signs (well, that’s understandable).

I wonder how it works. If trained on a normal corpus of texts, the difference would be very clear. Why are vowels separated from consonants? Apparently, from the network’s mathematical perspective, ‘a’ and ‘o’ serve the same function: they “trigger” the prediction of the consonant following them, so the alternation of vowels and consonants is to blame. But damn, it’s interesting 🙂

And since the model can predict the next letters, you might try running it on Russian. On a model with 30-dimensional embeddings, it invents names like: Byaketta, Afsena, Erakey, Zasbat, Daraya, Gaiomahad, Rain, Razhul, Gzhatsiy, Reben, Vureb, Durodira, Turuzhul, Regravgava, Razsan, Gabila, Avganzh, Raksi, Khalebkokhorta, Rather. The model – for those who understand – is this: input of 6×33 characters (because we take up to 6 characters of context), encoded into embeddings of 60, goes to a layer of 100 neurons, and from there back to 33 characters. Some nonsense, but at least it’s clear how it all works at all levels.