Tag: MachineLearning

Navigating the Future: Embracing Earth’s Magnetic Field as a GPS Alternative | January 10 2026, 17:41

I learned today that there is and is actively used a technology for navigation using the Earth’s magnetic field. It is used as a replacement or an extension of GPS.

For example, there is the Scandinavian ferry Express 5 of Bornholmslinjen, which insures against GPS problems (which do happen) by using MagNav navigation. Unlike GPS, the Earth’s magnetic field cannot be jammed or spoofed—it simply exists. The ferry follows the same route, and generally, navigation could even be achieved through household fishing sonars.

But there are a few startups that use this technology for indoor navigation, where GPS signals cannot reach. It’s claimed that the navigation accuracy is within 1 meter. That’s more interesting.

GiPStech, Oriient, Mapsted.

The basis of this technology is a process called magnetic fingerprinting. Engineers or mapping robots walk through a building with a smartphone, recording unique distortions of the magnetic field at every point. These distortions are created by the steel frame of the building, rebar in the walls, and large electrical equipment. A database is formed where each coordinate (x, y, z) corresponds to its unique magnetic field vector (intensity, inclination, deviation).

The collected data is uploaded to the cloud platform of the provider company. There, they undergo noise cleaning and are “stitched” together with the digital floor plan. When a user walks through a shopping center, their smartphone reads data from the built-in magnetometer in real-time. Special software (SDK) compares the current readings with those stored in the database. For accuracy to be within 1–2 meters, the system relies not only on magnets. It uses sensor fusion—combining data from the magnetic field with inertial sensors (accelerometer counts steps, gyroscope determines turns) and sometimes Wi-Fi/Bluetooth signals for rough localization.

This technology is certainly being actively implemented for drones. The main technical difficulty there is dealing with their own interference and considering that the magnetic field changes, requiring constant map updates. Electrics, engines create strong magnetic fields, which “drown out” the natural background of the Earth. However, various filtering algorithms (including neural networks) are used, which in real-time “subtract” motor interference from the overall sensor readings. From what I understand, at high altitudes (kilometers), the magnetic field is more “smooth”, therefore the accuracy is lower (about 1–5 km). But if several drones fly together and exchange signals, overall they can provide very good accuracy each. Additionally, a group of drones can measure the gradient (rate of change) of the magnetic field in space, tying location not to absolute values, but to relative ones. Essentially, using a group of drones turns the navigation system from a set of individual receivers into a distributed phased array antenna, capable of filtering global interferences and working with much weaker useful signals. Considering that small drones capable of staying airborne for long periods can be released into the air by the hundreds (and cost pennies), this is a quite promising area for military.

There’s an interesting startup, Zerokey. They release QUANTUM RTLS 2.0. This device provides spatial accuracy to 1.5mm. It’s used in production, for example. Their video shows a “watch” on a worker’s hand that monitors the correctness of assembling something on a table. Here, the principle is ultrasonic, and it’s understandable that these “watches” are paired with stationary sensors and further multilateration.

Exploring ASML’s Advanced Chip-Making Equipment with Veritasium | January 02 2026, 00:47

Veritasium released a very cool report yesterday from ASML about the equipment used to print chips for your little phones, cameras, and laptops.

For those who aren’t familiar with the process. First, a monocrystal is grown from ultra-pure silicon and cut into thin wafers, then multiple layers of thin dielectrics, conductors, and semiconductors are repeatedly applied to the wafer surface, each time shaping the necessary areas using photolithography, etching, and ion doping, eventually creating billions of transistors and connecting metallic paths; finally, the wafer is tested, cut into individual crystals, and packaged into casings, making them into finished microchips.

This process had a limitation – the width of the paths and the distance to the next one are limited by the wavelength of the light used, and reducing it is difficult because there’s nothing to focus such a beam with – lenses simply absorb/reflect everything. In EUV lithography (extreme ultraviolet), the wavelength is 13.5 nm. This is virtually soft X-ray radiation.

The video explains details about the ASML machine costing 400 million dollars. Instead of refracting lenses, highly complex systems of reflecting mirrors are used. These mirrors are the smoothest surfaces ever created by humanity. If the mirror of this machine were enlarged to the size of the Earth, the largest bump on it would not be thicker than a playing card. To enable the mirrors to reflect X-rays, up to 76 alternating layers of tungsten and carbon, each less than a nanometer thick, are applied. All this is done by Zeiss. In addition, this mirror has a controlled curvature—it is constantly adjusted by robots with precision up to picoradians. The precision of the mirror control is so high that if a laser were mounted on it, directed at the Moon, the system could choose on which exact side of a 10-cent coin lying on the moon’s surface to hit with the beam.

But. We don’t have a “light bulb” that emits light in the EUV range.

To generate this light, a laser “shoots” at a droplet of molten tin the size of a white blood cell, traveling at 250 km/h. The first pulse flattens the droplet into a disc, the second and third turn this “disc” into plasma – and all this occurs within just 20 microseconds. When hit by the laser, the droplet heats up to 220,000 Kelvin — approximately 40 times hotter than the surface of the Sun. This plasma emits that very necessary light. And it does so 50,000 times a second. They say it’s been brought up to 100,000. Imagine, at a hundred thousand laser shots per second, it never misses a single one. All this happens in a deep vacuum. To clean the mirrors from tin particles, the chamber is constantly blown with hydrogen at a speed of 360 km/h — faster than a Category 5 hurricane. This process is described by the same formula (Taylor-von Neumann) that describes a nuclear explosion or supernova explosion.

The machine layers the chip with an error margin of no more than five atoms, while the matrix swings back and forth with an overload of 20G.

A single High-NA machine is transported in 250 containers on 25 trucks and seven Boeing 747 aircraft.

Link to the video – in the comments. Or search on YouTube on the channel veritasium.

Designing 3D Volleyball Training Tools on the Fly | January 01 2026, 21:21

What I did on the plane to/from vacation and sometimes in between: 3D visualization and editing volleyball schemes for Nadya (she’s a coach). This court in the attached image freely rotates, players can be placed on it, and the ball and player paths are shown – all in 3D.

The ball’s trajectory is calculated so that it does not cross the net when moving from A to B (Bezier formula). Players can take several poses – right now there are hastily made poses for serve, attack, block, pass/receive. Interestingly, in the code: I had to write a bit of “volleyball brains”. The system itself calculates the ball’s trajectory through Bezier curves so that it always passes over the net. Moreover, the height of the launch depends on the type of action: for an attack, the ball “launches” from a higher point than for a pass. I also added auto-rotation: the 3D model itself turns its face to where, according to the scheme, it needs to pass or run.

The longest and most difficult task was creating the 3D model of a female volleyball player. To generate a realistic volleyball player, I used the tripo3D service. It gave me a model in a neutral pose (for free). Theoretically, you can then use Blender and the Rigify plugin to attach an armature to it and move its arms and legs, which would recalculate the model.

However, in reality, this approach does not work well: the AI-generated model contains a large number of geometric errors, which the renderer forgives but Rigify does not. They can be roughly divided into two types — incorrect polygon normals and issues with non-manifold geometry, which are significantly more challenging to fix. Inside the body, there may be “floating” clusters of polygons or intersecting surfaces. When Rigify tries to calculate weights (which bone affects which part of the skin), this internal noise confuses the algorithm, and as a result, the weights are distributed chaotically (for example, moving the arm might start pulling the mesh on the stomach). Plus, the model is slightly asymmetrical.

Non-manifold is a geometry error where the topology of an object ceases to be correct in terms of a three-dimensional body: edges may belong to more than two polygons, polygons may only touch at vertices or edges without a common volume, and “hanging” surfaces or zero thickness may appear inside the model. Such geometry formally does not describe a closed volume, causing problems with rigging and deformations. Moreover, the model needs to be simplified because millions of polygons are not needed for rendering in real-time in a browser.

I fixed these using MashLab, additionally refining by hand (“with a file”). In the end, the model turns out slightly different from the original almost everywhere. The original model had “skin” in the form of textures – the face, shirt, and shorts had to be colored. How to transfer all this to a simplified model? For that, there’s a special operation in Blender called Baking. This also involves some tricks. In the end, it didn’t transfer perfectly, but perfection isn’t necessary yet.

Next, we attach the armature to the “joints”, and after about three hours of figuring out why everything does not work as it should, it finally worked. I made four poses, and now each circle (player) can be told which pose it is in.

I’ll also need to make dynamic changes to the uniform colors – that shouldn’t be difficult. There’s also an idea to transfer poses from photographs – this is more complicated, but generally feasible. Using MediaPipe/AlphaPose, you can detect key points in 2D, then some models like HMR/HybrIK can “lift” flat coordinates into 3D space, outputting relative joint rotation angles. The resulting data can be attempted to be projected onto a Rigify skeleton. Since the proportions of the generated volleyball player and the person in the photo may not match, that’s exactly why Inverse Kinematics (IK) is used. This part is quite complex, but overall it’s not strictly necessary – just interesting to figure out and make something functional.

Video in the comments

Decoding Complex Queries: A Transformative Approach to Search Functionality | December 17 2025, 03:25

Oh, I just solved a really cool problem. It’s tricky to explain though. But I’ll try.

So, the client has 10 search websites. They all use one index but throw different queries at it. To what the user enters, a very long and complex query is added, generated by a module on Sitecore. It includes template and page IDs that need to be included or excluded. Ultimately, it’s impossible to understand what’s going on there. There could be ten opening brackets and some randomly closing ones, but it worked with Coveo. Reformatting helped, but not much.

And each site has its own version of this. Meanwhile, the same IDs appear periodically. I first tried to manually figure this out, but it was a nightmare. Nothing helped. There are also nested conditions. For example, “exclude this template” not globally, but only if that field equals one.

Here’s what I did:

I wrote a script that parses this textual “mess” into an abstract syntax tree (AST). This allowed to turn an unreadable string into a structured JSON object, where it’s clear: here’s AND, there’s OR, and here — a specific condition.

Then I turned these conditions into Boolean algebra formulas. Using the SymPy library, I “fed” these formulas to simplification algorithms. Mathematics itself eliminated duplicates, collapsed excessive nesting, and removed conditions that are logically absorbed by others. As a result, the “trees” became flat and understandable.

In the attachment — the original tree and the simplified one.

To be sure that I didn’t break anything during simplification, I wrote a test generator. It takes the simplified logic, puts it back into a working curl, and checks whether the number of found documents (totalCount) matches the original request. The numbers matched — meaning, the logic is preserved 100%.

Having simplified and standardized structures for each site in hand, I built a comparison matrix. The script analyzed them and highlighted Common Core — conditions that are guaranteed to be required (or prohibited) on all sites without exception, and Specifics — unique “tails” that distinguish one site from another.

In the attached screenshot: REQ means that the condition is guaranteed to be met for any document that goes through this request. NOT — definitely not met. OPT — the condition is present in the request, but it’s not strict by itself. It only works in conjunction with something else. “.” — the condition is not mentioned in the request at all.

For 3 sites it responds instantly, for 10 it takes about 30 minutes.

And of course, all data in all screenshots are thoroughly obfuscated.

From Idea to Chess AI: Building a Neural Network to Predict Moves | December 15 2025, 04:33

While figuring out neural networks, I decided to come up with a game-related task for myself. What if I find some ready-made games, and train a neural net to predict moves based on the board situation. Said and done. Of course, generating code is faster with LLM, but I wrote the detailed assignment myself and designed the architecture on my own. In 40 minutes (!) from the idea to the result, I already had a working solution that, at least in the first half of the game, does not mess up too much.

In the screenshot is CuteChess – it works with any chess engine, and in my case, it’s a simple Python script. The script takes the board situation and feeds it to the model. It selects the top 5 moves, and only these top 5 are analyzed deeply for several moves ahead and assesses the position. That is, the neural network suggests possible moves based on the analysis of 20,000 games (534,453 positions). From the results, the best is chosen. It uses the minimax algorithm for this, if that means anything to anyone (it didn’t to me, so Gemini here helped me)

How the model is trained. On the lichess website, you can download games, there are hundreds of gigabytes. I took a file with 800,000 played games from the year 2014. From these 800,000, I select 20,000, specifically looking with a script for games where the result is not a draw (1-0 or 0-1). Next, I calculate the difference (Winner_Rating minus Loser_Rating). It’s not the best metric, but it’s better than nothing. The bigger this difference, the more “confident” the win should be (the strong punish the weak). Thus, I get 20,000 such games.

“Ignoring the moves of the weak” (to avoid teaching the model bad play) is implemented during the training stage of the model. Essentially, the logic is: “If it’s White’s turn now, and White won this game — we learn. If it’s Black’s turn now, and Black lost — we skip and don’t teach the net this move.”.

The neural network is trained in batches of 128 positions at a time. The network receives a board position as input and outputs 4096 — the probability assessment for each possible move.

Selecting games takes about 5 minutes. Training the model on my computer takes about 10 minutes for 20,000 games. You could leave it to train on 100K or a million, and it would definitely be better. No need anymore – I figured it out 🙂

You can view the game here:

https://lichess.org/JWeaIrVW

Exploring the Magic of Neural Networks in Letter Prediction and Visualization | December 14 2025, 23:35

I am currently experimenting with training simple neural networks – primarily to automate the existing toolkit, and some things just seem like magic.

There is a database of 32,000 names. There is a neural network filled with random numbers. I start training, with only this list of names as input. The first layer of the neural network is embeddings, and I set the number of dimensions to 2 for easy visualization. And after 200,000 iterations of training, the system clearly separates vowels from consonants, and for some reason, places the letter “q” slightly apart from other consonants. It seems that this is because the letter ‘q’ almost exclusively predicts the letter ‘u’ (Queen, Quincy, Quentin).

It also very reliably separates vowels and consonants in Russian names. In Russian names, the letters b and l are somewhat away from the other consonants, as are the soft and hard signs (well, that’s understandable).

I wonder how it works. If trained on a normal corpus of texts, the difference would be very clear. Why are vowels separated from consonants? Apparently, from the network’s mathematical perspective, ‘a’ and ‘o’ serve the same function: they “trigger” the prediction of the consonant following them, so the alternation of vowels and consonants is to blame. But damn, it’s interesting 🙂

And since the model can predict the next letters, you might try running it on Russian. On a model with 30-dimensional embeddings, it invents names like: Byaketta, Afsena, Erakey, Zasbat, Daraya, Gaiomahad, Rain, Razhul, Gzhatsiy, Reben, Vureb, Durodira, Turuzhul, Regravgava, Razsan, Gabila, Avganzh, Raksi, Khalebkokhorta, Rather. The model – for those who understand – is this: input of 6×33 characters (because we take up to 6 characters of context), encoded into embeddings of 60, goes to a layer of 100 neurons, and from there back to 33 characters. Some nonsense, but at least it’s clear how it all works at all levels.

Harnessing GPU Power Beyond Machine Learning: A Data Processing Experiment | December 13 2025, 01:16

Torturing my supercomputer. Illustration that the GPU is not just for machine learning and some complex math.

My script takes a thick English dictionary (Webster) and multiplies it by 30, creating a list of 12 million words. Then, the algorithm looks through all 12 million words and replaces all the vowels with asterisks using regex. To add more load, a “word length” column is added, and then we take words longer than 10 letters and find the most frequent (top 5).

So, in Python this is

df[‘masked’] = df[‘text’].str.replace(r'[aeiou]’, ‘*’, regex=True)

df[‘len’] = df[‘masked’].str.len()

res = df[df[‘len’] > 10][‘masked’].value_counts().head(5)

and this code is executed first through the main processor, then through a GPU.

The main processor (I have the top-tier Intel i9 285k) completes this task in 24 seconds, while the Nvidia RTX 5090 does it in 0.51 seconds. That’s a 46 times difference!

[Pandas CPU] Top Patterns:

masked

s*r w. sc*tt. 23280

s*r t. br*wn*. 23220

j*r. t*yl*r. 16140

bl*ckst*n*. 10860

b***. & fl. 10830

Name: count, dtype: int64

[Pandas CPU] Computation Time: 23.5596 sec.

Transferring data to GPU…

Transfer complete in 1.16s

— Running Benchmark: cuDF GPU —

[cuDF GPU] Top Patterns:

masked

s*r w. sc*tt. 23280

s*r t. br*wn*. 23220

j*r. t*yl*r. 16140

bl*ckst*n*. 10860

b***. & fl. 10830

Name: count, dtype: int64

[cuDF GPU] Computation Time: 0.5108 sec.

TOTAL SPEEDUP: 46.12x

Data Science: The Modern Alchemy of the 21st Century | November 16 2025, 04:02

A cryptic post today. While writing a book on RecSys, I caught myself thinking that modern data science is essentially the alchemy of the 21st century. Half of the “best practices” in algorithms lack a solid mathematical framework. It’s a set of heuristics that “just work”. Much like in the 17th century where they mixed everything indiscriminately, it happens now, and if something works better, everyone else starts doing the same. There’s just no answer to the question “why”.

Take, for example, the NCF/NeuMF (Neural Collaborative Filtering) algorithm. The logic goes like this. Say, there are a million movie ratings by users. And 100 million ratings by users yet given – users can’t watch every movie in the world. But out of these 100 million, you need to choose candidates for advertising for a particular user. The algorithm, of course, has a training phase, where weights are calculated, and a prediction stage, where these weights are used on the incoming data.

(What the algorithm does. Essentially, it’s an ensemble of three sub-algorithms, two of which generate their own conclusions, and then their decisions go to a new neural network, the third algorithm, which provides the final recommendation. Smartly, it’s a hybrid of GMF (matrix factorization) and MLP (Multi-Layer Perceptron). The first of these two is based on matrix decomposition, and the second represents a neural network with multiple layers. Weights are adjusted on training data.)

For one positive example, it takes 4 negative ones. Why four? Just because it’s “not too many and not too few”. Would 8 be better? Unknown, but it would definitely take longer to learn.

Why are embedding dimensions 32? or 64? There’s no formula. It’s the “golden mean” between a “dumb” model (few k) and an “overtrained” (many k).

Now about the neural network. Why is the MLP block built as a “tower” (64 -> 32 -> 16)? Why not (50 -> 25 -> 10)? Why ReLU between them (and not tanh for example)? Pure empiricism. The number of layers in the tower is also adjusted.

Why do GMF and MLP parts have different embeddings at the input? Because the authors of the paper tried it, and it “worked out better”. No mathematical proof. Why do they go to the final layer with equal weights? Because they just do.

Why are the outputs of the two paths “concatenated” (concat), and not added or multiplied? “Experience showed that this way the result is more accurate.”

And so it is with everything, up to the choice of optimizer Adam or the “magical” learning_rate=0.001, although at least these have some mathematical basis.

That is, at least a dozen parameters of one algorithm are empirically chosen, with no clear confidence that they are independent of each other. But many of them depend on the dataset, but no one knows how 😉

In general, alchemy.