Programming – Page 3 – Hi, I'm Rauf Aliev.

Decoding the Beast: Migrating from Excel to Code | December 17 2025, 18:56

We’ve all encountered it — the “Main Excel Spreadsheet Managing the Business.” The very one B2B companies use to calculate million dollar quotes. It has 12 tabs, 1000+ nested formulas, and zero documentation. For ten years, it had “quick fixes” slapped on and constants hidden away. It’s no longer just a file, but a living organism that no one fully understands except for the guy who quit years ago. That’s how puzzled I was. Moreover, there was uncertainty whether even half of the formulas were needed, or if they were vestiges of the past.

Typical cell:

=IF($D11=$D10,””, IF(ISNUMBER( INDEX(Data!$T$10:$U$17,

MATCH(TabCalc!$F11,Data!$T$10:$T$17,0),2)),

INDEX(Data!$T$10:$U$17, MATCH(TabCalc!$F11,Data!$T$10:$T$17,0),2),

INDEX(TabProd!$C$8:$U$112,TabCalc!$D11,I$1)))

I was tasked with transferring this logic into code so that it was all computed by software. The Excel file seemed to have everything it needed, but in reality — it was a complicated black box. 1069 formulas.

The challenge was in how to translate a thousand interdependent formulas into clean code without losing any edge cases.

Here’s what I ended up doing.

Instead of rewriting everything from scratch at once with uncertain prospects of bug proliferation, I used a strategy of lazy computations and mocks.

I built a structure on Groovy that mimicked Excel’s behavior. Each computation (from a cell) I defined as a function that executed only when it was called. And the functions were a multidimensional dictionary.

I started from the end of the computation graph: from results to inputs. If a formula depended on something I hadn’t yet written, I “mocked” it in the code, simply substituting the value from the Excel sheet.

Bit by bit I replaced these mocks with real logic. Comparing the output of my code to the Excel at each step, I could clearly see where my logic diverged.

In other words, I moved from the result to the input data. At each step, it was clear which mocks needed to be turned into code, and I could compare version +1 with version -1 — the result had to match. As soon as all mocks were replaced with calls — the task was done.

The real “secret ingredient” was the dynamic nature of Groovy for creating a multidimensional map of functions. Instead of static variables, I used a deeply nested structure, where each “leaf” was a closure. This allowed access to any part of the table — be it an input parameter, a config constant, or a complex intermediate result — through a simple, unified syntax, and some components were dynamic.

Here’s an example:

conf[“group”] = { x -> [“a”, “b”, “c”] }

conf[“group”]().each {

calculate[“Group”][“Subgroup”][it][“TotalQuantity”] =

{

x -> calculate[“Group”][“Subgroup”][it][“Someparameter”]() * conf[“someConstant”]()

}

Using dynamic keys and closures, I could iterate through product groups or data sets. Since these were dynamic functions, not stored values, the entire system worked like a living graph of dependencies.

Testing was possible right from the start of transferring the formulas. The charm was that you were kind of addressing a cell through syntax like calculate[“Totals”][“A”](), but in reality, you were launching an entire tree of calculations at that moment. And this was incredibly convenient for debugging.

In two weeks, the “Black Box” was transformed into a transparent, modular library with clear logic, which produced exactly the same result as the original table.

P.S. Of course, all the data in all the screenshots are thoroughly obfuscated, or rather, written from scratch for this text.

Decoding Complex Queries: A Transformative Approach to Search Functionality | December 17 2025, 03:25

Oh, I just solved a really cool problem. It’s tricky to explain though. But I’ll try.

So, the client has 10 search websites. They all use one index but throw different queries at it. To what the user enters, a very long and complex query is added, generated by a module on Sitecore. It includes template and page IDs that need to be included or excluded. Ultimately, it’s impossible to understand what’s going on there. There could be ten opening brackets and some randomly closing ones, but it worked with Coveo. Reformatting helped, but not much.

And each site has its own version of this. Meanwhile, the same IDs appear periodically. I first tried to manually figure this out, but it was a nightmare. Nothing helped. There are also nested conditions. For example, “exclude this template” not globally, but only if that field equals one.

Here’s what I did:

I wrote a script that parses this textual “mess” into an abstract syntax tree (AST). This allowed to turn an unreadable string into a structured JSON object, where it’s clear: here’s AND, there’s OR, and here — a specific condition.

Then I turned these conditions into Boolean algebra formulas. Using the SymPy library, I “fed” these formulas to simplification algorithms. Mathematics itself eliminated duplicates, collapsed excessive nesting, and removed conditions that are logically absorbed by others. As a result, the “trees” became flat and understandable.

In the attachment — the original tree and the simplified one.

To be sure that I didn’t break anything during simplification, I wrote a test generator. It takes the simplified logic, puts it back into a working curl, and checks whether the number of found documents (totalCount) matches the original request. The numbers matched — meaning, the logic is preserved 100%.

Having simplified and standardized structures for each site in hand, I built a comparison matrix. The script analyzed them and highlighted Common Core — conditions that are guaranteed to be required (or prohibited) on all sites without exception, and Specifics — unique “tails” that distinguish one site from another.

In the attached screenshot: REQ means that the condition is guaranteed to be met for any document that goes through this request. NOT — definitely not met. OPT — the condition is present in the request, but it’s not strict by itself. It only works in conjunction with something else. “.” — the condition is not mentioned in the request at all.

For 3 sites it responds instantly, for 10 it takes about 30 minutes.

And of course, all data in all screenshots are thoroughly obfuscated.

Exploring Open Data: A Deep Dive into Loudoun County’s 1.5 Million Trees | December 15 2025, 15:40

I’m checking out what open data we have in our county to play with data analysis over the weekend, and discovered, for instance, an open database of all 1.5 million trees in the county. The screenshot shows just a tiny part around my house.

From Idea to Chess AI: Building a Neural Network to Predict Moves | December 15 2025, 04:33

While figuring out neural networks, I decided to come up with a game-related task for myself. What if I find some ready-made games, and train a neural net to predict moves based on the board situation. Said and done. Of course, generating code is faster with LLM, but I wrote the detailed assignment myself and designed the architecture on my own. In 40 minutes (!) from the idea to the result, I already had a working solution that, at least in the first half of the game, does not mess up too much.

In the screenshot is CuteChess – it works with any chess engine, and in my case, it’s a simple Python script. The script takes the board situation and feeds it to the model. It selects the top 5 moves, and only these top 5 are analyzed deeply for several moves ahead and assesses the position. That is, the neural network suggests possible moves based on the analysis of 20,000 games (534,453 positions). From the results, the best is chosen. It uses the minimax algorithm for this, if that means anything to anyone (it didn’t to me, so Gemini here helped me)

How the model is trained. On the lichess website, you can download games, there are hundreds of gigabytes. I took a file with 800,000 played games from the year 2014. From these 800,000, I select 20,000, specifically looking with a script for games where the result is not a draw (1-0 or 0-1). Next, I calculate the difference (Winner_Rating minus Loser_Rating). It’s not the best metric, but it’s better than nothing. The bigger this difference, the more “confident” the win should be (the strong punish the weak). Thus, I get 20,000 such games.

“Ignoring the moves of the weak” (to avoid teaching the model bad play) is implemented during the training stage of the model. Essentially, the logic is: “If it’s White’s turn now, and White won this game — we learn. If it’s Black’s turn now, and Black lost — we skip and don’t teach the net this move.”.

The neural network is trained in batches of 128 positions at a time. The network receives a board position as input and outputs 4096 — the probability assessment for each possible move.

Selecting games takes about 5 minutes. Training the model on my computer takes about 10 minutes for 20,000 games. You could leave it to train on 100K or a million, and it would definitely be better. No need anymore – I figured it out 🙂

You can view the game here:

https://lichess.org/JWeaIrVW

Exploring the Magic of Neural Networks in Letter Prediction and Visualization | December 14 2025, 23:35

I am currently experimenting with training simple neural networks – primarily to automate the existing toolkit, and some things just seem like magic.

There is a database of 32,000 names. There is a neural network filled with random numbers. I start training, with only this list of names as input. The first layer of the neural network is embeddings, and I set the number of dimensions to 2 for easy visualization. And after 200,000 iterations of training, the system clearly separates vowels from consonants, and for some reason, places the letter “q” slightly apart from other consonants. It seems that this is because the letter ‘q’ almost exclusively predicts the letter ‘u’ (Queen, Quincy, Quentin).

It also very reliably separates vowels and consonants in Russian names. In Russian names, the letters b and l are somewhat away from the other consonants, as are the soft and hard signs (well, that’s understandable).

I wonder how it works. If trained on a normal corpus of texts, the difference would be very clear. Why are vowels separated from consonants? Apparently, from the network’s mathematical perspective, ‘a’ and ‘o’ serve the same function: they “trigger” the prediction of the consonant following them, so the alternation of vowels and consonants is to blame. But damn, it’s interesting 🙂

And since the model can predict the next letters, you might try running it on Russian. On a model with 30-dimensional embeddings, it invents names like: Byaketta, Afsena, Erakey, Zasbat, Daraya, Gaiomahad, Rain, Razhul, Gzhatsiy, Reben, Vureb, Durodira, Turuzhul, Regravgava, Razsan, Gabila, Avganzh, Raksi, Khalebkokhorta, Rather. The model – for those who understand – is this: input of 6×33 characters (because we take up to 6 characters of context), encoded into embeddings of 60, goes to a layer of 100 neurons, and from there back to 33 characters. Some nonsense, but at least it’s clear how it all works at all levels.

Harnessing GPU Power Beyond Machine Learning: A Data Processing Experiment | December 13 2025, 01:16

Torturing my supercomputer. Illustration that the GPU is not just for machine learning and some complex math.

My script takes a thick English dictionary (Webster) and multiplies it by 30, creating a list of 12 million words. Then, the algorithm looks through all 12 million words and replaces all the vowels with asterisks using regex. To add more load, a “word length” column is added, and then we take words longer than 10 letters and find the most frequent (top 5).

So, in Python this is

df[‘masked’] = df[‘text’].str.replace(r'[aeiou]’, ‘*’, regex=True)

df[‘len’] = df[‘masked’].str.len()

res = df[df[‘len’] > 10][‘masked’].value_counts().head(5)

and this code is executed first through the main processor, then through a GPU.

The main processor (I have the top-tier Intel i9 285k) completes this task in 24 seconds, while the Nvidia RTX 5090 does it in 0.51 seconds. That’s a 46 times difference!

[Pandas CPU] Top Patterns:

masked

s*r w. sc*tt. 23280

s*r t. br*wn*. 23220

j*r. t*yl*r. 16140

bl*ckst*n*. 10860

b***. & fl. 10830

Name: count, dtype: int64

[Pandas CPU] Computation Time: 23.5596 sec.

Transferring data to GPU…

Transfer complete in 1.16s

— Running Benchmark: cuDF GPU —

[cuDF GPU] Top Patterns:

masked

s*r w. sc*tt. 23280

s*r t. br*wn*. 23220

j*r. t*yl*r. 16140

bl*ckst*n*. 10860

b***. & fl. 10830

Name: count, dtype: int64

[cuDF GPU] Computation Time: 0.5108 sec.

TOTAL SPEEDUP: 46.12x

Misadventures in AWS: Misusing aws-nuke for Configuration Exports | December 12 2025, 16:29

Just for laughs. I asked Gemini how to export the entire AWS configuration for local analysis, and they recommended using the aws-nuke command for permanently deleting everything, but if you add a dry-run flag, you’ll get the configuration… and someone actually follows such advice 🙂 and then we wonder

Two Weeks on Linux: From Mac to ArchLinux+KDE Bliss | December 12 2025, 16:24

Two weeks on Linux, wildly satisfied. After a Mac. I specifically have a setup of ArchLinux+KDE/Plasma 6.5. Everything here is customizable. For instance, I made a program from scratch in half an hour (no lie, thirty minutes) using Gemini that translates selected text to English or corrects errors if the selected text is already in English when ScrollLock is pressed. There seems to be an app for every situation in life, at least in my field. Everything flies (even though this is an Intel i9 285K/64Gb). I just enter a folder that contains 470,000 files, and it opens instantaneously. I’ve never seen anything like this anywhere else. I launch IntelliJ Idea, and there is practically no delay between clicking the icon and the editor being ready with the loaded project. All devices connected perfectly, unlike with the Mac, for which there are simply no drivers for my HP LaserJet 1018 and I need to perform tricks.

Now I occasionally switch to a Mac, and it drives me crazy that the hotkeys are different. Of course, they can be reconfigured for Mac, and probably I will do that. Muscle memory builds up, and switching quickly doesn’t work out. I miss iMessage a bit – I’m used to writing and responding to messages from the computer. Apple iMusic works, through a browser.

Overall, the impression is very good so far.

Stages of Understanding Scientific Papers | December 10 2025, 19:38

As I periodically read scientific papers on my topic, I will try to articulate the levels of understanding the truth.

Level 0: “Read Later Folder” Downloaded the PDF, the title sounds genius, the abstract seems like the solution to all my problems. The file is forever buried in the ~/Downloads/Papers/ToRead folder.

Level 1: “Sumerian Cuneiform” Don’t understand anything at all. Random symbols, the Greek alphabet is over. “Orthogonal extrapolation of cognitive entropy within a quasi-stationary discourse inevitably induces a bifurcation of transcendental synergism.” Such materials really lower self-esteem. Most often from this level, you either fall back to zero, or gradually move to the second level.

Level 2: “Illusion of Competence” The Abstract is clear, the Introduction reads like a good detective story. But as soon as the main section starts, the text turns into a pumpkin. I can’t paraphrase it in my own words, only in general phrases: “Well, they trained a neural net… kind of.”

Level 3: “Formulas where needed and where not” The Abstract is clear, the first half of the article is also okay (architecture, pictures). But then comes formula (4), where “magic” happens. I take the authors’ word for it that equation (3) leads to (4) because, of course, I won’t check it. Beyond that — sheer horror and belief in a miracle.

Level 4: “Goldfish Effect” While reading — everything is crystal clear. The logic is solid, conclusions are obvious, the authors are smart. I close the tab, someone asks me, “What was the article about?” — and I freeze. My mind goes blank. If you take away the paper, I can’t reproduce even the idea because there essentially isn’t an idea, there is a process.

Level 5: “Armchair Expert” Everything’s clear, I can retell the essence over a beer. I know that Input transforms into Output, but the “black box” inside is still black. Give me a computer, I wouldn’t be able to reproduce even the skeleton because, it turns out, the article lacks half of the important stuff.

Level 6: “Critic-Practitioner” Everything is clear, I can recount, understand how to reproduce (even without their code). I see where they cut corners. I definitely know that the “state-of-the-art” result is achieved only thanks to a lucky seed or dataset and this strange trick in preprocessing, mentioned in the footnote on page 12.

Level 7: “Deconstructor” Hooray, I’ve understood everything and implemented it myself. It works worse than in the article, but I know why. However, I understand this work better than the second author (who just made charts). I see that all this complex mathematics over 5 pages boils down to two paragraphs in the middle.

Level 8: “Nirvana” The article is trivial. The idea is secondary, it was all in the ’90s with Schmidhuber, just named differently. Formulas are overcomplicated for importance. I can write the same in 10 lines of code and it will work faster. Reject.

If anything — I’m stuck somewhere between 2 and 4.

Nostalgia and Innovation: The Story of Starchat.ru | December 09 2025, 23:41

2003. We had a chat, my creation, Starchat.ru, where people constantly hung out and communicated with each other. It had a Java applet! Nobody even remembers what that is nowadays, probably. Initially, some programmer I found on the internet wrote this thing, who then disappeared, and I took over the support.

Just for laughs, I made a bot that you could chat with by simply sending it a private message. It was always online, and not everyone realized that it was a bot. When the robot received a message, it searched through massive chat logs for messages that contained the most words from the query and had some response. A response is the next message directed at the user by someone (like “Vasya: oh just go you know where!” is a response to Vasya’s message). In the chat interface, you had to click on a message and then reply to it. In the presence of several options (and there were always several options, given the traffic of chatters), a random one was chosen.

It turned out to be a robot that very amusingly answers questions. If you ask it what its name is, it always replies with different names but appropriately, with emojis and suffixes, often swearing. Also, the bot always gave adequate responses to standard questions like “where do you live” or “how old are you.” Since there was a huge history, and they talked about everything in general, it was hard to find a question to which the system did not give an interesting/correct/funny answer.

So, the bot had an interesting side effect. If you start swearing at it offensively, it begins to swear back even more offensively. And generally, it often reacts inadequately to attacks and reproaches. Simply because in real conversations, a polite question is answered politely, and a rude one — of course, rudely. The audience had a lot of fun with this bot.

It was especially interesting to read the bot’s logs afterward. People there didn’t understand that it was a robot. They asked it questions, quarreled and made up with it. It was fun)