I wonder if there exists an agent that takes an Excel table significantly larger than the context window and begins to document its essence. Here are several tabs. Here on tab 5, there is a table with a million rows and five columns. The columns are as follows. We take random data from the table, looks like there are numbers, and there – surnames. We assume that there are numbers everywhere – we write a code that checks this assumption and at the same time calculates min/max and a set of unique values. So, few values, only five. We record it. Now we check the surnames. Yes, these are just strings, new sampling showed that they are indeed surnames. Here’s a formula. We see where it points. And so on. And this column – unclear purpose. We look at the data – these are some numbers from 0 to 1. We measure the average and the spread. We ask the user – maybe they’ll provide some comments. They did. It turned out to be a KPI issued to this user from an external system. We record it. And so on. Documentation emerges. Later, when there is documentation, one can request to perform some operations with all this, since the LLM now more or less understands the purpose of the data and their connection, and can build some hypotheses on detecting outliers and verifying them.
Category: Software
The Crucial Role of Data Quality Oversight in Development Projects | May 06 2026, 16:07
Almost every development project features a dedicated functional testing automation team, yet surprisingly, a similar emphasis on Data Quality is rarely found. Regardless of whether data comes from external integrations, users, or is generated by the system itself, it often remains without proper control simply because no one seems to consider it important, and later they struggle with the consequences – they accumulate like a snowball. The longer such issues persist, the harder they are to resolve, eventually leading to a situation where people just resign themselves to the “irreparable” state of the database. It is much better to identify these problems at the moment they arise, while the technical debt has not yet become insurmountable, rather than later figuring out how to prevent them from causing everything to crash;
In essence, there needs to be a constant “supervisor” over all types of databases used by the system (relational, NoSQL, search indexes, or graph databases) — essentially, this is a layer of data quality checking over processes. Of course, there must be clear rules – specifically what to check and which flags to use to mark specific anomalies.
There must be a responsible party for the process (a human, not AI), who will integrate these reports into the development and support workflows. Many data integrity issues cannot just be resolved through an interface — they require the engineering team to develop scripts for mass correction and data cleansing.
Incidentally, this also transitions into the realm of anomaly detection (outlier detection). Machine learning and LLMs for identifying subtle “bad” patterns that traditional rule-based systems might miss.
What do you think about this? Are similar mechanisms implemented in your processes?
Repurposing Components from a Broken Air Purifier | May 03 2026, 15:00
The air purifier broke down, so I bought a used one with a new cartridge for the price of a replacement cartridge plus $40. I completely disassembled the old one, extracted the reusable components, and figured out how it works. Just like in school 🙂
Inside, it comprises:
– an ESP32-WROOM-32D controller. But a part of the board responsible for voltage burned out, so it’s trash now.
– a CO sensor MQ-7 (unfortunately soldered to the board, but can be desoldered). Though, it needs a heating cycle for correct operation. First 5V (60 sec) for sensor cleaning, then 1.5V (90 sec) for measurement. But, it can also be used elsewhere.
– Plantower PMS9103M — a high-precision laser sensor for measuring airborne particulate matter concentrations (PM1.0, PM2.5, PM10). Can be connected to Arduino, specification available.
– a microwave motion sensor (radar), model RCWL-0516. Can be connected to Arduino, very simple interface. Detects motion up to 5-7 meters around within 360 degrees.
– 200W Snowfan YY225H310B motor. Also quite simple to connect, but it requires 310V DC plus 15V for speed control. But that’s all.
– a Hall sensor (magnet)
The motor is the most valuable part. It’s priced at $100 on eBay. Though, it should probably be tested first to see if it hasn’t burned out.
Harnessing Chat Data for Semantic Q&A Search | April 30 2026, 04:05
In one evening, I created a simple utility that extracts the Natural Language Processing chat for a year and a half – there are 65,000 messages, and converts it into question-answer pairs with semantic search available. Clicking on a search result (on the left) opens the dialogue in the chat. The messages that are responses to the question are highlighted. And at the top, the original phrasing of the question is highlighted as well.
How it works: the system assumes that people mainly reply to messages that are relatively close in the past. If several replies are made to one message, then it is likely useful and caught the interest of others in the chat. The system takes messages starting from the one many have replied to, ending with the last in the reply-to chain – and among such messages, it selects those that have at least 3 reply-tos to the original question. In essence, it cuts a piece from the chat starting with a popular question so that after the bottom cut, most likely, irrelevant content follows. Such blocks can overlap each other – for example, if someone asked a question while others were replying to something else.
So, if user A asked what the weather was like, and they received answers like “good,” “bad,” “rain,” and there were five messages without a reply-to, and then someone replied to “rain” with the question “why rain”, and five more people replied to this question, then the first question about the weather makes it into the system – the piece ends with 13 messages.
Afterwards, these pieces are summarized into question-answer pairs.
It turns out quite cool.
P.S. In the screenshot, the search query has nothing to do with the search result because I foolishly took the screenshot after I changed the query but before I hit send.
Misadventures in Keyboard Layouts: Searching for Gremlin, Finding Surprises | April 28 2026, 20:33
This is me typing the word gremlin, without switching the keyboard layout. Wanted to read about the query language for graph databases, need it for work. Google surprises, it does surprise
CPU vs GPU: A Speed Challenge in Embedding Creation | April 11 2026, 18:08
When working with certain tasks, the difference between a CPU and a GPU is simply astounding. For example, I need to create many (millions) of embeddings, model BGE M3. Running this on my quite powerful 24-core Intel Core Ultra 9 285K processor takes 45.85 seconds to create 500 embeddings, while using an NVIDIA 5090 GPU, the same task is completed in just 0.36 seconds. It is so fast that I specifically wrote this benchmark to figure out whether my GPU is being utilized at all. The program that sends requests to TEI does it in test mode not actively enough (roughly a couple of times per second), and the GPU load graphs are practically zero.
— Testing http://localhost:8080/embed — <– CPU version
Requests completed: 500
Total time: 45.85 sec
Throughput: 10.90 req/sec
Average latency (Avg Latency): 4386.11 ms
P95 latency: 5021.88 ms
— Testing http://localhost:8090/embed — <– GPU version (NVIDIA 5090)
Requests completed: 500
Total time: 0.36 sec
Throughput: 1398.69 req/sec
Average latency (Avg Latency): 31.38 ms
P95 latency: 53.18 ms
========================================
RESULT: http://localhost:8090/embed is 99.22% faster
Smartfolio.me: Revolutionizing Knowledge Organization with Advanced Features | March 19 2026, 04:01
My creation – the knowledge organization tool Smartfolio.me – has gained new features. I’m attaching a five-minute video overview.
It’s like Google Docs, but you can embed documents within each other, creating a network of connected knowledge, and these documents can be PDFs and regular texts.
Upload a PDF, the program converts it into images, and you can highlight any sections right on the pages to leave a comment or ask a question.
If something in the text is unclear, you highlight the area and press “elaborate” — the LLM will detail everything thoroughly, taking into account the context of the entire document, and the explanation will stay linked to the highlighted fragment.
You can simply cut out a piece from a PDF, and the LLM extracts clean text or a ready-made formula from it.
In the PDF window, there is now a small panel — all comments and explanations are immediately visible there, so you can quickly jump to the necessary parts.
You can cut out a diagram or graph from a PDF, copy it as a picture, and paste it into your text. It will automatically crop “on the fly” and save in the database, not as a copy but as a link to the page with crop parameters.
If you delete the page link in the text, it won’t disappear completely but will go into a special list, from where you can reattach it somewhere else or delete it finally. The same document can be inserted in several places. If you add a comment to it, it updates everywhere where this document is linked.
Mathematics is fully supported — LaTeX formulas can be not only viewed but also clicked to adjust them in the editor.
You can generate formulas by description. Just write in words what formula you need (for example, “binomial distribution”), and the system itself outputs the ready formula code.
Now there is a system of plugins – essentially isolated experimental functions separate from the main program. For instance, there is a plugin that recursively collects all subpages into one long document — convenient if you need to read or print everything at once.
Or consider the “YouTube Transcript Cleaning” plugin. If there is a dirty lecture text from YouTube, the plugin will punctuate, paragraph, and create neat headers.
If you insert a link to a website, it opens in a column next to it — you can read the source and simultaneously take your notes. However, some websites do not allow embedding on foreign pages. The system recognizes such sites, and they open in a new tab.
The left panel with the list of pages can be hidden or resized with the mouse, so it doesn’t take up space on the screen.
You can simply copy and paste an image or screenshot, and it will not just insert, but also upload to the database.
It supports working from a mobile phone. On the phone, the interface switches to a single-column mode for convenient reading and commenting on the go.
Multiple databases are supported – you can switch between them. You can connect different databases and different LLMs and switch between them.
Crafting Nabokov’s Dictionary: A Multilingual Lexical Journey | March 15 2026, 18:30
I’m reading Nabokov and decided to take a break to create a convenient app “Nabokov’s Dictionary” and am considering selling it on Amazon as a book. Essentially, it looks like this (see screenshot) – definitions of complex words in English, Russian, German, and French, in the same order they appear in the original book.
Would you buy such a book?
To accurately make their definitions, I also wrote an aligner – a program that matches sentences and paragraphs in English with their translations (Nabokovian) into Russian. And when a word’s definition is created, it uses not only the knowledge of LLM but also the Russian translation by the author. It’s worth separately discussing how the algorithm works (I invented it myself because everything I found online did not work as I needed). It first finds long sentences and matches the longest sentences with their pair through cosine similarity of embedding vectors created through the multilingual e5 model. These sentences become anchors. Then, assuming that for long sentences the error is almost excluded, the longest sentence between anchors is found, and everything repeats recursively. There are many situations where a sentence in Russian has no equivalent in English and vice versa, where a sentence is split into two, or conversely two are merged into one. The algorithm handles this as best as it can. The result is quite a good quality of alignment. To such an extent, that errors in alignment can hardly be found (but they are likely still there). Either way, it is only needed for the context for translating words, even if there are rare errors, it’s not a big deal.
Would you buy such a book?
NFC Smart Lock Review: Battery Woes and Unexpected Vendor Response | March 13 2026, 18:49
At the beginning of the year, I bought an NFC smart lock for the front door for 170 bucks. Recently, I wrote a review on Amazon stating that the batteries lasted only a month and a half, and if it continues like this, I will end up paying almost the same amount annually. The manufacturer has responded saying they will refund the money. They didn’t ask to remove the review, and I don’t even know if that’s possible.
Unlocking the Mystery: Dual Voltage Needs in Smart Locks | March 07 2026, 22:43
Update: figured it out, looks like the lock needs 6v + 6v for different purposes. Maybe the power part and electronics.
Anyone who knows electronics, help me understand. Red wires are connected to contacts that respond to the tester. A total of 8 batteries. I can’t see a classic snake configuration here. Can’t understand why the lower right ones are responding. I want to connect an external adapter
Hi, I'm Rauf Aliev. 