I wonder if there exists an agent that takes an Excel table significantly larger than the context window and begins to document its essence. Here are several tabs. Here on tab 5, there is a table with a million rows and five columns. The columns are as follows. We take random data from the table, looks like there are numbers, and there – surnames. We assume that there are numbers everywhere – we write a code that checks this assumption and at the same time calculates min/max and a set of unique values. So, few values, only five. We record it. Now we check the surnames. Yes, these are just strings, new sampling showed that they are indeed surnames. Here’s a formula. We see where it points. And so on. And this column – unclear purpose. We look at the data – these are some numbers from 0 to 1. We measure the average and the spread. We ask the user – maybe they’ll provide some comments. They did. It turned out to be a KPI issued to this user from an external system. We record it. And so on. Documentation emerges. Later, when there is documentation, one can request to perform some operations with all this, since the LLM now more or less understands the purpose of the data and their connection, and can build some hypotheses on detecting outliers and verifying them.
Tag: Data
The Crucial Role of Data Quality Oversight in Development Projects | May 06 2026, 16:07
Almost every development project features a dedicated functional testing automation team, yet surprisingly, a similar emphasis on Data Quality is rarely found. Regardless of whether data comes from external integrations, users, or is generated by the system itself, it often remains without proper control simply because no one seems to consider it important, and later they struggle with the consequences – they accumulate like a snowball. The longer such issues persist, the harder they are to resolve, eventually leading to a situation where people just resign themselves to the “irreparable” state of the database. It is much better to identify these problems at the moment they arise, while the technical debt has not yet become insurmountable, rather than later figuring out how to prevent them from causing everything to crash;
In essence, there needs to be a constant “supervisor” over all types of databases used by the system (relational, NoSQL, search indexes, or graph databases) — essentially, this is a layer of data quality checking over processes. Of course, there must be clear rules – specifically what to check and which flags to use to mark specific anomalies.
There must be a responsible party for the process (a human, not AI), who will integrate these reports into the development and support workflows. Many data integrity issues cannot just be resolved through an interface — they require the engineering team to develop scripts for mass correction and data cleansing.
Incidentally, this also transitions into the realm of anomaly detection (outlier detection). Machine learning and LLMs for identifying subtle “bad” patterns that traditional rule-based systems might miss.
What do you think about this? Are similar mechanisms implemented in your processes?
Harnessing Chat Data for Semantic Q&A Search | April 30 2026, 04:05
In one evening, I created a simple utility that extracts the Natural Language Processing chat for a year and a half – there are 65,000 messages, and converts it into question-answer pairs with semantic search available. Clicking on a search result (on the left) opens the dialogue in the chat. The messages that are responses to the question are highlighted. And at the top, the original phrasing of the question is highlighted as well.
How it works: the system assumes that people mainly reply to messages that are relatively close in the past. If several replies are made to one message, then it is likely useful and caught the interest of others in the chat. The system takes messages starting from the one many have replied to, ending with the last in the reply-to chain – and among such messages, it selects those that have at least 3 reply-tos to the original question. In essence, it cuts a piece from the chat starting with a popular question so that after the bottom cut, most likely, irrelevant content follows. Such blocks can overlap each other – for example, if someone asked a question while others were replying to something else.
So, if user A asked what the weather was like, and they received answers like “good,” “bad,” “rain,” and there were five messages without a reply-to, and then someone replied to “rain” with the question “why rain”, and five more people replied to this question, then the first question about the weather makes it into the system – the piece ends with 13 messages.
Afterwards, these pieces are summarized into question-answer pairs.
It turns out quite cool.
P.S. In the screenshot, the search query has nothing to do with the search result because I foolishly took the screenshot after I changed the query but before I hit send.
Misadventures in Keyboard Layouts: Searching for Gremlin, Finding Surprises | April 28 2026, 20:33
This is me typing the word gremlin, without switching the keyboard layout. Wanted to read about the query language for graph databases, need it for work. Google surprises, it does surprise
Dinner, Dialogues, and Decor: A Cozy Evening at an Uzbek Restaurant | April 26 2026, 21:42
Navigating the Depths of High-Dimensional Spaces | April 13 2026, 23:17
I am now working a lot with high-dimensional vectors, and some things that I hadn’t fully realized before are really starting to tickle my brain. Our 3D intuition doesn’t just not work there—it lies.
It turns out that any two random vectors in high-dimensional space are almost certainly nearly perpendicular to each other. Almost all the space is one continuous “equator”.
Much of machine learning is built on exactly this. If your embeddings suddenly show high cosine similarity (for example, 0.8 — this is not a statistical error, but a powerful signal. It’s almost impossible to randomly converge like this in a 1000-dimensional world.
In such spaces, almost all the mass of data is concentrated in an extremely thin surface layer. The “insides” of objects are mathematically empty.
This can be easily verified with such an imaginary example. Take the “skin” of a multidimensional sphere with a thickness of just 1% of the radius. The volume of the sphere is proportional to the radius raised to the power of its dimensionality.
• In three-dimensional space, the pulp (0.99 of the radius) occupies 97% of the volume, you raise 0.99 to the third power.
• In 1000D, the pulp occupies just 0.000043%.
You can understand it differently. For a point to be closer to the origin, it requires that along all axes the coordinates need to be close to the origin. If one axis has a high value, that’s it, the point has gone. If you take points randomly, the mere probability that they all at once will be below any value decreases with the growth of dimensionality, and decreases quickly.
All the “meat” of the data always ends up in the skin. Any sample in High-D is essentially a set of boundary values.
For white noise in high dimensions, the distance between the closest and the farthest neighbor becomes almost the same. The concept of “closeness” simply degrades.
CPU vs GPU: A Speed Challenge in Embedding Creation | April 11 2026, 18:08
When working with certain tasks, the difference between a CPU and a GPU is simply astounding. For example, I need to create many (millions) of embeddings, model BGE M3. Running this on my quite powerful 24-core Intel Core Ultra 9 285K processor takes 45.85 seconds to create 500 embeddings, while using an NVIDIA 5090 GPU, the same task is completed in just 0.36 seconds. It is so fast that I specifically wrote this benchmark to figure out whether my GPU is being utilized at all. The program that sends requests to TEI does it in test mode not actively enough (roughly a couple of times per second), and the GPU load graphs are practically zero.
— Testing http://localhost:8080/embed — <– CPU version
Requests completed: 500
Total time: 45.85 sec
Throughput: 10.90 req/sec
Average latency (Avg Latency): 4386.11 ms
P95 latency: 5021.88 ms
— Testing http://localhost:8090/embed — <– GPU version (NVIDIA 5090)
Requests completed: 500
Total time: 0.36 sec
Throughput: 1398.69 req/sec
Average latency (Avg Latency): 31.38 ms
P95 latency: 53.18 ms
========================================
RESULT: http://localhost:8090/embed is 99.22% faster
Navigating the Lexical Complexity of Nabokov’s “Lolita” | April 02 2026, 15:56
I’ve finished the first version of a dictionary-style book on Nabokov’s “Lolita”. The chart shows how the complexity of vocabulary is distributed across the pages of the book. The lower chart averages 25 sentences, displaying the number of complex words on the vertical axis, with colors indicating their complexity/rarity (purple – the most complex, red – less complex, yellow – even less so). But I have already removed two levels, and overall, for a foreigner, all five levels are challenging. In the book, level 3 is marked with a dashed line, level 4 with a simple frame, and level 5 with a double frame. Currently, there are 5794 words, of which 541 are fifth level, 1070 are fourth, 1883 are third, 1393 are second, and 54 are first (the simplest ones). Considering that the first version ended up being 1148 pages, the dictionary will need to be significantly streamlined by removing what can be dispensed with. This mainly pertains to the first and second levels, and some from the third and fourth. The rarity of words is calculated in three ways: through LLM, and through two lists of word frequencies in the English language corpus (300K words).
Not all words are complex. For instance, in the sentence “With the ebb of lust, an ashen sense of awfulness, abetted by the realistic drabness of a gray neuralgic day, crept over me and hummed within my temples.” someone well-acquainted with English might not know the words ebb, abet, drabness, while everything else is familiar, but lower the requirements for the reader, and the dictionary might not be very useful for such cases.
Or consider the sentence:
Homo pollex of science, with all its many sub-species and forms; the modest soldier, spic and span, quietly waiting, quietly conscious of khaki’s viatric appeal; the schoolboy wishing to go two blocks; the killer wishing to go two thousand miles; the mysterious, nervous, elderly gent, with brand-new suitcase and clipped mustache; a trio of optimistic Mexicans; the college student displaying the grime of vacational outdoor work as proudly as the name of the famous college arching across the front of his sweatshirt; the desperate lady whose battery has just died on her; the clean-cut, glossy-haired, shifty-eyed, white-faced young beasts in loud shirts and coats, vigorously, almost priapically thrusting out tense thumbs to tempt lone women or sadsack salesmen with fancy cravings.
My browser even highlights four words here.
I have definitions of words in English, German, French, and Russian. I’ve encountered the issue that different words from the text are considered complex in different languages, yet they are unified for me. So, I’ll have to mark, for example, French words in the English text separately, so they are not included in the French version, since there, the reader knows, for instance, what quel mot means.
Overall, this weekend I’ll be manually removing about half, and then I can make the cover and list it on Amazon.
Mapping Global Friendships and Rivalries: A Color-Coded Matrix Analysis | March 12 2026, 03:29
For fun, I decided to make a matrix of who is friends with whom and who is enemies with whom. For each country-country pair, I asked Gemini which of the five categories the relations fall into: “at daggers drawn” (purple), “predominantly unfriendly” (red), “neutral” (yellow), “predominantly friendly” (blue), “friends” (green). Lisa said that “neutral” should be purple. Overall, the quality of Gemini’s assessments is quite good.
Among all countries, three red lines stand out. These are countries that are on very bad terms with many others. Well, you guessed Russia right. And what is the second country? Israel? No, it’s Belarus and Venezuela.
In the top five countries that everyone is friends with and who have many friends themselves, LLM included the USA, United Kingdom, Canada, France, and Germany. There is an anti-rating – these are countries that have very bad relations (“at daggers drawn”) with many others. In this rating, Russia is in first place with 21 countries, and Israel is in second place with 18 enemies. Following them, with a significant gap, are Syria and the USA with 9 enemies each. There is also a separate Conflict Zone rating – this is the sum of red and purple. Russia, Venezuela, Belarus, Israel, USA, Iran, Ukraine.
There is a “pacifists’ club”. These are the ones who have no enemies at all, sorted by the number of friends. Rating: Bahamas, Vatican, Luxembourg, Angola, Singapore, Iceland, Jamaica, Tanzania, Zambia.
I was curious, what if I apply the formula: the enemy of my enemy is my friend? What would change? This led to new colors on the matrix – logic friends.
The most unexpected leader of the Master Pragmatists ranking was Taiwan (25 logical connections). Why so? In the logic of LLM, Taiwan is a country that is officially recognized by few, but because of its global opposition to China, it automatically becomes a “logical friend” for everyone who has strained relations with Beijing. This is confirmed in the Shadow Bridges section: Taiwan has 23 connections beyond its region. It literally “stitches” different parts of the world together through a common problem.
The report “Secret Partners” – a list of geopolitical oxymorons. These are pairs that are “at daggers drawn” in official news but are forced to be friends by Gemini’s calculation. For example, Afghanistan – USA/United Kingdom. Despite the status “rather bad relations”, Gemini’s logic sees them as “logical friends”. Possibly due to common regional threats (like ISIS) or dependence on humanitarian and back channels. Or here’s a strange alliance “Belarus — Hungary”. Nominal — different camps, factually — similar style of rhetoric and common “enemies” in Brussels. Eritrea — Ethiopia: Status “at daggers drawn”, but at the same time, they became logical friends.
In the report “Most Controversial,” the first places are taken by the USA, and then with a significant gap, Russia, and even larger – United Kingdom, Canada, Ukraine. These are countries with the highest Love x Hate product value. That is, countries that have many friends and enemies at the same time.
Another report – the indifferent ones. About them, LLM couldn’t say much, apparently because they bother no one (both literally and figuratively). There are, for example, Madagascar and Haiti.
I also tried to cluster by the strength of friends and got four groups of countries.
The largest cluster. Core: China, Russia, Iran, India, and BRICS+ countries, as well as almost the entire African continent (from Egypt to South Africa) and a significant part of the Middle East (UAE, Saudi Arabia, Qatar).
The second cluster mainly included European countries. Core: France, Germany, United Kingdom. The algorithm determined Ukraine and Israel to be here. Logically: their survival depends on “predominantly friendly relations” with the European core. In this same club are Armenia, Georgia, and Serbia. Apparently, despite all the political swings, Gemini considers their ties to Europe more fundamental than any others.
The third cluster included the USA, Canada, Brazil, Mexico, and, for example, Taiwan. Officially, it can be a “logical friend” to all of China’s enemies, but by “strength of friends,” it is permanently sewn to the American block. The Vatican also ended up here, which makes this club not only economic but also somewhat “values-based.”
The fourth cluster, the most compact and specialized, included countries of Oceania and Southeast Asia. Leaders: Australia, Japan, New Zealand, Singapore. This turned out to be a club of countries trying to balance in the most complex region of the planet. Here are almost all island states (Fiji, Samoa, Tonga).
What else could we extract from this information?
Seeking Alpha Testers for a Revolutionary Text and PDF Management Tool | March 03 2026, 03:02
Looking for alpha-testers. As part of R&D and for my own tasks, I wrote a productivity tool (I actually wrote about this in my last post, but Facebook said that because I put a link in the post, only 12% saw it). Now I want to check if it will be useful to anyone else. If the idea resonates with you — let me know, and I will share access.
Website smartfolio dot me. What’s the main idea?
It’s an online notebook for working with text and PDFs, organized as a graph. It looks like Google Docs, but there’s an important difference: you can attach “child” documents to specific parts of the main text to expand on details or clarify concepts. These “comments” themselves are full documents and can have their own nested branches.
If there’s a fragment in the text that is unclear, you can ask the system to explain it (this will require your Google Gemini API key).
The system uses the full context of the document to generate a response.
Explanations are permanently attached to a specific place in the text.
This is super convenient when reading complex scientific articles. For instance, you can highlight the authors’ surnames in a PDF and instantly get a background on them — the information will be attached right to that fragment on the page.
Typical workflow
Upload a complex text and read it right in the app from either a mobile or a computer. As you go, add manual or AI-generated notes to important or unclear sections for future reference.
I do not store your documents, PDFs, images, or API keys on my servers. All data is stored in Turso DB (SaaS, free up to 5 GB).
Screenshots on the website’s main page best describe the project.
How to try?
To register in the app, you need an invite code. Just write me in the comments or in a private message, and I will send it.
Website smartfolio-dot-me
Hi, I'm Rauf Aliev. 