Tag: Data

Revolutionizing Research: Introducing a Web-Based Notebook Integrated with AI and PDF Support | February 19 2026, 16:19

I’ve further developed a new tool for myself for working with information and organizing it. The main idea is a web-based notebook for research, studying subjects, working on them, integrated with AI and PDF support.

The main problem with typical PDF readers and notes is that the context is lost as soon as you switch to a new tab. In my tool, each text fragment or PDF becomes a node in a “live” hypertext tree, which I can access from multiple computers at any time.

Work process:

– Contextual AI. I can ask the AI to clarify complex passages right within the document. The explanation stays right where the question was asked. Moreover, it is a separate document, linked to the specific spot in the source. When clicked, you see both the original and the explanation on the screen at the same time.

– Panels instead of windows. If the explanation itself requires clarification, a new panel opens to the right. This allows for an endless chain of queries, never losing the place in the original text. That is, you see several panels at once, and unnecessary ones can be closed.

– PDF support. I can upload a PDF, select an area on the page (e.g., a complex diagram or a list of authors), and the LLM instantly extracts data, supplements, or explains them. The explanation is attached to the spot where it was requested, just like with non-PDFs.

– Nested annotations. My comments are not just static text. They can contain their own PDFs, links, and further sub-tasks for AI, maintaining a depth of nesting that reflects how we actually think.

This is not just a file storage system, but an “engine” for building knowledge.

The tool suits me personally very well, but perhaps it only solves my specific tasks. What do you think, would something like this be useful to others? Would it be useful to you? Should I develop the project into a fully-fledged product and give it to other users for testing?

Interactive Text Enhancer: A Tool for Embedding Clarifications | February 12 2026, 16:11

I whipped up this thing in just an hour. Do you think anyone besides me needs it?

Here’s the idea. Take any text – a Wikipedia article, for example. Highlight any segment, say something unclear. The LLM gives us an explanation, and instantly inserts a box right in the text which you can click to open the explanation. In this explanation, there might be something unclear too. We highlight it with the mouse from this explanation, and a box appears there too. This continues until everything is clear. All the boxes remain in the text, so you can always return to them. So, if the idea was unclear to me, maybe it will be to others, and then a ready link with explanations will come in very handy. The result can be shared with colleagues.

For explanations, not just the fragment is used, but also the context. For example, otherwise, the highlighted word Terrier would yield text about a dog breed, not about the search system.

Navigating the Future: Embracing Earth’s Magnetic Field as a GPS Alternative | January 10 2026, 17:41

I learned today that there is and is actively used a technology for navigation using the Earth’s magnetic field. It is used as a replacement or an extension of GPS.

For example, there is the Scandinavian ferry Express 5 of Bornholmslinjen, which insures against GPS problems (which do happen) by using MagNav navigation. Unlike GPS, the Earth’s magnetic field cannot be jammed or spoofed—it simply exists. The ferry follows the same route, and generally, navigation could even be achieved through household fishing sonars.

But there are a few startups that use this technology for indoor navigation, where GPS signals cannot reach. It’s claimed that the navigation accuracy is within 1 meter. That’s more interesting.

GiPStech, Oriient, Mapsted.

The basis of this technology is a process called magnetic fingerprinting. Engineers or mapping robots walk through a building with a smartphone, recording unique distortions of the magnetic field at every point. These distortions are created by the steel frame of the building, rebar in the walls, and large electrical equipment. A database is formed where each coordinate (x, y, z) corresponds to its unique magnetic field vector (intensity, inclination, deviation).

The collected data is uploaded to the cloud platform of the provider company. There, they undergo noise cleaning and are “stitched” together with the digital floor plan. When a user walks through a shopping center, their smartphone reads data from the built-in magnetometer in real-time. Special software (SDK) compares the current readings with those stored in the database. For accuracy to be within 1–2 meters, the system relies not only on magnets. It uses sensor fusion—combining data from the magnetic field with inertial sensors (accelerometer counts steps, gyroscope determines turns) and sometimes Wi-Fi/Bluetooth signals for rough localization.

This technology is certainly being actively implemented for drones. The main technical difficulty there is dealing with their own interference and considering that the magnetic field changes, requiring constant map updates. Electrics, engines create strong magnetic fields, which “drown out” the natural background of the Earth. However, various filtering algorithms (including neural networks) are used, which in real-time “subtract” motor interference from the overall sensor readings. From what I understand, at high altitudes (kilometers), the magnetic field is more “smooth”, therefore the accuracy is lower (about 1–5 km). But if several drones fly together and exchange signals, overall they can provide very good accuracy each. Additionally, a group of drones can measure the gradient (rate of change) of the magnetic field in space, tying location not to absolute values, but to relative ones. Essentially, using a group of drones turns the navigation system from a set of individual receivers into a distributed phased array antenna, capable of filtering global interferences and working with much weaker useful signals. Considering that small drones capable of staying airborne for long periods can be released into the air by the hundreds (and cost pennies), this is a quite promising area for military.

There’s an interesting startup, Zerokey. They release QUANTUM RTLS 2.0. This device provides spatial accuracy to 1.5mm. It’s used in production, for example. Their video shows a “watch” on a worker’s hand that monitors the correctness of assembling something on a table. Here, the principle is ultrasonic, and it’s understandable that these “watches” are paired with stationary sensors and further multilateration.

Designing 3D Volleyball Training Tools on the Fly | January 01 2026, 21:21

What I did on the plane to/from vacation and sometimes in between: 3D visualization and editing volleyball schemes for Nadya (she’s a coach). This court in the attached image freely rotates, players can be placed on it, and the ball and player paths are shown – all in 3D.

The ball’s trajectory is calculated so that it does not cross the net when moving from A to B (Bezier formula). Players can take several poses – right now there are hastily made poses for serve, attack, block, pass/receive. Interestingly, in the code: I had to write a bit of “volleyball brains”. The system itself calculates the ball’s trajectory through Bezier curves so that it always passes over the net. Moreover, the height of the launch depends on the type of action: for an attack, the ball “launches” from a higher point than for a pass. I also added auto-rotation: the 3D model itself turns its face to where, according to the scheme, it needs to pass or run.

The longest and most difficult task was creating the 3D model of a female volleyball player. To generate a realistic volleyball player, I used the tripo3D service. It gave me a model in a neutral pose (for free). Theoretically, you can then use Blender and the Rigify plugin to attach an armature to it and move its arms and legs, which would recalculate the model.

However, in reality, this approach does not work well: the AI-generated model contains a large number of geometric errors, which the renderer forgives but Rigify does not. They can be roughly divided into two types — incorrect polygon normals and issues with non-manifold geometry, which are significantly more challenging to fix. Inside the body, there may be “floating” clusters of polygons or intersecting surfaces. When Rigify tries to calculate weights (which bone affects which part of the skin), this internal noise confuses the algorithm, and as a result, the weights are distributed chaotically (for example, moving the arm might start pulling the mesh on the stomach). Plus, the model is slightly asymmetrical.

Non-manifold is a geometry error where the topology of an object ceases to be correct in terms of a three-dimensional body: edges may belong to more than two polygons, polygons may only touch at vertices or edges without a common volume, and “hanging” surfaces or zero thickness may appear inside the model. Such geometry formally does not describe a closed volume, causing problems with rigging and deformations. Moreover, the model needs to be simplified because millions of polygons are not needed for rendering in real-time in a browser.

I fixed these using MashLab, additionally refining by hand (“with a file”). In the end, the model turns out slightly different from the original almost everywhere. The original model had “skin” in the form of textures – the face, shirt, and shorts had to be colored. How to transfer all this to a simplified model? For that, there’s a special operation in Blender called Baking. This also involves some tricks. In the end, it didn’t transfer perfectly, but perfection isn’t necessary yet.

Next, we attach the armature to the “joints”, and after about three hours of figuring out why everything does not work as it should, it finally worked. I made four poses, and now each circle (player) can be told which pose it is in.

I’ll also need to make dynamic changes to the uniform colors – that shouldn’t be difficult. There’s also an idea to transfer poses from photographs – this is more complicated, but generally feasible. Using MediaPipe/AlphaPose, you can detect key points in 2D, then some models like HMR/HybrIK can “lift” flat coordinates into 3D space, outputting relative joint rotation angles. The resulting data can be attempted to be projected onto a Rigify skeleton. Since the proportions of the generated volleyball player and the person in the photo may not match, that’s exactly why Inverse Kinematics (IK) is used. This part is quite complex, but overall it’s not strictly necessary – just interesting to figure out and make something functional.

Video in the comments

Decoding the Beast: Migrating from Excel to Code | December 17 2025, 18:56

We’ve all encountered it — the “Main Excel Spreadsheet Managing the Business.” The very one B2B companies use to calculate million dollar quotes. It has 12 tabs, 1000+ nested formulas, and zero documentation. For ten years, it had “quick fixes” slapped on and constants hidden away. It’s no longer just a file, but a living organism that no one fully understands except for the guy who quit years ago. That’s how puzzled I was. Moreover, there was uncertainty whether even half of the formulas were needed, or if they were vestiges of the past.

Typical cell:

=IF($D11=$D10,””, IF(ISNUMBER( INDEX(Data!$T$10:$U$17,

MATCH(TabCalc!$F11,Data!$T$10:$T$17,0),2)),

INDEX(Data!$T$10:$U$17, MATCH(TabCalc!$F11,Data!$T$10:$T$17,0),2),

INDEX(TabProd!$C$8:$U$112,TabCalc!$D11,I$1)))

I was tasked with transferring this logic into code so that it was all computed by software. The Excel file seemed to have everything it needed, but in reality — it was a complicated black box. 1069 formulas.

The challenge was in how to translate a thousand interdependent formulas into clean code without losing any edge cases.

Here’s what I ended up doing.

Instead of rewriting everything from scratch at once with uncertain prospects of bug proliferation, I used a strategy of lazy computations and mocks.

I built a structure on Groovy that mimicked Excel’s behavior. Each computation (from a cell) I defined as a function that executed only when it was called. And the functions were a multidimensional dictionary.

I started from the end of the computation graph: from results to inputs. If a formula depended on something I hadn’t yet written, I “mocked” it in the code, simply substituting the value from the Excel sheet.

Bit by bit I replaced these mocks with real logic. Comparing the output of my code to the Excel at each step, I could clearly see where my logic diverged.

In other words, I moved from the result to the input data. At each step, it was clear which mocks needed to be turned into code, and I could compare version +1 with version -1 — the result had to match. As soon as all mocks were replaced with calls — the task was done.

The real “secret ingredient” was the dynamic nature of Groovy for creating a multidimensional map of functions. Instead of static variables, I used a deeply nested structure, where each “leaf” was a closure. This allowed access to any part of the table — be it an input parameter, a config constant, or a complex intermediate result — through a simple, unified syntax, and some components were dynamic.

Here’s an example:

conf[“group”] = { x -> [“a”, “b”, “c”] }

conf[“group”]().each {

calculate[“Group”][“Subgroup”][it][“TotalQuantity”] =

{

x -> calculate[“Group”][“Subgroup”][it][“Someparameter”]() * conf[“someConstant”]()

}

}

Using dynamic keys and closures, I could iterate through product groups or data sets. Since these were dynamic functions, not stored values, the entire system worked like a living graph of dependencies.

Testing was possible right from the start of transferring the formulas. The charm was that you were kind of addressing a cell through syntax like calculate[“Totals”][“A”](), but in reality, you were launching an entire tree of calculations at that moment. And this was incredibly convenient for debugging.

In two weeks, the “Black Box” was transformed into a transparent, modular library with clear logic, which produced exactly the same result as the original table.

P.S. Of course, all the data in all the screenshots are thoroughly obfuscated, or rather, written from scratch for this text.

Decoding Complex Queries: A Transformative Approach to Search Functionality | December 17 2025, 03:25

Oh, I just solved a really cool problem. It’s tricky to explain though. But I’ll try.

So, the client has 10 search websites. They all use one index but throw different queries at it. To what the user enters, a very long and complex query is added, generated by a module on Sitecore. It includes template and page IDs that need to be included or excluded. Ultimately, it’s impossible to understand what’s going on there. There could be ten opening brackets and some randomly closing ones, but it worked with Coveo. Reformatting helped, but not much.

And each site has its own version of this. Meanwhile, the same IDs appear periodically. I first tried to manually figure this out, but it was a nightmare. Nothing helped. There are also nested conditions. For example, “exclude this template” not globally, but only if that field equals one.

Here’s what I did:

I wrote a script that parses this textual “mess” into an abstract syntax tree (AST). This allowed to turn an unreadable string into a structured JSON object, where it’s clear: here’s AND, there’s OR, and here — a specific condition.

Then I turned these conditions into Boolean algebra formulas. Using the SymPy library, I “fed” these formulas to simplification algorithms. Mathematics itself eliminated duplicates, collapsed excessive nesting, and removed conditions that are logically absorbed by others. As a result, the “trees” became flat and understandable.

In the attachment — the original tree and the simplified one.

To be sure that I didn’t break anything during simplification, I wrote a test generator. It takes the simplified logic, puts it back into a working curl, and checks whether the number of found documents (totalCount) matches the original request. The numbers matched — meaning, the logic is preserved 100%.

Having simplified and standardized structures for each site in hand, I built a comparison matrix. The script analyzed them and highlighted Common Core — conditions that are guaranteed to be required (or prohibited) on all sites without exception, and Specifics — unique “tails” that distinguish one site from another.

In the attached screenshot: REQ means that the condition is guaranteed to be met for any document that goes through this request. NOT — definitely not met. OPT — the condition is present in the request, but it’s not strict by itself. It only works in conjunction with something else. “.” — the condition is not mentioned in the request at all.

For 3 sites it responds instantly, for 10 it takes about 30 minutes.

And of course, all data in all screenshots are thoroughly obfuscated.

Exploring Open Data: A Deep Dive into Loudoun County’s 1.5 Million Trees | December 15 2025, 15:40

I’m checking out what open data we have in our county to play with data analysis over the weekend, and discovered, for instance, an open database of all 1.5 million trees in the county. The screenshot shows just a tiny part around my house.

Modern Reading: More Words, Digital Shifts, and Surprising Data Insights from 2008 | December 14 2025, 22:33

An interesting study caught my eye, dating back to 2009. According to it, the modern human indeed reads significantly more than in the past, although the format of this reading has changed. The study suggests that in 2008, an average American consumed about 100,000 words a day (approximately a quarter of “War and Peace”) – this is an approximate number of words that passed through consciousness per day (via ears or eyes), calculated based on activity chronometry. This is 140% more than in 1980.

Therefore, contrary to the myth about the degradation of reading, at least in 2008, we processed 2.4 times more textual information than our parents’ generation. Moreover, the study only considered information consumed outside of work (at home, in transit, during leisure).

The structure of reading – if in 1960, 26% of words came from paper, by 2008 this share had fallen to 9%. However, digital media (internet, email, social networks) not only compensated for this decline but also tripled the total reading time. The reason — the internet, as it is predominantly a textual environment (web surfing, email).

But it’s interesting that although the Internet accounts for 25% of consumed words, it only makes up for 2% of bytes (since video on the internet in 2008 was of low quality). Thus, they estimated the information flow from different channels and converted it into bytes 🙂 Radio accounted for 19% of the time but only generated 0.3% of bytes (as audio requires less data). Voice communication (telephone) — accounted for only 5% of words and a negligible share of bytes, but it was the only fully interactive channel before the internet era. TV remained the main source of information by time in 2008 (41% of all hours) and quantity of words (45%), however, in terms of data volume (bytes), television was only second (35%), behind computer games.

Now about games, quite interesting. The main finding from the report: Games generated (or did in 2008) 55% of all “bytes” consumed by households. Meanwhile, they only accounted for 8% of user time. This is quite a controversial topic in their report.

Those 100,500 words — that’s an assessment of actual words that a person either read or heard. This is not a metaphorical “equivalent,” but an attempt to calculate the verbal information precisely. They took the consumption time of each media and multiplied it by the average word inflow rate for that channel. Reading (books, newspapers, internet texts): 240 words per minute. Email and web surfing – 240 words per minute. Television (dialogues in shows/movies): 153 words per minute. Radio: 80 words per minute (less because of many pauses and music). Music: 41 words per minute (song lyrics).

Link in the comments

Living in the Valley of Data Centers: The NSA’s Gigantic Utah Facility | September 20 2025, 20:06

I live right in the valley of data centers, like 80% of internet traffic goes through us (a dangerous place!). I drove by one of them today, and later at home, while Googling stuff about data centers, stumbled upon the NSA’s data center in Bluffdale, Utah.

It serves as a data repository for the U.S. intelligence community. Capacity — something like 5 trillion terabytes. 5,000,000,000,000,000 gigabytes. Back in 2013, it was 100-1000 times less, but 12 years have passed, Moore’s Law and all that. Hard drives in data centers usually have a lifespan of 3-5 years. Meaning, since the launch of the data center, they have all been replaced several times with obviously greater capacity.

It is expected that the data center will be able to process “all types of communications, including the full content of private emails, mobile phone conversations, internet browsing, as well as all types of personal data: parking receipts, travel itineraries, purchases in bookstores, and data of other transactions made using digital technologies.”

The amount of data this facility is able to store is, of course, classified, but estimates “several yottabytes”. Yottabyte = 1000 zettabytes = 1,000,000 exabytes = 1 trillion terabytes. To store all the books ever written in any language would require just 400 terabytes.

In 2013 it consumed no less than 65 MW with a potential of 100 MW. Water — ~1.5–1.7 million gallons (5.7–6.4 million liters) per day for cooling servers. The water is treated with chemicals (to prevent corrosion) and discharged, leading to criticism in arid Utah — especially amid the record heat from 2022–2025 and the shortage of fresh water. There’s no closed-loop system, and it remains a “hot” topic in local discussions.