Tag: science

LLMs Struggle with Simple Word Filtering Task | May 28 2024, 13:56

I recently encountered a task that no LLM can solve. It should be super simple for an LLM, but somehow they can’t manage it.

There’s a list of about 1000 words. I need to keep only the most functional words from it, like which, should, would, etc.

Request: I have a list of words: …. Select only 50 words from this list that are primarily functional and carry minimal meaning in the context of keyword searches (for example, which generate significant noise in the case of partial matches). Example – which, shall, very. Do not add any words not present on the list above. The resulting experienced list should contain only words, one word per line.

ChatGPT-4o: started outputting some words alphabetically, ending at the word asking. Thus, it did not even go past asking.

Google Gemini: began inventing words not in the list, despite clear instructions not to do so.

Google Gemini Pro: produced something, but again, invented words that weren’t on the list. Almost half invented.

Anthropic Claude also listed words alphabetically, and stopped at words starting with the letter d.

Mistral 8x7B Instruct also made up half.

In fact, no LLM has managed the task. And it’s about words, not mathematics.

https://pastebin.com/5B8w96au

Programming Quirks: Moon Phases and Easter Eggs in Code | May 26 2024, 11:19

Two fun stories about the daily life of programmers.

The first one:

Researchers (@maciejwolczyk, @CupiaBart) trained a neural network to play NetHack, an old role-playing game where everything is represented by text characters. It’s a very old RPG from the days (1987) when there were no normal user interfaces, and everything happened in the console. The player goes through levels, collects items and rewards, participates in battles, and scores points — all expressed through the simplest characters.

In general, they trained it. The model consistently scored 5000 points. However, suddenly something went wrong — the model started scoring only 3000 points. That is, it showed a significantly worse result. Debugging solutions is always fun, so the thread author tried:

— to find a problem in the agent model loading code

— to roll back the code a few days ago

— to roll back the code several weeks ago (well, surely everything works there?)

— to rebuild the environment

— to change the version of CUDA (drivers for running neural networks on a video card)

— to run the code on a personal laptop, not a server

Nothing helped — the model consistently showed 3000 points.

In desperation, the author wrote to the creator of the model @JensTuyls, and received an unexpected response:

— Maybe it’s a full moon today 🌕

What?? 😑

Upon checking the lunar calendar, it turned out that indeed it was a full moon that day. The author launched the game and saw the message: “You’re lucky! It’s a full moon today.”

In NetHack, there is a mechanic that changes the gameplay during a full moon, based on the system time. The character becomes luckier, werewolves appear in their beastly guise, and dogs start howling. The model was not trained with data from the full moon, so its scores dropped to 3000 points. By changing the system time, the author confirmed that the model again scored 5000 points.

This did not make the game more difficult, but the model simply did not understand how the rules had changed and tried to play as usual — hence the drop in points. To check, you can change the time on your computer — and the model again scores 5000 points.

Moral: When faced with an unexpected error, don’t forget to check the lunar calendar.

* * *

The second story is about the ‘man’ command in the console.

This is a command that outputs documentation about what is entered as the second parameter. For example, “man ls” provides documentation on ls, which shows a list of files and subdirectories of the current directory, and “man man” provides documentation about itself.

On StackExchange, someone was wondering why their tests were failing. Answers

Marnanel Thurman:

“Uh, that’s my fault, I suggested it. Sorry. Almost the entire story is outlined in the commit. The programmer maintaining the ‘man’, a good friend of mine, and one day, six years ago, I jokingly told him that if you called ‘man’ after midnight, it should print “gimme gimme gimme”, because of the Abba song “Gimme gimme gimme a man after midnight”:

Well, he actually added it. It was fun for someone to discover this, and we mostly forgot about it until today.

I can’t speak for Cola, of course, but I never expected that it would ever cause any issues: what test would break down parsing man output with no specified page? I suppose I shouldn’t have been surprised that such a test was eventually found, but it took six years.”

Marble Statues | May 25 2024, 12:44

Today, to my surprise, I discovered that marble is not a durable material. After 150 years, signs of decay appear, and after 300-400 years, it completely deteriorates. Under the open sky, especially in places with variable humidity, it quickly gets covered with stains and crumbles. Granite and basalt, on the other hand, are a completely different matter.

Old marble statues that you see in museums are often heavily “patched” by restorers, both modern and from past centuries, plus they spent most of their history in a dry and stable climate (often artificially created, in those same museums).

I’ll also throw in another fact—apparently well-known, but relevant to the discussion. It seems that many Greek statues, probably even all of them, were painted, and what we now see is not what their contemporaries saw or how they were intended by the sculptor. I assume that we wouldn’t like the painted version now; they seem too gaudy for us. There are convincing evidences that many were painted, and it can be confidently assumed that painting was a common practice at the time, but it seems unclear whether there was a context in which the sculptures remained unpainted from the beginning, and why.

Illustration: “Ugolino and His Sons” by Jean-Baptiste Carpeaux

Surdin and Semikhatov Interview Severinov | May 19 2024, 03:42

Seems like I have a lot of posts today, sorry about that. Nice format. In Russian. Surdin (an astronomer) and Semikhatov (a physicist) are quizzing the guest (in this case, Severinov) on his topic. Since both are educated, the level of questioning is generally pretty good (with rare exceptions).

The first two-thirds are filled with a lot of interesting stuff from Severinov (mostly high school level, but still). In the last third, Semikhatov jumps in with quantum theory and totally outshines Severinov.

Listening to the rest of it now. They’re discussing Student’s t-test – it’s about statistical significance. Just a brief mention, no big deal. I decide to check out some details on Wikipedia and find something amusing:

“…This test was developed by William Gosset to assess the quality of beer at the Guinness company. Due to confidentiality obligations (the Guinness management considered the use of statistical methods a trade secret), Gosset’s article was published in 1908 in the journal ‘Biometrika’ under the pseudonym ‘Student’.“

A380 landing | May 19 2024, 02:14

We have an international airport nearby, and planes are almost always in the sky (plus, even closer by, a small airfield where small private planes are constantly taking off and landing). And I’ve noticed something interesting several times—a plane virtually hanging in the air. Just driving along, you look up—at some point in the sky, there’s a plane. Look up a few seconds later, and it’s exactly in the same spot!

Yes, it’s almost always flying almost directly towards me, so the angular velocity is not noticeable. But again, there are many planes flying, and I have something to compare it with. Encountered it again today, but I went the extra mile and checked on flightradar to see what this four-engine aircraft was.

I investigated. It was an Airbus A380—the largest airliner in the world. It’s just big, the wings are huge, and with a strong headwind, and specifically the one I saw was decelerating to 155 km/h according to flightradar24 while approaching for landing. But the following B777 also landed at a similar speed, give or take. The difference is, the B777 is significantly smaller than the A380 (30% shorter in fuselage height and nearly 20% in wingspan), and the same visual angular size for A380 appears at a higher altitude, where angular velocity is lower. Meaning, I saw the A380 when it was further away than I am used to seeing Boeings, and therefore its speed seemed very slow.

Perhaps, in addition to this, there might be a headwind. It can really drastically reduce the plane’s speed relative to the ground. Here I’m “out of my depth” and don’t know what pilots do when landing against the wind.

While I was figuring this out, I read that apparently, the A380 engages its thrust reversers (which direct the engine’s thrust forward) quite late, when the speed drops to 80 km/h. Although a fully loaded Airbus A380 weighs 573 tons, its braking system does not rely heavily on reversers. They are not a mandatory element for airliners, and on the A380, they are installed only on the two inner engines.

Although reversers help slow down the A380, their contribution to braking is not as significant as on most airliners, including the Boeing 747, where thrust reversers do significantly reduce the braking distance compared to brakes and spoilers.

It turns out, A380’s main braking effort is performed by huge Honeywell composite brakes on 16 of the 20 main landing gear wheels. The brakes on the A380 are equipped with a car-like anti-lock braking system. The aerodynamic braking, thanks to 16 large spoilers on the wings, which create drag and reduce lift, also plays an important role. Reducing lift improves mechanical braking, increasing pressure on the wheels.

The A380 slows down from cruising speed of 500 knots (926 km/h) to a complete stop within a few minutes thanks to the overall design of the plane. The descent begins at cruising altitude at about 0.85 Mach (1235 km/h). Below 10,000 feet, the speed decreases to 250 knots (463 km/h), and on the approach glide, the plane moves at 180 knots (333 km/h). The A380 crosses the landing threshold at 140 knots (259 km/h) and touches down at 130 knots (240 km/h).

By the way, here’s how beautifully the landing gear of the A380 extends:

Rediscovering Raytburt: A Look at “Physics at Half Past Nine” | May 14 2024, 22:21

A truly excellent short documentary from 53 years ago (in Russian). Highly recommended viewing. It features a bald man who perplexes a young woman with various follies. The original title is “Physics at Half Past Nine”. Tsentrnauchfilm, 1971. Scriptwriter and director: S. Raytburt. Raytburt, incidentally, received an award at the Venice International Film Festival for “The Development of Reflex Activity in Ontogeny”. It’s a shame that almost all his films are unavailable online. 

Echoes of Ancient Egyptian Innovations in Modern Science | May 10 2024, 23:23

A very interesting podcast about Ancient Egypt. https://www.youtube.com/watch?v=EDz45ELiquc

There, Viktor discusses how over thousands of years, at least two scientific disciplines—astronomy and medicine—were systematically developed in Egypt. In another segment, the host asks Viktor what he would do if he had a time machine. Viktor answered that he would like to hear how the speech of a king sounded (because we do not understand how the speech of the ancient Egyptians sounded, although we are familiar with their writing).

And here’s what I thought. Indeed, given a time machine, what could we bring back from the past that would be truly useful in the present? Obviously, 100% of Egyptian astronomy is useless to us from a scientific standpoint. But it’s a different story with medicine.

The fact is that the discovery of drugs is now largely done by trial and error (scientifically called “high-throughput screening” or HTS). HTS is a process of automated testing of a large number of chemical compounds for activity against a specific biological target (often a molecule, frequently a protein). The “hits” are then analyzed, followed by many more tests on organisms, where 99% are filtered out for various reasons. And there is hope that eventually, this will lead to a working drug.

Thus, in ancient medicine, things were simpler ethics-wise, and they immediately tested on humans. Take a poisonous mushroom and feed it to the sick in the hope of curing them. If the patient died, but (hypothetically) his hair grew, then it would make sense to give half the dose to someone who needed hair. Oh, he died too. So, next time we give a quarter.

So, it is generally useful for modern medicine to look back at ancient books.

One notable example is the development of artemisinin, a drug against malaria, inspired by an ancient Chinese text. Under the leadership of Tu Youyou, the team analyzed about 2000 ancient recipes and based on 640 selected prescriptions, they produced 380 extracts, which were tested on animals. In 1971, one of these, artemisinin, showed good antimalarial properties. It was manufactured from the plant Artemisia annua, following a recipe from the third part of Ge Hong’s book “Zhouhou Beiji Fang” (“Emergency Prescriptions at an Elbow’s Reach”), dating back to AD 340. Tu Youyou and her team isolated artemisinin from the plant, which proved to be very effective against malaria. This discovery earned Tu Youyou the Nobel Prize in Physiology or Medicine in 2015.

About Egypt, look it up. Viktor Solkin is very knowledgeable.

https://www.youtube.com/watch?v=EDz45ELiquc

Echoes of Ancient Wisdom: The Evolution of Science and Medicine in Egypt | May 10 2024, 23:23

A very interesting podcast about Ancient Egypt. https://www.youtube.com/watch?v=EDz45ELiquc

There, Viktor discusses how, over the course of thousands of years, at least two scientific disciplines — astronomy and medicine — systematically evolved in Egypt. And in another segment, the host asks Viktor what he would do if he had a time machine. Viktor answered that he would like to hear how the speech of a king sounded (because we do not know what the speech of the ancient Egyptians sounded like, though we are familiar with their writing).

And it got me thinking. Indeed, given a time machine, what could we bring back from the past that would actually be useful today? Clearly, 100% of Egyptian astronomy is scientifically useless to us now. But it’s a different story with medicine.

The fact is that the discovery of drugs nowadays largely occurs through a trial-and-error method (scientifically called “high-throughput screening” or HTS). HTS is a process of automated testing of a large number of chemical compounds for activity against a specific biological target (often a molecule, frequently a protein). Subsequently, “hits” are analyzed followed by many more tests on organisms, where 99% are filtered out for various reasons, in hopes that this eventually leads to a functioning drug.

In ancient medicine, ethical considerations were simpler, and they tested directly on humans. Let’s take a poisonous mushroom and feed it to the sick in hopes of curing them. If the patient died but, hypothetically, grew hair, then that mushroom should be given, but in half the dosage, to someone who needs hair growth. Oh, he died too. Next time, we give a quarter.

Thus, it’s generally beneficial for modern medicine to look into ancient books.

One notable example is the development of artemisinin, an anti-malarial drug, inspired by an ancient Chinese text. Led by Tu Youyou, the team analyzed about 2000 ancient recipes and from 640 selected prescriptions, they produced 380 extracts tested on animals. In 1971, one of these, artemisinin, was found to have good anti-malarial properties. It was derived from sweet wormwood (Artemisia annua), based on a recipe from the third part of Ge Hong’s book “Zhouhou Beiji Fang” (“Emergency Formulae at an Elbow’s Length”), dated to AD 340. Tu Youyou and her team isolated artemisinin from the wormwood, which proved to be very effective against malaria. This discovery earned Tu Youyou the Nobel Prize in Physiology or Medicine in 2015.

And about Egypt, do watch. Viktor Solkin is very knowledgeable.

https://www.youtube.com/watch?v=EDz45ELiquc

Global Variations in Finger Counting Practices | May 06 2024, 20:55

It turns out there are differences in how people count on their fingers (there’s even a term for it — dactylonomy). It could easily give you away as a spy 🙂

In Russia and countries of the former USSR, counting to ten on fingers starts by bending the pinky of the left hand and progresses sequentially to the bent thumb of the right hand. However, when it’s necessary to visibly show a number, the hand clenches into a fist and then opens first with the index finger, followed by the middle, ring, pinky, and thumb.

But in the American convention, for example, counting starts with the index finger. The thumb is kept pressed against the palm to show numbers from 1 to 4, and only extended (along with all four fingers) to indicate the number 5.

In countries like Germany, France, Italy, and others, it seems that locals generally start counting with the thumb (=1), adding one finger for each subsequent number up to five.

In Eastern countries, such as Iran, they often begin with the pinky of the right hand, and finish with the thumb. But these are all linear systems.

In Japan, however, the finger counting system is reversed. Instead of showing numbers by raising fingers, it’s the fingers that are hidden in the hand that indicate the number. It starts with the thumb and hides the subsequent fingers in the hand, counting upwards until a closed palm shows the number five.

In China, counting up to five is more or less the same as in the West, but after six it becomes peculiar; there are special configurations indicating 7, 8, 9, 10.

The way to show “2” might lead to misunderstandings if someone accustomed to the Chinese method sees the German “two” with a thumb and index finger, because it would mean “8” in Chinese finger counting (see below). I find it very fascinating (and useful!) that you can count up to 10 in Chinese using just one hand!

When showing the number “2” in the UK the American way (with the index and middle fingers), be careful not to turn your palm inward toward yourself, as this gesture is perceived (or used to be perceived, but who knows whom you’re dealing with) as an insult in the United Kingdom, Ireland, Australia, New Zealand, and South Africa.

Discovering Free Bio Sludge Fertilizer at the Local Water Treatment Plant | May 06 2024, 18:35

Near our house, there’s a water treatment plant. It receives sewage water from all over the city. Today, I decided to stop by to see what those bags near the entrance were about. It turns out, they’re giving away free fertilizer, bio sludge. Essentially, it’s a by-product of the sewage water treatment process, and, apparently due to its by-product nature and relatively small quantities (and clearly the impossibility of scaling these volumes), they’re distributed for free. If you consider buying such fertilizer, there is a sign that suggests taking no more than 5 bags if possible. However, critics point out a slight potential toxicity of these fertilizers – apparently, they contain a slight exceedance of PFAS (polyfluoroalkyl sulfonates – chemical compounds also known as “forever chemicals” due to their durability and resistance to decomposition in the environment). But on the other hand, for fertilizing lawns, this isn’t really an issue, even if it exists.