December 2023 – Hi, I'm Rauf Aliev.

An interesting story is described in Musk’s biography. There are many tales there, but this one delighted me because it’s not about cars and rockets, but about servers, which is closer to me.

Just a couple of days before Christmas, the admins rush in saying, “one of our hosts started doubting whether Twitter would survive and reconsidered the moving conditions.” Twitter cost this host more than $100 million a year, and it was decided to move the servers to Portland, where Musk had better terms and a lower price. Apparently, the host initially agreed to give some preferential rates during the moving period, and then retracted it, fearing that the company would collapse sooner.

“We need six to nine months to move. The servers are in prod, handling traffic, everything needs to be switched over intelligently.” Musk: “Guys, you have 90 days for this. If you can’t handle it, don’t bother coming back to work.”

The admins try to explain that it’s not that simple. There are various densities of racks, electrical consumption, the place needs to be rebuilt for the servers.

Musk listened and listened, then interrupted with, “guys, you’re blowing my mind. You remember that emoji? This is exactly what’s happening. Don’t bother me with all this nonsense, just take the servers and move them to the new place.”

The admins continue to talk about the difficulties. “So, can someone go to the server room and send some photos from there?”. Mind you, this was just before Christmas, they say it’ll take a week for photos. “No, I need them tomorrow. I built those damn data centers myself, and I can tell whether or not more can be fitted in. So I ask you, have you ever been there? If you haven’t, there’s nothing to discuss.”

“All you need to do is load these damn servers onto a truck, and bring them from Sacramento to Portland. If that takes more than 30 days, it’ll definitely blow my mind. So.. Get a carrier, it takes a week to move the servers and another week to connect them. Two weeks in total. That’s how it should look. Just grab a U-Haul and go.”

The next day, December 23rd, almost on Christmas Eve, Musk with his family – wife Grimes, two-year-old son named X, two engineer brothers who were also involved in the Twitter story, suddenly decide en route to Austin over Las Vegas to move the servers right now.

The pilot turns around, Musk looks for a rental car, they find a Toyota Corolla. The child back, the rest cram in as best they can. It’s unclear if anyone is in the data center for Christmas.

There was one engineer from Twitter there, Alex-Uzbek. The data center was ultra-modern, with retina scanners and all that. Twitter’s server room looked like 5200 racks each the size of a refrigerator. “Okay, looks like they should manage it.” Okay, should manage, you think, 6200 tons of metal.

“Actually, a special contractor is needed to lift the floor – for that, special suction cups are required. Then another contractor is needed to disconnect the right cables under the floor and not disconnect the wrong ones.”

Musk pulls out a Swiss army knife and through the ventilation crawls into the space under the floor. Disconnects the servers. Done!

The next day, on Christmas Eve, there’s a buzz in San Francisco. Ross Norden hops into an Apple store and buys all the AirTags in stock for $2000, so they can track the servers en route. Then to Home Depot, where he spends another $2500 on various tools and cutting equipment. Steve Davis finds trucks, some from Boring Company, some random ones.

Fortunately, the racks are on wheels, so the team successfully loads four out of 5200 in the first hours. Everyone’s sweating, but at least it’s clear that the entire network can be loaded in days, not months, if more people and machines are connected.

The rest of the data center staff watch this and are flabbergasted.

If not to say more.

In the meantime, the management of NTT, the data center owner, learns about what’s happening and they’re flabbergasted too. And they order to end the chaos.

It turns out that the data center floors were built for a specific load and it’s not advisable to carry several heavy racks without special equipment. The entire load exceeds the permissible load fivefold.

To which Musk replies, they’re actually on wheels, so the load is distributed on four points, and we almost fit.

NTT realizes they’ve just lost $100 million in revenue for the next year.

NTT staff do everything possible to complicate their work and make them postpone it at least until after the holidays. One lady there was tailing them and being a pain. “She was the most unbearable character I’ve had to work with. But I can understand her – because of us she misses Christmas”

Moving carriers, usually recommended by NTT, often charge something like $200 an hour. James Mask found a company on Yelp, Extra Care Movers, for twenty bucks. The owner lived on the street, then had a child, and decided to start earning somehow. He didn’t have a bank account, and James paid him five-figure sums in cash. The workers didn’t have documents, so getting them into the data center was a separate problem. “I pay an extra dollar tip for each additional server.”

Interestingly, there’s a regulation that personal data must be deleted from servers before moving. By the time they learned about it, the servers had already been loaded. There was a risk that if one of the servers arrived wrong, the reputational losses would be monstrous. James sent someone to Home Depot to buy barn door locks and sent an Excel sheet with codes to Portland.

Moving 700 racks took three days. The previous data center record was thirty a month. The rest were moved in January.

“Looking back, it can be said that the closure in Sacramento was a mistake,” Musk told Isaacson. “I was told we had redundancy between our data centers. But I wasn’t told we had 70,000 hard-coded links to Sacramento. And because of that, things are still broken.”