If you know anything about astronomy, you have probably heard about something called [the heat death of the universe](https://en.wikipedia.org/wiki/Heat_death_of_the_universe). In a nutshell, it hypothesizes that everything in the universe (stars, planets, nebulae, etc.) becomes so far apart that it all just cools down to a point where equilibrium is reached, and nothing new would be created. This isn't a particularly exciting way for the universe to go out. Perhaps a much more eventful fate for the universe is the [big crunch](https://en.wikipedia.org/wiki/Big_Crunch), which hypothesizes that the observed expansion reverses and everything starts moving towards each other, until it all meets in one big crunch. According to Wikipedia, the big crunch isn't very likely to happen. But only time will tell. Regardless, both ideas lend themselves nicely to interesting internet analogies.

## The Heat Death of the Internet

The article linked in this submission, titled [The Heat Death of the Internet](https://www.takahe.org.nz/heat-death-of-the-internet/), applies the idea of the heat death of the universe to the internet. The author, Gregory Bennett, lists a series of internet situations that convey this analogy, such as

> You want to order from a local restaurant, but you need to download a third-party delivery app, even though you plan to pick it up yourself. The prices and menu on the app are different to what you saw in the window. When you download a second app the prices are different again. You ring the restaurant directly and it says the number is no longer in service. You go to the restaurant and order in person. You mention that their website has the wrong number and the woman behind the counter says they have to contact the company who designed the site for changes, which will cost them, but most people just order through an app anyway.

and 

> You want to watch the trailer for an upcoming movie on YouTube but you first have to sit through an ad. Then you sit through a preview for the trailer itself. Then you watch the trailer, which is literally another ad. When it ends, it cues up a new trailer, with a new ad at the start of it.

and 

> The first page of Google results are links to pages that have scraped other pages for information from other pages that have been scraped for information. All the sources seem to link back to one another. There is no origin. The photos on the page look weird. The hands are disfigured. There is no image credit.

I think that Bennett, in essence, was attempting to emphasize that the internet is becoming so fragmented with different services, polluted with advertisements, and confusing to navigate that any user of the internet can no longer easily make progress towards their goal. This analogy between the universe and the internet fits rather nicely. An internet user's ability to make progress towards their goal could be considered analogous to the universe's processes that increase entropy (e.g., star formation). And both the internet and the universe are showing no signs of slowing down.

## The Big Crunch of the Internet

But maybe the internet has other forces at play. In one of the examples, Bennett references ChatGPT:

> Your coworker sends you a PowerPoint pack to support a presentation you are giving to the executive committee, but you can’t make heads or tails of it. You call them over Zoom and they tell you they used ChatGPT to write it. You point out that it is near-unreadable, and they ask what specifically is wrong with it. You mention that, for starters, there are too many words on each slide. They tell you they’ll take care of it. They send you a new pack within the hour saying they asked ChatGPT to remove 30% of the text. It makes even less sense. You tell them you’ll just rewrite it yourself.

And this got me thinking - ChatGPT (or more generally, any of the popular, generative large language models) is trained from years and years of historical internet data: terabytes of everything that we've ever written, published, and shared on the internet, given to the model to "learn" in its model weights. Speculating on these model sizes is difficult due to their proprietary nature, but it is probably safe to say that the sizes of these models (hundreds of GB) are much smaller than the sizes of their training data (tens of TBs).

Sticking with our universe-based analogies, this could be thought of as the "big crunch" of the internet: terabytes of internet data being compressed into a single generative large language model. 

## The Big Crunchy Heat Death? 
The universe doesn't fair too well in either case of heat death or big crunch. Luckily for the universe, only one outcome can happen.

But for the internet, we seem to be seeing both heat death and big crunch happening at once. And to make matters worse, the big crunch's language models are *generative*, meaning that the output of these language models can be put back on the internet. This is what I think Bennett was getting at in the quoted example with ChatGPT. Not only does this referential loop [pollute the internet](https://boehs.org/node/llms-destroying-internet), but it may even make language models [less useful over time](https://arxiv.org/pdf/2305.17493). 

Does this mean that the internet is doomed in a big crunchy heat death? My guess is that probably not. Intuition tells me that progress in these types of areas tend to be a series of actions and reactions, and that it is only a matter of time before new innovations come along that help mitigate the "crunches" and "heat deaths". Of course, I don't have a clue of *what* or *when*, but fortunately for us, the need is high, and [necessity is the mother of invention](https://en.wikipedia.org/wiki/Necessity_is_the_mother_of_invention).







Submitted

by

Mentions