Eigen Hacks: what one threshold was hiding

Eigen Hacks is Eigen Times pointed at a different archive. Same mathematics — each day's stories as coordinates in the eigenbasis of everything the archive has seen — but where Eigen Times reads 16.5 million articles over 175 years of newspapers, Eigen Hacks reads 4,763,697 Hacker News submissions since 2006, with their scores, and the text of the pages the best of them link to.

The two are sister papers, cross-linked from each other's masthead, and the contrast is much of the point: what a basis considers a recurring shape of news differs sharply when the archive is a newspaper of record and when it is a community of programmers voting on links.

This post is mostly about a mistake. The corpus, the fit and the site went up in a day; the interesting part was the twelve hours spent discovering that the paper it produced was quietly nonsense, and why the ordinary signals of success all said otherwise.

The register problem

Hacker News titles carry a register. Show HN, Ask HN, Tell HN, the trailing (2019) on a resurfaced article, [pdf] and [video]. These are labels about how to read a post, not about what the post concerns.

The first fit, hn1, embedded titles as they stood, and the register dominated. On an ordinary day it clustered roughly 520 Show HN posts into a single story — a "story" whose members had nothing in common but their prefix. The fix seemed obvious: strip the register before embedding, and while we are at it, crawl the pages the highest-voted stories link to, so a bare headline is measured together with the article it points at. That is hn2: register-free titles plus 183,810 crawled article texts.

Ten hours of embedding later, 240 monthly partitions, both bases fitted, 4.76 million stories clustered, a site exported and deployed. Every page returned 200. The axis labels read plausibly — code · review · write, languag · model · chang, generat · hack · site. Nothing looked wrong.

Counting the clusters

The one number worth checking was whether the register fix had worked, so it needed hn1's cluster sizes to compare against. The comparison, over the 92 days of summer 2026:

The register stripping had worked perfectly and changed nothing. No Show HN post headed a giant cluster any more; a giant cluster was still there, wearing a different hat. The register was a symptom.

Here is one day, 21 July 2026, under hn2:

591 stories drawn from 1,256 articles
  553 singletons · 28 pairs · 6 triples · 2 quads · 1 six
  and ONE CLUSTER OF 615

Half the day's articles in a single story, and essentially nothing between that and singletons. Three explanations would have been comforting, and all three are wrong:

• Duplicate rows? No: 603 distinct titles among the 606 unique member ids.
• A scoring artefact? No: median score 3 inside the cluster, 3 outside.
• Missing article text? No: 87% of members were title-only, against 85% outside it.

It was genuine over-merging of unrelated posts, and its shape — one giant component, a long tail of singletons, almost nothing in between — is the signature of a similarity threshold sitting at its percolation point, where transitive linking abruptly joins everything to everything.

The threshold was inherited, not chosen

Stories are formed by linking articles whose embeddings are closer than a threshold. Eigen Times uses 0.72, tuned on a newspaper corpus where many outlets cover one event. That number had been carried over to Hacker News by analogy, and never questioned.

Sweeping it on a single day:

The giant component dissolves smoothly. But the table alone cannot distinguish a threshold that finds real stories from one that merely shatters everything, so the clusters have to be read:

At 0.83 the largest clusters are topics: one holds a benchmark for coding agents, a secrets manager for CLIs, a private-inference service and a parallel-session board — different stories that share a subject. At 0.86 they are events: eight submissions about a single security breach, six about one model launch, four about one court approving one settlement, several of them exact reposts of the same link.

Push higher and the opposite failure appears. The decisive measurement is what happens to a known single event — a breach covered ten times:

Above 0.86 the paper would print the same story seven times. The usable band is narrow, 0.85 to 0.86, bounded on one side by topics swallowing events and on the other by events shattering into fragments. Refitted at 0.86, the corpus yields 4,566,858 stories, of which 4,421,560 are singletons — 145,000 real clusters averaging 2.3 articles each.

A 97% singleton rate would be alarming in a newspaper. On Hacker News it is correct: most links are posted once, and a genuine cluster means a repost or several outlets covering one event.

A second inheritance

The same analogy had broken something else. A story's prominence is its energy, defined as the number of distinct sources times the log of the article count. On a newspaper corpus that is a good measure: an event covered by the Guardian, Wikinews and the Times is bigger than one covered once.

Hacker News has exactly one source. distinct_sources is always 1, energy collapses to cluster size, and since 97% of stories are singletons, almost every story on the front page has identical energy. The lead story of the day was an arbitrary tie-break among thousands — a 1,600-point thread and a 1-point submission ranked the same.

The signal was there the whole time and unused: points, and the comment count, which the adapter had been carrying as a tag since it was written and which nothing ever read back. Energy is now

sources · ln(1 + articles) · ln(e + engagement)

with engagement the points and comments summed over the members. Where neither is reported the last factor is exactly ln(e) = 1, so every newspaper source keeps its previous energy to the bit — a property worth a test rather than a comment.

Comments may be the better half of that signal. Points say a thing was approved of; comments say it was argued about, and the two are not the same story.

What is not yet known

Two questions this work opened and did not close.

Does the crawled article text earn its cost? It was invisible under a broken threshold — the title-only share inside the degenerate cluster was indistinguishable from outside it. Only 232,995 of 4.76 million stories carry full text, so for 95% of the corpus hn2 is only the register fix. The crawl is the most expensive stage of the pipeline and deserves a measurement now that one is meaningful.

What is a Hacker News story, actually? Arguably not the link: the argument underneath it. 1,576,563 submissions carry at least one comment, 40.8 million comments in all, and none of them were in the corpus — the adapter recorded how many there were and threw away what they said. That crawl is now running, into its own table rather than merged into the stories, so that a discussion can be treated as a separate document, or concatenated with its story, or weighted by its own scores, without a week of re-crawling to change one's mind.

The moral

Nothing about the broken paper looked broken. The site was up, the pages rendered, the axis labels were sensible, the exit codes were zero, and half of every issue was one fabricated story.

The only thing that caught it was counting the sizes of the things it produced, and the only thing that fixed it was reading the contents of a dozen clusters by hand. Both bases, both thresholds, both energy definitions had been inherited by analogy from a corpus with different properties — many sources per event, few reposts, no register. Every one of those inheritances was wrong, and none of them announced itself.

Eigen Hacks is live at eigenhacks.com, rebuilt daily.

Read more

• The sister paper and the method: Eigen Times: a newspaper in the eigenbasis of the news on this blog, and the second edition, Eigen Times v2: 175 years of news without drowning the present.
• The books, on the First Pair library shelf: the paper Eigen Times: A Newspaper in the Eigenbasis of the News (design, dataset, the two bases, evaluation, engineering) and its illustrated companion The Mathematics of Eigen Times, whose chapter on the clustering thresholds is where the figure above comes from, and whose chapter on streaming explains how the fits run in flat memory.