New feature: URL-based deduplication

What is the change?

Lighthouse now deduplicates items if the have the same URL. Before the change, when multiple sources published the same content, multiple items were added.

Now, items are added only once, and show which sources added it when.

Note here that Hacker News Frontpage published that website twice, once in 2024 and once two days ago, and Tildes also published it yesterday.

When does this occur?

There are publications which link to the content of other websites. Link communities are a prime example, like Hacker News and Tildes.

These communities may publish the same content multiple times, like in the second example. Another case is if you’re subscribed to a source, and the link community also publishes the same URL, like in the first example.

What is the behavior?

In general, if you already have an item with the same URL in Lighthouse, and an additional source publishes it, the source is added.

If the source that just published the article has tags, then these tags will also be applied to the (existing) item.

To make sure there are no disruptions to your workflow, if the existing item is in the library, it will stay in the library. And if it’s archived it will move back to the inbox, to ensure you see it again.

Future improvements

There are also link digest newsletters and publications. They curate content from a wide array of sources and send out a selected shortlist of articles.

Currently you have to go through these posts and bookmark the relevant articles yourself. In the future, Lighthouse will provide a rule that extracts the mentioned articles and puts them into your inbox.

Since it often happens that multiple link digests publish the same articles, URL-based deduplication will help reducing the item count by showing each one only once.