LighthouseBot info

User agent in robots.txt: LighthouseBot

Behavior

LighthouseBot requests web feeds (RSS, Atom, JSON Feed) and the URLs of every item.

It also fetches the URLs users added manually to their account.

In all cases robots.txt is considered before the request.

robots.txt is requested before the request is made, and then cached for 24h. Within those 24h, LighthouseBot does not request robots.txt again. Consequently, changes to the robots.txt file might take up to 24h to propagate.

Contact

If there are any issues, please contact us at dominik@lighthouseapp.io.