Scraping - hifiwi-fi/breadcrum.net GitHub Wiki

Twitter no longer serves up super basic tweet info when you wget it. It gives you more if you make the request as the google bot.

--user-agent="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36"