ExpandURLsPlugin - eric011/ThinkUp GitHub Wiki

extends Plugin implements CrawlerPlugin

crawl

This function will be called through Crawler interface. Flickr and Bitly API are managed by this function, not accessible to external classes, so they have different function names for expanding URL.

implement

1. get link_limit for a crawler;
2. get flickr api key;
3. call expandOriginalURLs;
4. call acquireBitlyClickStats to get click counts;

expandOriginalURLs

for each links:

1. call expandFlickrThumbnail if the URL comes from flic.kr;
2. call expandURL;
3. save results to database;

acquireBitlyClickStats

for each bitly link:

1. call getBitlyLinkData;
2. save the result;