Google News RSS Scraping - OpenDemocracyManitoba/winnipegelection GitHub Wiki

Old ManitobaElection.ca Search URL String

'http://news.google.ca/news?pz=1&cf=all&ned=ca&hl=en&as_maxm=3&q=' +
'"Candidate Name" location:winnipeg' +
'&as_qdr=a&as_drrb=q&as_mind=25&as_minm=2&cf=all&as_maxd=27&scoring=r&output=rss&num=50'

GET Parameters Used

pz=1 - Unknown

cf=all - Retrieve all content (deprecated) Was in the url twice.

ned=ca - restrict to Canadian edition

hl=en - language is english

q= - the actual search query. We append 'location:winnipeg' to all queries to search source location of news.

as_drrb=q - Retrieves articles added at any time when used with as_qdr=a.

scoring=r - sorts search results. Default: "r". Alias: "as_scoring"

r by relevance
n by date (newest first) (We use to use n. It stopped working.)
d by date (newest first) with duplicates
o by date (oldest first)

output=rss - atom also possible

num=50 - Number of results 1 to 100.

To be removed as they only work with as_drrb=b:

as_maxm=3 - Only the last three months (1 - 12)
as_mind=25 - Minimum day
as_minm=2 - Max month
as_maxd=27 - Max day

New Simplified Search Query

'http://news.google.ca/news?ned=ca&hl=en&as_drrb=q&as_qdr=a&as_nloc=winnipeg' +
'&scoring=r&output=rss&num=50&q="Candidate Name"'

New GET Parameters Used

ned=ca - restrict to Canadian edition

hl=en - language is english

q= - the actual search query. We append 'location:manitoba' to all queries.

as_drrb=q - Retrieves articles added at any time when used with as_qdr=a.

scoring=r - sorts search results. Default: "r" which is by relevance.

output=rss - atom also possible

num=50 - Number of results 1 to 100.

as_nloc=winnipeg - Restrict the source location of the news. Equivalent to appending location:winnipeg to the q vaule.

Candidates with multiple names.

If a candidate has more than one name used by the media we can use the boolean OR operator in the q search value. This will mean adding another column to the people name.

Restricting Source Location Doesn't Work

Restricting source location using as_nloc or location:winnipeg does not work anymore in RSS based results, leading to lots of false positives.

There is also a geo=winnipeg query option, but it may result in the loss of some valid articles. More research is required to see which valid articles the geo flag is dropping.

Geo Parameter Investigation

I ran a query for Harvey Smith with and without the query parameter "geo" being set to "winnipeg".

Note that the "geo" query has no false positives, and includes five articles missed by the "no-geo" query. Even though the "no-geo" query was full of false positives it actually found five valid articles missed by the "geo" query!

Looks like the geo results are only Freep and Wpg Sun. The non-geo query has results from the Metro, ChrisD and CTV in addition to Freep and Sun.

Ruby script and RSS files used to investigate.

Search for "Harvey Smith" with `geo=winnipeg`

Link to query.

Audit aftermath - Winnipeg Free Press
Consultants defend report under grilling - Winnipeg Free Press
City to province: review audit - Winnipeg Free Press
Katz's plan to revamp EPC passes - Winnipeg Sun
Gloves off over city hall audit - Winnipeg Free Press
Candidates register to run for city council - Winnipeg Free Press
A change gonna come to Winnipeg city council - Winnipeg Sun
Council votes no to rapid transit referendum - Winnipeg Sun
Council votes to reform EPC selection - Winnipeg Free Press

Only in `geo=winnipeg` Results

Consultants defend report under grilling - Winnipeg Free Press
City to province: review audit - Winnipeg Free Press
Katz's plan to revamp EPC passes - Winnipeg Sun
Gloves off over city hall audit - Winnipeg Free Press
Candidates register to run for city council - Winnipeg Free Press

Search for "Harvey Smith" without `geo=winnipeg`