Scripts are configured to interface with a MongoDB database. As such a local or hosted MongoDB instance is required. Script assumes localhost (default settings) as default. For custom configurations or remote server, set environmental variable MONGOURL to the URI of the MongoDB database.

export MONGOURL='[url..]

URI Format is: mongodb:// + username : password @ host : port / database

For example

export MONGOURL='mongodb://user123:[email protected]:27017/scholarly

A "source" collection is required, scholars. Optionally, create several scholars collections (to keep different groups separate, for example), following the pattern: scholars_[suffix] Then, point the script to the collection by setting the environmental variable collection_suffix Note: output collections are consolidated into one by default, even if you have multiple source collections. This prevents any double-scraping if content of collections might overlap. If desired to have separate output collections, add the collection_suffix argument to the relevant connect_mongo() functions in scrape_control.R

renv is used for package management. On first run, renv will bootstrap and prompt installation of required packages: renv::restore() NOTE: if using conda, install packages from conda instead. Some packages will not install properly from within R.

Complete one-time configuration (link needed) to define an aliased or macro command such as scrape

To run scraper, open a terminal (Mac/Linux) or command prompt (Windows) and simply enter scrape to begin.

Scraping of the Google Scholar profiles is handled by these scripts:

• scrape_control.R : The higher-level control script which would be activated by the user. Defines MongoDB connectivity and manages the scrape job
• scrape_requirements.R : Load packages and custom functions
• /R/ : Collection of functions to facilitate various aspects of scraping. Refer to individual script files for explanations

The scraper first collects the google scholar profile information which includes basic information such as name, affiliation, citations, and fields of study. This is inserted to the google_profiles collection. This webpage also exposes a list of publications. This list seeds the get_publication_details() function, which collects publication-level information from individual publication pages that is not exposed on the scholar profile page. This information is inserted to the publications collection. The process then repeats for the next scholar.

Use scrape_control.R to define the parameters of the job:

• (optional) Specify custom collection suffix to scrape into
  • Default: no suffix
• (optional) Specify inclusion/exclusion criteria to build scrape job
  • Default: Scrape all scholars who have a GSID but do not have a document in google_profiles collection
• (optional) Specify log / temp file directory as argument to scrape_looper(X, log_tmp_dir = [path])
  • Default: [project]/mongo_and_scrape/log

This command starts scraping and ensures the process will not terminate with your session.

nohup Rscript Scholarly/scrape_control.R \

</dev/null>Scholarly/log/`date '+%m-%d-%y'`_consoleScrape.out \

 2>Scholarly/log/`date '+%m-%d-%y'`_ScrapeErrorMessages.err & disown

Assumes proper configuration of .bash_profile. See:

Check for active scrape jobs with:

status

To terminate scrape job, note the PID from the above ps command, then:

kill [PID#]

To peek into the tail of the message log and error logs respectively, use the following commands: checklog checkerror

Check the current progress of an ongoing or completed scrape by referring to the log file.

• Scholarly/log/scraper_log_[date].log is the main log of the file which records the timestamped status updates of the script, it can be opened mid-scrape to check status
• nohup logs are a dump of the console and error/warning respectively, and are recommended for debugging as necessary.

An error, loss of power, kicked out by Google, etc, may cause a scrape job to not finish completely—for example, only 50 of a scholar's 85 publications are scraped.

On starting the script again, the script will continue where it was interrupted. It should not require any intervention.

If more scholars are added, or scholar documents that lacked GSID are updated with a GSID, just run the script to scrape them. By default, scholars who have been previously scraped in the given year collection will not be re-scraped.

You can scrape from an XLSX file containing Google Profile URLs using Google Scholar Profile Collector app. From R script: To scrape only one scholar, scholar_scrape() can be called directly.

The scrape can omit either the profile or the publications by modifying the corresponding argument from the default: scholar_scrape(scholar_query, fetch_profile=TRUE, fetch_publications=TRUE)

The scholar_query argument expects a data frame or named list that contains:

• scholar_query$_id
• scholar_query$gsid

This collection is scraped from the Google Scholar "Profile Page." Example profile page

This collection is scraped from publication pages. Example publication page.