README.md - abought/osf.io GitHub Wiki

OSF

Development

Help

Solutions to many common issues may be found at the OSF Developer Docs.

Running the OSF

If you have already installed all of the required services and Python packages, then you can start a working local test server with the following sequence:

invoke mongo -d  # Runs mongod as a daemon
invoke mailserver
invoke rabbitmq
invoke celery_worker
invoke elasticsearch
invoke assets -dw
invoke server

Note that some or all of these commands will run attached to a console, and therefore the commands may need to be run in separate terminals.

Optional extras

Some functionality depends on additional services that will not be started using the sequence above. For most development tasks, it is sufficient to run the OSF without these services, except as noted below.

TODO: Write this part! TODO: Sharejs TODO: Waterbutler

Installation

These instructions assume a working knowledge of package managers and the command line. For a detailed step-by-step walkthrough suitable for new programmers, consult the COS Development Docs.

Pre-requisites

Before attempting to run OSF setup commands, be sure that your system meets the following minimum requirements.

Mac OS

The following packages must be installed:

  • XCode command line tools (xcode-select --install)
  • Homebrew package manager (run brew update and brew upgrade --all before running installation)
  • Java (if not installed yet, run brew install Caskroom/cask/java)
  • Python 2.7
    • pip
    • virtualenv (pip install virtualenv)

Linux

TODO: Mention GPG etc

Quickstart

These instructions should work on Mac OSX >= 10.7

  • Clone the OSF repository to your computer. Change to that folder before running the commands below.
  • Create and activate your virtualenv.
virtualenv env
source env/bin/activate
  • Copy website/settings/local-dist.py to website/settings/local.py. NOTE: This is your local settings file, which overrides the settings in website/settings/defaults.py. It will not be added to source control, so change it as you wish.
$ cp website/settings/local-dist.py website/settings/local.py
  • On MacOSX with homebrew, there is a script that should automate much of the install process:
$ pip install invoke
$ invoke setup

To verify that your installation, follow the instructions to load the OSF and run unit tests. (TODO: add links)

  • You will need to:

    • Create local.py files for addons that need them.
    • Install TokuMX.
    • Install libxml2 and libxslt (required for installing lxml).
    • Install elasticsearch.
    • Install GPG.
    • Install requirements.
    • Create a GPG key.
    • Install npm.
    • Install node and bower packages.
    • Build assets.

  • On Linux systems, you may have to install python-pip, TokuMX, libxml2, libxslt, elasticsearch, and GPG manually before running the above commands.

  • If invoke setup hangs when 'Generating GnuPG key' (especially under linux), you may need to install some additonal software to make this work. For apt-getters this looks like:

sudo apt-get install rng-tools

next edit /etc/default/rng-tools and set:

HRNGDEVICE=/dev/urandom

last start the rng-tools daemon with:

sudo /etc/init.d/rng-tools start

__source: http://www.howtoforge.com/helping-the-random-number-generator-to-gain-enough-entropy-with-rng-tools-debian-lenny __

Starting Up

  • Run your mongodb process.
$ invoke mongo
  • Run your local development server.
$ invoke server
  • Run your local sharejs server.
$ invoke sharejs

Running the shell

To open the interactive Python shell, run:

$ invoke shell

Running Tests

To run all tests:

$ invoke test --all

To run a certain test method

$ nosetests tests/test_module.py:TestClass.test_method

Run OSF Python tests only:

$ inv test_osf

Run addons Python tests only:

$ inv test_addons

Run Javascript tests:

$ inv karma

By default, inv karma will start a Karma process which will re-run your tests every time a JS file is changed. To do a single run of the JS tests:

$ inv karma --single

By default, Karma will run tests using a PhantomJS headless browser. You can run tests in other browsers like so:

$ inv karma -b Firefox

If you want to run cross browser tests with SauceLabs, use "sauce" parameter:

$ inv karma --sauce

Testing Addons

Addons tests are not run by default. To execute addons tests, run

$ invoke test_addons

Testing Email

First, set MAIL_SERVER to localhost:1025 in you local.py file.

website/settings/local.py

...
MAIL_SERVER = "localhost:1025"
...

Sent emails will show up in your server logs.

Optional: fire up a pseudo-mailserver with:

$ invoke mailserver -p 1025

Using TokUMX

TokuMX is an open-source fork of MongoDB that provides support for transactions in single-sharded environments. TokuMX supports all MongoDB features as of version 2.4 and adds beginTransaction, rollbackTransaction, and commitTransaction commands.

Installing with Mac OS

$ brew tap tokutek/tokumx
$ brew install tokumx-bin

Installing on Ubuntu

$ apt-key adv --keyserver keyserver.ubuntu.com --recv-key 505A7412
$ echo "deb [arch=amd64] http://s3.amazonaws.com/tokumx-debs $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/tokumx.list
$ apt-get update
$ apt-get install tokumx

Migrating from MongoDB

TokuMX and MongoDB use different binary formats. To migrate data from MongoDB to TokuMX:

  • Back up the MongoDB data
    • invoke mongodump --path dump
  • Shut down the MongoDB server
  • Uninstall MongoDB
  • Install TokuMX (see instructions above)
  • Restore the data to TokuMX
    • invoke mongorestore --path dump/osf20130903 --drop
  • Verify that the migrated data are available in TokuMX

Using Celery

Installing Celery + RabbitMQ

  • Install RabbitMQ. On MacOSX with homebrew,
$ brew update
$ brew install rabbitmq

The scripts are installed to /usr/local/sbin, so you may need to add PATH=$PATH:/usr/local/sbin to your .bash_profile.

For instructions for other OS's, see the official docs.

Then start the RabbitMQ server with

$ invoke rabbitmq

If you want the rabbitmq server to start every time you start your computer (MacOSX), run

$ ln -sfv /usr/local/opt/rabbitmq/*.plist ~/Library/LaunchAgents
$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.rabbitmq.plist

Starting A Celery Worker

invoke celery_worker

Using Search

Elasticsearch

  • Install Elasticsearch

Mac OSX

$ brew install elasticsearch

note: Oracle JDK 7 must be installed for elasticsearch to run

Ubuntu

$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.deb
$ sudo dpkg -i elasticsearch-1.2.1.deb

Using Elasticsearch

  • In your website/settings/local.py file, set SEARCH_ENGINE to 'elastic'.
SEARCH_ENGINE = 'elastic'
  • Start the Elasticsearch server and migrate the models.
$ invoke elasticsearch
$ invoke migrate_search

Starting a local Elasticsearch server

$ invoke elasticsearch

NPM

The Node Package Manager (NPM) is required for installing a number of node-based packages.

# For MacOSX
$ brew update && brew install node

Installing Node on Ubuntu is slightly more complicated. Node is installed as nodejs, but Bower expects the binary to be called node. Symlink nodejs to node to fix, then verify that node is properly aliased:

# For Ubuntu
$ sudo apt-get install nodejs
$ sudo ln -s /usr/bin/nodejs /usr/bin/node
$ node --version      # v0.10.25

Install NPM requirements

To install necessary NPM requiremnts, run:

$ npm install

In the OSF root directory. This will install a number libraries for both the front-end and for building the assets (e.g. webpack).

To build assets with webpack

Use the following command to update your requirements and build the asset bundles:

$ inv assets -dw

The -w option puts you in "watch": assets will be built when a file changes.

Downloading citation styles (optional)

To download citation styles, run:

$ invoke update_citation_styles

Setting up addons

To install the python libraries needed to support the enabled addons, run:

$ invoke requirements --addons

Getting application credentials

Many addons require application credentials (typically an app key and secret) to be able to authenticate through the OSF. These credentials go in each addon's local.py settings file (e.g. website/addons/dropbox/settings/local.py).

For local development, the COS provides test app credentials for a number of services. A listing of these can be found here: https://osf.io/m2hig/wiki/home/ .

Livereload support

You can run the app server in livereload mode with:

$ invoke server --live

This will make your browser automatically refresh whenever a code change is made.

Troubleshooting

Trying to install libxml fails with error

If libxml installation fails with the error

"include libxml/xmlversion.h"