OS X Setup - otwcode/otwarchive GitHub Wiki
Note: We strongly encourage using Docker for your development environment. Due to limited resources, we can't offer support for all local installs, so please proceed with the understanding that you'll have to do most of the troubleshooting on your own.
These instructions assume you are comfortable with your shell of choice. They also assume that you've installed Xcode and therefore have the GNU toolchain. None of this needs to be run as root.
These instructions will assume you have installed the package manager Homebrew. To get Homebrew:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
While macOS comes with Ruby pre-installed, you'll want to use the same version as us to avoid compatibility issues.
There are two competing ways of managing Ruby environments on Unixes these days: rbenv and RVM. If you're already using one of these, you can set it to Ruby 3.2.7 and you're done. Otherwise, you'll need to install one and get yourself rubied up.
These instructions will assume you're using rbenv. To install rbenv using Homebrew:
brew install rbenv rbenv-gemset openssl
Follow the setup instructions on the rbenv page to get it into your shell environment. Then run rbenv to install Ruby 3.2.7:
rbenv install 3.2.7 rbenv global 3.2.7 rbenv rehash ruby -v
If ruby -v does not return Ruby version 3.2.7, try quitting and restarting your terminal. If that doesn't work, you may still need to add rbenv to bash.
brew install git imagemagick [email protected] [email protected]
Note: If you intend to allow downloads (ePub, PDF, etc) from your archive, you will need to add "calibre" to your package install list.
In the output will be two caveat blocks containing additional information on setting up Redis and MariaDB. A version of that information is included in the following sections, but make sure to follow the directions from your console.
We use Redis 3.2, which is no longer available for Homebrew, and our Redis setup for Cucumber tests is not compatible with the latest version of Redis. Fortunately, [email protected] works instead.
==> Caveats [email protected] is keg-only, which means it was not symlinked into /opt/homebrew, because this is an alternate version of another formula. If you need to have [email protected] first in your PATH, run: echo 'export PATH="/opt/homebrew/opt/[email protected]/bin:$PATH"' >> ~/.zshrc To start [email protected] now and restart at login: brew services start [email protected] Or, if you don't want/need a background service you can just run: /opt/homebrew/opt/[email protected]/bin/redis-server /opt/homebrew/etc/redis.conf
Because we're on an alternate version of Redis, we'll need create a symlink before we can follow the other setup instructions:
brew link [email protected]
Once you've created the symlink, follow the brew startup instructions for Redis by copying and pasting the desired command from the caveat.
==> Caveats ==> [email protected] A "/etc/my.cnf" from another install may interfere with a Homebrew-built server starting up correctly. MySQL is configured to only allow connections from localhost by default [email protected] is keg-only, which means it was not symlinked into /opt/homebrew, because this is an alternate version of another formula. If you need to have [email protected] first in your PATH, run: echo 'export PATH="/opt/homebrew/opt/[email protected]/bin:$PATH"' >> ~/.zshrc For compilers to find [email protected] you may need to set: export LDFLAGS="-L/opt/homebrew/opt/[email protected]/lib" export CPPFLAGS="-I/opt/homebrew/opt/[email protected]/include" For pkg-config to find [email protected] you may need to set: export PKG_CONFIG_PATH="/opt/homebrew/opt/[email protected]/lib/pkgconfig" To start [email protected] now and restart at login: brew services start [email protected] Or, if you don't want/need a background service you can just run: /opt/homebrew/opt/[email protected]/bin/mysqld_safe --datadir\=/opt/homebrew/var/mysql
Because we're on an alternate version of MariaDB, we'll need create a symlink before we can follow the other setup instructions:
brew link [email protected]
Now you can start MariaDB:
brew services start [email protected]
Then run mariadb-secure-installation and follow the instructions for setting up your database. Make a note of the user name and password you've used. You'll need it to configure your database.
You'll also need to install and start Elasticsearch. Unfortunately, Elasticsearch no longer supports Homebrew, so we'll need to take a different approach.
- Download Elasticsearch 8.18.1 from the Elasticsearch website, unzip it, and drag the elasticsearch-8.18.1 folder somewhere easy to find, e.g., your Documents folder.
- Download Java 8 from java.com and install it.
- In the console, navigate to your Elasticsearch bin folder, e.g.,
cd Documents/elasticsearch-8.18.1/bin - Start Elasticsearch:
./elasticsearch- You might need to disable Gatekeeper if you run into errors like these. While you're in Privacy & Settings to confirm your choice, make sure to allow jdk. Then you will be able to rerun
./elasticsearchsuccessfully. - You can set $JAVA_HOME if it complains about that.
- You might need to disable Gatekeeper if you run into errors like these. While you're in Privacy & Settings to confirm your choice, make sure to allow jdk. Then you will be able to rerun
Now you can clone the source:
git clone [email protected]:otwcode/otwarchive.git
If you want to participate in the project and create pull requests the easy way, you should probably make yourself an account here on GitHub if you don't already have one. Fork the repo! Get a copy of your repo locally using either the GitHub for Mac client or the command line:
git clone [email protected]:YOURGITHUB/otwarchive.git git remote add upstream [email protected]:otwcode/otwarchive.git
You add the original repository as a remote so you can track changes made to official repo.
cd to the project directory. You're now looking at a typical Rails app. It uses Bundler to manage dependencies, so you can install those:
gem install bundler rbenv rehash bundle install
cp config/database.example config/database.yml
Edit the database.yml file and give it the username and password combo you used when setting up MariaDB. Test that your setup is working by running the rake task to build the database tables:
bundle exec rake db:create:all
cp config/redis.example config/redis.yml
If you're not already using Redis in other projects, change dev.ao3.org to localhost and move to the next step. If you are, read on.
Unfortunately, the Redis initialization forces you to set up Redis using a hostname:port string instead of just taking a hash like you might expect. This doesn't let you specify which Redis database to use, which if you've already got data in a Redis instance is unacceptable. Let's fix that.
Edit config/initializers/gem-plugin_config/redis.rb. Replace these two lines:
redis_host, redis_port = redis_config[rails_env].split(":")
$redis = Redis.new(:host => redis_host, :port => redis_port)
With this one:
$redis = Redis.new(redis_config[rails_env])
Put something like this into your redis.yml. Your db numbers might vary.
test:
:host: localhost
:port: 6379
:db: 17
development:
:host: localhost
:port: 6379
:db: 16
production:
:host: localhost
:port: 6379
:db: 16
bundle exec rake db:schema:load bundle exec rake db:migrate
Note: If you're getting an error on schema:load, you may need to comment out factory_girl and factory_girl_rails in the Gemfile, bundle, and try running bundle exec rake db:schema:load again. When you're done, don't forget to uncomment the lines and run bundle install again.
Edit config/local.yml and add anything to it that you want to change from config/config.yml. The file must be present even if empty.
bundle exec rails server
If you're on Yosemite (OS X 10.10) or newer, you will need to use bundle exec rails server -d.
You might now need to fix a YAML engine problem. If Rails eventually exits with a yaml/psych parse error, edit config/boot.rb and add this line after Bundler.setup:
YAML::ENGINE.yamler = 'syck'
Newer versions of Ruby are using a YAML engine that doesn't allow symbols in YAML files, and the OTW localization files are full of symbols. The syck engine is tolerant of symbols. Another way to solve this problem is to use rbenv to install an older Ruby. 1.9.0 and 1.9.1 should work, but will have bugs fixed in later versions of Ruby.
You can now visit http://localhost:3000/ in your browser of choice and see an empty local archive.
bundle exec rake db:otwseed bundle exec rake work:missing_stat_counters bundle exec rake skins:load_site_skins
You may also wish to create additional admin and user accounts.
To create your search indexes (and fill them with data), run the tasks in search.rake:
bundle exec rake search:index_tags bundle exec rake search:index_works bundle exec rake search:index_pseuds bundle exec rake search:index_bookmarks bundle exec rake search:index_admin_users
Note: If the above commands error with Failed to open TCP connection to 127.0.0.1:9400, you might need to reconfigure to the port your Elasticsearch is running on (i.e. Homebrew uses :9200 by default). Add the port ES_URL: "http://localhost:9200" to local.yml.
To make sure your indexes get updated (e.g., when you create a new work), you'll need to start up a Resque worker and make sure Resque Scheduler is running:
RAILS_ENV=development QUEUE=* bundle exec rake environment resque:work bundle exec rake resque:scheduler
Now is also a good time to load autocomplete data:
bundle exec rake autocomplete:load_data
Now that you know you have a working install, you can install Puma-dev and use prettier URLs like http://otwarchive.test/. Recommended for anybody working on more than one rack app project at a time.
bundle exec rake db:test:prepare
You'll need to edit redis-cucumber.conf to get Redis to work with cucumber tests. Remove the line that says
dbfilename log/redis-cucumber-dump.rdb
If you're using a MacBook and it won't bundle install the nokogiri v.1.6.8 gem, you need to follow these instructions from http://john.hu/node/89:
Check your version of libxml2:
brew list libxml2
If it is not installed and/or isn't libxml2 2.9.4, please use the following commands to upgrade it:
brew update brew upgrade libxml2
After that, if you cannot install nokogiri 1.6.8 with the same error, you may also need to upgrade code-selected:
xcode-select --install
Then run bundle install again.
If you receive an error like the following:
Mysql2::Error: Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'otwarchive_development.subscriptions.id' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by: SELECT `subscriptions`.* FROM `subscriptions` WHERE ((subscribable_id = 1677550 AND subscribable_type = 'Work')
OR (subscribable_id IN (202514) AND subscribable_type = 'User')
OR (subscribable_id IN (NULL) AND subscribable_type = 'Series')) GROUP BY user_id
You may need to edit your MySQL config settings as described in Anuj Gakhar's "MySQL Error caused by sql_mode=only_full_group_by" blog post.
1. Copy the default config file:
sudo cp $(brew --prefix mysql)/support-files/my-default.cnf /etc/my.cnf
If my-default.cnf is missing, try:
sudo cp /usr/local/etc/my.cnf /etc/my.cnf
2. Edit the file to remove to remove ONLY_FULL_GROUP_BY and STRICT_TRANS_TABLES from sql_mode (you may only have one of these):
sudo nano /etc/my.cnf
Example addition to my.cnf:
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
3. After updating my.cnf, restart your MySQL server.
In December 2018, we converted the Archive's database to utf8mb4 (AO3-4056). To convert your development database:
1. Edit your MySQL config file (see the previous section for its location) to have:
[mysqld] innodb_file_format = Barracuda innodb_large_prefix = 1 innodb_file_per_table = ON
Note: If your MySQL server is greater than 5.7.7, innodb_file_format is already set to Barracuda, and does not need to be set.
2. Restart your MySQL server:
brew services restart mysql
3. Update your config/database.yml. If you find a section like this:
encoding: utf8 collation: utf8_general_ci
change it to:
encoding: utf8mb4 collation: utf8mb4_unicode_ci
3. Create a new migration ConvertDatabaseToUtf8mb4, as seen in the guide "Converting a Rails database to Utf8mb4 without downtime or data loss". The migration has a bug (AO3-5611), so you need to replace both instances of column.default.blank? with column.default.nil?. Then run ConvertDatabaseToUtf8mb4.
4. Repeat the previous step for your test database. To check if your test database is converted correctly, you can run the test to create a work with emojis in its title and content.
If you've switched your existing Homebrew setup from [email protected] to [email protected] and are getting errors when running `rails server`, you may need to restore your mysql2 gem to its original state:
gem pristine mysql2