11. Installing on Vanilla MicroSoft Azure Ubuntu 16.04 box - findsorguk/findsorguk GitHub Wiki
To install the system on a Vanilla installation of MicroSoft Azure Ubuntu 16.04 (Canonical) you will need to run the following commands after you have SSH'd into the box using an SSH client or terminal on OSX:
ssh username@hostname
Run all commands as sudo user to save entering sudo and password constantly:
sudo su -
If you are new to Linux, you may want to think about keeping a record of the commands you have issued. To do this at the end of your session issue this command:
history > history_DDMMYYYY_username.txt
If you ever release upgrade or have a long running process to use, do not use SSH but use Screen instead.
Domain names have been temporarily configured for you to test installation:
- https://azure.micropasts.org - main site which would mirror https://finds.org.uk
- http://discourse.micropasts.org - forum which would mirror https://forum.finds.org.uk
- https://azure.micropasts.org/counties - blogs for HLF project which would mirror https://finds.org.uk/counties
You cannot change the DNS settings of these, as they are run via UCL.
Update the box:
apt-get update
apt-get upgrade
Format the data and mount the storage disk you created in Azure management console:
mkfs.ext4 /dev/sdc
mkdir /data
mount /dev/sdc /data
Warning about disk arrays when upgrading packages:
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
To get rid of this:
nano /etc/mdadm/mdadm.conf
Add:
ARRAY <ignore> devices=/dev/sda
Change PHP version from 7.0 to 5.6:
add-apt-repository ppa:ondrej/php
apt-get install php5.6 php7.0
a2dismod php7.0
a2endmod php5.6
update-alternatives --set php /usr/bin/php5.6
apt-get install php5.6-mysql libapache2-mod-php5.6
Check php version is interpreting as 5.6:
php -v
Install all software and enable (note the MySQL and phpMyadmin sections are interactive and requires setting up passwords):
apt-get install default-jdk
apt-get install git
apt-get install sendmail
apt-get install mysql-server
apt-get install apache2
apt-get install tomcat7
apt-get install tomcat7-admin
apt-get install curl php5.6-gd php5.6-imagick php5.6-curl php5.6-mcrypt php5.6-xml php5.6-xmlrpc php5.6-mbstring php5.6-common php5.6-apcu unzip php5.6-curl
apt-get install phpmyadmin
On Ubuntu 16.04
ln -s /etc/phpmyadmin/apache.conf /etc/apache2/conf-available/phpmyadmin.conf
a2enconf phpmyadmin.conf
service apache2 reload
- Think about changing the URL via aliasing
- Make the interface SSL enabled
- Add basic auth
- Disable root user login to phpmyadmin interface.
Tips on the above can be found here: https://poweruphosting.com/blog/install-secure-php-myadmin-ubuntu/
a2enmod rewrite
a2enmod headers
a2enmod expires
a2enmod xml2enc
a2enmod deflate
a2enmod proxy
a2enmod proxy_http
a2enmod proxy_ajp
a2enmod rewrite
a2enmod proxy_balancer
a2enmod proxy_connect
a2enmod proxy_html
This package speeds up website serving considerably:
dpkg -i mod-pagespeed-stable_current_amd64.deb
apt-get install -f
service apache2 restart
Configure as detailed here https://www.dreamvps.com/knowledgebase/tutorials/install-mod_pagespeed-module-apache-ubuntu-16-04/
The PAS software depends on APC cache to run. APC is deprecated so use APCu instead. Now you need to edit apcu.ini file to get it working:
cd /etc/php/5.6/conf.d/
nano 20-apcu.ini
Add:
extension=apcu.so
apc.enabled=1
apc.enable_cli=1
Now amend php settings for file uploads etc:
nano /etc/php/5.6/apache2/php.ini
Change following to these settings:
post_max_size = 30M
upload_max_filesize = 30M
memory_limit = 256M
Now restart Apache2, which will get everything like the PHP curl extension working and change memory and upload settings:
service apache2 restart
Install codebase:
cd /var/www
git clone https://github.com/findsorguk/findsorguk.git finds.org.uk
cd finds.org.uk
git submodule update --init --recursive
mkdir public_html/images
mkdir app/cache
chmod 777 public_html/images
chmod 777 app/cache
chown -R www-data:www-data /var/www/finds.org.uk
Set up mysql database:
mysql -u {username} -p`
At mysql command prompt:
CREATE DATABASE {dbName} CHARACTER SET utf8 COLLATE utf8_general_ci;
quit
cd /var/www/finds.org.uk/sql
mysql -u username -p {dbName} < database.sql
mysql -u username -p {dbName} < populateTables.sql
wget {pathToLatestBackup}
mysql -u username -p {dbName} < {latestBackupFileName}
This should create all data and users as needed.
If you ran the database now as a public user, you will see 401 errors for any page that requires elevated privileges. You can only see the true error if you have admin status and then you get 503 error eg:
Service Unavailable: 503 SQLSTATE[42000]: Syntax error or access violation: 1055 Expression #95 of SELECT list is not in GROUP BY clause and contains nonaggregated column
To fix this turn off ONLY_FULL_GROUP_BY mode (see https://stackoverflow.com/questions/23921117/disable-only-full-group-by)
SET GLOBAL sql_mode=(SELECT REPLACE(@@sql_mode,'ONLY_FULL_GROUP_BY',''));
And to make this permanent (https://www.sitepoint.com/quick-tip-how-to-permanently-change-sql-mode-in-mysql/)
which mysqld
<pathToMySQL> --verbose --help | grep -A 1 "Default options"
Via MySQL command look up the default options:
select @@sql_mode
Which will return:
ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
Edit the mysql conf file:
nano /etc/mysql/my.cnf
Add the parameters for constraints you do not need:
[mysqld]
# ... other stuff will probably be here
sql_mode = "STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
Now restart mysql:
service mysql restart
Create a user for each application and ensure their privileges are appropriate. These user accounts should have password changed regularly. Suggested users for:
- PAS database - SELECT, UPDATE, INSERT, DELETE
- Wordpress install - SELECT, UPDATE, INSERT, DELETE, ALTER, INDEX, CREATE TABLE, DROP TABLE
- SOLR data import - SELECT only
To create the user:
CREATE USER 'username'@'localhost' IDENTIFIED BY 'password';
Or use the GUI in phpmyadmin setting the required privileges.
Copy and use ini files for system that have all values needed for it to run (save manually entering data):
cd /var/www/finds.org.uk/app/config/
rsync -avzhP --stats --log-file=/var/log/rysncBackup.log --omit-dir-times --no-perms [email protected]:/var/www/finds.org.uk/app/config/ /var/www/finds.org.uk/app/config/
Adjust MySQL user details appropriately (remember to edit the production values):
nano config.ini
Install and configure SOLR after tweaking Tomcat7:
nano /etc/tomcat7/tomcat-users.xml
Add the following within the tags:
<user username="admin" password="password" roles="manager-gui,admin-gui"/>
Then restart Tomcat7
service tomcat7 restart
Now install and config Solr:
mkdir /var/solr
cd /var/solr
wget https://archive.apache.org/dist/lucene/solr/4.2.1/solr-4.2.1.zip
unzip solr-4.2.1.zip
mkdir lib solr
cp solr-4.2.1/example/lib/* lib
cd lib
wget http://central.maven.org/maven2/org/apache/solr/solr-dataimporthandler/4.2.1/solr-dataimporthandler-4.2.1.jar
wget https://dev.mysql.com/get/archives/mysql-connector-java-5.1/mysql-connector-java-5.1.23.zip
unzip mysql-connector-java-5.1.23.zip
mv mysql-connector-java-5.1.23/mysql-connector-java-5.1.23.jar ../
rm -R mysql-connector-java-5.1.23
rm mysql-connector-java-5.1.23.zip
cd /var/solr
cp solr-4.2.1/dist/solr-4.2.1.war solr.war
git clone https://github.com/findsorguk/findsorguk-solr.git findsorguk
cp findsorguk/solr.xml.template solr.xml
nano /var/lib/tomcat7/conf/Catalina/localhost/solr.xml
You need to make some slight adjustments to the config for data import. For each core (objects, images, geodata, people, bibliography, coinsummary, publications, tags):
cd /var/solr/findsorguk/solr/<core>/conf/
cp solrcore.properties.template solrcore.properties
nano solrcore.properties
Enter database name, username and password. There is no need to restart tomcat7 for these changes to be picked up.
Tweak config:
<?xml version="1.0" encoding="utf-8"?>
<Context docBase="/var/solr/solr.war" debug="0" crossContext="true" >
<Environment name="solr/home" type="java.lang.String" value="/var/solr" override="true" />
</Context>
cd /var/solr
sudo chown -R tomcat7:tomcat7 *
sudo service tomcat7 restart
nano /etc/tomcat7/Catalina/localhost/solr.xml
Edit the file to look like the below with pipe delimited IP address ranges or specific IP:
<?xml version="1.0" encoding="utf-8"?>
<Context docBase="/var/solr/solr.war" debug="0" crossContext="true">
<Environment name="solr/home" type="java.lang.String" value="/var/solr" override="true" />
<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="127.0.0.1|xxx.xxx.xxx.xxx|xxx.xxx.xxx.*" />
</Context>
Suggested IPs to use:
Localhost (127.0.0.1)
BM public IP
BM proxied IP
IP of other boxes in cluster
Set heap size for tomcat7:
cd /usr/share/tomcat7/bin
nano setenv.sh
Enter:
export CATALINA_OPTS="-Xms512M -Xmx1024M"
Restart Tomcat7.
service tomcat7 restart
Set up vhosts:
cd /etc/apache2/sites-available
wget https://raw.githubusercontent.com/findsorguk/vhostsConfigs/master/finds.org.uk.conf
a2ensite finds.org.uk.conf
service apache2 reload
Note this is Apache 2.4 so your config file must end in .conf
Go to the web address for the site - this should now work. If you get errors then:
tail -f /var/log/apache2/findsorguk_error.log
Look for the PHP errors or config errors that are thrown.
If all is well, you'll now need to copy images from one server to another and also index the databases for Solr.
To get up and running quickly use Letsencrypt:
add-apt-repository ppa:certbot/certbot
apt-get update
apt-get install python-certbot-apache
certbot --apache -d <domainName>
Follow the prompts for set up. If you have multiple domains do:
certbot --apache -d <domainName> -d <domainName2> -d <domainName3>
Set up autorenewal
crontab -e
Enter:
15 3 * * * /usr/bin/certbot renew --quiet
Save the crontab file.
To index the tables you can either use the GUI for solr (remember if you've restricted access by IP address, you won't be able to do this from one outside the config range). You will need to have set the db config details in the dataconfig.xml file for each core.
Before updating the objects table, make sure that no 0000-00-00 values have crept in. Either via command line or phpmyadmin issue these two commands for MySQL:
UPDATE finds SET `datefound1` = NULL WHERE `datefound1` = 0000-00-00;
UPDATE finds SET `datefound2` = NULL WHERE `datefound2` = 0000-00-00;
Check for dodgy dates:
SELECT * FROM `finds` WHERE `datefound1` LIKE '%-00-00%'
SELECT * FROM `finds` WHERE `datefound2` LIKE '%-00-00%'
SELECT * FROM `finds` WHERE `datefound1` LIKE '%-00%'
SELECT * FROM `finds` WHERE `datefound2` LIKE '%-00%'
Fix these to YYYY-01-01 (there should not be many, a validation class needs writing for the data input form for finds.)
Log onto the Solr GUI at:
http://<hostname>:8080/solr
Then choose core admin from drop down menu. Navigate to data import and just press the execute button. Repeat for each core (objects - you have two entities, make sure you do one import for each).
Indexing the data tables via the command line is the way to go!
http://<hostname>:8080/solr/<coreName>/dataimport?command=full-import&verbose=true
For example to update the people table:
http://<hostname>:8080/solr/people/dataimport?command=full-import&verbose=true
Repeat for each core.
To copy images across will take a very long time as the image store is over 2TB in size. Use Screen to do this and rsync. The below commands set up a screen session and then issues the rysnc command to copy images. replace username in the below with appropriate one and enter password when prompted. The server will have to be allowed to connect to Asgard on the UFW settings on that server.
screen -S imageTransfer
touch /var/log/rysncBackup.log
rsync -avzhP --stats --log-file=/var/log/rysncBackup.log --omit-dir-times --no-perms [email protected]:/var/www/finds.org.uk/public_html/images/ /var/www/finds.org.uk/public_html/images/
You can reconnect to screen if your session drops by using:
screen -r imageTransfer
or use:
screen -x imageTransfer
The system needs to have a firewall in place to restrict access. We use the Uncomplicated Firewall which is preinstalled on this version of Ubuntu. To find out more on this software: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-16-04
Check status:
ufw status
See what apps it can work with:
ufw app list
Now configure it for correct values.
22/tcp ALLOW Anywhere
Apache ALLOW Anywhere
OpenSSH ALLOW Anywhere
115/tcp ALLOW Anywhere
21/tcp ALLOW Anywhere
8080 ALLOW Anywhere
2812 ALLOW Anywhere
1080 ALLOW Anywhere
3306/tcp ALLOW 81.29.66.141
443/tcp ALLOW Anywhere
4567 ALLOW 81.29.66.140
5671 ALLOW 81.29.66.140
4567 ALLOW Anywhere
5671 ALLOW Anywhere
5672 ALLOW Anywhere
Anywhere DENY 74.14.238.115
22/tcp ALLOW Anywhere (v6)
Apache (v6) ALLOW Anywhere (v6)
OpenSSH (v6) ALLOW Anywhere (v6)
115/tcp ALLOW Anywhere (v6)
21/tcp ALLOW Anywhere (v6)
8080 ALLOW Anywhere (v6)
2812 ALLOW Anywhere (v6)
1080 ALLOW Anywhere (v6)
443/tcp ALLOW Anywhere (v6)
4567 ALLOW Anywhere (v6)
5671 ALLOW Anywhere (v6)
5672 ALLOW Anywhere (v6)
Now enable ufw:
ufw enable
Check status:
ufw status
You will find the rules in:
cd /etc/ufw
ls
Docker is used for the forum software PAS runs (Discourse). To install and configure follow these steps (see https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-16-04 for explanations):
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get update
apt-cache policy docker-ce
apt-get install -y docker-ce
systemctl status docker
Discourse is a discussion board software that comes from the creators of Stackoverflow. It requires maintenance as it is frequently updated. Images and uploads for this are stored on an AWS S3 bucket. Changing this storage is painful, I don't recommend it.
The default webserver for this package is nginx (installed earlier) which we put Apache2 in front of. Follow full guidelines here: https://www.digitalocean.com/community/tutorials/how-to-install-discourse-on-ubuntu-16-04
You will need to have a domain name that resolves to this server for the forum (http://discourse.micropasts.org)
mkdir /var/discourse
git clone https://github.com/discourse/discourse_docker.git /var/discourse
cd /var/discourse
./discourse-setup
You will need to apache first whilst you configure the forum as it will throw a fit about ports being used (80).
This process may tell you do not have docker installed, press enter and let it run. Then run:
./discourse-setup
You need to enter:
- Hostname
- Email address for admin
- SMTP settings for mail
- Letsencrypt registered email (optional)
Basic install will take 10 minutes or more as there's a lot of code to pull and write.
If you need to change or fix these settings after bootstrapping, edit your /containers/app.yml file and run:
./launcher rebuild app
So to get the correct port running:
nano /var/discourse/containers/app.yml
And change:
- "80:80"
To
- "3000:80"
Rebuild the app, restart apache2
Now stop Apache2 and install nginx and then restart apache2:
service apache2 stop
apt-get install nginx
service apache2 start
We now need to set up proxying to the underlying nginx config in docker. This proxies to port 3000.
Go to:
nano /etc/apache2/sites-available/forum.finds.org.uk.conf
Enter:
<VirtualHost *:80>
ServerAdmin [email protected]
ServerName <domainName>
CustomLog ${APACHE_LOG_DIR}/forum_access.log combined
ErrorLog ${APACHE_LOG_DIR}/forum_errors.log
ProxyPreserveHost On
ProxyRequests Off
ProxyPass / http://localhost:3000/
ProxyPassReverse / http://localhost:3000/
</VirtualHost>
Then enable site:
a2ensite forum.finds.org.uk.conf
Then reload apache2:
service apache2 reload
Otherwise, your changes will not take effect.
To migrate the content from one server to another, follow exactly the guide here: https://meta.discourse.org/t/move-your-discourse-instance-to-a-different-server/15721
You must be on the same version of Discourse if you don't want problems.
Once you have the discussion board cloned and running, change the url in app.yml and point the dns to this server. Rebuild the app and you will get all topics running off the correct urls.
You must update this frequently from /admin/upgrade
The county pages use Wordpress as their CMS and are part of the HLF packages that the Scheme is committed to. This is relatively easy to configure. You'll need to create a MySQL database and get the latest version of wordpress.
This assumes you already have configured the server base url https://finds.org.uk to point to the new server.
CREATE DATABASE countypages CHARACTER SET utf8 COLLATE utf8_general_ci;
quit
cd /var/www/finds.org.uk/public_html
ln -s /var/www/counties.finds.org.uk /counties
Simplest method to get up and running is to do a fresh install of Wordpress. The below is also best practise for secure installation.
cd /tmp
curl -O https://wordpress.org/latest.tar.gz
tar xzvf latest.tar.gz
touch /tmp/wordpress/.htaccess
chmod 660 /tmp/wordpress/.htaccess
cp /tmp/wordpress/wp-config-sample.php /tmp/wordpress/wp-config.php
mkdir /tmp/wordpress/wp-content/upgrade
mkdir /var/www/counties.finds.org.uk
cp -a /tmp/wordpress/. /var/www/counties.finds.org.uk
chown -R www-data:www-data /var/www/counties.finds.org.uk
find /var/www/counties.finds.org.uk -type d -exec chmod g+s {} \;
chmod g+w /var/www/counties.finds.org.uk/wp-content
chmod -R g+w /var/www/counties.finds.org.uk/wp-content/themes
chmod -R g+w /var/www/counties.finds.org.uk/wp-content/plugins
Now for the security bit:
curl -s https://api.wordpress.org/secret-key/1.1/salt/
Copy the output to your clipboard.
nano /var/www/counties.finds.org.uk/wp-config.php
Find the section that contains the dummy values for those settings. It will look something like this:
define('AUTH_KEY', 'put your unique phrase here');
define('SECURE_AUTH_KEY', 'put your unique phrase here');
define('LOGGED_IN_KEY', 'put your unique phrase here');
define('NONCE_KEY', 'put your unique phrase here');
define('AUTH_SALT', 'put your unique phrase here');
define('SECURE_AUTH_SALT', 'put your unique phrase here');
define('LOGGED_IN_SALT', 'put your unique phrase here');
define('NONCE_SALT', 'put your unique phrase here');
Replace this section with your clipboard values. Note this will cause all old passwords to be invalid. If you want to maintain old access, copy the values from the old wp-config.php on Asgard.
Now look for the database values below and replace them with your MySQL user details (not a root user!!)
define('DB_NAME', 'dbName');
/** MySQL database username */
define('DB_USER', 'userName');
/** MySQL database password */
define('DB_PASSWORD', 'password');
define('FS_METHOD', 'direct');
Make sure you set the database prefix to cp_ not wp_
Export the MySQL database from phpmyadmin gzipping the content from the extended options.
Import the database into MySQL via PhpMyadmin or the commandline interface.
Install the theme and child theme via git clone:
cd /var/www/counties/wp-content/themes/
git clone https://github.com/findsorguk/countypages-theme-child.git countypages-child
git clone https://github.com/findsorguk/countypages-theme.git countypages
Using rysnc, copy the wp-content/uploads folder to your installation directory. Make sure to change the username below to yours:
rsync -avzhP --stats --log-file=/var/log/rysncBackup.log --omit-dir-times --no-perms <username>@81.29.66.140:/var/www/finds.org.uk/public_html/counties/wp-content/ /var/www/counties.finds.org.uk/wp-content/
Check all the plugins that are needed for the PAS wordpress site. Do a fresh install to get latest versions or copy the folder if lazy.
- Akismet Anti-Spam
- Breadcrumb NavXT
- Google Analytics Dashboard for WP (GADWP)
- Jetpack by WordPress.com
- Multi-Site Site List Shortcode
- Network Latest Posts
- Network Plugin Auditor
- Redirection
- UpdraftPlus - Backup/Restore
- User Switching
- Wordfence Security
- WP Force SSL
You should now be done migrating.