Table of Contents Deploy Prepare a debian chrooted (Optional) Prerequisite Config Apache and Django project with wsgi.py Import the basic data Add VisiOmatic iipsrv server Install IIPsrv Configure Apache for iipserv Install Oracle client Switch to develop branch * for TESTING env Update code Setup apache reload, via ssh Adding des basic user Adding DataDiscovery daemon Configure cron daemon to run by cron every 12 hours Install Celery system Install Wordpress Home site Php for wordpress Mysql Roteiro desenvolvidor a partir da imagem de produção Server Restart procedure = Ports per image DRI master DRI develop DRI masterlist References

Deploy

Prepare a debian chrooted (Optional)

this step is util to 'standardize' the installation steps

Follow the instruction in http://www.campisano.org/wiki/en/Debian_chrooted_server

Remember to create the link in /etc/init.d and configure the service to startup at boot time:

ln -s DEBIAN_jessie_amd64 CHROOT_master
ln -s /srv/CHROOT_master/srv/config/chroot.sh /etc/init.d/chroot_master.sh
cat /etc/init.d/chroot_master.sh | sed -e 's|/srv/CHROOT|/srv/CHROOT_master|g' > chroot.sh
cat chroot.sh > /etc/init.d/chroot_master.sh
rm -f chroot.sh
update-rc.d chroot_master.sh defaults
# for centos:   chkconfig chroot_master.sh on
cat /etc/resolv.conf > /srv/CHROOT_master/etc/resolv.conf
hostname > /srv/CHROOT_master/etc/hostname
echo 127.0.0.1       localhost `hostname` > /srv/CHROOT_master/etc/hosts

NOTE: the folder /HOST must be enabled to be mounted in the chroot.sh script:

        # { == NCSA specific mount points ==
        #mount -o rbind / "${CHROOT}/HOST";
        mount -o bind /cluster_scratch/users/des-dri/storage/data "${CHROOT}/storage/data";
        #mount -o bind /des003/desarchive/ACT/multiepoch "${CHROOT}/storage/desarchive/ACT/multiepoch";
        mount -o bind /archive_data/desarchive "${CHROOT}/storage/desarchive";

        for DES_DEV in /des0*
        do
            if `test -d ${DES_DEV}`;
            then
                mkdir -p "${CHROOT}${DES_DEV}";
                mount -o bind ${DES_DEV} "${CHROOT}${DES_DEV}";
            fi;
        done;
        # } == NCSA specific mount points  ==

        # { == NCSA specific mount points ==
        for MOUNTED_DES_DEV in "${CHROOT}/des0"*
        do
            if `test -d ${MOUNTED_DES_DEV}`;
            then
                umount "${MOUNTED_DES_DEV}";
                rmdir "${MOUNTED_DES_DEV}";
            fi;
        done;

        umount "${CHROOT}/storage/desarchive";
        #umount "${CHROOT}/storage/desarchive/ACT/multiepoch";
        umount "${CHROOT}/storage/data";
        #umount "${CHROOT}/HOST";
        # } == NCSA specific mount points ==

and create the folder inside the chroot env

mkdir -p /srv/CHROOT_master/storage/desarchive

For the following step, you must start your chrooted server and chroot inside it

/etc/init.d/chroot_master.sh start
chroot /srv/CHROOT_master

Prerequisite

Install apache 2 with WSGI support:

apt-get update                                      # update apt, test connection and dns
apt-get dist-upgrade
apt-get install apache2 libapache2-mod-wsgi-py3     # ~40MB of additional packages
apt-get clean
/etc/init.d/apache2 stop
ln -s /etc/init.d/apache2 /srv/config/init.d
ln -s ../init.d/apache2 /srv/config/rc.d/S50_apache2
ln -s ../init.d/apache2 /srv/config/rc.d/K50_apache2
# for centos:   yum install httpd mod_wsgi

Config Apache and Django project with wsgi.py

####
# REMEMBER TO CHANGE DEFAULT PORTS IN /etc/apache2/ports.conf, ie 8080 and 80443
####
nano /etc/apache2/ports.conf
cd /etc/apache2/sites-available
# for centos:   cd /etc/httpd/conf.d/
mv 000-default.conf 000-default.conf_orig
# for centos:   NONE
cat > 000-default.conf << 'EOF'
# WSGIPythonPath cannot be defined inside VirtualHost, it must be global
# so, VirtualHost was commented, it's an illusion
#<VirtualHost *:8080>

    # apache site
    Define SITE desportal.cosmology.illinois.edu
    ServerName ${SITE}
    ServerAdmin root@${SITE}

    # apache paths
    Define BASEPATH /srv/domain/${SITE}
    Define DATA_PATH /storage/data
    Define APPS_PATH ${BASEPATH}/apps

    # wsgi user and group
    Define EXEC_USER des-dri
    Define EXEC_GROUP des-dri

    # django and wsgi configs
    Define PROJECT api
    Define MAIN_APP dri
    Define ALIAS_MAIN_APP_URL /dri/api
    Define ALIAS_APPS_URL /dri/apps
    Define ALIAS_DATA_URL /data
    Define PYTHON_PATH ${BASEPATH}/${PROJECT}:${BASEPATH}/env/lib/python3.4/site-packages

    <IfModule mod_headers.c>
        #Header set Access-Control-Allow-Origin "http://desportal.cosmology.illinois.edu"
        #Header set Access-Control-Allow-Origin "http://desportal.cosmology.illinois.edu:8080"
        #Header set Access-Control-Allow-Origin "http://dri.com"
        #Header set Access-Control-Allow-Origin "http://dri.com http://desportal.cosmology.illinois.edu http://desportal.cosmology.illinois.edu:8080"
        # from http://stackoverflow.com/a/22331292
        # tested at http://regexr.com/3e33k DOES NOT WORK
        #SetEnvIf Origin "(http)(s)?(:\/\/)([\w.-]*)(:\d+)?(\/\w*)?$" AccessControlAllowOrigin=$1$2$3$4$5
        #SetEnvIf Origin "http(s)?://(www\.)?(dri.com|desportal.cosmology.illinois.edu|desportal.cosmology.illinois.edu:8080)$" AccessControlAllowOrigin=$0$1
        #Header set Access-Control-Allow-Origin %{AccessControlAllowOrigin}e env=AccessControlAllowOrigin
        #SetEnvIf Origin ^(https?://.+\.mywebsite\.com(?::\d{1,5})?)$   CORS_ALLOW_ORIGIN=$1
        #SetEnvIf Origin ^(https?://.+\..+(?::\d{1,5})?)(?:/.*)$ CORS_ALLOW_ORIGIN=$1
        #SetEnvIf Referer "^(https?://.+\..+(?::\d{1,5})?)(?:/.*)$" CORS_ALLOW_ORIGIN=$1
        #SetEnvIf Referer "^(https?://(?:[\w-]+\.?)+(?::\d{1,5})?)(?:/.*)$" CORS_ALLOW_ORIGIN=$1
        #### Referer is used to test the regular expression: the correct is to test the Origin var, that I think must be sent by the client!!!
        #SetEnvIf Origin "^(https?://(?:[\w-]+\.?)+(?::\d{1,5})?)(?:/.*)$" CORS_ALLOW_ORIGIN=$1
        #Header set Access-Control-Allow-Origin %{CORS_ALLOW_ORIGIN}e env=CORS_ALLOW_ORIGIN
        #Header merge Vary "Origin"
        #Header set Access-Control-Allow-Credentials "true"
        #Header set Access-Control-Max-Age "1728000"
        SetEnvIf Origin "^(https?://(?:[\w-]+\.?)+(?::\d{1,5})?)(?:/.*)?$" CORS_ALLOW_ORIGIN=$1
        Header always set Access-Control-Allow-Origin %{CORS_ALLOW_ORIGIN}e env=CORS_ALLOW_ORIGIN
        Header merge Vary "Origin"
        Header set Access-Control-Allow-Credentials "true"
        Header set Access-Control-Max-Age "1728000"
        Header set header_version 5
    </ifmodule>

    DocumentRoot ${BASEPATH}/apps
    <Directory ${BASEPATH}/htdocs>
        Require all granted
    </directory>

    LogLevel warn
    ErrorLog ${BASEPATH}/log/error.log
    CustomLog ${BASEPATH}/log/access.log combined

    <IfModule mod_wsgi.c>
        # Django wsgi*.py
        <Directory ${BASEPATH}/${PROJECT}/${MAIN_APP}>
            <Files wsgi.py>
                Require all granted
            </files>
        </directory>

        WSGIScriptAlias ${ALIAS_MAIN_APP_URL} ${BASEPATH}/${PROJECT}/${MAIN_APP}/wsgi.py
        WSGIDaemonProcess ${SITE} python-path=${PYTHON_PATH} user=${EXEC_USER} group=${EXEC_GROUP} threads=64
        WSGIApplicationGroup ${SITE}
        WSGIProcessGroup ${SITE}
        WSGIPassAuthorization On

        # useful for WSGIAuthUserScript script:
        WSGIPythonPath ${PYTHON_PATH}

        # Generic Data static files
        Alias ${ALIAS_DATA_URL} ${DATA_PATH}
        <Directory ${DATA_PATH}>
            SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
            Options FollowSymLinks
            AuthType Basic
            AuthName "${SITE} login"
            Require valid-user
            AuthBasicProvider wsgi
            WSGIAuthUserScript ${BASEPATH}/${PROJECT}/${MAIN_APP}/wsgi.py application-group=${EXEC_GROUP}
        </directory>
    </ifmodule>

    # ExtJS apps static files and wordpress home/.htaccess
    Alias ${ALIAS_APPS_URL} ${APPS_PATH}
    <Directory ${APPS_PATH}>
        Require all granted
        AllowOverride All
    </directory>

    # Django static files
    Alias /static ${BASEPATH}/${PROJECT}/${MAIN_APP}/static
    <Directory ${BASEPATH}/${PROJECT}/${MAIN_APP}/static>
        Require all granted
    </directory>
#</virtualhost>
EOF
####

First deploy

apt-get install python3-pip virtualenv git      # ~220MB of additional packages
apt-get clean
OLD_UMASK=`umask`
umask 0022
pip3 install --upgrade setuptools pip virtualenv
umask ${OLD_UMASK}
unset OLD_UMASK
# for centos:   yum install https://centos6.iuscommunity.org/ius-release.rpm
# for centos:   yum install yum install python34u python34u-pip
# for centos:   pip3.4 install --upgrade pip
# for centos:   pip3.4 install --upgrade virtualenv
#### create a user to have access to www-data
groupadd --gid 10001 des-dri
useradd --uid 10001 --gid des-dri --no-user-group --groups www-data --shell /bin/bash --create-home --home-dir /home/des-dri des-dri
# for centos:   useradd --gid des-dri --no-user-group --shell /bin/bash --create-home --home-dir /home/des-dri des-dri
mkdir -m 0755 /srv/domain
mkdir -m 2755 /srv/domain/desportal.cosmology.illinois.edu
chown des-dri:des-dri /srv/domain/desportal.cosmology.illinois.edu
echo "source /etc/profile" >> /home/des-dri/.bashrc
echo "umask 0022" >> /home/des-dri/.bashrc
su - des-dri
#### as des-dri user
export DRI_HOME=/srv/domain/desportal.cosmology.illinois.edu
echo "export DRI_HOME=$DRI_HOME" >> ~/.bashrc
# for centos:   echo "export DRI_HOME=$DRI_HOME" >> ~/.bashrc
export DJANGO_SETTINGS_MODULE=dri.settings.local
echo "export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE" >> ~/.bashrc
# for centos:   echo "export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE" >> ~/.bashrc
cd $DRI_HOME
git clone --branch master https://github.com/linea-it/dri.git .
mkdir -p db log
chmod 2775 db log
virtualenv --no-site-packages --always-copy --python python3 env
source env/bin/activate
#### inside virtualenv
cd api
pip3 install --upgrade -r requirements.txt
# for centos:   pip3.4 install --upgrade -r requirements.txt
python -c "import django; print(django.get_version())"      # Test django version, should be >= 1.9.4
####
cat > dri/settings/local.py << 'EOF'
from dri.settings.production import *
EOF
####
####
cat > dri/settings/local_vars.py << 'EOF'
import os

# Just the 3 vars that the specific environment will use:
# BASE_PROJECT, LOG_DIR, DATABASES

BASE_PROJECT = '/srv/domain/desportal.cosmology.illinois.edu'

LOG_DIR = os.path.join(BASE_PROJECT, 'log')

DATA_DIR = '/storage/data/portal'

DATA_TMP_DIR = 'tmp'

DATA_SOURCE = '/data/portal'

DOWNLOAD_DIR = os.path.join(DATA_DIR, 'download')

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_PROJECT, 'db/dri.db'),
    },
    'catalog': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_PROJECT, 'db/catalog.db'),
    },
}

EMAIL_HELPDESK = '[email protected]'
EMAIL_HOST = 'localhost'
EMAIL_PORT = '25'

# Target Viewer Rating, Reject Schema
SCHEMA_RATING_REJECT = 'brportal'

# DES Cutout Service
DES_CUTOUT_SERVICE = {
    'HOST': 'http://descut.cosmology.illinois.edu',
    'USER': '',
    'PASSWORD': '',
    # Path onde ficaram os arquivos de cutout, esse parametro sera usado em conjunto com DATA_DIR para criar o path
    # absoluto para os arquivos.
    'CUTOUT_DIR': 'targets/cutouts',
    # Url base que sera usada para exibir as imagens geradas esse parametro deve ser mapeado no dri.conf no apache
    'CUTOUT_SOURCE': '/data/portal',
    # Tempo de delay para a task check_jobs em minutos
    'CUTOUT_TASK_CHECK_JOBS_DELAY': 1
}

EOF
####
#### python manage.py makemigrations    #### [CMP] the makemigrations step can be tricky: it should be done by the developer user and the result should be committed
python manage.py migrate                                #### [CMP] the migrate step can ask something to the user
python manage.py migrate catalog --database=catalog     #### [CMP] the migrate step can ask something to the user
python manage.py createsuperuser    # use dri-admin desbrazil
python manage.py collectstatic --clear --noinput
deactivate
exit
#### as root again
a2enmod headers
apache2ctl configtest
/etc/init.d/apache2 start           # test it in localhost/dri/api/
exit                                # exit chroot environment

Import the basic data

Database ingestion procedure

# TO BE DONE
chroot /srv/CHROOT_master
su - des-dri
cd $DRI_HOME
source env/bin/activate
cd api
python manage.py loaddata initial_data.json
mkdir -p coadd/fixtures
cd coadd/fixtures
rm -f *.json
git clone --branch master https://github.com/linea-it/dri-coadd_data .
cd ../../
python manage.py loaddata filter
python manage.py loaddata release
python manage.py loaddata tag
python manage.py loaddata tile
python manage.py loaddata dataset
python manage.py loaddata survey
rm -rf coadd/fixtures
git checkout coadd/fixtures
deactivate
exit
exit

Add VisiOmatic iipsrv server

Install IIPsrv

chroot /srv/CHROOT_master
apt-get install wget libapache2-mod-fastcgi unzip g++ autoconf automake libtool libtiff5-dev libopenjpeg-dev libzthread-dev libmemcached-dev  # ~40MB of additional packages
apt-get clean
a2enmod fastcgi
su - des-dri
cd /tmp
#### wget https://codeload.github.com/ruven/iipsrv/zip/iipsrv-1.0
#### wget https://github.com/cmarmo/iipsrv-astro/archive/iipsrv-0.9.9.zip
wget https://github.com/cmarmo/iipsrv-astro/archive/74e17e2e124f5d7af0eddc020cd973588c784a1b.zip
#### unzip iipsrv-1.0
#### unzip iipsrv-0.9.9.zip
unzip 74e17e2e124f5d7af0eddc020cd973588c784a1b.zip
#### /bin/rm -f iipsrv-1.0
#### /bin/rm -f iipsrv-0.9.9.zip
/bin/rm 74e17e2e124f5d7af0eddc020cd973588c784a1b.zip
#### cd iipsrv-iipsrv-1.0
#### cd iipsrv-astro-iipsrv-0.9.9
cd iipsrv-astro-74e17e2e124f5d7af0eddc020cd973588c784a1b
sh autogen.sh
sh configure
make
exit
#### cd /tmp/iipsrv-iipsrv-1.0
#### cd /tmp/iipsrv-astro-iipsrv-0.9.9
cd /tmp/iipsrv-astro-74e17e2e124f5d7af0eddc020cd973588c784a1b
mkdir -m 0775 /srv/local
mkdir -m 0775 /srv/local/apache2
mkdir -m 0775 /srv/local/apache2/modules
cp src/iipsrv.fcgi /srv/local/apache2/modules
chown root:root /srv/local/apache2/modules/iipsrv.fcgi
chmod 0775 /srv/local/apache2/modules/iipsrv.fcgi
cd ..
#### rm -rf iipsrv-iipsrv-1.0
#### rm -rf iipsrv-astro-iipsrv-0.9.9
rm -rf iipsrv-astro-74e17e2e124f5d7af0eddc020cd973588c784a1b
exit

Configure Apache for iipserv

chroot /srv/CHROOT_master
cd /etc/apache2/sites-available
nano 000-default.conf

REMEMBER TO CHANGE DEFAUlT PORT, ie 9080, 9000 is default

    <IfModule mod_fastcgi.c>
        # to use, iip = '/visiomatic?FIF=/tmp/des2359.tif'
        # will use /storage/tmp/des2359.tif
        Alias /visiomatic /srv/local/apache2/modules/iipsrv.fcgi
        <Location /visiomatic>
            SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
            AuthType Basic
            AuthName "${SITE} login"
            Require valid-user
            AuthBasicProvider wsgi
            WSGIAuthUserScript ${BASEPATH}/${PROJECT}/${MAIN_APP}/wsgi.py application-group=${EXEC_GROUP}
        </location>

        AddHandler fastcgi-script fcgi

        FastCgiServer /srv/local/apache2/modules/iipsrv.fcgi -port 9000 \
            -initial-env FILESYSTEM_PREFIX=/storage/ \
            -initial-env LOGFILE=/var/log/iipsrv.log \
            -initial-env VERBOSITY=2 \
            -initial-env JPEG_QUALITY=90 \
            -initial-env MAX_IMAGE_CACHE_SIZE=10 \
            -initial-env MAX_CVT=3000
    </ifmodule>

Restart and exit

apache2ctl configtest
/etc/init.d/apache2 restart
ps aux | grep iipsrv                # check that iipsrv is running
exit

Note about STIFF image creation IT IS NOT PART OF THE INSTALL PROCESS

cd /tmp
####
cat > stiff.conf << 'EOF'
# Default configuration file for STIFF 2.3.0
# EB 2014-02-06
#
OUTFILE_NAME           DES2246-4457.ptif       # Name of the output file
IMAGE_TYPE             AUTO            # Output image format: AUTO, TIFF,
                                       # or TIFF-PYRAMID
BITS_PER_CHANNEL       -32             # 8, 16 for int, -32 for float
BIGTIFF_TYPE           AUTO            # Use BigTIFF? NEVER,ALWAYS or AUTO
COMPRESSION_TYPE       NONE            # NONE,LZW,JPEG,DEFLATE or ADOBE-DEFLATE
COMPRESSION_QUALITY    90              # JPEG compression quality (%)
TILE_SIZE              256             # TIFF tile-size
PYRAMID_MINSIZE        256             # Minimum plane size in TIFF pyramid
BINNING                1               # Binning factor for the data
FLIP_TYPE              NONE            # NONE, or flip about X, Y or XY
FITS_UNSIGNED          N               # Treat FITS integers as unsigned

#------------------------------- Dynamic range ------------------------------

SKY_TYPE               AUTO            # Sky-level: "AUTO" or "MANUAL"
SKY_LEVEL              0.0             # Background level for each image
MIN_TYPE               GREYLEVEL       # Min-level: "QUANTILE", "MANUAL"
                                       # or "GREYLEVEL"
MIN_LEVEL              0.001           # Minimum value, quantile or grey level
MAX_TYPE               MANUAL          # Max-level: "QUANTILE" or "MANUAL"
MAX_LEVEL              500,1200,2100,2400,1800,100 # Maximum value or quantile
SATUR_LEVEL            10000000.0         # FITS data saturation level(s)
GAMMA_TYPE             POWER-LAW       # Gamma correction: POWER-LAW, SRGB or
                                       # REC.709
GAMMA                  2.2             # Display gamma
GAMMA_FAC              1.0             # Luminance gamma correction factor
COLOUR_SAT             1.0             # Colour saturation (0.0 = B&W)
NEGATIVE               N               # Make negative of the image

#------------------------------ Memory management -----------------------------

VMEM_DIR               .               # Directory path for swap files
VMEM_MAX               2000000         # Maximum amount of virtual memory (MB)
MEM_MAX                10000           # Maximum amount of usable RAM (MB)

#------------------------------ Miscellaneous ---------------------------------

VERBOSE_TYPE           NORMAL          # QUIET, NORMAL or FULL
COPY_HEADER            Y               # Copy FITS header to description field?
DESCRIPTION            "RXT 2248.7-4431"    # Image content description
COPYRIGHT              "Dark Energy Survey" # Copyright notice
WRITE_XML              Y               # Write XML file (Y/N)?
XML_NAME               stiff.xml       # Filename for XML output
XSL_URL                file:///usr/share/stiff/stiff.xsl
                                       # Filename for XSL style-sheet
NTHREADS               0               # Number of simultaneous threads for
                                       # the SMP version of STIFF
                                       # 0 = automatic
EOF
####
svn co https://dessvn.cosmology.illinois.edu/svn/desdm/devel/terapix/tags/stiff-2.6.1 -r 42258 --username ogando   #### ask the password to ogando and DO NOT STORE IT
cd stiff-2.6.1
./configure
make
cd ..
time ./stiff-2.6.1/src/stiff -c stiff.conf fits/DES2208-0041_g.fits fits/DES2208-0041_r.fits fits/DES2208-0041_i.fits fits/DES2208-0041_z.fits fits/DES2208-0041_Y.fits
### ...

Install Oracle client

Install oracle, copy the archive from ...

http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html

... into the server

cp oracle_software/instantclient-basiclite-linux.x64-12.1.0.2.0.zip /srv/CHROOT_master/tmp/
cp oracle_software/instantclient-sdk-linux.x64-12.1.0.2.0.zip /srv/CHROOT_master/tmp/
cp oracle_software/instantclient-sqlplus-linux.x64-12.1.0.2.0.zip /srv/CHROOT_master/tmp/
chroot /srv/CHROOT_master
mkdir /srv/oracle
chmod 755 /srv/oracle
cd /srv/oracle
unzip /tmp/instantclient-basiclite-linux.x64-12.1.0.2.0.zip
rm /tmp/instantclient-basiclite-linux.x64-12.1.0.2.0.zip
unzip /tmp/instantclient-sdk-linux.x64-12.1.0.2.0.zip
rm /tmp/instantclient-sdk-linux.x64-12.1.0.2.0.zip
unzip /tmp/instantclient-sqlplus-linux.x64-12.1.0.2.0.zip
rm /tmp/instantclient-sqlplus-linux.x64-12.1.0.2.0.zip
chmod 755 instantclient_12_1
cd instantclient_12_1
ln -s libclntsh.so.12.1 libclntsh.so
ln -s libocci.so.12.1 libocci.so
echo /srv/oracle/instantclient_12_1 > /etc/ld.so.conf.d/oracle.conf
ldconfig
apt-get install libaio1
apt-get clean
exit

Install python cx_Oracle

chroot /srv/CHROOT_master
su - des-dri
cd $DRI_HOME
source env/bin/activate
ORACLE_HOME=/srv/oracle/instantclient_12_1 pip3 install --upgrade cx_Oracle

Change the new database entries 'catalog' and 'dessci' to 'api/dri/settings/local_vars.py' file

    'catalog': {
        'ENGINE': 'django.db.backends.oracle',
        'NAME': 'desdb.ncsa.illinois.edu:1521/dessci',
        'USER': '',
        'PASSWORD': '',
    },
    'desoper': {
        'ENGINE': 'django.db.backends.oracle',
        'NAME': 'leovip148.ncsa.uiuc.edu:1521/desoper',
        'USER': '',
        'PASSWORD': '',
    },
    'dessci': {
        'ENGINE': 'django.db.backends.oracle',
        'NAME': 'desdb.ncsa.illinois.edu:1521/dessci',
        'USER': '',
        'PASSWORD': '',
    }

Test

cd api
PATH=$PATH:/srv/oracle/instantclient_12_1 python manage.py dbshell --database=catalog
#SQL> SELECT table_name FROM user_tables;
#SQL> exit
deactivate
exit
exit

Follow the = Update code =

Switch to develop branch * for TESTING env

Switch branch

sudo chroot /srv/CHROOT_master
su - des-dri
cd $DRI_HOME
git checkout develop
cat > api/dri/settings/local.py << 'EOF'
from dri.settings.testing import *
EOF

Follow the = Update code =

Update code

sudo chroot /srv/CHROOT_???
su - des-dri
cd $DRI_HOME
git pull
source env/bin/activate
# do you want an homologation site? then do 'git checkout develop'
#### python manage.py makemigrations    #### [CMP] the makemigrations step can be tricky: it should be done by the developer user and the result should be committed
cd api
pip3 install --upgrade -r requirements.txt
python manage.py migrate                                #### [CMP] the migrate step can ask something to the user
python manage.py migrate catalog --database=catalog     #### [CMP] the migrate step can ask something to the user
#### REMEMBER to eventually update specific data models, e.g. python manage.py loaddata initial_data.json
python manage.py collectstatic --clear --noinput --verbosity 0
# * NOTE, if you have major data update, go to == Import the basic data == , however this procedure should be changed
deactivate
exit
/etc/init.d/apache2 restart
exit
exit

Setup apache reload, via ssh

NOTE: this was done only for testing environment

sudo chroot /srv/CHROOT_???
apt-get install sudo openssh-server
apt-get clean
#### sshd config
/etc/init.d/ssh stop
nano /etc/ssh/sshd_config       #### Change defaul port! e.g. 8081
ln -s /etc/init.d/ssh /srv/config/init.d
ln -s ../init.d/ssh /srv/config/rc.d/S10_ssh
ln -s ../init.d/ssh /srv/config/rc.d/K90_ssh
/srv/config/rc.d/S10_ssh start
#### sudo config
echo 'dri-update:x:10001:10001::/home/des-dri:/srv/domain/desportal.cosmology.illinois.edu/update.sh' >> /etc/passwd
echo 'dri-update:!:16979:0:99999:7:::' >> /etc/shadow
####www-data:x:33:des-dri,dri-update /etc/group
echo "des-dri, dri-update ALL=NOPASSWD: /etc/init.d/apache2 stop, /etc/init.d/apache2 start" >> /etc/sudoers
echo "des-dri, dri-update ALL=NOPASSWD: /etc/init.d/rabbitmq-server stop, /etc/init.d/rabbitmq-server start" >> /etc/sudoers
echo "des-dri, dri-update ALL=NOPASSWD: /etc/init.d/celerybeat stop, /etc/init.d/celerybeat start" >> /etc/sudoers
echo "des-dri, dri-update ALL=NOPASSWD: /etc/init.d/celeryd stop, /etc/init.d/celeryd start" >> /etc/sudoers
su - des-dri
mkdir -p .ssh
chmod 750 .ssh
nano .ssh/authorized_keys       #### ADD your auth keys
exit
exit

Adding des basic user

open http://localhost/dri/api/admin/ in the browser
login as dri-admin/desbrazil
go to http://localhost/dri/api/admin/auth/user/add/
add des/line@70chips

1. 1. - go to http://localhost/dri/api/admin/auth/user/2/change/
    - add NCSA

logout

WARNING: some things changed, login does not work at this point but must!

         using a NCSA version of the db works

Adding DataDiscovery daemon

Configure cron daemon to run by cron every 12 hours

OBSOLETE!!! desativado, foi incluido um parâmetro de entrada, não estamos conseguindo usar de forma sistematica, os releases mudam sempre de estrutura

sudo chroot /srv/CHROOT_???
su - des-dri
crontab -e
#### Add following:
# SHELL=/bin/bash
# [email protected]
# 0 0,12 * * * /srv/domain/desportal.cosmology.illinois.edu/datadiscovey.sh > /srv/domain/desportal.cosmology.illinois.edu/log/datadiscovey.log 2>&1
#### End add
exit
exit

Install Celery system

sudo chroot /srv/CHROOT_???
# update and dist-upgrade
apt-get install rabbitmq-server
apt-get clean
ln -s /etc/init.d/rabbitmq-server /srv/config/init.d
ln -s ../init.d/rabbitmq-server /srv/config/rc.d/S60_rabbitmq-server
ln -s ../init.d/rabbitmq-server /srv/config/rc.d/K40_rabbitmq-server
/etc/init.d/rabbitmq-server restart
su - des-dri
cd $DRI_HOME
source env/bin/activate
pip3 install --upgrade celery==4.0.2
pip3 install --upgrade amqp==2.1.4
exit
wget -c https://raw.githubusercontent.com/celery/celery/4.0/extra/generic-init.d/celerybeat -O /srv/config/init.d/celerybeat
chmod 755 /srv/config/init.d/celerybeat
ln -s ../init.d/celerybeat /srv/config/rc.d/S70_celerybeat
ln -s ../init.d/celerybeat /srv/config/rc.d/K30_celerybeat
wget -c https://raw.githubusercontent.com/celery/celery/4.0/extra/generic-init.d/celeryd -O /srv/config/init.d/celeryd_tmp
cat /srv/config/init.d/celeryd_tmp | sed -e 's|CELERY_DEFAULTS=${CELERY_DEFAULTS:-"$CELERY_CONFIG_DIR/${SCRIPT_NAME}"}|CELERY_DEFAULTS=/etc/default/celeryd|g' > /srv/config/init.d/celeryd
rm -f /srv/config/init.d/celeryd_tmp
chmod 755 /srv/config/init.d/celeryd
ln -s ../init.d/celeryd /srv/config/rc.d/S70_celeryd
ln -s ../init.d/celeryd /srv/config/rc.d/K30_celeryd
cat > /etc/default/celeryd << 'EOF'
CELERYD_NODES="worker1"
CELERY_BIN="/srv/domain/desportal.cosmology.illinois.edu/env/bin/celery"
CELERY_APP="dri"
CELERYD_CHDIR="/srv/domain/desportal.cosmology.illinois.edu/api"
CELERYD_OPTS="--time-limit=300 --concurrency=8"
CELERYD_LOG_LEVEL="INFO"
CELERYD_PID_FILE="/srv/domain/desportal.cosmology.illinois.edu/log/celery/%n.pid"
CELERYD_LOG_FILE="/srv/domain/desportal.cosmology.illinois.edu/log/celery/%n%I.log"
CELERYD_USER="des-dri"
CELERYD_GROUP="des-dri"
CELERY_CREATE_DIRS=1
EOF
/srv/config/init.d/celeryd start
/srv/config/init.d/celerybeat start
ln -s /srv/config/init.d/celeryd /etc/init.d/
ln -s /srv/config/init.d/celerybeat /etc/init.d/
exit

Install Wordpress Home site

Php for wordpress

from https://packages.debian.org/jessie/wordpress

sudo chroot /srv/CHROOT_???
# update and dist-upgrade
apt-get install libapache2-mod-php5 php5-gd php5-mysql libphp-phpmailer
apt-get clean
a2enmod rewrite

Mysql

Install

# install mysql
apt-get install mysql-server  # choose a root password #
apt-get clean
# config mysql to start at boot
ln -s /etc/init.d/mysql /srv/config/init.d
ln -s ../init.d/mysql /srv/config/rc.d/S30_mysql
ln -s ../init.d/mysql /srv/config/rc.d/K70_mysql

Setup

mysql -u root -p
mysql> create database `dri-dev` CHARACTER SET 'utf8';
mysql> create user 'wordpress'@'localhost';
mysql> set password for 'wordpress'@'localhost' = password('YOUR_PASSWORD');
mysql> grant all on `dri-dev`.* to 'wordpress'@'localhost';
mysql> quit;

Database restore from site backup

mysql -u wordpress -p dri-dev < dri-dev.sql

Site install from site backup

rm -f /srv/domain/desportal.cosmology.illinois.edu/apps/home   # this should not be necessary anymore
mv dri /srv/domain/desportal.cosmology.illinois.edu/apps/home
chown -R des-dri:des-dri /srv/domain/desportal.cosmology.illinois.edu/apps/home

Database site configuration
NOTE: in the follow example we are using testing URL (:8080), and the http://dri-dev.linea.gov.br as original backup URL

mysql -u wordpress -p dri-dev
mysql> update wp_options set option_value = replace(option_value, 'http://dri-dev.linea.gov.br', 'http://desportal.cosmology.illinois.edu:8080/dri/apps/home/') where option_name = 'home' or option_name = 'siteurl';
mysql> update wp_posts set guid = replace (guid, 'http://dri-dev.linea.gov.br', 'http://desportal.cosmology.illinois.edu:8080/dri/apps/home/');
mysql> update wp_posts set post_content = replace(post_content, 'http://dri-dev.linea.gov.br', 'http://desportal.cosmology.illinois.edu:8080/dri/apps/home/');
mysql> update wp_postmeta set meta_value = replace(meta_value, 'http://dri-dev.linea.gov.br', 'http://desportal.cosmology.illinois.edu:8080/dri/apps/home/') where meta_key = '_menu_item_url';
mysql> quit

Roteiro desenvolvidor a partir da imagem de produção

id -u > /tmp/MYUSER
id -g > /tmp/MYGROUP
sudo su -
mkdir -p -m 0755 /storage/data /srv
cd /srv
wget http://devel2.linea.gov.br/~riccardo.campisano/dri_images/DEBIAN_jessie_amd64_DRI-master_2016-06-29.tgz
tar -xzf /srv/DEBIAN_jessie_amd64_DRI-master_2016-06-22.tgz
mv DEBIAN_jessie_amd64_master DEBIAN_jessie_amd64_develop
ln -s DEBIAN_jessie_amd64_develop CHROOT_develop
ln -s /srv/CHROOT_develop/srv/config/chroot.sh /etc/init.d/chroot_develop.sh
nano /etc/init.d/chroot_develop.sh   # sobstitute 'CHROOT=/srv/CHROOT_master;' to 'CHROOT=/srv/CHROOT_develop;'
update-rc.d chroot_develop.sh defaults
# for centos:   chkconfig chroot_develop.sh on
cat /etc/resolv.conf > /srv/CHROOT_develop/etc/resolv.conf
hostname > /srv/CHROOT_develop/etc/hostname
echo 127.0.0.1       localhost `hostname` > /srv/CHROOT_develop/etc/hosts
/etc/init.d/chroot_develop.sh start
mv /tmp/MYUSER /srv/CHROOT_develop/tmp
mv /tmp/MYGROUP /srv/CHROOT_develop/tmp
chroot /srv/CHROOT_develop
export MYUSER=`cat /tmp/MYUSER`
export MYGROUP=`cat /tmp/MYGROUP`
rm -f /tmp/MYUSER
rm -f /tmp/MYGROUP
####grep -q "\:$MYGROUP\:" /etc/group || groupadd --gid $MYGROUP dri-dev
####export MYGROUPNAME=`grep "\:$MYGROUP\:" /etc/group | cut -d ':' -f 1`
####usermod -a -G $MYGROUPNAME des-dri
####usermod -a -G $MYGROUPNAME www-data
nano /etc/passwd # use $MYUSER:$MYGROUP
nano /etc/group # use $MYGROUP
chown -R des-dri:des-dri /home/des-dri/
chown -R des-dri:des-dri /srv/domain/desportal.cosmology.illinois.edu/
su - des-dri
cd $DRI_HOME
git reset --hard HEAD
git fetch
git checkout develop
git pull
source env/bin/activate
cd api
pip3 install --upgrade -r requirements_dev.txt
python manage.py migrate                                #### [CMP] the migrate step can ask something to the user
python manage.py migrate catalog --database=catalog     #### [CMP] the migrate step can ask something to the user
python manage.py loaddata initial_data.json
python manage.py collectstatic --clear --noinput
####
cat > dri/settings/local.py << 'EOF'
from dri.settings.development import *
EOF
####
deactivate
exit
/etc/init.d/apache2 restart
exit
exit
ln -s /srv/CHROOT_develop/srv/domain/desportal.cosmology.illinois.edu/ ~/Desktop/

Server

Restart procedure

mount points to be remounted manually - /des001 - /des002 - /des003 - /des006 - /des008

for any similar problem, stop all, umount all, mount all and start all:

sudo /etc/init.d/chroot_master.sh stop sudo /etc/init.d/chroot_develop.sh stop sudo /etc/init.d/chroot_masterlist.sh stop umount -f -l /des001 umount -f -l /des002 umount -f -l /des003 umount -f -l /des006 umount -f -l /des008 mount /des001 mount /des002 mount /des003 mount /des006 mount /des008 sudo /etc/init.d/chroot_master.sh start sudo /etc/init.d/chroot_develop.sh start sudo /etc/init.d/chroot_masterlist.sh start

= Ports per image

firewall allowed port range: 22, 80, 8080-8090
main sshd: 22

DRI master

apache 80 iipsrv 9000 sshd none

DRI develop

apache 8080 iipsrv 9080 sshd 8081

DRI masterlist

apache 8088 sshd 8089

References

http://www.campisano.org/wiki/en/Debian_chrooted_server
http://test-driven-django-development.readthedocs.org/en/latest/
http://arunrocks.com/understanding-tdd-with-django/
https://github.com/MaZderMind/django-vs-extjs
https://www.vultr.com/docs/setup-django-on-debian-8
http://www.django-rest-framework.org/api-guide/testing/

Chroot how to - linea-it/dri GitHub Wiki

Table of Contents

Deploy

Prepare a debian chrooted (Optional)

Prerequisite

Config Apache and Django project with wsgi.py

Import the basic data

Add VisiOmatic iipsrv server

Install IIPsrv

Configure Apache for iipserv

Install Oracle client

Switch to develop branch * for TESTING env

Update code

Setup apache reload, via ssh

Adding des basic user

Adding DataDiscovery daemon

Configure cron daemon to run by cron every 12 hours

Install Celery system

Install Wordpress Home site

Php for wordpress

Mysql

Roteiro desenvolvidor a partir da imagem de produção

Server

Restart procedure

= Ports per image

DRI master

DRI develop

DRI masterlist

References

⚠️ GitHub.com Fallback ⚠️

Chroot how to - linea-it/dri GitHub Wiki

Table of Contents

Deploy

Prepare a debian chrooted (Optional)

Prerequisite

Config Apache and Django project with wsgi.py

Import the basic data

Add VisiOmatic iipsrv server

Install IIPsrv

Configure Apache for iipserv

Install Oracle client

Switch to develop branch * for TESTING env

Update code

Setup apache reload, via ssh

Adding des basic user

Adding DataDiscovery daemon

Configure cron daemon to run by cron every 12 hours

Install Celery system

Install Wordpress Home site

Php for wordpress

Mysql

Roteiro desenvolvidor a partir da imagem de produção

Server

Restart procedure

= Ports per image

DRI master

DRI develop

DRI masterlist

References

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️