Understanding the 8.3 Controller - tooltwist/documentation GitHub Wiki

In v8.3 of ToolTwist, to Controller functionality has a major change.

Phil August 2014: This needs updating

Previously the Controller pulled source from git and compiled it from scratch using Ant scripts.

In v8.3 the Controller uses Gradle to build and run the server, and then Grunt to post-process the website files before transferring them to the remote server. This is all initiated using the tooltwist command line tool.

This is a transitional version, part old and part new, so the process flow is a little convoluted. However the transition to a streamlined process is straightforward and will occur in the next version.

1. The entry point is now the tooltwist command line, which can be called directly from Hudson.

    tooltwist --controller <launchpad-name>
   

This must be run in the launchpad directory /ControllerV8/launchpads/<launchpad-name>.

This downloads a webdesign project from git as defined by the payloads mapped onto the launchpad (stored in deploymentConfig.launchpad.xml). The difference now is that everything in the payload mapping is ignored except the webdesign project. The extension projects are determined by a file named project.json in top level of the webdesign project.

2. The tooltwist command checks that prerequisite software is installed, including Java, Gradle, ImageMagick, and Ruby.

3. Config files get created using Mustache templates. By default the templates in the directory /usr/local/lib/node_modules/tooltwist/templates. If you wish to override any of these templates, create a directory named /ControllerV8/config_files/templates.<launchpad-name> and place the modified template there, then run tooltwist with `--template-dir .

4. The tooltwist command starts Gradle to build the tomcat server using JAR files downloaded from Gradle and Maven repositories. , and trigger the code generation process.

5. Gradle calls ${TTC_CONFIG}/bin/launchpad-8.3.sh <launchpad-name> generate to generate the website, using a similarly named Ruby script. This is a stripped down version of the previous script in v8.2. The workload is divided into threads according to the launchpad configuration, and depending upon how many navpoints there are.

• Calls http://localhost:<port>/ttsvr/listNavpoints to get the navpoints
• Creates filter definitions in /ControllerV8/launchpad/<launchpad-name>/filters
• Starts multiple threads, each calling curl to call http://localhost:<port>/ttsvr/generator.
• Sends an email if an error occurs.

6. The generator servlet in the Tomcat server (still running within Gradle) runs the filter plugins defined in the filter files. This includes

• installing WebContent files
• widgets generating web assets
• generating navpoints
• converting URLS to a different webapp.

7. The tooltwist command calls npm install and then grunt to do file processing.

• copies jar files to image directory
• other user-defined processing, if template.Gruntfile.js is provided.

8. ZZZ _This might have changed._The tooltwist command calls gradle tooltwist/installerUtil to install the files to the remote server using it's matching Ruby script. This script in turn calls ${TTC_CONFIG}/bin/installLaunchpad <launchpad-name> which uses the Java utility java -cp ... tooltwist.controller.util.InstallerUtil <launchpad-name>. If an error occurs an email is sent.

  Hudson
    +-- tooltwist --controller <launchpad-name>
         +-- Check prerequisites
         +-- Create config files and directories
         |

    (BUILD PHASE)

         |
         +-- Gradle
         |    +-- Download jar files
         |    +-- Build Tomcat server
         |    +-- Explode WAR file into 'exploded'
         |    +-- Extract config, widget and WebContent from extension project jar files
         |    +-- Start Tomcat server
         |    |

    (GENERATE PHASE)

         |    |
         |    +-- ${TTC_CONFIG}/bin/launchpad-8.3.sh <launchpad-name> generate
         |    |    +-- ${TTC_CONFIG}/bin/launchpad-8.3.rb
         |    |    |    +-- http://localhost:<port>/ttsvr/listNavpoints
         |    |    |    +-- filters placed in /ControllerV8/launchpad/<launchpad-name>/filters
         |    |    |    +-- http://localhost:<port>/ttsvr/generator
         |    |    |         +-- install WebContent files
         |    |    |         +-- widgets generate web assets (e.g cropImages and stylesheets)
         |    |    |         +-- navpoints generate web assets (eg. frame sprites)
         |    |    |         +-- converting URLS to a different webapp.
         |    |    +-- send email if error
         |    +-- Stop Tomcat server
         |

    (GRUNT PHASE)

         |
         +-- npm install, to check Grunt has modules it needs
         |    +-- grunt
         |         +-- copy jars from 'exploded' to image/tomcat/webapps/ttsvr/WEB-INF/lib
         |

    (TRANSFER PHASE)

         |
         +-- ${TTC_CONFIG}/bin/launchpad-8.3.sh <launchpad-name> transfer
         +    +-- ${TTC_CONFIG}/bin/launchpad-8.3.rb
         |    |   +-- Run Gradle in tooltwist/installerUtil
         |    +-- send email if error
         |
       Finish

Extension projecs jar files contain web content files. These get extracted to extension-projects/<project>/WebContent and during the generate process have URLs converted and get copied to image/tomcat/webapps/<webapp>.

Currently each web content file gets deployed twice - once in the extension project jar, and once in exploded format. One has URLs converted while one does not. By default Tomcat will use the exploded (and converted) file. To be consistent however, one of the following should occur:

• Place the converted webcontent files back into the extension project jar, and don't deploy them in exploded form. This results in easier handling, with just jar files.

• Remove the web content files from the jar file. This results in more efficient incremental deployment.

Server-side configuration is now done on the production server. We no longer attempt to create deployed config files on the Controller.

• Config files are now placed in ~/site-conf/conf, including filemap.xml, logback.xml, tooltwist.xml, and wbd.conf.

• The directories under ~/server/tomcat do not get changed, except for under webapps. Of most importance, ~/server/tomcat/conf/* do not get changed.

These files must get set up by hand, and do not get changed when the controller transfers files to the remote machine.

If you wish to protect other files being modified or removed you can change the transfer rules defined on the Controller at /ControllerV8/config-files/rules.<launchpad-name>.txt.

An alternate way to prevent files being changed is to add an entry to ~/server/.fip-destination.

The Tomcat log file is now written to ~/site-conf/logs/tooltwist.log. The log file is moved to a date-related filename and restarted every 24 hours.

--