Server refactoring

At the time of the pilot competition in July 2011, there is a very strong interest in transitioning the server away from Grails/Groovy to Spring/Java, and in developing an agent platform in straight Java. The reasons for this are many:

• Architectural quality -- Grails is a web-app framework, optimized for webserver interaction patterns. Power TAC is a long-running simulation with a simple web front-end, and a (conceptually) separate visualizer. It's a poor match, and many Grails features have been a clear hindrance to achieving our goals. For example, the breakdown into domain, services, controllers, and views has not been particularly useful for much of what we are doing.
• Robustness -- The Power TAC server is a somewhat flimsy prototype at this point, and it needs to become a rock-solid simulation platform that can run for months at a time without getting into trouble. The qualities that make Grails/Groovy a good platform for quick-response Web programming seem to be antithetical to this kind of robustness.
• Accessibility -- Student developers need to be productive on specific problems with a relatively low learning curve. Most students know Java and can learn a component API, but do not know Groovy or Grails, both of which have significant learning curves, especially on the debugging side. Even experienced Grails developers are often mystified by the error messages they see.
• Performance -- Groovy's casual programming style comes at the cost of a very significant performance penalty. This has already been an issue, and will be a bigger issue as models become more complex.

Specific decisions and details on process and conventions will be added as they are derived and tested.

Tasks and assignments for getting the refactoring finished.

Goal

The goal of this phase of the project, then, is to migrate the existing Power TAC server away from Grails toward a pure Java platform. Ideally this can be done incrementally so the server will remain usable during the transition, we can use the existing server as an integration-testing framework, and the project will not be in jeopardy if some portions are not refactored by some deadline date.

Principles

1. Extensibility demands a robust, well-supported and well-documented IOC (inversion of control) component framework. An example is Spring, which is also the foundation of Grails.
2. Each simulation must record enough information about its operation to enable ex post data analysis to reconstruct (relevant) simulation events. Relevant events are those that immediately or eventually produce broker-visible effects, possibly with the exception of clock pauses. Note that this requirement does not imply the need for a relational database.
3. Since server state recording is not known to be complete at this point, it is acceptable if full server state reconstruction is infeasible during the refactoring process.
4. In order to manage the learning curve, the basic server and all its APIs will be expressed in Java and Java annotations. Other languages (Groovy, Clojure, etc.) can be used in experimental modules for specific research purposes, but all the functionality of the basic server will be implemented in Java.
5. Dependencies among modules must be strictly controlled to avoid dependency cycles and minimize the cognitive burden of understanding the system. Lower-level modules should have no need to know about the structure or even the existence of higher-level modules. This will likely require liberal use of IOC patterns.
6. The use of xml serialization for broker interaction should be preserved in order to maximize flexibility in technology choices for broker developers.
7. Objects intended to be serialized as xml should be value holders, with minimal behavior that is not directly related to their (abstract) structures. Behavior specific to server or brokers can then be "wrapped" around these data holders as necessary.
8. Data representation in domain objects should not be changed unless necessary, to minimize impact on broker developers. When changes are necessary, they need to be clearly documented, with code when feasible. Methods will of course change in many cases.

Ideas

• Before we can eliminate the database, we must catalog existing relations and queries and develop a small number of "standard" refactorings for them.
• Server state recording can be a separate process from the existing server logging, which is primarily intended to support understanding and debugging the server. This means that state recording can be done with binary serialized objects. It also means that some thought must be given to cascade effects in serializable objects. The SCM server uses a binary format for its .slg files, but it's not serialized objects; instead it's events (method calls in many cases) and their associated data. There has been some discussion between John and Prashant about the idea of merging the "trace" log and the "data" log, because trace information could be informative in understanding the data. The downside is that we are then potentially stuck with huge logfiles for each game, when all most researchers would need is the data.
• It might be easier to convert services (like TariffMarket and Accounting) after the domain types are converted.
• We could start by changing domain types in grails-app/domain to java objects in src/java within existing modules. Update: probably this would be a waste of time, because too much of the existing structure depends on Grails and will need to be changed.
• Possibly the first task is to work out the state logging process, because it will need to be invoked in all the places where domain objects are created or modified. With that in hand, we can start converting domain objects.
• We may want to specify visualizer requirements as early as possible to avoid lock-out situations - for example, in the current setup we are missing a message type for consumption broadcasting which also needs to be done by customer models. (JEC - This info is in the tariff transactions.)

Practical issues

It's not clear that an incremental refactoring will work. Grails may be built on Spring, but it is not really designed to co-exist with Spring components. If we drop the database, none of the existing services will work without significant change. The directory structure of components is different, and potentially much simpler for the Spring versions. Spring does not support anything quite like Grails plugins. Much of the dependency structure for the existing plugins is really only needed for running tests, and is ignored when running the simulator.

Better, perhaps, to just get a basic Spring framework running with the Clock and CompetitionController, and then start pulling pieces over by converting to Java, replacing the database functionality with an in-memory repository design, adding the state logging as we go.

Testing in the Spring/Java environment is a bit different from Grails, partly because the "integration tests" are not automated to the same extent, and partly because dependency relationships are more rigid.