Development policies - Pyosch/powertac-server GitHub Wiki

Development Policies

Writing Code

License

Project code is licensed under the Apache License, version 2.0. All modules, as well as other code-like artifacts, need to contain a license notice at the top in appropriate comment format. For Java/Groovy modules, the notice shall read as follows:

/*
 * Copyright (c) <current-year> by the original author or authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

This works as long as the actual author(s) are identified further down in the code commentary; otherwise it's fine to replace "original author" with actual names.

NOTE: This may not be correct for work done under grant sponsorship in the U.S. because of University copyright policies. Please check to make sure you can claim copyright personally; otherwise you will need to claim copyright for your institution instead of yourself.

In IntelliJ Idea, go to Settings > Copyright > Copyright Profiles and create a new profile with the copyright text that is not wrapped into a Java comment. In Settings > Copyright add a new entry with the scope All and save. This will automatically add the copyright information to files you create or are working on.

In Eclipse (or STS), simply load powerac-server/powertac-style.xml.

Version Control

If you do not have access credentials to the project repos, you will have to fork the module(s) you want to work on, make and test your changes, and then issue a pull request to have your changes incorporated in the project repo. Otherwise, after cloning the git repository, it's a good idea to start a new local branch before you start working. When you're done with your work you may pull the latest changes from the server into your master branch, then merge your working branch with the master branch and finally commit and push the master branch.

For larger changes or inexperienced/new developers or if you just want another set of eyes to look at your work, it's a very good idea to submit changes through pull requests instead of committing directly to the master branch.

Coding Guidelines

  • Please follow our Coding Guidelines.
  • Consistency and readability are at least as important as performance, except in very unusual circumstances. If you find yourself in one of those circumstances, please include commentary that explains why you felt a need to diverge from project standards.
  • For small standalone tools (shell scripts, Python programs), include a header that explains how to use your code.
  • For all other code, include a header comment that explains what the code is, what role it plays, and how to use it.

Unit Testing

For server and broker modules, we use JUnit 4.x as a unit-testing framework. For modules using the Maven directory layout, each class under src/test/java will be assumed to be a unit test if it has at least one method annotated with @Test. Maven will run them in nor particular order given the test goal (as in mvn clean test) or any goal that includes test as a prerequisite.

Unit tests are normally packaged in the same Java packages as their targets, to allow access to package-visible resources in units under test. Thus it often makes sense to provide test-accessible methods (and sometimes fields) in your production code under package visibility. Most modern IDEs will generate test stubs for you, given a class with a set of methods. However, good unit testing is generally focused on features rather than just on method coverage, so a better approach is to write test methods for each significant feature, and test methods for simple setters and getters are less important.

Package (namespace) usage

The top-level package name for all Power TAC code will be org.powertac. Because of the large number of institutions involved in development, it makes no sense to use the names of individual institutions to name packages, and the project owns the powertac.org domain.

Dependency management

We are using Apache Maven for dependency management and build automation. One place to start learning it is Maven by Example. Specific tips and procedures for its use are at Using Maven.

Third-Party packages

Third-party components may be used after careful testing and after consideration of architectural impact, as long as they are automatically resolvable with the maven dependency resolution process. If a component is going to be used by multiple Power TAC components, be sure you test with all of them, and please keep dependency versions in sync (in other words, we don't want two different versions of the same dependency to get sucked into a build).

Issue Tracking

We use GitHub's issue tracking system on the powertac-server repo.

Raising an issue

If you see something that does not work, something that needs to be done, or something that seems incorrect or confusing, and there is not already an issue that describes the problem, please create a new issue. Describe the problem as clearly as you can, and include relevant error messages and details on how you can re-create the problem. If you suspect the problem is due to your own configuration or lack of preparation (it's a complex system, so that applies to all of us on occasion), then you may of course ask for clarification on the developers Nabble list first.

When you raise an issue, do your best to classify it by attaching the appropriate green and red labels. If you think you need a label that's not there, then you may add a label, but it's also important to keep the number of labels under control.

Clarifying and refining an issue

If you see an issue for which you have ideas or information to contribute, please feel free to add comments to the issue. Just make sure that you poke the "comment" button and not the "comment and close" button when you are finished. Note that you can link wiki pages from issue comments as ../wiki/PageName.

Assigning issues

If you see an issue that you would like to work on, feel free to assign it to yourself. Important issues that are not claimed by project members will be assigned by project leadership.

Working an issue

Generally, this involves a series of steps, many of which should result in comments added to the issue so we can see your status.

  • Understand and confirm the issue. For software defects and enhancement requests, the best way to do this is to write a test case that demonstrates the issue, under the assumption that software defects are indications of missing tests. There can be several outcomes to this process:
    • It's not really an issue, but a misunderstanding or miscommunication. In this case, it may make sense to treat it as a documentation issue, or simply to close it after clearing up the miscommunication and adding an appropriate comment to the issue.
    • It's a duplicate of some other issue. Add a comment with the other issue number, and close it.
    • There is some other issue that must be understood and resolved before progress can be made. If this issue is already posted, then add a comment about the dependency, otherwise raise the new issue and note the dependency in the original issue. It may make sense at this point to remove the assignment.
    • It's one that you understand and can start work on. If appropriate, add a comment that describes your plan for resolving the issue, and gives an estimate of when you expect to complete the work.
  • Before modifying or adding code, create an environment for the work that will not impact others until you have completed and tested the fix. If you have commit access to the github repo, that means creating a local branch for the work with git checkout -b branch-name. Otherwise you will need to work on your own fork of the repository. You may need to update your fork before starting work.
  • Develop the fix. This may involve software and unit-test changes as well as documentation changes. Make sure you have included one or more test cases that failed before the fix and succeed after the fix. That way the same problem won't arise again.
  • Once your fix is complete, add a comment to the issue describing what you did and the outcome of your testing. Also make sure any relevant documentation has been updated to reflect the changes you have made.
  • Issue a pull request to get your changes incorporated into the trunk.
  • The module maintainer will inspect your work and run tests before merging your work into the trunk, and will then close the issue.

Dealing with unexpected problems

If you have made a completion estimate and are unable to meet it, please add a comment that either gives a new completion estimate, or explains why you are unable to make progress. Please do this before the expected completion date if possible. If necessary, we can defer (lower the priority) or reassign the issue in these cases.

Resolving an issue

For all but the most trivial issues, the fix should be confirmed by a second person before the issue is closed. If the fix is the subject of a pull request, then the second person is the one responding to the pull request. In any case, the issue should be closed with a comment describing how the fix was confirmed.

Testing

Thoughtful design and careful implementation is the first line of defense against defects. Next is attention to compiler errors and warnings. Careful unit and integration testing is also a strong defense against nasty surprises at the system level.

Committing Code

  • Get familiar with Git and GitHub.
  • All substantial work, especially work that involves multiple modules, should be done on a branch or fork (forks are generally easier), and preferably merged by someone else, for two reasons: (1) it allows you to commit partially-completed work and share development with a colleague before the work is complete, and (2) it forces someone else to look at your code and run your tests before it becomes generally available.
  • When new work or bugfixes are ready for general consumption, issue a pull request on the fork, and one of the "owner group" will do the inspection, merge, and testing, and commit your changes to the master branch (the development trunk).
  • In order to commit code, you should be a collaborator on the repository you are working on. You can always commit code to your own fork. In order to become a collaborator on one of the shared archives, ask for collaboration rights.
  • If you intend to push changes to a shared master branch rather than your own fork, be sure you know what you are doing and completely understand the consequences.

Local installation and SNAPSHOT deployment

After updating server code, it is important to run full-system tests in a local development environment. As long as code depends on the current SNAPSHOT version, the Maven install goal will install artifacts in your local Maven repository ~/.m2/repository where they will override downloads from the Sonatype SNAPSHOT repository. If anything in powertac-core has been changed, then in general it is necessary to do this in two steps: powertac-core, powertac-server. Installation is done with maven:

  mvn clean install

SNAPSHOT deployments are done by substituting deploy for install in these commands, except that you need credentials for committing to the Sonatype Nexus system. Note that whenever any of the modules in powertac-core is re-installed or re-deployed, you must also install or deploy powertac-server, because otherwise the updates to common or other powertac-core modules will not show up in the running code. It is a very bad idea to do partial deployments because of possible negative impacts on colleagues.

Releasing code and other artifacts

A release is a stable foundation on which our stakeholders can base their work. A particular release may contain all or many of these elements:

  • A release identifier, typically of the form a.b.c, where a is incremented for major releases, b for minor releases, and c for bugfix releases. The first release of the Java/Spring version is 0.1.0. The version for the 2019 competition is 1.6.x.
  • A compatible set of versions of all repos needed to run the server and sample broker (powertac-core, powertac-server, server-distribution, sample-broker). The top-level modules must be labeled with the release identifier.
  • A written description of the release, (here's an example) with a list of features and known issues. The issues should all be in the github issue tracker, and the release notes should link to the individual issues. The feature list in most cases should concentrate on differences from the previous release.
  • A tag, with the name of the release, in the git repos of all modules included in the release.
  • For all major releases, a new branch called release-x.y in the git repos of all included modules. This is the branch on which maintenance of the release will take place, from where bugfix releases will originate.
  • A set of jars, including binary, source, and javadoc, as required for promotion to Maven Central.
  • A release of the server-distribution module. This is created and distributed through the server-distribution release page. Sample-broker is also distributed in source form, and therefore is treated as server-distribution.
  • An update of the game specification, if necessary, that matches the code.

Creating a release package

A release package contains packaged versions of the server with its components, the broker framework, and the common package, along with descriptive documentation in the form of release notes that describe the capabilities of the release and list known outstanding issues.

We use the Sonatype maven repo for snapshot artifacts, and Maven Central for release artifacts. The process of posting snapshots and releases is described in detail on the Sonatype site. If you need access to post snapshots and release packages, please contact John. Once you have access, you will also need to register your code-signing certificate with your Sonatype account. For the code, the maven-release-plugin takes care of generating many of the artifacts required for a release. It does not generate the distribution package for the server or the source package for the sample broker.

Released artifacts cannot depend on SNAPSHOT artifacts. Major and minor releases are always created from the release branch created for the major release; and bugfix snapshots and releases are always created from the release branch. In order to satisfy dependencies at each stage of the release process, the server must be released in three stages:

  1. powertac-core comes first, unless a server release is being made without a corresponding release of one or more of the powertac-core modules. Start by creating the release branch; the rest of the process takes place on the branch. The standard maven release:prepare - release:perform process works well.
  2. Close and release powertac-core using the Sonatype Nexus server before proceeding.
  3. If the release fails, you will need to abandon the commits made during the failed release process. This is easily done by renaming the release branch with git branch -m branch-name new-name and then creating a new release branch for the next attempt. You will then need to remove the tags from your local environment, and remove the release and the release branch from github. You also need to remove release artifacts from the master branch, specifically pom.xml.releaseBackup (at the top level and in every submodule) and release.properties.
  4. Next we need to prepare powertac-server for release. On the release branch, it should be enough to edit powertac-server/pom.xml and fix the dependencies on powertac-parent to the release version (there should be two of these; one is the powertac.version property). Don't update other dependencies at this point.
  5. Before you can release powertac-server, you need to create a directory that the release process can use to dump the site artifact. In powertac-server/pom.xml, this is specified in the clause distributionManagement/site. Currently it requires a writeable directory /usr/local/share/powertac-site. If you cannot create such a directory, you will need to modify pom.xml (and commit and push it) before proceeding.
  6. Commit the updated powertac-server module and push it to github as git push origin release-branch.
  7. The powertac-server super-module is released in two steps using release:prepare and release:perform.
  8. Close and release powertac-server on the Sonatype Nexus server.
  9. The server-distribution module is not released using maven; rather, it's a download package. So all that's necessary is to create the release branch, edit the dependencies by hand, test, commit, tag (the tag name should be just the release ID, such as "1.7.1"), and push both the commit and the tag. At this point, you should be able to go to the server-distribution and see the new release (select tags, then switch to releases). It's still necessary to "create a release" using the tag; make sure the correct release artifacts are listed.
  10. The sample-broker module is distributed in source form, so if it is being released, it also needs to be hand-edited on the release branch to update dependencies, tagged, committed, and pushed.
  11. Push all changes up to github, including the tags (use git push --tags).
  12. If you have created a major release on a branch release-x.y, then you will need to update the master branch of powertac-core, powertac-server, server-distribution, and sample-broker to x.y+1.0-SNAPSHOT. In powertac-core and powertac-server you will need to make this change in all the submodules.
  13. Deploy the new snapshot versions of powertac-core and powertac-server.

Once all the components are available on Maven Central, you can create and post the actual release package. You do that by tagging the server-distribution release version, then create the release in github.

Post-release development

After a release, there will always be a main development branch for work on the next release, and a release branch for maintenance of the release. Depending on the project state, after a release x.y.z, the pom versions on the main development branch (the master branch) will be x.y+1.0-SNAPSHOT. The pom versions on the maintenance release branch will start at x.y.1-SNAPSHOT, and only the bugfix id will be incremented as new versions of individual modules are released. The release branch will be named release-x.y.

Maintaining backward compatibility

A core idea of the competitive-simulation approach to research is that broker implementations can be shared among the community. In many cases, these shared brokers will be binary packages rather than sources, either because research groups don't want to give away their code, or perhaps because they don't want to have to document it for others. As a result, it is very important to maintain backward compatibility between the broker and server, such that older broker implementations can work with newer versions of the server. There are several basic practices that can maintain backward compatibility, although in general it's very hard to test for without giving server developers in-depth knowledge of individual broker designs and their data and sequence dependencies. At the very least, server developers should observe at least the following principles when making changes or adding features to the server:

  1. Existing message types must not be removed.
  2. Data fields in messages should not be removed, nor should their meanings be changed. It should be OK to add fields; XStream should ignore extra fields when deserializing messages. If in doubt, test.
  3. Remember that not all the messages sent to brokers are in the common package. Almost all of them are either in the common or common.msg packages, but server configuration is transmitted to brokers in the form of a Properties instance.
  4. Server configuration data is generated for a module just in case (1) there are fields or setter/getter methods decorated with @ConfigurableValue annotations having publish = true; and (2) the module calls serverProps.publishConfiguration(this). So if it is necessary to move, rename, or change the meaning of a published bit of configuration, the original name and semantics must be retained, keeping in mind that the property name is generated from the package and classname of the module that originally published the item.