Contributing - oilshell/oil GitHub Wiki

NOTE: If anything on this page doesn't work for you, please file a bug!

(Related: Oil Native Quick Start / Where Contributors Have Problems / Oil Dev Cheat Sheet / Oil Dev Tips / Making Pull Requests)

Requirements

You should use Linux to make changes to OSH and test them. One reason for this is that the spec tests run against four other shells (bash/dash/zsh/mksh), and we don't have a way of building the right versions of these shells on other OSes.

When I'm on a Mac, I use Ubuntu on Virtualbox to develop.

  • Note that OSH does work on OS X for end users, but development is a different story.
  • 3/2020 update: there's a shell.nix in the root of the repo that may work on OS X. Ping us on Zulip if you need help with it. (Developing Oil With Nix)

Quick Start with "minimal" build (~2 minutes)

Note that this is the minimal dev build, which is very different than the release build, due to Oil's use of code generation! (In particular, don't run ./configure when developing Oil.)

git clone https://github.com/oilshell/oil.git    # please use this repo, not your own fork
                                                 # this helps us use a Github Action secret on PRs

cd oil                    # enter the base directory

build/py.sh ubuntu-deps   # Python 2 dev headers, etc.
                          # Uses sudo apt-get, so it's for Ubuntu/Debian.
                          # may need python2-dev instead of python-dev
build/py.sh minimal       # MINIMAL dev build, including native/libc.c Python extension

bin/osh                   # sanity check to make sure it works. 
                          # Make sure you have Python 2 installed.
                          # Ctrl-D to quit.

test/unit.sh unit frontend/lexer_test.py     # Run an individual unit test
test/unit.sh minimal      # run most unit tests, takes about a second or two

test/spec.sh smoke        # run a single file; there are more but this is a good start

This is all you need to make most changes to Oil.

Note on oilshell/oil vs. forks

Right now we have some awkwardness due to CI permissions.

We want you to send PRs from oilshell/oil and not your own fork, so that Github Actions can get our $TOIL_KEY, and publish HTML results to http://travis-ci.oilshell.org/ .

But this is a hiccup for first time contributors. I have to add you to the Github project so the CI can run. Ping me on Zulip!

Full Python build: build/py.sh all

I left a few things out of build/dev.sh minimal to make it easier to get started:

  1. It doesn't generate a lexer with re2c.
  2. It doesn't generate help text, which depends on CommonMark.

Once you have that working, please do the following:

build/deps.sh fetch  # re2c, CommonMark, etc.
build/deps.sh install-wedges

Now you're ready to run:

build/py.sh all

This is also documented in Oil Dev Cheat Sheet

Running Tests

There are many different kinds of tests: unit tests and spec tests are the most important.

Unit Tests

Now that you have the full Python build, run unit tests like this:

$ test/unit.sh all

You can run a single test like this:

$ test/unit.sh unit osh/word_parse_test.py

test/unit.sh is a simple wrapper that sets PYTHONPATH.

Spec Tests

Spec Tests test Oil against specific versions of existing shells via the test/sh_spec.py framework, and require more setup. Ping us on Zulip if you have problems.

First, set up the spec tests as described in test/spec-bin.sh and Spec Tests:

$ test/spec-bin.sh download     # Get the right version of every tarball                                                                                     
$ test/spec-bin.sh extract-all  # Extract source                                                                                                             
$ test/spec-bin.sh build-all    # Compile                                                                                                                    
$ test/spec-bin.sh copy-all     # Put them in _deps/spec-bin                                                                                                 
$ test/spec-bin.sh test-all     # Run a small smoke test

Now you can run the spec tests.

$ test/spec-py.sh osh-all     # all OSH tests in parallel
$ test/spec-py.sh ysh-all     # all YSH tests in parallel
$ test/spec.sh smoke       # a single file -- look at the list of functions

Output is in _tmp/spec.

Cheat Sheet

See Oil Dev Cheat Sheet for a quick summary of these commands.

Other Test Suites

There are many other test suites which are reported on:

  1. The /release/$VERSION/quality.html page on every release
  2. The "Soil" continuous build at http://travis-ci.oilshell.org/github-jobs/

For example, you can run

test/parse-errors.sh all

To see a big demonstration of parse errors. I use this to supplement the spec tests when changing the syntax of the language.

Working on C++: oils-cpp

If you did the above, then the Python bin/osh should be working.

If you want to work on C++, then you should get _bin/cxx-asan/osh to work. See Oil Native Quick Start, which links to the mycpp README.md.

Tips for Developing

  • Make sure to build/dev.sh minimal (or all) after you git pull ! The Python extension modules (.so files) that OSH uses may be out of date, and they have to match the Python code.
  • Use build/clean.sh to start over.
    • It deletes all the temp files created by the tools, in underscore dirs like _build and _tmp.
    • See Oil Dev Cheat Sheet
  • Spec Tests are very important! Make sure you know how to run them quickly and maintain a fast, iterative development style.
    • Where Do I Put Spec Tests? If I'm going to change the behavior of the source builtin, I do grep source spec/*.test.sh and that leads to the spec test that should verify the new behavior.
  • Debugging Completion Scripts -- Use --debug-file!

Older tips:

  • Debugging OSH / OVM / CPython itself
    • make _bin/oil.ovm-dbg and misc/bin.sh which creates _bin/osh-dbg may be helpful in debugging... although it's not something I do very often!
  • make clean is meant for the end user source tarball and will just remove the binaries in _bin. make clean-repo is more aggressive and removes the "pre-build" in _devbuild.

Pull Requests

Most people use Github pull requests to send patches. I will use Github's system to make comments. After addressing them, please:

  1. push the commits so I can see them
  2. reply with a message, like "All done", or "I disagree because ..."

If you only push commits, I won't look at the PR again, because I'll assume it's in progress.

I aim for 24-hour response time

Please feel free to ping andychu on Zulip or Github if you're waiting for a pull request review! (or to ask questions)

Usually I can respond in 24 hours. I might be traveling, in which case I'll respond with something like I hope to look at this by Tuesday.

I might have also missed your Github message, so it doesn't hurt to ping me.

Thank you for the contributions!

Code Style

Python code follows Google's Python style guide, which is essentially PEP 8, but with CapWords for function names.

Most python files can be formatted with test/lint.sh run-yapf-2.

There are notable exceptions in the opy/ dir, because I didn't write that code.

  • For shell code, also follow the existing style. It has 2 space indents, funcs-like-this (unless POSIX shell is required), and vars_like_this.
  • C code generally follows Python style (except 2 space indents?)
  • C++ is formated with clang-format -- see test/lint.sh

Code Overviews

Links