Git Guild Technical Overview - GitGuild/gitguild_website GitHub Wiki
Git Guild Technical Overview
The Git Guild is a team of engineers and designers working on open source projects. So far, we have built an authentication standard, a trustless application framework, and a number of example applications focused on digital currency market access.
By starting with the base of the application, our framework consistently chooses a modular and peer to peer (p2p) structure. We believe that smaller, single purpose software tools have the most re-usability value. Furthermore, we believe that the user experience should be paramount in any project. Users should be given control, and none of the user's information or valuables can be lost under any circumstances. This means p2p, and client side management of information and valuables, with the user ultimately in control.
Authentication needs to be done with asymetric cryptography, so no secrets are shared, and messages are provably authenticated by the user. Strong data typing with an ORM will facilitate any popular SQL database, as well as Swagger for API documentation and client development. Use Redis to send messages and cache data for processing or retransmission via http or websocket. Boom! You've got yourself a framework.
Not so fast. There are many small choices and details to work out for the above to work. Most of them are described in this document, but some are still open issues. If you find any problems with this document, please contact us, open a Pull Request, or an Issue.
Project Organization
The Git Guild organizes itself and its projects on github. For those non-techy readers, git is a "version control" tool, and github is a popular git server host. Version control means that it tracks changes to documents over time. Git and github answer questions like:
- Who made the changes?
- What exactly was changed?
- Which version of the document was changed?
Additionally, github provides related project management tools like Issues and Pull Requests. Issues are just what they sound like: a way to document some task or bug that needs to be worked on. Pull Requests are a bit more complicated. You can think of it as a request to make changes. You first specify exactly which changes you want to make to the documents, then you request that those changes be integrated.
In the Git Guild, sponsors pay our members to document and resolve Issues, make Pull Requests, and otherwise participate in the development process. This honor is only possible by maintaining the highest standards. In this section, we explore the standard Git Guild project repository.
README
This document should give an overview of the project, including purpose, installation, usage, and organization. The main audience for this document is usually contributors to the project, so the contents are an introduction to would-be collaborators.
The README document and many others use the markdown format. This human writeable and readable format is used extensively by the Git Guild, and we recommend practising it hands on with perhaps a 15 minute introduction on their site. This document is also written in markdown.
Wikis
Each github project can have its own wiki, and we make heavy use of this. Though wikis can have many purposes, Git Guild software projects use wikis for storing roadmaps, and specifications (aka specs).
Home w/ Roadmap
The standard Git Guild software project should have a wiki Home which includes a project description and roadmap. After the README, this is the next place a new contributor should look for information about a project.
Requirements
A requirements document describes what the program is supposed to achieve. What can the user do with the program, from a high level? Don't worry about the how.
Functional Spec
A functional specification defines the behavior of the program, as seen from the outside. It should be exhaustive, testable, and eventually tested.
Design Spec
A design specification describes the internal operations of the program. How will the program achieve the functionality described in the functional spec? What sub-components and data models are used?
Git Governance
Git Guild repositories are governed by voluntary charters and contracts. Contributors, maintainers, testers, reporters, and others who wish to collaborate on the project must agree to general working terms, including pay, roles, and responsibilities. To facilitate this, the Git Guild is developing a toolset that will make it easier for Guild members to manage their contracts, pay and other responsibilities. If a repository is under the management of a Guild, it will have a .gg directory with charter, contracts, and other governance documents inside.
Trustless Application (TAPP) Framework
Running Services
TAPPs that are runnable should include a Command Line Interface (CLI), and/or a daemonized process.
When a CLI is called for, it is strongly recommended to use Python's Argparse.
When daemonizing a process, a number of different choices are available. Recommended ones are: supervisord, gunicorn, and python-daemon. Whatever is chosen, well behaved start and stop commands must be available and documented in the README.
Configuration
Currently, most TAPPs are configured via an ini file, usually specified in an environment variable. The contents of this file and the name of the configuration variable should be specified in the project README.
Some TAPPs, like the soon-to-be-published Trade Manager, have been moved to a database configuration system. This allows the app to be re-configured without restart, but the method is probably not safe for storage of sensitive information like private keys.
Authentication
After months worth of research and experimentation, we decided to create a new authentication standard: bitjws. Bitjws is really a combination of two existing standards. It is JWS (JSON Web Signature) using Bitcoin message signing as the algorithm.
As we're initially designing services for a digital currency environment, Bitcoin message signing carries a number of advantages. Users can store (or generate using PBKDF2) an extended private key, which allows them to generate a number of keys for both Bitcoin and/or bitjws use. This unified keyring method will let TAPPs into many existing bitcoin wallets and services with minimal cryptographic change.
JSON Web Signatures is actually more obscure than Bitcoin. Still only a draft standard, we feel that it has the functionality we need to format and sign messages in a safe and secure manner. Our use of JWS is not typical, and their documentation refers to OpenID tokens, not our authentication standard.
A bitjws-compatible JWT message is not really a token, but a timestamped message with a specific audience.
{
"aud": "/api",
"exp": 1465950625.869843,
"echo": "hello"
}
A bitjws-compatible JWS header will have typ JWT, alg CUSTOM-BITCOIN-SIGN, and a bitcoin address for kid.
{
"typ": "JWT",
"alg": "CUSTOM-BITCOIN-SIGN",
"kid": <bitcoin_address>
}
An example of a signed and serialized (base64 encoded) bitjws message.
eyJhbGciOiAiQ1VTVE9NLUJJVENPSU4tU0lHTiIsICJraWQiOiAiMUc5c0NoYnlEU211QVhOVnBqdVJ3YWtUbWNIZHhLR3FwcCIsICJ0eXAiOiAiSldUIn0.eyJhdWQiOiBudWxsLCAiZWNobyI6ICJoZWxsbyIsICJleHAiOiAxNDY1OTUwNjI1Ljg2OTg0M30.SDRqV3NhdUE2TXlvczd6ck9EMmh2NC92amhkMGh6V0swNHMxNjNuSFRTNUpaeG90dk8zakNjSUROWjA2bnNlTlE3YU0rcGFrR2MzSEIydTFOZXlrNzM4PQ
Bitjws is in live production use by licensed financial services companies. It is a tried and tested method of authenticating API messages. As such, it has a number of libraries and packages available.
Core bitjws Libraries
Swagger
Swagger is an API documentation standard, which was recently chosen as the basis for the Open API Initiative. Swagger specs are a way to document your API in so much detail that public-facing, interactive documentation explorers and self-programming clients are feasible. By requiring all TAPPs to public a Swagger spec, we ensure a smoothly interoperable, and well-documented app ecosystem.
The TAPP toolbox includes a number of helpers for generating and interpreting Swagger specs.
-
alchemyjsonschema This helpful tool creates json schemas out of your SQLAlchemy models. This allows automatic synchronization between your data tables, ORM models, and json schemas. The json schemas are then used to generate Swagger specs, which are used by clients to construct requests and interpret responses.
-
flask-swagger By putting YAML-formatted Swagger spec fragments in request handler docstrings, the bulk of a Swagger spec can be automatically generated using flask-swagger.
-
bravado-bitjws A Python client which programs itself based on Swagger specs. Adapted for bitjws authentication.
-
bitjws-js-client A (beta) Javascript client which programs itself based on Swagger specs. Adapted for bitjws authentication.
TAPP clients like bravado-bitjws and bitjws-js-client are self-programming and can work with multiple TAPPs. Just grab the spec and public key for the TAPP you wish to use, and you're ready to start making requests.
Database (via SQLAlchemy)
stable
SQLAlchemy is a stable and well known Database manager and Object Relational Map (ORM). TAPPs use SQLAlchemy to give operators flexibility in database choices, as well as to integrate into other layers of the stack.
Alchemyjsonschema is a helpful tool that creates json schemas out of your SQLAlchemy models. This allows automatic synchronization between your data tables, ORM models, and json schemas. The json schemas are then used to generate Swagger specs, which are used by clients to construct requests and interpret responses.
Sqlalchemy-login-models provides generic user and login models for SQLAlchemy and json schema. It also has a handle signature table generator in generate_signature_class. It is planned but unimplemented to store all important bitjws signed messages in such signature tables for all TAPPs. The data models in sqlalchemy-login-models need review, and comparison with the needs of sponsors and partners.
Redis
stable
TAPPs use Redis for a variety of purposes, including message queue, orderbook storage and sorting, and data cache. Partially, this is because we think Redis is awesome, but also because using it in multiple capacities reduces complexity and TAPP pre-requisites. We won't duplicate information about the well known redis service here, but instead list some resources for further reading regarding TAPP use of it.
-
de-exchange-node Uses Redis to store and sort orderbooks.
-
bitjws-sockjs-server Will use Redis as a message queue.
Websockets
beta
The bitjws-sockjs-server will publish websocket notifications to subscribed users. Users will be required to authenticate using bitjws when subscribing to a channel.
Notifications will be read from a message queue, and should be able to originate from a number of publishers. At first the message queue will be AMQP and after Redis. The publishers are likely but not always going to be instances of flask-bitjws servers.
Bitcore + Insight
stable
The Git Guild uses and recommends Bitpay's bitcore stack for multi-signature key management. This rich ecosystem of open source libraries and services has become an industry standard. Colu has implemented colored coins for Copay (bitcore UI). Dash has also implemented the required bitcore libs, though it is not known if a public Dash Copay wallet is available.
TAPPs
The first batch of TAPPs are all digital currency microservices. Together, these provide a complete set of market access ramps with banking, cash, digital currency on and offramps using various exchange pricing choices. Though not all of the first batch are published, the full MVP set shown below is coming very soon.
DeSW (De Shared Wallet)
beta
De Shared Wallet aka desw is the simplest, most trusting implementation of a cryptocurrency wallet (a hot wallet). The MVP will be a single key signer setup, but even after cosigning is introduced, the functional requirement here is to allow fully automated transfers. Properly authenticated transfer requests will be processed automatically.
The basic functionality of a wallet will be implemented: storing, sending, and receiving assets. Additionally, transaction networks will be integrated to transfer assets outside of the desw wallet. Users should be able to control their network fees, which they are required to pay.
Since DeSW is not a "trustless" service, it is ironic that it is one of the first TAPPs to be completed. Still, machines, such as Lamassu ATMs, have need of such fully automated wallet services. Whenever possible, multiple signers should be facilitated using Bitcore.
De Exchange Node
beta
Each de-exchange-node instance is a service that provides bid/ask exchange matching for a single market pair. It maintains the orderbook and matches orders into trades. This service has a bitjws API, and does not provide either wallet or user interface. For a wallet, each exchange node instance expects to have its own account with a DeSW server.
Also a heavy and clever user of redis, de-exchange-node instances should be ready to plug into cached and/or streaming data feeds.
De Broker
alpha
The de-broker project will let API users convert one asset to another, using approved transaction networks. The general concept is an API-only service with the conversion functionality of Coinapult, Shapeshift or Coinbase. Similar to Shapeshift, guests will be able to complete transactions without registration, using public blockchain transactions.
Fixed price quotes should be provided upon user request, which are valid for a limited period of time. If the user initiates a payment within the time window, the quote is considered valid, even if the transaction takes longer to confirm. No funds will be sent out until confirmations are received, however.
The de-broker is the heaviest user of other TAPPs, potentially using DeSW, De Exchange Node, and Trade Manager all at once.
Transaction Networks
Payments should be accepted and made using desw. This means that de-broker only needs to be aware of Transaction networks enough to let desw know what to do.
Trade Clearing
To clear trades and avoid running out of one asset or another, de-broker expects to be integrated with an instance of de-exchange-node that it can place limit orders with. Since Trade Manager is coming soon, here is a de-broker architecture showing de-exchange-node instead.
Trade Manager
Coming soon
The trade-manager will communicate with cryptocurrency exchange APIs, and provide a stable and uniform interface for exchange users to manage their trading activities. The MVP release will include a TAPP API and a CLI for managing trades and configuration. This project makes extensive use of the older, stable bitcoin_exchanges project.
When completed, the trade-manager will be substitutable in any place communication with exchanges is needed. In particular, this means de-broker will have new and exciting settlement possibilities.
De Cosigner
Coming soon
An automated cosigning service to compliment bitcore or other multisignature wallet clients. Will look for 2fa (TOTP) code and other factors before deciding whether or not to cosign a transaction or bitjws request.
Other Projects
DashLamassu Upgradable
The Git Guild has committed in the DashLamassu project to maintain an easily upgradable toolset to add Dash to a standard Lamassu machine. This means that the code must comply with lamassu's latest and greatest, regardless of what other Git Guild teams, such as Tigo CTM, are working on. Though MVP software has already been developed using dashd and Poloniex, this is significantly more complicated than the standard Lamassu operation.
MVP (current)
The current status of the Dash Lamassu integration can be considered a Minimum Viable Product or MVP. All of the functionality is there, and it is in use on at least one production machine. There are a few relatively minor bugs, and install scripts/instructions to write, but otherwise, it is ready for more deployments.
There are some known issues with this architecture. These are not easy to solve, as they go outside of the code and scope of Lamassu.
- As there are no known Dash wallet services available via API (ala Coinapult, bitgo), a Dash node must currently be run for each machine.
- As there are no good USD/DASH markets available (at least not on Poloniex, our first exchange plugin), operators are expected to have a supply of USDT (Tether).
While each of these can be overcome by a technically and financially capable machine operator, it is significantly more difficult than running a Bitcoin-only machine, which does not involve node operation, or finding a supply of a third party token.
Dedicated Dash Services
To address the issues indicated with the MVP architecture, a next generation of Dash services and components are being built. After the services are complete, and the js client is ready, Lamassu plugins will need to be developed for each.
The first to be completed is De Shared Wallet aka desw. Along with the desw-dash plugin, this service will make hot wallet, send and receive functionality available via API.
The second component, still under construction, is De Broker. The MVP version of this service will provide guaranteed price conversions between BTC, DASH, and USD. The USD will be accessed via the Crypto Capital payment network.
Tigo CTM stack (Lamassu-based)
Tigo CTM is a Cryptocurrency Teller Machine vendor that sponsors and distributes Git Guild software.
Git Guild is planning to hard fork from the mainstream lamassu-machine code, and build a much more versatile interface. Eventually, this interface will be host to more advanced functionality like games and shopping. At first, the main difference will be cutting out the lamassu-server component entirely, and connecting directly to the dedicated services developed for the prior iteration.
It is important to note that backwards compatibility will be broken in this fork, but Git Guild will still maintain the mainstream plugins and components for the Dash community.
Coin Launches
The Git Guild is helping at least one project launch a digital currency aka a coin. Though no special software has been commissioned yet, it is likely that an Initial Coin Offering (ICO) platform will need to be developed. This will serve as a dashboard both for the issuer and the buyers during the ICO period.
Following conversations with Rootstock, the short term plan for coins requiring smart contract functionality is to use Ethereum. When Rootstock releases their production network, it will be possible to switch any coins that prefer the Bitcoin blockchain.
Appendices
Repositories
Trustless App Framework
-
bitjws - Bitcoin message signing and JWS combined to an authentication standard similar to bitid.
-
flask-bitjws - A flask plugin for creating bitjws and swagger powered API servers. The standard Git Guild http server.
-
bravado-bitjws - A universal Python client for flask-bitjws servers.
-
bitjws-js - (beta) A Javascript library for bitjws authentication, signing and verification.
-
bitjws-js-client - (beta) A universal Javascript client for flask-bitjws servers.
-
bitjws-sockjs-server - (beta) A SockJS server that authenticates using bitjws.
-
sqlalchemy-login-models - (beta) Generic user and login models for SQLAlchemy and json schema.
TAPPs
-
swagxample - An example TAPP that demos the framework.
-
desw - (beta) A multi-currency hot wallet service available via API.
-
desw-dashd - A dashd plugin for the desw service.
-
desw-bitcoind - A bitcoind plugin for the desw service.
-
desw-cryptocap - (alpha) A Crypto Capital plugin for the desw service.
-
de-broker - (alpha) a fixed-price cryptocurrency brokerage ala Shapeshift
-
exchange-node - (beta) a bid/ask exchange engine and orderbook.
-
trade-manager - (coming soon) An exchange client manager for manual or automated trading on popular cryptocurrency exchanges.
Third Party TAPPs
- Crypto Capital v3 API - Crypto Capital's latest API (private source code) uses the Trustless Application framework.
Git Guild Meta Projects
-
gitguild command software (pre-alpha)
-
gitguild.com website (not live)
Cryptocurrency Teller Machine (CTM)
-
lamassu-admin w/ Dash (beta)
-
lamassu-server w/Dash (beta)
-
lamassu-machine w/Dash (beta)
-
lamassu-poloniex (beta)
-
lamassu-dashd (beta)
Misc
- bitcoin_exchanges - A collection of cryptocurrency exchange clients
Glossary
trustless application - An application that does not require the user to trust the server. A polite application.
TAPP - An application using the Git Guild application framework, which is not bad at being "trustless."
digital currency - A digital token that has value and is spendable on a public blockchain.
p2p - Peer to peer is the term used for services that allow users to interact directly with each other.
ORM - Object Relation Managers are a method for standardizing and converting data between different types.
SQL - Structured Query Language is the most popular database language in the world.
API - An Application Program Interface is a set of routine definitions, protocols, and tools for building software and applications. (Wikipedia)
Redis - A memory datastore and message queue.
SQLAlchemy - A database client and ORM for Python.
Flask - A popular web-framework written on Python.
Swagger - An API description specification.
websocket - A streaming API for constantly delivery of data.
Issue - A way to document some task or bug that needs to be worked on
Pull Request - A request to change some document(s).
git - A version control system. Tracks changes to documents over time.
github - A popular host for git servers. See Issues and Pull Requests
CLI - Command Line Interface, a way for technical users to interact with a program
JWS - JSON Web Signatures, a draft web standard
broker - a digital currency asset exchange that guarantees fixed prices
exchange - a digital currency asset exchange that offers limit orders