Open Index Protocol - oipwg/wiki GitHub Wiki
#Introduction The Open Index Protocol (OIP) is an application protocol for decentralized, collaborative information and content archiving and distribution systems. OIP is the foundation of digital media artifact publishing and digital information record archiving in the Universal Index.
The Universal Index, or UniDex, is a collaboratively managed open, unified database of browsable, searchable Signed Messages contributed by registered Rights-Holders to a second-generation proof-of-work blockchain and made usable through the software and services provided by an eco-system of registered Service-Providers. OIP is the protocol that governs the rules for publishing-to and consuming-from the UniDex.
#An Open and Unified Index Scores of servers around the world running HTTP software by a large collective of servers results in the World Wide Web, a single thing in our minds, but in reality not actually a single thing at all, but instead, a collection of individual services running on privately hosted servers all around the world. Links connect the pieces of the web, but it has never had any sort of unified center - no index of its contents. Search engines attempt to fill this need by crawling and cataloging the various branches of the web so that it can be browsed and discovery of newly published content can happen - but this has been done as a layer on top of HTTP, usually using privately owned and controled indexes created by proprietary web-crawlers.
It is worth noting that the high cost of crawling and mapping the web has effectively worked to stiffle competition amongst search and suggestion algorithms - that is, while the real value offered by a search engine is in how effectively it's algorithm matches search terms with relevent results, the cost of simply mapping the web first is so high that many startups with interesting approaches to search and suggestion algorithms were never able to get off the ground. One of the most significant benefits of OIP, on the other hand, is that it establishes a fully decentralized - and also unified - index of published content, and anyone can develop search and suggestion algorithms for it. As a wise man once said "why reinvent the wheel, if your innovation is the rubber in the tire."
#Balanced Protocols Blockchain technology is changing the value capture relationship between protocols and applications. In his post, “Fat Prototols,” Joel Monegro discusses the shifting landscape. He describes this shift by analyzing the historically “thin” value capture protocols of the world wide web (TCP/IP, HTTP, SMTP) which enabled a “fat” value capture application layer where companies like Google and Facebook provide and extract such a large portion of the produced value. He then compares this to a “fat” value capture protocol, Bitcoin, which provides a significantly valuable service, but does so almost entirely in protocol, thus leaving very little value to be captured by applications built on top of it. The result is that Bitcoin's application layer can be described as "thin", in regards to how much value it is able to capture.
Insert illustration side by side
However, not all decentralized protocols will behave this way. Bitcoin is ingenius and incredibly useful, but its designed to facilitate one specific use-case; the sending and receiving of tokens representing monetary value. As the world begins to explore applying decentralized computing technologies to new industries, it behoves us to understand that sometimes Bitcoin's "fat protocol/thin application" value relationship will apply, and in some cases the web's "thin protocol/fat application" value relationship will apply. And in others, we propose that a third possibility for the protocol/application value capture relationship: balanced. The Open Index Protocol facilitates value capture at both the protocol level and the application level.
PROTOCOL VALUE CAPTURE
Value capture from OIP’s protocol layer aligns with Monegro’s discussion of Bitcoin/Ethereum value capture. OIP’s unified index blockchain (Florincoin) has two main components that drive value capture: 1) a shared data layer 2) “tokens” with speculative value.
- The value of the shared data layer is that it reduces the barrier of entry for new projects to create competing front end products and services. The network is open for permissionless use to both creators and consumers, increasing the potential quantity and value of the data contained in the blockchain. Each new creator that contributes to it increases its value for all previous creators and increases the visibility of this platform for potential future creators.
- Publishing into this shared data layer requires spending its tokens. This relationship leads to speculating on the value of the tokens, incentivizing both protocol development (which Albert Wenger discusses in this article) and protocol adoption (link to JM's article about this).
In the OIP ecosystem, the “rights-holder” category of users populate the index blockchain with data. “Rights-holders” are defined as:
- publishers. (independent content creators/industry majors)
- promoters. This category of user performs their contribution to the ecosystem a single time - either by announcing their identity, publishing new content, or sharing a link.
APPLICATION VALUE CAPTURE
Value capture from OIP’s application layers differs from Monegro’s analysis. There is significant value capture available from the application layers. The OIP ecosystem requires the services of security, storage, distribution and discovery. Unlike “rights-holders” above who contribute a single time, these ecosystem services must be provided on an ongoing basis. This category of OIP users are called “service-providers”
this paragraph is unfinished
Insert OIP balanced illustration
#Technical Overview
The UniDex is a decentralized application (DApp) that facilitates the permissionless publishing and consumption of digital content. Behind-the-scenes, it relies on a peer-to-peer distribution network (IPFS), a micropayments blockchain (Bitcoin), and a blockchain designed specifically to store messages (Florincoin), but the applications designed to browse it are written in familiar web-technology, so they appear to function -from the users perspective- exactly the same way as web-based applications. The first two applications built for OIP/the UniDex are Alexandria’s browser and Tokenly Music.
Unlike HTTP, OIP does not function as a request-response protocol, but rather a permissionless-publishing-and-consumption protocol. In other words, the protocol describes the set of rules one must follow in order to publish to the UniDex, and the set of rules one must follow in order to consume from it, and neither of these actions depend on responses from a remotely-hosted server. The result is a very real possibility of zero-downtime functionality and published content that won't disappear because someone stops paying a server bill somewhere.
Valuable content and information is provided by users called Rights-Holders, and various protocol-regulated services are offered by entities called Service-Providers. Value, in the form of direct cryptocurrency payments, is contributed into the eco-system by end users and divided in real time between Rights-Holders and Service-Providers according to the rules of the protocol and the preferences expressed by each.
OIP is designed to facilitate the browsing, discovery and consumption of any of the content in its index using any protocol-compliant software. OIP software can be run as a hosted service on the World Wide Web, as a secure and private local service, or by executing smart contracts in the Ethereum World Computer.
OIP is intended to balance the mutual needs for redundancy and efficiency. High-demand content benefits from significantly reduced file distribution costs, end users can earn incremental income by contributing needed permissionless services, and say more about efficiency here.
By sending a special type of blockchain transaction called a Signed Message, Rights-Holders retain full control over the distribution rights of the content they publish, but because the Universal Index is stored in a blockchain, the database itself is not centrally controlled by anyone and is completely transparent to the community at-large. The actual Signed Messages in the index describe (1)commerce & tipping preferences, (2)how media files can be found and distributed to end users, (3)cataloguing and descriptive information, (4)meta tags and more.
OIP resources like songs, videos, history archives, blog entries and many other kinds of digital media are identified and located in the UniDex by Unified Index Identifiers (UIIs), using the Uniform Resource Identifiers (URI's) schemes oips, oipw and oipe.
#History Development of OIP was initiated in February 2014 by Devon Read and Amy Oliveira, founders of Blockchain Technology Group LLC1 2. Standards development was initially managed under the initial title "Project Alexandria" and then "The Decentralized Library of Alexandria" by the parners and employees of Blocktech, with significant early contributions from Ryan Taylor, Joseph Fiscella, Ryan Jordan, Skyler Ostler and Jeremiah Buddenchen. Starting in late 2016 the title Open Index Protocol was adopted and Blocktech began collaborating on OIP with other like-minded projects. The Open Index Protocol Working Group was formed to manage standards development going forward. The formal definition of OIP/1.0 is expected to be published in early 2017.
#Rights-Holders A Rights-Holder (RH) is a user or a user agent who maintains positive-control of the private key corresponding to a particular public key and therefore assumes the rights and responsibilities of the messages in the UniDex signed by that key. All interactive users (meaning users who make changes to the UniDex itself, not including users who only consume content, even if it is payment-required content) in the UniDex must be registered Rights-Holders in some form or another. An end user who contributes original content to the UniDex is called a Publisher and they are an example of a Rights-Holder for which the rights they hold are somewhat obvious; the right to distribute the content they publish.
For other types of Rights-Holders, this isn't quite as clear. For example, an end user who periodically looks up, validates and then pays to store various kinds of historical records in the UniDex is another kind of Rights-Holder called a Historian. Since they make no claim to originality of, or the distribution rights surrounding the data they help to archive, some may argue that this is an innapropriate label for this class of user. However, another way of thinking of a right is as a responsibility, and describing them as Responsibility-Holders makes much more sense. In their case they have a responsibility to archive honest information; the Open Index Protocol has, in truth, no means of forcing any user to take seriously that responsibility, but the market functions it facilitates certainly does, by allowing a particular Historians record of accuracy to be evaluated and scored in a completely transparent manner (this is one of those areas where it is highly significant to know that we are evaluating 100% of the data that existed about a subject, and not just the data that exists at the current point in time, like web services are limited by). Therefore, it becomes possible to hold a Historian responsible for being innacurate or dishonest. This is why all rights-holders (responsibility-holders) must register the public key corresponding to the private key which they will be signing their contributions with.
To become a Rights-Holder, a user simply sends a properly formed Rights-Holder Registration Message (RHRM), a particular kind of Signed-Message to the UniDex, after which they can be identified by their public key address.
###Publishers Publishers are paid by end users when they connect with an audience that finds their content valuable, at whatever price they and their market agree upon
- How to register as a Publisher in the UniDex
- How to format a publish artifact messages properly
- Calculating the required Publish Fee
- Calculating the trade value of tokens from AutoMiner nodes
- Using TradeBot to exchange Bitcoin for Florincoin tokens to use for publishing
- Hosted API endpoints for checking tradable balance & starting a trade
#Signed Messages A Signed Message (SM) is a transaction-comment or series of multipart-linked transaction-comments in the OIP-specified blockchain, conforming to a protocol-specified message schema, and signed, cryptographically, using the private keys of a specified Rights-Holder. Some Signed Message formats facilitate the registering of various kinds of rights-holders in the UniDex, while others allow registered rights-holders make contributions or changes to the UniDex.
artifactPublish is one common example of a Signed Message. Other examples of Signed Messages include publisherRegister, artifactDeactivate, artifactEdit, artifactReassign, historianRegister, and recordArchive.
###Rights-Holder Registration Messages
- Historian
- Publisher
- Names
###Service-Provider Registration Messages
- Retailers (of artifacts published in the Unidex)
- Promoters (of artifacts published to the Unidex)
- Autominer (contributing proofs-of-work to the mutual protection of the unidex)
- AutoDistributors (provide storage and distribution services for the media files published to the unidex)
- Pool
###Historian Messages
- Blockchain Data
- Social Media
- Public API Data
###Artifact Messages
- Publish:
{
"oip-041":{
"artifact":{
"publisher":"$PublisherAddress",
"timestamp":1234567890,
"type":"$ArtifactType",
"info":{
"title":"$ArtifactTitle",
"description":"$ArtifactDescription",
"year":1234,
"extraInfo":{
...
}
},
"signature":"$IPFSAddress-$PublisherAddress-$timestamp"
}
}
- Edit:
{
"oip-041":{
"editArtifact":{
"txid":"$artifactID",
"timestamp":1234567890,
"patch":{
"add":[
{
"path":"/payment/tokens/mtcproducer",
"value":""
}
],
"replace":[
{
"path":"/storage/files/3/fname",
"value":"birthdayepFirst.jpg"
},
{
"path":"/storage/files/3/dname",
"value":"Cover Art 2"
},
{
"path":"/info/title",
"value":"Happy Birthday"
},
{
"path":"/timestamp",
"value":1481420001
}
],
"remove":[
{
"path":"/payment/tokens/mtmproducer"
},
{
"path":"/storage/files/0/sugBuy"
}
]
}
}
},
"signature":"$txid-$MD5HashOfPatch-$timestamp"
}
- Transfer:
{
"oip-041":{
"transferArtifact":{
"txid":"$artifactID",
"to":"$newPublisherAddress",
"from":"$oldPublisherAddress",
"timestamp":1234567890
},
"signature":"$artifactID-$newPublisherAddress-$oldPublisherAddress-$timestamp"
}
}
- Deactivate:
{
"oip-041":{
"deactivateArtifact":{
"txid":"$artifactID",
"timestamp":1234567890
},
"signature":"$txid-$publisher-$timestamp"
}
}
#Service Providers ##Retailers A Retailer is a web-hosted front end of some or all of the original content published to the Unidex. Retailers are paid directly by end users when they pay for a piece of media the retailer put in front of them, mostly using suggestion algorithms within a given front end/marketplace, percent of total sale they get is limited by preferences determined by publishers.
Since all Retailers have the same library of content available to them at the same market prices, it is incumbent on them to compete with each other on other factors such as interface and discovery mechanisms. The interface and how users interact with content will be up to each front end marketplace, but the back-end will be the same for all of them - they will run full nodes of all OIP services, which includes the Florincoin index blockchain and LibraryD. A server-hosted wallet only stores the blockchain, not the users keys - those are signed on the user’s side of a web session and then transmitted into the blockchain by the hosted wallet, keeping each user’s private keys in their control. Often, front ends will run a webserver with a large hard drive and IPFS running on it, as well as a script that pins artifacts in the library to the server (this is only a standalone JS application during this period of development - working title PinBotWizard - it will ultimately be merged into the functionality of LibraryD), providing one full seed on a high speed HTTP server for users browsing the library over the web.
Publishers have the choice to incentivize front end marketplaces to feature their content by allowing an optional cut of the sales facilitated by the marketplace, with the percentage amount controlled by the publisher. Due to this incentive mechanism, a very important function for front ends is to provide a discovery and suggestion mechanism. This is not required, but it will be a significant driver of user “stickiness,” as well as a source of direct revenue for the Retailer. This service provider role requires registration because of the relatively high risk of malicious front ends and their ability to steal a great deal of money if no mechanism existed to warn users of them. To register, a service provider sends a message to the Florincoin index with the URL of their service, if on the web, or App Store ID if on the Apple or Android app stores and their cryptocurrency address of choice to receive payments. This can be a Bitcoin or other cryptocurrency payment address, or a NetKi address, as they can be looked up over global DNS and allow users to dynamically change their chosen payment addresses without needing to update their Alexandria registration.
##Promoters A Promoter is paid directly by end users when they pay for a piece of media the promoter put in front of them, mostly over social media platforms, percent of total sale they get is limited by preferences determined by publishers. Promoters can be individuals who share links over various social media platforms or through other online marketing efforts, or special social media marketing agencies structured to provide the service.
For individual promoters the program works similarly to Amazon’s affiliates system. Promoters use a custom link to spread the piece of content via social media, blogs, websites, even their own custom applications, and included in the link is their payment address, so that any time a payment is made through the link, the designated percent goes directly to the promoter, and the rest goes to the publisher. For example, a filmmaker may chose to give 15% of their sales to those that help them widen their audience with their social media graphs. Because publishers can chose any percent to award, or even opt out of the program entirely, they can chose exactly how much financial reward they wish to offer that best suits their individual needs.
##Mining Pools Software for running an OIP-compliant mining pool to support the UniDex.
- Alexandria's Mining Pool: a Node.js pool portal customized to write proper Historian records to all blocks that it wins.
- To check the average pool_margin for the past 24 hours,
- Lookup the current block height using the OIP Working Group's web-hosted API for the getMiningInfo endpoint (Source Code) and store the results as block
- Send a POST to the OIP Working Group's web-hosted API for the historian summary endpoint along with the following input:
{
"max-block":*block*,
"min-block":*block*-2160
}
Example Results:
{
"averages": {
"block": 0,
"url_of_node_or_pool": "",
"mrr_last_10": 0.00005073155,
"pool_hashrate": 1006632.96,
"fbd_networkhashps": 1555391217,
"fmd_weighted": 0.00000332,
"fmd_usd": 0.00303,
"flo_spotcost_btc": 0.0000014612482832369695,
"flo_spotcost_usd": 0.0013336091259662705,
"offer_btc": 0.0000017534979398843634,
"offer_usd": 0.0016003309511595247,
"pool_margin": 20
},
"data-points": [
{
"block": 2034996,
"url_of_node_or_pool": "pool.alexandria.io",
"mrr_last_10": 0.00005073155,
"pool_hashrate": 1006632.96,
"fbd_networkhashps": 1555391217,
"fmd_weighted": 0.00000332,
"fmd_usd": 0.00303,
"flo_spotcost_btc": 0.0000014612482832369695,
"flo_spotcost_usd": 0.0013336091259662705,
"offer_btc": 0.0000017534979398843634,
"offer_usd": 0.0016003309511595247,
"pool_margin": 20
}
]
}
- The averages array will include a field labeled pool_margin. If this amount is in the ballpark of your target margin on top of mining costs, you may wish to become an autominer.
##Autominers autominers are paid by publishers for their actual costs plus their requested margin, governed by data-driven market feedback mechanisms, limited by demand for publishing
- Create an account on miningrigrentals.com
- Make a new “pool profile” using the following information:
algo: Scrypt host: api.alexandria.io port: 3032 worker: *yourflorincoinaddress* password: *anythingyouwant*
- Create a miningrigrentals API key
- Install and start the autominer_api application. Source Code
- Fund your miningrigrentals.com wallet with some bitcoin, and the autominer-api will automatically start renting rigs if market conditions allow your minimum margin to be met
##Ad-Providers Fill in some deets
#The Unidex Namespace ###Rules and Prereserved Names
- applications exist within a single global namespace
- Several generic application names will be registered by the Open Index Protocol Working Group, but most will be commercial trade names
- publishernames exist within a single global namespace
- publishernames are not required in order to publish artifacts. If a publisher wishes to not register a “human readable” name, they will be automatically given the publishername corresponding to the first 6 characters of their publisheraddress
- artifactnames exist within a unique global namespace for each publisheraddress
- As such, in all of Alexandria, there can only be one .../[application]/BobDylan/…
- and each publishername can have their own .../[application]/[publishername]/Featured/
- or even their own .../[application]/[publishername]/Titanic, but it won’t be able to be the mediatype “feature film” because only .../[application]/Paramount/Titanic (or perhaps …[application]/JamesCameron/Titanic) would resolve to the popular feature film by that name
- Likewise, there could be many different .../[application]/[publishername]/TinyHuman, but they won’t be able to be the mediatype “music” unless the publisher has used verified-publisher to connect their publisheraddress to a publishername reasonably similar to the name associated with the song in MusicBrainz and provide a social media verification of a connection between them all. As such, any musician can publish a song called “Tiny Human” but it will either already have to be in MusicBrainz or they’ll have to add it.
- Before a protocol compliant application will send a publisher name registration message, it confirms that the name is available according to a series of checks:
- Is the name currently unregistered or has its most recent registration expired?
- Is the name associated with a currently existing rights holder found in IMDB or MusicBrainz?
- If the requested publishername is unregistered and is not associated with any currently existing rights holder of feature films, tv shows or music, the application allows the name registration message to be sent to the blockchain
- The cost to register a name will be $8 per year, and all registration periods will be for 2 years. Applications can be designed with features that allow automatic re-registration when they expire, but a name cannot be registered for longer than 2 years at one time.
- Until 1.0, this price will be reduced by 90% to $0.80 per year, or $1.60 per 2-year registration period.
- If a publishername owner wishes to sell to another party, they can do so by sending a message that signs the rights for the name over to a different publisheraddress. The transfer cost will be $8 per year, or $16 per 2-year registration period, but if the two parties involved agree on a purchase price to transfer the name, that is entirely between them.
- Registration costs are paid directly to blockchain miners by being attached to the registration message itself as its transaction fee (TX-fee)
###Currently Existing Rights Holders
- If a name is found to be a rights holder in IMDB or MusicBrainz, the name is considered an existing rights holder
- In IMDB, existing rights holder are limited to the roles of:
- Production Companies, Distributors, Director
- In MusicBrainz, existing rights holder are limited to the roles of:
- Producer, Labels, Artists
- In this case, the user must use verified-publisher, which will have them go to social media to connect themselves to the publisheraddress before a publishername registration message for this name would be valid - you can read more about this in the section “Social DRP” (Digital Rights Protection)
- Once they have shared an authorization message on a publicly accessible social media account which is reasonably similar to the name being requested, they will be allowed to register the publishername
###SQUATTING!
- publishername “squatting” is defined as registering a publishername and not actually “using” it for a period of at least one year.
- “Using” a publishername involves getting users to resolve the name or contribute comments to artifacts resolved by the publishername
- So, if a publishername has gotten statistically zero name resolutions in at least a year…
- And the artifacts it resolves to have received statistically zero comments in at least a year…
- Then someone could claim that the name is being squatted upon and attempt to take it over from them by sending a publishername registration message, with the proper registration fee, and including evidence that the name was being squatted in the message itself
- Most often, this evidence would be links to quarterly public use reports published to the UniDex by the Open Index Protocol Working Group
- Within these quarterly public use reports are summaries of end user activities provided by OIP/UniDex front ends, including the following data:
- Total name resolution requests for each publishername
- Number of plays and purchases for each artifact
- Revenue for each artifact
- Revenue for each promoter
- Total number of comments for each artifact (note, comments include actual words, as well as ratings and simple up/down votes)
- If a user is able to link to at least 4 consecutive quarterly public use reports which can verify that there has been statistically zero resolutions of a name and comments on artifacts linked by a name, the publishername will be found to be squatted, and the new publishername registration message will be automatically considered (by protocol-compliant libraryd nodes) to be tentatively valid.
- This will begin a period of 60 days to contest the squatter claim. If the original publishername owner does not challenge the claim for this 60 day period, the new publishername registration message will be considered valid - ie, the name will be transferred to the new publisher.
- If the users evidence is more subjective (ie, the claimed squatter seems to have written a script to get some number of name resolutions to be reported per quarter, but there are very few if any new artifacts published and they have statistically zero comment activity), they can submit their argument as a document.
- In this case, Libraryd will not be able to automatically evaluate the evidence, and it will ignore the new publishername registration message, considering the earlier message the only valid one (given that its registration fees are up to date).
- The Open Index Protocol Working Group will commit some time to reviewing these subjective claims with a public discussion and review of evidence
- The Open Index Protocol Working Group will be responsible for reviewing these claims by a governance structure to be determined, and if they find the publishername to be squatting, it can add these to a list called “namesquatters”.
- All protocol-compliant applications will be subscribed to this list, so when a name has been challenged successfully, the earlier registration message will be considered void
- The recently sent publishername registration message (which opened the case) will be considered valid, as Libraryd will see no earlier claims to the name.
#Fees ##Calculating the cost of a token
##Name registration fee calculation
##Artifact Publishing Fee calculation
Artifact Publish
The OIP formula for calculating the publish fee is closed-loop and based on inputs found either in the Florincoin blockchain or in the publish message. It is calculated as a function of either (1) commercial value to the Publisher or (2) if it is being offered for free, as a function of its size in the blockchain.
Formula for calculating the fee to publish a free artifact:
[V * s / blockchainsize](https://github.com/oipwg/oip-sdk/blob/master/formulae.md#pf)
Formula for calculating the fee to publish a commercial artifact:
[if ( C < Cv, C, (( log (C) - log (Cv) ) x (Cv / C ) x D ) + Cv )](https://github.com/oipwg/oip-sdk/blob/master/formulae.md#pc)
#Subscriptions Publisher Subscriptions
Taste-Makers
##Blacklists 1. DMCA takedown notices & other pirated content 2. child pornography & other widely unlawful content
#OIP Software ##Daemons
- OIP Daemon
- API endpoints & documentation
- Florincoind
- Why UniDex is on Florincoin blockchain
- Florincoin network for micropayments
- Bitcoind
- Bitcoin network for payments
- Counterparty assets on Bitcoin blockchain
- IPFS Daemon
##Web-Applications and User Interfaces
###Media Players
####Tokenly Music
####ΛLΞXΛNDRIΛ (reference client)
-
Browser: Interface Source Code Includes Paywall-Web
-
Publisher: Includes TradeBot & Coinbase Buy Widget Interface Source Code
-
OIP-NPM: a node.js module built to facilitate publishing and submitting changes to the unidex. Source code
###Media Search Engines ####WeGetMusic
- WeGetMusic: from the founder of musicsupervisor.com, Barry Coffing, a meta-data driven search engine based upon the lessons learned from years of running one of the most popular music supervisor services in Hollywood.
###Web Browser Extensions ####Tokenly's Pocket
####ProTip
#Full Sequence of Operations 1. Mine tokens 2. Trade tokens for Bitcoin (Reserve Cryptocurrency) 3. Use tokens to Publish 4. Re-mine tokens