Open Index Protocol
|An 'open jungle' blockchain specification for a worldwide database|
|Initially Developed By||Blockchain Technology Group LLC|
The Open Index Protocol is a specification for decentralized publishing, distribution and payments of any digital media. The specification includes:
- How a proof-of-work blockchain and interoperable transport protocols create a shared data layer for digital content
- How to search and browse the shared data layer
- How the blockchain security incentive is connected to the value of the content published to the shared data layer
- How file persistence & legal compliance are ensured
Open Index Protocol uses the Flo blockchain for the meta-data index and interoperable transport protocols for file storage/distribution and payments. Current implementation supports IPFS for file storage/distribution and Bitcoin, Litecoin and Florincoin tokens for payments; support for additional file and value transport protocols is on the roadmap.
In February 2014 Devon & Amy James proposed decentralized applications ArchiveChain and MovieCoin on the Ethereum community forum. Early development proved a shared data layer for any digital content is more efficient than separate appcoins for each, so these initial ideas were combined to become Open Index Protocol. Significant contributions to the specification have been made by Ryan Jordan, Ryan Taylor, Skylar Young, Jeremiah Buddenhagen and lead Florincoin developer Joseph Fiscella.
The initial proof of concept archived social media data into a blockchain and visualized the data as an interactive word-cloud. It was demonstrated in October 2014 at Inside Bitcoins, Las Vegas. Next, streaming audio and video were added using the BitTorrent network. On Feb 19, 2015 the first piece of media was published and retrieved using a 100% decentralized system. Bitcoin has been used for paid content since the initial public release in April 2015 . In May 2015, the primary file storage/distribution network was changed to IPFS and support for BitTorrent was deprecated. At each stage, a user-facing browser or visualizer was built alongside specification development . Initially the browser was known as ‘The Decentralized Library of Alexandria’ and the specification as the ‘Alexandria protocol.’ In a demonstration at the Decentralized Web Summit hosted by the Internet Archive in June 2016, Sir Tim Berners-Lee said the names were confusing and suggested they be changed . The browser name was changed to ‘Alexandria’ and the specification name changed to ‘Open Index Protocol.’
- 1 History
- 2 Features
- 3 Design Philosophy
- 4 Economic Principals
- 5 Comparison of digital distribution services
- 6 See also
- 7 References
ProblemCentral points of failure plague current digital distribution architecture (Figure 1). The system is vulnerable to attack demonstrated by problems like hacked personal information, spying, deplatforming, demonetization and censorship, as well as direct technical attacks on network infrastructure itself. Current architecture also suffers from wasteful redundancy which increases operational costs. Network speed is fragile because speed and popularity are negatively correlated (Figure 2).
Zero central points of failure. Every component of Open Index Protocol is distributed. The specification is a permissionless system that creates an ‘open jungle’ where anyone can publish, distribute and sell. The Open Index Protocol changes the fundamental economics of content distribution by transforming individually-negotiated, contract-based services into digital commodities that can be exchanged with fungible tokens. The system expands the sharing economy to new services including front end retailers, social media promoters, proof-of-work miners and file storage/distribution providers.
Twenty years ago, print media changed forever when the ‘open jungle’ of the internet democratized information exchange. Walled garden services like America Online, CompuServe & Prodigy did not initially use HTTP, but eventually adopted it because the open jungle of the World Wide Web was growing in popularity. Today, walled garden content distribution services like YouTube, Spotify & Netflix are ripe for similar disruption by the open jungle of Open Index Protocol.
The current digital content distribution industry is in crisis. Artists blame confusing contract terms. Audiences are so frustrated by the difficulty of accessing content they resort to piracy. The industry as a whole is struggling.
The root of these problems is twofold: 1) current decentralized hub and spoke distribution architecture cannot support market demand as efficiently as a distributed network, and 2) current solutions were constructed when the technology required for a decentralized and open system did not yet exist.
Since 1999, p2p file sharing, blockchain and other decentralized technologies have been invented to solve these problems. Open Index Protocol defines a specification for how to use these technologies to create a permissionless system for decentralized publishing, distribution, access and payments of any digital media.
- Open - Permissionless system for publishing, distribution & payments of any digital media. The shared data layer functions as a public open index; blockchain protects metadata, peer-to-peer file storage/distribution ensures access, and digital currency streamlines payments. Transport layers for file storage/distribution and payments are fully interoperable.
- Efficient - Distributed networking reduces system overhead, cryptographic tokens decrease administrative management of content & payments, p2p file storage/distribution increases playback performance and user payment process is simplified.
- Anticensorship - Open Index Protocol is fully transparent. Full global state replication of index data, transparency of all publish attempts including fails, and human readable index data ensure content data is transparent and auditable. Prevents deplatforming, demonetization and censorship.
- Secure - Index is protected by proven proof-of-work blockchain and contributing to security is incentivized. The commercial value of content is connected to and drives the security incentive. System enables collective defense against attack, ensures liquidity of tokens for publishers, and automates reliable profit margins for miners and traders.
- Antifragile - System has positive sensitivity to increases in volatility. Free market economics strengthened by interdependent incentive structure; system fees drive network security. Chaotic market inputs cause convex response, improving network incentive for all participants (publishers, retailers, index mining and storage/distribution of files). Value capture is split between subjective/objective services which benefits overall network growth. Interoperable transport layers can adapt to changes in market preferences.
The system design objective is to serve the digital media distribution market with an open specification using decentralized technology. To meet the needs of this large and diverse industry, the specification is:
- An "open jungle"
- As interoperable as possible with as few rules as possible
- Capable of serving all distribution and monetization models: free, ad based, pay per view, subscription, metered, non-financial exchange (seeding), rental, lifetime access, stream, download.
Example uses: backend shared data layer for blog entries, Wikipedia articles, Netflix shows, YouTube videos, Spotify songs, SoundCloud remixes.
- A sufficiently diverse marketplace will include rational actors acting in their own best interest who understand it will benefit them to use standards & obey the law
- A proof-of-work blockchain-based shared data layer is the most efficient technology available to protect information freedom and resist entropy of information access
- Incentive structures influence outcomes. A trust-minimized, permissionless system with a sustainable market-based incentive structure fosters cooperation, competition, and iteration; continuously improving product-market-fit
- At present, some human governance is necessary. Open Index Protocol Working Group is a potential central point of failure, vulnerable to social attack. This risk is minimized with a transparent governance structure, weighing feedback from all system participants: publishers, retailers, promoters and miners
To ensure system integrity, the specification was built with two inviolable principles: permissionless system design and limited standardization
Permissionless system design
|“||The number of participants on the network is unlimited, and no one needs to get permission from another user in order to take part. It's a free and open system, like the Internet.||”|
|— Peter Van Valkenburgh, What does “permissionless” mean?|
All networks used by Open Index Protocol are permissionless, anyone can join the networks without requiring authorization from another party. Examples of other permissionless networks include BitTorrent, Bitcoin, IPFS and Ethereum. Permissioned layers can be built on top of a permissionless system for use-cases where identity and permissioning are appropriate, however the opposite is not possible.
Open Index Protocol is a permissionless system because freedom of information requires a free and open shared data layer for digital media to ensure media persistence, censorship resistance, security, and sustainability.
Media persistence and censorship resistance are the foundational values behind the permissionless system design. Index and file persistence are secured via: decentralized networks, incentive alignment with positive sensitivity to market chaos, and transparent governance. Censorship resistance is protected via: full global state replication of index, human readable data, auditable records (all publish attempts including fails), and transparent immutable ledger.
Security vs convenience friction is reduced through permissionless system design. API endpoints are universal; they can be looked up through a localhost or through any protocol compliant host. When security is prioritized over convenience, the end user or service provider runs a local protocol daemon and full node. When convenience is prioritized over security, the end user or service provider accesses the system through hosted full nodes. Both local and hosted addresses have the same url format, making host addresses interchangeable.
Sustainability of the system is balanced and maintained with interdependent incentive alignment; the commercial value of the media is connected to and drives the system security incentives.
Open Index Protocol specification limits standardization such that the incentive structure benefits assimilation, and does not threaten negative restriction. A salutary approach is used; the specification defines standard use of objective data (index, file addressing, payments), it does not standardize subjective, service-based uses of the data (proprietary discovery algorithms, user experience, user interface). The co-opetitive model benefits from cooperation on objective data, competition of subjective services, and collaboration for iteration; system incentive design improves potential for product-market-fit.
Standardization is powerful; Open Index Protocol benefits from all three fundamental characteristics of standardization:
|“||Standardization can help to (1) maximize compatibility, interoperability, safety, repeatability, or quality. It can also (2) facilitate commoditization of formerly custom processes. In social sciences, including economics, the idea of standardization is close to the solution for a coordination problem, a situation in which (3) all parties can realize mutual gains, but only by making mutually consistent decisions.||”|
|— "Wikipedia.org:Standardization, emphasis added|
An example of the power of standards is the rate of google searches compared to emails sent. Google is the most popular individual website in the world, but it’s index and search algorithm are proprietary, whereas email is a standard; for each google search performed, about 40 emails are sent.
Thin and Fat Protocols
Websites compared to Blockchains: How value is distributed across systems and corresponding value capture opportunity, as coined by Joel Monegro.
|“||The previous generation of shared protocols (TCP/IP, HTTP, SMTP, etc.) produced immeasurable amounts of value, but most of it got captured and re-aggregated on top at the applications layer, largely in the form of data (think Google, Facebook and so on). The Internet stack, in terms of how value is distributed, is composed of "thin" protocols and "fat" applications. As the market developed, we learned that investing in applications produced high returns whereas investing directly in protocol technologies generally produced low returns. This relationship between protocols and applications is reversed in the blockchain application stack. Value concentrates at the shared protocol layer and only a fraction of that value is distributed along at the applications layer. It's a stack with "fat" protocols and "thin" applications.||”|
|— Joel Monegro, Fat Protocols|
- Thin Protocol: HTTP/World Wide Web is a thin protocol because all of the value (usually in the form of data - ie Google, Facebook) is captured at the application layer and none is captured at the protocol layer.
- Fat Protocol: Bitcoin is a fat protocol because the majority of value capture happens in the protocol layer (by miners), and the application layer can only capture value by offering additional services beyond the protocol function.
The Web's thin protocol has caused the walled-garden content distribution marketplace, and the recent introduction of fat protocols threatens to perpetuate the problem. As explained by Vitalik Buterin, fat protocol systems are
|“||ultimately quite brittle. Protocol tokens using this model may well be sustained for some time due to irrationality and temporary equilibria where the implicit cost of holding the token is zero, but it is a kind of model which always has an unavoidable risk of collapsing at any time.||”|
|— Vitalik Buterin, On Medium-of-Exchange Token Valuations|
Fat protocols offer no direct incentive for the developer community to create application layer services, thus network effect is limited by lack of competitive services to draw users at the application layer.
The pioneers of the internet understood that open source standardization is key to information freedom. Standardized specifications open new markets for competitive services.
|“||The layered design of the Internet is not accidental. It is modular, with an open lower layer, in order to enable flexibility. One can always build identified and permissioned layers on top of a permissionless system—as TLS/SSL (a closed, identified layer) is built on top of TCP/IP (an open, pseudonymous layer). The reverse is not possible, however. Had the Internet originally been architected to be permissioned and identified, it would have imposed costs and limitations on open public participation, and it would have ossified the possible range and diversity of future higher level protocols for identity and permission. When lower layers are permissionless and pseudonymous, on the other hand, the costs of participating are low (merely the cost of hardware and free Internet-protocol-ready software), and such an open platform enables a variety of closed or identified higher level layers to emerge and compete for particular use cases where identity and permissioning are essential. For example, PGP and the Web of Trust compete with TLS/SSL as methods for enabling secure and identified communications built on top of TCP/IP.||”|
|— Coincenter.org, What does permissionless mean|
- Salutary protocol: Open Index Protocol is a salutary protocol because value capture is divided between the protocol and application layers.
Objective services like index security and file storage/distribution capture value at the protocol layer; subjective services like user interface and content discovery capture value at the application layer. Value at the protocol layer comes from blockchain tokens awarded for fungible kinds of work; value at the application layer comes from salutary requirement that a percent of value transmitted is assigned to the application layer for subjective kinds of work.
Both thin and fat protocols govern a single class of asset exchange (ie; hypertext transport or value transport). Salutary protocols employ these single purpose protocols and standardize their use for dual or multi purpose asset exchange (ie; file transmitted in exchange for value transmitted). All three protocol types have the components associated with positive network effect: limited standardization and interoperability. Salutary protocols have the unique characteristics of metered system fees, user-based governance and an eco system in which a unified open set of data (back end) is financially incentivized to be used/distributed by more than one competing interface (front end).
Salutary protocols are more resilient.
- Free market-determined components:
- Market determined selection of single purpose protocols used within salutary system. Increases stability and flexibility within the multi-equilibrium token market & eliminates de facto fee associated with “implicit cost” of holding tokens in fat protocol systems. Example: Current version of OIP uses Bitcoin & IPFS because they are the most popular, but it can support payment & storage/distribution in any single asset class protocol, and can adapt to changes in market preferences.
- System fees:
- Open market with closed loop system fees incentivizes a sustainable ecosystem. OIP stabilizes the relationship between 1) the value of media in the system with 2) the security of the system by using metered fees. Salutary protocols have an interdependent fee structure with market pressures from all stakeholders; the effect is similar to the “sink” (fee) as defined by Vitalik Buterin because salutary protocol system fees are “transparent and explicit.” Salutary protocol system fees are also stable and easy to calculate “instead of the highly variable and difficult to calculate ‘defacto fee’" associated with the ‘implicit cost’ of holding tokens. In this way, salutary protocol system fees function like a sink, because they are “a more transparent and explicit way to figure out what the value of the protocol tokens should be.” Example: Open Index Protocol calculates & enforces the minimum cost associated with publishing the data to the blockchain and associated storage/distribution cost and then connects the markets of: front end/artist, front end/audience, miner/artist, miner, front end, artist/promoter, artist/audience.
Comparison of digital distribution services
|Service||Offering||Cost to Users||Revenue||Creator Share of Revenue|
|Open Index Protocol||Self Publishing of Downloadable and Streaming Music, TV, Film and Videos||A La Carte, Subscription and Ad-Based||Varies based on use||Completely up to creator||As much as 100% to creator|
|iTunes||Purchasable Music||A La Carte Only||$0.99-$2.19 per track||Varies based on use||70% to creator|
|Spotify||Streaming Music||Subscription & Ad-Based||$9.99 per month||~$0.0072 per play||70% to creator|
|Pandora||Algorithm-Based Streaming Music||Subscription & Ad-Based||$4.99 per month||~$0.0014 per play||70% to creator|
|SoundCloud||Self Publishing of Streaming Music||Subscription & Ad-Based||$7-$15 per month||Unknown||Unknown|
|Netflix||Streaming TV/Film||Subscription Only||$7.99 per month||Unknown||Unknown|
|YouTube||Self Publishing of Streaming Video||Subscription & Ad-Based||$9.99 per month||~$0.0005 per play||55% to creator|
|Crackle||Streaming TV/Film||Ad-Based Only||$0||Unknown||Unknown|
- Alice and Bob: An explanation of the various kinds of users and service providers in the Open Index Protocol eco-system.
- Specification: The rules, variables, formulae and thresholds that make up Open Index Protocol.
- Message Protocol: The JSON schema of various kinds of OIP messages and definitions of all terms used within them.
- Security Considerations: the questions, solutions & security considerations that have been analyzed.
- Back to the Main Page
Other Open Index Protocol Applications in Development
- Token.fm streaming music player
- Caltech/Jensen Lab's Tomography Public Database
- Robo3D's "3DRM" marketplace
- YouTubexit.com video archiver
- eVue Digital Labs
- Alexandria v0.4.0 Video Demo
- Alexandria v0.5.1 Video Demo
- Presentation at the Decentralized Web Summit
- Washington Post: Data of 143 million Americans exposed in hack of credit reporting agency Equifax
- 9 Examples of How Smartphones Are Eavesdropping on You
- Wired: A Hacker Can Turn an Amazon Echo into a Wiretap
- BBC: Turkey social media ban raises censorship fears
- Juan Benet IPFS Presentation
- YouTube: 1 Billion Viewers, No Profit; The Wall St Journal
- Blockchain – What is Permissioned vs Permissionless?
- Sharing Economy;wikipedia.org
- What does “permissionless” mean?