From abd7c3ffb494a7154d731ebec7d65b6f1d17842c Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 10 Feb 2021 10:42:54 +0100 Subject: [PATCH 01/66] Create DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- .../DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 509 ++++++++++++++++++ 1 file changed, 509 insertions(+) create mode 100644 OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md new file mode 100644 index 0000000..562ea2e --- /dev/null +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -0,0 +1,509 @@ +# Decentralized Data Delivery Markets (3DMs) - Open Problem Statement + +DO NOT MERGE WITHOUT CREATING DOCTOC https://www.npmjs.com/package/doctoc + +# Overall + +## Short description + +With the emergence of Decentralized Storage Networks and the rapid decrease in the price of storage services and hardware, there is a rapidly growing need to leverage the additional storage capacity contributed to Decentralized Storage Networks by several new players, such as end-users and use it to deliver a reliable and high quality storage but also delivery services. Similarly to Content Delivery Networks (CDNs) for the traditional Cloud Storage market, we now have the opportunity to build **Decentralised CDNs **for Decentralised Storage Networks. The Decentralized Data Delivery Markets (3DMs) Open Problem covers all the essential areas of work that need to be studied in order to create **a fully permissionless free market for data delivery that supports fair data exchange** on the service provided. + +## Long description + +Serving content globally at scale is a hard technical problem as evidenced by the multiple decades of innovation and improvements in the Content Delivery Networks field. This challenge becomes even more interesting once we move away from centralized cloud infrastructures, which is typically managed and monitored by a single party, to a decentralized network that is permissionless, likely anonymous, constantly changing (i.e. high node churn) and without access to the convenience of a third party mediator system that facilitates the fair exchange of goods (e.g. credit providers). The benefits of moving to a decentralised setting are significant, however: cheaper storage and delivery services, resilience against business failures (i.e., the network and all its components are not dependent on business decisions made by a single entity), moving away from the personal data-based business models, as well as a significantly lower barrier to entry for new players who want to contribute to the system. + +At the same time, common problems addressed in the past, in traditional CDNs, reappear, albeit in a different dimension as we move to a decentralized network design space. Some of these problems are: the global orchestration of the system, the efficient allocation and use of the resources, or a clear visibility of the content being retrieved from the network. + +We introduce the field of Decentralized Data Delivery Markets (3DMs) to the world as a new field of research and innovation and one that has a rapidly increasing necessity of existence. Decentralised storage networks, such as Filecoin, have reached unprecedented storage capacity commitments (of over 2.5EB of cold storage in Jan 2021) and continue to grow rapidly. There is an urgent need to complement decentralised storage with decentralised data delivery as these storage networks are seeking for a solution to deliver the data stored in their network to their end users, while meeting the expectations that end-users are used to with the centralised services of today. + +We envision a 3DM as being: + +* A permissionless and open data delivery market, composed of one or more crypto-economic models +* A fair exchange relationship between service providers and users +* A cost efficient way for publishers to distribute their content +* A unique set of business opportunities including: + * Delivering data to end users + * Carrying data between multiple disconnected locations or with high latency between them (Data Muling and/or Package Switching) + * Create ephemeral distribution networks for large gatherings + * Power the next generation of Web 3.0 applications + +This new field is ripe with unique business opportunities given the growing demand from users for access to large datasets, videos and so on, the panoply of super-powered mobile devices that end users have access to and the limitations imposed by physics in ultimately moving increasingly large data at high speeds through fiber. + +In this Open Problem definition, we introduce the multiple areas of research (or subOpenProblems), what we know so far for each one of them and link to some new ideas being explored. + +### Definition of the actors + +For convenience and shared understanding, we define here the agents within a 3DM network. These are: + +* **Clients** - Fetching data from Providers +* **Providers** - Offer the service of data delivery to clients +* **Content Publishers** - The creators of content (or intermediaries) that want to have content distributed. There might be situations in which Content Publishers are willing to pay for the service consumed by Clients + + +### Expected requirements + +We present this list as a guide to protocol designers of what we expect a 3DM implementation to meet. This list is non-exhaustive and presents some flexibility between MUST and SHOULD. The requirements identified are: + +* **_MUST be Decentralized and not just federated_** + * Anyone should be able to join and leave at any time, without requiring permission +* **_The exchanges of value_** **_MUST be verifiable_** + * The payment for bandwidth/latency in token should match what has been agreed and fulfilled + * The payment should be fulfilled if the SLA is fulfilled + * Parties should be able to verify that the service provided was correct (e.g. received the right file) +* **_SHOULD be Trustless_** + * Ideally, the network can perform the operations without having to leverage Trust in third parties and/or the participants of the exchange + * Nevertheless, Trust can, and should, be considered as many designs might benefit from an element of Trust to increase the quality of service (e.g. reputation supports relaxing constraints) +* **_The network SHOULD do resource allocation in an efficient way_** (or as efficiently as possible) + * Should not be an afterthought process +* **_Providers SHOULD coordinate and accept pre-fetching of content (warm up the catches)_** + * This is in contrast to IPFS’s core principle of replicating only after explicit request by a peer + * This is done in order to make the network more efficient. However it is not mandatory with the setting being left up to the Provider. + +Additional expectations (i.e. bonus points): + +* Data SHOULD be readily available without having to wait for the [unsealing process](https://spec.filecoin.io/#section-glossary.seal) at storage nodes (e.g. Filecoin Storage Miners). +* Data retrieval SHOULD be anonymous, privacy-preserving. + +Other considerations (not requirements but good to think about): + +* It is expected that content will always be identified and retrieved by CIDs +* The provider network will likely not provide random access to files, i.e., items will be named at the object level. +* In case there is an auction market, it should always be available (Liveness) + * There needs to be a clever and fast way to match offers to bids + +# Areas of Work + +Throughout the exploration of the design space, we identified 3 (+1 bonus opportunity) areas of work that need to be explored in order to make 3DMs Fair, Decentralized and Efficient. The areas are listed below. + +## Data Delivery Metering & Fair Exchange + +We believe that this area of work is where the main crux for Decentralized Data Delivery Markets come to reality. Without it (i.e. without relying on a verified mediator to serve as escrow and referee), it will be impossible to have a fully permissionless, decentralized and open market for data delivery. + +#### How it is done traditionally + +In traditional or Web 2.0 setups (e.g. CDNs, Mirrors, Static Servers and so on), the correct metering happens on the server side and it is done by the provider which the clients trust to do the right measurement. This measurement translates into a statement of how much the service got used and a respective invoice and request for payment is issued. This payment is facilitated by a trusted third party (e.g. credit card company) and a legal contract that can be used to resolve disputes in case the client fails to pay to the provider and/or the provider fails to deliver the service promised to the client. + +In a decentralized environment that strives to be permissionless, we can’t expect to rely on such legal agreements and third parties and so, we essentially need a way to prove fair exchange. That is, we need to be able to prove that the service was delivered correctly and the metering of the provided service was done correctly. This will, in turn, verify the exchange between the two parties and issue the correct payment (i.e. fair exchange). + +This trustless property is really important as we know that markets with no mediators, escrows and other assurance services (e.g. underground markets) are prone to scams, as the users have to put a lot of trust that the provider will execute on the agreement. In a trustless environment, once the transfer is done, there is no way to contest. + +#### Properties desired + +* The exchanges of value MUST be verifiable and correct + * Fairness: + * The payment MUST be fulfilled if the SLA is fulfilled + * The payment for bandwidth/latency in token SHOULD match what has been agreed and fulfilled + * Verifiability: + * Both parties MUST be capable of verifying that the exchange was performed correctly + * Bonus: Anyone SHOULD be able to verify that the exchange was performed correctly +* SHOULD be Trustless + * Ideally, the network can perform the operations without having to leverage Trust + * Nevertheless in many designs it might be needed to leverage reputation as a trust vehicle to guarantee high quality of service and/or reliability. + +#### How is Data Delivery Metering different from Fair Exchange + +The field of Fair Exchange overlaps in part with Service Metering. Some notable differences are: + +* Fair Exchange can be used for non-digital goods also +* Fair Exchange is traditionally between 2 parties +* Metering should be available to properly measure the service quality and service provided by one or more parties (e.g. streaming from multiple endpoints) + +In review, the Metering & Fair Exchange fields end up contributing to each other and it is likely that the intersection of both is needed for a sound solution. + +#### Known challenges + +* Fairness + * Making it Trustless + * How to verify that the provider has the ability to delivery the file (e.g. stored locally or that can pull it from the network) + * How to verify that the file being transferred is the one requested, before a large amount of resources has been invested (i.e., before the entirety of the file has been transmitted) + * How to verify that the client has received the file + * How to verify and reward that SLA were guaranteed? + * How to avoid collusion when adding a third-party (e.g. Referee)? + * How to avoid grieving attacks (make one party pay for fees)? + * How to avoid a malicious actor causing un-rewarded work, hence wasting the others’ resources? +* Experience + * How to make the transfers start instantaneously? + * How to support others (e.g. Content Publishers) paying for the usage? + * How to make it private (e.g. that others don’t know who is requesting what)? +* Performance + * How to overcome the send-and-halt in order to max bandwidth throughput + * How to support multipath (i.e. fetching from multiple sources) + +### State-of-the-art + +When it comes to the state of the art, we found that the existing solutions fall within one of the following categories: + +#### Pay-per-packet + +INSERT DESCRIPTION + +[(David) Original Filecoin payment channels](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.pc4bfmnw2ke0) [4](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.pc4bfmnw2ke0) +[(David) Theta Whitepaper](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.82adp2bb1rys) [4](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.82adp2bb1rys) + +**Known shortcomings of these approaches:** +* TOFILL + + +#### Lock/Unlock access to the resource + +INSERT DESCRIPTION + +[(David) Solving the Buyer and Seller’s Dilemma: A Dual-Deposit Escrow Smart Contract for Provably Cheat-Proof Delivery and Payment for a Digital Good without a Trusted Mediator](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.nnf8jfjvpmgc) [4](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.nnf8jfjvpmgc) +[(David+Alfonso) FairSwap: How to fairly exchange digital goods](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.cbnzknlpc633) [5](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.cbnzknlpc633) +(David) Zero-Knowledge Contingent Payments Revisited: Attacks and Payments for Services [5](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.it3yajc2rdz0) + +**Known shortcomings of these approaches:** +* TOFILL + +#### Optimistic Fair Exchange + +INSERT DESCRIPTION + +[(David) OptiSwap: Fast Optimistic Fair Exchange](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.f2mby9dpar4f) [6](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.f2mby9dpar4f) +[(Irene) Asynchronous Protocols for Optimistic Fair Exchange](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.6ac4zrrdyhqs) [6](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.6ac4zrrdyhqs) +[(Irene) Optimistic Fair Exchange with Multiple Arbiters](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.sjnrx6peoi0w) [7](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.sjnrx6peoi0w) + +**Known shortcomings of these approaches:** +* TOFILL + +#### Reputation based + +INSERT DESCRIPTION + +[(David) Proof of Delivery in a Trustless Network](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.g219mq8bs8ad) [7](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.g219mq8bs8ad) +[(David) Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.9sqsu17nv29v) [8](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.9sqsu17nv29v) +[(David) Mechanisms for Outsourcing Computation via a Decentralized Market](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.8lcwzhd2jj73) [9](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.8lcwzhd2jj73) +[(Irene) Gradual Fair Exchange Class of Constructions](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.nq0pu1otlwlb) [10](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.nq0pu1otlwlb) + +**Known shortcomings of these approaches:** +* TOFILL + +#### Privacy focused + +INSERT DESCRIPTION + +[(David) Blockchain based Privacy-Preserving Software Updates with Proof-of-Delivery for Internet of Things](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.3n5ydbvkkmoc) [10](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.3n5ydbvkkmoc) +[(David) SilentDelivery: Practical Timed-delivery of Private Information using Smart Contracts](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.2q9trcjswmyw) [10](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.2q9trcjswmyw) +[(David) Bulletproofs+: Shorter Proofs for Privacy-Enhanced Distributed Ledger](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.bweiezm16dvh) [10](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.bweiezm16dvh) + +**Known shortcomings of these approaches:** +* TBD + +#### Traditional approaches (close to Web 2.0 world) + +INSERT DESCRIPTION + +[(David) The multimedia blockchain: a distributed and tamper-proof media transaction framework](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.tsv8bvhhclyh) [10](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.tsv8bvhhclyh) +[(Alfonso) Reliable Client Accounting for P2P-Infrastructure Hybrids](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.mnagl72zaw3u) [11](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.mnagl72zaw3u) + +**Known shortcomings of these approaches:** +* TOFILL + +### Known attacks to be mitigated + +Here we list the attacks to consider when designing a solution. These are: + +* Malicious actor forces provider to spend bandwidth without issuing payment (also known as Grieving Attack) + * Type: client fraud + * Consequence: bandwidth-spent +* Malicious actor forces provider to pull the file from storage point (e.g. Filecoin, Cloud Storage, etc), paying the cost without the provider ever getting paid + * Type: client fraud: + * Consequence: currency is spent +* Provider claims it has sent the file, but actually never did + * Type: Provider fraud + * Consequence: Client gets penalized without receiving the service +* Metering Inflation + * Providers report to have used more resources to serve content to clients than what they have actually done. +* Sybil / Throttle attacks + * Description: A set of nodes collude against a content-provider to try and make him go bankrupt (loss of all its “SLA stake”). They all try to download content from that content provider at the same time, so that Providers start consuming the stake from the content providers. If clients are not paying for the service we take the risk of enabling this attack. + +### New ideas being explored + +ResNetLab organized a Research Intensive Workshop on 3DMs, out of which, the following ideas, structured as RFCs emerged: + +* Link to RFCs: + * TOFILL + +## Distribution Graph Forming + +The area of graph forming for Decentralised Data Delivery Markets revolves around three main areas, which simultaneously compose the different goals of the Graph Forming problem: + +* **Content discovery & routing:** the system can forward requests to find content +* **Content placement:** the system should have ways to proactively distribute/replicate copies of data across different Providers in different geographic regions in order to be able to serve content fast. +* **Content copy selection:** the system can choose the optimal copy of the content to serve if multiple copies exist in several Providers, where potentially: + * every copy has a different “asking” price (defined by the cryptoeconomic model) and latency to deliver the content item, and + * every miner achieves different performance and has a different reputation profile. + +In order to achieve the above goals the system needs to have an architectural structure, that is, where do end-users connect (e.g., to a retrieval miner, or to a separate content resolution system), which entity in the architecture makes content resolution and request forwarding decisions and finally, which entity(ies) has(ve) the required knowledge to forward requests to the closest (to the requesting user) copy, _aka_ nearest replica routing. + +We generally consider that retrieval miners are relatively powerful devices with local storage dedicated to storing hot copies of the content, high uptime (close to zero churn) and ideally public IP (i.e., reachable from anywhere). The architecture should ideally be able to take advantage of storage in less-powerful and more ephemerally-connected end-user devices, such as laptops and mobile phones - this is discussed as a different area further down. Although not a strict requirement, this is what will make the decentralised storage network take the full benefit of planetary-scale unused storage. + +It is worth highlighting that the system should be able to serve different use-cases and therefore different setups and architectures could apply depending on the use-case. For instance, the content resolution system can be different between the cases where: i) the application operates based on a closed-content ecosystem (e.g., subscriber-based music or video streaming, where control of the content is solely with the content publisher) and ii) the application operates on a totally open content space, e.g., web. + +Last but not least, the system of overlay retrieval miner nodes has to be permissionless and decentralised. No entity has full control of the network of nodes, as is the case with traditional CDNs, where a single entity is in charge of the network setup and the content served by the system. + +How it is done traditionally (CDNs, P2P CDNs) + +Traditionally, content is stored, hosted, replicated and delivered by centralised CDNs. In those setups, **the CDN entity has full control over: i) the formation of the network**, e.g., where to place servers and how to interconnect them, both in terms of topology and in terms of bandwidth deployed between servers, **ii) the placement and replication of content**, i.e., where to put hot copies of content, as well as **iii) the network elements that are responsible for content resolution**. + +In a centralized setup it is much easier to optimise performance and provide guarantees, compared to the case where **every network node is operated by a separate (legal) entity that operates rationally in a profit-maximizing way**. + +Closer to our setup are P2P CDNs, where end-user equipment is “employed” to contribute to the storage and dissemination of content. Node churn, unreliable connectivity and low bandwidth speeds become a reality in P2P CDNs, but **again the setup of P2P CDNs is controlled and monitored by a single entity/company.** This makes decisions with regard to content placement and replication easier. The most important advantage of a single entity taking care of the P2P CDN setup is **monitoring**, that is, being able to observe the performance of peers and make decisions on where and when to replicate content or assign content distribution decisions. + +Properties + +* The system MUST always be able to discover content and satisfy content requests + * There should never be a “404 Content not Found” error. +* The system MUST replicate content to different storage points in order to reduce delivery times and maximize performance. +* Providers MUST follow the crypto-economic model and the system MUST make sure that Providers do not misbehave. +* The network MUST be a content-addressable network and operate based on content identifiers. +* The system MUST be permissionless and trustless + * Anyone should be free to join and set up a retrieval miner to contribute to the network. + + +### State-of-the-art + +The target area and expected outcome of this topic is to form the network, the interactions between different network entities, as well as the basic protocols through which network entities will communicate. Given this target, we split our investigation in the following three main areas: + +**P2P CDNs:** the network architecture setup in these systems is similar in some ways to that of a 3DM + +* LiveSky: Enhancing CDN with P2P +* P2P as a CDN: A new service model for file sharing (2012) +* OblivP2P: An Oblivious Peer-to-Peer Content Sharing System + +**P2P VoD:** video delivery is an ideal target use-case, with very strict requirements; therefore, the mechanisms proposed in this field and used for P2P VoD can be inspiring + +* Challenges, Design and Analysis of a Large-scale P2P-VoD System +* A Framework for Lazy Replication in P2P VoD +* InstantLeap: Fast Neighbor Discovery in P2P VoD Streaming (2008) +* Proactive Video Push for Optimizing Bandwidth Consumption in Hybrid CDN-P2P VoD Systems + +**Content-Centric Networking:** there are several network architectures and protocols proposed for content-/information-centric networks which can be great inspiration for a permissionless, content-addressable P2P network. + +* Named Data Networking (NDN) +* iCDN - An NDN based CDN +* Analysis and Improvement of Name-based Packet Forwarding over Flat ID Network Architectures (2018) +* High Throughput Forwarding for ICN with Descriptors and Locators (2016) + +Our initial, but thorough investigation resulted in the following potential groups of designs for content discovery and resolution: + +#### DHT-based: + +DHTs are very popular constructions in P2P networks. The DHT system is used as a content resolution, as well as a content and peer routing system. The routing table is split between peers that participate in the DHT, providing higher resilience and scalability. In a DHT-based system, clients “walk” the DHT (iteratively, or recursively) to find the provider record and then directly connect to the peer included in the provider record. + +* **Pros:** tested in the past, engineers have lots of experience with these structures, can become faster and less expensive than the IPFS DHT setup, if: i) re-publishing is done at coarser granularities, which is reasonable if we assume stable connectivity, i.e., low peer churn (reasonable to assume for Providers), and ii) some payment is associated with publishing. It is also reasonable to assume public IP connectivity for Providers. +* **Cons:** slow, several round-trips needed to resolve content (especially in case of iterative DHT lookup). The solution will not scale in the longer term. + +#### Name-based routing: + +In Name-based routing systems, routing hints are integrated as part of the content name; routing tables are filled with routing hints at network setup time by a routing protocol. Routers make hop-by-hop forwarding decisions based on content names seen in requests and routing hints in their routing tables.. Matching between hints and names depend on the structure of the names, e.g., hierarchical vs flat. + +* **Pros:** very fast, enables multicast-like opportunities and on-path caching, which can result in significant savings in case of popular and heavy content (e.g., HD VoD). Also makes routing from browser/limited connectivity clients more feasible. +* **Cons:** design can be(come) complex, security properties not fully studied in the literature. + +#### DNS-style, object-level indexing service: + +The Domain Name System (DNS) uses index tables to resolve names (URLs) to IP addresses of hosts that store the requested name. It is considered by many to be a decentralised system, in the sense that DNS servers can be deployed by anyone running an ISP. At the top-level, however, it is administered by [ICANN](https://www.icann.org/) which is in charge of assigning names and therefore, does not comply with the permissionless nature of 3DMs. We could envision a similar, index-based naming system, which is governed by the [Ethereum Name Service](https://docs.ens.domains/) (ENS) to bypass centralization concerns. Cloudflare has recently introduced a [Name Resolver for the Distributed Web](https://blog.cloudflare.com/cloudflare-distributed-web-resolver/), where IPFS content can be accessed through ENS. + +* **Pros:** can be very fast +* **Cons:** need to rely on ENS (which is fine given the ENS system is deployed and tested in the wild, albeit in smaller scales), or build a similar service + +#### PubSub-style: + +In pubsub systems information propagates in the network based on topics that nodes subscribe to. Notifications about new content being published in the system can propagate through a dedicated channel, or more realistically, applications can have their own topics to disseminate information about content published within the remit of their application. + +* **Pros:** simple approach, lots of literature and testing in dozens of systems, engineers have lots of experience with similar protocols +* **Cons:** too slow for real-time content delivery, can take seconds to find content, can’t scale in the longer term + +### Known attacks to be mitigated + +* Providers provide false content discovery and routing information +* Providers serve bogus content +* Providers provide false provider records (i.e., they claim that they have and can serve content that they actually do not have) +* Cache poisoning +* Privacy attacks: there is a bunch of privacy-related to be taken into account + +### New ideas being explored + +ResNetLab organized a Research Intensive Workshop on 3DMs, out of it, the following ideas, structured as RFCs emerged: + +* Link to RFCs TO FILL + +## CryptoEconomic model for Data Delivery + +The aim of the cryptoeconomic model is to build an incentive system to ensure that all actors in the network are aligned towards the same goal: to deliver data efficiently. The cryptoeconomic model is tightly coupled to all of the areas presented above, offering a way to incentivize desired behaviors in the network. Ideally, the economic model should offer the substrate to build a decentralized and trustless infrastructure for data delivery able to compete against traditional CDN infrastructures in terms of cost, scalability, and price. + +#### How it is done traditionally + +In traditional and centralized setups, there is no underlying need to align the incentives of all the participants in a data delivery network. The relationships between entities in the system are governed by a set of multilateral legal agreements between the different parties. These agreements offer the basic level of trust to ensure the good behavior of all entities in the system (clearly stating the SLA, and punishments for misbehaving or not fulfilling the contract). We currently see this setup in the multilateral agreements between CDNs, ISPs, and Cloud Providers to share resources from their infrastructures for a certain price in order to build their network of relationships that allow them to provide their content delivery services efficiently. + +In decentralized setups, it is not possible to use legal agreements to orchestrate the relationship between entities in the system. In recent deployments of hybrid CDNs, traditional, centralised CDNs leverage the resources of end users to deliver data with enhanced QoS reducing the load on their infrastructure and as a result the cost of upgrading/extending their infrastructure. End users are incentivized to contribute to the system with their upload bandwidth by making them eligible for enjoying a better service. This additional reward is used to incentivize a behavior in the system (in this case users contributing with bandwidth in hybrid CDNs). However, the fact that there is still a centralized infrastructure managed by an entity with a greater power over the rest of actors in the system is enough to prevent misbehaviors. This removes the need to design stronger reputation or reward systems to prevent misbehaviors. This does not represent a trustless setup. + +For decentralized and trustless setups, two main schemes have been traditionally used to align the incentive of the system’s participants and prevent misbehaviors: + +* Trustless networks governed by a common consensus, like public blockchains, use economic models based on token rewards and clear punishments built upon a secure consensus algorithm. The consensus algorithm gives a basic layer of trust where all participants “keep an eye on one another” preventing misbehaviors, and upon which the economic model is built to align all their goals. A good example of this model is Bitcoin, where the Proof of Work consensus algorithm prevents attacks, and the mining rewards and deflationary nature of the currency ensures that Providers will be incentivized to keep serving, and users to keep using and holding the network’s currency. +* For trustless P2P networks where there is no common protocol run by all the participants of the network, other schemes such as reputation systems, or credit networks need to be implemented to orchestrate the relationships and good behaviors of all entities in the system. + +In the same way public blockchains leverage the schemes and security properties in place to design their economic model, 3DMs will have to leverage the metering and graph forming designs in place to design the right economic model. + +Finally, auction markets and game theoretical models have been traditionally used to study and achieve efficient resource allocation in P2P networks and decentralized systems. + +#### Properties desired + +* The model MUST foster fair competition and avoid the creation of monopolies. + * We should prevent superlinear advantages from winning/completing a high percentage of retrieval deals in the system. +* The model SHOULD discourage, prevent, and punish misbehaviors. + * Or the other way around, good behaving entities should be rewarded by the model. + * In its different layers, the system will include schemes to prevent attacks in the network (client metering, sybil attacks, data-ransoming, etc.). However, this economic model should disincentivize these kinds of misbehavior in advance (from a game theoretical approach, the system should target a Nash Equilibrium in the system where misbehaviors are disincentivized). + * It should also avoid “malicious economic attacks” such as a content provider trying to bankrupt its competition by draining the stake of his retrieval deal. +* The model SHOULD foster collaboration between entities to benefit users perceived QoS / QoE. + * The model should reward entities that collaborate to serve content efficiently and with high QoS to users. + * This will also result in an efficient allocation of resources in the network. +* If any part of the economics of the system ends up being orchestrated by an auction market, this market MUST be always available. + * And should match offers to bids in a fast and efficient way. + +### State-of-the-art + +As described above, the outcome of this topic should be to design an economic model that aligns the goals of all profit-maximizing entities in the system while preventing misbehaviors. Given this target, we split our investigation in the following areas: + +##### Economics of Hybrid CDNs and P2P networks + +The economics of current Hybrid CDN deployments give a good understanding of the system and cost models involved in the delivery of content at scale. This, added to the different analysis and design proposals of economic models for content delivery in P2P networks represent a good foundation to understand the complexity of the problem and potential ways to tackle it. + +**_Related Papers:_** + +* An economic mechanism for request routing and resource allocation in hybrid CDN–P2P networks +* An economic replica placement mechanism for streaming content distribution in Hybrid CDN-P2P networks +* How Neutral is a CDN? An Economic Approach +* Value Networks and Two-Sided Markets of Internet Content Delivery +* Bilateral and Multilateral Exchanges for Peer-Assisted Content Distribution + +**_Learnings: _** + +* Protocols and systems involved in the cost model of a Hybrid-CDN. +* Models of P2P content delivery networks. + +##### Game theoretical models and reward/reputation systems in P2P networks + +How to design the right incentive and reputation models to incentivize certain behaviors in P2P networks has been a widely studied problem for some time now. From reputation systems where nodes track the level of good behavior of their peers and cooperate with them according to their grade; to incentive systems where good behavior is explicitly rewarded. To theoretically evaluate the strength of these schemes, they are generally mod game theoretical models are generally an objective way to evaluate the strength of these schemes, and to understand if the proposals can achieve Nash Equilibrium. + +**Related Papers:** + +* Scrivener: Providing Incentives in Cooperative Content Distribution Systems +* Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks +* Incentivizing Peer-Assisted Services:A Fluid Shapley Value Approach +* Incentive and Service Differentiation in P2P Networks: A Game Theoretic Approach +* On Incentivizing Caching for P2P-VoD Systems +* Bar Gossip + +**_Learnings: _** + +* Use different metrics to evaluate the level of good behavior of peers. +* Examples of game theoretic models to approach the evaluation of these protocols. +* Schemes to avoid attacks to which these systems can be vulnerable (such as sybil attacks). + +##### Credit networks and Token Designs + +Credit networks provide liquidity in markets where transaction volumes are close to balanced in both directions. They offer a way of performing payments and rewarding behaviors without the need of a common consensus or dedicated agreements between all the entities in the systems. Credit networks require no “a priori” trust relationships when they are backed by on-chain escrows, so they represent an interesting approach to tackle the economics of 3DMs. + +**_Related Papers:_** + +* Collusion-resilient Credit-based Reputations for Peer-to-peer Content Distribution +* Liquidity in Credit Networks: A Little Trust Goes a Long Way +* Liquidity in Credit Networks with Constrained Agents +* CAPnet: A Defense Against Cache Accounting Attacks on Content Distribution Networks +* MicroCash: Practical Concurrent Processing of Micropayments +* Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks + +**_Learnings: _** + +* Economics of credit networks. +* Using credit networks for fast payments and reputation systems. +* Credit networks for content distribution in P2P networks. + +##### Auctions, decentralized markets, and efficient resource allocation + +Auctions and decentralized markets have traditionally been a good way of allocating resources efficiently in a decentralized manner. These papers analyze different proposals of auction systems and decentralized markets that can aid the design of efficient resource allocation in 3DMs. + +**_Related Papers:_** + +* Edge-MAP: Auction Markets for Edge Resource Provisioning +* A Market Protocol for Decentralized Task Allocation +* On the Efficiency of Sharing Economy Networks +* Resource Allocation in Market-based Grids Using a History-based Pricing Mechanism +* Content Pricing in Peer-to-Peer Networks + +**_Learnings: _** + +* Use of auctions as an efficient way to allocate resources in a decentralized system. +* Local auction protocols. to drive the offer and demand of resources. +* Decentralized market protocols for efficient pricing and distribution of resources. + +### Known attacks to be mitigated + +* **Sybil attacks:** Actors trying to game the system by indiscriminately generating a large amount of entities and gaining large influence over the system (forging client retrievals, overestimating the use of resources of a provider, preventing access from some resource in the network, making actors in the system dedicate resources to useless work, etc.). +* **Collusion attacks:** Several independent actors in the system colluding to perform attacks in order to gain large influence in the network. The type of attacks that can be performed with a collusion attack are similar to the ones performed in a sybil attack, but without having to generate “pseudonymous identities”. +* **Data ransoming:** This attack takes place when a provider agrees to serve some data from a publisher, but ends up preventing its retrieval from clients until a ransom is paid. An alternative form of this attack occurs when a provider is serving chunks of content to a client, and doesn’t deliver the last couple of chunks until it pays a ransom. +* **Malicious economic attacks:** Attacks aimed at economically harming actors in the systems. For instance, clients trying to get a provider or content publisher bankrupt, providers offering abusive prices to content publishers, providers and clients colluding to avoid the economic rewards of target entities, etc. + +### New ideas being explored + +ResNetLab organized a Research Intensive Workshop on 3DMs, out of it, the following ideas, structured as RFCs emerged: + +* Link to RFCs TO FILL + +## Additional: Opportunistic deployment + +An area adjacent to the Distribution Graph Forming one is that of extending the network beyond Providers, as defined earlier, to also include more ephemerally-connected, end-user devices. Such devices can include laptops, desktops, (futuristic) storage-equipped WiFi Access Points, or even mobile smartphone and tablet devices. + +We call these environments “opportunistic deployments” to reflect their unpredictability in terms of availability, uptime, resource quality and quantity. In contrast to RMs, as defined earlier that are expected to have stable, public and high-bandwidth connectivity, 3DM opportunistic deployments can utilise everyday user devices to extend the storage footprint of RMs and create a wealth of new network connectivity and business opportunities. + +Although this area seems to sit on the periphery of 3DMs, it is actually a highly impactful area, as it realises the vision of Decentralised Storage Networks in general and Filecoin, in particular, of regular end-users sharing their own resources to contribute to the network. That said, we place high value in capturing this opportunity and offering to end-users the opportunity of being rewarded for their contribution to the network. + +### State-of-the-art + +The literature in the field of Opportunistic and Delay-Tolerant Networks is vast. We have narrowed down the scope of the literature that we reviewed and identified the following most-promising 3DM applicability areas. + +#### Transient Providers + +We consider end-users that store content in their ephemerally connected devices (laptops, or mobile phones) and provide them to the network through (one or more) Providers, as defined in the “Distribution Graph Forming” area. These service providers are called “Transient Providers (TRMs) to depict their ephemeral nature in resource availability and time. TRMs can have some economic relationship with one or more RMs, e.g., RMs can “recruit” TRMs to expand their storage capacity and footprint. In this case, RMs have to maintain their own monitoring mechanisms and do resource allocation to ensure that they are getting desirable levels of service from their group of TRMs. + +Opportunistic D2D + +This area of research and deployment is closer to traditional Delay-Tolerant Networks (DTNs). We envision applications where mobile devices contribute to the distribution of content in the mobile domain. These can be akin to previously proposed [User-Operated Mobile Content Distribution Networks](https://drive.google.com/file/d/1vvtQ7MJb4YFRXxeo2GewGh8EVYEqV-iJ/view?usp=sharing), or applications to realise concepts such as “[Floating Content](https://drive.google.com/file/d/1yOniHbAjLYv3q09pff7gnulOwT_NYCYG/view?usp=sharing)”. + +### Known shortcomings + +**User Mobility** + +When a user is “registered” under one address/network location to serve content (at least as far as its Retrieval Miner is concerned) and then moves to another network location, the content is not possible to find anymore. + +* Brings up content mobility issues and “content churn” +* Similar to node churn, this is a very tricky problem: how can you find content that was linked to some network address, when this address is not valid anymore. You can, of course, update the record if you know where the record lives, but if the record is “allowed” to be provided by anyone, then it’s difficult to even find it. +* Simple solution: route through “home” gateway, similar to Mobile IP. Increases delay, is not ideal, but can work +* There is a lot of work on “producer mobility” in the literature that we should look at. + +**Privacy - Efficiency Tradeoff** + +In mobile opportunistic communications, the two are often conflicting. On one extreme, you have efficient (low-replication) solutions that make use of a lot of node data (geolocation history/patterns, contact history/patterns, current geographic destination, current velocity vector). On the other extreme, you have naive epidemic dissemination solutions that require no node profiling but introduce data redundancy. There is a continuum in between, as well as a third variable, which is QoS: in the extreme, if willing to wait forever for delivery, you can blindly pass a message around without replication. + +Keeping with the ethos of our projects, we believe that the most suitable solutions are those that don’t require sensitive data, including data related to location or contact history, from which identity and behaviour can be easily extracted. Aggregate transitive metrics (e.g. total expected time to destination measures) can provide a measure of delivery likelihood without disclosing the full contact vector or the nature of the contact (direct vs. indirect), and may be a useful compromise. Other data, including interests/subscriptions, is akin to what is already used e.g. in pubsub protocols and not considered to be any more privacy-invading. + +**Security** + +Like with any other distributed routing/forwarding approach, malicious nodes can interfere with message propagation and delivery. Because of the disconnected nature of the network, such attacks are harder (or at least slower) to detect and counteract. Secure-by-design approaches are likely to suffer from additional overhead (either in data or complexity) and _may_ be precocious when compared to the security model of other, more critical, parts of the network. + +**Energy Efficiency and Battery Consumption** + +Opportunistic communications that rely heavily (or exclusively) on user devices always come with the challenge of usability from the energy/battery perspective. Of course, incentives through FIL will alleviate some of it, but even then the incentive would have to be such that it overcomes the “running out of battery” shock. Techniques to overcome this can be as simple as disabling user participation when the battery level is below X%. + +**Incentives** + +The incentive to participate and the cryptoeconomic model behind the opportunistic part of the network is quite likely going to be incompatible with the one in the main Distribution Graph Forming area that applies to large/stable Providers. Nevertheless, incentives is a core part of this area and has traditionally been a stumbling block for the deployment of opportunistic networks. + +### New ideas being explored + +ResNetLab organized a Research Intensive Workshop on 3DMs, out of it, the following ideas, structured as RFCs emerged: + +Link to RFCs +* TO FILL From c4abc36f7a15e283e0386a920471c198c998565a Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 10 Feb 2021 11:54:36 +0100 Subject: [PATCH 02/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 562ea2e..80137c6 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -6,7 +6,7 @@ DO NOT MERGE WITHOUT CREATING DOCTOC https://www.npmjs.com/package/doctoc ## Short description -With the emergence of Decentralized Storage Networks and the rapid decrease in the price of storage services and hardware, there is a rapidly growing need to leverage the additional storage capacity contributed to Decentralized Storage Networks by several new players, such as end-users and use it to deliver a reliable and high quality storage but also delivery services. Similarly to Content Delivery Networks (CDNs) for the traditional Cloud Storage market, we now have the opportunity to build **Decentralised CDNs **for Decentralised Storage Networks. The Decentralized Data Delivery Markets (3DMs) Open Problem covers all the essential areas of work that need to be studied in order to create **a fully permissionless free market for data delivery that supports fair data exchange** on the service provided. +With the emergence of Decentralized Storage Networks and the rapid decrease in the price of storage services and hardware, there is a rapidly growing need to leverage the additional storage capacity contributed to Decentralized Storage Networks by new players, including end-users, and use it to deliver reliable and high-quality storage and delivery services. Similarly to Content Delivery Networks (CDNs) for the traditional Cloud Storage market, we now have the opportunity to build **Decentralised CDNs** for Decentralised Storage Networks. The Decentralized Data Delivery Markets (3DMs) Open Problem covers all the essential areas of work that need to be studied in order to create **a fully permissionless free market for data delivery that supports fair data exchange** on the service provided. ## Long description From cec294794a9e997fe0bbad2c873d60b25f3e1a77 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 10 Feb 2021 11:55:55 +0100 Subject: [PATCH 03/66] Apply suggestions from code review Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- .../DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 39 +++++++++---------- 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 80137c6..f0b0cba 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -10,11 +10,11 @@ With the emergence of Decentralized Storage Networks and the rapid decrease in t ## Long description -Serving content globally at scale is a hard technical problem as evidenced by the multiple decades of innovation and improvements in the Content Delivery Networks field. This challenge becomes even more interesting once we move away from centralized cloud infrastructures, which is typically managed and monitored by a single party, to a decentralized network that is permissionless, likely anonymous, constantly changing (i.e. high node churn) and without access to the convenience of a third party mediator system that facilitates the fair exchange of goods (e.g. credit providers). The benefits of moving to a decentralised setting are significant, however: cheaper storage and delivery services, resilience against business failures (i.e., the network and all its components are not dependent on business decisions made by a single entity), moving away from the personal data-based business models, as well as a significantly lower barrier to entry for new players who want to contribute to the system. +Serving content globally at scale is a hard technical problem, as evidenced by the multiple decades of innovation and improvements in the Content Delivery Networks field. This challenge becomes even more interesting once we move away from centralized cloud infrastructure, which is typically managed and monitored by a single party, to a decentralized network that is permissionless, possibly anonymous, constantly changing (i.e. with high node churn) and lacking access to the convenience of a third party mediator system that facilitates the fair exchange of goods (e.g. credit providers). The benefits of moving to a decentralised setting are significant, however: cheaper storage and delivery, resilience against business failures (i.e., the network and all its components are not dependent on business decisions made by a single entity), independence from the personal data-driven business models, as well as a significantly lower barrier to entry for new players who want to contribute. -At the same time, common problems addressed in the past, in traditional CDNs, reappear, albeit in a different dimension as we move to a decentralized network design space. Some of these problems are: the global orchestration of the system, the efficient allocation and use of the resources, or a clear visibility of the content being retrieved from the network. +At the same time, problems already addressed in the past for traditional CDNs reappear, albeit in a different dimension, as we move to a decentralized network design space. These problems include the global orchestration of the system, the efficient allocation and use of the resources, and clear visibility of the content being retrieved from the network. -We introduce the field of Decentralized Data Delivery Markets (3DMs) to the world as a new field of research and innovation and one that has a rapidly increasing necessity of existence. Decentralised storage networks, such as Filecoin, have reached unprecedented storage capacity commitments (of over 2.5EB of cold storage in Jan 2021) and continue to grow rapidly. There is an urgent need to complement decentralised storage with decentralised data delivery as these storage networks are seeking for a solution to deliver the data stored in their network to their end users, while meeting the expectations that end-users are used to with the centralised services of today. +We introduce Decentralized Data Delivery Markets (3DMs) as a new field of research and innovation and one with rapidly increasing relevance. Decentralised storage networks, such as Filecoin, have reached unprecedented storage capacity commitments (of over 2.5EB of cold storage in Jan 2021) and continue to grow rapidly. There is an urgent need to complement decentralised storage with decentralised data delivery as these storage networks seek a solution to deliver the data stored in their network to end-users while meeting the expectations users have of the centralised services of today. We envision a 3DM as being: @@ -24,20 +24,20 @@ We envision a 3DM as being: * A unique set of business opportunities including: * Delivering data to end users * Carrying data between multiple disconnected locations or with high latency between them (Data Muling and/or Package Switching) - * Create ephemeral distribution networks for large gatherings - * Power the next generation of Web 3.0 applications + * Creating ephemeral distribution networks for large gatherings + * Powering the next generation of Web 3.0 applications -This new field is ripe with unique business opportunities given the growing demand from users for access to large datasets, videos and so on, the panoply of super-powered mobile devices that end users have access to and the limitations imposed by physics in ultimately moving increasingly large data at high speeds through fiber. +This new field is ripe with unique business opportunities given the growing demand from users for access to large datasets, videos, and other data-heavy content, the panoply of powerful mobile devices that users carry, and the limitations imposed by physics in ultimately moving increasingly large data at high speeds. In this Open Problem definition, we introduce the multiple areas of research (or subOpenProblems), what we know so far for each one of them and link to some new ideas being explored. ### Definition of the actors -For convenience and shared understanding, we define here the agents within a 3DM network. These are: +For convenience and shared understanding, we first define the agents within a 3DM network. These are: -* **Clients** - Fetching data from Providers -* **Providers** - Offer the service of data delivery to clients -* **Content Publishers** - The creators of content (or intermediaries) that want to have content distributed. There might be situations in which Content Publishers are willing to pay for the service consumed by Clients +* **Clients** - Agents that fetch data from Providers. +* **Providers** - Agents that offer data delivery services to Clients. +* **Content Publishers** - Agents that create content and want to have it distributed (or intermediaries thereof). ### Expected requirements @@ -49,31 +49,30 @@ We present this list as a guide to protocol designers of what we expect a 3DM im * **_The exchanges of value_** **_MUST be verifiable_** * The payment for bandwidth/latency in token should match what has been agreed and fulfilled * The payment should be fulfilled if the SLA is fulfilled - * Parties should be able to verify that the service provided was correct (e.g. received the right file) + * Parties should be able to verify that the service provided was correct (e.g. that they received the right file) * **_SHOULD be Trustless_** - * Ideally, the network can perform the operations without having to leverage Trust in third parties and/or the participants of the exchange - * Nevertheless, Trust can, and should, be considered as many designs might benefit from an element of Trust to increase the quality of service (e.g. reputation supports relaxing constraints) + * Ideally, the network can perform the operations without having to leverage trust in third parties and/or the participants of the exchange + * Nevertheless, designs might benefit from an element of trust to increase the quality of service (e.g. reputation mechanisms may allow for relaxed constraints) * **_The network SHOULD do resource allocation in an efficient way_** (or as efficiently as possible) - * Should not be an afterthought process -* **_Providers SHOULD coordinate and accept pre-fetching of content (warm up the catches)_** + * Should not be an afterthought +* **_Providers SHOULD coordinate and accept pre-fetching of content (warm up the caches)_** * This is in contrast to IPFS’s core principle of replicating only after explicit request by a peer * This is done in order to make the network more efficient. However it is not mandatory with the setting being left up to the Provider. Additional expectations (i.e. bonus points): * Data SHOULD be readily available without having to wait for the [unsealing process](https://spec.filecoin.io/#section-glossary.seal) at storage nodes (e.g. Filecoin Storage Miners). -* Data retrieval SHOULD be anonymous, privacy-preserving. +* Data retrieval SHOULD be anonymous and privacy-preserving. -Other considerations (not requirements but good to think about): +Other considerations: * It is expected that content will always be identified and retrieved by CIDs * The provider network will likely not provide random access to files, i.e., items will be named at the object level. -* In case there is an auction market, it should always be available (Liveness) - * There needs to be a clever and fast way to match offers to bids +* In case there is an auction market, it should always be available and have a fast way to match offers to bids # Areas of Work -Throughout the exploration of the design space, we identified 3 (+1 bonus opportunity) areas of work that need to be explored in order to make 3DMs Fair, Decentralized and Efficient. The areas are listed below. +Throughout the exploration of the design space, we identified 3 areas of work that need to be explored in order to make 3DMs Fair, Decentralized and Efficient, as well as a bonus opportunity. The areas are listed below. ## Data Delivery Metering & Fair Exchange From ab83e766dda6dceee3eac7b90fd7924f4f248410 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 10 Feb 2021 12:10:32 +0100 Subject: [PATCH 04/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index f0b0cba..8380a1c 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -134,13 +134,15 @@ When it comes to the state of the art, we found that the existing solutions fall #### Pay-per-packet -INSERT DESCRIPTION +Pay-per-packet solutions offer granular control over what gets paid, enabling the option to verify if the SLA is being fulfilled. However, their setup cost and latency overhead puts them behind on being able to max out the available underlay resources. + +These type of solutions have been found/highlighted: -[(David) Original Filecoin payment channels](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.pc4bfmnw2ke0) [4](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.pc4bfmnw2ke0) -[(David) Theta Whitepaper](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.82adp2bb1rys) [4](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.82adp2bb1rys) +- [Filecoin Retrieval Market](https://spec.filecoin.io/#section-systems.filecoin_markets.retrieval_market) +- [Theta Whitepaper](https://s3.us-east-2.amazonaws.com/assets.thetatoken.org/Theta-white-paper-latest.pdf?v=1612955013.791) **Known shortcomings of these approaches:** -* TOFILL +* send-and-halt - Because the next packet (unit of service) will only be issued after the payment of the previous, these solutions often miss to max out the bandwidth available in the connection available, increasing the latency of delivery. #### Lock/Unlock access to the resource From b42d13b0d8eb7e77dfd7cfac905f018aecc50a78 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 10 Feb 2021 12:29:37 +0100 Subject: [PATCH 05/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- .../DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 8380a1c..96f8d3f 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -147,14 +147,17 @@ These type of solutions have been found/highlighted: #### Lock/Unlock access to the resource -INSERT DESCRIPTION +These type of solutions focus give the possibility for the client and the provider to verify the actual service before unlocking the payment. One of the main tools used is locking the access to the service by encrypting it with a key and then performing an exchange over the key that gives access to the service. -[(David) Solving the Buyer and Seller’s Dilemma: A Dual-Deposit Escrow Smart Contract for Provably Cheat-Proof Delivery and Payment for a Digital Good without a Trusted Mediator](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.nnf8jfjvpmgc) [4](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.nnf8jfjvpmgc) -[(David+Alfonso) FairSwap: How to fairly exchange digital goods](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.cbnzknlpc633) [5](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.cbnzknlpc633) -(David) Zero-Knowledge Contingent Payments Revisited: Attacks and Payments for Services [5](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.it3yajc2rdz0) +These solutions can be found in several forms: +- [Solving the Buyer and Seller’s Dilemma: A Dual-Deposit Escrow Smart Contract for Provably Cheat-Proof Delivery and Payment for a Digital Good without a Trusted Mediator](INSERT URL) +- [FairSwap: How to fairly exchange digital goods](INSERT URL) +- [Zero-Knowledge Contingent Payments Revisited: Attacks and Payments for Services](INSERT URL) **Known shortcomings of these approaches:** -* TOFILL +* Vulnerable to Grieving attacks. +* Generally, these solutions focus on the fulfilment of the delivery, more so than measuring the speed and rate of the transfer. + #### Optimistic Fair Exchange From 5ca37edb8f18d703586045612036613a55a3ee3d Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 10 Feb 2021 12:32:28 +0100 Subject: [PATCH 06/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 96f8d3f..2580fd4 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -161,14 +161,14 @@ These solutions can be found in several forms: #### Optimistic Fair Exchange -INSERT DESCRIPTION +These solutions build on top of the Fair Exchange ones and adopt an Optimistic approach where some sort of Reputation or Stake is used to give both parties trust that the exchange will occur correctly. If one misbehaves, they can still rely on a dispute system. -[(David) OptiSwap: Fast Optimistic Fair Exchange](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.f2mby9dpar4f) [6](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.f2mby9dpar4f) -[(Irene) Asynchronous Protocols for Optimistic Fair Exchange](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.6ac4zrrdyhqs) [6](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.6ac4zrrdyhqs) -[(Irene) Optimistic Fair Exchange with Multiple Arbiters](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.sjnrx6peoi0w) [7](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.sjnrx6peoi0w) +- [OptiSwap: Fast Optimistic Fair Exchange](INSERT URL) +- [Asynchronous Protocols for Optimistic Fair Exchange](INSERT URL) +- [Optimistic Fair Exchange with Multiple Arbiters](INSERT URL) **Known shortcomings of these approaches:** -* TOFILL +* Setup + Dispute resolution time is non insignificant and non-trivial #### Reputation based From ec1499ca930754427c3d1aeca90e83644fd9ba13 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 10 Feb 2021 12:34:53 +0100 Subject: [PATCH 07/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 2580fd4..437bd41 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -184,14 +184,11 @@ INSERT DESCRIPTION #### Privacy focused -INSERT DESCRIPTION - -[(David) Blockchain based Privacy-Preserving Software Updates with Proof-of-Delivery for Internet of Things](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.3n5ydbvkkmoc) [10](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.3n5ydbvkkmoc) -[(David) SilentDelivery: Practical Timed-delivery of Private Information using Smart Contracts](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.2q9trcjswmyw) [10](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.2q9trcjswmyw) -[(David) Bulletproofs+: Shorter Proofs for Privacy-Enhanced Distributed Ledger](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.bweiezm16dvh) [10](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.bweiezm16dvh) +There are multiple solutions with the goal of delivering readers privacy and writers privacy. These incur an additional cost in setup and/or latency in favor of the additional privacy preserving benefits. Disclaimer: At the time of writing, we haven't dived as deep into these set of solutions. -**Known shortcomings of these approaches:** -* TBD +- [Blockchain based Privacy-Preserving Software Updates with Proof-of-Delivery for Internet of Things](INSERT URL) +- [SilentDelivery: Practical Timed-delivery of Private Information using Smart Contracts](INSERT URL) +- [Bulletproofs+: Shorter Proofs for Privacy-Enhanced Distributed Ledger](INSERT URL) #### Traditional approaches (close to Web 2.0 world) From 8fa69bcaf083e4d945b2435545c783bc662f10c8 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 10 Feb 2021 12:40:03 +0100 Subject: [PATCH 08/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 437bd41..e7fdd91 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -172,15 +172,15 @@ These solutions build on top of the Fair Exchange ones and adopt an Optimistic a #### Reputation based -INSERT DESCRIPTION +Reputation based solutions give the possibility to speed up the transfer and improve the quality of the service, by leveraging the trust built over time between clients & providers. -[(David) Proof of Delivery in a Trustless Network](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.g219mq8bs8ad) [7](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.g219mq8bs8ad) -[(David) Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.9sqsu17nv29v) [8](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.9sqsu17nv29v) -[(David) Mechanisms for Outsourcing Computation via a Decentralized Market](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.8lcwzhd2jj73) [9](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.8lcwzhd2jj73) -[(Irene) Gradual Fair Exchange Class of Constructions](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.nq0pu1otlwlb) [10](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.nq0pu1otlwlb) +- [(David) Proof of Delivery in a Trustless Network](INSERT URL) +- [(David) Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks](INSERT URL) +- [(David) Mechanisms for Outsourcing Computation via a Decentralized Market](INSERT URL) +- Other Gradual Fair Exchange Class of constructions **Known shortcomings of these approaches:** -* TOFILL +* Not a direct shortcoming, but a challenge is that each participant my have different levels of sensitivity with regards to how much they are willing to rely on trust vs. strict verifiability. #### Privacy focused From 5b6b118270f52a52a51bc2f9391e3ef68b4dbd75 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 10 Feb 2021 12:40:53 +0100 Subject: [PATCH 09/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index e7fdd91..4c8ce37 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -174,9 +174,9 @@ These solutions build on top of the Fair Exchange ones and adopt an Optimistic a Reputation based solutions give the possibility to speed up the transfer and improve the quality of the service, by leveraging the trust built over time between clients & providers. -- [(David) Proof of Delivery in a Trustless Network](INSERT URL) -- [(David) Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks](INSERT URL) -- [(David) Mechanisms for Outsourcing Computation via a Decentralized Market](INSERT URL) +- [Proof of Delivery in a Trustless Network](INSERT URL) +- [Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks](INSERT URL) +- [Mechanisms for Outsourcing Computation via a Decentralized Market](INSERT URL) - Other Gradual Fair Exchange Class of constructions **Known shortcomings of these approaches:** From d8a4d4804886a766426f8456f23f1896cb4b4ad1 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 10 Feb 2021 12:42:26 +0100 Subject: [PATCH 10/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 4c8ce37..d77c84c 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -192,13 +192,14 @@ There are multiple solutions with the goal of delivering readers privacy and wri #### Traditional approaches (close to Web 2.0 world) -INSERT DESCRIPTION -[(David) The multimedia blockchain: a distributed and tamper-proof media transaction framework](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.tsv8bvhhclyh) [10](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.tsv8bvhhclyh) -[(Alfonso) Reliable Client Accounting for P2P-Infrastructure Hybrids](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.mnagl72zaw3u) [11](https://docs.google.com/document/d/1hXTP0VRmQsSFcnbxk3Z4sVmdW0CPajogJ6SXTe7zJ34/edit#heading=h.mnagl72zaw3u) +These approaches suggest novel ways to add integrity checks to data transferred or to promote new ways for content distribution, but they don’t depart from the Web 2.0 world of assumptions such as trusted third parties and central points of control. + +- [The multimedia blockchain: a distributed and tamper-proof media transaction framework](INSERT URL) +- [Reliable Client Accounting for P2P-Infrastructure Hybrids](INSERT URL) **Known shortcomings of these approaches:** -* TOFILL +* Not fully aware of the challenges of building a decentralized network ### Known attacks to be mitigated From 78d8096cd08d96152a3c04f5939236078eae9bc2 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:30:15 +0200 Subject: [PATCH 11/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index d77c84c..6c6127d 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -80,7 +80,7 @@ We believe that this area of work is where the main crux for Decentralized Data #### How it is done traditionally -In traditional or Web 2.0 setups (e.g. CDNs, Mirrors, Static Servers and so on), the correct metering happens on the server side and it is done by the provider which the clients trust to do the right measurement. This measurement translates into a statement of how much the service got used and a respective invoice and request for payment is issued. This payment is facilitated by a trusted third party (e.g. credit card company) and a legal contract that can be used to resolve disputes in case the client fails to pay to the provider and/or the provider fails to deliver the service promised to the client. +In traditional or Web 2.0 setups (e.g. CDNs, Mirrors, Static Servers and so on), the correct metering happens on the server and is handled by the provider, which the clients trust to perform correct measurements. This measurement translates into a statement of how much the service was, ultimately resulting in an invoice and request for payment. This payment is facilitated by a trusted third party (e.g. credit card company) and a legal contract that can be used to resolve disputes in case the client fails to pay to the provider and/or the provider fails to deliver the service promised to the client. In a decentralized environment that strives to be permissionless, we can’t expect to rely on such legal agreements and third parties and so, we essentially need a way to prove fair exchange. That is, we need to be able to prove that the service was delivered correctly and the metering of the provided service was done correctly. This will, in turn, verify the exchange between the two parties and issue the correct payment (i.e. fair exchange). From a36816679db5d500faf3b68388e91d0bca88a03b Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:50:07 +0200 Subject: [PATCH 12/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 6c6127d..dd498ce 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -478,7 +478,7 @@ This area of research and deployment is closer to traditional Delay-Tolerant Net **User Mobility** -When a user is “registered” under one address/network location to serve content (at least as far as its Retrieval Miner is concerned) and then moves to another network location, the content is not possible to find anymore. +When a user is “registered” under one address/network location to serve content (at least as far as its Retrieval Miner is concerned) and then moves to another network location, the content is not discoverable/retrievable anymore. * Brings up content mobility issues and “content churn” * Similar to node churn, this is a very tricky problem: how can you find content that was linked to some network address, when this address is not valid anymore. You can, of course, update the record if you know where the record lives, but if the record is “allowed” to be provided by anyone, then it’s difficult to even find it. From 247386c645ec63739d65319c524fcba54c7a1652 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:53:22 +0200 Subject: [PATCH 13/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index dd498ce..7723650 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -82,7 +82,7 @@ We believe that this area of work is where the main crux for Decentralized Data In traditional or Web 2.0 setups (e.g. CDNs, Mirrors, Static Servers and so on), the correct metering happens on the server and is handled by the provider, which the clients trust to perform correct measurements. This measurement translates into a statement of how much the service was, ultimately resulting in an invoice and request for payment. This payment is facilitated by a trusted third party (e.g. credit card company) and a legal contract that can be used to resolve disputes in case the client fails to pay to the provider and/or the provider fails to deliver the service promised to the client. -In a decentralized environment that strives to be permissionless, we can’t expect to rely on such legal agreements and third parties and so, we essentially need a way to prove fair exchange. That is, we need to be able to prove that the service was delivered correctly and the metering of the provided service was done correctly. This will, in turn, verify the exchange between the two parties and issue the correct payment (i.e. fair exchange). +In a decentralized environment that strives to be permissionless, we can’t expect to rely on such legal agreements and third parties and, therefore, need a way to prove fair exchange. That is, we need to be able to prove that the service was delivered correctly and the metering of the provided service was done correctly. This will, in turn, verify the exchange between the two parties and issue the correct payment (i.e. fair exchange). This trustless property is really important as we know that markets with no mediators, escrows and other assurance services (e.g. underground markets) are prone to scams, as the users have to put a lot of trust that the provider will execute on the agreement. In a trustless environment, once the transfer is done, there is no way to contest. From d37fb77aaa9ee20fe66513286d9e14fa3446d7da Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:53:44 +0200 Subject: [PATCH 14/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 7723650..ae2d786 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -84,7 +84,7 @@ In traditional or Web 2.0 setups (e.g. CDNs, Mirrors, Static Servers and so on), In a decentralized environment that strives to be permissionless, we can’t expect to rely on such legal agreements and third parties and, therefore, need a way to prove fair exchange. That is, we need to be able to prove that the service was delivered correctly and the metering of the provided service was done correctly. This will, in turn, verify the exchange between the two parties and issue the correct payment (i.e. fair exchange). -This trustless property is really important as we know that markets with no mediators, escrows and other assurance services (e.g. underground markets) are prone to scams, as the users have to put a lot of trust that the provider will execute on the agreement. In a trustless environment, once the transfer is done, there is no way to contest. +This trustless property is really important as we know that markets with no mediators, escrows, and other assurance services (e.g. underground markets) are prone to scams, as the users have to trust that the provider will execute on the agreement. In a trustless environment, once the transfer is completed, there is no way to dispute it. #### Properties desired From ed12f3712851f00159b7ee265fa498236164417f Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:54:25 +0200 Subject: [PATCH 15/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index ae2d786..a7a3283 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -91,7 +91,7 @@ This trustless property is really important as we know that markets with no medi * The exchanges of value MUST be verifiable and correct * Fairness: * The payment MUST be fulfilled if the SLA is fulfilled - * The payment for bandwidth/latency in token SHOULD match what has been agreed and fulfilled + * The payment for bandwidth/latency SHOULD match what was agreed and provided * Verifiability: * Both parties MUST be capable of verifying that the exchange was performed correctly * Bonus: Anyone SHOULD be able to verify that the exchange was performed correctly From 878df15451b095384e2261355181631f2f2a8712 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:54:48 +0200 Subject: [PATCH 16/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index a7a3283..82f2d7c 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -94,7 +94,7 @@ This trustless property is really important as we know that markets with no medi * The payment for bandwidth/latency SHOULD match what was agreed and provided * Verifiability: * Both parties MUST be capable of verifying that the exchange was performed correctly - * Bonus: Anyone SHOULD be able to verify that the exchange was performed correctly + * Bonus: Third parties SHOULD be able to verify that the exchange was performed correctly * SHOULD be Trustless * Ideally, the network can perform the operations without having to leverage Trust * Nevertheless in many designs it might be needed to leverage reputation as a trust vehicle to guarantee high quality of service and/or reliability. From 2792e7209119f7e670b0cd270e0345a1b16672ff Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:55:39 +0200 Subject: [PATCH 17/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 82f2d7c..a4fd2e8 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -114,7 +114,7 @@ In review, the Metering & Fair Exchange fields end up contributing to each other * Fairness * Making it Trustless * How to verify that the provider has the ability to delivery the file (e.g. stored locally or that can pull it from the network) - * How to verify that the file being transferred is the one requested, before a large amount of resources has been invested (i.e., before the entirety of the file has been transmitted) + * How to verify that the file being transferred is the one requested, before a large amount of resources has been invested (i.e. before the entirety of the file has been transmitted) * How to verify that the client has received the file * How to verify and reward that SLA were guaranteed? * How to avoid collusion when adding a third-party (e.g. Referee)? From bcc8f67c1f814231e389a5eee8bf70d173574a76 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:56:13 +0200 Subject: [PATCH 18/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index a4fd2e8..2614af7 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -113,7 +113,7 @@ In review, the Metering & Fair Exchange fields end up contributing to each other * Fairness * Making it Trustless - * How to verify that the provider has the ability to delivery the file (e.g. stored locally or that can pull it from the network) + * How to verify that the provider has the ability to delivery the file (e.g. has a local copy or is able to pull it from the network) * How to verify that the file being transferred is the one requested, before a large amount of resources has been invested (i.e. before the entirety of the file has been transmitted) * How to verify that the client has received the file * How to verify and reward that SLA were guaranteed? From 9cc3e1ff4a5ea54ae171141f30ec561fe6017c25 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:57:01 +0200 Subject: [PATCH 19/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 2614af7..b92cea3 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -116,7 +116,7 @@ In review, the Metering & Fair Exchange fields end up contributing to each other * How to verify that the provider has the ability to delivery the file (e.g. has a local copy or is able to pull it from the network) * How to verify that the file being transferred is the one requested, before a large amount of resources has been invested (i.e. before the entirety of the file has been transmitted) * How to verify that the client has received the file - * How to verify and reward that SLA were guaranteed? + * How to verify and reward that SLA were met? * How to avoid collusion when adding a third-party (e.g. Referee)? * How to avoid grieving attacks (make one party pay for fees)? * How to avoid a malicious actor causing un-rewarded work, hence wasting the others’ resources? From 3fc8ee880b5ec46a6877eb42d4995698f4f4621e Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:57:30 +0200 Subject: [PATCH 20/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index b92cea3..2833f5b 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -118,7 +118,7 @@ In review, the Metering & Fair Exchange fields end up contributing to each other * How to verify that the client has received the file * How to verify and reward that SLA were met? * How to avoid collusion when adding a third-party (e.g. Referee)? - * How to avoid grieving attacks (make one party pay for fees)? + * How to avoid griefing attacks (make one party pay for fees)? * How to avoid a malicious actor causing un-rewarded work, hence wasting the others’ resources? * Experience * How to make the transfers start instantaneously? From a1235195191897f2b733bbfa91f1ae91cfdfe88c Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:57:42 +0200 Subject: [PATCH 21/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 2833f5b..df37073 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -209,7 +209,7 @@ Here we list the attacks to consider when designing a solution. These are: * Type: client fraud * Consequence: bandwidth-spent * Malicious actor forces provider to pull the file from storage point (e.g. Filecoin, Cloud Storage, etc), paying the cost without the provider ever getting paid - * Type: client fraud: + * Type: client fraud * Consequence: currency is spent * Provider claims it has sent the file, but actually never did * Type: Provider fraud From d9d430114a957957a8b1623ce88eccafbf829c41 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:58:01 +0200 Subject: [PATCH 22/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index df37073..2fed433 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -228,7 +228,7 @@ ResNetLab organized a Research Intensive Workshop on 3DMs, out of which, the fol ## Distribution Graph Forming -The area of graph forming for Decentralised Data Delivery Markets revolves around three main areas, which simultaneously compose the different goals of the Graph Forming problem: +The area of graph forming for Decentralised Data Delivery Markets revolves around three main areas, which together compose the different goals of the Graph Forming problem: * **Content discovery & routing:** the system can forward requests to find content * **Content placement:** the system should have ways to proactively distribute/replicate copies of data across different Providers in different geographic regions in order to be able to serve content fast. From 2f08df3971a2e71e82b57f4c35074892f56af4ca Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:58:19 +0200 Subject: [PATCH 23/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 2fed433..efbe632 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -233,7 +233,7 @@ The area of graph forming for Decentralised Data Delivery Markets revolves aroun * **Content discovery & routing:** the system can forward requests to find content * **Content placement:** the system should have ways to proactively distribute/replicate copies of data across different Providers in different geographic regions in order to be able to serve content fast. * **Content copy selection:** the system can choose the optimal copy of the content to serve if multiple copies exist in several Providers, where potentially: - * every copy has a different “asking” price (defined by the cryptoeconomic model) and latency to deliver the content item, and + * every copy has a different “asking” price (defined by the cryptoeconomic model) and delivery latency, and * every miner achieves different performance and has a different reputation profile. In order to achieve the above goals the system needs to have an architectural structure, that is, where do end-users connect (e.g., to a retrieval miner, or to a separate content resolution system), which entity in the architecture makes content resolution and request forwarding decisions and finally, which entity(ies) has(ve) the required knowledge to forward requests to the closest (to the requesting user) copy, _aka_ nearest replica routing. From fb104823612b653b08426bcd5f62ec57fe2df4c6 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:59:14 +0200 Subject: [PATCH 24/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index efbe632..dda948e 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -236,7 +236,7 @@ The area of graph forming for Decentralised Data Delivery Markets revolves aroun * every copy has a different “asking” price (defined by the cryptoeconomic model) and delivery latency, and * every miner achieves different performance and has a different reputation profile. -In order to achieve the above goals the system needs to have an architectural structure, that is, where do end-users connect (e.g., to a retrieval miner, or to a separate content resolution system), which entity in the architecture makes content resolution and request forwarding decisions and finally, which entity(ies) has(ve) the required knowledge to forward requests to the closest (to the requesting user) copy, _aka_ nearest replica routing. +In order to achieve the above goals, we need to answer certain architectural questions: where do end-users connect (e.g., to a retrieval miner, or to a separate content resolution system), which entity in the architecture makes content resolution and request forwarding decisions and finally, which entity(ies) has(ve) the required knowledge to forward requests to the closest (to the requesting user) copy, _aka_ nearest replica routing. We generally consider that retrieval miners are relatively powerful devices with local storage dedicated to storing hot copies of the content, high uptime (close to zero churn) and ideally public IP (i.e., reachable from anywhere). The architecture should ideally be able to take advantage of storage in less-powerful and more ephemerally-connected end-user devices, such as laptops and mobile phones - this is discussed as a different area further down. Although not a strict requirement, this is what will make the decentralised storage network take the full benefit of planetary-scale unused storage. From 02a9a3f87d94a5d15fc4fdbdfe8fc00d287be27b Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:59:29 +0200 Subject: [PATCH 25/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index dda948e..7d292e3 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -238,7 +238,7 @@ The area of graph forming for Decentralised Data Delivery Markets revolves aroun In order to achieve the above goals, we need to answer certain architectural questions: where do end-users connect (e.g., to a retrieval miner, or to a separate content resolution system), which entity in the architecture makes content resolution and request forwarding decisions and finally, which entity(ies) has(ve) the required knowledge to forward requests to the closest (to the requesting user) copy, _aka_ nearest replica routing. -We generally consider that retrieval miners are relatively powerful devices with local storage dedicated to storing hot copies of the content, high uptime (close to zero churn) and ideally public IP (i.e., reachable from anywhere). The architecture should ideally be able to take advantage of storage in less-powerful and more ephemerally-connected end-user devices, such as laptops and mobile phones - this is discussed as a different area further down. Although not a strict requirement, this is what will make the decentralised storage network take the full benefit of planetary-scale unused storage. +We generally consider that retrieval miners are relatively powerful devices with local storage dedicated to storing hot copies of the content, high uptime (close to zero churn) and, ideally, a public IP address (reachable from anywhere). The architecture should be able to take advantage of storage in less powerful and intermittently-connected end-user devices, such as laptops and mobile phones - this is discussed as a different area further down. Although not a strict requirement, this is what will make the decentralised storage network take full benefit of planetary-scale unused storage. It is worth highlighting that the system should be able to serve different use-cases and therefore different setups and architectures could apply depending on the use-case. For instance, the content resolution system can be different between the cases where: i) the application operates based on a closed-content ecosystem (e.g., subscriber-based music or video streaming, where control of the content is solely with the content publisher) and ii) the application operates on a totally open content space, e.g., web. From f6985b1af7c2376acd8f8751cb00be9ad8ccdfcc Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:59:51 +0200 Subject: [PATCH 26/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 7d292e3..4ea0b41 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -323,7 +323,7 @@ In pubsub systems information propagates in the network based on topics that nod * Providers serve bogus content * Providers provide false provider records (i.e., they claim that they have and can serve content that they actually do not have) * Cache poisoning -* Privacy attacks: there is a bunch of privacy-related to be taken into account +* Privacy attacks: there are a number of privacy considerations and threat vectors to be taken into account ### New ideas being explored From 0daa66b9e62eb57be2f992a04b75937723a5a900 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 12:00:04 +0200 Subject: [PATCH 27/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 4ea0b41..a94d847 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -327,7 +327,7 @@ In pubsub systems information propagates in the network based on topics that nod ### New ideas being explored -ResNetLab organized a Research Intensive Workshop on 3DMs, out of it, the following ideas, structured as RFCs emerged: +ResNetLab organized a Research Intensive Workshop on 3DMs, from which the following ideas emerged and were structured as RFCs: * Link to RFCs TO FILL From a4c09a87b2e36770e942ce616496b71afbb75851 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 12:07:10 +0200 Subject: [PATCH 28/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index a94d847..c08cb21 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -339,7 +339,7 @@ The aim of the cryptoeconomic model is to build an incentive system to ensure th In traditional and centralized setups, there is no underlying need to align the incentives of all the participants in a data delivery network. The relationships between entities in the system are governed by a set of multilateral legal agreements between the different parties. These agreements offer the basic level of trust to ensure the good behavior of all entities in the system (clearly stating the SLA, and punishments for misbehaving or not fulfilling the contract). We currently see this setup in the multilateral agreements between CDNs, ISPs, and Cloud Providers to share resources from their infrastructures for a certain price in order to build their network of relationships that allow them to provide their content delivery services efficiently. -In decentralized setups, it is not possible to use legal agreements to orchestrate the relationship between entities in the system. In recent deployments of hybrid CDNs, traditional, centralised CDNs leverage the resources of end users to deliver data with enhanced QoS reducing the load on their infrastructure and as a result the cost of upgrading/extending their infrastructure. End users are incentivized to contribute to the system with their upload bandwidth by making them eligible for enjoying a better service. This additional reward is used to incentivize a behavior in the system (in this case users contributing with bandwidth in hybrid CDNs). However, the fact that there is still a centralized infrastructure managed by an entity with a greater power over the rest of actors in the system is enough to prevent misbehaviors. This removes the need to design stronger reputation or reward systems to prevent misbehaviors. This does not represent a trustless setup. +In decentralized setups, it is not possible to use legal agreements to orchestrate the relationship between entities in the system. In recent deployments of hybrid CDNs, traditional, centralised CDNs leverage the resources of end users to deliver data with enhanced QoS reducing the load on their infrastructure and as a result the cost of upgrading/extending their infrastructure. End users are incentivized to contribute to the system with their upload bandwidth by making them eligible for better service. This additional reward is used to incentivize a behavior in the system (in this case users contributing with bandwidth in hybrid CDNs). However, the fact that there is still a centralized infrastructure managed by an entity with greater power over the rest of actors is enough to prevent misbehavior. This removes the need to design stronger reputation or reward systems. This does not represent a trustless setup. For decentralized and trustless setups, two main schemes have been traditionally used to align the incentive of the system’s participants and prevent misbehaviors: From d9c1fc6d4f53d2cf67458b468c5282fdb4876699 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 12:46:36 +0200 Subject: [PATCH 29/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index c08cb21..90a65d3 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -468,7 +468,7 @@ The literature in the field of Opportunistic and Delay-Tolerant Networks is vast #### Transient Providers -We consider end-users that store content in their ephemerally connected devices (laptops, or mobile phones) and provide them to the network through (one or more) Providers, as defined in the “Distribution Graph Forming” area. These service providers are called “Transient Providers (TRMs) to depict their ephemeral nature in resource availability and time. TRMs can have some economic relationship with one or more RMs, e.g., RMs can “recruit” TRMs to expand their storage capacity and footprint. In this case, RMs have to maintain their own monitoring mechanisms and do resource allocation to ensure that they are getting desirable levels of service from their group of TRMs. +We consider end-users that store content in their ephemerally connected devices (laptops, or mobile phones) and provide them to the network through (one or more) Providers, as defined in the “Distribution Graph Forming” area. These service providers are called Transient Providers (TRMs) to depict their ephemeral nature in resource availability and time. TRMs can have some economic relationship with one or more RMs, e.g., RMs can “recruit” TRMs to expand their storage capacity and footprint. In this case, RMs have to maintain their own monitoring mechanisms and do resource allocation to ensure that they are getting desirable levels of service from their group of TRMs. Opportunistic D2D From a23e4cfb8d2cdf1ccb56ad9605d384c0dbd18d4d Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 12:59:23 +0200 Subject: [PATCH 30/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 90a65d3..1f4c1e0 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -250,7 +250,7 @@ Traditionally, content is stored, hosted, replicated and delivered by centralise In a centralized setup it is much easier to optimise performance and provide guarantees, compared to the case where **every network node is operated by a separate (legal) entity that operates rationally in a profit-maximizing way**. -Closer to our setup are P2P CDNs, where end-user equipment is “employed” to contribute to the storage and dissemination of content. Node churn, unreliable connectivity and low bandwidth speeds become a reality in P2P CDNs, but **again the setup of P2P CDNs is controlled and monitored by a single entity/company.** This makes decisions with regard to content placement and replication easier. The most important advantage of a single entity taking care of the P2P CDN setup is **monitoring**, that is, being able to observe the performance of peers and make decisions on where and when to replicate content or assign content distribution decisions. +Closer to our setup are P2P CDNs, where end-user equipment is “employed” to contribute to the storage and dissemination of content. Node churn, unreliable connectivity and low bandwidth speeds become a reality in P2P CDNs, but **the operation of a P2P CDN is still largely controlled and monitored by a single entity**. This makes decisions with regard to content placement and replication easier. The most important advantage of a single entity taking care of the P2P CDN setup is **monitoring**, that is, being able to observe the performance of peers and make decisions on where and when to replicate content or assign content distribution decisions. Properties From 8dce8343f5e0311e531862fb88fc99311d3bf8d0 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:00:27 +0200 Subject: [PATCH 31/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 1f4c1e0..c847731 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -255,7 +255,7 @@ Closer to our setup are P2P CDNs, where end-user equipment is “employed” to Properties * The system MUST always be able to discover content and satisfy content requests - * There should never be a “404 Content not Found” error. + * There should never be a “404 Content not Found” error for content available anywhere on the network * The system MUST replicate content to different storage points in order to reduce delivery times and maximize performance. * Providers MUST follow the crypto-economic model and the system MUST make sure that Providers do not misbehave. * The network MUST be a content-addressable network and operate based on content identifiers. From 665b4e3430b55a82057bbaa81e484ef426de301a Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:00:45 +0200 Subject: [PATCH 32/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index c847731..e85ecb1 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -258,7 +258,7 @@ Properties * There should never be a “404 Content not Found” error for content available anywhere on the network * The system MUST replicate content to different storage points in order to reduce delivery times and maximize performance. * Providers MUST follow the crypto-economic model and the system MUST make sure that Providers do not misbehave. -* The network MUST be a content-addressable network and operate based on content identifiers. +* The network MUST be content-addressable and operate based on content identifiers. * The system MUST be permissionless and trustless * Anyone should be free to join and set up a retrieval miner to contribute to the network. From 5cb8493152b1fe57b766a8a9a28bcea30358e7c2 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:01:05 +0200 Subject: [PATCH 33/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index e85ecb1..acddf35 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -291,7 +291,7 @@ Our initial, but thorough investigation resulted in the following potential grou #### DHT-based: -DHTs are very popular constructions in P2P networks. The DHT system is used as a content resolution, as well as a content and peer routing system. The routing table is split between peers that participate in the DHT, providing higher resilience and scalability. In a DHT-based system, clients “walk” the DHT (iteratively, or recursively) to find the provider record and then directly connect to the peer included in the provider record. +DHTs are very popular constructions in P2P networks. The DHT system is used as a content resolution mechanism, as well as a content and peer routing system. The routing table is split between peers that participate in the DHT, providing higher resilience and scalability. In a DHT-based system, clients “walk” the DHT (iteratively, or recursively) to find the provider record and then directly connect to the peer included in the provider record. * **Pros:** tested in the past, engineers have lots of experience with these structures, can become faster and less expensive than the IPFS DHT setup, if: i) re-publishing is done at coarser granularities, which is reasonable if we assume stable connectivity, i.e., low peer churn (reasonable to assume for Providers), and ii) some payment is associated with publishing. It is also reasonable to assume public IP connectivity for Providers. * **Cons:** slow, several round-trips needed to resolve content (especially in case of iterative DHT lookup). The solution will not scale in the longer term. From 1a29e74a5af24bc53c92046ee1d1dafc25c2ed21 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:01:31 +0200 Subject: [PATCH 34/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index acddf35..3606f5e 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -298,7 +298,7 @@ DHTs are very popular constructions in P2P networks. The DHT system is used as a #### Name-based routing: -In Name-based routing systems, routing hints are integrated as part of the content name; routing tables are filled with routing hints at network setup time by a routing protocol. Routers make hop-by-hop forwarding decisions based on content names seen in requests and routing hints in their routing tables.. Matching between hints and names depend on the structure of the names, e.g., hierarchical vs flat. +In Name-based routing systems, routing hints are integrated as part of the content name; routing tables are filled with routing hints at network setup time by a routing protocol. Routers make hop-by-hop forwarding decisions based on content names seen in requests and routing hints in their routing tables. Matching between hints and names depend on the structure of the names, e.g., hierarchical vs flat. * **Pros:** very fast, enables multicast-like opportunities and on-path caching, which can result in significant savings in case of popular and heavy content (e.g., HD VoD). Also makes routing from browser/limited connectivity clients more feasible. * **Cons:** design can be(come) complex, security properties not fully studied in the literature. From 2840013b1537f09ebcfeb9182a395d2894eee41c Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:06:31 +0200 Subject: [PATCH 35/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 3606f5e..9e10886 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -119,7 +119,7 @@ In review, the Metering & Fair Exchange fields end up contributing to each other * How to verify and reward that SLA were met? * How to avoid collusion when adding a third-party (e.g. Referee)? * How to avoid griefing attacks (make one party pay for fees)? - * How to avoid a malicious actor causing un-rewarded work, hence wasting the others’ resources? + * How to avoid a malicious actor causing un-rewarded work, hence wasting others’ resources? * Experience * How to make the transfers start instantaneously? * How to support others (e.g. Content Publishers) paying for the usage? From afe361530814476e6f65bc2ac22346aa2ac6fd51 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:07:01 +0200 Subject: [PATCH 36/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 9e10886..bb49bc8 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -123,7 +123,7 @@ In review, the Metering & Fair Exchange fields end up contributing to each other * Experience * How to make the transfers start instantaneously? * How to support others (e.g. Content Publishers) paying for the usage? - * How to make it private (e.g. that others don’t know who is requesting what)? + * How to make it private (i.e. so that others don’t know which users are requesting what content)? * Performance * How to overcome the send-and-halt in order to max bandwidth throughput * How to support multipath (i.e. fetching from multiple sources) From 8fcb7a2229dcc99f064a6ca3ea476a383c492f18 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:09:08 +0200 Subject: [PATCH 37/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index bb49bc8..94f20c7 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -125,7 +125,7 @@ In review, the Metering & Fair Exchange fields end up contributing to each other * How to support others (e.g. Content Publishers) paying for the usage? * How to make it private (i.e. so that others don’t know which users are requesting what content)? * Performance - * How to overcome the send-and-halt in order to max bandwidth throughput + * How to overcome send-and-halt in order to maximize bandwidth throughput * How to support multipath (i.e. fetching from multiple sources) ### State-of-the-art From c4614798d68442debe386fa642859d5d639a7039 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:10:07 +0200 Subject: [PATCH 38/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 94f20c7..0699659 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -142,7 +142,7 @@ These type of solutions have been found/highlighted: - [Theta Whitepaper](https://s3.us-east-2.amazonaws.com/assets.thetatoken.org/Theta-white-paper-latest.pdf?v=1612955013.791) **Known shortcomings of these approaches:** -* send-and-halt - Because the next packet (unit of service) will only be issued after the payment of the previous, these solutions often miss to max out the bandwidth available in the connection available, increasing the latency of delivery. +* send-and-halt - Because the next packet (unit of service) will only be issued after payment for the previous is received, these solutions often fail to max out the bandwidth available in the connection, increasing the latency of delivery. #### Lock/Unlock access to the resource From a8627a0eb38b05bee542d8a14401dee040eb4708 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:10:37 +0200 Subject: [PATCH 39/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 0699659..27efc37 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -147,7 +147,7 @@ These type of solutions have been found/highlighted: #### Lock/Unlock access to the resource -These type of solutions focus give the possibility for the client and the provider to verify the actual service before unlocking the payment. One of the main tools used is locking the access to the service by encrypting it with a key and then performing an exchange over the key that gives access to the service. +Solutions of this type enable the client and the provider to verify the actual service before unlocking the payment. One of the main tools used is locking the access to the service by encrypting it with a key and then performing an exchange of the key that gives access to the service. These solutions can be found in several forms: - [Solving the Buyer and Seller’s Dilemma: A Dual-Deposit Escrow Smart Contract for Provably Cheat-Proof Delivery and Payment for a Digital Good without a Trusted Mediator](INSERT URL) From 01fc84c1cbf282eb42f2cb85cf3d7085aa533fc2 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:11:04 +0200 Subject: [PATCH 40/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 27efc37..16358c9 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -155,7 +155,7 @@ These solutions can be found in several forms: - [Zero-Knowledge Contingent Payments Revisited: Attacks and Payments for Services](INSERT URL) **Known shortcomings of these approaches:** -* Vulnerable to Grieving attacks. +* Vulnerable to griefing attacks. * Generally, these solutions focus on the fulfilment of the delivery, more so than measuring the speed and rate of the transfer. From 23032162ba934567753f5d14c59d75168fc8bbf4 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:11:33 +0200 Subject: [PATCH 41/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 16358c9..ee5da7c 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -168,7 +168,7 @@ These solutions build on top of the Fair Exchange ones and adopt an Optimistic a - [Optimistic Fair Exchange with Multiple Arbiters](INSERT URL) **Known shortcomings of these approaches:** -* Setup + Dispute resolution time is non insignificant and non-trivial +* Setup + Dispute resolution is non-trivial and takes significant time #### Reputation based From 06559a86e931ff254334f20ec250cdc9e78ced58 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:11:55 +0200 Subject: [PATCH 42/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index ee5da7c..e548996 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -172,7 +172,7 @@ These solutions build on top of the Fair Exchange ones and adopt an Optimistic a #### Reputation based -Reputation based solutions give the possibility to speed up the transfer and improve the quality of the service, by leveraging the trust built over time between clients & providers. +Reputation-based solutions give the possibility to speed up the transfer and improve the quality of the service by leveraging the trust built over time between clients & providers. - [Proof of Delivery in a Trustless Network](INSERT URL) - [Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks](INSERT URL) From 9206c1bd351bc43a03ce26ea3ed9ddd8a1d45f68 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:12:22 +0200 Subject: [PATCH 43/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index e548996..3cad32f 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -184,7 +184,7 @@ Reputation-based solutions give the possibility to speed up the transfer and imp #### Privacy focused -There are multiple solutions with the goal of delivering readers privacy and writers privacy. These incur an additional cost in setup and/or latency in favor of the additional privacy preserving benefits. Disclaimer: At the time of writing, we haven't dived as deep into these set of solutions. +There are multiple solutions with the goal of delivering reader privacy and writer privacy. These incur an additional cost in setup and/or latency. Disclaimer: At the time of writing, we haven't dived deep into this set of solutions. - [Blockchain based Privacy-Preserving Software Updates with Proof-of-Delivery for Internet of Things](INSERT URL) - [SilentDelivery: Practical Timed-delivery of Private Information using Smart Contracts](INSERT URL) From 6d5be4cc87fcd92ad38295641fc7286358bf2772 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:13:42 +0200 Subject: [PATCH 44/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 3cad32f..15e238d 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -205,7 +205,7 @@ These approaches suggest novel ways to add integrity checks to data transferred Here we list the attacks to consider when designing a solution. These are: -* Malicious actor forces provider to spend bandwidth without issuing payment (also known as Grieving Attack) +* Malicious actor forces provider to spend bandwidth without issuing payment (also known as Griefing Attack) * Type: client fraud * Consequence: bandwidth-spent * Malicious actor forces provider to pull the file from storage point (e.g. Filecoin, Cloud Storage, etc), paying the cost without the provider ever getting paid From f9a248ddf2a5a10e0c9205051661bdaa1d818ef2 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:13:57 +0200 Subject: [PATCH 45/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 15e238d..ccee67f 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -207,7 +207,7 @@ Here we list the attacks to consider when designing a solution. These are: * Malicious actor forces provider to spend bandwidth without issuing payment (also known as Griefing Attack) * Type: client fraud - * Consequence: bandwidth-spent + * Consequence: bandwidth is spent * Malicious actor forces provider to pull the file from storage point (e.g. Filecoin, Cloud Storage, etc), paying the cost without the provider ever getting paid * Type: client fraud * Consequence: currency is spent From 0262cdeb30761872026235a30d7b923258c41da5 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 13:14:15 +0200 Subject: [PATCH 46/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index ccee67f..367ae9b 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -208,7 +208,7 @@ Here we list the attacks to consider when designing a solution. These are: * Malicious actor forces provider to spend bandwidth without issuing payment (also known as Griefing Attack) * Type: client fraud * Consequence: bandwidth is spent -* Malicious actor forces provider to pull the file from storage point (e.g. Filecoin, Cloud Storage, etc), paying the cost without the provider ever getting paid +* Malicious actor forces provider to pay to retrieve the file from storage point (e.g. Filecoin, Cloud Storage, etc) without the provider itself ever getting paid * Type: client fraud * Consequence: currency is spent * Provider claims it has sent the file, but actually never did From 7757b9e49fa04da8cfc66f4b904481e9d92aa450 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 11:36:17 +0000 Subject: [PATCH 47/66] Metering section links to papers Adds links to all papers referenced in the metering and fair exchange section. --- .../DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 367ae9b..a55c82d 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -150,9 +150,9 @@ These type of solutions have been found/highlighted: Solutions of this type enable the client and the provider to verify the actual service before unlocking the payment. One of the main tools used is locking the access to the service by encrypting it with a key and then performing an exchange of the key that gives access to the service. These solutions can be found in several forms: -- [Solving the Buyer and Seller’s Dilemma: A Dual-Deposit Escrow Smart Contract for Provably Cheat-Proof Delivery and Payment for a Digital Good without a Trusted Mediator](INSERT URL) -- [FairSwap: How to fairly exchange digital goods](INSERT URL) -- [Zero-Knowledge Contingent Payments Revisited: Attacks and Payments for Services](INSERT URL) +- [Solving the Buyer and Seller’s Dilemma: A Dual-Deposit Escrow Smart Contract for Provably Cheat-Proof Delivery and Payment for a Digital Good without a Trusted Mediator](http://anrg.usc.edu/www/papers/Dual_Deposit_ICBC_2019.pdf) +- [FairSwap: How to fairly exchange digital goods](https://eprint.iacr.org/2018/740.pdf) +- [Zero-Knowledge Contingent Payments Revisited: Attacks and Payments for Services](https://acmccs.github.io/papers/p229-campanelliA.pdf) **Known shortcomings of these approaches:** * Vulnerable to griefing attacks. @@ -163,9 +163,9 @@ These solutions can be found in several forms: These solutions build on top of the Fair Exchange ones and adopt an Optimistic approach where some sort of Reputation or Stake is used to give both parties trust that the exchange will occur correctly. If one misbehaves, they can still rely on a dispute system. -- [OptiSwap: Fast Optimistic Fair Exchange](INSERT URL) -- [Asynchronous Protocols for Optimistic Fair Exchange](INSERT URL) -- [Optimistic Fair Exchange with Multiple Arbiters](INSERT URL) +- [OptiSwap: Fast Optimistic Fair Exchange](https://dl.acm.org/doi/10.1145/3320269.3384749) +- [Asynchronous Protocols for Optimistic Fair Exchange](https://ieeexplore.ieee.org/abstract/document/674826?casa_token=K7FXNFUKD7AAAAAA:cAcsxIQvxTqvUpRZ73PYBtG2lGtyrA0qMAEuBa46q4Zas4d3aD7yATmZhxYrhem0RxKGwlqn4g) +- [Optimistic Fair Exchange with Multiple Arbiters](https://eprint.iacr.org/2009/069.pdf) **Known shortcomings of these approaches:** * Setup + Dispute resolution is non-trivial and takes significant time @@ -174,9 +174,9 @@ These solutions build on top of the Fair Exchange ones and adopt an Optimistic a Reputation-based solutions give the possibility to speed up the transfer and improve the quality of the service by leveraging the trust built over time between clients & providers. -- [Proof of Delivery in a Trustless Network](INSERT URL) -- [Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks](INSERT URL) -- [Mechanisms for Outsourcing Computation via a Decentralized Market](INSERT URL) +- [Proof of Delivery in a Trustless Network](https://ieeexplore.ieee.org/document/8751417) +- [Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks](https://www.ee.ucl.ac.uk/~ipsaras/files/Proof_of_Prestige-icbc19.pdf) +- [Mechanisms for Outsourcing Computation via a Decentralized Market](https://scopelab.ai/files/eisele2020mechanisms.pdf) - Other Gradual Fair Exchange Class of constructions **Known shortcomings of these approaches:** @@ -186,17 +186,17 @@ Reputation-based solutions give the possibility to speed up the transfer and imp There are multiple solutions with the goal of delivering reader privacy and writer privacy. These incur an additional cost in setup and/or latency. Disclaimer: At the time of writing, we haven't dived deep into this set of solutions. -- [Blockchain based Privacy-Preserving Software Updates with Proof-of-Delivery for Internet of Things](INSERT URL) -- [SilentDelivery: Practical Timed-delivery of Private Information using Smart Contracts](INSERT URL) -- [Bulletproofs+: Shorter Proofs for Privacy-Enhanced Distributed Ledger](INSERT URL) +- [Blockchain based Privacy-Preserving Software Updates with Proof-of-Delivery for Internet of Things](https://www.sciencedirect.com/science/article/abs/pii/S074373151930098X) +- [SilentDelivery: Practical Timed-delivery of Private Information using Smart Contracts](https://arxiv.org/abs/1912.07824) +- [Bulletproofs+: Shorter Proofs for Privacy-Enhanced Distributed Ledger](https://eprint.iacr.org/2020/735.pdf) #### Traditional approaches (close to Web 2.0 world) These approaches suggest novel ways to add integrity checks to data transferred or to promote new ways for content distribution, but they don’t depart from the Web 2.0 world of assumptions such as trusted third parties and central points of control. -- [The multimedia blockchain: a distributed and tamper-proof media transaction framework](INSERT URL) -- [Reliable Client Accounting for P2P-Infrastructure Hybrids](INSERT URL) +- [The multimedia blockchain: a distributed and tamper-proof media transaction framework](http://www.cs.stir.ac.uk/~dbh/downloads/Multimedia-Blockchain-DSP17.pdf) +- [Reliable Client Accounting for P2P-Infrastructure Hybrids](https://www.cis.upenn.edu/~ahae/papers/accounting-nsdi2012.pdf) **Known shortcomings of these approaches:** * Not fully aware of the challenges of building a decentralized network From 16cbf67207e19aeabc63afaf973996e6a874fae7 Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 12:06:43 +0000 Subject: [PATCH 48/66] Graph Forming links to papers added --- .../DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index a55c82d..2404775 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -269,23 +269,23 @@ The target area and expected outcome of this topic is to form the network, the i **P2P CDNs:** the network architecture setup in these systems is similar in some ways to that of a 3DM -* LiveSky: Enhancing CDN with P2P -* P2P as a CDN: A new service model for file sharing (2012) -* OblivP2P: An Oblivious Peer-to-Peer Content Sharing System +* [LiveSky: Enhancing CDN with P2P](https://dl.acm.org/doi/pdf/10.1145/1823746.1823750) +* [P2P as a CDN: A new service model for file sharing](https://www.sciencedirect.com/science/article/pii/S1389128612002290?casa_token=uNw-gpbx5sIAAAAA:VYBa5QErOCS7DcduOwf1lsl3C174YlOIMmQii9NhKippX_Hm7I3QYnfeZ53LTVcp7Uj8y1JX4DI) +* [OblivP2P: An Oblivious Peer-to-Peer Content Sharing System](https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_jia.pdf) **P2P VoD:** video delivery is an ideal target use-case, with very strict requirements; therefore, the mechanisms proposed in this field and used for P2P VoD can be inspiring -* Challenges, Design and Analysis of a Large-scale P2P-VoD System -* A Framework for Lazy Replication in P2P VoD -* InstantLeap: Fast Neighbor Discovery in P2P VoD Streaming (2008) -* Proactive Video Push for Optimizing Bandwidth Consumption in Hybrid CDN-P2P VoD Systems +* [Challenges, Design and Analysis of a Large-scale P2P-VoD System](https://dl.acm.org/doi/pdf/10.1145/1402946.1403001) +* [A Framework for Lazy Replication in P2P VoD](https://dl.acm.org/doi/pdf/10.1145/1496046.1496068) +* [InstantLeap: Fast Neighbor Discovery in P2P VoD Streaming](https://dl.acm.org/doi/pdf/10.1145/1542245.1542251) +* [Proactive Video Push for Optimizing Bandwidth Consumption in Hybrid CDN-P2P VoD Systems](https://ieeexplore.ieee.org/abstract/document/8485962?casa_token=41CKY7irz4kAAAAA:V2cwdpnSCny5GB2GGn0Zj1-3-BefgmpJTs0qTPZiAdzMINYIJryr7tU2L5ue8S3kVcRRdyfOtg) **Content-Centric Networking:** there are several network architectures and protocols proposed for content-/information-centric networks which can be great inspiration for a permissionless, content-addressable P2P network. -* Named Data Networking (NDN) -* iCDN - An NDN based CDN -* Analysis and Improvement of Name-based Packet Forwarding over Flat ID Network Architectures (2018) -* High Throughput Forwarding for ICN with Descriptors and Locators (2016) +* [Named Data Networking (NDN)](https://dl.acm.org/doi/pdf/10.1145/2656877.2656887) +* [iCDN - An NDN based CDN](https://dl.acm.org/doi/pdf/10.1145/3405656.3418716) +* [Analysis and Improvement of Name-based Packet Forwarding over Flat ID Network Architectures](https://dl.acm.org/doi/pdf/10.1145/3267955.3267960) +* [High Throughput Forwarding for ICN with Descriptors and Locators](https://dl.acm.org/doi/pdf/10.1145/2881025.2881032) Our initial, but thorough investigation resulted in the following potential groups of designs for content discovery and resolution: From 80a8e86bf60115ec0d971732c6bb56202eebb16b Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 12:19:04 +0000 Subject: [PATCH 49/66] Cryptoeconomics section links to papers added Cryptoeconomics section links to papers added --- .../DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 44 +++++++++---------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 2404775..4d2e93a 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -374,11 +374,11 @@ The economics of current Hybrid CDN deployments give a good understanding of the **_Related Papers:_** -* An economic mechanism for request routing and resource allocation in hybrid CDN–P2P networks -* An economic replica placement mechanism for streaming content distribution in Hybrid CDN-P2P networks -* How Neutral is a CDN? An Economic Approach -* Value Networks and Two-Sided Markets of Internet Content Delivery -* Bilateral and Multilateral Exchanges for Peer-Assisted Content Distribution +* [An economic mechanism for request routing and resource allocation in hybrid CDN–P2P networks](http://www.cloudbus.org/papers/CDN-P2P-NetMan.pdf) +* [An economic replica placement mechanism for streaming content distribution in Hybrid CDN-P2P networks](https://www.sciencedirect.com/science/article/pii/S014036641400228X?casa_token=99HEhHX4Qx4AAAAA:HGUBKkgzqtRn-AwdIpjQmxFMmUd6YIwuWH4ESn3MC1Wv1xYjI5JemUOtJlPDpI20CqFwwlYbVqM) +* [How Neutral is a CDN? An Economic Approach](https://files.ifi.uzh.ch/stiller/CNSM%202014/pdf/54.pdf) +* [Value Networks and Two-Sided Markets of Internet Content Delivery](http://www.leva.fi/wp-content/uploads/ICN_2SM_TelecomPolicy_accepted_manuscript.pdf) +* [Bilateral and Multilateral Exchanges for Peer-Assisted Content Distribution](https://www.cs.princeton.edu/~mfreed/docs/bilateral-tr10.pdf) **_Learnings: _** @@ -391,12 +391,12 @@ How to design the right incentive and reputation models to incentivize certain b **Related Papers:** -* Scrivener: Providing Incentives in Cooperative Content Distribution Systems -* Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks -* Incentivizing Peer-Assisted Services:A Fluid Shapley Value Approach -* Incentive and Service Differentiation in P2P Networks: A Game Theoretic Approach -* On Incentivizing Caching for P2P-VoD Systems -* Bar Gossip +* [Scrivener: Providing Incentives in Cooperative Content Distribution Systems](http://cs.brown.edu/courses/csci2950-g/papers/scrivener.pdf) +* [Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks](https://www.ee.ucl.ac.uk/~ipsaras/files/Proof_of_Prestige-icbc19.pdf) +* [Incentivizing Peer-Assisted Services: A Fluid Shapley Value Approach](https://dl.acm.org/doi/pdf/10.1145/1811099.1811064) +* [Incentive and Service Differentiation in P2P Networks: A Game Theoretic Approach](http://www.cse.cuhk.edu.hk/~cslui/PUBLICATION/incentive_tons.pdf) +* [On Incentivizing Caching for P2P-VoD Systems](http://www.cse.cuhk.edu.hk/~cslui/PUBLICATION/netecon12.pdf) +* [Bar Gossip](https://static.usenix.org/event/osdi06/tech/full_papers/li/li.pdf) **_Learnings: _** @@ -410,12 +410,12 @@ Credit networks provide liquidity in markets where transaction volumes are close **_Related Papers:_** -* Collusion-resilient Credit-based Reputations for Peer-to-peer Content Distribution -* Liquidity in Credit Networks: A Little Trust Goes a Long Way -* Liquidity in Credit Networks with Constrained Agents -* CAPnet: A Defense Against Cache Accounting Attacks on Content Distribution Networks -* MicroCash: Practical Concurrent Processing of Micropayments -* Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks +* [Collusion-resilient Credit-based Reputations for Peer-to-peer Content Distribution](https://dl.acm.org/doi/pdf/10.1145/1879082.1879085) +* [Liquidity in Credit Networks: A Little Trust Goes a Long Way](https://dl.acm.org/doi/pdf/10.1145/1993574.1993597) +* [Liquidity in Credit Networks with Constrained Agents](https://dl.acm.org/doi/pdf/10.1145/3366423.3380276) +* [CAPnet: A Defense Against Cache Accounting Attacks on Content Distribution Networks](https://arxiv.org/abs/1906.10272) +* [MicroCash: Practical Concurrent Processing of Micropayments](https://arxiv.org/pdf/1911.08520.pdf) +* [Proof-of-Prestige: A Useful Work Reward System for Unverifiable Tasks](https://www.ee.ucl.ac.uk/~ipsaras/files/Proof_of_Prestige-icbc19.pdf) **_Learnings: _** @@ -429,11 +429,11 @@ Auctions and decentralized markets have traditionally been a good way of allocat **_Related Papers:_** -* Edge-MAP: Auction Markets for Edge Resource Provisioning -* A Market Protocol for Decentralized Task Allocation -* On the Efficiency of Sharing Economy Networks -* Resource Allocation in Market-based Grids Using a History-based Pricing Mechanism -* Content Pricing in Peer-to-Peer Networks +* [Edge-MAP: Auction Markets for Edge Resource Provisioning](https://www.ee.ucl.ac.uk/~ipsaras/files/edgeMap-wowmom18.pdf) +* [A Market Protocol for Decentralized Task Allocation](https://www.computer.org/csdl/proceedings-article/icmas/1998/85000325/12OmNwlZu42) +* [On the Efficiency of Sharing Economy Networks](https://ieeexplore.ieee.org/document/8665937) +* [Resource Allocation in Market-based Grids Using a History-based Pricing Mechanism](http://ce-publications.et.tudelft.nl/publications/646_resource_allocation_in_marketbased_grids_using_a_historyba.pdf) +* [Content Pricing in Peer-to-Peer Networks](https://www.usenix.org/legacy/event/netecon10/tech/full_papers/Park.pdf) **_Learnings: _** From 5111b5a24061f0bc0702f673c8e66149c7f59a1c Mon Sep 17 00:00:00 2001 From: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> Date: Thu, 11 Feb 2021 12:30:07 +0000 Subject: [PATCH 50/66] Replaced term Retrieval Miner and RM to Provider --- .../DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 4d2e93a..af447e5 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -236,13 +236,13 @@ The area of graph forming for Decentralised Data Delivery Markets revolves aroun * every copy has a different “asking” price (defined by the cryptoeconomic model) and delivery latency, and * every miner achieves different performance and has a different reputation profile. -In order to achieve the above goals, we need to answer certain architectural questions: where do end-users connect (e.g., to a retrieval miner, or to a separate content resolution system), which entity in the architecture makes content resolution and request forwarding decisions and finally, which entity(ies) has(ve) the required knowledge to forward requests to the closest (to the requesting user) copy, _aka_ nearest replica routing. +In order to achieve the above goals, we need to answer certain architectural questions: where do end-users connect (e.g., to a Provider, or to a separate content resolution system), which entity in the architecture makes content resolution and request forwarding decisions and finally, which entity(ies) has(ve) the required knowledge to forward requests to the closest (to the requesting user) copy, _aka_ nearest replica routing. -We generally consider that retrieval miners are relatively powerful devices with local storage dedicated to storing hot copies of the content, high uptime (close to zero churn) and, ideally, a public IP address (reachable from anywhere). The architecture should be able to take advantage of storage in less powerful and intermittently-connected end-user devices, such as laptops and mobile phones - this is discussed as a different area further down. Although not a strict requirement, this is what will make the decentralised storage network take full benefit of planetary-scale unused storage. +We generally consider that Providers are relatively powerful devices with local storage dedicated to storing hot copies of the content, high uptime (close to zero churn) and, ideally, a public IP address (reachable from anywhere). The architecture should be able to take advantage of storage in less powerful and intermittently-connected end-user devices, such as laptops and mobile phones - this is discussed as a different area further down. Although not a strict requirement, this is what will make the decentralised storage network take full benefit of planetary-scale unused storage. It is worth highlighting that the system should be able to serve different use-cases and therefore different setups and architectures could apply depending on the use-case. For instance, the content resolution system can be different between the cases where: i) the application operates based on a closed-content ecosystem (e.g., subscriber-based music or video streaming, where control of the content is solely with the content publisher) and ii) the application operates on a totally open content space, e.g., web. -Last but not least, the system of overlay retrieval miner nodes has to be permissionless and decentralised. No entity has full control of the network of nodes, as is the case with traditional CDNs, where a single entity is in charge of the network setup and the content served by the system. +Last but not least, the system of overlay Provider nodes has to be permissionless and decentralised. No entity has full control of the network of nodes, as is the case with traditional CDNs, where a single entity is in charge of the network setup and the content served by the system. How it is done traditionally (CDNs, P2P CDNs) @@ -260,7 +260,7 @@ Properties * Providers MUST follow the crypto-economic model and the system MUST make sure that Providers do not misbehave. * The network MUST be content-addressable and operate based on content identifiers. * The system MUST be permissionless and trustless - * Anyone should be free to join and set up a retrieval miner to contribute to the network. + * Anyone should be free to join and set up a Provider node to contribute to the network. ### State-of-the-art @@ -458,7 +458,7 @@ ResNetLab organized a Research Intensive Workshop on 3DMs, out of it, the follow An area adjacent to the Distribution Graph Forming one is that of extending the network beyond Providers, as defined earlier, to also include more ephemerally-connected, end-user devices. Such devices can include laptops, desktops, (futuristic) storage-equipped WiFi Access Points, or even mobile smartphone and tablet devices. -We call these environments “opportunistic deployments” to reflect their unpredictability in terms of availability, uptime, resource quality and quantity. In contrast to RMs, as defined earlier that are expected to have stable, public and high-bandwidth connectivity, 3DM opportunistic deployments can utilise everyday user devices to extend the storage footprint of RMs and create a wealth of new network connectivity and business opportunities. +We call these environments “opportunistic deployments” to reflect their unpredictability in terms of availability, uptime, resource quality and quantity. In contrast to Providers, as defined earlier that are expected to have stable, public and high-bandwidth connectivity, 3DM opportunistic deployments can utilise everyday user devices to extend the storage footprint of Provider nodes and create a wealth of new network connectivity and business opportunities. Although this area seems to sit on the periphery of 3DMs, it is actually a highly impactful area, as it realises the vision of Decentralised Storage Networks in general and Filecoin, in particular, of regular end-users sharing their own resources to contribute to the network. That said, we place high value in capturing this opportunity and offering to end-users the opportunity of being rewarded for their contribution to the network. @@ -468,7 +468,7 @@ The literature in the field of Opportunistic and Delay-Tolerant Networks is vast #### Transient Providers -We consider end-users that store content in their ephemerally connected devices (laptops, or mobile phones) and provide them to the network through (one or more) Providers, as defined in the “Distribution Graph Forming” area. These service providers are called Transient Providers (TRMs) to depict their ephemeral nature in resource availability and time. TRMs can have some economic relationship with one or more RMs, e.g., RMs can “recruit” TRMs to expand their storage capacity and footprint. In this case, RMs have to maintain their own monitoring mechanisms and do resource allocation to ensure that they are getting desirable levels of service from their group of TRMs. +We consider end-users that store content in their ephemerally connected devices (laptops, or mobile phones) and provide them to the network through (one or more) Providers, as defined in the “Distribution Graph Forming” area. These service providers are called Transient Providers (TPs) to depict their ephemeral nature in resource availability and time. TPs can have some economic relationship with one or more Providers, e.g., Providers can “recruit” TPs to expand their storage capacity and footprint. In this case, Provider nodes have to maintain their own monitoring mechanisms and do resource allocation to ensure that they are getting desirable levels of service from their group of TPs. Opportunistic D2D @@ -478,7 +478,7 @@ This area of research and deployment is closer to traditional Delay-Tolerant Net **User Mobility** -When a user is “registered” under one address/network location to serve content (at least as far as its Retrieval Miner is concerned) and then moves to another network location, the content is not discoverable/retrievable anymore. +When a user is “registered” under one address/network location to serve content (at least as far as its Provider is concerned) and then moves to another network location, the content is not discoverable/retrievable anymore. * Brings up content mobility issues and “content churn” * Similar to node churn, this is a very tricky problem: how can you find content that was linked to some network address, when this address is not valid anymore. You can, of course, update the record if you know where the record lives, but if the record is “allowed” to be provided by anyone, then it’s difficult to even find it. From e9deaedddfe180a9b66e8e23c69266578967132e Mon Sep 17 00:00:00 2001 From: Alfonso de la Rocha Date: Thu, 11 Feb 2021 15:54:59 +0100 Subject: [PATCH 51/66] Apply suggestions from code review Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index af447e5..e20d4c8 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -387,7 +387,7 @@ The economics of current Hybrid CDN deployments give a good understanding of the ##### Game theoretical models and reward/reputation systems in P2P networks -How to design the right incentive and reputation models to incentivize certain behaviors in P2P networks has been a widely studied problem for some time now. From reputation systems where nodes track the level of good behavior of their peers and cooperate with them according to their grade; to incentive systems where good behavior is explicitly rewarded. To theoretically evaluate the strength of these schemes, they are generally mod game theoretical models are generally an objective way to evaluate the strength of these schemes, and to understand if the proposals can achieve Nash Equilibrium. +How to design the right incentive and reputation models to incentivize certain behaviors in P2P networks has been a widely studied problem for some time now. From reputation systems where nodes track the level of good behavior of their peers and cooperate with them according to their grade; to incentive systems where good behavior is explicitly rewarded. Game-theoretical models are a way to evaluate the strength of these schemes and to understand if the proposals can achieve Nash Equilibrium. **Related Papers:** From e897a24e34e73c381e44a13f315a796ea84b122f Mon Sep 17 00:00:00 2001 From: Alfonso de la Rocha Date: Thu, 11 Feb 2021 16:02:11 +0100 Subject: [PATCH 52/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index e20d4c8..acbd02d 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -343,7 +343,7 @@ In decentralized setups, it is not possible to use legal agreements to orchestra For decentralized and trustless setups, two main schemes have been traditionally used to align the incentive of the system’s participants and prevent misbehaviors: -* Trustless networks governed by a common consensus, like public blockchains, use economic models based on token rewards and clear punishments built upon a secure consensus algorithm. The consensus algorithm gives a basic layer of trust where all participants “keep an eye on one another” preventing misbehaviors, and upon which the economic model is built to align all their goals. A good example of this model is Bitcoin, where the Proof of Work consensus algorithm prevents attacks, and the mining rewards and deflationary nature of the currency ensures that Providers will be incentivized to keep serving, and users to keep using and holding the network’s currency. +* Trustless networks governed by a common consensus, like public blockchains, use economic models based on token rewards and clear punishments built upon a secure consensus algorithm. The consensus algorithm gives a basic layer of trust where all participants “keep an eye on one another” preventing misbehaviors, and upon which the economic model is built to align all their goals. A good example of this model is Bitcoin, where the Proof of Work consensus algorithm prevents attacks, the mining rewards and deflationary nature of the currency ensures that Providers will be incentivized to keep serving, and users to keep holding the network’s currency. * For trustless P2P networks where there is no common protocol run by all the participants of the network, other schemes such as reputation systems, or credit networks need to be implemented to orchestrate the relationships and good behaviors of all entities in the system. In the same way public blockchains leverage the schemes and security properties in place to design their economic model, 3DMs will have to leverage the metering and graph forming designs in place to design the right economic model. From f35340a6f283c3919d9edb631a81e02868f46200 Mon Sep 17 00:00:00 2001 From: Alfonso de la Rocha Date: Thu, 11 Feb 2021 16:05:05 +0100 Subject: [PATCH 53/66] Update DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index acbd02d..82f7626 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -343,7 +343,8 @@ In decentralized setups, it is not possible to use legal agreements to orchestra For decentralized and trustless setups, two main schemes have been traditionally used to align the incentive of the system’s participants and prevent misbehaviors: -* Trustless networks governed by a common consensus, like public blockchains, use economic models based on token rewards and clear punishments built upon a secure consensus algorithm. The consensus algorithm gives a basic layer of trust where all participants “keep an eye on one another” preventing misbehaviors, and upon which the economic model is built to align all their goals. A good example of this model is Bitcoin, where the Proof of Work consensus algorithm prevents attacks, the mining rewards and deflationary nature of the currency ensures that Providers will be incentivized to keep serving, and users to keep holding the network’s currency. +* Trustless networks governed by a common consensus, like public blockchains, use economic models based on token rewards and clear punishments built upon a secure consensus algorithm. The consensus algorithm gives a basic layer of trust where all participants “keep an eye on one another” preventing misbehaviors, and upon which the economic model is built to align all their goals. A good example of this model is Bitcoin, where the Proof of Work consensus algorithm prevents attacks, while the mining rewards incentivize miners to keep serving the network. + * For trustless P2P networks where there is no common protocol run by all the participants of the network, other schemes such as reputation systems, or credit networks need to be implemented to orchestrate the relationships and good behaviors of all entities in the system. In the same way public blockchains leverage the schemes and security properties in place to design their economic model, 3DMs will have to leverage the metering and graph forming designs in place to design the right economic model. From bad975edff70495d3699ea3b4c9b1dc6f204a64f Mon Sep 17 00:00:00 2001 From: Alfonso de la Rocha Date: Thu, 11 Feb 2021 18:07:53 +0100 Subject: [PATCH 54/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 82f7626..53304cf 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -337,7 +337,7 @@ The aim of the cryptoeconomic model is to build an incentive system to ensure th #### How it is done traditionally -In traditional and centralized setups, there is no underlying need to align the incentives of all the participants in a data delivery network. The relationships between entities in the system are governed by a set of multilateral legal agreements between the different parties. These agreements offer the basic level of trust to ensure the good behavior of all entities in the system (clearly stating the SLA, and punishments for misbehaving or not fulfilling the contract). We currently see this setup in the multilateral agreements between CDNs, ISPs, and Cloud Providers to share resources from their infrastructures for a certain price in order to build their network of relationships that allow them to provide their content delivery services efficiently. +In traditional and centralized setups, a set of legal agreements is used to govern the relationships between entities, align the incentives of all participants, and discourage disruptive behaviour. These agreements offer the basic level of trust to ensure good behavior by all entities in the system (clearly stating the SLA, and punishments for misbehaving or not fulfilling the contract). We currently see this setup in the multilateral agreements between CDNs, ISPs, and Cloud Providers to share resources from their infrastructures for a certain price in order to build a network of relationships that allow them to provide their content delivery services efficiently. In decentralized setups, it is not possible to use legal agreements to orchestrate the relationship between entities in the system. In recent deployments of hybrid CDNs, traditional, centralised CDNs leverage the resources of end users to deliver data with enhanced QoS reducing the load on their infrastructure and as a result the cost of upgrading/extending their infrastructure. End users are incentivized to contribute to the system with their upload bandwidth by making them eligible for better service. This additional reward is used to incentivize a behavior in the system (in this case users contributing with bandwidth in hybrid CDNs). However, the fact that there is still a centralized infrastructure managed by an entity with greater power over the rest of actors is enough to prevent misbehavior. This removes the need to design stronger reputation or reward systems. This does not represent a trustless setup. From e2c40a4def226841c5a2b1f09b8813e374618484 Mon Sep 17 00:00:00 2001 From: Alfonso de la Rocha Date: Thu, 11 Feb 2021 18:09:32 +0100 Subject: [PATCH 55/66] Update DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 53304cf..b7ee660 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -337,7 +337,7 @@ The aim of the cryptoeconomic model is to build an incentive system to ensure th #### How it is done traditionally -In traditional and centralized setups, a set of legal agreements is used to govern the relationships between entities, align the incentives of all participants, and discourage disruptive behaviour. These agreements offer the basic level of trust to ensure good behavior by all entities in the system (clearly stating the SLA, and punishments for misbehaving or not fulfilling the contract). We currently see this setup in the multilateral agreements between CDNs, ISPs, and Cloud Providers to share resources from their infrastructures for a certain price in order to build a network of relationships that allow them to provide their content delivery services efficiently. +In traditional and centralized setups, a set of legal agreements is used to govern the relationships between entities, align the incentives of all participants, and discourage disruptive behaviour. These agreements offer a basic level of trust to ensure good behavior by all entities in the system (clearly stating the SLA, and punishments for misbehaving or not fulfilling the contract). We currently see this setup in the multilateral agreements between CDNs, ISPs, and Cloud Providers to share resources from their infrastructures for a certain price in order to build a network of relationships that allow them to provide their content delivery services efficiently. In decentralized setups, it is not possible to use legal agreements to orchestrate the relationship between entities in the system. In recent deployments of hybrid CDNs, traditional, centralised CDNs leverage the resources of end users to deliver data with enhanced QoS reducing the load on their infrastructure and as a result the cost of upgrading/extending their infrastructure. End users are incentivized to contribute to the system with their upload bandwidth by making them eligible for better service. This additional reward is used to incentivize a behavior in the system (in this case users contributing with bandwidth in hybrid CDNs). However, the fact that there is still a centralized infrastructure managed by an entity with greater power over the rest of actors is enough to prevent misbehavior. This removes the need to design stronger reputation or reward systems. This does not represent a trustless setup. From d51ba278fe88b2926f4213a23beae5b167ed3cd2 Mon Sep 17 00:00:00 2001 From: David Dias Date: Fri, 12 Feb 2021 11:24:18 +0100 Subject: [PATCH 56/66] add dottoc --- .../DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 105 +++++++++++++----- 1 file changed, 75 insertions(+), 30 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index b7ee660..7a92179 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -1,6 +1,53 @@ # Decentralized Data Delivery Markets (3DMs) - Open Problem Statement -DO NOT MERGE WITHOUT CREATING DOCTOC https://www.npmjs.com/package/doctoc + + + +- [Overall](#overall) + - [Short description](#short-description) + - [Long description](#long-description) + - [Definition of the actors](#definition-of-the-actors) + - [Expected requirements](#expected-requirements) +- [Areas of Work](#areas-of-work) + - [Data Delivery Metering & Fair Exchange](#data-delivery-metering--fair-exchange) + - [How it is done traditionally](#how-it-is-done-traditionally) + - [Properties desired](#properties-desired) + - [How is Data Delivery Metering different from Fair Exchange](#how-is-data-delivery-metering-different-from-fair-exchange) + - [Known challenges](#known-challenges) + - [State-of-the-art](#state-of-the-art) + - [Pay-per-packet](#pay-per-packet) + - [Lock/Unlock access to the resource](#lockunlock-access-to-the-resource) + - [Optimistic Fair Exchange](#optimistic-fair-exchange) + - [Reputation based](#reputation-based) + - [Privacy focused](#privacy-focused) + - [Traditional approaches (close to Web 2.0 world)](#traditional-approaches-close-to-web-20-world) + - [Known attacks to be mitigated](#known-attacks-to-be-mitigated) + - [New ideas being explored](#new-ideas-being-explored) + - [Distribution Graph Forming](#distribution-graph-forming) + - [State-of-the-art](#state-of-the-art-1) + - [DHT-based:](#dht-based) + - [Name-based routing:](#name-based-routing) + - [DNS-style, object-level indexing service:](#dns-style-object-level-indexing-service) + - [PubSub-style:](#pubsub-style) + - [Known attacks to be mitigated](#known-attacks-to-be-mitigated-1) + - [New ideas being explored](#new-ideas-being-explored-1) + - [CryptoEconomic model for Data Delivery](#cryptoeconomic-model-for-data-delivery) + - [How it is done traditionally](#how-it-is-done-traditionally-1) + - [Properties desired](#properties-desired-1) + - [State-of-the-art](#state-of-the-art-2) + - [Economics of Hybrid CDNs and P2P networks](#economics-of-hybrid-cdns-and-p2p-networks) + - [Game theoretical models and reward/reputation systems in P2P networks](#game-theoretical-models-and-rewardreputation-systems-in-p2p-networks) + - [Credit networks and Token Designs](#credit-networks-and-token-designs) + - [Auctions, decentralized markets, and efficient resource allocation](#auctions-decentralized-markets-and-efficient-resource-allocation) + - [Known attacks to be mitigated](#known-attacks-to-be-mitigated-2) + - [New ideas being explored](#new-ideas-being-explored-2) + - [Additional: Opportunistic deployment](#additional-opportunistic-deployment) + - [State-of-the-art](#state-of-the-art-3) + - [Transient Providers](#transient-providers) + - [Known shortcomings](#known-shortcomings) + - [New ideas being explored](#new-ideas-being-explored-3) + + # Overall @@ -8,7 +55,7 @@ DO NOT MERGE WITHOUT CREATING DOCTOC https://www.npmjs.com/package/doctoc With the emergence of Decentralized Storage Networks and the rapid decrease in the price of storage services and hardware, there is a rapidly growing need to leverage the additional storage capacity contributed to Decentralized Storage Networks by new players, including end-users, and use it to deliver reliable and high-quality storage and delivery services. Similarly to Content Delivery Networks (CDNs) for the traditional Cloud Storage market, we now have the opportunity to build **Decentralised CDNs** for Decentralised Storage Networks. The Decentralized Data Delivery Markets (3DMs) Open Problem covers all the essential areas of work that need to be studied in order to create **a fully permissionless free market for data delivery that supports fair data exchange** on the service provided. -## Long description +## Long description Serving content globally at scale is a hard technical problem, as evidenced by the multiple decades of innovation and improvements in the Content Delivery Networks field. This challenge becomes even more interesting once we move away from centralized cloud infrastructure, which is typically managed and monitored by a single party, to a decentralized network that is permissionless, possibly anonymous, constantly changing (i.e. with high node churn) and lacking access to the convenience of a third party mediator system that facilitates the fair exchange of goods (e.g. credit providers). The benefits of moving to a decentralised setting are significant, however: cheaper storage and delivery, resilience against business failures (i.e., the network and all its components are not dependent on business decisions made by a single entity), independence from the personal data-driven business models, as well as a significantly lower barrier to entry for new players who want to contribute. @@ -37,8 +84,7 @@ For convenience and shared understanding, we first define the agents within a 3D * **Clients** - Agents that fetch data from Providers. * **Providers** - Agents that offer data delivery services to Clients. -* **Content Publishers** - Agents that create content and want to have it distributed (or intermediaries thereof). - +* **Content Publishers** - Agents that create content and want to have it distributed (or intermediaries thereof). ### Expected requirements @@ -46,7 +92,7 @@ We present this list as a guide to protocol designers of what we expect a 3DM im * **_MUST be Decentralized and not just federated_** * Anyone should be able to join and leave at any time, without requiring permission -* **_The exchanges of value_** **_MUST be verifiable_** +* **_The exchanges of value_** **_MUST be verifiable_** * The payment for bandwidth/latency in token should match what has been agreed and fulfilled * The payment should be fulfilled if the SLA is fulfilled * Parties should be able to verify that the service provided was correct (e.g. that they received the right file) @@ -78,18 +124,18 @@ Throughout the exploration of the design space, we identified 3 areas of work th We believe that this area of work is where the main crux for Decentralized Data Delivery Markets come to reality. Without it (i.e. without relying on a verified mediator to serve as escrow and referee), it will be impossible to have a fully permissionless, decentralized and open market for data delivery. -#### How it is done traditionally +#### How it is done traditionally -In traditional or Web 2.0 setups (e.g. CDNs, Mirrors, Static Servers and so on), the correct metering happens on the server and is handled by the provider, which the clients trust to perform correct measurements. This measurement translates into a statement of how much the service was, ultimately resulting in an invoice and request for payment. This payment is facilitated by a trusted third party (e.g. credit card company) and a legal contract that can be used to resolve disputes in case the client fails to pay to the provider and/or the provider fails to deliver the service promised to the client. +In traditional or Web 2.0 setups (e.g. CDNs, Mirrors, Static Servers and so on), the correct metering happens on the server and is handled by the provider, which the clients trust to perform correct measurements. This measurement translates into a statement of how much the service was, ultimately resulting in an invoice and request for payment. This payment is facilitated by a trusted third party (e.g. credit card company) and a legal contract that can be used to resolve disputes in case the client fails to pay to the provider and/or the provider fails to deliver the service promised to the client. -In a decentralized environment that strives to be permissionless, we can’t expect to rely on such legal agreements and third parties and, therefore, need a way to prove fair exchange. That is, we need to be able to prove that the service was delivered correctly and the metering of the provided service was done correctly. This will, in turn, verify the exchange between the two parties and issue the correct payment (i.e. fair exchange). +In a decentralized environment that strives to be permissionless, we can’t expect to rely on such legal agreements and third parties and, therefore, need a way to prove fair exchange. That is, we need to be able to prove that the service was delivered correctly and the metering of the provided service was done correctly. This will, in turn, verify the exchange between the two parties and issue the correct payment (i.e. fair exchange). This trustless property is really important as we know that markets with no mediators, escrows, and other assurance services (e.g. underground markets) are prone to scams, as the users have to trust that the provider will execute on the agreement. In a trustless environment, once the transfer is completed, there is no way to dispute it. #### Properties desired * The exchanges of value MUST be verifiable and correct - * Fairness: + * Fairness: * The payment MUST be fulfilled if the SLA is fulfilled * The payment for bandwidth/latency SHOULD match what was agreed and provided * Verifiability: @@ -107,7 +153,7 @@ The field of Fair Exchange overlaps in part with Service Metering. Some notable * Fair Exchange is traditionally between 2 parties * Metering should be available to properly measure the service quality and service provided by one or more parties (e.g. streaming from multiple endpoints) -In review, the Metering & Fair Exchange fields end up contributing to each other and it is likely that the intersection of both is needed for a sound solution. +In review, the Metering & Fair Exchange fields end up contributing to each other and it is likely that the intersection of both is needed for a sound solution. #### Known challenges @@ -125,7 +171,7 @@ In review, the Metering & Fair Exchange fields end up contributing to each other * How to support others (e.g. Content Publishers) paying for the usage? * How to make it private (i.e. so that others don’t know which users are requesting what content)? * Performance - * How to overcome send-and-halt in order to maximize bandwidth throughput + * How to overcome send-and-halt in order to maximize bandwidth throughput * How to support multipath (i.e. fetching from multiple sources) ### State-of-the-art @@ -152,13 +198,12 @@ Solutions of this type enable the client and the provider to verify the actual s These solutions can be found in several forms: - [Solving the Buyer and Seller’s Dilemma: A Dual-Deposit Escrow Smart Contract for Provably Cheat-Proof Delivery and Payment for a Digital Good without a Trusted Mediator](http://anrg.usc.edu/www/papers/Dual_Deposit_ICBC_2019.pdf) - [FairSwap: How to fairly exchange digital goods](https://eprint.iacr.org/2018/740.pdf) -- [Zero-Knowledge Contingent Payments Revisited: Attacks and Payments for Services](https://acmccs.github.io/papers/p229-campanelliA.pdf) +- [Zero-Knowledge Contingent Payments Revisited: Attacks and Payments for Services](https://acmccs.github.io/papers/p229-campanelliA.pdf) **Known shortcomings of these approaches:** * Vulnerable to griefing attacks. * Generally, these solutions focus on the fulfilment of the delivery, more so than measuring the speed and rate of the transfer. - #### Optimistic Fair Exchange These solutions build on top of the Fair Exchange ones and adopt an Optimistic approach where some sort of Reputation or Stake is used to give both parties trust that the exchange will occur correctly. If one misbehaves, they can still rely on a dispute system. @@ -180,7 +225,7 @@ Reputation-based solutions give the possibility to speed up the transfer and imp - Other Gradual Fair Exchange Class of constructions **Known shortcomings of these approaches:** -* Not a direct shortcoming, but a challenge is that each participant my have different levels of sensitivity with regards to how much they are willing to rely on trust vs. strict verifiability. +* Not a direct shortcoming, but a challenge is that each participant my have different levels of sensitivity with regards to how much they are willing to rely on trust vs. strict verifiability. #### Privacy focused @@ -206,18 +251,18 @@ These approaches suggest novel ways to add integrity checks to data transferred Here we list the attacks to consider when designing a solution. These are: * Malicious actor forces provider to spend bandwidth without issuing payment (also known as Griefing Attack) - * Type: client fraud - * Consequence: bandwidth is spent -* Malicious actor forces provider to pay to retrieve the file from storage point (e.g. Filecoin, Cloud Storage, etc) without the provider itself ever getting paid + * Type: client fraud + * Consequence: bandwidth is spent +* Malicious actor forces provider to pay to retrieve the file from storage point (e.g. Filecoin, Cloud Storage, etc) without the provider itself ever getting paid * Type: client fraud * Consequence: currency is spent -* Provider claims it has sent the file, but actually never did +* Provider claims it has sent the file, but actually never did * Type: Provider fraud * Consequence: Client gets penalized without receiving the service -* Metering Inflation +* Metering Inflation * Providers report to have used more resources to serve content to clients than what they have actually done. * Sybil / Throttle attacks - * Description: A set of nodes collude against a content-provider to try and make him go bankrupt (loss of all its “SLA stake”). They all try to download content from that content provider at the same time, so that Providers start consuming the stake from the content providers. If clients are not paying for the service we take the risk of enabling this attack. + * Description: A set of nodes collude against a content-provider to try and make him go bankrupt (loss of all its “SLA stake”). They all try to download content from that content provider at the same time, so that Providers start consuming the stake from the content providers. If clients are not paying for the service we take the risk of enabling this attack. ### New ideas being explored @@ -289,14 +334,14 @@ The target area and expected outcome of this topic is to form the network, the i Our initial, but thorough investigation resulted in the following potential groups of designs for content discovery and resolution: -#### DHT-based: +#### DHT-based: DHTs are very popular constructions in P2P networks. The DHT system is used as a content resolution mechanism, as well as a content and peer routing system. The routing table is split between peers that participate in the DHT, providing higher resilience and scalability. In a DHT-based system, clients “walk” the DHT (iteratively, or recursively) to find the provider record and then directly connect to the peer included in the provider record. * **Pros:** tested in the past, engineers have lots of experience with these structures, can become faster and less expensive than the IPFS DHT setup, if: i) re-publishing is done at coarser granularities, which is reasonable if we assume stable connectivity, i.e., low peer churn (reasonable to assume for Providers), and ii) some payment is associated with publishing. It is also reasonable to assume public IP connectivity for Providers. * **Cons:** slow, several round-trips needed to resolve content (especially in case of iterative DHT lookup). The solution will not scale in the longer term. -#### Name-based routing: +#### Name-based routing: In Name-based routing systems, routing hints are integrated as part of the content name; routing tables are filled with routing hints at network setup time by a routing protocol. Routers make hop-by-hop forwarding decisions based on content names seen in requests and routing hints in their routing tables. Matching between hints and names depend on the structure of the names, e.g., hierarchical vs flat. @@ -323,7 +368,7 @@ In pubsub systems information propagates in the network based on topics that nod * Providers serve bogus content * Providers provide false provider records (i.e., they claim that they have and can serve content that they actually do not have) * Cache poisoning -* Privacy attacks: there are a number of privacy considerations and threat vectors to be taken into account +* Privacy attacks: there are a number of privacy considerations and threat vectors to be taken into account ### New ideas being explored @@ -335,7 +380,7 @@ ResNetLab organized a Research Intensive Workshop on 3DMs, from which the follow The aim of the cryptoeconomic model is to build an incentive system to ensure that all actors in the network are aligned towards the same goal: to deliver data efficiently. The cryptoeconomic model is tightly coupled to all of the areas presented above, offering a way to incentivize desired behaviors in the network. Ideally, the economic model should offer the substrate to build a decentralized and trustless infrastructure for data delivery able to compete against traditional CDN infrastructures in terms of cost, scalability, and price. -#### How it is done traditionally +#### How it is done traditionally In traditional and centralized setups, a set of legal agreements is used to govern the relationships between entities, align the incentives of all participants, and discourage disruptive behaviour. These agreements offer a basic level of trust to ensure good behavior by all entities in the system (clearly stating the SLA, and punishments for misbehaving or not fulfilling the contract). We currently see this setup in the multilateral agreements between CDNs, ISPs, and Cloud Providers to share resources from their infrastructures for a certain price in order to build a network of relationships that allow them to provide their content delivery services efficiently. @@ -359,7 +404,7 @@ Finally, auction markets and game theoretical models have been traditionally use * Or the other way around, good behaving entities should be rewarded by the model. * In its different layers, the system will include schemes to prevent attacks in the network (client metering, sybil attacks, data-ransoming, etc.). However, this economic model should disincentivize these kinds of misbehavior in advance (from a game theoretical approach, the system should target a Nash Equilibrium in the system where misbehaviors are disincentivized). * It should also avoid “malicious economic attacks” such as a content provider trying to bankrupt its competition by draining the stake of his retrieval deal. -* The model SHOULD foster collaboration between entities to benefit users perceived QoS / QoE. +* The model SHOULD foster collaboration between entities to benefit users perceived QoS / QoE. * The model should reward entities that collaborate to serve content efficiently and with high QoS to users. * This will also result in an efficient allocation of resources in the network. * If any part of the economics of the system ends up being orchestrated by an auction market, this market MUST be always available. @@ -407,7 +452,7 @@ How to design the right incentive and reputation models to incentivize certain b ##### Credit networks and Token Designs -Credit networks provide liquidity in markets where transaction volumes are close to balanced in both directions. They offer a way of performing payments and rewarding behaviors without the need of a common consensus or dedicated agreements between all the entities in the systems. Credit networks require no “a priori” trust relationships when they are backed by on-chain escrows, so they represent an interesting approach to tackle the economics of 3DMs. +Credit networks provide liquidity in markets where transaction volumes are close to balanced in both directions. They offer a way of performing payments and rewarding behaviors without the need of a common consensus or dedicated agreements between all the entities in the systems. Credit networks require no “a priori” trust relationships when they are backed by on-chain escrows, so they represent an interesting approach to tackle the economics of 3DMs. **_Related Papers:_** @@ -421,7 +466,7 @@ Credit networks provide liquidity in markets where transaction volumes are close **_Learnings: _** * Economics of credit networks. -* Using credit networks for fast payments and reputation systems. +* Using credit networks for fast payments and reputation systems. * Credit networks for content distribution in P2P networks. ##### Auctions, decentralized markets, and efficient resource allocation @@ -445,7 +490,7 @@ Auctions and decentralized markets have traditionally been a good way of allocat ### Known attacks to be mitigated * **Sybil attacks:** Actors trying to game the system by indiscriminately generating a large amount of entities and gaining large influence over the system (forging client retrievals, overestimating the use of resources of a provider, preventing access from some resource in the network, making actors in the system dedicate resources to useless work, etc.). -* **Collusion attacks:** Several independent actors in the system colluding to perform attacks in order to gain large influence in the network. The type of attacks that can be performed with a collusion attack are similar to the ones performed in a sybil attack, but without having to generate “pseudonymous identities”. +* **Collusion attacks:** Several independent actors in the system colluding to perform attacks in order to gain large influence in the network. The type of attacks that can be performed with a collusion attack are similar to the ones performed in a sybil attack, but without having to generate “pseudonymous identities”. * **Data ransoming:** This attack takes place when a provider agrees to serve some data from a publisher, but ends up preventing its retrieval from clients until a ransom is paid. An alternative form of this attack occurs when a provider is serving chunks of content to a client, and doesn’t deliver the last couple of chunks until it pays a ransom. * **Malicious economic attacks:** Attacks aimed at economically harming actors in the systems. For instance, clients trying to get a provider or content publisher bankrupt, providers offering abusive prices to content publishers, providers and clients colluding to avoid the economic rewards of target entities, etc. @@ -473,9 +518,9 @@ We consider end-users that store content in their ephemerally connected devices Opportunistic D2D -This area of research and deployment is closer to traditional Delay-Tolerant Networks (DTNs). We envision applications where mobile devices contribute to the distribution of content in the mobile domain. These can be akin to previously proposed [User-Operated Mobile Content Distribution Networks](https://drive.google.com/file/d/1vvtQ7MJb4YFRXxeo2GewGh8EVYEqV-iJ/view?usp=sharing), or applications to realise concepts such as “[Floating Content](https://drive.google.com/file/d/1yOniHbAjLYv3q09pff7gnulOwT_NYCYG/view?usp=sharing)”. +This area of research and deployment is closer to traditional Delay-Tolerant Networks (DTNs). We envision applications where mobile devices contribute to the distribution of content in the mobile domain. These can be akin to previously proposed [User-Operated Mobile Content Distribution Networks](https://drive.google.com/file/d/1vvtQ7MJb4YFRXxeo2GewGh8EVYEqV-iJ/view?usp=sharing), or applications to realise concepts such as “[Floating Content](https://drive.google.com/file/d/1yOniHbAjLYv3q09pff7gnulOwT_NYCYG/view?usp=sharing)”. -### Known shortcomings +### Known shortcomings **User Mobility** From e035a8b0a67dce875ac6963da6eac335d2103ef3 Mon Sep 17 00:00:00 2001 From: David Dias Date: Fri, 12 Feb 2021 11:26:52 +0100 Subject: [PATCH 57/66] add Open problem to the table --- README.md | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index a7b5ad8..d5a77c3 100644 --- a/README.md +++ b/README.md @@ -15,8 +15,8 @@ - [Motivation & Description](#motivation--description) - [Research](#research) - [Areas](#research) - - [Projects](#research) - - [Collaborations](#collaborations) + - [Projects](#research) + - [Collaborations](#collaborations) - [Collab projects tracking threads](#collab-projects-tracking-threads) - [Publications, Talks & Trainings](#publications-talks--trainings) - [Team](#team) @@ -48,31 +48,37 @@ The lab's genesis comes from a need present in the IPFS and libp2p projects to a Open Problem(s) Short Description - + - Networks Observability + Resilliency in Adversarial Networks NEEDs OPEN PROBLEM - + - Resilliency in Adversarial Networks + Networks Observability NEEDs OPEN PROBLEM - + + + + Decentralized Data Delivery Markets + General open problem + + Heterogeneous Runtimes General open problem Making libp2p / IPFS modules and protocols the de-facto distributed substrate for connected devices in the near-future Internet. Enable the execution of libp2p nodes and its underlying protocols anywhere (any device and any runtime). Allow global connectivity between devices. Enable “offline-first” applications. - + Content Addressing Routing at Scale (1M, 10M, 100M, 1B.. nodes) Content-addressable networks face the challenge of routing scalability, as the amount of addressable elements in the network rises by several orders of magnitude compared to the host-addressable Internet of today. - + @@ -84,37 +90,37 @@ The lab's genesis comes from a need present in the IPFS and libp2p projects to a Mutability Mutable Data (Naming, Real-Time, Guarantees) Enabling a multitude of different patterns of interactions between users, machines and both. In other words, what are the essential primitives that must be provided for dynamic applications to exist, what are the guarantees they require (consistency, availability, persistancy, authenticity, etc) from the underlying layer in order create powerful and complete applications in the Distributed Web. - - + + Human Readable Naming You can only have two of three properties for a name: Human-meaningful, Secure and/or Decentralized. This is Zooko's Trilemma. Can we have all 3, or even more? Can context related to some data help solve this problem? - + PubSub at Scale (1M, 10M, 100M, 1B.. nodes) As the IPFS system is evolving and growing, communicating new entries to the IPNS is becoming an issue due to the increased network and node load requirements. The expected growth of the system to multiple millions of nodes is going to create significant performance issues, which might render the system unusable. Despite the significant amount of related literature on the topic of pub/sub, very few systems have been tested to that level of scalability, while those that have been are mostly cloud-based, managed and structured infrastructures. - + Data Exchange Enhanced Bitswap/GraphSync with more Network Smarts Bitswap is a simple protocol and it generally works. However, we feel that its performance can be substantially improved. One of the main factors that hold performance back is the fact that a node cannot request a subgraph of the DAG and results in many round-trips in order to “walk down” the DAG. The current operation of bitswap is also very often linked to duplicate transmission and receipt of content which overloads both the end nodes and the network. - - + + - Distributed Type Systems + Distributed Type Systems Improved layouts to represent data in hash-linked graphs (using IPLD) Future™ ⚙️ - + ### Projects - Hydra Booster -- Gossipsub v1.1 +- Gossipsub v1.1 - drand - [Beyond Bitswap](https://github.com/protocol/beyond-bitswap) - P2P Observatory @@ -151,4 +157,4 @@ The lab's genesis comes from a need present in the IPFS and libp2p projects to a ### Contact -You can reach out to us anytime with your question and interest in these projects by emailing [resnetlab@protocol.ai](mailto:resnetlab@protocol.ai) \ No newline at end of file +You can reach out to us anytime with your question and interest in these projects by emailing [resnetlab@protocol.ai](mailto:resnetlab@protocol.ai) From 6038871291d0cda36835b658f00b4f2f1a8927a7 Mon Sep 17 00:00:00 2001 From: David Dias Date: Fri, 12 Feb 2021 11:28:24 +0100 Subject: [PATCH 58/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 7a92179..88312ed 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -122,7 +122,7 @@ Throughout the exploration of the design space, we identified 3 areas of work th ## Data Delivery Metering & Fair Exchange -We believe that this area of work is where the main crux for Decentralized Data Delivery Markets come to reality. Without it (i.e. without relying on a verified mediator to serve as escrow and referee), it will be impossible to have a fully permissionless, decentralized and open market for data delivery. +We believe that this area of work is the main crux for Decentralized Data Delivery Markets. Without it, it will be impossible to have a fully permissionless, decentralized, and open market for data delivery. #### How it is done traditionally From 724145342b99c7f04de53fb8de8edfacb012d3f5 Mon Sep 17 00:00:00 2001 From: David Dias Date: Fri, 12 Feb 2021 11:29:28 +0100 Subject: [PATCH 59/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 1 - 1 file changed, 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 88312ed..80ccbad 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -107,7 +107,6 @@ We present this list as a guide to protocol designers of what we expect a 3DM im Additional expectations (i.e. bonus points): -* Data SHOULD be readily available without having to wait for the [unsealing process](https://spec.filecoin.io/#section-glossary.seal) at storage nodes (e.g. Filecoin Storage Miners). * Data retrieval SHOULD be anonymous and privacy-preserving. Other considerations: From c78a7427238caa184c94bca68977692f096b3c4f Mon Sep 17 00:00:00 2001 From: David Dias Date: Fri, 12 Feb 2021 11:30:06 +0100 Subject: [PATCH 60/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 80ccbad..dd994fe 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -133,7 +133,7 @@ This trustless property is really important as we know that markets with no medi #### Properties desired -* The exchanges of value MUST be verifiable and correct +* The exchanges of value MUST be verifiable and fair * Fairness: * The payment MUST be fulfilled if the SLA is fulfilled * The payment for bandwidth/latency SHOULD match what was agreed and provided From 79c7a10b2520a90dcf7ad623082a4971a4720b0e Mon Sep 17 00:00:00 2001 From: David Dias Date: Fri, 12 Feb 2021 11:30:44 +0100 Subject: [PATCH 61/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Jorge Soares <547492+jsoares@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index dd994fe..2c9abe9 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -179,7 +179,7 @@ When it comes to the state of the art, we found that the existing solutions fall #### Pay-per-packet -Pay-per-packet solutions offer granular control over what gets paid, enabling the option to verify if the SLA is being fulfilled. However, their setup cost and latency overhead puts them behind on being able to max out the available underlay resources. +Pay-per-packet solutions offer granular control over what gets paid, enabling the option to verify if the SLA is being fulfilled. However, their setup cost and latency overhead put them at a disadvantage in terms of maximizing utilization of the underlying resources. These type of solutions have been found/highlighted: From 0d9e6e16c4d7cdf00a5e3c38a9d1c8e092a80a8c Mon Sep 17 00:00:00 2001 From: David Dias Date: Fri, 12 Feb 2021 11:31:24 +0100 Subject: [PATCH 62/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md Co-authored-by: Yiannis Psaras <52073247+yiannisbot@users.noreply.github.com> --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 2c9abe9..89ac9ef 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -290,7 +290,7 @@ Last but not least, the system of overlay Provider nodes has to be permissionles How it is done traditionally (CDNs, P2P CDNs) -Traditionally, content is stored, hosted, replicated and delivered by centralised CDNs. In those setups, **the CDN entity has full control over: i) the formation of the network**, e.g., where to place servers and how to interconnect them, both in terms of topology and in terms of bandwidth deployed between servers, **ii) the placement and replication of content**, i.e., where to put hot copies of content, as well as **iii) the network elements that are responsible for content resolution**. +Traditionally, content is stored, hosted, replicated and delivered by centralised CDNs. In those setups, **the CDN entity has full control over: i) the formation of their network of servers**, e.g., in which geographic areas to place servers and how to interconnect them, both in terms of topology and in terms of bandwidth deployed between servers, **ii) the placement and replication of content**, i.e., where to put hot copies of content, as well as **iii) the network elements that are responsible for content resolution**. In a centralized setup it is much easier to optimise performance and provide guarantees, compared to the case where **every network node is operated by a separate (legal) entity that operates rationally in a profit-maximizing way**. From 8fd90d5406b695c12d14e72aad64bcb5a7db9a50 Mon Sep 17 00:00:00 2001 From: David Dias Date: Fri, 12 Feb 2021 11:46:02 +0100 Subject: [PATCH 63/66] Update OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 89ac9ef..7dc7896 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -267,8 +267,9 @@ Here we list the attacks to consider when designing a solution. These are: ResNetLab organized a Research Intensive Workshop on 3DMs, out of which, the following ideas, structured as RFCs emerged: -* Link to RFCs: - * TOFILL +* [RFC: ZKCP with Fair Exchanges of 1 bit](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_ZKCP%20with%20Fair%20Exchanges%20of%201%20bit.md) +* [RFC: ZKCP Optimizations](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_ZKCP%20Optimizations.md) +* [RFC_Optimistic ZKC(S)P](https://github.com/protocol/ResNetLab/pull/15) ## Distribution Graph Forming From 8bec6348dfe440cc215bf8cc68b524a09efdd6b1 Mon Sep 17 00:00:00 2001 From: Alfonso de la Rocha Date: Fri, 12 Feb 2021 11:52:44 +0100 Subject: [PATCH 64/66] Added cryptoecon RFCs --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index 7dc7896..c6b788a 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -551,7 +551,8 @@ The incentive to participate and the cryptoeconomic model behind the opportunist ### New ideas being explored -ResNetLab organized a Research Intensive Workshop on 3DMs, out of it, the following ideas, structured as RFCs emerged: +ResNetLab organized a Research Intensive Workshop on 3DMs, out of which, the following ideas, structured as RFCs emerged: -Link to RFCs -* TO FILL +* [RIW2021|RFC: QFIL Closed Retrieval Economy](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_QFIL%20Closed%20Retrieval%20Economy.md) +* [RIW2021|RFC: No market 3DMs](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_NoMarkets3DM.md) +* [RIW2021|RFC: Credit-based Retrieval Network](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_Credit-based%20Retrieval%20Network) From 61ce8d563d1276569e698a31be883310271c5c93 Mon Sep 17 00:00:00 2001 From: Alfonso de la Rocha Date: Fri, 12 Feb 2021 11:56:54 +0100 Subject: [PATCH 65/66] Added RFCs Graph Forming --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index c6b788a..ac7c0ff 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -374,7 +374,9 @@ In pubsub systems information propagates in the network based on topics that nod ResNetLab organized a Research Intensive Workshop on 3DMs, from which the following ideas emerged and were structured as RFCs: -* Link to RFCs TO FILL +* [RFC: Incentive alignment for collaborative routing](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_Incentive%20alignment%20for%20collaborative%20routing.md) +* [RFC: Name-based Request Forwarding for Nearest Replica Routing](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_Name-based%20Request%20Forwarding%20for%20Nearest%20Replica%20Routing.md) +* [RFC: Omniscient Routers](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_Omniscient%20Routers.md) ## CryptoEconomic model for Data Delivery @@ -553,6 +555,6 @@ The incentive to participate and the cryptoeconomic model behind the opportunist ResNetLab organized a Research Intensive Workshop on 3DMs, out of which, the following ideas, structured as RFCs emerged: -* [RIW2021|RFC: QFIL Closed Retrieval Economy](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_QFIL%20Closed%20Retrieval%20Economy.md) -* [RIW2021|RFC: No market 3DMs](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_NoMarkets3DM.md) -* [RIW2021|RFC: Credit-based Retrieval Network](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_Credit-based%20Retrieval%20Network) +* [RFC: QFIL Closed Retrieval Economy](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_QFIL%20Closed%20Retrieval%20Economy.md) +* [RFC: No market 3DMs](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_NoMarkets3DM.md) +* [RFC: Credit-based Retrieval Network](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_Credit-based%20Retrieval%20Network) From 707a5f2dc05a6d904cbb25824eda788f5153139b Mon Sep 17 00:00:00 2001 From: Alfonso de la Rocha Date: Fri, 12 Feb 2021 11:59:37 +0100 Subject: [PATCH 66/66] Added RFC Opportunistic Deployments --- OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md index ac7c0ff..579c8a4 100644 --- a/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md +++ b/OPEN_PROBLEMS/DECENTRALIZED_DATA_DELIVERY_MARKETS.md @@ -500,7 +500,9 @@ Auctions and decentralized markets have traditionally been a good way of allocat ResNetLab organized a Research Intensive Workshop on 3DMs, out of it, the following ideas, structured as RFCs emerged: -* Link to RFCs TO FILL +* [RFC: QFIL Closed Retrieval Economy](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_QFIL%20Closed%20Retrieval%20Economy.md) +* [RFC: No market 3DMs](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_NoMarkets3DM.md) +* [RFC: Credit-based Retrieval Network](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_Credit-based%20Retrieval%20Network) ## Additional: Opportunistic deployment @@ -555,6 +557,6 @@ The incentive to participate and the cryptoeconomic model behind the opportunist ResNetLab organized a Research Intensive Workshop on 3DMs, out of which, the following ideas, structured as RFCs emerged: -* [RFC: QFIL Closed Retrieval Economy](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_QFIL%20Closed%20Retrieval%20Economy.md) -* [RFC: No market 3DMs](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_NoMarkets3DM.md) -* [RFC: Credit-based Retrieval Network](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_Credit-based%20Retrieval%20Network) +* [RFC: Hybrid CDN with Recruiter Providers](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_Hybrid%20CDN%20with%20Recruiter%20Providers.md) +* [RFC: OS-level OppNet Component](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_OS-level%20OppNet%20Component.md) +* [RFC: On-Demand Opportunistic Resource Deployment](https://github.com/protocol/ResNetLab/blob/master/3DM_RFC/RIW2021_RFC_On-Demand%20Opportunistic%20Resource%20Deployment.md)