From 29b80218ffc4216a637509acd87982a2f127533c Mon Sep 17 00:00:00 2001 From: Andrew-Clews Date: Fri, 13 Sep 2024 11:19:58 -0400 Subject: [PATCH] Updated with more GIPs to expand test sample --- ...7-allow-batching-calls-staking-contract.md | 49 ++++ GIP-Test/0005-gas-costing.md | 89 ++++++ GIP-Test/0006-gip-withdraw-helper.md | 104 +++++++ GIP-Test/0028-subsidized-query-settlement.md | 113 ++++++++ GIP-Test/003-gip-rewards-nosignal.md | 63 ++++ ...acing-Bonding-Curves-with-Indexing-Fees.md | 274 ++++++++++++++++++ .../0065-subgraph-availability-manager.md | 125 ++++++++ GIP-Test/0073-Reserved.md | 36 +++ 8 files changed, 853 insertions(+) create mode 100644 GIP-Test/00017-allow-batching-calls-staking-contract.md create mode 100644 GIP-Test/0005-gas-costing.md create mode 100644 GIP-Test/0006-gip-withdraw-helper.md create mode 100644 GIP-Test/0028-subsidized-query-settlement.md create mode 100644 GIP-Test/003-gip-rewards-nosignal.md create mode 100644 GIP-Test/0058-Replacing-Bonding-Curves-with-Indexing-Fees.md create mode 100644 GIP-Test/0065-subgraph-availability-manager.md create mode 100644 GIP-Test/0073-Reserved.md diff --git a/GIP-Test/00017-allow-batching-calls-staking-contract.md b/GIP-Test/00017-allow-batching-calls-staking-contract.md new file mode 100644 index 0000000..8d17f4f --- /dev/null +++ b/GIP-Test/00017-allow-batching-calls-staking-contract.md @@ -0,0 +1,49 @@ +--- +GIP: "0017" +Title: Allow batching calls in Staking contract +Authors: Ariel Barmat +Created: 2021-09-25 +Stage: Candidate +Implementations: https://github.com/graphprotocol/contracts/pull/475 +--- + +# Abstract + +Expose a way to batch multiple calls into a single transaction. It provides great flexibility for indexer agents to combine multiple functions in different ways. It also reduce the gas cost by saving the initial gas, and in some cases, accessing a "used" slot by the other bundled transactions. + +# Specification + +A new contract called MultiCall is introduced, inspired by the one used by Uniswap. The payable keyword was removed from the `multicall()` as the protocol does not deal with ETH. Additionally, it is insecure in some instances if the contract relies on msg.value. + +The Staking contract inherits from MultiCall that expose a public `multicall(bytes[] calldata data)` function that receives an array of payloads to send to the contract itself. This allows to batch ANY publicly callable contract function. + +By using a multicall a user can batch an arbitrary number of operations into a single transaction. The indexer agent can combine close, open, claim transactions in any way it sees more effective. + +# Implementation + +See [@graphprotocol/contracts#475](https://github.com/graphprotocol/contracts/pull/475) + +# Backwards Compatibility + +The proposal introduces some breaking changes to save bytecode. + +The following functions are removed as they can be constructed using a +multicall: + +- closeAllocationMany() +- closeAndAllocate() +- claimMany() + +# Validation + +### Audits + +The implementation was audited by Consensys Diligence. + +### Testnet + +The implementation has not yet been deployed to Testnet. + +# Copyright Waiver + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). \ No newline at end of file diff --git a/GIP-Test/0005-gas-costing.md b/GIP-Test/0005-gas-costing.md new file mode 100644 index 0000000..92addca --- /dev/null +++ b/GIP-Test/0005-gas-costing.md @@ -0,0 +1,89 @@ +--- +GIP: "0005" +Title: Gas costing for handlers +Authors: Leonardo Yvens and Zac Burns +Created: 2021-05-04 +Stage: Candidate +Implementations: https://github.com/graphprotocol/graph-node/pull/2414 +--- + +## Abstract + +This GIP proposes that the computational cost of a handler be measured in units of gas and limited to a maximum cost. If a handler attempts to consume more gas than the maximum, the subgraph will fail with a deterministic error. This is necessary so that it is possible to prove that a subgraph cannot be indexed past the block which triggers such a handler. + +## Motivation + +Subgraph handlers that run for too long are an issue for operators, since they excessively consume resources, and also a issue for protocol security since they make the subgraph impossible to sync in a reasonable time, if it can be synced at all. Graph Node currently implements a timeout for handlers, which is useful to prevent excessive resource consumption, however timeouts are not deterministic. For protocol security, it is necessary to deterministically measure the cost of a handler. This GIP proposes gas costing for handlers, similar to the gas costing for transactions that exists in blockchain protocols. + +## Detailed Specification + +Gas is defined as the unit of execution cost. As a reference, one unit of gas should correspond to 0.1ns of execution time, making for 10 billion gas in a second. Close correlation with execution time is not a primary goal of gas costing, at least not in this first iteration. Still gas needs to have some correlation to execution time in order to be useful. + +The maximum gas cost for a handler should be sufficient for any reasonable subgraph, and be limited by the cost a Fisherman or Arbitrator would be willing to incur in order to check an attestation. The proposed limit is 1000 seconds of execution time, which corresponds to 10 trillion gas. + +The costs incurred by executing a handler can be categorized as either WASM execution costs or host function execution costs. WASM execution costs must be measured by injecting a callback at the block, see the documentation for this technique in the [pwasm-utils crate](https://docs.rs/pwasm-utils/0.17.1/pwasm_utils/fn.inject_gas_counter.html). For the gas costing of individual WASM instructions, refer to the implementation in [this commit](https://github.com/graphprotocol/graph-node/blob/30720a8f1fb074b71fe76729d4e0dc4ba3c3e955/runtime/wasm/src/gas_rules.rs). + +The cost of most host functions is calculated as a base cost added to a cost per byte of input. To +account for host functions that have non-linear complexity, the input bytes are first combined +through a complexity function and then multiplied by the cost per byte. To calculate the size of the +input bytes, a `gas_size_of` function exists for each data type that is an input to a host +function, see implementations of `GasSizeOf` in the reference implementation for details. Note that +`gas_size_of` is an approximation of the true in-memory size of the data, the actual size may vary +between platforms and implementations but still this approximation should be sufficient for gas +costing purposes. + +#### Constants + + +##### Table 1: + +| Constant | Value (in gas) | Description | +| ---------------------- | ------------------------------------ | ------------------------------------------------------------ | +| GAS_PER_SECOND | 10_000_000_000 | Relates time and gas units. | +| MAX_GAS_PER_HANDLER | 3600 * GAS_PER_SECOND | The limit is one hour worth of gas. | +| HOST_EXPORT_GAS | 10_000 | Cost of the callback that measures gas. | +| DEFAULT_BASE_COST | 100_000 | Base cost of most host functions. | +| DEFAULT_GAS_PER_BYTE | 1_000 | Equivalent to 10MB/s, so 36GB in the 1 hour limit. | +| BIG_MATH_GAS_PER_BYTE | 100 | Equivalent to 100MB/s. | +| ETHEREUM_CALL | 25_000_000_000 | Gas cost of a contract call. Allows for 400 contract calls. | +| CREATE_DATA_SOURCE | MAX_GAS_PER_HANDLER / 100_000 | Cost of `dataSource.create`. | +| LOG_GAS | MAX_GAS_PER_HANDLER / 100_000 | Base cost of `log.log`. | +| STORE_SET_BASE_COST | MAX_GAS_PER_HANDLER / 250_000 | Base cost of `store.set`. | +| STORE_SET_GAS_PER_BYTE | MAX_GAS_PER_HANDLER / 1_000_000_000 | Allows 1GB of entity data for `store.set`. | +| STORE_GET_BASE_COST | MAX_GAS_PER_HANDLER / 10_000_000 | Base cost of `store.get`. | +| STORE_GET_GAS_PER_BYTE | MAX_GAS_PER_HANDLER / 1_000_000_000 | Allows 10GB of entity data for `store.get`. | + +#### Costs for host functions + +This section specifies the costs attributed to each host function. + +Most host functions have linear complexity on a single parameter `N` and are costed as: + +```` +DEFAULT_BASE_COST + gas_size_of(N)*DEFAULT_GAS_PER_BYTE +```` + +Host functions that differ from this are listed in the table below: + +| Host function | Cost | +| --------------------------- | ------------------------------------------------------------ | +| abort | DEFAULT_BASE_COST | +| store.set(key, data) | STORE_SET_BASE_COST + STORE_SET_GAS_PER_BYTE * (gas_size_of(key) + gas_size_of(data)) | +| store.remove(key) | STORE_SET_BASE_COST + STORE_SET_GAS_PER_BYTE * gas_size_of(key) | +| store.get(key) -> data | STORE_GET_BASE_COST + STORE_GET_GAS_PER_BYTE * (gas_size_of(key) + gas_size_of(data)) | +| ethereum.call | ETHEREUM_CALL | +| dataSource.create | CREATE_DATA_SOURCE | +| log.log(m) | LOG_GAS + DEFAULT_GAS_PER_BYTE * gas_size_of(m) | +| bigInt.plus(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * max(gas_size_of(x), gas_size_of(y)) | +| bigInt.minus(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * max(gas_size_of(x), gas_size_of(y)) | +| bigInt.times(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * gas_size_of(x) * gas_size_of(y) | +| bigInt.divided_by(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * gas_size_of(x) * gas_size_of(y) | +| bigInt.mod(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * gas_size_of(x) * gas_size_of(y) | +| bigInt.pow(x, exp) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * gas_size_of(x) ^ exp | +| bigInt.bit_or(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * max(gas_size_of(x), gas_size_of(y)) | +| bigInt.bit_and(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * min(gas_size_of(x), gas_size_of(y)) | +| bigDecimal.plus(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * (gas_size_of(x) + gas_size_of(y)) | +| bigDecimal.minus(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * (gas_size_of(x) + gas_size_of(y)) | +| bigDecimal.times(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * gas_size_of(x) * gas_size_of(y) | +| bigDecimal.divided_by(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * gas_size_of(x) * gas_size_of(y) | +| bigDecimal.equals(x, y) | DEFAULT_BASE_COST + BIG_MATH_GAS_PER_BYTE * min(gas_size_of(x), gas_size_of(y)) | \ No newline at end of file diff --git a/GIP-Test/0006-gip-withdraw-helper.md b/GIP-Test/0006-gip-withdraw-helper.md new file mode 100644 index 0000000..71b1943 --- /dev/null +++ b/GIP-Test/0006-gip-withdraw-helper.md @@ -0,0 +1,104 @@ +--- +GIP: "0006" +Title: Withdraw helper contract to support collecting funds from Vector channels. +Authors: Ariel Barmat +Created: 2021-05-21 +Stage: Candidate +Implementations: https://github.com/graphprotocol/contracts/pull/454 +--- + +# Abstract + +Consumers in the Graph Network set up state channels for sending query fees to Indexers using Vector. [Vector](https://github.com/connext/vector) is an ultra-simple, flexible state channel protocol implementation. This proposal introduces a contract required to encode custom logic in the process of withdrawing query fees from Vector state channels. The **GRTWithdrawHelper** contract receives query fees from a Vector state channel, then it forwards them to the **Staking** contract where they are distributed according to the protocol economics. + +# Motivation + +Vector state channels are contracts that hold funds used as collateral for consumer-to-indexer payments. When an Indexer closes an allocation, both parties sign a withdrawal commitment to transfer the query fees accumulated for the allocation back to the network. Vector, by default, supports plain transfers from the channel to any destination defined in the withdrawal commitment. In the case of the Graph Network, all funds should be collected by calling the collect() function in the Staking contract. For the purpose of calling custom logic when transferring from channels, Vector supports using a **WithdrawHelper** contract. In this proposal, we describe the implementation of such a contract for connecting a Vector withdrawal transfer to The Graph smart contracts. + +# Detailed Specification + +We created a contract called **GRTWithdrawHelper** that extends the base **WithdrawHelper** provided by the Vector framework. This contract has a function `execute(WithdrawData _wd, uint256 _actualAmount) external` that is called in the same transaction when the indexer withdraw funds from the channel. The execute function evaluates the validity of the `WithdrawData` struct, and then approve and transfer the received funds to the **Staking** contract by calling `collect()`. + +### Transaction Flow + +```sequence +Channel->GRTWithdrawHelper: transfer(_actualAmount) +Channel->GRTWithdrawHelper: execute(WithdrawData _wd, uint256 _actualAmount) +GRTWithdrawHelper->Staking: approve(_actualAmount) +GRTWithdrawHelper->Staking: collect(_actualAmount, allocationID) +Staking->GRTWithdrawHelper: transferFrom(_actualAmount) +``` + +## Features + +- The **GRTWithdrawHelper** is deployed with the token address that will be used for transfers, in our case it is the GRT token address. It can't be changed at later stage but this can easily be solved deploying a new contract if needed. +- The contract expects to have available tokens in balance when `execute(WithdrawData calldata wd, uint256 actualAmount)` is called by the Channel. For that to happen the Channel needs to transfer funds to the **GRTWithdrawHelper** before execute is called. +- On execute, the **GRTWithdrawHelper** will approve the funds to send to the Staking contract and then execute `collect()` passing the proper allocationID and amount. The Staking contract will then pull the tokens. +- If the call to `collect()` fails, the funds are returned to a return address. This is important for Vector accounting of funds. + +## Requirements + +- Set the **GRTWithdrawHelper** contract as **AssetHolder** in the **Staking** contract. This way we allow the network contracts to accept funds from this contract. +- The cooperative withdrawal commitments must have the following data (indicated with arrows): + +``` +struct WithdrawData { + address channelAddress; + address assetId; // -> GRT token address + address payable recipient; // -> GRTWithdrawHelper contract address + uint256 amount; + uint256 nonce; + address callTo; // -> GRTWithdrawHelper contract address + bytes callData; // -> CollectData struct defined below +} +``` + +``` +struct CollectData { + address staking; // -> Staking contract address + address allocationID; // -> AllocationID to send collected funds + address returnAddress; // -> Address where to return funds in case of errors +} +``` + +The parties must agree and sign the WithdrawalCommitment with that information to achieve proper execution of the withdrawals. + +## Exceptions + +The following conditions will result in reverts of the execution: + +- `assetId` in `WithdrawData` not matching the token address configured in the **GRTWithdrawHelper**. +- **GRTWithdrawHelper** not having enough balance to transfer the **Staking** contract as stated in `actualAmount`. +- Setting an invalid staking contract address. + +Note: To avoid unexpected conditions in the Vector off-chain logic, it is encouraged that the parties verify the WithdrawData does not revert by doing a call/estimateGas. + +## Handled Exceptions + +These reverts are handled and will make the contract to return the funds to the return address. + +- Doing a withdrawal for a non-existing `allocationID` as defined in `CollectData` when calling `collect()`. + +## Assumptions + +- This contract does not know if a `WithdrawalCommitment` is correct, it just accepts a transfer and `WithdrawData` that defines a proper destination. + +- A properly formatted `WithdrawData` can be crafted to send outside funds to the **Staking** contract. In that case, the person will get those funds distributed with Curators + Delegators + taxed a protocol percentage. There is no rational economic incentive to do so. + +# Backwards Compatibility + +The contract in this proposal is new and peripheral to the rest of the network contracts. No breaking change is introduced. The Graph Council will need to set the **GRTWithdrawHelper** as an **AssetHolder** in the Staking contract. + +# Validation + +## Audits + +An audit of the changes described in this document was performed by ChainSafe (see [`assets/gip-0006/TheGraph-GRTWithdrawHelper-Audit.pdf`](../assets/gip-0006/TheGraph-GRTWithdrawHelper-Audit.pdf)). + +## Testnet + +The implementation was deployed and tested on Rinkeby testnet. https://rinkeby.etherscan.io/address/0xe5fa88135c992a385aaa1c65a0c1b8ff3fde1fd4 + +# Copyright Waiver + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). \ No newline at end of file diff --git a/GIP-Test/0028-subsidized-query-settlement.md b/GIP-Test/0028-subsidized-query-settlement.md new file mode 100644 index 0000000..facacf2 --- /dev/null +++ b/GIP-Test/0028-subsidized-query-settlement.md @@ -0,0 +1,113 @@ +--- +GIP: "0028" +Title: Subsidized Query Settlement +Authors: Ariel Barmat +Created: 2022-03-24 +Stage: Candidate +Category: Protocol Logic +Implementations: https://github.com/graphprotocol/contracts/pull/557 +--- + +# [GIP-0028] Subsidized Query Settlement + +# Abstract + +This proposal extends the indexer reward subsidy to take into account both indexing subgraphs as well as query fee collection to ensure we close the loop of the lifecycle of an allocation. + +# Motivation + +The network provides incentives to bootstrap the operation of indexers for new subgraphs which may not have pre-existing demand to attract indexers. The way it works is that each subgraph in the network is allotted a portion of the total network token issuance, based on the proportional amount of total curation signal that subgraph has. That amount, in turn, is divided between all the Indexers staked on that subgraph proportional to their amount of allocated stake. + +Today, indexers collect indexing rewards when they close the allocation to a subgraph. Consequently, the subsidy only covers half of the allocation lifecycle, and indexers might choose never to collect query fees due to changing network conditions or any other strategy. By placing all rewards distribution at the end of the allocation lifecycle, we align the incentives to ensure that the network subsidy includes all the actions that indexers need to perform to end up with a lively network. + +# Specification + +### Allocation Lifecycle + +The lifecycle of an Allocation starts by setting aside tokens from the global indexer stake into a subgraph and finishes when claiming the query fees from the rebate pool. + +**Current:** + +1. Create Allocation +2. Close Allocation -> **_distribute indexing rewards_** +3. Collect query fees through payment vouchers +4. Claim Allocation Rebate -> **_distribute query rewards_** + +**Proposed:** + +1. Create Allocation +2. Close Allocation +3. Collect query fees through payment vouchers +4. Claim Allocation Rebate -> **_distribute both indexing and query rewards_** + +### Store Indexing Rewards + +The main change is that indexing rewards are no longer distributed when indexers call `closeAllocation()` but it will be delayed to a later stage in the Allocation lifecycle. Rewards will still accrue until closeAllocation is called, but we will set aside those rewards when the indexer presents a non-zero POI. + +To support this, we will add an `indexingRewards` state variable in the **Allocation** struct and initialize it to zero when indexers create the allocation in `allocateFrom()`. + +Then, after at least one epoch has passed, the indexer will close the allocation, the protocol will use the `RewardsManager.takeRewards()` function to mint the accrued tokens to the Staking contract and get the amount of tokens created. We will store those tokens in the `Allocation.indexingRewards` state variable instead of sending them directly to the indexers & delegators. + +### Distributing Indexing Rewards + +The lifecycle of an Allocation ends when the `claim()` function is called. Claiming is a public function that anyone can call passing a valid AllocationID. + +Any token distribution, both for query (from rebate pool) and indexing rewards will happen when claiming the Allocation. + +Changes: + +- After claiming from the rebate pool, calculate the totalRewards by checking if **_query rewards_** received are greater than zero, if that is the case we will let the indexer get their **indexing rewards**. If **_query rewards_** are zero the indexer will forfeit indexing rewards. The formula is `totalRewards = queryRewards + indexingRewards` + +- Collect delegation rewards for both stored indexing rewards and queries coming out from the rebate pool. The delegation rewards are then sent into the delegation pool for the indexer that created the allocation. + +- Reset the `indexingRewards` state variable in the Allocation struct. + +- Send the indexer part of rewards to the indexer (both indexing + query rewards). The formula is `indexerRewards = totalRewards - delegationRewards`. + +### Additional interface changes + +The function to claim an allocation currently has an extra parameter to decide whether to restake the rewards or not. However, this is not necessary and in fact can be troublesome as anyone can call the public `claim()` function. + +We propose to use the internal `_sendRewards()` function along with the existing **Rewards Destination** feature to decide whether to restake or not. + +``` +# old interface +function claim(address _allocationID, bool _restake) external override; + +# new interface +function claim(address _allocationID) external override; +``` + +### Gas Considerations + +- We are using 20,000 more gas to store the new `indexingRewards` variable in the Allocation struct during `closeAllocation()`, some of that is recovered when the variable is cleaned up during `claim()`. +- Doing all token transfers during `claim()` instead of having it split into two different calls is more efficient, saving around 10,000 gas. + +### Special Considerations + +- Indexers that haven't collected any query fees are not elegible for indexing rewards. Those rewards stored during close allocation will be burned after claiming. +- Claiming rebates requires waiting a seven-day period to ensure that all funds are collected properly from payment channnels, this has the side effect of delaying indexing rewards to when all the conditions are met. + +# Implementation + +See [@graphprotocol/contracts#557](https://github.com/graphprotocol/contracts/pull/557) + +# Backwards Compatibility + +If we remove the unused `_stake` from the `claim()` function that would require any agent using the contracts to update. + +Once the upgrade to the new mechanism happens, all of the Allocation structs for created allocations will have the `indexingRewards` variable set to zero, which is convenient as we don't need to initialize them. Whenever indexers close any of those allocations the protocol will calculate the accrued rewards and fill that variable for use during `claim()`. + +# Validation + +### Audits + +The implementation is not yet audited. + +### Testnet + +The implementation has not yet been deployed to Testnet. + +# Copyright Waiver + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). \ No newline at end of file diff --git a/GIP-Test/003-gip-rewards-nosignal.md b/GIP-Test/003-gip-rewards-nosignal.md new file mode 100644 index 0000000..2c623cd --- /dev/null +++ b/GIP-Test/003-gip-rewards-nosignal.md @@ -0,0 +1,63 @@ +--- +GIP: "0003" +Title: Fix accumulated subgraph rewards being reset to zero +Authors: Ariel Barmat +Created: 2021-03-01 +Stage: Candidate +Discussions-To: https://forum.thegraph.com/t/accumulated-subgraph-rewards-reset-to-zero-on-edge-case/1537 +Implementations: https://github.com/graphprotocol/contracts/pull/452 +--- + +# Abstract + +This proposal resolves an issue that prevents indexing rewards from being collected in certain cases. The RewardsManager contract's main responsibility is to calculate rewards accrued for allocations based on the total token supply, tokens signaled on subgraphs, accumulated rewards on a subgraph and allocations. In a specific edge case, calculation could fail and block closing an allocation in a way that collects rewards. + +# Motivation + +There is a particular edge case that happens when there are running allocations on a subgraph deployment ID, the total amount of curation signal is removed back to zero and `closeAllocation()` is called by an Indexer. If this happens, the Indexer won't be able to close that allocation unless it is opting for no rewards by sending POI=0x0. The reason for this is that `getAccRewardsForSubgraph()` is reseting the accumulated rewards for a subgraph when the curation pool is empty. The consequence is that `subgraph.accRewardsForSubgraphSnapshot` can be larger than `subgraph.accRewardsForSubgraph` leading to a revert when calculating new accrued rewards. + +# Detailed Specification + +## Proposed Changes + +By removing the early check within `getAccRewardsForSubgraph()` that returns zero for the subgraph accumulated rewards when the signal is zero we will always return the accumulated amount, as it should be an ever increasing number. + +#### 1) Update the calculation for accumulated rewards of a subgraph. + +`function getAccRewardsForSubgraph(bytes32 _subgraphDeploymentID)` + +**Remove early exit returning zero when no signalled tokens:** +``` +uint256 subgraphSignalledTokens = curation().getCurationPoolTokens(_subgraphDeploymentID); +if (subgraphSignalledTokens == 0) { + return 0; +} +uint256 newRewards = newRewardsPerSignal.mul(subgraphSignalledTokens).div(TOKEN_DECIMALS); +return subgraph.accRewardsForSubgraph.add(newRewards); +``` + +Removing that check will result in always keeping the accumulated rewards for the subgraph. In the cases when subgraphSignalledTokens = 0, no new rewards will be accrued per the below calculations. + +**New implementation:** +``` +uint256 subgraphSignalledTokens = curation().getCurationPoolTokens(_subgraphDeploymentID); +uint256 newRewards = newRewardsPerSignal.mul(subgraphSignalledTokens).div(TOKEN_DECIMALS); +return subgraph.accRewardsForSubgraph.add(newRewards); +``` + +# Validation + +## Tests + +Added a test to reproduce the condition that fails to pass with the older implementation and works with the new one. + +https://github.com/graphprotocol/contracts/pull/452/files#diff-9a2435f8c93e0efee7cee8b77fdc92daa626cc3058366c9e021d1dce33dcc389R687 + + +## Audits + +An audit for this change is pending. + +# Copyright Waiver + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). \ No newline at end of file diff --git a/GIP-Test/0058-Replacing-Bonding-Curves-with-Indexing-Fees.md b/GIP-Test/0058-Replacing-Bonding-Curves-with-Indexing-Fees.md new file mode 100644 index 0000000..a1cf3ab --- /dev/null +++ b/GIP-Test/0058-Replacing-Bonding-Curves-with-Indexing-Fees.md @@ -0,0 +1,274 @@ +--- +GIP: "0058" +Title: Replacing Bonding Curves with Indexing Fees +Authors: Justin Grana, Howard Heaton +Created: 2023-08-08 +Updated: 2023-12-01 +Stage: Draft +Discussions-To: https://forum.thegraph.com/t/gip-0058-replacing-bonding-curves-with-indexing-fees/4425 +Category: "Protocol Logic" +--- + + + +Amended 2023-12-01. Original document unchanged except for “***(see [amendment](#amendment-2023-12-01))***” notations. New “[Amendment 2023-12-01](#amendment-2023-12-01)” section added. + +## Abstract + +A key step to creating unstoppable dapps is creating a system whereby Indexers and data consumers can coordinate to get subgraphs indexed. The current coordination mechanism, curation with bonding curves, is rife with inefficiencies. This document provides a detailed description and rationale for indexing fees via a new mechanism that would replace ***(see [amendment](#amendment-2023-12-01))*** curation to enable efficient coordination among Indexers and developers. With indexing fees, Indexers publicly post their price *per unit of subgraph gas* and consumers (or someone acting on behalf of the consumer), choose Indexers to perform indexing at the posted price. Security of the payments is ensured through on-chain collateralization contracts. While consumers can select Indexers directly, the mechanism also supports automated Indexer selection for scalability. + +## Executive Summary + +**Problem**: The Graph’s current curation mechanism is inefficient for several reasons. + + 1. Subgraphs cost varying amounts to index, making it difficult for Indexers to predict GRT rewards. + 2. Indexer payments are uncertain and volatile due to staking decisions in disparate parts of the network. + 3. There is a noisy relationship between curation signal and the quality of indexing service for a subgraph. + +**Proposed Solution**: Indexing fees are collected via a new mechanism that proceeds as follows. + +1. Indexers publicly post a price per unit of work to index a subgraph. +2. A consumer chooses which Indexers it would like to contract to index the subgraph. (More likely, an algorithm acting on behalf of the consumer can choose for them.) +3. The selected Indexers index the subgraph and then submit a POI that verifiably and deterministically states how many units of work it took to index the subgraph. +4. The consumer pays the Indexer according to the posted price and the units of work. + +**Benefits**: In addition to its simplicity, indexing fees yield the following favorable properties. + +1. Indexer compensation depends directly on how resource intensive it is to index a subgraph. +2. Indexer revenue per unit of work is perfectly predictable and is not volatile. +3. The relation between consumer payment (per unit of work) and number of Indexers is perfectly predictable. This makes the relationship between consumer payment and quality of service more transparent. + +Although indexing fees represent a radical departure from curation, it adds clarity and predictability to the protocol by limiting uncertainty. This will enable an efficient and scalable marketplace for indexing services. + +## Motivation + +Decentralized and permission-less data services provide the robustness and censorship resistance required to build unstoppable apps. However, the pseudo-anonymous nature of a permission-less decentralized environment presents new challenges, especially in regard to novel attacks such as sybil attacks (i.e. where participants are able to improve their outcomes by splitting their identities) and ensuring parties are able to establish sufficient trust. To address these concerns, decentralized protocols (including The Graph), aim to implement mechanisms to deter behavior that is not in the spirit of the protocol while maintaining an efficient market. The Graph’s current curation system is meant to serve that purpose. + +Today, coordination for indexing uses bonding curves. Each subgraph has a bonding curve on which curation shares are minted when a user deposits GRT. Each subgraph’s bonding curve is unique. Indexers are incentivized to index subgraphs with high signal (i.e. much GRT) so that they can collect indexing rewards. Under this mechanism, the rewards Indexers get on a particular subgraph fluctuate depending on the signal of other subgraphs and the staking decisions of other Indexers; this leads to volatility and uncertainty for Indexers and unpredictable quality of service for consumers. + +## Proposed Solution + +To address the shortcomings of curation, we propose using indexing fees whereby Indexers publicly post a price per unit of work for which they are willing to index subgraphs. Consumers are able to choose Indexers to contract to index their subgraphs; often, we expect an algorithm with automate this process on behalf of consumers. The selected Indexers index the subgraph and submit a POI that verifiably and deterministically states how many units of work it took to index the subgraph. The consumer pays the Indexer according to the posted price and the units of work. + +A key departure from curation is that, with indexing fees, the incentive to index does not come from indexing rewards, but rather from a direct transfer of GRT from a consumer to an Indexer. Today, Indexers are primarily compensated for indexing activity via indexing rewards. With Horizon, indexing rewards will be used to subsidize protocol security in the form of payments to collateral ***(see [amendment](#amendment-2023-12-01))***. Therefore, indexing rewards will no longer be used to incentivize indexing activities. Instead, the incentive to index is provided through payments that flow directly from the consumer to the Indexer. + +To fully document and explain the proposed mechanism, the remainder of this document proceeds as follows. First we identify key terms and then desired properties for a mechanism. The main details of indexing fees follows with an illustration on how they meets the desiderata. The next section is dedicated to an example automated Indexer selection algorithm. Frequently asked questions are addressed in following section, which are followed by the conclusion. An appendix (to be linked) includes an important analysis of subgraph gas and how it relates to computational resources required to index a subgraph. + +## Terms and Background + +**Indexers**: Indexers are node operators in The Graph Network that stake Graph Tokens (GRT) in order to provide indexing and query processing services. Indexers earn query fees and indexing rewards for their services. + +**Consumers**: The core users of data services are referred to as consumers. Specifically, the consumer is the one that receives a benefit (and thus is willing to pay) for data services. In some cases, this may be the developer. In other cases, a consumer may want to leverage the subgraph of another developer. In general, the consumer is the party that interacts with Gateways/Indexers for the indexing services. + +**Subgraph Gas**: There are two notions of subgraph gas: sync gas and storage gas. Storage gas is simply the size of entities stored in the database, which correlates with the size of a subgraph on disk. Sync gas captures the amount of compute and disk operations required to sync a subgraph, write it to disk, and perform network operations (e.g. JSON RPC). Both storage gas and sync gas are verifiable quantities. A detailed relationship between subgraph gas and compute resources is given in Appendix A. + +**Horizon**: ***(see [amendment](#amendment-2023-12-01).)*** [Graph Horizon](https://forum.thegraph.com/t/graph-horizon-explained-a-proposal-for-the-evolution-of-the-protocol/5169) introduces a general purpose scheme for securing bilateral transactions in The Graph ecosystem. The core aim is to establish trust via on-chain means; to do this, Horizon utilizes an escrow-like service with the capability of penalizing bad behavior (via burning tokens of the party at fault). Horizon “establishes trust” via economic incentives from smart contracts and allows “freely evolving services” due to a fundamental shift in how governance operates. With respect to the scope of this document, the only essential information to know about Horizon is that it enables on-chain agreements to be formed for coordinating procurement of services and arbitration. The details of Graph Horizon will be defined in a separate GIP. + +**Agreement**: An agreement is used to identify the terms of service between an Indexer and a consumer. For example, a caricature agreement might be “Indexer will index a subgraph within 2 days for 1 GRT per unit of subgraph gas.” Agreements are often formalized via smart contracts; however, an off-chain agreement may also be formed between a consumer and a Gateway acting on the consumer’s behalf. + +## Desiderata + +The desiderata — desired properties of a new mechanism — reflect addressing the issues with the current curation mechanism. Specifically, once agreements are reached, the Indexer should have high certainty in regard to its profit and revenue, and that certainty should not wane with time. On the consumer side, once agreements are reached, their price paid and received quality of service should both be predictable and steady over time. Reducing these uncertainties yields the desirable higher level properties simplicity and efficiency. Of course, this must also be done in a sybil resistant manner to preserve the robustness and censorship resistance benefits of a decentralized platform. The remainder of this section details each of the desiderata. + +**High Certainty – Indexer Revenue**: +Indexers should be able to forecast how much revenue they will achieve by indexing a subgraph. Uncertainty may arise when indexing revenue is influenced by actions of other participants (e.g. curators adjusting signal with bonding curves, Indexers unexpectedly allocating a large amount of stake on a subgraph). + +**High Certainty – Indexer Profitability**: +Indexers cannot know the total cost to index subgraphs beforehand. To ensure Indexers are profitable, it follows that there must be a way to compensate them proportional to their incurred indexing costs. + +**Low Price Volatility**: +Even if an Indexer knows the price today, how much payments fluctuate over time for indexing services can be complicated to reason about. This similarly affects developers when making plans to use data services for their dapps. Cognitive load and pricing efficiency can typically be improved by reducing volatility. + +**Low QoS Uncertainty**: +The consumer can easily see how much it needs to pay to achieve a given quality of service. Furthermore, the consumer can have the flexibility of obtaining a high quality of service by selecting several competent Indexers or selecting fewer but more effective Indexers. + +**Sybil Resistant**: +A key property of blockchains is identities can be easily duplicated across several wallet addresses. To ensure developers have their dapps serviced by multiple Indexers (e.g. for robustness to failure of any single Indexer), developers must be able to ensure, with reasonable confidence, they are able to attract +distinct Indexers to index their subgraphs. + +**Efficiency**: +The mechanism (approximately) maximizes the number of mutually beneficial transactions. + +**Simple**: +We aim to create a simple market. We consider this along two dimensions: actions and consequences are simple to understand (low cognitive overhead) and tasks are simple to perform (efficient workflows). + +## Proposal + +### Sequence of Events and Details + +Under the indexing fees mechanism, the order of events can be summarized as follows. We then provide a detailed description of each step. + +**[Step 1](#step-1-indexers-post-prices)**: Indexers post prices per unit of subgraph gas with a sync-speed warranty per unit of subgraph gas. + +**[Step 2](#step-2-indexer-selection)**: Consumer (or agent acting on behalf of consumer) selects Indexers. + +**[Step 3](#step-3-consumer-and-indexer-enter-agreement)**: Consumer and Indexers both enter an on-chain agreement that specifies: + +1. Required Indexer collateral. +2. Terms of agreement (price per unit of subgraph gas, correctness warranties, slashable events, arbitrating party, collateral thawing period, etc.). The contract is enforced when the consumer executes a transaction to accept the terms and deposits funds to pay for the indexing services and the Indexer deposits the time-locked collateral. + +**[Step 4](#step-4-indexing-occurs)**: Indexer indexes the subgraph. + +**[Step 5](#step-5-indexer-posts-poi)**: Indexer posts POI. + +**[Step 6](#step-6-enter-dispute-period)**: Dispute period begins. + +1. Within the thawing period defined in the agreement the consumer can choose to raise a dispute that, if the arbitrator determined to be valid, would result in the Indexer being slashed according to the terms of agreement. +2. The Indexer cannot withdraw its payment nor its collateral until the dispute period ends. + +**[Step 7](#step-7-indexer-retrieves-funds)**: Indexer retrieves any funds (collateral and payment) after the dispute period. + +**[Step 8](#step-8-continual-syncing-and-payment)**: Indexer periodically reports ongoing sync gas and ongoing storage gas and withdraws funds from the contract accordingly. + +**[Step 9](#step-9-indexer-retrieves-remaining-funds)**: When the contract is over, the Indexer can withdraw collateral for the sync speed warranty. + +#### Step 1: Indexers Post Prices + +Simply stated, Indexers will submit prices to the Gateway that it charges to index a subgraph. This is similar to the current process for the Indexer selection algorithm for choosing queries. Importantly, these prices are in terms of metered computational units of work and also contain a sync speed warranty. The motivation for metered pricing is to account for subgraph heterogeneity and to incentivize dapp developers to optimize their subgraphs. One of the main challenges with previous versions of curation was that payment to Indexers did not depend on the amount of work (compute, storage, etc) required to index a subgraph. With indexing fees, Indexer pricing is a function of computational resources required to index a subgraph. As an initial protocol design choice, prices will be in terms of “subgraph gas,” an approximation of computational resources needed to index a subgraph. See section A for more details on the relationship between subgraph gas compute costs. The sync-speed warranty provides a channel for Indexers to credibly differentiate the quality of their hardware and thus earn more revenue by deploying better hardware. By posting sync-speed warranties per unit of compute (subgraph gas), Indexers with better hardware can post faster sync-speed warranties and thus charge a higher price. For more details on how different hardware configurations impact sync-time, see Appendix A. Furthermore, Indexers can post multiple prices with multiple sync-speed warranties if they want to offer different levels of service at different prices. + +#### Step 2: Indexer Selection + +Indexer selection can happen in two ways 1) directly by the consumer or 2) through an automated Indexer selection algorithm that optimizes for consumers preferences (e.g. quality of service). An automatic Indexer selection algorithm is discussed in detail below. This section focuses on the case where consumers choose their own Indexers. Under manual Indexer selection, a consumer is presented with a menu of Indexers and relevant features (price per unit of gas, geographical location, quality of service statistics, etc.). Note quality of service stats are trusted data (in the case of Gateway usage) and untrusted if manually collected. The consumer can then select one or more Indexers based on its preference to index its subgraph. Importantly, at this point neither the consumer nor the Indexer have entered into an agreement through smart contract. The selection step notifies Indexers that there is a smart contract that if they choose to enter would make them eligible for a payment upon successfully indexing of a subgraph. + +#### Step 3: Consumer and Indexer Enter Agreement + +After the Indexer is notified, the consumer and the Indexer must both opt into the agreement. The consumer opts into the agreement by depositing GRT into a smart contract that would then be transferred to the Indexer upon successful indexing. The Indexer deposits collateral into the smart contract that would be slashed if indexing did not happen according to the agreements parameters. The smart contract specifies several parameters of the agreement including the price per unit of subgraph gas, the required amount of Indexer collateral, the maximum amount of gas the Indexer should spend indexing the subgraph, slashable events and parameters (how much an Indexer should be slashed for not meeting their sync-speed warranty, for example), arbitrating entity and collateral thawing period. + +#### Step 4: Indexing Occurs + +This step is identical to how the Graph functions today. + +#### Step 5: Indexer Posts POI + +This step is identical to how the Graph functions today + +#### Step 6: Enter Dispute Period + +After the Indexer submits a POI the dispute period begins. Within this period, the consumer has the option to submit a dispute to the arbitrator (specified in step 3). If the arbitrator determines that the terms in step 3 were violated by the Indexer, the Indexer is slashed according to the agreement. The portion of slashed GRT that is returned to the consumer versus the portion that is burned is specified in step 3. During the dispute period, both the consumer’s funds and the Indexer’s funds are frozen in the contract. + +#### Step 7: Indexer Retrieves Funds + +After the dispute period, the Indexer retrieves its entitled funds (both collateral and payment) that remain after any slashing events. Of course, because the collateral and slashing provides adequate incentive for Indexers to meet the terms in step 3, step 7 will often consist of the Indexer withdrawing its full collateral and the consumer’s payment for indexing services. + +#### Step 8: Continual Syncing and Payment + +Since subgraphs undergo continual syncing, the Indexers post the incremental sync gas and disk space required to fully sync the subgraph. After a dispute period and assuming no slashable events, the Indexer can withdraw additional funds as compensation for the additional syncing resources. + +#### Step 9: Indexer Retrieves Remaining Funds + +Although a correctness warranty can be reclaimed at the end of a dispute period (e.g. whenever a PoI is submitted + fixed time,) the sync speed warranty can only be reclaimed at the end of the agreement. A PoI may be required periodically to checkpoint syncing, but submitting a PoI and collecting payment for the subgraph synced so far does not obviate the need for the sync speed warranty to continue for the subgraph yet to sync past that checkpoint. + +## Desiderata Revisited + +Given the full description of indexing fees in the previous section, it is now possible to show how the new mechanism meets the desiderata: + +**High Certainty – Indexer Revenue**: +The Indexer’s price (per unit of “work”) is perfectly predictable. It is simply the posted price. This is not affected by the behavior of other Indexers. + +**High Certainty – Indexer Profitability**: +Because Indexers post prices per unit of gas, they can charge more for more expensive subgraphs, thus reducing profitability uncertainty. + +**Low Price Volatility**: +The price an Indexer receives is fixed and does not change over time as a result of other Indexers’ staking decisions. + +**Low QoS Uncertainty**: +Each Indexer’s price and sync-speed warranty are publicly posted. Therefore, if a consumer knows the size of its subgraph, it can compute exactly how much it will cost for each additional Indexer. + +**Sybil Resistant**: +Because consumers select Indexers, they will select Indexers that they know to be distinct. Therefore, Indexers have no incentive to split their identity. + +**Efficiency**: +It is documented that in a general class of large markets, posted prices are the most efficient pricing mechanism. + +**Simple**: +The new indexing fees mechanism mirrors standard market mechanisms: Sellers post a price and consumers select what to buy based on product quality and cost. + +## Automated Indexer Selection + +Although consumers may manually choose their own Indexers, those seeking a simpler experience could use an automatic Indexer selection algorithm. Such an algorithm may abstract away the need for consumers to screen individual Indexers. Instead, after a consumer provides their desired a) number of Indexers and b) sync speed preferences (high, medium, low), an automated Indexer selection algorithm can report the price and select the Indexers on behalf of the consumer. + +An example algorithm works as follows. First, Indexers are sorted based on their sync speed warranties and then categorized into high, medium or low sync speed (based on pre-defined thresholds). Indexer’s that are dominated, i.e. have a higher price for a lower sync speed warranty than some other Indexer — are removed from selection. The number of requested Indexers in the tier are selected sequentially. + +The first Indexer within the chosen sync speed tier is selected at uniform random. To provide robustness and guard against sybil attacks, the subsequent Indexers are selected at uniform random as long as their quality of service is not sufficiently correlated with previously selected Indexers. In other words, robustness and redundancy relies on Indexer performance being uncorrelated; this ensures, if one Indexer fails, there is no loss in service. By selecting Indexers that are uncorrelated, the automated algorithm ensures robustness. Furthermore, sybil identities are also likely to have a correlated quality of service, and so selecting Indexers with low correlation protects against sybil attacks. + +This example algorithm is admittedly crude and imprecise. However, it already gives consumers greater control over their quality of service than they have today and, thus, makes the protocol more efficient. Nevertheless, there is nothing preventing other agents from developing more sophisticated Indexer selection algorithms, incorporating both on-chain and off-chain data to improve consumer experience. + +## Frequently Asked Questions + +In this section, we address some of the frequently asked question that arise when replacing curation with indexing fees. + +**How do Indexers know which subgraphs to index?** + +They can be notified (e.g. via Discord, Telegram, POI radio, email) that they have been selected to index a subgraph at their posted price. We leave it open to the community to decide the best approach for notifications. + +**What is an Indexer’s incentive to index the subgraph if they are selected?** + +They will receive a payment equal to their posted price multiplied by the subgraph gas used to index. + +**Once selected, what actions must an Indexer take before making the indexing agreement valid?** + +An Indexer must opt-in to the agreement. This gives an Indexer the opportunity to verify the subgraph is available before accepting the indexing contract. + +**Can an Indexer post different prices for different subgraphs?** + +No, the price per unit of gas is network-wide. This is subject to change in the future. + +**Can an Indexer post a price based on a quantity other than subgraph gas?** + +Initially no, though this is subject to change. + +**Does this impact query fees and how Indexers are selected to serve queries?** + +No. Indexing fees only pertain to indexing and not serving queries. + +**Can an Indexer that is not selected to index a subgraph still index the subgraph and compete for queries?** + +Yes! Indexing fees does not preclude any Indexer from indexing a subgraph, it only provides extra incentive. + +**How much gas will a subgraph consume?** + +This is impossible to know exactly before indexing. You can find some statistics for common subgraphs that relate subgraph properties to gas costs in Appendix A. + +## Summary + +Indexing fees are an improvement over the current curation mechanism. These add transparency and predictability which begets simplicity and efficiency for both consumers and Indexers. Together with Graph Horizon ***(see [amendment](#amendment-2023-12-01))***, indexing fees are general enough to allow for future services on the network and to scale, independent of issuance. + +## Detailed Specification + +The detailed specification of the smart contracts for this GIP will be added at a later time. + +## Copyright Waiver + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). + +## Amendment 2023-12-01 + +After considering community feedback, we are proposing an amendment to this GIP. We propose: + +1. having Indexing Fees exist ***alongside*** curation. +2. removing the dependency on “Graph Horizon” but only requiring a staking and dispute mechanism described in Step 3 and Step 6, above. + +**Importantly, this proposal now includes no change to issuance distribution.** + +### Indexing Fees Alongside Curation + +One of the original motivations for indexing fees was that consumers expressed frustration in predictably having their subgraph indexed with a stable quality of service. Adding indexing fees alongside current curation gives those customers an alternative channel to attract and incentivize indexers. However, since indexing fees will now live alongside current curation, the disruption to current consumers that are comfortable with curation will be minimal. For complex cases, consumers can use both current curation/signaling as well as indexing fees to incentive indexers with both issuance and direct payments. + +### Break from “Graph Horizon” + +Replacing curation with indexing fees required a new mechanism for issuance distribution. This was described as a part of “Graph Horizon.” However, now that the proposal is for indexing fees to exist alongside curation, the need to repurpose issuance is obviated. To reiterate, there will be ***no change to issuance and the distribution mechanism as part of this proposal***. + +The other main component of “Graph Horizon” was the collateralization mechanism exactly as described in Steps 3 and 6 above. While in this proposal, a collateral contract is used to ensure trusted indexing, the concept of collateralized transactions with dispute periods is far more powerful and can enable a wide array of services on The Graph network. General collateralized transactions is an avenue we will continue exploring and communicating with the community as we obtain insight. + +### So what about Graph Horizon? + +Briefly, Horizon had two main components 1) A general collateralization contract for secure transactions and 2) a change to the issuance mechanism. Currently, the plan is to continue exploring additional uses for collateralized transactions independently from improving the efficiency of the issuance mechanism. \ No newline at end of file diff --git a/GIP-Test/0065-subgraph-availability-manager.md b/GIP-Test/0065-subgraph-availability-manager.md new file mode 100644 index 0000000..7d47341 --- /dev/null +++ b/GIP-Test/0065-subgraph-availability-manager.md @@ -0,0 +1,125 @@ +--- +GIP: 0065 +Title: Subgraph Availability Manager +Authors: Miguel de Elias +Created: 2024-01-10 +Stage: Draft +Discussions-To: https://forum.thegraph.com/t/gip-0065-subgraph-availability-manager/5501 +Category: Protocol Interfaces +Implementations: https://github.com/graphprotocol/contracts/pull/882 +Audits: https://github.com/graphprotocol/contracts/blob/main/packages/contracts/audits/OpenZeppelin/2024-02-subgraph-availability-manager-and-minimum-allocation-duration-removal.pdf +--- + +# Abstract + +This GIP proposes increasing the decentralization in The Graph by introducing a new smart contract `SubgraphAvailabilityManager`. + +# Motivation + +The Subgraph Availability Oracle (SAO) plays a crucial role in The Graph's ecosystem. Its main function is to verify the availability and validity of subgraph files. Currently there is a single instance of the SAO in operation executing open-source code accessible at [The Graph's Subgraph Oracle repository](https://github.com/graphprotocol/subgraph-oracle). This SAO instance runs at five-minute intervals, performing a series of checks to assess the state of subgraph files. In instances where the SAO identifies unavailability or invalidity in a subgraph's files, it sends a transaction to the `RewardsManager` contract to deny indexing rewards for that subgraph. By default, subgraphs are presumed to be available, and typically, no initial action or vote by the SAO is necessary when a new subgraph is added to the network. + +The proposed change aims to enhance the decentralization of the protocol by introducing five separate oracles operators and requiring a minimum of three votes among them before any decisive action is taken. All oracles operators will be bound by the **Subgraph Availability Oracle Operator Charter** included in this GIP which outlines their roles, responsibilities, and operational guidelines. + +# High-Level Description + +The new smart contract `SubgraphAvailabilityManager` will be exclusively deployed to the Arbitrum One network and will function as a mediator between five distinct oracle operators and the `RewardsManager` contract. It will feature a vote aggregation system and enforce an execution threshold requiring three votes to alter the state of a subgraph deployment within the `RewardsManager` contract. Additionally, it will incorporate a `voteTimeLimit` timeframe to monitor current votes, after which they expire and become invalid. + +It's important to note that this system operates effectively only if a minimum of three oracles are honest and running. The system will be able to report subgraph availability whenever three or more of the oracle instances are running correctly. According to the Charter guidelines, all oracles are expected to operate in compliance with established standards. In the event that any oracles are found not to be performing up to these standards, measures will be taken to replace them, ensuring the system's integrity and continuous effective operation. + +The `SubgraphAvailabilityManager` contract will be governed by the council. While the contract itself will not be upgradeable, a new contract can be deployed, and the council can set the `RewardsManager` oracle address to the new deployment. The council will have the authority to add or remove oracles operators and update the `voteTimeLimit` timeframe. + +# Detailed Specification + +The number of oracle operators will be strictly enforced as five. Upon contract initialization, five valid accounts must be provided, and each oracle must maintain their index position, crucial for passing it as a parameter when casting votes. + +Vote aggregation will be separately tracked for positive or negative votes using the following data structures: + +```solidity +mapping(uint256 currentNonce => mapping(bytes32 subgraphDeploymentID => uint256[NUM_ORACLES] timestamps)) public lastDenyVote; +mapping(uint256 currentNonce => mapping(bytes32 subgraphDeploymentID => uint256[NUM_ORACLES] timestamps)) public lastAllowVote; +``` + +The initial mapping key corresponds to a public variable `currentNonce`. This nonce serves as a mechanism to invalidate all votes whenever a configuration is altered—be it updating the oracle operators list or modifying the `voteTimeLimit` parameter. The inner mapping stores the `subgraphDeploymentId` against the timestamp of each oracle's vote, using an array denoting the oracle index. + +The `executionThreshold` will be set to three during contract construction. Whenever an oracle casts a vote, the `SubgraphAvailabilityManager` contract will query the corresponding vote aggregation storage to verify if enough votes have been registered. Upon the third vote a transaction to `RewardsManager` is triggered to make the subgraph rewards state change. + +A `voteTimeLimit` parameter will be used to invalidate old votes, initially set to ten minutes. This duration is slightly longer than the current subgraph availability oracle configuration, which runs every five minutes, allowing time for oracles to cast their votes and accounting for potential delays in execution. This minimizes the need for frequent transactions. When comparing oracle votes, the `SubgraphAvailabilityManager` will use the `voteTimeLimit` to ensure the last recorded vote occurred within this timeframe. Votes older than the limit will not be considered valid, necessitating a new vote from the oracle if required. It's crucial to note that any adjustments to `voteTimeLimit` must always exceed the oracle's execution period to prevent the premature invalidation of votes before all oracles have participated. However, this timeframe should be kept as brief as possible, since an excessively long `voteTimeLimit` undermines its intended purpose. The council can modify this timeframe if desired but it must be set with consideration to the oracle’s iteration period to avoid repeated voting or expired votes. + +# Rationale and Alternatives + +The presented proposal serves as a temporary solution; we will continue our efforts to explore additional methods for further decentralization or even the complete elimination of the subgraph availability oracle. We believe this implementation provides a short-term fix that significantly improves decentralization compared to the previous 1-of-1 approach. + +--- + +*This section contains the body of the Subgraph Availability Oracle Operator Charter.* + +# Subgraph Availability Oracle Operator Charter + +## 1. Role of the Subgraph Availability Oracle. + +The primary role of the Subgraph Availability Oracle in The Graph Network is to ensure the availability and validity of subgraph manifests. Oracles are responsible for verifying subgraphs availability and conducting validity checks. + +Oracles operate as part of the `SubgraphAvailabilityManager` smart contract, contributing to the decentralized decision-making process by casting votes regarding the availability and validity of subgraph files. + +## 2. Appointment and Governance of Subgraph Availability Oracles Operators + +Oracle operators are appointed by The Graph Council, which retains the authority to add or remove oracle operators, ensuring compliance with the guidelines and responsibilities outlined in this charter. For this initial phase, the Oracle operators will be Core Developers Teams, although future extensions to include other community members are not discounted. + +The number of oracle operators is strictly limited to five. + +## 3. Responsibility and Integrity + +Operators must ensure their oracles remain operational, acknowledging that while occasional outages are part of normal operations, efforts to minimize these are crucial. Operators should communicate their infrastructure choices with one another to prevent risks associated with common dependencies, thereby enhancing the network's overall resilience. Active participation in a group chat to discuss operational challenges and coordinate on mitigating risks is required. Operators who are unresponsive or consistently underperform may be recommended for replacement by the technical advisory board or fellow operators to the Graph Council. + +## 4. Subgraph Availability Oracle Software and Versioning + +Subgraph Availability Oracles operators must use the software available at [The Graph's Subgraph Oracle repository](https://github.com/graphprotocol/subgraph-oracle). This ensures uniformity and accuracy in their assessments across the network. A specific commit or release version from this repository will be designated as the required version for all oracles. This version is subject to change over time as updates and improvements are made. + +Oracle operators are responsible for reaching consensus on the specific commit and configuration to be used. To facilitate this process, a group chat channel will be established for operators to communicate and reach consensus. Once agreed upon, operators must align their operations with these standards by posting a transaction with the details to a `DataEdge` smart contract within no more than three business days. A dedicated subgraph will track these declarations, flagging any deviations for prompt resolution to maintain network consistency. + +**Note:** In case a security fix is needed, operators should treat this with the highest priority. Oracles are required to be updated as soon as possible and no later than 24 hours after consensus. + +The transaction to the `DataEdge` should follow the format: + +```json +{ + "commit-hash": "XXXXX", + "config": { + "ipfs-concurrency": "4", + "ipfs-timeout": "10000", + "min-signal": "100", + "period": "300", + "grace_period": "0", + "supported-networks": "mainnet,gnosis,xdai,eip155:1,eip155:100,avalanche,eip155:43114,arbitrum-one,eip155:42161,celo,eip155:42220,fantom,eip155:250,matic,eip155:137", + "supported_data_source_kinds": "ethereum,ethereum/contract,file/ipfs,substreams,file/arweave", + "subgraph": "https://api.thegraph.com/subgraphs/name/graphprotocol/graph-network-arbitrum", + "subgraph_availability_manager_contract": "CONTRACT_ADDRESS", + "oracle_index": "ORACLE_INDEX", + } +} + +``` + +It is strongly recommended that Oracle operators diversify their sources for `IPFS_NODE_URL` and `RPC_URL` configurations. Utilizing varied sources for these critical components not only enhances the robustness and reliability of the network but also mitigates potential risks associated with single points of failure. + +## 5. Testnet + +In addition to their operations on Arbitrum One, all oracle operators are required to run a corresponding version on the Arbitrum Sepolia testnet. This setup is necessary to create a testing environment that closely mirrors the production environment, facilitating thorough testing and verification before deployment to production. Part of this requirement involves sending a transaction with the commit hash and configuration to the `DataEdge` smart contract on the testnet, following the format mentioned in the section above. + +## 6. Security + +Oracle operators are expected to follow best practices in security to protect their systems from unauthorized access and potential vulnerabilities. + +## 7. Conflict Resolution and Recourse + +Given that all oracle operators are mandated to run the same software version, uniformity in voting outcomes is expected. Discrepancies in oracle votes are therefore an area of concern and warrant investigation. In instances where voting differences occur or concerns are raised to the foundation or the core developers, the Technical Advisory Board and the Subgraph Availability Oracles operators will appoint an investigator to look into the issue. This investigation aims to determine whether the discrepancy was due to an honest failure or a result of malicious intent. The investigators decision will be final. + +If an oracle operator is found to be acting maliciously, the Graph Council will nominate a new operator to replace them. In scenarios where an oracle consistently deviates from the expected voting pattern due to honest errors, The Graph Council will evaluate its performance. Persistent non-compliance with the charter's guidelines, even if unintentional, may lead to the replacement of the oracle operator to maintain the integrity and reliability of the network. + +*This concludes the body of the Subgraph Availability Oracle Operator charter.* + +--- + +# Copyright Waiver + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). \ No newline at end of file diff --git a/GIP-Test/0073-Reserved.md b/GIP-Test/0073-Reserved.md new file mode 100644 index 0000000..aac14b8 --- /dev/null +++ b/GIP-Test/0073-Reserved.md @@ -0,0 +1,36 @@ +--- +GIP: "0073" +Title: Reserved +Authors: Rembrandt Kuipers +Created: 2024-TBD +Updated: 2024-TBD +Stage: Reserved +Discussions-To: TBD +Category: "Protocol Logic" +--- + +This GIP number has been reserved for a series of GIPs currently being drafted. + +## Abstract + +## Motivation + +## Prior Art + +## High-Level Description + +## Detailed Specification + +## Backward Compatibility + +## Dependencies + +## Risks and Security Considerations + +## Validation + +## Rationale and Alternatives + +## Copyright Waiver + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). \ No newline at end of file