Merge branch 'pyth-network:main' into stylus-sdk

README.md

@@ -35,7 +35,20 @@ and [examples](https://github.com/pyth-network/pyth-examples/tree/main/price_fee
 
 Fortuna is an off-chain service which can be used by [Entropy](https://pyth.network/entropy) providers.
 
-## Development
+## Local Development
+
+### Setup
+
+Please install the following tools in order to work in this repository:
+
+- [NVM](https://github.com/nvm-sh/nvm?tab=readme-ov-file#installing-and-updating) to manage your node version, then run `nvm use 20` to ensure you are using node version 20.
+- [Foundry](https://book.getfoundry.sh/getting-started/installation) in order to use `forge` for Ethereum contract development
+- [Solana CLI](https://solana.com/docs/intro/installation) for working with Solana programs.
+  - After installing, please run `solana keygen new` to generate a local private key.
+- [Anchor](https://www.anchor-lang.com/docs/installation) for developing Solana programs.
+- [Pre-commit](https://pre-commit.com/) is used to automatically format and lint the repository.
+  - After installing, please run `pre-commit install` in the root of the repo to configure the checks to run on each git commit.
+- [Rust](https://www.rust-lang.org/tools/install)
 
 ### Pull requests
 
@@ -45,14 +58,16 @@ to update the package versions following the [Semantic Versioning](https://semve
 
 ### Releases
 
-The repository has a CI workflow that will release javascript packages whose version number has changed.
-To perform a release, follow these steps:
+The repository has several CI workflows that automatically release new versions of the various components when a new Github release is published.
+Each component's workflow uses a specific tag format including the component name and version number (e.g., Fortuna uses the tag `fortuna-vX.Y.Z`).
+The general process for creating a new release is:
 
-1. Update the version number in the `package.json` file for the package(s) you wish to release. Please follow [Semantic Versioning](https://semver.org/) for package versions.
+1. Update the version number of the component in the repo, e.g., in `package.json` or `Cargo.toml` or wherever. Please follow [Semantic Versioning](https://semver.org/) for package versions.
 2. Submit a PR with the changes and merge them in to main.
-3. Create a new tag `pyth-js-v<number>` and push to github. You can simply increment the version number each time -- it doesn't affect any of the published information.
-4. Pushing the tag automatically triggers a CI workflow to publish the updated packages to NPM.
+3. Create a new release on github. Configure the release to create a new tag when published. Set the tag name and version for the component you wish to release -- see the [Releases](https://github.com/pyth-network/pyth-crosschain/releases) page to identify the relevant tag.
+4. Publish the release. This step will automatically trigger a Github Action to build the package and release it. This step will e.g., publish packages to NPM, or build and push docker images.
 
+Note that all javascript packages are released together using a tag of the form `pyth-js-v<number>`. (The `number` is arbitrary.)
 If you have a javascript package that shouldn't be published, simply add `"private": "true"` to the `package.json` file
 and it will be excluded from the publishing workflow. If you are creating a new public javascript package, you should add
 the following config option to `package.json`:
@@ -63,16 +78,6 @@ the following config option to `package.json`:
   },
 ```
 
-### pre-commit hooks
-
-pre-commit is a tool that checks and fixes simple issues (formatting, ...) before each commit. You can install it by following [their website](https://pre-commit.com/). In order to enable checks for this repo run `pre-commit install` from command-line in the root of this repo.
-
-The checks are also performed in the CI to ensure the code follows consistent formatting.
-
-### Tilt CI
-
-Integration tests run in Tilt (via the `tilt ci` command). The Tilt CI workflow requires approval from a member of the Pyth team. If you are a member, click on "Details" next to the "Workflow / ci-pyth-crosschain" check in a pull request, and then on the "Resume" button on the workflow page.
-
 ### Typescript Monorepo
 
 All of the typescript / javascript packages in this repository are part of a

apps/api-reference/jsx.d.ts

@@ -0,0 +1,17 @@
+/**
+ * This file only exists because in react 19, the JSX namespace was moved under
+ * the React export.  However, some libraries (e.g. react-markdown) still have
+ * some things typed as `JSX.<Something>`.  Until those libraries update to
+ * import the namespace correctly, we'll need this declaration file in place to
+ * expose JSX via the old global location.
+ */
+
+import type { JSX as Jsx } from "react/jsx-runtime";
+
+declare global {
+  namespace JSX {
+    type ElementClass = Jsx.ElementClass;
+    type Element = Jsx.Element;
+    type IntrinsicElements = Jsx.IntrinsicElements;
+  }
+}

apps/api-reference/next-env.d.ts

@@ -2,4 +2,4 @@
 /// <reference types="next/image-types/global" />
 
 // NOTE: This file should not be edited
-// see https://nextjs.org/docs/app/building-your-application/configuring/typescript for more information.
+// see https://nextjs.org/docs/app/api-reference/config/typescript for more information.

apps/api-reference/src/markdown-components.tsx

@@ -31,9 +31,6 @@ export const MARKDOWN_COMPONENTS = {
         </Code>
       );
     } else {
-      // @ts-expect-error react-markdown doesn't officially support react 19
-      // yet; there's no issues here in practice but the types don't currently
-      // unify
       return <pre {...props} />;
     }
   },

apps/fortuna/Cargo.toml

@@ -1,40 +1,43 @@
 [package]
-name    = "fortuna"
-version = "6.5.5"
+name = "fortuna"
+version = "6.7.1"
 edition = "2021"
 
 [dependencies]
-anyhow      = "1.0.75"
-axum        = { version = "0.6.20", features = ["json", "ws", "macros"] }
-axum-macros        = { version = "0.3.8" }
-base64             = { version = "0.21.0" }
+anyhow = "1.0.75"
+axum = { version = "0.6.20", features = ["json", "ws", "macros"] }
+axum-macros = { version = "0.3.8" }
+base64 = { version = "0.21.0" }
 bincode = "1.3.3"
-byteorder   = "1.5.0"
-clap        = { version = "4.4.6", features = ["derive", "cargo", "env"] }
-ethabi      = "18.0.0"
-ethers      = { version = "2.0.14", features = ["ws"] }
-futures            = { version = "0.3.28" }
+byteorder = "1.5.0"
+clap = { version = "4.4.6", features = ["derive", "cargo", "env"] }
+ethabi = "18.0.0"
+ethers = { version = "2.0.14", features = ["ws"] }
+futures = { version = "0.3.28" }
 hex = "0.4.3"
-prometheus-client  = { version = "0.21.2" }
+prometheus-client = { version = "0.21.2" }
 pythnet-sdk = { path = "../../pythnet/pythnet_sdk", features = ["strum"] }
-rand        = "0.8.5"
-reqwest     = { version = "0.11.22", features = ["json", "blocking"] }
-serde       = { version = "1.0.188", features = ["derive"] }
-serde_qs           = { version = "0.12.0", features = ["axum"] }
-serde_json  = "1.0.107"
+rand = "0.8.5"
+reqwest = { version = "0.11.22", features = ["json", "blocking"] }
+serde = { version = "1.0.188", features = ["derive"] }
+serde_qs = { version = "0.12.0", features = ["axum"] }
+serde_json = "1.0.107"
 serde_with = { version = "3.4.0", features = ["hex", "base64"] }
 serde_yaml = "0.9.25"
-sha3        = "0.10.8"
-tokio       = { version = "1.33.0", features = ["full"] }
-tower-http         = { version = "0.4.0", features = ["cors"] }
-tracing            = { version = "0.1.37", features = ["log"] }
+sha3 = "0.10.8"
+tokio = { version = "1.33.0", features = ["full"] }
+tower-http = { version = "0.4.0", features = ["cors"] }
+tracing = { version = "0.1.37", features = ["log"] }
 tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }
-utoipa             = { version = "3.4.0", features = ["axum_extras"] }
-utoipa-swagger-ui  = { version = "3.1.4", features = ["axum"] }
+utoipa = { version = "3.4.0", features = ["axum_extras"] }
+utoipa-swagger-ui = { version = "3.1.4", features = ["axum"] }
 once_cell = "1.18.0"
 lazy_static = "1.4.0"
 url = "2.5.0"
-chrono = { version = "0.4.38", features = ["clock", "std"] , default-features = false}
+chrono = { version = "0.4.38", features = [
+  "clock",
+  "std",
+], default-features = false }
 backoff = { version = "0.4.0", features = ["futures", "tokio"] }
 thiserror = "1.0.61"
 futures-locks = "0.7.1"

apps/fortuna/config.sample.yaml

@@ -12,6 +12,10 @@ chains:
     # How much to charge in fees
     fee: 1500000000000000
 
+    # Multiplier for the priority fee estimate, as a percentage (i.e., 100 = no change).
+    # Defaults to 100 if the field is omitted.
+    priority_fee_multiplier_pct: 100
+
     # Configuration for dynamic fees under high gas prices. The keeper will set
     # on-chain fees to make between [min_profit_pct, max_profit_pct] of the max callback
     # cost in profit per transaction.

apps/fortuna/src/chain/eth_gas_oracle.rs

@@ -39,11 +39,15 @@ pub const EIP1559_FEE_ESTIMATION_THRESHOLD_MAX_CHANGE: i64 = 200;
 #[must_use]
 pub struct EthProviderOracle<M: Middleware> {
     provider: M,
+    priority_fee_multiplier_pct: u64,
 }
 
 impl<M: Middleware> EthProviderOracle<M> {
-    pub fn new(provider: M) -> Self {
-        Self { provider }
+    pub fn new(provider: M, priority_fee_multiplier_pct: u64) -> Self {
+        Self {
+            provider,
+            priority_fee_multiplier_pct,
+        }
     }
 }
 
@@ -61,10 +65,19 @@ where
     }
 
     async fn estimate_eip1559_fees(&self) -> Result<(U256, U256)> {
-        self.provider
+        let (max_fee_per_gas, max_priority_fee_per_gas) = self
+            .provider
             .estimate_eip1559_fees(Some(eip1559_default_estimator))
             .await
-            .map_err(|err| GasOracleError::ProviderError(Box::new(err)))
+            .map_err(|err| GasOracleError::ProviderError(Box::new(err)))?;
+
+        // Apply the multiplier to max_priority_fee_per_gas
+        let max_priority_fee_per_gas = max_priority_fee_per_gas
+            .checked_mul(U256::from(self.priority_fee_multiplier_pct))
+            .and_then(|x| x.checked_div(U256::from(100)))
+            .unwrap_or(max_priority_fee_per_gas);
+
+        Ok((max_fee_per_gas, max_priority_fee_per_gas))
     }
 }
 
@@ -79,12 +92,14 @@ pub fn eip1559_default_estimator(base_fee_per_gas: U256, rewards: Vec<Vec<U256>>
                 U256::from(EIP1559_FEE_ESTIMATION_DEFAULT_PRIORITY_FEE),
             )
         };
+
     let potential_max_fee = base_fee_surged(base_fee_per_gas);
     let max_fee_per_gas = if max_priority_fee_per_gas > potential_max_fee {
         max_priority_fee_per_gas + potential_max_fee
     } else {
         potential_max_fee
     };
+
     (max_fee_per_gas, max_priority_fee_per_gas)
 }
 

apps/fortuna/src/chain/ethereum.rs

@@ -211,7 +211,8 @@ impl<T: JsonRpcClient + 'static + Clone> SignablePythContractInner<T> {
         provider: Provider<T>,
     ) -> Result<SignablePythContractInner<T>> {
         let chain_id = provider.get_chainid().await?;
-        let gas_oracle = EthProviderOracle::new(provider.clone());
+        let gas_oracle =
+            EthProviderOracle::new(provider.clone(), chain_config.priority_fee_multiplier_pct);
         let wallet__ = private_key
             .parse::<LocalWallet>()?
             .with_chain_id(chain_id.as_u64());

apps/fortuna/src/config.rs

@@ -166,6 +166,10 @@ pub struct EthereumConfig {
     /// Maximum number of hashes to record in a request.
     /// This should be set according to the maximum gas limit the provider supports for callbacks.
     pub max_num_hashes: Option<u32>,
+
+    /// The percentage multiplier to apply to the priority fee (100 = no change, e.g. 150 = 150% of base fee)
+    #[serde(default = "default_priority_fee_multiplier_pct")]
+    pub priority_fee_multiplier_pct: u64,
 }
 
 /// A commitment that the provider used to generate random numbers at some point in the past.
@@ -215,6 +219,10 @@ fn default_chain_sample_interval() -> u64 {
     1
 }
 
+fn default_priority_fee_multiplier_pct() -> u64 {
+    100
+}
+
 /// Configuration values for the keeper service that are shared across chains.
 #[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
 pub struct KeeperConfig {

apps/fortuna/src/keeper.rs

@@ -2,7 +2,6 @@ use {
     crate::{
         api::{self, BlockchainState, ChainId},
         chain::{
-            eth_gas_oracle::eip1559_default_estimator,
             ethereum::{
                 InstrumentedPythContract, InstrumentedSignablePythContract, PythContractCall,
             },
@@ -21,7 +20,7 @@ use {
     futures::StreamExt,
     prometheus_client::{
         encoding::EncodeLabelSet,
-        metrics::{counter::Counter, family::Family, gauge::Gauge},
+        metrics::{counter::Counter, family::Family, gauge::Gauge, histogram::Histogram},
         registry::Registry,
     },
     std::{
@@ -63,7 +62,6 @@ pub struct AccountLabel {
     pub address: String,
 }
 
-#[derive(Default)]
 pub struct KeeperMetrics {
     pub current_sequence_number: Family<AccountLabel, Gauge>,
     pub end_sequence_number: Family<AccountLabel, Gauge>,
@@ -75,6 +73,33 @@ pub struct KeeperMetrics {
     pub requests_processed: Family<AccountLabel, Counter>,
     pub requests_reprocessed: Family<AccountLabel, Counter>,
     pub reveals: Family<AccountLabel, Counter>,
+    pub request_duration_ms: Family<AccountLabel, Histogram>,
+}
+
+impl Default for KeeperMetrics {
+    fn default() -> Self {
+        Self {
+            current_sequence_number: Family::default(),
+            end_sequence_number: Family::default(),
+            balance: Family::default(),
+            collected_fee: Family::default(),
+            current_fee: Family::default(),
+            total_gas_spent: Family::default(),
+            requests: Family::default(),
+            requests_processed: Family::default(),
+            requests_reprocessed: Family::default(),
+            reveals: Family::default(),
+            request_duration_ms: Family::new_with_constructor(|| {
+                Histogram::new(
+                    vec![
+                        1000.0, 2500.0, 5000.0, 7500.0, 10000.0, 20000.0, 30000.0, 40000.0,
+                        50000.0, 60000.0, 120000.0, 180000.0, 240000.0, 300000.0, 600000.0,
+                    ]
+                    .into_iter(),
+                )
+            }),
+        }
+    }
 }
 
 impl KeeperMetrics {
@@ -142,6 +167,12 @@ impl KeeperMetrics {
             keeper_metrics.requests_reprocessed.clone(),
         );
 
+        writable_registry.register(
+            "request_duration_ms",
+            "Time taken to process each callback request in milliseconds",
+            keeper_metrics.request_duration_ms.clone(),
+        );
+
         keeper_metrics
     }
 }
@@ -343,6 +374,8 @@ pub async fn process_event_with_backoff(
     gas_limit: U256,
     metrics: Arc<KeeperMetrics>,
 ) {
+    let start_time = std::time::Instant::now();
+
     metrics
         .requests
         .get_or_create(&AccountLabel {
@@ -373,6 +406,16 @@ pub async fn process_event_with_backoff(
             tracing::error!("Failed to process event: {:?}", e);
         }
     }
+
+    let duration_ms = start_time.elapsed().as_millis() as f64;
+    metrics
+        .request_duration_ms
+        .get_or_create(&AccountLabel {
+            chain_id: chain_state.id.clone(),
+            address: chain_state.provider_address.to_string(),
+        })
+        .observe(duration_ms);
+
     metrics
         .requests_processed
         .get_or_create(&AccountLabel {
@@ -1208,9 +1251,11 @@ pub async fn estimate_tx_cost(
             .try_into()
             .map_err(|e| anyhow!("gas price doesn't fit into 128 bits. error: {:?}", e))?
     } else {
-        let (max_fee_per_gas, max_priority_fee_per_gas) = middleware
-            .estimate_eip1559_fees(Some(eip1559_default_estimator))
-            .await?;
+        // This is not obvious but the implementation of estimate_eip1559_fees in ethers.rs
+        // for a middleware that has a GasOracleMiddleware inside is to ignore the passed-in callback
+        // and use whatever the gas oracle returns.
+        let (max_fee_per_gas, max_priority_fee_per_gas) =
+            middleware.estimate_eip1559_fees(None).await?;
 
         (max_fee_per_gas + max_priority_fee_per_gas)
             .try_into()