Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Context Deadline fails w/ statesync + arm #999

Open
4 tasks
dreamcodez opened this issue Jun 2, 2022 · 3 comments
Open
4 tasks

Context Deadline fails w/ statesync + arm #999

dreamcodez opened this issue Jun 2, 2022 · 3 comments
Labels
T:Bug A regression or bug in the Umee codebase.

Comments

@dreamcodez
Copy link
Contributor

Summary of Bug

Context deadline failure which impacts only arm with statesync

5:41PM INF error from light block request from primary, removing... error="post failed: Post \"https://rpc.alley.umeemania-1.network.umee.cc:443\": context deadline exceeded" height=831700 module=light primary={}
5:41PM ERR error on light block request from witness, removing... error="post failed: Post \"https://rpc.cope.umeemania-1.network.umee.cc:443\": context deadline exceeded" module=light primary={}
5:41PM ERR failed to remove witnesses err="no witnesses connected. please reset light client" module=light witnessesToRemove=[0]
Error: failed to start state sync: failed to set up light client state provider: post failed: Post "https://rpc.cope.umeemania-1.network.umee.cc:443": context deadline exceeded

I tested this with linux/amd64 via docker emulation and it does not have this problem.

Version

Please provide the output of the following commands:

  • $ umeed version => 2.0.3-rc2
  • $ go version => N/A (release build)
  • $ uname -a => Darwin 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:46:32 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T6000 arm64

Steps to Reproduce

Steps to reproduce the behavior:

  1. use an arm64 platform build
  2. run umeed start w/ statesync
  3. example genesis: https://raw.githubusercontent.com/umee-network/testnets/main/networks/umeemania-1/genesis.json
  4. example config.toml:
[p2p]
persistent_peers = "[email protected]:26656,[email protected]:26656"
[statesync]
rpc_servers = "[https://rpc.alley.umeemania-1.network.umee.cc:443](https://rpc.alley.umeemania-1.network.umee.cc/),[https://rpc.cope.umeemania-1.network.umee.cc:443](https://rpc.cope.umeemania-1.network.umee.cc/)"
trust_height = 831700
trust_hash = "546155C86814F3DBFA31F0DBB6626C4160A73668763C7F8A43E8FCC1B01FD84A"
trust_period = "3000h"

For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@dreamcodez dreamcodez added T:Bug A regression or bug in the Umee codebase. P:Medium labels Jun 2, 2022
@brianosaurus
Copy link
Contributor

This is an HTTP-RPC timeout.

@dreamcodez
Copy link
Contributor Author

This is an HTTP-RPC timeout.

I agree. However:

  1. on arm it consistently fails after the timeout, every time -- no matter how many times i run
  2. umeed status seems to also have a similar problem on arm, it hangs when trying to contact rpc
  3. when i run emulated amd64 the problem dissappears

@brianosaurus
Copy link
Contributor

brianosaurus commented Jun 3, 2022

You just need to increase the rpc_write_timeout in app.toml. I got the same error in an Intel i7 until I increased that. Alternatively, it could be the rpc_read_timeout. I haven't figured out which one exactly is the problem but it occurs on a POST so I figure write timeout is the culprit. It's also set to 0 seconds by default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T:Bug A regression or bug in the Umee codebase.
Projects
None yet
Development

No branches or pull requests

3 participants