RPC trace_block response speed

[hypnagonia]

Summary

RPC trace_block response speed is slow. request takes ~1 second for a block with ~60 call txs. which is not enough for indexing apps.
fetching monthly data would take 2 weeks

Current Design

actual design is rerunning all tx calls in a block to get call traces

Problems

takes long time to form response

Proposal

cache when executes first time and never rerun

[LeoHChen]

@peekpi , can you add some metrics in the trace_block call to measure the time consumption?

@hypnagonia , do you have an example of the rpc call using curl?

[hypnagonia]

@LeoHChen
curl

curl --location --request POST 'https://a.api.s0.t.hmny.io' \
--header 'Content-Type: application/json' \
--data-raw '{
"jsonrpc": "2.0",
  "method": "trace_block",
  "id": 1,
  "params": ["0xd776b3"]
}'

[LeoHChen]

This query runs for a long time and it seems it has only 35 tx in the block.
https://explorer.harmony.one/#/block/0xd1be382e849a77f93bddac782d7e1e6424cf73ef7b5c903a76a3b16661c25de9/txs

Do you know why it takes so long for the query to return all the data? @peekpi

[peekpi]

This query runs for a long time and it seems it has only 35 tx in the block.
https://explorer.harmony.one/#/block/0xd1be382e849a77f93bddac782d7e1e6424cf73ef7b5c903a76a3b16661c25de9/txs

Do you know why it takes so long for the query to return all the data? @peekpi

this is the trace result of the block: https://gist.github.com/peekpi/de46dbee349c3664dfbabe0b86c17d13

below are the statistics:

• index: index of the tx in the block
• hash: tx hash
• count: number of internal calls. (number of all nodes in the call tree.)
• depth: depth of call tree
• max: The number of nodes of the layer with the highest number of nodes.

for a normal contract transaction, it has around 5-10 internal calls. but there are 16 txs that have 876 internal calls, trace these txs takes up most of the time.
Also, the size of the trace result is nearly 9M.

index	hash	count	depth	max
0	0xc5ed805fb02c30beb077687db52a004114a5b5fa5df460bc5812349e50a16623	10	2	7
1	0xe45bbadea7c2b9f1112299b6b8ffe9fb5707866d08c4100ee3753eaffbed1b59	876	2	874
2	0x46765361fe7e2a76f08404a16525206b5f18d0bde11d272752f2d468ada92a45	876	2	874
3	0xce92f54a965b1e4607578ec04e0845f1c747e218d334eec1ddd751c0265bcd36	876	2	874
4	0xa68e1004d4e471b049dc9066e26be6dd4ee54fd882f5dcb5126a26fb07a23c51	876	2	874
5	0x41e807f95128a6bf99b1cdc01779ff186e2af39492be9eddafafebc77b4561e2	876	2	874
6	0xcbe53f6fb4dd2ee13f24e4cf1a39c32209c2d998bb5698965c9b05cd4e5dc61e	876	2	874
7	0x7e17e141780e377c6669199a5f990a503f62a7eff49b08ae044fae2132ff0f92	876	2	874
8	0xb7294f63dc8770844662a9c34fddc56fac850226e9dae8c8b7a004704a705d5e	876	2	874
9	0x1d22b67dc5517683340c3bd5b936cf07fdd8a2f311e1a63be98a0047bbdae6f9	876	2	874
10	0x81660fa4fd164b66dc0ca466607089d0b304fb5acd522afb8bb71329999a749a	876	2	874
11	0x76a040e002594714625fdf9a829676d63ee8f3932727227e5cc06efa6e03f478	876	2	874
12	0x24894376fc02f5f45d12c57f389d8a15bcda34438cf8daeb6a539efb08e3826c	876	2	874
13	0xe8ce7f8a477d35cb6333735973e469aa7eac3a660d2575b085a359efbc92edca	876	2	874
14	0xf52911d363daa2de14d59a3eb709388fd07a13ad2732f7c441d9178faa2756ef	876	2	874
15	0x07d1d43be96aa4c822e34ed3a9edcba91376cb397da0deaee26e6324b76a3ee0	876	2	874
16	0x2049711a28c8a3406a6471981b1306a1d55ddaf719227508c105f1f682abf5d8	876	2	874
17	0xe101f0ed2eb051d8545d4855d791551f424c55715ce7a1a3cdca741bc4b6638a	5	2	2
18	0x36201d6cae468c8fdc73371e3c0d5ee77a368981ca23a2d66b50968030e18d9f	5	2	2
19	0x82504075d51f7fcf8e448d72ffa67cdedcd566f145bb446fca3e44f8205b4e0f	5	2	2
20	0x9a9b580a705c5e7a11509186546ce8db6a2dca2aefcc7ff1894d52857c5be092	5	2	2
21	0xaade0c6f48cf6f1785e2d6b33b8f1d84f26f146f28f405c7cb1e9bc307ddd157	5	2	2
22	0x6762b3a0d252eb323a15cc4c45f65f2f08fb285adf07e5b2bb9544229aefdd33	5	2	2
23	0x5c3d567a065e5ec2c3908eee7fc316385dc900763d1e1680de661114b23cf62e	5	2	2
24	0x19f75ee20d1c25886c77eb75786e14c79268c912d8ca44a921450852155de473	5	2	2
25	0x7c6407945553286f2f6017b1ef8ebd129f1208f8dbc0b4964df85587561e626c	5	2	2
26	0xa305f8a5a9a625677d4bd85f83b29337b034ce42215b5f79b2400a4ab65afe35	5	2	2
27	0x11a7bf4c48b57afe89ec14fb666283c51d65b06b191425d541678104a1266935	5	2	2
28	0x84799d9263154194f2a73f224bdd4166bcda157f58c178052ce5b20e4842aac0	5	2	2
29	0xba61d574ec5d55971bdf30a1a44eb56991e5704924dd5305e0a637edab5f69d6	5	2	2
30	0x5a12f2a8ee7e0db68f1e22ce09d5b1c99269ecc6192e1b134f96efb80fcdbd2b	5	2	2
31	0xe6ca38f3162276adc8be5a00baa587736214c3dca1a89ce6e7e69b2626bf5c8b	5	2	2
32	0x0af99855c11c2e83a063973b104739e5c8a116094238a27eb9beaa6f452a0d7e	5	2	2
33	0x0332bbd0857ac61c5aec78f205d6c6ad5133df9324c46f13da69292fff8bb71a	5	2	2
34	0x5ce308d559d4f77793836d464467db4961f91d85109ffcb1db48c49320af527c	5	2	2

[peekpi]

@LeoHChen and I can store the trace result into DB to speed up the query, but this may cause high usage of disk.
I use tar -zcf to compress the trace result of this block, the original file is 9,482,514 bytes, and compressed is 334,256 bytes.

[jleeqn]

Hey team, we use trace_block to ingest Harmony trace data, but the process is relatively slow. Is there any way we can do to speed up the process? Thanks!

curl http://endpoint:9501 \
  -X POST \
  -H "Content-Type: application/json" \
  --data '{"method":"trace_block","params":["0xccb93d"],"id":1,"jsonrpc":"2.0"}'