Improve synthetic monitoring and alerting of RPC methods

[givp]

Using AWS synthetic canary tests, we can simulate typical user experiences as it relates to RPC request and response flows. This will not only allow us to monitor response times but will also allow us to perform regression testing by ensuring response schemas are in tact and accurate returned values are present. Refer to this issue for a list of methods to be tested.

1. Go to AWS > Cloudwatch > Synthetics > Canaries
2. Create a new Canary and select "API Canary" blueprint

3. Add a new HTTP request

4. Add request URL, headers, and request body. Make sure to select "Capture headers and response body"

5. Using the script editor, use Javascript to inspect the response. For example, ensure a 2xx status code is returned and inspect the response payload to ensure certain fields are present and they match expected results.

6. Specify the frequency of your test

7. Finally, enable alarms for when success rates or endpoint response times fall below/above an acceptable threshold

You can now use the alarm like any other Cloudwatch alarm. It can trigger an email, send a message to PagerDuty and update https://status.harmony.one

[LeoHChen]

I tried it once to create a canary to test one RPC API. This is super nice.

[LeoHChen]

We can open a bounty to use either AWS CLI or CloudFormation to build a complete RPC monitoring hub using canary automation. The list of RPC calls can be found in https://api.hmny.io