Merge pull request #149 from HathorNetwork/master

* chore: updated wallet-service docs to root

* chore: updated SOP and feature toggles document

* chore: increased memory and lambda timeout (#145)

undefined

* feat: added handler to handle DLQ messages from loadWalletAsync (#146)

* feat: added handler to handle DLQ messages from loadWalletAsync

* tests: improved dlq test

* fix: sendAlert was failing because no region was set

* tests: passing full event

* feat: added try/catch and an alarm in case the processing failed

* tests: checking for wallet error

* tests: fixed tests

* chore: sending request id properly

* chore: added sns topic on lambdas

* refactor: setting retries to max

* docs: improved failed to handle loadWalletFailed alert

* refactor: removed xpubkey from logs and alerts

* chore: removed publish permission on handleLoadWalletFailed

* refactor: added alert on no xpubkey received

* docs: added event message example

* docs: spelling mistake

* chore: using region from provider instead of hardcoded

* fix: typo

* chore: bumped to v1.5.0 (#147)

* fix: fixed invalid 'local' being passed to lambda client on multiple … (#148)

* fix: fixed invalid 'local' being passed to lambda client on multiple lambdas

* fix: tests failing because of missing aws_region mock env

packages/wallet-service/docs/2021-07-29-infrastructure-design.md → docs/2021-07-29-infrastructure-design.md

@@ -2,7 +2,7 @@
 
 Date: 2021-07-29
 
-Issue: https://github.com/HathorNetwork/hathor-wallet-service/issues/80
+Issue: https://github.com/HathorNetwork/hathor-wallet-service-old/issues/80
 
 ## Summary
 

packages/wallet-service/docs/2022-04-18-feature-toggles.md → docs/2022-04-18-feature-toggles.md

@@ -10,18 +10,13 @@ This document describes the main features we are using and how to interact with
 
 ## Feature toggles
 
-We have a list of [feature toggles](https://docs.getunleash.io/advanced/feature_toggle_types) that are queried by the wallets:
+We have two [feature toggles](https://docs.getunleash.io/advanced/feature_toggle_types) that are queried by the wallets:
 
-* `wallet-service-mobile-android-mainnet.rollout`
-* `wallet-service-mobile-android-testnet.rollout`
-* `wallet-service-mobile-ios-mainnet.rollout`
-* `wallet-service-mobile-ios-testnet.rollout`
-* `wallet-service-wallet-desktop-mainnet.rollout`
-* `wallet-service-wallet-desktop-testnet.rollout`
+* `wallet-service-mobile.rollout`
+* `wallet-service-desktop.rollout`
 
 Those feature toggles are `Release` toggles and represent each wallet and the network they are connected to, e.g. when the mobile wallet on iOS on the `mainnet` wants to know wether to use or not the wallet-service facade, it will request the `wallet-service-mobile-ios-mainnet.rollout` feature-flag that will answer "Yes" or "No" based on a list of strategies.
 
-
 ## Stategies
 
 > It is powerful to be able to turn a feature on and off instantaneously, without redeploying the application. The next level of control comes when you are able to enable a feature for specific users or enable it for a small subset of users. We achieve this level of control with the help of activation strategies. The most straightforward strategy is the standard strategy, which basically means that the feature should be enabled to everyone.
@@ -39,7 +34,7 @@ Activates for users with a `userId` defined in the `userIds` list. We are curren
 
 1. [Gradual rollout](https://docs.getunleash.io/user_guide/activation_strategy#gradual-rollout)
 
-This is a `percentage` based strategy, it will answer the feature toggles depending on the percentage of users that already received a positive or negative answer. 
+This is a `percentage` based strategy, it will answer the feature toggles depending on the percentage of users that already received a positive or negative answer.
 
 For [stickness](https://docs.getunleash.io/advanced/stickiness), we are currently using `userId` on all the feature toggles, so if an user receives a positive response to the feature toggle request, it will continue receiving a positive response on consecutive requests
 

packages/wallet-service/docs/SOP.md → docs/SOP.md

@@ -4,14 +4,8 @@
 
 The deployment is partially automated with CodeBuild and CodePipeline in AWS.
 
-It's triggered when commits are made to `dev` or `master` branches, and when tags with names like `v*` are created.
-
-Each case deploys to a different environment:
-- `dev` branch -> `dev-testnet` environment
-- `master` branch -> `testnet` environment
-- `v*` tags -> `mainnet` environment
-
-All of them require manual approval to proceed. You should keep an eye in the `#wallet-service-deploys` channel in Slack, the approval requests are sent there.
+Please refer to the wallet-service's release guide for a more in-depth expalanation of which branches/tags trigger builds:
+https://github.com/HathorNetwork/ops-tools/blob/1092092ba840c7492436ee092ef2d0a274006c5b/docs/release-guides/wallet-service.md
 
 If you need to know the exact steps that take place during deployment, check [this document](2021-07-29-infrastructure-design.md#how-the-process-works)
 
@@ -30,7 +24,7 @@ Let's say we want to add the `ENV_VAR_1` env var.
 
 First step would be to add it to the [serverless.yml](https://github.com/HathorNetwork/hathor-wallet-service/blob/master/serverless.yml) file, under `provider.environment`.
 
-Then, you need to add it in [.codebuild/buildspec.yml](https://github.com/HathorNetwork/hathor-wallet-service/blob/master/.codebuild/buildspec.yml). If it's not a secret, just add it under `env.variables`. 
+Then, you need to add it in [.codebuild/buildspec.yml](https://github.com/HathorNetwork/hathor-wallet-service/blob/master/.codebuild/buildspec.yml). If it's not a secret, just add it under `env.variables`.
 
 If it's a secret, you'll need to add it to `env.secrets-manager`, and one for each environment we have (`dev`, `testnet` and `mainnet`). You should use the same name for it as you did in the `serverless.yml` file, but adding a prefix indicating the name of the environment. The value should be the path to a key in AWS Secrets Manager. Ask some account admin for help on adding the secrets there and providing you with the key path.
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "hathor-wallet-service",
-  "version": "1.4.6",
+  "version": "1.5.0",
   "workspaces": [
     "packages/daemon",
     "packages/wallet-service"

packages/daemon/src/config.ts

@@ -74,6 +74,7 @@ export const PUSH_NOTIFICATION_LAMBDA_REGION = process.env.PUSH_NOTIFICATION_LAM
 export const ACCOUNT_ID = process.env.ACCOUNT_ID;
 export const ALERT_MANAGER_REGION = process.env.ALERT_MANAGER_REGION;
 export const ALERT_MANAGER_TOPIC  = process.env.ALERT_MANAGER_TOPIC;
+export const AWS_REGION = process.env.AWS_REGION;
 
 // Healthcheck configuration
 export const HEALTHCHECK_ENABLED = process.env.HEALTHCHECK_ENABLED === 'true';
@@ -104,6 +105,7 @@ export default () => ({
   WALLET_SERVICE_LAMBDA_ENDPOINT,
   STAGE,
   ACCOUNT_ID,
+  AWS_REGION,
   ALERT_MANAGER_REGION,
   ALERT_MANAGER_TOPIC,
   ON_TX_PUSH_NOTIFICATION_REQUESTED_FUNCTION_NAME,

packages/daemon/src/utils/aws.ts

@@ -61,8 +61,17 @@ export const invokeOnTxPushNotificationRequestedLambda = async (walletBalanceVal
  * @param messageBody - A string with the message body
  * @param queueUrl - The queue URL
  */
-export const sendMessageSQS = async (messageBody: string, queueUrl: string, messageAttributes?: Record<string, MessageAttributeValue>): Promise<SendMessageCommandOutput> => {
-  const client = new SQSClient({});
+export const sendMessageSQS = async (messageBody: string, queueUrl: string, messageAttributes?: Record<string, MessageAttributeValue>, region?: string): Promise<SendMessageCommandOutput> => {
+  const { AWS_REGION } = getConfig();
+
+  if (!region) {
+    region = AWS_REGION;
+  }
+
+  const client = new SQSClient({
+    endpoint: queueUrl,
+    region,
+  });
   const command = new SendMessageCommand({
     QueueUrl: queueUrl,
     MessageBody: messageBody,

packages/wallet-service/serverless.yml

@@ -36,10 +36,10 @@ custom:
     topics: # SNS Topics to send alerts to
       major:
         alarm:
-          topic: arn:aws:sns:eu-central-1:${self:provider.environment.ACCOUNT_ID}:opsgenie-cloudwatch-integration-production-major
+          topic: arn:aws:sns:${self:provider.region}:${self:provider.environment.ACCOUNT_ID}:opsgenie-cloudwatch-integration-production-major
       minor:
         alarm:
-          topic: arn:aws:sns:eu-central-1:${self:provider.environment.ACCOUNT_ID}:opsgenie-cloudwatch-integration-production-minor
+          topic: arn:aws:sns:${self:provider.region}:${self:provider.environment.ACCOUNT_ID}:opsgenie-cloudwatch-integration-production-minor
     definitions: # Definition of alarms
       majorFunctionErrors:
         description: "Too many errors in hathor-wallet-service. Runbook: https://github.com/HathorNetwork/ops-tools/blob/master/docs/runbooks/wallet-service/errors-in-logs.md"
@@ -120,10 +120,16 @@ resources:
       Properties:
         QueueName:
             WalletServiceNewTxQueue_${self:custom.stage}
+    WalletServiceLoadAsyncFailedTopic:
+        Type: "AWS::SNS::Topic"
+        Properties:
+          DisplayName: 'Messages published when the loadWalletAsync lambda fails'
+          TopicName: WalletServiceLoadAsyncFailed_${self:custom.stage}
 
 provider:
   name: aws
   runtime: nodejs18.x
+  region: ${opt:region, 'eu-central-1'}
   # In MB. This is the memory allocated for the Lambdas, they cannot use more than this
   # and will break if they try.
   memorySize: 256
@@ -239,12 +245,40 @@ functions:
           - lambda:InvokeFunction
           - lambda:InvokeAsync
         Resource:
-          arn:aws:lambda:eu-central-1:${self:provider.environment.ACCOUNT_ID}:function:hathor-explorer-service-${self:custom.explorerServiceStage}-create_or_update_dag_metadata
+          arn:aws:lambda:${self:provider.region}:${self:provider.environment.ACCOUNT_ID}:function:hathor-explorer-service-${self:custom.explorerServiceStage}-create_or_update_dag_metadata
   loadWalletAsync:
     handler: src/api/wallet.loadWallet
+    # This lambda is currently running out of memory when wallets with a big
+    # number of transactions (> 300k) is being loaded. I chose 1024 because the
+    # average memory usage of the largest test wallet we have is 800mb.
+    memorySize: 1024
+    timeout: 600 # 10 minutes should be enough for most wallets
+    onError: arn:aws:sns:${self:provider.region}:${self:provider.environment.ACCOUNT_ID}:WalletServiceLoadAsyncFailed_${self:custom.stage}
     warmup:
       walletWarmer:
         enabled: false
+    iamRoleStatements:
+      - Effect: Allow
+        Action:
+          - SNS:Publish
+        Resource:
+          arn:aws:sns:${self:provider.region}:${self:provider.environment.ACCOUNT_ID}:WalletServiceLoadAsyncFailed_${self:custom.stage}
+  handleLoadWalletFailed:
+    handler: src/api/wallet.loadWalletFailed
+    events:
+      - sns: arn:aws:sns:${self:provider.region}:${self:provider.environment.ACCOUNT_ID}:WalletServiceLoadAsyncFailed_${self:custom.stage}
+    warmup:
+      walletWarmer:
+        enabled: false
+    iamRoleStatementsInherit: true
+    iamRoleStatementsName: hathor-wallet-service-${self:custom.stage}-handleLoadFailed-snsRole
+    iamRoleStatements:
+      - Effect: Allow
+        Action:
+          - SNS:Subscribe
+          - SNS:Unsubscribe
+        Resource:
+          arn:aws:sns:${self:provider.region}:${self:provider.environment.ACCOUNT_ID}:WalletServiceLoadAsyncFailed_${self:custom.stage}
   loadWalletApi:
     role: arn:aws:iam::${self:provider.environment.ACCOUNT_ID}:role/WalletServiceLoadWalletLambda
     handler: src/api/wallet.load

packages/wallet-service/src/api/wallet.ts

@@ -5,7 +5,7 @@
  * LICENSE file in the root directory of this source tree.
  */
 
-import { APIGatewayProxyHandler, Handler } from 'aws-lambda';
+import { APIGatewayProxyHandler, Handler, SNSEvent } from 'aws-lambda';
 import { LambdaClient, InvokeCommand, InvokeCommandOutput } from '@aws-sdk/client-lambda';
 import 'source-map-support/register';
 
@@ -21,7 +21,7 @@ import {
   updateWalletStatus,
   updateWalletAuthXpub,
 } from '@src/db';
-import { WalletStatus } from '@src/types';
+import { Severity, WalletStatus } from '@src/types';
 import {
   closeDbConnection,
   getDbConnection,
@@ -38,6 +38,7 @@ import middy from '@middy/core';
 import cors from '@middy/http-cors';
 import Joi from 'joi';
 import createDefaultLogger from '@src/logger';
+import { addAlert } from '@src/utils/alerting.utils';
 
 const mysql = getDbConnection();
 
@@ -380,6 +381,75 @@ interface LoadResult {
   xpubkey: string;
 }
 
+/*
+ * This lambda will be started by a SNSMessage on the load failed SNS configured
+ * in serverless.yml. It will receive all wallet load failed events published on
+ * the loadWalletAsync DLQ SNS topic
+ *
+ * The event will be a SNSEvent, here is an example of the Message attribute:
+ * {"xpubkey":"xpub","maxGap":20}
+ */
+export const loadWalletFailed: Handler<SNSEvent> = async (event) => {
+  const logger = createDefaultLogger();
+  const records = event.Records;
+
+  try {
+    for (let i = 0; i < records.length; i++) {
+      const snsEvent = records[i].Sns;
+      const { RequestID, ErrorMessage } = snsEvent.MessageAttributes;
+
+      // Process each failed load wallet event
+      const loadEvent: LoadEvent = JSON.parse(snsEvent.Message) as unknown as LoadEvent;
+
+      if (!loadEvent.xpubkey) {
+        logger.error('Received wallet load fail message from SNS but no xpubkey received');
+        await addAlert(
+          'Wallet failed to load, but no xpubkey received.',
+          `An event reached loadWalletFailed lambda but the xpubkey was not sent. This indicates that a wallet has failed to load and we weren't able to recover, please check the logs as soon as possible.`,
+          Severity.MAJOR,
+          {
+            RequestID: RequestID.Value,
+            ErrorMessage: ErrorMessage.Value,
+          },
+        );
+        continue;
+      }
+
+      const walletId = getWalletId(loadEvent.xpubkey);
+
+      // update wallet status to 'error' and set the number of retries to MAX so
+      // it doesn't get retried
+      await updateWalletStatus(mysql, walletId, WalletStatus.ERROR, MAX_LOAD_WALLET_RETRIES);
+
+      logger.error(`${walletId} failed to load.`);
+      logger.error({
+        walletId,
+        RequestID: RequestID.Value,
+        ErrorMessage: ErrorMessage.Value,
+      });
+
+      await addAlert(
+        'A wallet failed to load in the wallet-service',
+        `The wallet with id ${walletId} failed to load on the wallet-service. Please check the logs.`,
+        Severity.MINOR,
+        {
+          walletId,
+          RequestID: RequestID.Value,
+          ErrorMessage: ErrorMessage.Value,
+        },
+      );
+    }
+  } catch (e) {
+    await addAlert(
+      'Failed to handle loadWalletFailed event',
+      `Failed to process the loadWalletFailed event. This indicates that wallets failed to load and we weren't able to recover, please check the logs as soon as possible.`,
+      // This is major because the user will be stuck in a loading cycle
+      Severity.MAJOR,
+      { event },
+    );
+  }
+};
+
 /*
  * This does the "heavy" work when loading a new wallet, updating the database tables accordingly. It
  * expects a wallet entry already on the database

packages/wallet-service/src/utils/alerting.utils.ts

@@ -48,7 +48,10 @@ export const addAlert = async (
 
   const QUEUE_URL = `https://sqs.${ALERT_MANAGER_REGION}.amazonaws.com/${ACCOUNT_ID}/${ALERT_MANAGER_TOPIC}`;
 
-  const client = new SQSClient({});
+  const client = new SQSClient({
+    endpoint: QUEUE_URL,
+    region: ALERT_MANAGER_REGION,
+  });
   const command = new SendMessageCommand({
     QueueUrl: QUEUE_URL,
     MessageBody: JSON.stringify(preparedMessage),

packages/wallet-service/src/utils/nft.utils.ts

@@ -85,7 +85,7 @@ export class NftUtils {
   static async _updateMetadata(nftUid: string, metadata: Record<string, unknown>): Promise<unknown> {
     const client = new LambdaClient({
       endpoint: process.env.EXPLORER_SERVICE_LAMBDA_ENDPOINT,
-      region: 'local',
+      region: process.env.AWS_REGION,
     });
    const command = new InvokeCommand({
       FunctionName: `hathor-explorer-service-${process.env.EXPLORER_SERVICE_STAGE}-create_or_update_dag_metadata`,
@@ -141,7 +141,7 @@ export class NftUtils {
   static async invokeNftHandlerLambda(txId: string): Promise<void> {
     const client = new LambdaClient({
       endpoint: process.env.WALLET_SERVICE_LAMBDA_ENDPOINT,
-      region: 'local',
+      region: process.env.AWS_REGION,
     });
     // invoke lambda asynchronously to metadata update
    const command = new InvokeCommand({

packages/wallet-service/src/utils/pushnotification.utils.ts

@@ -28,6 +28,7 @@ try {
     'FIREBASE_TOKEN_URI',
     'FIREBASE_AUTH_PROVIDER_X509_CERT_URL',
     'FIREBASE_CLIENT_X509_CERT_URL',
+    'AWS_REGION',
   ]);
 } catch (e) {
   logger.error(e);
@@ -49,6 +50,7 @@ export enum FunctionName {
 }
 
 const STAGE = process.env.STAGE;
+const AWS_REGION = process.env.AWS_REGION;
 const WALLET_SERVICE_LAMBDA_ENDPOINT = process.env.WALLET_SERVICE_LAMBDA_ENDPOINT;
 const SEND_NOTIFICATION_FUNCTION_NAME = buildFunctionName(FunctionName.SEND_NOTIFICATION_TO_DEVICE);
 const ON_TX_PUSH_NOTIFICATION_REQUESTED_FUNCTION_NAME = buildFunctionName(FunctionName.ON_TX_PUSH_NOTIFICATION_REQUESTED);
@@ -228,7 +230,7 @@ export class PushNotificationUtils {
 
     const client = new LambdaClient({
       endpoint: WALLET_SERVICE_LAMBDA_ENDPOINT,
-      region: 'local',
+      region: AWS_REGION,
     });
 
     const command = new InvokeCommand({
@@ -263,7 +265,7 @@ export class PushNotificationUtils {
 
     const client = new LambdaClient({
       endpoint: WALLET_SERVICE_LAMBDA_ENDPOINT,
-      region: 'local',
+      region: AWS_REGION,
     });
 
     const command = new InvokeCommand({