Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adam/ct 948 add functionality to auxo to create new kafka topics #1756

Closed
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
b9f6cca
Add block height message to BlockProposer
adamfraser Jun 13, 2024
f7481b2
Add ability for socks to forward block height messages
adamfraser Jun 17, 2024
fd27c82
Add block-height topic to docker config
adamfraser Jun 18, 2024
ce5dfcf
Allow subscribe and unsubscribe to block height without id
adamfraser Jun 18, 2024
f6c19e3
Fix incorrect block height url
adamfraser Jun 18, 2024
62acfd9
Use BlockHeightMessage type
adamfraser Jun 18, 2024
a7b2538
Correctly format block height message in kafka helper
adamfraser Jun 19, 2024
2251142
Remove unused import
adamfraser Jun 19, 2024
914c51d
Modify workflow to push to dev4 from this branch
adamfraser Jun 19, 2024
4fe5a8d
Add kafka publisher tests
adamfraser Jun 19, 2024
a3a840d
Handle block height id is same manner as markets
adamfraser Jun 19, 2024
eb64d70
Revert "Modify workflow to push to dev4 from this branch"
adamfraser Jun 19, 2024
1bcc69d
Rename V4_BLOCK_HEIGHT constant for consistency
adamfraser Jun 20, 2024
f224ef1
Add example to documentation
adamfraser Jun 20, 2024
ec3c1c0
Add block height websocket topic to docker compose local
adamfraser Jun 20, 2024
23a5e41
Rename & add some more tags to stats (#1715)
dydxwill Jun 18, 2024
c1d4f76
chore: `x/subaccounts` move helper functions to to `lib` package (#1711)
BrendanChou Jun 18, 2024
7f9dec0
[OTE-419] Add GB_GEO to compliance reasons (#1697)
Christopher-Li Jun 18, 2024
1e2a44a
[OTE-379] Move Indexer SendOnchainData from Endblocker to Precommitte…
teddyding Jun 18, 2024
34f6c3d
Optimize GetSettlementPpmWithPerpetual, Remove GetSettlementPpm (#1722)
BrendanChou Jun 18, 2024
41d32e3
chore: remove `BigInt0()` & `BigFloat0()` (#1724)
BrendanChou Jun 18, 2024
7eb0f44
[OTE-438] Allow users to read data if they are in FIRST_STRIKE_CLOSE_…
Christopher-Li Jun 18, 2024
473e615
[TRA-354] Add a hard cap to the number of markets listed for PML (#1644)
shrenujb Jun 18, 2024
707cd90
[TRA-414] Add x/revshare module skeleton (#1719)
shrenujb Jun 18, 2024
edc419c
optimize perpInfos (#1723)
BrendanChou Jun 19, 2024
1aaef3a
[OTE-420]: Deprecate ONBOARD from Indexer (#1728)
Christopher-Li Jun 19, 2024
dfbc8a8
[CT-856] Add order replacement to fix vault causing orderbook flicker…
chenyaoy Jun 19, 2024
def4635
Send order remove subaccount message on order replace (#1735)
chenyaoy Jun 20, 2024
8d78baa
chore: add `margin.Risk` type, make margining functions stateless (#1…
BrendanChou Jun 20, 2024
a4cf38d
[TRA-415] implement market mapper rev share gov msg (#1733)
shrenujb Jun 20, 2024
8b6c3d8
[CT-946] only send snapshots to uninitialized streams (#1738)
jayy04 Jun 20, 2024
a2777f3
Optimize `ContainsDuplicates` (#1637)
BrendanChou Jun 21, 2024
a3dea6f
Cherry-pick "Handle invalid price level updates gracefully. (#1747)" …
vincentwschau Jun 21, 2024
59cb777
[TRA-433] Add MarketMapperRevShareDetails state and associated functi…
shrenujb Jun 21, 2024
b49bf48
[TRA-442] Add query for market mapper revenue share params (#1746)
shrenujb Jun 21, 2024
d357ac7
[CT-941] introduce state methods for managing the safety heap (#1731)
jayy04 Jun 22, 2024
b97b568
add MapToSortedSlice function (#1753)
BrendanChou Jun 22, 2024
0228ca8
Perf: add an software override (1.5sec) for `TimeoutPropose` (#1751)
teddyding Jun 24, 2024
a59b27e
Fix logger using incorrect function
adamfraser Jun 24, 2024
c100f2d
Add functionality to create new kafka topics
adamfraser Jun 24, 2024
09a18d4
Deploy to dev5
adamfraser Jun 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[OTE-420]: Deprecate ONBOARD from Indexer (#1728)
Christopher-Li authored and adamfraser committed Jun 24, 2024
commit 1aaef3ae4de435ef9f1adad7b5657b009cfd8027
Original file line number Diff line number Diff line change
@@ -250,7 +250,7 @@ describe('ComplianceV2Controller', () => {
const body: any = {
address: testConstants.defaultAddress,
message: 'Test message',
action: ComplianceAction.ONBOARD,
action: ComplianceAction.CONNECT,
signedMessage: 'signedmessage123',
pubkey: 'asdfasdf',
timestamp: 1620000000,
@@ -340,10 +340,11 @@ describe('ComplianceV2Controller', () => {
expect(response.body.status).toEqual(ComplianceStatus.COMPLIANT);
});

it('should set status to BLOCKED for ONBOARD action from a restricted country with no existing compliance status', async () => {
it('should set status to BLOCKED for CONNECT action from a restricted country with no existing compliance status and no wallet', async () => {
(Secp256k1.verifySignature as jest.Mock).mockResolvedValueOnce(true);
getGeoComplianceReasonSpy.mockReturnValueOnce(ComplianceReason.US_GEO);
isRestrictedCountryHeadersSpy.mockReturnValue(true);
await dbHelpers.clearData();

const response: any = await sendRequest({
type: RequestMethod.POST,
@@ -365,7 +366,7 @@ describe('ComplianceV2Controller', () => {
expect(response.body.updatedAt).toBeDefined();
});

it('should set status to FIRST_STRIKE_CLOSE_ONLY for CONNECT action from a restricted country with no existing compliance status', async () => {
it('should set status to FIRST_STRIKE_CLOSE_ONLY for CONNECT action from a restricted country with no existing compliance status and a wallet', async () => {
(Secp256k1.verifySignature as jest.Mock).mockResolvedValueOnce(true);
getGeoComplianceReasonSpy.mockReturnValueOnce(ComplianceReason.US_GEO);
isRestrictedCountryHeadersSpy.mockReturnValue(true);
@@ -446,59 +447,6 @@ describe('ComplianceV2Controller', () => {
expect(response.body.updatedAt).toBeDefined();
});

it('should be a no-op for ONBOARD action with existing COMPLIANT status', async () => {
await ComplianceStatusTable.create({
address: testConstants.defaultAddress,
status: ComplianceStatus.COMPLIANT,
});
(Secp256k1.verifySignature as jest.Mock).mockResolvedValueOnce(true);
isRestrictedCountryHeadersSpy.mockReturnValue(true);

const response: any = await sendRequest({
type: RequestMethod.POST,
path: '/v4/compliance/geoblock',
body,
expectedStatus: 200,
});

const data: ComplianceStatusFromDatabase[] = await ComplianceStatusTable.findAll({}, [], {});
expect(data).toHaveLength(1);
expect(data[0]).toEqual(expect.objectContaining({
address: testConstants.defaultAddress,
status: ComplianceStatus.COMPLIANT,
}));
expect(response.body.status).toEqual(ComplianceStatus.COMPLIANT);
});

it('should be a no-op for ONBOARD action with existing FIRST_STRIKE_CLOSE_ONLY status', async () => {
await ComplianceStatusTable.create({
address: testConstants.defaultAddress,
status: ComplianceStatus.FIRST_STRIKE_CLOSE_ONLY,
reason: ComplianceReason.US_GEO,
});
(Secp256k1.verifySignature as jest.Mock).mockResolvedValueOnce(true);
isRestrictedCountryHeadersSpy.mockReturnValue(true);

const response: any = await sendRequest({
type: RequestMethod.POST,
path: '/v4/compliance/geoblock',
body,
expectedStatus: 200,
});

const data: ComplianceStatusFromDatabase[] = await ComplianceStatusTable.findAll({}, [], {});
expect(data).toHaveLength(1);
expect(data[0]).toEqual(expect.objectContaining({
address: testConstants.defaultAddress,
status: ComplianceStatus.FIRST_STRIKE_CLOSE_ONLY,
reason: ComplianceReason.US_GEO,
}));

expect(response.body.status).toEqual(ComplianceStatus.FIRST_STRIKE_CLOSE_ONLY);
expect(response.body.reason).toEqual(ComplianceReason.US_GEO);
expect(response.body.updatedAt).toBeDefined();
});

it('should update status to CLOSE_ONLY for CONNECT action from a restricted country with existing FIRST_STRIKE status', async () => {
await ComplianceStatusTable.create({
address: testConstants.defaultAddress,
Original file line number Diff line number Diff line change
@@ -9,9 +9,12 @@ import {
ComplianceStatus,
ComplianceStatusFromDatabase,
ComplianceStatusTable,
WalletFromDatabase,
WalletTable,
} from '@dydxprotocol-indexer/postgres';
import express from 'express';
import { matchedData } from 'express-validator';
import _ from 'lodash';
import { DateTime } from 'luxon';
import {
Controller, Get, Path, Route,
@@ -28,14 +31,15 @@ import { getIpAddr } from '../../../lib/utils';
import { CheckAddressSchema } from '../../../lib/validation/schemas';
import { handleValidationErrors } from '../../../request-helpers/error-handler';
import ExportResponseCodeStats from '../../../request-helpers/export-response-code-stats';
import { ComplianceRequest, ComplianceV2Response, SetComplianceStatusRequest } from '../../../types';
import {
ComplianceRequest, ComplianceV2Response, SetComplianceStatusRequest,
} from '../../../types';
import { ComplianceControllerHelper } from './compliance-controller';

const router: express.Router = express.Router();
const controllerName: string = 'compliance-v2-controller';

export enum ComplianceAction {
ONBOARD = 'ONBOARD',
CONNECT = 'CONNECT',
VALID_SURVEY = 'VALID_SURVEY',
INVALID_SURVEY = 'INVALID_SURVEY',
@@ -235,98 +239,31 @@ router.post(
);
}

/**
* If the address doesn't exist in the compliance table:
* - if the request is from a restricted country:
* - if the action is ONBOARD, set the status to BLOCKED
* - if the action is CONNECT, set the status to FIRST_STRIKE_CLOSE_ONLY
* - else if the request is from a non-restricted country:
* - set the status to COMPLIANT
*
* if the address is COMPLIANT:
* - the ONLY action should be CONNECT. ONBOARD/VALID_SURVEY/INVALID_SURVEY are no-ops.
* - if the request is from a restricted country:
* - set the status to FIRST_STRIKE_CLOSE_ONLY
*
* if the address is FIRST_STRIKE_CLOSE_ONLY:
* - the ONLY actions should be VALID_SURVEY/INVALID_SURVEY/CONNECT. ONBOARD/CONNECT
* are no-ops.
* - if the action is VALID_SURVEY:
* - set the status to FIRST_STRIKE
* - if the action is INVALID_SURVEY:
* - set the status to CLOSE_ONLY
*
* if the address is FIRST_STRIKE:
* - the ONLY action should be CONNECT. ONBOARD/VALID_SURVEY/INVALID_SURVEY are no-ops.
* - if the request is from a restricted country:
* - set the status to CLOSE_ONLY
*/
const complianceStatus: ComplianceStatusFromDatabase[] = await
ComplianceStatusTable.findAll(
{ address: [address] },
[],
);
let complianceStatusFromDatabase: ComplianceStatusFromDatabase | undefined;
const [
complianceStatus,
wallet,
]: [
ComplianceStatusFromDatabase[],
WalletFromDatabase | undefined,
] = await Promise.all([
ComplianceStatusTable.findAll(
{ address: [address] },
[],
),
WalletTable.findById(address),
]);

const updatedAt: string = DateTime.utc().toISO();
if (complianceStatus.length === 0) {
if (isRestrictedCountryHeaders(req.headers as CountryHeaders)) {
if (action === ComplianceAction.ONBOARD) {
complianceStatusFromDatabase = await ComplianceStatusTable.upsert({
address,
status: ComplianceStatus.BLOCKED,
reason: getGeoComplianceReason(req.headers as CountryHeaders)!,
updatedAt,
});
} else if (action === ComplianceAction.CONNECT) {
complianceStatusFromDatabase = await ComplianceStatusTable.upsert({
address,
status: ComplianceStatus.FIRST_STRIKE_CLOSE_ONLY,
reason: getGeoComplianceReason(req.headers as CountryHeaders)!,
updatedAt,
});
}
} else {
complianceStatusFromDatabase = await ComplianceStatusTable.upsert({
address,
status: ComplianceStatus.COMPLIANT,
updatedAt,
});
}
} else {
complianceStatusFromDatabase = complianceStatus[0];
if (
complianceStatus[0].status === ComplianceStatus.FIRST_STRIKE ||
complianceStatus[0].status === ComplianceStatus.COMPLIANT
) {
if (
isRestrictedCountryHeaders(req.headers as CountryHeaders) &&
action === ComplianceAction.CONNECT
) {
complianceStatusFromDatabase = await ComplianceStatusTable.update({
address,
status: COMPLIANCE_PROGRESSION[complianceStatus[0].status],
reason: getGeoComplianceReason(req.headers as CountryHeaders)!,
updatedAt,
});
}
} else if (
complianceStatus[0].status === ComplianceStatus.FIRST_STRIKE_CLOSE_ONLY
) {
if (action === ComplianceAction.VALID_SURVEY) {
complianceStatusFromDatabase = await ComplianceStatusTable.update({
address,
status: ComplianceStatus.FIRST_STRIKE,
updatedAt,
});
} else if (action === ComplianceAction.INVALID_SURVEY) {
complianceStatusFromDatabase = await ComplianceStatusTable.update({
address,
status: ComplianceStatus.CLOSE_ONLY,
updatedAt,
});
}
}
}
const complianceStatusFromDatabase:
ComplianceStatusFromDatabase | undefined = await upsertComplianceStatus(
req,
action,
address,
wallet,
complianceStatus,
updatedAt,
);

const response = {
status: complianceStatusFromDatabase!.status,
reason: complianceStatusFromDatabase!.reason,
@@ -360,6 +297,102 @@ function generateAddress(pubkeyArray: Uint8Array): string {
return toBech32('dydx', ripemd160(sha256(pubkeyArray)));
}

/**
* If the address doesn't exist in the compliance table:
* - if the request is from a restricted country:
* - if the action is CONNECT and no wallet, set the status to BLOCKED
* - if the action is CONNECT and wallet exists, set the status to FIRST_STRIKE_CLOSE_ONLY
* - else if the request is from a non-restricted country:
* - set the status to COMPLIANT
*
* if the address is COMPLIANT:
* - the ONLY action should be CONNECT. VALID_SURVEY/INVALID_SURVEY are no-ops.
* - if the request is from a restricted country:
* - set the status to FIRST_STRIKE_CLOSE_ONLY
*
* if the address is FIRST_STRIKE_CLOSE_ONLY:
* - the ONLY actions should be VALID_SURVEY/INVALID_SURVEY/CONNECT. CONNECT
* are no-ops.
* - if the action is VALID_SURVEY:
* - set the status to FIRST_STRIKE
* - if the action is INVALID_SURVEY:
* - set the status to CLOSE_ONLY
*
* if the address is FIRST_STRIKE:
* - the ONLY action should be CONNECT. VALID_SURVEY/INVALID_SURVEY are no-ops.
* - if the request is from a restricted country:
* - set the status to CLOSE_ONLY
*/
// eslint-disable-next-line @typescript-eslint/require-await
async function upsertComplianceStatus(
req: express.Request,
action: ComplianceAction,
address: string,
wallet: WalletFromDatabase | undefined,
complianceStatus: ComplianceStatusFromDatabase[],
updatedAt: string,
): Promise<ComplianceStatusFromDatabase | undefined> {
if (complianceStatus.length === 0) {
if (!isRestrictedCountryHeaders(req.headers as CountryHeaders)) {
return ComplianceStatusTable.upsert({
address,
status: ComplianceStatus.COMPLIANT,
updatedAt,
});
}

// If address is restricted and is not onboarded then block
if (_.isUndefined(wallet)) {
return ComplianceStatusTable.upsert({
address,
status: ComplianceStatus.BLOCKED,
reason: getGeoComplianceReason(req.headers as CountryHeaders)!,
updatedAt,
});
}

return ComplianceStatusTable.upsert({
address,
status: ComplianceStatus.FIRST_STRIKE_CLOSE_ONLY,
reason: getGeoComplianceReason(req.headers as CountryHeaders)!,
updatedAt,
});
}

if (
complianceStatus[0].status === ComplianceStatus.FIRST_STRIKE ||
complianceStatus[0].status === ComplianceStatus.COMPLIANT
) {
if (
isRestrictedCountryHeaders(req.headers as CountryHeaders) &&
action === ComplianceAction.CONNECT
) {
return ComplianceStatusTable.update({
address,
status: COMPLIANCE_PROGRESSION[complianceStatus[0].status],
reason: getGeoComplianceReason(req.headers as CountryHeaders)!,
updatedAt,
});
}
} else if (complianceStatus[0].status === ComplianceStatus.FIRST_STRIKE_CLOSE_ONLY) {
if (action === ComplianceAction.VALID_SURVEY) {
return ComplianceStatusTable.update({
address,
status: ComplianceStatus.FIRST_STRIKE,
updatedAt,
});
} else if (action === ComplianceAction.INVALID_SURVEY) {
return ComplianceStatusTable.update({
address,
status: ComplianceStatus.CLOSE_ONLY,
updatedAt,
});
}
}

return complianceStatus[0];
}

if (config.EXPOSE_SET_COMPLIANCE_ENDPOINT) {
router.post(
'/setStatus',