Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EPIC] V1 #28

Open
wants to merge 83 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
5be6805
puppeteer scrape address page properly
dawsbot Feb 5, 2024
514baa8
scrape account pages effectively
dawsbot Feb 5, 2024
429c95a
Add todo
dawsbot Mar 6, 2024
de8526c
Copy readme over from main
dawsbot Mar 6, 2024
7261b25
Update contributing readme
dawsbot Mar 6, 2024
135ae11
fix error parsing and add dynamic hostname generation (#18)
kylewandishin Mar 9, 2024
3715502
working log in info from dotenv. scraping pages of 25 frmo each label
kylewandishin Mar 12, 2024
75588f1
added automatic sign in and scraping capabilities for signed in user\…
kylewandishin Mar 12, 2024
b9780a2
scrape every account 'main' labels into json files
kylewandishin Mar 12, 2024
e94f3b7
pull all main accounts from etherscan. parse as json in /data\n\nclos…
kylewandishin Mar 12, 2024
3638263
add runtime automated testing for scraping and speed up pulling proce…
kylewandishin Mar 19, 2024
d008efb
fix timeout
kylewandishin Mar 19, 2024
47ebe10
data checkpoint
kylewandishin Mar 19, 2024
4618c6a
sorting added
kylewandishin Mar 19, 2024
acbebef
sorted data
kylewandishin Mar 19, 2024
ac57f73
rename data folders
kylewandishin Mar 19, 2024
0466900
pull data from all labels with all subcatagories. depreciates length …
kylewandishin Mar 19, 2024
f628968
format all code files
kylewandishin Mar 19, 2024
235e770
clean up workspace
kylewandishin Mar 19, 2024
c0f9dd6
turned etherscan pulling into a component to begin refactoring
kylewandishin Mar 19, 2024
1832675
turned etherscan-pulling into a component to begin refactoring
kylewandishin Mar 19, 2024
2d93cfc
restyle code with prettier
kylewandishin Mar 19, 2024
4961767
Move etherscan to class and add eslint
dawsbot Mar 20, 2024
e4b0073
Fresh pull without lowercase re-ordering
dawsbot Mar 20, 2024
450d9ca
improve function names
dawsbot Mar 20, 2024
2c5186f
Pull accounts
dawsbot Mar 24, 2024
cd1e4b3
Sort all tokens, improve logging
dawsbot Mar 25, 2024
f5bdfab
alternative oop approach in components folder
kylewandishin Mar 26, 2024
d03c616
Revert "alternative oop approach in components folder"
dawsbot Mar 26, 2024
60323d2
🤓 Stricter eslint rules (#32)
dawsbot Mar 26, 2024
5e2c9b4
🧪 Add unit testing via vitest (#30)
dawsbot Mar 26, 2024
367e5f7
npm audit fix
dawsbot Mar 26, 2024
fb07c71
🤓 Add pull script to package.json (#38)
kylewandishin Mar 26, 2024
b6c0f6c
✨🤓 Pull Basescan and use OOP inheritance (#39)
dawsbot Mar 26, 2024
4de8c76
✨ Pull optimism labels (#41)
dawsbot Mar 26, 2024
53b4dfa
Update readme to organize labels (#43)
dawsbot Mar 26, 2024
6dee432
✨ Combine label files in new combined.json (#44)
kylewandishin Mar 27, 2024
a89f0f9
✨ Add Arbitrum and lowercase all addresses (#46)
dawsbot Mar 28, 2024
e4f08a9
✨ celoscan init (#48)
dawsbot Apr 12, 2024
10f29da
Add celo image to readme
dawsbot Apr 12, 2024
239ce57
✨ Fix celo image in readme (#49)
dawsbot Apr 12, 2024
7d94baa
Insert last edited date to readme (#50)
dawsbot Apr 13, 2024
dc8e4af
✨ Add Gnosis chain data (#55)
kylewandishin Apr 30, 2024
490d722
✨ Add bscscan (#56)
kylewandishin Apr 30, 2024
bb973e1
📓 Add star graph to readme
dawsbot May 3, 2024
f66e979
🧪 Test BSCScan (#61)
dawsbot May 18, 2024
b5a6687
🧪 add gnosis test close #57 (#63)
kylewandishin May 20, 2024
be3cc7c
🧪🐛 Test & expand token names for arbiscan (#64)
dawsbot May 22, 2024
69b31a7
✨ Add token image etherscan (#67)
kylewandishin May 22, 2024
7f21273
🤓 Concurrently test (#69)
kylewandishin May 22, 2024
e32d5c9
🤓 migrate to bun (#71)
kylewandishin May 29, 2024
45cbca7
🤓 Replace rest of npm with bun
dawsbot May 29, 2024
c3fa9dd
Merge branch 'v1' of github.com:Earnifi/etherscan-labels into v1
dawsbot May 29, 2024
1a41382
✨ Add etherscan api (#70)
kylewandishin May 29, 2024
bab0453
🤓 Remove old npm module bloat (#72)
kylewandishin May 29, 2024
0101b4f
🤓 Add api unit tests (#74)
kylewandishin Jun 3, 2024
0b0b198
🤓 Feat: Add pre-alpha API docs
dawsbot Jun 4, 2024
d6c1f6a
add config menu (#76)
kylewandishin Jun 10, 2024
33a7fc0
🧼 Pull expanded token names for Etherscan (#79)
kylewandishin Jun 13, 2024
97227d5
Remove fileformatting
dawsbot Jun 14, 2024
44def3d
🧼 cleanup: Scan config refactor (#85)
kylewandishin Jun 14, 2024
af153cd
✨ feat: Combine all accounts from fs into one json object (#90)
dawsbot Jun 17, 2024
40edf8a
🧪 chore: Test filesystem reading (#91)
dawsbot Jun 19, 2024
2011ec2
Rename to Eth Labels (#92)
dawsbot Jun 19, 2024
9cf94c0
📓 feat: add rest of chain logos to readme (#88)
kylewandishin Jun 19, 2024
9eb9d09
✨ feat: accounts label searching api init
michaelgreen06 Jun 23, 2024
ffbe616
🐛 Pull in PORT from process.env for remote deployments
dawsbot Jun 24, 2024
f850111
🐳 Dockerize init (#97)
dawsbot Jun 24, 2024
a999124
🤓 feat: sqlite as data storage replicate (#93)
dawsbot Jun 26, 2024
dafae2e
🤓 Fix spelling in error message
dawsbot Jun 26, 2024
8406a7b
📓 feat: added loadAllTokensFromFS() and updated the address api so it…
michaelgreen06 Jun 26, 2024
8e33887
🤓 stricter eslint
dawsbot Jun 28, 2024
4226962
🤓 Remove rollup from deps
dawsbot Jun 28, 2024
d183fc5
🤓 Remove excess typing
dawsbot Jun 28, 2024
98b046a
Move from vitest to bun:test
dawsbot Jun 28, 2024
2a3aa9c
✨ Move to sqlite source of truth (#98)
dawsbot Jun 30, 2024
f88858d
Add labels API endpoint (#100)
dawsbot Jun 30, 2024
61ee228
🐛 fix: Bun run pull (no more missing tsx)
dawsbot Jul 1, 2024
846c741
✨ Pull missing tokens and addresses on-chain (#101)
dawsbot Jul 3, 2024
71d8a50
🐛 Deduplicate labels on GET to /labels (#102)
dawsbot Jul 3, 2024
0015700
🧼 Remove all browser related code & pull full pages for tokens (#103)
dawsbot Jul 3, 2024
85a15be
🤓 move html fetching to shared function
dawsbot Jul 5, 2024
bdbdccc
Api search (#104)
kylewandishin Jul 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions .eslintrc.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
module.exports = {
ignorePatterns: ["lib/**", "**/*.js"],
parser: "@typescript-eslint/parser",
plugins: ["@typescript-eslint", "tsdoc"],
extends: [
"eslint:recommended",
"plugin:@typescript-eslint/recommended-type-checked",
"prettier",
],
parserOptions: {
project: ["./tsconfig.json"],
},
env: {
node: true,
},
rules: {
"@typescript-eslint/no-unnecessary-boolean-literal-compare": "error",
"@typescript-eslint/no-unnecessary-condition": "error",
"@typescript-eslint/no-unnecessary-qualifier": "error",
"@typescript-eslint/no-unnecessary-template-expression": "error",
"@typescript-eslint/no-unnecessary-type-arguments": "error",
"@typescript-eslint/no-unnecessary-type-assertion": "error",
"@typescript-eslint/no-unnecessary-type-constraint": "error",
"@typescript-eslint/explicit-member-accessibility": [
"error",
{
accessibility: "explicit",
overrides: {
accessors: "explicit",
constructors: "explicit",
methods: "explicit",
properties: "explicit",
parameterProperties: "explicit",
},
},
],
"no-shadow": "error",
"@typescript-eslint/consistent-type-imports": "error",
"@typescript-eslint/naming-convention": [
"error",
{
selector: "variable",
types: ["boolean"],
format: ["PascalCase", "UPPER_CASE"],
prefix: [
"is",
"should",
"has",
"can",
"did",
"will",
"does",
"IS",
"SHOULD",
"HAS",
],
},
{
selector: "typeLike",
format: ["PascalCase"],
},
],
"@typescript-eslint/array-type": ["error", { default: "generic" }],
"new-cap": "warn",
"tsdoc/syntax": "error",
},
};
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,8 @@ node_modules
dist
lib
types
.env
.eslintcache

package-lock.json
yarn.lock
5 changes: 3 additions & 2 deletions .husky/pre-commit
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#!/bin/sh
. "$(dirname "$0")/_/husky.sh"

npx lint-staged
npm run test
bun run scripts/db/generate-db-json.ts
npx lint-staged --concurrent false
npx concurrently -g -n "eslint,test:unit,build" -c "#341BAB,#A6CC66,#3077C6" "bun run lint" "bun test unit" "bun run build"
16 changes: 16 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Use the offical Bun image
FROM oven/bun as base

# Set the Docker working directory as /usr/src/app
# Copy everything from here into Docker's /usr/src/app
WORKDIR /usr/src/app
COPY . /usr/src/app

FROM base AS install
RUN cd /usr/src/app && bun install --frozen-lockfile

# The port that Elysia will listen on
EXPOSE 3000

# Run the Elysia webserver
CMD bun run api/index.ts
116 changes: 116 additions & 0 deletions api/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
import { swagger } from "@elysiajs/swagger";
import { Elysia, t } from "elysia";
import type {
AccountSearchOptions,
TokenSearchOptions,
} from "../scripts/db/types";
import { selectAllLabels } from "./services/select-all-labels";
import {
selectMatchingLabels,
selectMatchingLabelsFromQueryAccounts,
selectMatchingLabelsFromQueryTokens,
} from "./services/select-matching-labels";

const PORT = process.env.PORT || 3000;
export const app = new Elysia();

app.use(
swagger({
documentation: {
info: {
title: "Eth Labels API",
version: "0.0.0",
},
},
}),
);

app.get("/labels", () => {
return selectAllLabels();
});
app.get(
"/labels/:address",
async ({ params }) => {
const { address } = params;
return selectMatchingLabels(address);
},
{ params: t.Object({ address: t.String() }) },
);

app.get(
"/accounts",
async ({ query }) => {
const {
chainId,
address,
label,
nameTag,
offset,
limit,
}: AccountSearchOptions = query;
const conditions: AccountSearchOptions = {
chainId,
address,
label,
nameTag,
offset,
limit,
};
return selectMatchingLabelsFromQueryAccounts(conditions);
},
{
query: t.Object({
chainId: t.Optional(t.String()),
address: t.Optional(t.String()),
label: t.Optional(t.String()),
nameTag: t.Optional(t.String()),
offset: t.Optional(t.String()),
limit: t.Optional(t.String()),
}),
},
);

app.get(
"/tokens",
({ query }) => {
const {
chainId,
address,
label,
name,
website,
symbol,
offset,
limit,
}: TokenSearchOptions = query;
const conditions: TokenSearchOptions = {
chainId,
address,
label,
name,
website,
symbol,
offset,
limit,
};
return selectMatchingLabelsFromQueryTokens(conditions);
},
{
query: t.Object({
chainId: t.Optional(t.String()),
address: t.Optional(t.String()),
label: t.Optional(t.String()),
name: t.Optional(t.String()),
website: t.Optional(t.String()),
symbol: t.Optional(t.String()),
offset: t.Optional(t.String()),
limit: t.Optional(t.String()),
}),
},
);

app.get("/health", () => "OK");

app.listen(PORT, () => {
console.log(`Listening on port ${PORT}. Open /swagger to see the API docs.`);
});
66 changes: 66 additions & 0 deletions api/index.unit.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
import { describe, expect, it } from "bun:test";
import { z } from "zod";
import { app } from ".";

const fetchLocally = async (path: string) => {
return app
.handle(new Request(`http://localhost:3000${path}`))
.then((res) => res.json());
};
describe("Elysia", () => {
it("/labels", async () => {
const labels = z.array(z.string()).parse(await fetchLocally("/labels"));

expect(labels.length).toBeGreaterThan(702);
expect(labels).toContain("coinbase");
});

it("/accounts", async () => {
const coinbaseAccounts = z
.array(z.object({ address: z.string(), nameTag: z.string() }))
.parse(await fetchLocally("/accounts?nameTag=cOinBase%208&chainId=1"));

expect(coinbaseAccounts.length).toBe(1);
expect(coinbaseAccounts).toContainEqual({
address: "0x02466e547bfdab679fc49e96bbfc62b9747d997c",
nameTag: "Coinbase 8",
});

const accounts = z
.array(z.object({ address: z.string(), nameTag: z.string() }))
.parse(
await fetchLocally(
"/accounts?address=0x000037bb05b2Cef17c6469f4BCdb198826CE0000",
),
);

expect(accounts.length).toBe(5);
expect(accounts).toContainEqual({
address: "0x000037bb05b2cef17c6469f4bcdb198826ce0000",
nameTag: "Fake_Phishing8",
});
});

it("/tokens", async () => {
const tokens = z
.array(
z.object({
address: z.string(),
label: z.string(),
name: z.string(),
symbol: z.string(),
}),
)
.parse(
await fetchLocally("/tokens?name=UbEsWaP&symbol=ubE"), // test case sensitivity
);

expect(tokens.length).toBe(3);
expect(tokens).toContainEqual({
address: "0x00be915b9dcf56a3cbe739d9b9c202ca692409ec",
label: "defi",
name: "Ubeswap",
symbol: "UBE",
});
});
});
12 changes: 12 additions & 0 deletions api/services/select-all-labels.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import { AccountsRepository } from "../../scripts/db/repositories/AccountsRepository";
import { TokensRepository } from "../../scripts/db/repositories/TokensRepository";

export const selectAllLabels = async (): Promise<ReadonlyArray<string>> => {
const [allTokenLabels, allAccountLabels] = await Promise.all([
TokensRepository.selectAllLabels(),
AccountsRepository.selectAllLabels(),
]);
const sortedLabels = allTokenLabels.concat(allAccountLabels).sort();
// deduplicate repeats
return Array.from(new Set(sortedLabels));
};
57 changes: 57 additions & 0 deletions api/services/select-matching-labels.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
import type { Address } from "viem";
import { createPublicClient, http, isAddress } from "viem";
import { mainnet } from "viem/chains";
import { normalize } from "viem/ens";
import { AccountsRepository } from "../../scripts/db/repositories/AccountsRepository";
import { TokensRepository } from "../../scripts/db/repositories/TokensRepository";
import type {
AccountSearchOptions,
TokenSearchOptions,
} from "../../scripts/db/types";

const publicClient = createPublicClient({
chain: mainnet,
transport: http(),
});
async function findMatchingRows(address: Address) {
address = address.toLowerCase() as Address;
const matchingRows = await Promise.all([
TokensRepository.selectTokensByAddress(address),
AccountsRepository.selectAccountsByAddress(address),
]);
return matchingRows.flat();
}

export const selectMatchingLabels = async (address: string) => {
if (address.includes(".")) {
try {
const ensAddress = await publicClient.getEnsAddress({
name: normalize(address),
});
if (ensAddress === null) {
return { error: "ENS address not found" };
}
return findMatchingRows(ensAddress);
} catch (error) {
console.error("Error resolving ENS address:", error);
return { error: "Failed to resolve ENS address" };
}
} else {
if (!isAddress(address)) {
return { error: "Invalid address format" };
}
return findMatchingRows(address);
}
};

export const selectMatchingLabelsFromQueryAccounts = (
accounts: AccountSearchOptions,
) => {
return AccountsRepository.selectAccountsByObj(accounts);
};

export const selectMatchingLabelsFromQueryTokens = (
tokens: TokenSearchOptions,
) => {
return TokensRepository.selectTokensByObj(tokens);
};
Binary file added bun.lockb
Binary file not shown.
Loading