Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider deprecating or removing the enrichment module entirely #70

Open
JuxhinDB opened this issue Nov 2, 2023 · 4 comments
Open

Consider deprecating or removing the enrichment module entirely #70

JuxhinDB opened this issue Nov 2, 2023 · 4 comments
Assignees
Labels
help wanted Extra attention is needed question Further information is requested

Comments

@JuxhinDB
Copy link
Collaborator

JuxhinDB commented Nov 2, 2023

Currently the library is a mix between async and non-async modules. The permutation engine is trivial to use and integrate, making it pleasant to work with. The enrichment module however locks into a specified implementation which is (a) inflexible and (b) a pain to maintain.

After some thoughts, it seems that enrichment does not necessarily belong in the library at all, besides perhaps some traits to help library users extend the functionality to their liking.

@JuxhinDB JuxhinDB added help wanted Extra attention is needed question Further information is requested labels Nov 2, 2023
@JuxhinDB JuxhinDB self-assigned this Nov 2, 2023
@nix2intel
Copy link

nix2intel commented Jan 16, 2025

First of @JuxhinDB thank you so much for this amazing library! i'm currently building a tool in elixir and loved the permutation engine so much that i'm going to use twistrs as a native implemented function to build the rest of my tooling off of. I found what you said to be 100% accurate and had a really difficult time getting the enrichment module to do what I wanted, may be worth splitting off into its own thing imho. You can see what I did for the data enrichment here:
https://github.com/nix2intel/domaintwistex

And seriously thank you, so happy to not be using the python version in production anymore! What you are doing is incredible and amazing.

@JuxhinDB
Copy link
Collaborator Author

Appreciate the kind words @nix2intel. Love seeing the ffi integration with elixir, had no idea it was that smooth. Might actually prompt me to think about elixir when building Have I Been Squatted's services.

Re.: enrichment, after a few years building on top of it, for the most part it's best to just integrate your data sources the way you need/want to rather than relying on a separate crate for the same reasons I mentioned above. Instead, exposing some traits that allows user to easily add new enrichment methods that interoperate nicely (e.g., having enrichment modules that have dependencies on other modules, such as http_banner depending on the domain being resolved first). Will try to find some time to work on this.

On another note @nix2intel - if you come across any permutation methods you think are worth adding, please feel free to include them. One that comes to mind is localised keywords, for example servizio instead of service, which might capture some very specific sites.

@nix2intel
Copy link

I do have a few ideas but they are very centric to the industry i work in, so might be interesting to find a way to create rules based on industries or rulesets, do ya'll have a discord or forum where everyone chats? don't want to pollute the github issue for yah @JuxhinDB.

@JuxhinDB
Copy link
Collaborator Author

Totally, feel free to join the Have I Been Squatted discord. We have a #twistrs channel in there or just chat in #general

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants