Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace CIDR list with java cdr-ip-trie for faster lookups #12

Closed
wants to merge 4 commits into from

Conversation

brandond
Copy link

The current code is very slow, in that it simply iterates the list of CIDRs and checks them individually to see if the IP is contained within it. This PR replaces the list with cdr-ip-trie:

The CIDR Trie type provides lookups that range from 10-50x faster than using a TreeMap, and 100-500x faster than using sorted list (but at a cost of using around 20-30% more memory than a TreeMap). The Trie scales much better, getting faster and using less memory in comparison with a TreeMap the more CIDR's are added to it (tested with one hundred million unique CIDR's).

There is a minor breaking change here in that network lists specified within the logstash config (as opposed to sourced from an external file) are no longer sprintf'd from event fields, but since this was never documented as supported, I'm not sure it matters. If this needs to be retained, I can add add it back, with some additional refactoring.

@brandond
Copy link
Author

Oh, I guess this totally breaks IPv6 support too. I opened veqryn/cidr-ip-trie#7 to add IPv6 support to the upstream library, but I understand if this is a blocker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants