Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement KANNDI #4

Open
DmitryKey opened this issue Oct 3, 2021 · 1 comment
Open

Implement KANNDI #4

DmitryKey opened this issue Oct 3, 2021 · 1 comment

Comments

@DmitryKey
Copy link
Owner

Implement KANNDI
DmitryKey added a commit that referenced this issue Oct 3, 2021
@DmitryKey
#4 sharding by distance + hnsw indices per shard
ba14663
DmitryKey added a commit that referenced this issue Oct 3, 2021
@DmitryKey
#4 refactoring + fixes to logic + helper script
3db1039
DmitryKey added a commit that referenced this issue Oct 9, 2021
@DmitryKey
#4 added debug logging
7a91c79
DmitryKey added a commit that referenced this issue Oct 10, 2021
@DmitryKey
#4 1. fixed distance computation logic 2. added distance multiplier
2c83531
DmitryKey added a commit that referenced this issue Oct 10, 2021
@DmitryKey
#4 sample code for illustrating usage of the distance_matrix() method
4ee4a19
DmitryKey added a commit that referenced this issue Oct 12, 2021
@DmitryKey
#4 fixed with Leo
dfe6046
DmitryKey added a commit that referenced this issue Oct 16, 2021
@DmitryKey
#4 refactoring + moved to own running shard ids to avoid key collisions
f15de68
DmitryKey added a commit that referenced this issue Oct 16, 2021
@DmitryKey
#4 completed refactoring ;)
d50e28d
DmitryKey added a commit that referenced this issue Oct 17, 2021
@DmitryKey
#4 described an experiment with 100M sample of BIGANN
f428322
DmitryKey added a commit that referenced this issue Oct 17, 2021
@DmitryKey
#4 described an experiment with 100M sample of BIGANN
30bf3b6
DmitryKey added a commit that referenced this issue Oct 17, 2021
@DmitryKey
#4 described an experiment with 100M sample of BIGANN
a9a1b72
DmitryKey added a commit that referenced this issue Oct 17, 2021
@DmitryKey
#4 described an experiment with 100M sample of BIGANN
1d9960f
DmitryKey added a commit that referenced this issue Oct 17, 2021
@DmitryKey
#4 added minimum saturation that qualifies for a shard
fdab720
DmitryKey added a commit that referenced this issue Oct 18, 2021
@DmitryKey
#4 fixed typos
cc130c8
DmitryKey added a commit that referenced this issue Oct 18, 2021
@DmitryKey
#4 1. added distance recomputation logic, and distance adjustment wit…
3bc03c0 
…h a multiplier in case new distance did not increase 2. saved special shard
DmitryKey added a commit that referenced this issue Oct 21, 2021
@DmitryKey
#4 1. used del to see the impact on memory 2. added memory tracer
225eb58
DmitryKey added a commit that referenced this issue Oct 22, 2021
@DmitryKey
#4 impl'd argparse to parameterize input, output and number of shards
0fd9fe5
DmitryKey added a commit that referenced this issue Oct 22, 2021
@DmitryKey
#4 implemented gc collection
71dcac5
DmitryKey added a commit that referenced this issue Oct 24, 2021
@DmitryKey
#4 added dtype parameterization; added more one-liners for bigann and…
90790be 
… ssnpp datasets
DmitryKey added a commit that referenced this issue Oct 26, 2021
@DmitryKey
#4 implemented hsnw graph for centroids (shard seeds)
844407c
DmitryKey added a commit that referenced this issue Oct 26, 2021
@DmitryKey
#4 indexing wrapper script
26349b7
DmitryKey added a commit that referenced this issue Oct 26, 2021
@DmitryKey
#4 first stab at searcher, mostly copy of Max's code
4a54c77
DmitryKey added a commit that referenced this issue Oct 27, 2021
@DmitryKey
#4 fixed couple of bugs and made searcher work
721b4be
DmitryKey added a commit that referenced this issue Oct 27, 2021
@DmitryKey
#4 logged SSNPP experiment to README.md
236e4a4
DmitryKey added a commit that referenced this issue Oct 27, 2021
@DmitryKey
#4 wrapper scripts for indexing and searching
843581c
DmitryKey added a commit that referenced this issue Oct 27, 2021
@DmitryKey
#4 introduced in-loop points to reduce mem usage
e0b159e
DmitryKey added a commit that referenced this issue Oct 27, 2021
@DmitryKey
#4 trivial cleanup
76aef57
DmitryKey added a commit that referenced this issue Oct 30, 2021
@DmitryKey
#4 1. distance_matrix -> pdist 2. added transposition before reshapin…
bc8ac93 
…g to avoid array copy
DmitryKey added a commit that referenced this issue Oct 30, 2021
@DmitryKey
#4 use numpy pre-allocation for a pair of points
d9d1343
DmitryKey added a commit that referenced this issue Oct 30, 2021
@DmitryKey
#4 use numpy pre-allocation for shard and special_shard points
2dfd116
DmitryKey added a commit that referenced this issue Oct 31, 2021
@DmitryKey
#4 use numpy pre-allocation for shard_point_ids
8fc0c0a
DmitryKey added a commit that referenced this issue Oct 31, 2021
@DmitryKey
#4 bug fix + reduced logging
bd8a14f
DmitryKey added a commit that referenced this issue Nov 1, 2021
@DmitryKey
#4 mem release moved under the for loop
dc6efdf
DmitryKey added a commit that referenced this issue Nov 2, 2021
@DmitryKey
#4 implemented interval tree approach instead of keeping int ids in a…
57fff2a 
…n ever growing set
DmitryKey added a commit that referenced this issue Nov 4, 2021
@DmitryKey
#4 logging + typo fixing
a09a340
DmitryKey added a commit that referenced this issue Nov 4, 2021
@DmitryKey
#4 super-simplified the tracking of processed point ids -- thanks Max!
88a82dd
@DmitryKey
Copy link
Owner Author

Reached 1B points of BIGANN indexed in 14h:02m:29s.

DmitryKey added a commit that referenced this issue Nov 6, 2021
@DmitryKey
#4 bug fix for centroids and shards
e1d0944
DmitryKey added a commit that referenced this issue Nov 7, 2021
@DmitryKey
#4 refactored main method shard_by_dist
818aa4e
DmitryKey added a commit that referenced this issue Nov 7, 2021
@DmitryKey
#4 dropped obsolete code: fixes a bug with over-sharding
d95e083
DmitryKey added a commit that referenced this issue Nov 10, 2021
@DmitryKey
#4 moved to global shard id
7ed6f49
DmitryKey added a commit that referenced this issue Nov 15, 2021
@DmitryKey
#4 1. fix for global shard id 2. switched from copy() to view()
3e93232
DmitryKey added a commit that referenced this issue Nov 15, 2021
@DmitryKey
#4 inlined batch method and fixed a bug!
84f7dc7
DmitryKey added a commit that referenced this issue Nov 15, 2021
@DmitryKey
#4 filename fix
d46ef6d
DmitryKey added a commit that referenced this issue Nov 17, 2021
@DmitryKey
#4 got rid of shard object wrapper
2f38b5c
DmitryKey added a commit that referenced this issue Nov 17, 2021
@DmitryKey
#4 another fix for special shard id
740e8b6
DmitryKey added a commit that referenced this issue Nov 20, 2021
@DmitryKey
#4 nmslib->hnswlib
af7a563
DmitryKey added a commit that referenced this issue Dec 10, 2021
@DmitryKey
#4 pdist -> cdist; fixed logic for special shard; added numba (Alex S…
dbbe1cc 
…htuchkin's suggestion)
DmitryKey added a commit that referenced this issue Dec 10, 2021
@DmitryKey
#4 pdist -> cdist; fixed logic for special shard; added numba (Alex S…
89444a4 
…htuchkin's suggestion)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@DmitryKey