Questions about index size

[Amphetaminewei]

Summary

Platform

OS: Ubuntu 2004

Faiss version: commit 52ce3f5

Installed from: compiled by myself

Faiss compilation options: cmake -DBUILD_TESTING=OFF -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_PYTHON=OFF ../

Running on: CPU

Interface: C++

Hi, I have some questions when verifying the index size.
In the material I consulted, HNSWFlat index use more memory than Flat indexe, HNSWFlat is a very memory-consuming index, but in my tests, the HNSWFlat index didn't use much more memory than the Flat index.

• vector dim：1024
• vector count：1000000
• hnsw M：16
• hnsw efConstruction：100
• hnsw efSearch：10

IndexHNSWFlat use memory 4043.84 MB, IndexFlatL2 use memory 3928.93 MB

They don't differ much, is this normal? Is it my hnsw index M that is too small and the index doesn't need more extra memory?

The code I used is as follows:

static const int DEFAULT_M = 16;
static const int DEFAULT_CONSTRUCTION_EF = 100;
static const int DEFAULT_SEARCH_EF = 10;

void test_save_hnsw_flat_index() {
    const int num_vectors = 1000000;
    const int dim = 1024;

    faiss::IndexHNSWFlat index(dim, 48);
    index.hnsw.efConstruction = DEFAULT_CONSTRUCTION_EF;

    std::vector<std::vector<float>> vectors = generate_random_normalized_vectors(num_vectors, dim);
    std::cout << "完成向量生成，当前时间: " << current_time_str() << std::endl;

    std::vector<float> batch_vector = merge_vectors(vectors);

    std::cout << "完成向量合并，当前时间: " << current_time_str() << std::endl;

    index.add(num_vectors, batch_vector.data());

    std::cout << "完成索引构建，当前时间: " << current_time_str() << std::endl;

    std::string index_file = "hnsw_index.faiss";
    faiss::write_index(&index, index_file.c_str());

    std::cout << "完成本地存储，当前时间: " << current_time_str() << std::endl;

    std::cout << "索引构建并保存到 " << index_file << std::endl;
}

void test_save_flat_index() {
    const int num_vectors = 1000000;
    const int dim = 1024;

    faiss::IndexFlatL2 index(dim);

    std::vector<std::vector<float>> vectors = generate_random_normalized_vectors(num_vectors, dim);
    std::cout << "完成向量生成，当前时间: " << current_time_str() << std::endl;

    std::vector<float> batch_vector = merge_vectors(vectors);

    std::cout << "完成向量合并，当前时间: " << current_time_str() << std::endl;

    index.add(num_vectors, batch_vector.data());

    std::cout << "完成索引构建，当前时间: " << current_time_str() << std::endl;

    std::string index_file = "flat_index.faiss";
    faiss::write_index(&index, index_file.c_str());

    std::cout << "完成本地存储，当前时间: " << current_time_str() << std::endl;

    std::cout << "索引构建并保存到 " << index_file << std::endl;
}

[mdouze]

The amount of memory used by HNSWFlat is
(vector dim) * 4 + M * 4 * 2

• overheads due to higher graph layers + geometric reallocation.