Introducing the Graph module #22

FlorentinD · 2020-06-19T16:12:25Z

Result of two consecutive projects (first one by @TimPF).

2 Graph Storage Formats (CSR and AdjacencyGraph), which use MorphStore columns for compression
Graph Algorithms: Breadth-First Search and Page-Rank
a couple of "micro-benchmarks" (maybe not really micro)
Introduces a morph-flag for the path to resource directory of an LDBC-Graph (needed for tests and benchmark)
a morph operator which saves the block-offsets
a template free column-base class (currently used by the graph formats)

Also lifts the needed C++ version to 17.

It might make sense to introduce an graph build flag.

… in test

…enerate vertices) -> works

… last attribute)

finding a current bug (fixed in the following)

.. added the offset to early

actually need to decompress the blocks one by one .. (see `morph_saving_offsets_test` for reasoning) Lead to a couple of changes: * morph_saving_offsets now accepts column_with_offsets as input * dedicated morph_saving_offset for decompression

* changing cache in csr -> not const function anymore

catching a current bug regarding the offset_column (not fixing it yet) Failure case: last vertex has no edges -> no edges added -> no offset set

slimming down graph.h unique ptr didn't make sense, as the algorithm does not own the graph here

use-case: bfs_benchmark

for evaluating impact of graph compressions

and correcting the number of block-offsets to be reserved

looks like the ldbc graph schema is not final .. they added a forum type in the dev branch

pfeiffer-tim and others added 30 commits April 30, 2019 11:25

test of commitment in CLion

e4023c0

remove test commitment in gitignore

b496864

added graph.h with structs, cmakelists.txt

1272bd8

first test file with print

7160b74

Added subdir for graph test in cmakelists

50c0446

change dir of graph test file, cmakelists changes..

e1125ce

read LDBC files into dicts (intermediate dictionary)

79c4c23

added graph structure with class graph, vertex, edge; generated graph…

934c2bd

… in test

removed edge.h (edge is struct in vertex.h); fixed printing vertex infos

f79e8ce

added some comments...

29b77fd

changed unsigned long int's to size_t

c7205e7

changed namespace 'graph -> morphstore'; replace size_t with uint64_t

f8ec509

new directory in Engine/core/storage

6f5c62c

started ldbc_import.h; changed cmake to handle experimental::filesystem

f5da8f5

little changes...

11afae8

buffer allocation for vertex files

3100ea9

vertex data structure for intermediate results in ldbc importer

a17146f

first attempt of loading ldbc-vertex-data into graph structure (i.e g…

cab0446

…enerate vertices) -> works

little changes; added comments

5c652bf

added hash function for pair as key in unordered map

a197aad

added global id lookup when ldbc generating

dfcecb0

little changes; started relation-import functionality (to be continued)

d5b72bb

forgotten comments and little changes

24915f7

working edge-generating from ldbc files

abc4750

little changes in ldbc importer

6d9924e

remove unused include's

1f3c739

added simply functionality for multi-value attributes (just write the…

61d2ff9

… last attribute)

comment-out stuff

80f1c2c

little changes; added time measuring

4da862f

declaring some function parameters as const

8e6a881

Florentin Doerre added 28 commits June 2, 2020 23:15

WIP add template free morph_saving_offsets to use for csr columns

17a996f

Improve morph_saving_offsets_test

52dbb5c

finding a current bug (fixed in the following)

Add test for morphing column blocks and fix bug regarding last block

fe9967b

Fix offset last block if it is uncompressed

0e754ff

.. added the offset to early

Add decompress_column_block function

be7abcf

Use decompress_column_block in csr format

be9e511

Cache decompressed column block

d64d01c

* changing cache in csr -> not const function anymore

Cache edgeId_column in CSR + bug fix for getting edge-ids

4a4ea87

Add instructions for LDBC_DIR flag

853db0d

Fix delete error in AdjacencyList format and correct size_of_graph func

68b544a

Assert offsets in csr make sense

0b07069

catching a current bug regarding the offset_column (not fixing it yet) Failure case: last vertex has no edges -> no edges added -> no offset set

Fix title of benchmark

252a2b2

Move degree measurement into a seperate operator and use shared ptr

0e96431

slimming down graph.h unique ptr didn't make sense, as the algorithm does not own the graph here

Do not save graph in BFS class as volatile graphs dont work then

b124286

use-case: bfs_benchmark

Add bfs benchmark

bcec970

for evaluating impact of graph compressions

Add naive page rank

61d9938

Add tests for page rank

a7b787e

Add page-rank benchmark

448cea4

Comment more code

6639a51

Fix measurement unit description in benchmark

974c756

Fix documented file names

11dedfb

Merge remote-tracking branch 'origin/master' into graph

aa1a4bf

Add a getting started section for the graph module

4943d76

Comment new morph operators

4ce2b7e

and correcting the number of block-offsets to be reserved

Allow different compression formats for the different csr columns

0c46f8f

Benchmark partial csr compressions

f031f63

Allow fallback to string if data-type could not be found in schema

6f731e5

looks like the ldbc graph schema is not final .. they added a forum type in the dev branch

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introducing the Graph module #22

Introducing the Graph module #22

FlorentinD commented Jun 19, 2020

Introducing the Graph module #22

Are you sure you want to change the base?

Introducing the Graph module #22

Conversation

FlorentinD commented Jun 19, 2020