c++ usage as a library

[feeka]

Dear metagraph community,
I was trying to use the succinct de Bruijn graph of the project for graph construction, traversals and etc. After multiple trials of including the library(built statically, included the cmake file, tried to modify, almost every way possible) I simply gave up. It would be great to have some kind of a tutorial or manual on how to configure the library for external c++ project.
Best regards.

[hmusta]

Thanks for your interest! You can use ExternalProject_Add to add MetaGraph to your project, then link to libmetagraph-core.a

[hmusta]

If you run into any issues with my suggestion, could you please post your CMakeLists.txt here?

[feeka]

Dear @hmusta,

thank you very much for the reply. Unfortunately I got an error:
In file included from /vol/d/development/git/cat2/build/metagraph-src/metagraph/src/graph/representation/succinct/dbg_succinct.hpp:4,
from /vol/d/development/git/cat2/src/main.cpp:4:
/vol/d/development/git/cat2/build/metagraph-src/metagraph/src/common/vectors/bit_vector.hpp:6:10: fatal error: sdsl/int_vector.hpp: No such file or directory
6 | #include <sdsl/int_vector.hpp>
| ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/MyExecutable.dir/build.make:76: CMakeFiles/MyExecutable.dir/src/main.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:138: CMakeFiles/MyExecutable.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

Here is my CMakeLists.txt using ExternalProject_Add command:

cmake_minimum_required(VERSION 3.19)
project(MyExecutable)

include(ExternalProject)

ExternalProject_Add(metagraph
    GIT_REPOSITORY https://github.com/ratschlab/metagraph.git
    GIT_TAG master 
    SOURCE_DIR "${CMAKE_BINARY_DIR}/metagraph-src"
    BINARY_DIR "${CMAKE_BINARY_DIR}/metagraph-build"
    CONFIGURE_COMMAND "${CMAKE_COMMAND}" -G "${CMAKE_GENERATOR}" "<SOURCE_DIR>/metagraph"
    BUILD_COMMAND "${CMAKE_COMMAND}" --build "<BINARY_DIR>"
    INSTALL_COMMAND ""
    TEST_COMMAND ""
    STEP_TARGETS build
)

add_executable(MyExecutable src/main.cpp)

add_dependencies(MyExecutable metagraph)

target_include_directories(MyExecutable PRIVATE "${CMAKE_BINARY_DIR}/metagraph-src/metagraph/src")

target_link_libraries(MyExecutable PRIVATE metagraph-core)

Thanks for the help again,
Fikrat

[feeka]

Additional point: seems like my configuration is missing the external libraries that are provided by metagraph. My setup in general might be faulty.

[adamant-pwn]

Hi @feeka! It seems that you're missing installation of sdsl-lite step, which is currently not implemented in metagraph's CMakeLists.txt and should be done separately. Please let us know if this helps.

[feeka]

Hello @adamant-pwn , yes I have already executed this step as well. But, I think it is not only this(sdsl) library that causes trouble but other libraries as well. I checked it by exchanging the lines with #includes and re-executed again. I got the same error but with a different library.

[adamant-pwn]

I tried reproducing it on my side. Here's what I did:

mkdir external_test
cd external_test
mkdir src
nano src/main.cpp # just write int main() {} there
nano CMakeLists.txt # Use your CMakeLists.txt
mkdir build
cmake ..
make -j

This resulted into

/usr/bin/ld: cannot find -lsdsl: No such file or directory
collect2: error: ld returned 1 exit status

After that, I did

cd metagraph-src
pushd metagraph/external-libraries/sdsl-lite
./install.sh $PWD
popd
cd ..
make -j

This resulted into

[100%] Linking CXX executable MyExecutable
/usr/bin/ld: cannot find -lmetagraph-core: No such file or directory

Which is because you didn't specify link path in CMakeLists.txt. I've added target_link_directories(MyExecutable PRIVATE "${CMAKE_BINARY_DIR}/metagraph-build") to CMakeLists.txt, and it built correctly after that.

Could you try following the same steps? You may want to integrate sdsl-lite installation in your CMakeLists.txt though, if you want to later ship this as a standalone software. If you still experience some difficulties, please provide some more context on what you're doing exactly and what kind of errors you get.

[adamant-pwn]

UPD: Okay, I managed to reproduce your error by adding #include <graph/representation/succinct/dbg_succinct.hpp> into main.cpp. It seems that this is due to misconfiguration of include directories. For example, adding target_include_directories(MyExecutable PRIVATE "${CMAKE_BINARY_DIR}/metagraph-src/metagraph/external-libraries/sdsl-lite/include") to CMakeLists.txt fixes this, but then it produces another error for another library.

[adamant-pwn]

I will look into what's the proper way to handle it, for now maybe you can try adding things from

metagraph/metagraph/CMakeLists.txt

Lines 260 to 273 in 82f35fc

	include_directories(
	PUBLIC
	external-libraries/KMC/kmc_api
	external-libraries/sdsl-lite/include
	external-libraries/hopscotch-map/include
	external-libraries/ordered-map/include
	external-libraries/cpp_progress_bar
	external-libraries/Priority-Deque
	external-libraries/rollinghashcpp
	external-libraries/spdlog/include
	external-libraries/zlib
	external-libraries/sdust
	${PROJECT_SOURCE_DIR}/src
	)

to your CMakeLists.txt? Most likely a more proper way to do it would be to provide install directory to ExternalProject_Add, I will update with further details when I figure it out.

[adamant-pwn]

@hmusta is it really the intended way to install headers?

metagraph/metagraph/CMakeLists.txt

Lines 496 to 504 in 82f35fc

	file(GLOB header_files
	"src/*.hpp"
	"src/*/*.hpp"
	"src/*/*/*.hpp"
	)
	install(TARGETS metagraph-core DESTINATION lib)
	install(FILES ${header_files} DESTINATION include/metagraph)
	install(TARGETS metagraph DESTINATION bin)

I feel like a proper way would be to do something like

install(DIRECTORY ${CMAKE_SOURCE_DIR}/src/
        DESTINATION include
        FILES_MATCHING PATTERN "*.hpp")

And maybe also do the same with the stuff in external-libraries? So that the header directory structure (and also headers that are more than 3 levels deep) is not lost and one can use /include folder from the install dir as an include directory.

[feeka]

@adamant-pwn thank you very much for the suggestions. I am going to try it and report it as well.

[adamant-pwn]

@feeka you can try using #489 with the following CMakeLists.txt:

cmake_minimum_required(VERSION 3.19)
project(MyExecutable)

include(ExternalProject)

ExternalProject_Add(metagraph
    GIT_REPOSITORY https://github.com/ratschlab/metagraph.git
    GIT_TAG include_dirs
    SOURCE_DIR "${CMAKE_BINARY_DIR}/metagraph-src"
    BINARY_DIR "${CMAKE_BINARY_DIR}/metagraph-build"
    CONFIGURE_COMMAND
        ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR} <SOURCE_DIR>/metagraph
        -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR}/metagraph-install
        COMMAND ${CMAKE_COMMAND} -E chdir <SOURCE_DIR>/metagraph/external-libraries/sdsl-lite bash ./install.sh .
    STEP_TARGETS build
)

add_executable(MyExecutable src/main.cpp)

add_dependencies(MyExecutable metagraph)

target_include_directories(MyExecutable PRIVATE "${CMAKE_BINARY_DIR}/metagraph-install/include")
target_link_directories(MyExecutable PRIVATE "${CMAKE_BINARY_DIR}/metagraph-install/lib")

target_link_libraries(MyExecutable PRIVATE metagraph-core)

You might want to provide D_DNA_GRAPH to your compilation command (or write #define _DNA_GRAPH 1 in your code?).

For some reason you might need to call make -j twice, as the first one would fail (presumably related to how sdsl is configured). Even more unfortunately, when you get past issues with include files, you may get into linking issues.

[feeka]

Dear @adamant-pwn
thanks for suggestions. To be honest I am not even past the compilation step... Still can't figure it out

[adamant-pwn]

Hi @feeka, thanks for the update! Could you please provide some details on what exactly you're doing and what kind of errors you get?

[feeka]

Hello @adamant-pwn, sorry for late reply. So, I have replicated CMakeLists.txt as you suggested. However even during the compilation I got following: make[5]: *** [external-libraries/googletest/googletest/CMakeFiles/gtest.dir/build.make:76: external-libraries/googletest/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o] Error 1
make[4]: *** [CMakeFiles/Makefile2:979: external-libraries/googletest/googletest/CMakeFiles/gtest.dir/all] Error 2
make[3]: *** [Makefile:146: all] Error 2
make[2]: *** [CMakeFiles/metagraph-build.dir/build.make:73: metagraph-prefix/src/metagraph-stamp/metagraph-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:112: CMakeFiles/metagraph-build.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

I think something with gtest. It seems like metagraph itself compiled but the tests that are later on introduced are not. I am not sure though.

EDIT: I have commented out the part from #UNIT TEST to the end and it compiled. But right now, there is another error: /usr/bin/ld: CMakeFiles/moon.dir/src/main.cpp.o: in function spdlog::logger::sink_it_(spdlog::details::log_msg const&)': main.cpp:(.text._ZN6spdlog6logger8sink_it_ERKNS_7details7log_msgE[_ZN6spdlog6logger8sink_it_ERKNS_7details7log_msgE]+0x26b): undefined reference to fmt::v10::vformat[abi:cxx11](fmt::v10::basic_string_view, fmt::v10::basic_format_args<fmt::v10::basic_format_context<fmt::v10::appender, char> >)'
/usr/bin/ld: CMakeFiles/moon.dir/src/main.cpp.o: in function spdlog::logger::flush_()': main.cpp:(.text._ZN6spdlog6logger6flush_Ev[_ZN6spdlog6logger6flush_Ev]+0x262): undefined reference to fmt::v10::vformat[abi:cxx11](fmt::v10::basic_string_view, fmt::v10::basic_format_args<fmt::v10::basic_format_context<fmt::v10::appender, char> >)'
/usr/bin/ld: CMakeFiles/moon.dir/src/main.cpp.o: in function std::make_unsigned<long>::type fmt::v10::detail::to_unsigned<long>(long)': main.cpp:(.text._ZN3fmt3v106detail11to_unsignedIlEENSt13make_unsignedIT_E4typeES4_[_ZN3fmt3v106detail11to_unsignedIlEENSt13make_unsignedIT_E4typeES4_]+0x25): undefined reference to fmt::v10::detail::assert_fail(char const*, int, char const*)'
/usr/bin/ld: CMakeFiles/moon.dir/src/main.cpp.o: in function fmt::v10::detail::format_decimal_result<char*> fmt::v10::detail::format_decimal<char, unsigned int>(char*, unsigned int, int)': main.cpp:(.text._ZN3fmt3v106detail14format_decimalIcjEENS1_21format_decimal_resultIPT_EES5_T0_i[_ZN3fmt3v106detail14format_decimalIcjEENS1_21format_decimal_resultIPT_EES5_T0_i]+0x3d): undefined reference to fmt::v10::detail::assert_fail(char const*, int, char const*)'
/usr/bin/ld: CMakeFiles/moon.dir/src/main.cpp.o: in function fmt::v10::detail::format_decimal_result<char*> fmt::v10::detail::format_decimal<char, unsigned long>(char*, unsigned long, int)': main.cpp:(.text._ZN3fmt3v106detail14format_decimalIcmEENS1_21format_decimal_resultIPT_EES5_T0_i[_ZN3fmt3v106detail14format_decimalIcmEENS1_21format_decimal_resultIPT_EES5_T0_i]+0x40): undefined reference to fmt::v10::detail::assert_fail(char const*, int, char const*)'
/usr/bin/ld: CMakeFiles/moon.dir/src/main.cpp.o: in function std::back_insert_iterator<fmt::v10::basic_memory_buffer<char, 250ul, std::allocator<char> > > fmt::v10::vformat_to<std::back_insert_iterator<fmt::v10::basic_memory_buffer<char, 250ul, std::allocator<char> > >, 0>(std::back_insert_iterator<fmt::v10::basic_memory_buffer<char, 250ul, std::allocator<char> > >, fmt::v10::basic_string_view<char>, fmt::v10::basic_format_args<fmt::v10::basic_format_context<fmt::v10::appender, char> >)': main.cpp:(.text._ZN3fmt3v1010vformat_toISt20back_insert_iteratorINS0_19basic_memory_bufferIcLm250ESaIcEEEELi0EEET_S7_NS0_17basic_string_viewIcEENS0_17basic_format_argsINS0_20basic_format_contextINS0_8appenderEcEEEE[_ZN3fmt3v1010vformat_toISt20back_insert_iteratorINS0_19basic_memory_bufferIcLm250ESaIcEEEELi0EEET_S7_NS0_17basic_string_viewIcEENS0_17basic_format_argsINS0_20basic_format_contextINS0_8appenderEcEEEE]+0x42): undefined reference to void fmt::v10::detail::vformat_to(fmt::v10::detail::buffer&, fmt::v10::basic_string_view, fmt::v10::detail::vformat_args::type, fmt::v10::detail::locale_ref)'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/moon.dir/build.make:97: moon] Error 1
make[1]: *** [CMakeFiles/Makefile2:138: CMakeFiles/moon.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

Not sure why...
Trying to backtrack from the source code in dbg_succinct
Thanks,
Fikrat