Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
1136a2b
I propose to empirically find the m_maxSrs (maximum number of send re…
KADichev Aug 20, 2024
219372a
Decrease the number of messages to use for same reason as the decreas…
KADichev Aug 20, 2024
37cd6eb
Trying to modernize LPF to use FindGTest/GoogleTest combination, but …
KADichev Aug 27, 2024
2886606
Make tests compile again
KADichev Aug 28, 2024
c7dbc7d
In a middle of a big mess of changes, which I hope will end well
KADichev Aug 30, 2024
9eab088
Working my way through Gtest-ifying the tests
KADichev Aug 30, 2024
47294a7
Finished porting functional tests to gtest
KADichev Aug 30, 2024
359c512
Added a script converting the source line with P into a process count…
KADichev Aug 30, 2024
7319765
Fixing how the default process count is parsed (some parsing errors) …
KADichev Aug 30, 2024
02b14d8
Fix reading in probe argument, plus use lpfrun now instead of mpirun …
KADichev Sep 4, 2024
e8052c1
Finished with the tests/functional directory, tests passing. Now star…
KADichev Sep 4, 2024
71c1785
Commit current state as I can't deal with this enormous change
KADichev Sep 6, 2024
2c98499
Compiles again, will not run okay because of EXPECT_DEATH + MPI
KADichev Sep 6, 2024
e2a758d
Slow progress, now I need to implement in my Python script the differ…
KADichev Sep 6, 2024
7220e76
Almost got it, now need to fix the debug tests not to issue EXPECT_DE…
KADichev Sep 6, 2024
6f84f8f
Use GoogleTest but without death tests
KADichev Sep 8, 2024
80f0392
Got IB Verbs tests to work again by setting LPF_MPI_AUTO_INITIALIZE=0…
KADichev Sep 10, 2024
758b8de
All tests passing now - omitting the huge runs
KADichev Sep 10, 2024
4021f1f
Rename c99 folder to collectives, as the folder tests collectives, an…
KADichev Sep 10, 2024
703fde7
First step towards making it work for many engines
KADichev Sep 11, 2024
d6eebab
Go back to only ibverbs for now, have to think how to fix this
KADichev Sep 11, 2024
7081c78
This version runs all tests, but fails because I need one more fix --…
KADichev Sep 13, 2024
3cd577a
Oops, missing test
KADichev Sep 13, 2024
8cdfa67
Working on having different aborts for MPI and pthreads. Unfortunatel…
KADichev Sep 14, 2024
4db403d
I think I figured how to tell hybrid engine to call MPI abort without…
KADichev Sep 14, 2024
6efc47e
Request CMake 3.29 if building with tests, and clean up a bit bootstr…
KADichev Sep 16, 2024
2f42d4c
Improve Pthread abort to return exit(6) instead of calling std::abort…
KADichev Sep 16, 2024
afa1043
I am for now removing the gtest_discover_tests call because ultimatel…
KADichev Sep 17, 2024
15aea88
Eliminate remaining DEATH statements in debug folder
KADichev Sep 17, 2024
bfafa5f
A very annoying bug that took ages to find.
KADichev Sep 17, 2024
33c481d
Revert "Decrease the number of messages to use for same reason as the…
KADichev Sep 23, 2024
50cabd1
Revert "I propose to empirically find the m_maxSrs (maximum number of…
KADichev Sep 23, 2024
e3a20c9
Small fix to keep the refactored IBVerbs tests but still revert the t…
KADichev Sep 23, 2024
c0d8d28
Bring back GoogleTest download functionality
KADichev Oct 11, 2024
da922d5
Reflect changes to require CMake 3.29 and C++17 standard compatible c…
KADichev Oct 11, 2024
6143340
Resolve some compiler warnings
anyzelman Oct 11, 2024
a8da59e
Remove the cmake/mpi_open_port.c test, because this test passes both …
KADichev Oct 15, 2024
89d7001
Remove the disabling of IB Verbs -- it was only meant to test setup o…
KADichev Oct 15, 2024
d39d2cd
Use the latest IBVerbs test suite, which explicitly asks IBVerbs abou…
KADichev Oct 15, 2024
7fecf7c
Dramatic improvement in test generation time. Do not use anymore bash…
KADichev Oct 16, 2024
ec0a6e9
Do not generate IB Verbs tests if no Infiniband device is found
Oct 19, 2024
125631d
Unfortunately, I need to fix the setting for our cluster, where just …
KADichev Oct 19, 2024
2d7211c
Slightly improve the fix for single-process run to list tests
Oct 19, 2024
a08c0c5
Avoid one compiler warning (narrowing)
anyzelman Oct 21, 2024
4a6c887
Gtest macros implicitly causing ambiguous else-statements, fixed
anyzelman Oct 21, 2024
ca6d8c5
The test directly inspects the return code, rather than storing it in…
anyzelman Oct 21, 2024
99ac217
Suppress an unused-variable warning
anyzelman Oct 21, 2024
810a42a
The dynamic hook test was not fully GoogleTest compliant. Now it is. …
KADichev Oct 21, 2024
7c81328
Reduce to C++11 standard without tests, C++17 with tests
KADichev Oct 21, 2024
0a3c5cd
Albert Jan reported non-deterministic issues with pthread backend. Af…
KADichev Oct 23, 2024
540a91b
Bring back GoogleTest license agreement dialogue
KADichev Oct 23, 2024
eb245e7
Suppress some warnings encountered by GCC 11.5
anyzelman Oct 23, 2024
30cc344
Do some cleanup of gtest building macros in the CMake files. Also, do…
KADichev Oct 23, 2024
a606a1a
Minimal comment for internal abort
KADichev Oct 23, 2024
13ee0e8
Format all tests using clang-formatter in default mode
KADichev Oct 23, 2024
d32bc6b
Accept their changes plus use formatting with Clang
KADichev Oct 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
203 changes: 125 additions & 78 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# limitations under the License.
#

cmake_minimum_required(VERSION 3.1)
cmake_minimum_required(VERSION 3.29.0 FATAL_ERROR)
project(LPF C CXX ASM)

# Version info
Expand Down Expand Up @@ -105,18 +105,6 @@ set( INSTALL_HEADERS "${prefix}/include" CACHE PATH
"Installation path for header files" )
message( STATUS "Installation directory prefix is ${prefix}")

# C++ standard
find_file(TR1_ARRAY "tr1/array")
if (TR1_ARRAY)
message(STATUS "Governing C++ standard is C++98/TR1")
set(CMAKE_CXX_STANDARD 98)
set(CMAKE_CXX_STANDARD_REQUIRED YES)
else()
message(STATUS "Governing C++ standard is C++11")
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED YES)
endif()

# Dependencies
set(ENGINES)
find_library( LIB_POSIX_THREADS
Expand Down Expand Up @@ -173,9 +161,6 @@ if ( MPI_FOUND )
if ( NOT MPI_IS_THREAD_COMPAT OR NOT MPI_IS_NOT_OPENMPI1 )
message( WARNING "MPI implementation does not tolerate any threading. Hybrid implementation will not be built")
endif()
if ( NOT MPI_OPEN_PORT )
message( WARNING "MPI implementation does not support dynamically connecting separate MPI processes. Hence, lpf_mpi_initialize_over_tcp will always fail.")
endif()
if ( NOT MPI_IBARRIER )
message( WARNING "MPI implementation does not have MPI_Ibarrier, which is required to use the dense all-to-all algorithm on large (> 2 GB) meta-data exchanges")
endif()
Expand All @@ -189,7 +174,7 @@ if ( LIB_MATH AND LIB_DL AND MPI_FOUND )
list(APPEND ENGINES "mpirma")
endif()

if (LIB_IBVERBS)
if (ENABLE_IBVERBS)
list(APPEND ENGINES "ibverbs")
endif()

Expand All @@ -198,7 +183,7 @@ endif()
#enable the hybrid engine
if ( LIB_POSIX_THREADS AND LIB_MATH AND LIB_DL AND MPI_FOUND
AND MPI_IS_THREAD_COMPAT AND MPI_IS_NOT_OPENMPI1
AND LIB_IBVERBS )
AND ENABLE_IBVERBS )
list(APPEND ENGINES "hybrid")
set(HYBRID_ENGINE_ENABLED on)
endif()
Expand Down Expand Up @@ -246,59 +231,19 @@ add_definitions(-DBSPLIB_DLL=1)
option(LPF_ENABLE_TESTS
"Enable unit and API tests. This uses Google Testing and Mocking Framework"
OFF)
option(GTEST_AGREE_TO_LICENSE
"Does the user agree to the GoogleTest license"
OFF)

# C++ standard -- Google tests require newer C++ standard than C++11
if (LPF_ENABLE_TESTS)
message(STATUS "Unit and API tests will be built")

# set testing timeout to 60 seconds
set(CMAKE_TESTING_TIMEOUT 60)

# import Google Testing Framework
include(cmake/googletest.cmake)

# Have directory to gather all the tests results
file(MAKE_DIRECTORY ${CMAKE_BINARY_DIR}/junit)
set(test_output "${CMAKE_BINARY_DIR}/junit")

# Have a macro to add a unit test
function(add_gtest testName)
add_executable(${testName} ${ARGN})
target_link_libraries(${testName} gtest_main)
add_test(${testName} ${testName} --gtest_output=xml:${test_output}/ )
endfunction(add_gtest)

# Have a macro to add a unit test that should run with MPI
if (MPI_FOUND)
function(add_gtest_mpi testName nprocs)
add_executable(${testName} ${ARGN})
target_link_libraries(${testName} ${MPI_C_LIBRARIES} gtest_main)
foreach( p ${nprocs})
set(mpmd)
foreach( i RANGE 1 ${p})
if (i GREATER 1)
set(mpmd ${mpmd} ":")
endif()
set(mpmd ${mpmd} ${MPIEXEC_NUMPROC_FLAG} 1 ${MPIEXEC_PREFLAGS}
./${testName} --gtest_output=xml:${test_output}/${testName}_${i}of${p}.xml)
endforeach(i)
add_test(NAME ${testName}_${p}
COMMAND ${MPIRUN} ${mpmd}
)
endforeach(p)
endfunction(add_gtest_mpi)
endif(MPI_FOUND)

# Enable testing in CMake
enable_testing()
else(LPF_ENABLE_TESTS)
message(STATUS "Unit and API tests will *not* be built")
function(add_gtest testName)
# Do nothing because tests are disabled
endfunction(add_gtest)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED YES)
else()
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED YES)
endif()

function(add_gtest_mpi testName nprocs)
# DO nothing because tests are disabled
endfunction(add_gtest_mpi)
endif(LPF_ENABLE_TESTS)

# Handling of compiler flags
function(target_add_compilation_flags target visibility)
Expand Down Expand Up @@ -366,10 +311,7 @@ endfunction(target_compile_flags)
# Source
set(lpf_cflags)
set(lpf_lib_link_flags)
set(lpf_exe_link_flags)
include_directories(include)
include_directories(src/common)
add_subdirectory(src)
set(lpf_exe_link_flags "-rdynamic")

# Collating all compile & link flags
set(LPF_CORE_COMPILE_FLAGS "${lpf_cflags}" CACHE STRING "Compilation flags for all user code" )
Expand All @@ -384,14 +326,119 @@ function( target_link_exe_with_core target )
endif()
set(corelib "lpf_core_univ_${engine}_${LPFLIB_CONFIG_NAME}")

target_link_libraries(${target} ${corelib})
if ("${engine}" STREQUAL "pthread")
target_link_libraries(${target} ${corelib})
else()
target_link_libraries(${target} ${corelib} ${MPI_C_LIBRARIES})
endif()
target_compile_flags(${target} PRIVATE ${LPF_CORE_COMPILE_FLAGS})
set_target_properties(${target} PROPERTIES
LINK_FLAGS "${LPF_CORE_LIB_LINK_FLAGS} ${LPF_CORE_EXE_LINK_FLAGS}"
LINKER_LANGUAGE CXX
)
LINK_FLAGS "${LPF_CORE_LIB_LINK_FLAGS} ${LPF_CORE_EXE_LINK_FLAGS}"
LINKER_LANGUAGE CXX
)
endfunction()

if (LPF_ENABLE_TESTS)
message(STATUS "Unit and API tests will be built. This requires CMake version 3.29 or higher, since we use recent features of the GoogleTest package in CMake.")

if (NOT GTEST_AGREE_TO_LICENSE)
message(FATAL_ERROR "The user needs to agree with the GoogleTest license to use tests (option GTEST_AGREE_TO_LICENSE=TRUE)")
endif()
# Enable testing in CMake
enable_testing()
find_package(GTest)
include(GoogleTest)
if(NOT GTest_FOUND) # if not found, download it and pull it in
include(FetchContent)
FetchContent_Declare(
googletest
GIT_REPOSITORY https://github.com/google/googletest.git
# This tag corresponds to GoogleTest 1.15.0 release
GIT_TAG e397860
)
FetchContent_MakeAvailable(googletest)

endif()
# set testing timeout to 60 seconds
set(CMAKE_TESTING_TIMEOUT 60)

# Have directory to gather all the tests results
file(MAKE_DIRECTORY ${CMAKE_BINARY_DIR}/junit)
set(test_output "${CMAKE_BINARY_DIR}/junit")

set(MY_TEST_LAUNCHER ${CMAKE_BINARY_DIR}/test_launcher.py)
configure_file( ${CMAKE_SOURCE_DIR}/test_launcher.py ${MY_TEST_LAUNCHER} @ONLY FILE_PERMISSIONS WORLD_EXECUTE OWNER_EXECUTE OWNER_WRITE OWNER_READ GROUP_EXECUTE GROUP_READ)
if( NOT Python3_FOUND )
find_package( Python3 REQUIRED)
endif()

# Macro for adding a new GoogleTest test
function(add_gtest testName ENGINE debug testSource )
if ("{$ENGINE}" STREQUAL "")
message(FATAL_ERROR "engine cannot be empty, ever!")
endif()
add_executable(${testName} ${testSource} ${ARGN})
target_compile_definitions(${testName} PUBLIC LPF_CORE_IMPL_ID=${ENGINE})
target_compile_definitions(${testName} PUBLIC LPF_CORE_MPI_USES_${ENGINE})
if (debug)
target_include_directories( ${testName} BEFORE PRIVATE ${CMAKE_SOURCE_DIR}/include/debug )
target_link_libraries(${testName} lpf_debug lpf_hl_debug GTest::gtest GTest::gtest_main)
else(debug)
target_link_libraries(${testName} GTest::gtest GTest::gtest_main)
endif(debug)


# Extract test-specific information from comments of tests
file(READ ${testSource} fileContents)
string(REGEX MATCH "Exit code: ([0-9]+)" _ ${fileContents})
set(retCode ${CMAKE_MATCH_1})
string(REGEX MATCH "pre P >= ([0-9]+)" _ ${fileContents})
set(minProcs ${CMAKE_MATCH_1})
string(REGEX MATCH "pre P <= ([0-9]+)" _ ${fileContents})
set(maxProcs ${CMAKE_MATCH_1})
string(REGEX MATCH "-probe ([0-9]+.[0-9]+)" _ ${fileContents})
set(lpfProbeSecs ${CMAKE_MATCH_1})

target_link_exe_with_core(${testName} ${ENGINE})


if ("${minProcs}" STREQUAL "")
set(minProcs "1")
endif()
if ("${maxProcs}" STREQUAL "")
set(maxProcs "5")
endif()
if ("${lpfProbeSecs}" STREQUAL "")
set(lpfProbeSecs "0.0")
endif()
if ("${retCode}" STREQUAL "")
set(retCode "0")
endif()

# Most recent approach to Gtests, recommended!
set_property(TARGET ${testName} PROPERTY TEST_LAUNCHER ${MY_TEST_LAUNCHER};-e;${ENGINE};-L;${CMAKE_BINARY_DIR}/lpfrun_build;-p;${minProcs};-P;${maxProcs};-t;${lpfProbeSecs};-R;${retCode})
gtest_discover_tests(${testName}
TEST_PREFIX ${ENGINE}_
EXTRA_ARGS --gtest_output=xml:${test_output}/${ENGINE}_${testName}
DISCOVERY_MODE POST_BUILD
DISCOVERY_TIMEOUT 15
)


endfunction(add_gtest)

else(LPF_ENABLE_TESTS)
message(STATUS "Unit and API tests will *not* be built")
function(add_gtest testName ENGINE debug testSource )
# Do nothing because tests are disabled
endfunction(add_gtest)

endif(LPF_ENABLE_TESTS)

include_directories(include)
include_directories(src/common)

add_subdirectory(src)
# Apps
add_subdirectory(src/utils)

Expand Down Expand Up @@ -432,7 +479,7 @@ set( lpf_proxy_dummy ${CMAKE_CURRENT_BINARY_DIR}/src/MPI/lpf_proxy_dummy)
set( lpf_probe ${CMAKE_CURRENT_BINARY_DIR}/src/utils/lpfprobe)
set( lpfrun ${CMAKE_CURRENT_BINARY_DIR}/lpfrun_build)
set( lpfcore ${CMAKE_CURRENT_BINARY_DIR}/src/*/liblpf_core_univ_ENGINE_${LPFLIB_CONFIG_NAME}${SOSUFFIX} )
configure_file( lpfrun.in lpfrun_build @ONLY)
configure_file( lpfrun.in lpfrun_build FILE_PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE @ONLY)
configure_file( lpfproxy.in lpfproxy_build @ONLY)
configure_file( lpfprobe.in lpfprobe_build @ONLY)

Expand Down
4 changes: 2 additions & 2 deletions README
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@ Prerequisites
Mandatory
- GNU/Linux,
- GNU C compiler,
- GNU C++ compiler (C++03/TR1 or C++11),
- GNU C++ compiler (C++11 compatible; however, C++17 compatible with tests enabled)
- GNU Make,
- CMake 3.1 or better.
- CMake 3.29.0 or better.

Optional MPI engine requires
- MPI-3, such as 'MPICH', 'OpenMPI', or 'MVAPICH'.
Expand Down
10 changes: 5 additions & 5 deletions bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ installdir="$builddir"
config=Release
doc=OFF
functests=OFF
googletest_license_agreement=NO
googletest_license_agreement=FALSE
perftests=OFF
reconfig=no
CMAKE_EXE=cmake
Expand Down Expand Up @@ -133,7 +133,7 @@ do
------------------------------------------------------------------------------
The functional test suite requires Google Testing Framework which comes with
its own license. The license can be viewed via
https://github.com/google/googletest/blob/release-1.8.1/LICENSE
https://github.com/google/googletest/blob/v1.15.x/LICENSE
Do you agree with the license [yes/no] ?
EOF
read answer
Expand All @@ -154,15 +154,15 @@ User agrees with Google Testing Framework license. It will be downloaded during
the build.
==============================================================================
EOF
googletest_license_agreement=YES
googletest_license_agreement=TRUE
fi

shift
;;

--functests=i-agree-with-googletest-license)
functests=ON
googletest_license_agreement=YES
googletest_license_agreement=TRUE
shift
;;

Expand Down Expand Up @@ -281,7 +281,7 @@ ${CMAKE_EXE} -Wno-dev \
-DCMAKE_BUILD_TYPE=$config \
-DLPFLIB_MAKE_DOC=$doc \
-DLPFLIB_MAKE_TEST_DOC=$doc \
-DLPF_ENABLE_TESTS=$functests \
-DLPF_ENABLE_TESTS=$functests \
-DGTEST_AGREE_TO_LICENSE=$googletest_license_agreement \
-DLPFLIB_PERFTESTS=$perftests \
-DLPFLIB_CONFIG_NAME=${config_name:-${config}}\
Expand Down
59 changes: 0 additions & 59 deletions cmake/googletest.cmake

This file was deleted.

Loading