Skip to content

[ML] do not fail to start analytics process if memory estimate is lower than predicted memory usage #1465

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 1, 2020
Merged

[ML] do not fail to start analytics process if memory estimate is lower than predicted memory usage #1465

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c694515
[ML] do not fail to start analytics process of memory estimate is low
benwtrent Aug 31, 2020
c384d9f
fixing test
benwtrent Sep 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions lib/api/CDataFrameAnalysisRunner.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,14 +99,14 @@ void CDataFrameAnalysisRunner::computeAndSaveExecutionStrategy() {
auto roundMb = [](std::size_t memory) {
return 0.01 * static_cast<double>((100 * memory) / BYTES_IN_MB);
};

// Report rounded up to the nearest MB.
HANDLE_FATAL(<< "Input error: memory limit " << roundMb(memoryLimit)
<< "MB is too low to perform analysis. You need to give the process"
<< " at least " << std::ceil(roundMb(memoryUsage))
<< "MB, but preferably more.");

} else if (m_NumberPartitions > 1) {
// Simply log the limit being configured too low.
// If we exceed the limit during the process, we will fail and the user
// will have to update the limit and attempt to re-run
LOG_DEBUG(<< "Memory limit " << roundMb(memoryLimit) << "MB is configured lower than estimate "
<< std::ceil(roundMb(memoryUsage)) << "MB."
<< "Analytics process may fail due to low memory limit");
}
if (m_NumberPartitions > 1) {
// The maximum number of rows is found by binary search in the interval
// [numberRows / m_NumberPartitions, numberRows / (m_NumberPartitions - 1)).

Expand Down