llama: increase MEM_REQ_EVAL for MODEL_3B

jxy · jxy · commit b2fda181ae00 · 2023-07-03T21:35:41.000-05:00
It avoids crashing for quantized weights on CPU.
Better ways to calculate the required buffer size would be better.
diff --git a/llama.cpp b/llama.cpp
@@ -122,7 +122,7 @@ static const std::map<e_model, size_t> & MEM_REQ_KV_SELF()
 static const std::map<e_model, size_t> & MEM_REQ_EVAL()
 {
     static std::map<e_model, size_t> k_sizes = {
-        { MODEL_3B,   512ull * MB },
+        { MODEL_3B,   640ull * MB },
         { MODEL_7B,   768ull * MB },
         { MODEL_13B, 1024ull * MB },
         { MODEL_30B, 1280ull * MB },

Original file line number	Diff line number	Diff line change
`@@ -122,7 +122,7 @@ static const std::map<e_model, size_t> & MEM_REQ_KV_SELF()`
`122`	`122`	`static const std::map<e_model, size_t> & MEM_REQ_EVAL()`
`123`	`123`	`{`
`124`	`124`	`static std::map<e_model, size_t> k_sizes = {`
`125`		`- { MODEL_3B, 512ull * MB },`
	`125`	`+ { MODEL_3B, 640ull * MB },`
`126`	`126`	`{ MODEL_7B, 768ull * MB },`
`127`	`127`	`{ MODEL_13B, 1024ull * MB },`
`128`	`128`	`{ MODEL_30B, 1280ull * MB },`