Multi-thread DB Queue

[pentaflops]

Problem

After transferring the databases to Amazon, we were faced with the problem of a long DB response. Research shown that request queue is quickly filled, the reason is query response to Amazon database can take 2 seconds or more. The current DatabaseManager creates a new queue using the database connection string, and not for a connection element. Thus, a single queue will be created for one database, processing requests many connections in one thread.

Solution

The logic of DatabaseManager and DatabaseJobQueue has been changed. Each individual connection is processed in a separate thread and doesn't block the processing of requests from other connections. The dbGetConnectionQueueSize function was also added, usage example is shown below.

New function

int dbGetConnectionQueueSize(element dbConnection)

Returns false on failure.

---

This change has been tested and successfully operated on our project.
With love from NEXTRP Team

[pentaflops]

Example

--- RESOURCE: db_manager

CONST_QUEUE_SIZE_TO_CREATE_NEW_CONNECTION = 100

CONNECTIONS_POOL = { }
ACTIVE_CONNECTION = nil

function CheckAndCreateNewConnection( )
	local min_queue_size = CONST_QUEUE_SIZE_TO_CREATE_NEW_CONNECTION
	local db_with_min_queue_size = nil

	for i, db in pairs( CONNECTIONS_POOL ) do
		if isElement( db ) then
			local queue_size = dbGetConnectionQueueSize( db )
			if queue_size < min_queue_size then
				min_queue_size = queue_size
				db_with_min_queue_size = db
			end
		end
	end

	if not db_with_min_queue_size then
		db_with_min_queue_size = CreateNewConnection( )
	end

	return db_with_min_queue_size
end

function CreateNewConnection( )
	local db = dbConnect( "mysql", ... )
	table.insert( CONNECTIONS_POOL, db )
end

function KeepConnection( )
	ACTIVE_CONNECTION = CheckAndCreateNewConnection( )
end

KeepConnection( )
setTimer(KeepConnection, 5000, 0)

function GetDB( )
	return ACTIVE_CONNECTION
end

--- RESOURCE: test

addEventHandler( "onResourceStart", resourceRoot, function( )
	dbQuery( function( qh )
		local result = dbPoll( qh, 0 )
		iprint( result )
	end, exports.db_manager:GetDB( ), "SELECT * FROM users" )
end )

[qaisjp]

This is some good work, thank you!

Review:

• It doesn't build on Windows, so we can't merge this until that works. Don't worry about macOS if you can't get it to work, but you'll save me a little bit of work later if you are able to fix it.
• Left a few minor comments

[qaisjp]

This has been removed and not reimplemented in the new system. I assume this is intended as it's no longer a bottleneck.

Is there other dead code or no-longer-useful stuff you can rip out too, such as m_uiConnectionCountWarnThresh?

[qaisjp]

@pentaflops what about this one? (and the comment by @Citizen01)

[ccw808]

The message is to help scripters who inadvertanly call dbConnect multiple times. I think it should not be removed

Changes addressed

[qaisjp]

Please see #1242 (comment)

#1242 (review)

[4O4]

This PR was reverted, but after studying the current source code I've found that this issue was kind of already addressed since this commit 58e993d which added a queue option for dbConnect. It allows you to specify a queue name, so multiple connections to the same db can have separate job queues. The thing is that this was not documented on the wiki until today.

Btw changing the order of parameters in the host string can probably also do the trick, because the host string is used by default as the queue name. But this is probably not a good idea to rely on something like that, it's better to customize the queue name manually.