Add async/await as better way to call blocking functions

[CrosRoad95]

Is your feature request related to a problem? Please describe.
Callbacks hell, its often hard and uncomfortable to pass all data you need to next function and some of them just can't be passed

Describe the solution you'd like
add build in mta async/await style functionality, eg.:

invokeAsync(function()
  response = fetchRemoteAsync("mtasa.com/api")
  data = fromJSONAsync(response.data)
  print(data)
  dbQueryAsync("insert into...", data)

  data = triggerClientEventAsync("getFoo", getRandomPlayer())
  print("data", data) -- prints "foo"

  while true do -- equivalent of setTimer(print, 100, 0, "hi")
    sleep(100)
    print("hi")
  end
end)

-- clientside:
addEventHandlerAsync("getFoo", root, function()
  return "foo"
end)

less code, more readable, easy to use, non blocking

Describe alternatives you've considered
/

Additional context
https://forum.mtasa.com/topic/54787-simple-sleep-function/ example how to do this

Related issues #1815

[Pirulax]

For this MTA would need to support MT properly, and it doesn't.
Also Lua doesn't either, although this can be solved by adding GIL, which would slow down single threaded scripts a little, but nothing horrible, MTA is the blocker here.

Also, this approach isn't the greatest. Python's is way better, as we wouldn't have duplicate functions, which can be confusing, especially to new scripters.

[CrosRoad95]

Someone, most experienced person should make an example, template of how to make async, multithreaded function and then everyone else should follow the same rules

yes, i know 2 functions, one sync, second async may be confusing. Solution for that would be api versioning, but idk

[Pirulax]

The biggest problem here isn't the function naming.
It's that MTA doesn't support multi threading by default.
If you were to call any of the function's from a different thread, there's a high chance of it crashing. So unless we reprogram the whole MTA to be async this won't work.
And even then GTA isn't multi threaded either, so we'd still have a bottleneck somewhere.
Those few functions that can be run totally async already have the callback option.

[4O4]

For this MTA would need to support MT properly, and it doesn't.
Also Lua doesn't either, although this can be solved by adding GIL, which would slow down single threaded scripts a little, but nothing horrible, MTA is the blocker here.

I don't like to say people they are wrong, but you are wrong on this one mate :) You actually don't need multi threading for async/await at all. Javascript is a single threaded environment running an event-loop with occasional APIs which are running in separate threads (like IO), and there is a great async / await support there. This is very similar to how MTA works. We have a main game loop, a lot of events and also some APIs which are processed in a separate threads (like fetchRemote, dbQuery, triggerLatent...). What's worth pointing out is that the coroutines in Lua are basically the exact equivalent of async / await functionality in JS. Both a/a and coroutines allow you to write an async code that is flat like a synchronous code (no nested callbacks hell, if you don't know what callback hell is then you are one lucky and happy man 😄 See: https://callbackhell.com ).

This issue by @CrosRoad95 is actually related to what I was talking about yesterday on the #development channel and I wanted to publish a similar proposal about adding a first class support for coroutines in the existing asynchronous APIs in the MTA. By first class support I mean the functions that yield the currently running thread, and this means that these functions could only be invoked from the coroutine threads (not actual real OS-level threads, it's all about plain Lua coroutines!).

To get some broader picture of that please take a Roblox game as an example, look at their docs: https://developer.roblox.com/en-us/api-reference/function/HttpService
You will notice that some function names are ending with Async word and are marked with [yields], i.e. GetAsync, PostAsync. Here's a full signature from the docs:

string GetAsync ( string url , bool nocache , Variant headers ) [Yields] 

AFAIK the Roblox environment is a bit different because everything there is based on coroutines and they actually have a way of spawning OS-level threads, but as I said earlier this is completely not needed. You can combine the ideas and have the best of both worlds. The existing MTA Lua APIs which already take callbacks as arguments could be used this way. Of course you can do wrappers on the Lua level to make it work the same way, but I'll repeat what I said earlier - I suggest adding a first class support for that. This can contribute to the overall code quality on the Lua side and will help the developers to avoid callback hell if / when more async APIs are implemented especially around IO and other heavy operations.

Another Lua platform worth taking a closer look at is Lusty web framework running on OpenResty. Quoting from their website (https://leafo.net/lapis/):

Nginx’s event loop is used for all asynchronous actions, including HTTP requests and database queries. With the power of Lua coroutines code is written synchronously but runs asynchronously, without all that callback spaghetti seen in other asynchronous platforms. It’s fast, easy to read, and easy to write.

Here's also an excellent blog post about that: https://leafo.net/posts/itchio-and-coroutines.html

So to summarize: Async/await is not tied to OS-level multithreading support. Async/await suits well in every environment which is driven by events, and MTA is for sure such environment. Lua coroutines are basically a drop-in replacement for async/await functionality as known from other languages such as Javascript. The proposal is to bring a first class support for coroutine based async APIs which would improve the overall development experience and would help avoid spaghetti code when a lot of async code is in play on the Lua level. There are existing codebases that we can take inspiration from,

[qaisjp]

Implementing coroutine-versions of the MTA API should already be possible, in plain Lua. Please can someone build a proof of concept?

[CrosRoad95]

can you provide details how it should work? i could try do this

[qaisjp]

--
-- async.lua (library)
--

-- async returns a version of fn that has MTA functions running in an async context
function async(fn)
    -- to be implemented by CrosRoad95
end

--
-- client.lua (user code)
--

function hash()
    local str = passwordHash("test", "bcrypt", { cost = 20 })
    outputChatBox(str)
end

addEventHandler("onClientResourceStart", resourceRoot, async(hash))

This should run the hash asynchronously without locking up the client.

[CrosRoad95]

do this need to be in c++, or plain lua the same as inspect, exports ?

[Pirulax]

Implementing coroutine-versions of the MTA API should already be possible, in plain Lua.

[CrosRoad95]

yep, it is possible

_passwordHash = passwordHash
function passwordHashAsync(...)
    local args = {...}
    
    local co = coroutine.running() 
    local result;
    local function resumeThisCoroutine(...)
        result = {...}
        coroutine.resume(co)
    end
    _passwordHash(args[1], args[2], args[3], resumeThisCoroutine)
    coroutine.yield()
    return unpack(result)
end

function async(fn)
    local co = coroutine.create(fn)
    coroutine.resume(co)
    return function()end
end

function await(func, ...) 
    local co = coroutine.running() 
    local function resumeThisCoroutine(arg)
        coroutine.resume(co)
    end 
    setTimer(resumeThisCoroutine, 100, 1)
    coroutine.yield()
    return "Asdf"
end 

function hash()
    local str = passwordHashAsync("test", "bcrypt", { cost = 15 })
    iprint("result",str)
end

addEventHandler("onResourceStart", resourceRoot, async(hash))

[Pirulax]

That seems very over-complicated just to not use a callback directly.

[qaisjp]

It looks like your script runs hash immediately, and does not return an async version of the function. Try a command handler, addCommandHandler("test", async(hash)).

You should use setfenv to modify the function environment, i.e. to translate passwordHash to an async version in just that function's environment.

[4O4]

It looks like your script runs hash immediately, and does not return an async version of the function. Try a command handler, addCommandHandler("test", async(hash)).

You should use setfenv to modify the function environment, i.e. to translate passwordHash to an async version in just that function's environment.

Wouldn't it be better to have a clear visual distinction between sync and async functions though? The existing code needs to be modified anyway to get rid of the callback so it's not like it's going to magically convert someone's callback based code to coroutine based one with just one call of async(). Can you elaborate a little bit more what are your further intentions with the PoC? Also, are there any cons of implementing it on the c++ level? Would be great to get some insights.

In the meantime I've edited the sample from @CrosRoad95 , fixed version below:

local _passwordHash = passwordHash
function passwordHashAsync(...)
    local args = {...}
    
    local co = coroutine.running() 
    local function resumeThisCoroutine(...)
        coroutine.resume(co, ...)
    end
    _passwordHash(args[1], args[2], args[3], resumeThisCoroutine)

    return coroutine.yield()
end

local asyncEnv = setmetatable({ passwordHash = passwordHashAsync }, { __index = _G })

function async(fn)
  return function() 
    setfenv(fn, asyncEnv)
    local co = coroutine.create(fn)
    coroutine.resume(co)
  end
end

function hash()
  iprint("hash started")
    local str = passwordHash("test", "bcrypt", { cost = 15 })
    iprint("result",str)
end

addEventHandler("onResourceStart", resourceRoot, async(hash))
addCommandHandler("testasync", async(hash))

BTW Keep in mind that test command is reserved internally by MTA I believe, so if you make a command handler for that it will not work. I changed the command to testasync in this PoC.

[Pirulax]

We need to keep backwards compatibility, so we can't require the current code to be changed.

[4O4]

We need to keep backwards compatibility, so we can't require the current code to be changed.

Sorry I don't follow By adding a new function let's say passwordHashAsync you avoid both parameter-overload-related headaches and compatibility issues.