Lua transform not able to find lua-http package

[atibdialpad]

Use-Case : Make an http call from the transform.
Since there is no such vector transform which does that, I resorted to using the LUA transform and use lua-http to make the http requests.

Steps:

1. Install and run vector
2. Install lua rocks via : apt-get install luarocks
3. Install lua headers: apt install liblua5.3-dev
4. Install lua http (which install all the dependencies compat53, bit32, cqueues, luaossl, basexx, lpeg, lpeg_patterns, binaryheap >= 0.3, fifo : luarocks --tree=/usr/share/lua/5.3/ install http
5. Now, when I use lua http from my transform, I get the below stacktrace :
   May 03 09:09:14.232 ERROR transform{component_kind="transform" component_name=my_transform_id component_type=lua}: vector::internal_events::lua: Error in lua script; discarding event. error=RuntimeErrorHooksProcess { source: RuntimeError("[string "?"]:4: module 'http.request' not found: no field package.preload['http.request'] no file '/usr/share/lua/5.3/http/request.lua' no file '/usr/local/share/lua/5.3/http/request.lua' no file '/usr/local/share/lua/5.3/http/request/init.lua' no file '/usr/local/lib/lua/5.3/http/request.lua' no file '/usr/local/lib/lua/5.3/http/request/init.lua' no file './http/request.lua' no file './http/request/init.lua' no file '/usr/local/lib/lua/5.3/http/request.so' no file '/usr/local/lib/lua/5.3/loadall.so' no file './http/request.so' no file '/usr/local/lib/lua/5.3/http.so' no file '/usr/local/lib/lua/5.3/loadall.so' no file './http.so' stack traceback: [C]: in ? [C]: in function 'require' [string "?"]:4: in function <[string "?"]:1>") } internal_log_rate_secs=30
6. I have a feeling it is something to do with where Vector is looking for the relevant packages and where it is installed but am not able to fix it.
   @jszwedko any ideas on why this can be happening ?

My transform config has search_dirs = ["/usr/share/lua/5.3"]

[atibdialpad]

I changed the search_dirs to "/usr/share/lua/5.3/share/lua/5.1" becasue :
>>VECTOR_DEV# ls /usr/share/lua/5.3/share/lua/5.1/ basexx.lua binaryheap.lua compat53 cqueues cqueues.lua fifo.lua http lpeg_patterns openssl openssl.lua re.lua
After which vector was able to find the http package, now it fails in finding the lpeg package

May 03 09:43:37.578 ERROR transform{component_kind="transform" component_name=my_transform_id component_type=lua}: vector::internal_events::lua: Error in lua script; discarding event. error=RuntimeErrorHooksProcess { source: RuntimeError("/usr/share/lua/5.3/share/lua/5.1/http/request.lua:1: module lpeg not found:
no field package.preload['lpeg']
no file '/usr/share/lua/5.3/share/lua/5.1/lpeg.lua'
no file '/usr/share/lua/5.3/lib/lua/5.1/lpeg.lua'
no file '/usr/local/share/lua/5.3/lpeg.lua'
no file '/usr/local/share/lua/5.3/lpeg/init.lua'
no file '/usr/local/lib/lua/5.3/lpeg.lua'
no file '/usr/local/lib/lua/5.3/lpeg/init.lua'
no file './lpeg.lua'
no file './lpeg/init.lua'
no file '/usr/local/lib/lua/5.3/lpeg.so'
no file '/usr/local/lib/lua/5.3/loadall.so'
no file './lpeg.so'
stack traceback:
[C]: in ?
[C]: in function 'require'
/usr/share/lua/5.3/share/lua/5.1/http/request.lua:1: in main chunk
[C]: in function 'require'
[string "?"]:4: in function <[string "?"]:1>") } internal_log_rate_secs=30

Wish there was a simple installation as part of vector rather than having to manually install packages and take care of correct versions and package paths.

[atibdialpad]

Managed to link the lpeg.so (it was a compiled library) by adding package.cpath to the lua code.
Current status :
May 03 10:27:23.921 ERROR transform{component_kind="transform" component_name=my_transform_id component_type=lua}: vector::internal_events::lua: Error in lua script; discarding event. error=RuntimeErrorHooksProcess { source: RuntimeError("error loading module 'lpeg' from file '/usr/share/lua/5.3/lib/lua/5.1/lpeg.so':
/usr/share/lua/5.3/lib/lua/5.1/lpeg.so: undefined symbol: lua_tointeger
stack traceback:
[C]: in ?
[C]: in ?
[C]: in function 'require'
/usr/share/lua/5.3/share/lua/5.1/http/request.lua:1: in main chunk
[C]: in function 'require'
[string "?"]:5: in function <[string "?"]:1>") } internal_log_rate_secs=30

[daurnimator]

4\. **luarocks --tree=/usr/share/lua/5.3/ install http**

you probably wanted: luarocks --tree /usr/ install http
inside of the tree luarocks adds share/lua/5.3 for you.

I changed the search_dirs to "/usr/share/lua/5.3/share/lua/5.1" becasue :
>>VECTOR_DEV# ls /usr/share/lua/5.3/share/lua/5.1/ basexx.lua binaryheap.lua compat53 cqueues cqueues.lua fifo.lua http lpeg_patterns openssl openssl.lua re.lua

looks like you may need to add --lua-version 5.3. your luarocks thinks it's compiling for lua 5.1

[atibdialpad]

Thanks @daurnimator will add the lua-version and check.
Also @jszwedko I do not want luarocks to install Lua for me. I want it to use the Lua already installed as part of Vector. Is it possible to do that ?

[jszwedko]

@atibdialpad Vector will use the version of Lua bundled with it. I see that the luarocks package has a dependency on the lua package so it will install it when you install luarocks, but Vector won't use that one.

[atibdialpad]

@daurnimator does "--lua-version 5.3" work in the luarocks cmd line ?
I still see luarocks installing packages for lua 5.1 even when I specify the lua-version. I see this is still open luarocks/luarocks#719

[atibdialpad]

[Update]
So I could get lua http work with lua5.3 but not with vector

1. install lua 5.3

2. install liblua5.3-dev

3. install luarocks

4. install luahttp (--tree=/usr/ --lua-version 5.3)
   ~# luarocks config --lua-ver
   5.3

5. http code works
   package.path = package.path .. ';/usr/share/lua/5.3/?.lua'
   package.cpath = package.cpath .. ';/usr/lib/lua/5.3/?.so'
   local lpeg = require "lpeg"
   local http_request = require "http.request"
   local my_req = http_request.new_from_uri("http://httpbin.org/get")
   local headers, stream = assert(my_req:go())
   for field, value in headers:each() do
   print(field, value)
   end
   local body = assert(stream:get_body_as_string())
   print(body)

   {
   "args": {},
   "headers": {
   "Host": "httpbin.org",
   "User-Agent": "lua-http/0.4",
   "X-Amzn-Trace-Id": "Root=1-6090fbf7-36933a012d2959b04e41d834"
   },
   "origin": "103.35.127.33",
   "url": "http://httpbin.org/get"
   }

6. However when I run Vector Lua transform with http req, I get this error
   May 04 07:49:32.101 ERROR transform{component_kind="transform" component_name=my_transform_id component_type=lua}: vector::internal_events::lua: Error in lua script; discarding event. error=RuntimeErrorHooksProcess { source: RuntimeError("error loading module '_cqueues' from file '/usr/lib/lua/5.3/_cqueues.so':
   /usr/lib/lua/5.3/_cqueues.so: undefined symbol: luaL_openlibs
   stack traceback:
   [C]: in ?
   [C]: in ?
   [C]: in function 'require'
   /usr/share/lua/5.3/cqueues/auxlib.lua:2: in function </usr/share/lua/5.3/cqueues/auxlib.lua:1>
   (...tail calls...)
   [C]: in function 'require'
   /usr/share/lua/5.3/http/client.lua:1: in main chunk
   [C]: in function 'require'
   /usr/share/lua/5.3/http/request.lua:5: in main chunk
   [C]: in function 'require'
   [string "?"]:6: in function <[string "?"]:1>") } internal_log_rate_secs=30

Looking into this however @jszwedko @daurnimator any leads ?

[atibdialpad]

Hey @daurnimator is there any dependency of lua cqueues that I am missing here ?

[atibdialpad]

[Update]
I was not able to make the vector pipeline work with lua http :-( I switched to socket and vector seems happy with it . Something like this
local io = require("io")
local http = require("socket.http")
local ltn12 = require("ltn12")
b,c,h = http.request{
url = "https://xyz",
sink = ltn12.sink.file(io.stdout)
}
event.log.http_b = b
event.log.http_c = c

[khvzak]

@atibdialpad
You probably need to re-build vector with .cargo/config:

[target.x86_64-apple-darwin]
rustflags = ["-C", "link-args=-rdynamic"]

[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "link-args=-rdynamic"]

to export Lua symbols.

[atibdialpad]

Hey @khvzak , thanks for the info. Can you please explain a bit what that will do. Sorry I new to both Lua and Vector.

[atibdialpad]

Hey @khvzak , I already see these in https://github.com/timberio/vector/blob/master/.cargo/config . I am using a x86_64 linux gnu machine.

[khvzak]

Sorry @atibdialpad I can confirm that symbol luaL_openlibs does not exported (you can check dynamic symbols using nm -D).
Unfortunately with bundled lua there is not simple way to include this symbol :(

[atibdialpad]

Thanks @khvzak
I will go ahead with the socket way.