More high level documentation

[coderedart]

I want to expose sort of a plugin system for my bevy app.
and i don't want different plugins to be using the same lua vm for four reasons:

1. mix up of different variable names and their data (atleast in global scope) leading to different weird (or even malicious) errors.
2. security. one plugin might require some kind of api key and this should not be accessed by other plugins.
3. one plugin might accidentally mess up the whole global table and cause others to go down with it.
4. run different plugins in different lua instances concurrently when they are not interacting with any global bevy data.

so, when i create a LuaScriptCollection, i want it to have its own lua vm.

is this possible right now?

also, I am wondering there could be a small Architecture.md explaining at a high level how this works. I am not familiar with what the terms ScriptHost or ScriptCollection really mean in the abstract concept sense.

[makspll]

Hi! Sorry if this is not made clear in the docs, I will keep this issue around as a tracking issue for doc improvement. but in the meantime:

ScriptHost

The best way to understand the project structure is to look at the interface for a ScriptHost (line 64 in hosts/mod.rs):

/// A script host is the interface between your rust application
/// and the scripts in some interpreted language.
pub trait ScriptHost: Send + Sync + 'static + Default {
    /// the type of the persistent script context, representing the execution context of the script
    type ScriptContext: Send + Sync + 'static;
    /// the type of events picked up by lua callbacks
    type ScriptEvent: ScriptEvent;
    /// the type of asset representing the script files for this host
    type ScriptAsset: CodeAsset;
    /// the type representing the target of api providers, i.e. the
    /// script engine or the thing which stores the API
    type APITarget;
    /// the type of each doc fragment
    type DocTarget: DocFragment;

    /// Loads a script in byte array format, the script name can be used
    /// to send useful errors.
    fn load_script(
        &mut self,
        script: &[u8],
        script_name: &str,
        providers: &mut APIProviders<Self::APITarget, Self::DocTarget>,
    ) -> Result<Self::ScriptContext, ScriptError>;

    /// the main point of contact with the bevy world.
    /// Scripts are called with appropriate events in the event order
    fn handle_events<'a>(
        &self,
        world: &mut World,
        events: &[Self::ScriptEvent],
        ctxs: impl Iterator<Item = (FlatScriptData<'a>, &'a mut Self::ScriptContext)>,
    );

    /// Loads and runs script instantaneously without storing any script data into the world.
    /// The script receives the `world` global as normal, but `entity` is set to `u64::MAX`.
    /// The script id is set to `u32::MAX`.
    fn run_one_shot(
        &mut self,
        script: &[u8],
        script_name: &str,
        world: &mut World,
        event: Self::ScriptEvent,
    ) {
        let providers: &mut APIProviders<Self::APITarget, Self::DocTarget> =
            &mut world.resource_mut();
        let mut ctx = self.load_script(script, script_name, providers).unwrap();

        let entity = Entity::from_bits(u64::MAX);

        let events = [event; 1];
        let ctx_iter = [(
            FlatScriptData {
                name: script_name,
                sid: u32::MAX,
                entity,
            },
            &mut ctx,
        ); 1]
            .into_iter();

        self.handle_events(world, &events, ctx_iter)
    }

    /// Registers the script host with the given app, and attaches handlers to deal with spawning/removing scripts at the given stage.
    ///
    /// Ideally place after any game logic which can spawn/remove/modify scripts to avoid frame lag. (typically `CoreStage::Post_Update`)
    fn register_with_app(app: &mut App, stage: impl StageLabel);
}

A ScriptHost essentially orchestrates how ScriptEvent's get acted on/distributed to your scripts while also providing utilities for loading them from byte sequences.

Script

Script related components like ScriptCollection and Script are essentially just markers describing which script assets are attached to which entities, they do not hold any script runtime data or contexts. The need for ScriptCollection arises since one can only have one instance of a component per entity as of bevy 0.7.

ScriptContexts

The actual script contexts, or script VM's are stored inside the ScriptContexts resource, which is generic over the type of context it stores. As you can see from this structure:

pub struct ScriptContexts<C> {
    /// holds script contexts for all scripts given their instance ids.
    /// This also stores contexts which are not fully loaded hence the Option
    pub context_entities: HashMap<u32, (Entity, Option<C>, String)>,
}

impl<C> Default for ScriptContexts<C> {
    fn default() -> Self {
        Self {
            context_entities: Default::default(),
        }
    }
}

impl<C> ScriptContexts<C> {
    pub fn script_owner(&self, script_id: u32) -> Option<Entity> {
        self.context_entities.get(&script_id).map(|(e, _c, _n)| *e)
    }

    pub fn insert_context(&mut self, fd: FlatScriptData, ctx: Option<C>) {
        self.context_entities
            .insert(fd.sid, (fd.entity, ctx, fd.name.to_owned()));
    }

    pub fn remove_context(&mut self, script_id: u32) {
        self.context_entities.remove(&script_id);
    }

    pub fn has_context(&self, script_id: u32) -> bool {
        self.context_entities
            .get(&script_id)
            .map_or(false, |(_, c, _)| c.is_some())
    }
}

There is a unique one-to-one mapping between script components (which all have unique ID's) and their contexts/vm's.

Now the actual meaning of what a ScriptContext is depends on the ScriptHost itself, but for Lua this is represented as a Mutex<mlua::Lua> as seen here

Answer

What does this imply for Lua scripts? It means that every instance of a Script corresponds to a unique Lua VM, so all of your points 1 through 3 are indeed upheld with this crate.

When it comes to your point 4 however, while it may be possible to run multiple Lua VM's concurrently in general, in the current implementation of RluaScriptHost, script events are handled sequentially as seen here:

fn handle_events<'a>(
        &self,
        world: &mut World,
        events: &[Self::ScriptEvent],
        ctxs: impl Iterator<Item = (FlatScriptData<'a>, &'a mut Self::ScriptContext)>,
    ) {
        let world_ptr = world as *mut World as usize;

        world.resource_scope(
            |world, mut cached_state: Mut<CachedScriptEventState<Self>>| {
                let (_, mut error_wrt) = cached_state.event_state.get_mut(world);

                ctxs.for_each(|(fd, ctx)| {
                    let success = ctx
                        .get_mut()
                        .map_err(|e| ScriptError::Other(e.to_string()))
                        .and_then(|ctx| {
                            let globals = ctx.globals();
                            globals.set("world", world_ptr)?;
                            globals.set("entity", fd.entity.to_bits())?;
                            globals.set("script", fd.sid)?;

                            // event order is preserved, but scripts can't rely on any temporal
                            // guarantees when it comes to other scripts callbacks,
                            // at least for now.
                            // we stop on the first error encountered
                            for event in events {
                                // check if this script should handle this event
                                if !event.recipients().is_recipient(&fd) {
                                    continue;
                                }

                                let mut f: Function = match globals.get(event.hook_name.clone()) {
                                    Ok(f) => f,
                                    Err(_) => continue, // not subscribed to this event
                                };

                                let ags = event.args.clone();
                                // bind arguments and catch any errors
                                for a in ags {
                                    f = f.bind(a.to_lua(ctx)).map_err(|e| {
                                        ScriptError::InvalidCallback {
                                            script: fd.name.to_owned(),
                                            callback: event.hook_name.to_owned(),
                                            msg: e.to_string(),
                                        }
                                    })?
                                }

                                f.call::<(), ()>(())
                                    .map_err(|e| ScriptError::RuntimeError {
                                        script: fd.name.to_owned(),
                                        msg: e.to_string(),
                                    })?
                            }

                            Ok(())
                        });

                    success
                        .map_err(|e| {
                            error!("{}", e);
                            error_wrt.send(ScriptErrorEvent { err: e })
                        })
                        .ok();
                });
            },
        );
    }

The reason for this is that there is no guarantee that two scripts won't modify the same parts of the bevy World. Script concurrency is something I am considering, but it would require a lot of copying of what's already present in bevy and some sort of query syntax script-side to ensure safety.

However if concurrency is something you desperately need and you're sure your scripts won't violate rust safety rules, then you could most certainly implement your own ScriptHost which does in fact run your scripts in parallel. If this is something that people might want from this crate I am also happy to introduce cargo features to enable such behaviour in the feature!

I will note however that Unity for example does not run C# in parallel at all, and so this isn't the end of the world.

Let me know if this helps!

[coderedart]

thank you. the README did not mention the scrip contexts, which finally made everything fall into place for me.

the main intended usage for scripts is to just provide egui to lua, so i won't lose much if the scripts run sequentially.

hope this crate gets published to crates.io soon :)

[makspll]

Yeah I need to improve some things in the presentation, hopefully once I publish to crates.io the presence of online documentation will help a bit.

The only reason I haven't published yet is that I am working on the bevy API which I believe to be an essential feature, and also because I want to be able to break public API's to develop quickly for now :P

Oh! That sounds great! do link me your crate if you end up using this one, I'd love showcase it in the readme when it's done!

[coderedart]

I was able to successfully use bevy + bevy_mod_scripting + luaegui. although, now i just need to start adding bindings to functions as i use them in my projects to eventually cover the majority of the egui api.

The LuaArg type was simply the luaegui::Context which i get from Res<bevy_egui::EguiContextMut> in a normal system and send it via the priority event writer.

the main problem was the stale branch reference in the tealr dependency entry of bevy_mod_scripting crate. had to fork / patch it to tealr master for it to work. I also just used Cargo doc to look at the docs, and it was super easy to understand how bevy_mod_scripting worked and how to use it :) .

one thing that is also missing at the moment for a simple plugin system is being able to set package.cpath and package.path to the directory of the plugin. the main init script of a plugin would probably do a bunch of require("mylib.lua") or load native modules, but there is no way to set them afaik except implementing my own custom RluaHost. default paths have current directory and std userlib directories like /usr/lib/lua.

maybe ApiProvider could provide the path of the script too? that would allow me to easily get the parent folder of the script and set it as the package.{path, cpath} before the script is loaded.

EDIT: might as well add a preview

luaegui.mp4

by using the asset_server.watch_for_changes() i can now script egui live. ofcourse, my app is also transparent and passthrough, so it feels like magic :)

[makspll]

Fantastic news! thanks a lot for the constructive feedback, much appreciated!
This is exactly why I love scripting, hot reloading still blows my mind!

I will fix the tealr dependency issue tonight, I forgot to switch it back after the PR got merged (i needed it initially for full lua doc features).

Now I am gonna ask a bunch of possibly irrelevant or already answered questions just to make sure I understand what you're saying ;)

I will investigate your use case/package.cpath problem, could you elaborate on what the problem is exactly ?
If memory serves me right scripts are able to require() paths relative to their own location, is this different for C dll's?
Or is the problem wanting to load libraries from a location other than the scripts folder?

Also lastly, does modifying the LUA_CPATH env variable at runtime help at all in this case ?

When it comes to script paths, i think the bevy loader does not actually give you those back, so I'd have to store them on the script assets themselves, but providing those paths is certainly possible.

Could you provide me with a concrete use case (and c lib) so I can experiment with it and try to come up with a solution?

[coderedart]

require starts searching from current directory. current directly is usually the project root where we run Cargo run

so, if there's two scripts in assets/scripts named sample.lua and hello.lua. then, if you want to require hello.lua inside sample.lua, then you write require("assets/scripts/hello") rather than require("hello").

usually, you would want plugins to be self contained and portable. so, i would just set package.path and package.cpath to the specific plugin root directory rather than just put all scripts inside the assets/scripts directory.

for example. assets/plugins be where the directory for my bevy game exists. a new plugin named "fps_counter" would be installed in such a way.

assets/
	plugins/
		fps_counter
			main.lua // the main script. this will have require("modules/counter") and require("native_modules/mmap")
			plugin.json // some details about version, download link, permissions required etc..
			modules/
				counter.lua // some library functionality in lua
			native_modules/
				whatever.so // on linux. probably made with mlua::module feature in rust for more performance like using machine learning to predict where the cake item is hidden :D

I would set the package.path and package.cpath to assets/plugins/fps_counter which would make sense. ofcourse, i might also want to set them to modules and native_modules directories directly too to restrict them even further.

LUA_CPATH would just set the env globally, while i want this to be specific to a particular script (plugin).

this will allow me to install, remove plugins cleanly and each plugin will exist in isolation. if something goes wrong, i can just tell users to go into this folder and manually delete the plugin that's causing issues.

[makspll]

@coderedart please have a look at #17.

Specifically of interest to you, I added an initialization hook:

    fn setup_script(&mut self, script_data: &ScriptData,ctx: &mut Self::ScriptContext) -> Result<(), ScriptError>;

which is guaranteed to run once per script (rhai instantiates the API once), and is given script data, namely its name. Now this name can be whatever you want, but good practice is for this to be the script path. I believe this should let you do script dependent import setup?

[coderedart]

aye that should work. once i have the path, i can get the rest of the settings from the directory of the script.

[makspll]

Fantastic! let me know if anything else pops up :)

[makspll]

Should be addressed in #67