Custom Call Signature Per Script Context

[jdconrad]

We should make each context execute scripts based on exactly what gets passed in and returned. This eliminates a huge number of potential boxes for something like scoring that returns a double value out of a script. This may also eliminate a large number of hash look ups as certain parameters will be passed directly into the method instead of accessed from a map. Over millions of documents these performance gains really add up. Another benefit will be prevention of accessing null variables that may not belong in a context. For this to work the context will have to be passed in at compile-time.

I need to run through all the script cases and figure out what the contexts should be as I'm not sure if the current ones cover all the situations.

For file based scripts, I would propose for Painless and Expressions that the script be named something like script.score.painless now to embed the context in the name. For something like Groovy and other languages we can have an "unknown" context so these remain backwards compatible. The "unknown" context would revert to the old call signatures. Once the older languages are fully removed then we can remove the "unknown" context.

[nik9000]

I agree with the idea of having the context at compile time. Are you thinking you'd have to pass in a Class and we'd would analyze the publicly visible getters? Something more explicit and less magic? Like a Map<String, ContextField> where ContextField has Method references for getting and (optionally) setting?

[jdconrad]

@nik9000 I can't say this is 100% thought-through yet as I need to go through the code and figure out what the contexts should be (I don't think the existing ones completely fit all the cases), but as I think you've seen in the past from me I prefer explicit over magic typically :) . I was thinking more along the lines of perhaps an interface per context with the signature that must be implemented by the script object that is returned from compile. CompiledScripts that implement the specific interface are what get's executed rather than a call back to the ScriptService. I envision a CompiledScript containing more than one interface possibly so we hopefully wouldn't end up with an excessive number of new classes.

[nik9000]

That makes sense. I think it'd be pretty cool to say "here is some text that a user sent me, please compile it and return something that implements this interface". We wouldn't have to pass in there ctx objects any more at all which'd be cool.

I like the flexibility of letting the user of the script engine specify the interface that it needs. It lines up pretty well with "what can this script engine do?" because implementations like Mustache and Expressions can just choke when the caller asks them for an interface they don't understand but groovy and painless can actually implement the interface. Groovy through dynamic stuff and painless with static stuff.

[jdconrad]

@nik9000 Exactly what I was thinking! As a bonus, I really think this will help clean up the ScriptService quite a bit too.

[nik9000]

++

[jdconrad]

So I began to modify the code to allow context at compilation for script engines that would like it, and ran into an issue with stored scripts.

In RestPutStoredScriptAction we add handlers like this:

    protected RestPutStoredScriptAction(Settings settings, RestController controller, boolean registerDefaultHandlers) {
        super(settings);
        if (registerDefaultHandlers) {
            controller.registerHandler(POST, "/_scripts/{lang}/{id}", this);
            controller.registerHandler(PUT, "/_scripts/{lang}/{id}", this);
        }
    }

I would like to optionally have a context as part of this API, so I would propose that we change the handlers to be /_scripts/{id} and take in JSON as we do for other API's. This would allow us to add an optional context parameter.

Definitely looking for some feedback on this issue.

[rjernst]

Changing to just key on id in the stored scripts api sounds right to me. The cache is based on script id alone right?

[jdconrad]

The cache is based on both id and lang. And with context added, it would be based on all three.

[s1monw]

I would like to optionally have a context as part of this API, so I would propose that we change the handlers to be /_scripts/{id} and take in JSON as we do for other API's. This would allow us to add an optional context parameter.

you can still add an optional parameter here even with keeping this URL?

 controller.registerHandler(POST, "/_scripts/{lang}/{id}", this);
 controller.registerHandler(PUT, "/_scripts/{lang}/{id}", this);
 controller.registerHandler(POST, "/_scripts/{lang}/{id}/{context}", this);
 controller.registerHandler(PUT, "/_scripts/{lang}/{id}/{context}", this);

I am not sure if that is sufficient but I thought I'd bring it up.

++ on this issue in order to get good performance from scripts we have to require this and I really wonder if we should make it non-optional and require it for all scripts. Existing ones can still use Unknown or All? We have the infra to do this added when we required the lang to be present on all scripts. WDYT?

[clintongormley]

I love the idea but i would prefer it if it were implicit (ie not user specified). Also, I think that specifying eg aggs is insufficient as some aggs scripts return doubles and some return strings/objects.

How about this:

We have a number of contexts defined internally, eg returns_double, whatever. When we execute (eg) a file script, we pass in the context plus the script file name to the script service. The service tries to retrieve a compiled script that is keyed off the file name PLUS the context. If it doesn't exist, it retrieves the script from the file, compiles it for the context, and caches it with the context.

So if the same script is used with different contexts, then it will be cached multiple times. That said, most scripts will only be cached once or maybe twice.

Our contexts can be as specific as we need and this is totally transparent to the user.

[nik9000]

I suppose the nice thing about knowing the context up front is that you can
compile the script in the context right away and get useful error messages.

I, too, had envisioned this be done when the script is first used and not
at storage time, but I'd jack likes it at storage time that is fine with me
so long as we can work it out. I wonder if we can do this at first usage
time in one PR so we and then look at the separate storage in a second PR?

On Sep 13, 2016 7:01 AM, "Clinton Gormley" notifications@github.com wrote:

I love the idea but i would prefer it if it were implicit (ie not user
specified). Also, I think that specifying eg aggs is insufficient as some
aggs scripts return doubles and some return strings/objects.

How about this:

We have a number of contexts defined internally, eg returns_double,
whatever. When we execute (eg) a file script, we pass in the context plus
the script file name to the script service. The service tries to retrieve a
compiled script that is keyed off the file name PLUS the context. If it
doesn't exist, it retrieves the script from the file, compiles it for the
context, and caches it with the context.

So if the same script is used with different contexts, then it will be
cached multiple times. That said, most scripts will only be cached once or
maybe twice.

Our contexts can be as specific as we need and this is totally transparent
to the user.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#20426 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANLohtYrmaoHlQl5EnFiiijXQZUoJHOks5qpoJygaJpZM4J6wjq
.