Support for environment variables as default

[mariocj89]

As I user I'd like to be able to specify some of my arguments as environment variables.

Quite often I have a single bash script performing many calls to a script written using python-fire. Many of the functions defined in the python script contain shared arguments which I am just effectively passing from my environment variables to each of the calls. Having a way to pass those through would be extremely useful.

Desired behaviour:

Given:

$ cat run.py
import fire
def function_1(arg1, arg2=1):
    print(arg1, arg2)
fire.Fire(function_1)

I'd like:

$ arg1=arg1 python3.6 run.py -- --use_environment
arg1 1
$ arg1=arg1env arg2=arg2env python3.6 run.py arg1 arg2 -- --use_environment
arg1 arg2
$ arg1=arg1env arg2=arg2env python3.6 run.py arg1 -- --use_environment
arg1 arg2env
$ arg1=arg1env arg2=arg2env python3.6 run.py -- --use_environment
arg1env arg2env
$ arg1=arg1env python3.6 run.py -- --use_environment
arg1env 1

This would be extremely useful as then I'd be able to have the following:

export HOST=mydesiredhost
./machine_manager ping -- --use_environment
./machine_manager shutdown -- --use_environment
# No need to pass --host=mydesiredhost on each command

[mariocj89]

I have a posible implementation in my fork. Happy to sign the CLA and send a PR but wanted to know if you like the idea first.

[dnaranjo89]

👍

[dbieber]

Interesting idea.
Can you help me to understand the motivation for this?

One thing that I don't get yet (and keep in mind I've only been thinking about this a few minutes so far) is why you don't just use the environment variables explicitly.
Your examples would become:

$ arg1=arg1 python3.6 run.py $arg1
arg1 1
$ arg1=arg1env arg2=arg2env python3.6 run.py arg1 arg2
arg1 arg2
$ arg1=arg1env arg2=arg2env python3.6 run.py arg1 $arg2
arg1 arg2env
$ arg1=arg1env arg2=arg2env python3.6 run.py $arg1 $arg2
arg1env arg2env
$ arg1=arg1env python3.6 run.py $arg1
arg1env 1

^^would that be sufficient for you, or does the --use_environment flag still help you?

[mariocj89]

Sure thing!

At the moment I have quite complex shell scripts to configure environment variables or to perform some environment-related set-up.

A good example at the moment is Solr. When you want to customize Solr (100% of the times unless you are just trying it out) you are encouraged to source this file. Many tools provide this as a hook for customization, just a file you source with your environment.

This leaves you with the following:

tool-name
  <sources> tool-name.in.sh

Quite often your .in.sh needs more than just "hardcoding the values". As an example, you might have parts of the configuration in different systems, a database, other services in your DC, etc.
At this point, it becomes quite often to have a script with functions you include in your .in.sh

# tool-name.in.sh
source too-name-plumbing.sh

XYZ = `function_in_plumbing_to_get_xyz`

# Notify of startup
function_in_plumbing_to_notify_startup ELAPSED_TIME
...

OK, I'd to use Python in those situations, the most basic idea is: Create a Python CLI that exposes those variables/functions and at that point, I remember python-fire.

I can know define all those nasty functions I was defining in my plumbing file with python and just expose all that I have defined in my file with python fire! And what is even better, those functions can be used from other python scripts (as fire is triggered only when the script is run as main).

There is only though one last nasty part missing. On one scenario it makes sense to pass the arguments as kwargs whilst in the other (sourcing and plumbing) all is usually done via environment variables. The start scripts set many of those and if I find quite "dirty" to have my plumbing scripts accessing os.environ all around.

If only we could have the arguments to default to environment variables! Then all functions nicely express their dependencies and can be used both when scripting/development and when used to enrich a sourced file.

I'd like to use this in many scenarios as such. Example .bashrc, I can have a .bashrc.py and code all my logic there. The functions in .bashrc.py beautifully expose their dependencies in the environment and those are magically passed with Fire with this PR.

If this kicks off I'd probably send another PR to have a documented/fixed way to list all commands available in your Fire app. As at that point, the .bashrc can become just a: Go to this fire app and set the result of all commands in environment variables.

That will allow me to create a bridge sourcing file as:

BRIDGED_PY ?= "plumbing.py"

for variable in `$BRIDGED_PY -- --get-commands`; then
  export $variable= "`$BRIDGED_PY $variable -- --use-environment`"
end

I can now create all my environment code to be sourced in Python and all the functions take (in a really nice documented way) all the variables from the function signature (which come from the environment) :D

Forget about the old days where you had no idea what a shell script function needed to work or going through the pain of testing those shell scripts because there was no better way for that use case.

Bye Bye Shell script

Closing:

I can "live" without it as at the moment I just have a decorator I put to all functions but that breaks other parts of Fire. Having it built-in would be amazing! Other tools like click have the option to default to env vars on a per argument basis. Given the nature of python-fire it would be amazing to have a generic solution.

If it sounds good I'd like to do a lightning talk in PyConUK about how to kill sh with Python and mention python Fire.

[dbieber]

Thanks for explaining:

I find it quite "dirty" to have my plumbing scripts accessing os.environ all around.

Totally agree with you there. (Aside: Even for writing shell scripts I think it's a good practice to grab all the env variables up front, and give them a default value if they haven't been defined. This way a reader of the code can quickly see all the environment variables that are used, and you lessen the risk of using an unset environment variable incorrectly.)

... the old days where you had no idea what a shell script function needed to work

Agree here too. I also definitely appreciate the appeal of having all inputs defined in a function signature, unlike with traditional shell scripts where the env variable usages are scattered throughout.

Question / Example
Question:
One thing I was hoping you would mention though is why you'd rather not reference the environment variables in the shell script that calls your Python Fire CLI. Why is this?

Example, illustrating the question
You said you'd like to use Fire like this:

export HOST=mydesiredhost
./machine_manager ping -- --use_environment
./machine_manager shutdown -- --use_environment
# No need to pass --host=mydesiredhost on each command

But you can already use it like this:

export HOST=mydesiredhost
./machine_manager ping --host=$HOST
./machine_manager shutdown --host=$HOST
# There's no need to pass --host=mydesiredhost on each command

So why do you prefer the former? One reason I can think of is it that there can be a large number of environment variables, and so the latter approach could become messy. A second reason I can think of is that you now have to keep the list of args in sync between the Python script and the shell script, which can be a maintenance burden. Let me know if there's a different reason you were thinking of (or maybe you didn't know the latter approach was possible and it is good enough :)).

---

Lastly, here are some concerns to consider for implementation:

• Python Fire supports functions that take **kwargs and *varargs.
  • If someone used the --use_environment flag with a function that accepted **kwargs, would we pass in the whole environment?!
• Python Fire supports calling functions on members returned by other functions.
  • Multiple functions might accept variables with the same name.
  • When using command line arguments, we can use the argument order.
  • How would we resolve this case when using --use_environment?

[mariocj89]

Thanks for the answer! Really appreciated.

One thing I was hoping you would mention though is why you'd rather not reference the environment variables in the shell script that calls your Python Fire CLI. Why is this?

Indeed, think about using it for your .bashrc. Many functions will depend on the user, also many depend on the host. You might also want to specify http_proxy as a possible input that your function might be able to use. At the beginning, I was thinking of using classes with Fire but it quickly blows up due to the combinations.

It quickly becomes a pain having to update 3 scripts whenever you want to pass a new variable through. It is more like the first script defines a pool of variables and now you can just pick from that explicitly.

If you want to use "host", SAY IT, that helps with future refactoring and retiring variables, leave alone the fact that you can reuse the same script from other python scripts.

---

About your concerns:

If someone used the --use_environment flag with a function that accepted **kwargs, would we pass in the whole environment?!

Really good point, I did not even think about it. Was going just to not override it but the idea of the whole environment might be quite powerful. I personally don't see myself using it but happy to implement it.

Python Fire supports calling functions on members returned by other functions.

Indeed, you'll need to specify all arguments, you can see it the same as having a default in your function definition.
If your fire app has def example(optional_var="optional") optional_var needs to be provided for the function chaining to work. isn't it?

If instead of having the environment provide default you want that to be the actual value we can make this work, but not sure if that is worth it :X (we might enable that via options in the flag? 😒 )

[jtratner]

Perhaps you could specify a prefix for environment variables (similar to how npm and pip have env vars like npm_config_foo to set the foo value)? The prefix could even be customizable. I'd be hesitant to accidentally leak, say, my AWS access key to a function just because it happened to have a matching keyword arg. So, say prefix was FIRE_ Then FIRE_HOST=myhost FIRE_NPROCS=3 FIRE_GENOME=hg19 python something.py Would translate to --host=myhost --nprocs=3 --genome=hg19 What happens with '-' by the way? Would that be transparently handled?

…

On Mon, Oct 9, 2017 at 11:23 AM Mario Corchero ***@***.***> wrote: Thanks for the answer! Really appreciated. One thing I was hoping you would mention though is why you'd rather not reference the environment variables in the shell script that calls your Python Fire CLI. Why is this? Indeed, think about using it for your .bashrc. Many functions will depend on the user, also many depend on the host. You might also want to specify http_proxy as a possible input that your function might be able to use. At the beginning, I was thinking of using classes with Fire but it quickly blows up due to the combinations. It quickly becomes a pain having to update 3 scripts whenever you want to pass a new variable through. It is more like the first script defines a pool of variables and now you can just pick from that *explicitly*. If you want to use "host", SAY IT, that helps with future refactoring and retiring variables, leave alone the fact that you can reuse the same script from other python scripts. ------------------------------ About your concerns: If someone used the --use_environment flag with a function that accepted **kwargs, would we pass in the whole environment?! Really good point, I did not even think about it. Was going just to now override it but the idea of the whole environment might be quite powerful. I personally don't see myself using it but happy to implement it. Python Fire supports calling functions on members returned by other functions. Indeed, you'll need to specify all arguments, you can see it the same as having a default in your function definition. If your fire app has def example(optional_var="optional") optional_var needs to be provided for the function chaining to work. isn't it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABhjq1rOL2ZH_lEedFxjd3QWavNDgPrJks5sqmSZgaJpZM4PxD7V> .

[mariocj89]

What happens with '-' by the way? Would that be transparently handled?

Not sure what you mean. You cannot have - in your variable names in Python neither in shell. Therefore fire translation of _ to - is safe (won't cause collisions).

Example:

$ sample_var=1 python3.4 sample.py -- --use_environment
1
(venv3) mario at SOBREMESA in ~/workspace/python-fire on env_variables*
$ cat sample.py
import fire
def test(sample_var):
    print(sample_var)
fire.Fire(test)

I'd be hesitant to accidentally leak, say, my AWS access key to a function
just because it happened to have a matching keyword arg.

Humm interesting. You mean you are worried to capture a value by mistake? You'd then have any way an issue of having two variables with the same name on the upper script, isn't it? You could fall into the same situation by just accessing the variable. Not sure I got your worry here. 😕

Also, note this is not the default behaviour, you explicitly need to specify that you want the environment to set your defaults.

Thanks for the comments!

[jtratner]

Not sure what you mean. You cannot have - in your variable names in Python neither in shell. Therefore fire translation of _ to - is safe (won't cause collisions).

Ah I was thinking of CLI interface - gotcha.

Humm interesting. You mean you are worried to capture a value by mistake? You'd then have any way an issue of having two variables with the same name on the upper script, isn't it? You could fall into the same situation by just accessing the variable. Not sure I got your worry here. 😕

I'm thinking this could be a route to leaking secrets unintentionally to functions (or, more specifically, accidentally passing through extra environment variables because the names happen to collide, e.g. tmp, path, or user) - tho secret leaking is less of a big deal given that the python process can always access the environment.

AWS_SECRET_ACCESS_KEY=1234
ARG1=blah
ARG2=somethingelse

def malicious_function(arg1, arg2, aws_secret_access_key=None):
    print(aws_secret_access_key)

Or alternatively, consider this:

def load_something(a, b, c, path=None):
    """load something important, if path is set load from there"""
    read_csv(path or DEFAULT_PATH)

you'd end up with some long string substituted in for path.

[mariocj89]

def malicious_function(arg1, arg2, aws_secret_access_key=None):

Exactly, you just cannot protect that, and if you print a variable named "aws_secret_access_key" well.. 🔐 same can be argued about printing env right? I might agree on the fact that you might have something like "token" that you dont want to go out and the name is not descriptive enough, but note that the caller choses whether to pass those as default or not.

def load_something(a, b, c, path=None):

We can try to look into more selective defaulting if you think it is an issue. Or maybe on enhancing how the help of fire is returned to the user (showing the defaults would mitigate this).

But basically, the idea of getting the path if you have path as an argument is the whole point 😕

[jtratner]

Gotcha - I think the standard most other tools use is a prefix, e.g. for puppet:

FACTER_hostname=some-override-for-hostname

so I'm suggesting that use environment would take a prefix like:

MYMOD_path=whatever
python mymodule.py -- --use-environment-prefix=MYMOD_

[mariocj89]

python mymodule.py -- --use-environment-prefix=MYMOD_

That is actually quite interesting.

If the user wants it can pass that or just leave it empty to get the default environment. 👍

[jtratner]

How do you feel about optionally specifying it to the Fire constructor? `Fire(environment_prefix='FOO_')`

…

On Mon, Oct 9, 2017 at 1:19 PM Mario Corchero ***@***.***> wrote: python mymodule.py -- --use-environment-prefix=MYMOD_ That is actually interesting. If the user wants it can pass that or just leave it empty to get the default environment. 👍 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABhjq38OiuuDpv9umFvglMwECAjjMWFyks5sqn_SgaJpZM4PxD7V> .

[mariocj89]

That'd work as well. I put it as a flag for no special reason, just because I saw other options pases like that. In most situations you probably want to specify it at the time you write the script so that would make sense.

Any works for me :)

@dbieber what do you think?

If you guys tell me what you prefer o can try to get the PR ready this weekend.