simple plugin support where fpm-NAME commands are searched for

[urbanjost]

Relating to #361 and #211

A simplistic plugin support method whereupon when a subcommand is
not found the PATH variable is used to search for commands of the
form fpm-NAME and those are executed in lieu of an error message
indicating an unknown subcommand was entered.

The executable is simply called. No data is passed via arguments,
environment variables, or files (yet).

This allows for concurrent development. This is an alpha feature
to support testing the pros and cons of plugins, aliases, a
built-in scripting language (see Fortran scheme, lua, ...),
response files (see M_CLI2), ...

The subcommand must currently support its own help. The list subcommand
does not currently list available plugins -- that is, nothing in fpm
documents that external subcommand exists.

This allows related commands such as repository search, formatting
tools, and other utilitites to develop independently of the core
fpm functionality without bloating the core program with the many
external packages that would be potentially useful to other functions,
to allow for easier development of system-dependent subcommands, and
so on.

It could become much more useful with the ability to pass categories
of pathnames, but some of that exists with the --runner option.
With pathnames, you could say "reformat all my source files" or
"expand tabs in all my source files" or "edit my source files"
without the complex syntax required by the --runner option, and so on.

Best example so far:

install fpm-search(1) in your path, and you can use it using

fpm search 'date|time'

Since fpm-search is a stand-alone utility is requires no data to be passed
to it but feels very much a part of fpm(1))); even though it requires
a very different set of packages to be installed than the core fpm(1)
package -- and development and installation can occur concurrently.

[ghost]

Hi @urbanjost . I like the idea. Currently there's an issue running it on Windows: "<ERROR> unknown subcommand [search]"

[urbanjost]

I do not have an MSWindows environment so I will try to simulate it or get hold of one. I am guessing the separator() function did not work as I expected it to. A debug mode will hopefully sort it out shortly. I have been using on Linux RedHat and it is working as expected. I made a few other simple ones like a few of the simpler cargo ones to select example programs with an ncurses(3f) menu and to create a README file based on the output of tree(1) and it is interesting. I was thinking maybe a line in the fpm-search(1) help text saying where the directions are to add your project to the repository would be nice, by the way.
If I cannot sort it out if I make a debug version could you run it on Windows? You are running in a cmd Window or mingw or Cygwin or WLS or ...?

I think I see it. On MSWindows an executable has to end in '.exe' or '.bat' if the suffix is left off (right?) and I did not check for that.

[ghost]

Hi @urbanjost , yes I'm running it in a cmd.exe window. I'll run it on GDB and report later.

And yes, I've just tested the patch in a linux vm and it works nicely, thanks!

[urbanjost]

Thank you for fpm-search. I was already using it as a stand-alone command but it is the perfect initial case for trying this type of plug-in method too.

[ghost]

Thank you @urbanjost , I've tested the latest patch. Here are the results:

• If I put fpm.exe, fpm-search.exe and all dependencies into a single place, run set path=C:\place, then fpm search package works
• If I run set path=C:\other;C:\place, then fpm search package fails.

[ghost]

It is working as expected on Windows. Thank you @urbanjost.

[urbanjost]

Thanks. I will add some test cases for some of the routines like 'which(3f)' and 'separate(3f)' based on that.

[milancurcic]

Thank you for starting it @urbanjost. I generally like and support this direction.

Is this approach safe? For example, if a user mistypes fpm knit instead of fpm init, it will run fpm-knit, is there any chance that fpm-knit is something that already exists and is potentially destructive? I think this is highly unlikely, but thought I'd ask. I don't have much experience with security.

If not, perhaps plugins should be whitelisted in a config file?

But if yes, I think it would a very nice user experience to just fpm new fpm-myplugin, add some code, and easily play with a 3rd party command without having to tweak fpm itself.

[urbanjost]

I like it better from a developmental standpoint in that even the standard package can be delivered and developed as a group of related utilities but still act in many ways like one monolithic application. So far I see something similar in cargo-* and git-* and their might be others; but I find something lacking in communicating data and extending existing commands that I think would make this far more useful. Have to start somewhere and this change can be made with essentially no impact on existing behavior and seems to have corollaries in everything from rust::cargo to git. So using fpm-search as an example:

1. The utility can and does stand on it's own as a separate program -- so why access it this way? The search command is ultimately a basic function needed by fpm so it makes sense to appear to the user as a standard fpm subcommand. But it's function and requirements are significantly different than the model building function of fpm. So doing it this way allows for concurrent development, minimizes the external dependencies such as an RE library, libcurl, a hash library (not that those might not all be needed by the module engine at some point) ... so I think that this model works for this particular command very well; could actually be used as-is even if fpm-search is included directly; and looses no functionality. Could list other pro/cons.

So personally I think the main functionality missing in the model build core is providing compiler options that are easily distributed with the package. This lacking feature is the show-stopper I have seen listed most often by people who look at but do not adopt fpm.

The second highest priority in my view is having a simple CLI interface that lets you put a package into a repository and lets you search the repository(s). And a local repository needs supported for in-house code development. The fpm-search program does the search of the current repository nicely.

But making usage and development extensible and modular is my personal third priority. So I think this is a reasonable first step too but think there is a missing functionality here. Let's say I want to make an 'fpm edit' command that has some options like 'edit source file that build fails on' or 'edit files changed since last commit' or 'edit file I click on from tree of files'; or commands like 'reformat all my source files' ... some nice way for an external program like this to query the model and git information would help a lot with that, although all these functions could be done as separate applications they could benefit from being able to "talk" to the model. I would do that now on a POSIX machine but trying to think of a "pure Fortran" model is alluding me, other than maybe NAMELIST files. Everything I think of needs a system or C interface; which I would like to see avoided if possible. So trying to think of a way to merge this and functionality like --runner and aliases provides as a next step.

After looking at git-* usage it looked like this would be primarily a hidden feature to further modular development; but seeing the cargo-* usage it looks like it could get a large amount of community plugins contributed -- will be interesting to see if either starts developing.

Got disconnected by the dogs in the middle of this. So the idea is: I want to make a plugin that could benefit from basic model information. Picturing that when fpm runs a plugin it calls a routine called dumpmodel(3f) that writes a file in build/model.nml perhaps that has the TOML metatadata in a METADATA group, and a group for each major file grouping like TEST, EXAMPLE, BUILD_TARGETS, and APPS. Without knowing anything about the internals of fpm or TOML files I can use standard Fortran to read the file (maybe file groups also have a MAX_LENGTH and MAX_NUMBER_OF_FILES value group) assuming at least for now the fpm command is always executed in the top directory of the package and write some "model-aware" plugins. If that was agreed to it would have to have long-term support for anyone to want to do much with it. That allows for Fortran-based plugins to be developed where the interface to the model uses standard Fortran and no other knowledge is required.

We really do not have a --list option in the current core fpm that lists target source files. That would be handy. That is not what ''build --list' produces.

Simple and standard to at least implement part of "build.nml". Anyone think that would be a good thing to provide long-term?

[ghost]

Regarding security, I've noticed that Windows Defender displays a notification the first time I run a newly built fpm.exe (with this PR #362 enabled). I was able to reproduce it:

fpm.mp4

[ghost]

Regarding security, I've noticed that Windows Defender displays a notification the first time I run a newly built fpm.exe (with this PR #362 enabled). I was able to reproduce it:

fpm.mp4

Yes, I'm sure fpm/fpm-search are not viruses/threats.

[ghost]

Although I like writing fpm-search, I'm aware that it's a prototype and it will have a very short life, at least in its current form (We don't have the fpm-registry specification yet).

If security is a concern and if we don't find a way to avoid the Windows Defender scary notification, I'd rather transfer fpm-search to Fortran-Lang Org. I believe Fortran-Lang Org/Community could crowdfund a security certificate so that all binaries could be digitally signed and that will make Windows Defender happy I think.

Let's see how that idea will evolve.

[LKedward]

Great stuff @urbanjost I like this approach! It's really nice to see it working already with fpm-search.

---

to query the model and git information would help a lot with that, although all these functions could be done as separate applications they could benefit from being able to "talk" to the model. I would do that now on a POSIX machine but trying to think of a "pure Fortran" model is alluding me...

Regarding access to the fpm model structure, a possible solution is to use fpm as a package dependency and to call the build_model procedure from your plugin package.

This isolates the plugin from whatever fpm version is installed on the system and doesn't require serialising then parsing the model structure (which would require maintenance and be dependent on the fpm version).

fpm is a big package to have as a dependency, but we could always isolate the 'middle-end' (?) code that constructs the internal model as a separate package to help plugin developers.

I would find this useful for prototyping ideas and/or customising fpmetc.

Unfortunately fpm cannot yet be used as a git dependency due to the folder structure, but I don't think there is anything preventing us from fixing that now.

[ghost]

I should report that the Windows Defender notification is unrelated to this PR. I noticed that Windows Defender started displaying a security notification when I build and run the master and other branches also.. My apologies.