generated pipelines incompatible with nextflow's pipeline sharing

[cimendes]

Nextflow seamlessly integrates with BitBucket, GitHub, and GitLab hosted code repositories and sharing platforms. In theory, a nextflow pipeline generated with flowcraft could be hosted in one of these code repos and then be directly pulled through this nextflow feature. Unfortunatly I've identified a couple of issues that make this impossible with our pipelines:

1). Nextflow only perses the nextflow.config file for the manifest field when the pipeline isn't called "main.nf", ignoring the $IncludeConfig parameter. I've opened an issue on Nextflow's repository (nextflow-io/nextflow#1032), but so far it's flagged as low priority. A work-around is to instead of writing the manifest data in a new config file (mainfest.config), it can be appended to the current nextflow.config. I don't like this solution as the nextflow.config file shouldn't change from repository to repository,

2). Even if the manifest information is included directly in the nextflow.config file, or the script is saved in main.nf, the nextflow will pull all the files into the $HOME/.nextflow/assets directory. The pipeline will break upon execution as the ".forkTree.json" and ".treeDag.json" aren't in the execution directory.

I would like to know your opinion in how we can support the sharing functionality of nextflow as idealy, a pipeline generated with flowcraft could be hosted on GitHub, for example, and then directly pulled with nextflow.

[cimendes]

A way to solve the first part of the issue is to append the manifest information (currently in the manifest.config) to the beginning of the nextflow.config file.

[cimendes]

One of the sources of issues with this is that hidden files (like ".forkTree.json" and ".treeDag.json") aren't staged by git unless you explicitly state them in the git add command. I suggest making these files visible and moving them to the resources folder, that already contains the "main.js.zip" file for the report generation (offline mode). What do you think @tiagofilipe12 ?

[tiagofilipe12]

Just have in mind that report and inspect mode require the file location to properly send DAG to the web service.

[cimendes]

I'm closing this issues as the two lastest PRs (#204 and #209) have addressed the problems discussed here. I've been able to run flowcraft's pipelines remotely with Nextflow without any issue in local machines and in the cloud. Feel free to re-open this issue is you run into this problem in any particular circumstance.