Add the AWS Athena JDBC driver for nf-sqldb plugin #2500

abhi18av · 2021-12-17T12:53:57Z

This PR follows up from the discussion on nextflow-io/nf-sqldb#5

Open database used for testing

I've relied on the Athena database generated by crawling the SRA Metadata as described here the instructions shared in https://www.ncbi.nlm.nih.gov/sra/docs/sra-athena.

The exact instructruction to setup the necessary tables/database on the AWS side are descibed in the official walkthrough https://www.youtube.com/watch?v=_F4FhcDWSJg

Instructions to use the `Athena` JDBC integration

Populate the JDBC URL correctly for the Athena driver with the correct

AWSRegion
S3OutputLocation

jdbc:awsathena://AwsRegion=us-east-1;S3OutputLocation=s3://MY_BUCKET_NAME/AWS_GLUE_OUTPUT_DIR/'

Add the configuration for your AWS account for the Athena JDBC driver


sql {
    db {
        athena {
            url = 'jdbc:awsathena://AwsRegion=us-east-1;S3OutputLocation=s3://MY_BUCKET_NAME/AWS_GLUE_OUTPUT_DIR/'
            user = 'AWS_SECRET_KEY_ID'
            password = 'AWS_SECRET_ACCESS_KEY'
        }
    }
}

Add a query for the Athena database in your Nextflow code (the example below assumes you have a dataset named sra-athena-results with a metadata table)

# main.nf

def sql = "SELECT * FROM \"sra-athena-results\".metadata WHERE organism = 'Homo sapiens' LIMIT 1;"
Channel.sql.fromQuery(sql, db: 'athena')
.view()

Run the Nextflow code using the locally compiled binary

../launch.sh run main.nf

The results are shown as below

(base) Abhinavs-MacBook-Pro:_scratch eklavya$ ../launch.sh run main.nf

N E X T F L O W  ~  version 21.11.0-edge
Launching `main.nf` [loving_northcutt] - revision: aa68ce1be0
log4j:WARN No appenders could be found for logger (com.simba.athena.amazonaws.auth.profile.internal.BasicProfileConfigLoader).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[SRR363812, WGS, BI, public, SRX105026, PRJNA70759.Subject 1, 454 GS 20, SINGLE, RANDOM, GENOMIC, LS454, SRS269932, SAMN00750733, Homo sapiens, SRP009245, 2011-11-10, PRJNA70759, 56, null, 111, 23, null, 454-134 Flow 1, null, null, null, null, null, null, Subject 1, sra, sff, ncbi, s3, gs, ncbi.public, gs.US, s3.us-east-1, {k=gssr_id_exp, v=146935.0}, {k=lsid_exp, v=BROAD:SEQUENCING_SAMPLE:146935.0}, {k=material_type_exp, v=Genomic DNA}, {k=project_exp, v=R81}, {k=bases, v=23383062}, {k=bytes, v=59372830}, {k=gssr_id_run, v=146935.0}, {k=lsid_run, v=BROAD:SEQUENCING_SAMPLE:146935.0}, {k=project_run, v=R81}, {k=region_run, v=1}, {k=run_barcode_run, v=611540051122}, {k=run_name_run, v=Flow1Flow2Run611540}, {k=geographic_location__country_and_or_sea_region__sam, v=missing}, {k=isolation_source_sam_ss_dpl262, v=missing}, {k=strain_sam, v=missing}, {k=primary_search, v=385256.454-134 Flow 1.611540051122.S}, {k=primary_search, v=611540051122.1.TCAG.sff}, {k=primary_search, v=70759}, {k=primary_search, v=750733}, {k=primary_search, v=B_Cell_Immune_Repertoire_Baseline}, {k=primary_search, v=PRJNA70759}, {k=primary_search, v=PRJNA70759.Subject 1}, {k=primary_search, v=SAMN00750733}, {k=primary_search, v=SRP009245}, {k=primary_search, v=SRR363812}, {k=primary_search, v=SRS269932}, {k=primary_search, v=SRX105026}, {"gssr_id_exp": "146935.0", "lsid_exp": "BROAD:SEQUENCING_SAMPLE:146935.0", "material_type_exp": "Genomic DNA", "project_exp": "R81", "bases": 23383062, "bytes": 59372830, "gssr_id_run": ["146935.0"], "lsid_run": ["BROAD:SEQUENCING_SAMPLE:146935.0"], "project_run": ["R81"], "region_run": "1", "run_barcode_run": ["611540051122"], "run_name_run": ["Flow1Flow2Run611540"], "geographic_location__country_and_or_sea_region__sam": ["missing"], "isolation_source_sam_ss_dpl262": ["missing"], "strain_sam": ["missing"], "primary_search": "385256.454-134 Flow 1.611540051122.S"}]

Concern (Log4J)

Given the recent adventures of log4j dependency, we can use exlude it using Gradle safely right?

pditommaso · 2021-12-17T17:54:16Z

Oh, the Athena driver does use Log4j?

abhi18av · 2021-12-17T17:55:40Z

Unfortunately yeah 🤷

pditommaso · 2021-12-17T18:02:57Z

Ok, not such big problem this what slf4j:log4j-over-slf4j is meant for.

However, I would like to avoid including the zip in the Git repo. Not sure if there's a way to download it automatically and include it in the build.

abhi18av · 2021-12-17T18:07:20Z

Ah, okay - yes that helps!

Not sure if there's a way to download it automatically and include it in the build.

I found a Gradle plugin for this one -> https://github.com/michel-kraemer/gradle-download-task/blob/master/examples/unzip.gradle.

Will experiment and circle back

plugins/nf-sqldb/build.gradle

abhi18av · 2021-12-19T19:48:47Z

plugins/nf-sqldb/build.gradle

+defaultTasks 'downloadAndCleanUpLibsFolder'
+
+build.dependsOn downloadAndCleanUpLibsFolder
+compileGroovy.dependsOn downloadAndCleanUpLibsFolder
+compileTestGroovy.dependsOn downloadAndCleanUpLibsFolder
+test.dependsOn downloadAndCleanUpLibsFolder
+copyPluginLibs.dependsOn downloadAndCleanUpLibsFolder


There must be a better way to do this no? 🤔

Paolo, as per your comment #2500 (comment) , I've added slf4j:log4j-over-slf4j library, but not sure how exactly to configure it.

Yeah, a bit ugly. Should not be enough

compileGroovy.dependsOn downloadAndCleanUpLibsFolder

and then it will pull the parent tasks?

Unfortunately, for this one, if I only include compileGroovy.dependsOn ... it fails to compile and it doesn't trigger the download and import of the JAR file.

Execution failed for task ':plugins:nf-sqldb:copyPluginLibs'. > Could not resolve all files for configuration ':plugins:nf-sqldb:runtimeClasspath'. > Could not find :AthenaJDBC42_2.0.25.1001:. Searched in the following locations: - file:/Users/eklavya/projects/code/nextflow/plugins/nf-sqldb/libs/AthenaJDBC42_2.0.25.1001.jar Required by: project :plugins:nf-sqldb * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org BUILD FAILED in 57s

The other task dependencies I added because the Gradle build system threw a warning that the build might not be reproducible if I don't add these dependencies.

plugins/nf-sqldb/build.gradle

pditommaso · 2021-12-21T17:31:21Z

Do you have a working Athena config to use as a showcase?

pditommaso · 2021-12-21T18:40:21Z

OK, think i've managed to simplify a bit the inclusion of the Athena driver. It would be nice to include a no brainer example to use it in the readme. Then I think we can merge it.

pditommaso · 2021-12-21T20:42:43Z

Now does not break, but for some mysterious reason dump these warnings. Give up

> Task :plugins:nf-azure:unzipPlugin
Execution optimizations have been disabled for task ':plugins:nf-azure:unzipPlugin' to ensure correctness due to the following reasons:
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-azure-0.11.2.zip'. Reason: Task ':plugins:nf-azure:unzipPlugin' uses this output of task ':plugins:nf-amazon:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.

> Task :plugins:nf-console:unzipPlugin
Execution optimizations have been disabled for task ':plugins:nf-console:unzipPlugin' to ensure correctness due to the following reasons:
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-console-1.0.3.zip'. Reason: Task ':plugins:nf-console:unzipPlugin' uses this output of task ':plugins:nf-amazon:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-console-1.0.3.zip'. Reason: Task ':plugins:nf-console:unzipPlugin' uses this output of task ':plugins:nf-azure:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.

> Task :plugins:nf-ga4gh:unzipPlugin
Execution optimizations have been disabled for task ':plugins:nf-ga4gh:unzipPlugin' to ensure correctness due to the following reasons:
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-ga4gh-1.0.3.zip'. Reason: Task ':plugins:nf-ga4gh:unzipPlugin' uses this output of task ':plugins:nf-amazon:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-ga4gh-1.0.3.zip'. Reason: Task ':plugins:nf-ga4gh:unzipPlugin' uses this output of task ':plugins:nf-azure:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-ga4gh-1.0.3.zip'. Reason: Task ':plugins:nf-ga4gh:unzipPlugin' uses this output of task ':plugins:nf-console:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.

> Task :plugins:nf-google:unzipPlugin
Execution optimizations have been disabled for task ':plugins:nf-google:unzipPlugin' to ensure correctness due to the following reasons:
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-google-1.1.3.zip'. Reason: Task ':plugins:nf-google:unzipPlugin' uses this output of task ':plugins:nf-amazon:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-google-1.1.3.zip'. Reason: Task ':plugins:nf-google:unzipPlugin' uses this output of task ':plugins:nf-azure:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-google-1.1.3.zip'. Reason: Task ':plugins:nf-google:unzipPlugin' uses this output of task ':plugins:nf-console:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-google-1.1.3.zip'. Reason: Task ':plugins:nf-google:unzipPlugin' uses this output of task ':plugins:nf-ga4gh:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.

> Task :plugins:nf-ignite:unzipPlugin
Execution optimizations have been disabled for task ':plugins:nf-ignite:unzipPlugin' to ensure correctness due to the following reasons:
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-ignite-1.2.3.zip'. Reason: Task ':plugins:nf-ignite:unzipPlugin' uses this output of task ':plugins:nf-amazon:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-ignite-1.2.3.zip'. Reason: Task ':plugins:nf-ignite:unzipPlugin' uses this output of task ':plugins:nf-azure:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-ignite-1.2.3.zip'. Reason: Task ':plugins:nf-ignite:unzipPlugin' uses this output of task ':plugins:nf-console:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-ignite-1.2.3.zip'. Reason: Task ':plugins:nf-ignite:unzipPlugin' uses this output of task ':plugins:nf-ga4gh:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-ignite-1.2.3.zip'. Reason: Task ':plugins:nf-ignite:unzipPlugin' uses this output of task ':plugins:nf-google:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.

> Task :plugins:nf-sqldb:unzipPlugin
Execution optimizations have been disabled for task ':plugins:nf-sqldb:unzipPlugin' to ensure correctness due to the following reasons:
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-sqldb-0.2.0.zip'. Reason: Task ':plugins:nf-sqldb:unzipPlugin' uses this output of task ':plugins:nf-amazon:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-sqldb-0.2.0.zip'. Reason: Task ':plugins:nf-sqldb:unzipPlugin' uses this output of task ':plugins:nf-azure:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-sqldb-0.2.0.zip'. Reason: Task ':plugins:nf-sqldb:unzipPlugin' uses this output of task ':plugins:nf-console:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-sqldb-0.2.0.zip'. Reason: Task ':plugins:nf-sqldb:unzipPlugin' uses this output of task ':plugins:nf-ga4gh:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-sqldb-0.2.0.zip'. Reason: Task ':plugins:nf-sqldb:unzipPlugin' uses this output of task ':plugins:nf-google:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-sqldb-0.2.0.zip'. Reason: Task ':plugins:nf-sqldb:unzipPlugin' uses this output of task ':plugins:nf-ignite:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.

> Task :plugins:nf-tower:unzipPlugin
Execution optimizations have been disabled for task ':plugins:nf-tower:unzipPlugin' to ensure correctness due to the following reasons:
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-tower-1.3.0.zip'. Reason: Task ':plugins:nf-tower:unzipPlugin' uses this output of task ':plugins:nf-amazon:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-tower-1.3.0.zip'. Reason: Task ':plugins:nf-tower:unzipPlugin' uses this output of task ':plugins:nf-azure:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-tower-1.3.0.zip'. Reason: Task ':plugins:nf-tower:unzipPlugin' uses this output of task ':plugins:nf-console:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-tower-1.3.0.zip'. Reason: Task ':plugins:nf-tower:unzipPlugin' uses this output of task ':plugins:nf-ga4gh:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-tower-1.3.0.zip'. Reason: Task ':plugins:nf-tower:unzipPlugin' uses this output of task ':plugins:nf-google:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-tower-1.3.0.zip'. Reason: Task ':plugins:nf-tower:unzipPlugin' uses this output of task ':plugins:nf-ignite:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-tower-1.3.0.zip'. Reason: Task ':plugins:nf-tower:unzipPlugin' uses this output of task ':plugins:nf-sqldb:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

abhi18av · 2021-12-23T16:42:12Z

Yeah, this is a bit confusing - the error message you've mentioned #2500 (comment) are exactly which prompted me to add the explicit dependencies between gradle tasks #2500 (comment)

Not sure how to proceed here without those explicit task dependencies.

pditommaso · 2022-01-17T20:15:43Z

I've found this possible explanation, but I was not able to make it work

https://discuss.gradle.org/t/gradle-7-0-seems-to-take-an-overzealous-approach-to-inter-task-dependencies/39656

abhi18av · 2022-01-18T06:46:29Z

Thanks for looking into this Paolo!

Since this is dependent on Gradle - do you think we can proceed - for the time being - with the explicit dependency declaration as done previously here #2500 (comment) ?

jorgeaguileraseqera · 2022-02-01T15:55:32Z

After rebasing this branch against master I can't see these warnings and the build process ends successfully

pditommaso · 2022-02-01T16:44:06Z

Oh, gradle dark magic! the warning went away

pditommaso · 2022-02-01T21:12:44Z

Well, it looks like only the proximity of @jorgeaguileraseqera was enough to fix the build. Quite promising! 😄

pditommaso · 2022-02-01T21:17:12Z

@abhi18av do we have a no brain example showing how to configure an Athena datasource and run a query?

abhi18av · 2022-02-02T05:27:36Z

Thanks guys for taking a look at this!

@pditommaso , for any example regarding Athena, I think the best documentation would be regarding an open dataset since for any private dataset, it is necessary to create a database first, the simplest instructions for this are https://docs.aws.amazon.com/athena/latest/ug/getting-started.html

However, for a public dataset such as SRA, I've relied on the Athena database generated by crawling the SRA Metadata as described here the instructions shared in https://www.ncbi.nlm.nih.gov/sra/docs/sra-athena.

The exact instructions to setup the necessary tables/database on the AWS side are descibed in the official walkthrough https://www.youtube.com/watch?v=_F4FhcDWSJg

Thoughts?

jorgeaguileraseqera · 2022-02-02T14:01:28Z

After some clean and builds, the messages

> Task :plugins:nf-tower:unzipPlugin
Execution optimizations have been disabled for task ':plugins:nf-tower:unzipPlugin' to ensure correctness due to the following reasons:
  - Gradle detected a problem with the following location: '/Users/pditommaso/Projects/nextflow/build/plugins/nf-tower-1.3.0.zip'. Reason: Task ':plugins:nf-tower:unzipPlugin' uses this output of task ':plugins:nf-amazon:copyPluginZip' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.3.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.

appears again.

Problem comes because we are using a task (unzip) in different subproject I'll create a fix for it

abhi18av commented Dec 19, 2021

View reviewed changes

plugins/nf-sqldb/build.gradle Outdated Show resolved Hide resolved

abhi18av commented Dec 19, 2021

View reviewed changes

pditommaso reviewed Dec 20, 2021

View reviewed changes

plugins/nf-sqldb/build.gradle Outdated Show resolved Hide resolved

pditommaso reviewed Dec 20, 2021

View reviewed changes

plugins/nf-sqldb/build.gradle Outdated Show resolved Hide resolved

pditommaso force-pushed the abhinav/nfsqldb-awsathena branch from 122731d to 487a50f Compare December 21, 2021 18:37

pditommaso force-pushed the master branch from 7882406 to 0890cee Compare January 8, 2022 18:53

abhi18av self-assigned this Jan 18, 2022

abhi18av added the plugin/sql label Jan 18, 2022

pditommaso force-pushed the master branch from eeb9ac3 to e1de8e0 Compare January 28, 2022 17:34

pditommaso assigned jorgeaguileraseqera Feb 1, 2022

abhi18av and others added 6 commits February 1, 2022 17:02

Add the local AWS Athena JDBC file

49fdf39

Add gradle tasks for downloading the Athena JDBC driver

aa61418

Update the driver - use the AWS included SDK

d84e602

Simplify build for Athena support

b5c7325

Fix Athena download tasks order

0ea5f8c

Add athena tasks

4b078ae

jorgeaguileraseqera force-pushed the abhinav/nfsqldb-awsathena branch from 24046a5 to 4b078ae Compare February 1, 2022 16:02

pditommaso merged commit 462575f into master Feb 1, 2022

pditommaso deleted the abhinav/nfsqldb-awsathena branch February 1, 2022 21:16

jorgeaguileraseqera mentioned this pull request Feb 3, 2022

remove gradle logs about tasks dependencies #2617

Closed

bentsherman added plugins and removed plugins/nf-sqldb labels Aug 14, 2023

Add the AWS Athena JDBC driver for nf-sqldb plugin #2500

Add the AWS Athena JDBC driver for nf-sqldb plugin #2500

Uh oh!

Conversation

abhi18av commented Dec 17, 2021

Open database used for testing

Instructions to use the Athena JDBC integration

Concern (Log4J)

Uh oh!

pditommaso commented Dec 17, 2021

Uh oh!

abhi18av commented Dec 17, 2021

Uh oh!

pditommaso commented Dec 17, 2021

Uh oh!

abhi18av commented Dec 17, 2021

Uh oh!

Uh oh!

abhi18av Dec 19, 2021

Choose a reason for hiding this comment

Uh oh!

abhi18av Dec 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pditommaso Dec 20, 2021

Choose a reason for hiding this comment

Uh oh!

abhi18av Dec 21, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pditommaso commented Dec 21, 2021

Uh oh!

pditommaso commented Dec 21, 2021

Uh oh!

pditommaso commented Dec 21, 2021

Uh oh!

abhi18av commented Dec 23, 2021

Uh oh!

pditommaso commented Jan 17, 2022

Uh oh!

abhi18av commented Jan 18, 2022

Uh oh!

jorgeaguileraseqera commented Feb 1, 2022

Uh oh!

pditommaso commented Feb 1, 2022

Uh oh!

pditommaso commented Feb 1, 2022

Uh oh!

pditommaso commented Feb 1, 2022

Uh oh!

abhi18av commented Feb 2, 2022

Uh oh!

jorgeaguileraseqera commented Feb 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Instructions to use the `Athena` JDBC integration

abhi18av Dec 19, 2021 •

edited

Loading