Automate dependency updates

[alamb]

As suggested by @returnString on #33 (comment)

rationale

Keep dependencies up to date, with as little manual intervention as possile

Proposed Changes

Add some sort of automation / bot that automatically updates dependencies. @returnString suggests that dependable is a good potential choice

https://docs.github.com/en/code-security/supply-chain-security/configuring-dependabot-security-updates

[returnString]

A few q's from my side:

Do we have a strict policy around dependency version bumps and their potential impact on dependent crates? For example, say Tokio releases a theoretical 2.0 which doesn't play nicely with the 1.x runtime (similar to the 0.2 => 1.0 transition), would we be ok upgrading to v2 and requiring that DataFusion users do the same? Would that require a major version bump for DataFusion?

Also, as a library crate, DataFusion doesn't include a lockfile - do we want to just raise the minimum supported version of dependencies where appropriate?

[returnString]

For science, I've just enabled this on my fork (enabling commit here) and it's only picked up one dep that needs updating, rustyline - here's the example PR, you can see how the bot interaction works too where I asked for a rebase: reservoirdb#1

edit: GHA builds are broken with macos provisioning failures, so ignore the CI failures

[alamb]

Do we have a strict policy around dependency version bumps and their potential impact on dependent crates?

I do not think we have any strict (or written) policy about this

For example, say Tokio releases a theoretical 2.0 which doesn't play nicely with the 1.x runtime (similar to the 0.2 => 1.0 transition), would we be ok upgrading to v2 and requiring that DataFusion users do the same?

This is what we did when tokio went to 1.0

Would that require a major version bump for DataFusion?

It seems like a good idea :)

Prior to 1 week ago the datafusion major release numbers were tied to the arrow major release numbers, so we didn't have the flexibility to implement such a policy.

Defining one seems like a good idea

[alamb]

@returnString reservoirdb#1 looks pretty cool to me

[martinitus]

Just stumbled over this when I saw, that there is arrow-rs 12.x.y but datafusion still depends on arrow-rs 9.1.

I think general best practice (given you use semver) is that if you have a major increase in a dependency that appears in your public API, then you also need a major increase in your library version, as the stuff that appear in your public API may break backwards compatibility for the users of your API.

[xudong963]

Just stumbled over this when I saw, that there is arrow-rs 12.x.y but datafusion still depends on arrow-rs 9.1.

I think datafusion always uses the newest arrow-rs, which is done by @alamb.

[xudong963]

Btw, the issue had solved!

[alamb]

Just stumbled over this when I saw, that there is arrow-rs 12.x.y but datafusion still depends on arrow-rs 9.1.

this is true for the version of datafusion we released to crates.io https://crates.io/crates/datafusion/7.1.0/dependencies

I filed #2327 to track the idea of doing more frequent releases. thank you for raising it @martinitus