Blog post for the DataFusion `51.0.0` release

[alamb]

Is your feature request related to a problem or challenge?

• Part of [Epic] DataFusion Blogs #14836
• Part of Release: Release DataFusion 51.0.0 (Nov 2025) #17558

Describe the solution you'd like

Let's tell the world what we have done since the DataFusion 50 release

Here is the last blog:

• https://datafusion.apache.org/blog/2025/09/29/datafusion-50.0.0/
• Here is the PR with the post Blog: Add blog post about DataFusion 50.0.0 release datafusion-site#115

We can start with the features listed on the current release ticket (maybe there should be more)

• Release DataFusion 50.0.0 (Aug/Sep 2025) #16799

Describe alternatives you've considered

No response

Additional context

No response

[pepijnve]

Not a very exciting topic, but if you want some info for the blog post on the case work, here are some (unscientifically collected) numbers I got by running some queries on an SF10 lineitems parquet file.

I can make some proper versions of the diagrams from #18075 (comment) if you want some more elaboration on what changed exactly in the implementation. In the end this is just memory shuffling reduction, so maybe "it's a bit faster now" is sufficient.

Just a case expression

select case l_returnflag
  when 'R' then date_part('year', l_shipdate)
  when 'A' then date_part('year', l_commitdate)
  when 'N' then date_part('year',l_receiptdate)
  else null end
from lineitem;

50.0.0: ~0.492 seconds
51.0.0: ~0.278 seconds

*

select * from lineitem

50.0.0: 0.668 seconds
50.1.0: 0.668 seconds

* + a case expression

select *, case l_returnflag
  when 'R' then date_part('year', l_shipdate)
  when 'A' then date_part('year', l_commitdate)
  when 'N' then date_part('year',l_receiptdate)
  else null end
from lineitem;

50.0.0: ~1.287 seconds -> case adds 0,619 seconds
51.0.0: ~0.850 seconds -> case adds 0,182 seconds

[alamb]

Not a very exciting topic, but if you want some info for the blog post on the case work, here are some (unscientifically collected) numbers I got by running some queries on an SF10 lineitems parquet file.

I am excited!

I can make some proper versions of the diagrams from #18075 (comment) if you want some more elaboration on what changed exactly in the implementation. In the end this is just memory shuffling reduction, so maybe "it's a bit faster now" is sufficient.

Thank you for the offer -- unless you already have one or would like to spend time writing up a more full features blog about how to optimize CASE evaluation, I think some unscientific numbers are sufficient

Speaking of a blog, I am not aware of any practical writeup of implementing optimized CASE (conditional) evaluation in a vectorized query engine (aka like DataFusion). I would be willing to help write such a post, but if so I would want to make it a high quality one:

1. background of CASE / conditional
2. Basic evaluation strategy
3. Why it is hard to make CASE fast
4. All the crazy stuff that we did to actually make it fast in DataFusion

[pepijnve]

I've placed a first draft written entirely by Claude AI at apache/datafusion-site#122 to get the ball rolling. It correctly captured the essence of the original issues and the changes the were made, but the post still needs quite some editing.

[alamb]

thanks @pepijnve -- I'll try and review it over the next few days

[Omega359]

PR's in this release (approx based on dates) - https://github.com/apache/datafusion/pulls?q=is%3Apr+merged%3A2025-09-16..2025-11-08

[alamb]

Another source of data is https://github.com/apache/datafusion/blob/branch-51/dev/changelog/51.0.0.md