.NET for Apache Spark v2.4.0-rc1

[SparkSnail]

We are excited to announce the prerelease of .NET for Apache Spark v2.4.0-rc1, introducing Apache Spark 4.0.0–4.0.4 integration! Thank you for trying it out, and we look forward to your feedback!

Please see the Release Notes for new features, breaking changes, compatibility requirements, and known issues.

This release candidate is intended for evaluation and feedback. Use the Microsoft.Spark NuGet package, bridge JAR, and Worker from this release together.

Download Microsoft.Spark.Worker:

Platform	.NET 8	.NET Framework 4.8
Linux	x64 (tar.gz | zip)	N/A
Windows	x64 (zip)	x64 (zip)
macOS	x64 (zip)	N/A

Spark 4 validation covers Windows and Linux x64 with .NET 8 and JDK 17. Spark 4 compatibility with macOS or .NET Framework has not been established.

---

This discussion was created from the release .NET for Apache Spark v2.4.0-rc1.

[mungojam]

Any chance of a blog post to show some examples of the new features?

[SparkSnail]

Any chance of a blog post to show some examples of the new features?

Thanks for the suggestion, @mungojam! A hands-on walkthrough covering Spark 4 setup and practical C# examples would be a useful follow-up to this release.
In the meantime, the [Variant guide] includes C# examples for working with semi-structured JSON through Spark SQL, and the [setup and migration guide explains how to try the 2.4.0-rc1 version.
Are there particular scenarios you’d like to see covered? For example, .NET UDFs, Structured Streaming, or Variant/JSON processing?

[mungojam]

I was intrigued by the UDF mention. To be honest I haven't used this library yet, but I think it needs some more marketing if it is to have traction against python. I'm keen to see how the benefits of C# can be utilised in this domain to add more type safety