Compile against spark 4.0.0 #146
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…version Update sbt to the latest version. Unlocks use of newer jvm versions.
Feature/codeowners
Codeowners file was not created correctly
…e-executions-in-a-pull-request Avoid double executions for each spark version when a pull request is…
…olored-diff Add color diff for small dataframe comparison
alfonsorr
force-pushed
the
feature/test-agains-spark-4.0.0
branch
from
6966dfd to
9d1e260
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is not a pull request that can be merged into the main branch. It uses the preview version of spark 4.0.0.
If you want to test this branch on your machine, use the latest version in this repository: https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-sql_2.13/4.0.0-SNAPSHOT/
And use it as the parameter to compile when you start your SBT repl, for example:
-Dspark.testVersion=4.0.0-20240831.001610-367
Also, spark 4.0 is not compatible with Java 8, only compatible with Java 17 and 21. I would recommend using 17 because it is also compatible with the latest versions of spark 3.x
At the moment of creation there are issues in a couple of tests that need fixing. One of them related to extracting a local date that could be a spark bug.