Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ZEPPELIN-5970][ZEPPELIN-5971] bugs occur when zeppelin.livy.tableWithUTFCharacters is enabled #4675

Merged
merged 6 commits into from
Nov 20, 2023
Merged

[ZEPPELIN-5970][ZEPPELIN-5971] bugs occur when zeppelin.livy.tableWithUTFCharacters is enabled #4675

merged 6 commits into from
Nov 20, 2023

Conversation

levi0090
Copy link
Contributor

@levi0090 levi0090 commented Oct 12, 2023 •
edited
Loading

What is this PR for?

Fixed 2 bugs occur when zeppelin.livy.tableWithUTFCharacters is set to True. ref. commit ID 0fc1d7b,

The errors occur when:

  1. tableWithUTFCharacters is set to True and INSERT/CREATE statement is executed.
  2. tableWithUTFCharacters is set to True and SELECT query of a table that has non-primitive data (e.g. arrays).

What type of PR is it?

[MINOR] Bug Fix - JIRA-ticket ZEPPELIN-5970
[MINOR] Bug Fix - JIRA-ticket ZEPPELIN-5971

What is the Jira issue?

How should this be tested?

  1. Enable tableWithUTFCharacters feature from livy interpreter.
  2. Create non-select SQL statement, such as: CREATE table statement , you will receive an error, however, the table will be created !
%livy.scala
spark.sql("CREATE TABLE  test ( id int, name String, destination []String)")
  1. INSERT a record with an array of values, will raise an exception as arrays are non-primitive data
%livy.scala
spark.sql("INSERT INTO test VALUES ('6', 'test', '[\'1'\,\'2\']')")

Screenshots (if appropriate)

N/A

Questions:

  • Does the license files need to update? No
  • Is there breaking changes for older versions? No
  • Does this needs documentation? No

@jongyoul
Copy link
Member

@levi0090 Thank you for your contribution. I checked the code quickly and looks reasonable. BTW, could you please change the title of the PR including a ticket or the prefix [MINOR]?

@pan3793
Copy link
Member

pan3793 commented Oct 12, 2023

@levi0090 I'm not sure the rules of Apache Zeppelin community, but generally, the bug fix requires a JIRA ticket

@pan3793
Copy link
Member

pan3793 commented Oct 12, 2023

@levi0090 would you mind providing the error stacktrace/message of the case you listed before applying this patch in the PR description? Such messages could help users who meet the same issue to identify the root cause.

@levi0090
Copy link
Contributor Author

levi0090 commented Oct 15, 2023 •
edited
Loading

@pan3793 I have created two Jira issues that describe both issues, for your reference.

@jongyoul
Copy link
Member

@levi0090 Thank you for making tickets. Could you please add your ticket numbers in your PR title?

Reamer
Reamer requested changes Oct 17, 2023
View reviewed changes
Copy link
Contributor

@Reamer Reamer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only one performance issue found. The rest looks good.

livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java Outdated Show resolved Hide resolved
@levi0090 levi0090 changed the title Fix Fix [ZEPPELIN-5970-5971] Oct 19, 2023
@jongyoul jongyoul changed the title Fix [ZEPPELIN-5970-5971] [ZEPPELIN-5970][ZEPPELIN-5971] ... Oct 19, 2023
@jongyoul
Copy link
Member

@levi0090 I've added ticket prefixes. Could you please edit the title summarizing your contribution?

@levi0090 levi0090 changed the title [ZEPPELIN-5970][ZEPPELIN-5971] ... [ZEPPELIN-5970][ZEPPELIN-5971] bugs occur when zeppelin.livy.tableWithUTFCharacters is enabled Oct 19, 2023
Reamer
Reamer previously approved these changes Oct 19, 2023
View reviewed changes
@levi0090 levi0090 closed this Oct 19, 2023
@levi0090 levi0090 reopened this Oct 19, 2023
jongyoul
jongyoul previously approved these changes Nov 4, 2023
View reviewed changes
@Reamer Reamer dismissed stale reviews from jongyoul and themself via 0b69180 November 17, 2023 10:26
@Reamer
Copy link
Contributor

Reamer commented Nov 17, 2023

I will merge the changes with the master on 20.11 if no further comments are received.

@Reamer Reamer merged commit c9c3437 into apache:master Nov 20, 2023
30 checks passed
akoira pushed a commit to akoira/zeppelin that referenced this pull request Feb 1, 2024
@levi0090 @Reamer
[ZEPPELIN-5970][ZEPPELIN-5971] bugs occur when zeppelin.livy.tableWit…
ae7fe23 
…hUTFCharacters is enabled (apache#4675)

* fix error occurs when non-select SQL statement is executed and tableWithUTFCharacter is True

* fix error occurs when non-primitive type of data is within json response such as array  and tableWithUTFCharacter is True

* rename custom function

* fix bug and improve the performance to iterate over the entrySet.

* remove nit

---------

Co-authored-by: Philipp Dallig <philipp.dallig@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pan3793 pan3793 pan3793 approved these changes

@Reamer Reamer Reamer approved these changes

@jongyoul jongyoul jongyoul left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@levi0090 @jongyoul @pan3793 @Reamer