Release 3.2.0

Release notes - Feature Store - 3.2.0

Epic

FSTORE-615 HSFS JavaDoc

Bug

FSTORE-22 Failed to insert data to streaming FG

FSTORE-313 Throw intelligible error message when calling save on unsupported external feature groups in python engine

FSTORE-456 Validation time in the future

FSTORE-516 the 'transformationFunction" fields empty in feature_view.json()

FSTORE-540 Fix shared fs integration tests

FSTORE-549 Uploading a large number of rows to onlinefs fails with TOPIC_AUTHORIZATION_FAILED

FSTORE-566 Feature store online metrics page returns dashboard not found

FSTORE-602 get_or_create_feature_group does not validate feature group name

FSTORE-607 Investigate hsfs java resource leaks

FSTORE-636 ExpectationSuite sonarkube smartUpdate transaction

FSTORE-639 Pin SQLalchemy version or upgrade to new 2.0.0 version

FSTORE-653 get_feature_vector is broken

FSTORE-660 Docs: Syntax issue for creating external feature group

FSTORE-661 Creating external feature group in Jupyter Notebook didn't work

FSTORE-682 Fix fg save_expectation_suite

FSTORE-689 Shallow dataframe copy hides TZ bug

FSTORE-693 Insert with validation suite throws date format exception

FSTORE-701 Fix loadtest datavalidation

FSTORE-730 Fix Multiple backfill job started in dataval loadtest

FSTORE-760 Fix the printing of log in the stack trace for loadtest jobs

FSTORE-763 Onlinefs handling avro exception after append

FSTORE-776 Floating Point Issue with Train Test Split Ratios

FSTORE-780 Documentation doens't mention type limitation for event_time

FSTORE-781 Documentation for extenral feature groups still refers to them as on-demand feature groups

FSTORE-794 Offline fg backfill fails due to the DelataStreamer config Class path(s) in HudiEngine.java are not set correctly

FSTORE-804 Resolve OnlineFS getAndCastValue NullPointerException

FSTORE-812 Enable Kafka topic auto creation

FSTORE-814 StackOverflowException if you provide statistics columns in the statistics config

Design Log

FSTORE-559 Test connection feature for storage connectors

FSTORE-572 Design Log: HSFS Javadocs API

FSTORE-609 Design Log for FM Extend Alerts API

FSTORE-610 Design Log for FM UI / UX

Task

FSTORE-41 From the storage connector; there is no way to know if you are actually connected

FSTORE-44 From a notebook the connector does not say either if it worked or failed

FSTORE-343 Separate HSFS java client build for different Frameworks

FSTORE-349 Add advanced tutorials to documentation

FSTORE-353 Extend the README.md files for the electricity prices and NYC Taxi tutorials

FSTORE-373 Remove job when a fg or fv is deleted.

FSTORE-405 Add entity type to fs job name

FSTORE-434 Improve Kafka message delivery callback error handling

FSTORE-483 Hiveserver2 query optimization config

FSTORE-487 Handle each feature group in its own retry loop

FSTORE-490 Add more assert to Feature Store load test

FSTORE-491 Backfilling user story Design Document

FSTORE-492 Change notebooks that use sklearn models to XGBoost and save them as JSON files, instead of as pickled python files

FSTORE-493 Add load test for online feature store to test data integrity and volume

FSTORE-495 CItibike Tutorial

FSTORE-506 update hopsworks tutorials for 3.1 APIs

FSTORE-528 Add single expectation API and 3.1 changes to data validation tutorials

FSTORE-546 Update several Tutorials to 3.1 hopsworks API

FSTORE-547 Update Tutorials to 3.1

FSTORE-554 cleanup training dataset statistics

FSTORE-557 Simplify getRowClass method

FSTORE-558 Attach transformation function to self join query

FSTORE-561 Add load test for concurrent creation and insertion of online feature groups

FSTORE-562 Add load test to integrate benchmark for vector lookup

FSTORE-568 Add meaningful exceptions to the tutorials, that use API keys

FSTORE-569 New version of Bitcoin Tutorial

FSTORE-570 Design log: Kafka Authoriser Update

FSTORE-571 3.0 Some tutorials Fix

FSTORE-573 documentation/guide about on-demand feature computation pipeline

FSTORE-575 write streaming feature computation pipeline example

FSTORE-581 Review feature view documentation

FSTORE-588 Kafka Authoriser Update

FSTORE-589 Document behaviour of get_feature_vector(s) for missing PK values

FSTORE-591 Fix for hops-kafka-authorizer Sonarqube issues

FSTORE-597 Fix for online-fs Sonarqube issues

FSTORE-611 Investigate how to visualise Feature Monitoring

FSTORE-613 Make Kafka Topic partition default configurable for APP upgrade

FSTORE-614 Auto Kafka Topic recreation needs to provide better error message in Hopsworks for APP upgrade

FSTORE-617 Fix java docstrings to build javadocs

FSTORE-618 Expand current java docstrings to cover new hsfs functionalities

FSTORE-619 make the changes to the github action, pom.xml and mkdocs.yml to publish javadocs

FSTORE-621 Make unit tests run on hsfs branch-3.1

FSTORE-623 Flying Duck: Fix integration into deployment process

FSTORE-624 Add examples from hops-examples repo to hopsworks-tutorials repo

FSTORE-627 Create dockerfile and python entrypoint script-snowflake connector

FSTORE-628 Hopsworks backend logic, bash script to start the container and REST API-snowflake

FSTORE-629 Add CI/CD, set up Jenkins to deploy docker image to docker registry and pull on new clusters-snowflake

FSTORE-658 Incomplete doc string of create_external_feature_group

FSTORE-666 Fix modeling notebook in Bitcoin Price Prediction tutorial

FSTORE-667 Rerun and Refactor all tutorials

FSTORE-669 Enable GCS and BigQuery storage connector feature flags by default

FSTORE-670 Add provenance workflow tests

FSTORE-678 Flying Duck - Logging / Monitoring Configuration in Server

FSTORE-679 Upgrade Air Quality Tutorial

FSTORE-680 Optimize HSFS for single row writes

FSTORE-681 Fix blockers in hopsworks noticed by sonarqube

FSTORE-685 Improvments to onlinefs

FSTORE-687 Pin numpy in nightly-build

FSTORE-694 Users should have control over the size of the connection pool for online feature view

FSTORE-695 Flying Duck - Integrate Monitoring in Server and Chef

FSTORE-697 Handle timezones in validation report timestamp

FSTORE-698 Write Hudi performance best practices guide

FSTORE-704 Flying Duck - Hudi Table Reader Implementation

FSTORE-706 HSFS java make FeatureGroupBase into abstract class

FSTORE-708 Flying duck: Logging

FSTORE-714 Bump onlinefs version

FSTORE-724 Separate workflow and load tests

FSTORE-726 when computing pandas statistics check if mean is numeric type

FSTORE-735 Kafka authorizer Sonarqube fix

FSTORE-739 Add batch scoring examples to all non-online tutorials

FSTORE-741 Resolve outstanding onlinefs sonarqube issues

FSTORE-743 Add online support for External Feature Groups

FSTORE-757 Investigate Describe requests on all kafka topics by users in workflow tests

FSTORE-758 Fix failed workflow test (failed to serialise timestamp)

FSTORE-759 Relax fastavro requirements in hsfs

FSTORE-762 Cherrypick FSTORE-467 to hopsworks 3.0

FSTORE-765 Simplify initialization of OnlineFS

FSTORE-772 BigQuery ExternalFG tutorial

FSTORE-774 Move gcsfs constraint to docker image

FSTORE-775 Improve OnlineFS markdown file

FSTORE-777 Remove MapValue from onlinefs

FSTORE-782 Allow resetting kafka offsets for stream feature groups after backup/restore

FSTORE-786 Pin PyMysql version in hsfs to < 1.0.3

FSTORE-787 FraudBatch Fix

FSTORE-788 FraudBatch Fix for dev branch

FSTORE-789 multi modular hsfs java client

FSTORE-795 Provide flink dependencies in Hopsowrks deployment and not in hsfs.

FSTORE-805 Bump OnlineFS version to 1.5

FSTORE-811 Bump python version in testconnector image

FSTORE-818 Flying Duck - HSFS Cloud Integration

FSTORE-820 FlyingDuck - Memory-Aware Result Streaming

FSTORE-822 Flying Duck - HSFS External Client Integration

FSTORE-831 Upgrade citibikes tutorials

Release notes - Hopsworks - 3.2.0

Bug

HWORKS-29 Remove "Logical Clocks" from the feed back page

HWORKS-315 Copy and move get interrupted if run for too long

HWORKS-320 Pypi library versions not sorted correctly

HWORKS-360 Upgrade org.owasp.dependency-check-maven to 7.4.4

HWORKS-368 Fix serving tests

HWORKS-372 explicit provenance missing community stereotype

HWORKS-379 LDAP users cannot download certificates

HWORKS-380 PKI.java class contains resource leaks

HWORKS-381 hopsworks-api connection.close() should clean up hsfs and hsml connections

HWORKS-382 Provenance input links are missing for models trained on python kernels

HWORKS-390 app_id field in execution table is too short

HWORKS-407 Setting the maximum number of Projects per user to 0 let the user create unlimited Projects

HWORKS-410 Pin jupyter-console in Sparkmagic

HWORKS-414 dockerfile:experimental should be pushed to our docker registry

HWORKS-425 Update Grafana to address CVE-2022-39328, CVE-2022-39307 and CVE-2022-39306

HWORKS-431 Git image version shoud not be hardcoded in Hopsworks settings

HWORKS-444 Log aggregation reports incorrect failed status under load

HWORKS-445 Increase polling time for log aggregation to finish in execution.wait_until_finished()

HWORKS-481 hopsworks-api create_job fails on app_path missing for flink jobs

HWORKS-483 Add CREATING status for deployments

HWORKS-484 Consecutive python uninstall commands uses incorrect tag

HWORKS-507 kagent_param for kube hopsmon certificate break the deployment

HWORKS-524 HSML ignores knative_domain parameter

Subtask

HWORKS-370 Upgrade TensorFlow to 2.11.0

HWORKS-373 Bump ray[serve] dependency in kserve package

HWORKS-400 Upgrade Jupyter(Lab) to address CVE-2021-32797

HWORKS-430 OnlineFS CVEs

Task

HWORKS-126 Add idempotent methods for creating deployments

HWORKS-219 Fix Proxy Servlet to use consul instead of localhost

HWORKS-220 Fix zip/unzip for clustered payara

HWORKS-253 Refactor loadtest project creation

HWORKS-313 Include model.deploy() method in the docs

HWORKS-333 Fix container vulnerabilities

HWORKS-352 Make model input_example consistent across data types

HWORKS-361 Increase TransactionInactiveTimeout to 30000

HWORKS-362 Update spark to depend on latest version of hopsfs

HWORKS-363 Bump Hops version to 3.2.0.9-SNAPSHOT on hopsworks master

HWORKS-364 Update flatbuffers dependency in Hive to address CVE-2020-36632

HWORKS-366 Remove unused ClusterUtili price multiplicator singleton

HWORKS-367 create zip/unzip job

HWORKS-378 Metrics for consul nodes

HWORKS-391 Add utility function to create directory using hopsworks python SDK

HWORKS-394 Remove cron job to copy logs to HopsFS

HWORKS-395 Remove admin UI

HWORKS-398 OpenSearch Dashboard - elastic cluster status is wrong

HWORKS-399 Improve project admin rest api

HWORKS-402 Add monitoring rules

HWORKS-403 Parallelize workflow and load tests

HWORKS-405 Installing library using hopsworks python SDK should wait for ongoing environment sync operation

HWORKS-411 CVE-2023-25194 - Kafka Connect Worker Vulnerability

HWORKS-416 Upgrade data science libraries in Project environment

HWORKS-420 Remove links from images

HWORKS-423 Fix links for public slack on docs and community edition

HWORKS-435 Enable ACL for Zookeeper

HWORKS-436 Helper for loadtest to show execution logs in case of failure

HWORKS-441 Bump RonDB version to 21.04.10

HWORKS-442 Add log aggregation load test

HWORKS-443 Disable Hiveserver2 Web UI

HWORKS-451 Upgrade payara to 5.2022.5 to fix CVE-2022-42920

HWORKS-455 Remove old tours

HWORKS-457 Create users_groups view outside flyway

HWORKS-462 Remove Gemfile.lock file from community tests

HWORKS-463 Bump activerecord dependency in hopsworks-IT to 6.0.5.1

HWORKS-464 Update Python job documentation to reflect the fact that a K8s cluster is needed

HWORKS-467 Disable SSL/TLS Diffie-Hellman keys less that 2048 bits

HWORKS-470 Bump NVIDIA driver version to support CUDA 11.8

HWORKS-471 Remove usage of docker/dockerfile:experimental

HWORKS-472 Workflows/load tests python dependencies should be defined in a requirements.txt

HWORKS-474 Remove hdfs_user foreign key from JupyterProject

HWORKS-476 Remove hdfs_user_id foreign key from TensorBoard

HWORKS-478 Add webhook configuration support for alertmanager

HWORKS-480 Remove inode foreign key from storage connectors

HWORKS-485 Do not create linux user/group in git container

HWORKS-488 Add support for storing docker registry artifacts on S3

HWORKS-497 Remove homebew dependency from java cookbook

HWORKS-502 Add silence for CVE-2023-28462

HWORKS-505 Upgrade GPU components to support Cgroups v2/Rhel8/Ubuntu22.04

HWORKS-506 Bump Hops version to 3.2.0.9-RC0

HWORKS-513 Add MLOps Dictionary link to documentation

HWORKS-514 loadtest should run in its own anaconda environment