Skip to content

Releases: logicalclocks/hopsworks

v2.2.0

09 Apr 17:05
85a5192
Compare
Choose a tag to compare

Release notes - Hopsworks - Version 2.2.0

Bug

HOPSWORKS-2482 Read Hudi Feature Group from Hive Engine fails with unknown fg0

HOPSWORKS-2481 Only default deployments must be enabled when KFServing is not installed

HOPSWORKS-2473 dataset read acl list does not get cleared on dataset delete

HOPSWORKS-2471 fix prepared statement parameter for string PKs

HOPSWORKS-2447 Pip conflicts modal not closing when clicking confirm

HOPSWORKS-2415 reset elastic client in case it uses the wrong certificate

HOPSWORKS-2414 Verify that API token used when starting Jupyter with AutoPushOnShutdown enabled has write access to git repo

HOPSWORKS-2352 Snowflake connector requires pytz <= 2021.0

HOPSWORKS-2303 Change python pip search to use pypi simple index and JSON API

HOPSWORKS-2293 Fix test for kvm

Improvement

HOPSWORKS-2484 Fix SSL/TLS settings for Glassfish network listeners

HOPSWORKS-2460 Create admin end-point for user management

HOPSWORKS-2433 Azure Root CA missing from glassfish trust store.

HOPSWORKS-2432 TLS for Resource Manager UI

HOPSWORKS-2423 Advertise Rondb managment node through consul

HOPSWORKS-2422 Advertise Kafka brokers through Consul

HOPSWORKS-2421 [ndb-chef] Fix config.ini for RonDB

HOPSWORKS-2408 Feature store activity should return validation results

HOPSWORKS-2405 Upgrade Python version we install in recipes

HOPSWORKS-2404 remove metadata designer code

HOPSWORKS-2396 Add AzureML Studio notebook

HOPSWORKS-2368 Do not require API token for working with public GitHub and Gitlab repos using Jupyter

HOPSWORKS-2367 Hopsworks Connector Docs

HOPSWORKS-2366 Replace service with systemctl in recipes

HOPSWORKS-2351 derived fg provenance tests - links/usage

HOPSWORKS-2345 Get Kafka Broker Endpoints through REST Api

HOPSWORKS-2336 It should not be mandatory to provide a version when installing a pip or conda library

HOPSWORKS-2314 Snowflake connector

HOPSWORKS-2247 Replace feature store job tracking with provenance

HOPSWORKS-2139 Use Hudi commit ID as statistics timestamp

HOPSWORKS-2071 Data validation v2

HOPSWORKS-2052 Save preferences for dataset and experiments filter

New Feature

HOPSWORKS-2474 Docker job

HOPSWORKS-2458 Online-FS Service Integration

HOPSWORKS-2399 Add new frontend deployment

HOPSWORKS-2384 Add support for coalesce in training dataset

HOPSWORKS-2195 Attach Jupyter configuration to notebook as Xattr

HOPSWORKS-1978 schematized tags

Sub-task

HOPSWORKS-2193 Add support for request batching

HOPSWORKS-2185 Implement inference controller

HOPSWORKS-2106 Implement storage initializer webhook for KFServing

Task

HOPSWORKS-2476 Bump hops dependencies to 3.2.0.3-RC0

HOPSWORKS-2348 Disable storage and compute quota by default

HOPSWORKS-2342 Add PyTorch Maggy hyperparameter optimization example notebook

HOPSWORKS-2326 Display alert if a more recent version of hops or hsfs exists in python service

HOPSWORKS-2317 Remove featurestore module from hops-util-py and hops-util

HOPSWORKS-2305 Remove security questions

Release 2.1.0

12 Feb 09:23
8f64925
Compare
Choose a tag to compare

Release notes - Hopsworks - Version 2.1.0

Bug

HOPSWORKS-2359 Always show GPU configuration fields for Jobs and Jupyter
HOPSWORKS-2358 Remove ROCM GPU monitoring code
HOPSWORKS-2346 Identifying if a project is using the base image on ECR does not work
HOPSWORKS-2338 handle empty tfrecord files in TFDataEngine
HOPSWORKS-2333 create_training_dataset method throws Connector not found exception
HOPSWORKS-2311 Template required ec2-init parameters for the unmanaged version
HOPSWORKS-2307 Docker garbage collection should not remove base or python37 images for any hopsworks version
HOPSWORKS-2302 Hide python pip search bar as underlying pip search is now deprecated
HOPSWORKS-2294 Installing a python library from git should enclose the url in quotes
HOPSWORKS-2283 Reading from the online feature store doesn't work on Databricks
HOPSWORKS-2249 provenance elasticsearch - count and aggregations should be separate methods
HOPSWORKS-2248 Python file path is not shown in Job details
HOPSWORKS-2238 Feature Group schema primary key info not set for partition keys
HOPSWORKS-2219 Remove NaN from statistics
HOPSWORKS-2202 Docker registry caches deleted manifests
HOPSWORKS-2199 Glassfish consul domain should be included in Kafka super.users
HOPSWORKS-2198 Cannot get the schema of a shared kafka topic
HOPSWORKS-2187 Add anaconda user to hadoop group only if the group exists. Create hops-system Anaconda env conditionally
HOPSWORKS-2182 Disable unattended-upgrades in Ubuntu
HOPSWORKS-2174 issues attaching xattr to shared files
HOPSWORKS-2164 Add API Key annotation on the commits endpoint
HOPSWORKS-2159 ruby tests fix
HOPSWORKS-2158 Query constructor ignores more than 2 joining conditions
HOPSWORKS-2151 Tag validation doesn't work for key only tags
HOPSWORKS-2145 Flask (sklearn) race condition when reading pid file
HOPSWORKS-2102 Conversion of Jupyter notebooks to Python programs should consider notebook kernel
HOPSWORKS-1984 Exception when creating online feature group without primary key

Task

HOPSWORKS-2348 Disable storage and compute quota by default
HOPSWORKS-2337 Upgrade hudi to 0.7.0 version
HOPSWORKS-2327 Ui fixed for generating training datasets
HOPSWORKS-2324 Display alert if project is using older docker image
HOPSWORKS-2320 Set PARQUET as default for non-hudi offline feature groups
HOPSWORKS-2287 Bump up docker.io version for ubuntu
HOPSWORKS-2275 Remove InodeEncodingStatus Bean
HOPSWORKS-2274 Feature store "NOT FOUND" errors should return 404
HOPSWORKS-2241 Consul security recipe should not try to get Hopsworks IP if not needed
HOPSWORKS-2206 HSFS profile to install with and without Hive dependencies
HOPSWORKS-2196 Remove hudiEnabled flag and sync endpoint from hopsworks
HOPSWORKS-2189 Update spark-avro package for to_avro/from_avro methods
HOPSWORKS-2188 Remove partition key requirements for HUDI feature groups and allow users to set precombine keys
HOPSWORKS-2178 Remove leftover ldconfig in hops::install
HOPSWORKS-2163 Add missing chef changes of HOPSWORKS-2057 in master
HOPSWORKS-2144 Bump hops dependencies to 3.2.0.2-RC0
HOPSWORKS-2138 remove slider and LLAP
HOPSWORKS-2132 Check that base environment does not contain broken requirements when building docker
HOPSWORKS-2131 Add pip check command to Python service
HOPSWORKS-2127 Bump TensorFlow to 2.4 and PyTorch to 1.7
HOPSWORKS-2103 Add documentation for connecting to the Hopsworks Kafka cluster from an external client

Improvement

HOPSWORKS-2341 allow hsfs to get all metadata from feature store within experiment function
HOPSWORKS-2315 [hops-hadoop-chef] Template in Chef application key size
HOPSWORKS-2280 Save statistics configuration also for training datasets
HOPSWORKS-2277 Add VARBINARY(N) to proposed types for online features
HOPSWORKS-2201 Move to delete API for removing images from local docker registry
HOPSWORKS-2200 set fs.azure.skipUserGroupMetadataDuringInitialization to true for ADLSv2
HOPSWORKS-2183 Add feature group, training dataset, storage connector counters to FeaturestoreDTO summary
HOPSWORKS-2180 Disable transactions when Spark writes to the online feature store
HOPSWORKS-2177 extend on-demand feature groups parquet/orc/delta/hudi/csv tables
HOPSWORKS-2176 Add API key annotations to provenance endpoint
HOPSWORKS-2172 Reduce default varchar size for string features
HOPSWORKS-2168 Add keyword support to feature groups and training datasets
HOPSWORKS-2157 Store information about primary and precombine keys for time travel enabled feature groups separately from Hive Metastore
HOPSWORKS-2143 experiments/models list for any user with access (including shared)
HOPSWORKS-2129 Should run conda clean after installing a package
HOPSWORKS-2120 enable full provenance by default
HOPSWORKS-2119 Create an admin rest endpoint for user registration
HOPSWORKS-2101 Add resource to download files using api key
HOPSWORKS-2026 Add "converting notebook" job state
HOPSWORKS-1817 Replace influxdb with prometheus + pushgateway for Spark applications
HOPSWORKS-1591 UI improvements for storage connectors
HOPSWORKS-1193 Make deleting a project more intuitive
HOPSWORKS-288 Allow editing a job's configuration property without clearing the rest

New Feature

HOPSWORKS-2197 Add ADLS storage connector for the feature store
HOPSWORKS-2184 Install Python libraries from Anaconda environment file
HOPSWORKS-2165 Sup...

Read more

Release 2.0.0

01 Feb 09:23
d42991c
Compare
Choose a tag to compare

Release notes - Hopsworks - Version 2.0.0

Task

HOPSWORKS-2287 Bump up docker.io version for ubuntu
HOPSWORKS-2186 Remove installation of slack operator in Airflow cookbook
HOPSWORKS-2161 Option to disable just the Registration UI
HOPSWORKS-2156 Pin all TensorFlow projects in Docker image
HOPSWORKS-2126 Cleanup old hops-util(-py) feature store notebooks
HOPSWORKS-2125 Bump hops dependencies to 3.2.0.1-RC0
HOPSWORKS-2109 Remove base_env from PythonDep
HOPSWORKS-2094 Setup GPU monitoring without enabling it even when it's not a GPU deployment
HOPSWORKS-2090 Bump up docker.io version for Ubuntu
HOPSWORKS-2072 [hops-hadoop-chef] Recipe should check if Hops distro has been build with correct profiles
HOPSWORKS-2043 Upgrade base and airflow conda environment to python 3.7
HOPSWORKS-2042 Unmanaged Hopsworks Image
HOPSWORKS-2041 Bump Hops Hadoop version to 3.2.0.1-SNAPSHOT
HOPSWORKS-2040 Docker registry does not need to be versioned
HOPSWORKS-2039 Bump hops dependencies to 3.2.0.0-RC3
HOPSWORKS-2037 Rename map_fun and wrapper_fun to train_fn
HOPSWORKS-1945 Store AWS access/secret key in the secret store.
HOPSWORKS-1944 Split feature store tests in several categories
HOPSWORKS-1387 Upgrade to NDB 8

Bug

HOPSWORKS-2202 Docker registry caches deleted manifests
HOPSWORKS-2182 Disable unattended-upgrades in Ubuntu
HOPSWORKS-2162 Airflow page does not open
HOPSWORKS-2151 Tag validation doesn't work for key only tags
HOPSWORKS-2150 S3 storage connector should not bother about secrets if iam role is configured
HOPSWORKS-2149 Disable generating and sharing ssh key freely
HOPSWORKS-2147 Add feature group api scope to statistics resource endpoint
HOPSWORKS-2146 Resolve namenode consul name before setting feature group location
HOPSWORKS-2145 Flask (sklearn) race condition when reading pid file
HOPSWORKS-2128 Create api key not adding the right scope
HOPSWORKS-2122 Backend not handling null TimeTravelFormat sent by client
HOPSWORKS-2116 Add --network option to docker build
HOPSWORKS-2095 Uninstalling conda libraries doesn't work in airgapped environments
HOPSWORKS-2092 Overwriting a feature group by dropping/recreating loses default value of appended features
HOPSWORKS-2081 Fix search for registration disabled
HOPSWORKS-2080 Hops services should wait for Hopsworks before they start
HOPSWORKS-2068 Singleton initialization failure
HOPSWORKS-2067 shared featuregroup creation
HOPSWORKS-2065 Restart systemd units after failure
HOPSWORKS-2064 UI fixes for data preview
HOPSWORKS-2062 ProjectException should set prefix usrMsg from the causing exception's messge
HOPSWORKS-2061 Datasets browser order files by name not working while paginating
HOPSWORKS-2058 Conda environments are not removed when deleting a project
HOPSWORKS-2056 Change DNS server for docker containers in Localhost deployments
HOPSWORKS-2054 Increment timeout when launching Jupyter locally
HOPSWORKS-2053 Fix systemd dependencies
HOPSWORKS-2051 Save preferences for jobs/page filter
HOPSWORKS-2048 Jupyter-git missing configuration for git_backend
HOPSWORKS-2047 Non LTS Nvidia Driver version doesn't work on Azure VMs
HOPSWORKS-2038 Python client can't handle disabled online feature store
HOPSWORKS-2015 Users should not be allowed to create Dataset 'Projects'
HOPSWORKS-1988 IPynb converter fails for notebooks which contain special characters in the filename
HOPSWORKS-1918 Spark executor memory not modifiable from UI when starting jupyter
HOPSWORKS-1876 Data validation UI doesn't fill in Job parameters
HOPSWORKS-1861 API keys not all selectable for Github Jupyter notebooks

Improvement

HOPSWORKS-2140 Increase retries for HDFS Mover in hops-hadoop-chef
HOPSWORKS-2130 Refactor on-demand feature groups for HSFS
HOPSWORKS-2124 Remove R installation from spark::yarn
HOPSWORKS-2123 Add option to disable datasource validation when deploying Hopsworks
HOPSWORKS-2113 Add endpoint to list a user's projects
HOPSWORKS-2112 Return the user activation time as part of the userProfileDTO
HOPSWORKS-2093 remove env/pkgs dir symlinks in conda installation
HOPSWORKS-2091 Label/Prediction feature metadata for training datasets
HOPSWORKS-2084 Remove default_storage option from feature groups
HOPSWORKS-2077 add support for reading csv files as tf data object and rename feeder to tf_data
HOPSWORKS-2046 Kafka broker should use zookeeper fqdn instead of IP
HOPSWORKS-2045 artifact usage
HOPSWORKS-2044 Add option to return an error message if a user starts a YARN job but no nodemanagers are available
HOPSWORKS-2025 Delete a job's executions
HOPSWORKS-2010 Upgrade hudi to 0.6.1-SNAPSHOT for hsfs
HOPSWORKS-1986 Add redshift/snowflake examples and libraries
HOPSWORKS-1982 Deequ statistics for Feature Groups/Training Datasets
HOPSWORKS-1878 Add default args for Jobs
HOPSWORKS-1847 online feature store storage connectors should not be stored in the database
HOPSWORKS-1833 Move beam related files from conf directory to lib
HOPSWORKS-1769 enable publishing models to other projects
HOPSWORKS-1374 Add Hudi API to HSFS and commits to the FeatureStore metadata

New Feature

HOPSWORKS-2014 Manage AWS roles per application in Hopsworks
[HOPSWO...

Read more

Releases 1.4.0 - 1.4.1

09 Oct 14:33
f19a9cc
Compare
Choose a tag to compare
    Release notes - Hopsworks - Version 1.4.0

Improvement

New Feature

Bug

  • [HOPSWORKS-1421] - Hopsworks ignores chef parameter to set default max number of user projects
  • [HOPSWORKS-1504] - Workflow builder should sanitize job names
  • [HOPSWORKS-1568] - Installing Node.js must work for airgapped deployments
  • [HOPSWORKS-1623] - Overwrite config.ini for ndb-chef
  • [HOPSWORKS-1633] - Sparkmagic links to Spark UI and driver logs are broken
  • [HOPSWORKS-1634] - MaggyFacade throws getSingleResult for findByAppId in parallel experiments
  • [HOPSWORKS-1671] - increase systemctl timeout for ndbmtd
  • [HOPSWORKS-1672] - Featurestore Descriptions containing ' cannot be handled by hive
  • [HOPSWORKS-1740] - HopsKafkaAdminClient initialisation fails if kafka is down/unreachable and then never recovers
  • [HOPSWORKS-1768] - Restrict the usage of scan operations
  • [HOPSWORKS-1782] - Set C_INCLUDE_PATH and CPLUS_INCLUDE_PATH when installing libraries in Anaconda
  • [HOPSWORKS-1793] - Increase certificate keysize
  • [HOPSWORKS-1794] - Make Hopsworks logo consistent with the branding
  • [HOPSWORKS-1795] - Close connections in AirflowDagFacade
  • [HOPSWORKS-1796] - hops-util(-py) should use consul domain to reach Hopsworks
  • [HOPSWORKS-1797] - Hopsworks should listen on an internal port with certificate signed by HopsCA
  • [HOPSWORKS-1799] - Is kubernetes enabled is not evaluated correctly in front-end after variables service refactoring
  • [HOPSWORKS-1800] - cache provenance-epipe ...
Read more

Release 1.3.0

03 Jun 17:52
d6c3239
Compare
Choose a tag to compare
    Release notes - Hopsworks - Version 1.3.0

Bug

  • [HOPSWORKS-1382] - Placeholders for auditing REST API requests
  • [HOPSWORKS-1471] - Increase MaxNoOfTriggers and MaxNoOfFiredTriggers to delete large no of FKs
  • [HOPSWORKS-1545] - Handle cases where Elasticsearch is down
  • [HOPSWORKS-1546] - ruby tests
  • [HOPSWORKS-1556] - Kafka controller fixes
  • [HOPSWORKS-1557] - Dataset menu hidden as browser is not scrollable
  • [HOPSWORKS-1558] - dataset - share with cluster - fails
  • [HOPSWORKS-1559] - Truncate conda commands before upgrading to 1.1.0
  • [HOPSWORKS-1560] - Fix migration of datasets (HOPSWORKS-295)
  • [HOPSWORKS-1561] - Restrict access to elasticsearch data directory to elastic user
  • [HOPSWORKS-1563] - Schema of a shared topic is shown in one line
  • [HOPSWORKS-1565] - Removing a shared dataset removes it also from the source project
  • [HOPSWORKS-1566] - provenance elastic client - index not found exception not handled
  • [HOPSWORKS-1568] - Installing Node.js must work for airgapped deployments
  • [HOPSWORKS-1569] - Can't deploy with disable password login flag set to true
  • [HOPSWORKS-1578] - epipe operation fail on missing index
  • [HOPSWORKS-1579] - Align services on the project service nav bar
  • [HOPSWORKS-1581] - [hops-hadoop-log] Touch hadoop.log with correct ownership and permissions
  • [HOPSWORKS-1583] - Incorrect path for PYSPARK_ARCHIVES_PATH in spark-env.sh
  • [HOPSWORKS-1589] - Cannot upload to a shared dataset
  • [HOPSWORKS-1590] - Kafka advertised listeners external port conflict among brokers
  • [HOPSWORKS-1596] - Incorrect sklearn logs format
  • [HOPSWORKS-1599] - Cannot start/stop services from admin UI due to missing CommunicationController property
  • [HOPSWORKS-1600] - Services Status enum missing INIT state of kagent
  • [HOPSWORKS-1601] - Append projectId to topic name in kafka tour to avoid clashes with other topics
  • [HOPSWORKS-1603] - Kagent JWTHelper should retry in case of hopsworks authentication failure
  • [HOPSWORKS-1604] - In hops python library, the s3 connector tries to read parquet files as type avro
  • [HOPSWORKS-1609] - [ePipe] Fix version conflicts in Elasticsearch
  • [HOPSWORKS-1612] - Can't view directories with name url encoded
  • [HOPSWORKS-1616] - Hopsworks ear download URL is incorrect when installing with Kubernetes
  • [HOPSWORKS-1619] - Can't zip/unzip directories with name containing url encoded chars
  • [HOPSWORKS-1631] - [tensorflow-chef] Kernel-devel not installed correctly on some Centos flavors
  • [HOPSWORKS-1636] - [hops-hadoop-chef] Configuration key in force renew-service-tokens script is wrong
  • [HOPSWORKS-1637] - Papermill creates connection file without kernel_id in the filename which breaks notebook versioning
  • [HOPSWORKS-1643] - Filebeat kagent multiline property is not configured for new ELK 7.2
  • [HOPSWORKS-1645] - duplicate class in hopswoks-common
  • [HOPSWORKS-1646] - Leftover state when updating experiments table
  • [HOPSWORKS-1650] - Search does not work when in a service other than Data Sets
  • [HOPSWORKS-1653] - Error from Provenance when sharing dataset to target project name including upper case characters
  • [HOPSWORKS-1654] - Feature store page stuck in loading view if project name contains upper case letters
  • [HOPSWORKS-1663] - ePipe crashes after rerunning featurestore tour job
  • [HOPSWORKS-1664] - Shared dataset showing up in move and copy modals without project-origin prefix
  • [HOPSWORKS-1669] - Only chief worker should return wrapper return value in distributed training
  • [HOPSWORKS-1679] - Pin version of Java used in Berksfile
  • [HOPSWORKS-1682] - Downloading a file results in renamed file with the full path prefixed in the filename
  • [HOPSWORKS-1684] - [hops-util-py] nvidia-ml-py should be included in install_requires, remove tensorflow-gpu
  • [HOPSWORKS-1685] - Livy server should start after ResourceManager
  • [HOPSWORKS-1686] - Calling create training dataset with non existing storage connector doesn't fail
  • [HOPSWORKS-1690] - nvml_monitor fails when devices does not support device calls
  • [HOPSWORKS-1696] - Increase timeout for listing conda environments when creating projects
  • [HOPSWORKS-1699] - OOM Exceptions when parsing feature store metadata in hops-util-py
  • [HOPSWORKS-1703] - Create training dataset endpoint does not return full entity of the new training dataset
  • [HOPSWORKS-1704] - fetching schema and data preview not working in featurestore shared with another project
  • [HOPSWORKS-1706] - Use system's nameserver for DNS service discovery
  • [HOPSWORKS-1707] - Activating users using User-Admin broken
  • [HOPSWORKS-1708] - When sharing a featurestore dataset training dataset should also be shared
  • [HOPSWORKS-1710] - Add permission inheritance configuration to hive-site
  • [HOP...
Read more

Release 1.2.0

10 Feb 12:36
Compare
Choose a tag to compare
    Release notes - Hopsworks - Version 1.2.0

Bug

  • [HOPSWORKS-1283] - Zip does not work when there is a space in the file path
  • [HOPSWORKS-1379] - Timers should be rescheduled when failed and handle exceptions
  • [HOPSWORKS-1382] - Placeholders for auditing REST API requests
  • [HOPSWORKS-1507] - [ndb-chef] List only TCP and UDP sockets when starting MGM server
  • [HOPSWORKS-1509] - Allow all OU in nodes DN in elasticsearch configuration
  • [HOPSWORKS-1511] - pci-tools package not available and torchvision broken for 0.4.1
  • [HOPSWORKS-1515] - airflow webserver , scheduler and sqoop should not be enabled when service.enabled is false
  • [HOPSWORKS-1534] - Kibana error when recreating the project enviroment
  • [HOPSWORKS-1541] - [ePipe] Skip non searchable fields for schemabased exmetadata
  • [HOPSWORKS-1543] - [ePipe] Retry in case of conflict when adding datasets and projects
  • [HOPSWORKS-1545] - Handle cases where Elasticsearch is down
  • [HOPSWORKS-1550] - Do not upgrade pip binary in tensorflow default recipe
  • [HOPSWORKS-1553] - provenance cleaner timer
  • [HOPSWORKS-1556] - Kafka controller fixes
  • [HOPSWORKS-1559] - Truncate conda commands before upgrading to 1.1.0
  • [HOPSWORKS-1560] - Fix migration of datasets (HOPSWORKS-295)
  • [HOPSWORKS-1561] - Restrict access to elasticsearch data directory to elastic user
  • [HOPSWORKS-1563] - Schema of a shared topic is shown in one line
  • [HOPSWORKS-1564] - Job args expat tries to set null instead of empty string
  • [HOPSWORKS-1565] - Removing a shared dataset removes it also from the source project
  • [HOPSWORKS-1578] - epipe operation fail on missing index
  • [HOPSWORKS-1589] - Cannot upload to a shared dataset
  • [HOPSWORKS-1590] - Kafka advertised listeners external port conflict among brokers
  • [HOPSWORKS-1594] - HOPSWORKS-1554 breaks conda search in Hopsworks
  • [HOPSWORKS-1599] - Cannot start/stop services from admin UI due to missing CommunicationController property
  • [HOPSWORKS-1600] - Services Status enum missing INIT state of kagent
  • [HOPSWORKS-1601] - Append projectId to topic name in kafka tour to avoid clashes with other topics
  • [HOPSWORKS-1603] - Kagent JWTHelper should retry in case of hopsworks authentication failure
  • [HOPSWORKS-1610] - Prometheus alerting-rules files owned by root

Improvement

  • [HOPSWORKS-836] - Send message in Hopsworks to a user that has been added to a project.
  • [HOPSWORKS-1257] - sudoers script should be owned by root and 700
  • [HOPSWORKS-1306] - [ePipe] Add support for HA Elasticsearch
  • [HOPSWORKS-1308] - [ePipe] Rewrite and fix the metrics code
  • [HOPSWORKS-1481] - Add endpoints to retrieve feature groups and training datasets by name
  • [HOPSWORKS-1483] - Refactor Admin API - Hosts
  • [HOPSWORKS-1494] - Feature store UI improvements
  • [HOPSWORKS-1527] - Check TensorBoard readiness using http in Experiments service and for running applications
  • [HOPSWORKS-1532] - No need to mount an S3 bucket for the certificates in Databricks
  • [HOPSWORKS-1538] - Automatically setup hive and hopsfs on databricks cluster
  • [HOPSWORKS-1544] - Add endpoint to retrieve featurestore by name
  • [HOPSWORKS-1554] - Add support for proxy configuration in condarc
  • [HOPSWORKS-1562] - Improve databricks docs for the feature store

Task

  • [HOPSWORKS-1330] - Remove deprecated LocalFsService (Charon) codebase
  • [HOPSWORKS-1354] - Import(upload) template saves files in /Projects/Uploads/<Sample.json>
  • [HOPSWORKS-1407] - Change error message when failing to export project-user certificates
  • [HOPSWORKS-1522] - Update support link in Hopsworks and docs
  • [HOPSWORKS-1525] - [hops-hadoop-chef] Add option to allow Yarn auto discover HW capabilities
  • [HOPSWORKS-1528] - Remove unnecessary code to set kibana default index
  • [HOPSWORKS-1536] - Right-click rename does not work in Datasets browser
  • [HOPSWORKS-1540] - Upgrade PyTorch to 1.4
  • [HOPSWORKS-1549] - [kagent-chef] Update Hops CA bundle after host certificate rotation
  • [HOPSWORKS-1551] - Dynamically create Airflow DB connection only if Airflow is installed
  • [HOPSWORKS-1575] - Replace hops.readthedocs with hopsworks.readthedocs
  • [HOPSWORKS-1577] - [ndb-chef] Restrict MySQL connections to localhost for localhost installations
  • [HOPSWORKS-1582] - Add a new db user for Kafka
  • [HOPSWORKS-1611] - Tensorflow::default fails to install avro-python3 due to upstream changes

Release 1.1.0

10 Feb 12:32
Compare
Choose a tag to compare
    Release notes - Hopsworks - Version 1.1.0

Bug

  • [HOPSWORKS-1214] - Airflow HopsworksLaunchOperator does not handle job state App Master Failed
  • [HOPSWORKS-1245] - [ePipe] Fixes for xattrs handling to be work with the changes introduced in HOPS-1434 and HOPS-1451
  • [HOPSWORKS-1246] - [ePipe] Check for the existence of elastic fields before delete
  • [HOPSWORKS-1296] - [ePipe] log replay fails on xattr op
  • [HOPSWORKS-1305] - Fix broken Anaconda export yml
  • [HOPSWORKS-1318] - Featurestore should throw an error if you try to create the same featuregroup twice
  • [HOPSWORKS-1323] - [ndb-chef] Fix bug in NDB cluster-init.sh
  • [HOPSWORKS-1325] - Online Feature store tests fail when project name contains uppercase letters
  • [HOPSWORKS-1342] - Add parameter for s3 encryption type in core-site.xml
  • [HOPSWORKS-1353] - Flink timer should handle case when services are not installed
  • [HOPSWORKS-1385] - HA elastic does not have a synchronization point between starting nodes and creating indices
  • [HOPSWORKS-1398] - User activation date is irrational
  • [HOPSWORKS-1408] - AirflowManager kills Hopsworks deployments if Airflow directory does not exist
  • [HOPSWORKS-1412] - Fix create featuregroup with partition keys
  • [HOPSWORKS-1413] - AWS S3 does not accept empty Authorization header when downloading from a public bucket
  • [HOPSWORKS-1425] - Update stack to handle hopsfs paths starting by hopsfs instead of hdfs
  • [HOPSWORKS-1426] - Avoid using 100% containers for yarn containers
  • [HOPSWORKS-1428] - hops_airflow fails to install pymssql 3.0.3, pin it to 2.1.4
  • [HOPSWORKS-1431] - Better handling of Jupyter/Git failures
  • [HOPSWORKS-1432] - Elasticsearch node should always use its private ip
  • [HOPSWORKS-1434] - Scale jvm -Xmx for Spark and Jupyter based on allocated memory
  • [HOPSWORKS-1442] - Update statistics job shows app file as undefined
  • [HOPSWORKS-1443] - [featurestore] Spark returning Infinity values for mean statistics of string type features
  • [HOPSWORKS-1451] - Storage connector ignored when creating a training dataset from the UI
  • [HOPSWORKS-1456] - [hops-hadoop-chef] Hopsworks security related properties are not templated correctly
  • [HOPSWORKS-1457] - Correct systemd scripts
  • [HOPSWORKS-1458] - spark should not bring any hops jar with it
  • [HOPSWORKS-1461] - The JWT signing key should be decoded first before using it to sign jwt
  • [HOPSWORKS-1463] - Hops ndb version not taken in account when downloading dal
  • [HOPSWORKS-1464] - fix chef for HOPS-1542
  • [HOPSWORKS-1484] - Shared feature stores do not appear in the UI
  • [HOPSWORKS-1485] - Sparkmagic upgrades not working
  • [HOPSWORKS-1488] - Deep Learning tour project doesn't create Kafka inferenceschema
  • [HOPSWORKS-1489] - Increase default Heap size and max shards per node for Elasticsearch
  • [HOPSWORKS-1490] - Handle Kibana failures in Hopsworks Kibana client
  • [HOPSWORKS-1496] - Elastic::default does not template correctly hosts in elasticsearch.yml
  • [HOPSWORKS-1497] - hops-util-py incorrectly sets Kafka config
  • [HOPSWORKS-1498] - hops-util-py can't find attributes in KafkaTopicDTO
  • [HOPSWORKS-1507] - [ndb-chef] List only TCP and UDP sockets when starting MGM server
  • [HOPSWORKS-1508] - [kagent-chef] Update kibana x.509 certificate when host certificate is rotated
  • [HOPSWORKS-1510] - Remove the code to delete index patterns since we delete kibana project index
  • [HOPSWORKS-1516] - [ePipe] handle the case if XAttrAddAll is received for a deleted inode
  • [HOPSWORKS-1518] - [livy-chef] Create root Hops directory
  • [HOPSWORKS-1519] - The subject of ELK JWT token should be always lowercase
  • [HOPSWORKS-1521] - Dataset browser refresh button not working
  • [HOPSWORKS-1523] - Kibana opendistro_security plugin is not idempotent
  • [HOPSWORKS-1524] - [hops-hadoop-chef] Hops services startup should be dependent on Glassfish
  • [HOPSWORKS-1526] - Can't update job when doing insert into featuregroup
  • [HOPSWORKS-1564] - Job args expat tries to set null instead of empty string

Sub-task

New Feature

  • [HOPSWORKS-1109] - Experiments and models v2
  • [HOPSWORKS-1334] - [hopsworks-cloud-sdk] SQL query for creating training datasets
  • [HOPSWORKS-1338] - [featurestore] Configurable in-bucket path for exporting training datasets to S3
  • [HOPSWORKS-1340] - Airflow DAG UI composer
  • [HOPSWORKS-1362] - Endpoint to securely get a project user's x.509 credentials
  • [
Read more

Release 1.0.0

28 Oct 21:32
Compare
Choose a tag to compare
    Release notes - Hopsworks - Version 1.0.0

Bug

  • [HOPSWORKS-432] - Clean Kafka ACLs when unsharing a topic or deleting a project.
  • [HOPSWORKS-1095] - Localhost (127.0.0.1) cluster definition support
  • [HOPSWORKS-1114] - Rework password recovery
  • [HOPSWORKS-1146] - Kagent stops reporting if env creation fails with non default repo
  • [HOPSWORKS-1152] - Dataset browser not displaying contents of shared datasets with naming conflicts
  • [HOPSWORKS-1172] - Upload progress not visible in the beginning of uploading large files
  • [HOPSWORKS-1175] - yarnapp/user attribute not propagated correctly to yarn-site.xml
  • [HOPSWORKS-1176] - Security risk for default admin password pops up even if the password has been changed
  • [HOPSWORKS-1178] - [airflow-chef] Add default timezone for Airflow
  • [HOPSWORKS-1180] - Chef upgrade mixes kafka brokers ids
  • [HOPSWORKS-1185] - Jupyter not working when Hopsworks runs behind a LB/reverse proxy w/ TLS termination
  • [HOPSWORKS-1190] - [featurestore] missing argument when writing petastorm datasets
  • [HOPSWORKS-1195] - Bad handling of Job resource missing
  • [HOPSWORKS-1197] - Fix broken admin and agent tests
  • [HOPSWORKS-1198] - Cleanup hadoop sbin scripts
  • [HOPSWORKS-1199] - ErrorCode range mismatch and TestRestCodes not testing all ErrorCodes
  • [HOPSWORKS-1203] - [ePipe] Fixes for JSON handling in the XAttr table
  • [HOPSWORKS-1207] - Hopsworks broken on Ubuntu 18.04
  • [HOPSWORKS-1209] - filebeat_kagent does not start with kagent start-all services script
  • [HOPSWORKS-1210] - Jupyter now requires the python kernel.json to be under python{2, 3} dir.
  • [HOPSWORKS-1215] - Use OSProcessExecutor for Python library search
  • [HOPSWORKS-1216] - tensorflow-chef install recipe CentOS hiccups
  • [HOPSWORKS-1218] - Python service not working when searching or installing libraries containing 'http'
  • [HOPSWORKS-1225] - add attribute for url to download spark in spark-chef
  • [HOPSWORKS-1228] - Remove livy session name from sparkmagic conf file
  • [HOPSWORKS-1233] - [ePipe] hivecleaner fail during project removal
  • [HOPSWORKS-1234] - Remove sudo command from sudoers scripts
  • [HOPSWORKS-1236] - Project cleanup does not work after a failed project creation
  • [HOPSWORKS-1246] - [ePipe] Check for the existence of elastic fields before delete
  • [HOPSWORKS-1251] - images not rendering in Jupyter notebooks
  • [HOPSWORKS-1252] - Airflow HopsworksLaunchOperator waits for wrong Job statuses
  • [HOPSWORKS-1253] - [ePipe] Rename operation should read the inodes info from the DB
  • [HOPSWORKS-1263] - Project cleanup fails if X.509 to be revoked does not exist
  • [HOPSWORKS-1268] - Sanitize python library name/version/reposity/url
  • [HOPSWORKS-1271] - Bump HopsHive to use ORC 1.5.5
  • [HOPSWORKS-1276] - Anaconda import yml file not working, removal of explicit enabling of python kernel
  • [HOPSWORKS-1278] - [ePipe] The tailer should guarantee the order of epochs during recovery and while waiting for events
  • [HOPSWORKS-1279] - [ePipe] Check for the existence of the xattr parent field before delete
  • [HOPSWORKS-1280] - [hops-util-py] add_module not working in python kernel
  • [HOPSWORKS-1282] - Exclude netty from Hudi hdfs dependency
  • [HOPSWORKS-1285] - Livy is missing Yarn GENERATING_SECURITY_MATERIAL state
  • [HOPSWORKS-1290] - [hops-util-py] numpy_helper and pandas_helper documentation not rendering
  • [HOPSWORKS-1297] - Model Serving view not sorted
  • [HOPSWORKS-1298] - Cannot remove job associated with featuregroups
  • [HOPSWORKS-1300] - [kagent-chef] Change deprecated node.set on install recipe
  • [HOPSWORKS-1315] - Hopsworks wrongly sets umask for HDFS client to 000
  • [HOPSWORKS-1326] - setup-chef is broken
  • [HOPSWORKS-1328] - [airflow-chef] Package libjpeg-devel is no longer available
  • [HOPSWORKS-1329] - [hopsworks-cloud-sdk] Fix connecting from Databricks
  • [HOPSWORKS-1336] - [kagent] Host private key is not converted to PKCS#1 if it's an upgrade
  • [HOPSWORKS-1343] - Airflow Hopsworks connection hook ignores configured connections
  • [HOPSWORKS-1350] - [featurestore] default training dataset connector not found
  • [HOPSWORKS-1355] - Conda solver is problematic
  • [HOPSWORKS-1357] - flyway_schema_history fix for versions < 0.6 doesn't work after 1.0.0
  • [HOPSWORKS-1358] - Handle glassfish upgrade from 4.1.2.174 to 4.1.2.181
  • [HOPSWORKS-1359] - MySQL server reinitialized during update to 1.0.0
  • [HOPSWORKS-1360] - Data validation JS controller does not properly include mainCtrl
  • [HOPSWORKS-1365] - When upgrading ignore anaconda base env's lib dir of previous installations
  • [HOPSWORKS-1368] - NullPointerException when deleting a project without Onlin...
Read more

0.10.0

02 Jul 14:22
Compare
Choose a tag to compare
    Release notes - Hopsworks - Version 0.10.0

Bug

  • [HOPSWORKS-197] - Turning on and off tours is a terrible user experience
  • [HOPSWORKS-715] - Hopsworks singletons are not single
  • [HOPSWORKS-770] - Disable http port and TLS 1.0 on glassfish
  • [HOPSWORKS-849] - kagent - conda can get stuck
  • [HOPSWORKS-903] - Jupyter cleanup and timer bugfixes
  • [HOPSWORKS-904] - Users should be able to specify version of Python library to install when behind PyPi proxy
  • [HOPSWORKS-905] - remote_material_references is not cleaned up during project deletion
  • [HOPSWORKS-913] - Feature store quota missing from admin UI in hopsworks
  • [HOPSWORKS-917] - Non HA admin UI when HA namenodes
  • [HOPSWORKS-920] - Job monitor thread gets stuck if job is killed while running
  • [HOPSWORKS-927] - TensorBoard could not bind to an unsupported address family
  • [HOPSWORKS-928] - [Featurestore] Bug: nested spark schemas are not automatically translatable to Hive
  • [HOPSWORKS-929] - [Featurestore] Don't allow hyphen in featuregroup names because Hive do not allow it
  • [HOPSWORKS-930] - [Featurestore] bug, in API retrieval code in feature-registry featurestore is undefined
  • [HOPSWORKS-932] - Jobs UI date selector should persist user-selected values
  • [HOPSWORKS-937] - Adjust Cgroup configuration to account for longer Cgroup deletion and make CPU utilization configurable
  • [HOPSWORKS-938] - Adding pyFiles to Jupyter does not add it to python path
  • [HOPSWORKS-941] - Fix dataset access request and related tests
  • [HOPSWORKS-945] - Zipping and unzipping is broken with Hops TLS
  • [HOPSWORKS-948] - [hops-hadoop-chef] Copy hadoop logs utility is not removed from crontab
  • [HOPSWORKS-949] - [airflow-chef] Install recipe does not create root hops installation directory
  • [HOPSWORKS-951] - sqoop and airflow bugfixes
  • [HOPSWORKS-953] - [kagent-chef] kagent falsely assumes service is dead when it restarts
  • [HOPSWORKS-955] - [airflow-chef] Use assigned roles in JWT to determine if a user is admin
  • [HOPSWORKS-958] - AirflowJWTManager randomly does not renew tokens
  • [HOPSWORKS-962] - copy_to_hdfs is broken in hops-util-py
  • [HOPSWORKS-965] - Certificate enddate should always be UTC
  • [HOPSWORKS-968] - Dataset browser pagination and select all not working
  • [HOPSWORKS-970] - upgrade chef ulimit cookbook
  • [HOPSWORKS-972] - Missing sasl/sasl.h breaks PyHopsHive installation
  • [HOPSWORKS-973] - Dela install should create the hops group before adding members to it
  • [HOPSWORKS-974] - Blank padded day in rspec breaks tests
  • [HOPSWORKS-975] - [airflow-chef] Change restart policy of airflow scheduler
  • [HOPSWORKS-976] - Airflow file manager does not refresh secret directory when navigate to another project
  • [HOPSWORKS-977] - Pin tornado version to 5.1.1
  • [HOPSWORKS-978] - dela::install should create /srv/hops/ before creating sub directories
  • [HOPSWORKS-979] - Cannot change default 'admin' password of [email protected]
  • [HOPSWORKS-980] - HOPSWORKS-950 breaks hops-util-py in python 2.7 environments
  • [HOPSWORKS-983] - [ePipe] Wrong type for long values in query condition handling
  • [HOPSWORKS-987] - User admin to slow and user search returning duplicated results
  • [HOPSWORKS-988] - Launching Jupyter server gradually gets slower and times out
  • [HOPSWORKS-990] - uploading big files from web ui can cause 'invalidated token'
  • [HOPSWORKS-992] - [kagent-chef] Invalid command in kagent's service status script
  • [HOPSWORKS-993] - [Feturestore] tf record schema infer for float arrays is not working correctly
  • [HOPSWORKS-994] - TensorBoard in experiments service fails to start if tensorflow-gpu is in the project environment
  • [HOPSWORKS-996] - Jobs monitor threads may saturate the system
  • [HOPSWORKS-1000] - Parameter servers in ParameterServerStrategy do not return Spark task when all workers are finished
  • [HOPSWORKS-1003] - Jupyter convert notebook should inject SparkSession and handle spaces in notebook name
  • [HOPSWORKS-1004] - YarnJobMonitor does not have a full list of Yarn running states
  • [HOPSWORKS-1007] - Install version specific kernel packages, not generic ones
  • [HOPSWORKS-1009] - Non-websocket requests in Jupyter notebook are blocked due to cross-origin policy
  • [HOPSWORKS-1011] - hops-metadata-dal-impl-ndb schema update renames hdfs_inode_attributes to hdfs_directory_with_quota_feature
  • [HOPSWORKS-1012] - Metadata designer does not work
  • [HOPSWORKS-1013] - Descrease websocket_ping_interval to avoid proxied websocket connections being dropped
  • [HOPSWORKS-1014] - ui bugs
  • [HOPSWORKS-1017] - Date in field "feature store created" is NaN
  • [HOPSWORKS-1022] - NOT_START is not a final state for Log Aggregation
  • [HOPSWORKS-1030] - [hops-util-py] copy_to_local bugfixes and removal of localize
  • [
Read more

0.9.1

04 Apr 09:03
Compare
Choose a tag to compare
    Release notes - Hopsworks - Version 0.9.1

Bug

  • [HOPSWORKS-197] - Turning on and off tours is a terrible user experience
  • [HOPSWORKS-903] - Jupyter cleanup and timer bugfixes
  • [HOPSWORKS-913] - Feature store quota missing from admin UI in hopsworks
  • [HOPSWORKS-920] - Job monitor thread gets stuck if job is killed while running
  • [HOPSWORKS-927] - TensorBoard could not bind to an unsupported address family
  • [HOPSWORKS-928] - [Featurestore] Bug: nested spark schemas are not automatically translatable to Hive
  • [HOPSWORKS-929] - [Featurestore] Don't allow hyphen in featuregroup names because Hive do not allow it
  • [HOPSWORKS-930] - [Featurestore] bug, in API retrieval code in feature-registry featurestore is undefined
  • [HOPSWORKS-932] - Jobs UI date selector should persist user-selected values
  • [HOPSWORKS-938] - Adding pyFiles to Jupyter does not add it to python path
  • [HOPSWORKS-941] - Fix dataset access request and related tests
  • [HOPSWORKS-945] - Zipping and unzipping is broken with Hops TLS
  • [HOPSWORKS-949] - [airflow-chef] Install recipe does not create root hops installation directory
  • [HOPSWORKS-951] - sqoop and airflow bugfixes
  • [HOPSWORKS-953] - [kagent-chef] kagent falsely assumes service is dead when it restarts
  • [HOPSWORKS-955] - [airflow-chef] Use assigned roles in JWT to determine if a user is admin
  • [HOPSWORKS-958] - AirflowJWTManager randomly does not renew tokens
  • [HOPSWORKS-968] - Dataset browser pagination and select all not working
  • [HOPSWORKS-974] - Blank padded day in rspec breaks tests
  • [HOPSWORKS-975] - [airflow-chef] Change restart policy of airflow scheduler
  • [HOPSWORKS-976] - Airflow file manager does not refresh secret directory when navigate to another project
  • [HOPSWORKS-977] - Pin tornado version to 5.1.1
  • [HOPSWORKS-978] - dela::install should create /srv/hops/ before creating sub directories
  • [HOPSWORKS-979] - Cannot change default 'admin' password of [email protected]
  • [HOPSWORKS-987] - User admin to slow and user search returning duplicated results
  • [HOPSWORKS-988] - Launching Jupyter server gradually gets slower and times out
  • [HOPSWORKS-990] - uploading big files from web ui can cause 'invalidated token'
  • [HOPSWORKS-992] - [kagent-chef] Invalid command in kagent's service status script
  • [HOPSWORKS-993] - [Feturestore] tf record schema infer for float arrays is not working correctly
  • [HOPSWORKS-996] - Jobs monitor threads may saturate the system
  • [HOPSWORKS-1004] - YarnJobMonitor does not have a full list of Yarn running states
  • [HOPSWORKS-1007] - Install version specific kernel packages, not generic ones
  • [HOPSWORKS-1012] - Metadata designer does not work
  • [HOPSWORKS-1017] - Date in field "feature store created" is NaN
  • [HOPSWORKS-1022] - NOT_START is not a final state for Log Aggregation

Task

  • [HOPSWORKS-959] - Remove apache2 dependency from cookbooks
  • [HOPSWORKS-984] - Remove commons-fileupload dependency from kmon and admin modules