This repository contains the official code for session-based recommender system Serenade, which employs VMIS-kNN. It learns users' preferences by capturing the short-term and sequential patterns from the evolution of user behaviors and predicts interesting next items with low latency with support for millions of distinct items. VMIS-kNN is an index-based variant of a state-of-the-art nearest neighbor algorithm to session-based recommendation, which scales to use cases with hundreds of millions of clicks to search through.
The VMIS-kNN implementation has a p90 prediction latency of <1.7ms in our micro benchmarks on private and public trainingsets up to 60M user-item interactions with 1.76 million distinct items. The Serenade recommender service using the VMIS-kNN algorithm easily handles 1000 predictions per second using only two vCPU's in total. The p90 prediction latency of the deployed system with kubernetes is < 7ms end-to-end, measured from a different node using a http client, including http overhead, network traffic, istio loadbalancers, fetching session information from RocksDb and filtering for product availablity and intimacy, the serializing of the results etc. Training data is 2.3 billion records of which 582 million training records are used after index pruning and contains about 6.5 million distinct products. The index requires about 11GB of memory per serving node.
- Downloads
- Find the best hyperparameter values
- Configure Serenade to use the hyperparameter values
- Start the Serenade service
- Retrieve recommendations using python
- Evaluate the testset
- Using your own train- and testset
- Citation
- Join us
- License
Serenade can be downloaded here. Binary executables are available for Windows, Linux and MacOS. Download the toy example project which contains toy datasets and a preconfigured configuration file.
Extract both downloaded files in the same directoy. You now have the following files:
serving
tpe_hyperparameter_optm
evaluator
train.txt
test.txt
valid.txt
example.toml
The next step is finding the hyperparameters using the train and test-datasets.
Serenade uses Tree-Structured Parzen Estimator (TPE) for finding the hyperparameters. TPE achieves low validation errors compared to Exhaustive Grid Search (Bergstra et al).
The section [hyperparam]
in the example.toml
contains the ranges of hyperparameter values that will be explored.
- The hyperparameter search can be started using:
./tpe_hyperparameter_optm example.toml
The results will be printed out in the terminal, for example:
===============================================================
=== HYPER PARAMETER OPTIMIZATION RESULTS ====
===============================================================
MRR@20 for validation data: 0.3197
MRR@20 for test data: 0.3401
enabled business_logic for evaluation:false
best hyperparameter values:
n_most_recent_sessions:1502
neighborhood_size_k:288
idf_weighting:2
last_items_in_session:4
HPO done
and also in the output file defined in the config file, for example:
out_path = "results.csv"
We now update the [model]
values in configuration file example.toml
to use the hyperparameter values and set the training_data_path with the location of the train.txt
.
This is the content of the example configuration file with the new [model]
paramer values.
config_type = "toml"
[server]
host = "0.0.0.0"
port = 8080
num_workers = 4
[log]
level = "info" # not implemented
[data]
training_data_path="train.txt"
[model]
m_most_recent_sessions = 1502
neighborhood_size_k = 288
max_items_in_session = 4
num_items_to_recommend = 21
idf_weighting = 1
[logic]
enable_business_logic = "false"
[hyperparam]
training_data_path = "train.txt"
test_data_path = "test.txt"
validation_data_path = "valid.txt"
num_iterations = 15
save_records = true
out_path = "results.csv"
enable_business_logic = false
n_most_recent_sessions_range = [100, 2500]
neighborhood_size_k_range = [50, 1500]
last_items_in_session_range = [1, 20]
idf_weighting_range = [0, 5]
Start the serving
binary for your platform with the location of the configuration file as argument
./serving example.toml
You can open your webbrowser and goto http://localhost:8080/ you should see an internal page of Serenade.
Serenade exposes Prometheus metrics out-of-the-box for monitoring.
import requests
from requests.exceptions import HTTPError
try:
myurl = 'http://localhost:8080/v1/recommend'
params = dict(
session_id='144',
user_consent='true',
item_id='13598',
)
response = requests.get(url=myurl, params=params)
response.raise_for_status()
# access json content
jsonResponse = response.json()
print(jsonResponse)
except HTTPError as http_err:
print(f'HTTP error occurred: {http_err}')
except Exception as err:
print(f'Other error occurred: {err}')
[2835,10,12068,4313,3097,8028,3545,7812,17519,1164,17935,1277,13335,8655,14664,14556,6868,13509,9248,2498,11724]
The returned json object is a list with recommended items.
The evaluator
application can be used to evaluate a test dataset. It reports on several metrics.
- The evaluation can be started using:
./evaluator example.toml
===============================================================
=== START EVALUATING TEST FILE ====
===============================================================
Mrr@20,Ndcg@20,HitRate@20,Popularity@20,Precision@20,Coverage@20,Recall@20,F1score@20
0.3277,0.3553,0.6402,0.0499,0.0680,0.2765,0.4456,0.1180
Qty test evaluations: 931
Prediction latency
p90 (microseconds): 66
p95 (microseconds): 66
p99.5 (microseconds): 66
A train- and testset must be created from historical user-item click data, outside of Serenade. Each row in the training- or test set should contain an historical user-item interaction event with the following fields:
SessionId
the ID of the session. Format: 64 bit IntegerItemId
the ID of the interacted item. Format: 64 bit IntegerTime
the time when the user-item interaction occurred. In epoch seconds: 32 bit Floating point.
The last 24 hours in the historical data can be used as test-set while the rest of the sessions can be used as the training-set and written as plain text using a '\t'
as field separator.
This is an example of a training data CSV file train.txt:
SessionId ItemId Time
10036 14957 1592337718.0
10036 14713 1592337765.0
10036 2625 1592338184.0
10037 7267 1591979344.0
10037 13892 1591979380.0
10037 7267 1591979504.0
10037 3595 1591979784.0
10038 6424 1591008704.0
Serenade - Low-Latency Session-Based Recommendation in e-Commerce at Scale
@inproceedings{Kersbergen2022SerenadeScale,
title = {{Serenade - Low-Latency Session-Based Recommendation in e-Commerce at Scale}},
year = {2022},
series = {SIGMOD '22}
author = {Kersbergen, Barrie and Sprangers, Olivier and Schelter, Sebastian},
booktitle = {Proceedings of the 2022 International Conference on Management of Data},
pages = {150–159},
numpages = {10},
location = {Philadelphia, PA, USA},
isbn = {9781450392495},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
doi = {10.1145/3514221.3517901},
}
Anyone who interested in our project is welcome to join us. Let us build a wonderful open source community for this session-based recommender system! We are always looking for the developers that contribute core code (Rust and python) or specific algorithms to the Serenade Project. Example ideas for improvement are: add specific algorithms such as:
- (STAN)[https://github.com/rn5l/session-rec/blob/master/algorithms/knn/stan.py]
- (V-STAN)[https://github.com/rn5l/session-rec/blob/master/algorithms/knn/vstan.py] make hardcoded design choices configurable
- let the user define the metric (MRR, Hitrate, NDCG etc) and its length for the evaluator.rs via the config file instead of hardcoded
- improve the duration for the hyperparameter.rs for larger files. Right now it uses just one thread which takes
- let the user define the percentiles in the configuration file that should be printed when using the evaluator.rs and more! Please contact us.
This project is licensed under the terms of the Apache 2.0 license.