CHANGELOG.md

Release Note (v0.0.46)

Release time: 2019-10-17 18:14:45

🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, felix, raccoonliukai, hanhxiao, Jem, 🙇

🆕 New Features

• [02941b6c] - parser: add ctrl_with_ipc to cli (hanhxiao)

🐞 Bug fixes

• [dbf1a5e7] - release: add some hints to the release script (Han Xiao)
• [a641d5c7] - shot-detector: rename shot detector (felix)
• [d48eb53a] - preprocessor: add max_shot_num for shotdetect (raccoonliukai)
• [9331ef58] - flow: use recommend flow api to reduce confusion (hanhxiao)
• [660f8f99] - flow: make base flow as class not object (hanhxiao)
• [3062c43c] - ci: remove cffi from gnes docker image (hanhxiao)
• [38147fe8] - helper: fix gpuutil exception (hanhxiao)
• [15d9b4fe] - ci: fix cffi version to 1.12.3 (hanhxiao)
• [cbac2de4] - ci: fix cffi to 1.12.3 (hanhxiao)
• [24e41bec] - preprocessor: add numpy transform (Jem)
• [6008f7d1] - service: revert socket log (Han Xiao)
• [474deddf] - control-sock: build control socket at the begining (felix)
• [707d9e96] - service-logging: show socket creating (felix)
• [01531f74] - stream-call: hungry mode to receive responses (felix)

🚧 Code Refactoring

• [f16c672e] - client: make query method as generator (hanhxiao)

📗 Documentation

• [ec03351d] - flow: add flow to readme as main api (Han Xiao)

🍹 Other Improvements

• [31c05416] - flow: mark flow as a breaking change (Han Xiao)
• [4efd414e] - changelog: update change log to v0.0.45 (hanhxiao)
• [14fa5a18] - ... (felix)

💥 Breaking Changes (v0.0.45 -> v0.0.46)

The new GNES Flow API introduced since v0.0.46 has become the main API of GNES. It provides a pythonic and intuitive way of building pipelines in GNES, enabling run/debug on a local machine. It also supports graph visualization, swarm/k8s config export, etc. More information about GNES Flow can be found at here.

As a consequence, the composer module as well as gnes compose CLI and GNES board web UI will be removed in the next releases.

GNES board will be redesigned using the GNES Flow API. We highly welcome your contribution on this thread!

Release Note (v0.0.45)

Release time: 2019-10-15 14:01:07

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, felix, Han Xiao, 🙇

🆕 New Features

• [166698ce] - flow: add index and query flow as common flow (hanhxiao)
• [8a60c261] - flow: flow can not export docker swarm config (hanhxiao)
• [80cb530e] - flow: add flow to python generator (hanhxiao)
• [f6536c87] - flow: add eq operator to the flow to enable comparison (hanhxiao)
• [9ca757b4] - flow: add set remove and set_last (hanhxiao)
• [3c3c54b5] - webp-encoder: support webp encoder (felix)
• [b94490f1] - flow: allow add service to be str (hanhxiao)
• [4055ad8e] - flow: add support to replicas plot (hanhxiao)
• [7265f76c] - grpc: add proxy argument to cli (hanhxiao)
• [3901078c] - incep_v4_encoder: add inception v4 encoder for video (felix)

🐞 Bug fixes

• [8911314b] - style: double quote to single quote (Han Xiao)
• [228a2b19] - flow: fix unit test assert in flow (hanhxiao)
• [7d2c681e] - flow: add warning to jpg downloader (hanhxiao)
• [fce94d94] - service: fix ServiceManager replicas router (hanhxiao)
• [2705c287] - video-decoder: none chunk spliter (felix)

🚧 Code Refactoring

• [48d1828c] - flow: reorganize file structure for flow (hanhxiao)
• [c8f1df5b] - flow: flow is now a trainable base (hanhxiao)

🍹 Other Improvements

• [5121d65a] - revert unittest (felix)
• [8ada2538] - changelog: update change log to v0.0.44 (hanhxiao)

Release Note (v0.0.44)

Release time: 2019-10-11 15:27:37

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, felix, Jem, 🙇

🆕 New Features

• [2fb0f4f9] - flow: add dump to jpg (hanhxiao)
• [552fcdfe] - indexer-cli: add as_response switcher to indexer cli (hanhxiao)
• [c8cedd04] - service: remove async dump for better stability (hanhxiao)
• [1739c7b6] - flow: add client to flow (hanhxiao)
• [43b9d014] - flow: add context manager to flow (hanhxiao)
• [ae0d4056] - flow: first version of gnes flow (hanhxiao)

🐞 Bug fixes

• [c23ea61f] - frontend: fix frontend blocking behavior (hanhxiao)
• [c880c9b0] - service: make service handler thread-safe (hanhxiao)
• [a3da0582] - flow: fix flow unit test (hanhxiao)
• [6d118404] - ffmpeg: threads=1 (felix)
• [bca5b5b7] - base: fix env expansion in gnes_config (hanhxiao)
• [72f4a044] - indexer: fix empty chunk and dump_interval (hanhxiao)
• [9b79cdf5] - memory-leak: try to fix memory leak danger (felix)
• [16097f3f] - video-decoder: fix name (felix)
• [199a71a6] - frontend: remove duplicate receive (hanhxiao)
• [73dae6bd] - service: minor fix on the dump_interval (hanhxiao)
• [6f401905] - client: fix bugs for client (Jem)
• [c5af9308] - parser: use str instead of textio stream to prevent serializer err (hanhxiao)
• [6a368335] - cli: show more detailed version info in cli (hanhxiao)

📗 Documentation

• [d60a24a9] - fix docs format (hanhxiao)

🍹 Other Improvements

• [3cc4b041] - fix unit test (felix)
• [57519198] - changelog: update change log to v0.0.43 (hanhxiao)

Release Note (v0.0.43)

Release time: 2019-09-30 17:37:58

🙇 We'd like to thank all contributors for this new release! In particular, felix, hanhxiao, raccoonliukai, 🙇

🆕 New Features

• [bbf1ed8e] - frontend: add max pending request to frontend (hanhxiao)
• [946df39b] - indexer: delay the num_dim spec on first add (hanhxiao)
• [fdc38d57] - cd: trigger benchmark in push and tag (hanhxiao)
• [f8b9e00e] - cd: smaller num document for benchmarking (hanhxiao)
• [3974a1ba] - cd: adding benchmark to cd pipeline (hanhxiao)
• [086a73cb] - proto: add vcs version to pb (hanhxiao)
• [285d9dde] - docker: add vcs version as env var (hanhxiao)
• [a3e22db3] - service: add healthcheck for arbitary service (hanhxiao)
• [dedb8ba2] - proto: add ready status for healthcheck (hanhxiao)

🐞 Bug fixes

• [47add702] - ffmpeg-threads: threads=0 (felix)
• [9365ddb9] - video-decoder: minor revision video-decoder chunk spliter (felix)
• [be09bb09] - cd: fix duplicate step name in cd (hanhxiao)
• [ccf4efc8] - cd: fix trigger in cd pipeline (hanhxiao)
• [ca73b702] - shotdetect: support get arguments from yaml (raccoonliukai)
• [17aa78da] - shotdetect: fix bug with thre_algo after histcmp (raccoonliukai)
• [a6d1484e] - shotdetect: fix bug with thre_algo (raccoonliukai)
• [b69591de] - shotdetect: fix shot boundary (raccoonliukai)
• [f6c263a7] - ffmpeg: use -threads = 1 for ffmpeg (felix)
• [beafdb3a] - docker: fix vcs ref url and add build date as env (hanhxiao)
• [0367334a] - service: fix error message (hanhxiao)
• [434bc8db] - service: styling (hanhxiao)

🚧 Code Refactoring

• [e588c946] - encoder: update the init func for flair (hanhxiao)
• [1b85375c] - client: remove benchmark client (hanhxiao)
• [535085ef] - video-decoder: update video decoder signature and add more options (felix)
• [aa0f994e] - cli: refactor parser for healthcheck (hanhxiao)

📗 Documentation

• [9ad7448f] - add env vars explain (hanhxiao)

🍹 Other Improvements

• [e180a9f0] - fix error (felix)
• [074eec0d] - changelog: update change log to v0.0.42 (hanhxiao)

Release Note (v0.0.42)

Release time: 2019-09-26 10:40:48

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, felix, raccoonliukai, Jem, 🙇

🆕 New Features

• [8bef90dd] - cd: docker images now push to github during merge and tag (hanhxiao)
• [0ea566ff] - service: send ndarray separately (hanhxiao)
• [09199d82] - preprocessor: add frame selector (Jem)
• [a2f10589] - parser: add raw_bytes_in_sep to cli (hanhxiao)
• [10788951] - proto: speedup send/recv by separating raw_bytes from pb (hanhxiao)
• [2326fe97] - preprocessor: add preprocessor for mp4 and gif decode (raccoonliukai)
• [803afb34] - snoflake-uuid: add snowflake uuid generator (felix)
• [fe7025f5] - frontend: dump route in the frontend (hanhxiao)
• [8fbb0945] - router: add a block router for benchmarking (hanhxiao)

🐞 Bug fixes

• [43145019] - unittest: fix unit test for send recv (hanhxiao)
• [b6f2cdaf] - service: fix send/recv for better compatability (hanhxiao)
• [8c6f2558] - fix route table sum time (raccoonliukai)
• [8a0beec8] - service: send single long message rather than multiple (hanhxiao)
• [3b1f963c] - preprocessor: add solution for raw_video (raccoonliukai)
• [1b4a04fe] - preprocessor: add videodecode in init (raccoonliukai)
• [7108460a] - memory_leak: try to fix memory leak (felix)
• [82951d95] - frontend: use poll for better efficiency (hanhxiao)
• [2f539b7a] - snowflake: fix error shift (felix)
• [84e67792] - frontend: fix progressbar and route table (hanhxiao)
• [9a65e4fe] - frontend: flush dump (hanhxiao)
• [2e326af5] - catch exception in hook function (hanhxiao)
• [402867cc] - fix route table total time (hanhxiao)
• [30976179] - docker: decoupling prerequest and gnes install (hanhxiao)
• [c5347a5b] - service: make route_table as option for all services (hanhxiao)
• [45a078d9] - docker: reduce the size of built image (hanhxiao)

📗 Documentation

• [95493b47] - readme: add github package to readme (hanhxiao)

🍹 Other Improvements

• [31f53bc2] - fix unit test (felix)
• [89945ef8] - changelog: update change log to v0.0.41 (hanhxiao)

Release Note (v0.0.41)

Release time: 2019-09-20 20:51:39

🙇 We'd like to thank all contributors for this new release! In particular, felix, raccoonliukai, hanhxiao, Jem, 🙇

🆕 New Features

• [e255bd48] - shot-detector: limit number of frames in shots (felix)
• [6e87afa4] - traffic-controller: network traffic controller in frontend (felix)
• [6833a27c] - preprocessor: add sframes for shots frame number (raccoonliukai)
• [ea89d8cb] - stream-call: only 1000 pending tasks (felix)
• [ca53c65f] - video-encoder: encode video from list of images (felix)

🐞 Bug fixes

• [6aa0c3ca] - ffmpeg-video: fig bug for scaling videos to stdout (felix)
• [780aad0d] - subprocess: close stdout and stderr to avoid memory leak (felix)
• [205962fb] - socket: raise socket rec/send message exception (felix)
• [64acb4cd] - preprocessor: fix bug when num_frames < 4 in shotdetect (raccoonliukai)
• [05db02f7] - stream-client: request queue size is limited by 1000 (felix)
• [4f389449] - socket-buffer: set hwm and buffer limit for zmq socket (felix)
• [092379e1] - preprocessor: fix type of index in shotdetect (raccoonliukai)
• [9023afcd] - test: add test to cover three runtimes (hanhxiao)
• [65fff1a9] - cli: fix progressbar (hanhxiao)
• [8828535c] - proto: fix version check in recv message (hanhxiao)

🚧 Code Refactoring

• [d7c65727] - scorefn: move query coord into doc indexer (Jem)
• [b4007ba6] - frontend: realtime response to reduce network overload (felix)
• [de5b3368] - shot-detector: merge code from hub (felix)

🍹 Other Improvements

• [6acff20b] - clean unused variable (felix)
• [373df7da] - changelog: update change log to v0.0.40 (hanhxiao)

Release Note (v0.0.40)

Release time: 2019-09-12 19:54:34

🙇 We'd like to thank all contributors for this new release! In particular, Han Xiao, Jem, hanhxiao, felix, 🙇

🆕 New Features

• [b5e10bcd] - scorefn: add offset tfidf bm25 qureycoord (Jem)
• [20e7db6b] - cli: show default value of each argument (hanhxiao)
• [c6960b52] - progressbar: add dynamic progress bar (felix)
• [6c8017c1] - frontend: new route time table (hanhxiao)
• [81639268] - frontend: add route time table (hanhxiao)

🐞 Bug fixes

• [8704331e] - proto: fix version check in recv message (hanhxiao)
• [563a48c7] - cli: fix cli client required (hanhxiao)
• [3db34449] - proto: fix merge route logic (hanhxiao)
• [c31f21db] - parser: fix default dump interval to 5 (hanhxiao)
• [00c25f39] - parser: remove limite on message size (hanhxiao)
• [f89b4363] - parser: set dump_interval to -1 (hanhxiao)

🚧 Code Refactoring

• [a7a0ec8b] - grpc-client: add response handler (felix)
• [ce43be80] - proto: remove num_replicas from routes (hanhxiao)
• [d007fd9b] - indexer: remove unused code in leveldbindexer (hanhxiao)

📗 Documentation

• [787d0f14] - readme: add poem search demo (hanhxiao)

🏁 Unit Test and CICD

• [3af95227] - cli: add unittest for cli (Jem)

🍹 Other Improvements

• [0b22029a] - indexer: fix styles in indexer (Han Xiao)
• [edba197a] - clean and format codes (felix)
• [2a781aee] - license: remove aiohttp from barebone GNES license (hanhxiao)
• [cc72cf2b] - docker: revert alpine docker to reduce size (hanhxiao)
• [9f58fb35] - changelog: update change log to v0.0.39 (hanhxiao)

Release Note (v0.0.39)

Release time: 2019-09-11 17:22:11

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, Jem, raccoonliukai, Larry Yan, 🙇

🆕 New Features

• [fdb53a09] - client: add status logging (hanhxiao)
• [621c06b3] - client: abstract query_callback (hanhxiao)
• [2c2a4d83] - indexer: add indexer info (Jem)
• [cfe97aab] - proto: add version info to proto (hanhxiao)
• [f1a187c1] - proto: versioning protobuf and validating in recv_message (hanhxiao)

🐞 Bug fixes

• [31c796d3] - client: fix weights in helper indexer (hanhxiao)
• [21c3a8a9] - client: rename stub to _stub (hanhxiao)
• [235d901a] - parser: add max_concurrency to client (hanhxiao)
• [c988b327] - client: fix sync client (hanhxiao)
• [54a252e5] - indexer: add helper indexer to registeration (hanhxiao)
• [a1aed8f4] - client: use StreamingClient as the parent class of CLIClient (raccoonliukai)
• [a5999828] - preprocessor: add vframe(frame_num) for video and gif frames capture (raccoonliukai)
• [8357754a] - encoder: fix PCAEncoder mean from fp64 to fp32 (raccoonliukai)
• [654a5ba4] - encoder: fix vlad to speed up centroids calculation (Larry Yan)
• [814b2ee6] - encoder: fix vald encocer (Larry Yan)
• [ffc822b3] - encoder: fix vlad unittest (Larry Yan)
• [ddf13ff1] - encoder: fix bug in vlad encoder (Larry Yan)
• [1ba4e11c] - encoder: fix vald encoder and add unittest (Larry Yan)
• [f8e18d06] - encoder: fix vald in numeric encoder (Larry Yan)
• [fbfa1e47] - transformer: add model eval (Jem)

🚧 Code Refactoring

• [421e21f7] - indexer: add counter and move key_only to chunk helper (hanhxiao)
• [2a3f1210] - router: refactor base embed reduce router (Jem)

📗 Documentation

• [d4f59211] - update component overview (hanhxiao)

🍹 Other Improvements

• [ded79bee] - changelog: update change log to v0.0.38 (hanhxiao)

Release Note (v0.0.38)

Release time: 2019-09-06 17:25:48

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, 🙇

🍹 Other Improvements

• [71a3073a] - bump version (hanhxiao)
• [9666a462] - changelog: update change log to v0.0.37 (hanhxiao)

Release Note (v0.0.37)

Release time: 2019-09-06 16:46:20

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, felix, Jem, raccoonliu, raccoonliukai, Han Xiao, Larry Yan, 🙇

🆕 New Features

• [105a0abf] - encoder: add debug hook (hanhxiao)
• [0f04877f] - service: add pre and post hooks to baseservice (hanhxiao)
• [92860848] - reducer: add concat reducer (Jem)
• [2e6e80db] - encoder: add PCAEncoder support in gnes buster image (raccoonliukai)
• [16fa80bd] - tests: add unittest for PCAEncoder (raccoonliukai)
• [5a745b1e] - tests: add unittest for EncoderService and IndexerService (raccoonliukai)
• [a0fec684] - service: logging elapsed time and body type change (hanhxiao)
• [57cc95ff] - encoder: add quantizer (Jem)
• [00e6280d] - score_fn: use numpy for score fn (hanhxiao)
• [201c27e7] - cli: add --sorted_response as cli argument (hanhxiao)
• [81b21093] - index: move sort logic to base (hanhxiao)
• [a2d55dda] - index: move sort logic out to base (hanhxiao)
• [674a9da2] - encoder: add lab video model (Jem)
• [50a944b6] - encoder: add yt8m feature extractor (Jem)
• [f908f381] - score_fn: make score_fn as a TrainableBase (hanhxiao)
• [14c7e522] - score_fn: make score_fn dumpable (hanhxiao)
• [0b78798d] - score_fn: add score_fn as a new module (hanhxiao)
• [da56544f] - encoder: add PCAEncoder for incremental pca training (raccoonliukai)
• [97bb6de2] - lab encoder: add vggish for audio (Jem)
• [8cdcb7e8] - chunk scorer: add offset divergence (Jem)

🐞 Bug fixes

• [d404b8a7] - tests: use lowercase for true (raccoonliu)
• [bb9bbe9d] - tests: modify EncoderService unittest (raccoonliukai)
• [cd53a24b] - indexer: fix numpy indexer (hanhxiao)
• [d70e877e] - shot-detector: fix case of only one shot in video (felix)
• [e631d396] - service: indexer service return empty when no chunk (hanhxiao)
• [67b211da] - encoder: remove image resize from TFInceptionEncoder (raccoonliukai)
• [40849abc] - indexer: fix is_sorted in response flush away the request (hanhxiao)
• [ab819387] - ffmpeg: use tempfile as input instead of pipe (felix)
• [a8d2acfd] - service: is input list is false when query (Jem)
• [ba21c4e7] - service: fix bug for doc type in encoder (Larry Yan)
• [a4658250] - scorer: fix np float conversion (hanhxiao)
• [2d6c70fc] - indexer: fix vec np.concat (hanhxiao)
• [2ba135db] - indexer: fix empty chunks indexing (hanhxiao)
• [40dd1d5a] - encoder: fix embed_chunks_in_docs function (hanhxiao)
• [d94329b3] - preprocess: fix offset in sentence splitter (hanhxiao)

🚧 Code Refactoring

• [a8e87d9f] - service: minimize event loop, move handling to handler (hanhxiao)
• [06aab813] - grpc-client: implement async client via multi-threaded (felix)
• [35fa3ba4] - pb: remove unused field (hanhxiao)
• [6bbfc993] - score_fn: rename score functions (hanhxiao)
• [e9feaa61] - score_fn: use post_init instead of property (hanhxiao)
• [f406f8f0] - score_fn: move normalize_fn and score_fn to the init (hanhxiao)

🍹 Other Improvements

• [c988c7dd] - remove sklearn from base dep (felix)
• [2fd8dab4] - minor fix on the styling (Han Xiao)
• [066d6e99] - pb2: do not reformat pb2 generated python file (hanhxiao)
• [1bbc4358] - service: revert encoder service (hanhxiao)
• [b97b718f] - changelog: update change log to v0.0.36 (hanhxiao)

Release Note (v0.0.36)

Release time: 2019-08-30 17:32:23

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, Jem, felix, 🙇

🆕 New Features

• [07534f89] - score: improve score explain for better interpretability (hanhxiao)
• [32d815d7] - filesys: keep doc meta info (felix)
• [92fc3d8c] - scale_video: keep ratio of video frame (felix)

🐞 Bug fixes

• [f1402f50] - cli: fix cli chanel close (hanhxiao)
• [b140cca9] - service: fix exception when no chunks (hanhxiao)
• [cee99a63] - logger: change the color semantic for loglevel (hanhxiao)
• [4efea726] - service: raise except when empty chunk (hanhxiao)
• [31bffeb7] - preprocessor: add min_len to split preprocessor (hanhxiao)
• [7b16354a] - style: fix style issues (hanhxiao)
• [c6183960] - service: fix training logic in encoderservice (hanhxiao)
• [5828d20a] - preprocessor: fix SentSplitPreprocessor (hanhxiao)
• [522c5a4e] - preprocessor: rename SentSplitPreprocessor (hanhxiao)
• [030d6c66] - setup: fix path in setup script (hanhxiao)
• [3818c9a3] - test: fix router tests (hanhxiao)
• [9d03441e] - proto: regenerate pb2 (hanhxiao)
• [f49f9a5b] - indexer: fix parsing in DictIndexer (hanhxiao)
• [0215c6bf] - ffmpeg: fix issue for start and durtion argument position (felix)
• [a735a719] - service: log error in base service (hanhxiao)
• [3263e96c] - service: move py_import from service manager to base service (hanhxiao)
• [990c879d] - client: fix client progress bar, http (hanhxiao)
• [d02cd757] - router: respect num_part when set (hanhxiao)
• [a76a4604] - ffmpeg-video: fig bug for scaling videos to stdout (felix)

🚧 Code Refactoring

• [42e7c13b] - indexer: separate score logic and index logic (hanhxiao)
• [0c6f4851] - preprocessor: use io utils in audio and gif (Jem)
• [bae75b8c] - router: separate router and scoring logics (hanhxiao)
• [c3ebb93a] - proto: refactor offset nd (Jem)
• [e3bbbd9b] - shot_detector: update ffmpeg api (felix)
• [10cef54e] - ffmpeg: refactor ffmpeg again (felix)

🏁 Unit Test and CICD

• [1e9ef35c] - pipeline: test pipeline load from yaml (hanhxiao)
• [5b7c9f19] - pipeline: add unit test for pipeline encoder (hanhxiao)
• [bef2bf9c] - indexer: add unit test for dict indexer as service (hanhxiao)
• [620cf3bd] - ffmpeg: add unittest for ffmpeg api (felix)

🍹 Other Improvements

• [c83448b5] - license: add license header to frontend (hanhxiao)
• [04deea3a] - license: remove unrelevant packages from license (hanhxiao)
• [10f4bedb] - fix error (felix)
• [4aa997c3] - changelog: update change log to v0.0.35 (hanhxiao)

Release Note (v0.0.35)

Release time: 2019-08-26 18:15:02

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, Jem, 🙇

🆕 New Features

• [b4444cc0] - encoder: separate pooling as an indep. encoder (hanhxiao)
• [ce0e65ae] - helper: batching decorator supports tuple (hanhxiao)
• [a584c7e5] - helper: add as_numpy_array decorator (hanhxiao)

🐞 Bug fixes

• [ff7926d8] - encoder: fix eager execution (hanhxiao)
• [27a7ca8c] - cli: add a small jitter to prevent div zero (hanhxiao)

🚧 Code Refactoring

• [39561000] - base: add on_gpu to replace use_cuda (hanhxiao)
• [928574cd] - encoder: replace gpt and elmo with transformer (hanhxiao)
• [52538276] - encoder: no for loop in torch encoder now (Jem)
• [7493af97] - preprocessor: add init, change signiture (Jem)

🍹 Other Improvements

• [20249afb] - changelog: update change log to v0.0.34 (hanhxiao)

Release Note (v0.0.34)

Release time: 2019-08-23 19:00:27

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, 🙇

🍹 Other Improvements

• [79a8effd] - changelog: update change log to v0.0.33 (hanhxiao)

Release Note (v0.0.34)

Release time: 2019-08-23 18:44:34

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, 🙇

🍹 Other Improvements

• [79a8effd] - changelog: update change log to v0.0.33 (hanhxiao)

Release Note (v0.0.33)

Release time: 2019-08-23 18:34:28

🙇 We'd like to thank all contributors for this new release! In particular, Jem, hanhxiao, felix, raccoonliukai, 🙇

🆕 New Features

• [9d488e3f] - client: add progress bar and speed metric to cli (hanhxiao)
• [829d148c] - scale_video: scale video use ffmpeg (felix)
• [bc2e441d] - compose: add minimum http server without flask dep (hanhxiao)
• [d420f348] - video preprocessor: add edge detect for shotdetect (raccoonliukai)

🐞 Bug fixes

• [6cfbda9d] - preprocessor: move dependency into function (Jem)
• [0e88b77a] - frontend: fix request_id zero is none (hanhxiao)
• [ca28ecb9] - video preprocessor: use rgb as standard color (raccoonliukai)
• [5b5feb0b] - video preprocessor: use dict update (raccoonliukai)
• [47721b1c] - video preprocessor: remove custom canny threshold (raccoonliukai)
• [16aaa777] - video preprocessor: modify inaccurate names (raccoonliukai)
• [dfb54b62] - video preprocessor: Remove incorrect comments (raccoonliukai)

🚧 Code Refactoring

• [3d63fac6] - proto: request_id is now an integer (hanhxiao)
• [4497d765] - shotdetector: use updated ffmpeg api to capture frames from videos (felix)
• [dbc06a85] - ffmpeg: refactor ffmpeg to read frames, vides and gif (felix)
• [a7b12cb6] - preprocessor: add gif chunk prep (Jem)
• [559a9971] - compose: unify flask and http handler (hanhxiao)

📗 Documentation

• [a2801d5c] - link gnes hub tutorial to readme (hanhxiao)

🍹 Other Improvements

• [02f70a03] - fix bug (felix)
• [c970bec3] - changelog: update change log to v0.0.32 (hanhxiao)

Release Note (v0.0.32)

Release time: 2019-08-21 17:23:13

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, Han Xiao, Jem, 🙇

🆕 New Features

• [38567b00] - indexer: add preprocessor and lvdb for storing gif (Jem)
• [35465e85] - base: later import module now override the earlier ones (hanhxiao)

🐞 Bug fixes

• [5c2b60a4] - remove target_image_size (hanhxiao)
• [944b8c09] - ci: fix unit tests for modules (hanhxiao)

🚧 Code Refactoring

• [5f1ca000] - fixing the imports of all base module (hanhxiao)
• [0d1bd4e2] - preprocessor: remove unnecessary init (Han Xiao)

🏁 Unit Test and CICD

• [3820db6a] - encoder: rename BasePytorchEncoder to TorchvisionEncoder (hanhxiao)

🍹 Other Improvements

• [3147a1d5] - changelog: update change log to v0.0.31 (hanhxiao)

Release Note (v0.0.31)

Release time: 2019-08-20 14:01:04

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, 🙇

🆕 New Features

• [f7beae7b] - cli: add py_path in parser to load external modules (hanhxiao)

🍹 Other Improvements

• [ec1eb787] - release: fix duplicate release notes (hanhxiao)
• [447756d5] - changelog: update change log to v0.0.30 (hanhxiao)

Release Note (v0.0.30)

Release time: 2019-08-19 14:13:03

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, 🙇

🆕 New Features

• [7b5cc86a] - contrib: no need to give module name in advance (hanhxiao)

🐞 Bug fixes

• [5f69c781] - contrib: allowing dump for contribued module (hanhxiao)

🍹 Other Improvements

• [565ef569] - changelog: update change log to v0.0.29 (hanhxiao)

Release Note (v0.0.29)

Release time: 2019-08-16 15:40:31

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, Jem, 🙇

🐞 Bug fixes

• [2f905168] - setup: fix encoding problem in setup.py (hanhxiao)
• [469bc51d] - ci: fix cd pipeline (hanhxiao)

🚧 Code Refactoring

• [66d020bd] - base: component renamed to components (hanhxiao)
• [3a2b85b6] - proto: assign doc id in request generator (Jem)

📗 Documentation

• [b854c697] - readme: fix description on images (hanhxiao)

🏁 Unit Test and CICD

• [f1658c92] - docker: docker image tag-alpine as default tag (hanhxiao)
• [8885512f] - docker: clean up the space after docker build (hanhxiao)

🍹 Other Improvements

• [00ca6919] - changelog: update change log to v0.0.28 (hanhxiao)

Release Note (v0.0.28)

Release time: 2019-08-14 20:54:26

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, Jem, raccoonliukai, Larry Yan, 🙇

🆕 New Features

• [0133905c] - client: add a client for benchmarking and testing (hanhxiao)
• [732f2e64] - encoder: add pytorch transformers support in text encoder (raccoonliukai)
• [6aab48c8] - docker: add buster image with minimum dependencies (hanhxiao)
• [da1bbc0d] - docker: add alpine image with minimum dependencies (hanhxiao)

🐞 Bug fixes

• [315bd16a] - doc sum router: use meta info instead of doc id to do doc sum (Jem)
• [c9e92722] - encoder: use offline model in ci-base for pytorch transformer (raccoonliukai)
• [d7b42d39] - setup: remove unused dependencies (hanhxiao)
• [5b8acf7c] - test: fix routes assert in tests (hanhxiao)
• [5fedf6df] - encoder: fix unused variable (raccoonliukai)
• [df616463] - cli: remove unnecessary argument (hanhxiao)
• [fd76aa79] - request_generator: send index request in index mode (Jem)
• [64163cb1] - batching: enable to process three dimension output in batching (Jem)
• [415456d6] - preprocessor: fix bug (Larry Yan)
• [c150ad59] - preprocessor: modify ffmpeg video pre add video cutting method (Larry Yan)
• [b0f22d04] - audio preprocessor: filter audio with zero length (Jem)
• [d1cfa539] - preprocessor: modify ffmpeg video preprocessor (Larry Yan)

📗 Documentation

• [e11a920e] - readme: add image explain to readme (hanhxiao)

🏁 Unit Test and CICD

• [a8700801] - drone: add self hosted drone (hanhxiao)
• [079d0a1a] - docker: move docker-build to a more controllable cd process (hanhxiao)

🍹 Other Improvements

• [5257259f] - add kai liu to core maintainers (hanhxiao)
• [8d318204] - changelog: update change log to v0.0.27 (hanhxiao)

Release Note (v0.0.27)

Release time: 2019-08-09 19:51:57

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, Jem, Larry Yan, raccoonliu, Han Xiao, raccoonliukai, 🙇

🆕 New Features

• [55126f2b] - grpc: add a general purpose grpc service (hanhxiao)
• [23c6e68a] - reduce router: add chunk and doc reduce routers for audio (Jem)
• [6d3d2b4c] - cli: use ServiceManager as default service runner (hanhxiao)
• [ccfd474a] - service: add ServiceManager and enable parallel services in one container (hanhxiao)
• [63f9173f] - service: enabling the choose of thread or process as the backend (hanhxiao)
• [2647b848] - audio: add preprocess and mfcc encoder for audio (Jem)
• [208e1937] - audio: add preprocess and mfcc encoder for audio, update protobuf (Jem)
• [77a2ea42] - parser: improve yaml_path parsing (hanhxiao)
• [762535ca] - vlad: add vlad and enable multiple chunks and frames (Jem)
• [64e948d4] - encoder: add onnxruntime for image encoder (raccoonliukai)
• [f03e6fc2] - encoder: add onnxruntime suport for image encoder (raccoonliukai)

🐞 Bug fixes

• [5ae46d61] - composer: rename grpcfrontend to frontend (hanhxiao)
• [4cb83383] - audio: restrict max length for mfcc encoding (Jem)
• [e516646f] - grpc: add max_message_size to the argparser (hanhxiao)
• [0493e6fc] - encoder: fix netvlad (Larry Yan)
• [e773aa33] - service manager: fix nonetype for service manager (Jem)
• [d5d15d7f] - compose: fix a bug in doc_reduce_test (hanhxiao)
• [6856cb0a] - compose: copy args on every request (hanhxiao)
• [f80e8c03] - cli: set default num_part is None (hanhxiao)
• [7031fe20] - preprocessor: add random sampling to ffmpeg (Larry Yan)
• [fd37e6d9] - encoder: fix bug caused by batching in inception_mixture (Larry Yan)
• [2191b27b] - composer: fix yaml generation (hanhxiao)
• [e5fefcee] - encoder: fix batching in encoder (hanhxiao)
• [e35e3b3c] - composer: fix composer router generation logic (hanhxiao)
• [7300e055] - preprocessor: quanlity improvement (Larry Yan)
• [47efaba4] - unittest: fix unittest of video preprocessor 2 (Larry Yan)
• [a6efb4af] - unittest: fix unittest of video preprocessor (Larry Yan)
• [dd1216bb] - unittest: fix unittest for video processor (Larry Yan)
• [8e6dc4c6] - encoder: add func for preprocessor (Larry Yan)
• [2b21dc5a] - encoder: fix unused import and variable (raccoonliu)
• [fd576915] - test: fix import (Han Xiao)
• [a0fdad36] - test: fix broken code (Han Xiao)
• [8ca07a74] - test: fix img_process_for_test (Han Xiao)
• [7c16fb8b] - preprocessor: fix bug in ffmpeg.py and add more func to helper (Larry Yan)
• [e6a37119] - preprocessor: fix bug in params in ffmepg (Larry Yan)
• [f8d2abe5] - preprocessor: fix bug in ffmpeg (Larry Yan)
• [67610f86] - preprocessor: add more method for cutting video (Larry Yan)

🚧 Code Refactoring

• [8516096d] - grpc: moving zmqclient to client module (hanhxiao)
• [5e3409e1] - grpc: hide private class inside gRPCfrontend (hanhxiao)
• [6407cc8d] - yaml: remove useless default yaml config (hanhxiao)
• [c1e406ae] - onnx: move batch_size to class attribute (Han Xiao)

🏁 Unit Test and CICD

• [5503dbe7] - skip joint indexer test as it is not even used (hanhxiao)
• [8ab101ca] - add mergify for auto merging (hanhxiao)
• [203d1697] - chore: exclude chore job from ci pipeline (hanhxiao)
• [24f9fd1c] - fix yaml_path missing in the test (hanhxiao)
• [23a83a40] - simplify yaml naming (hanhxiao)

🍹 Other Improvements

• [e8e3b9b9] - changelog: update change log to v0.0.26 (hanhxiao)

Release Note (v0.0.26)

Release time: 2019-08-02 18:18:45

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, Jem, Larry Yan, 🙇

🆕 New Features

• [d0b2ef0b] - composer: more interaction for gnes board (hanhxiao)
• [9c33dc66] - router: allow consecutive mapping and reducing ops (hanhxiao)

🐞 Bug fixes

• [fc5026da] - board: improve gnes board 500 message (hanhxiao)
• [823bdeda] - test: fix grpc gentle shutdown (hanhxiao)
• [f6a801f7] - test: fix preprocessor building for image test (hanhxiao)
• [50fdc041] - base: fix ref to CompositionalTrainableBase (hanhxiao)
• [54a931c7] - test: fix test images by removing mac stuff (hanhxiao)
• [14cdfabe] - sliding window: fix the boundary (Jem)
• [46b5c94e] - encoder: fix name for video encoder (Larry Yan)
• [15eb50b4] - encoder: fix params in basevideo encoder (Larry Yan)
• [5b0fe7c6] - preprocessor: fix FFmpegVideoSegmentor (Larry Yan)
• [d6a46fa6] - encoder: fix import path for mixture encoder (Larry Yan)
• [17779676] - encoder: fix mixture encoder (Larry Yan)
• [95f03c56] - encoder: fix bug in video mixture encoder (Larry Yan)
• [3fdf1c06] - encoder: fix mixture (Larry Yan)
• [67991533] - encoder: add netvlad and netfv register class (Larry Yan)
• [92500f0f] - encoder: add netvlad and netfv (Larry Yan)

🚧 Code Refactoring

• [c430ef64] - base: better batch_size control (hanhxiao)
• [58217d8c] - base: moving is_trained to class attribute (hanhxiao)
• [7126d496] - preprocessor: separate resize logic from the unary preprocessor (hanhxiao)
• [52f87c7f] - base: make pipelineencoder more general and allow pipelinepreprocessor (hanhxiao)

📗 Documentation

• [3ab3723e] - tutorial: fix image and code layout (hanhxiao)

🍹 Other Improvements

• [635ba37f] - changelog: update change log to v0.0.25 (hanhxiao)

Release Note (v0.0.25)

Release time: 2019-07-26 19:45:21

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, felix, Larry Yan, Jem, Han Xiao, Felix, 🙇

🆕 New Features

• [66aec9c9] - grpc: add StreamCall and decouple send and receive (hanhxiao)
• [5697441b] - indexer: consider offset relevance at query time (Jem)
• [04c9c745] - image preprocessor: calculate offsetnd for each chunk (Jem)
• [b34a765a] - compose: add interactive mode of GNES board using Flask (hanhxiao)
• [5876c15e] - base: support loading external modules from py and yaml (hanhxiao)

🐞 Bug fixes

• [a20672d3] - preprocessor: add logging in helper module (felix)
• [f9500c1f] - protobuffer: add doc_type as func argument in RequestGenerator (felix)
• [1c3bb01a] - service: fix bug in doc_type name in indexer service (Larry Yan)
• [d834f578] - service: add doc type to req generator (Larry Yan)
• [80e234e1] - service: fix bug in req Generator add doc_type (Larry Yan)
• [5743e258] - indexer: fix bug in indexer service (Larry Yan)
• [11dde2bf] - encoder: fix bug in tf inception (Larry Yan)
• [ded92c57] - indexer: fix bug for indexer service dealing with empty doc (Larry Yan)
• [1dff06f1] - encoder: fix bug for encoder service dealing with empty doc (Larry Yan)
• [7e43d5a2] - preprocessor: fix ffmpeg to deal with broken image (Larry Yan)
• [83ebaced] - preprocessor: move import imagehash to inside (hanhxiao)
• [7c669a70] - test: rename the yaml test file (hanhxiao)
• [2cc26342] - compose: change textarea font to monospace (hanhxiao)
• [e644e391] - encoder: fix gpu limitation in inception (Larry Yan)
• [89d8b70c] - grpc: fix bug in RequestGenerator query (Larry Yan)
• [c52c2cc6] - base: fix gnes_config mixed in kwargs (hanhxiao)
• [68c15fac] - base: fix redundant warning in pipeline encoder (hanhxiao)
• [aadeeefb] - composer: fix composer state machine (hanhxiao)
• [c0bffe6c] - indexer: normalize weight (Jem)
• [2c696483] - indexer: fix weight in indexer call (Larry Yan)
• [139a02d9] - compose: fix compose bug of pub-sub rule, duplicate yaml_path (hanhxiao)
• [649ed131] - encoder: add normalize option in cvae encoder (Larry Yan)
• [eb487799] - encoder: fix tf scope error in cvae encoder (Larry Yan)
• [ab6c88cc] - encoder: fix error in cvae encoder (Larry Yan)
• [a4b883ac] - indexer: add drop raw bytes option to leveldb (Larry Yan)
• [4b52bcba] - grpc: fix grpc plugin path (Larry Yan)
• [d3fbbcac] - weighting: add simple normalization to chunk search (Jem)
• [08a9a4e3] - grpc: fix grpc service (Larry Yan)
• [6e6bbf83] - grpc: add auto-gen grpc code (Larry Yan)
• [b89d8fa2] - grpc: add stream index and train in proto (Larry Yan)
• [15cd7e58] - base: fix dump and load on compositional encoder (hanhxiao)
• [bab48919] - encoder: fix tf inception (Larry Yan)
• [973672ef] - encoder: fix bug for encoder bin load (Larry Yan)
• [1bef3971] - setup: fix setup script (hanhxiao)
• [67fb5766] - compose: fix argparser (hanhxiao)
• [63c4515f] - compose: accept parser argument only (hanhxiao)
• [887d89cc] - release: ask BOT_URL before releasing (hanhxiao)

🚧 Code Refactoring

• [9973f600] - preprocessor: rename singleton to unary (hanhxiao)
• [a1a2b020] - proto: refactor request stream call (hanhxiao)

📗 Documentation

• [c853e3da] - tutorial: fix svg size (hanhxiao)
• [04cccdcd] - tutorial: fix svg path (hanhxiao)
• [8927cd4f] - tutorial: add yaml explain (hanhxiao)
• [5b52ce4c] - fix doc path (hanhxiao)
• [45751e1f] - readme: add quick start for readme (hanhxiao)
• [73891ecc] - readme: add install guide to readme and contribution guide (hanhxiao)

🏁 Unit Test and CICD

• [6ff3079b] - unittest: skip all os environ test (hanhxiao)
• [816fa043] - unittest: skip blocked test (hanhxiao)
• [79a9c106] - unittest: run test in verbose mode (hanhxiao)
• [83276f90] - torchvision: install torchvision dependency to enable tests (hanhxiao)
• [499682ce] - base: add unit test for load a dumped pipeline from yaml (hanhxiao)
• [26a7ad18] - composer: add unit test for flask (hanhxiao)
• [87ec1fd2] - base: move module delete to teardown (hanhxiao)
• [479b183d] - compose: skip unit test (hanhxiao)

🍹 Other Improvements

• [c30f39cc] - ... (felix)
• [2d5654c0] - license: add license (hanhxiao)
• [d3347910] - reformat code and optimize import (hanhxiao)
• [71491ffb] - changelog: update change log to v0.0.24 (hanhxiao)

Release Note (v0.0.24)

Release time: 2019-07-19 18:18:46

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, Jem, Larry Yan, felix, 🙇

🆕 New Features

• [9f6c0524] - fasterrcnn: add the original image to chunk list (Jem)
• [abb0841c] - encoder: add convolution variational autoencoder (Larry Yan)

🐞 Bug fixes

• [1b526832] - base: fix dump yaml kwargs (hanhxiao)
• [086f3cea] - base: fix ump instance (hanhxiao)
• [12dfde42] - base: move name setting to trainable base (hanhxiao)
• [16f1a497] - base: move set config to metaclass (hanhxiao)
• [b97acd6c] - base: fix duplicate warning (hanhxiao)
• [991e4425] - base: fix duplicate load and init from yaml (hanhxiao)
• [69a486e5] - compose: fix import (hanhxiao)
• [4977aa3c] - vector indexer: reorder relevance and chunk weight (Jem)
• [2448411d] - encoder: modify CVAE (Larry Yan)
• [b4bf0bf8] - indexer: add path check for dir and file (hanhxiao)
• [92f36c33] - fasterrcnn: handle imgs with 0 chunk (Jem)
• [a1329913] - fasterrcnn: fix bug for gpu (Jem)
• [38eca0ce] - grpc: change grpc client message size limit (felix)
• [3836020a] - preprocessor: fix preprocessor service handler function name error (felix)
• [599a3c3d] - compose: fix composer logic (hanhxiao)
• [7f3b2fb5] - release: fix git tag version (hanhxiao)

🚧 Code Refactoring

• [9bbb3c05] - compose: move compose template to resources (hanhxiao)
• [a4e153d7] - base: remove dump path and reorganize work dir (hanhxiao)

🏁 Unit Test and CICD

• [e088ea9c] - drone: turn off profiling in ci (hanhxiao)
• [33a570b9] - drone: remove pylint for faster ci (hanhxiao)
• [51eafac7] - indexer: fix data path in unit test (hanhxiao)

🍹 Other Improvements

• [43ef4108] - git: add tmp to ignore (hanhxiao)
• [44b1a0c9] - fix unittest (felix)
• [984a9a2d] - changelog: update change log to v0.0.23 (hanhxiao)

Release Note (v0.0.23)

Release time: 2019-07-17 18:28:08

🙇 We'd like to thank all contributors for this new release! In particular, hanhxiao, Jem, felix, Larry Yan, Han Xiao, 🙇

🆕 New Features

• [cb4d9cf2] - release: add auto release and keep change log (hanhxiao)
• [c667d874] - image_preprocessor: add fasterRCNN (Jem)
• [a6c2975b] - composer: improve the gnes board with cards (hanhxiao)
• [6ec4233d] - composer: add swarm and bash generator (hanhxiao)
• [08aa30f4] - composer: add shell script generator (hanhxiao)
• [033a4b9c] - composer: add composer and mermaid renderer (hanhxiao)

🐞 Bug fixes

• [2b7c3f18] - compose: resolve unclosed file warning (hanhxiao)
• [8030feb2] - compose: fix router logic in compose (hanhxiao)
• [736f6053] - gnesboard: fix cdn (hanhxiao)
• [fb07ff02] - doc_reducer_router: fix reduce error (felix)
• [a7236308] - image encoder: define use_cuda variable via args (felix)
• [cba5e190] - image_encoder: enable batching encoding (felix)
• [3423ec83] - composer: add compose api to api.py (hanhxiao)
• [70ba3fca] - composer: in bash mode always run job in background (hanhxiao)
• [054981ce] - composer: fix gnes board naming (hanhxiao)
• [743ec3b0] - composer: fix unit test and add tear down (hanhxiao)
• [64aef413] - composer: fix styling according to codacy (hanhxiao)
• [dca4b03b] - service: fix bug grpc (Larry Yan)
• [09e68da2] - service: fix grpc server size limit (Larry Yan)
• [3da8da19] - encoder: rm un-used import in inception (Larry Yan)
• [8780a4da] - bugs for integrated test (Jem)
• [38fff782] - preprocessor: move cv2 dep to pic_weight (Han Xiao)
• [37155bba] - preprocessor-video: move sklearn dep to apply (Han Xiao)
• [1f6a06a2] - encoder: rm tf inception unittest (Larry Yan)
• [eaffbbff] - encoder: register tf inception in init (Larry Yan)
• [d0099b79] - encoder: add necessary code from tf (Larry Yan)
• [b480774a] - encoder: add inception tf (Larry Yan)

📗 Documentation

• [54276c6a] - readme: improve readme image and structure (hanhxiao)

🏁 Unit Test and CICD

• [1dcfdfa7] - docker-image: optimze docker file (felix)
• [bda562d1] - drone: auto release with cron job (hanhxiao)

🍹 Other Improvements

• [0c737a94] - release: revert back master check (hanhxiao)
• [7b04697c] - changelog: revert the change log to empty (hanhxiao)
• [d02f320d] - revert docker file (felix)