Skip to content

Commit

Permalink
[feat] support easyrec config convert to tzrec config (#37)
Browse files Browse the repository at this point in the history
  • Loading branch information
chengaofei authored Nov 21, 2024
1 parent 9fc05ac commit 7b331c2
Show file tree
Hide file tree
Showing 7 changed files with 1,435 additions and 1 deletion.
4 changes: 3 additions & 1 deletion .pyre_configuration
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@
"tzrec/*/*_test.py",
"tzrec/tests/*.py",
"tzrec/utils/load_class.py",
"tzrec/acc/_*.py"
"tzrec/acc/_*.py",
"tzrec/tools/convert_easyrec_config_to_tzrec_config.py",
"tzrec/*/*_test_tmp.py"
],
"site_package_search_strategy": "all",
"source_directories": [
Expand Down
1 change: 1 addition & 0 deletions .ruff.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ lint.ignore = ["D100", "D104", "D105", "D107"]

[lint.per-file-ignores]
"*_test.py" = ["D100", "D101", "D102", "D103", "D104", "D105", "D106", "D107"]
"*_test_tmp.py" = ["D100", "D101", "D102", "D103", "D104", "D105", "D106", "D107"]

[lint.pydocstyle]
convention = "google"
1 change: 1 addition & 0 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ Welcome to TorchEasyRec's documentation!
usage/predict
usage/serving
usage/feature_selection
usage/convert_easyrec_config_to_tzrec_config

.. toctree::
:maxdepth: 1
Expand Down
30 changes: 30 additions & 0 deletions docs/source/usage/convert_easyrec_config_to_tzrec_config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# EasyRec迁移TorchEasyRec

推荐模型一般特征和模型配置较为复杂,TorchEasyRec提供了配置转换工具,可以便捷地将EasyRec的配置文件转换为TorchEasyRec文件。

## 转换命令

torcheasyrec的pipeline.config包含了feature generate的配置,因此需要有easyrec训练使用的pipeline.config和fg.json两部分才可以转换为torcheasyrec的pipeline.config

```bash
PYTHONPATH=. python tzrec/tools/convert_easyrec_config_to_tzrec_config.py \
--easyrec_config_path ./easyrec.config \
--fg_json_path ./fg.json \
--output_tzrec_config_path ./tzrec.config
```

- --easyrec_config_path: easyrec训练使用的pipeline.config路径
- --fg_json_path: easyrec训练和推理使用的fg.json路径
- --output_tzrec_config_path: 生成tzrec的config路径

如果使用自定义的EasyRec安装包,应使用如下转换命令

```bash
EASYREC_URL=http://xxx.whl \
PYTHONPATH=. python tzrec/tools/convert_easyrec_config_to_tzrec_config.py \
--easyrec_config_path ./easyrec.config \
--fg_json_path ./fg.json \
--output_tzrec_config_path ./tzrec.config
```

- EASYREC_URL: 是http开头的EasyRec tar包或者whl的url地址
3 changes: 3 additions & 0 deletions tzrec/constant.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,6 @@ class Mode(Enum):
TRAIN = 1
EVAL = 2
PREDICT = 3


EASYREC_VERSION = "0.7.5"
Loading

0 comments on commit 7b331c2

Please sign in to comment.