Adding LoKrModel Class to paddle.peft library #9269
Conversation
|
Thanks for your contribution! |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #9269 +/- ##
===========================================
+ Coverage 53.02% 53.04% +0.02%
===========================================
Files 657 680 +23
Lines 106311 108195 +1884
===========================================
+ Hits 56367 57388 +1021
- Misses 49944 50807 +863 ☔ View full report in Codecov by Sentry. |
|
按照建议我已经提交了只有Linear Layer的LoKr实现供你们整体查看。麻烦有时间审阅下并给出需要修改的意见。 |
|
好的收到，我们这边会尽快review |
paddlenlp/peft/lokr/lokr_envs.py
Outdated
|
|
||
| # This module is set to be in alignment with code design paradiam of ...utils.env | ||
|
|
||
| LOKR_WEIGHTS_NAME = "lokr_model_state.pdparams" |
There was a problem hiding this comment.
paddlenlp/peft/lokr/__init__.py
Outdated
| @@ -0,0 +1,19 @@ | |||
| # Copyright 2023-present the HuggingFace Inc. team. | |||
paddlenlp/peft/lokr/lokr_model.py
Outdated
| def add_lora_split_mapping(self, module_name, is_column=False): | ||
| self.lora_split_mapping[module_name] = is_column | ||
|
|
||
| def _get_tensor_parallel_mappings(self, config, is_split=True): |
There was a problem hiding this comment.
没有使用tensor_parallel 和pipeline parallel先把没用到的相关的逻辑删掉
paddlenlp/peft/lokr/lokr_model.py
Outdated
| self.quantized = True | ||
| if lokr_module is None: | ||
| raise ValueError("LoKr strategy only supports paddle.nn.Linear right now") | ||
| if getattr(lokr_module, "quant_weight", None) is not None: |
| if any(isinstance(layer, lokr_layer) for lokr_layer in AVAILABLE_LAYERS): | ||
| layer.disable_lokr = True | ||
|
|
||
| def enable_lokr(self): |
There was a problem hiding this comment.
可以参考lora，在lora_layer中加入disable_lokr这个参数
|
辛苦了！第一次参与开源。可能错误较多，我会尽早修改问题，重新提交，供你们审阅。 Have a good day! |
感谢对PaddleNLP的贡献，我们非常欢迎社区开发者参与到PaddleNLP的开发中来。我会在重新提交代码后尽快进行review，期待提交的代码能早日合入到项目中！ |
|
可以再次review请告知我，我会尽快开始review |
按照要求我已经
目前我可以想到的接下来可以做的是：
|
PR types
New features
PR changes
Others
Description
Adding LoKrModel, LoKrLinear and LoKrConfig to support a new lora-like adapter. Current implementation only supports contains Linear Modules. Motivation and discussion on such PR issue is at: #9226
Please provide suggestions on the current implementation!