[Feat]: Unified LLM Calling Management

[drcege]

Search before continuing 先搜索，再继续

• I have searched the Data-Juicer issues and found no similar feature requests. 我已经搜索了 Data-Juicer 的 issue 列表但是没有发现类似的功能需求。

Description 描述

Currently, some LLM-dependent operators support vllm, while others utilize Hugging Face or the OpenAI API for model calling. It is necessary to review and unify these calling capabilities across the codebase.

Furthermore, could we abstract these calling mechanisms, rather than repeating similar code? This would enable unified management and ease the addition of support for more inference engines, such as custom Post APIs, TensorRT, and ONNX.

Use case 使用场景

No response

Additional 额外信息

No response

Are you willing to submit a PR for this feature? 您是否乐意为此功能提交一个 PR？

• Yes I'd like to help by submitting a PR! 是的！我愿意提供帮助并提交一个PR！