[Feature request] Custom Engine for Openrouter

[sonzin]

I tried using ChatGPT to help create a custom engine for OpenRouter, but I wasn't successful in my attempts. It would be fantastic if you could either integrate it directly or provide a script for a Custom Engine that can be used with OpenRouter's API, especially for people like me who aren't skilled in programming. This would be particularly helpful since OpenRouter offers many models that are more affordable compared to using the OpenAI API. Thank you very much!

[llkiriell]

Hi, buddy! I managed to create a custom engine with Groq and its API. At first, I wanted to create a new class in the "engine" folder called GroqTranslate, following the developer's pattern. However, since I’m not familiar with the entire project structure (and I don’t have much time), I modified the anthrophic.py file and the ClaudeTranslate class instead. I changed parameters such as the endpoint, the prompt, the headers of the HTTP request, and the response format. It's relatively simple since the developer of this plugin (I’m not judging you, my friend; I really appreciate you making this plugin—it's very useful) repeats code, and the classes in the LLM models are basically the same.

I also tried to create a custom engine using the "custom" option, but I encountered many errors. If you want to create your custom engine, follow these steps:

1. Access the plugin zip file.
2. Locate anthrophic.py.
3. Understand the functions there (they have very intuitive names) and modify them according to your needs for the HTTP request header and your response.

Here’s the code showing how the modification turned out; you can use it as a guide:

import json

from .. import EbookTranslator

from ..engines.base import Base
from ..engines.languages import google


try:
    from http.client import IncompleteRead
except ImportError:
    from httplib import IncompleteRead

load_translations()


class ClaudeTranslate(Base):
    name = 'Groq'
    alias = 'Groq (Groq)'
    lang_codes = Base.load_lang_codes(google)
    endpoint = 'https://api.groq.com/openai/v1/chat/completions'
    api_key_hint = 'sk-ant-xxxx'
    # https://docs.anthropic.com/claude/reference/errors
    api_key_errors = ['401', 'permission_error']

    concurrency_limit = 1
    request_interval = 12
    request_timeout = 30.0

    # prompt = (
    #     'You are a meticulous translator who translates any given content. '
    #     'Translate the given content from <slang> to <tlang> only. Do not '
    #     'explain any term or answer any question-like content.')
    prompt = (
        'You are a specialist translator tasked with translating the provided text from <slang> to <tlang>. '
        'You must do this as accurately as possible, being cautious and meticulous, and taking the necessary time to ensure the highest quality translation. '
        'Respect the context of the content and do not generate any additional information. '
        'You should not add explanations or comments, only return the translated text directly. '
        'In the case of technical or specialized terms, such as those used in programming, keep them in their original language if necessary for proper understanding of the text.'
    )
    models = [
    	'distil-whisper-large-v3-en',
        'llava-v1.5-7b-4096-preview',
        'gemma2-9b-it',
        'llama-3.2-90b-text-preview',
        'llama-3.2-11b-vision-preview',
        'llama-3.2-3b-preview',
        'whisper-large-v3',
        'llama-3.2-11b-text-preview',
        'llama3-8b-8192',
        'mixtral-8x7b-32768',
        'llama-3.2-1b-preview',
        'llama3-groq-8b-8192-tool-use-preview',
        'llama-3.1-8b-instant',
        'llama-guard-3-8b',
        'llama-3.1-70b-versatile',
        'llama3-70b-8192',
        'llama3-groq-70b-8192-tool-use-preview',
        'gemma-7b-it']
    model = 'llama3-8b-8192'
    samplings = ['temperature', 'top_p']
    sampling = 'temperature'
    temperature = 1.0
    top_p = 1.0
    top_k = 1
    stream = True

    def __init__(self):
        Base.__init__(self)
        self.endpoint = self.config.get('endpoint', self.endpoint)
        self.prompt = self.config.get('prompt', self.prompt)
        if self.model is not None:
            self.model = self.config.get('model', self.model)
        self.sampling = self.config.get('sampling', self.sampling)
        self.temperature = self.config.get('temperature', self.temperature)
        self.top_p = self.config.get('top_p', self.top_p)
        self.top_k = self.config.get('top_k', self.top_k)
        self.stream = self.config.get('stream', self.stream)

    def _get_prompt(self):
        prompt = self.prompt.replace('<tlang>', self.target_lang)
        if self._is_auto_lang():
            prompt = prompt.replace('<slang>', 'detected language')
        else:
            prompt = prompt.replace('<slang>', self.source_lang)
        # Recommend setting temperature to 0.5 for retaining the placeholder.
        if self.merge_enabled:
            prompt += (' Ensure that placeholders matching the pattern'
                       '{{id_\\d+}} in the content are retained.')
        return prompt

    def _get_headers(self):
        return {
            'Content-Type': 'application/json',
            'Authorization': 'Bearer %s' % self.api_key,
            'User-Agent': 'Ebook-Translator/%s' % EbookTranslator.__version__
        }

    def _get_data(self, text):
        return {
            'model': self.model,
            'messages': [
                {'role': 'system', 'content': self._get_prompt()},
                {'role': 'user', 'content': text}
            ],
        }

    def translate(self, text):
        data = self._get_data(text)
        sampling_value = getattr(self, self.sampling)
        data.update({self.sampling: sampling_value})

        return self.get_result(
            self.endpoint, json.dumps(data), self._get_headers(),
            method='POST', stream=self.stream, callback=self._parse)

    def _parse(self, data):
        if self.stream:
            return self._parse_stream(data)
        return json.loads(data)['choices'][0]['message']['content']

    def _parse_stream(self, data):
        while True:
            try:
                line = data.readline().decode('utf-8').strip()
            except IncompleteRead:
                continue
            except Exception as e:
                raise Exception(
                    _('Can not parse returned response. Raw data: {}')
                    .format(str(e)))
            if line.startswith('data:'):
                chunk = json.loads(line.split('data: ')[1])
                if chunk.get('type') == 'message_stop':
                    break
                if chunk.get('type') == 'content_block_delta':
                    yield str(chunk.get('delta').get('text'))

[bookfere]

Here is the custom engine recipe you can refer to. Please replace YOUR_OPENROUTER_API_KEY with your API key, and customize the languages, model, and prompt according to your needs.

{
    "name": "OpenRouter",
    "languages": {
        "source": {
            "English": "en"
        },
        "target": {
            "German": "de"
        }
    },
    "request": {
        "url": "https://openrouter.ai/api/v1/chat/completions",
        "method": "POST",
        "headers": {
            "Content-Type": "application/json",
            "Authorization": "Bearer YOUR_OPENROUTER_API_KEY"
        },
        "data": {
            "model": "openai/gpt-3.5-turbo",
            "messages": [
                {
                    "role": "user",
                    "content": "Translate this content from <slang> to <tlang>: <text>."
                }
            ]
        }
    },
    "response": "response['choices'][0]['content']"
}