Smart responses for error logs

[sttephen]

Explain your idea.

I think at this stage we should be having responses for error logs to automate some tickets.

The bot should look at error logs given and present the exact line of the error.

It should also be able to provide a smart response for certain phrases. Spigot is not perfect with sharing error information, but I can think of at least 10-20 errors which are fairly descriptive but still confusing. Eg: "commands are of wrong type" = plugin.yml commands area fucked up. If a phrase is found, the bot should provide a response. If not, ignore. I am happy to help build the bank of phrases to check for -> responses.

Here's an example: (obviously it would look better than this)

A fix would only be provided if one can be found. If not, it can still display them the error and line!

How could you do this differently

No response

Anything else?

No response

[joshbker]

I personally think we should not create technical debt creating a custom system that simply formats the error nicely, and instead we should look at OpenAI integration, using a text model to properly explain the error and a fix. We've all seen how powerful this is when pasting an error in, think it would be a great addition and solves this issue

[sttephen]

Absolutely the preferred option, but I do not believe OpenAI's API is free and I would rather not backdoor it.

[joshbker]

Am happy to fund this, and hopefully at some point Discord revenue can take over the fee, I don't think it would be much more than £10-£20/month. I'll do a cost forecast and come back with it

[sttephen]

Here is the language model pricing if it helps: https://openai.com/api/pricing/

It would be neat to start training a model. However, I cannot ask you to fund this if you are not getting anything in return.

The difficult task is actually limiting abuse as each request costs X cent. There are several ways we can do this, but we would need to implement a way which keeps costs low.

@Devous If you find it to be viable, please create an issue for it and I'll drop this. OpenAI integration is preferred but not sure if it is worth it when we can provide similar support for free (there is not an error in Spigot I haven't seen).

[Aethonx]

I think sending the user with a message about how to understand and read the stack trace is going to be a lot more beneficial then just extracting all the information for them.

[sttephen]

I agree. But support tickets are supposed to be convenient. Before you even learn a drop of Spigot, there is a video on how to read stacktraces. If people have not watched this, they are not going to read and attempt to understand text on how to do it. There is nothing wrong with providing some help - after all, that's what the "help" channel is for.

[joshbker]

Here is the language model pricing if it helps: https://openai.com/api/pricing/

It would be neat to start training a model. However, I cannot ask you to fund this if you are not getting anything in return.

The difficult task is actually limiting abuse as each request costs X cent. There are several ways we can do this, but we would need to implement a way which keeps costs low.

@Devous If you find it to be viable, please create an issue for it and I'll drop this. OpenAI integration is preferred but not sure if it is worth it when we can provide similar support for free (there is not an error in Spigot I haven't seen).

Sadly, I think as cool as it would be because how great the technology is, you may be right that it's just not really feasible to have an outgoing on it when we can do it for free, if we're not going to find a great way to utilise this AI elsewhere then maybe we should just hold off and innovate down the line?

[sttephen]

Here is the language model pricing if it helps: https://openai.com/api/pricing/
It would be neat to start training a model. However, I cannot ask you to fund this if you are not getting anything in return.
The difficult task is actually limiting abuse as each request costs X cent. There are several ways we can do this, but we would need to implement a way which keeps costs low.
@Devous If you find it to be viable, please create an issue for it and I'll drop this. OpenAI integration is preferred but not sure if it is worth it when we can provide similar support for free (there is not an error in Spigot I haven't seen).

Sadly, I think as cool as it would be because how great the technology is, you may be right that it's just not really feasible to have an outgoing on it when we can do it for free, if we're not going to find a great way to utilise this AI elsewhere then maybe we should just hold off and innovate down the line?

I'm guessing you looked at the prices!

But yes, I think when the Discord is self-sustainable and we finally get access to that damned feature we will be able to justify this. There is no harm in waiting, it will still be there for us, just even better.

[joshbker]

Here is the language model pricing if it helps: https://openai.com/api/pricing/
It would be neat to start training a model. However, I cannot ask you to fund this if you are not getting anything in return.
The difficult task is actually limiting abuse as each request costs X cent. There are several ways we can do this, but we would need to implement a way which keeps costs low.
@Devous If you find it to be viable, please create an issue for it and I'll drop this. OpenAI integration is preferred but not sure if it is worth it when we can provide similar support for free (there is not an error in Spigot I haven't seen).

Sadly, I think as cool as it would be because how great the technology is, you may be right that it's just not really feasible to have an outgoing on it when we can do it for free, if we're not going to find a great way to utilise this AI elsewhere then maybe we should just hold off and innovate down the line?

I'm guessing you looked at the prices!

But yes, I think when the Discord is self-sustainable and we finally get access to that damned feature we will be able to justify this. There is no harm in waiting, it will still be there for us, just even better.

Yeah, unfortunately the nature of stack traces is quite large, like 100-250 odd words per, which with OpenAI's word/token payment structure soon adds up. Assuming an error is 250 words, 3 errors is about 1,000 tokens, and on Davinci that's $0.02, or $0.0066667(ish)/error. If we're looking at say 20 errors a day, that's only like $4 a month, but I fear we would see a lot more than 250 words in stack traces quite often, and we would also get a lot more than 20 a day once people start realising the feature exists.

I suppose it's not as much down to the money, more just the time and effort, and I think I'm just driven for it because it's new and exciting rather than needed and feasible.

Maybe when we get this Discord feature we can implement OpenAI integration as a premium feature, that is funded by revenue from their subscription, would be a neat incentive.

[Nuckerr]

Love the idea, however surely support is redundant if we just have chat gpt respond. There is no use for support because the times gpt fails can easily be managed by a small team.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

Maybe using gpt somehow but when to utilise it is the question

[sttephen]

@Nuckerr I think we decided that OpenAI is not the solution, too expensive to justify. I would listen to my original ticket. Suggestions are very welcome.

[MLGPenguin]

I started making something like this a while ago that uses regex to automatically find the line and class from an error- Perhaps with a bit of manual work it can be adapted into what you're looking for?

[joshbker]

I started making something like this a while ago that uses regex to automatically find the line and class from an error- Perhaps with a bit of manual work it can be adapted into what you're looking for?

Sounds good, if you're happy to, go for it!

[Nuckerr]

@MLGPenguin would you like to remain assigned to this issue?

[MLGPenguin]

@MLGPenguin would you like to remain assigned to this issue?

I'm not really getting very far 😅
You can unassign me and if I make any decent breakthroughs I'll let get you to reassign me. For now I think the way I'm doing it is a little flawed

[Dawsson]

Any updates? I can implement this with the bot framework and our solutions command?

[stephendotgg]

@Dawsson seems like a dead isssue id go for it