Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

500 internal server #214

Open
schmidtytg opened this issue Feb 11, 2021 · 9 comments
Open

500 internal server #214

schmidtytg opened this issue Feb 11, 2021 · 9 comments
Labels
Type/Question Further information is requested

Comments

@schmidtytg
Copy link

Mattermost Version: 5.28.1
Database Schema Version: 5.28.0
Database: mysql
Mysql 8.0.22
Zoom plugin: 1.5

I'm able to resolve the issue by disabling the plugin and then enabling it again. While this usually isn't a big deal, I'm not always available to reset the plugin.

I tried completely uninstalling and reinstalling the plugin a couple months ago, but the same behavior seems to be continuing.

The issue seems to have started around when I upgraded mysql from 5.7 to 8.0. Not seeing anything in the mysql logs to indicate an issue with mysql though. The mattermost.log doesn't give me much to go off of either.

From what I can tell there's nothing logged until the plugin seemingly stops working.

{"level":"error","ts":1613014032.4081192,"caller":"mlog/log.go:229","msg":"Plugin failed to ServeHTTP, RPC call failed","plugin_id":"zoom","error":"connection is shut down"}
{"level":"error","ts":1613014037.2250252,"caller":"http/h2_bundle.go:4218","msg":"http2: server connection error from 24.220.147.23:1628: connection error: PROTOCOL_ERROR","source":"httpserver"}
{"level":"error","ts":1613014037.4109783,"caller":"mlog/log.go:229","msg":"Plugin failed to ServeHTTP, muxBroker couldn't Accept request body connection","plugin_id":"zoom","error":"timeout waiting for accept"}
{"level":"error","ts":1613014037.4110491,"caller":"mlog/log.go:229","msg":"Plugin failed to ServeHTTP, muxBroker couldn't accept connection","plugin_id":"zoom","serve_http_stream_id":246,"error":"timeout waiting for accept"}

Any suggestions?

PS. This is my first bug report on github. Please let me know if there's anything I can do better in the future when reporting issues.

@schmidtytg
Copy link
Author

I forgot to add that OAuth is not enabled.

@aaronrothschild
Copy link
Contributor

Hey @schmidtytg Thanks for filing this (and welcome to Github)!

We'll review it in our team meeting on Monday/Tuesday to triage it and see if we need more info from you, etc.

@aaronrothschild
Copy link
Contributor

I forgot to add that OAuth is not enabled.

Sorry, could you elaborate? are you using JWT or OAuth for your Zoom app authentication method?

@schmidtytg
Copy link
Author

We're using JWT, not OAuth.

@schmidtytg
Copy link
Author

After doing an upgrade yesterday to server version 5.31.1, all hell broke loose. It turns out I hadn't updated the character set on all the tables after updating mysql to 8.0.

I've updated the collation of all the database tables to utf8mb4_unicode_ci. Things have been looking much better performance-wise since.

Given this major revelation/rtfm, it would seem prudent to wait and see if the issue with the zoom plug-in has been resolved as well.

@schmidtytg
Copy link
Author

schmidtytg commented Feb 26, 2021

Unfortunately this issue just reared its ugly head again.

{"level":"error","ts":1614354624.8175862,"caller":"mlog/log.go:232","msg":"Plugin failed to ServeHTTP, RPC call failed","plugin_id":"zoom","error":"connection is shut down"}
{"level":"error","ts":1614354627.7682946,"caller":"http/h2_bundle.go:4218","msg":"http2: server connection error from 69.67.17.128:4721: connection error: PROTOCOL_ERROR","source":"httpserver"}
{"level":"error","ts":1614354629.8191347,"caller":"mlog/log.go:232","msg":"Plugin failed to ServeHTTP, muxBroker couldn't accept connection","plugin_id":"zoom","serve_http_stream_id":346,"error":"timeout waiting for accept"}
{"level":"error","ts":1614354629.8191993,"caller":"mlog/log.go:232","msg":"Plugin failed to ServeHTTP, muxBroker couldn't Accept request body connection","plugin_id":"zoom","error":"timeout waiting for accept"}
{"level":"error","ts":1614354629.8880343,"caller":"mlog/log.go:232","msg":"Plugin failed to ServeHTTP, RPC call failed","plugin_id":"zoom","error":"connection is shut down"}

I'm not sure if this is relevant or not, but when I first set Enable Plugin to False and hit save, the radio button went back to True. When I set it False and saved a second time, False stayed selected. The plugin started working again after setting it to True again. Same symptoms with users getting the 500 internal server error again.

@schmidtytg
Copy link
Author

schmidtytg commented Mar 25, 2021

I just had another issue with the plugin crashing. This sent the CPU of my mattermost server up to 100% for about 10 minutes before I was able to disable/enabled the Zoom plugin. Once the zoom plugin was disabled and then reenabled, the CPU dropped back to to its normal levels.

This was the process that was running at high CPU: plugin-linux-am
There were two of these processes running at the same time. Both were the highest CPU processes at the time.

@schmidtytg schmidtytg reopened this Mar 25, 2021
@schmidtytg
Copy link
Author

This continues to happen. Any help would be appreciated.

@hanzei
Copy link
Collaborator

hanzei commented May 27, 2021

Could you please set the log level of your MM server to Debug and check if you see any related messages around the crash?

@hanzei hanzei added the Type/Question Further information is requested label May 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type/Question Further information is requested
Projects
None yet
Development

No branches or pull requests

4 participants