Poor uptime ( move to a better server? )

[jacobgkau]

When going through SRS review sessions, I regularly (just about every day) encounter one or more periods when definitions won't load; trying to go to the website in another tab fails too. This can last for a few seconds or several mintues.

I know this isn't on my end because it's happened on multiple computers (with different OS's and browsers) and in multiple cities on entirely different internet connections. It seems like it happens more often later at night in U.S. Mountain time.

I understand this may not entirely be related to the code for the website on GitHub, but I wanted to report it somewhere. Hosting my own instance of the software so I'm in control of the uptime would be a potential workaround, but then I wouldn't have access to community definitions when adding new cards, which (IMO) is a large part of the website's value.

[fabd]

Hey thanks for letting me know,.

I'm sorry about the server's poor uptime.

Koohii is currently running on a shared server in central US - which may explain why it could be less available late evening US time.

I've been postponing moving servers for years now... so I don't know. Maybe I should really put aside any ideas or incrementla improvements this year and just focus on moving to a better server.

I'm just not a backend/server kinda guy and I don't even know where to start. But I mean I can do it, I'm sure within weeks i can spin up a copy of the site on DigitalOcean or whatever that's not the issue. My worry is first off, knowing for SURE that I am actually moving to something more powerful as the actual specs of the shared server atm are arcane to me, I only have CPanel to have some basic idea of CPU time and whatnot. Then I need to figure out what is a good basic server on DigitalOcean or any of those services, then I need to figure out all the spam protection stuff and how the server can be maintained and updated with minimal maintenance - so I don't need to be monitoring it all the time.

But... I guess I'll have to get down to it.

To be fair I receive very little feedback in that regard. I honestly have received like what, five emails maybe last year about slowness or uptime issues? I'm guessing a lot of people are reviewing at non peak times for the server.

I'll look into it. In the meantime all I can suggest is either

• try reviewing at other times (and let me know if it helps)
• potentially use the Kanji Ryokucha app on iOS, it connects to your Koohii account, it will also have issues, but it works offline so you sync whenever you like (the Android app I'm afraid is not maintained, the iOS app is by German Buela, not me)
• use a separate SRS app

[fabd]

Hi Jacob

Interesting, I remember your channel actually but I'm not sure from where. Thought maybe you offered advice earlier in the related thread or email but I only find one email from May 2022.

Either way, I'm certainly not going to turn down help so long as I have your attention.

I have to be honest, right now handling the keys to the database etc. that 's a big step and I worry that I am unable to attend to things on my own should the need arise.

Perhaps right now the best way you can help me is with advice, and we can proceed from there.

I did in fact experiment a little bit with Amazon AWS way back. I remember one thing I definitely do not like, is the cost based on traffic. I really don't like that I could have bills down the line with no fixed limit. I have read stories of abuse where people get a large bill because bots have been using lots of bandwidth.

I'll be honest I don´t know if I'm ready to dive into this. But it could be interesting as well and good skills to have.

Every step I can make forward on this issue will increase the chances of it happening so with that in mind...

Perhaps the simplest, most practical question right now - where I will just go with your experience :

What current offerings at Linode or Digital Ocean would you recommend for Kanji Koohii ?

Keeping in mind the following:

• a typical php / mysql (mariadb also works) stack. I've always used Apache but no particular reason
• you can get an idea of the database size in this old blog post https://kanji.koohii.com/news/id/416
• 99% of the site doesn´t benefit from a "cloud" setup since users always look at dynamic pages reflecting their own state/data

Another practical question is

Are there like recipes or pre-made images/containers that cover a lot of ground?

Things I'd need to figure out:

• shared server has php preconfigured with optimizations such as "OPCache" o(or "memcached?" - which I never dealth with
• shared server has probably a bunch of builtin security such as protection against bots and DDOS - I think some of this is covered by DO / LINODE (at a higher level)
• shared server has a pre-configured mailing service - which I also never needed to configure
• shared server has some kind of default MariaDB / MySQL optimized settings - which I would need to figure out or have a template to go with

The third question that comes to mind is about Docker.

Should I go with containers?

• should I go with the "deploying container" which exists locally on my machine, is that the best way to do it?
• which is why I mentioned recipes earlier, because maybe I could have like an image that does a lot of the setup?
• and if not what is the other option? currently I have a bash script that simply uploads archives of the app & web folders with scp and then simply run a fewe commands via SSH like rm folders && unzip archive - is that even an option or not at all?

Thanks

[jacobgkau]

Hi Fabrice,

I did in fact experiment a little bit with Amazon AWS way back. I remember one thing I definitely do not like, is the cost based on traffic.

I actually have an AWS Cloud Practitioner certification as well, and am therefore somewhat familiar with AWS's pricing schemes, although I also prefer the simpler fixed-cost VPS services. AWS is (in my opinion) more suited for workloads where interdependent services are programmatically started and stopped, or for larger scales of operation.

What current offerings at Linode or Digital Ocean would you recommend for Kanji Koohii ?

Keeping in mind the following:

• a typical php / mysql (mariadb also works) stack. I've always used Apache but no particular reason
• you can get an idea of the database size in this old blog post https://kanji.koohii.com/news/id/416
• 99% of the site doesn´t benefit from a "cloud" setup since users always look at dynamic pages reflecting their own state/data

It's difficult to make a 100% accurate forecast, as this will mostly depend on exactly what CPU load your PHP code generates with the quantity of traffic that you have. If you want to be very safe, a Linode 4 GB ($20/month) plan might be comfortable to try out-- it's the cheapest plan with two CPU threads, and has RAM that sounds very sufficient for the amount of data being worked with. I personally use a 2 GB ($10/month) plan to run ~10 different web services, but I think they all have lower traffic than your site.

Another practical question is

Are there like recipes or pre-made images/containers that cover a lot of ground?

Things I'd need to figure out:

• shared server has php preconfigured with optimizations such as "OPCache" o(or "memcached?" - which I never dealth with

These are both one-time configuration tasks that are pretty simple to set up. See Nextcloud's configuration guide for each, for example.

• shared server has probably a bunch of builtin security such as protection against bots and DDOS - I think some of this is covered by DO / LINODE (at a higher level)

Linode advertises that they provide DDoS protectionby default. DigitalOcean does not seem to advertise this directly, I only found some posts from employees on their forums stating they take precautions and monitor for attacks.

• shared server has a pre-configured mailing service - which I also never needed to configure

Ah, sending email would be an additional complication. I use Elastic Email as a relay for sending, although I'm grandfathered into a cheaper plan than what is currently offered. It is possible to send email directly from the Linode/Droplet, but the level of deliverability (mainly whether or not you get marked as spam) will partially depend on who had the IP address before you (if they were sending spam email) and if anyone with IP addresses in the same block is sending spam.

• shared server has some kind of default MariaDB / MySQL optimized settings - which I would need to figure out or have a template to go with

I would need to know more about what kinds of optimizations are performed or required to discuss this. The default configuration shipped by Linux distributions are typically suitable for general use. MariaDB has some documentation about performance tuning, but it's more about how your queries are written/database is structured (and maybe the storage engine in use) than anything else.

The third question that comes to mind is about Docker.

Should I go with containers?

• should I go with the "deploying container" which exists locally on my machine, is that the best way to do it?
• which is why I mentioned recipes earlier, because maybe I could have like an image that does a lot of the setup?
• and if not what is the other option? currently I have a bash script that simply uploads archives of the app & web folders with scp and then simply run a fewe commands via SSH like rm folders && unzip archive - is that even an option or not at all?

If the server is only going to be used to host this one app, than the third bullet point (upload app/web folders via SSH) is simplest. In my opinion, containers are more useful when running multiple apps and needing to keep their environments separate from each other, or when deploying a particularly complex application. However, if you're already using Docker for development and it would be simpler for you to deploy the same image onto the server, then that would certainly be an option. You just may need configuration/tuning tweaks to be done in the Docker configuration instead of on the server.

(I usually use Podman to actually deploy Docker images instead of Docker itself; it's simply a different frontend maintained by Red Hat and not requiring as many external dependencies from regular Linux distribution ecosystems. The syntax is usually identical.)

Knowing how much the current shared server setup costs to compare might be useful for evaluating options.

I have to be honest, right now handling the keys to the database etc. that 's a big step and I worry that I am unable to attend to things on my own should the need arise.

Understandable.

[fabd]

Hi Jacob

Thanks this is really helpful!

I've been thinking about this issue last couple days - and now I realize I´d be a fool to turn down this opportunity. The main reason I've postponed this server move/upgrade for so long is because I was worried if I mess up something, that the site would be down for long times and it would be very frustrating for users as well as just generally hurting the service more than it would have been otherwise.

But with someone qualified like you as a support, I am more confident in going forward. Hopefully we can work out something whether it ś short term like a one-time setup to get me started, or a medium term thing where I let you manage the server for some time, or an on and off thing where I try to manage as much as I can but know that I have an expert to help me out if I'm stuck.

As we say in french "beat the iron while it's hot" - if I am going to do this sever upgrade thing, it's gotta be now. Especially as I am in between (real) jobs atm, I can commit to spend some time figuring out the basics - so if/when you need to move on or do not have the time available anymore - then I'll have the basics down to handle the common problem scenarios.

So anyway I will fire you an email soon to talk more specifics and figure out the next steps.

I would be willing to volunteer to help maintain a server if that is something that would be helpful.

Yes, I'm grateful and I´d be stoked to have you onboard :) So yeah let's do this!

A few more things come to mind :

First I'll need to signup for a server. I'll do some research with DO vs LINODE. Any other options? Put simply, if these services are relatively similar and it's new to me anyway I'm willing to go with whatever is easier/quicker for you to work with. So if you have a preference or like two options you favor I can look at these.

For me the more user friendly frontends there are to monitor the server - the better. For setting up CRON jobs and the like usually CPanel is just a bunch of text boxes and doesn´t help me much. So for me the main plus points here is the included monitoring features, but it looks like both DO and LINODE have something like email alerts,

Something that would be good but doesn't seem to be in either by default - is a way to restart the processes when they fail. It ś possible to do this via a CRON job as well apparently with some shell scripting.

In terms of server specs going with a suggestion in the old thread - I'm leaning towards getting a cheaper 2 GB 1 vCPU server and simply scaling up - since we can do that.

Thins I need to do :

• send you an email to continue the discussion with more specifics
• set up a DISCORD to exchange messages then (one option I saw on your Managed by NOTS intake form - I am not very familiar with most of these). Discord could be useful as well perhaps to update users when I'll move the database, to have at least some communication channel taht is live when the site is down.
• spend some more time on DO and LINODE

Cheers!

[fabd]

I have extensive experience with DigitalOcean and Linode, mostly Debian-based distributions, but I am also a Red Hat Certified Engineer.

Also re-reading that part,

My thinking is we can probably go with whatever distro you are more comfortable or which is most convenient for you like if you already have scripts to automate things and whatnot.

Looking up Debian, I see that is is the base for Ubuntu right? Since I am using Ubuntu as a host and is the only linux distro I'm familiar with - I guess if we can go with that it is the ideal as there will be less context switching in terms of - sometimes - CLI commands that don't behave the same way.

But other than that, what I have in mind is I'll want to reproduce this setup in a local Docker container. Although I won´t be deploying the container (quite a step for me, keep it simpler atm) - I would in theory be able to experiment with command line, packages, configs. and run scripts and make sure they work before doing any intervention on the remote server. Say, eg. if I make a script to automate my database backups.

Only other thing that comes to mind, is I am (somewhat) familiar with bash. So if I can use that one it's ideal - but I think you can use multiple shells right? So that shouldn't be an issue.

Thanks

[jacobgkau]

Hello,

I'll do some research with DO vs LINODE. Any other options? Put simply, if these services are relatively similar and it's new to me anyway I'm willing to go with whatever is easier/quicker for you to work with.

There is another VPS option called Vultr that I've heard of from viewers, but not used. I typically suggest Linode because I am very familiar with their management interface, have chatted with their employees at events, etc. Linode is slightly better than DigitalOcean about allowing one to scale down with their API and not just up. I've also seen them have slightly newer CPUs for the same class of server, although that naturally goes back and forth as each company updates their infrastructure.

So I would recommend Linode for now if you don't have a preference. If you don't have an account yet, sign up through linode.nots.co and I think they'll start you off with some free credit, and will give me some credit. (The same applies to digitalocean.nots.co or vultr.nots.co.)

The main option beyond these VPS services would be a more corporate cloud provider, such as AWS that you mentioned, or Microsoft Azure or Google Cloud Platform. They all have similar pricing and service structure to each other (but more differences than DO vs. Linode vs. Vultr have, simply because they offer a lot more types of services and have more brand recognition to rely on.)

Something that would be good but doesn't seem to be in either by default - is a way to restart the processes when they fail. It ś possible to do this via a CRON job as well apparently with some shell scripting.

I would do this within the server using a systemd service. systemd (the init system) can start the app on boot and restart it if it fails. The VPS provider would only monitor if the server is booted, and boot it again if it shuts down.

My thinking is we can probably go with whatever distro you are more comfortable or which is most convenient for you like if you already have scripts to automate things and whatnot.

Looking up Debian, I see that is is the base for Ubuntu right? Since I am using Ubuntu as a host and is the only linux distro I'm familiar with - I guess if we can go with that it is the ideal

Yes, Ubuntu is based on Debian. I typically prefer Debian because it doesn't have some of Canonical's for-profit apps and services that are basically bloat if you're not using them (the paid Ubuntu Advantage support software, the daemon for the Snap packaging format, etc.) I am perfectly comfortable administering either of these, though, and they usually behave the same in terms of CLI. It sounds like we should use one of these rather than an RHEL-based distribution, as those have more differences.

Only other thing that comes to mind, is I am (somewhat) familiar with bash. So if I can use that one it's ideal - but I think you can use multiple shells right? So that shouldn't be an issue.

Yes, bash is the standard shell for most distributions. There is no need to use anything else in this context.

Hopefully we can work out something whether it ś short term like a one-time setup to get me started, or a medium term thing where I let you manage the server for some time, or an on and off thing where I try to manage as much as I can but know that I have an expert to help me out if I'm stuck.

Sure, I'm not going to be finished learning Japanese any time soon. 🙂 I expect most of the work to be the initial setup, but will be available to assist with anything that comes up at least in the medium-term (will want to have the basics documented for you in case I get hit by a bus or something.)

So anyway I will fire you an email soon to talk more specifics and figure out the next steps.

Sure, I'll keep an eye out. Please use my NOTS address, [email protected]. (I think the one you have on file from the app may be my personal.)

[fabd]

Thanks again for the feedback. Converted the issue to a discussion, and hopefully with more input it'll help me get the ball rolling.

But also anyone reading this please let us know if the uptime is an issue - where are you located and around what time?

[fabd]

Hey thanks for the offer ot help @ajweitz . Yes we are progressing, and I hope we can move to a LINODE instance sometime during April.

Site is running fine. I have to finish STMP email via GMail, to avoid php mail() shenanigans - which weren´t too much of an issue on the shared hosting - but are being blocked with LINODE instances.

I cancelled Google Business Starter plan - as I found out free GMail SMTP via XOauth2 can send up to 500 emails per day which is sufficient - hopefully will be able to avoid this extra cost. While fabrice AT koohii.com would be nice, the gmail address is fine. It will still be an improvement over php mail(). Needs more work to refresh SMTP tokens.

For caching using the old Symfony 1.x framework, it ś pretty straightforward, we can cache full page or templates / partials. The Study page currently caches the Shared Stories area for each of RTK kanji.

After moving I will be exploring a little bit how to take advantage of memcached (It's pretty basic right now).

Let's just hope that the LINODE 1 CPU 2 GB will do at least as well as the current shared hosting. If it isn't speedier but overall more reliable then it is still a win. Then again I'll admit I don't really understand the difference. AFAIK LINODE instances are also a form of shared CPU etc.

Thanks for your offer to help with AWS. I will keep you in mind if Jacob's busy and of course your input is welcome. Once we do the actual move I expect more conversation to move here, gathering user feedback as well.

From what I understand the monthly cost with LINODE will be fixed. Actually I have doubts now. It sounds like there would be extra billing if it goes over the quota, but not for inbound quota ( https://www.linode.com/pricing/#compute-shared ). There is some hardcoded throttling for the Study pages. But hmm.

Yeah we might have to setup some hardcoded limit as well just in case some idiot is running a bad scraping script that would cause the instance to exceed the quota? @jacobgkau

[fabd]

I should note scarping definitely happened in the past and once caused the site to go down as the user's script was downloading all the Study pages with barely a pause between the pages... it is because of that that the Study page has a hardcoded throttling.

[jacobgkau]

The Study throttling is something I've run into sometimes when attempting to navigate during a study session and would be a nice annoyance to possibly lighten up on, once we see how much load the site actually generates for the VPS.

Then again I'll admit I don't really understand the difference. AFAIK LINODE instances are also a form of shared CPU etc.

With the Linode VPS, your virtual server is guaranteed a certain amount of performance up to the limit of the VPS plan. I can't speak to how HostGator manages their shared hosting plans, but generally, all websites on the HostGator server would be competing with each other for CPU time, RAM, etc. With a VPS, at least your RAM and storage is "carved out" and theoretically guaranteed if it's needed. The vCPU cores are managed/balanced by the hypervisor, and not directly competing for time at the per-process level. (I'm not sure how much Linode overallocates on their host machines; it is possible for Linodes to affect each others' performance, but they may have mechanisms to move VMs or resources around to try and minimize that impact.)

Just to clarify one other point, Linode is not blocking SMTP, but mail deliverability didn't appear to be reliable with a quick setup (likely due to inbound mail servers' anti-spam measures.)

[jacobgkau]

Just posting for visibility, the migration to Linode was abandoned last month due to time constraints and lack of long-term confidence about maintaining the server on @fabd's end.

It sounded like a different shared hosting provider more similar to the current HostGator configuration might be tried eventually (although the site has been performing alright for me personally the past few weeks.)

[fabd]

Yes. Thank you Jacob. I gained some knowledge out of this experiment, and made some improvements to the docker setup which will help once I want to try again.

I hope I can give it another shot when things quiet down here after the summer. Thanks for your kind help!

[m-jepson]

I have been noticing quite some issues in the morning lately (CEST), that would be shortly after midnight in the US. Could that be server maintenance being done there? It doesn't seem likely that the shared server is busy with US requests at that time on a weekday, right?