502 Bad Gateway I can't view any of the topics in the forum

futurebird · 2019-03-27 10:04:18

I can't view any of the topics in the forum get this error...

502 bad gateway nginx

futurebird · 2019-03-27 11:18:31

The solution is to log out then log in again twice. My linage is still messed up. IDK what happened...

Tarr · 2019-03-27 11:28:00

Every day around 5 am est the games website + login server go down for around 5-10 minutes, this is what you were experiencing at the time of the bad gateway.

breezeknight · 2019-03-27 16:15:31

the family tree

Jason changes the privacy stuff about emails as log in
not sure how it will work eventually
atm you can log in to your family tree directly from the game

- - -

jasonrohrer · 2019-03-28 00:05:06

Tarr, are you sure about this? Every day, same time? There are some jobs that run on the main server around that time....

5am is the "mainBackupPush" job, which rsyncs a bunch of big files to the backup server...

Tarr · 2019-03-28 01:51:54

jasonrohrer wrote:

Tarr, are you sure about this? Every day, same time? There are some jobs that run on the main server around that time....
5am is the "mainBackupPush" job, which rsyncs a bunch of big files to the backup server...

I can with 100% certainly claim that it happens every day, near same exact time where both the website and login servers go down for at least a five minute window. People constantly ask in the wee hours why they can't login or that the website is down.

Grim_Arbiter · 2019-03-28 05:28:16

Tarr wrote:

jasonrohrer wrote:
Tarr, are you sure about this? Every day, same time? There are some jobs that run on the main server around that time....
5am is the "mainBackupPush" job, which rsyncs a bunch of big files to the backup server...
I can with 100% certainly claim that it happens every day, near same exact time where both the website and login servers go down for at least a five minute window. People constantly ask in the wee hours why they can't login or that the website is down.

Can +1 this

There is that window where no children are born no matter what, and then they all suddenly pop into the village. I call it the baby wave.

Ive been trying to get eve from logging on right after its back up from all the birthmoms being on cooldowns, but I haven't gotten it yet

DestinyCall · 2019-03-28 05:39:51

I can also confirm this. For me, it happens about 1 am in my timezone every day.

I usually take it as a sign that I should go to bed. >.>

ruanna · 2019-03-28 07:00:42

it happens at 5pm for me so I see this being asked in discord constantly.

jasonrohrer · 2019-03-29 03:16:43

mysqldump is the culprit.

It locks the entire database globally, which of course blocks all kinds of operations during that time.

This is part of the backup operation, and is important for consistent backups with the MyISAM tables that I'm using. I've used InnoDB tables in the past when I needed transactions, but they're not needed here...

HOWEVER, even the global lock shouldn't take 5 minutes.

The reviewServer_log table currently has 2.8 GB of data in it. Mostly spurious requests for bad email addresses, often many per second. Maybe a DOS attack of some kind.

jasonrohrer · 2019-03-29 03:27:34

Yes, 220 MILLION rows in that log database... Yikes. Even trying to delete them all is taking FOREVER.

This may be a rather new problem. Something somewhere is going haywire and trying to request sequence numbers for invalid email addresses. This caused the log DB to baloon, which in turn caused the mysqldump operation to take forever. These are otherwise very small tables. There would have always been SOME downtime during this, but maybe a few seconds at most.

jasonrohrer · 2019-03-29 05:02:22

Dammit...

152.136.36.66 - - [29/Mar/2019:04:58:43 +0000] "GET /reviewServer/server.php?act
ion=get_sequence_number&email=applea38a00f8-733e-412f-b324-06933d7d2b6d HTTP/1.0
" 200 6 "-" "-"
152.136.37.88 - - [29/Mar/2019:04:58:43 +0000] "GET /reviewServer/server.php?act
ion=get_sequence_number&email=specialAndroidea6b959d-96e0-43d5-bea3-6ae260c3b56b
 HTTP/1.0" 200 6 "-" "-"

So, essentially, the mobile devs have their server trying to report user stats to me. The must have forgotten to turn that off when they installed my server code.

And because the game has been free in china for... 70 days.... I've been getting absolutely hammered with bad requests like this, without even realizing it.

These IP addresses are all in China.

And thus, my logs have been filling up. And thus, the backups of the database are taking FOREVER. And thus the timeouts on the website at 5am EST every day.

Whatever · 2019-03-29 05:09:01

jason if they are sending you their stats you could use this opportunity to collect them and see how they are doing, hehe

jasonrohrer · 2019-03-29 05:30:07

Well, they get rejected because they are not my customers, and they're not even using valid email addresses. So they never get to the point of actually sending the stats (I reject them when they ask for a sequence number).

Here's the full list of IPs that submitted such faulty requests, and how many requests each one submitted in the past 24 hours alone:

      2 73.202.160.20
      3 172.105.218.219
      4 52.194.4.167
     18 81.0.62.53
    355 139.162.171.247
    451 172.104.178.45
    830 74.207.248.62
   1111 104.237.154.207
   1588 172.105.205.154
   1614 172.105.196.125
   1887 172.105.237.213
   2023 172.104.65.91
   2029 172.105.214.204
   2103 172.105.231.235
   2199 172.105.230.200
   2316 139.162.126.219
   2325 172.104.91.13
   6097 172.105.213.124
  12263 62.234.179.155
  13081 152.136.38.223
  13956 152.136.37.131
  14045 152.136.34.23
  14504 152.136.36.10
  14792 152.136.36.27
  15629 152.136.36.243
  15865 152.136.37.88
  15917 62.234.223.28
  18415 152.136.36.146
  19183 152.136.36.135
  22126 152.136.36.24
  23472 152.136.36.66
  24853 152.136.37.176
  26094 152.136.35.157
  30058 152.136.37.188

The smaller ones are innocent (probably individuals running their own server who just didn't turn the reporting feature off).

But the ones that are hitting me 30K times a day are a pretty big problem.

I could firewall blacklist them... but I worry about those IPs being used for something else in the future.

I mean, obviously, I shouldn't be saving log entries in the DB each time this happens... I wonder if the request load itself will eventually be a problem, though...

Currently 322,861 bad requests per day

Whatever · 2019-03-29 05:45:44

Disable the reporting feature by default. And then manually turn it on for your servers.
If they then update, it should be fixed? And prevent more spamm in the future.

jasonrohrer · 2019-03-29 06:25:24

Yes, that's the plan.

One Hour One Life Forums

#1 2019-03-27 10:04:18

502 Bad Gateway I can't view any of the topics in the forum

#2 2019-03-27 11:18:31

Re: 502 Bad Gateway I can't view any of the topics in the forum

#3 2019-03-27 11:28:00

Re: 502 Bad Gateway I can't view any of the topics in the forum

#4 2019-03-27 16:15:31

Re: 502 Bad Gateway I can't view any of the topics in the forum

#5 2019-03-28 00:05:06

Re: 502 Bad Gateway I can't view any of the topics in the forum

#6 2019-03-28 01:51:54

Re: 502 Bad Gateway I can't view any of the topics in the forum

#7 2019-03-28 05:28:16

Re: 502 Bad Gateway I can't view any of the topics in the forum

#8 2019-03-28 05:39:51

Re: 502 Bad Gateway I can't view any of the topics in the forum

#9 2019-03-28 07:00:42

Re: 502 Bad Gateway I can't view any of the topics in the forum

#10 2019-03-29 03:16:43

Re: 502 Bad Gateway I can't view any of the topics in the forum

#11 2019-03-29 03:27:34

Re: 502 Bad Gateway I can't view any of the topics in the forum

#12 2019-03-29 05:02:22

Re: 502 Bad Gateway I can't view any of the topics in the forum

#13 2019-03-29 05:09:01

Re: 502 Bad Gateway I can't view any of the topics in the forum

#14 2019-03-29 05:30:07

Re: 502 Bad Gateway I can't view any of the topics in the forum

#15 2019-03-29 05:45:44

Re: 502 Bad Gateway I can't view any of the topics in the forum

#16 2019-03-29 06:25:24

Re: 502 Bad Gateway I can't view any of the topics in the forum

Board footer