RAM-based map caching to reduce server CPU usage

jasonrohrer · 2018-03-27 15:41:45

Summary: It worked.

Profiling has long shown that map reads consume the majority of CPU resources on the server, especially in areas of the map that have a lot of full containers (each container acts as a multiplier on the amount of information that needs to be fetched for a given tile). As people walk around the map, they look at big chunks of the map all at once, and this produces dramatically more load than map writes (which only happen when a player does something). A player usually changes (writes) about one tile per second on average, but as they walk around, they might be looking at hundreds of tiles every few seconds.

I added a hash table for caching recently-read tiles from the map to make reads much less expensive. Since writes are way less frequent, they can still be expensive: changed tiles are both written to the hash table and instantly pushed into the disk-based database. Thus, the map on the disk never lags behind reality.

Additional profiling also found a few more places where I could squeeze out a few redundancies. For example, substantial load was coming from checking whether each item in a container had decayed. But most things in most containers currently don't decay at all. Might as well remember whether a container holds any decaying items or not, so we don't have to keep checking every item every time we look at the container.

The impact on CPU load can be seen above, with the whole graph representing 40 concurrent players on a cluttered map.

InSpace · 2018-03-27 15:48:27

Ooh cool, is it going to be implemented on thursday or earlier during the week?

jasonrohrer · 2018-03-27 16:38:18

It's already live.

InSpace · 2018-03-27 16:39:09

latest?cb=20160316025719

emucat · 2018-03-27 18:36:18

Does that apply for personal servers?

Uncle Gus · 2018-03-27 19:29:59

That's an impressive result! Are you going to increase the population cap on the servers?

jasonrohrer · 2018-03-27 19:39:10

Yes, personal servers just need to pull from github and rebuild.

I will raise the pop cap at some point soon, but I'd like to let this settle for a few days first.

Also, I just tracked down the source of excessive RAM usage (maps grow in an unbounded way over time as people explore, and the resulting map files grow over time, and so do the underlying database's hash tables, which are stored in RAM.)

At home on a blank map, my game server uses 12MB of RAM.

On server1, it's using 253 MB. It's not a leak, because the same amount is used immediately after restart.

Profiling shows that this is all coming from the DB's RAM-based hash table.

So, I've implemented something that will revert portions of the map back to a natural state when they haven't been seen by anyone for 24 hours. This will allow those map databases to shrink whenever the server restarts, as parts of the map can be culled.

Right now, in terms of disk space, I'm seeing this on server1:

-rw-rw-r-- 1 jcr13 jcr13 383M Mar 27 17:27 biome.db
-rw-rw-r-- 1 jcr13 jcr13 2.2M Mar 27 19:04 eve.db
-rw-rw-r-- 1 jcr13 jcr13 1.3M Mar 27 19:34 floor.db
-rw-rw-r-- 1 jcr13 jcr13 1.3M Mar 27 19:34 floorTime.db
-rw-rw-r-- 1 jcr13 jcr13 153M Mar 27 19:36 map.db
-rw-rw-r-- 1 jcr13 jcr13 163M Mar 27 19:36 mapTime.db
-rw-rw-r-- 1 jcr13 jcr13 2.2M Mar 27 19:36 playerStats.db

A lot of space is taken up by portions of the map that were visited weeks ago during long expeditions.

Furthermore, as the DB's hash tables fill up with entries, they become less efficient to access. This may actually be part of the CPU problem, because all those container lookups will use more and more CPU as the map hash tables become full. In fact, the only time the DB files grow is in response to hash table collisions.

I'll post again here later with the updated DB file sizes after the new culling code has had a chance to operate (I've got to wait at least 24 hours after installing it to give people a chance to actually LOOK at the map before the cull runs).

Uncle Gus · 2018-03-27 19:48:26

So the world is going to revert back to its unexplored state? Even if someone has altered that natural state by putting something down or crafting something?

This also means that the world can "change" whenever a new biome is added, but I doubt anyone will notice that.

jasonrohrer · 2018-03-27 20:32:34

Yes, only in areas that no one has seen for more than 24 hours, though, and only when the server restarts.

So this won't affect active villages at all.

Zwilnik · 2018-03-27 22:08:34

It does limit the fun sub game of “find the lost city of where I left 2 bowls of gold flakes before losing connection” to a 24 hour treasure hunt though.

Uncle Gus · 2018-03-27 22:16:46

Well, depending on how far through the week you are, you might have a whole week to refind it before the server restarts.

Gederian · 2018-03-28 00:35:14

jasonrohrer wrote:

Yes, only in areas that no one has seen for more than 24 hours, though, and only when the server restarts.
So this won't affect active villages at all.

What if a large village on an inactive/personal server wasn't visited for 24 hours, would it disappear too? What dictates what is and is not a village. When I spawn as eve I try my best to get a bowl fired in an oven and some carrots going so the next person that arrives can immediately start farming.

EvilBlackCat · 2018-03-28 02:43:39

I dunno, I think having it go after 24 hours makes sense. 24 hours is 1440 years so it makes sense to me that small places would be retaken by the wilderness. Though perhaps an area with buildings should take longer if that's possible.

Uncle Gus · 2018-03-28 05:05:33

I'm guessing that if you're running a personal server, you will be able to change the time threshold or turn it off entirely.

Gederian · 2018-03-28 06:18:11

It does make sense that areas not ventured would be overtaken by wilderness and since it's open source you certainly could change that. Maybe a setting could support the time it takes with an infinite mode. I just set my EBS drive to automatically take a snapshot every 12 hours so it won't loose my village if I can't log in for a while. Glad I came across this before I was able to put the game down for more than 24hr!

Goliath · 2018-03-28 07:09:23

I don't mind. Thanks Jason!

emucat · 2018-03-28 13:19:00

jasonrohrer wrote:

Yes, only in areas that no one has seen for more than 24 hours, though, and only when the server restarts.
So this won't affect active villages at all.

I had the idea the "nature" could reclaim its space after a few centuries, the same way it happens in real life, but... not just wiping the whole thing...

This way you could find some ruins.

I know this is not the intention of this post.

One Hour One Life Forums

#1 2018-03-27 15:41:45

RAM-based map caching to reduce server CPU usage

#2 2018-03-27 15:48:27

Re: RAM-based map caching to reduce server CPU usage

#3 2018-03-27 16:38:18

Re: RAM-based map caching to reduce server CPU usage

#4 2018-03-27 16:39:09

Re: RAM-based map caching to reduce server CPU usage

#5 2018-03-27 18:36:18

Re: RAM-based map caching to reduce server CPU usage

#6 2018-03-27 19:29:59

Re: RAM-based map caching to reduce server CPU usage

#7 2018-03-27 19:39:10

Re: RAM-based map caching to reduce server CPU usage

#8 2018-03-27 19:48:26

Re: RAM-based map caching to reduce server CPU usage

#9 2018-03-27 20:32:34

Re: RAM-based map caching to reduce server CPU usage

#10 2018-03-27 22:08:34

Re: RAM-based map caching to reduce server CPU usage

#11 2018-03-27 22:16:46

Re: RAM-based map caching to reduce server CPU usage

#12 2018-03-28 00:35:14

Re: RAM-based map caching to reduce server CPU usage

#13 2018-03-28 02:43:39

Re: RAM-based map caching to reduce server CPU usage

#14 2018-03-28 05:05:33

Re: RAM-based map caching to reduce server CPU usage

#15 2018-03-28 06:18:11

Re: RAM-based map caching to reduce server CPU usage

#16 2018-03-28 07:09:23

Re: RAM-based map caching to reduce server CPU usage

#17 2018-03-28 13:19:00

Re: RAM-based map caching to reduce server CPU usage

Board footer