Trouble with writing implementation

jasonrohrer · 2018-08-20 20:47:01

I'm struggling to figure out how to implement writing, given constraints of the engine.

Each object is represented by a 32-bit object ID. This is the way it's represented on the map, in container slots, in sub container slots, when held, when contained in a backpack. etc.

Furthermore, the code that manipulates objects on the server is pretty complicated (moving them in and out of containers, etc.).

Obviously, written phrases would be longer than 32 bits, so they'd need to be stored in some kind of separate database. I envision a system where each written phrase has a 32-bit ID, and these "written phrase IDs" get attached to an object in the world.

But I'm stuck on where to put this phrase ID so that it stays with the object as it moves around, in and out of containers, etc. There is so much server code already written for moving objects around as 32-bit IDs that rewriting it all feels like it would be out of the question.

An interesting aspect of this is that the written phrase is only needed when someone is holding a written object (when they are reading it). The rest of the time, it doesn't need to be accessed at all (for example, it's not needed in map chunks).

However, it still needs to be tracked as the written object moves from place to place (for example, in a rail cart).

So you have a piece of paper, say ID 2301. It's put in a basket, and then the basket is put in a cart, and then the basket is taken out of the cart. How does the writing follow it along the way?

One idea is that we're not using the full 32-bit ID space.... that's enough for four billion unique objects, and the stated goal for this game is 10,000 (along with, maybe, 10x more space for meta objects, like those generated automatically for used objects, locks, etc). So maybe 100,000.

Here are stats from the most recent object data when my test server loads it:

  Loaded 1337 objects from objects folder
  Auto-generated 1020 'used' objects
  Auto-generated 210 'variable' objects

Yes, there are an elite number of objects already in the game, and roughly 2x as many if we count meta objects.

Could the rest of this address space be leveraged to track metadata?

And I suppose there could be other kinds of metadata attached to certain objects in the future as well, beyond writing.

The problem with sticking this stuff in the top end of the 32-bit ID is that it really limits how many unique phrases there can be. If we keep 17 bits for the object ID (for 131K total IDs), that only leaves 15 bits for metadata IDs, or 32K of them. Will people want to write more than 32K pieces of writing on a given server?

The other idea is to include the object ID in the metadata ID, so if there are different types of paper, each type will have it's own 15-bit metadata address space. That doesn't really help, though, because there's no guarantee that player writing activity will be evenly distributed among the different paper types.

I suppose if there are 32K writing IDs, they will just "wrap around" and get used over again when they run out, meaning that the oldest piece of writing on the server will be lost to make room for new writing. That seems like pretty reasonable behavior.

Uncle Gus · 2018-08-20 21:00:13

32,000 unique pieces of writing should be plenty I think, especially if writing is not cheap. My plan for writing that I never got around to developing was pretty much along the same lines as what you're talking about here. I was going to use an offset to the ID int (basically utilising the top end of the int's number range, as you are talking about here) to track unique book IDs. If the server/client was ever dealing with an ID that was outside of the range of normal object IDs, it would substitute in the ID of the normal book object for the purposes of transitions and containing and rendering and all that, but in the map database etc. it would really be using the unique ID of that book. Then the next step was going to be implementing some kind of UI that would display when looking at a book and would read the text content from a file, referenced by the unique ID of that particular book.

But yeah, I'm sure you would be able to do it much nicer than that, with a proper database etc.

Exciting!

Kinrany · 2018-08-20 21:12:47

Paper decays too, right?

jasonrohrer · 2018-08-20 21:49:12

Well, it would, yeah, I suppose. I hadn't gotten that far in my thinking about it yet.

Uncle Gus · 2018-08-20 21:55:31

Having writing decay would almost defeat the purpose of writing in the first place. Being able to write down the wisdom of the ancients in a book and hand it to the newbie will make a world of difference. I guess if the decay was a really long time, it would mean someone would need to copy the aging manuscript from time to time, much like real history. That could be pretty cool. What would be super neat, but probably a total headache to implement would be if the text would decay gradually, with characters smudging out every so often.

Uncle Gus · 2018-08-20 22:15:09

Another problem to solve is how to represent these unique objects for transitions, etc. In my solution, I was only going to implement one writing object, a book, so that whenever a book ID was being manipulated, it would substitute in the book object. But if books were to decay, say a book becomes ragged after a few epochs or something, then there would need to be some way of recording which unique book this is, and which book type it is. I guess that's what you were talking about when you said:

jasonrohrer wrote:

The other idea is to include the object ID in the metadata ID, so if there are different types of paper, each type will have it's own 15-bit metadata address space. That doesn't really help, though, because there's no guarantee that player writing activity will be evenly distributed among the different paper types.

Uncle Gus · 2018-08-20 22:17:18

Or perhaps the unique object's parent object could be recorded in the text database, but then that would need to be looked up from the database every time that object is manipulated. That might cause performance issues.

Solbusaur · 2018-08-20 22:38:23

Kinrany wrote:

Paper decays too, right?

That seems fine to me. The point of this game is to be part of a long chain, right? So how about someone takes it upon themselves to preserve the message by rewriting it, or fixing the paper. A visual decay, like tattered clothes, would notify the player that the message will soon be lost forever.

bluefootedpig · 2018-08-20 23:31:31

What about the idea of allowing ANYTHING to have text added to it. So it would be part of the base object. Maybe I give someone an item that says, "from dad to son"

You hold an item, and say like... "inscribe Hello World!" and to read you can just type "Read" and you will read what you are holding.

For signs, they use this text to display things.

Asheron's call (an MMO back in the 90s) allowed this, and people would inscribe all kinds of things, and find the item later, or someone else had it. It was interesting to buy an item that had an inscription on it. It will add a lot of "leaving my mark on the world" when you can pick up an item and read the inscription.

Inscriptions might not make as much sense on clothing, but I think that is minor.

Christoffer · 2018-08-21 00:25:02

I think there's a flaw in the logic: You don't need 15bits of meta data for each object ID, you merely need N object IDs that are associated with N meta-data IDs, where N could be very large.

Have your regular objects occupy IDs 0-99,999, your auto-generated ones occupy 100,000-999,999 and start objects with meta-data at ID #1,000,000. To find the corresponding meta-data container, deduct 1,000,000 from the object ID.

The current way of starting auto-generated object IDs directly after the highest "base" object ID, is actually a potential bug source every time the data version changes. Keeping a lot of room between the types and not having to move the boundaries ever again, seems like a good idea to me.

I don't think there's a reason to inscribe carrots or piles of soil, etc. This is primarily a graphical game and not a text driven one, so text should probably be reserved for dedicated objects, such as parchments, books and rune stones (maybe grave stones?).

When you implement text, I hope you will consider using unicode, so different languages and symbols could be supported.

Cheers!

CynderR · 2018-08-21 02:17:21

A small idea..
But since we know this will be used mostly to write racist comments (just being honest here), how about we have a limited list of content that can be written out.... Kinda like the dark souls communication.
It would still allow trolls to have fun by putting
"Danger, man but stop" and prevent things like "h**** ***** ** f*** *** *** *****"

Neo · 2018-08-21 06:16:16

I'm just excited about writing, Hope you can make it work.

Joriom · 2018-08-21 07:16:29

And thats exactly why you use oop to pass around object addresses instead of their direct values. Because you can shove as much data as you want behind the reference while still mostly operating on ints.

You could even simulate similar behaviour using malloc where you pass around memory address and dereference it for actual object ID but in special cases (some special IDs like a "writable object") you expect there to be more than memroy allocated. For example first 4 bytes are always object ID, but for writables its followed by X bytes of "text id", Y bytes of "text len", and Z bytes for the text itself. As for database - if you can't store the text ID itself along with object id - just store it in second, almost exactly the same table, just replace object ID with text id. And third table for the texts themselves.

At least thats how I would implement it.

If chunks were used to store larget parts of map, any metadata is stored at the end of chunk information most of the time. Data has 2 (or more) sections. Actual map tiles in 1st (including short, local metadata ids) and 2nd section that contains all the metadata for this part aligned according to those short local ids.

Old mmo servers used to always store 4 values (4 x 16 bit short int) for each tile / object.
Item ID
Item Detail (no. of uses OR rotation OR texture variant, etc - in case of OHOL no. of uses could easily be split into two chars - lower byte: current uses, higher byte: max uses)
Action ID (to bind scripts to those items - what you would call transitions but simply more robust, giving a lot of flexibility as it would be piece of actual code capable of full interaction - thats how doors worked for example or containers)
Unique ID (a misnomer really, as it was basically additional room for Item Detail, often used as "Key ID" for door, or "Quest ID" for reward chests, etc - most often used only in the Action ID scripts to determine outcome)

Fun fact about using Action ID to bind script instead of transitions - "generic scripts".
For example as long as every open door had its "closed" part as next ID, you could bind the same pair pd Action ID scripts to all doorts.
Open doors : script(ID += 1); Closed doors : script (ID -= 1);

Matbat · 2018-08-21 07:30:36

Perhaps an item that can contain x amount of key words?
I honestly don't know how any of the technical stuff works but an item that can contain key words that show above your chat bar/in place of your chat bar is an idea.
So let's say you implement writing utensils and paper and my idea is when you hold the writing gear you have to click on an object to write it's name into the paper, you can't actually type into it, but you can write the objects into it.
Say Parchment and inked quill + maple tree = Custom parchment (maple tree)
Custom parchment and inked quill + Eve Worth = Custom parchment (maple tree Eve Worth)
So on and so forth until the word limit is reached on the parchment.
(If this comes to pass perhaps allow directional key words to be injected somehow.)
This is just my thought on a way it could work, unfortunately I cannot help with any technical hurdles you guys are all discussing.

Blazier · 2018-08-21 07:48:54

Here's potential way you could implement it so that you get 32-bits of data instead of 15: make a writable object able to hold a single object, similar to a container, but the item it's holding is not treated as an item, but just as a 32-bit ID that you can use to look up entries in your writing table. You can't remove the object that your writable object is containing, and it gets created with an auto-incremented ID when the writable object is first created.

Then any object can contain one or more writing-IDs, the IDs will move around with their object the same way containers work, and then you'll be able to have 4 billion different writing entries per server, instead of 32k.

It's a pretty ugly hack, so probably don't do it this way. I can see pitfalls with server code thinking a writing-ID is a carrot, and tracking it under food stats, so you'd have to have code that asks if it's a real ID or a writable ID based on whether it's in a special container spot. It's also not very general, and would take some work to make objects able to contain things other than regular IDs and writing IDs.

To solve this, you could use the upper bits of the object ID to specify what type of object it is (0, for regular, 1 for writing, 2... and so on), that way at least you don't have to check whether its in a special container slot, and there's no way your code would think a writing object is a carrot. Depending on how many different object ID types you'd want, you'd only lose the upper few bits, so you'd still get way more than 32k different writing-IDs (not to mention way more than 130k regular IDs too) with this method.

Last edited by Blazier (2018-08-21 08:15:24)

bENdI · 2018-08-21 13:17:38

CynderR wrote:

A small idea..
But since we know this will be used mostly to write racist comments (just being honest here), how about we have a limited list of content that can be written out.... Kinda like the dark souls communication.
It would still allow trolls to have fun by putting
"Danger, man but stop" and prevent things like "h**** ***** ** f*** *** *** *****"

before the discovery of rubber erasers, they used bread to erase graphite pencil marks.

so simply just erase it.

i want to randomly walk up to someone, ask them to read this paper and the only thing on the paper is

"(ascii text of the finger circle) gottem"

Joriom · 2018-08-21 13:32:29

CynderR wrote:

A small idea..
But since we know this will be used mostly to write racist comments (just being honest here), how about we have a limited list of content that can be written out.... Kinda like the dark souls communication.
It would still allow trolls to have fun by putting
"Danger, man but stop" and prevent things like "h**** ***** ** f*** *** *** *****"

If all you have to work with is, lets say, 10, 20 or 30 predefined messages - how would you imagine building culture around that? No for limited speach.

If you wind racial comments or something that offends you - just don't read it or possibly throw it away so it does not bother you anymore.

You being offended by stuff offends me. I'm triggered right now by you trying to suggest hard censorship. I believe you should *** you ***** **** of **** ****. Should I act on it? No. We should both get over it. Take my words with a grain of salt just like I take yours and lets walk different ways like civilized human beings.

Neo · 2018-08-21 16:47:16

I'm just glad we're getting writing. I'd probably copy and paste some guides from forums onto books in the game for noobs in future generations.

Last edited by Neo (2018-08-21 16:51:16)

Kinrany · 2018-08-21 18:32:30

What would be super neat, but probably a total headache to implement would be if the text would decay gradually, with characters smudging out every so often.

Single ID per text no matter how many copies there are is great for mass producing pamphlets

But if books were to decay, say a book becomes ragged after a few epochs or something, then there would need to be some way of recording which unique book this is, and which book type it is.

Books could just randomly tear when used. Make copies if you don't want to lose useful information.

(What kind of information would you have to store in books though?)
(Recording prophecies?)

Last edited by Kinrany (2018-08-21 18:37:32)

jasonrohrer · 2018-08-22 15:36:54

Christoffer, the reason I didn't implement generated objects the way you are suggesting is that the object IDs are represented as ASCII in the protocol, so I didn't want to unnecessarily inflate the size of messages. Currently, all object IDs are under 9999 (even auto-generated ones), so that's 4 bytes per ID, max (but only 1 byte for 0 IDs, which are the most common). If I push generated objects up into a higher ID space (an obvious choice), like 100,000+, then a whole heap of objects will have 6-bytes minimum in the protocol. Also, this won't save us from bugs when there's a data change, because the objects IDs in that generated space will change anyway (making a half-used loaf of bread turn into a clay pit, or whatever). The only safe thing is to assume we know nothing about generated IDs between server restarts, and revert all of them to real IDs.

As for the idea of "just subtract 1M from object IDs to get metadata IDs." I don't think that works, does it?

I mean, say you have object ID 13.

Now you add some metadata to it.... say the next free metadata ID is.... 1,000,021.

So what do we do here? We add them? The problem is that the "21" part interferes with the portion of the address actually used for the object ID. Adding gives 1,000,034.

So how do you get either the metadata ID or the object ID back out of that, once you've done it?

I guess I'm probably not understanding what you mean here.

It's not just enumerating objects with metadata, which we could do in the space between 1,000,000 and 4B.

We also need to get the on-the-ground object ID back out of that somehow.

And sorry, but no unicode...

But I do have this working correctly by using the upper 14 bits of the object ID, and it "just works," following the object around correctly through most of the server code with no changes. And the changes that did need to be made were minimal.

Joriom, yeah, if I was rewriting the thing from scratch, I'd do it differently. I'm just one guy, and I've got 34K lines of server code written already that assumes an object is represented by a 32-bit int (in the map, in containers, when held, etc.)

And Blazier, same answer to what you're suggesting. Keeping track of what an object contains is already the most complicated part of the server code (as the object is picked up, put down, put in a cart, worn, etc), and as items are added/removed from the container, or items decay/change in the container. So adding another "meta" slot for objects, and having that follow the object around as it moves around, in and out of containers and sub containers, would be a non-trivial code change.

In general, you're looking at a project that has been in development for 3+ years, has accumulated a lot of technical debt during that time, and has only one dev. And shockingly, the server is mostly bug-free (it has only crashed something like 4 times in 6 months, and all those crashes have been fixed now).

I'm really careful about launching off on major code changes at this point. So the changes end up being little band-aides on top of the existing band-aides.

The thing ends up growing into a giant ball of band-aides by the end. But after shipping 19 games, I'm okay with that, assuming that the software functions correctly. Elegant architecture is a fantasy, I think. I mean, show me one working, robust, complex, long-terms software system with an elegant architecture.

We're not building bridges here. We're more like running a legislature over decades in a changing political landscape. "So you want writing now, do you?" Maybe the only way to give it to you at this point is through executive order.

Christoffer · 2018-08-22 18:48:08

Jason, good to hear that you have found a solution that you are satisfied with. I'll answer your questions anyway, just for the sake of having an interesting discussion:

My idea is that you don't add meta data to a pre-existing object ID, but rather create new object IDs which carry a meta data connection from a known range (in the example 1,000,000 and above). So if a player writes "Hello World!" on a paper, and there are already 13 texts recorded into meta data object #0-12, then meta data #13 will now contain "Hello World!" and the paper with the writing will get the previously unused object ID 1,000,013. Wherever this object ID goes, the connection to meta data #13 will remain.
The base object type info (used for drawing and transition logic), can be carried in the meta-data.

The issue with bloating the ASCII messages can be solved with Base64 encoding/decoding of the object IDs. This very simple method allows storing of large numbers in a few ASCII characters. Four bytes are enough to cover the range 0-16,000,000 and one byte is enough for 0-63.

If you create new object IDs by adding to the upper 14 bits instead, I imagine you might still want to use Base64 to keep the ASCII size of the object IDs down?

CynderR · 2018-08-22 19:25:31

Joriom wrote:

You being offended by stuff offends me. I'm triggered right now by you trying to suggest hard censorship. I believe you should *** you ***** **** of **** ****.

Nice.

Thanks for your hard work Jay.
Great moderator choices...
Peace.

One Hour One Life Forums

#1 2018-08-20 20:47:01

Trouble with writing implementation

#2 2018-08-20 21:00:13

Re: Trouble with writing implementation

#3 2018-08-20 21:12:47

Re: Trouble with writing implementation

#4 2018-08-20 21:49:12

Re: Trouble with writing implementation

#5 2018-08-20 21:55:31

Re: Trouble with writing implementation

#6 2018-08-20 22:15:09

Re: Trouble with writing implementation

#7 2018-08-20 22:17:18

Re: Trouble with writing implementation

#8 2018-08-20 22:38:23

Re: Trouble with writing implementation

#9 2018-08-20 23:31:31

Re: Trouble with writing implementation

#10 2018-08-21 00:25:02

Re: Trouble with writing implementation

#11 2018-08-21 02:17:21

Re: Trouble with writing implementation

#12 2018-08-21 06:16:16

Re: Trouble with writing implementation

#13 2018-08-21 07:16:29

Re: Trouble with writing implementation

#14 2018-08-21 07:30:36

Re: Trouble with writing implementation

#15 2018-08-21 07:48:54

Re: Trouble with writing implementation

#16 2018-08-21 13:17:38

Re: Trouble with writing implementation

#17 2018-08-21 13:32:29

Re: Trouble with writing implementation

#18 2018-08-21 16:47:16

Re: Trouble with writing implementation

#19 2018-08-21 18:32:30

Re: Trouble with writing implementation

#20 2018-08-22 15:36:54

Re: Trouble with writing implementation

#21 2018-08-22 18:48:08

Re: Trouble with writing implementation

#22 2018-08-22 19:25:31

Re: Trouble with writing implementation

Board footer