The poll feature, in progress.

jasonrohrer · 2019-06-19 01:07:00

Here's the web UI that I can use to make a poll:

You can see that I have pretty fine-grained control over who gets a given poll and when they see it.

Here's what the poll looks like in the game:

This is shown to you after you die and hit OK, before you are shown the GET REBORN screen. This means that checking for a poll will never slow you down when logging into the game at startup.

Your vote is secured by your account key, so it's impossible to vote twice, or for a non-owner to vote, etc.

Joseph Stalin · 2019-06-19 01:34:50

I love it!

futurebird · 2019-06-19 01:36:26

Nice! I hope you share some of the results with us. I've always been curious about the difference between the polls some of us post here and what might happen if everyone was asked about various game features.

I'm personally most curious about how people feel about the level of violence, in the game.

jasonrohrer · 2019-06-19 01:38:34

Yes, the plan is for [Polls] to be a feature on the home page, listing the results of every poll.

jasonrohrer · 2019-06-19 01:39:54

FYI, murder rate on Sunday was 3%

Around 6000 deaths, and 200 murders.

200 murders in 24 hours is not very many... 8 per hour, or less than 1 per active family per hour.

Bob 101 · 2019-06-19 03:26:04

I think this is a step in the right direction if you ask the right questions.

Could be useful if it recorded their lineage/ Gen number. Also perhaps the age they died at.

Weekly polls post updates would be positive too. Like maybe the next day after people have played around with it, You can ask if they think negatively or positively about the changes.

Last edited by Bob 101 (2019-06-19 03:32:33)

Spoonwood · 2019-06-19 05:31:49

I'm not clear on what gets asked by the poll. I read two questions which I regard as distinct.

Also, since I've played on low pop servers and bs2, what is the context for the question?

Amon · 2019-06-19 05:53:47

You forgot to include a very much and sometimes not at all option to go along the indifferance.

RedComb · 2019-06-19 06:30:59

jasonrohrer wrote:

Here's what the poll looks like in the game:

Strangely worded, biased, and imprecise choices for responses.

You seem to ask about two distinct things (how much do you care if you live or die, then how much you care if your children live or die). This is called a double-barrelled question and is a big no no in academic research questionnaires.

While those questions might be related, it's entirely possible a person might feel one way about their own life/death and a different way about their children's life/death. They really should be presented separately so people have the option to respond to each topic.

Also, you include an opinionated statement at the end that doesn't relate to the possible answers of I love it/I hate it. The point about the last girl being treated like a queen, but life seeming to be cheap in all other instances, should really offer responses like "I agree/I disagree/no opinion"

Again, all three of these topics should be presented by themselves.

I've already mentioned possible sample error/selection bias, as well (because you are not including people who may have stopped playing recently, eg the 13% drop in Steam players over the last 30 days), and I think your already small pool of respondents doesn't benefit from narrowing the field even more by only offering the question to those with a certain number of lives left.

If you want to really capture some statistically significant results, you should be trying to randomly sample as many players as possible and sampling from a representative range of players. Those with many lives left, those with few, those with a moderate number, and so on... And you should be asking current players of the game, as well former players of the game (perhaps delineating between those who quit very recently versus those who have not played for more than a month).

As long as you have a means of determining who is who, you can still group the results as you see fit (e.g. those who are actively playing vs those who aren't).

But if you are going to rely only on responses from active players of the game who fall within a very narrow range of possible respondents, your results are likely to be skewed by selection bias and the incredibly small sample size.

I admit that even sampling from players who quit playing is problematic because you will have to actively convince them to participate via email or steam posts, which introduces voluntary response bias. Even so, you might be able to mitigate this and still get a sense of what kind of response potentially disgruntled players might give by creating a flag that groups any player who has returned to playing after more than three weeks of not logging in, so when they respond, they're results can be looked at separately from those who have been actively playing (and might respond differently because they are likely to be happy with the game )as shown by their continual play).

Furthermore, I think if a player logs in after a long hiatus (eg, more than three weeks without a log in), it could trigger an extra survey question asking them to give the reason for their extended absence from the game. This could include some premade answers, some not directly related to the game (eg, too busy at work, was on vacation, needed a break, wanted to play another game), some being more about dissatisfaction with OHOL (too many griefers, bored with late game, didn't like a recent update -- with a prompt to name/describe the disliked update), or simply a choice that says "other" and let's them write a very brief reason, say 120 characters).

Finally, every survey question that pops up after a life ends must offer a quick and painless way for people to opt out of the survey without interrupting their play time (to avoid people making random selections just to bypass the question),

What can I say, though? My graduate degree (applied linguistics) only had us focus on the basics of statistics as it related to the methodology behind some of the research in second language acquisition we had to understand, so I don't really know much about this topic beyond that. Certainly not as much as a statistician or actual peer-reviewed researcher with years of experience crafting valid questionnaires and mitigating bias in their selection of samples.

Still, if you want meaningful results that actually give you an insight into these kinds of topics, you need to have sound questions and methods for sampling/collecting data (and not expect an amateurish exit poll to tell you a whole lot).

Edit to add:

Tips for creating good research questionnaires

https://courses.lumenlearning.com/atd-h … -research/

https://psr.iq.harvard.edu/files/psr/fi … heet_0.pdf

https://www.sciencebuddies.org/science- … n-a-survey

https://opentextbc.ca/researchmethods/c … ionnaires/

https://www.pewresearch.org/methods/u-s … re-design/

^many of the above mention things I've said, like keeping questions and responses to one issue at a time, using direct language, avoiding opinionated questions, including a large enough scale of responses (eg, Likert scale 1-5, where 1 = strongly disagree, 2 = disagree, 3 = neither agree or disagree, 4 = agree, 5 = strongly agree)

On bias:

https://cirt.gcu.edu/research/developme … s/sampling

https://stattrek.com/survey-research/survey-bias.aspx

http://www.scholarpedia.org/article/Sam … pling_bias

https://web.ma.utexas.edu/users/mks/sta … pling.html

All of the above are just random resources found from googling while on my phone so it's not easy to be picky or summarize their best points.

Still, I hope Jason will take this stuff into consideration when creating a survey or poll about certain issues in the game.

If he wants statistically significant results with correlations that actually mean something, crafting good questions and sampling a representative set of respondents is vital.

If he only intends the survey to provide something he can point to as justification that players like feature xyz so the critics should be silent, well... then only selecting from a very specific group of current players (and omitting those who may have quit because of a given feature) and using imprecise, double-barrelled questions laced with opinion will certainly suffice.

Last edited by RedComb (2019-06-19 10:05:52)

Twisted · 2019-06-19 11:14:54

RedComb wrote:

jasonrohrer wrote:
Here's what the poll looks like in the game:
Strangely worded, biased, and imprecise choices for responses.
*SNIP*

That's not an actual real poll, it's just a test for the character limit and an example of how the poll looks ingame.

jasonrohrer · 2019-06-20 00:01:02

Yeah, sorry for the confusion. That's not a real question. It's an unusually long poll question with dummy text to test the spacing, etc.

jasonrohrer · 2019-06-20 00:10:04

My understanding is that the point of "good sampling" is to handle cases where you really can't ask everyone due to issues of practicality. So you don't want to just send to age-42 men by accident and assume their results represent the entire population.

But if you can actually ask everyone, you should, right? You don't need to sample in that case. You just ask everyone. This is like the census in the United States. The goal is to actually get an answer from everyone. Or the ideal form of governmental voting.

(Yes, when you count votes, you're sampling the opinion of the subset that chose to vote).

(Obviously, I'm sampling the opinion of people who actually play the game.)

Chasing down people who doing play anymore is a separate issue, and easy enough to do (I can see who stopped playing, and send them an email with a web-based poll). However, this is impossible to do for Steam players, sadly, because I have no contact information for them.

RedComb · 2019-06-20 04:32:54

jasonrohrer wrote:

My understanding is that the point of "good sampling" is to handle cases where you really can't ask everyone due to issues of practicality. So you don't want to just send to age-42 men by accident and assume their results represent the entire population.
But if you can actually ask everyone, you should, right? You don't need to sample in that case. You just ask everyone. This is like the census in the United States. The goal is to actually get an answer from everyone. Or the ideal form of governmental voting.
(Yes, when you count votes, you're sampling the opinion of the subset that chose to vote).
(Obviously, I'm sampling the opinion of people who actually play the game.)

Unless every person who currently plays the game plays it in the next 4-5 days and takes your poll, you aren't asking everyone who actively plays it, just sampling from those who play over the next 4-5 days and who meet the minimum lived hours requirement you set.

By limiting the poll to ~5 days (and 10 minimum lives lived), those who are polled by definition represent a sample of those who happen to play and qualify for the poll while it's available. For whatever reason, they are able to play and respond, while those who still might be actively playing didn't play or qualify to take the poll for whatever reasons.

Maybe they were on vacation, maybe they've been busy at work, maybe they've been engrossed in a different game, etc. Whatever the case, you are not actually asking everyone who plays, just asking those who happen to play and qualify during the time frame of your poll (i.e., sampling from active players).

This isn't always a problem as long as the sample is representative of those who actively play, but the fact that some players will be omitted needs to be kept in mind when looking at the result. Essentially, you need to keep in mind that you are sampling from active players and not asking everyone, so take the responses at face value.

Nonresponse Bias
Sometimes, in survey sampling, individuals chosen for the sample are unwilling or unable to participate in the survey. Nonresponse bias is the bias that results when respondents differ in meaningful ways from nonrespondents. Nonresponse is often problem with mail surveys, where the response rate can be very low.
https://stattrek.com/statistics/diction … nse%20bias

Also....

Chasing down people who doing play anymore is a separate issue, and easy enough to do (I can see who stopped playing, and send them an email with a web-based poll).

And because only a subset of those people who receive emails will respond (possibly the most passionate), then you get:

Non-response bias – Occurs when the subjects that respond are perhaps different in some way from those that choose not to respond. In this case, it may be that the correct group is being sampled, but some are refusing to respond. For example, an email survey would be more likely to draw responses from those with desk jobs or those that spend more time at a computer than perhaps blue-collar workers or those who use computers much less.
https://cirt.gcu.edu/research/developme … s/sampling

Just important to realize that an email survey (especially for former players) is likely to produce skewed results because the most passionate people will be the ones likely to respond.

However, this is impossible to do for Steam players, sadly, because I have no contact information for them.

And again, through no fault of your own, this means you are certainly sampling and not able to ask everyone who no longer plays the game. This introduces the likelihood of Selection Bias

Not saying a lot of this is avoidable, just that it needs to be kept in mind when drawing possible conclusions from any results you get.

There are ways to test how good the samples are like taking multiple samples and then examining the variance in the responses between the two samples. In truly representative samples, there will be zero variance (and with infinite samples, the distribution of responses will eventually approach a normal distribution).

It's hard to explain what these means (more found at Statistics Sampling). Like I've said before, I'm not a statistician, just a lay person. Still, just trying to caution you that the results of these kinds of polls can be problematic based upon the types of bias and sampling error present.

Apologies for rambling. Might go play OHOL now just to see if I can trigger the poll.

Last edited by RedComb (2019-06-20 04:36:23)

jasonrohrer · 2019-06-20 05:11:10

Well, here's the thing:

What would be the advantage, if I'm going to ask EVERYONE who plays the game in the next 4 days, to randomly (or evenly) sample a subset of them? To get a more representative sample? Representative of what? How could it be more representative of the people who actually play than including everyone who actually plays?

I mean, yes, "Everyone who plays in the next four days" is a sample.... but a pretty unbiased one. If we're sampling in the game, we can't do any better than asking the people who are actually running the game.

And there is no non-response bias, if they are forced to answer (allowing for a "no opinion" option). They simply can't "not mail it back."

I agree that an email survey is fraught with many more problems.

Whatever · 2019-06-20 05:49:54

This is a great new feature +1

Random poll ideas:

Would be interesting to see how many people use mods to play the game, maybe a question like this
- Did you modify the textures of the game?
- Do you play using a zoom out mod?
(but say in the question that there is no punishment, so that people are not afraid to answer)

How many people like property fences? war swords? the curse system? language update?
its hard to really know if you dont make a poll, but maybe you dont want to hear the answers to these questions,
you already made the updates and you dont want to revert them

RedComb · 2019-06-20 06:58:58

jasonrohrer wrote:

Well, here's the thing:
What would be the advantage, if I'm going to ask EVERYONE who plays the game in the next 4 days, to randomly (or evenly) sample a subset of them? To get a more representative sample? Representative of what? How could it be more representative of the people who actually play than including everyone who actually plays?
I mean, yes, "Everyone who plays in the next four days" is a sample.... but a pretty unbiased one. If we're sampling in the game, we can't do any better than asking the people who are actually running the game.
And there is no non-response bias, if they are forced to answer (allowing for a "no opinion" option). They simply can't "not mail it back."

Not really sure I understand the first few questions. I'm pointing out surveying everybody who plays over the next 4 days is a sample of the overall population of players. And even then, those who play and respond over the next 4 days are merely a sample of active players within a 4 day time frame. As far as how representative this kind of sampling is: surveying ALL players is most representative (and pretty much impossible/impractical), then sampling a random sample of all players is the next best approach, then sampling a random sample of players who play within a 4 day time frame would be the next best, etc.

I am also pointing out that when you said (in a previous post) that you weren't taking a sample because you are surveying everyone who plays the game, you were in fact were taking a sample because everybody who plays the game might not necessarily play the game over the next 4 days.

This matters because this *MIGHT* skew the results and support an erroneous conclusion. It is good to be mindful of stuff like this because considering bias and sample error is important to researchers when considering how valid their results are.

And you can't be sure it is a "pretty unbiased" sample until you've done the work of looking at variance and checking if the sampling distribution follows the 68-95-99 percent rule. Again, please refer to the section Statistics Sampling for more explanation. This is beyond my ability to properly explain. I am more of an Language Arts person, but I can say, understanding and being mindful of sampling error and bias when conducting surveys is very important if you're going to look to the results for insight into a specific question.

As for non-response bias, I was highlighting that SOME of the target population (players of the game) are not able to take the survey while others can. This is the very definition of sampling bias. Perhaps it's more a case of selection bias/undercoverage since they aren't actually being presented with your survey if they don't log into the game and meet your requirements before the survey pops, but the fact remains, if you want to survey people who are playing your game, anybody who doesn't play within the 4 day window of your survey will be excluded from the survey even if they are otherwise active players of your game.

In other words, you might be inadvertently sampling more "hardcore" players of your game than sampling "casuals" who only play sporadically and randomly take breaks, you might be inadvertently omitting people who are on vacation (summertime), you might be inadvertently sampling more from people who are unemployed or students with time to burn versus those who work 70 hours a week, etc.

https://stattrek.com/survey-research/survey-bias.aspx

Again, it's not a certain thing these types of things will skew your results, but it should be considered when you look at your results. Things like this can and do distort results to all kinds of surveys. If it matters to OHOL, who knows, but that's what looking at variance and comparing sampling distributions illustrates.

That's the only point I've been trying to make. It's incredibly easy to draw inaccurate conclusions from skewed results if you're unaware of factors that might distort those results. I admit I'm no expert on this topic, but I don't think you are either (no offense meant). Just wanted to raise these points because they are the source of a lot of incorrect conclusions when people conduct surveys.

Last edited by RedComb (2019-06-20 13:13:11)

Sukallinen · 2019-06-20 09:40:17

jasonrohrer wrote:

Chasing down people who doing play anymore is a separate issue, and easy enough to do (I can see who stopped playing, and send them an email with a web-based poll). However, this is impossible to do for Steam players, sadly, because I have no contact information for them.

Contacting via email OR impossibility to contact steam players is why I suggested to put voting possibility (also) before you start to play. Maybe have that as a possibility, or allow them some other way (via your homepage ohol ?) to vote, then inform them of said possibility. Many people use separate email account for "business" and one for games - so games usually get NO response whatsoever (from me at least).

Also, do I need to /die ten times in order to vote, haven't seen voting possibility ? I usually live to 60 as many do. So that'd mean eight lives (two I get forgotten by mom riding horse or such).

Does the voting count all lives or just since some update ?

Last edited by Sukallinen (2019-06-20 09:42:13)

olooopo · 2019-06-20 12:11:11

@RedComb
How do you take the variance from a nominal scale or is there another way for checking the sample bias for the type of surveys Jason is using? The provided link only shows an example for a Likert-scale which is kinda obvious.

RedComb · 2019-06-20 13:08:31

olooopo wrote:

@RedComb
How do you take the variance from a nominal scale or is there another way for checking the sample bias for the type of surveys Jason is using? The provided link only shows an example for a Likert-scale which is kinda obvious.

I don't know.

I made my post before I saw that the actual poll responses Jason used weren't based on a Likert scale. I just assumed he would use a Likert scale (and suggested he do so in earlier posts) because it measures preference and can be plotted as a distribution. Just seemed to make sense if gauging the preferences of players was what he was after.

There are ways to check if two categorical variables in a sample are likely to reflect real associations between the variables in the population (chi-squared test), but I don't know much about that or how it relates to determining what degree, if any, of error exists in a sample.

Like I tried to say before, I'm a huge amateur when it comes to this stuff and honestly don't have a lot of grounds to post much about it. I'm not a "math person" at all, and this kind of stuff is way above my head. It's just something that is stressed in the Social Sciences (which Second Language Acquisition research tends to fall under).

I guess the TL;DR of what I was trying to say to Jason is that he should be careful when he looks at his results. They might provide a misleading view of what people think (or they might not).

I think there are a lot of people on this forum who are really familiar with statistics, so they probably could answer topics like this better than I can and maybe they have some good suggestions. I imagine it's probably overly cautious to care about this stuff for something as informal as an in-game poll. I was just worried a poll like this might turn out to be misleading and could be less helpful than intended.

Hard to explain, but last year I got to witness an independent game studio (Portalarium) send out a survey to its most loyal and fervent players, attempting to use the overly positive feedback they were likely to give to mislead one of its then publishing partners (Travian) in the hopes Travian would continue supporting Portalarium. The broader playerbase became aware of this survey (including disgruntled players, of which there are many), and the link to the survey was more widely distributed among the playerbase (much to the chagrin of Portalarium).

When Travian saw some of the scathing responses, I think it was an eye-opening experience for them (and Portalarium was pissed they had been caught doing something like this). Travian had a community manager take control of Portalarium's forums for awhile and it was obvious Portalarium had lost some degree of their autonomy. There were several angry posts made by Portalarium devs following the kerfuffle.

Ultimately, I think the experience was one factor (among many) that convinced Travian to drop their partnership with Portalarium altogether. However, the point remains, if Portalarium had been successful in only surveying its most diehard fans and excluding critical voices, the conclusions Travian might have drawn from such a misleading survey would have been very distorted.

That was clearly purposeful deception though, and I don't think Jason intends anything of the sort. I just think an extreme example like that illustrates how sometimes excluding groups of players (even if unintentionally) presents a distorted picture when it comes to measuring the sentiment of the overall playerbase.

Last edited by RedComb (2019-06-20 13:09:49)

Hstrike · 2019-06-20 15:30:30

Why is there no option to say that we are satisfied with 18 lives? Now I'm forced to say that I don't have an opinion. There's also no option for in between 12-18.

Last edited by Hstrike (2019-06-20 15:31:38)

MrsDuckGirl · 2019-06-20 15:36:10

There should be an option "ask me again later" for when people don't feel like replying right away or just don't have time for that.

RedComb · 2019-06-20 18:40:39

Hstrike wrote:

Why is there no option to say that we are satisfied with 18 lives? Now I'm forced to say that I don't have an opinion. There's also no option for in between 12-18.

Said the very same thing to him, in a more roundabout way, on Steam:

This is why it's better to directly ask people "how much do you like the limit on lives?" and then give them a 1-5 scale to express their preference (1=strongly dislike, 2=dislike, 3=neutral/no opinion, 4=like, 5=strongly like). This way, you can look at the results and group them by all kinds of categories (players with almost no play time vs players with tons of play time, long-term players vs newer players, etc). Furthermore, you eliminate all the noise about specific numbers of lives and keep the focus only on the actual question at hand: how people feel about you limiting lines.
If the results warranted it, you could always ask follow-up questions about specific numbers/solutions, but for now, if you're curious what people think of you limiting lives, you need to ask them that and only that and give them a way to respond that allows you to create a distribution (1-5 Likert scale). It is so much better to gauge opinions this way. It keeps people's response focused on what they think about limiting lives, not if they think 18 is enough, or too few, or whatever. It also lets you plot their responses as a distribution and see if it conforms to the 68-95-99 percent role.
As for specific solutions, those are questions for another time....
https://steamcommunity.com/app/595690/d … 4732310137

RedComb · 2019-06-20 18:53:35

MrsDuckGirl wrote:

There should be an option "ask me again later" for when people don't feel like replying right away or just don't have time for that.

Suggested to him by myself and Starknight_One here (see top two quotes).

Pretty disappointed he is already trying to point to the early results of a one day old survey (plagued by convoluted responses) as if they mean anything significant...

https://steamcommunity.com/app/595690/d … 4731837102

Tried to give him the benefit of the doubt, but am losing my confidence he was earnestly looking at what players thought instead of trying to use the survey responses to shame the "minority" of unhappy voices.

BladeWoods · 2019-06-20 21:44:10

Latest poll has no option for saying that 18 is enough.

jasonrohrer · 2019-06-20 22:32:53

Good catch, BladeWoods. Hmmm...

One Hour One Life Forums

#1 2019-06-19 01:07:00

The poll feature, in progress.

#2 2019-06-19 01:34:50

Re: The poll feature, in progress.

#3 2019-06-19 01:36:26

Re: The poll feature, in progress.

#4 2019-06-19 01:38:34

Re: The poll feature, in progress.

#5 2019-06-19 01:39:54

Re: The poll feature, in progress.

#6 2019-06-19 03:26:04

Re: The poll feature, in progress.

#7 2019-06-19 05:31:49

Re: The poll feature, in progress.

#8 2019-06-19 05:53:47

Re: The poll feature, in progress.

#9 2019-06-19 06:30:59

Re: The poll feature, in progress.

#10 2019-06-19 11:14:54

Re: The poll feature, in progress.

#11 2019-06-20 00:01:02

Re: The poll feature, in progress.

#12 2019-06-20 00:10:04

Re: The poll feature, in progress.

#13 2019-06-20 04:32:54

Re: The poll feature, in progress.

#14 2019-06-20 05:11:10

Re: The poll feature, in progress.

#15 2019-06-20 05:49:54

Re: The poll feature, in progress.

#16 2019-06-20 06:58:58

Re: The poll feature, in progress.

#17 2019-06-20 09:40:17

Re: The poll feature, in progress.

#18 2019-06-20 12:11:11

Re: The poll feature, in progress.

#19 2019-06-20 13:08:31

Re: The poll feature, in progress.

#20 2019-06-20 15:30:30

Re: The poll feature, in progress.

#21 2019-06-20 15:36:10

Re: The poll feature, in progress.

#22 2019-06-20 18:40:39

Re: The poll feature, in progress.

#23 2019-06-20 18:53:35

Re: The poll feature, in progress.

#24 2019-06-20 21:44:10

Re: The poll feature, in progress.

#25 2019-06-20 22:32:53

Re: The poll feature, in progress.

Board footer