Mitch's Poll - Wood Coaster Results

Lord Gonchar said:

sws:
A group of 14 coaster geeks go to Korea and have the time of their lives. They have such happy memories of their fabulous trip that they all immediately participate in Mitch's poll and their selections reflect the overall wonderfulness of their trip. Their participation could not be considered a random sampling given their preselcetion bias. Thus bias is entered into the data collection. There doesn't even have to be any malice involved. It could be all warm and fuzzy bunnies and sweet smelling flowers.

But asking people at random and still including those 14 in the end would be statistically valid?

I have a hard time with that.

Let me see if I can explain it in a way that makes sense. It's been >25 years since I took a college statistics course, so I may not be completely accurate.

For the time being forget about the sample size. Regardless of the sample size, it would still not be considered statistically valid because of the way the sample was obtained. By using this self-reporting method, bias has been entered into the selection of participants. People who respond to the poll are more likely to have strong opinions one way or the other. With a large sample size, these biases would tend to cancel out, but regardless the bias selection would make the study statistically invalid.

Random sampling implies that all participants are created equal. To obtain a random sample, the study population is defined in advance, then a computer (or some other method) randomly determines who from that population is selected to participate. This is how you generate a random sample. A random method is used to determine who is allowed to participate in the study. The participants do not determine in advance if they participate - random chance makes that selection.

So the point I was trying to make is that regardless of the sample size, the poll would still not be considered statistically valid, because the selection of participants was biased.

Please note I'm not trying to blame Mitch's poll. It's a fun little distraction. I'm just suggesting people not wage religious wars based on the results. It's not that scientific for all of the reasons that people have previously discussed. It's better than alot of the other polls out there.

Scott

Lord Gonchar said:
Again, it seems you just can get past the idea that 14 people could respresent the whole.

Because they can't. Hundreds of years of math tell us that. That, and Brian Noble. :) It has nothing to do with what passes my scrutiny.

sws said:
Let me see if I can explain it in a way that makes sense.

No, you guys are doing great.

For the time being forget about the sample size. Regardless of the sample size, it would still not be considered statistically valid because of the way the sample was obtained.

Still with you. 100%

Random sampling implies that all participants are created equal. To obtain a random sample, the study population is defined in advance, then a computer (or some other method) randomly determines who from that population is selected to participate. This is how you generate a random sample. A random method is used to determine who is allowed to participate in the study. The participants do not determine in advance if they participate - random chance makes that selection.

But the participant still has to agree after they're chosen at random, correct? Or at least agree to potentially be chose at random. At some point in the process the subject has to 'opt-in' so to speak.

That's my first question - What's the difference in the two ways of opting in? Because to me, the people who agree to participate probably want their opinion heard in both cases...that's why they participate.

My second is:

What if Mitch used the entire pool of enthusiasts as his study population and then a computer chose 629 at random to participate. The computer could conceivably choose the exact 629 people who voted by choice in Mitch's poll...and they would presumably vote the same way.

Why is that result (which is identical) valid?

Jeff said:
Because they can't. Hundreds of years of math tell us that. That, and Brian Noble. :)

Actually Brian said 14 could be valid...and so would the entire history of math.

On top of that, what if the whole is only 15 or 20 or 25 (which it very well may be in this case) - then 14 is plenty.

Your argument seems to be that 14 can never be valid. What did 14 ever do to you? :)

Gonch, believe it or not I think we are actually making progress here. Little grasshopper, let us continue. :)

But the participant still has to agree after they're chosen at random, correct? Or at least agree to potentially be chose at random. At some point in the process the subject has to 'opt-in' so to speak.

Yes, you are correct.

That's my first question - What's the difference in the two ways of opting in? Because to me, the people who agree to participate probably want their opinion heard in both cases...that's why they participate.

Ok. I will agree with you there also. The difference is that in Mitch's poll, participants have selected themselves, and therefore bias is entered into the selection process. It is not a random process. That is different from a nonbiased computer randomly selecting people to participate.

My second is:

What if Mitch used the entire pool of enthusiasts as his study population and then a computer chose 629 at random to participate. The computer could conceivably choose the exact 629 people who voted by choice in Mitch's poll...and they would presumably vote the same way.

Why is that result (which is identical) valid?

Because the chances of that happening on a random basis would probably be a gadzillon to one. You would stand a better chance betting your house on the Detroit Lions this weekend.

Jeff said:

Because they can't. Hundreds of years of math tell us that. That, and Brian Noble. :)

Actually Brian said 14 could be valid...and so would the entire history of math.

On top of that, what if the whole is only 15 or 20 or 25 (which it very well may be in this case) - then 14 is plenty.

Your argument seems to be that 14 can never be valid. What did 14 ever do to you? :)

There is nothing inherently wrong with 14. Take this bizarre example: After consuming massive amounts of alcohol this holiday season, I design a ridiculous study. My null hypothesis is that if I pay people 20 bucks, they will allow me to cut off their legs with a chainsaw. I randomly dial 14 phone numbers, and to my surprise all 14 people say no and hang up on me. Note the sample was selected randomly. Final results: "Yes 0, No 14." I can then run a non-parametric statistical analysis and I suspect the p-value will be less than 0.05, causing me to reject the null hypothesis. I'm depressed to learn my study will never be published and resume drinking heavily.

So 14 is not necessarily evil. 13, on the other hand, I would never trust. :)

Scott

sws said:
Gonch, believe it or not I think we are actually making progress here. Little grasshopper, let us continue. :)

Yes, let us. I'm still not sure I'm getting the answer here.

The difference is that in Mitch's poll, participants have selected themselves, and therefore bias is entered into the selection process. It is not a random process. That is different from a nonbiased computer randomly selecting people to participate.

So because it's open to everybody that wants to participate rather than picking people at random until it finds enough people wanting to participate, it is biased. I understand that's the case, but still not why it is the case.

I can't wrap my head around that. Mostly because:

Because the chances of that happening on a random basis would probably be a gadzillon to one. You would stand a better chance betting your house on the Detroit Lions this weekend.

But that doesn't change the fact that it's one of the possible samples that would be enitrely valid if chosen at random. It seems to me that the selection process shouldn't make a difference if the sample ends up being the same...and valid.

Seriously, in that scenario the only difference is the selection process. The sample, the data, the conclusion - it's all identical. Using complete scientifically correct methods, you could achieve an identical poll to Mitch's.

But somehow Mitch's is not valid and statisically irrelevant. My logical self just won't accept that two identical outcomes aren't the same.

So 14 is not necessarily evil. 13, on the other hand, I would never trust.

13 is just misunderstood. :)

Jeff ignored my question. Can someone answer it? I want to know how this poll should be administered, if it is not being done correctly. No hypotheticals allowed. The goal is a poll for ranking coasters. How should it be done to be considered the "right" way?

I doubt it could be done. Not enough people have been on enough rides to ever create a large enough sample. I think it could be done with US coasters from a randomly selected group of American enthusiasts, but not for a global poll.

I know Mitch asks participants if they rode each coaster they rank in the year the poll is being taken. Here's my question: is a no answer weighted any differently than a yes?

Here what I'm getting at: people's memory of an event that happened years ago can become less accurate and vivid as time passes. How is it possible to rank a coaster based on the memory of riding it 4, 8, 10+ years ago? Case in point: I have not ridden the original Elitch Gardens Mr. Twister since the year it closed (1994) and yet I rank it against coasters I've ridden in the 14 years since. Mr. Twister was a kick-azz coaster...or at least that is how I fondly remember it. Could it be that it wasn't as good as I remember? Or was it better than I regard it? And how is my ranking affected?

Or consider this: what if my only memory of Hercules was during it's first year of operation and someone else's occured during its last year of operation. When we submit our ballot we indicate that we also both also rode Voyage this year. Which comparison is the more valid?

It's like when, years later you rewatch a movie you loved as a kid only to realize it is a pretty crappy movie. Would it be fair to rank it against a more recent film based solely on your feelings as a child?

Am I making sense? lol

But that doesn't change the fact that it's one of the possible samples that would be enitrely valid if chosen at random. It seems to me that the selection process shouldn't make a difference if the sample ends up being the same...and valid.

Seriously, in that scenario the only difference is the selection process. The sample, the data, the conclusion - it's all identical. Using complete scientifically correct methods, you could achieve an identical poll to Mitch's.

But somehow Mitch's is not valid and statisically irrelevant. My logical self just won't accept that two identical outcomes aren't the same.

Actually the selection process is the key to any study. Everything else follows from that step. Numbers and statistics can be manipulated to prove whatever your agenda happens to be. Numbers are actually quite evil by nature. :) The key is to randomly select the sample from the population to be studied.

Now there also has to be an accepted cut-off point that is considered "statistically significant." Generally this is considered p<0.05, which means there is still a 5% chance that this statistically significant result happened purely by random chance alone. You could have more stringent criteria, say p<0.01. This would increase the likelihood that your results are real and did not occur by random chance. However this would also increase the chance of missing an actual significant result.

So the results could be the same - purely by random chance. The better the design of the study, the larger the sample size, etc, all decrease this risk.

Even a well designed study can run into problems purely by chance. Next example: A new drug, Magic Bullet, is shown to increase survival in patients with cancer. Patients are randomly assigned to receive Magic Bullet or placebo by computer generated numbers. However when the data is analyzed, it was found that patients receiving placebo, on average, were 15 years older than the Magic Bullet patients, purely by chance. This could obviously explain the results of the study rather than Magic Bullet. This sometimes happens, although you would hope that if the number of participants is large enough, these differences will wash out in the end. The devil is in the details. There is a power test that can be done before you start your study to help determine the number of participants you need to detect a difference.

13 is just misunderstood. :)

No, I think 13 is actually quite evil. But then again, maybe I'm just biased. :) ;)

Jeff said:
I think it could be done with US coasters from a randomly selected group of American enthusiasts, but not for a global poll.

So how do you do it (randomly select the sample, that is) for a US coasters only poll offered to American enthusiasts? Don't forget, not everyone has ridden every coaster and not everyone (randomly selected or not) will give your poll the time of day.

Re: Chuck and the ballot with 12 coasters

My Mitch's ballots end up looking like this all the time. I really only care to weigh in for the top 25 so I never rank anything that doesn't have a fighting chance of getting in there. My steel ballot has entries for S:RoS SFNE and no other steel at SFNE (though I've ridden them all multiple times). If you looked at my steel ballot, you'd think I'd only ridden spectacular coasters and nothing medicore in my whole life.

I have no idea, Carrie. It's not something I've spent a lot of time thinking about (nor do I have much desire to). Just because I don't have a good idea about how to design a better study doesn't mean that I can't find the obvious flaws in this one.

Of course it doesn't. Who said it did? You spoke about your experience in media leading you to being sensitive/appreciative toward the right way to do polls and studies. I thought that meant you could be part of the solution.

At best, Mitch's poll is a quasi-scientific attempt to quantify an unquantifiable. Which doesn't mean it's not fun, or that effort isn't put into making it as objective and useful as possible. It's just that I wouldn't put any more stock in its results than I would any other unscientific method of acquiring and analyzing data.

All the fun happens when I'm away...the *issue* with small sample sizes (per statistical and epi methodologies) is that you cannot simply TRUST that your sample is "representative" of the population of interest, or that the results obtained are "generalizable" in the sense that other similar studies would yield the same results. Clearly, the representative idea gets hammered home when you consider that less than 5 percent of enthusiasts worldwide even fill out a ballot...and then 2 percent of that sample can even compare some rides? to be fair to Mitch, he does some very beneficial determinations, like including each comarison as a separate vote rather than each ballot - increases the power of the observations to some degree. How to "do better"? Require that every card-carrying member of every coaster club fill out a ballot, including every ride they've ridden. Obviously there are some flaws in that methodology as well... ;)

I'm trying really hard to remember where in this thread anyone said Mitch's poll was flawless. I can't seem to find it. ;)

ApolloAndy said:Re: Chuck and the ballot with 12 coastersMy Mitch's ballots end up looking like this all the time. I really only care to weigh in for the top 25 so I never rank anything that doesn't have a fighting chance of getting in there. My steel ballot has entries for S:RoS SFNE and no other steel at SFNE (though I've ridden them all multiple times). If you looked at my steel ballot, you'd think I'd only ridden spectacular coasters and nothing medicore in my whole life.

That really was only part of my questioning in that ballot. HE HAD NO OTHER Non US coaster listed other than T Express.Im not saying its a falsivication. Just quite intriguing that its with the list he has.

John Knotts said:I'm trying really hard to remember where in this thread anyone said Mitch's poll was flawless. I can't seem to find it. ;)

I never either. The cream rises and more importantly holds its possition near the top. I look at it this way. If your favoirte is 50th in this poll. I can look forward to 49 possible better coasters than I've ridden WOOHOO!

Unless I've already ridden them and they all stunk!

^ Yep. And if Voyage had won this poll, this thread would probably be about 4 pages shorter. ;)