Mitch's Poll - Wood Coaster Results

^We prefer the term "rocket surgey" - has that Dr. Strangelove appeal. ;) :)

Lord Gonchar said:
When parks base their entire marketing stategy on something like the Golden Tickets, this is rocket science and brain surgery combined in comparison.

Did I mention that I don't know what that has to do with the validity of this poll's methodology? :)

No, not at all. I had no idea. :)

It's a poll. Nothing more than a collection of opinion for analyzation. Interpret the results however you choose.

Again, put it in perspective. It's a poll...of roller coaster dorks...for roller coaster dorks...about which coasters the dorks likey the mostest.

This is one time I'm not hung up on the details. Regardless of the validity of the methodology or the statistic relavance or the intergrity of the sample or whatever the hell is supposedly wrong with the info or how it's collected or applied, the results are adequate (arguably outstanding) for the purpose they're meant to serve.

And yes, that's arguing adequacy against validity - but absolute validity doesn't matter in this case. So...

Jeff:
I don't know what that has to do with the validity of this poll's methodology?

My POV is that the current level of validity of this poll's methodology, while not perfect, is accurate enough to serve its purpose.

And I disagree, so there.

So what would you change? The way the info is collected or the way it's interpreted? Or both? And how?

That's not a challenge, just curious.

Mitch puts all the ballots up online. The data is there for the taking. It'd be interesting to see someone spit out some alternative results.

There isn't enough data... that has been my point all along. The weighting system itself can't compensate for a small sample. Comparing small samples to big samples is even more flawed, since in polling terms it creates a margin of error for each individual ride. In aggregate those margins add up to make the whole thing worthless.

Thinking back to what Brian was talking about, the sheer fact that the poll is so laborious to fill out causes a pretty serious hit on the kind of people who would be willing to fill it out as well. You have to feel invested in order to do it. No wonder the sample size is too small.

Myself, I wouldn't tally anything with n < 30....but that might be my inner statistician coming to the surface (thru the chimney of course). :)

Jeff said:
There isn't enough data... that has been my point all along. The weighting system itself can't compensate for a small sample. Comparing small samples to big samples is even more flawed, since in polling terms it creates a margin of error for each individual ride. In aggregate those margins add up to make the whole thing worthless.

But is it really comparing small data to large?

I mean, to me, what it comapres is each ballot to itself (your rankings giving wins and losses based on placement) and then to the whole (all the individual rankings being totaled).

I still maintain that 629 voters is plenty to make reasonable guesses at what enthusiasts think of coasters. The only question I have is where to draw the cutoff that makes the "not enough of a sample" line (See Gator's n < 30 comment) and if that's what you mean by comparing large and small samples, then ok I'm with you.

Thinking back to what Brian was talking about, the sheer fact that the poll is so laborious to fill out causes a pretty serious hit on the kind of people who would be willing to fill it out as well. You have to feel invested in order to do it.

I disagreed when Brian said it (although not directly) and still do. The poll is open to anyone willing to do a ballot. It's going to attract people who care enough about coaster rankings to want to participate and see the results. The sample base is exactly the people you want - coaster dorks who care strongly about ranking coasters. That's the opinion the poll is after. That's the entire point!

A poll by the dorks for the dorks.

No wonder the sample size is too small.

Yeah, I keep hearing that although no one can seem to explain exactly why 629 ballots isn't enough to generate a reasonably accurate representation of enthusiast opinion.

Yes they can, you're just choosing to ignore it. 14 people's opinion on a ride does not a consensus make. If 14 people can change the outcome of the poll, then the methodology is wrong.

Ok, then let me rephrase the question.

If 14 people can change the outcome of the poll, then the methodology is wrong.

Why?

Because 14 is statistically insignificant.

Based on what?

What I'm trying to get at is why 14 is statistically insignificant in this situation. No one seems to be able to tell me why it is...just that it supposedly is.

I feel like I'm banging my head against a wall. Statistics is a science, a discipline, something that PhD's sit around and think about all day. The general theory goes that the smaller the sample, the less it likely represents a bigger reality.

Think about it... as the number gets smaller and you get closer to 1, it's less and less likely that the statistic is representative of the whole. 1 would be skewed to 100% even if you got the one object that satisfied one condition out of millions. From 2, you could get 100% again, or at best 50%, even if the whole is in actuality 12% or 93%. And so on and so forth, until you get to 14, which out of thousands of enthusiasts is not significant.

And even beyond the pure numbers, as with any experiment, you have to be on the look out for variables which can skew and bias the results, which we've already talked about.

I disagreed when Brian said it (although not directly) and still do.

Which makes you wrong. The very nature of an "opt-in" poll leaves it open to selection bias. This isn't an opinion. It's something that anyone with even a modest training in social science and statistics will tell you.

That said, it is possible that "14" is big enough---it depends on the population the sample is drawn from, and the underlying distribution of preference. The problem is, figuring that out is hard.

At the end of the day, I like Mitch's methodology. I suspect he needs to think harder about his cut-off number, and reason a little about confidence values, etc.

Ok, so WHO is on first, right?

Jeff said:
I feel like I'm banging my head against a wall.

Me too.

The general theory goes that the smaller the sample, the less it likely represents a bigger reality.

Great. So why is 14 a statistically insignificant number in this case?

And so on and so forth, until you get to 14, which out of thousands of enthusiasts is not significant.

So which number is significant? 15? 30? 48? 62?

The point is that this is scientific and a discipline. There has to be a reason a number isn't relevant based on more than it seems too small.

Brian Noble said:
Which makes you wrong. The very nature of an "opt-in" poll leaves it open to selection bias. This isn't an opinion. It's something that anyone with even a modest training in social science and statistics will tell you.

Yup. I understand.

I think the point with this poll is that you want that bias - Well traveled enthusiasts who care strongly about ranking coasters.

Wouldn't that be like asking dentists which toothpaste they prefer and then, after finding that 4 of 5 dentists like Colgate, calling the results biased because you only asked dentists?

I don't think there is a bias because you're measuring the opinion of well traveled enthusiasts who care enough to send a ballot. That's the point of the poll.

That said, it is possible that "14" is big enough---it depends on the population the sample is drawn from, and the underlying distribution of preference. The problem is, figuring that out is hard.

Thank you. Now Jeff just needs to read it.

14 could very well be enough to form a reasonable conclusion under the circumstances. Figuring it out (scientifically; in a disciplined, actual way) is difficult.

At the end of the day, I like Mitch's methodology. I suspect he needs to think harder about his cut-off number, and reason a little about confidence values, etc.

Agreed.

How is a tour from TPR that rode T-Express any different than coaster enthusiasts traveling to Holiday World for the various events that happen there. If The Voyage were #1 I don't think anyone would be debating that when in fact a bias exists there as well.

Wouldn't that be like asking dentists which toothpaste they prefer and then, after finding that 4 of 5 dentists like Colgate, calling the results biased because you only asked dentists?

No. The difference is more subtle than this. Consider two polls. In one poll, you publish an advertisement in the dental society's magazine asking all interested dentists to submit their favorite toothpaste---and let's imagine that 20% answer. In another poll, you call 20% of the dentists on the magazine's subscription list, at random, and ask them about their favorite toothpaste.

The second poll is much more reliable than the first, because it does not suffer from selection bias.

Here's why it might matter: suppose that 15% of the dentists in the country are taking money under the table from Joe's Toothpaste Co, a disreputable firm that makes toothpaste with sugar. In the first poll, all of these dentists respond---they have an incentive to see that Joe's is viewed favorably---and only the 5% remaining dentists answer honestly. So, in this poll, "3 out of 4 dentists" prefer Joe's.

In the second poll, the 15% unscrupulous dentists are randomly sampled. So, of the 20% respondents, only 3% (.2 * .15) identify Joe's as their favorite, giving a much more accurate picture of the true population of "all dentists."

eightdotthree said:How is a tour from TPR that rode T-Express any different than coaster enthusiasts traveling to Holiday World for the various events that happen there. If The Voyage were #1 I don't think anyone would be debating that when in fact a bias exists there as well.

Much larger sample with a much greater variety in taste than just 14 people who generally preffer the same type of ride. Hers a question for you. Where did favorite steel rank in those 14 people? I Bet in every case its withing the top five. In my case the top steel isn't in my top ten, Why, because I like to leave the seat, get tossed about and latterals. .Like I said, this group done it before. They'll do it again. I don't really care. They are getting out and ridding coasters and having fun. If T express really is that good. It will hold its status. No worries.Chuck

Gotcha, Brian.

But the operative word is in the third paragraph - might. As in "Here's why it might matter:"

It might not matter. Correct?