Home English (UK) Ideas, Suggestions & Feedback

We need new Moderators!


ModeratorWe are always on the lookout for talented people to join the team. That means you! If you think you could help us organise and inform the community while entertaining everyone then apply. We need people to help out on the forum, behind the scenes with announcements, on Discord and on our other Social Media channels.


If this is something you think might be of interest to you, HERE

Chance of Smithing Gems - Bug or Deliberate Inaccuracy?

Chris2307 (GB1)Chris2307 (GB1) Posts: 944
This is probably the most important thread I've ever posted in this forum and it regards the chance of smithing gems.

The chance of smithing a level 3 gem to level 4 gem is supposed to be 60% (http://prntscr.com/4zg3bs).

For a while, a number of players, including myself, have felt that this percentage chance was inaccurate. There was a thread about it a while ago and we all conceded that not enough evidence was present to accuse GGS of deliberately misleading us or of a bug (if we wish to assume GGS is innocent). So I have gone out of my way to record every single level 3 -> 4 smith attempt since then.

I have not mis-recorded a single smith; everything has been noted down. I've not asked any other player to smith a gem and report the result to me. Every single attempt has been from myself. Each smith has been fairly random and occurred whenever I had enough gems to attempt a smith. I recorded the results in an Excel spreadsheet and used its standard functions to calculate the percentages. The results are as follows:

http://prntscr.com/4zg1p6

I want a CM to comment on these results as quickly as possible. Is this an inaccuracy in your system or a deliberate act? Either way, this needs to be fixed. I present results consisting of 40 level 3 -> 4 attempts. This is enough to be statistically significant. You have no comeback whatsoever.

40 smith attempts: 47.5% success instead of 60% success as you advertise.

As a side note, I also include the results of my level 4 -> 5 attempts. Whilst I concede that the amount of smiths for this level are not enough to warrant an accusation based on statistically significant evidence, I believe that as I carry on, I will produce similar results with regards to the advertised chance.

I await your reply GGS...
Post edited by Chris2307 (GB1) on
Chris2307 @ en 1

Comments

  • millwall97 (GB1)millwall97 (GB1) GB1 Posts: 2,513
    edited 25.10.2014
    Chris2307 wrote: »
    This is probably the most important thread I've ever posted in this forum and it regards the chance of smithing gems.

    The chance of smithing a level 3 gem to level 4 gem is supposed to be 60% (http://prntscr.com/4zg3bs).

    For a while, a number of players, including myself, have felt that this percentage chance was inaccurate. There was a thread about it a while ago and we all conceded that not enough evidence was present to accuse GGS of deliberately misleading us or of a bug (if we wish to assume GGS is innocent). So I have gone out of my way to record every single level 3 -> 4 smith attempt since then.

    I have not mis-recorded a single smith; everything has been noted down. I've not asked any other player to smith a gem and report the result to me. Every single attempt has been from myself. Each smith has been fairly random and occurred whenever I had enough gems to attempt a smith. I recorded the results in an Excel spreadsheet and used its standard functions to calculate the percentages. The results are as follows:

    http://prntscr.com/4zg1p6

    I want a CM to comment on these results as quickly as possible. Is this an inaccuracy in your system or a deliberate act? Either way, this needs to be fixed. I present results consisting of 40 level 3 -> 4 attempts. This is enough to be statistically significant. You have no comeback whatsoever.

    40 smith attempts: 47.5% success instead of 60% success as you advertise.

    As a side note, I also include the results of my level 4 -> 5 attempts. Whilst I concede that the amount of smiths for this level are not enough to warrant an accusation based on statistically significant evidence, I believe that as I carry on, I will produce similar results with regards to the advertised chance.

    I await your reply GGS...
    oh by the way, I would not count on a reply ;)
    millwall97 @ Somalia1

    tCp90gif

    Don't click: https://goo.gl/IC6U01
  • Chris2307 (GB1)Chris2307 (GB1) Posts: 944
    edited 25.10.2014
    millwall97 wrote: »
    oh by the way, I would not count on a reply ;)

    Sadly, I share your fear but I do expect a reply from them considering I have provided evidence that we have been actively mislead in the game.
    Chris2307 @ en 1
  • Colpin Cathar (US1)Colpin Cathar (US1) Posts: 195
    edited 25.10.2014
    Ah, How people can be mislead by percentages and statistics. The percentages are for EACH try. Not for over all success. Where your statistics are for over all success, not for individual tries. For example, take six six sided dice. Role them on the table. According to your argument, there should be one showing each number because there is six of them and there are six sides. That is not the way it works. Each die has a one in six chance of landing on a particular number. That is how things are calculated. Each try at the smithy has it's own percentage chance of success and it does not affect the next try.
    Colpin Cathar
    Of the House Of Dragons:
    Order of the Shadow Dragons
  • Lieutenant4Lieutenant4 Posts: 15
    edited 25.10.2014
    This is enough to be statistically significant. You have no comeback whatsoever.

    Please show your maths if you want to make a claim like this.

    I don't want to come off as patronising, but you haven't recorded enough examples to prove anything.

    A quick back-of-a-matchbox calculation suggests that about 10% of people will see a similar (or greater) difference between the expected success rate and actual success rate (~5% of those seeing more failure, ~5% seeing more success). You might just have been unlucky.

    It's also fair to say that those who see more success than expected don't come on the forum to complain, hence why the forum only contains anecdotal evidence of probabilities not matching up.

    It's not to say that there isn't a bug, just that you don't have enough evidence to prove it, you would need to get at least 100 results showing a 47.5% success rate in order to do so without any doubt.
    Lieutenant @ WWW 1
  • Kyle Ludwig WuKyle Ludwig Wu Posts: 1,053
    edited 26.10.2014
    Ah, How people can be mislead by percentages and statistics. The percentages are for EACH try. Not for over all success. Where your statistics are for over all success, not for individual tries. For example, take six six sided dice. Role them on the table. According to your argument, there should be one showing each number because there is six of them and there are six sides. That is not the way it works. Each die has a one in six chance of landing on a particular number. That is how things are calculated. Each try at the smithy has it's own percentage chance of success and it does not affect the next try.

    Spot on :P
    BSK-75 @ en 1


    We are the Battle Sacred Kittens :D
    (for whom who really want to know the exact meaning of BSK, PM me ingame :P)
  • neil33 (GB1)neil33 (GB1) Posts: 1,762
    edited 26.10.2014
    Please show your maths if you want to make a claim like this.

    I don't want to come off as patronising, but you haven't recorded enough examples to prove anything.

    A quick back-of-a-matchbox calculation suggests that about 10% of people will see a similar (or greater) difference between the expected success rate and actual success rate (~5% of those seeing more failure, ~5% seeing more success). You might just have been unlucky.

    It's also fair to say that those who see more success than expected don't come on the forum to complain, hence why the forum only contains anecdotal evidence of probabilities not matching up.

    It's not to say that there isn't a bug, just that you don't have enough evidence to prove it, you would need to get at least 100 results showing a 47.5% success rate in order to do so without any doubt.

    So basically, in a nutshell, you are saying that the gems have an independent probability. You are correct though, each forge is independent, but I do think that the success rate is inaccurate. more than half (50%) of mine fail.
    [SIGPIC][/SIGPIC]

    Proud member of Praetorians

    SO...I get conned out of rubies and support refuse to give them back. But when a ruby player does the same as me they get them back? This to me is point blank evidence that support and GGE don't give a fuck about us non ruby buyers...

    http://prntscr.com/93xafb

    Still think MeepMeep is one mad patty!
  • Chris2307 (GB1)Chris2307 (GB1) Posts: 944
    edited 26.10.2014
    Ah, How people can be mislead by percentages and statistics. The percentages are for EACH try. Not for over all success. Where your statistics are for over all success, not for individual tries. For example, take six six sided dice. Role them on the table. According to your argument, there should be one showing each number because there is six of them and there are six sides. That is not the way it works. Each die has a one in six chance of landing on a particular number. That is how things are calculated. Each try at the smithy has it's own percentage chance of success and it does not affect the next try.

    Actually, it is your explanation of statistics which is way off the mark here so let me explain, and of course it's percentage chance for each try - no one believes otherwise. In keeping with your 6 sided die example, I would not expect to roll the die 6 times and see each number. But over time, I would expect to see (and we would see) the results average out. With a coin flip - the same. Over X number of flips, we would see the flips average out over time. This is what I've attempted to do here. I have recorded 40 attempts and the overall result begins to average out. The more attempts there are, the more accurate the overall figure is.
    Please show your maths if you want to make a claim like this.

    I don't want to come off as patronising, but you haven't recorded enough examples to prove anything.

    I've actually showed my maths. You're issue is with the data set size and I take that on board.
    A quick back-of-a-matchbox calculation suggests that about 10% of people will see a similar (or greater) difference between the expected success rate and actual success rate (~5% of those seeing more failure, ~5% seeing more success). You might just have been unlucky.

    It's also fair to say that those who see more success than expected don't come on the forum to complain, hence why the forum only contains anecdotal evidence of probabilities not matching up.

    100% agree with you here. This is why I've come to the forums with numbers to back up my claim. What I present is not anecdotal evidence anyway, although your opinion is that it's too small a number (40 attempts). I'll carry on recording anyhow. Please note that I had a "feeling" that the smithing chance was incorrect and I noted down every single smith in order to make this post very anti-anecdotal.

    Again... your issue with my post seems to stem from the data-set size.
    It's not to say that there isn't a bug, just that you don't have enough evidence to prove it, you would need to get at least 100 results showing a 47.5% success rate in order to do so without any doubt.

    Fair enough - We'll see what we get when I go up to 100 then!
    Chris2307 @ en 1
  • Colpin Cathar (US1)Colpin Cathar (US1) Posts: 195
    edited 26.10.2014
    Chris2307 wrote: »
    Actually, it is your explanation of statistics which is way off the mark here so let me explain, and of course it's percentage chance for each try - no one believes otherwise. In keeping with your 6 sided die example, I would not expect to roll the die 6 times and see each number. But over time, I would expect to see (and we would see) the results average out. With a coin flip - the same. Over X number of flips, we would see the flips average out over time. This is what I've attempted to do here. I have recorded 40 attempts and the overall result begins to average out. The more attempts there are, the more accurate the overall figure is.

    Yes and no. The numbers would average out, but not all the way. You are again confusing percent chance with statistics. Since I work at a casino, I see people who don't understand this all the time and refuse to get it. If you role one six-sided die, it is entirely possible to not get a six for 10 to 15 roles of the die. Does that mean that the chance of getting that six at one in six wrong even tho the percentages of actually getting it are closer to one in ten? I have also done some computer programing and my friends and I have done work with random number generators. If they use a base 10 system (step it by 10 percent) then it would be ten numbers they use. 1-6 would be success and 7 thru 10 would be failures. I have seen some of my programs come up with some very interesting results as far as the base ten system. This would be like using a ten sided die to get your numbers. The results would be even more interesting if they used a base 100 system (numbers 1 thru 100) to get an actual percentage number because then you would have both the tens and the ones place to worry about and there is "more" numbers that can show up in the "failure slot. The point is that the numbers you are using are not going to 100% match up to the percentage chance of success at the smithy.
    Colpin Cathar
    Of the House Of Dragons:
    Order of the Shadow Dragons
  • Chris2307 (GB1)Chris2307 (GB1) Posts: 944
    edited 27.10.2014
    Yes and no. The numbers would average out, but not all the way. You are again confusing percent chance with statistics. Since I work at a casino, I see people who don't understand this all the time and refuse to get it. If you role one six-sided die, it is entirely possible to not get a six for 10 to 15 roles of the die. Does that mean that the chance of getting that six at one in six wrong even tho the percentages of actually getting it are closer to one in ten? I have also done some computer programing and my friends and I have done work with random number generators. If they use a base 10 system (step it by 10 percent) then it would be ten numbers they use. 1-6 would be success and 7 thru 10 would be failures. I have seen some of my programs come up with some very interesting results as far as the base ten system. This would be like using a ten sided die to get your numbers. The results would be even more interesting if they used a base 100 system (numbers 1 thru 100) to get an actual percentage number because then you would have both the tens and the ones place to worry about and there is "more" numbers that can show up in the "failure slot. The point is that the numbers you are using are not going to 100% match up to the percentage chance of success at the smithy.

    You're right about one thing. The results would never ever be 100% accurate and it's entirely possible that I could get 40 successful smiths in a row. However, you are still confusing statistics. I work as software development researcher so am very aware of both psuedo-random number generators and statistics. Whilst psuedo-random number generators are never truly random, they are random enough to not elicit the kind of inaccuracy I have presented thus-far. In fact, if you've programmed your programs correctly and have used proper built in RNGs instead of attempting to code your own, then you should not see "interesting" results from a random number generator at all. They do the job just fine. Anyway, the thread is digressing a bit.

    Back on to the statistics. By increasing the data-set, the percentage results average out further and further getting closer to the correct result. Whilst never achieving a purely accurate result, it is the job of statistics to try and explain these numbers. I've run a basic binomial test here which is a standard way of testing this kind of thing (You can Google it if you like). If you run my findings through any Binomial Probability calculator, you will find that the binomial probability of the results I have achieved against the dataset I have collected is good enough to report as evidence.

    I.e. Binomial Probability of the above = 0.03518. This is the same as saying that the chance of observing 19 successful smith attempts in 40 (based on a posted 60% success rate) is 3.518%. Anything less than 5% is considered to be statistically significant and therefore reliable.

    Despite this, I'll continue collecting to further strengthen my hypothesis.
    Chris2307 @ en 1
  • Eric768 (US1)Eric768 (US1) US1 Posts: 7,183
    edited 27.10.2014
    You want to talk about inaccuracy? I had a 14% fail rate on this piece of equipment I was forging, I click around 80 times, 10k coins a click before I had a success.
    Proud to have been one of the longest serving members on USA1
    Advocate of speaking up regarding mental health and seeking help
    ***Currently Retired from playing GGE***

    Rest well Jason; a fantastic player and an even greater person. Gone but not forgotten.
  • Colpin Cathar (US1)Colpin Cathar (US1) Posts: 195
    edited 27.10.2014
    *sigh* You are sounding more and more like a patron at a casino who plays a slot machine for 4 or 5 days straight, don't win a jackpot, and get upset because you know what the probability of winning is, but because you surpassed the number of spins indicated, you feel the machine is rigged and take it out on the casino staff. The machines are random. Each time you hit the button, you have a set chance of winning. The chance is the same each time you hit the button and doesn't affect the next push of the button. It is also not affected by the previous push of the button. How does it know that you haven't gotten a "win" for the last 30 pushes of the button? It doesn't. It only knows to come up with a result with this much probability, and gives you that result. Now the results may be that one day (a good day) there are a lot of successful results, and on another day, (a bad day) there may be a lot of failures. There are also those people who seem to have worse "luck" when it comes to these things. This was another test we preformed with our random number generators. It seems that some people are just luckier then others. In order for your statistics to have a spread for a test group, you would not only need you to be preforming the tests. You would need a minimum of 2 other people (the more the better), each preforming the tests at approximately the same stretch of time, from all times of the day. Remember, the more data, the more accurate you can be. But as you confirmed, it will never be 100% the same as what the percent chance is actually is.
    Colpin Cathar
    Of the House Of Dragons:
    Order of the Shadow Dragons
  • neuterable (US1)neuterable (US1) US1 Posts: 757
    edited 27.10.2014
    It's Technicus Statistics. In other words, GGS is basically lying. None of their probabilities are ever accurate and no matter how many times you click, they will always be able to say that the sample is just not quite large enough. Try smithing another five hundred thousand times and just maybe your luck will improve.
    neuterable, shameless slaughterer of peasants, collector of princesses and fan of homestuck


    α ЯTFM ¿¿¿Want Free $tuff??? Then Go Write Santa a Letter ЯTFM Ω
  • Chris2307 (GB1)Chris2307 (GB1) Posts: 944
    edited 27.10.2014
    [DISCLAIMER] This post is a continuation of the statistics debate this thread keeps turning in to. If you aren't interested... I wouldn't read it ;)
    *sigh* You are sounding more and more like a patron at a casino who plays a slot machine for 4 or 5 days straight, don't win a jackpot, and get upset because you know what the probability of winning is, but because you surpassed the number of spins indicated, you feel the machine is rigged and take it out on the casino staff.

    I understand you work at a casino and I really mean no disrespect by this but it by no means gives you even a basic understanding of fundamental statistics (you'd need to take a class or read a proper book on it). We're going around in circles here and you're unable to explain yourself without resulting back to your experience at work; which I am sure is a good enough explanation to your customers. As a side note, I am not a customer to any casino (I have more important things to spend my money on).

    You keep mentioning the example of a slots machine. Whilst this is fine and you're right; each individual spin is an independent chance, what you're failing to take in to account is the number of possible outcomes. With each additional outcome, the required dataset size increases. So right away you are comparing a slot machine with 3, 4 or possibly more outcomes against my test which has one of two outcomes. I've been able to show that 40 tests were enough (just) by presenting my p value. Again, maybe you should read up on this (as well as Binomial Testing - a statistically sound way to test for outcomes).
    The machines are random. Each time you hit the button, you have a set chance of winning. The chance is the same each time you hit the button and doesn't affect the next push of the button. It is also not affected by the previous push of the button. How does it know that you haven't gotten a "win" for the last 30 pushes of the button? It doesn't. It only knows to come up with a result with this much probability, and gives you that result.

    You've already said this. You're not thinking of the bigger picture. There is a way to test whether a system gives a fair (or expected outcome) by testing multiple trials. This is called binomial testing. I'm not testing each trial. (One of many good explanations here: http://stattrek.com/online-calculator/binomial.aspx#experiment) Again, look at my P value to satisfy yourself that my dataset is large enough.
    Now the results may be that one day (a good day) there are a lot of successful results, and on another day, (a bad day) there may be a lot of failures. There are also those people who seem to have worse "luck" when it comes to these things. This was another test we preformed with our random number generators. It seems that some people are just luckier then others.

    Now you're sounding like a casino customer ;) Never talk about luck. Luck doesn't exist. No one is luckier than anyone else. By the way, in order to carry out any kind of outcome testing on a slot machine, due to the many outcomes possible, you would need to test many many times. I highly doubt any one person has enough money to do this on there own. This is where your talk of "luck" comes from. Each player will play for such a relatively low number of times (relative to the number of outcomes) that they will never see a pattern in their own playing behaviour. Perhaps this is where your confusion comes from?
    In order for your statistics to have a spread for a test group, you would not only need you to be preforming the tests. You would need a minimum of 2 other people (the more the better), each preforming the tests at approximately the same stretch of time, from all times of the day. Remember, the more data, the more accurate you can be. But as you confirmed, it will never be 100% the same as what the percent chance is actually is.

    Actually, no. Since I am not testing the behaviour of people (or something which is affected by the behaviour of people), I do not need to test using multiple persons (like you would in psychology). I am testing the overal behaviour of the smithing system and whether it produces fair results. Therefore I must test this aspect over and over and over again. So whilst I agree with you; the more data the better, I am afraid you are completely wrong by stating that more than one person must carry out the same test.

    So I repeat to you, spend a couple of hours reading up about binomial testing and what it does. I didn't want to get in to this kind of debate on this forum as I want to focus on the results and what it means to my original questions; is GGS deliberately misleading us or is there a bug?

    And please also remember;

    You're experience/examples revolve around slot machines where many many outcomes can occur (i.e. Mini games on the modern style machines, different levels of payouts, different inputted amounts) which is much more complex and therefore requires much, much more testing.

    I am testing one of two outcomes. Much simpler and I have used a statistically sound method. Please do not compare the two again. You're comparing apples and oranges.
    Chris2307 @ en 1
  • Lieutenant4Lieutenant4 Posts: 15
    edited 27.10.2014
    Chris2307 wrote: »
    I.e. Binomial Probability of the above = 0.03518. This is the same as saying that the chance of observing 19 successful smith attempts in 40 (based on a posted 60% success rate) is 3.518%. Anything less than 5% is considered to be statistically significant and therefore reliable.

    I'm afraid you're mixing up the probability of a single outcome and the probability of a stated rate of success being flawed.

    As a quick, stupid, example, use your binomial calculator to toss a fair coin 1000 times (heads=0.5, tails=0.5)
    The probability of getting exactly 500 heads is 2.5%. It does not mean that the coin is unfair. It means that the probability of one outcome of one trial (trial: flip a coin 1000 times; outcome: observe exactly 500 heads) is low. It's improbable, but doesn't suggest that the coin is unfair, indeed, any other single outcome would have an even lower probability. It is on the contrary, strong evidence that the coin is fair - we observed the theoretical mean.

    What you need to do, is set up a hypothesis assuming that everything is ok, and then show that it is sufficiently unlikely for that hypothesis to be true (I'm pretty sure you have come across hypothesis testing before as you talked about the 5% confidence interval). This has to be in one direction or the other or both.

    Hypothesis: "The actual probability of upgrading a gem from 3 -> 4 is 60%"

    The probability that given this hypothesis, and 40 trials with a stated success rate of p=0.6 , you observe 19 or fewer successes is 7.43% This is low, but not low enough to reject the hypothesis with a confidence interval of 5%.


    Imo it is not enough to simply continue your experiment with your existing data - if you want to conduct a statistical test, you should decide the parameters of the experiment in advance - in this case, the number of trials. If you continue adding results to your data set from now, you are not calculating the probability of 50 trials being lower than a significant threshold, but the probability of 10 trials pushing your mean from 40 trials lower than it is now, which then becomes more likely, but ultimately meaningless when it comes to proving anything.
    Lieutenant @ WWW 1
  • Chris2307 (GB1)Chris2307 (GB1) Posts: 944
    edited 27.10.2014
    I'm afraid you're mixing up the probability of a single outcome and the probability of a stated rate of success being flawed.

    As a quick, stupid, example, use your binomial calculator to toss a fair coin 1000 times (heads=0.5, tails=0.5)
    The probability of getting exactly 500 heads is 2.5%. It does not mean that the coin is unfair. It means that the probability of one outcome of one trial (trial: flip a coin 1000 times; outcome: observe exactly 500 heads) is low. It's improbable, but doesn't suggest that the coin is unfair, indeed, any other single outcome would have an even lower probability. It is on the contrary, strong evidence that the coin is fair - we observed the theoretical mean.

    What you need to do, is set up a hypothesis assuming that everything is ok, and then show that it is sufficiently unlikely for that hypothesis to be true (I'm pretty sure you have come across hypothesis testing before as you talked about the 5% confidence interval). This has to be in one direction or the other or both.

    Hypothesis: "The actual probability of upgrading a gem from 3 -> 4 is 60%"

    The probability that given this hypothesis, and 40 trials with a stated success rate of p=0.6 , you observe 19 or fewer successes is 7.43% This is low, but not low enough to reject the hypothesis with a confidence interval of 5%.


    Imo it is not enough to simply continue your experiment with your existing data - if you want to conduct a statistical test, you should decide the parameters of the experiment in advance - in this case, the number of trials. If you continue adding results to your data set from now, you are not calculating the probability of 50 trials being lower than a significant threshold, but the probability of 10 trials pushing your mean from 40 trials lower than it is now, which then becomes more likely, but ultimately meaningless when it comes to proving anything.

    Okay, that's a fair point. I've reported the P(X = x) result and you are right; I need to report when P(X <= x) Again, this comes down to the low number of recorded trials. Whilst not at a significant level, I still believe my hypothesis (that is "The actual probability of upgrading a gem from 3 -> 4 is less than 60%) and I've still reported a low confidence interval, which I believe is an indicator that this is worth further investigation.

    However, I still see no reason why the original dataset cannot be used. As far as I can see, I have reported my results too soon (if my hypothesis is true) and adding to the original data recorded is OK considering that nothing has changed since reporting the data here. I have continued recording and still have not missed a single outcome. I.e. There would be no difference whether I started collecting data again now or adding on as my method has not changed.

    Anyway, thank you for pointing out the confusion between the P(X = x) and P(X <= x) results. I still stand by my previous points in regard to my conversation with Colpin Cathar as he was arguing a different point.
    Chris2307 @ en 1
  • Lieutenant4Lieutenant4 Posts: 15
    edited 27.10.2014
    Chris2307 wrote: »
    I still believe my hypothesis (that is "The actual probability of upgrading a gem from 3 -> 4 is less than 60%) and I've still reported a low confidence interval, which I believe is an indicator that this is worth further investigation.

    I know you believe it, which is why there is a lot of scope for bias (statistical or otherwise) if you don't run the experiment properly.
    However, I still see no reason why the original dataset cannot be used. As far as I can see, I have reported my results too soon (if my hypothesis is true) and adding to the original data recorded is OK considering that nothing has changed since reporting the data here.

    Maybe you can use it, maybe you can't - to be completely honest, it has been a long time since I had to do this kind of testing. But you may want to inquire a bit more with people more knowledgeable than myself - I strongly suspect that if you continue adding data to your data set, you will end up with a flawed experiment.

    A relevant quote from a statistics blog:
    It is a well-known fact of null-hypothesis significance testing (NHST) that when there is "optional stopping" of data collection with testing at every new datum (a procedure also called "sequential testing" or "data peeking"), then the null hypothesis will eventually be rejected even when it is true. With enough random sampling from the null hypothesis, eventually there will be some accidental coincidence of outlying values so that p < .05 (conditionalizing on the current sample size). Anscombe (1954) called this phenomenon, "sampling to reach a foregone conclusion."
    http://doingbayesiandataanalysis.blogspot.de/2013/11/optional-stopping-in-data-collection-p.html

    Reading up on prior/posterior probabilities, part of Bayesian probability, might help too - but personally, this wasn't my strong point.

    Ideally all your experiment design decisions should be made before any data collection, and more importantly, before any data analysis, i.e. clear criteria for accepting/rejecting a hypothesis, a clear pre-defined stopping point to an experiment, and any possible extensions (again pre-defined, before any data analysis).
    Lieutenant @ WWW 1
  • Chris2307 (GB1)Chris2307 (GB1) Posts: 944
    edited 27.10.2014
    I know you believe it, which is why there is a lot of scope for bias (statistical or otherwise) if you don't run the experiment properly.



    Maybe you can use it, maybe you can't - to be completely honest, it has been a long time since I had to do this kind of testing. But you may want to inquire a bit more with people more knowledgeable than myself - I strongly suspect that if you continue adding data to your data set, you will end up with a flawed experiment.

    A relevant quote from a statistics blog:

    http://doingbayesiandataanalysis.blogspot.de/2013/11/optional-stopping-in-data-collection-p.html

    Reading up on prior/posterior probabilities, part of Bayesian probability, might help too - but personally, this wasn't my strong point.

    Ideally all your experiment design decisions should be made before any data collection, and more importantly, before any data analysis, i.e. clear criteria for accepting/rejecting a hypothesis, a clear pre-defined stopping point to an experiment, and any possible extensions (again pre-defined, before any data analysis).

    Thanks for the link. I'll certainly have a read up on this.

    When we talk about bias, this is something which I am, of course, aware of and I've spent a lot of time in the past accounting for bias in other experiments (Completely different experiments this one) so I was aware of this going in. I struggle to account for any potential experimenter bias and the data collected thus far has adhered to some basic criteria in that I have recorded 100% of the results. I think that although you are right about defining all experiment design decisions before going in, I think I will be OK to carry on with this particular data collection.

    Either way, I appreciate the information you've given me. :)
    Chris2307 @ en 1
  • Colpin Cathar (US1)Colpin Cathar (US1) Posts: 195
    edited 28.10.2014
    Chris2307 wrote: »

    Now you're sounding like a casino customer ;) Never talk about luck. Luck doesn't exist. No one is luckier than anyone else. By the way, in order to carry out any kind of outcome testing on a slot machine, due to the many outcomes possible, you would need to test many many times. I highly doubt any one person has enough money to do this on there own. This is where your talk of "luck" comes from. Each player will play for such a relatively low number of times (relative to the number of outcomes) that they will never see a pattern in their own playing behaviour. Perhaps this is where your confusion comes from?

    I can tell you from experience that there are some people who come in, sit down at a machine and will not move for 3 days, and not hit a "jackpot". Then there are those who come in, sit down, and within 3 hours, they have won 5 of them. It is all random. If it the machine is going to hit, then it will hit, if it's not, well, then it won't. In a way, the idea is similar. A lot of players don't come in just to play. The want the "jackpot". If you look at the system in that respect, either "jackpot" or "no jackpot" then it is only two outcomes.
    Colpin Cathar
    Of the House Of Dragons:
    Order of the Shadow Dragons
  • Eric768 (US1)Eric768 (US1) US1 Posts: 7,183
    edited 28.10.2014
    I can tell you from experience that there are some people who come in, sit down at a machine and will not move for 3 days, and not hit a "jackpot". Then there are those who come in, sit down, and within 3 hours, they have won 5 of them. It is all random. If it the machine is going to hit, then it will hit, if it's not, well, then it won't. In a way, the idea is similar. A lot of players don't come in just to play. The want the "jackpot". If you look at the system in that respect, either "jackpot" or "no jackpot" then it is only two outcomes.

    I.. I want a jackpot.
    Proud to have been one of the longest serving members on USA1
    Advocate of speaking up regarding mental health and seeking help
    ***Currently Retired from playing GGE***

    Rest well Jason; a fantastic player and an even greater person. Gone but not forgotten.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file