Test of Comparison Sample size

[Lynn Long]

How do you determine what sample size in a test to failure to use when comparing two designs? There are two situations: (1) You changed the design to increase the life or (2) you changed the design for some other reason and you want to be sure you have not changed or decreased the life of the product.

In the first case, the more samples you run, the more likely you are to show a difference, if one exists. This is because increasing the sample size decreases the confidence band. In the second case, if you run a few samples, you are more likely to show no difference. You need some criteria to show a difference.

[RS Support]

Lynn, you are trying to prove/disprove is the hypothesis that two samples come from the same populations. To simplify this imagine that there are two buckets with black and white marbles in each of them -- and you want to find out whether the percent of white marbles in each bucket is the same. When one draws a sample out of each bucket, one is trying to prove or disprove that based on the sample taken. Obviously the larger the sample drawn from each bucket the more certain one would be as to the fact that the percentage is or is not the same. You mention this in your statement.

Towards the end you said that you need some criteria. Confidence bounds are one way to take sampling error into account, thus one could at a given confidence level say if there is a difference or not in a specific characteristic or metric (some information is provided at http://www.weibull.com/LifeDataWeb/tests_of_comparison.htm) Is this what you are asking or is what you are asking ‘What confidence level one should use?’

[Lynn Long]

When I looked at your example, it works well if I want to show that a new design has a higher reliability than an old design. Like you say, I could require a 90% or 95% confidence interval. However if I want to show that the two designs have the same reliability, smaller sample sizes are going to make the two designs look the same, when they may not. In your example, I could say that at the 95% confidence level, there is no difference in the reliability of the two designs. One way I could see to get around this would be to have a lower bound for the reliability at some time. For example the requirement could be that the lower CI at 1000 hrs must be greater than 90%. This would increase the sample size and then make a comparison more meaningful. This means I would be adding an additional requirement to a test of comparison. I was hoping to come up with a method that did not need an additional requirement.

[RS Support]

Yes, you need to clearly define the requirement and what you are comparing.