Statistics tell us: (1) Passing a typical 3-lot PQ is not easy; (2) What to do about it.
In medical device process validation (IQ OQ PQ) we are providing objective evidence the equipment and process are capable of successfully making the product consistently.
Three (3) consecutive lots have typically been used to demonstrate this in a PQ - although FDA does not specify a required number and it is up to the manufacturer to justify.
For key characteristics the capability is typically justified in statistical confidence and conformance (or reliability). For example, 95% confidence / 97% conformance means being 95% confident that greater than or equal to 97% of units made by the process meet the specification (i.e., the % defectives are < 3%).
Although the statistical confidence & conformance apply to the manufacturing process, very often the sample size and acceptance criterion is applied per lot (batch). In that case, if 3 consecutive lots are made what is the probability of passing the study?
Still Popular: a = 0
Assuming an attribute quality characteristic and accept on zero failures criterion (a = 0), the probability of passing all 3 lots using the binomial distribution is shown in the figure below.
As can be seen, having a successful process validation is NOT easy!
To have a reasonable chance at passing the study, the actual process % defective must be much lower (20x lower!) than would be indicated by the % conformance.
A Better Approach: a ≤ 1
Can we boost the odds in our favor while maintaining the same customer protection (e.g., %confidence /% conformance)? One option is to modify the acceptance criterion per lot from a = 0, to a ≤ 1 (accept on at most one failure) while increasing the sample size.
The corresponding probabilities using a ≤ 1 (assuming the sample size and acceptance criterion are applied to each LOT), are shown below.
Using a ≤ 1 results in a 58% increase in the sample size. However, this is most often a tradeoff we're willing to make because, for very reliable processes (i.e., very low % defectives), the chance of passing is greatly enhanced.
A comparison of the probabilities for a = 0 and a ≤ 1 is shown in the table below.
(Probabilities of passing the study: <50% are in red, 50% - 75% in orange, those >75% are green.)
Experience tells us the benefits of using the a ≤ 1 are often greater than the raw statistics would reveal. Since reproducibility (i.e., the variability due to different inspectors) is often a fairly large portion of measurement variation, and human beings are prone to error (due to fatigue, etc.), using a ≤ 1 may mean the difference between passing the study or failing it. This is especially true for those processes whose inspection methods are manually very delicate, complex, or having high variation.
Going to Extremes: Beyond a ≤ 1
The question now becomes: what happens as we increase the acceptance number (e.g., a ≤ 2, a ≤ 5, etc.)?
Taking 95% confidence and 97% conformance as an example and assuming the sample size & acceptance criterion are applied to each lot, the number of samples and probability of passing all 3 lots are shown below.
The ideal sampling plan (white trace in the graph above using 95% confidence / 97% conformance as an example) is one that would:
Pass 100% of processes with actual % defectives < 3%
Reject (pass 0% of the time) processes with actual % defectives ≥ 3%
Allowing a greater acceptance number with a bigger sample size means the plan increasingly behaves like the ‘ideal’: passing very reliable processes and rejecting those with % defectives greater than the minimum conformance.
The Modern Approach: Double Sampling Plans
Unless the lot sizes used during regular production are very large, we seldom use anything greater than a ≤ 1 in our studies: the number of samples required for inspection, the cost of the study and its duration simply become too great compared to the benefits.
There is a better alternative: double sampling. Double sampling plans increase the probability of passing the study while keeping the effective sample size comparable to a = 0 plans.
As an example, take an attribute double sampling plan that may be used to achieve 95% confidence / 97% conformance with the following parameters:
n1 = 106, a1= 0, r1 = 2, n2 = 83, a2 = 1
The plan works like this:
Take a first sample (stage 1) of 106 units (n1=106); if there are 0 defective units in the sample, the acceptance criterion (a1 = 0) has been satisfied (pass) and inspection stops. If there are 2 defective units found in the sample, the rejection criterion (r1 = 2) was reached (fail) and inspection stops. If there is only 1 defective unit found (a1 < x < r1), proceed to the second stage.
Second stage (stage 2), take an additional sample of 83 units; if there is still only 1 defective unit (a2 = 1) found in total [i.e., in 189 units (= 106 + 83)], the acceptance criterion has been satisfied (pass). Otherwise, the acceptance criterion has not been satisfied (fail).
Assuming, as we have for this analysis, that the sample sizes and acceptance criteria are applied to each lot, the probability of all 3 lots passing is shown below.
Notice how similar the probabilities for the double sampling plan and a ≤ 1 plan are. When the process percent defective is low (say ≤ 0.3%), the probabily of passing on the first stage is 4X higher compared to passing on the second stage.
For reliable processes, this means the lots will most likely pass during the first stage and the additional samples of the second stage won’t be needed. The result? Using double sampling means the study will be completed faster and with less inspection than with single a ≤ 1 plans.
Note that it is important to specify ahead of time (i.e., in the study protocol) that a double-sampling plan will be used and its parameters. Do not specify a single sampling plan and then inspect additional parts during study execution.
Key Takeaways
The process percent defective must be much lower (20x less) than what is indicated by the % conformance when using accept on zero failures (a = 0) sampling plans.
Accept on at most 1 failure (a ≤ 1) plans provide a much higher probability of passing the PQ study, but result in a significant increase in the sample size needed (+58%).
Use of double sampling plans provide essentially the same probability of passing that a ≤ 1 plans provide, while achieving the sample customer protection and having effective sample sizes closer to a = 0 sampling plans.
For the above reasons, it is highly recommended to use double sampling plans whenever possible for attribute analysis studies.
Learn more about Medical Device Process Validation with the FREE PDF below.