A Case of Contamination

When is a sample, in fact, not a sample?

By Paula Brown

A manufacturer acquired a botanical dietary ingredient from a certified organic farm. The manufacturer sent a contract laboratory nine roots for pesticide testing. The roots tested clean, the manufacturer accepted the lot, and proceeded to make a finished product. At a later date, this finished product was tested and found to contain high levels of DDT. The manufacturer was understandably upset, the farmer was adamant about its organic certification, and ultimately fingers were pointed at the laboratory that initially declared the roots clean. Clearly some detective work was in order; luckily the contract laboratory was up for the task.

“Come, Watson, come! The game is afoot.” (ABBE) The first revelation was that the nine roots sent for testing were supposed to represent a batch containing 68,000 POUNDS of raw material from a single farm. So while the lab could say with a measured degree of certainty that the nine roots tested were clean, given the total amount of biomass, this sample would not have been particularly representative of all the roots in the batch.

This non-representative sampling explains the discrepancy in test results, although it does not explain how pesticide contamination occurred on an organic certified farm. At this point the contract laboratory, being curious (not to say relentless), asked the farmer what had previously been grown on the farm.

“All knowledge comes useful to the detective.” (LION) Well, it turns out that long, long ago, when DDT was still a legal pesticide, the farm had grown cotton. DDT does not decompose and can persist for decades, especially in certain types of soils, such as clay. So when the question came up: “Are there any clay deposits on the farm?” The answer, not surprisingly, was “yes.” Sure enough, subsequent testing of the clay deposit revealed the presence of DDT at very significant levels. Even still, the farmer argued that the soil had been tested before the crop was planted. And so comes the million-dollar question, “What sampling plan for the soil was used?”

“In solving a problem, the grand thing is to be able to reason…” (STUD) The farmer had sent someone out to collect soil for testing. In piecing together a likely scenario, one can imagine that without a sampling plan and specific instructions, they went to the easy-to-dig sandy loam. Unfortunately, the soil sample was not representative of all of the soil on the farm and the DDT hotspots in the clay deposits were missed.

21CFR Part 111 Section 111.80:
What Representative Samples
Must You Collect?

“Sample” seems like an innocuous and useful word. However, when applied to laboratory testing, the term is a confusing and often misused one. The word first appears in § 111.3: What Definitions Apply to This Part? The text reads as follows: “Representative sample means a sample that consists of an adequate number of units that are drawn based on rational criteria, such as random sampling, and that are intended to ensure that the sample accurately portrays the material being sampled.” As with many provisions of the regulation, the wording is clear on what needs to be done, but not on how to actually do it.

The GMP rule defines “Batch” as a specific quantity of a dietary supplement that is uniform and that is produced during a specified time period according to a single manufacturing record during the same cycle of manufacture. “Lot” refers to a batch, or a specific identified portion of a batch that is uniform; or in the case of a dietary supplement, produced by continuous process, a specific identified amount produced in a specified unit of time or quantity in a manner that is uniform. Whew!

Interestingly, despite the helpful FDA language, these words—“sample,” “lot” and “batch”—are often used interchangeably and inappropriately. For example, a colleague relayed to me a definition for “lot” used by an herb wholesaler as “everything I can grind up that farmers bring to me in a day.” IUPAC (International Union of Pure and Applied Chemistry) and ISO (International Organization for Standardization) define a laboratory sample as “the material that is sent to or received by the laboratory.” Think of this as the bottle of supplements or ground herbal material that is sent to the laboratory for testing.

This seems pretty straightforward, but once we cross over into the world of sampling for the purpose of adhering to cGMPs, the world becomes a darker and more sinister place. The biggest threat to the largest number of companies in the dietary supplement industry is the FDA inspector who expects you to justify that your “laboratory sample” is in fact a “Sample” that suitably represents that larger batch or lot. If it does not, then what you have is a “specimen,” not a sample, and a potential recall on your hands.

The word “sample” is fraught with peril in this context because FDA states that the manufacturer must examine a representative sample of each batch of the packaged and labeled dietary supplement to determine whether the dietary supplement meets specifications established in accordance with § 111.70 (§111.415) and “for a subset of finished dietary supplement batches that you identify through a sound statistical sampling plan (or for every finished batch), you must verify that your finished batch of the dietary supplement meets product specifications for identity, purity, strength, composition, and for limits on those types of contamination that may adulterate or that may lead to adulteration of the finished batch of the dietary supplement” (§111.75). While these requirements are very prescriptive, the agency has yet to issue guidance on how to build a statistical sampling plan.

I have heard it said quite bluntly in conversation that, “if the agency wished to blast the kneecaps off the industry, all it would have to do is ask to see the statistical justification for a manufacturer’s sampling plan.” Alas, when you use the “s” word (statistical), you open a can of worms. Hell hath no fury greater than statisticians in disagreement. The fact is there are competing models and theories about “statistical design” and thus there is always going to be room to argue the validity of any individual design. However, that is not an excuse to ignore what has been found to be a common cause for inconsistencies in laboratory test results.

The language in §111.75 obscures the reality of sampling, which refers to the process of choosing a representative “sample” that exhibits characteristics similar to those of the population as a whole. The reality is that potential exists for a large batch or lot to be non-uniform. As such, companies should have a sampling plan in place to ensure that what is sent to the laboratory is representative of the whole batch. Once the laboratory receives the sample, it is up to the laboratory to ensure the test sample is representative of the laboratory sample. For the most part, 3rd party testing laboratories have no control over how the laboratory sample was obtained and thus should not “certify” single lots or products based on an analysis of the laboratory sample, let alone multiple lots. The onus is squarely on the manufacturer to justify or rationalize how the results obtained on the laboratory sample may be applicable to the entire lot.

We’re Almost There…

Well, if you have made it this far you should now have a good understanding of “sample” and “laboratory sample” and how they may or may not be related. Clearly, what is needed is a simple method for determining the number of samples that must be taken from a population in order to ensure that it is representative. This need was identified some time ago and during the past eight decades a rule of thumb has been used that is referred to as the “Square Root of N plus One” (√N + 1) sampling rule.

Now we arrive at the origin of this column topic, which was inspired by one of my students who questioned me on sampling plans, where this calculation came from, and what it actually meant. Well, the truth is the rule is not statistically motivated and was developed out of convenience by an AOAC committee in the 1920s from a need to provide agricultural regulatory inspectors with a simple rule for sampling (Izenman, AJ, Statist. Sci. 16:35-57, 2001). (Despite repeated warnings that the √N + 1 rule lacks a statistical basis and provides a false sense of security, it remains widely used.)

Several variations of the rule, as noted by Saranadasa (Pharmaceutical Technology, May 2003, 50-62), have been used for several applications. For example, the square root of the lot size plus one is taken to determine the number of units to inspect, with each selected unit tested individually; the lot is accepted if zero defectives are found (if measurements are continuous, the lot is accepted if the average falls within given specifications). Or the square root of the number of cartons plus one is taken to determine the number of cartons to examine, and the sample size is determined by other means, such as from the ANSI/ASQC Z1.4 military sampling plan tables (ANSI/ASQC Z1.4). The modern approach to statistical sampling plans is quite a bit more complex than a simple “rule of thumb” and depends on many factors, including the nature of the test article, the expected failure rate, and the acceptable failure rate.

Fortunately, there are some good resources out there for those who wish to (or are required to) learn more about sampling plans. To get started, SQC On Line (http://www.sqconline.com/sampling-plans.html) provides some calculators. For the dedicated, perhaps the greatest resource is Pierre Gy’s Sampling Theory and Sampling Practice by Francis Pitard. Although written for geological sampling and mining, this work provides authoritative information on sampling, is recognized worldwide, and is used extensively in fields where the relatively small laboratory sample must be representative.

Ultimately, it is up to the end-user to determine the suitability of any sampling plan for their particular purpose. A measure of common sense must be applied to any circumstance; the cost of accidentally adulterating plantain with digitalis is much greater than that of only meeting 90% of your label claim for vitamin C. Consulting with an expert in sampling plans may be the best option for those dealing with a potentially high-risk circumstance. I personally found the statisticians at FDA and Health Canada to be very willing—eager in fact—to talk about statistics and sampling plans.NW