One of the most important steps in designing an experiment is setting what are called “Data Quality Objectives” or “DQO’s.” This is a fancy term for figuring what we need to know and why. It goes beyond well simple generalizations. It has to reach to what data has to be known, and how precisely must it be known to meet our business objectives. Without having clear DQO’s it really isn’t practical to try to design an experiment.
One of the maddening parts of trying to design an efficient experiment is that we are forced to make many decisions when we really don’t know very much. We are in a quandary as to where to start. One approach is to start from your best guess at the end result and work backwards. That is to say, determining what we are trying to accomplish and representing that as a calculation. If we can say what we are trying to do as a mathematical equation, the variables of interest become obvious.
Let me illustrate with an example. Let’s suppose that we want to save money on a chemical process. We think that by substituting a less expensive raw material we might cut our overall costs. Before we go very far into running any lab or pilot tests, we need to define clearly what we mean by “saving money.” We probably can’t test “saving money” directly in our lab or our pilot plant and no one in their right mind would turn an operating plant to a bunch of geeks to play with. Hence, we would have to gather pertinent data from lab or pilot scale work and use that to predict what the impact would be on total costs at a plant level.
In order to make any real progress, we would need to be able to put all the results of our testing into some kind of mathematical formula that expresses the relationship between potential raw material cost savings and other (perhaps negative) results that we might have found in our testing. If, for example, we found that changing raw material reduced yield, we would want know how much. Likewise, if we found that the change increased the probability of having serious plant failures, we would need to have some way to factor that into our assessment. Of course we would want to know how all these factors worked together on the average, but we would also want to know the variability of the result would be impacted by the variability of individual factors and what their combined effects would likely be. If changing to a new raw material ruined the reliability of our plant we would certainly want to know that.
If we can write down a mathematical equation to calculate our final result, we actually have a “model” for our process. Some might complain that we really cannot do that until after we have done our experimentation. This is fallacious thinking. How factors impact results is always a necessary hypothesis. Failure to state that overtly or at least make provisional hypotheses, leaves the experimental design wide open. I would point out that all continuous functions approach linearity over small increments. Hence, without evidence to the contrary, it is always reasonable to start with a linear model.
Once we have an overall working model we can then begin to design an overall strategy for testing hypotheses. Furthermore, using techniques described later, we can assess the variability and uncertainty introduced by variability in things like the variability of raw product cost, yield, quality, down-time, etc. And finally we can assess the uncertainty created in our final result created by uncertainty in our measurements.
It is rare that all the pertinent factors are put into one giant test program. Development and the accompanying experimental designs usually progress in a more compartmentalized fashion. That is just nature of experimentation – especially in its early phases. The findings of individual groups are brought together and evaluated in the overall model. It is only later in the process that many factors are tested at the same time to ensure that factor interactions are understood.
The evaluation of how the uncertainty and variability of individual factors impacts a final, derived result is known as “propagation of error.” This is a mathematical technique applied to the generalized (and subsidiary) model(s). We will hold off on the mathematical details for a while, and just try to get the general idea of how this technique works and why it is important. The illustrations used here will be kept quite simple. At the end we will point to how this simple approach can be applied to very complex situations.
Let’s suppose that we have found a way to predict the average cost of a new raw material (CM) and the average manufacturing and distribution costs (CP) that would arise from using the new raw material. Our predicted total average costs (CT) would be:
We would, of course, insist that this new CT would be lower than our current CT. It is nice to know the average or expected value of CT, but we also need to know the uncertainty around this new CT. We will go more into the math later, but it turns out that in this type of equation the variance of CT is equal to the sum of the variances of CM and CT. That is:
We now have a way to assess the uncertainty of this new cost structure if we can measure or predict the uncertainty around CM and CP. We might see, for example, a slight improvement over our existing cost structure, but we might also see a lot of uncertainty around that. We might hesitate to implement such a change, especially if we had a lot of confidence that our current cost structure was very predictable and profitable.
Now Equation 2 obviously roles up a lot of experimental data into just a few simple variables. In actual practice the experimenter is working at a much less a grandiose scale. A more realistic case for an experimenter might be trying to assess the uncertainty of chemical yield. Of course the experimenter might run many trials in order to measure the variability of the yield. Nevertheless, there are many occasions where a “propagation of error” calculations can be very useful both before and after experimentation.
We might want to do an a priori calculation to help us evaluate which factors are likely to be the most important to the precision of future measurements. Some errors are multiplied greatly by how they are used in the final calculation. A propagation of error calculation would alert us to the sensitivity of the measurements. This kind of calculation might reveal that, under current conditions, we can’t possibly meet the precision requirements of our project. We might need to redesign some of our analytical equipment before wasting money on hopeless trials.
We might also use this type of calculation a posteriori when some of the factors of interest in the overall model were not or could not be included in our experiments. It is sometimes the case that important factors cannot be measured in the lab or in a pilot plant. This often occurs with scaling and market issues. Here we might have to use “expert” estimates (often called “SWAGS”). It is also possible that some factors are actually known well enough that they don’t have to be verified by our experimentation. The work may have already been done by vendors or industry groups. A propagation of error analysis can bring together very disparate information and give some objectivity in what could be a very complex analysis.
Let’s look at an example more typical of what we would see in a pilot plant. Suppose we want to measure the yield in our pilot plant. Yield is often a result that we derive from several independent measurements such as:
- The mass of mixed product produced (P)
- A compositional analysis of the mixed product (wt% A)
- A mass of mixed raw materials (R)
- A compositional analysis of the mixed raw materials (wt% B)
Let us also suppose that the key yield indicator is mass of A in the product per mass of B fed in the raw material. Our yield formula (as a fraction) would be:
But we don’t weight A or B directly. We calculate it from:
Now suppose we would like to estimate the error in the yield before doing any experiments for some of the reasons stated above. It turns out that this type of error propagation is also fairly simple. When formulas are simple ratios, the square of the coefficients of variation are additive. The coefficient of variation (CV) is the standard deviation divided by the mean. Hence:
Formulas like Equation 5 can be very useful for helping us design and interpret experiments. If we know, for example, that we will have a hard time measuring the composition of B, we know, before even doing the testing, that the variability/uncertainty around yield will be high. We might want to get a better way to measure B before we do a bunch of testing. On the other hand, if we have good data for mass measurements from our vendors of weigh belts, load cells, etc., we might conclude that it would be a waste of time to check and recheck the mass measurement systems. We could do a simple verification that they are working as designed and focus our attention on the more difficult measurements (such as measuring composition).
As we have said earlier, it is almost always the case that we have only part of the experimental data we need to fully “prove” a complex business proposition. We may have experimental data for yield. We might even have some experimental data on failure rates, product quality, etc. Nevertheless, we will almost always have to make guesses or extrapolations rather than being able to actually measure the bottom line numbers like profitability, total costs, etc.
Nevertheless, if we can state the result of interest in a mathematical formula based on variables we have measured or can reasonably estimate, then we can apply a general formula for propagation of error. This may help us predict the variability of our result of interest even when we are short on experimental data. The general approach to propagation of error is described mathematically in the following paragraphs and equations.1
Let x, y, z,... represent random variables whose true values are X, Y, Z.... Let u represent a derived quantity whose true value is given by:
Let ε1, ε2, ε3,... represent the statistically independent and relatively small errors2 of x, y, z, ... respectively. Then the error induced in u, which is denoted as ξ, as a result of the errors ε1, ε2, ε3,..., has a variance equal to:
Equation 7 looks daunting, but when the formula for the result of interest is a mix of sums and products, the evaluation usually isn’t too bad. It requires successively taking the partial derivative of the model equation relative to each variable of interest while treating the other variables as constants. It also requires reasonable knowledge of the variability/uncertainty of each variable at or near the operating point of the model.3
Although the analysis can become complex, evaluation of the total error of a process is an indispensable technique. It is essential to have a reasonable and unbiased estimate of the uncertainty of any result that will impact a business proposition. By its very nature, business turns on the issue of uncertainty. Businesses can often command a premium for handling uncertainty effectively. High stakes can be very rewarding. Nevertheless, those who find themselves with the wrong mix of uncertainty and reward will inevitably fail.
1From Mandel, John, The Statistical Analysis of Experimental Data, Dover Publications, Inc., New York, 1964.
2The random variables must be independent and the relative error generally 10%. With large relative error the approximation does not work well.
3It is very important to not apply this technique for conditions well outside of conditions that have been tested. It could be quite incorrect to assume that the variability/uncertainty of a random variable remains constant regardless of conditions. It is quite common for models to fail when applied to untested conditions.
Comments