Technology Development Network

Mass Balance for non-Engineers

In every pilot plant test the issue of “mass balance” comes up.  In just about every case there is a vague notion that, “We need a good mass balance.”  Unfortunately the concept is rarely well defined or well understood by managers, supervisors and operators.  These are the folks who have to run the pilot plant so that a “good mass balance” can be achieved.  Unfortunately, this often this leads to poor and even unusable test data.  This blog discusses mass balance in pretty simple terms for those folks who are not engineers, but need know about mass balance.

In simple terms, a mass balance is a calculation.  It is an accounting of what went into and came out of a process.  This purpose of the mass balance is to serve as a check of the measurement system.  Since mass is essentially conserved in any physical or chemical process, the amount of mass that goes into a process should equal the mass that comes out of a process.  If this is not the case, then the measurement system may be flawed.  Hence, a mass balance is not important in and of itself.  It is a check on the measurement processes.  It is not unlike a year-end retail store inventory.  We will use that as an example.

A year-end inventory is a counting of the actual physical inventory in a store.  During the year the store keeps track of merchandise that was received and merchandise that was sold.  We would want the starting number of various items (N_INT).  This would be the inventory at the start of the year.  We would also want the number of items received (as in purchased) in the year (N_PUR) and the number of items sent out (as in sold) in the year (N_SOLD).  We would compare the final number of items that we counted in the physical inventory (N_FINAL) with what we calculated should be the final count.  If N_FINAL = N_INT + N_PUR - N_SOLD all would be well and we would be home early for New Years. 

If the N_FINAL count was off a little bit we might recheck our count and calculations but we would not do much about the difference.  We would expect some amount of error, even when counting items.  It would be important to define what we meant by being off a little. 

If N_FINAL was not at all what we expected, we would start worrying that something was really wrong.  Perhaps the data were incorrect.  Perhaps some inventory counters were counting individual shoes while others were counting pairs of shoes.  Maybe there was returned merchandise that was being double counted.  Perhaps we were being shorted on some of our purchases.  Maybe some items were transferred to other stores and we didn’t get that information.  Of course there could have been a certain amount of shoplifting and even employee theft. 

When the inventory doesn’t balance as closely as we expect, we would start looking for problems.  The more detail we have about the items the better our chance for figuring out what went wrong.  This is something that we have to design into our inventory system long before doing our final inventory. 

At first we might want to keep our inventory system very simple.  As our store became larger and more complex we would want to collect more detailed information about our items.  We might be able to learn about our customer buying habits or our store’s ability to control particular inventory items.  We have to do this at the start of the year and collect the same data consistently throughout the year or we won’t be able to make any sense of what was going on.

A mass balance for a test run is very similar to an inventory.  First and foremost we must set up our system before the test so that we collect all the needed information consistently through the test.  A simple mass balance will tell us if everything was properly accounted for, but will not tell us much about what might have gone wrong if something does go wrong.  In a chemical process our inventory items can be molecules or atoms.  In most cases molecules change form and are not “conserved” through the process.  Atoms are not destroyed, they just change their associations.  For example, methane might react with oxygen to form water and carbon dioxide as shown below:

 CH₄ + 2O₂  ->  CO₂ + 2H₂O  (1)

Notice that the molecules change, but that we have the same number of carbon, hydrogen and oxygen atoms at the beginning and the end of the reaction.  This is typical of chemical processes.  If we “inventory” all the atoms at the beginning of the process and again at the end, we should find the same number of atoms.  If we express that inventory as mass then we are doing a “mass balance” by various elements.  If we don’t get the same mass of carbon, hydrogen and oxygen going into and out of the process we know that something went wrong.

In a chemical process things are a bit more complicated than in our inventory example.  There are at least three reasons:

1. Physical measurements are being taken rather than things being counted,
2. In many cases the physical measurements are not direct mass measurements but rather are several measurements that must be converted into a mass number (e.g. measuring concentration of a component and then multiplying by a measurement of flow to get the final estimate of mass) and
3. Some measurements are easy and accurate while others may be difficult or even impossible.

 The first item introduces measurement error.  It is an inescapable fact that every physical measurement is subject to measurement error.  The second item tends to multiply that error as multiple measurements are combined to get final answers.  Hence, the numbers never work out exactly right.  It is very important to think through what the measurement error will be and adjust expectations accordingly.  This can be done using a procedure called “Propagation of Error."  We need to know how close is “good enough” and this is never a black or white decision.

The third item will drive our thinking and our Sampling and Analysis Plan.  It makes no sense to try to draw any conclusions from data that are highly suspect.  We must design our testing around measurements that can be made with reasonable precision and accuracy.  This has to be a big part of planning when setting up any kind of experiment or trial. 

Let’s illustrate how a mass balance might be used in a relatively simple (imaginary) chemical process.  Let’s suppose that we are mixing together chlorine gas (Cl₂) and methane gas (CH₄) to make liquid methylene chloride (CH₂Cl₂, a solvent used in paint stripper).  Our purpose for the plant trial might be to try out some new operating conditions to see if we can improve yields.  The reactions of interest might be:

Cl₂ + CH₄ -> HCL + CH₃Cl (a gas we don’t want) (2)

CH₃Cl + Cl₂ -> HCl + CH₂Cl₂ (the liquid product we want) (3)

CH₂Cl + Cl₂ -> HCl + CHCl₃ (a liquid product that is OK if in small amounts) (4)

CHCl₃ + Cl₂ -> HCl + CCl₄ (a toxic liquid product we don’t want at all) (5)

 Notice that the first reaction is just a first step in getting to our product.  We really don’t want the intermediate, CH₃Cl (called methyl chloride which happens to be a gas).  What we might do is recycle the gas back through the reactor to get the CH₃Cl to become our desired product, liquid CH₂Cl₂.  Nevertheless, we would have to be careful that we don’t recycle things other than CH₃Cl.  If we do we will make too much CHCl₃ (chloroform) or CCl₄ (very toxic carbon tetrachloride).  How and how much we recycle the CH₃Cl might be a critical operating parameter that we would test.

Our simplified process might look like:

 

  Since we are interested in improving the yield of CH₂Cl₂ relative to CH₄ and Cl₂ we would certainly want to measure the amount of CH₂Cl₂ produced and the amounts of CH₄ and Cl₂ used.  Nevertheless, if we just measure these, we will never be sure of our data.  Our measurements of one or more of these might be way off and we might be fooling ourselves into believing something that wasn’t true.  If, for example, our measured yield was 80% based on measuring CH₄ and CH₂Cl₂, but our CH₄ number was actually 50% low, then our yield would be off by 50%.  Our true yield would only be 40%.  Sooner or later we would figure that out, but it might be a very costly error. 

The best way to avoid these kinds of serious errors would be to account for all the carbon, hydrogen and chlorine going into and coming out of the process.  In the case of the 40% instead of 80% yield, a good accounting of the amount of carbon going in and coming out of the system could be very helpful.  Our calculation might show that we seemed to be getting twice as much carbon out as we were putting in.  Since this is clearly impossible we would quickly be tipped off that something was wrong with our measuring system.  If we had collected redundant data through the test (perhaps measuring methane flow by two independent means), we might be able to “salvage” the test by using the alternative measurement method. 

If we can identify and measure all compounds we can make separate mass balance calculations based on the all the different types of atoms (i.e. elements).  That can give us redundant data that might help “salvage” a test should something go wrong.  If all of these independent mass balances work out well we would have high confidence in our data.  Unfortunately, it is rarely the case that we can measure all inputs and outputs with good confidence.  Our example helps illustrate this.

In our imaginary systems we would probably find it rather difficult to accurately measure the HCl (hydrochloric acid gas).  It is a very reactive toxic gas that easily dissolves in water and reacts with metals.  In our pilot plant we would be a lot more interested in removing the gas for safe disposal than collecting samples for accurate measurement.  If we found it impractical to measure HCl, then we could not do mass balance calculations on hydrogen and chlorine.  We would have to do our mass balance on carbon only.  We would have to be especially careful in identifying and measuring all of the hydrocarbons going into and coming out of the process.  Any mistakes with this would mean that we wouldn’t have a mass balance calculation of any kind.  If that should happen we would have no check on the measurement system and all the data from the test would be suspect.   The whole test might be ruined. 

In doing a proper mass balance it is critical that we select sampling points for composition and flow very carefully.  Figure 1 shows 7 possible points (yellow numbered boxes).  Nevertheless, in a real pilot plant there are usually huge constraints on where samples may be taken and flows may be measured.  Some points are physically inaccessible while others may be impractical because of temperature or pressure.  In Figure 1 we would want to select Points 1, 2 and 7 for our mass balance sampling and flow measurements.  We would certainly want to avoid Point 3 (the recycle loop) for a mass balance calculation.  That point might give us very useful process information, but if we included that flow in a mass balance calculation it would be like double counting returned merchandise in our inventory example.

There is one other point to make that sometimes gets lost in all the complications.  Just as in our store inventory example, the beginning and ending inventories of tanks, reactors, hoppers and etc. must be considered.  If in our imaginary plant, reactants can build up in the reactor or products can accumulate in the separator, we must take into account how much reactant or product was part of building up or draining off those levels.  In a surprising number of cases these inventory measurments turn out to be very difficult and can add tremendous error to overall calculations.

In this simple example we have illustrated some of strategies and concerns that need to be considered in any Sampling and Analysis Plan for a pilot plant trial.  The concepts are pretty simple, but there are devils in the details.  Flows in pilot plants are often complicated and subject to revision.  Such plants are often designed for multiple uses and may not be optimal for the current trial.  It is not uncommon to discover after the fact that vent, flare or recycle lines were left open unexpectedly or to find that measurement devices were improperly sized, were out of calibration or just were not of the proper type.  It is also not uncommon to find that data of all types were not properly recorded during the trial.  Whereas much time is often spent on evaluating laboratory methods and procedures, it is actually rare that the same kind of scrutiny and care is given to the much more important field and plant data collection and measurements. 

So what is a “good” mass balance?  Most measurement techniques can operate easily within an error bar of a few percent (1% or 2%).  When these errors are combined, the resulting error is often around 5% relative.  There are exceptions to this rule of thumb, but a “good” mass balance is usually around 95%.  That is to say the mass out is within 95% of the mass in.  Real pilot plants often fail to meet that 95% criteria due to an accumulation of constraints.  Nevertheless, any mass balance less than about 90% probably indicates significant measurement failures.  By the way, it is rare for accumulated error to overcome natural losses from leaks, adsorption, etc.  Hence, if the mass out exceeds the mass in this is usually an indication of a serious measurement or data collection error.

In summary:

1. Mass balances are crucial mathematical checks on the validity of all data collected during a pilot plant trial,
2. Careful planning and exacting implementation are required to get good mass balances,
3. Coordination with managers, supervisors and operators of the pilot plant is critical to achieving a good mass balance and
4. Laboratory data are very important, but careful attention to plant measurements are usually more important – failures in the field are more often the cause of things going awry.

The Power of Statistics is in Statistical Power

In a previous blog we introduced the concept of statistical power (see: “Protecting Against Experimental Failure”).  There we pointed out that calculating statistical power is one of the most important statistical techniques for designing useful experiments.  Nevertheless, it is seldom used and often used incorrectly.  There are two problems:

1. It can be difficult to conceptualize how statistical power works and
2. It is rather difficult to calculate. 

Modern statistical software makes the calculation fairly easy.  This blog will focus on the concepts rather than diving into complex mathematics.  We will deal with some of the mathematics, but we will try to stay conceptual and leave the detailed math to folks like Mark Anderson and Pat Whitcomb of Stat-Ease.

Let’s begin with a simple example.  Suppose that we are the junior basketball scouts for the University of Kansas.  Being the new folks, we are surprised to find that we have been assigned to the new international recruiting program.  The idea is to go out and find unknown international high school talent and offer key players scholarships to KU.  Naturally funds are short.  For reasons beyond our comprehension, we can choose to go to Senegal or Estonia for three months during basketball season.  We have to choose one or the other, but we cannot do both.  We have no idea what to do.

Fortunately the Chancellor is willing to help us out.  She has heard that the best indicator of basketball success is vertical leap.  Unfortunately, vertical leap data for high school players in Estonia and Senegal are not readily available.  She has managed to get some funding for limited trials from the statistics department at KU.  We can make a three day trip to each country to test the vertical leap of high school players to help us decide which country to go to for our recruiting.

Since this study is funded by the statistics department we think we need to approach our testing using what little we know about statistics.  We know we need to state our hypotheses.  We suspect that Senegalese and Estonians players will have similar vertical leaps.  If they do, we will go to Senegal since we hate the cold.  We create the following “truth table:”

The four highlighted boxes indicate the four decision possibilities.  In two of the boxes the experimental data will lead us to the correct conclusion.  In the other two boxes the data leads us to wrong conclusions.  We start by using the typical α = 0.05 and plan to crank through the numbers we get using a typical statistics program.  So far so good.

Now we need to determine how many players need to be tested to give us valid data.  This is trickier than it appears at first blush.  If we set the sample number too low we will have a poor estimate of the average vertical leap and might not be able to make a good judgment on what to do.  The way the math works, if we take even a few good samples we can generally avoid concluding that the vertical leaps are the same when they are actually different (a Type 1 Error).  This is because we have set α at a pretty low number.  Nevertheless, if we have too small of a sample set we could have a really good chance of concluding that the vertical leaps are the same even though there is a significant difference between Senegalese and Estonians players.  This is a Type 2 Error.  To avoid that kind of an error we need to think through β.    

We have described in words what statistical power is.  Mathematically it is:

      Power = 1 – β 

             where:  β is the probability of a Type 2 Error

It turns out that β is not that easy to calculate and it depends on a number of factors.  The actual calculations are better left to statistical programs.  Nevertheless, we can pretty easily figure out the important factors if we just think about this type of error.  As we said above, this type of error means that our data set doesn’t reveal a significant difference and we conclude there is no difference.  In other words, our data does not reveal that there is a difference in vertical leap between Senegalese and Etonians and we conclude that, for practical purposes, they have the same vertical leap. 

It is paramount to think through what we mean by “significant.”  Does it really matter that the average difference in vertical leap is ¼”?  How about 2”?  Surely a difference of 6” would be important.  In any case, it is intuitive that it takes “better” data to find a difference of ¼” than 6”.  We can think of this as trying to set the “resolving power” of our experiment.  This is one of the most important decisions to be made in setting up a useful test protocol.  It needs to be made before any testing is done.

It is also intuitive that the more scatter the population, the more samples we will need to get a reliable estimate of the averages.  This is a consequence of the Central Limit Theorem and can be expressed mathematically by:

      σ_mean = σ_population/SQRT(n)

                   where:  n = number of samples

Let’s think through these issues in our case study.  We will also look at these ideas graphically.  We plan to go to Senegal and Estonia and measure the vertical leaps of randomly selected high school basketball players.  We will get an average and variance for each of these groups.  The averages could be similar or far apart.  The variances could be small or big.  Let’s look at a few possibilities.  Let’s assume that one average is 18” and the other 20”.  Let’s also assume that the standard deviation of the both populations is 5” (by the way this is a pretty big assumption).  And finally, let’s say that we are going to pick just 5 players to test.  If we plot the probability distributions for these mean values it looks like Figure 1:

 
   Figure 1

This shows us the probability distribution for the mean values of our testing.  We can see that there is a pretty good chance that average calculated from the population with a higher mean will actually be the same or even less than the average calculated from the population with a lower mean.  This indicates poor resolving power.  It is highly unlikely that from these data we could convince us that the average leaps are different with any real confidence even though the averages really are different by 2”.

If we use the same model and increase the sample size to 40 the results are much different.  This is shown in Figure 2 below:

Figure 2

In only difference is that we took more samples.  This will get us a much better estimate of the means of the groups.  Here we see that there is a pretty good chance that the data would show that one group had better leap than the other.  The only difference here was that we took a lot more samples to get better estimates of both the means.

Now suppose the groups naturally have smaller standard deviations than we assumed earlier.  These data are shown in Figure 3 below:

 Figure 3

With a natural standard deviation of 2 it only takes 20 samples to get very good estimates for the means of the groups.  We would expect these data to tell us something pretty useful about the differences in the average vertical leaps.

And finally, let’s assume that the standard deviations are the same as in Figure 3, but that the average leaps are different by 6”.  The distribution of the means look like Figure 4 below:

Figure 4

These figures show that larger sample size, smaller standard deviations and a wider difference between the means all tend to give us better “resolving power.” 

Now let’s think about what we need to do to design an experiment with a specific resolving power.  Forget about how exactly to calculate it, the software will do that.  What do we need to know to properly design the experiment?  We need to know two of these three variables.  We are usually trying to decide how many samples to take.  Hence, we need to know the other two in order to do a good design.  Specifically we need to know:

1. The standard deviation of the results (in our case – the vertical leap) and
2. The difference between the means (at least the difference of interest).

At first this sounds like circular reasoning.  We need to know the answers before we do the test?  Well…in a way…yes.  If you design an experiment with no idea of what is going on you will probably be way off in the design.  You will almost certainly get Type 2 Error or you collect way more data than is really needed.  Getting reasonable estimates for these is not as difficult as it might seem at first.

Often we can look at previous data or similar data to get a reasonable estimate of the standard deviations.  In our particular case we might do a quick study of the vertical leaps of US high school players.  We might even be able to look up these data.  It would give a reasonable estimate of the variability we might see among Senegalese or Estonians.  Let’s suppose that we went back to the Chancellor and she found data in the study she quoted.  There she learned that US high school players had an average vertical leap of 20” with a standard deviation in vertical leap of 4”.  That would be extremely helpful in designing our experiment.

The second item is interesting.  We don’t really need to know the differences in the means between Senegalese and Estonians.  What we need to know is how important are differences.  It is really a sensitivity analysis.  It is an evaluation of how important is a small difference.  If a ¼” difference in vertical leap is not important, there is no reason to design an experiment that can reliably find a difference that small.   Let’s suppose that the Chancellor’s study really didn’t give us that information.  What would we do?  We might go to the basketball coach and his staff and get their opinions on the matter.  Let’s suppose that they thought anything less than ½” difference in vertical leap really didn’t matter as much as many other factors.  That is also very helpful.  Now we can put numbers into a program and see what it tells us.

We can now put these estimates into any of a number of statistical data packages and calculate β.  Use Stat-Ease Design Expert 8 we can construct the following table:

This tells us that if we test only 20 players we have almost a 90% chance of making a Type 2 Error (1 - 0.12 = 0.88).  If we test 1000 players we reduce the probability of a Type 2 Error to less than 1%, but the cost of testing that many players could be prohibitive – especially given the logistics of getting around in Senegal.  The “ideal” β is between 0.80 and 0.95.  For this experiment, the “ideal” sample size would be between 250 and 400. 

These are fairly high numbers, but it is not outrageous to think that we might be able to test 125 to 200 Senegalese and Estonian players for vertical leap in a matter of 3 days in each country.  It would require some planning and coordination, but it could be done.  We can now talk to coaches and administrators in Senegal and Estonia to see if these details can be worked out.  If not, we might need to find a different way to get the testing done (by folks in-country?) or find a different, more sensitive test.  That is the power of this kind of pre-planning.  We now know specifically what the problem looks like and can plan accordingly.  This is a great improvement over having “no idea what to do.”

This procedure has been described for a single test result (vertical leap).  Nevertheless, with good statistical software this approach can be applied to trials involving many results.  This is the beauty of modern statistical software.

Given A, What About B?

A surprising number of business decisions boil down to an exercise in Conditional Probability.  Often we observe one bit of data (a piece of news, a bit of test data, etc.) and then wonder about a related outcome.  Here are just a few examples:

• The price of crude oil just rose above $100/barrel.  Should we buy stock in an alternative fuels company?
• Monthly sales for model X dropped by 5%.  Should we cut production of model X?
• The latest trial on a new process indicated that yield improved by 15%.  Should we switch to the new process?

These decisions are never simple.  They always involve uncertainty.  What’s more, even when we have good data we often blunder in the logic.  Too often we put too much faith in the connection between the observed “fact” and the outcome.  This is especially true if the outcome is charged with emotion – good or bad.  We easily convince ourselves that the observed “fact” “proves” that we are in line to make a fortune or that the world as we know it will certainly end.  Often the outcome is much less certain than we think it is.

Consider the following example:

Suppose that large studies show that about 1% of the general population has a particular type of serious cancer.  Also suppose that there is a test for this cancer, but it is not perfect.  Again, suppose that we have good studies that show:

1. 80% of patients who have the cancer test positive for it by this method and
2. 9.6% of patients who don’t have the cancer test positive for it (i.e. ‘false positive”).

Now suppose that a patient tests positive for this cancer.  Should the patient immediately undergo expensive or risky treatment?  Part of the answer lies in what are the chances that the patient actually has the cancer?  Think about that for a moment before going on.  What would your guess be?

The correct answer is that there is only about a 7.8% chance that the patient actually has the cancer.  Additional testing should be done before starting any treatment.  Most people (including many doctors) get it wrong and guess a much higher percentage.  The solution can be calculated using Bayes’ Theorem for Conditional Probability, but it is complex and not very intuitive.  There’s a simple way to approach these types of problems without having to dig out the statistics book.  We can use a table to catalog all the various possibilities and fill in missing pieces.  Not only does it help us organize our data, it helps us determine if we have enough data to actually solve the problem. 

Consider Table I:

Where:

• T+ = Positive Test
• T- = Negative Test
• C+ = Having the Cancer
• C- = Not having the Cancer

Here we have listed the possibilities of having or not having the cancer (C+/C- respectively) and testing positive and testing negative (T+/T- respectively).  We have also listed what we know from our research and testing.  We show on the bottom row the fractions of the total population with and without cancer (0.01000 and 0.99000 respectively).  In the “C+” column we show the fractions of those who have cancer that will test positive and those who won’t (0.00800 and 0.00200 respectively).  This is simply 0.01 X 0.8 and 0.01 X 0.2.  In the “C-“ column we show  those who do not have the cancer.  Of these patients, 7.8% will still test positive.  This is .078 X 0.99 = 0.0945.  This is shown in the “T+” row of the “C-“ column.

Having filled in what we “know” in Table I, we can now use the table to calculate the missing pieces.  We sum across the “T+” row to get 0.10304.  This tells us that 10.3% of patients will test positive for all reasons (whether they have the cancer or not).  Then we can calculate the last number of the “T-“ row by subtracting 0.10304 from 1.  This tells us how many will test negative for all reasons.  Then we can calculate the middle number by 0.99000 – 0.09504 = 0.89496.  The results in Table II (calculated data highlighted in yellow):

 

The first row represents all those who will test positive broken down into those who have and those who do not have the cancer.  The total fraction is 0.10304.  The fraction with the cancer is 0.00800.  Hence, the probability of having cancer given a positive test is 0.00800/0.10304 = 7.8%.  This is not nearly as high a number as most people would guess.  Generally they give too much weight to the notion that the test is “80% accurate.”   There are three factors come into play complicating our thinking:

1. Reliability of the test to measure the outcome,
2. The number of “false positives” that the test gives and
3. The “real” distribution of the outcome under test.

It is hard to “visualize” all these factors.  Using the table helps sort them out and weigh them objectively.

By the way, we have stayed with fractions in this example.  It is sometimes more intuitive to use absolute numbers instead of fractions.  If instead of using fractions, we talked about what a population of 10,000 patients, our table would look like:

 

The result would be the same (80/1030 = 7.8%) but by using numbers of patients instead of fractions it is often easier to conceptualize the problem.  In fact, when the problem is stated in terms of individuals instead of fractions, a higher percentage of folks actually get the right answer.  Hence, it is often useful to state the problem in terms of numbers of individuals instead of using fractions (i.e. talk about “100 out of 10,000” instead of 1%). 

I will give one more example in the form of a “case study.”  This illustrates how this might be applied directly to a business decision.  Let’s assume that we are given percentage data, but we prefer to use a hypothetical population of 10,000 in our analysis. 

Suppose we have been told by the SBA that only 20% of all startups are successful.  Now let us suppose that Joe Venture has done a study of successful startups and has noted that 90% of the successful startups have CEO’s who are taller than 6 feet.  He promotes this as a new, secret formula for predicting success and begins to raise venture capital money to fund these “sure-fire” winners.  Now along comes Sally Skeptic.  She also does a study of startups.  She notes that of those that failed only about 30% of them had CEO’s less than 6 feet tall.  Just how “sure-fire” is Joe’s new investment strategy?

Let’s apply these percentages to our hypothetical 10,000 startups.  We expect that in 10,000 startups there will be 2000 successes and 8000 failures.  Joe’s study tells us that in the 2000 successes we should expect to see 1800 tall CEO’s.  Sally’s study tells us that in 8000 failures we should expect to see 2400 short CEO’s.  We can summarize what we should expect in Table IV:

Where:                                                

• T+ = Tall
• T- = Not Tall (“Short”)
• S+ = Successful Startup
• S- = Not Successful Startup

We can fill in the rest easily and calculate the probability of a Successful Startup given a Tall CEO.  See Table V below (calculated data highlighted):

 
Probability of Success Given a Tall CEO = 1800/7400 = 24.3%

Even if Joe’s data are correct, his investment strategy improves the success rate from 20% to about 24%.  That is an improvement, but not exactly “sure-fire.”  The confounding issue here is that most CEO’s (74%) are tall.  Hence, Joe’s data doesn’t mean as much as it appears at first blush.

This technique can be applied to even more complex problems provided there is clear thinking and enough data to enumerate and characterize all potential outcomes.  It is a great technique to organize thinking and to ferret out logical errors.

Although this seems straightforward enough, this technique is often not used.  It actually takes good business discipline to force objective analysis – especially when outcomes are emotionally charged.  If the outcome is a positive result from management’s pet project, questioning the notion that “A proves B” might be treated as heresy rather than good thinking.  Likewise, if the consequences of ”B” are dire, management might overreact and take action when there is little chance that “A proves B.”  When outcomes are extremely important or charged with emotion, intervention by a consultant may be the only way to get an objective analysis. 

 

Acknowledgements:

I would like to acknowledge my friend and colleague Bill Schafer, President and Chief Executive Officer of Resonant Management Solutions.   Bill brought up this important topic and asked me to try to explain it in simple – non-statistical terms.  He has run into this type of management error many times in his career and hopes to assist clients in dealing with these complex decisions effectively. 

 

Design of Experiment (DoE) Discipline – A Success Story

I recently returned from an extensive plant trial where things did not go as planned.  There were a number of issues, most of which were beyond the control of plant operations.  Analytical equipment failed, feed stock flow measurements did not go as planned and unexpected fuel feed problems occurred – along with the usual missed or spoiled sample.  Nevertheless, the trial turned out to be a success due to good planning guided by DoE principles.  By using the discipline of DoE, the company was able to meet several evaluation goals that might have otherwise not been achieved.

The experimental design for this trial was particularly difficult because there were many unknowns and potential problems.  There were also a wide variety of desired test conditions and results.  The pilot plant was an older plant with many known and suspected reliability issues.  Careful analysis of previous maintenance logs indicated that the plant was unlikely to operate at stable conditions for more than about 120 hours.   Based on preliminary estimates of flows this at first appeared to be ample time to get many different data points.  It was first thought that as many as 60 different points might be possible.  At first blush it seemed feasible to run a DoE of 3 numerical factors (temperature, feed rate and one feed component) at 3 levels and 1 additional categorical factor (another feed component on or off).  It was hoped that useful results on 6 output results (such as yield, product composition, etc.) could be obtained.  Initially this looked to be something like 54 runs.  This would have been within the number of runs that could have been achieved.

Fortunately, the experimenters did not move glibly forward but spent many hours agonizing over the details of the planned test.  When looking into the details of the measurement processes many things were learned.  These included:

• It could take several hours to reach a steady operational state
• It would take several hours to accurately measure some of the input and output variables
• The precision and sensitivity of several measurements began to show that many more runs would be needed to get useful precision on several key results

All these considerations would indicate the need for more runs.  As the number of runs grew, it became clear that objectives would have to be re-evaluated.  It was especially important to consider the risk of the 120 hours of run time turning out to be too optimistic.  The number of levels had to be reduced.  The statistical validity of some very desirable results had to be compromised or even abandoned.  It became essential to set high priority on some factors and low priority on others.  Hence, even the run order had to be considered when dealing with so many potential issues.  And finally, careful consideration was given to critical analytical measurements.  Where data was absolutely crucial, redundant measurement methods were instituted.  The bottom line was that the experimental design had to be altered very dramatically if there was to be much of a chance of success.  Furthermore, the trial was postponed until all these issues could be adequately addressed. 

As it turned out, almost everything that could go wrong did go wrong.  For a number of reasons, the stable testing period was only about half of what was hoped for.  Some automatic analytical equipment failed and had to be backed-up by manual methods.  Nevertheless, because crucial factors were tested at the beginning of the trial, stability of the operation was constantly monitored and verified before moving to the next run and back-up analytical systems were put in place, it became possible to draw significant conclusions about the most important factors. 

Not everything that was hoped for was achieved in this trial.  Nevertheless, had the experimenters not taken the time to look into the details and set priorities or had they been cavalier about the experimental risks, the trial would have been a complete failure – nothing would have been learned.  This was an excellent example of how good DoE planning can salvage the situation even when many things go wrong.

Protecting Against Experimental Failure Using Design of Experiment (DoE) Principles

If you are reading this Blog you might be thinking, “Do I really need to use DoE?”  Well, if you ask the question, the answer is probably a qualified, “Yes.”  It really doesn’t matter if you are just starting an investigation or about to begin what you think will be your final trial.  If you haven’t considered the issues of DoE you need to stop and do so.  As we have stated in earlier Blogs, how formal and how detailed you need to be with your DoE depends on where you are in the process and what is the cost of “experimental failure.”  By “experimental failure” I don’t mean blowing up a plant – that kind of risk is never justified.  What I mean is the risk of getting to the end of a trial or experiment and having ambiguous or conflicting data.    

The last thing an experimenter wants is to spend a million dollars on a trial and have ambiguous data.  The last thing an organization wants is to spend a million dollars on a trial and then make decisions on invalid data.  Unfortunately, some organizations end up spending millions of dollars chasing phantom “benefits” while others miss golden opportunities that were right in their backyard.  So, how does one prevent experimental failure?

There are a few relatively simple techniques that can help protect against “experimental failure.”  They are:

• Clear identification of hypotheses
• Randomization of trial conditions and
• Evaluation of the resolving power of the trial

Often ideas are best explained by using an example.  Let’s suppose that you are part of a group that wants to do a trial on a chemical pilot plant.  Let’s suppose that we want to study acetic acid yield in a catalytic process.  We want to vary the amount of moisture in the carbon monoxide feed and see what that does to acetic acid yield.  And finally, let’s suppose that there has been no formal DoE performed.  This last item is, of course, an immediate red flag.  The trial is going to be expensive in materials, energy and time.  If the trial was only going to take a few hours in a lab, then one might let the lab folks play around for an afternoon, but even at a pilot level, the costs of a single trial are tens if not hundreds of thousands of dollars.  Again, the costs can explode if the wrong conclusions are drawn and the organization invests in a “benefit” that turns out to be illusory.

If nothing has been done in the way of DoE, the experimenter must look at:

• The hypotheses being tested in the trial
• The trial factors (variables to be adjusted)
• The results to be reported
• The expected causal relationships between factors and results
• The expected precision and accuracy of measurements
• The procedures for adjusting factors and measuring results
• The amount of redundancy built into the trial or experiment

If these issues have not been looked at carefully then the trial is in danger of being ambiguous.  This is especially true if the trial has been designed by a group of persons who have not done a formal review of the trial.  Very often a committee based design fails because the committee has failed to do a final rationalization between the various points of views, assumptions and needs that have been discussed and the final “agreed upon” plan of action.  The resulting plan could be a “political” construct that is not even internally consistent.  

The first step is to ensure that the hypotheses have been identified, recorded and vetted.  There must be no “hidden agendas” or last minute modifications.  If, for example, a key stakeholder in the trial doesn’t care about acetic acid yield but only about acetic acid purity, that needs to be discussed and either included in the hypotheses or dismissed.  Furthermore, these hypotheses need to be prioritized.  There may be some essential hypotheses that are the justification for the trial, without which it makes no sense to run a trial at all.  There may be some “nice to have” data for secondary hypotheses that are optional but not essential.  Things never go exactly as planned – especially in a plant setting.  It is critical to keep focus on the essential hypotheses being tested.  Often it is necessary to make decisions during a trial that compromise or even eliminate data.  Those decisions should not compromise away the validity of the trial for crucial hypotheses.  If clear priorities are not set, the trial could be inadvertently compromised in the “heat of battle.”

The next step is to review the selected factors that will be varied.  They must make sense.  The variation must be verifiable or measureable.  The variation applied must be sufficient to result in a measurable variation of the result.  This last item is a key sticking point in many trials.  There must be some belief that the intended variation of the factor will have not only a measureable, but significant impact on the measured result(s).  This can get especially interesting when the intent is to prove that the factor does not cause a significant change in result being measured.  It is surprising how many trials are designed and implemented that have no hope of producing useful data.  Very often this can be traced to the setting of factor variation limits which turn out to be either impractical or irrelevant.

If, in our example, we expect the water content of the carbon monoxide stream to have little impact on the yield of acetic acid, it wouldn’t make much sense to vary the amount of water by only a very small amount.  We would only see the random variation in measurement of acetic acid yield.  We would need to vary it by an amount that we really believe should give a measurable change.  On the other hand, it would not make sense to try to put in so much water that it wouldn’t stay vaporized in the gas stream.  Even if we don’t know what the “right” amounts of water to put in the carbon monoxide stream,  we need to have sound thinking around thes and discuss our expectations before running ahead with an experiment that might be doomed to failure.

One of the key techniques used in effective DoE is the randomization of trial runs.  Instead of varying factors in a systematic way, it is important to vary them in a random order.  Plants hate this because it complicates operation, but the need to do this whenever possible cannot be emphasized enough.  This is because there are many factors that we either don’t know about or cannot control.  If we randomize the adjustment of our factors of interest then these other factors become random errors that are less likely to lead us astray. 

In our example, let us suppose that the water content has no real effect on acetic acid yield.  Let us also suppose that the yield is highly dependent on the age of the catalyst.  Let us also suppose that the catalyst starts with low yield, quickly goes to a maximum and then slowly drops off, but we don’t know that.  If we vary the water content from low to high during our trial and the catalyst is going through activation, we might incorrectly conclude that adding water improves yield.  If we vary the water from high to low during the trial, we might conclude that reducing water improves yield.  If we ran the trials when the catalyst is de-activating we would draw different conclusions.  If, on the other hand, we varied the water content between high and low values randomly, we would see wide fluctuations in yield, but we would not see a trend related to water content and would not erroneously conclude that changing water content changes yield.  Although randomization of a trial may be difficult, failure to do so often results in invalid results that are dangerous to have laying around.

One of the most important techniques is what is called evaluation of the power of resolution.  Here, several ideas come together.  We need to know (or estimate) the precision of our measurements and compare them to the importance of the measured change.  This can be used to estimate the number of measurements that will be needed to make a statistically valid conclusion.  Again, an example is the best way to illustrate.

Let us suppose that we can measure acetic acid yield with a precision of 1%.  Let us also assume that we would be very interested in a 2% increase in acetic acid yield, but anything much less than that just wouldn’t be very exciting.  We might conclude that we really want to be able to “see” a 1% change in yield.  Since this is very close to our random error in measuring yield, we know that it will take many observations to make a statistically valid conclusion about yield.  Estimating the number of trials needed to make statistically valid conclusions is a very important step in designing an experiment.  This is best done by using one of several computer programs (Design Expert by Statease being a good one) to estimate the power of resolution and the probability of achieving statistically valid results. 

The process for estimating resolution power is quite straightforward.  A trial run is designed based on the factors and results of interest.  We then estimate the measurement precision and amount of change that we want to be able to “see.”  We can input our trial design and these estimates into Design Expert (or similar software) to estimate the probability of achieving statistically valid results.  If the power is below a certain threshold (often 80%), the experiment is redesigned.  Failing to do this crucial review unnecessarily dooms many experiments and trials.

And finally, the trial should be evaluated for the level of redundancy that is built in.  No trial ever goes as planned.  It is often the case that key observations are lost or compromised.  Sometimes desired operating conditions cannot be met for a variety of reasons.  Every trial should have some redundant data to protect against invalidation of the entire trial by lack of sufficient data.  The chances of this happening should be discussed in detail and an appropriate amount of redundancy should be built in. 

To illustrate, let us suppose that in our experiment, measuring moisture content in the carbon monoxide stream is difficult.  We measure it in the lab after the fact and sometimes the lab data turn out to be unreliable.  Hence, occasionally we have data where we know the acetic acid yield, but we really don’t know the moisture content and have to throw out those data points.  Maybe that happens 1 in 10 runs.  If we estimate that it takes 10 runs at various moisture levels to get enough data to “see” the effect of moisture on yield, we might want to include 1 or 2 more runs to protect us from that occasion failure to get valid moisture data.  If we don’t, we might find that we have to do another trial of 10 runs to get the data needed. 

It is never possible to “guarantee” that an experiment or trial won’t fail.  That is the nature of doing research.  Nevertheless, by applying these simple techniques the number of failures can be reduced dramatically.

From Whence Does Uncertainty Come?

I am approaching this topic with a certain amount of trepidation.  I have been warned against talking about religion or politics and I hope to continue to avoid those topics for the most part.  Nevertheless, it is almost impossible to avoid philosophical issues when talking about uncertainty, randomness, statistics, knowledge and etc.  I had intended to post another discussion on how to reduce the probability of experimental failure.  I will do that later.  It dawned on me that I hadn’t talked about randomness and uncertainty in a general way.  Hence, I am going to venture into the philosophy of knowing for just a bit to help explain my position on things unknown and unknowable. 

Those of you who know me might ask, “How can someone who claims to be a devout Christian and associates himself with Fundamentalists and Calvinists deal with statistics?”  Those of you of a scientific bent will probably find this just an odd curiosity – perhaps even mildly amusing.  Those of you who are of a Fundamental or Calvinistic bent will find this a serious question.  

To sharpen the focus, let me quote to you from a Manifesto by the Trinity Association.  This is a Fundamentalist think tank that states in very stark terms a quintessential Fundamentalist view of knowledge and knowing.  It does not represent all Christians on this topic, but it does give insight into why many Fundamentalists consider this topic to be one of immense importance.  Early in the document the Trinity Association summarizes their view by stating (the italics are theirs):

The fundamental crisis of the twentieth century is neither political, nor social, nor economic. It is intellectual, and the primary intellectual problem is neither metaphysical nor ethical: It is epistemological…The Christian in the twentieth century is confronted with an overwhelming cultural consensus-sometimes stated explicitly, but most often implicitly: Man does not and cannot know anything truly…Unless knowledge is possible, Christianity is nonsensical, for it claims to be knowledge…If man can know nothing truly, man can truly know nothing.

There is much more in this Manifesto and I don’t want to short change them.  I suggest that you read it completely at http://www.trinityfoundation.org.  I have to admit that I am somewhat sympathetic to their position and in many ways wish it were true.  Nevertheless, I cannot agree with it.  The experimental evidence – in fact, the evidence from every experiment ever performed – is overwhelmingly on the side of the modern view that man cannot know anything with certainty.  But does that really mean that “man can truly know nothing?”  Does doubt make knowledge impossible?  I say, “No.”  I believe that admitting uncertainty is a crucial first step in knowing anything correctly.  It is exactly in trying to quantify and reduce our level of uncertainty that gives anything like clear direction on how to proceed toward a better – but always imperfect – understanding of anything.

This is where statistics and logical Design of Experiments (DoE) can help us with our learning.  By admitting that there is randomness and uncertainty and dealing with them honestly, we can more quickly and efficiently root out bias, superstition, fraud and charlatanism.  We cannot only learn approximately how things work, but we can begin to get a grip on just how good our approximations are and, very importantly, how much more we still need to learn.

OK…so perhaps this is a practical way to approach laboratory experiments…but what about other things in life?  Does this same clinical view of randomness and uncertainty apply elsewhere?  Why wouldn’t it…at least in some general way?  If we can’t know the exact position of an electron aren’t we doomed to be uncertain about everything else?  Should we expect the sum of all electrons and their individual randomness to somehow aggregate into something perfectly predictable?  Shouldn’t we expect the randomness that we can clearly demonstrate in the laboratory to “infect” the rest of creation?  Can we really believe that we can perfectly describe the way things really are by using language?  And when we try to pass those concepts to other languages separated by time and space can we really expect things to get better?  Can our network of brain cells really conceive with absolute clarity what it really is that stimulates our various nerves?  Is it not the true condition of man – certainly a “fallen” man – to be doomed to see “but through a glass darkly.”  And isn’t claiming otherwise nothing but hubris?

It appears to me that we have been born into a world of chaos.  Just as I don’t see water spontaneously running uphill, I don’t see things spontaneously getting better – they just seem to want to fall apart.  We may not agree on how the world got this way.  Some may say that it is a consequence of the “fall.”  Others may conclude that it is a necessary consequence of “free will.”  Still others may say that it is simply a consequence of creation.  Nevertheless, it seems undeniable that this creation is demonstrably prone to randomness and uncertainty.  It is full of things that happen and defy explanation as to why.  Get over it.

Now, having said all this, let me add, we do not have to be consumed by this concept.  We do not have to be defeated by it nor do we have to deny the reality of it.  Instead we should seek to understand and push back the uncertainty.  We should be ever mindful of the limits of our current understanding, but never despairing in our quest to understand all things more perfectly.  We should approach life and work with unshakeable optimism and determination.  This is the stuff of real faith.  It is a faith that is neither ignorant nor arrogant.  It is an active faith that wars against the chaos.  It is a faith that is certain of nothing except itself.  A faith that is sure that its hope will not be disappointed even though it hopes in what cannot be demonstrated or even fully articulated.  It is what the ancients were commended for and what believes will someday find Perfection.

Maturing with Design of Experiments (DOE) – Section III

INTRODUCTION:

In previous sections we talked about hypothesis testing and the general approach to R&D management.  In this installment we will start introducing the primary tools used in generating and evaluating experimental data.  As noted earlier, if done properly, the Intuitive Period results in strong suspicions about the factors that are important and how those factors are related to results.  In the Intuitive Period we should have also developed methods for measuring the impacts of these factors on results of interest.  And finally, we should have developed some hypotheses about how those factors impact the results of interest in quantitative ways.   We are now ready to apply some deductive reasoning, but first we should discuss where we are going.  That makes the path a bit clearer.

THE STATISTICAL MODEL:

Although often not stated overtly, most statistical analyses are based on least squared, linear regression.  That is to say, using experimental data to develop a linear model for how factors affect results by minimizing the square of the difference between experimental data points and the data points predicted by a linear model.  The least squared approach is very convenient since the mathematics of that model works well with Analysis of Variance (ANOVA).  This is primarily because variance is calculated from the square of residuals from the mean. 

A linear model is nothing more than a linear equation that is a “best fit” for the experimental data.  You should recall that a linear equation is of the form:

 y = mx + b

 where:

y = the result

x = the factor

m = the slope of the line

b = the intercept of the line

The “m” term is often interpreted as the interaction or correlation between the factor and the result.  The “b” is often interpreted as including a random “error” term expressing the difference between the data points predicted by the linear model and the actual data points.  The ratio of the square of b and the sum of the squares of these differences gives a measure of the “goodness of fit.”  When there is more than one factor the equation takes the form:

y = m₁x₁ + m₂x₂ + … + m_nx_n + b

 It can be seen that the linear model immediately begins to encourage statistical inference about the relationships between factors and results.  Positive “m” terms indicate a positive correlation and negative “m” terms indicate the opposite.  Large “m” terms for some factors and small “m” terms for other factors tend to indicate that some factors are more important than others.  Likewise, a large “b” term might indicate significant bias in the experiments.  From ANOVA analysis the error around these terms can give information about the utility of the model.   The ANOVA yields terms that can generally be interpreted as:

1. The R² values indicate the goodness of fit for the model.
2. The Adjusted and Predicted R² values indicate the robustness of the model.
3. The p factor for the F value indicates the significance of various factors.

These interpretations can be powerful, but they can also be delusional.  The model assumes linearity.  In many cases this is far from correct.  It is often case that the relationships between factors and results are known follow non-linear physical laws.  When this is the case, the factors should be transformed (often power or logarithm) before attempting to do linear modeling.

The model also assumes that the factors do not “interact” or if they do it is simply in a linear way.  This is often not the case – especially where reaction rates are important.  It is rare that factors such as temperature, pressure or voltage can be added to other variables in meaningful ways.  Often, however, linear models using the products or ratios of factors can be useful over the range of interest.  One can envision, for example, a linear model where pressure and temperature interact in a gas phase reaction as the product of absolute pressure and the inverse of absolute temperature.  In such a case, of course, one would want to transform these data from the experimental measurements to the absolute values of these factors.  From the above it can be seen that great care needs to be exercised in trying to interpret the results from a linear model.  If much is known about the system then the coefficients and the intercept might be meaningful.  If not, then the model should be viewed as simply empirical curve fitting.

So how does all this work together?  An example here might be useful so that we can be clear about where we are going.

Let us suppose that we believe that increasing pressure in a catalytic reaction will improve selectivity.  That is to say that at higher pressure we get a higher percentage of what we want instead of by-products.  This is our hypothesis.  Let us also suppose that we do well designed experiments to show that temperature (T), pressure (P) and purity (%X) are related according to the equation below:

%X = 0.00053*T – 0.000083*P + 12

And finally, let us assume that our confidence in these results meets our minimum acceptance criteria for reliability.  We would conclude that our hypothesis is wrong.  Instead of improving purity, the data would indicate that increasing pressure actually reduces product purity.  We would not be suggesting that the plant increase pressure without having additional data.  In this example our testing may have taught us that higher temperature increases purity.  In fact, we might have just learned that the anecdotal evidence that lead us to believe that increased pressure might help may have actually been increased temperature that was often associated with increased pressure.  We probably wouldn’t want to leap to that conclusion without reviewing the data carefully and testing that hypothesis independently since we designed our experiment around pressure and not temperature.  In any case, might have “discovered” something useful from our “failed” experiment.

EFFICIENT EXPERIMENTS:

“Efficient” experimental design is based on obtaining “just enough” information to solve a system of linear equations.  Determining what is “just enough” is not a simple task and much of DOE focuses on solving this problem.  The following discussion illustrates the point.

If the factors that cause effects are independent of each other and the effects are truly linear, then n factors require only n +1 independent data points to solve the system of linear equations for the coefficients and the intercept.  Such a miserly data set would give us a single, “best” solution with little ability to estimate the error in the model.  Furthermore, there would be no ability to test the model for linearity or for interaction between factors.  And finally, there would be no margin for mistakes in collecting data.  Nevertheless, it should be noted that n + 1 data points is the minimum to which are being added potentially redundant data points.  Additional data points always come with an associated cost that must not be ignored.  Hence, a useful balance needs to be struck. 

We will deal with this topic in the next installment of DOE.  For now remember that in our experiments we are attempting to generate just enough of the appropriate data to build and evaluate a model that can be used for decision making.  The model must have the appropriate factors and they need to give us the relationships between factors and results with enough reliability to make business decisions. 

 

Maturing with Design of Experiments (DOE) Section II

INTRODUCTION:

In the previous section we talked about the Intuitive Period in a research program.  This is where much is learned, but there is little real knowledge.  There is a transition from the Intuitive Period to the Deductive Period where things can go very badly wrong.  We will try to describe this transition with the hope that your research team and management organization can make this transition successfully.  It is important to note that this is written from the perspective of industrial research where there is a strong profit motive and serious constraints on time and resources.  Often researchers with academic backgrounds have difficulty in succeeding in this kind of environment.  I hope that some of these ideas help them in their work.

MATURING THE DESIGN – DEDUCTIVE PERIOD:

If done properly, the Intuitive Period results in strong suspicions about the factors that are important and how those factors are related to results.  These “relationships” are the basis for establishing hypotheses to be tested as part of a thorough research program.  There should also be a growing sense of the important measurement tools for testing those hypotheses.  In addition, these measurement tools should have been investigated or tested so that their precision and accuracy can be estimated.  And finally, there should emerge an emerging research strategy based on what is suspected about the number and importance of the factors impacting results.  This Intuitive Period gives way to a Deductive Period where hypotheses are stated in measureable terms that lead to statistically based evaluation.  Experiments are designed so that hypotheses can be tested and the relationships between factors and results can be measured and modeled.  In the case of industrial research (i.e. research and development for profit) the goal is to obtain a statistically valid model that can be used to do cost/benefit analysis for business purposes.  For scientific or philanthropic research (i.e. the goal is something other than profit) a model is still necessary, but the evaluation criteria is modified to satisfy these specific goals.

The transition from the Intuitive to the Deductive Periods is usually rather messy and awkward.  Because very little is really “known.”  Much is suspected, but it is difficult to make objective decisions on how to proceed.  Nevertheless, decisions must be made.  It is almost always the case that the Intuitive Period has generated interest in more factors than can be managed.  The quasi-formal testing of the Intuitive Period usually reveals variables that were not anticipated at the start of the project.  In many cases, even the number of pertinent results has grown.  It is often case that a desired result (e.g. yield) turns out to be a combination of two or more independent results (e.g. selectivity and activity).  Due to the nature of time and budget constraints in industrial research, it is rarely the case that more than a handful (4 to 6) factors and half as many result (2 or 3) can be managed simultaneously.  The number of independent analytical tests grows with more factors and results.  The number (dozens to hundreds) becomes too large for management to handle in the annual and often quarterly budgetary cycle.  Even in academic and philanthropic circles, the time and resources required to manage programs requiring hundreds of independent trial is rarely available.  Hence, it is often the case that a strategic decision becomes necessary.  The research team must reduce the number of factors and results or the number of independent tests.

The choice must be based largely on the confidence the research team has in the relative importance of the individual factors and results.  Where there is good reason to believe that some factors and results have minor relative impact on organizational goals they can be “shelved” until a later date.  The research team takes this decision knowing it risks overlooking important factors or results that can confound later decisions.  When the risk or consequences of this kind of error is too high, the research team can opt for a “screening” strategy where a simplified testing program (i.e. fewer independent tests) can elucidate the factors with the greatest impact on results.  Often the number of factors may be reduced dramatically once there is good evidence that they have little impact on pertinent results.  This can be a very useful strategy that can refocus the research efforts on to the key factors for which definitive information is needed.  In any case, realistic expectations for the research program can be set for the organization.

To this point we have been talking about factors and results as if they were independent objects that we wish to describe.  That is hardly the case.  Factors are important only in the way they impact results.  It is the relationship(s) between factors and results that are of interest.  These relationships are stated in the form of hypotheses.  These hypotheses must now be stated in quantifiable terms and tested for relevance in producing measurable results.  An example hypothesis might be:

H₀:  Increasing the temperature of a chemical reaction from 25C to 35C increases chemical yield.

In this case the factor is temperature and the result is yield.  It is important to evaluate this hypothesis and make certain that it makes no assumptions fallacious assumption about factors or results that are not overtly stated.  In the example above we are implicitly assuming that factors such as concentration and pressure can be held constant throughout the experiment or that they are, for this experiment, of no interest.  Every hypothesis will have a number of implicit assumptions.  Identifying and noting them could point out serious weaknesses in the hypothesis and the experimental design.

The hypothesis we are testing is called the null hypothesis (see Note 1).  It presumes that there is an unbiased way to measure both temperature and yield.  It is also usually assumed (though not required) that the error in measurement is low when compared to the variability of the effect.  

The negation of the null hypothesis is the alternative hypothesis symbolized by H₁.  In our example it could be stated as:

H₁:  Increasing the temperature of a chemical reaction from 25C to 35C does not increase chemical yield.

Either H₀ is true or H₁ is true, but both cannot be true.  Unfortunately, neither condition can be known with absolute certainty.  Hypotheses are tested with this understanding and acceptance or rejection of a hypothesis is made on probabilities rather than certainty.  Setting these acceptance and rejection criteria are a crucial step in the evaluation of hypotheses that should not be taken without considerable evaluation.  This is best illustrated by an example.

Suppose we test chemical reaction yield at 25C and get 45%.  Then we retest the chemical reaction yield at 35C and get 50%.  One question now is, “Is 50% result really greater than 45% or is that just random error?”  A follow-up question is, “How sure do we want to be that result of 50% is really greater than the result 45%?”  If we measure the yield at 25C and 35Cs many times and the answers are always very close to 45% and 50% respectively we have high confidence that the higher temperature will give us more yield.  If, on the other hand, the yields vary widely our confidence shrinks.  In most cases the null hypothesis is stated with a desirable outcome.  Hence, we are loath to reject it without good reason.  Nevertheless, at some point we must say that the risk is too high that temperature really has no impact yield as much as we would like it to be so.?

We resolve the situation by setting a statistical test that reflects our view of the risk.  In hypothesis testing parlance the situation can be mapped as shown in Table I:

TABLE I 

Hypothesis	Reality	Conclusion from Testing	Conclusion from Testing
H0:  Increasing Temp 10C Increases Yield	True	True (OK)	False (Wrong) Type I Error
H0:  Increasing Temp 10C Increases Yield	False	True (Wrong) Type II Error	False (OK)

Since we really want to increase yield, we don’t want to reject H₀ unless we are pretty sure that it is not true.  Hence, the acceptable probability for a Type I error is often set at 5% or 10%.  Let’s see how that would work in our example. 

Let’s suppose that our statistical analysis of the 25C and 35C data (that is 45% yield and 50% yield respectively) would happen by accident 8% of the time.  If we set our Type I rejection criteria at 5% we would reject the hypothesis and conclude that there was too big of a chance that this was just random.  We might do that if the consequences of raising the temperature without getting the yield were expensive.  Perhaps fuel costs are very high compared to the value of the product.  If we had set our Type I rejection criteria at 10% we would accept the hypothesis and turn up the heat.  Perhaps the chance of getting a little bit more product is worth risking a bit more fuel. 

It should be noted, however, as we relax the acceptance criteria for H₀, we reduce the chance of rejecting H₀ when it is actually true, but we increase our chances of accepting H_o as true when it is actually false.  That is to say, the probability of a Type II error increases and, in our case, we have a higher chance of just wasting fuel.  The consequences of the two types of errors could be quite different.  Suppose for example, we were testing the new method for reducing pollution at a plant.  We might wonder if the new method really was reducing the pollutant, but the potential for a stiff fine might make us more willing to try something that was much less certain.  In that case, it may be useful or even necessary to evaluate H₀ and H₁ independently and adjust business decisions accordingly.  

This Deductive Period is marked by careful evaluation of hypotheses to be tested and designing experiments based on the numbers of factors and results.  The number of independent tests (trials) can be established using DOE software such as Design Expert.  In many cases (especially when trying to optimize conditions or mixtures) it requires reasonably good understanding of the precision of testing protocols as well as the desired precision of prediction of results.  This often is an iterative process.  Rarely will the first set of trials give truly unambiguous results.  It should also be rare that the first set of trials gives completely ambiguous results.  That could indicate poor thinking and design.  Occasionally something really novel (“good” or “bad”) is discovered and the experimental team will have to go back to the Intuitive Period and learn more about the process in general. 

In the next installment we will discuss statistical modeling and how that plays a role in industrial research.

  

Note 1:  Sadly there is much inconsistency in the literature about what “null” means in this case and if H_o must be stated in the negative.  Likewise, some statistical purists insist on strict definitions of Type I and Type II errors.  Sometimes following these conventions is effective, but often it results in convolutions that confuse.  The key is to get a general understanding of setting useful and important hypotheses and dealing with the risks of the statistical testing leading you astray in two possible ways that are illustrated in the text.

Getting Started with Design of Experiments (DOE) Section I

INTRODUCTION:

Sampling and analysis should be guided by solid planning that is based on strategically relevant hypotheses that can be tested.  Failure to correlate sampling and analyses to relevant hypotheses that are amenable to testing usually results in ambiguous results that have little value.  Whereas it is generally true that design should precede experimentation, there exists an exceptional period during the earliest phases of experimentation where so little is known that it is almost impossible to distinguish relevant hypotheses from irrelevant ones.  This period is often accompanied by a lack of familiarity with the analytical tools available including their value and limitations.  Nevertheless, no serious progress can be made toward statistically valid understanding until experiments are designed with purpose and sampling and analysis plans are made an integral part of those designs.

EARLY PERIOD – INTUITIVE PEROID:

Early in an experimental process there is little or no understanding of the causal relationships that may be at play.  The old adage, “we don’t know what we don’t know” is sadly true.  Detailed design of experiments based on all possible causal relationships would be hopelessly complex and expensive.  Furthermore, good experimental design requires reasonable estimates of testing precision and accuracy if the “predictive power” of resulting models is to have any meaning.  Hence, the pertinent measurement tools need to not only be identified but their precision and accuracy need to be assessed. 

 A certain amount of semi-quantitative “groping” is both natural and useful.  Sam Kash Kachigan (Ref 1) calls this “fishing” – and that is an appropriate description.  I prefer to call it an “Intuitive Period” – somewhat in honor of Blaise Pascal.  This early, Intuitive Period should be marked by literature searches for probable causal relationships, discussion and debate, setting up and testing of analytical processes and simplified testing of proposed causal relationships – generally at “normal” and “extreme” levels to demonstrate “gross” effects.  It is a mistake to cut off such intuitive processes too soon.  It is a grave error to overlook a critical causal factor.  The value of all subsequent research will be diminished if not extinguished. 

It is also a mistake to believe that there is little purpose, value or method to the Intuitive Period.  The purpose is to identify relevant factors (input variables), results (output variables), hypotheses and useful measurement tools.  It is crucial to identify the key factors and dismiss those that are not important.  The amount of testing necessary to evaluate many factors and results grows geometrically.  An 8 factor/2 level/2 result DOE can require nearly 400 independent test runs.  When those tests are complex and/or expensive, the cost of the experimentation can be prohibitive.  A key outcome of the Intuitive Period is to learn what factors are important and what levels for these factors a relevant.  Activities that fail to contribute to these purposes are not useful and should be avoided.  Any finally, it is a grave error to stay in this period too long or to fail to progress to serious testing of the causal relationships “discovered” at this lower level of rigor.  The Intuitive Period results in no knowledge – only beliefs which are too often more a reflection of prejudices than anything akin to serious hypotheses.  Risking significant owned resources on outcomes based on such “information” is very imprudent.  Risking significant resources held in trust for others based on such “information” is unethical and, depending on the motives, could be fraudulent.

One last note concerning causality needs to be added.  In most cases what is observed is co-variance.  That is to say, factors (inputs) and results (outputs) are observed to vary together in some uniform way.  That is what is meant by co-variance here.  Causality is often inferred from these observations, but demonstrating causality is difficult.  Evidence of causality can be provided when it is possible to manipulate factors and observe uniform variance of results as in the case of experimental design.  This is not possible when observing historical information where factors and results are as they are found and are not under the control of the experimenter.  Hence, concluding causality from historical data, survey data and the like is rarely justified.  If the determination of causality is a goal, then factors should be under the control of the experimenter.

The next Section will begin with the Deductive Period and discuss formal Hypothesis Testing.

 

REFERENCES

1. Kachigan, Sam Kash, Multivariate Statistical Analysis, Second Edition, Radius Press, New York, 1991.

Why "Design" Your Experiments?

"Designing" an experiment forces the experimenter to think about hypotheses, factors (variables), outcomes (results) and error terms (uncertainty) in a systematic way.  Even when much is yet unknown the exercise is fruitful for focusing attention on key elements and encouraging objective consensus on important technical goals.  Furthermore, even preliminary designs help both experimenters and management begin the process of "managing" research rather than just "letting it happen."