Technology Development Network

Coming May 2022 New Book on Technology Development

    Technology Development

    Lessons from Industrial Chemistry

    and Process Design

 

    by Ron Stites

 

 

 

Ron has recently completed the manuscript for a new book on Technology Development under contract with Walter de Gruyter GmbH. De Gruyter is a German scholarly publishing house specializing in academic literature. The new book should be available May of 2022. The Table of Contents below gives you an idea of the topics covered:

    Chapter 1: Introduction

    Chapter 2: What is Technology and Technology Development?

        Some Common Misconceptions

        Types/Classes of Technology

        B2B Markets

        Consumer Markets

        Government Markets

        Social/Common Good

        Impact on Existing Markets

        The Benefits of Technology Development

    Chapter 3: Some Basics Axioms of Technology Development

        Axiom 1 – Nothing is Completely Predictable

        Axiom 2 – Collaboration can be a Four-Lettered Word

        Axiom 3 – Money is the Final Unit of Measure

        Axiom 4 – Everything is Urgent

        Axiom 5 – Size Matters

        Axiom 6 – The Quest for Knowledge Requires Intuition and Deduction

    Chapter 4: The Science of Technology Development

        Markets First

        Investors/Stakeholders Second

        Go/No-Go Decision Process Third

        Finally, The Technology Development Process

        Planning to Learn

        Designing Your Experiments

            The Intuitive Phase

            The Deductive Phase

            Making Sense of Uncertainty

            Simple Statistics – Population, Sample, Average, and Standard Deviation

            Guarding Against Major Blunders

        Execute Your Plan

        Evaluation and Reporting

        Improve Your Experiments or Wrap Up Your Conclusions

        Valuation Methodology (Present Value/IRR)

        Evidence-Based Decision-Making and Data Science

    Chapter 5: The Art of Technology Development

        The Art of Human Relationships

            With Employees

            With Funders and Potential Funders

            With Customers and Potential Customers

            With Stakeholders

        The Art of Matching Technology with Markets

        The Art of Monetizing the Technology

    Chapter 6: Funding and The Discipline of Technology Development

        How Discipline Impacts Funding and the Growth of Wealth

        The Biz Plan

        Funding in Large Organizations

        Funding in Small Organizations

        Technology Development Startups – A Peculiar Subset of Small Business

        The Funding Process

            Dispelling Some Myths about Raising Capital

            Assessing Capital Needs

            Set Capital Goals

            Detail Capital Goals and Strategy

            Identify Potential Capital Sources

            Guiding Principles for Contacting Potential Investors

            Structure Capital Offering(s)

            Manage the Capital Campaign

    Chapter 7: The Future of Technology Development

        Critical Technology Development Needs

        Capital Sourcing Challenges

        Evaluation of Technology

        Use of Capital

        Scarce Supply of Capital for Small Business

        Accountability

        Patience

    Chapter 8:  Where Do We Go from Here?

        Reducing the influence of academics and Higher Education

        Strengthening review of government Technology Development programs

        Specifically requiring statistically sound review

        Encouraging small, primary investments directly in small businesses

        Graduated Capital Gains Tax

        Capital Gains Tax on Non-profits

        Phasing out Non-profit Status for all Non-profits

        Phasing in Strict Financial Reporting for all Non-profits

        Reforming all of Education to make it eventually Competitive, For-profit

        Requiring Grade-appropriate Probability and Statistics Curricula

        Requiring Appropriate Field Proficiency Probability and Statistics STEM and Professions

        Deregulating Crowdfunding for Small, Technology Development Companies

    Chapter 9: Some Final Words

If you would like information on how to get a copy, contact us at Ron Stites at [email protected]

Turning Data into Knowledge

Not long ago a good friend of mine looked at my website and said, “Ron, you need a byline.”  That is to say, a catchy phrase that describes what I do.  Well, I’m hardly an expert on catchy phrases.  Most folks find my writing painfully convoluted – bordering on arcane (probably on the other side of the border).  As I thought about that, I tried to think of a simple phrase that described what I did.  I think it boils down to:

Turning Data into Knowledge

Naturally I feel compelled to explain this a bit.  I spent a career in the laboratory business.  I’ve done a lot of environmental analysis, but I’ve also done work in oil and gas, industrial development and even some forensic work.  It has seemed to me for a long time that we tend to rummage around with lots of random data points that frequently don’t make any sense.  That hasn’t kept folks from trying to make sense of their data, but often they see what they want to see.  It’s kind of like Dr. John Nash in the movie, “A Beautiful Mind”:

 

 

In the clip we see Dr. Nash (Russell Crowe) illustrating the mind’s ability to see order in chaos.  In fact, it is not at all unusual for our minds to infer order from chaos or causality from random events.  That is how our minds work and, though we are sometimes deceived, quite frequently, it is this ability that actually brings us to knowledge.

It is not enough to catalog random data.  If we are to have any useful knowledge, we must get at underlying causes in order to obtain any reasonable ability to predict future events.  This is the fundamental basis of useful knowledge.  If, for example, we could not depend on gasoline of certain specifications to work in predictable ways to power an automobile, knowing the composition and properties of the gasoline would be completely useless.  If causality were completely unhinged, all that could be known is that which is defined.  We might, for example, be able to define gasoline by its specifications, but that would be a completely useless bit of trivia of no greater consequence than defining a triangle as a polygon with three sides.

Philosophers have dealt with this ability to know in a very general way.  This study is known as epistemology – the theory of knowledge.  There are many subtleties from a philosophical perspective, but there are some issues of real practical importance that arise from these rather esoteric discussions.  The writings of Emmanuel Kant are especially interesting in that we learn that random data points are not real knowledge unless the relationships between these data points can also be understood.  That is to say, we understand how order can be seen in what first appears to be random.  And here, of course, is where things can get fuzzy.  Our mind tends to categorize and relate data points, naturally looking for recurring patterns and assigning order and causality.  In this we are both blessed and cursed.  We intuit order and causality, but is it really there?

We may observe that A and B always seem to appear together.  Our experience may be that A tends to appear before B and our minds begin to associate causality of B to A.  This may be true, but there are a lot of things that could be wrong.  Our perceptions could be wrong.  A may be perceived by us before B, but B might have actually existed before A without us knowing it.  In that case B might actually be causing A.  And, of course, both A and B may be caused by something totally beyond our perception and neither actually causes the other.

This whole process of knowing gets further complicated by inexact measurements, interactions between factors and the cost of knowing with greater confidence.  Inexact measurements can not only obscure relationships between factors, it can bias results such that apparent relationships are completely without objective reality.  Experiments must begin with proper analytical methods so that artifacts of measurements are eliminated from consideration.  Experiments must be designed so that important factors are included and we don’t waste time and money on irrelevant minutia.  Even the potential for two or more factors to work in concert to affect results must be considered.  And all the while the cost of knowing more must be balanced against the cost of knowing too little.

This is what I do.  I do look at the analytical challenges and devise better, more relevant and cost effective ways to get the right data.  I am especially keen on using GC/MS for organic analysis since it provides two independent methods for identifying compounds and avoiding much analytical confounding.  I am also very keen on using standard reference methods that have been proven time and time again.  Too often we leap to a new, slick, high tech method only to find that there was some interference or subtlety that we overlooked.

I also look at the possible relationships between factors and devise trials to test potential interactions in objective ways.  In this I use my years of experience in engineering settings to think through possibilities.  And I do look at the most cost effective ways to get the knowledge needed with the confidence required without chasing too many irrelevant issues.  I model these ideas in statistically based Design of Experiment software searching for optimal experimental designs.  This is both art and science.  It is an application of good Analytical Chemistry and shrewd Accounting and Finance to take statistically valid data and transform it into useful business knowledge.

The process always begins with the end in mind.  That end must be defined in quantitative terms that include both precision and accuracy terms, but also a sense of the monetary value of the knowledge.  It does no good to come up with a number and have no idea of its reliability.  Furthermore, it is unwise to begin any task without some idea what the cost will be.  Armed with basic ideas of the purposes of an investigation, the methods and experiments designed are much more likely to meet objectives.  Historical data can be used to intuit causal relationships and then these can be systematically tested to verify the reasonableness of our thinking.  It is only after planned experimentation that we have any hope of attaining something close to knowledge.  This is how data is systematically turned into knowledge.

One final comment should be made.  Outside of logical definitions (see “Analytic-Synthetic Distinction”), there is no such thing as knowing anything with perfect certainty.  In the simplest of circumstances we may come close to certainty – at least for all practical purposes.  Nevertheless, with complexity comes greater uncertainty.  We may, for example, be quite convinced that our experimental yield for a particular reaction will be between 85% and 90% at the lab bench.  Nevertheless, how much money we will make in plant running that process will be far from certain.  The key is understanding all the many factors and being able to make reasonable estimates with enough reliability to make shrewd decisions. 

This is what I mean when I say, “Turning Data into Knowledge.”  It is more than just careful analysis.  It includes a careful evaluation of the relationships between data from measurements and desired outcomes.

Ron Stites holds a BS in Chemistry and an MBA in Finance and Accounting.  Stites & Associates, LLC, is a group of technical professionals who work with clients to improve laboratory performance and evaluate and improve technology by applying good management judgment based on objective evidence and sound scientific thinking.  For more information see: www.tek-dev.net.

Monte Carlo Simulations (With Video Tutorial)

 

We often need to know how randomness in several variables interact to create randomness in a derived result. This can occur when we are trying to project complex outcomes from multiple factors or estimate the error in a measurement that is the result of several, independent measurements. An example of the first type might be trying to predict future profitability from estimated future costs and incomes. An example of the second type might be trying to estimate chemical yield when we have uncertainty in the measurements of both raw materials and finished products.

These are two typical examples of what is known as “propagation of error.” Sometimes the most practical approach is to resort to a Monte Carlo Simulation to estimate the “propagated error.”

Just about a year ago we gave a general method for estimating the propagated error in a mathematical model. We began with simple examples such as the difference between two measured or estimated values. One example might be the estimation of moisture content from a simple drying experiment. The moisture content might simply be the difference in mass of the sample before and after drying. This is the difference of two measured numbers. For sums and differences we showed that the variance of the total measurement is simply the sum of the variances of the individual measurements.

We then proceeded to more complex calculations and finally to a general statement. We stated the final conclusion in terms of a function of random variables. We repeat the conclusions below:

Let x, y, z,... represent random variables whose true values are X, Y, Z.... Let u represent a derived quantity whose true value is given by:

U = f(X, Y, Z,…)   Equation 1

Let ε1, ε2, ε3,... represent the statistically independent and relatively small errors of x, y, z, ... respectively. Then the error induced in u, which is denoted as ξ, as a result of the errors ε1, ε2, ε3,..., has a variance equal to:

V(ξ) = (δf/δX)²V(ξ₁) + (δf/δY)²V(ξ₂) +  (δf/δZ)²V(ξ₃) + …            Equation 2

             Where (δf/δX) is the partial derivative of f with respect to X.

You may want to review the full Blog by visiting:

http://tek-dev.typepad.com/technology-development/2015/02/estimating-total-process-error.html .

It was noted that for this approach to be valid the following conditions needed to be met:

1. The mathematical model must be known,
2. The variance of each factor must be known,
3. The model equation must be differentiable, and
4. The variance of each factor must be “small”

    The first two conditions are not trivial, but they often can be met. The third condition occasionally cannot be met with non-linear functions (logarithmic, inverse, or power functions). The fourth condition is frequently difficult to meet – especially with Binomial and Poisson distributed factors. When this occurs a Monte Carlo Simulation may give the best results.

A Monte Carlo Simulation consists of basically three parts:

1. Construct a mathematical model that expresses the interaction of all the variables of interest,
2. Measure or estimate the statistical distribution of the variables of interest,
3. Create independent tables of the variables of interest that represent a “fair” distribution of possible inputs and
4. Create a model for the results by a populating the model randomly selected possible inputs.

This is best illustrated by an example. Let us suppose that we have come up with a very spiffy model that predicts the recovery of a liquid product from a gas stream by a gas impingement system. Let us suppose that the recovery depends on the square of the particle velocity and the velocity of the gas stream. To keep it simple let us assume that the yield in useful units to us can be given by:

Yield (kg/hr) = X² + Y          Equation 3

    Where:
        X = Velocity of the particle (meters/sec)
        Y = Velocity of the gas (meters/sec)

Now let us suppose that we cannot control the liquid particle and gas flows perfectly and we are wondering about how much our yield would vary. Of course we could just build our plant and try things out, but this could be a pretty expensive way to go at the problem. Suppose we are told by a manufacturer of various nozzles and systems that we could expect a linear velocity of the particles to be 5 meters/sec with a variance of 0.25 meters/sec and a gas velocity of 8 meters/sec with a variance of 0.25 meters/sec. What would we expect the variability of the Yield to be? Let us do this using a Monte Carlo Simulation.
This particular example more or less meets the criteria for a propagation of error calculation. Hence, we can use Equations 1 and 2. When we do the partial derivatives and apply them to Equation 3 above we get:

Variance of Yield = (4X²) x Variance X + Variance Y Equation 4

Thus the Yield is estimated to be 33 kg/hr with a variance of 25.25 kg/hr (of course we would round to no more than 2 digits).

Now let us show how to solve this problem using a Monte Carlo simulation. Let us first create a table of values for X and Y that represent a number of “fair” values distributed like our manufacturer said. This is actually easily done in Excel using a Data Analysis Add-In. Rather than try to explain how this is done, the reader is encouraged to view the short YouTube video posted at https://www.youtube.com/watch?v=PXDRTl8_fVM or click on the graphic below:

 

 

If you view the YouTube video you will see that we created 300 random numbers (X’s) with a mean of 5 and standard deviation of 0.5, 300 random numbers (Y’s) with a mean of 8 and a standard deviation of 0.5 and then we created 300 results according to the equation X2 + Y. Then we calculated the mean, standard deviation and variance of the 300 results. We got:

    Mean = 33.38697
    STD =  4.957101
    VAR  = 24.57285

You can see that these results are very close to our propagation of error calculation. This makes sense, of course, because the criteria for the propagation of error calculation are met. If, however, these criteria had not been met, the Monte Carlo Simulation would have given us much more useful results.

The Monte Carlo Simulation methodology has a number of advantages over a simple propagation of error estimation. It can be especially helpful when the variables have much different behavior. This happens frequently when the variables are a mix of continuous variable which are usually Normally distributed, and discrete variables which are often Binomial or Poisson distributed. The Monte Carlo Simulation can be used with any variable where a pile of representative results can be tabulated and then used randomly to form multiple result calculations. Hence, it can be used when a random variable fits no particular statistical model at all. And finally, a Monte Carlo Simulation can give indications of skewness that is not apparent in a simple propagation of error estimation.

In our current example, the non-linear X term introduces skewness that would not be evident in a simple propagation of error estimation. If we again use an Excel Data Analysis Add In (this time the Histogram) we can plot a frequency chart that shows us that the distribution of results is not symmetric. This is shown below:

One word of caution is needed here. It can take a surprisingly large number of data points for a Monte Carlo Simulation to give good results – especially for the variance. Just remember from the Central Limit Theorem that the mean of any data set derived from normally distributed variables quickly approaches the same mean as that of a very large set. This is not true with variance. Even for normally distributed variables the number of simulations can be large before a stable variance is achieved. When the variables have a mixed distributions the numbers grow larger and less predictable. It is best to run the simulation several times with what “seems” should be a large number of data points and compare the resultant means and variances. If they seem stable then you probably have enough data points. If not, you should expand the number of points by 2X and try again. You should repeat the process until variability in outcomes meet your data quality objectives.

In conclusion, Monte Carlo Simulations can:

1. Be very helpful in estimating the error and distribution of error for derived calculations from experimental data or predicting the error and distribution of error around projections,
2. They can be easily set up in Excel based on your knowledge of input variables and their relationships,
3. They can be applied to a wide range of distributions and relationships and
4. They can give additional information about the resulting distribution curve that might not otherwise be apparent.

 Stites & Associates, LLC, is a group of technical professionals who work with clients to improve laboratory performance and evaluate and improve technology by applying good management judgment based on objective evidence and sound scientific thinking.  For more information see: www.tek-dev.net.

 

 

 

 

Ron Stites Publishes Article in Chemical Processing Magazine

An article on using Design of Experiments in Pilot Plants by Ron Stites appears in the November, 2015, issue of Chemical Processing.  The article describes the general approach used in Pilot Plants and gives several examples from industry.  The article gives several "how to" tips and warnings about what can go wrong.

The article is now posted on the web.  For the full article click Here.

 

 

Stites & Associates, LLC, is a group of technical professionals who work with clients to improve laboratory performance and evaluate and improve technology by applying good management judgment based on objective evidence and sound scientific thinking.  For more information see: www.tek-dev.net.

Estimating Total Process Error

One of the most important steps in designing an experiment is setting what are called “Data Quality Objectives” or “DQO’s.”  This is a fancy term for figuring what we need to know and why.  It goes beyond well simple generalizations.  It has to reach to what data has to be known, and how precisely must it be known to meet our business objectives.  Without having clear DQO’s it really isn’t practical to try to design an experiment.

One of the maddening parts of trying to design an efficient experiment is that we are forced to make many decisions when we really don’t know very much.  We are in a quandary as to where to start.  One approach is to start from your best guess at the end result and work backwards.  That is to say, determining what we are trying to accomplish and representing that as a calculation.  If we can say what we are trying to do as a mathematical equation, the variables of interest become obvious.

Let me illustrate with an example.  Let’s suppose that we want to save money on a chemical process.  We think that by substituting a less expensive raw material we might cut our overall costs.  Before we go very far into running any lab or pilot tests, we need to define clearly what we mean by “saving money.”  We probably can’t test “saving money” directly in our lab or our pilot plant and no one in their right mind would turn an operating plant to a bunch of geeks to play with.  Hence, we would have to gather pertinent data from lab or pilot scale work and use that to predict what the impact would be on total costs at a plant level.

In order to make any real progress, we would need to be able to put all the results of our testing into some kind of mathematical formula that expresses the relationship between potential raw material cost savings and other (perhaps negative) results that we might have found in our testing.  If, for example, we found that changing raw material reduced yield, we would want know how much.  Likewise, if we found that the change increased the probability of having serious plant failures, we would need to have some way to factor that into our assessment.  Of course we would want to know how all these factors worked together on the average, but we would also want to know the variability of the result would be impacted by the variability of individual factors and what their combined effects would likely be.  If changing to a new raw material ruined the reliability of our plant we would certainly want to know that.

If we can write down a mathematical equation to calculate our final result, we actually have a “model” for our process.  Some might complain that we really cannot do that until after we have done our experimentation.  This is fallacious thinking.  How factors impact results is always a necessary hypothesis.  Failure to state that overtly or at least make provisional hypotheses, leaves the experimental design wide open.  I would point out that all continuous functions approach linearity over small increments.  Hence, without evidence to the contrary, it is always reasonable to start with a linear model.

Once we have an overall working model we can then begin to design an overall strategy for testing hypotheses.  Furthermore, using techniques described later, we can assess the variability and uncertainty introduced by variability in things like the variability of raw product cost, yield, quality, down-time, etc.  And finally we can assess the uncertainty created in our final result created by uncertainty in our measurements.

It is rare that all the pertinent factors are put into one giant test program.  Development and the accompanying experimental designs usually progress in a more compartmentalized fashion.  That is just nature of experimentation – especially in its early phases.  The findings of individual groups are brought together and evaluated in the overall model.  It is only later in the process that many factors are tested at the same time to ensure that factor interactions are understood. 

The evaluation of how the uncertainty and variability of individual factors impacts a final, derived result is known as “propagation of error.”  This is a mathematical technique applied to the generalized (and subsidiary) model(s).  We will hold off on the mathematical details for a while, and just try to get the general idea of how this technique works and why it is important.  The illustrations used here will be kept quite simple.  At the end we will point to how this simple approach can be applied to very complex situations.

Let’s suppose that we have found a way to predict the average cost of a new raw material (C_M) and the average manufacturing and distribution costs (C_P) that would arise from using the new raw material.  Our predicted total average costs (C_T) would be:

We would, of course, insist that this new C_T would be lower than our current C_T.  It is nice to know the average or expected value of C_T, but we also need to know the uncertainty around this new C_T.  We will go more into the math later, but it turns out that in this type of equation the variance of C_T is equal to the sum of the variances of C_M and C_T. That is:

We now have a way to assess the uncertainty of this new cost structure if we can measure or predict the uncertainty around C_M and C_P.  We might see, for example, a slight improvement over our existing cost structure, but we might also see a lot of uncertainty around that.  We might hesitate to implement such a change, especially if we had a lot of confidence that our current cost structure was very predictable and profitable. 

Now Equation 2 obviously roles up a lot of experimental data into just a few simple variables.  In actual practice the experimenter is working at a much less a grandiose scale.  A more realistic case for an experimenter might be trying to assess the uncertainty of chemical yield.  Of course the experimenter might run many trials in order to measure the variability of the yield.  Nevertheless, there are many occasions where a “propagation of error” calculations can be very useful both before and after experimentation. 

We might want to do an a priori calculation to help us evaluate which factors are likely to be the most important to the precision of future measurements.  Some errors are multiplied greatly by how they are used in the final calculation.  A propagation of error calculation would alert us to the sensitivity of the measurements.  This kind of calculation might reveal that, under current conditions, we can’t possibly meet the precision requirements of our project.  We might need to redesign some of our analytical equipment before wasting money on hopeless trials.

We might also use this type of calculation a posteriori when some of the factors of interest in the overall model were not or could not be included in our experiments.  It is sometimes the case that important factors cannot be measured in the lab or in a pilot plant.  This often occurs with scaling and market issues.  Here we might have to use “expert” estimates (often called “SWAGS”).  It is also possible that some factors are actually known well enough that they don’t have to be verified by our experimentation.  The work may have already been done by vendors or industry groups.  A propagation of error analysis can bring together very disparate information and give some objectivity in what could be a very complex analysis. 

Let’s look at an example more typical of what we would see in a pilot plant.  Suppose we want to measure the yield in our pilot plant.  Yield is often a result that we derive from several independent measurements such as:

• The mass of mixed product produced (P)
• A compositional analysis of the mixed product (wt% A)
• A mass of mixed raw materials (R)
• A compositional analysis of the mixed raw materials (wt% B)

Let us also suppose that the key yield indicator is mass of A in the product per mass of B fed in the raw material.  Our yield formula (as a fraction) would be:

But we don’t weight A or B directly.  We calculate it from:

Now suppose we would like to estimate the error in the yield before doing any experiments for some of the reasons stated above.  It turns out that this type of error propagation is also fairly simple.  When formulas are simple ratios, the square of the coefficients of variation are additive.  The coefficient of variation (CV) is the standard deviation divided by the mean.  Hence:

Formulas like Equation 5 can be very useful for helping us design and interpret experiments.  If we know, for example, that we will have a hard time measuring the composition of B, we know, before even doing the testing, that the variability/uncertainty around yield will be high.  We might want to get a better way to measure B before we do a bunch of testing.  On the other hand, if we have good data for mass measurements from our vendors of weigh belts, load cells, etc., we might conclude that it would be a waste of time to check and recheck the mass measurement systems.  We could do a simple verification that they are working as designed and focus our attention on the more difficult measurements (such as measuring composition).

As we have said earlier, it is almost always the case that we have only part of the experimental data we need to fully “prove” a complex business proposition.  We may have experimental data for yield.  We might even have some experimental data on failure rates, product quality, etc.  Nevertheless, we will almost always have to make guesses or extrapolations rather than being able to actually measure the bottom line numbers like profitability, total costs, etc.  

Nevertheless, if we can state the result of interest in a mathematical formula based on variables we have measured or can reasonably estimate, then we can apply a general formula for propagation of error.  This may help us predict the variability of our result of interest even when we are short on experimental data.  The general approach to propagation of error is described mathematically in the following paragraphs and equations.¹

Let x, y, z,... represent random variables whose true values are X, Y, Z....  Let u represent a derived quantity whose true value is given by:

Let ε₁, ε₂, ε₃,... represent the statistically independent and relatively small errors² of x, y, z, ... respectively.  Then the error induced in u, which is denoted as ξ, as a result of the errors ε₁, ε₂, ε₃,..., has a variance equal to:

Equation 7 looks daunting, but when the formula for the result of interest is a mix of sums and products, the evaluation usually isn’t too bad.  It requires successively taking the partial derivative of the model equation relative to each variable of interest while treating the other variables as constants.  It also requires reasonable knowledge of the variability/uncertainty of each variable at or near the operating point of the model.³

Although the analysis can become complex, evaluation of the total error of a process is an indispensable technique.  It is essential to have a reasonable and unbiased estimate of the uncertainty of any result that will impact a business proposition.  By its very nature, business turns on the issue of uncertainty.  Businesses can often command a premium for handling uncertainty effectively.  High stakes can be very rewarding.  Nevertheless, those who find themselves with the wrong mix of uncertainty and reward will inevitably fail.     

 

 

 

¹From Mandel, John, The Statistical Analysis of Experimental Data, Dover Publications, Inc., New York, 1964.

²The random variables must be independent and the relative error generally 10%.  With large relative error the approximation does not work well.

³It is very important to not apply this technique for conditions well outside of conditions that have been tested.  It could be quite incorrect to assume that the variability/uncertainty of a random variable remains constant regardless of conditions.  It is quite common for models to fail when applied to untested conditions.

 

Sampling Error and Poisson Processes

 Whenever we are doing analytical work, we usually want to use statistics based on normal distributions.  This applies to not only the actual analysis, but also the sampling protocols.  We always hope that we are in a situation where the value of interest (usually a mean of several analyses) and the relative standard deviation of that value (again from several independent analyses) is small.  What we would like to see is a plot of analytical results that looks something like:

 
Figure 1

Figure 1 was created using Excel’s NORM.INV function set for a mean of 100 and a standard deviation of 5.  The RAND() function was used to create 20 random points.  These were plotted as a Histogram using the Excel Data Analysis Add-In.  When the mean and standard deviation of the “random data” was calculated I obtained 101.7 and 6.15 respectively.  This gave a relative standard deviation of 6.04%.  This would be typical of many analytical sampling and analysis protocols.

Of course, things never seem to work out as well as the “example” above.  Even when our sampling and analytical protocols are in control, we occasionally come across difficulties.  This is especially true with trace analysis or analysis of samples that contain small numbers of particles of interest.  When this happens we could be looking at Poisson processes.  This paper is a simplified look at this to help data users recognize the issues.

A Poisson process in any process that follows a Poisson distribution (wasn’t that helpful!).  The kinds of processes include the number of customers arriving per minute at Starbucks.  That is, it is a count of random events (like customers coming through the door per unit time) that are “rare” (as in “not too many”).  If we are looking at a handful of arrivals per minute then the Poisson distribution is a good model.  If there are hundreds of thousands of independent arrivals per minute, then the Poisson distribution is usually not a good model.  A graph of a Poisson process might look like:

  Figure 2

In Figure 2 we simulate the number of customers arriving at Starbucks using a Poisson distribution.  We assume that the average over our experiment is 3.7 customers per minute.  We would expect to count 3 arrivals per minute about 21% of the time (that is what is meant by “Frequency”).  We would expect to count 0 and 8 arrivals per minute about 2% of the time.  This “example” was generated in Excel using the POISSON.DIST function.  We can estimate the mean by:

     

Where:

λ = the mean

i = the number of arrivals

p_i = Poisson probability of i arrivals

 We can estimate the variance by: 
 
   When we do these calculations on our simulated data for n = 0 to 10, we find that the mean calculates out very closely to 3.7 (actually 3.63) and the variance is also very close to the same number (also being 3.63).  This is typical of a Poisson distribution.  The mean always equals the variance.  Hence, the standard deviation is the square root of the mean.  And finally, the relative standard deviation is:

 
Hence, when we have a limited number of counts, the relative standard deviation of the counts is simply the inverse of the square root of the mean.  When the number of counts is low, the relative standard deviation is high.  In our example the RSD is nearly 52%.

When we are taking samples that have a limited number of particles of interest, the variability coming from the variability in the number of particles in a sample is very high.  That variability often follows a Poisson distribution.  Let’s consider a mental example.

Suppose we have a large lot that contains random particles of interest scattered throughout.  Let’s also assume that those particles are very uniform with each particle contributing 10 concentration units based on our sampling protocol.  An example might be particles of lead each weighing 10 ug.  If we pulled 1.00 gram samples and found that, on the average, we got 2.5 particles per sample, we would naturally conclude that we have 25 ppm of lead in the large lot (25 ug per every 1 gram).  Because these occur as discrete particles, the distribution of sample results would probably be Poisson and look something like:

 Figure 3

We would have a lot of data points well away from our mean value of 25.  In fact, when we model this in Excel we obtain a mean of 24.99, a variance of 249.6, a standard deviation of 15.80 and a relative standard deviation of 63%.  The astute reader will note that the mean is not numerically equal to the variance because each particle contributes 10 units to the answer.  Hence, the Poisson mean is multiplied by 10 and the variance is multiplied by 10².  Interestingly, the relative standard deviation remains unaltered in this kind of unit transformation.  The relative standard deviation remains a reflection of the variability in the number of particles collected.  If, of course, there was also variability in the size or lead content of the particles, the variability would be even higher.

Now let us suppose that the average number of particles in our samples rose to 35.  We would expect our average lead result for the lot to go up to 350 ppm.  Our sampling histogram would look something like:

 Figure 4

Notice now that the histogram is starting to look like a normal distribution curve.  Nevertheless, when we calculate mean, variance, standard deviation and relative standard deviation from our model we obtain: 349.98, 3501, 59.17 and 16.9% respectively.  Although we seem to have a “Gaussian curve” we still have a very high relative standard deviation.  Furthermore, this large variability is all coming from variability in the number of particles in the samples we have been taking from the lot.  We should again see that the relative standard deviation is still predicted well by the inverse square root of the average number of particles.  That is, λ^-0.5 or 35^-0.5 = 16.9%. 

We could continue this process, but it should be clear by now that it requires quite a few particles in each sample for the counting error to “disappear.”  It turns out that the “rule of thumb” used by radiochemists is pretty good.  They like to have around 10,000 counts (i.e. detected radioactive disintegrations, also a Poisson process) for every radiometric analysis.  This puts the relative standard deviation from counting error alone at 1% (10,000^-0.5 = 0.01).  It is not usually hard to have 10,000 or more particles of interest in a sample unless we are doing trace analysis (well under 1%) or using very small sample sizes (milligrams, microliters, etc.).  The analyst and data users should be wary when low level analysis is being done using techniques that require very small samples.  The precision of the analysis could crater due to the appearance of Poisson counting error beginning to dominate the analytical methodology.  This can actually begin to be significant long before the data begins to look “non-normal” in histograms (e.g. 35 particle example). 

In designing your experiments please keep in mind:

1. Where particulates are important, try to ensure that well over 10,000 particles of interest are likely to be present in all samples and sub-samples.
2. Where relative standard deviations begin to exceed 15% and there are no reasonable analytical explanations (e.g. detection or control limits reached), begin to suspect Poisson counting errors.  That is, too few particles, counts, photons, etc., are being collected, generated or detected to ensure low counting error.
3. Remember that many of the data analysis schemes (sampling protocols, confidence limits, control charting, etc.) assume normal distribution.  If Poisson processes are in play many of these schemes fail to be reasonable methods for data generation and interpretation.  Very big errors could be in the offing.

 

 

 

Sampling Heterogeneous Materials – Lab Sub-Sampling Issues

Introduction:

One of the least well understood parts of the analytical process is the collection and handling of heterogeneous samples.  These include multi-phase liquids and gases, but especially heterogeneous solid samples such as soils, sediments and rocks for minerals, nutrients and contaminants.  There are essentially four separate but related steps.  These are:

1. Initial collection of the sample,
2. Transportation to the place of analysis,
3. Sub-sampling and preparation of the initial sample to an appropriate size for analysis and
4. Actual analysis

The last of these is dealt with exhaustively by universities and instrument manufacturers.  Both drive unceasingly toward speed, automation, precision, low cost, high sensitivity and smaller and smaller sample sizes.  Considerably less attention is paid in designing and implementing the collection of the initial sample.  This has been the purview of a relatively small and dedicated group of scientists, engineers and manufacturers.  Even less thought and effort is put into taking a proper sub-sample of the initial sample for the purpose of the actual analysis (Steps 2 and 3).  In this blog we will focus on Step 3 – the taking of a proper sub-sample for analysis.

General Principles:

Sub-sampling of a correctly taken sample from a much larger lot is best guided by the same principles that were used to take the initial sample.  Taking of the initial sample is, after all, a “sub-sampling” of a much bigger lot – e.g. the box car, the truck load, four hours of weigh belt flow, etc.  For heterogeneous solids, these principles have been thoroughly worked out by Dr. Pierre Gy, Dr. Francis Pitard and others (see Bibliography below).  Their focus was originally on the mining industry, but over the past several decades these principles have been applied to agricultural, pharmaceutical and even environmental fields.  In 2003, the USEPA published “Guidance for Obtaining Representative Laboratory Analytical Subsamples from Particulate Laboratory Samples,” EPA/600/R-027.  In 2012, ASTM updated D6323, “Laboratory Subsampling of Media Related to Waste Management Activities,” using the principles set forth by Gy and USEPA.  These principles are now well established, not only for initial samples but also for doing sub-sampling prior to analysis.

Sampling and Preparation Equipment:

It is essential that we have the proper sub-sampling equipment on-hand.  That equipment must be able to collect the sub-sample without biasing the result.  That includes the proper scoops, shovels, sieves, grinders and sample splitters.  Unfortunately it is not possible to select the right equipment until the actual sub-sample size and condition (especially particle size) are known.  This decision must be postponed until we work out an overall strategy and see what is required.  Hence, our first approach must be a mental exercise where we ASSUME that we have the right equipment.  Nevertheless, it is positively critical that we return to this topic after we have determined what it is that we are handling.

The Sub-Sampling Strategy:

In general we will begin by looking at the sample that we receive from the field and the final sub-sample that will be needed in the actual analysis.  In almost all cases, the sample received from the field is much too large to be used directly for analysis.  We might receive several kg of marble-sized rocks and the initial preparation step for analysis might call for 2.00 grams of sample ground to less than 1 mm in diameter.  The question is now how do we go from the initial sample to the lab analysis sample without adding bias or undue variability?  We will apply Gy’s principles for taking a “representative” sample from the initial sample, reducing the size of that aliquot and then taking a “representative” sample from that.  We might find that we have to do this several times.  Graphically we can look at this on a “nomagraph”:

 
        Figure 1

At each step we will want to stay well below the variance goal of our measurement.  Every time we make a new step we must “propagate” the error by adding the variances.  It is also extremely important that we take sub-samples “correctly” so that we only add to what Gy calls the “Fundamental Error” and we do not add to the problem with additional, uncontrolled errors or biases. 

When correctly taking a sub-sample, the variance of the “Fundamental Error” (s²_FE) may be described as:

 

Where:

s²_FE = The variance of the Fundamental Error

M_S = The mass of the sub-sample in grams

M_L = The mass of the “lot” from which the sub-sample is taken

C = The sampling constant in g/cm it includes several factors

d³ = The diameter of the largest particles in cm

When M_L >> M_S the Equation 1 reduces to:

 

The sampling constant factor C is the product of four additional factors, c (compositional factor), f (shape factor), l (liberation factor) and q (granulometric factor).  C is the product of these four factors:

The compositional factor, c, takes into account the concentration of the component of interest.  The lower the concentration of interest the more difficult it is to get a low relative standard deviation.  The compositional factor may be estimated by:

Where:

a_L = mass fraction of component

λ_M = density of the component bearing particles

λ_g = density of the background material

When a_L is small (<= 0.1), Equation 4 can be reduced to:

The liberation factor, l, is related to how completely the component of interest is separate from the components that are not of interest.  The liberation factor is highest when the component of interest exists as pure particles of substance in an environment of “inert” particles.  An example might be trying to determine the lead content of a berm at a shooting range.  The lead exists is slivers and clumps of nearly pure lead in a big pile of sand.  The liberation factor is based on the formula:

Where:

a_L = mass fraction of component in the lot

a_max = mass fraction of component in the largest fragment

There are other methods for estimating the liberation factor.  Nevertheless, the liberation factor is rather difficult to measure.  It can, however, be estimated using the following table for typical values: 

l	Type of Material
1.0	Analyte 100%   “liberated.”  This is recommended for   use with environmental samples contaminated with pure or nearly pure components.
0.8	Very heterogeneous   materials with easily distinguished particles with high concentration of the   component of interest
0.4	Heterogeneous   materials
0.2	Average materials
0.1	Homogeneous   materials
0.05	Materials known to   be very homogeneous

Table 1

The shape factor, f, is a measure of the particles divergence from a perfect cube.  Needles and flakes are the extremes of divergence.  The factor is hard to measure but can be estimated by:

f	Description
>1.0	Needle-like   materials.  Asbestos would be an   example.
1.0	All the particles   are cubes
0.5	All particles   spheres.  Most materials, including   minerals and environmental samples, are more or less in this category
0.2	Soft, homogeneous   solid particles, such as tar droplets or gold flakes
0.1	Flaky materials   such as mica

Table 2

The granulometric factor, g, accounts for the distribution of particle sizes.  The more uniform the particle size distribution the higher the granulometric factor.  This is difficult to measure but can be estimated from the table below:

g	Description
0.25	Undifferentiated   materials.  This includes most soils   and rocks that have been crushed to a maximum size but not differentiated by   sieving.
0.40	Materials passing   through a screen (i.e. some has been retained and rejected)
0.50	Materials retained   on a screen
0.60 –   0.70	Materials retained   between two screens
0.75	Naturally sized   materials such as cereal grains and some sands
1.00	Uniformly sized   materials such as ball bearings

Table 3

For laboratory sub-sampling, Equations 2 and 3 turn out to be the most useful.  Generally, a_L is around or below 0.1.  Furthermore, we can usually use Tables 1 through 3 to estimate the values needed for calculating C.  Our strategy would include evaluating our ability to take a correct sub-sample from the lot that was presented to the laboratory AND planning to take final sub-samples for analysis.  This is best illustrated by an example.

Example of Laboratory Sub-Sampling Strategy:

Let us suppose that a correct sample of a heterogeneous stream of crushed rock has be taken during loading of a train car.  Let us also suppose that our task is to measure the carbon content of the rock.  Let us also suppose that the content (a_L) is roughly 0.1.  And finally, let us target a relative standard deviation of the Fundamental Error at 5% or less.  We will use Equations 3 and 4 for our calculations.

Our first task would be developing a strategy for taking a manageable sub-sample for transporting to the laboratory.  It is likely that the correctly taken initial sample is many kilograms.  Our method of carbon analysis might be a CHN analyzer that prefers a few milligrams of sample.  Let us assume that our target is 0.004 gram for the final aliquot to go into the CHN analyzer.  Let’s also assume that we have additional tests that are less stringent and could use up to 100 grams of sample for the analysis.  Now our strategy might be two-fold:

1. Create a 100 gram sub-samples for some analyses and
2. Create a 0.004 g sub-sample from the above for CHN analysis.

We could do this by grinding the entire initial sample to a fairly course particle size for the 100 gram tests and then take 1 - 100 gram sub-sample and grind it to a much finer particle size for the CHN test.

Initially let’s start with our our target Fundamental Error is a relative standard deviation of 5%.  Our first grinding for sub-sampling the initial sample could be calculated by:

 

Sub-Sampling Example
Grind a Sampling to 100 g
sFE	0.05	Target Relative Standard Deviation
sFE^2	0.0025	Target Fundamental Variance
Ms	100	g
aL	0.1	fraction
λM	2.6	g/cm
c	26	g/cm (Equation 5)
l	1	(Table 1)
f	0.5	(Table 2)
g	0.25	(Table 3)
C	3.25	g/cm (Equation 3)
d^3	0.076923	cm
d	0.43	cm
d	4.25	mm

 

We could easily meet our objectives for the first set of tests by a very coarse grind.  Unfortunately, when we propagated the error for a second grind we would find that we could never do better than the 5% we accepted in this first step.  Hence, we would want to make this first step much better than 5% relative standard deviation.  A typical approach might be to grind the sample to pass through a Number 8 mesh (2.38 mm) for the first set of tests.  If we did that, we would modify our sFE to 2% (see below):

Sub-Sampling Example
Grind a Sampling to 100 g
Ms	100	g
aL	0.1	fraction
λM	2.6	g/cm
c	26	g/cm (Equation 5)
l	1	(Table 1)
f	0.5	(Table 2)
g	0.25	(Table 3)
C	3.25	g/cm (Equation 3)
d	0.238	cm
d^3	0.013481	cm
sFE^2	0.0004
sFE	0.020932

In order to have our overall propagated error less than 5% relative standard deviation, we need to shoot for an sFE² of .0025 – 0004 ~ .0021 in the second grind.  We would need to grind the 100 gram aliquot to 0.14 mm (100 mesh) and split it correctly down to 0.004 gram.  This is shown below:

Sub-Sampling Example
Grind a Sampling to 0.004 g
sFE^2	0.0021	Target Fundamental Variance
Ms	0.004	g
aL	0.1	fraction
λM	2.6	g/cm
c	26	g/cm (Equation 5)
l	1	(Table 1)
f	0.5	(Table 2)
g	0.25	(Table 3)
C	3.25	g/cm (Equation 3)
d^3	2.58E-06	cm
d	0.01	cm
d	0.14	mm

 Our strategy becomes:

1. Grind the initial sample to pass a Number 8 mesh
2. Split the ground material correctly into at least 2 – 100 gram aliquots
3. Grind one of the 100 gram aliquots to pass Number 100 mesh
4. Split the ground material correctly into several 0.004 gram aliquots

Back to Sampling Equipment:

Now that we have a strategy we can select equipment properly.  The first two steps are probably pretty easy.  There exists a lot of equipment to grind to 8 mesh and split into 100 gram aliquots (e.g. Riffle Splitters).  The last two steps could be challenging based on the nature of the material (too hard, thermally unstable, sticky, stringy, etc.).  The last step could be especially challenging.  There exist micro-Riffle Splitters that can split samples easily to a few grams.  Nevertheless, correctly selecting a 0.004 gram aliquot even from a 1 gram sub-sample can be difficult. 

It is likely that we will want to select this final 0.004 gram aliquot by taking multiple scoops from a 1 gram pile.  We should design our scoop with a flat bottom and square sides to avoid as much particle discrimination as possible.  Here we should plan to do the best we can, but we should test our procedure for variance before declaring it “good.”  A typical plan would be to run at least 10 replicates from a single pile to evaluate how we are doing.  If after several trials we see that we can reduce the number to fewer than 10 without exceeding our variance goal we can do so later. 

Summary:

This blog shows a typical approach for designing a laboratory sub-sampling plan.  Many features have been left out of the discussion.  There are a lot more details and guidance given by Gy, Pitard, USEPA and ASTM.  Especially important is a thorough knowledge of the equipment and techniques available for correct physical sampling.  The reader should consult these additional resources before declaring a sampling procedure good.  We have dealt with Fundamental Error, but we have not dealt with the tremendous bias that can result from improper physical techniques.  Improper technique can result in huge errors that are very difficult to detect and impossible to resolve.

 

Bibliography:

Gy, Pierre, Sampling for Analytical Purposes, John Wiley and Sons, New York, 1996.

Pitard, Francis, “Pierre Gy’s Theory of Sampling and C. O. Ingamells’ Poisson Process Approach,” Doctoral Thesis, Aalborg University, Campus Esbjerg, Denmark, June, 2009.

ASTM D6323-12, “Standard Guide for Laboratory Subsampling of Media Related to Waste Management Activities,” ASTM International, West Conshohockenn, PA, 2012.

Gerlach, Robert & Nocerino, John, “Guidance for Obtaining Representative Laboratory Analytical Subsamples from Particulate Laboratory Samples,” US Environmental Protection Agency, EPA/600/R-03/027, November, 2003.

 

 

Ron Stites co-Chairs Section at 34th Oil Shale Symposium

On October 15, 2014, Ron Stites co-Chaired the “Colorado Oil Shale Issues” section of the 34^th Oil Shale Symposium.  The Symposium was hosted by The Center for Oil Shale Technology and Research of the Colorado School of Mines, Golden, Colorado. 

Mr. Stites also presented a talk on “Using Design of Experiments in Pilot Plant Work.”  He explained how crucial pilot plant work is in the development of new technology -- especially as it relates to alternative fuels.  Without good experimental design the quality of pilot plant data can become very suspect.

The talk covered a variety of topics including:

• Why Good DoE is Needed for Pilot Plant Work
• What Exactly is a Pilot Plant and How They Differ from “Bench” & “Commercial Scale”
• What are the Primary Goals of Pilot Plant Work
• What are Some of the Myths about Pilot Plant Work
• An Overview of the DoE Process
• A Summary of Critical Points in Experimental Design

In addition to the above Mr. Stites gave several examples of how good experimental design has made pilot plant work much more effective.

For more information feel free to contact Ron Stites at:  [email protected]

A Critical Testing Issue: Pilot Plants and Type II Errors

Many testing groups avoid the painful process of hypothesis testing.  When they do, they take a high risk of making serious errors – especially Type II Errors.  These can be devastating for pilot plant work and can lead to significant “scope drift” as will be explained below.

It is important to note that whenever testing is done there are implied hypotheses – whether stated or not.  Even something as simple as looking at the gas gauge on our car involves a number of hypotheses or assumptions as we try to determine if we should stop and get gas or keep on driving.  We instinctively start evaluating the accuracy of our gauge, the reliability of our gas mileage, the probability of finding another station and the consequences of being wrong.  If we are about to cross Death Valley we are going to act much differently than if we are driving to the corner grocery.

In the case of a pilot plant the consequences can be very big.  They could involve safety issues or even the potential for financial ruin.  Hence, understanding the hypotheses and consequences in any testing program can be a serious matter.  Unfortunately, when we try to translate our intuitions about hypotheses or assumptions into objective statistical terms we often land in a state of confusion.  The primary cause of our confusion is the rather circular and somewhat contradictory language surrounding the topic.  We will try to keep it simple and point out some of the language traps while not losing the benefits of objective statistical terms.

Let’s begin with the idea of a hypothesis.  This is a statement that you believe, hope or at least assume to be true.  This is called the “null hypothesis” or H₀.  Here’s the first warning to the reader about language.  Some authors try to give meaning the term “null.”  By doing so they are hoping to influence the stating of hypotheses and “standardize” the process.  In many cases that really isn’t very helpful.  It often results in double negatives that add confusion.  It is actually better to think of it as the basic assumption and go from there.  The “alternative hypothesis” or H₁, is simply all other possibilities (the logical antithesis of H₀).  If H₀ is not true then H₁ must be true. 

We can give a simple example:

H₀:  Anna is the tallest girl in her math class

 H₁:  Anna is not the tallest girl in her math class

These are simple enough and even imply some things about how to “prove” or “dis-prove” them.  Notice that we could and probably should add more details to make it very clear on how we intend to measure and interpret results.  We could add, “…as measured by the school nurse using the usual method.”  It cannot be emphasized enough just how important it is to clearly state hypotheses.  This goes a long way toward driving a successful testing program.

Things get more complicated as soon as we start making measurements and realize that there seems to be some fuzziness around the measurements.  If we measure Anna’s height multiple times we will get slightly different numbers.  This is not a problem if Anna is 3 inches taller than everyone else, but it becomes quite a problem if one or more girls are very close to Anna’s height.  We now have to pass judgment on the probabilitythat Anna is actually the tallest.  This leads to two possible errors that we will show below in a “truth table”:

Hypothesis	Actual State	Measurements Say Anna is Tallest	Measurements Say that Anna is Not the Tallest
H0 = Anna is the tallest	H0 True	Right Answer	We incorrectly reject Ho Type I Error (α error)
H1 = Anna is not the tallest	H0 False	We Incorrectly Accept H0 Type II Error (β error)	Right Answer

In words the two types of errors are:

Type I:  H₀ is actually true but our data tells us that it is false.  Hence, we draw a wrong conclusion.  We think H₁ is true when it is not.

Type II:  H₀ is actually false but our data tells us that it is true.  Hence, we draw the wrong conclusion.  We think H₀ is true when it is not.

Let’s now think about a situation that could be a little more interesting.  Let’s think about an example that might occur with a pilot plant:

Case 1:  We believe that the yield in our pilot plant is highly dependent on how we start-up the catalytic reactor.  There are several different ways to do this that really add no significant costs or risks to our operation.  We decide we want to compare our current procedure (Procedure A) to a new procedure (Procedure B).  We realize that measuring yield is difficult and expensive, but small changes in yield could make us a lot of money in the long run.

Because we would really hate to lose out on potential earnings from a better start-up procedure, we set up our hypotheses and testing as follows:

1. We set our H₀ as, “Procedure B gives us higher yield than Procedure A” and
2. We set our α error quite low – say 0.05 (or 5%).

We set up our testing program so that Procedure B will be rejected only if we are 95% sure that it is not better than Procedure A.  This way we are pretty sure that we will not erroneously reject a better procedure.  This is a very typical approach in a development process.  There are many statistical programs for doing just this type of statistical analysis.  In fact, many statistical programs are set up to deal only with Type I Error.

There’s a problem here that often gets missed.  We have just biased our development program to accept hypotheses that may not be true.  As we reduce our α error (the chances of making a Type I Error), we actually increase our chances for making a Type II Error – the chances for accepting a hypothesis that is not true.  If we persist in this, we will be frequently making changes that are not demonstrably better and might even be worse than our current process.  This happens with surprising frequency in development projects.  It is a random scope drift that is actually a function of flawed analysis of hypotheses.  In the current example we could imagine many different start-up procedures that would give similar yields.  These might be accepted or rejected more or less randomly and we would never actually settle on a “best” procedure but would continually cycle between similar but hardly distinguishable procedures.

One approach would be to change the way we state our hypotheses.  We could state H₀ in the negative (i.e. Procedure B is NOT better than Procedure A).  This would have the effect of exactly reversing Type I and Type II Errors.  In other words, we would rarely accept a procedure that wasn’t demonstrably better than the current procedure.  This has some advantages, but we would also reject many procedures that actually were better than the current procedure.  This approach would do little to help us do a balanced assessment of the comparability of different procedures.  The only way to really do a balanced assessment is to deal with the issue of Type II Error directly.  This is rarely done in beginning statistics textbooks.  In fact, one intermediate university level text flatly states that “a thorough treatment of Type II Errors is beyond the scope of this book.”  This is not unexpected since the treatment of Type II Errors requires some pretty sophisticated mathematics to do it rigorously.  Fortunately, there are some useful ways to estimate the probability of Type II Errors without resorting to exacting treatment.

The key to controlling Type II Errors is setting values for our testing such that any actual, undetectable differences between H₀ and H₁ are so low that we don’t care anymore.  In other words, even if we erroneously reject H₀ it really doesn’t matter much to us.   We are setting a “level of indifference.”  We then look at our ability to measure the key responses and increase the number of tests such that we can just “see” that “level of indifference” with some measureable (or estimable) probability. 

In our current example, there is probably some level of difference between Procedure A and Procedure B, below which we really don’t care.  That is to say, we aren’t going to change to Procedure B for just any a slight improvement.  That becomes our “level of indifference” and we set up our testing so that we can detect down to that level.  In order to do this we must evaluate our ability to measure (standard deviation of our measurements) and combine that with an adequate number of tests (i.e. trials) to get an acceptable level of confidence.

We realize that the way we have our testing set up (α = 0.05), we will accept a lot of possible start-up procedures – even some that do us very little good.  In fact we are 95% sure that we won’t be throwing out a new procedure that might be good.  Nevertheless, we don’t want to accept any new procedure that doesn’t have a reasonably good chance of giving us some useful improvement.  We generally set this probability quite a bit higher than our α level.  This is called our β level.  For general screening work this is often set at 0.20.  This means that we have an 80% chance that the accepted new procedure will give us an improvement equal to or greater than our “level of indifference.”  When we approach Type II Error in this way we obtain a relatively simple procedure for dealing with it.

The process is:

1. Set the “level of indifference,” 
2. Measure or estimate the standard deviation of the pertinent measurement,
3. Set the level of acceptable Type II Error (often 20%),
4. Estimate the number of samples required using the formula: 

where:

n = number of replicates

z_α & z_β = “z scores” for the α and β probabilities respectively

s = the standard deviation of the measurement of u

(u_α – u_β) = the “level of indifference” expressed as difference in means

Several software programs will perform the calculation above including StatEase Design Expert 9.0.

Let us now go through a typical example using the scenario of Case 1 above.  Let’s say that we are indifferent to start-up procedures that produce less than a 1% change in yield.  Let’s also assume that we have measured yields many times in our pilot plant and can hold a standard deviation of 0.25%.  Let’s also assume that we are screening many start-up procedures and want to set our α at 0.05 and our β at 0.20.  When we analyze these data we see that we need 5 to 6 runs to reduce our Type II Error to less than 20%.  If this was a very simple test with only these two factors (Procedure A and Procedure B) we would set up our testing with 3 random comparisons of the two procedures for a total of 6 start-up tests.

Would we be done?  Probably not.  If our testing showed that Procedure B was much better than Procedure A – say by >5%, we would feel quite confident that we were on to something.  Even then, we would probably want to repeat a verification test from scratch.  There’s always the outside chance that our whole procedure was screwed-up (labeled procedures incorrectly?) or was compromised by some exogenous, uncontrolled factor (changes in catalyst?). 

If Procedure B was only slightly more than 1% better than Procedure A we would be even more cautious.  There is still nearly a 20% chance that we are wrong about Procedure B.  If we decide that we really want to go after that extra 1%+ in yield, we would want to repeat our test and maybe even redesign it to be a more convincing. 

Wanted – "Weird Projects"

Unlike many of my friends and colleagues in the commercial lab business, I’m looking for those weird projects that just don’t fit in a commercial lab setting.  In fact, that’s what I usually did even when working in the commercial lab business.  I was often the guy asked to “figure out” how to do the “weird project.”  The "weird project" sometimes requires a new analysis technique, but more often requires a new sampling or sample preparation technique that can be used with existing analytical techniques.  Another challenge is often validating the entire procedure to ensure that statistically valid data are being produced.  Many times the work can be done by the commercial lab once the modified sampling and sample preparation has been done.  That can result in synergy between large labs with plenty of analytical capabilities and Stites & Associates.  Let me give just a few examples from past experience to illustrate the point.

Tars in Syngas from Biomass:

Gasification of biomass is plagued by the formation of “tars.”  These are a wide variety of sticky, semi-solid substances that are generated primarily from the lignin in woody biomass.  It turns out that many of these substances can be seen on a standard BNA run done in an environmental laboratory.  Hence, a standard BNA run can be used as an indicator of tar formation and gasifier performance.  The biggest challenge is pulling a representative sample from the gas stream.  The “tars” have a wide variety of boiling points and solubilities.  They often form semi-solid droplets in the gas stream.  The key is designing an appropriate sampling system.  This may include heated lines, solvent traps and even filters depending on sample point and the purpose of the test.  No one sampling system is likely to give a complete and unambiguous “tar number.”  Nevertheless, with proper design the data can be very useful for troubleshooting purposes.  Stites & Associates has worked with several labs on several projects of this type with good success.

Water in Processes:

One of the most frustating analysis is water content of "real samples."  It is ubiquitous and often a very important compound that shows up in the most inconvenient places.  There are many ways to measure water content of a sample.  They include volumetric separation, co-distillation, Karl Fischer Titration, thermal drying, chemical adsorption, FTIR, dew point measurement and GC, just to name some of the more common.  Unfortunately, all of these can have serious limitations and interferences.  Depending on the matrix and sampling conditions, measuring water content often has very serious bias and/or precision problems.

Nearly every commercial laboratory has one or more of these techniques available.  Few labs have a complete suite of techniques.  In most cases the techniques they have are designed for the types of samples that they commonly receive.  The key is collecting the sample in a way that does not adulterate the water content, but can be analyzed by one of the laboratory techniques available.  Sometimes simple drying and weighing will be “close enough.”  Even then, proper sampling (including taking an aliquot in the lab) is crucial to reliable data.  Other times absorption in a solvent followed by analysis can work well.  Stites & Associates has worked with several labs in collecting and analyzing solid, liquid and even reactive gas samples at high temperature for water content. 

Carbohydrates in Enzymatic Processes:

Carbohydrates can be difficult to measure well, but there are now many reasonably reliable HPLC methods.  Unfortunately, most real world applications are taken from dynamic processes where carbohydrate concentrations are not stable.  One example would be the enzymatic hydrolysis of starch, cellulose and other macro-molecules.  To track the progress of an enzymatic process it is important to analyze the carbohydrates “real time.”  Often that means stopping the process in the sample until the analysis can be completed.  This must be done without adulterating the carbohydrates under consideration.  Stites & Associates has worked with labs to design sampling and preservation techniques that can be used to ship samples to remote labs – including overseas. 

These are just a few examples where Stites & Associates has worked with labs to make “weird projects” reasonable to perform in commercial labs.  We have focused on some of the sampling and sample preparation aspects, but there is a whole host of Experimental Design issues that we haven’t discussed here.  When it comes to validating the methodology used in a “weird project” or attempting to draw statistically valid conclusions from the results, Stites & Associates again can help.

If you have a “weird project,” we’d like to hear about it.  Maybe we can help.   We’d be happy to take a look at it and give you an honest assessment.  Frequently what is being asked turns out not to be feasible.  Nevertheless, it is usually possible to come up with an approach that goes a long way toward solving the real problem – often more cost effectively than the original approach.  Give us a call at (912) 247-6120 or write us at [email protected].