How Certain is Your Data?

An overview of areas of uncertainly that may affect your laboratory results

The awareness of uncertainty associated with laboratory data from contaminated sites has increased significantly in recent years, in the UK partly due to the implementation of the MCERTS standard by the Environment Agency for laboratories involved with testing soil samples, and partly due to general improvements of standards/legislation/controls within the industry itself, e.g. implementation of SiLC (Specialist in Land Contamination) qualifications. Awareness across Europe as a whole is much higher than in the early 1990s, when most countries were implementing Environmental Protection policies of one kind or another.

However, an understanding of the issues amongst environmental consultants, engineers and contractors is patchy, to say the least, (few companies employ a qualified chemist), and most good quality laboratories spend a significant part of their time explaining data to their clients, and providing technical support and advice. This article is an overview of most of the areas of uncertainty which may have an impact on the end result, and concludes with suggestions of how professionals within the industry can improve the level of certainty associated with their data.

Definitions

First, a brief explanation of some of the common terms in use:

Accuracy The closer a result is to the true value, the greater the accuracy

Precision The closer together a set of replicate data, the better the precision

These terms are sometimes confused, and the diagram in Figure 1 is a simple aid to an understanding of the difference between them.

Measurand (or analyte or determinand) The specific quantity subject to measurement (true result)

Uncertainty A parameter, associated with the result of a measurement, that characterises the dispersion of the values that could reasonably be attributed to the measurand

Standard uncertainty The estimated standard deviation

Combined standard uncertainty The result of the combination of standard uncertainty components

Expanded uncertainty Obtained by multiplying the combined standard uncertainty by a coverage factor

Coverage factor A number that, when multiplied by the combined standard uncertainty, produces an interval (the expanded uncertainty) about the measurement result that may be expected to encompass a large, specified fraction (e.g. 95%) of the distribution of values that could be reasonably attributed to the measurand

Variance A measure of the dispersion of a set of measurements; the sum of the squared deviations of the observations from their average, divided by one less than the number of observations

Standard deviation (SD) The positive square root of the variance of a data set

Relative standard deviation (RSD) (%) The standard deviation expressed as a percentage of the mean of a data set – this is equivalent to the precision

Bias (%) The mean value of the data set minus the assigned (true) value expressed as a percentage of the assigned value

In general, no measurement or test is perfect, and the accumulated errors and imperfections give rise to the expanded uncertainty as defined above. Tables of data are supplied to clients as absolute values, but it would be preferable to provide each result as a range, rather than a discrete value. However, this is difficult to deal with from a risk assessment point of view, and does not fit in well with standard models. Laboratories should supply spreadsheets of the uncertainty values determined during method validation, and these will provide clients with an understanding of the variation related to a particular method, but may not be applicable to all soils due to possible matrix interference effects. These may be caused by very high values of some contaminants or unusual background matrices, and a true bias for each specific sample can only be measured using spike and recovery techniques for every parameter. This is possible, but the costs (and timescales) associated with this are usually prohibitive. Also the spiked analyte is often much more readily extracted from the sample than the endogenous analyte so a somewhat optimistic value of the uncertainty is obtained.

Sampling errors on site

Before we can look in more detail at uncertainty within the laboratory, we need to consider the uncertainty associated with site sampling. An article in the June 2006 edition of AWE by Clive Griffiths of STATS Limited discussed this in some detail, and this article will not go over old ground. However, there are some additional points which are worthy of consideration.

The aim of a sampling exercise is to take samples which are as representative as possible of the area which is being sampled. This is not easy, given the size of an average site, and the figures below provide a good example of just how difficult this can be in reality.

This is not exactly what one would describe as representative… Please, take more samples.

Aside from the problems of possibly missing areas of contamination, other common problems on site include the following:

• Loss of volatiles due to poor handling or incorrect sample bottles
• Not compositing/mixing soil samples
• Not homogenising surface/borehole water samples containing a product layer (i.e. two phase sampling)
• Cross contamination caused by poor handling techniques
• Not using field and trip blanks to check against spurious contamination and potential analyte loss
• Labelling errors (more common than people realise)
• Filtering (or not filtering) incorrectly
• Not using correct sample bottles and preservatives
• Not storing samples at the correct temperatures
• Storing samples on site for too long, therefore exceeding permitted holding times

Sub-sampling in the laboratory

The 500 g – 1000 g supplied by most clients to the laboratory is considered to be representative of the location/depth on site (bearing in mind the above discussion), but only 100g or so of the soil will actually be used for testing. Technicians at the laboratory must therefore mix the soil and take a representative sub-sample of the submitted sample.

Soils must be mixed (and often tested) on an ‘as received basis’, as many parameters would be affected by drying and crushing, and dependent on the soil type and grain size, this can be very difficult. Loamy/topsoils, sands, silts and similar matrices are relatively easy to mix and sub-sample. Clays, gravels and samples containing bricks or hardcore/concrete/fill/tar/waste can be very difficult. A variety of mixing techniques may be used (jaw crusher, hammer, mixer mill, paddle mixer), but none will give a completely homogeneous sample – this can only be achieved by drying and crushing a sub-sample to a powder of less than 250 microns. Clients should therefore be aware that the following tests are performed on ‘as received’ samples, and will have a higher level of uncertainty associated with the results, in some cases, due to the potentially lower levels of homogeneity:

• Cyanides
• Phenols
• Leach tests
• Ammoniacal nitrogen
• Acid soluble sulphide
• Hexavalent chromium
• EPH or TPH
• Organic mercury
• Volatile organics (VOC)

Validation of methods

Under ISO 17025 (and MCERTS in the UK), all methods must be validated to prove they are fit for purpose, and with soils this involves performing the validation on three different matrices (possibly more, if routinely analysed). Most soils laboratories have carried out validation on clay, sand and a loamy topsoil.

Under MCERTS, validation is a very extensive procedure, involving testing in duplicate over eleven days for every parameter on three matrices at a high and low concentration. This amounts to one hundred and thirty two runs for every parameter – and that’s if it works first time. In reality, many methods needed changing to ensure good recovery over all the matrices and concentrations, and this has been a very time consuming (and costly) exercise for all laboratories.

Problems encountered by laboratories in the UK included:

• A lack of Certified Reference Materials (CRMs) containing the desired analytes
• A lack of CRMs in the correct matrices (clay is particularly difficult to find) and at appropriate analyte concentrations
• No such thing as an ‘as received’ CRM – all CRMs are dried, crushed and homogeneous. (i.e. best case samples)
• A lack of defined methods

However, it has hopefully allowed laboratories to validate methods which are empirical (where the result is defined by the method), e.g. all leaching methods; water soluble boron etc. For contaminated land analysis metals are reported as aqua regia soluble, not total. Thus metal results obtained by XRF, (especially for elements such as iron, manganese, nickel, titanium, sodium, aluminium etc.) are usually higher than after using an aqua regia digestion. However, for environmental risk assessment, if the metals are not released by vigorous aqua regia digestion, they are unlikely to be of environmental significance.

Bias and precision targets have been set by MCERTS, and laboratories must comply with these to receive accreditation. Generally, inorganic methods have more onerous targets than organic methods.

Instrument uncertainty

Once the samples are sub-sampled, digested/extracted, filtered, diluted, and have undergone any other preparative procedures, they are ready for analysis. Generally, instrument precision is very good – usually < 5 %, and often < 2 %. Higher variations in precision are often associated with the preparative procedures. Daily variation is monitored by the use of Analytical Quality Control (AQC) standards, which are run with every batch (ten to fifteen) samples, and are plotted on to Shewart charts.

The central line is the assigned (known) value of the standard, and during validation the Standard Deviation is calculated. Twice the SD and three times the SD are then plotted as warning and action limits on the chart, and 99.7% of the AQC values should fall within these. These charts are valuable tools for the laboratory to ensure data is kept under control, and are scrutinised closely by UKAS during audits. If the AQC values are not within the action limits, then the batch of samples is repeated.

One of the major areas of uncertainty with soil analysis is that most samples are not run in duplicate. In some other types of analytical laboratory (e.g. forensic and medical) all tests are carried out in duplicate, or even triplicate. The cost constraints within the contaminated land industry do not generally allow for this, and therefore most samples are only run once. This will significantly increase the level of uncertainty, e.g. a sample may give an arsenic result of 8 mg/kg, and on one result, this could mean the true value lies between 5 and 11 mg/kg. If a duplicate sample is run and gives an answer of 5 mg/kg, then the result will be given as 6.5 mg/kg (the mean of the two results). The more replicates which are performed, the greater the degree of certainty associated with the data.

Other problems which may occur, particularly with complex matrices, are spectral interference effects for metals being determined by ICP OES, where, for example, iron (which has a significant number of strong emission lines), may overlap the instrument spectral bandpass and thus interfere with an analyte metal emission line. With chromatography, one compound may co-elute with another compound, and it may not be possible for the system to separate them – the concentration of one compound is therefore reported as a falsely high result.

These potential problems can be resolved by specialised background correction routines or deconvolution techniques. For traditional “chemical interferences”, performing spike and recovery techniques can be used to correct for this type of interference. This is where the sample is analysed as usual, and then spiked with a known amount of every parameter of interest in turn and run again. Any differences between actual and expected recovery can then be applied to the result as a correction factor. Ideally the spike should be 1 to 2 times the analyte concentration. Again, the time and cost constraints do not allow this to happen routinely in the UK, although it is more common in laboratories in the USA.

Interpretation of data

There are problems which arise when data is reviewed, and some common misunderstandings are as follows:

Metals Normal (geological) background levels in some areas of a country may be classed as contaminated

EPH (or TPH) Extracts which are not cleaned up using silica/alumina columns may contain significant levels of indigenous organic material

Units Clients may assume these are all the same – often there is a thousand fold difference (e.g. mg/kg and ug/kg)

Elements/Compounds Comparisons of total values against compound values, e.g. total phosphorus concentrations are approx 33% of orthophosphate concentrations due to the oxygen in the PO4

Polyaromatics These are not always indicative of petroleum contamination, but can be derived from coal products as well

How can we improve the certainty of data?

On site protocols to follow and/or check:

• Ensure the correct bottles and preservatives are used, e.g. do not collect samples for organic analysis in plastic containers
• Ensure samples for volatile analysis do not contain a headspace and are airtight
• Where possible, composite soil samples from different areas, and send the mixed sample to the laboratory – this gives better representation
• Send some duplicate samples to the laboratory
• Send some duplicate samples to a second laboratory
• Sample from the cleanest areas of a site first, and work towards the most polluted areas – there is less likelihood of cross contamination
• Ensure there are sufficient cool boxes and ice packs (two to three per cool box), and that these are frozen – the laboratory does not normally send them out frozen
• Ensure the courier is arranged in good time – preferably at least the day before the collection
• Where manual submission systems are used, check the information on the chain of custody sheets matches the labels on the bottles
• If using acid preserved bottles for metals, then water samples must be filtered using a 0.45micron filter before filling the bottle, or suspended solids may dissolve in the acid and give falsely high results
• Ensure an adequate supply of deionised water is available on site for washing sampling equipment
• If a product layer is present within a borehole, then equipment must be rinsed with either a solvent or a detergent based product. This must then be rinsed with deionised water to remove any residues. A sample of this water can be sent to the laboratory as a rinsate blank to check there is no carry over
• Request trip blanks from your laboratory – these are filled by the lab and not opened by site personnel, but simply returned to the laboratory as a check on their bottles, water and handling techniques
• Prepare field blanks on site – fill a selection of sample bottles with deionised water and send to the lab – these will act as a check on your handling techniques and for any airborne contamination

Checks on the laboratory

• Ensure the laboratory is accredited to ISO 17025
• Ensure the tests of interest are also accredited – some laboratories have a limited suite of accredited and/or MCERTS tests
• Request copies of Proficiency Scheme testing data – again, some laboratories only enter a limited number of parameters
• Request copies of the relevant AQC charts for all parameters over the period of your analysis
• Visit the laboratory – check there is adequate space for cold storage of samples, check the preparation of samples, not just the analytical instruments, most uncertainty occurs in the preparative procedures
• Request as many duplicates as your budget will afford, and ensure you provide sufficient sample for the laboratory to do this
• On the Chain of Custody documentation, ensure you inform the laboratory of any particularly high levels of contaminants – these can have a significant impact on the methods and may result in cross contamination of uncontaminated samples in extreme cases. In addition, if any particularly high level of non-toxic compounds are present, e.g. saline conditions, these can also compromise the data if the laboratory is unaware of them
• If your samples consist of an unusual matrix not covered by the standard soil matrices, then you may require the laboratory to carry out validation on this matrix (by spike and recovery, as a CRM is unlikely to exist). If so, then you need to supply enough of the material for the laboratory to do this, and also be aware of the time it will take – up to three or four weeks in some cases
• Be very specific in the testing suites you request, particularly with poorly defined parameters such as TPH (Total Petroleum Hydrocarbons). Volatile petroleum hydrocarbons (carbon range C5 – C10) and extractable petroleum hydrocarbons (C10 – C44) are both needed to provide the full range. Mineral oil is only the aliphatic part of TPH – it will not include aromatics and polar compounds
• Most important of all, talk to your laboratory – discuss your testing requirements, give them any additional information you can, warn them when the samples are likely to arrive, be very specific over reporting times or non-routine reporting formats, and generally use them as a technical resource – they should be only too happy to be included from the outset (and preferably before you get on site)

Conclusion

This article will hopefully have raised the awareness of end users of laboratory data with respect to the uncertainty associated with their results. What is not intended, is a horror struck response of ‘Well, I might as well just make up some numbers.’ With an understanding of the limitations, care in the on-site sampling, and a good working dialogue with your laboratory, there are many ways to ensure the data clients finally receive will be accurate, reliable and timely.

Published: 10th Sep 2006 in AWE International