Updated: Oct 10, 2022
Lawyers and Judges need to get past their deep-seated fear of uncertainty. If we are to promote the use of good science, in Court, we, as lawyers, need to stop thinking in terms of absolute certainty in science. We are used to the concepts of "guilt beyond a reasonable doubt" and "on a balance of probabilities", why can't we attempt to wrap our heads around probability and uncertainty of a measurement result in the qunatitative analysis discussed every day in Court? Isn't it helpful in assessing proof beyond a reasonable doubt to have evidence about the probability that the measurement result alleged by the Crown is an accurate measurement result? Is 95% probability of guilt beyond a reasonable doubt? Is 98% probability of guilt beyond a reasonable doubt? An what if no one in the police (contrasted with a CFS lab) attempts to control the conditions precedent to calculating that 95% or 98%?
We learn from the CFS 8000C Information Sheet, (July 15, 2011, p. 5 of 8):
The Intoxilyzer(R) 8000C has been demonstrated to be at least as accurate and precise as the results of near-simultaneous breath tests conducted with other approved instruments. Breath testing typically underestimates an individual's BAC. Overestimation is rare and is highly unlikely to exceed 10 per cent.
Judges can accept this evidence from Crown experts who use terminology such as "demonstrated to be", "typically underestimates", "overestimation is rare", and "highly unlikely to exceed 10 per cent". Why shouldn't we lawyers ask good questions about what this terminology means, what is the published empirical testing that underlies these opinions, AND IN WHAT SPECIFIC CIRCUMSTANCES DOES THIS TERMINOLOGY APPLY? WHO IS RESPONSIBLE FOR CALCULATING THE BUDGET OF ITEMS THAT GO INTO SUCH CALCULATIONS? These concepts all relate to measures of uncertainty that are connected with any quantitative analysis. Lawyers need to learn about "Uncertainty of Measurement". Where do we start: perhaps with basic statistics?
To simplify the concept of Type A Uncertainty of Measurement to make it more understandable to a trial judge.
To obtain an admission from the CFS scientist that Uncertainty of Measurement is not "novel science".
To explain calculation of average, precision, and standard deviation in simple math terminology.
To connect CFS documents that talk about coverage factor with the international literature of Uncertainty of Measurement.
MR. BISS: Q. So, I was just going to say, I promised His Honour earlier – I don’t think I’ve given the Court one of these yet – frankly, those of us – the lawyers in the room, trained in law school, need all the help we can get in trying to understand what on earth measurement – well, not so much measurement uncertainty – but what – what is standard deviation, what is precision, and I just want to have you go – first of all, to page 4 of this document. It’s a paper by someone by the name of Stephanie Bell from National Physical Laboratory, which I think is in the U.K. Teddington, Middlesex, United Kingdom. It’s got a Crown copyright on the third page. 1999. A. So, page four?
Q. So, if you could go down to on the page that’s indicated number four. A. Yes. Q. In getting an average, there’s a suggestion at the bottom of the page in the last paragraph. It says, “Broadly speaking, the more measurements you use, the better the estimate you will have of the true value.” Would you, generally speaking, agree with that concept? A. Yes.
Q. And specifically, with respect to spread, standard deviation, paragraph 3.5 on page 5, gives an explanation of what standard deviation is, and I wonder if you’d just read that through, that – the bottom half of that page and tell me if you agree with it, and is – is that – for those of us who don’t have a background in statistics and are not scientists, is it a good explanation of what standard deviation is? A. Yes. Q. And page 6, how to calculate an estimated standard deviation, it gives an example of a calculation there, and I think you said that you would normally use an Excel spreadsheet. A. Yes.
Q. The methodology there, I can tell you, is what I used in trying to come up with my calculation of the 50 and also to confirm how it was that the C-M-I 8000C does its calculation of standard deviation, but doing it manually. And do you have any problem with – with the description here of the methodology for manually calculating estimated standard deviation for those of us who don’t have calculators in front of us? A. That's correct. As we discussed earlier.
Q. Right. And I’m just going to take it one step further. Now, standard deviation – the concept of standard deviation is that what – what - in a normal distribution we expect that what percentage of the results will be within one standard deviation of the mean. A. Sixty-seven percent. Q. Right. And then if we want to know – if we want to cover a larger area than that, page 16 there’s something called a coverage factor “K”. A. Yes.
Q. Paragraph 7.4. Could you just read – read through, please, what it says in paragraph 7.4? I want to make sure that’s correct. A. I agree with what’s written there. Q. All right. A. It’s taking me back, obviously, a few years to university statistics. I am concerned about the second K for – the K for 2.58. Q. I stumbled over that as well when I saw it, although you, I’m sure, have far more experience that me, education in statistics than I do. But my understanding is that you multiply the coverage factor – if you have a coverage factor of about 95.5 percent, which is what you identified this morning... A. Yes. Q. ...then you multiply by two. A. Yes. Q. If you are looking for a coverage factor of 99 it means you multiply by 2.58 and if you want a coverage factor of 99.7 you multiply it by 3, approximately. A. That is correct. Yes. Q. So, that does make sense.
A. Yeah, okay. Q. All right. At first, I thought there was something wrong there, it was a misprint, but you’re comfortable with that now? A. Yes. MR. BISS: So that Your Honour – if that could be an exhibit? THE COURT: Yes, 43. I never thought I’d receive an exhibit titled A Beginner’s Guide to Uncertainty of Measurement, but – until today. But now I have it. EXHIBIT NUMBER 43: A Beginner’s Guide to Uncertainty of Measurement – produced and marked. A. It’s a less offensive title than some of the books that are out there. THE COURT: Yeah, that feature Dummies at the end? Yes. A. The black and yellow ones, yes.