Purpose:
To obtain admissions about the use of historical control test data by the Centre of Forensic Sciences in Toronto to calculate standard deviation and uncertainty of measurement.
To obtain an admission that the same approach could be used with historical data of Intoxilyzer control tests to prepare statistical data of accuracy and precision at different points in time so as to assess drift in accuracy and precision over time, in other words, reliability.
Q. Let’s – let’s talk about – let’s suppose that a blood sample from my client had come in to the Centre of Forensic Sciences submitted for analysis and you’re the analyst. A. Yes. Q. And you are analysing it, as you indicated, you follow all the appropriate procedures, and you would come up with a – assuming that it was a measurement result that was worthy of a quantitative result, not just qualitative result,
but a quantitative result, so something in an interesting range, if I can use that terminology. A. Okay. Q. Let’s – let’s suppose that the result was 50 or 100 or 110 or 120 or 130 or 140. When you reported that measurement result in your ultimate report, you would also report something about the accuracy and the precision of the instrument that you were using to conduct that analysis, right? A. No. What you would have is a copy of a report that would show the blood analysis that was performed, the concentration that was obtained. Q. Yes. A. And the variability associated with that measurement. Q. See, the variability associated with that measurement. That variability, where’s that come – where does that information come from? A. That comes from... Q. That variability. A. ...running standards... Q. Yes. A. ...over time... Q. Yes. A. ...using a known concentration and looking at the variability associated with that standard at that particular concentration. Q. So, you would say for example, that you are 95 percent confident.... A. 95.5. Q. 95.5 percent confident. Do you use two standard deviations or approximate?
A. Yes. Q. Yeah. 95.5 percent confident that the result falls within this particular range, but the range wouldn’t be just plus or minus 10 milligrams per 100 mils. It could be something completely different from that. A. Correct. It would be based on the analytical standard that was run, that’s closest in proximation to the concentration that was determined in the unknown. So, for instance, if you had a result of 100 milligrams of alcohol in 100 millilitres of blood then the standard deviation for the standard at 80 milligrams of alcohol in 100 millilitres of blood would be used to determine the variability associated with that. Q. Yes. A. And so, you use that factor to multiply by the result to determine what the range would be. So, it would be 100 plus or minus 3 milligrams of alcohol in 100 millilitres of blood. Q. Right. And.... A. So, if you were to repeat that measurement multiple times... Q. Yes. A. ...the likelihood – 95.5 percent of the time, is that that result would be within that range of 97 to 103. Q. Right. So, the data that is used to create – calculate that variability is kept separately at the Centre of Forensic Sciences in Toronto, and it’s kept separately at the northern office of the
Centre of Forensic Sciences in Sudbury, I think it is?
A. Yes. Q. All right.
A. No, sorry, Sault Ste. Marie. Q. Sault Ste. Marie. My apologies. And that data is generated from the audit trail, if I can use that terminology, or historical data, from the control checks that have been run on the previous tests for other subjects, or just plain control checks? Right? A. No. It’s only run on – it’s compared to the standards that are run. Q. Oh. A. Not – the actual numerical value is an average of the four results. Q. I see. Okay. A. So, the result of 100 would be an average of the analysis of two samples that are analysed twice – sorry. Two samples that are analysed once, but there are four different detectors that are used to determine the concentration to make sure that there are no interferences with the analysis of alcohol in that sample. And those four results are then averaged and reported on the analysis. Q. That – that’s the report of.... A. So, the result of the variability is not associated with that, but associated with the standards that are run. Q. All right. The standards that are run at the same time of – of subject testing? A. No, of controls that were previously run the year before. So, it’s a static measurement uncertainty where you use the results of the previous year. You take the average and you take the variability associated with that, and that is used to apply the analysis of the blood sample in this particular case, until the next year. So, we’ve just switched over to the 2016 M-U data, for 2017.
Q. You just used the words ‘measurement uncertainty.’ A. Correct, yes. Q. That’s a reasonable scientific concept in measurement? A. It is, yes. It’s one way of measuring the variability. But it’s a measurement of – the analysis of the standards as opposed to a measurement of the variability associated with the analysis of the subject’s blood sample. Q. Ah. All right. It’s based on.... A. Which is an average of the four analyses that were measured when that analysis was done. Q. It’s based on the control tests over a period of time; the historical control tests. A. Yes. Q. And the number that’s used by the Centre in that calculation is 50. A. Sorry? Q. Fifty. Fifty control tests? A. Um, typically, yes. Q. All right. A. If there’s less than 50 analyses, so, if you had a new method then you would do it based on the number of analyses that were done to that date. Q. So, if you look at 50 control tests, all for the same value of standard, ‘cause you use a different standard for different points. A. Yes, there’s one at 25, one at 80 and one at 300. Q. Right. So – but these are control tests on the instruments that you are using of – for purpose of – purposes of the – of the blood, serum or urine analysis for
alcohol. A. Yes. So, every time I do an alcohol analysis, I run a series of control that bracket the samples that I analyse. Q. So, those are the control tests that are associated with the analysis you’re doing for each of these subjects. A. Correct. And they apply to this analysis that was done there. Q. All right. A. The control data from the previous year is used to determine the measurement uncertainty. Q. All right. But next year, that control data will be used, the control data that you generated when you were doing your – say, a blood analysis, on Mr. H. The control – that control data will be used in the following year for calculating measurement uncertainty associated with that particular instrument. A. Well.... Q. From the controls. A. With all different instruments that are used. There’s four different instruments that are used and there’s probably 18 different people who do alcohol analysis, so that variability would be associated with the various instruments as well as the people who are doing the analysis, as well as the pipets that are used, right? So, all that uncertainty from each of those different things that are used in the method go in to create the measurement uncertainty value. Q. All right, the point is you use historical data to generate that calculation of measurement uncertainty. A. That's correct.
Q. And that historical data relates to the control tests. A. Yes, from the previous year. Not the ones.... Q. And so, what – what – the steps that are taken in calculating that measurement uncertainty would be to take 50 control tests, total them, get an average. First step? A. Yes. Q. And then take that average and look at each of the individual – see how much control test number 3, what it’s – how much its result varies from the average. I just – I want to do what I can to try and explain standard deviation as best I can. And I’m going to need your help along the way, but basically, standard deviation is a kind of an average of the deviations of the control test results from their average. It’s an average of an average. An average of deviations. A. Yes. Q. And it’s always in a positive number, not a negative number... A. Well, it could be... Q. ...because deviations.... A. ...plus or minus. Q. Yeah, deviations can go either way. A. Correct. Q. That’s the whole point of them being standard deviations but in order to get that number, to come up with that number, for a standard deviation, you take an average of those? A. Yes. Q. All right. So, what the individual – and in fact, this appears to be exactly what is done when – this
is the same kind of methodology that’s used during the.... A. Page 38. Q. Page – oh. Well, page 32.
A. Thirty-two? Q. Start at page 32. All right, so at page 32. I’m going to let you borrow my page 32 for just a moment. I’ve got a piece of paper where we can actually do the math. What date was that? A. June 20th, 2012. Q. So, the methodology that’s used is we take each of the... A. Results. Q. ...June 20th, 2012 – we take each of the results, which are – if you just quickly compare the numbers here, the cal-checks of 97. A. Ninety-five, ninety-six, ninety-six, ninety-five, ninety-six, ninety-six, ninety-six, ninety-five and ninety-five. Q. All right. So, you total them up. That’s the 975. Divide by 10. A. Yes. Q. You get an average of 95.7. A. Correct. Q. Right? And then to calculate standard deviation what you do is, you look at the difference between each individual value and the average, and you get numbers like are in the second row under the heading of “Difference”. A. Correct. Q. And then we’ve got to turn those, essentially, into a positive number rather than a negative number and what we do is, we – we square those and then for some strange reason, and I’ve never understood this, instead
of dividing by 10 you divide by 9. You always divide by a number that is one less than the total of the squares. Is that – so in this case, I divided by 9 and got 4.1. And I think that’s consistent with the way that it’s done in the calculation on page 32 down by the machine. A. No, it isn’t. Q. No? A. So, that’s these numbers here? Q. Standard deviation of 0.6749. So, you – you take – you square them. A. Oh, yes, sorry, yes. Q. You square them. A. Yes. Q. Get an average of that square.
A. Yes. Q. Divide by 9. A. Yes. Q. And then you take that number and take what I think is the square root of that number, and that’s the 0.67. A. That’s right here. Yes. Q. All right. Right. MR. BISS: For purposes of the record Your Honour, in a minute what I’ll do is, I’ll introduce a paper that explains how to do this, so it’s easy. Q. But in any event event, that’s how you calculate....
THE COURT: I can hardly wait for that. I’ve got to tell you.
MR. BISS: Q. That’s how you calculate standard deviation. This is the methodology that’s used by [the local] Police, and they’re obviously using the stability function – the stability test...
A. Well, the....
Q. ...function on the – on the instrument.
A. The standard deviation’s calculated by the instrument.
Q. Yes.
A. From the data. It does – it’s all automatically.
Q. Right.
A. I haven’t had to calculate it this way since probably second year university.
Q. All right.
A. Because most spread sheets, you can use load – Excel. It’ll calculate it without having to know that you have to square it and then divide by this and do this. Right? Q. Right. All right, now.... A. So, I’m getting a refresher here. Q. All right. You see, one of the problems, I think, with calculation of standard deviation and calculation of precision is it depends on how many results you use, because you don’t know, do you divide by 9, do you divide by 49. Do you – again, your example at the Centre of Forensic Sciences, when you’re doing testing, when the measurement uncertainty calculation was done, what they do, is if they’ve got 50 historical records of control checks the division would be, I guess, by 49. They take that number, which gives them a standard deviation and then they multiply by the confidence interval level. Right? A. Yes. Q. Okay. And roughly, the confidence interval level for 95 percent probability is 2.
A. 95.5, yes.
Q. For 95.5 it’s 2.
A. Yes.
Q. Because it’s two standard...
A. Deviations.
Q. ...deviations.
A. Yeah.
Q. So, the problem that I see is that I think that defence lawyers are going to have to, in the future, in making arguments and talking about reliability, are going to have to talk about drift in accuracy and precision. So, if we have to make arguments about drift in accuracy and precision,we need data with which to do it. And I just want to suggest to you, that a methodology that might be used by defence lawyers in making that kind of submissions would be to follow exactly the same approach. We don’t need to calculate measurement uncertainty. Let’s stay out of that. Leave that one...
A. All right.
Q. ...for the [indiscernible]. But what we can do, is we can look at the control tests, the historical data of the control tests, just like the Centre of Forensic Sciences does when you do a blood analysis, and once we’ve got that information, and we look at – can look at 50 tests, we can take those 50 tests. We can average them. We can then calculate a standard deviation using the methodology that we just talked about.
A. Yes.
Q. And then we can see how that standard deviation compares with the expected standard deviation for the instrument and especially how it compares with standard deviation that’s reported by the local police service.
A. You could do it that way, yes.