Ignoring Inconvenient Data in Calculation of Precision
Updated: Aug 3, 2022
Purposes of this cross-examination:
To challenge the opinion that data outside maximum permissible error (outside +/- 10 mg%) should be excluded from any calculation of precision.
The cross-examiner should have called evidence in reply from a statistician or other expert in outliers.
Note: The cross-examiner and the Crown expert had vastly different perceptions of what data could be used to calculate precision/standard deviation. The defence was suggesting that if you use all the last 50 cal. checks for purposes of calculating standard deviation, then the standard deviation was well outside of the manufacturer's specification of 3.0. The expert took the position that some of the data should be excluded for purposes of the calculation and his calculated standard deviation was much less than 3.0.
Q. All right, and you would not use those because you would consider them to perhaps be outliers. A. No. They’re not outliers, they’re just not acceptable. If these results – so the results of 89, 89, 83, 86, 86, 87, 89, 83, 87, 89 and 89... Q. Yes. A. ...that are in the top part of this calculation – spreadsheet here, are outside the acceptable range, and therefore, no tests would have actually proceeded had these actually been part of a subject test. So, this indicates to me that there was a problem and that the breath tech was trying to troubleshoot that problem to get a result that was within the acceptable range. And so therefore, you wouldn’t use them in the calculation of a number like this because you’re using results that are obtained that have difficulty associated with them, compared to the ones where the tests actually proceeded and were acceptable. So, if you – if this was a 5000C and a result of 89 had been obtained, which it could happen on the 5000C but not on the 8000C and a numerical result was obtained for a subject test that test would be considered invalid. Q. Right. A. And so, you don’t use these values because they’re unacceptable and testing wouldn’t have proceeded. Q. But as a scientist....
A. But they’re also a part of the issue is that they are unacceptable because they have – they’re outside the acceptable range and there’s troubleshooting associated with these. So, the optimum scientific parameters for doing this kind of calculation have not been maintained here. Q. So, we need to find out the reason why they’re outside the acceptable range to see whether we should use them as part of precision or not. A. No. That would not...
Q. We need to see....
A. ...be my opinion.
Q. We need to see, and it appears then, that you differ from Mr. Kupferschmidt in that respect.
A. That's correct. Q. Mr. Kupferschmidt is suggesting that when it comes to control checks that are outside the acceptable limit, including these ones...
A. Yes. Q. ...that if we are going to try and look at what’s happened to this instrument in terms of drift in accuracy and precision, which is our thesis, then you need to have information about those results that don’t fit what we want to see. In other words, they’re – they’re unacceptable to you but my suggestion is, these are all control tests that are done by a qualified technician on this instrument using an instrument – using a simulator that’s running at the correct temperature, by a qualified technician. So, yes, on the face of it, there’s something wrong with it. But what we need to do is we need to find out why there’s something wrong with them, so that we can assess which ones should be used and which ones should not be used. That’s my suggestion. A. And I appreciate your suggestion. These
are not an accurate reflection of the operating of the instrument at that time. There is an issue here, and obviously, something’s happened where the breath tech has gotten these results, and they’re having to get the instrument into proper working order. And to do that, they have to get the result at least above 90 and they haven’t done that in this case. And so because these are associated with being beyond the lower end of the accepted range, and I would apply the same thing to any results that were greater than 110, that they would be excluded from that, and you would have to find yourself – 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 other points that were within the acceptable range and use those in your calculation, and I think you would find that your average and your standard deviation would drop significantly – or, your average would go up but your standard deviation would decrease significantly. Q. And I always thought those were called outliers.
A. Outliers are outside the average and the minimum and the maximum. So, outliers are bad data. Q. Well, isn’t that exactly what this is? What you’re – the way you’re describing these are outliers. A. So, you don’t use outliers. Q. Right. But you do investigate whether they’re outliers or not and you go back and you check the documentation to find out whether you should treat them as outliers or not treat them as outliers, for purposes of your calculation. A. In this case, these tests have no bearing on the tests that were done in this case because there’s no way to relate what happened here to the tests that were done in this case.
...
MR. BISS: Q. So, Mr. P, your approach would be to basically throw away 20 percent of the data, 10 out of the 50. A. Eleven.
Q. Eleven.
A. That's correct.
Q. So more than 20 percent of the data. And you are proposing that this should
be done without knowing the reasons why this data seems to be different from the rest of
the data.
Q. Yes.
A. And then quickly do the calculation.
Q. Yes.
A. All right? So, what I had her do, her name is Rachel Wallage. Q. Yes. the data. or.... or a mean. A. Just for – do you want a phone number Q. No. No.
A. Okay. So, I read out the numbers to her.
Q. Yes.
A. Minus the outliers.
Q. Yes.
A. Those 11 outliers, to calculate an average Q. Yes.
A. Which are the same thing. And the mean is 95.05128 if you want to go to that many decimal places. Q. Yes. A. The standard deviation is 1.276287. And that’s more than enough significant figures. So, that’s with the 39 data points. Q. Yes. A. All right? Now I asked her to do the same thing again, using those 39 data points and I added 11 more points to keep the number to 50.
Q. Yes.
A. So, I went back from page 37 and went back page 31 to find 11 data points...
Q. Yes.
A. ...in chronological order to add to bring the total number to 50.
Q. Yes.
A. And the mean or average is 94.52.
Q. Yes.
A. And the standard deviation is 2.052773.
Q. Two point zero....
A. Five two seven seven three. And again, from going backwards from page 37 to page 31, there is a calibration check that was 88 which I did not include. Q. All right. You – you....
A. That was the only unacceptable one going back. Q. You would never include a calibration check of 88 in calculating a standard deviation in trying to determine precision of an Intoxilyzer 8000C.
A. Correct. And I wouldn’t use a zero either. So, if you had a calibration check that was a zero I wouldn’t use that.
Comment: Isn't there a major problem with precision of the instrument if the indications on the instrument are drifting wildly?
Isn't there a requirement that we at least explore whether or not the cause of the wild fluctuations is trivial or non-trivial?
Maybe there is a simple explanation such as bad control tests on a particular date because of a leaky or malfunctioning wet-bath simulator. But isn't that also a cause for concern about reliability?
Please remember that under St-Onge the issue is reliability not causation of the results.
Quaere: If one always excludes data outside 10 mg% is it ever possible mathematically to have a standard deviation greater than 3 on an approved instrument?