top of page

Manipulating Control Test Data to Enhance Precision

Updated: Oct 5, 2022


To neutralize the government scientist's assertion that anomalous data should be excluded from calculation of precision.

To obtain an acknowledgment of drift in precision over time (see Hodgson definition of reliability)

To challenge the propriety of throwing out control test data outside 90 - 100 mg/100mls in any calculation of precision

This cross-examination shows the need for defence counsel to retain experts in statistics AND metrology.

This cross-examination shows the need for defence counsel to carefully review the study that the Crown expert is relying upon.

A. Just one thing before we get started. Let me

get organized here. On the previous day I spoke about a

paper. This was with respect to the data you referred to me

from South Simcoe Police Service.

Q. Yes.

Abstract from paper by Rong-Jen Hwang, Craig Rogers, Jada Beltran, Gerasimos Razatos, and Jason Avery, referred to by the witness
Excerpt from Abstract from paper by Rong-Jen Hwang, Craig Rogers, Jada Beltran, Gerasimos Razatos, and Jason Avery

A. Where the first calibration check was 88 and

then the subsequent ones were acceptable, and that it was a

phenomenon where we know that to happen with respect to

sometimes the first calibration check can be low, and that

wouldn't be improper to exclude that data point from the

calculation of the accuracy and precision of the instrument.

And I referred to a paper by Dubowski, and, in fact, it

turns out it's actually a paper by a gentleman of the name of

Hwang, H-W-A-N-G, et al, co-authors. And it's the Journal of

Analytical Toxicology, published in 2016, volume 40, page 338

to 344. And in that paper they refer to the fact that they

do a series of accuracy and precision testing that we talked

about, and that was shown in that case with respect to the

South Simcoe Police, and that they exclude the first point

because of they're aware of the limitations of the first

sample often being unacceptable due to

non-equaliberlation [sic] -- equilibration of the head space,

so they rely on nine data points, as opposed to ten.

Q. But, of course, they would never exclude

number two and number three and number four.

A. Correct.

Q. In other words, a significant proportion of

the data in any calculation and precision.

A. Yes.

Q. So if you're dealing with 50 points, it makes

no sense in doing a calculation of precision to exclude the

first 11, or 11 out of those 50?

A. Well, in this case... if I can find the data.

It wasn't the first 11 data points, but it was all the 11

data points that were below the low end of the acceptable

range of 90 milligrams of alcohol in 100 millilitres of


Calculation of standard deviation
Calculation of standard deviation by the cross-examiner using all of the last 50 cal. checks prior to subject tests, wet-bath temperatures in COBRA all 34.0 +/- .2 C.

Q. Right. But on the last day you talked about

the phenomenon that during training you inform the officers,

the new qualified technicians or the recertifying --

requalified technicians, that sometimes you have a problem

with the first cal check being low because...

A. Yes.

Q. ...the system needs to warm up. Maybe they

haven't waited long enough for the whole system to -- is

"equilibrate" the right word?

A. Yes.

Q. To stabilize. To reach an equilibrium of

temperature so that Henry's law applies?

A. Correct.

Q. So that's with respect to one test, not eight,

nine, ten, eleven of them?

A. Well, what we were doing here with respect to

this data that you are referring to is different than what is

done during an accuracy and precision test with respect to

the Intoxilyzer 8000C.1. Here we took field data that had been

obtained at various times, various calibration checks, and so

as I explained earlier, that had any of these been part of an

actual breath test, breath testing wouldn't have been able to

proceed, and that the results are outside of the range that's

considered acceptable of 90 to 110 milligrams of alcohol in

100 millilitres of blood, and that's why they would be

excluded, because they're not representative of the range

that is accepted to be in.

Q. I see. But there's still control tests, and

we've got no indication, without looking at the accompanying

documentation, there was anything not functioning in the

simulator. There are control tests. We assume, without

digging further and actually seeing the officer's notes,

there are control tests that would appear to be with a proper

alcohol standard at the temperature shown in the data. The

temperature is there of 34 degrees plus or minus 0.2 degrees

Celsius. So why shouldn't there be at least an indication of

the precision of the instrument, to look at the last 50 cal

checks prior to my client's tests?

A. Because they are outside that acceptable

range. There's obviously some other issue that's going on

during those particular standalone calibration checks that we

don't know anything about, and because it's aberrant data,

you don't include it in the actual calculation.

Q. But the issue may be that the instrument is

producing consistently low calibration checks.

A. Well...

Q. And this is an indication that --

A. We know that -- sorry. Sorry. Go ahead.

Q. This is, this is an indication. That change

in precision is an indication to us. I mean, subject to

review of the documentation by the officer who was dealing

with it on that day, this is an indication of consistently

low cal checks.

A. And that's what the data that we went through

last time appears to show. But the instrument can't proceed

with testing. So in this case -- again, I don't remember how

many of these are associated with one particular case, but,

again, it speaks to a time other than when the tests in this

case were done. It speaks to another time in question that

you can't relate to the tests done in this case. And

normally, as I said, probably 99 percent of the time you get

a calibration check standalone and everything is fine,

diagnostics are fine and a self-test that's fine, and breath

testing proceeds to get two tests that are in good agreement

and game over. But occasionally there will be difficulties

associated with the breath tech trying to get the instrument

ready for the purposes of conducting breath tests. And as

we've talked about consistently, that sometimes that first

calibration check is below the acceptable range, in which

case something has to be done to try to get it back into the

acceptable range, whatever that might be. Sometimes it might

take more than one calibration check to get the instrument

into proper working order. It may take some additional

effort on the part of the qualified breath tech to get that

instrument ready and get the calibration check into the

acceptable range to proceed with testing, because if they

can't, then they're supposed to find another instrument.

Q. They're supposed to find another instrument


A. Yes.

Q. would then, at that point in time,

appear that the accuracy and precision of the instrument may

be jeopardized?

A. Well, there could be any number of

explanations why the data is low. That's one possible

reason, yes.

Q. And so that's why we should be looking at the

contemporaneous documentation [see footnote 2. below]. that went with those control


A. I would disagree with that. Again, that

material speaks to times other than when the test in question

here was done, and you can't relate that information to the

tests that were done in this case in any way -

mathematically, statistically or otherwise.

Q. Well, we've just talked about a method

statistically of relating it to the time of the subject test

in a calculation of precision. I mean, either those control

tests are relevant to a calculation of precision or they are

not, right?

A. That is correct. In this case those 11 points

are outside the acceptable range, and, therefore, would be

excluded from any kind of accuracy and precision calculation.

Q. But if we're trying to determine the why of

why it happened, especially when they aren't all that far off

from the low 90s that we're seeing in some of the other

calibration checks, the only way to make that determination,

to give a proper scientific opinion, is to look at the

contemporaneous documentation.2.

A. You need to look at that information for the

purposes of the breath test that was done in that particular

case. It's irrelevant to the test that was done in this



The cross-examiner should have gone further and once again connected reliability with drift in accuracy and precision over time. An indication of drift in precision is therefore relevant to an assessment of reliability. A review of the contemporaneous documentation is therefore useful in studying whether or not there is a real drift in precision or rather some other cause for the anomalies.

1. Maybe the bigger problem is that no one is doing careful accuracy and precision tests (in an ISO 17025 certified lab) of approved instruments out in the field, including this particular instrument. Here we have at least an indication of a problem of precision. The instrument results are wandering. That problem requires disclosure of all accuracy and precision testing and contemporaneous documentation of anomalies.

2. See Motherisk Inquiry Report on Contemporaneous Documentation and ISO 17025.

6 views0 comments
bottom of page