This is a sample cross-examination of an Ontario CFS scientist on the procedure used to perform an initial factory calibration (or a subsequent re-calibration by the factory) of an individual Intoxilyzer 8000C.
Purposes of this cross-examination:
[respecting type approval by the Alcohol Test Committee and Minister of Justice]
To confirm that when new instruments are being evaluated, various solutions are used between 50 and 400 mg/100mls.
To confirm that when new instruments are being evaluated, the specified accuracy requirement is +/- 5 percent of the BAC with a precision requirement of +/- 3 mg/100mls under 100 mg/100mls and +/- 2.5 percent for readings greater than 100.
To confirm that the purpose of such evaluation is to ensure the accuracy and reliability of breath tests.
This process is "type approval" in the international literature.
[respecting manufacturer's advertising as to accuracy, precision, and reliability]
To obtain an admission that the manufacturer advertises that its instrument is accurate plus or minus 3 milligrams per 100 mls and standard deviation doesn’t exceed 3 milligrams per 100 mls.
[respecting initial verification of a new individual instrument]
To obtain an admission that under the ATC Best Practices document, new instruments must be individually inspected before being placed into service to ensure that they initially meet the manufacturer's specifications.
This process is "initial verification" in the international literature.
[respecting annual (subsequent) evaluation of an indiviual instrument out in the field]
To obtain an admission that under the ATC Best Practices document, instruments in service must be individually inspected, periodically, to ensure that they continue to meet the manufacturer's specifications.
This process is "subsequent verification" in the international literature.
[what does "manufacturer's specifications" mean?]
To obtain admissions as to the manufacturer's specifications for accuracy and precision:
in its advertising,
in the information given to the evaluator during type approval, and
in the servicing/maintenance documents given to the people who do work on these instruments on a regular basis.
[What are police services actually doing and not doing in Ontario?]
To obtain an admission that although the local police service appears to be doing annual and periodic inspections, they are doing so without reference to the manufacturer's specifications for accuracy and precision across the measuring interval of the instrument.
[What do "accuracy", "precision", "specificity", and "reliability" mean in science and in Canadian law? The following quote is from a paper written by Brian Hodgson of the ATC, presented to the Canadian House of Commons Justice Committee pre Bill C-2. This paper was relied upon by the SCC in R. v. St-Onge Lamoureux]
[Cross-examination of CFS scientist begins]
Q. I–well, can you just take a look for a moment at a paper by Brian Hodgson after the area that we looked at, page 88, this is Tab 2 of Exhibit 2, when new instruments are being evaluated, various solutions are used between 50 and 400 milligrams per 100 mils. That’s at page 88, right? A. Yes. Q. And actually, that’s where the reference is to the specified accuracy requirement that the Alcohol Test Committee is checking, is plus or minus 5 percent of the blood alcohol concentration with a precision requirement of plus or minus 3 milligrams per 100 mils for readings of 100 or less, and plus or minus 2.5 percent for readings greater than 100 milligrams.
A. That's correct. And this testing is done in a laboratory under controlled conditions to ensure the accuracy and reliability of the breath tests.
[He is referring to the "type approval" evaluation prior to ATC and Minister of Justice approval.]
Q. Yes.
A. But when breath testing is done by police services out in the field, the variability is always going to be larger, because there’s so many controls or so many parameters that aren’t controlled for, or can’t be controlled for like they can in a laboratory, where it can be precisely controlled, and therefore get an accurate idea of the reliability and accuracy of the results.
And a breath truck would be an even more difficult place to get results that are consistent over time because of the simple parameter that it’s a moving environment and things can change from time to time, very rapidly, depending on whether the door opens and the temperature outside and inside.
So laboratory studies, such as the one done by the Centre of Forensic Sciences as you said, and the R.C.M.P. would be done under controlled conditions, whereas the testing that’s been done by police officers here, is done in the field.
[So separate the concepts of CFS/RCMP/ATC laboratory evaluation of a sample instrument during "type approval" v. actual police use in the field of an individual instrument. In the field conditions are variable, particularly in a breath truck. Quaere: What makes the individual instrument in the field as "accurate", "precise", "specific", and "reliable" as the evaluated sample instrument]
Q. But the manufacturer advertises that its instrument is accurate plus or minus 3 milligrams per 100 mils and standard deviation doesn’t exceed 3 milligrams per 100 mils.
[Note that the manufacturer doesn't say that only the laboratory evaluated sample instrument will have this accuracy.]
A. That's correct. And that was tested by the A-T-C and the R.C.M.P. and that was determined to be the case. But you’re always going to have more variability associated with any test done out in the field by police officers who are not scientists.
[The defence concedes that any instrument tested by police in the field will have variability, but don't the manufacturer's specifications apply to the evaluated sample instrument AND the instruments in the field?]
Q. So, that’s what was required for type approval.
A. Yes.
Q. But how do we know that [the capabilities of] an individual instrument with a particular serial number, fits that [evaluated sample instrument] model that received the type approval?
Isn’t that what the initial inspection ["to ensure that they initially meet"] that's done of any instrument, and that we read about in the best practices document does?
A. Sorry. One more time?
Q. Well, let’s just take a look at the "Best Practices" document for a moment.
A. Excuse me, could I get some more water please? Thank you.
Q. That’s Exhibit 11 under – Exhibit 11, page11. Under ‘inspection’. All approved instruments, that would imply each – all individual approved instruments. Is that the way you read that document?
A. Correct. Yes. Thank you.
Q. Shall be individually inspected before being placed into service and periodically thereafter, to ensure that they initially meet and continue to meet the manufacturer’s specifications. Right?
A. That's correct.
Q. And those specifications are plus or minus 3, for purposes of accuracy, and plus or minus 3 for purposes of standard deviation. Right?
A. Well the plus or minus 3 percent or 3 milligrams is part of their advertising about how accurate the instrument is, as opposed to a specification that has to be met every single time.
[Note: Separate the concepts of ATC/CFS/320.31(1) +/- 10 mg/100 mls at time of use v. manufacturer's advertising v. "manufacturer's specifications" as described in the ATC "Best Practices" document.]
Q. Well, I’m not suggesting that that’s in the advertising brochure. I’m suggesting that’s in the documentation from the manufacturer as to specifications that’s given to the people who do work on these instruments on a regular basis. Those are the manufacturer’s specifications for the instrument of accuracy and precision. A. Can you show that to me one more time? Q. Sure.
A. I remember seeing it earlier this afternoon, and.... Q. That would be Exhibit 25 is one place. It’s page 1 in the area under specifications. A. Yes. And was there another document that had a – a black and white document? Q. No, that’s – that was a brochure that talked... A. Yes. Q. ...about the range. The range being – the advertising brochure that you’re speaking about mentioned range but it did not mention accuracy and precision. That would be – I think it was probably Exhibit 24. MR. O: I just wonder if the witness could be provided that one, Your Honour. THE COURT: The advertising brochure? MR. BISS: I’m just getting a copy. THE COURT: Exhibit 24. MR. BISS: In fact, I’ve got an extra copy. A. Thank you. All right, sorry. I thought there was something in here as well. Q. You can keep that. A. Okay, thank you.
[Exhibit 24: Manufacturer's advertising brochure re the 8000C:]
Q. So, that document, Exhibit Number 24, specifies range, measuring interval. Exhibit 25 is the document that has manufacturer’s specifications.
And also on one of our previous days, if I could just have the Court’s indulgence for a moment. Exhibit 10, which is an excerpt from the Evaluation of the Intoxilyzer 8000C, done by the Centre of Forensic Sciences, Toronto, Ontario. I think that’s Terry Martin’s assessment that went to the Alcohol Test Committee, and there’s a page – in this excerpt it’s the last page. Does Ms. Martin in her report mention the manufacturer’s stated accuracy and precision for the instrument?
A. So, on – it doesn’t say what page. One, two, three, four. On page 4 of 23, the last page, point number 3, performance data relating to the appropriate A-T-C standards. No specific performance data were provided by the manufacturer, however, the accuracy of the instrument is stated to be plus or minus 3 percent or .003 grams per 100millilitres, whichever is higher, and the standard deviation is listed as being 0.003 grams per 100 millilitres or better.
Q. All right, so that’s consistent with the specification sheet that I was showing you in Exhibit – I think it was 25. Right?
A. I know the – the page of that says specifications, but it’s actually performance data as opposed to the actual specifications that you must get the results within this specific value range. That would be my interpretation of that.
Q. I see. Well, I gather that the manufacturer must have told the evaluator, who is Ms. Martin, provided that information based on...
MR. O: Well, how....
MR. BISS: ...that document.
MR. O: I’m not sure how Mr. P
can answer that question. MR. BISS: Q. Any ideas where else this information could’ve come from that’s in the Alcohol Test Committee evaluation? Ah, it’s actually an evaluation by the Centre of Forensic Sciences for the Alcohol Test Committee, is my understanding. A. You would have to ask Terry Martin where she got that information from but I would assume it would’ve come from the manufacturer. Q. All right. A. In some form or another. Q. So, if police.... A. It could even be the same information that was on that document there. Q. Right. So, if police are complying with best practices document, of the Alcohol Test Committee, whether it’s the current one or whether it’s the older one, in each case inspections are supposed to be conducted periodically in relationship to then whether or not the instrument – the individual instrument initially complies with and ultimately whether it complies with the manufacturer’s specifications, right? A. That’s the language, yes.
[So police services have an obligation, that comes to them from the ATC, to initially inspect and subsequently inspect individual instruments, out in the field, to endure that each individual instrument complies with the "manufacturer's specifications."]
[In Canada, this is how "type approval" of an "approved instrument" links up with "initial verification" of a new instrument and "subsequent verification" of an instrument in the field.]
Q. And it looks like in this case [before the Court] there are inspections that are taking place, but they’re not being done – it appears – in reference to the manufacturer’s specifications. A. Sorry, the language you used again? Q. Well, it appears from Exhibit Number 12, the document that has all the periodic inspections... A. Yes. Q. ...by York Regional Police, it appears that
York Regional Police is doing annual inspections. A. Yes. Q. But it does not appear that they are doing those annual inspections with any reference to the manufacturer’s specifications for the instrument. A. The data from the specifications, yes.
[End of cross-examination of CFS expert.]
[Quaere: Are annual inspections not being done by this police service matching the individual instrument's performance against the manufacturer's specifications? Is the particular police service not complying with the ATC "Best Practices"document. Is there is a systemic problem with this police service?]
[The defence lawyer should consider calling evidence from an expert in metrology. The metrologist's evidence should include the concepts of "metrological control" and "metrological supervision, both of which appear to be in short supply in Ontario BAC testing.]
[Post Bill C-46, the defence lawyer should consider launching a section 8 Charter challenge, arguing a sytemic constitutional breach resulting in instruments which we normally would consider to be reliable, being unreliable.]