WEBVTT 00:00:08.066 --> 00:00:08.866 Hi, my name is 00:00:08.866 --> 00:00:11.966 Lauren Pearson and I am a laboratory director 00:00:11.966 --> 00:00:16.766 for University of Utah Health and AARP laboratories. 00:00:17.466 --> 00:00:21.100 It's my pleasure to present my thoughts 00:00:21.100 --> 00:00:25.466 about method validation and verification to you today. 00:00:26.500 --> 00:00:29.533 We'll go through some of the basics 00:00:29.866 --> 00:00:33.066 and apply the concepts that we learned 00:00:33.166 --> 00:00:36.533 as we go with some interesting laboratory challenges. 00:00:38.533 --> 00:00:41.400 So the objectives of today's 00:00:41.400 --> 00:00:44.333 didactic are the following. 00:00:44.333 --> 00:00:49.300 To identify the difference between method validation and method verification. 00:00:50.100 --> 00:00:54.333 To describe the studies required to document method performance 00:00:55.133 --> 00:00:58.133 and to interpret method performance data. 00:01:00.800 --> 00:01:01.600 So we'll start 00:01:01.600 --> 00:01:06.633 with some context and definitions, as well as an overview 00:01:06.633 --> 00:01:09.900 of the regulatory requirements in this area. 00:01:10.766 --> 00:01:14.866 As we'll spend the bulk of our time talking about the different studies 00:01:15.000 --> 00:01:20.133 required for assay verification or validation. 00:01:20.700 --> 00:01:23.400 So right off the bat let's talk about the difference 00:01:23.400 --> 00:01:26.933 between the terms validation and verification. 00:01:27.433 --> 00:01:31.166 Often in the laboratory we use these terms interchangeably. 00:01:32.033 --> 00:01:36.966 But if you go back to the books, validation is really a process 00:01:36.966 --> 00:01:42.166 by which you establish the performance specifications of a new diagnostic tool. 00:01:43.100 --> 00:01:45.266 So that could be a new instrument. 00:01:45.266 --> 00:01:47.833 It could be a new assay. 00:01:47.833 --> 00:01:52.266 It could be a laboratory developed test that you're inventing 00:01:52.800 --> 00:01:55.566 something that's being created for the first time. 00:01:55.566 --> 00:02:00.600 Or it could be a diagnostic tool that you have modified in some way. 00:02:01.833 --> 00:02:04.833 Contrast that with verification, 00:02:05.233 --> 00:02:08.233 which is really a one time process to determine 00:02:09.366 --> 00:02:12.100 how well a tool performs 00:02:12.100 --> 00:02:17.400 before it gets used for patient testing with verification. 00:02:17.533 --> 00:02:22.566 Typically, what laboratories are doing is they're taking an FDA 00:02:22.566 --> 00:02:26.000 cleared or FDA approved diagnostic tool 00:02:26.633 --> 00:02:29.933 and they're performing some initial studies on it 00:02:30.333 --> 00:02:33.666 just to verify that what they've been told by 00:02:33.666 --> 00:02:36.666 the manufacturer is true. 00:02:36.900 --> 00:02:39.900 Before testing patient samples. 00:02:40.200 --> 00:02:44.266 In general, the process of completing a full validation, 00:02:44.733 --> 00:02:48.766 which is typically what an instrument or assay manufacturer does 00:02:49.266 --> 00:02:51.833 before marketing a device, is 00:02:51.833 --> 00:02:54.833 much, much more labor intensive. 00:02:54.900 --> 00:02:57.900 There are more studies required 00:02:57.966 --> 00:03:01.333 and we will get into the differences as we go. 00:03:02.766 --> 00:03:06.466 Once we get into the heart of the various studies 00:03:06.466 --> 00:03:10.033 that are required to document method performance. 00:03:12.300 --> 00:03:13.100 So why do we 00:03:13.100 --> 00:03:16.100 evaluate a new method or a new tool? 00:03:17.233 --> 00:03:20.233 Well, let's start with the obvious one. 00:03:20.766 --> 00:03:25.566 It's required and we'll get into the details of that in the coming slides. 00:03:26.133 --> 00:03:29.133 We are required by clear 00:03:29.133 --> 00:03:33.333 as well as by other accrediting agencies to document 00:03:33.333 --> 00:03:36.700 method performance before patient testing. 00:03:37.566 --> 00:03:40.833 But there's other reasons to evaluate a method as well. 00:03:41.666 --> 00:03:46.200 So for example, as part of the normal quality assurance 00:03:46.833 --> 00:03:49.466 to ensure high quality results 00:03:49.466 --> 00:03:52.433 are being reported by the laboratory. 00:03:52.433 --> 00:03:56.733 It's also very useful to have some documentation 00:03:56.733 --> 00:03:59.733 about how the tool performed, 00:03:59.966 --> 00:04:03.100 in case you end up having to troubleshoot 00:04:03.100 --> 00:04:06.100 possible analytical errors down the road, 00:04:07.166 --> 00:04:11.333 and you want to go back to your original verification study 00:04:11.666 --> 00:04:15.066 to determine if there has been drift in the performance of 00:04:15.066 --> 00:04:16.633 the assay or the instrument. 00:04:17.766 --> 00:04:18.233 And then 00:04:18.233 --> 00:04:21.233 for clinical consultations, that can be quite helpful. 00:04:22.066 --> 00:04:25.133 So for example, if you get a call from a clinician saying 00:04:26.066 --> 00:04:30.000 yesterday my patient had a sodium result of 153 00:04:30.466 --> 00:04:34.100 and today their result is 154. 00:04:34.233 --> 00:04:37.233 Is this a clinically meaningful difference? 00:04:37.600 --> 00:04:42.433 Your understanding of how the assay or the instrument performs can help 00:04:42.433 --> 00:04:45.633 you answer those types of questions in a meaningful way. 00:04:47.933 --> 00:04:50.933 So let's talk about the laboratory regulations. 00:04:51.666 --> 00:04:55.633 In general, these regulations are very vague 00:04:56.166 --> 00:04:59.166 and they are open to interpretation. 00:04:59.800 --> 00:05:02.466 So they direct what must be done. 00:05:02.466 --> 00:05:05.466 But they don't tell you how to do that. 00:05:05.666 --> 00:05:09.400 And that's because it's hard to generate a one size fits 00:05:09.400 --> 00:05:14.700 all prescriptive approach to evaluating a new diagnostic tool. 00:05:15.500 --> 00:05:19.366 So while it may be a headache for a director, 00:05:19.366 --> 00:05:23.300 because now the onus is on us to decide how to accomplish it. 00:05:24.233 --> 00:05:28.933 It also benefits you as a laboratory, and to have a little bit of flexibility 00:05:29.566 --> 00:05:32.800 into how you're going to meet the requirements, 00:05:33.466 --> 00:05:38.166 as well as in your evaluation of whether a diagnostic tool 00:05:38.166 --> 00:05:42.700 is acceptable and in its performance for patients. 00:05:45.133 --> 00:05:45.566 So let's 00:05:45.566 --> 00:05:49.200 briefly talk about how tests are categorized. 00:05:49.700 --> 00:05:54.066 This may seem a little out of place in the context of the presentation, 00:05:54.066 --> 00:05:57.066 but it's important to understand because 00:05:58.033 --> 00:06:00.400 most of the discussion in this area is going 00:06:00.400 --> 00:06:04.166 to focus on requirements for non waived testing. 00:06:04.733 --> 00:06:07.733 And I just briefly wanted to define 00:06:08.033 --> 00:06:12.833 waived versus non waived testing and talk only for a moment 00:06:12.833 --> 00:06:16.233 about how to meet the requirements for waived testing. 00:06:17.766 --> 00:06:19.966 So test categorization 00:06:19.966 --> 00:06:23.033 or test complexity is another term that could be used 00:06:23.366 --> 00:06:26.633 is determined during premarket approval 00:06:27.766 --> 00:06:29.766 with the FDA 00:06:29.766 --> 00:06:32.066 and waive testing in 00:06:32.066 --> 00:06:35.200 general is testing that is approved 00:06:35.200 --> 00:06:39.333 for either home and or point of care use. 00:06:40.066 --> 00:06:42.000 And in the literature, 00:06:42.000 --> 00:06:47.233 you'll see terminology that waive testing in general has a low 00:06:47.233 --> 00:06:52.333 risk of patient mismanagement if it's even if it's performed incorrectly. 00:06:53.333 --> 00:06:55.400 I personally think 00:06:55.400 --> 00:06:58.533 that there is, some risk of patient 00:06:58.533 --> 00:07:02.266 mismanagement with waived testing, but that's just my opinion. 00:07:02.866 --> 00:07:08.133 Non waive testing encompasses moderate complexity and high complexity testing. 00:07:08.933 --> 00:07:12.000 An example of a high complexity test is a laboratory 00:07:12.000 --> 00:07:16.800 developed test for body fluid chemistry testing. 00:07:16.800 --> 00:07:20.000 For example modified test. 00:07:20.000 --> 00:07:25.300 That last bullet point there, really refers to a situation 00:07:25.300 --> 00:07:28.300 where a laboratory modifies 00:07:28.666 --> 00:07:31.666 an instrument or an assay 00:07:32.400 --> 00:07:35.400 in such a way that it becomes 00:07:35.733 --> 00:07:39.800 higher complexity, according to Clea, 00:07:40.100 --> 00:07:43.133 and would therefore require more thorough 00:07:43.166 --> 00:07:46.333 evaluation before patient testing commences. 00:07:47.533 --> 00:07:50.000 So big picture here. 00:07:50.000 --> 00:07:53.400 Very huge difference in terms of complexity 00:07:53.733 --> 00:07:56.733 between waived and non waived testing. 00:07:59.133 --> 00:08:00.766 As I mentioned, 00:08:00.766 --> 00:08:04.200 the rest of this is going to be very much 00:08:04.200 --> 00:08:07.433 directed at the requirements for non waived testing. 00:08:08.700 --> 00:08:09.966 But briefly 00:08:09.966 --> 00:08:14.300 regarding waived tests, at least according to clear, 00:08:14.400 --> 00:08:18.766 there's only three requirements that laboratories must meet. 00:08:19.966 --> 00:08:24.300 You have to pay a biennial fee for a clear certificate, 00:08:24.833 --> 00:08:27.833 follow the manufacturer's instructions for use 00:08:28.200 --> 00:08:30.800 and allow the lab to be inspected. 00:08:30.800 --> 00:08:33.600 And you'll notice that there's really no method 00:08:33.600 --> 00:08:37.100 evaluation required, according to Clea for waive tests. 00:08:37.833 --> 00:08:40.833 Now, if you're Cap accredited, 00:08:41.400 --> 00:08:44.400 there is a whole checklist of 00:08:44.800 --> 00:08:48.700 requirements that apply to waive testing that must be followed. 00:08:49.233 --> 00:08:52.233 And I would argue that, 00:08:52.566 --> 00:08:54.600 you know, the laboratory 00:08:54.600 --> 00:08:57.600 should go through some process 00:08:58.066 --> 00:09:02.733 of evaluating the method performance of waived tests 00:09:02.733 --> 00:09:07.033 because it's the best and the safest thing to do for patient care. 00:09:07.700 --> 00:09:10.766 So in my laboratory, we always make some effort 00:09:11.833 --> 00:09:16.566 to verify the performance of the tool before we use it on patients. 00:09:16.866 --> 00:09:21.700 Even though Clea may not explicitly require us to do much. 00:09:23.666 --> 00:09:24.533 Okay. 00:09:24.533 --> 00:09:27.533 Switching gears to non waive testing. 00:09:28.100 --> 00:09:32.233 So what does Clea say about method validation in this area? 00:09:32.933 --> 00:09:36.666 Well, Clea tells us what to do depending 00:09:36.666 --> 00:09:39.800 on the categorization 00:09:39.800 --> 00:09:42.800 of the test in question. 00:09:43.566 --> 00:09:48.033 So for example if the test or the assay is moderate 00:09:48.033 --> 00:09:52.800 complexity, Clea says we have to evaluate for things 00:09:53.666 --> 00:09:56.000 accuracy, precision, 00:09:56.000 --> 00:09:59.000 reportable range and rapid reference range. 00:10:00.233 --> 00:10:02.066 And I like this little mnemonic 00:10:02.066 --> 00:10:05.066 that I've indicated on the screen. 00:10:05.733 --> 00:10:08.000 Compare that to the requirements 00:10:08.000 --> 00:10:11.433 for high complexity testing which are more numerous. 00:10:12.000 --> 00:10:15.733 So in addition to the four that are required for moderate complexity, 00:10:16.800 --> 00:10:20.766 you're also required to make an evaluation 00:10:20.766 --> 00:10:25.233 for analytical sensitivity and analytical specificity. 00:10:26.200 --> 00:10:28.766 You'll also want to be sure 00:10:28.766 --> 00:10:31.766 that the calibration and control procedures 00:10:32.733 --> 00:10:35.933 and other performance criteria are considered 00:10:36.166 --> 00:10:40.500 before commencing patient testing for a high complexity assay, 00:10:40.500 --> 00:10:43.500 especially if it's a laboratory developed test. 00:10:45.300 --> 00:10:46.800 So we will 00:10:46.800 --> 00:10:49.600 go through these studies 00:10:49.600 --> 00:10:52.333 in this presentation. 00:10:52.333 --> 00:10:54.800 But remember at a minimum 00:10:54.800 --> 00:10:57.333 precision accuracy recordable 00:10:57.333 --> 00:11:00.333 range and reference range. 00:11:02.000 --> 00:11:03.366 So I mentioned briefly 00:11:03.366 --> 00:11:06.733 before the concept of test modifications. 00:11:07.200 --> 00:11:09.433 Why does it matter. 00:11:09.433 --> 00:11:13.500 The reason it matters is that any time you change 00:11:13.700 --> 00:11:17.766 something about a tool, it could affect the performance of that tool. 00:11:18.666 --> 00:11:23.100 And that's important because it will affect patients potentially. 00:11:24.366 --> 00:11:28.166 So that's why in the requirements 00:11:30.233 --> 00:11:31.633 you are required 00:11:31.633 --> 00:11:36.600 to verify that that change has not had an adverse effect 00:11:37.200 --> 00:11:40.200 on the diagnostic capability of the tool. 00:11:41.466 --> 00:11:43.800 So if I modify 00:11:43.800 --> 00:11:46.166 a moderate complexity, 00:11:46.166 --> 00:11:49.800 FDA approved assay in any meaningful way, 00:11:50.333 --> 00:11:55.433 that tool now is considered a high complexity tool. 00:11:56.000 --> 00:11:59.200 And I need to have the validation studies 00:11:59.200 --> 00:12:03.466 to back it up before testing patient specimens. 00:12:04.333 --> 00:12:08.233 So what are some examples of test modifications? 00:12:09.400 --> 00:12:11.533 There's a very vast list. 00:12:11.533 --> 00:12:14.666 But just to highlight some of the more common ones. 00:12:15.200 --> 00:12:18.600 So for example modifying the assay 00:12:18.966 --> 00:12:21.966 to test a different sample matrix. 00:12:22.800 --> 00:12:28.066 You know, let's say you took a serum assay and now you want to test urine using 00:12:28.066 --> 00:12:32.333 that assay, promoting a different clinical application, 00:12:33.100 --> 00:12:35.766 screening versus diagnosis. 00:12:35.766 --> 00:12:38.166 The type of analysis. 00:12:38.166 --> 00:12:41.833 So if you took something that was meant to be qualitative 00:12:41.833 --> 00:12:45.433 and you started to quantitate something, 00:12:46.200 --> 00:12:49.500 you changed an incubation time, a temperature, 00:12:50.733 --> 00:12:53.733 step in the standard operating procedure, 00:12:54.933 --> 00:12:57.933 diluting samples, reagents 00:12:58.366 --> 00:13:01.866 using a different calibration, etc. 00:13:02.100 --> 00:13:05.266 these are all things that one should 00:13:05.833 --> 00:13:08.833 verify aren't going to adversely affect 00:13:09.133 --> 00:13:12.133 the diagnostic performance of the tool. 00:13:15.900 --> 00:13:18.900 So here's the meat of what I've been 00:13:19.400 --> 00:13:21.300 saying thus far. 00:13:21.300 --> 00:13:24.300 Basically, laboratories are required 00:13:24.600 --> 00:13:27.600 to perform analytical validation. 00:13:27.900 --> 00:13:31.400 Verification of each non waived test method 00:13:31.400 --> 00:13:35.400 or instrument before beginning patient testing. 00:13:36.833 --> 00:13:37.633 And this does 00:13:37.633 --> 00:13:41.400 include any loaner or temporary instruments. 00:13:41.400 --> 00:13:46.900 Let's say your device breaks and the manufacturer sends you a loaner 00:13:46.900 --> 00:13:48.566 that you're going to have for a couple weeks 00:13:48.566 --> 00:13:53.000 while they figure out what's wrong with your device. 00:13:53.533 --> 00:13:56.700 You have to verify 00:13:57.433 --> 00:14:00.600 the performance of that loaner before you can test 00:14:00.600 --> 00:14:03.600 patients using it. 00:14:03.833 --> 00:14:08.266 And it's really important to retain records of everything that you do. 00:14:08.333 --> 00:14:11.333 That is also a clear requirement. 00:14:11.866 --> 00:14:14.866 So just wanted to point that out. 00:14:16.766 --> 00:14:19.300 So how do we meet the regulations. 00:14:19.300 --> 00:14:22.000 As I mentioned earlier 00:14:22.000 --> 00:14:23.566 they're not very prescriptive. 00:14:23.566 --> 00:14:25.333 How to do that. 00:14:25.333 --> 00:14:29.400 There is no single correct way to accomplish this stuff. 00:14:30.033 --> 00:14:32.700 But as a medical director, I often 00:14:32.700 --> 00:14:35.700 look to consensus protocols 00:14:36.700 --> 00:14:38.866 to the literature. 00:14:38.866 --> 00:14:41.000 It's great when other laboratories 00:14:41.000 --> 00:14:44.666 publish their verification or validation studies. 00:14:45.333 --> 00:14:49.866 I always enlist, the recommendations of the manufacturer. 00:14:51.666 --> 00:14:53.866 Ultimately, it is a balance 00:14:53.866 --> 00:14:59.266 between cost and other operational considerations. 00:14:59.800 --> 00:15:03.633 The amount of time and resources that your laboratory 00:15:03.633 --> 00:15:07.233 has to invest in the process. 00:15:09.700 --> 00:15:13.100 So we're going to dive right into the various studies 00:15:13.100 --> 00:15:16.100 that are required to document method performance. 00:15:16.733 --> 00:15:19.000 And I'm going to teach you, 00:15:19.000 --> 00:15:22.500 you know, best practice based on consensus. 00:15:23.133 --> 00:15:26.766 And I'm also going to show you an example 00:15:27.866 --> 00:15:30.633 of an assay that we 00:15:30.633 --> 00:15:34.600 did a verification study for in my laboratory and ended up 00:15:34.600 --> 00:15:37.833 publishing in one of the laboratory medicine journals. 00:15:39.166 --> 00:15:42.166 That nicely illustrates some of these concepts. 00:15:43.133 --> 00:15:46.133 So let's start with accuracy. 00:15:46.400 --> 00:15:46.633 Right. 00:15:46.633 --> 00:15:50.166 So this is the first of the four required studies 00:15:50.166 --> 00:15:53.166 for a moderate complexity tool. 00:15:53.933 --> 00:15:55.700 So how do we define accuracy. 00:15:55.700 --> 00:15:57.566 What is it in this context. 00:15:58.533 --> 00:16:02.833 So accuracy is a measure of bias 00:16:03.366 --> 00:16:07.633 and typically bias to some reference method. 00:16:09.300 --> 00:16:12.300 Now in my world where I'm primarily 00:16:12.300 --> 00:16:16.166 working with FDA approved tools, 00:16:16.766 --> 00:16:21.566 usually bias in that context is relative bias, 00:16:22.100 --> 00:16:26.400 meaning it's biased relative to the existing tool that I have. 00:16:26.766 --> 00:16:30.166 Perhaps I'm looking to replace one assay, 00:16:32.033 --> 00:16:33.933 with an updated, 00:16:33.933 --> 00:16:36.933 newer version of that same assay, for example. 00:16:37.266 --> 00:16:41.700 Or I'm replacing the latest version of an instrument with, 00:16:42.700 --> 00:16:46.600 a model of that instrument that is being sunsetting. 00:16:47.433 --> 00:16:51.300 So that would be relative bias between two tools, 00:16:52.300 --> 00:16:55.300 keeping in mind that in that situation, 00:16:55.400 --> 00:16:58.566 both tools can have error. 00:16:59.900 --> 00:17:03.600 So we may not always know which one is totally right. 00:17:04.700 --> 00:17:07.700 Then there's absolute bias, 00:17:08.000 --> 00:17:11.866 which would be bias of the tool that you're evaluating 00:17:12.066 --> 00:17:16.100 against some gold standard or reference method. 00:17:16.100 --> 00:17:19.333 That would be the definitive measure of truth 00:17:19.866 --> 00:17:23.400 in terms of quantification and recovery 00:17:23.400 --> 00:17:26.400 of whatever it is that you're trying to measure. 00:17:27.600 --> 00:17:30.600 Most laboratories aren't able to, 00:17:32.100 --> 00:17:36.100 conduct an accuracy study against, you know, the 00:17:36.666 --> 00:17:40.733 the true reference method for a particular test, but many do. 00:17:41.966 --> 00:17:45.300 How do you conduct an accuracy study? 00:17:46.000 --> 00:17:49.000 Well, you design a method comparison. 00:17:49.300 --> 00:17:51.800 And as I mentioned in the previous, 00:17:51.800 --> 00:17:54.900 slide, you want to be really careful what tool 00:17:55.133 --> 00:17:59.000 you compare your new assay or your new instrument against, 00:17:59.966 --> 00:18:02.566 because both tools are 00:18:02.566 --> 00:18:05.966 or can be subject to some measurement error. 00:18:06.000 --> 00:18:11.266 It's nice if you can choose a tool that is known to be operating 00:18:11.700 --> 00:18:15.033 as expected and operating stably 00:18:15.600 --> 00:18:18.600 for your method comparison study. 00:18:19.666 --> 00:18:20.966 The other really important 00:18:20.966 --> 00:18:23.966 consideration is the sample quality. 00:18:24.200 --> 00:18:28.000 So you're going to analyze a minimum of 40 specimens 00:18:28.333 --> 00:18:33.533 by both the test meaning the new tool and the reference method, 00:18:34.500 --> 00:18:36.766 preferably in duplicate over 00:18:36.766 --> 00:18:39.766 a period of many days. Now. 00:18:42.066 --> 00:18:45.066 What do you select for your specimens? 00:18:45.300 --> 00:18:48.600 It's obviously best to select specimens 00:18:48.600 --> 00:18:53.733 that are of the same matrix as patient samples. 00:18:55.166 --> 00:18:58.900 And there are some challenges inherent in that. 00:18:59.266 --> 00:19:02.900 So we all know that patient samples 00:19:04.066 --> 00:19:08.066 sometimes are not of sufficiently high quality. 00:19:08.366 --> 00:19:12.333 For example, it's not uncommon for laboratories to receive 00:19:12.600 --> 00:19:15.600 patient samples that have a low level of hemolysis 00:19:15.933 --> 00:19:19.633 or some other interfering substance that, affects 00:19:20.166 --> 00:19:23.166 the ability of the tool to measure 00:19:23.400 --> 00:19:25.566 the analyte of interest. 00:19:25.566 --> 00:19:28.766 So when you're doing a method comparison study, 00:19:29.333 --> 00:19:32.333 you want to make sure that whatever samples 00:19:32.666 --> 00:19:35.533 you procure for the method comparison 00:19:35.533 --> 00:19:40.200 are of high quality, that are free or as free as possible 00:19:40.200 --> 00:19:44.200 from, substances that can interfere with the measurement. 00:19:45.900 --> 00:19:48.066 It's also optimal 00:19:48.066 --> 00:19:53.533 if you curate samples for the study that covers a very wide 00:19:53.533 --> 00:19:57.433 range of analyte concentration, because you'd like to be able 00:19:57.433 --> 00:20:00.833 to understand how accurate your new tool is, 00:20:01.900 --> 00:20:03.933 both at high 00:20:03.933 --> 00:20:06.933 concentrations of the analyte as well as low 00:20:08.133 --> 00:20:11.600 and we'll talk about this 00:20:11.600 --> 00:20:14.600 in a little bit more detail with an example coming up. 00:20:15.333 --> 00:20:21.066 But I always find that it's it's very important to have a robust number 00:20:21.066 --> 00:20:24.066 of samples clustered around any 00:20:24.066 --> 00:20:27.200 clinically significant points of interest. 00:20:27.400 --> 00:20:30.300 Stay tuned on that point. 00:20:30.300 --> 00:20:32.666 Okay, so back to the drawing board. 00:20:32.666 --> 00:20:37.200 You're going to take a minimum of 40 specimens, preferably of the matrix 00:20:37.333 --> 00:20:40.333 that you will be testing for patients, 00:20:41.066 --> 00:20:43.300 spanning over the measurement 00:20:43.300 --> 00:20:48.000 range of the tool and of high quality and within stability. 00:20:48.000 --> 00:20:48.900 That's another, 00:20:49.833 --> 00:20:52.833 thing that can be important. 00:20:53.000 --> 00:20:56.000 So now at the end of this study, 00:20:56.000 --> 00:20:59.000 you have got an Excel spreadsheet 00:20:59.033 --> 00:21:02.133 that, lists a result for each specimen, 00:21:02.833 --> 00:21:06.833 for your existing method or your reference method quote. 00:21:07.233 --> 00:21:10.233 And your new tool or your test method, 00:21:10.266 --> 00:21:14.200 and you're going to perform a statistical analysis. 00:21:15.000 --> 00:21:18.300 Typically you'll make a scatterplot of the data and you'll 00:21:18.466 --> 00:21:22.200 calculate regression statistics and estimate bias. 00:21:23.166 --> 00:21:23.600 Okay. 00:21:23.600 --> 00:21:26.600 So now you've got some stats in front of you. 00:21:26.800 --> 00:21:29.600 How do you determine acceptability. 00:21:29.600 --> 00:21:32.600 Typically we compare the results 00:21:32.600 --> 00:21:36.900 with internal criteria to judge acceptability 00:21:37.033 --> 00:21:40.366 or manufacturers claims for accuracy 00:21:41.666 --> 00:21:42.533 for more detail. 00:21:42.533 --> 00:21:46.033 So there's a nice consensus document Klci 00:21:46.033 --> 00:21:49.033 Ep9 that can be referenced. 00:21:49.233 --> 00:21:52.233 So let's look at an example. 00:21:52.433 --> 00:21:55.366 This is a study that 00:21:55.366 --> 00:21:57.600 my laboratory published in 00:21:58.800 --> 00:22:01.233 a laboratory medicine journal. 00:22:01.233 --> 00:22:04.233 It was a validation study 00:22:04.400 --> 00:22:07.400 for a new pro calcitonin assay. 00:22:08.033 --> 00:22:11.100 At the time, our laboratory had the gold standard 00:22:11.100 --> 00:22:14.866 method, which was on the crypto instrument. 00:22:15.366 --> 00:22:19.266 And we were looking to replace the crypto 00:22:19.500 --> 00:22:22.500 with the, 00:22:22.766 --> 00:22:25.533 method offered by our, 00:22:25.533 --> 00:22:29.500 vendor who supplies our our automated chemistry assays. 00:22:30.766 --> 00:22:32.966 So what did we do 00:22:32.966 --> 00:22:34.500 for this particular study? 00:22:34.500 --> 00:22:37.500 We analyzed 192 00:22:38.000 --> 00:22:39.733 patient specimens. 00:22:39.733 --> 00:22:42.733 And these specimens were in the, 00:22:44.800 --> 00:22:46.133 collection container 00:22:46.133 --> 00:22:50.066 that is deemed to be acceptable for both assays. 00:22:50.100 --> 00:22:52.800 So that was the plasma separator tube. 00:22:52.800 --> 00:22:55.200 And serum separator tube. 00:22:55.200 --> 00:22:58.166 And these samples 00:22:58.166 --> 00:23:02.433 were collected and analyzed over a period of months, encompassing 00:23:02.466 --> 00:23:05.833 two lots of reagent and two lots of calibrator. 00:23:06.800 --> 00:23:09.766 So what you're looking at here is the scatter plot of those results. 00:23:12.400 --> 00:23:13.300 And when I 00:23:13.300 --> 00:23:17.100 look at method comparison data and in particular 00:23:17.100 --> 00:23:20.966 the scatter plot, I'm typically looking at a few things. 00:23:21.300 --> 00:23:24.433 The first is what is the overall spread 00:23:24.433 --> 00:23:27.433 or dispersion of the points 00:23:27.600 --> 00:23:32.266 around the identity line, which is the 1 to 1 line. 00:23:32.833 --> 00:23:36.266 So if all the points are on top of that 1 to 1 line, 00:23:36.266 --> 00:23:39.266 which is the gray one at the 45 degree angle, 00:23:39.833 --> 00:23:42.000 they're all clustered right on top of that line. 00:23:42.000 --> 00:23:45.266 That indicates a perfect correlation between the two methods. 00:23:45.266 --> 00:23:48.000 And that's like a medical director stream. 00:23:48.000 --> 00:23:49.300 But that rarely happens. 00:23:49.300 --> 00:23:52.300 Typically you will see some dispersion 00:23:52.633 --> 00:23:55.200 around that identity line. 00:23:55.200 --> 00:23:58.200 Then I'm looking for the spread of points, 00:23:58.866 --> 00:24:00.633 to indicate the overall range 00:24:00.633 --> 00:24:05.666 of analyte concentrations that we challenge the new tool with. 00:24:06.966 --> 00:24:09.300 And then the red dotted line 00:24:09.300 --> 00:24:13.000 is the passing linear regression. 00:24:14.600 --> 00:24:16.900 So going back to the example that's in front of you, 00:24:16.900 --> 00:24:20.166 I see that we have lots of samples. 00:24:21.233 --> 00:24:24.466 And they do appear to be spread out 00:24:24.466 --> 00:24:28.066 pretty nicely over a large range of target concentration. 00:24:28.500 --> 00:24:31.300 I noticed that at the low end, there's many, 00:24:31.300 --> 00:24:34.300 many more samples clustered there. 00:24:34.500 --> 00:24:37.700 And there does appear to be overall 00:24:37.833 --> 00:24:41.100 a negative bias between the two tools. 00:24:41.100 --> 00:24:43.933 And we'll talk about that in more detail. 00:24:43.933 --> 00:24:47.466 Now the regression statistics are listed in that box. 00:24:48.066 --> 00:24:51.900 So passing that Loc regression 00:24:51.900 --> 00:24:58.100 gives us a slope of 0.834 and an intercept of -0.013. 00:24:59.800 --> 00:25:02.200 And that will become important in a moment. 00:25:02.200 --> 00:25:05.666 So in the ideal world, you'd like it if your slope was one 00:25:06.066 --> 00:25:08.000 and your intercept is zero. 00:25:08.000 --> 00:25:10.500 So the slope is an assessment of your 00:25:11.800 --> 00:25:13.433 proportional bias. 00:25:13.433 --> 00:25:17.200 And then the intercept is your constant bias. 00:25:19.566 --> 00:25:22.400 So looking at those regression statistics 00:25:22.400 --> 00:25:25.800 it supports the picture that I'm seeing in the scatter 00:25:25.800 --> 00:25:28.800 plot graph. 00:25:29.100 --> 00:25:31.366 So let's talk for a second. 00:25:31.366 --> 00:25:34.166 I mentioned being strategic 00:25:34.166 --> 00:25:37.166 with how you curate samples for your method comparison. 00:25:37.900 --> 00:25:40.933 I like to make sure that I have lots of samples 00:25:40.933 --> 00:25:45.033 around medical decision points or clinically relevant cutoffs. 00:25:46.566 --> 00:25:49.300 Sometimes it isn't always possible 00:25:49.300 --> 00:25:53.466 to obtain samples around those values, 00:25:53.633 --> 00:25:56.700 but if it's at all possible, I highly advise it. 00:25:58.133 --> 00:26:01.700 Going back to our example of the pro calcitonin assay. 00:26:02.133 --> 00:26:05.533 There are two medically relevant cutoffs 00:26:05.966 --> 00:26:08.966 at the low end of the measurement range. 00:26:09.500 --> 00:26:13.866 So we strategically designed our method comparison study 00:26:14.833 --> 00:26:18.900 to cluster samples around those decision points to determine 00:26:19.566 --> 00:26:22.566 how great of the correlation 00:26:22.800 --> 00:26:25.800 was between the crypto assay and the new tool. 00:26:27.200 --> 00:26:30.933 And what we discovered is that in panel A, 00:26:32.200 --> 00:26:34.866 when we 00:26:34.866 --> 00:26:37.966 limit the scatter plot 00:26:37.966 --> 00:26:40.966 and the regression statistics 00:26:41.700 --> 00:26:47.000 to samples with a pro calcitonin target concentration 00:26:47.000 --> 00:26:50.000 between 0 and 3 nanograms per ML, 00:26:50.633 --> 00:26:53.333 that the negative bias 00:26:53.333 --> 00:26:56.333 becomes much more evident 00:26:56.866 --> 00:26:59.866 than if you were to not 00:27:00.400 --> 00:27:03.600 narrow the, data 00:27:04.200 --> 00:27:07.600 for your statistical analysis to that lower end. 00:27:09.900 --> 00:27:14.700 So what we saw is that the slope is 0.792 00:27:14.700 --> 00:27:18.166 and the intercept -0.008. 00:27:22.733 --> 00:27:25.100 We also put together a table 00:27:25.100 --> 00:27:28.666 showing agreement between the two tools at 00:27:30.466 --> 00:27:33.500 certain points in the measurement range. 00:27:33.500 --> 00:27:38.700 So at 0.10.250.5 and two nanograms per ML. 00:27:39.233 --> 00:27:42.100 So for this assay the important cutoffs 00:27:42.100 --> 00:27:45.100 are 0.5 and two. 00:27:45.133 --> 00:27:47.733 So looking at 00:27:47.733 --> 00:27:52.500 for this discussion I'll just go over the the latter two that I mentioned. 00:27:52.833 --> 00:27:57.600 Looking at the positive agreement between the crypt door, 00:27:57.966 --> 00:28:01.800 which remember was the gold standard at the time versus the new tool 00:28:02.400 --> 00:28:06.000 at the cutoff of 0.5 nanogram per ML. 00:28:06.500 --> 00:28:10.533 The positive agreement was only 78.9% 00:28:12.400 --> 00:28:14.666 at two nanogram per ML. 00:28:14.666 --> 00:28:17.666 It was 42.9%. 00:28:18.733 --> 00:28:23.166 So let's step back into the context 00:28:24.266 --> 00:28:26.733 for the clinicians. 00:28:26.733 --> 00:28:30.000 So what this would mean is that with the new tool, 00:28:30.866 --> 00:28:33.866 we would be. 00:28:34.133 --> 00:28:35.100 Having a difference 00:28:35.100 --> 00:28:40.300 of classification for patients around these two cutoffs 00:28:40.900 --> 00:28:43.833 compared to the old method. 00:28:43.833 --> 00:28:46.500 So it's really up to you as the medical director 00:28:46.500 --> 00:28:49.500 to decide how big of a deal that is. 00:28:49.800 --> 00:28:52.800 At the time we thought it was very interesting. 00:28:52.800 --> 00:28:56.300 And these cutoffs are used for determining 00:28:57.133 --> 00:29:00.133 risk for severe sepsis. 00:29:00.366 --> 00:29:03.166 And also they're used for, 00:29:03.166 --> 00:29:06.266 consideration for antibiotic stewardship, 00:29:07.400 --> 00:29:09.900 for lower respiratory tract infections. 00:29:09.900 --> 00:29:12.900 So we were concerned about this negative bias. 00:29:13.333 --> 00:29:16.833 And purportedly lower positive agreement at the low 00:29:16.833 --> 00:29:19.833 end around those cutoffs. 00:29:21.166 --> 00:29:24.166 There are other situations where 00:29:25.000 --> 00:29:26.466 these type of data 00:29:26.466 --> 00:29:30.566 would be more worrisome or potentially less worrisome, 00:29:30.566 --> 00:29:35.033 again, depending on what type of assay you're evaluating 00:29:35.666 --> 00:29:38.233 and how much weight is placed 00:29:38.233 --> 00:29:41.233 on the result of that particular assay 00:29:41.366 --> 00:29:44.533 for medical diagnosis or treatment. 00:29:46.466 --> 00:29:47.733 But what's interesting is 00:29:47.733 --> 00:29:50.733 that you go back to the overall 00:29:51.133 --> 00:29:54.700 method comparison data, and you go, but wait, 00:29:54.900 --> 00:29:57.900 when you look at the whole data set, things look better. 00:29:59.033 --> 00:30:02.266 And that's because in my experience, at least, 00:30:04.133 --> 00:30:07.133 the new tool has bias. But, 00:30:07.833 --> 00:30:09.500 when you're 00:30:09.500 --> 00:30:13.333 evaluating the high end of a measurement range compared to the low, 00:30:13.633 --> 00:30:18.200 that bias may not be, equivalent across the measurement range. 00:30:18.666 --> 00:30:21.566 So I think that this 00:30:21.566 --> 00:30:26.100 example highlights a few of the take home points about accuracy. 00:30:26.300 --> 00:30:28.200 Really well. 00:30:28.200 --> 00:30:31.200 And I'm going to flip back to my previous slide about this. 00:30:32.433 --> 00:30:35.433 So the take home points are 00:30:35.866 --> 00:30:38.166 make sure that you are selecting 00:30:38.166 --> 00:30:41.500 a sufficient quantity of specimens for your method. 00:30:41.500 --> 00:30:43.500 Comparison. 00:30:43.500 --> 00:30:48.000 And make sure that you are selecting specimens that, 00:30:48.600 --> 00:30:52.200 if at all possible, spanned the measurement range for your tool. 00:30:53.133 --> 00:30:55.466 And then never forget 00:30:55.466 --> 00:30:58.766 an assessment around the medical decision points. 00:31:00.466 --> 00:31:01.033 And this pro 00:31:01.033 --> 00:31:05.266 calcitonin case study does a nice job of highlighting that latter point 00:31:05.266 --> 00:31:08.566 and showing the value of a really thorough method 00:31:08.566 --> 00:31:11.566 comparison. 00:31:14.166 --> 00:31:15.133 Okay, 00:31:15.133 --> 00:31:19.533 so on to the second study, which is an assessment for precision. 00:31:20.633 --> 00:31:23.066 There's different types of precision. 00:31:23.066 --> 00:31:27.000 There's the within run or intra assay precision 00:31:27.800 --> 00:31:33.866 between run precision and date day to day or total precision. 00:31:33.866 --> 00:31:39.333 And we're going to focus on the intra assay or total precision 00:31:39.333 --> 00:31:42.333 for the purpose of this presentation. 00:31:44.166 --> 00:31:47.500 So we make an assessment for the precision of a new tool 00:31:47.500 --> 00:31:50.500 by conducting precision study. 00:31:51.366 --> 00:31:53.900 So again just as with the method comparison 00:31:53.900 --> 00:31:57.533 it's really important that you select appropriate material for this. 00:31:58.166 --> 00:32:00.233 And you have some options. 00:32:00.233 --> 00:32:02.500 So option one is to take 00:32:02.500 --> 00:32:05.500 patient samples and analyze them. 00:32:06.300 --> 00:32:09.000 Many replicates over time. 00:32:09.000 --> 00:32:12.566 Option two is to analyze commercially 00:32:13.766 --> 00:32:16.200 available material, 00:32:16.200 --> 00:32:19.200 and which to choose is going 00:32:19.200 --> 00:32:22.433 to depend on the stability of what you're trying to measure. 00:32:23.066 --> 00:32:26.066 So for example, in my laboratory 00:32:26.566 --> 00:32:31.133 we have a handful of analytes where the patient samples 00:32:31.133 --> 00:32:35.933 would just not be stable enough to complete out the full precision study. 00:32:37.033 --> 00:32:39.900 So typically what we do is 00:32:39.900 --> 00:32:43.933 we select an appropriate quality control material 00:32:45.100 --> 00:32:49.700 to conduct our precision study or I guess option three. 00:32:49.700 --> 00:32:51.800 And we've done this as well is you. 00:32:51.800 --> 00:32:54.900 You combine the two options and include 00:32:54.900 --> 00:32:57.900 some patient samples as well 00:32:57.900 --> 00:33:00.566 in your study design. 00:33:00.566 --> 00:33:03.566 So if you've got an FDA 00:33:03.933 --> 00:33:07.666 approved assay that you are looking to evaluate 00:33:08.366 --> 00:33:11.566 and you haven't modified that diagnostic tool in any way 00:33:12.133 --> 00:33:16.500 according to clear, you just have to do a verification study for precision, 00:33:16.500 --> 00:33:21.066 which is an abbreviated option compared to a full precision study. 00:33:22.066 --> 00:33:22.700 So that's 00:33:22.700 --> 00:33:25.800 typically called the five by five study design. 00:33:26.633 --> 00:33:29.633 So you can do five replicates 00:33:29.800 --> 00:33:32.800 per day over five days. 00:33:34.233 --> 00:33:37.433 Or you do five measurements and duplicate it. 00:33:37.433 --> 00:33:38.866 Or you do five measurements. 00:33:38.866 --> 00:33:40.500 Two runs a day over five days. 00:33:40.500 --> 00:33:42.566 But the idea being 00:33:42.566 --> 00:33:45.933 you're going to do some replicates each day for a total of five days. 00:33:46.633 --> 00:33:49.500 In contrast, a full precision study 00:33:49.500 --> 00:33:53.566 would be more on the level of what a manufacturer would do 00:33:54.200 --> 00:33:57.633 for their validation exercise. 00:33:58.233 --> 00:34:02.300 This is also the approach I would take if I had a modified FDA 00:34:02.300 --> 00:34:06.600 approved assay, or a laboratory developed test that I'm evaluating. 00:34:07.766 --> 00:34:10.533 So for a full precision study, 00:34:10.533 --> 00:34:13.000 you're going to get your within 00:34:13.000 --> 00:34:17.566 run precision by performing 20 consecutive replicates 00:34:17.566 --> 00:34:21.500 per sample in a single run, and your total precision 00:34:22.133 --> 00:34:25.133 two replicates per concentration level 00:34:25.233 --> 00:34:28.633 per run, two runs a day over 20 days. 00:34:29.733 --> 00:34:31.833 So the difference ends up being a five day 00:34:31.833 --> 00:34:34.833 study design versus a 20 day study design. 00:34:35.700 --> 00:34:39.766 You get a lot more information out of the full precision study. 00:34:39.766 --> 00:34:42.300 But remember, there's no free lunch. 00:34:42.300 --> 00:34:45.500 This is very time consuming. 00:34:45.533 --> 00:34:50.766 You've got tech time, you've got the cost of the materials, etc. 00:34:52.133 --> 00:34:54.866 so it ends up being a balance 00:34:54.866 --> 00:34:56.400 for your data analysis. 00:34:56.400 --> 00:34:59.833 You're going to calculate the mean standard deviation and coefficient 00:34:59.833 --> 00:35:01.200 of variation. 00:35:01.200 --> 00:35:05.266 And then you're going to compare the results with claims 00:35:05.266 --> 00:35:08.266 from the manufacturer or internal criteria 00:35:08.266 --> 00:35:11.266 to judge acceptability. 00:35:12.233 --> 00:35:15.233 So going back to that pro calcitonin assay 00:35:15.600 --> 00:35:17.133 how did we do this. 00:35:17.133 --> 00:35:22.600 What we did is we took three levels of control material. 00:35:22.600 --> 00:35:25.033 And by level I mean 00:35:25.033 --> 00:35:31.100 QC material that had differing target concentrations of pro calcitonin. 00:35:32.233 --> 00:35:35.300 We ran those twice per day in duplicate for 20 days. 00:35:35.566 --> 00:35:37.266 So we were a little bit more thorough 00:35:37.266 --> 00:35:40.266 with our precision study than we absolutely had to be. 00:35:41.500 --> 00:35:43.966 And these were the results. 00:35:43.966 --> 00:35:46.966 So you're going to see for each level of QC 00:35:47.466 --> 00:35:51.266 what the mean recovery was, what was the standard deviation. 00:35:51.266 --> 00:35:55.833 And CV for within the same run and the same for total precision. 00:35:57.133 --> 00:35:59.800 So for example 00:35:59.800 --> 00:36:03.300 for the low target concentration quality control material 00:36:03.633 --> 00:36:08.233 it was 2.9% CV and 3.4%. 00:36:08.266 --> 00:36:11.266 And then we compared these to the package insert 00:36:11.366 --> 00:36:15.033 given to us by the manufacturer of the assay. 00:36:15.600 --> 00:36:19.400 And these results exceeded the criteria. 00:36:19.400 --> 00:36:22.400 So looked great. 00:36:25.533 --> 00:36:25.900 I'll just 00:36:25.900 --> 00:36:29.033 briefly mention regarding acceptability criteria. 00:36:29.300 --> 00:36:32.400 Sometimes you've got one day 00:36:32.800 --> 00:36:37.233 in your study where the instrument was a little less precise, 00:36:37.666 --> 00:36:42.600 and that sort of affects the data analysis such that your, 00:36:43.766 --> 00:36:46.600 either within run or total precision 00:36:46.600 --> 00:36:49.533 is higher than what was expected. 00:36:49.533 --> 00:36:53.533 That's the downside of doing just a five day precision study, is that 00:36:54.100 --> 00:36:57.100 you'll be a little bit more prone to days like that, where you're 00:36:57.200 --> 00:37:00.500 considered somewhat of an outlier in terms of the precision. 00:37:01.500 --> 00:37:04.933 So your options at that point are to repeat the study. 00:37:05.833 --> 00:37:09.100 Or you can embark on that longer precision study 00:37:09.100 --> 00:37:12.100 that I described in a previous slide. 00:37:13.100 --> 00:37:16.100 However, you know, from occasionally, 00:37:16.400 --> 00:37:19.433 you just can't get the percent CV 00:37:20.066 --> 00:37:23.533 to come within the specification of the tool. 00:37:23.966 --> 00:37:26.966 And at that point you have to decide, 00:37:27.566 --> 00:37:29.333 does it matter? 00:37:29.333 --> 00:37:32.333 Obviously, you want to eliminate 00:37:32.766 --> 00:37:36.033 or troubleshoot what could be causing that. 00:37:36.766 --> 00:37:39.766 But if you exhaust all possibility 00:37:40.166 --> 00:37:43.400 and you're not able to identify a cause, you really have to come back to 00:37:43.400 --> 00:37:46.700 the question of is the precision clinically acceptable? 00:37:49.400 --> 00:37:50.400 Precision data are 00:37:50.400 --> 00:37:53.400 incredibly useful for troubleshooting. 00:37:54.333 --> 00:37:57.566 Also for clinical queries about significant change. 00:37:58.000 --> 00:38:00.733 And that is the scenario that I described earlier, 00:38:00.733 --> 00:38:03.733 where you get a call from a clinician saying 00:38:03.800 --> 00:38:07.866 my patient's value changed by two, 00:38:08.566 --> 00:38:12.566 and is this, clinically significant or meaningful? 00:38:13.500 --> 00:38:15.600 We also used precision data 00:38:15.600 --> 00:38:18.966 when we set ranges for QC. 00:38:20.700 --> 00:38:22.300 So you really do want to understand 00:38:22.300 --> 00:38:25.300 how precise your tool is. 00:38:25.500 --> 00:38:27.833 You wouldn't want to, for example, 00:38:27.833 --> 00:38:32.933 set your QC ranges narrower than the actualized precision 00:38:32.933 --> 00:38:35.933 of your instrument because you'd be failing QC all the time. 00:38:39.800 --> 00:38:41.433 So we're going to transition 00:38:41.433 --> 00:38:45.333 to the third required study for documenting method performance. 00:38:46.200 --> 00:38:49.133 And that is the reportable range. 00:38:49.133 --> 00:38:51.733 But let's start out with some definitions. 00:38:51.733 --> 00:38:54.400 First because 00:38:54.400 --> 00:38:57.566 this is an area of historical confusion. 00:38:58.633 --> 00:38:59.633 So the reportable 00:38:59.633 --> 00:39:04.333 range is sometimes used interchangeably with the term 00:39:05.200 --> 00:39:08.200 Amr or analytical measurement range. 00:39:08.800 --> 00:39:10.866 So what is the Amr. 00:39:10.866 --> 00:39:14.833 The Amr is the range of values that an instrument can report directly 00:39:15.133 --> 00:39:18.133 without alteration or pretreatment of the sample. 00:39:18.166 --> 00:39:21.433 The Amr is determined by the instrument 00:39:21.633 --> 00:39:24.633 or assay manufacturer. 00:39:25.800 --> 00:39:28.866 If you're in the laboratory developed test business, that's you. 00:39:28.900 --> 00:39:31.900 You're you're determining what the EMR is. 00:39:32.666 --> 00:39:35.966 The clinically reportable range is the range of values 00:39:35.966 --> 00:39:38.966 that can be reported with alteration of the sample. 00:39:39.133 --> 00:39:41.766 So for example, 00:39:41.766 --> 00:39:44.766 let's say I had a pro calcitonin 00:39:45.900 --> 00:39:47.700 in a patient sample. 00:39:47.700 --> 00:39:50.633 And the concentration was higher 00:39:50.633 --> 00:39:53.633 than my measurement range of my instrument. 00:39:54.433 --> 00:39:58.000 I could dilute that sample and re quantify the pro 00:39:58.000 --> 00:40:01.800 calcitonin, apply the dilution factor and report, 00:40:02.866 --> 00:40:05.700 a numerical value for that sample. 00:40:05.700 --> 00:40:08.966 So this gets confusing because in the lab medicine world 00:40:08.966 --> 00:40:11.966 we sometimes use these terms amr kra 00:40:12.333 --> 00:40:15.333 interchangeably. 00:40:16.066 --> 00:40:16.833 So what do you do. 00:40:16.833 --> 00:40:19.833 What do you decide to do for patient samples? 00:40:20.700 --> 00:40:25.400 In my laboratories I typically use the Amr, 00:40:25.433 --> 00:40:28.433 the analytical measurement range as the range 00:40:29.200 --> 00:40:32.200 of values for which I'm going to report on patients. 00:40:32.333 --> 00:40:34.466 But there are 00:40:34.466 --> 00:40:37.466 a handful of scenarios where I need to modify that 00:40:37.633 --> 00:40:40.800 to create an expanded capability for reporting. 00:40:41.400 --> 00:40:45.000 And that's typically at the high end when I'm trying to quantify something, 00:40:46.700 --> 00:40:49.700 that would be clinically meaningful. 00:40:49.700 --> 00:40:54.000 So if you decide to expand the range 00:40:55.266 --> 00:40:57.200 of values that you're going 00:40:57.200 --> 00:41:00.266 to report for patients, you've got to document 00:41:00.900 --> 00:41:04.266 that your tool can reliably quantify. 00:41:05.700 --> 00:41:09.700 So for example, in the pro calcitonin assay 00:41:09.700 --> 00:41:12.900 I would need to prove that diluting the patient sample 00:41:13.700 --> 00:41:17.033 and analyzing it and applying that dilution factor 00:41:17.033 --> 00:41:20.033 produced reliable results. 00:41:23.000 --> 00:41:26.333 Let me briefly mention that if you 00:41:28.100 --> 00:41:30.466 if you modify 00:41:30.466 --> 00:41:33.266 the low end as well, you're going to have to validate 00:41:33.266 --> 00:41:37.500 that your tool can accurately quantify at that low end. 00:41:38.400 --> 00:41:41.400 So your job is a lot easier if you just use 00:41:41.766 --> 00:41:44.400 the Amr that comes with the tool. 00:41:44.400 --> 00:41:48.733 The take home message is if you make modifications to expand that 00:41:48.733 --> 00:41:53.400 range, you're going to have to document that the method performance is acceptable. 00:41:56.166 --> 00:41:59.100 So how do you meet this requirement? 00:41:59.100 --> 00:42:02.100 How do you conduct a reportable range study. 00:42:02.700 --> 00:42:06.300 Another term for this would be a linearity study. 00:42:06.900 --> 00:42:09.366 And typically what is recommended. 00:42:09.366 --> 00:42:12.366 And again this is this is best practice 00:42:12.633 --> 00:42:16.833 is to take five or more concentrations of analyte 00:42:17.200 --> 00:42:20.100 throughout the stated range that you're trying to verify. 00:42:21.700 --> 00:42:24.700 And so a minimum of five 00:42:24.866 --> 00:42:27.866 samples at varying target concentrations. 00:42:28.966 --> 00:42:31.500 And then you're going to take 00:42:31.500 --> 00:42:35.033 two replicates at each for each sample. 00:42:35.966 --> 00:42:40.100 And then you're going to evaluate linear fit with an x y plot. 00:42:40.300 --> 00:42:43.300 And then calculate regression statistics. 00:42:43.433 --> 00:42:47.200 So how do you procure materials for this type of study. 00:42:48.166 --> 00:42:50.633 There's a bunch of different ways you can do it. 00:42:50.633 --> 00:42:55.666 And which you choose is going to depend on cost and resources, 00:42:55.833 --> 00:42:58.833 as well as availability of patient samples. 00:42:59.566 --> 00:43:01.566 But here are some ideas. 00:43:01.566 --> 00:43:05.400 So you could take a, patient sample 00:43:05.400 --> 00:43:08.400 with a low concentration of what you're trying to measure 00:43:08.400 --> 00:43:11.400 and spike it with 00:43:11.400 --> 00:43:14.100 the analyte of interest. 00:43:14.100 --> 00:43:17.233 You could take a high patient sample and dilute it. 00:43:17.633 --> 00:43:19.800 You could mix 00:43:19.800 --> 00:43:23.000 patient samples, you know, take one that's high and one 00:43:23.000 --> 00:43:26.833 that's low and create a variety of target concentrations. 00:43:26.866 --> 00:43:30.000 That way you can also buy, 00:43:30.866 --> 00:43:35.100 you know, human serum and spike it with a standard reference material. 00:43:36.200 --> 00:43:38.266 And then as another option, 00:43:38.266 --> 00:43:42.433 there are different vendors that sell, linearity products. 00:43:42.433 --> 00:43:47.533 They're basically kits with predetermined concentrations of the analyte 00:43:47.833 --> 00:43:51.000 that you can buy and then analyze and duplicate 00:43:51.166 --> 00:43:54.166 to satisfy this requirement. 00:43:56.533 --> 00:43:57.800 So what do we do for 00:43:57.800 --> 00:44:01.533 for the published study for pro calcitonin? 00:44:02.033 --> 00:44:06.366 Well, we took a pooled patient serum sample. 00:44:06.600 --> 00:44:09.600 And one of the collaborators was calibrator A 00:44:09.866 --> 00:44:12.900 and mixed them to get six sample concentrations. 00:44:13.433 --> 00:44:17.766 And then we ran each of those six in triplicate on each instrument. 00:44:19.166 --> 00:44:21.300 So we 00:44:21.300 --> 00:44:22.833 we went above and beyond here. 00:44:22.833 --> 00:44:25.600 As I mentioned you know best practice is five. 00:44:25.600 --> 00:44:27.933 We went with six. 00:44:27.933 --> 00:44:31.600 What was nice about this is that we were testing 00:44:31.600 --> 00:44:34.700 the native patient matrix, but it would have 00:44:34.700 --> 00:44:37.700 also been acceptable to, 00:44:37.766 --> 00:44:41.100 take any of the other approaches that I described on the previous slide. 00:44:42.133 --> 00:44:43.233 And we had good results. 00:44:43.233 --> 00:44:46.366 I didn't choose to put in a table here, but both of the 00:44:46.800 --> 00:44:50.166 the instruments that we were evaluating, the Sasson demonstrated, 00:44:51.133 --> 00:44:54.133 linearity consistent with the manufacturer's claims. 00:44:54.133 --> 00:44:57.900 So what you want to see is that, on your scatterplot, 00:44:58.266 --> 00:45:01.266 you want to see that there is a linear relationship 00:45:01.933 --> 00:45:04.266 between the, 00:45:04.266 --> 00:45:08.100 target concentration of the analyte and what your instrument 00:45:08.100 --> 00:45:12.066 actually measured over the entire stated measurement range. 00:45:12.600 --> 00:45:15.566 And then your regression statistics will prove to you 00:45:15.566 --> 00:45:18.566 that there is truly a linear fit. 00:45:21.566 --> 00:45:22.366 Okay. 00:45:22.366 --> 00:45:25.166 This is the fourth study that's required for method 00:45:25.166 --> 00:45:28.166 verification of a moderately complex assay. 00:45:29.000 --> 00:45:32.000 And that is an assessment for reference intervals. 00:45:32.800 --> 00:45:35.400 So laboratories are not actually required 00:45:35.400 --> 00:45:38.400 to establish their own reference interval 00:45:39.366 --> 00:45:41.866 for an FDA approved assay. 00:45:41.866 --> 00:45:46.700 However, good practice is to verify that the reference interval is appropriate 00:45:46.700 --> 00:45:51.000 for the patient population for which the tool is going to be used. 00:45:52.800 --> 00:45:55.800 We're going to talk about how to do that. 00:45:56.100 --> 00:45:58.700 You can also use a previously established 00:45:58.700 --> 00:46:01.700 reference interval at your home institution. 00:46:01.833 --> 00:46:04.833 You can create a new one that's sort of at your discretion. 00:46:05.500 --> 00:46:07.866 So if you do have a previously established 00:46:07.866 --> 00:46:10.866 reference interval you can transfer that one. 00:46:11.000 --> 00:46:15.333 If the subject population hasn't changed 00:46:15.666 --> 00:46:18.633 and if them method comparison shows 00:46:18.633 --> 00:46:21.700 that the two tools are equivalent or comparable. 00:46:23.100 --> 00:46:27.200 But best practice is really to just test. 00:46:27.900 --> 00:46:30.966 So what you would want to do to verify a reference interval 00:46:31.533 --> 00:46:34.800 is to analyze a minimum of 20 samples 00:46:35.366 --> 00:46:38.366 from a quote, normal population, 00:46:39.033 --> 00:46:41.600 and determine if the results for the test 00:46:41.600 --> 00:46:45.000 are within the proposed reference interval. 00:46:46.400 --> 00:46:51.633 If less than or equal to two of those patient samples are outside 00:46:51.633 --> 00:46:55.733 the limits, then you can accept the proposed reference interval. 00:46:56.733 --> 00:46:59.733 Whether that reference interval is coming from the package insert 00:47:00.000 --> 00:47:03.600 for the assay, or if it's a previously established reference 00:47:03.600 --> 00:47:06.600 interval already in use at your laboratory. 00:47:10.066 --> 00:47:13.033 So how are reference intervals determined? 00:47:13.033 --> 00:47:17.366 Typically they're the central 95% of the values 00:47:17.366 --> 00:47:20.366 for the quote normal study population. 00:47:22.433 --> 00:47:24.100 However, 00:47:24.100 --> 00:47:26.700 if you do undertake a verification study 00:47:26.700 --> 00:47:31.366 for a for reference interval, you're going to want to make sure 00:47:31.833 --> 00:47:34.800 that again, your samples 00:47:34.800 --> 00:47:37.600 are of sufficiently high quality 00:47:37.600 --> 00:47:40.800 that they have been processed, 00:47:41.000 --> 00:47:44.000 handled, stored appropriately. 00:47:44.033 --> 00:47:47.066 And depending on what you're trying to validate here, 00:47:47.266 --> 00:47:50.266 you're going to want to be careful about, 00:47:50.800 --> 00:47:53.500 special or unique patient populations. 00:47:53.500 --> 00:47:56.500 So there are some assays for which 00:47:57.000 --> 00:47:59.700 there could be consideration for sex 00:47:59.700 --> 00:48:03.233 or age bracketed reference intervals. 00:48:03.400 --> 00:48:07.033 And so all of these considerations go into your decision 00:48:07.033 --> 00:48:10.333 for how you're going to try to meet this requirement. 00:48:13.600 --> 00:48:16.600 The manufacturers go through 00:48:16.666 --> 00:48:19.766 at the time of assay development. 00:48:19.766 --> 00:48:22.766 And they go through a full reference interval study. 00:48:23.333 --> 00:48:25.466 And it's a real rigorous process. 00:48:25.466 --> 00:48:30.633 So there's predefined selection criteria for, quote, normal individuals. 00:48:31.300 --> 00:48:35.100 You've got to have a robust list of interferences or sources 00:48:35.100 --> 00:48:40.266 of biological variability for the tests that you want to make sure or not. 00:48:41.000 --> 00:48:44.900 Present in the samples that are included for the study. 00:48:46.233 --> 00:48:48.233 You've got to decide on the appropriate number 00:48:48.233 --> 00:48:51.700 of individuals to include in the reference interval study. 00:48:52.833 --> 00:48:55.833 So one of the magic numbers is 120. 00:48:56.166 --> 00:48:59.366 But depending on how much confidence you want to have, that number 00:48:59.366 --> 00:49:01.000 could be much greater. 00:49:01.000 --> 00:49:04.000 You've got to collect and analyze specimens 00:49:04.700 --> 00:49:07.533 and, make sure that that is all happening 00:49:07.533 --> 00:49:11.133 in the same way that you would be analyzing, 00:49:12.166 --> 00:49:14.633 patient specimens. 00:49:14.633 --> 00:49:18.800 So if you've got a reference interval that's going to be partitioned 00:49:18.800 --> 00:49:22.266 by age, for example, and you're a manufacturer, you've got to make sure 00:49:22.266 --> 00:49:27.366 that you have at least 120 specimens to include within each partition. 00:49:27.600 --> 00:49:30.600 So it gets to be a lot of work. 00:49:31.266 --> 00:49:33.700 As I mentioned, you don't have to do a full reference 00:49:33.700 --> 00:49:38.033 interval study for every tool that you're looking to integrate into your lab. 00:49:38.966 --> 00:49:42.266 But it is good laboratory practice to verify 00:49:42.266 --> 00:49:45.800 with a minimum of 20 samples from a normal population 00:49:46.233 --> 00:49:49.066 that the reference interval being proposed 00:49:49.066 --> 00:49:52.066 or transferred is appropriate. 00:49:53.266 --> 00:49:55.766 So what do we do for pro calcitonin? 00:49:55.766 --> 00:49:58.966 We took samples from 20 apparently healthy donors 00:49:59.400 --> 00:50:03.533 and collected blood or blood into Steve and Steve tubes. 00:50:04.200 --> 00:50:06.133 We did have exclusion criteria. 00:50:07.200 --> 00:50:09.600 So your exclusion 00:50:09.600 --> 00:50:12.600 criteria will be different depending on what you're trying to evaluate. 00:50:12.800 --> 00:50:19.133 So for pro calcitonin, because this is an assay that will be ordered 00:50:20.166 --> 00:50:22.633 for patients that are acutely ill. 00:50:22.633 --> 00:50:26.700 One of our exclusion criteria, for example, had to do with, 00:50:27.400 --> 00:50:29.900 you know, the the health 00:50:29.900 --> 00:50:32.900 at that moment in time of the donor 00:50:33.833 --> 00:50:36.500 making sure they didn't have some type of background, 00:50:36.500 --> 00:50:39.500 viral or bacterial infection, for example. 00:50:40.833 --> 00:50:45.900 So what we found from our study was that the samples have pro 00:50:45.900 --> 00:50:51.333 calcitonin concentrations of 0.01 to 0.03 nanograms per milliliter. 00:50:51.866 --> 00:50:54.933 And all of them fall within the manufacturer's 00:50:54.933 --> 00:50:57.933 proposed claims for reference intervals. 00:50:58.400 --> 00:51:02.766 So given these data, we concluded that the proposed 00:51:02.766 --> 00:51:05.766 reference interval was appropriate for our patients. 00:51:09.800 --> 00:51:11.400 So we've just covered the four 00:51:11.400 --> 00:51:15.000 basic studies that are required for method verification. 00:51:16.833 --> 00:51:17.700 It's those same 00:51:17.700 --> 00:51:20.966 four studies are required for method validation as well. 00:51:21.633 --> 00:51:24.633 But now we're going to talk about the two extra studies 00:51:24.633 --> 00:51:27.633 that are required for method validation. 00:51:28.400 --> 00:51:31.300 So accuracy precision 00:51:31.300 --> 00:51:34.300 reportable range and reference interval. 00:51:34.333 --> 00:51:38.166 That's the minimum you need for an FDA cleared or approved assay. 00:51:38.200 --> 00:51:41.200 As I mentioned in previous slides. 00:51:41.733 --> 00:51:44.033 So now we're entering the world of 00:51:44.033 --> 00:51:47.633 what additionally do you need to do for a modified 00:51:47.633 --> 00:51:51.800 FDA approved assay or a laboratory developed test? 00:51:53.000 --> 00:51:56.000 There's two extra studies that you have to do. 00:51:56.200 --> 00:52:00.733 And the first one is an assessment for analytical sensitivity. 00:52:01.700 --> 00:52:04.800 So what this does is it establishes 00:52:04.800 --> 00:52:07.800 the lower detection limit of the assay. 00:52:09.900 --> 00:52:13.300 What you do to meet this requirement is you acquire measurements 00:52:13.300 --> 00:52:16.300 from multiple independent blank 00:52:16.366 --> 00:52:19.200 and low concentration level samples. 00:52:20.500 --> 00:52:23.366 And you want to have at least four samples of each type. 00:52:23.366 --> 00:52:27.700 So for blinks and for low concentration samples 00:52:28.566 --> 00:52:30.733 how do you generate these samples. 00:52:30.733 --> 00:52:33.833 You can dilute samples so you can spike samples 00:52:35.166 --> 00:52:37.366 and try to 00:52:37.366 --> 00:52:41.466 to generate, samples around the targets 00:52:42.666 --> 00:52:46.433 at that low and you definitely want to have a low 00:52:46.433 --> 00:52:50.433 level sample around the presumed limit of detection of the assay. 00:52:51.200 --> 00:52:52.066 And then what you're going to do 00:52:52.066 --> 00:52:55.066 is you're going to test these samples and replicate 00:52:55.633 --> 00:52:58.633 and do some data analysis. 00:52:59.666 --> 00:53:00.333 So even though 00:53:00.333 --> 00:53:03.333 it wasn't strictly required for the pro calcitonin assay 00:53:03.333 --> 00:53:07.600 because again, it it's an FDA approved assay that we weren't modifying. 00:53:08.633 --> 00:53:11.833 We decided to do an analytical sensitivity study, 00:53:12.366 --> 00:53:15.800 and many laboratories would tell you 00:53:15.800 --> 00:53:18.800 that this is the best laboratory practice 00:53:18.900 --> 00:53:21.933 to do this study, even if it is an FDA approved one. 00:53:22.433 --> 00:53:25.366 But according to Clea, it's not absolutely required. 00:53:26.366 --> 00:53:29.233 But here's an example of how you would do this. 00:53:29.233 --> 00:53:33.033 So for the limit of blank determination, we took a calibrator calibrator 00:53:33.033 --> 00:53:36.600 A which had a target concentration of zero zero calcitonin. 00:53:37.100 --> 00:53:40.100 And we analyzed it ten times on each instrument. 00:53:40.466 --> 00:53:42.300 We also took calibrator B 00:53:42.300 --> 00:53:45.800 which had a target concentration of 0.1 nanograms per mole. 00:53:45.800 --> 00:53:48.800 So that would be the low level measure and sample 00:53:48.966 --> 00:53:51.966 and analyzed it three times on each instrument. 00:53:52.500 --> 00:53:54.566 For our limit of quantitation determination 00:53:54.566 --> 00:53:58.500 we took eight calibrator samples and that included four low level 00:53:58.500 --> 00:54:01.500 concentrations and analyzed them over ten days. 00:54:01.766 --> 00:54:05.866 And we compared the results from the statistics to the manufacturer's claim 00:54:06.933 --> 00:54:10.533 and showed that this assay was measuring 00:54:10.533 --> 00:54:13.766 as expected at that low end of quantification. 00:54:15.900 --> 00:54:17.700 These studies can be challenging to do. 00:54:21.166 --> 00:54:21.666 This is the 00:54:21.666 --> 00:54:24.666 last study that's required for method validation. 00:54:25.633 --> 00:54:28.633 And that is a study for analytical specificity, 00:54:28.933 --> 00:54:31.933 also known as interference study. 00:54:32.900 --> 00:54:35.733 So this is important because you're really getting 00:54:35.733 --> 00:54:39.233 at the ability of the assay to correctly identify 00:54:40.000 --> 00:54:43.000 correct to correctly quantify 00:54:43.566 --> 00:54:47.200 something of interest when there could be cross-reactive 00:54:47.200 --> 00:54:50.200 or interfering substances around. 00:54:52.133 --> 00:54:54.333 And as I mentioned, 00:54:54.333 --> 00:54:58.533 this is only a requirement for modified assays or DTS. 00:54:58.933 --> 00:55:02.933 But again, I would argue best practice is to just do it 00:55:03.000 --> 00:55:06.000 even if you have an FDA approved assay. 00:55:06.000 --> 00:55:09.000 And that's because in real life, 00:55:09.400 --> 00:55:11.700 there's almost always 00:55:11.700 --> 00:55:14.300 a percentage of samples that are going to come to your lab 00:55:14.300 --> 00:55:18.000 that have some type of interference present, whether that's 00:55:18.833 --> 00:55:22.066 some hemolysis, because there was traumatic thing, a puncture 00:55:22.500 --> 00:55:25.833 or the patient's on medication that can interfere, etc.. 00:55:28.633 --> 00:55:31.600 So let's talk about what what is an interference. 00:55:31.600 --> 00:55:34.600 An interference occurs when there's a significant difference 00:55:34.600 --> 00:55:38.200 in a test result because of some other component of the sample. 00:55:38.600 --> 00:55:42.300 So there's some other component in there other than the thing that you actually 00:55:42.300 --> 00:55:45.900 want to measure that's causing the measurement to be inaccurate. 00:55:46.433 --> 00:55:49.766 And this can cause a concentration dependent difference in the test. 00:55:50.966 --> 00:55:53.233 So manufacturers are required to screen 00:55:53.233 --> 00:55:56.233 for interfering substances during method development. 00:55:56.566 --> 00:56:01.500 But good idea to just make sure that that all holds up as expected. 00:56:01.666 --> 00:56:04.666 When you've got the tool in your hands. 00:56:04.966 --> 00:56:06.966 So how do you do this? 00:56:06.966 --> 00:56:09.966 You perform a pair of different study. 00:56:10.033 --> 00:56:13.200 So what you're going to do is you're going to take pairs of test samples, 00:56:13.966 --> 00:56:19.200 and you're going to spike one with a potential interfering substance. 00:56:19.500 --> 00:56:22.266 And then you're going to measure the analyte of interest. 00:56:22.266 --> 00:56:24.466 And calculate the difference from baseline. 00:56:24.466 --> 00:56:27.400 It's almost like having, you know, for each set 00:56:27.400 --> 00:56:31.266 you've got your control tube and your your test tube, right. 00:56:31.733 --> 00:56:34.866 So if you see no difference in the quantification of the analyte 00:56:34.900 --> 00:56:38.233 between the tube with the interfering substance 00:56:38.866 --> 00:56:42.566 or potential interfering substance compared to the control tube, 00:56:42.666 --> 00:56:45.766 you can conclude no error from interference. 00:56:47.266 --> 00:56:49.233 You can do this with patient samples. 00:56:49.233 --> 00:56:51.233 In fact, it's best to do it that way. 00:56:51.233 --> 00:56:54.233 You don't want to make sure that you, 00:56:54.600 --> 00:56:55.866 include a sufficient number 00:56:55.866 --> 00:56:58.866 of sample pairs over your measurement range, 00:56:59.100 --> 00:57:02.700 if at all possible, but also within the set. 00:57:02.700 --> 00:57:05.666 Let's say I take five pairs of test samples. 00:57:05.666 --> 00:57:08.700 You really only want one variable to be changing at a time. 00:57:08.933 --> 00:57:11.933 So let's say going back to your calcitonin, 00:57:12.400 --> 00:57:15.400 let's say I've got, 00:57:15.700 --> 00:57:18.900 samples that are artificially generated. 00:57:18.900 --> 00:57:20.300 And the target concentration 00:57:20.300 --> 00:57:23.566 of calcitonin in those samples is two nanograms per ML. 00:57:24.200 --> 00:57:27.700 What I'm going to do is I'm going to spike in increasing concentrations 00:57:27.700 --> 00:57:32.766 of bilirubin over those sets of samples. 00:57:33.100 --> 00:57:36.433 And again I'm quantifying profiles of tone in each pair. 00:57:36.800 --> 00:57:39.566 And I'm comparing a difference from baseline. 00:57:39.566 --> 00:57:42.566 So if I see no difference I can conclude that 00:57:43.366 --> 00:57:45.600 no interference from bilirubin. 00:57:45.600 --> 00:57:49.800 However if the difference from baseline becomes greater than allowable error 00:57:51.033 --> 00:57:52.266 at any point, 00:57:52.266 --> 00:57:57.800 that would be evidence for interference by billing room and on the inst. 00:57:57.900 --> 00:58:00.900 The assays ability to quantify pro calcitonin. 00:58:01.500 --> 00:58:03.733 And that information is helpful 00:58:03.733 --> 00:58:06.733 for having some criteria for sample rejection. 00:58:07.266 --> 00:58:10.800 For patients that have, hyperbole rubidium. 00:58:12.733 --> 00:58:14.866 So I've thrown a lot at you today. 00:58:14.866 --> 00:58:17.866 These are complex concepts. 00:58:19.000 --> 00:58:20.566 And in summary, 00:58:20.566 --> 00:58:25.000 I will remind you that the regulations require performance 00:58:25.000 --> 00:58:28.800 verification of new diagnostic tools prior to patient testing. 00:58:28.800 --> 00:58:32.500 But they don't tell you how to how to meet the regulations. 00:58:32.533 --> 00:58:35.533 There's no one size fits all approach. 00:58:36.700 --> 00:58:38.200 And the largest take home point 00:58:38.200 --> 00:58:41.700 here is really precision accuracy. 00:58:42.000 --> 00:58:44.000 Recordable range and reference intervals 00:58:44.000 --> 00:58:47.266 must be evaluated at minimum for all non WAV tests. 00:58:48.100 --> 00:58:51.133 And if you're in the laboratory developed test world 00:58:51.500 --> 00:58:54.166 or you're modifying an FDA approved assay, don't 00:58:54.166 --> 00:58:58.566 forget about analytical sensitivity and analytical specificity. 00:58:59.733 --> 00:59:00.933 Thank you for your attention.