For our first session this morning, we will have three speakers. Dr. Resak will speak on the IOM report, a call to action. Dr. Duncan will speak on an overview, scale and scope of error in diagnosis, and then Dr. Davenport will speak upon the diagnostic process more than just interpreting images. Thank you, Lane, and good morning, everybody. I was asked to address the audience on the IOM report, a call to action, as I was a member of the committee that wrote the report, and would like to share with you what really the report is all about and how does it affect us in radiology. The IOM is very committed to the improvement of the healthcare in America and actually has one of their very high priority is Quality Chasm Series. It is Institute of Medicine, which is now called National Institute of Medicine. The very first report, which was to error is human, was highly publicized in 2000. It followed by crossing the chasm and on quality chasm in 2001, and then the 2015 report that actually it's a continuum was a call to action. The very first report received high publicity from the media, healthcare professionals, as well as the policy makers. It was the first time that publicly it said that experts that it estimated that as many as 98,000 people die in any given year from medical errors that occur in the hospital. This did not include ambulatory errors. And this is more than those people that die from motor vehicle accidents, breast cancer, or AIDS, and all three of those always receive great attention. The report further go to say medical errors may be the country's third leading cause of death. And that is really what media tremendously picked up and public became aware of. So adding the financial cost to the human tragedy and medical error, it is seen how it went up to the top ranks of urgent widespread public problems. And it also brought the message to our own community in radiology. Then the second report crossing the quality chasm really looked at the elements that contribute to medical errors. And they include that we have to practice safe. Our practice has to be patient-centered, evidence-based, timely, efficient, cost-effective, and equitable. It's nice to identify the elements that are essential. But even more important is then to say, and what do we do about it? And this is why the 2015 report is called to action. And identify that the diagnostic process acknowledged is complex. It is collaborative. And it involves clinical reasoning and information gathering. Certainly diagnostic process is extremely complex. And it's getting more complex every day. It is complex because it used to be more or less confined to health care professionals. Today diagnostic process really shows a convergence of life science, physical science, and engineering. And we are not alone in that process. The process involves diagnostic tools from imaging, pathology, laboratory medicine. But equally important are clinical trials, machine learning, proteomics, bioinformatics. It's really today a team that participate in the diagnostic process. The larger the team, the greater probability for mistakes along the line. So improving diagnosis in health care and the call to action said that it will require collaboration and widespread commitment to change. And that change has to involve health care professionals, health care organization, policy makers, researchers, and extremely important patients and their families. So it's a team that now includes not only our own teams in the hospital or the community practice, but it has to involve policy maker, patients, and their families. So according to the 2015 IOM report, what percentage of adults who seek now outpatient care, not the hospital care, in the United States experience diagnostic errors? Do you think it is 1%, 5%, 10%, or 15%? The report did due diligence evaluation of the data and approximately 5% of adults who seek outpatient care in the United States experience diagnostic error. That's a large number considering the number of visits that occurs in the ambulatory care. What is the definition of diagnostic care? And that's very important to us as a radiologist. So the report, we had a long discussion. What is really the definition of the error? And then it really became obvious that we have to look at the definition of diagnostic error from patient perspective, not from the physician perspective, and not from the healthcare policy makers. So diagnostic error is a failure to establish accurate and timely, and that's where we are affected on both fronts, accurate and timely. A report that is given 10 days after the study was done, it's really an error because it causes delay in diagnosis. Second, you have to communicate that explanation. And very interesting, you have to communicate to the patient. So it isn't just communicating the report to the physician, but communicate to a patient. And that had a lot of discussion because especially in radiology, we are sometimes limited how to communicate to the patient. But if things go wrong, we are equally responsible. So it's something for us in our community to think about. Causes of diagnostic error include insufficient collaboration and communication between physicians, patients, and their family, and other inappropriate ordering and incorrect execution or interpretation of diagnostic tests. Number one cause for diagnostic errors is communication. Communication at every level. So what is listed as a key cause of diagnostic error in the report? Is it discussion between excessive number of the physicians? Is it discussion between physicians and patients? It is discussion between the physicians, patients, and their families? Or insufficient use of evidence-based practice guidelines? As we looked at the reports and the errors, it became obvious that the number one cause of diagnostic error actually includes failure to include family members. And we should really say family members or friends because patients don't always have family that comes with them. Family members in discussion of patient's health condition. It became so obvious if you discuss the problem with a patient alone, half of that he or she will not remember, will not truly understand, and very often it's uncomfortable coming back and asking more questions. So involving family or friends in the discussion of the healthcare problems is number one cause for the errors. Communication is not easy. There is an excellent quote, the single biggest problem in communication is the illusion that it has taken place. And this is why we have changed our rules for the urgent results where you give the results and somebody has to repeat, where you have to document that you gave the results. Because if you just talk, very often that message has not been communicated. So looking at the errors and what really happens, I like to look at the Crico data. Crico is a very large insurance company that covers Harvard Medical School. What we all like about Crico is not only that they have numbers of the errors or the problems, number of the claims, but they continuously analyze their data. In their studies, clinical judgment is the number one cause for medical errors. But look what clinical judgment includes. Includes frequency in ordering the incorrect diagnostic test. So how does that refer to radiology? Very much so, because inappropriate ordering is a second cause of medical errors. Inappropriate ordering, radiology has much higher volume than pathology, so diagnostic errors in radiology because of inappropriate ordering are very high on the list. But we have help. This time we have help from the government. And the help is really 2014 introduction of PAMA, Protecting Access to Medicare Act. Many in radiology, in our own community, really either didn't understand, didn't like, or think they can ignore PAMA. So PAMA is actually, for radiology, one of our biggest friends. Because PAMA, starting in 2020, will require mandatory use of appropriate use criteria. We are certified CDS mechanism. I know it is work. It is also additional resource. But in the long run, it is wonderful for radiology. The key requirement is to establish evidence between criteria by linking a clinical condition to appropriate imaging test. I know that not everybody is crazy about appropriate use criteria because you think it's too general. Maybe it doesn't apply to your practice. But look at the interest. So qualified provider-led entities. It was a call from the CMS. You had to apply to be a qualified provider-led entities. And as of July 2017, there are 18 sites, which means there are 18 CMS-certified sites to provide appropriate use criteria. I love the diversity of those sites. The diversity shows you that there are societies, such as American College of Cardiology, American College of Radiology, National Comprehensive Cancer Center. There are universities, of course. There are tertiary cancer centers, such as Memorial Sloan-Kettering. But there are also community sites, big community places, such as Intermountain Healthcare. And then there are now popping up institutes that want to do nothing but develop appropriate use criteria. As we move forward, we will really see what works and doesn't work. We at Memorial decided to be a qualified provider-led entities, while NCCN is wonderful. But it's static. It's at the time of diagnosis, at the time of staging, at the time you want to look for recurrence. But we need to provide continuum of appropriate criteria because, for example, a headache in a cancer patient or abdominal pain in a cancer patient should be dealt with differently. It's an emergency. Abdominal pain in a cancer patient, not after surgery, but sometimes down the line, needs direct CT. Doing anything else is a waste of time and delay in diagnosis. And this is why we want to provide a continuum of care. So I think this is one of the best things that happened to radiology because we are really going to work on appropriate use criteria, and it will diminish, not totally prevent, but diminish diagnostic errors. And then it's incurrent execution or interpretation of diagnostic tests. It is a huge problem. As you can see, it's a number two problem for CRICCO when CRICCO analyze their data. And that is really us. Pathology plays considerably smaller role. So CRICCO comparative benchmark system is called CBS. Their claims analysis of diagnosis-related allegations shows that the most is medicine. Interesting medicine ambulatory errors, those that go to the claims, are even higher than inpatient. Number two is surgery, and number three is us, radiology. In radiology, the larger number of errors is seen in the ambulatory setting than it is in inpatient. Because majority of radiology is done in the ambulatory setting, this is very important for us. And then you see pathology is actually very low on their list. So what in radiology really brings all those errors and the claims? You have to know that claims in radiology carry a very large price because you look back in the image and the finding is there. We carry a higher price per claim than, let's say, surgery, that it was never there. I didn't see it, so therefore it didn't exist. So in radiology, the largest number of claims are driven by events in diagnostic radiology, 86%, followed by interventional and then followed by nuclear medicine. We are already seeing a rise in nuclear medicine because of the theranostics and larger use of PET-CT. So it will get there fast. I personally was very surprised when they showed the data that the number one error in radiology and number one reason for the claims is misinterpretation or not seeing the findings on CT. It is no longer mammography. It is CT. Furthermore, in CT, in the ranking order, is abdomen, chest, head, and chest. So since CT of the abdomen, I'm sure denominator plays a role, but we always felt that, oh, mammography, it's a high risk. Not anymore. The highest risk is CT, followed by chest, head, by radiography. In radiography, the highest chance is trauma, missing subtle fractures that result in deformity. And then mammography is number three. Actually, mammography and MRI are at 10%, so they share three and four. So for all the mammographers, they say, well, they still have their claims, but the CT and the abdomen is really something we have to very hard work on. So how can we improve? What are the elements to make us make fewer errors as everybody is reading more and more abdominal CTs? So one, I'll just take one because there is an entire morning dedicated to the improving quality of care. And that is misinterpretation in diagnostic study, a high. Problems are multifactorial. We all know that. And they range from lack of clinical history to misinterpretation of diagnostic studies. So importance of obtaining adequate clinical history. It is difficult for us, but we really have to pay more attention and have greater efforts. We think that informatics and all the new tools will help in this area, but that is three to five years away. So I'll give you an example. This is an 80-year-old female with de-differentiated liposarcoma in the left shoulder. Sarcoma was resected. And a follow-up study, obviously, there is a finding in the chest, impressions are suspicious for minimally invasive adenocarcinoma versus infection. Patient was transferred to memorial for biopsy directly into the interventional. Biopsy was performed two weeks after chest CT, and it shows changes related to radiation therapy. When we looked at the history, there was a history of RT, which was not given to a radiologist that read the scan. It wasn't given to us either at the time of biopsy. So lack of adequate clinical information comes to communication, but there are so many arrows because we don't have correct information. So I call that now Monday morning quarterback, when somebody comes very smart and says, oh, the proximity of the lung finding to the surgical site should have prompted further inquiry. Easy to say after it was all done. But if we would have the diagnosis at the time the chest CT was read, the outcome would be very different. A different one. This is a case of melanoma. March and then follow-up in October. Now these findings in October, obviously they are findings, and in the patient says soft tissue in the right axilla, suspicious for tumor. Now this is a touch different. It is one, there was a lack of communication, and that is that patient had interval axillary lymph node dissection. But there is here a combination of Monday morning quarterback, but really also lack of knowledge. So we can blame lack of history, but we also have to understand that our knowledge has to be there. New finding doesn't look like melanoma. It should have prompted search for additional history, which it didn't. So it's those combinations. But I can tell, if this would ever go to the court, we would have very difficult time defending that this looks like melanoma metastasis. So there is a combination of both findings. Actually, when you look at the added value of second opinion review at a comprehensive cancer center, since I come from oncology, I only have cancer data and cancer examples. But you can see that actually when an expert reads the study, and those are all published studies, change in management is pretty drastic. Not just disagreement, but change in management. What this really tells me that when change in management can be as high as 50%, something is wrong with our system, our continuous education, and our ability to really understand the images, and also explain why misinterpretation is so high on the diagnostic claims. So let's look at this case and show how knowledge is absolutely essential to make the correct call, even when the information may be incomplete. This patient is 65-year-old, has Hodgkin lymphoma in the chest, and we have Dr. Casaruni there, and I'm sure when she looks at this, and the impression says metastasis, FDG, avid osseous lesion in the left iliac bone, she'll say, no way. And she'll, regardless of the fact that skip, so what's a knowledge-based? The knowledge-based is that skip metastasis to the pelvic bone in Hodgkin lymphoma in the chest, it's reportable. Single bone metastasis in a distant site, it's reportable. It's not something you would see. So search for more information, view serial imaging, and then really you have to investigate and code, and this is really FDG activity due to post-biopsy inflammation around the bone marrow biopsy tract. A huge difference for patient management. So as the complexity of imaging and medicine continues to increase, fostering subspecialty training and participation in multidisciplinary disease management is crucial and essential. It cannot be avoided. So how we do that and how we implement, I don't know. But developing culture and mechanism that promote discussing arrow, not only with administrators, but with each other is one way of continuously have a peer learning. So there are challenges, no question, but there are also unprecedented opportunities. Subspecialization, report standardization, lexicon, communication, visibility, MDT, but I really think that one of the greatest help as we look forward is going to come from the field of radiology informatics, AI machine learning, and cognitive computing. That analytical tools of cognitive computing are here. They are not evenly distributed, but they will be innovations and technology will facilitate reporting and bring us new ways to communicate. But we have to remember, they will never be able communicate for us. So knowledge and communication are going to remain a key factors in our ability to diminish diagnostic errors. Thank you very much for your attention. I'm lucky to have the opportunity and thankful to talk about the scale and scope of error in diagnosis. So some background, the IOM report in 2015, again, defines the error. And I like this, it's an operational definition. It's from the patient's perspective to accurately and a timely explanation of the health problem and communicate that explanation to the patient. And they estimate in our lifetimes, all of us will experience a meaningful diagnostic error. And I'd ask for just a quick show of hands, how many of you in the audience have already experienced a meaningful diagnostic error? I'm seeing maybe 30%. How many of you have had a family member that has had a meaningful diagnostic error? Probably 75 to 80%. And so if you're on Twitter and you're brave and want to put up what you think was a meaningful diagnostic error, please use the hashtag improved diagnosis for RSNA 2017. But it's worthwhile taking sort of an overview of the diagnostic process and thinking about it, it's more than data interpretation. I know we're radiologists, we like to look at the images and render an interpretation, but we need to think about more of the patient's journey through this process. And I've used this diagram before, that patients start out with some sort of need. If they can address it, and I am the biggest example, if I don't have to go to the hospital, if there's any way I can heal myself, I come, you know, do it, okay? It's only when it exceeds our internal capacity that we encounter the healthcare system. And with that, there's some clinical uncertainty, which is really the diagnosis. If there's no uncertainty, you progress quickly to treatment, but if there is uncertainty, there's typically a call for data, okay? And it might be additional history, it might be laboratory testing, and of course it involves imaging. And so with that request, the information gets, or data gets gathered, and then, I mean, we have to interpret data and transform it into information. And with that, if we have information, hopefully it'll address the diagnostic uncertainty. If it doesn't, typically go through another cycle. I mean, how many times is the first test, and then there's a second, a third, a fourth, and then arriving with enough answers, or enough reduction of clinical uncertainty that you progress to treatment, if the treatment's effective, the instigating symptom or need is quenched, and the patient will leave if it's not. And how often does this happen? That it might be a failure in diagnosis, it might be a failure of the treatment modality, or treatments are not 100% effective in every case, and sometimes that will lead to a failure in diagnosis. And sometimes that will lead to additional cycles of diagnostic testing. And you can see where you can go back and forth through this loop time and time again. There's a lot of thought, well, diagnosis, you know, it starts when the patient enters, it continues, this data is gathered continuously. There's a lot of errors that occur when we're just monitoring response to therapy, okay? That there's continuing updates on what the patient's clinical condition, and how might we get them really out of our healthcare system and back on with their lives? So the goal of diagnostic process is not attaining diagnostic certainty, okay? We're not trying to reach 100%. Really, the goal should be reducing uncertainty enough to allow the next step, which is often treatment. I mean, thinking about the procedures that I do in interventional radiology, I don't have certainty about where that catheter tip is. I have a pretty good idea, and I've reduced it by fluoroscopy or some other imaging modality, but it's enough for me to continue on with the procedure. Now, there's some really nice data from laboratory medicine about their steps in the diagnostic process, and they've broken it down into five separate steps. They call it pre-preanalytic, which is the choosing of a test, preanalytic, collecting the specimen, the analytic, where they actually put it in the machine and the machine whirls, and then it spits out a number. Post-analytic is taking that number and communicating it, and post-postanalytic is taking those results and applying it to the patient's care. What's interesting is that they say, and it's probably quite similar to us, at least in a couple places, that pre-pre and post are common, with an error rate probably above 5%. They actually have some really nice data from a study that I've referenced showing that the pre-analytic, collecting the specimen, they have an error rate, but it's less than 1%. When they put it into the machines to spit out that number, it's like .05%, okay? They're lucky, they have machines that do all their analysis. And then communicating, it's a little bit higher, but still quite low. And some examples, pre-pre and post-post from laboratory medicine. Given a clinical scenario, what diagnostic test should be ordered? Have any of you really gone on the lab recently and tried to order an exam, a blood test? I mean, it is pages and pages of opportunities, and it's pretty easy, you know, if I'm thinking factor V, am I testing for factor V Leiden or really the factor V level, okay? Two very different tests. They sound pretty similar to me, and I've made these type of errors quite commonly. And the error rate they estimate is greater than 5%. It used to be that a diagnostic test, it was pretty easy to interpret. It was a potassium of 5.7, and it's, oh my, that's hyperkalemia. You know, I know what the next step in treatment is. But now when you talk about things like the entire sequence for the hemophilia A gene, try and interpret that. That's probably closer to what we, I mean, we get the image. You have to go through it and interpret it and try and figure out what it means for this particular patient. And so their tests are actually getting complex, much like ours. Pre-analytic, okay? So these are now slightly lower, but one of the higher failure modes, at least failure rates for laboratory medicine. The orders placed and received in the lab. How many of you have done a biopsy and not known what container it goes into? Is it glutaraldehyde, formaldehyde, that purple, that red stuff, saline? You know, all these different choices, I've put it in the wrong container, okay? And so it's easy, even though I did a terrific biopsy, I put my specimen in the wrong container, it leads to a non-result. The volume, okay, that you, I mean, how many times have you done a biopsy and not had sufficient tissue, okay, for diagnosis? Patient preparation, especially for many of the lab tests. The specimen collection. It's not uncommon, at least when they were looking at their error rate and got 0.2%, that they found examples where a blood sample was collected in the antecubital fossa, there's a peripheral IV running down in the arm, and so it's diluting your sample and obviously changing the chemistry and the result because of that. Specimen handling, transporting those specimens to labs. Yeah, they get lost, but also they get mishandled before they're fed into the analyzer. We recently had a sentinel event, it was a venous sampling procedure. They did a terrific job of collecting multiple time samples, right, left, peripheral, a whole series after the stimulating event, getting them all to the lab on ice and handing them over to the lab technician who then spins them down and unfortunately has to take the serum off each one of those samples into another tube before it goes into the analyzer. Guess what happened? They mixed up which tube it went into and they lost track of right, left, and the timing and it really led to a non-diagnostic result. And so human error affects them just like it can affect us, that there are multiple opportunities every step of the way. So for radiology, what can happen to us? Well, clearly choosing the imaging study, clinical decision support, hopefully we'll address this, but boy, it's gonna be hard to get it down to 0.01% when you think about choosing the appropriate imaging study. Protocoling and acquiring the images. I don't know, I've seen some posters and people talking about repeat rates in radiography and some of the other failure modes and if it's 5%, it's probably a pretty good month if we have a 5% error rate for all the steps between getting an order and getting the images to you to do the analysis. And we'll talk a couple different examples from our practice of what kind of errors we've seen. The image interpretation, it depends. I guess in the second opinion, I saw numbers there as high as 50% of discordance with reading and interpreting the data that was available or it might be that there wasn't enough data available when you're interpreting the image and integrating and trying to tell a story. But one report looking at a whole series of different studies saw, you know, it varies widely and it probably depends on whether you're a zealot and count everything as a discrepancy in image interpretation or how much leeway you allow. Post-analytic, how many times do we not get the results to the right people? In time, okay? It's probably higher than this 0.7%. And finally, applying the results of the imaging study to the patient's care and leading them out of the healthcare system because of it. So upstream errors, pre-pre-analytic. I actually worked in a practice where they would routinely order skull X-rays instead of head CTs to evaluate for intracranial trauma. Boy, I mean, that was just frustrating. But it happens. Probably a more subtle exam would be, or subtle example would be child with minor head trauma, do they really need a head CT? If you follow certain guidelines, they don't. But many times, and I was an intern at one point, I probably would have ordered that head CT with and without contrast. That would go down as a wrong order, okay? That was an upstream error in sending this order to radiology. In the pre-analytic phase, wrong patient, okay? Somebody walks out into the waiting room, asks for Mrs. Smith to come back for her chest X-ray. There are two Mrs. Smiths sitting there, and the first one up gets to go. And we've seen that. Wrong body part, right versus left. Ankle instead of foot, ankle instead of calcaneus. Imaging protocols, you know, it's supposed to be with and without contrast, they only did the with, or, you know, again, all these different modes, you can start thinking about all the different opportunities for errors. We've had errors where all the images don't get to PAX, okay, that they're deleted from a scanner before they're uploaded to PAX. That would really go down as a type of diagnostic error. It's similar to the lab specimen not being sufficient when it gets to the lab. Incomplete studies, we've had examples where it's a pre and post contrast exam. The pre images get loaded to the PAX, and somebody opens that study, interprets it. With the pre, the post images don't come over until later, and now that study's actually closed and dictated, and it's found later on when somebody, when the neurosurgeon shows up and says, well, what did the contrast portion of the study show? And then hanging protocols, simple little things. We had a Sentinel event several years ago where there was a chest X-ray that was actually flipped as it was presented into the PAX because there was an opportunity to do that. It wasn't recognized. The mediastinal anatomy was such that it was pretty vertical. The patient had a pneumothorax. The chest tube goes in on the wrong side. It's not until the next morning that it's discovered that the pneumothorax is actually on the other side, and the patient actually had a complication from their bilateral chest tubes. Post-analytic, the downstream, because there's actually a separate session that will focus really on the image interpretation traps that we all fall into. But examples of some downstream errors, distributing your report. How many of you have had a typo in the last month in one of your reports? Okay. How many of you have had a report that went to the wrong provider? I'm the only one, okay. That doesn't get to the right person that really acts on it. We have examples where it goes to the subspecialist but not to the family practitioner who actually needs that result when the patient's discharged and following up on it. We don't have a good accounting of all the people that are involved that might need access to this report, and so therefore don't act on it. Errors in applying the results to your patient's care. So they have to decode your report, and I love the George Bernard Shaw quote as well, that the single biggest problem is communication. So here's a, oh, just a scenario. You're reading out a CT exam on this patient, they're post-op, they have a fever, they have a white count, and of course the surgical team is looking for the abscess. You're looking and you see that there's a DVT involving the superficial femoral vein. Choose your words, put together your report, this is important, so you're gonna call. And so when you call and get the surgical nurse practitioner on the phone, you say there's a clot in the superficial femoral vein, but it's a noisy nurse's station or it's a bad connection or something, leads the surgical nurse practitioner to only hear clot, superficial vein. They decode that, and they say, well, this is an incidental finding, it's a clot in a superficial vein, and you might actually even get the feedback that, yes, I've received the message that this is a venous clot and we'll address this after the acute problems have resolved for this particular patient, and think you've done your job, now this patient's on the right course. Unfortunately, and this has happened in our system, the nurse practitioner will say it's a superficial vein, it's sort of, you know, it's like, you know, a little venous bleb on the skin surface, and discount it, not recognize that the superficial femoral vein clot there is a DVT that warrants more aggressive treatment. And so it is this, you know, communication, it's tenuous, it breaks down, it's so easy to subvert. So there are numerous failure modes in the diagnostic process. This is an artist's rendering of me visiting the Midvale School for the Gifted. I sat there and I pushed on that door for the better part of five minutes. Because I'm a human and I make errors. There's a 2% error rate for simple tasks performed rapidly or given scant attention. And actually, reason in this book has some modifiers that go up and down if you wanna see how commonly humans make different types of errors under different situations. We definitely fall into traps because we have poorly designed systems. This one is a disaster waiting to happen, right? Liquor, guns, and ammo with easy credit so you can buy more, okay? And probably use these in this order, liquor, guns, and ammo. And then obviously, these are the right steps but they're probably out of sequence, the ready, fire, aim paradigm. Errors are common, we should expect them, okay? But we need a better process, probably for handling them. And so when you think about how you're going to improve the performance of the systems that you live and work in, there are all these potential errors, there are all these failure modes, right versus left. Our electronic medical records, somebody figured out that the scroll bar on the side, as you went down and filled out the order, the right versus left, there's actually a chance that the software would misinterpret your click on the scroll bar as a flip from right to left. And so we were getting a lot of reversals in our right versus left, yeah. And they actually knew about it. But the problem was that the new person, that wasn't part of their orientation, that you need to recheck your order after you've touched that scroll bar as you go down through this to make sure that it's still the study that you wanted. But the failure modes, the modes themselves, the underlying problems are fairly universal. Right versus left is the easy one. The failure rates and how often they impact your patients are really local because most of us have erected systems and control mechanisms to try and ameliorate or try and diminish the impact. And this leads me to this failure mode and effect analysis. This idea, entropy wins, okay? Failure is inevitable. Murphy was absolutely right, spot on. Failure is inevitable. But we can recognize and say we can minimize the impact. We can reduce the frequency of failure, okay? Design systems that make it a little bit harder to fail, okay, the gas tank, you know, that you can't fill up your cars now with diesel or unleaded gasoline because they've made the connections impossible, that reduces the frequency of failure. Minimize the impact of failure when it does occur. My favorite are the pop-off gas hoses at the station because they know that people will drive away before pulling the nozzle out of their tank. And so when that failure occurs, it doesn't lead to catastrophe, which was gas stations exploding. And then improve the ability to detect the failure mode before it leads to catastrophe. So detect it early enough in the process that you can actually step in and reverse it. In the classic priority score for a failure mode is to take its frequency, its severity, and its ability to escape detection, multiply them together, and the worst ones, the really ones that you should be working on should rise to the top. So thinking about what are the priority areas in your practice? Is it pre-pre-analytic? If it is, clinical decision support is probably part of the solution. But how well will it work? And the part would be really analyzing how often we have bad orders coming through our system that have to be corrected. And did clinical decision support, are the algorithms, is the system, is it working to improve performance? Pre-analytic, the protocoling and acquiring of images, we have to work upstream with our technologists, our referring clinicians. It really, it's the epitome of teamwork to actually get those studies into our hands that are ready for interpretation. Again, there's another session on the failure modes within image interpretation, but clearly knowledge, prior information, the upstream information about this patient's history so that it's more of a story rather than it is a single word on a page is important. Attention, avoiding interruptions, okay, as methods to improve the accuracy of image interpretation. Communicating results downstream, the informatics. We don't have a great system, at least at my institution, for knowing who all the people that need this report, how to get the important reports to the right people, and keeping it live with the follow-up, tracking follow-up recommendations until that signal is extinguished. We've created this patient has something that we're worried about. How good are we at really following to see if the problem has been taken care of? Do we just wait for the next study? You know, the DVT that you've seen in the last three CT scans that's not uncoagulated or doesn't have a filter. So the diagnostic process, it's complicated, it's prone to errors, and I think we really need to look beyond image interpretation. It's going to require a lot of teamwork. It's going to, leveraging the upstream information that we have, okay, and now that we have more and more electronic medical records that it actually flows to us so that we can actually use it. It's our technologists and the informatics systems that we're working with, and finally closing the loop for our patients so they can get out of the healthcare system and get on with their lives. I'm going to talk to you a little bit about the diagnostic process. I'm going to make a few assumptions just to make sure we're all on the same page. I'm going to assume that you're a diagnostic radiologist, but if you're an interventional radiologist, I think all this stuff remains relevant to you. I'm going to assume that you spend most of your days reading images and creating reports. I'm going to assume that you care about your craft and about your patients. So there are a lot of things that feed into the process of making an image. The thoughtfulness that goes into deciding what the right test is, all of the possible errors that can occur when you select that right test, but I'm going to focus for this talk on those two green boxes. We have imaging, we're interpreting it, and then we're going to give that interpretation to somebody else. So imaging's performed, we interpret it, we give that interpretation to somebody else, and then some clinical action is taken. And the question that sometimes you might have been asked, are we just middle managers here gumming up the works? I suspect in your institution you might have a neurosurgeon or an orthopedic surgeon or a fill-in-the-blank provider who feels like their diagnostic skills are probably equal or better than yours, and their argument might be, well, why are you here? I'm just going to overread your interpretation, and if you find something I didn't find, it probably wasn't actionable in the first place. So if that's the case, wouldn't lean thinking demand we just get chopped out of the process? What is it that we do here? So in my opinion, and I think other people may share this opinion, is that fundamentally we're translators. We take information from an image, and we translate that information into words. And this is the late Dr. Barron who said, no matter how many great new technological developments are implemented by radiologists, meaningful clinical effect and outcomes will only come because the radiologist is an outstanding translator of the language of images. And I think that that's true. But what does that really mean? It means that we translate imaging data into words that influence providers to take action on the basis of that information. But if all we do is translate pixel data into word data, then we are replaceable. This is a pigeon, and they are sitting in a PAX workstation, and they are sitting on a grid, and that grid probably shocks them if they get the wrong answer. And what they're looking at is a slide of breast cancer cells. And the pigeon has been asked, pigeon, please look at these cells and tell me if this is a cancer. That's a function that pathologists usually do, not pigeons. How well does the pigeon do? This is an ROC curve, and each of those letter and number combinations are individual pigeons. And you can see what their individual ROC curves look like. And that black line is a flock of pigeons. If you take all of the pigeons working together as a team to try and figure out if that's a breast cancer cell, look at that ROC curve. That's pretty amazing. And I don't know about you, but I wish I had that flock's ROC curve on basically any of the things that I do. And they're just birds. Now I think that most people would say our jobs are not in threat by birds. I think flocks of computers are probably a better threat or a more relevant threat to us. This is a study that we did at our institution where we tried to predict whether after a bladder cancer was taken out, if it would be pathologic T0, meaning I've taken the bladder out, there's no cancer detectable. Why is that information potentially useful? Well, if we could figure that out ahead of time, maybe we could do bladder sparing therapy. This is not a function that radiologists currently try to do, but we tried to predict it. How well does the computer do? Well, the computer is this green line here. On the y-axis, we have the area under the curve for their diagnostic performance. And on the x-axis, we have the different human observers, and this represented a combination of radiologists, oncologists, and urologists. How well did they do? Well, you can see not so great. And in fact, without the machine, the humans did substantially worse than the machine. And this is my favorite part of this study. If the humans were then told what the computer said, so the human gave their impression, they were then informed what the computer said. No, the computer thinks this is the right answer. The humans dragged down the computer's performance because they believed that they were right. I don't want to be replaced by a pigeon, and I don't want to be replaced by a machine. I don't think that I will be. I'm actually excited about machine learning and think it's going to be a huge advantage for our field. But I think there's never been a more important time for us to place value on what we do beyond just interpreting pixels on the screen. And there are several ways I think you can do this. You can integrate data across the medical record. You can solve problems, not just create reports. You can form relationships and be responsive to people's needs. If someone emails you at an off hour, it may be a good idea to answer that email. And think like your customers. And of course, we have many different customers, referring providers, patients, et cetera. This is an example of a way that we can push back against the idea that all we do is interpret images. This was a prospective QI study investigating the effect of in-person communication on diagnosis and care decisions. And this is what we did. We had 100 patients who were being managed by acute care surgeons who already had finalized radiology reports. And the surgical team came down and said, here's what we think is going on. We recorded that information. Here is our plan. We recorded that information. And then we said, let's have a conversation about what's going on. So we reviewed the entire radiology chart for this patient. We talked about their medical and surgical problems and tried to integrate that information together. And then after that was done, we said, OK, what's your diagnosis now? And what is your plan? Well, the key finding from this is that major changes were common. So of these 100 prospective consecutive patients, 43% of the time there was a change in management and 43% of the time they changed their impression. And interestingly, that was not the same 43 cases. They were substantially overlapped, but not exactly the same. And 19% had a change in their operative plan. So I'm going to make an incision here versus there, or I am going to operate versus not going to operate. So why does this kind of thing work? Well, we're forming this shared mental model about what's actually going on with the patient. This is an example from that study where we looked at two different time points on a patient who had a small bowel obstruction, small bowel obstruction on time point A, small bowel obstruction on time point B. And we talked about the case. They were currently pursuing conservative management because it worked well the first time. And during that discussion, the radiologist said, yeah, actually, intraluminally, that's actually the same transition point. It's really not different between the two when you look at it from an intraluminal standpoint. And the surgeon said, well, I didn't know that that was the case. I probably would have acted differently if I knew it was the same transition point. I would have operated rather than trying conservative management. And the radiologist said, oh, I didn't know that was important to you. We probably should have put that in our report. It may shock you, but we are human, and we are social, and we like to interact with each other. And sometimes you can't even predict what the result of that interaction is going to be until the interaction has occurred. Tumor boards and data like I just showed you are perfect examples of how radiologists aren't just pixel counters. If we talk to people and share information, we can add a tremendous amount of value. I think integrating data to support command-level decisions is where we do the best, where we shine the brightest. When someone's asking you for your expertise, and you share that by integrating information across a lot of different disciplines and say, this is what we should do next. It is important to translate imaging data into words, but it's not our only job. And I would argue it's not our most important job. Our job is to influence care through imaging. Now after we've interpreted an imaging study and created a report, we think, OK, great. Now we have the truth. The truth is my report. Well, the report is not the final arbiter of truth. You probably are aware of that. I think we saw a good table in one of the earlier talks about how a lot of times we disagree with each other about what the right thing is. So I want you to consider the following thought experiment. We have an omnipotent being, and they are crafting flawless radiology reports. No errors. Everything is 100% correct. They know exactly what they're doing. But the referring providers just don't trust them. Maybe they're new, or maybe they don't have as much experience, or maybe they rubbed someone the wrong way, or maybe they don't respond on their emails on time, or maybe they're just distant, or fill in the blank of the reason why they would be not trusted. I promise you that it doesn't matter what the report says. They're going to immediately ask that person's partner to re-review the case to make sure that that first person was right. So it is important to be correct, of course, but it's also important to have the trust of the people who are receiving that information, otherwise you're not delivering any real content or changing any behavior on the point of view of the provider. If you are untrusted, you're simply creating a report that can be billed. You're not actually influencing decision making. And you might be surprised, because even when we're saying the same things, I use a similar kind of word to you, words can be a little slippery, and maybe we're not saying the same thing. So do our words mean what we think they mean? This is a little study that was done looking at whether radiologists define concordance the same way. Many of you probably have heard of or used the Radpeer system, where someone assigns a score of one indicating concordance, and then everyone says, great, we're all saying the same thing. Well, not so fast. So this is a study of 119 consecutive preliminary and final reports which were prospectively assigned the concordant grade. Well, we took those cases and gave them to other people and blinded them and jumbled them all up and said, can you compare these two reports, which were originally assigned a concordant grade, we didn't tell them that, and tell me what you think is concordant or discordant about it. Now, of course, the original discrepancy rate is 0%, because they're all graded concordant. Well, basically, they don't agree. So when we show them to emergency medicine doctors or internal medicine doctors or, frankly speaking, even radiologists who weren't involved in that prospective assignment, we really disagree about what concordance actually means. And in order for you to be on the chart here, you have to have at least two of three people independently agree that the score is not a one. So at least two of three emergency medicine physicians thought that there was a major discrepancy 31% of the time. That's a huge number. It's not like it's 2%. That's a huge number. Internal medicine was something like 14%, major discrepancy present. So what are some examples of ways in which we think we're saying the same thing, but they might disagree? So I'll just pick one of these here. History. Sepsis, one week post-op. The resident report said, portal venous gas with pneumatosis and acute ischemia of the gastric fundus. The abridged faculty report said, portal venous gas with no evidence of gastric ischemia. Why would they rate that as concordant? Well, maybe they thought it's got, they said portal venous gas, that's kind of bad. And they sort of said pneumatosis, but I disagree with them a little bit about that, but the source of the portal venous gas we disagree with, but portal venous gas is bad, so concordant. And the referring provider said, no, actually knowing the source of the portal venous gas is highly relevant to us. Another example, dementia, abdominal pain. Resident report said no acute abnormality. Faculty report said, well, there's some little smuts in the lung base. Well, from our point of view, which we throw that disclaimer in probably on more cases than we should, from the point of view of the referring provider, that's a significant change because they're giving a diagnosis that might explain why the person's having troubles. This study interested me not because they disagree with this, I sort of expected them to disagree with this. It interested me because we oftentimes don't even know what has clinical importance. So when we say something that sounds a little different or we use a different word, the person receiving that information receives it differently depending on who the person is that's saying it and how they say it. And I think we can fix this through aggressive collaboration. By asking people how they want us to report things and standardizing what we say. So does it matter if we say different things sometimes? I would say yes. I'm going to give you some examples here. So I do some liver imaging. I'm going to give you an example of different ways 10 years ago you might describe a liver lesion in the setting of cirrhosis. You might say right lobe liver lesion cannot exclude HCC. What does that mean? I don't know. Small liver lesion, right lobe, HCC possible. Does that mean 1% possible, 99% possible? What does that mean? Right lobe nodule indeterminate for HCC and so on. You get the point. Different ways of trying to ascribe significance to a finding but without doing it any specific way leaves the referring provider uncertain of how to handle this. Descriptive text causes noise and uncertainty. Every cloud of uncertainty we put around a finding in our report gets added to the clouds of uncertainty already existing in the pretest probability setting which leads to more uncertainty and diagnostic error. So I'm going to share with you some emails that I got because they're funny. This was from 2011 pre-LIRADS. This is a hepatologist at my hospital. As discussed, I no longer know how to interpret radiology reports from our own hospital. The inconsistencies among our own radiologists make it very difficult for us to manage our patients. That's not a nice thing to get. Here's another one. I discussed his management plan at liver transplant eval meeting yesterday and the group agreed with my interpretation of your reread of another radiologist's read. Hope I didn't misinterpret you. And I think this probably resonates with a lot of people in the audience. How often do you get asked to look at a case from somebody else and then you play the telephone game with the person on the other end who then plays a telephone game with their subordinates or their team members and it's a very complicated process which leads to errors. In all those cases, we translated the imaging into words but then we made a mess of it. So we were using English but we were speaking different languages and not communicating things in a standardized way and we didn't understand the implications of what we're saying. What does it mean to a hepatologist when I say possible HCC? What are the actions that happen when I say that? So now we all speak LIRADS so we're all fixed, right? Everything's fine. No, actually not so well. So here is a study. I'm going to share with you a few others here in a moment. This is 10 blinded readers looking at 100 consecutive scans. The agreement for the different scoring systems and by the way, we have three of them, not just one, was not great. So for LIRADS, the agreement was fair. The OPTN agreement which is how we assign liver transplants was moderate. The agreement for LIRADS categories two, three, and four was only slight to fair, not so good. But we agreed in a substantial way about when we diagnose HCC. I would just point out it's not perfect agreement. It's just substantial agreement and huge decisions are made when someone says there's a definite HCC, no biopsy, we pretty much just go straight to therapy. This is a nice table from a published ahead of print article by Katie Fowler summarizing the different articles which have looked at agreement between radiologists interpreting liver observations with LIRADS. There's some important observations from this. First, when we're asked to measure something, we just knock it out of the park. I can measure something. I've been measuring things for many years now and I can assign a measurement and be pretty precise when I make my measurement. But when I'm asked to do something that's qualitative, is it enhancing faster than background liver? Is there a washout feature? Is there a capsule? We just don't do as well. In general, our agreement is moderate. Now most of the studies which have looked at agreement allow people to look at an entire image set and you can scroll through the images. When you look at static images and say just look at this one image or these two images, how well do we do? We actually do a lot better. So if you give me selected images to look at, we can improve our iterator agreement. The other funny thing about this table is we can't even agree on how much we agree and we also can't agree on how to measure our agreement, which is amazing. So we're still not speaking the same language here. This is another quote from Dr. Barron. Our field needs to continue to develop standardized lexicons and accurate reporting systems to provide meaningful and consistent clinical communications for patient care. I totally agree. These data show us our language is unclear to other doctors even though we're trying hard. I think Lyrides is an outstanding example of radiologists recognizing the seriousness of how important it is for us to standardize our language and yet still we're not really getting there yet. We're moving in a good direction though. Now imagine how the patients must feel. If the hepatologist can't figure out what I'm saying or my colleague is saying, what does the patient think? They have no clue. So this is a really interesting study published in AJR 2017, 104 patients reviewed eight radiology reports. So what did they think about those radiology reports? This is how they scored them in terms of things they thought were problems when they tried to consume that information. What are some highlights here? Unclear language. What on earth are you saying? Too long? Okay. But unclear language is pretty much the dominant one here because we're speaking jargon. We're not communicating effectively. Now granted, I would argue that my primary customer from the point of view of the report is not necessarily a direct line to the patient. It's probably I'm trying primarily to communicate to a foreign provider so maybe it's not my goal to communicate to the patient. Here's another one. This is what did they think about the reports? What are the two major things that patients complained about? Well, I would like to get a brief layman's explanation of what you're saying. Just break it down for me. What are you saying here? And then what is the significance of that finding? Can you tell me what that means for me? So you've put a lot of jargon there but just break it down for me. So this is an example of a report that they offered in their paper as maybe a model that could be considered going forward. We have a normal chest X-ray dictation and on the bottom there's what's called a patient summary statement. So the radiology impression is new mild pulmonary edema and bilateral small effusions, stable cardiomegaly. And the patient summary is there's a small amount of fluid in your lungs. Okay, I understand that as a patient even if I have no medical information. What are patients asking us to be from this study? They're asking us to translate medical imaging data into words that they can understand, the same issue that we've been talking about. So fundamentally radiologists are translators. We convert imaging data into actionable word data but the answer is actually not always on the image. Sometimes you have to use additional information like clinical history and integrating information across multiple imaging studies and the image is not what makes us doctors. So how can we enhance the diagnostic process in those two green boxes? All this soft stuff, standardization, trust, partnership, collaborating with people, being comprehensive, being accessible, having empathy for not just your patients but your referring providers and your partners in your practice. So if all we do is shuffle TPS reports around, that's an allusion to office space, from one person to the next person, then the machines I think can replace us actually. Think about it. Won't machines have faster turnaround times, better reliability, willingness to work 24 7, stoic dispositions? They're never going to complain. They're just going to do their job until they break like Epic does. Consistent reports, standardized lexicon, no typos. I mean that's the model radiologist from the point of view of a lot of people. There's never been a more important time for radiologists to create value beyond the pixels on the screen. And you can do this by integrating data across the medical record, solving problems, not just creating reports, forming relationships and being responsive and thinking like your customers. So I'll give you a little anecdote. So someone orders a liver MRI because someone called a liver lesion that needs to be worked up, some incidental finding. You're now looking at the liver MRI. The liver lesion was a hemangioma. No one cares about that. You go back to the original scan and say, why did they get that first scan in the first place that found that silly liver lesion? They say, well, they had recurrent left upper quadrant abdominal pain. And the CT scan that found the liver lesion found no cause of that. But now you know why they had the scan in the first place. Is my job to investigate the left upper quadrant pain on my liver MRI? No. But that's what the patient wants me to do. So I'm going to spend some time thinking about that. Now I have the MRI, and I have the CT, and now I have more a holistic picture of what's going on. I can investigate that, and you can make the diagnosis of the left peritoneal hernia that's been undiagnosed for multiple months or years. So trying to solve the problem, not just create a report. So I do think all these things take time. It's kind of annoying, I agree, sometimes to answer all of those emails and be responsive and paged and constantly be talking to people. But I think of them as a fundamental part of my job, not just something that irritates me. My job is not to make a report. That's what I get paid for. But my job is to advance patient care, and I do that by being an effective communicator. So try and figure out what prompted, solve the problem that prompted the test in the first place. So we have some SAM question type things. So this is not an audience response thing. I think your SAM is done a different way. What animal performs similarly to highly skilled medical pathologists in differating breast cancer cells from benign cells? You have pigeons, mice, toads, or beetles, maybe all of them, I don't know. Which of the following activities have been shown to have a substantial effect on the acute surgical diagnostic impression and plan beyond the written radiology report? Is it ignoring the surgeon when they ask you for clarification? Is it telling them to look at the report because it's already finalized? Is it telling them to use clinical correlation? Is it collaborating face-to-face with a surgeon? Okay. Thank you very much.

diagnostic errors

healthcare improvement

medical errors

communication

test ordering

diagnostic process

patient management

radiologists

standardized communication

teamwork

AI in diagnostics