I'll be their first speaker today, and I'm going to be talking to you about innovative education tools, including audience participation and the flipped classroom model. So the objectives of my talk today are to explore some new tools for audience engagement and new education models, and hopefully I will encourage you and empower you to implement some of these in your own practice. So I do have a few questions to get you going and perhaps convince you that these audience engagement tools are interesting, and if you're participating in the virtual audience, you can also participate in these. You don't need any special tools. You just need a phone or a laptop, so if you go to pollev.com forward slash RSNAE352, which is the room we're in, if you forget, you'll be able to answer the questions. So the first one is just to get us started asking, where are you participating from? And you can just click on the map and drop a little pin, and it should work. And it doesn't appear to be working, so I'm not sure why. Usually... Oh, there's... I see a few pins dropping. Okay, awesome. So a couple people dropping pins on your Chicago, I see. Awesome. I see one in Canada. Lots of pins in Chicago. So this is just to show you how you might use Poll Everywhere. We'll talk a little bit more about it in your education rounds. If you can imagine, if you had an image up or a case, you could ask your residents to point to where the abnormality was, and then you could see where people are looking. So awesome. I see a whole bunch of points being dropped across the US, Mexico, Canada, even in Europe. So that's fantastic. It's so exciting to see where people are presenting, where people are learning from. Awesome. So the second question is a multiple choice question. Do you use social media professionally for learning or education? A, yep, mostly Twitter. Yes, mostly other platforms other than Twitter. No, but I'd like to. No, I'm not interested. Or E, I'm not sure. Okay, so we have the results coming live. They're updating these percentages as other people respond. So awesome. So we see lots of answers across the board. A few people not interested. Lots of people already using social media, Twitter, and others, and quite a few people are interested in learning. So great. Hopefully, you will feel that you have learned and empowered to use these after our session today. Third question is, do you use audience engagement tools when delivering radiology education? So similar set of answers. Yes, no, but I'd like to. No, I'm not interested. And D, I'm not sure. Maybe you're not sure what those are, and we'll learn a little bit more about them. As a hint, you are using one right now if you're participating in the poll. Okay, so awesome. So lots of people already using these. That's fantastic. So over half of you are already using some sort of audience engagement tools. You may have been familiar with RSNA Diagnosis Live, Poll Everywhere. I'm sure many of you have participated in these types of questions at RSNA already. So to set the stage, to convince you, if you're not convinced already, we do need innovation in education. So the status quo is probably not going to cut it moving forward in medical education. And there have been a series of articles on this, not in radiology specifically, but across the board. This one was published in JAMA, really making a case for why it's inevitable that we need to reimagine medical imaging. So we know that there has been this incredible move towards asynchronous online learning in medical schools. At the University of Toronto, where my hospital is affiliated with, the last time I gave a lecture to our second year medical students, there was three medical students in the auditorium. It was mildly disconcerting. I was reassured that there were many more, several hundred of the 200 medical students at U of T who were participating online. And so we know that there is this move now that medical students in many centers don't need to attend in person. Some of the didactic lectures, many of them are taking advantage of that and attending the lectures on their own time or in a space that works for them. So we have seen decreased attendance at in-person lectures, and with the proliferation of virtual platforms, which we are all familiar with due to the COVID pandemic, this has certainly become a lot easier. And so we'll talk about some of these tools that we might use to engage our learners in a more innovative and engaging way. Some of the things we'll talk about today are these of social media, audience engagement, and we'll also talk about some alternative classroom models. So the most common social media platform that people use professionally and even for education is, of course, Twitter. But this isn't the only one. There are also educational accounts on Instagram, for example. Of course, we're a very imaging-heavy specialty, and so there are also accounts on there and other ones that you may not even think about as being social media, like LinkedIn. That's actually a way to interact with people and could be considered social media as well. So these are tools or platforms that might allow you to engage with your followers, and those followers could be trainees, or they could be colleagues, or they could be other people that you could be learning from or you could be providing educational content to. They are a potential tool to augment learning. You could share articles. You could share cases, if that's allowed at your institution. And there is a wealth of information on these sites. So one example would be FOMU, and we'll talk a little bit about how to find that, which refers to free open-access medical education. So if you're on Twitter, you could search the hashtag, hashtag FOMRAD, and this will provide you with examples of open-access medical education specific to radiology. You could search using other hashtags if you're a resident, RadRes, or MedTwitter more broadly. And one suggestion, if you're kind of overwhelmed about where to start, you may not have a Twitter account or you may, and you don't know where to find content, start by following accounts of content you trust. So people in your division, colleagues that might be giving talks at RS&A, for example, your department might have a Twitter account. And then other accounts that you may follow, you may already be familiar with, just not aware they're on Twitter. So for example, Radiopedia, of course, most people go to their website, but they also have a very active Twitter account. And you can see they have over 71,000 followers, and they're often tweeting short, interactive educational content that you might be able to scroll through and learn something from. So here's an example, just a screenshot of their recent Twitter account. This is another account that you may find interesting for more kind of general radiology education, Radiology Science. They also have 63,000 followers. FOMRAD is another one. They often retweet educational content. And of course, the content that you're going to be looking at can vary. This is not necessarily peer-reviewed, but start by following content experts that you already know and trust, and then you'll get a sense, and you can always unfollow people. Another good tip is to follow journals that you might normally read. So I like to follow Radiology, all of the subspecialty journals also have their own accounts. And they'll often tweet out when new articles come out, and so there'll be a link to it. You can even click on it and go directly to the article, often with a title or a little short synopsis to see, okay, this is a way for me to see what content is being generated. Another thing you might want to be familiar with in terms of getting content, consuming content yourself, but also consider creating it is tutorials. So these are a series of connected tweets on a particular topic. They might be used to review an article, discuss a case, or even use some kind of interactive tools like polls. So this is an example from a cardiothoracic radiologist that I follow, and she started with a poll asking, have you ever reported the cardiomegaly in a chest radiograph where the cardiothoracic ratio was more than 0.5? And people were able to respond to this poll in real time, and then, of course, the poll was locked, and she discussed the results of this in a series of tweets. Here you can see that they're numbered, and these were linked tweets where she talked about an article she published. So this is an example of a tutorial where you might be able to actually engage with the content and the author in real time. Here's an example from my Twitter. It's my pinned tweet. This is a recent review article I published with one of my fellows. And here I'm just going to show you a screenshot as I scroll through this series of linked tweets. I provide a link to the article at the beginning, discuss some of the background, and again, this is all linked. So if you were following me, you could just scroll through, and this is just an example where often people will include key points, kind of bite-size information that you can digest quickly, some figures from there, and usually ending with kind of a summary statement and often a link to the article. So these are just some examples of what a tutorial might be. The other thing, a concept you might want to be familiar with is a tweet chat. So this is a discussion on a topic that's often facilitated by a moderator or some sort of administrative account, often posing a series of questions, and the audience can engage using predefined hashtags and allowing, again, for interaction in real time. So the ACR likes to do these. This is an example from their resident and fellow section where they posed in advance that there would be a tweet chat at a pre-specified time. They gave the hashtag RadResChat here, and then they posed a series of questions, and people could respond to them right on Twitter by responding to the comment and also including that hashtag so that all you would have to do would be to search or click on that hashtag, and you could follow along in the conversation. So this is a way to engage with leaders in the field. You could also lead one of these. Another example of how potentially to use social media for education. So how to get started? If you already have a Twitter account or other professional accounts like LinkedIn, you're ahead of the curve. If not, sign up. You do it online. It's really easy to do. And just start by following a few accounts, again, people you know and trust or other accounts that you may already look at on their webpages. You could search for some hashtags, so for RSA-related content, hashtag RSA21, and the other hashtags we talked about. So next we're going to discuss some tools for audience engagement. The one I'm going to highlight as an example, but it's certainly not the only one, is Poll Everywhere. And I know many of you are familiar with it. You may have already used it at the beginning of my session and others. This allows you to do lots of different ways to interact with your audience, most commonly multiple choice questions. You could do true and false, word cloud, the pin dropping. I love to do that in my resident rounds. And perhaps as an additional disclosure, I do use Poll Everywhere frequently when I'm giving rounds to our residents and fellows. I find it's a great way to engage with them. If you want to get started, there's an intro account that's actually free. You can only have 25 people interacting, but depending on the number of residents you have, that may actually be enough for you. And often just check your department or your institution may have an account. So I find that these are very helpful, particularly in virtual platforms. So you can use these on Zoom. You just insert the slides for the Poll Everywhere, which are very easy to do, into your rounds. You launch it on however you're presenting, and then everyone can engage with the content, even if they're somewhere else, as we saw at the beginning of this session. I do find it's a really good way to kind of level the playing field and encourage more junior trainees to participate. As I mentioned, I'm a cardiac radiologist, and I find that the junior residents, when I try to give case-based rounds, will often say, like, but Dr. Hanneman, I don't know. I've never done a cardiac imaging rotation. But this will allow them to participate. They can answer the questions. They don't have to give their name. I don't check who answers what, but it allows them to actively engage in the content, even if they're not sure or they're embarrassed. There is a bit of a learning curve to learn these, but I have to say Poll Everywhere is pretty easy. You can sign up, as I said, for a free account and test it out. Finally, the last innovative tool that I want to talk about today is the flipped classroom model. And this is an instructional strategy and a type of blended learning that reverses the traditional learning environment, allowing you to deliver instructional content, often online outside of the classroom, and it moves the activities that have traditionally been considered kind of homework or engagement into the classroom. In this case, trainees or your audience might complete a lecture or some sort of educational activity at home, and then you would engage with the content with them by facilitating it during the time when you would normally give rounds or didactic lecture, for example. So here's an example of kind of putting these tools together in real practice that I've done for our residents. So here's a short YouTube video I created, actually, in 2016. I'm mildly horrified to see how long it's been. I always find our residents are a bit confused about the cardiac imaging accesses if they haven't done cardiac imaging. So here's a short video that I created, and if you watch it on YouTube, I've actually narrated it, talking about the accesses. Just an example, I did it on screen capture on my MacBook, nothing fancy, very not sophisticated. So what I did is I sent this link to my residents before the rounds, asked them to watch it. It was a few minutes long. And then when we came to rounds the next day, after they had presumably watched it, we actually applied this content in real life. So here's an example, a case. I showed them the images. We talked about the case. And then I asked them to use Poll Everywhere to actually engage with the content and ask a series of questions. So here, what's the top diagnosis? And here, this is just a screen capture, so this one you can engage with. I like to lock the results after I have the residents answer, and then show the results, because otherwise they change it to the right answer when they start seeing what the other answers are. Again, and I'm not tracking this, this is just to see. So you can see, I've locked it, and then I show the results, and here, everyone got the right answer. This was cardiac sarcoidosis. So this is an example of how you can use these tools in real life. You might want to, in terms of sharing the kind of more didactic learning, you could share online educational resources that are already available. You don't have to make your own YouTube video. You might share a rate of graphics article, for example, on a topic that you're going to present on or showcases on the next day. So it doesn't have to be something you create, but you could also develop online content, very simple things like video screen captures. We all know how to use Zoom now in terms of recording lectures, so you could just record something very short and upload it for free, for example, to YouTube. There are more sophisticated tools for creating more detailed educational content if you're interested. This can also be used to replace kind of traditional didactic lectures more broadly in the virtual environment. This is an invitation I recently received from the Society of Thoracic Radiology. I was excited about it because I like talking on this topic, but what I was really excited about is that the entire conference has moved to this flipped classroom format. So this is something that we are going to be hearing more about, whereas all of the lectures in the conference are actually going to be delivered in advance to the audience so they can learn and digest and think about it. And then in the time when we would normally be delivering our didactic lectures, the audience will actually engage with the presenters and ask questions and answers. So I think this is a really bold move, and I'm very excited to see where it goes. Okay, so in summary, we talked about some education tools you can hopefully use, even getting started right away, social media, including Twitter, audience engagement tools like Poll Everywhere, and think about some new models of delivering content. So my last question for you is, are you more likely to use one of these education tools after the presentation today? Yes or no? Fantastic. I like this answer. Okay, wonderful. And with that, I'm going to end my session today. We'll have a question and answer at the end of the session, and I'd be happy to take any questions. All right. So the topic that I'm going to hopefully enlighten you on is some tips for doing a better virtual conference, for teaching radiology virtually. So I don't know if you've noticed, but we are currently in a pandemic. And as such, here's a graph of the number of virtual lectures given in radiology over the last year and a half. I'm totally joking. Obviously, this is the number of COVID cases, but actually, probably the numbers of radiology virtual lectures looks even worse than that, right? It looks like this is something that's here to stay, that the conferences that we're giving more regularly are probably going to be virtual or at least a hybrid for a long time to come, if not permanently. And the reason that I say that this is, speak somewhat negatively about this is that if you're a speaker at a virtual conference, my guess is that you've experienced a lot of the challenges that come with speaking virtually. And it doesn't feel quite as good as a speaker. Ironically, I was given the option, obviously, of giving this conference virtually. And I decided to give it in person because I enjoy giving lectures in person, probably like you do. So, let's jump into what are some of the major challenges and then some of the tips I have. I think this is easily the biggest challenge of virtual conferences, engagement, right? You have no idea if your participants are taking a shower or doing any number of other things. Every once in a while, you'll hear the audio of somebody who's doing something that you didn't expect. But they aren't necessarily participating at the level of a learner who's sitting there in the classroom in front of you. If somebody isn't paying attention, you can maybe help them pay attention better by doing something even more engaging, sometimes even calling on them. But that's a little bit more difficult in the virtual setting. So, I've divided these tips up into four major sections as we go through this brief session. Those things, as you think forward to the virtual conference that you have to give, that can be done long beforehand, shortly beforehand, during or even after your conference to make you a better virtual presenter. One of the biggest things and most helpful things that I think you can do is to be as excited about a virtual conference as you are about an in-person conference. That excitement is infectious and you will... Is that an appropriate word to use in this context? Infectious? It's certainly something that will help your learners be more engaged as you're more engaged in your content. All right. The other big tip that I have is those things that make for a good in-person conference, most of them also make for a very, very good virtual conference. So, if you're preparing a very good lecture using all the tips that I had come up with on the back end, that will make for a great virtual conference. But limit the scope is one of the big things and that's what I'm doing by skipping ahead of this part. So, more specifically to giving a virtual lecture, one of the first things you need to do long beforehand is to decide on your virtual platform. So, if you're using Zoom, that's one obviously that is ubiquitous now and probably one of the best choices because unless your institution uses something different, your learners are going to have the most experience with this platform. And due to the probably massive influx of money that Zoom has gotten from its subscribers over the last year and a half or two years, they do have a very good platform that has a lot of resources to use. Microsoft Teams seems to be catching up. A lot of places still use WebEx and then Google Meets. I know my kids have been using this a ton for virtual, for their virtual schooling, but also a very good platform to use. Next, you'll need to decide on virtual interactive tools. And Kate, thanks for kind of sowing the seeds for this, right? Poll Everywhere is a great platform to use. I love it personally, but there are a lot of other ones out there as well that you can certainly use for making sure that your engagement in a virtual conference is very high, right? It's a big challenge and these are some of the tools that you might consider using. Helps your audience to be engaged. All right. So, moving on to those things that you can do shortly before your conference. So, one of the big factors I think that limits the level of engagement that you can have and the polishness of your lecture are technical factors. I think people underestimate how having a decent internet connection actually makes a huge difference. So, I'm the program director at Indiana and I've watched a lot of our faculty's lectures and I've given a lot of lectures myself. The ones that I see having the most problems are those where the internet connection wasn't taken into account beforehand. Rules of thumb, wired connection is usually better than wireless and a work connection is usually better than your home connection mostly because home internet often doesn't have the upload speeds that are needed to actually broadcast in a really effective, smoothly streaming method. So, think about that. Another big technical factor to consider is your microphone. I'm not proposing that you use something like this crazy gadget. But, if you're somehow separating the speaker audio from your microphone audio, you will have a much better, actually your learners will have a much better experience. So, plug in an earbud to listen to the audio while you talk into another microphone. A Dectaphone works okay, higher end microphones are even better, but these are the tools that are usually easily at our disposal. We're in the middle of interview season as well and the interviewees that seem to most effectively have a smooth experience with their audio are those who are using some sort of in-ear or over-ear speaker, audio speaker, and then a separate microphone, whether it's on the earbud cord or somewhere else. Those are usually the best experiences that I have. In Zoom, you can change speaker versus microphone audio by using that button next to the mute button. And on that topic, use the mute button, right? Use the mute button effectively not just for yourself, but if you can, if it's at all possible, be the host of the Zoom session that you're given so that you can mute those people who are taking a shower during your lecture, right? And actually, you can set this up beforehand. You can set it up so that when participants come in, you can mute them upon entry. So, if you didn't know about that feature beforehand, now you do. And that's the case with most of those online platforms. This is what it looks like in Zoom. Make sure that that is the feature that's checked on your meeting beforehand. All right, camera. I think the other thing people underestimate, speakers underestimate when they're giving conferences is how effective showing your face and having your camera turned on during the session is for increasing engagement during that session. Now, there are things that you can do to make the experience even better, right? You can, instead of using your old laptop, use either like a newer webcam or a newer laptop, and that'll increase the image quality. I don't think this is tremendously important, but having the camera turned on, I think, is. It lets your participants see that you're there. Actually, Microsoft Teams just recently implemented a feature that allows you to kind of hover in front of your presentation. It cuts out the background and lets you just, your person be in front of the presentation, which is kind of a cool little new feature that I like with that platform. All right. So, another thing that you can do shortly beforehand is practice. Whether it's with friends, whether it's with family members, whether it's with your pets, although they don't give quite as good a feedback, or with the IT person at your institution, if that's an opportunity you have available, this allows you to iron out some of the wrinkles when it comes to technical features that you may not be as familiar with. You're going to run into those bugs when you practice, and then hopefully get them sorted out beforehand. So, if you're not as familiar with the technology, practice with somebody that can help you with some of these bugs. The other thing that practice does is helps you see how you appear to your audience. So, you can do this, obviously, in real time. Watch yourself in the camera as you're giving the conference. And then if you record it, you can have an even better experience looking back, maybe at even just a short segment of your conference to see how you look when you're giving it. The final thing that practice helps with a ton is pacing. Knowing whether this 10-minute conference is going, stretching out to 15 minutes or 20 minutes, giving that conference over and over is very helpful. Sometimes if it's interactive, it's harder to gauge how fast it's going to go, though, obviously. All right. So, what should you be doing during your conference? I think as you get started, set your expectations. Giving a virtual-only conference feels a lot like giving a lecture to an empty room. Actually, Jesse, who's going to be talking in just a little bit, and I were talking about this before this conference, that, holy cow, it's a little disconcerting. Sometimes you may need to insert a laugh track, right, Jesse? I think that's a good option, to make sure that your corny jokes get at least a little bit of laughter. And, anyway, just set your expectations. Honestly, if you tell yourself this is how it's going to be, sometimes it's a little easier not only because it's less disconcerting, but also it's very similar to your practice experience, right, which was to an empty room. It feels very similar. You should be thinking about security concerns. Make sure that you have your notifications turned off whenever you're giving a conference. You definitely don't want something embarrassing to pop up on the screen. I actually had one of my residents have this happen while they were presenting from an iPad to, like, two or three hundred medical students, and that caused a little bit of a kerfuffle. So make sure you're turning those off. Make sure that if you're showing patient information, especially if you're presenting from PACS, that you are very certain that you're not showing protected patient information unless it's a multidisciplinary conference and there is really tight control over who's participating, right? So I would say unless, again, you're feeling very confident about your ability to exclude patient information, don't show this to a room, a virtual room, that you don't have complete control over the participants. All right. So one feature that I actually like that kind of mitigates this is when you're looking at your tools in Zoom and you're starting to share screen. Most people are sharing using Zoom, showing either their desktop or a single monitor, or they're showing the software presentation. You can click on either one of those and it works great. One of my favorite things to do with Zoom is instead of under the basic tab, you can go under the advanced tab and actually just show a portion of your screen. So what that will do is allow you to have a green box that you can move around and resize, have it set up to show only what you wanna show, and then your learners have a much cleaner experience when they're looking at their images on their screen, and it kinda limits the bandwidth that you're actually broadcasting and makes for a smoother broadcast with less delay in the Zoom. All right, so when it comes to small tips for engagement, if you have a camera that's pointed off in some wonky direction, it's gonna be weird. It's not gonna feel right for your participants. I recommend looking straight into the camera, or at least be aware of where your camera is placed so you can talk to your participants. We're not reading mammograms. We don't need the lights down while we're actually doing the conference. I recommend turning them up. Look how much better that is, right? Seems so much more engaging. When you're asking your participants to do things, I recommend giving them the heads up. So at the beginning of the conference, say, all right, I'm gonna call on people. I want some answers. Please be ready to turn your microphones on when I ask you to, or tell them be ready to chat, be ready to put something in the chat screen, right? This is a great and very simple way to engage your learners, have them participate in a way that doesn't let them just check out, okay? So when you're doing that, you might show a case, and then look, they can submit answers in the chat. This is very, very simple to do, very low-hanging fruit if you're a little intimidated by some of the more complex audience response tools. All right, and then tips for after your conference is finished, you should absolutely be soliciting feedback. If it's not built into the training program in which you're talking, you know, either through Med Hub or some other tool where you do get feedback from the learners, make sure you're asking your learners for directed feedback on how you can make a better virtual conference in the future. And then the final thing that I'll say as our tips is that if your lecture is recorded afterward, go back and view it, or again, view at least a small portion of it so that you can get a better sense of how you appear when you're giving a conference, and can make this a much better experience for your learners. All right, so there are the tips. Thanks so much, everybody. Again, I just encourage you to think about these things, and especially take some of the low-hanging fruit maybe that you haven't tried before, and make this a better experience for your virtual learners. Thank you. All right, thanks so much, Dr. Calmer, for that excellent talk. So a lot of really great tips there. So good morning, everyone. I'm Dr. Jesse Cordier. I'm a pediatric radiologist at the UCSF Benioff Children's Hospital. I'd like to start off by thanking RSNA for the opportunity to chat with you this morning. And I'm really excited to share with you some tips on virtual and augmented reality tools for radiology education and training. It's an area I'm very passionate about, so again, I'm excited to share this with you. And the overall idea that I would like you to come away with is that the incorporation of AR and VR technologies can provide an effective complement to existing methods in radiology education. I always like to start off with a quote to that end. Really, and this is by Dr. Richard Feynman, that the ultimate test of your knowledge is the capacity to convey it to another. And I think, you know, really, these types of tools that we'll talk about do provide us with some additional methods for conveying what's in our mind as educators into our trainees. This is just a little doodle that I often make for our radiology residents here, talking about transitional fractures of the ankle and about how we can get these unique fracture patterns with their pediatric patients due to their age and the incomplete fusion of the physis. And, you know, I often think this is helpful, but I'm not sure unless I see it missed in a QA or I ask them again later and they don't remember. But what if we had an additional method, a three-dimensional model that we could have at our fingertips that provides a really interactive way for the trainee to be able to take this fracture, move it around three-dimensionally to understand it? Would they have a better understanding of it? Would they retain the information better? And it turns out, yes, they actually would. This is just some information that some of our colleagues and I have put together and collaborated with on this work throughout the country, other institutions that are using these technologies. And again, I'd like to share with you what we have found. So we'll talk a little bit about the definition of the terms augmented, virtual, mixed reality, where do these fit, and what does the future hold for the technology? So let's talk a little bit about definitions and terms. And really this comes from the idea of the virtuality continuum. And that is on one end of the spectrum, the real environment, you and I here in the room talking, normal experience. On the other end of that spectrum, essentially the matrix where you're completely encased in virtual experience, a digital experience. And mixed reality is everything in between that, with augmented reality being a little close to the real world and virtual a little bit closer to the virtual world. There are a few different platforms that we can use to convey this. We have head-mounted devices like the HoloLens or Magic Leap augmented reality devices or virtual reality devices like the Oculus. And other methods of delivery include things like tablets or phones where you can use your phone as a picture window. Again, something a little bit more familiar or inserting your phone into a device using this with some special lenses to give you a virtual experience. And there are actually a number of benefits to this, the ability to interface in a three-dimensional environment. And we found that there are improvements in learner engagement, again, I think touching on some of the earlier talks as well, but increased motivation for learners, greater interactivity, and improved memory retention. Further, with virtual reality, it also enables you to train in environments that have limited access. So you can't always bring a group of radiology residents or medical students into a clinically busy environment. What this allows you to do is to have them as a virtual fly on the wall, as a virtual participate in here in these areas. So this is my collaborator, Dr. Upad here at MGH where he has this special 360 camera where during this contrast reaction simulation, you're able to be a virtual participant in this so that you can essentially be there and go through this without having to actually physically be present. All right, so where can we fit some of these technologies in? So here I'm just lecturing to our radiology residents at UCSF and I'm actually screen casting what I'm seeing on my iPad here and holding this merge cube. And then I can actually screen cast that to the laptop and then virtually throughout the other sites that we're discussing here. And so what this allows us to do is use tools like this to use this for improved anatomy education. So I think this is really where this technology shines is the ability to, on their own or at a conference, walk through and virtually manipulate these things, these labeled information to be able to better understand things. And we've actually shown this with a group of UCSF medical students where we did 27 students in the peer-to-peer learning session where they have a cadaveric session going over prosecuted specimens. And we compared that to our modules using augmented reality, these simple mobile devices, and found that there was no significant difference in their post-test scores between the group either using AR or the ones at a cadaver using this peer-to-peer learning method. And I think what this means is that they can study on their own at home and get just as an effective amount of information using this. And really with other trainees, we found that the mobile-based applications are helpful because there's a very quick learning curve readily available it's easy for them to figure out something that's familiar using a phone or iPad. And this novel experience, I think, is also helping to imprint the information. We've also used these things for our technologists. So the ability to look through a model here for rad text to like, how do you get a transcapular wide view and combining both the 2D and the 3D to better imprint this type of information for them. Similarly with our PICU residents, so our PEDS residents who are on a cardiac rotation here, we're able to show them very common congenital malformations and again, helped them to better understand being able to manipulate those on their own. There we go. And then further for procedural training, so things like virtual reality, where you can either use a stationary method or a GoPro camera, where you can actually be the participant and walk through this. And again, this is another thing where you can even incorporate things like haptics, where you can feel the different motions of using the different equipment. So understanding what tools are necessary and to have a virtual orientation into these various spaces here. And even you can use a 3D scanner to scan in the actual equipment and then have a virtual use of this. So this again is courtesy of Dr. Upad here, who is using this virtual 3D camera to give an orientation of this interventional suite here. So this is part of his orientation for his IR residents as they first come in, just as one piece of the component where you can walk through. And it's found that actually these types of simulations are effective. So neurosurgery has shown placement of ventriculostomy using a VR method, really shows improvement in being able to assess the learner being able to, for the learner to be more confident when they actually go to do the actual procedure. And particularly procedures that are sometimes difficult or stressful, this takes that stress away and allows them to practice in a more controlled environment. You may think, well, Jesse, this is very cool and all, but there are some challenges. What about the equipment cost? What about the software learning curve? And where can I find source material for this? And how did I get started? Actually learned everything that I did have done with this, developing software, all of this through YouTube. So there's a ton of information, great tutorials on there through YouTube that are free. Also great source material through things like the NIH 3D Print Exchange, where you can easily convert these models into an augmented reality format, as well as things like TurboSquid, where it is a pay model, but you do have a lot of pre-populated source material. And for funding, so I was fortunate to get some grants through UCSF to fund the initial purchase of a lot of this equipment, but I think because this is a novel area and there are a lot of areas where we can apply this, I think there's definitely a lot of opportunity for future funding. And then just to touch on things, you can screen cast these, so using things like Google Meet or other things, so I can cast my, as I was showing, my phone or my other equipment, what I'm seeing three-dimensionally as I walk through things, as I narrate with them. And you can also, if you're more, for me, I like to have the hardwire connection, so if you have a Mac, you can connect with a cable and then use QuickTime to go new movie recording and then select your phone, and then that way you're hardwired and you're not worrying about Bluetooth or anything like that, so that you can screen cast what you're showing here. So that is a possibility. Just to talk a little bit about what does the future hold for this, I think as the technology continues to develop, we'll continue to see more opportunities for greater interaction with other devices, so HoloLens, iPhone, all viewing and sharing the same three-dimensional models together. Things like no need for hand controllers, so ability to use hand interaction here, this is using the Microsoft HoloLens 2 here, we're at the Microsoft Reactor to be able to test out the HoloLens 2. When it was first put out, and so those abilities for you to recognize your hand so that you don't have to have any special controls I think is very exciting. And then HoloPresence finally, which is the ability to manipulate models in two different places and interact with three-dimensional information whether you be in one place or another, I think is another exciting growth opportunity. And so in summary, augmented and virtual reality technology, it's emerging technology, I think there's a lot of room for research and further innovation, and hopefully it sparked a few ideas about how you can incorporate that. For VR, really there's I think great applications for training, orientation, and simulation that can be used, and particularly I think that's really where it shines. And augmented reality, again, education, the ability to study on your own, to be able to learn and piece together, you get a lot of great information for training, education, and even more. But I think more importantly, choose what technology resonates with you, there's really no right answer, I think whatever area that you find is most effective is the one that is the best to use. So overall, hopefully you've come away with the idea that incorporation of AR and VR technologies can provide an effective complement to existing educational methods in radiology education. Thank you very much. Okay, so my name is David Sarkany, I'm the PD of the Radiology Residency at Staten Island University Hospital, and I want to thank Arsenae for inviting me to talk about feedback. I think it's a very nice ending to this group because we're talking a lot about education, and ultimately, when we give education, we want to give feedback to our trainees. So before we get started, we have to remember what's the purpose of feedback? It's essentially twofold, either we want to change some kind of behavior or we want to reinforce some kind of behavior. And this is really the cornerstone of education because if we teach and we don't say anything based on how those trainees are doing, they're not going to learn anything. So I actually have four objectives that I want to try to touch on. Let's talk about why ineffective feedback exists, what's the difference between constructive and destructive feedback, what's a good plan for success, and a little bit about the virtual setting. So why is it that feedback can be ineffective? And I found two major reasons. One is a perception versus reality situation. So there's an article written in 2005 where they did a survey of surgical attendings in residents, and approximately 90% of the surgical attendings felt that they gave good, satisfactory feedback. Those same residents, 17% felt like they received good, satisfactory feedback. So there's a disconnect between what we're saying and what they're hearing. So that's number one. The other thing is there is a fear from faculty to give feedback. It could be simply it's an awkward conversation, I don't want to ruin the relationship that I've developed with that student, I don't want to hurt their self-esteem. And one that I'm hearing more and more often is reprisal. In other words, you're going to give some kind of constructive feedback to a trainee, the trainee's going to go to your superior and complain about you, and then you may have to deal with the consequences. And you might say, what's the point of giving feedback then? But one thing that's really important to keep in mind is, just like you might have one image, you can see two different pictures in that one image. Many times when you're in that feedback situation in that scenario, there's more than one way to look at it, and more than one way to approach that feedback situation. So now we'll talk a little bit about constructive versus destructive feedback. So four important things with constructive feedback, you want it to be specific, in a safe environment, you want it to be good judgment, and conversational. So what do I mean by specific? So many times you want to say a nice word to your resident at the end of the day, and you say, good job. That's a meaningless statement, because what did I do good? Were my reports good? Did I make good findings? Do I have good differential diagnosis? I don't know what's good, so that I know what to reinforce to keep on doing. On the other hand, you might say, you know what, your reports, the histories are really sharp, and really are very helpful. Please keep on doing something like that. So at least then, that first year resident knows, okay, I'm doing this well with my reports, I will move forward with trying to do that. Clearly a safe environment, that makes sense. Good judgment, we'll talk about in one second, and conversational. So one thing I would like everyone to take home from this talk is, every time you give feedback, it has to be a conversation. It really shouldn't be somebody on high talking to somebody on low. It should be leveled playing field, and a two-way street. And when you do that, it'll be more effective. Because remember, when we're giving feedback, the reason why we're giving feedback is we wanna either affect change, or reinforce what's going on. So if we're not gonna do that, there's no reason to give it. So I'd like to give just an example. I think examples are very helpful. So let's say we're in a simulation for a contrast reaction. Kurt, I was so impressed the way you handled that contrast reaction. You were poised, calm, and professional. You knew your ABCs. Immediately established yourself as a leader. I noticed that you looked at the crash car for a few seconds before you acted. What were you thinking at that point? So that's good judgment, safe feedback. There's another concept, which is also good feedback, but not as good, which most of us tend to do, which is the feedback sandwich, which we like to say something positive, then put in the middle of the sandwich what we really wanna talk about. And at the end, follow up with, you're doing a good job. Something positive. And typically what's in the middle, or the meat of the sandwich, is a however or a but. And the problem with that, although it's not bad feedback, is that students, trainees, residents, employees will hone in on the however and but when they learn what kind of feedback you give, and the extraneous stuff is gonna be just extraneous, even if you don't mean it to be extraneous. Something like this, you're just stating the facts. And if you state the facts, sometimes as uncomfortable as it might be, you have a better chance of getting through to your trainee. So destructive feedback. So just a question out there, if people can raise hands. Anybody ever feel like, in med school, residency, fellowship, that you had a destructive feedback situation? Raise your hands. Right, so I think most of us, at one point in our life, have come across this, and it really doesn't feel good. And it doesn't make me feel like I wanna learn that thing more. It makes me angry, it makes me feel bad about myself, and a whole slew of other things. So what is destructive feedback? On the simplest term, it's general. So again, if I say you did a good job, that's not good feedback. But the primary focus of destructive feedback is really negative feedback. It's in a conflicted environment. Sometimes there's sarcastic tones, people are mean, harsh. It sometimes gets to a level where it's a personal level, like, I can't believe you became a radiologist, you don't have a good eye for radiology. And these things do happen. Now, I've noticed a little bit in the literature, just also in my observations, this tends to occur in two scenarios. Usually when something bad happens, that faculty member is stressed beyond belief, they have all this pressure, all these bad things are happening, and they kind of lash out at the trainee, which is not acceptable, but at least it's understandable, it's human nature. So that's something that we have to work on on a personal level. But there's also some teachers and mentors out there that feel that the destructive feedback paradigm is a good way of learning, and that to be essentially mean to your students and trainees will teach them to learn things the right way. And I think most of us who raised our hands would probably disagree with that statement. So just an example, this would be judgmental or destructive feedback. Kurt, what were you doing fumbling around that crash cart? Time is of the essence, the patient could have died, or couldn't anyone tell me Kurt's critical mistake? So those make you feel this small, and you're not gonna listen to any of the feedback. So let's talk a little bit about a few tips for a successful feedback session. I'm not gonna go through everything, some of the things we've discussed, but a few important things. It should be given by the individual involved. Many times as program director, I'll have the MSK section chief come to me and say, so-and-so's having trouble with A, B, and C, and can you please talk to him or her? So I'll go to that resident, and the resident says to me, oh, but they said I did a very good job during that rotation, that I understood my anatomy and everything else, and there's a disconnect, and the problem is I'm the middleman, so I don't know where the disconnect is occurring. So really, the person who's involved in that feedback situation should be the one giving it. Another thing is we talked about that disconnect between perception versus reality. So one thing that I think is a little helpful is before you actually give feedback to tell your trainee, now I am giving you feedback. There's no way for them to feel like it's just a conversation between two friends or over coffee, that where you're actually trying to affect change. The other thing that you want to look at is you would like it to be immediate. So another thing that I get as program director is two or three months later, the section chief or something comes to me and says, oh, so-and-so really doesn't know their anatomy of the skeleton, and the rotation was two or three months ago. What use is it to go now for me to go, or for them to actually go and talk? Because you have nothing, remember, we want to talk about specific. What are you going to tell them specific? You just didn't do a good job? That's a meaningless statement. Again, we're going to talk about conversational. So we're going to talk about that next. So again, conversational is really, really important. And I think this requires practice, and depends on your own style of how you want to do things. The important thing is that you want it to be a two-way street. You want to invite feedback. You want to level the playing field, because it's going to make your trainee more comfortable with the situation, therefore more willing to accept what you have to say. You would like the students to reflect on the situation, because many times they can become their own critic of what happened, and you don't even have to talk about it. Now, clearly we've all been in scenarios where we have people who don't listen to feedback well. Those are a little bit more unique experiences, but if you follow these basic ideas, you might lead them down the right way. And sometimes it might require you to show that you're a human being. Maybe talk about something that is vulnerable to you, with the residency or clinical. Obviously, you have to be smart, because you are still the teacher, the mentor. But when you do that, then the trainees tend to be more receptive to what you have to say. So this, I think, is really important, establishing a feedback culture. So this you want, ideally, in your residency, and you want, ideally, in your department, your business, whatever it is. So what I mean by this is, want it to be a safe and trusting environment, meaning if I get feedback from my chairperson that I know he or she has my best interest, and it's not a matter of attacking me or anything like that. The other thing is, feedback should be normal. It should be every day. It shouldn't be just when something is bad. It needs to be balanced. So again, don't only talk to people when there's something wrong, which is what we tend to do, but give positive reinforcement. Tell them, oh, you know, you did a really great job on A, B, and C because A, B, and C. If you make it normal and every day, and you talk about positive and negative, no one is gonna fear going into your office because you're gonna have that conversation. And finally, accountability. So something I referred to a little bit earlier, we're humans, we're giving feedback, but we should receive feedback as well. We're not perfect in everything that we do. If we approach these kind of scenarios in that way, then we'll be more successful. And finally, virtual. So most of the stuff I spoke about applies to virtual, but the problem is, how did most of us feel during virtual readouts or virtual graduation or tumor board or department meetings? We definitely feel like we missed something. Okay, we were forced into the situation, but we lost nonverbal and social cues, hard to react to certain people's reactions. So my feeling for the virtual is, we have to be more on top of the communication, make it much more constant, talk about positive and negatives all the time, level the playing field, have a cup of coffee in your Zoom meeting just so things are relaxed. And if you tend to do that, I think you might be successful. And thank you.

innovative tools

education enhancement

audience participation

flipped classroom

virtual reality

audience engagement

online learning

augmented reality

feedback culture