GR 02 09 15 Feedback in Medical Education Dr Patrick Nguyen



while we go ahead and get started this morning I guess our medical students won't be here because there they are taking a test some bad planning if we're going to give a medical feedback maybe this way we can do it behind their back as usual it's a pleasure to introduce dr. Patrick Wynn who is an assistant professor in the Department of Surgery in the division of general and laparoscopic or mentally invasive surgery dr. Wynn spent the early years of his life after being born in Belgium and then his family moved to Austin he is a graduate undergraduate of UT Austin graduate of this medical school and a graduate of this general surgery residency and recently he became the full-time clerkship director for the junior clerkship please help me welcome dr. winter all right good morning thank you for the introduction dr. Surran a all right so the timing it's been poorly planned for whatever reason but our students are taking their midterm exam but they actually had to suffer through two hours of feedback this year and from talking to the ume office and the students they get pretty beat down when it comes to evaluations in their first and second year so although it's a good time for them to be involved in did they get there they're two hours worth this year so my talk today is on feedback in medical education I have no financial disclosures really important disclosure that I have is I don't feel like I have much business talking about feedback in medical education mainly because it's not an expertise of mine you know being a clerkship director you kind of run the cursory administrative clerkship you get a a good understanding of Education but if you look at meet feedback in education it's a very broad topic that's been researched heavily a lot of the research is done by psychologists they look at feedback in grades K through 2 of college education medical education so I've done my best to over the last month or so review the literature and put together something that our faculty and residents will be able to use as far as giving feedback to medical education and this applies not just to the students but also to to resident education as well so three goals to define feedback and its importance in medical education let me go to to define summative and formative assessments and then to be able to give effective or develop some strategies to give effective or improve on your effective and constructive feedback so definition of feedback if you look at the Webster definition feedback is defined as helpful information or criticism that is given to someone to say what can be done to improve a performance or product and really the the most important thing here is it's basically the goal of feedback is to improve performance you know our students they see the evaluation process as a part of their grade you know the evaluation process for our students is about 26% of their final grade but the ultimate goal of feedback both from the learners perspective and the educators perspectives perspective is to improve their performance if you look at the literature that's out there a ton of definitions but one that I found that that I felt was was a pretty descriptive is specific information about the comparison between the trainees performance and a standard given with the intent to improve the trainees performance okay so again the goal is to improve the goal of the feedback processes to improve to improve the performance of the training so a lot of the broad information that that I used to put this together really was shared to me by two individuals one is dr. Aaron Nelson she's a clerkship director for the OB gen clerkship and then the other one is dr. Christy Costa who is the clerkship director for the for the medicine clerkship and so I got a chapter out of a textbook the chapter is titled feedback for medical education and the very first opening paragraph was pretty powerful the problem is on a scale of 1 to 10 10 being too many words for one slide it's about at 9 and so instead of putting you up on the slide and reading it to y'all I'm just gonna read it to you first just because I found out when you're sitting in the audience and you've got a slide full of words and someone's reading it to you you end up just not getting anything out of it and so I'm gonna read it to you feedback is essential for learning medicine it should happen all the time it seems simple that the more experienced should inform the less experienced about what has been done well or one might but what might be done better the next time it seems simple that the feedback should be frequent objective expected and timely it seems simple that students should get such feedback frequently from teachers and that medical educators should be well practiced at giving it however feedback remains a significant challenge in medical education so these are kind of the highlights of that paragraph and I think everyone realizes that feedback is important you know when you look back on your training as a student as a resident some of the bigger memories are the criticism that was given to you heater in evaluations or in person or if you had to get called at somebody's office to discuss your performance or some issues right so those are the things that stand out and that's feedback okay the goal of feedback is to inform your performance if you look back at your men direct medical education regardless of where you are in your career the folks that gave you good constructive criticism or the folks that you remember because they actually took your performance and altered it in some way and then that last sentence is important to you feedback remains a significant challenge in medical education I think it is a challenge at our institution and if you look at the literature pretty much every single paper that you read about feedback tells you that number one it opens it by saying it's really important and then at some point in the paper we're not very good at it okay there continues to be challenges in medical education as far as feedback goes so why am I talking to you about feedback today right so the way the for the most part the way the Grand Rounds speakers get their presentation on the calendar is you get an email from Silvia dr. Stewart's assistant so I got this email way back before October and she basically gives you a list of dates and asks you if you'd be able to participate in the Grand Rounds and then at some point you get asked what your topic is and you're seeing me questions and so on so I signed up for this date and initially my plan was well I had some interesting cases I'm going to talk about poor venous gas okay and so because no one really says well I'm gonna talk about feedback you know it's not really to me you know it is not exciting it wasn't that exciting putting it together but it's clearly important right so somebody's gotta do it but why me today I'll said very nice so the reason I'm giving this talk is it's really twofold number one they'll see me number two the curriculum committee okay when I think back have been here for nine years as a student excuse me as a faculty and resident I was here for four years as a student I'm pretty sure I don't remember a Grand Rounds topic being on feedback I'm almost certain that we haven't had a Grand Rounds topic on feedback so why me and then why you guys today so they'll see me Elsie me is pretty important you know our school went through its LCME crisis in October of 2011 we were placed on probation it made the newspapers and it was a big deal and everybody in the dean's office freaked out and then the rest of the faculty body had to suffer through a million committee meetings you know there are a ton of committee meetings and the reason it's so important is if you're gonna go to medical school in the US and your school is not accredited you can't sit for step one you can't sit for step two you can't enter a GME program you can't get financial aid so it's important to be LCME accredited and it's also a matter of pride right we don't think of ourselves as a bad medical school that there's a poor job of training students that's why we got put on probation because we did a poor job training students that wasn't a case if you look at at the time there are 126 standards and the data we were given is basically whenever school undergoes LCME accreditation for the most part on average there are going to be about 67 standards that you're deemed not acceptable on and so foreign institution we had 10 and that was enough to get on probation so it's not that we had a horrendous program it's just that we were above average in terms of not meeting acceptable and in our case it was 10 the average is 67 6 to 7 so that's the and then the other thing is if you look at the EDS or educational directors there's about 8 to 10 educational directors that apply to the to the clerkship so to the 3rd year education of our students and 4 of those 8 to 10 basically talk about assessment okay so talk about feedback is a form of assessment feedback is a formative assessment so clearly important in the LCME is and we were not dinged on on assessments this is just to give you an example of again it's critical to to the education of our students and residents the feedback processes and lastly you know when somebody tells you what to do and they're in a position of power and what they're telling you makes sense then you just have to do it so their curriculum committee they meet annually one of our big weak points was that there was no central oversight of the curriculum okay so the the ume office reorganized and they they developed a very robust curriculum committee and that curriculum committee is basically responsible for reviewing every single year of the medical school assessing it assessing that year and making recommendations and so on so this year one of the recommendations for the Department of Surgery in the surgery clerkship was faculty should've should receive development on useful specific feedback driven by learning objectives whether via discussion by the ume office or identification of resources separately so I'm that separate resource for you guys I'll be the separate resource so basically it is being important enough that the curriculum committee said do it so that's why we're doing it today so to understand why feedback is important as I reviewed this basically there are two types of assessments there's a summative assessment formative assessment and it took me forever to truly understand these two even though and maybe it sounds simple to you guys but when I read stuff you know you're used to reading clinical stuff surgical stuff operations procedures surgical diagnosis whatever you know I don't babble in in feedback research until now and so summative assessment just two slides on summative assessment because we're summative assessment is not really feedback feedback is a formative assessment so summative assessment is three different definitions as I as I reviewed the papers on this it is assessment of learning it's used for promotion and certification it's the gathering of information at the conclusion of a course or program to improve learning or to meet accountability demands that's how summative assessment is defined it was so confusing for me so I use Google all right and once I found this website Carnegie Mellon Institute their definition was was way easier for me to understand so basically the goal of a summative assessment is to evaluate student learning at the end of an instructional unit by comparing it against some standard or benchmark these tend to be high stakes which means they have a high point value and then they give some examples so examples on the surgery clerkship this is one of those see me questions for a faculty we've got weekly quizzes the auskey exam the midterm exam the nvme and of course your shelf exam okay just throw your for loop any of these summative assessment tools can actually be used as a formative assessment as well so under LCME forms one we filled them out they asked what kind of formative assessment tools you have what kind of summative assessment tools you have so we actually put down our weekly quizzes as formative assessment tools because the students take these quizzes they get the answers back right away and then based on their performance they use those answers to direct their learning so that the quiz can serve as a formative assessment off-site exam or your mock offsite exams can count as a summative assessment if you're using the mock ones as a learning tool then you're also using in the formative fashion all right so formative assessment assessment for learning it has a priority intention of improving performance by making an assessment of how student performs on a task and then providing feedback to the student about how the performance can be improved and carnegie-mellon description on their website wasn't quite that helpful so I didn't put it up here but really the big picture for a formative assessment tool is to enhance learning or to improve the recipients performance and these are very important to understand that if whenever we complete evaluations whenever we get feedback to a resident a student informally immediately then the goal of that is to as you're doing that it should be to enhance their learning it should be to improve their recipients performance I think I think everyone gets that I mean when we're giving feedback at the bedside or in the clinic or in the o.r you're really doing it for the most part – to improve performance maybe once in a while a very small minority of individuals may have different different goals and and feedback but for the most part it's to educate and enhance learning so on the surgery clerkship we have two formal formative assessment tools one is the mid rotation evaluation really the faculty members will be familiar with that one not the residents because for us it's an LCME mandate that students get clerkship or mid rotation evaluation whether you're a four-week clerkship six week eight weeks it's a mandatory thing that that a formative process is in place really so that a student can self assess and the educators can assess the students and then they have time to improve on that versus they go through eight weeks of surgery nomid rotation evaluation who knows what kind of feedback is given they get their end of clerkship evaluation and then they failed and there I oh I didn't even know I was supposed to do this or that whatever okay so that's that's they CD the oversight says you must do this and so and because they have to have good evidence you say that it works formative formative feedback does work and so that's why we do it because it does help our students and really maybe it doesn't the top students the above-average students maybe doesn't help them because they're going to self assess themselves and do what they need to do to improve but they're struggling students may not have the insight as to what they need to do better and so they could go on struggling through the entire third week third year of medical school become fourth years and then all these issues become evident so that's the mid rotation evaluation they self assess and complete a form in twelve categories self assess their overall performance have a sit down or stand up with a faculty member and then submit that to their clerkship office and then you've got the clerkship clinical performance evaluation at the end of each four-week rotation there's two of them total the faculty the residents and interns may complete a end of clerkship evaluation on the student and if you look at the categories they essentially match up to the ACGME competencies okay about two or three years ago the ume office wanted every single clerkship to develop a unified evaluation tool they wanted this tool to be the same because back then there are eight separate evaluation tools for the eight separate clerkships and so we decided to do it in in the same categories as what the students are going experience in residency so the core competencies and professional and in theirs these are mandatory responses we we want you to include comments on the strength in the areas needing improvement and describing specific behaviors and skills that were observed and including comments on professional behavior if improvement is needed all right so the purpose of feedback so again remember the most important thing really is feedback is to enhance learning we'll set out a few times feedback is to enhance learning the purpose of feedback is to document adequate performance and progression from the big picture standpoint the purpose of feedback is to affirm the achievement of the competencies required for medical school graduation or in the case of residency education for GME program completion and independent practice and in the in the GME setting you've got terminology such as milestones in the undergraduate medical education setting you've got terminology that's fairly new but in trust about professional activities what should a student do by the time they graduate and trust more professional activities and so how do we how do we ensure that our students are achieving these these milestones or these epa's they see the evaluation tools is one important aspect of it although I'm not going to touch on it that much here it's it's important to state that when you're giving good feedback an important part of good giving good feedback is having established goals and objectives and and establishing at the beginning of a rotation what your goals are for the students what they're expected to do so that when you sit down with them and do their their evaluations or when you give them informal feedback there's something that they were supposed to to strive for okay so it's important that when you're completing evaluations you're doing it based on goals and objectives and so our school has a set of goals and objectives and they he's broken down into altruism or professionalism knowledge and skills and if you look at the terminology each category is stated as by the time of graduation a medical student will and with part of the LCME oversight issue our our you and me office now is very big on on being able to map out everything that happens they want to be able to map out everything that happens in the medical school so they want to be able to pull up any of these objectives and competencies and and then identify exactly where and before your curriculum is taught and how do you assess it what assessment tools are being used for every single one of these objectives and so Roz Willis and I are in the process of basically one of the other two do's was please please the redefine every single goal and objective you have on this on the surgery clerkship in in exquisite detail and map it out to these objectives and competencies and map it out to some kind of assessment tool and so we'll be in the process of doing that but essentially we send out their goals and objectives as a mandatory part T to all the faculty residents interns once a year and they're also available on new innovation but I can tell you that very few people look at those you know if raise your hand at for the residents raise your hand if you even know there's goals and objectives for every rotation transplants got them C know about them raise your hand if you've actually read them when you start a rotation so that's no one right it's just a lot of effort no one's read them except for dr. Dan Ross Willis other program directors and then whoever had to write them right but no one seems to read them right and maybe it's because ours for the student standpoint maybe it's because they're not user-friendly as defined by the UM II office so hopefully in the next few months we'll have user-friendly goals and objectives that someone will consider opening in the future all right so the purpose of feedback from the learners perspective it's an important tool to cultivate the learners ability to self evaluate it provides I mean it's basically constant practice right somebody tells them you're not great at doing a history and physical you keep leaving out important aspects of the patient's history for example you didn't ask about the surgical history in a patient that presented with nausea vomiting can't have a bowel movement can't pass wages okay and this students will self assess and I'll say well you're right I'm not very good at that I'm gonna improve on that or you'll be in denial and say I think they're full of it who cares about past surgical history irrelevant we're gonna get a cat scan it's gonna be fine so it's a it's constant practice you know you self evaluate where am i how am i doing am i doing this well am I not doing this port am I not doing this well and then you've got residents and interns and faculty members that are then assessing you and you can compare yourself to a an external Assessor it also helps build confidence over time you know if a student is routinely being evaluated on their history skills physical exam skills ability to develop a diagnostic plan and and identifying a critically ill patient over time it builds the level of confidence that they need to eventually be interns residents and independent practitioners we're not going to talk about this much but you've got formal versus informal feedback formal would be for example the to evaluation tools we talked about and informal is just the day-to-day feedback that we receive when we're working with faculty members or senior residents and so on immediate versus delayed clearly there's a in the literature immediate feedback immediate brief feedback sessions or or have a bigger impact on performance than delayed at the end of my clerkship six weeks later overly too long evaluation tools so so delayed and too much detail is not as effective as frequent small pieces of feedback alright so how about feedback on its impact on performance it's actually hard to to measure the impact of feedback on performance it's easier to measure the satisfaction was the feedback harder to measure the impact on performance so a number of factors go into it the feedback has to be it's gotta be good feedback so he's got to be specific and actionable it can't just be gets to you good resident did well and we get that a lot unfortunately okay but it's got to be specific and actionable otherwise you really don't have anything you can get better on if somebody says you did a good job during your six weeks on transplant that's not quite as helpful as you know you need to understand the hepatic Billy Anatomy better you need to read up on it or you need to understand the steps of a Whipple better so it's gotta be specific is gobby actionable it's important that the learner understands and accepts that feedback okay and so that's why the School of Medicine decided that one of the longitudinal sessions for the students was going to be a two-hour session on feedback how to give it but also how to receive it learners got to be motivated to change and then their learner has to take action so these are factors that the some of the factors can be controlled by the person giving the feedback but others are completely out of out of the educators control and as far as satisfaction with feedback there's an article that looked at the AMC graduation questionnaire and for some of the core clerkships in the 2012 questionnaire as many as a third of responding students did not agreed that faculty provided sufficient feedback on their performance as far as the 2013 ACGME resident survey they found that satisfaction with feedback after assignments had the third lowest rating among the 41 items used to assess GME programs if you look at the san antonio data and i only have the data for the students but it's it's older data for whatever reason in 2014 they didn't ask this question anymore so I only have 2010 data available but in 2010 for the surgery clerkship for the question or statement faculty provided me with sufficient data on my performance if you look at all all respondents all schools in the u.s. about nineteen percent disagreed with that statement at our institution for 2010 it was about thirteen percent so not quite as bad as the up to a third in certain clerkships don't provide sufficient feedback it seems that for the surgical clerkship were actually doing better than the national mean for for those that disagree with with us providing sufficient feedback on performance all right so if you look at the literature there's actually some discrepancy between those giving the feedback and those receiving the feedback so this study was done at McGill University in Canada and their goal was to implement they're gonna implement a workshop on feedback and so before implementing the workshop they wanted to assess you know their program what do our residents think about feedback what do our faculty you think about feedback and it seems that the faculty the residents didn't agree on much when it came to feedback so for the statement feedback was given often or always immediately after the fact or activity 86% of the faculty member felt that they did that whereas only 12% of the residents as far as the feedback that's given to the learner is based only on behavior that reserves directly by the faculty 85% of the fact that he felt that that was the case only 54% of the residents felt that that was the case for other things that didn't agree on okay 69 percent of the faculty felt that the feedback they gave was complete and incorporated all aspects of the performance versus only 16 percent of the residents 96 percent of the faculty felt that they gave concrete suggestions for improvement versus 13 percent of the residents 97 percent of the faculty felt that they allowed residents to to the feedback that was given versus only half of the residents and then 90% of the facts he felt that the feedback they gave was non-judgmental versus 41% of the residents there seems to be a discrepancy between the learner and the feedback giver and perhaps that's what's driving the the dissatisfaction with the feedback all right so things that they did agree on they did agree that follow-up is never or seldom arranged to verify that the feedback was received or implemented all of them the surgeons and the the learners agreed that surgeons are very comfortable giving negative feedback and then residents or never or seldom asked what they wish to receive feedback on so clearly the teachers and learners differ in their perceptions on how feedback is given and received and this is this is kind of the driving point towards having developmental feedback in the development needs to be both ways the learners need to get some faculty development on how to give effective feedback and then the or see the the dowser teachers and then learners need to understand how to receive that feedback and what it's meant for and how to how to incorporate in your learning so some theories as to why there's feedback failure faculty lack the necessary skills there's not much time a lot of for the activity the supervisors based on how the how the learning environment is set up perhaps the supervisors have inadequate contact with the students or inadequate opportunities to directly observe the students and then lastly the learners aren't well primed to receive the feedback or even accept it and some majors deny the feedback okay particularly if the information is critical in a negative sense so that implies that maybe we need a phrase the feedback a little bit differently okay now a couple studies feedback is better than praise this is common sense it's interesting that you can you can do this study and published but basically they they took early years medical students and and were basically thought them two-handed knot tying technique and that's we groups in one group all they did was praise the students okay didn't give any kind of criticism or feedback Holliday was praised as seniors and in the other group that gave constructive feedback on how the student could improve and it's really not surprising that the performance was better in the group that received constructive feedback and also not surprising that those who receive praise were more satisfied with their feedback versus those that receive constructive feedback so that's a common sense one positive framing of criticism results in better performance different study they took first-year medical students learning how to use the Webber Renee tuning fort procedures on standardized patients and one group got standardized to positive framing of negative feedback and then the other group basically received negative framing of the negative feedback okay so positive framing you did this well here are some tips where you can improve negative framing you didn't do this well you should change etc and they found that the subjects in the positive framing group of the negative criticism had were actually half more satisfied with their feedback had higher self-efficacy or self confidence scores and actually their performance was improved compared to the students who received a negatively framed feedback their performance proved both immediately and then when they brought the students back two weeks later so how you frame your feedback whether it's the immediate feedback or the delayed feedback and evaluation process is important so st essentially just feed the sandwich feedback technique and there's other terminology if you go sandwich feedback technique there's another for another s word feedback technique that shows up with with interesting images so this is a cartoon that was pretty funny this was a highlight of my talk so I showed this to my five-year-old and she's starting to learn how to read so she now knows this saying we'll see what happens at school today alright so this means this to the last part of the talk I've got these little feedback cards if y'all can pass them around and ask alia our academic coordinator for the closure it's basically make them pocket-sized alright and these are from Johns Hopkins so I didn't it's well vetted and from the from the stuff that I write about feedback everything that that's said in these cards apply okay so rather than coming up with my own fancy cart I figured I will use the the Air Jordan approach right if you're playing basketball you're wearing your Jordans you can maybe hustle a little more jump a little higher and then play better so now that you've got a Johns Hopkins feedback card that might that might work for us so and essentially I'm just gonna go through the card and then and then we'll finish things up so good narrative comments okay we have all sorts of examples of narrative comments even though it's a mandatory box somehow there are times where there's nothing that shows up in there so somehow y'all are eating hitting the spacebar or something but the mandatory box is not working for some evaluators so good narrative comments are going to be constructive concise specific and actionable okay specific was using in one of the definitions on feedback and again remember as you're filling out these evaluation or as you're giving feedback on a day to day basis it's really the goal of your feedback is just to enhance learning it all it is is to improve performance reviewing the learning objectives and desired competencies for the rotation and you know it is a it is a task to read through all that stuff where can you find it you can find it on new innovations this is new innovations if you have not checked it out but all our faculty members and and residents and interns should have access to new innovation and so both the students goals and objectives and also the residency goals and objectives for all the rotations are going to be on there remember the feedback sandwich okay do you have some what is the student or learner doing well and then what can the student improve on okay positive framing of criticism appears to work better than negative framing of criticism different modifiers that you can use ok consistent reliable making progress or if it's the opposite so-and-so is not consistent not reliable not making progress and then giving examples ok giving examples is important that way the learner can see that it was based on direct observation what the what the negative criticism might be avoiding vague in global statements such as great to work with or a good student will make a great surgeon although it's nice you know you can use that as the praise that alone is actually not going to help someone improve their performance even if they're already outstanding even outstanding individuals can improve so important to not use just global statements as as when giving feedback sit on the back side of it the groundwork when when you're ready to give the feedback important to even set the stage so at the start of the rotation give your expectations goals and objectives what you expect them to do let them know to expect feedback and then when you do give the feedback open it up by saying let me give you some feedback let me tell you what you can do better okay we've got one faculty here that that's extremely good at opening or setting the stage for giving negative feedback okay and so the way it usually works and y'all are gonna be able to guess who it is usually usually the way this individual says let me let me give you some feedback usually it's gonna be I meant what the right y'all know who that is I mean what the f dr. sir any okay huh yes what the fine what the feedback what the feedback and so what the feedback it actually works quite well right you know for a fact when you're about to get what the feedback from dr. serenity it's gonna be it's gonna be criticism on your negative performance and guarantee that you're not gonna do it again guaranteed right and so through the years I think I can't remember if I've been a recipient of what the feedback because I've heard it too many times right so because I can't remember it was it me or was it the the student next to me or was it that when I was a seen was it the chief in the case getting what to feed that but you freaked out you know that's trained you have freaked out a lot of visiting for your students you know when you talk to the Caesar there don't like tell these stores mean I was in the or and dr. serene just went off and you know so the what the feedback actually works quite well because you know exactly what you're gonna get when it's preface what's that weeding out the weak those have those that can't stand there what the feedback will not survive so that's how you prepare for it for your feedback meaning organize your thoughts and observations sometimes it's appropriate to do it in a group setting that way the whole group can learn from how you can present better or how can how you can gather appropriate data for coming out for diagnosis and plan other times it's better to do in a quiet setting one-on-one with the student and then the rest you can read for yourself and remember frequent feedback limit the feedback to no more than four areas and in assessing this just the learners perspective what do you think you did well what do you think you can improve on giving the feedback what do you think you're going to do to now improve on the performance if you could we're supposed to say or very proud of human it is for sure all right thank you all for your time [Applause] while we have a moment before we start mmm everyone has spoken at the during the app site for the election of the new administrative chief residents and the results are in yes so y'all have selected got taller versions of listen myself taya Travis and Ashley are the new administration

Leave a Reply

(*) Required, Your email will not be published