5 Keys to Successful CME Outcomes Measurement | Continuing Medical Education



good afternoon and welcome to today's discussion entitled five keys to successful cme outcomes measurement I am your host jeremy Lumbergh CEO of ethos e-learning management system and dlc solutions just a little bit about ethos ee ethos ee is an enterprise level learning management system that has been specifically designed and developed to streamline and automate the delivery of cma by medical associations health systems and academic centers ethos ee is ACCME pars compliant and integrates assessments evaluations and transcripts it supports the delivery of live enduring and RSS of courses and has a built-in reporting engine designed to facilitate outcomes measurement it despite the availability of platforms such as ethos EE and advances in cm II technology we continue to hear from our cm II colleagues that they feel challenged about how to advance and implement formal outcomes assessment methodologies within their organizations so today I've invited our friends and panelists Derek Dietz and dr. Chris Christopher Urich from improved cm EECOM to discuss their 5 keys to successful cm e outcomes measurement so by way of introduction Derek is the president of improved CM e-comm which partners with leading healthcare organizations to help them integrate analyze and report on practical outcomes measurement within their CME programs he is a 20-plus year CM e professional a frequent speaker and accomplished author a fellow of the Alliance for Cee and the health professions and past the surveyor of ACC ma Christopher is the chief scientific officer for improved cm EECOM and has over 10 years of experience including his role as the vice president of medical education at sanofi-aventis he has also served as a clinical research associate clinical trial manager and global head of medical writing at Merck sanofi-aventis and burying cheering Christopher holds a PhD in developmental neurobiology has published several research articles and as a regular reviewer for contemporary clinical trials thank you both for joining us today we greatly appreciate everyone's time here and having said that Derek I will turn the conversation over to you thanks so much I can't tell you how many times as a kid my mother told me about this with respect to my homework when it was the eleventh hour and the night before it was due so I I've tried to learn this but it's still a challenge for all of us but planning ahead way ahead is really important with respect to CEC Emmy outcomes measurement really having some sort of plan described articulated in the document at the earliest stages of activity development whether you're developing a grant request to submit or you're just in developing an activity internally is to have that plan in place and the the best approach I think is to think you know plan with the end in mind so it's it's more about asking questions about the outcomes that you're planning to to design and planning backwards so that you have a way to get where you need to go and oftentimes the practical aspects of that mean developing your outcomes assessment questions in concert with content parallel with your content not as an afterthought thinking of those oh my gosh we finished the activity now I've got to come up with the outcomes questions you have a much better chance of success if you if you do that in concert with content development also planning with the end in mind helps you establish whether you have the resources you need to really perform the outcomes as you've planned them and you know we'll look at you know we've sometimes had folks who will say that they want to look at performance and patient outcomes in early documents but really have no specific plan for that they haven't asked who is going to help us with this when is it going to be done how is it going to be done when you ask those questions you really get a reality check on what what's feasible and what's not so second key is measuring to your educational design and here's a couple other questions that really help with outcomes planning what do you expect to change in your learner's it's surprising how often when we ask this question of our clients how this question and the next question can actually change and have an impact on the nature of the content and it's surprising how few people ask these questions so when you ask what do you expect to change in your learner's it's really are you focused on knowledge or filling knowledge gaps that you've identified or confidence or competence gaps or have you seen performance issues that you're trying to address have you found that patient outcomes are what they need to be and you're working to address those so what do you expect to change and it's interesting how that can change the content an example of this had somebody who's in early planning stages for an activity on depression and one of the things that they found was that in their needs assessment gap analysis was that clinicians weren't using depression screening tools the way they should or as often as they should and so when they identified that for us we said well what do you want to change in your one of the answers was we want them to use this depression screening tool more often and we said well how does your content address that and the answer was well you know we're gonna we'll tell them about the tool I said how about giving them the tool perhaps another idea is to show them how it's used have a practice using it so you'll find that as you ask these questions what do you expect to change and what domains do they fall that will help you make sure that your content is on track another thing is for higher outcomes levels to distinguish between subjective and objective assessment in your plan there's a whole lot more resources that typically have to be used for objective outcomes assessment meaning things like chart polls other forms of measurement perhaps Quality Assurance quality improvement data patient safety data those kinds of hard-hitting factual actual kinds of data versus objective self-reported kinds of levels and it's important to define those because they have a lot to do with the resources that you'll use Christopher any any comments on on that yeah I think yeah Derek I think the other the the point regarding accurate accurately representing the more levels I don't know we were you going to get to that no go ahead go ahead what I was I think I think what we we had discussed I think for the audience to know is that the more levels are not always entirely clear everyone's mind in other words there there may have been what we use these we use the more levels 2 3 4 5 6 a lot but we're not always clear on what exactly we are referring to so it's it is important to be very specific about this and when you're submitting especially if you're submitting to a commercial supporter for ground support it is typically required that you do indicate which more level you are going to be measuring so that's just something else to bear in mind yeah and if you're if you're not using a more level if you've got some other kind of model that you're working with just be specific so the the people can anticipate where you're headed with things so again continuing on with measure to your design here's a question to ask and it's a it's a common thing that people will just assume that yeah we got to measure performance change and we'll do it self-report it so it'll be subjective in nature and so we got to have this follow-up survey now it's really important to make that decision understanding what your educational design was so if you're the gaps you found we're in the knowledge domain and in the competence domain or performance domain where do your objectives land and how far you're going to go with your measurement if you're only focused on changing knowledge and competence people's intention to do something good more often then why would you have a follow-up survey to ask about performance and patient outcomes now you can always measure beyond your design but if your content is designed for knowledge and intention to change you don't necessarily need to go further in performance and patient outcomes and you may have success with that but again it's a question of resources and timing and you you if you're if your design is focused on changing knowledge then perhaps measuring beyond that doesn't do you too many favors as may have less of a chance of optimal result the results so here's an example of this of a mismatch so a group that was having Grand Rounds in a variety of hospitals and it was a one-hour Grand Rounds and they promised performance and patient outcomes results from chart pulls for one hour Grand Rounds we were so excited at first when we heard that you know there were we might be able to help them with some Foreman's change and so on but then we start asking questions about what's the nature of the content and it was when our Grand Rounds focused on knowledge and we said did you have any did you talk about performance change did you talk about patient care strategies and reinforce the need for those and so what we discovered was there was a misalignment so their measurement was not aligned with their design and had less then positive results so when you do measure to your design you establish enhance credibility and a more effective use of resources you're not going to waste money on unnecessary measurement and some of you who are submitting grants for commercial support Christopher how does how does measuring to your design enhance your credibility yeah I think and I think with this Derrick I think what this what this comes down to is you know the credibility is very important it is it is in the old adage it is in fact better to under promise and then over deliver than it is to over promise and under deliver so it may look great that you're requesting a grant and it's a relatively modest amount of money and for this modest amount of money you're claiming that you're going to be able to measure a performance change but in reality that may be very very difficult to do if you really think everything through and the next time you submit a grant the commercial supporter may say well let's let's go back and have a look at what this provider has done in the past and that may come back come back to haunt you so it certainly is it is really important that you that you do deliver to your to your design and that that will help you in the long run and and we realize of course that grants that are small amounts of money twenty five hundred five thousand dollars have different expectations from a commercial supporter in terms of outcomes assessment generally and would grants of much more tens of thousands or hundreds of thousands of dollars and so but being realistic here and and you know measuring to design we believe gets you the credibility so I guess we move on to another polling question so what we'd like you to answer here is we like it a rate self self assessment here rate rate your confidence what is your confidence in aligning your outcomes questions to your learning objectives and the learning gaps in in the program so please select one of these one being not confident at all I'm not constant at all in aligning outcomes to number five the other end which is I am extremely confident in aligning outcomes so off you go eagerly awaiting the results few more seconds you'd give you a couple more seconds yeah okay so it looks like we are somewhere kind of right in the middle I guess it looks like sixty about two-thirds are somewhat confident with a pretty broad broad tail going off one in not very confident others in very confident so very few at the two extreme so so direct that's I think that's a pretty good good good place to continue on with great yeah really if there is one key out of the five that we're talking about today this perhaps maybe the most important for outcomes success and it's really about alignment align align align so recently I was driving down the street in my car and I have a van and what I noticed was I was going straight but as I was holding the steering wheel the steering wheel was off to the left even though I was going straight so obviously obviously something was wrong with my steering I was worried about my alignment I took it into the shop and they checked it out now I've had misalignment problems before I've been on the highway misaligned in my car and it things started vibrating it's starting to pull me off to one side of the other pulling me off course it wears my tires unevenly and their safety implications of course when you're misaligned it's kind of similar with CM ECE and these items that we've listed here with alignment when these things are are not aligned you can go off course to varying extents to the extent that you're misaligned so we are really suggesting one of the the major keys to success in outcomes measurement is this alignment that the gaps you've identified and we we all work on gaps and identifying gaps in knowledge and performance and patient outcomes identifying those in a variety of ways but when you nail the gaps with with real tangible information that goes beyond people stating an interest in a topic when you get down to verifying that gaps really do exist that's the firm foundation that you have for your outcomes assessment and if your learning objectives a realistic set of learning objectives are meant to address the gaps and we realize that you're not you can't necessarily address all of the gaps you've found you're going to pick the ones that are realistic for you in your setting but when those are aligned that's adds to a firm foundation identifying desired results what do we want to change in our learners aligning that also with instructional design what kind of design in live or web-based or however we're going to do this is going to deliver us or help us to fulfill those objectives fill those gaps that you've actually got content that addresses the learning objectives that focuses on the gaps that you've got a format that facilitates all of this and that your outcomes questions are directly linked back if you're asking outcomes questions about learning about things other than are focused on in your learning objectives and about things other than where you're trying to fill the gaps you have less much less of a chance of success and so how can you plan this alignment so it's helpful in planning documents to demonstrate that alignment and I know that we've seen folks very effectively use a chart right Christopher right yeah and direct what what where we what we like to suggest is is it is a chart that has for a given gap a given learning gap some very specific learning objectives that are directly tied to the gap and along the same row looking at what it is you're trying to change for that gap in that learning objective whether it's a knowledge competence whether it's performance and then you continue on with the other components instructional design making sure that you have content as well and making sure that the format is amenable and that you're assessing this gap with the right types of outcomes questions that line up with all of the other components yeah and just a generic example a misalignment example is that just somebody who writes outcomes questions that aren't really tied to the content they're they're done with no context written with no context of the content the learning objectives in the gap and they may be cool interesting questions but there's they're not aligned there's no content that really specifically hits on those questions and they're not linked to objectives and gaps and what happens is if you're using for instance pre post questions that aren't aligned you're not going to see any change because the mists the alignment was way off and there's no way to tell if you the degree to which you met your learning objectives because the questions you asked aren't linked to them so a practical kind of approach to this under examples on this slide is really if you can link each learning objective to a specific gap that's often done way early in the planning process also if each outcomes assessment question is linked to a specific learning objective and what I do when I write these questions is I have in front of me on my desk either on paper or on a screen it's nice I have two screens I have the gaps that were identified the learning objectives and the content and those are the things I need in front of me to really craft meaningful outcomes measurement questions and to make sure that the linkages there and I know the software the learning management system software that Jared Jeremy works in facilitates this awesome so when your alignment is good you have we believe the best chances of success and and Derek I guess we're not telling people that they have to have two screens to do this right no just maybe a couple pieces of paper a big piece of paper will do it or you can have three screens like Jeremy yeah well some people in fact will will have all of this outlined way in advance and in a chart that you described and they will then just work from that to craft the questions or have a medical writer quiet craft the questions or a faculty member can use those but honestly it doesn't require a little bit of work if you're having faculty write questions there's often some some training or education needed to help them with that alignment it does take some time to make that happen right so we're gonna we're gonna take to take another polling question here from from you and what we'd like you to to do here is we're going to talk about some of the specificity issues and some of the ambiguity issues that we alluded to earlier and discuss those a little bit further so this is the question when when you promise level 5 outcomes results in a grant proposal what does this mean to you so there are various options here so one is does this mean you're going to do you're going to include chart polls in your grant application based on participant performance does it mean you're going to look at performance changes self-reported some of those changes by the participants does it mean you're going to ask them what performance changes they plan on undertaking are you going to ask them about patient outcomes reported by a participant or is your answer I have absolutely no idea and this is great great we have a very with a bunch of very honest people out there so we're kind of we have a chart poll we have self-reported subjective objective or I should say objective subjective respectively we have about 42 percent saying I have absolutely no idea very very very nice results actually yeah I think the the two the first two answers that when you say this level 5 outcomes results means chart pull data we got the group of people indicating that and others saying performance changes self-reported by participants that's that's really what we're getting at here we purposely had all this ambiguity in this because level five can mean different things to different people and as a result if you for instance promise this in the grant request or even internally what does it really mean so you know more at all defined differences in in level five outcomes results a subjective and objective where your response number one would be chart pull data would be objective data on performance change from chart and number two would be subjective so it's really important in your planning to specify kind of where you're headed with this any other comments Christopher no I think I think that's it if some some supporters may consider level five outcomes results as being a little ambiguous if they think that what you mean is you're gonna include chart polls and what in fact you had intended was that you were just going to have some subjective self-reported changes ordered by that by the participants then you're you're you're you makers you may get into a credibility problem with further for the future so it is really important it Derek would appear that our needs assessment for the web and Jeremy our needs assessment for the webinar was was spot-on because a good proportion of people said responded to this one very honestly and we do appreciate that I have absolutely no idea so well then that could also mean that they've identified that this is a particularly ambiguous question it early they can't they you know it's not specific enough so right no no right answer per se but thank you for recognizing of course that that was not patient outcomes so we're on to key number four which is to keep the plan simple and specific so this is really about addressing in your planning the what the how and the when a lot of people will miss the wind also so you can talk about what you're going to measure and generally that means at what level you're going to discuss things or or measure things how is a little bit more detail on the practical methods that you're going to use either we're going to use quality assurance quality improvement or patient safety data in some particular way and here's when we're going to collect it or we're going to be asking people about their commitments to change in an evaluation form at the end of the activity or we're going to use an ARS system to ask pre and post knowledge confidence and competence questions we think there'll be about five questions pre and post so getting down to those levels of specificity really help you in early planning stages and if you want a detailed plan and this is particularly necessary and we think helps in a commercial support when you're asking for commercial support is to be tinted to include the items that we've listed here so outcomes objectives those are your learning objectives those are what you want to accomplish with your outcomes we want to measure XYZ what we want to measure levels changes in knowledge and competence we want to use those results to establish the effectiveness of the activity that you agreed to which we met two learning objectives and we wanted to use it to improve future CME those could be some of your objectives your overall methods which is how you're generally going to approach your met your your measurement activity specific outcomes details so if you have multiple components in an initiative why not describe a little bit about how you're measuring and assessing each one rather than listing your activities and saying we're going to measure knowledge change well how are you going to do it live versus how you're going to do it online are you going to do some people say we're going to measure up to level five on all of these activities well how are you going to do that thinking through those details really helps you again understand the resources you need don't be scared by needing to explain your statistical analysis in many cases you're going to be using descriptive statistics which might be percentages and numbers of people or we call frequencies means you know mean scores or averages of responses that kind of thing but go ahead and describe that that sets up expectations but if you're going to go further than that and you need to assess changes say on your knowledge questions pre to post well what tests are you going to use what statistical tests so it's helpful to describe those and then what is in a results report people say well will report the knowledge results well is it going to be in PowerPoint is it going to be in Word how long is it going to be and what are you going to include in the report how far is it going to go and we highly recommend that there is always an executive summary section that is very very brief and focused almost like one slide or one page that can be used to crystallize the results of what what you did and also specifying when you're going to be delivering the results so if you've got an online activity that's going to last two years one to two years when does your first report on that going to be is going to be after two years or are you going to do an interim report on pre post knowledge change at six months when the vast majority of respondents are going to be there already that just helps you internally as you're as you're planning things also how are you going to share your results increasingly people are learning that just checking the box to say we did our outcomes assessment is insufficient and clearly it is if you want to learn and get better in cm e if we all want to do that we're going to need to spend more I'm looking at our results and figuring out what to do because of them so do you have internal audiences that eat the results your CME committee the faculty that helped you with this activity do you need to get more resources can you demonstrate effectiveness to get more resources internally what about your learners do they need to see this how will this help them clearly in a hospital setting where you've been able to convert your Grand Rounds from interest-based CME to quality improvement and performance improvement dialogues over time with with people in a particular specialty or department seeing those results evaluating them and making more plans for change are just key so those are some thoughts anything else that I missed there Christopher I I think it's all these are all really really important points I I think the the last one I mean these actually I think for all of these there really have been some big changes with all the pressure on the sources of funding for CME these five points here have become very very important and and the better you are at doing these things clearly the more likely you are to be successful especially I think it's a real paradigm change to begin to really formally report on your results whether it's a poster or a publication that's very new and that will actually help you a lot if you apply for another grant and saying that you've actually had something from a previous grant published yeah thanks I think we've covered these points on the next slide I when you have a really nice specific outcomes plan it becomes an implementation guide it just it's there it helps with timing it helps those project managers or CME coordinators who are involved really implement what you have in mind and I think we've established also that if you're overly ambiguous in your plans and/or what you describe it creates misunderstandings people will think you're delivering objective level performance data when you really had only in mind to ask clinicians in a follow-up survey how they changed their baby their their things in their practice or if you're too ambitious it's really important so many commercial supporters are a complain that CME providers over promise and under deliver and that that's a that's a challenge to overcome and and having a clear plan really helps you make sure you have the internal and external resources lined up to do what you're promising so key number five use multiple types of assessment questions I call this if you just use one type of assessment question meaning if you have only say two multiple choice knowledge questions you're putting all your eggs in one basket if you're measuring knowledge and competence change mix it up use a variety of question types and we certainly can help with and follow up with the kinds and types of questions but agreement scale questions confidence questions case questions no mix it up so you're not using only one type of question and always always make sure that they are aligned with the gaps and the learning objectives so just to recap here here are the five keys again plan ahead way ahead to eliminate surprises ending up at the end of an activity and realizing you don't have a plan or the resources to complete your outcomes assessment number two measure to your educational design / measurement is a drain on resources and often a waste of time has less chance of success align align a line that really is the most important key to ensure your xs/s success key alignments would be gaps to content two questions of course there's other and others in between keep the plans simple and specific really reduce your ambiguity here and be realistic also using multiple types of assessment questions don't put all your eggs in one basket with the types of questions that you're using so what do you do then to leverage the value of your outcomes results we've talked about this a little bit but what can you do with them how can you use them really if you summarize your results succinctly and I talked about executive summary type type of results describing them that way it can can be used in a variety of ways number one you can validate your needs assessment if you're a priest chorus for instance on a pre survey showed that knowledge levels we're a little lower than they ought to be or significantly lower than they ought to be you've you've nailed your needs assessment you can kind of quantify the degree to which you met your learning objectives if you've got good linkage there between your questions so you can say because people answered increased correct responses on this quiz pre to post we met XYZ learning objectives or this is the degree to which we met them you can look at different question types and over time establish which types of questions you like to use which are affected which are with your audience you can identify content which isn't quote performing well if you believe your content is linked to your objectives and gaps but you're finding that your outcomes measurement questions are not seeing changes positive changes in your learner's maybe you have some content or to review embolden the evidence for that content shift the content in some fashion and of course all of this is meant also to help inform future content development make improvements in your content and process over time Christopher you mentioned the first one on this next slide folding your outcomes measurement results back into a grant request definitely helps a lot if a if a someone you're submitting a grant to sees that you've actually incorporated the results of a previous grant into a new one it really it really forms begins to provide a cycle here where you're you're learning you're applying what you're learning in to the next program so that you're continuing to evolve your learner's further and further along the continuum yeah and here are just listed some other things I'll just mention the last one why not share your results in the clinical and see Emmy community to and and that could have variety to you sorry they're going to add value to your organization in a variety of ways Derrick one other thing we're talking about the the assists commercial support isn't selling the value of CM e within an organization sometimes just having you know having your results reduced to five four or five PowerPoint slides which you can just readily present to an audience and so that you summarize your results it really helps you to engage you know internal people internal stakeholders who have a say in your future budget it can be used to communicate your your external stakeholders so sometimes just you know being able to say something in a simple way and having a PowerPoint because in some organizations powerpoints are the currency of communication so yeah yeah thank you for that yeah sorry Jeremy continued on please yeah so that's that's really it Jeremy you wanted to talk about some tools that can be used to achieve success in yeah absolutely and thank you both for a wonderful discussion you know obviously as we've mentioned at the beginning you know tools you know such as prea OCE learning management system or other types of tools but one of the things that I wanted to touch on it and I really appreciate not just I mean the points that you've made about how this can be used for gaining commercial support I thought we're wonderful but also focusing on how this really can be used to increase the perceived value of the CME Department within an institution it made me think of you know some of the barriers that that my my colleagues face who do you think would be some of the the because I think collaborations in addition to technology for collaboration whether it's with external partners or internally is certainly a key to success with any type of strategic plan and certainly the implementation but within an organization who who would you say would be some of the key stakeholders that should be at the table and actively engaged to in order to make outcomes assessment not just going from a you know an idea or a plan that's been discussed but actually becomes you know ingrained in the DNA of your organization well you know for for has nationally accredited accredited CME providers I realize we have a diverse group listening in here many hospitals and health systems CME providers that may be state accredited and then we have some national nationally accredited CME providers but what we've found is that when outcomes measurement and kind of the levels that you're targeting to measure are part of your mission part of your vision for what you want to do as an organization and are established with your CME director and even the you know the deans the executives and other executives in your organization when that when that mission is defined and includes outcomes the kinds of outcomes you're pursuing that can become very powerful and a guiding force for you and so measuring to your design is what we've talked we focused on but measuring to your mission is really important also there's an important thing happening within health systems and in hospitals with respect to see Emmie and their many folks are trying to move from a model where they are cost center and our and are you know giving out what some people refer to as CMEs which hasn't been as good a good an approach as possible in that setting moving to a situation where they're engaged with patient safety quality improvement quality assurance folks and they are now a partner in achieving those goals those quality goals and many as they've been able to do that have facilitated it by being on the quality assurance or Quality Improvement Committee and vice versa inviting those people to be on their committees it makes eme more valuable so and I'm hoping we can get to some questions here or was one of that was what was that one of the coin yeah well and that's been kind of a common theme and there's also been a number of questions around getting a little bit more in depth around some of the Moore's laws that that you had been discussing earlier more arrivals yeah most levels of you some people consider them laws they are level and they're there are is it's a good framework to work from there are other kind of models that people work from stages of change stages of learning kinds of models but can you be any more specific with what needs to be addressed with more levels well actually that was the question was and I think in these obviously the questions were coming in as you were discussing yeah what are the appropriate you know methods to use and and I think the basis was the desire for some elaboration on that yeah so or is there or other there are other methods that you would recommend yeah I think there's if you look carefully at Moore's article and many of us have studied it but we often don't go beyond the nice than the nice table that's shown there and we kind of we nail down those levels and what's subjective and what's objective but what we really need to do overall as a CME community is move from measuring just knowledge change to competence change and then further up the ladder so any effort that you can make to do that is helpful so how do you measure competence change for instance in in folks that are coming to an activity say a grand round or a satellite symposium how do you measure that well some case many cases and yet questions are good for this because if you ask those pre and post then you'll be able to assess what pay what your learner's would likely do with the patient if they saw that patient and that's considered an objective measure of competence change using a case vignette there's also measuring competence change people have been doing that already to varying levels of quality or sophistication so some people will ask in an evaluation form do you plan to change your practice well that's not quite good enough to measure competence change go beyond that to measure how do you plan to change your practice based on what you learn today the next step would be give them a list please check all that apply I plan to make the following changes in practice check all that apply this the changes that you believe your content was going to motivate people to do the things that you knew from your gap analysis they're not doing enough of but now you're going to see if they will check those and do them at least an other category for them to specify what they might be changing so that's another step up we use an approach that's been published in in several places called current and planned frequency of use so it's as simple as to measure a competence change how often do you currently use each of the following clinical practice strategies so be a list of behaviors or things that you knew were deficient in your learner's your content addressed that they should be doing these things according to the evidence evidence-based practice and you ask them how often are you currently doing them never – always 1 to 5 scale and then how often do you now plan to do these things in your practice based on completing the activity today and you can see that you get a measure of current frequency of use planned frequency of use and when that increases for something you intended them to do more often that's a change in competence they intend to do something good more often great and so that's just an example we should go read another question yeah and then just because we are coming up on the hour so I'm going to kind of consolidate – about three questions into one which is around resources and I guess the way I would sum it up is obviously a lot of you know CMU departments are under a lot of financial stress and the resources limitations and someone asked you know is is what's being discussed here realistic if you don't have a bio stats person on staff or or available to you so how realistic is what we're discussing here today in the context of having very limited resources right so in an organization where you have very limited resources I guess you have to not only measuring to your design or assessing to your design you're assessing to your resources but what I found is that through including the right kinds of people on your committee you sometime you sometimes can draw people into helping you with this kind of thing and that allows you to expand your resources clearly there's less that you can do when you don't have those external resources and you have to be realistic but there are believe me there are very very simple ways that don't require a statistician for you to move your measurement your assessment from knowledge change which sometimes can actually be more complicated if you got pre and post questions and you've got to figure out was there a statistically significant increase in knowledge changed knowledge pre to post that can be fairly complicated for some folks but sure there are there are other ways to assess change that are very simple and incremental from perhaps where you're going that require you only to be looking at percentages and and frequencies or numbers of respondents and maybe some mean scores and I realize that's a quick answer but we'd certainly be happy to follow up on what could be done there sure absolutely well I would like to thank everyone in our audience today I do want to be mindful of everyone's time and absolutely again would like to thank Derek and Christopher for their insights today thank you thank you

1 comment

Leave a Reply

(*) Required, Your email will not be published