How to Turn Learner Feedback Surveys Into Better Learning Outcomes

Show Kickoff

[00:00:00]

Chris Van Wingerden: Welcome to Instructional Designers in Offices Drinking Coffee, #IDIODC. It's Wednesday morning. Hope you're feeling a a a good way, in the week so far. As always, this episode of IDIODC's brought to you by the team here at dominKnow One, helping L&D teams to develop, scale, and deliver learning that maximizes employee value. Folks, today, our guest is Dr. Will Thalheimer.

Meet Will Thalheimer

Chris Van Wingerden: Will, you've been with us a couple of times previously, but for folks who haven't met you before, share a little bit about yourself.

Will Thalheimer: So I've been in the L&D field for over forty years now. It's pretty frightening. And, I've done a lot of different roles over the years. One of the things I do, I've been a consultant for myself. My practice is called Work-Learning Research.

Will Thalheimer: And I started out doing the learning science stuff, helping people build more effective learning interventions by looking at the learning science, to align with what's good with that and to, you know, get rid of things in our learning designs that aren't working that well. Along the way, I got very interested in learning evaluation. I noticed some things that we were doing that were not aligned with the science of learning, and I got curious and, went down a rabbit hole. And so I've been focusing on that as well as the learning science stuff for the past, fifteen years or so.

[00:01:00]

Chris Van Wingerden: Yeah. And I will For folks who haven't, you know, encountered some of the research projects, your The work that you've shared on spaced repetition in particular has always been a very, helpful, helpful resource for me. I've always appreciated that.

Chris Van Wingerden: And I think in in prepping and talking to some folks on this side here about today's session, I've more than once referred to you as, probably the most generous person in in the L&D space for the the research that you do provide off of your website. So valuable for so many of us to to make sure that we're always improving our practice and thinking about those things. So, both of it.

[00:02:00]

Will Thalheimer: Thank you. Thank you. Appreciate that.

Chris Van Wingerden: Yeah. Well, we appreciate it more. Trust me. Awesome. So we are talking about one of the areas that you've been, you know, focused around for for a while, including some some new stuff that you've been working on in this space.

The Trouble with Smile Sheets

Chris Van Wingerden: But, taking, like, you know, learner feedback surveys. So many folks do these. I mean, there's a standard. We call them smile sheets, and that's not always a smile face that we say it with because there's a That's a bit of a, dismissive or almost disdainful name for them.

Chris Van Wingerden: So I think, you know, in terms of the value that so many people actually perceive in them. But that's what we're gonna talk about here today, making sure that you can take better advantage of that type of feedback and to and to do actual improvements to learning outcomes, etcetera. Let's maybe start with, you know, what's the problem with smile sheets? What do people do wrong?

Will Thalheimer: Yeah. Well, first, I I want you to know, Chris, that I know we're gonna talk about some momentous things today. So I really prepped for this. I'm I'm I don't usually have a second cup of coffee, but I bought my second cup of coffee.

[00:03:00]

Will Thalheimer: I had cold pizza for breakfast, so I'm really I'm really raring to go.

Chris Van Wingerden: Sounds like you were out partying last night if you're if

Will Thalheimer: you're having cold pizza. We had that big event here in this Right. Oh, fair enough. So, so smile sheets.

Will Thalheimer: Yeah, sort of derogatory term. There's a whole bunch of things we do wrong with them. In the in the research, I know we're gonna talk about the GROWLE research, which stands for the global research on workplace learning evaluation. We ask people, and organizations, L&D people throughout the world, what practices they actually use on their learner surveys. And, came up with a lot of things, but a lot of them were poor practices, which is not great.

[00:04:00]

Will Thalheimer: I got I actually got started in this thinking about learner surveys when I saw some research on, some meta analysis on the correlation between our learning results and our smile sheet ratings. And I will say that, the the finding was very startling. The correlation was 0.09. So, what statisticians tell us, anything below 0.30 is a weak correlation, so 0.09 is virtually no correlation at all.

Will Thalheimer: So what that means, we get high marks. We could have a very good course or equally likely a poor course. So we get low marks. We could have an ineffective, poorly designed course, equally likely a well designed course. So with our traditional smile sheets, we just can't tell.

Will Thalheimer: The biggest thing we do wrong is we use Likert scales and numeric scales, which are very fuzzy. Mhmm. It makes it hard for our learners. Well, first, it makes it demotivates them. Right?

[00:05:00]

Will Thalheimer: Ever seen people circle the same number down their smile sheets? Well, paying attention. And if they're not really paying attention, then we're They're You know, we're getting data that's not meaningful.

Will Thalheimer: So number one, our Likert scales, numeric scales are boring for people. Their eyes glaze over. They know we're not getting good information, so they don't give it very much. But also we're not supporting them in thinking. They're struggling between strongly agree and agree, neither agree nor disagree.

Will Thalheimer: And so, if they're struggling, then the data we're getting is not that useful. And and then another thing is we're taking an average. And if you ever looked at smile sheet averages, they all go from, like, 3.8 to 4.5, and there's no differentiation. Well, what does it mean? Are we doing good?

Will Thalheimer: Are we not doing good? We're never sure. We have arguments about it. We get paralyzed. We don't do anything, and that's no good because we evaluate so we can do something, so we can make better decisions, take better actions.

[00:06:00]

Will Thalheimer: So there's a whole bunch of things going on there, a whole bunch of problems, and yet it's the most common thing that we do. Learner surveys are the way we survey everything, including impact. On the GROWLE survey, one of the stunning results is we ask people, how is your organization measuring learning impact? And by far, the number one thing people do is measure that with a survey. And surveys are a terrible way to measure learning impact.

Will Thalheimer: So, yeah, a a lot going on there. I'll throw it over to you. We drilled it down over to

Chris Van Wingerden: Well, and I think, you know, most folks who have, I mean, you know, the It's nice to have a number or something that you can show to the bosses that, oh, you know, we get an average 4.2 rating, I should say, you know, on our content, or on our course that, that people took. But, you know, does it It doesn't give us anything either, you know, substantive to improve the content necessarily unless we're actually the exceptions, you know, when people, and if they've even got an opportunity. If it's nothing but a a set of likerts that people just tick a box, you don't actually have anything concrete to incorporate as actual feedback then either.

[00:07:00]

Chris Van Wingerden: Like, what doesn't tell you something that they felt was wrong with, you know, the course, or the project too?

Four Pillars of Training Effectiveness

Will Thalheimer: Yeah. Absolutely. So one of the things I encourage us to do, with my performance-focused smile sheet stuff, And by the way, there is a funny story about this is the second edition of the book, performance-focused learner surveys. The first edition of the book said performance-focused smile sheets. I was trying to resurrect the name smile sheets, you know, to bring it back from its negative connotation, then I failed.

[00:08:00]

Will Thalheimer: So I went back to learner surveys. But, yeah, one of the, one of the, issues we have is that, we're asking questions that are focused on the wrong things. We're asking about their general satisfaction, we're asking about the reputation of the course, we're using the NPS, the Net Promoter Score, and those don't really relate to effectiveness. So one of the things I recommend is, you know, what kind of questions can we ask that really relate to effectiveness? So I talk about the four pillars of training effectiveness.

Will Thalheimer: And this is an oversimplification, but, number one, we hope that our learners are able to understand the content. That they got it, that it's clear, that it wasn't confusing or contradictory. So understanding is one pillar. Another pillar is, are they motivated to apply it? Not only motivated to learn, but are they motivated to apply what they've what they've learned?

[00:09:00]

Will Thalheimer: Number three, are they able to remember something? Right? We've taught them things, that's great, but if they can't remember it an hour after the course or a day after or two days after, that's not gonna be very useful. So that's another pillar of, training effectiveness. And the the fourth one is, are they getting after training supports?

Will Thalheimer: We know from the research on learning transfer, that what happens after training could be just as viable is what happens during training. And so that's the fourth pillar. So we can ask questions related to those things. And you know, obviously, in in the book, I say that at least at least 20 times, Don't just use learner surveys. Please.

Will Thalheimer: This is just the starting point. It's a good starting point, especially if we're using the ones that are effective, but don't just use learner surveys. We can we can go deeper with that. Mhmm.

[00:10:00]

Chris Van Wingerden: I mean, even so so those four pillars, some of those to gather that kind of info, you know, trying to understand if someone has has actually taken something and applied it. Well, that takes time. Right? No one's gonna finish up a course in the LMS turnaround. In very rare cases, would they turn around and immediately need the information they might have consumed?

Chris Van Wingerden: They might have gotten 20 out of 20 on the on the multiple choice quiz, partly because they could pick up You know, remember just enough to to pick the right answers or or whatever. But, having someone actually apply something, that takes time. That's not something that you could even do in an immediate You know?

Chris Van Wingerden: Hey, congratulations. Now finish up the You know, you got 20 out of 20. Now here's the survey for you.

You Can't Survey Everything

Will Thalheimer: Well, and we have to be careful. You know, let let me step back a minute and tell your listeners, your viewers something that's kinda scary. In the L&D field, we have a lot of misconceptions about learning evaluation. You know, most of us, focus our time on developing learning, and so we spend less time thinking about evaluation. So there's a lot of misconceptions.

[00:11:00]

Will Thalheimer: One misconception, probably one of the biggest problems we have, is that you can survey everything. You can ask the learners about their knowledge. You can ask them whether they're better able to make decisions. You can ask them about whether they're better able to do tasks. You can ask them about whether they put things into practice or whether they're getting higher revenues or lower cost.

Will Thalheimer: You know, these are, these are, this is a false idea. You know, some things we can survey and some things people have good intuitions on. But there's this really strong set of research that shows that learners, and we're all learners, but when we think about learning, we have some misconceptions about how learning works. And so if we're asked to give our feedback on learning, we might give poor feedback because we don't have the right insights. And so we have to be very careful about the questions that we ask.

[00:12:00]

Will Thalheimer: We have to ask questions that our learners actually have intuitions about or accurate intuitions about. So if you get a question like, and this is one that's actually used, I'll see if I can remember the wording exactly, but something like, the design of the program, helped me learn, or the organization of the program made it easy for me to learn. Well, learners aren't good at knowing that. Right? They just don't know it.

Will Thalheimer: So why are we asking that question? We're getting false information, when we do that. So, like, if I wanna measure, whether behavior has changed in the workplace, A survey, we might have to do that, but it should be our last, the last thing we try. Better to see You know, better to get more, objective data if we can.

[00:13:00]

Will Thalheimer: So if we're measuring sales people, well, how many sales calls did they actually make? Or if we measure call center reps, how long is it taking them to finish their calls? Did the people call back? Do the customers call back? You know, these are more objective measures if we can do that.

Will Thalheimer: And then when we when we measure workplace performance, workplace behavior, we have to be very careful that it's the training that made a difference, not something else. And that's where, you know, that's where your focus, your question about all the effort, that's a major effort. Mhmm. You know, people come to me and say, hey, Will, we wanna do an ROI, or we wanna measure, Kirkpatrick's level four, we wanna look at, you know, business results, or, and as a good consultant I say, that's great, I can help.

[00:14:00]

Chris Van Wingerden: Mhmm.

Will Thalheimer: And, then I We start talking about it, and they begin to realize what it's gonna take to do, an analysis of, you know, actual, workplace outcomes. And they go, oh, no, no, no, no. That's not what we wanna do. Everybody has this idea that it's all about surveying.

Will Thalheimer: In fact, what do we call our surveys sometimes? We call them evals. And that That's like No.

Will Thalheimer: Evaluation, a survey is one part of what we can do for evaluation. It's not the only thing we should be thinking about doing.

Proving the Training Made the Difference

Chris Van Wingerden: I was thinking, as you were first mentioning it, and then you took, you know, that that additional tangent, about, you know, do do we know that the training is actually the, the thing that caused a a behavior change? Had a a conversation back, I think, maybe in the summer, with a with a team who had rolled out a new training program. It was around customer service, reps, etcetera. And, saw changes, you know, times improved, you know, all those things that we would be looking for. They were looking for that.

[00:15:00]

Chris Van Wingerden: They're looking at that external data. But then they were challenged by the fact that, like, someone openly challenged them and said, yeah. But how do we know? Because we actually made changes to the CSM, the CSR tool, you know, that they're working in. So there were different workflows and stuff.

Chris Van Wingerden: How do we know that that wasn't that wasn't? And I forget the the This is where my memory starts to fade a little bit, but they were able to use, I think they were using, like, x a p X API data and and some other things as well as some reporting and basically did Were able to demonstrate that it was indeed, the training.

Chris Van Wingerden: They might have even had some AB, you know, groups without With and that did and didn't take that. But basically, it had to to demonstrate that, yeah, it was the training that resulted in, these these improvements and not just simply that the the tool got an update coincidentally with the timing of of people taking the training.

[00:16:00]

Will Thalheimer: Well, just think about all the things that can affect workplace performance. Right? I mean, maybe you have a change of management. Maybe you have a company as a merger. Maybe the economy gets better or worse.

Will Thalheimer: Maybe your company introduces a new product. Maybe someone gets sick in your office. Maybe the weather changes. You know, there's so many different things that could affect, our results. And, you alluded to it, but sort of the gold standard is randomized controlled trials.

Will Thalheimer: So let's say I'm gonna put a 100 learners through a training and, I don't take the first 50 and put them in one group and then put the next 50 in another group because that could be biased. The people that sign up might be the more go getters. Right? Mhmm. I gotta randomly assign them.

Will Thalheimer: So I take those 100 people. I randomly assign them to getting the training now or getting it later. So maybe I train one group in March. I do my study in April, and then I train the other group in May. But, yeah, that's the kind of thing that's sort of the gold standard, what we wanna do.

[00:17:00]

Will Thalheimer: Now we can't always do that. Sometimes we need time series analysis where we measure before and after and then later and later and later or pre and post. When when I'm talking with clients about this, I take them through the strengths and weaknesses of all these designs. But the one thing you can see through this whole list is that this takes effort. Right?

Will Thalheimer: This is an investment. And, not of us Not a lot of us wanna make that investment. And so we default to our learner surveys, which give us some good information, but we have to be very careful, about that information.

Design Evaluation In from the Start

Chris Van Wingerden: Yeah. So, you know, when you're working with, you know, the client, you know, teams, and you're walking them through all of this, but, you know, what kind of a role or or what kind of steps in the process and and where do you put, you know, the evaluation aspect of things? Like, it seems to me that you need to be thinking about the evaluation as you're designing, you know, the program.

[00:18:00]

Will Thalheimer: Bingo. Bingo. Now think about ADDIE. Think about ADDIE. What did we do with ADDIE, right?

Will Thalheimer: Yeah. As ADDIE sort of hints that maybe we should do evaluation at the end. Well, yeah, but we really ought to be designing evaluation in from the very beginning. In fact, I have this learning evaluation framework. It's a replacement for their Kirkpatrick, Katzell four level model.

Will Thalheimer: It's called LTEM. It's a one pager. It's got eight tiers in it. It brings sort of learning wisdom back into the design. And, you know, I I tell people, okay, we're gonna start, you know, with the general, overview.

Will Thalheimer: What are we trying to accomplish, etcetera, you know. And then we're gonna, you know, we do our needs analysis. And right then in the very beginning, we lay out, well, let's brainstorm all the ways we might measure. We're not gonna do all those because that's way too much of an investment. But what are all the things we might measure?

[00:19:00]

Will Thalheimer: And that's important because then we're not just defaulting to surveys or, you know, measuring completion rates. So we brainstorm all the things and then we take it back and says, well, what what are our most important things? So maybe we're, you know, if you're, training, sales managers to coach their sales reps. So you might brainstorm, you know what, a good measure of quality of coaching is whether the coachies, feel they're getting a benefit from it. Yeah.

Will Thalheimer: You know, the coach is not the best judge in that, the coach he might be. And so that's one thing we can measure and that would be up here at tier seven, you know, how the the the coaches rating it. And maybe we have some, deep, well written scenario questions that we validated. We might give that pre to post and see if people are improving their decision making on coaching situations. That would be a tier five.

[00:20:00]

Will Thalheimer: We might measure some knowledge stuff at tier four. We might use a performance-focuseded learner survey at tier three. So we brainstorm that first. What are the All the things we're gonna measure?

Will Thalheimer: And only then do we think about what our learning objectives are gonna be. One of the problems we've had in the learning objective thing is that, you know, the action verbs, the learners will be able to list or explain or describe. Those are very low level and that pushes us to think about evaluation of measuring knowledge And sometimes knowledge is important But too often we measure trivial knowledge not important knowledge And we probably wanna measure more than knowledge, you know. Can they make decisions? Can they actually perform the tasks?

Will Thalheimer: Have they actually put it into practice in the job? Are they getting the business results? Are they getting the results for the learners or the beneficiaries of the program, etcetera? You know, there's a there's a a lot to think about in evaluation. But one of the most important things we can do is to think about evaluation from the very beginning.

[00:21:00]

Chris Van Wingerden: Mhmm. And even just that evaluation, having it centered around, you know, what are the actual goals? Not, you know, not about information. It's, you know, do we want to improve CSR response times? You know?

Chris Van Wingerden: Are we looking to improve, you know, various sales metrics? Like, we might You know, we have a great amount of information, but by setting our goals around and understanding that that's what we're going to evaluate, then that makes sure that we're giving the right information and not just all the information.

Don't Only Measure Impact

Will Thalheimer: Let's This is actually tricky. This is actually tricky. So, if we measure just the goals, if we measure just for the learning impact, that's fine. That's good.

[00:22:00]

Will Thalheimer: But what are we missing then? Well, one thing we're missing is all the leverage points that enable you and I as learning designers, as instructional designers in our offices drinking coffee to, you know, make good decisions about the learning design. What should we keep? What should we get rid of? What should we improve?

Will Thalheimer: If you just measure the impact, then we're blind to all the learning things. I wrote a article recently, called 10 reasons not to measure impact. And that's a provocative title, and I don't mean that we shouldn't measure impact, but my my message is we shouldn't just measure learning impact. But we tend to default to that. People think that this is what our senior leaders want.

Will Thalheimer: You know, one of the really interesting findings from the GROWLE research is that, you know, in thinking about what what does senior What do our senior business leaders want from us in doing learning evaluation? So, I asked this two part question. One was, hey, have you actually talked to your senior leaders about what they want to learn evaluation? And only 31% of us said yes.

[00:23:00]

Will Thalheimer: And then if you drill down on those people and you say, okay, what what what have you heard from your senior leadership? 37% did hear that, they want, a focus on, workplace results and, you know, the benefits of that. But almost an equal amount said, you know, we actually haven't heard what our senior leaders say and actually we're pretty sure they have not said anything about learning evaluation. So But even those who said yes, you know that 37% that said yes, that's 37 of 31%.

[00:24:00]

Will Thalheimer: If you do the math, that's 11% of us have heard from our senior leadership. So it's not really true that all of our senior leaders are advocating for a certain kind of learning evaluation. And when I think about what a senior leader might want us to do, some certainly are gonna want that. But I I think others, you know, let's say that they're giving us a 100,000 or no, a $100,000 to do learning evaluation, more or less, whatever it is for your relevant for your organization. They're gonna give us a certain amount of money, And, you know, you ask them, well, would you like us to use that $100,000 to try to prove that we are being beneficial?

Will Thalheimer: Or would you rather us use that $100,000 to improve, to find out what's going on in our training, to seeing what's works and what doesn't work, and to improve it. I think a smart business leader is gonna lean toward that second one. They're gonna want us to use it for our benefit, so we can make better decisions, take better actions, so we're gonna actually create more value, create a better result, than to create performative metrics to make us look good. You know, I don't think a senior I don't think a senior leader, a sensible one really wants that. I I do think that's a big myth in our in our space.

LTEM and the Kirkpatrick-Katzell Model

[00:25:00]

Chris Van Wingerden: You were showing us the LTEM card. Let's let's talk a little bit about More about it. Like, how is it different? You mentioned it in in sort of contrast with the Kirkpatrick's structure.

Chris Van Wingerden: Let's talk a bit about it and and, you know, how you came up with it, that sort of a thing.

Will Thalheimer: So I started working on this, I think, in 2016 or so. Ramped it up. I got feedback from a lot of really smart people like learning experts like Julie Dirksen, Clark Quinn, evaluate excellent Evaluation experts like, Rob Brinkerhoff, Ingrid Gera Lopez, El Hammurabi. And, I

[00:26:00]

Will Thalheimer: One of one of the versions that I created was a 15 level model of learning evaluation and the world is glad that I didn't end up with that. But, so you know, the And so you you might hear me call it the Kirkpatrick Katzell model and the reason I do that is because it turns out that Raymond Katzell was actually the inventor of the four level idea. And then Donald Kirkpatrick, put labels on the four levels and popularized it. So I feel like both gentlemen deserve credit.

Will Thalheimer: One of the strengths of the four level model is it reminded us that we're not just aiming for learning results. We're aiming for behavior change. We're aiming for organizational results as well. Brilliant, you know, very important. But it has some weaknesses as well.

[00:27:00]

Will Thalheimer: And one of the weaknesses is that it puts all learning in that one box, level two learning. And what could we measure in a box like that? Well, we could measure the regurgitation of trivia, the recall or recognition of meaningless information or meaningful information. We can measure decision making competence or task performance. We could measure skills.

Will Thalheimer: Now that's a wide continuum from trivia to skills. What did we used to do when we looked at that four level model? We go, oh, you know, level two, we need a knowledge check. Knowledge is not good enough most of the time to know whether people are able to make decisions, whether they're able to perform the task. So, one of the things I wanted to do with LTEM was bring back some learning wisdom into our deliberations.

Will Thalheimer: So, instead of, one bucket, now we have measuring tier four knowledge, tier five decision making. I know you can't see this, but I like a visual. Neither

[00:28:00]

Chris Van Wingerden: you're coming through pretty I mean, we see the At least the structure, not the words. Okay. Yeah.

Will Thalheimer: Alright. Tier six task performance. So task performance, decision making, and knowledge. So that reminds us as L&D professionals, oh, well, maybe we shouldn't just measure knowledge. Maybe we should measure decision making and task performance as well.

Will Thalheimer: Also, tier three is learner perceptions. This is where we put learner surveys. So it makes a distinction between, asking learners questions that reveal insights related to learning effectiveness or, asking learners questions about, things not related to learning effectiveness. So that, you know, it it sends us a message. I'm also in here in LTEM.

Will Thalheimer: It reminds us about the learning and forgetting in some sense You and I as instructional designers, we're not trying to create learning Learning is a fuzzy word. We're really trying to create remembering we want our learners to be able to remember when they encounter work situations, when they need that information that they learn. So LTEM has some hints in here about, hey, don't forget to measure not just at the end of training, but also after a delay, so you know whether your training is actually helping people remember, not just helping them learn. So there's a bunch of hints like that, in in LTEM that, you know, remind us to think some certain things. I'm a big believer in nudging and sending messages, you know, the behavioral economic notion of nudging.

[00:29:00]

Will Thalheimer: If we if we can have a framework or a model or a job aid or anything that reminds us of things to think about that we might otherwise forget, that's a good thing. And so that's sort of the design, idea behind LTEM is it's trying to nudge us to think more intelligently, about our learning evaluation.

AI and Learning Evaluation

Chris Van Wingerden: And so some of those extended tools like nudging, etcetera, I mean, we haven't yet in in a half an hour conversation. We haven't yet mentioned, AI, which is really weird in the elearning space these days to go a half an hour and not even say the phrase.

[00:30:00]

Will Thalheimer: Scandalous. Scandalous. Crazy.

Chris Van Wingerden: But, you know, are you seeing some ways that people are up Using AI to apply some of that stuff or put that in motion?

Will Thalheimer: Yeah. So, in the GROWLE report And by the way, GROWLE was done this year. We gathered data in, January, February, and a little bit of March. So the data is fresh, but AI is moving fast, so things could have changed a bit.

Will Thalheimer: But we asked people Asked L&D professionals, is your organization using AI and learning evaluation yet? And the split was about fifty fifty. Some people were doing it, but a lot of organizations were not doing it. Incidentally, one of the answer options was, well, we're not doing it, but our learning evaluation vendors are doing it for us.

[00:31:00]

Will Thalheimer: Hardly anybody said that so I think this could be a big business opportunity for learning vendors to go out there and do it. But generally, people are not doing it And those who are doing it are not doing it very extensively when I saw that, I said, well, you know, we really need to explore this. And I've been exploring AI a little bit. I'm not an expert, but I've been exploring it a little bit to do learning evaluation. One of the obvious things that everybody's doing because it's been a problem in the past is open ended responses on our surveys.

Will Thalheimer: In the past, we'd have to, like, read through all the answers. We'd have to sort of try to figure out what the patterns are. If we were sophisticated, we do a category sort and put people's responses into categories and then report on those. But now AI can very quickly categorize our data, make sense of it for us. So that's that's a real winner.

Will Thalheimer: You know, there are there are some concerns about AI. You know, the general concerns is are we Is it really telling us what we think it's telling us? Is it accurate, etcetera? So, obviously, we have to think about, those things.

[00:32:00]

Will Thalheimer: One of the big opportunities, I think, that came out of this work, and what people said was using AI, to create, you know, like an, an evaluation sandbox, you know. So you put p You have AI take people through a series of decisions, and that can simulate the realistic decision making they might have to make in their real real work. So a few people are doing that. It wasn't many yet, but I think that's an opportunity, going forward.

Will Thalheimer: But I think experimentation. In fact, I've got I gathered a group of folks together, Stella Lee, Paige Bradbury, and Mike Sanderson. We have now got a group going, for, talking about AI and evaluation. We've created a set of principles, which we're calling AIL.

[00:33:00]

Will Thalheimer: AIL stands for AI supported learning evaluation. In fact, we're meeting November 12. I forgot to bring the link with me. Mhmm. But, maybe I can I can sneak that in here at the end, if I can find that?

Will Thalheimer: But, yeah, come join us. We don't think that we have all the answers. We wanna We're sort of having, you know, developing a community around this. I want to see a lot of examples.

Will Thalheimer: What do people do with AI and learning evaluation and I teach a course called the LTEM Boot Camp. It's a very extensive challenging six week course there's six live sessions. It's actually over seven weeks. Talk about LTEM, but learning evaluation principles in general. And a lot of the folks in my course are now thinking about using AI or they're using AI, and I've seen some really interesting things that they're doing.

[00:34:00]

Will Thalheimer: But, you know, overall, they're still figuring it out. You know, I'm still figuring out. We as a field are are still figuring this out. The group with Stella, Paige, and and Mike and I, we have some concerns. There should be some guardrails and some principles, about data protection, you know, data accuracy and things like that, that we need to think about.

Will Thalheimer: We should shouldn't think this is a panacea that we can all run toward this without putting some guardrails up. But, I think it's a real opportunity for us going forward.

Chris Van Wingerden: Yeah. Mark, who's joining us, on LinkedIn today, did did note in the chat there that, that he's used AI AI for sentiment analysis. So what you were, you know, saying And now analyzing the, you know, the open ended answers and responses and being able to understand patterns and, etcetera. Oh, and we

Will Thalheimer: have just our link in there too.

Chris Van Wingerden: Right on.

Will Thalheimer: There you go. That's our that's our link. Super. So, yeah. Very good.

[00:35:00]

Will Thalheimer: Yeah. I think that's

Chris Van Wingerden: Saves you searching.

Will Thalheimer: Yeah. Yeah. No. That's that's Well, that's amazing.

Will Thalheimer: It's good to have friends. It's a it's a lifeline. Thank you, Mark.

Three Steps to Start

Chris Van Wingerden: Yeah. Just thinking, I guess, you know, you know, more more broadly, we've covered a lot of, you know, different things. But if someone was looking to do, let's say, some initial steps towards some of the stuff that we were talking about so far today, what would be, I don't know, three things that you would recommend people doing after today's session?

Will Thalheimer: Well, you know, let me let me give the big big picture answer to that first. Like when when a client comes to me and says, hey, Will, we wanna do something with you around learning evaluation. The thing I sort of almost require is, okay. This is great. I don't wanna make the decisions for you.

[00:36:00]

Will Thalheimer: You know your organization best. You know your needs. You know your political situation. What I'm gonna do is I'm gonna give you an evaluation backgrounder. And whether I can convince them to go up to two and a half hours or just ninety minutes or sixty minutes, they ought to have some background so they can think more clearly about learning evaluation.

Will Thalheimer: As I mentioned before, we all have misconceptions about it. And so, it's good to get the basics. So we should all just be sort of learning the basics about learning evaluation, you know, thinking about, like, the learning and forgetting curves. Well, if we relate that to learning evaluation, we can see how important it is to measure after a delay, not just immediately, because, forgetting doesn't happen right away. Right?

Will Thalheimer: So that's number one. And then if you're thinking about, well, what can my organization do better? How how can we get started? The best thing to do is build a better learner survey. Get rid of your Likert scales, your numeric scales.

[00:37:00]

Will Thalheimer: I mean, if you have to keep a few questions, to fight off the resistance, that's fine. But try to use better questions. Your learners will like them more, they'll be more motivated to fill out your learner surveys. You'll get better better data. And when you get better data, you'll be able to make better decisions, etcetera.

Will Thalheimer: The other thing is think about what decisions you're trying to make. I I wrote about this in my latest book, the CEO's Guide to Training, eLearning, and Work.

Chris Van Wingerden: By the

Will Thalheimer: way, this this book is not just for CEOs, it's for us as well. And this 50 short chapters, because we all have, you know, very limited, time and capacity now. But six of those chapters are on learning evaluation because it's so important. And one of the things I talk about in the book, I've been thinking about it for, like, ten years, talking about it for ten years, but I didn't write anything about it. It's called LEADS.

[00:38:00]

Will Thalheimer: And LEADS stands for learning evaluation as decision support. And here's the idea. If we're doing evaluation to help us make better decisions, then why don't we first start and figure out what decisions we have to make, and then work backwards from that and build our learning evaluation practices to enable us to make those decisions. So I encourage people to do that, to think about, what decisions you're trying to make and then make sure that everything you're doing in learning evaluation, helps you make those Gets the data you need to make those decisions.

Will Thalheimer: And and then the third thing is, you know, bring learning wisdom back into your, learning evaluation practices, use a model, and I think that LTEM I could be biased, but I think that LTEM is the only one that really has the little sort of learning wisdom baked into it. You know, use it to start thinking about, your your learning practices. Through a doctoral dissertation out there done by Elham Merabi. She's now Dr. Elham Merabi.

[00:39:00]

Will Thalheimer: She's Was successful. She had two hypotheses in her research. This was done with nurses in a hospital professional development. And she said, you know, if I introduce LTEM to a learning team, they'll do better learning evaluation.

Will Thalheimer: Well, that's not that remarkable because it's a learning evaluation framework. But she also had this second one was, if I introduce LTEM to a learning team, they'll do better learning design. And you can see how that might work, you know. We and our teams, we're gonna developing this new course, and we decide we're gonna measure decision making competence. And we're gonna do a pretest to posttest.

Will Thalheimer: We're gonna use validated scenario questions. And so, you could see us saying, you know what? Hey guys, we're gonna be evaluated on whether our learners can make better decisions. You know, we ought to give them a lot of realistic decision making practice. And we know from the learning sciences that that's one of the best things you can do.

[00:40:00]

Will Thalheimer: Give them retrieval practice, put it in a realistic situation, you know, use that context alignment notion, maybe space the learning out over time, the practice out over time. So you can see how if you look at, framework like LTEM, which has learning wisdom baked into it, then it can inspire us to do better learning design as well. So those three things, better learner survey, start with decisions, and, you know, build back, learning, wisdom into your learning evaluation practices would be my, takeaway for everybody.

What's Next for Will

Chris Van Wingerden: Right on. We've added a whole pile of, different resources in the chat if folks are, just joining us, all kinds of things. Will, what's what's next on your your list of to dos or your next I mean, you were talking about the the AI evaluation group. What else has got your interest peaked as as things that you're going to next?

[00:41:00]

Will Thalheimer: Well, I've been, I've been, working to design a course on really a a clinic on how to write scenario questions. Scenario questions are really valuable. I've used AI to, actually use AI in the Boot Camp to give people feedback on scenario questions that they develop in the Boot Camp. So that's I've been looking at that.

Will Thalheimer: I do have a side project I'm working on, which is unrelated to our field, But really important to me, you know, I've been seventy years born and bred American and I've looked at my country and said, no, we're not doing that well right now. So I started about, I don't know, a year or two years ago to write a book on how to make American democracy work better. And so I've been focused on that. So I don't know. We'll see.

[00:42:00]

Will Thalheimer: It's good for my mental health, if nothing else.

Chris Van Wingerden: It gives you a place to get some of those thoughts maybe out of your brain so that you you can allow for other space for other things too, maybe. Yeah. Exactly. I actually, I saw a comment in one of my feeds. Someone was talking about the number of emails that they write and then delete, because just the writing process gets it

Chris Van Wingerden: Gets their reaction out of their brain so that they've dealt with it emotionally, and then they don't send the email even though they typed out everything they wanted to say. So it sounds like at the very least that that could be a project for you, the cathartic aspect of things on that front.

Will Thalheimer: Yeah. And I see, someone in the chat has asked for the name of Elham, and, I put it in our chat for you guys there. Yeah. I will tell you that

Chris Van Wingerden: That up then.

Will Thalheimer: Elhams, she's given me permission to share her dissertation. If you like reading dissertations, and you should be warned, there's a lot of research stuff in there. Statistics and that kind of thing. But she's given me permission to share that. So if you get in touch with me, I can, I can give you, access to that?

Wrap-Up

[00:43:00]

Chris Van Wingerden: Fabulous. Right on. Will, as always, been a great conversation, and I'm looking at the chat and lots of great, pop ups in the chat too, which is always a good, a good measure of of people's engagement and interest in this stuff. And it's it's super important. So thanks as always for joining us here, and, we look forward to watching what you do next.

Will Thalheimer: Great. Well, thank you. It's my pleasure always. Great to be on with you.

Chris Van Wingerden: Excellent folks Don't forget instructional designers and offices drinking coffee drinking tea drinking hot beverages drinking cold beverages Anyway, no matter what you're drinking, it's We're brought to you by the team here at dominKnow, makers of My gosh. Look at you go. You had a rough night last night, didn't you?

Chris Van Wingerden: Maybe. Or a long night maybe, depending on Yeah. dominKnow One, helping a learning and development team is to develop scale and deliver learning that maximizes employee value. Folks, it's time to dance on out of here.

[00:44:00]

Chris Van Wingerden: Have a great rest of your Wednesday, and we'll see you again in two weeks for the next time we all gather for Instructional Designers in Offices Drinking Coffee. Hey. Have a great day, folks.