Does Your AI-Generated Learning Work? Can You Prove It?

Show Kickoff

[00:00:00]

Chris Van Wingerden: hey folks If it's Wednesday, every second Wednesday, let's say, technically, it must be time for instructional designers in offices drinking coffee. Hey, welcome everybody to this week's session. As always, keep in mind that instructional designers in offices drinking coffee, drinking hot beverages, drinking cold beverages, whatever suits your fancy, whatever suits your taste buds.

either way, the episode is brought to you by the team here at dominKnow One, helping L&D teams develop, [00:01:00] scale, and deliver learning that maximizes employee value. Boy, we've been on a real, buzz lately, about AI. But then again, our whole space is on about AI.

Meet Margie Meacham

Chris Van Wingerden: and so joining us this week for a really important conversation, we have Margie Meacham back, joining us again.

Margie, you've been with us on several occasions now, and they've always been great conversations. Anybody who may not have met you yet or, or whatever, tell our crowd a little bit about yourself.

Margie Meacham: Yeah, sure. So I'm coming to you guys from Southern Arizona. beautiful, sunny, still awfully hot, but, and early, so my lighting is a little harsh because, the sun isn't quite up yet.

So, but always a delight to, see you guys. And, what I do is I teach people how to use AI, particularly in the education and learning and development fields, and that is such a moving target these days. you really have to pay attention, so what I try and do is do that for you- Mm ... [00:02:00] and help you recognize trends.

up until a couple of weeks ago, I would talk about governance a lot. I always include it in all my training, and everyone was kind of like, "Yeah, yeah. Okay. Let's go to the sexy stuff." "Let's talk about all the cool stuff we could do." Now I'm getting people stop me and say, "Hey, can you talk to me about this?"

So, interesting how, it's caught our attention lately. So we actually chose this topic, Chris, didn't we, gosh, more than a month ago? Yeah. And, so now we, we look, psychic. our timing is excellent to, uh, talk about this and, what's coming up from us, tomorrow, so we'll talk about that as

Chris Van Wingerden: well.

Yeah. Yeah. You mentioned how this has popped up to the surface, governance, et cetera. maybe elaborate on why you think that is.

Margie Meacham: Yes.

Hugging Face Incident

Margie Meacham: Well Something happened that is being called the Hugging Face incident, although [00:03:00] now, news reports are coming out that

This behavior is happening in a lot of places. It's happening because we're all experimenting with how far can we push these autonomous agents? And so OpenAI was in a test environment, and they built many, many agents, but they gave them different pieces of a task, and then their intent was to have them put it together later.

But initially, the guardrails all said, "Just stay in your lane. Just do the task we've given you." but these agents were running on common platforms and were able to see each other's activity and figured out they were working on the same project, and they started communicating. And they started sharing access and sharing work product and sharing ideas.

And this can all be, it's all in the logs. You can watch it happening. This is both exciting and terrifying at the same time because this is where they actually wanted to [00:04:00] go, they just hadn't provided that instruction yet. And so it raises a lot of questions about just how autonomous are these agents, and can we really control them like we say we can?

and whereas, just a, a more simple, breach in the past makes the news for two or three days, the company addresses it, they say they're sorry, they figured it out, everybody moves on. This is different because this is a fundamental concern about how agents work because of how we're building them, and that's a conversation that's not gonna go away.

Chris Van Wingerden: And it seems like since that initial, report from around the Hugging Face incident, pretty much every other major AI, vendor organization has also revealed- Yes ... that, you know,

Margie Meacham: that-

Chris Van Wingerden: It's happened elsewhere. Yeah ... they're gonna too.

Not gonna lie, it almost feels like they gotta keep up with the Joneses in a sense. Like, "Oh, our, ours are terrifying as [00:05:00] well."

Margie Meacham: Yeah. You know, that's an interesting way to look at it, Chris. Because there is a power in that capability, and if you can harness it, that is truly exciting. So admitting that, "Oh yeah, we had similar behavior," I hadn't thought of it as a kind of weird, quirky marketing response, but I guess it could be.

Chris Van Wingerden: It, it's hard to not s- for me, it's hard to not see that- Yeah ... as a bit of the pattern. I'm not suggesting that anybody- Yeah ... but this sense that everybody, oh, putting up their hand going, "Oh look, ours are powerful too. Ours can go rogue too."

Margie Meacham: Yeah. "

Chris Van Wingerden: Don't, don't forget us," almost in a sense.

Margie Meacham: Yeah.

Chris Van Wingerden: It's, it feels a little bit of that to me, a little bit,

Margie Meacham: Well, I have seen it said that going rogue is a feature, not a bug.

Chris Van Wingerden: Yeah.

Margie Meacham: A- and it may be a feature for the agent, but it's very definitely a bug for the humans. it means somebody failed to anticipate something. And with these, with AI, it's difficult to anticipate it all.

AI Governance Stakes

Margie Meacham: Sometimes you just don't know until you run it and see how it works, [00:06:00] so- I, I'm not blaming the humans entirely, but it's certainly alerted everyone to the need for governance, which is what we're talking about today.

Chris Van Wingerden: in our e-learning world, so not just what, you know, controlling- Mm-hmm ... agents in a sense, but we also have some things I guess that apply to our world of, of learning, and development, et cetera, that also fall under the category of-

Margie Meacham: Yes ...

Chris Van Wingerden: of governance too.

So, where do you wanna start with this part of

Margie Meacham: the conversation then? Well, let's get ... I have a tiny deck mainly to keep me on track, so- Right

Chris Van Wingerden: on ...

Margie Meacham: there it is.

Polished Versus Effective

Margie Meacham: And so started with the question, you know, the great thing about the tools we have now is that it can make great looking slides in minutes, and we don't have to spend as much time on it.

Matter of fact, this deck, in full disclosure, was generated by Copilot inside PowerPoint. I gave it a simple outline, it generated the slides. It did take multiple, back and forth to get it the way I wanted, but, looks [00:07:00] pretty good. So, that's the question. It is so deceiving, these large language models and other tools can make things look good, but do we still have the fundamental learning design behind them, and how would we know?

how do we know anybody's learning, as we say? So let's get, going, and, what I've been alluding to is really the illusion that, and this can happen w- even with a human designer. I'm sure we all know someone who is a terrific graphic designer, and their decks always look nice, but they may not be interactive, they may not have the right scaffolding underneath them.

And I'm not just talking about do you know the buzzwords, but do you apply them? And so there's a big difference between looking polished and being effective, and of course we'd like both. So, the challenge comes when you are [00:08:00] using a tool that has maybe some built-in, AI capabilities to build your draft or suggest an outline or generate images, then a little bit of the control, I mean the whole idea of, an autonomous or semi-autonomous system, is that you're giving up some control.

It's just like you hired an assistant. And when you do that- Your assistant might not build the deck exactly like you would, or the handouts, or whatever learning experience you are building. And so where do you come in, and how do you ensure that learning experience is quality?

So that's what we're gonna talk about today. I'm not necessarily saying we have any answers. Chris and I are just gonna point out some things you should be talking about- For sure ... and thinking about. And, certainly dominKnow has been a leader in that field for a while. And, I built my first chatbot for [00:09:00] the United Nations in 2013, so in this field that's a long time.

we'll both share our insights as we go along.

Measuring Learning Impact

Margie Meacham: So, of the people who are joined us so far, I'm just curious, How are you, determining the, the effectiveness of a learning intervention now? Have you changed it in any way given that some of your content is now AI generated?

Chris Van Wingerden: Yeah. And I mean, this is kind of like the tale as old as time too, right? it's a question that we've always needed to ask ourselves even prior to AI.

Margie Meacham: Absolutely.

Chris Van Wingerden: Right? Like how do we know that the things we make- when people take them result in organizational or behavioral change that improves things, or can people do something more effectively-

when they come out on the other side? How do we measure that? How do we, determine that? How do we know that we're not just putting people through something that takes an hour of their time and doesn't result in something?

Margie Meacham: Absolutely.

Chris Van Wingerden: How do we, how do we validate the cost of t- Yeah ... of all of that [00:10:00] time spent, our L&D team's- Yeah

time making stuff, the staff time to take it? how do we know that that's actually something that, that is worth it in the long run?

Margie Meacham: Right.

Chris Van Wingerden: Yeah.

Margie Meacham: It, it ultimately in the end we wanna change behavior. Yeah. If we haven't done that, it, because behavior we can measure. so yeah. And, and I love that you said that.

I say it often is nothing has really... AI hasn't changed the, what we do and what's important, but it may have changed how quickly, the timeline- Mm-hmm ... the workflow. So it might change the how, but n- it didn't change the importance of needing to have these metrics.

Attention Engagement Transfer

Margie Meacham: So, most of you know that I love to pull in the learning science, behind what we do, which, and as I know many of you, have that background.

In order for people to learn, they do have to pay attention. But paying attention is not the same as being engaged. It's just the start. So we make the slides pretty, we make them [00:11:00] interesting, we might introduce movement or, in a classroom we might have things that they do.

We gain attention so that they get engaged with the content, and the instructor, and their own brains. that's part of engagement. so don't assume just because people... Sometimes our metrics lean to the side of just measuring, okay, they logged in, they stayed logged in, they responded in the chat, they were paying attention.

That's not really engagement yet, is it? So how do we go beyond that? And it's the same for just because they attended this doesn't mean they remember it or could apply it. Mm-hmm. And that's the last one. Even if they could, say, pass a multiple choice quiz, that's not the same as using that information and using it well.

And that takes practice and it takes follow-up. Yeah ... and, I'm sure, [00:12:00] Chris, you know a thing or two about that, in your work at

Chris Van Wingerden: dominKnow. For sure. I mean, putting someone through some content and then immediately giving them a 10 question quiz, first of all, what's the quality of the quiz?

You know, is it assessing something that's truly, going to make a difference, or is it assessing, you know, are you asking questions just about trivial information that may not really actually be consequential? and it also follows the pattern of like cramming for an exam. you cram for an exam, you go in the next day- Mm

two weeks later you haven't retained any of that because- Right ... it's been very short-term memory focused ... So yeah, just because, folks complete something in the LMS, and, get a, achieve the passing score, does not indicate any way that, that they're actually gonna take something back and use it on the job or improve how they do things on the job- Absolutely

or change behavior.

Margie Meacham: And that's where we humans always come in, is we need to design those follow-up learning activities and a, [00:13:00] a learning curve that continues to happen after the initial exposure to the content. Mm-hmm. And then come back in and measure periodically so you should see an improvement.

Agentic Risks Guardrails

Margie Meacham: Now, the reason that we talk about governance, particularly with agents, but as Chris said, it really isn't just about agents.

They're just in the news right now. What makes agentic AI a, a heightened risk is because the agents Will take actions on your material, and you want them to do that. You buy these tools for a reason, and that's because they're very efficient and they help us become more productive. And there's a difference between efficiency, which is, "Okay, I got that course designed in an hour instead of five days."

But productive is, "I got the right things done in that class, and it's going to be effective, and I've, I've got [00:14:00] sound, grounded, research-based instruction that I've built." that takes paying attention to what the agent is doing, not just letting it run. And, so if you're doing something like a content engine where you get to pull in slides and then say, "Update these slides, combine these decks, structure it this way," and then you go get a cup of coffee and you come back, that's when your work starts, is now checking it, editing it, making the course your own from there.

How do you know otherwise what choices... So in order to take those actions and make those cool materials, the agent is actually making choices based on the knowledge base it's been given, and that is just never going to be as extensive as your own. Yeah. So that's where you have to ask it. Well,

Chris Van Wingerden: and just e- even that phrasing that you just used, the knowledge- Hmm

base that you've given it,

Margie Meacham: Mm-hmm ...

Chris Van Wingerden: one risk that we constantly hear [00:15:00] from the folks that we work with around any of our, Yeah ... AI agentic services is, "How do I know, that it's not just going out on the internet and pulling in things?" Yeah. "How do I make sure that it's exactly and only..."

and those guardrails are something, that we've been very focused on making sure that, yeah, it's not pulling in content that isn't- Right ... what you provided it. It's not going out to the broader world. It's also not taking your content and putting it out in the broader world as well.

It's not- Right ... you know, exposing your content to any other level of risk, et cetera, outside of your organization, too. So that's another key governance risk that is on a- on the top of minds of a lot of our client teams, for sure.

Margie Meacham: Yeah, absolutely, and even more so a- as we continue down this journey.

Not many- Mm-hmm ... people were talking about this two years ago. They were just so impressed with the capabilities. Now we're talking about it.

Designer Role Evolution

Margie Meacham: But I'm so glad you mentioned that because it, it shows how our role is changing, is now we have to think, step back and look at a bigger picture as [00:16:00] designers.

We've really become... It's much more of a leadership role where it's, I know I started as a sole contributor. I, somebody said, "Hey Margie, we need a class." I sat down, I got on my computer, I got my information, and I designed it, and then we talked about it and, I didn't have all these tools at first.

And orchestrating these tools, and there's actually a whole discipline around how you orchestrate and use those tools. plus there are humans involved. Another thing that, that happens, we might actually have it going to the right knowledge base, now we have to manage our own knowledge base. Maybe the content is out of date.

Maybe we have more than one course that was on the same subject and they conflict. so now an agent doesn't know that, it's just gonna pull what you give it. Mm-hmm. So we also have to be worried about, where is this information coming from, even within our own [00:17:00] knowledge base. We need a way to track that so we can keep it, fresh and current. So how does that look like?

You need to rethink, your role as really a team leader even if there's no humans on that team. there are things on that team that function, very much like humans, and, that's the danger, right? Is, our brains want to see fellow humans everywhere. We're wired that way, and it's a wonderful thing about humans.

That's how, we recognize each other, that's how we form friendships. That's why even though we're not in the same room, we're looking at pixels on a screen, but our brains know that I know Chris is a fellow human being. Now, I might be fooled if he's running an avatar. I, I could be fooled.

I could be tricked into, thinking I'm talking to a real human, and maybe I'm [00:18:00] not. that's still, most of us can still usually tell, but I just taught a class to senior citizens about how to be safe online, and I was shocked how many of them automatically assume any face is a human. That they weren't quite up to, speed on the fact that could be AI generated.

I recently learned of a scam going around, was told to me by someone who, uh, operates a grocery store. And, in the scam, the phone call comes into the office and, it's, a friend of the owner or the franchisee or whatever, explaining that they're with that person and it's somebody sort of known, right, first of all.

Chris Van Wingerden: Mm-hmm. So it's not a stranger. It's so-and-so. Like a celebrity or- So-and-so. Yeah. Yeah, so-and-so from, you know, the other store. Like, like- Oh ... it's, you know, it's, it's, it's a known- Wow ... but it's not the owner. Yeah. and they need some payments made quickly, blah, blah, blah. And then eventually in, in part of this scam, the actual owner then is, "Oh, he's, he's done his other call [00:19:00] right now.

And they've got the AI voice imitation of these folks down pat. Yeah, yeah. And, an unsuspecting, you know, employee in an office reacting to this sense of urgency, ends up handing over some money. Yeah, yeah. and, and it's, the way it was described to me was that the, the person just didn't know, and it's led to, in that industry, a lot of discussion about, having rails, guards- Yes

guards in place- Absolutely ... for any kind of a- Yeah ... if this, if you get this kind of a thing, call, call... It sounds simple. Call your boss on his- Yeah ... cellphone or her cellphone and confirm that they're really the person,

Margie Meacham: Yeah ...

Chris Van Wingerden: on the other side.

Margie Meacham: But- ... you might have to sound silly sometimes.

Like, "Yes, darn it, do... That's me. Do it."

Chris Van Wingerden: Yeah.

Margie Meacham: But-

Chris Van Wingerden: You know?

Chris Van Wingerden: but being- Yeah ... caught in that and being manipulated in that moment, Yes ... with the re- you know, the realism of that imitation, so

Margie Meacham: yeah. Yeah. And so very often as leaders in the organization and as people who are familiar with these risks, we might b- have to be in a role of coaching our teams as well to, to watch for [00:20:00] these things.

And even just, even if there's no nefarious actor, there still can be mistakes from what- Mm-hmm ... are very beautiful, end result looks like, we still need to check it for factual and current information as well as the effective, teaching of it. So, we have really evolved from the individual who does it all to an architect who's using a lot of tools but manually pulls it together still, to a leader who's doing some things in an automated way and other things manually.

So, our, our role is changing, and I feel it's gotten much more strategic. So to me, it's exciting that instructional designer really is something very different today. but with it comes responsibility to stay informed and to choose the right tools. and you're probably wondering, how do we know the right tools to, to buy?[00:21:00]

Well, we're not gonna tell you today because we do have, some, options for some more information, and, that's where, dominKnow has some, some free resources that we'll tell you about, but not, not until the end. So- ... let's, let's keep going.

Proving Value Metrics

Margie Meacham: So, I'm sure that all of you know these questions, but we boiled it down to just two.

we've all worked with, having a stakeholder who says, "Oh, okay, can you prove it? What are your metrics?" And in fact, our industry has come a long way in terms of finding ways to prove the effectiveness of a learning intervention, whether it's legacy training, or maybe it's a, a high-stakes learning game, or we've built an interactive website.

We have so many options these days to deliver compelling content, but there's a difference between making something look cool, and even [00:22:00] having people say, "I enjoyed that," versus- Did they actually come away with usable, tangible changes in behavior, and are they doing those things? So, I'm curious, Chris, what are your clients asking you?

And I didn't tell him I was gonna ask him this. I just made it up. So, what kind of evidence are they looking for to, justify all the great work they're doing?

Chris Van Wingerden: Well, it, you know, in our space we've long had the, the Kirkpatrick's model, and then the Phillips model that kind of elaborates on- Yeah

on that. and one of the challenges is that, you mentioned positive reaction a minute ago, the smile sheets, level one of, did people enjoy the, the experience, and we can get that info easy enough. Did people demonstrate something knowledge-wise? Typically, we, you know, look to an assessment.

Well, that's also something that, our tools can do. But when we start going up level three and level four, did it result in a change in behavior? and [00:23:00] level four, did it, result in, what was the business results? That starts moving us outside of the realm of

what an L&D team can even understand or know, et cetera. So those higher levels of determining value become an organization level, operation, right? Mm-hmm. you do something, you put something out there people learn. Well, people take it, sorry, I should say.

But, they're not doing their job in the LMS, right? Yeah. They're not. ... And in a lot of cases there are tools. Think of, the world of customer support for instance that measure, that have metrics built into just the, the way that, you know, folks work in those tools, et cetera.

But there's lots of other things that, you need to look at outside metrics. or metrics I should say outside of the L&D team. Sales results- Right ... and et cetera. And, and so these aren't, though they ... it's the L&D team trying to prove that it's demonstrating value, but it requires, the whole organization to be on board to help prove that.

Margie Meacham: I'm so glad you said that, and I'm glad you brought up, you know, our, our classic models. Kirkpatrick [00:24:00] and Phillips, which it's, either Chris or I are happy to point you to- Mm ... some information if you wanna get in touch with us, if you'd like to refresh your knowledge or that's new to you.

The challenge we're facing really is the same challenge we've always had, is how, how do you prove it? It's just that now we have new tools, and sometimes that messes with our workflow or our thinking process, and we don't realize the capabilities we may have available to us in the modern LMS.

or we don't know how to present it in a coherent way with our colleagues. So, a starting point is what evidence, what reporting are you doing now, and is it enough? Is it really proving your case? Is it tied to those business metrics, as Chris says.

Chris Van Wingerden: Mm-hmm.

Webinar And Playbook

Margie Meacham: So, where can you get more practical information on how to do this?

We promised we would tell you. we're gonna put the link in the chat, it is not the playbook. [00:25:00] It is a full-blown webinar that Chris and I are, doing tomorrow for Training Magazine Network.

And as a result of attending that, you will be able to download our playbook for L&D leaders. And so what that does is it, tells you what questions to ask, questions of yourself, questions of a vendor if you're considering buying a new LMS or upgrading one.

Of course, we're all interested and concerned. But what we do tomorrow, is, we'll give you some practical, things that you can do to determine, the state of your content, and how your LMS is performing, and whether or not you are in fact, generating change in behavior.

So please join us, and remember that even if you can't make it live, , there'll be a recording later for all you folks that didn't get to join live. If you register, then you'll have access [00:26:00] to all the handouts and all the, recordings, even if something happens and you can't join us live tomorrow.

Uh Oh Story Testing

Chris Van Wingerden: Well, you said we do have a little bit more time that we can spend, here. And I was thinking, we've covered a lot, but it's been a little bit, abstract in a sense.

have you had an experience yourself, in working with these tools where you've had an uh-oh moment as opposed to an aha moment?

Margie Meacham: Um, oh yeah, certainly. Now I have to think of one. That payback is, payback is fair play. So, one of the things that, happened in, and this happened in my early days, so non-technology related, discussion.

I really started my career in sales. I was actually gonna be a teacher and I needed a summer job. So I had a teaching degree, but it was summer and I needed to make some money, and I got hired as a sales assistant. I was supposed to, just generate leads and process paperwork, and they [00:27:00] very quickly started asking me to teach other salespeople, to actually train people because, I kept calling to confirm details and I would close additional business.

So, they said, "Oh, we have a salesperson here. Let's, let's use her." And it turned out that that's when I discovered that teaching and sales are two sides of the same coin, that if you really want people to be motivated to change behavior, you need to sell them on it. You need to get them to believe in it.

and then they offered me a full-time job and that, then I was a trainer. So I get, one of the first, assignments I got in my new full-time job as a training specialist was to design a class for new salespeople, and I decided they need to understand how they get paid. So I added a whole bunch of content about their compensation plan and how it worked and what they got paid for, and apparently I did it a little [00:28:00] too well.

And this is be careful what you wish for, because they did only those things that paid the most. Sure. And that would, it, something I probably should have predicted, right? I should have realized that was a logical consequence. And so while sales went up, customer satisfaction did not. and I had personally- generated that change in behavior.

So sometimes our, our best intentions lead us to, to teach the wrong things or too much of the right thing and miss other things. So that was a big lesson for me. luckily the company, recognized that at least I had, done something well. I had taught them about compensation so we revised it, and that's why you test.

We, we didn't run a pilot. The other thing I learned is the [00:29:00] importance- Mm-hmm ... of testing your materials, running a small contained pilot. there are a lot of things that experience, taught me, and one was certainly humility, that I had a little more to learn about the business, that side of things than I realized.

Chris Van Wingerden: I love how frank and open you are with sharing that. That's fabulous. You started with a teaching degree, but so many folks in our space don't even start with that background, right?

Yeah. Um, so many of us have that, similar experience where you're doing something and people say, "Oh, you know a lot about this. Can you run a training class?" Yeah ... we have that sideways transference into the L&D world without, initially having the background that we really do need to have, and then gradually we, oh, you do it for a couple of years and you realize just how much you don't know.

Margie Meacham: Yeah.

Chris Van Wingerden: Right? And so then you seek out the education, the substantive background, whether that's a degree program or going to conferences to build your knowledge. And, and in, in, in our space, not only do we try to deliver something called learning, but learning is what we do constantly ourselves, too.

Margie Meacham: [00:30:00] Yeah.

Chris Van Wingerden: that really is the one of the key motivators for me in what we get to do. But that, that idea that we come in from somewhere else to start doing something. I, I've often joked, like back in high school, the career day, there wasn't a table that said instructional design or become a trainer, right?

Margie Meacham: True.

Chris Van Wingerden: There were all kinds of other careers, including teaching potentially, you know? But no one, no one knows about this as a thing, as a, a possible career until you're actually into the working world, typically anyway.

Margie Meacham: Yeah. In fact, I was, very lucky that one of the first sources of information I found was Training Magazine, and I thought-

there's a whole magazine I can get that tells me, you know, what to do? And I became so fascinated with the learning science and went back and got my, master's in learning science and learning technologies, and that, really kind of focused, a lot of the stuff that I'm doing today.

And, it's a great resource, a self-education resource, and it's all, peer sourced. So [00:31:00] you know it's reviewed, it's quality, as opposed to just jumping online and searching and then wondering if you're getting, the best practice.

So you don't have to spend a lot of money, but you do have to put in your time, and you do have to be engaged if you wanna keep learning, and that's especially true right now. So we're happy to support that with what we're doing here today and the webinar tomorrow, which will focus, on some of the same topics, but, much more,

Chris Van Wingerden: Yeah

Margie Meacham: give you a methodology, that you can use. So- You know what? I don't wanna give away too much.

Chris Van Wingerden: we gotta tease them. We gotta leave them wanting more, right? Yeah. That's right. your description though of the sheer breadth of stuff for instance, in the Training Mag Network sites, it makes me think of them, or an organization like that, and there are several other organizations as well, but, there, there are humans in the loop in a sense.

In, in- Hmm ... in this space where we can do [00:32:00] a search, and the AI agent at the top of our search field can give us lots of info, but, things can be conflated, things can be inaccurate, et cetera. So it, it really places a greater emphasis to me on an organization like the Training Mag Network to, to be that guarantor of quality.

Building that trust and, making sure that the quality is there, the value, the accuracy, all that stuff. So it's a really important role for the human aspect of what we need to rely on still.

Human Judgment Boundaries

Margie Meacham: I'm just- I was just hopping over to the chat for a minute, Chris, to, um, see, if we had any questions. Oh, okay, great question. Chris, we'll both tackle this. Mm-hmm. Which learning decisions require human judgment, review, and accountability? So where is that division of responsibility between the humans and the AI tools?

Chris Van Wingerden: what popped to my mind was, for instance, ensuring legal requirements [00:33:00] are met. Ah. Right? Right. Mm-hmm. I don't know that an organization should be entirely comfortable not having that vetted, where there are things like legal requirements and risk, you still need the humans to make sure that, that what's being, you know, put out there meets those requirements.

Margie Meacham: I agree. For example. You, you can certainly ask an AI tool to point out possible areas to bring- Mm-hmm ... to your legal counsel. but I wouldn't wanna rely on that. remember they are our assistants, these, agents and tools. that's what they are, they're tools. So, we're the ones who have to decide how to use it.

I would also say at the very front end, the decisions you make as a designer are still so important. just even doing a needs analysis and identifying what exactly, So if I have a bunch of salespeople who are only selling high-end products, just to go back to my own [00:34:00] example, why is that happening?

you have to investigate that. You can use some tools, some surveys, some analysis of, past sales patterns. You can talk to customers. But ultimately you have to drill down and recognize what behaviors are bringing us the business results that we wanna modify, and then how are we gonna change those behaviors?

How ingrained are they? Is there personal bias coming in? Have they just not been trained, or do we have a lot of maybe tribal knowledge that's influencing it? So your experience and your ability to look at the organization, really determines the instructions you give those tools, and that determines the end result.

So a lot of times, if I end up with ineffective training, the analysis of what happened here may go all the way back to very early decisions [00:35:00] made with the stakeholders and the designer and, that we may have missed something Very early on in understanding our audience or our business problem, maybe we were trying to solve the wrong problem, for example.

those are things that while certainly a tool might support you with data and analysis, but it's not going to connect the dots. That's a human process. That's my take, is that it really, from the very beginning, I put human judgment at every step. Absolutely every step has to involve a human. Is this still making sense? Is this still effective?

Chris Van Wingerden: Yeah, for sure.

Wrap Up And Goodbye

Chris Van Wingerden: Folks, as we've mentioned, we've got this, webinar tomorrow afternoon.

We'd love to see all of you there, picking up some of the threads of what we've discussed here and, and taking things a little bit further as well. don't forget, gang, Instructional Designers in Offices Drinking Coffee is brought to you by the team here at dominKnow, makers of dominKnow ONE, helping L&D teams to develop scale and deliver learning that [00:36:00] maximizes employee value.

would love to, have you check us out if, if you're, looking for assistance in that world as well. Margie, as always, thanks so much for joining us. Thank you. What another great conversation. so much fun, and so much, important and helpful stuff to put on people's radar.

Really appreciate you joining us here today.

Margie Meacham: I'm looking forward to seeing everybody in future sessions, just as one of the members over there in the chat.

Chris Van Wingerden: Awesome. Folks, it's time for us to dance on out of here.

Have a great rest of your week, and we'll join you again on the next episode, the next session of Instructional Designers in Offices Drinking Coffee, #IDIODC. Looking forward to that. Catch you all soon. Thanks so much. Have a great rest of your day, folks.

[00:37:00] [00:38:00]