Vibe SCORMing: Govern AI Learning Before It Governs You

IDIODC #275 feat. Adam Reynolds - Sep 09 2026

[00:00:00]

Welcome and Guest Intro

Chris Van Wingerden: Hey, everybody. welcome to instructional designers in offices drinking coffee, #IDIODC. Oh boy, gang, we got a little bit of inside baseball opportunity here today. joining us as our special guest for conversation is Adam Reynolds, who is actually a member of our dominKnow team here.

He's our lead learning systems architect. and what we wanted to do is we wanted to have a conversation, that, that came out of a, a separate conversation that we were just having internally not too long ago when we said, "Hey, let- let's talk about that. let's bring that forward to folks and think about it in the big picture, et cetera."

Adam, this is your first time joining us, so, introduce yourself to the gang here.

Adam Reynolds: Hi, gang. I'm Adam Reynolds. I've been at dominKnow now since 2007, so 19 years. actually started here in support, doing product support, answering phones, doing emails and email support for various sort of training tasks of helping customers really get the most value out of our LCMS, And then, yeah, I mean, just sort of [00:01:00] came through and-

Chris Van Wingerden: Mm-hmm

Adam Reynolds: moving away into development and, you know, slowly but surely worked my way up and I'm now, an architect here.

DominKnow Origins and Flash Era

Chris Van Wingerden: And it's kinda cool, 'cause we could really go off on tangents today because we you- Adam and I spend several hours a week on calls together in different meetings and plannings and triages and all kinds of things because of the nature of what we both do here at dominKnow.

but we've both got long time track record here w- with dominKnow as well, and we've seen lots of things change over time in the industry. I'm thinking about like, oh, the original version of dominKnow ONE, which wasn't even called, you know, dominKnow ONE, but- No ... the, the LCMS. people were uploading Flash files as pages.

Adam Reynolds: It's... Well, even, even most of the product back then was all built in Flash. I mean, Flash was the way, right? Exactly. I mean, we're- Right ... an instructional design company building learning content then. I mean, our LCMS was an evolution of our own problems. Yeah, yeah ... was actively developing courseware, and we needed some way to manage the courseware you were building.

Chris Van Wingerden: the roots of the company were actually based on, having a catalog of off-the-shelf courseware [00:02:00] for, surface map technologies for soldering circuit boards and those sorts of things in a manufacturing environment, as well as a catalog of content from, wireless technologies, whether those were cell phones or other things.

the catalog was actually on CD-ROM. And that CD-ROM was sent to somebody, the shiny plastic disc, and, it was then, set up on a network internally, and it had tracking functionality within it, et cetera, to make sure that people were, completing the content, passing the tests.

And then this internet thing came along and it was like, "Well, how do we do this without shipping those shiny CDs around all over the place?" so the original root version of what's become dominKnow ONE was based around taking that content, Developing it as Flash files, but then having a tool that then gave structure and tracking, et cetera, in the online context.

And we actually had a student-facing side of things, at that point in time originally. and then realized that what we also created was a learning [00:03:00] content management system, and kind of took our, our destination down that pathway, through various iterations.

Adam Reynolds: Yeah.

Chris Van Wingerden:

Adam Reynolds: It's one of the in- interesting things too with the LCMS side of it because

Right now we have a very powerful authoring capability. but that was not where we started, right? We started deep in content management, learning content management, understanding the structure, how do you put together content in a way, you know, allowing instructional designers to build instructionally sound content with various labeling objectives and terminal objectives you need to do that.

Mm-hmm.

Chris Van Wingerden: Um,

Adam Reynolds: authoring was such a side thought back then. Content creation was such a side thought back then. It was mostly programmers, instructional designers writing Flash.

Chris Van Wingerden: Yep. We had a team of folks working on every page was a separate Flash file with all of its stuff in it. David's chiming in in the chat, "I sent my e-learning on floppies back in the day."

Right on.

Adam Reynolds: There you go. Right

Chris Van Wingerden: on.

Adam Reynolds: I used my floppy forever.

Chris Van Wingerden: Yeah. David- Although I do have a couple of USB back there ... like, three and, three-and-a-half inch floppies or five-and-a-quarters? we do really actually wanna know that. That's the geeky side of our brain on that front.

Mobile Shift and SCORM Staying Power

Chris Van Wingerden: [00:04:00] So, I didn't mean to take us down a sort of a history lesson so much as to say, like, there's a clear change that hit the e-learning industry was when, mobile devices came along and suddenly Flash weren't it anymore.

And- Yep ... so much had to change and adapt to that, technology change at the time. Working with, um, HTML5, even though it was pretty limited in its early stages and its early days, to be able to even just, make sure that somebody could actually run something. And think about how, limited Flash usage can even be these days, that most devices don't, don't run it anymore.

We're not a s- an industry or a space that really ever is able to stand still. Well, it's interesting though because there are some things that seem to have stood still, such as SCORM, right?

Adam Reynolds: Yeah. You know what? When the wheel turns, why replace it, right? There's better- Yeah

technologies now, xAPI and rich analytics and data analytics and content analysis you could be doing. In fact, the learning content analysis market is massive and growing. I believe it's got, like, a 15%, CAGR year over year. by 2030 it's gonna be a massive ballooning-

Chris Van Wingerden: Hmm ...

Adam Reynolds: area. And [00:05:00] it is set to outpace learning content management from general market trends and what you're looking at.

It's interesting. but SCORM still delivers. You know what I mean? Like, when you have compliance, you need to know somebody actually passed this thing and they got through those questions. If those test questions in that SCORM package are solid and they demonstrate, you know, that you read this content and that you remember it.

And okay, they can't really demonstrate knowledge synthesis, but as best as we can get digitally. You can lean back onto that. One of the nice things about the new tech too is, like, you can combine it. You can stay on SCORM, and then just side load analytics to an LRS or to somewhere else to be able to report on that side of it too.

You kind of have your cake and eat it too It's interesting though, Chris, you bring up, sort of the history of SCORM and where it's at. But one of the things that with HTML5 that was an interesting, reflection from my perspective as a developer, HTML5 was something that working on mobile devices with media queries and bootstrap and grid CSS and all that stuff back then, like that was happening very rapidly for marketing.

When people wanna have websites that work on devices, they wanna be able to get you marketing material that works on devices. [00:06:00] The uptake in e-learning was a lot slower. There was companies out there being like, "Hey, we're HTML5," that really still produced Flash. We were right on the edge of that right away into HTML5.

We, really killed Flash in our system early. but it takes... There's always sort of that flow, right? So software development, building, like we have this new technology coming out, we're gonna take advantage of it digitally, and then it sort of rolls its way into the learning and development space, and it's always sort of a couple years behind in when those new developer techs come out.

AI Arrives in Development

Adam Reynolds: we're seeing some of the same transition now, right? Even as- Right on ... as fundamentally different as HTML5 and responsive was for mobile with, AI, what it's bringing to the table with this technology too, right?

Chris Van Wingerden: And I mean, I don't think there's an episode here on IDIODC where if it isn't the main topic, it's the subtopic of- Yeah

every conversation that we've had for many, many, many, many months, or at least it's, it's ended up forming at least some part of the conversation because it is so pervasive, currently in our space and a lot of, astounding changes that are happening because of it.

But like you say, the programming world had AI [00:07:00] as kind of a, a leading edge place to explore, to kick around, to understand, to work with before it started sort of trickling into, the e-learning world. from your perspective, let's talk a bit about that, the EI and, AI in the programming world.

Adam Reynolds: Yeah.

Chris Van Wingerden: Where it started and, and the things that, the programming world encountered about it.

Adam Reynolds: No, I have my own sort of take on that. I was playing with AI technologies, end of 2024, early 202-- I mean, way back when I had a... I was using PyTorch and TensorFlow. I don't know if anybody here knows that kind of technology.

Using PyTorch and TensorFlow, I taught my computer how to play Mario Kart. I trained it on, be able to run the Mario Royal Raceway, which is basically an oval track. But that took... It was funny because by the time I was done, I'd put hundreds of hours into this thing, and I'd realized only that I'd have to actually spend, like another 2,000 hours of playing all the other Mario Kart tracks just to get the training data sets I needed for it to be successful.

That was a fun little lesson. But no, even all these years later, 2024 and '25, I was looking at it from the perspective of software development, right? What can it do now that we've gotten to agentic [00:08:00] AI? What can it do with, agentic AI to build and run good software? And I actually was very disappointed.

I was very disappointed through '24 and into '25 because the promise is there that it can reason and understand what you're doing. but the lack of the ability to provide context to those AI systems, like we're talking, you know, Sonnet 3, and you know, way before, even before. You can't even get access to these models on the frontier models anymore because they're just, end of lifed.

But they didn't have the reasoning capabilities for coding. They were great for chatbots that you'd go online to talk to, and it could sound like it was doing, pretty well. But the coding stuff didn't really come, until into 2025. There's a couple of sort of inventions that came out, innovations like Claude Code, for example.

there's also Codex and Copilot, that are coming into like VS Code IDEs. And these are all, code development softwares that you use to build, build your code. And we have these integrated coding assistants now. they come in and they're much more useful. The integrated coding assistants are great.

They can help you autocomplete and finish things. [00:09:00] And then, with Claude and the Anthropic team, one of the things that they're sort of going is that it's not so much you wanna have the coding copilot with you. They have a sort of a different model, looking at sort of do it for you in a way.

It's like you have your own, junior development team that isn't all that great, but if you supply enough information to it, the other side of it, at these days in '26, we're getting, much more qual- higher quality outputs. You can look at the system, it understands how to navigate your code base and how to produce things.

Vibe Coding and Context Windows

Adam Reynolds: One of the challenges though, of course, in between that we had, and I think it's actually part of the title of, this IDIODC, right? But the, there's a whole world of vibe coding. have you heard of vibe coding, Chris, or? Oh, for sure.

Chris Van Wingerden: Mostly through memes, but,

Adam Reynolds: Right. The memes are legitimate though.

Chris Van Wingerden: And a lot of the memes were not positive viewpoints of it- Yeah ... you know? Uh-

Adam Reynolds: It, it... That's legitimate criticism, you know? Like, that's what we ran into in development a couple of years ago. People were saying, "Okay, the soft development space is dead. We don't need engineers anymore. We don't need architects.

We don't need people that can do this work because the AI's here and it can write all the code for you." That's [00:10:00] true, but it doesn't mean that the code meets the product standards that you need. It doesn't mean that you're solving the right problems of the market. and the code it writes, you know, let me tell you, I live and breathe AI development pipelines these days.

context, augmented retrieval, and, all kinds of... Cognitive augment generation actually for code, but all kinds of different RAG and CAG patterns that we're using, to develop code, and I spend most of my time in code review, human in the loop, because I have yet to see any solution that says, "I'm gonna produce the code, then I'm gonna review it myself, and then I'm gonna go and put it out there, and I'm gonna QA it myself, and then I'm gonna..."

It doesn't work. You gotta have a person at every step of the way up front. you gotta have that brainstorming, and you gotta have that discovery. The ideation phase, so important. You know, for me, I have my AI ask me questions and interview me. Basically, I give it a topic and it starts saying, "Okay, what do you mean by this?"

And it goes out and does research, and it continually engages with me to pull more and more domain knowledge out of my head, and it- it's context, and it's context window. well, let me just describe what that means, because it's an important part of vibe coding and important part of this whole [00:11:00] conversation.

Context is sort of an empty space. An AI is trained, an LLM is basically it's a predictive algorithm. So when you type in something, it's looking through what the most likely word that should come next with what it's about to say is. It's actually- Mm-hmm

kinda trained on how we talk and communicate. But, you know, the argument is, so are we by our parents listening to them talk and by reading our language, like we learn how to communicate the same way. so it doesn't really know how to code. What it knows is it's been trained on code, and it's been trained on things, and it knows what should come after this.

So an if statement may have an else if there's another connection, or, a feature should have a unit test if you need to have a test written. All of that stuff is all this training data that lives inside the LLM. But then what comes on top of this when you're actually working with it is the context.

So you've got these companies selling context windows. One million token context windows, is a profits Opus 5 right now, for example. And, that's a lot of context that you could store together. You can say, I wanna build a website, and I want that website to [00:12:00] be branded in red and orange.

I mean, terrible branding, okay? But, you know, you put that together, I wanna have this font, and I wanna have this look and feel, and I want my website to be about this kind of company. Or maybe it's gonna be a sales website, and you can select things, you can buy them. And it'll go out and it'll build the front end, it'll build the back end, it'll build you a container on your local that stores that, whoever clicked the buy button.

It, it'll do all of that for you. but if you don't have a good amount of context transferred at the beginning up front in that discovery phase, it'll kinda run away with where you want it to go. It won't know where you want it to start or stop, and it won't have all that information in between. And what the software development community has discovered with AI is that that context is actually the most valuable part.

The, output isn't valuable at all. It is in that the output is what we sell or what gets used in the products. But, Having the output isn't as valuable as how, having how you got there, which is also case in point in the entire industry why it doesn't replace software developers, because those, we have the context.

We know how the system was built and why for one end and what [00:13:00] purpose. what doors we're leaving open as we're designing software to be able to do things later. and I find, the funny thing is, there's a lot of parallels with, like learning and development and that sort of way of working.

I talk about ideation and discovery. I mean, I'll give you the, you know, the five bullet points. You come up with, you discover, you plan, you implement, you accept or you review what you did, and then you ship it. That's basically software development in a nutshell, and that is exactly what e-learning people do and breathe and like live and breathe every day.

That's the job. It's a little bit harder actually. L&D I think as far as if I were to reflect on that, meaning, I mean 19 years between support and development, I would think that L&D has a harder job because the human in the loop on the other side is behavioral, right? You not only have to use and design e-learning and it- instructionally sound things, like you have this, say, a rocket propulsion PDF, and you've gotta convert this thing into courseware to teach people this information.

So you actually have to know what knowledge comes first to build off of what, in what [00:14:00] order you're providing people good checks for knowledge, checks and opportunities for different kinds of learners along the way to be able to, really absorb this thing. You need to be aware of the psychology and behaviors of others.

That's something in software we don't care about. I mean, we do in UX and XD, we care about people using the software and how they interact with it, but not whether or not they really learned, or have to walk away with knowledge synthesis that's at a regulated- Yeah ... compliant level.

Chris Van Wingerden: You know, from a programming perspective, even outside of an e-learning tool, what you're trying to do is just have the program that you're creating carry out a function.

Adam Reynolds: Yeah.

Chris Van Wingerden: it doesn't have that extra additional layer of, e-learning has to have the buttons work, but it also has to have, an end result of behavior change.

Adam Reynolds: From, from a human.

Chris Van Wingerden: Yeah.

Adam Reynolds: Yeah. For a human. It's not just, it's not just a system you can rely on. I can't write a unit test to say you learned something, right?

I wish I could, but you know, you can't.

Chris Van Wingerden: Yeah.

Risks of Shipping Prototypes

Chris Van Wingerden: So thinking about then that experience in the programming world and how we're now starting to see, some of the parallels in the e-learning space with AI, what were some of the risks and I guess even the [00:15:00] costs that the programming world has been discovering-

Adam Reynolds: Right

Chris Van Wingerden: so

Adam Reynolds: Well, this is where all the memes come from, from vibe coding, right? So all the memes and that whole marketing deal and that whole everybody, I mean, that is literally the other side of that where you think, "Hey, I'm gonna rapidly build all these prototypes, and I can take these prototypes to market.

My marketing department is salivating over these things. They're amazing. I can put them out there. People can use them." And then it's, "Oh, and okay, now we need permissions," or, "Now we need some kind of a back end," or, "Now we need to actually productize or actually make like something you're selling or useful out of it.

It's not just a one-off thing you're gonna do." That's the whole meme of vibe coding, people sitting down that don't understand the complexity of delivering software that can... And maybe they do understand that complexity, maybe they don't, but the whole vibe coding thing is I'm just going to sort of feel my way through it.

That's the idea. I'm just gonna keep talking at this thing until the output becomes something that I was sort of aiming at, what I wanna have. and it works. It really does work. But then the costs and risks that you have in that world of vibe coding is [00:16:00] that on the other side you've got this prototype that nobody else knows how to use.

The person that generated it, they didn't review all that code. They don't know the software development practices to actually read the code and understand what this thing does. They understand it from the outcome side of it. It's like the best, product manager is probably going to be one of the best, vibe coders.

But on the other side of that, how does that get maintained? How does it get fixed? Do you just keep going back to that product manager over and over again? And then what happens when every single time you have to make a bug fix, you're coming back and it's like, okay, now I have to teach it everything about this outcome again, this prototype that I built, just so it can find where the bugs are.

That was real, and that is a real problem, and software development has really worked at, fixing a lot of that. You might see a lot of things in marketing now about, like I was talking about context augmented generation or retrieval augmented generation or, any of these sort of graphs, knowledge graphs or these things.

They talk about AI pipelines and how you need to be running loops. All of these things are the understanding and having respect for those five phases I was talking about, that ideation, [00:17:00] planning, implementation, review and acceptance, and ship. That is... When you start to look at things in that sort of model, That's when things can start to make sense and what you're getting shipped is really good because you have to review and approve and understand and accept the output, of this thing.

But the thing with vibe coding was people were shipping prototypes by telling Claude Code, "Hey, put this up on Amazon for

Chris Van Wingerden: me." Hmm.

Adam Reynolds: And is it scalable? Is it secure? Is your data gonna be safe? What kind of database are you using? How much is it gonna cost you? people run into all sorts of problems like that going to production with these things.

Vibe SCORMing and Lost Context

Adam Reynolds: Just one of the things that I've seen and one of those costs with vibe coding that comes up, and it's the same sort of cost that we have with AI development of SCORM packages in an L&D, right? when you see people online talking about, "I took a subject matter expertise," like, like whatever it is, a PDF or PowerPoint, Word document, a zip file of assets and stuff, and they threw this at Claude Code and it said, you know, "Make me a course."

And there's a lot of work. You can't just say, "Make me a course." You can say, "Put it in SCORM." SCORM is a known standard. It'll understand [00:18:00] how to connect that stuff up together. But, anybody who's done that, who's tried to do that, knows that not, that's not particularly successful. You have to spend a lot of time in review and turns.

When you're working with an AI system, a turn is one chat back. So every time you talk to it, it's a turn, it comes back to you, and then... So you're spending, you know, hundreds of turns with these things, and throwing tokens at understanding how to layer this context or this content in the right way, how to, organize it in the right way, how the assets of visuals, are balanced against the text, the font, the...

E- everything that gets done from branding all the way through to actual instructional s- sound instructional design, you've gotta teach it. And then on the other side of that, just like with vibe coding, if you're doing something like that, when you're done, you have the SCORM package. You put that in the LMS and it works.

But then all that context, everything you taught it, how you got there, it's gone. You lose it. It's not saved with that a- artifact. It doesn't come along unless you're doing something really specific in a pipeline that says, "I wanna learn from this and store everything that I'm doing." but that's just not what I'm hearing the market say.

A [00:19:00] lot of people are really happy about how quickly they can produce stuff. Mm-hmm. And we know how that works. There was a whole world of, of rapid content authoring. Do you remember, like Rapid Intake, for example? or I think Articulate Studio back then before Storyline was released. Those tools were all about producing learning very quickly, so you can get your training development teams up to speed, uh, as

Chris Van Wingerden: They were about creating content quickly.

Adam Reynolds: Yeah. Yeah. I mean, there's

Chris Van Wingerden: a word out there to use. Yeah. Which is always the wrinkle, right? with, with everything that anybody can put out. You can put out lots of content, but, is it helpful?

Is it, something that's actually going to assist with behavior change? David back in the chat was, was chiming in, with some reflection on this importance of context. "I work with SMEs, and they rejected my content because it didn't fit in real life." it... This was in the context of generative content.

So, those humans, in this case plural in the loop are what's still required to make sure that the output is gonna be sufficient to the task and et cetera.

Adam Reynolds: Yeah.

Chris Van Wingerden: Um, and

Adam Reynolds: you know, I think- And the great thing that he had SMEs to review that content, right? I mean, a lot of teams [00:20:00] don't.

Mm-hmm. They don't have review, approval, or versioning of what's being put out. They don't understand even the changes that it's making. they're just taking it in the LMS and saying, "All right. Yeah, it looks good to me

Chris Van Wingerden: Yeah.

Survey Data on AI Governance

Chris Van Wingerden: We've been doing, earlier this spring here at dominKnow, we actually released, our state-of-the-art of, of learning content management report.

Some really fascinating, data points that we found. y- we did a survey in the e-learning space in organizations across a number of different, content and, and topic and, and, business types, et cetera.

interesting, one of the questions was, it was looking at the anticipated impact of AI on learning content creation and management in the next two to three years. 7% of folks said, saw it as a transformational impact, 42% saw it as a significant, as going to have a significant impact, 37%, had a moderate impact.

So, almost half of, of the respondents to that question clearly saw, and I don't think that's any surprise to anybody. In fact, it might even be that only half saw it as having a significant or [00:21:00] transformational impact in our space. It's almost surprising that, well, the 2% of the people responded and said it was gonna have no impact, and I'm kind of thinking, "Hmm".

Some other points that we found,

um, the question was about what level of confidence you have that AI-generated or assisted learning content is governed by organizational policies. only 23% people were fully confident or completely confident. 35% of the people responded that they felt fairly confident, which is wishy-washy in a sense too, right?

Yeah ... and then when we found, thinking about the AI processes that we're adding in, just the sheer quantity of, review, security review, et cetera, that, that so many of our clients need to work through 'cause they have to meet governance standards, whether it's protection of their content, being su- certain that something's not being sent out into the world to be added to something that an LLM will then use in its own [00:22:00] knowledge.

Making sure that these things are closed and, and having confidence that their content is being treated securely. and the obverse, people worrying that the processes and tools are actually just gonna go randomly out to the world and grab other content, that's not from the organization itself, which could introduce errors and, and hallucinations and all kinds of things.

And the data, security models around that, wrapping around those things too. we worked through some pretty hefty paperwork with a lot of our client teams to just, to process that. And it also seems, this is just my qualitative reflection, that this is the bigger organizations, that we work with that are, that have probably been working on these kinds of policies and processes for several years watching the horizon.

but smaller teams may not, be in a position or a capacity to even think about some of these things because they are a smaller organization, a smaller team of people. it's, So, I mean, William Gibson said, "The future is here. It's, just not distributed evenly or equally."

some of these concerns and the policies, the procedures, [00:23:00] those safeguards, et cetera, are also not distributed evenly across organizations. And it could be a real problem for some of those organizations, the smaller ones especially that don't have things in place.

Adam Reynolds: Yeah, that's right. You said about half, give or take, half of learning teams right now believe that AI will have a significant impact, and then you'd said something, I believe you said like a third or 35%?

Chris Van Wingerden: Yeah.

Adam Reynolds: So people are understanding AI is here. Like this tells us in data AI is here, and it's gonna transform our industry or at least have a major or a moderate impact to our industry, in learning and development. But then you're saying on the other side that only a quarter of teams have confidence that those, changes are gonna be reviewed and understood?

Chris Van Wingerden: Yeah. their AI content is governed by policy. So not the changes that are coming to our industry, but the content that they're currently using AI for. The 23% people in our survey, expressed complete confidence that the AI content is governed successfully [00:24:00] by policy.

Adam Reynolds: and 35% were fairly confident. So, That's, that sounds like a lot of risk to me when you're looking at if people are even recognizing that, and that's great.

one of the things with the software development space with vibe coding, we didn't really see it coming, right? We did like a lot of developers, myself, when I'm looking at... I would tell you the 2020, early 2025, you know, Claude Code and when that stuff, that was actually later in 2025 came out.

But those sort of toolings weren't great. But there's a lot of people swearing by, you know, "Hey, I could just vibe code my way through anything. I can have the billion-dollar startup, one man billion-dollar startup." That was a thing that was happening. We were really blind to it. and we sort of all are waiting in I remember the, the circles of developers just being like, "Yeah, okay, we'll see how this actually fleshes out because this doesn't make any sense."

Seems like e-learning is a little bit smarter about it, because L&D teams are know, like at least they know that it's not being regulator covered. They don't really know where it's gonna have that impact. So that's good. It's good to have that foresight.

Chris Van Wingerden: Mm.

Building Better AI Pipelines

Adam Reynolds: It's one of the things we can kind of take advantage of, right?

When you look at the vibe coding and you look at the mistakes that developers have made with this technology, [00:25:00] you know that those same mistakes are gonna happen here in the L&D space, and get out ahead of it. You can even look at what developers are doing about it. I talked about some of the, the things like, you know, lost context, maybe finding a way to not lose that context.

I talked about loops and I talked about, successful pipelines that have to do with following, interview statuses, discovery into planning, and that matters. Discovery and planning are not the same thing. You gotta have a phase where it's just open, where you're reaching about how you can structure this stuff and what kind of content and how you're gonna teach people, what interactions you wanna have.

what kind of branding you need to have, what's look and feels, work on devices only. Is it gonna be like a presentation? there's so many different ways to output learning. It's like you gotta have all that upfront and then you really need to go through those phases. It makes a lot of sense.

I think that if L&D sort of followed that same path and looked at the mistakes of vibe coding, we could get away from the world of vibe SCORMing. we won't have the same sort of impact. Because the only thing it's gonna do is, We were talking about it earlier, but talking about the older tools when it was about producing rapid [00:26:00] courseware.

How fast can you get a course built from that PowerPoint? that gap from how fast you can go, now we're at that point where if you have an AI, like say you're using Cloud code and you have a skill that is, you know, make SCORM, and you're just gonna have your make SCORM skill that you've already trained it, you've talked to it, you know exactly what, it knows exactly what it needs to do.

and that's one of the ways, guys, how you can give context to an AI is through skills if you wanna have repeated context or through customized agents you can build yourself. That's one way you can kind of get around that stuff. But, when you just sort of say, "Okay, I've got another PowerPoint. Here, throw it at the skill.

I've got another PowerPoint, throw it at the skill," you're gonna produce content very, very quickly. And the, sort of the more context you load yourself with... So what ends up happening with people is, and I know this from my own personal experience, you can have many AI pipelines running. You can have it going and saying, "I'm producing three different SCORM files right now."

And your own mental load is involved, and it gets strained because it asks questions, it does things wrong. You've gotta pay attention to what it's reasoning and how it's going [00:27:00] about building this content. so the cognitive load on the human is still there, so you can get stretched really thin.

And then you get to these points, and this is what happens in code review with developers, is they've done all of this agentic development. They've worked with the AI enough that they... The AI didn't hallucinate anything. It's almost the human is hallucinating. They're like, "It must have built what I wanted because I said it so many times and because I, it knew what I meant," and you have all these assumptions, but it doesn't really.

So you really had to do all of that due diligence on the code review side afterwards. and it's exactly the same sort of problem in learning. You really have to have systems and management systems in place that allow you to review and approve changes and review and approve new content coming in.

Otherwise you're just gonna hit that brick wall faster if I produce 55 SCORM courses and now I don't even remember what any of them were about

Chris Van Wingerden: Mm-hmm. Some other, stats and information that our survey gathered, earlier in the year. fewer than half, 47% of, and this ties right into what you're saying, 47, only 47% of the [00:28:00] organizations that we were surveying had a mandatory human review and approval, of AI-generated content before publishing.

Less than half of, of the teams- that were responding had a mandatory human in the loop. Think about, I guess, the things that

Adam Reynolds: I wouldn't want that kind of liability. Yeah, no kidding. Especially in the L&D spe- space, the regulations and stuff that are there. Anybody listening to this, if you are- Mm

you're thinking, "Hey, I'm part of that half that really doesn't know what my AI is producing," please get a review. Mm. follow David's example, I believe you had said earlier. he's working with his SMEs, and they're actually rejecting his content. Thank you for that. Exactly the right way to work and the right way to think.

It's so important.

Chris Van Wingerden: If only 39% of the respondents said that they had visibility into where the AI generated content is actually used. that seems like a lot of risk too, for an organization.

Adam Reynolds: actually our frontier models now, the companies like Anthropic and OpenAI, they've been, engaging what's been called like AI watermarking.

So AI generated code and AI generated [00:29:00] things, images, it's AI generated anything, is getting watermarked digitally with certain signatures and certain things that they can use to reverse engineer to say, "Yeah, you actually produced this," because of liability. And it's not that they're having legal concerns now, but you know, that's the, the fear for them in the future is like suddenly there's a piece of information out there, something illegal that says something and they don't wanna be on the hook as a company being the one that gave the model that produced that output.

so they're watermarking everything. It's the same sort of problem, right? If you don't know what's AI generated, how are you ever gonna take accountability for it or take responsibility and own it? 'Cause ultimately, you know, from my point of view, if I use AI, much like if I use a keyboard to write code, if I use an AI to write code, that's still my code.

I'm using the tooling that produced it. I'm responsible. I own this. I need to make sure it's reviewed properly. I need to make sure it's high quality.

Chris Van Wingerden: Yeah. I mean, heaven forbid you've got safety training that's going out that does not have, accurate, helpful information. Yeah, exactly. You know? Yeah.

Adam Reynolds: Clicking the button doesn't work, But exactly, in L&D, like that [00:30:00] training matters a lot.

Chris Van Wingerden: Lives and health and safety, could all be at risk, very quickly and very easily. a couple of other stats from the same set of questions. only 38% of the respondents said they had a managed control, over what content can be used as AI source material.

So that's, sounds like the Wild West potentially right there. only 34% said that they had the ability to provide feedback to the AI if information isn't correct. And that goes back to that content review, content revision, s- those, those cycles. So.

Adam Reynolds: Yeah. At that lost context, right?

Like even if you have the ability to review it and you come back- I mean, if you don't have that same session open or that session got compressed too many times, you'll never be able to go back and change that content around. And that's the most expensive part too, is once the AI's already built a thing and it's built up, say, you know, a 4 to 1,000, token context, now you're adding turns, that are saying like, "Okay, make those tap sets green."

And these little sort of things that might be brand [00:31:00] related or, you know, I don't like the video interaction, or the audio's not coming through and it's not timed properly. These little things that you sort of work with, and all that back and forth drains more context. And then later on, you've actually delivered this thing, you feel proud of it, and the SMEs are looking at it saying, "Well, you know, you missed our formula entirely."

Now you're going back to an 800,000 token window context, and you're trying to tell it to fix the math, and it doesn't really understand what you're talking about anymore. So it has to go back and read the thing and drain more tokens. It gets expensive, and hard to manage, which is why, you know, like I was saying, there's, there are strategies around that.

Having actual, Well, I mean, content management strategies being one thing, but then even from the development perspective, not losing that context, finding a way to mine your sessions for, you know, knowledge and for information, finding ways to, store that and save that for later is super important.

Mm-hmm. It's something that's at the core of a lot of, Domin- like right now as we're architecting Domino's AI, solutions and products and that side of things, it's, always at the core of our mind as an LCMS, right? Because we have a lot of respect for content management, that review process, [00:32:00] understanding workflows going through, and how important it is not to lose it.

Chris Van Wingerden: Yeah. Of course.

Token Costs and Puppy Analogy

Chris Van Wingerden: You've used the word tokens several times. Mm-hmm ... but you finally, in that last set of phrasings there, you actually referred to the expense. So, tokens is almost a way of distracting from the fact that there's a dollar value behind each of these turns.

Adam Reynolds: Every amount of time you do a turn or you go back and forth, it has to reread or understand your context. It needs more and more and more. and they do. It's, it adds up fa- it can't, it can add up fast. But then the thing is, the argument is the amount of... If you have a proper pipeline and a proper setup that allows for that saved context to be able to use that context again, potentially things like a RAG situation where the e-learning you're producing is actually going out and looking at, okay, these are the other 12 courses I have managed somehow, and I take all 12 courses these, and it's all loaded into like a, a RAG database.

Then you've got the common, tone and style interactions, the way that learning is constructed. You can repeat that over and over again because you're [00:33:00] using, it, it's called RAG, retrieval augmented generation. You could use that, to basically produce the same kind of stuff again and again.

So as long as you're kind of thinking about how you don't lose The magic of how that was built from you as the person, the person actually working with the AI, you've got the magic, you've got the, the expertise, you've got the domain knowledge. The problem is you don't have the time. So you're trying to get this AI situation to deal with the time problem for you with sacrificing, you know, your own, ability to do it really well.

You don't have to sacrifice that. You have to find a meet in the middle, but you have to build proper process and pipelines around it.

Chris Van Wingerden: Mm-hmm.

Adam Reynolds: Tokens do have a large cost too. I mean, like when you do get into those millions of tokens, I could tell you, in having subscription accounts, I could tell you the number of times, you know, you hit the weekly limits after a day or two because you've given it some like, hey, you know, I wanna... Whatever it is.

Whatever it is, you could give it these tasks and it just runs on it. If you're not paying attention to the fact that it's stuck in some kind of a loop trying to [00:34:00] reason around a bug or whatever, it'll just go and go and drain everything right away. the, on the other side, if you're actually paying for like Bedrock tokens through AWS or you're paying for an enterprise account, if you're paying like actual dollars for tokens and API access, that adds up very quickly.

Should know that off the top of my head. I think it's, yeah, five bucks for a million input tokens and 25 bucks for a million output tokens, and that matters.

So then the, the things that you're typing in, is a lot cheaper than the stuff it's pumping out.

Chris Van Wingerden: Hmm.

Adam Reynolds: And when you're looking at the 25 on a million, you may use, like, I could tell you with a small feature, I might drain through 10, 15, 20 million tokens in context, and maybe that's at 20 million, $25 per million.

I mean, you're looking at, I guess, like 500 bucks. So that's not great, but also the amount of time it can save you if you have a good system in place, that could be well worth the investment.

Chris Van Wingerden: Mm-hmm. A lot of times I think of this as like, and maybe it's just top of mind because we have a, a new puppy in, in the [00:35:00] family at large.

But it's a lot of time and investment up, up, up front to make sure that that puppy gets the behaviors and the- That's right ... and, and the things that you need it to do. Gotta teach it its

Adam Reynolds: catalog.

Chris Van Wingerden: commands, training, and all of that stuff to get to a, an adult dog that, that fits what you're, what you're needing or wanting.

Uh, uh, and-

Adam Reynolds: Yeah, what a wonderful analogy. You're absolutely right. The thing with the dog, when they're young, they... You have like a six-week period or something like that when they're 6 to 12 weeks old where you really wanna load their brain up with everything that they're gonna experience in their life, if you can.

So like lawnmowers, skateboards, anything that you know triggers a dog, experience it then. and they're building their catalog of interactions and experiences they're okay with. it's so interesting 'cause it's exactly the right frame of mind. Take your AI like a new puppy. Teach it all the things you need to know up front.

Ideate on it, teach it, catalog it, give it these skills, give it all these things, and then let it, you know, let it run.

Chris Van Wingerden: Mm-hmm. And you have to endure the fact that it's still gonna chew the furniture. the messes, et cetera, that, that [00:36:00] it will leave behind in that timeframe. You have to be able to move beyond that, so...

Wrap Up and Final Takeaways

Chris Van Wingerden: Adam, this has been a really, really cool conversation. and I think it's been really awesome to bring that programming perspective and that set of experiences. We've had lots of sessions on AI, and even some sessions recently, you know, on governance. if folks wanna reach back into our catalog a couple of, sessions ago, a- and another couple of sessions at different times of this past year in particular about AI and AI and governance, go back and check out some of our past Instructional Designers in Offices Drinking Coffee episodes.

Adam, thank you so much for joining us here today. It's been fabulous having you here.

Adam Reynolds: It was a pleasure to be here, Chris. Thanks

Chris Van Wingerden: for having me. Um, I, I have a funny feeling that you and I are gonna go into our next meeting, and half this conversation's gonna trickle into that too.

Adam Reynolds: Probably.

Chris Van Wingerden: We're probably not done here- It never ends, right?

the two of us. It never

Adam Reynolds: ends.

Chris Van Wingerden: Folks, remember, as always, Instructional Designers In Offices Drinking Coffee, #IDIODC, is brought to you by the team here at dominKnow, makers of dominKnow ONE, helping learning and development teams develop, scale, and deliver learning that maximizes employee value.

The end [00:37:00] goal is to change how people do things so that, your organization can function better, can earn more, be safer, all of those things. And even if you're throwing stuff into Claude, you still gotta meet those goals at the end of the day. That same stuff still has to happen, so.

Adam Reynolds: Yeah, you still own them.

Chris Van Wingerden: Gang, thanks so much for joining us. Thanks for the conversation in the chat. It was, really nice to see a robust discussion going on in there today too. catch us again in a couple of weeks, folks. Talk with you all soon. In the meantime, let's dance on out of here.