Thoughtful AI Integration in E-Learning: Lessons from Welbee

In a recent episode of IDIODC with Tim Handley, Chief Product and Technology Officer at Welbee and dominKnow | ONE customer, we gained valuable insights into how artificial intelligence (AI) can be meaningfully integrated into an organization’s training experiences. Welbee, a UK-based company focused on improving well-being in educational institutions, has successfully implemented AI across their learning curriculum, and offers some important lessons for training teams and instructional designers looking to enhance their content with AI.

Starting with Purpose

Tim emphasized that a key first step with any AI implementation is to avoid being driven by the latest shinny AI tool or even the demands of the C-suite to “use AI”. Instead, the first step with an AI journey should begin identifying specific challenges that exist in one’s own learning ecosystem. For Welbee, their initial focus aimed to address two key issues:

1. Helping users find relevant content within their extensive well-being resources.
2. Making sure that this “help” was available to an audience that spanned multiple time zones.

This led to the development of their first AI application - a chatbot trained specifically on their course content.

The Evolution of AI Integration

While the chatbot was the first step, this experience then led to additional refinement and inclusion of AI into various aspects of their eLearning content. Today, Welbee's learning content features AI in a variety of forms:

• Course-specific chatbots trained on relevant content
• Interactive knowledge checks with immediate, contextual feedback
• Dynamic role-playing scenarios for coaching practice
• And the beginnings of personalized learning path recommendations

In seeking out where AI might potentially make the next biggest impact, they identified their current coaching exercise. Previously this was a handout for the learners which provided some key guidance and steps they would then utilize to practice what they learned. While these types of on-the-job learnings can provide an excellent way to incorporate training into the real world, for learners they can be intimidating, which can result in minimal utilization and thus stunt the overall training impact.

Enter AI. This seemed to be a ripe situation to enable a learner to practice in a safer and realistic situation. In this case, the AI plays the role of a teacher seeking guidance, allowing learners to practice multiple coaching conversations in a safe, realistic environment. The AI not only participates in the conversation but also provides structured feedback on these conversations based on established coaching models that Welbee fed into the AI scenario.

Technical Implementation Best Practices

In the process of implementing these AI scenarios, several key technical considerations that should be considered emerged:

1. Content Control: Rather than relying on general AI knowledge, they grounded their AI interactions in carefully curated content that was specific to each course, or in some situations, a specific scenario within the course.
2. Temperature Settings: By adjusting the AI's "temperature" (creativity level), they were able to ensure responses remained consistent and aligned with learning objectives. For most of their AI Scenario’s they decreased the “temperature” considerably, so it more strictly adhered to the given content.
3. Prompt Engineering: While each Scenario utilized different prompting for the given situation, one key aspect was that the prompts were always clear and structured. Some of the key considerations Tim recommended were:
   1. Defined persona/role of the AI engine
      1. E.g., “You are an expert instructor who specializes in helping education staff improve their leadership skills, with a focus on the GROW model.  You are acting as a guide during their course 'An Introduction to Coaching'.”
   2. Context and background information
      1. E.g., “You are based in the UK, and speak UK English, and use terms relevant to the UK primary to post-16 education system. Any relevant content from the course and your background reading will be given to you below. Use this to help answer the user’s question.”
   3. Specific response parameters
      1. E.g., “You should response in a supportive manner, and reference resources from the course where they are relevant. You should seek to clarify the users question if needed, and in most cases ask a follow up question to check their understanding and/or help them explore the topic in more depth.”
   4. Clear rules and boundaries
      1. E.g., “You should ALWAYS:  Speak in British English.  Use a Supportive Tone.  Focus questions on coaching and leadership.”
4. Quality Assurance: Welbee’s extensive testing includes attempting to provoke unexpected responses and ensuring appropriate handling of sensitive topics. Tim noted that the more similar scenarios you created, the less additional testing was needed and, on the flipside, the more creativity you allow, greater testing was required.

Addressing Bias and Safety

One problem with AI is that bias can creep into your AI answers and interactions. As noted by Brookings Institute, bias is built into the data, the AI engines themselves can be biased, and the engines are trained on human rules. Welbee took this problem seriously and sought to manage AI bias by:

• Restricting the AI to only utilize carefully vetted content provided by Welbee
• The use of low temperature settings in the AI engine to maintain consistent responses more tightly tied to the provided content
• Implementing strict guardrails through precise prompting
• Regular testing and monitoring of AI interactions
• Clear labeling of AI-powered features for the users of the content

Last, for some of the more sensitive topics like mental health and suicide prevention, additional safeguards were added to ensure responses appropriately addressed concerns about self-harm or other serious issues. Taking it one step further, responses to these situations would always include specific required information (e.g., hotline number).

Impact on Learning Outcomes

Like any other tool, adding AI for AI’s sake is something to be avoided. One of the key aspects to avoiding this pitfall is including evaluation of the changes and their impact for your learners. In implementing their solution Welbee did just this and utilized the results to drive additional forays into AI. Tim noted that their efforts have been encouraging with results such as:

• Increased course completion rates
• Higher engagement levels compared to passive learning experiences
• Positive user feedback, particularly regarding the main chatbot
• Increased time spent using the learning resources
• Provision of more interactive and personalized learning experiences

Best Practices for Implementation

Are you excited to plunge forward into utilizing AI with your learning connect? From our conversation with Tim, several recommendations emerged:

1. Start Small: Begin with a specific challenge rather than trying to implement AI everywhere.
2. Maintain Transparency: Clearly identify AI-powered features and how they are used, while avoiding generic "AI" labels.
3. Focus on Enhancement: Use AI to augment or improve existing learning experiences rather than straight out replace them.
4. Limit Frequency: Avoid oversaturating courses with AI interactions; Welbee aims for max of 3-4 AI touchpoints in a typical two-hour course, clearly focusing on aspects that tie to point number 3.
5. Test Thoroughly: Invest time in testing different scenarios and edge cases before deployment. The amount of testing needed will increase with the more creativity (temperature) you allow and the greater use of outside content you permit.

Looking Forward

As AI technology continues to evolve, the opportunities for meaningful integration in eLearning will expand. However, Welbee's experience demonstrates that success lies not in technology itself but in its thoughtful application to specific learning challenges. As with any technology or technique, the key is maintaining focus on your learning outcomes and leveraging AI (or any other approach) to create more engaging, interactive, and effective learning experiences.

For instructional designers looking to incorporate AI, the message is clear: start with purpose, maintain control through careful content curation and prompting, and always prioritize the learning experience over technological novelty.

Next Steps

At the foundation of any great learning organization are systems and tools and teams that empower your learning curriculum. dominKnow | ONE provides an all-in-one LCMS and authoring solution that enables your team to work together better and fully embrace their skills and creativity, thus enabling your entire team to create highly engaging and effective learning experiences for your organization. To learn more about Welbee’s use of dominKnow | ONE, review an excerpt of some of the learning experiences. Interested in learning how dominKnow | ONE can help your organization maximize their potential? Contact us now.

‍