How AI Can Support or Prevent True Learning With Justin Bruno

Full Transcript

Dr. Cristi Ford:

Welcome to Teach and Learn, a podcast for curious educators, brought to you by D2L. I’m your host, Dr. Cristi Ford VP of Academic Affairs at D2L. Every two weeks I get candid with some of the sharpest minds in the K through 20 space. We break down trending educational topics, discuss teaching strategies, and have frank conversations about the issues plaguing our schools and higher education institutions today. Whether it’s ed tech, personalized learning, virtual classrooms, or diversity inclusion, we’re going to cover it all. Sharpen your pencils, class is about to begin.

Hello and welcome. If you’re a teacher student or someone who works in education or actually in any area at this point, you know that the topic is on everyone’s mind is around artificial intelligence. That’s why we’ve been speaking with many experts in the emerging and evolving field to really get a better sense of what AI is, what it’s capable of, and what to consider when applying it to teaching and learning. Today’s conversation with Justin Bruno, AI strategist at Michigan Virtual, who is helping establish high quality personalized learning experiences for students and professionals in the K two 12 space.

Welcome to Teach and Learn. Justin,

Justin Bruno:

Thank you for having me, Cristi. I’m so excited to have this conversation.

Dr. Cristi Ford:

I’m really happy to have you here. I feel like in the last couple of months we’ve really delved deeply to talk about AI in the realm of higher education, so I’m really excited to talk with you and talk about applicability in K-12. And so when I saw your title, Justin, you are the AI strategist at Michigan Virtual. I was really intrigued. I’d really love to hear from you a little bit more about what this role entails. And then for our listeners who may not know very much about Michigan Virtual, if you can share a little bit more about the organization as well.

Justin Bruno:

Absolutely. So yeah, everyone is intrigued by the title. I’m excited by that. It sounds very cool and it is very cool. I get to do some really exciting cutting edge work. So I had been as of about three or four months ago, working on our team at Michigan Virtual that does professional learning. So we would design, develop, deliver professional learning across a number of different areas for all sorts of educators across the K-12 spectrum.

On the side though, I had been playing around with AI tools, so this was in May of last year. I was playing with something called the OpenAI Playground API, which was powered by GPT-3 at the time, and had started to wave my arms around internally and say, “Look what these tools can do.” I would give it prompts from our social studies courses that we host through Michigan Virtual and say, “Look, these are writing good answers to our assessment questions. We should be thinking about this pretty deeply.” Then in November, ChatGPT is released and it’s all anyone can talk about at that point in time.

Dr. Cristi Ford:

Yes.

Justin Bruno:

And so we get asked more frequently like, Hey, you’ve got some knowledge and expertise about this. I was able to write some blog posts and share some expertise through some of our presentations. I ended up writing some professional development courses about ChatGPT early this year, early last year now in January and February of 2023. And so as the momentum kept building, the more and more questions kept coming our way. More and more time was being asked of us. At Michigan Virtual, we thought, “Hey, we should have somebody full-time be dedicated to this work.” And that happened to be me just because I was maybe a half step ahead of most people and thinking about these things and writing about them and exploring these tools.

So it’s a nice crossover. I still do professional development work. I’m on the road now two or three days a week doing a training, a presentation or a workshop. But we’re also trying to get into the real actuality of the strategist role is to sit down with leaders and think at a deep level, how is this going to impact what your organization does, your school does in five years from now? How do we scaffold that future? Think about steps we can take to get staff up to speed about AI, what it is, what the impact will be, how that’s going to impact your goals, your strategic plan as a district, as an organization. And that’s really in the nascent stages right now. We’re just starting to go into those deep partnerships, but it’s really exciting work.

It’s work that we get to do at Michigan Virtual because we have a lot of different activities at Michigan Virtual. What we’re primarily known for is doing K-12 online learning. So that’s where we get to partner with D2L to host our online courses. We facilitate those courses, develop, design, deliver those courses asynchronously with mostly ninth through 12th graders. But then our other big area of focus is professional development, like I mentioned that I had worked on as well. So we do that face-to-face, synchronously, as well as online asynchronously. We deliver hundreds of thousands of hours of PD in a given year, mostly for educators who are looking to earn recertification, but we go deep in a number of different areas including technology and now AI.

Dr. Cristi Ford:

I love that. And as I listened to you talk about your experiences and your position, I hope my superintendents and principals are listening in terms of thinking about this role as an AI strategist. I love the combination of you thinking about professional learning and really the application, but also the strategy pieces of that. I think that’s why I really love to spend a little time talking about where AI is headed, how you’re thinking about AI, and for the listeners who haven’t thought deeply about the difference between just general AI and large language models, I really want to focus on large language models in this conversation like you talked about with ChatGPT and some of the work you’ve been doing.

And so, one of the blog postings that you wrote that I thought was really fantastic, you really emphasize that large-language models could help or hinder learning, and you pose a question and a piece that you wrote about AI learning and performance. And so I’m just going to go right in deep here and really have a conversation about the distinction between where AI can be a proponent versus where AI can be a distractor and how we can think about the importance of distinguishing learning from performance. And so if you can just share a little bit more with the audience and where you sit in all this, we’d love to hear your perspective.

Justin Bruno:

Yeah, absolutely. I think professionals outside of the education space see ChatGPT mostly as a real productivity enhancer, a performance booster. It can help me write my emails, write my reports, draft my presentations, those sorts of things. Education’s much different. Education’s not solely about output, it’s about learning, which is a difficult, nuanced, different process. So I’ve been really trying to tease out the differences and the necessary contexts and conditions under which the use of something like ChatGPT or a large-language model can really benefit learning. And so that means a structured process, an informed process. It means thinking about how we measure the output when students use an LLM to create that output. And so there’s a ton of different considerations that have to go into that process. The difficulty there is that we’re in such a new and emerging space that we don’t have a lot of good research to point to say, this is when it works and this is when it doesn’t.

So we’re all trying to figure that out together. There was one study that was published just last month actually that showed under some specific conditions, you can get at what researchers have long called this idea of desirable difficulty. So LLMs are like a magic trick. They take a lot of difficulty away. They give you an answer right away, they give you an explanation right away. But when you engage in the learning process, you do want some level of struggle and difficulty and grappling with something in order to really be able to internalize a concept and be able to express it long term.

So this one study basically showed that if you give students math problems without the use of an LLM, then you give them math problems with the use of an LLM, an explanation about those math problems. They can then do better on an assessment at the end of the intervention.

And that makes total sense. That’s how we teach and learn in schools now. We want a guided experience rather than something that helps students and provide students with all the answers or maybe even overwhelms students with all the answers right away. So if you think ask ChatGPT a query, it gives long bulleted answers that might be exactly what you’re looking for, but it’s not necessarily helping us in the true process of learning. And so, that’s where I think we as educators are all going to have to define that space and what it looks like over time and hopefully build more evidence and research to help us get to that promising and practice this notion of when should we allow the use of LLMs? How should we guide students to use them? Teaching them about what they’re good for, what they’re no., and then again, displaying the benefits for teachers to say, “Look, this can be a helpful tool in your back pocket when you aren’t available to answer some of those questions.” That’s where I see a lot of potential benefit in this space.

Dr. Cristi Ford:

Yeah. As you’re talking, I think this issue is so complex and when we talk about LLMs or just AI in general, the thing that I struggle with is I’m thinking about even at pre-LLM days, when we know that AI has been around for decades. And I remember seeing a study back in 2019, 2020 that focused on middle school foreign language. And the study focused on the students setting aside 30 minutes for an AI assisted personalized review of material. And what they found was a 16% increase in retention. And so I started to think about, well, when we think about learning sciences, when we talk about retrieval, we talk about interleaving. Some of the ways in which we reinforce learning, there is this notion of this automatic AI instructor refreshing our knowledge base that really does feel a bit compelling. And so, how do we really distinguish that opportunity versus some of the other pieces that your discussion?

Justin Bruno:

I think, knowing how to structure the introduction of material, the assessment of the learning of that material is going to be a totally different conversation than what we’ve typically thought of. As the technology just gets more and more embedded into what we do. So you won’t just be going to ChatGPT for assistance. If you’re presenting any kind learning material through a technological interface like Google Docs or elsewhere, you’ll have some tool that can summarize that information for you, that can reword and present that information in a different way. So we really have to be mindful of talking to students, teaching students about how to use those functions and how their interaction with those tools through those lenses really impacts what they are able to learn and what they need to learn long term. I think that’s a whole other question about what will we need to learn once we have more and more capable models that do that work for us in a sense. But right now, at least in the medium term, we still need to be thinking hard about the interface between the human and the information in the process of learning.

Dr. Cristi Ford:

And when you talk about the human interface, I think that’s one of the things for us at D2L, we really focus on keeping that human in the loop. So as we develop and outfit Brightspace and other entities with AI, we really are starting to also think about how do we make sure to keep that human in the loop? How do we make sure that educators, instructors, faculty members, learning design, and in corporate America, how do they make sure that they are responsible for making those final decisions? And so I want to go back to your piece around performance. Agreed. We know that increased performance doesn’t necessarily mean that something’s been learned in ways that we know that time on task doesn’t necessarily mean that something has been learned. But is there really an understanding behind this increased performance? What are your thoughts around, does AI actually facilitate a better grasp of material of subject matter, or is it a skill equalizer for students who are coming in with a deficit in a subject area? How do we grapple with all of those parts of this conversation?

Justin Bruno:

I think the research just isn’t there yet. So in the early phases of research that we’ve seen, we do see some benefits through the use of the AI to help in that retention process, but we haven’t yet set up good experiments to contrast that process with a human in the loop or a teacher in the loop. So what we’ll need to do is more of those RCTs to figure out exactly how we can compare those experiences, which I am sure people are working on as we speak. I do think though there’s just a fairly obvious benefit that anytime a human isn’t available that this could be an option for that learning process for that student. And so there’s the standard that Ethan Mollick at the University of Pennsylvania talks about is like, is it better than the best available human? But if there isn’t an available human at that point in time, then this could be a pretty good option in those cases and in that process.

So your point about skill leveling, that same study of the consultants in the workforce found that the lower performing consultants got the most benefit out of the use of the AI tools. They were able to perform more tasks more quickly and with higher quality results compared to consultants who had been considered high performers at the firm. So you think about how that hypothesis might be reflected in school and education. I’m certain it could be a skill leveler if students come in with deficits in certain areas, it can help them produce those outputs more quickly or on the same plane. But the question is, are they still learning, maintaining that and retaining that information and going through that deep learning process that we want them to have to be able to be successful outside of the school walls?

Dr. Cristi Ford:

Yeah. I love that you talk about this partnership around this research piece, which is so, so critical, and I agree with you, the research isn’t there yet, and then the practical application in terms of professional learning. So it’s really great to hear that you are on the road talking with teachers in classrooms, in schools, buildings, talking with individuals who are teaching our future’s brightest. Can you talk to us a little bit about those kinds of conversations? I know that you have some thoughts around strategies that educators should be employing as you’re starting to grapple with AI in the classroom setting.

Justin Bruno:

Basically, if someone’s vaguely surface level familiar with ChatGPT, I talked to them about think about something that you are having a problem or an issue with right now. Maybe it’s engagement, maybe it’s differentiation, how to think about something that you’ve been teaching for the last couple of weeks but hasn’t worked for all of your students. How to think about maybe differentiating that experience for that student. And then we talk about ways to craft a prompt or to input some of their own existing curriculum materials into a ChatGPT or another tool with some goals based on real problems of practice that they have. And then when they see what it can do to help them in that way, you’re usually on a better glide path about, okay, these can help you in your job. And again, not just help you write your emails faster or write your lesson plans, but they can really help you tackle deep problems of practice in the classroom.

Then you ask the question, well, why shouldn’t your students have access to some of these tools? And what are some things that they are struggling with that a tool like this might help them? Again, knowing that we have to guardrail that process to ensure learning and not just task performance, but that usually breaks the barriers down and gets people thinking a little bit harder about what to do. There’s also tons of tools that are built around ChatGPT, so Magic School, Diffit, these other tools that do that generative work for them. I think those are great starting points, but shouldn’t be the end points of those conversations because it’s not just going to be standalone tools. Like I mentioned, this technology is going to be embedded in computing as we think about it or as we know it right now. So thinking about the ways that we’re all going to have to change our interactions with computers and how that impacts learning is going to be a big process, and that’s a big elephant to try to swallow-

Dr. Cristi Ford:

One bite at a time, Justin, one bite at a time.

Justin Bruno:

Right.

Dr. Cristi Ford:

Well, one of the things you share in your opening remarks about your position was a focus around really your interest in looking at AI LLMs and thinking about assessment. And so when you’re talking with educators, when you’re talking with teachers about assessment as it relates to AI, what are the guardrails around those kinds of conversation?

Justin Bruno:

So there’s a few different lenses there. The first is that for the foreseeable future, big high stakes summative assessments are not going away anytime soon. So end of term exams, standardized tests from the state and from national tests and things like that, those probably aren’t going away. And so, you have to think about, still having to, you have the mandate to prepare that student for those assessments. And so if they are using an LLM in an unguided way as a shortcut, they’re really not preparing themselves for that eventual high stakes assessment. So you do want to make sure that if you just are, let’s say you’re an English teacher, you see that you’re getting tons of AI generated work back through the essays and things like that, and you don’t flunk the students, you grade them and you just assume like, “Oh, well, maybe this is their work, maybe it’s not.” But then they get to an end of term exam or a high stakes assessment and they’ve not really learned anything through the course of that year because they’ve been outsourcing all their work to the LLM. Then you’ve set that student up for failure, they’ve set themselves up for failure.

Again, thinking about the ways to incorporate it, the one thing I hate is that there are folks and teachers especially who don’t even want to broach the subject because they feel like it’s opening a Pandora’s box. And so having those really difficult conversations and knowing how to incorporate the use into an assessment are really important.

Then you think about, okay, in terms of my maybe formative assessments or lower stakes assessments, how do I change that formatting, knowing that I’m going to need to know that these students have these tools at their disposal? So making those assessments multimodal. So use an image generator to help illustrate the point, and that’s just one portion of a project. Use an LLM to revise your writing or be a writing thought partner in that way.

Putting all those pieces together and submitting it as one project or as one assessment, you can then assess the coherence between all those pieces, and that’s not something yet that we’re outsourcing to AI, that’s still a human skill, communicating really complex ideas and in different ways. So thinking about how to change your assessments in that way, make them broader, bigger in scope and in scale, I think is important.

And then the last thing is just incorporating more metacognitive objectives into your assessments, right?

Dr. Cristi Ford:

Yes.

Justin Bruno:

Asking students questions that ChatGPT cannot answer. Because you are the human, you are the student who has a lived experience and something to share, and so motivating them to share that is really important. It’s even more important now. Helping them clarify their own points of view in their assignments and their assessments, helping them identify their beliefs. We are be incorporating those kinds of strategies and steps into our assessments now as just one component of assessment rather than just content knowledge or demonstration of proficiency of a skill.

Dr. Cristi Ford:

I love everything that you shared around the importance of thinking about cognition and thinking about strategies to really engage learners differently. As I’m thinking about this role and what teachers have to play here, it sounds like a lot of great opportunities, but it sounds really complex and it seems like it may take me as a teacher more time to really go back and look at the prep that I’ve done maybe for the last five years, and teaching that social study lesson. I have to maybe engage in a different way with the curriculum and the content. And so how do you grapple with those conversations when you’re having those with teachers and principals?

Justin Bruno:

Yeah, you can’t sugar coat it. I mean, it’s tough. My wife is a teacher and thankfully she’s an elementary art teacher, so she may be one of the last ones to be really disrupted by this technology, but I do not envy the position that they’re in. It’s going to take, and I am upfront, I’m saying it’s going to take years to think about the ways that we restructure this and reformat this so that we are not totally ignoring these tools, we are trying to derive and harness the benefits of them in the learning process too. And again, I hate to say it’s another thing added to teachers plates, but in some senses it is.

Now the makers of a lot of these tools will say, “Well, with the productivity gains that you can get, you can maybe find some savings and time and efficiency to be able to reinvest that time into rethinking curriculum, rethinking assessments and things like that.” But again, I would say, can’t sugar coat it, just acknowledge that it’s going to be a long process. But I do think it’s necessary, right?

Dr. Cristi Ford:

I would agree with

Justin Bruno:

That. There’s this concept, it’s a Japanese word called Ikigai, which is like the overlap of, I think the actual translation is something like worth or self-worth or something to that extent. But it’s basically where the overlap is between what good at, what you enjoy doing, and what the world finds valuable. And if you can find that sweet spot, there are probably only a few fortunate people in this world who can really find that sweet spot who love what they do day in and day out. I think about schools as essentially trying to be facilitators of that process of finding what students love to do, what they’re good at, and what’s really going to be valuable for the world at large. And so thinking about what that means, it means making sure that students aren’t closed off from this technology.

Dr. Cristi Ford:

I wonder for you, one of the things that you have on the website that you’ve recently published is a great illustration of a couple of different scenarios of where students can use AI as one of three things. Let me see if I get this right. I think you talk about it being an accelerator, an assistant or a crutch. And so if you can share with me a little bit about each of those and the importance of this resource that you all have created.

Justin Bruno:

Sure. So this was our attempt to honor some of the concerns that folks in the classroom have, and I think they’re justified. I think we don’t have a lot of great research yet to paint clear pictures about where these use cases fall and what they look like. But I think when you think about students using AI tools, everything has to be contextual and situated in a different context. So what they’re using the AI for, what that student brings with them as a competency or a skill already, and what they’re using the AI tool to maybe further that skill or build that skill, but that there’s a potential for the AI to shortcut the building of that skill. And so what we think about this is no matter where a student is in terms of the skills or the competencies they have, they can use AI as a crutch.

They might just use it to have some shortcuts through the process that they they’re learning on. So maybe they procrastinated, they need to pump out an essay really quickly, they use it as a crutch. Even though they could have written that essay if they had better time management skills, whatever that might be. There’s other examples of using as a crutch to say, I didn’t really engage with this material. I might have gone to the LLM and asked it questions that I wanted to answer in the context of my learning, but I didn’t then apply it in any way. I still just used the output from the LLM to get my grade. So that’s still not helping that student in the learning process in any way.

What we would like to see is more use of AI as either an assistant in that process, so being a thought partner, and that really gets at what we talked about earlier, is giving students guidance about what these tools are capable of doing, how to prompt them in the best way to get their best output, and then having that student generate their own thoughts, ideas, maybe doing a first draft of a project in some way and then consulting an LLM for feedback or for suggestions or revisions and things like that. Knowing where in the process of any given assignment or learning they can use the LLM, could either be just an assistant to help them get that task done, or ultimately what we ideally like to see is an accelerator.

So maybe you’ve got a student who is so far advanced in your class, but you just don’t have time to give them some enrichment opportunities. If they’ve built that skill of using an LLM, they can go to that LLM to co-design a project with ChatGPT for themselves as long as they know the standards that they’re going to be assessed on. So that’s ultimately where we see a different a spectrum, and I’d like to see more research verify those assumptions, but again, I think that’s a more nuanced way to think about how AI can and should be used in a student context.

Dr. Cristi Ford:

So very helpful. What comes to mind is ChatGPT or any LLM, the output is only as good as the prompt. And so as we are talking with teachers who are really trying to dip their toe into exploring this, what are you offering them in terms of thinking about what we call prompt engineering? Or how do they develop really good prompts?

Justin Bruno:

Yeah, I think prompt engineering right now, based on the capabilities of the systems we have, is a really important skill. And this is verified by research, you’ll get better output by giving the LLM some context, being really clear about your goal for its output, and then even better results you get when you do something called few shot prompting or you basically are giving examples of something that you would like the LLM to produce. So it’s got a reference in that way. But just being as detailed as you can, context specific as you can, and then providing examples in your prompt, those are the best ways to get good output on the first try.

The second tip I would offer is to say, you can get over being a bad prompt engineer if you just spend more time and don’t really expect, and this is sometimes where the magic wears off, because admittedly, it is more work to do this, but if you just spend more time asking ChatGPT or an LLM to give you more iterations, it will do that too. You’ll say like, “No, that wasn’t exactly what I wanted from you. I need you to reformat it in this way or emphasize this piece a little bit more.” And so it is more of like a conversational skill you have to develop, and it’s a weird skill to develop because people are not used to interacting with computers in really human language. And so I encourage people to get over that hump, spend more time with the tools that way, and ultimately they’ll see a pretty big payoff, I think.

Dr. Cristi Ford:

It’s really been a real treat and pleasure having this conversation with you today, Justin. I really appreciate having your thoughtfulness around what you’re seeing K-12 and the ways in which we all need to be not just thinking about the practical applications, but around the research. And so I would urge our listeners to visit michiganvirtual.org/ai. We’re much of Justin’s research and writing on the subject is published. You can also follow him on the platform formerly known as Twitter @JustinBruno, and you can find him on LinkedIn @JustinBruno.

And for those of you who are coming to follow us at Teach and Learn, remember to follow us on social media. You can also find us at LinkedIn and Facebook or at Instagram @D2L, and you can find our YouTube channel at Desire to Learn Inc. Make sure to check us out at D2L’s Teaching and Learning Studio for blogs, master classes, and thought-provoking content for educators by educators.

Thanks so much for listening. Thank you, Justin, for joining us.

Justin Bruno:

Thank you, Cristi. This was amazing.

Dr. Cristi Ford:

You’ve been listening to Teach and Learn, a podcast for curious educators. This episode was produced by D2L, a global learning innovation company, helping organizations reshape the future of education and work. To learn more about our solutions for both K through 20 and corporate institutions, please visit www.d2l.com. You can also find us on LinkedIn, Twitter, and Instagram. And remember to hit that subscribe button so you can stay up to date with all new episodes. Thanks for joining us, and until next time, school’s out.