How AI Changes the Expert-Beginner Relationship and Stunts Skill Development With Dr. Matt Beane

Full Transcript

Dr. Cristi Ford:

Welcome to Teach and Learn, a podcast for curious educators, brought to you by D2L, a global ed-tech company committed to transforming the way the world learns. I’m your host, Dr. Cfour-and-a-half-risti Ford, Vice President of Academic Affairs. In each episode, either myself or one of my colleagues will meet up with some of the sharpest minds in the K to 20 space. We’ll break down trending educational topics, discuss teaching strategies, and have frank conversations about issues plaguing our schools and higher education institutions today. Whether it’s ed tech, personalized learning, virtual classrooms, or diversity, inequity, inclusion, we’re going to cover it all. Sharpen your pencils. Class is about to begin.

Hello and welcome. As you know, recent advancements in AI, in addition to its popularity and availability, mean that educators and faculty are thinking a lot about its usefulness. As we explore this topic through various lenses, it can be easily get swept under the rug in terms of the opportunities and promises, the hope and the potential. So today, we’re going to really turn to some sobering research on how AI is creating a gap between experts and students when it comes to on-the-job training. Today’s guest is someone who is a renowned thought leader in the field of machine intelligence and robotics. He’s a member of the technology management faculty at UC Santa Barbara, a digital fellow at Stanford, MIT, a writer of Substack, which we’re going to talk a lot about in a minute here, investigating the world of work, learning intelligent technologies, and somehow found time to add author to his resume. Welcome, Dr. Matt Beane.

Matt Beane:

Thank you. Thank you very much and please, just call me Matt.

Dr. Cristi Ford:

All right. Well, Matt, before we dive in and talk about your research and your upcoming book, I just want to share with listeners how I first became acquainted with the work that you’re doing. Last summer, I had the opportunity to meet you at the Fusion Conference for the executive summit, and after meeting you and really aligning to the ways in which you’re thinking about where the future of work is going, I started immediately to subscribe to your Substack, the Wild World of Work.

And if you recall, last fall, when you started to really get some traction in this space, you talked about not letting AI dumb you down, and I really appreciated that positioning. And one of the things that you talked about was really interesting to me because I’ve been reading reports that talk about AI being a skill leveler, but your notion was very different. You talked about de-skilling, and so I thought maybe we could start the episode today talking a little bit about the ways you’ve chatted in terms of using AI and how is it causing us to lose skills and maybe some of the advantages that come from the retrieval practice of those skills.

Matt Beane:

Happy to start there. We obviously have plenty of fun stuff we can talk about. So the first thing that is really important to say is that no one has yet proven through scientific study that using AI will drain your skills away. So that piece that I wrote in the Substack uses previous literature that we have. A lot of previous research I think, strongly suggests that we might be missing a subtle, slow de-skilling process for many, many millions of people with this technology. But we don’t have studies yet. And who knows by the time this airs, we may be there.

But I think the basic way to think about, and I don’t think this just applies to AI, about the impact of using a technology like generative AI to write a work report to buff up on a resume, to create some images or a blog post even, is that in many ways, it is absolutely fantastic. If you’ve actually used it and tried to do that kind of work, at least for myself, it’s amazing to watch what you can get out of these systems so quickly and how pretty good it is, actually.

And that’s what a lot of these studies show, right? So one of the big recent ones show consultants putting this technology to work and getting speed gains at the very least, they just can produce their content that much faster. The other part of it though is that they can get quality boosts, especially if you’re in the lower end of the skill distribution. You’re not terrible at your job if you’re in these consulting firms by and large, but there are some folks who do better than others. Those folks who are at that lower end of the distribution get a bump up. This all sounds fantastic and I’m not going to stand here and say that it’s not good. The hidden cost associated with that though has to do with the fact that it is a much more self-serve way of getting your work done that involves less struggle than doing work collaboratively with people without the technology. It’s not to say we should give it up. In fact, I use this technology probably more than most people, every day, hours.

So there’s two things to unpack there. The one is the self-serve component, and this is the theme in all of my work since maybe 2012, 2013, all my research and so on. So that can be dumbing down for the novice, the dumbing down that comes for the person actually using it, who picks it up and thinks, my goodness, this is absolutely incredible technology, is that you are not really forced to encounter your limitations in a way that drives you to the kind of struggle that the research really shows drives further curiosity and motivation to learn, build new skills, but also, visibility into problems that are adjacent to the one you’re looking at right now. In some cases, it’ll do that, but in other cases, you’ll just get your quick saccharine hit of productivity, you’ll get your result and you’ll be done and you don’t get the time space and pain really, difficulty, that drives the human organism to look for better, more creative ways of solving problems.

So that’s the kind of dumbing down that I’ve definitely seen. I’m at ground zero with my students undergrads and master’s students and have been for the last, I don’t know, eight months using this technology and I have seen it up close and personal. I have quotes in that piece that you mentioned in the Substack that really blew my hair back about some of my very best students in some ways getting great results real fast. But what they say about how it felt to do it is really revealing and I think it should give us all pause. A little bit of, I don’t even think anymore, I just couldn’t turn over my problems kind of thing.

Dr. Cristi Ford:

Yeah. You have this fantastic Ted Talk where I learned the term shadow learning. Can you share a bit more with us about this premise and the context of what is shadow learning and what does it mean?

Matt Beane:

Sure, happy to. And actually, so for me, that was quite a moment. I got called to Ted’s headquarters in New York. They do them in Vancouver or out west and then in New York. And that was early days in my research compared to now, right? So that was in 2018 and I’ve gathered a lot of data since then, with a lot of diverse kinds of technology. But in a way, the story’s the same. And the basic story that I presented in that talk was that if we’re not careful for the productivity gain that we’re getting with all of these amazing automating technologies, what we’re sacrificing is the novice expert bond that has been behind almost all skilled work for… Its fun. The historical records go back about 160,000 years. So back down to the beginning of language, we see evidence of this in the archeological record. That collaboration across expertise levels is so fundamental to the way that we get work done, interact with each other at work and build skill that we take it for granted.

However, I’m just using ChatGPT and Gen AI as an example. The more you can self-serve, the less that’s necessary. The less that person. You might need a novice’s, extra hands, extra eyes, extra ears to get this work done, the less they’re going to get involved. In surgery, this is the first place I went to go really study this. And yeah, that meant two and a half years of being in an operating room with a laptop on a surgical stand, typing as fast as I could about who was saying what to who, who was doing what around the operating room in old-fashioned surgical procedures and robotic surgical procedures of the same kind, to compare how a lot of things happen. But the main one was how are you learning how to do this thing over here versus how you’re learning to do do it over here.

And this has been independently investigated by a number of news outlets, but showed up in a number of the stories that came out that got me that Ted Talk. That for most surgical residents, the net effect of that new technology, it allows that surgeon to do more by themselves. They essentially have three arms to control with their hands and feet actually. And that just means that new person sits down at a trainee console if they’re lucky if there’s a second one there and they just watch, they don’t get their hands on the controls much. And when they do, that other surgeon is watching them like a hawk. And if they make a single mistake, by the way, it’s up on TVs for everybody to see, throughout the operating room and to give you feedback. Let’s say you’re the new resident and I’m the senior surgeon, I’ve got to yell at you from across the room like do this, don’t do that.

So everybody hears everything I’ve got to say to you and I can’t use my body to communicate with you anymore. Whereas we’d be standing side by side. Bottom line, for most residents, even at the top, I only studied the top programs. Forgive the metaphor, they were cut out of the action. They couldn’t learn by doing alongside an expert. There were about one in eight, and I think this is probably less so at not top hospitals, about one in eight surgical residents managed to learn anyway. They got really, really good at robotic surgery, but they were not doing it in the approved normal way you ought to learn. And there are papers about this in surgery, it’s called Dwell Time. And the basic assumption is if I get you in the operating room with a senior surgeon, you’re going to learn stuff. You’re going to get better at surgery.

And in the old-fashioned mode of surgery where four hands were needed to do the job, that was really actually true. So a four and a four-and-a-half-hour procedure, you’re on task four hours, sometimes five and a half before and after. And you’re learning the whole time, boy, oh boy, do you have to focus? That’s engaging work and you’ve got a mentor right there, moment by moment. Robotic surgery, maybe you got 15 minutes and it’s going to be on the easiest part of the procedure. But the people who got really good did a different set of things that let them build skill and they didn’t talk with each other. That was what made it an interesting scientific finding. They were all independently cooking up this new way of learning and they were doing it in isolation. They thought they were the only one, so to speak.

And to give you just the examples that played out in surgery. But I’ve since seen them in many other occupations, they all strained against the norms or rules on how one ought to go about learning. No one would quite approve of these if they saw the light of day. The one that really gets folks attention, it certainly got my attention, is operating without a senior surgeon in the room. So in the paper, I called that under supervised struggle. But another one that’s maybe, still surgeons do not like, did not approve of, it’s getting better now is learning by watching YouTube. I called it digital rehearsal in the paper. But bottom line, a surgeon who learned a lot, a junior surgeon, really fast was very prepared, they found a good quality YouTube clip of a real surgery, say a couple hours long, and they would watch it 150 times very carefully. And what they were doing at time 122 was very different than time two. They were taking notes, writing notes to themselves and so on.

So that bundle of practices that allowed them to build enough skill that when they got into the operating room the first time, the senior surgeon took a look at them and went, “Well, you really know what you doing? I’m going to give you more rope to play with. I’m going to let you do more.” And that positive snowball allowed them to build skill. But it all came through practices that really were basically rule breaking, like naughty, naughty. You should not be doing this stuff.

Dr. Cristi Ford:

By any means necessary.

Matt Beane:

Nobody punished them ever and nobody forbid them to do these things. But when these got listed in my findings, in my talk, in various news outlets, there were plenty of folks who said, “Yeah, that stuff is true and that’s not too good.”

Dr. Cristi Ford:

Yeah. Well, and when you talk about this, what I thought was so fascinating when you talk about the lengths that students were going to fight to protect this space, you talk about these three Cs being fundamental for skill development. Can you share a little bit about those three Cs? And then I think I’d also love to hear as you share that, we have lots of curious educators that listen to this podcast. How is this applicable to the work that educators are doing every day?

Matt Beane:

Yeah. So the journey to the three Cs was another four years after I did that original surgery paper because if somebody was to go read this, the paper is called Shadow Learning because it was about… The discovery to me was, wow, you can learn by cheating. That was the basic insight in the paper, which is, the minute you realize that you think, well, folks have been doing that for thousands of years. And I’m like, well, yeah, we just hadn’t quite named that thing. Then I started to look in other professions because I thought to myself, well, how common is this that you should break the rules of your own profession or occupation to learn? Are we seeing it in other places with different kinds of automation? So the next three, four years of work was trying to spider out to different data sets, different occupations, different kinds of technologies. Saw that same pattern very consistently. It started to get… Well take this in exactly the right way, boring because it’s the same story over and over.

Then came this period of folks telling me, “Boy, you should probably write a book.” And I started to take that seriously. But what the book needed is a positive thesis on what it is that really enables skill development. And by the way, skill is a fuzzy target. So I think it’s worth saying, from my point of view, from the point of view the science I’m familiar with, skill is, if you have it, you can do something where other folks look at it and it looks like you doing it very effortlessly in really strange and difficult conditions. And to you, it feels like you can just get results pretty much no matter what somebody throws at you. It’s grace under fire-level skill. If you have that, what was part of your journey in order to get that skill?

The shadow learners I studied, obviously were doing something, but not everybody does digital rehearsal. Not everybody practices without the expert in the room. So those practices that they use that were breaking norms, they were seeking to protect certain aspects of their work experience that allowed them to build skill. I didn’t name them as such in that paper. I was just like, here are these three practices. They seem to work. So the three Cs that you mentioned, which are challenge, complexity and connection, really came as a result of me looking not just in surgery but across more than 30 occupations and professions, different kinds of technology and trying to figure out, okay, we have shadow learners in all these places, what are they trying to protect in their experience that gives them real skill that’s valuable under pressure?

And it took, right up until the 11th hour of getting a book contract, actually, talking back and forth with my agent where he’s like… So I sent him a proposal with these three Cs in there, but it was like two months before I owed him that proposal that I figured out, all of these folks and all these occupations and professions were trying to protect, to put a fine point on it, the ability to struggle in their work in a healthy way, finding challenge in their situation. Not too much, not too little. The book really details this out and draws on all the research, gives you a 10-point checklist for what healthy challenge looks like.

Everyone from all these professions, occupations, and so on, including school by the way, we’re finding ways to protect healthy challenge. And the same with complexity where challenge is, are you close to but not at or beyond your level of capacity. You should feel really totally focused and you should not be performing at your best. You should be messing up occasionally. It should be hard. Complexity is a different dimension. That is, a lot of the literature on skill development and learning is about am I getting good at this thing? Am I learning how to use my iPhone better to do X or am I learning how to write better? That’s a focus skill development trajectory that’s about progressing sort of closer to the core.

But there’s another thing about human learning, which is that we are in a web of knowledge and skills and as we progress towards expertise, we also look broadly about like, okay, I’m learning about how to run a good assembly line in a warehouse. Maybe I’m also interested about where these supplies are coming from or how do I resolve people problems that come up or different people will have different interests, but there’s this aspect of spidering out into our context and understanding that better. Part of what that allows you to do is do the single focus task better because you understand the surprises and shocks that can come at you from the complex system you’re sitting in. So there’s part of this preserving your healthy encounters with complexity that everybody was fighting for and protecting as shadow learners.

And the last one’s human connection. So it says connection, but that’s about bonds of trust and respect, really. There’s two components to that. One is you don’t get to do the work unless you earn the trust and respect of somebody with the authority to give you a shot. And the person who has that authority wants the trust and respect of that new person coming in too. There’s a very human component to whether you even get to play that is part of… So that’s part of it, but also, what counts as a worthy goal? Where did I come up with the idea of learning that it would be cool to learn how to do X, Y or Z, or you, mentors people we looked up to, and then getting socialized into that system?

One of my favorite studies I talk about in the book is a great study of Disneyland done by one of my mentors, John Van Maanen, it’s called the Smile Factory. Incredible study. Bottom line, you start out wanting to work at Disney and as you get socialized into that system, you learn a whole bunch of new skills like you don’t call people customers at Disney, they’re guests. There’s this way of thinking and these skills associated with being effective at Disney, at the park, that you learn by getting socialized into that system. So that human connection and the social fabric of the organization is as much a part of your ability to be competent as, can I work this cash register or can I deal with a lost child? That kind of thing. So those are the three Cs and they really came out of looking across all those contexts. This technology is making it harder and harder for people to learn across all these contexts. A rare few are managing to do it anyway. What are they protecting that’s letting them get ahead?

Dr. Cristi Ford:

That’s so, so good. And you’ve alluded to the book. So listeners, I’m excited to mention your upcoming book called The Skill Code, How to Save Human Ability in an Age of Intelligent Machines. And so listeners, it will not be out till later this spring in June 2024. So make sure you pre-order a copy today or mark your calendar. So you’ve started to really talk about the impetus around that book and how that book came to be. But are there other things that you can share with us about this book and the ways in which we need to be approaching our thinking as it relates to AI and robotics and the future of work when reading these chapters?

Matt Beane:

Well, I can tell you maybe a little bit of an origin story that I hadn’t already covered that might be fun and interesting for folks because it’s not a straight-arrow story, from point A to point C in terms of getting the book done. But then also, I think there’s a twist in the book or a punchline that might not be evident from the outside. I think that’s on purpose. It’s a Trojan horse, this book. So the first thing to say is that I did my original research on robotic surgery and had these original findings about how folks were managing to learn in these ways that really, seemed inappropriate, from the point of view of all these other surgeons, that was going to get published in the top management journal on the planet really, which that blew my hair back.

And then I reached out to a few mentors about this because my dissertation committee said, “Boy, get ready. This is an important finding and it has a lot of broad applicability.” And one of those folks that I reached out to was Adam Grant. He and I have been connected before I started my doctoral program and I said, “I think this is going to be a big deal. What should I do?” And he said, “Okay.” Well, he actually made an initial connection for me to the TED organization for instance. So that actually is part of what led to that talk. And I said, “Well, I got that done. I was working on a Harvard Business Review article related to that.” And I said, “I’ve been thinking about a book, and a few have mentioned that to me.” And he said, “Don’t you dare.”

He doesn’t talk that way, that’s me paraphrasing him. But he was like, “Look, I wrote books early in my career and it was really risky thing to do. You don’t have tenure yet. You need to cool your jets. Wait until you get tenure and so on. It doesn’t really count for…” Inside of an academic enterprise, you have to write papers. So I continued to do that and I had a lot of people that I respected a lot, strongly encouraging me to consider the book. And so I as a side hustle decided to write up a proposal, which took me about a year and a half because I was trying to do my day job of doing research, teaching and all the rest.

And to his credit, when I had a proposal that was in pretty good shape, I came back to Adam and I said, “I did exactly what you told me not to do.”

Dr. Cristi Ford:

I did the opposite.

Matt Beane:

Yeah. And I really need to do this. And he instantly turned on the jets to help me get connected to potential agents and help me get up to speed and so on. I was compelled to do it, I think is the fair thing to say, and it was not a smart or appropriate thing to do on many dimensions. I don’t yet have tenure at Santa Barbara.

So what happened for me though is that that time period between surgery and discovering this problem and opportunity in so many occupations and professions, it made me realize… I said it on the TED stage and I meant it at the time, but now, I really know it and mean it, this is a very, very serious problem and it’s very subtle. We don’t get that we’re trading productivity for human capital, basically, the next generation of human capital. We’re sacrificing the future to get productivity in the present. And so that’s a hard challenge to notice, and I feel like the purpose of this book is to try to get folks to pay attention to the issue and maybe allocate their resources differently. And this is the third part of the book that I think is this Trojan horse that I was mentioning, which is, technology, we often think… It’s not a sky’s falling book about technology. It’s the opposite.

Actually, it’s currently, often unnecessarily part of the problem, but it’s absolutely within our power to redesign technology and make it part of the solution. In fact, to make skill development faster, more efficient, more deeply human than it ever was before. In fact, I think we have to. So part one of the book is, here’s this skill code. We’ve talked about that. Part two is by the way, it’s deeply under threat right now because we’re not really understanding this trade-off we’re making. That shows how that happens. But the third part is to fix this thing. We can’t go back. It’s much more a make technology part of an enabling system that just by touching it, enriches our skills and enriches human connection, enriches our healthy complexity, makes it a little more of a healthy challenge.

In the last chapter of the book does the best I could do, just taking the available technologies that we have today, everything from large language models to robotics to virtual augmented reality, the whole nine to come up with a very realistic eight years out, seven years out kind of a solution that makes for a brighter skills future for us all because of these technologies.

Dr. Cristi Ford:

As I’m listening to you offer this perspective, one of the things I’m just thinking about and wonder where your take is here, oftentimes, when a new technology comes out that’s shiny and exciting and new, and we are novices at it, we don’t focus on what is the challenge that we’re trying to solve. So if we could focus on the root cause to your point, around solutioning and thinking about the next five to seven years, maybe just maybe, we could maybe be a lot more directed and strategic around thinking about how can we use this technology to be an enabler to help us to be able to keep the human in the loop, but to be an enabler, to move us where we need to go.

Matt Beane:

No question about it. You’re absolutely preaching to the choir on that one. There’s two parts to that and the back end of each of these chapters basically, especially to do with these three Cs, I talked to different parties who have different ways of helping on these spaces and challenges. One of them though is technologists, the folks who are actually building the tech, and there’s two moves there. One is that every single technology vendor, in my opinion, doesn’t matter what you’re building, if it automates work in some way, which is a wide variety of technology. The basic question to that person, to that firm, is can you get those productivity results for your client and make it so that just by interacting with it, dealing with the new workflow associated with that technology, all users have more skill at the end of a day, a week, a month-

Dr. Cristi Ford:

Oh, that’s good.

Matt Beane:

Than they did before they touched that thing. It’s not always possible. So to be real candid about it, there are plenty of places where you’re going to put in a series of conveyors in a warehouse to move parcels around and you’re not going to have people carrying them around anymore. We have emails, so we don’t write physical letters. There are some examples where you really shouldn’t try. At the same time, we’re not really even trying. And so any vendor that decides to say, we built new features into our system that mean that sure, your surgery is going to go better and faster to lower complication rates, et cetera. And the surgeons, and by the way, maybe even the nurses in the room are going to learn some stuff that they didn’t know by the end of the day because our technology is sitting there and it’s not going to be the main reason for the tech, it’s not the show, but they’ll out-compete their competitors. Who doesn’t love that story? They’re going to get clients, they’re going to get in the news because we’re all worried about skills. So that’s part one.

I think everybody can try for that, but part two I think is maybe, well, they’re certainly both necessary. Part two is about a new infrastructure for skill development that uses these technologies, not as part of some tool that I’d use to do my work, but as a way of connecting people who have skill, want to build it in various occupations in various workplaces around the globe in a giant network where instead of just relying on a potential mentor or mentee locally or a collaborator, you might get matched up, if you’re interested in spot welding in plastic, which is a new skill, you might get matched up with a mentor in Jakarta.

A system like that, that could use AI to make smart matches between people at the right time, at the right moment, to get the right mentorship for the right person. There’s plenty of lo-fi solutions for that kind of thing that organizations do now, like informal apprenticeships, mentorships, that kind of thing. But if it was supercharged by AI and globally connected, in addition to the tools themselves amplifying skill, we could have an infrastructure that was only possible because of all this tech, that could leave us all far richer by way of skill.

Dr. Cristi Ford:

Yeah. Matt, this has been a conversation in the making, so thank you for taking the time. You have a Ted Talk, you’re publishing a Substack, you’ve written a book, which I really look forward to reading. Thank you for making the time to be here with us today, and every time I listen to you give a talk or I’m inspired to think about the ways in which we as educators, we as technology providers can make a difference in the way that we’ll learn. So thank you for joining me today. Really, really happy to have this conversation.

Matt Beane:

It’s an honor and a pleasure. I’m really grateful and you all can lay claim to this being the very first podcast I’ve been on where I’ve even talked about the book, you caught me ahead of time as the book was beginning to happen, and now, here we are, first conversation in the can about it. So it just feels really great and a good full-circle moment to be talking with you about it here.

Dr. Cristi Ford:

Really, really appreciate that. Listeners, make sure to follow Matt Beane on X Platform or LinkedIn @mattbeane, that’s B-E-A-N-E or visit his website mattbeane.com. For those who want to learn more about his work, learning in tech, make sure to subscribe to his Substack. Let me tell you, colleagues, it is wild. It is so wickedly good and such an inspiration. It’s called Wild World of Work at www.wildworldofwork.org, and remember to look for his new book, the Skill Code available for purchase in June 2024. Thank you for listening today. Thank you for the Curious Educators everywhere. Remember to follow us on social media. You can find us on LinkedIn or Facebook @D2L and check out our YouTube channel @Desire to Learn Inc. If you like what you’ve heard, remember to subscribe, rate us, review us, give us some feedback, and let us know about this episode, share with a colleague. For now, this is all we have. Thanks for joining us.

You’ve been listening to Teach and Learn, a podcast for curious educators, brought to you by D2L, a global learning innovation company, helping organizations reshape the future of education and work. To learn more about our K through 20 and corporate solutions, visit d2l.com. Visit the Teaching and Learning studio for more material for educators by educators, including masterclasses, articles and interviews @d2l.com/teaching-and-learning-studio. Remember to hit that subscribe button and please, take a moment to rate, review and even share the podcast. Thanks for joining us. Until next time, school’s out.

Resources Discussed in the Episode

• The Skill Code