How KSU Built an Analytics Platform That Faculty Love, With Dr. Anissa Vega

Full Transcript

Dr. Cristi Ford (00:00):

If the idea of analyzing data sends you into a tailspin, you’re going to want to hear this.

Hi, I’m Dr. Cristi Ford, and today I welcome Dr. Anissa Vega, Associate Vice Provost at Kennesaw State University. For a long time in higher education, we’ve been discussing the importance of data-driven decision-making, but it’s not always an easy thing to execute on. In this episode, Dr. Vega shares how her team successfully shifted the culture around data, leading to positive student outcomes. Join us.

Intro:

Welcome to Teach & Learn, a podcast for curious educators, brought to you by D2L. Each week, we’ll meet some of the sharpest minds in the K-200 Space. Sharpen your pencils. Class is about to begin.

Dr. Cristi Ford (00:42):

Listeners, you are in for a real treat today. I’m Dr. Cristi Ford. I’m so pleased to welcome Dr. Anissa Vega, Associate Vice Provost at Kennesaw State University. In her role, Dr. Vega provides oversight from everything from curriculum to digital learning to academic policy and academic innovations. Dr. Vega, really glad to have you here today.

Dr. Anissa Vega (01:03):

Thank you so much. It’s an honor.

Dr. Cristi Ford (01:07):

As I have watched all of the amazing work that you do, it is interesting to see all the different pieces that make up your role. But today we’re going to really talk about data. You’ve been doing some amazing things. I first learned about some of the work that you’re doing from a learning impact award that you all received in the spring, but you’re doing some amazing things with data-driven decision-making at KSU with your analytics platform called uHoo Analytics. So to kick things off for our listeners, can you first explain what the heck is uHoo Analytics and how it was built and what you use it for?

Dr. Anissa Vega (01:42):

Well, uHoo is an alerting system for our faculty. It’s been a whole campus-wide effort to bring this to fruition, and it’s built pulling our D2L data into an Azure platform where we have Power BI feeding out analytics for faculty insights across the campus.

Dr. Cristi Ford (02:08):

It’s really fantastic in terms of what you’re doing here, and I guess I would love to start with the impetus of what were some of the main challenges Kennesaw State was facing, what faculty were facing, that led really to the development of the analytic tool?

Dr. Anissa Vega (02:23):

Student success has become an institutional priority, and that really requires a lot of faculty time and dedication in any context, but specifically in higher education, the process of gathering student teaching and learning data, analyzing that data, it’s incredibly time-consuming, and so we typically push that off to between semesters. And we really wanted to find a way to support the current students within the courses during the semester, and so we needed an analytics platform that could provide that to faculty.

Dr. Cristi Ford (03:05):

So this is interesting, what I’m hearing often from faculty: just the workload, these kinds of things, the course administrative workload. And so to your point, they get pushed aside or they happen in the summer or during the lull times of the academic year. But can you talk about how these challenges really impact faculty workflow and student success, as you talked about student success being a really big part of Kennesaw State?

Dr. Anissa Vega (03:34):

Yes. So typically, when we wait until after the semester to review our course data… Now, reflective practice is fantastic and we encourage our faculty, and most of them do practice reflective practice. However, typically, that reflective practice supports our future student: the student in the next semester or the following academic year. And so by providing faculty with data and insights every single day, that gives them the opportunity to impact the student who’s in their classroom today. And so that’s where analytics and uHoo Analytics is further advancement. And really because we’re feeding the analytics, faculty don’t have to do anything extra beyond using their D2L platform for any modality that they’re teaching in, actually. It doesn’t have to just be fully online courses, but if they’re using that platform, that learning management system, then they can help their students in any class throughout the semester.

Dr. Cristi Ford (04:44):

I love this because what you’re offering is you’re creating this just-in-time data to be able to pivot and create these interventions for students as needed in the current semester. And so we long talk about in higher education the need for data-driven decision-making and a culture of data-driven decisions, but sometimes these challenges are the reasons why we see them happening in fits and starts.

And so as you’re chatting with us today, I guess I’d love to understand how you’re building capacity around this use of data, and I’ve read a little bit about the work that you’ve done in this realm, and you talk about the use of data is building patterns and behaviors. And this is one of the things that I saw that you have instituted, Kennesaw State has modeled so well, for other institutions to learn from. And so can we talk a little bit about the Monday Measures emails? I first, I’d love for you to explain to them what they are to listeners, because I’m just a huge fan of how you’ve pulled this concept together.

Dr. Anissa Vega (05:50):

So we approached this really as a design-thinking project as a team, and putting faculty up front. Their motivations, their processes. And we knew it was challenging to get a large number of faculty to adopt a new technology. And so we wanted to draw them into the uHoo Analytics system in a way that was natural to their day or work week. And so we decided to take the alerts that we are generating within the uHoo Analytics system from the D2L data, and we’re automatically sending them to our faculty on Monday mornings, and it provides them with a list of key data points that we want to prioritize their attention on that Monday morning to give them some curiosity about what’s going on in their class. And right from that Monday Measures email, they can click on it. That’s all they have to know how to do to be able to access uHoo Analytics and it pops it right up in their browser, pulled straight to that particular alert and the student that is alerting.

Dr. Cristi Ford (07:09):

When I learned about these Monday emails that you send out, it really reminded me of the importance of making things easy for faculty drives adoption. So having that Monday morning set email come into their inbox, really targeting those prioritization-around-the-data pieces. But I love what you shared here today is it’s also about thinking about design thinking, and doing this practice around… I imagine when you talk to faculty, you started to develop this. You wanted to figure out what challenges they needed to solve. What were the root cause and the problems that you needed them to help to solve so they could do their jobs better? So I really appreciate the focus that you share in that. And so can you maybe share with our listeners some of the specific data points or trends that go into the Monday Measures emails, how you prioritize them, and how do you help faculty identify students who might be struggling in that particular semester?

Dr. Anissa Vega (08:10):

Sure. So we had to start with generalizable alert triggers. So we as a group really had to decide, and faculty had to provide input, what alerts would be the most useful to the most faculty across campus. And so the first of those alerts is any student who has failed an assignment or received less than 60% of the possible points in the past week. So that helps identify students who might be struggling. And faculty then, of course, can click on that Monday Measure alert and look at the rest of the data that that student has generated, including previous alerts that the student has generated.

The second type of alerts is alerts related to more than half the class failing an assignment, rubric, or quiz. So we want to bring faculty’s attention to those particular engagements that are not being successful, where the students on a whole are struggling. That way, faculty can consider revising or looking for errors within those particular assignments.

And then the third type of alert is really at the item analysis. So we’re talking about a rubric criterion where more than half the class failed that criterion, or a quiz question where more than half the class has failed that quiz question, triggering faculty to really go back in, check to see that they marked the correct answer… I know I’ve made that mistake a couple of times where I marked the incorrect answer as the correct answer. Or maybe the question was worded poorly. It’s worth throwing out, perhaps, or just revisiting.

Dr. Cristi Ford (10:07):

I love all of these approaches because they are so comprehensive. One, you talked about the opportunity to have an intervention with a specific student who might be struggling. Two, when you talked about a specific assignment that maybe more than half the class is not doing well on, you’re offering that opportunity for continuous improvement: for that faculty to really go back and look and say, is that the best way to assess the learning outcomes for this particular topic or content or material?

And then three, when you’re talking about the other level here, it’s really, really important that faculty think about the fact that we weren’t trained… I mean, you and I, as I know, even what I know of you, we weren’t trained to be psychometricians, right? So thinking about that item analysis, understanding a particular question and what might not be testing well or may not be valid in terms of really being able to assess the thing that we think it’s assessing. And so this now gives us three nice comprehensive data points to be able to bring to faculty members’ attention on the Monday morning. I just love this work. I just love this work. I think that you all are doing such amazing work around this.

Dr. Anissa Vega (11:19):

Thank you.

Dr. Cristi Ford (11:20):

So I talked about continuous improvement. Can you talk a little bit about uHoo, and you got to share with the audience before we move any further how you figured out this uHoo around your mascot, because I want you to share that with them. But uHoo Analytics is said to provide course improvement insights. Can you give some examples of such insights, and how have you seen your faculty utilizing them, Kennesaw State’s faculty utilizing them, to enhance their courses?

Dr. Anissa Vega (11:50):

So we are the Owls. So Hooty Hoo, who is one of our slogans, I guess, and so anything with Hoo in it, or hoot, we employ it. So trying to gather someone’s attention, uHoo seemed to make a lot of sense. Some faculty like it. Some make fun of it, and I get that. That’s totally fine.

Dr. Cristi Ford (12:14):

I love it.

Dr. Anissa Vega (12:15):

So as far as our course insights, we have several different dashboards available to faculty when they click into it, and what they’ll find at the very top is our most popular dashboard, and it’s called course analysis. And in course analysis, it’s pulling across the data in their D2L course and putting it all on one screen. So they get the broad view, multiple data points at one time that they can consume. So that includes class averages on assignments. It includes attendance. It includes data about whether or not students are accessing the modules and the module content. It includes the distribution of grades, as well as even the list of alerts that have been generated that week within that course. So it gives that broad overview.

But we have another dashboard that’s very popular. That’s called the Student Analysis Dashboard. And in the Student Analysis Dashboard, faculty can select one or multiple students that they want to look at in comparison to the class at large. So this dashboard allows faculty to see how that student or subset of students are doing in comparison to the class average on the class assignments. It helps them review their attendance, also whether or not that student is generating any alerts, whether or not they’re accessing the content. So it gives a broad overview of that particular student as well.

Dr. Cristi Ford (13:53):

Thank you for sharing that, because I think it’s going to get some of our listeners to be thinking about the kinds of data they’d like to be pulling from Brightspace or the ways in which they might be able to utilize the data to be able to empower them as faculty or for others to be empowered around this work. And so you talked just now a little bit about individual courses and faculty members being able to see those instances, but can we talk a little bit about uHoo Analytics supports may be program coordinators in assessing program-level outcomes and making really data-informed decisions at that level?

Dr. Anissa Vega (14:28):

Sure. So we’re piloting right now a few dashboards that are pulling that Brightspace data from key assessments.

Dr. Cristi Ford (14:39):

Okay.

Dr. Anissa Vega (14:39):

So what we have here is where faculty can choose to put their program coordinator into their course in a role that we’ve created called program coordinator. And with that, that program coordinator can’t actually see the content within D2L and Brightspace. What they can see is in uHoo Analytics, it’s pulling key assessments.

Dr. Cristi Ford (15:05):

Okay.

Dr. Anissa Vega (15:06):

And so that program coordinator, that job of gathering all that data, handing out the key assessment and analyzing that data, is historically super time-consuming,

Dr. Cristi Ford (15:20):

Tedious but required, right? For our accrediting bodies, for our professional associations and the like, we need that data.

Dr. Anissa Vega (15:30):

And it’s valuable.

Dr. Cristi Ford (15:31):

It’s valuable.

Dr. Anissa Vega (15:32):

We learn a lot, but it is tedious.

Dr. Cristi Ford (15:34):

It’s tedious. It’s-

Dr. Anissa Vega (15:34):

I don’t know anyone who likes to do it. So what’s fantastic about this is now we can automate that whole process. The program coordinator needs to build those key assessments within a Brightspace shell that their instructors can copy over, and then that creates the scenario and the setting where we can automatically pull that data for them and provide them with those insights. And of course, we’re hoping that, because this is so much easier for our program coordinator and less time-consuming, that now our program coordinators are going to be our biggest champions to encourage faculty to use Brightspace for all of their courses and uHoo Analytics.

Dr. Cristi Ford (16:23):

So I love that you’re focused on the faculty member first and really getting just-in-time information and data to faculty to be able to create interventions to help students right now. And then you moved out to this next level or circle to program coordinators to say, “Okay, hey. We now see that there’s some opportunities to be able to support your work and create some efficiencies of scale to be able to do that.”

You’ve talked about all the great things you’re doing, and as people are listening and thinking, “Okay, that’s really great, but I’m wondering how many faculty are using it,” I want to talk about your adoption, because one of the things that I think is you’ve had significant growth in uHoo Analytics adoption at Kennesaw State. And so I’d love to understand what factors contributed to this wide acceptance among faculty, and what was your secret ingredient to getting such great adoption numbers?

Dr. Anissa Vega (17:18):

So today, we’ve got about 370-plus-

Dr. Cristi Ford (17:24):

Major.

Dr. Anissa Vega (17:25):

… faculty using uHoo Analytics, which is a large portion of our faculty.

Dr. Cristi Ford (17:31):

That’s major.

Dr. Anissa Vega (17:33):

We started with a pilot group of faculty. Well, we actually initially started with a beta group, and those are the early adopters who are excited, who will forgive me if something doesn’t work right and won’t immediately throw the baby out with the bathwater. And those individuals were able to provide us with a lot of feedback to get the tool working, and then we pulled in a 50-member faculty group to become the pilot group last fall.

Dr. Cristi Ford (18:06):

Oh, wow.

Dr. Anissa Vega (18:07):

And they were working with it closely. They were leading in focus groups, giving us feedback, making recommendations for feature changes and fixes. So they were critical to what it actually looks like today.

Dr. Cristi Ford (18:20):

Okay.

Dr. Anissa Vega (18:22):

And then we took that to our faculty senate, and we shared with our faculty senate how these 50 faculty members have shaped this tool and helped prepare it for them. And then in addition, following that, we opened it up broadly to the faculty last February, really. So we’ve really just had since February to August now to bring in this 100 or 2-

Dr. Cristi Ford (18:55):

Yeah. More than 200.

Dr. Anissa Vega (18:55):

320 faculty on board. And so there has been, I would say, relatively rapid adoption by our faculty on campus. And one of the things that were really important that we got questions right away, whether that was the beta group, the pilot group or the faculty senate, they wanted to know, is this data going to be used against them? Is this for surveillance? Right? They don’t want to use a system that is just going to feed data into their administrator’s inbox.

Dr. Cristi Ford (19:36):

Right. Then that can be used for P&T or for other evaluative things.

Dr. Anissa Vega (19:39):

Right. And I get it.

Dr. Cristi Ford (19:42):

Yeah.

Dr. Anissa Vega (19:42):

I understand that. And I also recognize that they’ll just stop using the systems that feed the data into a surveillance system. So we have been committed from the beginning, with that design-thinking approach starting with the faculty, that we were going to protect this data such that it would be used in the way that it’s intended: for the faculty member to learn, grow, and improve their courses and have an impact on student success.

And with that, we’ve also helped faculty create templates in how they can share how they’re using the data to improve student success and student outcomes, and that’s really important, especially for certain subgroups of our faculty, because the student evaluations, as wonderful as they can be, they also can have a lot of bias for certain subgroups. And so having additional data points and tools to help those faculty make an accurate representation of their impact on student success is really valuable. So those templates are helpful for them to use, and they get to present the story themselves and how they want to show that they are using uHoo Analytics to improve student success.

Dr. Cristi Ford (21:07):

So this is such a critical piece. I love that you created a pilot group, but what I’m hearing is you created a pilot group of 50 faculty, I’m assuming from across the institution. You thought through making sure that every college within the institution was represented. The fact you went to the faculty senate and had that level of transparency and buy-in was really, really critical. And I imagine those 50 faculty were also some of your champions to help other faculty know, “Hey, this can really be helpful for you.”

Dr. Anissa Vega (21:42):

Absolutely, yes. And of course, they were early adopters as well. And the reason for selecting early adopters… I know that sometimes we want the resistant faculty, but I knew something-

Because I knew things needed to be fixed and changed early on. I needed faculty who would also forgive us our errors.

Dr. Cristi Ford (22:06):

And a little grace.

Dr. Anissa Vega (22:07):

And give us a little grace as we were getting it up and rolling. Yeah.

Dr. Cristi Ford (22:11):

Yeah. So for many of those listening who are working in institutions, working in organizations, that are considering implementing learning analytics tools for their faculty or have started down that path… I mean, you’ve done such a good job of optimizing this at Kennesaw State. I just wonder what lessons learned could you share with your experiences with uHoo Analytics to this audience? And maybe from a couple of different lenses as you think about this.

Dr. Anissa Vega (22:43):

Sure. So as I mentioned before, this project has been a massive collaboration across campus. The faculty wanted to be involved and needed to be involved. That would be one of the first things that I would recommend. Involve the faculty early. Help them articulate the goals and the questions that they would have of this data. Commit to protecting the data. And then start with that pilot group of faculty. Keep the lines of communication open with your shared governance bodies. I think that’s really important as well. Faculty don’t like to be surprised.

Dr. Cristi Ford (23:32):

Yeah.

Dr. Anissa Vega (23:34):

We also engaged our university information-technology team very early as well, talking to them about the idea, the goals, and supporting them as they upskilled to be able to join us in the endeavor. They were amazing in that process.

And then also listening to what other institutions have done. I had heard of other institutions making the mistake of using learning-management data for dean dashboards or centralized dashboards and the outcomes of that, which was faculty upset. So I had heard that and knew that that needed to be an early criteria of the tool.

And then there is no substitute for executive leadership support. You really need to garner that support up front and listen to what they perceive as success measures, and make sure that you can target those success measures and maintain their support.

Dr. Cristi Ford (24:40):

That’s fantastic. So I think I heard you talk about keeping faculty in the loop and at the heart of these kinds of initiatives. You talked about making IT your best friend and bringing them cookies and donuts if you have to to bring them along as you build up these infrastructure. You talked about learning from other institutions, so talking with colleagues around where there was some lessons learned that they had to share where you could find some successes. And the last thing, you talked about the importance of executive leadership and really making sure that you had champions and buy-in at that level being really, really critical for institutions to be able to do this well. Did I hit them all?

Dr. Anissa Vega (25:18):

You hit them. That’s great.

Dr. Cristi Ford (25:21):

Well, I really appreciate you joining me today. Thank you so much for being here. Thank you so much for what you’re doing. Thank you for being willing to share and really talk about the work that you’re doing at Kennesaw State University and the ways in which you are really catapulting forward student success at your institution.

Dr. Anissa Vega (25:38):

Well, we couldn’t have done it without you guys, so thank you to Brightspace too.

Dr. Cristi Ford (25:42):

Absolutely. Great to see you, and bye for now to my guests.

Thank you to our dedicated listeners and curious educators everywhere. Remember to follow us on social media. You can find us on X, Instagram, LinkedIn or Facebook @D2L, and subscribe to our D2L YouTube channel. You can also sign up to the Teaching & Learning email list for the latest updates for new episodes, articles, and masterclasses. And if you like what you heard, remember to rate, review, share the episode, and remember to subscribe so you never miss what we’ve got in store.

Dr. Cristi Ford (26:14):

You’ve been listening to Teach & Learn, a podcast for curious educators brought to you by D2L. To learn more about our K-20 and corporate solutions, visit d2l.com. Visit the Teaching and Learning Studio for more material for educators by educators, including masterclasses, articles, and interviews.

Dr. Cristi Ford (26:33):

And remember to hit that subscribe button, and please take a moment to rate, review and share the podcast. Thanks for joining us. Until next time, school’s out.