Editor’s Note: This article is Part 2 of a three-part Student Affairs Executive series examining how colleges and universities are turning data into evidence of institutional impact. Part 1 explored the shift from documenting activity to demonstrating what changes because of student affairs’ work. In Part 2, we examine what it takes to build a division capable of doing this well — from assessment expertise and leadership strategy to data infrastructure and a culture of evidence. Part 3 will explore the next step: turning evidence into action, including how institutions are using data to identify what isn’t working, intervene earlier and make better decisions on behalf of students.
Prior to when the current vice chancellor for Student Affairs joined the University of North Carolina at Charlotte (UNC Charlotte), assessment was not prioritized within the division.
“That was not a thing — and if it was happening, it’s because whoever the director was of that department at the time saw the value in it and knew it should be something that their office should be doing,” recalled Ellissa Brooks Nelson, the divisional director of Student Affairs Research and Assessment at UNC Charlotte.
When the current vice chancellor came to UNC Charlotte, he recognized the value of collecting data and assessing impact. He mandated all departments create their own assessment plans, beginning a period of transformation for the institution.
That transformation did not happen overnight. At first, UNC Charlotte provided departments with an assessment template and a deadline to submit their plans. According to Brooks Nelson, it became clear fairly quickly that more guidance would be needed.
“It was clear they just had never done assessment,” explained Brooks Nelson. “They didn’t know how to write an assessment plan or how to think about it. So I went back to our vice chancellor and explained to him what was happening, and I asked, ‘Can you give us a year? Just give us a year to do professional development with all the departments.’ And he gave me the green light for that.”
With the administration’s support, Brooks Nelson and her team developed assessment learning workshops that taught each department the basics of assessment and how to create an assessment plan from beginning to end.
Investments were also made in the assessment and data team, with Brooks Nelson moving into the divisional director role and additional staff brought on to focus on assessment, quantitative research and analysis.
The experience demonstrated an important distinction: Mandating assessment does not automatically create the capacity to do it well. Before UNC Charlotte could become more data-driven, leadership had to establish the expectation, invest in expertise and give staff the time and support to develop new skills. The technology that would eventually make the work significantly easier came later.
One of the biggest breakthroughs in enabling proper assessment at UNC Charlotte was the creation of the assessment council model.
The division shifted from relying on a now four-person assessment team to building assessment expertise across all 21 departments. Through an assessment council, each department designates a primary and backup representative who receive regular updates, professional development and guidance to serve as internal assessment consultants. Supported by senior leadership, the model enables departments to help one another while freeing the central assessment team to focus on broader strategy and continuous improvement.
But for the assessment council model to be successful, staff also needed access to data. Prior to investment in a data lake and proper technological infrastructure, that information was scattered or extremely difficult to access.
“A lot of requests would need to go to IR, and it would really depend on if they could even entertain that request, depending on the capacity and what else they had going on,” explained Brooks Nelson. “And even if we could answer those questions, it was taking us a substantial amount of time to get them the data they needed. And then my team would have to analyze all the data for them because that’s not something they’ve been trained to do. So it just wasn’t a sustainable process.”
A data lake has mitigated this issue, with nearly all departments within student affairs contributing data to the student affairs data lake on a regular basis.
“So now those questions around impact — where we’re really focused on things like retention and graduation — we can pull that data in a matter of maybe an hour, whereas before, it was taking us forever just to identify who the data owner was, where the data lived, and actually getting access to that data in the first place,” said Brooks Nelson.
This has enabled the student affairs division at UNC Charlotte to transition from traditional assessment to showing impact. All departments in the division now must have at least one impact outcome on their assessment plan, such as evaluating retention rates compared to those who did not participate in certain programs.
“Since we’ve brought in the impact outcomes, I’m starting to see a change in how our directors respond to assessment, because they’re seeing the value in it now,” said Brooks Nelson. “They’re able to see the value of students that engage with their offices, and really, the end result.”
UNC Charlotte’s experience offers one model for building the capability to assess impact across an entire division.
At the University of South Carolina (USC), Renée Delgado-Riley, the assistant vice president of Planning, Assessment, Innovation and Strategy for Student Affairs and Academic Support, approaches that capability from another angle: helping departments determine what they are trying to accomplish in the first place and how the data they already have can help them understand whether they are getting there.
Instead of just collecting metrics, Delgado-Riley approaches assessment as a human-centered strategy for understanding and communicating impact.
“I really help all our colleagues in our division think beyond just the metric, and really collecting data for the sake of being able to tell our story,” said Delgado-Riley. “What is the reason that we come to work every day? Why does our program exist? We build our assessment from that, so we can tell our story more strategically.”
Through individualized assessment coaching, she helps departments at USC begin with their mission and strategic goals, define the experience they want to create and determine what evidence will show whether they achieved it.
Because each department contributes differently to student success, the process is customized rather than one-size-fits-all. The goal is to connect existing data, surveys and institutional outcomes into a fuller story that helps teams demonstrate their value, identify opportunities and make better decisions.
One specific example of this in practice at USC involved Beyond the Classroom Matters (BTCM), the institution’s homegrown database containing 10 years of co-curricular engagement records and five years of curricular engagement data from experiential-learning courses.
The database captures activity for approximately 95% to 97% of students, but its complexity had historically limited how deeply staff could analyze the information. Delgado-Riley reformatted the data and integrated it with institution-level records, allowing her to distinguish among students with engagement records, students without records and the overall student population. That distinction was important because previous reports had sometimes compared demographic information from the engagement database with demographics for the entire student body, despite the two datasets representing slightly different populations.
The work has provided a new lens through which USC can examine an issue already receiving attention at the institution: male student engagement.
Conversations about male engagement, including at the Board level, began well before Delgado-Riley arrived at USC. At the broadest level, the data showed men participating less than women. However, USC also enrolls fewer men, meaning a simple comparison between male and female participation could obscure important differences within those populations.
When Delgado-Riley isolated the male population, separated those without engagement records and examined the programs in which men participated most often, a more nuanced picture emerged. Men who were participating demonstrated strong engagement in certain areas; their engagement simply looked different from that of women.
For a campuswide task force already focused on male engagement, the analysis provided another way to examine the issue. Rather than looking only at whether men were participating at lower rates, leaders could dig deeper: Where are male students already participating? What is drawing them to those experiences? Should USC expand similar opportunities or restructure existing programs? What do male students themselves say they need?
Delgado-Riley encouraged the task force to supplement the quantitative findings with focus groups and direct conversations with male students.
The example illustrates a larger opportunity for institutions investing in their data capabilities. Integrating information across sources and applying more sophisticated analysis can allow leaders to move beyond simply identifying a disparity or pattern. They can begin examining how student experiences intersect, where meaningful differences emerge and what those patterns may suggest about student success.
USC’s experience also exposes a limitation of a technology-first approach to assessment. An institution can have years of data at its disposal without necessarily having the full picture. The value comes from being able to connect that information, examine it from different angles and understand the experiences represented within it.
For Delgado-Riley, that deeper understanding should ultimately help leaders make better decisions about programs and resources. Sometimes, it may also provide evidence that something is no longer working.
That type of insight can be useful for sunsetting programs, something Delgado-Riley admitted the profession can sometimes shy away from. For this to happen, she advocated for the concept of psychological safety within teams — environments where staff feel safe to admit if a program or service is not achieving its intended goal.
“How can we create psychologically safe environments where we can make mistakes?” asked Delgado-Riley. “Because a lot of the times, there’s some results in this world of data where not every story is great. I think we have that compliance framework where we’re like, ‘Oh, let’s brush those to the side. Let’s only talk about the positive. Let’s only put this great data out here.’ And we have to be in a mode where, if we’re all great, then we have nowhere to go. And so it’s OK to look at it from a continuous improvement standpoint.”
Her point gets at perhaps the most difficult part of building a culture of evidence. Data will not always confirm that a program is working, that resources are being used in the right place or even that an institution understands its students as well as leaders thought it did. Sometimes it will challenge an assumption, expose a gap or suggest that a longstanding program needs to change.
For assessment to improve student affairs rather than simply document it, staff have to feel safe bringing those findings forward, and leaders have to be willing to hear them.
UNC Charlotte’s experience shows the infrastructure required to get there: leadership commitment, assessment expertise, professional development, accessible data and technology that makes analysis possible. USC demonstrates what has to happen once that capability exists. Leaders have to ask better questions, interrogate what the data is actually showing and be willing to change course when the evidence calls for it.
A true culture of evidence, then, isn’t one where every assessment produces a success story. It’s one where the findings — positive or not — can be used to make the work better.
LEAD WITH BETTER INSIGHT
Join senior student affairs leaders receiving practical strategies, peer perspectives and the trends shaping higher education.
With more than a decade of experience in B2B journalism, Rachel Chonko is Editor-in-Chief of Student Affairs Executive and Peake Media. She specializes in connecting leaders with the ideas, strategies and peers that can help them navigate complex challenges and strengthen their organizations.
Through Student Affairs Executive, Rachel works closely with senior student affairs leaders across higher education to explore the issues commanding executive attention — from student success, data and assessment to organizational strategy, financial leadership, technology and stakeholder engagement. Her reporting is grounded in conversations with the leaders doing the work, with an emphasis on practical insights, institutional case studies and peer perspectives.
Rachel also helps shape the Student Affairs Executive Summit, an intimate, discussion-driven event designed to bring senior student affairs leaders together for candid conversations, relationship building and shared problem-solving.
Connect with Rachel on LinkedIn or explore her latest articles.
