Northern Arizona University (NAU) pursues student learning assessment for the dual purposes of improvement of student learning and accountability to our stakeholders. Faculty, in collaboration with staff, work together to understand, document, and enhance student learning. Over the past three years, a number of initiatives have energized the conversations regarding assessment on campus: the adoption of NAU university-level student learning outcomes, the approval of university-wide degree program expectations for curriculum and assessment, and a proposal to refresh all senior capstone courses.
In fall 2012, the Faculty Senate approved NAU university-level student learning outcomes, statements about what knowledge and skills NAU expects all undergraduates to have achieved upon graduation, The “NAU Graduates Can…” statements articulate NAU’s shared values and goals of degree programs, liberal studies, and our diversity and global learning initiative. University-wide student learning outcomes provide benefits to all university stakeholders by clearly expressing what it means to earn a degree from NAU. The university-level outcomes are posted on the NAU website at http://nau.edu/University-Learning-Outcomes/.
In May 2014, the Faculty Senate established that all degree programs achieve and maintain Degree Program Expectations for curriculum and assessment. The Degree Program Expectations are designed to collectively engage degree program faculty members in developing, implementing, or maintaining the following:
1. A Degree Program Purpose
2. Degree Program Student Learning Outcomes
3. A Curriculum Map or Matrix (identifies what courses address the Learning Outcomes)
4. Strategic Course Design (course assignments and activities are intentionally designed to achieve the course’s purpose and learning outcomes) of courses in the degree program
5. Systematic Assessment of Degree Program Student Learning Outcomes
6. Use of Assessment Findings for Continual Improvement
The six elements of the Degree Program Expectations, when linked, provide a framework for intentionally designing and implementing faculty members’ shared vision of a degree program’s curricular goals, which is linked to high-quality student learning and meaningful assessment efforts. When faculty members collectively design and implement a degree program curriculum (Degree Program Expectations 1, 2, and 3), they aggregate and prioritize aspects of the curriculum, leading to the intentional development of common curricular goals in the courses and class sections designed by individual faculty (Degree Program Expectation 4). In this manner, intentional connections across concepts and skills are developed throughout the student’s undergraduate career, forming a stronger web of learning than could be expected from isolated design of individual courses developed outside of a degree program’s purpose. Once a program has developed an intentional design, assessments (deliberately designed class assignments) of student learning (Degree Program Expectation 5) can provide information about how well students achieve the curricular goals of the degree program, and findings direct faculty members to either improve the program through specific, well-informed intervention, or celebrate student learning (Degree Program Expectation 6).
In Spring 2015, the Liberal Studies Committee drafted a proposal requesting that the Faculty Senate require all degree programs to refresh their senior capstone courses. Capstone courses at NAU were originally designed to integrate students’ disciplinary and Liberal Studies experiences through a culminating experience in the major. Extensive course “drift” of capstone syllabi from the requirement’s original purpose has occurred. This proposal was also based in part on recommendations from the 2011-2012 ETS Proficiency Profile report, including seeking out ways to better connect writing with critical thinking in the senior capstone course. Since the capstone is designed to be a culminating experience in the major, refreshing the capstone has the added benefit of providing the opportunity for degree programs to refresh the assignments in the course to efficiently and, simultaneously, effectively assess student learning in the major. Through this initiative, programs will be on their way to achieving the Degree Program Expectations of Systematic Assessment of Degree Program Student Learning Outcomes and the Use of Assessment for Continual Improvement, requirements for NAU’s accreditation visit in AY2017-18.
Northern Arizona University administered the ETS Proficiency Profile in 2014 - 2015.
Northern Arizona University conducted a Senior-only benchmarked administration of the ETS Proficiency Profile in 2014 - 2015. The results are displayed below in the SLO Results tab.
For additional information on NAU’s process for administering ETS Proficiency Profile, please click on the Assessment Process Tab below. For information on the students included in the administration, please click the Students Tested Tab.
The ETS Proficiency Profile was selected by NAU for VSA reporting because of sampling design and administrative flexibility as well as affordability. Most importantly, the ETS Proficiency Profile provides score information regarding the general education skills of critical thinking, reading, writing, and mathematics and context areas of the social sciences and natural sciences that are aligned to NAU’s Liberal Studies Program student learning outcomes. As a result, faculty can use ETS Proficiency Profile results to make modifications to curriculum and learning design, recommend improvements to assessments, and ask future assessment questions.
Senior students were assessed in capstone courses across the curriculum in fall 2014 and spring 2015. As recommended by the VSA, seniors were tested as close to graduation as possible. The fall semester 2014 testing window was October 27 – November 21; the spring semester 2015 testing window was March 23 – April 24. NAU used a modified stratified random sample strategy to target specific senior capstone courses within each college, aiming to ensure that the percentage of capstone students sampled in each college approximated the percentage of the entire student population of seniors enrolled in each college. Capstone instructors were invited to volunteer to participate. If instructors agreed to participate, their students were required to take the assessment. Each semester, all students who scored in the top 25% of test-takers were entered into a drawing for four $400 gift cards to the University Bookstore. Students in 30 capstone courses in all colleges across the university as well as extended campuses participated in the ETS Proficiency Profile.
Following confirmation of participation by senior capstone instructors, student cohorts were constructed in the online ETS Proficiency Profile Administrative System. Cohorts corresponded to the sections of capstone courses taught by participating instructors. As students participated in the online assessment, either face-to-face in a campus computer lab or at a distance online, their test results were entered into the Administrative System. Students received their individual results immediately after completing the assessment. Instructors received the results for the students in their capstone courses at the close of the testing window. All results were available to the test administrator for data analysis and reporting purposes.
Results were obtained through reports available in the ETS Proficiency Profile Administrative System, including the Custom Comparative Data Report. A series of reports were designed and shared with faculty, administrators, and the university community. Results were reported by Liberal Studies Essential Skills outcomes that were aligned to the ETS Proficiency Profile score categories and skill dimensions. The results focused the senior total score and subscore means and standard deviations, percent of a selected comparison group of Doctoral/Research Universities I and II scoring below NAU seniors, and percent of NAU seniors classified as proficient compared to percent of the selected comparison group institutions classified as proficient on the ETS Proficiency Profile skill dimensions.
The results of ETS Proficiency Profile informed and guided university-level conversations regarding teaching and learning of the Liberal Studies Essential Skills in courses across the curriculum. In particular, discussions focused on refreshing the senior capstone courses to better integrate these skills. Additionally, student performance on the essay and multiple-choice writing assessments informed discussions about reconsidering course design in the junior-level writing courses. At the program level, assessment results prompted focused conversations regarding the expectations for student performance on these skills and how learning opportunities can be scaffolded to help students successfully achieve these expectations.
The senior students enrolled in capstone courses across the curriculum who participated in the ETS Proficiency Profile were reasonably similar to the population of all eligible seniors in terms of the demographic characteristics of gender, race/ethnicity, number of Federal Pell Grant recipients, and ACT and SAT interquartile ranges. The sampling goal was to ensure that the percentage of capstone students sampled in each college approximated the percentage of the entire student population of seniors enrolled in each college. That goal was partially met, with students in the College of Engineering, Forestry, and Natural Sciences and the W.A. Franke College of Business being somewhat overrepresented and students in the College of Education and Extended Campuses being somewhat underrepresented.
The VSA advises institutions to follow assessment publisher guidelines for determining the appropriate number of students to test. In the absence of publisher guidelines, the VSA provides sample size guidelines for institutions based on a 95% confidence interval and 5% margin of error. So long as the tested sample demographics represent the student body, this means we can be 95% certain that the "true" population learning outcomes are with +/- 5% of the reported results. For more information on Sampling, please refer to the Research Methods Knowledge Base
At NAU, senior students who completed the ETS Proficiency Profile Critical Thinking test (n=453) scored higher than 49% of seniors at all other ETS Proficiency Profile-participating institutions in Spring 2014.
At NAU, senior students who completed the ETS Proficiency Profile Written Communication test (n=453) scored higher than 33% of seniors at all other ETS Proficiency Profile-participating institutions in Spring 2014.
As NAU did not participate in a value-added administration, scores are not adjusted to account for the incoming ability of NAU students.
The chart below shows the distribution of student scores on the ETS Proficiency Profile Critical Thinking test. Students are scored as Not Proficient, Marginal, or Proficient.
The charts below show the distribution of student scores on the three levels of the ETS Proficiency Profile Writing Test. Students are scored as Not Proficient, Marginal, or Proficient on each level. Writing 3 represents more advanced writing skill than Writing 2, which represents more advanced writing skill than Writing 1.