STEM Education – FEW NGCI

NEXT GENERATION CONCEPT INVENTORY

Project Synopsis 

To better inform those who teach, make curricular decisions, and manage college-level environmental programs, we are developing a tool for assessing environmental students’ foundational knowledge and their ability to grasp complex systems-level concepts. Specifically, we apply an established machine learning method of evaluating constructed response (short answer) questions to create a Next Generation Concept Inventory (NGCI). This new approach to concept inventory construction creates a new set of constructed-response items and associated automated scoring models focused on the complex systems typically addressed in EPs. This work focuses on Food-Energy-Water Nexus systems as an initial pilot for NGCI development.

Press Release: https://www.unco.edu/news/newsroom/releases/unc-leads-grant-students-learn-sustainability.aspx

Context – Concept Inventories for Assessing Student Learning

There is growing support and interest in interdisciplinary environmental education that integrate concepts and disciplines in addition to providing varied perspectives. Most of these EPs do not incorporate systematic evaluation and assessment, and especially non-summative evaluations. One challenge to evaluation is finding reliable and valid assessments geared toward interdisciplinary concepts. There is a need to assess student learning as well as rigorous evaluation of EP practices, especially of complex synthesis concepts. Concept inventories are validated assessments designed to evaluate student understanding of concepts and learning, but traditionally have focused on individual concepts rather than integration of concepts. Regardless, no valid and reliable concept inventory currently exists for interdisciplinary EP content. Without a concept inventory or similar assessment instrument, we are not able to evaluate how students’ ability to understand complex EP concepts, nor can we confidently evaluate and compare learning within and between EPs nationwide. Development of the NGCI using constructed response (CR) items and associated automated scoring models would allow evaluation of the relationships between foundational concept understanding and integration of those concepts, as well as more nuanced understanding of student comprehension of complex interdisciplinary concepts.

Collaborators

Collaborator: Kevin Haudek, Automated Assessment of Constructed Response Lab, MSU
https://beyondmultiplechoice.org/

CO-PI: Steve Anderson, Professor of Earth Sciences and co-developer of the Geoscience Concept Inventory
https://www.unco.edu/nhs/earth-atmospheric-sciences/faculty-staff/anderson.aspx
The Geoscience Concept Inventory

Co-PI: Shirley Vincent, Vincent Consulting
https://www.vincentconsult.com/

This research includes partnerships between universities for design and development of an assessment tool that can be used to evaluate and improve student learning. Universities across the United States have joined our group to provide materials and participate in student learning research. 

IUSE_Collaborators_v2

Grants

NSF IUSE-EHR Collaborative Research: A Next Generation Concept Inventory using Constructed Response for evaluating student knowledge of complex systems in environmental education. (funded 2020, IUSE:EHR Award # 2013373)
Role: PI

NSF INFEWS RCN: Cultivating a National Collaborative for Research on Food, Energy, and Water Education (NC-FEW). (funded 2018; ECR-EHR Core Research Award #1856040)
Role: Subaward.

Publications

(in development). Alignment among environmental programs in higher education: What are we covering in introductory courses? Journal of STEM Education

OTHER EVIDENCE BASED STEM EDUCATION RESEARCH

As part of a National Science Foundation funded program called Widening Implementation & Demonstration of Evidence-Based Reforms (WIDER), I have been testing the efficacy of classroom activities in achieving teaching and learning goals. Specifically my project responded to a recently revealed disconnects between environmental science instructors’ learning goals and student learning. Under my
supervision and direction, a STEM undergraduate with graduate school aspirations is analyzing survey data from 400 undergraduate students and 8 laboratory instructors. We currently have one publication in the Athens Journal of Education   (1 ) and a second in the Journal of College Science Teaching (2).

(1) Schwebach, J.R., Gerasimova, D., Luther, D.A., Verhoeven, A.B., Davis, C., Gostel, M., Romulo, C., Schreffler, L., Seshaiyer, P., Nelson, J.K. 2016. Advancing Graduate Education and Faculty Development with Discipline Based Education Research and the SIMPLE Framework: Design Memos in Biology for Active Teaching. Athens J. of Education

(2) Romulo, C., Raoufi, A., Largen, K.,and Schwebach, J.R. 2018. Using peer review to improve lab report assignments. American Biology Teacher. Vol. 80 No. 4 (pp. 301- 304) DOI: 10.1525/abt.2018.80.4.301