UIC Professor teaches students to collect quantitative data in measurement course

The online Measurement, Evaluation, Statistics, and Assessment (MESA) program at the University of Illinois Chicago (UIC) equips students with the skills to conduct research to make informed, data-driven decisions. Within the program, the measurement courses offer essential knowledge, empowering MESA students to design studies and determine how data will be collected.

As an introduction to measurement in the MESA program, the EPSY 546: Educational Measurement course familiarizes students to the fundamental concepts and methods of measurement, such as classical test theory, which is a framework in psychometrics, item analysis, factor analysis, plus the major extensions and alternatives to classical test theory, such as generalizability theory and item response theory.

Instructed by UIC Professor Yue Yin, her goal of this perquisite is to offer students a practical approach that is useful for researchers and practitioners who need to collect quantitative data. She emphasizes that regardless of the industry, if a professional is looking to collect and interpret data, this course will help students choose the correct research instrument to provide reliable and valid information.

Learn more about the EPSY 546 course from Dr. Yin, including the computer programs MESA students will become proficient in.

Why is it important for MESA students to be introduced to the concepts and methods used in measurement?

Measurement is everywhere. Measurement is a fundamental building block for research or practice in both natural science, social science, and daily life. We need to ensure that the measurements have high reliability and validity so that any conclusions are based on reliable and accurate data. Examples where measurement is used, includes:

• To determine whether an educational program is effective, we need to measure students’ learning outcomes, satisfaction, and job placement of graduates.
• To compare different products, we need to define the evaluation criteria and measure the indicators for those criteria.
• To evaluate whether teachers, administrators, and employees are effective, we need to define the constructs and measure them.
• To examine how human behavior impacts species in nature and predict future trends, we need to determine the variables and collect data about them.

What will students learn in the EPSY 546: Educational Measurement course? Are there any projects or real-world applications?

EPSY 546 is a practical course that is useful for any researchers and practitioners who need to collect quantitative data. As an introductory measurement course, we will have a variety of practical hands-on tasks for students to complete, such as calculating test-retest reliability, internal consistency of a survey, inter-rater reliability, and interrater agreement. Additionally, students will examine the evidence for criterion validity, predictive validity, and construct validity. They will also need to apply their measurement knowledge and skills to analyze realistic problems, consider whether threats to measurement reliability and validity might exist, and determine how to address these issues. Students will also learn some frequently used terminology or procedures in measurement, such as stanine score, grade-equivalent score, item bias, and standard setting. They will gain a conceptual understanding of item response theory, which will prepare them for their item response theory course (EPSY 551).

MESA students will use four different computer programs in the EPSY 546 course: Excel, SPSS, GENOVA, and IRTPRo. Why are these programs important when studying measurement?

Excel: It is very helpful for students to understand critical concepts by doing hand calculations. Compared to calculators, Excel can help deal with calculations involving many observations, which is often encountered in measurement. For example, when examining the correlation between a group of students’ SAT test scores and GPA, Excel allows us to simulate how computer software runs the analyses. Doing calculations in Excel can help MESA students understand the mathematical formulas and the concepts behind them.

SPSS: It is widely used statistical software that is handy and practical. It can be used to run frequently used tests for reliability and validity, such as inter-rater reliability and agreement, internal consistency, predictive validity, criterion validity, and factor analysis for construct validity.

GENOVA: This software is used for generalizability studies. Generalizability theory can handle multiple sources of measurement error simultaneously.

IRTPro: This software is used to demonstrate how item response theory works. It introduces students to this modern measurement method frequently used in the testing industry.

What is classical test theory, and what are the benefits of learning psychometrics in the EPSY 546 course?

Classical Test Theory (CTT) is a framework in psychometrics that provides a way to understand and analyze the reliability and validity of psychological and educational measurements. It is based on the idea that observed scores on a test consist of true scores and error scores.

Observed score (X) = True Score (T) + Error Score (E)

In CTT, the two most important concepts are reliability and validity. Reliability is the extent to which a test produces consistent and stable results over time and across different conditions. Validity refers to the extent to which a test measures what it is intended to measure. In CTT, various methods can be used to estimate or evaluate reliability and validity. Although modern test theory has advanced and improved ways to study reliability and validity, CTT remains the foundation for measurement, serving as a prerequisite to understand more advanced psychometric methods.

How can students apply their measurement skills from the EPSY 546 course to a future job role?

No matter what you do, if you collect data and/or use data to make any decisions, either for research or practical purposes, you need to make sure that the measurement instrument you use will provide reliable and valid information. Otherwise, you may arrive at misleading, inappropriate, or even incorrect conclusions. The EPSY 546 course familiarizes students with essential concepts and measurement methods, guiding them to make those informed decisions.