Home : Report Link : 95 Testing Center Report
The Testing Center's mission is to provide assessment and evaluation support through the collection and processing of test data, creation of assessment instruments, and the lending of measurement expertise to constituencies throughout the campus community. Our vision is to provide integrated assessment and evaluation information in ways that will continuously improve IUPUI.
All Testing Center activities incorporate the following values:
Placement testing refers to the activities related to the administration of IUPUI-sponsored placement tests. These tests are administered with the express purpose of enrolling students in classes that correspond to their ability level. As the numbers of students taking the respective IUPUI placement tests continues to increase1, so do the improvements made at the Testing Center. Apart from the hiring of new staff, several significant changes were implemented to enhance the efficiency and economy of placement testing procedures at the Testing Center. With regard to placement testing, the major changes include the cutover to computerized adaptive testing (CAT) math, individualized testing in chemistry, and the transfer of ESL (English as a Second Language) testing responsibilities to the ESL Program. A brief description of each change is presented in turn. The activities performed in this area are related to IUPUI's internal goals for quality and collaboration. The IUPUI Microcomputer Testing Facility was featured in the videotape Assessment 101: A Videoguide to Designing and Implementing an Assessment Program.
Computerized Adaptive Math Test.
The computerized adaptive test in mathematics was developed, piloted, and implemented during the past year. Development of the test was completed in Spring 1995 and piloting for the new adaptive math test was completed in early summer. Full implementation of the new math placement test occurred in late October. The choice to convert the conventional computerized math test into an adaptive procedure was mainly to achieve increased efficiency in the use of both students' time and test items. By efficiency we mean obtaining the same accuracy as a linear test while using fewer test items. The implementation of CAT procedures at the Microcomputer Testing Facility also offers a valuable demonstration of how incoming students might be better served through use of computer technology and advanced measurement techniques such as Item Response Theory (IRT). Furthermore, the computerized adaptive math test should facilitate the needed improvements in the observed correlation coefficients between the placement test scores and the math outcome measures. The new adaptive math test fits the instructional objectives for the math sequence more completely than its pencil-and-paper counterpart. Figure 2 shows an example of a hypothetical math problem presented by the computerized adaptive math test system, HyperCAT.
Chemistry Placement Examination.
Effective Fall 1995, the Testing Center expanded its placement testing operations by introducing the chemistry placement exam. The purpose of the exam is to assist with placement of students in the college-level general chemistry courses. The overall aims of the chemistry placement exam are: (a) to improve student progress in chemistry courses offered by the Department of Chemistry at IUPUI, (b) to meet the requirements of the students' majors in a more timely manner, and (c) foster greater retention and success.
The chemistry placement exam was initially administered as a group paper-and-pencil test. On October 21, 1995, the chemistry placement exam was converted into an individually administered paper-and-pencil test. The test administration procedures for the chemistry exam are now similar to those currently in use for the Reading placement exam. Therefore, the chemistry placement exam is administered at the Microcomputer Testing Facility as an individualized test with computerized test instructions. Students who sit for the chemistry placement exam now receive copies of the test reports upon completion of the exam. A total of 198 students took the chemistry placement exam in 1995.
Transfer of ESL Testing Responsibilities to the ESL Program.
Effective November 16, 1995, all English as a Second Language (ESL) testing operations were transferred from the Testing Center to the ESL Program. This transfer was made part in possible by a grant from the IUPUI Assessment Committee2 that permitted two of the three assessments to be converted to a computerized form. The model underlying the grant was designed to demonstrate that the Testing Center can act in a consulting fashion and help units independently administer their own exams using their own resources. A similar model was applied to the development of the Nursing Measurement Exam described in the next section, and will give units the flexibility in deciding how they want to spend their resources.
As a consequence of this development, all ESL testing responsibilities including test registration, arrangements for test sites, test administration/proctoring, scoring and uploading of students' ESL test scores on IUTS are currently being handled by the ESL Program. From June to November 1995, the Testing Center had administered ESL tests to a total of 166 students.
IUPUI Nursing School CAT Measurement Assessment Exam.
A computerized adaptive test in arithmetic was developed, piloted, and implemented in late October 1995 for the School of Nursing. Thus, the results of the Nursing Measurement Assessment (NMA) instrument are used to identify conceptual weaknesses for incoming nursing students. The test results are also used by the School of Nursing to tutor students in need of remediation courses. The test was designed to serve both as an assessment for overall measurement skills appropriate for nursing students, and also provide diagnostic information regarding individual areas of strength and weakness.
The NMA test also serves as a placement exam in that it reflects the program objectives of the School of Nursing and provides information concerning requisite skills essential for success within the nursing academic programs. The School of Nursing is currently using the IBM version of HyperCAT, a computer program developed and supported by the Testing Center. This assessment activity was supported by Perkins grant money.
Validity Study of the IUPUI Placement Test Scores.
A validity study of the IUPUI placement test scores in English, reading, and math was conducted and the report was completed in October, 1995. Although the report indicated that the current validity coefficients for the IUPUI placement tests were generally weak, several positive developments and recommendations were initiated on the basis of the investigation.
Math.
To improve the validity coefficients for math, the new math placement exam utilizing computerized adaptive testing procedures was implemented to replace the conventional math test. This was actually accomplished in October 1995. The choice to convert a computerized test into a CAT was mainly to achieve increased efficiency in the use of both students' time and test items. The other recommendations made for math include: (a) a revision of the current math placement cut-off scores, and (b) a reminder that decisions regarding math placement should include a review of students' previous academic (math) record and not be made on the basis of placement test scores alone.
Reading.
Because of the low correlations between the Nelson-Denny (N-D) Form E and N-D Form F, a new reading placement test utilizing CAT procedures needs to be developed to replace the current Nelson-Denny Reading Test Form F. This is important because previous pilot information on the new reading test designed to function in a CAT environment has produced promising results. Meanwhile, a non-adaptive version of the reading test is scheduled for implementation at the Testing Center early this year.
English.
Because a comprehensive validity study for English placement test scores is still pending, no recommendations were made with respect to the English placement exam in the 1995 placement test validity report. They did implement a new rating scale which was piloted this past year and will permit the calculation of validity coefficients.
An overall recommendation from the placement test validity report was that the Registrar create a "barring mechanism" to enforce compliance with placement recommendations. The bar would prevent a student from enrolling in a class that was inappropriate for the student based on placement test scores unless there was a special request by the concerned department.
Training of Work Study Employees.
On-the-job training and formal training sessions continue to be provided to the work study students in order to facilitate professionalism, thoroughness, and efficiency among the Microcomputer Testing Center (MTF) staff. While one full-time and one part-time clerical staff member were added to the Microcomputer Testing Facility during late summer of 1995, there is still a heavy reliance on quality work-study students to staff the facility. Several topics (e.g., new placement testing procedures, test registration or scheduling of students for testing, customer service, generation of placement test reports, test security, confidentiality of information, Testing Center emergency procedures, etc.) are usually covered during formal training sessions. The topics are selected on the basis of the needs of the Testing Center proctors and receptionists and/or on the basis of a placement testing exit survey conducted at the conclusion of placement test administration. The MTF survey was designed to solicit information such as expertise and disposition of the test proctors, examinees' perceptions of the testing situation, examinees computing background, and so on. In addition to providing suggestions for training topics, information from the MTF exit survey is used internally to monitor proctor performance improvements.
MTF Satisfaction Survey.
The placement testing exit survey was designed to solicit information about students' computing background, their perceptions of the testing experience, expertise and disposition of the proctors, and some demographic characteristics. The information obtained from the survey is used internally to monitor proctor behavior, suggest training topics, and specify possible technical innovations and recommendations that might improve the efficiency and effectiveness of the testing operations. The survey also provides valuable information for external communication purposes with other service units by demonstrating how the testing operations fit in with other IUPUI enrollment activities.
The following is a summary of the survey results for data collected between May to December 1995. The results are based on a total of 1752 respondents. Overall, the results of the present survey suggest that most students have a very favourable disposition towards our computerized testing operations. For instance, approximately 90% (or 1575 of 1752) of the students found the behavior of our MTF staff to be courteous or very courteous. In addition, a majority of the students (94%) reported that our staff had sufficient or very extensive knowledge of computers. In contrast, only 6% and 1% of the students, respectively, found our MTF staff to have inadequate computer knowledge and somewhat impatient or rude. To correct this situation and improve the competence and customer relations of our employees, bi-weekly group meetings and/or training sessions have been implemented for our work-study students. Attendance at the work-study meetings and training sessions is mandatory, and is one of the requirements for continued employment at the Testing Center.
Regarding students' experience in using computers, the 1994 survey results indicate that about 3% of the respondents (in contrast to 2% in 1995) had no computer experience at all. On the contrary, approximately 29% of the 1995 exit survey respondents reported having "a great deal" of computer-related experience, this number is up from 20% in 1994. Nearly 56% and 51% of the 1994 and 1995 survey respondents, respectively, reported having some experience in using computers. Perhaps this might help explain why there is currently an observed increase in the proportion of students who reported feeling quite at ease in taking the computerized tests. Although a majority of students now report having adequate knowledge of computers, there is only a small (but noticeable) proportion of students who report feeling very confused with computerized testing. Thus, the development and implementation of improved sets of computerized instructions (e.g., the adaptive math instructions currently being implemented at MTF) should make the students' testing experience to be pleasant, less confusing, and "user friendly". In addition, thorough training of our test proctors in test administration procedures as well as proctors' vigilance or alertness during test administration, can help to minimize or eliminate unnecessary examinee confusion.
Student Phone Survey.
In addition to the placement testing survey, the Testing Center administers a phone survey to students who call to register for placement tests. Placement testing reservations can be made monday-friday from 8-5. The phone survey was designed to obtain specific information pertaining to (a) how easy it was for students to call the Testing Center for test registration, (b) what (if any) problems were encountered by students in calling the Testing Center, (c) how long it took the receptionist to register the student, (d) reason for a lengthy phone conversation (if applicable), (e) number of students requesting rescheduling, and (f) students' reasons for rescheduling the placement test(s) (if applicable). The overall results indicate that most students were able to contact the Testing Center in a relatively easy manner and with no problems at all. Of those who reported any problems (10%), the vast majority (80%) said that obtaining our phone number was difficult. This is curious since we are listed in the undergraduate bulletins produced by the various academic units, in the materials produced by UEC and the Admissions Office, and are displayed along with other enrollment management units in the yellow pages. We recently listed contact numbers on our World Wide Web site which might be helpful to some. More importantly, however, we plan on switching our reservation activities to the Coordinated Student Entry Services in mid-1996 where one phone number should allow the student to accomplish multiple tasks. In addition, for the majority of the students, the test registration process was conducted in a quick and efficient way (i.e., without unnecessary delay).
External testing refers to the part of the IUPUI Testing Center that administers tests which are not part of the placement testing program. Such tests include Independent studies tests and exams that are done in behalf of profit and non-profit testing companies. This responsibility area also includes scanning, course evaluation activities, and forms design. This area is linked to IUPUI's goals of quality and collaboration.
Strong Interest Inventory Testing.
In 1995, the Consulting Psychology Press computerized the Strong Interest Inventory, a vocational interest inventory that is utilized primarily by the School of Business. The IUPUI Testing Center was one of the first organizations to use the new version of this assessment. The MTF tested 750 students from both the School of Business and the Division of Continuing Studies. The new assessment technology was well-received both by students and their professors. Students were able to take the test and receive immediate feedback. Prior to the conversion, the tests were sent off to Minnesota to be scored, resulting in delays as long as a month. This coming year, we hope to offer the computerized Myers-Briggs Type Indicator (MBTI) as a supplement to the Strong.
In conjunction with CPP, the Testing Center also co-sponsored a campus-wide workshop geared for counselors and advisors in the area of vocational interest and personality assessment. The workshop was attended by 14 staff members and was part of our continuing series in staff training.
Independent Studies Testing.
In 1994, we computerized the administration of the independent studies and began administering the tests in the MTF lab. We have continued to give the Independent Studies exams from IU as well as from other institutions in the MTF lab. We gave approximately 750 of these exams over the past year. This operation brought in approximately $10,500 in soft money last year. As a consequence of this extra-mission activity, we were able to continue providing the students with more flexible hours and better service by using the MTF lab.
In the spring of 1994, we moved the Independent Studies testing to computerized administration. In late 1994, we implemented an exit survey similar to the one that is given to students taking placement tests. This quick survey queries students about the manner in which they were greeted, the checking of identification, and how quickly they were tested after arriving at the MTF. This information is used to both provide feedback to our work-study proctors and our clients in the Division of Continuing Studies. For 1995, out of a sample of 131 responses, 118 students felt that they had been greeted in a friendly manner. One hunded and eighteen students reported they were able to start their test within five minutes of arriving at the facilty. One hundred ten students felt that they were given adequate instructions on how to use the computer prior to starting the test. Eighty-four percent of the students were comfortable or neutral about using the computer for testing activities. The students taking these tests tend to be comparable with our clientele for placement testing with a median age in the 20-24 bracket. Sixty-one percent of these students were female and 87.8% were Caucasian.
National and State Testing.
In addition to our internal testing operation, we also administered state and national tests for the benefit of students and staff. Over 6,000 students took exams under this program which serves the needs of the students and professional individuals. Tests administered under this program include the following:
Scan Forms Design.
The Scanning Office of the Testing Center offers scoresheet design on a cost-recovery basis. This service was implemented to permit faculty and staff to inexpensively design their own survey and measurement instruments in scannable form. In 1995, this service was used by both academic and administrative units and generated approximately $10,000 of income. We are currently collaborating with Printing Services for remote high speed printing so that scoresheets can be produced at a more reasonable rate with an even quicker turnaround time.
Course Evaluation.
The SET program was developed in 1994 with the School of Public and Environmental Affairs using their 5 IU campuses. In Fall 1995, we also added the School of Medicine to the program. Course evaluation activites for the School of Medicine covers 6 campuses throughout the state. We will devote more time this coming year to speeding up the reporting of processed data to the units. In addition, our goal is to be able to offer customized reporting that would be generated by the unit itself. Approximately $9,000 of income was attributable to this area.
Scanning.
In 1993 we proposed moving scanning operations from OIT to the Testing Center. Even though the plan was well received, it was not funded. However, scanning has become an increasingly important part of External Testing Operations. Last year we scanned approximately 60,000 documents with an income stream of approximately $9,000.
Development refers to two components of Testing Center activity. One component focuses on test development and is staffed by two graduate student research assistants. The second component centers around the development of computer-based tests, supplemental multi-media, and data collection mechanisms. This section is staffed by a .75 FTE programmer and two work-study students. Both components are linked to the IUPUI goals of quality and collaboration.
General Support.
The computer development is tasked with responding immediately to problems that might arise from any one of the operational areas of the Testing Center. In addition to providing general software support, the team also addresses hardware and communications problems as well.
Adaptive Math Test.
This is on-going programming and research activity which uses HyperCAT, a computerized adaptive testing package. Both Macintosh and IBM versions of the Adaptive Math Test are supported. Planned improvements include the development of testlets or "mini-tests", a mechanism to better distribute item selection, and a screening questionnaire developed for initial item selection. A survey is being constructed in order to receive feedback from our students on the success and/or short-comings of the test.
Nursing Measurement Assessment Test.
This test has been installed on the nursing network in the Nursing building. This was developed to be used as an assessment on the needs of the students already accepted into the nursing program, and is not being used as a replacement of the required placement tests given in the Testing Center.
Reading Test.
In addition to computerizing the math placement test, we are in the process of computerizing the reading assessment test also. By using advanced computer software packages such as LXRTest and C-Tree, the reading test is being completed to make the testing process less tedious and more comprehensible for clients.
HTML Programming.
A recently added capability of the development team is the formulation of Web pages for the both the Testing Center and the National Council of Measurement in Education (NCME) server. Development for the NCME server is a reimbursed activity done on behalf of that non-profit educational organization. The Testing Center Web Server can be reached at
http://www.assessment.iupui.edu/testing/tc.html.
Teaching, Research, and Service
In this final section, we list some of the external activities that speak to our contributions beyond the student service mission of the Testing Center.
Teaching
Mark Shermis taught two courses for the Department of Psychology: Tests & Measurement (Psych B307) and Clinical Rehabilitation Psychology Assessment I (Psych I664).
Research
(Articles)
Shermis, M.D., Falkenberg, B., Appel, V.M., & Cole, R.W. (in press). Construction of a faking detector scale for a biodata survey instrument. Military Psychology.
(Articles Under Review)
Shermis, M. D. (1995). Using computerized adaptive testing for college mathematics. Manuscript submitted to NACADA Journal.
Shermis, M. D., & Chang, S. H. (1994). The use of IRT to investigate the hierarchical nature of a college mathematics curriculum. Manuscript submitted to Educational and Psyhological Measurement (2nd Review).
Shermis, M. D., Fulkerson, J., & Banta, T. W. (1995). Computerized adaptive math tests for elementary talent development selection. Manuscript submitted to Applied Measurement in Education.
Shermis, M. D., Stemmer, P. M., & Webb, P. M. (1995). Computerized adaptive skill assessment in a statewide testing program. Manuscript submitted to the Journal of Educational Computing Research.
Shermis, M. D., Wolting, M., & Lombard, D. (1995). Development of a Computerized Test for College Reading Placement. Manuscript submitted to Journal of Developmental Education.
Shermis, M. D., Webb, P. M., & Mzumara, H. R. (1995, December). An assessment of the concurrent validity and reliability of the Merkler Style Preference Inventory (MSPI). First review, Journal of Counseling Psychology.
(Papers)
Shermis, M. D. (1995, April). Ensnaring NCME through the World Wide Web. Paper presented at the annual meetings of the National Council on Measurement in Education.
Service
Shermis, M. D. (1995, January). Constructing better assessment instruments. Invited workshop for the faculty of Nichols State University Thibadaux, LA.
Shermis, M. D. (1995, March). Tunneling for treasure: Measurement Resources on the Internet. Presentation given at the annual meetings of the Indiania Association of Institutional Research, West Lafayette, IN.
Shermis, M. D. (1995, June) Computerized adaptive testing for college mathematics. Presentation given to the Department of Computer Science, IUPUI, Indianapolis, IN.
Shermis, M. D. (1995, November) Assessment technology support. Workshop conducted at the Assessment Conference, Indianpolis, IN.
Shermis, M. D., & Kiger, B. (1995, August). Using LXRTest to enhance testing practice. Presentation given to the School of Nursing, IUPUI, Indianapolis, IN.
Consultant Presentations
Mark Shermis gave lectures last year in test construction, statistics, research design, and I/O psychology for the Association for Advanced Training in the Behavioral Sciences (Ventura, CA) in various cities throughout the country.
Journal Reviewer
Mark Shermis was a journal reviewer for the following scholarly publications:
Journal of Educational Measurement (2 manuscripts)
Educational Measurement: Issues and Practice (1 manuscripts)
Conference Reviewer
Shermis was also a reviewer for conference papers for the following organizations:
American Educational Research Association (AERA)
National Council on Measurement in Education (NCME)
Mark Shermis Committee Work
(National Committees)
Chair, NCME Committee on Electronic and Alternate Communications
(IU System-wide Committee)
Joint Sub-committee on Learning
(University-wide Committees)
Academic Affairs Committee
Academic Policy and Planning Committee
Administrative Council
Gopher Task Force Committee
Faculty Evaluation Committee
Professional Communications Committee
Program Review and Assessment Committee
Testing Center Advisory Committee
(School of Science Committees)
Undergraduate Curriculm Task Force
(Department of Psychology Committees)
Methodology Group
(TQM)
Executive Steering Group for Student Enrollment Support Systems
Sponsor, Internal Recruitment Action Team
IMIR/Systems Management Steering Group
Betsy Kiger and Howard Mzumara Committee Work
In addition, both Betsy Kiger and Howard Mzumara are members of TQM Action Teams. Betsy is part of the Management Informtion Action Team while Howard sits on the Entry Process Action Team. In addition, Howard is a member of the Strategic Directions Planning Committee on Quality (Study Group 4).
1 The total numbers of students who sat for the respective IUPUI placement tests in 1995 are as follows:
Chemistry: 198
English: 5766
Foreign Languages: 114;
Math: 6023
Reading: 5492.
2 Now called the Program Review and Assessment Committee.
