Background and Intent of the Tool:
This online computer-based decision making tool was designed by CCSSO’s Technical Issues in Large Scale Assessment (TILSA) consortium during 2006 with the assistance of Pacific Metrics. State testing directors and assessment personnel with experience in computer-based testing provided guidance and input to the tool’s content.

The tool is intended to assist state department of education personnel in making decisions and in implementing plans to move from a paper and pencil testing environment to a computer-based testing environment.

Directions for Use
The tool is designed to be simple and convenient to use.

Users must first register and create a log-in id and password. After that, the user should go into the tool and work through the sections in order. These sections are:

1. Starting Points
2. Goals
3. Decision Points:
Players and the Environment
Assessment and Psychometrics
Technology
4. Risk Analysis
5. Cost/Benefit Analysis
6. Implementation Plan

It is not necessary to complete the entire content of the tool in one sitting. Nor is it necessary to complete an entire section. The program will save the user’s entries as long as the same log-in id is used each time. The user may also go back and change or update responses as appropriate. In addition to the program entries, the user may wish to keep a separate list of comments as additional documentation of their situation.

At the completion of each of Sections 1 through 5 a summary report is generated based on the user’s input. That report can be printed out or accessed online as needed. The summary report will help users make decisions about what steps to take in planning and implementation related to each topic. Users considering a computer-based testing environment for more than one assessment component e.g., a formative assessment and an end-of-course assessment, may find it helpful to create a separate log-in for each type of assessment.

Section 6: Implementation Plan can be completed online, or alternatively printed out and completed by hand or used as a guide to develop a comprehensive implementation plan for the user’s particular situation.

Annotated Bibliography and Literature Review
CCSSO also conducted a literature review of documents and studies related to computer based testing. A description of each article is available in .pdf format by clicking on the article’s title; to go to the web and access an entire document click on the hyperlink (where available). Note that if you use a hyperlink, you may exit the CBT tool environment. The properties of the site you reach may prevent you from using the back button to re-access the tool. If this happens, you may need to re-access the CBT tool manually.

Literature Review

1:ARTICLES AND REPORTS ON DECISION CONSIDERATIONS

Bennett, R.E. (2002). Inexorable and inevitable: The continuing story of technology and assessment. Journal of Technology, Learning, and Assessment, ı(ı). Available from http://www.jtla.org.

Editors, Education Week on the Web (2003), Legal Twists, Digital Turns, 2003 Editorial Projects in Education, Vol. 22, number 35, page 11-14,16. Available online at http://counts.edweek.org/sreports/tc03/article.Cfm?slug=35impact.h22

Editors, Education Week on the Web (2005), Tech’s Answer to Testing, 2003 Editorial Projects in Education, Vol. 22, number 35, page 8,10. Available online at http://counts.edweek.org/sreports/tc03/article.cfm:slug=35exec.h22

Row, Neil C., (2005), Cheating Online Student Assessment: Beyond Plagiarism

Trotter, Andres, (2005), Published May 23, 2001. Testing Computerized Exams. Education Week on the Web.

Wall, Janet E., (2000) Technology-Delivered Assessment: Diamonds or Rocks? ERIC/CASS Digest. ERIC Identifier: ED446325, ERIC Clearinghouse on Counseling and Student Services Greensboro, NC.

2. ARTICLES AND REPORTS ON DESIGN CONSIDERATIONS

CTB/McGraw-Hill, April, 2003. Computer-Based Testing – Issues and Considerations.

CTB/McGraw-Hill, April 11, 2003, The Computer-Based or Online Administration of paper-Pencil Tests, Journal of Technology, Learning, and Assessment, 2(1).

Quenemoen, R. F., & Lehr, C. A. (2002). Access to computer-based testing for students with disabilities (Synthesis Report 45). Minneapolis, MN: University of Minnesota, Thompson, S. J., Thurlow, M. L., National Center on Educational Outcomes. Retrieved [January, 2006] from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Synthesis45.html

(RAND Education)Hamilton, Laura S., Klein, Stephen P., and Lorie, William (2000), National Science Foundation, Division of Elementary, Secondary and Informal Education grant ESI-9813981. http://www.rand.org/pubs/issue_papers/IP196/

Stover, Del, (2005), Computer-Graded Essays Gaining a Foothold, 2005 National School Board News, National School Boards Association

Thompson, S. J., Thurlow, M. L., Quenemoen, R. F., & Lehr, C. A. (2002). Access to computer-based testing for students with disabilities (Synthesis Report 45). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved January, 2006 from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Synthesis45.html

U.S. Department of Education, National Center for Education Statistics. NAEP Validity Studies: Implications of Electronic Technology for the NAEP Assessment. NCES 2003–16, by Richard P. Durán. Project Officer: Patricia Dabbs. Washington, DC: 2003

3. ARTICLES AND REPORTS ON RESEARCH CONSIDERATIONS: PAPER AND PENCIL VS. ONLINE

Amoroso, Donald L., (2004). Appalachian State University, Boone, North Carolina. Use of Online Assessment Tools to Enhance Student Performance in Large Classes. Proc ISECON 2004, v21 (Newport): §3142 (refereed) © EDSIG, pages 1-8.

Brown , Carol J., (2002), Online Classroom Assessment Project Evaluation Report #1, Prepared for the Bill & Melinda Gates Foundation.

Harcourt Assessment, Inc., (2004), Wang, Shudong, Young, Michael J. , and Brooks, Thomas E., Harcourt Technical Report: SDRT 4/SDMT 4 Administration Mode Comparability Study/Online or Paper: Does Delivery Affect Results?.

Loggia, J., Glasnapp, D. R., Yang, X., & Poggio, A. J. (2005). A comparative evaluation of score results from computerized and paper and pencil mathematics testing in a large scale state assessment program. Journal of Technology, Learning, and Assessment, 3(6). Available from http://www.jtla.org

Pearson Educational Measurement., Paek, Pamela, (2005), Recent Trends in Comparability Studies Research Report. Available: http://www.pearsonedmeasurement.com/research/research.htm

Poggio, J., Glasnapp, D. R., Yang, X., & Poggio, A. J. (2005). A Comparative Evaluation of Score Results from Computerized and Paper and Pencil Mathematics Testing in a Large Scale State Assessment Program. Journal of Technology, Learning, and Assessment, 3(6). Available from http://www.jtla.org

Pommerich, M. (2004). Developing computerized versions of paper-and-pencil tests: Mode effects for passage- based tests. Journal of Technology, Learning, and Assessment,2(6).Available from http://www.jtla.org

Wresch, W. (1984) The Computer in Composition Instruction: A Writer’s Tool. Urbana, IL, National Council of Teachers of English.

4. ARTICLES AND REPORTS ON RESEARCH CONSIDERATIONS: SCORING

Zhang, Yuli (Lilly), Powers, Donald E., Wright, Wendi, & Morgan, Rick. Applying the Online (OSN) to Advanced Placement Program ® (AP©) Tests. Educational Testing Service (ETS), April, 2003.

5. ARTICLES AND REPORTS ON RESEARCH CONSIDERATIONS: EFFECTS OF STUDENT FAMILIARITY WITH TECHNOLOGY

Educational Testing Service (ETS), National Center for Education Statistics. (2003), Sandene. Brent., Bennett, Randy., Kaplan, Bruce., and Braswell, James. Online Assessment in Writing.

National Board on Education Testing Public Policy, ), Russell, Michael, (1997), Boston College Massachusetts, Testing Writing on Computers: An Experiment Comparing Student Performance on Tests Conducted via Computer and via Paper-and-Pencil, Volume 5 Number 3 , Available: http://epaa.asu.edu.

O’Dwyer, L.M., Russell, M., Bebell, D., & Tucker-Seeley, K.R. (2005). Examining the Relationship Between Home and School Computer Use and Students’ English/Language Arts Test Scores. Journal of Technology, Learning and Assessment, 3(3). Available from http://www.jtla.org

Russell, Michael., (1997), Testing On Computers: A Follow-up Study Comparing Performance On Computer and On Paper, Volume 7, Number 21. Education Policy Analysis Archives Available: http://epaa.asu.edu.

6. ARTICLES AND REPORTS ON USE OF COMPUTERS IN INSTRUCTION

(ERIC) Kerka, Sandra., Wonacott, Michael E., Grossman, Cheryl., and Judy Wagner. (2000) Clearing House on Adult, Career and Vocational Education, No. Assessing Learners Online – Online Assessment: Principles and Practices, ED-99-CO-0013.

Goldberg, A., Russell, M., & Cook, A. (2003). The effect of computers on student writing: A meta-analysis of studies from 1992 to 2002. Journal of Technology, Learning, and Assessment, 2(1). Available from http://www.jtla.org

Liang, Xin & Kim Creasy (2004). Classroom assessment in web-based instructional environment: instructors’ experience. Practical Assessment, Research & Evaluation, 9(7). Retrieved August 12, 2005 from http://pareonline.net/