This study investigates the score comparability of paper-based and computer-based testing of reading comprehension achievement tests. More precisely, it examines the new test mode effect, i.e. computers, on both sides of the testing process, i.e. the test and the test taker, through examining the effect of the new testing mode on the essential unified concept of test score validity which encompasses reliability, concurrent validity, and truct validity and investigating the interactive effect of variables such as prior computer familiarity, computer attitudes, and pre and post testing mode preference with students' performance on the new testing mode.