A series of studies were conducted to investigate the validity of a group oral test as administered at a university in Japan. The research investigated the sources of test score variance and asked to what extent candidates' group discussion test scores were reliable measures of their proficiency. The research also investigated to what extent the discourse in the group discussion test task was generalizable to discourse on other kinds of group oral tasks. Three studies were conducted. Study 1 was a repeated measures design in which 113 candidates were tested on the discussion task on two successive occasions, and the consistency of their test scores was analysed. In Study 2,188 candidates were tested on the group discussion task and other estimations of their proficiency were additionally collected. The amount of talk candidates produced in the test was analysed in order to find whether this affected the scores they received. Study 3 was also a repeated measures design in which 64 candidates were tested on three different group oral tasks: - II -discussion,c onsensus-reachinga, nd picture difference. The candidates' test scores and discourse were analysed to find how generalizable their performances were across the task types. Additionally, the candidates' reactions to the tasks were collected through questionnaires and interviews. The findings show that contextual factors such as discussion topic, group interlocutors, and task characteristics led to significant variation in discourse and test scores across test occasions. The findings further show that the group discussion task elicited only a limited range of interactions, and therefore may not be representative of the language use domain. The implications are that test designers can elicit a wider range of interactions from candidates in group tests by introducing task characteristics which increase goal orientation and reactivity. This also lessens the impact of discussion topic and inter-personal group dynamics on candidates' performances