1、英语测试 专四口语考试A Critique to the Validity and Reliability of Oral Test in TEM4 in 2012June 2014Course Name: Language TestingLecturer: Dr. Liu MinStudent ID&Name: A Critique to the Validity and Reliability of Oral Test in TEM4 in 20121. IntroductionListening, speaking, reading, and writing are the 4 basi
2、c skills of learning a foreign language. Listening, reading, and writing, these 3 language skills have already got high attention in TEM4 and TEM8. However, the oral test, representing the speaking skill, is hard to work on effect. As a subjective exam, reliability assess always restricts the develo
3、pment of oral tests. Reliability assess and validity assess are of equal importance in examination theory. TEM 4 Oral English Test consists of three parts: retelling a story (listen to the story twice and retell for 3 minutes), talking on a given topic for 3 minutes and role-playing for 4 minutes. T
4、he scoring method synthesizes the advantage of speaking test of TOEFL and oral test in Hongkong. It gives specific explanatory notes for every parts and macro control of unfair scoring problems of the judgment. The audiotapes of the candidates are randomly assigned in to groups. Each audiotape is sc
5、ored by two teachers and the final score is the average grade of the two teachers.Through test-theory study and statistic analysis, this paper probe into the validity and reliability of oral test in TEM4 in 2012. Reliability particularly emphasizes on the test content, testing and grading. Validity
6、will focus on the content validity, face validity, standard validity and theory validity. In addition, by using reliability coefficient to test reliability, this paper will come to a conclusion of reliability and validity of the content and testing in Oral Language Test through data analysis.Oral la
7、nguage test is a vital part in Language Testing. Also, it is a part of linguistic study, that is, application of foreign language oral theory in foreign language oral teaching testing. Oral Language Test of TEM4 has already spread more than 10 years. But whether it efficiently reflects oral levels o
8、f examinees? Guided by the language testing theory and oral linguistics theory, this paper will analyze quality of this oral language test in 2012 from reliability and validity.2. The Reliability Henning (2001) defines reliability as “a measure of accuracy, consistency, dependability or fairness sco
9、res resulting from administration of a particular examination” . The tendency toward consistency found in repeated measurements is referred to as reliability (Carmines & Zeller, 1979). A test is reliable if it is consistent across different characteristics of the testing situation.2.1 Content Reliab
10、ilityFactors that affect reliability are the length, difficulty and discrimination of the content. (Bachman, 1999). Basically speaking, the more questions the test contains, the larger it covers and the the longer of its length, the higher of the test will be. The oral test with a fixed length not o
11、nly offers abundant language using examples, but also limits the influence from the prejudice of the judgment (Huang Yonghong, 2006). From this perspective, the reliability of TEM4 oral English test is well guaranteed. The total time of the oral test is about 19 minutes and reaches the length requir
12、ement. If the test is too easy or too difficult, the discrimination will be declined. TEM4 oral English test has a good command of difficulty and discrimination (Li Zhaoqing, 2005). The former two parts are rather easy and the last part is rather difficult, which makes sure of the discrimination of
13、the scores on the whole.2.2 Administrative ReliabilityAdministrative reliability is defined as the reliability of the preparation form and test procedure in the test. In this aspect, TEM4 oral English test has achieved high reliability. Firstly, the oral test is taken at the same time. Second, the c
14、andidates take the test in language laboratory and start recording at the same time to ensure fair and confidentiality (Liu Runqing, 1991).2.3 Scorer ReliabilityIn the first place, scorer reliability depends on whether the scoring criteria is simple to operate, concrete and accurate. TEM4 oral test
15、has very concrete scoring standards. The test paper in 2012 for example offers 25 points in the first part of retelling, one point for one right. The other two parts also have specific scoring criteria.Moreover, scorer reliability depends on the scoring foundation. Jin Yan (2002) in her research of
16、reliability and validity of tape-assisted oral English test points out that oral English test carried out by recording achieves admirable consistency . This test method not only saves time and human resources, but also provides a objective scoring foundation. The judgment can listen to the recording
17、 over and over again and make careful comparison to choose the excellences (Liu Runqing, 1991). As a result, some adverse impacts can be avoided such as the pre-conceived image of candidates and ignorance of some contents out of fatigue. 24 The research on the reliability of Speaking Test in TEM4Rel
18、iability is the overall consistency of a measure. A measure is said to have a high reliability if it produces similar results under consistent conditions. In order to test the reliability for research purpose , the precious scholars usually employed The Retesting Method or The Reevaluating method. B
19、ut the fact that student just take the Speaking Test in TEM4 once makes The Retesting Method seem not so practical. As for the method of reevaluation, only the Examination Center has the authority of using it. Therefore, how to test the reliability of Speaking Test in TEM4? Wen Qiufang, a professor
20、in Nan Jing University ，based on reliability coefficient (if Reliable coefficient is lower than 0.4, it means the reliability of the test is low to some extent.), uses the method of format: Reliable Coefficient = Note: N refers to the total number of testing sections; m refers to the average score o
21、f the examinees; x refers to standard deviation(Sd)S. d. =Note: d refers to the deviation to the average score of every examinee.(Cai Zhengying1999:167)to test the reliability of Speaking Test in TEM4. And she gets the reliability coefficient of two classes in1999 grade, which are 2.26 and 1.68 resp
22、ectively and both are greatly higher than 0.4. So, she draw to conclusion that Speaking Test in TEM4 is highly reliable. However, from the perspective of our group, it is still doubtful to say that Speaking Test in TEM4 is highly reliable since that the figures which base on only two classes don hav
23、e the representation.3The validity of Speaking Test in TEM4Test validity is the extent to which a test accurately measures what it purports to measure. And according to Bachman and Liu Runqing, the validity can be clarified into four parts: (a) content validity, (b) face validity, (c) criterion-rela
24、ted validity, and (d) construct validity. (Bachman 1999 ：243255 ；Liu Runqing 1991 ：16 18)。31 content validityContent validity includes two aspects: the relevance of the content and the courage of the content.As for the relevance of content, Popham points out it should include three elements: (a) the
25、 purpose of the test(b) the traits of giving examinees inspirations(c) the traits of knowing possible questions raised by examinees. According to the College Teaching Outline for English Major, the examinees must meet some requirements in TEM4 Speaking test such as they ought to have the ability of
26、communicate with the native english speakers in general social occasions. And they should be able to express their ideas accurately with a right pronunciation and a natural tone. Whats more, they should make sentence without grave grammatic errors. Based on such an Outline,the Speaking Test in TEM4
27、is designed to include three parts, which effectively test the examinees ability of conveying their thoughts to others, speaking with a right pronunciation and natural tone and making no serious grammatic-error sentences. Huang Yonghong, a young scholar coming from Hei Longjiang University, however
28、claims that the Speaking Test effectively test the examinees ability except for communicating with native English speakers, which is limited by some objective conditions. In terms of the the courage of content, Wen qiufang, Wu Caixia who study in Nan Jing University and Lydia So who comes from the U
29、niversity of Hong Kong, maintain that Speaking Test in TEM4 usually employs several but not just one questions in order to test the true oral English level of examinees. But, its content ignores cultural element to some extent, which has been improved in The New Teaching Outline that it requires the
30、 examinees to have better acquaintance with the Geography, History, Literature and Culture of the English speaking countries. And as for the decency of language, Speaking Test in TEM4 is more reasonable since the decency of language is closely associated with the different contexts of the conversati
31、ons, which Speaking Test in TEM4 covers a lot.32 Face ValidityFace validity refers to a test appears to be appropriate at least on the surface. The speaking test in TEM-4 is a semi direct oral test, for it guarantees its fairness through tape recording. According to Pang Jixian and Chen Chan(2005),
32、the exam content in a semi direct oral test is unified and the process of test and scoring is seperate from each other, therefore it is not easily affected and has a high level of validity. They claim that because of the lack of interaction, the face validity is low. Huang yonghong(2006) argues the
33、quality of the recorded tape should be improved because candidates can hear some whisper during the break, which may have some effect on the performance of candidates. Besides, the test-maker makes it clear in the test specification that the candidates must use their own words when retell and recitation w
