Effects of subtitles on vocabulary learning through videos: An exploration across different learner types

Sijia Chen1, Guangdong University of Foreign Studies

The Journal of Specialised Translation 42 (2024), 257-276

https://doi.org/10.26034/cm.jostrans.2024.5992

Creative Commons Attribution 4.0 International

ABSTRACT

This study examines the effect of four subtitling conditions — no subtitle, intralingual subtitles, interlingual subtitles, and keyword subtitles — on vocabulary learning among diverse learner groups, as well as their perception of the effectiveness of subtitles. The study involved four participant types, all learning English as a foreign language: second-year primary school students (N = 147), fourth-year primary school students (N = 97), middle school students (year 8, N = 38), and high school students (year 10, N = 50). Participants watched videos under different subtitling conditions, followed by a vocabulary test. Our findings revealed that middle and high school students scored higher on vocabulary meaning recall when subtitles were provided. Meanwhile, second- and fourth-year primary school students did not benefit from the subtitles in terms of meaning recognition. Among the subtitle types, bilingual keyword subtitles were the most effective for middle school students, while interlingual subtitles proved to be the best for high school students. Generally, the students' perception of the effectiveness of the subtitles in the experiment aligned with their actual learning outcomes.

KEYWORDS

Subtitle, vocabulary learning, effectiveness, learner types, video.

1. Introduction

One of the focuses of audiovisual translation is to transfer multimodal and multimedia content across languages (Pérez-González, 2019). Such content provides abundant resources for foreign language learning, and its accessibility continues to proliferate. There is a long-standing interest in using subtitled videos to support second and foreign language learning. Ever since Price’s (1983) pioneering work on the potential of closed-captioned TV, researchers have extensively examined the effects of same-language subtitles on language learning, with many focusing on vocabulary acquisition (for reviews see Montero Perez et al., 2013; Vanderplank, 2010, 2016). Subtitled audiovisual input has generally been demonstrated to bolster vocabulary learning (Montero Perez et al., 2013). Certain subtitle types, such as intralingual subtitles in the foreign language, have been subjected to more comprehensive investigation, whereas other types, such as interlingual subtitles in the native language and keyword subtitles, have been less researched. Furthermore, many of these studies focused on older students, often those enrolled in high school or higher educational institutions (Vanderplank, 2016). There is relatively limited research focusing on younger students who have limited language proficiency and literacy.

The purpose of this research is to examine the effectiveness of different subtitle types, namely intralingual, interlingual, and keyword subtitles, in enhancing vocabulary learning among diverse learner types, including primary, middle, and high school students. The findings could potentially help optimise language teaching and learning strategies to harness the potential of videos and subtitles.

2. Literature review

2.1. Intralingual subtitles

Based on their linguistic characteristics, subtitles can be divided into two broad categories: intralingual and interlingual (Díaz-Cintas, 2020). Intralingual subtitles, otherwise referred to as captions, are in the same language as the spoken dialogue of the audiovisual content, presenting the audio-verbal information in visual-verbal form. Typical forms of intralingual subtitles include subtitling for the deaf and hard-of-hearing (SDH, including live subtitling) and subtitles devised as a didactic tool for teaching and learning foreign languages. The application of intralingual subtitles has been extensively explored within the realms of second and foreign language acquisition (for a review, see Vanderplank, 2016). A meta-analysis carried out by Montero Perez et al. (2013) synthesised the findings from ten studies (six journal articles and four unpublished doctoral dissertations) that focused on the impact of intralingual subtitles on vocabulary learning. It was found that intralingual subtitles had a large effect on vocabulary learning, regardless of the type of assessment employed — recognition or recall — and across both beginning and intermediate learners.

Using videos with intralingual subtitles to enhance language learning is supported by Paivio’s (1986) dual-coding theory. It proposes that by presenting the same information in two modes — aural and written — both the verbal and imagery system can be stimulated, leading to greater depth of processing and better recall. The presence of intralingual subtitles could also guide learners to consciously focus on form, aiding them in establishing a connection between form and meaning. Furthermore, intralingual subtitles delineate word boundaries (Bird & Williams, 2002) and assist learners in segmenting the speech stream, isolating words, and locating important information (Danan, 2004; Winke et al., 2010).

The potential effectiveness of intralingual subtitles is related to the learner’s foreign language proficiency (e.g., Suárez & Gesa, 2019). Intralingual subtitles may only be truly effective in enhancing vocabulary learning when the gains from this additional (and redundant) verbal-visual source outweigh the effort required for its processing. For example, Webb and Rodgers (2009) proposed a general guideline for language learning through watching movies and suggested a baseline knowledge of the most frequent 3000 word families. Given that low foreign language proficiency potentially limits the effect of intralingual subtitles on vocabulary learning, a subsequent question arises: what if learners are provided with subtitles in their native language? This question is addressed by studies on interlingual subtitles.

2.2. Interlingual subtitles

Interlingual subtitles translate the spoken (and sometimes written) message of the original audiovisual content into a different language. A video with interlingual subtitles thus offers a triple connection between visual images, audio-verbal dialogue in one language, and visual-verbal subtitles in another language, with the two sources of verbal information linked by translation. In the context of foreign language learning, interlingual subtitles often denote standard subtitles in learners’ native language while the video is in the foreign language, although reversed subtitling involving videos in the native language and subtitles in the foreign language is also studied (e.g., Danan, 1992). Given the added effects of both image and translation, videos with interlingual subtitles can potentially enhance the communication and comprehension of word meanings, thus supporting vocabulary learning.

Several studies involving English as a Foreign Language (EFL) learners have explored the impact of standard interlingual subtitles on vocabulary learning. Koolstra and Beentjes (1999) observed that Dutch primary school students performed better in a vocabulary test when provided with interlingual subtitles compared to no subtitle. In this test, target words were presented in auditory form, and the children needed to choose the correct Dutch translation from four alternatives. The subtitled group also performed better in an auditory word recognition test, which involved recognising 20 words that appeared in the stimulus out of 30 alternatives. Peters et al. (2016) compared the effects of intralingual and interlingual subtitles on vocabulary learning among two participant groups: intermediate level learners from a general secondary school with Dutch as their L1 and low-proficiency learners from a vocational school with diverse native languages. The study found that for intermediate learners, interlingual subtitles were less effective in enhancing form recognition, while both subtitle types were comparable in facilitating meaning recall. For low-proficiency learners, the interlingual subtitles group outperformed in meaning recognition, underperformed in form recall, and showed no difference in form recognition compared to the intralingual subtitles group. Some studies have compared three conditions: no subtitle, intralingual subtitles, and interlingual subtitles. Peters (2019) used Dutch-speaking learners in a secondary school as participants and found that the intralingual subtitles group made more gains in vocabulary learning compared to the interlingual subtitles and no subtitle groups. Experimenting with Polish high school students, Baranowska (2020) found that intralingual subtitles assisted learners in vocabulary acquisition more than interlingual subtitles, but no significant difference was detected between intralingual subtitles and no subtitle, or between interlingual subtitles and no subtitle. In contrast, Birulés-Muntané and Soto-Faraco (2016) worked with Romance language speakers (Catalan, Spanish, and Italian) at a university, and found no reliable differences between the three conditions in vocabulary learning. A study conducted by Bisson et al. (2014) featured a group of special participants: native English speakers without any knowledge of Dutch in a university. Their results showed no vocabulary acquisition, regardless of the subtitling condition.

On the whole, studies investigating the impact of interlingual subtitles on vocabulary learning have yielded more diverse results than those focusing on intralingual subtitles. The effectiveness of interlingual subtitles might depend on factors such as age, learners’ proficiency, the language pair, and the closeness between the two languages.

2.3. Bilingual subtitles

Bilingual subtitles can be regarded as a variant of interlingual subtitles (Díaz-Cintas, 2020). They are originally a type of screen translation to render a foreign language into two domestic languages, used in geographical areas where more than one language is spoken, such as Belgium where Flemish and French are both used, Israel where Hebrew and Arabic are both used (Gottlieb, 2004), and Switzerland where movie theatres often show subtitles in two or even three languages (Wissmath et al., 2009). Bilingual subtitles are also employed in international film festivals, where foreign films are sometimes screened with two sets of subtitles: one in English and one in the language of the country where the film festival takes place (Díaz-Cintas, 2020). Furthermore, film companies attempting to appeal to the international market may also utilise bilingual subtitles. For example, when Chinese films are shown in China, bilingual subtitles are often already added (Jin & Yves, 2018), with the original dialogue in Chinese transcribed and translated (often into English) simultaneously. In addition, viewers in certain countries, such as China and Qatar, are interested in using bilingual subtitles for language learning purposes (Díaz-Cintas, 2020). Audiovisual content in foreign languages is watched with the original dialogue transcribed in the original language and translated into the viewers’ native language.

Despite the application of bilingual subtitles in the range of settings mentioned above, the question of whether bilingual subtitles are effective in enhancing vocabulary learning remains under-researched (Wang & Pellicer-Sánchez, 2022). Existing research efforts are mainly dedicated to bilingual subtitles that present the transcription of the original foreign language dialogue together with its translation into the native language of the viewer. On the one hand, this type of bilingual subtitles can potentially offer a combined benefit of intralingual and interlingual subtitles. On the other hand, they could pose more severe risks of split-attention, extraneous processing, cognitive overload, and redundancy (Ayres & Sweller, 2014; Kalyuga & Sweller, 2014).

Among the few studies that have empirically investigated the topic, Lwo and Lin (2012) found no significant difference among intralingual, interlingual, bilingual, and no subtitle conditions in Chinese-speaking EFL learners in junior high school. In contrast, Li (2016) observed that bilingual subtitles yielded better outcomes for English majors in a Chinese university compared to intralingual, interlingual and no subtitle conditions in terms of meaning recognition and recall on both immediate and delayed post-tests. Wang and Pellicer-Sánchez (2022) worked with intermediate to advanced Chinese learners of English studying in a university in the UK and found an advantage for bilingual subtitles over intralingual subtitles for meaning recognition and over interlingual subtitles for meaning recall. However, bilingual subtitles were less effective than intralingual subtitles for form recognition. From a subjective perspective, García (2017) surveyed users of a tool for creating bilingual subtitles and found that they perceived bilingual subtitles as useful for incidental vocabulary learning.

The limited empirical evidence so far does not provide a clear consensus on whether bilingual subtitles are better or worse than other subtitle types for enhancing vocabulary learning. Further research is needed to deepen our understanding, especially considering the various settings where this type of subtitles can be applied.

2.4. Keyword subtitles

Keyword subtitles distinguish themselves from conventional intralingual and interlingual subtitles in that they do not constitute full subtitles; rather, they contain only those words crucial to the meaning of a sentence (Montero Perez et al., 2018). This particular type of subtitles is studied on the premise that presenting less visual-verbal information to process may facilitate learners' capability to distribute their attention among the various sources of information present in a subtitled video, such as the visual image, the audio, and the subtitles.

Keyword subtitles can be either intralingual (in the same foreign language as the video) or interlingual (typically bilingual, featuring the keyword in the foreign language accompanied by a gloss offering the L1 translation). Montero Perez et al. (2014) compared the impacts of no subtitle, intralingual (full) subtitles, and intralingual keyword subtitles on vocabulary learning among university students. They found that intralingual keyword subtitles yielded superior results in terms of vocabulary form and meaning recognition when compared to no subtitle, yet they did not significantly enhance meaning recall. Teng (2019) experimented with grade six primary school students in Hong Kong and identified differences across the three subtitling conditions. Their findings suggested that intralingual subtitles led to the best outcomes, followed by intralingual keyword subtitles and then by no subtitle. Montero Perez et al. (2018) added interlingual keyword subtitles to the comparison and found that university students scored best on the form recognition and meaning recall tests when provided with this type of subtitles. Similarly, Teng (2022) found interlingual keyword subtitles to work best in enhancing form recognition, meaning recognition, and word use when working with grade six students from primary schools in Hong Kong.

The investigation into keyword subtitles provides initial evidence of enhanced vocabulary language learning. This innovative form of subtitles appears to offer a promising tool for optimising attention distribution and facilitating more efficient vocabulary learning experiences, but more empirical research is needed before any final conclusions can be drawn.

2.5. Aim and research questions

Decades of research underscore the effectiveness of subtitles in enhancing vocabulary learning; however, certain aspects warrant further investigation. To the best of the author's knowledge, no study has yet compared the effects of intralingual, interlingual, and bilingual keyword subtitles on vocabulary learning across various learner types. Additionally, many previous studies have primarily focused on university-level students (Montero Perez et al., 2013), potentially leading to an underrepresentation of younger learners, especially those whose language proficiency may not sufficiently support subtitle processing in the foreign language.

This study aims to bridge some existing gaps by examining the effects of four subtitling conditions — no subtitle, intralingual subtitles, interlingual subtitles, and bilingual keyword subtitles — on vocabulary learning across four distinct learner groups: second-year primary school students, fourth-year primary school students, middle school students, and high school students. The study addresses the following research questions:

1. What are the language learners’ perceptions of the effectiveness of subtitles on vocabulary learning?

2. What are the effects of intralingual subtitles, interlingual subtitles, and bilingual keyword subtitles on foreign language vocabulary learning among primary school, middle school, and high school students?

3. Method

3.1. Participants

This study involved four types of participants: 147 second-year students from a primary school in the central region of China (Mage = 7.27, SD = 0.47), 97 fourth-year students from a primary school in the south-eastern region (Mage = 9.13, SD = 0.36); 38 students from a middle school (eighth-year) in the eastern region (Mage = 13.13, SD = 0.41), and 50 students from a high school (tenth-year) in the central region (Mage = 15.14, SD = 0.41). All participants were learning English as a foreign language. The English proficiency level of the second and fourth-year primary school students was estimated by their teachers to be at the A1 level based on the Common European Framework of Reference for Languages (CEFR) scale, while that of the middle and high school students was estimated to be at A2 and between A2 and B1, respectively. Ethical clearance was obtained before the commencement of the study, and informed consent was provided by both the participants and their parents by signing a consent form.

For the second and fourth-year primary school students, the study adopted a between-subject design, with four parallel classes randomly assigned to the four subtitling conditions. We used their most recent end-of-term English exam scores to ensure that the four classes in each group had comparable foreign language proficiency. For the middle school and high school students, owing to a smaller sample size, a within-subject design was implemented by exposing all participants to the four subtitling conditions. The four types of participants were analysed separately.

3.2. Questionnaires

A pre-questionnaire was administered one week prior to the experiment. It included ten questions, collecting information on demographic background, how often the participants watched English videos outside of class, and what type of subtitles they usually chose when watching English videos. When watching subtitled videos, learners have access to a rich combination of multimodal and multiple-source information, including visual images, spoken dialogue, subtitles, and background sounds (Gottlieb, 1998; Liao et al., 2020). Therefore, the questionnaire also asked the participants to rate the effectiveness of various components in English videos for supporting vocabulary learning, based on their everyday experience of watching such videos. On a scale of 1 to 5, they rated the effectiveness of English audio, visual images, English subtitles, Chinese subtitles, and bilingual subtitles. The decision to exclude bilingual keyword subtitles from the questionnaire stemmed from our aim to assess participants’ preferences regarding everyday subtitle usage. As bilingual keyword subtitles are not commonly found in the videos typically viewed by participants, they were not included in the survey.

After completing the experiment, a post-questionnaire comprising 15 questions was administered. These questions aimed to gather participants’ feedback on the experiment and their perception of the effectiveness of various video components in supporting vocabulary learning, drawing from their experience during the experiment. Specifically, participants were asked to rate the suitability of the experiment videos for their language level, the level of difficulty, and their level of interest, as well as their ability to read subtitles, follow the pace, and allocate attention between audio, visual images, and subtitles. They were also asked to assess the subtitle speed and font size. Additionally, participants were prompted to re-evaluate the effectiveness of English audio, visual images, and different subtitles in supporting vocabulary learning, based on their experiment experience. An open-ended question allowed participants to provide any additional feedback about the experiment. Both the questionnaire and responses were in Chinese.

3.3. Videos and subtitles

Videos for this study were sourced from a website called Little Fox, which provides animated stories for English language learners in various grades. Permission was granted by Little Fox for the author to use their videos for research purposes. The structured nature of the videos on the Little Fox platform, categorised into nine levels, facilitated the control of video difficulty and the selection of appropriate videos for the different types of participants in our study. The selection of videos involved three steps, performed separately for each type of participant. The first step involved identifying the video level. The researcher, together with the participants' English teacher, explored various video levels by watching some videos in each level and comparing them to the videos typically used in school. Second, once a level was determined, the researcher and the teacher selected candidate videos from that level that were comparable in topic, difficulty, and duration. Third, a final screening procedure was applied to ensure that the selected videos contained at least five potential target words (see Section 3.4).

In the end, one video (Little Fox Level 1, 1.38 minutes) was selected for second-year primary school students; one video (Little Fox Level 2, 2.08 minutes) for fourth-year primary school students; four videos (Little Fox Level 5, averaging 4.76 minutes each) for middle school students; and four videos (Little Fox Level 7, averaging 5.99 minutes each) for high school students. The variation in video lengths primarily stemmed from differences in students’ ages and English proficiency levels. Furthermore, this study aimed to align the video lengths with participants’ typical exposure.

For each of the two videos selected for primary school students, four versions were created: no subtitle, intralingual subtitles, interlingual subtitles, and bilingual keyword subtitles. The four videos selected for middle school students were randomly assigned to a subtitling condition and relevant subtitles were added. The same process was applied to the four videos for high school students. Intralingual subtitles comprised a verbatim transcription of the dialogue, synchronised with the speaker and ensured that the speed did not exceed 18 characters per second. Interlingual subtitles were created by translating the intralingual subtitles from English into Chinese, taking into consideration spatial and temporal limitations. For quality assurance, the translated subtitles underwent checks by the four teachers participating in this study and a research assistant (a student studying for a Master of Translation and Interpreting degree). Necessary modifications were implemented to guarantee accuracy.

For bilingual keyword subtitles, we first identified the English keywords by watching the videos and reading the transcripts. Discussions were held and a consensus was reached between the teachers and the researcher, following steps outlined in previous studies (Montero Perez et al., 2018; Teng, 2022). On average, the English keywords represented around 20.8% of the English transcript. Subsequently, these English keywords were translated into Chinese and made into subtitles that displayed both languages side by side (English on the left and Chinese on the right). All target words were included in the keywords. Keyword subtitles stayed on the screen for an average of 2.58 seconds, with the subtitle appearance coinciding with the onset of the relevant word(s) in the audio.

3.4. Target words and pre-tests

For each video, target words that were predicted to be unknown to the participants were selected, and the following procedure was applied to each type of participant independently. First, the researcher identified potentially known words by surveying the English textbooks used by the participants, with particular focus on the glossary section, which typically contains words emphasised for learning and assessment and are thus likely to be familiar to the participants. Second, the teacher selected candidate words from the video based on her assessment of the students’ proficiency. Third, the target words were determined. The candidate words were checked against the list of potentially known words, and discussions were held between the researcher and the teacher to determine the target words, settling on five for each video (see Table 1). Because the study used existing English videos rather than creating them from scratch, controlling variables related to the target words (e.g., word class) was challenging. While these decisions may limit generalisability, they were necessary to ensure feasibility in a non-laboratory setting. These factors are incorporated into the Discussion section.

Student type Target words Subtitle condition Student type Target words Subtitle condition
PS2 axe All conditions PS4 cabin All conditions
hose castle
ladder brick
mask straw
firefighter igloo
MS tadpole No subtitle HS rash No subtitle
metamorphosis smallpox
gill cowpox
amphibian antibody
pond vaccine
nocturnal Intralingual subtitles chlorophyll Intralingual subtitles
branch sap
dingo pigment
pouch radish
joey shining
backyard Interlingual subtitles limb Interlingual subtitles
tear gill
chainsaw shrimp
sawdust moulting
slab starfish
diver Bilingual keyword subtitles puny Bilingual keyword subtitles
tentacle skyscraper
octopus snap
tank vertically
clownfish sway

Note:

PS2 = primary school year 2; PS4 = primary school year 4; MS = middle school; HS = high school.

Table 1 Target words used in this study

To ensure that the study only tested vocabulary learning based on previously unknown words, a pre-test was conducted. Participants were provided with a list of words and asked to indicate whether they knew the meaning of each word by ticking ‘yes’ or ‘no.’ To avoid guessing, they were asked to provide a translation, a synonym, or an explanation of the word in their first language (L1) for any word marked as known. In line with previous studies (Montero Perez et al., 2018; Teng, 2022), distractors (high-frequency words) were mixed with the target words in the pre-test to prevent students from identifying the target words and to avoid undermining their confidence. Specifically, primary school students encountered five distractors, while middle and high school students encountered ten. This pre-test was administered one week before the experiment.

3.5. Post-tests

For primary school students, a picture choice test was used to measure vocabulary meaning recognition. The test followed the method outlined by Koolstra and Beentjes (1999), using an audio recording in which the words were pronounced twice by a native speaker. Participants were asked to select the picture corresponding to the spoken English word from two options. The purpose for using a picture choice test was mainly to align the post-test with the typical English tests encountered by the participants in school. The teachers suggested that the students were more accustomed to tests involving picture selection rather than translation. Additionally, considering the young age of the participants, the researcher aimed to create a video-watching and testing experience that was enjoyable and engaging. The decision to use an auditory format was also influenced by teachers' observations that primary school students, especially those in the second year, tended to have stronger listening skills compared to reading skills. The teacher of the second-year students emphasised that they were in the early stages of learning English, and a substantial portion of their daily learning materials were delivered in audio format. Even during written tests, the teacher sometimes had to read the questions aloud to assist the students in selecting their answers.

For middle and high school students, their vocabulary meaning recall was assessed by providing the L1 translation for the target word, a test format familiar to the participants. Both the questions and answers were presented in written format. Each target word was embedded within a sentence and marked by underline. The sentences offered limited context to prevent guessing but provided enough information to ensure that the meaning matched the usage in the video. For example, to test the target word ‘pouch,’ referring specifically to a koala bear's pouch, the following sentence was provided: “It needs to finish developing in its mother's pouch.”

The post-test score was calculated by dividing the number of words answered correctly by the total number of target words. If a participant answered a target word correctly in the pre-test, that word was subsequently excluded from the calculation of the post-test score. For example, if a participant answered one target word correctly in the pre-test and correctly answered three out of the remaining four target words in the post-test, the score would be computed as 3/(5–1), which equates to 0.75.

The design of the post-tests meant that primary school and middle/high school participants were exposed to different forms of target words (auditory vs. written) and assessed on different aspects of vocabulary learning (meaning recognition vs. meaning recall). Consequently, the results were not directly comparable. In addition, this study did not include delayed post-tests, which could help offer a more accurate picture of the effectiveness subtitles. Given these limitations, the data collected from different types of participants were analysed separately, and discussions were carefully organised while keeping these limitations in mind.

3.6. Procedure

A week prior to the experiment, participants completed both the pre-questionnaire and the pre-test. During the experiment, the participants watched the video twice and then completed the post-test, following common practice in previous research (Montero Perez et al., 2013). After the experiment, the participants completed the post-questionnaire. For primary school students, due to the between-subject design, each group only watched one video under one subtitling condition. For middle and high school students, they watched four separate videos, each under a different subtitling condition, spread over the course of a week.

3.7. Data analysis

To address the variations in between-subject and within-subject designs, video length, and test format, the data were analysed separately for each type of participant. One-way ANOVA was employed for primary school participants, while repeated measures ANOVA was used for middle and high school participants. Some of the data were not normally distributed, but the selected parametric tests have been proven robust to violations of the normality assumption (Tabachnick & Fidell, 2007). To ensure the robustness of the results, validation was performed by applying the nonparametric Kruskal-Wallis test and the Friedman test to data that did not conform to a normal distribution. The results of the nonparametric tests concurred with the parametric tests. Consequently, the study reported the results of the parametric statistics, as the interpretation of the data would be more straightforward.

4. Results

4.1. Perception of subtitle effectiveness

Participants were asked what type of subtitles they usually chose when watching English videos, and the preference for bilingual subtitles was consistent across all learner types (Table 2).

PS2 (%) PS4 (%) MS (%) HS (%)
No subtitle 9 1 0 0
Intralingual subtitles 22 7 0 2
Interlingual subtitles 16 10 16 20
Bilingual subtitles 52 76 84 76
Other 0 5 0 2

Notes:

1. PS2 = primary school year 2; PS4 = primary school year 4; MS = middle school; HS = high school.

2. Percentages may not add up to 100 due to rounding.

Table 2 Subtitle preferences

When watching subtitled English videos, learners have access to a combination of information sources, such as visual images, spoken dialogue, and various types of subtitles. Therefore, this study asked the participants to rate the effectiveness of these sources in terms of supporting vocabulary learning. On a scale of 1 to 5, the participants rated the effectiveness of English audio, visual images, English subtitles, Chinese subtitles, and bilingual subtitles. Repeated measures ANOVAs were applied to data collected from the four groups of participants separately, and the results are summarised in Table 3. The main effect of source was significant for all learner types. Post hoc analysis with a Bonferroni adjustment revealed that participants rated bilingual subtitles to be the most effective, and no significant difference was found among the other sources. This result was consistent across all learner types (Table 4).

df1 df2 F p
PS2* 3.33 486.83 7.69 < 0.001
PS4* 2.97 275.71 20.94 < 0.001
MS* 2.44 90.41 9.68 < 0.001
HS 4 196 7.45 < 0.001

Notes:

1. PS2 = primary school year 2; PS4 = primary school year 4; MS = middle school; HS = high school.

2. * Greenhouse-Geisser correction was applied.

Table 3 A summary of repeated measures ANOVA results on perceived effectiveness for the four types of participants

95% CI
MD Lower Upper p
PS2 Bilingual subtitles English audio 0.52 0.17 0.88 <0.001
Visual images 0.72 0.31 1.14 <0.001
Intralingual subtitles 0.63 0.23 1.03 <0.001
Interlingual subtitles 0.43 −0.00 0.86 0.054*
PS4 Bilingual subtitles English audio 0.98 0.64 1.32 <0.001
Visual images 1.12 0.73 1.50 <0.001
Intralingual subtitles 1.13 0.75 1.50 <0.001
Interlingual subtitles 1.16 0.75 1.57 <0.001
MS Bilingual subtitles English audio 0.90 0.38 1.41 <0.001
Visual images 1.03 0.35 1.71 <0.01
Intralingual subtitles 1.03 0.56 1.50 <0.001
Interlingual subtitles 0.84 0.48 1.21 <0.001
HS Bilingual subtitles English audio 0.72 0.22 1.22 <0.01
Visual images 0.98 0.38 1.58 <0.001
Intralingual subtitles 0.72 0.22 1.22 <0.01
Interlingual subtitles 0.64 0.10 1.18 <0.05

Note:

1. PS2 = primary school year 2; PS4 = primary school year 4; MS = middle school; HS = high school; MD = mean difference.

2. * This is only approaching statistical significance.

Table 4 A comparison of the perceived effectiveness of bilingual subtitles against English audio, visual images, intralingual subtitles, and interlingual subtitles

After the experiment, the participants rated the effectiveness of the sources again, based on their experience during the experiment. For second and fourth-year primary school participants, three classes were exposed to subtitled conditions and one class was assigned to the no subtitle condition. The three classes exposed to subtitled videos were asked to rate the effectiveness of English audio, visual images, and subtitles (irrespective of their type).

For second-year primary school participants, a one-way ANOVA showed that the rating score given to subtitles did not differ between the three classes (F(2,104) = 0.19, p = 0.83), meaning that the participants gave the source ‘subtitles’ similar rating scores regardless of whether they were exposed to intralingual, interlingual, or bilingual subtitles. For a subsequent analysis comparing subtitles, English audio, and visual images, data from the three classes were merged. A repeated measures ANOVA with a Greenhouse-Geisser correction determined that perceived effectiveness differed statistically significantly between English audio, visual images, and subtitles (F(1.80, 191.23) = 4.41, p = 0.016). Post hoc analysis with a Bonferroni adjustment revealed that the effectiveness of subtitles (M = 3.95, SD = 1.02) was rated to be significantly higher than that of English audio (M = 3.65, SD = 1.23, p = 0.025) and visual images (M = 3.57, SD = 1.30, p = 0.027), but the latter two were not significantly different from each other (p = 1.000).

For fourth-year primary school participants, a one-way ANOVA showed that the three classes, although exposed to different types of subtitles, gave similar rating scores to the source ‘subtitles’ (F(2,74) = 2.59, p = 0.082). Data from the three classes were merged for subsequent analysis to compare English audio, visual images, and subtitles. A repeated measures ANOVA with a Greenhouse-Geisser correction showed that the three sources differed statistically significantly in their rating scores (F(1.83, 136.97) = 8.52, p < 0.001). Post hoc analysis with a Bonferroni adjustment showed that the perceived effectiveness of English audio (M = 3.58, SD = 1.18) was significantly lower than that of visual images (M = 4.17, SD = 1.10, p = 0.003) and subtitles (M = 4.07, SD = 1.08, p = 0.006), but the latter two did not differ significantly (p = 1.000).

For middle and high school participants, who were exposed to all subtitling conditions, they were asked to rate the effectiveness of five different sources in subtitled videos: English audio, visual images, intralingual subtitles, interlingual subtitles, and keyword subtitles. In the case of middle school participants, repeated measures ANOVA with a Greenhouse-Geisser correction did not reveal any significant difference in perceived effectiveness between the five sources (F(3.20, 118.22) = 2.12, p = 0.097).

As to high school participants, a repeated measures ANOVA determined that the perceived effectiveness differed statistically significantly between the five sources (F(4, 196) = 47.66, p < 0.001). Post hoc analysis with a Bonferroni adjustment revealed that the perceived effectiveness of English audio (M = 2.50, SD = 0.58) was significantly lower than that of visual images (M = 3.88, SD = 0.66, p < 0.001), interlingual subtitles (M = 3.76, SD = 0.82, p < 0.001), and keyword subtitles (M = 3.98, SD = 0.80, p < 0.001). Moreover, the perceived effectiveness of intralingual subtitles (M = 2.76, SD = 0.77) was lower than that of visual images (p < 0.001), interlingual subtitles (p < 0.001), and keyword subtitles (p < 0.001).

4.2. Effects of subtitles indicated by post-test scores

The post-test score of second-year primary school students did not differ significantly between the four subtitle conditions, as determined by a one-way ANOVA (F(3,143) = 0.92, p = 0.43). For fourth-year students, a one-way ANOVA showed that the post-test score differed significantly between the subtitling conditions (F(3,93) = 3.55, p = 0.017). A Tukey post hoc test revealed that significant difference only existed between intralingual and interlingual subtitles, with the former (0.74 ± 0.25) leading to higher scores than the latter (0.55 ± 0.26, p = 0.022).

For middle school students, a repeated measures ANOVA determined that the post-test score differed statistically significantly between the four subtitling conditions (F(3, 108) = 17.56, p < 0.001). Post hoc analysis with a Bonferroni adjustment revealed that the score in the no subtitle condition (M = 0.40, SD = 0.28) was significantly lower than that in the intralingual (M = 0.56, SD = 0.23, p = 0.011), interlingual (M = 0.62, SD = 0.24, p < 0.001), and keyword subtitles condition (M = 0.75, SD = 0.23, p < 0.001). Furthermore, the keyword subtitles condition led to significantly higher scores than the intralingual subtitles condition (p = 0.002). No significant difference was found between the intralingual and interlingual conditions or between the interlingual and keyword conditions. A summary of the test statistics is presented in Table 5.

95% CI
Mean difference Lower Upper p
No subtitle Intralingual subtitles −0.16 −0.29 −0.03 0.011
Interlingual subtitles −0.22 −0.35 −0.08 <0.001
Keyword subtitles −0.35 −.499 −0.20 <0.001
Intralingual subtitles Interlingual subtitles −0.06 −0.29 −0.03 1.000
Keyword subtitles −0.19 −0.35 −0.08 0.002
Interlingual subtitles Keyword subtitles −0.13 −0.27 0.01 0.071

Table 5 Comparing the post-test scores among middle school students under different subtitling conditions

For high school students, a repeated measures ANOVA with a Greenhouse-Geisser correction determined that the four subtitling conditions differed significantly in post-test scores (F(2.56, 125.55) = 75.82, p < 0.001). Post hoc analysis with a Bonferroni adjustment revealed that significant difference was found in all pairwise comparisons, with the intralingual subtitles condition leading to the lowest score (M = 0.32, SD = 0.17), followed by the no subtitle condition (M = 0.45, SD = 0.27), and then by the keyword subtitles condition (M = 0.69, SD = 0.25), while the interlingual subtitles condition led to the highest score (M = 0.86, SD = 0.18). The test statistics are summarised in Table 6.

95% CI
Mean difference Lower Upper p
No subtitle Intralingual subtitles 0.14 0.02 0.25 0.010
Interlingual subtitles −0.41 −0.53 −0.29 <0.001
Keyword subtitles −0.24 −0.34 −0.14 <0.001
Intralingual subtitles Interlingual subtitles −0.55 −0.63 −0.46 <0.001
Keyword subtitles −0.38 −0.48 −0.27 <0.001
Interlingual subtitles Keyword subtitles 0.17 0.05 0.30 0.003

Table 6 Comparing the post-test scores among high school students under different subtitling conditions

5. Discussion

This study aimed to address two key issues: (1) the perceptions of different types of learners regarding the effectiveness of subtitles in supporting vocabulary learning through videos; and (2) the effects of different subtitle types on vocabulary learning across learner types.

The data collected from the pre-experiment questionnaire indicated that, given a range of subtitle options commonly available when viewing English videos (i.e., no subtitle, intralingual subtitles, interlingual subtitles, and bilingual subtitles), the participants expressed a preference for bilingual subtitles. This preference was consistent across all learner types, including primary, middle, and high school students. Among the multiple information sources available in a subtitled video that could potentially support vocabulary learning — English audio, visual images, and the diverse subtitle types — participants’ perceived effectiveness was highest for bilingual subtitles, matching their preferred subtitle choice. These findings lend empirical support to the widespread adoption of bilingual subtitles in China (Jin & Yves, 2018), a trend which is particularly noteworthy considering the geographical diversity of the participants in this study. Further research efforts should be directed towards exploring the effectiveness of this subtitle type in the context of foreign language learning.

The results of the experiment generally support the effectiveness of subtitles in enhancing vocabulary learning from videos for middle and high school students. More often than not, both middle and high school students performed better in vocabulary meaning recall under subtitled conditions compared to the no subtitle condition. These findings are consistent with previous research that has highlighted the effectiveness of various subtitle types, including intralingual (Montero Perez et al., 2013), interlingual (Koolstra & Beentjes, 1999), and keyword subtitles (Montero Perez et al., 2018; Teng, 2022).

For middle school students, the level of vocabulary recall was comparable between the intralingual and interlingual subtitle conditions, as was their perception of the effectiveness of the two subtitle types. This observation aligns with findings from some previous studies (Birulés-Muntané & Soto-Faraco, 2016; Bisson et al., 2014; Peters et al., 2016; Pujadas & Muñoz, 2019). Previous research suggests that while intralingual subtitles present congruent verbal information in both auditory and visual forms and may thus benefit students due to the dual-coding effect (Paivio, 1986), interlingual subtitles may offer more accessible meaning and reduce comprehension costs (Koolstra & Beentjes, 1999).

Interestingly, high school students scored the lowest under the intralingual subtitles condition. This objective measurement concurs with the post-experiment subjective rating, where high school students rated the effectiveness of intralingual subtitles lower than that of visual images, interlingual subtitles, and keyword subtitles. Previous studies that detected a difference between intralingual and interlingual subtitles often reported higher effectiveness of intralingual subtitles (Baranowska, 2020; Peters, 2019). One possible explanation for this surprising finding could be related to the stimuli used. It is plausible that the video used for the intralingual subtitling condition may not have been of the same difficulty level as the others, despite attempts to ensure comparability across the four videos (refer to the Method section). In future research, efforts could be made to address this issue by adding trials and counterbalancing the conditions across the videos.

Bilingual keyword subtitles were found to enhance vocabulary learning better than no subtitle or intralingual subtitles among both middle and high school students, corroborating previous research on this specific subtitle type (Montero Perez et al., 2018; Teng, 2022). By increasing visual saliency and access to meaning for target words while concurrently reducing reading demands, bilingual keyword subtitles should be considered a valuable tool for augmenting vocabulary learning when watching videos.

In contrast to the generally positive impact of subtitles on vocabulary learning observed among middle and high school students, this study found no discernible benefit from any type of subtitles for primary school students, including both second- and fourth-year students. This finding was reflected not only in the objective post-test scores but also in their subjective perception of the effectiveness of the different subtitles. Previous research on subtitled videos and language learning has seldom incorporated such young learners with such low foreign language proficiency (Montero Perez et al., 2013).

Koolstra and Beentjes (1999) and Teng (2019) are among the few studies that have examined subtitles and vocabulary learning among primary school students. Koolstra and Beentjes (1999) experimented with Dutch children in fourth and sixth grades, finding that vocabulary scores in the interlingual subtitles condition surpassed those in the no subtitle condition. The authors noted that Dutch children were accustomed to learning English through radio and television, and some students even reported learning more English from these sources than in school. This familiarity with learning English through subtitled audiovisual content may explain the discrepancy between their fourth graders and the participants of this study. Teng (2019) studied six graders who were learning English as a second language and found that vocabulary learning was better with intralingual subtitles than without. Given that participants in the current study were at an even earlier stage of English language acquisition (grade two and four) and were learning English as a foreign language rather than a second language, their English proficiency was likely to be lower.

Participants in this study not only had low foreign language proficiency but also were limited in their native language literacy. Young learners often struggle with reading fluency (Kuhn et al., 2006). This issue is potentially more pronounced with Chinese, because it is a highly orthographic language with little transparency in mapping symbols to sounds (Li, 2014). As outlined by the Compulsory Education Chinese Language Course Standards issued by the Ministry of Education of the People's Republic of China, first and second-year students would learn approximately 1600 Chinese characters, with third and fourth-year students targeting about 2500 Chinese characters. The presence of unfamiliar Chinese characters in interlingual or bilingual keyword subtitles could undermine the effectiveness of the subtitles.

This study does not elucidate why primary school students did not benefit from the subtitles. One possibility is that due to the overwhelming input from multiple sources in a subtitled video, students may have disregarded the subtitles to avoid cognitive overload. However, in the post-questionnaire, participants rated the perceived effectiveness of the audio, images, and subtitles based on their experience during the experiment. They rated subtitles as more effective than audio, suggesting that they likely processed the subtitles, but this did not enhance vocabulary learning. An alternative explanation is that their low foreign language proficiency (and also limited native language literacy) resulted in high cognitive costs when reading and processing subtitles, thereby outweighing the benefits. Further research employing process-oriented methods like eye-tracking might help us gain a deeper understanding of this topic.

The different findings on primary school students and middle/high school students should also be considered in light of the study design. The primary school participants were exposed to shorter videos for reasons elaborated in Section 3.3. It remains to be tested if subtitles could lead to better learning outcomes if primary school students are exposed to longer videos. Furthermore, primary school participants were exposed to auditory tests focusing on meaning recognition, while middle and high school participants were exposed written tests focusing on meaning recall. To comprehensively explore the dynamics of vocabulary learning from subtitled videos, future research should systematically examine various facets of vocabulary knowledge using a diverse array of assessment formats. The different types of participants also varied in their general cognitive development. Due to their superior cognitive skills, older learners show a faster learning rate in foreign language learning (Muñoz, 2008). They might also be more aware of the challenging nature of foreign language learning, place more emphasis on vocabulary, and have a language-learning rather than solely entertainment-orientated purpose in video watching (Vanderplank, 2016).

6. Conclusion

This study provides fresh evidence for the effectiveness of subtitles in enhancing vocabulary learning through videos. Given the popularity of bilingual subtitles among language learners in China, further research efforts towards this less-explored subtitle type are warranted. While most of the subtitle types examined in this study demonstrated potential to enhance vocabulary learning among middle and high school students, their relative effectiveness varies. We need more empirical data before offering specific suggestions to students on the optimal subtitle type to use when watching videos to learn a foreign language. The most researched intralingual subtitles might not always represent the most beneficial choice under all circumstances. The study also encourages caution regarding subtitle usage when younger learners engage in vocabulary learning through videos. It postulates that factors such as video length, language proficiency, literacy skills, and cognitive skills could play a role in determining whether they can benefit from subtitles.

Several limitations prevent us from drawing definitive conclusions from the present study. This study did not subject the participants to the same study design, making it difficult to compare the findings between different participant groups. Using authentic video materials means that it is difficult to control the number of occurrences of the target words and their word frequency levels. A one-time video watch does not provide us with information about how students adapt to learning in the presence of different subtitle types. The study did not administer language proficiency tests to the participants, nor did it include delayed post-tests, both of which could have helped offer a more accurate picture of the effectiveness of subtitles. Furthermore, the participants’ subtitle preferences only became apparent subsequent to data collection via the pre-questionnaire, precluding the inclusion of their preferred subtitle type — bilingual subtitles — in this study. These limitations underscore the need for future research exploring the impact of subtitles on vocabulary learning and, more broadly, language learning through videos.

Acknowledgements/Funding

The author would like to express her gratitude to Little Fox (https://www.littlefox.com/) for granting permission to use their online resources for research purposes.

This work was supported by the National Social Science Foundation of China under grant 20CYY005.

References

Ayres, P., & Sweller, J. (2014). The split-attention principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge Handbook of Multimedia Learning (Second Edition) (pp. 206–226). Cambridge University Press.

Baranowska, K. (2020). Learning most with least effort: Subtitles and cognitive load. ELT Journal, 74(2), 105–115. https://doi.org/10.1093/elt/ccz060

Bird, S. A., & Williams, J. N. (2002). The effect of bimodal input on implicit and explicit memory: An investigation into the benefits of within-language subtitling. Applied Psycholinguistics, 23(4), 509–533. https://doi.org/10.1017/S0142716402004022

Birulés-Muntané, J., & Soto-Faraco, S. (2016). Watching subtitled films can help learning foreign languages. PLoS ONE, 11(6), e0158409. https://doi.org/10.1371/journal.pone.0158409

Bisson, M.-J., van Heuven, W., Conklin, K., & Tunney, R. (2014). Processing of native and foreign language subtitles in films: An eye tracking study. Applied Psycholinguistics, 35(2), 399–418. https://doi.org/10.1017/S0142716412000434

Danan, M. (1992). Reversed subtitling and dual coding theory: New directions for foreign language instruction. Language Learning, 42(4), 497–527.

Danan, M. (2004). Captioning and subtitling: Undervalued language learning strategies. Meta, 49(1), 67–77. https://doi.org/10.7202/009021ar

Díaz-Cintas, J. (2020). The name and nature of subtitling. In Ł. Bogucki & M. Deckert (Eds.), The Palgrave Handbook of Audiovisual Translation and Media Accessibility (pp. 149–171). Palgrave Macmillan.

García, B. (2017). Bilingual subtitles for second-language acquisition and application to engineering education as learning pills. Computer Applications in Engineering Education, 25(3), 468–479. https://doi.org/10.1002/cae.21814

Gottlieb, H. (1998). Subtitling. In M. Baker (Ed.), Routledge Encyclopedia of Translation Studies (pp. 244–248). Routledge.

Gottlieb, H. (2004). Subtitles and international Anglification. Nordic Journal of English Studies, 3(1), 219–230.

Jin, H., & Yves, G. (2018). Audiovisual translation in China: A dialogue between Yves Gambier and Haina Jin. Journal of Audiovisual Translation, 1(1), 26–39. https://doi.org/10.47476/jat.v1i1.42

Kalyuga, S., & Sweller, J. (2014). The redundancy principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge Handbook of Multimedia Learning (Second Edition) (pp. 247–262). Cambridge University Press.

Koolstra, C. M., & Beentjes, J. W. J. (1999). Children's vocabulary acquisition in a foreign language through watching subtitled television programs at home. Educational Technology Research and Development volume, 47, 51–60. https://doi.org/10.1007/BF02299476

Kuhn, M. R., Schwanenflugel, P. J., Morris, R. D., Morrow, L. M., Woo, D. G., Meisinger, E. B., Sevcik, R. A., Bradley, B. A., & Stahl, S. A. (2006). Teaching children to become fluent and automatic readers. Journal of Literacy Research, 38(4), 357–474. https://doi.org/10.1207/s15548430jlr3804_1

Li, H. (2014). Teaching Chinese Literacy in the Early Years: Psychology, pedagogy and practice. Routledge. https://doi.org/10.4324/9781315814285

Li, M. (2016). An investigation into the differential effects of subtitles (first language, second language, and bilingual) on second language vocabulary acquisition [Doctoral thesis, The University of Edinburgh]. ERA. https://era.ed.ac.uk/bitstream/handle/1842/22013/Li2016.pdf?sequence=2&isAllowed=y

Liao, S., Kruger, J.-L., & Doherty, S. (2020). The impact of monolingual and bilingual subtitles on visual attention, cognitive load, and comprehension. The Journal of Specialised Translation, 33, 70–98.

Lwo, L., & Lin, M. C.-T. (2012). The effects of captions in teenagers’ multimedia L2 learning. ReCALL, 24(2), 188–208. https://doi.org/10.1017/S0958344012000067

Montero Perez, M., Peters, E., Clarebout, G., & Desmet, P. (2014). Effects of captioning on video comprehension and incidental vocabulary learning. Language Learning & Technology, 18(1), 118–141.

Montero Perez, M., Peters, E., & Desmet, P. (2018). Vocabulary learning through viewing video: The effect of two enhancement techniques. Computer Assisted Language Learning, 31(1–2), 1–26. https://doi.org/10.1080/09588221.2017.1375960

Montero Perez, M., Van Den Noortgate, W., & Desmet, P. (2013). Captioned video for L2 listening and vocabulary learning: A meta-analysis. System, 41, 720–739. https://doi.org/10.1016/j.system.2013.07.013

Muñoz, C. (2008). Age-related differences in foreign language learning. Revisiting the empirical evidence. IRAL, 46, 197–220. https://doi.org/10.1515/IRAL.2008.009

Paivio, A. (1986). Mental representations: A dual coding approach. Oxford University Press.

Pérez-González, L. (2019). Audiovisual translation. In M. Baker & G. Saldanha (Eds.), Routledge Encyclopedia of Translation Studies (3rd ed., pp. 30–35). Routledge.

Peters, E. (2019). The effect of imagery and on-screen text on foreign language vocabulary learning from audiovisual input. TESOL Quarterly, 53(4), 1008–1032. https://doi.org/10.1002/tesq.531

Peters, E., Heynen, E., & Puimège, E. (2016). Learning vocabulary through audiovisual input: The differential effect of L1 subtitles and captions. System, 63, 134–148. https://doi.org/10.1016/j.system.2016.10.002

Price, K. (1983). Closed-captioned TV: An untapped resource. MATESOL Newsletter, 12, 1–8.

Pujadas, G., & Muñoz, C. (2019). Extensive viewing of captioned and subtitled TV series: A study of L2 vocabulary learning by adolescents. The Language Learning Journal, 47(4), 479–496. https://doi.org/10.1080/09571736.2019.1616806

Suárez, M. d. M., & Gesa, F. (2019). Learning vocabulary with the support of sustained exposure to captioned video: do proficiency and aptitude make a difference? The Language Learning Journal, 47(4), 497–517. https://doi.org/10.1080/09571736.2019.1617768

Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th edition). Pearson.

Teng, F. (2019). Incidental vocabulary learning for primary school students: The effects of L2 caption type and word exposure frequency. The Australian Educational Researcher, 46, 113–136. https://doi.org/10.1007/s13384-018-0279-6

Teng, F. (2022). Vocabulary learning through videos: Captions, advance-organizer strategy, and their combination. Computer Assisted Language Learning, 35(3), 518–550. https://doi.org/10.1080/09588221.2020.1720253

Vanderplank, R. (2010). Déjà vu? A decade of research on language laboratories, television and video in language learning. Language Teaching, 43(1), 1–37. https://doi.org/10.1017/S0261444809990267

Vanderplank, R. (2016). 'Effects of' and 'effects with' captions: How exactly does watching a TV programme with same-language subtitles make a difference to language learners? Language Teaching, 49, 235–250. https://doi.org/10.1017/S0261444813000207

Wang, A., & Pellicer-Sánchez, A. (2022). Incidental vocabulary learning from bilingual subtitled viewing: An eye-tracking study. Language Learning, 72(3), 765–805. https://doi.org/10.1111/lang.12495

Webb, S., & Rodgers, M. P. H. (2009). The lexical coverage of movies. Applied Linguistics, 30(3), 407–427. https://doi.org/10.1093/applin/amp010

Winke, P., Gass, S., & Sydorenko, T. (2010). The effects of captioning videos used for foreign language listening activities. Language Learning & Technology 14(1), 65–86.

Wissmath, B., Weibel, D., & Groner, R. (2009). Dubbing or subtitling? Effects on spatial presence, transportation, flow, and enjoyment. Journal of Media Psychology, 21(3), 114–125. https://doi.org/10.1027/1864-1105.21.3.114

Data availability statement

The data that support the findings of this study are available from the corresponding author upon reasonable request. The data are not publicly available due to privacy or ethical restrictions.


  1. ORCID 0000-0001-8294-9284; e-mail sijia.rachel.chen@gmail.com↩︎