‘Assessing Graded Readers’ by Peter Viney

This is the author’s personal viewpoint and should not be read as the official view of the ERF.

(This article was first published on the ERF website on 22 May 2009)

So you have a box of graded readers to consider for the ERF shortlist. What criteria can you use to judge these fruits of considerable labour by authors, editors and illustrators? The first task is to decide on sections for evaluation. The ones listed here can be subdivided or extended.

1. Stated Level

2. Readability for Language Learners at the Level

3. Quality of writing

4. Appropriateness and quality of illustration and graphic presentation

5. Production factors including general design, typography and standard of editing

6. Overall summary

There are no sections for exercise material (some readers have no exercises), nor for accompanying audio material, computer exercises or DVD video. These are not evaluated although it might be noted if the material fails to work totally without its accompaniments. There is an argument that there should be an award for exercise material, and another for audio recording and another for illustrator, but that’s all for the future. There’s some argument over the quantity of exercises, and some readers appear to be designed for intensive reading work in class with exercises breaking into chapters throughout, paired and group activities and transfer / extension material moving aside from the narrative. In a way, they’re supplementary course books rather than readers for extensive use. The stories are evaluated as if the exercises weren’t there, though it may be noted if exercises or glosses break the reading flow excessively.

1. Stated Level

Level is a potential minefield, even though the young learners section is separate from the elementary section. Each “level” contains sublevels. Many publishers add Council of Europe levels, which are vague, too wide for most purposes at lower levels, as well as irrelevant in East Asia, The Middle East or Latin America.

A brief structural list is more useful for the teacher combining books from different series, and some recent readers have provided this in an appendix.

2. Readability for Language Learners at the Level

This has several areas: lexical control (or headword level), structural control, sentence length and complexity, cultural assumption.

Some publishers keep their grading systems away from general view. Personally, I find this annoying. When I was developing graded reader library systems thirty years ago, the Heinemann Graded Readers Guide and Longman Structural Readers Handbook were regarded as essential information for the teacher, and the word lists and structural guidelines were there for everyone to see. Some readers give broad structural guidelines, which are most helpful to teachers. Others give the word lists which are great for developing accompanying exercises if teachers wish to do so (or even writing tests).  Most importantly, no library system relies only on one publisher. The institution needs to apply its own grading system to encompass books from different schemes and background information is helpful in doing this. When I built up a library system, we added our own colour coded stickers to books from different schemes, so we had a red level, blue level and so on.

When you see only that a reader has 250 or 400 or 1000 or 1500 headwords, it’s hard to know how the grading has been applied. Teachers have said to me that it’s simple. You just use corpus frequency. That doesn’t work unfortunately. The corpus has words based on frequency not simplicity, so a purely corpus based headword count would include will, been, would, should, must at the earliest levels. Obviously the structures these are found with would not have been taught at the level. You would also find that the proportion of structural words (verb forms, determiners, prepositions etc) to lexical (i.e. content) words (nouns, adjectives, adverbs) would be too high to tell a story.

Graded readers word lists owe a debt to Michael West’s General Service List of English Words, which dates back to 1947. Subsequent lists like L.A. Hill’s early list for OUP readers, or the Longman Structural Readers lists were developments. West was based on a written sample, and pretty out of date. For example he found that seldom was more frequent than hardly ever. The reverse is true in speech now, and probably was back then too. More importantly, modern written samples would approximate more closely to speech. Dialogue is a vital part of most graded readers.

When you go back to the version of the Longman Structural Reader wordlist in the 1970s, it included words based on West’s sample of mainly children’s stories. This meant that lion, mouse, chalk and blackboard all appeared in level one. That list was modernized in the late 70s, before being completely replaced by the Penguin Readers list.

When lists are kept secret, it’s hard to judge how well they’ve been adhered to, though it’s often possible to guess whether a word would be “out” of any sensible list at the level.

Traditionally, graded readers were designed round a base list (say 750 words) with a story-specific supplementary count. This varied between schemes, but was typically twenty extra words allowed at lower levels, to forty at higher levels. The scheme I worked on (Storylines) allowed thirty per level. The additional words were essential to the story, and in most schemes were glossed. Writing a graded reader isn’t just a matter of keeping to the word lists. Most series editors would want the quota of twenty, thirty or forty extra words to be used in full. If the quota was thirty and the author produced thirty-five, they would be asked to cut five. But if the quota was thirty, and the author only had (say) fifteen extra words, they would be encouraged to add more, and to make full use of the quota so that students met new words.

The degree of severity in applying quotas varied. In my schemes, I often allowed thirty-one or thirty-two. Other schemes were strict on exact numbers.

The guidelines for additional words in traditional graded readers said something like Words in addition to the word list should be glossed, and should be used more than once in the story.  This is to give the reader a chance to work out the meaning without reference to a gloss. It’s worth noting where new words appear only once in a reader.

The words in a list should be defined. So you should see:

pound (money, weight)

I have seen readers at low level where She pounded her fist on the table appears without a gloss, and this is bad editing (probably with a Search and Find computer facility!) pound was in the list, but not in that sense. Years ago, we tried creating a custom dictionary for each word list in a scheme, so then you could spell-check using the custom dictionary and it would highlight any words outside the custom dictionary. We quickly found that’s how you end up passing She pounded her fist on the table.

In the schemes I was involved with, we had a further supplementary possibility: international words like television, football, jet, internet, world wide web, pizza and spaghetti. Though these might not be in the base list, they were allowed without adding to the additional word quota if the series editor agreed that they were truly international.

In recent years, some readers have added eighty, a hundred or a hundred and fifty words as “glossed in addition to the list.” In my opinion, this is too high for an international readership. However, these have tended to come from Europe and are intended for a European secondary audience. They would not be useable at the same perceived level in Asia or the Middle East because the guessability rate would be far lower. It would be taken into consideration that they were designed for Europe, and readability for learners at the level would be viewed in this context.

In one example there were eight glossed words on a page with a picture and ten lines of print only. This is way above “readability” for a student at the headword level of the book. I would simply consider this badly-graded wherever it was designed for.

Some readers rely on “intuitive grading” rather than a set structural or lexical list. I could write intuitively to a level, and know many experienced authors of readers who could do the same. In these cases though, it’s important to compare books at the same advertised level within a scheme to see that the intuitive grading is consistent for the level. We all have a different view of what “high beginner” means, but a scheme with HIGH BEGINNER emblazoned on it should be internally consistent.

Structural control is a problem if applied simplistically. Schemes may list their structural control in the broadest terms: “present simple”, “indefinite article”, “infinitive form” and so on. The authors’ notes should go into more detail. An infinitive is in level one for going to do, but does that include an infinitive of purpose He went to get some bread? Does it include I’m sorry to hear that? Does it include There’s a lot of work to do? Does it include To be, or not to be, that is the question? Structural control should extend to infinitives and –ing forms, which is the area where some schemes fail to be precise.

Authors seem pretty good at sticking to broad tense guidelines, but suffer from an overall ELT tendency, which is to see everything divided up in terms of verb forms. When I started teaching, an ageing supervisor had designed two separate courses, one labelled “verb grammar” and one “non-verb grammar.” At the time, most of us saw the division as bizarre, but in retrospect it made us more aware of all the other structural areas.

Sentence length and complexity is an area where the long established schemes seem very good. Some newer schemes read to me as if sentence length has not even been considered. As in listening, students need those full stops as pause points. I was appalled when a beginner’s level coursebook which I was working on had semi-colons inserted all over the place by an American editor. I said “We do not use semi-colons at beginner level.” She was as appalled as I had been, but it’s true. At beginner level you don’t use them because you’re not writing complex sentences. You can write within a 600 word level, and with a limited palette of tenses and still be hard work for a student:

Having finished washing up, she went to make the bed, which was old and cheap; they bought it at Habitat in the sixties when Sean, her husband of forty years, was playing at being a student and they paid nine pounds ten shillings for it; and a tenner was a weeks wages in those days, when she worked in an office from 9 to 5 every day and they were young and poor, but happy and in love.

The past perfect was avoided, but what about having finished washing up? Apart from the time problem, while wash and up would be in the word list, that doesn’t mean wash up was. Make the bed? While make and bed would be there, the primary meaning of make would suggest she had an Allen key, bolts and an instruction booklet to hand. What’s a shilling? What’s a tenner?  Is it worth knowing nowadays? What about She went to make … which is an infinitive of purpose? play and be are in the list, but playing at being? What’s Habitat? It means nothing to someone who didn’t live in England.

The gloss, assuming there is one, would have to include wash up, to make a bed and wages. Shilling is a cultural information note, as is tenner. So is habitat. A sensible series editor would ask for it to be rewritten in shorter sentences. They would rewrite those  –ing form sections. Tenner would be replaced by the less colloquial ten pounds. Weights, measures and money are an ongoing problem. Native speakers find It was two hundred metres away jarring in (say) a Dickens adaptation but there’s an argument that the reader needs an idea of the distance, and two hundred metres is immediately transparent, while yards might have to be looked up. In British English we might argue that yards are fast going. In American English we might argue they’re an essential fact.

Some schemes have rules such as ‘only one subordinate clause’. In fact, experienced writers can get around the rules, write longer sentences and still be clear. They can use semi-colons in lists without anyone noticing too.

A minor point on readability is that I wince when I read of Sinead, Illona and José going to meet François and Xavier. Even if there is an audio enclosed, the proper names are going to cause problems in other language groups. Maybe it doesn’t matter if the lone reader without an audio thinks of Sinead as Sin-Ed or Sin-Eed rather than Shin-ade.  A personal irritation is names that are new to me and not obviously gender-marked.

Cultural assumptions cause problems. Years ago, there was a continuing ELT story in a coursebook where a plot hinge relied on knowing that Tottenham is north of Chelsea. If you live in London, the geographical relation of these two districts is obvious. In Guadalajara it will be mystifying. Assumptions have to be avoided and replaced with explanation, which means that adaptations often have to add material. If the British original said They were looking for the nearest Boots the adaptation would be They were looking for the nearest pharmacy. (Boots is the largest high street chain pharmacist). If the original said They drove into Sedona … the adaptation might need to say They drove into Sedona, a small tourist town in the Arizona desert.

Readability consists of largely objective areas. That makes it comparatively easy to assess.

There is a growing discrepancy between readers designed for Western Europe and for International use. Western European based publishers tend to have looser grading, very high vocabulary loads in addition to the headword counts, and often glosses dotted all over the place. Over-marked glosses detract from developing reading flow, and hinder the development of vital guessing abilities. To me, glosses should not break text or dot the page, but are better placed in the back of the book. Others disagree. Some now have translated glosses which make them market specific and impossible to test properly in a variety of countries.

3. Quality of Writing

This is subjective. I’ve read critics who believe Tolkien is a terrible writer and others who look down on Hemingway. A well-told story has flow, pace and a sense of style. Writers can demonstrate a strong style even within the confines of graded readers. They can get complex ideas across simply. Good graded reader writing is a skill, and with some writers it jumps out of the page.

The biggest issue on quality of writing is on the story itself. Here adaptors have an advantage over original writers, in that we already “know” that The Great Gatsby or Wuthering Heights or The Sign of The Four is a good story. It takes considerable skill to change a full length novel into a graded reader, which may only be 5% of the length.

Hollywood producers found early on that short stories by the likes of Ernest Hemingway and Somerset Maugham often made better movies than full length novels. It was creatively more interesting to add rather than subtract for a start. It’s surprising how often adaptors take on the full length novel rather than short stories. Conan Doyle, Saki, Hemingway, Somerset Maugham all provide stories of the length you need for a reader at the lower levels. Film adaptations also rely on a basic pre-existing storyline.

There’s a difference between adapting a “great work of literature” and a short story by a relatively unknown author. Here I think there’s a disadvantage to the great works adaptation. We tend to know the ending! Some secondary school systems see an intrinsic cultural value in students knowing the stories of major English language literary works, and will choose adapted great works of literature over original stories. 

Sometimes, one feels that adaptation, original and non-fiction are three different categories and should be treated as such. In the end, judges have to assess the overall reading experience at the level, regardless of whether it’s original or adapted; fiction or non-fiction. However, there may be a bias in favour of fiction. Several times the comment “You can find this on the internet” comes up with non-fiction. As an author myself, and the series editor of a now out-of-print series of original readers, I find I have to suppress an inherent bias towards originals.

Genre is another factor. Many years ago I was on an author tour of Greece, and lost my carefully-selected reading matter on the way out. With ten days on the road ahead of me, I ransacked OUP’s travelling book exhibition daily and read right through the entire Alpha Books series  of graded readers (which pre-dated Bookworms). They were all adaptations. I found that crime / mystery and sci-fi stood up to adaptation far better than more weighty novels, because the storyline was the most important thing. Something of Jane Austen is lost in a beginner level adaptation. Alpha Books adapted a number of Mills & Boone romance titles in the seventies and we had them as part of a library system in the late 70s. To our surprise we found they were mainly being borrowed by young adult males, not by females. Then we realized they were being read as light porn. I was told by an editor in the 90s that sci-fi “wasn’t popular anymore”. I think “space sci-fi” gets borrowed less, but sci-fi (of the spookily mysterious kind) in a contemporary setting goes extremely well. Again, the story and the twist on reality is the important point.

4. Appropriateness and quality of illustration and graphic presentation

Illustration in a graded reader should support the text. Many illustrations in ERF submissions are simply filler with two talking heads, or a whole page picture of a baby (presumably to illustrate “baby” which was probably in the basic word list anyway).  Good illustration aids the reader in guessing vocabulary without a gloss, and often does so subliminally. Some books have a picture dictionary style pre-page, which is rather overwhelming and indigestible unless done carefully. So often, I can see a plan “Illustrate pages 6, 12, 18, 24, 30 and 36” and that’s about as much thought as goes into it. My illustration notes for readers usually ran to a page of A4 notes per picture. Some graded readers get by without a single useful picture.

Quality of illustration is subjective, but not entirely. Because I’ve been heavily involved in commissioning illustration, I’ve seen portfolios where individual artists have various price levels depending on degree of detail, colour, research needed and so on. Colour doesn’t make bad art good. A sense of style is welcome, but sometimes puts off individual readers who dislike the style. Some readers (especially non-fiction) have a commendably rich mix of illustration, photo, diagram and picture dictionary. However, it does make them look more course book than reader sometimes.

Graphic presentation can be overdone. A reader is a story in a book. Shaded colour bars, glosses in sidebars, colour coded chapters might look nice, but may not assist the task or story. A designer will dislike different fonts or text formats mixed on a page. An excellent rule is “use one marker, or two at most” for headings etc. That is, use bold, or underline, or italic, or colour, or size or insetting or boxing or shading … but you do not need a title to be bold, italic, 20 pt, underlined, coloured, shaded and boxed.

Cover design is important and some cover designs are too abstract, too mundane and too uninviting. Two of the best-selling ELT books ever had dull covers. English Grammar in Use might be the best-known but most boring cover of all. Headway’s abstract cover design told you nothing about what to expect inside. In both cases though, the design said “serious.” A reader is different, in that a true extensive reading program will allow choice, and the reader needs clues as to content and style. In a library system, the cover must invite the reader in.

On several shortlisted entries judges have noted “in spite of the cover” and “in spite of the illustration.” This is heartening because it means the story was judged on its own merits. It’s a pity though, that in library systems, fewer students might read a book because of an uninformative or uninviting cover. On the other hand, the best illustrated books often fail to make the shortlists, so illustration is not given as much weight as other factors. Perhaps there should be a separate award.

Readers are different and should give an idea on the cover of genre, content, style; whether they’re funny or serious, realistic or fantastic, contemporary or historical, fiction or non-fiction (mark non-fiction with photos).

5. Production factors including general design, typography and standard of editing

Who knows whether editor or author is to blame for mishaps? The editor is in the unfortunate blame position in that their job is to catch the errors, and convoluted sentences and departures from the wordlist. The author can blithely follow F. Scott Fitzgerald, who when the editor complained of his constant spelling errors, said that if he didn’t make mistakes, the editor wouldn’t have anything to do.

Paper quality might be commented on, typography, binding … all of it is up for remark

6. Overall summary

This can cover everything above, and anything else of note.


It’s my strong feeling that authors and adaptors should be adequately credited on the front cover of books competing for a writing award. Some publishers give insufficient author credit on an inside page, and in one case, the author was noted in tiny print after the editor and printer at the bottom of the page.


This year, at just a single level, only one out of seventeen readers lacked an audio CD (and maybe the 17th just forgot to supply the CD). This often means shrink wrapping, therefore the front and back covers become a major factor in student choice … they can’t flick the book for pictures or read the first page. Not a single publisher has addressed this adequately. A few years ago, some readers had free bundled audio, some had audio for sale separately, some didn’t have audio, so it was excluded as a consideration. When every single reader has an audio, the quality of the reading (too fast? too slow? Supplying two versions at different speeds?) becomes a factor the ERF might have to assess.

Nor, for library systems, has any publisher given any assistance on storing the CDs with the books. A pocket in the back cover would be nice, but not very useful unless every publisher provides it, because a library would still have to find other ways of dealing with this even if one scheme missed out on the pocket.

This entry was posted in Articles and tagged . Bookmark the permalink.

Comments are closed.