Writing a Literature Review - Barbara Steward

Literature reviews are often criticized for several reasons such as failing to identify or fulfil a clear purpose, being too narrow in focus or limited in the extent of synthesis and analysis they contain, and uncritical and unstructured descriptions of unexplained collections of publications. The purpose of this paper is to consider how to report a review for journal publication.

Types of Literature Review

A review is an attempt to arrive at some conclusions about a body of evidence through the process of synthesis and analysis.

Systematic Literature Review: A systematic literature review may be a research project in its own right that follows a specific methodology to combine reliable results to create a new database for analysis. The Cochrane Collaboration (2003) stated that a systematic review aims to establish where the effects of health care are consistent and where research results can be applied across populations, settings and differences in treatment. The methods aim to limit bias and reduce chance effects. This provides more reliable results which can be used to draw conclusions and make decisions. For example, a meta-analysis of quantitative research evidence uses statistical methods to summarise the results of independent studies to offer more precise estimates (of the effects of healthcare) than those derived from individual studies.

Literature Reviews As Secondary Data Analysis Research Projects: In this type of research project, secondary data analysis of all types of published and grey data (unpublished documents, conference papers, theses and internal reports) is attained to identify important themes and debates and synthesise research evidence to reach new forms of understanding based on diverse evidence. It uses more qualitative thematic analytical procedures. Sometimes referred to as extended literature reviews.

Literature Reviews As Part Of Primary Research: This form of literature review aims to map the broad context in which the study/problem lies, identify key theories and theorists, and locate and define concepts and terms. It explains the selection of the research question and the methodology. It forms as a basis for discussing the results.

A literature review serves to offer the reader a response to the research question through an overview of the evidence, key themes, debates and controversies, with a fully supported interpretation of the data. The focus is on an analysis of the discussion and theoretical aspects of the topic. A good literature review should be (according to Steward):

• Comprehensive: collating evidence from all relevant disciplines.

• Fully referenced: it offers reliable access routes to the evidence.

• Selective: it uses appropriate inclusion and exclusion criteria to identify key evidence.

• Relevant: a focus on evidence that is pertinent to the research question.

• A synthesis of ideas and key themes.

• Balanced: a comparative account of differing theories and practices.

• Critical: evaluate the rigour of evidence or argument in the data.

• Analytical: developing new understandings from the available evidence.

A literature review should understand the context under which a review has been undertaken, and the protocols applied to access, select, and evaluate the trustworthiness of this knowledge. “A review must offer more than a confirmation of the views held by the researcher or the evidence located within a narrow discipline; it must introduce alternative perspectives from the widest range of academic disciplines, fairly representing opposing views”. “Too often reviews come across as dry and impenetrable lists of poorly debated facts, randomly selected and referenced, making the reader feel deprived of real substance, swamped by over-inclusion or frustrated by unchallenged opinion”.

Literature reviews as secondary research are sometimes seen as the poor relatives of primary research. “Those who can, research; those who can’t, review”. Sometimes they are seen as the boring prelude to the ‘real’ substance of primary research. This paper hopes to encourage authors to publish reviews as articles by "conveying the fascination of historical developments and controversies and seizing the opportunity to offer a reasoned and balanced but personal interpretation of the evidence on a topic".

Literature Reviews As Secondary Data Analysis Research Projects.

Because reviews can be research projects, they should begin with a research question or specific focus inquiry. The author needs to ground the review within a context to explain why a secondary research design is appropriate, now and in this manner. They need to justify the choice of data, the choice of amplifying or limiting a selection of evidence and the method planned for analysis. A review can be undertaken where there is very little published data or where the wealth of evidence is substantially large, which will then determine the depth or breadth of the analysis.

There are several methods to conducting literature reviews which offer repertoires of procedures to be followed and frameworks in which to report the methods and findings. Different review protocols are also welcomed as they encourage the notion that reviewing is not a prescribed activity that is achieved through rigid adherence to a single set of rules.

The position and bias of the reviewer must be explained and debated within a background or rationale statement in the same way that a researcher states the position taken in terms of theory generation, theory exploration and theory verification in primary research. Everyone has bias; "it is important to make the reader aware of who the reviewer is, why he or she is doing the review on this topic at this time, and his or her understanding of the topic before undertaking the review".

In a research report, the author must indicate the precise methods used to search and gather literature, identify themes and appraise the quality of the evidence and the analytical approach. "The reviewer has to satisfy the reader of the rigour of the review and the trustworthiness of the conclusions. Bruce (1994) suggested that reviews may be understood as being both process and product. The process relates to the formulation of a protocol to investigate the topic and a defence of that choice, as well as a reasoned debate comparing the strategies by which others have generated their opinions with those of the reviewer. The product is a synthesis of the evidence, which enables the reader to grasp quickly and easily the nature of the evidence available and to follow the critical reasoning that has led to its final analytical interpretation. A literature review collates and summarises the evidence, renders it comprehensible and appraises its weight and value in order to offer an opinion grounded within the evidence".

Literature Reviews As Part Of Primary Research

Literature reviews as part of primary research are almost the same in process, but the product serves to explain, justify (rationalise) and contextualise the research question and its methodology. By the end of the review, the reader should be clear about what is known, with the concepts and language defined and the relevant gaps in the existing data (which the planned research aims to fill) identified. "The synthesis and critical appraisal of the selected evidence remains the same as for secondary data analysis research projects, although the level of description might be more superficial and the analysis more tentative".

It is crucially important that the reader is adequately informed about the context and the historical and/or developmental evidence in which the current research is located and based. It is anticipated that the findings of the primary research project will be discussed in relation to existing evidence. "Depending on the methodology, the literature review may form an essential discrete preliminary investigation, as in most quantitative research, or a subsequent aspect of a qualitative study, as in grounded theory. The author has to inform the reader how and when the review of existing evidence has been sought and will be introduced into the report or article".

Common Strategies For Secondary Research And Reviews For Primary Research.

Searching

The method should state in detail the databases explored, the key words and search terms used and the inclusion and exclusion criteria applied in the sample selection of data. If specific decisions were made to limit the search to a certain discipline, these choices must be stated and justified. Other limits to the searches, such as language or date of publication, have to be similarly explained. The search should differentiate research-based from non-research-based evidence.

The search method has to be as reproducible as the methods in primary research. Readers should be satisfied with the thoroughness and rigour of the search and selection procedure. They also need to be impressed by how the author has captured a world of knowledge for them. Barbara likens it to a travel book, where the author offers a concise but fascinating account of where they have been, what was found, what sense was made out of it and how, if other readers went there too, the work that would help them navigate themselves and communicate more efficiently.

A useful procedure is to refine the process of searching from the general to the specific: Reviews can offer a generalised map of the topic, providing an indication of the widest parameters of the search and the way in which the topic is broadly viewed and understood. The review can then increasingly focus on the specifics of the topic under investigation, funnelling down further to a small number of key articles that offer direct evidence to answer the research question and are described and discussed in detail.

"Reviews should not only draw from diverse academic disciplines, but also from a wide range of sources, including citation searches, government documentation, in-house reports, conference abstracts and internet publications". Searching should go beyond what appears at a formal published level to what is going on in practice. All data are admissible, but must be identified and referenced appropriately. "Where specific evidence about the topic is not available, other related evidence must be sought". If there are gaps where the reviewer would expect evidence, this also needs to be stated.

A report of a review must describe the route of the exploration. "It must explain how search terms were selected, refined and combined to ensure a comprehensive exploration of relevant and related data. It is disappointing to read reviews that have limited their search to three or four terms, searches limited to titles, or key-word searches in only Medline and CINAHL, for example. A good review should show evidence of catholic inquiry, including attempts to use MeSH (Medical Subject Headings) terms (Fink 1998), capture relevant concepts or synonyms, or use key author and citation searches. It is important to mention where hand searches have been undertaken to fill the gaps resulting from the inevitable limitations of electronic searching, which often relies on title, abstract, author and key-word searches. A review should indicate the strategies adopted when using the internet to limit these problems (Isaacs and Isaacs 1998)".

"There has been a longstanding and well known publication bias towards the submission and acceptance of positive research findings (Dickersin 1990, Easterbrook et al 1991). A reviewer may wish to identify how he or she has sought to identify negative research evidence through other sources to provide balance. Reviews should also try to report minority opinions through the investigations and views of lesser known people from less expected places".

Appraising

It is common to establish criteria against which the type and quality of evidence is evaluated. "It is important to explain how and why a critical method was applied to different types of evidence located in phenomenological, ethnographic, action research, randomised controlled trial, case or clinical studies. The critical method must be sensitive to the format of the publication. It is not uncommon, for example, to see qualitative papers being negatively evaluated for having too few participants, suggesting that the inclusion of more people would have produced more reliable results. Reviewers need to show how they used the right ‘horses for courses".

Appraisal is a process of critiquing the evidence of other authors, not attacking their work or character. To this end, articles that are peer-reviewed, approved by ethics or research governance committees or cited by others have value. All research projects have weaknesses, some caused or exacerbated by the limitations imposed in publication. A reviewer should be moderate in his or her criticism. Several professional bodies now offer systems to classify the rigour of research articles for review purposes, particularly quantitative research. "Various guidelines for qualitative research (Blaxter et al 1996, Mays and Pope 2000, Morse 2003, Seale et al 2004) can be used to provide a process and structure to the report".

An appraisal may tabulate the articles with their hierarchical grades or enumerate them in some other way. This is called a literature matrix, which is a simple chart listing the author and title, the research question, method, sample size and key findings as well as its ‘grade’. However, these proceduralist methods have their restrictions and the procedures of qualitative research may divert attention away from the essence of its findings.

Synthesising and Analysing

A review is more than a matrix or list. It is an exploration that brings evidence together under themes or constructs. This can reflect histories, conflicts, revolutions or evolutions. The reader should get a taste of how the topic is lived, perceived, experienced and shaped by its past. It is structured account of evidence that has been brought together to make it easier to understand and, importantly, to show how the author has made sense of it. It should enable the reader to take a position or support their opinion in the matter/topic.

Because each reviewer will interpret articles differently, the report has to describe the abalytical process to assure readers that the reviewer has a professional and trustworthy command of the evidence. The analysis of both quantitative and qualitative research involves looking at the various individual and combined results and the subsequent discussion of the data to suggest new working theories.

A suggested conceptual frame: generalised and focused mapping

So, a good review can work like a historical atlas and tourist guide, showing the reader what is out there, offering different perspectives of the same world, indicating the complexity of its layout and providing a course through which he or she is guided to see the most interesting and important sites. A review should describe and debate:

• How the topic has been explored and within which disciplines it has been charted (e.g. databases, key words, dates searched).

• The key/significant features that have been identified.

• How the boundaries of the topic are currently established.

• How the topic has developed or changed over time.

• The key explorers/authors and their perspectives of the topic.

• The contested areas and discontents.

• Where and how these debates play out.

• The language used, including define certain words.

• What ordinary people are doing and whether practice matches theory.

The reader should be left knowing more about the topic, aware of the evidence on which the reviewer bases their new analysis and be able to find the relevant information themselves.

Literature Reviews As Part Of Primary Research Reports

All of the above issues also apply to these reviews but, as they are constrained by word limits, must attempt to summarise and condense the method, findings and analysis. In addition, the review must justify why the research is needed. Its role is to show that there is no duplication of previous research, but that there is a continuation and development of existing investigations and theory building. The review provides the research context, identifies the gaps and indicates the methodology and method that will fill that void.

A review should end with a response to the question “So what?”. What is the outcome of reading this body of literature in relation to the research question. The end result should offer a ‘good read’, easily accessible without academic aloofness, targeted at a given audience and providing a good reflection of the interest, involvement and perhaps even the passion of searching and analysing. The following summarises information that might be considered for inclusion:

• The research question or topic.

• The electronic databases used.

• Electronic search strategies and referencing software used.

• Other search methods such as citations or hand searching.

• Methods used to locate and access grey dat and other non-journal literature.

• Other forms of gathering data, such as consultations with experts or authors.

• Establishing and justification of inclusion and exclusion criteria.

• Method of synthesis. E.g. categorisation of evidence.

• Critical appraisal and analytical methods.

• Appraisal matrix/list.

• Critical evaluation of review method.

• Analysis

• Implications for research and practice. “So what?”

A poor review is:

• Opinionated.

• Unbalanced (only positive evidence).

• Uncritical.

• Overgeneralised.

• Lacking in focus or outcome.