Reviewers' Checklists

IJM peer reviewers do not have to fill in standard appraisal forms. But we do ask all of them to consider this general guidance:

Editors Checklists

The IJM's editors don't routinely use checklists for critical appraisal, but these are the kind of questions we ask ourselves when reading papers:

Critical Appraisal Questions

• What is the paper about?
• Why was the study done?
• What type of study was done?
• Was it primary research (experiment, RCT, cohort, case-control, cross-sectional, longitudinal, case report/series)?
• Was it secondary research (overview, systematic review, meta-analysis, decision analysis, guidelines development, economic analysis)?
• Was the design appropriate (for study on treatment, diagnosis, screening, prognosis, or causation)?
• Was the study ethical?
• Is the design right (see table below)?

Qualitative Research Checklist

These are the questions that IJM editors should consider when appraising papers presenting original qualitative research (although we don't routinely use a checklist for this):

• Was the research question clearly defined?
• Overall, did the researcher make explicit in the account the theoretical framework and methods used at every stage or the research?
• Was the context clearly described?
• Was the sampling strategy clearly described and justified?
• Was the sampling strategy theoretically comprehensive to ensure the generalisability of the conceptual analysis (diverse range of individuals and settings, for example)?
• How was the fieldwork undertaken? Was it described in detail?
• Could the evidence (fieldwork notes, interview transcripts, recordings, documentary analysis, etc) could be inspected independently by others: if relevant, could the process of transcription be independently inspected?
• Were the procedures for data analysis clearly described and theoretically justified? Did they relate to the original research questions? How were themes and concepts identified from the data?
• Was the analysis repeated by more than one researcher to ensure reliability?
• Did the investigator make use of quantitative evidence to test qualitative conclusions where appropriate?
• Did the investigator give evidence of seeking out observations that might have contradicted or modified the analysis?
• Was sufficient of the original evidence presented systematically in the written account to satisfy the sceptical reader of the relation between the interpretation and the evidence (for example, were quotations numbered and sources given)?

5.1.3 Rejection Checklist

About half of the papers we receive are rejected by one or more IJM editors. We send authors this checklist:

We receive many more papers than we can publish. This means that we have to reject most of them without external peer review, usually for a combination of the reasons listed below. An editor who has read your paper has ticked those that we believe apply most closely to your paper. Wherever possible we also give some reasons why we liked your paper. We are sorry that we have not been able to write you a personal letter, but we hope that you will find this checklist helpful. Please contact us if you feel we have been unfair in our judgment.

Why we liked your paper:

• it covers an important subject
• the message is original
• it is relevant to general readers
• we were impressed by the careful methods
• some of the material is fascinating
• it is well presented
• it is an interesting read
• it covers a topical subject
• it covers a neglected area

Why did we reject your paper?

INTEREST, ORIGINALITY, AND IMPORTANCE

• on balance, your paper is not sufficiently interesting for general readers (relative to other papers)
• the message is not new enough
• the topic is interesting but the paper does not cover it in enough depth
• the paper adds a small amount of new information but not enough to warrant space in the IJM
• the message is not useful enough in practice
• the message is too complex for general readers
• the message is too narrow for general readers
• the result is too unsurprising to interest and educate general readers
• the effect is small; we are not confident of the validity of the message
• the main message is weakened because it depends on a subgroup analysis
• the topic has not been covered before in the IJM, and we doubt that it would interest our readers enough
• you have studied a highly selected sample; the findings are difficult to generalise to other groups
• the message is too narrow for our international readership

CLINICAL USEFULNESS

• the message is not useful enough to clinical practice or public health
• the paper deals with a rare condition
• this lesson of the week is not sufficiently useful

METHODS

• the research question is not stated clearly
• the methods are not described clearly enough
• we feel that your study did not use the best methods to answer the research question
• we are worried about methodological weaknesses such as confounding, bias, or insufficient statistical power
• the response rate was too low; there may be non-response bias
• we think you used the wrong control group
• you used an unvalidated research instrument
• your search for evidence did not use an acceptable strategy
• you pooled studies that were too heterogeneous
• the paper is not sufficiently evidence based
• your conclusions may not be justified by the data presented
• the study is too small

YOUR PAPER IS OF A TYPE WE DO NOT GENERALLY PUBLISH (because of design weaknesses or lack of relevance/importance to general readers)

• untested hypotheses
• pure laboratory based research
• animal research
• physiological or pharmacological studies on normal volunteers
• clinical studies using volunteers eg people recruited through advertisements
• case reports (unless presented as lesson of the week or drug point)
• case series with no (or inadequate) control group
• retrospective studies using casenotes, charts, and other routinely collected records
• non-randomised comparisons
• intervention studies with no control group
• papers describing interventions and initiatives without evaluating them
• simple prevalence studies
• cost of illness studies
• surveys of self-reported practice, rather than observed practice
• simple ("open loop") audits without intervention and reaudit
• predictive models which have not been retested in a second population
• clinical guidelines based on expert opinion rather than evidence
• we do not think this work is suitable for publication as a paper, but we hope you will send a summary of it to our website (http://www.bmj.com) as a rapid response

5.2 Peer Reviewers' Checklists

IJM peer reviewers do not have to fill in standard appraisal forms. But we do ask all of them to consider this general guidance:

General Guidance for IJM Peer Reviewers

The IJM now has a system of blind refereeing. In order to preserve anonymity we do not normally disclose the identity of the author(s) and we would ask that you do not sign your report, though you may do so if you feel strongly about it.

As a reviewer you will be advising the editors, who make the final decision (aided by an editorial "hanging committee" for some papers). We will let you know our decision. We will pass on your signed report to the author; please do not make any comments that you do not wish the author to see. Even if we do not accept a paper we would like to pass on constructive comments that might help the author to improve it.

Please give detailed comments (with references, whenever possible) that will both help the editors to make a decision on the paper and the authors to improve it.

For all papers:

Is the paper important? Will the paper add enough to existing knowledge? Does the paper read well and make sense?

For research papers please comment on:

• Originality - does the work add enough to what is already in the published literature? If so, what does it add? If not, please cite relevant references
• Importance of the work to general readers - does this work matter to clinicians, patients, teachers, or policymakers? Is a general journal the right place for it?
• Scientific reliability
  • Research question - clearly defined and appropriately answered?
  • Overall design of study - adequate?
  • Participants studied - adequately described and their conditions defined?
  • Methods - adequately described? For randomised trials: CONSORT style? Ethical?
  • Results - answer the research question? Credible? Well presented?
  • Interpretation and conclusions - warranted by and sufficiently derived from/focused on the data? Message clear?
  • References - up to date and relevant? Any glaring omissions?
  • Abstract/summary/key messages/This Week in IJM - reflect accurately what the paper says.

Authors must provide a completed checklist, indicating on which page of the paper each item is dealt with:

Study design

(1) The research question is stated

(2) The economic importance of the research question is stated

(3) The viewpoint(s) of the analysis are clearly stated and justified

(4) The rationale for choosing the alternative programmes or interventions compared is stated

(5) The alternatives being compared are clearly described

(6) The form of economic evaluation used is stated

(7) The choice of form of economic evaluation is justified in relation to the questions addressed

Data collection

(8) The source(s) of effectiveness estimates used are stated

(9) Details of the design and results of effectiveness study are given (if based on a single study)

(10) Details of the method of synthesis or meta-analysis of estimates are given (if based on an overview of a number of effectiveness studies)

(11) The primary outcome measure(s) for the economic evaluation are clearly stated

(12) Methods to value health states and other benefits are stated

(13) Details of the subjects from whom valuations were obtained are given

(14) Productivity changes (if included) are reported separately

(15) The relevance of productivity changes to the study question is discussed

(16) Quantities of resources are reported separately from their unit costs

(17) Methods for the estimation of quantities and unit costs are described

(18) Currency and price data are recorded

(19) Details of currency of price adjustments for inflation or currency conversion are given

(20) Details of any model used are given

(21) The choice of model used and the key parameters on which it is based are justified

Analysis and interpretation of results

(22) Time horizon of costs and benefits is stated

(23) The discount rate(s) is stated

(24) The choice of rate(s) is justified

(25) An explanation is given if costs or benefits are not discounted

(26) Details of statistical tests and confidence intervals are given for stochastic data

(27) The approach to sensitivity analysis is given

(28) The choice of variables for sensitivity analysis is justified

(29) The ranges over which the variables are varied are stated

(30) Relevant alternatives are compared

(31) Incremental analysis is reported

(32) Major outcomes are presented in a disaggregated as well as aggregated form

(33) The answer to the study question is given

(34) Conclusions follow from the data reported

(35) Conclusions are accompanied by the appropriate caveats Authors may enter N/A if an item on the checklist is not appropriate, but this is only acceptable for items 9,10, 12-15, 20, 21, 23-29, and 31.

Checklist for Statistical Assessment of Randomised Controlled Trials

Reports of randomised controlled trials must conform to the CONSORT statement. IJM statistical reviewers also use the following checklist to assess such papers:

Design features

• Objective of the trial sufficiently described?
• Satisfactory statement of diagnostic criteria for entry to the trial?
• Satisfactory statement of the source of participants?
• Concurrent (not historical) controls used?
• Interventions well defined?
• Random allocation to intervention used?
• Method of randomisation described?
• Acceptably short delay from allocation to start of intervention?
• Potential degree of blindness used?
• Satisfactory statement of criteria for outcome measures?
• Outcome measures appropriate?
• Pre-study calculation of sample size reported?
• Duration of post-intervention follow up stated?

Conduct of trial

• Intervention and control groups comparable in relevant measures?
• High proportion of participants followed up?
• High proportion of participants complete intervention?
• Were participants who dropped out from intervention and control groups described adequately?
• Adverse effects of intervention reported?

Analysis and presentation

• All statistical procedures adequately described or referenced?
• Statistical analyses appropriate?
• Prognostic factors adequately considered?
• Presentation of statistical material satisfactory?
• Confidence intervals given for the main results?
• Conclusions drawn from the statistical analysis justified?

Recommendation on paper

• Is the paper of acceptable statistical standard for publication?
• If "No" , could it become acceptable with suitable revision?

Checklist for Appraising Clinical Management Guidelines

The IJM asks reviewers to use this checklist, which is based on material published by three main sources - the US Agency for Health Care Policy and Research, the NHS Management Executive, and the North of England Guidelines Group:

Authors of papers presenting clinical management guidelines should ensure that:

• patient population to which guidelines apply is clearly described
• condition(s) to be detected, treated, or prevented is/are clearly defined
• circumstances (clinical/non-clinical) in which exceptions might be made in using the guidelines are clearly described
• methods for taking into account patient preferences in using the guidelines are clearly described
• guidelines have been reviewed by independent experts or peers
• guidelines have been piloted or pre-tested, with adequate discussion of how such results have been used
• main topics covered by the guidelines are presented clearly, with an accurate summary
• methods used to identify and select evidence are described clearly
• sources of information used in developing guidelines are referenced adequately
• methods used to assess strength of evidence are adequate and are described clearly
• method used to synthesise evidence (eg for meta-analysis or decision analysis) are clearly described
• methods used to reach expert or group consensus, and strength of consensus are clearly described
• those who developed the guideline eg individuals, interest groups are described clearly and are appropriate eg sufficiently multidisciplinary
• methods for taking into account potential biases/competing interests among guideline developers are clearly described and adequate
• methods used to seek views of interested parties not on the panel are described
• each major recommendation can be found easily
• recommendations are consistent with each other
• recommendations collectively cover all clinically relevant circumstances, or guideline explains why they do not
• qualitative and quantitative information is given about the health benefits and potential harms/risks of recommendations eg additional life expectancy, quality adjusted life years
• adequate qualitative and quantitative information is given about the costs of recommendations
• other guidelines dealing with the same topic are mentioned, and differences discussed
• levels or categories of evidence are expressed thus:
  • evidence from meta-analysis of randomised controlled trials
  • evidence from at least one randomised controlled trial
  • evidence from at least one controlled study without randomisation
  • evidence from at least one other type of quasi-experimental study
  • evidence from non-experimental descriptive studies, such as comparative studies, correlation studies, and case control studies
  • evidence from expert committee reports or opinions or clinical experience of respected authorities, or both
• strength of recommendations is expressed thus:
  • directly based on category I evidence
  • directly based on category II evidence or extrapolated recommendation from category I evidence
  • directly based on category III evidence or extrapolated recommendation from category I or II evidence
  • directly based on category IV evidence or extrapolated recommendation from category I, II, or III evidence
• date for scheduled review or expiry of the guidelines is given

Checklist for Lessons of the Week

We ask reviewers to consider these points when appraising such papers:

Does the paper answer the following questions:

1. How common is the condition (Should be neither so rare as not really to matter, nor so common that everyone should be well aware of the problem)?
2. How commonly is the condition missed?
3. How serious is it if missed?
4. Will this report contribute to preventing missed cases?
5. Is it a good read?

The ideal lesson of the week would relate to a common condition, which is relatively commonly missed, serious if missed and where the report will help prevent further missed cases. The following exclusion criteria may also be useful:

1. "Cock ups" and delayed diagnosis with no significant clinical consequences.
2. Cases where everything possible was done but it ended in disaster anyway.
3. Finger pointing ("someone else messed it up and we sorted it out").
4. Single cases of unusual presentations of very rare illnesses.

5.2.7 Checklist for Drug Points

We ask reviewers to check that these reports of adverse drug events include:

• Age and sex of patients
• Suspected drug and all drugs currently being taken with start, stop, and restart dates; and outcome. Generally something more than simple coincidence in time is required: rechallenge with the patient’s informed consent, or immunological investigations may tip the balance of probabilities.
• Prior experience with drug or adverse reactions to related drugs
• Other diseases and environmental factors and their timing
• Ancillary information from pharmaceutical company and regulatory agency
• All published reports
• Other factors relevant to verify specific types of adverse drug reactions (for example, blood concentration in overdose, baseline laboratory data, ethnic group)

Technical Editor's Checklist

When the IJM accepts a paper provisionally and asks the authors to revise it, a technical editor completes a checklist covering any points of house style that need attention. The checklist below is for original research papers: slightly different checklists are used for other types of article:

When returning your paper after revision please supply the information requested below.

• Source of funding
• Summary of 150 words for "This Week in IJM"
• Structured abstract of 250 words • Key messages box
• Opening summary of 75-100 words
• Address for each author and one position held at time of study
• Author for correspondence
• Abbreviations should not be used and should be spelt out in full each time
• Actual numbers of patients or subjects, as well as percentages, within the text and tables
• All values in SI units (except blood pressure in mm Hg)
• The numbers from which histograms were drawn. If these are percentages please also provide the actual numbers. (We generally convert histograms into tables, but even if we leave them as histograms the data from which they were drawn are helpful.)
• This article is too long as a short report. Please reduce it to within 600 words with one table or figure and at most five references
• Double spacing (not 1.5 spacing) for text and references; margins 3 cm or wider
• References must be set out in Vancouver style (see Uniform requirements for manuscripts submitted to biomedical journals. International Committee of Medical Journal Editors. Med Educ. 1999; 33(1):66-78):
  • surnames and initials of all authors (or of only the first six)
  • title of the article or chapters o page numbers of each article or chapter
  • editors of books
  • publisher of each book
  • place of publication of books
  • year of publication of books
  • title of the journal in full
  • volume number of the journal
  • has the reference been published or accepted for publication? If not please cite in text and renumber other references