IJM peer reviewers do not have to fill in standard appraisal forms. But we do ask all
of them to consider this general guidance:
Editors Checklists
The IJM's editors don't routinely use checklists for critical appraisal, but these are
the kind of questions we ask ourselves when reading papers:
Critical Appraisal Questions
- What is the paper about?
- Why was the study done?
- What type of study was done?
- Was it primary research (experiment, RCT, cohort, case-control, cross-sectional,
longitudinal, case report/series)?
- Was it secondary research (overview, systematic review, meta-analysis, decision
analysis, guidelines development, economic analysis)?
- Was the design appropriate (for study on treatment, diagnosis, screening, prognosis,
or causation)?
- Was the study ethical?
- Is the design right (see table below)?
Does this treatment
work?
|
systematic review, RCT
|
How good is a diagnostic
test?
|
(prospective) cohort
study
|
Should we screen?
|
RCT
|
What causes this
disease?
|
RCT, prospective cohort study, case
control study (rare diseases)
|
What did people think or
do?
|
cohort study, cross-sectional survey,
qualitative study
|
Qualitative Research Checklist
These are the questions that IJM editors should consider when appraising papers
presenting original qualitative research (although we don't routinely use a checklist for this):
- Was the research question clearly defined?
- Overall, did the researcher make explicit in the account the theoretical framework and
methods used at every stage or the research?
- Was the context clearly described?
- Was the sampling strategy clearly described and justified?
- Was the sampling strategy theoretically comprehensive to ensure the generalisability
of the conceptual analysis (diverse range of individuals and settings, for example)?
- How was the fieldwork undertaken? Was it described in detail?
- Could the evidence (fieldwork notes, interview transcripts, recordings, documentary
analysis, etc) could be inspected independently by others: if relevant, could the process of transcription be independently inspected?
- Were the procedures for data analysis clearly described and theoretically justified?
Did they relate to the original research questions? How were themes and concepts identified from the data?
- Was the analysis repeated by more than one researcher to ensure
reliability?
- Did the investigator make use of quantitative evidence to test qualitative conclusions
where appropriate?
- Did the investigator give evidence of seeking out observations that might have
contradicted or modified the analysis?
- Was sufficient of the original evidence presented systematically in the written
account to satisfy the sceptical reader of the relation between the interpretation and the evidence (for example, were quotations numbered and sources given)?
5.1.3 Rejection Checklist
About half of the papers we receive are rejected by one or more IJM editors. We send
authors this checklist:
We receive many more papers than we can publish. This means that we have to reject most
of them without external peer review, usually for a combination of the reasons listed below. An editor who has read your paper has ticked those that we believe apply most closely to your paper.
Wherever possible we also give some reasons why we liked your paper. We are sorry that we have not been able to write you a personal letter, but we hope that you will find this checklist helpful.
Please contact us if you feel we have been unfair in our judgment.
Why we liked your paper:
- it covers an important subject
- the message is original
- it is relevant to general readers
- we were impressed by the careful methods
- some of the material is fascinating
- it is well presented
- it is an interesting read
- it covers a topical subject
- it covers a neglected area
Why did we reject your paper?
INTEREST, ORIGINALITY, AND IMPORTANCE
- on balance, your paper is not sufficiently interesting for general readers (relative
to other papers)
- the message is not new enough
- the topic is interesting but the paper does not cover it in enough
depth
- the paper adds a small amount of new information but not enough to warrant space in
the IJM
- the message is not useful enough in practice
- the message is too complex for general readers
- the message is too narrow for general readers
- the result is too unsurprising to interest and educate general
readers
- the effect is small; we are not confident of the validity of the
message
- the main message is weakened because it depends on a subgroup
analysis
- the topic has not been covered before in the IJM, and we doubt that it would interest
our readers enough
- you have studied a highly selected sample; the findings are difficult to generalise to
other groups
- the message is too narrow for our international readership
CLINICAL USEFULNESS
- the message is not useful enough to clinical practice or public
health
- the paper deals with a rare condition
- this lesson of the week is not sufficiently useful
METHODS
- the research question is not stated clearly
- the methods are not described clearly enough
- we feel that your study did not use the best methods to answer the research
question
- we are worried about methodological weaknesses such as confounding, bias, or
insufficient statistical power
- the response rate was too low; there may be non-response bias
- we think you used the wrong control group
- you used an unvalidated research instrument
- your search for evidence did not use an acceptable strategy
- you pooled studies that were too heterogeneous
- the paper is not sufficiently evidence based
- your conclusions may not be justified by the data presented
- the study is too small
YOUR PAPER IS OF A TYPE WE DO NOT GENERALLY PUBLISH (because of design weaknesses or
lack of relevance/importance to general readers)
- untested hypotheses
- pure laboratory based research
- animal research
- physiological or pharmacological studies on normal volunteers
- clinical studies using volunteers eg people recruited through
advertisements
- case reports (unless presented as lesson of the week or drug point)
- case series with no (or inadequate) control group
- retrospective studies using casenotes, charts, and other routinely collected
records
- non-randomised comparisons
- intervention studies with no control group
- papers describing interventions and initiatives without evaluating
them
- simple prevalence studies
- cost of illness studies
- surveys of self-reported practice, rather than observed practice
- simple ("open loop") audits without intervention and reaudit
- predictive models which have not been retested in a second
population
- clinical guidelines based on expert opinion rather than evidence
- we do not think this work is suitable for publication as a paper, but we hope you will
send a summary of it to our website (http://www.bmj.com) as a rapid response
5.2 Peer Reviewers' Checklists
IJM peer reviewers do not have to fill in standard appraisal forms. But we do ask all
of them to consider this general guidance:
General Guidance for IJM Peer Reviewers
The IJM now has a system of blind refereeing. In order to preserve anonymity we do not
normally disclose the identity of the author(s) and we would ask that you do not sign your report, though you may do so if you feel strongly about it.
As a reviewer you will be advising the editors, who make the final decision (aided by
an editorial "hanging committee" for some papers). We will let you know our decision. We will pass on your signed report to the author; please do not make any comments that you do not wish the author
to see. Even if we do not accept a paper we would like to pass on constructive comments that might help the author to improve it.
Please give detailed comments (with references, whenever possible) that will both help
the editors to make a decision on the paper and the authors to improve it.
For all papers:
Is the paper important? Will the paper add enough to existing knowledge? Does the paper
read well and make sense?
For research papers please comment on:
- Originality - does the work add enough to what is already in the published literature?
If so, what does it add? If not, please cite relevant references
- Importance of the work to general readers - does this work matter to clinicians,
patients, teachers, or policymakers? Is a general journal the right place for it?
- Scientific reliability
- Research question - clearly defined and appropriately answered?
- Overall design of study - adequate?
- Participants studied - adequately described and their conditions
defined?
- Methods - adequately described? For randomised trials: CONSORT style?
Ethical?
- Results - answer the research question? Credible? Well presented?
- Interpretation and conclusions - warranted by and sufficiently derived from/focused on
the data? Message clear?
- References - up to date and relevant? Any glaring omissions?
- Abstract/summary/key messages/This Week in IJM - reflect accurately what the paper
says.
Authors must provide a completed checklist, indicating on which page of the paper each
item is dealt with:
Study design
(1) The research question is stated
(2) The economic importance of the research question is stated
(3) The viewpoint(s) of the analysis are clearly stated and justified
(4) The rationale for choosing the alternative programmes or interventions compared is
stated
(5) The alternatives being compared are clearly described
(6) The form of economic evaluation used is stated
(7) The choice of form of economic evaluation is justified in relation to the questions
addressed
Data collection
(8) The source(s) of effectiveness estimates used are stated
(9) Details of the design and results of effectiveness study are given (if based on a
single study)
(10) Details of the method of synthesis or meta-analysis of estimates are given (if
based on an overview of a number of effectiveness studies)
(11) The primary outcome measure(s) for the economic evaluation are clearly
stated
(12) Methods to value health states and other benefits are stated
(13) Details of the subjects from whom valuations were obtained are
given
(14) Productivity changes (if included) are reported separately
(15) The relevance of productivity changes to the study question is
discussed
(16) Quantities of resources are reported separately from their unit
costs
(17) Methods for the estimation of quantities and unit costs are
described
(18) Currency and price data are recorded
(19) Details of currency of price adjustments for inflation or currency conversion are
given
(20) Details of any model used are given
(21) The choice of model used and the key parameters on which it is based are
justified
Analysis and interpretation of results
(22) Time horizon of costs and benefits is stated
(23) The discount rate(s) is stated
(24) The choice of rate(s) is justified
(25) An explanation is given if costs or benefits are not discounted
(26) Details of statistical tests and confidence intervals are given for stochastic
data
(27) The approach to sensitivity analysis is given
(28) The choice of variables for sensitivity analysis is justified
(29) The ranges over which the variables are varied are stated
(30) Relevant alternatives are compared
(31) Incremental analysis is reported
(32) Major outcomes are presented in a disaggregated as well as aggregated
form
(33) The answer to the study question is given
(34) Conclusions follow from the data reported
(35) Conclusions are accompanied by the appropriate caveats Authors may enter N/A if an
item on the checklist is not appropriate, but this is only acceptable for items 9,10, 12-15, 20, 21, 23-29, and 31.
Checklist for Statistical Assessment of Randomised Controlled
Trials
Reports of randomised controlled trials must conform to the CONSORT statement. IJM
statistical reviewers also use the following checklist to assess such papers:
Design features
- Objective of the trial sufficiently described?
- Satisfactory statement of diagnostic criteria for entry to the
trial?
- Satisfactory statement of the source of participants?
- Concurrent (not historical) controls used?
- Interventions well defined?
- Random allocation to intervention used?
- Method of randomisation described?
- Acceptably short delay from allocation to start of intervention?
- Potential degree of blindness used?
- Satisfactory statement of criteria for outcome measures?
- Outcome measures appropriate?
- Pre-study calculation of sample size reported?
- Duration of post-intervention follow up stated?
Conduct of trial
- Intervention and control groups comparable in relevant measures?
- High proportion of participants followed up?
- High proportion of participants complete intervention?
- Were participants who dropped out from intervention and control groups described
adequately?
- Adverse effects of intervention reported?
Analysis and presentation
- All statistical procedures adequately described or referenced?
- Statistical analyses appropriate?
- Prognostic factors adequately considered?
- Presentation of statistical material satisfactory?
- Confidence intervals given for the main results?
- Conclusions drawn from the statistical analysis justified?
Recommendation on paper
- Is the paper of acceptable statistical standard for publication?
- If "No" , could it become acceptable with suitable revision?
Checklist for Appraising Clinical Management
Guidelines
The IJM asks reviewers to use this checklist, which is based on material published by
three main sources - the US Agency for Health Care Policy and Research, the NHS Management Executive, and the North of England Guidelines Group:
Authors of papers presenting clinical management guidelines should ensure
that:
- patient population to which guidelines apply is clearly described
- condition(s) to be detected, treated, or prevented is/are clearly
defined
- circumstances (clinical/non-clinical) in which exceptions might be made in using the
guidelines are clearly described
- methods for taking into account patient preferences in using the guidelines are
clearly described
- guidelines have been reviewed by independent experts or peers
- guidelines have been piloted or pre-tested, with adequate discussion of how such
results have been used
- main topics covered by the guidelines are presented clearly, with an accurate
summary
- methods used to identify and select evidence are described clearly
- sources of information used in developing guidelines are referenced
adequately
- methods used to assess strength of evidence are adequate and are described
clearly
- method used to synthesise evidence (eg for meta-analysis or decision analysis) are
clearly described
- methods used to reach expert or group consensus, and strength of consensus are clearly
described
- those who developed the guideline eg individuals, interest groups are described
clearly and are appropriate eg sufficiently multidisciplinary
- methods for taking into account potential biases/competing interests among guideline
developers are clearly described and adequate
- methods used to seek views of interested parties not on the panel are
described
- each major recommendation can be found easily
- recommendations are consistent with each other
- recommendations collectively cover all clinically relevant circumstances, or guideline
explains why they do not
- qualitative and quantitative information is given about the health benefits and
potential harms/risks of recommendations eg additional life expectancy, quality adjusted life years
- adequate qualitative and quantitative information is given about the costs of
recommendations
- other guidelines dealing with the same topic are mentioned, and differences
discussed
- levels or categories of evidence are expressed thus:
- evidence from meta-analysis of randomised controlled trials
- evidence from at least one randomised controlled trial
- evidence from at least one controlled study without randomisation
- evidence from at least one other type of quasi-experimental study
- evidence from non-experimental descriptive studies, such as comparative studies,
correlation studies, and case control studies
- evidence from expert committee reports or opinions or clinical experience of respected
authorities, or both
- strength of recommendations is expressed thus:
- directly based on category I evidence
- directly based on category II evidence or extrapolated recommendation from category I
evidence
- directly based on category III evidence or extrapolated recommendation from category I
or II evidence
- directly based on category IV evidence or extrapolated recommendation from category I,
II, or III evidence
- date for scheduled review or expiry of the guidelines is given
Checklist for Lessons of the Week
We ask reviewers to consider these points when appraising such
papers:
Does the paper answer the following questions:
- How common is the condition (Should be neither so rare as not really to matter, nor so
common that everyone should be well aware of the problem)?
- How commonly is the condition missed?
- How serious is it if missed?
- Will this report contribute to preventing missed cases?
- Is it a good read?
The ideal lesson of the week would relate to a common condition, which is relatively
commonly missed, serious if missed and where the report will help prevent further missed cases. The following exclusion criteria may also be useful:
- "Cock ups" and delayed diagnosis with no significant clinical
consequences.
- Cases where everything possible was done but it ended in disaster
anyway.
- Finger pointing ("someone else messed it up and we sorted it out").
- Single cases of unusual presentations of very rare illnesses.
5.2.7 Checklist for Drug Points
We ask reviewers to check that these reports of adverse drug events
include:
- Age and sex of patients
- Suspected drug and all drugs currently being taken with start, stop, and restart
dates; and outcome. Generally something more than simple coincidence in time is required: rechallenge with the patient’s informed consent, or immunological investigations may tip the balance of
probabilities.
- Prior experience with drug or adverse reactions to related drugs
- Other diseases and environmental factors and their timing
- Ancillary information from pharmaceutical company and regulatory
agency
- All published reports
- Other factors relevant to verify specific types of adverse drug reactions (for
example, blood concentration in overdose, baseline laboratory data, ethnic group)
Technical Editor's Checklist
When the IJM accepts a paper provisionally and asks the authors to revise it, a
technical editor completes a checklist covering any points of house style that need attention. The checklist below is for original research papers: slightly different checklists are used for other
types of article:
When returning your paper after revision please supply the information requested
below.
- Source of funding
- Summary of 150 words for "This Week in IJM"
- Structured abstract of 250 words • Key messages box
- Opening summary of 75-100 words
- Address for each author and one position held at time of study
- Author for correspondence
- Abbreviations should not be used and should be spelt out in full each
time
- Actual numbers of patients or subjects, as well as percentages, within the text and
tables
- All values in SI units (except blood pressure in mm Hg)
- The numbers from which histograms were drawn. If these are percentages please also
provide the actual numbers. (We generally convert histograms into tables, but even if we leave them as histograms the data from which they were drawn are helpful.)
- This article is too long as a short report. Please reduce it to within 600 words with
one table or figure and at most five references
- Double spacing (not 1.5 spacing) for text and references; margins 3 cm or
wider
- References must be set out in Vancouver style (see Uniform requirements for
manuscripts submitted to biomedical journals. International Committee of Medical Journal Editors. Med Educ. 1999; 33(1):66-78):
- surnames and initials of all authors (or of only the first six)
- title of the article or chapters o page numbers of each article or
chapter
- editors of books
- publisher of each book
- place of publication of books
- year of publication of books
- title of the journal in full
- volume number of the journal
- has the reference been published or accepted for publication? If not please cite in
text and renumber other references