Introduction

The conduct of a scholarly literature review, particularly a formal systematic review, ideally involves the a priori identification of criteria for selection of studies to be reviewed [1]. Typically the identification process focuses on the language and year of publication, whether an abstract is readily viewable upon an initial digital search, the quality of the journal and the country in which either the study was performed or where the primary investigators are based. A second set of criteria typically focuses on the measurable quality of the studies being reviewed; these include whether included papers should embrace studies on human or animal subjects, the type and rigour of study design, sample size and other appreciable metrics of methodological quality [2].

The latter set of criteria filters candidate studies according to their scientific virtues. The former set are, in large part, applied for reasons of convenience. This paper draws attention to an element of the first set, the criterion of country of study origin, though assumptions surrounding the other criteria of convenience also warrant some scrutiny. Reviews conducted in a particular nation by researchers of a particular culture, dealing with subject populations representing a generalisable culture or community, defensibly select review studies originating from the same or similar countries, cultures or communities.

In the specific case of Canadian researchers seeking to review studies pertinent to a Canadian population, it is not unusual to a priori decide to include only papers originating from jurisdictions assumed to be similar to the Canadian population. Typically, these jurisdictions are: USA, UK, Australia and New Zealand. The inclusion of non-English speaking nations in Western Europe occurs less frequently, but is not uncommon.

There are unspoken and unexamined assumptions inherent in using the papers of these nations alone to approximate the Canadian case. The nations of Canada, USA, UK, Australia and New Zealand enjoy a shared colonial history and dominant language, and comparable political structures, economies and alliances. However, other factors may be more pertinent in establishing whether a given nation’s population is an appropriate comparator for the Canadian case; a more systematic approach is thus indicated.

In light of a tendency to reflexively assume that nations with shared colonial histories are more likely to be more comparable, in terms of population health research needs, this study was developed demonstrate an approach for comparing all putative nations, such that more objective criteria can be applied to determine overt comparability.

We present one example of a systematic approach for deciding which nation’s populations are sufficiently similar to Canada’s, for the purposes of inclusion in a systematic review of health literature. As described in the methods section, a checklist of nine criteria, both systemic and sociodemographic, was developed for establishing appropriate similarity of nations; and this checklist was then applied to 68 test jurisdictions. While the example herein is specific to Canada, the grander methodology and motivation are applicable for other nations and other contexts, as well. The authors’ intent is to introduce the problem of unexamined assumptions implicit in the extant process of selecting studies, and to demonstrate one approach for addressing the problem.

Methods

Since this study did not involve research on human subjects, no ethical approval was required. As part of a larger systematic review on paediatric mental illness in Ontario, Canada, (based on the methods employed in an earlier review [3]), the authors engaged in a qualitative dialogical process to identify descriptors of the Canadian population that were most relevant to the topic of the systematic review, and that could be used to assess the relevance of peer-reviewed published studies.

This dialogical process was a focused discussion informed by the authors’ expertise in both the theory of international health and development and the conduct of systematic reviews. Its purpose was to identify markers and indicators of nations’ development status which would be descriptive of a country’s economic wealth and overall population health. To inform this process, reference was made to the United Nations Development Programme’s international human development indicators [4], whose list of appropriate markers incorporates measures of health, education, income, inequality, poverty, gender, sustainability and demographics. The World Bank’s list of development indicators [5], which comprise a similar list with somewhat larger scope, were also examined. Refinement of the list of criteria was informed by comparison with indicators identified in the literature [6], which tended to stress health system responsiveness and inclusivity as appropriate benchmarks for performance.

Each putative indicator was discussed by the authors, championed by a specific author, then voted upon by the three authors with expertise in quantitative global health research (Deonandan, Schacter and Barrowman).

At the end of the dialogical process, the characteristics of any society, including the Canadian case, that were deemed most pertinent for establishing meaningful comparisons could be divided into two broad categories: sociodemographic factors (relating to the nature of the population) and systemic factors (relating to the nature of the government or society overall). All factors can be roughly mapped to the UNDP’s general list of human development indicators.

The sociodemographic factors landed upon were: population age distribution, ethnic distribution, dominant language, the existence of a marginalised Aboriginal community and high literacy rate (particularly among women). The systemic factors were: the existence of socialized medicine somewhat comparable to Canada’s, the existence of a mental health system separate from basic primary care, a measure of industrialization comparable to Canada’s and policy around formal public education.

A further dialogical process was employed to establish measurable indicators mapped to the aforementioned factors. Indicator cut-offs were based upon Canadian values, when appropriate within one standard deviation of the acknowledged national mean, according to both the CIA World Factbook [7] and the Census of Canada [8]. Table 1 summarizes the factors, indicators and cut-offs to be employed.

Figure 1

Table 1 – Factors and indicators comprising the checklist for relevance

As the primary systematic review on paediatric mental illness proceeded, and papers originating from many nations and jurisdictions were identified, these indicators were measured for each region in question. Data sources included the individual nations’ national censuses, the CIA World Factbook [7], the World Bank’s published list of nations’ industrial output [9] and from queries of the United Nations’ online statistical database [10] of development indicators.

The minimum number of positive outcomes for each indicator needed to ensure inclusion of papers from the traditionally acknowledged comparator nations of the USA, UK, Australia and New Zealand would be the formal threshold for deeming a tested nation to be sufficiently similar to Canada to warrant inclusion in the systematic review. This threshold was found to be a minimum of 4 of the 5 sociodemographic factors, and a minimum of 3 of the 4 systemic indicators. In addition, given the focus of the systematic review on child/youth mental health, it was felt that for a nation’s study to be included, it would be mandatory for that nation’s age distribution to be comparable to Canada’s. Note that these thresholds were established before the commencement of the systematic review.

Results

In the course of conducting a systematic review on the prevalence of youth mental health issues in the Canadian province of Ontario, papers from 68 jurisdictions, including Canada, were considered for inclusion and were therefore subject to the application of the developed checklist for relevance. Almost all the jurisdictions analysed were nation states, excepting Puerto Rico and Dubai. The indicator breakdowns for the top 10 and bottom 10 scoring nations are summarised in table 2.

Some additional jurisdictions were omitted from these analyses for reasons of problematic or imprecise political and jurisdictional definition, and due to a lack of readily available data. These were: Holland, Greenland, Bavaria, Palestine, Anatolia and Hong Kong.

According to the checklist, a total of 19 jurisdictions were deemed sufficiently similar to Canada to warrant including their studies in the systematic review on paediatric mental illness.

Figure 2

Table 2 – Results of the application of the relevance checklist on jurisdictions from which reviewed papers originated. Of 68 countries reviewed, the top 10 and worst 10 scores are presented. Legend: x – satisfies requirement, o – does not satisfy requirement, ~ - unclear or no data available.

Discussion

Not surprisingly, economically vibrant Western democracies like Norway, France and Sweden were deemed appropriately comparable to Canada for inclusion in the review. Unexpectedly, though, the less wealthy nation of Barbados was included, while Canada’s free trade partner and hemispheric neighbour, Mexico, was not; nor were many Western powers, like Japan, Spain, Belgium, Germany, Greece and Italy. The application of this set of criteria, then, had a profound impact on the conduct of the systematic review on paediatric mental illness, dramatically reducing the number of articles that otherwise likely would have been reflexively included.

It seems reasonable that population factors that define a nation’s character, relevant to most conceptualisations of population health, must include that population's age distribution, aspects of its wealth distribution, ethnic and linguistic mix and some measure of ease of access to services. The latter is in essence a measure of how similar that jurisdiction's medical system is to Canada's socialized system, which is important in the context of the grander systematic review because societies with desperately poor access to preventative services may manifest elevated rates of many types of illness, mental and otherwise, relative to Canada's rate. These factors, on the whole, affect whether prevalence estimates from a foreign nation can be used to model prevalence estimates in Canadian communities.

The dialogical process of identifying and developing the nine criteria for inclusion, while qualitative, was nonetheless was informed by existing international health and development literature. The general categories of indicators chosen is defensible, due to their use by the UNDP and World Bank. However, the import of each indicator, as well its cutoff, is tailored to the specific needs of this stidy. The criteria herein, then, are not necessarily the most pertinent for every case; and cut-offs representing these criteria are most definitely not the only possible ones, and not necessarily the most appropriate selections for many reviews.

The decision to set the checklist threshold at 4/5 sociodemographic factors and 3/4 systemic factors, based on the limit required to include the USA, UK, Australia and New Zealand, can be rationally argued to be both self-serving and self-defeating. Toggling the cut-offs for education, literacy and for the existence of socialized medicine could easily have resulted in disqualifying the USA as an appropriate comparator for Canada, immediately calling into question the appropriateness of a host of existing reviews that routinely lumped the two nations’ studies into an indistinguishable soup of transnational culture.

Additionally, there are additional criteria that could have been brought to bear. For example, a nation’s type of government was not considered. It could be argued, quite convincingly, that the characteristics that most pertinently bind the nations of Canada, USA, UK, Australia and New Zealand are their secular, Western liberal democratic traditions. Nor was the possibility of some internal national crisis considered, such as requiring that any included nation must not be in a state of war or of civil domestic strife.

Developing criteria for conducting systematic reviews pertinent to public health has been discussed by others [13]. It is well accepted that publication language biases must be addressed, or at least acknowledged, in any truly transparent systematic review. And yet the extent to which political and social bias, manifesting in the largely innocuous practice of excluding studies from dissimilar nations, affects reviews is unknown.

The point of this study was not to establish a benchmark for comparing the relevance of studies from competing nations. Rather, it was to suggest that for any future review, systematic or otherwise, researchers should consider developing a priori criteria, specific to their context, research question and study population, to more defensibly and rationally select international studies for inclusion. This is important for avoiding any biases, however unconscious, of a political, economic, linguistic or ethnic bent, that may influence the selection of relevant papers.

Conclusions

Within systematic reviews, the practice of including studies from nations deemed similar to the test nation, while excluding studies from those deemed dissimilar, requires a defensible and systematic process. One possible process is the development of a set of criteria for relevant similarity. And given the variability among review topics, such criteria should be developed uniquely for each distinct review. As demonstrated in these results, the application of such criteria can significantly alter the number and types of studies to be included or rejected from a systematic review, adding both rigour and external validity.

Acknowledgments

The authors wish to thank the Directors and staff of the Provincial Centre of Excellence for Child and Youth Mental Health at CHEO, in Ottawa, Canada, for their support.