Introduction

Until the last decade, poor children in Peru had limited or no access to health services. In 1997, the government established a system to deliver free health care to children attending government-run schools (children from private schools were excluded), which covered children from the age of three years until their 18th birthday. This health care system for school children from poor families lasted until January 2001, when a new, more integrated health system was established to cover children of all ages. Our research study was performed in a paediatric intensive care unit (PICU) during the period when health care services were free for poor children from their initial entry to school through secondary school. This study aimed to evaluate if the use of costly critical care to poor children was effective, as measured by outcomes, when compared to patients who were as ill in other developed countries.

Intensive care for children has contributed to the marked reduction in the rate of mortality and morbidity for certain diagnostic categories in Peru and in developed countries. Increased demand for health care, new technologies, and awareness of limited resources have become prominent issues, and intensive care paediatricians are interested in new ways to evaluate health care programs^1,2,3. This has fostered the need for quantitative methods to evaluate medical activities in the paediatric population. Severity of illness scoring systems in intensive care have been developed and validated for all ages in the United States^4,5,6,7, Europe⁸ and other settings^9,10,11, and these systems have been improved and simplified^12,13. Very few studies using these severity of illness scoring systems have been carried out in Latin American countries^{14,15,16,17,18}.

When PICU was not available at our institution mortality was over 50 %. After establishing the necessity of intensive care and demonstrating that mortality could be lowered with skills, procedures and technologies proven in developed countries, paediatric mortality progressively decreased in our institution and in our country. However, we also saw that inadequate care could result in varying degrees of morbidity without necessarily resulting in death. In infants and children, poor care can have long-term effects on the dependent or independent-functioning life expectancy, and these effects are particularly important to physicians and society^19,20,21. Studies assessing the performance, quality²², efficiency of paediatric intensive care²³, effectiveness²⁴, and patient outcome are relevant in our country because of the high social and economic costs of survival with disability^{25,26,27,28,29,30}. We could not find studies measuring disability in countries comparable to our reality and since we considered important to have an approximate idea of our performance, we evaluated disability using the Pediatric Overall Performance Category (POPC) ¹⁹.

Our unit is the most important PICU in Peru. The lack of comparative analysis studies in Latin America and the need to maximize the efficiency of the scarce resources prompted us to evaluate our critical care and to compare our performance with units in other countries. In structuring our study, we considered three important aims: (1) to evaluate survival of critically ill patients and the factors related to their survival, (2) to establish the validity of the Paediatric Risk of Mortality (PRISM I) score in a Peruvian setting, and (3) to assess the effectiveness and efficiency of care in our PICU as compared to patients who were as ill in other developed countries.

Materials And Methods

Setting. This study was performed at the National Institute of Child Health in Lima, Peru, an urban, university-based tertiary care paediatric centre with a diverse patient population. During the study, the institute functioned as a 515-bed centre that admitted infants and children from the entire country.

Intensive Care Unit Characteristics. Our PICU is a 16-bed unit distributed in three wards. It admits all patients younger than 18 years of age who require intensive care, including neonates referred from other centres and cardiovascular surgical patients (after our study, the latter patients were treated in a separate unit).

Study Period. Data were collected from 1 August 1997 to 31 January 2001, when government health insurance covered medical care for poor, school-age children in Peru.

Patient Population. We included all patients three years of age or older but less than 18 years of age (n = 819) admitted consecutively to the PICU during the study period. Patients younger than three years were excluded because the study included only patients covered by the government-financed free health care services for pre-school and school age children, which at that time did not cover patients younger than three years of age. Patients already admitted to the PICU at the start of the study period or with incomplete data (n = 6) were not enrolled. Patients admitted more than once to the PICU during the same hospitalization and those discharged from the hospital and readmitted to the unit were considered as separate admissions. Regardless of the primary service managing the patient, patients were considered surgical if they were post-operative, trauma patients were considered surgical if they were post-operative, and patients were considered not surgical if their condition did not require surgery. All other patients were considered medical.

Data Collection. On admission, demographic data (age, sex, place of origin, schooling), primary clinical specialty, main diagnostic category, and operative status were recorded. Each admission diagnosis was coded according to the International Classification–10–Clinical Diagnosis (ICD-10). Even though we would have preferred to use PRISM III, during the time of the study the calculation for this score was not within our possibilities

for which we used PRISM I. The worst scores for the Paediatric Risk of Mortality (PRISM I)⁴ and Simplified Therapeutic Intervention Scoring System (STISS) scores^30,31 were registered on admission or at 48–72 hours after admission. Other data collected included vital status, length of stay at the unit, and need of mechanical ventilation. To safeguard the quality of the data, the data were collected by the main researcher only. The study was approved by the Direction of Investigation and Technology Development of the Institute.

Survival Curves and Survival Tables. Survival curves and tables were constructed from the time of admission to the time of discharge from the PICU. We set the final follow-up time at 30 days of PICU discharge to other services to avoid the influence of a small number of patients with a prolonged stay at the unit. We have chronic patients of more than 6 months to several years. Deaths after PICU discharge were not included. The survival curves were constructed and analysed in relation to the type of disease diagnosed at admission, PRISM scores, and the need to use mechanical ventilation.

Paediatric Risk of Mortality Score. The PRISM score is an established, institutionally independent measure of the severity of illness that has a high predictive accuracy for mortality that has been established in multicentre studies of PICUs in the United States⁴ and Europe⁸. The PRISM score is based on 14 routinely measured physiological parameters. For each parameter, the most abnormal values observed on the day of admission at the PICU were used for scoring. We excluded unreliable or undetermined items that might influence the risk of mortality. The risk of dying at the unit within the next 24 hours, the paediatric death rate, was determined using the Dynamic Objective Risk Assessment scored derived from the PRISM score on admission.

Simplified Therapeutic Intervention Scoring System. The STISS³⁰, also called the TISS-28, is the simplified system derived from the TISS-76³¹. The TISS-28 consists of 84 interventions provided in the intensive care and intermediate care units. The interventions in the simplified scoring applied best to our PICU; this scoring system assigns 1 to 4 points depending on the characteristics of the interventions used on the patients. Based on the scoring, the time of nurse’s care in minutes was calculated. One point on the TISS-28 scale equals 10.6 minutes of each eight-hour shift.

Medical Effectiveness. To estimate effectiveness, we followed the same design used by Gemke et al.⁸, who compared the severity of illness-corrected mortality rates in their unit with the accepted U.S. standard based on the PRISM score⁴. The overall medical effectiveness was determined using the standardized mortality ratio (SMR) by comparing the cumulative predicted mortality risk with the total actual mortality rate. According to Gemke et al., the SMR can be determined reliably in samples of > 200 patients⁴ and is an established parameter to analyse mortality rates in diverse studies^4,9,15. An SMR > 1.0 means that the actual mortality rate is greater than the predicted rate, and an SMR < 1.0 means the opposite. We used the five conventional mortality risk intervals ( 1%, 1–4.9%, 5–14.9%, 15–29.9%, and > 30%) in a stratified comparison of predicted and observed mortality.

Medical Efficiency. Medical efficiency was evaluated according to the two criteria used by Pollack et al. ⁴. Admission was regarded as efficient if: (a) the mortality risk based on the PRISM score on admission was > 1% ,or (b) the patient received a PICU-dependent therapy during the first 24 to 72 hours of admission. All PICU-dependent therapies were derived from the TISS-28.

Statistical Analysis. We used simple descriptive statistics to describe groups (mean, median, SD, and quartiles). Data are expressed as mean ± SD unless otherwise indicated. Analysis of variance or t tests were used to compare continuous variables (age, length of stay, PRISM and TISS-28 scores, and mortality risk). Medians of continuous variables were compared using the Mann–Whitney test. To estimate the accumulated survival of the patients at the unit, survival tables and survival curves were constructed using the Kaplan–Meier method. Admission to the PICU was set at t = 0, and the follow-up was truncated at 30 days (t = 1). In the Cox regression analysis, the association between a variable and survival is expressed as a risk rate. A receiver operating characteristic curve was constructed. We used the z-statistic to assess the overall agreement of expected and observed mortality and the chi-square goodness-of-fit test to evaluate the agreement in the distribution number of expected and observed deaths over the five risk strata.

Results

Patient Demographic Characteristics. During the three and a half year study period, 825 patients three to 17.9 years of age were admitted to the intensive care unit. The mean age was 9.1 years (109.7 ± 1.7 months), the mean length of stay was 9.2 days, and the mortality in the unit was 16.2%. More than 50% of admitted patients were boys from primary schools in urban metropolitan areas who were admitted through the emergency service. The patients’ characteristics are shown in Table 1.

Figure 1

Table 1. Patient characteristics

Patient Clinical Characteristics. Patients with medical morbidities comprised 43.5% (n = 355) of the admissions to the unit, patients with surgical morbidities comprised 40.3% (n = 329), and patients with trauma comprised 16.5% (n = 135). The distribution of patients into primary diagnosis, diseases at admission and severity of illness is shown in Table 2. The system most affected on admission was the neurological system (44.7%), followed by the cardiovascular (16.3%), respiratory (15.4%), and multi-organ (10.2%) systems. Trauma (27.7%), infections (23.3%) and congenital pathologies (13.1%) were the most frequent causes of primary dysfunction. The mean values of the scores for the patients on admission were: PRISM, 10.7  6.9; paediatric death rate, 8.15  11.9; TISS-28, 26.5  9.8; and time of nurse’s care, 279.9 ± 104.5 minutes.

Figure 2

Table 2. Patient admission characteristics: primary diagnosis, diseases and severity of illness

Survival Curves. _{Survival was related significantly to the type of diagnosis on admission (P = 0.008), sex (P = 0.006), primary service of reference P = 0.002), and the use of mechanical ventilation (P = 0.001). The Kaplan–Meier survival curves for patients with medical, surgical, and trauma pathologies are shown in Figure 1. The survival curves had the same tendency for all patients with different types of diagnosis during the first 10 days of stay in the unit. However, the probability of survival was greater in post-operative patients or trauma patients who did not require surgery, and post-operative trauma patients showed less probability of survival before the first week of stay in the unit. After the 10th day of stay, the curve showed that the patients with medical pathologies and some post-operative pathologies had a lesser tendency to survive.}

Figure 3

Figure 1. Survival curve in relation to the types of pathologies ( < 0.001): medical (green), Surgical (pink), trauma (yellow), and trauma with surgery (grey). The crosses in the curves (+) indicate deaths.

The survival curves for the patients according to the PRISM scores on admission are shown in Figure 2. To construct the curves, PRISM scores were divided into quartiles (0–6, 7–10, 11–15, and > 15 points), with larger scores showing greater physiological instability. We observed that patients with low PRISM scores (< 10 points), showing less physiological instability and had a greater probability of survival, and those with scores between 11 and 15 had a better probability of survival than patients with PRISM scores > 15.

Figure 4

Figure 2. Survival curve in relation to Pediatric Risk of Mortality (PRISM) score ( < 0.001). PRISM scores are indicated by colours: 0–6 (green), 7–10 (pink), 11–15 (yellow) and  15 (grey).

In our population, the PRISM score performed well in predicting overall PICU mortality. The calculated SMR was 1.75 (95% confidence interval, 1.72–1.78), where the SMR by definition is the observed mortality rate divided by the expected mortality rate^4,8. The stratified performance of the PRISM score in predicting the PICU mortality rate is illustrated in Figure 3; this was formally evaluated using the goodness-of-fit test, which showed that the observed and expected mortality rates were not in agreement [² (5) ^{= 11.07; P = 0.05]. The observed mortality rates were higher than those predicted by the PRISM score in all five intervals. We constructed a receiver operating characteristic curve (Figure 4) to analyse the predictive power of the PRISM scores. The curve showed that the shape and the area under the curve (Az = 0.82 ± 0.02) were similar to the original validation groups4.}

Figure 5

Figure 3. Observed (pink bars) and expected (blue bars) mortality rates based on the Pediatric Risk of Mortality score at admission. The number of patients in each risk interval are shown above each pair of bars. ? (5) = 11.07 ( = 0.05) in the goodness-of-fit test.

Figure 6

Figure 4. Receiver operating characteristic curve relating the true-positive rate (sensitivity; correct prediction of death) to the false-positive rate (specificity; correct prediction of survival). The area under the (Az) curve is a measure of the overall discriminatory performance of the prognostic model. Az = 0.82 ± 0.02 (SE).

Admission Efficiency. Retrospective analysis of admission efficiency demonstrated that admission to the unit was justified in 84.0% (n = 688) of patients. The remaining 16% (n = 131) did not have a mortality risk > 1%, did not receive a PICU-dependent therapy (low risk monitoring patients), or were potential early discharge patients (patients who stayed in the PICU even though the PICU-dependent therapy had been stopped or had become stable without any risk of the patient’s dying).

Medical Effectiveness. Medical effectiveness represented as the SMR was 1.75 (95% CONFIDENCE INTERVAL, 1.72–1.78). The SMR for our population was higher than for the U.S. and European validation groups.

Figure 7

Table 3. Comparative analysis of PICU patients in our study with patients from American and Dutch populations

Discussion

Children under three years of age were excluded from the study mainly to minimize any bias that might arise from the difference between government coverage of critical care and the cost assumed by the families of patients not covered by the insurance. We did this because the lack of services for children younger than three years might have delayed diagnostic procedures and treatment, or might have prevented parents from buying the medicines to treat their children, despite the help of social services and private institutions. Even though our study did not cover the full child population (i.e., none younger than three years), we believe our data are relevant to the study of health problems in school-age children because they experience a unique array of diseases and conditions (e.g., trauma), which could be useful to orient health care policies and allocation of resources.

Our country and other countries in our region are becoming increasingly concerned about the costs and questions of effectiveness and quality of care using high-technology medicine in PICUs. Until the development of the PRISM score for mortality prediction and the subsequent validation by other studies, comparing performance of different PICU settings was nearly impossible because of the wide variability in the types and conditions of the patients and the multiple factors influencing their outcomes. The PRISM score is a major step forward in standardizing these conditions.

Because of the shortage of available beds in PICUs³² and the cost of critical care borne by the families, the selection of patients has always been a factor determining patient care. Factors such as availability of beds and cost frequently delay the hospitalization and transfer of patients³³ and delay the use of diagnostic procedures and therapeutic measures. Together, these factors may increase the length of stay, cause the patient’s condition to deteriorate, increase the risk of adverse events, and increase hospital costs. Government insurance for school-age children provided us with the opportunity to determine whether performance at the unit would improve once critical care was determined by factors other than family finances. We are addressing this new line of research in another work, which we hope to complete soon.

Survival curves have been used widely in cohort studies and for very specific diseases, but they have not been used in critical care studies^34,35. We used survival tables and curves to evaluate the probability of patient survival during their stay in the PICU and the prognostic value of these data according to the diagnosis and physiological stability at admission. The survival curves for the PRISM score and for groups of diseases performed well in the prediction of overall PICU mortality, and, consequently, in overall survival. We found them to be useful tools to show tendencies and prognostic values in relation to mortality and when assessed with other factors related to survival. The probability of survival was better for non-medical conditions (trauma and surgery) and after the first week of admission at our unit. The probability of survival decreased for children with high PRISM scores, which indicate greater physiological instability.

We found heterogeneity in the cumulative observed and expected mortality rates, and in separate risk strata. We observed that the cumulative observed mortality rate was higher than the expected rate and that this difference was reflected in the SMR. These data indicate that the effectiveness of care in our population was equivalent to, but not better than, the reference population.

Patients most severely ill (PRISM>30) were patients with septic shock and those after cardiac interventions mainly due to congenital heart diseases. We cannot really explain why our outcomes were better in the sickest groups (PRISM >30), but we think that probably we gave more attention and we directed more efforts and resources trying to do the utmost for them than we did with the other patients (PRISM <30).

We consider that the observed mortality in our unit was considerably greater in the moderately and severely ill due to several reasons: (1) there was an asymmetry of knowledge and abilities between those intensivists dedicated to teaching and those who were not; (2) nurses were periodically rotated to other areas of the Institute according to the needs of the different services; (3) increased complications occurring during the night shifts or during the change of shifts; (4) physicians and nurses were on duty 24 hours per day on 6 and 12 hours shifts; and (5) complications due to nosocomial infections.

(6) during the study period, the medical staff at the unit were paediatricians without formal training in critical care, but with varying degrees of experience in paediatric intensive care, the same as the nurses. After this study was completed and as the need for pediatric intensivists increased, several universities started training programs for intensivists lasting two to three years. At the same time, several training programs in critical care were initiated for nurses.

The efficiency of resource allocation has been evaluated using two objective criteria in previous studies^4,8. The first criterion of an arbitrary mortality risk level of 1% seemed to be an appropriate and safe lower margin; our observed mortality for this stratum was 3.57%, which is higher than that observed in these previous reports. The other criterion was the administration of at least one ICU-dependent therapy, based on the therapies found within the TISS-28^30,31 and their appropriate application on the patients. Even though the turnover in our unit was good considering the perceived shortage of beds, the length of stay was somewhat longer than in other PICUs, causing an inappropriate prolongation of ICU-dependent therapies, which may have caused an artificially increased efficiency rate.

As critical care physicians, we have a duty to care for and to save children with life-threatening illnesses, as well as trying to maintain their functional status. Efficiency and effectiveness should be assessed by the final product^34,35. Children have a long life expectancy, and it is important to try to increase their chances of living independently rather than with disability. In our paediatric population, more than 60% of the patients showed moderate to severe disability as measured by the Paediatric Overall Performance Category (POCP)¹⁹ , which could have been explained by the type of pathology predominant in this age group, such as trauma; infectious, congenital, and neurological diseases; adverse events before and during their stay in the unit; and previous disability before entering the unit. However, because of factors beyond our control, we could not adequately assess baseline functional status at admission or follow-up status at discharge, and we have only estimated their functional outcome at egress. Thus, these results reflect only partially the quality of care of the unit; full assessment requires further study and should be taken with caution because we do not have follow-up data after the patients left the PICU.

The risk of mortality is high in small children and infants. Having excluded from the study children younger than three years for reasons mentioned above, we acknowledge that our results may be difficult to compare with those from other units. Regardless, we still believe it important to have separate data for school-age children and adolescents, who experience a different array of diseases and risks.

We tried to compare our PICU with those in American and European reference populations (see Table 3). Because we included only school-age patients, the average age was greater in our population than in other studies. Our length of stay was longer and the PRISM score was somewhat higher, even though the risk of mortality was about the same as in other studies. Our mortality and SMR almost doubled those of Gemke et al. ⁸ and Seferian et al.³. Our SMR was higher than in the other studies and would have been even higher if we had included younger children and infants. The efficiency at admission would not have been influenced much by age and seems to be similar to that reported by Gemke.

In summary, in addition to PICU mortality, other factors such as survival, long-term mortality, morbidity, and functional outcome seem to be important parameters for measuring the effectiveness and quality of care in PICUs. Our study demonstrates the validity of PRISM score-based mortality prediction in a Peruvian PICU for children between the ages of three and 18 years. The efficiency of our unit met the standards established in American and European studies, but our overall performance was lower than that in the American and European reference populations. Our study confirms that these methods are transferable to units in other countries. One important contribution of our study is that we have presented for the first time survival curves in relation to two main factors that affect survival in a PICU—the type of disease on admission and PRISM scores, which proved to be useful tools in assessing critically ill children.