Aimee Verdisco, Lead Education Specialist, Bolivia, and Jennelle Thompson, Senior Education Specialist, Ecuador, Inter-American Development Bank; Katelyn Hepworth, PRIDI Research Assistant 2011–2012
The Regional Project on Child Development Indicators, or PRIDI (Programa Regional de Indicadores de Desarrollo Infantil), is currently collecting data on child development outcomes in Costa Rica, Nicaragua, Paraguay and Peru. It is the first international study of its kind, seeking to generate high-quality and regionally comparable data which currently do not exist. Results are expected to be published in 2014. This article summarises the processes and results to date. 1
PRIDI was launched in 2009 with the aim of generating high-quality data on child development that are comparable across countries. These data will allow countries to identify gaps in child development among populations and areas, and to plan to mitigate them. PRIDI collects data on cognition, language, motor skills, and socio-emotional development, evaluating children between 2 years and 4 years 11 months in their homes. It is not a screening instrument.
A division is made within that age range, with one set of instruments for children aged 2 to 3.5 years and another for children aged 3.5 to 4 years 11 months. Children under 2 are excluded because it would require greater levels of training of enumerators than the budget could accommodate at scale; children aged 5 or older are excluded because the effects of formal schooling would be hard to control for. The final samples will include at least 2000 randomly selected children who are nationally representative including, for example, indigenous children.
The PRIDI Conceptual Framework defines child development as:
An integral process which includes not only verbal skills and knowledge and intellect, but also social skills and motor development, and strategies for learning, such as attention and inhibition of impulsive behaviours, as well as basic notions of health and nutrition. It is the process through which a child is prepared for new levels of responsibility and progressively gains new levels of autonomy.
(Inter-American Development Bank, 2011)
Participating countries were already measuring child development outcomes, using existing scales such as the Escala Abreviada de Nelson Ortiz and Escala de Desarrollo Integral del Niño. Many of these scales, however, were outdated. The PRIDI team therefore decided to develop a new test, building on existing national measures and international surveys (such as Woodcock-Muñoz, Denver and Multiple Indicator Cluster Surveys (MICS)).
This process requires attention to the smallest details. It took over a year to map the factors PRIDI seeks to measure against possible candidates for measuring them, and then to design and validate an initial set of development scales and surveys of the home environment. The design accounted for the fact that observation may be sufficient to evaluate a given developmental outcome in older children, while in younger children a response from the mother or caregiver may also be needed. In terms of instrument design, the first example (observation of a child) is included in a scale; the second (response from a mother/caregiver) is in a survey.
Two surveys of associated factors were also developed. One captures characteristics of the household that are known to influence child development. The other includes the mother’s (or caregiver’s) responses for items such as the child’s socio-emotional state, attendance in preschool, and disciplinary methods used in the home.
Phase I: Piloting on small groups
A three-phase plan was laid out. The objective of Phase I was to observe the functionality of the PRIDI instrument when administered to a small sample of children in two countries. The PRIDI team asked Peru and Paraguay to create a small, non-randomised sample that over-sampled indigenous children and included children in both age cohorts in urban, rural and indigenous areas. In Paraguay, the sample consisted of 50 children; in Peru, of 41 children. Each country could modify the instruments used to measure each competency, and include additional items (for example, Paraguay added items on musicality), but could not eliminate any activities from the evaluation scales.
The PRIDI team deliberately chose to keep each scale long, to identify the specific questions within the survey and evaluation scale that best captured the desired information. In Paraguay, the scale for children in the younger age group included 105 items, and for children in the older age group it included 99 items. In Peru, those figures were 33 and 36 respectively. PRIDI trained teams of consultants in both countries, including Guaraní speakers in Paraguay and Quechua speakers in Peru.
In general, results were encouraging. Among children in the younger age group, the items were functioning as expected based on the literature. Scores were evenly distributed and strongly correlated with age, thus creating a base and ceiling for the scale.
It was expected that a strong correlation would emerge between each child’s score and the average level of education of all adults living in the household. This was the case in Paraguay, but not in Peru – although this is probably explained by the small sample size and the over-sampling in indigenous areas, reducing variation in the levels of education within the sample.
In general, the scores of indigenous children were considerably lower than the scores of urban or rural children. This outcome was also consistent with expectations, but the magnitude of the disparity suggests the need to ensure that the instruments are appropriate to indigenous children and not underestimating their abilities. For example, field reports indicated that, in indigenous populations, some children did not know what ‘shoulders’ were, but could identify all other body parts.
The results for children in the older age group were different in important ways, primarily in the lack of variation in the distribution of scores. In Paraguay, the correlation between age and scores is essentially flat. That most children completed all tasks successfully suggests that there were too few difficult tasks. These results pointed to the need to revise the scale.
In general, the surveys for the child and household functioned largely as anticipated. In both countries, children in rural areas received higher scores than children in urban areas on a number of items (for example: the child worries if someone is crying; if the child is doing something and makes a mistake, he or she persists and keeps on trying without getting frustrated or angry; the child can play for 15 minutes or more without requiring the attention of an adult).
Feedback from enumerators indicated that too much time was required to complete the surveys. The application of the development scales ran anywhere from 20 to 55 minutes; the surveys required anywhere from 9 to 40 minutes. It often required two or three visits to the same home to apply the complete set of instruments, which created some logistical difficulties. More importantly, many children seemed to be fatigued by the end of the test or got bored during it, which influenced their performance on later tasks.
The pilot revealed important issues that needed to be dealt with in the training of enumerators. One issue was the difficulty of finding suitable spaces for the evaluation of the children, away from siblings or adults who tried to help them come up with the right response. Another was that the children were interested in the whole kit of materials, and insisted in knowing ‘what else’ the enumerator had with her, forcing enumerators to reveal objects before using them in the evaluation.
Phase I also served to inform the PRIDI team about the appropriateness of the materials used for applying the development scales. For example, some of the drawings used to describe sequences on both scales appeared to confuse children, and not all children could name objects that were assumed to be ‘daily use’ items across the board. In the latter case, it was impossible to know whether the lack of identification stemmed from a lack of vocabulary or from simply not knowing what the object was.
On the basis of results from Phase I, both instruments were shortened and the one for older children almost completely revised, with items modified to increase difficulty. For example, in Phase I, a child was asked to count to five; in the revised version the task is to count as high as 20. In Phase I, a child was asked to differentiate between above and below, and in front of and behind; in the revised version, he or she was also asked to differentiate between right and left.
The revised scales were applied to a further sample of 12 children in Cusco, Peru, and, as expected, displayed much more variation in results, while the average application time for scales and surveys combined was reduced. With these results, the PRIDI team felt confident in moving forward to the second phase: validation on samples of 200 children in each of four countries.
Phase II: Validation
A meeting of national coordinators and the PRIDI team in November 2011 approved the scales and surveys for application in Phase II. The International Association for Educational Achievement, which provided technical assistance to PRIDI, returned to Lima in January 2012 to give a training seminar to country data managers on the use of the software it had developed for the entry and management of data. The PRIDI team provided training on the application of the internationally normed Peabody Picture Vocabulary Test (Test de Vocabulario en Imagenes Peabody, TVIP), which would also be applied in Phase II to validate the PRIDI instruments by checking that the results bore an association. The team distributed a training manual and DVD and a detailed curriculum for the training of enumerators.
Each country was asked to adapt the instruments for idiomatic expressions or language (for example, using media instead of calcetín (sock) in the case of Nicaragua) without changing the substance or intent of the item. The cultural adaptation of items is important to ensure that all children have an equal opportunity to show what they know and are capable of doing, as is the consistent application of items. How an item is applied has a direct relationship with what a child does in response to a given prompt.
In Nicaragua, Paraguay and Peru the local team met with representatives of indigenous people to collect qualitative information on the daily routines of children aged 2 to 5 in indigenous areas. Enumerators were trained who spoke Guaraní (Paraguay), Miskito (Nicaragua) and Quechua (Peru), and instruments were translated. Samples of 200 children were chosen, stratified by urban/rural, level of mother’s education and – in the cases of Nicaragua and Peru – indigenous and non-indigenous.
Results from Phase II were presented in San José, Costa Rica, in January of 2013 to the country coordinators and representatives from each of the national firms responsible for the field work. This included more and more sophisticated analyses of the data than in Phase I. For each item on each scale, Rasch model and item response theory analyses were undertaken by item by country, and by item across countries, to examine variability and validation. Items displaying low variation were dropped, as were those that appeared too hard or easy. From these ‘reduced scales’ (21 items for the younger group and 22 for the older group), factor analyses were undertaken to ensure that items theoretically mapped onto each domain actually did hang together in an empirical analysis.
Several additional analyses were undertaken to ensure the validity of the instruments, including plotting the distributions of responses for each domain (area), and regressing results of each domain by age, level of education, access to services and other factors that the literature identifies as important correlates of ECD; correlations with the TVIP and height-for-age data were strong. These analyses provided an empirical basis upon which to streamline the instruments and to ensure their validity.
Alpha reliability scores, which measure internal consistency, for each domain were acceptably high, ranging from .55 to .78. For example, reducing the socio-emotional scale from 27 to 16 items increased the reliability coefficient to .74. Cognition among young children is Normally distributed and correlates well with the age of the child and the level of maternal education. The same occurs for motor skills and language. Similar trends emerge in the older group, albeit to a slightly lesser magnitude.
Phase III: Data collection
Results from Phase II again showed that the time needed to administer all components – the scales, surveys, TVIP, and height-for-age – continued to be an issue. Average application time again proved to be too long, ranging from up to 90 minutes for younger children to up to 120 minutes for the older age group. More comprehensive training of enumerators could increase their efficiency, but the main technique to decrease the time needed is to shorten the components of the instrument.
As well as the dropping of items described above, the surveys for the child and family were both considerably shortened and consolidated into a single survey, with changes in formatting introduced to facilitate more efficient scoring of items. This streamlining significantly reduced the time required. Field testing early in 2013 of the reduced and final versions indicates that application time ranges between about 30 and 40 minutes.
Phase III is now underway, with data from 2000 children in each country being collected and analysed over the course of 2013. The scales have been formally named the Engle Scales of Child Development (las Escalas Engle de Desarrollo Infantil) in memory and appreciation of Patrice Engle who passed away in September of 2012. Final results will be presented in the first quarter of 2014.
Equipo Gerencial del PRIDI. (2011). Programa Regional de Indicadores de Desarrollo Infantil (PRIDI): Marco conceptual. Washington, DC: Inter-American Development Bank. Available at: http://iadb.org/es/publicaciones/detalle,7101.html?id=31906&dcLanguage=es&dcType=All (accessed April 2013).
1 The ministries with lead responsibility for young children in Costa Rica, Nicaragua, Paraguay and Peru are working on PRIDI with the Inter-American Development Bank’s education division, a management team of experts led by Patrice Engle (Cal Poly University, USA; UNICEF) and Santiago Cueto (GRADE, Peru), and with technical assistance from the International Association for Educational Achievement on issues such as sampling and data collection.