CIS-2008: Study methodology

Click here to download PDF version

The CIS-2008 is the third national study examining the incidence of reported child abuse and neglect in Canada. This summary presents the methodology, discussing its strengths and limitations and impact on interpreting CIS-2008 estimates.
 

Sampling

The CIS-2008 sample was drawn in three stages (Figure 1): (1) a representative sample of child welfare sites from across Canada was selected, (2) cases were sampled over a three-month period within the selected sites, and (3) child investigations that met the study criteria were identified from the sampled cases.
 

Figure 1: Three stages of sampling in the CIS-2008

Forms of Maltreatment Included in the CIS-2008

The CIS-2008 definition of child maltreatment includes 32 forms of maltreatment grouped into five categories: (1) physical abuse, (2) sexual abuse, (3) neglect, (4) emotional maltreatment, and (5) exposure to intimate partner violence. This classification reflects a fairly broad definition of child maltreatment and includes forms that are not specifically indicated in some provincial and territorial child welfare statutes (e.g., educational neglect). Up to three forms of maltreatment for each investigation were tracked in the CIS-2008.
 

Investigated Maltreatment vs. Substantiated Maltreatment

The investigation phase is designed to determine whether a child has in fact been maltreated. The CIS uses a three-tiered classification system for investigated incidents of maltreatment including: (1) substantiated, (2) unsubstantiated, and (3) suspected. The latter provides an important clinical distinction in cases where there is not sufficient evidence to substantiate maltreatment, but where maltreatment cannot be ruled out.

Maltreatment-related investigations include specific allegations of maltreatment as well as allegations of risk of future maltreatment. Risk only investigations are those where a specific incident of maltreatment has not been alleged to have occurred, but where there are significant indications that maltreatment could occur in the future (e.g., caregiver with substance abuse issues). These estimates are included in Chapter 3 of the CIS-2008 report.

Cases of substantiated maltreatment include those cases where maltreatment of a child was confirmed (or verified) to have occurred after an investigation. Substantiated maltreatment includes cases of confirmed maltreatment. These estimates are included in Chapter 4 and Chapter 5 of the CIS-2008 report. The CIS-2008 was redesigned to track maltreatment investigations versus cases that were opened only to assess the risk of future maltreatment. Substantiated investigations in previous cycles of the CIS may include risk of future maltreatment; therefore, previous cycles cannot be compared to the CIS-2008.
 

Instrument

Because investigation procedures vary considerably across Canada, a key challenge in designing the CIS-2008 survey instruments was to identify elements common across jurisdictions that could provide data in a standardized manner. Given the time constraints faced by child welfare workers, the instrument also had to be kept as short and simple as possible.
The main data collection instruments used for the study were the CIS-2008 Maltreatment Assessment Form; and in Québec the Québec Maltreatment Assessment Form. The forms were completed by the primary investigating child welfare worker upon completion of each child welfare investigation. The data collection form consisted of an Intake Face Sheet, a Household Information Sheet, and two identical Child Information Sheets.

The CIS-2008 Maltreatment Assessment Form was based on the CIS-2003, CIS-1998, and the Ontario Incidence Study of Reported Child Abuse and Neglect 1993 (OIS-1993) data collection instruments in order to maximize comparisons across cycles of the study. A key challenge in updating instruments across cycles was to find the right balance between maintaining comparability while making improvements based on the findings from previous cycles. Changes to the data collection instrument were based on a case file validation study, validation focus groups, and a test re-test reliability study. The changes were made in close consultation with the Research Working Group and included: (1) the addition of a series of questions designed to distinguish maltreatment investigations from risk of future maltreatment cases,(2) a more detailed procedure to identify the relationship between each child and the caregivers in the home, (3) a more elaborate housing safety question, (4) a new proxy measure for poverty, (5) more specific exposure to intimate partner violence maltreatment codes, and (6) revised emotional maltreatment categories. The final version of the data collection instrument is in Appendix F of the CIS-2008 report.
 

Data collection and verification procedures

Site Researchers coordinated training and case selection at each CIS-2008 site and visited the sites on a regular basis (average of six times) to collect forms, respond to questions and monitor study progress. Forms were completed at the point when workers finished their written report of the investigation and were verified twice for completeness and consistency in responses. The overall participation rate was 96% and the completion rates for the forms were over 98% on most items.
 

Estimation procedures: Weighting, case duplication and sampling error

Regionalization and annualization weights were combined so that each case was multiplied by an annualization weight and by a regionalization weight. National incidence estimates were calculated by dividing the weighted estimates by the child population. Child population counts for sites and strata were obtained using Statistics Canada Census 2006 data.

Although the CIS-2008 estimates are based on a relatively large sample of 15,980 child maltreatment investigations, sampling error is primarily driven by the variability among the 112 sites. Sampling error estimates were calculated to reflect the fact that the survey population had been stratified and that primary sampling units (sites) had been selected randomly from each stratum.
 

Ethics procedures

The CIS‑2008 data collection and data-handling protocols and procedures were reviewed and approved by McGill University, the University of Toronto, and the University of Calgary Ethics Committees. Written permission for participating in the data collection process was obtained from the Provincial/Territorial Directors of Child Welfare as well as from each site administrator or director. Where a participating site had an ethics review process, the study was also evaluated by that site. The First Nations CIS-2008 Advisory Committee, which mediates Aboriginal ownership of and control over the project, had a mandate of ensuring that the CIS-2008 respects ownership, control, access and possession (OCAP) principles to the greatest degree possible. The First Nations CIS-2008 Advisory Committee is responsible for guiding and approving analyses of First Nations data, including potential comparisons with non-Aboriginal sites.
 

Study limitations

Although every effort was made to make the CIS‑2008 estimates precise and reliable, several limits inherent in the nature of the data collected must be taken into consideration: (1) comparisons between study cycles must be made with caution as a result of changes in the way risk of future maltreatment cases are identified; (2) the weights used to derive annual estimates include counts of children investigated more than once during the year, therefore the unit of analysis for the weighted estimates is a child investigation as opposed to an investigated child; (3) the CIS tracks information during the first 6 weeks of case activity; however there were slight provincial and territorial differences in this length of time; (4) data from Québec could not be included in some tables as a result of differences in data collection procedures; (5) in some instances, sample sizes were too small to derive publishable estimates; (6) the CIS tracks only reports investigated by child welfare sites and does not include reports that were screened out, cases that were investigated only by the police and cases that were never reported; and (7) the study is based on assessments provided by the investigating child welfare workers and could not be independently verified. For updates and more information on the CIS-2008, visit the Child Welfare Research Portal at www.cwrp.ca and PHAC’s Injury and Child Maltreatment Section: www.phac-aspc.gc.ca/cm-vee/public-eng.php.