_-The-Breakthrough-Antipsychotic-That-Could-Change-Everything.jpg?1729528747 "KarXT (Cobenfy)_ The Breakthrough Antipsychotic That Could Change Everything.jpg")
Effect Size Matters: The Seismic Shift Toward Naturalistic and Developmental Interventions in Autism
Micheal P Sandbank, PhD
Assistant professor of special education at the University of Texas, Austin. Dr. Sandbank has disclosed no relevant financial or other interests in any commercial companies pertaining to this educational activity.
CCPR: Welcome, Dr. Sandbank. Tell us about your work.
Dr. Sandbank: My current research is in social communication and language interventions for young children with disabilities, including those on the autism spectrum. I am lead researcher on Project AIM, a vast systematic review and meta-analysis of all group design studies of interventions for young children with autism (www.tinyurl.com/dda6pzpv).
CCPR: What’s the state of research in autism?
Dr. Sandbank: Autism treatment is a huge industry, and yet randomized controlled trials (RCTs) have been rare in the autism field, making it hard to show that treatment is effective. Despite a tidal wave of low-quality studies, RCTs were almost nonexistent in the early 2000s. In 2011, a systematic review in Pediatrics found only two high-quality RCTs. Then in 2017, another systematic review found 48. That’s a major change in just six years.
CCPR: What is the purpose of a meta-analysis?
Dr. Sandbank: Studies that show “significant change” do not tell you if that change is meaningful for patients. Meta-analysis allows you to compute an effect size across multiple studies to get a sense of the magnitude of the clinical impact. With so many kinds of autism treatment, people are looking at many different outcomes. We used a recently developed technique that lets us compare all the effect sizes for the different outcome measures. In our 2019 Project AIM study, we sorted interventions and outcomes into broad domains and calculated effect sizes across outcomes (Sandbank M et al, Psychol Bull 2019;146(1):1–29).
CCPR: What population did you look at in the study, and what were the categories of interventions?
Dr. Sandbank: We looked at studies of children 8 years and under because that’s the defining age range of early childhood. Most of the studies focused on three categories: traditional applied behavioral analysis (ABA), developmental relationship-based interventions (DRBI or “developmental”), and naturalistic developmental behavioral intervention (NDBI).
CCPR: So what did you find?
Dr. Sandbank: For the developmental interventions and NDBI, we found moderate effect sizes for improving social communication, which is the central problem area in autism. But by far the most interesting finding of our study was that we couldn’t find compelling evidence for traditional applied behavioral analysis (ABA) interventions. There weren’t enough RCTs to compute summary effects on any outcome for traditional ABA interventions. This is concerning because traditional ABA interventions are far and away the most commonly recommended approach in the field.
CCPR: This could change how we look at autism treatment.
Dr. Sandbank: Yes. And even the few positive findings we computed were based on the rare very high-quality studies. The majority of studies were impacted by potential bias, largely when participants were aware of what group they were assigned to. When we excluded those studies, we saw no significant effects on any intervention approach for any outcome for any treatment.
CCPR: Can you talk more about the outcomes you were looking at in these studies?
Dr. Sandbank: Sure. When a child learns a skill and then can use it in several contexts, we call that generalization. When a child learns a skill and then builds on that to develop other new skills, we call that distal development. We wanted to see whether interventions were causing meaningful change in generalized and distal outcomes, or if the child had simply acquired a very discrete skill that they could only use in the exact circumstances where they were taught. If you teach the child 10 words, you may not see growth in general communicative ability. On the other hand, teaching social communication may cascade into language development not just within the intervention, but across multiple contexts.
CCPR: How does this relate to the different categories of treatment?
Dr. Sandbank: The traditional ABA behavioral approaches teach specific skills. The developmental approaches work on foundational skills, such as emotional regulation and reciprocal communication, that support both generalization across domains and cascading distal development. NDBI marries the two theories. (Editor’s note: For a succinct tutorial on the differences between these three approaches, see Dr. Diane Cullinane’s article in this issue.)
CCPR: Why have behavioral approaches dominated treatment?
Dr. Sandbank: It is far easier to do research in traditional ABA approaches that teach specific skills such as new phrases. These are narrow proximal outcomes. It has been harder to conduct research on developmental or NDBI approaches. You need to measure change across a whole domain on a validated, standardized assessment administered by a naïve assessor. That would be a distal effect. Proximal effects appear large, even though they may not generalize nor result in distal growth. Distal and generalized effects are more clinically meaningful but likely to appear smaller.
CCPR: What are the implications for treatment?
Dr. Sandbank: Since the traditional ABA approaches have seemed to be more effective, they have dominated treatment and also insurance reimbursement. And when therapists are trained, they tend to be trained in behavioral approaches. Students train in traditional ABA approaches to become Board Certified Behavioral Analysts (BCBAs). These are BA or MA certifications that are separate from usual mental health specialties and focused on autism treatment without training in more broad mental health conditions or treatments. Therapists learn to teach autistic children specific skills. For instance, they learn to teach children to point to specific objects, a common activity in these treatments. It turns out from our research that this is unlikely to generalize to new kinds of pointing nor other communication. Developmental and NDBI approaches have certifications provided by the training institutions but no central board like the BCBA. Still, NDBI and developmental interventions may have better outcomes. For the same example, the child may learn to use pointing better because they learn it in the context of meaningful interactions. The newer research supports this.
CCPR: The recent American Academy of Pediatrics (AAP) guidelines seem to favor the traditional behavioral approaches.
Dr. Sandbank: Current practice is typically the use of an Autism Diagnostic Observation Schedule (ADOS) as a chief diagnostic tool, which is taken to a psychiatrist, psychologist, or pediatrician who often doesn’t have expertise in autism but recommends 40 hours per week of intensive behavioral intervention. We just wrote an opinion piece in JAMA Pediatrics that urges clinicians to learn about the entire range of treatments and collaborate with families to figure out what might work best for them rather than routinely recommending just the one approach (Sandbank M et al, JAMA Pediatr 2020. Epub ahead of print). Clinicians are gatekeepers who need to know about the different intervention approaches, particularly those supported by RCTs.
CCPR: Can you comment on the recommended intensity of treatment?
Dr. Sandbank: We did an analysis of intensity to see if more hours of intervention was associated with greater effect sizes. We did this for behavioral, developmental, and NDBI studies and found no association between the total hours of intervention received in the study and the size of the effect. That doesn’t mean the amount of intervention doesn’t matter, but it means that we lack the kind of evidence needed to support recommendations for the very high-intensity treatments. It may be problematic to recommend that a toddler receive 40 hours of intervention. This is quite taxing for children and families and it may not be necessary.
CCPR: Families often come to us asking for 40 hours of ABA.
Dr. Sandbank: They hope that this will result in the best outcomes, and yet mere access to these hours is not what the research suggests. About a third of kids with autism who are preverbal at 2 are not likely to develop phrase speech by the end of elementary school, and this is related to effective intervention—not access to traditional ABA (Anderson DK et al, J Consult Clin Psychol 2007;75(4):594–604). Sadly, when kids haven’t made progress, it is common for parents to feel guilty and ask, “Why isn’t my child speaking? Was it because I didn’t start intervention early enough? Was it because I didn’t do the 40 hours?”
CCPR: Does the research support any particular time commitment for these interventions?
Dr. Sandbank: For many treatments, the intensity and duration of intervention really depends on the intervention. There’s a parent-mediated communication-focused treatment called PACT, a developmental approach, that is only 18 one-hour sessions—no more, no less. But we have a really high-quality study that shows it improves the core challenges related to autism and development over time. There are few RCTs comparing intervention intensity. My former advisor, Paul Yoder, just completed a study comparing the Early Start Denver Model (ESDM, a form of NDBI) and a traditional ABA intervention, where children were randomly assigned to receive either 15 or 25 hours per week, and they showed no differences in outcomes by intervention or intensity across the whole group (Rogers SJ et al, J Am Acad Child Adolesc Psychiatry 2020;S0890-8567(20):31350–31352). It is the first study to ask this question in this way. We need more studies like that. A more recent paper found that more hours of intervention led to better outcomes for higher-functioning children. This is interesting since most of the time, it is the children with more severe challenges who are offered higher numbers of hours.
CCPR: How do we decide what kind of approach to recommend? Can we predict which child responds to which intervention?
Dr. Sandbank: It’s not one size fits all. We need to look within the groups for mediators. Maybe kids with better social attention do better with developmental interventions or NDBI. And maybe kids who don’t have that social awareness need more formal, explicit instruction in how to do these things. A lot of families feel that their ABA therapy has been very helpful, in which case we should let that support continue. Does it depend on study-entry autism symptomology, social communication skills, or age? Those are the kinds of studies that we need.
CCPR: What should we say to families about how to choose among the various options for autism treatment?
Dr. Sandbank: We certainly have more high-quality evidence supporting developmental and naturalistic approaches as our research is improving. Autism is a spectrum, and different kids have different abilities and strengths. Some families prefer certain interventions, and some kids improve with certain interventions. I recommend that clinicians describe the different approaches, which means you have to be familiar with them and what’s available in your community; you can’t recommend an intervention that a family cannot access. Identify the family’s priorities. Do they want direct structured intervention from a clinician, or do they think that won’t fit into their life? You have to have buy-in from the family, and it has to be something that the child experiences positively.
CCPR: Thank you for your time, Dr. Sandbank.
Child PsychiatryDr. Sandbank: My current research is in social communication and language interventions for young children with disabilities, including those on the autism spectrum. I am lead researcher on Project AIM, a vast systematic review and meta-analysis of all group design studies of interventions for young children with autism (www.tinyurl.com/dda6pzpv).
CCPR: What’s the state of research in autism?
Dr. Sandbank: Autism treatment is a huge industry, and yet randomized controlled trials (RCTs) have been rare in the autism field, making it hard to show that treatment is effective. Despite a tidal wave of low-quality studies, RCTs were almost nonexistent in the early 2000s. In 2011, a systematic review in Pediatrics found only two high-quality RCTs. Then in 2017, another systematic review found 48. That’s a major change in just six years.
CCPR: What is the purpose of a meta-analysis?
Dr. Sandbank: Studies that show “significant change” do not tell you if that change is meaningful for patients. Meta-analysis allows you to compute an effect size across multiple studies to get a sense of the magnitude of the clinical impact. With so many kinds of autism treatment, people are looking at many different outcomes. We used a recently developed technique that lets us compare all the effect sizes for the different outcome measures. In our 2019 Project AIM study, we sorted interventions and outcomes into broad domains and calculated effect sizes across outcomes (Sandbank M et al, Psychol Bull 2019;146(1):1–29).
CCPR: What population did you look at in the study, and what were the categories of interventions?
Dr. Sandbank: We looked at studies of children 8 years and under because that’s the defining age range of early childhood. Most of the studies focused on three categories: traditional applied behavioral analysis (ABA), developmental relationship-based interventions (DRBI or “developmental”), and naturalistic developmental behavioral intervention (NDBI).
CCPR: So what did you find?
Dr. Sandbank: For the developmental interventions and NDBI, we found moderate effect sizes for improving social communication, which is the central problem area in autism. But by far the most interesting finding of our study was that we couldn’t find compelling evidence for traditional applied behavioral analysis (ABA) interventions. There weren’t enough RCTs to compute summary effects on any outcome for traditional ABA interventions. This is concerning because traditional ABA interventions are far and away the most commonly recommended approach in the field.
CCPR: This could change how we look at autism treatment.
Dr. Sandbank: Yes. And even the few positive findings we computed were based on the rare very high-quality studies. The majority of studies were impacted by potential bias, largely when participants were aware of what group they were assigned to. When we excluded those studies, we saw no significant effects on any intervention approach for any outcome for any treatment.
CCPR: Can you talk more about the outcomes you were looking at in these studies?
Dr. Sandbank: Sure. When a child learns a skill and then can use it in several contexts, we call that generalization. When a child learns a skill and then builds on that to develop other new skills, we call that distal development. We wanted to see whether interventions were causing meaningful change in generalized and distal outcomes, or if the child had simply acquired a very discrete skill that they could only use in the exact circumstances where they were taught. If you teach the child 10 words, you may not see growth in general communicative ability. On the other hand, teaching social communication may cascade into language development not just within the intervention, but across multiple contexts.
CCPR: How does this relate to the different categories of treatment?
Dr. Sandbank: The traditional ABA behavioral approaches teach specific skills. The developmental approaches work on foundational skills, such as emotional regulation and reciprocal communication, that support both generalization across domains and cascading distal development. NDBI marries the two theories. (Editor’s note: For a succinct tutorial on the differences between these three approaches, see Dr. Diane Cullinane’s article in this issue.)
CCPR: Why have behavioral approaches dominated treatment?
Dr. Sandbank: It is far easier to do research in traditional ABA approaches that teach specific skills such as new phrases. These are narrow proximal outcomes. It has been harder to conduct research on developmental or NDBI approaches. You need to measure change across a whole domain on a validated, standardized assessment administered by a naïve assessor. That would be a distal effect. Proximal effects appear large, even though they may not generalize nor result in distal growth. Distal and generalized effects are more clinically meaningful but likely to appear smaller.
CCPR: What are the implications for treatment?
Dr. Sandbank: Since the traditional ABA approaches have seemed to be more effective, they have dominated treatment and also insurance reimbursement. And when therapists are trained, they tend to be trained in behavioral approaches. Students train in traditional ABA approaches to become Board Certified Behavioral Analysts (BCBAs). These are BA or MA certifications that are separate from usual mental health specialties and focused on autism treatment without training in more broad mental health conditions or treatments. Therapists learn to teach autistic children specific skills. For instance, they learn to teach children to point to specific objects, a common activity in these treatments. It turns out from our research that this is unlikely to generalize to new kinds of pointing nor other communication. Developmental and NDBI approaches have certifications provided by the training institutions but no central board like the BCBA. Still, NDBI and developmental interventions may have better outcomes. For the same example, the child may learn to use pointing better because they learn it in the context of meaningful interactions. The newer research supports this.
CCPR: The recent American Academy of Pediatrics (AAP) guidelines seem to favor the traditional behavioral approaches.
Dr. Sandbank: Current practice is typically the use of an Autism Diagnostic Observation Schedule (ADOS) as a chief diagnostic tool, which is taken to a psychiatrist, psychologist, or pediatrician who often doesn’t have expertise in autism but recommends 40 hours per week of intensive behavioral intervention. We just wrote an opinion piece in JAMA Pediatrics that urges clinicians to learn about the entire range of treatments and collaborate with families to figure out what might work best for them rather than routinely recommending just the one approach (Sandbank M et al, JAMA Pediatr 2020. Epub ahead of print). Clinicians are gatekeepers who need to know about the different intervention approaches, particularly those supported by RCTs.
CCPR: Can you comment on the recommended intensity of treatment?
Dr. Sandbank: We did an analysis of intensity to see if more hours of intervention was associated with greater effect sizes. We did this for behavioral, developmental, and NDBI studies and found no association between the total hours of intervention received in the study and the size of the effect. That doesn’t mean the amount of intervention doesn’t matter, but it means that we lack the kind of evidence needed to support recommendations for the very high-intensity treatments. It may be problematic to recommend that a toddler receive 40 hours of intervention. This is quite taxing for children and families and it may not be necessary.
CCPR: Families often come to us asking for 40 hours of ABA.
Dr. Sandbank: They hope that this will result in the best outcomes, and yet mere access to these hours is not what the research suggests. About a third of kids with autism who are preverbal at 2 are not likely to develop phrase speech by the end of elementary school, and this is related to effective intervention—not access to traditional ABA (Anderson DK et al, J Consult Clin Psychol 2007;75(4):594–604). Sadly, when kids haven’t made progress, it is common for parents to feel guilty and ask, “Why isn’t my child speaking? Was it because I didn’t start intervention early enough? Was it because I didn’t do the 40 hours?”
CCPR: Does the research support any particular time commitment for these interventions?
Dr. Sandbank: For many treatments, the intensity and duration of intervention really depends on the intervention. There’s a parent-mediated communication-focused treatment called PACT, a developmental approach, that is only 18 one-hour sessions—no more, no less. But we have a really high-quality study that shows it improves the core challenges related to autism and development over time. There are few RCTs comparing intervention intensity. My former advisor, Paul Yoder, just completed a study comparing the Early Start Denver Model (ESDM, a form of NDBI) and a traditional ABA intervention, where children were randomly assigned to receive either 15 or 25 hours per week, and they showed no differences in outcomes by intervention or intensity across the whole group (Rogers SJ et al, J Am Acad Child Adolesc Psychiatry 2020;S0890-8567(20):31350–31352). It is the first study to ask this question in this way. We need more studies like that. A more recent paper found that more hours of intervention led to better outcomes for higher-functioning children. This is interesting since most of the time, it is the children with more severe challenges who are offered higher numbers of hours.
CCPR: How do we decide what kind of approach to recommend? Can we predict which child responds to which intervention?
Dr. Sandbank: It’s not one size fits all. We need to look within the groups for mediators. Maybe kids with better social attention do better with developmental interventions or NDBI. And maybe kids who don’t have that social awareness need more formal, explicit instruction in how to do these things. A lot of families feel that their ABA therapy has been very helpful, in which case we should let that support continue. Does it depend on study-entry autism symptomology, social communication skills, or age? Those are the kinds of studies that we need.
CCPR: What should we say to families about how to choose among the various options for autism treatment?
Dr. Sandbank: We certainly have more high-quality evidence supporting developmental and naturalistic approaches as our research is improving. Autism is a spectrum, and different kids have different abilities and strengths. Some families prefer certain interventions, and some kids improve with certain interventions. I recommend that clinicians describe the different approaches, which means you have to be familiar with them and what’s available in your community; you can’t recommend an intervention that a family cannot access. Identify the family’s priorities. Do they want direct structured intervention from a clinician, or do they think that won’t fit into their life? You have to have buy-in from the family, and it has to be something that the child experiences positively.
CCPR: Thank you for your time, Dr. Sandbank.
Recommended
