Just two days after the Government Accountability Office sharply criticized the Safety Measurement System’s ability to identify high-risk motor carriers, the Federal Motor Carrier Safety Administration released an analysis concluding that there is “solid evidence” that SMS effectively supports FMCSA’s mission to reduce crashes, injuries, and fatalities. SMS is a key element of FMCSA’s Compliance, Safety, Accountability (CSA) program.
In research conducted for FMCSA, the Dept. of Transportation’s Volpe National Transportation Systems Center analyzed the association between historical carrier data and future crash involvement by examining two years of pre-SMS safety data for a subset of carriers and the carriers’ crash performance over an 18-month period. According to this analysis, the companies that SMS would have identified for interventions - roadside inspections, warning letters, on-site investigations, etc. – had a future crash rate of more than double the national average, 4.82 crashes per 100 power units rather than 2.69 crashes per 100 power units.
The Volpe analysis also found that 79% of the carriers that SMS would have tagged as high risk in at least one of the seven Behavior Analysis and Safety Improvement Categories (BASICs) had higher future crash rates than those that would not have been identified as high risk.
In announcing the findings, FMCSA said that the research had concluded that SMS is more effective at identifying such carriers than the former SafeStat system, which had been used from the 1990s until the agency launched the SMS in December 2010. But the Volpe report doesn’t actually address the effectiveness of SMS relative to SafeStat. In fact, one of three researchers who conducted a peer review of the Volpe research noted a brief mention of SafeStat in the report. Independent researcher Ian Noy, asked, “What is the relevance of the discussion of SafeStat, and how does that relate to the current analyses?”
Noy questioned the fundamental conclusion that the analyses “provide solid evidence that the Carrier SMS is effectively supporting FMCSA in its mission to reduce crashes, injuries, and fatalities involving large trucks and buses by improving safety and compliance.” That conclusion “is over-reaching,” Noy said. “I think the analyses themselves indicate that the program may be suboptimal.”
Another peer reviewer, independent researcher Kristen Monaco, also questioned the conclusions, which she termed “incredibly limited (one paragraph of the entire report).” Monaco argued that the report doesn’t measure whether interventions actually decreased crash rates, “merely that those who should be identified as high risk from the earlier data actually have higher crash rates later.”
In its own review of what it termed “an overly rosy self-assessment,” the American Trucking Assns. said that “it is important to accurately assess the program and its shortcomings in a balanced and honest fashion.” Even so, the self-assessment identified but downplayed several findings that call CSA’s effectiveness into question, ATA said.
For example, the Volpe report mirrored GAO’s findings regarding the limited amount of data available on smaller carriers, ATA noted. Despite the fact that carriers with five or fewer trucks represented 75% of those in the study, the study’s authors acknowledge that there is very little available safety information on these carriers to “make a meaningful safety assessment.”
ATA also pointed out that the report acknowledged that while the aggregated CSA data may be helpful for enforcement purposes, it can be misleading with respect to assessing the performance of individual carriers. The report found that 93% of carriers monitored in the study had no crashes.
“The report demonstrated that if you compare the performance of the relatively few fleets that have scores in the system against other carriers, including those with minimal data in the system or no scores, you can paint a positive picture of the program,” said Bill Graves, ATA’s president and chief executive officer. “ATA continues to support FMCSA’s efforts to rid our nation’s highways of unsafe carriers, but the concerns independent researchers, the GAO and ATA have raised show that while CSA may be a modestly better tool for enforcement prioritization, it can and must be improved.”
