Problems with the data and methodology deployed by the Federal Motor Carrier Safety Administration’s Compliance, Safety, Accountability (CSA) safety-rating program “produce an imperfect and unreliable measure of a carrier’s safety record,” according to the American Trucking Assns. (ATA).
A white paper just issued by ATA discusses data and research on the connection between CSA scores and crash risk.
Per research cited by ATA, scores in at least three of the system’s measurement categories don’t bear a positive correlation to crash risk. Even in those categories that generally have a positive correlation to crash risk, the paper points out that there are tens of thousands of real-world “exceptions”-- carriers with high scores and low crash rates and vice-versa.
“ATA continues to support the objectives of CSA and to call for improvements to the program,” said ATA president & CEO Bill Graves. “However, data and methodology problems continue to plague the system and the accuracy and reliability of companies’ scores.
“It may make sense for FMCSA to use scores in those categories that correlate positively with crash risk to prioritize companies for enforcement review,” he continued. “In the process, FMCSA can verify whether or not the scores paint an accurate picture.
“But third parties need to know that for the purposes of drawing conclusions about individual carriers, the scores are unreliable,” cautioned Graves.
CSA is used to identify unsafe carriers and prioritize them for future safety-enforcement interventions. In its white paper, ATA points out that FMCSA also encourages third parties (shippers and brokers) to use CSA Safety Measurement System (SMS) scores as a tool for making safety-based business decisions and thus compel carriers to improve their safety performance.
In addition, SMS scores can be used by plaintiffs’ attorneys and prosecutors in the context of post-crash litigation.
Given how SMS scores can be used by third parties and evaluated by judges “raises obvious questions about the accuracy and reliability of the data,” the white paper contends.
So, ATA argues, “the question is whether or not the scores can be routinely relied upon to make sound, beneficial judgments about the safety posture of individual carriers. Similarly, courts must be concerned with whether or not SMS data meet Federal and jurisdictional rules of evidence which require that the data be ‘trustworthy’ and rest on a reliable foundation.”
The paper relates that the American Transportation Research Institute (ATRI) has found a positive relationship between scores and crash risk in three of CSA’s publicly available BASICs measurement categories.
But ATRI also determined that scores in two others bear an inverse relationship to crash risk. Of the non-publicly available categories, scores in one-- the Crash Indicator BASIC-- likely correspond well to future crash involvement, but scores in the other—the HM Compliance BASIC) do not.
ATRI also discerned that the number of alerts that a carrier has been assigned is a strong indicator of crash risk.
However, the strength of the relationship varies depending on the BASICs in which the carrier has alerts-- since scores in some BASICs more strongly correlate with crash risk than those in others.
“The relationship between scores and crash risk is impacted by a number of data and methodology problems that plague the system,” states ATA.
Per the white paper, these problems include:
- A substantial lack of data, particularly on small carriers who comprise the bulk of the industry
- Regional enforcement disparities
- Questionable assignment of severity weights to individual violations
- Underreporting of crashes by states
- Inclusion of crashes not caused by motor carriers
- Increased exposure to crashes experienced by carriers operating in urban environments
Therefore, ATA advises, while there are statistical correlations between SMS scores in certain categories and crash risk, as well as between the total number of alerts assigned and crash risk, individual carriers’ scores can be unreliable indicators of their safety performance.
What’s more, the identified correlations between scores and crash risk “represent industry-wide trends that often don’t hold true for individual carriers.”
The white paper relates that, in most BASICs, there are thousands of carriers (‘exceptions’) whose scores contradict the trends—that is, carriers with high scores but low crash rates and vice-versa.
“The sheer number of [these] ‘exceptions’ and the presence of numerous data and methodology problems lead to the conclusion that SMS scores alone as measures of individual carrier safety performance are, at a minimum, unreliable,” according to ATA.
To review the complete white paper, click here.
