The Government Accountability Office (GAO) released its long awaited effectiveness analysis of FMCSA's Safety Status Measurement System (SafeStat) last month. The GAO analysis found that SafeStat's effectiveness in targeting carriers with high crash rates could be increased about nine basis points by using a statistical regression model rather than the current “expert judgment” approach.
Researchers noted that while an increase of nine basis points may seem modest, the regression model identified a group of high-risk carriers that had nearly twice as many future crashes as the group identified by the current method.
The study looked at 4,989 high-risk carriers, i.e., those identified as SafeStat categories A & B during a July 2004 SafeStat run. Using crash data from the Motor Carrier Management Information System (MCMIS), GAO determined that these carriers, which operated 98,619 vehicles, were involved in 10,076 crashes during the 18 months following the July 2004 report (August 2004 to December 2005). That equates to a crash rate of 102 crashes per 1,000 vehicles.
The GAO team next tested several statistical models known to be effective forecasting tools when using data such as number of crashes, out-of-service inspections and driver violations. Most effective was a technique known as “negative binomial regression,” which assumes that average crash rates and the variance in crash rates are not equal.
Using this statistical model on the July 2004 SafeStat data set, the team identified 4,989 carriers most at risk for future crash involvement, and then determined how many crashes they were involved in between August 2004 and December 2005. The number was 19,580, or 111 crashes for every 1,000 vehicles, which is a net increase of 9,514 crashes.
In other words, SafeSat's current “expert judgement” method sometimes fails to identify carriers with unacceptably high crash rates. (In statistician speak, this is known as a Type II error, or false negative.)
In 2004, the Oak Ridge National Laboratory produced nearly identical findings.
The potential for Type II errors was also noted in a 1998 National Private Truck Council petition that asked FMCSA to employ an independent team of researchers to determine the most effective statistical method of identifying at-risk carriers.
FMCSA rejected the findings, however, because such an approach could shift the focus from carriers with known safety violations to those with high crash involvement rates. In reviewing the agency's opposition to this approach, GAO did find a correlation between future crashes and such non-crash carrier data as driver- and vehicle-inspection violations and compliance review violations. Nonetheless, it concluded that crash data was the best single predictor of at-risk carriers.
FMCSA could readily incorporate these statistical techniques into the SafeStat algorithm. Granted, the agency is conducting a complete overhaul of its safety ranking system, part of an effort known as Comprehensive Safety Analysis 2010. However, that overhaul will not be complete until 2010.
Given that research has repeatedly proven that the current expert judgement approach sometimes fails to identify high-risk carriers, FMCSA should act immediately to implement these improvements.
Go to www.gao.gov; search for report GAO-07-585.
Jim York is the ass't. vice president of technical services for Zurich Services Corp. Risk Engineering in Schaumburg, IL.
