Study on Traffic Forecasting Urges Embrace of Uncertainty

Last week in a note about the Technical Advisory Committee at the MPO, I appended a brief note about a study on a favorite topic, "Traffic Forecasting Accuracy Assessment Research" from the Transportation Research Board.

That's a bell-ish looking curve with a bias to the left

The note was too brief, it needed follow-up, and a reader even offered a cranky comment in the absence of more discussion. The comment is dismissive and I am not sure it was offered in good faith, but I will use it here as a jumping off point.

What I find extremely funny about this post is adding a graphic from the city that forecasts biking and walking in 2035 and accepting it as a fact or accurate prediction,* but in the same post adding a graphic to show that model forecasts are not accurate but instead have a distribution (everyone who does forecasting knows this, its not a revelation!!!)

It appears that the author likes to rely on forecasts when it fits their viewpoints and biases, but rebuke forecast when it doesn't!

BTW - that bell-shape forecast distribution show that the majority of forecasts are within 20% plus or minus of the actual future volumes. I have no problem with that level of accuracy. How many people do you think can predict the Dow Jones Index 20 years from now within 20% +/-??

One reason the comment does not seem to be offered in good faith is that its assumed audience is a fellow forecaster or someone with similar knowledge. Of course "everyone who does forecasting" knows about a distribution on forecasts. Here I am not concerned with professional, guild knowledge addressed to other professionals. This kind of insider talk is not the issue. The relevant issue here is the outward-facing information, what public and electeds "know," and they see only a single number in a forecast.

Here is a forecast from the SRC, for example. (You can read a longer discussion of it, with references, here.)

2040 counts from a January 2019 SRC report

If those who generated the forecast understand there is a distribution and margin of error, they have hid it. The forecast as it was presented to City Council is full of false certainly and precision.

The claim here, is that this kind of forecast needs visible statements of uncertainty.

And, indeed, it turns out this paper recommends that. This is the big takeaway: Use a range of forecasts to communicate uncertainty. (See from 2015, "Like Weather Forecasting, our Traffic Forecasting Needs Error Bars," which in light of this study holds up pretty well.)

Recommendation: Include a range to communicate uncertainty

Recommendation: Compare forecast to actual

Recommendation: Report on accuracy

This is important because there is quite a range of uncertainty. Look at the gap between a 5th percentile actual and 95th percentile actual. That is an empirically generated 95% confidence interval - and it's pretty wide!

The spread on a 95% confidence interval is wide

Here are the numbers. Unsurprisingly, the variation is larger with smaller counts and is smaller with larger counts. I am going to use as a rule of thumb +/- 50%. Maybe you would insist on +/- 40%. But it's easy to do half in one's head, and so that's the back-of-the-envelope calculation I will use.

The error is bigger on smaller roads, but as a rule of thumb
we could say +/- 50%

One important reason this matters is that we trigger planning for road construction by a volume/capacity ratio of 1.0 (greater than 0.99). The capacity of a road is theoretically fixed, and if the volumes in a forecast range +/- 50%, then the v/c ratio will similarly vary, since c is a constant value for any road. That is, if the forecast is 100, and the capacity is 100, the distribution is likely to be something like 50/100 to 150/100. Even if you limit it to +/-20% , that's a lot of variance! And if the upper range might apparently call for widening, the lower range emphatically says "stay put."

This table on volume/capacity is from 2015

And we have seen this kind of variance on the historic projections from the previous century. Consequently, we should, as the paper suggests, expect variance on our current projections, and planners should be up-front about this. A few years ago a commenter said, "To expect a model to be 100 percent accurate when forecasting events 20-years in the future is a bit naive." That is true. But again, the issue is not a silly expectation for 100 percent accuracy; the issue is a reasonable expectation that forecasters will communicate appropriately probabilistic levels of uncertainty.

---

* Bike count and bike forecast accuracy is not the main point here, but it has been discussed before, most recently last year. There is no question that bike traffic forecasting has a greater margin of error. The daily variation on bike counts is large also. The point on the chart referenced in the comment, however, was to show a difference of such magnitude (5% goal vs. 0.4% estimated actual, a whole order of magnitude) that no margin of error would close the gap. In that particular analysis, margin of error on a forecast was irrelevant, and proves nothing about cherry-picking.