Tuesday, January 21, 2020

Study on Traffic Forecasting Urges Embrace of Uncertainty

Last week in a note about the Technical Advisory Committee at the MPO, I appended a brief note about a study on a favorite topic, "Traffic Forecasting Accuracy Assessment Research" from the Transportation Research Board.

That's a bell-ish looking curve with a bias to the left
The note was too brief, it needed follow-up, and a reader even offered a cranky comment in the absence of more discussion. The comment is dismissive and I am not sure it was offered in good faith, but I will use it here as a jumping off point.
What I find extremely funny about this post is adding a graphic from the city that forecasts biking and walking in 2035 and accepting it as a fact or accurate prediction,* but in the same post adding a graphic to show that model forecasts are not accurate but instead have a distribution (everyone who does forecasting knows this, its not a revelation!!!)

It appears that the author likes to rely on forecasts when it fits their viewpoints and biases, but rebuke forecast when it doesn't!

BTW - that bell-shape forecast distribution show that the majority of forecasts are within 20% plus or minus of the actual future volumes. I have no problem with that level of accuracy. How many people do you think can predict the Dow Jones Index 20 years from now within 20% +/-??
One reason the comment does not seem to be offered in good faith is that its assumed audience is a fellow forecaster or someone with similar knowledge. Of course "everyone who does forecasting" knows about a distribution on forecasts. Here I am not concerned with professional, guild knowledge addressed to other professionals. This kind of insider talk is not the issue. The relevant issue here is the outward-facing information, what public and electeds "know," and they see only a single number in a forecast.

Here is a forecast from the SRC, for example. (You can read a longer discussion of it, with references, here.)

2040 counts from a January 2019 SRC report
If those who generated the forecast understand there is a distribution and margin of error, they have hid it. The forecast as it was presented to City Council is full of false certainly and precision.

The claim here, is that this kind of forecast needs visible statements of uncertainty.

And, indeed, it turns out this paper recommends that. This is the big takeaway: Use a range of forecasts to communicate uncertainty. (See from 2015, "Like Weather Forecasting, our Traffic Forecasting Needs Error Bars," which in light of this study holds up pretty well.)

Recommendation: Include a range to communicate uncertainty

Recommendation: Compare forecast to actual

Recommendation: Report on accuracy
This is important because there is quite a range of uncertainty. Look at the gap between a 5th percentile actual and 95th percentile actual. That is an empirically generated 95% confidence interval - and it's pretty wide!

The spread on a 95% confidence interval is wide
Here are the numbers. Unsurprisingly, the variation is larger with smaller counts and is smaller with larger counts. I am going to use as a rule of thumb +/- 50%. Maybe you would insist on +/- 40%. But it's easy to do half in one's head, and so that's the back-of-the-envelope calculation I will use.

The error is bigger on smaller roads, but as a rule of thumb
we could say +/- 50%
One important reason this matters is that we trigger planning for road construction by a volume/capacity ratio of 1.0 (greater than 0.99). The capacity of a road is theoretically fixed, and if the volumes in a forecast range +/- 50%, then the v/c ratio will similarly vary, since c is a constant value for any road. That is, if the forecast is 100, and the capacity is 100, the distribution is likely to be something like 50/100 to 150/100. Even if you limit it to +/-20% , that's a lot of variance! And if the upper range might apparently call for widening, the lower range emphatically says "stay put."

This table on volume/capacity is from 2015
And we have seen this kind of variance on the historic projections from the previous century. Consequently, we should, as the paper suggests, expect variance on our current projections, and planners should be up-front about this. A few years ago a commenter said, "To expect a model to be 100 percent accurate when forecasting events 20-years in the future is a bit naive." That is true. But again, the issue is not a silly expectation for 100 percent accuracy; the issue is a reasonable expectation that forecasters will communicate appropriately probabilistic levels of uncertainty.

* Bike count and bike forecast accuracy is not the main point here, but it has been discussed before, most recently last year. There is no question that bike traffic forecasting has a greater margin of error. The daily variation on bike counts is large also. The point on the chart referenced in the comment, however, was to show a difference of such magnitude (5% goal vs. 0.4% estimated actual, a whole order of magnitude) that no margin of error would close the gap. In that particular analysis, margin of error on a forecast was irrelevant, and proves nothing about cherry-picking.


Salem Breakfast on Bikes said...

By email, a reader forwards a comment about a kind of control that the study omits:

"While it is frank about the errors in making projections for projects that are actually built, I think there's a subtle 'dog that didn't bark' bias in the research design here.

As you know well from the SRC, the policy question isn't so much the accuracy of the build forecast, as it is being able to accurately compare states of the world (congestion, emissions, etc) between the build and no-build (and possible other options).  This study didn't look at the accuracy of no-build forecasts for projects that weren't built. 
My experience (and other analyses) have concluded that traffic models tend to way over-predict traffic growth under the no-build scenario.  Wes Marshall attributes this to the use of static assignment, rather than dynamic assignment, which allows the model to overload successive links in a system without regard to whether capacity is available.  So in the 'no build' models, they forecast an amount of traffic that one link can't possiblly handle, and then allow that impossible traffic to show up on subsequent links, when, in fact, there would be a feedback loop that would lead to some combination of route changes, mode changes, time shifts, and reduced trip making.

The clue with this particular study is that the word 'induced' appears, as far as I can tell, exactly once in the text, and not in the context of induced demand.

So yes, the 'build' forecasts are inaccurate, and biased.  But what may be more important from a policy standpoint is that the 'no-build' forecasts--which this report doesn't consider--are even more biased, and in the opposite direction, which leads policy makers to too dour a view of conditions under the no-build, relative to the build condition.

That's a good point!

Anonymous said...

Here is a critique of "four step" algorithms -


"...nearly everyone agreed the biggest question is not whether the models can yield better results, but why we rely on them so much in the first place. At the heart of the matter is not a debate about TDMs or modeling in general, but the process for how we decide what our cities should look like.

TDMs, its critics say, are emblematic of an antiquated planning process that optimizes for traffic flow and promotes highway construction. It’s well past time, they argue, to think differently about what we’re building for."