everyone and welcome to
our its seminar and today we are very
delighted to have professor fred mannery
as our speaker for today uh professor
mannering
is the current associate dean for
research
in the college of engineering and a
professor of civil
and environmental engineering at the
university of south florida
he also holds an apprenticeship
appointment in economics
before joining university of south
florida he was a professor at penn state
a professor and department chair at the
university of washington
seattle and school head and chair
professor
at purdue university all in civil and
environmental engineering
and dr matters research focuses on
applications of economic and statistical
method
to the analysis of highway safety
transportation economics
vehicle demand travel behavior and a
variety of
other engineer related problems
he has published extensively in these
areas and his work
is highly cited he also co-authored two
popular textbooks one is principles of
hybrid engineering and traffic analysis
which i use in my undergraduate course
here for more than 20 years
and another book is called a statistical
and econometric
method for transportation data analysis
professor mannering is currently the
editor-in-chief
and also the founding editor of the el
severe science journal
analytic method in accident research
and a previous editor-in-chief and
current distinguished editorial board
member
of the sl severe science journal
transportation research
part b the logical
one of the premier uh transportation
journals in the world
so this is his official bio but i would
like to
let you know that when he's not doing
all the above
he plays rock music and also races
motorcycles
uh i'll let him to uh tell you how he
managed to do all this
you know the time he has uh fred and now
the zoom room is yours
okay so maybe uh well i'll talk about
the
rock and roll and the motorcycle at the
end of the lecture right
although i do have my guitars couple
guitars out in the background
and in the corner here i guess this
corner that's actually a cutout of
johnny depp
from pirates of the caribbean it doesn't
quite show up in the zoom screen but it
does in the microsoft teams is a wider
angle it usually shows up
so today um you know before
before we got started we were talking
about machine learning so
today's talk i'm going to talk about
different methodological approaches from
traditional statistics
to more advanced what we call
heterogeneity models that use mixing
distributions
to causality models and then to sort of
full out machine learning
so we're going to look at the
differences in these and
the application that i'm going to
provide today
is for highway safety but
a lot of these same issues that i'm
going to talk about today
come up with almost any transportation
related data
and even beyond transportation related
data these uh
elements will come up so first i'm going
to
uh look at and mention
that these today's talk is really based
on
two papers one is a um
a paper that i recently wrote with uh
chandra bad at ut austin
venki shankar and mohammed abdallah t
who is actually a
university a california davis graduate
i should mention is a little departure i
was on sabbatical leave
at the university of california davis in
1992-93
academic year probably a lot of the
people here today weren't even born then
but muhammad abdellati was a doctoral
student at that time and we
shared an office um and so that's how we
got to know each other
but so that's going to be the primary
paper
that will contain the material that i'm
talking about today but i also have a
2018 paper that talks about temporal
instability
in the analysis of highway accident data
but there's temporal instability and a
lot of transportation related data and
i'll talk about that a bit in this talk
as well
and i want to special thanks to clifford
winston
at the brookings institution who's
actually my doctoral advisor
at mit because we talked for
years about some of the concepts that
i'm going to be talking about today
because you know he's a uh a full
uh economist and you know i'm sort of an
engineer slash economist so we have a
lot of debate about
endogeneity self-selectivity all these
things
we'll chat briefly about in today's talk
so let's look at highway accidents
and if we think you know when you when
you look at the
infrastructure bill that might be coming
out of washington d.c
at some point you know safety is one of
the key issues that would be addressed
in that
but there's a lot of elements like that
are coming up in highway safety like
autonomous vehicles and how people's
behavior will change around autonomous
vehicles because we're going
until we go to fully autonomous vehicles
we're going to have an
interesting mix of people-driven
cars human-driven cars and autonomous
vehicles and
it's going to be an exciting 10 to 20
years of
research until we get fully autonomous
vehicles then the mode is going to
change but
the way people's behavior is going to
change around autonomous vehicles is
going to be a fascinating
research topic for many years so
if we look at highway accidents uses
about the general consensus you know 1.2
million people per year probably 50
million more injured worldwide
and safety researchers have sought for
decades to reduce this carnage by
doing some sort of statistical analysis
mostly statistical analysis
although neural networks and other
methods data-driven methods were also
used and i'll talk about that in a
moment
when we look at the in the analysis of
highway accident data studies have
really focused on two elements
the likelihood of an accident and then
the resulting
injury severity and when you look at the
likelihood of an accident
you usually have accident counts per
some
period of time and then the resulting
injury severity
usually police reported data are used
and
they use a five-point injury severity
scale no injury possible injury
evident injury disabling injury and
fatality
and the way that the accident data
recorded
you have 30 days to die so if you're in
a crash and you don't die within 30 days
you'll be
listed as a disabling injury if you die
in the 31st day you won't be listed as a
fatality
but when we look at this and again
mostly from police reported data
we can you can safely say that you know
many insights
have been gained by the analysis of data
these data
but there are some very critical
elements
one is the police you know using police
reported data
is going to have some issues and i'll
talk about that in a second
we have vehicle driver data because we
now have some naturalistic
driving studies where you collect you
know very large amounts of data
you know even we could look at like eye
movements of the driver very detailed
data so you know far beyond what's
police reported data
then we have site observational data
where we can use videos and sensors to
see
not necessarily if there is a crash but
how close
vehicles came to crashing and so there's
a whole
body of research that looks at time to
collision and sort of
methods like that that are based on
observational data
when we look at police reported data
that's the most widely used and
available
and if you look at a police reported
data there's a lot of information
in there but there are some problems
with police reported data so
the police i'm sure some of you have
been in accidents the police go to the
scene they fill out the age of the
driver you know they get a lot of
information on road conditions
so there's probably hundreds of
variables and if you link it to
state databases you can link it to the
coefficient of friction of the highway
and all sorts of things so you can vary
large
amounts of data but there's
an under reporting of minor crashes you
know because a lot of minor crashes
aren't reported to the police so that's
always a concern
there's potential inaccuracies in the
attributes and injury severity levels
like i mentioned at the beginning if you
die in your 31st
day in the hospital you're not recorded
as a fatality be recorded as a disabling
injury
so there's possible some inaccuracy in
that
in other attributes and most importantly
and this is going to be one of the
concepts i'm going to talk about
throughout today's presentation is it's
only gathered
for people that are observed in the
crash and so this is something
economists spend a lot of time on is
sort of this
self-selectivity and identification
issue and i'll you know again i'll
explain this a little more
in the slides forthcoming but we do not
have
the people that are actually observed
involved
in crashes are a self-selected group of
probably more crash prone drivers
we do not you know ideally what an
economist would do in experimental
economists is they would develop an
experiment
and have everybody drive into a brick
wall or something and then they would
have a fully
uh a sample that fully captured you know
a complete
sample of the population but since we're
dealing with a self-selected sample of
risky drivers
that has implications for the types of
models we use
and the conclusions that we draw so i'm
not going to
get into that in a little more detail in
a second for naturalistic driving data
here you gather massive amounts of data
because you have a fully instrumented
car so you're going to get
acceleration deceleration yaw rates you
know all sorts of things
but this has problems with the subject
selection
uh your gen you know the people that
actually use these cars have to be
generalized to the entire population
the behavior of drivers includes route
and time of day choices
that are self-selected so you still have
even the safer drivers are going to
probably go
on what they feel is a safer route so
the route choice and all of these things
still have sort of a self-selectivity
in them because we don't observe the
safest drivers maybe on the most risky
routes
so you might think it's like oh that one
intersection is dangerous and i just
try to avoid that so that's going to
create a potential self
selectivity problem as well that's going
to affect the identification
of parameters when we estimate these
models and the other things with
naturalistic drivers
is you tend to have few observation of
catastrophic events because
you know you have a limited number of
vehicles that are fully instrumented and
then you don't actually observe them in
a crash
i should tell you that it reminds me you
know
for a minute because we talked about
motorcycle racing and
one of my crashes was on the las vegas
motor speedway
i was invited by a honda
they were testing back then it was the
cbr 600 rr i think these were in the
early 2000s but anyway i crashed
and i've they figured oh yeah so
they had some racers go to the las vegas
motor speedway to test this
new motorcycle and they were collecting
all sorts of data on the fuel injection
and acceleration deceleration
but after i crashed i figured well
that's it i'm going to be out
but it said no no no go back out go back
up because when you crash you get an
incredible amount of information so they
were hoping i'd go out and crash again
or something but
anyway that's i hadn't thought about
that story until now but
so there are uh well i've thought about
the story before but i haven't thought
about the analysis here there's
great information when you actually have
a catastrophic event because you get a
lot more data than
if you don't have a catastrophic event
so there's also
site observational data so here
as i mentioned previously we have
massive amounts of data gathered from
video and potential sensors and other
things we may have in the roadway
so these data have problems because we
don't gather the social economic
characteristics of the drivers
you know we usually don't observe all
the details of the drivers
the behavior of observed drivers still
has this route and time of tape choices
that are self-selective
again you have the case where it's a
dangerous intersection you're not
observing
all of the people because some people
may avoid dangerous intersections
and again we also have here a few
observations of catastrophic events
because your probability
of observing a crash at a specific
intersection you may get one a day or
one a week so there's not going to be a
lot of catastrophic
events so let's look at what are the
analytic alternatives now these are the
analytic alternatives that
have been applied in safety studies but
you can think of this in almost any data
that you might analyze that's in
transportation or
really in any field so we have
what i would call traditional
statistical models so these are
things models that you probably had in a
lot of your statistics courses
ordinary least squares multinomial logit
all of these standard fixed parameter
models that you probably
hazard based duration models if you look
at my textbook which was mentioned you
know statistical and econometric methods
you know almost all of these models are
in there of
for the first 15 chapters but starting
in chapter 16
and this is sort of 16 17 and 18 we
start looking at the heterogeneity
models now
heterogeneity models probably the one
that
you may have been exposed to is like a
mixed logic model so
these models have mixing distributions
in the parameters to try to capture what
is commonly referred to as unobserved
heterogeneity
so these heterogeneity models at
a level of complexity that tends that
attempts to capture
all sorts of uh unobserved elements of
the data
so these heterogeneity models have
become the most popular model
used in sort of advanced safety work you
know there's random parameter models
latent class
models all of these models i would
classify as heterogeneity models
next we have the data driven method so
this is the machine learning neural
networks
all of the stuff that just analyzes a
correlation
and patterns in the data without
necessarily much
thought to underline causality
and i'll i'll talk about what i mean
there in just a second
but lastly we have what i would call the
causal inference model
so this is if you look at the economics
profession
this is where modern economics is
really focused on what i would refer to
as causal inference models
economists focus on uncovering causality
and in so doing they're going to have to
make compromises
and stuff but they really want to
understand what is driving the
underlying process
where some of these other models to
varying degrees
ignore causality and i'll explain the
details of that in a second
so let's first look at traditional
statistical models so if you've had
a course in high school or undergraduate
this is probably
what you've ordinary least squares
regression three stages squares two
stages all that stuff
i would classify as what is as a
traditional
statistical model so it uses
conventional statistical methods
and this is if you use the highway
safety manual or you're familiar with
the safety field
these are the models that are used for
safety practitioners
there's a claim of predictive
capabilities and causal explanation
but there's a general lack of
fundamental support for these claims and
i'll talk about that in a second
and a lot of times you can easily show
that the parameter estimates are biased
because
to develop models that can be used in
the field a lot of times you
don't have access to very large amounts
of data
that you need so there's an emitted
variable bias so there's
there's a a lot of times if you use a
traditional statistical model
you end up with omitted variable bias
and other issues that i'll talk about
soon now if we look specifically
at the case of highway safety analysis
we have
poisson and negative binomial
regressions because remember what i
talked about we're looking at the
likelihood of a crash
and then the injury severity of a crash
once a crash has occurred
so when we look at the likelihood of
crashes we usually want to look at the
number of crashes per month
number of crashes per year on a highway
segment
or perhaps at an intersection and since
these are
non-negative integers we usually use
count regressions you don't want to use
a normal ordinary least squares
regression because you don't account for
the fact that that's a non-negative
integer and you end up with a continuous
variable
so poisson and negative binomial
regressions are most often used for that
but there's also been applications of a
tobit model
where you have censoring at zero because
a lot of intersections or
highway segments that you analyze over
some length of time may not have a
crash during that time so there's going
to be a zero so tobit models have
been applied and you know just sort of a
background the tobit model was developed
by tobin who was a
economist i think he won the nobel prize
in economics
he developed the model tobit model in
1958 won the nobel prize maybe in the
1970s but not for the tobit model we'll
talk about
other nobel laureates later in this talk
that actually
won the nobel prize for the methods that
i'm going to uh
talk about today then we have ordered
probit model and multinomial logit
models that are often used to determine
the injury severity of crashes so as
again
if you recall i mentioned there's five
injury severity levels
no injury possible injury evident injury
disabling injury fatality
these are discrete alternatives so we
apply an ordered problem ordered probit
model or probability model could be
ordered logit or whatever
in a multinomial logic to determine the
probability of those outcomes
now the heterogeneity models use these
traditional models as a starting point
and then add mixing distributions to
account for unobserved effects
so these the idea with these models is
that well
you know i know that there's uh
everybody has a beta
some beta in the population but some
people may have different betas than
other people so
using these mixing distributions and
this gets to the mixed logic
all sorts of random parameter models you
can
mitigate potential parameter estimation
by
bias and provide additional insights but
at the downside the model complexity can
be very extensive because you can look
at
using mixing distributions with
heterogeneity and the means
heterogeneity and the variance of the
distributions
and you can even have correlated random
parameters it can become a very complex
thing
but what i'm going to show you though is
that these heterogeneity models are sort
of a
link between traditional statistical
models
and almost like a machine learning you
know there's
they fall somewhere in the middle and i
have a nice little diagram at the end
that
will show where these various methods
end up so
the heterogeneity models expand
traditional statistical models with
random parameters random parameter
models heterogeneity and the means and
variance correlated random parameter
models
latent class models also called finite
mixture models
latent class models with random
parameters markov switching models
and you know if you want to get into
this uh
you know into these mixing models just
send me an email and i'll send you some
you know great papers on these things
but
you know these are going to have issues
with them as
all of the methods i'm going to talk
about today have so let's look at data
driven methods because this is the big
fad now right because
this can handle extremely large amounts
of data
provide a high level of prediction
accuracy i mean if you look at machine
learning
even neural networks and stuff the
prediction accuracy is amazing
the downside is that you don't
necessarily provide
insights into underlying causality so if
there's some
fundamental shift in behavior these
models do not predict well
do not forecast well in real time
they're very good because you're
unlikely to have these
what we might view as a temporal shift
in the betas
but if you if you're using machine
learning to predict two or three years
out it can be problematic because you're
not getting
insights into causality so
data-driven methods data mining
artificial intelligence machine learning
neural network these are methods
that analyze the correlations and the
complexities of data
but without really putting an emphasis
on
underlying causality and that gets us to
what the
you know the economists are using a lot
and this has become a big thing
in the economics field with experimental
economics and so on
is they explicitly recognize what an
econ
economic model would an economist would
do is expectedly recognize that
accidents are only observed for portion
of the driving population
and that this can lead to erroneous
interpretation
now to able when you have this sort of
self-selected uh problem is that you're
not observing acts everybody getting
into an accident
there are corrections for that there's a
mathematical correction
and then there are methods where you can
use fixed effects and other ways to
mitigate this potential problem the
downside with these causal inference
models and
as i'll explain in a few minutes is you
cannot provide the level of detail
that's often needed by practitioners
for a causal inference model to be
implemented
usually you have to simplify the data to
such an extent
to to be able to undercut uncover the
underlying causality
so conf you know causal inference models
are
typically based on ordinary least
squares regression which has great
properties it has great small sample
properties
uh that make it ideally suited to this
and
usually to account for the endogeneity
and self-selectivity that's encountered
they can use instruments fixed effects
there's a lot of tricks that
can be applied to try to avoid this
issue that you're not observing
everybody getting into
an automobile crash so let's look
more at this causality and the
identification problems
so here's if you
when you want to determine causality
we're giving you safety examples
again i mentioned crash data are
selective sample
because we only observe individuals who
are in crashes
individuals observed in crashes may not
be a random sample of the population
you know for sure they're not a random
sample the population
but on one level everybody has a
probability of getting into a crash but
since we don't have a
fully random sample this can create
problem in identifying causality
and this identification problem has
dominated
the economics field i think they've gone
overboard
in economics but we've the if you look
at engineers and how we've analyzed data
we probably have not given enough
thought to this
so somewhere in the middle of what we're
doing with our
heterogeneity models and what the
economists are doing with their
causality models
as sort of a happy medium so
this uh has dominated many fields
and has also given rise to like
experimental economics so
let me give the great example of this
and because i'm a motorcycle rider this
really makes this sort of
self-selectivity
problem really clear so
let's say we estimate an injury severity
model so we're developing a model that
predicts a probability
of a motorcyclist that you know
condition on them being in the crash
what is the likely outcome of that crash
you know and we could use the five
no injury possible injury evident injury
disabling injury and fatality
suppose we have an indicator variable
whether it was
raining at the time of the crash so and
we find that that indicator variable
has a positive effect on being severely
injured so
we find that rain the rain indicator
variable
increases the severity of the crash so
the problem is is that becomes difficult
to interpret
because we don't know if one the rainy
conditions
actual rain increases the pro you know
the severity of the crash or
what we're observing is that safer
riders just don't ride in the rain
and so the people that we observe in the
rain
are just dangerous riders so when we get
that result
in our model estimation we can't
interpret what the effect of rain is
because we're not sure
if it's actually the reduced friction
that's causing the increase in severity
or the fact that now we're just
observing the riskiest drivers and the
riskiest drivers are more likely to be
severely injured when they crash
so this is the problem that
economists try to uncover with these
causal inferences they try to untangle
the effects
of the actual physical element of rain
and the self-selectivity of riders not
riding in the rain
and i can tell you as a motorcycle rider
i did not ride in the ring
yeah it's like i uh you know i know how
the bike performs in dry conditions
same thing on race tracks i don't race
in the rain so if it's raining i'm out
of the race
so um i just you know don't want to go
there so
with the motorcycle example this creates
identification problem because as i just
mentioned
does rainy weather increase the injury
severity of all riders or does rainy
weather have no effect on the injury
severity
and our finding simply reflects the fact
that injury-prone riders are more likely
to ride in the ring
so in this case it may be a combination
of the two effects but we're not going
to be able to uncover that unless we
explicitly address the self-selectivity
issue
so identification is an issue in other
transportation data analysis so this is
not just
um you know injury severity and
highway safety so if you look at equity
studies
your underprivileged are going to be
underrepresented in many
samples so if you're looking at uh you
know how's
uber going to be applied to
disadvantaged people
um you're going to have a self
selectivity because any
data you uh observe you're going to
probably
under privileged people are going to be
less represented in the uber sample and
this is going to create this
self-selectivity problem
individuals observed on specific routes
are a self-selected group and chapter 15
of my
statistic and econometric methods uh
textbook actually addresses this
specific sample
so there if you look at the chapter 15 i
have
people from a single origin to a single
destination choosing
three different routes a freeway an
arterial and a rural road
but the thing is you only observe people
taking one of these routes probably the
fastest drivers go on the freeway
and if you close the freeway you don't
know how they're going to behave if
the fastest drivers were going other
routes ideally you would set up an
experiment
where you had everybody go on each of
the routes and then
observe their travel times and behavior
but since we don't have that we have an
inherent self-selectivity problem
now this you know the example 15 in the
book is a simple ordinary least squares
model that tries to predict the
average speed of people on these
different routes
and there is a correction for that jane
james heckman won the nobel prize shared
the nobel prize
in 2000 with dan mcfadden um you know on
his heckman style correct you know
selectivity bias correction
so that's what the chapter 15 looks at
that but i'm going to talk about the
heckman correction in a few minutes so i
just want to mention that now
so individuals observed owning driving
uh only one type of vehicle
uh actually going way back to 1983
when i did my dissertation on
vehicle ownership and vehicle use back
in 1983 i was like
five years old no that's a joke let me
see wait i got my
um actually yeah again probably you know
way more than happy more born in 1983
that was uh an issue that i had to uh
address in my dissertation because again
you only
you you don't observe uh you you can
look at it
we know that people that own new cars
drive more than people that don't use
cars but we don't observe people that
have
always have used cars we never absorbed
uh
observe them driving new cars so
we have a problem is that for old and
new cars you don't observe everybody so
it's the same sort of self-selectivity
problem that we're talking about with uh
route
and then also with people getting into
accidents so you can
there's similar examples that pervade
transportation modeling
modeling based and observed data so
dealing with self-selectivity
identification
causality we can look at the traditional
models that i talked about which is like
the ordinary least squares
they do not address the self-selectivity
the data-driven methods machine learning
and so on
does not address the self-selectivity
heterogeneity models
mitigate the effects with mixing
distributions
but they do not solve the problems and
you know you have to make distributional
assumptions there's a lot
you know so you're getting some
mitigation of the effect
but you're still going to have it causal
inference models
explicitly solve the problem but to do
so often they must
greatly simplify the data using fixed
effects instrumental variables and other
things
so if we look at this sort of cell
selectivity
identification causality in
heterogeneity models the problem can be
theoretically solved with this heckman
style correction so this is a chapter 15
from my textbook
you can if you have ordinary least
squares you can
develop this heckman selectivity
correction again you want to know about
price for this
but if you extend that to the non-linear
models
like logit models probit models
that we use in highway safety it's a
mathematical nightmare
to do a selectivity correction in a
non-linear model
especially with a non-linear model with
mixing distributions
so it's just not possible to do a
traditional heckman style
selectivity correction so when you're
deciding
what methodological approach to use any
almost any of you that are doing like
you know dissertation thesis or whatever
if you're
using some methodological approach you
probably say oh yeah i'm going to use
machine learning because everybody's
doing that now
you should think about the slides in
this presentation and
what that means you know because it'd be
a nice sort of summary to put the
method that you use in the context
of the available methods but when you're
thinking of which
methodological approach you can think of
the following considerations
one is the predictive capability the
ability to provide
you know detailed usable findings
the second would be causality
the ability to understand the causal
elements
of the occurrence and injury severity to
guide policy decisions so this is
something
that economists would focus on and then
just the data characteristics i mean if
you have uh
you know hundreds of millions of
observations
using traditional statistical methods or
heterogeneity models just not going to
be possible so the data characteristics
and size
also plays an element now i have this
diagram
that sort of shows how these them
how each of the approaches that i
discussed
sort of fit into this uh sort of you
know these
uh predictive capability big data
suitability
and causality inference capability
so if we look at the causal inference
models that are outlined in the yellow
here let me i guess i can
move my pointer over here if we look at
these in the yellow this is what
economists when you think of economists
uh the data that they analyze they're
well
they are very interested in causality
and
since they focus so much on causality
they
uh compromise predictive capabilities
because they don't
they can't have the very large amount of
data
that is often required for good
predictions
and they also are not suitable to big
data if we look at the second
sort of model which would be traditional
safety models which are in this blue
area
you get a little bit of causality a
little bit of predictive capability
with the heterogeneity models the
heterogeneity models are pretty good
with predictive capability it's a
pretty good with big data and they're
very you know better than causality not
as good as a causality model but
you know better than traditional
statistical models
and if we look at data driven methods so
we have the heterogeneity models are
green are going to be tilted toward the
right of the triangle and the data
driven methods have
excellent predictive capabilities unless
you go
unless you go like five six months you
know
years seven years out then you start
getting shifting
temporal effects but they're very
obviously with the large data the big
they have excellent big data
capabilities but
on the downside they're not going to
uncover causal effects so you can sort
of
look at these four basic methodological
approaches and see how they fit into
the predictive capability causality so
really
what you when you're selecting a method
you're probably you're making some
compromise no matter what method
you choose now the method that i've you
know i've been using in my research is
sort of this heterogeneity model so
you can see it doesn't do anything the
best
but it does everything a little bit you
know is a little bit
well it does everything a little bit
well but not anything great
so when we look at the current
state of traditional safety models so
let's look at
the these methods and see where
everything stands
simple simple statistical methods
continue
to dominate safety practice because
they're real easy to use
and these models form the basis of the
highway safety manual in efforts to
improve it
for heterogeneity models heterogeneity
models have rapidly evolved to dominate
the safety literature but in practice
people have not understood their
application very well
so one of the problems with um
heterogeneity models so if you have a
random parameters model
a lot of people have been erroneously
using the mean
of the random parameter that was
estimated in the heterogeneity model
but really it's there's a couple papers
that have come out recently that show
you have to use the variance as well as
it provides obviously biased results so
the practical application
of heterogeneity models even though
they've taken off in the academic
literature
um they're just it's still going to
probably be like five years before they
hit the practical literature
for data-driven methods you know these
are the the current
fad you know gaining wider acceptance
have obvious applications in
real time data processing where issues
of causality are
a little concerned you know there's
going to be a big place for data-driven
models
obviously in the future for causality
models
if they really have not gained a lot of
traction
in the safety field for two reasons
they do not have the potential for
example to handle
detailed predictions so for example a
lot of times
you have a specific highway it's like
okay i want to increase the shoulder
width from four feet to six feet
how's that going to affect the frequency
of crashes
to be able to make that kind of
prediction you need
pretty detailed data and usually if you
want to
undertake a causality model that
addresses that self-selectivity
question you're not going to be able to
answer a question like that
but if you don't account for the
causality you'll be able to answer that
question
there's probably going to be some error
in your estimate though
so safety researchers have not really
thought much about
the selectivity problem and the impact
that it has on estimates
so the extent of this problem is not
known yet although
there are a lot of recent papers that
are going to start looking at this
self-selectivity problem a couple that
have already been published in a couple
that are under review
uh in the journal i edit now so
the future when you think of these
different methods you're probably going
to
end up with some mix between
heterogeneity models and data-driven
approaches
that seems to be the most fruitful
direction
so heterogeneity models can still
potentially
unravel some causality effects but
there's probably going
you're probably going to see some hybrid
approach between machine learning
and heterogeneity models which sort of
link back to the
fundamental statistical approaches but
there also needs to be very serious
thought
has to be given to this self-select
self-selection identification problem
which again has largely been ignored uh
in the safety field
so again if you look at this uh sort of
diagram you can see here
that again all the methods are going to
have the trade-off
but we've really been short changing the
causality
inference capabilities of the model and
if we look at
the individual ones you can see that the
data-driven methods
if you look at what data-driven methods
and heterogeneity could potentially
bring
you can see we get to almost all the
ends of the triangle i mean we're still
losing some on causal inference and
tada but it seemed that some combination
of data-driven plus heterogeneity
methods
could get a pretty big area in this
triangle
so sort of a summary and conclusions
um i think you know an ideal model would
be one that
uncovers uh causality has excellent
predictive capabilities and scalable to
very large data
but with currently available methods
safety analysts and
analysts in other fields are going to be
forced to make a trade-off that can
entail serious compromises there's a
clear need in safety
probably other transportation fields as
well to ground
intrinsically predictive models which
are like machine lear
learning with causal frameworks so
intrinsically predictive models
especially for big data
can can also improve causal structures
through insights from association
involvement variables not typically
available in traditional safety data
the promising direction for future
research i believe
is a hybrid approach for data driven and
statistical methods with strong
consideration for causal elements
such a hybrid approach is likely to be
uh
perfected over time as an integrative
technique
um and access to more big data becomes
available
so that's uh sort of the
outline i'm not sure how we're doing for
time i guess we're okay
we're doing great yeah good good
well thank you fred for a you know just
really a great overview
of the landscape of uh you know safety
uh
modeling using statistical tools and as
you mentioned
uh this uh also you know similar
problems also
exist in other data analysis for
transportation like if you have travel
surveys
for you know for example coupling or
other
applications yeah um i um
so i would like you know now open up the
q a and i want to remind our students
that
you know you use your wrist hand
sort of function to uh to ask questions
and for our
outside audience you can use the q a to
post your questions
since some of our audience are the
students
be rich for the class and so we'll have
them to
ask questions first uh but before we uh
you know start the q a you know remember
in the start i think uh
uh we would like to hear a little bit
more about fred
other interests and that uh you know and
then
sort of he's not your uh you
your your your sort of usual professor
and uh
um so it will be uh interesting good to
know how
he you know his trajectory
uh as an academic and and i know
many of our students uh who have an
interest
you know in the academic career will be
interesting to know and also know that
a professor is not just writing papers
okay let me uh let me share my website
here
so um yeah
you know because again you know
everybody has a life outside of
academics so a couple you know
interesting things that i've done over
the years
um is motorcycle racing so most of this
was uh actually in the early 1990s
although i
still did track days and stuff until the
2000s but
this is a a picture of me testing
i think yeah this isn't that the one i
crashed on but i could yeah this was
during that
this this i think i'm not sure how many
are motorcycle especially older
motorcycles i think this is
the this was the cbr600rr that i
actually crashed on
maybe a day after this was taken but
anyway
i did it i always had the really cool
helmet so
you know i had the dragon with the fire
coming out and stuff so
but you know motorcycle racing is uh
very dangerous but it's it's very
attractive
because you know motorcycles go zero to
sixty the bike that i have now
which is this uh cbr954r
is like zero to sixty in two and a half
seconds but i was reading today
that tesla is coming out with a
roadster that goes zero to 60 in 1.1
seconds
so you may have you may have seen that
uh story i think it just came out today
actually
so um and these are some of the
uh cars and motorcycles i currently have
but the other
sort of interesting element and you know
everybody can think about this in the
in the early 90s uh i was in a heavy
metal band
in seattle and this was when um
nirvana soundgarden oh you know
everybody that was anybody
was from seattle so and we did
uh we did a cd we had interest uh from
there was metal blade records in los
angeles was interested inside
so i at the time i had actually i just
returned
from uc davis my uc davis sabbatical
and while i was on sabbatical at uc
davis
i was i was taking guitar lessons was i
can't remember the place
in uc davis now but there was this
guitarist
and we were really experimenting with
some you know
interesting uh guitar methods but this
so i can attribute my time at uc davis
for launching my music career
as it was so about a year after i got
back to the university of washington i
started getting serious about music
we created a cd
um again there was and if you want to go
to the website you can download it we
have some full videos of some of the
songs and some of them are just on mp3s
i should show you i have um it's
it's heavy metal so i have my heavy
metal guitars with
uh you know skulls and crossbones and
all the cool stuff
so um but uh yeah
just give it a little history uh
we uh we kicked by the time metal blade
records was interested in us we kicked
the singer out of the band
because and you'll understand why we
kicked him out when you listen to the cd
but
um we did get a really good singer but
he was a real heavy metal guy you know
with drug problems and everything and
i i had written three new songs i said
look we got to get this
get this guy in the studio send the
demos to metal blade records we'll be
millionaires and they'll all be over but
the guy had you know he's missing
rehearsals and stuff
but he had to go into rehab and then we
couldn't find another singer
and then i realized this is taking a lot
of time so i better go back and be a
professor
but the students the students often ask
me
if um if you had been signed uh
would you cons you know what would you
do would you be a professor or go into
rock and roll and say it's obvious i
would be a rock star anybody can be a
professor
but um yeah so that didn't work out
although i should mention the album was
released in 1994
and technically my band is eligible for
the rock and roll hall of fame
so if you want to do a write-in vote i
think we sold like
2 000 cds so we're not quite up there
with jethro tall and some of the other
people that haven't been inducted yet
but you know there's like a a one in a
billion chance right who's to say
there's never a chance
but you know for rock and roll
eligibility you have to be 25 years out
from your first cd
so we felt uh lucky that you you made
the
i think you made the right choice for
the transportation
yeah yeah but it yeah i think it's it's
important to rem
and when you think about it if i look
back on my career all of these
sort of hobbies you know the motorcycle
racing
and the music they all bring something
back
to what you actually do as a professor
and you can think about that as you're
going through your career
you know nothing is really a waste of
time because when you do something
like you know there's a lot of uh you
know things that i've done when i was
racing motorcycles that gave me ideas to
write papers
um you know that actually ended up you
know some motorcycle safety papers that
i've written in others
so you know nothing is really a waste of
time even binging on netflix
series you can probably get some ideas
and
and you know it's been my experience is
that the researchers that come up with
really good ideas usually it's
other aspects of their life that
motivate them to bring
something from another aspect of life
and bring it in and
something that you know people have
never thought about uh
viewing a problem that way before um so
you know when somebody says ah you're
watching too many netflix series that's
like no no
you know i'm gaining information that
will help me you know propel my
dissertation to new excellence or
whatever
yeah that's that's good to hear and so
this netflix don't feel now on the other
effect if you're not actually processing
data it could be just a waste of time
right it could be uh we've got a data
gap
job or you have to uh to
use the information intelligently yes um
so ray i i saw you raise your hand a few
minutes ago
so ray you can ask your question
yeah sure um thank you fred it was
amazing we have one more seminar this
quarter
but i think your representation would
have been
an excellent conclusion for this this
quarter
um you mentioned
you talked about all this amazing
information about models and also how to
be a good academic
um person but
i was wondering if you can give us some
advice
on how to be a good modeler um
given your background and your career
um what it takes to be a fair
and conscious modeler in transportation
field
yeah that's um you know that's a
difficult if i
if i look over the many years
um it's you know because
when you collect data and when you
develop
models you you have inherent biases
uh and so i um i remember had one
student a few years back
where you know the statistical modelers
was
telling her that a variable she thought
would be significant wasn't significant
but she insisted that it be in the model
because i know it has to be significant
so what i think is important
when you're developing statistical
models or doing
whatever is to let the
data speak to you and then think about
what it's saying because a lot
you know when you go in to develop a
model you have an expectation of what's
going to be an
important explanatory variable and
invariably when you start estimating the
model
you find that what your expectations are
not
100 correct or your hypothesis or theory
is not being supported but there's often
reasons for that
that may not necessarily mean that your
hypothesis or theory was incorrect
but they're different the way that data
become statistically significant there
could possibly be
different explanations and what i have
noticed
uh in recent years probably since the
early 2000s
when we start using the heterogeneity
models with mixing distributions is a
lot of times
the contradictory findings that we were
coming up with
using fixed parameters can now be
explained because you have a
distribution of parameters over the
population
so i guess my recommendation for being a
good modeler
would be let the data talk to you and
then
think about what it's saying and then
try to reconcile
your theories and hypothesis before you
started analyzing the data
try to reconcile that with what the data
is actually
telling you and a lot of times you're
getting the contradictory
well not a lot of times but some of the
times you get a contradictory result
simply because of the estimation process
you know there's something you're not
accounting for in the estimation process
so you may find a variable ah it's
statistically insignificant
but the reason why it's statistically
insignificant is you may be modeling it
as a
fixed parameter when in fact it actually
it's significant but it varies across
the population
so if you use like a random parameters
model it is significant
but if you use a fixed parameters model
it's insignificant
because you don't you're not accounting
for the distribution of effects across
the population
so it's a constant balance between your
hypothesis and theory and then what the
data actually tells you now sometimes
they perfectly
collide you know you have an hypothesis
your data supports the hypothesis
and you're done but almost always when
you start analyzing the data
um it tells but you have to be aware of
the method that you're using because the
method that you're using
uh constrains what the data can tell you
so it's a it's a constant balance
between those two
your hypothesis and what the data is
telling you
but it's you know every data set is
different
and so uh you know a lot of times you
know people say well
you know what modeling approach is
different a lot of times it just depends
on the variance in the data
you know sometimes a mixed logic model
would be better and sometimes an ordered
probability model would be better you
know you can't
it's often not a single method
you know completely dominates another
awesome thank you
any other questions from our student
audience you can raise your hand if you
have
and they can be questions on motorcycle
racing and music too
i can ask the question um
hi fred thank you very much for a very
interesting talk
i think both your your presentation was
really good and then after
after that when you shared about your
history that was really interesting too
um so we've actually been learning about
uh predictive and inference models in
our applied
data analysis class but i wasn't too
sure about the heterogeneity models that
you spoke of
would you mind expanding a little bit on
that
so yeah the the heterogeneity models
are models that use
mixing distributions so i'm not sure
if did you cover like the mixed logit
models or random parameter models in
your course maybe not but
these are um the uh
these models i guess the one paper
then i i don't have it up here i wrote a
paper
in 2016 with chandra bat did chandra
give a presentation
and uh not yet we okay okay well
in the future but there's a 2016
paper that i wrote uh called unobserved
heterogeneity in the analysis of highway
accident data
so that paper would explain what these
heterogeneity models are but really what
they are
is their random parameter models and
latent class
models and what they try to do is
they add a mixing distribution
that allows the beta that you estimate
to not be the same for everybody in the
population
so for example let's say
i have i'm estimating a uh
give a good example um i'm estimating a
model
that uh predicts what type of car
a person is going to buy um not
you know so let's go for a specific make
and model of a car so i know that
rich people um are going to be less
sensitive to the price variable
than poor people so i mean one way that
you do that with a fixed parameter
models is
you uh divide the price of the card
divided by income
and then so that reduces the x for rich
people and in in the makes the x larger
for poor people because they'll have a
smaller
income in the denominator so what these
mixing models do is say
okay look not everybody has the same
beta
for price and i'm assuming that that
beta is distributed over a population
by some distribution maybe a normal
distribution or whatever
so when you use these heterogeneity
models
they allow the individual parameters
to be unique to each observation
according to some distribution
so these heterogeneity models account
for
unobserved heterogeneity in the data you
know for some reason
um you know rich people are going to be
less sensitive
to price so they're going to have a
lower beta and poor people are going to
be more sensitive to price so they're
going to have a higher beta
so these mixing distributions and i
guess
these these really started catching on
in the safety and transportation fields
maybe in the early 2000s
when dan mcfadden and kenneth train um
you know wrote some papers on the mix
logit and kenneth train
also has a book that's available uh
online
that looks at random parameter logit
models
uh but these these heterogeneity models
if you look at the
chapters in my book from chapter 16 on
they're all heterogeneity models and it
it just
allows you a lot more flexibility
because it allows you to look at betas
instead of them being fixed across
everybody gets the same data potentially
everybody could get a different beta
or if you use a latent class or finite
mixture model
groups of people have the same beta so
um these you know these models have
become extremely popular
in academic circles not so much in
economics even though economists are the
ones that first started developing this
because because economists uh realize
that these heterogeneity models
don't completely resolve the issues of
endogeneity and self-selectivity
in fact they can confuse them because
they you get these mixing distributions
that
capture some of it but not all of it so
the economics field has sort of moved
away from these extremely
and they're extremely complex and
difficult
potentially difficult to estimate
although there are
a lot of packages that well you know
there's packages in r that estimate
random parameter models
enlogit limdep is a package that
you know i use when i teach my classes
using my book
so these heterogeneity models are just
what i would classify as random
parameter models
and latent class models are the two most
popular heterogeneity models
thank you and and i should mention if
you if you look at my journal
analytic methods and accident research
if you look at any paper
in there it has heterogeneity models
because
if you don't use a heterogeneity model i
reject your paper
because you have because you can show
mathematically that the heterogeneity
model fits better
almost always not always but almost
always
uh but yeah so that if you uh
want to get caught up on that literature
that 2016 paper
unobserved heterogeneity in acts in
highway safety data is really uh
is really good place to start because it
not it not only does the safety
literature but it talks about
the evolution of heterogeneity models
and i think it's a it's a web of science
highly cited paper so it's
it should be easy to find
uh thanks uh michael uh fred thank you
so much for the presentation very
interesting uh
one question or two questions uh one
related to your motorcycle racing world
uh
have has that world and your modeling
world collided
to figure out better ways literally how
we do some of this
uh just just interested to know if there
were any
you know practical applications that
came out of it and a
more academic question is
with increasing amount of technology
driven inputs into the vehicles now
are the current models good enough to be
able to capture some of
these kind of failures that lead to
fatalities or crashes
as compared to modeling for human error
that's been more prevalent so far
so let me answer the first question um
yeah my i think my motorcycle there was
uh my interest in motorcycling
uh one time i saw a uh
there was a rider magazine the
motorcycle
safety writer for writer magazine did a
survey
so i saw his survey because i was
reading motorcycle
magazines this is like in the early 90s
i saw a survey and said hey
i could do an analysis of this and so
we ended up writing a paper um based on
the data that i saw in this
motorcycle publication now unfortunately
he was killed in motorcycle crash
um i think in texas they you know he
he was very famous because he would
teach motorcycling
safety to like uh mike tyson and all
sorts of celebrities a
very famous guy but tragically i mean
there are cases where
you know a deer comes out of nowhere and
there's nothing you can do so there's a
you know there's some risk involved
and i've also uh
estimate you know i've done a lot a
number of motorcycle
safety uh you know papers
personally i've experienced multiple
crashes that one's been on the highway
and then um the other two or three
crashes and mostly on racetracks where
um
but unfortunately in you know there's a
saying when you race motorcycles
if you're um if you're not crashing
you're not going fast enough
so um i was going fast enough a couple
times
so uh yeah so i think at some level
uh those two uh you know do collide
um so you know they they have helped
motivate
now your second question with regard to
the technology
actually if i did a paper in
2006 where we argued that
safety features would actually
be offset by more risky driving behavior
and actually i think if you go to my
website i
actually did a there's an interview i
did for good morning america like in
2006 or something
where um our paper came out and
we argued that there's offsetting
behavior
that in other words you can think of it
like this i like to drive and i have a
i like to drive with a 1 in 10 000
chance of dying
so if you give me a safety feature like
you know anti-lock brakes or side impact
airbags whatever
that increases or decreases my chance of
dying to one in a hundred thousand then
i would just drive faster
so i don't take the benefit in that
there's still benefit to society because
i'm driving faster getting to my
destination quicker
but i don't take it out as safety and
ends up as being this offsetting
behavior
so i think when we evaluate the
effectiveness
of vehicle attributes we have to
consider the potential offsetting
behavior
another interesting point and this is
something very important to consider
is there because this fits into what we
are talking about with self selectivity
when you look at a new safety feature
and how effective it is a lot of times
when
the national highway traffic safety
administration does this
they look at people that are in crashes
with side impact airbags for example
and people in crashes with outside
impact airbags
and they say well side impact airbags
are you know eighty percent effective in
reducing fatality
but that's something you do not want to
do
because that's something that's 100
wrong because
the people that buy cars with side
impact airbags are probably the safest
drivers
okay and if you look at the effect of
the estimated effectiveness of side
impact airbags
over time there i think their initial
estimates were like 70 percent
then it became 50 40 until you have the
entire sample of the population because
the riskiest drivers
are not going to have the site impact
aircraft airbag cars
for quite some time you know it takes a
while for it and so
that self selectivity that we were
talking about
affects results because if you think of
what nissa reports they just look at the
cars with side impact airbags the cars
without
but the drivers in those cars are much
different if you put the drivers and the
cars with side impact airbags
in the non-side impact airbag cars they
would probably have a higher
um you know a lower probability of a
crash because the riskier drivers don't
have the side impact cars
at some point everybody will have them
and then you'll be able to assess the
full effectiveness
but that's the same sort of
self-selectivity identification problem
they were talking about
earlier with the causality now what's
also going to be interesting
and is when you're driving a car with
advanced safety features
or anti-lock brakes that allow you to
stop faster or
the tesla that goes zero to 60 in one
point one second they accelerate quicker
if your behavior does not only affect
your accident risk
everybody else around you starts
adjusting toward
to you so that's you know another paper
that i
wrote uh is this temporal instability
of highway crack of pirate crash data
and the argument there
is that technology is changing so
quickly
in on in the automotive industry that
the betas that we estimate
in you know one year are going to be
different than in another year because
everything is changing
everybody's behavior is adapted and
you're not only responding to the safety
features in your car
and the safety features that others have
in cars around you
but then there's always social changes
right your cell phone distraction
everything is changing and there have
been a lot of papers that come out
recently in my journal that
show you know statistical differences
from year to year in uh the frequency of
crashes and the severity of crashes
so this is what you know we call
temporal instability that you can test
for it
and that's another big future direction
is we want to find out how people's
betas change over time
and that's going to be something you
know the next generation of people you
and others
are going to have to deal with you know
you have this machine learning and stuff
coming
but you have the great opportunities you
have the betas changing over time
because of the new technologies and
you're going to have this
era where you have the automated
autonomous vehicles mixed with human
vehicles that's going to be fascinating
you're good there's going to be like
years of research on that
it's going to be great thank you
all right i think uh i can open uh to
the uh um
outside audience i've noticed there are
a few questions
in q a um so please ask your question
please
you know as concise as you can so we can
go over
all these questions if possible so linda
can you ask your question
um okay uh so i
probably linda is uh longer with um he's
probably signed up and signed uh sanjo
are you still there
well i think
they probably have already signed up and
so um so fred i
um you know it's
not a very a question it's it's a
problem that we're
actually encountering it's you know
keltons
they're interested in um
[Music]
the uh like hot spots and also
you know if they did something whether
it improved the city or not
they want to evaluate project based
sort of safety impact and the
the thing that you know for any
particular location
um you know these crashes are very small
probability events
and oftentimes you don't have enough
observations
um and and there are multiple factors
contributing to
the safety so
from your vantage point like if you're
going to
evaluate you know before and after
uh what kind of method would be more
suitable like the heterogeneity
or the traditional methods
so yeah this is um this is a a very good
point
and it sort of uh gets back to
what i was talking about with this
temporal instability
let me give you an example for um
if we look at crashes during the great
recession you know there's this general
observation
when the economic conditions of the
country
get bad we end up with fewer
crashes and fatalities per mile driven
yeah so there's a fundamental
and then when the economy is good you
end up getting more vehicle crashes
uh per mile driven now there's a lot of
theories and potential explanations for
that about
how people's risk perception evolved you
know there's neuroscience a lot of
um my 2018 people my 2018 paper gets
into a lot of details about this
now the problem is you have these sort
of
global trends that are based on
economics
or over time we know that
there's a general downward trend in the
number of fatalities per mile driven
so the problem when you do like before
and after analysis
is you have to see where you are in this
economic trend
so for example when we were in the great
recession in the late 2000s all
if you did a before and after analysis
almost anything you did
was found to be very effective because
the number of crashes per mile driven
were naturally going down globally okay
over time so everything is affected
now we we i noticed that problem when we
got to
when i got to the university of south
florida here we started
you know they were doing people in the
center for urban transportation research
here
we're doing studies where the economy
was starting to pick up so this is like
2016 2017 and you're getting more
uh you know crashes per mile driven so
now you're doing these safety features
but you're finding almost every safety
feature you use if you do before and
after analysis
it's gotten worse because globally
the risk of drivers is getting worse so
when you do before and after analysis
you have to take into account
this sort of uh you know this underlying
temporal element that could sometimes
give you
contradictory findings and a lot of
traditional before and after analysis
like uh
using empirical bays and stuff runs into
this
temporal instability problem uh straight
on
now i have not done a lot of hot spot
you know related you know high accident
location related research
but i if i was giving a multi-million
dollar contract
and we can talk to talk about that later
um i think you could do a really
interesting
thing using heterogeneity models to
account for those temporal effects
i'm currently doing a national
cooperative highway research project
that's looking at speed limits being
raised in kansas
and we developed this really interesting
uh heterogeneity model
uh that studies the effect of this and
we have a lot of
temporal shifting uh going on so it's
it's not an easy problem because of
these
global trends so you know even if you
if you knew that the betas were constant
that the overall
crash rate per mile driven was the same
um you would still potentially have a
before and after problem
but when you have this global changing
in the
um you know rate crash rates due to
economic conditions or other factors it
becomes a more complicated problem
but it's not easy to solve but it can be
done
uh with i think heterogeneity models
would be the way to go
if you're talking about a specific uh
project idea you have in mind we can
talk about it
yeah it's a 150 150 k dollar projects
definitely we a tenth of the
the budget that you mentioned uh okay
we're yeah so we're looking at
uh actually more um you know
like if caltrans have done something and
what what's the effectiveness of on the
safety improvement
yeah it's it it's still you know going
to be sort of a
difficult problem to address but it's
not impossible
and i think you can make um you know you
can make some
you can definitely make some inferences
um you know perhaps we can
talk at some point about that and i can
give you i think you'll again uh
talk offline and on that i think we'll
probably can
schedule a meeting to meet you with you
with my guys
but that uh that that general phenomenon
of uh you know fewer crashes per mile
during recession then this goes
it's not just the great recession this
goes back to recessions
in like 1980 1960 anytime there's a
recession
you get fewer crashes per mile driven
and again if you look at my 2018
temporal instability paper there's a lot
of neuroscience
reasons and stuff for that um but it it
it makes it
it's it makes the problem intellectually
interesting
but empirically it makes it difficult
yeah it's wondering you know
during the recession a lot of for
example you know construction work or
delivery
so a lot of the people who probably
drive fatigue and yeah maybe there's a
higher proportion of those
but when in a recession you have a
bigger reduction of those kind of travel
um well that's just you know some yeah
and that's
that's one of the arguments is that the
people driving
now this is very this brings up a very
interesting point and i'm actually
doing some work on this now is what
happened during covet
because you know in this case during
covet
the types of drivers that are on the
road are a lot different
during the covet lockdowns and so
in this case we have in getting back to
what i was talking about earlier
here you have the self-selectivity
problem who's getting involved in
crashes
so who's getting involved in crashes
before covet
is going to be a lot different than
who's getting involved in crashes
during the covet lockdowns because the
types of people that are going to be on
the road are going to be a lot different
because you know the people that can
work remotely are going to be working
remotely
so this is and you know we've collected
data from florida on this um but again
this is a research project that i have
in the upcoming year
but it's really going to be you know
target that sort of self-selectivity
issue
because the types of people being on the
road are much different during the
coveted lockdown
and it's sort of the idea that you're
getting to is that during a recession
you have different people using
different proportions of the vmt
proportions of the mile travel but
during covet is going to be a great
example of that because we're going to
have like three or four months
where the vmt being driven by certain
people is going to be a lot
different than it was pre-coveted yeah
and this is a fascinating topic that we
can go on on
but our time the three has passed
and so we'll stop here and thank you
fred again for
you know a really exciting talk and
we'll stop here and i wish everybody
have a nice weekend