9 Asking a New Question, Getting New Answers: Evaluating Results
The Inevitability of Evaluating
Evaluating social policies is essential, although social scientists often face skepticism regarding their interpretations. People inevitably make claims about the effects of social programs, which should be based on specific measurements of what has been achieved. The key question is whether a policy will produce its intended result. To answer this, methods similar to those used by social scientists are often employed.
Policy analysis largely relies on quantitative techniques, which can be complex and debated, but a crucial aspect often overlooked is the "dependent variable problem." In evaluation statistics, the dependent variable is the effect, while the independent variable is the cause. For social policies, the program acts as the independent variable, and the intended outcome is the dependent variable. The success of the evaluation depends on how well the dependent variable is measured.
The way the "real" dependent variable is represented—through what are known as operational measures—is critical to the evaluation's results. For instance, grades (GPA) are used as an operational measure of learning, even though they may not perfectly reflect actual learning progress. If the measure does not correspond accurately to the outcome we care about, it can lead to misleading conclusions about the effectiveness of a program.
Additionally, choosing a unit of aggregation (the group used for evaluation) can significantly impact the understanding of results. For example, when assessing a program like Job Corps, the evaluation might focus on overall employment outcomes or individual trainees' experiences, leading to different implications for policy effectiveness.
The dependent variable problem highlights how social policies often miss measuring some outcomes while overvaluing others, resulting in inaccurate assessments of progress. This flaw can mislead policymakers and analysts, making it difficult to identify actual successes or failures in social initiatives.
The Pursuit of Happiness and the 55-MPH Speed Limit
The debate surrounding the 55-mph speed limit, established in 1974, provides an interesting insight into how policies can impact happiness and safety in society. Originally introduced as a temporary measure to save fuel during the Arab oil embargo, the law was later thought to also significantly reduce fatalities in car accidents. However, while many people professed support for the law, a significant number did not follow it, leading to a complex situation where public opinion and actual behavior did not align.
When the law was first enacted, it was aimed at addressing a national crisis by conserving fuel. Congress believed that by lowering the speed limit, fuel consumption would decrease, thus alleviating the fuel shortage. However, the anticipated savings from this limit turned out to be considerably less than predicted. As time passed, the initial gas crisis dissipated, but it was reported that this law had positive side effects, such as saving lives. As a result, in 1974, Congress made the speed limit permanent, believing it to be beneficial for public safety. Yet, this created a kind of national confusion, as many people opposed the law while publicly showing support for it.
A Gallup survey conducted in 1981 perfectly illustrates this dissonance. While 75 percent of respondents expressed their support for the 55-mph speed limit, only 29 percent admitted to following it consistently. In fact, 42 percent of those who favored the law acknowledged that they complied “not very often or never.” This contradiction arose because, if individuals openly opposed the speed limit, it implied they were in favor of allowing more deaths in car accidents, creating a social pressure to conform to the law regardless of personal behavior.
In 1987, after years of public frustration, Congress enacted a bill allowing states to raise the speed limit to 65 mph on rural interstate highways. This shift in legislation did not ultimately result from a rejection of the law's claimed life-saving benefits but came about largely because many Americans disliked the rule, even while they claimed to support it. Opponents of the speed limit did not necessarily prove that it did not save lives; instead, they voiced frustration over the law's restrictions. The situation exemplifies the problem of defining what constitutes “the good” in public policy discussions.
If we were to examine the 55-mph speed limit purely from a data-driven perspective, without political bias, we could systematically analyze whether this law is beneficial or harmful. We would need to look at several dependent variables:
1. Human deaths and injuries (expected to decline).
2. Fuel costs (expected to decline).
3. Other economic costs associated with travel (expected to increase).
4. Noneconomic and non-health-related costs associated with travel (expected to increase).
The first three dependent variables are straightforward to quantify, although precise calculations can be tricky due to the complexities of causal attribution, especially regarding lives saved. The collective data indicates that since the 55-mph limit was enacted, approximately 7,466 lives have been saved annually, leading to a grand total of almost 90,000 lives saved by the end of 1986. This finding is significant and may suggest that the speed limit is justified based on lives saved.
However, this analysis does not address the fourth dependent variable, which considers the noneconomic costs to individuals. Depending on location, the impact of the speed limit could vary greatly. For example, individuals traveling from one state to another may find that longer trips at 55 mph can be frustrating and unproductive. Also, other feelings associated with long drives—boredom or the enjoyment of driving fast—become less important when viewed solely from an aggregate data standpoint. Economists know how to assign dollar values to these opportunity costs, but doing so can overlook the personal significance those costs hold for individual drivers.
Imagine a driver limited to 55 mph who wishes to reach a destination in time. He may want to drive faster to enjoy his time or to have a full day planned; thus, the lack of freedom to choose could impact his overall happiness. The difference here is that the driver looks at the law as a limitation on personal freedom rather than as a protective measure.
After discussing the effectiveness of the speed limit, we should question why the limit hasn’t been lower, like to 50 mph, if it could potentially save even more lives. The slippery slope of evaluating potential speed limits reveals that while lower limits might save some lives, there would be an increasing backlash from the public and a perceived loss of personal freedom.
Reasoning against continually lowering the speed limits often relies on emotionally charged arguments. People may resist limits they see as extreme, despite rational data showing that further reductions would save more lives. This indicates that public perception of costs, both in terms of time spent and the experience of driving, becomes more valuable than the calculated safety benefits.
With this understanding, it becomes crucial to evaluate the law through the lens of personal happiness rather than just national data. When looking at factors affecting an individual driver, the analysis should combine safety, economic costs, and the noneconomic aspects of driving.
For instance, if we consider a 250-mile trip, a driver might spend more in gas if he chooses to drive faster. Assuming gasoline costs and mileage rates, this drives down overall happiness when drivers are forced to comply with the 55-mph limit. Additionally, even when examining fatalities associated with the speed limit, the odds of being involved in a serious accident remain extremely low at either speed.
In conclusion, while the 55-mph speed limit has saved many lives, its negative impact on personal freedom and happiness should also be considered. Individuals often have to weigh their need for safety against their desire for choice on the road. By shifting the focus from aggregate benefits to individual experiences, discussions about traffic laws and regulations can better account for the values and preferences of all drivers. The overall lesson is that evaluating public policies requires understanding how these laws affect individual happiness and freedoms while still aiming for safety outcomes. Whether or not to maintain such laws ultimately reflects broader societal values regarding individual rights versus collective safety.
Recasting the Criteria of Success
The way we usually evaluate social programs, like saving lives or providing jobs, can be too basic and doesn’t help us create good policies. These goals, while important, don’t give enough information to understand if a program is truly successful. Analysts have become lazy by focusing on these simple measures instead of the deeper goals behind them. This laziness leads to evaluations that miss the real impact of programs on people's lives.
We should evaluate programs based on how they affect individuals rather than looking only at overall results. For instance, when thinking about a program like the Job Corps, it’s important to ask what a young person facing challenges should do next and whether this program will actually help them find a job. Just looking at unemployment rates after the program ends doesn’t show how effective it is for each person.
To improve job training programs, we need to ask why working hard in these programs often doesn’t lead to good job opportunities. This helps us understand if the programs are set up correctly and if they serve individual needs. Evaluations should go deeper to check how well programs help individuals rather than just sharing average results.
Focusing on individuals and their happiness helps create better discussions about policies and leads to better results. It encourages decision-makers to think about what really helps people instead of relying on numbers that may support programs that don’t work well.
The Inevitability of Evaluating
Evaluating social policies is essential, although social scientists often face skepticism regarding their interpretations. People inevitably make claims about the effects of social programs, which should be based on specific measurements of what has been achieved. The key question is whether a policy will produce its intended result. To answer this, methods similar to those used by social scientists are often employed.
Policy analysis largely relies on quantitative techniques, which can be complex and debated, but a crucial aspect often overlooked is the "dependent variable problem." In evaluation statistics, the dependent variable is the effect, while the independent variable is the cause. For social policies, the program acts as the independent variable, and the intended outcome is the dependent variable. The success of the evaluation depends on how well the dependent variable is measured.
The way the "real" dependent variable is represented—through what are known as operational measures—is critical to the evaluation's results. For instance, grades (GPA) are used as an operational measure of learning, even though they may not perfectly reflect actual learning progress. If the measure does not correspond accurately to the outcome we care about, it can lead to misleading conclusions about the effectiveness of a program.
Additionally, choosing a unit of aggregation (the group used for evaluation) can significantly impact the understanding of results. For example, when assessing a program like Job Corps, the evaluation might focus on overall employment outcomes or individual trainees' experiences, leading to different implications for policy effectiveness.
The dependent variable problem highlights how social policies often miss measuring some outcomes while overvaluing others, resulting in inaccurate assessments of progress. This flaw can mislead policymakers and analysts, making it difficult to identify actual successes or failures in social initiatives.
The Pursuit of Happiness and the 55-MPH Speed Limit
The debate surrounding the 55-mph speed limit, established in 1974, provides an interesting insight into how policies can impact happiness and safety in society. Originally introduced as a temporary measure to save fuel during the Arab oil embargo, the law was later thought to also significantly reduce fatalities in car accidents. However, while many people professed support for the law, a significant number did not follow it, leading to a complex situation where public opinion and actual behavior did not align.
When the law was first enacted, it was aimed at addressing a national crisis by conserving fuel. Congress believed that by lowering the speed limit, fuel consumption would decrease, thus alleviating the fuel shortage. However, the anticipated savings from this limit turned out to be considerably less than predicted. As time passed, the initial gas crisis dissipated, but it was reported that this law had positive side effects, such as saving lives. As a result, in 1974, Congress made the speed limit permanent, believing it to be beneficial for public safety. Yet, this created a kind of national confusion, as many people opposed the law while publicly showing support for it.
A Gallup survey conducted in 1981 perfectly illustrates this dissonance. While 75 percent of respondents expressed their support for the 55-mph speed limit, only 29 percent admitted to following it consistently. In fact, 42 percent of those who favored the law acknowledged that they complied “not very often or never.” This contradiction arose because, if individuals openly opposed the speed limit, it implied they were in favor of allowing more deaths in car accidents, creating a social pressure to conform to the law regardless of personal behavior.
In 1987, after years of public frustration, Congress enacted a bill allowing states to raise the speed limit to 65 mph on rural interstate highways. This shift in legislation did not ultimately result from a rejection of the law's claimed life-saving benefits but came about largely because many Americans disliked the rule, even while they claimed to support it. Opponents of the speed limit did not necessarily prove that it did not save lives; instead, they voiced frustration over the law's restrictions. The situation exemplifies the problem of defining what constitutes “the good” in public policy discussions.
If we were to examine the 55-mph speed limit purely from a data-driven perspective, without political bias, we could systematically analyze whether this law is beneficial or harmful. We would need to look at several dependent variables:
1. Human deaths and injuries (expected to decline).
2. Fuel costs (expected to decline).
3. Other economic costs associated with travel (expected to increase).
4. Noneconomic and non-health-related costs associated with travel (expected to increase).
The first three dependent variables are straightforward to quantify, although precise calculations can be tricky due to the complexities of causal attribution, especially regarding lives saved. The collective data indicates that since the 55-mph limit was enacted, approximately 7,466 lives have been saved annually, leading to a grand total of almost 90,000 lives saved by the end of 1986. This finding is significant and may suggest that the speed limit is justified based on lives saved.
However, this analysis does not address the fourth dependent variable, which considers the noneconomic costs to individuals. Depending on location, the impact of the speed limit could vary greatly. For example, individuals traveling from one state to another may find that longer trips at 55 mph can be frustrating and unproductive. Also, other feelings associated with long drives—boredom or the enjoyment of driving fast—become less important when viewed solely from an aggregate data standpoint. Economists know how to assign dollar values to these opportunity costs, but doing so can overlook the personal significance those costs hold for individual drivers.
Imagine a driver limited to 55 mph who wishes to reach a destination in time. He may want to drive faster to enjoy his time or to have a full day planned; thus, the lack of freedom to choose could impact his overall happiness. The difference here is that the driver looks at the law as a limitation on personal freedom rather than as a protective measure.
After discussing the effectiveness of the speed limit, we should question why the limit hasn’t been lower, like to 50 mph, if it could potentially save even more lives. The slippery slope of evaluating potential speed limits reveals that while lower limits might save some lives, there would be an increasing backlash from the public and a perceived loss of personal freedom.
Reasoning against continually lowering the speed limits often relies on emotionally charged arguments. People may resist limits they see as extreme, despite rational data showing that further reductions would save more lives. This indicates that public perception of costs, both in terms of time spent and the experience of driving, becomes more valuable than the calculated safety benefits.
With this understanding, it becomes crucial to evaluate the law through the lens of personal happiness rather than just national data. When looking at factors affecting an individual driver, the analysis should combine safety, economic costs, and the noneconomic aspects of driving.
For instance, if we consider a 250-mile trip, a driver might spend more in gas if he chooses to drive faster. Assuming gasoline costs and mileage rates, this drives down overall happiness when drivers are forced to comply with the 55-mph limit. Additionally, even when examining fatalities associated with the speed limit, the odds of being involved in a serious accident remain extremely low at either speed.
In conclusion, while the 55-mph speed limit has saved many lives, its negative impact on personal freedom and happiness should also be considered. Individuals often have to weigh their need for safety against their desire for choice on the road. By shifting the focus from aggregate benefits to individual experiences, discussions about traffic laws and regulations can better account for the values and preferences of all drivers. The overall lesson is that evaluating public policies requires understanding how these laws affect individual happiness and freedoms while still aiming for safety outcomes. Whether or not to maintain such laws ultimately reflects broader societal values regarding individual rights versus collective safety.
Recasting the Criteria of Success
The way we usually evaluate social programs, like saving lives or providing jobs, can be too basic and doesn’t help us create good policies. These goals, while important, don’t give enough information to understand if a program is truly successful. Analysts have become lazy by focusing on these simple measures instead of the deeper goals behind them. This laziness leads to evaluations that miss the real impact of programs on people's lives.
We should evaluate programs based on how they affect individuals rather than looking only at overall results. For instance, when thinking about a program like the Job Corps, it’s important to ask what a young person facing challenges should do next and whether this program will actually help them find a job. Just looking at unemployment rates after the program ends doesn’t show how effective it is for each person.
To improve job training programs, we need to ask why working hard in these programs often doesn’t lead to good job opportunities. This helps us understand if the programs are set up correctly and if they serve individual needs. Evaluations should go deeper to check how well programs help individuals rather than just sharing average results.
Focusing on individuals and their happiness helps create better discussions about policies and leads to better results. It encourages decision-makers to think about what really helps people instead of relying on numbers that may support programs that don’t work well.