Indices exist in the social sciences for a variety of subjects—some attempt to compare freedom on a global level by comparing policies in every country while others compare freedom domestically by evaluating policies across the 50 states.
Some indices compare cities across the country on superficial or a narrow set of metrics, such as the "Best Places to Live" ranking, but none focus on cities in a particular state comparing policies from one locality to the other.
Liberty at the local level is not easy to measure. Turning qualitative differences into a quantitative metric is challenging and is only one way to provide a snapshot of freedom in any given city. Cities are inherently different and have different populations with different cultural makeup. Local economies are different, development history is different—even the different geography plays a role. Each of these differences can have an impact on the policies that exist in a given city. However, it is important to try and evaluate these differences to create a baseline that cities can be measured against. Our index attempts to do that by looking at a sample cross section of local policies and programs and measuring each in a quantitative way. We do this through a comparative composite index.
The index is a composite index of 22 metrics across three categories. Each individual metric was normalized using standard deviation and statistical z-scores. The z-scores in each metric were then weighted and combined for each category. The category sub-scores were then combined equally and each category makes up one-third of the total index score respectively. Thus, each category is made up of a composite of data points that were individually scored, normalized, and compounded using a weighting system. Overall scores are weighted composites of these component composites.
We chose this methodology as it has been used in a number of other similar composite indices like that of the Mercatus Center's index of "Freedom in the 50 States." We also followed the industry standard for composite indicators as outlined in the Handbook on Constructing Composite Indicators: Methodology and User Guide.
Normalization is important in constructing a composite index where the component metrics in the data have different measurement units. We used statistical z-scores to normalize our data. A z-score measures how many standard deviations from the mean a particular value is. It represents how different a particular city is from the average of all cities for a particular topic. A z-score of 0 means the observed value is the same as the average, a positive z-score means a value is above the average, and negative values are below the average.
[z-score = (individual city value - mean of all values) / standard deviation of values]
We purposely did not use robust z-scores (based on the median instead of the mean) because the composite index is inherently comparative and outliers are important in a comparative index. Because the data is not sampling data but actual comprehensive data of the population studied, the impact of outliers is actually desirable on the index. This is because many localities may tend to adopt common regulatory approaches such that there could be less variance. Thus, when a locality deviates significantly from the norm, this should be reflected in the index. Moreover, the effect of outliers is muted by the differential weights that are applied to each component of the index.
Metrics are groupings of data points that measure policies governing a particular topic in each city. Data points used were either numerical continuous values, as in the example of city budget expenses and tax rates, or the data points were transformed into ordinal, interval, or ratio-based scores. None of the data points were nominal. In the case of ordinal scores that evaluate a policy on a numerical scale, the higher number is associated with the more favorable policy approach.
Data points are observed values or observed conditions that reflect individual properties for a single city. For example, whether or not a city has a policy regulating the discharge of firearms is a single data point. In some cases, multiple data points were combined to create a metric score. For example, the taxes and cost of government metric includes data points like population figures, tax revenue across multiple years, city budget figures, and more. Data points that are relevant to each other were combined arithmetically where appropriate—for example, census income per capita was multiplied by population to arrive at a total income estimate.
When multiple data points were used for the same metric but could not appropriately be combined arithmetically, they were normalized first and combined using weighting—for example, whether or not a city permits chickens in any residential zones (a binary data point) was combined with the data point for the actual limit on the number of chickens a city allows. In this example, each data point was normalized using a z-score first and multiple z-scores were weighted and combined to create the final metric score.
All data analysis for the 2015 index is based on information collected from cities as of September 2015.
Weighting was used to emphasize more important policy areas over minor ones. For example, the impact of taxes on a property owner is more significant than the cost of building permit fees and was weighted accordingly. Weighting was also used to adjust for areas where data is less available, less useful, or less reliable.
For a more in-depth view of the information used to compile the Index, you can access the original data.