```{r load, echo=FALSE}
d <- read.csv("C:/Users/Jonny/Google Drive/Academic/USP/Class/Week 1 - Intro/Homework/atlas2013_dadosbrutos_pt.csv")

```

### 1. What is the IDHM of Sao Paulo in 1991? What is the percentage change in Sao Paulo's IDHM between 1991 and 2010?
The IDHM of Sao Paulo in 1991 is $`r d[d[,"ANO"]==1991 & d[,"UFN"]=="São Paulo","IDHM"]`$. 

The percentage increase in the IDHM from 1991 to 2010 is $`r round(((d[d[,"ANO"]==2010 & d[,"UFN"]=="São Paulo","IDHM"] - d[d[,"ANO"]==1991 & d[,"UFN"]=="São Paulo","IDHM"])/d[d[,"ANO"]==1991 & d[,"UF"]==35,"IDHM"])*100,1)`$%.

### 2. Produce a clear table of the IDHM by state and by year.

```{r changes, echo=FALSE,message=FALSE}
library(tidyverse)
library(knitr)
d_wide <- d %>% select(UFN,ANO,IDHM) %>% spread(ANO,IDHM) %>% mutate(change=`2000`-`1991`)
kable(d_wide)

#library(reshape2)
#d_wide <- dcast(d,UFN~ANO,value.var="IDHM")
#d_wide[,"Change"] <- d_wide[,"2010"] - d_wide[,"1991"]

```

### 3. Which state has made the greatest absolute gains in the IDHM between 1991 and 2010?
The state with the largest improvement in the IDHM between 1991 and 2010 is `r d_wide %>% top_n(1,change) %>% select(UFN)`. 

### 4. Produce a chart showing how the IDHM has changed over time in four states: Sao Paulo, Rio Grande do Norte, Roraima and Mato Grosso.

```{r chart, echo=FALSE,message=FALSE,warning=FALSE}
d_select <- d %>% filter(UFN %in% c("São Paulo","Rio Grande do Norte","Roraima","Mato Grosso")) %>% select(UFN,ANO,IDHM)

#d_wide_select <- d_wide[d_wide[,"UFN"] %in% c("São Paulo","Rio Grande do Norte","Roraima","Mato Grosso"),c("UFN","1991","2000","2010")]
#d_wide_select_long <- melt(d_wide_select)

ggplot(data=d_select) + geom_line(aes(x=ANO,y=IDHM,group=UFN,color=UFN)) + theme_classic() + xlab("Year") + ylab("IDHM")
```

### 5. What is the correlation between state population (variable POP) and IDHM in 2010?

The correlation between the variables IDHM and Population are: $`r round(cor(d[d[,"ANO"]==2010,"IDHM"],d[d[,"ANO"]==2010,"POP"]),3) `$

### 6. Do you think that a higher state population causes a higher state IDHM? Why?

From simple correlations we must be extremely cautious in drawing any causal conclusions. It may be that there is reverse causation: more populated places are more developed because more educated, healthier people choose to move to (or stay in) more populated urban areas. Or there may be an ommited variable that causes both population and development. However, it is possible there is a real causal connection if population - or at least population \textit{density} - allows people to create more wealth and human capital (eg. Ed Glaeser's arguments).

### 7. What is the correlation between state education (for example variable I_ESCOLARIDADE) and IDHM in 2010?

The correlation between IDHM and Schooling is: $`r round(cor(d[d[,"ANO"]==2010,"IDHM"],d[d[,"ANO"]==2010,"I_ESCOLARIDADE"]),3)`$

### 8. Do you think that a higher state education causes a higher IDHM? Why?

We can be more definitive in saying higher state education causes a higher IDHM because the variable I_ESCOLARIDADE is used in calculating the IDHM - so increases in schooling \textit{automatically} produce increases in the IDHM.

### 9. What is the correlation between state education (variable I_ESCOLARIDADE) and state population (variable POP) in 2010?

The correlation between Population and Schooling is: $`r round(cor(d[d[,"ANO"]==2010,"POP"],d[d[,"ANO"]==2010,"I_ESCOLARIDADE"]),3)`$

### 10. How does your answer to (9.) affect your answer to (6.)?

Knowing that municipal population is correlated with schooling suggests a number of possibilities. This could simply be because schooling is a mechanism connecting population and the IDHM -  it is 'on the causal path' -  higher population increases education and therefore the IDHM. But it may also be because schooling is an omitted (confounding) variable, causing both population and IDHM to raise. For example, perhaps places with high education or good schools attract more people so this is one causal effect, and separeately we already know schooling is mechanically linked to a higher IDHM, a second and separate causal effect. The relationship between population and IDHM may not be a causal relationship, but a spurious relationship reflecting the effect of education on both population and IDHM separately.

### 11. Can you think of any methodology (field experiment, natural experiment, observational study etc.) for testing whether an increase in population causes a higher municipal IDHM?

To test whether an increase in population causes an increase in IDHM, ideally we might want to run an experiment and increase the population of some states and not others, choosing at random. However, that would be infeasible and unethical. We might be able to find a natural experiment where some unexpected shock created an increase in population in some states but not others - perhaps a disruption to migration flows in Europe which affected migration to, by chance, Rio but not Sao Paulo. But that might not be available, or might be impossible to measure. And is migration the same as a population increase? More likely, we would need to do an observational study to compare trends over time as population changed in comparable states. For example, a difference-in-differences methodology that compared changes in population to changes in the IDHM across all states. We would still need to check carefully the relevant assumptions, of course.