Or to plot the year over year change in average GPA, with some missing values interpolated:

I’ve never tried to scrape a website with R before, but it turns out for this it was pretty easy (with some help).

Using the XML package and readLines() from the base package you can read the html file which has the grade inflation data. The result of this is not really useful yet since it contains all of the original html and xml tags, but with another function, readHTMLTable one can pull out just the table itself. Since R will by default convert character vectors to factors when creating a data frame, there are a few extra lines in the code below to create a data frame with two numeric vectors for year and GPA:

library(XML)
library(plyr)

# Get and format html file
duke.html <- readLines("http://www.gradeinflation.com/Duke.html")
duke.doc <- htmlParse(duke.html)

# Get table as data frame
duke <- readHTMLTable(duke.doc, header=F, as.data.frame=F)
duke <- data.frame(duke, stringsAsFactors=F)
colnames(duke) <- c("year", "gpa")

# Format columns
duke$year <- as.numeric(duke$year)
duke$gpa <- as.numeric(ifelse(duke$gpa=="n.d.", NA, duke$gpa))

So now we have a data frame with correct types for year and GPA:

 > head(duke)
  year  gpa
1 1932 2.25
2 1933 2.28
3 1934 2.27
4 1935 2.23
5 1936 2.21
6 1937 2.26

But if you look at the table on the original website, you might notice something funny with the years listed…there are gaps, or missing years, like the skip between ’47 and ’56. After a few lines to add the missing years (link to the full code is below), we can plot the average undergrad GPA for Duke from 1932 on:

plot(duke$year, duke$gpa)

With a little bit of extra code we can make the plot look a bit nicer: add grid lines, mark every decade instead of every 20 years on the x-axis, axis labels, and set the y-axis limits to round numbers:

par(cex=1.2)
plot(duke$year, duke$gpa, ylim=c(2, 4), type="p", pch=20, xaxt="n", xlab="year",
     ylab="GPA")
x_ticks <- seq(round_any(min(years_covered), 10), 
               round_any(max(years_covered), 10), 10)
axis(1, at=x_ticks)
grid(col="gray50", lty=3)

Coding aside, two things stand out from this plot. First, there are significant gaps in the data, e.g. in the 50′s and 60′s. Second, something crazy happened with grade inflation in the 1960′s. We can do something about the missing values by using splines to interpolate them:

duke$gpa_inter <- spline(duke$gpa, n=length(duke$year))$y
lines(duke$year, duke$gpa_inter, col="gray80")

Finally, since in the context of grade inflation we might be less interested in the absolute values for the average GPA, it might make more sense to plot the changes in GPA from year to year. It looks like there was a period of grad inflation in the late ’40s (WW2?) and another, bigger period of grade inflation in the 1960s. This actually seems to have happened nation-wide in the 1960s, if you look at the writeup at http://www.gradeinflation.com/.

plot(duke$year[2:dim(duke)[1]], diff(duke$gpa_inter), col="gray50", pch=20,
     xlab="year", ylab="Change from prev. year", type="h", lwd=2, xaxt="n")
axis(1, at=x_ticks)
grid(col="gray50", lty=3)
abline(h=0, col="red")

Here is a gist of the complete code.

A nice thing about duration analysis though is that a lot of the key concepts are already explicitly graphical, like survival curves (wikipedia) and hazard rates. Below, for example, is a survival curve for cancer patients diagnosed with acute lymphoblastic leukemia between 1988 and 2008 in the US, from SEER fast stats:

Starting from the moment at which a patient is diagnosed with cancer (year 0), it shows the proportion of patients who survive without relapse or death any given number of years from diagnosis. Two years from diagnosis, 50 percents of patients are still event free. Five years from diagnosis about 35 percent are event free (what you might call cured), etc. Alternatively, one could interpret it as the probability that a given ALL patient will be alive 2 or 5 years from diagnosis, with probabilities of 0.5 and 0.35 respectively.

So far so good, except that in practice one has to estimate survival functions on the basis of limited empirical data, e.g. using a Kaplan-Meier estimate (wikipedia). The resulting estimated survivor curves are not smooth like the curve above, but ragged. Using another example of lightbulbs, with data for the number of days until five bulbs burned out, one might get something like this:

The black line shows the survivor curve estimate, based on data for the five lightbulbs. I added the grey bars to represent the number of days each of the five bulbs lasted to show how the survivor estimate is built up from individual failures. E.g. the top bar is the first bulb, which lasted for ~80 days. Thus on day 25 all 5 bulbs are still burning and our survivor curve is at 1.0. Around day 80, after the first bulb has failed, 4 of 5 or 0.8 of the original bulbs are still “alive”, bringing the curve down to 0.8. And so on for each additional failure.

Taking a cue from Bayesian Biologist’s video of Bayesian updating, and Drew Conway’s video of Chicago crime , I thought it would be nice to create a video that shows how an empirical survivor curve like this is built up from observed failure data, and how it changes as you add more data.

First, a simple backstory: Imagine you move into a new apartment, and having an obsession with measurement, you start keeping track of how long the lightbulbs in your five light fixtures last (it’s a small apartment). In the video below, the top half shows the five fixtures, where each bar represents a lightbulb and the number of days it is in operation now. On day 0 all 5 fixtures have new bulbs, i.e. there are no failures yet. Thus the survivor curve estimate, shown at the bottom, is not very useful (we disregard bulbs that haven’t burned out yet, i.e. no censoring).

Time goes by and after 80 days bulbs are starting to burn out. With data on failures we can now start updating our survivor curve to reflect those failures. The first updates still leave us with a very rough survivor curve estimate, but as more bulbs fail the curve starts getting a nicer shape. Note also that the mean time to failure (MTTF) in the bottom left corner starts getting closer to it’s theoretical value. The video ends after a year’s worth of simulation, but the longer we let it run the smoother the KM estimate would get. Eventually the KM estimate should converge to the theoretical survivor curve shown in red at the end of the video.

I created the video using R, with the code below. You’ll need ffmpeg to combine the individual frames into a video at the end.

library(ggplot2)
library(gridExtra)

## functions for simulated dgp
createSurvivalFrame <- function(f.survfit) {
 ## Create data frame to pass to ggplot2 using survfit result
 if (class(f.survfit)!='survfit') stop('Need "survfit" class object.')
 f.surv <- data.frame(time=f.survfit$time, surv=f.survfit$surv)
 f.start <- data.frame(time=c(0, f.surv$time[1]), surv=c(1, 1))
 # add row at end (0 to end of time)
 f.end <- data.frame(time=125, surv=0)
 f.surv <- rbind(f.start, f.surv, f.end)

 return(f.surv)
}

qplot_survival <- function(f.surv) {
 require(ggplot2)
 p <- ggplot(data=f.surv) + geom_step(aes(x=time, y=surv), direction='hv', lwd=1.5)
 p <- p + theme_bw() + ylim(c(0, 1)) +
 scale_x_continuous(breaks=c(0, 25, 50, 75, 100, 125), limits=c(0,125)) +
 labs(y='Survivors', x='Day')
 return(p)
}

## function to simulate lightbulbs
lights_sim <- function(max.days=100) {
 require(survival)
 require(ggplot2)
 require(gridExtra)

 # Initalize
 obs.bulbs <- NULL
 fixtures <- data.frame(no=1:5, bulb=NA, bulb.life=NA, bulb.spell=0, event=F)
 bulb <- 0
 true.mttf <- round(gamma(1+1/10)*100)
 mttf <- 'NA'

 # Initial survival plot
 f.surv <- data.frame(time=c(0, 125), surv=c(1, 1))
 p <- qplot_survival(f.surv) +
 annotate('text', x=0, y=0.2, hjust=0,
 label='Estimated MTTF:', size=4) +
 annotate('text', x=35, y=0.2, hjust=0,
 label=mttf, size=4) +
 annotate('text', x=0, y=0.1, hjust=0,
 label='Theoretical MTTF:', size=4) +
 annotate('text', x=35, y=0.1, hjust=0,
 label=true.mttf, size=4)

 # Simulate through max.days
 for (day in 1:max.days) {
 # Update spell counters, reset events
 fixtures$bulb.spell <- fixtures$bulb.spell + 1
 # Place bulbs in empty fixtures
 while (any(is.na(fixtures$bulb))) {
 bulb <- bulb + 1
 fixtures[match(NA, fixtures$bulb), c('bulb', 'bulb.life')] <- c(bulb, round(rweibull(1, shape=10, scale=100)))
 }
 # Check for bulbs to burn out now
 fixtures$event <- with(fixtures, ifelse(bulb.life==bulb.spell, T, F))
 if (any(fixtures$event)) {
 obs.bulbs <- rbind(obs.bulbs, fixtures[fixtures$event==T, c('bulb', 'bulb.life', 'event')])
 fixtures[fixtures$event==T, c('bulb', 'bulb.life', 'bulb.spell', 'event')] <-
 matrix(rep(c(NA, NA, 0, F), sum(fixtures$event)), ncol=4, byrow=T)
 # Mean time to failure estimate
 mttf <- round(mean(obs.bulbs$bulb.life), digits=1)
 # Kaplan-Meier surv curve
 surv.data <- with(obs.bulbs, Surv(bulb.life, event))
 f.surv <- createSurvivalFrame(survfit(surv.data ~ 1, surv.data))
 p <- qplot_survival(f.surv) +
 annotate('text', x=0, y=0.2, hjust=0,
 label='Estimated MTTF:', size=4) +
 annotate('text', x=35, y=0.2, hjust=0,
 label=mttf, size=4) +
 annotate('text', x=0, y=0.1, hjust=0,
 label='Theoretical MTTF:', size=4) +
 annotate('text', x=35, y=0.1, hjust=0,
 label=true.mttf, size=4)

 }

# fixture plot for each day
 p.fix <- ggplot(data=fixtures) +
 geom_bar(aes(x=factor(no), y=bulb.spell), fill=rgb(0,0,0.61), width=0.3) +
 scale_y_continuous(limits=c(0, 125), name='', breaks=c(0, 25, 50, 75, 100, 125)) +
 scale_x_discrete(name='Lamp') + coord_flip() + theme_bw() + theme(axis.title.y=element_text(vjust=0.1)) +
 theme(plot.margin=unit(c(1,1,0.1,1.45), 'lines')) + ggtitle(paste0('Day: ', day))

 # Plot frames for each day
 png(paste0('graphics/frames/', sprintf('%03d', day), '.png'))
 grid.arrange(p.fix, p)
 dev.off()

 # Progress bar
 pb <- txtProgressBar(min=0, max=max.days, style=3, width=50)
 setTxtProgressBar(pb, day)
 }

 ## Add last frame with true survivor curve
 # unobserved data-generating process
 dgp <- data.frame(t=1:125, f=dweibull(1:125, shape=10, scale=100))
 dgp$F <- cumsum(dgp$f)
 dgp$S <- 1 - dgp$F

 true <- p + geom_line(data=dgp, aes(x=t, y=S), col='red', lwd=1, linetype='dashed') +
 annotate('text', label='Empirical', x=70, y=0.65) +
 annotate('text', label='Theoretical', x=105, y=0.72)

 png(paste0('graphics/frames/', sprintf('%03d', max.days+1), '.png'))
 grid.arrange(p.fix, true)
 dev.off()

# done, return failures
 return(list(obs = obs.bulbs, current = fixtures, dgp = dgp))
}

### End of functions ###

## Simulate
set.seed(1152359)
sims <- lights_sim(365)

system('ffmpeg -f image2 -r 10 -i ~/path/to/frames/%03d.png ~/path/to/video/lightbulbs.mp4')

r code and github links

We are working on a draft paper, so I don’t want to go through the whole story, but if you’d like to try it out yourself and know how to use JAGS, all the R and JAGS code is available on github.

Basically we are using a state-space model which consists of two separate equations: (1) a state transition equation that models a latent system state and describes how that latent state changes over time, and (2) a measurement equation that describes the process by which we observe actual outcomes (y) at any given time point. As a result, this model separates error in the temporal process that creates our data from the measurement error with which we observe it. An implication is that you can have covariates in either equation, where variables in the state transition equation have effects that propagate through time, i.e. short and long-run effects, while variables entering the measurement equation only have an instantaneous impact for that measurement period.

Monthly Iraqi civilian deaths (black=total, red=Baghdad, blue=rest of Iraq).

We’ve tested this model against simulated data we create, and replication code for these simulations is included as well. For an empirical application, I ran the model against some data on Iraqi civilian deaths from the Iraq Body Count that I was working on earlier. The graph on the right shows the  monthly civilian deaths in Iraq we are working with. The black line shows totals, and red and blue show deaths in Baghdad and the rest of Iraq respectively. I’ve highlighted some the of the spikes in deaths, like during the invasion or government offensives.

In-sample fitted values (red) and observed civilian deaths (black). The red band shows 80% interval.

We estimated a very basic random walk negative binomial model for total monthly civilian deaths, with four binary indicators for the initial invasion period, elections, government offensives, and Ramadan. The in-sample model fit is in the graph on the right. I think this is a pretty amazing fit considering how simple this model is, and the mean predicted values seem to be a pretty good reflection of local levels at any given time. At this stage I would feel fairly confident in using this for hypothesis testing, but fore forecasting it would be nice to have a more explicit temporal structure, e.g. running estimates of trends and changes in trends.

library(maptools)

plot(bosnia)

The function map.scale() from the maps package adds a scale. The position is in map units, latitude/longitude in this case:

library(maps)
map.scale(x=15.5, y=42.75, ratio=FALSE, relwidth=0.2)

And GISTools north.arrow for the north arrow. Units are also in map units. This package has a map scale function as well, which looks nicer but is a little bit more complicated to set up.

library(GISTools)
north.arrow(xb=15.75, yb=43.25, len=0.05, lab="N")

This will produce the following map:

Adding these to a function I wrote for producing thematic maps of Bosnia produces this pretty nice map (with scale and north arrow!) of document per capita civil war deaths:

Here is the function (which is pretty specific to the data I use):

ThematicMap<-function(vector, breaks, title, legend) {
  require(maptools)
  require(shape)
  require(RColorBrewer)
  require(GISTools)
  require(maps)
  
  plotvar <- unlist(vector)
  nclr <- 9
  plotclr <- brewer.pal(nclr, "Reds")
  fillRed <- colorRampPalette(plotclr)
  plotvar[plotvar >= maxy] <- maxy -1
  colcode <- fillRed(maxy)[round(plotvar) + 1]
  plot(bosnia, col = colcode, lty = 0, border = "gray")
  plot(bosnia.st, add=TRUE, lwd=1, border = "gray30")
  plot(bosnia.front93, add = TRUE, lty="solid", lwd=1.5, col="darkblue")
  map.scale(x=15.5, y=42.75, relwidth=0.2, ratio=FALSE)
  north.arrow(xb=15.75, yb=43.25, len=0.05, lab="N")
  title(main = title)
  colorlegend(posy = c(0.05,0.9), posx = c(0.9,0.92),
              col = fillRed(maxy),
              zlim=c(0, maxy), zval = breaks,
              main = legend,
              main.cex = 0.9)
  par(bg='white')
}

killed.pk <- (killed/pop91*1000)
maxy <- 50
breaks <- c(0, 10, 20, 30, 40, Inf)
png("images/map_dead.png")
print(ThematicMap(killed.pk,breaks,"Documented killings during the Bosnian War","per 1,000"))
dev.off()

Using the incident data from 2003 to February 2012, about 27,500 records, I’ve written an R script that assign provinces to ~95 percent of the records, 26,000.

Here’s a basic overview of how it works:

The process basically consists of four steps:

1. Split the location string into separate words
2. Identify a candidate word
3. Strip if of unnecessary parts (“al-”, “‘s”,…)
4. Check against known city-province list
5. Repeat until a match is found

Two functions cover steps 3 and 4. The first strips out the “al-” prefix, which just is an article (“the”), the possessive suffix “‘s”, and a couple of other characters that are in a words sometimes.

DelChar <- function(x) {
    word <- sub("al-", "", x, ignore.case = TRUE)
    word <- gsub("'[s]", "", word)
    word <- gsub("[?,']", "", word)
    return(word)
  }

The other function takes a word and checks it against a list of known city and province pairs (“city.prov”) to see whether it matches either, and if so returns that province name:

ProvLook <- function(x) {
  province <- NULL
  repeat {
    if (x %in% city.prov[, 2]) {
      province <- x
      break
    } else if (x %in% city.prov[, 1]) {
      province <- city.prov[(x==city.prov[, 1]), 2]
      break
    } else {
      province <- "no match"
      break
    }
  }
  return(province)
}

The rest is a loop over the words in location to evaluate each one. It’s not very efficient and takes 7 seconds to run over the 27,500 records I’m using, but it’s effective enough. Any ideas on how to improve efficiency? I don’t know enough to have any clue on this.

I’m pretty happy for now with getting 95 percent of the records, but it would be nice to get all in the future. Here are some ideas for how to do that:

1. Figure out how to accommodate common misspellings. There are certain patterns in how place names get misspelled. For example, “h” at the end of words that end in a vowel (Basra, Basrah), and repeated or substituted vowels (Qadisiya, Qadisiyya; Baiji, Baaji)
2. Accomodate place names that consist of more than one word, like “Salman Pak”.
3. Extent the list of known city-province pairs. Ultimately this could become a brute force solution, although unpleasant considering there are ~1,500 uncoded places.

The Ill-Treatment and Torture (ITT) project by Courtenay Conrad and Will Moore codes Amnesty International (AI) allegations of government torture, including the perpetrator, motive, and judicial response. The aggregated, country-year version of their data shows whether AI made allegations against a country in a given year and if so, what the extent of alleged torture or ill-treatment was, on a 5-point scale from “infrequent” to “systematic”.

Here is a video showing the AI torture allegations from 1995 to 2005 using their country-year data and shape files for world borders from Thematic Mapping.

Government torture or ill-treatment, 1995 to 2005 from Andreas Beger on Vimeo.

The initial impression I had from this is the sheer extent of (alleged) torture and ill-treatment. It looks like pretty much all major states engaged in torture at some point between 1995 and 2005. Only 8 out of 151 states had no allegations of torture at all (Costa Rica, Uruguay, Finland, Benin, Gabon, Quatar, Singapore, and New Zealand), and in those remaining states with AI allegations of torture, on average there were allegations for 7 out of 10 years. More than a quarter of states were accused of torture or ill-treatment in all 10 years covered by the data.

That doesn’t necessarily mean that a lot of torture or ill-treatment is going on in any specific country, nor that it is systematic. It doesn’t reflect what the specific acts of torture or ill-treatment were, e.g. whether someone was tortured to death or water-boarded (which may not be different). But, nevertheless, unpleasant stuff happens.

R code and source. This produces images for each year that I strung together in iMovie.

The term Yugoslav came about sometime in the 19th century while the Ottoman Empire still ruled the Balkans, and was associated with attempts to unite all South Slavs in the Balkans (i.e. Croat, Muslim, Serb, and at some point Bulgarians). The Kingdom of Serbs, Croats, and Slovenes, which had formed in the aftermath of World War 1, was renamed Yugoslavia in 1929 after imposition of a royal dictatorship. It was explicitly used in a way opposed to nationalism.

After World War 2 the communists also used the name and technically every citizen of Yugoslavia was a Yugoslav. But six traditional nationalities, Serb, Croat, Slovene, Montenegrin, Macedonian, and Muslim (by nationality), were officially recognized and most people identified with those or other ethnic minorities. The concept of a Yugoslav was kind of a nascent nationalism, similar to the way Bavarians and Saxons eventually came to identify as Germans and the way people from Virginia or Massachusetts eventually came to see themselves as American. Just not quite that accepted.

In the 1991 census 5 percent of people in Bosnia choose to identify themselves as Yugoslav, the highest percentage in any of the former Yugoslav republics. The map not he right shows the percentage of Yugoslavs in each of the 109 pre-war municipalities. It was highest in parts of Sarajevo and Tuzla, upwards of 15 percent, and lower in rural areas in the south and east. The blue line on the map shows the approximate 1993 front lines during the war by the way. (The R code for this map is originally from Diego Valle-Jones.)

For ethnic conflict research this is pretty nice since in a way it is a direct measure of anti-nationalist sentiment: you could either identify as a member of your particular ethnic group (or nationality as it was used there), or as a Yugoslav, someone associated with the federal state encompassing all the nationalities. Most people who did so probably had married across ethnic lines or were children who did not fit quite into the traditional ethnic categories (like one of my parents), as well as communists and anti-nationalist idealists. Either way, if extreme nationalism is what leads to violence and “ethnic conflict”, then we should expect to see less violence in areas with low nationalist sentiment, i.e. high Yugoslav identification.

This is a pretty particular argument specific to Bosnia and other Yugoslav republics. Conflict research in studies examining more countries than that typically use a summary measure of ethnic diversity that is more generalizable. The most common one is the ethnolinguistic fractionalization index, with ranges from 0 to 1. Higher values indicate more diversity (technically, the chance that two randomly picked individuals will belong to different ethnic groups), and usually are expected to increase the potential for violence.

The same index can be calculated for the 109 municipalities in Bosnia. The bubble chart on the right shows the relationship between this ethnic diversity index, Yugoslav self-identification, and violence. Each red circle is one of the municipalities, with it’s diversity score on the y-axis (ELF), and proportion of self-identified Yugoslavs on the x-axis. Why are there no circles on the bottom right quarter of the plot? Few people identified as Yugoslavs in ethnically homogenous municipalities. In ethnically mixed municipalities (the top half of the chart), there is more variation in how many Yugoslavs there were. So there were some municipalities that were ethnically diverse but where few people thought of themselves as Yugoslav. We might call these “nationalist” municipalities. In others a lot of people identified as Yugoslav.

Now consider that the area of each circle is proportional to the number of deaths per 1,000 in that municipality during the war from 1992 to 1995 (data is from the Research and Documentation Center). Few people died in areas where one ethnic group clearly dominated. In ethnically diverse places, the death rate was highest in “nationalist municipalities”, and much lower in “Yugoslav” municipalities.

One should be careful to infer anything about causes here, especially since the census was conducted a few months before war broke out. It may be that in places where there was a lot of tension, people choose to respond to the census as non-Yugoslavs. Still, it’s a pretty interesting pattern and shows that although ethnic diversity can be associated with high violence, it doesn’t necessarily have to.

I’ve been looking into this stuff as part of my dissertation, and it’s part of a paper you can find here.

An alternative is to use Amazon’s EC2 cloud computing service. A few weeks ago I started playing around with it, and running basic instances for limited time is actually free. I use R/RStudio for the most part, but was overwhelmed by which AMI to use without having to instal R/RStudio. Fortunately someone has created AMI’s (Amazon Machine Images) with RStudio Server, which, once running, lets you use RStudio through your web browser.

To begin, get an Amazon Web Services account and log in to the management console. Louis Aslett has created a number of AMI’s–choose one and start an instance on EC2 with that AMI id.

If you started the instance with the default security group settings like me, you will also have to open port 80 to get access. Go to the security group settings in Amazon management console, select whichever group your instance runs under (e.g. default), and add a custom TCP rule for port 80 (i.e. port range 80). Add the rule and apply. Find out your instance address (instances, at the bottom, it’s the string that ends with amazonaws.com, e.g. ec2-184-88-8-888.compute-1.amazonaws.com), paste into your browser and you should get to a RStudio log in. The defaults are “rstudio” for both. And, there.

Iraq is a democracy, sort of. Although it has a parliament, prime minister, elections and that sort of stuff, it is also pretty corrupt. There has not yet been a peaceful transition of power, a key feature of democracy, as prime minister al-Maliki was reelected to a second term in a troubled election in 2010. It has a polity score  of 3 on a scale from -10 to 10, which doesn’t quite qualify it as a democracy in common usage. According to the Economist Intelligence Unit’s Democracy Index 2010 it is a hybrid regime just a tad above authoritarianism and according to Freedom House it is not free.

In 9 years of war, 111,000+ civilian deaths (Iraq Body Count), and taking into account excess mortality maybe even upwards of 1 million (Opinion Research Business survey) out of a population of 31 million. The Iraq Body Count data is based on press reports of violent deaths and actual deaths are most likely higher. But basically around 1 out of every 50 was killed.

Upwards of 15,00o Iraq Security Forces deaths and around 24,000 insurgent deaths (Iraq War Logs summary at IBC).

About 4,800 U.S. and coalition deaths (iCasualties.org) and 32,000 U.S. wounded (DoD). Also, direct cost of $800 billion or so. But since most of that was financed through debt, the actual cost will probably end up in the low trillions.

Surprisingly, Iraq has surpassed it’s pre-war GDP per capita, even during the war. But then, oil prices have risen as well and most of Iraq’s economy is based on oil.

Weapons of mass destruction. Never mind.

the government generally should increase investment in social science disciplines (anthropology, ethnography, human geography, sociology, social-psychology, political science, and economics) to inform a whole-of-government approach to understanding local cultures and customs and to support future COIN campaigns.

and the report notes that:

the United States government is not investing adequately in the development of social and behavioral information that is critically important for COIN.

Great news for someone hopeful to have a degree in political science in the near future! But beyond that, the report deals with larger issues with modern intelligence and information collection by the military. Much of what people in the military think of first in association with ISR, like unmanned drones, battlefield radars, or ground moving target indicators (GMTI), was developed in the context of the Cold War with the purpose of identifying concentrations of troops, tanks, and so on. It might be great to see that a hexagonal array of tracks on the ground likely is a specific type of surface to air missile site. Knowing that there are a lot of moving ground targets traveling on certain paths in Baghdad probably on the other hand is not so useful. One would maybe rather know about lone targets heading into remote areas (weapons caches?). So a first challenge is figuring out how to appropriately use that type of ISR in a counterinsurgency campaign.

More importantly though, as the report notes, this technical kind of ISR is not well suited to supporting warfare when the population, rather than terrain or equipment, is the focus. Rather, there is more need for the kind of research and analysis that is common in the social sciences, with questions appropriately focused on the local population. In Afghanistan this has been happening under the Humman Terrain System program, which among other things collects local surveys to inform military policy.

Most exciting for me is not only the acknowledgement the people are the key, but a concurrent admission that more rigorous, scientific analytical methods are called for: “analysts will have to operate with a better balance of regional knowledge and theoretical/methodological competence,” compared to the current approach based on indication, specific instances of an event and its analogues, and ultimately subjective intuition. Having done some very limited work along the lines this suggests when I was an intelligence officer in Iraq, and seeing how it is usually received, it will take a large change in attitude for a shift like this to be accepted. Unfortunately.

Some other comments:

• “there are few effective, temporally‐acceptable methodologies for the integration (or fusion) of current levels of data streaming from the many space‐based, airborne, mobile, in situ, and terrestrial remote sensors, let alone real‐ time integration…” Yes! I would love to have a dashboard with choice apps like live video (e.g. unmanned drone), custom satellite imagery, GMTI, …
• In response to a question about emerging technologies and methodologies, the report directly mentions computation social sciences, social network analysis, behavior modeling and simulation, and associated programs like ICEWS. Most of the programs I am familiar with are still at a fairly high level, and might for example try to predict which states are under threat of collapse. But there is a lot that can be done at a much more immediate level with the heaps of data collected every day by U.S. military forces in Iraq and Afghanistan, beyond threat-specific efforts like JIEDDO COIC.