Muhlberger & Stromer-Galley on Measurement of Deliberative Quality in Online Policy Discussions

Professor Peter Muhlberger of the Texas Tech University Center of Communications Research and Professor Jennifer Stromer-Galley of the University of Albany Department of Communication have published Automated and Hand-coded Measurement of Deliberative Quality in Online Policy Discussions, in dg.o 2009: Proceedings of the 10th Annual International Conference on Digital Government Research: Social Networks: Making Connections between Citizens, Data and Government 35-41 (2009). Here is the abstract:

A number of research projects are now underway to assess how IT tools and deliberative techniques could enhance public input into government at all levels. The success of such projects depends to an important degree on objective measurement of the quality of discussion under different utilizations of tools and techniques. This paper explores the value of two techniques for determining the quality of policy discussion in the context of federal agency rulemaking: a human-coded technique and a technique involving statistical bootstrapping of natural language processing data. The hand-coded technique utilizes counts of verbal behaviors that are indicative of intellectual engagement. This includes such behaviors as raising questions, disagreeing, and introducing new topics. The automated technique develops a measure of sophistication of reasoning based on significant co-occurrence of concepts in participants’ speech. Such co-occurrence clarifies the network of conceptual relations utilized by participants. The more connections participants make between policy-relevant concepts, the more sophisticated their speech should be. Data consist of discussion of the federal rulemaking issue of network neutrality regulation by a sample of 53 volunteers. Correlations and ordinary least squares regression find that the conceptual connection and hand-coded measures, despite being qualitatively quite different, significantly predict each other and several other measures of sophistication, including two indicators of network neutrality knowledge and sophistication of views of government.