Intelligence Analysis Methods in Scientific Literature

Intelligence analysis methods in scientific literature

In a case study I recently completed I had the welcome opportunity to explore in more detail an interesting phenomenon in scientific publications: selective presentation of study results to scientific audience. The example below illustrates publication bias and multiple publication bias at its “best”.

Whilst I do not underestimate ability of clinicians and scientists to screen information from wide variety of sources and process it accordingly, I doubt that overloaded physicians have the time and capacity to scrutinize the origin of every published study in their respective area of interest, and search registries for unpublished studies to get a clear unbiased picture.

In an effort to improve confidence and uncertainty parameters in the results of review of scientific literature I took the time and tried to match all relevant hits in search for publications to the original study ID. There were several registries I could use; and most of the time the results of studies were already posted, including listings of publications. This was a good starting point because especially with new entities this is not always the case. Not all identified publications could be traced back to the original study. In fact majority of them could not.

In this particular example, the manufacturer conducted 84 studies and declared 154 publications for the compound in question based on them. Of these 84 studies in the manufacturer’s registry, 56 were listed as sponsored by that particular company in ClinicalTrials.gov. This does not come as a surprise, because some of them were conducted in collaboration with other entities or completed before the database was in operation. On 9 occasions the manufacturer declared that the results were not published in scientific press at all and on 23 occasions the number of publications was not disclosed. Three of these 84 studies were published extensively: 20, 14, and 12 times, respectively. Much more effort would be required to identify the origin (study ID) of all 575 published studies identified in PubMed, including those originating from other sources.

Scientific literature collected from traditional databases can be examined using target-centric approach as in intelligence analytics. The target is not the substance (active ingredient) itself but a complex system which includes multiple stakeholders and participants whose behavioral patterns in information production and sharing differ substantially from one another. Target as a system comprises of structure, function, and a process. Academic environment, a research hospital, or a commercial entity, produce information of interest on a particular compound for differing reasons. Each follows its own unique set of motivators and incentives.

Review of scientific literature follows the same principles as any other intelligence collection and analysis activity. The main reasons for intelligence failures are failure to share information or its selective sharing, failure to analyze the collected material objectively, and reluctance/inability of the customer to act on provided results.

Incorrect, distorted or misrepresented findings are not without impact: published studies translate into clinical guidelines, benefit: risk assessments on medicines and procedures, and S.W.O.T. analyses in project reviews.