The Cochrane Collaboration

Simply they are developed without due knowledge or understanding of prior art.

I read papers every day, and I have a long memory. In fact I’ve been reading papers for 15 years both papers for my specific research, and then again those that just interest me. And for the last eight years I’ve kept notes on each paper to help me remember. I got into the habit by taking a lesson from one of my students, Darren Lunn. I dare say Darren can remember every paper he’s ever read, so copious where his notes! But this isn’t the case for a vast amount of the research I see in my field at least.

Anecdotally, I surmise that the way research software gets developed is that first an observed need is identified, then a technical idea forms, from this a development ensues.

Then there is a disconnect!

Now as its research we need to publish, and only at this point is a literature survey performed. And performed with the explicit intention, not of understanding prior art, but of fulfilling the criteria of the publication and review process.

So why is this bad? Well it’s bad for two reasons:

First, because we seem to keep reinventing the wheel. Incremental developments are normal in science, we all stand on someone else’s shoulders after-all. But it’s becoming obvious that in certain areas science is not advancing, it is static, each development is not making incremental advancements, but redeveloping the same thing, just slightly differently, this is looping not advancement.

Second, because I approximate, we seem to have a 10 year domain memory. If we only ever look back to satisfy, what some consider onerous, expectations imposed by the review process then we only ever get those references that pop up at the top of the list, and those are often within the last 10 years. Indeed, this is not only the authors fault, I’ve seen some terrible reviewing which ask not for a seminal paper 20 years old, but for a new incremental development 5 years old! Seminal papers, and old papers are never out of date as they represent steps along the path! Only in CS does this stupid mindset persist!

So what can be done? Well, observe the need and form a technical idea by all means. But then perform a structured or more strict Cochrane review. These kind of reviews are common in many other fields, maybe not so match in engineering related domains, but for medicine and the life sciences they are super necessary.

But what if you think a structured review is over kill? Well build up a mental library of what is relevant and being published! How? By signing up for email alerts from every good publisher, and then read read read, more reading produces better ideas, and allows you to have a command of the background to your wider domain. Finally, test this out at conferences by suggesting relevant work to presenters, they will normally be happy to get the reference, and it provides for a coffee break conversation ice breaker!