Category Archives: Technology

Signal to noise

When people talk about information analysis, there’s often a lot of worry about noise in the data, and the reliability of the data sources. So when you’re building an information analysis system, there are often requirements that have to do with “filtering out bad data” or assigning “reliability scores” to data sources.

But in practice this isn’t usually necessary. With enough data, noise suffers from destructive interference, and signal interferes constructively. I first learned about this in physics class in school, and first encountered it while doing radio astronomy. I was a research assistant on a team that realized we had an opportunity to make the highest quality radio maps ever made of certain galaxies. See, when big stars go supernova, astronomers like to aim the Very Large Array of radio telescopes in New Mexico at them to study how the massive explosions grow and change. They get a lot of data, write their papers, and move on. But supernovas happen in galaxies. And scientists share data. So our PIs realized they had very long exposures of the galaxies containing the supernovae. Radio astronomy “pictures” tend to be noisier than pictures taken by optical telescopes, and noise in a picture of a distant galaxy can overwhelm the detail of star formation regions and the like. However, noise is random, whereas the bright spots (except for the supernova) don’t change noticeably. So if you take the sum of enough radio data, the random noise cancels out, and the actual bright regions become clear. Note, this is not the same thing as astronomical interferometry, although that was a technique we made use of.

The same thing is true of many other types of noisy data. If you only ever look at the data at a point in time, or watch or listen to it as it changes, it’s difficult to see the signal through the noise, but if you have a system that allows this summation to happen, and you can look at the sum, suddenly the picture becomes clear.

Suppose the “ground truth” looks like this:

A draftsman I'm not.

But we have noisy data that looks like this:

I hope that's not an EKG.

If we layer lots of noisy data, we can start to see that the signal’s there…

I drew this better on the whiteboard last week.

But if we can sum the data it looks something like this:

"The hands acquire shakes, the shakes become a warning. It is by caffeine alone I set my mind in motion." --

Now we can clearly see the signal! Is it truth? Not necessarily, but the analyst can now see that there is agreement across the data. If you want more information, you also need a system that gives you a way to dive into the details of what data contributed to the peaks. And now you also have guidance as to where to collect more data, ideally from additional sources. More on that in another post.

This is related to why Google is so good and your organization’s internal Search is so bad. Even though Google’s data source, the Internet, is way noisier than your organization’s intranet (I hope), Google is still better. This is true even if you have an in-house Google appliance. It’s because of Google’s second big data source (equally noisy): the billions of user clicks. Google doesn’t show you the sum of that data, but it does use that aggregation to decide what to show you. In essence, Google finds the peaks of agreement among billions of clicks and shows you just the peaks. Your IT department doesn’t have enough click-data to do that for your organization, no matter how good their software is.

Reliability Scores

You also probably don’t need to assign reliability scores to your data sources, even though it seems like a perfectly logical, even prudent, thing to do. The problem is that the scores will be fairly arbitrary, hard to agree on, and may present a false sense of rigor where there isn’t any. There’s lots of ways a data source can be unreliable, but we’ve found different ways to handle them that avoid these problems. For example:

Problem: There’s hardly any signal (useful information) in the data.
Solution: You’re not going to get any peaks even when you sum it all together. Don’t score it, ditch it.

Problem: There’s a lot of mistakes or inconsistencies in the data.
Solution: That’s the kind of noise that cancels out if you have enough data. If you do, then don’t sweat it.

Problem: The data has been deliberately redacted to remove what you’re looking for.
Solution: The more data there is, the harder it is to do this perfectly. If you get enough of it, you can find what was missed. Also, if you have enough of it, you’ll see mysterious quiet areas of data, because not only is the signal gone, but so is the noise. So you can detect the obfuscation, and you might even be able to catch the deceivers in a mistake.

Problem: The data is out-of-date.
Solution: This is absolutely relevant to the analysts, and it should be documented, but it’s not something you score. The analysts just need to know because data timeliness matters more for some questions than others.

Problem: There are gaps in the data coverage.
Solution: Again, it’s relevant, and should be documented, but it’s not a “reliability” issue. Maybe there weren’t enough 18-25 year-olds in the medical study you’re analyzing. Even so, if there’s a statistically significant result visible for 25-35 year-olds, you’ve still found something; you just don’t know if it works for young people.

Problem: The data’s useful, but its noise is obscuring the signal of other, cleaner data sets.
Solution: Let the users turn data sets on and off as they choose. See, this is actually what people have in mind with reliability scores – they imagine they’ll either weight more reliable data higher, or the user will use the score to decide what to look at. It’s true, you might end up doing some form of weighting. For example, a search engine might weight clicks higher for users who appear to come from a part of the world that speaks the language the results are in, so clicks from Italy have more effect on ranking Italian search results. But you don’t want to do this before you’ve had a chance to work with the data in the system. As for showing the reliability scores to the analysts…believe me, your analysts already have strong opinions on the reliability of individual data sources, and they will ignore your well-intentioned ratings. If you just give them the ability to turn them on and off, they’ll be happy and productive.

In short, signal reveals itself in noisy data if you have enough of it. And have tools that let you work with all of it in aggregate, while still letting you quickly get the details of the revealed signal.

My story?

My dad bought a TRS-80 Color Computer when I was about 5. I didn’t learn to code, but I saw a modem, heard binary being played on our cassette drive, and learned what a kilobyte is.

Later, I learned Logo and BASIC when I was 8 and 9. Just very simple toy programs. I learned more sophisticated programming in Pascal in high school. I did have books, and Dad got me started, but my schools’ programming classes get at least half the credit.

I started getting paid to work with computers while still in high school. I have made money ever since from working with computers. Even the years I taught ballroom dancing full-time, I wrote software part-time and brought in new revenue at the studio by setting up the website and our first online sales of gift certificates.

Today I live in Vienna and manage a significant software project at the International Atomic Energy Agency. As a job, it’s amazing, and the work is important. I’m writing this from a lovely apartment in Venice where I’m vacationing with my family while the team works without me.

It’s a good life, and I’m incredibly grateful. I wouldn’t be here if I didn’t know how to code.

But it’s not just about the good work you can do and the good life you can have. It’s fun. The things we can do now with software are amazing. A programmer in the 80s would be awed by what’s possible to coders now. It’s not just faster computers, it’s the fact that so much of the world is now online. Take something simple like flight bookings: they were computerized in the 80s (probably earlier), but in closed systems. Today, there are so many ways to tie that information together that travel booking sites abound, and the best ones are so good that we can be near-omniscient about our options. We think little of booking, from our couch, vacations with airlines and hotels we’ve never heard of.

Coders regularly produce apps which do things that weren’t possible a few years ago. My phone (an anachronistic name for the hyper-connected supercomputer I carry in my pocket) can augment my reality in countless ways, but the latest is holding it up and looking through it so that all the Italian writing is replaced with English.

What’s next? Imagine writing code to do this:

  • social apps that allow you to point your finger and write in the sky…where all your friends can see it through their glasses or contact lenses.
  • designing toys and selling them online where buyers click to print them out on their 3D printers
  • building the apps to do the designing I just mentioned or building the site to broker the transactions
  • writing code to control swarms of tiny flying/crawling robots to…well, frankly the first of these will all have military or intelligence applications which may appeal to some, but, after that, there will be plenty of environmental and scientific uses.