The Rightful Use of Big Data

People often express concern about the role “big data” and predictive analytics will hold in the near and long term future of our society.  Some fear that critical decisions will be made automatically with little to no human intervention.  The reality is that computers are often able to make life-saving decisions much quicker than humans.  The systems in Google’s self-driving cars are able to recognize a vehicle breaking or swerving into its lane and properly react far quicker than even the most attentive human driver (none of whom drive in Manhattan). Automated control systems in our energy grids are able to quickly and efficiently reroute power in the event of a system problem, restoring electricity before most customers have even noticed there was a problem.

Like with almost every other technological advance, innovation in big data is best seen as a handmaiden to humanity, enriching our potential and furthering our survival. Some decisions are better made by a computer, particularly when it involves making the same decision any rational human would make given the same information the computer is able to process.  In other words, if the self-driving car is going to react to someone swerving into their lane in almost the exact same way as a human driver (minus perhaps the one-finger salute), but do it in a fraction of the time and have a likely beneficial effect for the driver and any passengers (due to a decreased likelihood of an accident), it’s right that a computer should be entrusted to make that decision.

Should the computer then be allowed to tell us where to go and when we should be there? Obviously not.  In this case, the computer fails at the ability to make the same or reasonably similar decision as the human driver even with the same level of information.  We don’t want to become slaves to the systems and the data that’s fed into them, but there is a sphere of decisions that can be easily made by automated systems operating on easy to process information.

There are other important applications for automated systems to effectively utilize insights from data: education.  A recent blogpost from the CEO of Knewton, Jose Ferreira, made an interesting point about the use of big data in education. Having been through the meat-grinder of constant testing throughout my academic career, I understand to some extent the impact that standardized tests have on education, at least from the students’ point of view.  Having a number of friends teaching in public schools, I’m also aware of the impact it has on educators who want to be inspirational role-models but end up being nothing more than test proctors.

Which is why I found this blogpost so interesting.  He doesn’t argue that standardized tests are a waste.  They are indicators of how well students are doing in school, but they aren’t the finely-crafted assessment tools many college admissions people think.  With a 28-point margin of error on the SAT, a score cut-off for school admissions becomes almost meaningless, if not cruel. State standardized testing is likewise fraught with issues in how they assess student ability. Add onto that the variability of performance. A test administered once a year really only assesses how well the student was doing on that particular day (a day fraught with all kinds of anxiety and fear) rather than throughout the entire school year.

What Ferreira argues for is a continuous measure of student performance.  Obviously he’s interested in promoting the Knewton adaptive learning solution, but I think the point is clear: as with self-driving cars and smart-grid control systems, the ability to quickly collect, analyze, and act on available data allows for a better holistic understanding of how our children are learning and provide quicker, more effective interventions in their learning process to address issues before they become problems.  The data doesn’t tell students what to learn but assesses how well they’re doing with the material they’ve been given, helping educators and parents make better-informed decisions.

This isn’t to shill for Knewton, though in full disclosure I do own a “Knerd” T-shirt.  I think this is how we rightfully use big data, as a means to better enable the survival and success of our society by optimizing the collection and analysis of information important to better decision making.  The constraints of size and complexity no longer make this an impossible task, and for applications such as education and transportation, there is also the hope of making a beneficial change from which we all can benefit.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s