Advances in data science have changed the way we communicate, share, and receive information. Consider how big data has changed our TV and movie experiences for instance. Companies like Netflix collect thousands of data points from several places to make suggestions to users with the help of a tool known as a recommender engine.
Netflix’s recommendation algorithm has a RMSE (root mean squared error) of 0.88, which basically means that the differences between suggestions presented by this model and the user needs are hardly any. This state-of-the-art algorithm is the result of a machine learning and data mining competition with a 1 million dollar prize money. But how did the winning engineering team build it? And how does it work?
Before we dive deeper, let’s say that Netflix is a subscription based streaming service and thus its success can be seen as a function of acquisition rate of new users, cancellations and the rate at which former members rejoin. Simply put, the more members Netflix has, the higher its revenue. That’s why the company holds dear the satisfaction of its customers and it has the recommendation algorithm to thank for it. To demonstrate how important the recommendation engine is for Netflix’s incomes, 80% of stream time is achieved through this system which translates to 1 Billion dollars of savings on customer acquisition.
Netflix uses machine learning to give the platform the ability to automate millions of decisions based on user activities. When Netflix recommends “How I Met Your Mother” because I like “Friends”, machine learning was behind that decision. This suggestion is the Netflix recommendation engine at work: it uses your past activity and returns movies and shows it thinks you would enjoy. But how is that any different from any other recommendation system?
There are multiple potential methods for creating a recommendation algorithm. A basic implementation would be the editorial method in which the platform would make suggestions based on a small number of users such as critics. Another easy one is named the simple collection method where suggestions are based on the top products across the platform. Netflix doesn’t use these methods because they do not allow personalization, nor cover the breadth of the movie catalogs and user preferences. Instead, Netflix slays the recommendation game by the use of personalized methods where movies and shows are suggested to the users who are most likely to enjoy them based on their personal ratings and people with similar interests. This way, the Netflix methodology accounts for the diversity in its audiences and its very large catalog.
Thanks to machine learning, the Netflix engineering team was able to create a “smart” platform that calculates the likelihood of a user liking a product. To understand the probability aspect of the algorithm, let’s consider this example: a utility matrix that is obtained by placing a score on the relationship between users and a movie type to predict their preferences.
Each user has seen titles on Netflix, liked some and disliked some. Based on the utility matrix, Netflix may recommend to user A a “whodunit” since they liked the thrilling plot of a horror movie they have seen recently whereas the algorithm would recommend a soft and light-hearted movie to user B because they did not like a horror movie they saw last week and give user C romance movies suggestions. And thus, each user will receive a suggestion that’s personalized for them!
Netflix interface also uses video ranking algorithms like Personalized Video Ranker (or PVR) which usually filters down the catalog by a certain criteria (violent, romance, thriller…), Top-N Video Ranker which is similar to PVR but looks only at the head of the ranking and Trending Now Ranker which captures temporal trends which Netflix deduces to be strong predictors. These rankers are used to form a row based interface where it is easy to get feedbacks as a right scroll on a row would indicate interest whilst a scroll down would indicate non-interest or irrelevance.
In a nutshell, Netflix’s recommender system collects a huge amount of data from customer behavior that is then used by machine learning algorithms in order to perfect the user experience.
Share your thoughts
Episode 7: How AI could save the planet’s biodiversity?
Innovations in Data Science and machine learning have brought the benefits of Artificial Intelligence to bear on our daily lives. By working alongside machines, people can now accomplish more by doing less. Putting AI into good use can help solve some of the world’s most urgent and difficult problems rather than helping organize our calendars, order our groceries, or play games.
One of the most fundamental global problems today is the exponential loss of biodiversity. In fact, scientists say that our planet is in the middle of its sixth mass extinction, the worst one since the extinction of dinosaurs, 65 million years ago. And that is due to countless direct and indirect human actions, such as poaching (illegal hunting or capturing of animals), overpopulation, habitat destruction, and climate change. Our planet’s rich biodiversity is taking such a huge hit.
In this fight against human greed, AI models have turned into an unlikely ally, helping save our planet, paradoxically from our own hands.
Today, environmentalists are facing unfair odds as hunters call on the latest technology in night-vision goggles, military-grade weaponry, and sophisticated transportation. But thanks to the immense power of deep learning, AI has unlocked the ability to rapidly scan, process, and analyze a variety of signals, identify risks accurately, and provide almost immediate alerts to the authorities.
It is a type of AI system that is particularly effective at pattern recognition and identification. For example, when these models are given thousands of pictures of whale sharks, they can learn to spot a unique whale shark from a future sighting, with remarkable accuracy let alone handle unstructured data such as images, videos, and audio clips. This incredible feature can help solve another urgent matter which is species collection.
To this day, scientists have discovered and described only 1.5 million species of the estimated 10 million on earth. At current rates, we will have to wait almost 500 years to collect all the estimated species and by then, most of them may be extinct. AI and associated technologies have the ability to close this information gap cost effectively and efficiently with hardware becoming increasingly cheap and power-efficient enough to deploy monitoring systems on the ground, on animals, in the sky, and up in space. Early work is proving that algorithms can sift through the massive amounts of data streaming back from these monitoring systems. In turn, humans and machines can begin to identify the plants, birds, fish, and other species captured by these remotely deployed cameras, microphones, and more sometimes down to the unique individual. And we are finding new ways to deploy these technologies every day. For example, Microsoft is working on ways to use organisms such as mosquitoes as small, self-powered data collection devices that can help us better understand an ecosystem through the animals they feed on.
Artificial intelligence can help us understand land-use patterns as well. Microsoft and others are experimenting with ways to turn high-resolution imagery into land cover maps. These maps provide an unprecedented view of what is where, and how it is changing. This in turn helps governments, organizations, and researchers make more informed decisions about when, where, and how to deploy conservation efforts most effectively for the greatest impact. This creates a virtuous cycle of learning, as all this information can then be fed back into AI systems, making them smarter. Thus, AI methods make it possible to build a digital dashboard for the planet, allowing us to monitor, model, and manage environmental systems at a scale like never seen before.
The most obvious use of AI indeed seems to be for further extraction, consumption, and production. However, in the middle of a climate crisis, and with a deteriorating ecosystem, species are dying. Artificial intelligence can be a magical silver bullet that will help us restore the planet. It won’t be an easy journey, but by applying the power of AI to help both humans and our natural systems thrive, we can help provide a better and healthier future for the planet.