Recommendation engines are becoming very popular over last few years. As you know websites like Amazon, Walmart, Ebay etc have ton of content. These websites are constantly looking for ways to surface the content to user in several innovative ways. One way is to provide ability to search the products that users are interested.
Here user enters camera as search term and Ebay shows top 10 results on page 1. Although Ebay has several thousand search results for camera, users would hardly navigate beyond first few pages. Hence “Search” is often termed as needle in the haystack problem. In other words, what you see on first page is just tip of the iceberg. How would we make the rest of the iceberg visible to user thereby improving sales revenue.
Enter Recommendation Engine. Recommendation engine is powerful technique where some ideas like popularity, collaborative filtering and personalization are used to provide suggestion to user as he/she navigates through the site. Here are few techniques along with examples
Discovery: Even when users are not searching, can we recommend products to user based on popularity, nicheness, innovativeness? Amazon uses this technique to help user discover new products. If you visit https://www.amazon.com/discover , you will find may interesting products that you would not normally find on amazon.com first search result page. If you visit Netflix, you will find several panels even before you initiate search, one such panel where you movie/tv shows recommendation is by “Trending” nature of these movies/shows. If a movie/tv show become viral , then it gets categorized into Trending nature.
The goal here is to upsell content based on property of content which happens to uniqueness for Amazon and Trending for Netflix.
2. Collaborative filtering: Another useful technique is collaborative filtering. Here the basic idea is to use crowdsource information like what other users have bought or liked and then correlating with user’s habits with other users (user based collaborative filtering) or product’s that are similar to product that user is about to buy (Item based collaboraive filtering).
User Based Collaborative filtering: Here the idea is similar people like similar movies. So say if Peter has liked “Mission Impossible”, “Rambo”, “Terminator” and “alien” and Say I have liked Mission Impossible”, “Rambo”, “Terminator” and I would like to watch next movie, since my watching habits are most similar to that of Peter, obviously the next movie recommendation for me would be “alien”.
Although this approach is easy to implement, it is inaccurate at times because user habits keeps changing more often. Also user’s frequency of watching movies is variable and data is very sparse ad not every user watches every movie.
A better alternative is item based collaborative filtering. Here instead of computing similarity between users, we compute similarity between items as rated by users. Here we start with creating vectors for movies instead of users.Each row is a move and each column is user and each cell value is movie rating by a user for that specific movie. When we plot it in a pivot table , each movie row represents a numeric vector which can be used to compute similarity between 2 movies using KNN or Pearson Cofficient matrix.
Before we compute a similarity vector, as you can see, the matrix is very huge and computationally very expensive. So we use dimension reduction like SVD (Singular Value Decomposition) to reduce number of columns in above matrix to reasonable number like 20. Once we do that , the matrix would be changed as follows
What SVD does is it reduces the columns in above matrix thereby creating a latent variables like concept1, concept2 and concept3 to densely express movie similarity. One could try to create analogies to express these concepts but it may always not so obvious. In above example, we could treat concpets as movie genres and verify if our analogies holds true.
Once we have reduced matrix, then if I have watched movie Rambo, then suggestion for recommendation could be set of vetors which are closer to Rambo vector above. This can be easily computed by using pearson r coefficient matrix.
If you would like to see step by step example of how to use item based similarity to compute recommendation web application
f you would like to learn how to build recommendation engine in under 2 hrs , sign up for following course
You can watch introductory video below to get an idea.
About Author Evergreen Technologies:
Active in teaching online courses in Computer vision , Natural Language Processing and SaaS system developmentOver 20 years of experience in fortune 500 companies
•Linked in: @evergreenllc2020
•Udemy: Evergreen Technologies