Algorithms don’t apply to outliers. But what happens when an outlier event becomes normal?
Machine learning predictive models were left confused when COVID-19 hit the world. As consumer behavior changed 180-degree, and nothing reflected usual.
The normal working of machine learning algorithms relies on the normal distribution of attributable values. Data outliers can spoil their results and mislead training process. This is what ensued in the pandemic.
COVID-19 served as an eye-opener for the data scientists and machine learning engineers.
Data science recommender systems became unreliable and less accurate and gave poorer results. But what happened?
Data Science Council of America (DASCA) in its blog on “Data Science Driven Recommender Systems in the Post-COVID Era,” elaborates on the fresh understanding of data science professionals about shortcomings of current recommender systems and outlays what the future recommender systems would encompass. Below given are the highlights. The complete blog is available on the official website of DASCA.
About recommender systems and their uses
In a nutshell, recommender systems convert browsers into buyers. How?
By mirroring the website or other offerings as per the needs of the user (reflected through her browser). Browsers eventually are customers who buy via browsing for a product or service on a website. What recommender system algorithms do is improve the chances of a purchase by showing products that one is really interested in.
The first recommender systems were based on the content that focussed on product rather than user. New age recommender systems are more advanced with a knack for filtering products as per several of a user’s characteristics – age, sex, occupation, hobbies, and more.
Examples of famous brands using recommender systems machine learning algorithms.
• Amazon
Amazon uses data from its vast universe of customers to identify the items that are brought together – and based on that it recommends to users’ the items when they make any purchase. Furthermore, the profile of a user is also captured and built as per the buying behavior, browsing history, and ratings provided that is eventually used to make recommendations.
• Netflix
It uses data from users when they watch a movie, rate it or set up their preferences. These actions are then used by Netflix to recommend shows to the users.
LinkedIn uses recommender systems to pick jobs as per one’s liking and suggest people they would like to connect with.
Similarly, Spotify, Google, YouTube, and most other digital platforms leverage recommender systems to collect data, and automatically make suggestions or recommendations.
How COVID brought havoc on recommender systems
Generally, recommender system machine learning models are trained as per normal behavior, but the pandemic pushed people and organizations across the globe to act and behave differently than their normal.
DASCA quotes Nozzel’s CEO with a fitting advice for data scientists – A data science team must know what is going on in the world. Algorithms cannot pick this.
The disorder brought by COVID-19 outbreak proved how an outlier can become a normal and remain so.
When machine learning models are confused and show signs of cracking, it makes it imperative for humans to step in.
Breakdown of recommender algorithms during COVID outbreak
Amazon’s top items were tech gadgets until April. However, later customer preferences saw a U-turn post-COVID and recommender systems were confused.
The popularity of eCommerce broke records after a decade of growth.
Toiletries, and Masks became the new stars of eCommerce world.
Toilet paper jumped from preference of over 2000 to number four in predictive models.
Similarly, a pack of masks jumped over 35,000 ranks to make it to top 10.
Some dubbed it a “madhouse.”
COVID-19 shook the systems and jolted recommender systems machine learning algorithms.
The decades of confidence in predictive models of recommender systems came crashing down.
Machine learning models as a result had to redefine their normal and address consumer behavior that was no longer normal.
Visit DASCA Insights to get more statistics and details on COVID’s impact on data science recommender systems.
Future of recommender systems
There is a need for data scientists and machine learning experts to better their strategy of creating and training recommender systems.
The ROI lost by companies, which could be saved by more efficient and advanced recommender systems was huge.
New age recommender systems must be grounded in cognitive aspects of consumers – including behavior, attitude, and personas – if they are to accurately predict changes in big events.
New age recommender systems should be driven by five trends – curated around detailed aspects of a consumer – as mentioned by DASCA.
Complete details about them is available on DASCA Insights.
How should recommendation engine strategy be modified?
In this changed reality, recommender systems in eCommerce must become adaptable as soon as the awareness of an outlier event (such as an outbreak) is realized.
Predictive models can consider, and map changed user behavior in such situations.
Data science professionals need to understand the overall ecosystem – including algorithms and search modules, their interplay, and user engagement in different circumstances.
Five trends (centered around cognitive aspects of a consumer) should be included by the data scientists to make smarter recommender systems. Know about them on DASCA Insights.
