Opening up new encounters for users with ambiguous requirements
As information retrieval tools battle for dominance, Megagon Labs
, Recruit Group's AI research institute, has entered the fray with "OpineDB,"a system for searching natural language in text. Wang-Chiew Tan, who leads the US research team, discusses the development process, the potential for use in Recruit Group's services, and what the future holds.
Megagon Labs' Mountain View office located in the heart of Silicon Valley, USA.
From keyword search to natural language search
Recent years have seen progress soar in the natural language processing (NLP) field, namely in the analysis of everyday words and sentences. Dramatic progress in language models means performance on tasks such as automatically answering questions and understanding logic is now far more accurate. In 2019, Google announced a strategy shift away from keyword-based search toward natural language search, sparking a major transition in the info-searching arena.*1
Anticipating these changes, the US research team at Megagon Labs, led by Wang-Chiew Tan, has been working on a new system for searching text that contains subjective and experiential expressions in natural language, such as user and customer reviews. The Megagon Labs team wrote a paper about OpineDB that appeared in VLDB 2019, the world's premier database conference. Their research was subsequently invited and will appear in a special issue of VLDB*2 Journal soon.
*1. Source: Pandu Nayak, Google Fellow and Vice President, Search "Understanding searches better than ever before" google blog, published Oct 25, 2019
*2. VLDB (Very Large Data Bases) is the top international conference on data management. The VLDB Woman in Database Research Award honors female researchers who have made outstanding contributions to the database research community. To date, there have been five recipients of the Woman in Database Research Award, and Wang-Chiew was the fourth winner.
Q&A:
How will the user search experience evolve with OpineDB?
Wang-Chiew: "Well, when it comes to job-seeking, for example, most people just enter the job title, location, annual salary, employment status, and other specifications of choice into the search box on the site. But the only way to find out about vague and diverse intangibles outside of the job description, such as corporate culture, work-life balance, and benefits, is by reading tons of reviews. OpineDB's game-changing feature is its ability to find all the relevant information for a particular query and display that information in a meaningful way. For example, if a user searches for a "good working environment," OpineDB will parse reviews for relevant phrasing such as "flexible hours," "fun company culture," etc. Even users with somewhat ambiguous requirements can find helpful information that previously would have been inaccessible. This way, people can make better informed choices. Megagon Labs wants to empower people with the right information to make their best decisions by harnessing the right technologies. This is the mission of our lab."
Wang-Chiew Tan and Eser Kandogan (Head of Engineering, Megagon Labs Inc.) are speakers at Recruit's in-house knowledge sharing event called FORUM with a presentation titled, "AI technology R&D that distills, accumulates, and searches for information in large volumes of natural text sources."
Several Recruit Group companies have begun assessing the feasibility of using OpineDB in various Japanese and English applications. Jalan.net has even released a dataset of 120,000 reviews to contribute to the development of Japanese NLP research. How else will Recruit Group use the system?
Wang-Chiew: "Recruit Group has a wealth of unstructured data, including reviews, job descriptions, and Q&A posts. OpineDB is capable of analyzing all of this data and providing multiple and unrivaled insights. Improving search experience based on such insights should open up abundant business opportunities. OpineDB is just one of many tools we are researching, and we are constantly seeking ways to explore new technologies. Nothing would please us more than to see our cutting-edge technology contribute to service development and innovation in Recruit's matching businesses. We will continue our rigorous research so that we can deploy Megagon Labs' technology in services that have a tangible impact on the future of Recruit, our communities, and the world."
Wang-Chiew Tan in an online meeting with members of the Tokyo office.
Wang-Chiew Tan
Head of Research, Megagon Labs Inc.
After her 14-year tenure as a Professor of Computer Science at the University of California, Santa Cruz and with a two-year stint in between as a researcher at IBM Almaden Research Center in Silicon Valley, Wang-Chiew joined Megagon Labs in 2016. In 2015, she was named a Fellow of the Association for Computing Machinery (ACM), the world's leading computing society*3. In 2019, she won the VLDB's Woman in Database Research Award.
*3. The ACM (Association for Computing Machinery) is one of the most prestigious academic associations for information technology. Outstanding ACM members are named to Fellow status in recognition of exceptional accomplishments and technical, professional, and leadership contributions to computing.