Effective Utilization of Machine Learning Marketplaces

Speaker: Lingjiao Chen
         Stanford University

Title:   "Effective Utilization of Machine Learning Marketplaces"

Date:    Wednesday, 13 March 2024

Time:    10:00am - 11:00am

Zoom link:
https://hkust.zoom.us/j/96688516988?pwd=Z3YzcVJ4RVB2L25WakhaVFd6TngxQT09

Meeting ID: 966 8851 6988
Passcode: 202425

Abstract:

Traditionally, machine learning (ML) researchers and practitioners have
focused on building and refining models starting from a dataset they have.
Today, this paradigm has undergone a significant transformation: a growing
number of users now utilize ML as a service, through cloud services like
Google AI, as well as foundation models like GPT-4. The shift to this
service-centric paradigm introduces a suite of new challenges that are
intellectually interesting and crucial for today's ML users. For example,
many AI services are available for a given task, but there is a large
heterogeneity in their price and performance. Given specific budget and
data requirements, how should one decide which services to utilize and how
to use them? Both AI services and user data are continuously updated, and
thus there is no absolute rank of the same service over time. How should
one monitor an AI service's performance over time in a data-efficient and
compute-efficient manner?

In this talk, I will describe my work on addressing these fundamental
challenges arising from this new ML paradigm. My first line of work,
including FrugalML and FrugalGPT, shows that for a range of tasks,
adaptively deciding which ML services to use and how to use them can match
or exceed the performance of the best individual ML API (such as GPT-4)
with over 90% cost reduction. I will also talk about my discovery of model
drift, the phenomenon that the behavior and performance of many AI
services (e.g., ChatGPT) drift over time, and explain how to monitor model
drift efficiently via a paradigm called MASA. My research has been
explored and deployed by high-tech companies including Databricks and
Celonis, and covered by mainstream media such as the Wall Street Journal,
Fortunes, and the New York Times.


********************
Biography:

Lingjiao Chen is a fifth-year PhD candidate in the Computer Science
Department at Stanford University, co-advised by Professor Carlos
Guestrin, Professor Matei Zaharia, and Professor James Zou. His research
interest lies broadly in machine learning and data systems, with a recent
focus on the efficient and reliable utilization of machine learning
marketplaces. His work has been supported in part by a Google PhD
fellowship.