A Missing Link in the ML Infrastructure Stack // Josh Tobin // MLOps Meetup #57

MLOps community meetup #57! Last Wednesday we talked to Josh Tobin, Founder, Stealth-Stage Startup. // Abstract: Machine learning is quickly becoming a product engineering discipline. Although several new categories of infrastructure and tools have emerged to help teams turn their models into production systems, doing so is still extremely challenging for most companies. In this talk, we survey the tooling landscape and point out several parts of the machine learning lifecycle that are still underserved. We propose a new category of tool that could help alleviate these challenges and connect the fragmented production ML tooling ecosystem. We conclude by discussing similarities and differences between our proposed system and those of a few top companies. // Bio: Josh Tobin is the founder and CEO of a stealth machine learning startup. Previously, Josh worked as a deep learning & robotics researcher at OpenAI and as a management consultant at McKinsey. He is also the creator of Full Stack Deep Learning (fullstackdeeplearning.com), the first course focused on the emerging engineering discipline of production machine learning. Josh did his PhD in Computer Science at UC Berkeley advised by Pieter Abbeel. // Other Links https://josh-tobin.com course.fullstackdeeplearning.com ----------- Connect With Us ✌️-------------    Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Josh on LinkedIn: https://www.linkedin.com/in/josh-tobin-4b3b10a9/ Timestamps: [00:00] Introduction to Josh Tobin [01:18] Background of Josh into tech [08:27] We're you guys behind the Rubik's Cube? [09:26] Rubik's Cube Project [09:51] "Research is meant to show you what's possible to solve." [11:07] "That's one of the things that's started to change and I think the MLOps world is maybe a part of that. What I'm excited about this is that people are focusing on the impact of their models." [13:18] Insights on Testing [17:11] Evaluation Store [18:33] "Production Machine Learning is data-driven products that have predictions in the loop." [23:40] Analyzing and moving forward [24:02] "My medium term mindset how machine learning is created is that is there's still gonna be humans involved but humans will be more efficient by tools." [25:50] Is there a market for this? [27:40] "The long tale of machine learning use cases is becoming part of every products and service more or less the companies create but it's the same way the software part of the products and services the companies create these days. It's going to create an enormous amount of value." [30:09] Talents [32:52] Organizational by-ends and knowledge [35:16] Tools used for Evaluation Store 39:59] Difference from Monitoring Tool [42:10] Who is the right person to interact in Evaluation Store? [50:05] Technical challenges of Apple and Tesla [53:30] "As Machine Learning use cases are getting more and more complicated, higher and higher dimensional data, bigger and bigger models, larger training sets many companies would need in order to continually improve their systems over time."