ML Tests // Svet Penkov // Coffee Sessions #61

MLOps Coffee Sessions #60 with Svet Penkov, ML Tests. // Abstract How confident do you feel when you deploy a new model? Does improving an ML model feel like a game of "whack-a-mole"? ML is taking over all sorts of industries and yet ML testing tools are virtually non-existent. Drawing parallels from software engineering and electronic circuit board design to the aviation and semiconductor industries, the need for principled quality assurance (QA) step in the MLOps pipeline is long overdue. Let's talk about why ML testing is hard, what can we do about it and what place should ML QA take in the future? // Bio Svet has been building robots ever since he was a kid. At some point, Svet got interested in not just how to build them, but actually how to make them think, and so he did a Ph.D. in AI & Robotics at the University of Edinburgh, UK. Towards the end of Svet's Ph.D., he joined FiveAI as a Research Scientist and led the motion prediction team for 3 years. Throughout his career, Svet spent endless hours fixing model regressions and fighting with edge cases and so at some point he had enough of it and decided it's time to do something about it. That's how Svet started Efemarai where they are building a platform for testing and improving ML continuously. // Relevant Links --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, Feature Store, Machine Learning Monitoring and Blogs: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Svet on LinkedIn: https://www.linkedin.com/in/svpenkov/ Timestamps: [00:00] Introduction to Svet Penkov [02:10] Svet's background in tech [04:34] Testing on robotics vs areas of machine learning [05:21] What's missing in testing right now? [08:56] Who should test?            Step 1. Figuring out the requirements [12:04] Edge cases            Steps 2. Access of variation [13:29] Step 3. Validation and Verification [16:15] New challenges that need to be addressed [18:25] Test-driven development viability argument   [20:26] Software engineering tests vs machine learning engineering tests [23:23] Rule of tools in MLOps [26:15] Figuring out the difficulty in designing the API's [27:48] Svet's vision for the future [29:15] Moving goal post [31:00] 10 data points being realistic [31:27] Getting less [32:20] Efemarai: Where it came from and Why? [33:53] Efemarai - Functional Magnetic Resonance Imaging   [35:21] A perfect world journey [36:22] Value of tests [37:55] Get ready for the MLOps Community Slack testing channel!