Back To Schedule
Thursday, May 23 • 15:55 - 16:30
Scaling and Securing Spark on Kubernetes at Bloomberg - Ilan Filonenko, Bloomberg

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
In the management of its Data Science Platform, Bloomberg has always focused on providing tenants with secure, reliable, and scalable solutions for their machine learning workflows and ETL pipelines. In adapting Kubernetes to support a diverse set of machine learning workloads, we decided to also support Apache Spark with Native Kubernetes integration. In this talk we'll discuss how we designed: a scalable and resilient External Shuffle Service for Dynamic Resource Allocation, a pluggable interface for secure worker creation, and a token renewal service that handles privacy and security across Spark jobs. These topics will address multi-tenancy, data security and privacy, and elastic resource scalability in the context of running Spark natively on Kubernetes, with an emphasis on disaggregated compute.

avatar for Ilan Filonenko

Ilan Filonenko

Software Engineer, Bloomberg
Ilan Filonenko is a member of the Data Science Infrastructure team at Bloomberg, where he has designed and implemented distributed systems at both the application and infrastructure level. He is one of the principal contributors to Spark on Kubernetes, primarily focusing on enabling... Read More →

Thursday May 23, 2019 15:55 - 16:30 CEST
Hall 8.0 C1