Back To Schedule
Thursday, May 23 • 15:55 - 16:30
Scaling and Securing Spark on Kubernetes at Bloomberg - Ilan Filonenko, Bloomberg

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
In the management of its Data Science Platform, Bloomberg has always focused on providing tenants with secure, reliable, and scalable solutions for their machine learning workflows and ETL pipelines. In adapting Kubernetes to support a diverse set of machine learning workloads, we decided to also support Apache Spark with Native Kubernetes integration. In this talk we'll discuss how we designed: a scalable and resilient External Shuffle Service for Dynamic Resource Allocation, a pluggable interface for secure worker creation, and a token renewal service that handles privacy and security across Spark jobs. These topics will address multi-tenancy, data security and privacy, and elastic resource scalability in the context of running Spark natively on Kubernetes, with an emphasis on disaggregated compute.

avatar for Ilan Filonenko

Ilan Filonenko

Engineering Team Lead, Bloomberg
Ilan Filonenko is an Engineering Team Lead with the Enterprise Data Science Infrastructure team at Bloomberg, where he has designed and implemented distributed systems at both the application and infrastructure level. Previously, Ilan was an engineering consultant and technical lead... Read More →

Thursday May 23, 2019 15:55 - 16:30 CEST
Hall 8.0 C1