The Zwicky Transient Facility Alert Distribution System. (arXiv:1902.02227v1 [astro-ph.IM])
<a href="http://arxiv.org/find/astro-ph/1/au:+Patterson_M/0/1/0/all/0/1">Maria T. Patterson</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Bellm_E/0/1/0/all/0/1">Eric C. Bellm</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Rusholme_B/0/1/0/all/0/1">Ben Rusholme</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Masci_F/0/1/0/all/0/1">Frank J. Masci</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Juric_M/0/1/0/all/0/1">Mario Juric</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Krughoff_K/0/1/0/all/0/1">K. Simon Krughoff</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Golkhou_V/0/1/0/all/0/1">V. Zach Golkhou</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Graham_M/0/1/0/all/0/1">Matthew J. Graham</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Kulkarni_S/0/1/0/all/0/1">Shrinivas R. Kulkarni</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Helou_G/0/1/0/all/0/1">George Helou</a>
The Zwicky Transient Facility (ZTF) survey generates real-time alerts for
optical transients, variables, and moving objects discovered in its wide-field
survey. We describe the ZTF alert stream distribution and processing
(filtering) system. The system uses existing open-source technologies developed
in industry: Kafka, a real-time streaming platform, and Avro, a binary
serialization format. The technologies used in this system provide a number of
advantages for the ZTF use case, including (1) built-in replication,
scalability, and stream rewind for the distribution mechanism; (2) structured
messages with strictly enforced schemas and dynamic typing for fast parsing;
and (3) a Python-based stream processing interface that is similar to batch for
a familiar and user-friendly plug-in filter system, all in a modular, primarily
containerized system. The production deployment has successfully supported
streaming up to 1.2 million alerts or roughly 70 GB of data per night, with
each alert available to a consumer within about 10 s of alert candidate
production. Data transfer rates of about 80,000 alerts/minute have been
observed. In this paper, we discuss this alert distribution and processing
system, the design motivations for the technology choices for the framework,
performance in production, and how this system may be generally suitable for
other alert stream use cases, including the upcoming Large Synoptic Survey
Telescope.
The Zwicky Transient Facility (ZTF) survey generates real-time alerts for
optical transients, variables, and moving objects discovered in its wide-field
survey. We describe the ZTF alert stream distribution and processing
(filtering) system. The system uses existing open-source technologies developed
in industry: Kafka, a real-time streaming platform, and Avro, a binary
serialization format. The technologies used in this system provide a number of
advantages for the ZTF use case, including (1) built-in replication,
scalability, and stream rewind for the distribution mechanism; (2) structured
messages with strictly enforced schemas and dynamic typing for fast parsing;
and (3) a Python-based stream processing interface that is similar to batch for
a familiar and user-friendly plug-in filter system, all in a modular, primarily
containerized system. The production deployment has successfully supported
streaming up to 1.2 million alerts or roughly 70 GB of data per night, with
each alert available to a consumer within about 10 s of alert candidate
production. Data transfer rates of about 80,000 alerts/minute have been
observed. In this paper, we discuss this alert distribution and processing
system, the design motivations for the technology choices for the framework,
performance in production, and how this system may be generally suitable for
other alert stream use cases, including the upcoming Large Synoptic Survey
Telescope.
http://arxiv.org/icons/sfx.gif
