# What is the Data Services Hub (DSH)? The DSH is a <!-- .element: class="fragment" data-fragment-index="1" --> Streaming Data <!-- .element: class="fragment" data-fragment-index="2" --> Platform <!-- .element: class="fragment" data-fragment-index="2" -->

## What is a “platform”? - Something you can build (applications) on. <!-- .element: class="fragment"--> - It provides reusable infrastructure. <!-- .element: class="fragment"--> - It takes care of recurring and tedious tasks. <!-- .element: class="fragment"--> - It shouldn’t not hamper creativity. <!-- .element: class="fragment"-->

## What is “streaming data”? > … data that is generated continuously by many independent data > sources. Typically, this data is of small size (order of kilobytes). <ul> <li>Data is <strong>streaming</strong> if it is produced and transmitted <strong>without delays or batching</strong>.</li><!-- .element: class="fragment"--> <li>The smallest unit of streamed data is called a <strong>message</strong> or <strong>event</strong>.</li><!-- .element: class="fragment"--> <li>Streaming data is sorted, collected, or aggregated in a <strong>stream</strong> or a <strong>topic</strong>.</li><!-- .element: class="fragment"--> </ul>

## Types of streaming data Not all datastreams are created equal: <img src="./images/figures/leaky-tap.gif" style="width: 40%" /> <img src="./images/figures/spillway.gif" style="width: 40%" />

## What should a streaming data platform be able to do? - Handle hundreds of thousands of sources. <!-- .element: class="fragment"--> - Send data to hundreds of thousands of sinks. <!-- .element: class="fragment"--> - Process data: clean, refine, aggregate, combine… <!-- .element: class="fragment"--> - Share data streams with other parties. <!-- .element: class="fragment"--> - Do all of this, with high security standards. <!-- .element: class="fragment" -->

## Key concepts - Scalable <!-- .element: class="fragment"--> - Data as low-latency events (streams) <!-- .element: class="fragment"--> - Real-time processing <!-- .element: class="fragment"--> - Data sharing <!-- .element: class="fragment"--> - Secure <!-- .element: class="fragment"-->

## Types of streaming data, for the DSH <table> <tr> <td style="text-align: center"><img src="./images/figures/leaky-tap.gif" style="width: 100%" /></td> <td style="text-align: center"><img src="./images/figures/spillway.gif" style="width: 100%" /></td> </tr> <tr> <td style="text-align: center"><p>Many sources, low volume<br/>Single sensor<br /><strong>MQTT</strong></em></p></td> <td style="text-align: center"><p style="text-align: center">Few sources, high volume<br />Processing of streams<br /><strong>Kafka</strong></p></td> </tr> </table>

# How does the DSH work? Let’s go through the DSH one step at a time.

## The DSH step by step The Kafka cluster is the core of messaging in the DSH. The MQTT adapter allows easy input and output of data. <img src="images/figures/dsh-step-1.png" class="kpn-figure" />

## The DSH step by step As ‘Tenant A’, you receive an isolated environment on the DSH. <img src="images/figures/dsh-step-2.png" class="kpn-figure" />

## The DSH step by step The DSH manages your access to Kafka topics and DSH streams via an Access Control List (ACL). <img src="images/figures/dsh-step-3.png" class="kpn-figure" />

## The DSH step by step As a tenant, you can deploy applications (containers) in your isolated environment. <img src="images/figures/dsh-step-4.png" class="kpn-figure" />

## The DSH step by step You also have access to resources, which are assigned specific limits. <img src="images/figures/dsh-step-5.png" class="kpn-figure" />

## The DSH step by step The DSH offers services for monitoring, tracing, Public Key Infrastructure etc. <img src="images/figures/dsh-step-6.png" class="kpn-figure" />

## The DSH step by step The DSH offers ways to interact with Kafka easily, such as the Kafka Proxy and adapter services. <img src="images/figures/dsh-step-7.png" class="kpn-figure" />

# Data flow of the DSH <br><br> How does the data flow to, on and from the DSH?

## Data flow of the DSH Data sources and data sinks connect to the DSH via MQTT bridges. <img src="images/figures/dsh-data-flow-1.png" class="kpn-figure" />

## Data flow of the DSH The DSH stores messages from data sources on a so-called public stream. Data sinks consume from this public stream. <img src="images/figures/dsh-data-flow-2.png" class="kpn-figure" />

## Data flow of the DSH A stream is a collection of Kafka topics, and the DSH manages access to the stream. For example, the DSH gives Tenant A permission to consume from the public stream. <img src="images/figures/dsh-data-flow-3.png" class="kpn-figure" />

## Data flow of the DSH Tenant A processes the data, and produces new data. However, Tenant A wants this data to be accessible on the DSH only. <img src="images/figures/dsh-data-flow-4.png" class="kpn-figure" />

## Data flow of the DSH Tenant A publishes the new data on a so-called internal stream. Tenant A can give other tenants permission to consume from this internal stream. <img src="images/figures/dsh-data-flow-5.png" class="kpn-figure" />

## Data flow of the DSH For example, Tenant B has permission to consume from the internal stream. However, Tenant C can’t even see this stream. <img src="images/figures/dsh-data-flow-6.png" class="kpn-figure" />

## Data flow of the DSH Tenant A produces a separate set of data. Only Tenant A should be able to access these data, and no-one else. <img src="images/figures/dsh-data-flow-7.png" class="kpn-figure" />

## Data flow of the DSH Tenant A writes the data to a private Kafka topic. Only Tenant A can access this topic, and the other tenants can’t even see it. <img src="images/figures/dsh-data-flow-8.png" class="kpn-figure" />

## Data flow of the DSH This is only one example of the many possible data flows in the DSH. Data isn’t shared by default, and even MQTT access is on an explicit opt-in basis. <img src="images/figures/dsh-data-flow-9.png" class="kpn-figure" />

## Stream Processing > … is the processing of data in motion, or in other words, > computing on data directly as it is produced or received. https://docs.risingwave.com/reference/key-concepts#stream-processing

## Where to process The data source doesn’t have the resources to process the data, and the data sink can’t handle the huge number of data sources. The DSH can act as an intermediate to receive the data from the source, process it in real time, and send it on to the sink. <img src="images/figures/dsh-data-flow-10.png" class="kpn-figure" />

## Many ways to process the data - There are many frameworks for stream processing. - However, no framework fits all use cases. - Luckily, the DSH doesn’t dictate a framework: you can bring your own. No one framework to rule them all, but the DSH to bind them.

## Sharing securely - Tenants are completely separated, using Calico. <!-- .element: class="fragment"--> - The DSH enforces the correct use of Docker containers. <!-- .element: class="fragment"--> - Tenant containers use certificates to authenticate towards Kafka.<!-- .element: class="fragment"-->