material view + replaying streams

[mntnclmbr]

Hi,

Here what I am trying :

1. We will have a lot of IoT devices( millions ) connecting to our backend
2. When a device connects to our backend, it will create a stream with the device serial number as the "topic".
3. Each device will push say 1 msg every 5 minute with timeseries data and the message will get routed to the stream for the device created in # 2 above.
4. As part of # 2 , we will create material views on each stream to capture the last message and also some avg values that will be calculated as the data comes in.
5. Our internal dashboards can then query these material views when we need to show the data for each individual device without having to wait for the actual query to process every time we need information.
6. We do also want to have the ability to "zoom" in to a specific time slice , say 1 hour long and see how the data changed over time. For this we would have to replay the stream for that time slice, potentially going back in time where the data may not be on the tier-1 storage anymore.

So the questions that I am trying to answer are:

1. Does the above make sense in general (specifically # 2-5 above )or am I making bad design decisions based on my limited understanding ?
2. for # 6 above, i have no clue what to do here and if this is even possible ?
   a. Can streams be replayed for a given time slice ? Also, how do we retrieve data from tier-2 storage ?

Appreciate any pointers I can get and apologies in advance if I am asking stupid questions.

[Commelina]

Thanks for your asking!

1. Yes, the design is reasonable. In fact, HStream is specifically designed for such scenes (e.g. very large number of streams).
2. It is a part of our future design to support queries on time series data. As for the current version, maybe you'd like to create a new subscription whose offset is EARLIEST or an estimated value and read from it.