Parsing dynamic data

[NateEaton]

I have a device that's likely not something widely used (SensorBug, reports temperature, light level and accelerometer data) but I've previously adopted other code to add a worker for it to bt-mqtt-gateway and now I'm in the process of switching over to Theengs Gateway so I'm looking into creating a decoder for it.

I've already drafted a json structure that includes property for battery level. For the other three, one challenge is that while they are included in the manufacturer data, it's a dynamic structure: three properties but can be in any order so for a given property (e.g., light) it takes checking the code in first position and if it indicates that property, decode using the next few bytes but otherwise checking further into the dynamic data block, iterating till finding the code for that property.

It's complicated a bit because for one property the data can be either 2 or 3 bytes so there is another check to do before decoding.

Just thought before I charge ahead and risk ending up thrashing about for days on this I would check in for suggestions.

Here's the spec for the device:
http://www.blehome.com/SensorBug%20Interface%20Specification%20v1.3.0.0a.pdf

Also, a question on creating a decoder: is there a specific tool/technique for going from raw json to the c++ const char* with escaped quotes?

[koenvervloesem]

Currently Theengs Decoder isn't able to specify how to decode dynamic structures. So we would probably need some changes in Theengs Decoder before a SensorBug decoder can be implemented. We have the same issue with the BTHome format.

Maybe you could work around it by trying to check all possible orders of the three properties, but I guess that would make the decoder quite complex.

[DigiH]

I agree, with the current Decoder capabilities this would not be that easy and straightforward. Looking at the device and all its possibilities, among them also being able to have it broadcast in iBeacon, Eddystone and AltBeacon protocols, it reminds me a lot of the Puck.JS another device that through its versatility and the resulting dynamic data was not a real candidate for integration.

Having a private decoder for yourself though is also an advantage, as I'm certain that you always have the same services turned on, so the possible variations would already be less, even if the data would be sent differently every time.

@koenvervloesem idea to cover all possible combinations is likely the best approach to get this started.

Also, a question on creating a decoder: is there a specific tool/technique for going from raw json to the c++ const char* with escaped quotes?

Some script on your preferred platform will do the trick, where it needs to
• delete all returns
• delete all tabs (if present at all) and spaces (apart from spaces within quotes)
• escape all quotation marks

[NateEaton]

Thanks. Yes, in fact to start with I will build the decoder to just handle static data as it does appear that the key/value pairs are always in sequence (41, 42, 43) but I'd figured that for a decoder that anyone could use it would have to handle the dynamic nature of the data described in the spec.

[NateEaton]

Also, a question on creating a decoder: is there a specific tool/technique for going from raw json to the c++ const char* with escaped quotes?

Some script on your preferred platform will do the trick, where it needs to • delete all returns • delete all tabs (if present at all) and spaces (apart from spaces within quotes) • escape all quotation marks

Just thought I would share...

After spending enough time manually updating the c++ code to reflect revised json code as I modified existing decoder definitions and created a new one, I figured I would write something to simplify that step.

However, in part because my python skills are modest (I used many languages over the course of my career but have only used python for personal use) but also because I have been spending a lot of time following and exploring advances in AI, I decided to see what ChatGPT could do to help. In fact, it was a huge help, creating code that didn't work as provided but was close enough that it didn't take me long to extend it to something that does work.

I've added two files to my decoder fork under scripts:

• decoder_prep_raw.py - this has the instructions I gave ChatGPT and the code it produced
• decoder_prep.py - this has the working script I created building on what ChatGPT provided

NateEaton : /decoder/scripts

I thought this might be of interest both because of its potential usefulness to others involved in writing decoder device definitions and also as an interesting example of the power of AI, not from the standpoint of it replacing us but as a tool that can help accelerate our work as programmers. It does have issues (including adding functionally incorrect code - see comments in the scripts) but the work to adapt and use what it produces isn't that different (in my experience) from a practice many of us already use, copying and adapting code written by other programmers.

[DigiH]

Hi @NateEaton,

very interesting approach, I'll have to have a closer look at these.

Thanks :)

[koenvervloesem]

Thanks @NateEaton I implemented this now, based on your script above, as a script, pre-commit hook and GitHub Actions job in the decoder repository: #380

[NateEaton]

Glad to be able to help!

…

On Thu, Jun 29, 2023 at 10:16 AM Koen Vervloesem ***@***.***> wrote: Thanks @NateEaton <https://github.com/NateEaton> I implemented this now, based on your script above, as a script, pre-commit hook and GitHub Actions job in the decoder repository: #380 <#380> — Reply to this email directly, view it on GitHub <#255 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABNGQYOSLFFG2LUTXOYC2TXNWL45ANCNFSM6AAAAAATWD52DA> . You are receiving this because you were mentioned.Message ID: ***@***.***>