Handling the manufacturer id separately from the manufacturer data

[1technophile]

Currently the manufacturerdata string contains the manufacturer id, for example:

{"id":"F3","mac_type":1,"name":"F3","manufacturerdata":"330117540e101776666663f801370100","rssi":-85,"brand":"BlueMaestro","model":"TempoDisc","model_id":"BM_V23","tempc":30.9,"tempf":87.62,"hum":101.6,"dp":31.1,"volt":2.56}

Where nRF Connect gives:
Company 0x0133
Data 0x17540e101776666663f801370100

This is due to the first usage of the library with ESP32 but can cause issues with other implementations.

Do you think we should fix this and differentiate these 2 fields in the decoder and impact the client projects like OMG and Theengs Gateway?

[DigiH]

@1technophile if you remember the long discussion with the initial H505 decoder, when nRF Connect and even I/we thought that the beginning on the manufacturerdata was the company ID, but it turned out to be the ending part of the MAC address?

Unless doing an online check for 'proper' registered company IDs only I don't think this is a viable, secure idea. For decoder, nor for nRF Connect ;) EDIT, even with the online lookup it doesn't work, as in the below case!

#92

nRF Connect screenshot halfway down - can't seem to get an anchor for it ;(

[h2zero]

I think the issue is that the first 2 bytes of the manufacturerdata being the company identifier is NOT mandatory

Actually it is mandatory, that's why nRFconnect displays it that way.

Here's what the spec says:

1.4.1 Description The Manufacturer Specific data type is used for manufacturer specific data.
The first two data octets shall contain a company identifier from Assigned Numbers. The interpretation of any other octets within the data shall be defined by the manufacturer specified by the company identifier.

Unfortunately there are a lot of non-conforming devices out there because they don't want to pay for their own ID and try to keep their specs closed source.

[DigiH]

Actually it is mandatory, that's why nRF Connect displays it that way.

Ouch :( then that's not good that quite a few don't follow this. Not that I have any of their devices, but from a first look at their sites and devices I wouldn't have taken Inkbird or Govee for 'cheap' AliExpress-like devices ;)

For real world usage though this means that either only allow 'conforming' devices, or state that displaying the company identifier is not always a certain thing.

[h2zero]

What it does say about those devices is that they have not undergone the specifications testing and would fail if they did. This means that they cannot use the Bluetooth trademarks on their products.

They get away with it though because they are operating within their own ecosystem where their apps etc ignore this fault. Only people like us who are collecting data from these without knowing their specs would notice this.

[emericg]

The list of manufacturer is not easily extensible, I'm sure it's harder and takes more time to get into that official list than to develop an entire BLE device from scratch...
But the manufacturer field is mandatory to get a valid packet, even if the value is 0 or just garbage...
It's easy enough to bypass that problem and just use service data instead of manufacturer data though.

I'm using a desktop Bluetooth stack (all three BLE stacks actually), and the manufacturer data does not contain the manufacturer ID, which is usually an int16 you can get separately. That make it weird to re-make a big endian hexadecimal string (even saying it is weird :) representation of the manufacturer ID and put it before the actual data for theengs.

[h2zero]

Yes, it's up to the application programmer to use the correct ID in the manufacturer data. Not a problem for testing things or custom applications but commercial products should adhere to the specs, when they don't, we get conversations such as this lol.

Anyway, we should just ignore this data IMO.

[DigiH]

Also a bit confused now with situations like

https://community.openmqttgateway.com/t/is-it-possible-to-please-add-inkbird-ibt-6x-to-compatibility-list-for-openmqttgateway/1850

as a similar thing happened with the IBT_2X.

@1technophile do you have an IBT_2X and implemented the original decoder yourself with its leading 4 "0000" at the beginning of the manufacturerdata, or was the original 4 "0000" manufacturerdata user reported?

In the latter case it could be easily missed that the 'suposed' company identifier was actually another four "0000" before, making it a total of 8 "00000000" as is this current case with the IBT_6X(S).

So maybe there are only 8 "00000000" versions, but depending on where the manufacturerdata reports come from it can easily be mistaken for just 4 "0000".

Or there really are two different IBT_2X versions and IBT_6XS and IBT_6X differ there.

Just asking before I create similar conditions and underscore variants for the IBT_6XS decoder.

[1technophile]

@1technophile do you have an IBT_2X and implemented the original decoder yourself with its leading 4 "0000" at the beginning of the manufacturerdata, or was the original 4 "0000" manufacturerdata user reported?

It was reported by a user here with the 0000
And a different here with 00000000
So 2 different users, 2 different types of advertisements.... Seems that we need to handle both also for the 6XS

I have only an IBT-4XS.

[1technophile]

Maybe we just put the 2 models in the doc (.md) file and let the decoder with Inkbird brand and model. Thoughts?

[h2zero]

Cool! Why not just make the brand Inkbird/Solis? Then we can skip the docs and have minimal impact on size.

[1technophile]

I'm ok with this. Just wanted to reduce the payload size for esp.

[h2zero]

12 bytes is hopefully not a concern 😄, there are bigger fish to fry in that regard.

[DigiH]

I left the docs separately, as they are uniquely different hardware, both linking to the same merged decoder.

[DigiH]

Picking this up again with the new TheengsExplorer also showing the company names, as many other BLE scanning apps do, and my comment in the initial Explorer discussion.

#132 (reply in thread)

Since we pretty much know if a device adheres to the company ID convention or not when creating decoders. How about adding a simple static boolean flag to each decoder, something like "cidc" for "company ID compliant", so that every app using the Decoder library can easily decide to display a proper company name or not. This would be short from actually doing a company name lookup in Decoder itself, which would also require to keep the list updated.

Or adding

"mfid":{
         "decoder":["string_from_hex_data", "manufacturerdata", 0, 4]
      },

to all decoders, with non compliant ones having an empty or "FFFF" value.

Just an idea to get to grips with this issue in some way.

[koenvervloesem]

Good idea to try to distinguisgh these non-compliant devices, so tools such as Theengs Explorer can take this into account.

As non-compliant devices are the exception, I would just add a flag for those devices that are known to use the company ID for their values, to not overcomplicate all other decoders, no?

[DigiH]

As non-compliant devices are the exception

Not quite unfortunately ;) so I think it makes it even more important to decide on a solution in Decoder, so that any app using it can make confident decisions about displaying the manufacturer ID or not.

[DigiH]

So either

"mfid":{
         "decoder":["static_value", "FFFF"]
      },

to align with the official
This value is reserved as the default vendor ID when no Device ID service record is present in a remote device.

Leaving all compliant devices not sending a mfid, apart from iBeacons. Which again would make a double check necessary for any Decoder using app, for mfid AND IS "FFFF, while a separate

"cidc":{
         "decoder":["static_value", false]
      },

could allow only checking for the cidc existence, when only added to non-cidc decoders.

[DigiH]

Did a sample branch, also for testing and eventual implementation with Explorer, with a few obvious (cid is actual temperature) and non-registered sample decoders

https://github.com/DigiH/decoder/tree/cidc-introduction