Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Output of webserver file #2

Open
MediaVan opened this issue Sep 9, 2017 · 24 comments
Open

Output of webserver file #2

MediaVan opened this issue Sep 9, 2017 · 24 comments

Comments

@MediaVan
Copy link

MediaVan commented Sep 9, 2017

Hi, thanks a lot for these insights. I was looking for something like this since I want to log the data on my synology NAS system at home. Anyway, I noticed some difference in the data I'm receiving and therefor was wondering, where did you get the output file structure of? For example the data I'm receiving is 276 long and has a different header 687c51b0. Do you have any idea where to find documentation about the structure of the data?
Thanks again!
@graham0
Copy link
Owner

graham0 commented Sep 9, 2017

Hi MediaVan,

Thanks for getting in touch. The Synology NAS is a completely different animal. I decoded the info coming from a Solar inverter which is used to to convert the DC power from solar panels to AC power that can be used in the home. So the information I found was related to voltages amps and kilowatts etc. I would imagine the information from a NAS will be to do with processor usage, drive temperatures, memory usage etc.

I couldn't find any information to help me so had to backward engineer it myself. If you can't find anything to help decode the data stream the best approach is to look at it in a hex editor and try to spot known values. bare in mind the values may span multiple elements.

Grabbing multiple sets of data over a few minutes and comparing changes is a good way to start. once you spot a couple of values it starts to make more sense. Numbers that are different every time could be a system clock or uptime counter. If you spot values that could relate to disk space copy some more files to the NAS to see if the value changes.

It took quite a few hours to decode my hex dumps, I can't remember exactly how long but it can be a long process. I managed to decode enough of it to be able to use it, but there is also more data that I didn't manage to decode.

Hope this helps
graham0

@graham0 graham0 closed this as completed Sep 9, 2017
@MediaVan
Copy link
Author

hi graham0,

Thanks for your response. I am capturing the data from the solis solar inverter to.
I'm using the Synology to log all the data coming from this wifi stick : https://www.civicsolar.com/product/ginlong-solis-data-logging-wifi-stick-solis-dls-wifi

So I'm actually trying to accomplish the same thing as you. But using my Synology server with node.js etc. I was also planning to put the source code on github once I got it running. Your efforts are a great start from what I'm trying to do. 👍

Thanks for the tips, this helps me.

Regards,
Geoff

@graham0
Copy link
Owner

graham0 commented Sep 10, 2017

Hi Geoff,

Ah, sorry I misunderstood. Re-read your email and I realise what you meant now.
My setup was installed about 2 years ago so it's possible that Ginlong have changed the data format. The Wi-fi stck you have looks the same as mine. My inverter is a Solis 3.6kW (2nd generation) 240v single phase unit.

I can't log onto the inverter at the moment because it's dark and it goes off line. I will check the firmware version on the stick tomorrow, I think it's somewhere on the webpage. I also think there are two modes you can put it in, I think mine is in 'transparent' mode. Again I will check this tomorrow.

If you could check your equipment details and get back to me we may be able to work out the best approach to working out what has gone on.

Regards
Graham0

@graham0 graham0 reopened this Sep 11, 2017
@graham0
Copy link
Owner

graham0 commented Sep 11, 2017

Hi Again Geoff,

The wi-fi stick is running firmware: H4.01.44Y4.0.01W1.0.21(2014-12-231) and I was wrong about the working mode which is 'Data collection' and not 'Transparent'

It would be interesting to see if your firmware is a later version and if the available modes are the same.

Regards graham0

@MediaVan
Copy link
Author

Hi Graham0,

Here is the firmware version mine is running : H4.01.51Y4.0.02W1.0.57(GL17-07-261-D)

It has the two options: data collection and transparency. It's on data collection right now.

My inverter is the 3.2 4G.

Looking forward to be able to put some more time in "cracking the code". ;-)

Regards,
Geoff

@graham0
Copy link
Owner

graham0 commented Sep 12, 2017

Hi Again Geoff,

Yours is two generations newer than mine!
Get as many known values directly from the inverter and try to spot them in the data stream.
Let me know how it goes.
Good luck.

Once you get your data collection running you can start to display it and crunch the numbers, you can see my attempt here: https://www.whiteside.me.uk/solar/

Regards Graham0

@francsw
Copy link

francsw commented Feb 27, 2018

I have firmware version 4.01.51Y4.0.02W1.0.57(2017-07-261-D)
So far I managed to figure out the following:
Char 1-64 stays constant
31-60 is the Inverter serial no.
305-378 is the firmware version.
The bits inbetween I haven't been able to figure out.
Here is a line of my stick's output:
687c51b05edcca255edcca258103053031304142303137423137303030322001fb0d640a590000001c00220000004c00000000091a00000000137106ea00000000000c09060852000004920000000000030000beab041f007500000000000000000000000000000000000000001027140700e10b00d40d00ff010000000000000000000b0001000dc816682951b15edcca255edcca25800148342e30312e353159342e302e303257312e302e353728323031372d30372d3236312d44292c00aa16

@ashleysommer
Copy link

ashleysommer commented Mar 1, 2018 •
edited
Loading

hey guys, I came across this thread while googling for something different.

I can help out!

I reverse-engineered the newer (4G) Ginlong data format a couple of months ago.
From the header start byte (0x68) and the stop (0x16) and some other stuff in the payload, I was able to determine that it is a form of MBUS Protocol (specifically the LONG-form of the protocol), however most of the content between the start byte and stop byte are not MBUS compliant:

  • the Length field is one byte instead of two
  • the checksum is calculated in a different way than MBUS
  • there is no start2 byte (a 2nd 0x68 to mark the start of the payload), it just uses a fixed-length header instead.
  • The control field and address field are messed up or missing.

So the ginlong protocol needed to be manually reverse-engineered and unpacked.

They way I did it, was I downloaded the official ginlong android mobile app (the one you can use to live-monitor your ginlong device over wifi), and I decompiled the java .class files in the apk package.

I then combined what I found there, with what I know about industrial serialization formats.

I wrote a python script that monitors a tcp port, and every time it receives a packet, it deconstructs it, and pushes it to my pvoutput.org account. It also saves the values to a local daily .csv files, and zips the csv files into a .tar.gz every 7 days.

See the code in this gist here:
https://gist.github.com/ashleysommer/2e11f232abc5509243ea408d5a33dbc0

The magic is in the decode_payload function.

@graham0
Copy link
Owner

graham0 commented Mar 1, 2018 via email

@Richard-g8jvm
Copy link

Hi
I've recently had a solis inverter fitted , I can read the remote data with the android app and remote web site. but I would like to be able to monitor locally. I'm tottaly linux here, and do have a Hardkernel C2 SBC running ubuntu16.04 which is used as a TV kodi box and motion detection on 3 cameras.
I think it has spare capacity, I had a look at Grahams s/w , I would like to real time , or as close as possible monitor the system output. Is there a total noob, nerd, idiot guide lurking somewhere.
I'm in the UK , thanks sorry for going a tad off topic
Richard

@graham0
Copy link
Owner

graham0 commented Jun 5, 2018

Hi Richard,
My inverter is about 3 years old, known as a 2nd generation model. Ginlong have made quite a few changes so my version of the software will not work on the new inverters. If you have a 4th generation unit take a look at this project https://github.com/dpoulson/ginlong-mqtt.

With the wi-fi 'stick' installed data is only updated every 6 minutes. If you want real-time information you can remove the wi-fi 'stick' and connect directly to the inverter, and read the data directly, I've never attempted this myself. Other people have backward engineered the android app, but I don't know enough about Java to do that.

You could look at meter pulse counting. https://learn.openenergymonitor.org/electricity-monitoring/pulse-counting/introduction-to-pulse-counting?redirected=true I pulse count my electricity and gas. I found it reliable and instant so I installed one on my solar meter as well. I still read additional information using data from the inverter. Combining the data from gas , electricity and solar meters can give you an overall view of usage and consumption.

You can take a look at my efforts at monitoring here https://www.whiteside.me.uk/solar/

Hope this helps,

Graham

@Richard-g8jvm
Copy link

Hi Graham
Yes that does help, somewhere to start
many thanks

@RichardDL99
Copy link

Hi, I want to say thanks @ashleysommer for the code. Following that I looked at the UDP messages from my Ginlong Data Logging Stick, first found the frequency = 50.00, easy, and from that the instantaneous power. The update is every 6 minutes, but I'd really like it faster. I guess there are more options with 'Transparent Mode' but I can't find any information about it? Anyone know?

@RobertSmart
Copy link

you can pull any data you want via modbus over the RS485 port. You can do it with a raspberry pi and a usb-rs485 converter that costs a couple of quid.

@RichardDL99
Copy link

I only have WiFi interface on the Data Logging Stick. (I've used Serial RS232 and RS485 many times in other places.) I can select 'Transparent Mode' with either UDP or TCP, but what enquiry does it need?

@Richard-g8jvm
Copy link

Richard-g8jvm commented Jun 28, 2019 via email

@ashleysommer
Copy link

I've tried to use Transparent mode and never been able to get it to work.
I think you'd need to put T-line between the inverter and the wifi stick and sniff the traffic between them. It is RS485 on that port. In normal mode the wifi stick will issue the requests to the inverter, so if you can capture and decode them, I think you should be able to put it in Transparent mode and use the same requests.

@RichardDL99
Copy link

Oh that's interesting. I've had the cover off the DLS (just push in the 4 plastic hooks on the circular part of the case). In there it will be quite easy to connect something. There are 4 wires, so logically that would be +5V, 0V, TX, RX. RS485 uses 4 wires where I come from, unless it's alternate direction on a pair of wires, never usd that option. I'm hoping it's TTL serial. Don't know when I'm going to try this.

@RobertSmart
Copy link

We started by using modbus through the same port the wifi stick plus into. just run those cables into the A/B of the RS485 converter, and you can just poll it like a normal modbus register. If you use node-red you can pull whatever value you want. It can't do much faster than 2 second intervals though. any faster and it stops being very reliable.

We are just experimenting with using the data stick for another project, hence why we're not just sticking with the modbus approach. Has anyone actually successfully decoded the transparent mode messages?

@RichardDL99
Copy link

Hello again,

I’ve done some serial testing as I said in my last post, the results are not very encouraging. Using '9600,8,1,None' what appeared on putty on the request side was:

            P0'­¹

Á(110AM$¨©4¾mj103 192.168.1.51'¡¢~¡PÏ
P0'¢
¡¢¡PÒF'¨©t¾mj1

And on the response side:

Òdî~(¶þòVéþþºÖ¾<¾¾ù>þþþþþþþþ¾b¢¸þâÞþ~ùþîöþþþþþþþþþþþúþʺüúz|É¢Ï;Æ´:þ^ú^4þþòþþþZ
ÖdþûVúþþºÖ¾<¾¾þ>þþþþþþþþ¾bªüþòÞþiþîöþþþþþþþþþþþúþÊþúz|ÙQÏ纴:þ^ú^4þþòþþþZ

That's two bursts from each side.

I tried a few other baud rates and parity options, but everyting seemed to make it worse. (That IP address is its IP address, but why would the stick send that to the Inverter? - maybe it's in one of the diag displays in the inverter.)

Maybe if I read it as bytes into a programming language I could work out what's going on. It's not a very attractive route.

I tried TCP mode, The DLS connected to my PC each 6 minutes. That's too infrequent so I didn't bother to look at the message.

The power generation meter has a LED that flashes every 0.1kWh. I've built a light-dependant-resistor + Arduino that times the pulses and this looks like a more predictable development path.

@graham0
Copy link
Owner

graham0 commented Oct 19, 2019

Hi RichardDL99,
This is the sort of output that you get unfortunately. If you think you have a valid set of data, the next stage is to analyze it.
Grab a few sets of data over a couple minutes then load them into a hex editor. Investigate the values that change to see if any match known values from the inverter display. Most of the values use 2 consecutive hex values so a bit of maths may be required. That's the way I did it. It takes a while but becomes easier as you find more values.

I also went down the route of pulse counting. Initially to my mains electricity then eventually the gas and solar to complete the picture. The meter is a more accurate measure as the inverter rounds values down.

I used raspberry pi and some photo transistors to read the pulses on the electricity and solar, and some hall effect devices to read the gas meter pulses.

Good luck
Graham0

@RobertSmart
Copy link

Why not use modbus? you can poll it every 2 seconds for real time data. It doesn't require a huge investment, a little adapter board from ebay for a few dollars will do the trick. I can give you the modbus table registers for any data your looking for.

@craigcurtin-dev
Copy link

Guys,

A couple of things here - if you have the wifi stick you can setup a 2nd server to output the data to - the secret sauce if to go to

http://your_IP_for_YOUR_SOLIS/config_hide.html - in there you can set the 2nd server to send data to - more importantly you can change the mode to be TCP Client so it will attach - i am just testing this out now (just got my 2 x Solis up and running today) and have them outputting to this project https://github.com/XtheOne/Inverter-Data-Logger/

I have just changed the TCP timeout to be 60 instead of the default 300 but it looks like it ignores that. I am going to continue to play around (my inverters have shutdown for the night) and see what other files are in there to call

Craig

@NibblyPig
Copy link

Does anyone have any updates on this?

I've been trying to get data out of the darn wifi stick with little success. The thing seems to work for a bit (30-60 seconds) then completely packs up pretty much until I reboot it. All I'm doing is proxying the connection between it and the chinese servers, even though I've figured out from various posts how to emulate what the server is doing.

There's suggestions about plugging a pi or whatever into the port but it uses a non-standard connector and you'd still need to know the protocol to talk to the inverter presumably. It's probably a bit much for me to manage that kind of task.

I wish the darn thing was just a bit more reliable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@ashleysommer @MediaVan @NibblyPig @francsw @RobertSmart @graham0 @RichardDL99 @Richard-g8jvm @craigcurtin-dev