Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MEDM not responsive after acquiring image in "continuous" mode and Mono12Packed Pixel format #4

Open
LeeYangLBLBCS opened this issue Jul 28, 2020 · 106 comments
Open

MEDM not responsive after acquiring image in "continuous" mode and Mono12Packed Pixel format #4

LeeYangLBLBCS opened this issue Jul 28, 2020 · 106 comments

Comments

@LeeYangLBLBCS
Copy link

issue moved from:
epics-extensions/medm#6
@LeeYangLBLBCS LeeYangLBLBCS mentioned this issue Jul 28, 2020
Closed
@MarkRivers
Copy link
Member

Is this issue still happening? Can you describe the symptoms in more detail? Are there any error messages on the IOC console when this happens?

@LeeYangLBLBCS
Copy link
Author

yes, this is still happening.
I just recreated it with the "new" computer without any problems.
There is no error on IOC terminal, except it does not respond to any command. The only way to exit is ctrl-C.
It happens in continuous mode fairly quickly.
It also happens in multiple image mode when the number of images is large enough, such as 1000 images.
It did not happen in multiple image mode with up to a few hundred images.
screen attached shows multiple image mode with 1000 images, where the "acquiring" is frozen at image number 993.
image

@MarkRivers
Copy link
Member

There is no error on IOC terminal, except it does not respond to any command. The only way to exit is ctrl-C.

Do you mean that if you press on the keyboard you don't get a new iocsh prompt? If so, that it is a problem I have never seen, and it sounds like it could be a resource depletion issue.

Please post a screen shot of the Plugins/All screen when the system has collected about 900 images in Multiple mode, or when it has collected more than 1000 images in Continuous mode.

@MarkRivers
Copy link
Member

@LeeYangLBLBCS how much memory does your computer have, and how many cores?

@LeeYangLBLBCS
Copy link
Author

correction: the IOC responds to command input.
I checked epicsMutexShowAll 1, the results show 0:
epics> epicsMutexShowAll 1
ellCount(&mutexList) 37822 ellCount(&freeList) 0

However, I found that It also happens in mono16 pixel format.
image
image

More information:
In Mono12Packed mode, if I stop acquisition before it reaches 1000 images, usually it will stop normally after a few more frames taken.
If I click stop more than once, it will usually show "acquiring" permanently.
At this time, I see two mutex in IOC terminal:
epics> epicsMutexShowAll 1
ellCount(&mutexList) 37719 ellCount(&freeList) 0
epicsMutexId 0x55d7011c5970 source ../../asyn/asynPortDriver/asynPortDriver.cpp line 3700
epicsMutexId 0x55d7011c7c20 source ../../asyn/asynDriver/asynManager.c line 1975
epics>

@LeeYangLBLBCS
Copy link
Author

LeeYangLBLBCS commented Aug 27, 2020 via email

@LeeYangLBLBCS
Copy link
Author

LeeYangLBLBCS commented Aug 27, 2020 via email

@MarkRivers
Copy link
Member

I want to be clear on exactly what symptoms you are seeing.

If you configure your system as I show below:
GC_AdcBitDepth=Bit10
PixelFormat=Mono12Packed
ConvertPixelFormat=Mono16
FrameRate=105 (actual 104.984)
ImageMode=Multiple
NumImages=1000

image
image

If you acquire in this mode does it ever fail to acquire 1000 images? It does not for me.

However, it you try to stop the acquisition before it completes normally, does it sometimes get into this state where it is stuck with Acquire=1 (says Collecting)?

image

That is the problem I see.

At that point epicsMutexShowAll 1 does show a deadlock:

epics> epicsMutexShowAll 1
ellCount(&mutexList) 38759 ellCount(&freeList) 0
epicsMutexId 0x556e97136c80 source ../../asyn/asynPortDriver/asynPortDriver.cpp line 3712
epicsMutexId 0x556e97138950 source ../../asyn/asynDriver/asynManager.c line 1986

Typing "exit" at the iocsh prompt does not work, I have to press ^C.

The same thing happens in Continuous mode. When I press Stop it often hangs.

@MarkRivers
Copy link
Member

Maybe it could be the motherboard?

I think I am a seeing the same problem as you. It is a bug in my driver. I am working on it now. I will let you know when I think I have it fixed.

@LeeYangLBLBCS
Copy link
Author

LeeYangLBLBCS commented Aug 27, 2020 via email

@MarkRivers
Copy link
Member

yes, I see exactly what you saw.
Plus, it happens in Mono16 format also. It never can reach 1000 images. But
if I can always stop it without problems, either before stuck acquiring at
the end, or in the middle of acquiring.

I am confused by what you are saying. In my case if I set Multiple mode and NumImages=1000 it always acquires 1000 images with no problem. You seem to be saying that it never collects all 1000 images?

My only problem is trying to stop it manually by pressing Stop, either in Multiple mode or Continuous mode.

@MarkRivers
Copy link
Member

I just tried PixelFormat=Mono16 and ConvertPixelFormat=None. I see a few frames are dropped with these messages:

2020/08/27 13:31:56.391 ADSpinnaker::grabImage error GetImageStatus returned 9
2020/08/27 13:31:56.393 ADSpinnaker::grabImage error GetImageStatus returned 9

In this case 2 frames had missing packets, and so it only collected 998 frames, not 1000. That is "normal". When I pressed Stop it changed back to the Idle state and I could collect again.

@LeeYangLBLBCS
Copy link
Author

LeeYangLBLBCS commented Aug 27, 2020 via email

@MarkRivers
Copy link
Member

Yes, for some reason the maximum frame rate it not updating as it should for Mono16 and Mono12Packed.

I have isolated the problem to what appears to be a bug in their SDK. When I call pCamera->EndAcquisition() when acquiring Mono12Packed and converting to Mono16 that call hangs forever. There are some comments in the code indicating that I have seen this before.

ADSpinnaker is currently using Spinnaker 1.20.0.14. That is quite old now, and I am going to try the latest version, 2.0.0.147 and see if we still have the problem.

@MarkRivers
Copy link
Member

I have created a new SDK_2.0 branch. It uses the most recent Spinnaker SDK from FLIR (2.0.0.147).

It seems to fix the problem of stopping acquistion hanging the IOC with Acquire=1 when using PixelFormat=Mono12Packed and ConvertPixelFormat=Mono16. I can now stop acquistion in Multiple or Continuous mode with those parameters with no problem now, it never hangs.

It seems like it might drop a few more frames compared to the older SDK with the Oryx camera, but I need to test this more to be sure. @LeeYangLBLBCS I would be interested in you testing to see if this fixes the problem for you, and whether you see more dropped frames. By dropped frames I mean Multiple mode asking for 1000 frames and seeing it only collect 995, etc.

@LeeYangLBLBCS
Copy link
Author

LeeYangLBLBCS commented Aug 27, 2020 via email

@LeeYangLBLBCS
Copy link
Author

LeeYangLBLBCS commented Aug 28, 2020 via email

@MarkRivers
Copy link
Member

If you use Multiple mode and 1000 images does it collect exactly 1000 images. Try 8, 12, 16 bit, fixed rate and free running.

@LeeYangLBLBCS
Copy link
Author

All pixel formats, mono8/16/12pack/12p work correctly in "fixed frame" rate.
All 1000 images collected and acquisition stopped any time.

For "non fixed frame rate", some pixel format can't be stopped. Though they all finish 1000 frames correctly.
Among them,mono8/mono16 worked (ADC=8/10/12).
mono12pack/monop (ADC =8/10), acquired 1000 images correctly but can't be stopped.
momo12pack/monop in ADC=12bit worked fine in all cases.

Apparently the non fixed frame rates are programmed to use higher values that cause problems. Are they calculated in the SDK based on some kind of theoretical settings? I believe they are not going to be correct at all times for all systems.

We acquire images on a fixed interval triggered by motor output pulses at a steady rate, so this will not be the problem for our application. (We always have to calibrate the highest frame rate achievable right before the tomography run.)

image

@LeeYangLBLBCS
Copy link
Author

LeeYangLBLBCS commented Aug 28, 2020 via email

@MarkRivers
Copy link
Member

Which should allow more than 65535 number of images.
Is this limit 65535 caused by MEDM?

No, this must be a limit enforced by the camera firmware. The GenICam specification (https://www.emva.org/wp-content/uploads/GenICam_SFNC_2_3.pdf page 166) says that AcquisitionFrameCount is an IInteger, which is actually a 64-bit integer. But vendors are free to limit the values to a specific range. In ADSpinnaker if I write 65536 to NumImages (which is the AcquisitionFrameCount feature) the value then read back from the camera is 65535, indicating it won't allow more than that.

@LeeYangLBLBCS
Copy link
Author

LeeYangLBLBCS commented Aug 28, 2020 via email

@MarkRivers
Copy link
Member

Interestingly I cannot reproduce your result of failing to stop. Here are some results I obtain when FrameRateEnable=No and AcquireTime=0.001.

Pixel format ConvertPixelFormat ADC bits FLIR specified frame rate Actual frame rate # Images collected Can stop?
Mono8 None 8 162 162 991-993 Yes
Mono12Packed Mono16 8 122 120 978-983 Yes
Mono16 None 8 81 79 965-966 Yes
Mono8 None 10 144 144 996-997 Yes
Mono12Packed Mono16 10 130 130 983-984 Yes
Mono16 None 10 85 84 978-979 Yes
Mono8 None 12 89 89 1000 Yes
Mono12Packed Mono16 12 89 89 1000 Yes
Mono16 None 12 89 89 989-1000 Yes

In the Mono12Packed mode there is up to 2 second delay between pressing Stop and it actually stopping. I don't see the problem you do of not being able to stop it all. I think the problem is that the time to convert the pixel format from Mono12Packed to Mono16. Currently the mutex is locked during this operation, which is probably preventing the Stop command from getting enough time to execute. I will try changing that and see if it fixes your problem.

@MarkRivers
Copy link
Member

Once in a while we do TIMBIR scan (Time Interlaced Model-Based Iterative
Reconstruction), which can get close to this number, though we are limited
by our current motor controller first (~13k position array), at this time.

I think you can work around that problem by using Continuous mode rather than Multiple. Once you have collected the correct number of images your software just stops the camera.

@MarkRivers
Copy link
Member

MarkRivers commented Aug 28, 2020 •
edited
Loading

We always have to calibrate the highest frame rate achievable right before the tomography run.

You should not need to do that. I also trigger from motor pulses, and I adjust the motor speed to generate the next trigger as close as possible to previous image readout being completed. I don't do that empirically for each run, I have a table for each camera, PixelFormat, ADC bits, etc. I recently converted my software from IDL to Python, and it is here:

Code:
https://github.com/tomography/tomoscan

That code for the readout time is here:
https://github.com/tomography/tomoscan/blob/d997a247407640b96d7ce9d124ab432f274b9254/tomoscan/tomoscan.py#L748-L790

Documentation:
https://tomoscan.readthedocs.io/en/latest/

@LeeYangLBLBCS
Copy link
Author

LeeYangLBLBCS commented Aug 28, 2020 via email

@MarkRivers
Copy link
Member

There is no need to worry about the network traffic for file saving with areaDetector. If the file saving cannot keep up with the detector then you just set the queue size for the file plugin to be large, up to the number of images you are collecting. That way the detector runs at its maximum speed and the files to be written will be queued, but they will eventually all be written. That way the detector sets the acquisition speed (time resolution of measurement) while the disk/network speed sets the time between datasets.

@MarkRivers
Copy link
Member

@LeeYangLBLBCS I have pushed a change that may fix the problem that you cannot stop acquisition with Mono12Packed. I am not sure it is safe, I did observe it to lock up once. It seems to reduce the delay for me between pressing Stop and when it actually stops.

My suspicion is that your system is a little less powerful and the mutex is never unlocked long enough to allow the Stop command to be processed. Please try the master branch.

@LeeYangLBLBCS
Copy link
Author

LeeYangLBLBCS commented Aug 28, 2020 via email

@decarlof
Copy link

decarlof commented Sep 29, 2020 •
edited
Loading

Also, when you updated from Ubuntu to RH8 did you increase the system settings described here:
https://areadetector.github.io/master/ADGenICam/ADGenICam.html#linux-usb-and-gige-system-settings

@MarkRivers IT had to set other parameters to get the FLIR to work correctly. I am not sure if these are FLIR specific but if not probably should be added to the ADGenICam docs. Here they are:

  1. Prerequisites:
    64GB memory
    Cat 6A cable
    Intel X550T2 ETHERNET CONVERGED Network Adapter X550-T2
  2. Enable jumbo packet
  3. Disable DHCP and set a fixed IP address on the Ethernet port connecting to the FLIR
  4. Increase the receive buffer size
  5. Set the NIC tx queue length

(2. 3.). are documented at:
FLIR doc: https://www.flir.com/support-center/iis/machine-vision/knowledge-base/lost-ethernet-data-packets-on-linux-systems/

(4. already documented) edit /etc/sysctl.conf and add:

net.core.rmem_default=26214400
net.core.rmem_max=268435456

(5.) edit /etc/rc.local and add:

#NIC camera settings and  10GB nic settings  In this example the camera is attached to  ens1f1    
/usr/sbin/ifconfig ens1f1 txqueuelen 3000

@MarkRivers
Copy link
Member

MarkRivers commented Oct 3, 2020 •
edited
Loading

I have released ADSpinnaker R3-1 and also ADAravis R2-1.

The new version of ADSpinnaker allows increasing the number of buffers used in the Spinnaker library. It also has much improved statistics reporting on the transport layer to aid in diagnosing problems. This is a screen shot when running the new version with the ORX-10G-51S5M in 10-bit mode, Mono12Packed, 130 frames/s.

image

Note that the input buffers is 77. I specified a total of 100 buffers, whereas in the previous release the value was fixed at the Spinnaker default of 10. This means that 23 buffers are currently in use, because the thread that reads the camera frames and does the Mono12Packed to Mono16 conversion is barely keeping up. That can also be seen with top -H

image

Note that the thread called ADSpinnakerImag is using 99.9% of a core. That is the thread the reads the images and does the conversion. On my 3.7 GHz machine that thread just barely keeps up, and I don't drop any frames as long I have 100 buffers.

This is a screen shot of ADAravis running the Oryx cameras with the same settings as above.

image

This is top -H when adAravis is running:
image

In this case aravisPoll is the thread that is reading the images and doing the Mono12Packed to Mono16 conversion, using functions that I added in ADGenICam R1-6. It is only using 65.5% of a core, compared to 99.9 in ADSpinnaker.

@decarlof this means that ADAravis may be able to keep up with the Oryx 10-bit, Mono12Packed on your machine. An advantage of ADAravis over ADSpinnaker is that it runs on RHEL7, it does not require RHEL8 or Ubuntu 18. Now that I have added Mono12Packed and Mono12p support to ADAravis it really has all the functions needed for mono cameras and ADSpinnaker provides no addtional benefit that I know of. However, for color cameras ADSpinnaker does Bayer to RGB conversion that ADAravis currently lacks.

@decarlof
Copy link

@MarkRivers your FLIR/Spinnaker screen shows you are using Driver version 3.0.0. I am not sure if this is related but on the 3.1.0 version installed at 2-BM the max number I get for the input buffer is still 10 (see here).

@MarkRivers
Copy link
Member

You need to change your startup script as documented in the R3-1 notes in RELEASE.md.

The arguments in the startup script are documented here: https://areadetector.github.io/master/ADSpinnaker/ADSpinnaker.html#ioc-startup-script

@LeeYangLBLBCS
Copy link
Author

I updated ADSpinnaker to 3.10 but I still see quite a few lost frames even at at conservative frame rate of 60fps and ADC=10 bits. (see attached screen shot that shows 4990 of 5000 frames acquired).
Also the improved statistics is not available. Maybe I need to compile ADAravis to get that?
image

tttt.txt

@MarkRivers
Copy link
Member

I updated ADSpinnaker to 3.10 but I still see quite a few lost frames even at at conservative frame rate of 60fps and ADC=10 bits. (see attached screen shot that shows 4990 of 5000 frames acquired).
Also the improved statistics is not available. Maybe I need to compile ADAravis to get that?

Your screen shot above shows that you are running ADSpinnaker 3.0.0. You need to update to ADSpinnaker 3.1.0. You are running ADCore 3.10, which is fine.

The second problem is that you are not using the latest medm screen from ADSpinnaker 3.1.0, so you are not seeing all of the statistics.

Be sure to also change your startup script, as I noted to @decarlof above.

This is what the screen should look like. Note that the Driver version is 3.1.0 and there are additional statistics values.

image

@MarkRivers
Copy link
Member

In the above screen I am collecting 5000 frames at 129 frames/s, Mono12Packed, 10-bit, converting to Mono16.

I see the input buffers start at 95 and gradually decrease during the acquisition. When it reaches 5000 frames input buffers is down to ~20. This means I likely would have run out of buffers if I had collected 10000 frames. I could avoid this by increasing the maximum number of buffers in the startup script from the default of 100.

@decarlof
Copy link

decarlof commented Oct 29, 2020 •
edited
Loading

@LeeYangLBL my detector is in use at the beamline so I could not complete the test, @MarkRivers was correct after installation the old start-up script was copied over to the new version. I will update next week.

@LeeYangLBLBCS
Copy link
Author

@MarkRivers v3.10 still drops frames.
FLIR released two more versions, 2.1.0.95/2.2.0.48. I wonder if that'll work with this driver?
You mentioned ADAravis driver works better compared to ADSpinnaker.
Do you recommend ADArvis instead?

@MarkRivers
Copy link
Member

Please post a screen shot of the new version.

@LeeYangLBLBCS
Copy link
Author

screen shot with 6 lost frames:
image
all plugin settings:
image
Feature #2 screen shot:
image

@MarkRivers
Copy link
Member

Please do it again. When it gets to about 4500 frames complete take the screen shot of just the main ADSpinnaker screen. What value does the Input buffers have at that point?

@LeeYangLBLBCS
Copy link
Author

here is at 4567 frame:
image

@MarkRivers
Copy link
Member

At the end of that acquisition how many images were complete? It looks like InputBuffers never drops below 100?

You are only trying to do 80 frames/s. As my screen shot above shows with 10-bit Mono12Packed I can do 130 frames/s and not drop any frames.

@LeeYangLBLBCS
Copy link
Author

it lost 16 frames at the end.
this is the end of acquisition:
Uploading image.png…

@LeeYangLBLBCS
Copy link
Author

last message seems to be stuck at "Uploading image". Uploading again:
image

@MarkRivers
Copy link
Member

What happens if you try to run at 130 frames/s?

It looks like you are getting a lot of FailedPackets. That indicates a problem at the lower-level Ethernet interface. I don't have any failed packets in my screen shot.

@LeeYangLBLBCS
Copy link
Author

maybe there is something with the brand of the computer we are using (supermicro):
CPU is 6 core, 3.5GHz.
What computer are you using? I wonder if we should get the same one.
cpuinfo.txt

@LeeYangLBLBCS
Copy link
Author

at 112 fps, lost more frames, ~220 frames.
image
image

@MarkRivers
Copy link
Member

Note that in your first screen shot above InputBuffers is only 2, while it started at 100. The was when it was acquiring because Image rate=108. This strongly suggests that you are dropping frames because you are running out of input buffers. Here are things to do:

  • Run camonitor on InputBuffers while you are acquiring. See if the value is dropping
  • Run "top -H" while you are acquiring. When you get to 4000 frames complete capture the top -H screen and post it. We can see if any threads are using 100% CPU.
  • Increase input buffers in the st.cmd file to 1000 and see if it can complete 5000 frames OK.

@decarlof
Copy link

decarlof commented Nov 3, 2020

I set the numSPBuffers to 100 with mono12Pack, 10bit and I run out of input buffer and loose frames if I set frame rate above 80.

I increased the buffer to 400 and get:

100fps_during

and at the end

100fps_end

I never see the failed packet counter to increase, just at the end I get 4178/5000 frames.

If I set buffer > 400 the IOC crashes with:

epicsThreadSleep(1.0)
epics> /net/s2dserv/xorApps/PreBuilts/areaDetector-R3-10/ADSpinnaker-R3-1/iocs/spinnakerIOC/iocBoot/iocSpinnaker/softioc/2bmbSpinnaker.sh: line 185: 2932845 Segmentation fault      (core dumped) ${IOC_CMD}

I think is because my MemTotal: 64460268 kB

@MarkRivers
Copy link
Member

Hi Francesco,

Your screen shot above shows that InputBufferCount has decreased from 400 down to 2. Can you repeat the test with 400 input buffers, but this time run this command in another window:

camonitor 2bmbSP1:cam1:NumImagesCounter_RBV 2bmbSP1:cam1:InputBufferCount

Then we can see how the number of free buffers changes as it is acquiring.

Can you also run "top -H" in another window and capture that screen when it acquiring?

@decarlof
Copy link

decarlof commented Nov 3, 2020

Hi Mark,
I restarted the camera with 400 input buffer and start the monitoring, first in 10-bit /Mono12Pack/80fps and I could see the
InputBufferCount stabilizing around 200.

2bmbSP1:cam1:InputBufferCount  2020-11-03 16:54:59.135727 215  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:54:59.147795 982  
2bmbSP1:cam1:InputBufferCount  2020-11-03 16:54:59.147804 216  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:54:59.182009 983  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:54:59.217220 984  
....
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:01.897469 2769  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:01.932332 2770  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:01.967699 2771  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.003755 2772  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.037956 2773  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.072963 2774  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.107975 2775  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.143357 2776  
2bmbSP1:cam1:NumImagesCounter_RBV 2020-11-03 16:56:02.178097 2777

I then, before increasing the fps, I wanted to see the same parameters when running in 8-bit/160fps but I end up with the camera in strange state. I can now only run it a very slow speed (~30fps). I have seen this in the past and solved it by adjusting ISP Enable parameter or increasing the DeviceLinkThroughputLimit. (see warning for more details) but now nothing seems to work ... so I stock at 30fps even if I set 100fps and I know the camera can do 160. Here are the current settings:

Screen Shot 2020-11-03 at 4 19 47 PM

Screen Shot 2020-11-03 at 4 20 00 PM

@MarkRivers
Copy link
Member

You are only getting 30 fps because your AcquireTime is 0.035. You need to change it to 0.001 for testing.

@decarlof
Copy link

decarlof commented Nov 3, 2020

ops my bad.

@decarlof
Copy link

decarlof commented Nov 3, 2020

OK I run out of buffer with 100fps here are the logs.

CPU load is:

Screen Shot 2020-11-03 at 4 39 46 PM

@MarkRivers
Copy link
Member

Yes, so your computer can just barely do 80 fps, and cannot do 100 fps. Mine can do 130 fps for 5000 frames, but it would use more than 100 buffers with 10000 frames.

You need a faster computer to be able to run the Oryx in 10-bit mode at its maximum frame rate of 130 frames/s.

You should try using ADAravis. I suspect it will work on your existing computer, because my unpacking method is a bit faster than FLIR's.

@LeeYangLBLBCS
Copy link
Author

My test with 5000 images (mono12Packed/10 bit) :
Top -H shows ADSinnakerImage CPU 99%.
image
Acquisition halted at 4782:
image

camonitor 13SP1:cam1:NumImagesCounter_RBV 13SP1:cam1:InputBufferCount result:
Initially inputbufferCount is small, about 1-3. Then suddenly increased quickly to 100 at about 4800 image frame.
I am using the default buffer size of 100.
Does this mean I need to get a faster computer?

13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.750602 46
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.761084 4428
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.761093 47
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.772023 4429
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.772032 48
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.781118 4430
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.781135 49
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.790217 4431
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.790227 50
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.798920 4432
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.798952 51
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.807732 4433
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.807748 52
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.816455 4434
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.816464 53
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.825071 4435
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.825093 54
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.833765 4436
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.833788 55
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.843426 4437
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.843433 56
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.852360 4438
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.852367 57
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.863787 4439
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.863796 58
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.872291 4440
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.872300 59
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.881827 4441
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.881844 60
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.893546 4442
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.893556 61
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.901867 4443
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.901878 62
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.910319 4444
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.910327 63
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.919646 4445
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.919656 64
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.928691 4446
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.928703 65
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.937118 4447
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.937125 66
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.945626 4448
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.945634 67
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.954398 4449
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.954406 68
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.963895 4450
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.963912 69
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.973792 4451
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.973808 70
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.983065 4452
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.983073 71
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:18.991749 4453
13SP1:cam1:InputBufferCount 2020-11-03 18:25:18.991756 72
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.000137 4454
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.000145 73
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.008424 4455
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.008432 74
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.016743 4456
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.016758 75
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.025515 4457
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.025538 76
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.034754 4458
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.034774 77
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.043120 4459
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.043136 78
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.051368 4460
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.051380 79
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.060071 4461
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.060080 80
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.069033 4462
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.069057 81
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.079125 4463
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.079135 82
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.088257 4464
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.088268 83
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.097112 4465
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.097133 84
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.107207 4466
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.107214 85
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.115673 4467
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.115682 86
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.124066 4468
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.124085 87
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.132786 4469
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.132801 88
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.141944 4470
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.142003 89
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.152274 4471
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.152280 90
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.162566 4472
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.162575 91
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.173289 4473
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.173299 92
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.184600 4474
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.184629 93
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.195185 4475
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.195197 94
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.206228 4476
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.206247 95
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.217257 4477
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.217350 96
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.227804 4478
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.227818 97
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.238104 4479
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.238113 98
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.248467 4480
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.248485 99
13SP1:cam1:NumImagesCounter_RBV 2020-11-03 18:25:19.259684 4481
13SP1:cam1:InputBufferCount 2020-11-03 18:25:19.259702 100

@decarlof
Copy link

decarlof commented Nov 4, 2020

You should try using ADAravis. I suspect it will work on your existing computer, because my unpacking method is a bit faster than FLIR's.

thank you Mark, will do.

@MarkRivers
Copy link
Member

Initially inputbufferCount is small, about 1-3. Then suddenly increased quickly to 100 at about 4800 image frame.
I am using the default buffer size of 100.

No, it did not start small (1-3), it started at 100, then it went down from 100 to 1-3. At that point it was dropping buffers. Then when the detector is actually done acquiring those buffers get processed and it goes back to 100.

Does this mean I need to get a faster computer?

Yes. Or try ADAravis and see if can keep up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@decarlof @MarkRivers @LeeYangLBLBCS