External repository for header specification #11

clbarnes · 2023-01-20T12:41:50Z

I started https://github.com/clbarnes/jeiss-specs/ as a way to share, in a language-agnostic way, information about the header format. This could then be used as a concrete base for reader implementations, either with code generation or to read dynamically. Having a single source of truth in this way seemed valuable to me. I implemented a lossless HDF5 converter (which can provably regenerate the original .dat from the HDF5) based on that: https://github.com/clbarnes/jeiss-convert .

To me, the main missing piece is actually comparing the data in that repo to the labview implementation. @mkitti , is there a way I can do that without a labview license/ digging handfuls of spaghetti out of the SVG representation, or might you be able to take a look? I'm sure most users are onto v8 or higher by now so I'd be happy to call v1-7 support experimental if we could just get v8 right.

mkitti · 2023-01-20T17:34:10Z

@trautmane is currently handling dat files. @virginiascarlett is thinking about overarching data flow issues particularly in the realm of open data and recently started to tackle these issues.

I'm not sure if there is another way other than LabView license or a PDF / SVG print out of the "code". That said I think we could compare it against more data that we have on hand. We could also compare to other non-LabView code.

Should we organize a coordination meeting with @trautmane and @virginiascarlett?

virginiascarlett · 2023-01-23T16:30:05Z

I'm not too savvy with the technical questions, but I would be happy to facilitate a meeting about this. Mark @mkitti has a lot on his plate right now, but maybe Eric @trautmane or I can help you out.

trautmane · 2023-01-23T19:49:37Z

Hi @clbarnes - I'll defer to @mkitti regarding anything related to LabView since I can't help much there.

I am happy to draft/add a v9 TSV spec to your jeiss-specs repo based on what I know/use from @d-v-b 's fibsem tools repo. With a v9 spec, I can then try to integrate your HDF5 converter into my pipelines and replace the fibsem tools based code I currently use. I support the idea of there being one definitive dat parsing and low-level conversion library (and am happy for that to be your library).

I don't have time this week to work on this, but will try to take a look next week. I'm also available to meet next week or later if you think that would help in any way.

Best,
Eric

mkitti · 2023-01-25T02:15:50Z

@clbarnes Here is the internal documentation from LabView. I looked over the LabView code quickly and I do not see anything missing from this list. Let me know if you have any specific questions.

The next thing I would examine is perhaps the MATLAB and Python code.

Tracking File Headers:
BLN: boolean (1 byte)
U8: unsigned 8-bit (1 byte)
U16: unsigned short (2 bytes)
U32: unsigned long (4 bytes)
DBL: double (8 bytes)
I16: signed short (2 bytes)
I32: signed long (4 bytes)

Byte	       Type		      Significance
0              U32	        Magic Number, should be 0xd3edf5f2 (3555587570)
4   	          U16	        Version number (9)
6   	          U16	        File type. 1: SEM image
8              STR         SW date
24     	      DBL         AI sampling time increment
32     	      U8           Number of channels
33            BLN         8-bit data
36            SGL         AI device scaling coefficients or 8-bit conversion coefficients (Io and k)
68            U8           Restart flag
69            U8           Stage move flag
70            I32         X coordinate of the first pixel
74            I32         Y coordinate of the first pixel

100          U32        X resolution
104          U32        Y resolution
108          U16        AI oversampling     
111          U8          Scan speed (Zeiss #)
112          SGL        Actual AO (scanning) rate
116          SGL        Frame line rampdown ratio
120          SGL        X min (V)
124          SGL        X max (V)
128          SGL        Det min (V)
132          SGL        Det max (V)
136          U16        Decimating factor 

151          BLN        AI Ch1          
152          BLN        AI Ch2
153          BLN        AI Ch3
154          BLN        AI Ch4
 
155          STR        Sample ID
180          STR        Notes

SEM parameters
380          STR       Detector A
390          STR       Detector B
410          STR        Detector C
430          STR        Detector D

460          SGL        Magnification
464          SGL        Pixel size (nm)
468          SGL        Working distance (mm)
472          SGL        EHT (kV)
480          U8          Aperture number
481          U8          High current mode
490          SGL        Probe current (A)
494          SGL        Scan rotation (deg)
498          SGL        Chamber vacuum (Torr)
502          SGL        Gun vacuum (Torr)
510          SGL        Beam shift X
514          SGL        Beam shift Y
518          SGL        Stigmation X
522          SGL        Stigmation Y
526          SGL        Aperture aligment X
530          SGL        Aperture aligment Y
534          SGL        Stage X (mm)
538          SGL        Stage Y (mm)
542          SGL        Stage Z (mm)
546          SGL        Stage T (deg)
550          SGL        Stage R (deg)
554          SGL        Stage M (mm)
560          SGL        Brightness (%), detector A
564          SGL        Contrast (%),  detector A
568          SGL        Brightness (%), detector B
572          SGL        Contrast (%),  detector B


FIB parameters
600          U8         Mode
604          SGL        Focus (V)
608          U8         Probe current number
620          SGL        Emission current (A)
624          SGL        rotation (deg)
628          SGL        Aperture alignment X
632          SGL        Aperture alignment Y
636          SGL        Stigmation X
640          SGL        Stigmation Y
644          SGL        Beam shift X (um)
648          SGL        Beam shift Y (um)


Milling parameters
652          U32        X resolution
656          U32        Y resolution
660          SGL        X size (um)
664          SGL        Y size (um)
668          SGL        UL-Ang (deg)
672          SGL        UR-Ang (deg)
676          SGL        Line milling time (s)
680          SGL        FIB FOV (um)
684          U16        Lines auto pause
686          BLN       FIB PID on
689          U8          FIB PID measured (0: specimen, 1: beam dump)
690          SGL        FIB PID target
694          SGL        FIB PID target slope
698          SGL        FIB PID P
702          SGL        FIB PID I
706          SGL        FIB PID D


800          STR        Machine ID

850          SGL        Temperature (F)
854          SGL        Faraday cup current (nA)
858          SGL        Specimen current during FIB (nA)
862          SGL        Beam dump 1 current (nA)
866          SGL        Specimen current during SEM (nA)
870          SGL        Milling Z voltage (V)
874          SGL        Focus index
878          U32        FIB slice #
882          SGL        Beam dump 2 current (nA)
886          SGL        Milling current (nA)

1000        DBL        File length, position for the beginning of recipe variant

1024        I16          1-D array of n channels of unscaled measurement from DAQ

mkitti · 2023-01-25T02:19:04Z

Here is the MATLAB code readfibsem.m:

function FIBSEMData = readfibsem(FullPathFile)
% Read raw data file (*.dat) generated from Neon
% Needs PathName and FileName
%
% Rev history
% 04/17/09 
%   1st rev.
% 07/31/2011
%   converted from script to function
% 11/25/2012
%   added support for file version 5
% 6/20/2013
%   read raw data up to
%   [FIBSEMData.ChanNum,FIBSEMData.XResolution*FIBSEMData.YResolution]
% 6/25/2013
%   added support for file version 6
% 7/10/2013
%   added decimating factor
% 7/1/2014
%   added file version 7 for 8-bit data support
% 7/4/2015
%   added file version 8 support
% 7/25/2017
%   added support for files with partial image
% 4/15/2020
%   revised version 8 16-bit electron counts conversion
% 6/14/2020
%   added a scaling factor of electron counts of 16-bit version 8
%   *.dat files
% 11/17/2020
%   added file version 9 support
%

%% Load raw data file
fid = fopen(FullPathFile,'r', 's'); % Open the file written by LabView (big-endian byte ordering and 64-bit long data type)

% Start header read
fseek(fid,0,'bof'); FIBSEMData.FileMagicNum = fread(fid,1,'uint32'); % Read in magic number, should be 3555587570
fseek(fid,4,'bof'); FIBSEMData.FileVersion = fread(fid,1,'uint16'); % Read in file version number
fseek(fid,6,'bof'); FIBSEMData.FileType = fread(fid,1,'uint16'); % Read in file type, 1 is Zeiss Neon detectors
fseek(fid,8,'bof'); FIBSEMData.SWdate = fread(fid,10,'*char')'; % Read in SW date
fseek(fid,24,'bof'); FIBSEMData.TimeStep = fread(fid,1,'double'); % Read in AI sampling time (including oversampling) in seconds
fseek(fid,32,'bof'); FIBSEMData.ChanNum = fread(fid,1,'uint8'); % Read in number of channels
fseek(fid,33,'bof'); FIBSEMData.EightBit = fread(fid,1,'uint8'); % Read in 8-bit data switch
switch FIBSEMData.FileVersion
  case 1
    fseek(fid,36,'bof'); FIBSEMData.Scaling = single(fread(fid,[4,FIBSEMData.ChanNum],'double')); % Read in AI channel scaling factors, (col#: AI#), (row#: offset, gain, 2nd order, 3rd order)
  case {2,3,4,5,6}
    fseek(fid,36,'bof'); FIBSEMData.Scaling = fread(fid,[4,FIBSEMData.ChanNum],'single');
  otherwise
    fseek(fid,36,'bof'); FIBSEMData.Scaling = fread(fid,[4,2],'single');
end
switch FIBSEMData.FileVersion
  case {1,2,3,4,5,6,7,8}
  otherwise
    fseek(fid,68,'bof'); FIBSEMData.RestartFlag = fread(fid,1,'uint8'); % Read in restart flag
    fseek(fid,69,'bof'); FIBSEMData.StageMove = fread(fid,1,'uint8'); % Read in stage move flag
    fseek(fid,70,'bof'); FIBSEMData.FirstPixelX = fread(fid,1,'int32'); % Read in first pixel X coordinate (center = 0)
    fseek(fid,74,'bof'); FIBSEMData.FirstPixelY = fread(fid,1,'int32'); % Read in first pixel Y coordinate (center = 0)
end
fseek(fid,100,'bof'); FIBSEMData.XResolution = fread(fid,1,'uint32'); % X resolution
fseek(fid,104,'bof'); FIBSEMData.YResolution = fread(fid,1,'uint32'); % Y resolution
switch FIBSEMData.FileVersion
  case {1,2,3}
    fseek(fid,108,'bof'); FIBSEMData.Oversampling = fread(fid,1,'uint8'); % AI oversampling
    fseek(fid,109,'bof'); FIBSEMData.AIDelay = fread(fid,1,'int16'); % Read AI delay (# of samples)
  otherwise
    fseek(fid,108,'bof'); FIBSEMData.Oversampling = fread(fid,1,'uint16'); % AI oversampling
end
fseek(fid,111,'bof'); FIBSEMData.ZeissScanSpeed = fread(fid,1,'uint8'); % Scan speed (Zeiss #)
switch FIBSEMData.FileVersion
  case {1,2,3}
    fseek(fid,112,'bof'); FIBSEMData.ScanRate = fread(fid,1,'double'); % Actual AO (scanning) rate
    fseek(fid,120,'bof'); FIBSEMData.FramelineRampdownRatio = fread(fid,1,'double'); % Frameline rampdown ratio
    fseek(fid,128,'bof'); FIBSEMData.Xmin = fread(fid,1,'double'); % X coil minimum voltage
    fseek(fid,136,'bof'); FIBSEMData.Xmax = fread(fid,1,'double'); % X coil maximum voltage
    FIBSEMData.Detmin = -10; % Detector minimum voltage
    FIBSEMData.Detmax = 10; % Detector maximum voltage
  otherwise
    fseek(fid,112,'bof'); FIBSEMData.ScanRate = fread(fid,1,'single'); % Actual AO (scanning) rate
    fseek(fid,116,'bof'); FIBSEMData.FramelineRampdownRatio = fread(fid,1,'single'); % Frameline rampdown ratio
    fseek(fid,120,'bof'); FIBSEMData.Xmin = fread(fid,1,'single'); % X coil minimum voltage
    fseek(fid,124,'bof'); FIBSEMData.Xmax = fread(fid,1,'single'); % X coil maximum voltage
    fseek(fid,128,'bof'); FIBSEMData.Detmin = fread(fid,1,'single'); % Detector minimum voltage
    fseek(fid,132,'bof'); FIBSEMData.Detmax = fread(fid,1,'single'); % Detector maximum voltage
    fseek(fid,136,'bof'); FIBSEMData.DecimatingFactor = fread(fid,1,'uint16'); % Decimating factor
end
fseek(fid,151,'bof'); FIBSEMData.AI1 = fread(fid,1,'uint8'); % AI Ch1
fseek(fid,152,'bof'); FIBSEMData.AI2 = fread(fid,1,'uint8'); % AI Ch2
fseek(fid,153,'bof'); FIBSEMData.AI3 = fread(fid,1,'uint8'); % AI Ch3
fseek(fid,154,'bof'); FIBSEMData.AI4 = fread(fid,1,'uint8'); % AI Ch4
switch FIBSEMData.FileVersion
  case {1,2,3,4,5,6,7,8}
  otherwise
    fseek(fid,155,'bof'); FIBSEMData.SampleID = fread(fid,25,'*char')'; % Read in Sample ID
end
fseek(fid,180,'bof'); FIBSEMData.Notes = fread(fid,200,'*char')'; % Read in notes

switch FIBSEMData.FileVersion
  case {1,2}
    fseek(fid,380,'bof'); FIBSEMData.DetA = fread(fid,10,'*char')'; % Name of detector A
    fseek(fid,390,'bof'); FIBSEMData.DetB = fread(fid,18,'*char')'; % Name of detector B
    fseek(fid,700,'bof'); FIBSEMData.DetC = fread(fid,20,'*char')'; % Name of detector C
    fseek(fid,720,'bof'); FIBSEMData.DetD = fread(fid,20,'*char')'; % Name of detector D
    fseek(fid,408,'bof'); FIBSEMData.Mag = fread(fid,1,'double'); % Magnification
    fseek(fid,416,'bof'); FIBSEMData.PixelSize = fread(fid,1,'double'); % Pixel size in nm
    fseek(fid,424,'bof'); FIBSEMData.WD = fread(fid,1,'double'); % Working distance in mm
    fseek(fid,432,'bof'); FIBSEMData.EHT = fread(fid,1,'double'); % EHT in kV
    fseek(fid,440,'bof'); FIBSEMData.SEMApr = fread(fid,1,'uint8'); % SEM aperture number
    fseek(fid,441,'bof'); FIBSEMData.HighCurrent = fread(fid,1,'uint8'); % high current mode (1=on, 0=off)
    fseek(fid,448,'bof'); FIBSEMData.SEMCurr = fread(fid,1,'double'); % SEM probe current in A
    fseek(fid,456,'bof'); FIBSEMData.SEMRot = fread(fid,1,'double'); % SEM scan roation in degree
    fseek(fid,464,'bof'); FIBSEMData.ChamVac = fread(fid,1,'double'); % Chamber vacuum
    fseek(fid,472,'bof'); FIBSEMData.GunVac = fread(fid,1,'double'); % E-gun vacuum
    fseek(fid,480,'bof'); FIBSEMData.SEMStiX = fread(fid,1,'double'); % SEM stigmation X
    fseek(fid,488,'bof'); FIBSEMData.SEMStiY = fread(fid,1,'double'); % SEM stigmation Y
    fseek(fid,496,'bof'); FIBSEMData.SEMAlnX = fread(fid,1,'double'); % SEM aperture alignment X
    fseek(fid,504,'bof'); FIBSEMData.SEMAlnY = fread(fid,1,'double'); % SEM aperture alignment Y
    fseek(fid,512,'bof'); FIBSEMData.StageX = fread(fid,1,'double'); % Stage position X in mm
    fseek(fid,520,'bof'); FIBSEMData.StageY = fread(fid,1,'double'); % Stage position Y in mm
    fseek(fid,528,'bof'); FIBSEMData.StageZ = fread(fid,1,'double'); % Stage position Z in mm
    fseek(fid,536,'bof'); FIBSEMData.StageT = fread(fid,1,'double'); % Stage position T in degree
    fseek(fid,544,'bof'); FIBSEMData.StageR = fread(fid,1,'double'); % Stage position R in degree
    fseek(fid,552,'bof'); FIBSEMData.StageM = fread(fid,1,'double'); % Stage position M in mm
    fseek(fid,560,'bof'); FIBSEMData.BrightnessA = fread(fid,1,'double'); % Detector A brightness (%)
    fseek(fid,568,'bof'); FIBSEMData.ContrastA = fread(fid,1,'double'); % Detector A contrast (%)
    fseek(fid,576,'bof'); FIBSEMData.BrightnessB = fread(fid,1,'double'); % Detector B brightness (%)
    fseek(fid,584,'bof'); FIBSEMData.ContrastB = fread(fid,1,'double'); % Detector B contrast (%)
    
    fseek(fid,600,'bof'); FIBSEMData.Mode = fread(fid,1,'uint8'); % FIB mode: 0=SEM, 1=FIB, 2=Milling, 3=SEM+FIB, 4=Mill+SEM, 5=SEM Drift Correction, 6=FIB Drift Correction, 7=No Beam, 8=External, 9=External+SEM
    fseek(fid,608,'bof'); FIBSEMData.FIBFocus = fread(fid,1,'double'); % FIB focus in kV
    fseek(fid,616,'bof'); FIBSEMData.FIBProb = fread(fid,1,'uint8'); % FIB probe number
    fseek(fid,624,'bof'); FIBSEMData.FIBCurr = fread(fid,1,'double'); % FIB emission current
    fseek(fid,632,'bof'); FIBSEMData.FIBRot = fread(fid,1,'double'); % FIB scan rotation
    fseek(fid,640,'bof'); FIBSEMData.FIBAlnX = fread(fid,1,'double'); % FIB aperture alignment X
    fseek(fid,648,'bof'); FIBSEMData.FIBAlnY = fread(fid,1,'double'); % FIB aperture alignment Y
    fseek(fid,656,'bof'); FIBSEMData.FIBStiX = fread(fid,1,'double'); % FIB stigmation X
    fseek(fid,664,'bof'); FIBSEMData.FIBStiY = fread(fid,1,'double'); % FIB stigmation Y
    fseek(fid,672,'bof'); FIBSEMData.FIBShiftX = fread(fid,1,'double'); % FIB beam shift X in micron
    fseek(fid,680,'bof'); FIBSEMData.FIBShiftY = fread(fid,1,'double'); % FIB beam shift Y in micron
  otherwise
    fseek(fid,380,'bof'); FIBSEMData.DetA = fread(fid,10,'*char')'; % Name of detector A
    fseek(fid,390,'bof'); FIBSEMData.DetB = fread(fid,18,'*char')'; % Name of detector B
    fseek(fid,410,'bof'); FIBSEMData.DetC = fread(fid,20,'*char')'; % Name of detector C
    fseek(fid,430,'bof'); FIBSEMData.DetD = fread(fid,20,'*char')'; % Name of detector D
    fseek(fid,460,'bof'); FIBSEMData.Mag = fread(fid,1,'single'); % Magnification
    fseek(fid,464,'bof'); FIBSEMData.PixelSize = fread(fid,1,'single'); % Pixel size in nm
    fseek(fid,468,'bof'); FIBSEMData.WD = fread(fid,1,'single'); % Working distance in mm
    fseek(fid,472,'bof'); FIBSEMData.EHT = fread(fid,1,'single'); % EHT in kV
    fseek(fid,480,'bof'); FIBSEMData.SEMApr = fread(fid,1,'uint8'); % SEM aperture number
    fseek(fid,481,'bof'); FIBSEMData.HighCurrent = fread(fid,1,'uint8'); % high current mode (1=on, 0=off)
    fseek(fid,490,'bof'); FIBSEMData.SEMCurr = fread(fid,1,'single'); % SEM probe current in A
    fseek(fid,494,'bof'); FIBSEMData.SEMRot = fread(fid,1,'single'); % SEM scan roation in degree
    fseek(fid,498,'bof'); FIBSEMData.ChamVac = fread(fid,1,'single'); % Chamber vacuum
    fseek(fid,502,'bof'); FIBSEMData.GunVac = fread(fid,1,'single'); % E-gun vacuum
    fseek(fid,510,'bof'); FIBSEMData.SEMShiftX = fread(fid,1,'single'); % SEM beam shift X
    fseek(fid,514,'bof'); FIBSEMData.SEMShiftY = fread(fid,1,'single'); % SEM beam shift Y
    fseek(fid,518,'bof'); FIBSEMData.SEMStiX = fread(fid,1,'single'); % SEM stigmation X
    fseek(fid,522,'bof'); FIBSEMData.SEMStiY = fread(fid,1,'single'); % SEM stigmation Y
    fseek(fid,526,'bof'); FIBSEMData.SEMAlnX = fread(fid,1,'single'); % SEM aperture alignment X
    fseek(fid,530,'bof'); FIBSEMData.SEMAlnY = fread(fid,1,'single'); % SEM aperture alignment Y
    fseek(fid,534,'bof'); FIBSEMData.StageX = fread(fid,1,'single'); % Stage position X in mm
    fseek(fid,538,'bof'); FIBSEMData.StageY = fread(fid,1,'single'); % Stage position Y in mm
    fseek(fid,542,'bof'); FIBSEMData.StageZ = fread(fid,1,'single'); % Stage position Z in mm
    fseek(fid,546,'bof'); FIBSEMData.StageT = fread(fid,1,'single'); % Stage position T in degree
    fseek(fid,550,'bof'); FIBSEMData.StageR = fread(fid,1,'single'); % Stage position R in degree
    fseek(fid,554,'bof'); FIBSEMData.StageM = fread(fid,1,'single'); % Stage position M in mm
    fseek(fid,560,'bof'); FIBSEMData.BrightnessA = fread(fid,1,'single'); % Detector A brightness (%)
    fseek(fid,564,'bof'); FIBSEMData.ContrastA = fread(fid,1,'single'); % Detector A contrast (%)
    fseek(fid,568,'bof'); FIBSEMData.BrightnessB = fread(fid,1,'single'); % Detector B brightness (%)
    fseek(fid,572,'bof'); FIBSEMData.ContrastB = fread(fid,1,'single'); % Detector B contrast (%)
    
    fseek(fid,600,'bof'); FIBSEMData.Mode = fread(fid,1,'uint8'); % FIB mode: 0=SEM, 1=FIB, 2=Milling, 3=SEM+FIB, 4=Mill+SEM, 5=SEM Drift Correction, 6=FIB Drift Correction, 7=No Beam, 8=External, 9=External+SEM
    fseek(fid,604,'bof'); FIBSEMData.FIBFocus = fread(fid,1,'single'); % FIB focus in kV
    fseek(fid,608,'bof'); FIBSEMData.FIBProb = fread(fid,1,'uint8'); % FIB probe number
    fseek(fid,620,'bof'); FIBSEMData.FIBCurr = fread(fid,1,'single'); % FIB emission current
    fseek(fid,624,'bof'); FIBSEMData.FIBRot = fread(fid,1,'single'); % FIB scan rotation
    fseek(fid,628,'bof'); FIBSEMData.FIBAlnX = fread(fid,1,'single'); % FIB aperture alignment X
    fseek(fid,632,'bof'); FIBSEMData.FIBAlnY = fread(fid,1,'single'); % FIB aperture alignment Y
    fseek(fid,636,'bof'); FIBSEMData.FIBStiX = fread(fid,1,'single'); % FIB stigmation X
    fseek(fid,640,'bof'); FIBSEMData.FIBStiY = fread(fid,1,'single'); % FIB stigmation Y
    fseek(fid,644,'bof'); FIBSEMData.FIBShiftX = fread(fid,1,'single'); % FIB beam shift X in micron
    fseek(fid,648,'bof'); FIBSEMData.FIBShiftY = fread(fid,1,'single'); % FIB beam shift Y in micron
end

switch FIBSEMData.FileVersion
  case {1,2,3,4}
  otherwise
    fseek(fid,652,'bof'); FIBSEMData.MillingXResolution = fread(fid,1,'uint32'); % FIB milling X resolution
    fseek(fid,656,'bof'); FIBSEMData.MillingYResolution = fread(fid,1,'uint32'); % FIB milling Y resolution
    fseek(fid,660,'bof'); FIBSEMData.MillingXSize = fread(fid,1,'single'); % FIB milling X size (um)
    fseek(fid,664,'bof'); FIBSEMData.MillingYSize = fread(fid,1,'single'); % FIB milling Y size (um)
    fseek(fid,668,'bof'); FIBSEMData.MillingULAng = fread(fid,1,'single'); % FIB milling upper left inner angle (deg)
    fseek(fid,672,'bof'); FIBSEMData.MillingURAng = fread(fid,1,'single'); % FIB milling upper right inner angle (deg)
    fseek(fid,676,'bof'); FIBSEMData.MillingLineTime = fread(fid,1,'single'); % FIB line milling time (s)
    fseek(fid,680,'bof'); FIBSEMData.FIBFOV = fread(fid,1,'single'); % FIB FOV (um)
    fseek(fid,684,'bof'); FIBSEMData.MillingLinesPerImage = fread(fid,1,'uint16'); % FIB milling lines per image
    fseek(fid,686,'bof'); FIBSEMData.MillingPIDOn = fread(fid,1,'uint8'); % FIB milling PID on
    fseek(fid,689,'bof'); FIBSEMData.MillingPIDMeasured = fread(fid,1,'uint8'); % FIB milling PID measured (0:specimen, 1:beamdump)
    fseek(fid,690,'bof'); FIBSEMData.MillingPIDTarget = fread(fid,1,'single'); % FIB milling PID target
    fseek(fid,694,'bof'); FIBSEMData.MillingPIDTargetSlope = fread(fid,1,'single'); % FIB milling PID target slope
    fseek(fid,698,'bof'); FIBSEMData.MillingPIDP = fread(fid,1,'single'); % FIB milling PID P
    fseek(fid,702,'bof'); FIBSEMData.MillingPIDI = fread(fid,1,'single'); % FIB milling PID I
    fseek(fid,706,'bof'); FIBSEMData.MillingPIDD = fread(fid,1,'single'); % FIB milling PID D
    fseek(fid,800,'bof'); FIBSEMData.MachineID = fread(fid,30,'*char')'; % Machine ID
    fseek(fid,980,'bof'); FIBSEMData.SEMSpecimenI = fread(fid,1,'single'); % SEM specimen current (nA)
end

switch FIBSEMData.FileVersion
  case {1,2,3,4,5}
  otherwise
    fseek(fid,850,'bof'); FIBSEMData.Temperature = fread(fid,1,'single'); % Temperature (F)
    fseek(fid,854,'bof'); FIBSEMData.FaradayCupI = fread(fid,1,'single'); % Faraday cup current (nA)
    fseek(fid,858,'bof'); FIBSEMData.FIBSpecimenI = fread(fid,1,'single'); % FIB specimen current (nA)
    fseek(fid,862,'bof'); FIBSEMData.BeamDump1I = fread(fid,1,'single'); % Beam dump 1 current (nA)
    fseek(fid,866,'bof'); FIBSEMData.SEMSpecimenI = fread(fid,1,'single'); % SEM specimen current (nA)
    fseek(fid,870,'bof'); FIBSEMData.MillingYVoltage = fread(fid,1,'single'); % Milling Y voltage (V)
    fseek(fid,874,'bof'); FIBSEMData.FocusIndex = fread(fid,1,'single'); % Focus index
    fseek(fid,878,'bof'); FIBSEMData.FIBSliceNum = fread(fid,1,'uint32'); % FIB slice #
end

switch FIBSEMData.FileVersion
  case {1,2,3,4,5,6,7}
  otherwise
    fseek(fid,882,'bof'); FIBSEMData.BeamDump2I = fread(fid,1,'single'); % Beam dump 2 current (nA)
    fseek(fid,886,'bof'); FIBSEMData.MillingI = fread(fid,1,'single'); % Milling current (nA)
end

fseek(fid,1000,'bof'); FIBSEMData.FileLength = fread(fid,1,'int64'); % Read in file length in bytes
% Finish header read

if FIBSEMData.EightBit==1
  fseek(fid,1024,'bof'); Raw = (fread(fid,[FIBSEMData.ChanNum,FIBSEMData.XResolution*FIBSEMData.YResolution],'*uint8'))'; % Read in raw AI the "*" is needed to read long set of data
  missing = uint8(zeros(FIBSEMData.XResolution*FIBSEMData.YResolution-size(Raw,1),2)); % creates missing element array
  Raw = vertcat(Raw, missing); % concatenate zeros to the correct size of raw data
else
  fseek(fid,1024,'bof'); Raw = (fread(fid,[FIBSEMData.ChanNum,FIBSEMData.XResolution*FIBSEMData.YResolution],'*int16'))'; % Read in raw AI the "*" is needed to read long set of data
  missing = int16(zeros(FIBSEMData.XResolution*FIBSEMData.YResolution-size(Raw,1),2)); % creates missing element array
  Raw = vertcat(Raw, missing); % concatenate zeros to the correct size of raw data
end
fclose(fid); % Close the file

%% Convert raw data to electron counts
if FIBSEMData.EightBit==1
  RawTemp=Raw;
  Raw=int16(Raw);
  if FIBSEMData.AI1
    DetectorA = RawTemp(:,1);
    Raw(:,1)=int16(single(Raw(:,1))*FIBSEMData.ScanRate/FIBSEMData.Scaling(1,1)/FIBSEMData.Scaling(3,1)/FIBSEMData.Scaling(4,1)+FIBSEMData.Scaling(2,1));
    if FIBSEMData.AI2
      DetectorB = RawTemp(:,2);
      Raw(:,2)=int16(single(Raw(:,2))*FIBSEMData.ScanRate/FIBSEMData.Scaling(1,2)/FIBSEMData.Scaling(3,2)/FIBSEMData.Scaling(4,2)+FIBSEMData.Scaling(2,2));
    end
  elseif FIBSEMData.AI2
    DetectorB = RawTemp(:,1);
    Raw(:,1)=int16(single(Raw(:,1))*FIBSEMData.ScanRate/FIBSEMData.Scaling(1,2)/FIBSEMData.Scaling(3,2)/FIBSEMData.Scaling(4,2)+FIBSEMData.Scaling(2,2));
  end
else
  switch FIBSEMData.FileVersion
    case {1,2,3,4,5,6}
      if FIBSEMData.AI1
        DetectorA = FIBSEMData.Scaling(1,1)+single(Raw(:,1))*FIBSEMData.Scaling(2,1); % Converts raw I16 data to voltage based on scaling factors
        if FIBSEMData.AI2
          DetectorB = FIBSEMData.Scaling(1,2)+single(Raw(:,2))*FIBSEMData.Scaling(2,2); % Converts raw I16 data to voltage based on scaling factors
          if FIBSEMData.AI3
            DetectorC = FIBSEMData.Scaling(1,3)+single(Raw(:,3))*FIBSEMData.Scaling(2,3);
            if FIBSEMData.AI4
              DetectorD = FIBSEMData.Scaling(1,4)+single(Raw(:,4))*FIBSEMData.Scaling(2,4);
            end
          elseif FIBSEMData.AI4
            DetectorD = FIBSEMData.Scaling(1,3)+single(Raw(:,3))*FIBSEMData.Scaling(2,3);
          end
        elseif FIBSEMData.AI3
          DetectorC = FIBSEMData.Scaling(1,2)+single(Raw(:,2))*FIBSEMData.Scaling(2,2);
          if FIBSEMData.AI4
            DetectorD = FIBSEMData.Scaling(1,3)+single(Raw(:,3))*FIBSEMData.Scaling(2,3);
          end
        elseif FIBSEMData.AI4
          DetectorD = FIBSEMData.Scaling(1,2)+single(Raw(:,2))*FIBSEMData.Scaling(2,2);
        end
      elseif FIBSEMData.AI2
        DetectorB = FIBSEMData.Scaling(1,1)+single(Raw(:,1))*FIBSEMData.Scaling(2,1);
        if FIBSEMData.AI3
          DetectorC = FIBSEMData.Scaling(1,2)+single(Raw(:,2))*FIBSEMData.Scaling(2,2);
          if FIBSEMData.AI4
            DetectorD = FIBSEMData.Scaling(1,3)+single(Raw(:,3))*FIBSEMData.Scaling(2,3);
          end
        elseif FIBSEMData.AI4
          DetectorD = FIBSEMData.Scaling(1,2)+single(Raw(:,2))*FIBSEMData.Scaling(2,2);
        end
      elseif FIBSEMData.AI3
        DetectorC = FIBSEMData.Scaling(1,1)+single(Raw(:,1))*FIBSEMData.Scaling(2,1);
        if FIBSEMData.AI4
          DetectorD = FIBSEMData.Scaling(1,2)+single(Raw(:,2))*FIBSEMData.Scaling(2,2);
        end
      elseif FIBSEMData.AI4
        DetectorD = FIBSEMData.Scaling(1,1)+single(Raw(:,1))*FIBSEMData.Scaling(2,1);
      end
    case {7}
      if FIBSEMData.AI1
        DetectorA = (single(Raw(:,1))-FIBSEMData.Scaling(2,1))*FIBSEMData.Scaling(3,1); % Converts raw I16 data to voltage based on scaling factors
        if FIBSEMData.AI2
          DetectorB = (single(Raw(:,2))-FIBSEMData.Scaling(2,2))*FIBSEMData.Scaling(3,2);
        end
      elseif FIBSEMData.AI2
        DetectorB = (single(Raw(:,1))-FIBSEMData.Scaling(2,2))*FIBSEMData.Scaling(3,2);
      end
    case {8,9} 
      ElectronFactor1=0.1; % 16-bit intensity is 10x electron counts
      FIBSEMData.Scaling(4,1)=ElectronFactor1;
      ElectronFactor2=0.1; % 16-bit intensity is 10x electron counts
      FIBSEMData.Scaling(4,2)=ElectronFactor2;
      if FIBSEMData.AI1
        DetectorA=(single(Raw(:,1))-FIBSEMData.Scaling(2,1))*FIBSEMData.Scaling(3,1)/FIBSEMData.ScanRate*FIBSEMData.Scaling(1,1)/ElectronFactor1;
        if FIBSEMData.AI2
          DetectorB=(single(Raw(:,2))-FIBSEMData.Scaling(2,2))*FIBSEMData.Scaling(3,2)/FIBSEMData.ScanRate*FIBSEMData.Scaling(1,2)/ElectronFactor2;
        end
      elseif FIBSEMData.AI2
        DetectorB=(single(Raw(:,1))-FIBSEMData.Scaling(2,2))*FIBSEMData.Scaling(3,2)/FIBSEMData.ScanRate*FIBSEMData.Scaling(1,2)/ElectronFactor2;
      end 
  end
end
%% Construct image files
if FIBSEMData.AI1
  FIBSEMData.ImageA = (reshape(DetectorA,FIBSEMData.XResolution,FIBSEMData.YResolution))';
  FIBSEMData.RawImageA = (reshape(Raw(:,1),FIBSEMData.XResolution,FIBSEMData.YResolution))';
  if FIBSEMData.AI2
    FIBSEMData.ImageB = (reshape(DetectorB,FIBSEMData.XResolution,FIBSEMData.YResolution))';
    FIBSEMData.RawImageB = (reshape(Raw(:,2),FIBSEMData.XResolution,FIBSEMData.YResolution))';
    if FIBSEMData.AI3
      FIBSEMData.ImageC = (reshape(DetectorC,FIBSEMData.XResolution,FIBSEMData.YResolution))';
      FIBSEMData.RawImageC = (reshape(Raw(:,3),FIBSEMData.XResolution,FIBSEMData.YResolution))';
      if FIBSEMData.AI4
        FIBSEMData.ImageD = (reshape(DetectorD,FIBSEMData.XResolution,FIBSEMData.YResolution))';
        FIBSEMData.RawImageD = (reshape(Raw(:,4),FIBSEMData.XResolution,FIBSEMData.YResolution))';
      end
    elseif FIBSEMData.AI4
      FIBSEMData.ImageD = (reshape(DetectorD,FIBSEMData.XResolution,FIBSEMData.YResolution))';
      FIBSEMData.RawImageD = (reshape(Raw(:,3),FIBSEMData.XResolution,FIBSEMData.YResolution))';
    end
  elseif FIBSEMData.AI3
    FIBSEMData.ImageC = (reshape(DetectorC,FIBSEMData.XResolution,FIBSEMData.YResolution))';
    FIBSEMData.RawImageC = (reshape(Raw(:,2),FIBSEMData.XResolution,FIBSEMData.YResolution))';
    if FIBSEMData.AI4
      FIBSEMData.ImageD = (reshape(DetectorD,FIBSEMData.XResolution,FIBSEMData.YResolution))';
      FIBSEMData.RawImageD = (reshape(Raw(:,3),FIBSEMData.XResolution,FIBSEMData.YResolution))';
    end
  elseif FIBSEMData.AI4
    FIBSEMData.ImageD = (reshape(DetectorD,FIBSEMData.XResolution,FIBSEMData.YResolution))';
    FIBSEMData.RawImageD = (reshape(Raw(:,2),FIBSEMData.XResolution,FIBSEMData.YResolution))';
  end
elseif FIBSEMData.AI2
  FIBSEMData.ImageB = (reshape(DetectorB,FIBSEMData.XResolution,FIBSEMData.YResolution))';
  FIBSEMData.RawImageB = (reshape(Raw(:,1),FIBSEMData.XResolution,FIBSEMData.YResolution))';
  if FIBSEMData.AI3
    FIBSEMData.ImageC = (reshape(DetectorC,FIBSEMData.XResolution,FIBSEMData.YResolution))';
    FIBSEMData.RawImageC = (reshape(Raw(:,2),FIBSEMData.XResolution,FIBSEMData.YResolution))';
    if FIBSEMData.AI4
      FIBSEMData.ImageD = (reshape(DetectorD,FIBSEMData.XResolution,FIBSEMData.YResolution))';
      FIBSEMData.RawImageD = (reshape(Raw(:,3),FIBSEMData.XResolution,FIBSEMData.YResolution))';
    end
  elseif FIBSEMData.AI4
    FIBSEMData.ImageD = (reshape(DetectorD,FIBSEMData.XResolution,FIBSEMData.YResolution))';
    FIBSEMData.RawImageD = (reshape(Raw(:,2),FIBSEMData.XResolution,FIBSEMData.YResolution))';
  end
elseif FIBSEMData.AI3
  FIBSEMData.ImageC = (reshape(DetectorC,FIBSEMData.XResolution,FIBSEMData.YResolution))';
  FIBSEMData.RawImageC = (reshape(Raw(:,1),FIBSEMData.XResolution,FIBSEMData.YResolution))';
  if FIBSEMData.AI4
    FIBSEMData.ImageD = (reshape(DetectorD,FIBSEMData.XResolution,FIBSEMData.YResolution))';
    FIBSEMData.RawImageD = (reshape(Raw(:,2),FIBSEMData.XResolution,FIBSEMData.YResolution))';
  end
elseif FIBSEMData.AI4
  FIBSEMData.ImageD = (reshape(DetectorD,FIBSEMData.XResolution,FIBSEMData.YResolution))';
  FIBSEMData.RawImageD = (reshape(Raw(:,1),FIBSEMData.XResolution,FIBSEMData.YResolution))';
end

mkitti · 2023-01-25T02:19:37Z

Note that we are currently on version 9.

clbarnes · 2023-01-25T14:49:35Z

That's very useful, thanks! The names in my TSVs are from the MATLAB implementation although it would be really helpful to have the units as that original doc from the labview code does. My TSVs were initially generated from @d-v-b 's code so, barring any fixes after that date, should be in sync there.

I'm available for a meeting next week! I've just updated my converter to optionally ingest the CSVs the microscopes generate, as I think the metadata provided there has overlap with but isn't entirely contained by the header metadata.

P.S. Is that a temperature in Fahrenheit to go with the non-ISO dates 😁 Also a floating point file length?

mkitti · 2023-01-25T19:43:20Z

Err... I think the "file length" at byte 1000 is really a 64-bit integer. By the way, the "file length" is not the actual file length. It's where the data ends. After the "file length" byte offset is where the recipe starts.

mkitti · 2023-01-25T19:47:20Z

P.S. Is that a temperature in Fahrenheit to go with the non-ISO dates 😁 Also a floating point file length?

Apparently. Now I'm curious what scientific sensor actually reports in Fahrenheit now.

mkitti · 2023-01-26T04:12:45Z

I'm available for a meeting next week!

I am relatively free on Wednesday morning.

clbarnes · 2023-01-26T10:33:48Z

I think the "file length" at byte 1000 is really a 64-bit integer

I'm glad it's signed, at least, need to account for those files of negative length.

Wednesday afternoon works for me!

clbarnes · 2023-01-26T11:52:46Z

@mkitti : that MATLAB code seems to read SEMSpecimenI in two places in versions 6+; once at 866 and once at 980. The v9 LabView docs show it at 866.

trautmane · 2023-01-26T13:52:15Z

@mkitti : that MATLAB code seems to read SEMSpecimenI in two places in versions 6+; once at 866 and once at 980. The v9 LabView docs show it at 866.

yeah - I ran into that bug last year ( see janelia-cellmap/fibsem-tools#17 )

mkitti · 2023-01-26T20:46:40Z

is there a way I can do that without a labview license/ digging handfuls of spaghetti out of the SVG representation, or might you be able to take a look?

I spoke with @dmilkie about this, and he pointed to LabView community edition: https://www.ni.com/en-us/shop/labview/select-edition/labview-community-edition.html

clbarnes · 2023-01-31T19:03:02Z

@mkitti @trautmane would you still be available to talk tomorrow about finalising the external spec? I'm free until around 1930 UTC.

trautmane · 2023-01-31T19:40:00Z

I can meet tomorrow anytime between 1300 and 1800 UTC (I have another meeting at 1830).

clbarnes · 2023-02-01T13:03:49Z

Shall we say 1600 UTC?

trautmane · 2023-02-01T13:22:29Z

sure - I just emailed you a zoom link - let me know if you did not receive it

clbarnes mentioned this issue Jan 20, 2023

Actually check that all these are correct clbarnes/jeiss-specs#1

Open

clbarnes mentioned this issue Jan 26, 2023

Improve names clbarnes/jeiss-specs#5

Open

clbarnes mentioned this issue Jan 26, 2023

v9 spec clbarnes/jeiss-specs#6

Closed

clbarnes mentioned this issue Feb 2, 2023

add v9 spec based upon https://github.com/janelia-cosem/fibsem-tools/… clbarnes/jeiss-specs#7

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

External repository for header specification #11

External repository for header specification #11

clbarnes commented Jan 20, 2023

mkitti commented Jan 20, 2023

virginiascarlett commented Jan 23, 2023

trautmane commented Jan 23, 2023

mkitti commented Jan 25, 2023

mkitti commented Jan 25, 2023

mkitti commented Jan 25, 2023

clbarnes commented Jan 25, 2023 •

edited

Loading

mkitti commented Jan 25, 2023

mkitti commented Jan 25, 2023

mkitti commented Jan 26, 2023

clbarnes commented Jan 26, 2023

clbarnes commented Jan 26, 2023

trautmane commented Jan 26, 2023

mkitti commented Jan 26, 2023

clbarnes commented Jan 31, 2023

trautmane commented Jan 31, 2023 •

edited

Loading

clbarnes commented Feb 1, 2023

trautmane commented Feb 1, 2023

External repository for header specification #11

External repository for header specification #11

Comments

clbarnes commented Jan 20, 2023

mkitti commented Jan 20, 2023

virginiascarlett commented Jan 23, 2023

trautmane commented Jan 23, 2023

mkitti commented Jan 25, 2023

mkitti commented Jan 25, 2023

mkitti commented Jan 25, 2023

clbarnes commented Jan 25, 2023 • edited Loading

mkitti commented Jan 25, 2023

mkitti commented Jan 25, 2023

mkitti commented Jan 26, 2023

clbarnes commented Jan 26, 2023

clbarnes commented Jan 26, 2023

trautmane commented Jan 26, 2023

mkitti commented Jan 26, 2023

clbarnes commented Jan 31, 2023

trautmane commented Jan 31, 2023 • edited Loading

clbarnes commented Feb 1, 2023

trautmane commented Feb 1, 2023

clbarnes commented Jan 25, 2023 •

edited

Loading

trautmane commented Jan 31, 2023 •

edited

Loading