Example input data format throws errors

[isabelhurley]

Hi, I've tried running the command line tool with json data files and csv data files and neither has worked for me.
I tried running the fillingaps command and the vt-classify command from the command line passing in a data file containing the example json given in #4 (comment). This is part of the following error message: Unhandled Exception: System.AggregateException: One or more errors occurred. ---> System.FormatException: Input string was not in a correct format. at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal) at System.Number.ParseInt64(String value, NumberStyles options, NumberFormatInfo numfmt) at System.Int64.Parse(String s) at UXI.GazeToolkit.Serialization.Converters.TimestampFromTicksConverter.Convert(String value).
I also tried using the sample csv file format given for raw gaze data (#4 (comment)) and hit errors as it seems to get deserialized as a JSON object (the error begins with: Unhandled Exception: System.AggregateException: One or more errors occurred. ---> Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: T. Path '', line 0, position 0. at Newtonsoft.Json.JsonTextReader.ParseValue() at Newtonsoft.Json.JsonTextReader.Read() at UXI.Serialization.Json.JsonDataReader.TryRead(Object& data)).
How can I get my csv file to be interpreted as a csv file? Can you provide updated data file format examples if there have been updates?

[martinkonopka]

Hello, thanks for reporting this problem. To read input data in CSV format either use --format CSV parameter or do not specify it if the file has .csv extension. To specify format for output, use --output-format CSV, or again, just set the output file with .csv extension. Formats of input and output files may differ, if you need.

Example:
Contents of the input.csv file - note, that Timestamp is in a format with date and time:

Timestamp,LeftValidity,LeftGazePoint2DX,LeftGazePoint2DY,LeftGazePoint3DX,LeftGazePoint3DY,LeftGazePoint3DZ,LeftEyePosition3DX,LeftEyePosition3DY,LeftEyePosition3DZ,LeftPupilDiameter,RightValidity,RightGazePoint2DX,RightGazePoint2DY,RightGazePoint3DX,RightGazePoint3DY,RightGazePoint3DZ,RightEyePosition3DX,RightEyePosition3DY,RightEyePosition3DZ,RightPupilDiameter
2019-02-20T14:30:19.8526299+01:00,Valid,0.249588146805763,0.363745391368866,-130.047988891602,208.728134155273,59.1125564575195,-41.76611328125,8.7668342590332,661.870300292969,3.10238647460938,Valid,0.216417774558067,0.350670248270035,-147.382720947266,212.744323730469,60.574333190918,21.4870147705078,4.3505334854126,660.007873535156,3.25660705566406
2019-02-20T14:30:19.8693049+01:00,Valid,0.247222796082497,0.358571588993073,-131.284103393555,210.317321777344,59.6909790039063,-41.8769149780273,8.7861385345459,661.918762207031,3.11991882324219,Valid,0.229601517319679,0.363623380661011,-140.492935180664,208.765609741211,59.1261978149414,21.4036102294922,4.41953754425049,659.91650390625,3.25010681152344
2019-02-20T14:30:19.8859769+01:00,Valid,0.236037373542786,0.344207912683487,-137.129577636719,214.729309082031,61.2968101501465,-41.9852180480957,8.88303756713867,662.0673828125,3.09461975097656,Valid,0.222043097019196,0.367220818996429,-144.442947387695,207.660614013672,58.7240104675293,21.3197116851807,4.52923631668091,660.086669921875,3.21722412109375
2019-02-20T14:30:19.9026389+01:00,Valid,0.253329932689667,0.359348505735397,-128.092529296875,210.078689575195,59.6041221618652,-42.0834197998047,8.90143585205078,661.941101074219,3.08680725097656,Valid,0.222759857773781,0.348396211862564,-144.068359375,213.442825317383,60.8285675048828,21.2234134674072,4.60443639755249,659.994323730469,3.26002502441406
2019-02-20T14:30:19.9193049+01:00,Valid,0.244058474898338,0.356473416090012,-132.937774658203,210.961807250977,59.9255523681641,-42.200813293457,8.89363098144531,662.165649414063,3.07200622558594,Valid,0.225430101156235,0.364816665649414,-142.672912597656,208.399078369141,58.992790222168,21.1293144226074,4.62313508987427,660.202392578125,3.27789306640625
2019-02-20T14:30:19.9354099+01:00,Valid,0.269463747739792,0.37244439125061,-119.661056518555,206.056121826172,58.1400260925293,-42.3988380432129,8.91371726989746,662.519775390625,3.13050842285156,Valid,0.259013205766678,0.35579439997673,-125.12247467041,211.170379638672,60.00146484375,20.9335346221924,4.64631128311157,660.319458007813,3.47819519042969
2019-02-20T14:30:19.9520849+01:00,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN
2019-02-20T14:30:19.9687419+01:00,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN
2019-02-20T14:30:19.9853949+01:00,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN
2019-02-20T14:30:20.0020669+01:00,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN
2019-02-20T14:30:20.0187439+01:00,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,Valid,0.225678563117981,0.773198366165161,-142.543060302734,82.9596099853516,13.3364915847778,20.1933059692383,4.84471797943115,660.452941894531,2.87591552734375
2019-02-20T14:30:20.0354199+01:00,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,Valid,0.271445840597153,0.398023754358292,-118.625213623047,198.199096679688,55.2803001403809,20.3727340698242,4.92081499099731,660.839416503906,3.12071228027344
2019-02-20T14:30:20.0520849+01:00,Valid,0.299531936645508,0.409739166498184,-103.94751739502,194.600570678711,53.9705390930176,-42.9782333374023,8.91861820220947,662.953125,2.94776916503906,Valid,0.28294438123703,0.36534595489502,-112.616119384766,208.23649597168,58.9336166381836,20.3946361541748,4.90711498260498,660.902770996094,3.17532348632813
2019-02-20T14:30:20.0687609+01:00,Valid,0.300280123949051,0.344572871923447,-103.556510925293,214.617202758789,61.2560081481934,-43.0859375,8.9193172454834,662.773010253906,3.09466552734375,Valid,0.292255342006683,0.346888810396194,-107.750228881836,213.905838012695,60.9970893859863,20.3046340942383,4.94291496276855,660.793640136719,3.25553894042969
2019-02-20T14:30:20.0854289+01:00,Valid,0.30443349480629,0.345005482435226,-101.385963439941,214.484329223633,61.2076416015625,-43.1788368225098,8.91031265258789,662.943725585938,3.12303161621094,Valid,0.288352221250534,0.343738108873367,-109.789993286133,214.873611450195,61.3493309020996,20.2462310791016,4.9574146270752,660.925048828125,3.24212646484375
2019-02-20T14:30:20.1020969+01:00,Valid,0.304627418518066,0.333515554666519,-101.284622192383,218.013595581055,62.4921951293945,-43.237133026123,8.86701774597168,662.796936035156,3.07875061035156,Valid,0.288599461317062,0.36536380648613,-109.660797119141,208.231018066406,58.9316215515137,20.1889362335205,4.94511127471924,660.793151855469,3.24139404296875
2019-02-20T14:30:20.1187519+01:00,Valid,0.309577882289886,0.337784349918365,-98.6975250244141,216.702377319336,62.014949798584,-43.3080368041992,8.82761573791504,663.017211914063,3.05610656738281,Valid,0.291314989328384,0.367952406406403,-108.241668701172,207.435897827148,58.642219543457,20.1552352905273,4.94071245193481,660.792846679688,3.31416320800781
2019-02-20T14:30:20.1360949+01:00,Valid,0.300848722457886,0.335498183965683,-103.25936126709,217.404602050781,62.2705383300781,-43.3376350402832,8.78061771392822,663.044555664063,3.07589721679688,Valid,0.282489657402039,0.379741191864014,-112.85375213623,203.814819335938,57.3242568969727,20.1322326660156,4.92041683197021,660.8388671875,3.24394226074219
2019-02-20T14:30:20.1538459+01:00,Valid,0.304312169551849,0.36512616276741,-101.449371337891,208.304000854492,58.9581871032715,-43.3578338623047,8.76451301574707,663.096130371094,3.09585571289063,Valid,0.283008337020874,0.377561151981354,-112.58268737793,204.484451293945,57.5679817199707,20.1173324584961,4.9076132774353,660.904724121094,3.22706604003906
2019-02-20T14:30:20.1709059+01:00,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN
2019-02-20T14:30:20.1884769+01:00,Valid,0.308903098106384,0.358667194843292,-99.0501708984375,210.287948608398,59.6802864074707,-43.362232208252,8.760910987854,663.158264160156,3.12155151367188,Valid,0.275484323501587,0.386370539665222,-116.514724731445,201.778533935547,56.5831069946289,20.1219387054443,4.90001535415649,660.958374023438,3.19432067871094

To run fillingaps.exe and save results to output.csv you can run any of these commands, they behave equally. Because of the Timestamp field being the DateTime string, we have to specify --timestamp-format argument with date value too:

fillingaps.exe input.csv --timestamp-format date --output output.csv 

fillingaps.exe input.csv --timestamp-format date --format CSV --output output.csv 

fillingaps.exe input.csv --timestamp-format date --format CSV --output-format CSV --output output.csv 

Contents of the output.csv:

Timestamp,LeftValidity,LeftGazePoint2DX,LeftGazePoint2DY,LeftGazePoint3DX,LeftGazePoint3DY,LeftGazePoint3DZ,LeftEyePosition3DX,LeftEyePosition3DY,LeftEyePosition3DZ,LeftPupilDiameter,RightValidity,RightGazePoint2DX,RightGazePoint2DY,RightGazePoint3DX,RightGazePoint3DY,RightGazePoint3DZ,RightEyePosition3DX,RightEyePosition3DY,RightEyePosition3DZ,RightPupilDiameter
2019-02-20T14:30:19.8526299+01:00,Valid,0.249588146805763,0.363745391368866,-130.047988891602,208.728134155273,59.1125564575195,-41.76611328125,8.7668342590332,661.870300292969,3.10238647460938,Valid,0.216417774558067,0.350670248270035,-147.382720947266,212.744323730469,60.574333190918,21.4870147705078,4.3505334854126,660.007873535156,3.25660705566406
2019-02-20T14:30:19.8693049+01:00,Valid,0.247222796082497,0.358571588993073,-131.284103393555,210.317321777344,59.6909790039063,-41.8769149780273,8.7861385345459,661.918762207031,3.11991882324219,Valid,0.229601517319679,0.363623380661011,-140.492935180664,208.765609741211,59.1261978149414,21.4036102294922,4.41953754425049,659.91650390625,3.25010681152344
2019-02-20T14:30:19.8859769+01:00,Valid,0.236037373542786,0.344207912683487,-137.129577636719,214.729309082031,61.2968101501465,-41.9852180480957,8.88303756713867,662.0673828125,3.09461975097656,Valid,0.222043097019196,0.367220818996429,-144.442947387695,207.660614013672,58.7240104675293,21.3197116851807,4.52923631668091,660.086669921875,3.21722412109375
2019-02-20T14:30:19.9026389+01:00,Valid,0.253329932689667,0.359348505735397,-128.092529296875,210.078689575195,59.6041221618652,-42.0834197998047,8.90143585205078,661.941101074219,3.08680725097656,Valid,0.222759857773781,0.348396211862564,-144.068359375,213.442825317383,60.8285675048828,21.2234134674072,4.60443639755249,659.994323730469,3.26002502441406
2019-02-20T14:30:19.9193049+01:00,Valid,0.244058474898338,0.356473416090012,-132.937774658203,210.961807250977,59.9255523681641,-42.200813293457,8.89363098144531,662.165649414063,3.07200622558594,Valid,0.225430101156235,0.364816665649414,-142.672912597656,208.399078369141,58.992790222168,21.1293144226074,4.62313508987427,660.202392578125,3.27789306640625
2019-02-20T14:30:19.9354099+01:00,Valid,0.269463747739792,0.37244439125061,-119.661056518555,206.056121826172,58.1400260925293,-42.3988380432129,8.91371726989746,662.519775390625,3.13050842285156,Valid,0.259013205766678,0.35579439997673,-125.12247467041,211.170379638672,60.00146484375,20.9335346221924,4.64631128311157,660.319458007813,3.47819519042969
2019-02-20T14:30:19.9520849+01:00,Invalid,0,0,0,0,0,0,0,0,0,Valid,0.252346277236939,0.439275193214416,-128.606591796875,185.528225708008,50.6684701919556,20.7854888916016,4.68599262237549,660.346154785157,3.3577392578125
2019-02-20T14:30:19.9687419+01:00,Invalid,0,0,0,0,0,0,0,0,0,Valid,0.245679348707199,0.522755986452102,-132.09070892334,159.886071777344,41.3354755401611,20.6374431610108,4.7256739616394,660.3728515625,3.23728332519531
2019-02-20T14:30:19.9853949+01:00,Invalid,0,0,0,0,0,0,0,0,0,Valid,0.23901242017746,0.606236779689789,-135.574826049804,134.24391784668,32.0024808883667,20.4893974304199,4.76535530090332,660.399548339844,3.11682739257813
2019-02-20T14:30:20.0020669+01:00,Invalid,0,0,0,0,0,0,0,0,0,Valid,0.23234549164772,0.689717572927475,-139.058943176269,108.601763916016,22.6694862365722,20.3413516998291,4.80503664016723,660.426245117187,2.99637145996094
2019-02-20T14:30:20.0187439+01:00,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,Valid,0.225678563117981,0.773198366165161,-142.543060302734,82.9596099853516,13.3364915847778,20.1933059692383,4.84471797943115,660.452941894531,2.87591552734375
2019-02-20T14:30:20.0354199+01:00,Invalid,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,Valid,0.271445840597153,0.398023754358292,-118.625213623047,198.199096679688,55.2803001403809,20.3727340698242,4.92081499099731,660.839416503906,3.12071228027344
2019-02-20T14:30:20.0520849+01:00,Valid,0.299531936645508,0.409739166498184,-103.94751739502,194.600570678711,53.9705390930176,-42.9782333374023,8.91861820220947,662.953125,2.94776916503906,Valid,0.28294438123703,0.36534595489502,-112.616119384766,208.23649597168,58.9336166381836,20.3946361541748,4.90711498260498,660.902770996094,3.17532348632813
2019-02-20T14:30:20.0687609+01:00,Valid,0.300280123949051,0.344572871923447,-103.556510925293,214.617202758789,61.2560081481934,-43.0859375,8.9193172454834,662.773010253906,3.09466552734375,Valid,0.292255342006683,0.346888810396194,-107.750228881836,213.905838012695,60.9970893859863,20.3046340942383,4.94291496276855,660.793640136719,3.25553894042969
2019-02-20T14:30:20.0854289+01:00,Valid,0.30443349480629,0.345005482435226,-101.385963439941,214.484329223633,61.2076416015625,-43.1788368225098,8.91031265258789,662.943725585938,3.12303161621094,Valid,0.288352221250534,0.343738108873367,-109.789993286133,214.873611450195,61.3493309020996,20.2462310791016,4.9574146270752,660.925048828125,3.24212646484375
2019-02-20T14:30:20.1020969+01:00,Valid,0.304627418518066,0.333515554666519,-101.284622192383,218.013595581055,62.4921951293945,-43.237133026123,8.86701774597168,662.796936035156,3.07875061035156,Valid,0.288599461317062,0.36536380648613,-109.660797119141,208.231018066406,58.9316215515137,20.1889362335205,4.94511127471924,660.793151855469,3.24139404296875
2019-02-20T14:30:20.1187519+01:00,Valid,0.309577882289886,0.337784349918365,-98.6975250244141,216.702377319336,62.014949798584,-43.3080368041992,8.82761573791504,663.017211914063,3.05610656738281,Valid,0.291314989328384,0.367952406406403,-108.241668701172,207.435897827148,58.642219543457,20.1552352905273,4.94071245193481,660.792846679688,3.31416320800781
2019-02-20T14:30:20.1360949+01:00,Valid,0.300848722457886,0.335498183965683,-103.25936126709,217.404602050781,62.2705383300781,-43.3376350402832,8.78061771392822,663.044555664063,3.07589721679688,Valid,0.282489657402039,0.379741191864014,-112.85375213623,203.814819335938,57.3242568969727,20.1322326660156,4.92041683197021,660.8388671875,3.24394226074219
2019-02-20T14:30:20.1538459+01:00,Valid,0.304312169551849,0.36512616276741,-101.449371337891,208.304000854492,58.9581871032715,-43.3578338623047,8.76451301574707,663.096130371094,3.09585571289063,Valid,0.283008337020874,0.377561151981354,-112.58268737793,204.484451293945,57.5679817199707,20.1173324584961,4.9076132774353,660.904724121094,3.22706604003906
2019-02-20T14:30:20.1709059+01:00,Valid,0.306607633829116,0.361896678805351,-100.249771118164,209.295974731445,59.3192367553711,-43.3600330352783,8.76271200180054,663.127197265625,3.10870361328125,Valid,0.279246330261231,0.381965845823288,-114.548706054688,203.131492614746,57.0755443572998,20.1196355819702,4.90381431579589,660.931549072266,3.210693359375
2019-02-20T14:30:20.1884769+01:00,Valid,0.308903098106384,0.358667194843292,-99.0501708984375,210.287948608398,59.6802864074707,-43.362232208252,8.760910987854,663.158264160156,3.12155151367188,Valid,0.275484323501587,0.386370539665222,-116.514724731445,201.778533935547,56.5831069946289,20.1219387054443,4.90001535415649,660.958374023438,3.19432067871094

Unfortunately, the project's readme is out of date and Wiki would be helpful too. I have opened issue #10 for that, and have unfinished work left in the feature-documentation branch. Hopefully, I can get back to the work on this soon and finish it.

You can find all available options for each filter using the --help argument.

Please, if you have still problem with this, or encounter any other issue, report it.

[isabelhurley]

Thanks, that's been very helpful.
Could you also specify what formats are acceptable for the timestamp, and if piping commands is still possible given csv files?

[martinkonopka]

Accepted formats for the timestamp and their aliases are:

• date, d, or dt - date and time string in the round-trip "o" format (see MSDN docs)
• time, t, or tm - time string in the Constant "c" format (see MSDN docs)
• ticks, tick, k, c - long integer value representing ticks from an arbitrary point in time in selected unit which can be specified with ticks:[precision], e.g. ticks:us. Available precisions:
  • us - microsecond (default precision)
  • ms - millisecond
  • 100ns, ns - hundreds of nanosecond, the highest precision for the internal DateTimeOffset format used in the library.

Examples for timestamp format:

• --timestamp-format ticks - sets to the ticks in microseconds;
• --timestamp-format ticks:ns - sets to the hundreds of nanosecond;
• --timestamp-format time - sets to the time string.

For your other question about piping commands - Yes, it is still possible, but with some caveats. Use --quiet argument to suppress log messages. Log messages are currently written to standard output stream (not error), so they are piped too and it produces error in reading the input from pipe. This was changed in previous updates and an issue #17 is open to fix that.
Each filter logs how many samples were on input and output. The log can be also redirected to a file with --log [PATH] argument.

Note, if you use a custom timestamp format, you have to specify it for each command in the pipeline.
For the file format, default is JSON, so the filters will use this format when chained using the pipes, if not specified otherwise. You can use CSV with pipelines too, but have to always add --format CSV and --output-format CSV to each command (disadvantage). But CSV serialization is faster (advantage). For example, you can chain fillingaps and select:

fillingaps.exe input.csv --timestamp-format date --quiet | select.exe --timestamp-format date --quiet --output-format CSV --output output.csv

The input is read by fillingaps as CSV file, then serialized into JSON, written to the output (pipe), then read by the select as JSON from the input (pipe), serialized into CSV and written to the output file. Explicit output format argument could be omitted, if a file with supported extension is specified.

To force CSV format through the pipe, you would use:

fillingaps.exe input.csv --timestamp-format date --quiet --output-format CSV | select.exe --format CSV --timestamp-format date --quiet --output-format CSV --output output.csv

Same applies for all other commands, you have to maintain the same --timestamp-format setting and suppress log messages with --quiet switch argument.

However, creating a pipeline is no longer necessary because complete filter is now available as an executable i-vt.exe. The same example from the project's readme but using the i-vt would then be:

i-vt.exe dataset.json --fillin --select Average --denoise --denoise-alpha 0.3 --frequency 60 --threshold 30 --merge --merge-max-gap 75 --discard --discard-min-duration 60 --output fixations.json

To change format and specify custom timestamp format, use the same arguments as before, e.g.:

i-vt.exe dataset.csv --format CSV --timestamp-format date --fillin --select Average --denoise --denoise-alpha 0.3 --frequency 60 --threshold 30 --merge --merge-max-gap 75 --discard --discard-min-duration 60 --output-format CSV --output fixations.csv

Arguments --fillin, --denoise, --merge, --discard are optional. If ommitted, these steps would not appear in the processing pipeline.
Arguments --select [EYE] and --frequency [SAMPLES_PER_SECOND] are required.

[martinkonopka]

Closing this issue since the readme was updated.