Ingest files to update fields

[miketaormina]

Edit: tagging @dyf since I'm not sure if anyone will see this otherwise

I've created a file for the AIBS SmartSPIM (still needs updating), but noticed that it can't then be parsed to make a new object:

In [57]: new_inst = instrument.Instrument.parse_file(json_fp)
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
Cell In [57], line 1
----> 1 new_inst = instrument.Instrument.parse_file(json_fp)

File ~/opt/anaconda3/envs/aind_schema/lib/python3.8/site-packages/pydantic/main.py:569, in pydantic.main.BaseModel.parse_file()

File ~/opt/anaconda3/envs/aind_schema/lib/python3.8/site-packages/pydantic/main.py:526, in pydantic.main.BaseModel.parse_obj()

File ~/opt/anaconda3/envs/aind_schema/lib/python3.8/site-packages/pydantic/main.py:342, in pydantic.main.BaseModel.__init__()

ValidationError: 3 validation errors for Instrument
motorized_stages
  'NoneType' object is not iterable (type=type_error)
additional_devices
  'NoneType' object is not iterable (type=type_error)
daq_ao_names_to_channel
  'NoneType' object is not iterable (type=type_error)

Even though these fields are optional. It seems like we should be able to parse one of these files, update a field, and create a new file as needed. Am I just doing it wrong?

Here is the raw text of my JSON file:

{
    "version": "0.1.0",
    "describedBy": "https://github.com/AllenNeuralDynamics/aind-data-schema/blob/main/src/aind-data-schema/imaging/instrument.py",
    "instrument_id": "SmartSPIM01",
    "microscope": {
        "type": "smartSPIM",
        "manufacturer": "LifeCanvas",
        "model": null,
        "serial_number": null,
        "notes": null,
        "location": "440 Westlake"
    },
    "temperature_control": false,
    "humidity_control": false,
    "vibration_control": false,
    "optical_table_model": "VIS2424-IG2-125A",
    "optical_table_size": 24.0,
    "objectives": [
        {
            "type": null,
            "manufacturer": null,
            "model": null,
            "serial_number": null,
            "notes": "Thorlabs TL4X-SAP with LifeCanvas dipping cap and correction optics.",
            "numerical_aperture": 0.2,
            "magnification": 3.6,
            "immersion": "multi"
        }
    ],
    "detectors": [
        {
            "type": "Camera",
            "manufacturer": null,
            "model": "C14440-20UP",
            "serial_number": "001107",
            "notes": null,
            "data_interface": "USB",
            "cooling": "air"
        }
    ],
    "light_sources": [
        {
            "type": "laser",
            "manufacturer": "Vortran",
            "model": null,
            "serial_number": "VL01222A11",
            "notes": null,
            "coupling": "SMF",
            "wavelength": 488.0,
            "max_power": 150.0,
            "filter_wheel_index": 0
        },
        {
            "type": "laser",
            "manufacturer": "Coherent",
            "model": null,
            "serial_number": "417927",
            "notes": null,
            "coupling": "SMF",
            "wavelength": 561.0,
            "max_power": 150.0,
            "filter_wheel_index": 1
        },
        {
            "type": "laser",
            "manufacturer": "Vortran",
            "model": null,
            "serial_number": "VL01222A10",
            "notes": null,
            "coupling": "SMF",
            "wavelength": 647.0,
            "max_power": 160.0,
            "filter_wheel_index": 2
        }
    ],
    "fluorescence_filters": [
        {
            "type": "Band pass",
            "manufacturer": "Semrock",
            "model": "FF03-525/50-25",
            "serial_number": null,
            "notes": null,
            "diameter": 25.0,
            "thickness": 2.0,
            "description": null
        },
        {
            "type": "Band pass",
            "manufacturer": "Semrock",
            "model": "FF01-600/52-25",
            "serial_number": null,
            "notes": null,
            "diameter": 25.0,
            "thickness": 2.0,
            "description": null
        },
        {
            "type": "Band pass",
            "manufacturer": "Chroma",
            "model": "ET690/50m",
            "serial_number": null,
            "notes": null,
            "diameter": 25.0,
            "thickness": 2.0,
            "description": null
        }
    ],
    "motorized_stages": null,
    "additional_devices": null,
    "calibration_date": null,
    "calibration_data": null,
    "daq_device_name": null,
    "daq_model_name": null,
    "daq_update_frequency": null,
    "daq_ao_names_to_channel": null,
    "com_ports": [
        {
            "hardware_name": "Laser Launch",
            "com_port": "COM3"
        },
        {
            "hardware_name": "ASI Tiger",
            "com_port": "COM5"
        },
        {
            "hardware_name": "MightyZap",
            "com_port": "COM4"
        }
    ],
    "notes": null
}

[dyf]

@miketaormina Can you send me the script that generated the JSON?

[miketaormina]

AIBS SmartSPIM schema creation script

This script has no input parameters -- you may want to change the output file path before running

[miketaormina]

I also found a bit of a bug in the while loop for saving a file -- the counter doesn't increment, so you could get stuck in a loop.

[dyf]

@miketaormina looks like you found a pydantic bug. pydantic/pydantic#4568

This is getting fixed in the next pydantic release (it's already been fixed in a branch). Looks like the class is saving to JSON correctly, so I think we don't need to do anything.