Issues writing to manifestdb

[jlashner]

Hi sotodlib experts, I'm trying to make a caldb based off of bias-step and iv data from sodetlib, and I'm running into an issue. Currently I'm trying to run the following function:

def update_bias_step_cal(ctx, smurf, idx_path, h5_path, log=None):
    if log is None:
        log = default_logger

    if not os.path.exists(idx_path):
        scheme = metadata.ManifestScheme()
        scheme.add_exact_match('obs:obs_id')
        scheme.add_data_field('dataset')
        db = metadata.ManifestDb(scheme=scheme)
        db.to_file(idx_path)
    else:
        db = metadata.ManifestDb(idx_path)
    
    existing_obsids = db.get_entries(['dataset'])['dataset']
    all_obsids = ctx.obsdb.query("type='obs'")['obs_id']
    remaining_obsids = \
        sorted(list(set(all_obsids) - set(existing_obsids)),
            key=(lambda s: s.split('_')[1]))

    for obs_id in remaining_obsids:
        try:
            ## Gets calibration results that correspond to this obs
            rset = get_bias_step_resset(obs_id)
        except Exception:
            log.error(f"Error running on {obs_id}")

      # This happens if there are no applicable results (i.e. if bias has been changed since taking bias_steps)
        if not len(rset):  
            continue

        log.info(f"writing dataset for {obs_id}")
        io_meta.write_dataset(rset, h5_path, obs_id, overwrite=True)
        if obs_id not in existing_obsids:
            db.add_entry({
                'obs:obs_id': obs_id,
                'dataset': obs_id,
            }, filename=h5_path,)

This runs well, however consistently, the first time its run, no files are actually added to the sqlite database, while the second time its run, files are added and I'm able to load metadata. I might be missing something, but can someone help me figure out what I'm doing wrong? Thanks!

[skhrg]

I had the same problem in the past. I think its because when it saves it makes a new database connection (see here).

I just tossed in a db = metadata.ManifestDb(idx_path) after the to_file and that worked for me. So in your case you could just get rid of the else.