Skip to content

Commit

Permalink
[BUG] PullLog Operator should update the next pull offset to be the l…
Browse files Browse the repository at this point in the history
…ast record offset + 1 of previous pull (#2056)

## Description of changes

*Summarize the changes made by this PR.*
 - Improvements & Bug fixes
- The PullLogOperator does not correctly update the offset after the
first fetch as it might return less records than the batch size.
 - New functionality
	 - ...

## Test plan
*How are these changes tested?*

- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust

## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
  • Loading branch information
Ishiihara authored May 3, 2024
1 parent 70eab1e commit e96c545
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion rust/worker/src/execution/operators/pull_log.rs
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,8 @@ impl Operator<PullLogsInput, PullLogsOutput> for PullLogsOperator {
}

num_records_read += logs.len();
offset += batch_size as i64;
// unwrap here is safe because we just checked if empty
offset = logs.last().unwrap().log_offset + 1;
result.append(&mut logs);

// We used a a timestamp and we didn't get a full batch, so we have retrieved
Expand Down

0 comments on commit e96c545

Please sign in to comment.