Skip to content

Commit

Permalink
Remove comment on doc attribute handling during sharding, as this is …
Browse files Browse the repository at this point in the history
…done by spaCy's slicing directly.
  • Loading branch information
rmitsch committed Nov 6, 2023
1 parent dddfaab commit 3af21b5
Showing 1 changed file with 0 additions and 6 deletions.
6 changes: 0 additions & 6 deletions spacy_llm/tasks/util/sharding.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,12 +68,6 @@ def map_doc_to_shards(
)
fraction /= 2

# todo doc properties, such as .ents, have to be included for some tasks (e. g. REL, EL) to work. how
# should this be done in cases where the properties transcend shard limits?
# - should sharding never cut across entities/other properties?
# - should entities or all other properties be dropped if they transcend shard properties? this seems
# like the most pragmatic solution for now.
# - which properties should be copied to shards other than .ents?
assert shard is not None
shards.append(shard)
fraction = 1
Expand Down

0 comments on commit 3af21b5

Please sign in to comment.