You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We should be able to speed up feature generation a lot even without using windowing by:
Passing in or sniffing an end_date for old data. Choose start_date to be dt(1?, 7?)
days earlier than the date we care about.
Compute features only from start_date onward
Merge the features with old data
All of this may get simpler by switching to the simpler 5 minute rule that David uses. Just
pick one point out of every 5 minute interval of the day. (Perhaps use median across values,
or point most at Median time-wise).
It may be better / simpler to switch to sharding by date, then have a utility that converts date shards into mmsi shards (for some date range).
The text was updated successfully, but these errors were encountered:
We should be able to speed up feature generation a lot even without using windowing by:
Passing in or sniffing an
end_date
for old data. Choosestart_date
to bedt
(1?, 7?)days earlier than the date we care about.
Compute features only from
start_date
onwardMerge the features with old data
All of this may get simpler by switching to the simpler 5 minute rule that David uses. Just
pick one point out of every 5 minute interval of the day. (Perhaps use median across values,
or point most at Median time-wise).
It may be better / simpler to switch to sharding by date, then have a utility that converts date shards into mmsi shards (for some date range).
The text was updated successfully, but these errors were encountered: