Scheduled Messages

[Poofjunior]

Migrating discussion from Harp-Core-RP2040 repo.

Currently, the protocol does not support scheduled behavior, i.e: "do-this-at-some-future-time-X." Generally, having this behavior opens the door to simultaneous actions to occur across devices on the Harp clock, and it lets us perform certain device behaviors at specific times without PC-to-device communication latency.

One implementation would be to interpret a PC-to-device read/write message that includes the timestamp as a scheduled message to be executed at the specified time. Numerous events could be queued up on the device this way such that the device can perform certain tasks on exact times.

Here are some high-level specification details and edge-cases to keep in mind that we discussed at a recent meeting:

• device reply conditions upon receiving a scheduled message and what the reply contains
  • Do we reply upon receiving the message in addition to when we execute it?
• querying the state of the device's scheduled message queue
  • How many messages are in the queue?
• queued device messages that need to execute simultaneously
  • How do we handle this?
  • If we execute multiple messages within 32 microseconds, is that considered "simultaneous enough?"
  • Could this behavior be implementation-dependent? i.e: some devices can handle multiple simultaneous messages while others cannot?
• Canceling messages that haven't happened yet
  • Do we need to implement this? If yes, then...
  • How do we cancel a specific message? Does it have a unique identifier? How do we get this identifier if queuing a message does not issue a reply
• Related: how do we clear the queue?
  • Similar to canceling, do we need to implement this?
• Receiving scheduled messages who'd deadline has already elapsed
  • Is this an error, or do we just execute it ASAP
• How to indicate whether a device supports scheduled messages
  • Is there a register in the harp core that could represent the size of the buffer or zero if the device does not scheduled messages?
• Does this get added on top of the existing protocol or get pushed to the next protocol?

I'll chime in with my thoughts after this message.

[Poofjunior]

Here's a proposed specification based on our previous discussions to date:

Proposed Spec

PC-to-device messages that include a timestamp are interpreted by the device as messages to be executed no earlier than that specified time. If the message can be queued, no reply is issued to these PC-to-device messages until the message has been handled, in which case, a Harp reply message is issued. If the message cannot be queued, a Harp reply message is issued immediately indicating an error attempting to queue the message.

This behavior is added to the existing protocol. Devices that support it indicate that they do so by means of a specific register or a spec for a particular device that exists a priori. Attempting to send timestamped messages to a device that does not support scheduled messages results in undefined behavior for that device.

Handling edge cases

• queued device messages that need to execute simultaneously
  • First, we consider "within 32 microseconds" simultaneous.
  • We make support for this feature implementation specific, since it may only make sense to have this feature on certain boards.
    • Devices with enough bandwidth to execute multiple messages simultaneously simply execute them in sequence as fast as possible.
    • Devices without enough bandwidth reply with an error. Devices can easily identify this scenario by attempting to queue two messages with the same timestamp.
• Receiving a scheduled message who's deadline elapsed
  • In this case, the device handles is ASAP. Since all Harp Device replies issue the timestamp at which they executed the message, the PC could identify this reply timestamp as an error. It is the PC's responsibility to flag this missed deadline as an error.
• Canceling a message
  • I propose we do not support canceling a message. The need to do this suggests that the PC is closing the loop with a device, and the latency from pc-to-device will introduce the chance that some cancellation requests will just fail if they fall within a certain unbounded time period.

Downsides

According to the above spec, there's no way to throttle the flow rate of queued messages other than sending messages until you get an error. At that point, you can simply retry sending messages starting from the message that failed.

Accommodations

The required changes are:

PC Side

• PC must be able to issue read/write messages to a device that include the timestamp field.
• PC must be able to receive Harp replies that occur far later in the future (after a requested scheduled message is processed by the device).
• PC must be able to handle rejected messages

Harp Device Side

• Device must indicate, via register or register field, that it supports queuing messages. Devices that do not support this should have a way of indicating that. (This could be a read-error from a register that does not exist, a reply of no payload or payload of 0, etc.)
• Device must indicate, via register, how many bytes are left in the message buffer
• Device must buffer a certain number of bytes to act hold scheduled messages
• Device must implement a priority queue to insert new messages
• Device must issue a reply with an error message only if it cannot queue a scheduled message
• Device must handle a scheduled message at or later than the specified timestamp
• Upon the deadline of a scheduled message, the device must handle it like any other Harp read/write message (i.e: do the action, issue a reply, etc.)

[Poofjunior]

Cancelling Messages
One possible implementation: since all scheduled messages include a timestamp and a register, we use the aggregate timestamp and register as a way to index into the harp queue of messages to cancel a particular message.

Cancelling Everything
We need to be able to cancel all messages. Otherwise, the experimenter could restart a bonsai program with the prior state of the hardware devices affecting the next program.