"row-wise" execution plans in static branches

[gofford]

Consider the following. I have a pipeline that can be depicted this. It is created using tar_map:

               D1 --> E1 --> F2 --> G2
             /                                   
           /
A -> B -> C - D2 --> E2 --> F3 --> G3
           \
             \
               D3 --> E3 --> F4 --> G4

The numbered branches are independent after C. Is there a way to configure this pipeline to execute the DAG row-wise rather than column-wise? I.e., to complete A->C as it normally would, then proceed to execute the 1st row (D1, E1, F1, G1) before moving to the 2nd row (D2, E2, F2, G2), etc.?

As far as I can tell, the targets DAG executes in a column-wise fashion (left-to-right), where D1, D2, D3 are completed before moving on to E1, E2 .. EN, and so on. This is generally fine, but when the serialised objects for the D steps are large (comparable >50% available disk on a remote machine), then this either incurs storage problems or massive network transfer to/from S3.

Being able to execute row-wise would alleviate this, but it's not clear if this is currently possible? Adjusting priority with the HPC back-ends such as tar_make_future is the closest I can see but this still seems to determine priority from left-to-right.

[wlandau]

Please have a look at #437. The ‘priority’ argument of tar_target() can handle this. Should work in the development version of the package.

[gofford]

Fantastic -- unclear how I missed that lengthy discussion. Great package, and awesome engagement!