Massive suspend data clogs up the diff_suspend_data task

[christianp]

We had an attempt whose suspend data was over 20MB long because they wrote python code that printed like 5 million lines, and the output is saved in the pre-submit cache.

When a huey worker tried to diff the suspend data elements, it took so long that the database lock timed out. The attempt was still not diffed, so the next time the diff job came up a worker tried again. This ended up locking up all the workers.

We could try compressing very large suspend data instead of diffing it. Somewhere around 500KB might be a cut-off for compressing instead of diffing. I don't think it's feasible to diff two compressed values.

[christianp]

We should also consider making the programming extension truncate output after a certain point - maybe 1MB.