Remove generated files before build and detect modified files

[salkinium]

This enhances the lbuild build and lbuild clean commands to detect modified generated files before deletion, via modified timestamp comparison (and content hash as backup) in the buildlog. The commands fail if a modification is detected and users can lbuild clean --force to clear them by force.

TODO:

• Implement content change detection in buildlog
• Clean before building to remove files from previous configuration
• Testing

Fixes #33.

cc @dergraaf @WasabiFan

[salkinium]

Example errors:

[WasabiFan]

Nice!

If I am using source control and don't check in the build log, how will this behave?

[salkinium]

If I am using source control and don't check in the build log, how will this behave?

There used to be an option to generate a buildlog via silently simulating a build on the current config and then use that for deletion. However, that cannot detect any content changes and it's a fragile solution anyways.

[salkinium]

If there is no build log, then lbuild build will simply overwrite your files again, but it cannot remove the unnecessary ones. It's best to commit the build log too, but perhaps it could be made a little slimmer? This is currently not using source and module file hashes or build time.

[WasabiFan]

My initial reaction is to oppose checking in the build log. That being said, it isn't an educated opinion, it's rather just based on patterns I usually follow. My main reasoning is that the build log would be a single point of failure for collaboration, as if multiple people added modules when implementing their own features then tried to merge together, I imagine it would be easy to "break" the file (i.e., either render it unparsable or make it not match reality).

For my project at least, I'd probably prefer to wipe out the generated files and regenerate them all from scratch each time, but I also don't yet have any custom modifications to modm and don't know how that would work. In the past, the way I've preferred autogen systems to work is that they're a pure map of config -> code without any state, so it's harder to mess it up or get into an inconsistent state.

[salkinium]

multiple people added modules when implementing their own features then tried to merge together, I imagine it would be easy to "break" the file

Hm, though you may have merge conflicts in the generated code too, since wildly different modules/options can still change the same code location. But I get your point, one less file to worry about.

a pure map of config -> code without any state

That's a good point. It is possible to simulate the build and get the buildlog from that, however, only if the config still corresponds to config used for generating the files.

I'm kinda just looking for a "build receipt" of sorts, simply copying the project.xml config to project.xml.backup would suffice, however, you'd need to simulate a full build everytime you call, which can be annoying. I think I'll wait until I've implemented #37 and then see what the resulting capabilities of lbuild are and what information is really required, and what is derivable from config.

[WasabiFan]

I'm kinda just looking for a "build receipt" of sorts, simply copying the project.xml config to project.xml.backup would suffice, however, you'd need to simulate a full build everytime you call, which can be annoying

You could view this a bit differently -- think of it as including an inventory in the build output. Rather than a backup, it's the same information encoded as part of the output rather than the input.

---

Separately...

The build log has a strange mix of stuff. Some of it is like a manifest of the module contents, while other parts are actual log info (such as the build time). It would be helpful if it were made into just the former, so it was more conducive to version control if that's the goal. If it were both small and deterministic/consistent, that would make versioning it better.

Something to think about: as far as I can tell, most of this is about trying to support modifications to generated code. Some possible larger changes to the design/model which you could consider:

• Replace the build log with a module-level hash of the contents. Check the contents against the hashes to know what must be re-built. This would a) shrink that file and b) make it less weird.
• You could have a new way to mark (in the project file) modules or individual files which shouldn't be auto-generated. Overwrite everything else.