Find a way to communicate the ordering of a file back with the existing listing table implementation

[zhuqi-lucas]

Is your feature request related to a problem or challenge?

This is the follow-up for:
#13874 (review)

We add support (order by / sort) for DataFrameWriteOptions, but when a user try to query the table which the file already ordered, we can't get info from the table.

We need to find a way to communicate the ordering of a file back with the existing listing table implementation.

Describe the solution you'd like

It is also conceivable that DataFusion itself could write custom metadata in paquet and other formats that support that custom metadata with the ordering, but that seems like we can use iceberg and other table formats.

Describe alternatives you've considered

No response

Additional context

No response

[zhuqi-lucas]

take

[alamb]

One way to do this would be to write DataFusion specific metadata into the files (e..g add something to https://docs.rs/parquet/latest/parquet/file/properties/struct.WriterProperties.html#method.key_value_metadata)

It would be great if we can avoid making something DataFusion specific. Maybe someone could do some research and find how other systems handle this