Import Twitter ToS history

[clementbiron]

Twitter publishes its previous ToS https://twitter.com/fr/tos/previous and it will be interesting to add them in the OTA history.

[MattiSG]

Hopefully this will be done in April through our upcoming partnership with PGA 😃

[ckatzenbach]

Yes, once we are fully on board, we are happy to contribute this. Our Twitter archive goes back to 2006 (www.pga.hiig.de – full archive soon at https://github.com/PlatformGovernanceArchive). I think it makes sense that we get first fully on board with regard to the current OTA scraping process, then try to understand its peculiarities and also compare the results with our PGA method, before we bulk import our data. I'd like to understand the differences and commonalities first, otherwise the dataset might get a bit "dirty".

[MattiSG]

This issue is not related to the engine, but to specific collections.
Currently, the PGA collection is the reference for Twitter. Moving to PGA, whose team can then decide to handle however they see fit 🙂

[ckatzenbach]

Thanks, Matti – yes. We still need to find a way to merge history PGA data with the current and ongoing PGA/OTA data. We do not want to just merge without having a proper idea of the side /output effects of the two different collections methods. This is on our roadmap for this year.