Url preview endpoint

[adnull]

This PR introduces the missing URL Preview functionality ( Issue matrix-org/dendrite#1303 )

According to the specification that endpoint requests require authentication and are rate limited. There are both endpoints (v1 and v3) as some clients still use deprecated endpoints for url preview requests.

Key features:

• takes info from html pages or image links. Max image size downloadable is configurable. Timeout is configurable
• stores the url's image as a thumbnail in media storage if the thumbnail size for url previews is configured. Otherwise stores the original image (to make it sy-test compatible).
• uses the maxThumbnailGenerators variable to prevent overload. Waits 30s for free generator.
• keeps an in-memory cache of url preview responses. The cache entry expiration time is configurable
• stores the the url preview json responses in the media storage to prevent any extra load
• url preview blacklist is configurable as a list of regexps in the config

Missing:

• some tests
• synapse's ability to query/extract oEmbed resources
• ip blacklist checks

I'd like somebody to take a look at the code.

• I have added Go unit tests or Complement integration tests for this PR or I have justified why this PR doesn't need tests
• Pull request includes a sign off below using a legally identifiable name or I have already signed off privately

Signed-off-by: adnull <[email protected]>

[CLAassistant]

All committers have signed the CLA.

[S7evinK]

Quick notes:
We just merged e9cc37a, which makes me much more comfortable for something like this.

Also, I'd prefer URLPreviewDenyList over UrlPreviewBlacklist.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 71.33655% with 178 lines in your changes missing coverage. Please review.

Project coverage is 49.71%. Comparing base (f4506a0) to head (78281d5).

Files with missing lines	Patch %	Lines
mediaapi/routing/url_preview.go	72.19%	105 Missing and 41 partials ⚠️
internal/netcontext.go	60.86%	13 Missing and 5 partials ⚠️
mediaapi/thumbnailer/thumbnailer_nfnt.go	58.62%	9 Missing and 3 partials ⚠️
mediaapi/routing/download.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3444      +/-   ##
==========================================
+ Coverage   49.40%   49.71%   +0.30%     
==========================================
  Files         524      526       +2     
  Lines       59814    60434     +620     
==========================================
+ Hits        29554    30042     +488     
- Misses      26776    26856      +80     
- Partials     3484     3536      +52     

Flag	Coverage Δ
unittests	49.71% <71.33%> (+0.30%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[adnull]

Quick notes: We just merged e9cc37a, which makes me much more comfortable for something like this.

Also, I'd prefer URLPreviewDenyList over UrlPreviewBlacklist.

Implemented allow/deny lists checking from federation config. Maybe to move these lists to make them global?