freshclam !Update failed for database: daily

[alexxroche]

# freshclam
Mon Mar 15 19:50:38 2021 -> !downloadFile: Unexpected response (429) from https://database.clamav.net/daily.cvd
Mon Mar 15 19:50:38 2021 -> !getcvd: Can't download daily.cvd from https://database.clamav.net/daily.cvd
Mon Mar 15 19:50:39 2021 -> Giving up on https://database.clamav.net...
Mon Mar 15 19:50:39 2021 -> !Update failed for database: daily
Mon Mar 15 19:50:39 2021 -> ^fc_update_databases: fc_update_database failed: HTTP GET failed (11)
Mon Mar 15 19:50:39 2021 -> !Database update process failed: HTTP GET failed (11)
Mon Mar 15 19:50:39 2021 -> !Update failed.

# freshclam -v
<snip>
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 429
< date: Mon, 15 Mar 2021 19:27:00 GMT
< content-type: text/plain; charset=UTF-8
< content-length: 16
< retry-after: 18919
< x-frame-options: SAMEORIGIN
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< set-cookie: __cfduid=d94db69550497dc647181bee958e87da91615836420; expires=Wed, 14-Apr-21 19:27:00 GMT; path=/; domain=.clamav.net; HttpOnly; SameSite=Lax
< cf-request-id: 08d8f430ed000032bcd28b1000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< strict-transport-security: max-age=15552000
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 630822fb1e2232bc-CDG
<
Time: 0.2s, ETA: 0.0s [=============================>] 16B/16B
* Connection #0 to host database.clamav.net left intact
Mon Mar 15 20:27:00 2021 -> ^downloadFile: Unexpected response (429) from https://database.clamav.net/daily.cvd
Mon Mar 15 20:27:00 2021 -> ^getcvd: Can't download daily.cvd from https://database.clamav.net/daily.cvd

# freshclam --list-mirrors
WARNING: Deprecated option --list-mirrors. Individual mirrors are no longer tracked, as official signature distribution is now done through the CloudFlare CDN.

"Well there's your problem..." #lets_not_put_all_of_our_eggs_in_one_basket

# uname -mrsp
Linux 4.12.0-1-amd64 x86_64 unknown

# clamconf -n
Checking configuration files in /etc/clamav

Config file: clamd.conf
-----------------------
PreludeAnalyzerName = "ClamAV"
LogFile = "/var/log/clamav/clamav.log"
LogFileMaxSize = "4294967295"
LogTime = "yes"
LogRotate = "yes"
ExtendedDetectionInfo = "yes"
LocalSocket = "/var/run/clamav/clamd.ctl"
LocalSocketGroup = "clamav"
LocalSocketMode = "666"
MaxConnectionQueueLength = "15"
MaxThreads = "12"
ReadTimeout = "180"
SendBufTimeout = "200"
SelfCheck = "3600"
User = "clamav"
BytecodeTimeout = "60000"
MaxScanTime = "120000"
PCREMatchLimit = "10000"
PCRERecMatchLimit = "5000"

Config file: freshclam.conf
---------------------------
LogFileMaxSize = "4294967295"
LogTime = "yes"
LogRotate = "yes"
UpdateLogFile = "/var/log/clamav/freshclam.log"
Checks = "24"
DatabaseMirror = "db.local.clamav.net", "database.clamav.net"
MaxAttempts = "5"
ReceiveTimeout = "30"

clamav-milter.conf not found
Software settings
-----------------
Version: 0.102.4
Optional features supported: MEMPOOL IPv6 FRESHCLAM_DNS_FIX AUTOIT_EA06 BZIP2 LIBXML2 PCRE2 ICONV JSON

Database information
--------------------
Database directory: /var/lib/clamav
main.cld: version 59, sigs: 4564902, built on Mon Nov 25 14:56:15 2019
bytecode.cld: version 331, sigs: 94, built on Thu Sep 19 18:12:33 2019
daily.cvd: version 25887, sigs: 3681654, built on Tue Jul 28 17:44:20 2020
Total number of signatures: 8246650

Platform information
--------------------
uname: Linux 4.12.0-1-amd64 #1 SMP Debian 4.12.6-1 (2017-08-12) x86_64
OS: linux-gnu, ARCH: x86_64, CPU: x86_64
Full OS version: Debian GNU/Linux 10 (buster)
zlib version: 1.2.11 (1.2.11), compile flags: a9
platform id: 0x0a2173730800000000080300

Build information
-----------------
GNU C: 8.3.0 (8.3.0)
CPPFLAGS: -Wdate-time -D_FORTIFY_SOURCE=2
CFLAGS: -g -O2 -fdebug-prefix-map=/build/clamav-sSz0eR/clamav-0.102.4+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64  -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
CXXFLAGS: -g -O2 -fdebug-prefix-map=/build/clamav-sSz0eR/clamav-0.102.4+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64
LDFLAGS: -Wl,-z,relro -Wl,-z,now -Wl,--as-needed
Configure: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--libexecdir=/usr/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode' '--disable-dependency-tracking' 'CFLAGS=-g -O2 -fdebug-prefix-map=/build/clamav-sSz0eR/clamav-0.102.4+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fdebug-prefix-map=/build/clamav-sSz0eR/clamav-0.102.4+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now -Wl,--as-needed' '--with-dbdir=/var/lib/clamav' '--sysconfdir=/etc/clamav' '--disable-clamav' '--disable-unrar' '--enable-milter' '--enable-dns-fix' '--with-libjson' '--with-system-libmspack' '--with-libcurl=/usr' '--with-gnu-ld' '--with-systemdsystemunitdir=/lib/systemd/system' 'build_alias=x86_64-linux-gnu' 'OBJCFLAGS=-g -O2 -fdebug-prefix-map=/build/clamav-sSz0eR/clamav-0.102.4+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security'
sizeof(void*) = 8
Engine flevel: 115, dconf: 115

# ldd --version
ldd (Debian GLIBC 2.28-10) 2.28

# freshclam -V
ClamAV 0.102.4/25887/Tue Jul 28 17:44:20 2020

Out of curiosity I opened https://database.clamav.net/daily.cvd in firefox and after cloudfair faffed about for most of a minute I was presented with:

 Error 1015 Ray ID: 6308357a9c5d331e • 2021-03-15 19:39:38 UTC
You are being rate limited
What happened?

The owner of this website (database.clamav.net) has banned you temporarily from accessing this website.

"not exactly poggers".

[micahsnyder]

Hi @alexxroche

The Error 429 and message from Firefox "You are being rate limited" indicates that your IP address is on a sort of cool-down for trying to download the same files from database.clamav.net too frequently.

Background: A growing number of people had been using wget and similar tools to frequently download the whole database set, or are deploying containers or VMs that don't contain a baseline set of databases and thus when started download the whole database sets. This has become increasingly costly for the ClamAV project. So the ClamAV project is making a best effort to require users to use FreshClam to update existing databases and to not download the entire database set unless absolutely necessary. To that end, programs like wget have been blocked entirely (error code 403) and the whole database files like daily.cvd, main.cvd, and bytecode.cvd are being rate limited (error code 429).

So it might not be your fault exactly. But you can take steps to prevent being affected. If you're using a public cloud or are behind a large NAT, it's likely that others are frequently downloading the whole database files and triggering the rate limiting.

The update patch files (the *.cdiff files) are not rate limited, so good netizens can easily update their existing ClamAV installs by running FreshClam.

If you work for a large organization has a lot of ClamAV installations, it might be a good idea to set up a private mirror for your organization. This will also eliminate the rate limiting issue, and will help save the ClamAV project some $$. See https://www.clamav.net/documents/private-local-mirrors and https://pypi.org/project/cvdupdate/ for more details.

[alexxroche]

If you're using a public cloud or are behind a large NAT,

I'm on a dedicated single static IPv4 and an IPv6 /64. I use NAT for the IPv4 stack of my LAN, but I control every node within that LAN and ensure that no more than two machines are trying to download daily.cvd or main.cvd (and any other device can then be synchronised using rsync or by reading the cvd files from the LAN's NAS.)

it might be a good idea to set up a private mirror

I agree. (Which is why I've been doing that since 1999 with anti-virus databases and why I propose at each company that I work that they should be good netizens and offer to mirror important projects such as ClamAV. I believe that every ISP should have their own mirror of ClamAV specifically so that we don't have a single point of failure such as a CDN that could be silently dropping requests without anyone noticing. How can we as the ClamAV community know how many ClamAV instillation are now out of date because they are unable to update?)

[micahsnyder]

@alexxroche are you still having issues downloading signature updates?

[alexxroche]

Not currently.
I found a copy of main.cld that wasn't on the CDN and downloaded and installed it. I just tested freshclam and it updated from 0.102.4 to 0.103.2 without error. (I still think that freshclam should fall back to using the old network of mirrors if the CDN times out N times.

[toxi22]

@alexxroche are you still having issues downloading signature updates?

I'm still getting database update error via freshclam