not all Get operations taken into account

[davidoesch]

Hi
IMHO you have to take into account all operations flag, not only "WEBSITE.GET."
missing
"REST.GET.OBJECT"
"SOAP.GET.“

Otherwise you will miss m2m requests..

[metaodi]

@davidoesch I guess you're refering to the parse code, which currently only takes REST.GET.OBJECT into account.

I'm not really an expert, I guess the reason for this is to filter out "true" downloads of a file. Does WEBSITE.GET and SOAP.GET have a similar meaning?

[davidoesch]

according to http://docs.aws.amazon.com/AmazonS3/latest/dev/LogFormat.html

a SOAP or a REST based download is IMHO a download as well... and we have some datasets on data.geo.admin.ch which are only consumed by REST

[davidoesch]

another solution is to download all the log files, merge/cat them into one file and then do some AWK magic
grep -E "REST.GET.OBJECT|WEBSITE.GET|SOAP.GET" YOURLOG.txt | awk '$13 ~ /200/ {array_bytes[$5 " " $9] +=$15;array_counter[$5 " " $9]++;} END {for (i in array_bytes) {printf "%s %.0f %.0f\n",i,array_counter[i],array_bytes[i]}}' > YOURRESULTS.log