Event submission rejected by django CSRF: Forbidden (Referer checking failed - no Referer.)

[razvan286]

Self-Hosted Version

24.8.0

CPU Architecture

x86_64

Docker Version

27.2.0

Docker Compose Version

2.29.2

Steps to Reproduce

Clone the self-hosted Sentry repository.

Sentry configuration:

cat sentry/config.yml:

###############
# Mail Server #
###############

# mail.backend: 'smtp'  # Use dummy if you want to disable email entirely
mail.host: 'smtp'

###################
# System Settings #
###################

# If this file ever becomes compromised, it's important to generate a new key.
# Changing this value will result in all current sessions being invalidated.
# A new key can be generated with `$ sentry config generate-secret-key`
system.secret-key: '*******'

################
# File storage #
################

# Uploaded media uses these `filestore` settings. The available
# backends are either `filesystem` or `s3`.

filestore.backend: 'filesystem'
filestore.options:
  location: '/data/files'
dsym.cache-path: '/data/dsym-cache'
releasefile.cache-path: '/data/releasefile-cache'

# The URL prefix in which Sentry is accessible
system.internal-url-prefix: 'http://web:9000'
symbolicator.enabled: true
symbolicator.options:
  url: "http://symbolicator:3021"

transaction-events.force-disable-internal-project: true

system.url-prefix: 'https://sentry.mydomain.com'

mail.host: 'smtp.mail.com'
mail.port: 587
mail.username: '[email protected]'
mail.password: '****'

---

cat sentry/sentry.conf.py:

# This file is just Python, with a touch of Django which means
# you can inherit and tweak settings to your hearts content.

from sentry.conf.server import *  # NOQA

BYTE_MULTIPLIER = 1024
UNITS = ("K", "M", "G")

def unit_text_to_bytes(text):
    unit = text[-1].upper()
    power = UNITS.index(unit) + 1
    return float(text[:-1]) * (BYTE_MULTIPLIER**power)

# Generously adapted from pynetlinux: https://github.com/rlisagor/pynetlinux/blob/e3f16978855c6649685f0c43d4c3fcf768427ae5/pynetlinux/ifconfig.py#L197-L223
def get_internal_network():
    import ctypes
    import fcntl
    import math
    import socket
    import struct

    iface = b"eth0"
    sockfd = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    ifreq = struct.pack(b"16sH14s", iface, socket.AF_INET, b"\x00" * 14)

    try:
        ip = struct.unpack(
            b"!I", struct.unpack(b"16sH2x4s8x", fcntl.ioctl(sockfd, 0x8915, ifreq))[2]
        )[0]
        netmask = socket.ntohl(
            struct.unpack(b"16sH2xI8x", fcntl.ioctl(sockfd, 0x891B, ifreq))[2]
        )
    except IOError:
        return ()
    base = socket.inet_ntoa(struct.pack(b"!I", ip & netmask))
    netmask_bits = 32 - int(round(math.log(ctypes.c_uint32(~netmask).value + 1, 2), 1))
    return "{0:s}/{1:d}".format(base, netmask_bits)


INTERNAL_SYSTEM_IPS = (get_internal_network(),)

DATABASES = {
    "default": {
        "ENGINE": "sentry.db.postgres",
        "NAME": "postgres",
        "USER": "postgres",
        "PASSWORD": "",
        "HOST": "postgres",
        "PORT": "",
    }
}

# You should not change this setting after your database has been created
# unless you have altered all schemas first
SENTRY_USE_BIG_INTS = True

# If you're expecting any kind of real traffic on Sentry, we highly recommend
# configuring the CACHES and Redis settings

###########
# General #
###########

# Instruct Sentry that this install intends to be run by a single organization
# and thus various UI optimizations should be enabled.
SENTRY_SINGLE_ORGANIZATION = True

SENTRY_OPTIONS["system.event-retention-days"] = int(
    env("SENTRY_EVENT_RETENTION_DAYS", "90")
)

#########
# Redis #
#########

# Generic Redis configuration used as defaults for various things including:
# Buffers, Quotas, TSDB

SENTRY_OPTIONS["redis.clusters"] = {
    "default": {
        "hosts": {0: {"host": "redis", "password": "", "port": "6379", "db": "0"}}
    }
}

#########
# Queue #
#########

# See https://develop.sentry.dev/services/queue/ for more
# information on configuring your queue broker and workers. Sentry relies
# on a Python framework called Celery to manage queues.

rabbitmq_host = None
if rabbitmq_host:
    BROKER_URL = "amqp://{username}:{password}@{host}/{vhost}".format(
        username="guest", password="guest", host=rabbitmq_host, vhost="/"
    )
else:
    BROKER_URL = "redis://:{password}@{host}:{port}/{db}".format(
        **SENTRY_OPTIONS["redis.clusters"]["default"]["hosts"][0]
    )


#########
# Cache #
#########

# Sentry currently utilizes two separate mechanisms. While CACHES is not a
# requirement, it will optimize several high throughput patterns.

CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.memcached.PyMemcacheCache",
        "LOCATION": ["memcached:11211"],
        "TIMEOUT": 3600,
        "OPTIONS": {"ignore_exc": True},
    }
}

# A primary cache is required for things such as processing events
SENTRY_CACHE = "sentry.cache.redis.RedisCache"

DEFAULT_KAFKA_OPTIONS = {
    "bootstrap.servers": "kafka:9092",
    "message.max.bytes": 50000000,
    "socket.timeout.ms": 1000,
}

SENTRY_EVENTSTREAM = "sentry.eventstream.kafka.KafkaEventStream"
SENTRY_EVENTSTREAM_OPTIONS = {"producer_configuration": DEFAULT_KAFKA_OPTIONS}

KAFKA_CLUSTERS["default"] = DEFAULT_KAFKA_OPTIONS

###############
# Rate Limits #
###############

# Rate limits apply to notification handlers and are enforced per-project
# automatically.

SENTRY_RATELIMITER = "sentry.ratelimits.redis.RedisRateLimiter"

##################
# Update Buffers #
##################

# Buffers (combined with queueing) act as an intermediate layer between the
# database and the storage API. They will greatly improve efficiency on large
# numbers of the same events being sent to the API in a short amount of time.
# (read: if you send any kind of real data to Sentry, you should enable buffers)

SENTRY_BUFFER = "sentry.buffer.redis.RedisBuffer"

##########
# Quotas #
##########

# Quotas allow you to rate limit individual projects or the Sentry install as
# a whole.

SENTRY_QUOTAS = "sentry.quotas.redis.RedisQuota"

########
# TSDB #
########

# The TSDB is used for building charts as well as making things like per-rate
# alerts possible.

SENTRY_TSDB = "sentry.tsdb.redissnuba.RedisSnubaTSDB"

#########
# SNUBA #
#########

SENTRY_SEARCH = "sentry.search.snuba.EventsDatasetSnubaSearchBackend"
SENTRY_SEARCH_OPTIONS = {}
SENTRY_TAGSTORE_OPTIONS = {}

###########
# Digests #
###########

# The digest backend powers notification summaries.

SENTRY_DIGESTS = "sentry.digests.backends.redis.RedisBackend"

###################
# Metrics Backend #
###################

SENTRY_RELEASE_HEALTH = "sentry.release_health.metrics.MetricsReleaseHealthBackend"
SENTRY_RELEASE_MONITOR = (
    "sentry.release_health.release_monitor.metrics.MetricReleaseMonitorBackend"
)

##############
# Web Server #
##############

SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    "protocol": "uwsgi",
    # This is needed in order to prevent https://github.com/getsentry/sentry/blob/c6f9660e37fcd9c1bbda8ff4af1dcfd0442f5155/src/sentry/services/http.py#L70
    "uwsgi-socket": None,
    "so-keepalive": True,
    # Keep this between 15s-75s as that's what Relay supports
    "http-keepalive": 15,
    "http-chunked-input": True,
    # the number of web workers
    "workers": 3,
    "threads": 4,
    "memory-report": False,
    # Some stuff so uwsgi will cycle workers sensibly
    "max-requests": 100000,
    "max-requests-delta": 500,
    "max-worker-lifetime": 86400,
    # Duplicate options from sentry default just so we don't get
    # bit by sentry changing a default value that we depend on.
    "thunder-lock": True,
    "log-x-forwarded-for": False,
    "buffer-size": 32768,
    "limit-post": 209715200,
    "disable-logging": True,
    "reload-on-rss": 600,
    "ignore-sigpipe": True,
    "ignore-write-errors": True,
    "disable-write-exception": True,
}

###########
# SSL/TLS #
###########

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
USE_X_FORWARDED_HOST = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

# End of SSL/TLS settings

########
# Mail #
########

SENTRY_OPTIONS["mail.list-namespace"] = env("SENTRY_MAIL_HOST", "localhost")
SENTRY_OPTIONS["mail.from"] = f"sentry@{SENTRY_OPTIONS['mail.list-namespace']}"

############
# Features #
############

SENTRY_FEATURES["projects:sample-events"] = False
SENTRY_FEATURES.update(
    {
        feature: True
        for feature in (
            "organizations:discover",
            "organizations:events",
            "organizations:global-views",
            "organizations:incidents",
            "organizations:integrations-issue-basic",
            "organizations:integrations-issue-sync",
            "organizations:invite-members",
            "organizations:metric-alert-builder-aggregate",
            "organizations:sso-basic",
            "organizations:sso-rippling",
            "organizations:sso-saml2",
            "organizations:performance-view",
            "organizations:advanced-search",
            "organizations:session-replay",
            "organizations:issue-platform",
            "organizations:profiling",
            "organizations:monitors",
            "organizations:dashboards-mep",
            "organizations:mep-rollout-flag",
            "organizations:dashboards-rh-widget",
            "organizations:metrics-extraction",
            "organizations:transaction-metrics-extraction",
            "projects:custom-inbound-filters",
            "projects:data-forwarding",
            "projects:discard-groups",
            "projects:plugins",
            "projects:rate-limits",
            "projects:servicehooks",
        )
        # Starfish related flags
        + (
            "organizations:deprecate-fid-from-performance-score",
            "organizations:indexed-spans-extraction",
            "organizations:insights-entry-points",
            "organizations:insights-initial-modules",
            "organizations:insights-addon-modules",
            "organizations:mobile-ttid-ttfd-contribution",
            "organizations:performance-calculate-score-relay",
            "organizations:standalone-span-ingestion",
            "organizations:starfish-browser-resource-module-image-view",
            "organizations:starfish-browser-resource-module-ui",
            "organizations:starfish-browser-webvitals",
            "organizations:starfish-browser-webvitals-pageoverview-v2",
            "organizations:starfish-browser-webvitals-replace-fid-with-inp",
            "organizations:starfish-browser-webvitals-use-backend-scores",
            "organizations:starfish-mobile-appstart",
            "projects:span-metrics-extraction",
            "projects:span-metrics-extraction-addons",
        )
        # User Feedback related flags
        + (
            "organizations:user-feedback-ingest",
            "organizations:user-feedback-replay-clip",
            "organizations:user-feedback-ui",
            "organizations:feedback-visible",
            "organizations:feedback-ingest",
            "organizations:feedback-post-process-group",
        )
    }
)

# Temporary flag to mark User Feedback to be produced to the dedicated feedback topic by relay.
# This will be removed at a later time after it's considered stable and fully rolled out.
SENTRY_OPTIONS["feedback.ingest-topic.rollout-rate"] = 1.0

#######################
# MaxMind Integration #
#######################

GEOIP_PATH_MMDB = "/geoip/GeoLite2-City.mmdb"

##############################################
# Suggested Fix Feature / OpenAI Integration #
##############################################

# See https://docs.sentry.io/product/issues/issue-details/ai-suggested-solution/
# for more information about the feature. Make sure the OpenAI's privacy policy is
# aligned with your company.

# Set the OPENAI_API_KEY on the .env or .env.custom file with a valid
# OpenAI API key to turn on the feature.
OPENAI_API_KEY = env("OPENAI_API_KEY", "")

SENTRY_FEATURES["organizations:open-ai-suggestion"] = bool(OPENAI_API_KEY)

##############################################
# Content Security Policy settings
##############################################

# CSP_REPORT_URI = "https://{your-sentry-installation}/api/{csp-project}/security/?sentry_key={sentry-key}"
CSP_REPORT_ONLY = True

# optional extra permissions
# https://django-csp.readthedocs.io/en/latest/configuration.html
# CSP_SCRIPT_SRC += ["example.com"]

#################
# CSRF Settings #
#################

# Since version 24.1.0, Sentry migrated to Django 4 which contains stricter CSRF protection.
# If you are accessing Sentry from multiple domains behind a reverse proxy, you should set
# this to match your IPs/domains. Ports should be included if you are using custom ports.
# https://docs.djangoproject.com/en/4.2/ref/settings/#std-setting-CSRF_TRUSTED_ORIGINS

CSRF_TRUSTED_ORIGINS = ["*", "https://*sentry.mydomain.com", "https://*", "http://*"]

#################
# JS SDK Loader #
#################

JS_SDK_LOADER_DEFAULT_SDK_URL = "https://browser.sentry-cdn.com/%s/bundle%s.min.js"


# If you would like to use self-hosted Sentry with only errors enabled, please set this
SENTRY_SELF_HOSTED_ERRORS_ONLY = env("COMPOSE_PROFILES") != "feature-complete"

Caddy configuration

I use Caddy for setting up the reverse proxy by adding the caddy container in a docker-compose.override.yml file.
cat docker-compose.override.yml:

services:
  caddy:
    image: caddy:2.8.4
    container_name: caddy
    restart: always
    ports:
      - 80:80  # Needed for the ACME HTTP-01 challenge.
      - 443:443
      - 443:443/udp # Needed for HTTP/3.
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy-config:/config
      - ./caddy-data:/data
    environment:
      DOMAIN: "sentry.mydomain.com"
      EMAIL: "[email protected]"  # The email address to use for ACME registration.
      LOG_FILE: "/data/access.log"

This is the Caddyfile available in the root directory of the cloned repo.

cat Caddyfile:

{
    debug
}

sentry.mydomain.com {

    log {
        output file /data/access.log
        format json
    }
    reverse_proxy web:9000 {
        health_uri /_health/
        health_status 2xx
        header_up Host {upstream_hostport}
    }

    # By default, the TLS is acquired from Let's Encrypt
    tls [email protected]

    # If you have self-signed certificate
    # tls /path/to/server-certificate.crt /path/to/server-certificate.key

    # To enable rate limiter, install additional module from
    # https://github.com/mholt/caddy-ratelimit
    # rate_limit {
    #     zone sentry {
    #         key {remote_host}
    #         window 1s
    #         events 100
    #     }
    # }

    # To expose only ingest endpoint publicly, add the named matcher below before `reverse_proxy` directive
    # @ingest_endpoint {
    #     path_regexp /api/[1-9]\d+/(envelope|minidump|security|store|unreal)/
    # }
}

Run ./install.sh and then spin up docker container docker compose up. All containers running then set up a Sentry python project on sentry.mydomain.com. Get the DSN and create the python script:

Test connection using a simple python script:

cat test-self-hosted-sentry.py:

import sentry_sdk

sentry_sdk.init(
    dsn="https://*****@sentry.mydomain.com/3",
    # Set traces_sample_rate to 1.0 to capture 100%
    # of transactions for tracing.
    traces_sample_rate=1.0,
)

division_by_zero = 1 / 0

Expected Result

A new issue is reported under the Sentry python project at URL: https://sentry.mydomain.com/organizations/sentry/projects/python/?project=3.

Actual Result

docker compose logs -f web:

web-1  | 11:07:26 [WARNING] django.security.csrf: Forbidden (Referer checking failed - no Referer.): /api/3/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/3/envelope/'>)
web-1  | 11:07:26 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/3/envelope/' caller_ip='*****' user_agent='sentry.python/2.13.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.018622398376464844 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')

Event ID

No response

[aldy505]

On your Caddyfile, the reverse_proxy directive should point to nginx, which binds to port 80.

self-hosted/docker-compose.yml

Line 455 in b8b4aa2

- "$SENTRY_BIND:80/tcp"

That should probably fix it. But I see your sentry.conf.py having this:

CSRF_TRUSTED_ORIGINS = ["*", "https://*sentry.mydomain.com", "https://*", "http://*"]

I'm not sure whether Django will accept that. If I were you, I'll just do this:

CSRF_TRUSTED_ORIGINS = ["http://sentry.mydomain.com", "https://sentry.mydomain.com"]

[razvan286]

Thank you for your response. I have updated the Caddyfile based on your suggestion, and also the CSRF_TRUSTED_ORIGINS value.
cat Caddyfile:

{
    debug
}

{$DOMAIN} {

    log {
        output file /data/access.log
        format json
    }

    reverse_proxy nginx:80 {
        health_uri /_health/
        health_status 2xx
        header_up Host {upstream_hostport}
    }
    # By default, the TLS is acquired from Let's Encrypt
    tls [email protected]
}

By checking the docker containers logs it seems that the requests now go through caddy and also nginx and relay until they reach the sentry instance (web-1). However, now there is a different error reported by django:

web-1                                           | 09:30:29 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/3/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/3/envelope/'>)
web-1                                           | 09:30:29 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/3/envelope/' caller_ip='****' user_agent='sentry.python/2.13.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.016865015029907227 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')

[aldy505]

By checking the docker containers logs it seems that the requests now go through caddy and also nginx and relay until they reach the sentry instance (web-1). However, now there is a different error reported by django:

web-1 | 09:30:29 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/3/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/3/envelope/'>)
web-1 | 09:30:29 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/3/envelope/' caller_ip='****' user_agent='sentry.python/2.13.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.016865015029907227 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')

@razvan286 after changing your sentry.conf.py file, most of the time you'll need to re-run ./install.sh script.

[razvan286]

@aldy505 yes, I always run the ./install.sh and then the docker compose up command.

[aldy505]

Oh I just realized this message

09:30:29 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/3/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/3/envelope/'>)

The /api/3/envelope should be pointed to the relay container instead of the web container.

Can you confirm from your Caddy instance that it really did a reverse proxy to the nginx container, and then correctly did a reverse proxy to the relay container?

To sum up (the name of the container):

caddy -> nginx -> relay

Then data ingested by relay should be published to kafka as individual messages, then it would be consumed by one of sentry worker (in which this becomes more complicated)

[razvan286]

@aldy505 Here is the full container output from when the request reaches caddy first and in the end the sentry instance that returns the CSRF missing token error:

caddy                                           | {"level":"debug","ts":1725955821.389444,"logger":"events","msg":"event","name":"tls_get_certificate","id":"fcf0f3b8-9a92-4ca3-8ae7-94fa1e16f8c7","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,49195,49199,52393,52392,49188,49192,49187,49191,159,158,107,103,255],"ServerName":"sentry.mydomain.com","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"****","Port":37110,"Zone":""},"LocalAddr":{"IP":"*****","Port":443,"Zone":""}}}}
caddy                                           | {"level":"debug","ts":1725955821.3895462,"logger":"tls.handshake","msg":"choosing certificate","identifier":"sentry.mydomain.com","num_choices":1}
caddy                                           | {"level":"debug","ts":1725955821.3895824,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"sentry.mydomain.com","subjects":["sentry.mydomain.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"eae69292c5535321b6dc540f182c76d910120f4a074a11bf66600357806d2d1b"}
caddy                                           | {"level":"debug","ts":1725955821.389603,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"*****","remote_port":"37110","subjects":["sentry.mydomain.com"],"managed":true,"expiration":1733305272,"hash":"eae69292c5535321b6dc540f182c76d910120f4a074a11bf66600357806d2d1b"}
caddy                                           | {"level":"debug","ts":1725955821.4079857,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"nginx:80","total_upstreams":1}
relay-1                                         | 2024-09-10T08:10:21.409512Z DEBUG request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: tower_http::trace::on_request: started processing request
relay-1                                         | 2024-09-10T08:10:21.410400Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-10T08:10:21.410441Z DEBUG relay_server::services::project: project 6c77219228f5973a3f9efc00978f35ae state requested 1 times
relay-1                                         | 2024-09-10T08:10:21.410558Z TRACE request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: relay_server::endpoints::common: Sending envelope to project cache for V1 buffer
relay-1                                         | 2024-09-10T08:10:21.410748Z DEBUG request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: tower_http::trace::on_response: finished processing request latency=1 ms status=200
relay-1                                         | 2024-09-10T08:10:21.411020Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-10T08:10:21.411044Z TRACE relay_server::services::project_cache: Enqueueing envelope
nginx-1                                         | *** - - [10/Sep/2024:08:10:21 +0000] "POST /api/3/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.13.0" "***"
caddy                                           | {"level":"debug","ts":1725955821.41155,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"nginx:80","duration":0.003337724,"request":{"remote_ip":"***","remote_port":"37110","client_ip":"***","proto":"HTTP/1.1","method":"POST","host":"nginx:80","uri":"/api/3/envelope/","headers":{"Content-Type":["application/x-sentry-envelope"],"X-Forwarded-Proto":["https"],"Accept-Encoding":["identity"],"Content-Length":["1079"],"X-Forwarded-Host":["sentry.mydomain.com"],"Content-Encoding":["gzip"],"User-Agent":["sentry.python/2.13.0"],"X-Sentry-Auth":["Sentry sentry_key=6c77219228f5973a3f9efc00978f35ae, sentry_version=7, sentry_client=sentry.python/2.13.0"],"X-Forwarded-For":["***"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"sentry.mydomain.com"}},"headers":{"Connection":["keep-alive"],"Access-Control-Allow-Origin":["*"],"Vary":["origin","access-control-request-method","access-control-request-headers"],"Cross-Origin-Resource-Policy":["cross-origin"],"Server":["nginx"],"Date":["Tue, 10 Sep 2024 08:10:21 GMT"],"Content-Type":["application/json"],"Content-Length":["41"],"Access-Control-Expose-Headers":["x-sentry-error,x-sentry-rate-limits,retry-after"]},"status":200}
relay-1                                         | 2024-09-10T08:10:21.512620Z DEBUG relay_server::services::project_upstream: updating project states for 1/1 projects (attempt 1)
relay-1                                         | 2024-09-10T08:10:21.512811Z DEBUG relay_server::services::project_upstream: sending request of size 1
relay-1                                         | 2024-09-10T08:10:21.539993Z DEBUG relay_server::services::project: project state 6c77219228f5973a3f9efc00978f35ae updated
nginx-1                                         | 172.18.0.57 - - [10/Sep/2024:08:10:27 +0000] "GET /_health/ HTTP/1.1" 200 12 "-" "Go-http-client/1.1" "-"
caddy                                           | {"level":"info","ts":1725955827.935051,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"host is up","host":"nginx:80"}
relay-1                                         | 2024-09-10T08:10:35.557499Z TRACE relay_server::services::metrics::aggregator: flushing 1 partitions to receiver
relay-1                                         | 2024-09-10T08:10:35.557858Z TRACE relay_server::services::processor: sending envelope to sentry endpoint
relay-1                                         | 2024-09-10T08:10:35.558493Z TRACE relay_server::services::cogs: recording measurement: CogsMeasurement { resource: Relay, feature: MetricsSessions, value: Time(724.815µs) }
web-1                                           | 08:10:35 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/1/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/1/envelope/'>)
web-1                                           | 08:10:35 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/1/envelope/' caller_ip='' user_agent='sentry-relay/24.8.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.03174233436584473 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')

[aldy505]

That looks fine to me, the error you're trying to send from the Python app went through the projectId of 3 (as seen on /api/3/envelope). Yet the web container is trying to make a request to /api/1/envelope, in which I guess there is a frontend or backend error happening from the web container, but it wrongly being sent to web container instead of the relay or nginx container.

I hope this illustration explain the situation better:

For now, my suggestion is to modify the docker-compose.yml on this section:

self-hosted/docker-compose.yml

Lines 43 to 62 in b8b4aa2

	environment:
	PYTHONUSERBASE: "/data/custom-packages"
	SENTRY_CONF: "/etc/sentry"
	SNUBA: "http://snuba-api:1218"
	VROOM: "http://vroom:8085"
	# Force everything to use the system CA bundle
	# This is mostly needed to support installing custom CA certs
	# This one is used by botocore
	DEFAULT_CA_BUNDLE: &ca_bundle "/etc/ssl/certs/ca-certificates.crt"
	# This one is used by requests
	REQUESTS_CA_BUNDLE: *ca_bundle
	# This one is used by grpc/google modules
	GRPC_DEFAULT_SSL_ROOTS_FILE_PATH_ENV_VAR: *ca_bundle
	# Leaving the value empty to just pass whatever is set
	# on the host system (or in the .env file)
	COMPOSE_PROFILES:
	SENTRY_EVENT_RETENTION_DAYS:
	SENTRY_MAIL_HOST:
	SENTRY_MAX_EXTERNAL_SOURCEMAP_SIZE:
	OPENAI_API_KEY:

to be:

  environment:
    # ...
    OPENAI_API_KEY:
    SENTRY_DSN:

Then on your .env or .env.custom, set the SENTRY_DSN to the DSN of the project named "internal" on your self-hosted instance. The URL to accessing this should be http://your-sentry-instance.com/settings/your-organization-name/projects/internal/keys/

Re-run ./install.sh and let's see what will happen next

[razvan286]

I added the DSN value of the internal sentry project to the .env file. It looks like this: https://******@sentry.mydomain.com/1. Ran the ./install.sh and then docker compose up.

Logs right after I run the python script: docker compose logs -f:

caddy                                           | {"level":"debug","ts":1726043825.853992,"logger":"events","msg":"event","name":"tls_get_certificate","id":"466cbdef-1c4e-45b0-ad5e-c6df8e4e7705","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,49195,49199,52393,52392,49188,49192,49187,49191,159,158,107,103,255],"ServerName":"sentry.mydomain.com","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"****","Port":49580,"Zone":""},"LocalAddr":{"IP":"***","Port":443,"Zone":""}}}}
caddy                                           | {"level":"debug","ts":1726043825.8540905,"logger":"tls.handshake","msg":"choosing certificate","identifier":"sentry.mydomain.com","num_choices":1}
caddy                                           | {"level":"debug","ts":1726043825.854142,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"sentry.mydomain.com","subjects":["sentry.mydomain.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"eae69292c5535321b6dc540f182c76d910120f4a074a11bf66600357806d2d1b"}
caddy                                           | {"level":"debug","ts":1726043825.8541822,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"****","remote_port":"49580","subjects":["sentry.mydomain.com"],"managed":true,"expiration":1733305272,"hash":"eae69292c5535321b6dc540f182c76d910120f4a074a11bf66600357806d2d1b"}
caddy                                           | {"level":"debug","ts":1726043825.8728714,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"nginx:80","total_upstreams":1}
relay-1                                         | 2024-09-11T08:37:05.874100Z DEBUG request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: tower_http::trace::on_request: started processing request
relay-1                                         | 2024-09-11T08:37:05.875026Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.875193Z TRACE request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: relay_server::endpoints::common: Sending envelope to project cache for V1 buffer
relay-1                                         | 2024-09-11T08:37:05.875284Z DEBUG request{method=POST uri=/api/3/envelope/ version=HTTP/1.1}: tower_http::trace::on_response: finished processing request latency=1 ms status=200
relay-1                                         | 2024-09-11T08:37:05.875504Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.875532Z TRACE relay_server::services::project_cache: Sending envelope to processor
nginx-1                                         | **** - - [11/Sep/2024:08:37:05 +0000] "POST /api/3/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.13.0" "****"
caddy                                           | {"level":"debug","ts":1726043825.875985,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"nginx:80","duration":0.002951075,"request":{"remote_ip":"*****","remote_port":"49580","client_ip":"*****","proto":"HTTP/1.1","method":"POST","host":"nginx:80","uri":"/api/3/envelope/","headers":{"X-Forwarded-Proto":["https"],"Accept-Encoding":["identity"],"X-Forwarded-For":["****"],"User-Agent":["sentry.python/2.13.0"],"Content-Encoding":["gzip"],"X-Sentry-Auth":["Sentry sentry_key=6c77219228f5973a3f9efc00978f35ae, sentry_version=7, sentry_client=sentry.python/2.13.0"],"Content-Length":["1079"],"Content-Type":["application/x-sentry-envelope"],"X-Forwarded-Host":["sentry.mydomain.com"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"sentry.mydomain.com"}},"headers":{"Date":["Wed, 11 Sep 2024 08:37:05 GMT"],"Content-Type":["application/json"],"Access-Control-Expose-Headers":["x-sentry-error,x-sentry-rate-limits,retry-after"],"Cross-Origin-Resource-Policy":["cross-origin"],"Server":["nginx"],"Content-Length":["41"],"Connection":["keep-alive"],"Access-Control-Allow-Origin":["*"],"Vary":["origin","access-control-request-method","access-control-request-headers"]},"status":200}
relay-1                                         | 2024-09-11T08:37:05.875548Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.875738Z TRACE relay_server::services::processor: Processing error group
relay-1                                         | 2024-09-11T08:37:05.875780Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.875803Z TRACE relay_server::services::processor::event: processing json event
relay-1                                         | 2024-09-11T08:37:05.876220Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.876267Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.876726Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string
relay-1                                         | 2024-09-11T08:37:05.877681Z TRACE relay_server::services::processor: sending envelope to sentry endpoint
relay-1                                         | 2024-09-11T08:37:05.879065Z TRACE relay_server::services::cogs: recording measurement: CogsMeasurement { resource: Relay, feature: Errors, value: Time(3.160312ms) }
web-1                                           | 08:37:05 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/3/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/3/envelope/'>)
web-1                                           | 08:37:05 [INFO] sentry.access.api: api.access (method='POST' view='django.views.generic.base.RedirectView' response=403 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/3/envelope/' caller_ip='172.18.0.56' user_agent='sentry.python/2.13.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.058341026306152344 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')

It looks like there is the same CSRF cookie not set error while the request is only made to /api/3/envelope now.

Maybe also this relay log is important?

relay-1                                         | 2024-09-11T08:37:05.875504Z DEBUG relay_server::envelope: failed to parse sampling context error=invalid type: floating point `1`, expected a string

[razvan286]

Thank you again for your time and ideas @aldy505 . However, I am still blocked on this issue, is there anything else that I can do or also provide you with more information so that maybe you can guide me further towards finding a solution?