[BUG] Merge does not work properly with required variables

[m1ker1n]

Description

I suppose when starting application with docker compose -f compose.yaml -f local.yaml up interpolation happens before merging compose files.

Since that if compose.yaml has required environment variable which must be overrode by local.yaml, docker compose up will fail.

My expectation:
docker compose up will be applied to config from docker compose -f compose.yaml -f local.yaml config --no-interpolate.

Steps To Reproduce

Compose files:

#compose.yaml
services:
  postgres:
    image: postgres
    environment:
      POSTGRES_USER: ${PG_USER?PG_USER must be set}
      POSTGRES_PASSWORD: ${PG_PASSWORD?PG_PASSWORD must be set}
      POSTGRES_DB: ${PG_DB?PG_DB must be set}

#local.yaml
services:
  postgres:
    environment: !override
      POSTGRES_USER: ${PG_USER:-postgres}
      POSTGRES_PASSWORD: ${PG_PASSWORD:-postgres}
      POSTGRES_DB: ${PG_DB:-postgres}

Merged config with no interpolation (which is what I want) docker compose -f .\compose.yaml -f .\local.yaml config --no-interpolate:

name: merge-bug
networks:
  default:
    name: merge-bug_default
services:
  postgres:
    environment:
      POSTGRES_DB: ${PG_DB:-postgres}
      POSTGRES_PASSWORD: ${PG_PASSWORD:-postgres}
      POSTGRES_USER: ${PG_USER:-postgres}
    image: postgres
    networks:
      default: null

What I get with docker compose -f compose.yaml -f local.yaml up:

error while interpolating services.postgres.environment.POSTGRES_DB: required variable PG_DB is missing a value: PG_DB must be set

Compose Version

Docker Compose version v2.27.0-desktop.2

Docker Environment

Client:
 Version:    26.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0-desktop.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)   
    Version:  0.0.29
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe 
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe  scout: Docker Scout (Docker Inc.)
    Version:  v1.8.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 9
  Running: 2
  Paused: 0
  Stopped: 7
 Images: 16
 Server Version: 26.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521    
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.761GiB
 Name: docker-desktop
 ID: ca4bc2c3-5e6f-4c3b-a0e4-fd1e001a6724
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Registry Mirrors:
  https://mirror.gcr.io/
  https://daocloud.io/
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Anything else?

No response

[ndeloof]

re-opening as compose-spec/compose-go#651 was reverted

[ndeloof]

Earlier attempt to fix this issue introduced significant regression. I guess with current compose-go architecture we as stuck with this limitation. This will be considered if we eventually start working on next-gen compose architecture, but for now there's no such plan.