How to clone cloud project for review environments?

[iBobik]

Hi, for review environments I need to clone project wilt its database, including tenant database.

Current data import-export works only with content, but for proper testing I will need to test also with authentication, actions etc.

Is it possible? Or review apps workflow is possible only on a subproject level - content tables?

Btw, how deployment should look like to create subproject, copy database from live and apply migrations to this new project?

Can I have single deployment token for all projects?

[iBobik]

Btw, I love how Cloudflare did review apps super simple :-)
https://developers.cloudflare.com/pages/platform/preview-deployments/

[iBobik]

What are review apps:
https://docs.gitlab.com/ee/ci/review_apps/

[iBobik]

I would like to use Contember cloud like this:

$ firebase hosting:channel:deploy new-awesome-feature --expires 7d
Deployed to: https://myproject.new-awesome-feature.eu.contember.cloud/

https://firebase.google.com/docs/hosting/manage-hosting-resources

[honzasladek]

Hi @iBobik! Thanks for a very good questions. I'll in part answer however will raise a few questions to you as this is currently something we're looking more into.

1. Exporting tenant database is currently not supported and we know it's an issue. Thanks to your question we raised it's priority and plan to allow this ASAP.
2. Review apps workflow is possible but a bit complicated. (And I and David both agree that the way Firebase does it is awesome, would love to have something similar. And actually we're not that far away in terms of how things work.) One step in that direction is to revive stages (which Contember supports for some) and allow easily to clone your content DB to test for example new content on your website. (This would allow easy review apps for frontend.) Anyway yes, currently you can export your data, create new project (within same project group), import data (we actually support data transfer for this) and apply migrations to the new version. It will share tenant database - but you need to give your chosen users access (membership) to this project. That way you can safely test new features with only selected beta testers. However you need to create the project by hand in Selfcare at the moment. But things are moving this way fast - we already have an API to this (because we needed that for Studio) and thinking about allowing it to be used via CI.
3. You can't have a single deployment token for all projects. Do I understand it right that you would like to automatically create review app for each feature and that's what you're after and that's why you want this? It does make sense and I'll add it to requested features.

Feel free to reach out in DMs if we can help to go around current Cloud limitations. Happy to work with you and make Cloud better for everyone!

[iBobik]

Hi, just wanted to try it, but I cannot find a token which has permission to create projects.

{
  "errors": [
    {
      "message": "You are not allowed to create a project",
      "locations": [
        {
          "line": 2,
          "column": 3
        }
      ],
      "path": [
        "createProject"
      ],
      "extensions": {
        "code": "ForbiddenError"
      }
    }
  ],
  "data": {
    "createProject": null
  }
}

I also checked docs and found there is project:create CLI command mentioned, but is is not in the 1.3 version.

[honzasladek]

@iBobik Oh sorry, I see how my reply could be read this way. The command works internally and we use it but it required authentication through Selfcare which is currently not public API. We'll make it available & possible but can't commit to specific timeline yet. I'll bring it up on next week internal planning session.

[iBobik]

Aha, could I do it manually now? I haven't found it in the Selfcare.

[honzasladek]

@iBobik You want to create new project within the same project group, correct?

If so: go to Selfcare, click "Create Project" and in Advanced Settings you can see select "Project Group" where you can select already existing Project Group. Looks like this:

[iBobik]

Aha, I expected Project Group is something else :-)

[iBobik]

Thank you for answer and info.Yes, you understand me correctly.Sharing tenant database is not critical issue for this workflow, because most projects will not need to test interaction with tenant API. It is just nice to have.Most critical is to streamline content database cloning. Ideally to solve all parts in one CLI command - create subproject, transfer database to it from live, add selected members to it (all, or e.g. with a role "testers" on a live project) and deploy frontend to a predictable subdomain (so it can be easily linked).This could be probably done with current API, it is just about usability. Unusable feature means it will not be used, or incorrectly, so people will test new featurest on live.

[honzasladek]

(Edited you reply, not sure from where you answered but it included my reply also with some gibersish.)

Thank you for clarification. I does make a lot of sense and I'm raising it with the team how we can deliver it for you (and others).

Just a thought @ViliamKopecky is this something you'd like as well?

[ViliamKopecky]

Of course I would like this as well :)

[honzasladek]

Thought so :) Thanks!

[iBobik]

Another use-case good to cover:

I would like to protect live environment agains accidental db import, but allow import to review environments.

export const adminRole = acl.createRole('admin', {
  content: {
    import: process.env['CI_REF_NAME'] !== 'master'
  }
})

There should be some env variables on the cloud what can be used for things like this.

Useable also for configuring webhooks baseurl etc.

[ViliamKopecky]

Actions have support for variables - https://docs.contember.com/reference/engine/actions/definition/#using-variables-in-webhook-urls-and-headers.

But variable support for those acl rules would be great.

[honzasladek]

Really like your idea! Might not be that complicated @matej21 ? :)

[iBobik]

Hi, I am experimenting with automated DB import on stage environment and end up with this error:

Incompatible schema version (import version 2023-10-24-153813 does not match server version 2024-01-08-172131)
I want to test new features on the stage which involves also some schema changes, so I want to test if db will be migrated correctly when merged.

In Drupal and SQL database I usually added some schema changes and it's migrations scripts, commited it into the branch and let CI deploy it. It deployed code, imported DB dump from production (old schema) and then ran migration which migrated DB in the review environment. Then I was sure migration process will pass even on production when branch will be merged and the same migration script will be executed.

So what steps the CI script should do with Contember to test new schema in the staging environment?

This is my draft of the .gitlab-ci.yml which resulted in the error in contember data:import:

image:
  name: contember/cli:1.3
  entrypoint: [""]

variables:
  CONTEMBER_SKIP_VERSION_CHECK: 1

contember cloud backup:
  stage: test
  services:
    - docker:dind
  script:
    - cd contember
    - contember data:export contember://${CONTEMBER_SUBPROJECT}:${CONTEMBER_TOKEN}@api-${CONTEMBER_PROJECT}.eu.contember.cloud --include-system
  artifacts:
    name: backup
    untracked: true
  only:
    - master

contember cloud stage db:
  stage: test
  services:
    - docker:dind
  script:
    - cd contember
    - 'wget -O db.jsonl.gz --header "PRIVATE-TOKEN: $GL_API_TOKEN" "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/jobs/artifacts/master/raw/contember/thebeast.jsonl.gz?job=contember%20cloud%20backup"'
    - contember data:import db.jsonl.gz contember://${CONTEMBER_SUBPROJECT}:${CONTEMBER_TOKEN}@api-${CONTEMBER_PROJECT}.eu.contember.cloud --yes
  except:
    - master
  when: manual

contember cloud deploy: &deploy_cloud
  stage: deploy
  rules:
    - if: $CI_PIPELINE_SOURCE != "schedule" && $CI_COMMIT_BRANCH == "master"
  services:
    - docker:dind
  script:
    - cd contember
    - contember deploy contember://${CONTEMBER_SUBPROJECT}:${CONTEMBER_TOKEN}@${CONTEMBER_PROJECT}.eu.contember.cloud --yes
  environment:
    name: production

contember cloud deploy stage:
  <<: *deploy_cloud
  rules:
    - if: $CI_COMMIT_BRANCH != "master"
  environment:
    name: stage

[matej21]

Hi,
you can extend your CI pipeline to manage this more dynamically. Here's how you can do it:

• Start the Contember Server: Begin by starting up the Contember server in your CI environment.
• Execute Migrations Until a Specific Point: Use the --until option with your migration command to execute migrations up to a certain point. This is useful if you want to bring your database schema to a specific state that is compatible with the data you are about to import.
• Import Data from Production: Once the schema is prepared, import your production data. This step should now work without the schema version error.
• Run the Rest of the Migrations: After the import, run the remaining migrations without the --until option. This step will apply any new schema changes that are part of your new features.
• Export the Database: Post migration, export the database. This creates a snapshot of your database in its latest schema state.
  Import into Staging Environment: Finally, import this latest snapshot into your staging environment. Since the schema versions will now match, this should resolve the error you're facing.