Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: release candidate #1025

Open
wants to merge 65 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
beabb81
refactor: container configurations
m8vago Nov 6, 2024
839cc1f
fix: secret handling
m8vago Nov 25, 2024
76bc1de
fix: build
m8vago Nov 25, 2024
2a6dc53
refactor: remove unused ws messages
m8vago Nov 26, 2024
be6c022
fix: migration
m8vago Nov 27, 2024
b0c93cb
feat(web): api & ui
robot9706 Nov 27, 2024
95e13d6
fix: migration
m8vago Nov 27, 2024
54e16d0
fix: trying to fix config-bundle related tests
m8vago Nov 28, 2024
a6fe6d5
Merge branch 'develop' into feat/deployments-paginated
robot9706 Nov 28, 2024
4badf49
fix: lint
robot9706 Nov 28, 2024
63a2cb9
Merge branch 'feat/deployments-paginated' into refactor/configurations
robot9706 Nov 28, 2024
20ff448
fix: merge
robot9706 Nov 28, 2024
8a28439
feat: deploy list by configbundle
robot9706 Nov 28, 2024
66bfe63
feat: config bundle deployment list
robot9706 Nov 28, 2024
2a20744
feat: use pagination
robot9706 Nov 29, 2024
322ba35
Merge branch 'develop' into refactor/configurations
robot9706 Nov 29, 2024
9c6f837
fix: lint
robot9706 Nov 29, 2024
c2b38c9
fix: update test
robot9706 Nov 29, 2024
4bf2c8f
fix: go lint
robot9706 Dec 2, 2024
c5f654f
fix: deploy bug & separate config schema
robot9706 Dec 2, 2024
a2482b8
Revert "fix: deploy bug & separate config schema"
m8vago Dec 3, 2024
a780d10
fix: deployment null secrets
m8vago Dec 3, 2024
a942ad0
fix: config bundle e2e
m8vago Dec 3, 2024
0b37410
fix: golang unit tests
m8vago Dec 3, 2024
885cbd1
fix: setting prefix and container name
m8vago Dec 3, 2024
293a2e2
Merge branch 'develop' into refactor/configurations
m8vago Dec 3, 2024
353b164
fix: duplicated images
m8vago Dec 3, 2024
038495d
fix: container name defaults
m8vago Dec 3, 2024
8b21704
fix: storage saving
m8vago Dec 4, 2024
f42e87f
build: disable e2e
m8vago Dec 4, 2024
44d73ca
fix: remove e2e need
m8vago Dec 4, 2024
e6b8580
fix: lint
m8vago Dec 4, 2024
37755f0
Merge branch 'refactor/configurations' into feat/release-candidate
m8vago Dec 4, 2024
3d3afac
fix: go_push needs
m8vago Dec 4, 2024
7ed3f95
fix: retag_needs
m8vago Dec 4, 2024
3d1f2b8
feat: specify pipeline target branch
robot9706 Dec 4, 2024
900010d
feat: deployments list pagination & filtering
robot9706 Dec 4, 2024
447029f
build: remove e2e
m8vago Dec 9, 2024
630fe52
build: fix push requirements
m8vago Dec 9, 2024
9b080b1
build: fix push requirements
m8vago Dec 9, 2024
d4ae54e
build: fix needs
m8vago Dec 9, 2024
6f2ec6c
feat: docker labels & secret
robot9706 Dec 9, 2024
916ac57
fix: null exceptions in crux
m8vago Dec 9, 2024
59f3004
build: add deploy job
m8vago Dec 9, 2024
f62a28c
fix: ui label validation
robot9706 Dec 9, 2024
c0876dd
fix: rule validation
robot9706 Dec 10, 2024
9d5607c
feat: merge docker-labels
robot9706 Dec 12, 2024
413a0bf
fix: lint
robot9706 Dec 12, 2024
f2ce2fd
fix: ui label & secret validation
robot9706 Dec 12, 2024
c1a09d9
fix: minor ui fixes
robot9706 Dec 12, 2024
f58c601
fix: unable to reset storage section
robot9706 Dec 13, 2024
66ed46d
feat: versionName template
robot9706 Dec 14, 2024
e6bf95a
fix: config bundle storage conflict
robot9706 Dec 14, 2024
9ebd8c6
fix: template
robot9706 Dec 14, 2024
fd7eb2e
fix: null tag info
robot9706 Dec 16, 2024
912d850
fix: remove v2 http api debug logs
robot9706 Dec 16, 2024
3616c8d
fix: handle v2 http api exceptions
robot9706 Dec 16, 2024
fc97901
fix: batched v2 http api fetch & fix deployment update
robot9706 Dec 16, 2024
f59f9a6
fix: env merge & deploy start validation
robot9706 Dec 16, 2024
04a1cf2
fix: remove label processing
robot9706 Dec 16, 2024
3421b19
fix: deployment copy
robot9706 Dec 17, 2024
8504f57
Merge branch 'develop' into feat/release-candidate
m8vago Jan 14, 2025
2174896
revert: pipeline changes
m8vago Jan 14, 2025
4cb53fd
Merge branch 'develop' into feat/release-candidate
m8vago Jan 14, 2025
975758a
fix: crux build
m8vago Jan 15, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions package-lock.json

This file was deleted.

1 change: 1 addition & 0 deletions web/crux-ui/locales/en/container.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"json": "JSON",

"sensitiveKey": "Sensitive config data should be stored as secrets",
"templateTips": "Available templates in environments and storage: {{versionName}}",

"base": {
"all": "All",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,9 @@ const EditDeploymentCard = (props: EditDeploymentCardProps) => {
t,
onSubmit: async (values, { setFieldError }) => {
const body: UpdateDeployment = {
...values,
note: values.note,
prefix: values.prefix,
protected: values.protected,
configBundles: values.configBundles.map(it => it.id),
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ export interface VersionStateOptions {
projectId: string
version: VersionDetails
initialSection: VersionSectionsState
fetchTagInfo?: boolean
setSaveState?: (saveState: WebSocketSaveState) => void
}

Expand Down Expand Up @@ -176,11 +177,18 @@ export const useVersionState = (options: VersionStateOptions): [VerionState, Ver
const editor = useEditorState(versionSock)

const registriesSock = useWebSocket(routes.registry.socket(), {
onOpen: () =>
onOpen: () => {
// TODO(@robot9706): Not sure why we fetch all the tags here,
// it is super slow with our 8+ images / version, where each image has 200+ tags, so disable it
if (options.fetchTagInfo === false) {
return
}

refreshImageTags(
registriesSock,
version.images.filter(it => it.registry.type !== 'unchecked'),
),
)
},
})

registriesSock.on(WS_TYPE_REGISTRY_IMAGE_TAGS, (message: RegistryImageTagsMessage) => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ const VersionlessProjectSections = (props: VersionlessProjectSectionsProps) => {
version,
projectId: project.id,
initialSection,
fetchTagInfo: false,
setSaveState,
})

Expand Down
2 changes: 1 addition & 1 deletion web/crux-ui/src/elements/dyo-table.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ type SortState = {

export const sortString = (a: string, b: string) => (a ?? '').localeCompare(b)

export const sortNumber = (a: number, b: number) => b - a
export const sortNumber = (a: number, b: number) => a - b

export const sortDate = (a: string, b: string): number => {
if (a && b) {
Expand Down
10 changes: 10 additions & 0 deletions web/crux-ui/src/models/container-merge.ts
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,16 @@ export const mergeConfigs = (strong: ContainerConfigData, weak: ContainerConfigD
expectedState: strong.expectedState ?? weak.expectedState,
})

// TODO(@robot9706): Validate
// export const squashConfigs = (configs: ContainerConfigData[]): ContainerConfigData =>
// configs.reduce((result, conf) => {
// const merged = mergeConfigs(conf, result)
// return {
// ...merged,
// environment: mergeUniqueKeyValues(conf.environment, result.environment),
// }
// }, {} as ContainerConfigData)

export const squashConfigs = (configs: ContainerConfigData[]): ContainerConfigData =>
configs.reduce((result, conf) => mergeConfigs(conf, result), {} as ContainerConfigData)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -422,14 +422,16 @@ const ContainerConfigPage = (props: ContainerConfigPageProps) => {
</PageHeading>

<DyoCard className="p-4">
<div className="flex mb-4 justify-between items-start">
<div className="flex mb-1 justify-between items-start">
<DyoHeading element="h4" className="text-xl text-bright">
{getName()}
</DyoHeading>

{getViewStateButtons()}
</div>

<DyoMessage className="text-xs mt-2 mb-4" message={t('templateTips')} messageType="info" />

{viewState === 'editor' && <ContainerConfigFilters onChange={setFilters} filters={filters} />}
</DyoCard>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,10 @@ const EnvironmentDetailsPage = (props: EnvironmentDetailsPageProps) => {

const submit = useSubmit()

const onEnvEdited = (newEnv: PackageEnvironmentDetails) => setEnv(newEnv)
const onEnvEdited = (newEnv: PackageEnvironmentDetails) => {
setEnv(newEnv)
setEditing(false)
}

const onDelete = async () => {
const res = await fetch(routes.package.api.environmentDetails(pack.id, env.id), {
Expand Down Expand Up @@ -88,7 +91,12 @@ const EnvironmentDetailsPage = (props: EnvironmentDetailsPageProps) => {
{!editing ? (
<PackageEnvironmentCard packageId={pack.id} environment={env} />
) : (
<EditPackageEnvironmentCard packageId={pack.id} environment={env} onEnvironmentEdited={onEnvEdited} />
<EditPackageEnvironmentCard
submit={submit}
packageId={pack.id}
environment={env}
onEnvironmentEdited={onEnvEdited}
/>
)}

{!editing && <PackageEnvironmentVersionList environment={env} />}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ const VersionDetailsPage = (props: VersionDetailsPageProps) => {
initialSection,
projectId: project.id,
version,
fetchTagInfo: false,
})

const onVersionEdited = async (newVersion: EditableVersion) => {
Expand Down
50 changes: 34 additions & 16 deletions web/crux-ui/src/validations/container.ts
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,7 @@ const expectedContainerStateRule = yup
.optional()
.label('container:dagent.expectedState')

// TODO(@robot9706): Fix labels & config bundles conflicting
type KeyValueLike = {
key: string
value: string
Expand Down Expand Up @@ -507,30 +508,45 @@ const testRules = (
return true
}

const testEnvironmentRules = (imageLabels: Record<string, string>) => (envs: UniqueKeyValue[]) => {
const rules = parseDyrectorioEnvRules(imageLabels)
if (!rules) {
return true
}
const testEnvironmentRules =
(imageLabels: Record<string, string>) =>
(envs: UniqueKeyValue[]): boolean | yup.ValidationError => {
const rules = parseDyrectorioEnvRules(imageLabels)
if (!rules) {
return true
}

const requiredRules = Object.entries(rules).filter(([, rule]) => rule.required)
const envRules = requiredRules.filter(([_, rule]) => !rule.secret)
const requiredRules = Object.entries(rules).filter(([, rule]) => rule.required)
const envRules = requiredRules.filter(([_, rule]) => !rule.secret)

return testRules(envRules, envs, 'environment')
}
const validationError = testRules(envRules, envs, 'environment')
if (validationError) {
return validationError
}

const testSecretRules = (imageLabels: Record<string, string>) => (secrets: UniqueSecretKeyValue[]) => {
const rules = parseDyrectorioEnvRules(imageLabels)
if (!rules) {
return true
}

const requiredRules = Object.entries(rules).filter(([, rule]) => rule.required)
const secretRules = requiredRules.filter(([_, rule]) => rule.secret)
const testSecretRules =
(imageLabels: Record<string, string>) =>
(secrets: UniqueSecretKeyValue[]): boolean | yup.ValidationError => {
const rules = parseDyrectorioEnvRules(imageLabels)
if (!rules) {
return true
}

const requiredRules = Object.entries(rules).filter(([, rule]) => rule.required)
const secretRules = requiredRules.filter(([_, rule]) => rule.secret)

return testRules(secretRules, secrets, 'secret')
}
const validationError = testRules(secretRules, secrets, 'secret')
if (validationError) {
return validationError
}

return true
}

// eslint-disable-next-line @typescript-eslint/no-unused-vars
const createContainerConfigBaseSchema = (imageLabels: Record<string, string>) =>
yup.object().shape({
name: matchContainerName(yup.string().nullable().optional().label('container:common.containerName')),
Expand All @@ -539,6 +555,7 @@ const createContainerConfigBaseSchema = (imageLabels: Record<string, string>) =>
.nullable()
.optional()
.label('container:common.environment')
// TODO(@robot9706): Fix labels & config bundles conflicting
.test(
'labelRules',
'Environment variables must match their image label rules.',
Expand Down Expand Up @@ -591,6 +608,7 @@ export const createContainerConfigSchema = (imageLabels: Record<string, string>)

export const createConcreteContainerConfigSchema = (imageLabels: Record<string, string>) =>
createContainerConfigBaseSchema(imageLabels).shape({
// TODO(@robot9706): Fix labels & config bundles conflicting
secrets: uniqueKeyValuesSchema
.default(null)
.nullable()
Expand Down
23 changes: 21 additions & 2 deletions web/crux/src/app/container/container-config.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import { EditorLeftMessage, EditorMessage } from '../editor/editor.message'
import EditorServiceProvider from '../editor/editor.service.provider'
import ContainerConfigDomainEventListener from './container-config.domain-event.listener'
import { ConfigUpdatedMessage, WS_TYPE_CONFIG_UPDATED } from './container-config.message'
import { ContainerConfigProperty } from './container.const'
import {
ContainerConfigDetailsDto,
ContainerConfigRelationsDto,
Expand Down Expand Up @@ -265,13 +266,13 @@ export default class ContainerConfigService {
},
})

const data: ContainerConfigData = this.mapper.configDtoToConfigData(
let data: ContainerConfigData = this.mapper.configDtoToConfigData(
config as any as ContainerConfigData,
req.config ?? {},
)

if (req.resetSection) {
data[req.resetSection] = null
data = this.resetSection(data, req.resetSection)
}

await this.prisma.containerConfig.update({
Expand Down Expand Up @@ -319,6 +320,24 @@ export default class ContainerConfigService {
return message
}

private resetSection(config: ContainerConfigData, resetKey: ContainerConfigProperty) {
config[resetKey] = null

switch (resetKey) {
case 'storage':
config.storageSet = false
config.storageId = null
config.storageConfig = null
break

default:
config[resetKey as ContainerConfigProperty] = null
break
}

return config
}

private async checkMutability(configId: string): Promise<boolean> {
const deploymentSelect: Prisma.DeploymentSelect = {
status: true,
Expand Down
26 changes: 26 additions & 0 deletions web/crux/src/app/deploy/deploy.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import {
instanceConfigOf,
mergePrefixNeighborSecrets,
} from 'src/domain/start-deployment'
import { applyStringTemplate, applyUniqueKeyValueTemplate } from 'src/domain/template'
import { DeploymentTokenPayload, tokenSignOptionsFor } from 'src/domain/token'
import { collectChildVersionIds, collectParentVersionIds, toPrismaJson } from 'src/domain/utils'
import { copyDeployment } from 'src/domain/version-increase'
Expand Down Expand Up @@ -70,6 +71,10 @@ import {
WS_TYPE_INSTANCE_DELETED,
} from './deploy.message'

type InstanceConfigTemplate = {
versionName?: string
}

@Injectable()
export default class DeployService {
private readonly logger = new Logger(DeployService.name)
Expand Down Expand Up @@ -512,6 +517,10 @@ export default class DeployService {
})
}

const templateParams: InstanceConfigTemplate = {
versionName: deployment.version.name,
}

const invalidSecrets: InvalidSecrets[] = []

// deployment config
Expand All @@ -531,6 +540,8 @@ export default class DeployService {
deployment.instances.map(instance => {
const instanceConfig = instanceConfigOf(deployment, deploymentConfig, instance)

this.applyInstanceConfigTemplate(instanceConfig, templateParams)

return [instance.id, instanceConfig]
}),
)
Expand Down Expand Up @@ -1256,6 +1267,21 @@ export default class DeployService {
return versions.flatMap(it => it.deployments)
}

private applyInstanceConfigTemplate(config: ConcreteContainerConfigData, template: InstanceConfigTemplate) {
if (config.environment) {
applyUniqueKeyValueTemplate(config.environment, template)
}

if (config.storageSet && config.storageConfig) {
config.storageConfig.bucket = config.storageConfig.bucket
? applyStringTemplate(config.storageConfig.bucket, template)
: null
config.storageConfig.path = config.storageConfig.path
? applyStringTemplate(config.storageConfig.path, template)
: null
}
}

private static listInclude = {
version: {
select: {
Expand Down
Loading
Loading