Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass groupids to core #6628

Merged
merged 11 commits into from
Aug 2, 2024
Merged

Pass groupids to core #6628

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
09c38a6
Pass group ids to core when creating runs
flvndvd Aug 1, 2024
2e23618
Add BaseAuthenticator type
flvndvd Aug 1, 2024
5b42e29
:see_no_evil:
flvndvd Aug 1, 2024
f705369
:scissors:
flvndvd Aug 1, 2024
f674fff
:sparkles:
flvndvd Aug 1, 2024
cad9040
Address comments from review
flvndvd Aug 1, 2024
cf1147f
Use key workspace and groups
flvndvd Aug 1, 2024
38d60df
:shirt:
flvndvd Aug 2, 2024
c0f10e7
Address comments from review
flvndvd Aug 2, 2024
3c8d833
:sparkles:
flvndvd Aug 2, 2024
4195df3
:book:
flvndvd Aug 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 14 additions & 11 deletions front/pages/api/v1/w/[wId]/apps/[aId]/runs/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ async function handler(
if (keyRes.isErr()) {
return apiError(req, res, keyRes.error);
}
const { auth, keyWorkspaceId } = await Authenticator.fromKey(
const { auth } = await Authenticator.fromKey(
keyRes.value,
req.query.wId as string
);
Expand Down Expand Up @@ -285,16 +285,19 @@ async function handler(
"App run creation"
);

const runRes = await coreAPI.createRunStream({
projectId: app.dustAPIProjectId,
runAsWorkspaceId: keyWorkspaceId,
runType: "deploy",
specificationHash: specificationHash,
config: { blocks: config },
inputs,
credentials,
secrets,
});
const runRes = await coreAPI.createRunStream(
auth.getNonNullableWorkspace(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we already have owner from above ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah good catch. Let me fix.

auth.groups(),
{
projectId: app.dustAPIProjectId,
runType: "deploy",
specificationHash: specificationHash,
config: { blocks: config },
inputs,
credentials,
secrets,
}
);

if (runRes.isErr()) {
return apiError(req, res, {
Expand Down
29 changes: 16 additions & 13 deletions front/pages/api/w/[wId]/apps/[aId]/runs/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,19 +111,22 @@ async function handler(
);
const inputDataset = inputConfigEntry ? inputConfigEntry.dataset : null;

const dustRun = await coreAPI.createRun({
projectId: app.dustAPIProjectId,
runAsWorkspaceId: owner.sId,
runType: "local",
specification: dumpSpecification(
JSON.parse(req.body.specification),
latestDatasets
),
datasetId: inputDataset,
config: { blocks: config },
credentials: credentialsFromProviders(providers),
secrets,
});
const dustRun = await coreAPI.createRun(
auth.getNonNullableWorkspace(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, we already have owner ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW here at the top we shouldn't use getNonNullableWorkspace because it throws while we want to 404 if we don't have a ws

auth.groups(),
{
projectId: app.dustAPIProjectId,
runType: "local",
specification: dumpSpecification(
JSON.parse(req.body.specification),
latestDatasets
),
datasetId: inputDataset,
config: { blocks: config },
credentials: credentialsFromProviders(providers),
secrets,
}
);

if (dustRun.isErr()) {
return apiError(req, res, {
Expand Down
69 changes: 36 additions & 33 deletions types/src/front/lib/core_api.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ import {
} from "../../front/run";
import { LoggerInterface } from "../../shared/logger";
import { Err, Ok, Result } from "../../shared/result";
import { GroupType } from "../groups";
import { LightWorkspaceType } from "../user";

export const MAX_CHUNK_SIZE = 512;

Expand Down Expand Up @@ -89,7 +91,6 @@ export type CoreAPITokenType = [number, string];

type CoreAPICreateRunParams = {
projectId: string;
runAsWorkspaceId: string;
runType: RunRunType;
specification?: string | null;
specificationHash?: string | null;
Expand Down Expand Up @@ -280,28 +281,29 @@ export class CoreAPI {
return this._resultFromResponse(response);
}

async createRun({
projectId,
runAsWorkspaceId,
runType,
specification,
specificationHash,
datasetId,
inputs,
config,
credentials,
secrets,
}: CoreAPICreateRunParams): Promise<CoreAPIResponse<{ run: CoreAPIRun }>> {
// TODO(GROUPS_INFRA): use the auth as argument of that method instead of `runAsWorkspaceId`
// and pass both X-Dust-Workspace-Id and X-Dust-Group-Ids.

async createRun(
workspace: LightWorkspaceType,
groups: GroupType[],
{
projectId,
runType,
specification,
specificationHash,
datasetId,
inputs,
config,
credentials,
secrets,
}: CoreAPICreateRunParams
): Promise<CoreAPIResponse<{ run: CoreAPIRun }>> {
const response = await this._fetchWithError(
`${this._url}/projects/${encodeURIComponent(projectId)}/runs`,
{
method: "POST",
headers: {
"Content-Type": "application/json",
"X-Dust-Workspace-Id": runAsWorkspaceId,
"X-Dust-Workspace-Id": workspace.sId,
"X-Dust-Group-Ids": groups.map((g) => g.sId).join(","),
},
body: JSON.stringify({
run_type: runType,
Expand All @@ -319,33 +321,34 @@ export class CoreAPI {
return this._resultFromResponse(response);
}

async createRunStream({
projectId,
runAsWorkspaceId,
runType,
specification,
specificationHash,
datasetId,
inputs,
config,
credentials,
secrets,
}: CoreAPICreateRunParams): Promise<
async createRunStream(
workspace: LightWorkspaceType,
groups: GroupType[],
{
projectId,
runType,
specification,
specificationHash,
datasetId,
inputs,
config,
credentials,
secrets,
}: CoreAPICreateRunParams
): Promise<
CoreAPIResponse<{
chunkStream: AsyncGenerator<Uint8Array, void, unknown>;
dustRunId: Promise<string>;
}>
> {
// TODO(GROUPS_INFRA): use the auth as argument of that method instead of `runAsWorkspaceId`
// and pass both X-Dust-Workspace-Id and X-Dust-Group-Ids.

const res = await this._fetchWithError(
`${this._url}/projects/${projectId}/runs/stream`,
{
method: "POST",
headers: {
"Content-Type": "application/json",
"X-Dust-Workspace-Id": runAsWorkspaceId,
"X-Dust-Workspace-Id": workspace.sId,
"X-Dust-Group-Ids": groups.map((g) => g.sId).join(","),
},
body: JSON.stringify({
run_type: runType,
Expand Down
Loading