-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature: custom runtimes to define function #1602
Feature: custom runtimes to define function #1602
Conversation
🦋 Changeset detectedLatest commit: aa6762e The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @MarlonJD thanks for taking the time to open this. We will likely not merge this in its current form. There are a few reasons for this:
- We cannot depend on CDK alpha packages in our stable release line. The approach outlined in the RFC you commented on moves the alpha CDK dependency to the customer's project so they are explicitly aware they are using alpha functionality.
- Changing the
runtime
parameter from a number union to a string union is a breaking API change which we cannot do without a major version bump of our libraries - E2E tests that exercise building and calling python and go functions need to be added
- The dependency on docker to build the functions needs more consideration of the implications (eg for backend deployments on Amplify hosting and getting started locally)
If you are interested, you can work with @josefaidt to define an interface that is appropriate here and then we can have further discussion on the CDK and docker dependencies.
Hi @edwardfoyle. |
Hey @MarlonJD 👋 thanks for taking the time to file this! To add to @edwardfoyle 's note we'd like to avoid taking a dependency on Docker and the alpha distributions of L2 constructs for Go and Python. While those two runtimes are top of mind, the escape hatch is desirable to offer the capability of using |
Hello @josefaidt. I'm trying to change my PR, I'm trying to create defineFunctionWithAnyLanguage in @aws-amplify/backend. Now I'm struggling about creating callback function for Construct/scope and use this function, any help in here? I can create directly with Stack parameter like this:
But it's will depend to |
Hey @MarlonJD you can define a union on the the props here will need adjustments and the scope would be passed to the We wouldn't want to necessarily change the scope but change how "props" (in this case, the callback) is called with the existing |
9a96a36
to
bb5e0df
Compare
@josefaidt I changed PR to this:
or
Golang function will be usable out of box, python function will be require docker, so python function cannot be used on CI/CD, it can be deploy with custom pipeline on local. ie: disabling auto deploy, What are you thinking about this changes? If it's okay, I'll try to write e2e tests, can you guide me again? |
I hope you can review this changes @edwardfoyle |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is heading in the right direction! Another thing we need to do is turn off generating the typed process.env file when using custom runtimes. We can probably do that by creating a new noop FunctionEnvironmentTypeGenerator
if a callback is specified to defineFunction
We should also have an integration and/or e2e test that asserts permissions and environment variables are correctly attached to functions with custom runtimes.
@edwardfoyle I did changes, if it's okay about code, can you help me how and where should write for integration and/or e2e tests? |
@edwardfoyle Hey there, I hope you can review new changes |
Hi @MarlonJD, sorry for the delay here. While this PR is on the right track in terms of implementation, we are concerned about the implications when deploying using the Amplify hosting service. Specifically, CDK requires Docker to build some runtimes and currently Docker is not installed in the hosting build container. Docker also cannot be installed using a custom build script because it requires permissions that the execution environment doesn't have. This means that customers using custom runtimes would not be able to use Hosting deployments. We want to consider this feature holistically rather than only offer partial support across our product offering. As a result, we need to do some internal prerequisite work to ensure custom runtime deployments can work in Amplify hosting deployments. |
@josefaidt @edwardfoyle Hey there! I have good news about python functions. There is already a
I just tried any it's works. I used pure |
@edwardfoyle Any updates? |
PTAL |
Great work @MarlonJD! Hope this gets merged quickly as currently im trying to achieve same but in a really whacky way |
Hey @josefaidt, I commented how we can use python functions without docker build, but there is no local parameter in What are you thinking about this solution to using python functions like this, should we add local parameter to |
Any updates on this PR? @edwardfoyle, has @MarlonJD addressed your request? It would be super, super helpful to get this merged! |
I was able to build my python custom function with my requirements. The only problem I have now is that this seems to be never updated even if I alter this line. Seems that a missing force/cache somewhere. Only workaround I've found now: remove the |
@guikcd oh it's a good review, let me check how nodejs changes triggers, maybe we can found fix about it |
@josefaidt Hey there, I was looking into tracking changes on lambda functions, but I couldn't found this yet, I found FilesChangesTracker class but it's for sandbox file changes, can you help me how can I found how lambda functions triggering bundling again when function have changes? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MarlonJD
I looked at latest diff and it looks good overall.
Couple of areas I left comments:
- test project - I ran e2e tests here https://github.com/aws-amplify/amplify-backend/actions/runs/12552245657 - they failed, I left comments with changes to make them passing.
- The error messages. For these please keep in mind that they need that docs PR to stabilize, so this might not be final iteration.
packages/integration-tests/src/test-projects/advanced-auth-and-functions/amplify/function.ts
Outdated
Show resolved
Hide resolved
throw new AmplifyUserError( | ||
'FunctionProviderError', | ||
{ | ||
message: e instanceof Error ? e.message : JSON.stringify(e), | ||
// TODO better resolution | ||
resolution: 'Ensure that function provider is working.', | ||
}, | ||
e instanceof Error ? e : undefined | ||
); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
throw new AmplifyUserError( | |
'FunctionProviderError', | |
{ | |
message: e instanceof Error ? e.message : JSON.stringify(e), | |
// TODO better resolution | |
resolution: 'Ensure that function provider is working.', | |
}, | |
e instanceof Error ? e : undefined | |
); | |
throw new AmplifyUserError( | |
'CustomFunctionProviderError', | |
{ | |
message: e instanceof Error ? e.message : JSON.stringify(e), | |
resolution: "Ensure that callback passed to 'defineFunction' executes without error. See https://docs.amplify.aws/react/build-a-backend/functions/custom-functions for more details.", | |
}, | |
e instanceof Error ? e : undefined | |
); |
@josefaidt @Amplifiyer please take a look at proposed wording here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what are a few examples of errors that would fall into this category? The wording seems fine but I'm concerned with whether it would be tough to debug
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Anything that function passed to defineFunction
can throw.
Including:
- Bundling errors,
- Invalid properties/validation errors
- Mistakes in function itself.
The underlying error will be printed in the console.
- We copy it's message
- We include actual root cause as
cause
, it is reported in std streams (see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error/cause ).
The idea here is that the troubleshooting section can be expanded on the docs page without changing this code. I.e. information about root cause and the docs page should be enough to investigate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Btw. technically docker errors could be handled this way as well, but for now in PR they get their own category as we expect it to be common problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- When we are wrapping the error, we should provide a different error message than the one being wrapped, specially as it is being passed in the cause. This error message should describe what happened at the call site e.g. "Failed to instantiate the custom function provider", or something like that.
Ensure that callback passed to 'defineFunction' executes without error
, are we conflating the concept ofcallback
here?
throw new AmplifyUserError( | ||
'FunctionBundlingDockerError', | ||
{ | ||
message: e.message, | ||
// TODO better resolution | ||
resolution: | ||
'Ensure that docker is present and works correctly. See https://some.link.about.docker.', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
throw new AmplifyUserError( | |
'FunctionBundlingDockerError', | |
{ | |
message: e.message, | |
// TODO better resolution | |
resolution: | |
'Ensure that docker is present and works correctly. See https://some.link.about.docker.', | |
throw new AmplifyUserError( | |
'CustomFunctionProviderDockerRequiredError', | |
{ | |
message: e.message, | |
resolution: | |
'See https://docs.amplify.aws/react/build-a-backend/functions/custom-functions for more details about current limitations and troubleshooting steps.', |
@josefaidt @Amplifiyer please take a look at proposed wording here. The goal here is to:
- refrain from stating docker support in error message/resolution in case we find a way to support it (avoid hardcoding).
- point to web page that we can edit out of bound with most recent information about the state of support. (providing support should not involve code change in this repo, therefore need for decoupling).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have that callout at the top of this page that calls out limitations with docker, but perhaps this warrants more clarity at the top of that page -- prereqs, etc.
There's a note I left on the docs PR to (separately) create a troubleshooting dedicated to docker errors which we can use to add more color for support
Co-authored-by: Kamil Sobol <[email protected]>
Co-authored-by: Kamil Sobol <[email protected]>
Co-authored-by: Kamil Sobol <[email protected]>
Co-authored-by: Kamil Sobol <[email protected]>
Hi @sobolk, I’ve implemented the changes based on your suggestions. Please let me know if there’s anything else that needs adjustment. |
Thank you. The change in test project needs imports, see https://github.com/aws-amplify/amplify-backend/actions/runs/12556641214/job/35018558350?pr=1602 |
Hi @sobolk, I've added the missing imports to address the issue. |
Thanks. The e2e look good here https://github.com/aws-amplify/amplify-backend/actions/runs/12584741475 . Could you please address |
Hi @sobolk, I’ve resolved the linting errors and committed the changes. I ran |
Co-authored-by: Amplifiyer <[email protected]>
Co-authored-by: Amplifiyer <[email protected]>
@MarlonJD could you please merge latest |
And please |
Hi @sobolk, I'm getting this error after
|
It's likely related to workspace being in inconsistent state after You can try the following:
Or just run |
Hi @sobolk, Thank you for pointing this out. I followed your recommendations, which resolved the inconsistency, and I’ve committed the necessary changes. I also want to express my gratitude for your guidance and support—it’s much appreciated! |
Happy to hear it's finally possible to define a python function! Can you also please add documentation about how a Python function can interact with the Amplify Data? Thanks! |
Problem
#1543, #1486
Changes
I changedruntime
parameter to string from int. It was only nodejs version. Now it can be:'GO_1_X_PROVIDED_AL2023' | 'NODEJS_16_X' | 'NODEJS_18_X' | 'NODEJS_20_X' | 'PYTHON_3_8' | 'PYTHON_3_9' | 'PYTHON_3_10' | 'PYTHON_3_11' | 'PYTHON_3_12'
It's defaultNODEJS_18_X
same as old functions, people only need to specifyruntime: "NODEJS_16_X"
if they want another version, it's giving lint error, so it's easy to inform people if it's changed. If it's empty, it's still creating nodejs version, I also add required toentry
parameter when it's go or python runtimes.Edit: I modified
defineFunction
to props and callback function can be parameters at the same time:can be usable like proposal in RFC, go function will be usable out of box, python function will be require docker, so python function cannot be used on CI/CD, it can be deploy with custom pipeline on local.
ie: disabling auto deploy,
npm ci && export CI=1 && npx ampx pipeline-deploy --branch BRANCH_NAME --app-id AMPLIFY_APP_ID
will do the trick for python.Checklist
run-e2e
label set.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.