[email protected] #264
Conversation
Switch to Amazon Linux 2022: https://docs.aws.amazon.com/linux/al2022/ug/what-is-amazon-linux.html. Brotli is now a package that is installed.
Some of the args are already in /build/args/headless.gn
|
@Sparticuz - do you mind to commit your |
|
After cleaning everything up, I'm receiving the same WebGL error's that @alixaxel was getting. (I'm unsure why my initial compile seemed to work, but I haven't been able to reproduce that). It looks like in Chrome 96, swiftshader was dropped and replaced with swangle, so I'm doing some research into what needs to be done to switch to that. I even got vulkan working on my local box and got better performance, but I'm unsure if that would work on lambda or not. |
|
I believe I've found the files needed to be bundled for Chrome 96+.
Notice that the Also the cli flags that needed to be changed are:
With those files and those flags, chromium runs without error and an inspect shows a working webgl. I'm running a compile with the latest ansible instructions in order to bundle the needed files. I won't commit the brotli files. I'll let @alixaxel run a build for security reasons, but I will post them so they can be tested. The compile takes about an hour. @alixaxel I also won't commit the changes I made to the ec2 instance, but I'll tell you I upgraded to a After I test this out I'll push the latest changes as Ready for review |
Also need to change a few flags
|
EDIT: If you use these, you'll need to edit chrome-aws-lambda's index file to include the flags in the previous comment. Edit2: Please see my branch here for the latest binaries https://github.com/Sparticuz/chrome-aws-lambda/tree/13.5_bin/bin |
|
I've tested it with the SAM application in the _/amazon folder and it worked. I'll be testing it internally as well. EDIT: Please see my branch here for the latest layer: https://github.com/Sparticuz/chrome-aws-lambda/blob/13.5_bin/chrome_aws_lambda.zip |
Also drop node 10
|
@Sparticuz - just to let you know, I've merged in with your _bin branch on top of my changes for webpack and so far so good! I'm doing some larger sized testing today, but nothing to complain about with the updates! (I don't require webgl so I can't speak to that, but basic functionality looks good!) |
|
@Sparticuz great contribution! I am looking forward to see this PR getting approved. |
|
@Sparticuz thanks so much for getting this up and running it will be a huge help to the community! Are there any particular lambda config changes that are required to support the new layer/version? |
|
Hey @Sparticuz, I'm testing Your compilation on AWS Lambda and with DEBUG enabled i got this on chromium boot.
I have chromium flags you suggested above. EDIT: flag that is not included in chrome-aws-lambda flags - |
|
It looks like it can't find libglesv2.so. are you using the zip image, or did you build and upload? |
I was using brotli compressed archives from Your fork :) Those You have linked above :) I have it all packed to lambda layer and then attached to lambda. EDIT: I've just found out that my custom decompression code for lambda layer build extracts swiftshader libnraries to bin/swiftshader directory - not in bin itself. It is probably an issue - paths mismatch :) EDIT 2: Yup, fixed paths, removed |
|
I've updated my PR to be Puppeteer 13.6.0, and my _bin branch to have 13.6.0 as well as chromium 101. Layer here. I have not tested it |
|
I gave the 13.6.0 layer a shot, and first few invocations seem to work fine (albeit quite a bit slower than the version on main for some reason), but afterwards I get errors that look like this when trying to take a screenshot: FWIW, I'm using playwright instead of puppeteer, but curious if others had success with taking screenshots using this version (puppeteer or playwright)? |
|
Hey @Sparticuz, |
Bfcache was disabled in puppeteer 13.6, Try that and see if you are getting the same results puppeteer/puppeteer#8182 |
|
A few more updates: I gave disabling bfcache a try, but unfortunately it didn't change anything. Also tried an earlier version of the layer at #264 (comment), and it had the same issue. Afterwards, I tried migrating the script to puppeteer, which worked (albeit with a screenshot time of ~1s compared to ~200ms with playwright on main)! So this seems to be a playwright specific thing. Going to dig into playwright's default flags to see if there's anything I'm missing that might resolve this. EDIT: Another update: Figured it out! Apparently this was caused by my custom pre-inflating layer I was using with the version on main to improve cold starts. Seems like that approach no longer worked with this version for some reason. Everything started working again once I reverted back to using the standard |
|
I'm using a build based on this fork and it seems to work perfectly. Am running it in a container image based on Hard to know when this fork will get merged in, so in case anyone needs it, here are instructions for installing from the fork: git clone https://github.com/Sparticuz/chrome-aws-lambda.git
# ^^^ note: almost 2GB. Maybe do a depth=1 clone of just the [email protected] branch. (I don't remember that git command though)
git checkout [email protected]
# ^^^ keep an eye for newer, in case it moves since this was posted.
npm install
npm pack
# next, replace $your_branch below with actual path:
mv chrome-aws-lambda-13.6.0.tgz $your_branch && cd $your_branch
npm i --save ./chrome-aws-lambda-13.6.0.tgz |
I'll keep the branch the same for the purposes of this PR, but I'll update the PR title and comment when there is a new version of Puppeteer that requires a new build of chromium. Ideally we can get this merged, then get some automated testing on it. I still don't know enough about the typescript that @alixaxel has written to be confident enough to say one way or the other if what I've done is production ready because of the lack of automated testing. There is only the one SAM test, but it's only testing that chromium takes a screenshot. It doesn't test any other functionality (webgl, etc...) That said, a number of users ARE using it in production. |
|
@Sparticuz Maybe running the core puppeteer test suite against this build might be a good solution? From a skim of the code, it looks like puppeteer is using an integration test approach, hitting an actual browser for everything. It uses process.env.BINARY to set executablePath (in test/mocha-utils.ts) so plopping their entire repo into a Lambda and running |
|
does this fix this issue #276 ? |
Yes |
|
@Sparticuz Hi there, is there anyway to implement this on my codes/dep without manually copy pasting? Should i make a private npm package using this pr? |
|
|
@Sparticuz thanks for the PR. After updating my code to use these changes there is a drop in performance when taking screenshots with Puppeteer in AWS (from '< 1' to '~8' seconds). Also, total memory consumption almost double from ~450mb to ~800mb. Any clue of what could have trigger this problem or possible fixes? |
Are you using this PR or my fork? This branch does not have the updated swiftshader code, so chromium is likely searching for the right files and not finding them. I believe someone further up had similar problems. I'm still trying to figure out how to update this PR with the needed changes. My fork includes the needed changes to get webgl working. That would be my guess, but I haven't measured anything. |
Implement lambda to generate documents (e.g., DOW) that returns a base64 encoded string of the pdf binary. Template files are contained in a layer and are under the /templates folder. Juice is a third-party package to combine the html and css files. Still troubleshooting trying to get puppeteer to run inside of lambda using puppeteer-core and having a compressed chromium binary added as a dependency or some alternative. The chrome-aws-lambda package does not appear to be maintained anymore, but someone has forked the repo and has been updating recent versions of node (v16) and puppeteer (v14). - https://acloudguru.com/blog/engineering/serverless-browser-automation-with-aws-lambda-and-puppeteer - https://github.com/alixaxel/chrome-aws-lambda - https://github.com/Sparticuz/chrome-aws-lambda - alixaxel/chrome-aws-lambda#264 - alixaxel/chrome-aws-lambda#275 Tikcet: AT-7345
|
Sorry, I didn't specified, I'm using the fork. So it does have the webGL changes. |
|
Thanks for putting this together! I tried using the binary from your v14.3.0 branch with Playwright v1.22.2 but I keep getting the following error whenever I try to create a new page: Is anybody else hitting this? Am I doing something wrong? |
Implement lambda to generate documents (e.g., DOW) that returns a base64 encoded string of the pdf binary. Template files are contained in a layer and are under the /templates folder. Juice is a third-party package to combine the html and css files. Still troubleshooting trying to get puppeteer to run inside of lambda using puppeteer-core and having a compressed chromium binary added as a dependency or some alternative. The chrome-aws-lambda package does not appear to be maintained anymore, but someone has forked the repo and has been updating recent versions of node (v16) and puppeteer (v14). - https://acloudguru.com/blog/engineering/serverless-browser-automation-with-aws-lambda-and-puppeteer - https://github.com/alixaxel/chrome-aws-lambda - https://github.com/Sparticuz/chrome-aws-lambda - alixaxel/chrome-aws-lambda#264 - alixaxel/chrome-aws-lambda#275 Tikcet: AT-7345
|
@codykaup I had the same error when using Lambda docker image on Apple M1, you need to add the |
|
@Sparticuz I can confirm the issue @bjassael is having with performance degradation with the WebGL fix, I can see screenshots being 10x slower (I see numbers going from 150ms to 1.2s+). Maybe using angle with swiftshader is causing this degradation given without the webgl fix it was not using swiftshader libs right ? |
Implement lambda to generate documents (e.g., DOW) that returns a base64 encoded string of the pdf binary. Template files are contained in a layer and are under the /templates folder. Juice is a third-party package to combine the html and css files. Still troubleshooting trying to get puppeteer to run inside of lambda using puppeteer-core and having a compressed chromium binary added as a dependency or some alternative. The chrome-aws-lambda package does not appear to be maintained anymore, but someone has forked the repo and has been updating recent versions of node (v16) and puppeteer (v14). - https://acloudguru.com/blog/engineering/serverless-browser-automation-with-aws-lambda-and-puppeteer - https://github.com/alixaxel/chrome-aws-lambda - https://github.com/Sparticuz/chrome-aws-lambda - alixaxel/chrome-aws-lambda#264 - alixaxel/chrome-aws-lambda#275 Tikcet: AT-7345
|
Thanks for the heads up @Antoine-C, it's working locally now! I tried running this in Lambda but it throws a different error for some reason. I tried multiple versions (v14.1.0, v14.1.1, and v14.3.0) but all of them throw this same error. Any ideas on why the binaries from @Sparticuz's fork does this with Playwright? |
@Antoine-C Where do you add that flag? |
|
I'm wondering if it might not be better to run puppeteer in a container now that Lambda supports it? This PR has been open and struggling for months now :( or will the same issues be present running it in a container? PS. I am currently running this on a Lambda in production and don't want to go the container route. |
In your Dockerfile next to the |
I haven't used Playwright, but I also haven't changed anything that would affect it. For people seeing slowness, does turning webgl off help? This may just be related to the new driver. If someone is running x-ray on their function, I'd love to see any info about what exactly is slow. |
|
Hi @Sparticuz, I make tests for the page.screenshot issue and these are the numbers: Puppeteer 14.3 with WebGL (based your repo, on tag v14.3.0, not contains the PR of decrease chrome size): Old puppeteer version (5.5, without this PR): Puppeteer 14.3 without WebGL (--disable-webgl and remove --use-gl=angle, --use-angle=swiftshader): |
|
My guess on the slowness would just be from Chrome's version creep? Those numbers are interesting @yakirza17, however, I'm looking more for a nodejs firegraph that could tell us whether the slowdown is occurring in chrome_aws_lambda, puppeteer, or chromium. FWIW, I really haven't changed any chrome_aws_lambda code besides the patch making it compatible with node16 and ts4.4+. Edit: Puppeteer 5.5 is Chromium 88, Puppeteer 14.3 is Chromium 103. That being said, I would recommend running with webgl disabled if you don't need it. |
|
@Sparticuz I was able to run 0x in a Lambda docker container and the flamegraph doesn't show any unoptimized bits regarding the screenshot in playwright nor chrome_aws_lambda so IMO the slowdown is coming from chromium. When I was using swiftshader directly I came up across these possible flags for ANGLE : https://chromium.googlesource.com/angle/angle/+/main/doc/DebuggingTips.md, I don't have much knowledge here but do you think it's possible to use another renderer like Vulkan (given we're including some vk_* files) ? Or swiftshader is the only way to go ? |
|
Sorry all, I'm out of the country at the moment. I will update to the latest puppeteer next week. |
|
@Sparticuz No worries, have you had the chance to look at it ? On my end I've noticed quite some "Browser closed" errors with |
|
Hello all, I've made a new package that breaks the link between
|
|
It works for me @Sparticuz 🤘🏼 |
This PR updates the build process for chromium and gets puppeteer support up to 14.1.0.
Closes: #240 #248 #258 #254 #242 #274 #275 #276
Notes:
Due to some changes in WebGL/SWAngle, this PR does not included the needed fixes for lambdafs. If you need WebGL, please use my scoped package.