Refactor Load and Performance Doc #2140
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,200 +1,75 @@ | ||
| --- | ||
| title: Load and Performance Testing | ||
| description: Learn how to monitor internal execution performance of your Pantheon Drupal or WordPress site. | ||
| tags: [performance] | ||
| categories: [performance] | ||
| --- | ||
| Load and performance tests are critical steps in going live procedures, as they help expose and identify potential performance killers. These tests provide insight for how a site will perform in the wild under peak traffic spikes. | ||
|
|
||
| ## Load vs Performance Testing | ||
| Before you start, it's important to understand the difference between load and performance testing and know when to use each. | ||
| ### Performance Testing | ||
| Performance testing is the process in which you measure an application's response time to proactively expose bottlenecks. These tests should be regularly executed as part of routine maintenance. Additionally, you should run these test before any load testing. If your application is not performing well, then you can be assured that the load test will not go well. | ||
|
|
||
| The scope of performance tests should be limited to the application itself on a development environment (Dev or [Multidev](/docs/multidev)) without caching. This will give you an honest look into your application and show exactly how uncached requests will perform. You can bypass cache by [setting the `no-cache` HTTP headers](/docs/cache-control) in responses. | ||
|
There was a problem hiding this comment. Offer alternatives to bypass cache by setting a no-cache header? How about just disabling cache completely on Dev/Multidev during testing through Drupal/WordPress Admin UI? There was a problem hiding this comment. My understanding is that the dev environment has a default time-to-live of zero for dev, which implies no caching, but that things like Pantheon Advanced Page Cache may override this to be non-zero value. While a no-cache header may help, this may depend on when this get executed. Suggesting to disable caching via the UI is an option, with an emphasis to remember to re-enable prior to pushing to prod. |
||
|
|
||
| ### Load Testing | ||
| Load testing is the process in which you apply requests to your site that will represent the most load that your site will face once it is live. This test will ensure that the site can withstand the peak traffic spikes after launch. This test should be done on the Live environment before the site has launched, after performance testing. | ||
|
|
||
| If your site is already live, then you should run load tests on the Test environment. Keep in mind that the Test environment has one application container, while Live environments on sites with a service level of Business and above can have multiple application containers serving the site. So try to run a proportionate amount of traffic based on how many containers you currently have on your Live environment. | ||
|
There was a problem hiding this comment. Offer concrete example with math? There was a problem hiding this comment. The EOM team is the best source for the algorithm we use. |
||
|
|
||
| ## Preparing for Tests | ||
| The procedure for executing a load test and a performance test are similar: | ||
|
|
||
| 1. Enable [New Relic Pro](/docs/new-relic) within the Site Dashboard on Pantheon to ensure you have clear reporting to monitor response times. | ||
|
|
||
| * Set your [apdex](https://docs.newrelic.com/docs/apm/new-relic-apm/apdex/apdex-measuring-user-satisfaction#score) threshold according to your business rules (.5 is the default). Be careful not to set this too high, otherwise you will not get as many transaction traces in New Relic. | ||
| * If you have particular transactions that you want to ensure are traced, set them up as [key transactions](https://docs.newrelic.com/docs/apm/transactions/key-transactions/key-transactions-tracking-important-transactions-or-events). | ||
|
|
||
| 2. Select a load testing tool: | ||
|
There was a problem hiding this comment. Seems like verifying Varnish is working is still important before doing a load test? Maybe this can be more concise? There was a problem hiding this comment. Is this still the case now that Global CDN is in place? |
||
|
|
||
| * SaaS Solutions | ||
| * [Blazemeter](https://www.blazemeter.com) | ||
| * [Load Impact](https://loadimpact.com) | ||
| * Open Source tools | ||
| * [Jmeter](http://jmeter.apache.org) | ||
| * [Locust](http://locust.io/) | ||
|
|
||
| The Pantheon onboarding team uses Locust, an open source load testing tool. Locust makes it easy to build out test scripts, and it allows you to crawl the site instead of using predefined URLs. Crawling the site has the added benefit of loading every page that is linked to anywhere on the site. This exposes edge case performance bottlenecks that would have gone undetected under tests with predifined URLs. | ||
|
There was a problem hiding this comment. "makes it easy" -- link to example script? There was a problem hiding this comment. The EOM team should be asked to update this section. |
||
|
|
||
| Ultimately, it doesn't matter what tool you use as long as you to test your site properly. Be sure to allow for any authenticated traffic as well as anonymous. | ||
|
There was a problem hiding this comment. "Be sure to allow for any authenticated traffic as well as anonymous" - Not sure we should just assert this in passing. Load testing authenticated users can be difficult. There was a problem hiding this comment. I agree that authenticated user testing is a complex task and thus the generic statement should be along the lines of "It is important for Load Testing to test against the anticipated traffic patterns of the site, both in terms of traffic volume and authenticated/anonymous proportion. Note that testing authenticated workflows is considerably more complex requiring more time, skills and iterations." |
||
|
|
||
| 3. Determine how much load to apply. | ||
|
|
||
| * **Performance Tests**: Smaller loads should suffice, as you should be able to see transactional bottlenecks with 10-20 concurrent users. | ||
|
There was a problem hiding this comment. Why 10-20? A single request can give you all you need, no? There was a problem hiding this comment. IMO, you want to generate more than single request to tease out potential bottlenecks. Also, I know that we have a Quicksilver example that will use free loader.io account to automatically run this level of test on each push to Test environment. Not only does this result in automated testing procedures, it provides a standard profile that you can see in New Relic. Here's a related link, but we need better: pantheon-systems/quicksilver-examples#110 There was a problem hiding this comment. IMO, this is good to go (i.e. no edits needed). A separate issue should be created, if/when we want to include reference to the loader.io Quicksilver example. |
||
| * **Load Tests**: Determine how many concurrent users the site is expected to serve based on historical analytics for the site. Identify the peak hourly sessions and average session duration, then do some math: `hourly_sessions / (60 / average_duration) = Concurrent Users` | ||
|
There was a problem hiding this comment. @bentekwork How do I determine load to apply in the test after calculating concurrent users? There was a problem hiding this comment. Let's reiterate difference between load test on Live vs non-live, and include app containers in calculation for scenario. There was a problem hiding this comment. Load tests should not be run on Test, rather performance test can/should be run there. In terms of providing formulas, it is complicated by the fact that to run "proportionate amount of traffic" on Test involves knowing the number of appservers on Live, which clients can't determine on their own (other than asking Support, or looking at New Relic, which will include decommissioned appservers for some time). |
||
|
|
||
|
|
||
| ##Running the Tests | ||
| If this is a performance test, be sure to run the test on a development environment (Dev or [Multidev](/docs/multidev)) without caching. Run load tests on the Live environment before launching the site. If the site is already launched, use the Test environment instead. | ||
| <div markdown="1" class="alert alert-danger" role="alert"> | ||
| ###Warning {.info} | ||
| We do not recommend load testing on the Live environment if the site has already launched because you risk overwhelming your live site and causing downtime. | ||
| </div> | ||
| Note the start time for the test. As the test executes, it's a good idea to keep a close eye on [log files](/docs/logs). Make note of any errors and warnings that pop up during the test so that you can fix them. | ||
|
|
||
| Once the test is running, execute common tasks done by editors and administrators and note the time. Example tasks may include: | ||
|
|
||
| * Clear the drupal cache | ||
| * Clear the edge cache (if this is a load test, performance tests should not be cached) | ||
| * Run Drupal cron | ||
| * Run any scripts that could be triggered while users are on the site. | ||
|
|
||
| ##Assess Results | ||
| Now that the test is complete, examine the New Relic data. The **Overview** tab will give you an average response time for the duration of the test. Times above 750ms are good indicators of performance optimization opportunites. | ||
|
|
||
| Next, review the **Transactions** tab in New Relic and sort by **Slowest average response time**. Click on the slowest transaction to pull up the transaction trace. Review the transaction trace to find the performance bottleneck. | ||
|
|
||
| Finally, review the **Error analytics** tab in New Relic. PHP errors often indicate huge performance bottlenecks. If you have errors, fix them. | ||
|
|
||
| ### Calculating Load Capacity After Launch | ||
|
There was a problem hiding this comment. How can we highlight this scenario? And flesh it out with concrete example explaining how to collect RPM and response time from New Relic? |
||
| After launch, you can establish a baseline that `X` response time will let you handle `Y` traffic. If `X` degrades in Dev/Test, that will impact how much traffic Live can handle. | ||
|
|
||
| ## See Also | ||
|
|
||
| * [Load Testing Drupal and WordPress with BlazeMeter](/docs/guides/load-testing-with-blazemeter/) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why should these tests be run regularly as part of routine maintenance? To ensure performance doesn't degrade with a code or configuration change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, I'd favor suggesting that clients:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some of these notions.