Posted tagged ‘load testing’

Some of the best practices in the performance testing

October 27, 2008

Defining the number of Virtual Users.

The number of Virtual Users must be close to the number of real users once the application is in production, with a realistic think time applied between pages. Avoid testing with too few Virtual Users and a reduced think time. It could be assumed that the result would be the same, as the number of requests played per second is identical. However, this is not the case, for the following reasons:

  • The memory burden on the server will be different: Each user session uses a certain amount of memory. If the number of user sessions is underestimated, the server will be running under more favorable conditions than in real-life and the results will be distorted.

  • The number of sockets open simultaneously on the server will be different. An underestimation of user numbers means the maximum threshold for open server sockets cannot be tested.

  • The resource pools (JDBC connection pools) will not be operating under realistic conditions. An inappropriate pool size setting might not be detected during the test.

Using different user accounts and values

Use variables to dynamically modify key values such as user account logins or certain form parameters (such as productID in an e-business application). The main idea of this is to bypass the use of the various server caches, for the following reasons:

  • Playing the same requests with the same values produces an unrealistically high performance, due to the use of various caches: preloading into memory cache, connection pools, system swap…

  • On the other hand, completely disabling the caches (when available) will produce an unrealistically poor performance.

“Warm up” the server before you start

  • After a re-start, don’t hesitate to “warm up” the server with a few calls before generating a sudden, high load which, in addition to being unrealistic, may cause the server to crash. Sending a short, light load beforehand allows certain resources, such as connection pools or thread pools, to be pre-allocated.

  • Run the test for a significant length of time in order to iron out any outliers.

  • Make sure the Load Generators are not overloaded; CPU and memory usage are displayed in real time throughout the test.

Stop any Virtual Users containing errors

When a Virtual User receives an error, it should normally stop running. If this does not happen, it could continue playing requests that have no meaning. For example, if the user login fails, there is little point sending further browsing or search requests to the application as it will only distort the response time statistics for those pages.

Each Virtual User type may be configured to stop running in case of error or failed assertion.

Make Scenarios and Transaction Definitions Granular

Where possible, break scenarios into several smaller scenarios to focus the tests. Make sure transaction definitions are granular enough to be able to pinpoint performance issues to specific GUI actions.

Preserve Environment During Recording and Running Load Tests

Make sure that the environment when running a load test is in the same state as it was when the test was recorded. Changes to the operating environment might require tests to be rerecorded.

Validate test data before you start

If you have got an excel with user accounts, do verify them so that you make sure they are all OK. It can happen that some of the them can be disabled and will cause unnecessary errors on login page. It’s also worth having a script that would verify user accounts automatically. You can set it executed before each of runs.

Use different workloads to recognize system behavior

It’s beneficial to do several workloads that can reveal application thresholds, bottlenecks. This can be done using simple increasing models with varying paces and maximum amount of users.

Make sure that the system can handle less amount of users than a maximum one.

Sometime it can happen that customer wants to load the system with, for example, 1000 of virtual users.  In this case it’s reasonable to do one test for 500 vUsers beforehand that shows the system can handle half of maximum amount.

Of course this is far from being a complete, full-fledged list of best practices from the world of performance testing.

Related content: