Archive for April 2008

Approaches to Performance Testing. Part 2

April 23, 2008

Approaches to Performance Testing. Part 1

18. Load/Volume Testing. Concept

  • In the testing literature, the term “load testing” is usually defined as the process of exercising the system under test by feeding it the largest tasks it can operate with.
  • Load testing is used to determine whether the system is capable of handling various anticipated activities performed concurrently by different users.
  • Load testing ensures the level of confidence with which the customer uses the system efficiently under normal conditions.
  • Usually load tests generate 80% of traffic (amount of load) a system can potentially handle.
  • There is an extreme importance of having large datasets.

19. Load/Volume Testing. Goals

  • Expose bugs that do not surface in cursory testing, such as memory management bugs, memory leaks, buffer overflows, etc.
  • Ensure that the application meets the performance baseline established during performance testing. This is done by running regression tests against the application at a specified maximum load.

20. Load/Volume Testing. While Executing

  • During the execution of the load test, the goal is to check whether the system is performing well for the specified load or not.
  • To achieve this, system performance should be captured at periodic intervals of the load test.
  • Performance parameters like response time, throughput, memory usage, and so forth should be measured and recorded.
  • This will give a clear picture of the health of the system.

21. Stress Testing. Concept

  • Stress testing goes one step beyond the load testing and identifies the system’s capability to handle the peak load.
  • In stress testing, think time is not important as the system is stressed with more concurrent users beyond the expected load.
  • Stress testing tries to break the system under test by overwhelming its resources or by taking resources away from it (in which case it is sometimes called negative testing).
  • The main purpose behind this madness is to make sure that the system fails and recovers gracefully this quality is known as recoverability.

22. Stress Testing. Examples

  • Double the baseline number for concurrent users/HTTP connections.
  • Randomly shut down and restart ports on the network switches/routers that connect the servers (via SNMP commands for example).
  • Take the database offline, then restart it.
  • Rebuild a RAID array while the system is running.
  • Run processes that consume resources (CPU, memory, disk, network) on the Web and database servers.
  • This list can be enhanced with your favorite ways of breaking systems.



Approaches to Performance Testing. Part 1

April 16, 2008

1. Diversity of Approaches to Performance Testing

  • There are many variations within the broad framework of performance testing.
  • There is no universal or consistent set of terminology, and many organizations have their own terms such as “work load testing” and “sweet spot testing”

2. Amount of Load that is put onto the server

  • It can come from two different areas:
    • the number of connections (or virtual users) that are hitting the server simultaneously
    • the amount of think-time each virtual user has between requests to the server
  • The more users hitting the server, the more load will be generated.
  • The shorter the think-time between requests from each user, the greater the load will be on the server.
  • Keep in mind that as you put more load on the server, the throughput will climb, to a point.

3. Baseline/Performance Testing. Concept

  • Baseline — a range of measurements that represent acceptable performance under typical operating conditions.
  • Testers have a baseline for how the system behaves under normal conditions.
  • Baseline can then be used in regression tests to gauge how well a new version of the software performs.
  • Baseline provides a reference point that makes it easier to spot problems when they occur.

4. Benchmark Testing. Concept

  • The key to benchmark testing is to have consistently reproducible results.
  • Benchmark tests should be used to determine if any performance regressions are in the application.
  • Benchmark tests are great for gathering repeatable results in a relatively short period of time.
  • The best way to benchmark is to change one and only one parameter between tests.

5. Benchmark Testing. “Flat” and “Ramp-Up”. Run Modes

  • In case of “Flat” run mode, all of the users are loaded at once, and then run them for a predetermined amount of time.
  • In case of “Rump-Up” run mode, users are loaded step by step.