http://www.lovemytool.com/blog/2007/08/span-ports-or-t.html

http://en.wikipedia.org/wiki/Network_tap

http://www.infosecwriters.com/hhworld/hh9/roc/node4.html

http://www.netoptics.com/lp/tapspan.asp

• Do not place too much stress on the client.

• Create baselines for your test setup.

• Allow for think time in your test script.

• Consider test duration.

• Minimize redundant requests during testing.

• Consider simultaneous versus concurrent users.

• Set an appropriate warm up time.

The next sections describe each of these considerations.

Do Not Place Too Much Stress on the Client
Do not overly stress the client computer used to generate application load. The processor and memory usage on your client computers should be well below the threshold limit (CPU: 75 percent). Otherwise, the client computer may end up as a bottleneck during your testing. To avoid this, consider distributing the load on multiple client computers. Also, monitor the network interface bandwidth utilization to ensure that the network is not getting congested.
Create Baselines for Your Test Setup
Create baselines for your test setup for all types of tests you need to perform. The setup should be representative of the real life environment. This has two advantages. First, the results from various categories of tests do not reflect the type of hardware you are using. This means that your application modifications are the only variable. Second, you can depend on the test results because the test setup closely mirrors the real life environment.
Allow for Think Time in Your Test Script
Think time reflects the amount of time that a typical user is likely to pause for thought while interacting with your application. During this time, the user views the information displayed on the page or enters details such as credit card numbers or addresses. You should average the think time across various operations.
If you do not include think time in your script, there is no time gap between two subsequent requests on a per-user basis. This directly translates to all users firing requests to the server concurrently. So, for a concurrent load of 200 users, there will be 200 requests fired at a given instance to the server. This is not a true representation of how your typical users use your application.

Note: Omitting think time can be helpful when you want to generate excessive load on the server.

Make sure that the think time in your test script is based on real life conditions. This varies according to the operations performed by the user, and it varies depending on the page the user is interacting with.
For your test scripts, you can program for either a fixed think time between consecutive requests or a random think time ranging between minimum and maximum values.
Consider Test Duration
You should base the time duration for your tests on the end goal. If the goal is to load test and monitor the application behavior for your workload pattern, the test duration might range from 20 minutes to as long as one week. If the site is expected to experience users of similar profile, and the average user is expected to perform the same set of operations during the intraday activity, a test of 20 minutes to one hour is sufficient for generating data for load testing. You may want to run load tests for an extended period — four to five days — to see how your application performs on the peak operating capacity for a longer duration of time.
However, to generate test data for your site if your site expects users of different profiles during various hours of operation, you may need to test for at least eight to 10 hours to simulate various user profiles in the workload pattern.
For stress testing purposes, the end goal is to run tests to identify potential resource leaks and the corresponding degradation in application performance. This may require a longer duration, ranging from a couple of hours to a week, depending on the nature of the bottleneck.
For tests used to measure the transaction cost of an operation using transaction cost analysis, you might need to run test for only approximately 20 minutes.
More Information
For more information about capacity planning and transaction cost analysis, see ”How To: Perform Capacity Planning for .NET Applications” in the “How To” section of this guide.
Minimize Redundant Requests During Testing
Make sure that your test load script simulates an appropriate load for your particular scenario and does not generate additional redundant requests. For example, consider a logon scenario. The complete operation typically consists of two requests:
* A GET request used to retrieve the initial page where the user supplies a logon name and password.
* A POST request when the user clicks the Logon button to verify the credentials.

The GET request is the same for all users and can easily be cached and served to all users. However, it is the POST request that is critical from a performance perspective. In this example, your test script should avoid sending a GET request before submitting a POST request. By avoiding the GET request within your test script, the client threads of the load generator are able to iterate faster through the loop for a given set of users, thereby generating more POST requests. This results in a more effective stress test for the actual POST operation. However, there may be conditions when even the response to a GET request is customized for a user; therefore, you should consider including both the GET and POST requests in the stress tests.
Consider Simultaneous vs. Concurrent Users
Simultaneous users have active connections to the same Web site, whereas concurrent users connect to the site at exactly the same moment. Concurrent access is likely to occur at infrequent intervals. Your site may have 100 to 150 concurrent users but 1,000 to 1,500 simultaneous users.
To stress test your application by using tools such as ACT, use a think time of zero in your test scripts. This allows you to stress your application without any time lag, with all users concurrently accessing the site. By using an appropriate think time, however, you can more accurately mirror a real life situation in which a user takes some time before submitting a new request to the server.
High numbers of simultaneous users tend to produce spikes in your resource usage and can often cause interaction that is beyond the concurrent load a server can handle. This results in occasional “server busy” errors. Tests with simultaneous users are very useful because they help you identify the actual load your system can handle without sending back too many server busy errors.
Set an Appropriate Warm-up Time
ACT supports warm-up times. The warm-up time is used in a test script to ensure your application reaches a steady state before the test tool starts to record results. The warm-up time causes ACT to ignore the data from the first few seconds of a test run. This is particularly important for ASP.NET applications because the first few requests trigger just-in-time (JIT) compilation and caching.
The warm up time is particularly relevant for short duration tests, so that the initial startup time does not skew the test results.
To determine an appropriate warm-up time for your application, use the ASP.NET Applications\Compilations Total counter to measure the effects of JIT compilation. This counter should increase every time a user action triggers the JIT complier.
In some cases you may want to know how long it takes to compile and cache. It should be a separate test; it should not be averaged into your steady state measurements.
Best Practices for Performance Testing
When you test, consider the following best practices.
Do
When performing performance testing, make sure you do the following:

• Clear the application and database logs after each performance test run. Excessively large log files may artificially skew the performance results.

• Identify the correct server software and hardware to mirror your production environment.

• Use a single graphical user interface (GUI) client to capture end-user response time while a load is generated on the system. You may need to generate load by using different client computers, but to make sense of client-side data, such as response time or requests per second, you should consolidate data at a single client and generate results based on the average values.

• Include a buffer time between the incremental increases of users during a load test.

• Use different data parameters for each simulated user to create a more realistic load simulation.

• Monitor all computers involved in the test, including the client that generates the load. This is important because you should not overly stress the client.

• Prioritize your scenarios according to critical functionality and high-volume transactions.

• Use a zero think time if you need to fire concurrent requests,. This can help you identify bottleneck issues.

• Stress test critical components of the system to assess their independent thresholds.

Do Not

• Do not allow the test system resources to cross resource threshold limits by a significant margin during load testing, because this distorts the data in your results.

• Do not run tests in live production environments that have other network traffic. Use an isolated test environment that is representative of the actual production environment.

• Do not try to break the system during a load test. The intent of the load test is not to break the system. The intent is to observe performance under expected usage conditions. You can stress test to determine the most likely modes of failure so they can be addressed or mitigated.

• Do not place too much stress on the client test computers.

Related content: http://feelsgood11.wordpress.com/2008/10/27/some-of-best-practices-in-performance-testing/

Source: http://channel9.msdn.com/Wiki/PerformanceWiki/PerformanceTestingGuidelines/

Some of the results:

Data/test performance

HP LOADRUNNER took the most votes in the data/test performance category, which groups products in terms of their ability to apply test data to an application or system and evaluate how
it handles the processing.

Functional test

One again, HP’s QUICKTEST PROFESSIONAL takes the top prize for functional testing.

Test/QA Management

HP is once again on top with TESTDIRECTOR FOR QUALITY CENTER, which testers voted this year and last their favorite tool for test and QA management.

Defect/Issue Management

Last year there were two winners in the defect and issue management category: HP TestDirector for Quality Center and the Mozilla Foundation’s Bugzilla were tied. This year, TESTDIRECTOR alone took the top spot and Bugzilla moved to third, edged out by Microsoft’s Visual Studio Team Edition for Software Testers.

To read more, you have to download a November issue of ST&P

Well, that’s a good PR for HP. Good tools for “good” prices

Here are the pictures:

More shots here

The number of Virtual Users must be close to the number of real users once the application is in production, with a realistic think time applied between pages. Avoid testing with too few Virtual Users and a reduced think time. It could be assumed that the result would be the same, as the number of requests played per second is identical. However, this is not the case, for the following reasons:

You can get it here: http://www.stpmag.com/issues/stp-2008-01.pdf , Page 18.

Since I love biking and everything that relates to it, it was lovely to see a lot of people riding a bike and commuting. There is a mature infrastructure for this. Helsinki has more than one thousand kms of bike paths. Most of people while riding have a helmet on.

I didn’t waste my free time just sitting at home after work. I visited a lot of places. Fortunately, Helsinki has a lot of this stuff. The Government is paying a lot attention on the tourism development. You can easily get free maps, tourist guides and necessary info about anything.
Moreover I learnt some phrases like “Hyvaa Paivaa”, “Kiitos” and more. I used them when I met our finnish colleagues. Actually, they know some Russian too
Everybody speaks English in more or less extent. It’s cool. I didn’t have any difficulties with communication. Even elder people know it

I lived in the downtown area, not far from the Kamppi bus terminal and railway station. It was easy to get around from there. Most of evenings I spent walking on the city.
I stumbled across like a problem with the food in stores – I didn’t understand food titles So I had to pick some just by how it looks

You can get more shots here: http://picasaweb.google.com/vadim.feelsgood/HelsinkiFinland

18. Load/Volume Testing. Concept

• In the testing literature, the term “load testing” is usually defined as the process of exercising the system under test by feeding it the largest tasks it can operate with.
• Load testing is used to determine whether the system is capable of handling various anticipated activities performed concurrently by different users.
• Load testing ensures the level of confidence with which the customer uses the system efficiently under normal conditions.
• Usually load tests generate 80% of traffic (amount of load) a system can potentially handle.
• There is an extreme importance of having large datasets.

19. Load/Volume Testing. Goals

• Expose bugs that do not surface in cursory testing, such as memory management bugs, memory leaks, buffer overflows, etc.
• Ensure that the application meets the performance baseline established during performance testing. This is done by running regression tests against the application at a specified maximum load.

20. Load/Volume Testing. While Executing

• During the execution of the load test, the goal is to check whether the system is performing well for the specified load or not.
• To achieve this, system performance should be captured at periodic intervals of the load test.
• Performance parameters like response time, throughput, memory usage, and so forth should be measured and recorded.
• This will give a clear picture of the health of the system.

21. Stress Testing. Concept

• Stress testing goes one step beyond the load testing and identifies the system’s capability to handle the peak load.
• In stress testing, think time is not important as the system is stressed with more concurrent users beyond the expected load.
• Stress testing tries to break the system under test by overwhelming its resources or by taking resources away from it (in which case it is sometimes called negative testing).
• The main purpose behind this madness is to make sure that the system fails and recovers gracefully this quality is known as recoverability.

22. Stress Testing. Examples

• Double the baseline number for concurrent users/HTTP connections.
• Randomly shut down and restart ports on the network switches/routers that connect the servers (via SNMP commands for example).
• Take the database offline, then restart it.
• Rebuild a RAID array while the system is running.
• Run processes that consume resources (CPU, memory, disk, network) on the Web and database servers.
• This list can be enhanced with your favorite ways of breaking systems.

23. Stress Testing. Goals

• However, stress testing does not break the system purely for the pleasure of breaking it, but instead it allows testers to observe how the system reacts to failure.
• Does it save its state or does it crash suddenly?
• Does it just hang and freeze or does it fail gracefully?
• On restart, is it able to recover from the last good state?
• Does it print out meaningful error messages to the user, or does it merely display incomprehensible hex codes?
• Is the security of the system compromised because of unexpected failures?
• And the list goes on.

24. Configuration Testing. Concept

• Integrated with performance testing to identify how the response time and throughput vary as the configuration of infrastructure varies and to determine the reliability and failure rates.
• Configuration tests are conducted to determine the impact of adding or modifying resources.
• Verifies whether a system works the same, or at least in a similar manner, across different platforms, Database Management Systems, Network Operating Systems, network cards, disk drives, memory and central processing unit settings, and execution or running of other applications concurrently.
• Compatibility testing is a term which is used synonymously with configuration testing since compatibility issues are the matter of interest here.

25. Scalability Testing. Concept

• Scalability Testing, part of the battery of non-functional tests, is the testing of a software application for measuring its capability to scale up or scale out – in terms of any of its non-functional capability – be it the user load supported, the number of transactions, the data volume etc.
• The purpose of scalability testing is to determine whether the application automatically scales to meet the growing user load.

26. Scalability Testing. Scale Up / Scale Vertically

• To scale vertically (or scale up) means to add resources to a single node in a system, typically involving the addition of CPUs or memory to a single computer.
• A server twice as fast is more than twice as expensive.
• Taking advantage of such resources can also be called “scaling up”, such as expanding the number of Apache daemon processes currently running.

27. Scalability Testing. Scale Out / Scale Horizontally

• To scale horizontally (or scale out) means to add more nodes to a system, such as adding a new computer to a distributed software application.
• An example might be scaling out from one web server system to three (organizing a cluster system).

28. Scalability Testing. Ideal Scalability

29. Scalability Testing. Tradeoffs between Scale Up and Scale Out models

• Larger numbers of computers means increased management complexity, as well as a more complex programming model and issues such as throughput and latency between nodes; also, some applications do not lend themselves to a distributed computing model.
• However, the price differential between the two models is increasingly in favor of “scale out” computing for those applications that fit its paradigm.
• Super computers are scaled out!

30. Testing which is driven by what we want to measure.

• Response time testing
• Throughput testing
• Availability testing
• Measurement of resource utilization
• Capacity testing
• Measurement of delays (latency)
• Measurement of losses in networks
• Error rate measurement

31. Testing which is based on source or type of load.

• Usage-based testing
• Standard benchmark testing
• Load variation testing
• Ramp-up testing
• Component-specific testing
• Calibration testing

32. Testing which focuses on the impact of changes.

• System change impact assessment
• Infrastructure impact assessment
• Baseline testing
• Volume testing
• Parallel testing
• Live patch and change testing
• Extreme configuration testing

33. Testing which seeks to stress the system or find its limits.

• Scalability testing
• Bottleneck identification and problem isolation testing
• Duration or endurance testing
• Hot spot testing
• Spike and bounce testing
• Breakpoint testing
• Rendezvous testing
• Feature interaction / interference testing
• Deadlock testing
• Synchronization testing
• User scenario, bad day or soap opera testing
• Disaster recovery testing
• Risk-bases testing
• Hazard and threat identification
• Environmental testing
• Compatibility and configuration testing

Sources:

• “Integrated Approach to Web Performance Testing: A Practitioner’s Guide“, B. M. Subraya, Infosys Technologies Limited, Mysore, India
• “System Performance Testing. A Case Study. Part 1: Developing the Test Strategy”, Ross Collard, Collard & Company, New York, NY
• http://en.wikipedia.org/wiki/Performance_Testing
• http://en.wikipedia.org/wiki/Performance_Engineering
• Approaches to Performance Testing. A best-practices approach to maximize your performance test effort
• Performance vs. load vs. stress testing

]]> http://feelsgood11.wordpress.com/2008/04/23/approaches-to-performance-testing-part-2/feed/ 1 feelsgood11 http://feelsgood11.wordpress.com/2008/04/16/approaches-to-performance-testing-part-1/ http://feelsgood11.wordpress.com/2008/04/16/approaches-to-performance-testing-part-1/#comments Wed, 16 Apr 2008 09:41:36 +0000 feelsgood11 http://feelsgood11.wordpress.com/?p=52 ]]> 1. Diversity of Approaches to Performance Testing

• There are many variations within the broad framework of performance testing.
• There is no universal or consistent set of terminology, and many organizations have their own terms such as “work load testing” and “sweet spot testing”

2. Amount of Load that is put onto the server

• It can come from two different areas:
  • the number of connections (or virtual users) that are hitting the server simultaneously
  • the amount of think-time each virtual user has between requests to the server
• The more users hitting the server, the more load will be generated.
• The shorter the think-time between requests from each user, the greater the load will be on the server.
• Keep in mind that as you put more load on the server, the throughput will climb, to a point.

3. Baseline/Performance Testing. Concept

• Baseline — a range of measurements that represent acceptable performance under typical operating conditions.
• Testers have a baseline for how the system behaves under normal conditions.
• Baseline can then be used in regression tests to gauge how well a new version of the software performs.
• Baseline provides a reference point that makes it easier to spot problems when they occur.

4. Benchmark Testing. Concept

• The key to benchmark testing is to have consistently reproducible results.
• Benchmark tests should be used to determine if any performance regressions are in the application.
• Benchmark tests are great for gathering repeatable results in a relatively short period of time.
• The best way to benchmark is to change one and only one parameter between tests.

5. Benchmark Testing. “Flat” and “Ramp-Up”. Run Modes

• In case of “Flat” run mode, all of the users are loaded at once, and then run them for a predetermined amount of time.
• In case of “Rump-Up” run mode, users are loaded step by step.

6. Benchmark Testing. Accuracy of Results

• An average should be taken of the response time and throughput for a given test.
• The ramp-up run does not allow for accurate and reproducible averages because the load on the system is constantly changing as the users are being added a few at a time.
• Therefore, the flat run is ideal for getting benchmark numbers.

7. Capacity Planning. Definition

• Capacity testing measures the overall capacity of the system and determines at what point response time and throughput become unacceptable.
• Capacity testing is conducted with normal load to determine the extra capacity where stress capacity is determined by overloading the system until it fails, which is also called a stress load to determine the maximum capacity of a system.

8. Capacity Planning. Goals

• For capacity planning-type tests, your goal is to show how far a given application can scale under a specific set of circumstances.
• Often the specific goal is to find out how many concurrent users the system can support below a certain server response time.
• As an example, the question you may ask is, “How many servers do I need to support 8,000 concurrent users with a response time of 5 seconds or less?”

9. Capacity Planning. Factors to consider

• How many of total users will be hitting the server concurrently?
• What the think-time or time between requests for each user will be?
• It is also important to note that in the real world users won’t be clicking at exactly that interval every time they send a request.

10. Capacity Planning. Running Tests

• How do I load the users to simulate the load?
• The best way to do this is to try to emulate how users hit the server during peak hours.
• Does that user load happen gradually over a period of time?
• If so, a ramp-up-style load should be used, where x number of users are added ever y seconds.
• Do all of the users hit the system in a very short period of time all at once?
• If that is the case, a flat run should be used, where all of the users are simultaneously loaded onto the server.
• Different styles will produce different results that are not comparable.

11. Capacity Planning. Inaccuracy Explanation

• The system is able to continually adjust over time during the ramp-up run.
• If a ramp-up run is done and you find out that the system can support 5,000 users with a response time of 4 seconds or less.
• If we follow that test with a flat run with 5,000 users, we’ll probably find that the average response time of the system with 5,000 users is higher than 4 seconds.
• This is an inherent inaccuracy in ramp-up runs that prevents them from pinpointing the exact number of concurrent users that a system can support.

12. Capacity Planning. The best way to determine capacity

• Taking the best of both load types and running a series of tests will yield the best results.
• For example, using a ramp-up run to determine the range of users that the system can support should be used first.
• Then, once that range has been determined, doing a series of flat runs at various concurrent user loads within that range can be used to more accurately determine the capacity of the system.

13. Soak/Longevity/Duration/Endurance Tests. Concept

• This type of testing is conducted for different durations to find out the health of the system in terms of its consistent performance.
• These tests will show any performance degradations over time via memory leaks, increased garbage collection (GC), the accrual of uncommitted database transactions in a rollback buffer which impact.
• System resources, or other problems in the system.
• The longer the test, the more confidence in the system you will have.
• Endurance testing is conducted either on a normal load or on a stress load.

14. Soak/Longevity/Duration/Endurance Tests. Running

• These tests should be run for several days to really get a good idea of the long-term health of the application.
• Make sure that the application being tested is as close to real world as possible with a realistic user scenario (how the virtual users navigate through the application), testing all of the features of the application.
• Ensure that all the necessary monitoring tools are running so that problems will be accurately detected and tracked down later.
• Performance parameters like response time, throughput, memory usage, and so forth should be measured and recorded.

15. Peak-Rest/Spike Tests. Concept

• A spike is an unexpected load which stresses the system.
• Peak-rest tests are a hybrid of the capacity-planning ramp-up-style tests and soak tests.
• The goal here is to determine how well the system recovers from a high load (such as one during peak hours of the system), goes back to near idle, and then goes back up to peak load and back down again.
• This testing ensures whether the system will be stable and responsive under unexpected variations in load.
• If an unexpected surge appears in the user base, the performance of the system should degrade gracefully rather than come crashing down all of a sudden.
• Spike testing starts with a less number of users, say one user and then 50 concurrent users and then suddenly the user base is increased.
• Unexpected surges can make the system become unstable since the system might not be prepared to service a sudden surge of concurrent users.
• During these tests the possibility of system crashing is very high.

16. Peak-Rest/Spike Tests. During Run

• A couple of things can be determined from Peak-Test/Spike Tests:
• Does the system recover on the second “peak” and each subsequent peak to the same level (or greater) than the first peak?
• Does the system show any signs of memory or GC degradation over the course of the test?
• The longer this test is run (repeating the peak/idle cycle over and over), the better idea you’ll have of what the long-term health of the system looks like.
• If an unexpected surge appears in the user base, the performance of the system should degrade gracefully rather than come crashing down all of a sudden.

17. Peak-Rest/Spike Tests. Example

• In the online shopping application, there could be variable level of activities throughout the day (24 hours).
• We can anticipate that the activities will be at a peak during midday as compared to the rest of the day.
• The next figure depicts the different spikes when the system is tested across 24 hours with a surge in activities during midday.

Approaches to Performance Testing. Part 2

Sources:

• “Integrated Approach to Web Performance Testing: A Practitioner’s Guide“, B. M. Subraya, Infosys Technologies Limited, Mysore, India
• “System Performance Testing. A Case Study. Part 1: Developing the Test Strategy”, Ross Collard, Collard & Company, New York, NY
• http://en.wikipedia.org/wiki/Performance_Testing
• http://en.wikipedia.org/wiki/Performance_Engineering
• Approaches to Performance Testing. A best-practices approach to maximize your performance test effort
• Performance vs. load vs. stress testing