PerfTestingGuide Wiki & Documentation Rss Feed

Updated Wiki: Home

mycodeplexuser — Thu, 28 Aug 2008 00:15:20 GMT

patterns & practices Performance Testing Guidance for Web Applications

Welcome to the patterns & practices Performance Testing Guidance for Web Applications! This guide shows you an end-to-end approach for implementing performance testing. Whether you are new to performance testing, or looking for ways to improve your current performance testing approach, you will find insights that you can tailor for your specific scenarios. This guide is related to our Performance Testing Guidance Project . - J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Download the Guide

The Final Release is Available! Start using the guide today, while we continue to make improvements.

Download the Performance Testing Guidance for Web Applications Guide

Parts

Part 1, Introduction to Performance Testing
Part II, Exemplar Performance Testing Approaches
Part III, Identify the Test Environment
Part IV, Identify Performance Acceptance Criteria
Part V, Plan and Design Tests
Part VI, Execute Tests
Part VII, Analyze Results and Report
Part VIII, Performance Testing Techniques

Forewards

Chapters

Introduction

Part 1, Introduction to Performance Testing

Part II, Exemplar Performance Testing Approaches

Part III, Identify the Test Environment

Chapter 8 – Evaluating Systems to Increase Performance-Testing Effectiveness

Part IV, Identify Performance Acceptance Criteria

Part V, Plan and Design Tests

Part VI, Execute Tests

Chapter 14 – Test Execution

Part VII, Analyze Results and Report

Part VIII, Performance-Testing Techniques

Reference

Glossary

Team

Core Team: J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea
Contributors and Reviewers

UPDATED WIKI: Home

sbarber — Sat, 15 Dec 2007 04:50:42 GMT

patterns & practices Performance Testing Guidance for Web Applications

Download the Guide

The Final Release is Available! Start using the guide today, while we continue to make improvements.

Download the Performance Testing Guidance for Web Applications Guide

Parts

Part 1, Introduction to Performance Testing
Part II, Exemplar Performance Testing Approaches
Part III, Identify the Test Environment
Part IV, Identify Performance Acceptance Criteria
Part V, Plan and Design Tests
Part VI, Execute Tests
Part VII, Analyze Results and Report
Part VIII, Performance Testing Techniques

Forewards

Chapters

Introduction

Part 1, Introduction to Performance Testing

Part II, Exemplar Performance Testing Approaches

Part III, Identify the Test Environment

Chapter 8 – Evaluating Systems to Increase Performance-Testing Effectiveness

Part IV, Identify Performance Acceptance Criteria

Part V, Plan and Design Tests

Part VI, Execute Tests

Chapter 14 – Test Execution

Part VII, Analyze Results and Report

Part VIII, Performance-Testing Techniques

UPDATED WIKI: Foreword By Rico Mariani

prashantbansode — Wed, 29 Aug 2007 20:09:34 GMT

Foreword By Rico Mariani

It’s hard to imagine anything than is considered a more arcane art than performance tuning – unless perhaps it is performance testing.
If you were to go door to door between groups just within Microsoft you would find many different approaches with various different degrees of quality or success. Pretty much everyone will vow that their approach is certainly the one that is best for them – except maybe an honest few, who might say something more modest. Some have good reason to be confident because they really have studied the space very well. In my own experience at least, the situation is not altogether different outside of Microsoft than it is inside where I do my work. It’s a mixed bag, on a good day.
If I had to describe the most common problem I see in this space with one word it would imbalance. There are many aspects to testing and teams tend to unduly focus on one or another and then sometimes get blindsided by the ones they missed. Perhaps they’re only thinking about throughput – what about consumption? Perhaps only latency – what about smooth delivery? Perhaps only cost -- what about scalability?
You get great performance by balancing the key factors, considering them in your designs and then tracking them carefully. So perhaps the greatest service that a book like Performance Testing Guidance for Web Applications can provide to you is a broader understanding of what all the factors might be so that you have an excellent menu of considerations to choose from in your testing plan. Luckily, that is just what you’re going to get.
The Guidance that follows provides a great survey of the most important considerations: From how to understand and quantify your desired end user experience, how to choose key resources for study, to advice on summarizing results in a statistically meaningful way, and how to fit these practices into different software lifecycles. And even though the focus is squarely on web applications, the teachings are actually much more general and can easily be applied for many different kinds of applications.
Great engineering comes from creating predictable results at predictable costs. In fact, I like to say that if you’re not measuring you’re not engineering. This volume will provide you with the performance testing fundamentals to give you the ongoing metrics you need to do great engineering.

Rico Mariani
Chief Architect of Visual Studio
Microsoft Corporation
July, 2007

Rico Mariani began his career at Microsoft in 1988, working on language products beginning with Microsoft® C version 6.0, and contributed there until the release of the Microsoft Visual C++® version 5.0 development system. In 1995, Rico became development manager for what was to become the "Sidewalk" project, which started his 7 years of platform work on various MSN technologies. In the summer of 2002, Rico returned to the Developer Division to as a Performance Architect on the CLR team. His performance work led to his most recent assignment as Chief Architect of Visual Studio. Rico's interests include compilers and language theory, databases, 3-D art, and good fiction.

UPDATED WIKI: Foreword By Alberto Savoia

prashantbansode — Wed, 29 Aug 2007 20:07:53 GMT

Foreword By Alberto Savoia

Testing the performance of web applications is easy. It’s easy to design unrealistic scenarios. Easy to collect and measure irrelevant performance data. And, even if you manage to design a sound scenario and collect the right data, it’s easy to use the wrong statistical methods to summarize and present the results.

Starting in the late 90s, through the peak of the Internet bubble and beyond, I spent a lot of time testing the performance of web applications. During that period, I designed and led several mission-critical web performance and load tests for high-profile Internet companies. Working with the in-house performance experts at each company was very revealing – and quite frightening. Most of the engineers assigned to work on web application performance were smart, hard-working, and dedicated; they invested in expensive software and hardware, read the right books, and followed the best practices of the day. But, somehow, the results of their performance measurements and predictions did not match reality. In some cases the performance tests overestimated the performance and scalability of the web applications – leading to embarrassing and costly crashes when the web application was deployed. In other cases, they underestimated capacity and scalability – leading to unnecessary spending on hardware and infrastructure. The errors in these tests were not small; some tests overestimated or underestimated actual performance and capacity by an order of magnitude or more! How is this possible?

Based on my experience, the majority of gross errors in web application performance testing are the result of oversimplification. More precisely, they are the result oversimplification of user behavior and oversimplification in summarizing and reporting test results. Imagine a transportation engineer estimating traffic patterns for a proposed stretch of highway by assuming that most drivers will drive at the same average speed, break and accelerate with the same response time and at the same rate, and never change lanes. A simple – but completely worthless – scenario. Or imagine the same transportation engineer reporting that there are no traffic issues because the average speed is 57mph – without bringing up that during rush-hour the average speed is 25mph. A simple, but very misleading, result. Unfortunately, most web application performance testers commit errors of oversimplification as bad, or worse, as the ones committed by our hypothetical transportation engineer.

I am all for simplicity but, as Albert Einstein once said: “Make everything as simple as possible, but not simpler.” When it comes to testing the performance of web applications, that’s exactly what this remarkable – and much needed – book teaches you. The authors leverage their passion, experience, and hard-earned knowledge and provide you with the broad, thorough, and extensible foundation you need to tackle web performance testing the right way. Performance Testing Guidance for Web Applications does not get bogged down with unnecessary details, but it does make sure that you know about – and don’t overlook – the key parameters and variables that you need to take into account in designing, conducting, and analyzing your tests.

If you are new to web performance testing, this book will get you started on the right path and save you a lot of time and embarrassment. Even if you are a seasoned web performance testing veteran, I am confident that this book will provide you with new insights and, most likely, have you slap your forehead a few times as you read about some common and familiar mistakes. In either case, Performance Testing Guidance for Web Applications, is a must-have for any web performance engineer bookshelf.

Alberto Savoia
Founder and CTO, Agitar Software Inc.
July, 2007

Author of: “The Science and Art of Web Site Load Testing”, “Web Load Test Planning”, and “Trade Secrets from a Web Testing Expert”.

UPDATED WIKI: Home

prashantbansode — Wed, 29 Aug 2007 20:04:56 GMT

patterns & practices Performance Testing Guidance for Web Applications

Download the Guide

The Final Release is Available! Start using the guide today, while we continue to make improvements.

Download the Performance Testing Guidance for Web Applications Guide

Parts

Part 1, Introduction to Performance Testing
Part II, Exemplar Performance Testing Approaches
Part III, Identify the Test Environment
Part IV, Identify Performance Acceptance Criteria
Part V, Plan and Design Tests
Part VI, Execute Tests
Part VII, Analyze Results and Report
Part VIII, Performance Testing Techniques

Forewards

Chapters

Introduction

Part 1, Introduction to Performance Testing

Part II, Exemplar Performance Testing Approaches

Part III, Identify the Test Environment

Chapter 8 – Evaluating Systems to Increase Performance-Testing Effectiveness

Part IV, Identify Performance Acceptance Criteria

Part V, Plan and Design Tests

Part VI, Execute Tests

Chapter 14 – Test Execution

Part VII, Analyze Results and Report

Part VIII, Performance-Testing Techniques

Reference

Glossary

Team

Core Team: J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea
Contributors and Reviewers

Related Sites

patterns & practices Performance Testing Guidance

UPDATED WIKI: Home

prashantbansode — Mon, 27 Aug 2007 20:24:33 GMT

patterns & practices Performance Testing Guidance for Web Applications

Download the Guide

The Final Release is Available! Start using the guide today, while we continue to make improvements.

Download the Performance Testing Guidance for Web Applications Guide

Parts

Part 1, Introduction to Performance Testing
Part II, Exemplar Performance Testing Approaches
Part III, Identify the Test Environment
Part IV, Identify Performance Acceptance Criteria
Part V, Plan and Design Tests
Part VI, Execute Tests
Part VII, Analyze Results and Report
Part VIII, Performance Testing Techniques

Chapters

Introduction

Part 1, Introduction to Performance Testing

Part II, Exemplar Performance Testing Approaches

Part III, Identify the Test Environment

Chapter 8 – Evaluating Systems to Increase Performance-Testing Effectiveness

Part IV, Identify Performance Acceptance Criteria

Part V, Plan and Design Tests

Part VI, Execute Tests

Chapter 14 – Test Execution

Part VII, Analyze Results and Report

Part VIII, Performance-Testing Techniques

Reference

Glossary

Team

Core Team: J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea
Contributors and Reviewers

Related Sites

patterns & practices Performance Testing Guidance

UPDATED WIKI: Contributors and Reviewers

prashantbansode — Thu, 23 Aug 2007 19:08:24 GMT

Contributors and Reviewers

Microsoft Contributors / Reviewers

Alan Ridlehoover; Clint Huffman; Edmund Wong; Ken Perilman; Larry Brader; Mark Tomlinson; Paul Williams; Pete Coupland; Rico Mariani

External Contributors/Reviewers

Alberto Savoia; Ben Simo; Cem Kaner; Chris Loosley; Corey Goldberg; Dawn Haynes; Derek Mead; Karen N. Johnson; Mike Bonar; Pradeep Soundararajan; Richard Leeke; Roland Stens; Ross Collard; Steven Woody

UPDATED WIKI: Chapter 18 – Stress-Testing Web Applications

prashantbansode — Thu, 23 Aug 2007 19:06:51 GMT

Chapter 18 – Stress Testing Web Applications

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Understand the key concepts of stress testing.
Learn how to stress-test a Web application.

Overview

Stress testing is a type of performance testing focused on determining an application’s robustness, availability, and reliability under extreme conditions. The goal of stress testing is to identify application issues that arise or become apparent only under extreme conditions. These conditions can include heavy loads, high concurrency, or limited computational resources. Proper stress testing is useful in finding synchronization and timing bugs, interlock problems, priority problems, and resource loss bugs. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not expected to process the overload without adequate resources, but to behave (e.g., fail) in an acceptable manner (e.g., not corrupting or losing data).

Stress tests typically involve simulating one or more key production scenarios under a variety of stressful conditions. For example, you might deploy your application on a server that is already running a processor-intensive application; in this way, your application is immediately “starved” of processor resources and must compete with the other application for processor cycles. You can also stress-test a single Web page or even a single item such as a stored procedure or class.

This chapter presents a high-level introduction to stress-testing a Web application. Stress testing can help you identify application issues that surface only under extreme conditions.

Examples of Stress Conditions

Examples of stress conditions include:

Excessive volume in terms of either users or data; examples might include a denial of service (DoS) attack or a situation where a widely viewed news item prompts a large number of users to visit a Web site during a three-minute period.
Resource reduction such as a disk drive failure.
Unexpected sequencing.
Unexpected outages/outage recovery.

Examples of Stress-Related Symptoms

Examples of stress-related symptoms include:

Data is lost or corrupted.
Resource utilization remains unacceptably high after the stress is removed.
Application components fail to respond.
Unhandled exceptions are presented to the end user.

How to Use This Chapter

Use this chapter to understand the key concepts of stress testing and the steps involved in stress-testing a Web application. To get the most from this chapter:

Use the “Input” and “Output” sections to understand the key inputs for stress-testing a Web application and the key outcomes of this type of testing.
Use the “Approach for Stress Testing” section to get** an overview of the approach for stress-testing a Web application, and as quick reference guide for you and your team.
Use the various steps sections to understand the details of each step involved in stress-testing a Web application.
Use the “Usage Scenario for Stress Testing” section to understand various real-world scenarios where stress testing is employed.

Input

To perform stress testing, you are likely to use as reference one or more of the following items:

Results from previous stress tests
Application usage characteristics (scenarios)
Concerns about those scenarios under extreme conditions
Workload profile characteristics
Current peak load capacity (obtained from load testing)
Hardware and network architecture and data
Disaster-risk assessment (e.g., likelihood of blackouts, earthquakes, etc.)

Output

Output from a stress test may include:

Measures of the application under stressful conditions
Symptoms of the application under stress
Information the team can use to address robustness, availability, and reliability

Approach for Stress Testing

The following steps are involved in stress-testing a Web application:

Step1 - Identify test objectives. Identify the objectives of stress testing in terms of the desired outcomes of the testing activity.
Step 2 - Identify key scenario(s). Identify the application scenario or cases that need to be stress-tested to identify potential problems.
Step 3 - Identify the workload. Identify the workload that you want to apply to the scenarios identified during the “Identify objectives” step. This is based on the workload and peak load capacity inputs.
Step 4 - Identify metrics. Identify the metrics that you want to collect about the application’s performance. Base these metrics on the potential problems identified for the scenarios you identified during the “Identify objectives” step.
Step 5 - Create test cases. Create the test cases in which you define steps for running a single test, as well as your expected results.
Step 6 - Simulate load. Use test tools to simulate the required load for each test case and capture the metric data results.
Step 7 - Analyze results. Analyze the metric data captured during the test.

These steps are graphically represented below; the following sections discuss each step in detail.

_Figure 18.1 Stress Testing Steps_

Step 1 - Identify Test Objectives

Asking yourself or others the following questions can help in identifying the desired outcomes of your stress testing:

Is the purpose of the test to identify the ways the system can possibly fail catastrophically in production?
Is it to provide information to the team in order to build defenses against catastrophic failures?
Is it to identify how the application behaves when system resources such as memory, disk space, network bandwidth, or processor cycles are depleted?
Is it to ensure that functionality does not break under stress? For example, there may be cases where operational performance metrics meet the objectives, but the functionality of the application is failing to meet them — orders are not inserted in the database, the application is not returning the complete product information in searches, form controls are not being populated properly, redirects to custom error pages are occurring during the stress testing, and so on.

Step 2 - Identify Key Scenario(s)

To get the most value out of a stress test, the test needs to focus on the behavior of the usage scenario or scenarios that matter most to the overall success of the application. To identify these scenarios, you generally start by defining a single scenario that you want to stress-test in order to identify a potential performance issue. Consider these guidelines when choosing appropriate scenarios:

Select scenarios based on how critical they are to overall application performance.
Try to test those operations that are most likely to affect performance. These might include operations that perform intensive locking and synchronization, long transactions, and disk-intensive input/output (I/O) operations.
Base your scenario selection on the specific areas of your application identified as potential bottlenecks by load-testing data. Although you should have fine-tuned and removed the bottlenecks after load testing, you should still stress-test the system in these areas to verify how well your changes handle extreme stress levels.

Examples of scenarios that may need to be stress tested separately from other usage scenarios for a typical e-commerce application include the following:

An order-processing scenario that updates the inventory for a particular product. This functionality has the potential to exhibit locking and synchronization problems.
A scenario that pages through search results based on user queries. If a user specifies a particularly wide query, there could be a large impact on memory utilization. For example, memory utilization could be affected if a query returns an entire data table.

Step 3 - Identify the Workload

The load you apply to a particular scenario should stress the system sufficiently beyond threshold limits to enable you to observe the consequences of the stress condition. One method to determine the load at which an application begins to exhibit signs of stress is to incrementally increase the load and observe the application behavior under various load conditions. The key is to systematically test with various workloads until you create a significant failure. These variations may be accomplished by adding more users, reducing delay times, adding or reducing the number and type of user activities represented, or adjusting test data.

For example, a stress test could be designed to simulate every registered user of the application attempting to log on during one 30-second period. This would simulate a situation where the application suddenly became available again after a period of downtime and all users were anxiously refreshing their browsers, waiting for the application to come back online. Although this situation does not occur frequently in the real world, it does happen often enough for there to be real value in learning how the application will respond if it does.

Remember to represent the workload with accurate and realistic test data — type and volume, different user logins, product IDs, product categories, and so on — allowing you to simulate important failures such as deadlocks or resource consumption.

The following activities are generally useful in identifying appropriate workloads for stress testing:

Identify the distribution of work. For each key scenario, identify the distribution of work to be simulated. The distribution is based on the number and type of users executing the scenario during the stress test.
Estimate peak user loads. Identify the maximum expected number of users during peak load conditions for the application. Using the work distribution you identified for each scenario, calculate the percentage of user load per key scenario.
Identify the anti-profile. As an alternative, you can start by applying an anti-profile to the normal workload. In an anti-profile, the workload distributions are inverted for the scenario under consideration. For example, if the normal load for the order-processing scenario is 10 percent of the total workload, the anti-profile would be 90 percent of the total workload. The remaining load can be distributed among the other scenarios. Using an anti-profile can serve as a valuable starting point for your stress tests because it ensures that the critical scenarios are subjected to loads beyond the normal load conditions.

Step 4 - Identify Metrics

When identified and captured correctly, metrics provide information about how well or poorly your application is performing as compared to your performance objectives. In addition, metrics can help you identify problem areas and bottlenecks within your application.

Using the desired performance characteristics identified during the “Identify objectives” step, identify metrics to be captured that focus on potential pitfalls for each scenario. The metrics can be related to both performance and throughput goals as well as providing information about potential problems; for example, custom performance counters that have been embedded in the application.

When identifying metrics, you will use either direct objectives or indicators that are directly or indirectly related to those objectives. The following table describes performance metrics in terms of related performance objectives.

Performance metrics	Category
Base set of metrics
Processor	Processor utilization
Process	Memory consumption
	Processor utilization
	Process recycles
Memory	Memory available
	Memory utilization
Disk	Disk utilization
Network	Network utilization
Transactions/business metrics	Transactions/sec
	Transactions succeeded
	Transactions failed
	Orders succeeded
	Orders failed
Threading	Contentions per second
	Deadlocks
	Thread allocation
Response times	Transactions times

Step 5 - Create Test Cases

Identifying workload profiles and key scenarios generally does not provide all of the information necessary to implement and execute test cases. Additional inputs for completely designing a stress test include performance objectives, workload characteristics, test data, test environments, and identified metrics. Each test design should mention the expected results and/or the key data of interest to be collected, in such a way that each test case can be marked as a “pass,” “fail,” or “inconclusive” after execution.

The following is an example of a test case based on the order-placement scenario.

Test 1 – Place Order Scenario

Workload: 1,000 simultaneous users.
Think time: Use a random think time between 1 and 10 seconds in the test script after each operation.
Test Duration: Run the test for two days.

Expected results:

Application hosting process should not recycle because of deadlock or memory consumption.
Throughput should not fall below 35 requests per second.
Response time should not be greater than 7 seconds for 95 percent of total transactions completed.
“Server busy” errors should not be more than 10 percent of the total response because of contention-related issues.
Order transactions should not fail during test execution. Database entries should match the “Transactions succeeded” count.

Step 6 - Simulate Load

After you have completed the previous steps to an appropriate degree, you should be ready to simulate the load executing the stress test. Typically, test execution follows these steps:

Validate that the test environment matches the configuration that you were expecting and/or designed your test for.
Ensure that both the test and the test environment are correctly configured for metrics collection.
Before running the test, execute a quick “smoke test” to make sure that the test script and remote performance counters are working correctly.
Reset the system (unless your scenario is to do otherwise) and start a formal test execution.

Note: Make sure that the client (a.k.a. load generator) computers that you use to generate load are not overly stressed. Utilization of resources such as processor and memory should remain low enough to ensure that the load-generation environment is not itself a bottleneck.

Step 7 - Analyze Results

Analyze the captured data and compare the results against the metric’s accepted level. If the results indicate that your required performance levels have not been attained, analyze and fix the cause of the bottleneck. To address observed issues, you might need to do one or more of the following:

Perform a design review.
Perform a code review.
Run stress tests in environments where it is possible to debug possible causes of failures, during test execution.

In situations where performance issues are observed, but only under conditions that are deemed to be unlikely enough to warrant tuning at the current time, you may want to consider conducting additional tests to identify an early indicator for the issue in order to avoid unwanted surprises.

Usage Scenarios for Stress Testing

The following are examples of how stress testing is applied in practice:

Application stress testing. This type of test typically focuses on more than one transaction on the system under stress, without the isolation of components. With application stress testing, you are likely to uncover defects related to data locking and blocking, network congestion, and performance bottlenecks on different components or methods across the entire application. Because the test scope is a single application, it is common to use this type of stress testing after a robust application load-testing effort, or as a last test phase for capacity planning. It is also common to find defects related to race conditions and general memory leaks from shared code or components.
Transactional stress testing. Transactional stress tests aim at working at a transactional level with load volumes that go beyond those of the anticipated production operations. These tests are focused on validating behavior under stressful conditions, such as high load with same resource constraints, when testing the entire application. Because the test isolates an individual transaction, or group of transactions, it allows for a very specific understanding of throughput capacities and other characteristics for individual components without the added complication of inter-component interactions that occurs in testing at the application level. These tests are useful for tuning, optimizing, and finding error conditions at the specific component level.
*Systemic stress testing.* In this type of test, stress or extreme load conditions are generated across multiple applications running on the same system, thereby pushing the boundaries of the applications’ expected capabilities to an extreme. The goal of systemic stress testing is to uncover defects in situations where different applications block one another and compete for system resources such as memory, processor cycles, disk space, and network bandwidth. This type of testing is also known as integration stress testing or consolidation stress testing. In large-scale systemic stress tests, you stress all of the applications together in the same consolidated environment. Some organizations choose to perform this type of testing in a larger test lab facility, sometimes with the hardware or software vendor’s assistance.

Exploratory Stress Testing

Exploratory stress testing is an approach to subjecting a system, application, or component to a set of unusual parameters or conditions that are unlikely to occur in the real world but are nevertheless possible. In general, exploratory testing can be viewed as an interactive process of simultaneous learning, test design, and test execution. Most often, exploratory stress tests are designed by modifying existing tests and/or working with application/system administrators to create unlikely but possible conditions in the system. This type of stress testing is seldom conducted in isolation because it is typically conducted to determine if more systematic stress testing is called for related to a particular failure mode. The following are some examples of exploratory stress tests to determine the answer to “How will the system respond if…?”

All of the users logged on at the same time.
The load balancer suddenly failed.
All of the servers started their scheduled virus scan at the same time during a period of peak load.
The database went offline during peak usage.

Summary

Stress testing allows you to identify potential application issues that surface only under extreme conditions. Such conditions range from exhaustion of system resources such as memory, processor cycles, network bandwidth, and disk capacity to excessive load due to unpredictable usage patterns, common in Web applications.

Stress testing centers around objectives and key user scenarios with an emphasis on the robustness, reliability, and stability of the application. The effectiveness of stress testing relies on applying the correct methodology and being able to effectively analyze testing results. Applying the correct methodology is dependent on the capacity for reproducing workload conditions for both user load and volume of data, reproducing key scenarios, and interpreting the key performance metrics.

UPDATED WIKI: Chapter 18 – Stress-Testing Web Applications

prashantbansode — Thu, 23 Aug 2007 19:06:15 GMT

Chapter 18 – Stress Testing Web Applications

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Understand the key concepts of stress testing.
Learn how to stress-test a Web application.

Overview

Examples of Stress Conditions

Examples of stress conditions include:

Excessive volume in terms of either users or data; examples might include a denial of service (DoS) attack or a situation where a widely viewed news item prompts a large number of users to visit a Web site during a three-minute period.
Resource reduction such as a disk drive failure.
Unexpected sequencing.
Unexpected outages/outage recovery.

Examples of Stress-Related Symptoms

Examples of stress-related symptoms include:

Data is lost or corrupted.
Resource utilization remains unacceptably high after the stress is removed.
Application components fail to respond.
Unhandled exceptions are presented to the end user.

How to Use This Chapter

Use this chapter to understand the key concepts of stress testing and the steps involved in stress-testing a Web application. To get the most from this chapter:

Use the “Input” and “Output” sections to understand the key inputs for stress-testing a Web application and the key outcomes of this type of testing.
Use the “Approach for Stress Testing” section to get** an overview of the approach for stress-testing a Web application, and as quick reference guide for you and your team.
Use the various steps sections to understand the details of each step involved in stress-testing a Web application.
Use the “Usage Scenario for Stress Testing” section to understand various real-world scenarios where stress testing is employed.

Input

To perform stress testing, you are likely to use as reference one or more of the following items:

Results from previous stress tests
Application usage characteristics (scenarios)
Concerns about those scenarios under extreme conditions
Workload profile characteristics
Current peak load capacity (obtained from load testing)
Hardware and network architecture and data
Disaster-risk assessment (e.g., likelihood of blackouts, earthquakes, etc.)

Output

Output from a stress test may include:

Measures of the application under stressful conditions
Symptoms of the application under stress
Information the team can use to address robustness, availability, and reliability

Approach for Stress Testing

The following steps are involved in stress-testing a Web application:

Step1 - Identify test objectives. Identify the objectives of stress testing in terms of the desired outcomes of the testing activity.
Step 2 - Identify key scenario(s). Identify the application scenario or cases that need to be stress-tested to identify potential problems.
Step 3 - Identify the workload. Identify the workload that you want to apply to the scenarios identified during the “Identify objectives” step. This is based on the workload and peak load capacity inputs.
Step 4 - Identify metrics. Identify the metrics that you want to collect about the application’s performance. Base these metrics on the potential problems identified for the scenarios you identified during the “Identify objectives” step.
Step 5 - Create test cases. Create the test cases in which you define steps for running a single test, as well as your expected results.
Step 6 - Simulate load. Use test tools to simulate the required load for each test case and capture the metric data results.
Step 7 - Analyze results. Analyze the metric data captured during the test.

These steps are graphically represented below; the following sections discuss each step in detail.

Cannot resolve link: [image:Fig18.1.GIF]

_Figure 18.1 Stress Testing Steps_

Step 1 - Identify Test Objectives

Asking yourself or others the following questions can help in identifying the desired outcomes of your stress testing:

Is the purpose of the test to identify the ways the system can possibly fail catastrophically in production?
Is it to provide information to the team in order to build defenses against catastrophic failures?
Is it to identify how the application behaves when system resources such as memory, disk space, network bandwidth, or processor cycles are depleted?
Is it to ensure that functionality does not break under stress? For example, there may be cases where operational performance metrics meet the objectives, but the functionality of the application is failing to meet them — orders are not inserted in the database, the application is not returning the complete product information in searches, form controls are not being populated properly, redirects to custom error pages are occurring during the stress testing, and so on.

Step 2 - Identify Key Scenario(s)

Select scenarios based on how critical they are to overall application performance.
Try to test those operations that are most likely to affect performance. These might include operations that perform intensive locking and synchronization, long transactions, and disk-intensive input/output (I/O) operations.
Base your scenario selection on the specific areas of your application identified as potential bottlenecks by load-testing data. Although you should have fine-tuned and removed the bottlenecks after load testing, you should still stress-test the system in these areas to verify how well your changes handle extreme stress levels.

Examples of scenarios that may need to be stress tested separately from other usage scenarios for a typical e-commerce application include the following:

An order-processing scenario that updates the inventory for a particular product. This functionality has the potential to exhibit locking and synchronization problems.
A scenario that pages through search results based on user queries. If a user specifies a particularly wide query, there could be a large impact on memory utilization. For example, memory utilization could be affected if a query returns an entire data table.

Step 3 - Identify the Workload

Identify the distribution of work. For each key scenario, identify the distribution of work to be simulated. The distribution is based on the number and type of users executing the scenario during the stress test.
Estimate peak user loads. Identify the maximum expected number of users during peak load conditions for the application. Using the work distribution you identified for each scenario, calculate the percentage of user load per key scenario.
Identify the anti-profile. As an alternative, you can start by applying an anti-profile to the normal workload. In an anti-profile, the workload distributions are inverted for the scenario under consideration. For example, if the normal load for the order-processing scenario is 10 percent of the total workload, the anti-profile would be 90 percent of the total workload. The remaining load can be distributed among the other scenarios. Using an anti-profile can serve as a valuable starting point for your stress tests because it ensures that the critical scenarios are subjected to loads beyond the normal load conditions.

Step 4 - Identify Metrics

Performance metrics	Category
Base set of metrics
Processor	Processor utilization
Process	Memory consumption
	Processor utilization
	Process recycles
Memory	Memory available
	Memory utilization
Disk	Disk utilization
Network	Network utilization
Transactions/business metrics	Transactions/sec
	Transactions succeeded
	Transactions failed
	Orders succeeded
	Orders failed
Threading	Contentions per second
	Deadlocks
	Thread allocation
Response times	Transactions times

Step 5 - Create Test Cases

Test 1 – Place Order Scenario

Application hosting process should not recycle because of deadlock or memory consumption.
Throughput should not fall below 35 requests per second.
Response time should not be greater than 7 seconds for 95 percent of total transactions completed.
“Server busy” errors should not be more than 10 percent of the total response because of contention-related issues.
Order transactions should not fail during test execution. Database entries should match the “Transactions succeeded” count.

Step 6 - Simulate Load

After you have completed the previous steps to an appropriate degree, you should be ready to simulate the load executing the stress test. Typically, test execution follows these steps:

Validate that the test environment matches the configuration that you were expecting and/or designed your test for.
Ensure that both the test and the test environment are correctly configured for metrics collection.
Before running the test, execute a quick “smoke test” to make sure that the test script and remote performance counters are working correctly.
Reset the system (unless your scenario is to do otherwise) and start a formal test execution.

Step 7 - Analyze Results

Perform a design review.
Perform a code review.
Run stress tests in environments where it is possible to debug possible causes of failures, during test execution.

Usage Scenarios for Stress Testing

The following are examples of how stress testing is applied in practice:

Application stress testing. This type of test typically focuses on more than one transaction on the system under stress, without the isolation of components. With application stress testing, you are likely to uncover defects related to data locking and blocking, network congestion, and performance bottlenecks on different components or methods across the entire application. Because the test scope is a single application, it is common to use this type of stress testing after a robust application load-testing effort, or as a last test phase for capacity planning. It is also common to find defects related to race conditions and general memory leaks from shared code or components.
Transactional stress testing. Transactional stress tests aim at working at a transactional level with load volumes that go beyond those of the anticipated production operations. These tests are focused on validating behavior under stressful conditions, such as high load with same resource constraints, when testing the entire application. Because the test isolates an individual transaction, or group of transactions, it allows for a very specific understanding of throughput capacities and other characteristics for individual components without the added complication of inter-component interactions that occurs in testing at the application level. These tests are useful for tuning, optimizing, and finding error conditions at the specific component level.
*Systemic stress testing.* In this type of test, stress or extreme load conditions are generated across multiple applications running on the same system, thereby pushing the boundaries of the applications’ expected capabilities to an extreme. The goal of systemic stress testing is to uncover defects in situations where different applications block one another and compete for system resources such as memory, processor cycles, disk space, and network bandwidth. This type of testing is also known as integration stress testing or consolidation stress testing. In large-scale systemic stress tests, you stress all of the applications together in the same consolidated environment. Some organizations choose to perform this type of testing in a larger test lab facility, sometimes with the hardware or software vendor’s assistance.

Exploratory Stress Testing

All of the users logged on at the same time.
The load balancer suddenly failed.
All of the servers started their scheduled virus scan at the same time during a period of peak load.
The database went offline during peak usage.

Summary

UPDATED WIKI: Chapter 17 – Load-Testing Web Applications

prashantbansode — Thu, 23 Aug 2007 18:49:14 GMT

Chapter 17 – Load-Testing Web Applications

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Understand the key concepts of load testing.
Learn how to load-test a Web application.

Overview

This chapter explains how to load-test a Web application. Load testing helps to identify the maximum operating capacity of an application as well as any bottlenecks that might interfere with its operating at capacity. The basic approach to performing load testing on a Web application is:

Identify the performance-critical scenarios.
Identify the workload profile for distributing the entire load among the key scenarios.
Identify the metrics that you want to collect in order to verify them against your performance objectives.
Design tests to simulate the load.
Use tools to implement the load according to the designed tests, and capture the metrics.
Analyze the metric data captured during the tests.

By using an iterative testing process, these steps should help you achieve your performance objectives.

There are many reasons for load-testing a Web application. The most basic type of load testing is used to determine the Web application’s behavior under both normal and anticipated peak load conditions. As you begin load testing, it is recommended that you start with a small number of virtual users and then incrementally increase the load from normal to peak. You can then observe how your application performs during this gradually increasing load condition. Eventually, you will cross a threshold limit for your performance objectives. For example, you might continue to increase the load until the server processor utilization reaches 75 percent, or when end-user response times exceed 8 seconds.

How to Use This Chapter

Use this chapter to understand the key concepts of load testing and the steps involved in load-testing Web applications. To get the most from this chapter:

Use the “Input” and “Output” sections to understand the key inputs for load testing a Web application and the key outcomes of doing so.
Use the “Approach for Load Testing” section to get** an overview of the approach for load testing a Web application, and as quick reference guide for you and your team.
Use the various steps sections to understand the details of each step involved in load-testing a Web application.

Input

The following are useful inputs for load-testing a Web application:

Performance-critical usage scenarios
Workload models
Performance acceptance criteria
Performance metrics associated with the acceptance criteria
Interview feedback from the designer or developer of the Web application
Interview feedback from end users of the application
Interview feedback from the operations personnel who will maintain and manage the application

Output

The main outcomes that load testing helps you to accomplish are:

Updated test plans and test designs for load and performance testing
Various performance measures such as throughput, response time, and resource utilization
Potential bottlenecks that need to be analyzed in the white-box testing phase
The behavior of the application at various load levels

Approach for Load Testing

The following steps are involved in load-testing a Web application:

Step 1 - Identify performance acceptance criteria
Step 2 - Identify key scenarios
Step 3 - Create a workload model
Step 4 - Identify the target load levels
Step 5 - Identify metrics
Step 6 - Design specific tests
Step 7 - Run tests
Step 8 - Analyze the results

These steps are graphically represented below. The sections that follow discuss each step in detail.

_Figure 17.1 Load Testing Steps_

Step 1 - Identify Performance Acceptance Criteria

Identifying performance acceptance criteria is most valuable when initiated early in the application’s development life cycle. It is frequently valuable to record the acceptance criteria for your application and store them in a place and format that is available to the entire team for review and comment. Criteria are typically determined by balancing your business, industry, technology, competitive, and user requirements.

Test objectives frequently include the following:

Response time. For example, the product catalog must be displayed in less than 3 seconds.
Throughput. For example, the system must support 100 transactions per second.
Resource utilization. A frequently overlooked aspect is the amount of resources your application is consuming, in terms of processor, memory, disk input output (I/O), and network I/O.
Maximum user load. This test objective determines how many users can run on a specific hardware configuration.
Business related metrics. This objective is mapped to business volume at normal and peak values; for example, the number of orders or Help desk calls handled at a given time.

Step 2 - Identify Key Scenarios

Scenarios are anticipated user paths that generally incorporate multiple application activities. Key scenarios are those for which you have specific performance goals, those considered to be high-risk, those that are most commonly used, or those with a significant performance impact. The basic steps for identifying key scenarios are.

Identify all the scenarios for a Web application. For example, even the most basic e-commerce application must support the following user scenarios:

Browse catalog
Search for a product
Place an order

Identify the activities involved in each of the scenarios. For example, a “Place an Order” scenario will include the following activities:

Log on to the application.
Browse the product catalog.
Search for a specific product.
Add items to the shopping cart.
Validate credit card details and place an order.

Identify the scenarios that are most commonly executed or most resource-intensive; these will be the key scenarios used for load testing. For example, in an e-commerce application, browsing a catalog may be the most commonly executed scenario, whereas placing an order may be the most resource-intensive scenario because it accesses the database.

The most commonly executed scenarios for an existing Web application can be determined by examining the log files.
The most commonly executed scenarios for a new Web application can be obtained from market research, historical data, market trends, and so on.
Resource-intensive scenarios can be identified by using design documents or the actual code implementation. The primary resources are:

Processor
Memory
Disk I/O
Network I/O

Once they have been identified, you will use these key scenarios to create workload profiles and to design load tests.

Step 3 - Create a Workload Model

When defining workload distribution, consider the following key points for determining the characteristics for user scenarios:

A user scenario is defined as a navigational path, including intermediate steps or activities, taken by the user to complete a task. This can also be thought of as a user session.
A user will typically pause between pages during a session. This is known as user delay or think time.
A session will have an average duration when viewed across multiple users. It is important to account for this when defining the load levels that will translate into concurrent usage, overlapping users, or user sessions per unit of time.
Not all scenarios can be performed by a new user, a returning user, or either; know who you expect your primary users to be and test accordingly.

Step 4 - Identify Target Load Levels

Identify the load levels to be applied to the workload distribution(s) identified during the previous step. The purpose of identifying target load levels is to ensure that your tests can be used to predict or compare a variety of production load conditions. The following are common inputs used for determining target load levels:

Business volume (both current and projected) as it relates to your performance objectives
Key scenarios
Distribution of work
Session characteristics (navigational path, duration, percentage of new users)

By combining the items above, you can determine the remaining details necessary to implement the workload model under a particular target load.

Step 5 - Identify Metrics

There is a virtually unlimited number of metrics that can be collected during a performance test execution. However, collecting too many metrics can make analysis unwieldy as well as negatively impact the application’s actual performance. For these reasons, it is important to identify the metrics that are most relevant to your performance objectives and those that you anticipate will help you to identify bottlenecks. Only well-selected metrics that are analyzed correctly and contextually provide information of value.

The following are a few suggestions for identifying the metrics that will provide the most valuable information to your project:

Define questions related to your application performance that can be easily tested. For example, what is the checkout response time when placing an order? How many orders are placed in a minute? These questions have definite answers.
With the answers to these questions, determine quality goals for comparison against external expectations. For example, checkout response time should be 30 seconds, and a maximum of 10 orders should be placed in a minute. The answers are based on market research, historical data, market trends, and so on.
Identify the metrics. Using your list of performance-related questions and answers, identify the metrics that provide information related to those questions and answers.
Identify supporting metrics. Using the same approach, you can identify lower-level metrics that focus on measuring the performance and identifying the bottlenecks in the system. When identifying low-level metrics, most teams find it valuable to determine a baseline for those metrics under single-user and/or normal load conditions. This helps you determine the acceptable load levels for your application. Baseline values help you analyze your application performance at varying load levels and serve as a starting point for trend analysis across builds or releases.
Reevaluate the metrics to be collected regularly. Goals, priorities, risks, and current issues are bound to change over the course of a project. With each of these changes, different metrics may provide more value than the ones that have previously been identified.

Additionally, to evaluate the performance of your application in more detail and to identify potential bottlenecks, it is frequently useful to monitor metrics in the following categories:

Network-specific metrics. This set of metrics provides information about the overall health and efficiency of your network, including routers, switches, and gateways.
System-related metrics. This set of metrics helps you identify the resource utilization on your server. The resources being utilized are processor, memory, disk I/O, and network I/O.
Platform-specific metrics. Platform-specific metrics are related to software that is used to host your application, such as the Microsoft .NET Framework common language runtime (CLR) and ASP.NET-related metrics.
Application-specific metrics. These include custom performance counters inserted in your application code to monitor application health and identify performance issues. You might use custom counters to determine the number of concurrent threads waiting to acquire a particular lock, or the number of requests queued to make an outbound call to a Web service.
Service-level metrics. These metrics can help to measure overall application throughput and latency, or they might be tied to specific business scenarios.
Business metrics. These metrics are indicators of business-related information, such as the number of orders placed in a given timeframe.

Step 6 - Design Specific Tests

Using your scenarios, key metrics, and workload analysis, you can now design specific tests to be conducted. Each test will generally have a different purpose, collect different data, include different scenarios, and have different target load levels. The key is to design tests that will help the team collect the information it needs in order to understand, evaluate, or tune the application.

Points to consider when designing tests include:

Do not change your test design because the design is difficult to implement in your tool.
If you cannot implement your test as designed, ensure that you record the details pertaining to the test that you do implement.
Ensure that the model contains all of the supplementary data needed to create the actual test.
Consider including invalid data in your performance tests. For example, include some users who mistype their password on the first attempt but get it correct on a second try.
First-time users usually spend significantly more time on each page or activity than experienced users.
The best possible test data is test data collected from a production database or log file.
Think about nonhuman system users and batch processes as well as end users. For example, there might be a batch process that runs to update the status of orders while users are performing activities on the site. In this situation, you would need to account for those processes because they might be consuming resources.
Do not get overly caught up in striving for perfection, and do not fall into the trap of oversimplification. In general, it is a good idea to start executing tests when you have a reasonable test designed and then enhance the design incrementally while collecting results.

Step 7 - Run Tests

Poor load simulations can render all of the work in the previous activities useless. To understand the data collected from a test execution, the load simulation must reflect the test design. When the simulation does not reflect the test design, the results are prone to misinterpretation. Consider the following steps when preparing to simulate load:

Configure the test environment in such a way that it mirrors your production environment as closely as possible, noting and accounting for all differences between the two.
Ensure that performance counters relevant for identified metrics and resource utilization are being measured and are not interfering with the accuracy of the simulation.
Use appropriate load-generation tools to create a load with the characteristics specified in your test design.
Using the load-generation tool(s), execute tests by first building up to the target load specified in your test design, in order to validate the correctness of the simulation. Some things to consider during test execution include:

Begin load testing with a small number of users distributed against your user profile, and then incrementally increase the load. It is important to allow time for the system to stabilize between increases in load while evaluating the correctness of the simulation.
Consider continuing to increase the load and record the behavior until you reach the threshold for the resources identified in your performance objectives, even if that load is beyond the target load specified in the test design. Information about when the system crosses identified thresholds is just as important as the value of the metrics at the target load of the test.
Similarly, it is frequently valuable to continue to increase the number of users until you run up against the service-level limits beyond which you would be violating your SLAs for throughput, response time, and resource utilization.

Note: Make sure that the client computers (agents) you use to generate load are not overly stressed. Resource utilization such as processor and memory must remain well below the utilization threshold values to ensure accurate test results.

Step 8 - Analyze the Results

You can analyze the test results to find performance bottlenecks between each test run or after all testing has been completed. Analyzing the results correctly requires training and experience with graphing correlated response time and system data.

The following are the steps for analyzing the data:

Analyze the captured data and compare the results against the metric’s accepted level to determine whether the performance of the application being tested shows a trend toward or away from the performance objectives.
Analyze the measured metrics to diagnose potential bottlenecks. Based on the analysis, if required, capture additional metrics in subsequent test cycles. For example, suppose that during the first iteration of load tests, the process shows a marked increase in memory consumption, indicating a possible memory leak. In the subsequent iterations, additional memory counters related to generations can be captured to study the memory allocation pattern for the application.

Summary

Load testing helps to identify the maximum operating capacity of the application and any bottlenecks that might be degrading performance.

The basic methodology for performing load testing on a Web application is to identify the performance-critical key scenarios; identify the workload profile for distributing all the load among the key scenarios; identify metrics that you want to collect in order to verify them against your performance objectives; create test cases that will be used to simulate the load test; use tools to simulate the load according to the test cases and capture the metrics; and finally, analyze the metrics data captured during the tests.

UPDATED WIKI: Chapter 17 – Load-Testing Web Applications

prashantbansode — Thu, 23 Aug 2007 18:48:08 GMT

Chapter 17 – Load-Testing Web Applications

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Understand the key concepts of load testing.
Learn how to load-test a Web application.

Overview

Identify the performance-critical scenarios.
Identify the workload profile for distributing the entire load among the key scenarios.
Identify the metrics that you want to collect in order to verify them against your performance objectives.
Design tests to simulate the load.
Use tools to implement the load according to the designed tests, and capture the metrics.
Analyze the metric data captured during the tests.

How to Use This Chapter

Use this chapter to understand the key concepts of load testing and the steps involved in load-testing Web applications. To get the most from this chapter:

Use the “Input” and “Output” sections to understand the key inputs for load testing a Web application and the key outcomes of doing so.
Use the “Approach for Load Testing” section to get** an overview of the approach for load testing a Web application, and as quick reference guide for you and your team.
Use the various steps sections to understand the details of each step involved in load-testing a Web application.

Input

The following are useful inputs for load-testing a Web application:

Performance-critical usage scenarios
Workload models
Performance acceptance criteria
Performance metrics associated with the acceptance criteria
Interview feedback from the designer or developer of the Web application
Interview feedback from end users of the application
Interview feedback from the operations personnel who will maintain and manage the application

Output

The main outcomes that load testing helps you to accomplish are:

Updated test plans and test designs for load and performance testing
Various performance measures such as throughput, response time, and resource utilization
Potential bottlenecks that need to be analyzed in the white-box testing phase
The behavior of the application at various load levels

Approach for Load Testing

The following steps are involved in load-testing a Web application:

Step 1 - Identify performance acceptance criteria
Step 2 - Identify key scenarios
Step 3 - Create a workload model
Step 4 - Identify the target load levels
Step 5 - Identify metrics
Step 6 - Design specific tests
Step 7 - Run tests
Step 8 - Analyze the results

These steps are graphically represented below. The sections that follow discuss each step in detail.

Cannot resolve link: [image:Fig17.1.GIF]

_Figure 17.1 Load Testing Steps_

Step 1 - Identify Performance Acceptance Criteria

Response time. For example, the product catalog must be displayed in less than 3 seconds.
Throughput. For example, the system must support 100 transactions per second.
Resource utilization. A frequently overlooked aspect is the amount of resources your application is consuming, in terms of processor, memory, disk input output (I/O), and network I/O.
Maximum user load. This test objective determines how many users can run on a specific hardware configuration.
Business related metrics. This objective is mapped to business volume at normal and peak values; for example, the number of orders or Help desk calls handled at a given time.

Step 2 - Identify Key Scenarios

Identify all the scenarios for a Web application. For example, even the most basic e-commerce application must support the following user scenarios:

Browse catalog
Search for a product
Place an order

Identify the activities involved in each of the scenarios. For example, a “Place an Order” scenario will include the following activities:

Log on to the application.
Browse the product catalog.
Search for a specific product.
Add items to the shopping cart.
Validate credit card details and place an order.

Identify the scenarios that are most commonly executed or most resource-intensive; these will be the key scenarios used for load testing. For example, in an e-commerce application, browsing a catalog may be the most commonly executed scenario, whereas placing an order may be the most resource-intensive scenario because it accesses the database.

The most commonly executed scenarios for an existing Web application can be determined by examining the log files.
The most commonly executed scenarios for a new Web application can be obtained from market research, historical data, market trends, and so on.
Resource-intensive scenarios can be identified by using design documents or the actual code implementation. The primary resources are:

Processor
Memory
Disk I/O
Network I/O

Once they have been identified, you will use these key scenarios to create workload profiles and to design load tests.

Step 3 - Create a Workload Model

When defining workload distribution, consider the following key points for determining the characteristics for user scenarios:

A user scenario is defined as a navigational path, including intermediate steps or activities, taken by the user to complete a task. This can also be thought of as a user session.
A user will typically pause between pages during a session. This is known as user delay or think time.
A session will have an average duration when viewed across multiple users. It is important to account for this when defining the load levels that will translate into concurrent usage, overlapping users, or user sessions per unit of time.
Not all scenarios can be performed by a new user, a returning user, or either; know who you expect your primary users to be and test accordingly.

Step 4 - Identify Target Load Levels

Business volume (both current and projected) as it relates to your performance objectives
Key scenarios
Distribution of work
Session characteristics (navigational path, duration, percentage of new users)

By combining the items above, you can determine the remaining details necessary to implement the workload model under a particular target load.

Step 5 - Identify Metrics

Define questions related to your application performance that can be easily tested. For example, what is the checkout response time when placing an order? How many orders are placed in a minute? These questions have definite answers.
With the answers to these questions, determine quality goals for comparison against external expectations. For example, checkout response time should be 30 seconds, and a maximum of 10 orders should be placed in a minute. The answers are based on market research, historical data, market trends, and so on.
Identify the metrics. Using your list of performance-related questions and answers, identify the metrics that provide information related to those questions and answers.
Identify supporting metrics. Using the same approach, you can identify lower-level metrics that focus on measuring the performance and identifying the bottlenecks in the system. When identifying low-level metrics, most teams find it valuable to determine a baseline for those metrics under single-user and/or normal load conditions. This helps you determine the acceptable load levels for your application. Baseline values help you analyze your application performance at varying load levels and serve as a starting point for trend analysis across builds or releases.
Reevaluate the metrics to be collected regularly. Goals, priorities, risks, and current issues are bound to change over the course of a project. With each of these changes, different metrics may provide more value than the ones that have previously been identified.

Additionally, to evaluate the performance of your application in more detail and to identify potential bottlenecks, it is frequently useful to monitor metrics in the following categories:

Network-specific metrics. This set of metrics provides information about the overall health and efficiency of your network, including routers, switches, and gateways.
System-related metrics. This set of metrics helps you identify the resource utilization on your server. The resources being utilized are processor, memory, disk I/O, and network I/O.
Platform-specific metrics. Platform-specific metrics are related to software that is used to host your application, such as the Microsoft .NET Framework common language runtime (CLR) and ASP.NET-related metrics.
Application-specific metrics. These include custom performance counters inserted in your application code to monitor application health and identify performance issues. You might use custom counters to determine the number of concurrent threads waiting to acquire a particular lock, or the number of requests queued to make an outbound call to a Web service.
Service-level metrics. These metrics can help to measure overall application throughput and latency, or they might be tied to specific business scenarios.
Business metrics. These metrics are indicators of business-related information, such as the number of orders placed in a given timeframe.

Step 6 - Design Specific Tests

Do not change your test design because the design is difficult to implement in your tool.
If you cannot implement your test as designed, ensure that you record the details pertaining to the test that you do implement.
Ensure that the model contains all of the supplementary data needed to create the actual test.
Consider including invalid data in your performance tests. For example, include some users who mistype their password on the first attempt but get it correct on a second try.
First-time users usually spend significantly more time on each page or activity than experienced users.
The best possible test data is test data collected from a production database or log file.
Think about nonhuman system users and batch processes as well as end users. For example, there might be a batch process that runs to update the status of orders while users are performing activities on the site. In this situation, you would need to account for those processes because they might be consuming resources.
Do not get overly caught up in striving for perfection, and do not fall into the trap of oversimplification. In general, it is a good idea to start executing tests when you have a reasonable test designed and then enhance the design incrementally while collecting results.

Step 7 - Run Tests

Configure the test environment in such a way that it mirrors your production environment as closely as possible, noting and accounting for all differences between the two.
Ensure that performance counters relevant for identified metrics and resource utilization are being measured and are not interfering with the accuracy of the simulation.
Use appropriate load-generation tools to create a load with the characteristics specified in your test design.
Using the load-generation tool(s), execute tests by first building up to the target load specified in your test design, in order to validate the correctness of the simulation. Some things to consider during test execution include:

Begin load testing with a small number of users distributed against your user profile, and then incrementally increase the load. It is important to allow time for the system to stabilize between increases in load while evaluating the correctness of the simulation.
Consider continuing to increase the load and record the behavior until you reach the threshold for the resources identified in your performance objectives, even if that load is beyond the target load specified in the test design. Information about when the system crosses identified thresholds is just as important as the value of the metrics at the target load of the test.
Similarly, it is frequently valuable to continue to increase the number of users until you run up against the service-level limits beyond which you would be violating your SLAs for throughput, response time, and resource utilization.

Step 8 - Analyze the Results

Analyze the captured data and compare the results against the metric’s accepted level to determine whether the performance of the application being tested shows a trend toward or away from the performance objectives.
Analyze the measured metrics to diagnose potential bottlenecks. Based on the analysis, if required, capture additional metrics in subsequent test cycles. For example, suppose that during the first iteration of load tests, the process shows a marked increase in memory consumption, indicating a possible memory leak. In the subsequent iterations, additional memory counters related to generations can be captured to study the memory allocation pattern for the application.

Summary

UPDATED WIKI: Chapter 16 – Performance Test Reporting Fundamentals

prashantbansode — Thu, 23 Aug 2007 18:40:00 GMT

Chapter 16 – Performance Test Reporting Fundamentals

Objectives

Learn how to apply principles of effective reporting to performance test data.
Learn when to share technical results versus produce stakeholder reports.
Learn what questions various team members expect performance reports to answer.

Overview

Managers and stakeholders need more than simply the results from various tests — they need conclusions based on those results, and consolidated data that supports those conclusions. Technical team members also need more than just results — they need analysis, comparisons, and details of how the results were obtained. Team members of all types get value from performance results being shared more frequently. In this chapter, you will learn how to satisfy the needs of all the consumers of performance test results and data by employing a variety of reporting and results-sharing techniques, and by learning exemplar scenarios where each technique tends be well received.

How to Use This Chapter

Use this chapter to understand the principles of effective performance test results reporting, and as a reference for exemplars of effective data presentation. To get the most from this chapter:

Use the “Principles of Effective Reporting” section to understand the key concepts and principles behind effective reporting.
Use the “Frequently Reported Performance Data” section to learn about various ways that performance data can be presented and the types of results to which those methods are most effectively applied.
Use the “Questions to Be Answered by Reporting” section to understand how reports are designed for various audiences, and how to deliver the right information to the right audience in a format that they find intuitive.

Principles of Effective Reporting

The key to effective reporting is to present information of interest to the intended audience in a quick, simple, and intuitive manner. The following are some of underlying principles of effective reporting:

Report early, report often
Report visually
Report intuitively
Use the right statistics
Consolidate data correctly
Summarize data effectively
Customize reports for the intended audience
Use concise verbal summaries
Make the data available

Report Early, Report Often

Continual sharing of information and data is critical to the efficiency and overall success of a performance-testing project. However, not all of the information and data to be shared needs to take the form of a formal or semiformal report. One effective approach is to send stakeholders summary charts and tables every day or two in an e-mail message that contains a concise statement of key points. Use the feedback and questions you receive from those stakeholders when deciding what to put in the next formal or semiformal report. In this way you can gauge the needs of your audience before writing what is intended to be a stand-alone or final document.

Sharing information and data with the technical team can be an even more straightforward process. It may be as simple as posting the location of the new results files to a team wiki before you begin analyzing them, and then posting links to any charts and graphs that derive from your analysis.

Report Visually

Most people find that data and statistics reported in a graphical format are easier to digest. This is especially true of performance results data, where the volume of data is frequently very large and most significant findings result from detecting patterns in the data. It is possible to find these patterns by scanning through tables or by using complex mathematical algorithms, but the human eye is far quicker and more accurate in the vast majority of cases.

Once a pattern or “point of interest” has been identified visually, you will typically want to isolate that pattern by removing the remaining “chart noise.” In this context, chart noise includes all of the data points representing activities and time slices that contain no points of interest (that is, the ones that look like you expect them to). Removing the chart noise enables you to more clearly evaluate the pattern you are interested in, and makes reports more clear.

Report Intuitively

Whether formal or informal, reports should be able to stand on their own. If a report leaves the reader with questions as to why the information is important, the report has failed. While reports do not need to provide the answers to issues to be effective, the issues should be quickly and intuitively clear from the presentation.

One method to validate the intuitiveness of a report is to remove all labels or identifiers from charts and graphs and all identifying information from narratives and then present the report to someone unfamiliar with the project. If that person is able to quickly and correctly point to the issue of concern in the chart or graph, or identify why the issue discussed in the narrative is relevant, then you have created an intuitive report.

Use the Right Statistics

Even though there is a widespread need to understand many statistical concepts, many software developers, testers, and managers either do not have strong backgrounds in or do not enjoy statistics. This can lead to significant misrepresentations of performance test results when reporting. If you are not sure what statistics to use to highlight a particular issue, do not hesitate to ask for assistance.

Consolidate Data Correctly

While it is not strictly necessary to consolidate results, it tends to be much easier to demonstrate patterns in results when those results are consolidated into one or two graphs rather than distributed over dozens. That said, it is important to remember that only results from identical test executions that are statistically similar can be consolidated into performance report output tables and charts.

Additional Considerations

In order for results to be consolidated, both the test and the test environment must be identical, and the test results must be statistically equivalent. One approach to determining if results are similar enough to be consolidated is to compare results from at least five test executions and apply the following rules:

If more than 20 percent (or one out of five) of the test execution results appear not to be similar to the rest, something is generally wrong with the test environment, the application, or the test itself.
If a 95th percentile value for any test execution is greater than the maximum or less than the minimum value for any of the other test executions, it is not statistically similar.
If every page/timer result in a test execution is noticeably higher or lower on the chart than the results of all the rest of the test executions, it is not statistically similar.
If a single page/timer result in a test execution is noticeably higher or lower on the chart than all the rest of the test execution results, but the results for all the rest of the pages/timers in that test execution are not, the test executions are probably statistically similar.

Summarize Data Effectively

Summarizing results frequently makes it much easier to demonstrate meaningful patterns in the test results. Summary charts and tables present data from different test executions side by side so that trends and patterns are easy to identify. The overall point of these tables and charts is to show team members how the test results compare to the performance goals of the system so they can make important decisions about taking the system live, upgrading the system, or even, in some cases, completely reevaluating the project.

Additional Considerations

Keep the following key points in mind when summarizing test data:

Use charts and tables that make your findings clear.
Use text to supplement tables and charts, not the other way around.
If a chart or table is confusing to the reader, don’t use it.

Customize Reports for the Intended Audience

Performance test results are most commonly read by one of three audiences: technical team members, non-technical team members, and stakeholders outside of the core team. These three groups tend to look for very different things in a performance report and are inclined to prefer different presentation methods. When reporting, make sure that you identify which group or groups you are reporting to and what their expectations are before deciding on the best way to present the results you have collected.

Use Concise Verbal Summaries

Results should have at least a short verbal summary associated with them, and some results are best or most easily presented in writing alone. What you decide to include in that text depends entirely on your intended audience. Some audiences may require just one or two sentences capturing the key point(s) you are trying to make with the graphic. For example:
“From observing this graph, you can see that the system under test meets all stated performance goals up to 150 hourly users but at that point degrades quickly to an essentially unusable state.”

Other audiences may also require a detailed explanation of the graph being presented. For example:

“In this graph, you see the average response time in seconds, portrayed vertically on the left side of the graph, plotted against the total number of hourly users simulated during each test execution, portrayed horizontally along the bottom of the graph. The intersection points depict ”

Make the Data Available

There is a disturbingly popular belief that performance testing (or other testing) data should not be shared in its raw form out of fear that the consumers of that data will use or analyze it improperly. While this concern is not invalid, of much greater concern is the fact that it is simply not reasonable to expect any one person or team to be able to extract all of the value from a set of data at one point in time. Data provides different value to different people at different times, and the only way to get the most out of the data is to make that data continually available to the team. Additionally, making the data available tends to minimize some people’s perception that the performance results are simply fabrications based on a set of tools and processes that they do not understand.

Frequently Reported Performance Data

The following are the most frequently reported types of results data. The sections that follow describe what makes this data interesting to whom, as well as considerations for reporting that type of data.

End-user response times
Resource utilizations
Volumes, capacities, and rates
Component response times
Trends

End-user Response Times

End-user response time is by far the most commonly requested and reported metric in performance testing. If you have captured goals and requirements effectively, this is a measure of presumed user satisfaction with the performance characteristics of the system or application. Stakeholders are interested in end-user response times to judge the degree to which users will be satisfied with the application. Technical team members are interested because they want to know if they are achieving the overall performance goals from a user’s perspective, and if not, in what areas those goals not being met.

Exemplar1

_Figure 16.1 Response Time_

Exemplar2

_Figure 16.2 Response Time Degradation_

Considerations

Even though end-user response times are the most commonly reported performance-testing metric, there are still important points to consider.

Eliminate outliers before reporting. Even one legitimate outlier can dramatically skew your results.
Ensure that the statistics are clearly communicated. The difference between an average and a 90th percentile, for example, can easily be the difference between “ship it” and “fix it.”
Report abandonment separately. If you are accounting for user abandonment, the collected response times for abandoned pages may not represent the same activity as non-abandoned pages. To be safe, report response times for non-abandoned pages with an end-user response time graph and response times and abandonment percentages by page on a separate graph or table.
Report every page or transaction separately. Even though some pages may appear to represent an equivalence class, there could be differences that you are unaware of.

Resource Utilizations

Resource utilizations are the second most requested and reported metrics in performance testing. Most frequently, resource utilization metrics are reported verbally or in a narrative fashion. For example, “The CPU utilization of the application server never exceeded 45 percent. The target is to stay below 70 percent.” It is generally valuable to report resource utilizations graphically when there is an issue to be communicated.

Exemplar for Stakeholders

_Figure 16.3 Processor Time_

Exemplar for Technical Team Members

_Figure 16.4 Processor Time and Queue_

Additional Considerations

Points to consider when reporting resource utilizations include:

Know when to report all of the data and when to summarize. Very often, simply reporting the peak value for a monitored resource during the course of a test is adequate. Unless an issue is detected, the report only needs to demonstrate that the correct metrics were collected to detect the issue if it were present during the test.
Overlay resource utilization metrics with other load and response data. Resource utilization metrics are most powerful when presented on the same graph as load and/or response time data. If there is a performance issue, this helps to identify relationships across various metrics.
If you decide to present more than one data point, present them all. Resource utilization rates will often change dramatically from one measurement to the next. The pattern of change across measurements is at least as important as the current value. Moving averages and trend lines obfuscate these patterns, which can lead to incorrect assumptions and regrettable decisions.

Volumes, Capacities, and Rates

Volume, capacity, and rate metrics are also frequently requested by stakeholders, even though the implications of these metrics are often more challenging to interpret. For this reason, it is important to report these metrics in relation to specific performance criteria or a specific performance issue. Some examples of commonly requested volume, capacity, and rate metrics include:

Bandwidth consumed
Throughput
Transactions per second
Hits per second
Number of supported registered users
Number of records/items able to be stored in the database

Exemplar

_Figure 16.5 Throughput_

Additional Considerations

Points to consider when reporting volumes, capacities and rates include:

Report metrics in context. Volume, capacity, and rate metrics typically have little stand-alone value.
Have test conditions and supporting data available. While this is a good idea in general, it is particularly important with volume, capacity, and rate metrics.
Include narrative summaries with implications. Again, while this is a good idea in general, it is virtually critical to ensure understanding of volume, capacity, and rate metrics.

Component Response Times

Even though component response times are not reported to stakeholders as commonly as end-user response times or resource utilization metrics, they are frequently collected and shared with the technical team. These response times help developers, architects, database administrators (DBAs), and administrators determine what sub-part or parts of the system are responsible for the majority of end-user response times.

Exemplar

_Figure 16.6 Sequential Consecutive Database Updates_

Additional Considerations

Points to consider when reporting component response times include:

Relate component response times to end-user activities. Because it is not always obvious what end-user activities are impacted by a component’s response time, it is a good idea to include those relationships in your report.
Explain the degree to which the component response time matters. Sometimes the concern is that a component might become a bottleneck under load because it is processing too slowly; at other times, the concern is that end-user response times are noticeably degraded as a result of the component. Knowing which of these conditions applies to your project enables you to make effective decisions.

Trends

Trends are one of the most powerful but least-frequently used data-reporting methods. Trends can show whether performance is improving or degrading from build to build, or the rate of degradation as load increases. Trends can help technical team members quickly understand whether the changes they recently made achieved the desired performance impact.

Exemplar

_Figure 16.7 Response Time Trends for Key Pages_

Additional Considerations

Points to consider when reporting trends include:

Trends typically do not add value until there are at least three measurements. Sometimes trends cannot be effectively detected until there are more than three measurements. Start creating your trend charts with the first set of data, but be cautious about including them in formal reports until you have collected enough data for there to be an actual trend to report.
Share trends with the technical team before including them in formal reports. This is another generally good practice, but it is particularly relevant to trends because developers, architects, administrators, and DBAs often will have already backed out a change that caused the trend to move in the wrong direction before they are able to compile their report. In this case, you can decide that the trend report is not worth including, or you can simply make an annotation describing the cause and stating that the issue has already been resolved.

Questions to Be Answered By Reporting

Almost every team member has unique wants, needs, and expectations when it comes to reporting data and results obtained through performance testing. While this makes sharing information obtained through performance testing challenging, knowing what various team members expect and value in advance makes providing valuable information to the right people, at the right level of detail and at the right time, much easier

All Roles

Some questions that are commonly posed by team members include:

Is performance getting better or worse?
Have we met the requirements/service level agreements (SLAs)?
What reports are available?
How frequently can I get reports?
Can I get a report with more/less detail?

Executive Stakeholders

Executive stakeholders tend to have very specific reporting needs and expectations that are often quite different from those of other team members. Stakeholders tend to prefer information in small, digestible chunks that clearly highlight the key points. Additionally, stakeholders like visual representations of data that are intuitive at a glance, as well as “sound bite”–size explanations of those visual representations. Finally, stakeholders tend to prefer consolidated and summarized information on a less frequent (though not significantly less frequent) basis than other team members. The following are common questions that executive stakeholders want performance test reports to answer:

Is this ready to ship?
How will these results relate to production?
How much confidence should I have in these results?
What needs to be done to get this ready to ship?
Is the performance testing proceeding as anticipated?
Is the performance testing adding value?

Project-Level Managers

Project-level managers — including the project manager, development lead or manager, and the test lead or manager — have all of the same needs and questions as the executive stakeholders, except that they want the answers more frequently and in more detail. Additionally, they commonly want to know the following:

Are performance issues being detected efficiently?
Are performance issues being resolved efficiently?
What performance testing should we be conducting that we currently are not?
What performance testing are we currently doing that is not adding value?
_Are there currently any blockers? If so, what are they? _

Technical Team Members

Although technical team members have some degree of interest in all of the questions posed by managers and stakeholders, they are more interested in receiving a continual flow of information related to test results, monitored data, observations, and opportunities for analysis and improvement. Technical team members tend to want to know the following:

What do these results mean to my specialty/focus area?
Where can I go to see the results for the last test?
Where can I go to get the raw data?
Can you capture metric X during the next test run?

Types of Results Sharing

In the most basic sense, there are three distinct types of results sharing: raw data display, technical reports, and stakeholder reports. While all are based on timely, accurate, and relevant communication of results, observations, concerns, and recommendations, each type targets a different audience, and the most effective methods of communicating data differ dramatically.

Raw Data Display

While not explicitly a reporting scenario, the sharing of raw data for collaboration purposes involves many of the same principles of data presentation that are applied to reports in order to improve the effectiveness of the collaboration.

In general, most people would rather view data and statistics in graphical form instead of in tables. In some cases, however, tables are the most efficient way to show calculated results for all of the data. It is recommended that you use tables sparingly in reports, while including the tabular form of the data used to create charts and graphs as an appendix or attachment to a report, so that interested stakeholders can refer to it.

Results from the following types of tests can be well represented in a tabular format:

Baseline
Benchmark
Scalability
Any other user-experience–based test

Tables are an excellent way to present volumes of data in a clean and orderly manner and to support the findings they ultimately lead to. However, you should be careful not to overuse tables in your reports. Many people quickly skip over tables and read only the surrounding text or examine only the charts that go with them. Be certain that whether you use the tables discussed below or other types, you present in your report only those tables that clearly make an important point. Huge tables containing all of the supporting data may be of interest to a few individuals, but not to most, and thus should be included only in an appendix to a report. Raw data is most commonly shared in the following formats:

Spreadsheets
Text files (and regular expression searches)
Data collection tools (“canned” reports)

Technical Reports

Technical reports are generally more formal than raw data display, but not excessively so. Technical reports should stand on their own, but since they are intended for technical members of the team who are currently working on the project, they do not need to contain all of the supplemental detail that a stakeholder report normally does. In the simplest sense, technical reports are made up of the following:

Description of the test, including workload model and test environment
Easily digestible data with minimal pre-processing
Access to the complete data set and test conditions
Short statements of observations, concerns, questions, and requests for collaboration

Technical reports most commonly include data in the following formats:

Scatter plots
Pareto charts
Trend charts
Summary spreadsheets

Stakeholder Reports

Stakeholder reports are the most formal of the performance data sharing formats. These reports must be able to stand alone while at the same time being intuitive to someone who is not working on the project in a day-to-day technical role. Typically, these reports center on acceptance criteria and risks. To be effective, stakeholder reports typically need to include:

The acceptance criteria to which the results relate
Intuitive, visual representations of the most relevant data
A brief verbal summary of the chart or graph in terms of criteria
Intuitive, visual representations of the workload model and test environment
Access to associated technical reports, complete data sets, and test conditions
A summary of observations, concerns, and recommendations

When preparing stakeholder reports, consider that most stakeholder reports meet with one (or more) of the following three reactions. All three are positive in their own way but may not seem to be at first. These reactions and some recommended responses follow:

“These are great, but where’s the supporting data?” This is the most common response from a technical stakeholder. Many people and organizations want to have all of the data so that they can draw their own conclusions. Fortunately, this is an easy question to handle: simply include the entire spreadsheet with this supporting data as an appendix to the report.
“Very pretty, but what do they mean?” This is where text explanations are useful. People who are not familiar with performance testing or performance results often need to have the implications of the results spelled out for them. Remember that more than 90 percent of the times, performance testers are the bearers of bad news that the stakeholder was not expecting. The tester has the responsibility to ensure that the stakeholder has confidence in the findings, as well as presenting this news in a constructive manner.
“Terrific! This is exactly what I wanted! Don’t worry about the final report — these will do nicely.” While this seems like a blessing, do not take it as one. Sooner or later, your tables and charts will be presented to someone who asks one of the two preceding questions, or worse, asks how the data was obtained. If there is not at least a final report that tells people where to find the rest of the data, people will question the results because you are not present to answer those questions.

Creating a Technical Report

Although six key components of a technical report are listed below, all six may not be appropriate for every technical report. Similarly, there may be additional information that should be included based on exactly what message you are trying to convey with the report. While these six components will result in successful technical reports most of the time, remember that sometimes creativity is needed to make your message clear and intuitive.

Consider including the following key components when preparing a technical report:

A results graph
A table for single-instance measurements (e.g., maximum throughput achieved)
Workload model (graphic)
Test environment (annotated graphic)
Short statements of observations, concerns, questions, and requests for collaboration
References section

Exemplar Results Graph

_Figure 16.8 Consolidated Statistics_

Exemplar Tables for Single-Instance Measurements

_Figure 16.9 Single Instance Measurements_

Exemplar Workload Model Graphic

_Figure 16.10 Workload Model_

Exemplar Test Environment Graphic

_Figure 16.11 Test Environment_

Exemplar Summary Statement

“The results graph shows both response times and resource utilization together. Close examination shows that Application Server CPU Usage and queue length coincide with significantly degraded response time. It appears as if the application server CPU usage was the catalyst to the degradation, but this has yet to be confirmed. The remaining charts and graphs are included as supplemental information for easy reference.”

Exemplar References Section

“Raw data and additional supporting information is checked into the version-control system with the build and tagged as PerfTest-{date}-{issue number}.”

Creating a Stakeholder Report

Although eight key components of a stakeholder report are listed below, all eight may not be appropriate for every stakeholder report. Similarly, there may be additional information that should be included based on exactly what message you are trying to convey with the report. While these eight components will result in successful stakeholder reports most of the time, remember that sometimes creativity is needed to make your message clear and intuitive.

Consider including the following key components when preparing a stakeholder report:

Criteria to which the results relate
A results graph
A table for single-instance measurements (e.g., maximum throughput achieved)
A brief verbal summary of the chart or graph in terms of criteria
Workload model (graphic)
Test environment (annotated graphic)
Summary of observations, concerns, and recommendations
References section

Exemplar Criteria Statement

“This report relates to end-user response time compliances as documented in the requirements management system as requirements Perf### through Perf??? at one-half of the expected peak load with the most commonly expected usage scenario.”

Exemplar Results Graph

_Figure 16.12 Response Time Compliance Summary_

Exemplar Tables for Single-Instance Measurements

_Figure 16.13 Single Instance Measurements_

Exemplar Criteria-Based Results Summary

“All metrics collected achieved their required values except for the response times of pages 8 and 10.

Page 10 failed to achieve its required value by 2 percent.
Page 8 failed to achieve its required value by 38 percent.”

Exemplar Workload Model Graphic

_Figure 16.14 Workload Model_

Exemplar Test Environment Graphic

_Figure 16.15 Test Environment_

Exemplar Observations and Recommendations Statement

“Based on the test conditions and results, the performance testing and tuning team recommends the following.

Continue performance testing with increasingly strenuous scenarios and loads.
Priority should be given to determining the root cause of pages 8 and 10 not achieving their acceptance criteria, and subsequently tuning those root causes.
At such time as additional pages demonstrate a failure to achieve their acceptance criteria, a dedicated root cause and tuning cycle should be undertaken.”

Exemplar References Section

“All of the data used to create this report and execute the tests that generated that data is checked into the version-control system as read-only with the release candidate and tagged as PerfTest-{date}-{RC number}-Validation.

“The same data has been temporarily copied to {\\shared-resource\location} for individuals without access to the version-control system.”

Summary

Performance test reporting is the process of presenting results data that will support key technological and business decisions. The key to creating effective reports is to consider the audience of the data before determining how best to present the data. The most effective performance-test results will present analysis, comparisons, and details behind how the results were obtained, and will influence critical business decision-making.

UPDATED WIKI: Chapter 16 – Performance Test Reporting Fundamentals

prashantbansode — Thu, 23 Aug 2007 18:12:24 GMT

Chapter 16 – Performance Test Reporting Fundamentals

Objectives

Learn how to apply principles of effective reporting to performance test data.
Learn when to share technical results versus produce stakeholder reports.
Learn what questions various team members expect performance reports to answer.

Overview

How to Use This Chapter

Use this chapter to understand the principles of effective performance test results reporting, and as a reference for exemplars of effective data presentation. To get the most from this chapter:

Use the “Principles of Effective Reporting” section to understand the key concepts and principles behind effective reporting.
Use the “Frequently Reported Performance Data” section to learn about various ways that performance data can be presented and the types of results to which those methods are most effectively applied.
Use the “Questions to Be Answered by Reporting” section to understand how reports are designed for various audiences, and how to deliver the right information to the right audience in a format that they find intuitive.

Principles of Effective Reporting

Report early, report often
Report visually
Report intuitively
Use the right statistics
Consolidate data correctly
Summarize data effectively
Customize reports for the intended audience
Use concise verbal summaries
Make the data available

Report Early, Report Often

Report Visually

Report Intuitively

Use the Right Statistics

Consolidate Data Correctly

Additional Considerations

If more than 20 percent (or one out of five) of the test execution results appear not to be similar to the rest, something is generally wrong with the test environment, the application, or the test itself.
If a 95th percentile value for any test execution is greater than the maximum or less than the minimum value for any of the other test executions, it is not statistically similar.
If every page/timer result in a test execution is noticeably higher or lower on the chart than the results of all the rest of the test executions, it is not statistically similar.
If a single page/timer result in a test execution is noticeably higher or lower on the chart than all the rest of the test execution results, but the results for all the rest of the pages/timers in that test execution are not, the test executions are probably statistically similar.

Summarize Data Effectively

Additional Considerations

Keep the following key points in mind when summarizing test data:

Use charts and tables that make your findings clear.
Use text to supplement tables and charts, not the other way around.
If a chart or table is confusing to the reader, don’t use it.

Customize Reports for the Intended Audience

Use Concise Verbal Summaries

Make the Data Available

Frequently Reported Performance Data

End-user response times
Resource utilizations
Volumes, capacities, and rates
Component response times
Trends

End-user Response Times

Exemplar1

Cannot resolve link: [image:Fig16.1.GIF]

_Figure 16.1 Response Time_

Exemplar2

Cannot resolve link: [image:Fig16.2.GIF]

_Figure 16.2 Response Time Degradation_

Considerations

Even though end-user response times are the most commonly reported performance-testing metric, there are still important points to consider.

Eliminate outliers before reporting. Even one legitimate outlier can dramatically skew your results.
Ensure that the statistics are clearly communicated. The difference between an average and a 90th percentile, for example, can easily be the difference between “ship it” and “fix it.”
Report abandonment separately. If you are accounting for user abandonment, the collected response times for abandoned pages may not represent the same activity as non-abandoned pages. To be safe, report response times for non-abandoned pages with an end-user response time graph and response times and abandonment percentages by page on a separate graph or table.
Report every page or transaction separately. Even though some pages may appear to represent an equivalence class, there could be differences that you are unaware of.

Resource Utilizations

Exemplar for Stakeholders

Cannot resolve link: [image:Fig16.3.GIF]

_Figure 16.3 Processor Time_

Exemplar for Technical Team Members

Cannot resolve link: [image:Fig16.4.GIF]

_Figure 16.4 Processor Time and Queue_

Additional Considerations

Points to consider when reporting resource utilizations include:

Know when to report all of the data and when to summarize. Very often, simply reporting the peak value for a monitored resource during the course of a test is adequate. Unless an issue is detected, the report only needs to demonstrate that the correct metrics were collected to detect the issue if it were present during the test.
Overlay resource utilization metrics with other load and response data. Resource utilization metrics are most powerful when presented on the same graph as load and/or response time data. If there is a performance issue, this helps to identify relationships across various metrics.
If you decide to present more than one data point, present them all. Resource utilization rates will often change dramatically from one measurement to the next. The pattern of change across measurements is at least as important as the current value. Moving averages and trend lines obfuscate these patterns, which can lead to incorrect assumptions and regrettable decisions.

Volumes, Capacities, and Rates

Bandwidth consumed
Throughput
Transactions per second
Hits per second
Number of supported registered users
Number of records/items able to be stored in the database

Exemplar

Cannot resolve link: [image:Fig16.5.GIF]

_Figure 16.5 Throughput_

Additional Considerations

Points to consider when reporting volumes, capacities and rates include:

Report metrics in context. Volume, capacity, and rate metrics typically have little stand-alone value.
Have test conditions and supporting data available. While this is a good idea in general, it is particularly important with volume, capacity, and rate metrics.
Include narrative summaries with implications. Again, while this is a good idea in general, it is virtually critical to ensure understanding of volume, capacity, and rate metrics.

Component Response Times

Exemplar

Cannot resolve link: [image:Fig16.6.GIF]

_Figure 16.6 Sequential Consecutive Database Updates_

Additional Considerations

Points to consider when reporting component response times include:

Relate component response times to end-user activities. Because it is not always obvious what end-user activities are impacted by a component’s response time, it is a good idea to include those relationships in your report.
Explain the degree to which the component response time matters. Sometimes the concern is that a component might become a bottleneck under load because it is processing too slowly; at other times, the concern is that end-user response times are noticeably degraded as a result of the component. Knowing which of these conditions applies to your project enables you to make effective decisions.

Trends

Exemplar

Cannot resolve link: [image:Fig16.7.GIF]

_Figure 16.7 Response Time Trends for Key Pages_

Additional Considerations

Points to consider when reporting trends include:

Trends typically do not add value until there are at least three measurements. Sometimes trends cannot be effectively detected until there are more than three measurements. Start creating your trend charts with the first set of data, but be cautious about including them in formal reports until you have collected enough data for there to be an actual trend to report.
Share trends with the technical team before including them in formal reports. This is another generally good practice, but it is particularly relevant to trends because developers, architects, administrators, and DBAs often will have already backed out a change that caused the trend to move in the wrong direction before they are able to compile their report. In this case, you can decide that the trend report is not worth including, or you can simply make an annotation describing the cause and stating that the issue has already been resolved.

Questions to Be Answered By Reporting

All Roles

Some questions that are commonly posed by team members include:

Is performance getting better or worse?
Have we met the requirements/service level agreements (SLAs)?
What reports are available?
How frequently can I get reports?
Can I get a report with more/less detail?

Executive Stakeholders

Is this ready to ship?
How will these results relate to production?
How much confidence should I have in these results?
What needs to be done to get this ready to ship?
Is the performance testing proceeding as anticipated?
Is the performance testing adding value?

Project-Level Managers

Are performance issues being detected efficiently?
Are performance issues being resolved efficiently?
What performance testing should we be conducting that we currently are not?
What performance testing are we currently doing that is not adding value?
_Are there currently any blockers? If so, what are they? _

Technical Team Members

What do these results mean to my specialty/focus area?
Where can I go to see the results for the last test?
Where can I go to get the raw data?
Can you capture metric X during the next test run?

Types of Results Sharing

Raw Data Display

Baseline
Benchmark
Scalability
Any other user-experience–based test

Spreadsheets
Text files (and regular expression searches)
Data collection tools (“canned” reports)

Technical Reports

Description of the test, including workload model and test environment
Easily digestible data with minimal pre-processing
Access to the complete data set and test conditions
Short statements of observations, concerns, questions, and requests for collaboration

Technical reports most commonly include data in the following formats:

Scatter plots
Pareto charts
Trend charts
Summary spreadsheets

Stakeholder Reports

The acceptance criteria to which the results relate
Intuitive, visual representations of the most relevant data
A brief verbal summary of the chart or graph in terms of criteria
Intuitive, visual representations of the workload model and test environment
Access to associated technical reports, complete data sets, and test conditions
A summary of observations, concerns, and recommendations

“These are great, but where’s the supporting data?” This is the most common response from a technical stakeholder. Many people and organizations want to have all of the data so that they can draw their own conclusions. Fortunately, this is an easy question to handle: simply include the entire spreadsheet with this supporting data as an appendix to the report.
“Very pretty, but what do they mean?” This is where text explanations are useful. People who are not familiar with performance testing or performance results often need to have the implications of the results spelled out for them. Remember that more than 90 percent of the times, performance testers are the bearers of bad news that the stakeholder was not expecting. The tester has the responsibility to ensure that the stakeholder has confidence in the findings, as well as presenting this news in a constructive manner.
“Terrific! This is exactly what I wanted! Don’t worry about the final report — these will do nicely.” While this seems like a blessing, do not take it as one. Sooner or later, your tables and charts will be presented to someone who asks one of the two preceding questions, or worse, asks how the data was obtained. If there is not at least a final report that tells people where to find the rest of the data, people will question the results because you are not present to answer those questions.

Creating a Technical Report

A results graph
A table for single-instance measurements (e.g., maximum throughput achieved)
Workload model (graphic)
Test environment (annotated graphic)
Short statements of observations, concerns, questions, and requests for collaboration
References section

Exemplar Results Graph

Cannot resolve link: [image:Fig16.8.GIF]

_Figure 16.8 Consolidated Statistics_

Exemplar Tables for Single-Instance Measurements

Cannot resolve link: [image:Fig16.9.GIF]

_Figure 16.9 Single Instance Measurements_

Exemplar Workload Model Graphic

Cannot resolve link: [image:Fig16.10.GIF]

_Figure 16.10 Workload Model_

Exemplar Test Environment Graphic

Cannot resolve link: [image:Fig16.11.GIF]

_Figure 16.11 Test Environment_

Exemplar Summary Statement

Exemplar References Section

“Raw data and additional supporting information is checked into the version-control system with the build and tagged as PerfTest-{date}-{issue number}.”

Creating a Stakeholder Report

Criteria to which the results relate
A results graph
A table for single-instance measurements (e.g., maximum throughput achieved)
A brief verbal summary of the chart or graph in terms of criteria
Workload model (graphic)
Test environment (annotated graphic)
Summary of observations, concerns, and recommendations
References section

Exemplar Criteria Statement

Exemplar Results Graph

Cannot resolve link: [image:Fig16.12.GIF]

_Figure 16.12 Response Time Compliance Summary_

Exemplar Tables for Single-Instance Measurements

Cannot resolve link: [image:Fig16.3.GIF]

_Figure 16.13 Single Instance Measurements_

Exemplar Criteria-Based Results Summary

“All metrics collected achieved their required values except for the response times of pages 8 and 10.

Page 10 failed to achieve its required value by 2 percent.
Page 8 failed to achieve its required value by 38 percent.”

Exemplar Workload Model Graphic

Cannot resolve link: [image:Fig16.14.GIF]

_Figure 16.14 Workload Model_

Exemplar Test Environment Graphic

Cannot resolve link: [image:Fig16.15.GIF]

_Figure 16.15 Test Environment_

Exemplar Observations and Recommendations Statement

“Based on the test conditions and results, the performance testing and tuning team recommends the following.

Continue performance testing with increasingly strenuous scenarios and loads.
Priority should be given to determining the root cause of pages 8 and 10 not achieving their acceptance criteria, and subsequently tuning those root causes.
At such time as additional pages demonstrate a failure to achieve their acceptance criteria, a dedicated root cause and tuning cycle should be undertaken.”

Exemplar References Section

Summary

UPDATED WIKI: Introduction

prashantbansode — Thu, 23 Aug 2007 17:52:26 GMT

Introduction

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Performance Testing Guidance for Web Applications provides an end-to-end approach for implementing performance testing. Whether you are new to performance testing or looking for ways to improve your current performance-testing approach, you will gain insights that you can tailor to your specific scenarios.

The information in this guide is based on applied use in customer scenarios. It reflects the lessons learned from multiple performance-testing professionals. The guidance is task-based and presented in the following parts:

Part 1, “Introduction to Performance Testing,” gives you an overview of common types of performance testing, key concepts, and a set of common terms used in performance testing.
Part II, “Exemplar Performance Testing Approaches,” shows you seven core activities for performance testing. This section also contains information designed to show you how to apply performance testing to different environments, including Agile and CMMI® software development.
Part III, “Identify the Test Environment,” shows you how to collect information about your project that you will need for your performance tests. This includes collecting information on system architecture, the physical deployment, user activities, and any relevant batch processes.
Part IV, “Identify Performance Acceptance Criteria,” shows you how to determine your performance testing objectives. You will also learn how to achieve clarity around your various performance goals and requirements, from a performance testing perspective.
Part V, “Plan and Design Tests,” shows you how to model the workload and user experience to design more effective performance tests.
Part VI, “Execute Tests,” walks you through the main activities of actual performance testing.
Part VII, “Analyze Results and Report,” shows you how to organize and present your findings in a way that is useful based on the audience and the intent of the report.
Part VIII, “Performance-Testing Techniques,” shows you the core techniques for performing load and stress testing.

Scope of This Guide

This guide is focused on Web application performance testing. It provides recommendations on the following:

Managing and conducting performance testing in both dynamic (e.g., Agile) and structured (e.g., CMMI) environments.
Performance testing, including load testing, stress testing, and other types of performance related testing.
Core activities of performance testing: identifying objectives, designing tests, executing tests, analyzing results, and reporting.

Even though many of the topics addressed in this guide apply equally well to other types of applications, the topics are all explained from a Web application perspective for consistency and to ensure that the concepts are presented in a manner that is most intuitive to the majority of anticipated readers.

This guide is intended to be tool-agnostic. What that means is that none of the concepts presented in this guide require any specific tool to accomplish, though some techniques or concepts will require the use of a certain class of tools.

This guide does not directly address performance tuning. Performance tuning is extremely application- and technology-specific and thus is not a good fit for the style and format of the guide. The guide does, however, address high-level approaches around how performance testing and performance tuning activities overlap and feed one another.

Why We Wrote This Guide

We wrote this guide to accomplish the following:

To consolidate real-world lessons learned around performance testing.
To present a roadmap for end-to-end performance testing.
To narrow the gap between state of the art and state of the practice.

Features of This Guide

Approach for performance testing. The guide provides an approach that organizes performance testing into logical units to help you incrementally adopt performance testing throughout your application life cycle.
Principles and practices. These serve as the foundation for the guide and provide a stable basis for recommendations. They also reflect successful approaches used in the field.
Processes and methodologies. These provide steps for managing and conducting performance testing. For simplification and tangible results, they are broken down into activities with inputs, outputs, and steps. You can use the steps as a baseline or to help you evolve your own process.
Life cycle approach. The guide provides end-to-end guidance on managing performance testing throughout your application life cycle, to reduce risk and lower total cost of ownership (TCO).
Modular. Each chapter within the guide is designed to be read independently. You do not need to read the guide from beginning to end to benefit from it. Use the parts you need.
Holistic. The guide is designed with the end in mind. If you do read the guide from beginning to end, it is organized to fit together in a comprehensive way. The guide, in its entirety, is better than the sum of its parts.
Subject matter expertise. The guide exposes insight from various experts throughout Microsoft and from customers in the field.

Who Should Read This Guide

This guide is targeted at providing individuals with the resources, patterns, and practices they need to conduct effective performance testing.

How to Use This Guide

You can read this guide from beginning to end, or you can read only the relevant parts or chapters. You can adopt the guide in its entirety for your organization or you can use critical components to address your highest-priority needs.

Ways to Use the Guide

There are many ways to use this comprehensive guidance. The following are some suggestions:

Use it as a mentor. Use the guide as your mentor for learning how to conduct performance testing. The guide encapsulates the lessons learned and experiences gained by many subject matter experts.
Use it as a reference. Use the guide as a reference for learning the do’s and don’ts of performance testing.
Incorporate performance testing into your application development life cycle. Adopt the approach and practices that work for you and incorporate them into your application life cycle.
Use it when you design your performance tests. Design applications using the principles and best practices presented in this guide. Benefit from lessons learned.
Create training. Create training based on the concepts and techniques used throughout the guide.

Organization of This Guide

You can read this guide from end to end, or you can read only the chapters you need to do your job.

Parts

The guide is divided into eight parts:

Part 1, Introduction to Performance Testing
Part II, Exemplar Performance Testing Approaches
Part III, Identify the Test Environment
Part IV, Identify Performance Acceptance Criteria
Part V, Plan and Design Tests
Part VI, Execute Tests
Part VII, Analyze Results and Report
Part VIII, Performance-Testing Techniques

Part 1, Introduction to Performance Testing

Part II, Exemplar Performance Testing Approaches

Part III, Identify the Test Environment

Chapter 8 – Evaluating Systems to Increase Performance-Testing Effectiveness

Part IV, Identify Performance Acceptance Criteria

Part V, Plan and Design Tests

Part VI, Execute Tests

Chapter 14 – Test Execution

Part VII, Analyze Results and Report

Part VIII, Performance-Testing Techniques

Approach Used in This Guide

The primary task of any testing activity is to collect information in order to be able to help stakeholders make informed decisions related to the overall quality of the application being tested. Performance testing additionally tends to focus on helping to identify bottlenecks in a system, tuning a system, establishing a baseline for future testing, and determining compliance with performance goals and requirements. In addition, the results from performance testing and analysis can help you to estimate the hardware configuration required to support the application(s) when you “go live” to production operation.

The performance-testing approach used in this guide consists of the following activities:

Activity 1. Identify the Test Environment. Identify the physical test environment and the production environment as well as the tools and resources available to the test team. The physical environment includes hardware, software, and network configurations. Having a thorough understanding of the entire test environment at the outset enables more efficient test design and planning and helps you identify testing challenges early in the project. In some situations, this process must be revisited periodically throughout the project’s life cycle.
Activity 2. Identify Performance Acceptance Criteria. Identify the response time, throughput, and resource utilization goals and constraints. In general, response time is a user concern, throughput is a business concern, and resource utilization is a system concern. Additionally, identify project success criteria that may not be captured by those goals and constraints; for example, using performance tests to evaluate what combination of configuration settings will result in the most desirable performance characteristics.
Activity 3. Plan and Design Tests. Identify key scenarios, determine variability among representative users and how to simulate that variability, define test data, and establish metrics to be collected. Consolidate this information into one or more models of system usage to be implemented, executed, and analyzed.
Activity 4. Configure the Test Environment. Prepare the test environment, tools, and resources necessary to execute each strategy as features and components become available for test. Ensure that the test environment is instrumented for resource monitoring as necessary.
Activity 5. Implement the Test Design. Develop the performance tests in accordance with the test design.
Activity 6. Execute the Test. Run and monitor your tests. Validate the tests, test data, and results collection. Execute validated tests for analysis while monitoring the test and the test environment.
Activity 7. Analyze Results, Report, and Retest. Consolidate and share results data. Analyze the data both individually and as a cross-functional team. Reprioritize the remaining tests and re-execute them as needed. When all of the metric values are within accepted limits, none of the set thresholds have been violated, and all of the desired information has been collected, you have finished testing that particular scenario on that particular configuration.

Feedback on the Guide

We have made every effort to ensure the accuracy of this guide and its companion content. If you have comments on this guide, send e-mail to
mailto:PerfTest@microsoft.com .

We are particularly interested in feedback regarding the following:

Technical issues specific to recommendations
Usefulness and usability issues

The Team Who Brought You This Guide

This guide was created by the following team members:

J.D. Meier
Carlos Farre
Prashant Bansode
Scott Barber
Dennis Rea

Contributors and Reviewers

Alan Ridlehoover; Clint Huffman; Edmund Wong; Ken Perilman; Larry Brader; Mark Tomlinson; Paul Williams; Pete Coupland; Rico Mariani

External Contributors and Reviewers

Tell Us About Your Success

If this guide helps you, we would like to know. Please tell us by writing a short summary of the problems you faced and how this guide helped you out. Submit your summary to:
mailto:MyStory@Microsoft.com.

UPDATED WIKI: Chapter 15 – Key Mathematic Principles for Performance Testers

prashantbansode — Thu, 23 Aug 2007 17:51:28 GMT

Chapter 15 – Key Mathematic Principles for Performance Testers

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Learn the uses, meanings of, and concepts underlying common mathematical and statistical principles as they apply to performance test analysis and reporting.

Overview

Members of software development teams, developers, testers, administrators, and managers alike need to know how to apply mathematics and interpret statistical data in order to do their jobs effectively. Performance analysis and reporting are particularly math-intensive. This chapter describes the most commonly used, misapplied, and misunderstood mathematical and statistical concepts in performance testing, in a way that will benefit any member of the team.

Even though there is a need to understand many mathematical and statistical concepts, many software developers, testers, and managers either do not have strong backgrounds in or do not enjoy mathematics and statistics. This leads to significant misrepresentations and misinterpretation of performance-testing results. The information presented in this article is not intended to replace formal training in these areas, but rather to provide common language and commonsense explanations for mathematical and statistical operations that are valuable to understanding performance testing.

How to Use This Chapter

Use this chapter to understand the different metrics and calculations that are used for analyzing performance data results and preparing performance results reports. To get the most from this chapter:

Use the “Exemplar Data Sets” section to gain an understanding of the exemplars, which are used to illustrate the key mathematical principles explained throughout the chapter.
Use the remaining sections to learn about key mathematical principles that will help you to understand and present meaningful performance testing reports.

Exemplar Data Sets

This chapter refers to three exemplar data sets for the purposes of illustration, namely.

Data Set A
Data Set B
Data Set C

Data Sets Summary

The following is a summary of Data Sets A, B, and C.

_Figure 15.1 Summary of Data Sets A, B, and C_

Data Set A

_Figure 15.2 Data Set A_

100 total data points, distributed as follows:

5 data points have a value of 1.
10 data points have a value of 2.
20 data points have a value of 3.
30 data points have a value of 4.
20 data points have a value of 5.
10 data points have a value of 6.
5 data points have a value of 7.

Data Set B

_Figure 15.3 Data Set B_

100 total data points, distributed as follows:

80 data points have a value of 1.
20 data points have a value of 16.

Data Set C

_Figure 15.4 Data Set C_

100 total data points, distributed as follows:

11 data points have a value of 0.
10 data points have a value of 1.
11 data points have a value of 2.
13 data points have a value of 3.
11 data points have a value of 4.
11 data points have a value of 5.
11 data points have a value of 6.
12 data points have a value of 7.
10 data points have a value of 8.

Averages

An average ? also known as an arithmetic mean, or mean for short ? is probably the most commonly used, and most commonly misunderstood, statistic of all. To calculate an average, you simply add up all the numbers and divide the sum by the quantity of numbers you just added. What seems to confound many people the most when it comes to performance testing is that, in this example, Data Sets A, B, and C each have an average of exactly 4. In terms of application response times, these sets of data have extremely different meanings. Given a response time goal of 5 seconds, looking at only the average of these sets, all three seem to meet the goal. Looking at the data, however, shows that none of the data sets is composed only of data that meets the goal, and that Data Set B probably demonstrates some kind of performance anomaly. Use caution when using averages to discuss response times and, if at all possible, avoid using averages as the only reported statistic. When reporting averages, it is a good idea to include the sample size, minimum value, maximum value, and standard deviation for the data set.

Percentiles

Few people involved with developing software are familiar with percentiles. A percentile is a straightforward concept that is easier to demonstrate than define. For example, to find the 95th percentile value for a data set consisting of 100 page-response-time measurements, you would sort the measurements from largest to smallest and then count down six data points from the largest. The 6th data point value represents the 95th percentile of those measurements. For the purposes of response times, this statistic is read “95 percent of the simulated users experienced a response time of the 6th-slowest value or less for this test scenario.”

It is important to note that percentile statistics can only stand alone when used to represent data that is uniformly or normally distributed with an acceptable number of outliers (see “Statistical Outliers” below). To illustrate this point, consider the exemplar data sets. The 95th percentile of Data Set B is 16 seconds. Obviously, this does not give the impression of achieving the 5-second response time goal. Interestingly, this can be misleading as well because the 80th percentile value of Data Set B is 1 second. With a response time goal of 5 seconds, it is likely unacceptable to have any response times of 16 seconds, so in this case neither of these percentile values represent the data in a manner that is useful to summarizing response time.

Data Set A is a normally distributed data set that has a 95th percentile value of 6 seconds, an 85th percentile value of 5 seconds, and a maximum value of 7 seconds. In this case, reporting either the 85th or 95th percentile values represents the data in a manner where the assumptions a stakeholder is likely to make about the data are likely to be appropriate to the data.

Medians

A median is simply the middle value in a data set when sequenced from lowest to highest. In cases where there is an even number of data points and the two center values are not the same, some disciplines suggest that the median is the average of the two center data points, while others suggest choosing the value closer to the average of the entire set of data. In the case of the exemplar data sets, Data Sets A and B have median values of 4, and Data Set C has a median value of 1.

Normal Values

A normal value is the single value that occurs most often in a data set. Data Set A has a normal value of 4, Data Set B has a normal value of 3, and Data Set C has a normal value of 1.

Standard Deviations

By definition, one standard deviation is the amount of variance within a set of measurements that encompasses approximately the top 68 percent of all measurements in the data set; in other words, knowing the standard deviation of your data set tells you how densely the data points are clustered around the mean. Simply put, the smaller the standard deviation, the more consistent the data. To illustrate, the standard deviation of Data Set A is approximately 1.5, the standard deviation of Data Set B is approximately 6.0, and the standard deviation of Data Set C is approximately 2.6.

A common rule in this case is: “Data with a standard deviation greater than half of its mean should be treated as suspect. If the data is accurate, the phenomenon the data represents is not displaying a normal distribution pattern.” Applying this rule, Data Set A is likely to be a reasonable example of a normal distribution; Data Set B may or may not be a reasonable representation of a normal distribution; and Data Set C is undoubtedly not a reasonable representation of a normal distribution.

Uniform Distributions

Uniform distributions ? sometimes known as linear distributions ? represent a collection of data that is roughly equivalent to a set of random numbers evenly spaced between the upper and lower bounds. In a uniform distribution, every number in the data set is represented approximately the same number of times. Uniform distributions are frequently used when modeling user delays, but are not common in response time results data. In fact, uniformly distributed results in response time data may be an indication of suspect results.

_Figure 15.5 Uniform Distributions_

Normal Distributions

Also known as bell curves, normal distributions are data sets whose member data are weighted toward the center (or median value). When graphed, the shape of the “bell” of normally distributed data can vary from tall and narrow to short and squat, depending on the standard deviation of the data set. The smaller the standard deviation, the taller and more narrow the “bell.” Statistically speaking, most measurements of human variance result in data sets that are normally distributed. As it turns out, end-user response times for Web applications are also frequently normally distributed.

_Figure 15.6 Normal Distribution_

Statistical Significance

Mathematically calculating statistical significance, or reliability, based on sample size is a task that is too arduous and complex for most commercially driven software-development projects. Fortunately, there is a commonsense approach that is both efficient and accurate enough to identify the most significant concerns related to statistical significance. Unless you have a good reason to use a mathematically rigorous calculation for statistical significance, a commonsense approximation is generally sufficient. In support of the commonsense approach described below, consider this excerpt from a StatSoft, Inc. (http://www.statsoftinc.com) discussion on the topic:

There is no way to avoid arbitrariness in the final decision as to what level of significance will be treated as really ‘significant.’ That is, the selection of some level of significance, up to which the results will be rejected as invalid, is arbitrary.

Typically, it is fairly easy to add iterations to performance tests to increase the total number of measurements collected; the best way to ensure statistical significance is simply to collect additional data if there is any doubt about whether or not the collected data represents reality. Whenever possible, ensure that you obtain a sample size of at least 100 measurements from at least two independent tests.

Although there is no strict rule about how to decide which results are statistically similar without complex equations that call for huge volumes of data that commercially driven software projects rarely have the time or resources to collect, the following is a reasonable approach to apply if there is doubt about the significance or reliability of data after evaluating two test executions where the data was expected to be similar. Compare results from at least five test executions and apply the rules of thumb below to determine whether or not test results are similar enough to be considered reliable:

If more than 20 percent (or one out of five) of the test-execution results appear not to be similar to the others, something is generally wrong with the test environment, the application, or the test itself.
If a 90th percentile value for any test execution is greater than the maximum or less than the minimum value for any of the other test executions, that data set is probably not statistically similar.
If measurements from a test are noticeably higher or lower, when charted side-by-side, than the results of the other test executions, it is probably not statistically similar.

_Figure 15.7 Result Comparison_

If one data set for a particular item (e.g., the response time for a single page) in a test is noticeably higher or lower, but the results for the data sets of the remaining items appear similar, the test itself is probably statistically similar (even though it is probably worth the time to investigate the reasons for the difference of the one dissimilar data set.

Statistical Equivalence

The method above for determining statistical significance actually is applying the principle of statistical equivalence. Essentially, the process outlined above for determining statistical significance could be restated as “Given results data from multiple tests intended to be equivalent, the data from any one of those tests may be treated as statistically significant if that data is statistically equivalent to 80 percent or more of all the tests intended to be equivalent.” Mathematical determination of equivalence using such formal methods as chi-squared and t-tests are not common on commercial software development projects. Rather, it is generally deemed acceptable to estimate equivalence by using charts similar to those used to determine statistical significance.

Statistical Outliers

From a purely statistical point of view, any measurement that falls outside of three standard deviations, or 99 percent, of all collected measurements is considered an outlier. The problem with this definition is that it assumes that the collected measurements are both statistically significant and distributed normally, which is not at all automatic when evaluating performance test data.

For the purposes of this explanation, a more applicable definition of an outlier from a StatSoft, Inc. (http://www.statsoftinc.com) is the following:
Outliers are atypical, infrequent observations: data points which do not appear to follow the distribution of the rest of the sample. These may represent consistent but rare traits, or be the result of measurement errors or other anomalies which should not be modeled.
Note that this (or any other) description of outliers only applies to data that is deemed to be a statistically significant sample of measurements. Without a statistically significant sample, there is no generally acceptable approach to determining the difference between an outlier and a representative measurement.

Using this description, results graphs can be used to determine evidence of outliers — occasional data points that just don’t seem to belong. A reasonable approach to determining if any apparent outliers are truly atypical and infrequent is to re-execute the tests and then compare the results to the first set. If the majority of the measurements are the same, except for the potential outliers, the results are likely to contain genuine outliers that can be disregarded. However, if the results show similar potential outliers, these are probably valid measurements that deserve consideration.

After identifying that a dataset appears to contain outliers, the next question is, how many outliers can be dismissed as “atypical infrequent observations?”

There is no set number of outliers that can be unilaterally dismissed, but rather a maximum percentage of the total number of observations. Applying the spirit of the two definitions above, a reasonable conclusion would be that up to 1 percent of the total values for a particular measurement that are outside of three standard deviations from the mean are significantly atypical and infrequent enough to be considered outliers.

In summary, in practice for commercially driven software development, it is generally acceptable to say that values representing less than 1 percent of all the measurements for a particular item that are at least three standard deviations off the mean are candidates for omission in results analysis if (and only if) identical values are not found in previous or subsequent tests. To express the same concept in a more colloquial way: obviously rare and strange data points that can’t immediately be explained, account for a very small part of the results, and are not identical to any results from other tests are probably outliers.

A note of caution: identifying a data point as an outlier and excluding it from results summaries does not imply ignoring the data point. Excluded outliers should be tracked in some manner appropriate to the project context in order to determine, as more tests are conducted, if a pattern of concern is identified in what by all indications are outliers for individual tests.

Confidence Intervals

Because determining levels of confidence in data is even more complex and time-consuming than determining statistical significance or the existence of outliers, it is extremely rare to make such a determination during commercial software projects. A confidence interval for a specific statistic is the range of values around the statistic where the ‘true’ statistic is likely to be located within a given level of certainty.

Because stakeholders do frequently ask for some indication of the presumed accuracy of test results ? for example, what is the confidence interval for these results? ? another commonsense approach must be employed.

When performance testing, the answer to that question is directly related to the accuracy of the model tested. Since in many cases the accuracy of the model cannot be reasonably determined until after the software is released into production, this is not a particularly useful dependency. However, there is__ a way to demonstrate a confidence interval in the results.

By testing a variety of scenarios, including what the team determines to be “best,” “worst,” and “expected” cases in terms of the measurements being collected, a graphical depiction of a confidence interval can be created, similar to the one below.

_Figure 15.8 Usage Models_

In this graph, a dashed line represents the performance goal, and the three curves represent the results from the worst-case (most performance-intensive), best-case (least performance-intensive), and expected-case user community models. As one would expect, the blue curve from the expected case falls between the best- and worst-case curves. Observing where these curves cross the red line, one can see how many users can access the system in each case while still meeting the stated performance goal. If the team is 95-percent confident (by their own estimation) that the best- and worst-case user community models are truly best- and worst-case, this chart can be read as follows: the tests show, with 95-percent confidence, that between 100 and 200 users can access the system while experiencing acceptable performance.

Although a confidence interval of between 100 and 200 users might seem quite large, it is important to note that without empirical data representing the actual production usage, it is unreasonable to expect higher confidence in results than there is in the models that generate those results. The best that one can do is to be 100-percent confident that the test results accurately represent the model being tested.

Summary

UPDATED WIKI: Chapter 15 – Key Mathematic Principles for Performance Testers

prashantbansode — Thu, 23 Aug 2007 17:50:23 GMT

Chapter 15 – Key Mathematic Principles for Performance Testers

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Learn the uses, meanings of, and concepts underlying common mathematical and statistical principles as they apply to performance test analysis and reporting.

Overview

How to Use This Chapter

Use the “Exemplar Data Sets” section to gain an understanding of the exemplars, which are used to illustrate the key mathematical principles explained throughout the chapter.
Use the remaining sections to learn about key mathematical principles that will help you to understand and present meaningful performance testing reports.

Exemplar Data Sets

This chapter refers to three exemplar data sets for the purposes of illustration, namely.

Data Set A
Data Set B
Data Set C

Data Sets Summary

The following is a summary of Data Sets A, B, and C.

_Figure 15.1 Summary of Data Sets A, B, and C_

Data Set A

_Figure 15.2 Data Set A_

100 total data points, distributed as follows:

5 data points have a value of 1.
10 data points have a value of 2.
20 data points have a value of 3.
30 data points have a value of 4.
20 data points have a value of 5.
10 data points have a value of 6.
5 data points have a value of 7.

Data Set B

_Figure 15.3 Data Set B_

100 total data points, distributed as follows:

80 data points have a value of 1.
20 data points have a value of 16.

Data Set C

_Figure 15.4 Data Set C_

100 total data points, distributed as follows:

11 data points have a value of 0.
10 data points have a value of 1.
11 data points have a value of 2.
13 data points have a value of 3.
11 data points have a value of 4.
11 data points have a value of 5.
11 data points have a value of 6.
12 data points have a value of 7.
10 data points have a value of 8.

Averages

Percentiles

Medians

Normal Values

A normal value is the single value that occurs most often in a data set. Data Set A has a normal value of 4, Data Set B has a normal value of 3, and Data Set C has a normal value of 1.

Standard Deviations

Uniform Distributions

Normal Distributions

Statistical Significance

If more than 20 percent (or one out of five) of the test-execution results appear not to be similar to the others, something is generally wrong with the test environment, the application, or the test itself.
If a 90th percentile value for any test execution is greater than the maximum or less than the minimum value for any of the other test executions, that data set is probably not statistically similar.
If measurements from a test are noticeably higher or lower, when charted side-by-side, than the results of the other test executions, it is probably not statistically similar.

_Figure 15.7 Result Comparison_

If one data set for a particular item (e.g., the response time for a single page) in a test is noticeably higher or lower, but the results for the data sets of the remaining items appear similar, the test itself is probably statistically similar (even though it is probably worth the time to investigate the reasons for the difference of the one dissimilar data set.

Statistical Equivalence

Statistical Outliers

Confidence Intervals

Summary

UPDATED WIKI: Chapter 15 – Key Mathematic Principles for Performance Testers

prashantbansode — Thu, 23 Aug 2007 17:41:28 GMT

Chapter 15 – Key Mathematic Principles for Performance Testers

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Learn the uses, meanings of, and concepts underlying common mathematical and statistical principles as they apply to performance test analysis and reporting.

Overview

How to Use This Chapter

Use the “Exemplar Data Sets” section to gain an understanding of the exemplars, which are used to illustrate the key mathematical principles explained throughout the chapter.
Use the remaining sections to learn about key mathematical principles that will help you to understand and present meaningful performance testing reports.

Exemplar Data Sets

This chapter refers to three exemplar data sets for the purposes of illustration, namely.

Data Set A
Data Set B
Data Set C

Data Sets Summary

The following is a summary of Data Sets A, B, and C.

Cannot resolve link: [image:Fig15.1.GIF]

_Figure 15.1 Summary of Data Sets A, B, and C_

Data Set A

Cannot resolve link: [image:Fig15.2.GIF]

_Figure 15.2 Data Set A_

100 total data points, distributed as follows:

5 data points have a value of 1.
10 data points have a value of 2.
20 data points have a value of 3.
30 data points have a value of 4.
20 data points have a value of 5.
10 data points have a value of 6.
5 data points have a value of 7.

Data Set B

Cannot resolve link: [image:Fig15.3.GIF]

_Figure 15.3 Data Set B_

100 total data points, distributed as follows:

80 data points have a value of 1.
20 data points have a value of 16.

Data Set C

Cannot resolve link: [image:Fig15.4.GIF]

_Figure 15.4 Data Set C_

100 total data points, distributed as follows:

11 data points have a value of 0.
10 data points have a value of 1.
11 data points have a value of 2.
13 data points have a value of 3.
11 data points have a value of 4.
11 data points have a value of 5.
11 data points have a value of 6.
12 data points have a value of 7.
10 data points have a value of 8.

Averages

Percentiles

Medians

Normal Values

A normal value is the single value that occurs most often in a data set. Data Set A has a normal value of 4, Data Set B has a normal value of 3, and Data Set C has a normal value of 1.

Standard Deviations

Uniform Distributions

Normal Distributions

Statistical Significance

If more than 20 percent (or one out of five) of the test-execution results appear not to be similar to the others, something is generally wrong with the test environment, the application, or the test itself.
If a 90th percentile value for any test execution is greater than the maximum or less than the minimum value for any of the other test executions, that data set is probably not statistically similar.
If measurements from a test are noticeably higher or lower, when charted side-by-side, than the results of the other test executions, it is probably not statistically similar.

Cannot resolve link: [image:Fig15.7.GIF]

_Figure 15.7 Result Comparison_

If one data set for a particular item (e.g., the response time for a single page) in a test is noticeably higher or lower, but the results for the data sets of the remaining items appear similar, the test itself is probably statistically similar (even though it is probably worth the time to investigate the reasons for the difference of the one dissimilar data set.

Statistical Equivalence

Statistical Outliers

Confidence Intervals

Because determining levels of confidence in data is even more complex and time-consuming than determining statistical significance or the existence of outliers, it is extremely rare to make such a determination during commercial software projects. A confidence interval for a specific statistic is the range of values around the statistic where the ‘true’ statistic is likely to be located within a given level of certainty.

Because stakeholders do frequently ask for some indication of the presumed accuracy of test results ? for example, what is the confidence interval for these results? ? another commonsense approach must be employed.

When performance testing, the answer to that question is directly related to the accuracy of the model tested. Since in many cases the accuracy of the model cannot be reasonably determined until after the software is released into production, this is not a particularly useful dependency. However, there is__ a way to demonstrate a confidence interval in the results.

By testing a variety of scenarios, including what the team determines to be “best,” “worst,” and “expected” cases in terms of the measurements being collected, a graphical depiction of a confidence interval can be created, similar to the one below.

Cannot resolve link: [image:Fig15.8.GIF]

_Figure 15.8 Usage Models_

In this graph, a dashed line represents the performance goal, and the three curves represent the results from the worst-case (most performance-intensive), best-case (least performance-intensive), and expected-case user community models. As one would expect, the blue curve from the expected case falls between the best- and worst-case curves. Observing where these curves cross the red line, one can see how many users can access the system in each case while still meeting the stated performance goal. If the team is 95-percent confident (by their own estimation) that the best- and worst-case user community models are truly best- and worst-case, this chart can be read as follows: the tests show, with 95-percent confidence, that between 100 and 200 users can access the system while experiencing acceptable performance.

Although a confidence interval of between 100 and 200 users might seem quite large, it is important to note that without empirical data representing the actual production usage, it is unreasonable to expect higher confidence in results than there is in the models that generate those results. The best that one can do is to be 100-percent confident that the test results accurately represent the model being tested.

Summary

UPDATED WIKI: Chapter 14 – Test Execution

prashantbansode — Thu, 23 Aug 2007 17:34:48 GMT

Chapter 14 – Test Execution

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Understand common principles and considerations of performance test execution.
Understand the common activities of performance test execution.

Overview

Performance test execution is the activity that occurs between developing test scripts and reporting and analyzing test results. Much of the performance testing–related training available today treats this activity as little more than starting a test and monitoring it to ensure that the test appears to be running as expected. In reality, this activity is significantly more complex than just clicking a button and monitoring machines. This chapter addresses these complexities based on numerous real-world project experiences.

How to Use this Chapter

Use this chapter to understand the key principles and considerations underlying performance test execution and the various activities that it entails. To get the most from this chapter:

Use the “Approach for Test Execution” section to get** an overview of the approach for performance test execution and as quick reference guide for you and your team.
Use the various activity sections to understand the details of each activity involved in performance test execution.

Approach for Test Execution

The following activities are involved in performance test execution:

Validate the test environment
Validate tests
Run tests
Baseline and benchmark
Archive tests

The following sections discuss each of these activities in detail.

Validate the Test Environment

The goal is for the test environment to mirror your production environment as closely as possible. Typically, any differences between the test and production environments are noted and accounted for while designing tests. Before running your tests, it is important to validate that the test environment matches the configuration that you were expecting and/or designed your test for. If the test environment is even slightly different from the environment you designed your tests to be run against, there is a high probability that your tests might not work at all, or worse, that they will work but will provide misleading data.

The following activities frequently prove valuable when validating a test environment:

Ensure that the test environment is correctly configured for metrics collection.
Turn off any active virus-scanning on load-generating machines during testing, to minimize the likelihood of unintentionally skewing results data as a side-effect of resource consumption by the antivirus/anti-spyware software.
Consider simulating background activity, when necessary. For example, many servers run batch processing during predetermined time periods, while servicing users’ requests. Not accounting for such activities in those periods may result in overly optimistic performance results.
Run simple usage scenarios to validate the Web server layer first if possible, separately from other layers. Run your scripts without think times. Try to run a scenario that does not include database activity. Inability to utilize 100 percent of the Web server’s processor can indicate a network problem or that the load generator clients have reached their maximum output capacity.
Run simple usage scenarios that are limited to reading data to validate database scenarios. Run your script without think times. Use test data feeds to simulate randomness. For example, query for a set of products. Inability to utilize 100 percent of the Web server’s processor can indicate a network problem or that the load-generator clients have reached their maximum output capacity.
Validate the test environment by running more complex usage scenarios with updates and writes to the database, using a mix of test scripts that simulate business actions.
In Web farm environments, check to see if your load tests are implementing Internet Protocol (IP) switching. Not doing so may cause IP affinity, a situation where all of the requests from the load-generation client machine are routed to the same server rather than being balanced across all of the servers in the farm. IP affinity leads to inaccurate load test results because other servers participating in the load balancing will not be utilized.
Work with key performance indicators (KPIs) on all the servers to assess your test environment (processor, network, disk, and memory). Include all servers in the cluster to ensure correct evaluation of your environment.
Consider spending time creating data feeds for the test application. For example, database tables containing production data such as number of users, products, and orders shipped, so that you can create similar conditions to replicate problems in critical usage scenarios. Many scenarios involve running queries against tables containing several thousands of entries, to simulate lock timeouts or deadlocks.

Additional Considerations

Consider the following key points when troubleshooting performance-testing environments:

Look for problems in the load-generation clients from which load is simulated. Client machines often produce inaccurate performance-testing results due to insufficient processor or memory resources. Consider adding more client computers to compensate for fast transactions that may cause higher processor utilization; also consider using more memory when this becomes the bottleneck. Memory can be consumed when test data feeds are cached in load generators, or by more complex scripting in load tests.
Some network interface cards (NICs) when set to auto mode will fail to negotiate with switches in proper full-duplex mode. The result is that the NICs will operate in half-duplex negotiation, which causes inaccurate performance-testing results. A typical perimeter network with a Web server and database server in different layers will be deployed with the Web server having two NICs, one facing your clients and another using a different route to communicate with the database layer. However, be aware that having one NIC in the Web server facing both the clients and the database tier may cause network bottleneck congestion.
The database server in the production environment may be using separate hard drives for log files and data files associated with the database as a matter of policy. Replicate such deployment configurations to avoid inaccurate performance-testing results. Consider that if DNS is not properly configured, it might cause broadcast messages to be sent when opening database connections by using the database server name. Name-resolution issues may cause connections to open slowly.
Improper data feeds consumed by your scripts will frequently cause you to overlook problems with the environment. For example, low processor activity may be caused by artificial locking due to scripts querying the same record from the database. Consider creating test data feeds that simulate the correct business actions, accounting for variability of data sent from the post request. Load-generation tools may use a central repository such as a database or files in a directory structure to collect performance test data. Make sure that the data repository is located on a machine that will not cause traffic in the route used by your load-generation tools; for example, putting the data repository in the same virtual local-area network (VLAN) of the machine used to manage data collection.
Load-generation tools may require the use of special accounts between load-generator machines and the computers that collect performance data. Make sure that you set such configurations correctly. Verify that data collection is occurring in the test environment, taking into consideration that the traffic may be required to pass through a firewall.

Validate Tests

Poor load simulations can render all previous work useless. To understand the data collected from a test run, the load simulation must accurately reflect the test design. When the simulation does not reflect the test design, the results are prone to misinterpretation. Even if your tests accurately reflect the test design, there are still many ways that the test can yield invalid or misleading results. Although it may be tempting to simply trust your tests, it is almost always worth the time and effort to validate the accuracy of your tests before you need to depend on them to provide results intended to assist in making the “go-live” decision. It may be useful to think about test validation in terms of the following four categories:

Test design implementation. To validate that you have implemented your test design accurately (using whatever method you have chosen), you will need to run the test and examine exactly what the test does.
Concurrency. After you have validated that your test conforms to the test design when run with a single user, run the test with several users. Ensure that each user is seeded with unique data, and that users begin their activity within a few seconds of one another — not all at the same second, as this is likely to create an unrealistically stressful situation that would add complexity to validating the accuracy of your test design implementation. One method of validating that tests run as expected with multiple users is to use three test runs; one with 3 users, one with 5 users, and one with 11 users. These three tests have a tendency to expose many common issues with both the configuration of the test environment (such as a limited license being installed on an application component) and the test itself (such as parameterized data not varying as intended).
Combinations of tests. Having validated that a test runs as intended with a single user and with multiple users, the next logical step is to validate that the test runs accurately in combination with other tests. Generally, when testing performance, tests get mixed and matched to represent various combinations and distributions of users, activities, and scenarios. If you do not validate that your tests have been both designed and implemented to handle this degree of complexity prior to running critical test projects, you can end up wasting a lot of time debugging your tests or test scripts when you could have been collecting valuable performance information.
Test data validation. Once you are satisfied that your tests are running properly, the last critical validation step is to validate your test data. Performance testing can utilize and/or consume large volumes of test data, thereby increasing the likelihood of errors in your dataset. In addition to the data used by your tests, it is important to validate that your tests share that data as intended, and that the application under test is seeded with the correct data to enable your tests.

Dynamic Data

The following are technical reasons for using dynamic data correctly in load test scripts:

Using the same data value causes artificial usage of caching because the system will retrieve data from copies in memory. This can happen throughout different layers and components of the system, including databases, file caches of the operating systems, hard drives, storage controllers, and buffer managers. Reusing data from the cache during performance testing might account for faster testing results than would occur in the real world.
Some business scenarios require a relatively small range of data selection. In such a case, even reusing the cache more frequently will simulate other performance-related problems, such as database deadlocks and slower response times due to timeouts caused by queries to the same items. This type of scenario is typical of marketing campaigns and seasonal sales events.
Some business scenarios require using unique data during load testing; for example, if the server returns session-specific identifiers during a session after login to the site with a specific set of credentials. Reusing the same login data would cause the server to return a bad session identifier error. Another frequent scenario is when the user enters a unique set of data, or the system fails to accept the selection; for example, registering new users that would require entering a unique user ID on the registration page.
In some business scenarios, you need to control the number of parameterized items; for example, a caching component that needs to be tested for its memory footprint to evaluate server capacity, with a varying number of products in the cache.
In some business scenarios, you need to reduce the script size or the number of scripts; for example, several instances of an application will live in one server, reproducing a scenario where an independent software vendor (ISV) will host them. In this scenario, the Uniform Resource Locators (URLs) need to be parameterized during load test execution for the same business scenarios.
Using dynamic test data in a load test tends to reproduce more complicated and time-sensitive bugs; for example, a deadlock encountered as a result of performing different actions using different user accounts.
Using dynamic test data in a load test allows you to use error values if they suit your test plan; for example, using an ID that is always a positive number when testing to simulate hacker behavior. It may be beneficial to use zero or negative values when testing to replicate application errors, such as scanning the database table when an invalid value is supplied.

Test Validation

The following are some commonly employed methods of test validation, which are frequently used in combination with one another:

Run the test first with a single user only. This makes initial validation much less complex.
Observe your test while it is running and pay close attention to any behavior you feel is unusual. Your instincts are usually right, or at least valuable.
Use the system manually during test execution so that you can compare your observations with the results data at a later time.
Make sure that the test results and collected metrics represent what you intended them to represent.
Check to see if any of the parent requests or dependent requests failed.
Check the content of the returned pages, as load-generation tools sometimes report summary results that appear to “pass” even though the correct page or data was not returned.
Run a test that loops through all of your data to check for unexpected errors.
If appropriate, validate that you can reset test and/or application data following a test run.
At the conclusion of your test run, check the application database to ensure that it has been updated (or not) according to your test design. Consider that many transactions in which the Web server returns a success status with a “200” code might be failing internally; for example, errors due to a previously used user name in a new user registration scenario, or an order number that is already in use.
Consider cleaning the database entries between error trials to eliminate data that might be causing test failures; for example, order entries that you cannot reuse in subsequent test execution.
Run tests in a variety of combinations and sequences to ensure that one test does not corrupt data needed by another test in order to run properly.

Additional Considerations

Consider the following additional points when validating your tests:

Do not use performance results data from your validation test runs as part of your final report.
Report performance issues uncovered during your validation test runs.
Use appropriate load-generation tools to create a load that has the characteristics specified in your test design.
Ensure that the intended performance counters for identified metrics and resource utilization are being measured and recorded, and that they are not interfering with the accuracy of the simulation.
Run other tests during your performance test to ensure that the simulation is not impacting other parts of the system. These other tests may be either automated or manual.
Repeat your test, adjusting variables such as user names and think times to see if the test continues to behave as anticipated.
Remember to simulate ramp-up and cool-down periods appropriately.

Questions to Ask

What additional team members should be involved in evaluating the accuracy of this test?
Do the preliminary results make sense?
Is the test providing the data we expected?

Run Tests

Although the process and flow of running tests are extremely dependent on your tools, environment, and project context, there are some fairly universal tasks and considerations to keep in mind when running tests.

Once it has been determined that the application under test is in an appropriate state to have performance tests run against it, the testing generally begins with the highest-priority performance test that can reasonably be completed based on the current state of the project and application. After each test run, compile a brief summary of what happened during the test and add these comments to the test log for future reference. These comments may address machine failures, application exceptions and errors, network problems, or exhausted disk space or logs. After completing the final test run, ensure that you have saved all of the test results and performance logs before you dismantle the test environment.

Whenever possible, limit tasks to one to two days each to ensure that no time will be lost if the results from a particular test or battery of tests turn out to be inconclusive, or if the initial test design needs modification to produce the intended results. One of the most important tasks when running tests is to remember to modify the tests, test designs, and subsequent strategies as results analysis leads to new priorities.

A widely recommended guiding principle is: Run test tasks in one- to two-day batches. See the tasks through to completion, but be willing to take important detours along the way if an opportunity to add additional value presents itself.

Keys to Efficiently and Effectively Running Tests

In general, the keys to efficiently and effectively running tests include:

Revisit performance-testing priorities after no more than two days.
Remember to capture and use a performance baseline.
Plan to spend some time fixing application errors, or debugging the test.
Analyze results immediately so that you can modify your test plan accordingly.
Communicate test results frequently and openly across the team.
Record results and significant findings.
Record other data needed to repeat the test later.
At appropriate points during test execution, stress the application to its maximum capacity or user load, as this can provide extremely valuable information.
Remember to validate application tuning or optimizations.
Consider evaluating the effect of application failover and recovery.
Consider measuring the effects of different system configurations.

Additional Considerations

Consider the following additional points when running your tests:

Performance testing is frequently conducted on an isolated network segment to prevent disruption of other business operations. If this is not the case for your test project, ensure that you obtain permission to generate loads during certain hours on the available network.
Before running the real test, consider executing a quick “smoke test” to make sure that the test script and remote performance counters are working correctly.
If you choose to execute a smoke test, do not report the results as official or formal parts of your testing.
Reset the system (unless your scenario is to do otherwise) before running a formal test.
If at all possible, execute every test twice. If the results produced are not very similar, execute the test again. Try to determine what factors account for the difference.
No matter how far in advance a test is scheduled, give the team 30-minute and 5-minute warnings before launching the test (or starting the day’s testing). Inform the team whenever you are not going to be executing for more than one hour in succession.
Do not process data, write reports, or draw diagrams on your load-generating machine while generating a load because this can corrupt the data.
Do not throw away the first iteration because of script compilation or other reasons. Instead, measure this iteration separately so you will know what the first user after a system-wide reboot can expect.
Test execution is never really finished, but eventually you will reach a point of diminishing returns on a particular test. When you stop obtaining valuable information, change your test.
If neither you nor your development team can figure out the cause of an issue in twice as much time as it took the test to execute, it may be more efficient to eliminate one or more variables/potential causes and try again.
If your intent is to measure performance related to a particular load, it is important to allow time for the system to stabilize between increases in load to ensure the accuracy of measurements.
Make sure that the client computers (also known as load-generation client machines) that you use to generate load are not overly stressed. Utilization of resources such as processor and memory should remain low enough to ensure that the load-generation environment is not itself a bottleneck.
Analyze results immediately and modify your test plan accordingly.
Work closely with the team or team sub-set that is most relevant to the test.
Communicate test results frequently and openly across the team.
If you will be repeating the test, consider establishing a test data restore point before you begin testing.
In most cases, maintaining a test execution log that captures notes and observations for each run is invaluable.
Treat workload characterization as a moving target. Adjust new settings for think times and number of users to model the new total number of users for normal and peak loads.
Observe your test during execution and pay close attention to any behavior you feel is unusual. Your instincts are usually right, or at least valuable.
Ensure that performance counters relevant for identified metrics and resource utilization are being measured and are not interfering with the accuracy of the simulation.
Use the system manually during test execution so that you can compare your observations with the results data at a later time.
Remember to simulate ramp-up and cool-down periods appropriately.

Questions to ask:

Have recent test results or project updates made this task more or less valuable compared to other tests we could be conducting right now?
What additional team members should be involved with this task?
Do the preliminary results make sense?

Baseline and Benchmark

When baselines and benchmarks are used, they are generally the first and last tests you will execute, respectively. Of all the tests that may be executed during the course of a project, it is most important that baselines and benchmarks be well understood and controlled, making the validations discussed above even more important.

Baselines

Creating a baseline is the process of running a set of tests to capture performance metric data for the purpose of evaluating the effectiveness of subsequent performance-improving changes to the system or application.

With respect to Web applications, you can use a baseline to determine whether performance is improving or declining and to find deviations across builds and versions. For example, you could measure load time, number of transactions processed per unit of time, number of Web pages served per unit of time, and resource utilization such as memory and processor usage. Some considerations about using baselines include:

A baseline can be created for a system, component, or application.
A baseline can be created at different layers: database, Web services, etc.
A baseline can be used as a standard for comparison to track future optimizations or regressions. When using a baseline for this purpose, it is important to validate that the baseline tests and results are well understood and repeatable.
Baselines can help product teams articulate variances that represent degradation or optimization during the course of the development life cycle by providing a known starting point for trend analysis. Baselines are most valuable if created using a set of reusable test assets; it is important that such tests are representative of workload characteristics that are both repeatable and provide an appropriately accurate simulation.
Baseline results can be articulated by using combinations of a broad set of key performance indicators such as response time, processor, memory, disk, and network.
Sharing baseline results across the team establishes a common foundation of information about performance characteristics to enable future communication about performance changes in an application or component.
A baseline is specific to an application and is most useful for comparing performance across different builds, versions, or releases.
Establishing a baseline before making configuration changes almost always saves time because it enables you to quickly determine what effect the changes had on the application’s performance.

Benchmarking

Benchmarking is the process of comparing your system performance against an industry standard that is endorsed by some other organization.

From the perspective of Web application development, benchmarking involves running a set of tests that comply with the specifications of an industry benchmark to capture the performance metrics for your application necessary to determine its benchmark score. You can then compare your application against other systems or applications that have also calculated their score for the same benchmark. You may choose to tune your application performance to achieve or surpass a certain benchmark score. Some considerations about benchmarking include:

A benchmark score is achieved by working within industry specifications or by porting an existing implementation to comply with those specifications.
Benchmarking generally requires identifying all of the necessary components that will run together, the market where the product exists, and the specific metrics to measure.
Benchmark scores can be published publicly and may result in comparisons being made by competitors. Performance metrics that may be included along with benchmark scores include response time, transactions processed per unit of time, Web pages accessed per unit of time, processor usage, memory usage, and search times.

Archive Tests

Some degree of change control or version control can be extremely valuable for managing scripts, scenarios, and/or data changes between each test execution, and for communicating these differences to the rest of the team. Some teams prefer to check their test scripts, results, and reports into the same version-control system as the build of the application to which they apply. Other teams simply save copies into dated folders on a periodic basis, or have their own version-control software dedicated to the performance team. It is up to you and your team to decide what method is going to work best for you, but in most cases archiving tests, test data, and test results saves much more time than it takes over the course of a performance-testing project.

Additional Considerations

Consider the following additional points when creating baselines and benchmarking:

You can use archived test scripts, data, and results to create the baseline for the next version of your product. Archiving this information together with the build of the software that was tested satisfies many auditability standards.
In most cases, performance test scripts are improved or modified with each new build. If you do not save a copy of the script and identify the build it was used against, you can end up doing a lot of extra work to get your scripts running again in the case of a build rollback.
With the overwhelming majority of load-generation tools, implementing the test is a minor software-development effort in itself. While this effort generally does not need to follow all of the team’s standards and procedures for software development, it is a good idea to adopt a sound and appropriately “weighted” development process for performance scripts that complements or parallels the process your development team employs.

Summary

Performance test execution involves activities such as validating test environments/scripts, running the test, and generating the test results. It can also include creating baselines and/or benchmarks of the performance characteristics.

It is important to validate the test environment to ensure that the environment truly represents the production environment.

Validate test scripts to check if correct metrics are being collected, and if the test script design is correctly simulating workload characteristics.

UPDATED WIKI: Chapter 13 – Determining Individual User Data and Variances

prashantbansode — Thu, 23 Aug 2007 17:04:21 GMT

Chapter 13 – Determining Individual User Data and Variances

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Learn how to determine realistic durations and distribution patters for user delay times.
Learn how to incorporate realistic user delays into test designs and test scripts.
Learn about key variables to consider when defining workload characterization.
Learn about the elements of user behavior that will aid with modeling the user experience when creating load tests.

Overview

This chapter describes the process of determining realistic individual user delays, user data, and abandonment. For performance testing to yield results that are directly applicable to understanding the performance characteristics of an application in production, the tested workloads must represent the real-world production environment. To create a reasonably accurate representation of reality, you must model users with a degree variability and randomness similar to that found in a representative cross-section of users.

How to Use This Chapter

Use this chapter to understand how to model variances such as user delays, user data, and user abandonment so that your workload characterization will create realistic usage patterns, thus improving the accuracy of production simulations. To get the most from this chapter:

Use the “User Delay” section, along with the sections that follow, to understand the key concepts of user delay modeling and its impact on workload characterization.
Use the “Determining Individual User Data” section to understand the key concepts of user data and its impact on workload characterization.
Use the “User Abandonment” section to understand the key concepts of user abandonment and its impact on workload characterization.

User Delays

The more accurately users are modeled, the more reliable performance test results will be. One frequently overlooked aspect of accurate user modeling is the modeling of user delays. This section explains how to determine user delay times to be incorporated into your workload model and subsequently into your performance scripts.

During a session, the user can be in a number of different states — browsing, logging onto the system, and so on. Customers will have different modes of interacting with the Web site; some users are familiar with the site and quickly go from one page to another, while others take longer to decide which action they will take. Therefore, characterizing user behavior must involve modeling the customer sessions based on page flow, frequency of hits, the amount of time users’ pause between viewing pages, and any other factor specific to how users interact with your Web site.

Consequences of Improperly Modeling User Delays

To ensure realistic load tests, any reasonable attempt at applying ranges and distributions is preferable to ignoring the concept of varying user delays. Creating a load test in which every user spends exactly the same amount of time on each page is simply not realistic and will generate misleading results. For example, you can very easily end up with results similar to the following.

_Figure 13.1 Results for Using Static User Delays_
In case you are not familiar with response graphs, each dot represents a user activity (in this case, a page request); the horizontal axis shows the time, in seconds, from the start of the test run; and individual virtual testers are listed on the vertical axis. This particular response graph is an example of “banding” or “striping.” Banding should be avoided when doing load or performance testing, although it may be valuable as a stress test. From the server’s perspective, this test is the same as 10 users executing the identical actions synchronously: Home page??? wait x seconds??? page1.
To put a finer point on it, hold a ruler vertically against your screen and move it slowly across the graph from left to right. This is what the server sees: no dots, no dots, no dots, lots of dots, no dots. This is a very poor representation of actual user communities.
The following figure is a much better representation of actual users, achieved by adding some small-range uniform and normally distributed delays to the same test.

_Figure 13.2 Results for Using Normally Distributed User Delays_
If you perform the same activity with the ruler, you will see that the dots are more evenly distributed this time, which dramatically increases both the realism of the simulated load and the accuracy of the performance test results.

Step 1 – Determine User Delays

Delays that occur while users view content on Web pages — also commonly known as think times — represent the answers to questions such as “How long does it take a user to enter their login credentials?” and “How much time will users spend reading this page?” You can use several different methods to estimate think times associated with user activities on your Web site. The best method, of course, is to use real data collected about your production site. This is rarely possible, however, because testing generally occurs before the site is released to production. This necessitates making educated guesses or approximations regarding activity on the site.

The most commonly useful methods of determining this include the following:

When testing a Web site that is already in production, you can determine the actual values and distribution by extracting the average and standard deviation for user viewing (or typing) time from the log file for each page. With this information, you can easily determine the think time for each page. Your production site may also have Web traffic–monitoring software that provides this type of information directly.
If you have no log files, you can run simple in-house experiments using employees, customers, clients, friends, or family members to determine, for example, the page-viewing time differences between new and returning users. This type of simplified usability study tends to be a highly effective method of data collection for Web sites that have never been live, as well as validation of data collected by using other methods.
Time yourself using the site, or by performing similar actions on a similar site. Obviously, this method is highly vulnerable to personal bias, but it is a reasonable place to start until you get a chance to time actual users during User Acceptance Testing (UAT) or conduct your own usability study.
In the absence of any better source of information, you can leverage some of the metrics and statistics that have already been collected by research companies such as Nielsen//NetRatings, Keynote, or MediaMetrix. These statistics provide data on average page-viewing times and user session duration based on an impersonal sample of users and Web sites. Although these numbers are not from your specific Web site, they can work quite well as first approximations.

There is no need to spend a lot of time collecting statistically significant volumes of data, or to be excessively precise. All you really need to know is how long a typical user will spend performing an activity, give or take a second or two. However, depending on the nature of your site, you may want to determine user delay times separately for first-time and experienced users.

Step 2 – Apply Delay Ranges

Simply determining how much time one person spends visiting your pages, or what the variance in time between users is, is not enough in itself — you must vary delay times by user. It is extremely unlikely that each user will spend exactly the same amount of time on a page. It is also extremely likely that conducting a performance test in which all users spend the same amount of time on a page will lead to unrealistic or at least unreliable results.

To convert the delay times or delay ranges from step 1 into something that also represents the variability between users, the following three pieces of information are required:

The minimum delay time
The maximum delay time
The distribution or pattern of user delays between those points

If you do not have a minimum and maximum value from your analysis in step 1, you can apply heuristics as follows to determine acceptable estimates:

The minimum value could be:

An experienced user who intended to go to the page but will not remain there long (for example, a user who only needs the page to load in order to scan, find, and click the next link).
A user who realized that they clicked to the wrong page.
A user who clicked through a form that had all of its values pre-filled.
The minimum length of time you think a user needs to type the required information into the form.
Half of the value that you determined was “typical.”

The maximum value could be:

Session time-out.
Sufficient time for a user to look up information for a form.
No longer than it takes a slow reader to read the entire page.
The time it takes to read, three times out loud, the text that users are expected to read. (This is the heuristic used by the film industry for any onscreen text.)
Double the value that you determined was “typical.”

Although you want your estimate to be relatively close to reality, any range that covers ~75 percent of the expected users is sufficient to ensure that you are not unintentionally skewing your results.

Step 3 – Apply Distributions

There are numerous mathematical models for these types of distributions. Four of these models cover the overwhelming majority of user delay scenarios:

Linear or uniform distribution
Normal distribution
Negative exponential distribution
Double hump normal distribution

Linear or Uniform Distribution

A uniform distribution between a minimum and a maximum value is the easiest to model. This distribution model simply selects random numbers that are evenly distributed between the upper and lower bounds of the range. This means that it is no more likely that the number generated will be closer to the middle or either end of the range. The figure below shows a uniform distribution of 1000 values generated between 0 and 25. Use a uniform distribution in situations where there is a reasonably clear minimum and maximum value, but either have or expect to have a distinguishable pattern between those end points.

_Figure 13.3 Uniform Distribution_

Normal Distribution

A normal distribution, also known as a bell curve, is more difficult to model but is more accurate in almost all cases. This distribution model selects numbers randomly in such a way that the frequency of selection is weighted toward the center, or average value. The figure below shows a normal distribution of 1000 values generated between 0 and 25 (that is, a mean of 12.5 and a standard deviation of 3.2). Normal distribution is generally considered to be the most accurate mathematical model of quantifiable measures of large cross-sections of people when actual data is unavailable. Use a normal distribution in any situation where you expect the pattern to be shifted toward the center of the end points. The valid range of values for the standard deviation is from 0 (equivalent to a static delay of the midpoint between the maximum and minimum values) and the maximum value minus the minimum value (equivalent to a uniform distribution). If you have no way to determine the actual standard deviation, a reasonable approximation is 25 percent of (or .25 times the range) of the delay.

_Figure 13.4 Normal Distribution_

Negative Exponential Distribution

Negative exponential distribution creates a distribution similar to that shown in the graph below. This model skews the frequency of delay times strongly toward one end of the range. This model is most useful for situations such as users clicking a “play again” link that only activates after multimedia content has completed playing. The following figure shows a negative exponential distribution of 1000 values generated between 0 and 25.

_Figure 13.5 Negative Exponential Distribution_

Double Hump Normal Distribution

The double hump normal distribution creates a distribution similar to that shown in the graph below. To understand when this distribution would be used, consider the first time you visit a Web page that has a large amount of text. On that first visit, you will probably want to read the text, but the next time you might simply click through that page on the way to a page located deeper in the site. This is precisely the type of user behavior this distribution represents. The figure below shows that 60 percent of the users who view this page spend about 8 seconds on the page scanning for the next link to click, and the other 40 percent of the users actually read the entire page, which takes about 45 seconds. You can see that both humps are normal distributions with different minimum, maximum, and standard deviation values.

_Figure 13.6 Double Hump Normal Distribution_

To implement this pattern, simply write a snippet of code to generate a number between 1 and 100 to represent a percentage of users. If that number is below a certain threshold (in the graph above, below 61), call the normal distribution function with the parameters to generate delays with the first distribution pattern. If that number is at or above that threshold, call the normal distribution function with the correct parameters to generate the second distribution pattern.

Determining Individual User Data

Once you have a list of key scenarios, you will need to determine how individual users actually accomplish the tasks or activities related to those scenarios, and the user-specific data associated with a user accomplishing that task or activity.

Unfortunately, navigation paths alone do not provide all of the information required to implement a workload simulation. To fully implement the workload model, you need several more pieces of information. This information includes:

How long users may spend on a page?
What data may need to be entered on each page?
What conditions may cause a user to change navigation paths?

Considerations

Consider the following key points when identifying unique data for navigation paths and/or simulated users:

Performance tests frequently consume large amounts of test data. Ensure that you have enough data to conduct an effective test.
Using the same data repeatedly will frequently lead to invalid performance test results.
Especially when designing and debugging performance tests, test databases can become dramatically overloaded with data. Periodically check to see if the data base is storing unrealistic volumes of data for the situation you are trying to simulate.
Consider including invalid data in your performance tests. For example, include some users who mistype their password on the first attempt but do it correctly on a second try.
First-time users usually spend significantly longer on each page or activity than experienced users.
The best possible test data is test data collected from a production database or log file.
Consider client-side caching. First-time users will be downloading every object on the site, while frequent visitors are likely to have many static objects and/or cookies stored in their local cache. When capturing the uniqueness of the user’s behavior, consider whether that user represents a first-time user or a user with an established client-side cache.

User Abandonment

User abandonment refers to situations where customers exit the Web site before completing a task, because of performance slowness. People have different rates of tolerance for performance, depending on their psychological profile and the type of page they request. Failing to account for user abandonment will cause loads that are highly unrealistic and improbable. Load tests should simulate user abandonment as realistically as possible or they may cause types of load that will never occur in real life — and create bottlenecks that might never happen with real users. Load tests should report the number of users that might abandon the Web site due to poor performance.

In a typical Web site traffic pattern, when the load gets too heavy for the system/application to handle, the site slows down, causing people to abandon it, thus decreasing the load until the system speeds back up to an acceptable rate. Abandonment creates a self-policing mechanism that recovers performance at previous levels (when the overload occurred), even at the cost of losing some customers. Therefore, one reason to accurately account for user abandonment is to see just how many users “some” is. Another reason is to determine the actual volume your application can maintain before you start losing customers. Yet another reason to account for user abandonment is to avoid simulating, and subsequently resolving, bottlenecks that realistically might not even be possible.

If you do not account for abandonment at all, the load test may wait indefinitely to receive the page or object it requested. When the test eventually receives that object, even if “eventually” takes hours longer than a real user would wait, the test will move on to the next object as if nothing were wrong. If the request for an object simply is not acknowledged, the test skips it and makes a note in the test execution log with no regard as to whether that object was critical to the user. Note that there are some cases where not accounting for abandonment is an accurate representation of reality; for instance, a Web-based application that has been exclusively created for an audience that has no choice but to wait because there are no alternative methods of completing a required task.

Considerations

The following are generally useful guidelines related to user abandonment:

Check the abandonment rate before evaluating response times. If the abandonment rate for a particular page is less than about 2 percent, consider the possibility of those response times being outliers.
Check the abandonment rate before drawing conclusions about load. Remember, every user who abandons is one less user applying load. Although the response-time statistics may look good, if you have 75-percent abandonment, load is roughly 75 percent lighter than it was being tested for.
If the abandonment rate is more than about 20 percent, consider disabling the abandonment routine and re-executing the test to help gain information about what is causing the problem.

Summary

The process of designing realistic user delays into tests and test scripts is critical for workload characterizations to generate accurate results. For performance testing to yield results that are directly applicable to understanding the performance characteristics of an application in production or a projected future business volume, the tested workloads must represent reality, replicating user delay patterns.

To create a reasonably accurate representation of reality, you must model user delays with variability and randomness by taking into account individual user data and user abandonment, similar to a representative cross-section of users.

UPDATED WIKI: Chapter 13 – Determining Individual User Data and Variances

prashantbansode — Thu, 23 Aug 2007 17:03:07 GMT

Chapter 13 – Determining Individual User Data and Variances

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Learn how to determine realistic durations and distribution patters for user delay times.
Learn how to incorporate realistic user delays into test designs and test scripts.
Learn about key variables to consider when defining workload characterization.
Learn about the elements of user behavior that will aid with modeling the user experience when creating load tests.

Overview

How to Use This Chapter

Use the “User Delay” section, along with the sections that follow, to understand the key concepts of user delay modeling and its impact on workload characterization.
Use the “Determining Individual User Data” section to understand the key concepts of user data and its impact on workload characterization.
Use the “User Abandonment” section to understand the key concepts of user abandonment and its impact on workload characterization.

User Delays

Consequences of Improperly Modeling User Delays

Step 1 – Determine User Delays

When testing a Web site that is already in production, you can determine the actual values and distribution by extracting the average and standard deviation for user viewing (or typing) time from the log file for each page. With this information, you can easily determine the think time for each page. Your production site may also have Web traffic–monitoring software that provides this type of information directly.
If you have no log files, you can run simple in-house experiments using employees, customers, clients, friends, or family members to determine, for example, the page-viewing time differences between new and returning users. This type of simplified usability study tends to be a highly effective method of data collection for Web sites that have never been live, as well as validation of data collected by using other methods.
Time yourself using the site, or by performing similar actions on a similar site. Obviously, this method is highly vulnerable to personal bias, but it is a reasonable place to start until you get a chance to time actual users during User Acceptance Testing (UAT) or conduct your own usability study.
In the absence of any better source of information, you can leverage some of the metrics and statistics that have already been collected by research companies such as Nielsen//NetRatings, Keynote, or MediaMetrix. These statistics provide data on average page-viewing times and user session duration based on an impersonal sample of users and Web sites. Although these numbers are not from your specific Web site, they can work quite well as first approximations.

Step 2 – Apply Delay Ranges

The minimum delay time
The maximum delay time
The distribution or pattern of user delays between those points

If you do not have a minimum and maximum value from your analysis in step 1, you can apply heuristics as follows to determine acceptable estimates:

The minimum value could be:

An experienced user who intended to go to the page but will not remain there long (for example, a user who only needs the page to load in order to scan, find, and click the next link).
A user who realized that they clicked to the wrong page.
A user who clicked through a form that had all of its values pre-filled.
The minimum length of time you think a user needs to type the required information into the form.
Half of the value that you determined was “typical.”

The maximum value could be:

Session time-out.
Sufficient time for a user to look up information for a form.
No longer than it takes a slow reader to read the entire page.
The time it takes to read, three times out loud, the text that users are expected to read. (This is the heuristic used by the film industry for any onscreen text.)
Double the value that you determined was “typical.”

Step 3 – Apply Distributions

There are numerous mathematical models for these types of distributions. Four of these models cover the overwhelming majority of user delay scenarios:

Linear or uniform distribution
Normal distribution
Negative exponential distribution
Double hump normal distribution

Linear or Uniform Distribution

Normal Distribution

Negative Exponential Distribution

Negative exponential distribution creates a distribution similar to that shown in the graph below. This model skews the frequency of delay times strongly toward one end of the range. This model is most useful for situations such as users clicking a “play again” link that only activates after multimedia content has completed playing. The following figure shows a negative exponential distribution of 1000 values generated between 0 and 25.

Cannot resolve link: [image:Fig.13.5.GIF]

_Figure 13.5 Negative Exponential Distribution_

Double Hump Normal Distribution

Determining Individual User Data

How long users may spend on a page?
What data may need to be entered on each page?
What conditions may cause a user to change navigation paths?

Considerations

Consider the following key points when identifying unique data for navigation paths and/or simulated users:

Performance tests frequently consume large amounts of test data. Ensure that you have enough data to conduct an effective test.
Using the same data repeatedly will frequently lead to invalid performance test results.
Especially when designing and debugging performance tests, test databases can become dramatically overloaded with data. Periodically check to see if the data base is storing unrealistic volumes of data for the situation you are trying to simulate.
Consider including invalid data in your performance tests. For example, include some users who mistype their password on the first attempt but do it correctly on a second try.
First-time users usually spend significantly longer on each page or activity than experienced users.
The best possible test data is test data collected from a production database or log file.
Consider client-side caching. First-time users will be downloading every object on the site, while frequent visitors are likely to have many static objects and/or cookies stored in their local cache. When capturing the uniqueness of the user’s behavior, consider whether that user represents a first-time user or a user with an established client-side cache.

User Abandonment

Considerations

The following are generally useful guidelines related to user abandonment:

Check the abandonment rate before evaluating response times. If the abandonment rate for a particular page is less than about 2 percent, consider the possibility of those response times being outliers.
Check the abandonment rate before drawing conclusions about load. Remember, every user who abandons is one less user applying load. Although the response-time statistics may look good, if you have 75-percent abandonment, load is roughly 75 percent lighter than it was being tested for.
If the abandonment rate is more than about 20 percent, consider disabling the abandonment routine and re-executing the test to help gain information about what is causing the problem.