Interview Extract - Performance testing

Reference: http://www.whatistesting.com/interviews/sbarber.htm

" Scott Barber is the person behind www.perftestplus.com, co-founder of WOPR (the Workshop On Performance and Reliability) and a prolific writer. In this interview he explains what performance testing is, how one should go about learning it, various resources that can help you and many more things. If you want to ask him more questions or you want to send any feedback on this interview, please send it to webmaster @ whatistesting.com. "

If one wants to gain expertise in performance testing what all should one do? What are the tools, books and sites that can be helpful?
There is very little consolidated information out there about the type of performance testing that most organizations need. There is no doubt that my site (http://www.perfetsplus.com) and articles are one of the most complete references for “Practical Performance Testing”. However, it is by no means complete. Some of the other places I have learned valuable lessons that I have applied to performance testing are:
* Everything I can find written by Alberto Savoia
* Lessons Learned in Software Testing, John Wiley & Sons, 2002 by Cem Kaner, James Bach and Brett Pettichord.
* Testing Computer Software (2nd Ed.), International Thomson Computer Press, 1993 by Cem Kaner, Jack Falk, & Hung Quoc Nguyen.
* Select Chapters from The Web Testing Handbook, STQE Publishers, 2001 by Steve Splaine and Stefan P. Jaskiel.
* Research published by performance tool vendors.
* Research published by testing and research organizations such as Nielsen//NetRatings, Keynote Systems, Inc., and IEEE
* Online forums and magazines such as StickyMinds.com, QAForums.com and stpmag.com and performancetester.com.
* Performance Engineering of Software Systems, Addison-Wesley, 1990 by Connie U. Smith, Ph.D.
* Scaling for E-Business: technologies, models, performance, and capacity planning, Prentice Hall, 2000 by Daniel A. Menascé, Ph.D.
* Speed Up Your Site, New Riders, 2003 by Andrew B. King
* Anything by Cem Kaner, James Bach, Jerry Weinburg and/or Edward Tufte.
These days, I continue learning through networking with experts, being a technical editor for new performance related publications, beta testing tools, hosting and/or attending workshops and conferences and assisting with research being led by other experts and students in the field.To really grow your career as a performance tester, I believe there are several things you can do that will gain you largest benefit.
* Become a “Mid-Level Everything” – Developer, DBA, Network Admin, Systems Admin, Architect, Business Analyst, etc.
* Become an expert in the following – your tools, performance analysis, relevant aspects of operational research and statistics, consolidating, interpreting, sorting, and graphically presenting complex information in an intuitive format.
* Become skilled/knowledgeable in the following areas – group facilitation, product management and training.
* Find a mentor. There are several exceptional performance testers out there who believe in Jerry Weinburg’s consulting principle “Give your best work away for free.” Befriending one (or more) of them to ask questions of, share ideas and brainstorm with is amazingly useful… to everyone involved.

What is your opinion about the state of performance testing tools? Are there some open source or free tools available?
Most tools on the market aren’t actually Performance Testing tools, they are Load Generation tools. No matter what the tool vendors claim, they do not make performance testing simple, they do not analyze results, and they do not pinpoint performance issues. They are virtually all grossly overpriced, and never believe the demo. That said, I have a favorite tool that I support heavily… with a long list of caveats. My favorite is my favorite because it allows me to create realistic loads by writing custom functions and procedures in C – rather than trying to guess how to make the silly GUI buttons duplicate the workload I have constructed. The best OpenSource tool for general web based load generation is OpenSTA. As long as you are only testing websites, it is competitive with the major (pay) tools in terms of actual ability to generate load and collect data. There are plenty of other free niche tools for specific platforms and protocols, but they are only useful to small groups of people doing very specific types of testing.

How do you combine performance testing with load and stress testing
Actually, I see Performance Testing as a superset of testing that includes load and stress testing. I often refer to it as “Performance Related Testing”. In general, load testing is testing your application under expected load conditions to determine the overall performance. Stress testing is testing your application under unexpected or extreme conditions to determine failure modes so you can protect against them. In my presentation “Introduction to Performance Testing; The Who, What, Where, When and Why.” (available here) I describe various components of Performance Testing, and the process in general.