Several popular test software packages were compared back-to-back in stress-loading JSP containers. Only half of tested packages produced meaningful results in short stress-load against simplest JSP page. Microsoft WAST and java-based Grinder and JBlitz consistently identified speed limits of web containers. CPU consumption was found to be limiting parameter both for web container and test software.
Computer-A runs JSP containers. AMD K6-3D. 350 MHz, 320Mb ram, Windows ME (SE), Java HotSpot(TM) Client VM (build 1.4.1-rc-b19, mixed mode).
Computer-B runs testing software. Intel P-III 866 MHz, 512Mb ram, Windows XP (Pro), Java HotSpot(TM) Client VM (build 1.4.0-b92, mixed mode).
Network: 10/100 LAN with LinkSys Cable/Router. Internet was disconnected to prevent any distraction.
JSP containers are not primer focus of this presentation. Latest (as of Aug 28, 2002) development and beta versions of Container-T, Container-R and Container-O were downloaded and installed without any tuning.
|
Name |
Full name and version |
Referrence |
|
Grinder |
Grinder 2.8.3 |
|
|
WAST |
Microsoft Web Application Stress Tool 1.1.293.1 |
|
|
StressTool |
Stress Tool 1.3 |
|
|
JMeter |
JMeter 1.7 |
|
|
JBlitz |
JBlitz Pro 3.0 (eval) |
|
|
OpenLoad |
OpenLoad 0.1.2-Win32 |
Jsp containers were run on weaker computer “A” to be sure that testing software is not a limiting factor. Also WAST does not run on Win-ME. Computer “A” was restarted before every new container is tested.
Test was conducted against one page “include.jsp” from JSP examples supplied with Tomcat container. The page outputs current time and than dynamically include current time calculation from another page. Thus, there is no access to business layer or database.
The code was present in most containers or was copies in demo directory. Http port was set to 80, because one package (WAST) has no port setting. Test duration was in 30-40 seconds range. Where possible, average results exclude first 5-10 seconds.
In addition to various test results from load software, special attention was paid to CPU consumption on both computers. Real-time graphs from “system monitor” of Win-ME and “windows task manager” were used. “Visual-average” values were recorded. RAM was less than 50% during all tests in computer-B (load software). RAM on computer-A (jsp container) was not considered.
Common for all test are:
# - test number. Some testes were done not in order of threads increase to prevent possible “training” effect.
Threads – number of parallel processes; same as number of users
CPU-T– approximate average CPU consumption of test machine
CPU-JSP – approximate average CPU consumption of machine with JSP container
TPS – transactions per second
Hits – number of successful requests for time of test
Err – number of errors (Socket errors)
% Err == (Err * 100) / Hits
All times are in milliseconds, unless stated otherwise.
Grinder is pure java software consisting of two parts: main console and any number of workers, possibly in several different machines. Workers must be set with xml file with number of parameters. The following time settings were used here:
grinder.thread.initialSleepTime=1000
grinder.thread.sleepTime=5000
grinder.thread.sleepTimeFactor=0.1
grinder.thread.sleepTimeVariation=0.005
Brief attempts to decrease time between requests caused huge increase in Socket failures. Grinder shows relatively small CPU demand producing high output load.
Grinder and Container-T
|
# |
Threads |
CPU-T |
CPU-JSP |
TPS |
Mean time |
Hits |
Err |
% err |
|
1 |
2 |
3 |
10 |
3.86 |
13.5 |
184 |
0 |
0 |
|
2 |
10 |
6 |
40 |
19.3 |
14.5 |
918 |
0 |
0 |
|
3 |
20 |
10 |
65 |
38.1 |
21.7 |
1813 |
0 |
0 |
|
4 |
30 |
25 |
82 |
56.9 |
24.1 |
2713 |
0 |
0 |
|
5 |
40 |
30 |
100 |
71.1 |
59.4 |
3421 |
3 |
0.09 |
|
6 |
50 |
35 |
100 |
70.9 |
141 |
3317 |
146 |
4.40 |
Main test results: between 30 and 40 users container reached 100 CPU. This results in sudden jump in response time and appearance of connection failures.
WAST is Microsoft software running only on Win-NT/2000/XP. All settings are done from user-friendly GUI. Tests were conducted in two modes:
“No” == no delay between requests
“500” == random delay in 0-1000 ms range
Test duration is 30 after 10 seconds of warm-up.
WAST reports time to first returning byte (“First” column) and time to last byte (“Last” column).
Some tests were repeated to determine repeatability (no formal statistical analysis).
WAST has small CPU demand in 10-30% producing high amount of threads and requests.
WAST and Container-T
|
# |
Threads |
Delay |
CPU-T |
CPU-JSP |
TPS |
First |
Last |
Hits |
Err |
% err |
|
4 |
1 |
No |
20 |
100 |
83 |
5.1 |
10.7 |
2472 |
0 |
0 |
|
3 |
2 |
No |
20 |
100 |
81 |
17.7 |
23.2 |
2439 |
0 |
0 |
|
2 |
5 |
No |
20 |
100 |
84 |
53 |
58 |
2523 |
0 |
0 |
|
1 |
10 |
No |
20 |
100 |
83 |
66 |
72 |
2482 |
60 |
2.42 |
|
13 |
10 |
No |
24 |
100 |
86.15 |
63.04 |
68.53 |
2580 |
89 |
3.45 |
|
5 |
10 |
500 |
14 |
30 |
17.3 |
23.16 |
29.14 |
519 |
1 |
0.19 |
|
8 |
10 |
500 |
8 |
30 |
18.31 |
5.40 |
10.39 |
550 |
0 |
0 |
|
6 |
20 |
500 |
10 |
55 |
38 |
9.53 |
15.42 |
1155 |
0 |
0 |
|
7 |
20 |
500 |
10 |
55 |
39 |
8.64 |
14.49 |
1174 |
0 |
0 |
|
9 |
30 |
500 |
20 |
70 |
55.72 |
13.45 |
19.30 |
1674 |
0 |
0 |
|
10 |
40 |
500 |
22 |
80 |
62.98 |
19.58 |
25.4 |
1892 |
12 |
0.63 |
|
11 |
50 |
500 |
22 |
88 |
67.87 |
24.57 |
30.52 |
2039 |
156 |
7.65 |
|
12 |
60 |
500 |
25 |
90 |
69.60 |
25.61 |
31.39 |
2091 |
368 |
17.60 |
Main test results: Connection failures start between 30 and 40 users. TPS levels off at 65 after 40 users. CPU level at the same user range at 80-90%.
WAST and Container-R
|
# |
Threads |
Delay |
CPU-T |
CPU-JSP |
TPS |
First |
Last |
Hits |
Err |
% err |
|
4 |
1 |
no |
25 |
98 |
240 |
1.57 |
3.10 |
7219 |
0 |
0 |
|
3 |
2 |
no |
28 |
100 |
244 |
5.40 |
7.04 |
7334 |
0 |
0 |
|
2 |
5 |
no |
30 |
100 |
240 |
18.17 |
19.60 |
7205 |
0 |
0 |
|
1 |
10 |
no |
27 |
100 |
222 |
25.94 |
27.71 |
6668 |
3 |
0.05 |
|
5 |
10 |
500 |
0 |
20 |
20 |
1.59 |
2.73 |
588 |
0 |
0 |
|
6 |
40 |
500 |
1 |
45 |
76.5 |
2.98 |
4.21 |
2300 |
0 |
0 |
|
7 |
70 |
500 |
10 |
70 |
137.5 |
6.12 |
7.42 |
4143 |
0 |
0 |
|
11 |
80 |
500 |
15 |
75 |
156 |
6.93 |
8.27 |
4696 |
0 |
0 |
|
10 |
90 |
500 |
15 |
80 |
173 |
10.09 |
11.45 |
5200 |
2 |
0.04 |
|
8 |
100 |
500 |
15 |
87 |
141 |
85.7 |
96.5 |
4291 |
191 |
4.45 |
|
9 |
120 |
500 |
18 |
90 |
133 |
168.4 |
171 |
3997 |
620 |
15.51 |
Main test results: Connection failures, CPU and TPS indicate 90 users and 150-170 TPS as practical limit. In strictly subsequent access mode (1 user, no delay) container manages 240 TPS with 3 ms per transaction!
WAST and Container-O
|
# |
Threads |
Delay |
CPU-T |
CPU-JSP |
TPS |
First |
Last |
Hits |
Err |
% err |
|
4 |
1 |
No |
20 |
100 |
84.08 |
1.43 |
10.57 |
2526 |
0 |
0 |
|
3 |
2 |
No |
22 |
100 |
84.31 |
13.25 |
22 |
2533 |
0 |
0 |
|
2 |
5 |
No |
23 |
100 |
84.18 |
48 |
58 |
2525 |
0 |
0 |
|
1 |
10 |
No |
20 |
100 |
73.49 |
79 |
95 |
2208 |
0 |
0 |
|
5 |
10 |
500 |
12 |
25 |
19.34 |
1.54 |
10.79 |
581 |
0 |
0 |
|
6 |
30 |
500 |
30 |
75 |
57.35 |
13.16 |
22.4 |
1723 |
0 |
0 |
|
9 |
40 |
500 |
21 |
90 |
71.80 |
36 |
47.02 |
2157 |
0 |
0 |
|
7 |
50 |
500 |
30 |
100 |
80.02 |
66.36 |
76.02 |
2404 |
4 |
0.17 |
|
10 |
60 |
500 |
20 |
100 |
75.69 |
119.5 |
131.5 |
2274 |
122 |
5.37 |
|
8 |
70 |
500 |
22 |
100 |
77.32 |
134 |
146 |
2336 |
303 |
12.97 |
Main test results: Connection failures, CPU and TPS indicate 45 users and 75-80 TPS as practical limit. In comparison with other containers, there is no failures at “10 users no delay” test.
JMeter is pure java package from Jakarta. No direct TPS and errors statistics were found by brief examination of this package. The package design is very modular, so appropriate plugins possibly exists.
Gaussian random timer was used with 500 ms offset and 300 ms deviation. JMeter has average appetite for CPU. It was OK with Container-T, but can be a problem with faster containers.
JMeter and Container-T.
|
# |
Threads |
CPU-T |
CPU-JSP |
Average |
|
1 |
10 |
20 |
30 |
16 |
|
6 |
20 |
30 |
65 |
29 |
|
8 |
20 |
- |
- |
22 |
|
2 |
30 |
50 |
80 |
46 |
|
5 |
30 |
50 |
80 |
38 |
|
7 |
40 |
45 |
85 |
65 |
|
3 |
50 |
80 |
90 |
77 |
|
4 |
70 |
80 |
90 |
78 |
Main test results: nothing.
StressTool is simple command line java program with rather straightforward properties file. In opposite to other packages, StressTool requires single timing setting: “Hits per seconds”. Amount of used threads is unknown. Row test results show huge distribution of response times from 10 ms to 1500 ms. StressTool does not consume CPU, but also unable to load other computer.
StressTool and Container-T
|
# |
Hit/sec |
CPU-T |
CPU-JSP |
Average |
TPS |
Hits |
Err |
% err |
|
1 |
10 |
8 |
15 |
184 |
9.73 |
389 |
5 |
1.25 |
|
2 |
20 |
12 |
40 |
232 |
19.33 |
773 |
26 |
3.36 |
|
5 |
30 |
14 |
40 |
268 |
21.68 |
867 |
43 |
4.96 |
|
6 |
40 |
14 |
40 |
294 |
22 |
880 |
37 |
4.20 |
|
3 |
50 |
14 |
40 |
422 |
24 |
960 |
160 |
16.7 |
|
4 |
70 |
20 |
40 |
419 |
30 |
1200 |
292 |
24.3 |
StressTool and Container-R
|
# |
Hit/sec |
CPU-T |
CPU-JSP |
Average |
TPS |
Hits |
Err |
% err |
|
6 |
10 |
7 |
- |
251 |
8.98 |
359 |
0 |
0 |
|
4 |
20 |
10 |
- |
385 |
17.85 |
714 |
8 |
1.18 |
|
2 |
30 |
14 |
30 |
381 |
21.93 |
877 |
18 |
2.05 |
|
3 |
50 |
15 |
- |
331 |
24 |
960 |
6 |
0.63 |
|
1 |
60 |
18 |
45 |
489 |
28.95 |
1158 |
73 |
6.30 |
|
5 |
70 |
18 |
30 |
440 |
30 |
1200 |
45 |
3.75 |
Commercial product JBlitz is a pure java package. All setting are done from GUI. Evaluation copy allows upto 5 users (threads). Time setting was No (zero) delay between users requests. JBlitz reports time for establish connections (“Connect” column) and for downloading (“Download” column). JBlitz takes a lot of CPU time. In case of fast Container-R JBlitz itself becomes a bottleneck, as seen from CPU-T and CPU-JSP.
JBlitz and Container-T
|
# |
Threads |
CPU-T |
CPU-JSP |
TPS |
Errors |
Connect |
Download |
total |
|
1 |
1 |
30 |
98 |
51.28 |
0 |
1 |
18 |
19 |
|
2 |
2 |
50 |
97 |
78.01 |
0 |
1 |
23 |
24 |
|
3 |
3 |
55 |
100 |
89.16 |
0 |
1 |
31 |
32 |
|
4 |
4 |
55 |
100 |
104.59 |
0 |
1 |
36 |
37 |
|
5 |
5 |
60 |
100 |
73.52 |
0 |
1 |
66 |
66 |
Main test results: 80-100 TPS limit with subsequent load. Connect time with 1 ms shows that network speed has no, or minimal effect on results.
JBlitz and Container-R
|
# |
Threads |
CPU-T |
CPU-JSP |
TPS |
Errors |
Connect |
Download |
total |
|
1 |
1 |
65 |
70 |
149.58 |
0 |
1 |
5 |
6 |
|
2 |
2 |
80 |
75 |
169.87 |
0 |
3 |
8 |
12 |
|
3 |
3 |
90 |
80 |
180.97 |
0 |
5 |
10 |
15 |
|
4 |
4 |
93 |
80 |
186.8 |
0 |
6 |
14 |
20 |
|
5 |
5 |
96 |
80 |
185.45 |
0 |
8 |
14 |
22 |
Main test results: none because of tester CPU consumption.
OpenLoad is non-java free package. This command line program takes only one setting – number of threads. For both tested containers OpenLoad shows unusual high response times > 200 ms. Also high CPU consumption was noted. Brief local test with OpenLoad in the computer “A” (where JSP container) shows reasonable response times in 10-20 ms. Thus, the way OpenLoad uses network causes unusual 200ms delays.
OpenLoad and Container-T
|
# |
Threads |
CPU-T |
CPU-JSP |
TPS |
Errors |
Time |
|
1 |
1 |
5 |
20 |
4.95 |
0 |
199 |
|
2 |
5 |
50 |
55 |
23.85 |
0 |
206 |
|
3 |
10 |
65 |
80 |
35.02 |
0 |
275 |
|
4 |
20 |
100 |
90 |
33.49 |
0 |
282 |
OpenLoad and Container-R
|
# |
Threads |
CPU-T |
CPU-JSP |
TPS |
Errors |
total |
|
1 |
1 |
20 |
3 |
4.89 |
0 |
200 |
|
4 |
20 |
100 |
45 |
73.39 |
0 |
225 |
Tools summary table
|
Name |
CPU demand |
Flexibility of settings |
Amount of results |
User friendly |
Main problem |
|
Grinder |
small |
average |
average |
GUI + xml |
Time settings |
|
WAST |
Very small |
high |
high |
Good GUI |
Windows only |
|
StressTool |
n/a |
low |
average |
Cmd line |
Inconsistent results |
|
JMeter |
average |
high |
low |
GUI |
Less reporting |
|
JBlitz |
high |
high |
high |
Good GUI |
CPU demand |
|
OpenLoad |
high |
low |
average |
Cmd line |
Network problems |
Considering their questionable or uninformative results, StressTool, OpenLoad and JMeter were omitted from further discussion. Main findings were pulled off from rest of the raw results and summarized in the table.
Results summary for selected tools
|
Tester |
Property |
T |
R |
O |
|
|
|
|
|
|
|
Grinder |
Min time (2 users 500 ms delay) |
13.5 |
|
|
|
|
Max users (500 ms delay) |
35 |
|
|
|
|
Max TPS (max users, 500 ms delay) |
70 |
|
|
|
|
|
|
|
|
|
WAST |
Min time (1 user, no delay) |
10.7 |
2.73 |
10.57 |
|
|
Max users (500 ms delay) |
40 |
90 |
45 |
|
|
Max TPS (max users with 500 ms delay) |
65 |
170 |
80 |
|
|
Max TPS (1-5 users no delay) |
82 |
240 |
84 |
|
|
(time to first byte) – (time to last byte) |
5 |
1.4 |
10 |
|
|
|
|
|
|
|
JBlitz |
Min time (1 user, no delay) |
19 |
6 |
|
|
|
Max TPS (2-5 users no delay) |
90 |
>=180 |
|
Among this set of tools, for this particular test design, WAST is the best stress load software: free, user-friendly GUI with flexible settings, powerful load with minimal CPU consumption, informative result statistics. Pure java (=multi-platform) Grinder requires xml coding of settings and does the job. Evaluation version of JBlitz produces meaningful results providing enough CPU power.
Main limiting factor (in this test) for web container speed appeared to be CPU consumption. In real systems it can be also database speed, or remote invocations, or RAM.
AlexV. (Montreal 2002)