Testing test software.

1              Summary

Several popular test software packages were compared back-to-back in stress-loading JSP containers. Only half of tested packages produced meaningful results in short stress-load against simplest JSP page. Microsoft WAST and java-based Grinder and JBlitz consistently identified speed limits of web containers. CPU consumption was found to be limiting parameter both for web container and test software.

2              Hardware, basic software and network

Computer-A runs JSP containers. AMD K6-3D. 350 MHz, 320Mb ram, Windows ME (SE), Java HotSpot(TM) Client VM (build 1.4.1-rc-b19, mixed mode).

 

Computer-B runs testing software. Intel P-III 866 MHz, 512Mb ram, Windows XP (Pro), Java HotSpot(TM) Client VM (build 1.4.0-b92, mixed mode).

 

Network: 10/100 LAN with LinkSys Cable/Router. Internet was disconnected to prevent any distraction.

 

JSP containers are not primer focus of this presentation. Latest (as of Aug 28, 2002) development and beta versions of Container-T, Container-R and Container-O were downloaded and installed without any tuning.

3              Testing software

Name

Full name and version

Referrence

Grinder

Grinder 2.8.3

Grinder

WAST

Microsoft Web Application Stress Tool  1.1.293.1

http://webtool.rte.microsoft.com/

StressTool

Stress Tool 1.3

servletsuite

JMeter

JMeter 1.7

JMeter 1.7

JBlitz

JBlitz Pro 3.0 (eval)

JBlitz

OpenLoad

OpenLoad 0.1.2-Win32

OpenLoad

4              Test setup

Jsp containers were run on weaker computer “A” to be sure that testing software is not a limiting factor. Also WAST does not run on Win-ME. Computer “A” was restarted before every new container is tested.

 

Test was conducted against one page “include.jsp” from JSP examples supplied with Tomcat container. The page outputs current time and than dynamically include current time calculation from another page. Thus, there is no access to business layer or database.

 

The code was present in most containers or was copies in demo directory. Http port was set to 80, because one package (WAST) has no port setting. Test duration was in 30-40 seconds range. Where possible, average results exclude first 5-10 seconds.

 

In addition to various test results from load software, special attention was paid to CPU consumption on both computers. Real-time graphs from “system monitor” of Win-ME and “windows task manager” were used. “Visual-average” values were recorded. RAM was less than 50% during all tests in computer-B (load software). RAM on computer-A (jsp container) was not considered.

5              Test results

5.1             General notes

Common for all test are:

# - test number. Some testes were done not in order of threads increase to prevent possible “training” effect.

Threads – number of parallel processes; same as number of users

CPU-T– approximate average CPU consumption of test machine

CPU-JSP – approximate average CPU consumption of machine with JSP container

TPS – transactions per second

Hits – number of successful requests for time of test

Err – number of errors (Socket errors)

% Err == (Err * 100) / Hits

All times are in milliseconds, unless stated otherwise.

5.2             Grinder

Grinder is pure java software consisting of two parts: main console and any number of workers, possibly in several different machines. Workers must be set with xml file with number of parameters. The following time settings were used here:

 

grinder.thread.initialSleepTime=1000

grinder.thread.sleepTime=5000

grinder.thread.sleepTimeFactor=0.1

grinder.thread.sleepTimeVariation=0.005

 

Brief attempts to decrease time between requests caused huge increase in Socket failures. Grinder shows relatively small CPU demand producing high output load.

 

Grinder and Container-T

#

Threads

CPU-T

CPU-JSP

TPS

Mean time

Hits

Err

% err

1

2

3

10

3.86

13.5

184

0

0

2

10

6

40

19.3

14.5

918

0

0

3

20

10

65

38.1

21.7

1813

0

0

4

30

25

82

56.9

24.1

2713

0

0

5

40

30

100

71.1

59.4

3421

3

0.09

6

50

35

100

70.9

141

3317

146

4.40

 

Main test results: between 30 and 40 users container reached 100 CPU. This results in sudden jump in response time and appearance of connection failures.

5.3             WAST

WAST is Microsoft software running only on Win-NT/2000/XP. All settings are done from user-friendly GUI. Tests were conducted in two modes:

“No” == no delay between requests

“500” == random delay in 0-1000 ms range

Test duration is 30 after 10 seconds of warm-up.

WAST reports time to first returning byte (“First” column) and time to last byte (“Last” column).

Some tests were repeated to determine repeatability (no formal statistical analysis).

 

WAST has small CPU demand in 10-30% producing high amount of threads and requests.

 

WAST and Container-T 

#

Threads

Delay

CPU-T

CPU-JSP

TPS

First

Last

Hits

Err

% err

4

1

No

20

100

83

5.1

10.7

2472

0

0

3

2

No

20

100

81

17.7

23.2

2439

0

0

2

5

No

20

100

84

53

58

2523

0

0

1

10

No

20

100

83

66

72

2482

60

2.42

13

10

No

24

100

86.15

63.04

68.53

2580

89

3.45

5

10

500

14

30

17.3

23.16

29.14

519

1

0.19

8

10

500

8

30

18.31

5.40

10.39

550

0

0

6

20

500

10

55

38

9.53

15.42

1155

0

0

7

20

500

10

55

39

8.64

14.49

1174

0

0

9

30

500

20

70

55.72

13.45

19.30

1674

0

0

10

40

500

22

80

62.98

19.58

25.4

1892

12

0.63

11

50

500

22

88

67.87

24.57

30.52

2039

156

7.65

12

60

500

25

90

69.60

25.61

31.39

2091

368

17.60

 

Main test results: Connection failures start between 30 and 40 users. TPS levels off at 65 after 40 users. CPU level at the same user range at 80-90%.

 

WAST and Container-R

#

Threads

Delay

CPU-T

CPU-JSP

TPS

First

Last

Hits

Err

% err

4

1

no

25

98

240

1.57

3.10

7219

0

0

3

2

no

28

100

244

5.40

7.04

7334

0

0

2

5

no

30

100

240

18.17

19.60

7205

0

0

1

10

no

27

100

222

25.94

27.71

6668

3

0.05

5

10

500

0

20

20

1.59

2.73

588

0

0

6

40

500

1

45

76.5

2.98

4.21

2300

0

0

7

70

500

10

70

137.5

6.12

7.42

4143

0

0

11

80

500

15

75

156

6.93

8.27

4696

0

0

10

90

500

15

80

173

10.09

11.45

5200

2

0.04

8

100

500

15

87

141

85.7

96.5

4291

191

4.45

9

120

500

18

90

133

168.4

171

3997

620

15.51

 

Main test results: Connection failures, CPU and TPS indicate 90 users and 150-170 TPS as practical limit. In strictly subsequent access mode (1 user, no delay) container manages 240 TPS with 3 ms per transaction!

 

WAST and Container-O

#

Threads

Delay

CPU-T

CPU-JSP

TPS

First

Last

Hits

Err

% err

4

1

No

20

100

84.08

1.43

10.57

2526

0

0

3

2

No

22

100

84.31

13.25

22

2533

0

0

2

5

No

23

100

84.18

48

58

2525

0

0

1

10

No

20

100

73.49

79

95

2208

0

0

5

10

500

12

25

19.34

1.54

10.79

581

0

0

6

30

500

30

75

57.35

13.16

22.4

1723

0

0

9

40

500

21

90

71.80

36

47.02

2157

0

0

7

50

500

30

100

80.02

66.36

76.02

2404

4

0.17

10

60

500

20

100

75.69

119.5

131.5

2274

122

5.37

8

70

500

22

100

77.32

134

146

2336

303

12.97

 

Main test results: Connection failures, CPU and TPS indicate 45 users and 75-80 TPS as practical limit. In comparison with other containers, there is no failures at “10 users no delay” test.

5.4             JMeter

JMeter is pure java package from Jakarta. No direct TPS and errors statistics were found by brief examination of this package. The package design is very modular, so appropriate plugins possibly exists.

 

Gaussian random timer was used with 500 ms offset and 300 ms deviation. JMeter has average appetite for CPU. It was OK with Container-T, but can be a problem with faster containers.

 

JMeter and Container-T.

#

Threads

CPU-T

CPU-JSP

Average

1

10

20

30

16

6

20

30

65

29

8

20

-

-

22

2

30

50

80

46

5

30

50

80

38

7

40

45

85

65

3

50

80

90

77

4

70

80

90

78

 

Main test results: nothing.

5.5             StressTool

StressTool is simple command line java program with rather straightforward properties file. In opposite to other packages, StressTool requires single timing setting: “Hits per seconds”. Amount of used threads is unknown. Row test results show huge distribution of response times from 10 ms to 1500 ms. StressTool does not consume CPU, but also unable to load other computer.

 

StressTool and Container-T

#

Hit/sec

CPU-T

CPU-JSP

Average

TPS

Hits

Err

% err

1

10

8

15

184

9.73

389

5

1.25

2

20

12

40

232

19.33

773

26

3.36

5

30

14

40

268

21.68

867

43

4.96

6

40

14

40

294

22

880

37

4.20

3

50

14

40

422

24

960

160

16.7

4

70

20

40

419

30

1200

292

24.3

 

StressTool and Container-R

#

Hit/sec

CPU-T

CPU-JSP

Average

TPS

Hits

Err

% err

6

10

7

-

251

8.98

359

0

0

4

20

10

-

385

17.85

714

8

1.18

2

30

14

30

381

21.93

877

18

2.05

3

50

15

-

331

24

960

6

0.63

1

60

18

45

489

28.95

1158

73

6.30

5

70

18

30

440

30

1200

45

3.75

 

5.6             JBlitz

Commercial product JBlitz is a pure java package. All setting are done from GUI. Evaluation copy allows upto 5 users (threads). Time setting was No (zero) delay between users requests. JBlitz reports time for establish connections (“Connect” column) and for downloading (“Download” column). JBlitz takes a lot of CPU time. In case of fast Container-R JBlitz itself becomes a bottleneck, as seen from CPU-T and CPU-JSP.

 

JBlitz and Container-T

#

Threads

CPU-T

CPU-JSP

TPS

Errors

Connect

Download

total

1

1

30

98

51.28

0

1

18

19

2

2

50

97

78.01

0

1

23

24

3

3

55

100

89.16

0

1

31

32

4

4

55

100

104.59

0

1

36

37

5

5

60

100

73.52

0

1

66

66

 

Main test results: 80-100 TPS limit with subsequent load. Connect time with 1 ms shows that network speed has no, or minimal effect on results.

 

JBlitz and Container-R

#

Threads

CPU-T

CPU-JSP

TPS

Errors

Connect

Download

total

1

1

65

70

149.58

0

1

5

6

2

2

80

75

169.87

0

3

8

12

3

3

90

80

180.97

0

5

10

15

4

4

93

80

186.8

0

6

14

20

5

5

96

80

185.45

0

8

14

22

 

Main test results: none because of tester CPU consumption.

5.7             OpenLoad

OpenLoad is non-java free package. This command line program takes only one setting – number of threads. For both tested containers OpenLoad shows unusual high response times > 200 ms. Also high CPU consumption was noted. Brief local test with OpenLoad in the computer “A” (where JSP container) shows reasonable response times in 10-20 ms. Thus, the way OpenLoad uses network causes unusual 200ms delays.

 

OpenLoad and Container-T

#

Threads

CPU-T

CPU-JSP

TPS

Errors

Time

1

1

5

20

4.95

0

199

2

5

50

55

23.85

0

206

3

10

65

80

35.02

0

275

4

20

100

90

33.49

0

282

 

OpenLoad and Container-R

#

Threads

CPU-T

CPU-JSP

TPS

Errors

total

1

1

20

3

4.89

0

200

4

20

100

45

73.39

0

225

 

6              Discussion

Tools summary table

Name

CPU

demand

Flexibility

of settings

Amount of results

User friendly

Main problem

Grinder

small

average

average

GUI + xml

Time settings

WAST

Very small

high

high

Good GUI

Windows only

StressTool

n/a

low

average

Cmd line

Inconsistent results

JMeter

average

high

low

GUI

Less reporting

JBlitz

high

high

high

Good GUI

CPU demand

OpenLoad

high

low

average

Cmd line

Network problems

 

Considering their questionable or uninformative results, StressTool, OpenLoad and JMeter were omitted from further discussion. Main findings were pulled off from rest of the raw results and summarized in the table.

 

Results summary for selected tools

Tester

Property

T

R

O

 

 

 

 

 

Grinder

Min time (2 users 500 ms delay)

13.5

 

 

 

Max users (500 ms delay)

35

 

 

 

Max TPS (max users, 500 ms delay)

70

 

 

 

 

 

 

 

WAST

Min time (1 user, no delay)

10.7     

2.73

10.57

 

Max users (500 ms delay)

40    

90

45

 

Max TPS (max users with 500 ms delay)

65     

170

80

 

Max TPS (1-5 users no delay)

82     

240

84

 

(time to first byte) – (time to last byte)

5

1.4

10

 

 

 

 

 

JBlitz

Min time (1 user, no delay)

19

6

 

 

Max TPS (2-5 users no delay)

90

>=180

 

 

7              Conclusion

 

Among this set of tools, for this particular test design, WAST is the best stress load software: free, user-friendly GUI with flexible settings, powerful load with minimal CPU consumption, informative result statistics. Pure java (=multi-platform) Grinder requires xml coding of settings and does the job. Evaluation version of JBlitz produces meaningful results providing enough CPU power.

 

Main limiting factor (in this test) for web container speed appeared to be CPU consumption. In real systems it can be also database speed, or remote invocations, or RAM.

 

AlexV. (Montreal 2002)