Recently I registered a new domain, rootuser.ninja, for the purpose of testing AWS using the 12 month free tier offer. That gives you 750 hours per month to run a micro server, 1GB RAM, 8GB hard drive and 1 vCPU basically for free for a year. What’s not to like about that?! After the 12 months, each micro server is $0.012 per hour, that’s cheaper than a lot of web hosting and you get a dedicated server.
After spinning up my little VM with a CentOS 7 image I installed:
- Apache with all the wordpress pre-requisites
- MariaDB
- Postfix for SMTP
- Dovecot for IMAP
- Amavis for Email scanning
- Spamassassin for SPAM scanning
- Clamav for Email antivirus
- WordPress
- Webmail
After configuring all the software, importing my WordPress database and installing my themes and plugins, this little free VM from Amazon is basically a web hosting service in a box. I get Email with antispam and antivirus, a web-based Email interface, IMAP and SMTP Email for my mobile devices and website hosting with WordPress; so this little 1GB server is doing a lot of work.
WordPress isn’t exactly a light-weight option for hosting web pages. If I wanted raw speed I would probably install something like Nginx with static pages. But, I wanted to put this little server through its paces, WordPress is very popular and this is an extremely standard configuration that anyone can support.
The next thing I did was went looking for benchmarking tools. I’ve used Siege before, but it’s labor intensive to run a lot of scenarios and extract the data in a manageable format. What I found was LoadImpact.com. This is a basic web stress testing tool that generates good reports, lets you customize the scenario, and exports the data to CSV if you want to do custom reporting. They integrate with NewRelic, so you can overlap your server statistics if you’re a NewRelic customer. And, even better there’s a free tier where you can spin up test scenarios to see how the service works.
This is basically the stock install, with very little tuning done, here are the results with 25 virtual users:
As you can see, the wheels start to come off between 14 and 16 virtual users. Here’s vmstat output from the server:
| procs | ———————–memory———————- | —swap– | —–io—- | -system– | ——–cpu——– | |||||||||||
| r | b | swpd | free | inact | active | si | so | bi | bo | in | cs | us | sy | id | wa | st |
| 0 | 0 | 128724 | 77520 | 423532 | 423464 | 0 | 0 | 2 | 1 | 27 | 72 | 0 | 0 | 100 | 0 | 0 |
| 0 | 0 | 128724 | 77396 | 423532 | 423464 | 0 | 0 | 0 | 4 | 40 | 97 | 0 | 0 | 100 | 0 | 0 |
| 0 | 1 | 195804 | 60720 | 450960 | 412876 | 0 | 13416 | 23894 | 13423 | 1095 | 744 | 20 | 4 | 63 | 13 | 0 |
| 0 | 0 | 300532 | 62640 | 452568 | 394268 | 0 | 20946 | 42329 | 21000 | 1591 | 1239 | 23 | 6 | 46 | 25 | 0 |
| 0 | 0 | 300532 | 68908 | 435772 | 402976 | 0 | 0 | 2900 | 0 | 126 | 137 | 0 | 0 | 98 | 1 | 0 |
| 0 | 0 | 300532 | 156012 | 432580 | 323940 | 0 | 0 | 2718 | 0 | 497 | 479 | 20 | 3 | 77 | 1 | 0 |
What you see is the server going along, completely idle, then the virtual users start. Immediately, there’s pressure on memory and the kernel starts swapping out to disk, the “so” column. At the same time there’s some CPU usage, the “us” column, and some stress on the I/O subsystem, the “wa” column, but not a lot of processes blocked on I/O, the “b” column. This is when the server starts spinning up httpd processes to handle the requests. Once those processes are started, then it’s quiet again.
| procs | ———————–memory———————- | —swap– | —–io—- | -system– | ——–cpu——– | |||||||||||
| r | b | swpd | free | inact | active | si | so | bi | bo | in | cs | us | sy | id | wa | st |
| 0 | 0 | 300532 | 191748 | 433244 | 290536 | 0 | 0 | 474 | 13 | 95 | 156 | 0 | 1 | 99 | 0 | 0 |
| 1 | 0 | 300532 | 95320 | 433596 | 390152 | 0 | 0 | 166 | 1 | 499 | 296 | 22 | 3 | 75 | 0 | 0 |
| 1 | 0 | 377184 | 62916 | 417864 | 407452 | 0 | 19766 | 23860 | 19785 | 1521 | 1241 | 38 | 7 | 46 | 9 | 0 |
| 0 | 4 | 467204 | 71552 | 409468 | 410024 | 6 | 13569 | 39147 | 13578 | 1257 | 785 | 11 | 5 | 63 | 20 | 0 |
| 0 | 0 | 473548 | 104864 | 410744 | 380584 | 0 | 1269 | 12700 | 1270 | 574 | 394 | 10 | 1 | 81 | 8 | 0 |
| 0 | 0 | 473548 | 109700 | 412584 | 379204 | 0 | 0 | 467 | 9 | 111 | 190 | 0 | 1 | 98 | 1 | 0 |
| 0 | 0 | 473548 | 105012 | 411360 | 389924 | 0 | 0 | 1788 | 18 | 565 | 530 | 20 | 3 | 76 | 1 | 0 |
Then, as a few more virtual users startup, there’s another hit, a little larger this time. And another quiet period.
| procs | ———————–memory———————- | —swap– | —–io—- | -system– | ——–cpu——– | |||||||||||
| r | b | swpd | free | inact | active | si | so | bi | bo | in | cs | us | sy | id | wa | st |
| 1 | 0 | 473548 | 98528 | 411900 | 396088 | 0 | 0 | 942 | 0 | 481 | 299 | 19 | 2 | 78 | 0 | 0 |
| 0 | 0 | 496108 | 98340 | 420072 | 389376 | 6 | 4513 | 2642 | 4514 | 527 | 379 | 22 | 3 | 75 | 1 | 0 |
| 0 | 0 | 534600 | 84128 | 415984 | 400040 | 0 | 7698 | 2881 | 7711 | 971 | 509 | 40 | 3 | 53 | 4 | 0 |
| 0 | 52 | 601956 | 65720 | 406936 | 402948 | 491 | 21736 | 76078 | 21756 | 2012 | 870 | 6 | 27 | 41 | 26 | 0 |
| 0 | 10 | 750884 | 68616 | 405672 | 405504 | 436 | 24171 | 70228 | 24208 | 1949 | 1221 | 19 | 23 | 0 | 57 | 1 |
| 2 | 15 | 830092 | 74364 | 403444 | 403420 | 211 | 13290 | 64972 | 13294 | 1867 | 1024 | 20 | 7 | 0 | 72 | 1 |
| 6 | 4 | 918572 | 65196 | 408888 | 413156 | 952 | 18221 | 48684 | 18230 | 2651 | 1327 | 69 | 13 | 0 | 18 | 0 |
| 0 | 0 | 937712 | 69840 | 419668 | 366812 | 207 | 3874 | 22123 | 3885 | 1353 | 713 | 27 | 5 | 58 | 10 | 0 |
| 0 | 8 | 1045236 | 75040 | 391872 | 391388 | 458 | 21505 | 46228 | 21516 | 1573 | 935 | 16 | 7 | 49 | 28 | 0 |
| 6 | 3 | 1092932 | 59632 | 393496 | 410188 | 765 | 10330 | 60089 | 10331 | 1784 | 1112 | 24 | 7 | 0 | 68 | 0 |
| 1 | 2 | 992984 | 154652 | 350388 | 364660 | 14298 | 0 | 26733 | 11 | 1582 | 1501 | 31 | 4 | 51 | 14 | 0 |
| 1 | 4 | 971284 | 70608 | 389528 | 410608 | 4701 | 7465 | 19646 | 7475 | 1444 | 958 | 43 | 7 | 46 | 5 | 0 |
| 0 | 0 | 1002300 | 125788 | 382248 | 367688 | 1703 | 6318 | 54590 | 6332 | 1499 | 989 | 13 | 4 | 24 | 57 | 1 |
| 0 | 16 | 1069088 | 70420 | 403344 | 403080 | 3590 | 16073 | 59748 | 16212 | 2275 | 1373 | 31 | 12 | 3 | 53 | 1 |
| 0 | 38 | 1096192 | 70276 | 406416 | 407696 | 4713 | 14312 | 70625 | 14500 | 2055 | 1096 | 19 | 11 | 0 | 69 | 1 |
| 2 | 28 | 1155908 | 63692 | 419576 | 402056 | 5941 | 15580 | 66120 | 15596 | 2094 | 1472 | 13 | 9 | 0 | 78 | 1 |
| 0 | 25 | 1190732 | 76336 | 407212 | 407328 | 17626 | 27545 | 63867 | 27556 | 2726 | 2134 | 24 | 9 | 0 | 67 | 1 |
| 0 | 30 | 1201480 | 66744 | 414152 | 413876 | 17462 | 21536 | 64895 | 21564 | 3241 | 2028 | 39 | 8 | 0 | 52 | 1 |
| 0 | 26 | 1225468 | 69684 | 414936 | 410704 | 16846 | 16354 | 68164 | 16368 | 2887 | 1989 | 27 | 7 | 0 | 65 | 1 |
| 0 | 29 | 1201136 | 63584 | 420580 | 411332 | 11280 | 10859 | 67350 | 10869 | 2440 | 1660 | 11 | 17 | 0 | 71 | 1 |
Finally, there’s another big spike as more virtual users come online. But, now the server is in trouble. You can see the available memory has dropped into the 5 digits, without really recovering, and the server has started swapping IN from disk, the “si” column. There’s also a lot of I/O going on, and the disk can’t keep up. The CPU is being used, but it’s not busy. At this time, for 25 virtual users, Apache had started over 80 processes!
This is a pretty typical memory constrained system, which is then causing an issue with the disks because of swapping; the server just can’t feed the CPUs fast enough. So, the fix would be to add memory until we have the swapping under control, then see if there’s still an issue with the disk subsystem. We’re not going to do that because then we’ll be into the next server tier, so lets start by limiting the Apache processes.
The web server will be mostly idle, but the server will need to process Email 24×7. I’m sizing it to start a maximum of 20 processes. To limit the number of servers that Apache starts I’ve removed some modules that I won’t be using and added these lines to the httpd.conf file:
StartServers 3 MinSpareServers 3 MaxSpareServers 4 ServerLimit 20
This is very small. It says we’re planning on the Apache server being mostly idle, starting just 3 processes, and we’ll run a maximum of 20 server processes. We will pay a penalty when the Apache processes startup, but we’ll keep more memory free for other processes when it’s not needed.
Here are the results of the second run:
This is a much more linear result, and the numbers are much better as the number of virtual users ramps up past about 16. Here’s that the server was doing:
| procs | ———————–memory———————- | —swap– | —–io—- | -system– | ——–cpu——– | |||||||||||
| r | b | swpd | free | inact | active | si | so | bi | bo | in | cs | us | sy | id | wa | st |
| 0 | 0 | 43656 | 74440 | 505044 | 350028 | 0 | 0 | 29 | 1 | 47 | 113 | 0 | 0 | 100 | 0 | 0 |
| 0 | 0 | 43656 | 74440 | 505048 | 350028 | 0 | 0 | 0 | 0 | 48 | 111 | 0 | 0 | 100 | 0 | 0 |
| 0 | 0 | 64808 | 76588 | 490660 | 361252 | 0 | 4230 | 6077 | 4231 | 290 | 334 | 2 | 1 | 90 | 7 | 0 |
| 0 | 0 | 64808 | 76464 | 490728 | 361256 | 0 | 0 | 6 | 6 | 52 | 122 | 0 | 0 | 100 | 0 | 0 |
Here’s a bump similar to what we saw with the first run, but a bit smaller. After this there’s another second wave as the processes spin up, very similar to the first run. But, look at what the numbers look like toward the end of the run:
| procs | ———————–memory———————- | —swap– | —–io—- | -system– | ——–cpu——– | |||||||||||
| r | b | swpd | free | inact | active | si | so | bi | bo | in | cs | us | sy | id | wa | st |
| 0 | 0 | 794712 | 104832 | 419764 | 388220 | 6 | 2910 | 3136 | 2923 | 816 | 551 | 38 | 3 | 56 | 2 | 0 |
| 0 | 26 | 849428 | 84784 | 416532 | 411036 | 18 | 10948 | 70797 | 10958 | 2143 | 781 | 24 | 45 | 16 | 14 | 0 |
| 0 | 0 | 848928 | 117556 | 435740 | 360032 | 82 | 0 | 5038 | 21 | 1236 | 748 | 62 | 4 | 31 | 4 | 0 |
| 1 | 0 | 848616 | 88356 | 445936 | 378980 | 77 | 0 | 2366 | 0 | 461 | 338 | 11 | 1 | 87 | 1 | 0 |
| 0 | 0 | 848608 | 103152 | 448652 | 361544 | 6 | 0 | 606 | 10 | 625 | 328 | 30 | 1 | 68 | 0 | 0 |
| 0 | 0 | 847308 | 102752 | 448372 | 362116 | 261 | 0 | 294 | 0 | 184 | 212 | 1 | 0 | 98 | 1 | 0 |
| 0 | 0 | 847300 | 100744 | 448896 | 363704 | 0 | 0 | 105 | 9 | 475 | 301 | 20 | 1 | 79 | 0 | 0 |
| 0 | 0 | 847296 | 100744 | 448896 | 363704 | 0 | 0 | 2 | 12 | 115 | 158 | 0 | 0 | 100 | 0 | 0 |
| 4 | 0 | 847288 | 77224 | 448896 | 387736 | 0 | 0 | 0 | 1 | 284 | 208 | 12 | 1 | 87 | 0 | 0 |
| 0 | 0 | 847264 | 117996 | 427260 | 367728 | 0 | 0 | 0 | 17 | 1118 | 477 | 66 | 4 | 30 | 0 | 0 |
Where before we had tons of swapping in and out, and high I/O Wait numbers, the server now looks relatively quiet. CPU usage is relatively low, there’s low disk I/O overall, and a fair amount of free memory even if we are using 800MB of swap.
Now the memory workload is tuned to about the maximum the server can handle and there’s no constraint on CPU or Disk. Unfortunately our response time goes over 20 seconds at about the 14 virtual user mark, that’s pretty slow.
Next we’ll look at how improve the performance of WordPress.