Configuring HttpArchive + Webpagetest (Part 2: Webpagetest)

Welcome to part 2! This post discusses installing Webpagetest.org. In
order to do so, we will need:

  • A windows machine (XP or Windows 7 have been tested with this method)
  • Apache2.2 or higher
  • PHP5 or higher
  • IE of some sort (IE8 or 9 would be best)
  • ffmpeg

It is possible to split up the web server and the testing server, but I
put them both on the same machine for ease of use. In addition I found
installing and configuring Apache and PHP together on windows was
surprisingly difficult, so I suggest installing Xampp. It’s a
single-install program that includes many of the tools used for serving
web pages and web development, such as Apache, PHP5, MySQL, and
Filezilla. Windows 7 machines have IE 8 installed by default, but
upgrading is straightforward for XP machines.

Once you have your machine set up properly, it’s time to install
Webpagetest! You can download the source here: Webpagetest.org
source
.

There are also installation instructions on the webpagetest.org google
site. I would recommend following these fora a complete guide, but what
I have written is a shorter version and will attain the same result.

https://sites.google.com/a/webpagetest.org/docs/private-instances

Configure Apache to point to the www directory of your source, or move
the contents of the folder to the “htdocs” folder under Xampp (I found
that the Virtualhost directive in Apache was having issues, so I just
threw everything into the htdocs folder, where Xampp is initially
configured to point to). I found that on the windows machine, giving
read/write permissions to the directories needed was not an issue.

You will then have to configure everything properly. This involves
basically copying and pasting everything in the settings folder to it’s
non-sample equivalent. I was able to do this because I wanted a basic
instance, but keep in mind you may need to do more if you want more
complex options.

Configuring the system to run the tests is best explained, verbatim from
the private-instance setup site linked above:

  1. Configure the test system to automatically log-on to an administrator
    account. Running “control userpasswords2” from the start menu is one way
    to configure it.
  2. Disable any screen savers (the desktop needs to remain visible for the
    video capture to work)
  3. Disable UAC (Vista or later – slide to “never notify”)
  4. Uninstall IE Enhanced-Security Mode (Windows Server)
  5. Copy the test software from the agent folder to the system (to
    “c:\webpagetest” for this example)
  6. Install the DUMMYNET ipfw driver
    • Pull up the properties for the Network Adapter that is used to access
      the Internet
    • Click “Install”
    • Select “Service” and click “Add”
    • Click “Have Disk” and navigate to c:\webpagetest\dummynet
    • Select the ipfw+dummynet service (and click through any warnings
      about the driver being unsigned)
  7. Create a shortcut to c:\webpagetest\dummynet\ipfw.cmd in the startup
    folder
  8. Create a shortcut to c:\webpagetest\urlblast.exe in the startup folder
  9. Make a copy of the settings file (urlblast.ini) based on the sample
    • Give it the path to the server (default configuration points to a
      server on the local machine)
    • Configure the location to match the location defined on the server in
      locations.ini (if modified)
    • Configure the location key to match the server in locations.ini (if
      modified)
  10. Reboot to make sure everything starts up correctly

Note:On windows 7, ipfw will not properly install (it will not
show up under installable services). If you want to use windows 7, you
must add a “Location = LAN” directive under test in settings.inc in
settings:

[Test]Location = LAN

After that, your instance should be set up! Now that wasn’t so bad, was
it?

Next time we’ll talk about installing HttpArchive!

Configuring HttpArchive + Webpagetest (Part 1: Introduction)

Testing is an integral part of developing code, and this is usually a
lesson that is learned the hard way. I can’t name all the times that
I’ve made a trivial mistake that goes unnoticed until the code has
ballooned to the point where rooting around and figuring it out is an
impossibility. For even the best tester, it’s a rare occurrence that
every single possible exception is captured the first time around. On
top of this, certain kinds of tests are also less frequently even
considered. Among these are performance tests for web developers.
Figuring out how quickly each part of a website responds is not an easy
task, and is not a concern for must until it goes wrong. Luckily, there
exists a pair of tools that provide a great look at these performance
issues: HttpArchive and Webpagetest.

Webpagetest

Webpagetest is a tool for testing website performance. Originally a
piece of in-house AOL software, the site is now available as both a
public website, and open-source for users to download. It is an
amazing tool that provides various tools that measure website
performance, from a waterfall chart (of both resources accessed and
connections):

image0

to Google’s Pagespeed app (gives a rating out of 100 for page
performance):

image1

to actual screenshots of the page:

image2

I would give it a shot on your own site to see how amazing of a product
it really is. It is a comprehensive at look at a site, and a timeline
for the time it takes to load, start to finish. Webpagetest by itself is
a great tool for debugging a slow site, but there is another level with
which one can examine the performance of a site: over time. If one could
take the information from Webpagetest, store it in a database of some
kind, and analyze the results, there would be another dimension through
which one can analyze where the site was, how it is now, and predict the
direction, performance wise, it is heading.

Well, that is exactly what HttpArchive is built to do!

HttpArchive

HttpArchive is another open-source project, meant to be similar to
archive.org, which snapshots various sites on the internet, providing
a look back into the past. HttpArchive archives the performance of
various websites, providing a different look back: at site
performance.Using Webpagetest as a backbone, HttpArchive saves snapshots
of information from webpagetest.org, and analyzes trends in various
websites. For example, take a look at it’s analysis of wordpress.com:

http://httparchive.org/viewsite.php?pageid=268424

The site is able to give insight that is interesting to both developers
and users alike!

Conclusion

The goal of this tutorial is to go over how to implement a private
instance of Webpagetest and HttpArchive, and how to link the pair
together. This will create a powerful and useful performance testing
suite.

Stay tuned for part 2, where we’ll go over installing and configuring
Webpagetest!

Part3:  going over installing HttpArchive.

Part4: putting them together and conclusion.

Port Forwarding To a VirtualBox VM

Posting this little tidbit for myself. Turns out VirtualBox has a lot of
interesting NAT options:

Nat Forwarding – VirtualBox

To forward a port from a host machine to a VM, all you need to do is
find VBoxManager (.exe at the end for windows) and type:

VBoxManage modifyvm "VM name" --natpf1 "guestssh,tcp,,2222,,22

And now you have forwarded port 2222 of your host machine to 22 of your
VM.

Would love to know how to delete these rules though.

EDIT: just found it, awesome: VBoxManage Command Reference

--natpf<1-N> delete <name>

Hello world!

Welcome to WordPress.com. After you read this, you should delete and write your own post, with a new title above. Or hit Add New on the left (of the admin dashboard) to start a fresh post.

Here are some suggestions for your first post.

  1. You can find new ideas for what to blog about by reading the Daily Post.
  2. Add PressThis to your browser. It creates a new blog post for you about any interesting  page you read on the web.
  3. Make some changes to this page, and then hit preview on the right. You can alway preview any post or edit you before you share it to the world.