Configuring HttpArchive + Webpagetest (Part 3: HttpArchive)

It’s time to install HttpArchive! So just as with Webpagetest, there’s
some requirements for HttpArchive as well.

HttpArchive must run on a unix-based machine, as HttpArchive uses pcntl,
a threading function in PHP currently available only on unix-based
machines. For this guide I will be using Ubuntu.

The following will be needed on your machine:

  • Apache2+
  • PHP5 or above
  • MySQL
  • Subversion
  • pcntl (PHP)

Most of these can be installed with a package manager. However with
pnctl, one must manually download the source, and either configure PHP
with the pcntl argument, or compile and install the pcntl extension
manually. I found an Ubuntu forum post from skout23 that explains a very
easy way to install pcntl for Ubuntu users. However I’m sure aside from
the package manager, BSD based Linux users can do the exact same thing:
http://ubuntuforums.org/showthread.php?t=549953

Here’s the relevant code for Ubuntu users:

mkdir phpcd phpapt-get source php5cd php5-(WHATEVER_RELEASE)/ext/pcntlphpize./configuremake

And don’t forget to restart Apache afterward!

Once everything is configured properly, you can checkout the HttpArchive
source from the googlecode repository:“ “

http://httparchive.googlecode.com/svn/trunk/

In addition, unless you want to download the downloads folder (which
contains over 1GB of data from the sites that HttpArchive tracks), it
would be best to checkout non-recursively, then check out all other
folders:

$svn co -N http://httparchive.googlecode.com/svn/trunk/ .$cd trunk$svn up images$svn up bulktest

Next, we will modify the settings.inc folder with the following
information:

  • $gMysqlServer = “YOUR_SERVER”
  • $gMysqlDb = “YOUR_DATABASE”
  • $gMysqlUsername = “ACCOUNT_USERNAME”
  • $gMysqlPassword = “ACCOUNT_PASSWORD”

Finally, Apache needs to interpret the .js files with PHP before being
served to the user. There exists a directive inside the .htaccess file
in the root of the repository that already accounts for this. However, I
had issues with this particular part, so I had to add the directive into
the php.conf file under /etc/apache2/mods-enabled (you will need root
permissions to modify this file):

<Filesmatch "(filmstrip|harviewer|interesting|interesting-images).js">SetHandler application/x-httpd-php</FilesMatch>

And you’ve configured HttpArchive! Unfortunately it won’t really work
without any data, but we’ll talk about the final steps in part 4:
Configuring the two to work with each other!

Configuring HttpArchive + Webpagetest (Part 2: Webpagetest)

Welcome to part 2! This post discusses installing Webpagetest.org. In
order to do so, we will need:

  • A windows machine (XP or Windows 7 have been tested with this method)
  • Apache2.2 or higher
  • PHP5 or higher
  • IE of some sort (IE8 or 9 would be best)
  • ffmpeg

It is possible to split up the web server and the testing server, but I
put them both on the same machine for ease of use. In addition I found
installing and configuring Apache and PHP together on windows was
surprisingly difficult, so I suggest installing Xampp. It’s a
single-install program that includes many of the tools used for serving
web pages and web development, such as Apache, PHP5, MySQL, and
Filezilla. Windows 7 machines have IE 8 installed by default, but
upgrading is straightforward for XP machines.

Once you have your machine set up properly, it’s time to install
Webpagetest! You can download the source here: Webpagetest.org
source
.

There are also installation instructions on the webpagetest.org google
site. I would recommend following these fora a complete guide, but what
I have written is a shorter version and will attain the same result.

https://sites.google.com/a/webpagetest.org/docs/private-instances

Configure Apache to point to the www directory of your source, or move
the contents of the folder to the “htdocs” folder under Xampp (I found
that the Virtualhost directive in Apache was having issues, so I just
threw everything into the htdocs folder, where Xampp is initially
configured to point to). I found that on the windows machine, giving
read/write permissions to the directories needed was not an issue.

You will then have to configure everything properly. This involves
basically copying and pasting everything in the settings folder to it’s
non-sample equivalent. I was able to do this because I wanted a basic
instance, but keep in mind you may need to do more if you want more
complex options.

Configuring the system to run the tests is best explained, verbatim from
the private-instance setup site linked above:

  1. Configure the test system to automatically log-on to an administrator
    account. Running “control userpasswords2” from the start menu is one way
    to configure it.
  2. Disable any screen savers (the desktop needs to remain visible for the
    video capture to work)
  3. Disable UAC (Vista or later – slide to “never notify”)
  4. Uninstall IE Enhanced-Security Mode (Windows Server)
  5. Copy the test software from the agent folder to the system (to
    “c:\webpagetest” for this example)
  6. Install the DUMMYNET ipfw driver
    • Pull up the properties for the Network Adapter that is used to access
      the Internet
    • Click “Install”
    • Select “Service” and click “Add”
    • Click “Have Disk” and navigate to c:\webpagetest\dummynet
    • Select the ipfw+dummynet service (and click through any warnings
      about the driver being unsigned)
  7. Create a shortcut to c:\webpagetest\dummynet\ipfw.cmd in the startup
    folder
  8. Create a shortcut to c:\webpagetest\urlblast.exe in the startup folder
  9. Make a copy of the settings file (urlblast.ini) based on the sample
    • Give it the path to the server (default configuration points to a
      server on the local machine)
    • Configure the location to match the location defined on the server in
      locations.ini (if modified)
    • Configure the location key to match the server in locations.ini (if
      modified)
  10. Reboot to make sure everything starts up correctly

Note:On windows 7, ipfw will not properly install (it will not
show up under installable services). If you want to use windows 7, you
must add a “Location = LAN” directive under test in settings.inc in
settings:

[Test]Location = LAN

After that, your instance should be set up! Now that wasn’t so bad, was
it?

Next time we’ll talk about installing HttpArchive!

Configuring HttpArchive + Webpagetest (Part 1: Introduction)

Testing is an integral part of developing code, and this is usually a
lesson that is learned the hard way. I can’t name all the times that
I’ve made a trivial mistake that goes unnoticed until the code has
ballooned to the point where rooting around and figuring it out is an
impossibility. For even the best tester, it’s a rare occurrence that
every single possible exception is captured the first time around. On
top of this, certain kinds of tests are also less frequently even
considered. Among these are performance tests for web developers.
Figuring out how quickly each part of a website responds is not an easy
task, and is not a concern for must until it goes wrong. Luckily, there
exists a pair of tools that provide a great look at these performance
issues: HttpArchive and Webpagetest.

Webpagetest

Webpagetest is a tool for testing website performance. Originally a
piece of in-house AOL software, the site is now available as both a
public website, and open-source for users to download. It is an
amazing tool that provides various tools that measure website
performance, from a waterfall chart (of both resources accessed and
connections):

image0

to Google’s Pagespeed app (gives a rating out of 100 for page
performance):

image1

to actual screenshots of the page:

image2

I would give it a shot on your own site to see how amazing of a product
it really is. It is a comprehensive at look at a site, and a timeline
for the time it takes to load, start to finish. Webpagetest by itself is
a great tool for debugging a slow site, but there is another level with
which one can examine the performance of a site: over time. If one could
take the information from Webpagetest, store it in a database of some
kind, and analyze the results, there would be another dimension through
which one can analyze where the site was, how it is now, and predict the
direction, performance wise, it is heading.

Well, that is exactly what HttpArchive is built to do!

HttpArchive

HttpArchive is another open-source project, meant to be similar to
archive.org, which snapshots various sites on the internet, providing
a look back into the past. HttpArchive archives the performance of
various websites, providing a different look back: at site
performance.Using Webpagetest as a backbone, HttpArchive saves snapshots
of information from webpagetest.org, and analyzes trends in various
websites. For example, take a look at it’s analysis of wordpress.com:

http://httparchive.org/viewsite.php?pageid=268424

The site is able to give insight that is interesting to both developers
and users alike!

Conclusion

The goal of this tutorial is to go over how to implement a private
instance of Webpagetest and HttpArchive, and how to link the pair
together. This will create a powerful and useful performance testing
suite.

Stay tuned for part 2, where we’ll go over installing and configuring
Webpagetest!

Part3:  going over installing HttpArchive.

Part4: putting them together and conclusion.