:::: MENU ::::
Browsing posts in: Platforms

Try/Catch for SQL!?

Thanks to Nick Sertis for this trick – who knew TSQL could do try/catch statements!

Very useful when you need to write data manipulation scripts for production databases.



        --Some SQL


-- Catch the errors on the inserts



Functions with side effects are just rude!

Today I fell into a trap when using a function that had a side effect – it unexpectedly changed an input parameter; causing a later statement to fail. Debugging took an age!

For example, consider the following function:

      string StringReplace(string haystack, string needle)

If this function is side-effect free, we can use it without fear like this:

        string menagerie = "cat,dog,bee,llama";
        string catFreeMenagerie = StringReplace(menagerie, "cat");
        string beeFreeMengerie = StringReplace(menagerie, "eric");

        Assert.AreEqual(",dog,fish,llama", catFreeMenagerie);
        Assert.AreEqual("cat,dog,,llama", beeFreeMengerie);

However, if StringReplace() had the side effect of also changing the passed in haystack, then the second Assert would fail, because the first StringReplace has the unexpected side effect of changing one of its arguments.

Evans in the DDD book has quite a bit to say about this; arguing that having side effect free functions goes a long way towards making a supple design

Side effect free functions also make testing & refactoring easier (less state to worry about etc)

Remember, a function that changes its parameters is rude, and should not be trusted!

PS: Eric the half a bee lyrics

Selenium gotcha – selenium.GetHtmlSource() returns processed HTML

Whilst writing some Selenium based acceptance tests today; I bumped into a hair pulling gotcha.  Hopefully this post will prevent you from the same pain.

The test was to check whether some tracking tag javascript was being inserted into the page correctly or not.

I assumed that I could get the page source as it was being delivered to the browser by calling selenium.GetHtmlSource(); and then check that for the javascript string I was expected.

Unfortunately, GetHtmlSource is just a proxy for the browsers DOM.InnerHTML method; and that returns the Html after it has been preprocessed by the browser.

Turns out that preprocessing does a couple of funky things, including

  • Changing line-endings (Firefox)
  • Changing capitalization (IE6)
  • Seemingly random removal / insertion of ” & ‘  (IE6)

So, when I was expecting a string like this:

   var amPid = '206'';
   var amPPid = '4803';
   if (document.location.protocol=='https:')

IE6 was presenting me with:

   var amPid = '206'';
   var amPPid = '4803';
   if (document.location.protocol=='https:')

A possible solution is to ignore case, whitespace and quotes when doing the comparison, with a helper method like this:

        /// Use this to compare strings to those returned from selenium.GetHtmlSource for an Internet Explore instance
        /// (IE6 seems to change case and inclusion of quotes, especially for Javascript.?)
        private static void AssertStringContainsIgnoreCaseWhiteSpaceAndQuotes(string expected, string actual)
            string expectedClean = Regex.Replace(expected, @"s", "").ToLower().Replace(""","").Replace("'","");
            string actualClean = Regex.Replace(actual, @"s", "").ToLower().Replace(""", "").Replace("'", "");
                                  string.Format("Expected string nn{0} nnis not contained within nn{1}", expected, actual));

It was the line endings that really floored me; because they were automatically normalized/corrected by my test runner when displaying the error. Aaargh!

Apache2 on Ubuntu 8.04LTS; restrict access to PAM authenticated users

I have a couple of static pages that I want to restrict access to.

I don’t want to manage another set of usernames & passwds, so I’d like apache2 to authenticate off the standard users on my system, via PAM.

To get this to work, you need to install and configure mod_auth_pam and mod_auth_shadow

aptitude install libapache2-mod-auth-pam libapache2-mod-auth-shadow

Ensure the www-data user is part of the shadow group, so apache2 can read the passwords

usermod -G shadow www-data

And set up the relevent virtual host:


                AuthPAM_Enabled On
                AuthShadow on
                AuthPAM_FallThrough Off
                AuthBasicAuthoritative Off
                AuthType Basic
                AuthName "Restricted to group: sysadmins"
                AuthUserFile /dev/null
                Require group sysadmins

Restart apache, and you’re done!

Self Cert SSL certificate for Apache2 on Ubuntu 8.04LTS

Generate a self cert certificate:


Create a new virtual host (you can only have one SSL virtual host / IP)

sudo cp /etc/apache2/sites-available/default /etc/apache2/sites-available/ssl

Edit ssl sothat it looks like this:
NameVirtualHost *:443

ServerName webangle-www1.everyangle.co.uk
ServerAdmin [email protected]

DocumentRoot /var/www/

SSLEngine on

SSLOptions +StrictRequire

SSLCertificateFile /etc/ssl/certs/server.crt
SSLCertificateKeyFile /etc/ssl/private/server.key

Finally, if you want to force redirect of all traffic to a certain folder via SSL (e.g, /phpmyadmin), add the following to /etc/apache2/sites-available/default

#Redirect traffic to /phpmyadmin through https
        RewriteEngine   on
        RewriteCond     %{SERVER_PORT} ^80$
        RewriteRule     ^/phpmyadmin(.*)$ https://%{SERVER_NAME}/phpmyadmin$1 [L,R]

Enable it:

sudo a2ensite ssl
sudo /etc/init.d/apache2 reload

Automount remote filesystem over SSH

Previously I posted on how I backup my server’s data to rsync.net’s remote storage.

A convienient way to access that remote storage is to configure rsync over sshfs:

sudo aptitude install sshfs
mkdir /mnt/sshfs
mkdir /mnt/sshfs/rsync.net
sshfs **username**@ch-s011.rsync.net: /mnt/rsync.net
Now, test that you can access /mnt/rsync.net, and copy a few files to your remote storage.  if all works well, the next step is to have sshfs automatically connect whenever we try to access the directory

First, unmount

fusermount -u /mnt/rsync.net

Then, install autofs, and edit the config file

sudo aptitude install autofs
sudo vi /etc/auto.master

Add the following line 

/mnt/sshfs /etc/auto.sshfs --timeout=30,--ghost


sudo vi /etc/auto.sshfs


rsync.net -fstype=fuse,rw,nodev,nonempty,noatime,allow_other,max_read=65536 :sshfs#**username**@ch-s011.rsync.net:


And finally restart autofs 

sudo /etc/init.d/autofs restart


Now, when you cd /mnt/sshfs/rsync.net, after a short delay you will automatically be connected to the remote filesystem over SSH.  After 30 seconds of inactivity, the connection will be closed.

Backup Ubuntu 8.04LTS to rsync.net using backup-manager (at linode.com)

I’m setting up a new linode360 VPS, based of the Ubuntu 8.04LTS image.

For backups, I want to do weekly backups and daily incrementals of the data files, and sync these off to an external backup location.

Broadly, there are two parts to the backup, creating the backed up files, and then copying them offsite.

Creating the backups

I’m using backup-manager 0.7.6-debian1, which handles backing up sets of files and MySQL databases to tar.gz files.

sudo aptitude install backup-manager
sudo /usr/sbin/backup-manager --version

The comments in the config file make editing it quite straight forward.

sudo vi /etc/backup-manager.conf

One minor points:

  • Separate multiple backup methods with a space, eg:
    export BM_ARCHIVE_METHOD="tarball-incremental mysql"

To test:

sudo /usr/sbin/backup-manager --verbose

The output folder you specified (/var/archives) should now contain some .tar.gz versions of your data. Hurrah!

Getting the files offsite

Originally I intended to use Amazon’s S3 as a backup store, following Michael Zehrer’s instructions on how to rsync with S3. However, I couldn’t get this to work reliably; so I opted instead for rsync.net which offers standard scp, ftp, WebDav and sshfs access to their geographic backup locations.

Backup-manager can rsync over ssh, which is a quick and efficient way to sync changes over to the remote host..

The first step is get your rsync.net account setup; and set up your ssh so you can access without typing in a password

Then, set the BM_UPLOAD_METHOD to rsync, and configure both the scp and the rsync settings in /etc/backup-manager.conf (pay attention not to prefix remote folders with / ).

Test with:

sudo /usr/sbin/backup-manager --verbose

Once its all working, set up a cron job to call backup-manager daily.

crontab -e

I run backup-manager once per day in the wee hours, and log output to /root/crontab/daily_backup-manager.logs

  0 3   *   *   *    /usr/sbin/backup-manager -v > /root/cronlogs/daily_backup-manager.log


ASP.NET MVC Beta – Setting properties on ViewControls

In ASP.NET MVC Beta, it isn’t possible to set properties on partials when calling them with Html.RenderPartial.

Rusty Zarse blogged about a useful ViewData helper class, which allows you to set properties by passing values to the partial through the ViewData.

I’ve extended this slightly to enable the following syntax:

Which sets properties on a ViewUserControl like this:

     public partial class YUIDataTable : ViewUserControl
        public string ConfigNamespace { get; set; }
        public string DataTableId { get; set; }
        public bool HideFilter { get; set; }

        protected void Page_Load(object sender, EventArgs e)

Here is the full helper code.

using System;

namespace MvcHelpers
    /// With thanks to http://www.vitaminzproductions.com/technology-blog/index.php/2008/11/12/setting-properties-using-aspnet-mvc/
    public static class ViewDataDictionaryBuilder
        public static System.Web.Mvc.ViewDataDictionary Create(object data, ModelType model) where ModelType : class
            return (System.Web.Mvc.ViewDataDictionary)CreateInternal(new System.Web.Mvc.ViewDataDictionary(model), data);

        public static System.Web.Mvc.ViewDataDictionary Create(object data, object model)
            return CreateInternal(new System.Web.Mvc.ViewDataDictionary(model), data);

        public static System.Web.Mvc.ViewDataDictionary Create(object data)
            return CreateInternal(new System.Web.Mvc.ViewDataDictionary(), data);

        private static System.Web.Mvc.ViewDataDictionary CreateInternal(System.Web.Mvc.ViewDataDictionary dictionary, object data)
            AddPropertiesToViewData(dictionary, data);
            return dictionary;

        private static void AddPropertiesToViewData(System.Web.Mvc.ViewDataDictionary dictionary, object data)
            if (data == null) return;

            System.Reflection.PropertyInfo[] properties = data.GetType().GetProperties();

            foreach (var property in properties)
                dictionary.Add(property.Name, property.GetValue(data, null));

        public static void SetPropertiesToViewDataValues(System.Web.Mvc.ViewUserControl viewUserControl)
            foreach (var property in viewUserControl.GetType().GetProperties())
                if (viewUserControl.ViewData[property.Name] != null)
                    property.SetValue(viewUserControl, Convert.ChangeType(viewUserControl.ViewData[property.Name], property.PropertyType), null);


Hope that’s useful to you!

Announcing the TDD TestHelpers opensource project

Whenever I start working on a project; I invariably find myself writing a collection of TDD test helper methods.  I quick survey of other TDDers reveals the same; and thus the birth of my latest opensource project, TestHelpers (http://code.google.com/p/testhelpers/).

The aim of the project is to centralise all those little test helper methods you end up creating into a useful assembly you can use to jumpstart your next project.  Things like:

  • Comparers
    • Generic object comparers
    • DataSet comparers
  • Test Data generators
    • Builder pattern
  • Automocking containers

For example, I’ve just added an “AssertValues” functor; which helps you check whether the values of who object instances are the same. 

One area I keep using asserts like this is in integration tests; where I want to check that the objects I’m persisting to the database via my ORM actually end up in the database in a non-mangled form.  In this case, I new up entityA, persist it, reload it into entityB and then need to check that all the values in entityB are the same as those in entityA.

A standard Assert.AreEqual will fail, because entityA and entityB are different instances.  But, my helper method AssertValues.AreEqual will pass, because it checks the (serialized) string values of entityA and entityB.

Here is another, simpler example to illustrate the concept.

    public class StandardObjectsTests
        public class StringContainer
            public string String1 { get; set; }
            public string String2 { get; set; }

        public void ObjectsWithSameValue_ShouldBeEqual()
            var stringContainer1 = new StringContainer {String1 = "Test String1", String2 = "Test String 2"};
            var stringContainer2 = new StringContainer {String1 = "Test String1", String2 = "Test String 2"};

            Assert.AreNotEqual(stringContainer1, stringContainer2);

            AssertValues.AreEqual(stringContainer1, stringContainer2);

I’m sure you have a bunch of similar helper methods lying about your projects.

How about contributing them to the TestHelper project?