I’ve been working on a project that requires handling a lot of data coming in from external sources – some of it in a delimited format (e.g. CSV) and some in XML format. When trying to decide how to handle the XML-formatted files, Mark suggested that I try using Hpricot instead of a more “traditional” XML parser like REXML. I really love how Hpricot handles traditional HTML documents, so having the same functionality for XML documents made my life much easier.

But, there was a catch. Given this sample XML file, finding the contents of the <link> tag wasn’t possible:

My first attempt to capture the URL (with XPath) wasn’t successful, it just returned an empty string:

As a test, I attempted to reconstruct the original document by using the to_html method:

Strange – it appeared that the parser made certain assumptions about the structure and validity of the document. Namely that the <link> tag could only contain attributes, not values. After digging deeper into the documentation, I found that there is an :xml option when parsing a document. The updated code looks like this (and returns the expected URL):

The moral? When parsing XML documents with Hpricot, you need to parse them as … well, XML.

So you botched a portion of your long-running data import (who hasn’t?) and you need to re-import that part. The problem is, the rest of your dataset relies on the auto-numbered primary key that ActiveRecord has already generated behind the scenes and you need re-use that ID. Not a problem:

Oops, that ID doesn’t look quite right! Let’s try a different way:

Much better – since ActiveRecord doesn’t allow setting the ID through mass-assignment, you’ll need to set it separately to make it stick.

I’ve been working on a project that requires loading a large dataset into MySQL so I’m using the acts_as_importable plugin that I created. This allows me to generate MySQL bulk insert statements instead of doing the inserts through ActiveRecord. Even though this approach was much faster (2 minutes instead of 30 for ~100k records), I still needed to speed things up a bit more. I ran across this post that addresses the performance of MySQL’s LOAD DATA INFILE statement, but I wasn’t sure if it would work for bulk inserts, so I gave it a shot.

Initial Settings & Benchmarks

I performed the benchmarking and tuning on my MacBook, so the numbers don’t actually refelect what I would see on a production server. Instead, it gave me a good baseline to compare against future benchmark results. Since I am using InnoDB tables, here are the initial settings for the options I wanted to change:

And here’s the benchmark when importing 100K records:

Not too bad, but if we extrapolate this across 10MM records it will take almost 2 hours to load the entire dataset!

Improving Performance

I took the recommendations I found and updated the MySQL configuration file:

After restarting the server I was ready to try the same benchmark again, but I was unable to access the database tables. I could connect to the database server but had issues when selecting from the table:

The solution (from the MySQL documentation) is to just remove the logfiles:

Once the logfiles were moved away, I could restart the server and have access to the tables.

Final Benchmarks

The result of the configuration changes was about a 70% speedup in loading my sample file:

Awesome!

When spec’ing out code for a Rails plugin I’m working on, I needed some methods that were only available as part of the current connection. Rather than create a connection, my first instinct was to start mocking:

Ouch, what am I testing here? This quickly took a turn for the worse – maybe creating a connection isn’t so bad to start with:

Much more succinct and I no longer have to worry about the internals of my implementation.

If the thought of typing “man grep” at the commandline doesn’t make you cringe, you might find options that you never knew existed:

I knew about the -n option, but -m came in handy today when I was trying to grab a slice of an import file that contained sequential IDs:

Since I knew the next record past what I was interested in, a simple grep would tell me the line number that I could parse with Ruby and then grab the correct number of lines:

Lesson: don’t be afraid to shell out.

Just a quick announcement – one of the recipes I submitted for the upcoming Advanced Rails Recipes book has been published in the current beta release. My 4-page tutorial steps you through the process of having your application respond to request that Rails doesn’t know about out of the box – in this case, audio files.

This is a concept that I pulled from the existing Lemurgene codebase, though the current PHP implementation isn’t quite as nice as the Rails version. Creating this same behavior in a Rails application relies on the config/initializers/mime_types.rb file along with the controller that’s used to serve up the audio files. First, the configuration:

And in the controller that the user hits to retrieve the file:

Obviously, this isn’t the complete version. Check out the book to see this recipe (and others) in their entirety.

While I was working on the caches_constants plugin for the upcoming Advanced Rails Recipes book, I had a bit of trouble adding a new method to the String class. Following some examples I found from the Rails source, I created a new module:

To hook it in as an instance method on the String class, I originally dropped this into the init.rb file for the plugin:

One final test, and I should have been ready to go (from ./script/console):

Strange. As a test, I re-pasted the include from init.rb into the console and tried again:

At this point it appeared there wasn’t anything wrong with my code, but for some reason the mix-in wasn’t included when run from inside the plugin initialization. My first fix was to drop the include code into the file that contained the module definition.

This produced the desired effect, but I wasn’t happy with the implementation. In a last-ditch effort, I decided to try using send to perform the mixin:

This functioned as expected and was consistent with how I was extending ActiveRecord::Base. To check out the full implementation, browse the source or install the plugin:

As James pointed out in the comments in my last post, Mocha has the ability to do parameter matching based on type. This is similar to the behavior of FlexMock that I discussed earlier and it’s available from SVN if you want to try it for yourself.

To grab the code and build the gem, head on over to the Mocha repository at RubyForge:

Now, let’s try it out by testing some basic code (yes, this is backwards – always test first):

Here’s the corresponding test:

Excellent! Thanks to James and the rest of the Mocha team for this one.

Ok, so I’m on somewhat of a “mocking” kick at the moment. I’m actually preparing a bit of material for an upcoming talk at the next NovaRUG meeting where I’ll be (briefly) talking about mocking libraries in Ruby.

A bit of history – I had switched to using Mocha after initially getting hooked on Jim Weirich’s FlexMock library back in January. This was the first time I had seen mocks used in Ruby and I was completely blown away by how powerful they were. This was definitely not something I was able to do in PHP.

Now that I’m actively comparing the two libraries, I’m noticing that the issues I previously had with FlexMock have disappeared with the newest version (0.6.2 as I write this). It’s still a bit verbose, but I’m now finding it useful to mock out methods based on the type of parameters that they receive instead of the actual value. For example, this is how you would mock out an ActiveRecord#find call with Mocha:

FlexMock allows you to be a little more liberal when matching against a method call:

This is something that Stuart Halloway mentioned during a talk he gave in Richmond a few months ago. I still like the syntax of Mocha, but this feature of FlexMock may be enough to bring me back.

While I was writing a simple URL validation plugin that checked the HTTP status of a resource, I needed to test some code that made heavy use of some Net::HTTP methods. Here is a simplified version of the code:

When writing tests I don’t need to test the HTTP methods, and I don’t want to access a resource that might not be available during my test run. So, how do I hit that code path that will make exists ‘true’? The solution here is to use Mocha to stub out the behavior that my code is using:

That works well for success, but what happens if URI.parse throws a URI::InvalidURIError?

I really like the concise syntax of Mocha as compared to other mocking libraries and the added bonus that it keeps the same type information for the instances that have stubbed methods. This is something that I struggled with for a while when trying to use FlexMock for this task.