Vermonster's Blog

Build-a-Gem Workshop Recap

2010-02-26T00:00:00-05:00

We had another Newer Yankee Workshop on Tuesday, following the success of our previous Cucumber session in our new office.

We started by reviewing gem basics – what it is, how it’s made, how it’d distributed, etc. Then we jumped into a simple hello-world project using the venerable Jeweler. Greg Sterndale then took the driver’s seat and walked us through “gemifying” an existing project (which happens to be a pretty useful library to talk to Nextbus, the provider of future MBTA bus data, among many other agencies).

We then paired up and started hacking our own gems. There were are few that used the Massachusetts RMV wait time feed (like Bouvier, by yours truly and Jay). I saw one team playing with Last.fm’s API and one that even ran python code (for some reason).

It was a good session and I’m looking a next one on a topic TBD.. Maybe using a NoSQL database, Rails 3, Rack-ifying a feature, or something else?

UPDATE: simple-twilio is another one by Pascal and Jeff

We Moved to 71 Broad St in Boston

2010-01-27T00:00:00-05:00

After 8ish years on Stuart St, we moved our office. We are still unpacking, setting things up and starting to settle in.

Our new office is an open floor space in what I believe is a building that once was used as a candle factory some 150+ years ago (yea, it’s that old). We will be posting pictures soon. In the meantime, update your address books:

Vermonster LLC
71 Broad St, 4th Floor
Boston, MA  02109

We have a new phone number too:

617-482-2020

Recommended Omnigraffle Guide for Rails ORM

2009-12-29T00:00:00-05:00

We recently developed a work-in-progress style guide for using Omnigraffle to describe Object Relationship Mappings (ORMs) in our Rails applications. We wanted to use simple, standard stencils while being able to exhaustively describe relationships. We also wanted to have them be fairly obvious to developers.

Has One / Belongs To

One of the more common relationships is belongs_to. We decided to show this as a line on the connector toward the belonging model. The sometimes-ambiguous sibling, has_one is shown as a block toward the belonging model. In both cases, we use the unfilled circle/square to show optional.

Has Many

The has_many relationship is shown as “crows feet” stretching toward the child model. We use the unfilled circle to show optional relationship. The fairly common combination of a belongs_to and has_many is shown as as a combination of the crows feet and line connector.

Through, HABTM and Polymorphic

The through relationship, or one that uses a joining table, is show shown as a block using the double -greater and -less than symbols <<through>>. Then we use an arrow to point at the joining model.

The has and belongs to many relationship is shows as double crows-feet with a perpendicular line on each end.

The polymorphic relationship is shown as a rectangle, like a model class, but the name of the relationship is shown in <brackets>. This lets us know the object is not an actual model.

And finally, a combination of all the relationships discussed here.

UPDATE: Jan 8, 2010 -- Changed the way through relationships are described. Swapped the belongs_to and has_many symbols to make room for the has_and_belongs_to_many (habtm) relationship.

Cucumber Workshop Recap

2009-12-16T00:00:00-05:00

Thanks to Greg Sterndale for running the show and Plectix for footing the food bill, we had a fantastic hands-on workshop last evening on Cucumber. The turnout was much better than expected – I think nearly 20 people attended!

Greg did a great job taking us through the Cucumber basics – walking through an example demonstrating pending and unimplemented features. Here are some take-aways and interesting conversation points.

Gem Bundler

We attempted to use modern tools and techniques for this project. The workshop project, called Code Stache, uses Rails 2.3.5 and the soon-to-be-de-facto gem bundler for gem specifying and inventorying (um, but it doesn’t work in ruby1.9). Gem bundler is fairly new, so it was not surprising that there was some discussion on this topic. The folks who use gem bundler seem to agree it is awesome but there are some missing features, including more usage instructions and visibility to what is actually going on during the bundle process.

After rake bundle, we were good to go. One important aspect (and feature) of gem bundler is any “binaries” that gems create (like cucumber or rspec) will end up in RAILS_ROOT/bin. These should be used instead of the system gems to ensure the correct version is being executed. So from RAILS_ROOT, instead of issuing:

cucumber

You should run:

bin/cucumber

Machinist vs. Factory Girl (vs. Fixtures)

Fairly quickly our of the gate, we had a discussion on using object factories instead of fixtures. Examples of object factories include factory_girl and Machinist - both work really well with Cucumber. In our example, we used factory_girl.

Some of the Machinist fans found the integration of Sham and Faker for completely random data, along with simple association declarations makes it a solid choice. It was also brought up that factory_girl could use the Machinist API (at least I think this was mentioned).

But without getting into a religious war, using an object factory pattern is clearly a win.

Features - Consumable by Non-Programmers?

The discussion of what the actual value of human-readable features is and if it’s worth the effort go through this DSL. Some developers out there believe it is not worth the effort, but the consensus at the workshop was cucumber human-readable features are worth it. It also seemed that developers are primarily responsible for authoring feature files; then QA or stakeholders review and provide acceptance.

Using IVARs

Along with non-programmer interaction with feature files, the use of instance variables is a common topic when discussing cucumber. Using instance vars does make individual steps less modular – you can’t cherry-pick a step if it relies on something being set, for one. But setting up a state, like an actor or current user, can simplify features with intentional and calculated deviation from pure “no-ivar” philosophy.

In the end, it seemed that being pragmatic and trying to minimize the use of instance variables was the consensus.

Pickle - Making Life less “Cucumbersum”

Pickle is a nice way to interact with your object factories in cucumber. Steve created a pickle branch in codestache where the manual factory step was replaced with Pickle steps.

features/share_code.feature
@@ -17,6 +17,6 @@ Scenario: Create a code snippet
     And I should be on the "sudo rm -rf /" snippet page

 Scenario: View a code snippet
-  Given a snippet exists with a body of "sudo rm -rf /"
+  Given a snippet exists with body: "sudo rm -rf /"

features/step_definitions/share_code_steps.rb
@@ -1,3 +0,0 @@
-Given /^a snippet exists with an? (.*) of "([^"]*)"$/ do |attr, value|
-  Factory(:snippet, attr.gsub(' ', '_') => value)
-end

Later today in IRC (#boston.rb), Dan Pickett pointed out that factory_girl actually has some undocumented pickle-like step definitions.

XPath Selectors (instead of CSS Selectors)

In the original codestache title branch, we had a feature like this to help us specify there ought to be two fields, body and title, using some webrat CSS selectors:

Scenario: Visit the home page
	Given I am on the home page
	 Then I should see "New snippet"	
	  And I should see "Title" within "label[for=snippet_title]"
	  And I should see "Body" within "label[for=snippet_body]"

Arguably, this exposes internal view implementation and not business requirements. Fortunately, ajvargo was in attendance and provided this step definition that uses XPath to marry up semantic form labels to their respective fields. It is still work-in-progress, and won’t work with the non-for, <label>Label<input id="foo" name="foo"/></label> style, but removes the CSS selector from the feature (and I believe he also mentioned this works well with Selenium).

    # Uses xpath to check for a form element based on the label,
    # instead of using a css selector.
    Then /^I should see a "([^\"]*)" for "([^\"]*)"$/ do |type, name|

      case type
        when 'Button'
          response.should have_xpath("//input[@type='submit' and @value='#{name}']")
        when 'Text Field'
          response.should have_xpath("//input[@type='text' and @id=..//label[normalize-space(text())=\"#{name}\"]//@for]")
        when 'Text Area'
          response.should have_xpath("//textarea[@id=..//label[normalize-space(text())=\"#{name}\"]//@for]")
        when 'Radio Button'
          pending
        when 'Select Field'
          response.should have_xpath("//select[@id=..//label[normalize-space(text())=\"#{name}\"]//@for]")
        else pending
      end

    end

So with this step definition, our feature could now like this:

Scenario: Visit the home page
  Given I am on the home page
   Then I should see "New snippet"	
    And I should see a "Text Field" for "Title"
    And I should see a "Text Area" for "Body"

Feedback

One area we didn’t get to much into was the reuse of scenarios in step definitions. For instance, some login feature could be cuke’d with ample of detail, then consolidated and referenced as a ‘When I successfully login’ step for subsequent new features.

Another item we missed talking about was cucumber tagging. As cucumber test suites can grow in size and run-time, using tagging can be a way to isolate tests locally during development (continuous integration should continue to run the entire suite). Also, we didn’t really cover the special @wip tag, which marks features as works-in-progress.

Other feedback included better instructions getting people setup before-hand. Our internet pipe was laboring as everyone was cloning the project at the same time.

The balance of roughly 50% discussion and 50% hands-on coding, seems to resonate well.

All-in-all, it was a very successful maiden “Newer Yankee Workshop” – we are looking forward (and looking for topic suggestions) for future ones!

Mysql::Error: Lock wait timeout exceeded; try restarting transaction

2009-12-11T00:00:00-05:00

This post pertains to MySQL with InnoDB.

With heavy concurrency it is common to see the “Lock wait timeout” error on some database transactions. This is a temporary error that can often be made to go away just by trying the transaction again. Here is a little helper method you can put in a Rails initializer, for instance, to give your model a concise way to retry a few times.

class ActiveRecord::Base

  def self.retry_mysql_error(tries=3, &b)
    count = 0
    begin
      count += 1
      return b.call
    rescue Mysql::Error
      if count < tries
        sleep(0.1 * count)
        retry
      end
      raise
    end
  end

end

Just pass it a block (and an optional number of tries) and it should help to reduce the number of timeouts you see. Be careful not to carry over inadvertently any assumptions across retries.

class Download < ActiveRecord::Base

  def Download.increment_download_count!(filename)
    Download.retry_mysql_error(5) do
      record = Download.find_by_filename(filename)
      record.download_count += 1
      record.save!
    end
  end

end

It also plays nice with RSpec.

describe Download do

  describe "increment_download_count!" do

    it "retries mysql error five times" do
      Download.should_receive(:find_by_filename).exactly(5).times.and_raise(Mysql::Error)
      lambda { Download.increment_download_count!(true) }.should raise_error(Mysql::Error)
    end

  end

end

Of course, this technique should be balanced against other considerations. If it is unable to complete successfully after all the retries, it will re-raise the Mysql::Error. If this happens often, you might need to make deeper changes to your application.

Fun with Tomatoi.st; the Pomodoro Technique

2009-11-11T00:00:00-05:00

I did a lightening talk at this past Boston.rb meeting about the Pomodoro Technique, including Stephen Caudill’s recently launched Tomatoi.st.

The Pomodoro Technique (PT) uses behavioral rules to help focus time and increase overall productivity. The rules attempt to build a nice mental rhythm while adding a feeling of time-importance to you and your workmates. But the main goal is to alleviating anxiety due to the perception of having not enough time via a bit of a mind hack – changing the way we use time.

Francesco Cirillo, creator of the technique, recommends that a modest measure of time, like 20-40 minutes, followed by a shorter break, like 5 minutes for a few cycles, then a longer break. He apparently did research to come to this time pattern.

The quickstart is:

Set a timer to 25 minutes
Take a 5 minute break
Repeat for 4 cycles
Take a 25 minute break

Tomatoi.st is a simple web-based timer to help you out. It is ideal as a Fluid application if you are using OSX (it even integrates with growl). Here is an icon you could use.

Full Version

This is a bit over-simplified, as the PT also involves a planning and progress worksheet, along with estimation techniques – I strongly recommend you read the PDF book. From my limited use and reading on the subject, the technique is something you can pick up fairly quickly, but to master and benefit from it, plan to spend at least a few months.

Amazon ELB - Root Domain Oversight!

2009-11-05T00:00:00-05:00

Amazon Elastic Load Balancer, or ELB, is a service from Amazon’s webservice group that allows you to setup a load balancer in the cloud without having to run a dedicated instance (or instances if you want failover) – which saves money and maintenance (it costs under $20 per month, plus bandwidth to use ELB, the smallest instance is close to $70).

So we wanted to us ELB and set it up in a staging environment, where it worked like a charm. Our staging environment had the subdomain prefix staging. But when we switched to production, we ran into a bit of a snafu.

From the ELB documentation:

You will still need a CNAME, or equivalent, in order to map 
the *generated* DNS name to a name that is meaningful to you 
and your customers (e.g. www.mywebsite.com).

So the DNS name for the loadbalancer ought to be used when mapping the actual domain of your application. This is likely because it is fairly straightforward to failover using a DNS name (verses an IP address). But the oversight is for root domains.

Root domains require an A record to point at an IP address. So if your application is intended to run from a root doamin (i.e http://myapp.com), using ELB out of the box is not straightforward. For now, we ended up making the application run on a www subdomain and used a designated instance with an elastic IP to forward any the root traffic to it.

There are a few other workarounds, including setting up DNS / named servers to use name-based failover or round-robin name resolution - but none seem that clean. This problem is apparently fairly known, and Amazon has indicated they are working on a solution, hopefully soon.

So I guess the moral of the story is if you want your application to run on a root domain, you may want to reconsider ELB, at least for the time being.

Vermonster.com, now with Jekyll

2009-10-24T00:00:00-04:00

It’s been a few years, so we decided to redesign our site again. Our original site was running Wordpress, which is a fine blogging/CMS package. As we’ve been simplifying our infrastructure, we decided to try Jekyll this time, a static site generator. We did this for a few reasons:

It is super simple, static site is simple HTML - any ole’ web server will do
The page sources are in markdown
Everything from templates to post content fits nicely into git, so we have fantastic version control
It’s written in Ruby

A few things did make us think:

Comments and contact form need 3rd party (WuFoo and Disqus respectively)
Embedded comments and contact form needs JS (but we do have degraded links)

But the yeas won, so we decided to go with it.

We did implement a few tricks, via a nice Rakefile. First, we are using rsync to publish the site (which is common for Jekyll -driven sites).

desc "Build then rsync the site to the public server"
task :publish do
  sh %{ rm -rf _site }
  sh %{ jekyll }
  sh %{ rsync -av --delete -e "ssh -i ./.ssh/dsa_key_file" ./_site/ user@server:/var/path/to/webroot }
end

If you use this code above, your key and destination will vary when rsync’ing.

We also have a task that uses a template to create a new post, prompting for a title and filling in the author.

def dasherize(title)
  title.gsub(/\s/, '-').gsub(/[\|\?\+\#\!\,\.\'\"\:\;\&]/,'')
end

desc "Create a new post using today's date, filled with instructions"
task :post do
  filename = File.dirname(__FILE__) + "/_posts/" + \
    Time.now.strftime("%Y-%m-%d")

  title = ENV['TITLE']
  if title.nil?
    print "\n>> What's the title? (skip by set env. var TITLE) : "
    title = STDIN.gets.strip
  end

  filename << "-#{dasherize(title)}".downcase + ".md"

  author = `git config --global user.name`.strip
  if author.nil?
    print "\nQUESTION: What's your name? "
    author = STDIN.gets.strip
  end

  content = File.read(File.dirname(__FILE__) + "/_posts/template")
  content.gsub!(/.*author:.*/, "author: \"#{author}\"")
  content.gsub!(/.*title:.*/, "title: \"#{title}\"")
  
  File.open(filename, 'w') {|f| f.write(content) }
  
  puts "\nMade new post by #{author}:"
  puts "  #{filename.gsub( File.dirname(__FILE__) + "/",'./')}"
  puts "\nCommit changes to git. Publish with 'rake publish'"
end

## ./_posts/template
__END__
---
layout: post
published: false
title:
#author: 
#tags:
#- "ruby"
---

Last, we implemented a fun changing theme, where the background images switch from day to night. We did thing with a simple crontab entry on the server that switches around a supplemental CSS style sheet.

All in all, we are very happy with the new site and hopeful the simplicity of Jekyll will promote more posts from our team (because we have lots to talk about!). We haven’t converted all our existing blog entries over yet, but plan to do so over the next week or so.

Shopify_API 0.0.1 Released

2009-05-29T00:00:00-04:00

You can see Shopify_API for a small library for driving the Shopify.com API. It is adapted from the official Shopify Rails plugin, but decoupled from the controller for use outside of web requests. It depends on ActiveResource, enjoy!

svn:mergeinfo - Addressing a Painfully Missing Subversion Feature

2009-02-12T00:00:00-05:00

I’ll preface this by saying article is about subversion and not git. Since release 1.5, subversion now includes a property, svn:mergeinfo, that is auto-applied when preforming merges. Before this, a merge was pretty much an ad-hoc diff between two files or lines of code. The best practice was to include the merge command used in the SVN comment, which was extremely unreliable. For clients and projects not yet on the git-train, svn:mergeinfo is a welcome and painfully overlooked feature.

We, like many other teams, typically follow the TRUNK = “bleeding edge”, branch-releases pattern to SCM. So at the end of each iteration, we cut a release branch from TRUNK. This release is deployed and we turn our efforts toward the next sprint. However, is is atypical to not have to patch the release. This is usually done in the release branch, tested on a staging environment, the ultimately deployed to production. Afterward, the patch is merged back to the mainline development trunk. This pattern works nice, but sometimes we do a series of patches in the release and there is definitely the opportunity to miss a commit when merging. Since subversion previously didn’t have a way to see merges (merges were not directly tracked), the process involved a session of reviewing commit logs and manual comparison. Enter git, which is built to merge, and the mass movement to git is inevitable.

However, with subversion 1.5, a new property is introduced svn:merginfo. This feature add merge tracking to svn. So when committing the results of a merge, this piece of metadata is recorded. This opens up many new features, like looking at the svn:merginfo property (you shouldn’t edit this property, only get it).
And one of my favorite features is the --reintegrate option. So we can now tell, with a single command, what merges in the release haven’t made their way back to trunk — no more missing merges!

There are a few points, echoed here (parting thoughts section at bottom), worth mentioning:

The subversion server and all developer clients ought to be at least running svn 1.5+
Do all the merges from a standard location, like the project root (RAILS_ROOT for RoR)
Inform developers of this property. After doing a merge, the svn:mergeinfo is put on the place the merge was preformed (preferably project root). So developers should know this and make sure to commit this property change.

Minimizing Software Entropy in the Agile Environment

2009-02-06T00:00:00-05:00

As part of the agile process, each development sprint has a planning prerequisite. During this planning process, user stories are scored using a consensus complexity and risk estimations. It is common that some stories are deemed “to risky” and either scaled-back, re-evaluated or hit the chopping block. Minimizing risk and defects is something desirable to all - management, developers and stakeholders. This mix of risk and defects is sometimes referred to as “software entropy”. From my thermodynamics class, I vaguely remember that entropy has something to do randomness, or was it heat - or is that enthalpy?

After a quick google + wiki session, I found that entropy is the amount of disorder in a system. And there is a principal that says in a closed system, one can’t decrease the amount of entropy, it either increases or could possible stay the same. So that doesn’t look good if we apply this to software. This basically says that each iteration the very best we can do is have the same unpredictability, but most likely our software will continuously get worse! Not good news for the software industry and not good news for agile practitioners.

However, we know software doesn’t always become more unpredictable, software can become more stable. So what’s the factor and how can being agile, which embraces change and uncertainty, be used to minimize software entropy?

The factor is actually refactor, as in software refactoring. So we follow the laws of thermodynamics in this house, our system is not a closed one (it all depends on how you draw your boundaries). Whenever we add or removed lines of code, we are doing so as an external influence. This can actually reduce software entropy. For instance, if we have a defect rate of say 3 defects per 1000 lines of code, and we remove 500 lines of code in an iteration, we’d have 1.5 less defects, right? Well that was pretty theoretical, but the concept does work in a very tangible, and common occurrence - bug fixing.

We consider any bug fix that turns out to be a software defect (i.e. one that is not misimplementation of some business requirement or feature) a refactoring effort. When a defect is addressed, the existing code is altered as we add or remove bits - and we are external influence. Fixing defects can have 3 possible outcomes: 1) the system is more stable, software entropy reduced (ah!, the fix works!), 2) the system is not any more or less stable, software entropy unchanged (the fix didn’t work, but at least you didn’t break anything), or 3) the system is less stable, software entropy increases (regression! whack a mole! uh-oh!). So if we are good bug fixers, we can reduce entropy.

Now for the agile part. As we work through our iterations, defects land on our plate. And properly fixing them does reduce entropy. But another way to more drastically reduce entropy is some good ole’, heads-down refactoring effort. Pure refactoring specifically targets minimization of software entropy. When we refactor, we are DRYing up methods, eliminating lines of code, cleaning and making our software more stable and manageable. We strongly recommend that each iteration ought to have some overhead earmarked for refactoring.

It is sometimes difficult to justify refactoring to stakeholders and non-developers, as the business value is difficult to measure. However, if a new feature is deemed “too risky”, a trade-off could be to add a pure refactoring effort to the iteration targeting the potential risk of the feature. Then build the feature.

So the moral is to minimize entropy and uncertainity in software, try to have refactoring a part of each development iteration.

Vermonster's Blog

Build-a-Gem Workshop Recap

We Moved to 71 Broad St in Boston

Recommended Omnigraffle Guide for Rails ORM

Has One / Belongs To

Has Many

Through, HABTM and Polymorphic

Links

Cucumber Workshop Recap

Gem Bundler

Machinist vs. Factory Girl (vs. Fixtures)

Features - Consumable by Non-Programmers?

Using IVARs

Pickle - Making Life less “Cucumbersum”

XPath Selectors (instead of CSS Selectors)

Feedback

Related Links

Mysql::Error: Lock wait timeout exceeded; try restarting transaction

Fun with Tomatoi.st; the Pomodoro Technique

Amazon ELB - Root Domain Oversight!

Vermonster.com, now with Jekyll

Shopify_API 0.0.1 Released

svn:mergeinfo - Addressing a Painfully Missing Subversion Feature

Minimizing Software Entropy in the Agile Environment