Full-stack webapp testing with Selenium and Rails

Alex Chafee and Brian Takita, Pivotal Labs

slides

They’ve done 15 ruby on rails app in the past 18 months.

Selenium scripts the browser (Mozilla, Safarie, IE) so it allows you to test the whole application. The use Selenium RC that lets you automatically start / stop / control any supported browser that allows you to do cross browser testing in one test run which is useful for CI.

What you want is a whole lot of unit tests (Test::Unit / jsunit) and a few integration tests (Selenium) that test the whole thing.

Demo of them running their selenium tests that are integrated into their rake build. Selenium lets you test AJAX / DHTML stuff.

Selenium IDE is a firefox plugin that lets you record selenium scripts by clicking through your web app.

You can write selenium tests in an internal Ruby DSL! You can also use Java, .Net, Python, Perl. Because you’re writing in ruby you can access activerecord objects, etc as well. You can stub email by using action mailer.

Don’t forget to extract common actions into a DSL (eg. login, logout) You can provide QA with login_as_user, login_as_admin etc and then they can either write low level (click on this button) or high level actions. This makes tests a whole hell of a lot more legible, debuggable, and generally easier to work with.

Showing the architecture of the Selenium RC Architecture. It is a bit complicated, but I imagine that it needs to be to pull off everything that they are doing.

Question: Can you only use Webrick?

No, they’ve used Mongrel before, but it doesn’t have a huge perf impact. Yeah, it’s not the exact same production stack, but that is by design. They want to be able to load fixtures and stub email, etc.

They have a virtual farm of parallels instances running on a mac mini that lets them do Unix, Windows, Mac Safari, Firefox, IE. Idea is to let them have there

Because AJAX and DHTML are asynchronous and dynamic they are hard to test. The way you do it is using wait_for (which runs in ruby) and wait_for_condtion.

Good slide showing the types of different tests. You only want a few Selenium tests. Why? they are slow, and they can kill CI. You do want to use it because it is incredibly powerful. They’re experience is that integration tests are the most important. It is a good idea to have a smoke test, just makes sure it doesn’t blow up. Even a single selenium test that walks through the whole app doing just the nominal case can have a huge test. Don’t use for every edge case.

To counteract the speed issues, break into two suites a slow suite and a fast suite. So run the fast suite before checkin, but kick the slow suite in the CI. so if something in the slow suite breaks you’ll know in the hour.

Selenium tests are less DRY than unit tests just as unit tests are less DRY than prod code. Selenium tests are supposed to exactly mimic user interaction.

Questions

I appreciate the value of this in a large team. Do you recommend this for a small team?

Still recommend Selenium RC so that you can do cross platform testing. Also it lets you write your tests in Ruby. Can still be add a lot of value for a small team.

Pros

• Top-down
• Test passes features done

Cons

• Brittle
• Slow

How to get started?

OpenQA. Documentation will be improved as they open source their rake tasks.

Can you script x/y coordinate clicks to test Flash?

Yes. Testing Flash with Selenium is apparently a huge pain.