Fluffy – Flex testing and automation tools and strategies

I just set up a QA site for a new project I’m working on called “Fluffy” — just because it’s a word that starts with the letters “FL”. It’s a research project born out of a conversation with my friend Nate at a BBQ last week.

Fluffy is an investigation into Flex testing and automation. Nate mentioned that tools and techniques for testing Flex are rudimentary, if used at all. Since he’s a Flex developer, it’s a pain he feels acutely. We agreed to work together to try to improve this.

A quick search on Google turns up FlexUnit and AsUnit. There is also the Automated Testing API on the other end.

How do you do functional testing with Flex? That’s the type of question we want to find answers to.

What kind of tools are needed? Do we want a recorder or UI scripting? Something like selenium? A test player? Performance tools? Randomized or parameterized data? Mocks or stubbing?

Just like on my other blogs, I’ll post links here for completeness and just in case anyone wants to get involved.

Here is the Fluffy QA Site with subversion, bugzilla, projectpier, trac, blogs, and forum.

What do I get out of it?

I get to learn Flex. And a project to work on with a friend. It’s my aim to also improve the QA site / PM site tools and process as well while working on this project. And maybe I’ll make some money out of it. Though if we do develop any tools, they’ll most likely be open sourced.

We welcome any other interested participants.

Pylons and WSGI

Pylons is next on my list of frameworks to look at, because I like the scalability characteristics of WSGI middleware servers.

The idea that you might have 50% of your requests using sessions, 25% using authentication, 10% some web service, 10% using some other web service (cached), and some other percentage using any of a number of other things that could involve caching, computation, instantiation, database and/or network connections, etc., and not using the mod_perl solution of loading everything into every server process, the PHP way of loading everything from scratch (or one big cache) on every request or the java solution of really big APP servers with lots of threads.

Instead of have 100 high memory and/or cpu worker processes (or threads) taking up resources so they can handle everything (except static content or database) or moving everything onto the network with webservices, splitting it up so you have just as many processes as you need for each service, saving 90 memory for each of your web service handlers, and 50 percent for your session handlers.

However, I’ve heard in practice that only a few things need offloaded, and WSGI in practice doesn’t do that much better than separating static, appserver, database, and cache, which is probably good enough for most. So the overhead of a common gateway interface isn’t that useful.

Yet I do like the idea of “feature-specific” caches or servers. Having a pool of memcached’s is really nice, but having pools of each of session caches, page caches, web service caches, business object caches, etc. seems like it would be more resource efficient, and possibly more secure.

Coming from PHP, the idea of offloading things like PDF and graphic processing from the webserver (or appserver) also appeals.

more reasons not to use RoR 2

from the same slideshow

“XML in, JSON out”

Why promote XML? That’s not very web 2.0. And more important, if you’re using one, why use the other. JSON is supposed to be A solution to XML, not a consumer of it. I realize you might need to consume XML, but if at any point your JS data structures look like SOAP, you’ve got problems. If ever an XML data structure that actually makes sense for the domain it is intended to be used it… well, I’ve never seen one.

Plus, there seems to be general trend towards more helper functions to write HTML snippets for you, like div_for, dom_class, dom_id, url_for, etc. It’s worse spaghetti than old school PHP. Worse than old school CGI or servlets.

I realize AJAX rendering is hard, but if this is Rails’ answer, give me JSF or ASP code behind.

real reasons not to use Rails

Mostly taste, like I don’t like ActiveRecord, database naming defaults, yaml, etc.  But here is something from a What’s new in Rails 2 slideshow:

validates_numericality_of :salary, :greater_than => 49_000

Now I’m not talking about the amount.  Although I’m aware as a RoR developer, you run the risk of failing that particular validation check, something you’d never worry about in Java or C++ land.

What I mean is, if this is an example of a dynamically typed language, I don’t want any of it.  Here is the same code in Java or C:

int salary;
salary > 49_000;

Much more concise, and more better checked too.  This will catch it at compile time (unless you have a cast from a hash — but that’s an argument in favor of generics.)

Of course, Rails has a bunch of code behind the scenes that is probably doing Javascript and serverside validation and displaying a pleasant and ambiguous error message to the user and log, but you can do all that with an annotation as well.  A more realistic comparison might use an annotation to tie your salary integer to your view and/or model and require a try/catch block for type cast errors.  Or it could wrap all that up in a static function that starts with “validates” that does all that.

I’m arguing for the syntax.  And I don’t think it takes a statically typed, pre-compiled language to get that.  I hope it doesn’t take operator overloading to achieve a > comparison.  Any decent language should be able to cast, throw a runtime error on type mismatch, and return false.  How 49_000 is cast is dependent on the language, and it doesn’t really matter as long as it is consistent and documented.  So guess I’m aguing against needing to use is_numeric() style functions as well.

With enough scaffolding and inheritance, you can do anything, even in C++.  As a matter of fact, with a hypothetical C++ framework like Rails (with other goodies like boost GC) you could get pretty close to achieving the ideal conciseness above.  Maybe we should all switch to C++ web frameworks once the DHH of C++ comes along.  Then our numerical validation really would be:

salary > 49_000

reasons for not using Rails

couple of reasons for not using Rails are performance and scalability.  I’m actually confident in the development community and momentum behind Rails that these will soon be non-issues.  Enough so that I can’t use them as arguments against using Rails.  By the time performance & scalability would become an issue to One Shore or most other small shops, you could throw hardware at the solution, and besides, there are some promising developments that I think will probably make them irrelevant.

First, are the performance improvements.  A lot work is being done on that end, though by a lot of disparate groups.  Besides Ruby 1.9, there is Yarv, Rubinius, IronRuby, Ruby.NET, and JRuby.  With all this enthusiasm, whether it’s there yet or not, engineers will make Ruby performant.  Some collaboration might be in order, but it’s the healthy competition that will bring the results.

Sun is backing JRuby pretty hard, but most forcefully with the Netbeans Ruby IDE.  You can now fairly confidently scale Ruby via Java, with the attractive benefit of the JEE stack.  The other benefit is that if you are running on JRuby, you can push your complexity where it belongs, into Java.  (IronRuby promises similar integration benefits on the .NET stack.)  So you don’t have to worry about integration, web services, caching, etc.  On the other hand, JEE is very un-Rubaic(?) and people use Rails to avoid things like Hibernate and JSF.

Which brings be to my next argument –  Rails 2.1 — which has, among other things, nice built in memcached support.  Rails 2 is by most accounts a subtle, but significant improvement, with supposedly a relatively harmless migration.  Due to the relative lack of documentation (and plethora of existing documentation) however, I would
be hesitant to jump in with Rails 2.1.  Starting with 1.2.6 and an older book, and then trying to add features and clear warnings would be a good strategy.  6 months or a year from now, that probably won’t be as big of an issue however, except you’ll have to spend $100 for new books — that is if they get written.  There is a risk that the bulk of Rails’ userbase will migrate gradually and not feel a need to update their documentation.  If there is still an influx of new users, then the new docs will get written.

And finally, there is progress on the deployment end.  Everyone has read a rant or two about hosting Rails.  Mod_ruby was never more than a proof of concept, and even Zed Shaw pissed on Mongrel.   But, as predicted, big providers like Dreamhost leapt in to meet the demand, even if a bit slowly.  I’m excited with developments like Phusion Passenger which also markets a Ruby VM they call Ruby Enterprise Edition, probably hoping to be the Zend (not Zed) of Ruby.

If PHP had more competition like this, it might have had a credible story.

Don’t get me wrong, PHP is good at what it does, and still my “favored” language, but unless you’re dealing with stateless apps, it doesn’t have a prayer.  Even with stuff like Zend Framework, CakePHP, and Symfony, PHP doesn’t have the runtime for complexity.  Ruby didn’t either until recently.

But maybe it’s the passion people have for Ruby (and especially Rails) that makes the environment that PHP lacks coalesce.  PHP is popular because it can “get it done”, and when Perl and Java were the only alternatives (and with much better stacks), PHP was an easy winner.  With Ruby (and Python) now coming into their own, who knows?