Fluffy - Flex testing and automation tools and strategies

I just set up a QA site for a new project I’m working on called “Fluffy” — just because it’s a word that starts with the letters “FL”. It’s a research project born out of a conversation with my friend Nate at a BBQ last week.

Fluffy is an investigation into Flex testing and automation. Nate mentioned that tools and techniques for testing Flex are rudimentary, if used at all. Since he’s a Flex developer, it’s a pain he feels acutely. We agreed to work together to try to improve this.

A quick search on Google turns up FlexUnit and AsUnit. There is also the Automated Testing API on the other end.

How do you do functional testing with Flex? That’s the type of question we want to find answers to.

What kind of tools are needed? Do we want a recorder or UI scripting? Something like selenium? A test player? Performance tools? Randomized or parameterized data? Mocks or stubbing?

Just like on my other blogs, I’ll post links here for completeness and just in case anyone wants to get involved.

Here is the Fluffy QA Site with subversion, bugzilla, projectpier, trac, blogs, and forum.

What do I get out of it?

I get to learn Flex. And a project to work on with a friend. It’s my aim to also improve the QA site / PM site tools and process as well while working on this project. And maybe I’ll make some money out of it. Though if we do develop any tools, they’ll most likely be open sourced.

We welcome any other interested participants.

Pylons and WSGI

Pylons is next on my list of frameworks to look at, because I like the scalability characteristics of WSGI middleware servers.

The idea that you might have 50% of your requests using sessions, 25% using authentication, 10% some web service, 10% using some other web service (cached), and some other percentage using any of a number of other things that could involve caching, computation, instantiation, database and/or network connections, etc., and not using the mod_perl solution of loading everything into every server process, the PHP way of loading everything from scratch (or one big cache) on every request or the java solution of really big APP servers with lots of threads.

Instead of have 100 high memory and/or cpu worker processes (or threads) taking up resources so they can handle everything (except static content or database) or moving everything onto the network with webservices, splitting it up so you have just as many processes as you need for each service, saving 90 memory for each of your web service handlers, and 50 percent for your session handlers.

However, I’ve heard in practice that only a few things need offloaded, and WSGI in practice doesn’t do that much better than separating static, appserver, database, and cache, which is probably good enough for most. So the overhead of a common gateway interface isn’t that useful.

Yet I do like the idea of “feature-specific” caches or servers. Having a pool of memcached’s is really nice, but having pools of each of session caches, page caches, web service caches, business object caches, etc. seems like it would be more resource efficient, and possibly more secure.

Coming from PHP, the idea of offloading things like PDF and graphic processing from the webserver (or appserver) also appeals.

more reasons not to use RoR 2

from the same slideshow

“XML in, JSON out”

Why promote XML? That’s not very web 2.0. And more important, if you’re using one, why use the other. JSON is supposed to be A solution to XML, not a consumer of it. I realize you might need to consume XML, but if at any point your JS data structures look like SOAP, you’ve got problems. If ever an XML data structure that actually makes sense for the domain it is intended to be used it… well, I’ve never seen one.

Plus, there seems to be general trend towards more helper functions to write HTML snippets for you, like div_for, dom_class, dom_id, url_for, etc. It’s worse spaghetti than old school PHP. Worse than old school CGI or servlets.

I realize AJAX rendering is hard, but if this is Rails’ answer, give me JSF or ASP code behind.

real reasons not to use Rails

Mostly taste, like I don’t like ActiveRecord, database naming defaults, yaml, etc.  But here is something from a What’s new in Rails 2 slideshow:

validates_numericality_of :salary, :greater_than => 49_000

Now I’m not talking about the amount.  Although I’m aware as a RoR developer, you run the risk of failing that particular validation check, something you’d never worry about in Java or C++ land.

What I mean is, if this is an example of a dynamically typed language, I don’t want any of it.  Here is the same code in Java or C:

int salary;
salary > 49_000;

Much more concise, and more better checked too.  This will catch it at compile time (unless you have a cast from a hash — but that’s an argument in favor of generics.)

Of course, Rails has a bunch of code behind the scenes that is probably doing Javascript and serverside validation and displaying a pleasant and ambiguous error message to the user and log, but you can do all that with an annotation as well.  A more realistic comparison might use an annotation to tie your salary integer to your view and/or model and require a try/catch block for type cast errors.  Or it could wrap all that up in a static function that starts with “validates” that does all that.

I’m arguing for the syntax.  And I don’t think it takes a statically typed, pre-compiled language to get that.  I hope it doesn’t take operator overloading to achieve a > comparison.  Any decent language should be able to cast, throw a runtime error on type mismatch, and return false.  How 49_000 is cast is dependent on the language, and it doesn’t really matter as long as it is consistent and documented.  So guess I’m aguing against needing to use is_numeric() style functions as well.

With enough scaffolding and inheritance, you can do anything, even in C++.  As a matter of fact, with a hypothetical C++ framework like Rails (with other goodies like boost GC) you could get pretty close to achieving the ideal conciseness above.  Maybe we should all switch to C++ web frameworks once the DHH of C++ comes along.  Then our numerical validation really would be:

salary > 49_000

reasons for not using Rails

couple of reasons for not using Rails are performance and scalability.  I’m actually confident in the development community and momentum behind Rails that these will soon be non-issues.  Enough so that I can’t use them as arguments against using Rails.  By the time performance & scalability would become an issue to One Shore or most other small shops, you could throw hardware at the solution, and besides, there are some promising developments that I think will probably make them irrelevant.

First, are the performance improvements.  A lot work is being done on that end, though by a lot of disparate groups.  Besides Ruby 1.9, there is Yarv, Rubinius, IronRuby, Ruby.NET, and JRuby.  With all this enthusiasm, whether it’s there yet or not, engineers will make Ruby performant.  Some collaboration might be in order, but it’s the healthy competition that will bring the results.

Sun is backing JRuby pretty hard, but most forcefully with the Netbeans Ruby IDE.  You can now fairly confidently scale Ruby via Java, with the attractive benefit of the JEE stack.  The other benefit is that if you are running on JRuby, you can push your complexity where it belongs, into Java.  (IronRuby promises similar integration benefits on the .NET stack.)  So you don’t have to worry about integration, web services, caching, etc.  On the other hand, JEE is very un-Rubaic(?) and people use Rails to avoid things like Hibernate and JSF.

Which brings be to my next argument –  Rails 2.1 — which has, among other things, nice built in memcached support.  Rails 2 is by most accounts a subtle, but significant improvement, with supposedly a relatively harmless migration.  Due to the relative lack of documentation (and plethora of existing documentation) however, I would
be hesitant to jump in with Rails 2.1.  Starting with 1.2.6 and an older book, and then trying to add features and clear warnings would be a good strategy.  6 months or a year from now, that probably won’t be as big of an issue however, except you’ll have to spend $100 for new books — that is if they get written.  There is a risk that the bulk of Rails’ userbase will migrate gradually and not feel a need to update their documentation.  If there is still an influx of new users, then the new docs will get written.

And finally, there is progress on the deployment end.  Everyone has read a rant or two about hosting Rails.  Mod_ruby was never more than a proof of concept, and even Zed Shaw pissed on Mongrel.   But, as predicted, big providers like Dreamhost leapt in to meet the demand, even if a bit slowly.  I’m excited with developments like Phusion Passenger which also markets a Ruby VM they call Ruby Enterprise Edition, probably hoping to be the Zend (not Zed) of Ruby.

If PHP had more competition like this, it might have had a credible story.

Don’t get me wrong, PHP is good at what it does, and still my “favored” language, but unless you’re dealing with stateless apps, it doesn’t have a prayer.  Even with stuff like Zend Framework, CakePHP, and Symfony, PHP doesn’t have the runtime for complexity.  Ruby didn’t either until recently.

But maybe it’s the passion people have for Ruby (and especially Rails) that makes the environment that PHP lacks coalesce.  PHP is popular because it can “get it done”, and when Perl and Java were the only alternatives (and with much better stacks), PHP was an easy winner.  With Ruby (and Python) now coming into their own, who knows?

Google Checkout

Have had trouble with Paypal. Both me and Kelsey have had our accounts locked, without ever having used them. Probably because we dared log into paypal from an IP address in Ecuador. Or whatever their flawed heuristic things our IP came from.

So I’m looking at Google Checkout. It’s much cheaper, and probably better service. I don’t know if that would cut it for One Shore business payments, but it’s a start. I’m not getting anything through Paypal as is anyway.

Email invoices seems interesting too, though if it gets popular will become a huge phishing problem.

Would love to hear anyone’s feedback from using Google Checkout or any other payment system.

VPSLand down again

My VPSLand Virual Server that One Shore is hosted on is down again.  That’s the second time since I’ve got back (in two weeks.)

It responds to pings, but not SSH or HTTP.  It’s possible I did it, or someone malicious, but not likely.   I’ll need to investigate.

New site design with ModX content management

I’m working on a redesign of One-Shore.com, with a blue and tan “nautical” theme.  I’ve run into the usual frustrations with Javascript, CSS, and DOM, but am working through them.

The new site is under development at http://new.one-shore.com.

I’m also using the QA Site for development (at http://one-shore.qa-site.com) , but right now that consists of just using Bugzilla and Trac (for design.)

I really liked my method of using the wiki for design, building a wireframe (that’s not the perfect term) of the links in the wiki, and then a wiki features list.  The as you flesh out the pages, you can link to features, components. I’d like a feature checklist app that can tie in to the wiki pages detailing each feature, and the ability to spin a discussion off of the actual documentation.

I’ll be using Project Pier for planning as well, but I don’t know how happy I am using lightweight PM tools.  The task lists seem too informal, more like todo lists, and I’d like to tie work to features, but keep the project->milestone->task->work and product->release->feature->requirement hierarchies separate.  (See my post on modeling projects and products.)

I’ll wait until after the site design to put anything valuable on the dashboard.  Things like recent changes and assigned bugs, tasks, tests executed, etc. kind of get unused on a one-man show, since everthing is assigned to you.  But I do need some help prioritizing.

Besides the new layout, I’m working on the site navigation and content.  I’ve got a new menu list, and am still debating whether to add drop down menus:

• about
• blog
• wiki
• demos
• products
• services
• support
• contact

I’m leaving the forum off (and may take the demos off) and added products and support.  Support can lead to the forum or contact. I’ve also thought about putting blog/wiki/forum/demo on a sidebar.  Maybe with the tools wiki links.

I’ll be making the tools menu tree into a dynamic widget (eventually) - though it seems more of a hassle than it’s worth.  I’d still have to keep it in sync with the wiki.  I’ve toyed with XML and am thinking of maybe creating a DB table for it.  Shoehorning a hierarchy into a table structure isn’t always best, but I think it might make sense in this situation.   The table could look like:

• Link
• Id
• Label
• Expandible (or hasChildren)
• Parent (or Category)

Or I could have flags (or associations) for QA, Dev, PM, Biz, etc.  as well as tool types such as bug tracker, IDE, etc.  Too much abstraction is the enemy of a useful model.

Content is the big one that needs updated, though I’ve been cutting and pasting from the existing site.  I am just not good at marketing copy.  In order to make the copy editing easier (and to impose some structure so I don’t have to make the pretty links myself, I’ve decided to use a CMS.

After a lot of evaluations, I’ve come up with a list of what’s wrong with CMSes.  Though as usual, it’s more of a ramble than a useful list.  The main issues are:

• too much of a learning curve — wasting time gaining knowledge just to use that specific tool
• letting me lay out my site the way I want — again, needing to learn a complex new templating system and arcane admin inteface
• making it easy to update content — not in a textarea in a browser and not stored in a database.

I’m using ModX for content management:  ModX has the nicest admin tool of any CMS I’ve tried.  It’s philosophy is to get out of your way.

ModX does a fairly good job of the first two.   It’s a little weak on structuring though, but that allows you to post a PHP snippet instead of learning a byzantine module format.  So there are pluses and minuses.  If you could plug in a component framework with things like authentication, etc.  then you might be cooking.  As is, you’ve got a bunch of one-off scripts that you have to cut and paste into the admin textarea.  So something like Joomla or Drupal wins in that regard, except that writing modules, plugins, etc. for either of them is too arcane, and all you get out of it is a sidebar box.

But after only a couple days, I’m really aching for a real content repository.  Something I can roll back, something I can FTP (or much better, checkin.)  Content should be under revision control.

And the develop/test/deploy cycle for textarea editing and programming is brutal.  I want to make a change in dev, run a quick script to put it in test, and then run the same script again with a different parameter to put it in staging.  Once it’s verfied in staging, I want to flip a switch (like change a VHOST and restart apache) and make it go live.

And I don’t want to do database dumps.  For one, that means losing production data every time I make a change.  And for another it makes the cycle that much longer and error prone.  I want to make a quick layout change in dev and push a button and see it show up on the production site.  For this reason, I’m thinking about giving up on CMS altogether.

What I’m really looking for is a CMS that uses a repository for content.  Including layout and formatting (themes.)   It can still use a database, especially for things like users.  But don’t put my content in there.  Or if you do, provide scripts so that I can put them in a file hierarchy and it slurps them up for search or admin or whatever.  A decent CMS outputs pages and components to a cache file anyway for display.

Need graphic design

Well, I tried posting on craigslist, but it won’t post. It exists, but I don’t know how to pay for it to show up on the listing:

http://seattle.craigslist.org/see/cpg/587815221.html

Oh well, it’s a crap site anyway. And yes, sometimes the grapes really are sour.

I’m looking for a graphic designer to help me spruce up my site and make it look more “businessy.”

Basically, come up with a logo, a few images having to do with the theme of “shore” (beach, ocean, waves, palm trees, sailing, etc.) or other stock photos that might be relevant to my business (computers, people in suits), and a new layout/color scheme. Because I’ll waste way too much time trying to do it myself, and honestly, the design is blocking my desire to complete the implementation and edit the content, which is desperately needed.

I’ve got $500 for the designer that can do it and make me happy.

Actually, I’m curious how the whole graphic design biz works. It doesn’t seem fair to pay someone by the hour until a monkey types King Lear, and it doesn’t seem fair to plunk down a wad of cash and take whatever crap the monkey flings at you.

Payment on satisfactory completion seems the most reasonable, but for sticklers (like me) that might not be reasonable. A contest seems a better bet, because each person is only doing a design once, and then moving on if they don’t get the bid. But design is a give and take, so the odds of making a client completely happy through a contest are small. I can’t see paying hourly rates unless you *really* trust the designer. Lots of designers are probably paranoid (justifiably) about their work being ripped off once shown.

So either a big fee, and basically indenturing yourself until the client is happy, or a big purse to make it worthwhile to risk throwing away your work.

I realize the amount of money people are willing to spend versus the amount a designer needs to have their creative leeway and satisfy the client are not lined up. But the way the market works is the value has to match the buyer’s perception, or there is no buyer.

I’d be interested to learn how designers do it. Maybe you just need to have the confidence that you can get it right, and figure they payouts will even out.

If a good designer is worth as much as a good developer, I’d say a week of his time is worth at least $1000. But is a logo, a couple of stock photos, and a layout concept really 40 hours of work?

What I really want to do

Okay, that last was a practical plan, but here’s what I really want to do:

1. Design the ultimate project management and quality assurance processes
2. Build my ideal web framework
3. Build the PM and QA functionality on my framework
4. Experiment with Ruby, Python, and Groovy more
5. Mess around with more framworks, including source level investigations
   • Wicket
   • Django
   • Seaside
   • Catalyst
   • Give CakePHP another shot
6. Mess around with more tools (for open source tools)
   • Bugzilla
     • Modular UI
     • Web Services and RSS
     • DB abstraction layer
     • More tests (perl tests & GUI tests)
     • Getting started dev docs
   • Mercurial, Git, Aegis
   • Eclipse
     • Jazz
     • GMF
     • Mylyn
   • HP/Mercury
   • Seapine
7. Work on my ideal tools
   • PM
     • Process/Ticket/Issue/Defect->Fix/Enhancement->Change Request/Work Item
     • Projects/Milestones/Tasks/Work
     • Products/Releases/Features/Requirements
   • Design & Modeling
   • Process (IBM Rational Jazz looks interesting)
   • Graphic Design
   • QA Tools
     • Defect Tracking
     • Test Case Authoring / Execution / Reporting
     • Automation
     • Builds
     • Unit Tests
     • Smoke Test
     • Performance Tests
     • Requirements Tracking
     • Change Management
8. Master infrastructure tools
   • Email
   • Authentication (LDAP, RADIUS, etc.)
   • DNS
   • Virtual Servers (VMWare & Xen)
   • Security (Monitoring, Lockdown,
   • Fix Spam (note, I have a solution, that’s not directly technical, but won’t go into it now.)
9. Play around with VOIP and WIFI
   • Asterisk
   • FreeWRT

So really what I want to do is build my framework and tools, and research frameworks, tools, and languages for how best to do it