Test Automation!

At some point we understood we must have a robust test automation harness which can at least cover our smoke test and regression. This will help us feel more confident in our releases in less time, and allow meeting the business needs.

As we are an appliance based file-system product, in essence an IT infrastructure product, all of the commercially available harnesses from CA, Mercury and the like are useless, as they focus on GUI/Web automation, and we need API automation and the ability to run and control file system operations and file system testing tools.
We considered home-grown approaches but decided that the time-to-market is too long for our needs.
We considered adopting STAF/STAX (http://staf.sourceforge.net/index.php ) but again the custom work needed around it was estimated to be too long and required human resources and expertise we didn't have, and weren't available in the neighbourhood.

What we eventually chose was a testing
automation harness called Aqua
(http://www.aquasw.com/) and we are very satisfied with it.

Its still requires significant customizations/development in a project mode, rather than an off-the-shelf product, but for some situations its the best and only available approach today, and is much better than developing on your own from scratch, or testing manually.