Automated tests for applications?

Hi, while I was following the recent decision to notify 3rd parties also on beta releases I was also wondering if there's some kind of automated tests for 3rd party application available/planned/considered? I'm talking about automation of compiling a source release and automatically running test on 3rd party software (phpmyadmin, etc.), of course depending on whether those projects provide such kind of thing. Thinking about unit tests on console but also on web tests via selenium. Is there or has there anything of this kind been done in the past? thanks, - Markus