#title #ubuntu-meeting Meeting Meeting started by mmrazik at 15:31:53 UTC. The full logs are available at http://ubottu.com/meetingology/logs/ubuntu-meeting/2012/ubuntu-meeting.2012-01-24-15.31.log.html . == Meeting summary == *Testability (aka automated testing in Qt environment) ''LINK:'' http://i.imgur.com/Kp7RA.png (greyback, 15:37:03) ''LINK:'' https://wiki.ubuntu.com/Unity2DTestability?action=AttachFile&do=get&target=testability.ogv (greyback, 15:41:41) ''LINK:'' https://projects.developer.nokia.com/Testabilitydriver/wiki (greyback, 15:43:58) Meeting ended at 15:59:01 UTC. == Votes == == Action items == * (none) == People present (lines said) == * greyback (60) * mmrazik (23) * AlanBell (10) * kalikiana (4) * gema (3) * meetingology (3) * Riddell (3) == Full Log == 15:31:53 #startmeeting 15:31:53 Meeting started Tue Jan 24 15:31:53 2012 UTC. The chair is mmrazik. Information about MeetBot at http://wiki.ubuntu.com/AlanBell/mootbot. 15:31:53 15:31:53 Available commands: #accept #accepted #action #agree #agreed #chair #commands #endmeeting #endvote #halp #help #idea #info #link #lurk #meetingname #meetingtopic #nick #progress #rejected #replay #restrictlogs #save #startmeeting #subtopic #topic #unchair #undo #unlurk #vote #voters #votesrequired 15:32:32 hi mmrazik 15:32:36 We have one agenda topic and that is the introduction to testability -- a test automation framework for Qt 15:32:37 hi 15:32:47 o/ 15:32:54 Hi all 15:32:57 [TOPIC] Testability (aka automated testing in Qt environment) 15:33:17 greyback was so kind to do the introduction :-) 15:33:33 greyback, your turn :) 15:33:39 mmrazik: ok, thank you 15:33:46 Hey all, I'm Gerry Boland, team lead of the Unity 2D project. I'm here to talk about automated testing! 15:34:22 We all use our desktops every day. We know it is important for the desktop shell to be reliable, consistent and fast. 15:34:22 From the programmer's side, we strive to maintain these qualities. But we're human, and we make mistakes. 15:34:35 Sometimes a bugfix will break something else, and it's easy to miss these breakages. 15:34:51 are you using at-spi to do the testing? 15:35:06 AlanBell: hold on, I'm getting to it (short answer: no) 15:35:14 We have documents called "manual User Experience tests" where human beings follow a step-by-step procedure to verify the software is doing exactly what it should. But why not automate it! 15:35:33 In Unity 2D, we have just introduced an automated User Experience test system, based on a test framework called "Testability Driver" (I'll just call it 'Testability' from now on) 15:35:53 Testability is for Qt-based applications. 15:36:02 Its core feature is that it allows inspection of the tree of QObjects in a Qt application while it is running. 15:36:23 It can read and write object properties, call methods and slots, verify signals are emitted, as well as faking mouse/keyboard/gesture inputs, grabbing visual outputs, and measure performance. 15:36:42 And best of all, Testability is open source and maintained by Nokia! 15:36:58 To show it off, here is a screengrab of the Testability Visualizer application which allows you to dig into the QObject tree and investigate what's going on (here, the Launcher): 15:37:03 http://i.imgur.com/Kp7RA.png 15:37:22 On the left is a preview of the app, in the center is the object tree which you can navigate, and on the right is the list of properties, methods and signals you can interact with. 15:37:47 All these properties, methods, signals, etc, are scriptable via Ruby. Thus we can emulate every interaction that a user can make with the software, and measure the reaction. 15:38:09 This forms the foundation for the Unity 2D User Experience testing suite. 15:38:20 . 15:38:23 Testability works as follows: 15:38:45 - *any* Qt application using Qt4.6+ can be tested by executing it with the "-testability" switch (and any previous with a little code) 15:39:01 - a standalone server "qttasserver" runs in the background 15:39:16 - With the -testability switch, the Qt library tries to load a "libtestability.so" plugin which establishes a connection between the application and qttasserver, giving access to the root node of the object tree. 15:39:39 - qttasserver then climbs the QObject tree, reads all the info it can and converts it to an XML format. It also can receive commands and cause the application to react upon them (click here, type, etc..). 15:40:01 - A series of Ruby scripts connect to qttasserver, parse this XML and allow us to script tests and interactions with the application. 15:40:31 - from here, can use existing test frameworks like Test::Unit or Cucumber to run tests 15:40:38 . 15:40:48 The Unity 2D test framework also includes an extra helper library to control the X server, to control the mouse and keyboard, and to manipulate windows on the desktop. As far as X is concerned, you are controlling it. 15:41:10 With the ability to fake any form of user input, and directly read the output from the shell applications themselves, we can test almost every behaviour of the desktop shell. And as a result, the quality of the experience will only go up. 15:41:36 As a demo, here is a video of part of the Unity 2D test suite in action. On the right is Ubuntu Oneiric running inside VirtualBox, where the Launcher will be tested. On the left is my host machine terminal running the test suite: 15:41:41 https://wiki.ubuntu.com/Unity2DTestability?action=AttachFile&do=get&target=testability.ogv 15:41:51 (I hope you can all see it) 15:42:07 It shows various hide/show tests are being performed, with windows in the way, mouse being controlled, keyboard shotcuts being pressed etc. 15:42:09 greyback, is the X helper library developed by the Unity-2D team or is it something "standard"? 15:42:30 mmrazik: it's a heavily customised version of a ruby library for xdotool 15:42:51 mmrazik: it's in the unity-2d source repo as a result (but some stuff I want to push upstream) 15:43:09 Our policy is that every new feature and bug fix in Unity 2D will now be tested like this. 15:43:17 greyback: is testability proprietary or free software? 15:43:23 Riddell: open source 15:43:53 Links: https://wiki.ubuntu.com/Unity2DTestability 15:43:56 greyback: so we can package it for ubuntu and have other people be able to test unity 2d and expand that to other qt uses? 15:43:58 https://projects.developer.nokia.com/Testabilitydriver/wiki 15:44:37 Riddell: exactly. I've packaged it up in my PPA (see first link), and hope to get it into Ubuntu proper 15:45:01 Riddell: idea is that everyone can run the tests we do, and help add to them 15:45:24 Ok, that's my spiel. Thanks for your attention everybody. Questions welcome! 15:45:30 I'm sure plenty of people would be happy to review the packaging if needed 15:45:41 greyback: it would be great to see tests for orca users too. With a dummy speech dispatcher module you can get the text orca would read out loud piped to a file, then the test can validate that non-visual users would be hearing something sensible 15:46:13 Riddell: I hope so. I'll need help to get it properly packaged (I made a few shortcuts) 15:46:50 AlanBell: I agree. Certainly we can read the accessibility text that is sent to Orca directly from the Qt application itself 15:47:07 AlanBell: but yes, test to ensure Orca is getting the right strings is important 15:47:49 reading the final output is sometimes more instructive than the individual strings 15:48:19 That's a good point. No certainly, that is something I will investigate more 15:48:38 There is still plenty to do 15:48:48 I can help you with the speech dispatcher module, I have one that just appends to /tmp/orca.out 15:49:09 That would be great 15:49:37 Unity2D is reasonable for A11Y, but can be better. Every bit helps 15:49:48 Thank you 15:50:12 is the HUD stuff included in this testing? 15:50:41 HUD needs to be written in Unity2D yet :) But yes, it will not be accepted unless there are tests for it 15:50:57 great 15:51:00 AlanBell, unfortunately testability is Qt specific 15:51:13 so unity 3D goes with custom solution - Autopilot 15:51:30 AlanBell: I'm assuming unity-core has unit tests to check HUD works at a lower level 15:51:34 right now the wiki is work in progress but to get some idea: https://wiki.ubuntu.com/Unity/QA/Autopilot 15:51:45 yes, that is exactly the case 15:51:53 mmrazik: any way to create an abstraction layer so that you can reuse the test cases between both? 15:51:59 in matter of fact Mark even published some coverage data on his blog 15:52:20 gema, we were thinking of cucumber 15:52:29 but those are very early thoughts ATM 15:52:33 not real work was done yet 15:52:34 greyback: can it run something like xephyr to avoid fiddling with the running system? 15:52:38 mmrazik: ack 15:52:46 greyback: yeah, I am just trying to push orca testing back in the development lifecycle as far as possible, hopefully as far as design one day! 15:53:03 kalikiana: yes. In fact, I have it set up to run tests on Ubutu inside a virtual box, but controlled by the host machine 15:53:20 kalikiana: it ensures as little changes as possible are made to the sustem being tested 15:53:29 ah, nice 15:53:48 kalikiana: Xephyr isn't quite a "pure" but something I have in mind to also support 15:54:32 greyback: I'm mainly saying xeyphr since it's less overhead and faster 15:54:53 not every potential tester may want to have a vm running 15:55:02 AlanBell: noted. I must admit, it's been some time since I've tested Unty2D and Orca. But we do test them manually during the bug-fixing stage of the cycle 15:55:18 yeah, it is harder then ;) 15:55:26 kalikiana: indeed. You can run these test on your machine too, it just mucks up your desktop a little afterwards :) 15:55:38 stuff like the new shortcut overlay in unity just plain doesn't work with orca as it can't be focussed 15:56:25 Ok. Any other questions? 15:56:55 AlanBell: yep. Unity2D has a problem with the Dash textbox, it can't be navigated with Orca. It needs to be fixed 15:57:36 AlanBell: yes if you should share where I could get that dummy speech dispatcher, I'd be grateful 15:58:02 As we are running out of time -- many thanks for joining us, for good questions and thanks greyback for the talk. 15:58:11 You're welcome! 15:58:15 We are also blogging about our activities on http://qualityhour.wordpress.com/ :) 15:58:32 And we want to make these public meetings on a monthly basis. 15:58:38 So see you next time! 15:58:57 byt 15:58:58 bye 15:59:01 #endmeeting Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/AlanBell/mootbot)