== Meeting information == * #ubuntu-meeting Meeting, 24 Jul at 15:01 — 15:42 UTC * Full logs at [[http://ubottu.com/meetingology/logs/ubuntu-meeting/2013/ubuntu-meeting.2013-07-24-15.01.log.html]] == Meeting summary == === Lightning round === The discussion about "Lightning round" started at 15:02. === Bugs === The discussion about "Bugs" started at 15:13. === Error tracking === The discussion about "Error tracking" started at 15:14. * ''LINK:'' https://code.launchpad.net/~ev/ubuntu-system-settings/diagnostics/+merge/174385 * ''LINK:'' https://wiki.ubuntu.com/ErrorTracker#line-37 * ''LINK:'' https://code.launchpad.net/~ev/apport/grouped_reports * ''LINK:'' http://randomascii.wordpress.com/2013/02/20/symbols-on-linux-part-three-linux-versus-windows/ * ''LINK:'' https://errors.ubuntu.com/retracers-results/ * ''LINK:'' https://errors.ubuntu.com seems to be doing well. We've had 36 people sign the * ''LINK:'' https://launchpad.net/~error-tracker-access * ''LINK:'' https://code.launchpad.net/~ev/errors/whats-unusual-architecture * ''LINK:'' https://wiki.ubuntu.com/ErrorTracker#When_an_update_is_available_to_fix_a_crash * ''LINK:'' https://wiki.ubuntu.com/ErrorTracker#Collecting_extra_information_for_particular_errors * ''LINK:'' https://wiki.ubuntu.com/ErrorTracker/ServerSideHooks * ''LINK:'' https://en.wikipedia.org/wiki/Weak_symbol * ''LINK:'' http://lwn.net/Articles/558600/ if you have a subscription * ''LINK:'' http://lwn.net/SubscriberLink/558106/5222e063b48fd989/ === AOB === The discussion about "AOB" started at 15:40. == Vote results == == Action items == * (none) == People present (lines said) == * ev (220) * cjwatson (60) * jodh (43) * stgraber (14) * barry (9) * ubottu (8) * meetingology (3) * stokachu (1) * doko (1) == Full Log == 15:01 #startmeeting 15:01 Meeting started Wed Jul 24 15:01:56 2013 UTC. The chair is cjwatson. Information about MeetBot at http://wiki.ubuntu.com/meetingology. 15:01 15:01 Available commands: #accept #accepted #action #agree #agreed #chair #commands #endmeeting #endvote #halp #help #idea #info #link #lurk #meetingname #meetingtopic #nick #progress #rejected #replay #restrictlogs #save #startmeeting #subtopic #topic #unchair #undo #unlurk #vote #voters #votesrequired 15:01 np 15:02 [TOPIC] Lightning round 15:02 $ echo $(shuf -e barry doko stgraber jodh ev bdmurray cjwatson xnox) 15:02 bdmurray doko jodh xnox barry stgraber cjwatson ev 15:02 win! 15:02 Hope everyone is here, I didn't have a chance to check the holiday calendar 15:03 cjwatson: I think xnox is on hols. 15:03 mumble was just jodh and barry 15:03 * stgraber waves 15:04 Brian is on holiday too 15:04 doko: around? 15:05 * ev watches the tumbleweeds roll by 15:06 I guess we should move on, maybe he'll catch up 15:06 jodh: 15:06 * foundations-1305-upstart-work-items: 15:06 - python module (for DEP-8 integration tests): 15:06 - Meeting with pitti, jibel and racb re bug 1158391. 15:06 bug 1158391 in Auto Package Testing "ability to have a DEP-8 test run a test in a separate full system environment" [High,Confirmed] https://launchpad.net/bugs/1158391 15:06 - Discussions with xnox+barry on how to get my sub-sub-class tests 15:06 to run correctly. 15:06 - Split tests out of existing module into: 15:06 I'm sure everyone is just at the bank, negotiating how they're going to pay for a "one of a kind" Ubuntu Edge 15:06 - Session-level Upstart Tests (non-priv and root user). 15:06 - System-level Upstart tests (root user only). 15:06 - Added a few new state/session methods to the module. 15:06 - Wrote a DEP-8 "test" to create a chroot environment that will be 15:06 used by the python tests. 15:06 - Made good progress on writing the System-level Upstart tests. 15:06 Currently hitting a D-Bus issue with the module. 15:06 * upstart: 15:07 - Reviewed and merged: 15:07 lp:~ted/upstart/dbus-configure-event 15:07 lp:~ted/upstart/dbus-arguments. 15:07 - Fixed a bug in the Upstart apport hook. 15:07 - upstart/android bridge: 15:07 - wrote a basic bridge over the w/e: 15:07 lp:~jamesodhunt/upstart/upstart-text-bridge 15:07 - still has a silly name. Possible contenders for a better one are: 15:07 - upstart-comms-bridge 15:07 - upstart-injection-bridge 15:07 - upstart-recv-bridge 15:07 - upstart-peer-bridge 15:07 - upstart-client-bridge 15:07 - upstart-host-bridge 15:07 - upstart-proxy-bridge 15:07 - upstart-external-bridge 15:07 Vote now! :) 15:07 upstart-london-bridge 15:07 - Modified watchprops to be an Android service that talks to the upstart-text-bridge: 15:07 http://phablet.ubuntu.com/gitweb?p=ubuntu/upstart-property-watcher.git;a=tree 15:07 ⟁ 15:08 :) 15:08 upstart-edge-bridge maybe? 15:08 assuming xnox is afk 15:09 system-image-updates: LP: #1192585, LP: #1202915, LP: #1202283, LP: #1204090. upload 0.7, working on 0.8. meetings (weekly, QA team). 15:09 Launchpad bug 1192585 in Ubuntu system image "Add a dbus API" [High,Fix released] https://launchpad.net/bugs/1192585 15:09 other: python bug 18531 (**kws and METH_KEYWORDS). LP: #1203374 (turns out, fubar w/ -4 kernel), patch piloting. 15:09 Launchpad bug 1202915 in Ubuntu system image "The client reboots the phone when there's no update" [Critical,Fix released] https://launchpad.net/bugs/1202915 15:09 Launchpad bug 1202283 in Ubuntu system image "system-image-cli -v should display the files that are being downloaded" [Medium,Fix released] https://launchpad.net/bugs/1202283 15:09 done 15:09 Launchpad bug 1204090 in Ubuntu system image "Drop 'device' from client.ini" [Critical,Fix committed] https://launchpad.net/bugs/1204090 15:09 bug 18531 in Ubuntu "lam: new changes from Debian require merging" [Medium,Fix released] https://launchpad.net/bugs/18531 15:09 Launchpad bug 1203374 in HPLIP "hplip fails between 3.12.6-3.1 and 3.13.4-1+b1" [Undecided,New] https://launchpad.net/bugs/1203374 15:09 barry: he's on holiday 15:10 jodh: upstart-simple-bridge? (anyway, whatever we choose is going to be vague and confusing ;)) 15:10 So I unfortunately don't have something to copy/paste 15:10 but I've been fixing system-image stuff last week 15:10 once everything worked I posted https://www.stgraber.org/2013/07/20/introducing-the-ubuntu-touch-image-based-upgrader/ 15:10 stgraber: yeah, maybe that might work ;) 15:10 I wrote a few more tools this week to help with cleanup of system-image and day to day maintenance 15:11 "The images usually boot and work" 15:11 otherwise, I've been doing release-related work this week 15:11 cute 15:11 and I'll be away pretty much all of next week 15:11 DONE 15:11 ev: well, the images I import currently come from "pending" not "current" so I'm ignoring Jenkins, hence the "usually boot" :) 15:11 foundations-1305-click-package: 15:11 - PackageKit backports. 15:12 - Redesigned the hooks specification to be actually implementable, and implemented it. 15:12 :) 15:12 - Helped out Steve Beattie and Jamie Strandboge with the AppArmor hook. 15:12 I haven't had one that didn't boot yet though (on nexus4 and nexus7) 15:12 - Wrote a first-pass desktop file hook. 15:12 - Reviewed, adjusted, and merged "click pkgdir" from Ted Gould. 15:12 Release engineering sprint: 15:12 - Image build pipeline review. 15:12 - Minor fiddling with changelogs.ubuntu.com. 15:12 - Designed livefs builds in Launchpad, with William Grant, Adam Conrad, and Steve Kowalik. 15:12 - Working on proper build cancellation, which we've worked out is a prerequisite for moving livefs builds. Attempting to inhale enough knowledge of how buildd-manager and the buildd slaves work. 15:12 Finally jammed Apache 2.4 into saucy. 15:12 .. 15:12 - Further Cassandra on prodstack fun. We really painted ourselves into a 15:12 corner. Fortunately, we now have a support contract with Acunu \o/, so happy 15:12 times soon. 15:12 - Further work on the Touch UI for error reporting configuration. We ran into 15:12 some interesting problems around using QDBus* in that it doesn't really 15:12 handle service activation well - you need to watch for NameOwnerChanged 15:12 rather than relying on isValid(). That probably makes no sense. Diffs: 15:12 http://bazaar.launchpad.net/~ev/ubuntu-system-settings/diagnostics/revision/133 15:12 http://bazaar.launchpad.net/~ev/ubuntu-system-settings/diagnostics/revision/134 15:12 - Fixes to whoopsie-preferences and shepherding it into main. 15:12 - Participated in Juju GUI user testing. It's looking amazing. I cannot wait to 15:12 use this in production, especially the deployment collections. 15:12 - Learned about weak symbols in GCC and put to use in a unit test of the 15:12 uploading code in whoopsie. This should make developing a broader API between 15:12 whoopsie and daisy a lot easier (need to support report uploads, core 15:12 uploads, server-side hooks, etc). 15:12 - Started learning Acunu Analytics. Deployed to canonistack and started data 15:12 modelling with its API. James is blocking us moving to this until after we've 15:12 finished the Cassandra cluster migration, understandably. We should be able 15:12 to make it fairly safe in production by tuning the Cassandra 1.2 ACLs to only 15:12 give it read access to the Error Tracker column families. It's very 15:13 responsive - this should solve a lot of problems for us, namely showing the 15:13 top problems in a rolling 24 hour period (rather than "20130724") in 15:13 realtime, as well as the "what's interesting about this problem?" section. 15:13 done! 15:13 [TOPIC] Bugs 15:13 I guess we can skip this this week, as we have no Brian, unless anyone has anything to ask about 15:13 nope 15:13 nope 15:14 [TOPIC] Error tracking 15:14 So I asked Evan to lead our discussion topic this week, on errors.ubuntu.com 15:14 grab a warm beverage, it's story time 15:14 I know we typically get novels in the lightning round :-) 15:14 oops, I spoke out of turn :) 15:15 * ev cracks his knuckles 15:15 Hi! 15:15 The biggest thing going on with the Error Tracker is the move from a six node 15:15 cluster running on traditional servers to a 12 node cluster running on juju, 15:15 prodstack, and Ceph (distributed storage, similiar to EBS but not shit). 15:15 But perhaps we can hear a more contextful tale of where we are and where we're going soon here 15:15 Go for it :) 15:15 This was needed for two reasons. 15:15 One, we've overtaxed the existing nodes to the point where they don't have 15:15 enough free space to run compaction (Cassandra doesn't do deletes per se, it 15:15 writes tombstones and cleans up the mess periodically). They're also holding 15:15 too much data. Each node carries about 3TB when they should have no more than 15:15 1TB. This makes everything slower and less fault-tolerant. 15:15 Two, we have no backup. This isn't as scary as it sounds - the data is 15:15 replicated on at least two nodes (previously 3), but it means we cannot upgrade 15:16 Cassandra to a modern version (we're on 1.0.7 hoping to go to 1.2/2.0) without 15:16 fear of it eating the world. We cannot turn on compression which will give us a 15:16 performance increase and probably significantly drive down the storage needed 15:16 per node (crash reports are just textual data - it compresses well). 15:16 In moving to this new 12 node cluster, we enabled Cassandra's multi-datacenter 15:16 replication and told the nodes in the new DC one by one to rebuild themselves 15:16 from pieces of the existing cluster. This fell over in a big heap, partially 15:16 because the existing nodes were already overtaxed (we really painted ourselves 15:16 into a corner) and possibly because we may have found a bug in the replication 15:16 code. This unfortunately stalled what was supposed to be a weekend move into a 15:16 many week affair. 15:16 We're at the point where we need outside help to fix this. We've signed on with 15:16 Acunu, a London-based Cassandra company to provide that support. They're 15:16 getting us to a safe place where we'll do the following: 15:16 - Move into the second cluster. 15:16 - Test upgrading to 1.2 and enable compression in the decomissioned 15:16 real-hardware cluster. 15:16 - Upgrade to 1.2 and enable compression in prodstack. 15:16 - Assimilate the old nodes into prodstack, build a hot-standby and analytics 15:16 (Hadoop in this case) Cassandra cluster. 15:16 Acunu also makes a product called Analytics 15:16 (http://www.acunu.com/uploads/1/1/5/5/11559475/070913_aa_datasheet_v5.pdf), 15:16 which we're going to use to *finally* build the top-k report you see on the 15:16 front page of errors.ubuntu.com over a rolling 24 hour period, 7 day period, 30 15:16 day period, 365 day period, and all time. What's more, we'll have it display in 15:16 real-time (finally as well). It looks likely that it can scale to handle the 15:16 complex set of combinations required to implement the "What's interesting about 15:16 this problem?" section (https://wiki.ubuntu.com/ErrorTracker#unusual) and may 15:16 someday form the basic for the generic metrics collection that mpt, ted, and I 15:16 have dreamed of. 15:16 Once the migration is settled we can get back to hacking on the database. We 15:16 have a few back-population jobs, namely weighting errors 15:16 (https://bugs.launchpad.net/errors/+bug/1069827) that have been languishing for 15:16 some considerable time. This one in particular may be replaced with Analytics 15:16 though. 15:16 James, Tom, and I will also be working on a talk for the London Cassandra 15:16 Ubuntu bug 1069827 in Errors "Error rate incorrectly spikes with any influx of machines" [High,Confirmed] 15:17 community on our experiences with Cassandra, Juju, and Ceph. There are some 15:17 doubts that Ceph will be successful, so it should make for a very interesting 15:17 discussion (and great use case for Juju). 15:17 The headaches around migrating these terabytes of data has got me thinking 15:17 again about internally opening up access to all the report data via Hadoop 15:17 (James is against giving Joe Random Community Member a turing complete system 15:17 with access to the production database). 15:17 I've implemented a Hadoop subordinate charm that we'll quickly deploy on all 15:17 the Cassandra nodes once we're on 1.2. There are also some interesting projects 15:17 around Hadoop and Pig I've been looking into which make monitoring and query 15:17 building a lot easier: 15:17 - http://cloudera.github.io/hue/ 15:17 - https://github.com/twitter/ambrose 15:17 - https://github.com/Netflix/Lipstick 15:17 A lot has changed in the past few months, and CQL 15:17 (https://github.com/datastax/python-driver) is increasingly becoming a 15:17 compelling alternative to the "iter world" use case that Hadoop currently 15:17 handles. It, like Netflix's Astyanax Cassandra client library (Thrift-based), 15:17 uses a thread pool to do a bunch of gets, rather than a blocking get_range. So 15:17 sorry, a bit late 15:17 implementing something on top of this or indeed Astyanax 15:17 (https://github.com/Netflix/astyanax/wiki/All-rows-query) may prove worthwhile. 15:17 If nothing else, we'll definitely start using this approach to speed up our 15:17 other APIs that wrap calls into Cassandra. 15:17 Meanwhile, work continues on getting error reporting up and running for Ubuntu 15:17 Touch. I've split the DBus service that manages error reporting out of 15:17 activity-log-manager/gnome-control-center so it can be used in 15:17 ubuntu-system-settings (the QML settings UI). That latter project now has UI 15:17 for error reporting: 15:17 https://code.launchpad.net/~ev/ubuntu-system-settings/diagnostics/+merge/174385 15:17 Apport now has a noninteractive frontend 15:17 (http://bazaar.launchpad.net/~apport-hackers/apport/trunk/view/head:/data/apport-noui), 15:17 which will handle error reporting for Touch, Server, and desktop systems where 15:17 power users just want to send us stuff: 15:17 https://wiki.ubuntu.com/ErrorTracker#line-37 15:17 We still need to implement enabling this for server via d-i. I've had some 15:17 discussions with the Juju GUI designers about the right place to put this. 15:17 As all of these pieces fall into place, and as click packaging comes online, 15:17 I'll be working to reduce the overhead of Apport. 15:17 It's also worth spending some time on improving the UI for the desktop, and 15:17 someday I'll finish off the branch to group system error reports with the next 15:17 application crash: 15:17 https://code.launchpad.net/~ev/apport/grouped_reports 15:17 With click packaging and the SDK, we'll need some way to get debugging symbols 15:17 for retracing. I've discussed this with Colin and the hope is that we can build 15:17 a symbol server: 15:17 http://randomascii.wordpress.com/2013/02/20/symbols-on-linux-part-three-linux-versus-windows/ 15:17 Working with what we have today, we're still getting ddebs added into the 15:18 Publisher. Last I heard we're waiting on storage. Once that's in place, our 15:18 retrace rate should increase quite dramatically: 15:18 https://errors.ubuntu.com/retracers-results/ 15:18 James and I also have plans to move the retracers into builddstack, hopefully 15:18 with autoscaling. There might be an intermediary of running some in prodstack 15:18 as we are very much unable to keep up with the retracer queue at present. 15:18 We still want to accept more types of errors. If we weren't using an ancient 15:18 version of apport on production (oops!) we'd be getting kernel OOPS reports. I 15:18 have plans to fix and back-populate this once the database migration is over. 15:18 I'm also going to be working with the X/Mir team to start collecting GPU hangs 15:18 and come up with some forward looking plan for porting my error reports from 15:18 application hangs code from compiz to the new world order. 15:18 https://errors.ubuntu.com seems to be doing well. We've had 36 people sign the 15:18 NDA for access: 15:18 https://launchpad.net/~error-tracker-access 15:18 Brian has been working hard on building an API in errors.ubuntu.com for phased 15:18 updates. I believe this is just waiting on some Launchpad bits to land. Brian? 15:18 As mentioned, we've started work on implementing "What's interesting about this 15:18 problem" which will give us a better idea of when a problem is in the library 15:18 underneath the package, or specific to a architecture, locale, or day of the 15:18 week. The code we have for this takes a simplistic (and thus often inaccurate) 15:18 approach to the problem while leveraging a threadpool to do the heavy lifting 15:18 of iterating over hundreds or thousands of reports: 15:18 https://code.launchpad.net/~ev/errors/whats-unusual-architecture 15:18 Moving to Acunu Analytics should fix this. 15:18 I have high hopes of finding time to implement the API for "is this problem 15:18 fixed in a later version of the package" call to be used by Apport: 15:18 https://wiki.ubuntu.com/ErrorTracker#When_an_update_is_available_to_fix_a_crash 15:18 I might build this as a microservice, following what I've learned from watching 15:18 Netflix OSS videos :). 15:18 I'm also hopeful that we'll find the time and resource to implement server-side 15:18 Phased updates: the Launchpad bits have landed, as have Brian's ubuntu-archive-tools patches 15:18 hooks soon, but it requires review from the security team: 15:18 https://wiki.ubuntu.com/ErrorTracker#Collecting_extra_information_for_particular_errors 15:18 https://wiki.ubuntu.com/ErrorTracker/ServerSideHooks 15:18 I think it may just have run into Brian's holiday 15:18 * * * AUDIENCE PARTICIPATION TIME! * * * 15:18 I'd be interested to know if any of you have thoughts on how we can allow 15:18 remote code execution (delivered over SSL, obviously) within the apport hook 15:18 framework, but still allow the hooks to be able to attach the contents of 15:19 Assuming we just say that hooks have to run with the permissions of the user 15:19 that the application crashed under, how would you implement running the hooks 15:19 under that user? Upstart user session jobs? 15:19 Finally, as a random aside, I discovered GCC's weak symbols: 15:19 https://en.wikipedia.org/wiki/Weak_symbol 15:19 This seems to solve all my crying over not being able to sparingly mock in C. 15:19 Expect lots of this to surface in whoopsie as I add support for server-side 15:19 hooks and problem fixed notifications. 15:19 Also, C++'s RAII kind of confuses me in it not being obvious where memory is 15:19 allocated for all these Qt objects, but does seem to produce some clean, safely 15:19 managed code. 15:19 As always, I could really use your help. If you want to volunteer for code 15:19 review, let me know and I'll add you to ~daisy-pluckers. If you want to try 15:19 your hand at hacking on it, let me know and I will gladly handhold you through 15:19 setting everything up. The same goes for anyone you know who might be 15:19 interested. It's pretty easy, it deploys with juju in a few short commands once 15:19 your instance limits on prodstack have been increased. 15:19 Thanks guys. 15:19 \o/ 15:20 ev: There was an article about "CMocka" in this week's LWN 15:20 http://lwn.net/Articles/558600/ if you have a subscription 15:20 oh? I didn't think we still had access 15:20 Some of us are Debian developers :) 15:20 :) 15:21 Ubuntu members still have access (at least I do) 15:21 I can't even make a beard joke anymore 15:21 because you no longer have one 15:21 http://lwn.net/SubscriberLink/558106/5222e063b48fd989/ 15:21 and I do 15:21 yay 15:21 thanks 15:22 Running hooks under the user> does it actually need anything from the user's session, or just need the user's file privileges? 15:24 I'm not sure I follow. 15:24 ev: can you explain why we can't encode the required logic in existing apport hooks? 15:24 presumably just the latter 15:25 jodh: they will largely be the existing hooks, but they cannot have interactive UI 15:25 Things you might need from the user's session: environment variables set by upstart, connections to dbus session bus, etc. 15:25 so no password prompts, no yes/no dialogs, etc 15:25 But if you just need to reliably read files owned by the user, switching uid is sufficient 15:25 ah right 15:26 and it's relatively easy to read the list of upstart user sessions and dump the environment from there if you need to 15:26 switching uid -> have something privileged that looks for the hooks and runs them, first dropping privs to the right uid? 15:27 Right, if you're planning on doing things as arbitrary users then you must have a privileged dispatcher anyway 15:27 prsumably upstart can be the privileged dispatcher? 15:27 Might be worth remembering to set $HOME. Aside from that it really depends what else you need 15:27 ev: still not clear. If apport now has a noninteractive frontend, what are we needing to do "dynamically"? Can you give some examples? 15:27 It could be, but that might be more trouble than it's worth if you just want to spawn a subprocess and wait for it to finish 15:27 yeah, good point 15:28 click certainly doesn't use Upstart jobs when it's executing user-level hooks, for instance 15:28 It would be possible but would really involve way too much runaround 15:28 * ev nods 15:29 Symbol server: ack, though this is at the handwave level right now. Is darkserver free software? 15:30 Apparently so, http://fedoraproject.org/wiki/Darkserver 15:30 May indeed be worth a look; the less we have to write new server code the better ... 15:30 jodh: the hooks will be dynamically executed in that whoopsie will send a report to daisy, daisy will reply back with "oh, a developer wants you to run this hook code. HERE", and then something will take care of launching a python subprocess as the right user with the hook code 15:30 folks in F/RH land were raving about this at pycon 15:31 this of course doesn't handle operations that require raised privileges, like give me xorg.0.log 15:31 ddebs/publisher> Blocked on the SAN move, which we really rather hope happens soon before the librarian runs out of space, but we're assured it will 15:31 I believe it's been decoupled from the precise upgrade now, which should help 15:31 cjwatson: following our discussion about software engineering largely being an exercise in shopping, I agree wholeheartedly 15:31 I'll have a look 15:31 ev: ok, but can you give a concrete of example of what a dev might want you to run that cannot be pushed out as a standard apport hook? 15:32 jodh: we don't want to ship them as standard apport hooks not because they can accomplish something more, but because we can deliver them without involving publisher cycles and the user hitting the upgrade button 15:33 also because they can accomplish more :) - we can target hooks to a specific problem 15:33 or package, or $world 15:33 I've certainly had cases of ad-hoc things I want people to run that I wouldn't want to push out to the wordl 15:33 Not that I can think of specific examples right now 15:33 ev: so if this feature is introduced, how would these dynamic chunks of code get tested reliably before delivery? 15:33 But I think it has happened with us all 15:33 Of course the security properties are rather scary 15:33 jodh: a code review approach, but if they catch fire, they wont bring down the user's system 15:34 see https://wiki.ubuntu.com/ErrorTracker#Collecting_extra_information_for_particular_errors for the nitty gritty 15:34 very scary 15:34 ev: I was about to use those same words :) 15:34 but it's no more scary than a rogue ubuntu developer 15:34 If it relates to click packages then I'd suggest that the extra chunk of code should run with the apparmor profile of the click app 15:34 we're only giving ubuntu-devs access to this, and they already have root on your machien 15:34 * ev nods 15:34 yes, definitely 15:37 OK, any other questions? I've certainly been gratified by the steadily increasing amount of information we're getting from errors, and the extent to which we're putting it to use 15:37 yay! thank you very much 15:37 Particularly in assorted SRUs 15:38 it helps to hear that it's useful 15:39 ev: very impressive, thanks for presenting this 15:39 aw shucks. Thanks 15:40 #topic AOB 15:41 Anything else people have today? 15:41 nope 15:42 I'm all chatted out 15:42 #endmeeting Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/meetingology)