== Meeting information == * #ubuntu-meeting Meeting, 07 Aug at 15:01 — 15:49 UTC * Full logs at [[http://ubottu.com/meetingology/logs/ubuntu-meeting/2013/ubuntu-meeting.2013-08-07-15.01.log.html]] == Meeting summary == === lightning round === The discussion about "lightning round" started at 15:01. === AOB === The discussion about "AOB" started at 15:20. === Phased updates === The discussion about "Phased updates" started at 15:24. * ''LINK:'' http://people.canonical.com/~ubuntu-archive/phased-updates.html * ''LINK:'' http://people.canonical.com/~brian/tmp/phased-updates.html == Vote results == == Action items == * (none) == People present (lines said) == * ev (70) * slangasek (55) * jodh (52) * bdmurray (43) * xnox (24) * cjwatson (24) * stokachu (15) * doko (12) * ubottu (8) * meetingology (3) * stgraber (1) == Full Log == 15:01 #startmeeting 15:01 Meeting started Wed Aug 7 15:01:19 2013 UTC. The chair is slangasek. Information about MeetBot at http://wiki.ubuntu.com/meetingology. 15:01 15:01 Available commands: #accept #accepted #action #agree #agreed #chair #commands #endmeeting #endvote #halp #help #idea #info #link #lurk #meetingname #meetingtopic #nick #progress #rejected #replay #restrictlogs #save #startmeeting #subtopic #topic #unchair #undo #unlurk #vote #voters #votesrequired 15:01 [TOPIC] lightning round 15:01 $ echo $(shuf -e barry doko stgraber jodh ev bdmurray slangasek cjwatson xnox stokachu) 15:01 bdmurray slangasek ev xnox barry jodh stokachu stgraber doko cjwatson 15:01 (and barry and stgraber are off today) 15:02 this covers some of the stuff I did before my vacation 15:02 updated errors not to return failed buckets (LP: #1202416) 15:02 updated daisy branch to cache packages for more teams 15:02 various package to team mapping work 15:02 set up mailing lists for new package to team mapping teams 15:02 arsenal modifications for package to team mapping work 15:02 Launchpad bug 1202416 in Errors "package-version-new-buckets should not return failed retraces" [Medium,Fix released] https://launchpad.net/bugs/1202416 15:02 modified arsenal to better handle the search criteria with negated tags 15:02 submitted merge proposal for unsubbed-packages to ubuntu-archive-tools 15:02 updated sru-release to set phased_update_percentage to 10 and submitted merge proposal for it 15:02 tested the phased-updater on lillypilly 15:03 worked on SRU verification of bug 1205374 15:03 upload of ubuntu-release-upgrader to saucy fixing a test failure 15:03 testing of ubuntu-release-upgrader unsupported release dialog 15:03 bug 1205374 in whoopsie (Ubuntu Raring) "Only attempts to retry the existing crash reports once, after two hours." [Undecided,Fix committed] https://launchpad.net/bugs/1205374 15:03 ✔ done 15:04 * client sprint on the IoM; it was nice to be in everyone's timezone last week :) 15:04 * prepping for DebConf next week 15:04 * prepping for 12.04.3, the week after DebConf 15:05 (done) 15:05 - Discussions with pitti and the QA team on automatically uploading to whoopsie 15:05 when there are crashes in jenkins: 15:05 http://bazaar.launchpad.net/~apport-hackers/apport/trunk/view/head:/data/whoopsie-upload-all 15:05 - Discussions with the MIR team on reporting errors for hanging applications in 15:05 Touch. Some investigation of Android's ANR (application not responding) and 15:05 the Mir architecture. Blocked on waiting for a Nexus 4 (nvidia on the 7 makes 15:05 running Mir a difficult task). 15:05 - Created and tested an upstart job for automatic error reporting. Updated the 15:05 Touch seeds to use this new package. 15:05 - Built out some functional tests for the whoopsie-preferences daemon, after 15:05 the MIR requested (but didn't block on) them: 15:05 https://code.launchpad.net/~ev/whoopsie-preferences/functional-tests 15:05 - Work on generating a unique system identifier on touch devices. Blocked on 15:05 waiting for the Nexus 4 (need a SIM for getting an IMEI): 15:05 https://code.launchpad.net/~ev/whoopsie/android-serial/+merge/178306 15:05 - Fighting errors that have cropped up in recent revisions of the charms 15:05 (haproxy, cassandra, etc). With these fixes in place, we're now running the 15:05 entire infrastructure (on canonistack) with gojuju. This should hopefully 15:05 speed up RT 58019 and was necessary to test running the retracers with the 15:05 latest and greatest apport: 15:05 - https://code.launchpad.net/~ev/canonical-marshal/cassandra-dont-assume-bzr/+merge/178520 15:05 - https://code.launchpad.net/~ev/canonical-marshal/haproxy-write-listen-stanza/+merge/178569 15:05 - Meeting with Katherine to discuss automatic error reporting on Touch. 15:05 Followed up with Cimi and Katie Taylor on getting a bit of text added to the 15:05 first use experience. Katie will get the exact text from Katherine, but I 15:05 think we're safe to enable this. 15:05 - Fixing kernel OOPS reporting to daisy.ubuntu.com by testing and updating 15:05 our version of apport. 15:05 http://bazaar.launchpad.net/~daisy-pluckers/daisy/trunk/revision/387 15:05 Updates from webops: 15:05 - The backup of the Cassandra DC cluster and initial Cassandra prodstack 15:05 cluster (etstack - what we built to migrate away from the ENOSPC disaster, 15:05 not to be confused with the production prodstack cluster) is now complete. We 15:06 have a meeting with Acunu in the office on Monday to have a little cry and 15:06 maybe fix some things. 15:06 - Moving the retracers on prodstack is still waiting for action by the webops 15:06 team (https://portal.admin.canonical.com/58019). Tom suggested it was a 15:06 resource problem on their end. I'll keep prodding. 15:06 Random: 15:06 - I'm going to start subscribing the team to merge proposals for projects I've 15:06 written in C. I'll do my best to get involved in upstart MPs in return. 15:06 (done!) 15:06 ev: you rock! :) 15:07 ev: N4> is the PO raised? 15:07 slangasek: it is - should be here by the end of the week 15:08 huzzah 15:08 * catched up on some merges/syncs, i TIL. 15:08 * finished multiarch of tk/tcl 8.6. 15:08 * multiarched boost-dev for cross-building (for doko) 15:08 * fixed privileges dropping in ubiquity since migration to pkexec 15:08 ( thus fixing U1 page ) 15:08 * Fighting the emulator for Ubuntu touch (legacy, unflipped, arm at 15:08 the moment): 15:08 - so far the emulator is winning 15:08 - it boots and mounts filesystems, but linker gets unresolved 15:08 symbols when trying to run ubuntuappmanager. (from AOSP build) 15:08 - or fails to mount filesystems. (from phablet build) 15:08 - trying to mix & match until in works 15:08 .. 15:08 * foundations-1305-upstart-work-items: 15:08 - upstart dep-8 integration tests: 15:08 - Reworked MP for lp:~jamesodhunt/upstart/python-upstart-module 15:08 - DEP-8 test code now finished, but not fully tested (read on). 15:08 Has to perform some interesting setup: 15:08 - Start a nested VM and timeout-wait for it to boot. 15:08 - Configure nested VM: 15:09 - Copy the source tree to the VM. 15:09 - Enable Upstart debug mode by updating grub config. 15:09 - Manually install test dependencies. 15:09 - Create a test job to check for a successful boot (yes, 15:09 cloud-init does this too, but just in case... :) 15:09 - Create a chroot environment. 15:09 - Reboot the VM and timeout-wait for it to boot. 15:09 - Run the tests via ssh in the background to avoid hanging 15:09 indefinately should the nested VMs kernel panic. 15:09 - Timeout if the test takes too long to run or the VM dies/cannot 15:09 be connected to. 15:09 - Handle the scenarios where the test *does* kill the nested VM: 15:09 - Reboot it, forcing an fsck and wait for it to come back up. 15:09 - Copy the test results data from the nested VM back into the 15:09 autopkg VM for collection and presentation via Jenkins. 15:09 - Some of the above setup will hopefully be added to autopkgtest 15:09 when we have this working well. 15:09 - However, this work item is blocked on bug 1208455: 15:09 bug 1208455 in linux (Ubuntu) "general protection fault running apt-get inside double nested kvm VM" [High,Incomplete] https://launchpad.net/bugs/1208455 15:09 - Discussions with kernel and QA teams. 15:09 - Tried forcing nested mode for all *three* environments (host, 15:10 autopkgtest vm and pristine "nested" (actually double-nested)) 15:10 vm, but still get kernel panic. 15:10 - Test code now updated to force nested mode in the 2 environments 15:10 it can configure now, just as a precaution. 15:10 - Part of the explanation for the double-nested vm running at 200% 15:10 cpu in OpenStack turned out to be bug 1208853. 15:10 - Currently trying with linux-image-3.11.0-0-generic from 15:10 bug 1208853 in byobu (Ubuntu) "dozens of byobu-status processes running on ubuntu server" [Critical,Fix released] https://launchpad.net/bugs/1208853 15:10 jodh: heard anything back from the kernel team yet wrt the nested kvm horrors? 15:10 '-proposed' as recommended by the kernel team. 15:10 * other: 15:10 - Reviewed lp:~ev/whoopsie/be-more-verbose. 15:10 - Worked on DebConf presentation. 15:10 ʡ 15:10 (oh and thanks xnox for the code review on the android system identifier stuff!) 15:10 jodh: also, do you have console output from the nested VM? I'm inclined to think this would be more reliable than just driving it over ssh 15:10 been chatting in #ubuntu-kernel about it most of today. 15:10 my "best hope" seems to be a 3.11 kernel (testing currently in progress), else ....? 15:11 bug 1121874 and bug 1207123 are uploaded and just need sru approval 15:11 bug 1121874 in mysql-5.5 (Ubuntu Raring) "MySQL launch fails silently if < 4MB of disk space is available" [Medium,Triaged] https://launchpad.net/bugs/1121874 15:11 done 15:11 bug 1207123 in gnutls26 (Ubuntu Precise) "Key usage violation in certificate has been detected" [Undecided,New] https://launchpad.net/bugs/1207123 15:11 ev: C merge proposal reviews is nice =) i <3 C 15:11 jodh: ok 15:11 don't we all 15:12 stokachu: ok, noted :) 15:12 we have the console log on the bug. 15:12 doko: your turn 15:12 thanks! 15:12 ev: i've been reading up on "The C++ programming language" (4th edition, updated with C++'11) and i'm starting to drink the kool-aid =) ps. i can share the book, if anyone else is interested. 15:12 - AArch64 bringup (now normal and buildd chroots can be, and finally are built natively) 15:12 - Test two Linaro changes to fix two regressions. 15:12 - Try to cross-build unity8, first getting b-d's all installed. 15:12 slangasek: we capture the console output as a "test artifact" 15:12 (done) 15:13 slangasek: I did think about running the tests directly as upstart jobs. we could explore that if required. 15:13 xnox: C++> I'm so sorry 15:13 xnox: I'm very tempted to give it another try, mostly because we're building our SDK on top of it. Also because I'm curious about RAII. 15:13 foundations-1305-click-package: 15:13 - File format tweak to support MIME type detection. 15:13 - Move MIME type declaration from shared-mime-info to click, per shared-mime-info upstream. 15:13 - Scary scary improvement in handling of removal of old links for single-version hooks. (Nobody seems to have noticed the previous bug yet ...) 15:13 - Add a sort of virtual hook facility (Hook-Name in .hook files), which should support Ted's Upstart-based desktop hook. 15:13 foundations-1305-arm64-bringup: 15:13 - Finally sorted out the auto-cross-builder again. 15:13 xnox: I'm interested :) 15:13 - Diagnosed and fixed http://bugs.debian.org/718482. 15:13 cjwatson: lol 15:13 Debian bug 718482 in apt "apt: CompareProviders ranks Priority above native architecture" [Normal,Open] 15:13 Prepared https://code.launchpad.net/~cjwatson/launchpad/series-alias/+merge/178103 to make "devel" symlinks work. 15:13 Review-induced tweaks to https://code.launchpad.net/~cjwatson/launchpad/buildstatus-aborted/+merge/176990. 15:13 jodh: I'm arguing that the console output should be used to detect VM crashes, and possibly even for driving the tests themselves 15:13 Tested bug 1205407 for raring. 15:13 bug 1205407 in shadow (Debian) "su: kill child process group on signal, not just immediate child" [Unknown,Fix committed] https://launchpad.net/bugs/1205407 15:13 foundations-r-phased-updates: Helped to deploy the phased updater. 15:13 PuTTY security update. 15:13 Fighting qemu to get 4K/4K disk tests working again. So far, failed. 15:13 Preparing daily quality talk for DebConf. 15:13 .. 15:14 jodh: because using anything other than the console makes detection more heuristic 15:14 jodh: if you could configure everything to run noninteractively within the VM, and capture all the needed results from the VM console output, I think that's going to be most reliable *and* most efficient 15:15 ev: RAII is quite intersting, given that it's scope save without requiring to wrap your code in a try/with block with a "finally:" as one does in python. There is no finally. And things like initialisations and "auto" types are all very nice. 15:15 slangasek: maybe we should discuss after the meeting? I'm not clear how we'd drive tests via the console but in the background. I know there are socket options on qemu but would need to dig into the manual. 15:15 (no races with ssh startup, for one thing) 15:15 ev: re: gojuju are you still writing all hooks in bash? 15:16 jodh: well, I'm perhaps assuming you aren't so much driving the tests via console as you are consuming the results 15:16 cjwatson: i started with a rather large article published by microsoft "why one shall not use c++ for drivers/kernel mode" and that set quite a few things straight to start with. 15:16 stokachu: for the moment, though I'd like to move to the new python libraries 15:16 it's just a matter of time 15:16 this is already working 15:16 ev: cool just curious what everyone is using 15:16 sure 15:16 jodh: yes, it should be possible to wire the VM's console up to a pty and drive it interactively, but I don't think that should actually be necessary - anyway, sure, we can talk more after the meeting 15:16 I think python is the way to go for future charms 15:16 stokachu: the u1 charms are worth looking at for best practices 15:16 slangasek: so I guess you're implying we do run the tests via an upstart job. I can look at doing that, but if this needs to happen fast, I guess we can use what we have as long as we can overcome this kernel issue? 15:16 doko: chroots built natively - huzzah! 15:17 ev: definately, we just got done writing our cts app deployment in bash 15:17 ev: cool ill take a look at those 15:17 stokachu: i have some in python and some in shell. anything beyond package installation & relationships, I preffer the python modules. 15:17 slangasek, well, last two or three packages still building ... 15:17 stokachu: sidnei is the guy to talk to for pointers on their codebase / access 15:17 slangasek: ok, note that I'm essentially emulating some of what the autopkg test infrastructure does atm. 15:17 and sure thing 15:17 thanks :D 15:18 ev: new python libraries for use with gojuju? curious 15:18 All my charms are still in shell, largely because I haven't written anything new since switching to Go Juju 15:19 slangasek: yeah somebody did a mini python module, with useful hooks / functions which has been cargo culted for a while. 15:19 slangasek: weird, right? :) 15:19 * xnox doesn't want to come anywhere near the "official" nginx charm. it's written in php. 15:19 ha! 15:19 jodh: yes, tests via upstart job; yes, what you currently have is enough to be getting on with - I just think you will find that moving it to be more noninteractive and capturing from the console will significantly simplify things and make it more robust, so we don't have to worry about false test failures 15:20 Any other questions wrt lightning round? 15:20 [TOPIC] AOB 15:21 php, because, well: https://github.com/search?p=1&q=extension%3Aphp+mysql_query+%24_GET&ref=searchresults&type=Code 15:21 wat 15:21 that's not a charm, that's a hex 15:22 LOL 15:22 :D 15:22 though I guess since it's php, it will be implicitly *cast* from hex to charm 15:22 half the time 15:22 ANYWAY 15:22 so I had one thing for AOB 15:22 ev: scary ;) 15:22 half the team is at DebConf next week 15:22 i sense dissatisfaction 15:22 wooooo party for the rest of us 15:22 I'm buying a hammock 15:23 that leaves just ev, bdmurray, barry, and stokachu here 15:23 if you guys want to have a meeting next week, feel free? 15:23 but don't expect the rest of us to show :) 15:23 they get to do everyone else's work then 15:23 eeeeexcellent 15:23 if bdmurray doesnt mind ill just ping him directly next week on any bugs 15:23 stokachu: that'd be fine 15:23 thanks man 15:24 cjwatson: I'm going to subcontract your work to the sales team. 15:24 any other Any Other Business? 15:24 nope 15:24 no topic this week? 15:24 ev: i have flashbacks.... 15:24 stokachu: shortly :) 15:24 ah ok 15:24 [TOPIC] Phased updates 15:24 * slangasek turns the floor over to bdmurray 15:25 who can tell us all about the awesome results of integrating errors.u.c with our SRU process! 15:25 For a while now we've wanted to gradually roll out packages in -updates to users. 15:26 update-manager has had this support for a while but until recently we were lacking server side support for this 15:26 Thanks to some help from cjwatson and ev this now done. 15:26 WOOHOOO! 15:27 Starting right now (for raring) when a SRU is released it will have the phased-update-percentage set to 10%. 15:28 Every 6 hours there is a job that checks for regressions in that version of the package and if there are none the p-u-p is incremented by 10%. 15:28 how are these users choosen? 15:28 If a user uses update-manager to install updates, update-manager chooses for them. 15:29 It's possible to choose to install all updates being phased or none. 15:29 doko: does that answer your question? 15:29 yes, thanks 15:30 so what are the regression checks? 15:30 so 10% of the phased ones 15:30 detail: update-manager generates a random number which is consistent across runs for each machine and package 15:30 So if the p-u-p is 10% then that random number [0,1] must be <=0.1 15:30 Or something along those lines 15:30 Thanks cjwatson 15:31 So we use errors.ubuntu.com to see if there are any crashes reported about the new version of the package (the one in -updates) that were not reported about the previous version of the package. 15:31 bdmurray: how was the value 6 hours chosen? 15:33 jodh: I just picked something that didn't seem too fast 15:33 yeah, it's a pretty arbitrary value; we may have to tune it later 15:33 bdmurray: I just wondered if it should be >24 hours to allow for those systems that are only updated daily? 15:33 but it's intended to ensure the roll-out is slow enough that we can collect meaningful data on errors.u.c in each interval 15:34 jodh: all systems are only updated daily, but there are a lot of systems 15:34 jodh: then it would take about 9 days for everybody to get an update, which seems really long 15:34 * slangasek nods 15:35 anyway, not everybody applies the updates as soon as they're visible... some users let updates linger for days or weeks before applying them 15:35 In addition to checking for new crashes, the error tracker is also queried to see if there is an increased rate of crashes about the package. 15:35 so we know there's a lag time between the update being published and reports of regressions coming in, though we don't know exactly how long that lag is 15:35 The initial run took about 10 minutes, so every six hours is probably reasonable enough load 15:37 If either type of regression is detected then the phasing of the update is stopped by setting it to 0. 15:37 This will prevent other users from receiving the updated version of the package. 15:38 The job also generates an html report of packages currently undergoing phasing which displays the p-u-p and any detected regressions. 15:39 Addiitionally, an email is sent to the signer of the package (uploader) and its creator (either the uploader or sponsee). 15:39 The email notifies them of the problem and that the phasing has been stopped. 15:39 bdmurray: do you have the link to that html report? 15:40 Yes, but its empty right now. 15:40 http://people.canonical.com/~ubuntu-archive/phased-updates.html 15:40 cjwatson: "intial run took about 10 minutes" what does that run mean? to generate report, or for a a first regression to be filed in errors since first phased publishing? 15:40 An example of its content can be found here 15:40 http://people.canonical.com/~brian/tmp/phased-updates.html 15:40 xnox: to generate the report and review the updates 15:41 ack. 15:41 There is also support in the phased-updater to override specific problems, if a crash is determined not to be a regression. 15:43 So once the SRU team starts releasing packages to raring-updates we'll see some activity. 15:43 I think that covers it. 15:43 congrats to bdmurray, ev, cjwatson for getting this landed 15:44 very exciting to see us leveraging errors.u.c in this way 15:44 thanks for all the heavy lifting, guys! 15:44 me too! :D 15:44 any more audience questions? :) 15:44 bdmurray: is there a plan for handling phased updates to server systems? 15:45 I'm not aware of one and I believe ev is just working on error reporting for servers. 15:45 FLAGRANT SYSTEM ERROR 15:45 Computer over. 15:45 Virus = Very Yes. 15:45 very yes? 15:45 it's a joke (that's sadly going away) 15:45 you've encountered a bug in the code 15:45 strOngBaD 15:45 * ev looks it up 15:47 doko: appears to be a bug in django_openid_auth (https://oops.canonical.com/oops/?oopsid=OOPS-180ace38c5a200b3c30ff08a1f98505c for those in ~daisy-pluckers). I'll see what I can do. 15:48 ?! 15:48 ok, sounds like that's it for the meeting 15:49 y'all can dissect django on another channel :) 15:49 #endmeeting Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/meetingology)