15:03 <robru> #startmeeting Foundations Team Meeting
15:03 <meetingology> Meeting started Thu Aug  7 15:03:10 2014 UTC.  The chair is robru. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
15:03 <meetingology> 
15:03 <meetingology> Available commands: action commands idea info link nick
15:03 * xnox o/
15:03 <robru> #topic Lightning round
15:03 <robru> ok who is actually here for the meeting?
15:04 <cjwatson> o/
15:04 <barry> o/
15:04 <xnox> o/
15:04 * stgraber waves
15:04 <bdmurray> \o
15:04 <sil2100> We can always shuffle from the whole list and just skip people that are not around
15:04 <robru> ok who has the whole list? ;-)
15:04 <sil2100> Not super optimal, but it worked ;)
15:04 * sil2100 looks at slangasek
15:04 <xnox> and doko
15:05 <slangasek> echo $(shuf -e barry doko stgraber jodh bdmurray slangasek cjwatson xnox caribou infinity mvo bhuey sil2100 robru)
15:05 <slangasek> caribou sil2100 doko infinity jodh bhuey bdmurray mvo xnox slangasek cjwatson robru stgraber barry
15:05 <slangasek> there :)
15:05 <barry> win!
15:05 <robru> slangasek, thanks, I need to make an alias for that ;-)
15:05 <robru> i guess no caribou then, so sil2100 starts?
15:06 <sil2100> One moment!
15:06 <robru> sil2100, sil2100, sil2100, sil2100!! go go go!
15:07 <sil2100> - Helping out with C++ debian packaging symbols handling
15:07 <sil2100> - Landing team work, landing e-mails, landing coordination - standard stuff
15:07 <sil2100> - Double trainguard shift on Monday
15:07 <sil2100> - CI Train maintenance and features:
15:07 <sil2100> * More work on enabling for RTM
15:07 <sil2100> * Adding many workarounds to the RTM branch to enable working with both LP and DF at once (no code-hosting on DF)
15:07 <sil2100> * More testing of a complete landing to ubuntu-rtm
15:07 <sil2100> * Enabling copy2distro to temporarily support the testing preprod RTM features
15:07 <sil2100> * Work on the twin package upload checks better
15:07 <sil2100> * Switching to proper distro-based silo names (e.g. ubuntu-rtm/landing-001)
15:07 <sil2100> - Packaging advice and support for some upstreams
15:07 <sil2100> - Discussions related to RTM support in the CI Train
15:07 <sil2100> - Discussions regarding beta testing readiness with QA
15:07 <sil2100> - Discussions on QA rules for ubuntu-rtm
15:07 <sil2100> - Announcing TRAINCON-0 due to the image situation
15:07 <sil2100> - Dealing with TRAINCON-0: pushing upstreams, information flow, decision-making etc.
15:07 <sil2100> * Also helping out with some of the fixes for blockers
15:07 <sil2100> - Started review of a ubuntu-keyboard branch
15:07 <sil2100> - Many many other things that got lost in the chaos
15:07 <sil2100> (done)
15:08 <robru> doko, you're up
15:08 <xnox> what's "copy2distro" ?
15:08 <doko> not soo fast
15:08 <doko> - three days of main merges
15:08 <doko> - various Debian NMUs to be able to drop the delta
15:08 <doko> - more pestering about component mismatches
15:08 <doko> - MIR's
15:08 <doko> - working on the aarch64 multilib toolchain
15:08 <doko> - finally finished the openmpi transition (which was started in April ...)
15:08 <doko> - more openjdk-6 / openjdk-7 patches, mentoring
15:08 <doko> - packaging review of some third party software
15:08 <doko> (done)
15:09 <sil2100> xnox: it's the CI Train side that runs on snakefruit, that actually pushes the packages to the archive
15:09 <cjwatson> xnox: from lp:cupstream2distro - it's the privileged component that CI Train talks to in order to copy stuff into Ubuntu
15:09 <xnox> right. thanks.
15:09 <cjwatson> I believe infinity and jodh are both off today
15:10 <cjwatson> Is bhuey around?
15:10 <robru> bhuey, around?
15:10 <sil2100> Most of my knowledge about copy2distro actually comes from this week, never saw it before as this part was a bit archive-admin specific ;)
15:10 <bdmurray> and I heard bhuey was out from slangasek
15:10 <bdmurray> so I'll start
15:10 <cjwatson> OK
15:10 <bdmurray> requested saved .crash files from a retracer for manual retracing
15:10 <bdmurray> research into armhf retracing failures on precise
15:10 <bdmurray> reported gdb bug 1351018 regarding threads on precise
15:10 <bdmurray> discovered an issue with the retracers trying to write CoreDump to the stacktrace column family (fixed in r507)
15:10 <ubottu> bug 1352591 in apport (Ubuntu) "duplicate for #1351018 apport-retrace does not update libraries in a sandbox" [Undecided,New] https://launchpad.net/bugs/1352591
15:10 <bdmurray> submitted RT to update retracers to r507
15:10 <bdmurray> updated and tested daisy using counters for "rootfs build" and "device image"
15:10 <bdmurray> submitted RT to have daisy updated to r508 (counters for images)
15:10 <bdmurray> verified new counters exist in DayBucketsCount
15:10 <bdmurray> pushed errors updates to be able to query for rootfs_build_version and device_image_version using the API on errors
15:10 <bdmurray> pinged webops about the retracers being stuck (lost cassandra connection on the 3rd)
15:10 <bdmurray> debugging apport-retrace / gdb issue on retracers with armhf crashes (LP: #1351018)
15:10 <bdmurray> reported apport bug 1352591 (sandbox libs not updated) source of bug gdb issue
15:11 <ubottu> bug 1352591 in apport (Ubuntu) "apport-retrace does not update libraries in a sandbox" [Undecided,New] https://launchpad.net/bugs/1352591
15:11 <bdmurray> reported apport bug 1352450 regarding apport-retrace options
15:11 <ubottu> bug 1352450 in apport (Ubuntu) "apport-retrace should indicate that -s and -o don't work together" [Low,Fix committed] https://launchpad.net/bugs/1352450
15:11 <bdmurray> pinged webops about updating ubuntu_assets_url to 492 (package version selection filtering)
15:11 <bdmurray> updated daisy's submit.py code to only retry retracing crashes without third-party-packages
15:11 <bdmurray> updated and tested creation of SystemImages column family in daisy (r511, r512)
15:11 <bdmurray> confirmed that SystemImage column family exists and is populated
15:11 <bdmurray> rewriting errors frontends to allow for selecction of rootfs build or device image
15:11 <bdmurray> ✔ done
15:11 <xnox> mvo is out.
15:11 <xnox> me.
15:12 <xnox> * split upstart package bug #1351306
15:12 <xnox> * merge proposals to stabilise whoopsie ids on ubuntu-touch & more
15:12 <xnox> * fixed bug #1326327
15:12 <xnox> * fixed bug #1351295
15:12 <xnox> * emailed outstanding systemd units (mostly cloud)
15:12 <ubottu> bug 1351306 in upstart (Ubuntu) "Cannot uninstall upstart and install systemd-sysv" [Low,Fix released] https://launchpad.net/bugs/1351306
15:12 <xnox> * looking into 12.04.5 picked up ubiquity issue:
15:12 <xnox> self.controller.get_string does not work at import time, hence this
15:12 <ubottu> bug 1326327 in debhelper (Ubuntu) "dh_installinit should generated update-rc.d remove to remove rc*.d symlinks" [Medium,Fix committed] https://launchpad.net/bugs/1326327
15:12 <ubottu> bug 1351295 in initramfs-tools (Ubuntu) "Boot fails if /sbin/init (or init= arg) is an absolute symlink" [Undecided,Fix released] https://launchpad.net/bugs/1351295
15:12 <xnox> broke OEM installation on 2.1 QEMU which returns a too long
15:12 <xnox> autogenerated hostname.
15:12 <xnox> * today mostly coordinating EOE & fixing other spotted ubiquity UX
15:12 <xnox> bugs
15:12 <xnox> * Todo for tomorrow:
15:12 <xnox> lvm2 merge, btrfs-tools update, mdadm update, ubiquity UX, and then
15:12 <xnox> EOE. :`(
15:12 <xnox> ..
15:12 <robru> xnox, EOE but not EOL!
15:12 <xnox> =) yeap End Of Employment
15:13 <cjwatson> last team meeting!  good luck at the next gig ...
15:13 <slangasek> indeed, good luck :)
15:13 <slangasek> we'll have to have the good-bye drinks in Portland :P
15:13 <slangasek> * work to make sure all the packages on the phone have debugging symbols
15:13 <xnox> =)
15:13 <robru> slangasek, so I know you said you wouldn't be here for this meeting, but since you're here.... ;-)
15:14 <slangasek> * removed Qt4 from the phone
15:14 <sil2100> ;)
15:14 <slangasek> * mid-cycle cloud sprint in Nuremberg
15:14 <slangasek> * demo of a transactionally-updated server image
15:14 <slangasek> * lots of good conversations around go, POWER, testing, image matrices, etc
15:14 <slangasek> * next week:
15:14 <slangasek> robru: yeah, a meeting disappeared from my schedule at the last minute :)
15:14 <slangasek> * on vacation Mon-Thu
15:14 <slangasek> (done)
15:14 <cjwatson> Cloud sprint.  Lots of discussions, some highlights:
15:14 <cjwatson> - Worked around the sides of the ubuntu-core system-image build project.
15:14 <cjwatson> - Discussed some image consolidation possibilities.
15:14 <cjwatson> - Various things about containerising apps.
15:14 <cjwatson> - Helped to generate click store keys.
15:14 <cjwatson> - Taught Ben about Launchpad livefs image building so that he can move cloud images into that system.
15:14 <cjwatson> Organised launchpad-buildd 125 deployment.  You now have more verbose build logs. \o/
15:14 <cjwatson> Spent about a day fixing parted bug 1352252.
15:14 <ubottu> bug 1352252 in parted (Ubuntu) "Exception during partitioning whilst utopic server installations" [Critical,Fix released] https://launchpad.net/bugs/1352252
15:14 <cjwatson> Fixed systemd dependency problem that broke a bunch of build-dep installation.
15:14 <cjwatson> Released click 0.4.30.
15:14 <cjwatson> ..
15:14 <cjwatson> Even more RTM discussion.
15:14 <cjwatson> Next week: on holiday Wed-(next)Mon.
15:15 <xnox> cjwatson: \o/ cloud images on real livefs builders would be awesome
15:15 <cjwatson> Yeah, it looks quite tractable and Ben's keen
15:15 <xnox> and verbose build logs \o/
15:15 <robru> * wrote graph definitions for NFSS data
15:15 <robru> - Long-Running-Test for Chris Gagnon
15:15 <robru> - memevent tests for Chris Lee
15:15 <robru> - fixed app-startup-benchmark for Max Brustkern
15:15 <robru> - which means NFSS will shortly have support for 4 data sets once this lands.
15:15 <robru> - landed massive refactoring that included Jasmine unit tests for NFSS code.
15:15 <robru> * landings, landings, landings
15:15 <robru> * various bugfixes and tweaks for citrain dashboard:
15:16 <robru> - stop displaying stale spreadsheet data when backend indicates silo is empty
15:16 <robru> - stop linkifying parentheses around URLs
15:16 <robru> - sort the SVG status bubbles by size
15:16 <robru> - when packages are in proposed, linkify them directly to the excuses permalink
15:16 <robru> (done)
15:17 <stgraber> Various RTM related discussions wrt system-image.
15:17 <stgraber> Worked on getting a basic ubuntu-core system-image.
15:17 <stgraber> Various LXC patches review and some fixes for the CI environment.
15:17 <stgraber> Preparation and work on Ubuntu 12.04.5 LTS.
15:17 <stgraber> (DONE)
15:17 <barry> phone: s-i getprop test suite fix.  LP: #1349478.  system-image 2.3.2.  discussing additional improvements for various other mobile toys.  LP: #1324241
15:17 <ubottu> Launchpad bug 1349478 in Ubuntu system image "/usr/sbin/system-image-dbus:sqlite3.OperationalError:_check_for_update:emit_signal:UpdateAvailableStatus:__init__:__enter__:_cursor" [High,Fix released] https://launchpad.net/bugs/1349478
15:17 <ubottu> Launchpad bug 1324241 in Ubuntu system image "Collect code coverage" [High,In progress] https://launchpad.net/bugs/1324241
15:17 <barry> other: looked a bit at LP: #1351018.  more dialer-app py3 ap merges.  reviewed xnox's lazr.authentication branch.  patch pilot.  pytest-instafail 0.2.0-2.  Python issue 21539 (Path.mkdirs() exist_ok argument).
15:17 <ubottu> Launchpad bug 1352591 in apport (Ubuntu) "duplicate for #1351018 apport-retrace does not update libraries in a sandbox" [Undecided,New] https://launchpad.net/bugs/1352591
15:17 <barry> done
15:18 <robru> yaaaay that's everybody
15:18 <sil2100> \o/
15:19 <xnox> oh. Who do i talk to to (a) make lazr.authentication project owned by lazr developers (b) make upstream release ?
15:19 <xnox> ~leonardr registered it, but ~benji is listed as maintainer.
15:19 <cjwatson> xnox: I'd suggest making it William's problem :)
15:19 <xnox> cjwatson: =)))))))))))
15:20 <cjwatson> We can probably track down people to reassign things to an appropriate current team
15:20 <xnox> cjwatson: excellent.
15:20 <cjwatson> inc. pypi
15:20 <barry> xnox: i think i have perms on the pypi project
15:20 <xnox> i feel like i want to finish the port up, so i'll ping wgrant/barry about things i don't have access to. =) sounds good.
15:21 <barry> xnox: yep.  i can admin a ton of lazr.* stuff
15:21 <robru> #topic Any Other Business
15:22 <xnox> Thanks everyone, it was a blast =) my last meeting for a while ;-)
15:22 <slangasek> does xnox get to tell us what he's working on?
15:22 <robru> xnox, you'll be back!
15:22 <xnox> robru: we'll see.
15:22 <sil2100> xnox: :<
15:23 <xnox> slangasek: which bits you'd want me to talk about? systemd? whoopsie ids? python port? rumors from the office?
15:23 <slangasek> xnox: the last!
15:23 <slangasek> or, I don't know
15:23 <slangasek> the whoopsie ids? :)
15:23 <xnox> slangasek: well, i emailed you brian and ev about whoopsie ids already =)
15:24 <xnox> slangasek: so in the office there are approximately 20 or so small boxes with bq written on them. I don't have a big enough bag to carry them all, what should I do?
15:24 * xnox ponders about the post-room in the building.
15:25 <slangasek> yes, but you haven't shared with the whole team about whoopsie ids
15:25 <xnox> ok.
15:25 <xnox> #topic Whoopsie IDs
15:26 <xnox> So whoopsie when it generates crashes and submits them to error tracker uses a SHA512 hash of a "unique id" for a given machine.
15:26 <xnox> Typically, we want to gather sets of errors comming from the same machine, but we don't really care how it's identified (hence the SHA512)
15:27 <xnox> and we'd want to be able to track crashes from same machine across reinstalls.
15:27 <xnox> for that we need to create unique ids. There are a few strategies implemented at the moment.
15:27 <xnox> on i386/amd64 product_uuid is used.
15:27 <xnox> however id generation is now a library which is used by other projects as well.
15:28 <xnox> for example post-office service on ubuntu-touch uses those ids.
15:28 <xnox> (post-office is something like push notifications to the devices I believe)
15:28 <xnox> so product_uuid is ./sys/devices/virtual/dmi/id/product_uuid
15:29 <xnox> do check yours to see what it's like =) on some of my hardware it's a generic 1234567890, but generally it's unique enough.
15:29 <xnox> For virtual machines, one can pass --uuid flag to specify a product_uuid.
15:29 <slangasek> mine says "44444444444444"
15:30 <xnox> On armhf, product_uuid does not exist hence other bits of uniqueness are used - MAC address and IMEI (unique symcard modem id on GSM phones)
15:30 <robru> mine looks mostly unique but it has a couple 'FFFF' clauses in it
15:30 <xnox> slangasek: hm, we should probably black list it.
15:30 <cjwatson> slangasek: chosen by fair dice roll
15:30 <slangasek> :-)
15:30 <xnox> =)))))))))))
15:30 <stgraber> root@sateda:~# cat /sys/devices/virtual/dmi/id/product_uuid
15:30 <stgraber> cat: /sys/devices/virtual/dmi/id/product_uuid: No such file or directory
15:30 <xnox> .....
15:31 <xnox> in cases where id cannot be determined whoopsie falls back through each one. However, one can also override the ID by supplying environment variable
15:31 <cjwatson> Basically sensible on the systems I can readily get at
15:31 <xnox> CRASH_DB_IDENTIFIER
15:32 <xnox> so all of above is good, however it's quite unstable ids through the lifetime of a single boot and/or other operating modes.
15:32 <stgraber> (that system ithout product_uuid is nothing fancy, it's a 1U supermicro server I use here as a router, it's got all the other product_* files, just not _uuid)
15:32 <slangasek> oh, see, it's a whitebox server, that's why
15:32 <slangasek> it's not unique
15:32 <xnox> For example: product_uuid is root owned file, thus regular userss for libwhoopsie cannot get the same id as whoopsie itself.
15:33 <xnox> also, if network interfaces are added or removed, and/or ofono (GSM/modem provider) is started or stopped ----- the unique id changes again.
15:33 <xnox> and lastly if CRASH_DB_IDENTIFIER was used it's also unexposed to libwhoopsie users.
15:33 <xnox> I've come up with a three step plan to address above issues in the context of rtm and ubuntu touch platform.
15:34 <xnox> Firstly for the ubuntu-emulator, i've implemented generating uuid per instance at create time and passing --uuid option to qemu such that product_uuid exists on ubuntu-touch emulator i386
15:35 <xnox> Secondly for ubuntu-touch armhf, I drop whoopsie.override to export and set CRASH_DB_IDENTIFIER cause armhf doesn't have product_uuid
15:35 <xnox> next, I propose changes to whoopsie to export a world readable /run/whoopsie-id with the ID it chose to use, with a matching change to libwhoopsie0 to read that "cache" file at top priority
15:36 <xnox> this way all root and non-root users are aligned on the same machine id through the lifetime of boot, irrespective of ofono starting/stopping and networking interfaces changing on the fly.
15:36 <slangasek> xnox: "export and set" - where does it get the value?
15:37 <xnox> and lastly, in addition to checking product_uuid, I propose to also check for /sys/class/android_usb/android0/iSerial which is Android specific serial number present on most recent devices.
15:37 <slangasek> xnox: oh - why in /run/whoopsie-id, instead of /var/lib so it doesn't have to write it each boot?
15:38 <xnox> slangasek: so $ ubuntu-emulator create -> uses libuuid to create ~/.local/share/ubuntu-emulator/$NAME/.uuid & .whoopsie-id (sha512 hash of .uuid)
15:38 <xnox> for i386 --uuid is passed from .uuid by ubuntu-emulator run.
15:38 <slangasek> yep - but you said armhf doesn't have uuid, so how do you inject it there?
15:39 <xnox> for armhf "env CRASH_DB_IDENTIFIER=sha512-from-whoopsie-id" is written into /etc/init/whoopsie.override as passing it as a kernel cmdline arg looked very ugly.
15:39 <slangasek> ah, so it's actually written by ubuntu-emulator into the filesystem
15:39 <slangasek> gotcha
15:39 <xnox> (so at create base image instance construction time)
15:39 <slangasek> nice
15:40 <slangasek> sounds pretty slick to me
15:40 <xnox> slangasek: re: /var/lib vs /run/ -> I am undecided. With all of the above fixes I believe ids are stable across reinstalls and hence stable on each boot, hence it can live in /run. I am open to writting ot /var/lib as well.
15:40 <xnox> which is similar to dbus/machine-id.
15:41 <slangasek> xnox: /var/lib guarantees its presence before whoopsie has started, or if whoopsie fails to start on a reboot for some reason
15:41 <xnox> slangasek: true. And e.g. installers could write that file out as well.
15:41 <slangasek> could still be a race if you needed it on first boot, but, well, you don't have much opportunity to crash things on first boot
15:41 <xnox> ..
15:42 <slangasek> xnox: thanks for pulling this together
15:42 <xnox> for an epilogue, I'd like to mention other machine IDs used on our systems.
15:42 <slangasek> and I think it's useful to have this explained before you go, as I imagine we'll be using it for some time to come
15:42 <slangasek> other machine IDs> uhoh :)
15:42 <bdmurray> xnox: so will you switch the merge proposal to use /var/lib?
15:42 <xnox> dbus uses /dev/urandom to essentially generate /var/lib/dbus/machine-id. All I know dbus system daemon does not start without it. But not much else about it.
15:42 <xnox> bdmurray: yeap.
15:43 <xnox> and there is a newish systemdish proposal for /etc/machine-id, which by default is generated by systemd first-boot system unit, and is /var/lib/dbus/machine-id compatible. In the future I can see it used more universally to identify "install id" (as it's persistant for a lifetime of a single installation)
15:44 <xnox> and systemd also offers "boot-id" to identify an individual boot. (helpful to sort out logs/events as to wether they belong to current or subsequent boot, when e.g. time jitters as well)
15:45 <xnox> /etc/machine-id is a bit broken at the moment as our livefs builders spit it out on disk image =) something to fix in the future.
15:45 <xnox> And that's it.
15:45 <xnox> More Q&A?!
15:45 <bdmurray> xnox: why don't you use‽
15:46 <xnox> bdmurray: а зачем? =)
15:47 <xnox> oh, there is whoopsie preferences dbus interface through which one can query their id.
15:47 <xnox> it's useful to browse your own crashes on errors.u.c when authenticated.
15:48 <slangasek> would be good to have that on the commandline
15:48 <xnox> dbus-send --system --print-reply --type=method_call --dest=com.ubuntu.WhoopsiePreferences /com/ubuntu/WhoopsiePreferences com.ubuntu.WhoopsiePreferences.GetIdentifier
15:48 <slangasek> fwiw I'm hearing reports from ogra_ + ev that the errors.u.c interface may not actually work
15:48 <xnox> slangasek: commandline just for you ^
15:48 <bdmurray> won't that be the identifier in /var/lib?
15:48 <slangasek> I don't want a commandline just for me
15:48 <slangasek> I want a commandline for the rest of us
15:48 <slangasek> ;)
15:48 <slangasek> bdmurray: yes
15:48 <xnox> bdmurray: once above is merged, it indeed will be =)
15:48 <xnox> all of my branches are not merged yet. I'll be sheparding for them to be merged.
15:49 * slangasek nods
15:49 <ogra_> the ID seems to be changed regulary on the device or some such
15:49 <slangasek> which I believe means that you'll be nagging me after you've left the company and while I'm on vacation, if I'm not mistaken
15:49 <ogra_> (someone pointed me to a bug, got to dig it up)
15:49 <slangasek> ogra_: that part is known and is the bit we're already fixing
15:49 <ogra_> ah, k
15:49 <bdmurray> slangasek: hmm? it might not work because the system identifier has changed on the device
15:49 <ogra_> well then it should just work, no ?
15:49 <slangasek> ogra_: I thought you were asserting that crashes from a *current* run were not findable on errors.ubuntu.com.  If it's just the other thing, then yeah, it's well in hand
15:49 <xnox> ogra_: well, i have 3 merge proposals to address whoopsie id stability as explained above =)
15:50 <slangasek> awesomesauce
15:50 <slangasek> ok, so, I need to run here
15:50 <ogra_> slangasek, well, i didnt kill any apps, so i only referred to the crashes that i saw with .uploaded suffix
15:50 <ogra_> for these i wasnt able to find anything
15:50 <slangasek> ogra_: yes, so it's an open question whether they happened during the current boot or not
15:50 <ogra_> right
15:50 <slangasek> anyway - thanks for the meeting all
15:51 <bdmurray> ogra_: did you check /var/log/upstart/whoopsie.log for the OOPS ID?
15:51 <slangasek> keep up the great work
15:51 <bdmurray> ogra_: then look up the corresponding oops?
15:51 <slangasek> robru: don't forget to #endmeeting on your way out :)
15:51 <robru> is it over? ;-)
15:51 <xnox> dunno, do we have AOB people?
15:52 <xnox> (.... before we jump)
15:52 <ogra_> bdmurray, nope, i only checked the return value from the debus call
15:52 <ogra_> *dbus
15:52 <xnox> http://goo.gl/ncjXiY
15:53 <xnox> ogra_: check that ofono is running; check id from the dbus call; stop ofono, check id again.
15:53 <bdmurray> ogra_: so the unique identifier for the crash, the OOPS ID, gets storted in the whoopsie log file. from that oops page there will be a link to the problem bucekt
15:53 <xnox> ogra_: i believe at the moment they will be different, but with my proposed patches it will stay the same.
15:56 <ogra_> will chefck both (once i have the time ... meetings ... )
15:57 <robru> #endmeeting