16:02 <blackboxsw> #startmeeting Cloud-init bi-weekly status meeting
16:02 <meetingology> Meeting started Mon Mar 19 16:02:30 2018 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
16:02 <meetingology> 
16:02 <meetingology> Available commands: action commands idea info link nick
16:03 <blackboxsw> ok, let's kick off this cloud-init bi-weekly meeting. welcome all!
16:04 <dpb1> o/
16:04 <blackboxsw> it's been a busy couple weeks for a few of us w/ planning meetings and vacation, but let's see what progress we've made on cloud-init.
16:05 <blackboxsw> #topic Recent Changes
16:05 <dpb1> smoser vacation specifically
16:05 <blackboxsw> hehe. Generally we're tracking high-points of what  lands in our trello board
16:05 <blackboxsw> #link http://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
16:06 <blackboxsw> but from changelogs folks have made progress on azure, vmware and FreeBSD deployment targets
16:06 <blackboxsw> - netplan: render bridge port-priority values (LP: #1735821)
16:06 <blackboxsw> - net: recognize iscsi root cases without ip= on kernel command line.
16:06 <blackboxsw> (LP: #1752391)
16:06 <blackboxsw> - util: Fix subp regression. Allow specifying subp command as a string.
16:06 <blackboxsw> (LP: #1755965)
16:06 <blackboxsw> - This commit fixes get_hostname on the AzureDataSource.
16:06 <blackboxsw> [Douglas Jordan] (LP: #1754495)
16:06 <ubot5`> Launchpad bug 1735821 in nplan (Ubuntu Artful) "netplan needs bridge port-priority support" [Medium,Fix committed] https://launchpad.net/bugs/1735821
16:06 <blackboxsw> - shellify: raise TypeError on bad input.
16:06 <blackboxsw> - FreeBSD: Set hostname to FQDN. [Dominic Schlegel] (LP: #1753499)
16:06 <blackboxsw> - Make salt minion module work on FreeBSD.
16:06 <blackboxsw> [Dominic Schlegel] (LP: #1721503)
16:06 <ubot5`> Launchpad bug 1752391 in cloud-init "cloud-init does not recognize initramfs provided network config in all cases" [Medium,Fix committed] https://launchpad.net/bugs/1752391
16:06 <blackboxsw> - set_hostname: When present in metadata, set it before network bringup.
16:06 <blackboxsw> (LP: #1746455) VMWare
16:06 <ubot5`> Launchpad bug 1755965 in cloud-init "util.subp regression: no longer accept commands as string" [High,Fix committed] https://launchpad.net/bugs/1755965
16:06 <blackboxsw> - cc_snap: Add new module to install and configure snapd and snap
16:06 <blackboxsw> packages.
16:06 <blackboxsw> - doc: fix all warnings issued by 'tox -e doc'
16:06 <blackboxsw> - tests: Make pylint happy and fix python2.6 uses of assertRaisesRegex.
16:06 <ubot5`> Launchpad bug 1755965 in cloud-init "duplicate for #1754495 util.subp regression: no longer accept commands as string" [High,Fix committed] https://launchpad.net/bugs/1755965
16:06 <blackboxsw> - tests: fix run_tree and bddeb
16:06 <blackboxsw> - tests: Fix some warnings in tests that popped up with newer python.
16:06 <ubot5`> Launchpad bug 1753499 in cloud-init "hostname in FreeBSD should prefere FQDN" [Undecided,Fix committed] https://launchpad.net/bugs/1753499
16:06 <blackboxsw> - tests: fix flakes warning for unused variable
16:06 <blackboxsw> - tests: patch leaked stderr messages from snap unit tests
16:06 <ubot5`> Launchpad bug 1721503 in cloud-init "salt module not able to be used on FreeBSD" [Medium,Fix committed] https://launchpad.net/bugs/1721503
16:06 <blackboxsw> - tests: Centralize and re-use skipTest based on json schema presense.
16:06 <ubot5`> Launchpad bug 1746455 in cloud-init "cloud-init vSphere cloud provider DHCP unique hostname issue" [High,Fix committed] https://launchpad.net/bugs/1746455
16:08 <blackboxsw> a big thanks to dojordan (Azure) and Dominic Schlegel (FreeBSD) for patching some gaps in support as cloud-init master progresses
16:10 <blackboxsw> On the ubuntu side of the house we got tip of tree published into Bionic thusday & friday, we are awaiting cloud-image builds which look like they are stale at 03-15-2018. once those builds are published, all clouds should be getting latest cloud-init on Bionic
16:11 <blackboxsw> I think that's probably it for 'done' work. We have a few things in flight at the moment
16:12 <blackboxsw> #topic In-progress Development
16:13 <blackboxsw> Ubuntu is getting a number of new cloud-config modules:
16:13 <blackboxsw> - new cc_snap module (deprecated cc_snappy and cc_snap_config modules) the ability to install and manage snap package
16:14 <blackboxsw> - new cc_ubuntu_drivers: support to install 3rd party drivers on install
16:15 <blackboxsw> - new cc_ubuntu_advantage: manage Ubuntu Advantage subscriptions for services such as Extended Security Mainenance (trusty), canonical livepatch and FIPS PPAs
16:15 <blackboxsw> these should be landing in the week(s) to come
16:15 <blackboxsw> and cc_snap landed already
16:17 <blackboxsw> also there are a couple of branches that we are trying to wrap up for first class chrony support (per rharper, inspired by robjo's work)
16:17 <rharper> blackboxsw: smoser: on the lander emails, the subject could include the git hash (or branch name); it's currently only in the body;
16:17 <smoser> rharper: i had suggested to blackboxsw that it should acutally change to *not* send a subject. so it threads in your email reader with the other MP mails.
16:18 <rharper> heh
16:18 <blackboxsw> maybe we can toggle between the two modulus 2 :)
16:18 <rharper> sorry, didn't meant to disturb the flow
16:19 <rharper> continue
16:19 <blackboxsw> yeah, we've also touched a little bit of our code landing automation this last week. powersj also is working on a git lander plugin that we might be able to use to automate landing of approved branches w/ tox test runs
16:19 <blackboxsw> anything to free up developer time will give us more time for reviews/code
16:19 <ajorg> should vendor-specific modules be shipped in a separate package?
16:20 <smoser> vendor specific modules ?
16:20 <ajorg> ubuntu_advantage
16:22 <blackboxsw> good question/point. I hand't thought about that separation as a lot of the modules cloud-init delivers support a subset of distros
16:23 <blackboxsw> each module has a distro attribute defined as to whether or not it will even run
16:23 <blackboxsw> so we have spacewalk, zypper_repos etc
16:24 <smoser> and cloud.cfg is rendered based on knowledge of the distro
16:24 <smoser> so ubuntu_advantage wont even be in the list of config modules
16:24 <smoser> having a static config module list is WIN in this case (but pain elsewhere)
16:25 <smoser> at some point whe may have a more dynamic config module list.
16:25 <smoser> but anyway... at the moment the only cost to non-ubuntu of that module being shipped is bytes on disk.
16:26 <blackboxsw> if/when we do define that more dynamic config module list, I'd like us also to look at having configurable/separate plugin directories defined for folks providing vendor-specific content.
16:26 <ajorg> agree that having a more dynamic config module list is prerequisite to being able to parcel out modules to other packages
16:26 <blackboxsw> so that we don't expect folks to add plugins directly into /usr/lib/python3/dist-packages/cloudinit/config/ for instance
16:26 <smoser> yeah. at the point when it is dynamic, the module would still declare its support for a list of distros and would be filtered out.
16:29 * ajorg is satisfied
16:30 <blackboxsw> :).    the only other thing I can think of in progress two more datasources  softlayer cloud support by smoser and hetzner cloud
16:30 <blackboxsw> so cloud-init is getting it's grubby hands into a couple of more clouds shortly.
16:31 <blackboxsw> s/it's/its/
16:31 <blackboxsw> it's nice to see the adoption continue to grow
16:31 <smoser> looks like someone followed up on hetzner
16:31 <smoser> so that hopefully is ready to land
16:31 <blackboxsw> #link https://code.launchpad.net/~lp-markusschade/cloud-init/+git/cloud-init
16:32 <blackboxsw> rharper: also has a couple of branches to allow cloud-init to work a bit better when rendering netplan configuration
16:33 <blackboxsw> I *think* that's all for in-progress development at the moment.
16:33 <smoser> man we need to fix that pylint thing.
16:33 <smoser> did you mention ?
16:33 <rharper> blackboxsw: yeah, I just pushed a fix for v1 global dns entries to get rendered under interfaces without any dns configuration
16:33 <blackboxsw> Anything else that should be noted by anyone?
16:33 <blackboxsw> #link https://code.launchpad.net/~lp-markusschade/cloud-init/+git/cloud-init/+merge/338439
16:33 <blackboxsw> oops
16:34 <smoser> cloudinit/config/cc_puppet.py:143: [W1505(deprecated-method), handle] Using deprecated method readfp()
16:34 <blackboxsw> #link https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/341662
16:34 <smoser> that needs fixing. it has come to us due to a new version of some of our tox environemnts.  we do not fully pin the versions , only the top level packages.  Ie, pylint's dependencies changed, but we only pin pylint version.
16:35 <blackboxsw> yeah how much should we freeze our deps?
16:35 <blackboxsw> it's kindof annoying to have your branch locally pass ci, and a fresh build of CI deps fail when you try to land
16:36 <blackboxsw> but I don't really know whether it's worth us 'pinning' everything
16:36 <smoser> i think pinning everything is generally the best practice for this sort of thing now.
16:37 <smoser> oh my.
16:37 <blackboxsw> so if we pin the world, should we also just make it a habit to occasionally tox -e tip-pylint?
16:37 <smoser> blackboxsw: did you know you accidently fixed that in trunk ?
16:38 <blackboxsw> smoser: I know I intentionally added a pylint ignore on that to come back and address it today.
16:38 <ajorg> i tend to believe it's better to stay current and take your punches a few at a time so you don't have a major upset when you have to upgrade.
16:38 <smoser> oh ko. i see.
16:39 <smoser> ajorg: well, sor tof. if you have c-i that you want to be green, and consider it bad when it is not, then you dont want dude-on-the-internet to break you
16:40 <smoser> there is the "good" break, where new upload to pypi identifies some lingering bug
16:40 <blackboxsw> +1 ajorg, but I'm good (on avoiding a avalanche) if we agree to run tip-pylint target fairy regularly to avoid the landslide
16:40 <smoser> but also the "bad" break where some upload breaks your c-i for invalid reason.
16:42 <smoser> one huge advantage to pinning is the ability to re-create things.
16:42 <blackboxsw> it definitely felt like last week was a lot of c-i breaks  for changes unrelated to the code up for review
16:42 <smoser> ie, if you were looking to it bisect something...
16:42 <smoser> git bisect
16:43 <blackboxsw> I should transition to the office hours topic so we can continue discussion
16:43 <smoser> you can't really do that if trunk from a point in the past does not pass C-I because an external dependency changed.
16:43 <smoser> sure we can transition to office  hours.
16:43 <blackboxsw> #topic Office hours (next ~30 mins)
16:43 <smoser> but yeah... i want c-i on tip to not just start failing.
16:44 <stanguturi> @blackboxsw, I have couple of requests. First, https://bugs.launchpad.net/ubuntu/+source/open-vm-tools/+bug/1724128 discussed this in last meeting as well. Any help is greatly appreciated
16:44 <ubot5`> Ubuntu bug 1724128 in open-vm-tools (Ubuntu) "Need a Success / Failure notification mechanism when cloud-init finishes." [Undecided,New]
16:44 <blackboxsw> that's fair. smoser can we maybe add a jenkins job to run tip-pylint then weekly. So, we don't have a huge backlog of lint failures against tip?
16:45 <smoser> blackboxsw: i'mi fine with that... thats why we added the tip-* targets. so it was easy enough to keep current.
16:45 <blackboxsw> +1 smoser I'll put a card up for that
16:46 <smoser> stanguturi: your suggestion there is not a bad idea at all.
16:46 <blackboxsw> #link https://trello.com/c/vD1em9WP/698-jenkins-job-to-run-tox-tip-pylint-weekly-pin-all-lint-versions-otherwise
16:46 <smoser> the desire to have cloud-init tell the platform/datasource that it failed or succeeded is valid.
16:47 <smoser> with MAAS, that his done via reporting
16:47 <blackboxsw> stanguturi: hiya. I think we talked after that meeting about trying to allow the datasource to subscribe to a callback when cloud-init exists
16:47 <blackboxsw> stanguturi: hiya. I think we talked after that meeting about trying to allow the datasource to subscribe to a callback when cloud-init exits
16:47 <smoser> cloud-init reports status and results via a Reporter.
16:47 <smoser> dojordan (i think) had also put up a request for a reproter module on azure.
16:47 <smoser> so we *do* kind of have the function you're after  in place.
16:48 <stanguturi> smoser: ok. Any inputs / examples of using it will be really great.
16:50 <smoser> stanguturi: well, the reporter interface is pretty simple. you can cloudinit/reporting/handlers.py
16:51 <smoser> blackboxsw: http://paste.ubuntu.com/p/6KjDX8WHQH/
16:51 <smoser> did you intend boht of those changes ?
16:51 <stanguturi> smoser: ok. Then do I need to write a new handler for our DataSource?
16:52 <smoser> stanguturi: well you write a reporting Handler, ankd then either system confi or optionally datasource config would turn that reporter on.
16:52 <blackboxsw> wow smoser, no
16:52 <blackboxsw> wow, ok, I'll put up a branch to fix that
16:52 <stanguturi> smoser: ok. Will work on that.
16:52 <smoser> ok.
16:53 <stanguturi> I have another quick request about https://code.launchpad.net/~sankaraditya/cloud-init/+git/cloud-init/+ref/set_hwclock_module
16:53 <smoser> stanguturi: do you find this is actually needed ?
16:53 <smoser> to my knowledge the only time anyone would ever set their hardware clock to something other than UTC would be dual booting with windows.
16:54 <smoser> which i can't seem to believe is all that a common situation in VMs
16:54 <stanguturi> smoser: Yeah. We need this for 'VMware guest customization workflow'.
16:54 <dpb1> smoser: oh man, I hope not
16:54 <stanguturi> smoser: if you think, this is not worth for the base cloud-init modules, I can modify to do this change in our datasource specific modules.
16:54 <smoser> stanguturi: does it actually solve a *current* problem for you ?
16:55 <stanguturi> smoser: For 'VMware managed VMs', customers can specify in the specification file if they want UTC or localtime for the hardware clock.
16:55 <smoser> or one that originally came in from a decade ago
16:55 <stanguturi> smoser: Our existing customization (non cloud-init) engine does it. If we want to move to cloud-init, we want to port all our changes from our engine to our datasource in cloud-init.
16:55 <smoser> hm... so my sugestion is really to stop allowing that :)
16:56 <stanguturi> smoser: Oh ok. Can you please add a comment to that merge request just for the record.
16:56 <smoser> i very well could be wrong
16:56 <smoser> but the only time that i ever had to deal with this was when dual booting
16:56 <smoser> with windows specifically
16:57 <stanguturi> smoser: ok.
16:57 <smoser> am i wrong there ?
16:57 <smoser> i really *could* be.
16:58 <stanguturi> smoser: I can discuss this within our team. But to be on par with our existing engine, want to port the changes.
16:58 <smoser> and even if you get it wrong, generally speaking you havhe some sort of ntpdate or ntp that will fix your system clock anyway.
16:58 <smoser> stanguturi: yeah. i understand that.
16:58 <stanguturi> I have another request. For Ubuntu 18.04, we are planning to set 'disable_vmware_customization' flag to False by default in /etc/cloud/cloud.cfg file.
16:59 <stanguturi> Want to know your opinion, shall we set it in cloud-init installation phase or request Ubuntu maintainers to set it in 18.04
17:00 <stanguturi> smoser: And when is the cloud-init 18.2 scheduled for release? 3/22?
17:01 <blackboxsw> probably a good time for us to bring that up
17:01 <smoser> yeah. whoops.
17:01 <smoser> :)
17:01 <smoser> 18.2 is scheduled for 3/22 (thursday)
17:02 <blackboxsw> We cloud-init 18.2 have it scheduled for an arbitrary 3/22 date, we'd like to slip that out to next week Tuesday 3/27
17:02 <smoser> but amoung our internall team we decided to push that to 3/27
17:02 <stanguturi> smoser: ok. Thanks for the update.
17:02 <blackboxsw> there a a few in flight branches, azure, softlayer etc that we'd like to get in and get tested before 18.2
17:02 <smoser> we will send an email today or tomrrow witih "pending release" like subject like we've done before.
17:03 <blackboxsw> dpb1: others any objections to cutting the 18.2 release on Tuesday 3/27?
17:03 <dpb1> none
17:04 * blackboxsw adds the upcoming date to the topic
17:05 <blackboxsw> ... ok, folks interested in discussing today?
17:08 <blackboxsw> #link https://cloud-init.github.io
17:09 <blackboxsw> The above link will have our captured logs for this meeting.
17:09 <blackboxsw> thanks again for tuning in
17:09 <blackboxsw> #endmeeting