16:05 <blackboxsw> #startmeeting Cloud-init 'bi-weekly' status meeting
16:05 <meetingology> Meeting started Mon Dec 11 16:05:16 2017 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
16:05 <meetingology> 
16:05 <meetingology> Available commands: action commands idea info link nick
16:05 <smoser> thanks for hosting blackboxsw
16:05 <blackboxsw> no problemo.
16:05 <blackboxsw> happy holidays folks and thanks for joining.
16:07 <blackboxsw> #topic In-progress Development
16:07 <blackboxsw> As mentioned @ our 17.1 release, we're promising more frequent cloud-init releases.
16:08 <blackboxsw> smoser has mailed the list informing cloud-init interested parties that we are targeting a 17.2 release for Thursday this week
16:08 <blackboxsw> It's been a few weeks since we've hosted the meeting (I think we missed last meeting), so I'll post some of the development that has landed in trunk
16:08 <smoser> #link https://lists.launchpad.net/cloud-init/msg00114.html
16:09 <blackboxsw> * All integration tests now function with the nocloud-kvm backend
16:09 <blackboxsw> * Fix apport for cloud-name options (LP: #1722564)
16:09 <blackboxsw> * Improve warning message when templates aren't found (Robert Schweikert) (LP: #1730135)
16:09 <blackboxsw> * Perform null checks for enabled/disabled Red Hat repos (Dave Mulford)
16:09 <blackboxsw> * Fix openSUSE and SLES setup of /etc/hosts (Robert Schweikert) (LP: #1731022)
16:09 <blackboxsw> * Catch UrlError when #include'ing URLs (Andrew Jorgensen)
16:09 <ubot5> Launchpad bug 1722564 in Apport "apport question will not accept multi-character responses" [Undecided,Confirmed] https://launchpad.net/bugs/1722564
16:09 <ubot5> Launchpad bug 1730135 in openstack-dev-sandbox ""Too much rain in Sydney"" [Undecided,New] https://launchpad.net/bugs/1730135
16:09 <ubot5> Launchpad bug 1731022 in cloud-init "host template expansion does not work on SUSE distros" [High,Fix committed] https://launchpad.net/bugs/1731022
16:09 <smoser> ajorg replied with a request for https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/329657
16:09 <smoser> that fell on deaf ears
16:09 <blackboxsw> * Released stable release update (SRU) of 17.1-27-geb292c18 (LP: #1721847)
16:09 <blackboxsw> * Cleanup dhclient background process after EC2 network discovery.
16:09 <blackboxsw> * ntp: fix configuration template rendering for openSUSE and SLES (Robert Schweikert) LP: #1726572
16:09 <blackboxsw> * fix manually running cloud-init after upgrade (LP: #1732917)
16:09 <ubot5> Launchpad bug 1721847 in cloud-init (Ubuntu Artful) "sru cloud-init 2017-10-06 (17.1-18-gd4f70470-0ubuntu1) updated to (17.1-27-geb292c18)" [Medium,Fix released] https://launchpad.net/bugs/1721847
16:09 <ubot5> Launchpad bug 1726572 in cloud-init "ntp config handling inconsistent for SLES and openSUSE" [Medium,Fix committed] https://launchpad.net/bugs/1726572
16:09 <ubot5> Launchpad bug 1732917 in cloud-init "17.1 update breaks EC2 nodes" [High,Fix committed] https://launchpad.net/bugs/1732917
16:09 <ajorg> truth
16:09 <smoser> ajorg: i will review shortly
16:09 <blackboxsw> * Queued upstream for merge into Bionic
16:09 <blackboxsw> * Queued 17.1.46 SRU for Xenial, Zesty, and Artful
16:09 <blackboxsw> * Fix EC2 race on sandboxed dhclient's pidfile during tempdir teardown (LP: #1735331)
16:09 <blackboxsw> * Enable Bionic in Integration Tests
16:09 <blackboxsw> * Create LXD and KVM Integration Tests in Jenkins
16:09 <ubot5> Launchpad bug 1735331 in cloud-init "ec2: zesty tempfile sandbox dhclient.pid file can't be created" [High,Fix committed] https://launchpad.net/bugs/1735331
16:10 <blackboxsw> As of end of last week, we are trying to blitz the review queue and dust off anything that has been sitting too long
16:12 <blackboxsw> So a couple fixes went into Amazon's initial network setup, IPv6 support is live for Ubuntu series Xenial, Zesty, Artful and Bionic
16:13 <ajorg> cool
16:14 <blackboxsw> heh I blew that last topic. it should have been #topic Recent Changes.
16:14 <blackboxsw> anyway I'll fix it in the logs when I publish
16:15 <blackboxsw> As always , for historical docs from this meeting check this link
16:15 <blackboxsw> #link http://cloud-init.github.io
16:15 <blackboxsw> #topic In-progress Development (for real)
16:15 <blackboxsw> So we have an active queue that is pretty healthy still
16:15 <blackboxsw> #link http://bit.ly/ci-reviews
16:16 <blackboxsw> smoser: rharper are we still trying to get through that queue as best we can for 17.2 or when do we think the window closes there?
16:16 <smoser> i think we can spend some more time on queue today.
16:16 <smoser> but that is about it really.
16:16 <blackboxsw> yeah, want some settle 'bake' time before the 17.2 cut on Thursday
16:17 <blackboxsw> We saw a couple Azure branches come in late last week.... Are there any branches folks are really excited about landing this week (today tomorrow?)
16:18 <blackboxsw> I had hoped to get through a couple of Robert's as they don't seem very contentious.
16:19 <smoser> the reporter bit seems pretty reasonable
16:19 <smoser> other than its not actually used anywhere in the mp
16:19 <smoser> ie, its non-contentious to add a reporter, but adding code that is not used is of not a lot of use :)
16:19 <blackboxsw> true
16:20 <ajorg> which mp is being discussed?
16:20 <smoser> (https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/334989)
16:20 <blackboxsw> #link https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/334989
16:21 <ajorg> thanks
16:23 <blackboxsw> With the upcoming holidays I expect things will be pretty slow after mid-next week, so we won't likely be landing a lot before the first of the new year.
16:25 <robjo> If it's slow for you more time to review open merge proposals ;)
16:25 <blackboxsw> This week we are also trying to get an SRU into ubuntu xenial, zesty and artful for some VMware/OVF datasource fixes for ds-identify and for pre-cusomization marker files courtesty (smoser & maitriyee)
16:26 <blackboxsw> *courtesy* rather
16:26 <smoser> ajorg: you could ping matthew on https://code.launchpad.net/~yeazelm/cloud-init/+git/cloud-init/+merge/331897
16:27 <ajorg> yup
16:27 <blackboxsw> and I know powersj is working on EC2 integration test support for cloud-init
16:27 <powersj> yep!
16:27 <powersj> Hoping to have an initial MP up this week
16:27 <blackboxsw> it's gonna be excellent to automatically test these releases
16:28 <blackboxsw> powersj: rharper smoser anything else in progress?
16:28 <rharper> nothing here
16:28 <ajorg> oh very nice.
16:28 <dpb1> powersj: \o/
16:28 <smoser> just the things that are in teh review queue. i put up one this morning for tmp file leakage
16:28 <smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/335034
16:29 <smoser> i think the yeazelm mp probably is just missing somethign simple bug haven't spent any time on it.
16:29 <blackboxsw> ahh also we landed the initial /run/cloud-init/instance-data.json which we had talked about with larsks. It captures all metadata and userdata and some standardized properties which could help people script instance data.
16:29 <smoser> yeah, that is neat.
16:30 <blackboxsw> Yeah, we have yet to write up docs on using it (and we have an inprogress branch to allow using jinja templates in #cloud-config modules). But I don't expect this to land by 17.2
16:31 <ajorg> did we? that's great!
16:32 <blackboxsw> yeah only basic standardized properties currently.  +            'local-hostname': self.get_hostname(),
16:32 <blackboxsw> 592	+            'instance-id': self.get_instance_id(),
16:32 <blackboxsw> 593	+            'cloud-name': self.cloud_name,
16:32 <blackboxsw> 594	+            'region': self.region,
16:32 <blackboxsw> 595	+            'availability-zone': self.availability_zone}}
16:32 <robjo> is that basically a re-implementation of python-ec2metadata? https://github.com/SUSE/Enceladus/tree/master/ec2utils/ec2metadata
16:32 <blackboxsw> but it's a first pass. We expect to add more
16:33 <blackboxsw> robjo: kindof, though generalized for all datasources
16:34 <rharper> robjo: long term, it's expected to be more than just ec2; rather a common baseline of instance metadata independent the actual cloud, but , IIRC, having a cloud-specific area (or at least access to the raw data)
16:34 <blackboxsw> it leaves a json-foramatted file containing any vendor data and user-data plus generalized/standardized fields extracted from that content which can be expected on all clouds
16:35 <robjo> problem with "all" clouds is that Azure is very different
16:35 <blackboxsw> since each datasource has that data already, it's essentially just formating it in a consumable file that others could levereage
16:35 <robjo> although one can argue that a "name" is an id it still looks weird when 'instance-id' and is a name
16:36 <blackboxsw> agreed robjo, some datasources may not provide different/less  content.
16:36 <smoser> ajorg: http://paste.ubuntu.com/26164503/
16:36 <ajorg> Is that one of the Azure differences? name vs. instance-id?
16:36 <smoser> thats a demo of instance-data.json
16:36 <robjo> yes, azure has names no numbers
16:37 <robjo> it's more about "user expectations" as a "name" is an "identifier"
16:37 <blackboxsw> I'll check that azure run now. I think I linked it to the branch originally
16:37 <blackboxsw> ok I think it's probably a good to transition to open office hours now for the next 30 mins
16:38 <smoser> what is 'name' versus 'instance-id' comment ?
16:38 <smoser> ajorg: ?
16:38 <blackboxsw> #topic Office Hours (for next 30 mins)
16:38 <blackboxsw> feel free to continue the discussion now
16:38 <robjo> I'd just caution of making the assumption that we can stick the information from that data sources straight into another format and then call it "generic instance information"
16:38 <ajorg> smoser: re robjo's comment about "Azure is very different"
16:39 <smoser> oh. yes. ok.
16:39 <blackboxsw> azure instance-data
16:39 <blackboxsw> #link http://pastebin.ubuntu.com/26075842/
16:39 <dpb1> smoser: standup
16:39 <smoser> yeah, they do have a 'id'
16:40 <blackboxsw> #link http://paste.ubuntu.com/26164503/
16:40 <ajorg> from DMI?
16:40 <robjo> Which is useless in any any command
16:40 <smoser> from the cd i think.
16:41 <ajorg> robjo: ah, so the ID is unique (is it?) but can't be used to call any Azure APIs?
16:41 <robjo> in EC2 the instance-id is useful to me if I want to run "aws" commands, but the instance ID shown in the pastebin is useless for any "az" command
16:41 <robjo> ajorg: correct
16:41 <robjo> in the "az" tools everything is a name
16:42 <robjo> and thus to make the data cloud-init produces useful the -id should be the name of the VM
16:42 <robjo> then I can parse that information and use it if I need to deal with the API
16:43 <smoser> hm.
16:43 <robjo> but providing that ID as its is basically just sticking information into the json to "fill a field" which is somewhat counter to the point I'd say
16:43 <ajorg> robjo: there's a uniqueness constraint on the name too? but per-account or at-a-time or what?
16:43 <smoser> i odnt knwo. although it is insteresting thought.
16:43 <smoser> the issue is 'instance-id' is supposed to be an instance id
16:43 <smoser> not a user provided name that can be provided mutliple times in a row.
16:44 <smoser> i realize name is per-group unique, but if i
16:44 <smoser> a.) launch
16:44 <smoser> a.) launch 'foobar'
16:44 <robjo> There is a uniqueness constraint in that one cannot run a VM with the same "name" in the same resource group
16:44 <ajorg> so the question is if that ID provides global uniqueness, or if it provides a reference to the instance to be used via APIs
16:44 <smoser> b.) create capture
16:44 <smoser> c.) delete foobar
16:44 <smoser> d.) launch foobar
16:44 <smoser> then 'd' wont look new
16:45 <robjo> yes, it will it just takes a long time in Azure until the backend reaches "eventual" consistency and knows "foorbar" has been deleted previously
16:45 <ajorg> It seems clear enough that cloud-init is looking for a unique ID
16:45 <ajorg> But a user might want either, and probably an ID for API use.
16:47 <robjo> Well if we provide a format of the data that is exposed to the user via documentation and expected to be used by the user than at that point, IMHO, user needs have higher priority than what cloud-init is looking for
16:47 <ajorg> To decide which APIs to use, a script has to first look at which cloud it's on, so it has a chance to decide which value to use.
16:47 <robjo> that cloud-init uses the id to make decisions about "pre-once", "per-always" is a different topic
16:49 <robjo> Well that then kind of defeats the "generic instance information" claim, IMHO
16:49 <robjo> you are basically saying 1.) look for the framework and then decide if on that framework the "generic instance information" is useful or not
16:50 <robjo> 2.) If you happen to be on a platform where the "generic instance information" is not useful, go and collect your own
16:50 <robjo> From a user perspective that is not very nice, IMHO
16:51 <ajorg> oh, I was presuming we'd also include the Azure name, not that we'd include only a useless instance-id in that case.
16:51 <ajorg> clouds that don't have a name, wouldn't include a value for it.
16:52 <robjo> the pastebin only has the ID
16:52 <ajorg> right, I'm saying we should add the name
16:53 <robjo> This is why I am pointing out that "generic instance information" is not necessarily so easy to come by
16:54 <blackboxsw> robjo: ultimately, I'd like the generalized content surfaced in instance-data.json to be something that external user's could get value from and script against. This first pass was a stripped down approach to some of that content.
16:54 <smoser> we could add 'name' and have it be none yes.
16:55 <smoser> the not-yet-written doc will state that consumers should not be confused by new field names.
16:55 <blackboxsw> There are some fixes that need to be proposed to all datasources to better standardize on things like  public vs private addresses, external hostnames etc. Those I expect will come in subsequent passes.
16:55 <robjo> it might be worth considering the concept of "equivalent instance information" where the entries in the json files get names/keys that are generic across all cloud frameworks and provide the euivalent information/usefulness to the user
16:55 <smoser> but inside the 'v1', then content of a key will not change.
16:55 <ajorg> robjo: that's a fair point, imho
16:55 <smoser> but 'instance-id' is in fact 'instance-id'.  not 'name'.
16:55 <blackboxsw> robjo: I think that is the intent of those 'v1' standardized fields.
16:56 <smoser> note that lxd shares the same generic problem in this regard as azure. it uses user-provided name for instance-id. but does not provide an actual instance id of any sort.
16:56 <blackboxsw> right per name/instance-id discussion, they feel separate, and I think there is value in adding a separate 'name' as smoser mentioned
16:58 <robjo> Lets look at it from an API perspective, if I were to use the .json file wouldn't it be nice if I could just say  json.load().get{'instance_api_id')
16:58 <robjo> for EC2 that returns the instance-id, for Azure it gives me the name
16:59 <robjo> part of the idea of cloud-init is to keep the ugly details of the cloud framework away from the user
16:59 <ajorg> if we were talking about the value of "region" we'd certainly want to yield the value that's useful for API calls.
17:00 <robjo> so why would the .json data then retrieve from that idea and make the user know if I am in EC2 I need to use instance-id and if I am in Azure I need to use instance-name?
17:01 <robjo> ajorg: agreed
17:01 <smoser> it seems somewhat non-sense that azure gives an instance a unique id, but cannot take that in as an identifier to the instance.
17:02 <robjo> AWS, GCE, and Azure all have the concept of "region" , not certain how IBM is handling that part in their setup but that may not be of interest to us at this point
17:02 <smoser> your point is good though. but instance-id i really think needs to be a unique identifier (as much as possible) for this *instance*
17:02 <ajorg> It sounded like smoser's 'v1' comment was meant to imply we could have a 'v2' that yields data differently than 'v1'.
17:02 <smoser> softlayer has "datacenters"
17:03 <robjo> smoser: I agree, but that's the way it is
17:03 <smoser> at some point, ajorg we will of course realize that we're all idiots
17:03 <smoser> and wonder What were we thinking!
17:03 <smoser> and have a 'v2'
17:04 <blackboxsw> <-- it takes some of us longer than others to realize that
17:05 <ajorg> smoser: you're not convinced that today is that day?
17:05 <smoser> i try to keep acknowledgement of that fact to be more than a few days later
17:06 <ajorg> good to let it sink in first :-)
17:06 <smoser> (compared to when i notice it, to allow for additional occurences)
17:06 <ajorg> I'm not going to say it has to be changed, but I do think at the very least the azure name should be available.
17:07 <blackboxsw> I think this discussion definitely sheds light on the fact that we should continue to bring these standardized instance-data discussions to this meeting for a quick feedback loop from you guys as it evolves :)
17:07 <ajorg> :)
17:08 <blackboxsw> #action blackboxsw bring up any updates in instance-data.json fields for discussion about common use-cases/patterns
17:08 * meetingology blackboxsw bring up any updates in instance-data.json fields for discussion about common use-cases/patterns
17:08 <ajorg> and it doesn't seem harmful to have the name only if the cloud provides one, just as if the cloud doesn't have a concept of an availability zone we'll skip that too.
17:09 <blackboxsw> +1
17:10 <blackboxsw> well i think this about wraps up our meeting for today
17:10 <blackboxsw> any other topics for today?
17:11 <ajorg> I pinged Matt Yeazel, but he didn't respond yet.
17:11 <smoser> i think 'api-id' would lmake sense as a name.
17:11 <ajorg> so nothing more from my end
17:11 <ajorg> smoser: or 'api-instance-id'
17:12 <smoser> that just seems confusing.
17:12 <smoser> hm..
17:12 <smoser> i see why you want the 'instance' portion there, but the thing i dont like is that implies that this is 'per instance'
17:12 <ajorg> well, it's an API instance identifier.
17:13 <smoser> which in fact it is not.
17:13 <smoser> hm.
17:13 <ajorg> Ah, okay, that's true, but if the cloud doesn't have a unique way to identify the instance to the API...
17:13 <smoser> yeah
17:14 <ajorg> someone should check that assumption... how do you refer to terminated instances? or how are they identified in logs?
17:16 <ajorg> smoser: I just worry that someone's going to say "but in my API an API ID is this other thing"
17:17 <blackboxsw> yeah before surfacing something like that we'd need to vet it
17:17 <ajorg> In general I think there are enough differences between clouds that it's probably a losing battle to try to come up with something that's one-size-fits-all.
17:17 <ajorg> The goal was to make the information available more readily than by calling out to metadata services, right?
17:18 <ajorg> It's much harder to implement meta-data pulling for every cloud than to implement some logic that pulls the right value out of a JSON object, so it's still a big improvement even if it can't provide a unified view.
17:21 <ajorg> anywho, I should go do other things.
17:21 <blackboxsw> ajorg: yes that is the primary goal:  more easily access cloud-provided metadata
17:21 <blackboxsw> if there is low-hanging fruit we can standardize I'm +1 on the concept
17:22 <blackboxsw> that's where the standard 'v1' key came from
17:22 <blackboxsw> but yeah I also don't think cloud-init needs to boil the ocean and standardize all fields
17:22 <blackboxsw> we'll capture what low-hanging fruit we can
17:22 <blackboxsw> and it'll take time
17:22 <blackboxsw> ok. Thanks for the great discusssions/suggestions ajorg and robjo. keep 'em coming
17:22 <blackboxsw> think I'll end meeting now
17:22 <blackboxsw> until next time...
17:22 <blackboxsw> #endmeeting