== Meeting information == * #cloud-init: Cloud-init 'bi-weekly' status meeting, 11 Dec at 16:05 — 17:22 UTC * Full logs at [[http://ubottu.com/meetingology/logs/cloud-init/2017/cloud-init.2017-12-11-16.05.log.html]] == Meeting summary == === In-progress Development === The discussion about "In-progress Development" started at 16:07. * ''LINK:'' https://lists.launchpad.net/cloud-init/msg00114.html * ''LINK:'' http://cloud-init.github.io === In-progress Development (for real) === The discussion about "In-progress Development (for real)" started at 16:15. * ''LINK:'' http://bit.ly/ci-reviews * ''LINK:'' https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/334989 * ''LINK:'' https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/335034 === Office Hours (for next 30 mins) === The discussion about "Office Hours (for next 30 mins)" started at 16:38. * ''LINK:'' http://pastebin.ubuntu.com/26075842/ * ''LINK:'' http://paste.ubuntu.com/26164503/ * ''ACTION:'' blackboxsw bring up any updates in instance-data.json fields for discussion about common use-cases/patterns == Vote results == == Action items, by person == * blackboxsw * blackboxsw bring up any updates in instance-data.json fields for discussion about common use-cases/patterns == Done items == * (none) == People present (lines said) == * blackboxsw (83) * smoser (54) * ajorg (38) * robjo (32) * ubot5 (7) * meetingology (4) * powersj (2) * dpb1 (2) * rharper (2) == Full Log == 16:05 #startmeeting Cloud-init 'bi-weekly' status meeting 16:05 Meeting started Mon Dec 11 16:05:16 2017 UTC. The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology. 16:05 16:05 Available commands: action commands idea info link nick 16:05 thanks for hosting blackboxsw 16:05 no problemo. 16:05 happy holidays folks and thanks for joining. 16:07 #topic In-progress Development 16:07 As mentioned @ our 17.1 release, we're promising more frequent cloud-init releases. 16:08 smoser has mailed the list informing cloud-init interested parties that we are targeting a 17.2 release for Thursday this week 16:08 It's been a few weeks since we've hosted the meeting (I think we missed last meeting), so I'll post some of the development that has landed in trunk 16:08 #link https://lists.launchpad.net/cloud-init/msg00114.html 16:09 * All integration tests now function with the nocloud-kvm backend 16:09 * Fix apport for cloud-name options (LP: #1722564) 16:09 * Improve warning message when templates aren't found (Robert Schweikert) (LP: #1730135) 16:09 * Perform null checks for enabled/disabled Red Hat repos (Dave Mulford) 16:09 * Fix openSUSE and SLES setup of /etc/hosts (Robert Schweikert) (LP: #1731022) 16:09 * Catch UrlError when #include'ing URLs (Andrew Jorgensen) 16:09 Launchpad bug 1722564 in Apport "apport question will not accept multi-character responses" [Undecided,Confirmed] https://launchpad.net/bugs/1722564 16:09 Launchpad bug 1730135 in openstack-dev-sandbox ""Too much rain in Sydney"" [Undecided,New] https://launchpad.net/bugs/1730135 16:09 Launchpad bug 1731022 in cloud-init "host template expansion does not work on SUSE distros" [High,Fix committed] https://launchpad.net/bugs/1731022 16:09 ajorg replied with a request for https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/329657 16:09 that fell on deaf ears 16:09 * Released stable release update (SRU) of 17.1-27-geb292c18 (LP: #1721847) 16:09 * Cleanup dhclient background process after EC2 network discovery. 16:09 * ntp: fix configuration template rendering for openSUSE and SLES (Robert Schweikert) LP: #1726572 16:09 * fix manually running cloud-init after upgrade (LP: #1732917) 16:09 Launchpad bug 1721847 in cloud-init (Ubuntu Artful) "sru cloud-init 2017-10-06 (17.1-18-gd4f70470-0ubuntu1) updated to (17.1-27-geb292c18)" [Medium,Fix released] https://launchpad.net/bugs/1721847 16:09 Launchpad bug 1726572 in cloud-init "ntp config handling inconsistent for SLES and openSUSE" [Medium,Fix committed] https://launchpad.net/bugs/1726572 16:09 Launchpad bug 1732917 in cloud-init "17.1 update breaks EC2 nodes" [High,Fix committed] https://launchpad.net/bugs/1732917 16:09 truth 16:09 ajorg: i will review shortly 16:09 * Queued upstream for merge into Bionic 16:09 * Queued 17.1.46 SRU for Xenial, Zesty, and Artful 16:09 * Fix EC2 race on sandboxed dhclient's pidfile during tempdir teardown (LP: #1735331) 16:09 * Enable Bionic in Integration Tests 16:09 * Create LXD and KVM Integration Tests in Jenkins 16:09 Launchpad bug 1735331 in cloud-init "ec2: zesty tempfile sandbox dhclient.pid file can't be created" [High,Fix committed] https://launchpad.net/bugs/1735331 16:10 As of end of last week, we are trying to blitz the review queue and dust off anything that has been sitting too long 16:12 So a couple fixes went into Amazon's initial network setup, IPv6 support is live for Ubuntu series Xenial, Zesty, Artful and Bionic 16:13 cool 16:14 heh I blew that last topic. it should have been #topic Recent Changes. 16:14 anyway I'll fix it in the logs when I publish 16:15 As always , for historical docs from this meeting check this link 16:15 #link http://cloud-init.github.io 16:15 #topic In-progress Development (for real) 16:15 So we have an active queue that is pretty healthy still 16:15 #link http://bit.ly/ci-reviews 16:16 smoser: rharper are we still trying to get through that queue as best we can for 17.2 or when do we think the window closes there? 16:16 i think we can spend some more time on queue today. 16:16 but that is about it really. 16:16 yeah, want some settle 'bake' time before the 17.2 cut on Thursday 16:17 We saw a couple Azure branches come in late last week.... Are there any branches folks are really excited about landing this week (today tomorrow?) 16:18 I had hoped to get through a couple of Robert's as they don't seem very contentious. 16:19 the reporter bit seems pretty reasonable 16:19 other than its not actually used anywhere in the mp 16:19 ie, its non-contentious to add a reporter, but adding code that is not used is of not a lot of use :) 16:19 true 16:20 which mp is being discussed? 16:20 (https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/334989) 16:20 #link https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/334989 16:21 thanks 16:23 With the upcoming holidays I expect things will be pretty slow after mid-next week, so we won't likely be landing a lot before the first of the new year. 16:25 If it's slow for you more time to review open merge proposals ;) 16:25 This week we are also trying to get an SRU into ubuntu xenial, zesty and artful for some VMware/OVF datasource fixes for ds-identify and for pre-cusomization marker files courtesty (smoser & maitriyee) 16:26 *courtesy* rather 16:26 ajorg: you could ping matthew on https://code.launchpad.net/~yeazelm/cloud-init/+git/cloud-init/+merge/331897 16:27 yup 16:27 and I know powersj is working on EC2 integration test support for cloud-init 16:27 yep! 16:27 Hoping to have an initial MP up this week 16:27 it's gonna be excellent to automatically test these releases 16:28 powersj: rharper smoser anything else in progress? 16:28 nothing here 16:28 oh very nice. 16:28 powersj: \o/ 16:28 just the things that are in teh review queue. i put up one this morning for tmp file leakage 16:28 https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/335034 16:29 i think the yeazelm mp probably is just missing somethign simple bug haven't spent any time on it. 16:29 ahh also we landed the initial /run/cloud-init/instance-data.json which we had talked about with larsks. It captures all metadata and userdata and some standardized properties which could help people script instance data. 16:29 yeah, that is neat. 16:30 Yeah, we have yet to write up docs on using it (and we have an inprogress branch to allow using jinja templates in #cloud-config modules). But I don't expect this to land by 17.2 16:31 did we? that's great! 16:32 yeah only basic standardized properties currently. + 'local-hostname': self.get_hostname(), 16:32 592 + 'instance-id': self.get_instance_id(), 16:32 593 + 'cloud-name': self.cloud_name, 16:32 594 + 'region': self.region, 16:32 595 + 'availability-zone': self.availability_zone}} 16:32 is that basically a re-implementation of python-ec2metadata? https://github.com/SUSE/Enceladus/tree/master/ec2utils/ec2metadata 16:32 but it's a first pass. We expect to add more 16:33 robjo: kindof, though generalized for all datasources 16:34 robjo: long term, it's expected to be more than just ec2; rather a common baseline of instance metadata independent the actual cloud, but , IIRC, having a cloud-specific area (or at least access to the raw data) 16:34 it leaves a json-foramatted file containing any vendor data and user-data plus generalized/standardized fields extracted from that content which can be expected on all clouds 16:35 problem with "all" clouds is that Azure is very different 16:35 since each datasource has that data already, it's essentially just formating it in a consumable file that others could levereage 16:35 although one can argue that a "name" is an id it still looks weird when 'instance-id' and is a name 16:36 agreed robjo, some datasources may not provide different/less content. 16:36 ajorg: http://paste.ubuntu.com/26164503/ 16:36 Is that one of the Azure differences? name vs. instance-id? 16:36 thats a demo of instance-data.json 16:36 yes, azure has names no numbers 16:37 it's more about "user expectations" as a "name" is an "identifier" 16:37 I'll check that azure run now. I think I linked it to the branch originally 16:37 ok I think it's probably a good to transition to open office hours now for the next 30 mins 16:38 what is 'name' versus 'instance-id' comment ? 16:38 ajorg: ? 16:38 #topic Office Hours (for next 30 mins) 16:38 feel free to continue the discussion now 16:38 I'd just caution of making the assumption that we can stick the information from that data sources straight into another format and then call it "generic instance information" 16:38 smoser: re robjo's comment about "Azure is very different" 16:39 oh. yes. ok. 16:39 azure instance-data 16:39 #link http://pastebin.ubuntu.com/26075842/ 16:39 smoser: standup 16:39 yeah, they do have a 'id' 16:40 #link http://paste.ubuntu.com/26164503/ 16:40 from DMI? 16:40 Which is useless in any any command 16:40 from the cd i think. 16:41 robjo: ah, so the ID is unique (is it?) but can't be used to call any Azure APIs? 16:41 in EC2 the instance-id is useful to me if I want to run "aws" commands, but the instance ID shown in the pastebin is useless for any "az" command 16:41 ajorg: correct 16:41 in the "az" tools everything is a name 16:42 and thus to make the data cloud-init produces useful the -id should be the name of the VM 16:42 then I can parse that information and use it if I need to deal with the API 16:43 hm. 16:43 but providing that ID as its is basically just sticking information into the json to "fill a field" which is somewhat counter to the point I'd say 16:43 robjo: there's a uniqueness constraint on the name too? but per-account or at-a-time or what? 16:43 i odnt knwo. although it is insteresting thought. 16:43 the issue is 'instance-id' is supposed to be an instance id 16:43 not a user provided name that can be provided mutliple times in a row. 16:44 i realize name is per-group unique, but if i 16:44 a.) launch 16:44 a.) launch 'foobar' 16:44 There is a uniqueness constraint in that one cannot run a VM with the same "name" in the same resource group 16:44 so the question is if that ID provides global uniqueness, or if it provides a reference to the instance to be used via APIs 16:44 b.) create capture 16:44 c.) delete foobar 16:44 d.) launch foobar 16:44 then 'd' wont look new 16:45 yes, it will it just takes a long time in Azure until the backend reaches "eventual" consistency and knows "foorbar" has been deleted previously 16:45 It seems clear enough that cloud-init is looking for a unique ID 16:45 But a user might want either, and probably an ID for API use. 16:47 Well if we provide a format of the data that is exposed to the user via documentation and expected to be used by the user than at that point, IMHO, user needs have higher priority than what cloud-init is looking for 16:47 To decide which APIs to use, a script has to first look at which cloud it's on, so it has a chance to decide which value to use. 16:47 that cloud-init uses the id to make decisions about "pre-once", "per-always" is a different topic 16:49 Well that then kind of defeats the "generic instance information" claim, IMHO 16:49 you are basically saying 1.) look for the framework and then decide if on that framework the "generic instance information" is useful or not 16:50 2.) If you happen to be on a platform where the "generic instance information" is not useful, go and collect your own 16:50 From a user perspective that is not very nice, IMHO 16:51 oh, I was presuming we'd also include the Azure name, not that we'd include only a useless instance-id in that case. 16:51 clouds that don't have a name, wouldn't include a value for it. 16:52 the pastebin only has the ID 16:52 right, I'm saying we should add the name 16:53 This is why I am pointing out that "generic instance information" is not necessarily so easy to come by 16:54 robjo: ultimately, I'd like the generalized content surfaced in instance-data.json to be something that external user's could get value from and script against. This first pass was a stripped down approach to some of that content. 16:54 we could add 'name' and have it be none yes. 16:55 the not-yet-written doc will state that consumers should not be confused by new field names. 16:55 There are some fixes that need to be proposed to all datasources to better standardize on things like public vs private addresses, external hostnames etc. Those I expect will come in subsequent passes. 16:55 it might be worth considering the concept of "equivalent instance information" where the entries in the json files get names/keys that are generic across all cloud frameworks and provide the euivalent information/usefulness to the user 16:55 but inside the 'v1', then content of a key will not change. 16:55 robjo: that's a fair point, imho 16:55 but 'instance-id' is in fact 'instance-id'. not 'name'. 16:55 robjo: I think that is the intent of those 'v1' standardized fields. 16:56 note that lxd shares the same generic problem in this regard as azure. it uses user-provided name for instance-id. but does not provide an actual instance id of any sort. 16:56 right per name/instance-id discussion, they feel separate, and I think there is value in adding a separate 'name' as smoser mentioned 16:58 Lets look at it from an API perspective, if I were to use the .json file wouldn't it be nice if I could just say json.load().get{'instance_api_id') 16:58 for EC2 that returns the instance-id, for Azure it gives me the name 16:59 part of the idea of cloud-init is to keep the ugly details of the cloud framework away from the user 16:59 if we were talking about the value of "region" we'd certainly want to yield the value that's useful for API calls. 17:00 so why would the .json data then retrieve from that idea and make the user know if I am in EC2 I need to use instance-id and if I am in Azure I need to use instance-name? 17:01 ajorg: agreed 17:01 it seems somewhat non-sense that azure gives an instance a unique id, but cannot take that in as an identifier to the instance. 17:02 AWS, GCE, and Azure all have the concept of "region" , not certain how IBM is handling that part in their setup but that may not be of interest to us at this point 17:02 your point is good though. but instance-id i really think needs to be a unique identifier (as much as possible) for this *instance* 17:02 It sounded like smoser's 'v1' comment was meant to imply we could have a 'v2' that yields data differently than 'v1'. 17:02 softlayer has "datacenters" 17:03 smoser: I agree, but that's the way it is 17:03 at some point, ajorg we will of course realize that we're all idiots 17:03 and wonder What were we thinking! 17:03 and have a 'v2' 17:04 <-- it takes some of us longer than others to realize that 17:05 smoser: you're not convinced that today is that day? 17:05 i try to keep acknowledgement of that fact to be more than a few days later 17:06 good to let it sink in first :-) 17:06 (compared to when i notice it, to allow for additional occurences) 17:06 I'm not going to say it has to be changed, but I do think at the very least the azure name should be available. 17:07 I think this discussion definitely sheds light on the fact that we should continue to bring these standardized instance-data discussions to this meeting for a quick feedback loop from you guys as it evolves :) 17:07 :) 17:08 #action blackboxsw bring up any updates in instance-data.json fields for discussion about common use-cases/patterns 17:08 * meetingology blackboxsw bring up any updates in instance-data.json fields for discussion about common use-cases/patterns 17:08 and it doesn't seem harmful to have the name only if the cloud provides one, just as if the cloud doesn't have a concept of an availability zone we'll skip that too. 17:09 +1 17:10 well i think this about wraps up our meeting for today 17:10 any other topics for today? 17:11 I pinged Matt Yeazel, but he didn't respond yet. 17:11 i think 'api-id' would lmake sense as a name. 17:11 so nothing more from my end 17:11 smoser: or 'api-instance-id' 17:12 that just seems confusing. 17:12 hm.. 17:12 i see why you want the 'instance' portion there, but the thing i dont like is that implies that this is 'per instance' 17:12 well, it's an API instance identifier. 17:13 which in fact it is not. 17:13 hm. 17:13 Ah, okay, that's true, but if the cloud doesn't have a unique way to identify the instance to the API... 17:13 yeah 17:14 someone should check that assumption... how do you refer to terminated instances? or how are they identified in logs? 17:16 smoser: I just worry that someone's going to say "but in my API an API ID is this other thing" 17:17 yeah before surfacing something like that we'd need to vet it 17:17 In general I think there are enough differences between clouds that it's probably a losing battle to try to come up with something that's one-size-fits-all. 17:17 The goal was to make the information available more readily than by calling out to metadata services, right? 17:18 It's much harder to implement meta-data pulling for every cloud than to implement some logic that pulls the right value out of a JSON object, so it's still a big improvement even if it can't provide a unified view. 17:21 anywho, I should go do other things. 17:21 ajorg: yes that is the primary goal: more easily access cloud-provided metadata 17:21 if there is low-hanging fruit we can standardize I'm +1 on the concept 17:22 that's where the standard 'v1' key came from 17:22 but yeah I also don't think cloud-init needs to boil the ocean and standardize all fields 17:22 we'll capture what low-hanging fruit we can 17:22 and it'll take time 17:22 ok. Thanks for the great discusssions/suggestions ajorg and robjo. keep 'em coming 17:22 think I'll end meeting now 17:22 until next time... 17:22 #endmeeting Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/meetingology)