16:03 <blackboxsw> #startmeeting Cloud-inin bi-weekly status meeting
16:03 <meetingology> Meeting started Mon Jan  8 16:03:53 2018 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
16:03 <meetingology> 
16:03 <meetingology> Available commands: action commands idea info link nick
16:04 <blackboxsw> Happy  2018 cloud-initers!   Thanks ajorg for helping kick us off.
16:04 <blackboxsw> Welcome back from break hope the holidays were good for folks.
16:04 <blackboxsw> It
16:05 <blackboxsw> It's been a while since we've held the meeting due to holidays and vacation time. So, not a ton of content to report for the last bit. Digging up those details now
16:06 <blackboxsw> Testing of 17.2 on EC2, Azure, and GCE and release to Ubuntu Bionic
16:06 <blackboxsw> Complete 17.1.46 SRU to Ubuntu Xenial, Zesty, and Artful
16:06 <blackboxsw> Fix documentation around 'init' mode for modules subcommand (LP: #1736600)
16:06 <blackboxsw> Tooling to merge community authored branches into master
16:06 <ubot5> Launchpad bug 1736600 in cloud-init "CLI: cloud-init modules -h documents unsupported --mode init" [Low,Fix committed] https://launchpad.net/bugs/1736600
16:07 <blackboxsw> So the canonical side of the team worked a bit on getting the latest SRU updates 17.1.46 into Xenial, Zesty and artful. The testing and verification of that release took a bit of time, but we are getting better(faster)
16:07 <blackboxsw> I think this last SRU only took us 2 weeks instead of 4 weeks. so that frees up more time on upstream reviews and increasing cloud-init's velocity
16:07 <ajorg> great
16:08 <blackboxsw> we also added team tools for streamlining community authored branches. so that we stop slowing folks down :/
16:08 <blackboxsw> then the only problem is the reviewer :)
16:09 <blackboxsw> #topic Recent changes
16:09 <blackboxsw> ^ should have been that topic.
16:10 <blackboxsw> Also 17.2 release was 'cut' prior to Christmas break, this opened master up for more changes to land. so we've pulled in good  fixes for VMWare NoCloud and SLES
16:11 <blackboxsw> digging up the changests now.
16:11 <blackboxsw> Also, keep in touch with our active development and the "done" lane on trello. It's out bookkeeper for anything we are working and Done represents anything landed
16:11 <blackboxsw> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
16:13 <blackboxsw> so high-level content that landed between 17.1.46 and 17.2:
16:14 <blackboxsw> * CLI added the clean and status subcommands
16:14 <blackboxsw> * Support for identifying OVF datasource provided by VMware
16:14 <blackboxsw> * NoCloudKVM tests now run in continuous integration
16:14 <blackboxsw> * Formalize DataSource get_data and related properties
16:14 <blackboxsw> * Remove prettytable dependency and introduce simpletable
16:14 <blackboxsw> * VMWare pre and post-customization script support
16:15 <blackboxsw> Thanks ajorg I think you were the author of note on simpletable stuff, it's nice to drop dependencies where we can to increase speed of cloud-init
16:15 <ajorg> it was done selfishly
16:15 <ajorg> we dislike taking on new dependencies :-)
16:17 <blackboxsw> and thanks to robjo(suse)  maitree(vmware) too and dojordan and Ryan McCabe(redhat) for recent branches too
16:17 <blackboxsw> :)
16:18 <blackboxsw> Post our 17.2 release we've started work on improved integration..... I think we just got powersj's ec2 integration tests landed right johs?
16:18 <blackboxsw> josh even
16:18 <powersj> \o/ yep!
16:18 <ajorg> nice
16:19 <blackboxsw> sweet, so an extra security blanked for us when we have significant changesets landed in master to ensure ec2 is happy.
16:19 <blackboxsw> powersj: what are out plans for continuous integration frequency
16:19 <blackboxsw> with ec2 specifically
16:19 <ajorg> Can those integration tests be run by others with EC2 accounts?
16:19 <blackboxsw> ajorg: yes they can
16:19 <powersj> I am working on the jenkins jobs this week and hope to have a weekly run as well as a manual run for backport testing
16:19 <blackboxsw> I'll get the cmdline
16:19 <ajorg> thanks!
16:20 <blackboxsw> tox -e citests -m tests.cloud_tests run --os-name=artful --platform=ec2 --preserve-data --data-dir=../results --verbose
16:20 <blackboxsw> or something like that
16:20 <ajorg> got it
16:20 <ajorg> thanks!
16:20 <blackboxsw> powersj: documented it too I think
16:20 <blackboxsw> getting link
16:20 <powersj> https://cloudinit.readthedocs.io/en/latest/topics/tests.html#ec2
16:20 <blackboxsw> #link https://cloudinit.readthedocs.io/en/latest/topics/tests.html#ec2
16:20 <blackboxsw> :)
16:21 <blackboxsw> excellent work  Josh
16:21 <powersj> thanks for all the reviews :)
16:21 <blackboxsw> anything else I'm missing about landed work?    rharper powersj smoser1 ?
16:22 <blackboxsw> otherwise next topic
16:22 <rharper> blackboxsw: nothing from me
16:22 <blackboxsw> #topic In-progress Development
16:23 <blackboxsw> So we've got a fairly healthy review queue that we need to get through as we get the year started....
16:24 <blackboxsw> we also have a few things we are in flight currently:
16:24 <blackboxsw> - continuous integration improvements per powersj
16:24 <blackboxsw> - dropping dependence on ifup ifdown utils where possible as that's not supported (or installed in some cases) in systemd world
16:24 <smoser1> blackboxsw: wow. sorry, missing.
16:25 <blackboxsw> who is that smoser1 guy anyway
16:25 <smoser1> yeah, i didnt see anything missing sorry.
16:25 <smoser> wonder how that happened.
16:25 <blackboxsw> welcome ;)
16:25 <blackboxsw> - netplan improvements per rharper and jinja template support for all cloud-config modules
16:26 <blackboxsw> - and softlayer support per smoser
16:27 <blackboxsw> know the Azure guys are also posting a couple branches on getting a pre-provisioning setup going for thier datasource which looks pretty exciting
16:27 <blackboxsw> I can't think of anything else off the top of my head.
16:28 <robjo> chrony support
16:28 <ajorg> we're only talking feature work in this topic?
16:29 <blackboxsw> any in progress development to highlight is fair game. bug work. refactoring, feature etc
16:29 <blackboxsw> +10 robjo and again thanks for working with us getting all those branches up and (hopefully soon) landed
16:29 <ajorg> what does "jinja template support for all cloud-config modules" mean?
16:31 <ajorg> I'd guess most modules don't need templating?
16:31 <blackboxsw> ajorg: two things. 1. since we have now landed /run/cloud-instance/instance-data.json    to store metadata/userdata it'd be that #cloud-config can  new be specified with ## template:jinja header and could leverage anything jinjia has to offer plus sourcing any of the instance-data.json metadata fields
16:33 <ajorg> Ah, right. Is that not being done above the module level?
16:33 <blackboxsw> so if people have repetitive or template-driven content in the runcmd   or write_files portion or their #cloud-config  they'd be able to leverage jinja templates etc
16:33 <smoser> ajorg: yes, above the module level.
16:33 <blackboxsw> ajorg: not anywhere in cloud-config currently
16:33 <blackboxsw> one sec I misunderstood the question
16:33 <blackboxsw> smoser: can you clarify what you mean?
16:33 <ajorg> I mean, shouldn't #cloud-config template expansion happen before the module sees the config?
16:34 <smoser> blackboxsw: we could/should also allow other part types to be rendered
16:34 <smoser> ttps://trello.com/c/xyqxyOxg
16:35 <smoser> er... bad url. in 2 ways
16:35 <ajorg> The the part handler would be the one to do that expansion.
16:35 <smoser> https://trello.com/c/AYaCdQyT
16:35 <blackboxsw> ahh ok, right that makes sense. I think the cut I made was limited in focus to cloud-config modules and custom scripts supporting the ## template:jinja header.. but nothing would preclude handling other parts
16:36 <blackboxsw> so the link to my WIP branch was
16:36 <blackboxsw> #link https://trello.com/c/xyqxyOxg
16:36 <blackboxsw> and the general feature per smoser
16:36 <blackboxsw> #link https://trello.com/c/AYaCdQyT/21-cloud-init-query-standardized-json-information
16:36 <ajorg> Is there a design doc of some kind of this?
16:37 <blackboxsw> not yet.. but we probably should have a spec as it'd be a good template for the docs we'll need to write
16:38 <blackboxsw> scott captured most of the use cases we'd be going for in that last trello link above
16:38 <ajorg> Small example of where some clarity is needed: if Jinja is interpreting {foo} in a user-script, what will it do when it sees a shell variable ${foo}
16:38 <ajorg> ?
16:39 <smoser> you declare that the content is a jinja template
16:39 <smoser> if you provide it something that is not renderable as a jinja template
16:39 <smoser> then it will fail
16:39 <smoser> it requires input to explicitly say "this is jinja". it does not just attempt to render anything.
16:39 <smoser> (unless explicitly told to)
16:39 <blackboxsw> some brief working examples are in the description of the branch @ https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/334030
16:40 <ajorg> Sure. But as a content author, I need to know if Jinja is going to try to render ${foo} or not.
16:40 <smoser> then as a content author you can read jinja docs :)
16:40 <blackboxsw> jinja would try to render {{ foo }}
16:40 <ajorg> :-
16:41 <smoser> ajorg: we'll document a simple case, and we can even document "for shell, you'll have to be aware that ...."
16:41 <smoser> but we're not going to document all of jinja
16:41 <ajorg> I see.
16:42 <ajorg> My understanding was that Jinja was highly customizable in what it interpreted and how, so that it's important to document how you've configured it to work.
16:42 <blackboxsw> and since to burden is on the #cloud-config or script writer to provide the header        ## template: jinja\n#cloud-config\n         they *should* understand what they are doing
16:42 <blackboxsw> we won't implicitly run the #cloud-config through jinja
16:43 <ajorg> I get that, no problem, what I'm saying is that Jinja is an engine that you configure to do something, not a markup that always does the same thing for everyone.
16:43 <ajorg> Am I making any sense?
16:44 <blackboxsw> understood (though I thought it was fairly constrained it it's application and functionality). We'll make sure that the mechanism by which jinja operates is well documented and confined as best we can... for our own sanity we don't want that template engine to be too flexible... too many tough support use cases
16:45 <blackboxsw> ok anything else for "In progress development"  otherwise we can move to Office hours for 30 mins
16:46 <blackboxsw> #topic Office Hours (next 30 minutes)
16:47 <blackboxsw> robjo: you've got quite a few branches of goodness up for us to review. Any prioritization on those branches or just take them as we can?
16:47 <rharper> I don't think  there are issues w.r.t jinja and shell; they use different variable escape methods, jinja uses {{ variable/expression }}; and it doesn't consume $  AFAIK, ajorg do you know differently ?
16:48 <blackboxsw> #link https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334992
16:48 <blackboxsw> I'm guessing is top of the list
16:48 <ajorg> I saw {instance_id} at https://trello.com/c/AYaCdQyT/21-cloud-init-query-standardized-json-information so I assumed it was being customized to look for { instead of {{
16:48 <robjo> blackboxsw: The chrony support should probably be the last as it will take longer over all and more back and forth
16:48 <ajorg> (for one thing)
16:49 <ajorg> rharper: also, there's the whole question of the "extends" feature
16:50 <ajorg> We integrated Jinja into an internal tool a few years back and we spent a very long time making sure the loaders did the right thing.
16:50 <blackboxsw> ajorg: I thought I read somewhere that you couldn't exend jinja for custom functions. maybe I was mistaken
16:50 <robjo> I am also not certain that the "re-write everything" on the first go around for chrony is really what we want to do initially
16:50 <ajorg> blackboxsw: I don't think I'm referring to custom functions
16:51 <robjo> That's probably where we ant to end up, but I am not certain that a "step function" approach is in order
16:51 <rharper> ajorg: hrm, I've always seen {{ variable }} or {% expression %};  so maybe blackboxsw can just update the templates;
16:51 <rharper> the examples in the cards
16:52 <ajorg> rharper: sure, that would have helped in this case.
16:52 <robjo> If we do go down the route of the step function I'll need more gudance then in rharper's comments
16:52 <ajorg> blackboxsw: I was referring to the ability of one template to extend another.
16:53 <ajorg> blackboxsw: and the question of where does the engine look when it's asked to extend another template. It can be tricky.
16:54 <blackboxsw> yeah I honestly hadn't gotten past step one of handling the template markup within an existing single template. so this may need a bit of thought/work
16:55 <ajorg> Personally, I'd be a lot happier with limiting things to Python format() templates, even though it means you can't have loops, but I won't get in the way as long as we're cognizant of the problems we can run into by accepting the full power of an advanced engine like Jinja.
16:56 <smoser> i'm not opposed to allowing ## template: python-format
16:56 <ajorg> heh
16:56 <smoser> honestly.
16:56 <smoser> you can pick  a differnt name if you dont like that one.
16:57 <smoser> but we already use jinja, so it makes sense to support jinja
16:57 * smoser has to run. sorry.
16:57 <rharper> I do feel that supplying the template means the user is opting in;  and specifically if we've got a good way to provide dry-run based on a instance.json and a script; that certainly can help folks work out the kinks in the template of their choosing
16:57 <ajorg> I'm really not opposed so much as wary of the extensive power of the thing
16:58 <rharper> ajorg: that's a fair warning; given you've experience here; help drawing the line is most welcome
16:58 <ajorg> I'm trying to think of a way to read in /etc/shadow using Jinja, you know?
16:58 <rharper> well, cloud-init is root anyhow; so, what's the deal with that ?
16:59 <blackboxsw> ajorg: heh, right though you can read that with your runcmd section in #cloud-config :)
16:59 <ajorg> If I can come up with a way to do it that doesn't make it look obvious that I'm doing it, and then post that as something others can copy, or use with #include <url> then I win.
16:59 <rharper> I don't think jinja makes that any more troublesome
17:00 <rharper> folks already wget | bash with shell they don't understand either
17:00 <ajorg> I suspect Jijna makes it more opaque.
17:01 <ajorg> The answer to "what file does Jinja read when I use {% extends foo %}" is a very lengthy "it depends"
17:02 <ajorg> anyway, I've said my piece
17:03 * ajorg is a bit of a template naysayer.
17:05 <blackboxsw> +1, there's one in every group. We'll try to keep that in mind as this feature evolves
17:05 <blackboxsw> :)
17:06 <ajorg> nice
17:06 <ajorg> :-)
17:06 <blackboxsw> any pet bugs, new features or burning reviews that need mention?
17:07 <blackboxsw> ajorg: we could do something simple like disable the extends option via policies
17:07 <blackboxsw> it looks like
17:08 <blackboxsw> #link http://jinja.pocoo.org/docs/2.10/api/#policies
17:08 <blackboxsw> or maybe I'm misunderstanding the issue   I'll read up more on it
17:08 <ajorg> thanks
17:09 <ajorg> It looked like https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/334074 was blocking https://code.launchpad.net/~yeazelm/cloud-init/+git/cloud-init/+merge/331897 but shouldn't be anymore.
17:09 <ajorg> I'll remind Matt to try it again now.
17:13 <blackboxsw> thanks good dela
17:13 <blackboxsw> dela
17:13 <blackboxsw> deal
17:13 <blackboxsw> geez
17:14 <blackboxsw> on that note. I think it's time for coffee
17:14 <blackboxsw> and time to end the meeting
17:14 <blackboxsw> Happy New Year again folks. Good to be back in the office.
17:15 <blackboxsw> thanks again for the chat, until next time..
17:15 <blackboxsw> #endmeeting