16:05 <smoser> #startmeeting cloud-init
16:05 <meetingology> Meeting started Mon Oct  2 16:05:47 2017 UTC.  The chair is smoser. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
16:05 <meetingology> 
16:05 <meetingology> Available commands: action commands idea info link nick
16:06 <smoser> #topic Recent Changes / Highlights
16:06 <smoser> hm..
16:06 <smoser> 17.1 released https://lists.launchpad.net/cloud-init/msg00106.html
16:06 <smoser> thats the biggest thing and only thing i have for this topic
16:06 <blackboxsw> #link  https://lists.launchpad.net/cloud-init/msg00106.html
16:06 <smoser> horay for a release, thanks to those who contributed.
16:07 <rharper> \o/
16:07 <smoser> 17.2 is set for 2017-12-14
16:08 <smoser> https://launchpad.net/cloud-init/+milestone/17.2
16:08 <ajorg> yay!
16:08 <smoser> feel free to target bugs to that release.
16:08 <rharper> #link https://launchpad.net/cloud-init/+milestone/17.2
16:08 <rharper> #info please target bugs to the next release
16:09 <smoser> does that do anyting ?
16:09 <rharper> in the meeting summary, it does
16:09 <blackboxsw> smoser: the links showed up in meeting minutes last time
16:09 <smoser> k. i always expected the bot to tell me that in a pm
16:09 <blackboxsw> meetingology didn't echo though
16:09 <rharper> well, smoser may have to do those
16:09 <smoser> #topic In Progress Development / Highlights
16:09 <blackboxsw> same. but something is up that needs attention.
16:10 <blackboxsw> I'll properly handle it when publishing
16:10 <rharper> cool
16:11 <smoser> Merge Proposals
16:11 <smoser> #link http://bit.ly/ci-reviews
16:11 <smoser> there are some there for sure. i know that robjo has some he's interested in, and i think ajorg's instance-identity deserves a look
16:12 <smoser> as well as simpletable for 17.2
16:12 <ajorg> I'd be grateful for both, yes.
16:12 <smoser> i am looking at the 'networkd' one which is a blocker as currently Ubuntu does not work properly on azure
16:12 <blackboxsw> we should have more bandwidth this week
16:12 <smoser> and all sysstemd-networkd systems will not work properly on CloudStack.
16:12 <smoser> anything else there?
16:13 <robjo> The task limit increase has been tested and solves our problem, thus merging would be great
16:13 <smoser> robjo, yeah. athat does seem unlikely to cause issue
16:13 <smoser> Trello Board
16:13 <smoser> # https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
16:13 <rharper> #link  ?
16:13 <smoser> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
16:14 <robjo> https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+ref/taskLimit that's the branch for the task limit
16:15 <robjo> Sorry, I can never find the link for the open merge proposals :( problem exists between the keyboard and the chair
16:16 <blackboxsw> will get eyes on that today robjo as your addZypper is about wrapped
16:16 <smoser> Bugs
16:16 <smoser> #link https://bugs.launchpad.net/cloud-init
16:16 <smoser> just mostly pointers... above.
16:16 <smoser> for Ubuntu in 16.04, there is one SRU in progress that blackboxsw and i need to verify today
16:16 <smoser> and then we will be soon looking at doing an SRU of 17.1-ish
16:17 <robjo> Thanks, still in europe, my day is about done, will pick it up tomorrow
16:17 <smoser> robjo, link is in topic
16:17 <smoser> http://bit.ly/ci-reviews
16:17 <smoser> ok. so open discussion i guess. no w?
16:17 <smoser> #topic Open Discussion / Office Hours
16:17 <smoser> we'll hang around and pay attention to pings and discussion fro the next 30  minutes or so.
16:18 <smoser> and after that, feel free to ping.
16:19 <ajorg> I'm writing a unit test for a bugfix I'm preparing to submit. It works, but it's slow because readurl retries and waits a second each time
16:20 <ajorg> I'm a bit of a mock / patch newb. Anyone can help me see how to patch that argument?
16:20 <smoser> you can feel free to  mock readurl
16:20 <smoser> and set its return_value and look at calls
16:21 <smoser> then you dont have to httppretty.
16:21 <ajorg> The only thing I want to change about it is the sleep though. Is there a straightforward way to just change one arg?
16:21 <blackboxsw> tests/unittests/test_handler/test_handler_chef.py is an example and tests/unittests/test_datasource/test_maas.py
16:21 <blackboxsw> I think
16:21 <ajorg> Hmm, I could do that, yeah. Would be sufficient for this.
16:22 <ajorg> I'd still like to know if there's an easy way to just change one argument to something that gets called somewhere else.
16:22 <smoser> you could also mock the time.sleep from url_helper
16:22 <ajorg> hahaha, yes. that's what I need to do.
16:22 <rharper> ajorg: if you want to mock the sleep you can decorate the test_ method with @mock.patch(time.sleep)
16:22 <smoser> we're suggesting other ways generally because there isn't :) at least that i know of.
16:23 <ajorg> that's cool, patching time.sleep will do nicely
16:24 <ajorg> is there an integration test that looks at what lands on the console?
16:24 <ajorg> (going back to my question from before the meeting)
16:25 <smoser> no. there coudl be on the nocloud-kvm backend
16:25 <smoser> but i think there is no console access currently on lxd
16:25 <smoser> and i think that we do not collect console access on nocloud-kvm
16:25 <smoser> but as you suggest we should for sure
16:25 <ajorg> k
16:26 <ajorg> I wondered if systemd might be swallowing my output, or maybe python (boothook in this case is a python script) is making a strange choice when it sets up logging.
16:27 <rharper> certainly possible;  I know there were issues with cloud-init starting before say rsyslog on non-systemd boots
16:28 <rharper> and the python logging has changed w.r.t the default configuration;  cloud-init main sort of expects this transition as it starts up in init-local and it has not yet read the cloud-config for logging configuration yet, so it reads that and then does some replay
16:28 <ajorg> interesting
16:29 <smoser> ajorg, you're writing to stdout/err with logging from a boothook ?
16:29 <ajorg> at least I would have thought cloud-init-output.log would contain my logs though, since it's more a redirect of stderr and stdout, right?
16:29 <ajorg> yeah
16:29 <rharper> correct
16:29 <rharper> this is from a bootcmd ?
16:29 <ajorg> no, #cloud-boothook
16:29 * rharper hasn't used boothook
16:29 <smoser> hm..
16:30 <ajorg> boothooks are super useful
16:32 <rharper> smoser: when do boot_hooks run ?  local ?  net ?
16:33 <rharper> I'm not sure yet where stdout/err for boot hooks occur, but you can dump each from the units via:  journalctl -u cloud-init-local.service   (or cloud-init.service)
16:33 <blackboxsw> "This is the earliest hook available. Note, that there is no mechanism provided for running only once"
16:34 <blackboxsw> I'm checking the code now (local/pre-local maybe)?
16:34 <rharper> yeah, saw that;  it runs in-image scripts programs
16:34 <ajorg> I did that too, and don't see anything.
16:34 <ajorg> (journalctl)
16:34 <rharper> you should see some output
16:34 <rharper> but not seeing your hook in there ?
16:34 <ajorg> I can tell by other means that the hook ran
16:34 <ajorg> just don't see that it printed anything
16:35 <ajorg> my prime suspect if systemd can't do something bad here is that python makes some decision about not actually logging.
16:35 <ajorg> and I need to be more explicit that it should log
16:36 <rharper> so, the boot_hook will sub out each part, I see no capture on the stderr/stdout ; so I would expect those to go to whatever is currently capturing those;
16:37 <rharper> s/sub/subprocess
16:38 <ajorg> yup, that's what I expect too, and what I recall seeing on Amazon Linux
16:38 <rharper> since it's using cloud-init's util.subp, there *should* be a debug level message saying 'Running command %s with allowed return codes %s' which maches a path to the boothook
16:38 <rharper> I would expect to see that in the cloud-init.log
16:39 <smoser> http://paste.ubuntu.com/25661692/
16:39 <ajorg> yup, I do see that
16:39 <rharper> the output of script, I *think* should go to cloud-init-output.log
16:39 <smoser> i'm testing ^ on serverstack now.
16:39 <smoser> i verified the commands run on lxd but not look at console there.
16:39 <ajorg> thanks
16:40 <smoser> actually.. yeah, BOOTHOOK does run, but doesnt seem to have stdout tied to same place as bootcmd
16:41 <ajorg> oh?
16:41 <ajorg> I'm slightly surprised it's not just me
16:42 <smoser> http://paste.ubuntu.com/25661714/
16:42 <rharper> it really should be in the journal/console if stdout is not redirected to a file, cloud-init-local service
16:43 <smoser> that reproduces in lxc
16:44 <rharper> smoser: in your  instance on serverstack, do you see your BOOTHOOK in journalctl -b -u cloud-init-local.service ?
16:44 <smoser> ajorg, please go ahead and open a bug.
16:44 <ajorg> cool (not cool :-P)
16:44 <ajorg> will do
16:44 <smoser> # journalctl -b -u cloud-init.service | grep BOOT
16:44 <smoser> Oct 02 16:37:41 a5 cloud-init[121]: BOOTCMD: Mon, 02 Oct 2017 16:37:41 +0000: a5
16:44 <smoser> ^ that is from lxd
16:45 <rharper> and what about the welcome message from cloud-init ?
16:45 <smoser> i think not.
16:45 <rharper> bootcmd does:  util.subp(cmd, env=env, capture=False) ; where boot_hook does: util.subp([filepath], env=env)
16:46 <smoser> yeah, so boot_hook definitely swallowed. and should not be.
16:47 <rharper> yeah
16:49 <ajorg> https://bugs.launchpad.net/cloud-init/+bug/1720841
16:49 <ubot5> Ubuntu bug 1720841 in cloud-init "Output from boothook is not logged" [Undecided,New]
16:50 <ajorg> Meanwhile I need to setup more explicit logging from that boothook anyway, so I'm okay.
16:51 <blackboxsw> #link https://bugs.launchpad.net/cloud-init/+bug/1720841
16:51 <ubot5> Ubuntu bug 1720841 in cloud-init "Output from boothook is not logged" [Medium,Confirmed]
16:55 <ajorg> Here's another one, which I've confirmed with a unit test and proposed a patch: https://bugs.launchpad.net/cloud-init/+bug/1720844
16:55 <ubot5> Ubuntu bug 1720844 in cloud-init "UrlError from #include aborts stage" [Undecided,New]
16:55 <ajorg> (what is this #link thing you're doing there?)
16:55 <smoser> well, the bot is supposed to care and do somethign with it.
16:56 <smoser> but /me thinks it doesnt work right :)
16:56 <blackboxsw> ajorg: since meetingology is still active, the theory is it is keeping track of links  during this meeting
16:56 <smoser> (during the meeting)
16:56 <ajorg> ah, okay
16:57 <ajorg> I think simpletable is completely ready to merge, btw. Any objections? https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/330525
16:57 <blackboxsw> The theory is it writes the meeting notes out for us so we can publish to https://cloud-init.github.io
16:57 <ajorg> #link https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/330525 ;-)
16:57 <blackboxsw> geg
16:57 <blackboxsw> heh
16:58 <blackboxsw> ajorg: I think we were good on that changeset, and we only wanted to wait post 17.1 cut to avoid potential regression
16:59 <ajorg> great
16:59 <blackboxsw> I'm +1 on that will give it a spin today and then I think we can land it
17:01 <smoser> blackboxsw, ajorg my thoughts on the simpletable...
17:01 <ajorg> here it comes...
17:01 <ajorg> :-P
17:01 <blackboxsw> :)
17:02 <smoser> i would like to have a machine friendly output available
17:02 <smoser> human friendly is good, but machine friendly solves the actual goal of writing the stuff.
17:02 <ajorg> So I modified it to display *exactly* as prettytable did
17:02 <smoser> oh really.
17:02 <smoser> wow
17:02 <ajorg> which is admittedly not machine friendly
17:02 <ajorg> not especially
17:02 <smoser> then i guess i can't object at all.
17:02 <ajorg> but it's at least as good as prettytable
17:03 <smoser> in that its backwards compat
17:03 <smoser> right.
17:03 <ajorg> ^ this was my goal, to get you to not object
17:03 <smoser> i still think we should probaly additionally write some machine friendly json
17:03 <ajorg> agreed, that would be better
17:03 <smoser> i tihnk having something human friendly is good though
17:03 <smoser> as i know *I* look at that output
17:04 <ajorg> it meets my goal of not depending on prettytable
17:04 <smoser> and parsing json would be less nice
17:04 <smoser> yeah
17:05 <smoser> so based on your assertion that it outputs the same as pretty table, i have no objections. only future hopes.
17:05 <smoser> and i do agree dropping pretty table is nice
17:05 <ajorg> winning
17:06 <blackboxsw> smoser: yeah I was wondering how we generally expect people/machines to parse cloud-init-output.log. Right now it's kindof hard to do machine parsing of cloud-init-output.log.
17:07 <smoser> blackboxsw, yeah. cloud-init-output.log is not intended to be machine friendly.
17:07 <blackboxsw> do we know already of non-human consumers of the formatting in cloud-init.output.log?
17:07 <smoser> but the console actually has value in having machien friendly things on it.
17:08 <smoser> my feeling is if you have access to /var/log/cloud-init.log, then you could very easily have written whatever you wanted to another file that was purely machine friendly.
17:08 <blackboxsw> was wondering where we would intend to dump machine-friendly json
17:08 <blackboxsw> yeah
17:08 <smoser> but the console (/dev/ttyS0) boot log is different
17:08 <smoser> in that it can give you a couple things you'd not find easy access to
17:09 <smoser> a.) ssh public keys for the system (providing out of band communication of this data)
17:09 <smoser> b.) network configuration info: providn useful bits of data on how you might get to this system
17:10 <smoser> systemd makes it "fun" to get that data to the console in a safe way
17:12 <ajorg> ugh yes. systemd like to stomp all over your consoles.
17:19 <blackboxsw> ok so did we want to iterate on simpletable to dump json, or just look for cloud-init to write supplementary json files under /run/cloud-init
17:20 <blackboxsw> .. after we land ajorg's branch
17:21 <ajorg> I think it's a good idea to dump json fragments to console for some things, but I suspect smoser will still not want to break anyone who might be using the tables, so probably best to land my branch as-is
17:23 <smoser> yeah, i agree with ajorg. for now, we can just take a replacement that drops prettytable
17:23 <smoser> there are improvments to be done there, but this is a simple win
17:29 <smoser> we'll go ahead and end meeting here.
17:29 <smoser> i'll still be around. thanks all!
17:29 <smoser> #endmeeting