19:06 <infinity> #startmeeting Ubuntu Technical Board
19:06 <meetingology> Meeting started Tue Oct  8 19:06:52 2019 UTC.  The chair is infinity. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
19:06 <meetingology> 
19:06 <meetingology> Available commands: action commands idea info link nick
19:06 <infinity> I haven't meetbotted in a while.
19:07 <infinity> So, who's here?  vorlon, mdeslaur, infinity?  Izzat it?
19:07 <infinity> Looks like.
19:07 <mdeslaur> yep
19:07 <xnox> no kees?
19:07 <infinity> #topic Action review!
19:08 * infinity vorlon to follow up w/ DMB on request to require 6-month expiry policy for flavor developer teams
19:08 <vorlon> last thing I was told is that there hasn't been a DMB meeting since my request, due to lack of quorum
19:08 <mdeslaur> :(
19:08 <infinity> Shiny.  I think we have something on the agenda to deal with that later. :P
19:08 * infinity Wimpress To follow-up on-list with design review to address MATE Boutique security/consent concerns.
19:09 <vorlon> I haven't seen anything on this yet
19:09 <infinity> Ditto.
19:09 <vorlon> but this can be Wimpress's regular fortnightly irc highlight nag
19:09 * infinity infinity to ask maas team to prepare SRU exception policy à la CurtinUpdates (see https://wiki.ubuntu.com/MAASUpdates) (awaiting response from rbasak)
19:09 <infinity> I meant to get together with my nemesis and discuss their plans for MAAS going forward (ie: are they killing the deb).  I'll do that.
19:10 * infinity vorlon to circle around with store, snapcraft, et all, and revise the snap source revision policy to be more clear with regards to rebuildability and GPL compliance.
19:10 <vorlon> mm carry-over
19:10 * infinity vorlon to reply to seeded snap upload permissions question on list
19:10 <vorlon> and my other two also :/
19:10 <infinity> Right, I won't paste the next one, then. :P
19:10 <infinity> #topic New and shiny things
19:11 <infinity> [cyphermox] Reinstate / extend membership of cyphermox and jbicha in DMB for elections
19:11 <infinity> It seems we entirely ignored cyphermox's email about this?
19:11 * Wimpress hangs head in shame
19:12 <vorlon> seems so
19:12 <vorlon> shall I go push the button right now, while you carry on with the meeting agenda?
19:12 <infinity> I can JFDI this request, if there are no objections?
19:12 <infinity> Or you can.
19:12 <mdeslaur> infinity: +1
19:12 <infinity> +1 from me too.
19:12 <infinity> Go for it, Steve.
19:12 <mdeslaur> heh
19:13 <infinity> [xnox] Please review proposal for OEM archive integration into Ubuntu Project mailing list post
19:13 <infinity> https://lists.ubuntu.com/archives/technical-board/2019-August/002456.html
19:13 <vorlon> done with a 3-month extension
19:14 <xnox> I am looking for guidance, if we Ubuntu Project wants to provide an opt-in upgrade path from Vanilla to Canonical-OEM-collaborated archives, or not. And if yes, how to do it.
19:14 <infinity> xnox: Are you proxying a request from a Canonical engineering team on this, or was this just you trying to make things better?
19:15 <xnox> The proposal, is to ship just enough shims in the Ubuntu Archive, that will offer users to ugprade to "Ubuntu Certified"
19:15 <vorlon> I had asked xnox to refer this to the TB, based on the nature of the request to add additional archives to an Ubuntu install and our precedent that such things should have TB oversight
19:15 <infinity> Except that the point of the OEM stuff is to move people as quickly as possible to generic stuff, not the other way around.
19:15 <xnox> infinity:  proxying request from Lenovo, stgraber (i bought certified machine, why nothings works), and the OEM team.
19:15 <xnox> infinity:  separate instances of the same thing, as to why "ubuntu.com/download" never leads to "certified experience"
19:16 <vorlon> infinity: right, the issue is that you buy a certified machine but do a stock Ubuntu install and don't get the certified experience
19:16 <mdeslaur> I see why this would be beneficial to users
19:16 <infinity> Discounting the external archive issue, I'd be concerned that if "the OEM archive" and "the OEM kernel" become the new hotness that everyone "upgrades" to, we're moving users in exactly the opposite direction form what we want.
19:16 <mdeslaur> as long as the incentive to get stuff into the archive doesn't go away
19:17 <mdeslaur> who has upload rights to those ppas?
19:17 <xnox> many machines get certified with a delay, of 3 months+, such that one might have "all of their hardware bugs fixed" if we push out a meta-package with metadata to pull in linux-oem kernel and other device specific fixes which haven't yet made their way to GA kernel, etc.
19:17 <xnox> mdeslaur:  canonical OEM controls the signing keys, whilst the software is opensource and created in collaboration by Canonical OEM, Canonical Kernel, Vendor, & Vendor suppliars teams.
19:17 <vorlon> infinity: upstreaming the per-device or per-OEM specific fixes into the Ubuntu archive takes non-zero time, and while we want to drive the set of oem customizations to zero post-certification, there should never be a point in time that a machine is certified but they can't make the hardware work with Ubuntu
19:18 <infinity> I'm also somewhat curious, on a technical level, how you intend to devise modaliases narrow enough to target the machines we want and not get, say, every thinkpad in the last three years.
19:18 <xnox> there might be non-open source suff, like documentation / manuals / guides from the Vendor.
19:18 <infinity> vorlon: Yeah, I understand the problem this is attempting to solve.
19:18 <xnox> infinity:  OEM teams have specific modaliases of specific revisions of specific hardware that is paid for to be certified. and those modaliases will be advertised.
19:18 <mdeslaur> xnox: that doesn't really answer my question...who has upload rights to the ppa?
19:18 <infinity> I just want to be sure it's not creating larger problems (like disincentivising moving things out of OEM silos)
19:19 <vorlon> infinity: ack
19:19 <vorlon> I think that's something we need to pay attention to, but I don't see how anything /other/ than a set of OEM archives can solve the root problem
19:19 <xnox> infinity:  by nature, those things spike at a product release / certification time, and then continiously make their way into generic kernel & vanilla packages, thus eventually the delta decreases.
19:20 <infinity> Also, these PPAs have no community oversight.   linux-oem is something we can exert some control over, random junk hacked up in a PPA is another story.
19:20 <xnox> infinity:  unless there is OEM hotfix for firmware / restrict wifi & bluetooth regulatory RF bands / etc.
19:21 <vorlon> prior art here included the ubuntukylin commercial archive; I don't recall there being community oversight for that?
19:21 <xnox> mdeslaur:  upload rights, i believe it is Canonical OEM team, and that excludes linux kernel, as the linux-oem* kernel flavours are in teh Ubuntu Archive proper - just currently never installed automatically / nor like offered to be opted into at all.
19:21 <vorlon> [LINK] https://lists.ubuntu.com/archives/technical-board/2014-April/001867.html
19:21 <infinity> Indeed not.  And there was much handwaving over allowing that.
19:21 <vorlon> (^^ reference)
19:22 <xnox> mdeslaur:  infinity: the archives are on the public internet right. So anybody has access to them, including sources.
19:22 <xnox> mdeslaur:  infinity: and that's why i want the OEM archive key be shipped by a package in the Ubuntu Archive, such that if we don't feel comfortable, we can push out an update that removes the key & those archive repositories too from the Ubuntu Project.
19:22 <mdeslaur> will this new archive be opt-in in the software properties gui?
19:23 <vorlon> from my POV it's a requirement that it be opt-in, not opt-out
19:23 <infinity> Very much so.
19:23 <xnox> mdeslaur:  i was more thinking ubuntu-drivers pages of software properties gui, but yeah somthing like popup "new software is available to support your Lenovo"
19:23 <xnox> mdeslaur:  which one can choose to opt-into, or ignore.
19:23 <mdeslaur> xnox: all vendors will be in the same archive?
19:23 <infinity> What sorts of things live in this PPA?  Do they fork Ubuntu packages?
19:24 <xnox> mdeslaur:  at the moment there is a different archive per-vendor, because they don't like to see each others / unrelated things.
19:24 <infinity> Cause downgrading from a PPA removal is gross.
19:24 <xnox> mdeslaur:  the contents is the same / harmonised across them. and all of them are signed by the same key as far as I was able to trace them.
19:25 <infinity> xnox: Are all the packages in these archive unique, or are some of them forks of things in the primary archive?
19:25 <xnox> infinity:  majority of it, is opt-into specific Xorg stack and opt-into specific linux-oem flavour and ship docs.
19:25 <infinity> "specific xorg stack" sounds like forks...
19:26 <infinity> Unless it's uniquely-named driver packages built to work with the xorg in the archive.
19:26 <xnox> infinity:  no, as in use the -hwe xorg stack from the Ubuntu Archive
19:26 <xnox> instead of the ga hwe stack
19:26 <xnox> (meta package dependencies)
19:26 <xnox> (and conflicts)
19:26 <infinity> Okay, none of that requires an external archive.
19:26 <infinity> So, what's in the external archive that we want?
19:27 <xnox> infinity:  they do fork some things, there was fwupd hotfix update, bluez update, etc.
19:27 <xnox> infinity:  things that are not yet ready to go into Ubuntu Archive as an SRU.
19:27 <infinity> Right, see, forks of stuff in the archive is a non-starter for me unless it's also renamed with appropriate conflicts, etc.
19:27 <mdeslaur> how do we make sure an archive update doesn't then break those things?
19:27 <xnox> it's a timing issue, rather than content.
19:28 <infinity> Since the SRU team has no control over these archives, random forks just don't work.
19:29 <xnox> mdeslaur:  once updates/fixes are shipped in the OEM archives under SLAs, the OEM team works on upstreaming those fixes into devel and SRUing it. Such that after a period of time, none of the extra packages or fixes are needed.
19:29 <mdeslaur> how do we make sure that actually happens?
19:29 <vorlon> infinity: what if there were a tightly constrained policy regarding package versioning in the oem archives, and a requirement that the packages must be kept in the ppa (and maintained wrt security updates etc) until such time as the changes are merged back into the Ubuntu archive?
19:29 <xnox> mdeslaur:  meaning that next-LTS is shinny, but you are running bionic, so tough shit please wait 9m for things to start working =)
19:30 <infinity> vorlon: If there was strict (and actually followed) policy in said archives, that would be fine.  Unfortunately, we go back to oversight.  Who's going to keep them in check?
19:30 <xnox> mdeslaur:  how do we make sure that actually happens => there is a good stick, if it doesn't happen OEMs don't renew contracts, and $$$ is gone, and the archive is gone too.
19:30 <infinity> We make sure the primary archive is consistent because many people (some of whom are in this meeting) prevent developers from doing stupid things.
19:31 <vorlon> infinity: aiui the ppas will all be public, not private; we could require as a precondition for exposing this in the ui that the responsible team also provide us archive checking tools to monitor?
19:31 <xnox> mdeslaur:  infinity: please note, that these archive are already enabled, and are shipped on ubuntu pre-installed certified devices.
19:31 <mdeslaur> yeah but once we have the ppa, why would they pay to get stuff into the archive?
19:31 <xnox> mdeslaur:  infinity: thus they are in use by all Ubuntu Dell laptops, Lenovo Ubuntu Thinkpads etc. Unless one reinstalls or like removes those archives and downgrades.
19:32 <infinity> xnox: Yes, I'm well aware, and they're also responsible for statements like "we can't upgrade OEM machines because they have no upgrade path" and other great oddities from the past.
19:32 <mdeslaur> vorlon: I like the requirement that tools should be provided...and I'd also like some sort of policy that each package in the PPA needs to have an SRU bug filed immediately
19:32 <infinity> xnox: If they want us to play nicely with them, this is a great opportunity to make sure they play nicely with us.
19:32 <xnox> infinity:  mdeslaur: another shiny thing, is BIOS settings, kernel cmdline parameters, and/or instructions on which BIOS settings must be set to be "good on linux, instead of good on windows"
19:32 <infinity> And then maybe OEM install users can actually upgrade when everyone else does, not randomly lose OEM fixes to security fixes trumping them, etc.
19:33 <xnox> i do agree there is currently little integration/oversight into what's currently in teh OEM archives overall. I.e. kind of like MoM on steroids, to see all the published packages & their version numbers, etc.
19:34 <infinity> Anyhow, I'm not conceptually against an addition to ubuntu-drivers to make opting into OEM archives easier than Googling, typing, and praying.
19:35 <infinity> But I want some indication that users who opt into that won't be opting into a situation worse than the one they left.
19:35 <xnox> and an easy way to roll-back/downgrade back to vanilla, right?!
19:35 <vorlon> (maybe keeping the prayer step wouldn't hurt)
19:35 <xnox> infinity:  to me, switching kernel flavours is scary.
19:36 <infinity> Downgrading is something that we really shouldn't do, generally.
19:36 <infinity> Which is why I prefer forked packages to updated ones.
19:36 <xnox> yeah, i was thinking ppa-purge style, which can break things more than they were before.
19:36 <infinity> s/forked/renamed/
19:36 <xnox> infinity:  renamed, with metapackage pulling in a renamed one, whilst conflicting on the original name?
19:37 <infinity> But if they can commit to making 0 packaging changes, and only doing code updates in those non-renamed packages, downgrading to archive versions should usually be harmless.
19:37 <xnox> (just so i understand the logic, if it's bluez that is forked)
19:37 * xnox conffiles
19:38 <infinity> Renamed packages are a lot more effort to maintain, and unless someone like Timo is driving that, I fear for the quality coming out of the other end of the sausage machine.
19:38 <infinity> So I wouldn't make it a requirement.
19:38 <infinity> It's just that it works well when done right, cause they're easier to shuffle around. :P
19:38 <xnox> if one used a point release bionic to install, one has linux-hwe kernel, whilst these OEM metapackages might try to upgrade to linux-oem kernel, which is of lower versions everything.
19:39 <infinity> Right, I wouldn't allow this for bionic for that reason.
19:39 * xnox is not sure how to deal with that, cause grub picks highest number betwen linux-hwe & linux-oem kernel images
19:39 <infinity> In FF, linux-oem will be rolling with linux-hwe.
19:39 <xnox> is linux-oem-oesp1 a rolling one? or just a newer one?
19:39 <infinity> osp1 is a one-off that you should pretend doesn't exist.
19:40 <infinity> In FF, linux-oem will be rolling with linux-hwe.
19:40 <xnox> lolz
19:40 <infinity> Almost like I just said that. :)
19:40 <xnox> i am a lot more comfortable with a rolling linux-oem kernel
19:40 <infinity> bionic is a lost cause because of contract we signed that required certain things be a certain way.
19:40 <vorlon> hmm so it sounds like this isn't necessarily a viable solution for the current problems on top of 18.04 due to the kernel version issue?
19:41 <xnox> such that it's >= linux-hwe one at all times, specifically
19:41 <infinity> I don't think it's useful to have this conversation in the context of bionic, no.
19:42 <xnox> infinity:  i was thinking to add support in grub.cfg for a notion of kernel flavours. Yes i have both linux-hwe & ga-lowlatency flavours installed, and yes the two should be grouped by flavour, and sorted by version numbers, but i want to pick that "ga-lowlatency" is the default flavour, unless I boot into linux-hwe by hand to play games on the weekend. or some such.
19:42 <infinity> But for 20.04, we can certainly make the world more pleasant, if we and Canonical OEM can all agree on how it should work, and provide a user experience that's guaranteed to degrade neither functionality or security (though erring on the side of degrading functionality for security if we have to, IMO)
19:43 <infinity> xnox: Because grub.cfg isn't confusing enough already.
19:43 <vorlon> ah.  well, the goal for this was certainly to try to fix these problems in the 18.04 timeframe as well, not just once oems start shipping and being certified against 20.04
19:43 <xnox> =) i know right
19:44 <xnox> infinity:  i like the path in 20.04. What can we do to make things better in 18.04, without switching kernel flavours & xorg stacks?! But that doesn't get one into certified state.
19:44 <infinity> vorlon: I mean, even just from a "cementing policy" POV, I'm not sure how realistic that goal is.  We could maybe hammer it out just in time to move to 20.04 anyway.
19:45 * xnox does wonder why ubuntu.com/downloads has no links to the OEM installer images (which do exist) to install Dell Certified Ubuntu for Thinkpad Sonicmaster 2019
19:45 <xnox> for the 18.04 LTS images as they are created / certified.
19:45 <infinity> xnox: I'm not sure how to get where we'd went to be for 18.04 without asking Timo to roll up a linux-oem-hwe kernel.  I don't think it's reasonable to wrench people down from an HWE stack to a GA stack to provide a certified experience.
19:46 <infinity> When, in reality, that will almost always be a functionality downgrade too, despite now earning a sticker.
19:46 <xnox> infinity:  true.
19:46 <vorlon> infinity: I don't know in practice how long after the LTS release it starts shipping from OEMs, but I assume that's also not zero and I wouldn't like us to preclude fixing problems affecting users now because we /think/ the policy + engineering will take a certain amount of time
19:47 <mdeslaur> do the ppas actually contain modified kernel now?
19:47 <infinity> They'd better not.
19:47 <infinity> All kernel mods are meant to live in linux-oem now.
19:47 <infinity> But, as stated, linux-oem in bionic is pegged at GA (4.15).
19:47 <vorlon> mdeslaur: AIUI the kernel used is the oem kernel from the archive, but something needs to drive selection of that kernel for the "certified experience"
19:47 <xnox> mdeslaur:  no, but they require that one runs the linux-oem or linux-oem-osp1 kernel flavours from teh Ubuntu Archive, as those machines are new and need new intel-stack support which is not in the GA kernel.
19:48 <infinity> With a weird one-off (linux-oem-osp1) at 5.0
19:48 <xnox> ie. think Cannon Lake / Ice Lake
19:48 <mdeslaur> so I guess I don't see what the issue is...either that ppa moved people to linux-oem or it moved people to -hwe, whatever is requires for that particular system
19:48 <mdeslaur> wow, I can't type apparently
19:49 <infinity> mdeslaur: The problem (well, the one I have) is that an OEM may have sticker certified on linux-oem, but the user experience is actually better with linux-hwe.
19:49 <infinity> Which is almost always going to be true right now, given the massive version skew.
19:49 <vorlon> I don't have a problem with that.
19:49 <mdeslaur> hrm
19:49 <mdeslaur> it's opt-in, I don't see a big issue either
19:49 <xnox> mdeslaur:  if one installed using vanilla ubuntu bionic point release, one needs to *purge* linux-hwe to boot installed linux-oem flavour.
19:49 <vorlon> the "certified" stack is opt-in, yeah
19:49 <xnox> mdeslaur:  this is where prayers come in, that the linux-oem one boots ;-)
19:50 <infinity> vorlon: But there's a belief that "certified" also means "better" somehow.
19:50 <vorlon> for me, the bigger concern is the practical integration issues with the downgrade of major kernel versions
19:50 <infinity> I don't think that would actually be true for most hardware on bionic downgrading from 5.3 to 4.15
19:50 <xnox> infinity:  vorlon: can we start experimenting with these repositories, including on bionic, but without xorg/kernel stack changes?
19:51 <xnox> such that one at least gets documentation, manuals, bluez hotfix / usersapce hotfixes?
19:51 <vorlon> infinity: ok but the certified stack is supported, so I don't see why the TB would need to be in the middle of trying to adjudicate whether the hardware support in one is better than the other
19:51 <mdeslaur> I think release linux to linux-oem is fine too
19:51 <xnox> and start working on how to do kernel flavour changes / xorg stack changes? and maybe it will not be an issue?
19:51 <mdeslaur> but let's leave hwe alone
19:52 * xnox ponders if we need a kernel flavour selector in the GUI
19:52 <infinity> vorlon: Because I think it's irresponsible to suggest to users "a certified stack for your hardware is available!" when it's quite likely to degrade their experience by installing it?
19:52 <infinity> As someone who actually does heavy 3D and such on laptops we certify, newer is nearly always better when it comes to kernel and X.
19:52 <xnox> infinity:  users would rollback, and "just use ubuntu" and normally users will never opt-into it, if they think everything is working anyway.
19:53 <infinity> xnox: You think?  ubuntu-drivers popups kinda imply that if you want things to work better, you should take their recommendations.
19:53 <xnox> infinity:  i think "certified" in practice means only the preinstalled machines.
19:54 <mdeslaur> I'm not convinced hwe kernels are necessarily better
19:54 <mdeslaur> as a user, I just want stuff to work, and certified means just that
19:54 <xnox> infinity:  we could offer this thing, and push people onto linux-hwe kernel and xorg-hwe stack. It's not certified, but we expect that that one must actually work, and by policy already include all the the OEM flavour fixes, no?
19:54 <xnox> in 18.04
19:54 <infinity> xnox: That would be nice if it were true.
19:54 <xnox> infinity:  i'm with mdeslaur on "just work, and when i call my dell support paid up option they do support me"
19:55 <infinity> xnox: Unfortunately, the OEM team's policy for 18.04 was "make sure it's all migrated to linux proper by 20.04".
19:55 <xnox> even if that means running an older kernel
19:55 <infinity> xnox: So linux-hwe isn't guaranteed to have all the linux-oem bits until 20.04's kernel is backported.
19:55 <xnox> infinity:  in that case, you advocate certified means a kernel below linux-hwe one. and we should be installing that one.
19:56 <infinity> Yes, that's what certified means.
19:56 <vorlon> infinity: history shows that the typical Ubuntu desktop user applies exactly zero security updates that aren't automatically installed by default; I am not convinced that most users are going to be enticed into changing out their kernel with a properly-framed ubuntu-drivers popup
19:56 <xnox> infinity:  and i was hoping to make this thing, less "nagging" to opt-into then the current additional drivers dialogs are.
19:56 <xnox> ie. make it dismissable.
19:56 <infinity> I'm arguing that, as well-informed users, you think "certified" means "stable, and I get support, even if it's not the best and shiniest".
19:56 <infinity> I don't think you represent what normal people do when they see a popup suggesting "better" drivers.
19:57 <vorlon> sure, and I'm arguing that low-information users click minimize on the dialog so it doesn't matter what it says
19:57 <xnox> infinity:  which is exactly what OEM teams and hardware vendors dow itht the linux-oem kernel flavour
19:57 <vorlon> I'm absolutely not using myself as the reference here
19:57 <mdeslaur> nobody cares about the dialog unless their bluetooth doesn't work
19:57 <infinity> I guess a good guage for that would be how many people have binary nvidia/amd drivers installed, if we actually had stats. :/
19:57 <vorlon> anyway, we're pretty much at time for this meeting; I think we're going to need further discussion.  What should next steps be?
19:58 <xnox> infinity:  we might have that with the next lts, because nvidia drivers are installable without enabling wifi & first-boot data has info on those.
19:59 <infinity> I think we might need to circle back to the OEM team to see if they have intent WRT kernel downgrades.  Maybe that was out of scope for them and we're overthinking it, or maybe that's the one critical piece they need.
19:59 <xnox> good action / question
19:59 <infinity> And we need to establish policy on patched userspace bits in the external archives.
19:59 <vorlon> I think we've agreed on some conditions that would make such archives acceptable for 20.04.  xnox, do you want to mine the scrollback and summarize to the mailing list what you understand the consensus to be?
19:59 <xnox> vorlon:  sure.
19:59 <mdeslaur> can we have a wiki page that states what the policy is on getting the ppa changes SRUed, and what will the policy be with kernel/xorg upgrades/downgrades?
19:59 <xnox> vorlon:  will circle it for proof-reading before posting publically
19:59 <xnox> mdeslaur:  ok, i can start a draft.
20:01 <infinity> #action xnox to start a draft summarizing the OEM archive portion of the meeting which vorlon, infinity, and mdeslaur will review, edit, and ratify before we move on to figuring out the next step :P
20:01 * meetingology xnox to start a draft summarizing the OEM archive portion of the meeting which vorlon, infinity, and mdeslaur will review, edit, and ratify before we move on to figuring out the next step :P
20:01 <infinity> [2/3 subquorum required if one of us slacks, to avoid holding up the process]
20:01 <infinity> Any objections?
20:02 <vorlon> wfm
20:02 <mdeslaur> on objection from me
20:02 <infinity> Righto.
20:02 <infinity> #topic mailing list archives.
20:02 <infinity> Looks like we've covered it all.
20:03 <infinity> #topic boogs
20:03 <infinity> Nein.
20:03 <infinity> #topic Next chair
20:03 <infinity> kees (hah), mdeslaur backup
20:03 <mdeslaur> ack
20:03 <infinity> #topic AOB
20:03 <infinity> Going once.
20:03 <mdeslaur> who's this kees guy I keep hearing about?
20:03 <infinity> Twice.
20:03 <infinity> Sold to the guy with the beard.
20:04 <infinity> #endmeeting