15:00:22 #startmeeting ESR-Transition 10/03 15:00:22 Meeting started Thu Oct 3 15:00:22 2019 UTC. The chair is pili. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:22 Useful Commands: #action #agreed #help #info #idea #link #topic. 15:00:27 hi everyone 15:00:36 let me create a pad quickly 15:01:20 https://pad.riseup.net/p/tor-browser-esr-transition-keep 15:01:31 is here to listen 15:01:38 hi! 15:03:05 I'm just adding some questions we should figure out the answers to 15:04:17 in part because we want to get this funded and will need to answer those questions 15:04:48 <- is here 15:04:58 hi 15:05:07 and in part because we will need to figure this out to do this transition! 15:05:15 did we have an other pad with the same discussion in the past? 15:05:46 boklm: good point, I think we may have had 15:05:58 * boklm vaguely remembers one 15:05:59 let me try to dig that up while someone else kicks off the discussion? :) 15:06:29 hi 15:06:42 okay, so one of the questions we need to answer is how we'll keep up with the release train 15:06:49 pad from our last esr convo https://pad.riseup.net/p/otf-tb-esr-migration-brainstorm 15:06:58 alsmith++ 15:06:59 alsmith: thanks 15:07:25 this is, honestly, a good question 15:08:01 but right now GeKo and Mike are spending some of their days looking at all of the changes since the last esr 15:08:20 alsmith: thanks :D 15:08:41 and that is something we can create roles for, and do it weekly or a few times per month 15:08:48 from what i have fresh in mind right now when doing the feature review 15:08:50 rebasing and resolving merge conflicts is another issue 15:09:01 i think it takes like 2 days per release for that part 15:09:46 okay, so with a 4-week cycle, that isn't too bad 15:09:55 yep 15:10:08 i expect a little less work in that case even 15:10:14 as the changes are not as many 15:10:17 (probably) 15:10:18 GeKo: That sounds about right, but sometimes more time is needed to dig deeper on some complex change (especially if it is an area of code that is unfamiliar). 15:10:18 yeah 15:10:18 Mozilla has three trains: Release, Beta, Nightly. The intention is that Tor Browser release follows release; and Tor Browser beta follows beta; right? Would there be a Tor Browser Nightly that follows Nightly? 15:10:18 It seems like tracking Beta and Release wouldn't be too hard with regards to ongoing work (e.g. point releases and patches landing in -release and -beta... 15:10:18 But it might be 50% or more of someone's time tracking Nightly. 15:10:38 Of course not tracking Nightly means that the work when we release (and bump the Beta version) is increased 15:10:53 But hopefully the Beta -> Release bump would be minimal 15:11:12 i doubt we have the capacity to track nightly closely 15:11:15 yeah, and not tracking nightly we don't catch bugs/issues early enough to fix them before Beta 15:11:19 but that's a tradeoff 15:11:37 and especially to resolve bustage in nightly builds by following mozilla's nightly closely 15:11:38 I think last time we discussed this we said we would only track 2 trains :) 15:11:47 yes 15:12:02 Okay, cool 15:12:12 maybe we could have two nightlies branches, one based on beta, and an other one on nightly 15:12:24 maybe 15:12:27 the one based on nightly being more often broken 15:12:32 we can always add the nightly train later if two trains is simple for us 15:12:37 but starting with the nightly on beta as a goal seems not too bad 15:12:44 yes 15:12:53 we can still add that later 15:13:25 this brings us to the question about automated rebasing 15:13:50 doing the rebasing is easy, but we'll need alerts when it fails 15:14:13 doing this manually once a week isn't something we can do long-term 15:15:07 but, overall, i think that shouldn't impact our capacity too much 15:15:35 unless there are significant merge conflicts or we need to backout a change that landed upstream 15:16:06 Do we have enough experience to know that the rebasing will be easy? Sometimes it will not be easy, right? 15:16:21 boklm: sent an email about quilt a little bit ago 15:16:24 ah, what you just said covers my comment 15:16:36 so part of increasing our capacity to move into this cycle is the autorebasing (which is in O2)? 15:17:27 i personally think this will be two-fold 15:17:42 While the phrase "has been rewritten in perl to make [it] more maintainable" made me simultaneously giggle and cry - I do actually agree that maintaining a series of patches to be applied each time - rather than trying to rebase - sounds good and is worth investigating 15:17:43 the first part is auto-rebasing, and the second part is testing 15:18:25 alsmith: yes 15:18:55 tjr: ah, i think when we say "rebase" we really mean "cherry-pcik" 15:18:59 *cherry-pick, too 15:19:20 maybe that is part of the confusion? 15:19:31 at least that was my thought on this 15:19:38 GeKo: ^ ? 15:19:52 Well; when youcherry-pick from branch a to branch b - and there's a merge conflict, the tool will require you to fix it. 15:20:00 no we actually rebase with every esr release right now 15:20:45 like git rebase -i --autosquash mozilla/esr68 15:20:54 actually i cherry-picked :) 15:21:03 which is nice in some scenarios 15:21:13 ah, but you mean from now on 15:21:29 like getting rid of all the !fixups and !squashs 15:21:44 and one can reorder the commits as one deems useful 15:22:10 but, yes, rebasing in that sense might not be what we want here 15:22:11 yeah, autosquash is a nice feature, and we should continue using that :) 15:22:32 i think we can post-process the branch 15:22:54 If you've got a script that (every day) grabs mozilla-beta/master and cherry-picks patches a,b,and c from tor/last-release - when a breaks on day N, it will break on day N+1, and N+2 15:22:54 You need to create a new a called a' and stick it in some branch and then cherry pick a' 15:22:54 And - as boklm/greg notes - the change from a -> a' is difficult to find - you need to track branches and compare across branches. 15:22:55 With the quilt approach, you have a git repo that contains patches a, b, and c - and every day you apply a, b, c on top of mozilla-beta/master. If on day N, a breaks; you fix up a and commit an update to patch a to the quilt repo - and you can see the history of a as it needs rebasing. 15:23:07 but for moving out patches onto the newest beta, it seems like we'd want to cherry-pick 15:24:27 tjr: yeah, i think that's a neat approach 15:24:28 However - I'm not very familiar with autosquash so if everyone thinks that's a better workflow... :) 15:24:52 and we should explore it for our "rebasing" efforts 15:24:57 we can use a git branch for creating the patches 15:25:15 but this is an interesting idea 15:25:26 boklm: did you have write an email or ticket about this? 15:25:43 i think if that works for a large project like a kernel 15:25:48 it might be for us, too 15:25:49 sysrqb: yes, I sent an email a few hours ago on tbb-dev 15:25:54 sysrqb, boklm: [tbb-dev] Nightly rebase experiment proposal 15:26:06 boklm: ah! :) 15:26:11 heh 15:26:12 (and we even don't have to reinvent the wheel) 15:27:07 ok, so it seems like we have some idea of the process we will need to take with each new release 15:27:08 more or less... 15:27:43 and we should figure out, on average how long that will take us with each release... 15:27:44 (FWIW in Debian, we use quilt as the storage/transfert format for patches on top of upstream; but many, if not most, people use other tools, such as gbp-pq, to manage this patch series with quilt, because they find quilt too bothersome; I'm part of these people) 15:28:21 (A side note - we don't need to talk about right now but is echoing very loudly in my head - is what amount of automation we want to put into Mozilla infrastructure and what into Tor infrastructure.) 15:29:22 i think that'll depend on what tooling we use 15:29:41 * tjr is now splitting attention between two meetings 15:29:54 but given we're a little limited on resources, that seems like a good discussion to have 15:29:57 tjr: that sounds like it could be a good discussion for the tor/mozilla sync? :) 15:30:10 we wanted to discuss some of this then anyway 15:30:28 I'm happy to discuss now also if people would prefer though 15:30:53 i don't think we have all the necessary people, right? 15:31:16 but we can discuss it on our side, at least, and get an idea of what we'd want to offload onto Mozilla 15:31:34 Sure. My starting point for discussion would be around "What do you want to automate?" and then "How do you want that to behave?" 15:32:26 I think any automation that happens in Mozilla infrastructure should probably be considered to be owned and maintained by Tor (and me) and just running there though. So if your build or whatever breaks, it might be disabled automatically, and it'll be up to us to fix it and re-enable it. 15:32:51 that's understandable 15:33:10 do mozilla provide this type of infra support for other orgs? 15:33:21 or would ew be a weird special case 15:33:53 (if you know, off hand) 15:33:58 I think something we could do is pushing our rebased branch every night on Mozilla Try, to run the tests, skipping the patch that changes all our prefs (which breaks a lot of tests) 15:34:12 yep! 15:34:25 yes, but that's a different can of worms 15:34:46 as tests will still be broken that way 15:35:09 and it might be hard to figure out whether that's due to some nightly feature or whatever 15:35:10 we'll need to audit every test 15:35:29 i guess it could be part of our myriad of qa efforts, though 15:35:34 maybe we can limit the tests that are run to the few that test the most important features we care about 15:35:47 could be 15:36:00 as a start 15:36:13 i'd just wary to spent too much time on this 15:36:26 i fear that's just a deep rabbit hole 15:36:32 ah yes 15:36:49 i am more in favor of setting up our own integration tests properly 15:36:58 and write more tests for that part 15:37:02 like we did a while ago 15:37:17 we still have that infrastructure running 15:37:32 yes 15:37:33 just the tests need to get fixed (again) and then expanded 15:38:24 we are basically avoiding all the infra setup overhead in that scenario 15:38:25 i think use Firefox tests and Try builds is helpful, but we can disable some of them (or maybe most of them) 15:38:32 sure 15:38:35 no doubt 15:39:00 at this point we don't know what is broken in Tor Browser compared with Firefox 15:39:20 but there will be hard choices to be made and i would opt to not fix all the tests on try for us 15:39:34 but rather invest in our own infra 15:39:37 okay, that's fair 15:39:52 but ideally we would have both :) 15:40:24 sysrqb: I think Thunderbird has some stuff running in Moz infrastructure but I'm not sure exactly... 15:40:25 yeah 15:41:34 tjr: okay 15:42:11 this is somethign we can thik about 15:42:53 Monitoring & evaluating: "Describe how the project's success will be evaluated and any metrics that will be collected related to the project's outcomes and impact" 15:43:13 we can decide on a point where we declare it complete 15:43:39 maybe after shipping two full releases? 15:44:01 it seems like this can be a little arbritrary, as long as it's defined 15:44:10 It wouldn't be hard to develop milestones. Metrics I'm less sure about. 15:44:30 yeah, i'm pausing on the metrics part 15:44:33 Perhaps "Average time between introduction of a mozilla patch that currently breaks things to time it is dealt with" 15:44:43 Current time: 8 months. New time: 2 days - 2 weeks! 15:44:54 :) 15:45:07 lol :) 15:45:08 that is a good way of showing it though 15:45:37 I wonder if there's something there about less/no reports of tor browser being outdated 15:45:44 but I'm not sure if we have any metrics of that right now 15:45:52 we can provide more defined improvements from the ETP portion of it 15:45:53 metrics could be how much time per month we spend to keep things working, instead of working on new features 15:46:14 boklm: ah, that's a good point, too 15:46:17 boklm: that's a tricky one... it may be that we have less time for new features :) 15:46:37 Metrics are geneally the things the customer cares about but I'm not sure who the customer is here 15:46:52 pili: not really 15:47:03 but i see this from time to time 15:47:07 we could check the metrics that we had for the TBA project 15:47:16 get inspiration there 15:47:17 good idea pili 15:47:22 in the begining we will spend a lot of time to just keep things working, but after a few releases we should improve and have more time for working on new features 15:47:29 boklm: I hope so! :) 15:47:59 (12 minutes warning) 15:48:21 i want to circle back to the ‘sustaining the pace’ question 15:48:30 (or, maybe let's say 7 minute warning, for those who have a meeting in the next hour) 15:48:37 alsmith: sure 15:48:47 I think overall the new approach will require more engineering resources… forever. But maybe I am pessimistic. 15:48:52 are there other ways we are preparing the team? 15:49:36 Regardless, we have to make this transition for mobile at least. 15:49:59 yeah 15:50:17 i don't know if this will necessarily require more engineering resources overall 15:50:29 we just spend the last two months rebasing onto the next esr 15:50:32 *spent 15:50:47 that's a lot of engineering resources 15:51:07 (that didn't take 100% capacity, but it was a large part of it) 15:51:12 It is hard to know at this point but I think we will be interrupted a lot to fix things as we go instead of working in bigger chunks… 15:51:22 and context switches have a cost 15:51:26 mcs: +111 15:51:31 that could be, i agree with that 15:51:49 Anyway, what else should we discuss now? 15:51:51 but, we honestly won't know until we start 15:51:58 +1 15:52:00 and we'll see how this goes with mobile first 15:52:12 (unfortunatelly, that'll be too late for this proposal) 15:52:35 alsmith: are there other specific questions we should answer? 15:53:05 i think the fact that the responsibilities will be distributed over the team, maybe/probably with rotating roles 15:53:14 is a large change for the team already 15:53:25 we do need to elaborate on the concrete actions — ie, question two in the pad, but i don’t know if there’s really time to do that all now 15:53:26 and we're getting mentally prepared 15:54:07 ok sysrqb i (am guessing) that staffing/role definition element is something they want to hear 15:54:36 we can enumerate the concrete changes we're expecting 15:55:13 they tie into the pieces from question 1, as well 15:55:21 as to who we'll maintain the pace 15:55:25 *how 15:57:12 okay, anything else for the next 3 minutes? 15:58:30 i’m realizing that this wasn’t clearly communicated in question 2 but we need to write more concrete steps for the Activities that start with ‘evaluate’ / ‘set up’ / ‘investigate’ — they are concerned about how we determined our estimations & want to see those details. 15:58:43 maybe thig is something pili & sysrqb could work on? 15:58:55 alsmith: yeah, sure thing 15:59:02 awesome thanks 15:59:05 ok 15:59:20 ok, I think I'll end the meeting here then 15:59:24 #endmeeting