18:00:51 <donuts> #startmeeting Tor Browser Release Meeting 2022-11-28
18:00:51 <MeetBot> Meeting started Mon Nov 28 18:00:51 2022 UTC.  The chair is donuts. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:51 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
18:01:10 <richard> #agreed
18:01:20 <richard> perhaps not
18:01:36 <donuts> lol
18:01:38 <donuts> hello all
18:01:46 <richard> o/
18:01:49 <donuts> let's take a couple of mins to update the pad before we get started
18:02:05 <donuts> "Need to fix Android's problems" made me lol
18:03:17 <donuts> ready?
18:03:23 <PieroV> Yes
18:03:54 <donuts> excellent
18:03:58 <donuts> let's start with gaba's question
18:04:12 <donuts> I think both richard and I may have talked to gaba about this individually last week?
18:04:29 <richard> yes most likely
18:04:51 <donuts> it sounds like if we've got another alpha and stable coming, we should just push the big release to Jan regardless?
18:04:54 <richard> so does it make sense kind of depends on the next alpha release which has been delayed due to a couple of issues
18:04:56 <donuts> however I believe Tails are already aware of the upcoming changes, yep?
18:05:06 <richard> well, ideally we don't have another full stable
18:05:17 <donuts> yeah just the android stable
18:05:29 <PieroV> Yes, Tails is already aware and already tested them
18:05:32 <richard> an android stable is fine compared to a full stable
18:05:33 <PieroV> They say it's okay
18:05:37 <donuts> awesome
18:05:42 <richard> yeah we looped them in early on
18:05:48 <PieroV> They found some problem initially, but we've already fixed them
18:06:06 <gaba> I'm worry that there may be other orgs having this dependency
18:06:16 <donuts> does gaba's point about gettor probably still stand? or has someone chatted to anticensorship team about that already?
18:06:22 <gaba> Sending a mail to tor-project@ would be a good idea
18:06:25 <PieroV> That said, I don't like the idea of having to backport other security fixes
18:06:33 <richard> yeah that^
18:06:42 <PieroV> We're already later than we've expected and Firefox 91 has eol
18:06:43 <boklm> we have a ticket for gettor
18:06:48 <gaba> meskio has the work on gettor on his plate but if there are bugs afteter the release it will be good to have this in january to be here to help/support people
18:06:54 <PieroV> 'ed almost 3 months ago
18:06:55 <donuts> yep we discussed adding an email to tor-project as a new item under the "notify stakeholders" checklist for major releases
18:06:58 <boklm> https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/109
18:07:02 <PieroV> I think security should be #1 priority
18:07:26 <richard> backporting security fixes gets harder/less reliable the further away we are from the intended release they are
18:07:35 <donuts> right
18:07:51 <gaba> Is there any difference between doing it around Decmeber 10th and January 10th?
18:08:00 <richard> so other things equal, a 12.0-based stable will be easier to release in dec then 11.5
18:08:22 <richard> yes, if we do another 11.5 in dec that's another round of security backports
18:08:40 <donuts> well breaking gettor would be very bad too
18:08:47 <richard> what are the other team's blockers?
18:08:52 <richard> why would gettor break?
18:09:04 <donuts> the links to localized versions?
18:09:09 <gaba> because the way gettor is getting TB
18:09:18 <gaba> meskio can fix it in december and coordiante with you all
18:09:21 <gaba> that is not a problem
18:09:28 <gaba> the issue is after the release if there is any problem
18:09:38 <donuts> and nobody is around to deal with it
18:09:42 <gaba> right
18:09:45 <PieroV> Could the other team try to update on the alpha just to test it?
18:10:07 <donuts> well how much time do we have between release and vacations? three weeks?
18:10:11 <gaba> give alpha versions to people instead of stable you mean?
18:10:31 <richard> yeah basically
18:10:34 <PieroV> gaba: no
18:10:38 <gaba> if we have 3 weeks between release and vacations then it may not be an issue
18:10:55 <gaba> but in that case the release should happen this week
18:11:10 <PieroV> I've meant: try to run dev instances with the alpha
18:11:10 <richard> if the 12.0a5 release goes smoothly then we'll have exercised all the machinery needed for 11.5 -> 12.0 no?
18:11:16 <donuts> hang on, I'm going to open the calendar
18:11:22 <richard> apart from the forcing people through 11.5.8 code path right boklm?
18:11:30 <PieroV> It's less than 3 weeks
18:11:34 <boklm> richard: yes
18:11:42 <donuts> Release was planned for the 5th
18:11:43 <PieroV> It's like 10 days after the release
18:11:48 <PieroV> Release is planned for 12
18:11:50 <PieroV> 13
18:11:50 <donuts> I'm guessing it's delayed because of the alpha delays
18:11:59 <PieroV> Nope, it's delayed to rebase on 102.6
18:12:07 <donuts> oh okay, release calendar needs updated
18:12:12 <donuts> 13 is a bit late tbh
18:12:14 <richard> sorry we discussed in the tb meeting this am
18:12:18 <donuts> ah sorry
18:12:22 <donuts> no worries
18:12:32 <PieroV> But I think we can do .1 based on 102.6
18:12:35 <gaba> yes, it is late
18:12:51 <gaba> for a release that will have a big change
18:12:53 <richard> so the current plan is to do 12.0 once we get 102.6 tags w/ i think you said is the 8th or something PieroV?
18:12:54 <donuts> it's like 1.8 weeks to deploy fixes
18:13:04 <donuts> i think we need to abandon ship
18:13:07 <PieroV> richard: it is, IIRC
18:13:16 <PieroV> https://wiki.mozilla.org/Release_Management/Calendar
18:13:25 <PieroV> Tags on the 8, release on the 13
18:13:30 <donuts> eject, eject
18:14:48 <richard> blech
18:15:16 <PieroV> Okay, then what about doing .1 based on 102.6 is 12.0a5 goes well?
18:15:17 <donuts> sorry :(
18:15:42 <donuts> so back to the original plan of releasing early next week, if based on 102.6?
18:15:58 <PieroV> 102.5
18:16:11 <PieroV> But yes
18:16:14 <richard> well i mean you're not wrong, it is risky
18:16:31 <PieroV> 12.0a5 should be very close to what we will ship
18:16:32 <donuts> pierov: ack
18:16:44 <PieroV> The only thing I'd change is the macOS Chinese fonts
18:17:00 <donuts> gaba: how do you feel about that?
18:17:04 <richard> tbh i'd rather leave everythign as is between 12.0a5 and 12.0
18:17:11 <richard> and update chiense in 12.0.1
18:17:15 * meskio is around if needed, I'm just noticing to be poked here
18:17:25 <gaba> seems fine
18:17:32 <PieroV> richard: that'd make 12.0.1 fingerprintable
18:17:37 <donuts> Option 1: 102.5-based release on the 5th, option 2: 102.6-based release in Jan
18:17:49 <PieroV> I'd rather re-build 12.0a5 entirely to include them
18:17:54 <donuts> not sure if i've got those FF numbers right but w/e
18:18:00 <gaba> thanks donuts for summarizing options
18:18:18 <PieroV> Option 2 needs also a risky security backport
18:18:38 <PieroV> How many users are using gettor compared to users of other channels?
18:18:48 <donuts> yeah, option 1 feels like a good compromise if we can patch gettor in time
18:19:07 <richard> gettor can also just keep folks on 11.5.8 if necessary
18:19:18 <richard> users would update to 12.0 after first botstrap
18:19:27 <PieroV> Yes, that's where I wanted to arrive :)
18:19:35 <donuts> ah yes of course
18:19:43 <donuts> so long as we don't delete 11.5.8?
18:19:44 <gaba> right! that could be an option. To keep gettor in an older version until 12 is release /cc meskio
18:19:50 <richard> but this is all predicated on 12.0a5 working as we think it should
18:19:50 <PieroV> We won't delete 11.5.8
18:19:54 <gaba> but we will not have issues also in TB if something is a problem
18:19:56 <PieroV> It's a forced step for users
18:20:04 <donuts> pierov: right, great
18:20:04 <richard> imagine deleting mars before you intend to lol
18:20:10 <donuts> hahahaha
18:20:37 <meskio> gettor can be ready for TB 12 by mid december if needed
18:20:37 <gaba> before I meant until January. Keep users in gettor to the older TB version
18:20:59 <donuts> I've added notes to the pad
18:20:59 <gaba> BUT there is no possible bugs or problems in TB 12 that will need to be supported?
18:21:05 <donuts> but I'm happy with this plan if everyone else is
18:21:06 <gaba> regardless of gettor
18:21:12 <donuts> (better start working on the blog post etc...)
18:21:44 <PieroV> donuts: I think we've already set pretty much anything on 12.0a4
18:21:54 <PieroV> 12.0a5 brings a major thing, too: macOS ARM
18:22:00 <richard> yeah 12.0a4 is very close to release
18:22:10 <donuts> finally \o/
18:22:15 <richard> yeah macOS ARM is the big thing, which we've validated in testbuids
18:22:20 <donuts> fantastic
18:22:22 <richard> just need to you know, actually ship it to users
18:22:41 <donuts> what else is comign in 12.0a5? is that it pretty much?
18:22:43 <donuts> *coming
18:22:47 <PieroV> Let me go through the changelog
18:22:54 <richard> i think letterboxing fixes?
18:22:58 <donuts> oh yes
18:23:03 <PieroV> Yes, and support for Albanian
18:23:07 <richard> which should lbe transparent to end users
18:23:26 <donuts> but there's a problem with reproducibility atm?
18:23:26 <richard> here's the release prep ticket: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40668
18:23:31 <donuts> lovely ty
18:23:50 <PieroV> A few minor fixes. The "restart to apply language
18:23:51 <donuts> "Rename OS X to macOS"
18:23:52 <donuts> wow
18:23:52 <PieroV> " above all
18:23:53 <donuts> finally
18:23:57 <donuts> :D
18:24:05 <PieroV> That might fix some problems alpha users might have found with multi-locale
18:24:17 <boklm> it seems we can't reproduce the reproducibility problem
18:24:22 <meskio> about gettor, I'm expected the telegram bot to have some issues if it gets released and we don't update, but we are working on the update and we can be ready if we have a date for it
18:25:01 <meskio> the gettor telegram bot update is not going to be backward compatible, that means I will need to deploy it somehow around the time you release
18:25:29 <meskio> I'm planning to make the gettor email backward compatible
18:25:51 <richard> meskio: how much lede time do you need?
18:26:03 <richard> i can ping you on the above ticket when we're signing/publishing/etc
18:26:53 <meskio> to deploy? will be nice to know the day before
18:27:37 <meskio> but I need to know how much I need to rush the development, I believe is ~2 days of work, but I have to sit down to do them
18:28:02 <meskio> so having a rough date will be really handy
18:28:21 <richard> so the rought date if all goes well is the 13th of Dec
18:28:33 <meskio> nice, I can work with that
18:28:35 <richard> if all does not go well, sometime in january
18:28:39 <donuts> wait, no
18:28:47 <donuts> 13th is too close to vacations
18:29:04 <PieroV> richard: I thought we were discussing for the 5th
18:29:06 <donuts> I thought we meant the 5th, or January?
18:29:07 <donuts> yeah
18:29:17 <gaba> not only for gettor but for you all people! what if there is an issue with v12 release?
18:29:26 <donuts> ^^^
18:29:38 <richard> hold up, 12.0 would be on 102.6, which is tagged on the 8th
18:29:38 <PieroV> Like: if 12.0a5 is okay let's just ship it as 12.0
18:29:52 <PieroV> richard: we can do 12.1 for that
18:29:53 <richard> hm that could work too
18:30:03 <PieroV> err, 12.0.1
18:30:57 <donuts> so meskio the real date (if we launch in December) is the 5th, i.e. next Monday :P
18:31:06 <donuts> If we miss that, it'll be January
18:31:20 <PieroV> I think we can do also Tue or Wed, if it's better for gettor
18:31:37 <richard> for that to work we would need to get 12.0a5 out wednesday or thursday, see how upgrades go over the weekend
18:31:52 <richard> and then have 12.0 ready on the 5th
18:32:09 <meskio> I'm confident I can be ready for the email support by monday, I'm looking into telegram as I'm not the main dev, but I'll do my best
18:32:17 <meskio> let's plan everything for monday
18:32:29 <richard> which is fine, we'd optimistically do the 12.0 release prep right after 12.0a5
18:32:38 <donuts> tyvm all <3
18:32:59 <gaba> We also need to coordinate with championquizzer[m] and user support team
18:33:01 <PieroV> richard: are you going to rebuild Firefox, too?
18:33:02 <gaba> for the weeks after
18:33:05 <PieroV> For 12.0a5
18:33:30 <richard> on tb-build-05 i'm currently rebuilding everything
18:33:31 <gaba> ^ ggus
18:33:45 <donuts> championquizzer is in the loop, I'll speak to them again tomorrow at the UX team meeting
18:33:49 <gaba> ok
18:33:51 <richard> and put my old tor-browser-build on ice until i can diff the two later
18:34:14 <richard> i'll do 11.5.10 locally or on tb-build-04
18:34:14 <PieroV> So, for the Chinese font I think it'd be very good to get it into 12.0, to prevent macOS users to be fingerprintable between 12 and 12.0.1
18:34:26 <PieroV> It's a change of a few prefs
18:34:31 <richard> it's just a font to bundle and add to the allow list?
18:34:34 <richard> kk
18:34:46 <PieroV> A system font to add to the list,
18:35:04 <PieroV> The MR has already been approved, but I didn't merge it
18:35:07 <richard> ah ok even easier
18:35:15 <PieroV> Because I wanted 12.0a5 out first
18:35:50 <PieroV> I think it'd be safe to add it directly for 12.0, but another possibility is to rebuild Firefox, too
18:35:50 <richard> ok
18:36:02 <PieroV> And tag build3 for 12.0a5
18:36:34 <richard> the release calendar has been updated with the tentative plan assuming 12.0a5 updates go as expected
18:36:48 <donuts> ty richard
18:37:03 <richard> 11.5.10 tomorrow, 12.0a5 thursday and 12.0 after if alpha doesn't explode
18:37:08 <donuts> is that it for your wishlist pierov? (given the tighter turnaround for next week)
18:37:29 <richard> if it looks at all sus we delay to january and do another backport stable after the 8th
18:37:37 <donuts> 👍
18:37:50 <PieroV> donuts: yes, we can sort it out later whether to add it directly in 12.0 or in 12.0a5
18:37:54 <donuts> sounds good
18:38:14 <donuts> since 12.0a5 won't be live long before stable, we should recruit tor-internal to test test test too
18:38:35 <PieroV> And 11.5.10 for tomorrow is goodish, though having to rebuild Firefox once again only because we're chaning version string is a bit on the annoying side :P
18:38:55 <richard> yes and we need to test every single platform for build-to-build upgrades
18:38:57 <donuts> what's this ominous android situation?
18:39:04 <donuts> with 11.5.10?
18:39:08 <richard> ugh
18:39:09 <richard> Google
18:39:16 <PieroV> So, Google bumps the required SDK version every year it seems
18:39:31 <PieroV> It might be possible to ask to be exempted, but maybe it's a manual thing?
18:39:34 <richard> ^that
18:39:40 <donuts> ah, fun
18:39:56 <PieroV> And we've discovered only today that we could ask for exemption, and went for the upgrade instead
18:40:05 <PieroV> But things broke on Android 12 and 13
18:40:06 <richard> caught us unawares and so pushed out a fix 11.5.9 which broke newer android deviecs becuase they added an API breaking change
18:40:37 <donuts> D:
18:40:53 <PieroV> Like, there was an optional flag. They've introduced the complementary flag and forced to use one of them in 12+
18:40:58 <richard> but we've a new fix that's been verified to work on both old and new devices
18:41:14 <donuts> oh great, phew
18:41:28 <richard> (we have an issue tracking this for next year w/ a reminder so it does't sneak up on us)
18:41:43 <PieroV> Since many users are reporting the problem, we could ask them to verify for a couple of days, and then push on the Play Store, if somehow possible
18:42:30 <richard> i hadn't seeen any issues related to this pop up in gitlab
18:42:33 <richard> apart from the one we opened
18:42:42 <PieroV> I think there were some in #tor?
18:42:56 <PieroV> But not sure
18:43:00 <donuts> I'll get the lowdown from user support tomorrow
18:43:14 <richard> ah ok
18:43:21 <PieroV> donuts: fantastic
18:43:28 <donuts> how much breakage is it resulting in?
18:43:30 <PieroV> Anyway we have the solution, just need to build
18:43:30 <richard> yes please donuts
18:43:36 <PieroV> donuts: crash at bootstrap D:
18:43:40 <donuts> ooft
18:43:40 <richard> crash on launch i'm p sure
18:43:45 <richard> bootstrap/launch
18:43:51 <PieroV> At bootstrap, not at lunch
18:44:13 <PieroV> Personally I got the same problem also on my phone, that's how I've discovered... And bad feelings from March/April started to arise :(
18:44:19 <donuts> i could crash some lunch rn
18:44:35 <richard> #same
18:45:04 <PieroV> richard: same of what Donuts wrote or of what I wrote? :D
18:45:15 <richard> both tbh
18:45:18 <donuts> okay godspeed with 11.5.10, I'll let user support know that fix is coming in the next couple of days?
18:45:36 <richard> hopefully tomorrow evening ish
18:45:43 <donuts> great
18:45:51 <PieroV> If your meeting is in the afternoon UTC maybe the fix will be already available B)
18:46:12 <donuts> yes, 1600 :D
18:46:50 <richard> hopefully we'll have buids by then :p
18:47:09 <donuts> I think that's us out of discussions
18:47:15 <donuts> unless there's anything we missed about 12.0a5
18:47:20 <PieroV> Not sure if we wanted to say something about 12.0a5
18:47:32 <richard> it's gonna be great
18:47:35 <PieroV> We've reproducibility problems that we cannot reproduce
18:47:42 <richard> best release of all times
18:47:47 <donuts> lol
18:47:50 <richard> I *am* sightly sus of localization changes
18:47:55 <PieroV> But apart from this it'll be great
18:48:11 <donuts> yeah, I'd feel a lot better if we can get tor-internal to mass test it
18:48:15 <donuts> the l10n changes I mean
18:48:30 <gaba> ^ yes
18:48:40 <donuts> and if there are any major issues we can pull the plug and go with plan B
18:48:40 <gaba> Can we send a mail to tor-project@ about it?
18:49:24 <richard> I can send the usual announce email to tor-project too
18:49:29 <richard> with a plea for testing
18:49:44 <richard> or do we want somethign less formulaic?
18:49:53 <donuts> thanks richard :D
18:50:00 <gaba> maybe something in the subject to call for attenition? :)
18:50:02 <gaba> thanks richard
18:50:11 <emmapeel> i am a bit scared about l10n as well
18:50:20 <gaba> richard: what is your worry about localization changes?
18:50:29 <emmapeel> but we were testing the multilocale and  'it worked'(tm)
18:50:33 <PieroV> the thing that can go worse of all isn't easy to test though
18:50:38 <PieroV> And it's the updater
18:50:43 <emmapeel> i dont know, changes scare me, for releases
18:50:52 <gaba> what could go wrong?
18:50:55 <richard> only that my unreproducibe reproducibiity problems are all langauge adjacent
18:51:05 <PieroV> The updater might break for $random_reason
18:51:19 <PieroV> But it worked when we switched from single locale to multi locale in the alpha
18:51:21 <richard> and there's that too, but that's what 12.0a5 is meant to test
18:51:26 <donuts> gaba: tor browser changes language?
18:51:28 <richard> it's true it did
18:51:32 <emmapeel> also, more enthropy with the translations. one bad translation can break all languages now, before it was only its language :D
18:51:43 <PieroV> emmapeel: what do you mean?
18:52:08 <emmapeel> i mean, now with all translations being on the same package
18:52:23 <gaba> i want to understand what could go wrong and what is the plan if it goes wrong :)
18:52:39 <donuts> I think best case scenario our migration strategy works, and most users don't notice a change – but worst case scenario we get a bunch of users asking why their TB just changed to English or something
18:52:52 <gaba> ok
18:53:02 <richard> or the update fails entirely somehow
18:53:08 <richard> but there's not really a reason to think that would happen
18:53:12 <donuts> well, yes, that is indeed worse
18:53:17 <richard> given our experience with the previous alpha ugprade
18:53:39 <richard> the big unknown currently is macOS x86_64 -> universal migration
18:53:54 <richard> but there isn't anything fundamentally different about that process
18:54:18 <richard> it's just somewhat scary :p
18:54:20 <donuts> I can make sure I have the last alpha installed on both my machines and update both to universal
18:54:36 <donuts> i'll scream if something goes wrong
18:54:43 <richard> please do :p
18:55:03 <meskio> mmm, this is another thing I need to handle in gettor, but should be easy
18:55:17 <PieroV> donuts: I'm sure I'll hear you across the ocean :D
18:55:21 * gaba going into other meeting
18:55:26 <donuts> oh of course, sorry meskio
18:55:27 <PieroV> bye gaba
18:55:33 <donuts> thanks gaba!
18:55:44 <donuts> we're getting close to the hour, shall I close the meeting?
18:55:50 <richard> yeah wfm
18:55:54 <PieroV> yes, I don't have anything else to add
18:56:00 <donuts> thanks everyone :D
18:56:03 <PieroV> Thanks!
18:56:08 <donuts> #endmeeting