17:59:07 <richard> #startmeeting Tor Browser Release Meeting 2024-12-11
17:59:07 <MeetBot> Meeting started Mon Dec 11 17:59:07 2023 UTC.  The chair is richard. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:59:07 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:59:15 <richard> pad: https://pad.riseup.net/p/tor-browser-release-meeting-keep
17:59:43 <donuts> /
18:00:19 <richard> ok PieroV looks like you have some discussion points :D
18:00:26 <PieroV> Yes
18:00:48 <PieroV> So, ruihildt[m] told us users complained about MB arriving one week later than TBB
18:01:03 <PieroV> And the reason might be the QA
18:01:15 <PieroV> So, I thought on how to provide QA builds sooner
18:01:41 <PieroV> And I think we can build MB from build1 tags
18:01:50 <PieroV> I.e., not wait for Android backports
18:02:08 <PieroV> Since usually they're Android-only, even though in the past we backported to ESR before Moz
18:02:16 * ruihildt[m] nodes his head (QA and even more QA around weekends)
18:02:43 <ma1> It does make sense. Even if we needed to backport something (like a PB / RFP bug which wasn't deemed critical enough for esr) it's unlikely to affect QA
18:02:45 <PieroV> (i.e., we backported a patch and Moz backported it just for the ESR that came after the one in which we backported)
18:02:58 <PieroV> Also what ma1 says
18:03:22 <PieroV> And since we don't need to wait for Android backports, we could slightly change our process for release builds
18:03:46 <richard> this seems reasonable
18:03:51 <PieroV> We more or less know a lot earlier which backports of our changes we want in a release
18:03:58 <PieroV> We don't have to wait for the rebase
18:04:08 <PieroV> But we can rebase after having already cherry-picked
18:04:20 <PieroV> Therefore, also changelogs are predictable enough
18:04:38 <PieroV> So, we could prepare the release before rebasing
18:04:44 <richard> I *do* like the idea of rebasing+tagging being the last step before building
18:04:58 <PieroV> E.g., we're expecting tags (they could come in a few hours, or tomorrow)
18:05:07 <PieroV> We could've prepared the release already
18:05:33 <richard> however long term we're going to run into this same problem again when mullvad-browser for android is a thing
18:05:36 <PieroV> Not too early, but ideally on Monday of the week before the release
18:05:47 <PieroV> Well, it depends if we switch to RR
18:06:03 <PieroV> Switching back to RR for Android eventually could be a wise choice
18:06:30 <PieroV> Or we could build twice
18:06:50 <PieroV> Hoping we're blessed and we don't have to backport
18:06:56 <PieroV> Then, if we have, we tag and build build2
18:07:21 <PieroV> In the meantime, we could wire Mullvad's CI
18:07:33 <PieroV> So that they don't have to wait for us for unsigned builds for testing
18:08:32 <ruihildt[m]> How do we know we're testing the right build? Is there a risk we test builds that aren't reproducible?
18:09:21 <richard> so there is risk that the build won't be reproducible
18:09:23 <PieroV> Repro problems usually are rare on the release channels
18:09:33 <PieroV> * channel
18:10:08 <PieroV> But we could also have a CI for release builds, FWIW
18:10:12 <richard> but historically i don't think i've ever seen a reproducibility bug which affected application behaviour
18:10:32 <PieroV> And hashes locations are quite predictable
18:10:37 <ruihildt[m]> PieroV: I like that idea.
18:10:48 <PieroV> I regularly test on my own if the build on tb-build-02/03 matches mine
18:10:54 <PieroV> I even wrote a quick script to do it
18:11:33 <PieroV> We could polish so that you can run it to verify if the builds produced in your ci match
18:11:50 <ruihildt[m]> If we could automate building browser on both infra, when it needs to be done, it seem sideal
18:13:50 <richard> we spoke w/ jag about this briefly in gothenburg; iirc if your infra built on new tor-browser-build tags then that will give you exactly what you need to test
18:14:08 <richard> in general, speaking of releasing and coordination
18:14:35 <richard> how can we improve the release pipeline on your end
18:15:01 <richard> i imagine it kind of sucks for your QA team to receive 'you've a build to test' emails unpredictably
18:16:15 <ruihildt[m]> To have a release date ahead of time would be nice. so we can make sure, our QA is available.
18:17:49 <ruihildt[m]> Lately we've had multiple small delays due to the template email not being followed, or because the process was somehow not documented? Or because tags were missing on our Github.
18:18:14 <ruihildt[m]> I'm not sure what there is to improve there.
18:18:29 <ma1> Given the process above, Thursday after the tags is the worst-case date for a test build, right?
18:18:40 <ma1> RC I mean
18:18:50 <ruihildt[m]> Or also, because we needed signed builds, but the right people were not   available.
18:18:56 <ruihildt[m]> to sign
18:19:23 <ma1> Ah right, because the MB rc needs to be signed (for MacOS testing)
18:19:54 <ruihildt[m]> If builds arrive without warning on Thursday, someone might or might not be able to test. If it collides with a Mullvad VPN App QA session, it might be an issue.
18:19:54 <richard> and release signing (currently) can only be done by myself and boklm
18:20:30 <richard> so stable releases are fairly reliably in-line with the firefox release calendar ( https://whattrainisitnow.com/calendar/ )
18:20:48 <ruihildt[m]> This is why I'm talking about giving us a date.
18:21:10 <ruihildt[m]> I know there's the caendar, but I haven't been able to estimate the MB release from it.
18:21:43 <PieroV> Yep, we're not very predictable in terms of starting a build
18:21:50 <ma1> So, how would "the Thursday before Firefox release (which is always on Tuesday)" work?
18:22:21 <PieroV> But so far we've been quite reliable on having the rebase ready on the Tuesday before the release week
18:22:36 <PieroV> So, doing the rebase as a last step should help for predictability also on builds
18:22:43 <ruihildt[m]> I'll needto ask the QA team about it.
18:23:40 <richard> agreed
18:23:44 <richard> re doing rebase last
18:25:27 <ruihildt[m]> Regarding the automation part of the build, what do you need from me?
18:25:27 <ruihildt[m]> I think JB mentioned we couldn't leverage our existing Drone CI instance, we might need to create another one.
18:26:02 <PieroV> Do you want us to setup a webhook to signal you?
18:26:21 <PieroV> I think it's the most we can do for active notifications, otherwise you'll have to poll in some way
18:26:50 <ruihildt[m]> I'm not the right person to talk about this I'm afraid. I'll talk again to JB about it tomorrow.
18:27:51 <PieroV> Yes, sorry, I thought you meant also needs from you as you relaying the request :)
18:27:55 <richard> we can easily add another email to the 'Email on push" webhook
18:27:55 <ruihildt[m]> And regarding the signing, since we know optimistically that a build is going to be released at a set date, can we make sure someone is available to do it?
18:28:47 <PieroV> Yes, also emails, and actually a few other integrations, even though I think webhook are the easiest to parse ^_^
18:29:13 <ruihildt[m]> I think I can use a webhook for slack.
18:29:16 <ruihildt[m]> also
18:29:38 <ruihildt[m]> if we can do without the email through my inbox stuff, that woud be good as well
18:30:08 <richard> oh yeah there's Slack notifacation integrations builtin to gitlab apparently
18:30:34 <PieroV> ruihildt[m]: would automation be completely automatic, or would JB or someone else still need to launch the build?
18:31:09 <ruihildt[m]> I don't know. :) I would prefer to launch it automatically obviously.
18:33:51 <richard> ok, i've added a set of TODOs on our end in the pad
18:34:57 <ruihildt[m]> I don't remember how we landed there, but would it make sense to revisit pushing the tags+branch automatically?
18:35:04 <richard> ruihidlt[m]: the main things we need from y'all is basically stuffs dependant on JB; how to notify you of new build tags to build and all the automation required to get that going
18:35:04 <ruihildt[m]> to Github
18:36:18 <ruihildt[m]> And QA for the Thursday release. ma1 Do you have a more precise time of release (morning or afternoon UTC)?
18:37:01 <richard> so the inefficiency there is bad release documentation; pushing branch+tags to github should happen as part of the rebase
18:37:06 <richard> rather than the *last* step in the whole process
18:37:20 <PieroV> I think it's also a problem of missing permissions
18:37:27 <richard> that too
18:37:34 <PieroV> I think I can't push to GitHub, whereas I can push everywhere on tpo/applications
18:37:53 <PieroV> And often I'm rebasing, so I'm also pushing/merging
18:37:56 <richard> we should probably add anyone who can push to tor-browser gitlab projects to the mullvad-browser github repo as well
18:37:57 <PieroV> Esp. if ma1 reviews
18:38:10 <ruihildt[m]> Is there a way to automate this?
18:38:33 <richard> not really, the tags are pgp signed by the devs doing the rebase
18:39:00 <ruihildt[m]> I meant once the tags are added to the gitlab, automatically push them to github
18:39:00 <PieroV> I think GitHub can set to mirror
18:39:04 <ma1> ruihildt[m], re time: since the last step is signing, it mainly depends on the timezone of the available person(s)
18:39:08 <PieroV> And maybe also GitLab can push
18:39:42 <richard> hmmm true
18:40:12 <ruihildt[m]> ma1: Because if signing happens during the afternoon, it the same thing as saying the release is happening on Friday from our POV.
18:40:45 <ruihildt[m]> People tend to go home from 3/4PM CET
18:41:55 <ruihildt[m]> To have any QA done on Thursday, a build should be ready by 1PM CET
18:42:22 <ma1> Soo, let's say we have fixed date to start relprep on Monday (Firefox tags or not), a fixed date to finish the rebase and start building on Tuesday night, and a fixed date to sign on Wednesday ~18 UTC (after the All Hands): is this too tight?
18:42:43 <PieroV> ruihildt[m]: would it be possible to test macOS a little bit later in case?
18:43:08 <PieroV> Does the build need to be signed by us? (e.g., a check on the signing details themselves)
18:43:22 <PieroV> Or would self-signed builds also be okay?
18:44:15 <ruihildt[m]> So once QA is done, then JB needs to manually upload the files on our cdn, and the signing check happens at the same time.
18:45:02 <ruihildt[m]> I'll come back to you as well on the self-signed builds.
18:45:26 <ruihildt[m]> Because note everybody has access to all test machines, QA is spread between multiple people.
18:45:47 <richard> self-signed for macOS would let QA get a head-start
18:46:16 <ruihildt[m]> self signed for Macos still needs to execute a script and whatnot, right?
18:46:38 <PieroV> Yes, but if it can help we can try to improve it
18:47:01 <ruihildt[m]> Ok, yes definitely.
18:47:15 * ma1 wonders whether Mullvad has their own Apple Developer cert and can sign their MacOS RC on their side
18:47:19 <ruihildt[m]> Having to go in some specific subfolder and so on is still a pain.
18:47:25 <PieroV> ma1: I was about to say that
18:47:36 <ruihildt[m]> Good question ma1, you get 5 internet points.
18:47:58 <ruihildt[m]> I'll ask.
18:48:13 <ma1> I assume the answer would be likely yes: https://mullvad.net/en/download/vpn/macos
18:48:13 <PieroV> (timecheck: 10 minutes and we have another quick point)
18:48:45 <ruihildt[m]> But it's another server altogether, so I'm not sure what would need to be done in practice.
18:49:00 <richard> and signing certs+notarisation tokens are pretty easy to acquire from the dev portal
18:49:01 <ruihildt[m]> another build server and environment
18:49:28 <ruihildt[m]> I will definitely ask
18:50:09 <richard> ok next point
18:50:30 <PieroV> What's the plan for 13.5a3?
18:50:35 <PieroV> Things to consider:
18:50:43 <PieroV> 1. the opt-in for the new backend
18:50:56 <PieroV> 2. we have a break, and if 1 goes badly we might not be very present to fix it
18:51:19 <PieroV> Also, 13.5a3 brings the fix to the wayland crash on alpha
18:52:36 <richard> so i don't think i'm too worried about breaking alpha users
18:52:55 <richard> re the new backend on android
18:54:04 <PieroV> Should we wait to get the opt-in merged?
18:54:18 <PieroV> I don't think it'll take a long time
18:55:27 <richard> i'm not sure
18:55:44 <PieroV> It'll help me play with it during the break :)
18:55:47 <richard> i think we should get it in asap for our own development purposes
18:55:49 <PieroV> I like dogfooding
18:55:59 <richard> perhaps we can gate the toggle on a pref
18:56:00 <PieroV> But I hate doing it on nightly on Android
18:56:05 <richard> yeah exactly^
18:56:09 <ruihildt[m]> (woof woof)
18:56:10 <PieroV> The backend is gated
18:56:16 <PieroV> Already with a pref
18:56:34 <richard> i meant gate the UX allowing the toggle on a pref
18:56:53 <richard> but honestly i'm ok with users playing around with it
18:56:57 <PieroV> That is less easy
18:56:59 <richard> it is the alpha channela fter all
18:57:09 <PieroV> Okay, so business as usual?
18:57:13 <richard> yeah I think so
18:57:16 <PieroV> I.e., best effort to get it before the break?
18:57:41 <richard> yeah if it comes late i'll sign over the weekend
18:57:58 <PieroV> ack
18:58:00 <richard> but let's get mullvad-browser alpha out the door asap next week so QA isn't sad over the holidays
18:58:13 <PieroV> also, if we break Android users are still able to get updates
18:58:17 <PieroV> It isn't as tragic as desktop
18:58:23 <PieroV> That once broken we can't even serve updates
18:58:23 <richard> truuue
18:58:38 <ma1> praise be Google...???
18:59:16 <richard> lol
18:59:17 <richard> ok
18:59:20 <richard> we're a minute over
18:59:26 <richard> sorry whoever comes after if anyone
18:59:30 <richard> later y'all o/
18:59:33 <richard> #endmeeting