14:59:45 <richard> #startmeeting Tor Browser Weekly Meeting 2022-04-04
14:59:45 <MeetBot> Meeting started Mon Apr  4 14:59:45 2022 UTC.  The chair is richard. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:59:45 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:59:49 <richard> hello everyone!
14:59:54 <richard> pad: https://pad.riseup.net/p/tor-tbb-keep
15:00:10 <boklm> hi!
15:00:40 <richard> as usual, please fill in your todos and todones, add any discussion points, and please update your boards to better reflect reality
15:00:59 <aguestuser> o/
15:01:04 <richard> o/
15:03:01 <donuts> o/
15:04:51 <richard> ok first up is easy hopefully
15:04:53 <richard> sysrqb
15:05:10 <richard> I need delete permissions on our git because we need to delete a branch
15:05:11 <richard> for reasons?
15:05:22 <richard> aguestuser: (why do we need to delete geckoview-99.0b3-11.50-1 ?)
15:05:34 <PieroV> richard: because it's 2.5GB of stuff
15:05:57 <PieroV> that we don't really need. It was introduced by some bug on a shell script to automatically find the commit to perform the rebase on
15:06:18 <richard> ahh I see
15:06:26 <aguestuser> richard: there was a whole discussion about how i accidentally created the branch based off of a cinnabar clone of the upstream mercurial branch
15:06:27 <PieroV> it copied some branch from the hg-git bridge, instead of getting it directly from gecko-dev
15:06:27 <boklm> it's a separate commit history for all firefox commits
15:06:47 <richard> well i'll keep poking sysrqb until i get delete access
15:06:51 <richard> ok PieroV
15:06:57 <PieroV> richard: you may ask tpa directly
15:07:05 <PieroV> I think they're the ones who set permissions
15:07:07 <aguestuser> (there is a script to use that to derrive the gecko-dev commit hash from the hg tag, but the script didn't work and i picked up after the weekend and just grabbed the commit from the cinnabar clone)
15:07:17 <boklm> it's also possible to open a ticket and ask gitolite admins to delete the branch
15:07:27 <aguestuser> (not really remembering enough context to recall that by doing so i was introducing an entirely different commit tree)
15:07:32 <richard> alright I'll do all those things
15:07:57 <richard> aguestuser/PieroV so is there any reason to not apply the alpha reording to alpha geckoview?
15:08:19 <PieroV> richard: time/more important stuff to do :)
15:08:54 <richard> yeah that's fair
15:09:25 <PieroV> but I should done soon with the bridges/about:preferences#connection
15:09:29 <richard> I would say it's p low priority atm though may be higher priority relatively soon vOv
15:10:00 <PieroV> richard: I reminded of it because we're getting the crashes on geckoview
15:10:17 <PieroV> (or, to be more precise, loading problems)
15:10:23 <richard> ah, so may be helpful for debugging you think?
15:10:35 <PieroV> and if vanilla FF works, I don't know if we can bisect geckoview
15:10:46 <PieroV> but it would be useful in case
15:11:08 <PieroV> in any case, that
15:11:10 <richard> so ignoring the patch reording
15:11:16 <richard> in the current alpha branch
15:11:25 <PieroV> that issue seems to be painful
15:11:33 <PieroV> okay
15:11:43 <richard> what's the difference in applied patches between geckoview alpha and tor-browser alpha
15:12:01 <PieroV> geckoview dropped some patches
15:12:27 <PieroV> and I think that some modifications were done only to tor-browser. That's why I was proposing to "merge" the patch sets on 102
15:12:47 <PieroV> But I think this has low priority wrt aguestuser's point, I think we could proceed with his first
15:12:58 <richard> yeah fair enough
15:13:18 <richard> i would say, sounds like a plan, but only after we get the rest of the android world under control
15:13:23 <richard> and speaking of which
15:13:28 <richard> how are things aguestuser?
15:13:28 <aguestuser> ayo
15:14:03 <aguestuser> getting toolchain working was hard but we thought we got it, then 3 runtime halting errors causing crash
15:14:13 <aguestuser> all resolved except a mystery "cannot find mozglue" bug
15:14:45 <aguestuser> there is a painfully long convo about it in tor-browser-build#40446
15:14:53 <PieroV> aguestuser: could it be related to us using old SDKs?
15:15:05 <PieroV> (Android SDKs, I mean)
15:15:20 <aguestuser> but useful input to help me keep moving would be: (1) any institutional memory about mozglue (how we build it)
15:15:37 <aguestuser> (2) any isntitutional memory about how/why we pick the flags that go into `.mozconfigs`
15:15:58 <aguestuser> currently trying out PieroV's idea to try using our build env to build vanilla firefox
15:16:30 <aguestuser> so we can determine whether mozglue crash is due to probelm in (a) our build tooling, (b) source code changes
15:16:46 <aguestuser> but that is failing on flags in mozconfig being off
15:16:50 <aguestuser> having dug into mozconfig
15:16:52 <PieroV> aguestuser: (2): we have some custom flags, that I've recently touched
15:17:01 <aguestuser> i see that there's actually guite a big surface area for bugs there
15:17:28 <richard> I have no idea re mozglue, that's always just sort of happened in tor-browser/firefox builds
15:17:28 <PieroV> for example the Tor Browser version, update, etc. They're the ones I usually comment to get a working build of Firefox
15:17:39 <aguestuser> and its also unclear to me which flags we introduce, which we borrow from moz, etc.
15:17:42 <Jeremy_Rand_Talos_> aguestuser, when I did the ARM/POWER mozconfigs, I routinely grabbed mozconfigs from Arch Linux ARM and Talospace, both of which are updated every major release.
15:18:14 <aguestuser> this is the set of all possible mozconfig flags used upstream: http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/tpo/applications/tor-browser/-/tree/geckoview-99.0b3-11.5-1/mobile/android/config/mozconfigs
15:18:19 <aguestuser> (there are a lot)
15:18:20 <richard> and re .mozconfig your best bet is most likely the commit history for those files in tor-browser-build
15:18:28 <aguestuser> some of the flags we set are the same as what upstream sets
15:18:30 <aguestuser> some are custom
15:18:31 <richard> for the 'why' q
15:19:07 <aguestuser> i don't know how close it is worth poking into this... but it occurs to me that a lot happens at this layer
15:19:25 <aguestuser> so drift in what flags we set vs. what flags are set upstream could be a source of a bug?
15:19:28 <Jeremy_Rand_Talos_> So basically, if you have a mozconfig issue that's specific to one platform, try to find a distro that builds for that platform and see what they do
15:19:54 <Jeremy_Rand_Talos_> If this is Android, F-Droid might be the distro to check
15:19:54 <PieroV> Jeremy_Rand_Talos_: in this case actually we're speaking of Android
15:20:33 <aguestuser> yes. and we have 5 different mozconfig files for android builds
15:20:40 <aguestuser> 1 for each arch and 1 for the "merge" step
15:20:48 <PieroV> aguestuser: but the 4 platform ones are very similar
15:20:52 <aguestuser> true
15:21:04 <PieroV> I think the only difference is --target
15:21:08 <richard> otoh if we have vanilla ff building in our system, it'll be 'easy' to check if lack of some flags is causing the current issues
15:21:10 <aguestuser> so one generic "android" one which borrows a lot from upstream but introduces other stuff
15:21:42 <Jeremy_Rand_Talos_> PieroV, yeah so I would check what F-Droid did when they bumped to the Firefox version that's breaking for you.
15:22:15 <aguestuser> here is a representative one: http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/tpo/applications/tor-browser-build/-/blob/master/projects/geckoview/mozconfig-android-x86
15:22:24 <Jeremy_Rand_Talos_> Might or might not yield useful insights but it's definitely the approach that worked for me with ARM/POWER
15:23:05 <aguestuser> several i obviously need to comment out if i'm tryig to build vanilla ff
15:23:08 <boklm> maybe looking what changed related to mozglue in upstream geckoview between the old and new release
15:23:10 <aguestuser> (eg anything mentioning tor)
15:23:32 <aguestuser> but then for others there is no "vanilla FF" version of the flag
15:23:44 <aguestuser> b/c they set the flag differently for beta/release/nightly etc
15:23:59 <aguestuser> it's perhaps a red herring (likely a red herring)
15:24:08 <PieroV> we're using beta (lines 25-26 of the mozconfig)
15:24:19 <aguestuser> but i guess i just wnated to raise it to (1) understand how we arrived at picking the flags we picked
15:24:20 <PieroV> so I think you should look the beta ones
15:25:15 <aguestuser> and (2) convince myself that flipping enough of these to get the build to work won't have destroyed the "control" nature of the basleine "control" build i'm trying to generate
15:25:17 <aguestuser> does that make sense?
15:25:19 <PieroV> many flags are the same we set on Firefox
15:25:27 <aguestuser> okay that's reassuring
15:25:34 <aguestuser> (hadn't had a chance to poke at them yet)
15:25:45 <aguestuser> also curious: *where* are these build flags picked up?
15:25:55 <PieroV> projects/firefox
15:26:01 <aguestuser> (which i can probably figure out on my own in tor-browser.git)
15:26:01 <PieroV> we have a bunch of mozconfigs also there
15:26:29 <aguestuser> PieroV: projects/firefox picks up the build flags? (is not what i think you meant?)
15:26:41 <aguestuser> like how does mach interpret our custom build flags?
15:26:45 <PieroV> --enable-optimize / --enable-rust-simd are optimization things
15:27:17 <PieroV> aguestuser: projects/firefox is like geckoview, we have the various mozconfig (one for each platform) that are automatically copied as .mozconfig when building TBB
15:28:22 <aguestuser> PieroV: gotcha. so just comparing mozconfigs between tb-build projects could help me get a sense of the general gist of how we use mozconfigs?
15:28:24 <PieroV> (so yes, the build flags are picked up from projects/firefox is what I meant)
15:28:37 <aguestuser> but ultimately mach picks up the flags, yes?
15:29:10 <aguestuser> like someone at mozilla wrote some code or we patched it to recognize what `-disable-tor-launcher` means?
15:29:46 <aguestuser> (or --with-tor-browser-version etc?)
15:29:48 <PieroV> from what I understood, mach first reads mozconfig / .mozconfig (you can only have one of them), then it reads the command line. In doubt, it uses the one specified in the command line
15:29:49 <boklm> we have some patch to recognize this flag
15:29:55 <richard> tor mozconfig stuff was added by us
15:30:02 <richard> exists in some python in the build somewhere iirc
15:30:02 <aguestuser> aha! where does that live?
15:30:07 <PieroV> they are in the patch that begins with TB3
15:30:13 <aguestuser> ty!
15:30:26 <PieroV> TB3: Tor Browser's official .mozconfigs.
15:30:37 <aguestuser> okay, i think that's enough for me to orient w.r.t. mozconfigs....
15:30:55 <aguestuser> curious to pick up on boklm's suggestion about comparing changes in mozglue between versions
15:31:08 <aguestuser> what are hotspots where such changes might occur
15:31:17 <aguestuser> the error we are seeing is opaque b/c...
15:31:21 <aguestuser> it is an error in the java code
15:31:28 <aguestuser> (GeckoLoader)
15:31:43 <aguestuser> trying to go find a mozglue binary it expects to be able to find
15:31:45 <aguestuser> then not finding it
15:31:57 <aguestuser> so it's almost certainly not an error in the java code
15:32:11 <aguestuser> but the java runtime error is all we have to go ff
15:32:21 <richard> ok
15:32:25 <boklm> I think I would start with checking for commit messages mentioning mozglue, and see if there's anything interesting there
15:32:55 <aguestuser> this is the error in q: http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/tpo/applications/tor-browser-build/-/issues/40446#note_2792777
15:33:05 <boklm> or a diff between the two versions, looking for mozglue stuff there
15:33:24 <Jeremy_Rand_Talos_> aguestuser, maybe a dumb idea but if you can run something like strace on Android, that might help you find what binary is missing
15:34:01 <PieroV> aguestuser: have you tried decompressing the .aar and see if we have libmozglue.so there? I haven't, yet
15:34:33 <aguestuser> PieroV: nope. that's a good idea! hypo being we built it, but put it in a different location than where gv expects to find it
15:34:47 <aguestuser> is it important / does anyone know what `mozglue` actually does?
15:35:40 <PieroV> aguestuser: fact is that we don't know where geckoview expects it, because I think that it's something tied to how Android loads stuff. That's why I was asking if something changed there, and if Moz is using some of the old SDKs that we're using
15:35:51 <Jeremy_Rand_Talos_> aguestuser, not 100% sure but I believe it's a "miscellaneous important stuff that didn't fit anywhere else" lib.
15:35:55 <boklm> I think we don't do anything to build it ourself, it's built as part of the geckoview build
15:36:05 <aguestuser> (i found this which wasn't terribly illuminating: https://wiki.mozilla.org/Modules/Core#Mozglue)
15:38:50 <PieroV> (btw, libmozglue.so is indeed in the .aar, in the jni directory)
15:38:52 <aguestuser> boklm: hmm... that seems to suggest it's something in the geckoview rebase we might want to pay more attention to? (ie: if we don't touch this in tor-browser-build, it's likely not drift between tor-browser-build and upstream mach that caused the issue? but if the issue is in geckoview build... how could it pertain to source code cahnges?)
15:40:06 <PieroV> wait, I've just discovered a thing: geckoview-beta-omni is only 1.2MB for my build
15:40:59 <aguestuser> was about to type: "at any rate: seems hopefully simple to get a vanilla FF build working as basis of comparision (given above discussion of flags), and looking through commit messages in gv seems like at least a new idea"
15:41:05 <aguestuser> but PieroV that seems interesting!
15:41:36 <PieroV> (btw, libmozglue.so is indeed in the .aar, in the jni directory) <-- this was for an arch build (the first one I've found)
15:41:54 <PieroV> so, I wanted to check that omni had the files for all the architectures
15:42:11 <aguestuser> PieroV: oh! and the omni does not?
15:42:35 <PieroV> No, and it's only 1.2MB, instead of being > 50MB
15:43:16 <PieroV> (I'm speaking of geckoview-7af7603f41ea-32c545.tar.gz, but you may check that you have a similar result on your out/geckoview)
15:43:37 <aguestuser> aha! okay! also a new (and most promising yet) direction to explore!
15:43:54 <richard> :)
15:44:05 <richard> ok a couple things from me i forgot:
15:44:18 <aguestuser> (might have to do w/ recent name change w/ `find <snip> *omni*` to avoid packaging exoplayer thing)
15:44:27 <aguestuser> thanks for hlep all!
15:44:49 <richard> boklm: our builds are all matching for 11.0.10 so we're good to go for signing (if you haven't already started)
15:45:06 <richard> i'm planning on trying out the signing process for the next desktop alpha, but we'll see how it goes
15:45:06 <boklm> I already started this weekend
15:45:14 <richard> boklm: yeah i figured :)
15:45:35 <richard> aguestuser: we had tentatively planned on waiting and having a joint android+dekstop alpha 11.5a10
15:45:51 <richard> i'm guessing that plan no longer makes sense?
15:47:05 <richard> well given all the anroid discussion, i'm going to assume no so I'll prep alpha desktop release this week
15:47:23 <PieroV> richard: the alpha was to be based on v100
15:47:33 <richard> mmhm
15:47:36 <PieroV> But we haven't rebased to it yet (nor we have rebased the alpha to v99)
15:47:52 <richard> right
15:48:23 <richard> 11.5a10 is desktop only and I'll get started on it today/tomorrow depending
15:48:36 <richard> ok does anyone have anything else?
15:48:37 <PieroV> (we also have the release meeting later)
15:48:59 <richard> oh right, Jeremy_Rand_Talos: it's been in there awhile, but thanks for the fix to our fonts project in tor-browser-build :)
15:49:35 <Jeremy_Rand_Talos_> richard, you are welcome!  This fix came about from my ongoing Cirrus experiments.
15:49:42 <richard> if nobody has anything else i'm going to call it here
15:50:15 <aguestuser> richard: okay! (sorry was poking at things PieroV pointed out)
15:50:23 <richard> alright then
15:50:26 <richard> #endmeeting