18:01:31 <ahf> #startmeeting gitlab migration meeting 18:01:31 <MeetBot> Meeting started Wed Apr 15 18:01:31 2020 UTC. The chair is ahf. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:31 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic. 18:01:40 <ahf> sorry, i ended up in a discussion in another room 18:01:51 <ahf> gaba is missing, she had to run somewhere 18:01:59 <ahf> let me just find the pad and stuff 18:02:31 <ahf> https://pad.riseup.net/p/gitlab-migration-problems is the pad 18:02:52 <ahf> so friday was not my most active day, since i spend it with family in the distance since i incremented my birth counter 18:03:27 <ahf> i did look a bit more at the wiki stuff and stole some parts from tracboat (again), but the list of tickets of course being the big problem with migration there 18:03:36 <ahf> i think last week a lot of progress was made on the irc bot though? 18:04:26 <anarcat> what's the list of tickets prblem sorry? 18:04:32 <anarcat> yes, progress on the bot: it's up and running! :) 18:04:44 <ahf> the list is on the pad 18:04:53 <anarcat> if you do foo/bar#1234 in a zwiebelbot tor channel, it will poke at gitlab.tpo/foo/bar/issues/1234 18:04:57 <anarcat> heh 18:05:10 <anarcat> i guess i just found a bug :p 18:05:29 <ahf> hm, that is neat 18:05:40 <ahf> can it be done to not need foo/bar ? 18:05:45 <ahf> or is it just that format one should use? 18:05:46 <anarcat> #info zwiebelbot has been patched to support gitlab, but some issues remain: private or non-existent issues are not handled 18:06:03 <anarcat> ahf: #foo will work for the legacy tickets, but it can't possibly work for all projects 18:06:28 <ahf> so we can make #xyz work to be like legacy/legacy/xyz ? 18:06:37 <anarcat> yes, eventually 18:06:46 <anarcat> for now that still points at trac, but we'll make the switch after the migration 18:07:06 <ahf> sweet 18:07:08 <ahf> that is really nice 18:07:43 <ahf> ggus: was mentioned by pili to talk about cypherpunks 18:07:47 <ggus> yes 18:07:48 <anarcat> i'm still confused as to what "list of tickets" means 18:07:57 <pili> yes thank you ahf and ggus 18:07:57 <ahf> anarcat: my todo list is on the pad 18:08:03 <ahf> it's not on trac if that is what you are asking 18:08:03 * pili jumps back to her other meeting 18:08:05 <anarcat> ahf: yes, i see the pad 18:08:18 <ahf> there is an "ISSUES NOT RESOLVED" 18:08:28 <anarcat> oh you're saying that the problem with the migration is the remaining not-resolved-issues? 18:08:30 <anarcat> got it 18:08:36 <ahf> yeah, formatting issues sorry 18:08:37 <arma2> anarcat: right, i found that bug earlier because prop#300 18:08:39 <ahf> it is all formatting things 18:08:42 <ggus> next week we will release community portal -community.tpo and we would like to collect feedback (ie bug report) on gitlab. do you htink it's possible to have an account like cypherpunks? 18:08:51 <arma2> (ok there is no gitlab issue 300 yet) 18:09:00 <ahf> ggus: i think there is people who thinks it is a bad idea 18:09:04 <ahf> partially myself including 18:09:15 <ahf> on gitlab we cannot restrict an account very much 18:09:28 <ahf> so for example the first troll could just login and change the pw 18:09:40 <anarcat> arma2: not sure i follow, you're saying the gitlab redirector overlaps with prop patterns? 18:09:42 <ahf> or delete the old comments from the other user 18:10:00 <anarcat> ggus: i understood that we were using RT for that now? 18:10:10 <anarcat> didn't we create queues for that already? or is that a different process? 18:10:21 <anarcat> in any case, i agree with ahf that gitlab is not the right place for now 18:10:29 <ggus> anarcat: different things. we created for training@ 18:10:35 <ahf> ggus: realistically i think it will be too early to experiment with that 18:10:45 <ahf> gitlab recently announced that they are open sourcing more private components 18:10:52 <ahf> so there will be a user support system 18:10:54 <ggus> ahf: ok, do you think it migh make sense to point to trac? 18:10:56 <ahf> where people can submit things anonymously 18:11:08 <ahf> ggus: i think a form that you guys an email might be the best? 18:11:12 <ahf> i don't know much about trac 18:11:15 <ahf> except the api :-/ 18:12:01 <anarcat> ggus: trac could work, but i'd suggest we discuss this outside this meeting and just say "not gitlab" for now :) 18:12:16 <ggus> we discussed three options: a) cypherpunk account on gitlab; b) cypherpunk account on trac.tpo ; c) a feedback survey on survey.tpo 18:12:35 <anarcat> survey.tpo sounds like what you need :) 18:12:41 <anarcat> but in any case it's not a) 18:12:53 <arma2> anarcat: (i will tell you about the bug post-meeting. no need to clutter things now.) 18:12:53 <ggus> antonela: pili: ^ 18:12:58 <anarcat> ahf: can we set an agenda for this meeting? i feel we're a bit all over the place :) 18:13:01 <anarcat> ahf: thanks 18:13:06 <anarcat> er 18:13:07 <anarcat> arma2: thanks 18:14:00 <ahf> anarcat: i think we are at it. i think the next steps now would be to talk about next steps in migration, but i will need gaba there 18:14:07 <ahf> and i don't have much of an update due to friday 18:14:10 <ahf> so i think we can just call it 18:14:37 <antonela> ggus: oki 18:15:04 <ahf> should we call it and aim for same time next week? 18:15:16 <anarcat> ahf: so i think we need to talk about 1) status update (what we're doing now) 2) next steps in migration 3) more specifically, next steps in infra plan 18:15:31 <anarcat> so we'd be done with the stauts update and can talk about the next steps 18:15:50 <ahf> (2) is "continue removing formatting issues" and "testing wiki migration" 18:15:57 <ahf> (3) i am not sure if there is anything 18:15:58 <ahf> ? 18:16:05 <anarcat> i had a question about the pad actually 18:16:07 <ahf> as far as i know the service seems to be running stable and so on right now? 18:16:10 <anarcat> lin94 you say *not possible ASFAIK* 18:16:17 <anarcat> what does that refer to? 18:16:34 <ahf> ah 18:16:37 <anarcat> the image stuff? 18:16:51 <anarcat> oh wait did you just update the status there? :) 18:16:52 <ahf> it is a note saying that i am not sure we can convert the inline Image() tags from trac to something sensible with attachments in GL 18:16:59 <anarcat> right 18:17:01 <ahf> no? 18:17:05 <anarcat> ok 18:17:13 <anarcat> so markdown supports images and we can definitely put images in gitlab comments 18:17:25 <anarcat> so we should be able to translate between those two, no? 18:18:37 <ahf> hm, that one should be updated after friday actually, because now i do the attachments at the same time as i do the ticket 18:18:44 <ahf> like i fetch attachments first, then post ticket to gitlab 18:18:44 <anarcat> awesome 18:18:54 <anarcat> so attachements and images should work... 18:18:58 <anarcat> i have test tickets for those if you like 18:19:10 <ahf> yeah 18:19:13 <ahf> there is in the pad 18:19:16 <anarcat> let me know if you want me to dig some out, not sure if i can find them quickly, but i definitely used that feature 18:19:29 <ahf> #30936 18:19:33 <ahf> is one of them 18:19:44 <anarcat> okay 18:19:58 <anarcat> seems like you got it covered 18:20:03 <anarcat> so i guess we can move on to point 3) 18:20:06 <anarcat> which i would like to talk about :) 18:20:14 <anarcat> the service is running stable now 18:20:16 <anarcat> but we don't have users 18:20:21 <anarcat> and i think it will not scale to many users 18:20:45 <ahf> i have only hosted small (few users) gitlabs before, so i have not much clue on that 18:20:48 <anarcat> this is gitlab right now 18:20:49 <anarcat> Instance Hypervisor OS Primary_node Status Memory 18:20:54 <anarcat> gitlab-02.torproject.org kvm debootstrap+buster fsn-node-03.torproject.org running 12.0G 18:20:55 <anarcat> gah 18:21:23 <anarcat> Instance ConfigVCPUs ConfigMaxMem Disk_sizes 18:21:23 <anarcat> gitlab-02.torproject.org 2 12.0G 30720,2048,20480 18:21:45 <ahf> it's memory, disk, swap? 18:21:57 <anarcat> it has as much memory as cupani, vineale and troodi (git-rw, gitweb and trac) together, but only two CPUs (vs 10 total for the other three) 18:22:21 <anarcat> Disk_sizes is basically / (root fs), swap, /srv 18:22:34 <ahf> /srv contains the repos? 18:22:39 <anarcat> i am guessing yeah 18:22:51 <anarcat> this is the legacy infra: 18:22:52 <anarcat> Instance ConfigVCPUs ConfigMaxMem Disk_sizes 18:22:52 <anarcat> cupani.torproject.org 2 2.0G 20480,4096,40960 18:22:52 <anarcat> troodi.torproject.org 4 4.0G 20480,8192,51200 18:22:52 <anarcat> vineale.torproject.org 4 8.0G 20480,4096,40960 18:23:13 <anarcat> cupani is: 18:23:14 <anarcat> Filesystem Size Used Avail Use% Mounted on 18:23:14 <anarcat> /dev/mapper/vg_cupani-srv 39G 31G 5.9G 84% /srv 18:23:21 <ahf> and your concern here is that it is now using a lot or it is now using a lot and in the future it might need even more? 18:23:25 <anarcat> so we already have a problem there that /srv is not big enough to have a copy of cupani 18:23:29 <anarcat> so we'll need to scale that up 18:23:42 <anarcat> my concern is that we haven't looked at this problem at all 18:23:50 <ahf> is all of that git repos? 18:23:57 <ahf> ah, the browser tree is of course yuge 18:24:00 <anarcat> yep 18:24:03 <ahf> makes sense 18:24:22 <anarcat> i'd be surprised, actually, if gitlab stays at 30GB 18:24:31 <anarcat> i expect it to explode in disk usage over time 18:24:41 <ahf> yeah because people will be much more just creating repos there 18:24:44 <ahf> and throw stuff into it 18:24:49 <ahf> today there is a bit of a process to get a repo 18:24:51 <anarcat> especially if we start CI, artifacts storage and containers registries 18:24:56 <anarcat> yep 18:25:03 <anarcat> but yeah, just git will eat a lot of I/O 18:25:19 <anarcat> both in terms of disk usage but also in terms of actual I/O in the cluster 18:25:23 <anarcat> that's one thing i'm worried about 18:25:26 <anarcat> i'm also worried about CPU usage 18:25:54 <anarcat> but the point is more that this is all an analysis that Someone(tm) needs to take care of 18:26:04 <anarcat> so far i've been kind of assuming that Someone Else(tm) would take care of it 18:26:21 <anarcat> but right now it certainly feels that if i don't do it, no one will and everyone kind of assumes we'll just wing it :p 18:26:32 <ahf> hm 18:26:42 <ahf> CI is not gonna be on that one for sure 18:26:52 <ahf> artifact storage and container storage i don't even have on my mind yet 18:27:09 <ahf> the CI is a bit of a security nightmare in my world so i want that off everything else 18:27:29 <anarcat> right 18:27:30 <ahf> i think you are right to think the someone else thing 18:27:46 <ahf> i think that is fine right now, you are already contributing a lot in these conversations 18:28:11 <anarcat> okay :) 18:28:20 <anarcat> well i just want to add that to our launch checklist i guess 18:28:22 <anarcat> we need to 18:28:24 <ahf> so 18:28:31 <ahf> you know we have the gift cards to digital ocean? 18:28:37 <ahf> or not gift cards, but a sponsorship 18:28:44 <ahf> that i haven't sorted out, but that dgoulet got us setup with 18:28:47 <anarcat> 1. have someone talk with other gitlab providers (riseup, debian, gnome) and see what kind of traffic they're seeing and hardware they're using 18:29:00 <ahf> i was hopeing to use that for CI, but since we don't have CI right now other than jenkins, CI is not on my priority list right now 18:29:02 <ahf> but is on nice to have 18:29:11 <anarcat> 2. plan extra hardware and related finances to make sure we can handle the load after launch 18:29:16 <anarcat> 3. deploy that 18:29:34 <anarcat> 2 and 3, by the way, might involve splitting the service into multiple servers (say run gitaly elsewhere) 18:29:42 <anarcat> and yes 18:29:47 <ahf> is micah the riseup gitlab person? 18:29:58 <anarcat> i want to make it really crystal clear to everyone that we don't do CI just yet, that's another launch 18:30:16 <ahf> yes! 18:30:17 <ahf> agreed 18:30:22 <anarcat> as much as i'd like to retire jenkins, i don't think we should do it as we retire trac :) 18:30:29 <anarcat> one retirement at a time :) 18:30:40 <ahf> yep 18:30:41 <ahf> agreed 18:30:41 <anarcat> i am not sure about gitlab 18:30:43 <ahf> we are in on that 18:30:45 <anarcat> but i can ask them 18:30:53 <anarcat> point is: who's bottomlining that side? 18:31:20 <ahf> bottomlining? 18:31:53 <anarcat> uh 18:31:59 <anarcat> "making sure this happens"? 18:32:20 <anarcat> a "bottomliner" makes sure things are taken care of, even if they don't necessarily do the work themselves 18:32:27 <anarcat> coordinating i guess is a good approximation 18:32:37 <ahf> are you up for talking with them about this? 18:32:43 <ahf> i think you have a better overview of the concerns there 18:32:49 <ahf> hiro knows some scaling issues from oniongit i guess 18:33:16 <anarcat> i guess i can bounce that problem around with hiro and see which lap it falls on at the end :p 18:33:33 <anarcat> i am also wondering if we have any sort of budget for this launch in terms of hardware provisionning 18:33:49 <anarcat> i would love to be able to throw hardware at this problem if we need to 18:34:00 <anarcat> as opposed to spending days of staff work trying to fine-tune gitlab's performance 18:34:07 <ahf> so it is good you are a bit more balanced on this because i'm a bit of a yes hat person in all of this and it might be there are some performance issues that is worth thinking into it 18:34:11 <ahf> that i am not seeing 18:34:20 <ahf> i have no clue on budget here 18:34:25 <anarcat> that's because you're not dealing with trac crap right now :) 18:34:31 <anarcat> you would definitely be worried if you would 18:34:33 <ahf> i think once we are migrated, we can get rid of some of the resources from the other services 18:34:40 <ahf> yep! 18:34:41 <ahf> for sure 18:36:23 <ahf> crawling is an issue with trac, no? 18:36:26 <anarcat> yes 18:36:29 <anarcat> it will be worse with gitlab 18:36:31 <anarcat> oh 18:36:33 <ahf> i would be 100% sure that gitlab probably have that issue too 18:36:34 <anarcat> and monitoring 18:36:50 <ahf> if i google torproject stuff now i already see a lot of things on gitlab.torproject.org 18:36:50 <anarcat> we need monitoring, particularly hooking gitlab into prometheus and grafana 18:36:52 <anarcat> we have nothing right now 18:36:53 <ahf> like individual commits and such 18:37:07 <ahf> we should probably disable robots.txt because every call to many of the sites are probably expensive git operations 18:37:36 <anarcat> i'll open tickets 18:37:54 <ahf> awesome 18:38:55 <anarcat> https://trac.torproject.org/projects/tor/ticket/33921 18:39:40 <yanmaani> Can't you just stick it behind caching? Or are these idiosyncratic operations? 18:39:49 <ahf> awesome 18:40:04 <ahf> yanmaani: maybe putting varnish in front could be a solution, but i don't know 18:40:11 <ahf> maybe just blocking crawling in the beginning is fine 18:40:24 <anarcat> we already have nginx in front of it, and yes we could setup caching there 18:40:26 <yanmaani> Do Things That Don't Scale™ 18:40:40 <anarcat> but as i said, that falls in the "let's spend a week tuning gitlab" solution :p 18:40:49 <yanmaani> (not sarcasm) 18:40:49 <anarcat> as opposed to "throw hardware at the problem" solution 18:41:00 <anarcat> https://trac.torproject.org/projects/tor/ticket/33922 18:41:08 <anarcat> alright, we can move the discussion to async 18:41:14 <anarcat> i'm happy with the meeting, and would move on :) 18:42:02 <ahf> oki 18:42:05 <ahf> let's close it down folks 18:42:10 <ahf> meeting in 1 week at same time 18:42:21 <anarcat> thanks ahf 18:42:23 <ahf> thanks folks, i will talk with gaba 18:42:25 <ahf> #endmeeting