18:01:31 <ahf> #startmeeting gitlab migration meeting
18:01:31 <MeetBot> Meeting started Wed Apr 15 18:01:31 2020 UTC.  The chair is ahf. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:01:31 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
18:01:40 <ahf> sorry, i ended up in a discussion in another room
18:01:51 <ahf> gaba is missing, she had to run somewhere
18:01:59 <ahf> let me just find the pad and stuff
18:02:31 <ahf> https://pad.riseup.net/p/gitlab-migration-problems is the pad
18:02:52 <ahf> so friday was not my most active day, since i spend it with family in the distance since i incremented my birth counter
18:03:27 <ahf> i did look a bit more at the wiki stuff and stole some parts from tracboat (again), but the list of tickets of course being the big problem with migration there
18:03:36 <ahf> i think last week a lot of progress was made on the irc bot though?
18:04:26 <anarcat> what's the list of tickets prblem sorry?
18:04:32 <anarcat> yes, progress on the bot: it's up and running! :)
18:04:44 <ahf> the list is on the pad
18:04:53 <anarcat> if you do foo/bar#1234 in a zwiebelbot tor channel, it will poke at gitlab.tpo/foo/bar/issues/1234
18:04:57 <anarcat> heh
18:05:10 <anarcat> i guess i just found a bug :p
18:05:29 <ahf> hm, that is neat
18:05:40 <ahf> can it be done to not need foo/bar ?
18:05:45 <ahf> or is it just that format one should use?
18:05:46 <anarcat> #info zwiebelbot has been patched to support gitlab, but some issues remain: private or non-existent issues are not handled
18:06:03 <anarcat> ahf: #foo will work for the legacy tickets, but it can't possibly work for all projects
18:06:28 <ahf> so we can make #xyz work to be like legacy/legacy/xyz ?
18:06:37 <anarcat> yes, eventually
18:06:46 <anarcat> for now that still points at trac, but we'll make the switch after the migration
18:07:06 <ahf> sweet
18:07:08 <ahf> that is really nice
18:07:43 <ahf> ggus: was mentioned by pili to talk about cypherpunks
18:07:47 <ggus> yes
18:07:48 <anarcat> i'm still confused as to what "list of tickets" means
18:07:57 <pili> yes thank you ahf and ggus
18:07:57 <ahf> anarcat: my todo list is on the pad
18:08:03 <ahf> it's not on trac if that is what you are asking
18:08:03 * pili jumps back to her other meeting
18:08:05 <anarcat> ahf: yes, i see the pad
18:08:18 <ahf> there is an "ISSUES NOT RESOLVED"
18:08:28 <anarcat> oh you're saying that the problem with the migration is the remaining not-resolved-issues?
18:08:30 <anarcat> got it
18:08:36 <ahf> yeah, formatting issues sorry
18:08:37 <arma2> anarcat: right, i found that bug earlier because prop#300
18:08:39 <ahf> it is all formatting things
18:08:42 <ggus> next week we will release community portal -community.tpo and we would like to collect feedback (ie bug report) on gitlab. do you htink it's possible to have an account like cypherpunks?
18:08:51 <arma2> (ok there is no gitlab issue 300 yet)
18:09:00 <ahf> ggus: i think there is people who thinks it is a bad idea
18:09:04 <ahf> partially myself including
18:09:15 <ahf> on gitlab we cannot restrict an account very much
18:09:28 <ahf> so for example the first troll could just login and change the pw
18:09:40 <anarcat> arma2: not sure i follow, you're saying the gitlab redirector overlaps with prop patterns?
18:09:42 <ahf> or delete the old comments from the other user
18:10:00 <anarcat> ggus: i understood that we were using RT for that now?
18:10:10 <anarcat> didn't we create queues for that already? or is that a different process?
18:10:21 <anarcat> in any case, i agree with ahf that gitlab is not the right place for now
18:10:29 <ggus> anarcat: different things. we created for training@
18:10:35 <ahf> ggus: realistically i think it will be too early to experiment with that
18:10:45 <ahf> gitlab recently announced that they are open sourcing more private components
18:10:52 <ahf> so there will be a user support system
18:10:54 <ggus> ahf: ok, do you think it migh make sense to point to trac?
18:10:56 <ahf> where people can submit things anonymously
18:11:08 <ahf> ggus: i think a form that you guys an email might be the best?
18:11:12 <ahf> i don't know much about trac
18:11:15 <ahf> except the api :-/
18:12:01 <anarcat> ggus: trac could work, but i'd suggest we discuss this outside this meeting and just say "not gitlab" for now :)
18:12:16 <ggus> we discussed three options: a) cypherpunk account on gitlab; b) cypherpunk account on trac.tpo ; c) a feedback survey on survey.tpo
18:12:35 <anarcat> survey.tpo sounds like what you need :)
18:12:41 <anarcat> but in any case it's not a)
18:12:53 <arma2> anarcat: (i will tell you about the bug post-meeting. no need to clutter things now.)
18:12:53 <ggus> antonela: pili: ^
18:12:58 <anarcat> ahf: can we set an agenda for this meeting? i feel we're a bit all over the place :)
18:13:01 <anarcat> ahf: thanks
18:13:06 <anarcat> er
18:13:07 <anarcat> arma2: thanks
18:14:00 <ahf> anarcat: i think we are at it. i think the next steps now would be to talk about next steps in migration, but i will need gaba there
18:14:07 <ahf> and i don't have much of an update due to friday
18:14:10 <ahf> so i think we can just call it
18:14:37 <antonela> ggus: oki
18:15:04 <ahf> should we call it and aim for same time next week?
18:15:16 <anarcat> ahf: so i think we need to talk about 1) status update (what we're doing now) 2) next steps in migration 3) more specifically, next steps in infra plan
18:15:31 <anarcat> so we'd be done with the stauts update and can talk about the next steps
18:15:50 <ahf> (2) is "continue removing formatting issues" and "testing wiki migration"
18:15:57 <ahf> (3) i am not sure if there is anything
18:15:58 <ahf> ?
18:16:05 <anarcat> i had a question about the pad actually
18:16:07 <ahf> as far as i know the service seems to be running stable and so on right now?
18:16:10 <anarcat> lin94 you say *not possible ASFAIK*
18:16:17 <anarcat> what does that refer to?
18:16:34 <ahf> ah
18:16:37 <anarcat> the image stuff?
18:16:51 <anarcat> oh wait did you just update the status there? :)
18:16:52 <ahf> it is a note saying that i am not sure we can convert the inline Image() tags from trac to something sensible with attachments in GL
18:16:59 <anarcat> right
18:17:01 <ahf> no?
18:17:05 <anarcat> ok
18:17:13 <anarcat> so markdown supports images and we can definitely put images in gitlab comments
18:17:25 <anarcat> so we should be able to translate between those two, no?
18:18:37 <ahf> hm, that one should be updated after friday actually, because now i do the attachments at the same time as i do the ticket
18:18:44 <ahf> like i fetch attachments first, then post ticket to gitlab
18:18:44 <anarcat> awesome
18:18:54 <anarcat> so attachements and images should work...
18:18:58 <anarcat> i have test tickets for those if you like
18:19:10 <ahf> yeah
18:19:13 <ahf> there is in the pad
18:19:16 <anarcat> let me know if you want me to dig some out, not sure if i can find them quickly, but i definitely used that feature
18:19:29 <ahf> #30936
18:19:33 <ahf> is one of them
18:19:44 <anarcat> okay
18:19:58 <anarcat> seems like you got it covered
18:20:03 <anarcat> so i guess we can move on to point 3)
18:20:06 <anarcat> which i would like to talk about :)
18:20:14 <anarcat> the service is running stable now
18:20:16 <anarcat> but we don't have users
18:20:21 <anarcat> and i think it will not scale to many users
18:20:45 <ahf> i have only hosted small (few users) gitlabs before, so i have not much clue on that
18:20:48 <anarcat> this is gitlab right now
18:20:49 <anarcat> Instance                            Hypervisor OS                  Primary_node               Status     Memory
18:20:54 <anarcat> gitlab-02.torproject.org            kvm        debootstrap+buster  fsn-node-03.torproject.org running     12.0G
18:20:55 <anarcat> gah
18:21:23 <anarcat> Instance                 ConfigVCPUs ConfigMaxMem Disk_sizes
18:21:23 <anarcat> gitlab-02.torproject.org           2        12.0G 30720,2048,20480
18:21:45 <ahf> it's memory, disk, swap?
18:21:57 <anarcat> it has as much memory as cupani, vineale and troodi (git-rw, gitweb and trac) together, but only two CPUs (vs 10 total for the other three)
18:22:21 <anarcat> Disk_sizes is basically / (root fs), swap, /srv
18:22:34 <ahf> /srv contains the repos?
18:22:39 <anarcat> i am guessing yeah
18:22:51 <anarcat> this is the legacy infra:
18:22:52 <anarcat> Instance               ConfigVCPUs ConfigMaxMem Disk_sizes
18:22:52 <anarcat> cupani.torproject.org            2         2.0G 20480,4096,40960
18:22:52 <anarcat> troodi.torproject.org            4         4.0G 20480,8192,51200
18:22:52 <anarcat> vineale.torproject.org           4         8.0G 20480,4096,40960
18:23:13 <anarcat> cupani is:
18:23:14 <anarcat> Filesystem                 Size  Used Avail Use% Mounted on
18:23:14 <anarcat> /dev/mapper/vg_cupani-srv   39G   31G  5.9G  84% /srv
18:23:21 <ahf> and your concern here is that it is now using a lot or it is now using a lot and in the future it might need even more?
18:23:25 <anarcat> so we already have a problem there that /srv is not big enough to have a copy of cupani
18:23:29 <anarcat> so we'll need to scale that up
18:23:42 <anarcat> my concern is that we haven't looked at this problem at all
18:23:50 <ahf> is all of that git repos?
18:23:57 <ahf> ah, the browser tree is of course yuge
18:24:00 <anarcat> yep
18:24:03 <ahf> makes sense
18:24:22 <anarcat> i'd be surprised, actually, if gitlab stays at 30GB
18:24:31 <anarcat> i expect it to explode in disk usage over time
18:24:41 <ahf> yeah because people will be much more just creating repos there
18:24:44 <ahf> and throw stuff into it
18:24:49 <ahf> today there is a bit of a process to get a repo
18:24:51 <anarcat> especially if we start CI, artifacts storage and containers registries
18:24:56 <anarcat> yep
18:25:03 <anarcat> but yeah, just git will eat a lot of I/O
18:25:19 <anarcat> both in terms of disk usage but also in terms of actual I/O in the cluster
18:25:23 <anarcat> that's one thing i'm worried about
18:25:26 <anarcat> i'm also worried about CPU usage
18:25:54 <anarcat> but the point is more that this is all an analysis that Someone(tm) needs to take care of
18:26:04 <anarcat> so far i've been kind of assuming that Someone Else(tm) would take care of it
18:26:21 <anarcat> but right now it certainly feels that if i don't do it, no one will and everyone kind of assumes we'll just wing it :p
18:26:32 <ahf> hm
18:26:42 <ahf> CI is not gonna be on that one for sure
18:26:52 <ahf> artifact storage and container storage i don't even have on my mind yet
18:27:09 <ahf> the CI is a bit of a security nightmare in my world so i want that off everything else
18:27:29 <anarcat> right
18:27:30 <ahf> i think you are right to think the someone else thing
18:27:46 <ahf> i think that is fine right now, you are already contributing a lot in these conversations
18:28:11 <anarcat> okay :)
18:28:20 <anarcat> well i just want to add that to our launch checklist i guess
18:28:22 <anarcat> we need to
18:28:24 <ahf> so
18:28:31 <ahf> you know we have the gift cards to digital ocean?
18:28:37 <ahf> or not gift cards, but a sponsorship
18:28:44 <ahf> that i haven't sorted out, but that dgoulet got us setup with
18:28:47 <anarcat> 1. have someone talk with other gitlab providers (riseup, debian, gnome) and see what kind of traffic they're seeing and hardware they're using
18:29:00 <ahf> i was hopeing to use that for CI, but since we don't have CI right now other than jenkins, CI is not on my priority list right now
18:29:02 <ahf> but is on nice to have
18:29:11 <anarcat> 2. plan extra hardware and related finances to make sure we can handle the load after launch
18:29:16 <anarcat> 3. deploy that
18:29:34 <anarcat> 2 and 3, by the way, might involve splitting the service into multiple servers (say run gitaly elsewhere)
18:29:42 <anarcat> and yes
18:29:47 <ahf> is micah the riseup gitlab person?
18:29:58 <anarcat> i want to make it really crystal clear to everyone that we don't do CI just yet, that's another launch
18:30:16 <ahf> yes!
18:30:17 <ahf> agreed
18:30:22 <anarcat> as much as i'd like to retire jenkins, i don't think we should do it as we retire trac :)
18:30:29 <anarcat> one retirement at a time :)
18:30:40 <ahf> yep
18:30:41 <ahf> agreed
18:30:41 <anarcat> i am not sure about gitlab
18:30:43 <ahf> we are in on that
18:30:45 <anarcat> but i can ask them
18:30:53 <anarcat> point is: who's bottomlining that side?
18:31:20 <ahf> bottomlining?
18:31:53 <anarcat> uh
18:31:59 <anarcat> "making sure this happens"?
18:32:20 <anarcat> a "bottomliner" makes sure things are taken care of, even if they don't necessarily do the work themselves
18:32:27 <anarcat> coordinating i guess is a good approximation
18:32:37 <ahf> are you up for talking with them about this?
18:32:43 <ahf> i think you have a better overview of the concerns there
18:32:49 <ahf> hiro knows some scaling issues from oniongit i guess
18:33:16 <anarcat> i guess i can bounce that problem around with hiro and see which lap it falls on at the end :p
18:33:33 <anarcat> i am also wondering if we have any sort of budget for this launch in terms of hardware provisionning
18:33:49 <anarcat> i would love to be able to throw hardware at this problem if we need to
18:34:00 <anarcat> as opposed to spending days of staff work trying to fine-tune gitlab's performance
18:34:07 <ahf> so it is good you are a bit more balanced on this because i'm a bit of a yes hat person in all of this and it might be there are some performance issues that is worth thinking into it
18:34:11 <ahf> that i am not seeing
18:34:20 <ahf> i have no clue on budget here
18:34:25 <anarcat> that's because you're not dealing with trac crap right now :)
18:34:31 <anarcat> you would definitely be worried if you would
18:34:33 <ahf> i think once we are migrated, we can get rid of some of the resources from the other services
18:34:40 <ahf> yep!
18:34:41 <ahf> for sure
18:36:23 <ahf> crawling is an issue with trac, no?
18:36:26 <anarcat> yes
18:36:29 <anarcat> it will be worse with gitlab
18:36:31 <anarcat> oh
18:36:33 <ahf> i would be 100% sure that gitlab probably have that issue too
18:36:34 <anarcat> and monitoring
18:36:50 <ahf> if i google torproject stuff now i already see a lot of things on gitlab.torproject.org
18:36:50 <anarcat> we need monitoring, particularly hooking gitlab into prometheus and grafana
18:36:52 <anarcat> we have nothing right now
18:36:53 <ahf> like individual commits and such
18:37:07 <ahf> we should probably disable robots.txt because every call to many of the sites are probably expensive git operations
18:37:36 <anarcat> i'll open tickets
18:37:54 <ahf> awesome
18:38:55 <anarcat> https://trac.torproject.org/projects/tor/ticket/33921
18:39:40 <yanmaani> Can't you just stick it behind caching? Or are these idiosyncratic operations?
18:39:49 <ahf> awesome
18:40:04 <ahf> yanmaani: maybe putting varnish in front could be a solution, but i don't know
18:40:11 <ahf> maybe just blocking crawling in the beginning is fine
18:40:24 <anarcat> we already have nginx in front of it, and yes we could setup caching there
18:40:26 <yanmaani> Do Things That Don't Scale™
18:40:40 <anarcat> but as i said, that falls in the "let's spend a week tuning gitlab" solution :p
18:40:49 <yanmaani> (not sarcasm)
18:40:49 <anarcat> as opposed to "throw hardware at the problem" solution
18:41:00 <anarcat> https://trac.torproject.org/projects/tor/ticket/33922
18:41:08 <anarcat> alright, we can move the discussion to async
18:41:14 <anarcat> i'm happy with the meeting, and would move on :)
18:42:02 <ahf> oki
18:42:05 <ahf> let's close it down folks
18:42:10 <ahf> meeting in 1 week at same time
18:42:21 <anarcat> thanks ahf
18:42:23 <ahf> thanks folks, i will talk with gaba
18:42:25 <ahf> #endmeeting