17:01:01 #startmeeting anti-censorship weekly checkin 2019-09-12 17:01:01 Meeting started Thu Sep 12 17:01:01 2019 UTC. The chair is phw. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:01:01 Useful Commands: #action #agreed #help #info #idea #link #topic. 17:01:05 hi everyone! 17:01:15 for the record, here's our meeting pad: https://pad.riseup.net/p/tor-censorship-2019-keep 17:01:28 hi o/ 17:02:14 * catalyst is kind of here 17:02:20 let me start with the first discussion item: are there services left that aren't monitored or (re)started automatically on crash/reboot? 17:03:13 i'm asking because we ran into an issue with gettor - it looks like it wasn't started automatically on boot? is this correct, hiro? 17:04:10 fwiw, anarcat once helped me set up a relatively simple systemd script to monitor/restart services: https://help.torproject.org/tsa/doc/services/ 17:04:22 * anarcat o/ 17:04:32 gettor should have a systemd --user process now, iirc 17:04:57 yes, thanks anarcat. i set it up for gettor and systemd now restarts the service when it crashes 17:05:02 anarcat before sysadmin were against doing that... that's probably why there wasn't a script like that 17:05:21 (we have yet to see if it also does its job when the host reboots) 17:05:24 phw: awesome 17:05:30 hiro: ah 17:05:39 good thing i messed things up then :) 17:05:50 the idea was that if a service would be restarted when it crashed nobody would fix the reason for the crash 17:05:59 hiro: there's no possibility to monitor the gettor process from a separate machine, right? because both the http and smtp server are separate processes? 17:06:18 the http server is apache serving a static html 17:06:23 hiro: oh right, that's a good point 17:06:32 hiro: i thought we were just starting the service in systemd 17:06:33 hiro: that's a good reason but that works only if we have sound monitoring that tells us when a service disappared. we don't have that for gettor aiui 17:06:38 but yeah, systemd can also restart the service on crashes 17:06:47 i think that as long as people get notified on crashes, it's okay 17:06:48 the smtp server is twistd 17:06:53 phw: to answer your original "are there any other services" question, i am hoping we get to the point where you can show me a web page with a bunch of green lights on it. until we're there, i don't know how to know what got skipped. :) 17:06:54 I am happy to have something that restarts the service 17:06:55 but i shouldn't hijack your meeting :) 17:07:13 arma2: working on that dashboard in grafana, actually 17:07:30 hiro: so if gettor crashes, nothing is listening on port 25 anymore? 17:07:52 the system will still get the email but it will not be processed 17:08:08 the point is that if gettor only sends email it doesn't need to be a twisted service 17:08:21 it can be a python script that is called when the email hit the system 17:08:35 it's just a postfix rule 17:08:36 i understand that but our problem right now is that gettor was dead for ~3 days and nobody noticed. how can we notice? 17:08:39 arma2: this is the current grafana homepage https://paste.anarc.at/publish/2019-09-12-7hOjxgi6jUg/screenshot.png 17:09:02 maybe we can have system checks on the twisted process 17:09:05 phw: we should be monitoring the actual service 17:09:12 phw: send an email, check if we get tor back 17:09:13 via nagios or something else 17:09:38 checking that all the bits underneath are all in place will not be as useful as a "does the thing actually work" test 17:09:47 yeah we can send an email 17:09:58 you need to have an inbox somewhere to check if you get the reply i guess 17:10:01 but that can be arranged 17:10:06 nagios has checks to do this 17:10:21 i don't know about the politics of putting this in our nagios 17:10:33 but that seems like the best solution, technically 17:10:53 i don't have a strong preference but i think that some sort of service monitoring should be a priority. what do you think, hiro? 17:11:12 looks good to me 17:12:37 ok, let's try to get this done asap. can you take the lead on that hiro? 17:12:42 (i'm happy to help however i can) 17:12:59 (same here) 17:13:16 I can check how to do this in nagios 17:13:23 thanks! 17:13:38 fwiw, our main monitoring system is sysmon, run by gman999. the config is here: https://dip.torproject.org/torproject/anti-censorship/sysmon-configuration 17:14:15 it's limited though because it cannot follow http redirects. still, it does a good job at basic tcp reachability tests and has notified us of default bridges going offline 17:14:38 i realise that a mix of sysmon, nagios, and others is not optimal but scattered monitoring is better than no monitoring 17:15:56 (i also experimented with monit on my laptop, which i use to monitor a set of private obfs4 bridges that we will hand out to an NGO) 17:17:00 ok, that's it from my side wrt monitoring and reliability. any more thoughts? 17:17:47 next is a link to google's reviewing guidelines, which i found interesting and worth a skim: https://google.github.io/eng-practices/review/reviewer/ 17:18:36 i think we can learn from some of their experience 17:18:40 thanks for the thoughts, i can be better about doing reviews earlier in the week for sure 17:19:11 the network team also periodically tries to prioritize reviews, 17:19:14 cohosh: right, i was thinking about gettor. if hiro has only thursday to work on it, we should be done with our reviews by wednesday the week after, so hiro doesn't block on us 17:19:26 especially because a new volunteer will get hooked if they get feedback and attention, and wander away if they don't 17:19:40 they've found it hard to be consistent with that priority though 17:19:47 yep maes sense 17:19:58 *makes 17:20:16 phw I tend to do more things on thu but I have always other stuff so I do gettor when I can ... as I do all the other things 17:20:28 so reviews do not really block me 17:20:56 if I am blocked I ping you guys and let you know 17:21:02 hiro: gotcha 17:21:34 hiro: thanks :) 17:21:51 i think our weekly cycle works reasonably well but we should speed up things a bit if it's helpful 17:22:24 yeah i think i will try to prioritize reviews a bit more, was putting them in a giant bucket of "things to do by the next meeting" 17:24:04 another thing i liked in google's reviewing guidelines is to prefix suggestions with "nitpick" if they're worth pointing out but not necessary to incorporate 17:24:43 i can sometimes think of nicer ways to accomplish something but i don't want to drown somebody in minor feedback. that may be discouraging 17:25:00 the idea of "nitpick" is to say "hey, this is worth noting but feel free to ignore" 17:25:46 cool 17:25:56 (and as reviewee i would like to learn about all the ways to improve my code, even if i don't end up incorporating everything) 17:26:11 anyway, it's a useful document! 17:26:39 shall we move on to our 'needs help with' sections? 17:27:15 sounds good 17:27:47 hiro has 'probably more reviews' :) 17:27:50 keep em coming! 17:28:38 yes I might have some more reviews as I incorporate your feedback and fix a few more pending things 17:28:58 cool, i'm happy to take a look 17:30:12 another thing related to gettor: i didn't mean to step on your toes with the systemd script, hiro. whenever i touch things on getulum, i try to document it and let you know. but please let me know if you can think of a better process 17:30:32 (generally, i prefer not to touch anything without talking to you first) 17:30:43 you need to feel free to touch it actually 17:30:51 that's why I am setting up the ansible recipe 17:31:19 ok, gotcha 17:31:23 my idea is hat the ansible playbooks can run via cron and restart or/update the system 17:32:05 if everthing runs via ansible there are no hidden scripts 17:32:25 and everyone working on the service can see and improve the code 17:32:38 that's great, thanks for working on this 17:33:34 coming back to reviews: i only have one for now, #31692. it's a minor change to our docker image. can you take a look, cohosh? 17:33:55 phw: yup 17:34:20 next up is #29206 17:34:30 i think dcf1 has started reviewing it 17:34:41 yes I am 17:34:41 right, that's good 17:34:54 thanks 17:34:54 oh, you aren't absent after all, dcf1! 17:35:26 no I amn't 17:36:02 the last review seems to be #31455. i sent sina an email last week but haven't heard back yet. 17:36:27 that's a bummer.. he usually responds swiftly to bridge-related emails 17:37:27 is there anyone else at cymru that we could ask? rabbi rob maybe? 17:38:55 who runs it there? is it sina or is it a generic cymru service? 17:39:14 i believe it's sina, right dcf1? 17:39:19 it's sina 17:40:08 ok. then other cymru folks won't be so helpful. maybe follow up and ask how it's going and cc me? 17:40:37 last night i started answering a batch of urgent mails from july. and my watch tells me it's no longer july. so, this happens. :/ 17:40:58 * phw sent a reminder 17:41:28 looks like that's it for today. anything else? 17:42:06 not from me 17:42:12 alrighty, let's wrap it up 17:42:14 #endmeeting