17:00:24 #startmeeting OONI weekly dev gathering 2016-05-09 17:00:24 Meeting started Mon May 9 17:00:24 2016 UTC. The chair is hellais. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:24 Useful Commands: #action #agreed #help #info #idea #link #topic. 17:00:37 hello Oonitarians! 17:00:42 who is here? 17:00:50 hello :) 17:01:25 hi 17:02:40 to speed up the initial report-back phase we moved that to a pad so we don't clutter your backlog. If you are curious to see what people have been up to you should check it out. 17:02:54 link? 17:03:01 #link https://storm.torproject.org/grain/xLzNzE64JiHuMERL25Gkrq/ report backs for 2016-05-09 17:03:19 feel free to add anything you did ooni related to that pad :) 17:03:28 hi all 17:04:03 Hi 17:04:27 catfish99, andresazp: hey there :) 17:04:39 so let's begin with the first item on the agenda 17:04:48 #topic Come to an agreement on what should be the canonical place to store issues 17:05:14 hello 17:05:30 I have a strong opinion for us to switch to using github entirely for the following reasons: 17:06:08 * It's tightly integrated with the code review process so it makes it easier to work between issues and code review (pull requests) 17:06:53 * A lot of other projects that are part of the OONI ecosystem already live there and are unlikely to move their issues elsewhere (I am thinking of measurement-kit/*, citizenlab/test-lists) 17:08:05 it's possible to clone/port github issues to gitlab/something else if github ever went goodbye? (I think that's true) 17:08:14 * Working with github is much more pleasant than working with trac 17:08:36 aagbsn: there's an api to get issue contents, and scripts to do migration. 17:08:55 i'd be happy to see github be the canonical place for issues 17:09:36 me too 17:10:07 I don't think that we should use Github as the canonical place for anything. 17:10:11 other than the obvious free software reasons, what are the advantages of moving everything to trac in comparison to github? 17:10:12 aagbsn: I have a script for migrating issues from github to trac: https://github.com/hellais/gh-to-trac 17:10:49 * Github has been banned quite some times repositories and projects. 17:11:24 * It's much easier for an authority to make Github block a project 17:12:36 it's easier to block *.torproject.org, with little undesirable collateral impact 17:12:45 i dunno if i agree with that argument. ooni seems very unlikely to be blocked - there's no copyright infringement going on, which is the only thing github has really complied with afaik 17:12:54 and github is available in many more countries than trac is 17:13:17 * It requires that all contributors should create a Github account 17:13:22 yes I was about to say that. It seems to me the blocking issue is more relevant for trac.torproject.org 17:13:41 just as an example I can't use trac.torproject.org from my university, because they blog *.torproject.org 17:13:42 aagbsn: If *.torproject.org is blocked it affects OONI anyway. 17:14:00 hrm? 17:14:18 what does ooni get from torproject.org? 17:14:24 anadahz: how does the blocking of *.torproject.org affect OONI? 17:14:44 willscott: Ref: https://github.com/shadowsocks/shadowsocks 17:14:52 that wasn't github 17:15:00 that was chinese police showing up at the guys door and telling him to take it down 17:15:47 willscott: and this occured to all other forks as well? 17:16:29 hi 17:17:05 I think it's reasonable to have a few ways to report an issue, such as an email + gpg key, as well as github, but to expect that most development would happen on the default platform of choice 17:17:08 anadahz: there are still many live forks of shadowsocks: https://github.com/RockerFlower/shadowsocks. 17:17:13 anadahz: like these forks: https://github.com/shadowsocks/shadowsocks/network/members 17:17:43 i feel like github handled the shadowsocks thing pretty well. same with the greatfire debacle 17:17:54 willscott: why can you then download it here: https://shadowsocks.org/ ? 17:18:08 set up by a 3rd party afaik 17:18:13 anadahz: yes I agree that we can provide some other options for reporting issues for people that don't want to use github. 17:18:20 *aagbsn 17:18:23 there was some shady stuff going on with regard to nytimes reporting on that greatfire incident though, and github did 503 the pages 17:18:42 'nothing to see here' 17:20:33 so as i see there's a trade-off here between control / ownership and accessibility 17:21:05 aagbsn: IRC that happenned when china started doing a DDOS to the greatfire pages, so they had to blackhole them for a while. 17:21:22 since the downside to 'something bad happening' is that we have to take someone's local version of the repo and put it somewhere else, that transition seems acceptable to me 17:21:29 yep it looked like a unicorn tarpit 17:21:37 hellais: many scripts use *.torproject to install ooni-* required dependencies 17:21:54 the arguments against delegating it seem to be the moral objections around needing a github account, and the bad-taste of proprietary code 17:22:04 tbh I am much more for addressing the imminent issue at hand, having a good issue tracker that is the canonical place to file issues related to OONI, rather than finding a solution for some hypothetical not well defined threat. 17:23:14 willscott: agreed, that's my only real issue here; has anyone looked at platforms that are better than github in any dimensions? 17:23:41 late to the party, anyway my opinion is that I'd rather use github. 17:23:50 issues will likely continue to still be reported out of band - email, irc - and existing developers will be able to post those as issues on whatever platform we use, so the overhead of needing an account wherever seems not overly critical 17:24:35 regarding the possibility of submitting issues and patches, perhaps ooni-dev could by used by whom does not want to use github? (i.e. one could send emails describing issues and git send-email patches?) 17:24:38 I don't want to support a company that treat employees in a bad way (sexual harrassment, discrimination) 17:24:51 i guess my main thing is that i'd prefer not to have (a) another account to remember (b) a new process to learn 17:25:01 and i guess high availability 17:25:31 so like, us hosting a gitlab instance seems worse to me because i don't want to maintain it and i don't believe we'll have the time to keep it up and running reliably. i'd really rather someone else deals with it 17:25:44 willscott: +1 17:25:48 willscott: +1 17:26:07 Not that I think my opinions should have any significant weight in the decition, but in my opinion github makes it easier for pleople delopying ooni (like myself) that are less involved than key contributors to participate more 17:26:31 willscott: +1, esp since I've never even used trac 17:26:54 sbs: OK, shall I start using ooni-dev for submiting patches and discuss issues? 17:27:12 Will the rest of the OONI team be happy with that? 17:27:14 deploying* 17:27:50 anadahz: well, if I see a diff floating on ooni-dev I'm definitely going to read it and provide feedback 17:29:14 I think using email is not the best alternative of that but what I want to say is that people that don't want to open a Github account will not send an email with a patch. 17:30:08 i don't think we're going to get any more consensus continuing this discussion now. 17:30:27 anadahz: is there actually people that do not have a github account and that would like to contribute to ooni, or is this just a hypothesis? 17:30:33 i'd propose the following: for those opposed to github, can you get a working alternative that others are willing to buy into 17:31:02 otherwise we'll inevitably have a continued stream of issues coming in through github. 17:31:27 and no one else will run a gitlab for you? 17:31:45 (if the issue is, maintenance, vs free software) 17:32:20 i bet someone would 17:32:34 but there's a bunch of work there 17:32:39 commercially, even 17:32:41 is someone volunteering to do that work? 17:32:56 because then we have 3 places where the code / issues could live :-/ 17:32:57 aagbsn: I think gitlab is not desirable for the reasons we have already mentioned before: * It's one more piece of infrastructure to run, that is likely to not be as reliable as github * It's one more place where you need to create an account * Other projects that we work with are not there 17:33:08 sbs: yes there were some people that's why OONI moved the tickets from Github to Trac two times already hellais can give more details on that 17:34:17 anadahz: actually moving it to trac turned out to not be a good move in the end. In the time when we moved to trac from github, it was mainly on ethical grounds and the hypotetical "people don't want to use github" argument. 17:34:57 sbs: a time where there were a bunch of OONI contributors 17:36:35 sbs: not only in terms of code but in general , enthusiasts, people that explore things, ... 17:37:10 when the issues were moved to trac and issues on github disabled I didn't really see that much increase in reported issues, while it did seem to go up a little bit when I reopenned them in github. Though this is a very un-scientific eyeball measure. 17:37:22 sbs: with lively IRC discussions 17:39:09 aren't most developers who would contribute to ooni most likely using github anyway? In terms of non-developers, to my understanding the plan is to provide a GUI through which they can contribute to test-lists 17:39:35 anadahz: what period are you referring to exactly? 17:40:29 anadahz: also I fail to see how this is relevant to the question of trac vs github. 17:40:32 okay. we've spent 40 minutes on this. we should continue with other agenda items rather than rabbit-holing? 17:40:42 willscott: yes, indeed. 17:40:45 willscott: agreed. 17:40:57 can we say that we have reached a consensus that we should be switching to github entirely? 17:41:15 hellais: i don't think so. anadahz and aagbsn dissent 17:41:34 but we should continue out of band. we don't need to take up all of our weekly meeting on this 17:41:36 sbs hellais: we can discuss this after the meeting 17:41:43 ok 17:41:48 we are not likely to get more consensus in the next 20 minutes. 17:41:49 yes let's continue 17:41:53 i don't dissent - just want to see if we've explored other options than trac/github 17:42:07 i think github is fine given the reasons mentioned and that we can ditch it if we change our minds 17:42:28 aagbsn: ok thanks for clarifying 17:42:33 though let's move on 17:42:36 #topic Ensure that OONI git repositories are mirrored and synced in different git servers 17:42:37 anadahz: ok 17:43:13 sounds good - should be a hook on each developers with push access? but how does this integrate with pull requests... hmm 17:43:22 aagbsn: yeah that is what I was thinking 17:43:28 #link https://trac.torproject.org/projects/tor/ticket/18606 17:43:59 the only reason why I haven't yet implemented a git hook for that though is that you can't force push to torproject git mirrors 17:44:14 that is a feature 17:44:17 and I generally prefer to squash and rebase commits with master before merging them 17:44:20 hellais: what about automatically sync-ing up master? 17:44:23 are we regularly force-pushing? 17:44:30 so probably it should only happen when pushing to master 17:44:34 that seems like it should be an extraordinary event, and be worthy of extra effort 17:44:35 willscott: only on feature branches 17:45:12 but on feature branches it does happen regularly, because it's generally ideal to not litter the commit history with merge commits 17:45:24 can you not delete a feature branch? 17:45:28 (on tpo infrastructure)? 17:46:07 do feature branches need the same level of replication as master? 17:46:31 We should also make sure that all OONI related git repositories are synced 17:46:39 aagbsn: no you can't 17:46:57 willscott: no feature branches don't need to be replicated 17:47:53 well, sometimes feature branches may be long-lived before merge. it would be a shame if something happened to them 17:49:50 aagbsn: yes I guess, so but I really don't want to loose the ability to be able to clean up history before merging into master. 17:50:14 they're already going to be in at least 2 places right? on the developer's machine and wherever the developer has chosen to make them public 17:50:34 willscott: yep 17:50:37 syncing of those seems needed only if we have multiple collaborators on a feature branch pushing to different replicas 17:50:44 another thing to consider is that a lot of the merges are done via the github interface, this means that a local git hook would not be sufficient. 17:52:03 hmm. so what more do we want? 17:52:49 willscott: currently the git repos hosted in torpoject are not in sync 17:53:34 either we decide to rip them and host a git backup somewhere else, or we keep them in sync 17:54:44 but having git repos that are 19 months unsynced it's not desireable 17:55:13 seems reasonable 17:55:44 so, the issue as i see it is that access on the tpo repos is limited based on some credentials that are not clear to me 17:56:54 but we can have a webhook from github that triggers an automatic pull to tpo with the right credentials 17:56:59 is there a vm where that could live? 17:57:35 or we could even give github a deploykey to auto-push directly to tpo, i think 17:57:57 maybe that's the other direction though 17:57:59 willscott: perhaps this gitlab instance needs to be created ;) 17:58:12 then we have 3 things to keep in sync :p 18:00:29 anadahz: i have no strong objection to such a thing existing, but hypothesizing about it isn't going to be productive. if this is the direction you think is right, propose a place for it to live, and who's going to do the work of setup and maintenance. 18:03:28 willscott: I'm just proposing it as an option no need for 3 way syncing, backup git repos can exist there together with the gitlab instance. I don't think that maintaining the Gitlab instance will be that much work since we 'll need to maintain the backup git repos anyway 18:04:08 who's volunteering to do it? 18:04:09 except if we find a viable solution with the torproject git repositories 18:05:24 I can volunteer to do that together with the person that will setup the backup/sync git repos process 18:07:10 who has access to the tpo repo besides you and hellais? 18:07:53 willscott: I dont have access to the tpo repos. 18:07:58 willscott: isis and aagbsn should also have access to them. 18:08:19 anadahz: you don't? 18:08:30 hellais: no 18:09:16 that seems like a pretty good indication that we shouldn't be using those as anything other than mirrored backup 18:10:26 related #17524 18:11:08 yeah, I think what we should decide is if it's acceptable for them to not be perfectly in sync and them only being synched when I remember to push to them (i.e. not that often as it turns out, though I can try to make an effort, but knowing myself I will sometimes forget to synch them and could be off by some weeks) or if we need this sync process to be automatic 18:12:40 hellais: if the goal is to have a backup, then months out of date probably doesn't do that 18:13:13 i guess i see 2 ways forward: 18:13:41 anadahz sets up a gitlab. that becomes backup initially or maybe eventually primary. we get rid of the tpo ones 18:14:43 i dont think we should have official repos that aren't sync'd 18:14:55 or hellais gets an auto-pull to happen to tpo or figures out a way to get credentials for someone else to do that 18:15:46 the process should be automatic, or a pull/push script a developer with access can run. github should not get to push to tpo automagically, though at least it cannot force push 18:16:07 willscott: I will look into a way to make it so the sync will happen automatically from my dev machine. 18:16:49 aagbsn: i was imagining a sync run on a developer machine that was triggered by a github webhook 18:17:23 isn't this just git pull github master && git push tpo master ? 18:17:34 sure is :) 18:18:05 hopefully takes about the same time as the 40 minutes we spent talking about it :) 18:18:09 what's hard about doing that periodically :)? 18:18:10 aagbsn: yeah the command is trivial, it's just that if some things are merged via github and I forget to run it for a while, these issues with it being out of sync happen 18:18:27 a cron job also seems totally fine 18:18:34 it only breaks if someone force pushes to master :) 18:18:41 hellais: anacron ? 18:18:58 yeah I will figure this out. I don't think we should continue this discussion 18:19:14 great. should we tackle another agenda item this meeting? 18:19:22 #action hellais will figure out syncing to tpo 18:19:33 I think we are a bit overtime 18:20:11 yeah I think at this point we should defer them to next week 18:20:15 and I would like to listen from andresazp and catfish99 if they have something to mention/comment/report back 18:21:01 On the VE deployment? 18:21:24 sure. anything since last week or things you'd like from ooni? 18:22:12 since last week - just that updated VE testing list was shared with the OONI team. request that data be merged with existing testing 18:22:35 so that future testing will include new/updated sites that were identified 18:22:45 I havent checked the conversation last week, I´m not working closely with catfish99 18:22:59 I added links to reports on the document linked i the email 18:23:03 catfish99: so can we publish the testling list? I was under the impression you wanted us to wait for the report to go public 18:23:19 Hi catfish! this is andres 18:23:20 wait until report goes public - later this month 18:23:34 sounds good. 18:24:34 sorry guys I really need to leave, but happy to discuss any pending issues later on jabber 18:25:21 catfish99: ack 18:25:22 I undestand that´s happening next week 18:25:43 great. lets end? 18:25:53 Question 18:26:00 go for it 18:26:05 catfish99: we still don't have the translated version 18:26:21 the topics like depricating DNS_consistency, etc are pushed for the next version? 18:26:29 yeah. until next week 18:27:06 andadahz: you meen the url test lists? 18:27:52 anadahz: yes we should talk about that next week, but very briefly I can tell you that we have been developing a new test that will be testing for most of the things that http_requests and dns_consistency does, except it's a bit more accurate. 18:27:53 andresazp: no it was about the report text 18:28:11 this is the test: https://github.com/TheTorProject/ooni-spec/blob/feature/web_connectivity/test-specs/ts-016-web-connectivity.md 18:28:20 andresazp: ^^ 18:29:49 if we don't have anything else I guess we can end this? 18:30:19 I can send detailed info on how the testing was peformed 18:30:41 and the incidents identified, the text of the report itself is out of my scope for this onw 18:30:43 one 18:31:52 I´m very keen in participating on the discussion about the web connectivity test next week 18:32:53 andresazp: excellent :) 18:35:07 Thankfully recent timzone changes in VE make it much easier for me to login to the weekly meetuo 18:36:00 andresazp: You can always sen an email to ooni-dev mailing list (https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-dev) and continue/start the discussion from there. 18:40:22 I think we are good to end this meeting? 18:40:52 sounds good 18:40:54 thanks for attending 18:40:57 #endmeeting