15:01:00 #startmeeting prop259 15:01:00 Meeting started Wed Mar 23 15:01:00 2016 UTC. The chair is asn. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:00 Useful Commands: #action #agreed #help #info #idea #link #topic. 15:01:03 what is better 15:01:04 hello there :) 15:01:06 hi 15:01:08 ok 15:01:13 (meetbot is logging for posterity) 15:01:32 very handy! 15:01:34 ok so summary here is 15:01:42 that the prop259 spec has kind of stabilized 15:01:48 and now the STRIKE team is implementing it 15:01:56 to see how it works in practice 15:01:59 the latest spec is 15:02:00 : 15:02:06 https://github.com/twstrike/torspec/blob/review/proposals/259-guard-selection.txt 15:02:13 hey all, I'm here 15:02:17 #action we should push the latest version of the spec on [tor-dev] 15:02:27 olabini: greetings 15:02:34 ok you guys want to brief us up how the implementation is going? 15:02:51 or you want to start off in a different way? 15:03:23 rjunior: you want to lead this? 15:03:34 sure 15:04:00 we have implemented the proposal in C 15:04:08 it's in this branch 15:04:09 https://github.com/twstrike/tor_for_patching/tree/prop259 15:04:32 I was more interested in finding the right place to plug things into tor 15:04:33 is there a guardsim branch that matches the proposal and matches the C implementation? 15:04:40 yep 15:04:45 https://github.com/twstrike/tor_guardsim/tree/develop 15:05:00 i still have not seen the graphs from this simulation. 15:05:06 and I also have some graphs for the simulation 15:05:15 in a google doc here: https://docs.google.com/document/d/1meFgonpSzz_FEa2LMSRyt6AOaEcq27PCZSyEi7kqZeE/edit 15:05:35 back to the C code 15:05:57 the strategy has been using as much as possible of the existing tor code 15:06:13 so we dont face any problem extern to the proposal 15:06:44 in summary, we replaced choose_random_entry_impl by the proposal 15:06:58 aha 15:07:30 trying to roughly keep its semantics (adding chosen guards to the global "entry_guards", for example) 15:07:52 (The travel network case is not in the graphs, eh?) 15:08:20 and plugging everything else needed to dont break tor (neither because of the proposal, or code outside of it) 15:09:04 we still need to decide if this is the better place to put the proposal 15:09:34 which place? 15:09:47 and I'm currently interested in finding existing behavior that might be important but we are ignoring due the way we plugged the proposal 15:10:05 replacing choose_random_entry_impl 15:10:32 or if we shoudl keep it, and replace more granular functions (like choose_random_entry and router_pick_directory_server) 15:10:38 well, you will definitely need to replace choose_random_entry_impl() to implement this proposal. you will probably have to touch other places as well. 15:10:47 choose_random_entry_impl seems to be what we want 15:10:52 i agree 15:10:58 so does the thing work now? 15:11:02 https://github.com/twstrike/tor_for_patching/issues 15:11:11 * asn looks 15:11:16 I think it might be a good idea to step back. 15:11:25 Which do we actually have more confidence in now, the python, or the proposal? 15:11:28 I work (it does not crash) 15:11:57 Not having read either too closely in a while, I am more confident that the Python describes something actually buildable in Tor, in sufficient detail. 15:12:11 I'm less sure that the proposal really specifies a complete algorithm, rather than just describing it. 15:12:25 But I don't actually know. What's the status on those? 15:13:08 the python implementation and the proposal are in sync 15:13:20 and the C implementation is in sync with the proposal too 15:13:29 I'm not asking that part. 15:13:34 I'm asking about the level of detail. 15:13:43 I'm not sure what is missing to make the proposal really specify a complete algorithm, rather than just describing it 15:13:47 Though, there are some details in C we need to confirm yet 15:14:02 exactly. 15:14:03 If three people sat down with the proposal, would they all come up with a python implementation that chose guards in the same way? 15:15:19 they should, assumptions have been added to the spec 15:15:45 nevertheless, some concepts in the spec might not be exactly equivalent to those in the tor code 15:16:10 these are the details which need confirmation fan was talking about 15:16:15 we have ignored details on the current implementation and focused on specifying a strategy for choosing guards. things like how specific tor configuration properties should change the strategy, how the guard selection integrates with other parts beyond building a circuit 15:16:40 ok 15:16:40 and this is where we are 15:16:51 nickm: any questions about the spec? any specific doubts? 15:17:13 otherwise maybe we can go into the spec / code questions. 15:17:24 that rjunior has 15:18:01 I guess I don't see how it corresponds to the "entry points" we see in the guardsim code. I would expect to see part of it written in terms of "When a router goes down...." and "When we're building a circuit...." and "When Tor starts for the first time..." 15:18:20 but maybe I need to read more closely 15:18:29 in any case, let's move on to the spec/code questions? 15:18:43 so in reply to nickm I believe no, the spec lacks details needed to make sure 3 people will implement the same thing 15:19:35 ok 15:19:54 rjunior: hm, what are these details? 15:20:52 or detail about where it fits into the entire action of connecting, rather than just specifying it in isolation 15:21:12 how to behave when state->need_uptime or state->need_capacity 15:21:25 is one specific example 15:21:38 For now, some missing part of spec/python according to C are some detailed options and boundary cases, as mentioned. Some are the unspecified entry points. 15:21:58 or how ReachableAddresses should influence the guards we choose 15:22:45 and every implementation detail we might find while reading choose_random_entry_impl 15:22:54 chelsea_komlo: indeed, i don't think the proposal currently specifies really well how it fits into the Tor networking system, 15:23:13 there is the pseudocode in the appendix, but that does not reflect the tor reality as close as we would like 15:23:18 (as discussed in the recent mailing list thread) 15:23:18 I think we should document every aspect of choose_random_entry_impl 15:23:37 great, that can be an action item 15:23:48 i was expecting that as you guys get more familiar with how tor works, the pseudocode in the end would change slightly. 15:24:07 so the idea is making the proposal more implementation oriented (giving the current tor networking system) rather than more descriptive 15:24:10 I would be happy for somebody (possibly with tor experience) to also take on making Tor's internals look more like the guardsim api 15:24:32 given that the guardsim api is actually almost sane, but the current internals that tor has are pretty messed up 15:24:48 nickm: indeed that would be nice 15:25:13 that'll need a careful look at the changes, though, and a little planning 15:25:20 #action figure out how exactly the prop259 algorithm fits into tor 15:25:23 there. an action time. 15:25:24 item. 15:25:33 [also I'm in two meetings at once now so please poke me again if there's a question i'm not answering] 15:25:39 how to behave when state->need_uptime or state->need_capacity 15:26:01 that's also a good point. the current proposal does not specify what happens when we need a guard for a circuit with restricted requirements. 15:26:28 like what happens if your top guard is not fast, and you need a fast circuit. 15:27:13 the current tor algorithm just walks through all the guards from the top, and chooses the next guard that is fast on your list. 15:27:38 do you think that's wrong behavior? or just unspecified? 15:27:54 armadev: am I wrong, or are all guards not fast-enough and stable-enough? 15:28:07 The guard requirements right now include speed and stability 15:28:08 there are some very few guards that are not fast. 15:28:13 or there used to be. 15:28:16 this is the kind of details we didnt not sure exist until we started to implement the C code 15:28:27 rjunior: doesn't surprise me, yes. 15:28:37 I think that using any guard for those is okay. but I'd like armadev's opinion too 15:29:10 so i created an issue to create a list of this specific things and discuss them with you: https://github.com/twstrike/tor_for_patching/issues/3 15:29:33 for now, need_uptime and need_capacity are the ones I remember 15:29:34 and aldo 15:29:49 how should ReachableAddresses influence the guard selection 15:29:59 right, i think everything with the Guard flag meets some pretty good standards for speed and being-there-often 15:30:16 (if that was the question) 15:30:17 At least, sampled guards may need some re-definition 15:30:34 ah, and also entry guards vs directory guards 15:30:57 hm i'm not sure how exactly the current guard algorithm works with ReachableAddresses. 15:31:37 rjunior: what is the uqestion about directory guards? 15:31:50 I believe that we shouldn't connect to any guard not included in ReachableAddresses. If we've picked one already, we shouldn't throw it away, but we shouldn't use it. 15:31:55 when creating the sample sets, we assume it is very likely that the sample will contains directory guards 15:31:58 (not sure whether this is the current behavior or not) 15:32:31 otherwise we would need to create additional sample sets (for directory guards) 15:32:34 rjunior: indeed it is very likely, since most guards are also directory guards. 15:33:13 great 15:33:14 rjunior: has this been a problem to you? 15:33:34 like, that you populate your sampeld guards list, and then there is a circuit that asks for a directory guard and you don't have one? 15:33:55 no because we have skipped the directory guards until yesterday =D 15:34:06 hello and sorry i'm late 15:34:13 rjunior: OK, I see. 15:34:13 hi isis 15:34:30 asn: should we then use the fallback directories listed in fallback_dirs.inc? 15:34:40 iapazmino: how would you use them? 15:34:41 hi isis 15:34:47 but this is the current strategy, assuming it's very likely we will find enough directory guards in the sample sets 15:35:08 rjunior: indeed. 15:35:12 rjunior: although the way the algo currently works 15:35:23 if the algo is called first with dir=True 15:35:33 then the lists will be populated only with dirguards 15:35:36 no? 15:35:46 this is my understanding 15:36:00 asn: I thought these were the routers to be used when nothing else was left 15:36:09 but then, the next time you call with Dir=False, this guard will be ignored 15:36:14 asn: did not go that far as to think on how to use them 15:36:25 unless there is no other live entry guard 15:36:33 rjunior: why? 15:36:38 all directory guards can be normal guards. 15:36:44 oh 15:36:47 I see 15:36:57 in this case, yes this will be used until it fails 15:37:04 directory guards are guards that can also answer directory requests. 15:37:14 we're moving towards a world where most relays are directory caches by default. 15:37:17 in the future all guards will be able to answer directory guards, and hence all guards will be directory guards. 15:37:39 currently about 80% of guards are directory guards. so there is a small chance your top guard will not be and you will need to move to the next one. 15:38:14 so in the future there should be no distinctions between directory and entry guards 15:38:36 yes. in an ideal world, your main guard will also be answering your directory queries. 15:38:58 ack 15:39:03 about ReachableAddresses, we have decided to only use it to determine the sample set of dystopic guards 15:39:41 plausible. 15:39:46 the idea being: a node is dystopic if it matches the ReachableAddresses (or if its port is 80/443 in the absence of ReachableAddresses9 15:39:53 plausible yes. 15:40:19 but according to nickm we shouldn't connect to any guard not included in ReachableAddresses 15:40:38 indeed, so if you have already added a guard and then ReachableAddresses changed... 15:41:11 and the new ReachableAddresses does not include that guard, you should not try to connect to it. 15:41:27 meaning that the utopic sample set should also obey ReachableAddresses, correct? 15:42:14 i guess so. but i'm not sure if this restriction should be performed on the "pick the guard" layer, or on the "connect to the guard" layer. 15:42:20 not sure where it happens now either. 15:42:27 otherwise we would try to connect to guards not included in ReachableAddresses 15:42:30 ack 15:42:42 asn: I just got this on my 0.2.7.6 hs 15:42:53 so we stil pick the guard, but dont connect to it 15:42:54 Introduction circuit established without a rend_intro_point_t object for service 15:43:01 and why making such distinction is important? 15:43:02 Isn't that supposed to be fixed? 15:43:31 asn: if you come up with a (hopefully short) clear list of "how does tor do X now" or "what function should I look at to understand Y", i'll try to be helpful. 15:43:38 sorry for interrupting, /me waits 15:43:41 re guards, entry node selection, etc 15:43:48 if it happened at the "connect to the guard" layer, then the connetion wouldn't happen, that layer would ask for a guard from the "pick a guard" layer, which would not understand why 15:44:03 asn: i confess i haven't read the (hidden, not sent to tor-dev) proposal that is being discussed here 15:44:24 everything has been happening on tor-dev, it's just that we didn't send an individual mail about the proposal. 15:44:32 armadev: but yes, I will take your offer on this. 15:44:49 armadev: next time rjunior et al. ask me about some functionality i don't know about. 15:45:09 great. expect high latency on the responses though (maybe this helps you pare down the list) 15:45:37 bleh sorry my mind drifted. where were we 15:45:58 rjunior: hmmm 15:46:10 rjunior: i'll need to look at how REachableAddresses works in tor right now to answer your question better. 15:46:39 rjunior: if you compile a list of questions after this meeting, I will answer them in the next two days. 15:46:59 asn: this is my plan for the next 2 days 15:47:10 rjunior: btw, did you find all the places to hook your code? 15:47:28 i remember you were puzzled about the "while True" loop and stuff on the mailing list. 15:47:36 everything else I've sent in last week email, we have find a working solution 15:47:51 oh wow that's nice 15:47:53 I didnt have to to anything about it 15:48:00 while true is a goto hah 15:48:03 rjunior: yes doesn't surprise me. 15:48:13 the current way tor keeps trying to build a circuit until it success, matched with that loop 15:48:20 rjunior: because the execution will just come back to you; you don't need to make the loop yourself. 15:48:38 ok 15:48:45 I assumed it to be an implicit rather than explicit loop, as your email implied 15:48:50 ack 15:49:02 i'm curious to see how you have fitted the proposal in tor. 15:49:06 do you think i should look at the code? 15:49:13 you can 15:49:31 #action finally look at the code 15:49:38 #action answer rjunior's implementation questions 15:49:45 if you configure with --enable-prop259 15:49:48 alright 15:49:52 you should be able to run it 15:49:53 interesting 15:49:58 does it have any nice debug logs? 15:50:12 that i can monitor my guard usage with? 15:50:13 not many, this is a good idea 15:50:28 yes please 15:50:32 to be more detailed in debug logging, for the moment at least 15:50:37 that could be an action item as well 15:50:59 #action write beautiful & intuitive debug logging to monitor guard usage 15:51:20 my plan for the future is to run the proposal on my laptop for a month or so and manually monitor the guard usage, so good logs will really help. 15:51:57 last question from my side 15:51:59 so like where did you put the SHOULD_CONTINUE thing? 15:52:12 nickm mentioned some work on the modularity documentation during the network team meeting 15:52:13 in which tor function? 15:53:45 I assumed it to be the same thing as how channel_do_open_actions() uses entry_guard_register_connect_status() 15:54:08 ack 15:54:21 if entry_guard_register_connect_status() returns false in this function, it drops the chosen entry_guard and "continue" 15:55:05 rjunior: ah nice. 15:55:06 because should continue exists to simulate this existing behaviour, afaik 15:55:07 got it. 15:55:35 btw, wrt implementation details like reachable addresses, i would suggest you do the same thing that the current tor algorithm is doing when possible 15:55:52 if it's blocking you from progress 15:55:55 so, yeah, the integration is spread over different places due how the tor networking code currently is 15:56:05 but then also note it down somewhere so that we can look deeper into it. 15:56:08 rjunior: right. 15:56:29 back to mmy last question, Is there something we can start taking into consideration for prop259 in the subject of modularity? 15:56:44 nickm: ^ 15:57:14 rjunior: obvious things is to make a new file for the proposal and hide in there as much functionalityu as possible 15:57:18 asn: you mentioned a "Updated draft: improved modularity in tor" during the tor networking team meeting 15:57:27 ideally your interface will be much more elegant than the current entrynodes.c . 15:57:30 but I suspect it is not public (I couldnt google it) 15:57:58 nickm: can we send your modularity doc to rjunior ? 15:58:07 rjunior: i think nickm is going to push it to the mailing list RSN 15:58:12 nice 15:58:24 no more questions from me 15:58:27 rjunior: also, *not* using the global entry_guards list would be nice. 15:58:43 yes, send to rjunior . Also, I plan to send today. 15:58:50 ack thx 15:59:11 rjunior: ideally the whole guard algorithm thing could be objectified. and then we could have multiple of them running in parallel. 15:59:27 rjunior: in the sense, that we could have guard layers (prop247), and each layer needs a different guardlist. 15:59:32 rjunior: you can't do that with global structures. 15:59:59 #action send modularity doc to rjunior and team 16:00:54 any other question from the team? 16:01:01 rjunior: also, what are you currently doing about needs_capacity, etc.? 16:01:09 or any other question / action item from the tor team to us? 16:01:17 just one around the metrics on the simulation 16:01:18 we are just ignoring 16:01:23 the ones that I have can be answered by email ;) 16:01:30 tdruiva: great :) 16:01:35 iapazmino: go for it 16:01:47 another question: how much time you have left on this task? 16:01:58 should we reconsider the numbers we are gathering in different scenarios? 16:02:06 we're planing one week momre 16:02:14 *more 16:02:27 tdruiva: and then you are gone for real. fun. 16:02:41 iapazmino: what do you mean? 16:02:45 i mean, time and successful connections are not equally good metrics on a stable network as in a network-down scenario 16:02:49 \o/ well, we're working for it 16:03:06 asn: I thinks it' more like "and then we are not full time on this" 16:03:21 ok 16:03:22 also, how do you wanna to test before to get live? 16:03:23 asn: we won't be gone 16:03:33 asn: we are committed to shepherding this to completion 16:03:41 asn: but our hope is that the bulk of the work will be done by then. 16:03:44 olabini: ack 16:03:57 iapazmino: what do you suggest instead? 16:04:15 tdruiva: what do you mean "before to get live"? 16:04:19 olabini, +1 16:04:26 tdruiva: before you push the patch for review? 16:04:47 asn: i guess this would be a per-scenario thing, depending in what success means to each 16:04:58 [another question: i don't see the traveling network on the graphs list. should i care?] 16:05:01 asn, yes 16:05:07 tdruiva: hmmm 16:05:14 tdruiva: well ideally you guys would run the patch on your laptop 16:05:22 with that nice debug logging we were talking about 16:05:31 and you would use your laptop as normal, and maybe also use tor 16:05:37 k 16:05:39 and then every now and then you would chekc your logs to see your guard usage 16:05:50 and ideally you should be happy about the security that your algorithm offers you 16:06:03 if you see the algo swithcing guards all the time, it's a problem. 16:06:12 if you see the algo moving to the 5th guard when your 1st guard is up, it's a problem. 16:06:13 etc. 16:06:18 nice! 16:06:20 or well, that's what I'm planning to do at least :) 16:06:27 :) 16:06:33 asn: I guess I have them on my computer 16:06:36 will send them nos 16:06:38 now 16:06:42 alright 16:06:47 iapazmino: indeed. 16:07:22 iapazmino: if you can find scenario-specific metrics that can be enlightining and we are missing 16:07:25 iapazmino: we should do them! 16:08:20 asn: good, i'll think on this and run it by in the list for feedback 16:08:27 iapazmino: great. 16:09:03 for example, if we spend time on dystopic mode while we are on Default network. it's a problem. 16:09:07 we should never be down there. 16:09:08 asn: done 16:09:15 ok 16:09:20 any other questions? 16:09:32 otherwise, I will be monitoring the mailing list in the following days. 16:09:47 also I will take a look at the code just to see what's going on. 16:09:57 asn: me done 16:10:06 great 16:10:09 thanks! 16:10:13 alright thanks guys!!! 16:10:14 ty asn 16:10:23 let's be in contact the following days :) 16:10:23 thanks! 16:10:24 take care! 16:10:31 good bye 16:10:38 #endmeeting