18:31:08 <mikeperry> #startmeeting tor-dev
18:31:08 <MeetBot> Meeting started Wed Nov 18 18:31:08 2015 UTC.  The chair is mikeperry. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:31:08 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
18:31:23 <asn> hello im here
18:31:25 <asn> but not for long!
18:31:46 <asn> may i go first with the reportbacks?
18:31:47 <nickm_mobile> Let's start with status reports?
18:31:54 <nickm_mobile> Please!
18:32:06 <asn> over the past week, i worked on prop250
18:32:09 <nickm_mobile> Afk 5 min
18:32:16 <asn> i did a few refactorings and moved the implementation forward.
18:32:35 <asn> i think we are done with the voting logic
18:32:41 <asn> and the commitment/reveal logic
18:32:54 <asn> the next step is computing the actual shared randomness
18:33:13 <asn> i made a post in the ML about making this more robust: https://lists.torproject.org/pipermail/tor-dev/2015-November/009903.html
18:33:30 <asn> this is what i was up to.
18:33:37 <asn> i also have space in my review queue
18:33:55 <asn> so if nickm_mobile or someone else has something that you want reviewed please tell me.
18:34:23 <asn> and that's that
18:34:31 <asn> i will be off now, but i will check backlog
18:34:38 <asn> so please let me know which ticket you want reviewed!!
18:34:40 <asn> cheers
18:34:50 <asn> next?
18:35:41 <mikeperry> I have been testing #175292, along with the rest of my netflow stuff
18:36:11 <mikeperry> I've been running into issues where chutney does not behave very much like a real tor network
18:37:00 <mikeperry> primarily because of the continual consensus creation+downloading. I also just hit an issue where it tends to use two guards very often, despite the consensus params. this might be considered a tor bug, though
18:37:43 <nickm> rehi
18:37:49 <nickm> back from bus
18:38:00 <mikeperry> circuit_establish_circuit() chooses an exit before it chooses the rest of the path, so if it chooses your guard as an exit first, you have to use a second guard. in small testing networks, it happens frequently that this exit is your guard, so you often end up using two guards
18:38:29 <nickm> makes sense.  I seem to remember a really long arguument as to whether it leaked more info to have a guard as your exit sometimes, or a guard as your exit never
18:39:04 <mikeperry> otherwise #17592 testing is going fairly well. it is mostly behaving how I want, modulo those issues
18:39:45 <mikeperry> I might play with exactly when it samples timeouts and stores them. I'm running into the max(X,Y,Z...) distribution bias problem again
18:39:54 <nickm> hmm
18:40:35 <nickm> as for the testing issue, I would love to have a couple of chutney networks with default + accurate timing.  The ones we use for the default test really do need to be accelerated though, or otherwise they'd take hours to run right
18:40:51 <mikeperry> I think it probably shouldn't be storing per-circuit timeouts, but should have one single timeout associated with port prediction that also governs circuit idle timeouts
18:42:16 <nickm> my turn?
18:42:38 <mikeperry> I think that's it for me. I was going to comment more on circuit_establish_circuit(), but it is probably a rabithole
18:42:41 <nickm> I've been going a bit mad with auditors and accountants and trying to get an ED hired and making bills get paid.  Hope hope hope.
18:43:15 <nickm> I've been trying to revise and merge old patches from others, and merge other stuff
18:43:27 <nickm> I think I've finally got the patches all written for the first round of the blob-removal:
18:43:43 <nickm> with all of them merged, we will no longer have any SCC in our callgraph over 19 functions in size
18:44:00 <nickm> next step on that plotline will be to take function pointers into account
18:44:50 <nickm> i'm taking a break there to review a few more things, and finish #15055 at long last
18:44:53 <nickm> so tired of that one
18:45:03 <nickm> also got to get 0.2.7.5 out
18:45:07 <nickm> hm
18:45:11 <nickm> I think that's it for me
18:46:44 <nickm> anyone else around?
18:47:07 <nickm> ok.  Any discussion topics?
18:47:14 <dgoulet> hi meeting!
18:47:21 <mikeperry> I'm the only one who likes the new meeting times it seems :/
18:47:22 <dgoulet> yes I can go quickly
18:47:41 <nickm> great!
18:49:01 <dgoulet> I'm back from DC which was the R meeting so I'm catching up on things, mostly emptying queue of "people need my attention", I've started this morning also review on Yawning's #13737 (will finish soon), been busy yesterday also with our jabber server that needed immediate attention, I'm expecting things to cool down by Friday so I can focus on little-t tor code and more dev.
18:49:17 <nickm> yay for review
18:49:23 <dgoulet> done --
18:50:16 <nickm> any more?
18:50:20 <nickm> anyone else?
18:51:26 <nickm> ok. any discussion topics?
18:51:40 <nickm> I'd like to talk about the meeting time, but I think maybe that's a better discussion to have with more people. :)
18:51:45 <mikeperry> I am wondering if I should try to write an official chutney test for the connection lifespan stuff and netflow padding.. it would take minutes-hours to run, even if I coulod figure out what to instrument to check for success/good operation
18:52:09 <mikeperry> right now I've just been watching log lines
18:52:40 <nickm> I'd love to have a minutes/hours-long tests that covers this stuff, though the instrumentation is hard.
18:52:59 <nickm> I wonder if it's an option to make the tor code self-testing somehow, so it can LD_BUG if there's a mistake
18:53:03 <nickm> not sure if that's doable here though
18:53:45 <mikeperry> armadev suggested such a thing for #17604, but that seems expensive.. we'd have to iterate through all the connections and look for dups
18:54:24 <nickm> there's already an index of conns by identity I hope...
18:54:33 <nickm> Or we could check only once every N minutes
18:54:42 <mikeperry> for the netflow padding itself, its tricky.. cause what I really want to know is how long orconns live, and when the padding actually happens, and how much of it there is, independent of what tor actually things
18:54:58 <mikeperry> but I could settle for logline scraping or something, if we already had tests that do that sort of thing
18:55:13 <mikeperry> yes, there is an index of orconns by identity digest
18:55:16 <nickm> sounds more to me like you'll have to do some kind of capture on it
18:55:34 <mikeperry> and none of that exists right now?
18:55:45 <mikeperry> (capture in chutney)
18:55:50 <nickm> there's no "chutney + protocol capture" framework yet
18:55:52 <mikeperry> how about control port logging/monitoring?
18:56:04 <nickm> that could work, but it relies on there being no bugs in tor's controlport stuff
18:56:36 <mikeperry> does that exist yet either?
18:56:56 <nickm> there's lots of stuff in stem you could adapt, but it's not integrated into chutney
18:57:05 <nickm> I'd be interested in collaborating if this is something you want to work on
18:58:29 <mikeperry> possibly. I think I need to think about what I even want here. packet capture itself might be useless due to TLS.. that's why I asked about the control port.. somehow using the two together or something
18:58:36 <mikeperry> sounds complicated though
19:02:05 <mikeperry> if I checked for duplicate connections, I guess I'd just add that to run_connection_housekeeping() or on a less frequent schedule in run_scheduled_events()?
19:04:28 <nickm> run_scheduled_events is now refactored, but yeah
19:08:10 <mikeperry> oh, I have another random question. how do I check for the number of currently scheduled libevent timers, and how many is "too many"?
19:08:13 <nickm> mikeperry: could you have a quick re-look at my #17590 btw?  I fixed the XXX and added a little more to it
19:08:38 <nickm> mikeperry: there's not a way to do that, I believe.  Too many is when you start getting towards "one per connection" or "one per circuit"
19:08:59 <nickm> there are tricks to make those work, but they're probably best avoided
19:10:07 <mikeperry> is it just as expensive to cancel a timer in that situation as it is to add one? or what is the expensive operation? delivering the event for the timer callback?
19:10:59 <nickm> it uses a heap-based priority queue, so basically everything is O(lg n)
19:11:07 <nickm> add, remove, de-queue
19:11:11 <nickm> check-next is O(1)
19:12:55 <mikeperry> I could do something crazy like make a bitarray with 1000 bits in it, and stack the things if more than one of them is scheduled for a given millisecond
19:13:11 <nickm> what the F kind of events are you thinking about here?
19:13:59 <nickm> there's a crazy data structgure that can be more efficient if you really want, and if the timeouts in question are always constant, you can tell libevent to put those events in a doubly-linked-queue rather than a heap...
19:14:45 <mikeperry> basically, the way the netflow patch works is that once per second, I call channelpadding_decide_to_pad_channel() from run_connection_housekeeping(). if the time-to-send-padding is less than 1 second remaining, I need to schedule a timer to send the packet when it should be sent
19:15:03 <mikeperry> otherwise, I wait until the next call to run_connection_housekeeping() and do not schedule a timer
19:15:55 <nickm> what horrible thing would happen if padding was added on 1-second boundaries?
19:15:56 <mikeperry> and active connections will never come close to that point, so it will only be relays with a lot of inactive orconns that will end up with lots of these timers scheduled
19:16:01 <nickm> or 200-ms boundaries
19:16:02 <nickm> or whatever?
19:16:56 <mikeperry> against the default netflow configs, not much, but I didn't really like the padding to be completely predictable, because it may also obscure things like HS circuit fingerprinting
19:17:22 <nickm> maybe batch the connections into buckets and add the padding at different offsets into the second?
19:18:35 <mikeperry> and pick those bucket offsets randomly every so often?
19:18:43 <nickm> yeah
19:19:14 <mikeperry> could work
19:19:59 <mikeperry> is there anything that already happens on sub-second scheduling inside of tor that I could use instead, btw?
19:20:09 <mikeperry> out of curiosity
19:20:20 <nickm> If you'd like to see the improved algorithm which allegedly libevent should use instead, which could handle way more events, see http://www.25thandclement.com/~william/projects/timeout.c.html
19:20:23 <nickm> It's a cool algorithm
19:20:48 <nickm> resurrected from 1987
19:21:00 <nickm> not sure if it's a good idea though
19:21:23 <nickm> as for sub-second: I'll be adding it soon now that 3199 is mereged.  There are some kludges too.  one second...
19:21:44 <nickm> btw endmeeting maybe? I think it's just us two now
19:21:51 <mikeperry> ok. sure
19:22:21 <mikeperry> #endmeeting *baf* (couldn't resist using the e-gavel :)