I like trac. we should use it jml: i need your help --> chaka (~chaka@ns2.toolshed51.com) has joined #twisted jml: i am going to do some refactoring on trial slyphon: awesome slyphon: I've been wanting to do that for some time. slyphon: honour to serve. how can I help. jml: so i could really use your input heh; can you explain the concept of the runners to me? or rathedr wait please detail the major chunks that make up trial and what each individual chunk does slyphon: ok. gimme a moment. sure I'm on a new install, so I have to checkout twisted and set up my .emacs no problem but, while that's happening, let's talk vision I'd like stuff like atop.test.unittest to be much easier. and, if possible, for trial to relate to the reactor in a simpler fashion. slyphon: what are your goals? we share some common goals i'd like to make trial understand componentized slyphon: in what sense? ITestCase --> norfolk (~alan@ip68-96-11-53.hr.hr.cox.net) has joined #twisted no issubclass() no istestcase() slyphon: I considered doing ITestCase when I first wrote trial. People said it wasn't necessary because the major point is that a class that runs as a testcase must provide an interface yeah slyphon: but now I agree with you. that's what they told me but screw them one sec slyphon: backwards compatibility is a consideration though yeah that's a big consideration i convinced glyph that ITestCase is a good idea slyphon: can you summarise the discussion? my gf is on the phone, will you be around in 20 min? slyphon: yes. ok, i'll talk to you than then :) ------- yo hey whew i'm glad you're still here i had to drive everyone back into DC and talk to my gf but now, i've got a nice block of time to actually get something _done_ ;) so, trial if you've got a minute yeah, I've got a minute or two :) YAY :D okay so we were talking about components first thing, as a design decision, yes, i want to make TestCase __implement__ ITestCase can you, at some stage, summarise the discussion you had with glyph wrt components and trial? well the basic consensus i had with glyph was that the runner stuff needed to be revamped, because nobody really understands what it does... ... ...and that there would be an IBenchmark :) IProfile, and ITest interface yeah. that way you could do ITest().allYourTests() IBenchmark().allYourBenchmarks() etc. to give the developer more control over what gets run how and when also, it would allow you to benchmark already-existing code and tests without worrying about how the methods were named which is kind of irritating, yet understandable well, the naming thing is a tradeoff between control and convenience yes which we agreed upon so, ITest(myTestObject).allYourTests() ? yet, we can make allYourTests() default to return all methods named test_ and if someone wants to change that the machinery is there to make it easy yes or how do you make it default? ITest(ITestCase(obj)).allYourTests() you make it a default by making a default adapter i think I see. i am _very_ concerned with backwards compatability so, the user would write a class that would implement ITestCase. What's ITest? slyphon: I'm not :) heh :D well, the user wouldn't see much of a change if they wanted to use trial.unittest because t.unittest.TestCase would implement ITestCase so the difference would be pretty transparent slyphon: yeah. I think we should be backwards compat. I just don't feel strongly about it at all. yeah i agree slyphon: so, what's ITest when it's at home? ITest would return all test_ methods, or if overridden all methods of a class that should be considered test methods which brings me to a point of confusion because i've hacked some edge cases where i messed with the test classes or methods (i can't remember the exact case) and wound up with "foo is not a test method" errors which i don't really understand slyphon: the runners are essentially adapters, I guess. --> psy (~joe@adsl-1-046.QLD.dft.com.au) has joined #twisted although, it's slightly more complex :) adapters in the IBenchmark, etc way? yeah ah what's a "Singleton" runner? TestClassRunner should really be an adapter to ITest slyphon: runs a single method. yes OH :) that confused the hell out of me slyphon: not singleton as in the pattern slyphon: sorry man what would be cool would be to make the ITest().allYourMehtods() return an /iterator/ :) slyphon: yes that way you don't have to worry about counting methods but, the thing is, what would the interface for IBench be? slyphon: counting methods is for progress bars. oh oh hmm hadn't thought of that IBenchmark would probably be just "return all methods that you want to be timed" during execution and it would return an iterator over a bunch of methods ok. so, it'd have one method. just like ITest yeah make it really simple ok, so what does ITestCase define? umm setUp, tearDown? yeah, it also defined raiseIf failUnless etc which was because i needed to change those for some of the functional tests in quotient slyphon: that's unnecessary. they should not be TestCase methods. oh i had some weird requirements about skipping the rest of a sequence why? --> zbir (~zbir@m41105e42.tmodns.net) has joined #twisted because the tests _had_ to run in a specified order slyphon: they still don't need to be TestCase methods. hmm what would they be then? i agree --> SamB (naesten@ts001d0655.wdc-dc.xod.concentric.net) has joined #twisted slyphon: whack em in a module IUnittest ? nah. yeah you're right soooo... twisted.trial.judges.failUnlessEqual heh slyphon: I don't know if ITestCase is well-thought out. probably not... but the idea was really to make the "is this a test case" stuff in the script easier for sure. it might only be a tag. yeah hmm but, I think perhaps we might want IBenchCase, ITestCase etc i'm not sure. yeah, neither am i the difference was that i wanted ITestCase to cover the other possibilities, i think the whole thing that's a bit weird to me is dealing with the TestSuite how you can add just random methods to it or classes TestSuite.. TestSuite.. what's that again. or modules or packages * jml looks stuff up heh * slyphon does the same slyphon: what's confusing about it? what is its purpose? slyphon: the script gets a bunch of params, they get added to the test suite. then it gets run. slyphon: it's purpose is collation ah yes, one of the things i need to do is sit down and write a design document just to try to understand what all the use cases are slyphon: well, nothing so formal yeah well, a "design document" (quotes added) would be more appropriate ;) :) TestSuite might not be needed hmm how that would be slick I want to wind up with a design that will cut 50% of the code out of trial also, I reckon we ditch the progress bar requirement. No UI uses it. slyphon: me too. heh slyphon: YAGNI. some people were suggesting that i build a gtk ui for it i was like, FUCK NO! slyphon: that's irrelevant. YOU BUILD IT yeah trial needs a gtk ui like a fish needs a bicycle slyphon: also, buildbot is an important consideration. yes i've been talking with warner --> chaka (~chaka@ns2.toolshed51.com) has joined #twisted he wants there to be a Problem object that can get passed via jelly that takes the relevant locals information and can pass it off to buildbot also *blink* i didn't see the "there" at first aside: to benchmark stuff, you'd need, essentially, a list of methods to benchmark, and a process that takes a method to benchmark, and actually benchmarks it. which is pretty trivial time.time, startmethod(), time.time - originalTime slyphon: well, the Tester class is the thing that takes a method and tests it * slyphon nods dreid: :) slyphon: it's actually not going out and killing people in its triviallity explain please ;) slyphon: look at unittest.Tester. It's 100 lines of code oh, you were talking about tester i thought you were talking about benchmark's triviality yeah Tester is gonna need some refactoring it seems a bit too "busy" slyphon: well, I'm thinking that a lot of that is intrinsic complexity intrinsic in what sense? because of the rest of the design, you mean? slyphon: because of tests raising different kinds of exceptions. ah hmm and the reactor yyeah warner and i had a _long_ talk about the reactor state sure, if we had a Bencher and a Profiler, there'd be a lot of stuff in common slyphon: I wish I were there. dreid: I still don't get it :( :) we both agreed that the framework should just VOMIT if the reactor was left in an unclean state jml: i thought warner wanted to be a Problem object that can be passed via jelly HA! dreid: right :) jml: so yeah ... it left me in a state of amused confusion slyphon: why did you agree? slyphon: as in, why should it vomit? well, it was a question of running asynchronous tests if you set up a callLater, and there's a code path that allows you to exit out the side without cleaning up that call later, you can get weird failures on unrelated methods which can be hard to track down i.e. the http logging "optimization" that just leaves timed calls lying around which is a major PITA slyphon: so, by vomit, you mean, fail and exit yes brb immediately ok --> z3p (~z3p@pcp04401522pcs.nrockv01.md.comcast.net) has joined #twisted perhaps ITestCase should have 3 methods allYourTests, allYourProfiles, allYourBenchmarks back slyphon: I was thinking about that yah, that seems more natural hmm. itamar was talking some crazy shit about how he wants to be able to adapt the /classes/ to ITestCase not the /instances/ so he would be able to use PyUnit tests with trial which i told glyph and he looked at me as if /I/ was crazy which is quite a compliment coming from glyph ;) slyphon: one of the problems with trial atm is that it gets all the methods _from_ classes right at the beginning and only instantiates those classes when they are about to be tested yes slyphon: I don't think we need to do that. that's the thing i was talking about before with "is not a test method" ahh, right. soo, i think that would be handled by the ITestCase thing we only really need to have a list of classes at the beginning, and then get the methods as we are about to test them. yeah which is an iterator over a list of methods and the list could be a list of modules, classes, and methods lemme rephrase that ok. TestSuite could be replaced by a list of stuff that would have for meth in ITestCase(foo).allYourTests(): run meth() called on it so then you register an adapter on ModuleType, ClassType, object and method that do The Right Thing (like return testcase classes, etc) words right out of my mouth :) i'm so glad that we're agreeing on this so many people were telling me YAGNI! at the top of their lungs i was thinking "FUCK YOU!" _I'M_ implementing it slyphon: it's not a matter of YAGNI at all. It's refactoring mercilessly yes YES! WITH A CHAINSAW! OF JUSTICE! slyphon: SO! yes slyphon: what of the case of setup/teardown ? hmm in what sense? oh shit hmm slyphon: well, we may want different ones for different "test" types (bench, profile, dbtest) etc yes hmm <-- Jerub (~stephen@CPE-138-130-223-236.qld.bigpond.net.au) has left #twisted that's why i was thinking of ITestCase, IBenchmark, IProfile because then you get IBenchmark.setUp() slyphon: then let's do that. yes slyphon: but how would the adapter work? your adapter class would make sure that the correct method got called self.original.setUp() or some such (in the simple case) hmm slyphon: I'm not quite sure I follow. Let's say we have a class that has tests and benchmarks, yes so that would implement ITestCase slyphon: and IBenchmark? then, you would register an adapter of that class to IBenchmark ok now I get it and that adapter would handle the calls to .setUp .tearDown, etc yeah it makes a lot of sense, when you think about it IBenchMark.setUp === self.original.benchSetUp yeah or whatever yeah. whatever :) that would probably be a good standard profileSetUp but so, ITestCase is a tag, and that's it. your adapter class should probably contain a slightly different implementation yes afaict it just lets you pick out the TestCase classes out of a module hmm yeah because there may be instances where you don't /want/ to run unittests using a certain class, only IBenchmark and then, for each of IBenchmark, ITest, IProfile, we have some code that actually benchmarks, tests, profiles yes how would that be structured? where there could be cases, where there's only one of those you want to do, that would implement both ItestCase and ITest well in the simplest case you'd have a class that defined prefixed methods test_ bench_ profile_ the adapter to IBenchmark().allYourMethods() would return all 'bench_' prefixed methods of self.original just as an example implementation so BigTestCaseClass has test_foo bench_foo and profile_baz ITest(BigTestCaseClass).allYourMethods() returns test_foo yeah, I've got all this so far ok, so what have i not explained that you're thinking of? well then nasty user comes along and says "trial profile BigTestCaseClass" hmm so we do IProfile(ITestCase(BigTestCaseClass)).allYourMethods() which returns only profile_baz or nothing if no profile methods are to be returned yeah, I was wondering about the code that actually does the profiling. does that need to be structure in any way. i.e. we laugh at him hmm i'm not really sure i'd need to talk to JP and Glyph we should make it so users could pick either the Hotshot profiler or the old-skool one slyphon: my guess is 'not really, but it'd be kind of nice' yeah well, our framework doesn't really give a shit the _user_ picks how profiling should work which is the genius <-- chmod has quit (Read error: 110 (Connection timed out)) now. one final issue. ok how to handle test failures and errors. hmm <-- teratorn has quit ("Leaving") this is one thing warner had a bunch to say about / results in general slyphon: good. slyphon: he's the one person who's had a non-trivial use case :) indeed :D shit, i took some notes from our conversation add multiple exception types # single failure per test case # list of all errors between starting test and failing tests # # intermittent the intermittent part was what i mentioned before about "test for clean reactor state" It's gonna take me a moment to parse this which consisted of a test leaving behind timed calls, open sockets, and installed signal handlers in fact, can you elaborate on the whole thing? yes indeed this was a 20 minute conversation and warner did most of the high-speed talking :) trial needs to keep track of all failures that occur during a test run in a list was the idea I fully agree. me too or an iterator ;) :) "add multiple exception types" i kinda forget exactly what he meant