[Cs5500] Fwd: [Cs4800] How does scoring work?

Wed Nov 9 10:14:30 EST 2011

Srinivas sent a good message to the undergraduate class. This is managing
software development in action! We have an angry group of users and
Srinivas found a good way to calm them.

Why are they angry? Because they really care about how our software works
because their grade will depend on it! And they found multiple bugs that
give them a bad grade although their avatar does perfect work.

This is not a good situation to be in. We use our users to debug our
software.
Our users want to learn about algorithms. Fortunately, our software is also
an algorithm: an avatar evaluation algorithm. And they see value in this
algorithm and help us to implement it correctly.

I made the mistake of not pushing you more to find bugs. I set up testing
projects and tournament projects in 3 playgrounds. I made the mistake that
I did not notice that there is a feature selection missing at the
playground design level.

You made the mistake of not testing seriously enough and of not
helping me to think more carefully about playground design.

We need to pool together to get this repaired.

-- Karl

---------- Forwarded message ----------
From: Srinivas N Jay <srinivasnjay at gmail.com>
Date: Wed, Nov 9, 2011 at 1:36 AM
Subject: Re: [Cs4800] How does scoring work?
To: CS4800 Mailing List <cs4800 at lists.ccs.neu.edu>
Cc: Karl Lieberherr <lieber at ccs.neu.edu>, Madhuvanthi Balasubramanian <
balasubramanian.m at husky.neu.edu>, Sandeep Upendra Kamath <
kamath.s at husky.neu.edu>, karan bhat <bhat.r.karan at gmail.com>, Ravishankar
Rajagopal <rajagopal.r at husky.neu.edu>

Hi Guys,

First thing I want to say is, take it easy! It is obvious that you wouldn't
be graded based on an erroneous system. So just relax.  The system is new
for us too and we are not familiar with entire code base. It was never been
tested with so many people participating in a tournament. So obviously
there will be some bugs.

I believe this is an awesome way to learn algorithms. So if you find any
bugs please let us know rather than being worried about your grades. We
together has to work and fix these bugs to make it a better system.

We are working continuously to improve the system. As you can see Tuesday's
tournament was much better than Monday's.

Thanks a lot for finding out bugs. We will fix them soon.

Best,
Srinivas

On Tue, Nov 8, 2011 at 11:26 PM, David Richards <dirich at ccs.neu.edu> wrote:

> To further support my theory that there's a bug:
>
> If you look in the "proceed" method of scg.protocolInterpreter.
> StrengtheningProtocolInterpreter
> you will see that it calls claim.getProtocol().strengthenP(originalClaim,
> claim)) as follows:
>
> > if (!claim.getProtocol().strengthenP(originalClaim, claim)) {
> >                       updateReputation(0, 1); // Alice wins
>
>
> If we look at the scg.protocol.ForAllExistsMin class we can see that it
> doesn't implement
> strengthenP(…) so it must inherit it from it's superclass which is
> scg.protocol.ForAllExists.
> The implementation of strengthenP(…) in that file is the following:
>
> >        // return true iff the strengthenedClaim is stronger than the old
> claim
> >       public boolean strengthenP(Claim oldClaim, Claim
> strengthenedClaim){
> >               return strengthenedClaim.getQuality() >
> oldClaim.getQuality();
> >       }
>
> Clearly this is wrong for the ForAllExistsMin protocol since strengthened
> claims have
> smaller qualities.
>
> -David Richards
>
> On Nov 8, 2011, at 11:13 PM, Greg I Kerr wrote:
>
> > After looking at the log, I agree with David. It's pretty clear that
> strengthening a proposal does not win any points (and that the server never
> asks for a solution to be provided). This is concerning as strengthening
> claims is a major part of the game.
> >
> > - Greg
> >
> > On Tue, Nov 8, 2011 at 10:59 PM, David Richards <dirich at ccs.neu.edu>
> wrote:
> > Could you explain how to defend my strengthening?  If the server were to
> ask for a solution I would provide it…
> > so clearly the server isn't asking me to defend the strengthening.
> >
> > -Dave
> >
> > On Nov 8, 2011, at 10:50 PM, srinivasnjay wrote:
> >
> > > I was with Matt when he replied you. Strengthening protocol requires
> you to defend your strengthened claims. If you fail to do so, opponent will
> again points.
> > >
> > > - N Jay
> > >
> > > On Nov 8, 2011, at 22:42, David Richards <dirich at ccs.neu.edu> wrote:
> > >
> > >> Again: how does the scoring work and why does strengthening a claim
> seem to lose you points?
> > >>
> > >> In this match:
> http://tank.ccs.neu.edu:7007/tournament_details?resource=smart_history&session=B0E44B907BCC44D0&tournament_id=43&history_file=usb2%20vs%20dirich11-08-11-22-18-54.txt
> > >> I successfully defended all my claims, refuted invalid claims, and
> strengthened poor claims, and
> > >> yet my opponent ended up with more points than I did…
> > >>
> > >> I don't think it's fair to grade us based on a buggy non-transparent
> scoring system.
> > >>
> > >> -David Richards
> > >> _______________________________________________
> > >> Cs4800 mailing list
> > >> Cs4800 at lists.ccs.neu.edu
> > >> https://lists.ccs.neu.edu/bin/listinfo/cs4800
> >
> >
> > _______________________________________________
> > Cs4800 mailing list
> > Cs4800 at lists.ccs.neu.edu
> > https://lists.ccs.neu.edu/bin/listinfo/cs4800
> >
>
>

-- 
***Race A Race Where You Are The Goal For Every Opponent! - *N Jay****

Srinivas *N Jay*
srinivasnjay at gmail.com
+1(617)-817-0913
-------------- next part --------------
HTML attachment scrubbed and removed