[Cs4800] How does scoring work?

David Richards dirich at ccs.neu.edu
Wed Nov 9 00:27:10 EST 2011 

I was just looking into that one.

I have a theory.  I think that what happens is a "game" goes on for a certain number of rounds.
Each round consists of a request to player one and then a request to player two.

Some actions in the game take place over several rounds.  For example, here's what happens
when two players agree.

Round 1) 5150 claims HSR(921, 16) with quality 0.010857763300760043
                  dirich agrees
Round 2) 5150 provides HSR(921, 16) as the instance for Bob to provide a solution to
		  dirich provides a solution to HSR(921, 16)

or it could look like the following (which is the case in the log below)

Round 1) dirich does some unrelated stuff….
                  5150 claims HSR(921, 16) with quality 0.010857763300760043
Round 2) dirich agrees
                  5150 provides HSR(921, 16) as the instance for Bob to provide a solution to
Round 3) dirich provides a solution to HSR(921, 16)
                  5150 does some unrelated stuff….

The problem is that the end of the game doesn't necessarily line up nicely with the end of a given
back and forth.  So looking at the example just above and the log below, it looks like the game ended
after round 2, so my avatar never got the chance to provide a solution to the claim.

The same thing happens sometimes with refutations, and when strengthenings are working I'm guessing
it will happen there too.

The good news is that as far as I can tell these don't negatively impact anyone's score.  The bad news is
that it means you could lose out on some points you could potential get during a refutation or a strengthening.

-Dave

On Nov 9, 2011, at 12:08 AM, Greg I Kerr wrote:

> Also, does anyone have insight as to why these entries from the log of David (dirich) versus Casey and I (5150) where we agree do not list solutions or award points:
> 
> claim hsr.HSRInstanceSet {{ HSR(921,16) }} scg.protocol.ForAllExistsMin {{ }} 0.010857763300760043 1.0 
> proposer {{ 5150 }} 
> opposer {{ dirich }} 
> action agree 
> responses provider {{ 5150 }} pr provide hsr.HSRInstance {{ HSR(921,16) }} 
> winner {{ }} 
> pointsWon 0.0
> 
> claim hsr.HSRInstanceSet {{ HSR(861,3) }} scg.protocol.ForAllExistsMin {{ }} 0.020905923344947737 1.0 
> proposer {{ 5150 }} 
> opposer {{ dirich }} 
> action agree 
> responses provider {{ 5150 }} pr provide hsr.HSRInstance {{ HSR(861,3) }} 
> winner {{ }} 
> pointsWon 0.0
> 
> claim hsr.HSRInstanceSet {{ HSR(845,9) }} scg.protocol.ForAllExistsMin {{ }} 0.011834319526627219 1.0 
> proposer {{ 5150 }} 
> opposer {{ dirich }} 
> action agree 
> responses provider {{ 5150 }} pr provide hsr.HSRInstance {{ HSR(845,9) }} 
> winner {{ }} 
> pointsWon 0.0
> 
> - Greg
> 
> On Tue, Nov 8, 2011 at 11:48 PM, Karl Lieberherr <lieber at ccs.neu.edu> wrote:
> Hi David:
> 
> you have found exactly the right spot where things go wrong. Thank you for
> finding the root cause for the wrong behavior. I apologize for all those bugs
> that come out now.
> 
> It is not for lack of testing: we had testing projects; we implemented three playgrounds
> and run tournaments in them with avatars written by the graduate students. But nobody noticed this behavior. We even had a teaching avatar
> in those playgrounds that was supposed to win always against the baby avatar and that worked too.
> 
> The graduate class needs to repair this which means we cannot have the counting tournament on
> Wednesday night.
> 
> Hold on to your avatars and we will inform the class soon after we have made a plan to repair the bugs.
> Get your solve function into good shape in the mean-time.
> 
> -- Karl
> 
> 
> On Tue, Nov 8, 2011 at 11:26 PM, David Richards <dirich at ccs.neu.edu> wrote:
> To further support my theory that there's a bug:
> 
> If you look in the "proceed" method of scg.protocolInterpreter. StrengtheningProtocolInterpreter
> you will see that it calls claim.getProtocol().strengthenP(originalClaim, claim)) as follows:
> 
> > if (!claim.getProtocol().strengthenP(originalClaim, claim)) {
> >                       updateReputation(0, 1); // Alice wins
> 
> 
> If we look at the scg.protocol.ForAllExistsMin class we can see that it doesn't implement
> strengthenP(…) so it must inherit it from it's superclass which is scg.protocol.ForAllExists.
> The implementation of strengthenP(…) in that file is the following:
> 
> >        // return true iff the strengthenedClaim is stronger than the old claim
> >       public boolean strengthenP(Claim oldClaim, Claim strengthenedClaim){
> >               return strengthenedClaim.getQuality() > oldClaim.getQuality();
> >       }
> 
> Clearly this is wrong for the ForAllExistsMin protocol since strengthened claims have
> smaller qualities.
> 
> -David Richards
> 
> On Nov 8, 2011, at 11:13 PM, Greg I Kerr wrote:
> 
> > After looking at the log, I agree with David. It's pretty clear that strengthening a proposal does not win any points (and that the server never asks for a solution to be provided). This is concerning as strengthening claims is a major part of the game.
> >
> > - Greg
> >
> > On Tue, Nov 8, 2011 at 10:59 PM, David Richards <dirich at ccs.neu.edu> wrote:
> > Could you explain how to defend my strengthening?  If the server were to ask for a solution I would provide it…
> > so clearly the server isn't asking me to defend the strengthening.
> >
> > -Dave
> >
> > On Nov 8, 2011, at 10:50 PM, srinivasnjay wrote:
> >
> > > I was with Matt when he replied you. Strengthening protocol requires you to defend your strengthened claims. If you fail to do so, opponent will again points.
> > >
> > > - N Jay
> > >
> > > On Nov 8, 2011, at 22:42, David Richards <dirich at ccs.neu.edu> wrote:
> > >
> > >> Again: how does the scoring work and why does strengthening a claim seem to lose you points?
> > >>
> > >> In this match: http://tank.ccs.neu.edu:7007/tournament_details?resource=smart_history&session=B0E44B907BCC44D0&tournament_id=43&history_file=usb2%20vs%20dirich11-08-11-22-18-54.txt
> > >> I successfully defended all my claims, refuted invalid claims, and strengthened poor claims, and
> > >> yet my opponent ended up with more points than I did…
> > >>
> > >> I don't think it's fair to grade us based on a buggy non-transparent scoring system.
> > >>
> > >> -David Richards
> > >> _______________________________________________
> > >> Cs4800 mailing list
> > >> Cs4800 at lists.ccs.neu.edu
> > >> https://lists.ccs.neu.edu/bin/listinfo/cs4800
> >
> >
> > _______________________________________________
> > Cs4800 mailing list
> > Cs4800 at lists.ccs.neu.edu
> > https://lists.ccs.neu.edu/bin/listinfo/cs4800
> >
> 
> 
> _______________________________________________
> Cs4800 mailing list
> Cs4800 at lists.ccs.neu.edu
> https://lists.ccs.neu.edu/bin/listinfo/cs4800
> 
> 
> _______________________________________________
> Cs4800 mailing list
> Cs4800 at lists.ccs.neu.edu
> https://lists.ccs.neu.edu/bin/listinfo/cs4800
> 
>

More information about the Cs4800 mailing list