Circle Judging System - Automated

Working on a footbag-related project? Ask for help, get feedback/advice here.
Post Reply
jay7
Flower Child
Posts: 1616
Joined: 22 Jun 2006 11:31
Location: Europe

Circle Judging System - Automated

Post by jay7 » 08 Jan 2017 04:16

Greetings Modified.

Background:
Once upon a time, I disagreed with some circle contest results. Having experienced many issues with judging in the past, I decided to try and break down the results as objectively as I could and throw it into a spreadsheet. My thought was that potentially a very high quality, very transparent, and very reliable system could be put in place which would encourage a very entertaining type of competition.

The first attempt can be seen here:

https://docs.google.com/spreadsheets/d/ ... 1665360758

The reception was mostly good, where the large exception stemmed from concerns that it under-minded judges and their efforts.

The Problem to Solve:
I believe that players deserve a reliable, high quality judging system which completely aligns with judging criteria, in every single competition. I believe that today this does not exist the majority of the time.

Variables
  • Players often do and can take competition extremely seriously. Training as hard as they can, making life sacrifices, etc.
    -A large amount of players and judges alike do not actually understand or agree with the judging criteria/system for circle. (1/3, as per my last public survey)
    -When I did a large scale public survey, asking if an online judging system (that was open to literally anyone to judge) was of interest, over 1/2 were in favour.
    -A system could be put into place which does not undermine judging efforts, as it would be done in place of judges.
    -The amount of tournaments I went to where judges where literally selected because they were hungover and unable to play, completely underqualified, or literally had no desire to judge in the first place was staggering.
    -It is a lose-lose situation for all when there is debatable, or obvious judging problems. The winner does not fully enjoy his win, the "real" winner does not have the title (or prize money, etc), the judges feel criticism from all, etc.
    -Internet speeds, camera availability and quality are reliable enough in arguably all of the places tournaments are held today to implement an online system.
    -Finding someone to judge shred 30 is often like pulling teeth.
    -Ties are annoying, and there are large discrepancies on how to solve them fairly.
Having a judging system which players are confident in, does not expose itself to require hungover, unqualified, or disinterested judging, will give the same results every time, reduces the chances of a tie to near zero, and gives the community results in a reasonably fast turn-around time I think would be accepted by the community.

The System:
The actual tool to implement a system to me would be secondary. (Spreadsheet, bespoke program, mobile app, whatever). My idea would roughly be as follows

Inputs:
Player Names with all associated trick names in order. This would also require a person to film, and film well (close enough, with nothing in the way between the lens and the players)

Criteria for Variety:
-All tricks by all players would be pooled, and then sorted and ranked by uniqueness to give them a score.
-Tricks which were never repeated by any other player
-unique up-time and unique downtimes (also pooled and ranked, similar to the tricks)
-Individual player's number of unique tricks
-Using both sides

Criteria for Density:
-Tricks would have a density ranking
-Links would have a density ranking
-The number of densely linked tricks in a row

By clicking the link at the top, you will see most of how I implemented a first draft of this system.

The Execution
This system would require that someone film and review the footage, and write down each string which each player did. From there, all the variables above would be calculated and ranked automatically and nearly instantly. This implies that someone is willing to review the footage and do this, but also that they would do this accurately and within an acceptable amount of time to receive the results.

There were 17 events in 2016 which had IFPA status, not all which even had a circle contest. While I suspect at least one person on site at the tournament could do this, online participants could also as well. For example, I would easily offer to do 5-10 tournaments per year with a turnaround time of 4 hours, assuming the tournament was done on a weekend (which they basically always are...)

The Demand
The system can be made, even if it is just me cramming formulas into a google spreadsheet. I think players would enjoy having a reliable, consistent judging system which is universal. I think that this system would promote a very unique style of play for variety, and a very dense style of play for density.

Unfortunately I did not think of this system based idea when I did my mass poll a few years ago, so I do not have data on the type of demand people would have for this. My suspicion would be (from similar questions and numbers) that over 50% would enjoy such a system... But my gut says that unless a huge majority really thought there was added value, coming to a conclusion and implementation stage would be unlikely, and a waste of time.

My question to you
Before I bother with a facebook poll, I'd like your input. Do you think this is worth pursuing? Why or why not?
Last edited by jay7 on 08 Jan 2017 10:39, edited 1 time in total.
Jay Boychuk

User avatar
F[uns]tylin' Eclectic
Post Master General
Posts: 4092
Joined: 05 Feb 2010 19:53
Location: Drumore, PA

Re: Circle Judging System

Post by F[uns]tylin' Eclectic » 08 Jan 2017 09:07

I think the world should know about this judging system, because people told me the current judging system failed me one time and that honestly made me not want to play footbag anymore, but then I just stated playing for fun and not competing. I might compete again if the judging was mathematically judged and everything was proven/nothing was based on a judge's hunch.
Nick Polini

Footbag is good for the SOLE

Funblog

"Yeah dude it's all mental. Then it's physical" ~Evan Gatesman

jay7
Flower Child
Posts: 1616
Joined: 22 Jun 2006 11:31
Location: Europe

Re: Circle Judging System

Post by jay7 » 08 Jan 2017 10:38

Just had an interesting chat with Matt Kemmer on this. As his wrist is now broken, he opted to video chat rather than write a response. For the sake of documentation, I'll write the key notes here...

Key findings:

1.) Variety would be much more reliable, and in the scope of what we currently enjoy if done with a numeric system. It matches what we as judges want with our brains and the rule book pretty well.

2.) Density on the other hand, does not match as well.

Key issues:

1.) For a reason that does not seem to be in the rulebook (although the Footbag.org/IFPA rulebook is not up at the moment, so I can't point any paragraph out :( ) Matt and I both agree this system would facilitate a boring style of play. Our example was:

a.) Blurry whirl x 15
b.) 15 unique fearless.

Trick Density: For individual trick density, blurry whirl has 5 adds = 5 components = 5 "density" points if you will. However, 15 unique fearless would by definition as well.

Link Density: Arguably the trick density for blurry whirl x 15 would be say "2" arbitrary not yet defined density points. For the unique fearless, a pdx-blender-ps whirl link would get say "1" point, but then a blurry drifter-blurry whirl might get "3" points. It could be very possible that they both equal out.

Number of tricks in a row: By definition, the 15 fearless and 15 bw's would be equal.

Again, for an indescribable reason Matt and I could not argue with the current rule book that they would be different either... But yet we both knew in our little hearts that we'd rather see the 15 unique fearless in density and somehow inherently feel that is more dense... Even though by definition it is wrong. In my opinion this is not just a fault with a calculated system, but also the rules as they stand today.

Either way, if this judging system promotes a less interesting style of play simply by exposing an already flawed rulebook, I'd rather work something else out to give us the competition that we do indeed want. We already have shred 30 for a calculated number mashing competition, and I do think we all would enjoy density more as it is played today than blurry whirl x 15

So, conclusions on density

1.) In order to maintain a competition style we enjoy and deploy a calculated density number crunching program, we might need a change in the rule book to give more depth to density which we could capture via formulas.
2.) At the moment, density is very ill defined in terms of numbers. ADDs are surprisingly good for trick density, but link density and that "I know it was dense when I saw it because tricks were one after another" gets tougher. Theoretically if this was figured out, no rule book change would be needed.

----

We also discussed the whole concept of humans vs machines judging. Human judging is interesting, because it is both various, as well as very limiting. As a whole if we all sort of judge in a way we agree, we might evolve naturally into judging in a way which promotes the sport growing in a way we want.

That being said, this is a natural process, and making any definitive changes in a rule book to reflect this, or to ensure the world is all evolving the same way is another story. Trying to convince people to change a rule book because of gut feelings and different opinions is not easily going to happen.

Where as, for comparison, a machine based program would never evolve by itself. If left alone you would be stuck with one system forever. However, assuming you had a machine program, it would be much easier to use numbers and facts to argue changes to the system as you would limit much of the "gut feeling" discussion.

-----

What we concluded is that the system, and people's understanding of the system today definitely has room for improvement. I'll be starting another topic on that.

Thanks for reading.
Jay Boychuk

User avatar
Tjuggles
Flower Child
Posts: 1746
Joined: 27 Mar 2005 20:03
Location: Wellington, NZ
Contact:

Re: Circle Judging System - Automated

Post by Tjuggles » 08 Jan 2017 12:45

I like a systematic approach to things, especially when it comes to footbag.

I feel like for an effective system to be in place, the requirements for circle need to be explicitly stated and/or agreed on. I tried to (quickly) find some concrete requirements, and the best I could do was the footbag wiki (as Jay mentioned many pages on footbag.org are down):
https://en.wikipedia.org/wiki/Freestyle_footbag#Circle_Competition wrote: Each player in the circle takes a turn with the bag, until they drop. The bag is then passed to the next player, who takes their turn, and so forth. There is a limit to the number of times the bag will go around the circle. For instance, there may be three difficulty rounds followed by three variety rounds, though organizers of different events may revise the structure of the format in different ways. Players are evaluated subjectively by a panel of judges.
Using this description it will be pretty difficult to create an automated system.
Jay wrote:...we both knew in our little hearts that we'd rather see the 15 unique fearless in density and somehow inherently feel that is more dense... Even though by definition it is wrong.
This is a good example of the importance of making clear what is meant by "dense". While 15 unique fearless is more appealing, blurry whirl x15 also has merit; being able to do 15 consecs of a 5-add trick, versus blurry whirl > osis for example, shows a strong level of control of the trick. Whether or not having a solid control over a trick is a requirement of "density" is another story though. Similarly, awarding points for a large amount of unique fearless seems to be more appropriate in the variety round, calling into question the importance of variety in the density rounds. The way that circle is typically employed in Japan combines the two rounds with the point being to hit a variety of tricks, with a range of adds, the higher the better. This is an interesting way to do circle, however it starts to resemble shred 30 more and more...

In short, I think that having specific, explicit requirements will make this awesome endeavor awesomer. I hope the above makes sense. Great idea!
TJ Boutorwick

"You can do anything" -Greg Nelson

boyle
Post Master General
Posts: 3146
Joined: 19 Sep 2007 23:42
Location: Canberra, Australia
Contact:

Re: Circle Judging System - Automated

Post by boyle » 09 Jan 2017 07:43

A quick thought on the blurry whirl discussion. ..I think if it was made that if you are repeating concepts, eg blurry whirl b2b, or even blurry whirl spinning whirl tomahawk, that this is not as difficult as changing concepts. This would do something about that discrepancy.

I think first though the numbers should do it, things like average add value, average link value, number of tricks, then take that further breakdown to make sure the results are correct.

Post Reply