TCO22 Stage 2 / 3 boundary
In the announcement email of the new Capacitated Vehicle Routing match, the schedule mentioned this match (and also the upcoming Real Time Cognitive State one) will both be part of Stage 3. In the past matches starting in December were still considered part of Stage 2. I can somewhat understand putting the two new ones into Stage 3 as they were both announced quite late (I never heard of the capacitated vehicle one until I got the email that it started), but it would be nice to have an official answer here.
Thanks.
Comments
It would be nice to have an answer, Especially the Vehicle Routing match is already over.
In addition, we are still waiting on the final result for MM130, stage 2 is about to end, or I think it is, it would be nice to know quickly either if some addition challenge are counting towards stage 2, or what are the final results to know who qualified.
Thanks
@vdave @sullyper checking on this - will get back to you
any updates?
also with stage 3 starting, MM132 starting soon, it would be nice to know who qualified from stage 2, this requires both MM130 and MM131 to have final official results, and to have a decision on which sponsored challenges were/are part of state 2.
And one more question on this topic: Does the "Cognitive State Determination Marathon Match" count towards the TCO'22 Marathon track? The name suggests it does, but in the "Submission Review" system it shows up as a "Cd" ("Code") competition, not as a "MM" one.
I guess Capacitated Vehicle Routing match belongs to stage 3 (as it was menionted in emails) since Stage 2 was already announced (with MM130 and MM131) when it was added.
Of course it is really bad decision to add a match exactly when another match is starting and even worse to make it part of TCO, but even Topcoder admins wouldn't add it to the stage that was already finishing.
The only source of truth I could find was the announcement email for Capacitated Vehicle Routing Part 3, which contained the following section:
. Based on that both sponsored matches belongs to Stage 3 (which is in contrast to how stage picking worked in previous years, e.g. for TCO20, the Mortgage Pooling marathon which started in December 2019 and ended in January 2020 counted towards the Oct - Dec 2019 stage). With the Cognitive State one still running and MM132 starting in 10 days, it would be nice to have official answers here.
The rules are pretty clear regarding how matches are allocated to stages: it's based on the date when the submissions open: "Only Tournament-eligible Marathon Competitions in which submission has opened during the four (4) Stages shall be included in determining points for that Stage. Note: that for Stage 4, matches in which submission opens during Stage 4 and ends after Stage 4 won’t be included in determining points for Stage 4."
Based on the published rules, the Capacitated Vehicle Routing match and the Cognitive State Determination match should count for Stage 2. It's a bit sad to just break the published rules - especially the Capacitated Vehicle Routing match makes no sense to me to be part of Stage 3, given that submissions both opened and closed during Stage 2. It doesn't matter when it was announced - in fact, there are easy arguments to be made that it shouldn't be part of any stage (it was announced too late for Stage 2 and too early for Stage 3 - presumably competitors planning to participate in Stage 3 weren't expecting a Stage 3 match taking place during Stage 2). So maybe the right thing to do is to not award any TCO points to the Capacitated Vehicle Routing match (that would also not be fair to competitors who participated hoping to get such TCO points).
I'm afraid at this point there cannot be any fair decision.
Making CVR part of Stage 2 is unfair for people who did not participate assuming it's in Stage 3 (from multiple Topcoder emails) and thought they already have enough points for TCO22.
Making it part of Stage 3 is indeed violating the rules, though given it was announced for Stage 3 multiple times (in the announcement emails of both MM131 and CVP), I feel this is the least bad decision.
Not giving TCO points for it is unfair for everyone who spent lots of time participating in it, especially that it was scheduled through the entire Christmas weekend.
My apology for the response delay. Over the holidays it's always a bit wonky.
Per the TCO22 Rules, the matches that start in Stage 2 are part of Stage 2. I missed the typo in the email so my apologies for the confusion. I'm working with a new team on the process of MMs going forward. Less mishaps and more consistency/improvements. I really appreciate your patience. You guys are so important to us. I'm looking forward to "housekeeping" this month and getting it right going forward.
Thanks again for your patience!
Thanks for the official answer, Jessie. Just to be crystal clear, this means that both the Capacitated Vehicle Routing 3 (which ended in December) and the currently running NASA Cognitive State Predictions (started in December, but still ongoing for ~2 weeks) will count towards Stage 2, is that correct?
Correct.
Thanks @jmpld40, could we also have the final result of MM130 and MM131 as well as the leaderboard for stage 2 updated with the results of the first 3 challenges.
Yes, the team is actively working on it. Hope to have an update from them soon.
@jmpld40 Sorry to bug you again with that, but any update? MM132 is going to start soon, so it would be nice to have a clearer picture of what happened in stage 2 (eventhough one DS MM is still running for stage 2, which is also quite unfortunate, the official results of the other 3 + the current leaderboard of the last DS MM running can provide some overview).
In general having the final result of the previous MM before the next one starts should be nice, and it would be nice to not have 2 MMs running in the same time.
ping
@jmpld40: Any update on the TCO'22 Marathon Stage 2 Leaderboard? It's still empty, although 3 out of 4 competitions have long been completed (with final results available).
By the way, I computed the TCO points for Stage 2 myself here, if that helps in any way (although I've done this manually, so I can't guarantee I haven't made any mistakes). If anything, it shows that it's mathematically impossible for @sullyper to not qualify from Stage 2 (congrats!). The other spots are still up for grabs, though.
Should be up today, Tuesday. Apologies for the delay. Seems a step was missed and it was unclear. But now we are all sorted out!
@jmpld40 The Marathon stage 2 leaderboard is still empty.
Tuesday? Like next week?
Tuesday as in Thursday
Sorry guys, truly am. Trust me we are now on top of it
Hi @jmpld40,
I now see that the Stage 2 Marathon leaderboard is not empty, which is great. But the points are wrong. For one, I see @RafbillFr with 240 points out of 2 challenges - this should be mathematically impossible, since the maximum number of points per challenge is 100.
After taking a closer look at the points, I think I reverse-engineered the issue: the TCO points for MM 130 are multiplied by 2. So in the published leaderboard:
The fix should be pretty simple.
I think there's something else as well, there are 16 people with the score of 1, which is quite impossible from 3 matches.
Actually this too many people with score 1 issue is also present on the Stage 1 leaderboard, I have the impression that a score of 1 is given to everyone with a valid solution, while based on the rules only people in the top 30 should receive TCO points.
The team that does this is currently out of office so I envision it will be after the weekend. Again, my apologies.
New MM problem: reverse engineer how the MM leaderboard was incorrectly calculated
It would be kind of cool if some MM gave you double points. Maybe that would encourage more participation
On a serious note, this leaderboard is clearly wrong and we will hopefully fix it soon.
If you want to double the points:
- Announce it at the beginning of the stage, that's it's fair for everyone
- Don't make it the first one, make it the last one, more fun like that
MM leaderboard for stage 2 is updated. Sincere apologies for all the delay.
@ufo_believer I think the duplicate points issue is indeed fixed, but there are still two problem with the leaderboard.
1. @iehn and @bowwowforeach have points, even though they both qualified for the TCO22 finals in Stage 1.
2. There are still lots of people with one point, with 3 matches, only up to 3 people could have ended up with 1 point. This issue is also there on the Stage 1 leaderboard (though not affecting the top 3 in either case).
Thanks for highlighting this. As Jessie mentioned, we're working with the new team and these issues will be fixed once we update the scores for Cognitive State Determination Marathon Match for STAGE 2.
I believe the Cognitive State marathon has final results now, it would be nice to fix the leaderboard (hopefully also for Stage 1 where it still contains some people with score 1 who never finished in the top 30).
My own calculations are here (but as I have personal interest, don't assume mine is correct): https://pastebin.pl/view/5290cfa4