EDIT: Final Results: 2nd Experiment results: AND shuttles are absolutely bugged

edited April 2018 in The Bridge
Summary: By looking at shuttles that have 1 crew slot, we can eliminate variables and more closely study the effects of the AND node. By doing this, 3 different testers, including @(FF) This Sisko1, @[TFA]Múspell and 1 other found nearly identical results where the 2nd skill of and nodes grossly under perform expected results. We can then suggest that you should treat AND nodes as 1st + 25% * 2nd, not Highest + 25% Lowest as was the prior communication.

Edit: Example for clarity:

in a DIPLOMACY AND COMMAND node, Cmdr Keras (890 CMD, 617 DIP) should score as:

890 + 25% * 617 = 1044 (highest of the 2 skills plus 25% of the second). This is the stat used to calculate the displayed chance of success.

Instead, when the game is calculating if you pass the shuttle, it uses the following formula:

617+ 25% * 890 = 839 (First skill (DIP) + 25% the second skill (CMD)).


Background: After investigating our findings from the last event and proposing the AND node failure mode, @Peachtree Rex suggested utilizing the federation shuttles to compare single skill shuttles against the single crew AND shuttles where there should be a huge difference in the data if AND nodes are being mis-calculated.

Experimental Design: To achieve this we looked at "Rescue Citizen Captives" DIP AND SEC and "Investigate corruption" DIP AND CMD. We needed to test for the hypothesis that the AND node is calculated 1st 2nd while controlling for potential other bugs in the mission or bugs in calculation. To do this:

3 testers ran the AND missions with only Skills in the 2nd of the AND skills (security and command) to see if our results far underachieved the displayed chance of success

1 tester ran the same AND missions with only a skill in the 1st spot (diplomacy and diplomacy) to see if the shuttles matched the chance of success

3 testers ran single skill missions to see if those matched the chance of success.

Results: The missions with crew in the 2nd AND slot drastically under performed expectations, showing a result of 13% compared to the expected 71%.

The crew in the 1st skill of the AND missions were successful 16/16 runs. The small sample size could account for the abnormally high success rate, although we cannot rule out that there is actually some error that is boosting the success of the 1st skill AND missions, although that was out of the scope of this experiment. We can however rule out that the shuttles are drastically under performing the displayed success rate

The crew in the single mission shuttles came back right in line with expected success rates across the 3 testers.

kvduv8hkaqkn.png


Conclusion: There is now a preponderance of data with other variables ruled out that the AND missions are clearly calculating as 1st + 2nd, not Highest + Lowest.

Additional notes: The expected success rates was an estimate from the testers, since the high success or failure rates would move the number and we all had slightly different crew, there was some variability, but in general all the shuttles had displayed ranges in the 68-75 range, sufficient for the scope of this trial.


ORIGINAL POST:
All, a team of 4 people led by @(FF) This Sisko1 have been comparing single slot AND shuttles with regular single slot shuttles. We don't have all the final data in, but with a faction event starting tomorrow we wanted to share the preliminary results.

While single skill missions are performing right around the displayed % of ~70%, the AND shuttles using only the 2nd skill have bombed at less than %10 success rate. We have a few more rounds of data to collect and some number crunching, but with an N for each group of between 50-100, there is just no doubt any more that the AND shuttles are not performing as intended.

It is our recommendation that you only use AND slots where you utilize the first skill in this next event. We will publish our final combined results soon.
Task Force April now has openings! Lvl 130 Starbase! Task Force April is Looking for You!
«1345

Comments

  • Thanks for wasting a bunch of shuttles and transmissions for the benefit of the community.

    Eh, I pretty much fail 2/4 every time anyway and I am sure Troi and Picard are going to need federation stuff, so probably wasted 100 merits. Besides, I love a good experiment
    Task Force April now has openings! Lvl 130 Starbase! Task Force April is Looking for You!
  • IvanstoneIvanstone ✭✭✭✭✭
    How would this interact with skill boosts?

    There was one mission last weekend called Adjusting the Emitters. It was:
    ENG AND CMD
    ENG AND SEC
    ENG AND DIP
    I filled each seat with a crew using the second skill (ex T'Mir, KalSpock, Sybok) and then used a 3* ENG boost even though none of these crew have ENG. Typically I would have a 90-95% displayed success chance. I didn't notice any unusual failures all weekend.
    VIP 9 - 165 Crew Slots - 412 Immortals
  • Peachtree RexPeachtree Rex ✭✭✭✭✭
    Ivanstone wrote: »
    How would this interact with skill boosts?

    There was one mission last weekend called Adjusting the Emitters. It was:
    ENG AND CMD
    ENG AND SEC
    ENG AND DIP
    I filled each seat with a crew using the second skill (ex T'Mir, KalSpock, Sybok) and then used a 3* ENG boost even though none of these crew have ENG. Typically I would have a 90-95% displayed success chance. I didn't notice any unusual failures all weekend.

    Unknown, at this point. Boost usage, to the best of my knowledge, was not tested. Boosts also add such a small % to overall success that it would require a LOT more shuttles to definitively say anything. Since the exact mechanics of the "AND" bug is not really known either, it's hard to say what might be broken with Boosts and what might be broken with the "AND" slots.

    To test boosts, you'd probably want to use low-power crew (so the boost has a bigger impact on shuttle success). But we're still talking a modest impact to expected success with a number of different permutations:

    AND Character Skill 1 with boost of Skill 1
    AND Character Skill 1 with boost of Skill 2
    AND Character Skill 2 with boost of Skill 1
    AND Character Skill 2 with boost of Skill 2
  • Most of us did, I mean you can still get lucky, but theoretically instead of the following score:

    (900+250)*2 = 2400 points per slot

    It was calculating as:
    (900/4+250)*2 = 950 per slot

    Those of us tracking it were getting results on that mission approaching 50%
    Task Force April now has openings! Lvl 130 Starbase! Task Force April is Looking for You!
  • Secret JourneySecret Journey ✭✭✭✭
    I’ve been paying closer attention to my shuttle results rather than mindless turning and burning which is what DB inspires in me and it’s absolutely true. The second skill is whacked!
    DB = Climbing up an endless wall...
  • Dirk GundersonDirk Gunderson ✭✭✭✭✭
    With this development, I’ve been paying more attention to who I choose for slots. For example, if I see a DIP/SEC slot and a SEC/DIP slot, I will put Kortar in the second and Ambassador Shras in the first rather than the other way around. Although I haven’t taken the time to track successes and failures, whether in an event or out, I seem to be doing better than I was previously.

    If this is actually a (putridly awfully described) feature rather than a bug, it does add a lot more strategy to choosing who goes on which shuttles. And, by extension, it puts even more strain on crew slots. Instead of having one crew who can do both permutations of a particular AND slot, you would theoretically need two separate crew to maximize your success rate unless both crew had relatively close primary and secondary skills (like how T’Kuvma or Mirror Picard have COM and SEC skills that are only a hundred or so points apart, while Robin Hood has a several hundred point disparity between his COM and SEC skills).
  • Dirk, if it was the intended action, then 1) Shan wouldn't have opened a bug ticket to investigate and 2) the displayed success rates would correspond with it. The displayed success rate is highest plus 25% of the lowest which was the communicated functionality of AND shuttles
    Task Force April now has openings! Lvl 130 Starbase! Task Force April is Looking for You!
  • Dirk GundersonDirk Gunderson ✭✭✭✭✭
    Dirk, if it was the intended action, then 1) Shan wouldn't have opened a bug ticket to investigate and 2) the displayed success rates would correspond with it. The displayed success rate is highest plus 25% of the lowest which was the communicated functionality of AND shuttles

    I leave open the possibility that DB has more than one person on the design team and that they communicate poorly with programmers, writers, and Shan. It would fit what many suspect already, no?
  • [S14] Elynduil[S14] Elynduil ✭✭✭✭
    How many different platforms were you using? It has occurred to me that the projected success rate could be displaying correctly in some client versions, and not others, which could account for the wide disparity in opinions on this subject... Or could a specific client perhaps be sending the data through to the server incorrectly?

    On iOS here, and I haven't yet noticed anything that resembles an anomalous pattern since I started keeping a closer eye on it...
  • (FF) This Sisko1(FF) This Sisko1 ✭✭✭✭
    edited April 2018
    TBH I'm just more mad. I knew something wasn't right when the AND formula changed and the shown percentage was higher but the actual success was still the same for me as the pre change results. I've tracked over a 1000 shuttles (in Excel, Line, and on paper). Got into fights on the formulas and some people didn't think I could have such a consistent shown percentage in the high 90s.

    Now to figure out that in a 3/4 crew mission, the AND setup was what caused the false fails. Totally makes sense that if the shown was 95%, but the actual was at 85%, then the difference was the slot or two that I used the wrong crew (1st vs 2nd skill). Had they explained to us that the first skill shown had to be the primary, I would have always setup my missions that way.

    Anyways, so far I see 5/40 for the second skill missions. While I am personally at a 20% success rate, others haven't passed a single one. Shoutout to GTMET for getting the ball rolling and not only sacrificing the time but also lowering his Federation difficulty because of the fails lol.

    I usually am 50/50 on weekly shuttles so this didn't hurt me and I have a ton of film as a result.
  • WebberoniWebberoni ✭✭✭✭✭
    Ivanstone wrote: »
    How would this interact with skill boosts?

    There was one mission last weekend called Adjusting the Emitters. It was:
    ENG AND CMD
    ENG AND SEC
    ENG AND DIP
    I filled each seat with a crew using the second skill (ex T'Mir, KalSpock, Sybok) and then used a 3* ENG boost even though none of these crew have ENG. Typically I would have a 90-95% displayed success chance. I didn't notice any unusual failures all weekend.

    Strange. I did the exact same thing and failed that mission 5 times in a row (4 as I kept repeating 2250 points, and the last time at 2750 points), and wound up avoiding it for the rest of the weekend. My average success rate was much better without risking that mission.
  • PallidynePallidyne ✭✭✭✭✭
    So it seems that when the "AND"s were changed due to forum outcry when it was changed on an event, they were just changed on the display percentage. That's pretty crappy if you think about it.
  • Pallidyne wrote: »
    So it seems that when the "AND"s were changed due to forum outcry when it was changed on an event, they were just changed on the display percentage. That's pretty crappy if you think about it.

    They were changed because they used to be an average, not highest plus 25% lowest. The change displays correctly, and I am pretty sure there is code to calculate it both ways, but there is clearly some bug that always uses the first case, not the highest case.
    Task Force April now has openings! Lvl 130 Starbase! Task Force April is Looking for You!
  • Webberoni wrote: »
    Ivanstone wrote: »
    How would this interact with skill boosts?

    There was one mission last weekend called Adjusting the Emitters. It was:
    ENG AND CMD
    ENG AND SEC
    ENG AND DIP
    I filled each seat with a crew using the second skill (ex T'Mir, KalSpock, Sybok) and then used a 3* ENG boost even though none of these crew have ENG. Typically I would have a 90-95% displayed success chance. I didn't notice any unusual failures all weekend.

    Strange. I did the exact same thing and failed that mission 5 times in a row (4 as I kept repeating 2250 points, and the last time at 2750 points), and wound up avoiding it for the rest of the weekend. My average success rate was much better without risking that mission.

    Same results here. This specific shuttle failed 90% of time despite it shows 91% success chance.
  • WebberoniWebberoni ✭✭✭✭✭
    They were changed because they used to be an average, not highest plus 25% lowest. The change displays correctly, and I am pretty sure there is code to calculate it both ways, but there is clearly some bug that always uses the first case, not the highest case.

    Would I be right in assuming that even if a crew member has both skills required on an "and" shuttle, they better be most proficient in the first skill listed, in order to approach the posted success rate?
  • So this is when the AND shuttle change was announced:
    https://forums.disruptorbeam.com/stt/viewthread/63133
    This was the info we got and have relied on for the last year
    3lsmc91r206l.png
    00339kmricvp.png


    By June of last year I had gone bananas with all the fails and finally started to ask:
    f681288kh8vk.png


    Grr if only we thought the of the issue sooner.....
  • I'm stealing this from someone else but IF the order is NOT supposed to matter, why would you have this?...
    wvlmspsal295.png
    Member of Rise of the Phoenix.
  • That is a heck of a question! Never thought of that. I am sure someone figured out this issue a long time ago and is mad that we finally caught up. I'm sorry but just couldn't lose anymore people to rage quitting lol.
  • MagisseMagisse ✭✭✭✭✭
    edited April 2018
    So should we still be trying to fill both skill requirements, just make sure that the first skill has our higher-power crew?
    Like for a "Dip And Com" node, you're showing that you shouldn't put someone with just high Com. But are you still better-off with a crew with high Dip & Com, as opposed to just a high Dip? Or does the second part not really matter at all?
    Thanks for all your work on this!
  • It apparently always calculates as

    First Skill + 25%* second skill

    So you are always better using 2 skills, but if your crew is 900 command 400 security but the slot is security AND command then you are better off putting a 700 security crew in the slot
    Task Force April now has openings! Lvl 130 Starbase! Task Force April is Looking for You!
  • Your updated post is outstanding! Great work, great reason, great explanation!
  • I'm stealing this from someone else but IF the order is NOT supposed to matter, why would you have this?...
    wvlmspsal295.png

    I always just chocked it up to programmers randomly assigning traits......
    Task Force April now has openings! Lvl 130 Starbase! Task Force April is Looking for You!
  •  Captain Captain ✭✭✭✭✭
    I'm stealing this from someone else but IF the order is NOT supposed to matter, why would you have this?...
    wvlmspsal295.png

    I always just chocked it up to programmers randomly assigning traits......

    Someone was just having a fun time at the dartboard.
  • *ST* Donatra*ST* Donatra ✭✭✭✭
    So maybe the order has always been supposed to matter and then it wouldn't be a bug, not working as intended? What about the OR shuttles and boosties from the opposite skill working just like the actual skill?
    "Everything about the Jem'Hadar is lethal!" - Eris (ST-DS9 Episode 2x26 "The Jem'Hadar")
  • I take it, for example, you put a single skill crew ,like Captain Scott, in a slot that is listed (SCI AND ENG), you will have that default (little value) + (his ENG*0.25), and we should see a minor addition to the overall percentage?

    Has anyone ever asked why the points needed for a 4000VP shuttle (or any) isn't shown?
  • Peachtree RexPeachtree Rex ✭✭✭✭✭
    So maybe the order has always been supposed to matter and then it wouldn't be a bug, not working as intended? What about the OR shuttles and boosties from the opposite skill working just like the actual skill?

    If the order is supposed to matter, then there is a bug with how the expected pass rate as well as sorting the ideal crew for the slot.

    There is a discrepancy between the client and server. They cannot both be right.
  • (FF) This Sisko1(FF) This Sisko1 ✭✭✭✭
    edited April 2018
    SSR is full of some smart cookies led by what I believe to be the finest Admiral in the game; you guys should legitimately consider yourselves lucky to be apart of it and if anyone has a chance to join, take it! I'm sure at some point the question was asked about why in the same mission is the order of the same skill different.

    I however push buttons because they light up and make sound. I don't stop to ask why or even notice a small key idea. I just tappy tap. I hate expedition events with a passion but I sure do well in them because I know how to mindlessly tap lol.

    However I have a lovable personality which I guess must be addictive to every great member in FF. Which is why I love our 100 members :)
Sign In or Register to comment.