There's a very easy, very simple fix for the second part of this problem: approval voting. Here's my two-sentence description of approval voting:
Each voter can put a check next to as many candidates as they approve of, leaving disapproved-of candidates blank. The candidate(s) with the most votes win(s).
That's it. Ballots look the same. It's not complicated to explain. And approval voting lends itself to virtually no strategic voting (i.e., faking your preferences on the ballot to try to induce your desired outcome to happen).
In the top-two primary, everything would work the same, except that voters wouldn't get one choice; they could vote for as many candidates as they like. ABBAs could vote for every ABBA candidate, and BeeGees could vote for every BeeGee candidate. Or ABBAs could vote for most ABBA candidates and some centrist BeeGees. Or a centrist could vote for some centrist ABBAs and some centrist BeeGees. Let's imagine a scenario in which a district is 51% ABBA voter and 49% BeeGees. Let's say each side nominates three candidates: A, B, and C for the ABBAs, and X, Y, and Z for the BeeGees.
In a hyper-partisan environment, 100% of ABBA voters approve of A, B, and C, and 0% approve of X, Y, and Z; vice versa for all the BeeGee voters. Because there are slightly more ABBA voters (51-to-49), therefore the top two candidates will always be some combination of A, B, and C (more on this in a second). The BeeGee would be locked out. However, this lock-out has nothing to do with how many candidates the BeeGees nominated. It would have happened whether they nominated two, three, four, or a hundred candidates. The lock-out is the result of the hyper-partisan environment, not the number of candidates nominated splitting the vote. No matter how many candidates the BeeGees nominate, they all get 49% of the vote and fall short of the general ballot.
Let's go back to that issue of which two of A, B, and C make the general ballot in the hyper-partisan environment. If it is truly a tie--all three got exactly the same number of votes--then some tie-breaking mechanism would have to be employed. They could draw straws, or the ABBA party chairperson could decide because all of the candidates are its own. But this three-way tie seems fairly unlikely. Would primary ABBA voters be so united in support of all three candidates that they give 100% approval to each? I guess this is an argument that such hyper-partisanship seems unlikely; it's more likely A gets 95% approval from ABBA voters, B gets 90%, and C gets 70% or some such split. If there aren't at least two of ABBA's candidates that get 100% approval from ABBA voters, it opens up the possibility that a BeeGees candidate can make it to the general election.
Furthermore, it seems unlikely that there are no unaffiliated voters exist and that none of the partisans ever cross-over. Even in today's highly partisan environment, people can and do split tickets, switch parties, and cross-over. (I reserve hyper-partisanship to mean zero behavior exists.) Some ABBA voters might approve of A, B, and Z. Some centrist voters might approve of B and X. Having unaffiliated voters and cross-over votes doesn't guarantee ABBA candidates or BeeGee candidates won't be locked out--but it does make it less likely. Even in a highly partisan environment, candidates with cross-over appeal might be at somewhat of a practical advantage. Winning 99% of 51% of the total votes (almost all ABBA voters) is 50.49% of the total; winning 90% of 51% of the total votes (most ABBA voters) and 10% of 49% of the total votes (a smattering of BeeGee voters) is 50.8% of the total votes. As a real-world matter, I think it's harder to get complete party unity behind a candidate (that is, 99%) than it is to attract a couple percentage points from the other party. Maybe I'm wrong, but look at this graph of presidential approval ratings. Of the twelve presidents of the modern era, nine were able to pick up more support from the opposition party than they lost from their own. Only two were better able at holding their own party together than at attracting opponents (Barack and the Donald). The twelfth case, Jimmy, did dismally with both parties. The average trend is that pulling in opposition is easier than preventing any defections.
In an approval voting scheme for a top-two primary, it's possible that a party gets locked out, but the cause would not be how few or many candidates they nominate. A party would get locked out if (1) the other party had more voters and had at least two candidates they completely unified behind or if (2) the other party had at least two candidates with cross-over appeal. Scenario 1 seems unlikely as an empirical matter; scenario 2 seems like it fulfills the exact purpose of top-two primaries of selecting the two best candidates overall--who just happen to be from the same party, but expanded their support beyond it.
By the way, the approval voting scheme makes sense for regular primaries too, or any time voters have more than two choices they need to whittle down. I use it in meetings whenever we have more than two options to consider to find out where the general consensus lays.
There's not much an individual voter can do to vote strategically. Some might consider giving an approval vote to the candidate I find least objectionable from the party I disagree with, if I think it's inevitable that the other party will get one candidate in. (In other words, it's inevitable, so chose the weakest opposition.) This seems an unlikely scenario, however, and a risky strategy. When do I know the other party is nearly guaranteed a spot in the general election? Only when my party has nominated only one candidate or only one strong candidate (so, unlikely). And it's a risky strategy: my approval vote for the weakest opposition might be enough to push TWO of the opposition party's candidates into the general election, excluding my candidate entirely. Let's say the standings look like this, including my vote for my candidate but not yet voting for the opposition candidate:
My candidate - 51%
Opposition candidate I hate - 49%
Opposition candidate I would prefer - 49%
In this case, I do get to decide which opposition candidate we face in the general election. But the scenario could just as easily be this:
My candidate - 49%
Opposition candidate I hate - 51%
Opposition candidate I would prefer - 49%
In this case, the opposition candidate I hate is inevitable. My vote for the opposition candidate I prefer knocks out my candidate. (At least before, my candidate might have won the tie-breaker for second place.)
Who can say which scenario is likely to happen in a close race before the voting is done? This strategy is incredibly risky when everyone votes before votes are tallied.