It was a huge fad, but now people have seen it done before. Rock Band 4 and GH Live brought nothing notable to the table (both cut features rather than include new ones) and thus mass appeal was lost. Imagine if Nintendo tried to copy the Wii's motion control gimmick in 2017 for the Switch and did absolutely nothing else with it other than that, nobody's going to care if you don't innovate.
Plus out of the two, Rock Band was the one who saved the most stuff out of the sinking ship, the only notable problem is the lack of keys as an instrument, despite that it's there as DLC, I think they would have done better if they did like GTA V: A next-gen port of Rock Band 3, then keep on with DLC, and Madcatz could have probably just sold well among those who already played and those who are nw to it, I introduced about 20+ people to Rock Band 3, a few of them being pianists, and they loved the concept of being able to play for real on a game, especially since they have me as a song request tool!
The second problem is that Harmonix is way too ambitious. Rock Band VR only supports Oculus Rift, so it only appeals to a few select people, the Oculus is really expensive for what it does, and buying it with the game for 600+€ just to feel "a bit more into the game" and only with guitar is what convinced me that it's not going to do well.
The third thing, not a problem but an opinion, is that the campaign for the PC port would have helped, a little or a lot, but it would have. There are a lot of people who only play Phase Shift (And probably a half with the keyboard) for a reason or another, and some of them would have probably bought it if it came out on PC, allowing a lot of stuff. The problem on this one, is that Harmonix should have used a more "trustworthy" platform with a bigger audience, like Kickstarter, I never heard of Fig before the announcement.