Once the webpage loaded, there was a percentage already going in the top left corner. After that finished, it installed something and my only option was "O". So I pressed it and now the screen is frozen on "Enabling HEN..."
This sounds a bit weird, the installer should not have automatically enabled HEN at the end. Or do you mean it got frozen when YOU enable HEN, after installation is completed?
It is true that a few times during installation you are shown some message and you only get the option to press "O". This was not documented and puzzled me too, since normally in PS3 "O" is the default button to go back and "X" to go forward, but pressing "O" in here is the correct (and only) choice.
Did the installer reboot your PS3 automatically? I see they added a "reboot" step in the latest guide, perhaps if you enabled HEN straight after installation without any reboot it might not work. But now it should be already installed, so you can boot and enable it from the XMB.
Edit: I checked some threads at psx-place on people getting the PS3 frozen when enabling HEN, and the most common reason seems to be browser stuff getting in the way, since both HEN installation and HEN activation use the browser... so try to clear as much as you can in the browser (cache, history, cookies and so on) both before installing and before enabling HEN.
Another thing which is not explained in the official guide is when you need to be online and when you should rather not.
Obviously being online is needed when running the installer, and I was actually signed in to a secondary PSN account in this phase. The instructions for the older HAN method suggested not to use your real PSN account but a throwaway account, just to be safe against any chance of ban, but the official instructions for HEN do not mention this.
However I recommend that for the time being, after installing HEN you stay signed out of your PSN when emabling HEN (for example, do not setup your PS3 to sign-in automatically). Nobody has reportedly been banned from PSN for using HEN so far, but better stay safe. After all, for the purpose of RB3 customs, we really need to enable HEN only when we want to use a file manager to add/change/remove customs to the game folders, and once you've done that, you can disable HEN (at the moment there is no manual disable option, so you have to reboot to disable it) and then it should be safe to go online. But if you want to be sure, always check the recommendations at psx-place!