Joolz, I am sorry you are injured again. I have had a very serious fracture of the Left ankle myself. I wouldn't let them just pass you around from person to person and tell you nothing. You don't know at this point if it is an old fracture or a new one.
You don't know the extent of the soft tissue damage.
You don't know if you need a surgical repair of the ligaments. (They are not likely to heal on their own if you have torn them and they are not attached like they should be)
You don't know if you are allowed to weight bear.
You don't know if you should do physio.
You don't know why your toes are numb.
That is a lot of things you don't know about how to get back on your feet. I would type it all up and hand this to your GP. Be calm and tell him/her that you deserve to see the specialist due to the previous injury and also due to the conflicting opinions. Point out that you really want to do your very best so you can walk again. EDS ligaments have poor collagen and they tear easily and heal slowly. Wrong treatment choices will impact you for years. All of this is absolutely true.
If it doesn't get you the appointment, I would pay to see the specialist. It really is your best shot at recovering from this injury. My personal thoughts are that you should wear the boot and that the numbness in the toes is due to swelling. The swelling doesn't have to be really evident right at the toes, the nerves could be compressed higher up toward the ankle. My toes were purple and numb after the surgery for a couple of months at least and the surgeon said it was just swelling. It lasted longer in me than most people. I swelled up when walking for several years, long after I was out of the cast. Stay off your feet as much as possible. Use the wheelchair if you can when you go out. You don't need to strain it any more than necessary right now.
Physio made me much worse with my shoulder. Exercise after my ankle had time to heal, just careful walking made my ankle worse. It was very weak and it just turned and sprained and reinjured. Right now it is a huge risk (my opinion) that it will make it worse. Give it a lot of time and if you do physio take it very slowly and carefully with the physio. "Just soft tissue damage" can be as bad an injury as a fracture according to the foot surgeon I used to work for. Bone heals in about 8 weeks. Torn ligaments can take a lot longer to heal depending on the exact injury. Don't let them send you away with no answers. That would mean you would be stuck off your feet in a lot of pain for no good reason. I would expect that you should have a good ankle brace or support after you are out of the boot and supposedly healed. Expect it to be weak and turn very easily.
I really am sorry you are injured. I just went through my whole injury and it wasn't any fun at the time. I also have had injuries since then but mostly in the foot. I get stress fractures and dislocations due to the unusual way my foot is now with some things pinned and fused that normally wouldn't be the way they are. One positive point, my foot is nowhere near my worst joint. I did heal fairly well but it took years to get there. Once you have your specialist appointment work with them and make a treatment plan.