In complaining that astrophysics was hard because of all the maths, a student recently told me:
"The way it's presented is: 'OK, you get that $A+B=C$? Excellent. Now derive $DQH$, use a matrix to get $Y$, find $M$ by mysterious means. What? Why can't you do that?"
Let's start with a story about legendary chef Escoffier - or rather, about one of his sauce recipes. Someone learning to cook followed Escoffier to the letter; the sauce came out inedible. He then followed Delia Smith's recipe for the same sauce, and it came out fine.
The difference? Escoffier said something like "zest the lemons into a bowl of water." Smith said "zest the lemons into a bowl of water and discard the water - it'll be really bitter."
No, not at all. Delia's recipe made more sense for the trainee chef because he hadn't made a thousand sauces before - and Delia is very good at taking every step slowly and clearly, in a way that would be excruciating for an experienced chef. Escoffier, on the other hand was writing for exactly that kind of experienced chef - someone who knew how food fit together already, who wanted to hone their skills, and just needed some extra notes to make things exceptional.
The wannabe didn't read Escoffier, decide "it's presented all wrong!" and decide he was rubbish at cookery; he found a more accessible entry point and got enough experience under his belt so that he could read Escoffier at a later point, and end up with edible sauces.
Similarly, if you were trying to teach someone English, you wouldn't start them with Finnegans Wake. That would be a Really Bad Idea. It's not an appropriate entry point to the English language, or even a sensible thing to read unless you have a solid grasp of all sorts of literary references and experience of tackling difficult literature.
Maybe you see where I'm going with this: any science text will assume a certain level of understanding, and a certain level of competence in maths. Even Basic Maths For Dummies assumes you can count; a serious astro book will most likely assume you have a decent level of linear algebra.
That doesn't mean "you'll need to recognise a matrix" - it means you'll need to be familiar with the methods, the vocabulary and the notation - and you'll probably have solved a lot of problems and developed a certain instinct for how things fit together. That's something that only comes with practice.
That's not to say you can't read it: my approach to any mathematical text has three prongs.
1. Skip the hard bits (for now). If there's something I don't get straight away, I'll skip over the proofs and take them on trust - possibly marking them to come back to later.
2. Ask for help. If there's a step I don't get, I'll call up a friend with better skills than I have.
3. Do some background work. If there's a whole topic I don't understand but need to, I'll find a good introduction to the subject and read it, and do the exercises until I'm happy with it.
The most important thing, though, is my attitude: Escoffier doesn't set out to make it hard to make a sauce; he just writes for the people he regularly deals with. Scientists don't set out to make it hard to understand their work - far from it, I don't know any scientist who doesn't want their work to be read and understood - they just present it for the audience most likely to read it.
They're not being deliberately obtuse.
They're not trying to keep secrets from you.
They're just making assumptions about what you know.
Speaking for myself: if someone reads something I wrote and can't see how I've taken a given step, I'm really happy to get an email saying so - because then I can see what unclear logical jumps I've made and, where possible, fill them in a bit.
Maths isn't your enemy, any more than cookery is - and there are plenty of people willing to help you with both.