Yellowing of prostrate rosemary can be caused by three factors we've found on green roofs. First is water stress, second is Iron deficiency and the third is Nitrogen deficiency. In a highly inorganic material you should check the pH. Calciferous soils with higher pH will bind iron and promote the yellowing. We add dry pine needles and oak leaves as a mulch around the root ball before we plant and then add the same to the soil around the plants. The organic matter provides micronutrients in ways, we've seen, industrially manufactured fertilizers can't. In fact, smart leaf compost application has allowed us to create thriving rosemary plantings.
The attached photo is of prostrate rosemary growing on a green roof in 50mm of a soil media with sharp sand for drainage and heavy leaf usage. Once the plant becomes established the leaf requirements diminish. I prefer to only plant rosemary where it can 'cascade' over the side of a sidewall or edging.
Finally, cow peas interspersed with the rosemary really helps from a NH4 availability perspective.