Androgen receptor (AR) signaling in stromal cells is important in prostate cancer, yet the mechanisms underpinning stromal AR contribution to disease development and progression remain unclear. Using patient-matched benign and malignant prostate samples, we show a significant association between low AR levels in cancer associated stroma and increased prostate cancer-related death at one, three and five years post-diganosis, and in tissue recombination models with primary prostate cancer cells that low stromal AR decreases castration-induced apoptosis. AR-regulation was found to be different in primary human fibroblasts isolated from adjacent to cancerous and non-cancerous prostate epithelia, and to represent altered activation of myofibroblast pathways involved in cell cycle, adhesion, migration, and the extracellular matrix (ECM). Without AR signaling, the fibroblast-derived ECM loses the capacity to promote attachment of both myofibroblasts and cancer cells, is less able to prevent cell-matrix disruption, and is less likely to impede cancer cell invasion. AR signaling in prostate cancer stroma appears therefore to alter patient outcome by maintaining an ECM microenvironment inhibitory to cancer cell invasion. This paper provides comprehensive insight into AR signaling in the non-epithelial prostate microenvironment, and a resource from which the prognostic and therapeutic implications of stromal AR levels can be further explored.