Rising interest rates, higher inflation, and rising construction costs are expected to slow the growth of affordable housing finance companies in the current fiscal year, according to a report.