New Zealand's Minister for Pacific Peoples says the agency can still deliver despite losing around $64.2m in three years, but the opposition claims government "doesn't care" about Pasifika.