The Scorecards: How ACS measures its providers' performance

In its annual scorecard of foster care and preventive services providers, the New York City Administration for Children’s Services replaced letter grades with rankings in an effort to increase competition among nonprofits and to reflect new Commissioner David Hansell’s emphasis on greater accountability.

The provider scorecards are an assessment of the quality of services provided by the small army of nonprofit agencies who do much of the work to manage the cases of roughly 9,000 children in the city foster care system and those who receive preventive services.

Foster care agencies work to provide placements outside of the home for children living in dangerous situations, while preventive services providers work proactively with families at risk of having children removed from their households.

The newest set of calculations, which measure providers’ progress toward meeting targets and improving outcomes during the 2016 fiscal year – as well as to identify areas where improvement is needed – were released exclusively to New York Nonprofit Media. The rating system ranks 62 family foster care and more than 100 preventive services programs.

The most recent metrics rank programs based on how well they compared with the work of other agencies across similar program areas, or the level of risk their clients’ cases presented. So, a provider might rank in first among its peers, instead of earning an A, per the letter grades issued in previous years.

The preventive categories include risk levels ranging from low to very high and foster care categories included program areas such as therapeutic family foster care, residential and specialized family foster care.

The methodology for ranking providers draws upon outcomes and measures gathered during recurrent interactions between ACS and its providers as well as data from the Provider Agency Measurement System comprehensive case review. ACS’ oversight procedures often involve reviewing the contacts providers have with children and families, measuring program usage, making safety and risk-related updates and following up on critical incidents. Each preventive and foster care program is measured against a baseline – the “category score,” or “program systems score” respectively – which is calculated using a median of about 4,000 cases from across the provider system. Then each provider is compared against the peer group for their program.

“We think competition is a good thing,” Hansell told NYN Media. “We want agencies to know where they stand in relation to each other and that there are performers providing the same service they are that may be doing better in some areas, and so that they know that it is possible to do better. It’s their obligation to figure out how they can improve their performance.”

Along with fostering competition, ACS has increased its emphasis on improving child safety and reducing the length of time children spend in foster care. The effort is designed to increase the rate at which children find permanent homes through family reunification, adoption or guardianship with relatives, known as KinGap. ACS officials said that KinGap placements have increased 25 percent, while adoptions increased 5 percent. The percentage of youths who re-enter the foster care system has also dropped by 1.6 percentage points to 7.7 percent over the past two years.

In recent years, ACS itself has faced intense pressure to improve its operations following the high-profile deaths of children in the agency’s care, the resignation of Commissioner Gladys Carrion, probes by the city comptroller and city Department of Investigation, failures involving its Close to Home program, and concerns about high caseloads and inadequately supported staff. In addition, the state appointed an outside monitor that reports to the state Office of Children and Family Services.

Hansell, a former KPMG executive and public health administrator who was appointed in February to replace Carrion, said the new approach reflected his priority of bringing more accountability to the agency. ACS also recently launched an Accountability Office to track data for outside nonprofits, staffers, providers and members of the public.

“While the scorecards obviously have existed here for some time, from my perspective, they’re now embedded in a stronger accountability framework than we had at the agency previously,” Hansell said.

While the metrics are an important indicator of how nonprofits are performing, the agency typically doesn’t publicize them. Nonprofits see their own performance data regularly, against the typical performance of other foster care providers delivering the same type of programming, or preventive services providers serving similar types of families.

Overall, foster care agencies received a median score of 77 of 100 on measures of permanency, such as the approval of a Family Assessment and Service Plan and reaching KinGap targets. They scored an 81 both on measures of safety – factoring in indications of maltreatment, children who are unaccounted for and other measures – and well-being, such as sibling placements and medical reviews.

The provider earning the highest score in permanency, HeartShare St. Vincent’s, received 91 points, with the lowest, Coalition for Hispanic Family Services, earning 63 points. NYN Media’s document takes a snapshot of this data by listing the rankings for all of the providers in key areas across various programs. Here are ACS’ complete scorecards for foster care and preventive services providers.

Providers are rarely deficient across the board. Instead, some exceeded the standard in one or two areas, while lagging in others. In one example, foster care provider the Coalition for Hispanic Family Services was ranked second in the safety and well-being categories, but last among 22 providers in permanency.

Those earning lower scores in certain outcomes – which, for preventative agencies include repeated maltreatment, placements and safety, among other metrics – will get targeted help in those areas. Every provider is required to create plans to improve any deficiencies.

For providers whose performance continues to raise concerns, they are targeted for heightened monitoring. If the situation continues to be below the agency’s standards, a provider can have their intake closed, corrective measures can be taken or its contracts can be terminated.

In 2014, the nonprofit Community Counseling & Mediation had a contract canceled, though it still operated a preventive services contract last year. Last year, the Women’s Prison Association was placed on a corrective action plan and it ranked 17th out of 18 preventive services providers this year that are working with a very high-risk population.

Three nonprofits are currently in corrective action status – about the standard for a typical year – though ACS officials declined to name the providers.

Julie Farber, the ACS deputy commissioner of family permanency services, said that while providers were competing against each other, the agency is broadly sharing best practices uncovered along the way. “We look at bright spots and challenges, and we have a very disciplined approach to looking at the practice, and then we work to scale the bright spots and mitigate the challenges,” she said.

Stephen Levin, who chairs the New York City Council’s General Welfare Committee, said that over the past several decades, the foster care system – which had more than 16,000 children in 2007 and more than 40,000 in the late 1990s – now receives far fewer cases. He was heartened that the bulk of resources are now being invested in preventive services. More recently, ACS has allowed nonprofit workers to participate in the ACS Workforce Institute, which offers professional skill development that only used to be open to city workers.

He said the death of Zymere Perkins – a 6-year-old boy whose case was being investigated by ACS’ child protective specialists and supervisors – showed where more training could help.

“You had across-the-board system failure there, where you had CPS managers, supervisors all missing important aspects of what was going on in his life, but also there were important gaps in his preventive case and with his preventive provider. They need that type of training; they need that ongoing professional development.”

He said he considered the scorecards an important tool for the agency and he said he’d considered legislation to publicly provide the information, but some providers resisted because of concerns that they presented an incomplete picture.

Mary Jane Dessables, who is the director of information, research and accountability at the Council of Family and Child Caring Agencies, said the umbrella organization has collaborated with nonprofit organizations to provide feedback on the scorecards. “Overall, it’s a comprehensive, generally fair, good measure of what’s going on in the system,” she said.

But she cautioned that a simple ranking cannot convey all of the various and complex program elements that factor into the measurements. For example, one of the components of the final score has roughly 180 questions. There could also be anomalies due to geography or program area that affect a provider’s progress.

”Every program is very different and has individual reasons why they might end up scoring better or worse on different things,” she said. “It doesn’t always tell the full story.”

One main benefit to the scorecard is letting providers see how they are doing and how they are driving performance on programs such as KinGap. “It keeps people from working in a vacuum,” she said.

Click to see ACS’ complete scorecards for foster care and preventive services providers.