Predictive analytics at family agencies (featuring Eckerd Rapid Safety Feedback) (Department of Health and Human Services)
| Multi Request | Predictive analytics at family agencies (featuring Eckerd Rapid Safety Feedback) |
| Submitted | May 23, 2019 |
| Est. Completion | None |
This request was filed by Beryl Lipton through MuckRock, a non-profit organization that helps journalists, researchers, and the public file, track, and share records requests. You may follow up directly with the requester by following the prompts below. Should this request be directed elsewhere? Have you already responded? Click Get Help above and let us know, or reach our team directly at info@muckrock.com . Frequently asked questions about MuckRock.
Upload documents and update the status of the request below, or choose an option to reach our team directly:
Communications
From: Beryl Lipton
To Whom It May Concern:
Pursuant to the New Hampshire Right to Know Law, I hereby request the following records:
Materials related to any predictive or automated analysis software and algorithms used by this agency in the performance of its responsibilities.
These materials should include, but not be limited to, those regarding the software and algorithm used in the Eckerd Rapid Safety Feedback Software. For reference and help retrieving responsive documents, this program was publicly discussed in the following news article: https://www.bostonglobe.com/2015/11/07/childwelfare-bostonglobe-com/AZ2kZ7ziiP8cBMOite2KKP/story.html
Please provide the following records related to the above program and any similar programs:
• Software and algorithm:
Any software and algorithms developed for the implementation of the program. We are particularly interested in any algorithms that have a public policy outcome, such as providing guidance on a policy or an assessment of an individual.
More generally, please provide any software that was developed by or with, given to, used by, purchased or licensed to this agency for implementation of the above referenced program. This would include original source code, any compiled binaries (also known as executables), spreadsheets, program scripts, and other digital materials used to calculate any data in the above program. It would also include the input training data for machine learning algorithms. Collectively, these responsive materials will be referred to as "the software" in the following paragraphs.
Please provide any description of input and output data fields that capture the type of information that is submitted to the software and that is produced by the software.
• Contracts and procurement materials:
Any and all contracts (including related amendments and exhibits), non-disclosure agreements, liability waivers, insurance agreements, Requests for Proposals, responses to Requests for Proposal, Memorandums of Understanding, Letters of Interest, and informal agreements that reference "the software" or otherwise guide its use or the relationship between this agency and the provider or developer of the software.
• Instructional and training materials:
All instructional materials, presentations and presentation materials (including recorded video and audio, PowerPoint files, prepared remarks, and slides formats), and other guidance on the use of "the software." This includes any notes taking during meetings that discussed the use of the software, any explanations (whether internally or externally generated) of how the software works, and any other document that has been used to help explain the use of "the software" to any party, including internal documentation, public relations materials, and executive summaries.
• Assessments, audits, and reports:
A copy of any privacy impact assessments, validation studies, reports to legislative bodies, annual reports that mention the use of the Software, as well as audit records (including but not limited to security audits of the software, misuse reports, and reports to oversight bodies), and other evaluations that reference "the software," otherwise pertain to its use, or were conducted with respect to the program or with respect to any software or algorithms used in connection with the program. In addition to a copy of any validation studies, please also provide any records pertaining to inquiries for the need of validation studies or discussion of validation studies (potential or actual).
By “validation study,” we mean any study designed to assess how accurate the program is in predicting what it is supposed to predict or any study designed to assess whether the program may err in the direction of either under- or overestimating likelihoods or predicted outcomes or may produce any results that are biased or unexpected in any way.
• Policies:
A copy of any usage policies, data retention policies, legal opinions, warranties, standard operating procedures, and other guiding materials that reference "the software" or otherwise pertain to its use.
• Funding materials:
A copy of any invoices, financial records, funding opportunity announcements, grant applications and grantor status/progress reports that reference "the software" or otherwise pertain to its use.
• Recent history:
For the aspects of the software that require an input (for example, to compute a value), we request a copy of the five most recent sets of data that were used for input, as well as the five most recent outputs of the software, in whatever their native format is. If these inputs or outputs include exempt information, we ask that you exempt only that information which is specifically exempted from law and provide the other material.
Please let me know if further clarification is needed for this request.
The requested documents will be made available to the general public, and this request is not being made for commercial purposes.
In the event that there are fees, I would be grateful if you would inform me of the total charges in advance of fulfilling my request. I would prefer the request filled electronically, by e-mail attachment if available or CD-ROM if not.
Thank you in advance for your anticipated cooperation in this matter. I look forward to receiving your response to this request within 5 business days, as the statute requires.
Sincerely,
Beryl Lipton
From: Muckrock Staff
To Whom It May Concern:
I wanted to follow up on the following New Hampshire Right to Know Law request, copied below, and originally submitted on May 23, 2019. Please let me know when I can expect to receive a response.
Thanks for your help, and let me know if further clarification is needed.
From: Department of Health and Human Services
An acknowledgement letter, stating the request is being processed.
From: Muckrock Staff
To Whom It May Concern:
I wanted to follow up on the following New Hampshire Right to Know Law request, copied below, and originally submitted on May 23, 2019. Please let me know when I can expect to receive a response.
Thanks for your help, and let me know if further clarification is needed.
From: Muckrock Staff
To Whom It May Concern:
I wanted to follow up on the following New Hampshire Right to Know Law request, copied below, and originally submitted on May 23, 2019. Please let me know when I can expect to receive a response.
Thanks for your help, and let me know if further clarification is needed.
From: Department of Health and Human Services
From: Martin, John
Sent: Friday, July 5, 2019 9:32 AM
To: '74013-69756670@muckrock.com' <74013-69756670@muckrock.com>
Cc: Nachman, Frank <Frank.Nachman@dhhs.nh.gov>; DHHS: PIO <PublicInformationOffice@dhhs.nh.gov>
Subject: RE: New Hampshire Right to Know Law Request: Predictive analytics at family agencies (featuring Eckerd Rapid Safety Feedback) (Department of Health and Human Services)
Good morning Beryl. For reasons I don’t fully understand, because I am not a technical person, your request is proving to be somewhat complicated. Today our Director of Information Services is meeting with one of his staff to discuss this request. By next week I should have a sense as to what the Department has that is responsive to your request and when you can anticipate receiving it.
From: DHHS: PIO <PublicInformationOffice@dhhs.nh.gov<mailto:PublicInformationOffice@dhhs.nh.gov>>
Sent: Tuesday, July 2, 2019 3:44 PM
To: Martin, John <John.Martin@dhhs.nh.gov<mailto:John.Martin@dhhs.nh.gov>>
Cc: Nachman, Frank <Frank.Nachman@dhhs.nh.gov<mailto:Frank.Nachman@dhhs.nh.gov>>; DHHS: PIO <PublicInformationOffice@dhhs.nh.gov<mailto:PublicInformationOffice@dhhs.nh.gov>>
Subject: FW: New Hampshire Right to Know Law Request: Predictive analytics at family agencies (featuring Eckerd Rapid Safety Feedback) (Department of Health and Human Services)
Hi, John, we received a follow up request below. Thanks, Kathy.
From: requests@muckrock.com<mailto:requests@muckrock.com> <requests@muckrock.com<mailto:requests@muckrock.com>>
Sent: Tuesday, July 2, 2019 5:03 AM
To: DHHS: PIO <PublicInformationOffice@dhhs.nh.gov<mailto:PublicInformationOffice@dhhs.nh.gov>>
Subject: RE: New Hampshire Right to Know Law Request: Predictive analytics at family agencies (featuring Eckerd Rapid Safety Feedback) (Department of Health and Human Services)
EXTERNAL: Do not open attachments or click on links unless you recognize and trust the sender.
________________________________
Department of Health and Human Services
RTK Office
129 Pleasant Street
Concord, NH 03301
July 2, 2019
This is a follow up to a previous request:
To Whom It May Concern:
I wanted to follow up on the following New Hampshire Right to Know Law request, copied below, and originally submitted on May 23, 2019. Please let me know when I can expect to receive a response.
Thanks for your help, and let me know if further clarification is needed.
Filed via MuckRock.com
E-mail (Preferred): requests@muckrock.com<mailto:requests@muckrock.com>
Upload documents directly: https://www.muckrock.com/
Is this email coming to the wrong contact? Something else wrong? Use the above link to let us know.
For mailed responses, please address (see note):
MuckRock News
DEPT MR 74013
411A Highland Ave
Somerville, MA 02144-2516
PLEASE NOTE: This request was filed by a MuckRock staff reporter. Also note that improperly addressed (i.e., with the requester's name rather than "MuckRock News" and the department number) requests might be returned as undeliverable.
---
On June 4, 2019:
Subject:
An acknowledgement letter, stating the request is being processed.
---
On May 23, 2019:
Subject: New Hampshire Right to Know Law Request: Predictive analytics at family agencies (featuring Eckerd Rapid Safety Feedback) (Department of Health and Human Services)
To Whom It May Concern:
Pursuant to the New Hampshire Right to Know Law, I hereby request the following records:
Materials related to any predictive or automated analysis software and algorithms used by this agency in the performance of its responsibilities.
These materials should include, but not be limited to, those regarding the software and algorithm used in the Eckerd Rapid Safety Feedback Software. For reference and help retrieving responsive documents, this program was publicly discussed in the following news article: https://www.bostonglobe.com/2015/11/07/childwelfare-bostonglobe-com/AZ2kZ7ziiP8cBMOite2KKP/story.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.bostonglobe.com_2015_11_07_childwelfare-2Dbostonglobe-2Dcom_AZ2kZ7ziiP8cBMOite2KKP_story.html&d=DwQFaQ&c=vYl7KJMDeuM7F-Nqf_hfailBifPmyspo7hrJGlNN7nU&r=gsAXtPvP27JFVSQkU2nMEjbheIvy7Hm-VSnsOMWIcJ8&m=N_03VMYETo43tZEQdXYC7bDRwQTrPXza3wPP3ZEzM98&s=0hrISIkz1sX5TSeLf5qVk-adoTHbJpwI6lNeG6OVqoE&e=>
Please provide the following records related to the above program and any similar programs:
• Software and algorithm:
Any software and algorithms developed for the implementation of the program. We are particularly interested in any algorithms that have a public policy outcome, such as providing guidance on a policy or an assessment of an individual.
More generally, please provide any software that was developed by or with, given to, used by, purchased or licensed to this agency for implementation of the above referenced program. This would include original source code, any compiled binaries (also known as executables), spreadsheets, program scripts, and other digital materials used to calculate any data in the above program. It would also include the input training data for machine learning algorithms. Collectively, these responsive materials will be referred to as "the software" in the following paragraphs.
Please provide any description of input and output data fields that capture the type of information that is submitted to the software and that is produced by the software.
• Contracts and procurement materials:
Any and all contracts (including related amendments and exhibits), non-disclosure agreements, liability waivers, insurance agreements, Requests for Proposals, responses to Requests for Proposal, Memorandums of Understanding, Letters of Interest, and informal agreements that reference "the software" or otherwise guide its use or the relationship between this agency and the provider or developer of the software.
• Instructional and training materials:
All instructional materials, presentations and presentation materials (including recorded video and audio, PowerPoint files, prepared remarks, and slides formats), and other guidance on the use of "the software." This includes any notes taking during meetings that discussed the use of the software, any explanations (whether internally or externally generated) of how the software works, and any other document that has been used to help explain the use of "the software" to any party, including internal documentation, public relations materials, and executive summaries.
• Assessments, audits, and reports:
A copy of any privacy impact assessments, validation studies, reports to legislative bodies, annual reports that mention the use of the Software, as well as audit records (including but not limited to security audits of the software, misuse reports, and reports to oversight bodies), and other evaluations that reference "the software," otherwise pertain to its use, or were conducted with respect to the program or with respect to any software or algorithms used in connection with the program. In addition to a copy of any validation studies, please also provide any records pertaining to inquiries for the need of validation studies or discussion of validation studies (potential or actual).
By “validation study,” we mean any study designed to assess how accurate the program is in predicting what it is supposed to predict or any study designed to assess whether the program may err in the direction of either under- or overestimating likelihoods or predicted outcomes or may produce any results that are biased or unexpected in any way.
• Policies:
A copy of any usage policies, data retention policies, legal opinions, warranties, standard operating procedures, and other guiding materials that reference "the software" or otherwise pertain to its use.
• Funding materials:
A copy of any invoices, financial records, funding opportunity announcements, grant applications and grantor status/progress reports that reference "the software" or otherwise pertain to its use.
• Recent history:
For the aspects of the software that require an input (for example, to compute a value), we request a copy of the five most recent sets of data that were used for input, as well as the five most recent outputs of the software, in whatever their native format is. If these inputs or outputs include exempt information, we ask that you exempt only that information which is specifically exempted from law and provide the other material.
Please let me know if further clarification is needed for this request.
The requested documents will be made available to the general public, and this request is not being made for commercial purposes.
In the event that there are fees, I would be grateful if you would inform me of the total charges in advance of fulfilling my request. I would prefer the request filled electronically, by e-mail attachment if available or CD-ROM if not.
Thank you in advance for your anticipated cooperation in this matter. I look forward to receiving your response to this request within 5 business days, as the statute requires.
Sincerely,
Beryl Lipton
Filed via MuckRock.com
E-mail (Preferred): requests@muckrock.com<mailto:requests@muckrock.com>
Upload documents directly: https://www.muckrock.com/
Is this email coming to the wrong contact? Something else wrong? Use the above link to let us know.
For mailed responses, please address (see note):
MuckRock News
DEPT MR 74013
411A Highland Ave
Somerville, MA 02144-2516
PLEASE NOTE: This request was filed by a MuckRock staff reporter. Also note that improperly addressed (i.e., with the requester's name rather than "MuckRock News" and the department number) requests might be returned as undeliverable.
[Image removed by sender.]
-
image001
From: Muckrock Staff
To Whom It May Concern:
I wanted to follow up on the following New Hampshire Right to Know Law request, copied below, and originally submitted on May 23, 2019. Please let me know when I can expect to receive a response.
Thanks for your help, and let me know if further clarification is needed.
From: Department of Health and Human Services
Good afternoon Beryl. The DCYF Rapid Safety Feedback is an ETL process from NH Bridges EDW to the operational data center of the contracted analytical company called MindShare. MindShare then posts reports of potential high-risk level 1 responses into a portal that may require a second review from a DCYF QA team. MindShare has a proprietary analytical tool\algorithm that they will NOT share with us. The output of from this tool is posted on a secure portal. This data is reviewed by a DCYF RSF team to identify specific high risk cases so the field can take a second look for risk mitigation.
I have 18 separate documents that are responsive to your Right to Know request. They total 231 pages. DHHS charges $.10 for electronic records. Following your receipt of the documents please submit a check or money order in the amount of $23.10 made payable to “Treasurer – State of New Hampshire.”
I will have to documents to you before the end of today.
-
~WRD000
From: Department of Health and Human Services
1-3 of 18
-
~WRD000
From: Department of Health and Human Services
4 of 18
-
~WRD000
From: Department of Health and Human Services
5 of 18
-
~WRD000
From: Department of Health and Human Services
6-8 of 18
-
~WRD000
From: Department of Health and Human Services
9 of 18
-
~WRD007
From: Department of Health and Human Services
10-12 of 18
-
~WRD114
From: Department of Health and Human Services
13 of 18
-
~WRD159
From: Department of Health and Human Services
14 of 18
-
~WRD286
From: Department of Health and Human Services
15 of 18
-
~WRD270
From: Department of Health and Human Services
17 of 18
From: Department of Health and Human Services
I see that I inadvertently sent you three copies of the same document (MOU with data Sharing Agreement). So you got an extra 12 pages you didn’t need. That means the total number of pages is 219, so the cost is $21.90.
-
~WRD000
From: Department of Health and Human Services
Also my document count was wrong. Not counting the two duplicates you should have received 15 documents.
-
~WRD000
From: Beryl Lipton
Hello Mr. Martin,
Thank you again for all of these materials. I really appreciate it, and I'll put a check in the mail as soon as possible.
Have a great evening,
Beryl
From: Beryl Lipton
To Whom It May Concern:
Please find enclosed a check for $23.10 to satisfy the fee associated with the attached public records request.
Thank you.
From: Department of Health and Human Services
Hi Beryl. Your check arrived today. Thank you so much.
-
~WRD000
Department of Health and Human Services
RTK Office
129 Pleasant Street
Concord, NH 03301
From: Muckrock Staff
To Whom It May Concern:
I wanted to follow up on the following New Hampshire Right to Know Law request, copied below, and originally submitted on May 23, 2019. Please let me know when I can expect to receive a response.
Thanks for your help, and let me know if further clarification is needed.
From: Muckrock Staff
To Whom It May Concern:
I wanted to follow up on the following New Hampshire Right to Know Law request, copied below, and originally submitted on May 23, 2019. Please let me know when I can expect to receive a response.
Thanks for your help, and let me know if further clarification is needed.
From: Department of Health and Human Services
I’m a bit confused. I responded to this request some time ago. I don’t believe we have any additional documents that are responsive to your request.
Files
pages