What is motivating the question? In our work we find that for some, this sort of question can come from a feeling of defensiveness: our findings might challenge someone’s leadership or identity and draw attention to inequities they may have perpetuated. This post is not for those readers. Further, the question “are those differences statistically significant” can be motivated by an assumption that if pain is not being broadly experienced, it doesn’t merit our attention. In response to that we offer our perspective that any pain should be acknowledged and attended to – whether or not the sample size or severity meets a bar for statistical significance in a pure data sense. Instead, this post is for those with a statistics, research or evaluation background who may be curious about our methodology, data samples and results.
As a social science researcher, I run rigorous statistical analyses beyond the percentage-based findings we share with clients and what we published in our Unrealized Impact reports. Below I give deeper context on our surveys, data and results — specifically those about race and ethnicity. I also include some technical language for those seeking to understand, at a statistically significant level, what our survey data tells us.
As of October 2021, Promise54 has collected over 60,000 DEI Staff Experience Surveys. We’ve used the data from these surveys to publish our latest Unrealized Impact 2.0 report, as well as our report from 2017. While our DEI Staff Experience Survey is anonymous, participants have the option of answering questions about demographics and identities. We use these responses to cross-tabulate identities, like race/ethnicity, with various questions from the survey. To analyze the surveys and generate insights, we’ve used both qualitative and quantitative research methods, with the majority of our findings coming from Likert scale questions and responses.
We began our latest report with over 3,000,000 data points that represented people from all 50 states, DC, and Puerto Rico, and representing over 500 different organizations – primarily across the education sector. We then ran a set of analyses starting with one simple research question: Are people of various races/ethnicities having different experiences within their organizations?
Running a Spearman rank correlation analysis across all of our survey questions, we were able to identify the questions that yielded the most efficient and holistic data in the fewest amount of questions possible. As we isolated these highly efficient and telling survey questions, a pattern became clear: they were the same questions our clients most often ask about — or worry about — when they receive our customized DEI Staff Experience Report.
The following four isolated survey questions arose:
- Bias: “I have been on the receiving end of bias (either overt or implicit) at our organization.”
- Intent to Stay: “I fully expect to be working in this organization three years from now.”
- Whole Self: “I can bring my “whole self” to work (i.e., I am able to engage at work with a free and authentic expression of who I really am).”
- Net Promoter Score: “On a scale of 0 to 10, how likely are you to recommend a qualified friend or colleague to work here?”
After running the statistical tests, we found that there, in fact, were significant differences along the following identities and measures:
- Race/ethnicity and how people experience bias, H(5)=887.69, p<.001.*
- Race/ethnicity and the Intent to Stay question (proxy for retention/turnover), H(5)=1030.353, p<.001*
- Race/ethnicity and how strongly people feel that they can be their “whole self” at their organizations/schools, H(5)=546.795, p<.001.*
- Race/ethnicity and how people score on the Net Promoter Score, H(5)=690.901, p<.001*
Once we verified that statistically significant differences existed among race/ethnicities along the identified four survey questions, we performed a post-hoc pairwise comparison to test each race/ethnicity group against one another and identify where the statistically significant differences existed. Declaring that a difference exists is one thing, but we also wanted to identify where those differences specifically existed. We wanted to know how groups of people and identities were spread across the spectrum of experiences. Then, controlling for Type I errors across tests by using an adjusted Bonferroni alpha level of .003 (0.05/15), we found the following (all ps<.003):
Experiencing bias within their organization
- Black survey participants experience bias (mean rank=24774.88) more than…
- White staff (mean rank=20918.90)
- AAPI staff (mean rank 22951.15)
- Latinx staff (mean rank=24031.59)
- There are no statistically significant differences between Black, Multiracial (mean rank=25129.25) and Native American (mean rank=25331.82) survey participants.
- White survey participants experience bias (mean rank=20918.90) less than every other race/ethnicity category of survey participants.
Feeling that they can bring their “whole self” to work
- Multiracial staff feel statistically significantly less able to bring their whole self to work (mean rank=19851.33) compared to…
- White staff (mean rank=22910.18)
- Black staff (mean rank=21437.34)
- AAPI staff (mean rank=22425.86)
- Latinx staff (mean rank=24864.36)
- Of the three largest race/ethnicity subgroups, Black staff score statistically significantly lower (mean rank=21437.34) than white staff (mean rank=22910.18) and Latinx staff (mean rank=24864.36).
Net Promoter Score
- Multiracial staff are statistically significantly less likely to recommend their organization to a friend or colleague (mean rank=19965.86) compared to…
- White staff (mean rank=22465.37)
- Black staff (mean rank=21475.01)
- AAPI staff (mean rank=22041.34)
- Latinx staff (mean rank=25574.32)
- Of the three largest race/ethnicity subgroups, Black staff score statistically significantly lower (mean rank=21475.01) than white staff (mean rank=22465.37) and Latinx staff (mean rank=25574.32).
Intent to Stay
- Staff that identify as Multiracial and AAPI scored statistically significantly lower (mean rank=20960.19 and mean rank=20774.55, respectively) on the Intent to Stay question compared to…
- White staff (mean rank=21826.99)
- Black staff (mean rank=22408.49)
- Native American staff (mean rank=24702.91)
- Latinx staff (mean rank=26431.37)
- Of the three largest race/ethnicity subgroups, white staff scored statistically significantly lower (mean rank=21826.99) than Black staff (mean rank=22408.49) and Latinx staff (mean rank=26431.37). Note: There are some sociological implications to explore on this point — but that is for another day.
This data serves as a critical reminder that we all have a long way to go on this road towards equity and justice. I hope these findings further amplify the disparities in experience that take place for People of Color and/or those with other historically marginalized identities, and drive action to close those gaps.
Finally, a reminder: when evaluating survey data, if you find yourselves or your colleagues asking questions like: “Okay, but is this just one person who feels this way?” or “Does this response accurately represent the overall experiences of our collective staff?”, we urge you to make sure that concerns around “statistical significance” or a “small sample size” do not lead to overlooking the pain or trauma that can be hiding in data. At Promise54, we believe there is validity (and significance!) in understanding what data is revealing … no matter what the sample size or p-value is.
If you have additional questions or want to break some virtual bread over data and theory, you can reach me at Michael@promise54.org.
* Each of the above mentioned statistically significant findings were found using a Kruskal-Wallis H Test to examine significant differences based on the identity of race/ethnicity and a specific Likert scale question on our DEI Staff Experience Survey from 1 (strongly disagree) to 5 (strongly agree) and 0-10 for the single NPS question.
Contact Michael: firstname.lastname@example.org