Last week I wrote a guest post here at Grounded Parents looking at what we can learn from a recently published study in the Journal of Midwifery and Women’s Health on the risks and benefits of homebirths in the U.S. In the days between when I wrote the post and when it was actually posted, Dr. Amy Tuteur asked a statistics professor to write an analysis at Skeptical OB comparing the results in the study to data on U.S. hospital births from the CDC. I wasn’t going to comment on it because I felt like I’d already stepped on a hornets’ nest with my first post, but then i09 picked up the story in a way that suggested that they read neither the original study they were criticizing or even the post at Skeptical OB that they were linking to. I feel like it’s my duty to step in and provide a little more thoughtful review.
So, how do I know that i09 did not bother to read the study or the Skeptical OB post? Their title was “Statistics Professor Challenges Midwives’ Math on Home Birth Safety.” There are so many things wrong just in this title that it’s hard to even know where to begin. First of all, let’s take the term “midwives’ math.” i09 also doubled down on this in the first sentence of their post which started out with “Recently a group of midwives published a paper about birth outcomes for home births…” If i09 had actually bothered to google the authors of the supposedly overturned study, they would have seen that the authors are medical anthropology researchers at Oregon State University. The head author is also a registered midwife in addition to being a medical anthropology professor, though that’s hardly “a group of midwives.” (Note: I’m not linking to the authors’ bios for their privacy and protection).
Also, if they’d read either the “Midwives’” study or the analysis that they wrote their entire post on, they may have realized that no one is actually challenging anything. The Journal of Midwifery and Women’s Health study did not attempt to make any relative risk comparisons between homebirths and hospital births. All it did was look at data from about 17,000 planned homebirths in the US and report on outcomes for those births. The analysis posted on Skeptical OB then takes the numbers from that study and compares them to CDC risk numbers on hospital births to see if they can tease out whether the outcomes of particular risks are much higher in homebirths than hospital births. No one is overturning anything or “challenging” anything. It’s just one analysis building on another.
If I were to edit i09’s title to make it more accurate, I would write something more along the lines of “Statistics Professor Writes a Blog Post that Builds on Math Regarding Homebirth Risks from a Peer Reviewed Study in a Well-Respected Journal by Researchers from Oregon State University.” But this is probably why I’m not a writer at i09.
Phew. Now that I’ve got that out of the way, I’m just going to jump right into the “statistics professor” analysis at Skeptical OB. (Note: quotes are because I think it’s weird that Prof. Orosz keeps being referred to as “a statistics professor” instead of by her name, not because I’m doubting that she’s really a statistics professor.)
Remember from my previous post, the study in the Journal of Midwifery and Women’s Health only looked at a cohort of women who planned homebirths. It did not have a comparison hospital birth cohort. Brooke Orosz (the previously mentioned statistics professor) then attempts to put together a comparable cohort of hospital births using the CDC’s Wonder Database, which collects data on birth-infant death certificates. The homebirth cohort had very few preterm or low-weight babies so Prof. Orosz removed preterm and low-weight babies from the hospital cohort. She also limited the cohort to only mothers who received some prenatal care, which also makes it a more comparable cohort to the homebirth mothers who also likely received prenatal care.
After a couple sentences about how she chose what filters to use on the CDC data, Prof. Orosz jumps right in to giving us her results for neonatal mortality:
For babies born in a hospital during the period 2007-2010, weighing at least 2500 grams, whose mothers received some prenatal care, the neonatal death rate was 0.71 per thousand. When deaths due to congenital anomalies are excluded, the rate drops to 0.40. The neonatal death rate for MANA STATS babies was 0.77 excluding congenital anomalies (13 deaths out of 16,950 births) and 1.30 including congenital anomalies (22 total neonatal deaths). The neonatal death rate excluding anomalies was significantly higher in the MANA STATS group (p=0.01). In addition, the number of neonatal deaths attributed to congenital anomalies was higher than expected, although the statistical significance of this difference was marginal (p=0.04).
Now, it’s all well and good to calculate a simple p-value to compare the two cohorts, but I’m not even sure why we are at the point of comparing them when I’m not even yet convinced that the cohorts are similar enough to be compared. If this were an actual scientific paper, the author would have given me a whole bunch of data about each cohort so that I can see exactly the ways in which they are extremely similar and the ways in which they differ and how that might affect the analysis, not to mention that this data is coming from two completely different sources. I want to know that all the definitions between the two datasets are the same. I want to know how problematic it may be that the hospital cohort defines “hospital births” as any birth taking place in a hospital while the homebirth cohort defines “homebirths” as a birth that was planned to be at home regardless of where it actually took place. In fact, over 10% of the homebirth cohort gave birth in a hospital. How might this affect the comparison? Was ethnicity or class taken into consideration? Urban vs rural? Were the homebirth mothers more likely to use alternative medicines and therapies in place of science-based interventions? In fact, the p-value part is the easiest part of the analysis, but it is meaningless if the cohorts aren’t similar enough to be compared. I want to see pages convincing me the hospital cohort is a good comparable control cohort and what weaknesses there may be in using that cohort before I see any p-values.
Prof. Orosz then immediately moves on to the comparison of intrapartum stillbirth rates:
The WHO estimates the intrapartum stillbirth rate across North America as 0.3 per thousand. It is likely that the true number of intrapartum stillbirths among low-risk infants delivered in the hospital is considerably lower. However, in the absence of any other solid data, I will use the value 0.3 per thousand as a maximum reasonable estimate. The intrapartum death rate for MANA STATS sample was 1.30, which is significantly higher. (p<.0001, highly significant.)
Somehow this is an even worse comparison. In the absence of any good control cohort, Prof. Orosz has just taken a big estimated stat that refers to a very large and diverse group of people (all US births) and compared it to a very small, very unique subset (17,000 planned homebirths) and then calculated a p-value and declared it is statistically significant. Of course it is statistically significant. This is a comparison between two completely different groups of individuals and is not controlled in any way. Not to mention that she makes this comparison without any link to where we can see the WHO’s estimates and evaluate them for ourselves. (EDIT: Commenters pointed out that if you click the link at the introduction to the Skeptical OB analysis where it says “you can find the complete analysis here” it brings you to a copy-paste of the blog post but with a couple sources at the end included the source of the WHO stat. So, when Dr. Tuteur converted it into a blog post she left out the sources at the end, not Prof Orosz)
Professor Orosz concludes the section on neonatal mortality rates with the following statement:
In other words, the expected number of deaths from causes other than congenital anomalies was at most 12, and the actual number of deaths was 35 (44 with anomalies included). It is clear that home birth substantially increases the risk of neonatal death and of intrapartum death.
Look, here’s the thing to remember here. We currently do not have a comparable hospital birth cohort to use to calculate relative risk between homebirths and hospital births. However, the absence of a good control doesn’t help parents who are trying to asses risk factors in order to make a decision on whether to have a home or hospital birth. Therefore, it’s ok to attempt to find a kinda, sorta, maybe, similarish cohort and do some back-of-the-envelope calculations.
The analysis that Dr. Orosz did is perfectly ok to do in the absence of better data. In the real world we don’t always have good data so we have to use whatever we can get. However, if you are going to do these fuzzy analyses, it is imperative you make the weaknesses very clear and use qualifying language. Just because your p-value is statistically significant doesn’t mean there’s actually an effect there if you are using a biased control group (Note: I’m using “biased” in the statistical sense – not as a knock against the author – frankly, I don’t care how biased a person is as long as their data is unbiased).
In this case the cohort groups are very different. They contain completely different types of people and do not even use the same definitions for very simple things like what counts as a hospital birth. In the case of intrapartum death, we don’t even have any cohort comparison group at all. Even so, this is the best data we’ve got. We can compare the numbers to get some rough estimates on potential risk increases, but to claim that “home birth substantially increases the risk of neonatal death” is absolutely not supported by the data. Instead, I would sum up Prof. Orosz’s results for this section as “risk of neonatal death and intrapartum death in the homebirth cohort seem slightly higher than what we generally see in hospital births. Therefore, there may be increased risk of neonatal and intrapartum death associated with homebirths although further research using a comparable hospital cohort will have to confirm.”
See what I did there? There’s no need to mislead. It’s ok to not have all the answers in the absence of clean data. Attempting to find correlation in real world data is a difficult and messy process. The messier and more unalike your cohorts, the more qualifiers you need when reporting your results. Doing anything less amounts to misrepresenting the results.
I also want to make it clear that it is ok to give your opinion after your results. This is a blog post and not a research study, after all. It’s ok to give your personal opinion that even if you can’t say for sure that there is a risk increase for having a homebirth based on the data, you believe there is a large risk increase. Just don’t try to pass off your personal beliefs about the data as if they were actual results.
The next section of Prof. Orosz’s post is far more egregious. She attempts to compare mortality rates for high-risk subsets in the homebirth cohort to…..well, she doesn’t really say what she’s comparing to. In some cases she does mention that she is comparing to the CDC hospital birth data again but in others she just throws out stats without mention of any source.
At one point she discusses breech births. In the homebirth cohort there were 5 deaths in 222 breech births. There were also a handful of breech births in which there was missing data. Prof. Orosz explains:
Disturbingly, the breech data were incomplete, as the authors explained: “There are 4 singleton pregnancies, 3 of which were breech presentations, for which all birth outcomes data are unavailable. These women began labor at home and then transferred to the hospital prior to birth. The midwives of record were contacted, and in each case the midwife did not accompany the mother, nor did the mother return to the midwife for postpartum care.” This missing data means that the breech death rate could in fact be as high as 36.0 per thousand, or 3.6%, which is similar to the breech birth death rate in the USA circa 1950.
In other words, Prof. Orosz is assuming that in all cases in which outcomes were missing for mothers who transferred to a hospital with a breech birth, the newborn died, though she gives no evidence for why this may be true. She then adds in these supposed deaths in order to come to her conclusion that 3.6% of the breech homebirths resulted in the death of the infant.
I don’t think I can possibly impress on you how not ok it is to just assume all missing data must mean dead babies. In the earlier section, the only thing I criticized Prof. Orosz for was not using enough qualifiers in her results statement to account for all the weaknesses in the data. In this case she’s going far beyond that to a point where it seems purposefully misleading.
Even if Prof. Orosz is trying to prove the point that homebirths are a bad idea in the case of breech births, there is no reason to manipulate the data. The 5 deaths in 222 breech births is already worryingly high. You don’t need to artificially inflate that number to get your point across.
Later, she does the same “assume all missing data is the worse possible outcome” trick when discussing the APGAR scores for the homebirth newborns. APGAR scores are commonly used immediately after birth to determine the health of the baby. The higher the score, the more signs of health the newborn is showing.
Only 245 infants (1.5%) had a recorded 5-minute APGAR of less than seven. Nationwide, 1.1% of babies above 2500 grams had a low APGAR as per CDC birth data, so the rate among the MANA STATS babies is significantly higher (p<0.0001 per binomial test) but not drastically higher.
However, APGARs were unavailable for 401 newborns. This missing data is concerning, as newborn health is a key outcome, and the only recorded variable with a greater number of missing data points was maternal education. If all 401 of those newborns in fact had low APGARs, the true rate of low-APGAR births would be 3.8%, which is three and a half times the national average.
No. Just no. Just because a baby’s APGAR score is missing in the dataset doesn’t mean it was a low score. Maybe some of them were, but we don’t know because they are missing. That’s kind of what “missing” means. Unless you have a really good reason to assume that all missing data = worst case scenario, you cannot just make that assumption.
In the piece I wrote last week I did not provide my personal opinion but there seems to be a lot of assumptions about it, pretty much all of them wrong. I’m going to give my opinion here if only so everyone will stop making assumptions. My personal opinion is that from what I’ve seen of the data reported on in the Journal of Midwifery and Women’s Health along with the CDC data as reported by Prof. Orosz and reviews of other studies I have read in various places (such as Emily Oster’s book Expecting Better, which I highly recommend) it seems as though there is either no increased risk or a very small increased risk to having a homebirth in a low-risk birth situation. High-risk births seem to have a possibly much higher risk of neonatal death in homebirths relative to hospital births. I think that almost no one should be having a high-risk birth at home except in exceptional cases. As for low-risk births, the situation is muddier but I would recommend not going with a homebirth. This is because I feel that in the case of uncertainty over the true risk of neonatal death, we should err on the side of assuming higher risk. I also don’t believe that the potential benefits of homebirths, such as less interventions or increased comfort, are in most cases worth any possibility of increased risk of neonatal death. I also understand that some women may be in a situation in which they value the benefits of homebirths more than I do or are not as worried about a possibility of increased risk, especially if they are minimizing those risks by using a qualified science-based midwife. I also think that it is problematic that some women who choose to have a homebirth do so for reasons that are not necessarily science-based or they don’t fully understand that there could be an increase in risk.
Again, this is my personal opinion and based partially in data, partially on guessing and assumptions regarding data I don’t have or doesn’t exist, and partially based on my personal valuation. Some of you may feel that benefits like comfort and lower rates of interventions are worth more than I am valuing them at. Some of you may feel that the chance that there could be even one death means that no one should ever be having a homebirth. Some of you may think that I’m not interpreting the data correctly or was too harsh on Prof. Orosz’s analysis and i09 for promoting it. Go in the comments and tell me why I’m wrong! Like everything in life, homebirths have some risks and some benefits. Let’s have an honest discussion about what these may be and how these should be weighted.
However, there is no reason to mislead on the real risks. If you are against homebirths and you think that any possibility of a risk is not worth taking, then argue that without resorting to inflating the numbers. It is my opinion as a skeptic that we should never, ever, ever misrepresent information in order to encourage others to take our side. We should never be personally attacking someone who has a differing opinion (and conversely not accusing others of personally attacking us when they merely disagreed with our views). We should never be threatening researchers or their families for publishing data we disagree with.
And i09, next time you come across a blog post that claims that they were able to do a complex cohort comparison study in just a couple days that completely overturns an established research paper done at a University and published in a respected peer reviewed journal, at least take the time to read the study first.
With that said, let’s have an honest discussion about homebirths.
Thank you Grounded Parents for letting me write one last guest post. <3