The Lord has shaped me to be someone who values data. Over the years I have realised that it is so easy to be driven by warm-hearted anecdotes, and generalise our views, and formulate strategy, accordingly. So, for many years, I have been delighted to dig into the depths of the rich NCLS data and seek to be driven, with God’s help, to be more objectively grounded.

However, at this time, I find myself somewhat ambivalent to delve into the depths of the 2021/2 NCLS data, the full reports of which have been released over the summer.

I realise that the source of this unease goes back to the extraordinarily COVID impacted circumstances in which the survey was done. Other than deferring the entire NCLS survey for a year or two, there is not much that could have been done.

However, this meant that the survey was done either during COVID lockdowns or at the very time that we were emerging from COVID lockdowns. At this time, there were a wide variety of reactions about COVID, vaccines, and about how church leaders handled the situation. At the time, in my own context, I was eager to push conducting the NCLS survey as far away from COVID as I could. I wanted to use the time, money and effort of the once every 5 years survey well, and that meant trying to minimise the one-off and extraordinary COVID circumstances.

The window in which the survey was done was a time when many were still on livestream, or whatever online offering respective churches were making available.

I remember years ago, as part of my Doctor of Ministry studies, doing a class on the importance and practice of data collection. Yes, it was a hard slog (apologies to those who love this kind of thing). However, a key lesson was that data analysis must be based on robust data collection. That includes minimising bias. In cases of comparison surveys, like the NCLS, that includes data collection under similar conditions. (Hey, I remember something from those classes!).

It seems to be a given that those watching online did not (I would argue probably cannot fully) engage in the same way as in person. So many have talked to me about how hard it was to do so, the ready distractions in all kinds of forms. It just seems too much to reasonably expect everyone sitting at home and tuning in to scan the QR code and do a 15-20 minute survey in the same way that someone would sitting in person, in church, in a dedicated slot to do so.

This reality seems to be borne out in the results. As I looked at my church’s results, I just found myself second-guessing at so many points. Is the data a genuine indication of a shift that I need to prayerfully engage with? Or, is the data an indication of the collection bias? At the level of the Sydney Anglican Diocese, there were a whopping 38% fewer NCLS forms completed. At my church, despite our best attempts, we saw 27% fewer forms completed.

Here are some of many, many examples of the consequential second-guessing that I found perusing the results:

  • 11% more people at my church are tertiary educated (the Diocese saw a similar result). Is this indicative of a trend in society and/or our churches? Or, does this mean that tertiary educated people were more inclined and able to complete the survey? I ended up in a similar place when I looked at the proposed shifts in ethnicity.
  • The all important “newcomer” statistic has apparently declined by some 40% – this is the same both for my church and the Diocese. This is important because it is some measure of those coming into church life. However, does this mean that there are fewer “newcomers”? What do I do with this given the COVID lockdowns, even if that is the case? Or, could it be that “newcomers” make up a disproportionate number of people who did not complete the survey?
  • 5% more people say they are at church every week. This is great, if it is correct. Those I know who have been able to do robust analysis in this area (from other data sources) have concluded that church attendance rates have declined post-COVID. Could it be that the more regular ones at church were more inclined to complete the survey?
  • The sense of belonging to a church has declined substantially. Similarly, faith-sharing actual rates (beyond just being willing) have declined. Here the temptation or danger is slightly different. That is, it is so easy to write this off to COVID, and it is very hard to get to underlying factors or trends.
  • I am delighted to see increases in awareness and commitment to our church’s vision. But how is this impacted, if at all, by the bias of those who chose to complete the survey?

I could go on. I found the same reactions to so much of my NCLS report. It was so easy for me to either rightly question the data given the self-selection bias, write off the shifts due to COVID, or just get confused. Perhaps controversially, this set of seemingly objective data makes it possible to justify whatever I might want. And, after the discouragements of the COVID years, many will find it impossible not to do exactly that.

Now, for the record, I do think that there is some value in the 2021/2 NCLS reports as we have them. They did, for example, indicate areas of respective value and desire in church life. My point is simply that the measures of historical comparison, the very things we have been accustomed to using NCLS for, are fraught with data collection and interpretive danger on this occasion. Perhaps a wise use of resources would be for each church to pay for a one-off full NCLS to be done toward the end of 2023.  

Finally, whilst the particular 2021/2 NCLS data set may be of limited value, we must continue to value data in church life. It is all too easy, and tempting, to be driven by selected anecdotes. But well-collected and interpreted data can help us use the limited resources that the Lord has entrusted to our care well and wisely as we seek to reach all people with the life changing Gospel of Jesus Christ.