Data science in local government
Data scientists are changing the world. Computing power and artificial intelligence are accelerating at a pace that will soon make today’s technology look like toys. This is pretty exciting. Particularly if you’re working in a tech startup. But what about if you work in local government?
Data science in the public sector is already lagging massively behind the private sector in terms of skills, technology and scope for innovation. And this gap is only going to grow wider without investment.
In the last few weeks I’ve visited 20 Local Authorities across England, from the rural southwest to northern city centres and seaside towns. I’m writing this as a quick reflection on the data science capabilities of local government.
I’ve been meeting Analysts who work with Children’s Services data. This is the data that informs decisions about the lives of the most vulnerable children in the country.
Some context: In 2016 about 390,000 children in England received support and protection from the Children’s Social Care system. Of these around 70,000 children are ‘looked after,’ which means that the state is their legal parent. More than 50% of these children entered state care as a result of abuse and neglect (data here).
So this is pretty important analysis that these guys are doing. Their ability to analyse this data set has direct implications for the lives of the most vulnerable children in our society.
Who are these analysts?
These local authority data analysts are the good guys. And they’re doing their jobs for all the right reasons. They care about what they do, and the results they get. But they’re also pretty overworked. Often they’ve seen big cutbacks to their teams, with skills and experience lost. And with analysis teams being consolidated across areas of expertise, it’s often the case that the analyst in charge of compiling and analysing a data set is completely new to it in the last year.
This generally results in quite a healthy and positive approach to change, as fresh sets of eyes bring ideas and optimism. But many have a lot of work to just to maintain current levels of reporting. There’s rarely time to take on exploratory analysis that might lead to new insights.
What skills do they have?
In terms of background there’s a broad range of skills and experience in these teams. This is a great positive because it means there isn’t a monoculture. But it also means that in some teams there are skills gaps.
The teams are often conscious of the skills they’re missing and keen to find ways to bridge the gaps. Many are taking to online platforms like Coursera and Futurelearn to up-skill themselves in their own time in areas like;
- Data analysis and statistics
- Process mining
- Machine learning
- Telling better stories with data
- Data visualisation
- Python
- SQL
From my conversations so far, I’ve started to identify 4 different types of analyst. These are just loose personas at this stage, but hopefully they’ll give you an idea of the types of people who are in analyst roles and the sorts of things they’re doing:
The expert administrator
A lot of analysts are from an expert administrative or generalist background. They often have deep expertise in the human side of the data (where it comes from) and have good relationships with the social workers who enter it. They tend to be focussed on the operational aspect of forming data sets into pre-defined reporting structures for senior management. They don’t tend to view the data outside of the framework that has already been defined. Their role is generally focussed on data management and the timely delivery of reporting. They’re not often asked to provide any detailed analysis or narrative alongside their reporting.
The data grinder
You can tell from the bags under their eyes that the data grinders know a strong cup of coffee when they see it. These guys have got mad excel skills. They build macros and interfaces and dashboards. Often on an insane scale. They’re building these as tools for their own analysis, and occasionally they’re shared with highly engaged senior managers who can use them to do their own self service analysis and performance tracking. These spreadsheets are things of beauty.
The data grinders are pushing themselves to build increasingly sophisticated analysis with the resources and skills they have at their disposal. This is often genius, because the whole sector pretty much works off spreadsheets, including the pre-formatted data submissions that local authorities deliver to central government. So these sheets can suck in standard templated data. And everyone sort of understands how they work and can use them. Until something in an obscure macro somewhere breaks…
Most data grinders are aware that they could do more if they were able to invest in using more advanced data analysis tools. They are often the ones learning new coding languages from online courses. But they hit blockers when they then try to use these internally, because the infrastructure (both technical and in terms of team support) might not exist to support them.
The data enabler
These guys are lucky enough to have data warehouses and analysis and reporting platforms at their disposal. But they’re under-resourced and in demand from their high performing management teams who have invested in their tech capacity. They’re working their socks off trying to provide all the reporting that’s being demanded to drive service improvement, and they’re making it as intuitive and self service as possible. Their hope is that in a year they can let that run itself…
They can see a glimmer of light at the end of the tunnel, that at some point they will have reporting under control and will be able to focus on what they really want to do — exploratory analysis that has the potential to transform the way their service is delivered by uncovering deep insights. Like being able to do predictive analysis to identify children who are at risk of abuse and neglect, or knowing the right intervention to commission at the right time to get positive social outcomes.
The data enablers are on the cusp of being able to move from doing power-reporting to doing analysis — and becoming proper data scientists rather than BI support.
The innovator
The lucky few. A small number of analysts have the training and resources to genuinely innovate as data scientists. But this is often as part of an externally supported programme rather than being as a result of a cultural change in their organisation. Often they are working closely with one senior advocate who has to work hard to protect their time and budget from ‘business as usual’.
What resources do they have at their disposal?
There’s a massive disparity between the resources available to analysts. You can see it when you first arrive at the council building. Some are working in dilapidated, downright depressing environments of stained ceiling tiles and once-temporary-now-permanent office furniture. They have large beige desktop computers running seriously out of date software. Others are in shining new glass and steel buildings full of jauntily coloured feature walls and touchscreen room booking systems.
The environment tends to reflect the technical infrastructure and resources available. Despite many analysts being engaged in the wider data science world, reading blogs and thought leadership on cutting edge technology, most don’t have access to it at work. There’s an increasingly bizarre disparity between the ‘personal tech stacks’ and ‘work tech stacks’ of local government employees. Teams might communicate on whatsapp or work on personal Python projects at home, but at work they’re using versions of Internet Explorer that remind me of the sound of dialup.
How do they see the future
Global corporations are building analytics and AI platforms to consume your online data and maximise their revenues. Those guys have got the best resources imaginable in terms of computing power, funding, and easy access to innovation networks to pick up on the next big thing. Public sector data scientists should have access to the same resources.
We’re on the cusp of a massive transformation in the ways that data can be used to improve social outcomes. Children’s Services is one of the areas where the biggest difference could be made.
Local authority analysts (I’m calling them data scientists) can see this potential and they’re pushing towards it. But often they simply don’t have the capacity right now to be focussed on much beyond the next few months. Many analysts are isolated and don’t feel like they’re part of a wider community who are thinking about these things. Skills, ideas, and best practice are not being shared widely.
What the sector needs is the resources for its data scientists to collaborate effectively as a community. That means shared skills, shared technology and a shared approach to innovation.
