back

Notes

Guide to data sources

All the data on this website comes from official government sources.

Most data comes from large, annual UK-wide (or Great Britain-wide) government surveys, where either the survey datasets themselves or tables derived from them are then made available.  This is the case for data about income, work, pay and jobs.

Some data comes from administrative sources, which are typically separate for England, Scotland, Wales and Northern Ireland.  This is the case for data about housing, health and education.  Data on benefits is separately maintained for Great Britain and for Northern Ireland.

Data about income

Households Below Average Income (HBAI)

The Office for National Statistic's (ONS's) recommended source for data on household income, including income poverty, is HBAI, a dataset which is it itself derived from the Family Resources Survey (FRS).  HBAI covers the whole of the United Kingdom.  Despite its name, it covers all levels of income, not just low income.

HBAI is a sophisticated dataset which combines data about individuals, benefit units and households to calculate every household's income on a like-for-like basis.  This involves a process (called 'equivalisation') whereby household incomes are adjusted to reflect the fact that, to have the same standard of living, a family of four (for example) requires more money than a family of one but not four times as much.  It also contains data about individuals' pay rates, working hours, tax credits, housing costs, etc, thus allowing complex analyses.  Furthermore, data from HBAI can be linked with the data for the same individuals/households in FRS.

Finally, despite its name, HBAI is actually a dataset which covers all levels of income, not just low income.  As such, its uses are much wider then income poverty and cover the whole of the income distribution.

Use of the HBAI datasets themselves, rather than the annual published HBAI reports, is recommended because the UK Government sometimes changes and re-publishes the datasets without re-publishing the reports.  So, for example, old HBAI reports use a different method of weighting than the most recent reports.  When analysing data for below the UK level (e.g. for Scotland, Wales or Northern Ireland), use of the datasets rather than the published reports becomes mandatory as the published reports only contain a very limited amount of information at this level.

Scope Sources
UK-wide

All the HBAI datasets, both the individual datasets and the household datasets, are available from the UK data archive.  This website requires registration.  The Department of Work and Pensions also publishes an annual report giving many statistical analyses from the dataset.

All the FRS datasets are available from the UK data archive.  This website requires registration.

Note that:

British Household Panel Survey (BHPS)

BHPS is only relevant in two particular circumstances:

Scope Sources
UK-wide The BHPS dataset is a single UK-wide dataset covering all years and is available from the UK data archive.  This website requires registration.

top

Data about work and lack of work

Labour Force Survey (LFS)

ONS's recommended source for data on work, at both the individual and household level, is LFS.  LFS is a quarterly survey of individuals which covers the whole of the United Kingdom.  Every other quarter, a further household version of LFS is produced which includes additional variables relating to the combined work status of the adults in the household.

Scope Sources
UK-wide All the LFS datasets, both the individual datasets and the household datasets, are available from the UK data archive.  This website requires registration.

 

Alternatively, the Nomis website provides a 'wizard query' facility which allows the user to select the data of interest (e.g. LFS) and to define the rows and columns for the table to be presented.  This website requires registration.

Note that:

Annual Population Survey (APS)

Many people are interested in work statistics relating to individual local authority areas but the sample sizes in LFS are not sufficient to do this for small local authorities.  To rectify this, the Annual Population Survey was invented for local authority-level analyses, effectively LFS with boosted samples for small local authority areas.

Scope Sources
UK-wide All the APS datasets are available from the UK data archive.  This website requires registration.

 

Alternatively, the Nomis website provides a 'wizard query' facility which allows the user to select the data of interest (e.g. APS) and to define the rows and columns for the table to be presented.  This website requires registration.

Note that

Out-of-work benefits

Data on benefit recipients is maintained separately for Great Britain and for Northern Ireland:

Scope Sources
Great Britain-wide The DWP tabulation tool, which allows the user to select the benefits of interest and to define the rows and columns for the table to be presented.  This provides data down to the level of the individual local authority.

 

For data at a lower geographic level, a variety of Excel spreadsheets are available from DWP neighbourhood statistics.

Northern Ireland Various tables are available from the Department for Social Development website

top

Data about pay

Annual Survey of Hours and Earnings (ASHE)

ONS's recommended source for data on pay, including low pay, is ASHE.  ASHE covers the whole of the United Kingdom.

The actual ASHE dataset is not publicly available.  Outside researchers therefore have to rely on the UK Government's published tables.  In addition, researchers can place data requests to ONS but these are largely restricted to geographic breakdowns for those tables which are only published on the UK-wide basis.

Scope Sources
UK-wide ONS ASHE tables
Northern Ireland Department of Enterprise, Trade and Investment website

Note that:

Labour Force Survey (LFS)

ONS does not generally recommend use of LFS for low pay statistics because of the high non-response rate.  However, because the dataset is publicly available, it allows for much more flexibility of analysis than ASHE.  It is, for example, the dataset to use for analysis by ethnic group.

Scope Sources
UK-wide All the LFS datasets are available from the UK data archive.  This website requires registration.

Tax credits

Data on tax credits is available on a UK-wide basis from HM Revenue & Customs.

Scope Sources
UK-wide HM Revenue & Customs website, which provides a number of pre-defined tables.

top

Data about jobs

Labour Market Statistics (LMS)

ONS's recommended source for data on jobs is Labour Market Statistics.  These statistics are, in turn, derived by combining data from a variety of surveys including the Labour Force Survey (a survey of people) and the Annual Business Inquiry (a survey of employers).  LMS covers the whole of the United Kingdom.

Scope Sources
UK-wide Labour Market Statistics are available at the ONS website.

Note that the LMS estimates are subject to substantial revision from time to time.

Annual Business Inquiry (ABI)

Labour Market Statistics only provide a limited number of job analyses.  For more detailed analyses, ONS's recommended source is ABI.  ABI covers Great Britain, with the equivalent survey in Northern Ireland being called the Quarterly Employment Survey.

Scope Sources
Great Britain-wide The Nomis website which provides a 'wizard query' facility which allows the user to select the data of interest (e.g. ABI) and to define the rows and columns for the table to be presented.  This website requires registration.
Northern Ireland Department of Enterprise, Trade and Investment website, which provides a number of pre-defined tables.

Labour Force Survey (LFS)

ONS does not generally recommend use of LFS for job statistics because it is a survey of people rather than jobs.  Difficulties therefore arise when people have more than one job or where a job is shared by more than one person.  Rather, it is precisely to deal with these difficulties that LMS combines data from a variety of sources including LFS to come up with job estimates.

Scope Sources
UK-wide All the LFS datasets are available from the UK data archive.  This website requires registration.

top

Data about education

Schools and their pupils

School-related education data is the responsibility of:

Each of these organisations has a website where most of the available data is published in a number of annual reports.

Scope Sources
England Department for Education website.  More detailed data, at both school-level (performance tables) and pupil-level (National Pupil Database) is available on request.
Scotland Scottish Government website
Wales Welsh Assembly Government website and the Welsh National Pupil Database
Northern Ireland Department of Education website

Labour Force Survey (LFS)

LFS includes data about the highest level of qualification of each working-age person surveyed.  As such, it provides a source for data on the overall educational qualifications of the working-age population.

Scope Sources
UK-wide All the LFS datasets are available from the UK data archive.  This website requires registration.

top

Data about health

Births and deaths

Data about births and deaths is the responsibility of:

In this context, the only UK-wide source of health data used to be an annual ONS publication entitled Key population and vital statistics, which provided data on births and deaths by local authority, but this publication has now been discontinued (apparently for cost reasons)

Scope Sources
UK-wide Key population and vital statistics
England and Wales the publications DR and child mortality
Scotland General Register Office for Scotland website
Northern Ireland Northern Ireland Statistics and Research Agency website

Other health data

Health data other than births and deaths is the responsibility of:

In addition, there is an annual health survey of the population in England and equivalent but less frequent surveys in Scotland and Wales.  Since 2003/04, the Welsh Health Survey has been conducted annually.

Finally, various ad hoc analyses can be found in the quarterly ONS publication entitled Health statistics quarterly.

Scope Sources
UK-wide Health statistics quarterly
England All the Health Survey for England datasets are available from the UK data archive.  This website requires registration.
Scotland ISD Scotland website
Wales All the Welsh Health Survey datasets are available from the UK data archive.  This website requires registration.
Northern Ireland Only limited data is available at the Department of Health, Social Services and Public Safety website but more data is available from them on request.

top

Data about housing

Housing statistics

Housing data is the responsibility of:

In addition, DCLG collates a variety of UK-wide housing data in its live tables.

Scope Sources
UK-wide and England DCLG live tables
Scotland Scottish Government website
Wales Welsh Assembly Government website
Northern Ireland Department for Social Development website

House Condition Surveys

These are surveys which combine a physical survey of the house with an interview of the households.  As such, they can be used to analyse how the characteristics of houses vary according to the characteristics of households (e.g. how does the energy efficiency of houses vary by household type or tenure).  There is an annual house condition survey in England and equivalent, but less frequent, surveys in Scotland, Wales and Northern Ireland.

Scope Sources
England All the English Housing Survey - stock data datasets are available from the UK data archive, with the datasets prior to 2008 being available from a different part of the archive.  This website requires registration.

Household Surveys

These are general annual household surveys which include questions about housing.  There are separate annual household surveys for each of:

In addition, there is a smaller annual survey covering the whole of Great Britain called the General Lifestyle Survey (formerly called the General Household Survey).

Scope Sources
England All the English Housing Survey - household data datasets are available from the UK data archive, with the datasets prior to 2008/09 being available from a different part of the archive.  This website requires registration.
Scotland All the Scottish Household Survey datasets are available from the UK data archive.  This website requires registration.
Wales All the National Survey for Wales datasets are available from the UK data archive.  This website requires registration.
Northern Ireland All the Continuous Household Survey datasets are available from the UK data archive.  This website requires registration.
Great Britain-wide All the General Lifestyle Survey datasets are available from the UK data archive.  This website requires registration.  ONS also publishes an annual report giving many statistical analyses from the dataset.

top

Data about crime

Crime data is the responsibility of:

Prevalence of crime and its victims

The Government's recommended source for data on the prevalence of crime is the official crime surveys rather than police-recorded crime statistics.  This is important because the analyses of the crime surveys tends to result in very different answers than analyses of police-recorded crime statistics, largely because the latter have been affected by changes in recording practice from year to year.

There is an annual crime survey for England and Wales - oddly called the British Crime Survey although it does not include Scotland - and equivalent but less frequent surveys in Scotland and Northern Ireland.

Scope Sources
England and Wales All the British Crime Survey datasets (which do not cover Scotland) are available from the UK data archive.  This website requires registration.  The Home Office also publishes an annual report giving many statistical analyses from the dataset.

Perpetrators of crime

As part of the recent move to locate all aspects of the criminal justice system with the Ministry of Justice, it is now responsible for statistics on the perpetrators of crime.  In this capacity, it publishes a variety of annual reports.

Scope Sources
England and Wales Ministry of Justice website - statistics section

top

General data about households

These are general annual household surveys where the precise set of questions asked, and thus the analyses possible, vary from survey to survey.  The major annual household surveys are:

Scope Sources
UK-wide All the Family Resources Survey datasets are available from the UK data archive.  This website requires registration.
England and Wales All the Citizenship Survey datasets are available from the UK data archive.  This website requires registration.
England All the English Housing Survey - household data datasets are available from the UK data archive, with the datasets prior to 2008/09 being available from a different part of the archive.  This website requires registration.
Scotland All the Scottish Household Survey datasets are available from the UK data archive.  This website requires registration.
Wales All the National Survey for Wales datasets are available from the UK data archive.  This website requires registration.
Northern Ireland All the Continuous Household Survey datasets are available from the UK data archive.  This website requires registration.

Note that some researchers erroneously use the British Household Panel Survey, General Lifestyle Survey or Living Costs and Food Survey for general UK-wide or Great Britain-wide household analyses.  Except in isolated cases, however, their use in this regard has been supplanted by the much larger Family Resources Survey.

European data

A wide range of indicators comparing the countries in the European Union are available from the Eurostat website.  Those relevant to poverty and social exclusion are called the 'Laeken indicators'.  It is not clear precisely how comparable all the data is between countries.

top

Scope Sources
EU-wide All the European Union indicators are available from the Eurostat website.

top