Notes
Guide to data sources
All the data on this website comes from official government sources.
Most data comes from large, annual UK-wide (or Great Britain-wide) government surveys, where either the survey datasets themselves or tables derived from them are then made available. This is the case for data about income, work, pay and jobs.
Some data comes from administrative sources, which are typically separate for England, Scotland, Wales and Northern Ireland. This is the case for data about housing, health and education. Data on benefits is separately maintained for Great Britain and for Northern Ireland.
Data about income
Households Below Average Income (HBAI)
The Office for National Statistic's (ONS's) recommended source for data on household income, including income poverty, is HBAI, a dataset which is it itself derived from the Family Resources Survey (FRS). HBAI covers the whole of the United Kingdom. Despite its name, it covers all levels of income, not just low income.
HBAI is a sophisticated dataset which combines data about individuals, benefit units and households to calculate every household's income on a like-for-like basis. This involves a process (called 'equivalisation') whereby household incomes are adjusted to reflect the fact that, to have the same standard of living, a family of four (for example) requires more money than a family of one but not four times as much. It also contains data about individuals' pay rates, working hours, tax credits, housing costs, etc, thus allowing complex analyses. Furthermore, data from HBAI can be linked with the data for the same individuals/households in FRS.
Finally, despite its name, HBAI is actually a dataset which covers all levels of income, not just low income. As such, its uses are much wider then income poverty and cover the whole of the income distribution.
Use of the HBAI datasets themselves, rather than the annual published HBAI reports, is recommended because the UK Government sometimes changes and re-publishes the datasets without re-publishing the reports. So, for example, old HBAI reports use a different method of weighting than the most recent reports. When analysing data for below the UK level (e.g. for Scotland, Wales or Northern Ireland), use of the datasets rather than the published reports becomes mandatory as the published reports only contain a very limited amount of information at this level.
| Scope | Sources |
|---|---|
| UK-wide | All the HBAI datasets, both the individual datasets and the household datasets, are available from the UK data archive. This website requires registration. The Department of Work and Pensions also publishes an annual report giving many statistical analyses from the dataset. All the FRS datasets are available from the UK data archive. This website requires registration. |
Note that:
- Whilst FRS comes with full documentation (like all UK data archive datasets), HBAI comes with very little. Furthermore, many of the HBAI variables have complex definitions and it therefore follows that one can only use HBAI after becoming deeply familiar with it. Such familiarity is difficult to develop but if you contact us (info@poverty.org.uk) then we might be able to help.
- Both HBAI and FRS make an important distinction between 'households' and 'benefit units', with most variables being defined in terms of one or the other but never both. The term 'benefit unit' is used to cover an adult and their spouse (if applicable) whereas the term 'household' is used to cover everyone living in a dwelling. So, a young adult living with their parents would count as one 'household' but two 'benefit units'. In analysing the rates of low income by work status or family type, the work status and family type is analysed by benefit unit whereas the income is analysed by household. These issues are discussed in more detail in the pages on choices of low-income threshold and households, families and benefit units.
British Household Panel Survey (BHPS)
BHPS is only relevant in two particular circumstances:
- When looking at persistent low income: BHPS is a panel survey and therefore can be used to look at how people's income changes over time.
- When comparing with the rest of the European Union: BHPS is part of an EU-wide set of surveys which are designed to be comparable.
| Scope | Sources |
|---|---|
| UK-wide | The BHPS dataset is a single UK-wide dataset covering all years and is available from the UK data archive. This website requires registration. |
Data about work and lack of work
Labour Force Survey (LFS)
ONS's recommended source for data on work, at both the individual and household level, is LFS. LFS is a quarterly survey of individuals which covers the whole of the United Kingdom. Every other quarter, a further household version of LFS is produced which includes additional variables relating to the combined work status of the adults in the household.
| Scope | Sources |
|---|---|
| UK-wide |
All the LFS datasets, both the individual datasets and the household
datasets, are available from the
UK data archive.
This website requires registration.
Alternatively, the Nomis website provides a 'wizard query' facility which allows the user to select the data of interest (e.g. LFS) and to define the rows and columns for the table to be presented. This website requires registration. |
Note that:
- LFS is a 'semi-panel' survey, with every household being interviewed for five consecutive quarters before being dropped. In reaction, some researchers, when producing annual results, merge the four quarters together and remove the duplicates before doing their analysis. This is, however, a time-consuming process which makes little difference to the answers and a more practical approach is to estimate the annual figures as the average for the four relevant quarters.
- Prior to 2006, the 'quarters' of LFS were Winter (December to February), Spring (March-May), Summer (June-August) and Autumn (September to November), rather than the four calendar quarters.
Annual Population Survey (APS)
Many people are interested in work statistics relating to individual local authority areas but the sample sizes in LFS are not sufficient to do this for small local authorities. To rectify this, the Annual Population Survey was invented for local authority-level analyses, effectively LFS with boosted samples for small local authority areas.
| Scope | Sources |
|---|---|
| UK-wide |
In principle all the APS datasets are available from the
UK data archive
but, in practice, some are either missing from, or late to, the archive.
This website requires registration.
Alternatively, the Nomis website provides a 'wizard query' facility which allows the user to select the data of interest (e.g. APS) and to define the rows and columns for the table to be presented. This website requires registration. |
Note that
- APS in only available for the years from 2004 onwards.
- APS also includes more general boosters for Scotland, Wales and Northern Ireland. For this reason, government departments in those countries often use it in preference to LFS. This is, however, problematic for outside researchers as these booster datasets are not readily available and a more practical approach for Scottish, Welsh and Northern Irish analyses - and one which gives very similar answers - is therefore simply to use LFS.
Out-of-work benefits
Data on benefit recipients is maintained separately for Great Britain and for Northern Ireland:
- Great Britain: Department of Work and Pensions (DWP).
- Northern Ireland: Department for Social Development.
| Scope | Sources |
|---|---|
| Great Britain-wide |
The
DWP
tabulation tool, which allows the user to select the benefits of interest
and to define the rows and columns for the table to be presented. This
provides data down to the level of the individual local authority.
For data at a lower geographic level, a variety of Excel spreadsheets are available from DWP neighbourhood statistics. |
| Northern Ireland | Various tables are available from the Department for Social Development website |
Data about pay
Annual Survey of Hours and Earnings (ASHE)
ONS's recommended source for data on pay, including low pay, is ASHE. ASHE covers the whole of the United Kingdom.
The actual ASHE dataset is not publicly available. Outside researchers therefore have to rely on the UK Government's published tables. In addition, researchers can place data requests to ONS but these are largely restricted to geographic breakdowns for those tables which are only published on the UK-wide basis.
| Scope | Sources |
|---|---|
| UK-wide | ONS ASHE tables |
| Northern Ireland | Department of Enterprise, Trade and Investment website |
Note that:
Non-vatable companies were only included in the ASHE survey for 2004 onwards. Data for 2003 and earlier is therefore not directly comparable with that from 2004 onwards. In particular, the data for 2003 and earlier is considered to underestimate the prevalence of low pay and should therefore not be combined with the data for 2004 onwards to produce time trend analyses. It is to partly fill this gap that ONS specifically publish overall UK-wide estimates of low pay by year, using ASHE data from 2004 onwards but a combination of ASHE and the Labour Force Survey for earlier years.
ASHE only goes back to 1997, and the data for 2001 and earlier is more limited than for 2002 onwards.
Many of the tables can also be obtained by the Nomis website, using the wizard query facility. This website requires registration.
Labour Force Survey (LFS)
ONS does not generally recommend use of LFS for low pay statistics because of the high non-response rate. However, because the dataset is publicly available, it allows for much more flexibility of analysis than ASHE. It is, for example, the dataset to use for analysis by ethnic group.
| Scope | Sources |
|---|---|
| UK-wide | All the LFS datasets are available from the UK data archive. This website requires registration. |
Tax credits
Data on tax credits is available on a UK-wide basis from HM Revenue & Customs.
| Scope | Sources |
|---|---|
| UK-wide | HM Revenue & Customs website, which provides a number of pre-defined tables. |
Data about jobs
Labour Market Statistics (LMS)
ONS's recommended source for data on jobs is Labour Market Statistics. These statistics are, in turn, derived by combining data from a variety of surveys including the Labour Force Survey (a survey of people) and the Annual Business Inquiry (a survey of employers). LMS covers the whole of the United Kingdom.
| Scope | Sources |
|---|---|
| UK-wide | LMS is published monthly and is available at the ONS website. |
Note that the LMS estimates are subject to substantial revision from time to time.
Annual Business Inquiry (ABI)
Labour Market Statistics only provide a limited number of job analyses. For more detailed analyses, ONS's recommended source is ABI. ABI covers Great Britain, with the equivalent survey in Northern Ireland being called the Quarterly Employment Survey.
| Scope | Sources |
|---|---|
| Great Britain-wide | The Nomis website which provides a 'wizard query' facility which allows the user to select the data of interest (e.g. ABI) and to define the rows and columns for the table to be presented. This website requires registration. |
| Northern Ireland | Department of Enterprise, Trade and Investment website, which provides a number of pre-defined tables. |
Labour Force Survey (LFS)
ONS does not generally recommend use of LFS for job statistics because it is a survey of people rather than jobs. Difficulties therefore arise when people have more than one job or where a job is shared by more than one person. Rather, it is precisely to deal with these difficulties that LMS combines data from a variety of sources including LFS to come up with job estimates.
| Scope | Sources |
|---|---|
| UK-wide | All the LFS datasets are available from the UK data archive. This website requires registration. |
Data about education
Schools and their pupils
School-related education data is the responsibility of:
- England: Department for Children, Schools and Families (formerly the Department for Education and Skills).
- Scotland: Scottish Government.
- Wales: Welsh Assembly (mostly) and the Local Government Data Unit Wales (pupil database).
- Northern Ireland: Department of Education.
Each of these organisations has a website where most of the available data is published in a number of annual reports.
| Scope | Sources |
|---|---|
| England | Department for Children, Schools and Families website. More detailed data, at both school-level (performance tables) and pupil-level (National Pupil Database) is available on request. |
| Scotland | Scottish Government website |
| Wales | Welsh Assembly website and the Welsh National Pupil Database |
| Northern Ireland | Department of Education website |
Labour Force Survey (LFS)
LFS includes data about the highest level of qualification of each working-age person surveyed. As such, it provides a source for data on the overall educational qualifications of the working-age population.
| Scope | Sources |
|---|---|
| UK-wide | All the LFS datasets are available from the UK data archive. This website requires registration. |
Data about health
Births and deaths
Data about births and deaths is the responsibility of:
- England and Wales: ONS.
- Scotland: General Register Office for Scotland (GRO).
- Northern Ireland: Northern Ireland Statistics and Research Agency (NISRA).
In this context, the only UK-wide source of health data is an annual ONS publication entitled Key population and vital statistics, which provides data on births and deaths by local authority.
| Scope | Sources |
|---|---|
| UK-wide | Key population and vital statistics |
| England and Wales | the publications DH2, DH3 and DH4 |
| Scotland | General Register Office for Scotland website |
| Northern Ireland | Northern Ireland Statistics and Research Agency website |
Other health data
Health data other than births and deaths is the responsibility of:
- England and Wales: ONS.
- Scotland: ISD Scotland.
- Northern Ireland: Department of Health, Social Services and Public Safety.
In addition, there is an annual health survey of the population in England and equivalent but less frequent surveys in Scotland and Wales. Since 2003/04, the Welsh Health Survey has been conducted annually.
Finally, various ad hoc analyses can be found in the quarterly ONS publication entitled Health statistics quarterly.
| Scope | Sources |
|---|---|
| UK-wide | Health statistics quarterly |
| England | All the Health Survey for England datasets are available from the UK data archive. This website requires registration. |
| Scotland | ISD Scotland website |
| Wales | All the Welsh Health Survey datasets are available from the UK data archive. This website requires registration. |
| Northern Ireland | Only limited data is available at the Department of Health, Social Services and Public Safety website but more data is available from them on request. |
Data about housing
Housing statistics
Housing data is the responsibility of:
- England: Department for Communities and Local Government (DCLG).
- Scotland: Scottish Government.
- Wales: Welsh Assembly (some) and the Local Government Data Unit Wales (mostly).
- Northern Ireland: Department for Social Development.
In addition, DCLG collates a variety of UK-wide housing data in its live tables.
| Scope | Sources |
|---|---|
| UK-wide and England | DCLG live tables |
| Scotland | Scottish Government website |
| Wales | Welsh Assembly website and the Local Government Data Unit Wales website |
| Northern Ireland | Department for Social Development website |
House Condition Surveys
These are surveys which combine a physical survey of the house with an interview of the households. As such, they can be used to analyse how the characteristics of houses vary according to the characteristics of households (e.g. how does the energy efficiency of houses vary by household type or tenure). There is an annual house condition survey in England and equivalent but less frequent surveys in Scotland, Wales and Northern Ireland.
| Scope | Sources |
|---|---|
| England | All the English House Condition Survey datasets are available on request from the Department of Communities and Local Government English House Condition Survey website, where the reports are also online. |
Household Surveys
These are general annual household surveys which include questions about housing. There are separate annual household surveys for each of:
- England: Survey of English Housing.
- Scotland: Scottish Household Survey (from 1999 onwards only).
- Wales: Living in Wales (from 2004 onwards only).
- Northern Ireland: Continuous Household Survey.
In addition, there is a smaller annual survey covering the whole of Great Britain called the General Lifestyle Survey.
| Scope | Sources |
|---|---|
| England | All the Survey of English Housing datasets are available on request from the Department of Communities and Local Government (david.wall@communities.gsi.gov.uk, 020 7944 3301). |
| Scotland | All the Scottish Household Survey datasets are available from the UK data archive. This website requires registration. |
| Wales | All the Living in Wales datasets are available on request from the Welsh Assembly (rhiannon.caunt@wales.gsi.gov.uk, 029 2082 5063). |
| Northern Ireland | All the Continuous Household Survey datasets are available from the UK data archive. This website requires registration. |
| Great Britain-wide | All the General Lifestyle Survey datasets are available from the UK data archive. This website requires registration. ONS also publishes an annual report giving many statistical analyses from the dataset. |
Data about crime
Crime data is the responsibility of:
- England: Home Office (prevalence and victims) and Ministry of Justice (perpetrators).
- Scotland: Scottish Government.
- Wales: Welsh Assembly.
- Northern Ireland: Department for Social Development.
Prevalence of crime and its victims
The Government's recommended source for data on the prevalence of crime is the official crime surveys rather than police-recorded crime statistics. This is important because the analyses of the crime surveys tends to result in very different answers than analyses of police-recorded crime statistics, largely because the latter have been affected by changes in recording practice from year to year.
There is an annual crime survey for England and Wales - oddly called the British Crime Survey although it does not include Scotland - and equivalent but less frequent surveys in Scotland and Northern Ireland.
| Scope | Sources |
|---|---|
| England and Wales | All the British Crime Survey datasets (which do not cover Scotland) are available from the UK data archive. This website requires registration. The Home Office also publishes an annual report giving many statistical analyses from the dataset. |
Perpetrators of crime
As part of the recent move to locate all aspects of the criminal justice system with the Ministry of Justice, it is now responsible for statistics on the perpetrators of crime. In this capacity, it publishes a variety of annual reports.
| Scope | Sources |
|---|---|
| England and Wales | Ministry of Justice website |
General data about households
These are general annual household surveys where the precise set of questions asked, and thus the analyses possible, vary from survey to survey. The major annual household surveys are:
- UK-wide: Family Resources Survey.
- England: Survey of English Housing.
- Scotland: Scottish Household Survey (from 1999 onwards only).
- Wales: Living in Wales (from 2004 onwards only).
- Northern Ireland: Continuous Household Survey.
| Scope | Sources |
|---|---|
| UK-wide | All the Family Resources Survey datasets are available from the UK data archive. This website requires registration. |
| England | All the Survey of English Housing datasets are available on request from the Department of Communities and Local Government (david.wall@communities.gsi.gov.uk, 020 7944 3301). |
| Scotland | All the Scottish Household Survey datasets are available from the UK data archive. This website requires registration. |
| Wales | All the Living in Wales datasets are available on request from the Welsh Assembly (rhiannon.caunt@wales.gsi.gov.uk, 029 2082 5063). |
| Northern Ireland | All the Continuous Household Survey datasets are available from the UK data archive. This website requires registration. |
Note that some researchers erroneously use the Expenditure and Food Survey, British Household Panel Survey or General Lifestyle Survey for general UK-wide or Great Britain-wide household analyses. Except in isolated cases, however, their use in this regard has been supplanted by the much larger Family Resources Survey.
European data
A wide range of indicators comparing the countries in the European Union are available from the Eurostat website. Those relevant to poverty and social exclusion are called the 'Laeken indicators'. It is not clear precisely how comparable all the data is between countries.
| Scope | Sources |
|---|---|
| EU-wide | All the European Union indicators are available from the Eurostat website. |