10 Insights You Should Know About the UK Housing Market
I presented my insights reports a few days ago. You can have a look if you want. — Link
Introduction
When the pandemic happened unexpectedly, the UK housing prices were soaring, followed by the increasing stock prices. I was very intrigued by the phenomena.
Some said increasing housing prices could be the “cheap money” floated into deflationary assets, as governments printed money to tackle the public health crisis in the short term. Others said more demands than supplies for nice houses as many companies were implementing ‘Working From Home’ policies. I started to ask why but the best way to study is to analyze historical patterns.
The real estate economy plays a vital role in a country. My father runs a real estate developing company. He told me how the real estate industry affects China economy. For example, the upper stream in the industry would be manufacturers for building materials, construction companies, etc, while the downstream is marketing&sales companies, etc. If a construction project started, everything runs like a vehicle moving forwards. More construction projects mean more job supplies. More employees gain incomes from their labours, investors achieve returns from the investment, governments have extra funds by involving legal regulations and taxation. The real estate industry is an important part of gross domestic product(GDP) in many countries.
My initial intention was to see how housing markets can reveal economic growth. However, this quest would cost too much time to reach the aim. Instead, the main focus of this project is data visualization regarding the UK housing market 12-year data analysis.
Data selection
I spared efforts into the dataset selection on Kaggle, a website for data science practitioners to explore open datasets and publish notebooks. I summarized what I thought about datasets into a tweet thread.
Having gone through the selection, I picked up UK Housing Price Paid data, from 1995 to 2017.
In this article, I’d try a different approach to demonstrate a few insights I concluded in this project. For a more structured method, you can check out my notebook. https://jovian.ai/shenghongzhong/exploratory-data-analysis-uk-housing-price
Understanding data
As for this dataset, we have 11 columns such as Transaction Unique Identifier, Price, Date of Transfer, Property Type, Old/New, Duration, Town/City, District, County, PPDCategory Type, Record Status — monthly file only.
Since this is a blog post, I won’t spend too much ink describing columns individually. To find out more, you can have a look at my notebook on jovian.ai.
I’d love to demonstrate some interesting learning points about the dataset:
Processing date
Data scientists often process data-wise data in the real world. Time is an important factor. Whenever we start a project, our boss would ask a question How much time can you get it done?
In general, we would create 4 extra columns to indicate year, month, day, weekday for further use, which is helpful to filter data by specifying year, month, day to use memory efficiently. In this case, we won't use day and weekday to save more RAM.
Property type
In the UK, 5 common types are shown in the slide. Since I’m not from here, I have done basic research about the dataset given by Kaggle.
According to the description on Kaggle, we have 5 property types. To demonstrate in a better way, I gathered the information into one slide.
Price
It’d be ridiculous if we didn’t discuss house prices. The sale prices of properties were stated on the transfer deed. I didn’t know the transfer deed until this project. It’s something like a contract approved by HM Land Registry, the department in the UK .It looks like this picture.
Mean V.S. Median housing prices
I learned that price is a continuous type, which means we can use a box plot to understand the distribution of prices. In the financial world, outliers are inevitable. Asset prices are varying and fluctuating in the market. Some properties in London could be ridiculously expensive, whereas some in rural areas in the UK could be unbelievably cheap.
It is less accurate to represent housing prices with the mean price. Note, the Michael Jordan Fallacy affects the mean prices. Bear in mind, we’re analyzing the housing market in the whole UK. Some properties could be sold at very high prices in London, whereas some properties might be sold at 1 pound in Liverpool.
To detect outliers, we use the box plot using the random sampling technique. We look at the population by using this sampling technique as long as each sample has an equal chance to be picked up. By the random sampling technique, we can roughly understand the shape of the whole dataset.
In our case, we randomly selected 10% out of our dataset, which is 2 million samples out of 22 million data points of housing prices.
As we can see from the screenshots, the statistics from 10% samples are as follows:
- Max = £98.9 million
- Min = £1
- 𝑄3 = £412.5K
- Median = £130K
- 𝑄1 = £75K
- 𝐼𝑄𝑅 = £337.5
After the random sampling, we aimed to grab a basic sense of the population for the dataset.
We know the maximum value is £98.9 million and the minimum value is £1. However, this doesn’t mean there is only 1 data point for £98.9 million. Multiple house prices could be equal to £98.9 million.
Next, we know the third quartile is £412.5K. This means in the sample dataset, 75% of housing prices are below £412.5K. In contrast, the first quartile is £75K meaning 25% of housing prices is below £75k.
Location
Notice: the dataset does not contain any information about Scottland, Northern Ireland.
As I’m from China, I taught myself geography knowledge about the UK. In China, we don’t use the term “country”. Instead, we say “province” to refer to “country”.
Britain consists of four countries: England, Scotland, Wales and Northern Ireland. England is divided into 48 ceremonial counties. Scotland was divided into 33 counties, Wales has 22 counties, and Northern Ireland has 6 counties.
Each county has their cities and towns. On Wikipedia, places had been granted city status by letters patent or royal charter. There are currently a total of 69 such cities in the United Kingdom: 51 in England, 7 in Scotland, 6 in Wales, and 5 in Northern Ireland.
It’d be interesting to see house sales data matching on the UK map. At the beginning of the project, I was concerned that the dataset was too tidy to practice data cleaning. Whenever you have a dataset outside you would certainly have chances to practice.
Tidying data for merging
One of the tricks I learned is to convert all location data into the same format. i.e. lower cases. This trick can save you lots of time but yet you should expect to clean data manually for text data. For example, You may encounter a location in your dataset, “City of Bristol”, whereas the dataset you attained somewhere else contained data like “Bristol, the city of.” This happened to me and I had to clean up manually due to the limited time.
Here is a screenshot for the output because the dataset is too large to render in Google Colab.
Other columns such as Duration, Old/New type, Record Status, PPDCategory Type, Record Status are not included in this post. However, you can check out them in my notebook to see how I explore them. My notebook: Analyzing 12-year Housing Prices in the United Kingdom
Insights
House sales were driven by political and financial events
Looking at monthly house sales, it’s easy to spot on swings during the financial crisis of 2007–08, also called the subprime mortgage crisis. In a nutshell, people were buying overvalued houses and investment banks designed Mortgage-Backed Securities (MBS) for reselling those debts. the financial instrument designed by the Wall Street drove the world into a panic mode. There were lots of discussions about the global financial crisis in this post on Reddit. Give a read if you want — Link
Another spike in house sales was around March 2016. My inference is Brexit. British people started to vote for leaving the EU or not around March, 2016. Here is what I think: The announcement of Brexit created an uncertain expectation towards the future of the United Kingdom. The business was planning to relocate and European businessmen sold properties for returning home countries. To some level, it explains why the spike happened.
50% house sales in the North of England
The first thing you might think is London is definitely the top 1 house sales city.
Yep, you’re right. London is the top 1 house sales city. Besides London, there were cities like Manchester, Bristol and Birmingham following with London. My educated guess would be policies in those cities attracted people for migration or cheaper expenses of living.
Another interesting point is out of the top 10 cities, 5 places are in the North of England, i.e. Nottingham, Leeds, Liverpool, Sheffield, Leicester. I’m intrigued to know the population growth over years in these places. It could be my next project!
The popular property type depends on locations
Without data, we would probably guess flats or terraced houses, as many make conclusions based on the living experience. However, they are both right and wrong. It depends on where we are taking about.
According to the description, we have 5 types as follows:
- D = Detached
- S = Semi-Detached
- T = Terraced
- F = Flats
- O = Others
The total number of house sales by property type in the UK
In the whole dataset, we can see the terraced house type is the most popular in the UK, but it’s a different scenario when it comes to London. Hence, where we are talking determines our answers to the most popular property type.
The total number of house sales by property type in London
From 1995 to 2017, flat had been sold the most in London. For this chart, my experience told me that I can trust this data. Because the population in London involves immigrants and other young people. Flat type is a good solution to the high intensity of population.
Old properties were more popular than new properties
Note that the dataset only provided data to the date of June 30th 2017
The stacked bar chart shows the total number of house sales by the old&new type across years.
Between 1995 to 2007, house sales were fluctuating in general and remained high at around 1.2 million house sales.
Next, this followed by a big drop in 2008. One of the explanations was the UK housing market was affected by the global financial crisis. In 2008, the house sales were decreased by 48% in house sales compared to 2007. Next, house sales in the UK market was levelled off between 2010 to 2012. Gradually, the housing market recovered as the house sales went up since 2013.
The Financial Crisis hit the UK housing market during 2008–2012
The colours of house sales from 2008 to 2013 are very dark regarding the heatmap. It suggests that during the global financial crisis, a few people participated in selling properties. From June 2008 to June 2009, the dark colour revealed that the financial crisis was in the middle.
Also, seemly there may be a pattern where January seems not a good month to selling houses.
Let’s reflect on the price trend chart.
The line chart of housing prices shows, Since 1995, the housing prices in the UK went up and reached the highest point in 2007. Next, it followed by a slight drop in housing prices affected by the global financial crisis in 2008, it fell steadily but still similar to the level in 2006, then the price was bottom out in 2012.
I think it’s safe to say the downturn in housing prices paused people’s action in selling properties.
Not a year new properties being sold than new property
Note: this excludes the incomplete data in the year of 2017 as the dataset didn’t include housing prices after June 2017.
The short answer is no. There is no single year the number of new properties more than old properties regarding the dataset.
In the bar chart, I grouped new old property sales instead of stacking them to compare whether the new property sales had a better sale performance.
However, the chart told us that every year there were old property sales more than new property sales.
Counties surrounding London were preferable
Using a log scale enable us to see changes over years clearly.
House sales by counties over years gave us another picture, using a log scale could allow us to see yearly changes clearly.
Kent is the county close to London as many people commute to London for work and live outside London. Kent has been a popular place as we can see the level of house sales in the heatmap.
Also, we can see the popularity of Greater Manchester had been increasing. I noticed the property sales performance in Yorkshire had been very strong over years. I was wondering if this was something associated with the population growth?
The minimum&maximum prices, the sky and the floor
From the chart, we can see the minimum price could vary. They are £1 and £100. However, we see the maximum price reached the highest point of £98.76 million.
£100 million property
Having googled, I found out the most expensive property in 2016 was the mansion located in central London. Link
It’s easy to understand why it was very expensive. Because the mansion is in central London with massive space, swimming pools, etc. My friend said it could be someone from Dubai. However, “what’s the data? ”, I asked.
£1 houses
I don’t know about you, but I was very intrigued by one pound houses. My first reaction was “ What? £1 for a property in the UK??” It couldn’t be wrong anymore.
“£1 must be a mistake.” I thought and plotted a bar chart for those house sold at £1.
This can’t be mistaken, I said surprisingly. Since we have statistics for the one pound houses, there were quite many one pound houses. I started to brainstorm before my investigation.
Parents’ gifts
I heard lots of parents in the UK gave gifts to their children. Maybe they sent a house to their children.
Divorce
I heard the divorce rate in the UK was quite high. So I wouldn’t be surprised for this reason.
Companies bankrupt
Running a business is hard and the biggest risk is to lose the company. When the company went bankrupt, they would sell some properties to somewhere else or bank if they loaned money from banks with collaterals.
Avoiding taxes
Possible. But I believe UK GOV would have sort of systems, experts to recognize unfavourable economy behaviour.
Mistakes
It happened sometimes. Humans make mistakes in data entry.
Always check the data source
While I exhausted all reasons, I started to find the data source on the UK GOV. https://www.gov.uk/guidance/about-the-price-paid-data#data-excluded-from-price-paid-data
All my assumptions were defeated after examing the description of how UK GOV processed the dataset I used.
Continue Researching the myth of £1 houses
I was hooked by these 1 pound houses. Why were there houses sold at £1?
Besides those reasons I listed, this would never happen in China in my opinion. Chinese people see properties as the most valuable investment, the hardest assets. Also, in the current marriage market, males have to own a property to be married. They would rather hold them and put properties aside until the housing market becomes a bull market.
Having continued my investigation, I found out actually the GOV in Liverpool tried to solve the housing crisis by launching the One Pound House Scheme years ago.
Furthermore, they made documentary films! The £1 Houses: Britain’s Cheapest Street.
The one-pound ($1.25) homes program run by Liverpool City Council — the subject of a new TV documentary — is one of a handful of schemes by cities across Britain and Europe searching for innovative ways to solve housing crises. Selling homes for token sums has become a popular, last-ditch strategy for towns, cities and villages fighting depopulation and decay.
House sales in Manchester and Bristol were potential
The pie chart shows the most popular property type is flat in the top 10 house sales cities, whereas terraced houses are being sold the most for England& Wales.
I was wondering if there was a correlation between the population of England and Wales and the top 10 cities.
We could look at flat sales if we consider the property investment for the purpose of reselling in the future.
On the other hand, detached houses were accounting for 10.5%, whereas semi-detached houses are 21.2%. I wonder if the price can be the main factor for this difference, as prices for detached houses are generally more expensive.
Detached houses were more expensive than Semi-detached
You might think the statement here is pointless. Bigger house = Higher price. However, being a data scientist involves asking “stupid” questions and “smart” questions.
In the box plot, we can see clearly the median price of detached houses is higher than the semi-detached house. It’s not that surprising because a detached house is in general bigger than a semi one. However, knowing the price range is very helpful. At least, we can answer a specific question “How large is large?”
Let’s look at the Q3.
Semi-detached had Q3 of £187K, comparing with the detached house having Q3 of £285K. That’s a 52% difference. That’s much large.
In terms of property type flat, what’s the ratio for new/old flats among the top 6 flat sales cities?
First, we need to find out what the top 6 flat sales cities are.
If we exclude London, we can see actually the new flat sales in cities like Bristol, Leeds, Liverpool accounting for 5% ish over 12 years. The statistics about the capital city is weighted too much. My guess is the population in London could be larger than any other place in the UK. Somehow, this reveals an increasing demand for properties to live in London.
The chart demonstrates that Manchester is the second city where flats was sold the most, accounting for almost 12%.
I’m not surprised to see this increase. London is becoming more and more expensive so that many young people consider moving to Manchester. North England has been attracting young people for job opportunities and promising futures.
The second following city is Bristol which is in the west of England. It’s a good place for young families. I have friends moving into Bristol after having a baby. Yet, it’s my personal experience and it could be biased. Also, I’d love to live in one day!
Summary
1. London is a unique case from anywhere
Terraced type property was a popular choice in the UK over years, accounting for 30%(6.6 million) of 22 million house sales across England and Wales over 12 years. However, it depends on different locations. In London, house sales in flat were the most, accounting for 64.93% (1.1 million) out of 1.7 million all house sales in London.
We need to be careful to validate statistics. Because there are various differences in populations and land resources across England and Wales. It determines the price behavior in the housing market and hard to generalize one-shoes-fit-all conclusions.
2. The use of median value for financial assets should be cautious
As mentioned earlier, the mean value is not a good example for representing financial assets. However, the price as an indicator can show the general trend over years and helped us to explore the underlying relationships between house prices and other features such as old/new, property types, etc.
We also discover the housing market was pessimistic during the global financial crisis, no matter what metrics we can use.
3. Paying attention to global financial structures and the local political climates
Besides the global financial crisis, the housing market is also sensitive to regional politics. When the announcement of voting for Brexit the house sales were soaring. The underlying trigger could be many businesses and people started to plan for relocating if Brexit happened unexpectedly ( In fact, it happened).
We should pay attention to political news, new policies, etc. Do not wait until it becomes a post-truth. In the financial world, the speed of acquiring information determines a portfolio’s profit margins, due to information asymmetry.
4. Some underdeveloped areas are awaiting transformation in the UK
The most interesting in this project is when I discovered one pound houses. It’s an unbelievable event. I was so curious about the reasons why there were properties sold at £1. Yet, I believe the house crisis is a real problem in any place in the world.
One pound houses scheme was an attempt for the house crisis in Liverpool, but I believe the long-term vision should be conducted by us together. We should echo to the government and support them. Maybe more young people are willing to take the challenges to transform societies. Then we can create a bright future for our young generations.
Future work
- You can exclude London for deep regional analysis. People say London is entirely different from any other places in the UK and also can’t represent the whole UK. Many immigrants gathered in London for job opportunities and education. It makes sense if we can exclude London in future work to reach a generalized conclusion.
- I’d recommend you to use the population data from 1995 to 2017. This can help us to form a clear picture of the price behavior.
- I’d recommend narrowing down 10 regions with respects to the desired time period. i.e. (2007–2012) This can generate more interesting insights.
- I’d recommend studying the UK economy and gather data from the banking sector to uncover the correlation between housing prices/sales and the interest rate/mortgage/GDP. You can check out on the site — the Office for National Statistics
- I’d recommend doing a second project based on the UK Housing Price Paid dataset to train a model with machine learning techniques. You can follow instructions of chapter 2 on Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow where Géron analyzed the housing price in California.
Thank you for your time!
Reference
[1] HM Land Registry. UK Housing Price Paid, 2017 . https://www.kaggle.com/hm-land-registry/uk-housing-prices-paid
[2] Osbornep. UK House Price Analysis: Part 1.1, 2018. https://www.kaggle.com/osbornep/uk-house-price-analysis-part-1-1
[3] Aakash N S. Analyzing Tabular Data with Pandas, 2021. https://jovian.ai/aakashns/python-pandas-data-analysis
[6] Aakash N S. Data Visualization using Python Matplotlib and Seaborn, 2021. https://jovian.ai/aakashns/python-matplotlib-data-visualization
[7] Aakash N S. Advanced Data Analysis Techniques with Python & Pandas, 2021. https://jovian.ai/aakashns/advanced-data-analysis-pandas
[8] Aakash N S. Interactive Visualization with Plotly, 2021. https://jovian.ai/aakashns/interactive-visualization-plotly
[9] Jaramillo,J. Mapping the UK and navigating the post code maze, 2020. https://focaalvarez.medium.com/mapping-the-uk-and-navigating-the-post-code-maze-4898e758b82f
[10] Zhong, SZ. Geography Discovery In the United Kingdom , 2021. https://jovian.ai/shenghongzhong/the-geography-of-the-uk
[11] Plotly Documentation. https://plotly.com/python/
[12] the R graph gallery. Parallel Coordinates chart, 2018. https://www.r-graph-gallery.com/parallel-plot.html
[13] Aakash N S. plotly-line-chart, 2021. https://jovian.ai/aakashns/plotly-line-chart
[14] Nussbaumer Knaflic, Cole. Storytelling with Data, 2015
[15] Tufte,Edward. Visual Explanations, 1997.
