Predict the Selling Price of HDB Resale Flats

Problem Statement

There are multiple factors affecting the selling price of HDB resale flats in Singapore. Therefore, by using linear regression, I am interested in finding out how the selling price of a HDB resale flat changes based on its following characteristics in this mini exercise:

• its distance to the Central Business District (CBD)

• its distance to the nearest MRT station

• its flat size

• its floor level

• its remaining years of lease

The resources for this mini exercise can be found on my GitHub which includes the dataset, and the Python notebook files - data preprocessing, and data processing (including the building of linear regression).

Dataset

The following are the data sources used in this mini exercise:

• HDB Resale Flat Prices from Data.gov.sg (as of 3 May 2021)

• OneMap API

Data Preprocessing

There are 5 comma-separated values (CSV) files in the HDB Resale Flat Prices dataset in terms of different time periods - 1990 to 1999, 2000 to 2012, 2012 to 2014, 2015 to 2016, and 2017 onwards respectively.

In this mini exercise, all time periods are considered in the analysis. Hence, there is a need to combine all 5 different CSV files as one whole dataset.

import globimport pandas as pd

df = pd.concat([pd.read_csv(f) for f in glob.glob("./data/*.csv")], ignore_index=True)

As we can see from the dataset, it is not comprehensive enough to answer the problem statement of this mini exercise. Henceforth, there is a need to compute the remaining lease from this year (2021) onwards as well as complementing with the OneMap API to do geocoding in order to compute the distance between each flat and its nearest MRT station, and the distance between each flat and the CBD (based on Raffles Place).

Prior to conduct geocoding, missing values and duplicate values of the raw dataset would be removed. Furthermore, the address of each HDB flat is constructed in order to retrieve its geo-location.

df['address'] = df['block'] + " " + df['street_name']address_list = df['address'].unique() # to be iterated in order to retrieve the geo-location of each address

Geocoding is done using JSON requests to conduct querying in order to obtain the geo-location of each HDB flat, and the geo-locations of the MRT stations as shown in the picture below. With these, we can compute the distance between each flat and its nearest MRT station. Furthermore, with the geo-locations of all the HDB flats, we can compute the distance between each flat and the CBD.

The followings are the code template to retrieve data from OneMap API as well as the MRT stations that were used as a list to retrieve their geo-locations.

import jsonimport requests

query_string = 'https://developers.onemap.sg/commonapi/search?searchVal='+query_address+'&returnGeom=Y&getAddrDetails=Y' # define your query_address variable (e.g. HDB address)resp = requests.get(query_string)data = json.loads(try_resp.content)

The final plate to add to our dataset is to compute the remaining lease of each flat from this year onwards. HDB leases are 99 years long so as to compute the remaining lease, a new variable is defined as the following where the starting lease date variable is used:

df['lease_remain_years'] = 99 - (2021 - df['lease_commence_date'])

Ta-da! The new processed dataset is generated!

Data Wrangling

One necessary step to handle the newly generated dataset is to make sure each of the variables is in its correct data type respectively.

df['resale_price'] = df['resale_price'].astype('float')df['floor_area_sqm'] = df['floor_area_sqm'].astype('float')df['lease_commence_date'] = df['lease_commence_date'].astype('int64')df['lease_remain_years'] = df['lease_remain_years'].astype('int64')

df.dtypes

One of the factors to look into is the floor level. The floor level variable is a categorical variable that has different floor level ranges.

Therefore, the median is applied to map each HDB flat's floor level.

import statistics

def get_median(x):    split_list = x.split(' TO ')    float_list = [float(i) for i in split_list]    median = statistics.median(float_list)    return median

df['storey_median'] = df['storey_range'].apply(lambda x: get_median(x))

Now it is time to extract the relevant variables that answer the problem statement as our new data frame that is used for building the linear regression model.

#cbd_dist = CBD distance#min_dist_mrt = Distance to the nearest MRT station#floor_area_sqm = Flat size#lease_remain_years = Remaing years of lease#storey_median = Floor level#resale_price = Selling price (dependent variable)

df_new = df[['cbd_dist','min_dist_mrt','floor_area_sqm','lease_remain_years','storey_median','resale_price']]

Correlation Analysis

From above, we can infer that the floor size has the highest strength relationship that impacts the resale price as this relationship is positively moderate. Distance to the nearest MRT station has the lowest strength relationship that impacts the resale price as this relationship is negatively weak. Regardless of a positive or a negative relationship, the strength of the rest of the variables that impact the resale price is moderate.

Linear Regression

The latest dataset will be split into 75% training dataset and 25% test dataset. In this mini exercise, the dependent variable (y) is the resale price variable while the others are independent variables (X).

from sklearn.model_selection import train_test_split

X=scope_df.to_numpy()[:,:-1]y=scope_df.to_numpy()[:,-1] #resale_price is at the last column of the latest dataset

X_train, X_test, y_train, y_test = train_test_split(X,y,random_state=42,test_size=0.25)

Now, it's time to build the linear regression model!

from sklearn.linear_model import LinearRegression

line = LinearRegression()line.fit(X_train,y_train)

line.score(X_train, y_train) # 0.8027920069011848

The R-squared score of the model is 0.803, which is actually considered pretty good!

Let's further examine the results of the model - Mean Squared Error (MSE), Statistical Significance of Coefficients, both the Mean Absolute Error (MAE) and the Root Mean Squared Error (RMSE), and the Variance Inflation Factor (VIF).

def MSE(ys, y_hats): # Mean Squared Error function    n = len(ys)    differences = ys - y_hats    squared_diffs = differences ** 2    summed_squared_differences = sum(squared_diffs)    return (1/n) * summed_squared_differences

MSE(line.predict(X_train),y_train) # 4490363021.170545

The MSE shows that, on average, the error of prediction predicting the selling price of a HDB resale flat is about 67010.171 (+/-).

MSE can be used as an indicator to check how close the predicted selling price is to the actual selling price.

From the table, the p-value in the model is 0, which is less than 0.05, which shows that the independent variables have a statistically significant relationship with the resale price variable.

By answering the problem statement, the model helps to estimate the following variables that impact the selling price of a HDB resale flat:

• For every 1 metre further away from the CBD, the selling price drops by $18.12

• For every 1 metre further away from the nearest MRT station, the selling price drops by $49.04

• For every 1 square metre of flat size increases, the selling price rises by $4353.13

• For every 1 remaining year lease, the selling price rises by $4079.25

• For every rise in 1 floor, the selling price rises by $5065.95

from sklearn import metrics

metrics.mean_absolute_error(scope_df["resale_price"], predictions)# 51060.924629381385np.sqrt(metrics.mean_squared_error(scope_df["resale_price"], predictions))# 66948.4376270297

As compared to the mean of the resale price of the dataset, the MAE is relatively very small as it is about 1% of the mean of the resale price.

For RMSE, the model's prediction will miss out $66948.44 on average where it consists of about a 15% error rate, as compared to the mean of the resale price of the dataset. Therefore, the error rate of the model's prediction is relatively high.

From the table, we can see that VIF values are all below 4. Hence, all the independent variables that should not be correlated with each other are not correlated.

Conclusion

To conclude, we can say that the explanatory variables have a statistically significant relationship with the resale price of a HDB flat. With that, it helps us explain how each explanatory variable impacts the changes in the selling price of a HDB resale flat. Furthermore, as compared to the rest of the explanatory variables, the strength of the relationship between the floor size and the resale price of a HDB flat is the highest which has a positive moderate relationship. However, to improve the analysis, factors such as the towns (planning areas) of the HDB flats as well as the political boundaries where the HDB flats are located can be considered to answer the problem statement.

References

NDR 2018: Why are HDB leases 99 years long? PM Lee explains