Providing schools data and school holidays

You know that website that provides school holidays? Well I have to make that work.. Delivering the holiday dates to you is really easy, no drama there. We use a bunch of technologies to solve the problem of handling lots of requests per day.

Building a database of all of the schools in the world

This really requires a separate post but I will touch on it lightly here.. We use a number of data sources to build our database, all of the data sources have an automated tool which extracts them, compares the data to our stored data then updates our database. Our database is rather large, we use sources such as edubase for UK, US, RU but things are very different for our Chinese brothers.. We have to call up the district offices and ask them to send us a list of schools, worst case scenario is they give us a list over the phone. We also use some open data maps to get some rough data. Finally we use web mining to “detect new schools”, this gives our database an organic life and allows it to grow without any government input. By combining all of this data into one source we have a huge data set that we can begin mining from. We have been able to solve the problem of schools changing name/closing by using some clever insight analytics and by looking at search patterns for certain school names.. Here you can see a school that has either closed or renamed, blue showing the old name, red showing the new.. When you correlate the lat/long you can calculate that it has been renamed.. Simples!

Mining

Get out your hard hat and your birdy… We’re going web mining… We rolled our own web mining platform, this took a lot of my time up, I have no idea why I did it this way because using tools such as rapidminer would have probably been quicker but that’s life.. Our mining platform is a product of 2 years hard work to press, it’s job is to go out onto the internet, find a school, find relevant data and store it. So for example, we go out onto the web, find a school and store the holiday dates as a “data dump”, this data dump is just a copy of the page in whatever format it is stored in. We use naive bayesian filters to cluster the data into different categories and basically go from there.. We wont be releasing our mining platform publicly, it’s not worth releasing because it’s so application specific..

Language support

MSH now supports most languages and is now classed as a global service. We used mygengo to gather translations and then we need PHP strings to render them based on the users URL IE de.myschoolholidays.com gets German results. It couldn’t be more simple. The hardest part was crafting the initial language file and finding a good partner to work with.

Cloudflare

Another recent new addition is cloudflare. Cloudflare sits in front of our varnish stack and provides us with the ability to compress Javascript, cache requests etc. One thing Cloudflare sucks at is providing round robin so it actually reduces our ability to provide high availability. Cloudflare’s always online feature sucks pretty bad too. Despite these things it does reduce page load times and takes ~20% load away from our cluster, this is a good thing, hopefully they will resolve their current issues (I have 4 bugs filed with them) and we can move forward.

Removing images

Our other task to increase page load speeds is to remove all images. We did this by replacing images with icons using a typography set. This sped up page load more than you would have thought, it’s pretty impressive!

Google Web fonts

Is a blight.. Y u need 150KB? So my trick w/ Google web fonts was instead of using the usual item in the head I load it in with javascript at the bottom of the page, this stops the on ready event from being blocked and means the user can use the page early on…

Potential company changes

We are in discussions about sharding MSH out of it’s current parent company PT. We have a bunch of reasons for this, the biggest one is that the two companies don’t match up, the skill sets required to deliver the MSH service is completely different required to deliver day to day PT services.

Outsourcing

Most of our workers on MSH are not based in the UK, we use a number of remote worker platforms to help us find the right employees. We actually require quite a lot of manpower to do “initial country learns”, this is the process of learning about the structure of a country, how their holidays / councils / districts are layed out and also the language/social nuances.

So there you have it, a huge update and a lot of information about how we go about getting our data, it’s resource intensive and it’s a pretty big job as there are ~50k holiday sets each year each containing ~200k points of data.

Anyway if you get a sec please check out our performance and let us know what you think.

MAJOR UPDATE (v1.1): My School Holidays (inc lots of freebies)

Today we are pleased to announce the launch version 1.1 of our popular School Holiday Dates website, My School Holidays.

With over 1 million people every month using My School Holidays it is crucial your school holiday dates are up to date.

Mobile Phone App

We’ve just released a FREE Android school holiday countdown widget for My School Holidays users which can be found on the Android Market here. Or from your phone scan this QR code (or visit the Android Market on your phone and search for My School Holidays).

msh_widget_qr_code

Printable Calendar Cards

Don’t have an Android? But want a way to remember your school holiday dates on the go? How about printing them onto something that fit right in your pocket? Maybe a business card? You can now select your school on My School Holidays and print business card sized calendars with your school’s holidays on. You could even print a few pages, laminate them and give to parents? It’s super easy to do:
Simply find your school on My School Holidays and click the Printable Business Cards link. Print it out double sided and off you go.

Autocomplete on Search

We revamped the search box on the website so our autocomplete search will suggest school names to you as you type in the name of your school. Try it out now at My School Holidays

Going global

We have added more countries dates to My School Holidays. We now currently support: Germany, Canada,
The United Kingdom(includes Wales), Australia, The United States of America and New Zealand with more countries due over the next few months. Also useful if you’re planning to go on holiday in another country and want to find out when the local schools are on holiday.

iCal & GMail support

GMail, Outlook and Thunderbird users can now enjoy a simple iCal import of their school holiday calendar.

To do this visit your school on My School Holidays then click “More options” then select the format you wish your holiday dates.

Edit your own school holiday dates

We want your school to update your school holiday dates but we know how difficult it can be to encourage administrators to do this so we have given this functionality to the hardest working people in the education system, the teachers.

Teachers can now use a really simple interface to insert their own school holiday dates. They can also add casual/training/inset days. To insert your days simply find your school on My School Holidays and Click “Edit the dates”.

Behind the scenes

We have developed a new way of predicting and validating school holiday dates that you will see come to life in Version 1.2. We are currently using this method to quickly collect school holiday dates and publish them live on the website.

We will be using this technique along side other techniques to gather Holiday Data from trusted sources and provide you with the most up to date information on school holidays from around the world.

MAJOR UPDATE (v1.1): My School Holidays (inc lots of freebies)

Today we are pleased to announce the launch version 1.1 of our popular School Holiday Dates website, My School Holidays.

With over 1 million people every month using My School Holidays it is crucial your school holiday dates are up to date.

Mobile Phone App

We’ve just released a FREE Android school holiday countdown widget for My School Holidays users which can be found on the Android Market here. Or from your phone scan this QR code (or visit the Android Market on your phone and search for My School Holidays).

msh_widget_qr_code

Printable Calendar Cards

Don’t have an Android? But want a way to remember your school holiday dates on the go? How about printing them onto something that fit right in your pocket? Maybe a business card? You can now select your school on My School Holidays and print business card sized calendars with your school’s holidays on. You could even print a few pages, laminate them and give to parents? It’s super easy to do:
Simply find your school on My School Holidays and click the Printable Business Cards link. Print it out double sided and off you go.

Autocomplete on Search

We revamped the search box on the website so our autocomplete search will suggest school names to you as you type in the name of your school. Try it out now at My School Holidays

Going global

We have added more countries dates to My School Holidays. We now currently support: Germany, Canada,
The United Kingdom(includes Wales), Australia, The United States of America and New Zealand with more countries due over the next few months. Also useful if you’re planning to go on holiday in another country and want to find out when the local schools are on holiday.

iCal & GMail support

GMail, Outlook and Thunderbird users can now enjoy a simple iCal import of their school holiday calendar.

To do this visit your school on My School Holidays then click “More options” then select the format you wish your holiday dates.

Edit your own school holiday dates

We want your school to update your school holiday dates but we know how difficult it can be to encourage administrators to do this so we have given this functionality to the hardest working people in the education system, the teachers.

Teachers can now use a really simple interface to insert their own school holiday dates. They can also add casual/training/inset days. To insert your days simply find your school on My School Holidays and Click “Edit the dates”.

Behind the scenes

We have developed a new way of predicting and validating school holiday dates that you will see come to life in Version 1.2. We are currently using this method to quickly collect school holiday dates and publish them live on the website.

We will be using this technique along side other techniques to gather Holiday Data from trusted sources and provide you with the most up to date information on school holidays from around the world.

MAJOR UPDATE (v1.1): My School Holidays (inc lots of freebies)

Today we are pleased to announce the launch version 1.1 of our popular School Holiday Dates website, My School Holidays.

With over 1 million people every month using My School Holidays it is crucial your school holiday dates are up to date.

Mobile Phone App

We’ve just released a FREE Android school holiday countdown widget for My School Holidays users which can be found on the Android Market here. Or from your phone scan this QR code (or visit the Android Market on your phone and search for My School Holidays).

msh_widget_qr_code

Printable Calendar Cards

Don’t have an Android? But want a way to remember your school holiday dates on the go? How about printing them onto something that fit right in your pocket? Maybe a business card? You can now select your school on My School Holidays and print business card sized calendars with your school’s holidays on. You could even print a few pages, laminate them and give to parents? It’s super easy to do:
Simply find your school on My School Holidays and click the Printable Business Cards link. Print it out double sided and off you go.

Autocomplete on Search

We revamped the search box on the website so our autocomplete search will suggest school names to you as you type in the name of your school. Try it out now at My School Holidays

Going global

We have added more countries dates to My School Holidays. We now currently support: Germany, Canada,
The United Kingdom(includes Wales), Australia, The United States of America and New Zealand with more countries due over the next few months. Also useful if you’re planning to go on holiday in another country and want to find out when the local schools are on holiday.

iCal & GMail support

GMail, Outlook and Thunderbird users can now enjoy a simple iCal import of their school holiday calendar.

To do this visit your school on My School Holidays then click “More options” then select the format you wish your holiday dates.

Edit your own school holiday dates

We want your school to update your school holiday dates but we know how difficult it can be to encourage administrators to do this so we have given this functionality to the hardest working people in the education system, the teachers.

Teachers can now use a really simple interface to insert their own school holiday dates. They can also add casual/training/inset days. To insert your days simply find your school on My School Holidays and Click “Edit the dates”.

Behind the scenes

We have developed a new way of predicting and validating school holiday dates that you will see come to life in Version 1.2. We are currently using this method to quickly collect school holiday dates and publish them live on the website.

We will be using this technique along side other techniques to gather Holiday Data from trusted sources and provide you with the most up to date information on school holidays from around the world.