What the last 4 days have tought me.

Disclaimer: These are my own words and do not reflect the view of my employer.

Some teachers are really supportive.  Huge thanks to @pimmsSmith, @LouMeethongsai, @janwebb21, @idletim, @deputymitchell, @enomilie, @purplelady1979, @anne_neal, @whorwe, @sraff79, @missbrownsword and @primaryt for your support and help.  The fact that you all got in touch asking for help and tested logins is great, we need testers just as much as we need developers or system admins so my hat goes off to you..

Some teachers are real not quite as nice (actually just one). Even though PrimaryBlogger is a free service I got an email saying “how disappointed they were with the support.”  To this teacher I remind you that PrimaryBlogger is unblocked in every UK LA, allows embed, open source, doesn’t enforce HTTPS only, spam free, ad free, not publicly funded and each blog post is reviewed by a CRB checked human from Primary Technology.  To combat this I think a community approach is needed, a wikipedia model of moderators should help.

PrimaryBlogger is as a service is spread across multiple servers and sites and isn’t run by monkeys or unskilled professionals, it takes a lot of expertise to deliver a scaled WordPress deployment.  Now you probably don’t care about how big it is but the fact is if we were selling PrimaryBlogger at £200 a school per year we would be making £400,000 a year, pretty good money right but that would mean the community wouldn’t grow and that would suck.  PrimaryBlogger will remain as a free blog service, last year we made £100 in total in sales of domains and additional storage purchases.   We thrive from the success of school blogs and we are rewarded by “inbound links” which in turn allows us to promote other Primary Technology products.  I have to justify the cost of PrimaryBlogger to our sales director and I do it based on the amount of respect it gets us inside of the teaching community, something not easily bought and something we don’t take for granted.

WordPress is open source and free and there are a number of hosts you can move your blog to.  I suggest everyone periodically backs their blog up anyway.  It’s really easy to do.  Tools -> Export..

If you really want to help PrimaryBlogger, buy a domain for your blogs through us, it’s only £50 per year and it will be an improvement to your blog.  I would like to employ someone full time to work on it and to build a community so if we can sell 400(one fifth of the # of blogs on PrimaryBlogger) domains that is one persons full time wage covered.  It would be way better though if we can encourage a community to become vocal and over the next few months that will be my goal.

I also want to comment quickly about how all of our findings/problems were put into the public domain.  How many other companies do you know that show you their inner workings and explain why a problem came about, how it was resolved and who resolved it?

So to reflect on what I have learned:

  1. Always trim old/stale blogs from the database
  2. Tell teachers that they are out of order if they are out of order
  3. Don’t do any development work on a live server the day before we’re due to have all of our staff on holiday
  4. Give PrimaryBlogger’s (and our other services) a community site they can go to to share experiences and knowledge
  5. We as a community need to discuss the benefits of having a blog site that has blogs with “search engines disabled”

PrimaryBlogger is all about giving school pupils and teachers a voice, we want you to be heard.

PrimaryBlogger – why it was down..

So I’m writing this post on PrimaryPad because PrimaryBlogger is currently down.. It’s been a nightmare, it really has.. Thankfully it’s currently 1/2 term in schools so activity is down 20% or so across the site.

We have 3 layers of backup for school blogs, our database is large and the file system is very large. None of this is surprising for a site like PrimaryBlogger but what is surprising is that 2 levels of our backups failed..

The problem stemmed from me playing with a plugin that is built to replicate one site to another, I noticed that it was playing up so I disabled it and didn’t proceed any further. The next day I was informed some people were getting a white screen when trying to access their blog..

I looked through the plugins source code (some bits I had written) and realized there was potential it could of been dropping the wrong tables from the database, no biggy.. I will just restore the database.. I had done a full backup of the file system and database just before I started playing with the plugin.. Now bare in mind I’m backing up 660GB here which takes a little while so I set it going and went off to play some Bad company 2..

I came back an hour later and it had finished backing up so proceeding playing with the plugin.. Things broke so I figured I will just restore from backup only to realize that the backup I had taken was completely useless.. It was 10MB! “What the deuce” I pondered.. mysqldump didn’t output any errors during backup so what is going on?! I checked the replication and I couldn’t use that as a backup source as the replication servers had replicated the error.. So I was left with only one option..

We take daily brick level backups off site, these brick level backups take a .sql backup of our current mysql state but because it was off site it took a long time to transfer, during that time I quickly brought up a local VM to the .sql file and went ahead at trying to restore primaryblogger’s database locally just to make sure the servers were up to the job.

The first thing I noticed is that WordPress really doesn’t like the base domain being changed after install, so I had to backtrack and begin installation w/ the correct base set.

The next thing I noticed is that my mysql reads were giving an error: ERROR 1153 (08S01) at line 218227: Got a packet bigger than ‘max_allowed_packet’ bytes. I fixed this by increasing the max_allowed_packet in my.cnf

I also have a problem when I do “use blog;” (blog is the name of my database) I get a delay and “Reading table information….” notification which takes a few minutes to get past..

When restoring to a fresh database I noticed that mysqldump is not dumping table data and is just dumping the table structure. I’m not sure why but I need to investigate this further too at some point.. Note: I was using “-d database” name like a tool.. In the future I know not to make this mistake.. It is easy to make though…

Another problem I had was that my admin password keeps resetting itself. There is no logic to this, I used a mysql update statement to update the password then check it using a select statement yet after I try to login it changes back to an unknown md5 hash. I think it’s due to the SALT values in wp-config but I may be wrong.

While I was watching the database file dump back into the database I noticed just how much crap wordpress puts into each blog.. I mean most of each blogs contents is wordpress guffing the space.. I recon that 40% of my entire blog database contents is wordpress putting links back to itself and documentation into each blog site. Not cool…

Usually I test plugins off site and this was no exception but this specific plugin needed to iterate over an array of 1000’s of blogs and I didn’t have that many records locally. The specific bug with the plugin was quickly isolated but the fall out was 12 hours of unavailility of Primary School blogs across the whole of the UK.

The only way I could of really avoided this is if it I had local snap shots in the form of a VM but even then recording from a snap shot would of given me all sorts of database and file system inconsistancies and headache.

Anyway a few things caused the problem, it was literally down to one row (out of millions) in our database and that’s why it took so long to diagnose and resolve.. I also had to completely restore the entire themes folder as for some reason this was empty..

So my apologies, I had extremely bad luck but worked my butt off over the weekend and early today to restore stability. This error could of happened to anyone and it’s very lucky we have a load of backups in place to restore all of the sites. At no point were any sites data or content at risk, credit is due to our remote backup service that saved the day.

Why we should be teaching blogging in school

Teaching kids how to use the technical aspects of blogs but what is more important is teaching them to be concious creators. The below video shows what happens if you lose focus and become consumed by the connectivity aspect of creativity. The video is 13 mins long but it’s an easy watch and I would recommend it if you are considering teaching blogging or any form of web based creativity..

Of course it’s slightly ironic that I’m blogging about blogging but you can ignore that based on the fact I have a financial interest in PrimaryBlogger

WordPress and Varnish comment IPs

If your WordPress install is behind Varnish you may have issues with Comments showing your Server IP. This will also cause problems with Spam as Akismet will not be able to check the source IP.

In the vcl_recv part of your varnish config add this

set req.http.X-Forwarded-For = client.ip;

Then edit your WordPress wp-includes/comment.php and replace REMOTE_ADDR with HTTP_X_FORWARDED_FOR
Then edit your wp-content/plugins/akismet/akismet.php and replace REMOTE_ADDR with HTTP_X_FORWARDED_FOR

Restart varnish

/etc/init.d/varnish restart

Finished, test by commenting.

Note: Users experiencing spam on PrimaryBlogger should now experience less spam and more accurate comment filtering.

If you are running Varnish 2.1.2 RPCXML will not work, you need to upgrade to 2.1.3..  You can find out your version by typing


varnishd -V

Using Classdroid with any WordPress site


Classdroid allows you to upload to any wordpress site.
PrimaryBlogger supports this by default.

Either a) Download & install this plugin

or b) Goto Settings, Writing then click Enable XML-RPC (Bottom option of the Writing settings page)