The New Zealand labour movement used to have its own newspaper. A group of us thought that now might be a good time for it to be digitally reborn: The Standard v2.0.
Another upgrade complete. As per usual let me know if any problems (that I didn’t find in testing) show up. Please let me know here. If I don’t know about them then I can’t fix them.
The next upgrade will happen on the weekend. This is a new module designed to make bans more effective and easier for the moderators to operate without expending my time. That is good. At present it looks like the steady-state for the site loading looks like it will settle at about 3 weeks prior to election. Looks like I’ll need the time for the steady enhancements to reduce site loadings before the next election.
BTW: The wingnut emulation project looks pretty good at present. Hopefully I’ll get that tested before the xmas.
I’ve just put another round of optimizations into the system. In this case caching the p-code generated by php into shared memory This should usually result in faster page display where content is changing rapidly. Posts with fast growing comments should get the major benefit from this. This should allow the site to run on its existing hardware for quite a while without the trauma of hardware upgrades.
This should be the last of the back-end optimization updates. The next round of updates will concentrate more on usability.
Some comments on Sunday about this site performing well induced me to have a look at the traffic over the last week. I was extremely surprised at how much load the system can now take without stalling. From the Wordpress blog stats, It turns out that the traffic for the week was 34.6% higher in page views compared to the week before. It was a staggering 61.9% higher than it was two weeks before. It makes me more confident about running the site at a technical level towards the 2011 (or earlier) election.
I’d love to take credit for this (and I will in part). However the reason people read the site is largely because of the content, both from our posters and our knowledgeable and informed commentators*. This is a completely voluntary site run on a very loose cooperative arrangement - it just shows what kind of talent is there to be tapped from the grass-roots. A special mention for Julie Fairey from The Hand Mirror for her election evening coverage who kept everyone coming back for more. Despite being offline for comments for 19 hours, it is still in our top ten days for traffic.
However that is just the visible portion of the site. We’re reliant on a number of technical supports.
Webfarm also deserves a strong mention. Their systems have been utterly reliable since I moved the site there in Feburary. Currently the site runs on virtual private server with 512MB of RAM running Fedora Linux with unlimited local traffic costing $159+GST per month.
However, my biggest kudo’s has to go to the open source people at Wordpress, Fedora project, K2 theme and the wordpress plugin authors without whom this system would have failed dramatically under the load. The site now runs without any code kludges apart from some css styles for the display. It is quite a different site from what was running last year and failing under load.
* Well almost all commentators - I specifically exclude the annoying, largely incoherent, and definitely anti-social trolls. To a man, they seem to mainly be act supporters** hyped on their badly self-assessed intelligence, in love with their own egos, and who take the credo of individualism to the point of being incapable of meaningful social interaction with others. It is the one area that I pity John Key in - he is going to have to deal with adult versions of these clowns. He also doesn’t have the “send to spam” button. However he has done well with an early ban.
There will be some scheduled maintenance at the hosting company tonight between 10pm and 6am. During that period the site may be offline for some short periods.
So if you cannot get the site for a period, then just try again a little time later.
Yet another record breaking day at The Standard yesterday. We keep breaking our own records for numbers of page views and numbers of visitors. The site does seem to be be running better during the peaks since the database tweaking.
I’ve brought the site up to date. All the various hacks that I put in this years have now been removed as the software and plugins have now been upgraded.
The main area of difference is in the re-edit, which is no longer in-line. I’m still tweaking that in places for look and feel. I’ve very interested if this gets rid of the problems that people have been having with the older version recently. There was one bug in it that I sent the bug and fix to the author.
Let me know of issues you’d like addressed in the current look and feel, and any bugs.Currently the only known bug is the missing caption graphic (working on it).
I’ll be doing some more of the wish-list stuff later, but it is unlikely to get into place before the election. I have tweaked the database parameters to give better performance when the system goes under HEAVY load, as happened twice last week.
There will be a server upgrade on the 16th of October about midday. I’ll remind people closer to the date.
After a lot of work both by a few e-mails from people here and by the good services of some people on nz.comp (google seems to be a little behind on the messages) I finally found the link to the malware site that was attached itself to the site footer.
The material that it was trying to introduce to people reading the site may include various forms of backdoors. It would be adviseable to run a good virus scan on your system if you have read the site in the last couple of weeks. Corporate systems shouldn’t have had an issue because the site it was linking to has been a well known chinese malware site for a long time.
The anti-virus/malware scans missed it at the server because it was a new variant of an old problem (the same one I had in march), targeted specifically at wordpress sites using what is evidently is still a open vunerability. My own checking of the site missed it because it had managed to leave all of the file attributes of the file (size, times, etc) exactly the same as the origionals. My attempts to see what people were reporting had failed because it only emitted the malware link out periodically. A dump of the web page at the client side by Stephen Worthington allowed me to see exactly what it was doing.
The vunerability it was exploiting was meant to have been fixed in wordpress 2.5, however they seem to have found another vunerability. The downside of having open source software is that it is possible to read the code looking for holes. I’ve done some things to reduce possible problems, but I now have MD5 hash check of the files running periodically which will fix the problem if it happens again. I’ve also reported the details to wordpress and a couple of other sites.
But there are some very creative people out there writing this stuff, and evidently this site is popular with them.
I’m on ‘holiday’ for the next few weeks to contribute some campaign work.
However it will also give me some time to do the next set of upgrades to the site sometime during that period. I will be putting in the acculmulated upgrades for existing packages and testing them.
Here is your opportunity to say what kinds of features you’d like added. I won’t promise to put them in, but I’ll at least have a look at them. Remember they have to stay consistent with the look and feel of the existing site. Have a look around the Wordpress plugin area.
I was just checking the links to the standard and I see that Whale is being his usual self. It looks like he has finally managed to read a DNS. He has discovered that I run a server at home, and that I’m a labour party supporter. Now I’m sure I’ve mentioned this a number of times both here and on KiwiBlog and on some of the other blogs.
Whale has found that I own labour.co.nz. That was registered by myself a long time ago (early 90’s from memory) to give uucp e-mail addresses to labour party members in my electorate. This was before any political party apart from the greens had figured out that there was a net. It was done to show my skeptical local politician of the potential of the net for NZ and its exports, and has probably helped a lot of IT companies as a consequence.
I changed the owner name of the domain to the NZ Labour Party (NZLP) when they did finally set up their own domains and redirected the web to labour’s site. This gives the NZLP rights over the name under domainz and then InternetNZ rules. I still pay for the domain since it is my fellow activists who use it, so the bills come to me.
Now in my book, that means at worst, that I’m guilty of giving a donation to the NZLP - a web redirection. But Whale seems to think this is significant - but as we’ve seen before Whale doesn’t understand the law very well. It’d be interesting to find out if he understands the cost of a 300 series redirection response to a HTTP GET command.
Technical skills are in short supply around the world. Almost every tech I know provides them gratis to someone else - if only family. Most help voluntary groups from the scouts to the PTA’s. I help the NZLP because I want to make sure we have literate politicians. I also help companies I’ve worked with, friends, family, and sometimes their voluntary organizations. Of course in the bloated ego of Whale this seems to mean that there is a vast conspiracy. He should really go and help someone (or someone should help him).
Well that happened fast. This is the 1001st public post on The Standard. That is assuming I get it in before the next post appears. So I thought we should have a post to celebrate the event. It has been just over 9 months in the gestation.
The first post was, appropriately enough, “Get your story straight John” on August 15th last year. It is a question that our posters are still asking. Like them or despise them, the blog posts have been interesting, thought provoking, and steadily improving.
It is a pleasure to help provide the space for our posters and our varied cohort of enthusiastic and generally well informed commentators.
There will be a hardware upgrade on monday during working hours (still trying to get the time). The site will be offline for up to an hour while the system gets transferred.
The upgrade should eliminate the capacity issues that have been showing every few days. I’ve limited the problem by reducing the number of simultaneous connections. This caused the system to stop swapping memory out while sending pages. However it will have reduced speed periodically during high demand.
This server side issue has been showing up since I started upgrades on the site. While the traffic per IP has reduced as a result of the upgrades, the total traffic has been increasing. This mainly seems to be from the multitudinous web-crawlers picking up every page in the site again.
The latest tweak on the system are Gravatar/Identicons. Read about it on this site in the Gravatar menu item. Apart from the graphics, it does have a serious purpose. That is to provide some level of personal trust/authentication for those who wish to take advantage of it.
This post is to provide a thread to discuss/help on how to use it. It is a support thread. Please leave the sniping outside.
Yesterday’s site issues were due to a combination of inadequate Expires (causing a lot more files being transmitted) and the web-crawlers grabbing the updated site.
It blew out the memory. I’ve limited the simultaneous sessions to prevent that until I get more RAM. Bandwidth usage for this month is indicating this would be a wise precaution to speed page delivery, even without the site updates.
I’ve done a few tweaks to the front page of the site to speed finding the current state of play and balance out the blogroll. Because of the volume of posts and comments, I’ve increased the size of the latest of both. This means you will be able to see who is talking across posts more easily.
The access to the older posts are now inside ‘Archives’ at the top menu.
There are a number of features that I’ll be turning on over the next week or so as things settle down. But none of those should structurally change the site. The back-end changes are quite effective.
Let me know of the gripes and suggestions either as comments in this post or e-mail (but avoid the political sniping). I will do what is feasible, while keeping a close eye on that bandwidth and memory. Hopefully this is the last of the upgrade issues apart from the disappearing IE comments box that I still have to fix.
I upgraded the site tonight. There have been a lot of security fixes since we pushed a test site together in August 2007. It was only meant to be a trial (as I keep saying plaintively to the posters), and it sort of got away on us.
This version is meant to be much more secure, and considerably faster. But I’m confident that the post and comments will ensure that doesn’t show up on your side. Verbiage seems to fill the space available - especially in an election year.
The biggest visible differences are in the Dashboard for the posters and anyone who logs in. That is because I wanted the site to look much the same, at least for now. Getting the similarity has deprived me of some needed sleep because I needed to move hacks to css.
I know about the missing selection of posts in the top right, but let me know in here if anyone sees any problems. It seems to work in all of my browsers. But the first set of comments in here will be a final test for common browsers.
Cheers
Lynn.
Update: Internet Explorer version 6 has problems with the layout on the reCaptcha. This will not be fixed until tuesday. If you are using Internet Explorer before version 7, you should upgrade or move to a better browser anyway - help your developers.
I’d suggest Firefox or Safari. Both confirm to the standards a lot better than any Internet Explorer.
A reader pointed out a set of hidden spam links that had attached themselves to the footer of the site.
I’d noticed them previously and thought that I’d corrected them (see comment). But they turned up again. Turns out it was from an interesting flaw in the website, and was being inserted from outside. I’ve hardened up the problem area(s) and I’ll monitor for a repitition.
It is likely that I’ll need to take the site down for a few hours in the early morning on Sunday to do some further maintenance. So don’t get too surprised to get a unexpected screen around that time.
There was a reboot on the server last night for some reason about 20 hours ago, and the database process failed to start. The cron job that was meant to restart the process didn’t work.
It would happen on a day when I was rushing around with a dead cellphone and no net access.
I’ll have a look at what happened and see how we prevent this from happening again.
Cyber-Santa came a little late to The Standard this year but we’re certainly not complaining - evidently we’ve got a New Year’s present instead.
He and the techno-elves have moved us to a brand spanking new server cluster that should give us plenty of breathing room and make those pesky traffic congestion problems we were having a thing of the past.
Seriously though, it wasn’t really Santa. Just like James Bond apparently we have our very own M and it’s him we have to thank instead.
We’re still secretly hoping for the pimped out Austin Aston Martin but this should tide us over in the meantime. Thanks M!
Well I didn’t get my pony but I hope everyone’s Christmas was as enjoyable as mine.
I’m just finishing some peanut butter on toast, by my reckoning the first non turkey- or ham-based meal I’ve had in five days. I’m hoping that it won’t trigger some sort of auto immune rejection reaction.
Things are likely to be pretty quiet here on The Standard for the next week or so, though I suspect people will be posting from time to time. I thought now might be a good time to thank you all for your support over the last few months.
Our first post was on August the 15th 2007. It’s been exciting for all of us to watch the rather humbling growth of this blog from tens to hundreds to thousands of daily visitors - of whom you’re probably one. It’s all of you that help to make this site a success - so thanks.
A particular thank you to all those who participate in the comments section - you’ve helped to build a community, and one that we’re going to be doing our best to help you grow even further in the new year. We’ve got new hardware sorted to cope with the site’s steadily increasing traffic and we’re kicking around a few other ideas we’ll hope you’ll like - more on this as it develops.
Merry Christmas, Happy New Year, and thanks again from all of us here at The Standard.
So the Herald links to us from the front page of their website this afternoon, and the resulting traffic knocks out our server. Bugger. We’re back online now, but I guess the moral of the story is we’re going to have to start paying for it like the big boys do. Did I say bugger already?
