That slows me down but it’s not currently the main issue with paste2.org.

The issue I’m having right now is a design nightmare. I’m not a designer, a long way off it, I like to work on complex backend problems and design work doesn’t suit me in any way. I did the current design for paste2.org originally, back in 2006. The problem is it’s not really holding it’s own now, and it needs replacing.

I did a new design for the new code but quite a lot of people who saw it complained that the design was too bright when they’re working with consoles mainly – the white/grey design that everybody but paste2 uses – it seems isn’t up to scratch for people who spend most of their time in a white on black console.

I didn’t know this at the time I first put paste2 together, don’t think for a second it was a genius design decision – it just seems like it may have been a happy accident.

So anyway, I’m trying to put together a design that works and is dark but isn’t a catastrophe like the current one is, ideally I’d like a dark and light design so people can have a choice, like the new highlighter offers a choice of colours too.

The good news is that actually the code is actually fairly close to done, then I’ll need to write a migration script. Once I have this new code done and running on the site I’ll be happier about adding new features, like the collaborative editing I’ve been talking about for so long now!

One of the obvious changes between the old code and new will be the URLs, which will change from a sequential number system to an 8 (+ if ever needed) case-sensitive alphanumeric system that is randomly generated. This means a bigger key space (shorter URLs because there will be 6,000,000,000 possible pastes instead of 99,999,999 possible pastes in 8 chars) and it will stop people from trawling for pastes so much.

The other change with URLs will be the removal of the /p/ from paste URLs. All pastes will be accessed by http://paste2.org/<tag> rather than http://paste2.org/p/<number>. Old pastes will redirect (301) to the new URLs, so there will be no need for users to change anything.

The next update here I’m planning related to paste2 will be a public beta test.

I return a value so the class which calls this can replace the value back in the form if there is an error, and it actually replaces it in the data that is used (sent to the DB or whatever), I also use a custom validation exception class, these are all things that will need to be considered if anybody who sees it plans on using. Caveat emptor and whatnot.

What it does is sends the value to the PHP filter functions for validating as looking remotely like an email address, which is handy but it will validate local addresses and for domains that don’t exist.

What my code does is check if there is a remote address part and then if it is an IP just allows it, if it’s a domain it checks if the domain has MX records using checkdnsrr() – if that’s all good it allows it.

What could be improved? It doesn’t work so well at preventing local IP addresses going through. It also won’t tell you if the user is legit at a given domain or if the user has any association at all, so address validation will be a must in some circumstances, but it will prevent the standard dfgdf@dfdgdf.com type stuff. At least you’ll know if the domain part looks even close to reasonable, which is one of the ‘failings’ (read: missing features?) of FILTER_VALIDATE_EMAIL if you want internet-only addresses entering.

	public function email_address($value) {
		$result = filter_var($value, FILTER_VALIDATE_EMAIL);
		if($result == false) {
			throw new validateException("Invalid email address");
		}
		$domain = explode("@", $value, 2);
		if(filter_var($domain[1], FILTER_VALIDATE_IP) !== false) {
			return $value;
		}
		if(checkdnsrr($domain[1]) == false) {
			throw new validateException("Email domain doesn't seem to exist or is incapable of recieving mail");
		}
		return $value;
	}

I was reading the Linux kernel code for software RAID1 because I was totally bored, and something caught my eye – the ability to prefer to write to certain disks in the array (mdadm –write-mostly). I  decided that I was going to find some use for it that was a bit outside the box.

After a little tweaking and configuring I came up with this Evil Plan:

mdadm --create --verbose /dev/md5 --level=1 --raid-devices=3 /dev/ram0 --write-mostly /dev/sda3 /dev/sdd3

This config gives you an array as in the diagram below:

The smart people reading this will probably have figured out by now that what you now have is a RAID1 array that tends to read from RAM and write to disk.

What they may not have figured is what we actually have here is a RAM disk that is mostly* safe on disk in the event of crashes and reboots. The RAM is effectively just a copy of what’s on the disk – but an automatic copy, there’s no manual syncing, it’s handled by the RAID code in the kernel – if something is written it’s marked as dirty and the code will go to the source disk (the one the data was written to until the changes have been synced across).

At this point you’re probably wondering what the downsides are. Well firstly on a reboot you lose a disk from the array. I haven’t actually rebooted the server yet so I’m not completely sure how this is going to respond to a reboot – if it’ll say oh hey here’s a blank disk so we can add it to the array or if I’m going to need a boot script to add it back into the array – either way it will automatically sync the data when that’s done – and really on a server you really don’t want to be rebooting too much anyway.

*The other issues is it may not be totally crash-proof all of the time. I can imagine a scenario where you’re write blocked on the two hard disks – and it’ll have to write to the RAM – the data from which will be copied back to the hard disks right when it can be. For that time it could be a little risky if you have a crash at that point. One of the answers is that if you’re tending to write a lot and it’s writing to RAM – add more disks to the array so the code doesn’t block writes to the array.

So what are the performance numbers?

Well, I’ve not done any write benchmarking, but I’d expect it to be generally standard hard disk write speed – until you start doing concurrent writes – the array will probably actually get faster in this example when you do 3 or more concurrent writes – which is actually very strange for a RAID array, usually you start getting performance loss with more concurrency – but with this array at some stage the code will decide the disks are IO blocked and write to the RAM drive. The speeds we’re talking about start to get extreme, but it’ll be 1000MB/sec plus judging by the read benchmarks:

Reading from this array will prefer to hit the RAM disk – this is where things get interesting. I’m starting to get the feeling that the speed may be IO-bound by the filesystem code and other parts of the kernel, but, a bog-standard hdparm benchmark:

hdparm -t /dev/md5

/dev/md5:
Timing buffered disk reads:  3986 MB in  3.00 seconds = 1328.17 MB/sec

Wow. If you think that’s useful, you should see the seeker results:

Benchmarking /dev/md5 [4102MB], wait 30 seconds…………………………
Results: 1149116 seeks/second, 0.00 ms random access time

My 4 (SATA HDD) disk RAID1 array for comparison:

Benchmarking /dev/md1 [200000MB], wait 30 seconds………………………..
Results: 99 seeks/second, 10.09 ms random access time

It’s not hard to see where the performance can come in useful. If you’re getting bogged down by a lot of random reads, want reasonably safe storage and have RAM to spare – this is going to take some beating. SSDs? Pah!

Why use this rather than relying on filesystem cache? Aside from the fact that it’s targeted so the data you want is always there as opposed to the 1% hit-rate-if-you’re-lucky that the filesystem cache will give you? Other reasons that I could list…

Like I said sometimes my crazy ideas actually work. As far as I can tell nobody has done this before.. Or at least I can’t find any evidence of it in Google. I’d love to hear if somebody has seen it done before though, be nice to compare notes.

Update:

So on reboot I now know for sure – it does totally kick the ram disk out the array – but this is an easy fix, just a case of having say an init script that runs a command like:

mdadm --add /dev/md5 /dev/ram0

It’ll then rebuild the array automatically.

After seeing this happening in a IRC channel not long back, being done inside an editor that is really designed for creating word processed style documents, but with code, I’m absolutely sure it’s the way forward.

The problem of course is that this stuff isn’t easy. I already have a plan to do it using some parts of google mobwrite, and building the server side in PHP because it’ll be faster – and I’ll trust myself more if a bug comes up to know the language ins and outs, which while I can write Python I don’t really trust it or myself. I could easily just do it easily using mobwrite in it’s entirety but the performance wouldn’t be great, and like I said – if something came up it’d probably push my Python knowledge.

Problem is getting it done. I was thinking about (and started) writing the whole thing into the new code, but I’ve changed my mind – I’m going to push the new code out on the new application server, then start working on that.

There’s some other good pastebins around, since paste2 came on the scene (at the time pastebin.com was down basically all the time) some other new ones have popped up and are helping improve the competition in the ‘market’, which can’t be a bad thing. Even pastebin.com has had a redesign now (after being sold).

The bootloader is effectively broken, it’s actually doing a netboot right now which is ugly and it doesn’t solve the problem I was trying to sort out in the first place.*

I’m planning some downtime in the next few days to move paste2.org to a different server so I can do an OS reload on that one and set it up how I want it. I may even end up putting Xen or ESX Server or something on it for a bit more flexibility and run servers out of that which will give me a bit more room to play around (probably some mirroring between multiple VMs, one of which running solaris so I can use dtrace on it).

Just to update on the new paste2 code, I’m currently polishing off the application server I’ve been working on for what seems like decades now. Whilst doing this I’m also trying to put together a realtime(-ish) collaberative editor – so multiple people can edit the same file at the same time, and see updates that other people have made in close-to realtime. By realtime I mean a few seconds later of course – the tech involved of course has latency and also, worse can’t be continously updated for server resources reasons (it’s not feasible to be continiously connected to a http server, though it is possible), you have to work smart and get as close to realtime as resources and the technogy allows. This isn’t new by any means, the likes of etherpad have done it before – but I want to target it at the people who use paste2 rather than people say putting together a rich text doc for example.

* If anybody cares – effectively Grub has made a horific mess of the server, no matter what I do I can’t get rid of it, even using dd on the MBR region of the disk isn’t getting the job done. I need a custom kernel for latencytop and so I don’t have to run VMWare workstation anymore, wanted to get VMWare server running in it’s place which I had issues with before but now the will to get it fixed.

Update (6 March 2010):

The issues have been resolved after a server reload yesterday – the original problem was easily cured once I got GRUB out the way. Now the box is very happily running on LILO with no issues that I know about.

The good news is that nothing is lost – just having an issue trying to get into the server, my KVM access isn’t working correctly right now. I could reboot the entire thing with root on one of the drives and it may work except for the fact / is mounted on RAID rather than /dev/sda1 so it’ll boot the kernel then give up.

Shouldn’t be long now, I have a ticket with the DC which I’m waiting on a response to.

Some things just aren’t worth the effort. This is one of those things.

It’s not the bandwidth, the CPU time or even frankly the illegality of what the content links to – it’s the fact that these idiots can’t write code that plays nice (well, they are Java developers, so what can you expect?), and on top of that – instead of using a real user agent which would give developers, server admins and the like (i.e. me, in this case) information about who it is doing the damage so it can be discussed and hopefully come to some kind of fix – they RANDOMISE their UA string.

If I may quote (and mangle) one of my favourite TV shows – “I let them play in my sandbox; and they went and shit in it”.

Please don’t ask me to restore these pastes, they’re gone forever. The risk was paste2 would become impossible for me to keep paying for out my own pocket.

It’s now been brought back from the dead, like Frankenstein.

I’ve been trying to fix a problem I thought of in my head, sort of a “how the hell am I going to solve this problem?” type deal.

The problem being essentially, because you maintain a connection with the DB server (because the application is persistent), if it gets restarted the application flaps – because the server kills the connections. It’s not really feasible to ask people to try/catch the specific exception, and because of one of the features of the server that isn’t available anywhere else (it’s essentially a minimal preforking HTTPD – though not intended to be used as one, I’m actually considering killing the process if there’s no forwarded-for header – to make it a “on your head be it” type deal), it’s very hard to catch this stuff further up the stack. You can’t just pretend it’s not happening and kill the child when you’re working with a forked server like this – if you have 10 children you’re going to output errors for 10 clients which is not really what you want to be doing.

I experimented with using semaphores to resolve the issue, which didn’t work too well and was kind of ugly – and it made the code that much more complex.

The solution I came up with was to catch the problem in the PDO class (currently only for MySQL), and create a new kind of exception that gets caught in the routing code (the chunk of code that decides where requests get sent). This then redirects the client to the same page (302/Location headers) and kills the child, the system then does what it usually does when children die – fires up a new child, which creates a new instance of the app class – which will reconnect to MySQL.

It also shows you the problem with being very hands-off with what people who use systems you write, if you’re not paying attention you can create problems that you don’t anticipate then the day you let people play with it somebody restarts a server in production and the world ends. So you basically have to force people to work like you want them to.

Paste2.org’s code is written to be fast, the problem with doing that is if I leave it alone for a day it can take large amounts of traffic that isn’t legitimate without really notifying me because the load doesn’t go high enough for the server to start alerting me that things are going wrong.

Take last night for example, I just happened to look at munin and I saw the first spike of this (the part with the big red updates block in the graph):

This event which peaked at almost 400 queries/second (and if I tell you paste2.org hardly does any SQL queries, you’ll get why I was pretty pissed off when I noticed this), was pretty massive traffic comming from a lot of different IPs – which a lot of people would assume is a DDoS attack, I’m pretty sure is somebody trying to mirror the site.

If I may slide slightly off-topic for a second it’s a bit of a win for the much-hated query cache – look at the numbers of cache hits – when your MySQL server is set up right and your code is asking the right questions.

You’ll notice that the number of queries drops off at around midnight, this is the point when I noticed something is amiss and did something about it.

I have a script that scours the access log and adds the IPs it pulls out to an IPTables Chain, which, naturally, stops all inbound connections.

The problem is until about 5 minutes ago it was all manually ran, because in the past people have got the idea after a few rounds of that.

Not this time, note what happens after midnight – it slowly picks up again until it’s just as bad as it was. Now the whole thing for the last few minutes has been completely automated.

In case you’re wondering, whilst it’s nice having the site load tested, there’s two main issues: firstly nobody has ever asked if they can have the paste files, or told me why they want them all, and secondly – as you’ll see from the first part with all the updates, they were triggering the code which determines if they’re a robot or not and decides if they should update the last viewed date – which in turn determines when old posts should be deleted. That’s probably the worst part of people doing stuff like this – that it screws up the reliability of a system which is essentially a spam removal process. Legit posts that people need will be visited and kept, spam won’t be visited and thus get deleted after a time – all these posts are now marked as updated last night and the 95% that will be actually spam, will survive in the site for another 60 days.

I wonder how long it will be until these clowns get the message. Anyways, I can go back to my day job now the script is chugging away on its own.

I feel for sites like TPB sometimes. That is I mean, they suck legal defence (“we’re just like Google” indeed), but they must get so many takedown requests for random crap every day than most people get spam.

I get one every few weeks. People like to post NFOs and lists of files on paste2, and somebody comes along and complains about it. The requests to take these posts down annoys me.

Instead of just asking politely at first they start off with the legal threats. “We demand you take blah blah down else we’ll sue you and the next 14 generations of your offspring, fuck you very much”. I’ve had one sat in my inbox for a while from Fox, I keep re-reading it to make sure I wasn’t imagining it’s content.

I’m not a big fan of taking pastes off the site generally, if you think something on the site is infringing somebody’s copyright you’re probably mistaken – it won’t let you posts content as long as books and people can’t upload movies and music, so what’s the deal?

Well it turns out Fox didn’t take too kindly to people posting Wolverine release NFO files. Now I hate TPB’s we’re just like Google defence, because frankly it’s pretty retarded – I don’t disagree in many way’s what they’re trying to do but I think we should all stop for a minute and say it like it is – their defence was pure idiocy. Back to my point though, the majority of the pastes these guys are getting upset about are so many steps removed from copyrighted content it’s not even funny.

Firstly, the stuff Fox got upset about was a bunch of NFO pastes, which if you don’t know what they are, essentially when a group releases something they ripped or whatever, they add a file in which basically says what that release is, which group did it, usually some ASCII art and usually some info about the group (yes I know, I’m trying to do a lazy idiot’s guide to here, ask wikipedia). Anyway, look at the link above, I defy anybody to tell me what IP that paste infringes on.

The other type of takedown I’ve been getting lately are ones where people have been pasting lists of links to download files, I’ve actually been deleting these as requests come in, but I’m getting a bit bored of it really. Paste2 isn’t the problem, it’s people hosting the files, talk to them.

I guess what I’m saying is two things, firstly, I got no issue with NFOs being posted, indeed they arguably have their own artistic merit, at worst are free advertising and it’s not my fault if the studios can’t find a real movie distribution platform that makes people turn to piracy.

Secondly, if I get many more “we demand you” emails for random crap that doesn’t infringe on anything, I’m going to start demanding I ignore such emails and demanding they go to /dev/null. Ask me nicely, and I might play nice. Keep the BS up and I’ll happily call your bluff and see you in court.

Last thing is, the email that I’ve been sat on for a while, that annoys me so much I’ve totally ignored except for looking at every now and then when I need cheering up.