I return a value so the class which calls this can replace the value back in the form if there is an error, and it actually replaces it in the data that is used (sent to the DB or whatever), I also use a custom validation exception class, these are all things that will need to be considered if anybody who sees it plans on using. Caveat emptor and whatnot.

What it does is sends the value to the PHP filter functions for validating as looking remotely like an email address, which is handy but it will validate local addresses and for domains that don’t exist.

What my code does is check if there is a remote address part and then if it is an IP just allows it, if it’s a domain it checks if the domain has MX records using checkdnsrr() – if that’s all good it allows it.

What could be improved? It doesn’t work so well at preventing local IP addresses going through. It also won’t tell you if the user is legit at a given domain or if the user has any association at all, so address validation will be a must in some circumstances, but it will prevent the standard dfgdf@dfdgdf.com type stuff. At least you’ll know if the domain part looks even close to reasonable, which is one of the ‘failings’ (read: missing features?) of FILTER_VALIDATE_EMAIL if you want internet-only addresses entering.

	public function email_address($value) {
		$result = filter_var($value, FILTER_VALIDATE_EMAIL);
		if($result == false) {
			throw new validateException("Invalid email address");
		}
		$domain = explode("@", $value, 2);
		if(filter_var($domain[1], FILTER_VALIDATE_IP) !== false) {
			return $value;
		}
		if(checkdnsrr($domain[1]) == false) {
			throw new validateException("Email domain doesn't seem to exist or is incapable of recieving mail");
		}
		return $value;
	}

After seeing this happening in a IRC channel not long back, being done inside an editor that is really designed for creating word processed style documents, but with code, I’m absolutely sure it’s the way forward.

The problem of course is that this stuff isn’t easy. I already have a plan to do it using some parts of google mobwrite, and building the server side in PHP because it’ll be faster – and I’ll trust myself more if a bug comes up to know the language ins and outs, which while I can write Python I don’t really trust it or myself. I could easily just do it easily using mobwrite in it’s entirety but the performance wouldn’t be great, and like I said – if something came up it’d probably push my Python knowledge.

Problem is getting it done. I was thinking about (and started) writing the whole thing into the new code, but I’ve changed my mind – I’m going to push the new code out on the new application server, then start working on that.

There’s some other good pastebins around, since paste2 came on the scene (at the time pastebin.com was down basically all the time) some other new ones have popped up and are helping improve the competition in the ‘market’, which can’t be a bad thing. Even pastebin.com has had a redesign now (after being sold).

It’s now been brought back from the dead, like Frankenstein.

I’ve been trying to fix a problem I thought of in my head, sort of a “how the hell am I going to solve this problem?” type deal.

The problem being essentially, because you maintain a connection with the DB server (because the application is persistent), if it gets restarted the application flaps – because the server kills the connections. It’s not really feasible to ask people to try/catch the specific exception, and because of one of the features of the server that isn’t available anywhere else (it’s essentially a minimal preforking HTTPD – though not intended to be used as one, I’m actually considering killing the process if there’s no forwarded-for header – to make it a “on your head be it” type deal), it’s very hard to catch this stuff further up the stack. You can’t just pretend it’s not happening and kill the child when you’re working with a forked server like this – if you have 10 children you’re going to output errors for 10 clients which is not really what you want to be doing.

I experimented with using semaphores to resolve the issue, which didn’t work too well and was kind of ugly – and it made the code that much more complex.

The solution I came up with was to catch the problem in the PDO class (currently only for MySQL), and create a new kind of exception that gets caught in the routing code (the chunk of code that decides where requests get sent). This then redirects the client to the same page (302/Location headers) and kills the child, the system then does what it usually does when children die – fires up a new child, which creates a new instance of the app class – which will reconnect to MySQL.

It also shows you the problem with being very hands-off with what people who use systems you write, if you’re not paying attention you can create problems that you don’t anticipate then the day you let people play with it somebody restarts a server in production and the world ends. So you basically have to force people to work like you want them to.

I’ve also either added or are working on adding a few new features.

Screenshot of the new template, I know, the logo is horrible!

The first one that’s pretty much done is diffing between pastes. Essentially you can diff between pastes and in the page you’d be able to quickly, on the click of one button, be able to show the diff of the two pastes without messing around entering numbers.

Secondly the site has a new theme that I’m pretty happy with now. It’s lighter and much cleaner, and will hopefully provide a better experience for users.

I also want to add (text) file uploading to paste2 when creating pastes. When you have a big file pasting it in a browser window can be really annoying.

Another major project I want to do is to have a remote API using probably SOAP for people that want to create tools for interacting with the site.

I’ll be trying to get a beta up as soon as I can but some current work commitments mean I can’t spend as much time on it as I’d like.

On the note of scripts interacting with paste2 I was contacted by Andy Stewart regarding the Emacs script he created for interacting with paste2.org. After a few emails back and forth I implemented a feature for getting raw content of pastes so they can be grabbed by his script. I don’t use Emacs personally, but it looks useful.

That presents a problem for performance, in that every time somebody requests a page you’re doing a lot of work to get the application to a state where it can start spitting out content – then at the end throwing away all that hard work you did. It wastes physical time and CPU cycles, not forgetting often IO and memory in the process. To resolve the issue you have to make your application persistent, which is something that PHP isn’t designed for in this context.

A while back I discovered that somebody wrote an SCGI handler in PHP – so that you can write your application in a way that you build it up and it serves requests from it’s already-running state, your HTTP server talks to your application via the SCGI class and dumps this output back to the HTTP server and handles the next request without it all being torn down. That’s useful, it’s what you want when you’re serving huge numbers of requests in short spaces of my time. The problem is this isn’t a complete answer to the issue.

Firstly the code can only handle 1 request at a time. If a concurrent request comes in it’ll get queued by the HTTP server until the previous request has finished. This is easy enough to resolve – if your HTTP server supports it you can just spawn multiple instances and make it round-robin the backends. Another problem is these different processes don’t know about each other, they’re completely dumb to the fact that there are more versions of themselves running – they can’t share data with their siblings, and if you want to have a process to watch over them and make sure everything is fine you need to write that code separately. Then you get even more problems if your traffic gets to the point where it’s too much for the number of processes you have spawned; you need to manually increase the number of processes.

I decided I needed to come up with a solution to the problem.

First on my agenda was to ensure it spoke a language that most if not all HTTP servers understand. SCGI is pretty well supported, but not by everything. FCGI uses a complex binary communications protocol, which is a good protocol and it’s well supported – the problem is when you’re attacking a problem like this you have no idea if in the end your solution is going to improve things at all – so it’s potentially a complete waste of time trying to implement a protocol if the project fails.

I settled with making the server code support HTTP, which aside from the fact it’s well supported, has other fringe benefits. Firstly you can point a browser at the application and test the code directly with no HTTPD needed, which creates all sorts of opportunities for doing fun things when debugging, not least because you can find out if your problem is caused by your code or the HTTP server. You can also stick hardware load balancers in between your frontend HTTP server and backend slave application servers if you need that kind of capacity. HTTP load balancers are easy to get hold of. You can even put squid between the app server backend(s) and the frontend HTTP server if you need to without making any changes to your code if you need to. Now bear in mind this is was never intended to be exposed to browsers directly, just you can because it speaks HTTP 1.1 – but it’s not the idea. This is not a HTTP server in it’s own right, not even close., it isn’t intended to be used this way. This is not Nanoweb.

The next step is was to resolve the 1 connection at a time issue. Unfortunately PHP doesn’t support threads in any kind of way, shape or form. It’s not the end of the world, but despite there being a PECL extension covering threads this isn’t going to work, it hasn’t been updated in years – and even when it was updated it never gone finished, which is a huge shame. Hopefully one day somebody recovers that extension because it has huge potential outside the traditional “PHP does this and only this” worldview.

The only opportunity for doing this stuff in PHP is to use pcntl_fork(), so I hacked in some forking code. Experience teaches me that it’s not sensible to fork our server on every request – you have to fork enough children to handle your load up-front. What I was left with after a few hours was a HTTP server class that is extended extend by your application class, in a similar way to the application server I found previously – you write a constructor that builds up stuff you always need, maybe a template engine, read in some data files, connect to a DB and whatever, then it returns to the server class which forks the process into a bunch of children. When a request comes in i hits one of the children which processes the HTTP headers and body (if it’s a POST), which in turn passes the request off to your code.

After bashing it with ab it was clearly very fast, stable and working well. The result was a fully-functioning web application (a rebuild of paste2.org) spitting out it’s home page in about 0.0004s on my crappy local development server. For the sake of unscientific comparison, the ‘same’ code on paste2′s production server manages to spit out the same page in about 0.01s. That’s a pretty major time improvement, on significantly worse hardware.

It has a few issues still else I’d be putting the code up for people to look at – they’re all solvable though with a bit of time which I don’t really have right now. The first is some of the code is a mess as it is. The second one is if MySQL (or more accurately, I’m using PDO right now, hence whatever you use) goes away the code isn’t aware and tries to keep banging away at the old connection/prepared statement – but that’s going to be easy to resolve. The next issue is I haven’t added any kind of communication between processes yet but that’s just a matter of getting to it, and the code doesn’t need it – it will just add more reasons to use it. Basically right now it’s not even as cool as it could be.

Another problem is that no opcode cacher for PHP supports CLI as far as I can see, not because of technical difficulties but that they, I guess, have been completely useless in the past. The reason why this would be useful is that some template engines (including mine) repeatedly try to include the same file which it seems isn’t cached anywhere. That could be speeded up by an opcode cacher. I could use the CGI SAPI or patch XCache and give it a try, and then see what happens. [Update: I tried this with APC in CGI SAPI instead of CLI and it does improve things, but with the nature of the beast milage may vary].

The last issue I can think of is that I haven’t written in a way yet for the code to monitor itself and spawn new children when it gets clogged up or kill off children when it has too many. It does spawn new children when another dies though, that is to say it ensures there’s always x children spawned.

Because the app is persistent there’s some other fringe benefits that one might not usually think about.

Around the time I was hacking this together, in ##php on Freenode, Rasmus was complaining about how there’s no cache for prepared statements in MySQL. A few hours later it occurred to me that because what I’ve been doing is persistent across requests and that it holds open connections to the database server if you make your app work that way (and you want it to work that way) it’d be possible to implement something resembling a prepared statement ‘cache’. I should say before I get slapped around for what follows – I know what I’m about to describe isn’t actually a cache but it’s the next-best thing.

I pulled out the application-specific PDO code from my test application (the paste2.org rebuild), and moved it to the main app server parent class. Then I added two methods, one creates a prepared statement from a string and an identifier string (whatever you like as long as it works as a PHP array key) and stores it, the other returns a reference to the ‘named’ prepared statement. Essentially you can now add prepared statements in your app’s constructor (or wherever, realistically) and reuse them time-after-time-after-time.