This weekend, I thought I'd extend my little Android usage tracking application to work on more ISPs than the one (Internode) that it already does. As my phone is (sadly) on Optus, I thought I'd write one for that.
In short order, I had a working prototype that used JTidy to clean up the HTML into something that I could use properly and then XPath to extract the elements of the document I needed. It works great, except that the document clean up and parsing into DOM takes a really long time on a resource constrained device such as a phone. It takes about 20 seconds to clean up and parse the document on my development emulator, which is too slow to produce a good mobile experience, especially if you have to parse multiple documents as I do.
So now I'm faced with a choice. I could write a man in the middle service that the phone sends the user's login details to which then performs the parsing on behalf of the user and sends the results on to the phone, but there are a number of drawbacks to this:
- This means that the user is sending his login details to a 3rd party, which is a security no no.
- It introduces a single point of failure into the equation. If my app gets popular the middle man service could get slammed. If Optus decides that they don't like what I'm doing, they could easily block it.
- It means I need to host a service, which means additional expense.
I don't want to do this, so what I'm left with is more hacky solutions, using regular expressions to find what I want in the HTML documents retrieved from the provider. This will take me longer to code, will be more prone to failure, and is just generally nasty. I'm not happy. Devices these days are very powerful, and there should not be the need for intermediary servers to help with processing.
Of course this would all be much easier if the providers published web services interfaces to their data, rather than just web applications. This has been the mantra of SOA and internet connected businesses since the terms were coined. It doesn't even cost them that much more to do it, and would lead to better designed web applications, but thats a subject for another rant. Optus doesn't do this because there's no economic incentive for them to do so. They gain nothing directly from publishing a usable web interface, so they can't be bothered... bah!
To be fair to Optus, they aren't the only ones that don't get it. No ISPs and telcos provide any decent interfaces, other than Internode.