Why So Many Clueless Programmers?

Recently, I’ve the distinct joy of dealing with incompetent programmers on two different, unrelated, projects. Or perhaps they’re competent but simply don’t give a damn.

First up: A new regional service provider for an association of professionals. I have nearly a dozen clients whose websites depend on a datafeed, which was provided by the previous service provider for this association. The old datafeed was excellent — clear and concise, comprehensive, and easy to use. The new provider first hemmed and hawed and offered vague responses to direct questions….. Somewhat important questions such as, “Will you still provide us with a datafeed?” If this service provider was not going to provide a datafeed, I would have to rewrite all the programming on my clients’ sites to use a different datafeed from a secondary source, which wasn’t as good as the one we were currently getting, but at least it was something. If the new provider would be providing a datafeed, I needed the schema for it as soon as possible, because the old datafeed was going to cease on a particular date, and any new datafeed would likely require rewriting my programming.

The new provider, though, simply couldn’t seem to tell us if they would be providing a datafeed or not. First the answer was “yes,” then it was “no,” then it was “maybe,” then it was “yes, and it will be compliant with the national standard,” then it was “if you pay us for the custom programming we will.” I was just about at the point where I would have to use the other available datafeed in order to get my clients’ sites updated by the deadline, when this new provider made a datafeed available.

And when I finally got a look at it: Oh boy, does it suck. First of all, and I’m actually very glad of this, the datafeed omits some of the crucial information that my clients need. I couldn’t believe that a large professional service provider such as this one would omit such critical information from the datafeed they provide to their members. Keep in mind, the members, including my clients, provide the data in the first place, and pay the service provider handsomely to maintain it for them. This isn’t some free service where you get what you pay for.

As I said, I was glad they omitted this information, because it allowed me to recommend to my clients that we use the secondary datafeed from a different provider. My clients wouldn’t have understood, or cared, about my other objections to the datafeed, but the missing information they understood and cared about.

And what are my other objections to this new feed? Well, first, it completely ignores the flexibility of relational databases, and has never even been in the same room with the concept of normalization. As an example, for a particular feature, the datafeed has 5 fields for characteristics of that feature. What will happen when 6 characteristics are needed? The datafeed will have to be restructured to add a 6th field, and my database that uses the datafeed will have to be restructured, and the programming code that selects the data and displays it in a web browser would have to be re-written (again) to accommodate the 6th characteristic. And then a 7th characteristic will appear at some point, and I’d have to go through that pointless exercise all over again. The datafeed is structured this way for about 9 different features — and it’s almost certain that these will need to be changed on a regular basis.

It also means that for a feature, say, “surface,” a particular characteristic, say “fuzzy,” might appear in field #10, or field #11, or field #12, or field #13, or field #14…. Well, you probably get the idea. There’s no consistency at all with respect to what goes where in the database. So that if I wanted to search the database for items that have a fuzzy surface, I have to write the code to search multiple different fields. This is both stupid, and highly inefficient.

The previous service provider understood relational databases and had structured their datafeed so that they could add a 6th characteristic, and a 7th, and even an 8th, 9th, and 10th, and it would continue working with my database “as is,” with no re-programming required. It’s not that I’m lazy — when re-programming is required, I do it. But I have better things to do with my time than waste it re-writing code that shouldn’t need to be rewritten, and my clients have better things to do with their money than pay me for that time.

And they did silly things like break up a single 1,200-character text field into 5 separate text fields, breaking up the text between words in a sentence, if needed, to fit them into the shorter text fields. That’s easy enough to handle in my code — I would simply concatenate all 5 fields into 1 in my database query. But it shouldn’t be necessary

The complete lack of normalization, and the moronic handling of the large text field, are indicative of a programmer who doesn’t know jack shit about databases. And that was my biggest concern of all. Even if I were to use the database as provided, what would they do with it down the road? They might decide to remove or add fields without telling anyone, or make other changes that would totally break the sites that depend on the datafeed.

We ended up using the secondary provider’s datafeed. It’s not as good as the previous one we were getting, but it’s infinitely better than the datafeed from the new provider.

Second: eBay, of all people! You would think a company the size of eBay would have highly competent programmers who know what they’re doing.

News flash: They don’t.

I have a number of sites that earn affiliate commissions from eBay’s Partner Network program. I can view reports on my earnings within eBay’s ePN website. I can also download those reports, for importing into a spreadsheet or database.

I want to be able to view and manipulate these reports in different ways than the ePN interface allows, so I’ve been working on a program that will import the downloaded reports into a database and allow me to search, sort, and analyze in a variety of ways. I think it’s a very cool program, and I may develop it further in order to release it publicly.

But the transaction reports that I get from eBay made my shake me head. One day, the report might contain 9 fields, and the next day it might have 11, and on another day it might have 13. I think there are some other possible permutations possible, too, but until I get such a report in my account I won’t know exactly how many fields it will have or in what order.

This makes it much more difficult to import the data into my database. I can’t just “LOAD DATA INFILE,” nor can I simply parse the files and do a standard INSERT query. Instead, I have to parse the files and check for the number of fields, and check to see what headings the fields have, and write several different MySQL queries to use, depending on the results of the parsing.

These are basic tab-delimited files. eBay’s programmers should know enough to just put a zero or a null value, along with the tab, for the fields that aren’t needed for a given report — such that all downloaded reports would have exactly 13 fields, always, and they would all have the same fields, in the same order, always.

I sometimes feel overwhelmed and outclassed on the programming forums where I hang out. There are so many extremely competent, highly skilled programmers who can toss off a complex INNER JOIN of 5 different tables as easily as I can type my name.

These programmers would laugh their asses off if they were to see these datafeeds and reports that I’m dealing with.