Ach, dammit. I don't like Twitter, mostly because it seems vapid, but then again I also will listen to anything Stephen Fry says, even if he's just muttering to himself.
But as silly as the Kutcher/CNN war was, the point is that a shitload of people not only heard about anti-malaria efforts, but could click on a link and immediately donate. For me, that's the exact opposite of the #amazonfail lunacy -- one was a viral and fundamentally misguided exercise in social networking, and the other is a way social networking can be used for good in supporting a worthy (and longstanding and researched) cause.
Kutcher and Oprah and King aside, read up on Malarianomore.org. They're good people, and anything that prevents a kid from contracting malaria (and stopping malaria from becoming drug-resistant) is worthwhile.
How many of you have ever dealt with really, really large databases and really, really complex software?
I came to Jezebel through Valleywag and Consumerist and stuck around because I really enjoy the articles and comments here. But today, folks, chill.
I work for one of the largest IT companies in the world. Not Amazon, but beyond that, I'm going to say who. I know only one person who works for Amazon, and only slightly, and I have not had any contact with that person about this matter. In other words, I have some background in this arena, but I don't have a dog in this race.
Here's how this sort of thing goes down.
Person 1 gets an assignment, or decides on his own to do something new and useful. Person 1 screws it up. Either he's a complete idiot with too much access to the system, or he just makes a really bad assumption. Perhaps he screws it up incrementally over an extended period of time. Anyhow...
Easter weekend (for the Christain folks) comes along AND someone notices the oddity. This being the internet age, real-time discussions start happening. Twitter. Blogs. Online news sites.
Some folks start asking Amazon customer support (though how they found the contact information escapes me) about what is happening. Support knows nothing about good old Person 1 and explains some semi-related aspect of the site (that the "adult" stuff is excluded from rankings, if I have it right). Because it is semi-related and something is obviously happening, much speculation ensues. Then an internet shit-storm forms.
Eventually someone at Amazon notices. Either customer support realizes that they are getting a lot of hits on the subject, or someone else in the company is reading the internet on Sunday, but a bug -- and probably a very high severity one -- is opened. People start getting called in.
At this point, the situation is probably going to be described as a "glitch". Cause until they find out what the reason behind the problem is, what else can they possibly call it?
And maybe they *are* working on some way to permit filtering on results, and they think that some of that code got into the production side. They speak too soon, so now you've gone from policy to glitch to new feature.
Then you've got some random person claiming that he did it by exploiting a weakness in their feedback system. (And maybe that is true and there is no Person 1, but let's go with Person 1 for this example.) Policy to glitch to new feature with a side order of hacking. Just in case it is a hacker exploiting the feedback mechanism, they disable the mechanism in question. (I think I read that they had done so.)
At this point, people of the VP rank and higher are probably standing in Amazon geeks' offices (or cubes, whatever) breathing fire. But until someone at Amazon has a chance to look at the data, the logs, and the applications, they don't know what the problem is.
Frankly, I'm amazed that they tracked it down as quickly as they did. Maybe Person 1 was reading the internet too, and realized that his project was probably to blame. (And if that's the case, I hope that they don't deal too harshly with him if it was an honest mistake.)
Amazon has nothing to gain from intentionally doing this and a lot to lose, as we can see.
Syrenia's rule for technology-related problems: Always assume incompetence until proven otherwise. For example: I didn't mean to destroy the entire knowledge base with one poorly written SQL query, but nonetheless I did. Oops. Good thing we had those backup tapes.
Syrenia's corollary for conspiracies: They are too much freaking work. It's probably just incompetence.
I was totally ready to cancel my Amazon Prime and drop them entirely. I was even looking for another source for my British sci-fi DVDs. But this sequence of events is totally believable to me, as someone with more than a decade in IT. Either this is what really happened, or they are the best liars in the world. If they had had one single explanation -- if the explanation hadn't evolved in the sort of way that it did, I wouldn't have believed them.
@Syrenia: I think that's quite plausible, but I'd feel better about it that explanation came from Amazon, not from people on the internet trying to fill in the gaps in their explanations. And even if that's the sequence of events, they should be apologetic about it. They don't have to grovel, but they've been very slow and not especially sympathetic in responding to the outrage, and it's not coming off well.
@samethingwedoeverynightpinky: But nobody was giving them the time to figure out what had happened. Customers cancelling accounts, news media pounding them for an explanation, all that. It's been what, 36 hours? People have been screaming for blood since yesterday.
The general attitude (not yours) that I've been seeing has been: "I want an explanation, and I want it now, and it better be one that I like. And I don't care that it's only been three hours since the problem was even brought to your attention and you've only gotten hold of three DBAs and one sys admin, and that it's Easter weekend and half your staff is on vacation. And even when you do explain, I've already made up my mind that you are evil because this happened at all, so there."
From the LA Times article on the subject: In striking contrast to all of the online uproar, Amazon -- a leading Internet company and media pioneer -- remained nearly silent. It issued a brief statement Sunday evening, citing "a glitch," then waited most of Monday before issuing the mea culpa. [ [www.latimes.com] ]
My post and many others like it (see Limber's excellent 10:58 comment) are just trying to provide context to people who are fortunate enough to not deal with these sorts of things on a regular basis.
The better initial response would have been: "We have become aware of this problem and have called in staff to investigate it. We will have no further comment until we have determined the cause and a course of corrective action." But unfortunately, even if you could get a PR person who was willing to issue such a statement, nobody was willing to give them even one full day to figure it out.
I have to disagree with you about the apology. I don't think that they need to be apologetic about it. Ultimately, what real harm has been done? For a few days, the search results are screwed up. (And I tested it yesterday with the book that is pictured in this article -- on a search of "dating guide lesbian" it was the number one hit Sunday evening. So it wasn't that the title couldn't be found at all.)
I think that we've reached a point as a society where too many apologies are made for too many things and they don't mean anything anymore. "I'm sorry" has become the most meaningless phrase in the English language.
Frankly, I was impressed by the phrasing of their statement, which is refreshingly blunt and conveys to me more real feeling than any corporate apology ever has or will.
@Syrenia: No matter what happened, or how accidental it was, yesterday Amazon told a population that has been told over and over and over again you're less than everyone, you're dirty, you deserve to be hidden, that they were less than everyone, dirty, and deserved to be hidden. It was very triggering for a lot of people. Not to mention that it's been happening on a smaller scale for months, and people's books have lost sales. If I trip and knock you down, then I'm going to apologize when I see you bleeding, even if I didn't push you on purpose. At best, they made a huge error that resulted in a deeply offensive outcome. Sorry is just basic decency when a) you screw up b) you hurt people you care about, especially when those people support you.
I guess from a PR perspective, I find it hard to believe that people who value what you call "bluntness," will stop shopping there if they apologize, whereas if I don't feel like they get that they hurt people and they're sorry and they mean it, I won't shop there again. So to me, it seems like it's in the interest of not being an asshole and in the interest of business to say sorry, but if they only want to cater to the consumers who prefer the blunt get over it attitude, I can take my business elsewhere. I think businesses, even anonymous internet businesses, should think of customers first as human beings. I worked in a big chain bbookstore, and if I felt the customer was upset in good faith, I apologized for all kinds of things that were not directly my fault, even if I couldn't change the policy or make it right in he way they wanted, because I didn't want them to think of our store as a space where they were humiliated or treated badly.
@Syrenia: The thing is: The only reason this "glitch" was able to happen is because things had already been classified in a way such that "Heather has Two Mommies" could be considered to be "dirtier" than "American Psycho."
It's not so much the human error (which is what I would call the "glitch" since it probably originated from a human judgment call rather than a bad line of code.) but the way that anything outside of the mainstream gets lumped together in a way that makes it all too easy to accidentally exclude.
I agree that incompetence is probably the best explanation -- but this incompetence was only possibly because everything about gays and feminists had already been lumped together in their taxonomy.
@Syrenia: Directly, no. However, I am related to someone who does, and in their opinion, they're skeptical because of previous incidences, albeit only a few. For it to suddenly explode from one or two minor incidences to a shitstorm is a little bit fishy.
(And no, I can't be any more specific than that, other than to say their job involves govt. controlled databases)
@probablyawkward: Exactly. I'm pretty sure the widespread exclusion of sales rankings was an error, but the fact that gay and lesbian content can so easily be tagged as potentially offensive or adult, even when it doesn't depict sex, comes from bigotry.
I'm not buying it. I'm a librarian and while it's not my specialization, I know a thing or two about indexing and cataloging. I'll grant that it's possible that there was a "glitch" in a search algorithm, but I can't imagine that someone wrote a code that decided to go after Heather Has Two Mommies and Virginia Wolf and Dan Savage all on its own. It seems like maybe they were trying to strip rankings from materials dealing explicitly with gay sexuality (which is bad enough) but ended up going further than they'd intended. The fact that they're even experimenting with this on a limited basis is disturbing. If anything comes of this, I hope it is that it's going to be harder for Amazon to use the LGBT collection to "experiment" on in the future.
@Miss. Money-Sterling: It's not necessarily code doing this, it's word classification and node alignment. I'm a search algorithm designer and taxonomist; I cannot describe how many different ways this error could have occurred.
You also have to keep in mind the purpose of matching algorithms. Part of tweaking the engine is trying to provide the most appropriate matches to the most receptive audience. It is not designed to be an egalitarian system, and tweaks will attempt to guide users towards the content they're most likely to buy. It's a marketing tool, not a faithful factual tree like animal classification.
I realise this is shouting into the wind at this point, but unless you have worked with complex algorithmic matching systems with hierarchical nodes, you're out of your depth in assessing the Amazon error. And a lot of people are making assumptions that are being taken as fact, and while something like "well, how else do you explain all the gay tags being triggers?!" may sound logical to the uninitiated, trust me, it's not that simple.
The scenario described by Amazon is totally feasible. Tagging can be like coding; a single error in a single node can have far-reaching consequences. It takes a long time to trace back that error. The lack of instant, flawless pinpointing of the problem... It's unrealistic.
It's invigorating to be angry and activist, and getting Amazon to fix the glitch is worthwhile, but damn. People are cancelling their accounts due to a simple and probably benign tagging error. The tagging error Amazon proposes is not only plausible, but it's enormously likely, and a scenario I have experienced time and again while working with matching algorithms. It fits.
@limber: I don't think it's infeasible, but I do think their explanations have been half-assed and inconsistent. It could be a case of people who don't have the authority or information to speak speaking on behalf of the whole company, but the PR people need to have been aggressive about a) offering a plausible explanation that answers the questions people have about why only certain material was affected (it may seem basic to people who work with code, but obviously it doesn't to everyone.) People claiming to reprsent Amazon have given multiple explanations for this. OK, yesterday was a holiday, but they had a full day to get it together, and the response still seems disjointed and half ass and not answering the most pressing questions people have or promising a more detailed response or investigation. b)saying they're sorry this happened. Hones mistake, sabotage, or deliberae malfeasance, this was a big offensive screw up, and it deserves more than an oops! ish happens!
@samethingwedoeverynightpinky: @carol.teater: If people are getting up in arms about the PR approach, damn, I don't know what to tell you. The first day was Easter, and likely no one was on-hand to access the logs and figure out what was going on. And then, even when they found the glitch, they had to go crawling through the algorithm to find what the hell was throwing the glitch. It's hugely time-consuming. (And unfortunately, in my experience, algorithm "tweaks" do not get QA'd anywhere near as rigorously as code, mainly because tweaks are usually needed quickly and the QA cycle takes 8 weeks.)
Their explanations were probably half-assed because they had no idea what the fuck was happening. It could be tagging, it could be trolling, it could be hacking -- what are they going to do, come out saying "we have no idea what the fuck's going on"? So they came out with "We have a glitch", which just meant they acknowledged the error, and bought them time to find the actual cause. The hasty relisting we saw yesterday is probably a quick fix that they'll eventually have to replace with the longer-term catalog fix.
If this is what they say it is, and I'm telling you that the patterns of product matches makes me think that is INCREDIBLY LIKELY, then all of this canceling and activist rage is largely misplaced. It's not intentional, it's an error in the process, they'll QA the hell out of it and it's fixed. Tagging and tweaking in these cases is more of an art form than a mechanical exercise.
@samethingwedoeverynightpinky: This is exactly my point. If they had said, "We're sorry, this was a mistake, here's what we're doing to fix it, and we'd also like to offer a 10% sale on all the books that have accidentally been incorrectly tagged, and we're gonna donate money to some big gay charity," then I'd give them the benefit of the doubt. But they haven't done any of that. They've danced around the issue, can't tell us what the real problem is (technological? cataloging? coding? deliberate attempt at not offending people? it depends on who's asking, who's answering, and when the question's getting asked), and are being sketchy about the whole thing.
If it was just a computational error, why couldn't they just straight up say that and move on? Apologize for the error (recognizing that it happened), and then do something to fix it. If it wasn't a deliberate and homophobic thing, then they wouldn't have had to be so sketchy about the whole issue, they could have just explained and apologized. They didn't. That means either that they are trying to hide something, or they are comfortable lying to their customers if they think nobody will notice. Neither response is acceptable, especially when it comes to something as serious as effecting censorship.
Lots of people make coding mistakes, and people understand that. But how, where, and when the mistakes happen do matter. If someone in the government made a mistake and deleted everyone in New York's social security number (or something), there would be a big fucking deal made about it. This isn't the same thing, but it's also not a commenting problem at Jezebel or something. It's a volatile subject for a lot of people, and deserves an apprpriate response. Theirs was not.
@limber: There have been books wihin these categories losing ther Amazon rank for months. Even if the huge switch was simple user error, it seems like it accelerated something that was already happening piecemeal. I haven't heard a single official explanation for why this has been going on for so long, or even an unofficial explanation, beyond the possibiliy of selective trolling. Amazon is now putting out an official explanation that claims this was the result of a single user error, and they know exacly what the error was. It doesn't wash. If that's not true, and they still don't know what caused it, they should not be lying about it to try to sweep it under the rug. I'm not canceling my account or heading to Seattle with a pitchfork, but I won't be shopping there until someone with authority gives an explanation for the earlier changes, and indicates that they take what happened seriously. I understand that people on the inside may be annoyed if they are being accused of something they didn't do, but the realities of PR mean that you can't respond to something like this by having a corporate hissyfit and assuming your customers ought to know better. The language they've used has been pretty blase about the whole thing, which seems like a pr misstep.
@samethingwedoeverynightpinky: I've only heard of two authors who contacted Amazon about their rank vanishing before this weekend; I'm not sure that would ever have made it back to the tagging teams as a verified glitch, it might've just been tweaked on a page-by-page basis. It's only this weekend that this appeared to explode, when many authors found books delisted.
Reserving judgment is absolutely reasonable, and hopefully Amazon will come out with a concise explanation once they're sure of what it is (though I assume people will still be suspicious). But expecting PR to be immediately omniscient and all-knowing strikes me as weird; I don't think they COULD have given a firm rationale for what was happening at first, I don't think they'd found the category error yet. If they'd come out immediately with a cause, they'd've been lying.
@samethingwedoeverynightpinky: and they don't have to give their whole algorithm to give a plausible explanation. I gave an explanation upthread for why it intuitively makes sense to me tha Queer interest books and and anti-gay books might not share the same tag. They have different target audiences, and both audiences would be offended if they got one kind of material while searching for the other. It may be that coding put in place to protect Queer folks from anti-queer propoganda popping up during searches backfired here. But if that's the explanation for why one kind of material was affected, and the other wasn't, why couldn't Amazon say that? It took me two minutes, and there was no math involved.
The biggest two questions people have are why this started months ago, and why only books with a certain political slant were affected. Amazon's official explanation ignores the first question, and glosses over the second. "It wasn't just gay books, it was feminist and sex positive books too," does not make the situation better or answer the questions about political bent. "We have systems in place to try to keep people browsing for GLBT interest books from being bombarded by anti-gay propoganda, so those books are categorized differently," might answer the question, but if it's that simple, why hasn't anyone said it yet?
@Cimorene: Bingo. If it was just a coding malfunction or whatever, then that's cool and I won't hold it against them. But the fact that it was such an emotionally charged fuck-up that offended so many people on so many levels should have warranted some major ass-kissing apologies by now.
@Cimorene: Exactly! The people that do the cataloging are not the same as the PR and legal team. Those people should have sat down FIRST THING and written up a statement for the Amazon front page.
"We are having problems with X,Y,Z. We apologize to the authors effected and for the inconvenience to our customers." That takes what? 30 minutes? An hour?
Instead they let the the internet come to it's own conclusions for almost two days. Amazon of all companies should know how fast information moves on the internet. The only way to handle these problems is to get out ahead of the controversy. They failed to do that and they still haven't apologized. Whether it was intentional or not, it still effected thousands of authors, publishers and customers. Amazon totally deserves blame for this, they dropped the ball in a major way.
@Laura Enriquez: Yes, this. If the coding people want to get huffy about the fact that people don't understand the system, or the work involved, or how fragile it is, or how long it will take them to fix it, that's fine. All they have to do is fix it as best they can-- if they want to grumble that people who think it was deliberate are confused or dumb or what have you, I don't care.
But if the PR people for a major corporation that just, intentionally or not, made an already atacked and marginalized demographic feel more attacked and marginalized, want to take the same huffy it-was-just-a-mistake-chillax-already tone, that's a huge problem. It's not like anyone who cares about this sort of thing hadn't heard about it yet. Along with plausible answers for the big questions (which they have not yet offered),there should have been an apology on the home page, a better worded form email that actually made it sound like they were sorry about what had happened and valued their offended customers' business, and possibly a sale or some kind of featured section with some of the deranked books, to show "See, how crazy is it to think that we would have done this on purpose? We value these books and authors and the customers who seek them out." Beat Powell's to the GLBT sale, show your customer base you mean it, and you're not catering to any element who would find a main page apology offensive.
@limber: Thanks for this input. It's pretty shocking that Amazon would intentionally do this, and really didn't seem to make a whole lot of sense, but then, there are a lot of things that don't make sense in this story.
You raise a good point, and maybe this will help some people (myself included) take a step back, see how Amazon deals with it (as it's only been about 48 hours since everyone realized what was going on) and then decide to support/not support amazon.
I've done the research for myself today (and I don't want to admit how much time I wasted doing so). I've decided that, for the time being, I will not place any orders with Amazon. I've also cut off all email they send to alert me to new releases and other items hat are of interest to me.
Officially we're done until they prove they were not at fault with this 'glitch'. I don't see that happening anytime soon, but the burden of proof lies with Amazon.
I don't understand - this was happening to a guy back in FEBRUARY and then magically it was fixed when he noticed it. And it wasn't a "cataloging error" then was it? So how does this new explanation make sense?
Did I miss something or is it exactly what I think it is?
@missnicolec: It probably was a cataloging error back in February -- but it also probably didn't reach the IT or content management team back then, either. More likely, someone went for a quick fix and either deleted the tag causing the problem or otherwise tweaked the author's book listing. Simple, quick, gets results.
It would take noting a pattern to make combing through the taxonomy worthwhile. And now a pattern's been noted -- unfortunately, it's being whipped up as a discrimination issue while the employees are likely desperately searching for the causal relationship that's making this happen.
Even if Amazon's "story" (such as it is) is true, there is still something terrible and offensive about this.
Even if you're d-listing hardcore S&M erotica, this is not okay. You're a retailer, not my mom, and you don't get to make moral judgments. If your corporation is not comfortable with these things, then don't sell them.
Instead of trying to hide the "adult" books away, they need to implement a filter like Google's, so one can opt in or out of returning "adult" items.
"Yes, taking gay books-or any books-off the rankings list seriously limits how many will sell, but isn't it up to the bookseller to decide what the market wants, what it will sell and how it will sell it"
Sily me, I thought we were against censorship in this country.
I want to give Amazon the benefit of the doubt, for now. I mean, even if they were actually all homophobic, they cannot be that... dumb. Hopefully this is all a mistake or some sort of internal conspiracy.
If this really WAS a 'ham-fisted cataloging error', it would have been fixed by today, you would have apologized to all of your customers and offered compensation to the authors and publishers of the books affected. Do I see that going on? No.
Do I think you had a secret agenda to de-rank and catalogue feminist and gay studies books in favor of homophobic shit? Abso-fucking-lutely.
@SO I HERD U LIEK MUDKIPZ (formerly Shamrockette): Yeah, it's a little sketchy, if it really was a matter of someone answering one question "true" instead of "false," that it would take so long to fix it, and they would have to do it book by book. And it doesn't explain the earlier episodes.
@SO I HERD U LIEK MUDKIPZ (formerly Shamrockette): Algorithms aren't as simple as you're implying. They had to find the error, then they had to game out fixes to find one that wouldn't make things worse. This was the most laborious, time-eating task possible when I worked with matching engines.
yeah, as i posted elsewhere, maybe they should have gamed it out in the first place, maybe?~
even if it was an honest mistake, their public response seems totes asshattery to me...i am getting more and more disgusted and hurt that the top hits are still books on how evil gay ppl are and/or how to coerce them into being str8...
ppl make mistakes, true character and values are shown by how ppl handle their mistakes...
@carol.teater: Gaming out every tweak is a huge, huge cost. And it's very rare that stuff like this happens so catastrophically; if this had started brewing on a Wednesday, I don't think we'd be having this conversation right now. They'd've made a tweak that covered (likely an ubertag that resurrected all the killed pages) until they could find the root of the problem.
The complaining about the PR -- well, hell, how do you deal with people feeling like your cataloging error has been a direct, personal insult to them? I feel like a great deal of rationality has left the building, and people are just thrilling off emotion. As I've said elsewhere, an instant response is only possible if the Amazon gang knew what was causing the glitch, and I don't think they did. I expect we'll hear more out of them about what happened and how sorry they are later, but this immediate gratification thing ("apologize for offending me before you even know what happened!") is weird.
@samethingwedoeverynightpinky: We agree that the PR team dropped the ball on anticipating the need to apologise for the result of the glitch; what I'm saying is that people are demanding explanation of how it happened, and Amazon might not have one yet.
Ham-fisted indeed. I emailed them yesterday and got the following: "We recently discovered a glitch in our systems and it's being fixed." (Yes, that's it in its entirety.) SO I am less than relieved by the current version of the response.
I just emailed them back suggesting they donate some money to HRC or PFLAG. Pipe dream, but it made me feel better.
@MisterMoonlight: The glitch and the catalog error are not different things. The glitch is an event that is causing an unexpected, unwanted result. The cataloging error is what is causing the glitch.
@limber: I'm not comment-stalking you, but it's a matter of definition -- for some of us "glitch" means something highly specific and different than "human error."
@probablyawkward: No worries, I'm not intentionally stalking anyone either -- the linguistic side of the algorithm is very different from the programmer side; I totally understand that one's error is another's specific definition.
In this case, I'm pretty sure code didn't cause this, and the PR usage of the word is in the more generic way rather than the programmer code.
04/18/09
But as silly as the Kutcher/CNN war was, the point is that a shitload of people not only heard about anti-malaria efforts, but could click on a link and immediately donate. For me, that's the exact opposite of the #amazonfail lunacy -- one was a viral and fundamentally misguided exercise in social networking, and the other is a way social networking can be used for good in supporting a worthy (and longstanding and researched) cause.
Kutcher and Oprah and King aside, read up on Malarianomore.org. They're good people, and anything that prevents a kid from contracting malaria (and stopping malaria from becoming drug-resistant) is worthwhile.
04/14/09
It's hardly the first time that Amazon's screwed up...
04/14/09
Thank you, I needed that.
04/14/09
How many of you have ever dealt with really, really large databases and really, really complex software?
I came to Jezebel through Valleywag and Consumerist and stuck around because I really enjoy the articles and comments here. But today, folks, chill.
I work for one of the largest IT companies in the world. Not Amazon, but beyond that, I'm going to say who. I know only one person who works for Amazon, and only slightly, and I have not had any contact with that person about this matter. In other words, I have some background in this arena, but I don't have a dog in this race.
Here's how this sort of thing goes down.
Person 1 gets an assignment, or decides on his own to do something new and useful. Person 1 screws it up. Either he's a complete idiot with too much access to the system, or he just makes a really bad assumption. Perhaps he screws it up incrementally over an extended period of time. Anyhow...
Easter weekend (for the Christain folks) comes along AND someone notices the oddity. This being the internet age, real-time discussions start happening. Twitter. Blogs. Online news sites.
Some folks start asking Amazon customer support (though how they found the contact information escapes me) about what is happening. Support knows nothing about good old Person 1 and explains some semi-related aspect of the site (that the "adult" stuff is excluded from rankings, if I have it right). Because it is semi-related and something is obviously happening, much speculation ensues. Then an internet shit-storm forms.
Eventually someone at Amazon notices. Either customer support realizes that they are getting a lot of hits on the subject, or someone else in the company is reading the internet on Sunday, but a bug -- and probably a very high severity one -- is opened. People start getting called in.
At this point, the situation is probably going to be described as a "glitch". Cause until they find out what the reason behind the problem is, what else can they possibly call it?
And maybe they *are* working on some way to permit filtering on results, and they think that some of that code got into the production side. They speak too soon, so now you've gone from policy to glitch to new feature.
Then you've got some random person claiming that he did it by exploiting a weakness in their feedback system. (And maybe that is true and there is no Person 1, but let's go with Person 1 for this example.) Policy to glitch to new feature with a side order of hacking. Just in case it is a hacker exploiting the feedback mechanism, they disable the mechanism in question. (I think I read that they had done so.)
At this point, people of the VP rank and higher are probably standing in Amazon geeks' offices (or cubes, whatever) breathing fire. But until someone at Amazon has a chance to look at the data, the logs, and the applications, they don't know what the problem is.
Frankly, I'm amazed that they tracked it down as quickly as they did. Maybe Person 1 was reading the internet too, and realized that his project was probably to blame. (And if that's the case, I hope that they don't deal too harshly with him if it was an honest mistake.)
Amazon has nothing to gain from intentionally doing this and a lot to lose, as we can see.
Syrenia's rule for technology-related problems: Always assume incompetence until proven otherwise. For example: I didn't mean to destroy the entire knowledge base with one poorly written SQL query, but nonetheless I did. Oops. Good thing we had those backup tapes.
Syrenia's corollary for conspiracies: They are too much freaking work. It's probably just incompetence.
I was totally ready to cancel my Amazon Prime and drop them entirely. I was even looking for another source for my British sci-fi DVDs. But this sequence of events is totally believable to me, as someone with more than a decade in IT. Either this is what really happened, or they are the best liars in the world. If they had had one single explanation -- if the explanation hadn't evolved in the sort of way that it did, I wouldn't have believed them.
04/14/09
04/14/09
The general attitude (not yours) that I've been seeing has been: "I want an explanation, and I want it now, and it better be one that I like. And I don't care that it's only been three hours since the problem was even brought to your attention and you've only gotten hold of three DBAs and one sys admin, and that it's Easter weekend and half your staff is on vacation. And even when you do explain, I've already made up my mind that you are evil because this happened at all, so there."
From the LA Times article on the subject: In striking contrast to all of the online uproar, Amazon -- a leading Internet company and media pioneer -- remained nearly silent. It issued a brief statement Sunday evening, citing "a glitch," then waited most of Monday before issuing the mea culpa. [ [www.latimes.com] ]
My post and many others like it (see Limber's excellent 10:58 comment) are just trying to provide context to people who are fortunate enough to not deal with these sorts of things on a regular basis.
The better initial response would have been: "We have become aware of this problem and have called in staff to investigate it. We will have no further comment until we have determined the cause and a course of corrective action." But unfortunately, even if you could get a PR person who was willing to issue such a statement, nobody was willing to give them even one full day to figure it out.
I have to disagree with you about the apology. I don't think that they need to be apologetic about it. Ultimately, what real harm has been done? For a few days, the search results are screwed up. (And I tested it yesterday with the book that is pictured in this article -- on a search of "dating guide lesbian" it was the number one hit Sunday evening. So it wasn't that the title couldn't be found at all.)
I think that we've reached a point as a society where too many apologies are made for too many things and they don't mean anything anymore. "I'm sorry" has become the most meaningless phrase in the English language.
Frankly, I was impressed by the phrasing of their statement, which is refreshingly blunt and conveys to me more real feeling than any corporate apology ever has or will.
04/14/09
I guess from a PR perspective, I find it hard to believe that people who value what you call "bluntness," will stop shopping there if they apologize, whereas if I don't feel like they get that they hurt people and they're sorry and they mean it, I won't shop there again. So to me, it seems like it's in the interest of not being an asshole and in the interest of business to say sorry, but if they only want to cater to the consumers who prefer the blunt get over it attitude, I can take my business elsewhere. I think businesses, even anonymous internet businesses, should think of customers first as human beings. I worked in a big chain bbookstore, and if I felt the customer was upset in good faith, I apologized for all kinds of things that were not directly my fault, even if I couldn't change the policy or make it right in he way they wanted, because I didn't want them to think of our store as a space where they were humiliated or treated badly.
04/14/09
It's not so much the human error (which is what I would call the "glitch" since it probably originated from a human judgment call rather than a bad line of code.) but the way that anything outside of the mainstream gets lumped together in a way that makes it all too easy to accidentally exclude.
I agree that incompetence is probably the best explanation -- but this incompetence was only possibly because everything about gays and feminists had already been lumped together in their taxonomy.
04/14/09
(And no, I can't be any more specific than that, other than to say their job involves govt. controlled databases)
04/14/09
04/13/09
04/13/09
You also have to keep in mind the purpose of matching algorithms. Part of tweaking the engine is trying to provide the most appropriate matches to the most receptive audience. It is not designed to be an egalitarian system, and tweaks will attempt to guide users towards the content they're most likely to buy. It's a marketing tool, not a faithful factual tree like animal classification.
04/13/09
The scenario described by Amazon is totally feasible. Tagging can be like coding; a single error in a single node can have far-reaching consequences. It takes a long time to trace back that error. The lack of instant, flawless pinpointing of the problem... It's unrealistic.
It's invigorating to be angry and activist, and getting Amazon to fix the glitch is worthwhile, but damn. People are cancelling their accounts due to a simple and probably benign tagging error. The tagging error Amazon proposes is not only plausible, but it's enormously likely, and a scenario I have experienced time and again while working with matching algorithms. It fits.
04/13/09
04/13/09
04/13/09
Their explanations were probably half-assed because they had no idea what the fuck was happening. It could be tagging, it could be trolling, it could be hacking -- what are they going to do, come out saying "we have no idea what the fuck's going on"? So they came out with "We have a glitch", which just meant they acknowledged the error, and bought them time to find the actual cause. The hasty relisting we saw yesterday is probably a quick fix that they'll eventually have to replace with the longer-term catalog fix.
If this is what they say it is, and I'm telling you that the patterns of product matches makes me think that is INCREDIBLY LIKELY, then all of this canceling and activist rage is largely misplaced. It's not intentional, it's an error in the process, they'll QA the hell out of it and it's fixed. Tagging and tweaking in these cases is more of an art form than a mechanical exercise.
04/13/09
If it was just a computational error, why couldn't they just straight up say that and move on? Apologize for the error (recognizing that it happened), and then do something to fix it. If it wasn't a deliberate and homophobic thing, then they wouldn't have had to be so sketchy about the whole issue, they could have just explained and apologized. They didn't. That means either that they are trying to hide something, or they are comfortable lying to their customers if they think nobody will notice. Neither response is acceptable, especially when it comes to something as serious as effecting censorship.
Lots of people make coding mistakes, and people understand that. But how, where, and when the mistakes happen do matter. If someone in the government made a mistake and deleted everyone in New York's social security number (or something), there would be a big fucking deal made about it. This isn't the same thing, but it's also not a commenting problem at Jezebel or something. It's a volatile subject for a lot of people, and deserves an apprpriate response. Theirs was not.
04/13/09
04/13/09
Reserving judgment is absolutely reasonable, and hopefully Amazon will come out with a concise explanation once they're sure of what it is (though I assume people will still be suspicious). But expecting PR to be immediately omniscient and all-knowing strikes me as weird; I don't think they COULD have given a firm rationale for what was happening at first, I don't think they'd found the category error yet. If they'd come out immediately with a cause, they'd've been lying.
04/13/09
The biggest two questions people have are why this started months ago, and why only books with a certain political slant were affected. Amazon's official explanation ignores the first question, and glosses over the second. "It wasn't just gay books, it was feminist and sex positive books too," does not make the situation better or answer the questions about political bent. "We have systems in place to try to keep people browsing for GLBT interest books from being bombarded by anti-gay propoganda, so those books are categorized differently," might answer the question, but if it's that simple, why hasn't anyone said it yet?
04/13/09
04/14/09
"We are having problems with X,Y,Z. We apologize to the authors effected and for the inconvenience to our customers." That takes what? 30 minutes? An hour?
Instead they let the the internet come to it's own conclusions for almost two days. Amazon of all companies should know how fast information moves on the internet. The only way to handle these problems is to get out ahead of the controversy. They failed to do that and they still haven't apologized. Whether it was intentional or not, it still effected thousands of authors, publishers and customers. Amazon totally deserves blame for this, they dropped the ball in a major way.
04/14/09
But if the PR people for a major corporation that just, intentionally or not, made an already atacked and marginalized demographic feel more attacked and marginalized, want to take the same huffy it-was-just-a-mistake-chillax-already tone, that's a huge problem. It's not like anyone who cares about this sort of thing hadn't heard about it yet. Along with plausible answers for the big questions (which they have not yet offered),there should have been an apology on the home page, a better worded form email that actually made it sound like they were sorry about what had happened and valued their offended customers' business, and possibly a sale or some kind of featured section with some of the deranked books, to show "See, how crazy is it to think that we would have done this on purpose? We value these books and authors and the customers who seek them out." Beat Powell's to the GLBT sale, show your customer base you mean it, and you're not catering to any element who would find a main page apology offensive.
04/14/09
You raise a good point, and maybe this will help some people (myself included) take a step back, see how Amazon deals with it (as it's only been about 48 hours since everyone realized what was going on) and then decide to support/not support amazon.
04/13/09
04/13/09
Officially we're done until they prove they were not at fault with this 'glitch'. I don't see that happening anytime soon, but the burden of proof lies with Amazon.
04/13/09
Did I miss something or is it exactly what I think it is?
04/13/09
It would take noting a pattern to make combing through the taxonomy worthwhile. And now a pattern's been noted -- unfortunately, it's being whipped up as a discrimination issue while the employees are likely desperately searching for the causal relationship that's making this happen.
04/13/09
Although I'd say this clusterfuck is more like "Side-of-beef-fisted."
04/13/09
Even if you're d-listing hardcore S&M erotica, this is not okay. You're a retailer, not my mom, and you don't get to make moral judgments. If your corporation is not comfortable with these things, then don't sell them.
Instead of trying to hide the "adult" books away, they need to implement a filter like Google's, so one can opt in or out of returning "adult" items.
04/13/09
Sily me, I thought we were against censorship in this country.
04/13/09
04/13/09
Is that a lesbian joke?
04/13/09
04/13/09
If this really WAS a 'ham-fisted cataloging error', it would have been fixed by today, you would have apologized to all of your customers and offered compensation to the authors and publishers of the books affected. Do I see that going on? No.
Do I think you had a secret agenda to de-rank and catalogue feminist and gay studies books in favor of homophobic shit? Abso-fucking-lutely.
04/13/09
04/13/09
04/14/09
yeah, as i posted elsewhere, maybe they should have gamed it out in the first place, maybe?~
even if it was an honest mistake, their public response seems totes asshattery to me...i am getting more and more disgusted and hurt that the top hits are still books on how evil gay ppl are and/or how to coerce them into being str8...
ppl make mistakes, true character and values are shown by how ppl handle their mistakes...
04/14/09
The complaining about the PR -- well, hell, how do you deal with people feeling like your cataloging error has been a direct, personal insult to them? I feel like a great deal of rationality has left the building, and people are just thrilling off emotion. As I've said elsewhere, an instant response is only possible if the Amazon gang knew what was causing the glitch, and I don't think they did. I expect we'll hear more out of them about what happened and how sorry they are later, but this immediate gratification thing ("apologize for offending me before you even know what happened!") is weird.
04/14/09
04/14/09
04/13/09
I just emailed them back suggesting they donate some money to HRC or PFLAG. Pipe dream, but it made me feel better.
04/13/09
There is no contradiction.
04/14/09
04/14/09
In this case, I'm pretty sure code didn't cause this, and the PR usage of the word is in the more generic way rather than the programmer code.