I’ve been meaning to write this post for a long time now and while I’m 2 days behind deadline to give one of my clients another monthly report I have to take a break and spit it off my chest.
Over the past 8 months I tried and tested practically all of the key players in ’social media monitoring services’: Radian6, BuzzMetricsBrandwatch, Attentio, Techrigy (SM2) etc. And I have very few good words to say about them. The few good words will be in relation to the people who work at these companies - these are usually nice people. But the software and service is verging on epic fail - very problematic and clunky solutions wrapped in a big bold shiny (but rather hollow) promises. (OK maybe I’m being a bit too harsh….but that’s the result of two long nights sorting out monitoring dashboards)
I don’t mean and don’t have the time to give you a fully detailed review on each one of the tools I’ve tested. Honestly I can’t be bothered. so the following points i will make are rather generic and represent similar issues I had with ALL of them. I will bullet point so to avoid rambling…
1. The technology is fairly stupid. It either don’t do what it says on the tin or do it quite badly. It is supposed to be simple: based on keyword configuration, the software scrolls the social web and collect mentions of your keyword(s). It then supposes to analyse themes and influence ranking and sentiment etc, which it does with limited accuracy . For example, will pick up any header, ad sense or footer mentions of your keywords even if it’s in the totally irrelevant context. If your brand name is pretty generic you are in deep sh*t. Hours of configuration and exclusions awaiting you.
2. Unreliable data. The most important thing to understand is that the software simply provides you with piles of data. Before you can extract anything meaningful from this data you have to go through hours and hours of spam filtering which can be very tedious if you are dealing with 1000s of mentions every week/month. In some occasions I had over 50% irrelevant data coming through my dashboard. Additionally, the spiders cannot access all social spaces and sometimes the most important conversations are blocked.
3. Sentiment analysis is flawed. Again, this is part of the limitation of the technology. The software analyses keywords, not human emotions and, on average, the software gets it wrong 30% of the data because human emotions are subtle and complex and not easily categorised by software - we are not there yet.
4. Region specific data: for global brands, social media have very strong global element as well as clear regional bent (forums, blogs, networks etc). This is tricky especially if you are working with a regional client (e.g Huggies UK). Problem is for the software it’s not about where you are but which domain are you using. So reliable geo / regional analysis is, in many cases imposible to carry or not complete so need to be complemented with manual search.
5. Influence analysis is flawed. Well, the concept of influenced is flawed so of course technologies of measuring it are flawed as well. Similar to sentiment, the technology is just not as clever as they want you to believe. It is based either on bogus metrics or just irrelevant, obsolete ranks .
6. Unreliable data + partial data = Unreliable stats and visualisation. Because of the above, the one good thing for which you actually want to use this software, i.e. the funky visualisation tools can’t be used before you make quite a lot of amends and refinements to the data
7. Time consuming. Because all of the above, the reality is that while thess companies provide you with piles of data and funky visualisations the profound unreliability of the software means you have to sit for hours and days and configure the dashboard, refine the data, correct the scores, filter the spam, get rid of irrelevant data AND THEN, AND ONLY THEN you can start making some meaningful analysis.
8. Price. This varies significantly but the fact is that you pay just for the data and license fees to use the software. For the level of service you don’t get value for your money.
The overall feeling is of a serious problem of reliability of the results across the different functionalities of the software. In essence it’s like paying a market research firm for 10000 people’s survey results + 300 focus groups transcripts + 687 depth interviews and what you get is just the raw data with 30% of it irrelevant/spam you actually never asked for. Now you have to invest a lot of time in sorting out the data, analysing and interpreting everything you have. It’s doable but sometimes not really cost effective.
If you want to get the best out of these tools you almost have to plan for someone to sit 2-5 hours a day and sort the data (depends on the volume it’s a full time job.
At the moment I’m working with both SM2 and Brandwatch on different clients for different reasons and I can’t say I’m satisfied with either of them - it’s more of a default until better solutions will be developed. Until the software will be better at sorting through context and semantics it will remain very problematic and unreliable.
Solutions:
- Pay for them to do the analysis for you or always plan/cost for more man hours than you expect
- DIY dashboard with free services- search online there are quite a lot of how-to posts
- Wait for google to provide this service for free. This will either be the end of the bubble or they all will turn into market research 2.0 where you pay for the reports not just the data
naff said. back to my dashboard.
Comments 55
Hey Asi,
I’d love to chat with you sometime. Feel free to shoot me an email.
Posted 05 Aug 2009 at 8:15 pm ¶Oh, man!
I’ve been looking into these types of tools (particularly ScoutLabs and Sysomos) not knowing whether to trust the claims and so looking to understand the underlying principles they work on.
Thanks for the heads up.
Posted 05 Aug 2009 at 8:48 pm ¶Asi:
Posted 05 Aug 2009 at 10:22 pm ¶You make a lot of good points in this. We are a text analytics vendor and have built specialized natural language processing technology that mitigates many of the issues you cite above. We get very accurate analysis of sentiment and then can drill into root cause, identifying actionable issues and opportunities in customer conversations/social media…..
When you come up for air - let’s talk.
Thanks,
Michelle
Brilliant post. There really are so many challenges we all have yet to overcome. But do not fear, there are countless coders and text analyzers and product gurus working hard to push the envelope and develop these tools so that some day, they will be, at least, more dependable, more thorough, and a little less messy. Thanks again for tearing us all a new one
Julie
Posted 05 Aug 2009 at 11:28 pm ¶Community Manager @scoutlabs
Hi Asi,
I couldn’t agree with you more! The tools are pretty much only good for aggregating the information. Any type of analysis has to be done manually in order to gain any type of reliable insight. Wrote a similar post on the same topic, with the addition of pros/cons on each tool. Check it out here http://ow.ly/jdLF. Google better hop on the bandwagon!
Posted 06 Aug 2009 at 12:33 am ¶WOW! The emperor has no clothes!
Thanks for this excellent piece Asi, this is bound to get the social media monitoring companies aflutter!
Wish someone can come out with such an objective review of the offerings of these other companies in this space: BrandIntel; BuzzLogic; Collective Intellect; Converseon; Nielsen BuzzMetrics; BuzzGain; TNS Cymfony; Visible Technologies; PR NewsWire.
Posted 06 Aug 2009 at 2:02 am ¶Damn Asi, you got pwned by these vendors. how much do you pay for these tools? give us that info and save us from doing meaningless calls with these companies. We are looking at this space right now and I am going to avoid all the vendors mentioned in your post
Nice post.. very well done.
Posted 06 Aug 2009 at 4:08 am ¶Asi,
You are inspiring me to finish my own blog post on: Asking the right (tough) questions when choosing an analysis platform.
For example, people should be asking us here at Scout Labs, “How good is your spam/porn suppression? How often does it catch what humans would label as spam and what percent of the time does it catch the wrong stuff? Does it learn / improve over time?” But no one ever asks! Good results are SO fundamental to accurate counts, but no one ever asks. Tip 2: When vendors say “Our sentiment is really good” you should ask: “What was the algorithm’s agreement rate with human testers in your previous 2 tests?”.
Social media = human expression = inherently messy. No platform will ever be perfect. But attention to detail and meticulous obsession with quality will separate the men from the boys (or the woman from the girls, in Scout Labs’ case)
Posted 06 Aug 2009 at 6:34 am ¶Asi,
You make some solid points but I would suggest it is still early days for the social media analytics “business” so the technology and players will evolve and mature in time.
As well, there are players providing different features and levels of service. While I’m clearly biased, Sysomos is offering some user-friendly and valuable services through its Heartbeat (monitoring and measurement) and MAP (analytics) services.
If you’d like a demo to learn more about Heartbeat and MAP, please let me know.
cheers, Mark
Mark Evans
Posted 06 Aug 2009 at 12:13 pm ¶Director of Communications
Sysomos
mark@sysomos.com
Great post!
I agree that there is a lot of problem regarding quality of both sentiment, duplicated posts and spam filters.
We are addressing and have solutions to this problems (on the B2B side directly to the social media monitoring companies).
I believe it’s time to have a standard quality measurement for how good the sentiment is. Maybe a standard test or something (please contact me if you agree so we can start doing something about it)…
Posted 06 Aug 2009 at 1:00 pm ¶I’ve seen use-based estimates that it takes 5-10x the cost of the software to get good, useful information from these tools. More traditional clip tools are way better than the SM tools.
Just remember, no matter what they say, the lowest priced data providers (Meltwater, CyberAlert, Factiva) end up costing the most in time and resources because the junk data rates are often 50%. There’s no time to think, just time trying to categorize and chart…
You are better off, as KDPaine usually suggests, doing traditional survey research, or partnering with a SW+People solution (Biz360, MediaHound, Delahaye) and paying the extra.
But the bottom-line is that SM is nearly impossible to use for analysis, at this stage of development.
Posted 06 Aug 2009 at 1:30 pm ¶Interesting feedback from someone actually using ORM solutions.
Have you tried Sysomos ? Check them out if not.
For the readers looking for a free solution, read here : http://philgo20.com/2009/06/googlereader-socialmention-postrank-orm-for-free/
Posted 06 Aug 2009 at 3:15 pm ¶I agree with your prediction about Google effectively wiping out most the technologies you’ve described. The intense downward price pressure on monitoring tools, which is how I’d describe the products you’ve used, is evidence enough of the limited value offered.
You didn’t really state what your goals in using those platforms. While there are use cases for monitoring (crisis management, directly engaging influencers, etc), I believe that the real value is thinking of social media like a massive dataset, much like what CRM has evolved to. It’s an incredible dataset, when you sit down and think about it, offering based on the sheer volume, authenticity, and real-time nature of the data. There is amazing value to be had by performing in-depth analytics on that data and using it to inform strategic marketing decisions. Far greater than simply counting how many times your brand is mentioned, and whether it’s good or bad.
Analytics is the key - not monitoring. The market is already starting to segment but today most people still lump everyone together as “social media monitoring.” I’m looking forward to when full bifurcation takes place. Google & Twitter will own the monitoring market and it will be free.
Posted 06 Aug 2009 at 5:31 pm ¶I don’t agree that Google, Twitter, Facebook etc. will ever wipe out social media specialists. It’s not their core competency to develop B2B analytics offerings and it’s not their primary product mandate. Google, for example, has never exhibited the slightest interest in tools for specialists- as opposed to generalists. Thus you get products like SERP rankings, not influence rankings, and Google Alerts, as opposed to the kind Scout Labs produces, which focus on recent influential content, and highlight positive and negative developments for a brand or product. While I do agree that Google has the money and talent to launch it’s own space program, if it so chooses, I just don’t think they will choose that direction. Twitter is having trouble handling denial of service attacks, much less developing SAAS apps that span multiple social media content sets, and Facebook’s main business model is advertising. So- I don’t see any of them owning the “monitoring” market anytime soon.
Posted 07 Aug 2009 at 4:03 pm ¶Asi - I do believe I told you this about 9 months ago?
Posted 08 Aug 2009 at 1:16 pm ¶Great discussion. I just wonder how precise the art of social media monitoring needs to be in order to produce the desired results. Do we need more than 70% accuracy to pick up general trends of discussion and sentiment? Think about real-life WOM. You can miss, mishear and misunderstand comments, but you’re likely to pick up the gist and that’s normally enough. How much you’re prepared to pay for that is another question entirely.
I’m organising a Social Media Monitoring conference in London (UK) in the autumn. Hope some of you can make it
Posted 09 Aug 2009 at 11:25 am ¶Asi:
This is one of those “thanks for writing this post so I don’t have to” blog posts.
I can’t write this, because we are tangentially a competitor of all the brand monitoring companies.
But 80% of our work is in the analysis - not the data collection, clean-up and characterization. People need to know why not just count brand mentions.
The easy part is collecting and presenting data. The hard part is doing it in a way that is interesting, useful or insightful.
Tom O’Brien
Posted 10 Aug 2009 at 4:48 pm ¶MotiveQuest LLC
http://www.ubervu.com/
Try that it’s free.
Posted 12 Aug 2009 at 10:36 am ¶Y’all are trying to solve world hunger with a rake and trowel. These tools barely scratch the surface of usefulness. Anyone with a working knowledge of SPSS, SurveyMonkey, and some web logs will clobber you with insight and actionable information.
Posted 12 Aug 2009 at 12:14 pm ¶Great post! In the interest of full transparency, let it be known that we are a full-service measurement house that uses a hybrid approach to analyze and code media items for clients, so yes, in some way, I compete with the companies you mention. But the reason we started this 7 years ago was precisely for the reasons you cite. Until and unless automated systems can achieve greater than 90% intercoder reliability scores, you are never going to adequately be able to explain to the board or the C-suite what those numbers actually mean.
Posted 12 Aug 2009 at 12:30 pm ¶I totally agree with this article. The technology can still be good though if it is told exactly what to look for. That is why companies like Biz360 that have human experts that are fantastic at developing elaborate search strings with multiple inclusions and exclusions so that only the most relevant content comes in are so important. They can build the topics for the client so that only relevant content comes in. It’s just like anything else - if you bring garbage in - you’ll get garbage out. If you bring clean relevant content it, you’ll get clean relevant analyses. The problem is that with a cheap tool - you won’t get the benefit of that expert setting up the account, QAing the data etc. You get what you pay for!
Posted 12 Aug 2009 at 5:25 pm ¶Data Wonk…The market research tools you’ve listed are good for answering specific questions, not for generating insights.
Posted 12 Aug 2009 at 6:38 pm ¶If we’re looking at mapping trends, hard facts etc, then market research is great but when we want to know what people are thinking, we turn to Social Media analysis.
….it stripped out my ml!!
hi
[dilema]
i’m the ceo of brandwatch - and i didn’t really know whether to respond or not here - the dilemma of social media?!
[/dilema]
[thanks class="through-clenched-teeth"]
so, thanks asi for sharing, even if a lot of this is maybe hard to hear without putting up my hand and saying, but, but, but etc
[/thanks]
Quick (unavoidable) defence response, then an observation.
[defence class="in-ours"]
i’m sure i speak for others as well as ourselves when i say our backend technology is certainly not string and glue put together by a couple of undisciplined amateurs. we have world-class developers at Brandwatch and our infrastructure is a proper piece of engineering. Whether we have gotten there or not (not, clearly!) with the application, we are at least staffed by professionals who are working very hard on the problem.
[/defence]
[observation]
which, by the way, is a difficult problem that involves
1. gathering all the information that shows up on the net every day
2. only keeping the important stuff (no spam, no ads, no navigation elements etc)
3. indexing it and make if findable
4. analysing it using the structured data (eg number of inbound links) and unstructured (eg sentiment expressed)
4. presenting it nicely to all sorts of different types of user
basically, it’s a hard thing we are trying to do. I think the issue is that the vendors have set expectations unrealistically high too soon. I see things like ‘80% sentiment accuracy’ and ‘we find the most influential people on the web’ and cringe a bit. It makes it sound like a solved problem when it’s really a work in progress and will never really be solved as there is always an element of opinion involved. Who’s the most influential person in all these responses you have had for example?
[/observation]
[falling-on-sword]
Posted 13 Aug 2009 at 7:47 am ¶So, i think you’re being a bit harsh Asi, but i think that maybe you weren’t sold these systems in the best of faith and your expectations were out of line with the current state-of-the-art. And if we were guilty of that - I’m sorry.
[/falling-on-sword]
cheers
giles
Couldn’t agree more with the article. Can’t tell you how many sales demos we sat through that blatantly overpromised, so had abslutely no credibility. We ended up going with Biz360. So far, so good.
Posted 13 Aug 2009 at 6:21 pm ¶i think it’s brilliant that most of the responses to this article are from social media monitoring companies. at the very least we know they’re drinking their own kool-aid.
although i don’t work for any of these companies, i am very interested in social media monitoring. but, as your article correctly points out, it is still a developing technology. indeed, it is one that i almost hope is never perfected (where will humanity be when machines can truly “understand” us? the thought gives me terminator/matrix shivers
regardless, it is getting better every day and will continue to. i’m just grateful there is a lot of competition!
thanks for the insightful piece. i hope you’re day isn’t too packed with sifting through spammy data.
jeff
Posted 13 Aug 2009 at 7:14 pm ¶Asi,
I agree with some of these points but it seems to differ from each service we have tested. For now, we are happy with WebDig. The GUI is definitely a “work in progress” but the depth of quality data is impressive. Frankly, I forget the details of their pitch, but it was enough to show their advantages, and motivate my agency to sign on.
Overall, this idea of “active listening” while in early stages, I think will prove invaluable in the way that we engage with our clients’ customers.
Thanks for the post!
Randy
Posted 13 Aug 2009 at 7:56 pm ¶Interesting no holds barred post.
Even though we’re a Text Analytics company I can’t argue with many of you’re points. It’s never wise to leave decisions entirely to computers, but they are really good at repetitive chores and volumes of work, two things people aren’t very good at. The trick is to match the technology to problem.
PR metrics is a tough technical problem because people want the sentiment of every story to be perfect, and that’s just not possible for a machine, but the machines are better at spotting sentiment trends, which is why the technology is taking off in stock trading where you care about the gross direction of a company more than the tone for a particular story, and in these uses the users tell us that the aggregate tones are incredibly accurate.
It’s undeniable that a hybrid approach is your best bet if you can afford it, in cases where you can’t the question should be am I better off with something like a Scoutlabs, Sysomos or others.
Posted 13 Aug 2009 at 8:02 pm ¶I’d like to chat with you about Overtone. Imagine taking all of the data those existing solutions find, and run that through a sophisticated set of algorithms designed to read and decipher UGC for thematic subjects, topics, and sentiment. MUCH deeper dive, still real-time, and no traditional pay-per-report market research.
Posted 13 Aug 2009 at 9:59 pm ¶I started analyzing buzz/social media monitoring tools in October of 2007 while working at Jive Software. During that same year I wrote a post explaining how to roll your own dashboard using a mashup of Netvibes, Yahoo! Pipes, and PostRank. The market has evolved considerably since that time.
Clearly you just stumbled upon this market and have little historic context. There is much work to be done, but it doesn’t mean vendors weren’t right to release version 1.0 of their software. The first versions of everyone’s software aren’t perfect, but that doesn’t means businesses aren’t profiting greatly from using them in their initial, imperfect form. It’s called first movers advantage. All early tech is expensive. Tech always offers more and more for less as time goes by.
You don’t look smart by being rude in your observations that vendors haven’t perfected their software yet. That’s been obvious for years to those following the space. There are ways to graciously talk about your observations. Instead, you read like an armchair critic with no historic reference whining about how things aren’t perfect and free yet.
Just so you don’t write another upset post in the future as the market evolves, social media monitoring software will look more and more like the systems used by call centers. If you think the amount of person hours spent using the current systems is a lot now, just wait for what is coming.
Posted 14 Aug 2009 at 1:01 am ¶The best about this post is twofold; the article itself and the responses it has generated. I’ll have to go with Julie from Scoutlabs on this one, it actually is a good thing that you’re tearing us a new one! Asi, I honestly think your kind of feedback is indispensible for companies like us. We need to know where we’re failing for the people who feel they need our type of solutions, because what we do is so new and uncharted. I say keep berating us, keep telling us where we make mistakes. You’ll certainly have my ear
Thanks
Posted 14 Aug 2009 at 10:46 am ¶Taking the role of Devil’s Advocate, I see your unreliability, flawed analysis, time productivity, technology, and price; is there anything about any of the web monitoring tools you like? Or more to the point, if reliability is the most important factor in determining one over another, how can that be improved?
Perhaps Almitra commented best that the cited tools are only good for aggregation but analysis requires manual labor–which is a time sink and for a small firm, less an option.
To Justin Kistner, you commented, “If you think the amount of person hours spent using the current systems is a lot now, just wait for what is coming.”
Can you elaborate?
Posted 14 Aug 2009 at 1:30 pm ¶Asi,
First off, I would encourage you to contact me to talk about some of your insights and experiences. A lot of your views about what hasn’t been working with competing monitoring technologies have worked to inform our firms aggressive push towards offering precision brand and reputation listening and monitoring services.
I liked Almitri’s take, though the content discovery isn’t what is wrong with monitoring technologies. If anything, its the content discovery which has had the most time to refine and evolve. Where it falters is how each firm markets its approach to filtering out spam, malware, and the nuances of brand names which are used as everyday regular expression.
The other area of obvious exploit is the way each firm tackles this notion of attaching meaning to actionable insights and whether they actually successfully achieve the accelerated analysis portion. In other words, it might not particularly excite subscribers when their inboxes are pelted with 150 new mentions per hour for the keywords “Amazon” - and those results include the bookseller, the river and links to nefarious vendors of smut. Worse is when none of the captured links refer to the brand for which the monitoring was originally assigned to track.
Also, #5 was bang on and I applaud you for articulating this point in the manner you have. The floodgates to standardizing an influence barometer have busted wide open, and IMHO, the advances - being hailed as the next big thing - are leaving most underwhelmed.
We hear this a lot from folks that come to us looking for something better, and frankly, the real meaning and context of brand mentions (lets call this the circle in the square) is often overlooked, and may I add in a manner that rather hastily favours flawed methodology over delivering on promise.
Finally, kudos to Attentio and Scoutlabs for gracefully accepting these calls for improvement in the manner they have. Anything less would be the antithesis of this spaces reason for being.
Joseph
Posted 14 Aug 2009 at 2:10 pm ¶@RepuTrack
Asi,
These programs provide a way to visualize and bring sets of data together, in that, the software can be limiting. These programs can be very complex and overwhelming. Based on the complexity, there is a need to be “trained” on how to maximize the information gleaned from the reporting. Additionally, a savvy marketer must facilitate the data mining to make the information actionable, specific to marketing strategies and tactics.
Building search queries is not an easy task. Filtering out spam is extremely time-consuming. Today’s spammers are working twice as fast as the software companies themselves. They are intelligent (bots) and will continue to get better. It may be asking too much of them to be perfect. That being said, if these “software” companies go directly to the client, they are responsible for making it an easy experience. I don’t believe they have done that.
In selling their software, they (won’t name names) make promises they simply can’t keep. Obviously, this is very problematic. This action is turning the “social monitoring” solutions into a commodity. That is a lose-lose situation for all parties involved. The client should be fully aware that the software doesn’t work magic (they are told that it does).
These charts hold little value without extensive analysis. I highly doubt they will provide that analysis for you, because it takes so much time. And again, there needs to be a competent marcom person to create action based on the data.
Posted 14 Aug 2009 at 2:18 pm ¶Great observations, and it’s definitely important to challenge an industry and it’s big players for the sake of improvement!
I had reviewed a few tools on my blog a while back and one of the reasons I came to work at MediaMiser is it does things differently (in a way I like)
For starters we offer social media monitoring and analysis in addition to traditional media (our core biz) and work to correlate the two. I think with all the hype surrounding social media, many are putting it in a silo and losing touch with the rest of PR/marketing when it should be a concerted effort.
Also what I love is the value add of a human element–we have dedicated analysts who hand pick the data for clients, determine tone/sentiment and provide consulting, so that the client doesn’t have to spend hours doing it themselves.
Of course this affects the costs, but if it’s saving a client hours of time, then it’s worth the different imo.
Not trying to be overly promotional here, but sharing my experience, but I did end up with a job at MediaMiser because I absolutely love what it does, so sometimes I can’t help it!
My point is there are companies constantly searching for a better way to do things, and I would imagine some of the big players you mentioned are in fact looking to improve. It’s a new concept and uncharted territory so it may take a while to settle the waters.
Kelly Rusk
Posted 14 Aug 2009 at 2:49 pm ¶Manager of marketing & communities
MediaMiser
Asi, I wondering if “suck” is a bit severe *S*. Certainly these tools are in the early stages of development. But, there is a lot more to come, especially as we enter the Web 3.0 world.
One thing to keep in mind is that the power of all these tools is still with the user, not the provider of the technology. They are only tools. People still (and always will) need to interpret the results to suit their own purposes. These products only provide the information we need to do what we do better. Even in the more evolved world of Business Intelligence and Predictive Analytics, highly skilled people still interpret results and act on those interpretations.
The problem you are experiencing is likely that your providers are not clearly understanding your requirements to identify the specific value they contribute to addressing those requirements.
There is, nor ever will be, a “Silver Bullet” in this kind of market.
Posted 14 Aug 2009 at 3:06 pm ¶Thanking him for letting you know the issues? Please. If anything in this post was news to a vendor, avoid them like a plague as it’s clear they don’t understand the challenges for the market they are attempting to serve.
This guy is 2 days behind his deadline and he’s pointing fingers. He is inexperienced in the discipline and mismanaged his time. Analysis is hard. It used to be even harder! If Asi didn’t know that or the strengths and limitations of analysis tools, he shouldn’t be selling his services as an analyst. Many companies are successfully selling reporting services because they know what it takes to get the job done. They also know that the current tools have made their job much, much easier than it was even a year ago (at least the good tools). I’m sure to those just starting their analyst practice because they overheard there were tools that did the job for you *are* disappointed. But that’s not about vendors, that’s about lack of experience and frame of reference. Talk to the people that have been in the game for years, and they’ll tell you how glad they are that their job is getting easier from improvements in tech.
No, current tech hasn’t made reporting instant. Vendors shouldn’t pretend that algorithms will replace people and time. Technology can make the process more efficient, but don’t expect AI for another decade or more. Run from any vendor saying otherwise.
Posted 14 Aug 2009 at 3:35 pm ¶@Ari, absolutely. I’ll write a post on it today.
Posted 14 Aug 2009 at 4:40 pm ¶wooooow
Who could have thought that a late night rant will cause such a stir…!
right…first, thanks ever so much for all the people who took the time to leave a comment and debate - nice 1
again to avoid rambling please allow me to reply to everyone with few bullet points:
1. the major issue to understand is that the promise of self-service tool can be somewhat misleading - if you want to produce quality, meaningful report you will have to plan your human resource accordingly - whether you do it yourself or outsource. so not self service but fairly high-maintainance. As any experienced data analysts will tell you it’s 80% sorting the data and 20% analysis and the tools require A LOT of training and practice until you get things right - just be prepared.
2. the technology and tools are far from perfect so they don’t give you the full picture - this is both the data that they do provide you with, i.e. a lot of spam/irrelevant data and with what they don’t provide you with i.e. facebook data and other walled gardens the technology cannot penetrate yet it is where valuable conversations still take place there.
3. the major flaws from my experience is with sentiment analysis as well as regional / country filters. it’s just not good enough. some of the vendors will admit off record that sentiment reliability is only 70%.
4. when the strength of the tool is visualisation of quantitative data but the data is unreliable (unless sorted properly over long hours) it is effectively unrepresentable .
Having said that, and i’m sorry i didn’t emphasised it in my post the fact is that i am still using SM2 and Brandwatch is a testimony that i’m still getting A LOT OF valuable information and insights. I cannot do my job without these tools - just wished they were less of a pain
thanks again for all your insightful comments
Posted 14 Aug 2009 at 7:20 pm ¶A
Yes, these new tools are rather kludgey, but I recall first setting-up Crystal Reports and that monster’s learning curve, so we’ll see.
Posted 14 Aug 2009 at 10:46 pm ¶They don’t call it nuance for nothing. Hybrid solutions are here to stay, these are the same issues AI face(d)(s) and how deep has that dry hole been drilled?
Posted 15 Aug 2009 at 4:02 pm ¶Good to know that Brandwatch is useful to you! We’re always working on making it better, so all feedback is very much welcome.
Posted 17 Aug 2009 at 9:12 am ¶A very excellent summation. It has been our experience as well: the nightmare of working with clients named “bell” and “rogers”. The monitoring was fine as far as the technology, but the sentiment index was inversely proportional to what we coded using our own people. I suspect the reason Google hasn’t touched this yet is because they can’t find a sufficiently reliable solution that would benefit their brand. And the person who posted about SPSS and other data analysis software hasn’t worked (as I have) with both these programs and monitoring social media for clients.
Posted 17 Aug 2009 at 4:33 pm ¶Brilliant post. Really informative. And I feel your pain! Can I just ask one specific question - in point 2 you say this, “sometimes the most important conversations are blocked” and I just wondered what you mean? why are they blocked?
Posted 18 Aug 2009 at 9:10 am ¶Asi,
Great post! I have myself been working on trying to track/monitor social media using Omniture. I think everyone of your points you made is spot on!
-Rudi
Posted 18 Aug 2009 at 12:12 pm ¶Hilarious. Here is a guy that has fatigue written all over him, using his last bits of strength to finish this painful chapter called “The Wonder Tools of Social Media Monitoring”, putting an end to the misery for once and for all. And then there’s all these comments like “contact us” and “have you tried…” and “you should try…”. Please, have mercy on this man. Leave him alone. He’s got better things to do. Hasn’t he suffered enough? To redeem all of mankind, I feel obliged to add. Great analysis!
Posted 18 Aug 2009 at 7:46 pm ¶Hi Asi,
Was my previous comment sent to you?
-Kelly
Posted 20 Aug 2009 at 5:52 am ¶Interesting post, although I think you have to separate self-service solutions (relatively cheap but altogether over-simplistic as technology, your own words ’simply is not there’) from solutions that involve a lot of manual analysis done by the provider.
Posted 24 Aug 2009 at 7:20 pm ¶The latter are of course much more expensive but they provide somewhat meaningful data and analysis.
They are not 100% perfect yet but then, as some other guy wrote, do we really need 100% accuracy?
Hey dude,
Read Write Web are big fans of yours. So much so they copied you… check this:
http://www.readwriteweb.com/archives/sentiment_analysis_is_ramping_up_in_2009.php#more
Ben
Posted 25 Aug 2009 at 1:28 pm ¶Hi Asi,
I appreciate the frustration you experienced. I have also tried out most of these tools, and our experience is not nearly half as bad. We have managed quite well and do realize that the data that we get is very relevant. It may not be perfect but it is not a waste by any means.
Not seeing any easy way for the data clean up to happen, we have done the next best thing. We have our own team of Social Media monitoring specialists who we use to read through all posts, remove spam, to trend spotting, do sentiment spotting and also to classify the conversations as per predefined tags. This makes the data very actionable.
We do this for ourselves and also offer the service to others, as our Social Media monitoring effort. More about this at our website at http://www.socialwavelength.com.
Posted 28 Aug 2009 at 11:07 am ¶Social Media Monitoring vendors are full of shite.
It is that simple.
Posted 17 Oct 2009 at 4:23 pm ¶Try ASOMO, http://www.asomo.net. There is a human involved in the service, so sentiment analysis is accurate. Results are great and I don´t have any of the above problems you mentioned. Good luck!
Posted 19 Oct 2009 at 3:24 pm ¶Sorry to arrive at this post so late in the day. Music to my ears. The problem with all of the black box solutions is they are about monitoring places - not spaces. In my experience you can only do this with a bespoke, DIY panel approach. See http://richardstacy.com/2009/07/16/is-much-of-social-media-monitoring-snake-oil-or-have-i-missed-something/ and http://richardstacy.com/2009/09/04/what-social-media-monitoring-and-the-english-channel-have-in-common/
Posted 03 Nov 2009 at 4:26 pm ¶You Can Try
Social Master Tool 2.0 - Social Media Management and Analytics Tool
Posted 11 Nov 2009 at 6:55 pm ¶Hey Asi,
I can’t agree with you more. Spot on with your post.
Posted 20 Jan 2010 at 6:30 am ¶Check out beRelevant (http://www.berelevant.com).
Really well thought out and thought provoking post. As an industry we have a lot of work to do, it’s clear. First of all, sentiment measurement is no trivial task. Especially in a medium like Twitter, sometimes I can’t even tell what the sentiment is, and I am human. We (Biz360) advise clients that this data is directional, and should be only part of the brand analysis. We, as well as our competitors, are constantly looking for ways to improve sentiment analysis. We actually allow to score sentiment on the topic / object or the article level, which helps a bit with accuracy.
One thing that is of value, in my opinion, (and we offer), is topic discovery, that helps you figure out most relevant phrases that occur around the search topic. To help uncover what’s going on with your product or brand, you can take a look at such a tag cloud, take a look at a trendline, and sentiment trendline and dig in deeper to see where engagement efforts should be applied. We allow our users to sort by impact and reach, so that they can focus in on the most relevant articles and mentions.
If you would like to give Biz360 Community a whirl, please feel free to ping me.
Maria Ogneva
Posted 05 Feb 2010 at 6:13 am ¶Director of Social Media, Biz360
@themaria @biz360
Trackbacks & Pingbacks 17
[...] No Man’s Blog – The problem(s) with social media monitoring technologies [...]
[...] No Man’s Blog – The problem(s) with social media monitoring technologies [...]
Scaling Social Media…
Jeff Katz recently sent me a link to a post from a blogger complaining about the state of technology in social media monitoring. While I agree with many of the points in the post about the challenges of the industry, I was very put off by the insulting…
[...] problem(s) with social media monitoring technologies: No Man’s Blog via [...]
[...] outils de veille des médias sociaux, qu’il me semble intéressant de partager avec vous : No Man’s Blog. L’auteur reproche essentiellement aux outils qu’il a testés : La non pertinence : la [...]
[...] No Man’s Blog – The problem(s) with social media monitoring technologies A list of issues found with all social media analytics software. Are they worth what you pay? (tags: socialmedia monitoring analytics tools metrics analysis) [...]
[...] have been several recent write-ups about social media monitoring tools. The majority of the reviews surround flaws and [...]
[...] There are a number of challenges when it comes to social media monitoring solutions for business and you can find out some things to watch out for in Asi Sharabi’s blog post. [...]
Plannersphere Top 20/20 - September 2009…
The top 20 planning blogs in September 2009 and the top 20 planning blog posts from August 2009….
[...] of these, entititled “The problems with Social Media Monitoring technologies“, seems to have decent web findability as it was sent to me a couple times by friends and [...]
[...] This rant has gained me few haterz in the industry and an invite to speak in a panel at the monitoring social media conference. Whether client or agency side, PR, market research or marketing if listening and monitoring ‘the conversations’ is something you should know about I highly recommend you to come along. [...]
[...] the dying days of the summer, Asi Sharabi wrote a late night rant about the state of social media monitoring. He directed his anger squarely at the many social media monitoring services that have emerged in [...]
[...] against such monitoring tools was also a point of discussion with Asi Sharabi’s controversial blog post inspiring a lively panel debate. No solution (right now anyway) is perfect and the current [...]
[...] followed on, what was wrong with social media monitoring , including Asi Sharabi, who’s blog post on the same topic, inspired the conference. Some of the main issues discussed [...]
[...] The problem(s) with social media monitoring technologies [...]
[...] No Man’s Blog - The problem(s) with social media monitoring technologies - On social media monitoring and failing services in that field. [...]
[...] //No Man’s Blog [...]
Post a Comment