Print Page   |   Contact Us   |   Your Cart   |   Sign In   |   Register
Dana Gardner's BriefingsDirect for Connect.
Blog Home All Blogs
Longtime IT industry analyst Dana Gardner is a creative thought leader on enterprise software, SOA, cloud-based strategies, and IT architecture strategies. He is a prolific blogger, podcaster and Twitterer. Follow him at


Search all posts for:   


Top tags: Dana Gardner  Interarbor Solutions  BriefingsDirect  HP  cloud computing  big data  The Open Group  HPDiscover  SaaS  VMWare  virtualization  data analytics  HP DISCOVER  Ariba  data center  enterprise architecture  HP Vertica  SOA  Ariba Network  SAP  Open Group Conference  security  Ariba LIVE  VMWorld  Tony Baer  desktop virtualization  Jennifer LeClaire  mobile computing  Business Intelligence  Converged infrastructure 

Agile on fire: IT enters the new era of 'continuous' everything

Posted By Dana L Gardner, Thursday, November 19, 2015

The next BriefingsDirect DevOps thought leadership discussion explores the concept of continuous processes around the development and deployment of applications and systems. Put the word continuous in front of many things and we help define DevOps: continuous delivery, continuous testing, continuous assessment, and there is more.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy.

To help better understand the continuous nature of DevOps, we're joined by two guests, James Governor, Founder and Principal Analyst at RedMonk, and Ashish Kuthiala, Senior Director of Marketing and Strategy for Hewlett Packard Enterprise (HPE) DevOps. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: We hear a lot about feedback loops in DevOps between production and development, test and production. Why is the word "continuous" now cropping up so much? What do we need to do differently in IT in order to compress those feedback loops and make them impactful?

Kuthiala: Gone are the days where you would see the next version 2.0 coming out in six months and 2.1 coming out three months after that.


If you use some of the modern applications today, you never see Facebook 2.0 is coming out tomorrow or Google 3.1 is being released. They are continuously and always making improvements from the back-end onto the platforms of the users -- without the users even realizing that they're getting improvements, a better user experience, etc.

In order to achieve that, you have to continuously be building those new innovations into your product. And, of course, as soon as you change something you need to test it and roll it all the way into production.

In fact, we joke a lot about how if everything is continuous, why don’t we drop the word continuous and just call it planning, testing, or development, like we do today, and just say that you continuously do this. But we tend to keep using this word "continuous" before everything.

I think a lot of it is to drive home the point across the IT teams and organizations that you can no longer do this in chunks of three, six, or nine months -- but you always have to keep doing this.

Governor: How do you do the continuous assessment of your continuous marketing?

Continuous assessment

Kuthiala: We joke about the continuous marketing of everything. The continuous assessment term, despite my objections to the word continuous all the time, is a term that we've been talking about at HPE.

The idea here is that for most software development teams and production teams, when they start to collaborate well, take the user experience, the bugs, and what’s not working on the production end at the users’ hands -- where the software is being used -- and feed those bugs and the user experience back to the development teams.

When companies actually get to that stage, it’s a significant improvement. It’s not the support teams telling you that five users were screaming at us today about this feature or that feature. It’s the idea that you start to have this feedback directly from the users’ hands.

We should stretch this assessment piece a little further. Why assess the application or the software when it’s at the hands of the end users? The developer, the enterprise architects, and the planners design an application and they know best how it should function.

Whether it’s monitoring tools or it’s the health and availability of the application, start to shift left, as we call it. I'd like James to comment more about this, because he knows a lot about the development space. The developer knows his code best; let him experience what the user is starting to experience.

Governor: My favorite example of this is that, as an analyst, you're always looking for those nice metaphors and ways to talk about the world -- one notion of quality I was very taken with was when I was reading about the history if ship-building and the roles and responsibilities involved in building a ship.


One of the things they found was that if you have a team doing the riveting separate from doing the quality assurance (QA) on the riveting, the results are not as good. Someone will happily just go along -- rivet, rivet, rivet, rivet -- and not really care if they're doing a great job, because somebody else is going to have to worry about the quality.

As they moved forward with this, they realized that you needed to have the person doing the riveting also doing the QA. That’s a powerful notion of how things have changed.

Certainly the notion of shifting left and doing more testing earlier in the process, whether that be in terms of integration, load testing, whatever, all the testing needs to happen up front and it needs to be something that the developers are doing.

The new suite of tools we have makes it easier for developers to have better experiences around that, and we should take advantage.

Lean manufacturing

One of the other things about continuous is that we're making reference to manufacturing modes and models. Lean manufacturing is something that led to fewer defects, apart from one catastrophic example to the contrary. And we're looking at that and asking how we can learn from that.

So lean manufacturing ties into lean startups, which ties into lean and continuous assessment.

What’s interesting is that now we're beginning to see some interplay between the two and paying that forward. If you look at GM, they just announced a team explicitly looking at Twitter to find user complaints very, very early in the process, rather than waiting until you had 10,000 people that were affected before you did the recall.

Last year was the worst year ever for recalls in American car manufacturing, which is interesting, because if we have continuous improvement and everything, why did that happen? They're actually using social tooling to try to identify early, so that they can recall 100 cars or 1,000 cars, rather than 50,000.

It’s that monitoring really early in the process, testing early in the process, and most importantly, garnering user feedback early in the process. If GM can improve and we can improve, yes.

Gardner: I remember in the late '80s, when the Japanese car makers were really kicking the pants out of Detroit, that we started to hear a lot about simultaneous engineering. You wouldn’t just design something, but you designed for its manufacturability at the same time. So it’s a similar concept.

Solutions That Unify Development and Operations
To Accelerate Business Innovation
Get More Information

But going back to the software process, Ashish, we see a level of functionality in software that needs to be rigorous with security and performance, but we're also seeing more and more the need for that user experience for features and functions that we can’t even guess at, that we need to put into place in the field and see what happens.

How does an enterprise get to that point, where they can so rapidly do software that they're willing to take a chance and put something out to the users, perhaps a mobile app, and learn from its actual behavior? We can get the data, but we have to change our processes before we can utilize it. 

Kuthiala: Absolutely. Let me be a little provocative here, but I think it’s a well-known fact that the era of the three-year, forward-looking roadmaps is gone. It’s good to have a vision of where you're headed, but what feature, function and which month will you release so that the users will find it useful? I think that’s just gone, with this concept of the minimum viable product (MVP) that more startups take off with and try to build a product and fund themselves as they gain success.

It’s an approach even that bigger enterprises need to take. You don't know what the end users’ tastes are.

I change my taste on the applications I use and the user experience I get, the features and functionality. I'm always looking at different products, and I switch my mind quite often. But if I like something and they're always delivering the right user experience for me, I stick with them.

Capture the experience

The way for an enterprise to figure out what to build next is to capture this experience, whether it’s through social media channels or engineering your codes so that you can figure out what the user behavior actually is.

The days of business planners and developers sitting in cubicles and thinking this is the coolest thing I'm going to invent and roll out is not going to work anymore. You definitely need that for innovation, but you need to test that fairly quickly.

Also gone are the days of rolling back something when something doesn’t work. If something doesn’t work, if you can deliver software really quickly at the hands of end users, you just roll forward. You don’t roll back anymore.

It could be a feature that’s buggy. So go and fix it, because you can fix it in two days or two hours, versus the three- to six-month cycle. If you release a feature and you see that most users -- 80 percent of the users -- don’t even bother about it, turn it off, and introduce the new feature that you were thinking about.

This assessment from the development, testing, and production that you're always doing starts to benefit you. When you're standing up for that daily sprint and wondering what are the three features I'm going to work on as a team, whether it’s the two things that your CEO told you you have to absolutely do it, because "I think it’s the greatest thing since sliced bread," or it’s the developer saying, "I think we should build this feature," or some use case is coming out of the business analyst or enterprise architects.

We have wonderful new platforms that enable us to store a lot more data than we could before at a reasonable cost.

Now you have data. You have data across all these teams. You can start to make smarter decisions and you can choose what to build and not build. To me, that's the value of continuous assessment. You can invest your $100 for that day in the two things you want to do. None of us has unlimited budgets.

Gardner: For organizations that grok this, that say, "I want continuous delivery. I want continuous assessment," what do we need to put in place to actually execute on it to make it happen?

Governor: We've spoken a lot about cultural change, and that’s going to be important. One of the things, frankly, that is an underpinning, if we're talking about data and being data-driven, is just that we have wonderful new platforms that enable us to store a lot more data than we could before at a reasonable cost.

There were many business problems that were stymied by the fact that you would have to spend the GDP of a country in order to do the kind of processing that you wanted to, in order to truly understand how something was working. If we're going to model the experiences, if we are going to collect all this data, some of the thinking about what's infrastructure for that so that you can analyze the data is going to be super important. There's no point talking in being data-driven if you don’t have architecture for delivering on that.

Gardner: Ashish, how about loosely integrated capabilities across these domains, tests, build, requirements, configuration management, and deployment? It seems that HPE is really at the center of a number of these technologies. Is there a new layer or level of integration that can help accelerate this continuous assessment capability?

Rich portfolio

Kuthiala: You're right. We have a very rich portfolio across the entire software development cycle. You've heard about our Big Data Platform. What can it really do, if you think about it? James just referred to this. It’s cheaper and easier to store data with the new technologies, whether it’s structured, unstructured, video, social, etc., and you can start to make sense out of it when you put it all together.

There is a lot of rich data in the planning and testing process, and all the different lifecycles. A simple example is a technology that we've worked on internally, where when you start to deliver software faster and you change one line of code and you want this to go out. You really can’t afford to do the 20,000 tests that you think you need to do, because you're not sure what's going to happen.

We've actually had data scientists working internally in our labs, studying the patterns, looking at the data, and testing concepts such as intelligent testing. If I change this one line of code, even before I check it in, what parts of the code is it really affecting, what functionality? If you are doing this intelligently, does it affect all the regions of the world, the demographics? What feature function does it affect?

We've actually had data scientists working internally in our labs, studying the patterns, looking at the data, and testing concepts such as intelligent testing.

It's helping you narrow down whether will it break the code, whether it will actually affect certain features and functions of this software application that’s out there. It's narrowing it down and helping you say, "Okay, I only need to run these 50 tests and I don't need to go into these 10,000 tests, because I need to run through this test cycle fast and have the confidence that it will not break something else."

So it's a cultural thing, like James said, but the technologies are also helping make it easier.

Gardner: It’s interesting. We're borrowing concepts from other domains in the past as well -- just-in-time testing or fit-for-purpose testing, or lean testing?

Kuthiala: We were talking about Lean Functional Testing (LeanFT) at HP Discover. I won't talk about that here in terms of product, but the idea is exactly that. The idea is that the developer, like James said, knows his code well. He can test it well before and he doesn’t throw it over the wall and let the other team take a shot at it. It’s his responsibility. If he writes a line of code, he should be responsible for the quality of it.

Gardner: And it also seems that the integration across this continuum can really be the currency of analysis. When we have data and information made available, that's what binds these processes together, and we're starting to elevate and abstract that analysis up and it make it into a continuum, rather than a waterfall or a hand-off type of process.

Before we close out, any other words that we should put in front of continuous as we get closer to DevOps -- continuous security perhaps?

Security is important

Kuthiala: Security is a very important topic and James and I have talked about it a lot with some other thought leaders. Security is just like testing. Anything that you catch early on in the process is a lot easier and cheaper to fix than if you catch it in the hands of the end users, where now it’s deployed to tens and thousands of people.

It’s a cultural shift. The technology has always been there. There's a lot of technology within and outside of HP that you need to incorporate the security testing and the discipline right into the development and planning process and not leave it towards the end.

In terms of another continuous word, I mean I can come up with continuous Dana Gardner podcast.

Governor: There you go.

Gardner: Continuous discussions about DevOps.

One of the things that RedMonk is very interested in, and it's really our view in the world, is that, increasingly, developers are making the choices, and then we're going to find ways to support the choices they are making.

Governor: One of the things that RedMonk is very interested in, and it's really our view in the world, is that, increasingly, developers are making the choices, and then we're going to find ways to support the choices they are making.

It was very interesting to me that the term continuous integration began as a developer term, and then the next wave of that began to be called continuous deployment. That's quite scary for a lot of organizations. They say, "These developers are talking about continuous deployment. How is that going to work?"

The circle was squared when I had somebody come in and say what we're talking to customers about is continuous improvement, which of course is a term again that we saw in manufacturing and so on.

But the developer aesthetic is tremendously influential here, and this change has been driven by them. My favorite "continuous" is a great phrase, continuous partial attention, which is the world we all live in now.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy. Sponsor: Hewlett Packard Enterprise.

You may also be interested in:

Tags:  Ashish Kuthiala  BriefingsDirect  Dana Gardner  DevOps  Hewlett Packard Enterprise  Interarbor Solutions  James Governor  RedMonk 

Share |
PermalinkComments (0)

Big data enables top user experiences and extreme personalization for Intuit TurboTax

Posted By Dana L Gardner, Wednesday, November 18, 2015

The next BriefingsDirect big-data innovation case study highlights how Intuit uses deep-data analytics to gain a 360-degree view of its TurboTax application's users’ behavior and preferences. Such visibility allows for rapid applications improvements and enables the TurboTax user experience to be tailored to a highly detailed degree.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy.

Here to share how analytics paves the way to better understanding of end-user needs and wants, we're joined by Joel Minton, Director of Data Science and Engineering for TurboTax at Intuit in San Diego. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: Let’s start at a high-level, Joel, and understand what’s driving the need for greater analytics, greater understanding of your end-users. What is the big deal about big-data capabilities for your TurboTax applications?

Minton: There were several things, Dana. We were looking to see a full end-to-end view of our customers. We wanted to see what our customers were doing across our application and across all the various touch points that they have with us to make sure that we could fully understand where they were and how we can make their lives better.


We also wanted to be able to take that data and then give more personalized experiences, so we could understand where they were, how they were leveraging our offerings, but then also give them a much more personalized application that would allow them to get through the application even faster than they already could with TurboTax.

And last but not least, there was the explosion of available technologies to ingest, store, and gain insights that was not even possible two or three years ago. All of those things have made leaps and bounds over the last several years. We’ve been able to put all of these technologies together to garner those business benefits that I spoke about earlier.

Gardner: So many of our listeners might be aware of TurboTax, but it’s a very complex tax return preparation application that has a great deal of variability across regions, states, localities. That must be quite a daunting task to be able to make it granular and address all the variables in such a complex application.

Minton: Our goal is to remove all of that complexity for our users and for us to do all of that hard work behind the scenes. Data is absolutely central to our understanding that full end-to-end process, and leveraging our great knowledge of the tax code and other financial situations to make all of those hard things easier for our customers, and to do all of those things for our customers behind the scenes, so our customers do not have to worry about it.

Gardner: In the process of tax preparation, how do you actually get context within the process?

Always looking

Minton: We're always looking at all of those customer touch points, as I mentioned earlier. Those things all feed into where our customer is and what their state of mind might be as they are going through the application.

To give you an example, as a customer goes though our application, they may ask us a question about a certain tax situation.

When they ask that question, we know a lot more later on down the line about whether that specific issue is causing them grief. If we can bring all of those data sets together so that we know that they asked the question three screens back, and then they're spending a more time on a later screen, we can try to make that experience better, especially in the context of those specific questions that they have.

HP Vertica Community Edition
Start Your Free Trial Now
Gain Access to All Features and Functionality

As I said earlier, it's all about bringing all the data together and making sure that we leverage that when we're making the application as easy as we can.

Gardner: And that's what you mean by a 360-degree view of the user: where they are in time, where they are in a process, where they are in their particular individual tax requirements?

Minton: And all the touch points that they have with not only things on our website, but also things across the Internet and also with our customer-care employees and all the other touch points that we use try to solve our customers’ needs.

During our peak times of the year during tax season, we have billions and billions of transactions.

Gardner: This might be a difficult question, but how much data are we talking about? Obviously you're in sort of a peak-use scenario where many people are in a tax-preparation mode in the weeks and months leading up to April 15 in the United States. How much data and how rapidly is that coming into you?

Minton: We have a tremendous amount of data. I'm not going to go into the specifics of the complete size of our database because it is proprietary, but during our peak times of the year during tax season, we have billions and billions of transactions.

We have all of those touch points being logged in real-time, and we basically have all of that data flowing through to our applications that we then use to get insights and to be able to help our customers even more than we could before. So we're talking about billions of events over a small number of days.

Gardner: So clearly for those of us that define big data by velocity, by volume, and by variety, you certainly meet the criteria and then some.

Unique challenges

Minton: The challenges are unique for TurboTax because we're such a peaky business. We have two peaks that drive a majority of our experiences: the first peak when people get their W-2s and they're looking to get their refunds, and then tax day on April 15th. At both of those times, we're ingesting a tremendous amount of data and trying to get insights as quickly as we can so we can help our customers as quickly as we can.

Gardner: Let’s go back to this concept of user experience improvement process. It's not just something for tax preparation applications but really in retail, healthcare, and many other aspects where the user expectations are getting higher and higher. People expect more. They expect anticipation of their needs and then delivery of that.

This is probably only going to increase over time, Joel. Tell me a little but about how you're solving this issue of getting to know your user and then being able to be responsive to an entire user experience and perception.

Minton: Every customer is unique, Dana. We have millions of customers who have slightly different needs based on their unique situations. What we do is try to give them a unique experience that closely matches their background and preferences, and we try to use all of that information that we have to create a streamlined interaction where they can feel like the experience itself is tailored for them.

So the most important thing is taking all of that data and then providing super-personalized experience based on the experience we see for that user and for other users like them.

It’s very easy to say, “We can’t personalize the product because there are so many touch points and there are so many different variables.” But we can, in fact, make the product much more simplified and easy to use for each one of those customers. Data is a huge part of that.

Specifically, our customers, at times, may be having problems in the product, finding the right place to enter a certain tax situation. They get stuck and don't know what to enter. When they get in those situations, they will frequently ask us for help and they will ask how they do a certain task. We can then build code and algorithms to handle all those situations proactively and be able to solve that for our customers in the future as well.

So the most important thing is taking all of that data and then providing super-personalized experience based on the experience we see for that user and for other users like them.

Gardner: In a sense, you're a poster child for a lot of elements of what you're dealing with, but really on a significant scale above the norm, the peaky nature, around tax preparation. You desire to be highly personalized down to the granular level for each user, the vast amount of data and velocity of that data.

What were some of your chief requirements at your architecture level to be able to accommodate some of this? Tell us a little bit, Joel, about the journey you’ve been on to improve that architecture over the past couple of years?

Lot of detail

Minton: There's a lot of detail behind the scenes here, and I'll start by saying it's not an easy journey. It’s a journey that you have to be on for a long time and you really have to understand where you want to place your investment to make sure that you can do this well.

One area where we've invested in heavily is our big-data infrastructure, being able to ingest all of the data in order to be able to track it all. We've also invested a lot in being able to get insights out of the data, using Hewlett Packard Enterprise (HPE) Vertica as our big data platform and being able to query that data in close to real time as possible to actually get those insights. I see those as the meat and potatoes that you have to have in order to be successful in this area.

On top of that, you then need to have an infrastructure that allows you to build personalization on the fly. You need to be able to make decisions in real time for the customers and you need to be able to do that in a very streamlined way where you can continuously improve.

We use a lot of tactics using machine learning and other predictive models to build that personalization on-the-fly as people are going through the application. That is some of our secret sauce and I will not go into in more detail, but that’s what we're doing at a high level.

Gardner: It might be off the track of our discussion a bit, but being able to glean information through analytics and then create a feedback loop into development can be very challenging for a lot of organizations. Is DevOps a cultural parallel path along with your data-science architecture?

HP Vertica Community Edition
Start Your Free Trial Now
Gain Access to All Features and Functionality

I don’t want to go down the development path too much, but it sounds like you're already there in terms of understanding the importance of applying big-data analytics to the compression of the cycle between development and production.

Minton: There are two different aspects there, Dana. Number one is making sure that we understand the traffic patterns of our customer and making sure that, from an operations perspective, we have the understanding of how our users are traversing our application to make sure that we are able to serve them and that their performance is just amazing every single time they come to our website. That’s number one.

Number two, and I believe more important, is the need to actually put the data in the hands of all of our employees across the board. We need to be able to tell our employees the areas where users are getting stuck in our application. This is high-level information. This isn't anybody's financial information at all, but just a high-level, quick stream of data saying that these people went through this application and got stuck on this specific area of the product.

We want to be able to put that type of information in our developer’s hands so as the developer is actually building a part of the product, she could say that I am seeing that these types of users get stuck at this part of the product. How can I actually improve the experience as I am developing it to take all of that data into account?

We have an analyst team that does great work around doing the analytics, but in addition to that, we want to be able to give that data to the product managers and to the developers as well, so they can improve the application as they are building it. To me, a 360-degree view of the customer is number one. Number two is getting that data out to as broad of an audience as possible to make sure that they can act on it so they can help our customers.

Major areas

Gardner: Joel, I speak with HPE Vertica users quite often and there are two major areas that I hear them talk rather highly of the product. First, has to do with the ability to assimilate, so that dealing with the variety issue would bring data into an environment where it can be used for analytics. Then, there are some performance issues around doing queries, amid great complexity of many parameters and its speed and scale.

HP Vertica Community Edition
Start Your Free Trial Now
Gain Access to All Features and Functionality

Your applications for TurboTax are across a variety or platforms. There is a shrink-wrap product from the legacy perspective. Then you're more along the mobile lines, as well as web and SaaS. So is Vertica something that you're using to help bring the data from a variety of different application environments together and/or across different networks or environments?

Minton: I don't see different devices that someone might use as a different solution in the customer journey. To me, every device that somebody uses is a touch point into Intuit and into TurboTax. We need to make sure that all of those touch points have the same level of understanding, the same level of tracking, and the same ability to help our customers.

Whether somebody is using TurboTax on their computer or they're using TurboTax on their mobile device, we need to be able to track all of those things as first-class citizens in the ecosystem. We have a fully-functional mobile application that’s just amazing on the phone, if you haven’t used it. It's just a great experience for our customers.

From all those devices, we bring all of that data back to our big data platform. All of that data can then be queried, because you want to understand, many questions, such as when do users flow across different devices and what is the experience that they're getting on each device? When are they able to just snap a picture of their W-2 and be able to import it really quickly on their phone and then jump right back into their computer and finish their taxes with great ease?

You need to be able to have a system that can handle that concurrency and can handle the performance that’s going to be required by that many more people doing queries against the system.

We need to be able to have that level of tracking across all of those devices. The key there, from a technology perspective, is creating APIs that are generic across all of those devices, and then allowing those APIs to feed all of that data back to our massive infrastructure in the back-end so we can get those insights through reporting and other methods as well.

Gardner: We've talked quite a bit about what's working for you: a database column store, the ability to get a volume variety and velocity managed in your massive data environment. But what didn't work? Where were you before and what needed to change in order for you to accommodate your ongoing requirements in your architecture?

Minton: Previously we were using a different data platform, and it was good for getting insights for a small number of users. We had an analyst team of 8 to 10 people, and they were able to do reports and get insights as a small group.

But when you talk about moving to what we just discussed, a huge view of the customer end-to-end, hundreds of users accessing the data, you need to be able to have a system that can handle that concurrency and can handle the performance that’s going to be required by that many more people doing queries against the system.

Concurrency problems

So we moved away from our previous vendor that had some concurrency problems and we moved to HPE Vertica, because it does handle concurrency much better, handles workload management much better, and it allows us to pull all this data.

The other thing that we've done is that we have expanded our use of Tableau, which is a great platform for pulling data out of Vertica and then being able to use those extracts in multiple front-end reports that can serve our business needs as well.

So in terms of using technology to be able to get data into the hands of hundreds of users, we use a multi-pronged approach that allows us to disseminate that information to all of these employees as quickly as possible and to do it at scale, which we were not able to do before.

There's always going to be more data that you want to track than you have hardware or software licenses to support.

Gardner: Of course, getting all your performance requirements met is super important, but also in any business environment, we need to be concerned about costs.

Is there anything about the way that you were able to deploy Vertica, perhaps using commodity hardware, perhaps a different approach to storage, that allowed you to both accomplish your requirements, goals in performance, and capabilities, but also at a price point that may have been even better than your previous approach?

Minton: From a price perspective, we've been able to really make the numbers work and get great insights for the level of investment that we've made.

How do we handle just the massive cost of the data? That's a huge challenge that every company is going to have in this space, because there's always going to be more data that you want to track than you have hardware or software licenses to support.

So we've been very aggressive in looking at each and every piece of data that we want to ingest. We want to make sure that we ingest it at the right granularity.

Vertica is a high-performance system, but you don't need absolutely every detail that you’ve ever had from a logging mechanism for every customer in that platform. We do a lot of detail information in Vertica, but we're also really smart about what we move into there from a storage perspective and what we keep outside in our Hadoop cluster.

Hadoop cluster

We have a Hadoop cluster that stores all of our data and we consider that our data lake that basically takes all of our customer interactions top to bottom at the granular detail level.

We then take data out of there and move things over to Vertica, in both an aggregate as well as a detail form, where it makes sense. We've been able to spend the right amount of money for each of our solutions to be able to get the insights we need, but to not overwhelm both the licensing cost and the hardware cost on our Vertica cluster.

The combination of those things has really allowed us to be successful to match the business benefit with the investment level for both Hadoop and with Vertica.

Gardner: Measuring success, as you have been talking about quantitatively at the platform level, is important, but there's also a qualitative benefit that needs to be examined and even measured when you're talking about things like process improvements, eliminating bottlenecks in user experience, or eliminating anomalies for certain types of individual personalized activities, a bit more quantitative than qualitative.

We're actually performing much better and we're able to delight our internal customers to make sure that they're getting the answers they need as quickly as possible.

Do you have any insight, either anecdotal or examples, where being able to apply this data analytics architecture and capability has delivered some positive benefits, some value to your business?

Minton: We basically use data to try to measure ourselves as much as possible. So we do have qualitative, but we also have quantitative.

Just to give you a few examples, our total aggregate number of insights that we've been able to garner from the new system versus the old system is a 271 percent increase. We're able to run a lot more queries and get a lot more insights out of the platform now than we ever could on the old system. We have also had a 41 percent decrease in query time. So employees who were previously pulling data and waiting twice as long had a really frustrating experience.

Now, we're actually performing much better and we're able to delight our internal customers to make sure that they're getting the answers they need as quickly as possible.

We've also increased the size of our data mart in general by 400 percent. We've massively grown the platform while decreasing performance. So all of those quantitative numbers are just a great story about the success that we have had.

From a qualitative perspective, I've talked to a lot of our analysts and I've talked to a lot of our employees, and they've all said that the solution that we have now is head and shoulders over what we had previously. Mostly that’s because during those peak times, when we're running a lot of traffic through our systems, it’s very easy for all the users to hit the platform at the same time, instead of nobody getting any work done because of the concurrency issues.

Better tracking

Because we have much better tracking of that now with Vertica and our new platform, we're actually able to handle that concurrency and get the highest priority workloads out quickly, allow them to happen, and then be able to follow along with the lower-priority workloads and be able to run them all in parallel.

The key is being able to run, especially at those peak loads, and be able to get a lot more insights than we were ever able to get last year.

HP Vertica Community Edition
Start Your Free Trial Now
Gain Access to All Features and Functionality

Gardner: And that peak load issue is so prominent for you. Another quick aside, are you using cloud or hybrid cloud to support any of these workloads, given the peak nature of this, rather than keep all that infrastructure running 365, 24×7? Is that something that you've been doing, or is that something you're considering?

Minton: Sure. On a lot of our data warehousing solutions, we do use cloud in points for our systems. A lot of our large-scale serving activities, as well as our large scale ingestion, does leverage cloud technologies.

We don't have it for our core data warehouse. We want to make that we have all of that data in-house in our own data centers, but we do ingest a lot of the data just as pass-throughs in the cloud, just to allow us to have more of that peak scalability that we wouldn’t have otherwise.

The faster than we can get data into our systems, the faster we're going to be able to report on that data and be able to get insights that are going to be able to help our customers.

Gardner: We're coming up toward the end of our discussion time. Let’s look at what comes next, Joel, in terms of where you can take this. You mentioned some really impressive qualitative and quantitative returns and improvements. We can always expect more data, more need for feedback loops, and a higher level of user expectation and experience. Where would you like to go next? How do you go to an extreme focus even more on this issue of personalization?

Minton: There are a few things that we're doing. We built the infrastructure that we need to really be able to knock it out of the park over the next couple of years. Some of the things that are just the next level of innovation for us are going to be, number one, increasing our use of personalization and making it much easier for our customers to get what they need when they need it.

So doubling down on that and increasing the number of use cases where our data scientists are actually building models that serve our customers throughout the entire experience is going to be one huge area of focus.

Another big area of focus is getting the data even more real time. As I discussed earlier, Dana, we're a very peaky business and the faster than we can get data into our systems, the faster we're going to be able to report on that data and be able to get insights that are going to be able to help our customers.

Our goal is to have even more real-time streams of that data and be able to get that data in so we can get insights from it and act on it as quickly as possible.

The other side is just continuing to invest in our multi-platform approach to allow the customer to do their taxes and to manage their finances on whatever platform they are on, so that it continues to be mobile, web, TVs, or whatever device they might use. We need to make sure that we can serve those data needs and give the users the ability to get great personalized experiences no matter what platform they are on. Those are some of the big areas where we're going to be focused over the coming years.


Gardner: Now you've had some 20/20 hindsight into moving from one data environment to another, which I suppose is equivalent of keeping the airplane flying and changing the wings at the same time. Do you have any words of wisdom for those who might be having concurrency issues or scale, velocity, variety type issues with their big data, when it comes to moving from one architecture platform to another? Any recommendations you can make to help them perhaps in ways that you didn't necessarily get the benefit of?

Minton: To start, focus on the real business needs and competitive advantage that your business is trying to build and invest in data to enable those things. It’s very easy to say you're going to replace your entire data platform and build everything soup to nuts all in one year, but I have seen those types of projects be tried and fail over and over again. I find that you put the platform in place at a high-level and you look for a few key business-use cases where you can actually leverage that platform to gain real business benefit.

When you're able to do that two, three, or four times on a smaller scale, then it makes it a lot easier to make that bigger investment to revamp the whole platform top to bottom. My number one suggestion is start small and focus on the business capabilities.

Number two, be really smart about where your biggest pain points are. Don’t try to solve world hunger when it comes to data. If you're having a concurrency issue, look at the platform you're using. Is there a way in my current platform to solve these without going big?

Frequently, what I find in data is that it’s not always the platform's fault that things are not performing. It could be the way that things are implemented and so it could be a software problem as opposed to a hardware or a platform problem.

HP Vertica Community Edition
Start Your Free Trial Now
Gain Access to All Features and Functionality

So again, I would have folks focus on the real problem and the different methods that you could use to actually solve those problems. It’s kind of making sure that you're solving the right problem with the right technology and not just assuming that your platform is the problem. That’s on the hardware front.

As I mentioned earlier, looking at the business use cases and making sure that you're solving those first is the other big area of focus I would have.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy. Sponsor: Hewlett Packard Enterprise.

You may also be interested in:

Tags:  big data  BriefingsDirect  Dana Gardner  data analysis  Hewlett Packard Enterprise  HP DISCOVER  HP Vertica  Interarbor Solutions  Intuit  Joel Minton  Tableau  TurboTax 

Share |
PermalinkComments (0)

Spirent Leverages Big Data to Keep User Experience Quality a Winning Factor for Telcos

Posted By Dana L Gardner, Tuesday, November 17, 2015

Transcript of a discussion on the use of big data to provide improved user experiences for telecommunications operators' customers.

Listen to the podcast. Find it on iTunes. Get the mobile app. Download the transcript. Sponsor: Hewlett Packard Enterprise.

Dana Gardner: Hello, and welcome to the next edition of the HP Discover Podcast Series. I'm Dana Gardner, Principal Analyst at Interarbor Solutions, your host and moderator for this ongoing discussion on IT innovation and how it’s making an impact on people’s lives.


Our next big-data case study discussion explores the ways that Spirent Communications advances the use of big data to provide improved user experiences for telecommunications operators.

We'll learn how advanced analytics that draws on multiple data sources provide Spirent’s telco customers’ rapid insights into their networks and operations.  That insight, combined with analysis of user actions and behaviors, provides a "total picture" approach to telco services and uses that both improves the actual services proactively -- and also boosts the ability to better support help desks.

Spirent’s insights thereby help operators in highly competitive markets reduce the spend on support, reduce user churn, and better adhere to service-level agreements (SLAs), while providing significant productivity gains.

HP Big Data Analytics Engines
Meet Complex Enterprise-scale OEM Requirements
Get More Information

To hear how Spirent uses big data to make major positive impacts on telco operations, we're joined by Tom Russo, Director of Product Management and Marketing at Spirent Communications in Matawan, New Jersey. Welcome, Tom.

Tom Russo: Hi, Dana. Thanks for having me.

Gardner: User experience quality enhancement is essential, especially when we're talking about consumers that can easily change carriers. Controlling that experience is more challenging for an organization like a telco. They have so many variables across networks. So at a high-level, tell me how Spirent masters complexity using big data to help telcos maintain the best user experience.

Russo: Believe it or not, historically, operators haven't actually managed their customers as much as they've managed their networks. Even within the networks, they've done this in a fairly siloed fashion.


There would be radio performance teams that would look at whether the different cell towers were operating properly, giving good coverage and signal strength to the subscribers. As you might imagine, they wouldn't talk to the core network people, who would make sure that people can get IP addresses and properly transmit packets back and forth. They had their own tools and systems, which were separate, yet again, from the services people, who would look at the different applications. You can see where it’s going.

There were also customer-care people, who had their own tools and systems that didn’t leverage any of that network data. It was very inefficient, and not wrapped around the customer or the customer experience.

New demands

They sort of got by with those systems when the networks weren't running too hot. When competition wasn't too fierce, they could get away with that. But these days, with their peers offering better quality of service, over-the-top threats, increasing complexity on the network in terms of devices, and application services, it really doesn't work any more.

It takes too long to troubleshoot real customer problems. They spend too much time chasing down blind alleys in terms of solving problems that don't really affect the customer experience, etc. They need to take a more customer-centric approach. As you’d imagine that’s where we come in. We integrate data across those different silos in the context of subscribers.

We collect data across those different silos -- the radio performance, the core network performance, the provisioning, the billing etc. -- and fuse it together in the context of subscribers. Then, we help the operator identify proactively where that customer experience is suffering, what we call hotspots, so that they can act before the customers call and complain, which is expensive from a customer-care perspective and before they churn, which is very expensive in terms of customer replacement. It's a more customer-centric approach to managing the network.

Automate Data Collection and Analysis
In Support of Business Objectives
With Spirent InTouch Analytics

Gardner: So your customer experience management does what your customers had a difficult time doing internally. But one aspect of this is pulling together disparate data from different sources, so that you can get the proactive inference and insights. What did you do better around data acquisition?

We integrate data across those different silos in the context of subscribers.

Russo: The first key step is being able to integrate with a variety of these different systems. Each of the groups had their different tools, different data formats, different vendors.

Our solution has a very strong what we call extract, transform, load (ETL), or data mediation capability, to pull all these different data sources together, map them into a uniform model of the telecom network and the subscriber experience.

This allows us to see the connections between the subscriber experience, the underlying network performance and even things like outcomes -- whether people churn, whether they provide negative survey responses, whether they've called and complained to  customer care, etc.

Then, with that holistic model, we can build high-level metrics like quality of experience scores, predictive models, etc. to look across those different silos, help the operators see where the hot spots of customer dissatisfaction is, where people are going to eventually churn, or where other costs are going to be incurred.

Gardner: Before we go more deeply into this data issue, tell me a bit more about Spirent. Is the customer experience division the only part? Tell me about the larger company, just so we have a sense of the breadth and depths of what you offer.

World leader

Russo: Most people, at least in telecom, know Spirent as a lab vendor. Spirent is one of the world leaders in the markets for simulating, emulating, and testing devices, network elements, applications, and services, as they go from the development phase to the launch phase in their lifecycle. Most of their products focus on that, the lab testing or the launch testing, making sure that devices are, as we call it, "fit for launch."

Spirent has historically had less of a presence in the live network domain. In the last year or two, they’ve made a number of strategic acquisitions in that space. They’ve made a number of internal investments to leverage the capabilities and knowledge base that they have from the lab side into the live network.

One of those investments, for example, was an acquisition back in early 2014 of DAX Technologies, a leading customer experience management vendor. That acquisition, plus some additional internal investments has led to the growth of our Customer Experience Management (CEM) Business Unit.

Gardner: Tom, tell me some typical use cases where your customers are using Spirent in the field. Who are those that are interacting with the software? What is it that they're doing with it? What are some of the typical ways in which it’s bringing value there?

Russo: Basically, we have two user bases that leverage our analytics. One is the customer-care groups. What they're trying to do is obtain, very quickly, a 360-degree view of the experience that a subscriber is seeing -- who is calling in and complaining about their service and the root causes of problems that they might be having with their services.

HP Big Data Analytics Engines
Meet Complex Enterprise-scale OEM Requirements
Get More Information

If you think about the historic operation, this was a very time-intensive, costly operation, because they would have to swivel chair, as we call it, between a variety of different systems and tools trying to figure out whether I had a network-related issue, a provisioning issue, a billing issue, or something else. These all could potentially take hours, even hundreds of hours, to resolve.

With our system, the customer-care groups have one single pane of glass, one screen, to see all aspects of the customer experience to very quickly identify the root causes of issues that they are having and resolve them. So it keeps customers happier and reduces the cost of the customer-care operation.

The second group that we serve is on the engineering side. We're trying to help them identify hotspots of customer dissatisfaction on the network, whether that be in terms of devices, applications, services, or network elements so that they can prioritize their resources around those hotspots, as opposed to noisy, traditional engineering alarms. The idea here is that this allows them to have maximal impact on the customer experience with minimal costs and minimal resources.

Gardner: You recently rolled out some new and interesting services and solutions. Tell us a little but about that.

Russo: We’ve rolled out the latest iteration of our InTouch solution, our flagship product. It’s called InTouch Customer and Network Analytics (CNA) and it really addresses feedback that we've received from customers in terms of what they want in an analytic solution.

We're hearing that they want to be more proactive and predictive. Don’t just tell me what's going on right now, what’s gone on historically, how things have trended, but help me understand what’s going to happen moving forward, where our customer is going to complain. Where is the network going to experience performance problems in the future. That's an increasing area of focus for us and something that we've embedded to a great degree in the InTouch CNA product.

More flexibility

Another thing that they've told us is that they want to have more flexibility and control on the visualization and reporting side. Don't just give me a stock set of dashboards and reports and have me rely on you to modify those over time. I have my own data scientists, my own engineers, who want to explore the data themselves.

We've embedded Tableau business intelligence (BI) technology into our product to give them maximum flexibility in terms of report authorship and publication. We really like the combination of Tableau and Hewlett Packard Enterprise (HPE) Vertica because it allows them to be able to do those ad-hoc reports and then also get good performance through the Vertica database.

And another thing that we are doing more and more is what we call Closed Loop Analytics. It's not just identifying an issue or a customer problem on the network, but it's also being able to trigger an action. We have an integration and partnership with another business unit in Spirent called Mobilethink that can change device settings for example.

If we see a device is mis-provisioned, we can send alert to Mobilethink, and they can re-provision the device to correct something like a mis-provisioned access point name (APN) and resolve the problem. Then, we can use our system to confirm indeed that the fix was made and that the experience has improved.

We're trying to tie it all together, everything from the subscriber transactions and experience to the underlying network performance, again to the outcome type information.

Gardner: It’s clear to me, Tom, how we can get great benefits from doing this properly and how the value escalates the more data and the more information you get, and the better you can serve those customers. Let's drill down a bit into how you can make this happen. As far as data goes, are we talking about 10 different data types, 50? Given the stream and the amount of data that comes off of a network, what size data we are talking about and how do you get a handle on that?

Russo: In our largest deployment, we're talking about a couple of dozen different data sources and a total volume of data on the order of 50 to 100 billion transactions a day. So, it’s large volume, especially on the transactional side, and high variety. In terms of what we're talking about, it’s a lot of machine data. As I mentioned before, there is the radio performance, core network performance, and service performance type of information.

We also look at things like whether you're provisioning correctly for the services that you're trying to interact with. We look at your trouble ticket history to try and correlate things like network performance and customer care activity. We will look at survey data, net promoter score (NPS) type information, billing churn, and related information.

We're trying to tie it all together, everything from the subscriber transactions and experience to the underlying network performance, again to the outcome type information -- what was the impact of the experience on your behavior?

Gardner: What specifically is your history with HPE Vertica? Has this been something that's been in place for some time? Did you switch to it from something else? How did that work out?

Finishing migration

Russo: Right now, we're finishing the migration to HP Vertica technology, and it will be embedded in our InTouch CNA solution. There are a couple of things that we like about Vertica. One is the price-performance aspects. The columnar lookups, the projections, give us very strong query response performance, but it's also able to run on commodity hardware, which gives us price advantage that's also bolstered by the columnar compression.

So price performance-wise and maturity-wise we like it. It’s a field-proven, tested solution. There are some other features in terms of strong Hadoop integration that we like. A lot of carriers will have their own Hadoop clusters, data oceans, etc. that they want us to integrate with. Vertica makes that fairly straightforward, and we like a lot of the embedded analytics as well, the Distributed R capability for predictive analytics and things along those lines.

Gardner: It occurs to me that the effort that you put into this at Spirent and being able to take vast amounts of data across a complex network and then come out with these analytic benefits could be extended to any number of environments. Is there a parallel between what you are doing with mobile and telco carriers that could extend to maybe networks that are managing the Internet of Things (IoT) types of devices?

We definitely see our solution helping operators who are trying to be IoT platform providers to ensure the performances of those IoT services and the SLAs that they have for them.

Russo: Absolutely. We're working with carriers on IoT already. The requirements that these things have in terms of the performance that they need to operate properly are different than that of human beings, but nevertheless, the underlying transactions that have to take place, the ability to get a radio connection and set up an IP address and communicate data back and forth to one another and do it in a robust reliable way, is still critical.

We definitely see our solution helping operators who are trying to be IoT platform providers to ensure the performances of those IoT services and the SLAs that they have for them. We also see a potential use for our technology going a step further into the vertical IoT applications themselves in doing, for example, predictive analytics on sensor data itself. That could be a future direction for us.

Gardner: Any words of wisdom for folks that are starting to do with large data volumes across wide variety of sources and are looking also for that more real-time analytics benefit? Any lessons learned that you could share from where Spirent has been and gone for others that are going to be facing some of these same big data issues?

Automate Data Collection and Analysis
In Support of Business Objectives
With Spirent InTouch Analytics

Russo: It's important to focus on the end-user value and the use cases as opposed to the technology. So, we never really focus on getting data for the sake of getting data. We focus more on what problem a customer is trying to accomplish and how we can most simply and elegantly solve it. That steered us clear from jumping on the latest and greatest technology bandwagons, instead going with the proven technologies and leveraging our subject-matter expertise.

Gardner: I'm afraid we'll have to leave it there. We've been exploring the ways that Spirent Communications advances the use of big data to provide improved user experiences for their telecommunications operator’s customers. We've identified some advanced analytics and how they're drawing on more data sources and providing their telco customers more rapid insights into their networks and operations.

So join me in thanking Tom Russo, Director of Product Management and Marketing at Spirent Communications in Matawan, New Jersey. Thanks so much.

Russo: Thanks very much, Dana. Thanks for having me.

Gardner: And a big thank you to our audience as well for joining us for this big data information innovation case study discussion.

I'm Dana Gardner; Principal Analyst at Interarbor Solutions, your host for this ongoing series of HPE-sponsored discussions. Thanks again for listening, and come back next time.

Listen to the podcast. Find it on iTunes. Get the mobile app. Download the transcript. Sponsor: Hewlett Packard Enterprise.

Transcript of a discussion on the use of big data to provide improved user experiences for telecommunications operators' customers. Copyright Interarbor Solutions, LLC, 2005-2015. All rights reserved.

You may also be interested in:

Tags:  big data  BriefingsDirect  Dana Gardner  data analytics  Hewlett Packard Enterprise  Interarbor Solutions  Spirent Communications  Tom Russo 

Share |
PermalinkComments (0)

Powerful reporting from YP's data warehouse helps SMBs deliver the best ad campaigns

Posted By Dana L Gardner, Thursday, November 12, 2015

The next BriefingsDirect big-data innovation case study highlights how Yellow Pages (YP) has developed a massive enterprise data warehouse with near real-time reporting capabilities that pulls oceans of data and information from across new and legacy sources.

We explore how YP then continuously delivers precise metrics to over half a million paying advertisers -- many of them SMBs and increasingly through mobile interfaces -- to best analyze and optimize their marketing and ad campaigns.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy.

To learn more, BriefingsDirect recently sat down with Bill Theisinger, Vice President of Engineering for Platform Data Services at YP in Glendale, California. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: Tell us about YP, the digital arm of what people would have known as Yellow Pages a number of years ago. You're all about helping small businesses become better acquainted with their customers, and vice versa.

Hewlett Packard Enterprise
Vertica Community Edition

 Start Your Free Trial Now

Theisinger: YP is a leading local marketing solutions provider in the U.S., dedicated to helping local businesses and communities grow. We help connect local businesses with consumers wherever they are and whatever device they are on, desktop and mobile.


Gardner: As we know, the world has changed dramatically around marketing and advertising and connecting buyers and sellers. So in the digital age, being precise, being aware, being visible is everything, and that means data. Tell us about your data requirements in this new world.

Theisinger: We need to be able to capture how consumers interact with our customers, and that includes where they interact -- whether it’s a mobile device or web device -- and also within our network of partners. We reach about 100 million consumers across the U.S and we do that through both our YP network and our partner network.

Gardner: Tell us too about the evolution. Obviously, you don’t build out data capabilities and infrastructure overnight. Some things are in place, and you move on, you learn, adapt, and you have new requirements. Tell us your data warehouse journey.

Needed to evolve

Theisinger: Yellow Pages saw the shift of their print business moving heavily online and becoming heavily digital. We needed to evolve with that, of course. In doing so, we needed to build infrastructure around the systems that we were using to support the businesses we were helping to grow.

And in doing that, we started to take a look at what the systems requirements were for us to be able to report and message value to our advertisers. That included understanding where consumers were looking, what we were impressing to them, what businesses we were showing them when they searched, what they were clicking on, and, ultimately what businesses they called. We track all of those different metrics.

When we started this adventure, we didn't have the technology and the capabilities to be able to do those things. So we had to reinvent our infrastructure. That’s what we did

Gardner: And as we know, getting more information to your advertisers to help them in their selection and spending expertise is key. It differentiates companies. So this is a core proposition for you. This is at the heart of your business.

Given the mission criticality, what are the requirements? What did you need to do to get that reporting, that warehouse capability?

Theisinger: We need to be able to scale to the size of our network and the size of our partner network, which means no click left behind, if you will, no impression untold, no search unrecognized. That's billions of events we process every day. We needed to look at something that would help us scale. If we added a new partner, if we expanded the YP network, if we added hundreds, thousands, tens of thousands of new advertisers, we needed the infrastructure to able to help us do that.

We need to be able to scale to the size of our network and the size of our partner network, which means no click left behind, if you will, no impression untold, no search unrecognized.

Gardner: I understand that you've been using Hadoop. You might be looking at other technologies as they emerge. Tell us about your Hadoop experience and how that relates to your reporting capabilities.

Theisinger: When I joined YP, Hadoop was a heavy buzz product in the industry. It was a proven product for helping businesses process large amounts of unstructured data. However, it still poses a problem. That unstructured data needs to be structured at some point, and it’s that structure that you report to advertisers and report internally.

That's how we decided that we needed to marry two different technologies -- one that will allow us to scale a large unstructured processing environment like Hadoop and one that will allow us to scale a large structured environment like Hewlett Packard Enterprise (HPE) Vertica.

Business impact

Gardner: How has this impacted your business, now that you've been able to do this and it's been in the works for quite a while? Any metrics of success or anecdotes that can relate back to how the people in your organization are consuming those metrics and then extending that as service and product back into your market? What has been the result?

Theisinger: We have roughly 10,000 jobs that we run every day, both to process data and also for analytics. That data represents about five to six petabytes of data that we've been able to capture about consumers, their behaviors, and activities. So we process that data within our Hadoop environment. We then pass that along into HPE Vertica, structure it in a way that we can have analysts, product owners, and other systems retrieve it, pull and look at those metrics, and be able to report on them to the advertisers.

Hewlett Packard Enterprise
Vertica Community Edition

 Start Your Free Trial Now

Gardner: Is there an automation to this as you look to present a more and better analytics on top of the Vertica? What are you doing to make that customizable to people based on their needs, but at the same time, controlled and managed so that it doesn't become unwieldy?

Theisinger: There is a lot of interaction between customers, both internal and external, when we decide how and what we’re going to present in terms of data, and there are a lot of ways we do that. We present data externally through an advertiser portal. So we want to make sure we work very closely with human factors and ergonomics (HFE) and the use experience (UX) designers as well as our advertisers, through focus groups, workshops, and understanding what they want to understand about the data that we present them.

Then, internally, we decide what would make sense and how we feel comfortable being able to present it to them, because we have a universe of a lot more data than what we probably want to show people.

We also do the same thing internally. We've been able to provide various teams internally whether its sales, marketing, or finance, insights into who's clicking on various business listings, who's viewing various businesses, who’s calling businesses, what their segmentation is, and what their demographics look like and it allows us a lot of analytical insight. We do most of that work through the analytics platforms, which is, in this case, HPE Vertica.

Small businesses need to be able to just pick up their mobile device and look at the effectiveness of their campaigns with YP.

Gardner: Now, that user experience is becoming more and more important. It wasn't that long ago when these reports were going to people who were data scientists or equivalent, but now we're taking the amount to those 600,000 small businesses. Can you tell us a little bit about lessons learned when it comes to delivering an end analytics product, versus building out the warehouse? They seem to be interdependent but we're seeing more and more emphasis on that user experience these days.

Theisinger: You need to bridge the gap between analytics and just data storage and processing. So you have to present them in-state. This is what happens. It’s very descriptive of what's going on, and we try to be a little bit more predictive when it comes to the way we want to do analysis at YP. We're looking to go beyond just descriptive analytics.

What has also changed is the platform by which you present the data. It's going highly mobile. Small businesses need to be able to just pick up their mobile device and look at the effectiveness of their campaigns with YP. They're able to do that through a mobile platform we’ve built called YP for Merchants.

They can log in and see their metrics that are core to their business and how those campaigns are performing. They can even see some details, like if they missed a phone call and they want to be able to reach back out to a consumer and see if they need to help, solve a problem, or provide a service.

Developer perspective

Gardner: And given that your developers had to go through the steps of creating that great user experience and taking it to the mobile tier, was there anything about HPE Vertica, your warehouse, or your approach to analytics that made that development process easier? Is there an approach to delivering this from a developer perspective that you think others might learn from?

Hewlett Packard Enterprise
Vertica Community Edition

 Start Your Free Trial Now

Theisinger: There is, and it takes a lot more people than just the analytics team in my group or the engineers in my team. It’s a lot of other teams within YP that build this. But first and foremost, people want to see the data as real time and as near real time as they can.

When a small business relies on contact from customers, we track those calls. When a potential customer calls a small business and that small business isn’t able to actually get to the call or respond to that customer because maybe they are on a job, it's important to know that that call happened recently. It's important for that small business to reach back out to the consumer, because that consumer could go somewhere else and get that service from a competitor.

To be able to do that as quickly as possible is a hard-and-fast requirement. So processing the data as quickly as you can and presenting that, whether it be on a mobile device, in this case, as quickly as you can is definitely paramount to making that a success.

Having the right infrastructure puts you in the position to be able to do that. That’s where businesses are going to end up growing, whether it's ours or small businesses.

Gardner: I've spoken to a number of people over the years and one of the takeaways I get is that infrastructure is destiny. It really seems to be the case in your business that having that core infrastructure decision process done correctly has now given you the opportunity to scale up, be innovative, and react to the market. I think it’s also telling that, in this data-driven decade that we’ve been in for a few years now, the whole small business sector of the economy is a huge part of our overall productivity and growth as an economy.

Any thoughts, generally about making infrastructure decisions for the long run, decisions you won't regret, decisions that that can scale over time and are future proof?

Theisinger: Yeah, for speaking about what I've seen through the job that we’ve had it here at YP, we reach over half a million paying advertisers. The shift is happening between just telling the advertisers what's happened to helping them actually drive new business.

So it's around the fact that I know who my customers are now, how do I find more of them, or how do I reach out to them, how do I market to them? That's where the real shift is. You have to have a really strong scalable and extensible platform to be able to answer that question. Having the right infrastructure puts you in the position to be able to do that. That’s where businesses are going to end up growing, whether it's ours or small businesses.

And our success is hinged to whether or not we can get these small businesses to grow. So we are definitely 100 percent focused on trying to make that happen.

Gardner: It’s also telling that you’ve been able to adjust so rapidly. Obviously, your business has been around for a long time. People are very familiar with the Yellow Pages, the actual physical product, but you've gone to make software so core to your value and your differentiation. I'm impressed and I commend you on being able to make that transitions fairly rapidly.

Core talent

Theisinger: Yeah, well thank you. We’ve invested a lot in the people within the technology team we have there in Glendale. We've built our own internal search capabilities, our own internal products. We’ve pulled a lot of good core talent from other companies.

I used to work at Yahoo with other folks, and YP is definitely focused on trying to make this transition a successful one, but we have our eye on our heritage. Over a hundred years of being very successful in the print business is not something you want to turn your back on. You want to be able to embrace that, and we’ve learned a lot from it, too.

So we're right there with small businesses. We have a very large sales force, which is also very powerful and helpful in making this transition a success. We've leaned on all of that and we become one big kind of happy family, if you will. We all worked very closely together to make this transition successful.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy. Sponsor: Hewlett Packard Enterprise.

You may also be interested in:

Tags:  big data  Bill Theisinger  BriefingsDirect  Dana Gardner  data analysis  Hewlett Packard Enterprise  HPE  HPE Discover  Interarbor Solutions  Vertica  Yellow Pages  YP 

Share |
PermalinkComments (0)

IoT brings on development demands that DevOps manages best, say experts

Posted By Dana L Gardner, Monday, November 09, 2015

The next BriefingsDirect DevOps thought leadership discussion explores how continuous processes around the development and deployment of applications are both impacted by -- and a benefit to -- the Internet of Things (IoT) trend.

Listen to the podcast. Find it on iTunesGet the mobile app. Read a full transcript or download a copy. Watch for Free: DevOps, Catalyst of the Agile Enterprise.

To help better understand the relationship between DevOps and a plethora of new end-devices and data please welcome Gary Gruver, consultant, author and a former IT executive who has led many large-scale IT transformation projects, and John Jeremiah, Technology Evangelist at Hewlett Packard Enterprise (HPE), on Twitter at @j_jeremiah. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: Let’s talk about how the DevOps trend extends not to just traditional enterprise IT and software applications, but to a much larger set of applications -- those in the embedded space, mobile, and end-devices of all sorts. Gary, why is DevOps even more important when you have so many different moving parts as we expect with the IoT?

Gruver: In software development, everybody needs to be more productive. Software is no longer just on websites and in IT departments. It’s going on everywhere in the industry. It’s gone to every product in every place, and being able to differentiate your product with software is becoming more and more important to everybody.

Gardner: John, from your perspective, is there a sense that DevOps is more impactful, more powerful when we apply it to IoT?


Jeremiah: The reality is it that IoT is moving as fast as mobile is -- and even faster. If you don’t have the ability to change your software to evolve -- to iterate as there is new business innovation -- you're not going to be able to keep up to be competitive. So IoT is going to require a DevOps approach in order to be successful.

Gardner: In the past, we've had a separate development organization and approach to embedded devices. Do we need to still to do that, or can we combine traditional enterprise software with DevOps and apply the same systems architecture and technologies to all sorts of development?

Learn how DevOps solutions unify development and operations
To accelerate business innovation

Gruver: The principles of being able to keep your code base more "releasable," to work under a prioritized backlog, to work through the process of adding automated testing, and frequent feedback to the developers so that they get better at it -- this all applies.


Therefore, for embedded systems you are going to need to develop simulators and emulators for automated testing. A simulator is a representation of the final product that can be run on a server. As much as possible, you want to be able to create a simulator that represents the software characteristics of the final product. You can then use this and trust it to find defects, because the amount of automated testing you are going to need to be running to transform your businesses is huge. If you don’t have an affordable place like a server farm to run that, it just doesn’t work. [Watch for Free: DevOps, Catalyst of the Agile Enterprise.]

If you have custom ASICs in the product, you're also going to need to create an emulator to test the low-level firmware interacting with the ASIC. This is similar to the simulator, but also includes the custom ASIC and electronics from the final product. I see way too many organizations that are embedded and are trying to transform their process giving up on using simulators and emulators because they're not finding the defects that they want to. Yet they haven’t invested in making them robust so they can be effective.

One of first things I talk about to people that have embedded systems is that you’re not going to be successful transforming your business until you create simulators and emulators that you can trust as a test environment to find defects.

Gardner: How about working as developers and testers with more of an operations mentality?

Gruver: At HPE and HP, we were running 15,000 hours of testing on the code base every day. When it was manual, we couldn’t do that and we really couldn’t transform our business until we fundamentally put that level of automated testing in place.

For laser printer testing, there's no way we would have been able to have enough paper to run that many hours of testing, and we would have worn out printers. There weren’t enough trees in Idaho to make enough paper to do that testing on the final product. Therefore, we needed to create a test farm of simulators and emulators to drive testing upstream as much as possible to get rapid feedback to our developers.

Gardner: Tell us how DevOps helped in the firmware project for HP printers, and how that illustrates where DevOps and embedded development come together?

No new features

Gruver: I had an opportunity to take over leading the LaserJet FW for our organization several years ago. It had been the bottleneck for the organization for two decades. We couldn’t add a new product or plans without checking the firmware, and we had given up asking for new features.

Then, when 2008 hit, and we were forced to cut our spending, as a of lot of people out in the industry at that time. We could no longer invest to spend our way out of problems. So we had to engineer our solution.

Discover how to use big data platforms
To unlock value of Internet of Things

We were fundamentally looking for anything that we could do to improve productivity. We went on a journey of what I would call applying Agile and DevOps principles at scale, as opposed to trying to scale small teams in the organization. We went through this process of continually trying to improve with a group of 400-800 engineers and working through that process. At the end of three years, firmware was no longer the bottleneck.

We had gone from five percent of our capacity going to innovation to 40 percent and we were supporting 1.5 times more products. So we took something that was a bottleneck for the business, completely unleashed that capability, and fundamentally transformed the business.

IoT is going to move so fast that nobody knows exactly what they need and what the capabilities are.

The details are captured in my first book, A Practical Approach to Large-Scale Agile Development. It’s available at all your finest bookstores. [Also see Gary's newest book, Leading the Transformation: Applying Agile and DevOps Principles at Scale.]

Gardner: And how does this provide a harbinger of things to come? What you’ve done with firmware at HP and Laser Printers several years ago, how does that paint a picture of how DevOps can be powerful and beneficial in the larger IoT environment?

Gruver: Well, IoT is going to move so fast that nobody knows exactly what they need and what the capabilities are. It's the ability to move fast. At HP and HPE, we went 2-3 times faster than we ever thought possible. What you're seeing in DevOps is that the unicorns of the world are showing that software development can go much faster than anybody ever thought was possible before.

That’s going to be much more important as you're trying to understand how this market evolves, what capabilities customers want, and where they want them in IoT. The companies that can move fast and respond to the feedback from the customers are going to be the ones that win. [Watch for Free: DevOps, Catalyst of the Agile Enterprise.]

Gardner: John, we've seen sort of a dip in the complexity around mobile devices in particular when people consolidated around iOS and Android after having hit many targets, at least for a software platform, in the past. That may have given people a sense of solace or complacency that they can develop mobile applications rapidly.

But we are now getting, to Gary's point, to a place where we don't really know what sort of endpoints we're going to be dealing with. We're looking at automated cars, houses, drones, appliances, and even sensors within our own bodies.

What are some of the core principles we need to keep in mind to allow for the rapid and continuous development processes for IoT to improve, but without stumbling again as we hit complexity when it comes to new targets?
New technologies

Jeremiah: One of the first things that you're going to have to do is embrace service virtualization and strategies in order to quickly virtualize new technologies and to be able to quickly simulate those technologies when they come to life. We don't know exactly what they're going to be, but we have to be able to embrace that and to bring that into our process and methodology.

And as Gary was talking about earlier, the strategies of going fast that apply in firmware, apply in the enterprise as well about building automated testing, failing as fast as you can, and learning as you go. As we see complexity increase, the real key is going to be able to harness that, and use virtualization as strategy to move that forward.

Gardner: Any other metrics of success? How do we know we're succeeding with DevOps? We talked about speed. We talked about testing early and often. How do you know you're doing this well? For organizations that want to have a good way to demonstrate success, how do they present that?

Gruver: I wouldn't just start off by trying to do DevOps. If you're going to transform your software development processes, the only reason you would go through that much turmoil is because your current development processes aren't meeting the needs of your business. Start off with how your current development processes aren't meeting your business needs.

The executives are in a best position to clarify exactly this gap and get the organization going down a continuous improvement process to improve the development and delivery processes.

Most organizations will quickly find that DevOps has some key tools in the toolbox that they want to start using immediately to start take some inefficiencies out of the development process.

Most organizations will quickly find that DevOps has some key tools in the toolbox that they want to start using immediately to start take some inefficiencies out of the development process.

But don't go off to do DevOps and measure how well you did it. We're all business executives. We run businesses, we manage businesses, and we need to focus on what the business is trying to achieve and just use the tools that will best help that.

Gardner: Where do we go next? DevOps has become a fairly popular concept now. It's getting a lot of attention. People understand that it can have a very positive impact, but getting it in place isn't always easy. There are a lot of different spinning variables -- culture, organization, management. In an enterprise that's looking to expand in the internet of things, perhaps they're not doing that level of development and deployment.

They probably have been a bit more focused on enterprise applications, rather than devices and embedded. How do you start up that capability and do it well within a software development organization? Let's look at moving from traditional development to the IoT development. What should we be keeping in mind?

Learn how DevOps solutions unify development and operations
To accelerate business innovation

Gruver: There are two approaches. One is, if you have loosely coupled architectures like most unicorns do, then you can empower the teams, add some operational members, and let figure it out. Most large enterprise organizations have more tightly coupled architectures that require large numbers of people working together to develop and deliver things together. I don't think those transformations are going to be effective until you find inspired executives who are willing to lead the transformation and work through the process.

Successful transformations

I've led a couple of successful transformations. If you look at examples from the DevOps Enterprise Summit that Gene Kim led, the common thing that you saw in most of those is that the organizations that were making progress had an executive that was leading the charge, rallying the troops, and making that happen. It requires coordinating work across a large number of teams, and you need somebody who can look across the value chain and muster the resources to make the technical and the cultural changes. [Read a recent interview with Kim on DevOps and security.]

Where a lot of my passion lies now, and the reason I wrote my second book is, that I don't think there are a lot of resources for the executives to learn how to transform large organizations. So I tried to capture everything that I knew about how best to do that.

My second book, Leading the Transformation: Applying Agile and DevOps Principles at Scale, is a resource that enables people to go faster in the organization. I think that’s the next key launch point -- getting the executives engaged to lead that change. That’s going to be the key to getting the adoption going much better. [Watch for Free: DevOps, Catalyst of the Agile Enterprise.]

Gardner: John, what about skills? It’s one thing to get the top-down buy-in, and it’s one thing to recognize the need for transformation and put in some of the organizational building blocks. But ultimately you need to be have the right people with the right skills.

Any thoughts about how IoT will create demand for a certain set of skills and how well we're in a position to train and find those people?

Jeremiah: IoT requires people to embrace skills and understand much broader than their narrow silo. They'll need to develop an expertise in what they do, but they have to have the relationships. They have to have the ability to work across the organization to learn. One of the skills is constantly learning as they go. As Gary mentioned earlier, it’s not a "done" for DevOps. It’s a journey of learning. It’s a journey of growing and getting better.

Then, as they apply their skills, they're focusing on how they deliver business value. That’s really the change.

Skills such as understanding process and understanding how things are working so you can continuously improve them is a skill that a lot of times people don’t bring to the table. They know their piece, but they don’t often think about the bigger picture. So it’s a set of skills. It’s beyond a single technology. It's understanding that that they are really not in IT -- they're really a part of the business. I love the way Gary said that earlier, and I agree with him. Seeing themselves as part of the business is a different mindset that they have to have as they go to work.

Then, as they apply their skills, they're focusing on how they deliver business value. That’s really the change. [Watch for Free: DevOps, Catalyst of the Agile Enterprise.]

Gardner: How do you do DevOps effectively when you're outsourcing a good part of your development? You may need to do that to find the skills.

For embedded systems, for example, you might look to an outside shop that has special experience in that particular area, but you may still want to get DevOps. How does that work?

Gruver: I think DevOps is key to making outsourcing work, especially if you have different vendors that you're outsourcing to because it forces coordination of the work on a frequent basis. Continuous integration, automated testing, and continuous deployment are the forcing functions that align the organization with working code across the system.

When you're enabling people to go off and work on separate branches and separate issues and you have an integration cycle late in the process, that’s where you get the dysfunction -- with a bunch of different organizations coming together with stuff that doesn’t work. If you force that to happen on a daily, or multiple times a day, basis, you get that system aligned and working well before people spend time and energy working on something that either don’t work together or won’t work well in production.

Listen to the podcast. Find it on iTunesGet the mobile app. Read a full transcript or download a copy. Watch for Free: DevOps, Catalyst of the Agile Enterprise. Sponsor: Hewlett Packard Enterprise.

You may also be interested in:

Tags:  BriefingsDirect  Dana Gardner  DevOps  Gary Gruver  HP Enterprise  Interarbor Solutions  Internet of things  John Jeremiah 

Share |
PermalinkComments (0)

Big data generates new insights into what’s happening in the world's tropical ecosystems

Posted By Dana L Gardner, Tuesday, November 03, 2015

The next BriefingsDirect big-data innovation case study interview explores how large-scale monitoring of rainforest biodiversity and climate has been enabled and accelerated by cutting-edge big-data capture, retrieval, and analysis.

We'll learn how quantitative analysis and modeling are generating new insights into what’s happening in tropical ecosystems worldwide, and we'll hear how such insights are leading to better ways to attain and verify sustainable development and preservation methods and techniques.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy.

To learn more about data science -- and how hosting that data science in the cloud -- helps the study of biodiversity, we're pleased to welcome Eric Fegraus, Senior Director of Technology of the TEAM Network at Conservation International and Jorge Ahumada, Executive Director of the TEAM Network, also at Conservation International in Arlington, Virginia. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: Knowing what’s going on in environments in the tropics helps us understand what to do and what not to do to preserve them. How has that changed? We spoke about a year ago, Eric. Are there any trends or driving influences that have made this data gathering more important than ever.

Fegraus: Over this last year, we’ve been able to roll out our analytic systems across the TEAM Network. We're having more-and-more uptake with our protected-area managers using the system and we have some good examples where the results are being used.


For example, in Uganda, we noticed that a particular cat species was trending downward. The folks there were really curious why this was happening. At first, they were excited that there was this cat species, which was previously not known to be there.

This particular forest is a gorilla reserve, and one of the main economic drivers around the reserve is ecotourism, people paying to go see the gorillas. Once they saw that these cats are going down, they started asking what could be impacting this. Our system told them that the way they were bringing in the eco-tourists to see the gorillas had shifted and that was potentially having an impact of where the cats were. It allowed them to readjust and think about their practices to bring in the tourists to the gorillas.

Information at work

Gardner: Information at work.

Fegraus: Information at work at the protected-area level.

Gardner: Just to be clear for our audience, the TEAM Network stands for the Tropical Ecology Assessment and Monitoring. Jorge, tell us a little bit about how that came about, the TEAM Network and what it encompasses worldwide?

No-Compromise Big Data Analytics
With HP Vertica OnDemand
Request Your 30-Day Free Trial

Ahumada: The TEAM Network was a program that started about 12 years ago and it was started to fill a void in the information we have from tropical forests. Tropical forests cover a little bit less than 10 percent of the terrestrial area in the world, but they have more than 50 percent of the biodiversity.


So they're the critical places to be conserved from that point of view, despite the fact we didn’t have any information about what's happening in these places. That’s how the TEAM Network was born, and the model was to use data collection methods that were standardized, that were replicated across a number of sites, and have systems that would store and analyze that data and make it useful. That was the main motivation.

Gardner: Of course, it’s super-important to be able to collect and retrieve and put that data into a place where it can be analyzed. It’s also, of course, important then to be able to share that analysis. Eric, tell us what's been happening lately that has led to the ability for all of those parts of a data lifecycle to really come to fruition?

Fegraus: Earlier this year, we completed our end-to-end system. We're able to take the data from the field, from the camera traps, from the climate stations, and bring it into our central repository. We then push the data into Vertica, which is used for the analytics. Then, we developed a really nice front-end dashboard that shows the results of species populations in all the protected areas where we work.

The analytical process also starts to identify what could be impacting the trends that we're seeing at a per-species level. This dashboard also lets the user look at the data in a lot of different ways. They can aggregate it and they can slice and dice it in different ways to look at different trends.

Gardner: Jorge, what sort of technologies are they using for that slicing and dicing? Are you seeing certain tools like Distributed R or visualization software and business-intelligence (BI) packages? What's the common thread or is it varied greatly?

Ahumada: It depends on the analysis, but we're really at the forefront of analytics in terms of big data. As Michael Stonebraker and other big data thinkers have said, the big-data analytics infrastructure has concentrated on the storage of big data, but not so much on the analytics. We break that mold because we're doing very, very sophisticated Bayesian analytics with this data.

One of the problems of working with camera-trap data is that you have to separate the detection process from the actual trend that you're seeing because you do have a detection process that has error.

Hierarchical models

We do that with hierarchical models, and it's a fairly complicated model. Just using that kind of model, a normal computer will take days and months. With the power of Vertica and power of processing, we’ve been able to shrink that to a few hours. We can run 500 or 600 species from 13 sites, all over the world in five hours. So it’s a really good way to use the power of processing.

We’d been also more recently working with Distributed R, a new package that was written by HP folks at Vertica, to analyze satellite images, because we're also interested in what’s happening at these sites in terms of forest loss. Satellite images are really complicated, because you have millions of pixels and you don’t really know what each pixel is. Is it forest, agricultural land, or a house? So running that on normal R, it's kind of a problem.

No-Compromise Big Data Analytics
With HP Vertica OnDemand
Request Your 30-Day Free Trial

Distributed R is a package that actually takes some of those functions, like random forest and regression trees, and takes full power of the vertical processing of Vertica. So we’ve seen a 10-fold increase in performance with that, and it allows us to get much more information out of those images.

Gardner: Not only are you on the cutting-edge for the analytics, you've also moved to the bleeding edge on infrastructure and distribution mechanisms. Eric, tell us a little bit about your use of cloud and hybrid cloud?

Fegraus: To back up a little bit, we ended up building a system that uses Vertica. It’s an on-premise solution and that's what we're using in the TEAM Network. We've since realized that this solution we built for the TEAM Network can also be readily scalable to other organizations and government agencies, etc., different people that want to manage camera trap data, they want to do the analytics.

So now, we're at a process where we’ve been essentially doing software development and producing software that’s scalable. If an organization wants to replicate what we’re doing, we have a solution that we can spin up in the cloud that has all of the data management, the analytics, the data transformations and processing, the collection, and all the data quality controls, all built into a software instance that could be spun up in the cloud.

In many of these countries, it's very difficult for some of those governments to expand out their old solutions on the ground. Cloud solutions offer a very good, effective way to manage data.

Gardner: And when you say “in the cloud,” are you talking about a specific public cloud, in a specific country or all the above, some of the above?

Fegraus: All of the above. We'll be using Vertica or we're using Vertica OnDemand. We're actually going to transition our existing on-premise solution into Vertica OnDemand. The solution we’re developing uses mostly open-source software and it can be replicated in the Amazon cloud or other clouds that have the right environments where we can get things up and running.

Gardner: Jorge, how important is that to have that global choice for cloud deployment and attract users and also keep your cost limited?

Ahumada: It’s really key, because in many of these countries, it's very difficult for some of those governments to expand out their old solutions on the ground. Cloud solutions offer a very good, effective way to manage data. As Eric was saying, the big limitation here is which cloud solutions are available in each country. Right now, we have something with cloud OnDemand here, but in some of the countries, we might not have the same infrastructure. So we'll have to contract different vendors or whatever.

But it's a way to keep cost down, deliver the information really quick, and store the data in a way that is safe and secure.

What's next?

Gardner: Eric, now that we have this ability to retrieve, gather, analyze, and now distribute, what comes next in terms of having these organizations work together? Do we have any indicators of what the results might be in the field? How can we measure the effectiveness at the endpoint -- that is to say, in these environments based on what you have been able to accomplish technically?

Fegraus: One of the nice things about the software that we built that can run in the various cloud environments, is that it can also be connected. For example, if we start putting these solutions in a particular continent, and there are countries that are doing this next to each other, there are not going to be silos that will be unable to share an aggregated level of data across each other so that we can get a holistic picture of what's happening.

So that was very important when we started going down this process, because one of the big inhibitors for growth within the environmental sciences is that there are these traditional silos of data that people in organizations keep and sit on and essentially don't share. That was a very important driver for us as we were going down this path of building software.

Gardner: Jorge, what comes next in terms of technology. Are the scale issues something you need to hurdle to get across? Are there analytics issues? What's the next requirements phase that you would like to work through technically to make this even more impactful?

Ahumada: As we scale up in size and  start  having more granularity in the countries where we work, the challenge is going to be keeping these systems responsive and information coming. Right now, one of the big limitations is the analytics. We do have analytics running at top speeds, but once we started talking about countries, we're going to have an the order of many more species and many more protected areas to monitor.

This is something that the industry is starting to move forward on in terms of incorporating more of the power of the hardware into the analytics, rather than just the storage and the management of data.

This is something that the industry is starting to move forward on in terms of incorporating more of the power of the hardware into the analytics, rather than just the storage and the management of data. We're looking forward to keep working with our technology partners, and in particular HP, to help them guide this process. As a case study, we're very well-positioned for that, because we already have that challenge.

Gardner: Also it appears to me that you are a harbinger, a bellwether, for the Internet of Things (IoT). Much of your data is coming from monitoring, sensors, devices, and cameras. It's in the form of images and raw data. Any thoughts about what others who are thinking about the impact of the IoT should consider, now that you have been there?

Fegraus: When we talk about big data, we're talking about data collected from phones, cars, and human devices. Humans are delivering the data. But here we have a different problem. We're talking about nature delivering the data and we don't have that infrastructure in places like Uganda, Zimbabwe, or Brazil.

No-Compromise Big Data Analytics
With HP Vertica OnDemand
Request Your 30-Day Free Trial

So we have to start by building that infrastructure and we have the camera traps as an example of that. We need to be able to deploy much more, much larger-scale infrastructure to collect data and diversify the sensors that we currently have, so that we can gather sound data, image data, temperature, and environmental data in a much larger scale.

Satellites can only take us some part of the way, because we're always going to have problems with resolution. So it's really deployment on the ground which is going to be a big limitation, and it's a big field that is developing now.

Gardner: Drones?

Fegraus: Drones, for example, have that capacity, especially small drones that are showing to be intelligent, to be able to collect a lot of information autonomously. This is at the cutting edge right now of technological development, and we're excited about it.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy. Sponsor: Hewlett Packard Enterprise.

You may also be interested in:

Tags:  big data  BriefingsDirect  cloud computing  Conservation International  Dana Gardner  data analysis  Eric Fegraus  Hewlett Packard Enterprise  HP  HP Vertica  Interarbor Solutions  Jorge Ahumada 

Share |
PermalinkComments (0)

DevOps and security, a match made in heaven

Posted By Dana L Gardner, Thursday, October 29, 2015

This next BriefingsDirect DevOps thought leadership discussion explores the impact of improved development on security and how those investing in DevOps models specifically can expect to improve their security, compliance, and risk-mitigation outcomes.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy.

To help better understand the relationship between DevOps and security, we're joined by two panelists: Gene Kim, DevOps researcher and author focused on IT operations, information security and transformation (his most recent book, The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win, will soon be followed by The DevOps Cookbook), and Ashish Kuthiala, Senior Director of Marketing and Strategy for Hewlett Packard Enterprise (HP) DevOps. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: Coordinating and fostering increased collaboration between development, testers, and IT operations has a lot of benefits. We've been talking about that in a number of these discussions, but security specifically. How specifically is DevOps engendering safer code and improved security?

Kuthiala: Dana, I look at security as no different than any other testing that you do on your code. Anything that you catch early-on in the process, fix it, and close the vulnerabilities is much simpler, much easier, and much cheaper to fix than when the end-product is in the hands of the users.

Learn the Four Keys

to Continuous DevOps

At that point, it could be in the hands of thousands of users, deployed in thousands of environments, and it's really very expensive. Even if you want to fix it there, if some trouble happens, if there is security breach, you're not just dealing with the code vulnerability, but you are also dealing with loss of brand, loss of revenue, and loss of reputation in the marketplace.

Gene has done a lot of study on security and DevOps. I would love to hear his point of view on that.

Promise is phenomenal

Kim: You're so right. The promise of DevOps for advancing the information security objective is phenomenal, but unfortunately, the way most information security practitioners react to DevOps is one of moral outrage and fear. The fear being verbalized is that Dev and Ops are deploying more quickly than ever, and the outcomes haven't been so great. You're doing one release a year, what will happen if they are doing 10 deploys a day? [See a recent interview with Gene from the DevOps Enterprise Summit.]


We can understand why they might be just terrified of this. Yet, what Ashish described is that DevOps represents the ideal integration of testing into the the daily work of Dev and Ops. We have testing happening all the time. Developers own the responsibilities of building and running the test. It’s happening after every code commit, and these are exactly same sort of behaviors and cultural norms that we want in information security. After all, security is just another aspect of quality.

We're seeing many, many examples of how organizations are creating what some people calling DevOps(Sec), that is DevOps plus security. One of my favorite examples is Capital One, which calls DevOps in their organization DevOps(Sec). Basically, information security is being integrated into every stage of the software development lifecycle. This is actually what every information security practitioner has wanted for the last two decades.


Kuthiala: Gene, that brings up an interesting thought. As we look at Dev and Ops teams coming together without security, increasingly we talk about how people need to have generally more skills across the spectrum. Developers need to understand production systems and to be able to support their code in production. But what you just described, does that mean that’s how the developers and planners start to become security specialist or think like that? What have you seen?

Kim: Let's talk about the numbers for a second. I love this ratio of 100 to 10 to 1. For every 100 developers, we have 10 operations people, and you have one security person. So there's no way you're going to get the adequate coverage, right? There are not enough security people around. If we can't embed Ops people into these project or service teams, then we have to train developers to care and know when seek help from the Ops experts.

We have the similar challenge in information security -- how we train, whether it's about secure coding, regular compliance, or how we create evidence that controls exist and are effective. It is not going to be security doing the work. Instead, security needs to be training Dev and Ops on how to do things securely.

Kuthiala: Are there patterns that they should be looking at in security? Are there any known patterns out there or are there some being developed? What you have seen with the customers that you work with?

Kim: In the deployment pipeline, instead of having just unit tests being run after every code commit, you actually run static code analysis tools. That way you know that it's functionally correct, and the developers are getting fast feedback and then they’re writing things that are potentially more secure than they would have otherwise.

And then alongside that in production, there are the monitoring tools. You're running things like the dynamic security testing. Now, you can actually see how it’s behaving in the production environment. In my mind, that's the ideal embodiment of how information security work should be integrated into the daily work of dev, test, and operations.

Seems contradictory

Kuthiala: It seems a little contradictory in nature. I know DevOps is all about going a little faster, but actually, you’re adding more functionality right up front and slowing this down. Is it a classic case of going slower to go faster? Walk before you can run, until you get to crawl? From my point of view, it slows you down here, but toward the end, you speed up more. Are you able to do this?

Kim: I would claim the opposite. We're getting the best of all worlds, because the security testing is now automated. It’s being done on demand by the developers, as opposed to your opening a ticket, "Gene, can you scan my application?" And I'll get back to you in about six weeks.

Learn the Four Keys

to Continuous DevOps

That’s being done automatically as part of my daily work. My claim would be not only is it faster, but we'll get better coverage than we had before. The fearful info sector person would ask how we can do this for highly regulated environments, where there is a lot of compliance regimes in place.

If you were to count the number of controls that are continuously operating, not only do you have orders and managing more controls, but they are actually operating all the time as opposed to testing once a year.

Kuthiala: From what I've observed with my customers, I have two kind of separate questions here. First, if you look at some of the highly regulated industries, for example, the pharmaceutical industry, it's not just internal compliance and regulations. It's part of security, but they often have to go to the outside agencies for almost physical paperwork kind of regulatory compliance checks.

Not only can you be compliant with all the relevant laws, contractual obligations, and regulations, but you can significantly decrease the amount of work.

As they're trying to go toward DevOps and speed this up, they are saying, "How do we handle that portion of the compliance checks and the security checks, because they are manual checks? They're not automated. How do we deal with external agencies and incorporate this in? What have you seen work really well?

Kim: Last year, at the DevOps Enterprise Summit, we had one bank, and it was a smaller bank. This year, we have five including some of the most well-known banks in the industry. We had manufacturing. I think we had coverage of almost every major industry vertical, the majority of which are heavily regulated. They are all able to demonstrate that not only can you be compliant with all the relevant laws, contractual obligations, and regulations, but you can significantly decrease the amount of work.

One of my favorite examples came from Salesforce. Selling to the Federal government, they had to apply with FedRAMP. One of the things that they got agreement on from security, compliance groups, and change management was that all infrastructure changes made through the automation tools could be considered a standard change.

In other words, they wouldn’t require review and approval, but all changes that were done manually would still require approvals, which would often take weeks. This really shows that we can create this fast path not just for the people doing the work, but also, this make some work significantly easier for security and compliance as well.

Human error

Kuthiala: And you're taking on the human error possibility in there. People can be on vacation, slowing things down. People can be sick. People may not be in their jobs anymore. Automation is a key answer to this, as you said. [More insights from HP from the DevOps Enterprise Summit.]

Gardner: One of things we've been grappling with in the industry is how to get DevOps accelerated into cultures and organizations. What about the security as a point on the arrow here? If we see and recognize that security can benefit from DevOps and we want to instantiate DevOps models faster, wouldn’t the security people be a good place to be on the evangelistic side of DevOps?

Kim: That’s a great observation, Dana. In fact, I think part of the method behind the madness is that the goal of the DevOps Enterprise Summit is to prove points. We have 50 speakers all from large, complex organizations. The goal is to get coverage of the industry verticals.

Learn the Four Keys

to Continuous DevOps

I also helped co-host a one-day DevOps Security Conference at the RSA Conference, and this was very much from a security perspective. It was amazing to find those champions in the security community who are driving DevOps objectives. They have to figure out how security fits into the DevOps ecosystem, because we need them to show that the water is not only just safe, but the water is great.

Kuthiala: This brings up a question, Gene. For any new project that kicks off, it’s a new company. You can really define the architecture from scratch, thus enabling you a lot of practices you need to put in place, whether it's independent deliverables and faster deliverables, all acting independent of each other.

But for the bigger companies and enterprise software that’s being released -- we've discussed this in our past talks -- you need to look at the architecture underneath it and see how we can modernize this to do this.

Just as marketing is too important to leave to the marketing people, and quality is too important to leave to the QA people -- so too security is too important to leave just to the security people.

So, when you start to address security, how do you go about approaching that, because you know you're dealing with a large base of code that’s very monolithic? It can take thousands of people to release something out to the customers. Now, you're trying to incorporate security into this with any new features and functions you add.

I can see how you can start to incorporate security and the expertise into it and scan it right from development cycle. How do you deal with that big component of the architecture that’s already there? Any best practices?

Kim: One of the people who have best articulated the philosophy is Gary Gruver. He said something that, for me, was very memorable. If you don’t have automated testing, and I think his context was very much like unit testing, automated regression testing, you have a fundamentally broken cost model, and it becomes too expensive. You get to a point where it becomes too expensive to add features.

That’s not even counting security testing. You get to a point where not only it is too expensive, but it becomes too risky to change code.

We have to fully empower developers to get feedback on their work and have them fully responsible for not just the features, but the non-functional requirements, testability, deployability, manageability, and security.

A better way

Gardner: Assume that those listening and reading here are completely swayed by our view of things and they do want to have DevOps with security ingrained. Are there not also concurrent developments around big data and analytics that give them a better way to do this, once they've decided to do it.

It seems to me that there is an awful lot of data available within systems, whether it's log files, configuration databases. Starting to harness that affordably, and then applying that back to those automation capabilities is going to be a very powerful synergistic value. How does it work when we apply big data to DevOps and security, Ashish?

Kuthiala: Good question Dana. You're absolutely right with data sources now becoming easy, bringing together data sources into one repository and at an affordable cost. We're starting to build analytics on top of that and this has being applied in a number of areas.

We're finding that we're about 80 to 85 percent accurate in predicting what to test and not to test and what features are reflected or not.

The best example I can talk about is how HP has been working on an IP creation of the area of testing using big data analytics. So, if we have to go faster and we have to release software every hour or every two, versus every six to eight months, you need to test it as fast as well. You can no longer afford to go and run your 20,000 tests based on this one-line change of code.

You have to be able to figure out what modules are affected, which ones are not, and which ones are likely to break. We're starting to do some intelligent testing inside of our labs and we're finding that we're about 80 to 85 percent accurate in predicting what to test and not to test and what features are reflected or not.

Similarly, using the big data analytics and the security expertise that Gene talked about, you need to start digging through and analyzing exactly the same as we run any test. What security vulnerabilities do you want to test, which functions of the code? And it’s just a best practice moving forward that you start to incorporate the big data analytics into your security testing.

Kim: You were implying something that I just want to make explicit. One of the most provocative notions that Ashish and I talked about was to think about all the telemetry and all the data that the build mechanisms create. You start putting in all the results of testing, and suddenly we have a much better basis of where we apply our testing effort.

Learn the Four Keys

to Continuous DevOps

If we actually need to deploy faster, even if we completely automate our tests, and even if we parallelize them and run them across thousands of servers and if that takes days, we may be able use data to tell us where to surgically apply testing so we make a informed decision on whether to deploy or not. That's an awesome potential.

Gardner: Speaking of awesome potentials, when we compress the feedback loops using this data -- when development and operations are collaborating and communicating very well -- it seems to me that we're also moving from a reactive stance to security issues to a proactive stance.

One of the notions about security is that you can’t prevent people from getting in, but you can limit the damage they can do when they do get in. It seems to me that if you close a loop between development operations and test, you can get the right remediation out into operations and production much quicker. Therefore you can almost behave as we had seen with anti-malware software -- where the cycle between the inception of a problem, the creation of the patch, and then deployment of the patch was very, very short.

Is that vision pie in the sky or is that something we could get to when DevOps and security come together, Gene?

Key to prevention

Kim: You're right on. The way an auditor would talk about it is that there are things that we can do to prevent: that’s code review, that’s automated code testing and scanning.

Making libraries available so that developers are choosing things and deploying them in a secured state are all preventive controls. If we can make sure that we have the best situational awareness we can of the production environment, those are what allow quicker detection recovery.

The better we are at that, the better we are at mitigating, effectively mitigating risk.

Kuthiala: Gene, as you were talking, I was thinking. We have this notion of rolling back code when something breaks in production, and that’s a very common kind of procedure. You go back into the lab, fix what didn’t work, and then you roll it back into production. If it works, it's fine. Otherwise, you roll it back and do it over again.

But with the advent of DevOps and those who are doing this successfully, there are no roll backs. They roll forward. You just go forward, because with the discipline of DevOps, if done well, you can quickly put a patch into production within hours, versus months, days, and weeks.

The more you talk about IoT, the more holes are open for hackers to get in.

And similarly like you talked about security, you know once a vulnerability is out there that you want to go fix it, you want to issue the patch. With DevOps and security, there are lot of similarities.

Gardner: Before we close out, is there anything more for the future? We've heard a lot about the Internet of Things (IoT), a lot more devices, device types, networks, extended networks, and variable networks. Is there a benefit with DevOps and security as a tag team, as we look to an increased era of complexity around the IoT sensors and plethora of disparate networks? Ashish?

Kuthiala: The more you talk about IoT, the more holes are open for hackers to get in. I'll give you classic example. I've been looking forward to the day where my phone is all I carry. I don’t have to open my car with my keys or I can pay for things with it, and we have been getting toward that vision, but a lot of my friends who are in high-tech are actually skeptical.

Learn the Four Keys

to Continuous DevOps

What happens if you lose your phone? Somebody has access to it. You know their counter argument against that. You can switch off your phone and wipe the data etc. But I think as IoT grows in number, more holes open up. So, it becomes even more important to incorporate your security planning cycles right into the planning and software development cycles.

Gardner: Particularly if you're in an industry where you expect to an have an Internet of Things ramp-up, getting automation in place, thinking about DevOps, thinking about security as an integral part of DevOps -- it all certainly makes a great deal of sense to me.

Kim: Absolutely, you said it better than I ever could. Yes.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy. Sponsor: Hewlett Packard Enterprise.

You may also be interested in:

Tags:  Ashish Kuthiala  BriefingsDirect  cybersecurity  Dana Gardner  DevOps  Gene Kim  Hewlett Packard Enterprise  Interarbor Solutions  security 

Share |
PermalinkComments (0)

How Sprint employs orchestration and automation to bring IT into DevOps readiness

Posted By Dana L Gardner, Thursday, October 15, 2015

The next BriefingsDirect DevOps innovation case study explores how telecommunications giant Sprint places an emphasis on orchestration and automation to bring IT culture and infrastructure into readiness for cloud, software-defined data center (SDDC) and DevOps.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy.

Learn how Sprint has made IT infrastructure orchestration and automation a pillar of its strategic IT architecture future from Chris Saunderson, Program Manager and Lead Architect for Data Center Automation at Sprint in Kansas City, Missouri. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: I'm intrigued by your emphasis on working toward IT infrastructure, of getting to more automation at a strategic level. Tell us why you think automation and orchestrations are of strategic benefit to IT.

IT automation is an urgent priority
Automation and orchestration can pay huge dividends
Read the report

Saunderson: We've been doing automation since 2011, but it came out of an appreciation that the velocity of change inside that data center is just going to increase over time.

In 2009, my boss and I sat down and said, "Look, this is going nowhere. We're not going to make a significant enough impact on the way that the IT division works ... if we just keep doing the same thing."


That’s when we sat down and saw the orchestrated data center coming. I encapsulated it as the "data center of things." When you look at the journey that most enterprises go through, right around 2009 is when the data center of things emerged. You then began to lose track of where things are, what they are, who uses them, how long they’ve been there, and what their state is.

When we looked at automation and orchestration in 2009, it was very solidly focused on IT operational efficiency, but we had the longer-term view that it that it was going to be foundational for the way to do things going forward -- if for nothing else than to handle the data center of things. We could also see changes coming in the way that our IT organization was going to have to respond to the change in our business, let alone just the technology change.

Gardner: So that orchestration has allowed you to not only solve the problems of the day, but put a foundation in place for the new problems, rapid change, cloud, mobile, all of these things that are happening. Before we go back to the foundational aspects, tell us a little bit about Sprint itself, and why your business is changing.

Provider of choice

Saunderson: The Sprint of today ... We're number three, with aspirations to be bigger, better, and faster, and the provider of choice in wireless, voice and data. We're a tier 1 network service provider of global IP network along with private network, MPLS backbone, all that kind of stuff. We're a leader in TRS -- Telecommunication Relay Services for the deaf.

The Sprint of old is turning into the Sprint of new, where we look at mobile and we say mobile is it, mobile is everything -- that Internet of Things (IoT). That's what we want to foster growth. I see an exciting company that’s coming in terms of connecting people not only to each other, but to their partners, the people who supply services to them, to their entertainment, to their business. That’s what we do.

Gardner: When you started that journey for automation -- getting out of those manual processes and managing complexity, but repeatedly getting the manual labor out of the way -- what have you learned that you might relate to other people? What are some of the first things people should keep in mind as they embark on this journey?

Saunderson: It’s really a two-part answer. Orchestration comes after automation, because orchestration is there to consume the new automation services. So let’s take that one first. The big things to remember is that change is hard for people. Not technology change. People are very good about doing technology change, but unwiring people’s brains is a problem, and you have to acknowledge that up-front. You’re going to have a significant amount of resistance from people to change the way that they're used to doing things.

Orchestration comes after automation, because orchestration is there to consume the new automation services.

Now addressing that is also a human problem, but in a certain sense, the technology helps because you're able to say things like, "Let's just look at the result and let's compare what it takes to get to the result. Was it the humans doing it, and what does it take to get to the result with the machines doing it?" Let’s just call it what it is. It’s machines doing things. If the result is the same, then it doesn't require the humans. That’s challenge number one, unwiring people’s minds.

The second is making sure that you are articulating the relevance of what you’re doing. We had an inbuilt advantage, at least in the automation space, of having some external forces that were driving us to do this.

It’s really regulatory compliance, right? Sarbanes-Oxley (SOX) is what it is. PCI is what it is --  SAS70, FISMA, those things. We had to recognize the excessive amount of labor that we were expending to try and keep up with regulatory change.

PCI changes every year or 18 months. So it's just going through every rule set and saying, "Yes, this doesn’t apply to me; I'm more restricted." That takes six people. We were able to turn that. We were able to address the requirement to continue to do compliance more effectively and more efficiently. Don’t lose that upward communication, the relevancy thing -- which is not only are we doing this more efficiently, but we are better at it?

When you get to orchestration, now you’re really talking about some interesting stuff because this is where you begin to talk about being able to do continuous compliance, for example. That says, "Okay, we used to treat this activity as once a quarter or maybe once a month. Let's just do it all the time, but don’t even have a human involved in it." Anybody who has talked to me about this will hear this over and over again. I want smart people working on smart things. I do not want smart people working on dumb things. Truth be told, 99 percent of the things that IT people do are dumb things.

Orchestration benefits

The problem with them is that they're dumb because they force a human to look at the thing and make a decision. Orchestration allows you take that one level out, look at the thing, and figure out how to make that decision without a human having to make it. Then, tie that to your policy, then report on policy compliance, and you're done.

The moment you do that, you’re freeing people up to go have the harder discussions. This is where we start to talk about DevOps and this is where we start to talk about some of the bigger blocks that grind against each other in the IT world.

Gardner: "Continuous" is very interesting. You use the PCI compliance issue, but it's also very important when it comes to applications, software development, test, and deploy. Is there anything that you can explain for us about the orchestration and automation that lends itself to that continuous delivery of applications? People might not put the two together, but I'm pretty sure there's a connection here.

IT automation is an urgent priority
Automation and orchestration can pay huge dividends
Read the report

Saunderson: There is. DevOps is a philosophy. There was a fantastic discussion from Adobe where it was very clear that DevOps is a philosophy, an organizational discussion. It’s not necessarily a technology discussion. The thing that I would say, though, is that you can apply continuous everywhere.

The successes that we're having in that orchestration layer is that it's a much easier discussion to go in and say, "You know how we do this over here? Well, what if it was a release candidate code?" The real trick there, when you go back to the things that I want people to think about, is that DevOps is a philosophy, because it requires development and operations to work together, not one hand off to the other, and not one superior to the other; it’s together.

If they’re not willing to walk down the same path together, then you have an organizational problem, but you may also have a toolset problem as well. We're an Application Lifecycle Manager (ALM) shop. We have it there. Does it cover all of our applications? No. Are we getting all of the value out of it that we could? No.

But that’s because we're spending time in getting ready to do things like connect ALM into the running environment. The bigger problem, Dana, is that the organization has to be ready for it, because your philosophical changes are way more difficult than technical changes. Continuous means everything else has to be continuous along with it.

If you're in the ITIL model, you’re still going to need configuration items (CIs). How do CIs translate to Docker containers? Do they need to be described in the same way? If the operations team isn't necessarily as involved in the management of continuously deployed applications, who do I route a ticket to and how do they fix it?

This is where I look at it and say that this is the opportunity for orchestration to sit underneath that and say it not only has the capability to enable people to deploy continuously -- whether it’s into test or production, disaster recovery, or any other environment.

To equip them to be able to operate the continuous operation (that’s coming after the continuous integration and development and deployment), that has to be welded on because you’re going to enforce dis-synergy if you don’t address it all at the same time as you do with integration and deployment.

Gardner: Let’s look at some other values that you can derive from better orchestration and automation. I'm thinking about managing complexity, managing scale, but also more of the software-defined variety. We are seeing a lot of software-defined storage (SDS), software-defined networking (SDN), ultimately software-defined data center (SDDC), all of which is abstracted and managed. How do you find the path to SDDC, vis-à-vis better orchestration and automation?

At the core

Saunderson: Orchestration is going to have to be at the core of that. If you look at the product offerings just across the space, you’re starting to see orchestration pop up in every last one of them -- simply because there's no other way to do it.

RESTFul APIs are nice, but it’s not enough because, at that point, you’re asking customers to start bolting things together themselves, as opposed to saying, "I'm going to give you a nice orchestrated interface, where I have a predefined set of actions that are going be executed when you poll that orchestration to make it work and then apply that across the gamut."

SDS is coming after SDN. Don’t misunderstand me. We're not even at the point of deploying software defined networks, but we look at it and we say, "I have to have that, if for no other reason than I need to remove the human hands out of the delivery chain for things that touch the network."

We should never lose sight of the fact that the whole reason to do this is to say, "Deploy the thing."

I go back to the data center of things. The moment you go to 10Gbit, where you are using virtual context, just anything that’s in the current lexicon of new networking as opposed to VLANs, versus all that stuff, switchboards, etc., you’re absolutely losing visibility.

Without orchestration, and, behind that, without the analytics to look at what's happening in the orchestration that’s touching the elements in your data center, you’re going to be blind. Now, we’re starting to talk about going back to the dark ages. I think we're smarter than that.

By looking at orchestration as the enabler for all of that, you start to get better capability to deliver that visibility that you’re after, as well as the efficiency. We should never lose sight of the fact that the whole reason to do this is to say, "Deploy the thing."

That’s fine, but how do I run it, how do I assure it, how do I find it? This keeps coming up over and over. Eventually, you’re going to have to do something to that thing, whether it’s deployed again, whether you have some kind of security event that is attached to it, or the business just decides not to do it any more. Then, I have to find it and do something to it.

Gardner: Given your deep knowledge and understanding of orchestration and automation, what would you like to see done better for the tools that are provided to you to do this?

Is there a top-three list of things you’d like to see that would help you extend the value of your orchestration and automation, do things like software-defined, do things like DevOps as a philosophy, ultimately to be have more of a data-driven IT of strategic operation?

Development shop

Saunderson: I'm not sure I have a top three. I can certainly talk about generic principal stuff, which is, I want open. That’s what I really want. Just to take the sideline for a second, it’s fascinating. It’s just absolutely fascinating. IT operations is starting to become a software development shop now.

I'm not resistant to that in the least because, just in this conversation, we've been talking about RESTFul APIs and we were talking about orchestration. None of this is IT operations stuff. This isn’t electrons flowing through copper anymore. It’s business process translated into a set of actions, open, and interoperable.

Then, just give me rich data about those things, very rich data. We’re getting to that point, just by the shear evolution of big data, that it doesn’t matter anymore. Just give it all to me, and I will filter it out to what I'm looking for.

Gardner: The thing that is interesting with Hewlett Packard Enterprise (HPE) is that they do have a big-data capability, as well as a leading operations capability and they're starting to put it all together.

Saunderson: In the same way the orchestration is starting to pop up everywhere. If you look at the HPE product portfolio and you look at network coordination, it’s going to have an operations orchestration interface into it. Server automation is welded into operations orchestration and it’s going to appear everywhere else. Big data is coming with it.

Server automation is welded into operations orchestration and it’s going to appear everywhere else.

I'm not hesitant on it. It's just that it introduces complexity for me. The fact that the reporting engine is starting to turn big data is good. I'm happy for that. It just has to get more. It’s not enough to just be giving me job results that are easy to find and easy to search. Now, I want to get some really rich metadata out of things.

Software-defined network is a good example. The whole open flow activity just by itself looks like network management until it goes into a big-data thing and then suddenly, now I have a data source that I can start correlating events to that turn into actions inside the control that turns into change on the network. 

Let’s extend that concept. Let’s put that into orchestration, into service management, or into automation. Give me that and it doesn’t have to be the single platform. Give me a way to anticipate HPE’s product roadmap. The challenge for HPE is delivery.

Gardner: Before we sign off, one of the important things about IT investment is getting the buy-in and support from your superiors or the other aspects of your enterprise. Are there some tangible metrics of success, returns on investment (ROIs), improvements and productivity that you can point to from your orchestration, not just helping smart people do smart things, but benefiting the business at large? 

Business case

Saunderson: So organizations often only do the things that the boss checks. The obvious priorities for us are straight around our business case.

If you want to look at real tangibles, our virtual server provisioning, even though it’s the  heavyweight process that it is today, is turning from days into hours. That’s serious change, that’s serious organizational cultural change, but it’s not enough. It has to be minutes not hours, right? 

Then there's compliance. I keep coming back to it as this is a foundational thing. We're able to continue to pass SOX and PCI every time, but we do it efficiently. That’s a cultural change as well, but that’s something that CIOs and above do care about, because it’s kind of important.

One gets your CFO in trouble, and the other ones stops you taking payments. That gets people's attention really quickly. The moment you can delve into those and demonstrate that not only are you meeting those regulatory requirements, and you're able to report all of them and have auditors look at it and say yes we agree, you are doing all those things that you should be doing.

IT automation is an urgent priority
Automation and orchestration can pay huge dividends
Read the report

Then, you can flip that into the next area which is that we do have to go look at our applications for compliance. We have rich metadata over here that was able to articulate things. So let’s apply it there.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy. Sponsor: Hewlett Packard Enterprise.

You may also be interested in:

Tags:  BriefingsDirect  Chris Saundserson  cloud computing  Dana Gardner  DevOps  Hewlett Packard Enterprise  Interarbor Solutions  IT automoation  orchestration  Sprint 

Share |
PermalinkComments (0)

How fast analytics changes the game and expands the market for big data value

Posted By Dana L Gardner, Monday, October 05, 2015

The next BriefingsDirect big-data thought leadership discussion highlights how fast analytics -- or getting to a big data analysis value in far less time than before -- expands the market for advanced data infrastructure to gain business insights.

We'll learn how bringing analytics to a cloud services model also allows smaller and less data-architecture-experienced firms to benefit from the latest in big-data capabilities. And we'll explore how Dasher Technologies is helping to usher in this democratization of big data value to more players in less time.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or  download a copy.

To share how a fast ramp-up for big data as a service has evolved, we're joined by Justin Harrigan, Data Architecture Strategist at Dasher Technologies, as well as Chris Saso, Senior Vice President of Technology at Dasher Technologies in Campbell, California. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: Justin, how have big-data practices changed over the past five years to set the stage for rapid leveraging of big-data capabilities?

Harrigan: Back in 2010, we saw big data become mainstream. Hadoop became a household name in the IT industry, doing scale-out architectures. Linux databases were becoming common practice. Moving away from traditional legacy, smaller, slower databases allowed this whole new world of analytics to open up to previously untapped resources within companies. So data that people had just been sitting on could now be used for actionable insights.


Fast forward to 2015, and we've seen big data become more approachable. Five years ago, only the largest organizations or companies that were specifically designed to leverage big-data architectures could do so. The smaller guys had maybe a couple of hundred or even tens of terabytes, and it required too much expertise or too much time and investment to get a big-data infrastructure up and running.

Today, we have approachable analytics, analytics as a service, hardened architectures that are almost turnkey with back-end hardware, database support, and applications -- all integrating seamlessly. As a result, the user on the front end, who is actually interacting with the data and making insights, is able to do so with very little overhead, very little upkeep, and is able to turn that data into business-impact data, where they can make decisions for the company.

Gardner: Justin, how big of an impact has this had? How many more types of companies or verticals have been enabled to start exploring advanced, cutting-edge, big-data capabilities? Is this a 20 percent increase? Perhaps almost any organization that wants to can start doing this.

Tipping point

Harrigan: The tipping point is when you outgrow your current solutions for data analytics. Data analytics is nothing new. We've been doing it for more than 50 years with databases. It’s just a matter of how big you can get, how much data you can put in one spot, and then run some sort of query against it and get a timely report that doesn’t take a week to come back or that doesn't time out on a traditional database.


Almost every company nowadays is growing so rapidly with the type of data they have. It doesn’t matter if you're an architecture firm, a marketing company, or a large enterprise getting information from all your smaller remote sites, everyone is compiling data to create better business decisions or create a system that makes their products run faster.

For people dipping their toes in the water for their first larger dataset analytics, there's a whole host of avenues available to them. They can go to some online providers, scale up a database in a couple of minutes, and be running.

They can download free trials. HP Vertica has a community edition, for example, and they can load it on a single server, up to terabytes, and start running there. And it’s significantly faster than traditional SQL.

It’s much more approachable. There are many different flavors and formats to start with, and people are realizing that. I wouldn’t even use the term big data anymore; big data is almost the norm.

Gardner: I suppose maybe the better term is any data, anytime.

Harrigan: Any data, anytime, anywhere, for anybody.

Gardner: I suppose another change over the past several years has been an emphasis away from batch processing, where you might do things at an infrequent or occasional basis, to this concept that’s more applicable to a cloud or an as-a-service model, where it’s streaming, continuous, and then you start reducing the latency down to getting close to real time.

Are we starting to see more and more companies being able to compress their feedback, and start to use data more rapidly as a result of this shift over the past five years or so?

Harrigan: It’s important to address the term big data. It’s almost like an umbrella, almost like the way people use cloud. With big data, you think large datasets, but you mentioned speed and agility. The ability to have real-time analytics is something that's becoming more prevalent and the ability to not just run a batch process for 18 hours on petabytes of data, but having a chart or a graph or some sort of report in real time. Interacting with it and making decisions on the spot is becoming mainstream.

We did a blog post on this not long ago, talking about how instead of big data, we should talk about the data pipe. That’s data ingest or fast data, typically OLTP data, that needs to run in memory or on hardware that's extremely fast to create a data stream that can ingest all the different points, sensors, or machine data that’s coming in.

Smarter analysis

Then we've talked about smarter analytic data that required some sort of number-crunching dataset on data that was relevant, not data that was real-time, but still fairly new, call it seven days or older and up to a year. And then, there's the data lake, which essentially is your data repository for historical data crunching.

Those are three areas you need to address when you talk about big data. The ability to consume that data as a service is now being made available by a whole host of companies in very different niches.

It doesn’t matter if it’s log data or sensor data, there's probably a service you can enable to start having data come in, ingest it, and make real-time decisions without having to stand up your own infrastructure.

Gardner: Of course, when organizations try to do more of these advanced things that can be so beneficial to their business, they have to take into consideration the technology, their skills, their culture -- people, process and technology, right?

Chris, tell us a bit about Dasher Technologies and how you're helping organizations do more with big-data capabilities, how you address this holistically, and this whole approach of people, process and technology.

Dasher has built up our team to be able to have a set of solutions that can help people solve these kinds of problems.

Saso: Dasher was founded in 1999 by Laurie Dasher. To give you an idea of who we are, we're a little over 65 employees now, and the size of our business is somewhere around $100 million.

We started by specializing in solving major data-center infrastructure challenges that folks had by actually applying the people, process and technology mantra. We started in the data center, addressing people’s scale out, server, storage, and networking types of problems. Over the past five or six years, we've been spending our energy, strategy, and time on the big areas around mobility, security, and of course, big data.

As a matter of fact, Justin and I were recently working on a project with a client around combining both mobility information and big data. It’s a retail client. They want to be able to send information to a customer that might be walking through a store, maybe send a coupon or things like that. So, as Justin was just talking about, you need fast information and making actionable things happen with that data quickly. You're combining something around mobility with big data.

Dasher has built up our team to be able to have a set of solutions that can help people solve these kinds of problems.

Gardner: Justin, let’s flesh that out a little bit around mobility. When people are using a mobile device, they're creating data that, through apps, can be shared back to a carrier, as well as application hosts and the application writers. So we have streams of data now about user experience and activities.

We also can deliver data and insights out to people in the other direction in that real-time of fashion, a closed loop, regardless of where they are. They don’t have to be at their desk, they don’t have to be looking at a specific business-intelligence (BI) application for example. So how has mobility changed the game in the past five years?

Capturing data

Harrigan: Dana, it’s funny you brought up the two different ways to capture data. Devices can be both used as a sensor point or as a way to interact with data. I remember seeing a podcast you did with HP Vertica and GUESS regarding how they interacted with their database on iPads.

In regards to interacting with data, it has become not only useful to data analysts or data scientists, but we can push that down into a format so lower-level folks who aren't so technical. With a fancy application in front of them, they can use the data as well to make decisions for companies and actually benefit the company.

You give that data to someone in a store, at GUESS for example, who can benefit by understanding where in the store to put jeans to impact sales. That’s huge. Rather than giving them a quarterly report and stuff that's outdated for the season, they can do it that same day and see what other sites are doing.

On the flip side, mobile devices are now sensors. A mobile device is constantly pinging access points over wi-fi. We can capture that data and, through a MAC address as an unique identifier, follow someone as they move through a store or throughout a city. Then, when they return, that person’s data is captured into a database and it becomes historical. They can track them through their device.

Read more on tackling big data analytics
Learn how the future is all about fast data
Find out how big data trends affect your business

It allows a whole new world of opportunities in terms of the way retailers interact with where they place merchandise, the way they interact with how they staff stores to make sure they have the proper amount of people for the certain time, what weather impact has on the store.

Lastly, as Chris mentioned, how do we interact with people on devices by pushing them data that's relevant as they move throughout their day?

The next generation of big data is not just capturing data and using it in reports, but taking that data in real time and possibly pushing it back out to the person who needs it most. In the retail scenario, that's the end users, possibly giving them a coupon as they're standing in front of something on a shelf that is relevant and something they will use.

Gardner: So we're not just talking about democratization of analytics in terms of the types of organizations, but now we're even talking about the types of individuals within those organizations.

Do you have any examples of some Dasher’s clients that have been able to exploit these advances and occurrences with mobile and cloud working in tandem, and how that's produced some sort of a business benefit?

Business impact

Harrigan: A good example of a client who leveraged a large dataset is One Kings Lane. They were having difficulty updating the website their users were interacting with because it’s a flash shopping website, where the information changes daily, and you have to be able to update it very quickly. Traditional technologies were causing a business impact and slowing things down.

They were able to leverage a really fast columnar database to make these changes and actually grow the inventory, grow the site, and have updates happen in almost real time, so that there was no impact or downtime when they needed to make these changes. That's a real-world example of when big data had the direct impact on the business line.

Gardner: Chris, tell us a little bit about how Dasher works with Hewlett Packard Enterprise technologies, and perhaps even some other HP partners like GoodData, when it comes to providing analytics as a service?

Once Vertica . . . has done the analysis, you have to report on that and make it in a nice human-readable form or human-consumable form.

Saso: HP has been a longtime partner from the very beginning, actually when we started the company. We were a partner of Vertica before HP purchased them back in 2011.

We started working with Vertica around big data, and Justin was one of our leads in that area at the time. We've grown that business and in other business units within HP to combine solutions, Vertica, big data, and hardware, as Justin was just talking about. You brought up the applications that are analyzing this big data. So we're partners in the ecosystem that help people analyze the data.

Once HP Vertica, or what have you, has done the analysis, you have to report on that and make it in a nice human-readable form or human-consumable form. We’ve built out our ecosystem at Dasher to have not only the analytics piece, but also the reporting piece.

Gardner: And on the as a service side, do you work with GoodData at all or are you familiar with them?

Saso: Justin, maybe you can talk a little bit about that. You've worked with them more I think on their projects.

Optimizing the environment

Harrigan: GoodData is a large consumer of Vertica and they actually leverage it for their back-end analytics platform for the service that they offer. Dasher has been working with GoodData over the past year to optimize the environment that they run on.

Vertica has different deployment scenarios, and you can actually deploy it in a virtual-machine (VM) environment or on bare-metal. And we did an analysis to see if there was a return on investment (ROI) on moving from a virtualized environment running on OpenStack to a bare-metal environment. Through a six-month proof of concept (POC), we leveraged HP Labs in Houston. We had a four-node system setup with multiple terabytes of data.

We saw 4:1 increase in performance in moving from a VM with the same resources to a bare-metal machine. That’s going to have a significant impact on the way they move data in their environment in the future and how they adjust to customers with larger datasets.

Gardner: When we think about optimizing the architecture and environment for big data, are there any other surprises or perhaps counter-intuitive things that have come up, maybe even converged infrastructure for smaller organizations that want to get in fast and don’t want to be too concerned with the architecture underlying the analytics applications?

That’s going to have a significant impact on the way they move data in their environment in the future and how they adjust to customers with larger datasets.

Harrigan: There's a tendency now with so many free solutions out there to pick a free solution, something that gets the job done now, something that grows the business rapidly, but to forget about what businesses will need three years down the road, if it's going to grow, if it’s going to survive.

There are a lot of startups out there that are able to build a big data infrastructure, scale it to 5,000 nodes, and then they reach a limit. There are network limits on how fast the switch can move data between nodes, constantly pushing the limits of 10 Gbyte, 40 Gyte and soon 100 Gbyte networks to keep those infrastructures up.

Depending on what architecture you choose, you may be limited in the number of nodes you can go to. So there are solutions out there that can process a million transactions per second with 100 nodes, and then there are solutions that can process a million transactions per second with 20 nodes, but may cost slightly more.

If you think long-term, if you start in the cloud, you want to be able to move out of the cloud. If you start with an open ecosystem, you want to make sure that your hardware refresh is not going to cost so much that the company can’t afford it three years down the road. One of the areas we help consult with, when picking different architectures, is thinking long-term. Don't think six weeks down the road, how are we going to get our service up and running? Think, okay, we have a significant client install base, how we are going to grow the business from three to five years and five to 10 years?

Gardner: Given that you have quite a few different types of clients, and the idea of optimizing architecture for the long-term seems to be important, I know with smaller companies there’s that temptation to just run with whatever you get going quickly.

What other lessons can we learn from that long-term view when it comes to skills, security, something more than the speeds and feeds aspects of thinking long term about big data?

Numerous regulations

Harrigan: Think about where your data is going to reside and the requirements and regulations that you may run into. There are a million different regulations we have to do now with HIPAA, ITAR, and money transaction processes in a company. So if you ever perceive that need, make sure you're in an ecosystem that supports it. The temptation for smaller companies is just to go cloud, but who owns that data if you go under, or who owns that data when you get audited?

Another problem is encryption. If you're going to start gaining larger customers once you have a proven technology or a proven service, they're going to want to make sure that you're compliant for all their regulations, not just your regulations that your company is enforcing.

There's logging that they're required to have, and there is going to be encryption and protocols and the ability to do audits on anyone who is accessing the data.

Gardner: On this topic of optimizing, when you do it right, when you think about the long term, how do you know you have that right? Are there some metrics of success? Are there some key performance indicators (KPIs) or ROIs that one should look to so they know that they're not erring on the side of going too commercial or too open source or thinking short term only? Maybe some examples of what one should be looking for and how to measure that.

If you implement a system and it costs you $10 million to run and your ROI is $5 million, you've made a bad decision.

Harrigan: That’s going to be largely subjective to each business. Obviously if you're just going to use a rule of thumb, it shouldn't cost you more money than it makes you. If you implement a system and it costs you $10 million to run and your ROI is $5 million, you've made a bad decision.

The two factors are the value to the business. If you're a large enterprise and you implement big data, and it gives you the ability to make decisions and quantify those decisions, then you can put a number to that and see how much value that big-data system is creating. For example, a new marketing campaign or something you're doing with your remote sites or your retail branches and it’s quantifiable and it’s having an impact on the business.

The other way to judge it is impact on business. So, for ad serving companies, the way they make money is ad impressions, and the more ad impressions they can view, for the least cost in their environment, the higher return they're going to make. The delta is between the infrastructure costs and the top line that they get to report to all their investors.

If they can do 56 billion ad impressions in a day, and you can double that by switching architectures, that’s probably a good investment. But if you can only improve it by 10 percent by switching architectures, it’s probably too much work for what it’s worth.

Read more on tackling big data analytics
Learn how the future is all about fast data
Find out how big data trends affect your business

Gardner: One last area on this optimization idea. We've seen, of course, organizations subjectively make decisions about whether to do this on-premises, maybe either virtualized or on bare metal. They will do their cost-benefit analysis. Others are looking at cloud and as a service model.

Over time, we expect to have a hybrid capability, and as you mentioned, if you think ahead that if you start in the cloud and move private, or if you start private you want to be able to move to the cloud, we're seeing the likelihood of more of that being able to move back and forth.

Thinking about that, do you expect that companies will be able to do that? Where does that make the most sense when it comes to data? Is there a type of analysis that you might want to do in a cloud environment primarily, but other types of things you might do private? How do we start to think about breaking out where on the spectrum of hybrid cloud set of options one should be considering for different types of big-data activity?

Either-or decision

Harrigan: In the large data analytics world, it’s almost an either-or decision at this time. I don’t know what it will look like in the future.

Workloads that lend themselves extremely well to the cloud are inconsistent, maybe seasonal, where 90 percent of your business happens in December. Seasonal workloads like that lend themselves extremely well to the cloud.

Or, if your business is just starting out, and you don't know if you're going to need a full 400-node cluster to run whatever platform or analytics platform you choose, and the hardware sits idle for 50 percent of the time, or you don’t get full utilization. Those companies need a cloud architecture, because they can scale up and scale down based on needs.

Companies that benefit from on-premise are ones that can see significant savings by not using cloud and paying someone else to run their environment. Those companies typically pin the CPU usage meter at 100 percent, as much as they can, and then add nodes to add more capacity.

The best advice I could give is, if you start in the cloud or you start on bare metal, make sure you have agility and you're able to move workloads around. If you choose one sort of architecture that only works in the cloud and you are scaling up and you have to do a rip and replace scenario just to get out of the cloud and move to on-premise, that’s going to be significant business impact.

One of the reasons I like HP Vertica is that it has a cloud instance that can run on a public cloud. That same instance, that same architecture runs just as well on bare metal, only faster.

Gardner: Chris, last word to you. For those organizations out there struggling with big data, trying to figure out the best path, trying to think long term, and from an architectural and strategic point of view, what should they consider when coming to an organization like Dasher? Where is your sweet spot in terms of working with these organizations? How should they best consider how to take advantage of what you have to offer?

Saso: Every organization is different, and this is one area where that's true. When people are just looking for servers, they're pretty much all the same. But when you're actually trying to figure out your strategy for how you are going to use big-data analytics, every company, big or small, probably does have a slightly different thing they are trying to solve.

That's where we would sit down with that client and really listen and understand, are they trying to solve a speed issue with their data, are they trying to solve massive amounts of data and trying to find the needle in a haystack, the golden egg, golden nugget in there? Each of those approaches certainly has a different answer to it.

Read more on tackling big data analytics
Learn how the future is all about fast data
Find out how big data trends affect your business

So coming with your business problem and also what you would like to see as a result -- we would like to see x-number of increase in our customer satisfaction number or x-number of increase in revenue or something like that -- helps us define the metric that we can then help design toward.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or  download a copy. Sponsor: Hewlett Packard Enterprise.

You may also be interested in:

Tags:  big data  BriefingsDirect  Chris Saso  Dana Gardner  Dasher Technologies  data analytics  Hadoop  Hewlett Packard Enterprise  HP DISCOVER  HP Vertica  Interarbor Solutions  Justin Harrigan 

Share |
PermalinkComments (0)

How content in context within apps and process strengthens marketing muscle

Posted By Dana L Gardner, Tuesday, September 15, 2015

The next BriefingsDirect discussion explores the changing role and impact of content marketing, using the IT industry as a prime example. Just as companies now communicate with their consumers and prospects in much different ways -- with higher emphasis on social interactions, user feedback, big data analysis, and even more content to drive conversations -- so too the IT industry has abruptly changed.

There's more movement to cloud models, to mobile applications, to leveraging data at every chance -- and they are also facing lower-margin subscription business models. The margin for error is shrinking in the IT industry. If any industry is the poster child for how to deal with rapid change on all fronts, including marketing, it's surely the global information technology market.

Listen to the podcast. Find it on iTunes. Get the mobile app for iOS. Read a full transcript or download a copy.

To examine how the IT industry is adjusting to the dynamic nature of marketing, we're joined by Lora Kratchounova, the Founder and Principal at Scratch Marketing and Media in Cambridge, Mass. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: Lora, you and I have been talking about marketing for years now. We're in an interesting field, and it’s been such a dynamic time. I have some interesting ideas about where technology is going and where marketing is intercepting, and how they are both changing.

So, let’s start at a high level. Content marketing has proven to be very successful, and you and I have had a hand in this. Creating compelling stories, narratives about what’s going on, and how people can learn from peers as they go through problems and solve them, has become a mainstay in marketing. From your perspective, why is content marketing so important? Why has it been so successful?

Kratchounova: There are couple of reasons for that. The pace of change is tremendous now. People are trying to get their bearings on what’s going on in their markets, and a lot of times, they need to get educated. What has changed with social media now, information is a lot more immediate and transparent, and you can get it from many more sources than the just online presence of a company, for example.


The top-down modeling in the marketing is changing. We used to rely on companies to tell us how to think about the world, and now we can form our own opinions. As we realize that the customer is in the driver’s seat, they educate themselves, and they make the right decisions about how to go about change, companies are realizing that they need to feed into that flow and be part of that discussion. So content marketing has been so successful, because you become an educator, not just selling to people, and especially in IT.

Gardner: And I think people have become much more accustomed to conversations, rather than just a one-direction information flow. "We're the seller and we're going to tell you what it is." Now, people want to relate. They want to hear what others have to think. It’s much more of an actual conversation.

Ongoing conversation

Kratchounova: Exactly. Look at any IT domain. It’s interesting when we look at who is influencing and who the main voices in it are, who the voices that people consider experts are. You pretty much consistently see reporters, journalists, and the analysts folks like you, but then we see that there are a lot of C-level executives from IT companies who are becoming that kind of a voice as well.

That just points to the need for that ongoing conversation, the need for sharing at all levels of the buyer funnel. Once people have bought into a selection, they need to make sure of adoption, and they are maximizing the investment.

So the conversation is very important, and the immediacy of having access to folks and having the ability to exchange a few thoughts on Twitter or LinkedIn has changed the dynamic completely. So it’s absolutely about conversations and storytelling, but it's still mapped to the buyer’s funnel.

Find out what Scratch Marketing and Media

Can Do for You
Click Here to Access A Free Webinar on Market Reputation

People are still educating and still looking at options for a change or for replacement, one or the other, until they select the people they want to work with. And it’s usually people in brands. It's not just that they want to work with this company, but the people behind it. We're moving more to a people economy.

Gardner: As you point out, you can get to the real source of the knowledge nowadays. Publishing is available to anybody whether they're tweeting, blogging, posting on Facebook, or putting something up on their company website. Anybody who has something to say can say it. It can get indexed and it can be made available to anybody who wants to hear about that particular topic.

The ability to publish is great, and it democratizes the means of how we communicate with each other and educate each other, but yet you still have to earn it.

Most people now don’t just sit back and wait for information to reach them. They're proactive. They go out, they start to search, they do hashtag searches on Twitter, and they can do Google or Bing on web.

It’s much more of, "I know something; I'm putting it out there." And there's another case of someone saying, "I need to know something; I am seeking it." They come together on their own. The content makes that possible. The better the content, the better the likelihood that those in a need to know and those in a need to tell come together.

Kratchounova: Exactly, but I think you hit on something very important. Everybody can publish, and a lot of people are publishing. Yet, we're interested in a love for your people, falling in love for your people, and what they have to say.

The ability to publish is great, and it democratizes the means of how we communicate with each other and educate each other, but yet you still have to earn it. This is very important. People who really are influential are usually domain experts and they're there to help other people. That’s the other aspect of it that both companies and their marketing teams and their executives need to think about. You have to actively participate and show your expertise, it doesn’t come for granted.

Important of curation

Gardner: And there's another aspect to greasing the skids between the knowledge and the acquirer of the knowledge, and that is content curation. There are people who point at things, give it credence, and say that it's a good thing, you should read it; or that’s a bad thing, don’t waste your time -- and that helps refine this.

Kratchounova: It’s pretty exciting.

Gardner: There are machines doing the same thing. There are algorithms, there's indexing, there's both human and machine aspects of winnowing down the good stuff and providing it to people in a need to know, and that’s when we are going to get more powerful.

Kratchounova: Great. I'm sure you know about Narrative Science. I've had a professional crush on this company for few years now. They take data, turn it into storytelling, and they think this is phenomenal. Obviously, that’s not going to replace some of the human storytelling that needs to happen, but some of the data storytelling will come from technology. This is one particular application where marketing and technology come together to bring something completely new into life.

Gardner: So we can get knowledge through expertise or we can get knowledge through experience, someone who has gone through it already and is willing to share that with you. If you're acquiring IT, it’s super important to avail yourself of everything, because it changes so rapidly and the costs are high.

IT depends on the IT buyer, because we can’t necessarily lump them together and ask how the IT buyer goes about it. There are people with different needs, and it depends on their role.

If you make a big mistake in how you're designing a data center, you're out millions of dollars, your products don’t work, and your front office are going to come screaming down on you. You have to make the big decisions and you have to make them correctly in IT. It’s not just a service to the business; it is the business.

So, let’s think about the IT industry in particular, and then think about how content marketing as we’ve discussed is powerful. How do IT people acquire content marketing? Do they get it through websites, emails, or tweets? Is it delivered to them at a webinar that they opt into? How does content marketing reach somebody who's an IT buyer?

Kratchounova: IT depends on the IT buyer, because we can’t necessarily lump them together and ask how the IT buyer goes about it. There are people with different needs, and it depends on their role. If you're CIO or CTO, there is a different mix of channels and sources you use. If you're on the dev or on the ops side and looking for specific solutions, you're going into completely different channels.

For example, if you're a DevOps professional, you're maybe on Stack Overflow and you might be seeking advice from other folks. You might be on GitHub and sharing open-source code and getting feedback on that.

If you're a CIO or CTO, what we have found working with number of different companies, be that global companies or maybe companies that are growing, is that they do seek their peers to validate what the peers are going through. One of the best things that companies can do, when they try to talk to the C-level, is expose some of those connections that they already have from their customers. Make sure that the customers are part of the discussion, and they can chime in.

Another important source of information for the C level in IT would be folks like you, analysts, and strategic system integrators like Accenture and Deloitte, because these folks are exposed to the kinds of challenges that a CIO or CTO would go through. So they have a lot to bring to the table in terms of risk mitigation, optimal deployment, and maximization of the investment in IT. Making those connections and sharing those experiences we have seen work really, really well.

Let me just throw this in as well. The other thing we have seen is that the C level is still going on Google. They're still doing the searches. We have compelling data, across the board, that in any B2B complex enterprise environment folks are self-educating as well. So it’s not a question of either/or; it’s what’s the right mix for each company depending on channels, depending on where people sit.

Spectrum of content

Gardner: So there is a spectrum of content, some highly technical and defined, on places like GitHub that are germane to a technologist. Then, there is that spectrum up from there to a higher level toward peer review of products and peer review of solutions. Then, there are more business topics about what is strategic, what’s the forward direction, how do I understand at an architectural-level decision processes, and where can I go for more information to find out what’s coming down the pike and then put it in place.

Kratchounova: Think about Spiceworks. They're probably at five million IT professionals at this point, and the community is there for a reason. So again, with each particular, there isn’t one size fits all. One thing that we always recommend to folks is that if you’re looking to develop an influential strategy and approach IT, it really depends on what domains you span.

You find that even if you're doing mobile application development, the folks who were really influential and set the standards of that stage are somewhat different from the folks who are concerned with security in mobile app development. So there isn’t necessarily one pool of influencers that you need to go then to develop a relationship and understand what’s in their mind. It really depends on your domain.

Gardner: So if you're a marketer and you recognize that quality content is super important, you need to have a spectrum of content. It needs to be some content that would be germane to a technologist that’s highly detailed, a how-to type. You need to have peer review and stories, case studies, testimonial type content where the customer is telling what they’ve done, why it benefited them, and what you can learn from that.

You also need to have higher-level discussions with experts to help people chart the next course, the strategic level. So content needs to come across a spectrum, and we recognize that the way in which people get that content might be through search. It might be through web, e-mail, webinars, webcasts, reading certain online sites, listening to certain Twitter feeds or groups, or having a select group of people that you follow. All of that happens.

But what’s interesting to me, Lora, is that all has to do with the web. But what we're seeing in IT is a rapid movement toward mobile apps, rather than just the web. And in many cases, they're starting to overtake the web as to where people spend their time. I'm sure you're using a smartphone and you have mobile apps. You're not going on the web to find a cab; you’re going to the Uber app to find a cab.

If you're looking for a restaurant review, you’re not necessarily going on the web and doing a search. You’re going into a specific app on Yelp, OpenTable, or somewhere else to find out where your restaurants are and you’re going into Google Maps to find out how to get there.

Find out what Scratch Marketing and Media

Can Do for You
Click Here to Access A Free Webinar on Market Reputation

So more-and-more, we're seeing, on the consumer side, people using mobile apps for more of their processes, for their inquiry, for their actual productivity. Then, on the enterprise side, the business-to-employee (B2E) side, we're seeing people using cloud services.

We're moving more toward mobile applications, cloud services, an API-driven world that leverages big data and analytics in order to put context into process. It's all about user experiences, and mobile delivers the best. How then does content continue to reach people? Do we lose the ability to deliver content when they are in apps?

Different perspective

Kratchounova: I have a different perspective on what you're describing. I don’t know that we are moving to a mobile app experience necessarily. When we think about the apps and the examples you gave -- Yelp or Uber -- yes, they're best-of-breed applications that we use because these are the most frequently used applications.

But what you're seeing is actually a digital transformation. Digital no longer means the web, as we know it, going online through your computer. You're actually navigating on a mobile device. So it’s this digital transformation that’s happening, and the trend that we're seeing is aggregation.

It’s not about one individual app, but it’s more about what is the Flipboard within the enterprise. You're seeing that sort of aggregation bubbling up to the top because information overload is a huge problem. People can’t prioritize anymore. They can’t toggle among those different applications and companies.

For example, one of our clients, not to necessarily add a plug for them, actually is very germane to the discussion. does exactly that.

Once you understand, then you understand what a partner is trying to do. Why are they are here, what’s the context, what’s the most logical next step or the optimal next step?

In those kinds of environments, what we're finding and where I totally agree with you, is the ability to read and understand context, so that you can support the user, be that an employee with internal work experience, or external customers, to support them to get the job done.

The role of content is actually merging with big data, because big data is helping us to understand context and say, "What do we serve this person here?" On the marketing side, and the lingo side it’s more about ongoing customer journeys. Think about the same thing on the employee side, ongoing employee journeys or partner journeys.

Once you understand, then you understand what a partner is trying to do. Why are they are here, what’s the context, what’s the most logical next step or the optimal next step? Now, content becomes both an ability for people to find something, but also for marketers or product development folks. I think those functions are emerging as well to deliver the right content in the right format so that the user can get the job done. That’s my perspective on that.

Gardner: There's no disagreement from me on this issue of context to process, context to location, context to need for knowledge all being much more granular and powerful going forward. What I am concerned about is that, when I talk to developers, the vast majority of them are much more interested in a mobile-first, cloud-first world.

They're not much interested in building what we used to think of as big honking applications in the enterprise. They're much more interested in how to bring services -- and microservices -- together in context to provide a better productive outcome and how to leverage low-cost services in APIs and from any cloud.

Discovering inference

So, to me, it becomes, on one hand, all the more important to have the ability to deliver content contextually into these processes, but at the same time these processes are becoming fragmented. They're going across hybrid-cloud environments, they include both what we call cloud and SaaS, and I'm not sure where the marketer now can get enough inference to support the injection of content appropriately.

The ways that it’s been done now is usually through the web where we have links, and we have code, and we can do cookies. It’s sort of like, it’s Web 1.0 mechanisms by which marketers are injecting content, but we are moving not only pass Web 2.0, we're into Web 3.0  cloud platform. To me this is a big question mark.

Kratchounova: It is a question mark. I don’t know that there is going to be one mode of delivering what we're talking about or one approach or one framework. I'll give you one example. Look at how web content management has changed. It used to be about managing pages and updating content. Now, web content management is becoming the Marketing Command Center, if you look at a web content management system like Sitefinity, for example.

Now, marketers can deal with the customer through his own mobile and on the web, so they can inject the content that needs to happen there. The reason they can do this now is because there is this ability, the analytics that come from all of these customer interactions of you, actually creating cohorts of people as they're going through your web experience or online experience. You know why they're there and what’s the optimal path for them to get where they need to be.

You're seeing this ability to distribute content to post content to people, but in a much more contextual way. So, there is going to be a pull and push, but the push is getting a lot smarter and very contextual.

So, you're seeing this ability to distribute content to post content to people, but in a much more contextual way. So, there is going to be a pull and push, but the push is getting a lot smarter and very contextual.

Gardner: So it’s incumbent upon us who are examining this marketing evolution in the context of the IT industry to create that spectrum of content to make it valuable, to make it appropriate and not too commercial or crass, but useful. And at the same time now, think about how to get this in front of right people at the right time.

It seems to me that if I'm an IT company, and more and more of my services, whether it’s a B2B, B2C, B2E, or all of the above, I need to be thinking about ways that I'm going to communicate with my existing universe or market and move them toward new products and services as they need them in context of their process.

Think about this in a B2C environment in retail, where I am walking through Wal-Mart. I have my smartphone and, as I turn the corner, they know that now I am interested in home goods, and they are going to start to incentivize me to buy something. That’s kind of an understood mechanism by which my location and the fact that I turned a corner and made a decision provides an inference that then they can react to with content or information.

But take that now to the B2B environment where I'm in a business setting. I'm in procurement, I'm in product development, or I'm looking for a supply chain efficiency. I want to move into a new geographic location and I need to find the means to do that. All of those things are also like turning a corner in a Wal-Mart, except you're in a business application using cloud services, using a mobile device and apps.

If I'm an IT vendor, I'm going to want to have content or information that I can bring to that situation, perhaps even through an example of what other people have done when they face that same process crossroads. So the content can be more important and more germane. These are multi-million-dollar decisions in some cases.

Don’t you think that big companies should be starting to make content with the idea that it’s going to become part of their application services, part of their cloud delivery services, and that they need to use big data and analytics to know when to inject it?

Understanding context

Kratchounova: I absolutely agree. I think that difference between the example you just gave for Wal-Mart and a B2B environment is that, in Wal-Mart, you don’t need to understand so much about who the person is, what their role is, whether they work at an accounting firm or whether they are a physician, for example.

In a B2B environment you do need to understand context, and context is the location or the point where they are in their journey, whatever that journey maybe, and their role as well, because different people do have different decisions to make.

It’s a little bit more complex to bring context in a B2B environment, but it’s absolutely essential. You used the word inference. We always get enamored by the concept of the big data and guess what, once the machines are there, they're going to analyze everything and it's going to be this perfect world of marketing where everyone is aligned. 

Just look at the history of marketing. We don’t know ourselves as people. We individually don’t know ourselves as well, let alone someone else getting to know us that well. Inference is very important, but it’s going to be a balance between inferring what the person needs and allowing the person to customize this experience as well. So it’s going to come both ways.

Some people still believe that it’s a relationship-based world and, therefore, there's no need for a digital experience for their customers or for their potential buyers, which is actually never the case.

Some people going to one extreme or the other. Some people still believe that it’s a relationship-based world and, therefore, there's no need for a digital experience for their customers or for their potential buyers, which is actually never the case. Other people believe that it’s all digital; therefore they don’t need to touch them in any other way, which is rarely the case, especially in IT. 

Gardner: I also suggest to you that the data is more readily available, because I, as an employer, as a corporation, control what’s going on. I know what that employee is doing. I know what apps they're using. I know what data they're seeking. 

They're going to provide a feed of data back to you about what’s going on, on those apps from your very own employees.

What I'm suggesting then, as we begin to think about closing out this fascinating conversation, is that you need to have content, stories, and customers lined up, so that you can uncover their path to truth, their path to value, and have that content context-ready. Not only you are going to be using it in webinars, webcasts, podcasts, blogs, but pretty soon, if my hypothesis is correct, you're going to be using that content in the context of process and inside of applications in cloud services and on mobile devices.

Way of the future

Kratchounova: Maybe this is an opportunity, because it is the way of the future, and some people are more mature and others are less mature, but maybe we can bring other people into the discussion and see what other folks in the field think about where the content is going, how to contextualize and how to deliver it. One of the biggest question is how do we scale this. You can still do a meaningful experience or create a meaningful experience one-on-one, but it’s hard to recreate that even if your customers are 200, 500, or even 5,000 within the IT space. 

Gardner: You also have to remember that people's connections to apps, cloud services and context-aware processes are only going to increase. The Internet of Things and new classes of devices like the Apple Watch are expanding the end points and ways to connect to them. One of the things that’s important with the Apple Watch functionally is that it’s very good at alerts and notifications. It can also detect a lot of context of what you're doing physically and your location, and it can relate, because it integrates to your phone, with what you're doing with applications and cloud services.

Wouldn’t it be interesting if you're wearing an Apple Watch or equivalent, you're in a business setting, and you come up against a problem that you might not even know yet, but all of these services working together are going to say, "That person is going to be facing a problem; they are going to need to make a decision. Let’s put some information, content, and use cases together for them that will help them as they face that situation to make a better decision." That’s the kind of role I think we're heading toward. 

Before we sign off, Lora, tell me more about Scratch Marketing and Media, what you do and why that’s related to this discussion we have had today.

Find out what Scratch Marketing and Media

Can Do for You
Click Here to Access A Free Webinar on Market Reputation

Kratchounova: Scratch Marketing and Media is an integrated marketing agency. We help B2B technology companies with market growth. Sometimes that means helping the sales folks within IT companies and sometimes it means working with the marketing folks on things like content marketing programs, PR, and all its relations, and influence their relations in social media.

Gardner: And how could they find out more information about Scratch Marketing Media?

Kratchounova: You can go online at

Listen to the podcast. Find it on iTunes. Get the mobile app for iOS. Read a full transcript or download a copy.

You may also be interested in:

Tags:  BriefingsDirect  content marketing  Dana Gardner  Interarbor Solutions  Lora Kratchounova  marketing  mobile computing  Scratch Marketing and Media 

Share |
PermalinkComments (0)
Page 1 of 62
1  |  2  |  3  |  4  |  5  |  6  >   >>   >| 
Page Title
Association Management Software Powered by®  ::  Legal