Too much CRAP – Big Data is here to stay

Too much CRAP – Big Data is here to stay

All the data.
All the time.

That pretty much sums up The Age of Big Data in which we now find ourselves. Organizations that learn how to make use of the data will prosper.
There are datastreams being generated by all the ways we are connected to the internet.

This data is being collected and stored, although there is too much of it to organize, so it’s CRAP (create, replicate, append, process) as Charles Fan calls it; “generated by machines, coming in large quantities at high velocity.”

The solution? Easy. Make the data accessible in a way that gives it structure. That’s where we come in.
 

So the great challenge is to use this data. If the data is not used then, there is little point in storing it, especially at cost.

Q: What should (big) data be used for?
A: For making decisions.

In other words, why bother to save all that data, unless to put it to good use, and that means helping organizations make decisions. Those decisions can be based directly on human behavior, for example to decide what content to display to a viewer, based on Social Media preferences, or based on machine signals.

Automated signals, such as all the purchases made by supermarket customers with bonus cards, which can help managers decide which products to stock & display, create datasets too large for humans to manage.

That’s where solutions such as Opentracker come in. You don’t need a big engineering team to get value out of the data if you use a suitable data collection and reporting system.

In our case, we will:

  • provide the engineering
  • host the data
  • provide a simple interface
  • allow the end-user to send signals (write)
  • and ask the data questions (read)

Click here for an article about understanding Big Data.

Because there are new and massive databases coming into existence, new technologies are needed in order to manage the data. The point of efficient data management is that it takes processing power to make calculations. Therefore, the new & efficient way which has evolved is to leave the data as it is; CRAP, and sort it out only on a need-to-know-basis.

That’s where Opentracker comes in. What we have built is a distributed-data technology that allows users to slice & dice at will.

In other words, you send the data, it’s stored, but not structured. The only thing that is structured is the tool (an API: application program interface) that allows you to ask the data questions.

Please get in touch if you have any questions.

 

PS- And why is Big Data not going away? Because with ever-increasing amounts of data, the likelihood of the data being organized in any meaningful way becomes increaingly smaller.

Other blog posts

Start your free, no-risk, 4 week trial!

Top Sites feature in Safari loads website repeatedly

Top Sites feature in Safari loads website repeatedly

Top Sites feature in Safari generates pageview(s) by repeatedly loading indexed sites

We’ve noticed that the Top Sites feature in Safari sometimes generates a pageview when Top Sites are displayed. We confirmed this by noticing that we were “generating” visits and pageviews (clicks) to a site that we were not visiting.

The explanation for the reason that our profile kept showing up in the stats is that the site was being disaplyed repeatedly in Top Sites.

This is potentially problematic because it might give the false impression that a person was visiting on a regular basis.
We assume that this happens when Safari generates the Webpage Preview Images.
Definition of Top Sites from Apple website:

Top Sites: Safari automatically identifies your favorite sites and displays them as a wall of graphic previews. To visit one of your top sites, just click any of the previews.

Other blog posts

Start your free, no-risk, 4 week trial!

Total Engagement in a Nutshell

Total Engagement in a Nutshell

Today we officially released Total Engagement. You can read the press release here.total engagement mobile app and web analytics

Q: What is Total Engagement? Give me the elevator pitch.
A: Our Total Engagement release is comprised of two parts;

i) We now support analytics for Mobile Apps – meaning that we collect and report app user events
ii) we combine app usage measurement with traditional web reporting. As far as we know, we are the only company on the market who can display app user eventstreams and website visitor clickstreams in the same report on the same page.

Q: What does that actually mean?
A: That means that if you make or publish apps and websites you can see all activity in one place, in realtime.

Q: How cool is that?
A: Very cool!

Q: Do you have a PDF that explains all that to me with visuals so that I can read it for myself?
A: Yes, click here.

Other blog posts

Start your free, no-risk, 4 week trial!

Trendspotting

Trendspotting 2014

How will your business respond
to the increase in available digital data?

With increasing data available, the winners will be those who understand the possibilities. Businesses are seeing that, increasingly, there is revenue to be earned through the intelligent use of data.

Trendwatching. We live in a time of great technical change. Big things are happening, things which can have enormous impact on products, development, and revenue models.

The question to ask is: what are important trends in business sectors that will drastically, disruptively, affect bottom lines?

Trends are important, as are people who have the ability to observe and understand trends. What trends will affect your business or sector?
What are we talking about? Here are a few big picture examples:

  1. The energy industry: people putting energy into the grid, as opposed to taking it off. This is increasing. Energy companies need to plan for people creating their own energy. In Germany more than 50% of electricity is currently produced by alternative power sources.
  2. Eastman-Kodak: creation of digital film led, ultimately, to their bankruptcy and reorganization.
  3. Nokia: trend of personal handheld computers (devices, iphones) led to their loss of leading market position.
  4. Banks: their position challenged by Bitcoin and crowdsourcing – will we still need banks? With the rise of crowdfunding, banks may need to rethink their position.

Q: What is the take home message? 
A: Companies ignore upcoming trends at their own peril. Plan to succeed.

 

2014: year of the Chief Digital Officer

In 2014 an increasing number of businesses are expected to appoint CDOs – Digital content chiefs; in short, somebody responsible for all the data.

Unforeseen trends and opportunities for profit (or loss) are the reason why there is such a high demand for people who can analyse data. Human resource managers are also seeing the shift; a need for people who can analyse data, and a new culture is emerging, whereby in-house knowledge is needed. Is your company making adequate use of the available data?

Opentracker. Don’t get left behind.

Other blog posts

Start your free, no-risk, 4 week trial!

Unlocking the value of data

Unlocking the value of data

Infinitely scalable read/write real-time data management

A recent article post on the BBC could have been written about Opentracker and the technology we have spent the past year developing. The piece, entitled ‘Data wars: Unlocking the data gold mine’ was written by Mike Lynch and ran on 12 April, 2012.

The premise of the article is that businesses and organizations wishing to stay ahead of the curve can do so by learning how to use technology in order to give meaning to the data they collect. Traditionally, data has been stored in databases and only later, if ever, consulted. With technology, it is now possible to consult data in real-time and make decisions based on what we learn.

“The availability of vast quantities of data on current or potential customers on e-commerce sites offers online retailers huge value – if they can gather and analyse this information efficiently and in a timely manner”

– a quote from the article which tells us about the potential value of (large scale) data. There is a catch: the information collected needs to be analyzed and understood today, as opposed to tomorrow.

Additional points;

  • Online customers generate a lot of data (data trails) which, combined with social media targeting, can be used to add value, and generate leads and sales
  • In order to be effective, decisions based on this data should be based on realtime reporting
  • This is because the information loses its value (quickly) and so should be used as quickly as possible
  • Keeping up with the demands of changing customers requires a structural solution ( like real-time data utilization)
  • This is a question of scale necessitated by the current internet environment and the volume of customers (with millions of associated data points) that many websites currently manage

The key is to understand the data – leaving it exactly where it is – in order to create an infinitely scalable platform, and a powerful basis for analysis and action…and get to the heart of the issue – being able to process 100% of the information, structured and unstructured, to unlock real business value.

– quote from the article’s conclusion.

The main reason that this article caught our attention is because, as mentioned above, we spent the last year or so building exactly what the author describes: an infinitely scalable read/write real-time data management solution. Not only that, but as of yesterday, it runs cross-platform, meaning that we can present universal activity across apps and websites.

Q: What is the goal?
A: Given the importance of understanding user audience and being able to react in a timely fashion, Opentracker’s goal is to provide a realtime solution which is easy to read and reports on Total Engagement across apps and websites.

Q: How cool is that?
A: Very cool!

Other blog posts

Start your free, no-risk, 4 week trial!

Upgrading from keystore/keytool to Tomcat Native APR with OpenSSL

Upgrading from keystore/keytool to Tomcat Native APR with OpenSSL

Today i had to upgrade a tomcat server that used a keystore file for SSL, to a native tomcat 6 with APR and OpenSSL. I thought this was easy until i realized i didn’t had the original private key anymore of when the certificate was created. The steps we took with Keytool to generate the the original certificate request, do not save the private key as a separate file.. After some digging around i found this nice java app called Portecle that does the trick: it opens your keystore and allows you to export the private key. It even lets you remove the encryption/password. After that it was easy: saving the private key and issued certificate in a separate folder, changing the server.xml to this:
<Connector port="443" maxHttpHeaderSize="8192"
maxThreads="150"
enableLookups="false" disableUploadTimeout="true"
acceptCount="100" scheme="https" secure="true"
SSLEnabled="true"
SSLCertificateFile="/ssl/wildcard.crt"
SSLCertificateKeyFile="/ssl/private.key"
/>
and voila:
Feb 9, 2010 4:04:27 PM org.apache.catalina.core.AprLifecycleListener init
INFO: Loaded APR based Apache Tomcat Native library 1.1.19.
Feb 9, 2010 4:04:27 PM org.apache.catalina.core.AprLifecycleListener init
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
Feb 9, 2010 4:04:27 PM org.apache.coyote.http11.Http11AprProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-80
Feb 9, 2010 4:04:27 PM org.apache.coyote.http11.Http11AprProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-443
Feb 9, 2010 4:04:27 PM org.apache.coyote.ajp.AjpAprProtocol init
INFO: Initializing Coyote AJP/1.3 on ajp-8009
Feb 9, 2010 4:04:27 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 525 ms
Feb 9, 2010 4:04:27 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Feb 9, 2010 4:04:27 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.24l
Feb 9, 2010 4:04:28 PM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor ROOT.xml
Feb 9, 2010 4:04:28 PM org.apache.coyote.http11.Http11AprProtocol start
INFO: Starting Coyote HTTP/1.1 on http-80
Feb 9, 2010 4:04:28 PM org.apache.coyote.http11.Http11AprProtocol start
INFO: Starting Coyote HTTP/1.1 on http-443
Feb 9, 2010 4:04:28 PM org.apache.coyote.ajp.AjpAprProtocol start
INFO: Starting Coyote AJP/1.3 on ajp-8009
Feb 9, 2010 4:04:28 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 953 ms
Thank you Portecle !!

Other blog posts

Start your free, no-risk, 4 week trial!

Location ‘Unknown’ significantly reduced – accuracy increased to 97%

Location ‘Unknown’ significantly reduced – accuracy increased to 97%

Accuracy of visitor location dramatically increased to 97%

With the addition of a second location database we have effectively reduced the number of occurrences of Location Unknown. A failover component has been introduced.

Until today, when we identified a visitor or user, there was a 17% chance of an inaccurate, outdated, or unknown classification, meaning an accuracy of 83%.

The margin of error has effectively been reduced to 3%, with the addition of a second database to improve upon our geo-location identification technology.

This development pertains to location of website visitors based on IP address assignment.

Other blog posts

Start your free, no-risk, 4 week trial!

EU Postal Codes added

EU Postal Codes added

As of June, 2013, we’ve added Postal Code (ZIP code) database information for six EU European Countries;

  • Germany
  • France
  • Sweden
  • Switzerland
  • Finland
  • The Netherlands

More countries can be implemented, please contact us if you require this enhancement.

These new databases have been placed in production and result in increased address location functionality.
Improved EU postal database identification means that now, alongside United States and Canada, postal and ZIP code information is displayed for every visit/ visitor from any of the named countries.

This is helpful with identifying visitor location and using the data to create geographically-oriented marketing efforts.

Q: What is this information for?
A: This data can be used for grouping of visitors within/ by postal code, for administrative, lead generation, sales mapping, marketing, geo-targeting, sales team responsibility for delineation of location, etc.

Case Study: For example, a company doing business in Europe may assign sales teams the task of responsibility per region. In the case where ZIP/ Postal codes are used to divide regions between efforts, teams are now able to identify and capture leads based on geography, which reflects how the teams actually work.

Other blog posts

Start your free, no-risk, 4 week trial!

CAO: Chief Analytics Officer

CAO: Chief Analytics Officer

There is a new CXO in town  … called the Chief Analytics Officer!

Authored by Tomas Higbey

This is just to get you thinking about what’s coming in the next couple of years so that when the CAO shows up in town, you’ll recognize that person and be able to take him or her on.

Just as Big Data is inevitable, so may be the creation of the Chief Analytics Officer or CAO. Companies that are neck-deep in Big Data wrestle with the challenges of multiple data-bases, overwhelming data storage issues and exponential data flows.  In the enterprise sector current trends tell us that individual departments within organizations are creating their own Big Data solutions and analysis. Infighting typically occurs and resources are wasted through the lack of a comprehensive and centralized plan. The smart pioneers that “get it” right from the start, and influence the leaders at the C level will have a major competitive advantage while the competition learns the hard way, wasting internal political capital in the process.

The CAO can create the Big Picture design on Big Data for the entire organization, find which tools work, what projects have performed best, which data sources can be trusted, and measure both the entire ROI for the company as well as rank and rate individual projects.

The reality is that Big Data is showing up in nearly every industry and in every department; in productivity, innovation, competition, and market knowledge. In other words Big Data is being used in Marketing, Operations, Human Resources, Product Development and Finance. There is no place within the organization where it isn’t being used.

Demand for Data Scientists and certified professionals will also increase. A CAO can create the Big Strategy to mitigate the foreseeable challenges and simultaneously keep Big Data’s importance in the minds of the entire leadership in the organization.  No doubt the elite companies are already beginning the planning and even implementation stages for the unavoidable future of Big Data.

But what does this really mean? We start with the issues of Big Data.  Big Data at the core is about the “art of the possible”. For the first time, even small companies have a fishing line, a fishing net and various tools to dive into the information ocean and extract all types of specific “species” that are most important for the business.  By recognizing patterns that would otherwise remain hidden, correlations can be identified. Think: identifying successful patterns.

Data scientists have discovered that orange painted cars are in better shape than other cars.  The “why” is not clear but that doesn’t really matter. When seeing it from the Big Data perspective, we can use this as a guide for buying used cars. This is a balancing between “intuition” and instinct (which are often wrong) the potential risk of which can be curtailed and mitigated by Big Data.

Here are the key reasons why the CAO might be critical for Big Data initiatives:

  • Revealing hidden strategies over-looked by departments
  • Educate organization (not just C level) about BD’s critical utility
  • Organized best talent and practices while attracting the best people from a scarce hiring pool
  • Data trumps intuition, meaning a new leadership philosophy is being born, one which must be evangelized within the organization
  • Establish standardization of reports / analytic models across the organization
  • Establish what is and isn’t possible from the Big Data perspective
  • Leverage Big data as a key asset / resource for everyone

To repeat: this is just to get you thinking about what is coming in the next couple of years so that when the CAO shows up in town, you’ll recognize him and be able to take him on!

Other blog posts

Start your free, no-risk, 4 week trial!

Mobile Version (Web App) Released

Mobile Version (Web App) Released

Mobile optimized edition of Opentracker now available.

We are pleased to announce that a mobile version of our reporting dashboard has been released and is now supported on mobile devices.

 

We have started by releasing the dashboard as the first report optimized for mobile devices. We are currently planning the next feature to enable.

What features should we add next? Please write and tell us what functionality you would like to have on your mobile.

In other words, what would be the best piece of information to access when you are on-the-go, at a conference, or any remote location. Need to check your Online & Recent Visits report when you are standing in line at the post office? The web doesn’t sleep, why should you?

 

 

 

Other blog posts

Start your free, no-risk, 4 week trial!