It's Dark Outside
1. What is Dark Data? Have you ever wondered about all those notes you’ve made, all those spreadsheets you’ve created, all those documents that you’ve written over your lifetime? Do you know how many you have? Thousands, I’d bet. 178,284, in my case. And have you ever wondered what insights all that collective data might give you if only you could ask it the right questions? How to count your office and text files find <path-to-files> -type f \( \ -iname '*.doc' -o -iname '*.docx' -o \ -iname '*.xls' -o -iname '*.xlsx' -o \ -iname '*.ppt' -o -iname '*.pptx' -o \ -iname '*.odp' -o -iname '*.pdf' -o \ -iname '*.eml' -o -iname '*.msg' -o \ -iname '*.adoc' -o -iname '*.md' -o \ -iname '*.rtf' -o -iname '*.txt' -o \ -iname '*.text' -o -iname '*.csv' \ \) | wc -l "What problem do I keep solving for customers?" "What types of challenges typically expand beyond their initial scope?" "What am I most commonly asked to do?" In my case, apparently, the answers were: Enterprise integration, organisational change and risk mitigation Infrastructure migration, security compliance projects, and any pre-sale engagement Translate technical complexity into business value Think how that power could benefit you or your business to target customer needs, put the right people in the right place, and streamline and improve efficiency for the work you deliver. Now, imagine those 178,284 files multiplied by all the people in your organisation. What a wealth of information that would be. This is an example of dark data, the information a company collects during its normal business operations but isn’t part of its analytics or decision-making. Instead, it’s hidden away, collecting dust. And it’s not just the unstructured notes of colleagues on your file shares; it can be unused database records, the contents of your email servers, or historical metadata stored in logs. It’s not just insights that you or your business can benefit from by mining your dark data. Think about the security risks you could uncover because the business is unaware of all the sensitive data it holds or, by the same measure, whether you could be out of compliance. Also, wouldn’t it be great to archive that data if it holds no value and reclaim some of those ever-increasing storage costs? These are questions smart companies are looking to artificial intelligence to help answer, and I decided to try and illuminate my own dark data to tackle a pressing problem of my own. ...