1. What is Dark Data?
Have you ever wondered about all those notes you’ve made, all those spreadsheets you’ve created, all those documents that you’ve written over your lifetime?
Thousands, I’d bet. 178,284, in my case.
And have you ever wondered what insights all that collective data might give you if only you could ask it the right questions?
"What problem do I keep solving for customers?"
"What types of challenges typically expand beyond their initial scope?"
"What am I asked to do, most commonly?"
In my case, apparently, the answers were:
Enterprise integration, organisational change and risk mitigation
Infrastructure migration, security compliance projects and any pre-sale engagement
Translate technical complexity into business value
Think how that power could benefit you or your business to target customer needs, put the right people in the right place, and streamline and improve efficiency for the work you deliver.
Now, imagine those 178,284 files multiplied by all the people in your organisation. What a wealth of information that would be.
This is an example of dark data, the information a company collects during its normal business operations but isn’t part of its analytics or decision-making, hidden away, collecting dust. And it’s not just the unstructured notes of colleagues on your file shares; it can be unused database records, the contents of your email servers, or historical metadata stored in logs.
It’s not just insights that you or your business can benefit from by mining your dark data. Think about the security risks you could uncover because the business is unaware of all the data it holds or, by the same measure, whether you could be out of compliance. Also, wouldn’t it be great to archive that data if it holds no value and reclaim some of those ever-increasing storage costs?
These are questions smart companies are looking to artificial intelligence to help answer, and recently I decided to try and illuminate my own dark data to tackle a pressing problem of my own.
2. Mining the Past - A Personal Usecase
Recently, I found myself back in the employment market.
It’s a different world from when I was a lad. I’ve done so much more, accumulated so many skills and held so many different positions that it’s difficult to imagine how to encompass all of that in a single résumé and a cover letter that would lead a prospective employer to see how perfect I may be for their role.
Further, the progression of online recruitment means that the employment marketplace is considerably larger than when job applications were solicited in magazines and newspapers and submitted by post. With potentially hundreds of opportunities that match my skills on any given day, sifting and applying for the right ones may no longer be something one person can achieve in a timely fashion.
It seemed that my own dark data combined with data mining and generative AI might be able to help.
"Well," I thought. "Perhaps I should apply the same approach to mine my dark data to help my own career aspirations?"
2.1. Where’s the Gold?
Over the years, I accumulated a lot of project documentation, emails, and technical notes saved to disk. So, I decided to build a local offline pipeline that ingests documents, chunks them into evidence-sized pieces, stores them in a SQLite database, and enriches them using local large language models (running on Ollama) to tag customers, projects, technologies, and outcomes. After running this over my source files, I could generate structured artefacts that transformed all of this higgledy-piggledy data into evidence-backed dossiers concentrating on customers I delivered solutions for, the projects I worked on, and the skills I accumulated.
I had some clear security concerns when considering this project. Even as a private citizen, my notes contained potentially sensitive information that I had no wish to see become part of a corporate AI’s next training cycle. So, it was important for me that the dissection and tagging of all this information never left my computer. To make sure this happened, I chose local LLMs to tag the data after it had been read into the database. Whilst this took considerably longer than using an online frontier model, I was pleased to discover that it could be achieved reliably with a comparatively small model such as qwen2.5:7b on a home server, even if it had to run overnight.
From here, I hope you can start to see how the skills and experience required in a job description could now be matched to real-world examples, making the path to writing a tailored curriculum vitae (CV) clear.
3. Forging the Future
Finding a new role means that you have first to learn some new roles yourself: job seeker, résumé specialist, recruitment consultant, interview coach, and ATS [2] specialist, amongst others. That’s a lot of work for one person, and nobody else was showing up to help me out. However, in this brave new world, this matters less; if you needed people to help you, you build them, not out of clay but from Markdown.
I constructed an agentic pipeline consisting of specialist agents governed by a team lead, an orchestrator coordinating three related workflows.
- Job Application
Specialist agents researched the employer and likely salary, checked data-mined evidence and qualifications against the job description, assessed fit to ferret out jobs I was not suited for, and built a rationale to link my proven skills to the employer’s requirements.
Coded guidance including a fixed work history, writing guide and CV formatting rules prevented the agents from hallucinating timelines or inventing employers and to keep their output format consistent with recruiting best practice.
A writing agent produced a CV and cover letter tailored for the role that could be submitted to the employer’s ATS [2] system.
- Interview
The same evidence could be combined with public information and notes from earlier conversations to create structured discussion, technical, and behavioural insights, each tailored to give an informed advantage for the interview based on the company, job role and my skills & experiences.
- Government Work Search Compliance
Finally (and this was unrelated to dark data but very relevant for the unemployed), each job-search activity, from application preparation through submission, interviews and the eventual outcome, got recorded as a separate dated entry in work-search logs required by the government should you be claiming an employment benefit.
The outcome of this exercise proved to be one of the most revealing and beneficial pieces of work I have undertaken in terms of the rediscovery of my own value, as well as allowing me to compete in an saturated employment marketplace where the electronic gatekeepers are strongly biased against all but the closest matching applicants.
The tool reminded me of my deep skills in ITIL [3], project management and disaster recovery, and of my work in asset management in the nuclear and energy industries. Whilst I’ve worked in financial markets the most it found I have years of experience in aerospace, GIS [4], gaming and entertainment, and many more.
These insights have allowed me to widen my net in terms of the roles I have felt confident applying for, knowing that I could attend interviews with strong, evidence-backed stories of how these experiences and skills could benefit future employers. The pipeline also built ATS-targeted documentation that was more likely to match with the ATS filters and land on the hiring manager’s desk and was instrumental in lining up more interviews than I could have achieved on my own.
4. All that Glisters … Did it work for me, could it work for you?
And so, having spent months building my mining operation, did I find gold? Did my dark data ultimately yield value?
I think so.
I had an immediate need and a practical problem to solve: how could I represent decades of experience in a modern employment marketplace? In the process of addressing that need, I unexpectedly rediscovered experiences and capabilities I had largely forgotten I had. These insights had real, tangible value, allowing me to target opportunities supported by well-documented successes, broadening my portfolio and appeal to a wider market. Just as a business could use dark data to support its business development, resource planning and skills assessments, I applied the same principles to my own career.
Of course, not everything that glisters is gold. AI is perfectly capable of producing convincing nonsense fuelled by decades of accretion: there is a lot of digital detritus in them there hills. Distinguishing real value from noise and ensuring that whatever conclusions the models reach can be traced back to something real is the trick. That’s why deterministic retrieval, keeping the original evidence accessible and, of course, human oversight are so important to realising the value.
Equally, this project took time, time that wasn’t finding a new role, that wasn’t "generating revenue." But the dividends were greater and the result was more valuable. From a black mountain of data, it yielded a searchable, reusable body of knowledge that I can continue to interrogate and put to work.
So, think on that. My 178,284 files are a drop in the ocean of information accumulated by even a modest-sized organisation. Somewhere in those dark waters may be answers to questions nobody has thought to ask, lessons already learned but subsequently forgotten, or risks nobody realises they’re carrying.
It may not all be gold, of course, but until you cast light on the shadows, how will you know?