Better organisations use AI and data to supercharge their people, allowing them to do things faster, cheaper, smarter. What does better mean for you?

Turn data into better

Our clients and partners

We identify opportunities for AI in your business and deliver solutions which transform day-to-day operations. This is about the practical application of data platforms, data science and machine learning to create real tools you’ll use every day to improve efficiency, effectiveness and, ultimately, create more value.

Data science meets design thinking

Turning data into better solutions to real-life problems

At Datasparq, you only get the A-team. We put together unique expertise for unique challenges, with the right blend of data science, product thinking, engineering and design.

Passionate practitioners

Datasparq people are smart, curious, skilled, experienced and creative. If you have what it takes to create data solutions that aren’t like everyone else’s, we’d love to hear from you.

Inspired? Join our team

Home

Welcome to Datasparq’s technology blog. At Datasparq, we design, deliver and run bespoke data science products to help organisations capture value from data. - Medium

Datasparq Technology - Medium

Imagine you’re building thousands of rocket engines and spending billions of dollars on a quest to make humanity a multi-planetary species. If your manufacturing process isn’t as efficient as possible, you’ll run out of money before you get there. You probably need to optimise this process like crazy right? So why don’t we see if we can optimise our software in the same way…
(please ignore the fact that nobody needs to manufacture software, that’s why it’s called software not hardware, just bear with me)
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qXCFgcRXosxgEthX"><figcaption>Photo by <a href="https://unsplash.com/@spacex?utm_source=medium&amp;utm_medium=referral">SpaceX</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure>In a recent interview, Elon Musk talked about how manufacturing was underrated and was the most difficult problem SpaceX had to solve.
<blockquote>“10,000% more work goes into the production system than the thing itself.”</blockquote>
<blockquote>“The amount of effort that goes into the design rounds down to zero relative to the amount of effort that goes into the manufacturing system.”</blockquote>
He went on to describe 5 steps used to optimise engineering for manufacturing, which emerged from the many successes and mistakes of rocket programs and Teslas in the past:
<h4>1. Make the requirements less dumb</h4>
The requirements are definitely dumb; it does not matter who gave them to you. Therefore, requirements get challenged, which allows for out-of-the-box thinking (see Cybertruck).
<blockquote>“Everyone’s wrong. No matter who you are, everyone is wrong some of the time.”</blockquote>
<blockquote>“All designs are wrong, it’s just a matter of how wrong.”</blockquote>
He also states that specific people should own requirements, not departments, so that they can be questioned much quicker.
<h4>2. Try very hard to delete the part or process</h4>
You can’t afford to add a part ‘just in case’. Musk encourages deleting as many parts as much as possible, even if you end up adding some back later.
<blockquote>“If you’re not occasionally adding things back in, you’re not deleting enough.”</blockquote>
<h4>3. Simplify and optimise the design</h4>
This step comes third, not first, for good reason. The critical point of this whole process is to not do this step until you’re absolutely sure it’s needed before you start wasting your time.
<blockquote>“The most common error of a smart engineer is to optimize something that should not exist.”</blockquote>
Elon stressed the importance of an engineer understanding the whole system to avoid optimising the wrong thing. He revealed that SpaceX engineers had made the mistake of putting enormous amounts of effort into reducing the weight of engine components, but not the weight of unused fuel, despite them being equivalent.
<h4>4. Accelerate cycle time</h4>
Here’s where you start to find issues, and have a chance to fix them before it’s too late. It’s like learning to run before you run a marathon.
<blockquote>“You’re moving too slowly, go faster! But don’t go faster until you’ve worked on the other three things first.”</blockquote>
<h4>5. Automate</h4>
Why is this separate from the fourth step? Because this assumes you wait until any and all issues have been found before removing all humans from the process.
It would be hugely costly to try to automate something that was overly complicated or wasn’t needed.
This is also the point where you should aim to remove in-process testing for parts that can be produced reliably. Musk states that a tester could be creating a choke point, and will always have some have false negatives.
<blockquote>“If things are getting to end-of-line testing, and are passing, then you don’t need to do in-process testing.”</blockquote>
We can summarise this process as the following:
Design → Question → Simplify → Optimise → Accelerate → Automate
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jmz58whKh3ky9qS3"><figcaption>Photo by <a href="https://unsplash.com/@spacex?utm_source=medium&amp;utm_medium=referral">SpaceX</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>How does this relate to software?</h3>
So let’s say instead of building thousands of rockets, you’re developing the next big app/website/API. It’s a huge task. If you do everything right you’ll be bought by Google within 5 years, else you’ll take 15 years, never get feature-complete, spend all your money, and get overtaken by somebody else. How does this situation compare to SpaceX’s?
In terms of the processes and ways of thinking, there are a lot of parallels: it’s easy to get carried away solving the engineering problems and ignoring the bigger picture. Engineers tend to want to add loads of functionality that isn’t essential. They like optimise for speed and efficiency without being required to. They stick to a set of best practices and don’t usually question them.
<h4>Manufacturing ≈ Going Live</h4>
In the design stage your code only needs to work once. When you go live you’re running the same code billions of times, on millions of different computers, for years or decades. There’s a lot that could go badly: will you need constant patches, or tiny changes for every new use case? Will you need to spend forever teaching people how to use it? Will you be able to keep up with demand and scale? Only by solving these issues can you hope to actually achieve anything with your software.
Like with rockets, this is an underrated part of programming. An inexperienced software developer might assume their cobbled together script can be used in production with no changes. This is like building a rocket engine in your garage from spare parts and deciding you’re going to build 100,000 more in the exact same way.
<h4>Part = Script, Service, or Program</h4>
Cutting out an entire service from your app (e.g. deleting an email server because you can just use an API) is undeniably a great way to reduce complexity and go live faster.
<h4>
Cost = Cost?</h4>
For software, material cost is much lower, personnel cost is much higher, and there are additional computing costs. People often worry about their compute/network/storage bill, but the salaries of the people fixing bugs all day long are likely to be larger.
There’s also the opportunity cost of complexity. If a low priority feature requires a day in total to develop, deploy, debug, document, etc. over the course of a month, then you’ve missed the opportunity to start a killer feature, or present your product at a conference, or have a much needed office party!
<h4>Weight = Complexity</h4>
For rockets this doesn’t just mean weight of the part, it’s also how much thrust you can get for that weight. Adding complexity to a piece of software means more complex documentation, technical debt, bugs, deployment, etc., all of which increase costs but more importantly, slow you down and kill your momentum.
Reducing complexity allows you to go further with the same amount of resource.
<h4>Requirement = Requirement</h4>
Requirements often don’t get questioned, and they rarely have an individual owner. It’s far too easy to make assumptions about how software will be used and not seek out the opinion of an actual potential customer.
<h4>Feature = Feature</h4>
We’re all familiar with dubious feature requests. The Agile process helps to prevent low priority features from being worked on but doesn’t stop us from adding unnecessary complexity.
<h4>In-process testing = Unit tests</h4>
This one has really made me think. Elon’s right, we don’t need most of these unit tests. We’ve got hundreds that pass every time, and they sometimes get false negatives which take time to investigate. We could probably replace 90% of them with a single integration test.
(I won’t suggest getting rid of them though, just running them less often)
<h4>Colony on Mars = __________</h4>
It helps to know what you’re working towards. A clear and ambitious goal has certainly helped keep SpaceX focused on what’s important. What’s the ultimate end goal of your project? What will make you say ‘Mission accomplished’?
The optimisation of the productionisation process is vital for any software company. There’s a huge impact on time but it’s also the difference between being proud of your accomplishments, and being embarrassed. We’ve all seen projects that are being carried by one developer who knows how to fix the many reoccurring issues, and has given up hope of ever working on something interesting again.
I guarantee no app ever went live without a single issue, so there’s plenty of room to improve.
<h3>What Should We Be Doing Differently?</h3>
With all the parallels described above, I think we can take some learnings from the five steps. Looking back at some of my own projects that were difficult to get live, I can now see that we skipped some steps or did them in the wrong order, just as Elon admitted Tesla had done in the past. Based on this, here’s what I think we should all be doing differently:
<ul>
<li>Challenge all requirements, no matter how small. A common example would be “We’ve got streaming data therefore we need a streaming data pipeline” but in reality there’s no requirement for data to be processed in realtime. Micro-batch processing is much simpler and cheaper to build.</li>
<li>Pause once you have a working concept to ask which processes aren’t needed, and just delete them. Don’t spend time on a feature nobody cares about just because you want everything ‘production ready’.</li>
<li>Recognise the true cost of complexity. Any ‘part’ in your app that could be replaced with something much simpler must also be deleted.</li>
<li>Don’t optimise for the wrong thing. Is speed your priority or is it lowering cost? Should you spend your time improving features or adding more? Find a real person in the business who can explain what the priority is and why.</li>
<li>
Don’t optimise too soon. Don’t invent a framework before trying something the simple way first. Don’t define a data model when the data isn’t finalised. Be flexible to change throughout.</li>
<li>
Accelerate your code before you automate. You’ll find which parts require further optimisation, and uncover problems that would be impossible to fix after going live.</li>
</ul>
Most importantly, appreciate just how challenging it is to go from a great concept to a live app for a mass audience. Your approach to productionisation could determine your success or failure. Keep complexity low and momentum high, else you’ll never get to Mars.
The full text of the interview referenced can be found here: <a href="https://everydayastronaut.com/starbase-tour-and-interview-with-elon-musk/">https://everydayastronaut.com/starbase-tour-and-interview-with-elon-musk/</a>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=a7963fef9887" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/elon-musks-5-steps-of-engineering-applied-to-software-a7963fef9887">What Can a Software Engineer Learn From a Rocket Engineer? Quite A Lot Actually.</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

What Can a Software Engineer Learn From a Rocket Engineer? Quite A Lot Actually.

A large part of our business is making valuable data science products a reality. I’m going to share some of what we have learnt on how to do this quickly whilst maintaining the highest quality.
Pandas alone won’t save you
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7mU7TnQZRcHy51xc"><figcaption>Photo by <a href="https://unsplash.com/@wuyyyying?utm_source=medium&amp;utm_medium=referral">Ying Wu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure>The first mistake is to believe that every data science project is only a data science problem. Multiple different roles have an important part to play.
The best and most efficient teams I have worked in are a combination of Data Science + Product + Engineering roles. That will not come as a surprise, but what makes a difference is ensuring that these roles have a part to play throughout the project. Unless the project is heavily standardised or a repeat of another — there is virtually no chance of finishing on time or to an acceptable standard by passing (or more commonly “throwing”) from one team to another.
A common example of where early team integration really shines is during feature generation.
<ul>
<li>Data features will often need to be re-run, re-configured and re-designed (multiple times)</li>
<li>New data sources (with valuable additional signal) need to be ingested and cleaned</li>
<li>Tests on various potential models will be repeated and reviewed when features change</li>
</ul>
A role diverse team will create repeatable processes from the outset and adhere to the boundaries of company infrastructure, ensuring that no costly logic “re-writes” are required when moving from test to live datasets. The benefits are often greater than the sum of parts — we develop data models supporting current project requirements and perhaps those of others too. This is just one of the areas where we have managed to cut weeks, if not months, out of projects without effecting quality.
MLOps products help, collaboration is better
Secondly, do not let anyone tell you that an MLOps product (Kubeflow, MLflow, Cloud provided, etc)+ Data Science are going to be all the tools your team will ever need.
Elements of Data Science projects are similar, especially in operations (monitoring, testing, processing, development environments etc). Although there are great open source and cloud based tools available, parts of your projects will be unique to your business.
If your business is software based this might be second nature. If not, having a strategy to share both code and documentation between projects is a fantastic accelerator. Make sure you bring relevant teams together to discuss lightweight governance, location and features of existing reusable parts.
Here is an example of our serverless components, we use these across several projects to reduce maintenance time and improve quality: <a href="https://datasparq.ai/elements-of-engineering">https://datasparq.ai/elements-of-engineering</a>
Think about the end at the beginning
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4PFHtzQVS7nvozbt"><figcaption>Photo by <a href="https://unsplash.com/@sigmund?utm_source=medium&amp;utm_medium=referral">Sigmund</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure>Finally, those with experience will know that finishing a data science project is as difficult as starting or building one. It turns out that APIs are not designed for human consumption. And, no team is likely to be happy manually running data processes, creating graphs or having similar conversations forever.
It is also possible that no two Data Science projects you run have the same results interface. A handful of examples we have completed include: APIs, web / mobile applications, static files, databases (all different kinds), messaging service notifications and more.
Finishing a project means delivering something that can be used in operations, sustainably. It’s well worth taking the time early on in a project to make sure your team either has or can acquire the skills needed to finish the project as they might be a little different to starting and developing. This is where product-oriented awareness and design helps round a project off in a way that drives impact for the customer.
Final thoughts
Do not let any of the above put you off! Data Science projects are extremely rewarding, fantastic investments and really enjoyable to work on. The extra effort invested in early team integration, code collaboration and building tricky results interfaces is often rewarded by increased adoption and usage.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=1b380ff13ea5" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/data-science-projects-at-pace-1b380ff13ea5">Data Science projects at pace</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Data Science projects at pace

As technology and investment market trends converge, now is the time for PE firms to learn what AI really is.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mwnMr4VIcm2l4Dxka4ohhA.jpeg"><figcaption>Photo by <a href="https://unsplash.com/@yukato?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Yu Kato</a> on <a href="https://unsplash.com/s/photos/lens?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure>One clear trend in the investment world right now is the <a href="https://www.ft.com/content/cd9571a3-726c-4995-9954-23a8dcf12b19">growth of Private Equity deals</a>. The first half of 2021 has seen PE deals worth more than $500bn; helping to propel global mergers and acquisitions activity to an all-time high.
<a href="https://www.idc.com/getdoc.jsp?containerId=prUS47482321">The other trend is AI technology growth</a> — despite being around for decades, and growing significantly since the arrival of cloud computing earlier this millennium— AI is still growing. This trend isn’t particularly new, but it means that AI capabilities are becoming more common place in more businesses; and leading businesses are taking advantage of the benefits of AI to outperform their lagging competition.
Combining these trends, it’s no surprise that there’s an increasing amount of investment activity involving companies that have AI as a core component of their proposition and enterprise value. <a href="https://www.cbinsights.com/research/report/enterprise-ai-trends-2021/">AI companies raised a record $33 billion in equity funding in 2020</a>, and what’s more, an increasing number of companies claim to be “AI-first”; using cloud technology, data science and engineering to deploy AI into different aspects of their business to operate better.
Time for PE to learn about AI
If AI is deemed to be a crucial differentiator for a company’s proposition, or a major value lever given the rich, unique dataset the company has, then it’s important this is well understood when assessing the company’s valuation and investment potential. When considering a deal with an “AI-first” company, any investor needs to be clear on the existing and potential value of AI within that company.
<ul>
<li>Do buyers understand exactly what AI capabilities they’re buying?</li>
<li>How do buyers evaluate and leverage the claimed AI capabilities?</li>
<li>For data-rich companies that aren’t even claiming to use AI, how do buyers assess the potential value that AI could add to the acquisition?</li>
</ul>
Many investment brochures will advertise the organisation’s good data governance, rich data assets, scalable data platform, and leading AI capabilities that makes it a unique investment opportunity — but in many cases the truth may be it’s not so unique. After all, despite AI maturing in the last few years, it’s still like teenage sex — more companies are talking about it than actually doing it.
PE companies will use their preferred IT due-diligence and Commercial due-diligence partners to confirm a potential investment’s true capabilities, potential and risk; but these organisations rarely have the AI practitioner experience to robustly assess AI; and the remit of a typical IT DD and CDD exercise is too broad to provide any deep analysis within the AI domain.
The answer: AI Due-diligence
Given the increasing importance of rigorously evaluating a company’s AI capabilities before investing, and the lack of detail IT DD and CDD studies provide for this, DataSparQ is finding itself developing valuable AI Due-diligence reports on potential assets for PE investors. Whilst a PPT slide deck isn’t the traditional output of a DataSparQ project (we pride ourselves on developing and running operational AI solutions in the real world; our outputs are usually more Python than PPT), our deep AI practitioner expertise means we know what good looks like, and because we understand what it takes to deliver AI successfully in the real world, we’re in a privileged position to help others.
Our AI DD service consists of both a “backward-looking” confirmatory assessment of a potential investment’s AI capability, as well as a “forward-looking” analysis of how AI could be used to unlock further hidden value in that business. Different investments often have a different focus; for businesses with established AI capabilities that give them a unique competitive advantage, the confirmatory assessment is more important. For data-rich companies early in their analytics journey, the forward-looking analysis is often more valuable.
The areas of investigation during an AI DD include:
<ul>
<li>
Data diagnostic — a review of the data collected, stored and used by the business. Is it done systematically or ad-hoc? How accurate, relevant and complete is it? How is it catalogued and governed?</li>
<li>
Algorithm diagnostic — an assessment of the algorithms and analysis used. Are they appropriate (meeting requirements for accuracy, explainability, auditability, bias, customisation)? Are they robust? Will they scale and remain competitive as the business grows and the market matures? Are they compliant with industry rules and regulations?</li>
<li>
Technology diagnostic — a review of the technology systems and architecture used to build and run these AI capabilities. Are they resilient? Will they scale to support future growth requirements? Are they cost efficient? Are they secure?</li>
<li>
Capability diagnostic — an analysis of the business’s way of designing, building and running AI solutions. Do the right skill-sets exist? Is the capability unique and sustainable? Does the way of working encourage innovation?</li>
<li>
AI Opportunity review — a process to identify, validate and quantify high-value opportunities for AI yet to be exploited. What hidden value levers can AI unlock? What incremental value does AI offer over a more low-tech solution? How feasible is each opportunity? What’s a realistic delivery timetable for value capture?</li>
</ul>
Given a modest amount of time with the management team, and other key individuals in the business who might not normally be involved in traditional DD exercises, these questions can be answered within the timelines for most DD processes. It can be efficient for the AI DD provider to work alongside the IT DD and CDD providers so the same questions don’t have to be asked twice.
Bottom line
As AI matures and becomes a differentiating value lever in organisations, it’s increasingly important for investors to put this capability under the microscope when considering deals. This means PE companies need to start asking new questions, and learning more about AI than the buzzwords and headlines in order to know what questions to ask, let alone answer them.
The innovation of AI technologies and methods is fast and shows no sign of slowing. It will be an ongoing learning journey for investors to ensure their assessments remain relevant. As a result, AI practitioners like us may find ourselves increasingly doing PPT as much as Python as we become the go-to service providers for AI DD.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=e2da04f94b8e" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/why-private-equity-needs-to-learn-about-ai-to-make-better-investment-decisions-e2da04f94b8e">Why Private Equity needs to learn about AI to make better investment decisions</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why Private Equity needs to learn about AI to make better investment decisions

Humans don’t like risk and, as with any technology investment, a Machine Learning solution comes with its own set of risks. Will it predict the right thing, will it be accurate, will it be useable? In this post I’m going to take you through how we at Datasparq reduce risk through Proof of Value.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*IwpfqunBgcA0693VDTVP4g.jpeg"><figcaption>A machine learning solution to demand forecasting can lead to pretty, ordered, stocked shelves… and higher revenues! But will it work?</figcaption></figure>Our machine learning solutions are designed to allow people to perform beyond their potential, whether that’s making more accurate sales forecasts, creating effective retention strategies for customers, or focusing business development effort on leads likely to convert to wins. Machine learning solutions give people the benefit of the predictive power hidden away in data.
Machine learning has proven time and again to be effective at predicting the future by learning from the past, whether that’s <a href="https://www.bbc.co.uk/news/technology-57157566">identifying potential skin cancer</a>, or <a href="https://towardsdatascience.com/deep-dive-into-netflixs-recommender-system-341806ae3b48">helping you decide what to watch next on TV</a>. That’s not entirely surprising as these solutions are fundamentally doing what humans have evolved to do — observe, learn and predict, just with the advantage of being able to process vast amounts more data.
But just because there’s evidence something has worked before doesn’t necessarily mean it’s going to work in every situation.
This is why we start all of our machine learning projects with a Prove phase. It’s a short, targeted mission to reduce ambiguity and test assumptions to minimise risk. Over the course of a few weeks, we validate or test the following:
<ol>
<li>That there is a clear target variable for the prediction - we know what we’re predicting.</li>
<li>That there is a viable means for measuring value from the prediction</li>
<li>That there is predictive power in the data (signal in the noise)</li>
</ol>
1. Clear target variable for the prediction
The target variable is the variable in a dataset whose value is to be predicted using other variables. It’s pretty fundamental to any machine learning solution, and picking the wrong target variable can lead to significant wasted effort down the line.
Our goal here is to define which fields in the data will be used to form the ‘target variable’. This is an iterative process where we identify a target variable, then validate how a prediction of this target variable will be beneficial. This in turn raises new questions which allows us to loop back and idenfity a better target variable.
It sounds like it should be easy. Let’s look at demand forecasting as an example. Imagine you sell jeans in a high street store. In order to effectively manage your stock levels you need to make an estimate in advance of how many you’re going to sell. So a machine learning solution just needs to predict your sales. Simple, right?
In reality, defining the scope of prediction tends to be a significant chunk of work that requires us to deep dive into an organisation’s processes and value chain.
<ul>
<li>What product or service are we predicting demand for? Is it one product, or hundreds, each with different demand patterns?</li>
<li>For what period are we predicting demand — next week, month, year? Or maybe combinations of these, or maybe different products would benefit from different levels of prediction?</li>
<li>What is ‘demand’? Is it a placed order? A product taken off a shelf? What if that order was later cancelled, returned, or not paid for?</li>
<li>Do we need to care about who is making the demand — do we need to predict demand from a specific customer or location, or a demand that will be fulfilled by a specific fulfilment centre?</li>
<li>How far in advance would the prediction be required in order to be valuable to the business? Predicting tomorrow’s demand today is much easier than predicting for a day next month.</li>
</ul>
Some of these questions (the known unknowns) will have been answered prior to the Prove phase as part of our <a href="https://www.datasparq.ai/products/">SparQshop™</a>, others arise from deeper exploration of the data (the unknown unknowns!). All need to be answered if we’re to precisely define the target variable.
2. Measuring value
In order to optimise a model we need to know what metrics we’re using to measure performance. Understanding the key performance metric gets to the heart of some of the difficult questions around model optimisation, for example:
<ul>
<li>In a regression problem (where we are predicting a number, like product sales), what’s the relative value in over predicting vs under predicting? Underpredicting by 10% might have a significantly bigger cost impact than overpredicting by 10% (you can’t sell what you don’t have, you may though be able to store some excess stock for selling later).</li>
<li>In a classification problem (where we are predicting a binary outcome, like customer churn), what’s the impact of a true positive vs a false positive? If we predict a customer to a subscription service is going to leave (churn), they may get offered a discount to stay. If they take up this discounted offer that’s great if they would have otherwise left. However, if it was a false positive prediction and they were actually never going to leave, it’s a cost to the business.</li>
</ul>
Then there may be other factors specific to the problem being solved. In the demand forecasting example, is it more important to accurately predict demand for some products over others (maybe some have a shorter shelf life, or are more expensive to keep in stock)?
The goal here is to come up with a metric to measure model performance which relates directly to the value being created for the business. For example, for customer churn, that metric may be:
Profit ($) = R*TP — C*FP
Where R is the revenue from a true positive, C is the cost of a false positive, and TP &amp; FP are the true positive and false positive rates.
3. Predictive power in the data
This is where we create a baseline machine learning model and test the predictive power inherent in the data. This starts with some basic evaluation of the data — specifically is there enough historical data to be able to train and test a model?
Next we need to know where to start with the data. We’ll typically run a hypothesis workshop with people in your organisation who understand the data and the target variable. In the case of demand forecasting this would involve those close to the buying/selling operations to understand, from their experience, what factors affect demand. For example, we may learn that they see sales go up in summer, or are higher at shops in certain locations. From these hypotheses, we extract <a href="https://en.wikipedia.org/wiki/Feature_(machine_learning)">features</a> from the data, for example “store location”, or “average temperature over the previous 7 days”.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*Uny-ym1ashJxGKzn1Gmwhw.png"><figcaption>thanks <a href="https://xkcd.com/license.html">XKCD</a></figcaption></figure>From here we can do some univariate analysis. Are there features which show predictive power on their own? <a href="https://www.tylervigen.com/spurious-correlations">Correlation ≠ causation</a> so success here isn’t proof of signal in the noise, but can help steer focus to the most useful areas of the data.
Once we have a handful of features we believe to be predictive, we can put them into a baseline model and test it. We measure the performance of the model by testing it against data it hasn’t seen. This is both out-of-time (given complete data up to 2018, can it predict what happened in 2019) and out-of-sample (given data for all time but removing data for certain shops, or products or price points at random, can it predict the gaps?).
The Prove phase isn’t designed to deliver a fully optimised model, our goal here is to demonstrate evidence that there is predictive power in the data and to get a sense of how much value might be gained through an optimised model. It’s not an exact science, but working closely with the subject matter experts gives the model the best chance of succeeding early on and it’s usually possible to prove value in this first phase, or at least get a good sense of whether it’s worth pursuing.
We’re good to go!
…not quite. The above activities bring clarity and definition to the model and it’s potential value but even the most accurate demand forecast is of absolutely no value unless it’s used.
The final piece we look at in the prove phase is how the solution will be used in operations. Are the stakeholders bought in to the idea of having a machine help with their jobs? Is is feasible to integrate the predictions with the systems/tools they use every day? What changes might be needed to operational processes once the predictions are live?
Even if these questions aren’t fully answered at this stage, having the discussion early on is another way to reduce risk.
If you think there may be an opportunity for machine learning to deliver value in your organisation and you’d like to chat about a low risk way to get started and prove the value, please get in touch at <a href="mailto:info@datasparq.ai">info@datasparq.ai</a>.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=e61f9700ef58" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/proving-the-value-of-machine-learning-e61f9700ef58">How to prove the value of machine learning and avoid worthless models</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to prove the value of machine learning and avoid worthless models

Well you probably do, it’s not that hard.
Rather than test your knowledge of the language, I’ll suggest seven different things you could do or learn next that will make you a better, more capable, Python developer. After all, it’s not the language itself that’s powerful, but how it’s used!
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2Uw3ZfvoggULk3qj"><figcaption>I promise I’m not gatekeeping (photo by <a href="https://unsplash.com/@lailagebhard?utm_source=medium&amp;utm_medium=referral">Laila Gebhard</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><h3>1. Master Unix and Learn (Some More) Bash</h3>
Hopefully you already love Bash more than Python and use it every day, but for many devs who started out with Python, it may never have been a major part of everyday operations. If Bash seems scary or clunky to you, I strongly recommend experimenting with some core commands and understanding loops, piping, and links.
The command line is the universal standard for getting programs to talk to each other, and if you can’t get your Python to talk to other programs, you’ll repeatedly hit barriers.
Bash has helped me get around awkward situations like not being able to use pip (in a serverless code execution environment), and not having admin rights on Windows (Git Bash was installed, thankfully).
You can always rely on Bash. With a strong grasp of it you can work around any problem. Not to mention you can run Bash commands with Python and vice versa.
<h3>2. Learn a tiny amount of HTML/JS/CSS</h3>
There are some things Python should not be used for, and Front End development is one of those things.
I’d also say the same about data visualisation. My favourite plotting libraries for Python (<a href="https://plotly.com/python/">Plotly</a> and <a href="https://bokeh.org/">Bokeh</a>) render charts within a browser, using JavaScript. These achieve far better results than Matplotlib in terms of appearance, ease of use, and interactivity.
JavaScript is even eating into Java and C#’s market share. <a href="https://www.electronjs.org/">Electron</a> allows you to build desktop applications with just HTML and JS (building on top of Chromium). It might surprise you to learn that MS Teams is an Electron app.
Don’t pigeonhole yourself as a purely back-end dev! Learn the basics of a framework like <a href="https://reactjs.org/docs/create-a-new-react-app.html#create-react-app">React</a> and start showing off how great your work is with some stunning user interfaces. Plus, putting some thought into the user experience of your app will help streamline your back-end code.
Best of all, you get to call yourself a Full Stack developer.
<h3>3. Use Docker to industrialise your code</h3>
<blockquote>“Well it works on my machine” -Bad Python developer</blockquote>
Hopefully this one seems obvious. A Python developer running everything on their laptop, or on a VM with a lengthy setup script, is not a good developer.
Code that provides value should be able to run on its own, without being babysat, and should be able to be deployed and redeployed by anyone, otherwise it will end up being thrown away as soon as the person who wrote it isn’t around.
The first step to industrialising code is to move the code and all dependencies into a container. Docker makes this fairly straightforward with Dockerfiles, and allows anyone to run the container on any machine, provided they have Docker installed.
You can then make use of serverless tools like Google’s Cloud Run or AWS Lambda, which will run your container on request without the need to provision compute resources. This is the easiest way to turn your code into an API, and allow anyone to use without having to install anything. From here you can plug it into a web front-end and reach a much larger audience.
I’ve seen many git repos from universities and hobbyists alike that contain some amazing code but have absolutely no thought put into industrialisation. This is a great shame since it prevents most people from being able to play with (and therefore potentially contribute to) the code. If they had only provided a requirements.txt and Dockerfile, someone might have taken it and created something truly mind-blowing.
Disclaimer: I’m a huge fan of Cloud Functions/Lambdas which don’t require any Docker knowledge, but some tasks require containers.
<h3>4. Learn PostgreSQL
</h3>
Everybody knows that Python is great for transforming data but that doesn’t mean it’s a great choice for an application backend or data pipeline. In fact, I would strongly discourage the use of Python (or pandas specifically) for any scenario in which a database could also be used. Database technologies like Postgres and MySQL have been designed to deal with data, and they do it really well.
Next time you think about joining two datasets with pandas, consider spinning up a PostgreSQL instance, importing your data into tables, and running a join on those tables (scripted by Python, using <a href="https://pypi.org/project/psycopg2/">psycopg2</a>). The setup may take longer, but when you find yourself needing to repeat the join again with updated data, you will see the benefits in terms of speed and reliability. You can then easily parameterise your SQL with Python formatting or Jinja2, massively reducing the amount of code needed to carry out all of your transformations.
SQL is dead easy to learn and Postgres is a great starting place. Once your data outgrows Postgres, you can apply the same knowledge to other, more powerful tools like Google BigQuery or Snowflake.
<h3>5. Don’t repeat yourself… ever again in your entire career</h3>
Have you ever copied code from one project to another? Perhaps you do this at the start of every project.
Save others from repeating your work! Make packages.
The first step is to collect together all the useful functions you use regularly. It might be best to group them up and make lots of separate packages to keep requirements simple.
Secondly make a <a href="https://docs.python.org/3/distutils/setupscript.html">setup.py</a> so that you can pip install your package locally.
Thirdly, and most importantly, put your package into a git repo and share it with your team*. Python packages can be pip installed directly from git repos like so:
<pre>pip install "git+ssh://git@github.com/&lt;username&gt;/&lt;repo&gt;.git"</pre>
Once you have a whole company using the same code across multiple projects, you start to see huge time savings both when starting new projects and when upgrading/fixing bugs in commonly used tools. I dread to think how long it would take us at DataSparQ to patch each of our projects separately if we weren’t using the same packages across all of them.
*Or, if you’re happy for it to be public, upload it to PyPI. That way you can save everyone from repeating your work!
<h3>6. Become a Git Expert</h3>
I’ve met developers who use Git every day and don’t actually understand it. As soon as something unexpected happens they don’t know what to do, and don’t know how to avoid problems in the first place.
Every team needs a Git expert. Someone who can confidently handle tricky merges, rebases, cherry picks, etc., and keep repos tidy.
If your releases are difficult and technical debt is piling up, the problem might be here. Make sure you understand branching strategies like Gitflow, and how to use Git + Agile + good documentation in the most complementary and efficient way.
<h3>7. Learn C to remove bottlenecks</h3>
I have come across problems that Python was too slow for. The solution was C, wrapped in Python (so you could say Python wasn’t too slow).
Ok you don’t need to learn C. I would suggest Go instead as it has a lot of support from cloud providers and is much easier to compile for different architectures, which was always the biggest pain with C for me.
Some of the most powerful packages like numpy, pillow, and TensorFlow are all just C modules wrapped in Python. That level of performance can also be achieved with very little effort these days. One could write a simple program in Go and call it from Python as part of a larger script, for example:
<pre>import subprocess</pre>
<pre>def run(cmd): p = subprocess.Popen(cmd, stdout=subprocess.PIPE, text=True) return list(iter(p.stdout.readline, "")) </pre>
<pre>x = run("count_emojis 🚀 --file wikipedia.zip")</pre>
If anything, writing a hello_world program in C is worthwhile purely to appreciate how easy Python is.
Disclaimer 1: I only suggest this method for simplicity. It doesn’t benefit from shared memory but is much less painful to write, and less likely to make Python crash.
Disclaimer 2: Python is definitely powerful enough to count emojis.
<h3>Conclusion</h3>
There are 0 job roles that require Python skills and nothing else. Branching out and learning complementary skills will make a huge difference to how effective your code can be.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=9c0d8dc8451e" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/so-you-think-you-know-python-9c0d8dc8451e">So you think you know Python?</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

So you think you know Python?

Synthetic datasets are boring. In most cases, we just generate them by drawing samples from a multivariate distribution with components from families of distributions that we have set previously, adding some noise at the end to make them more realistic. And of course, modelling a synthetic dataset is rather easy, as we have access to the underlying distribution of the data.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rlI8fkpCKdzYg7ljfQlclA.jpeg"><figcaption>Photo by <a href="https://unsplash.com/@photography_by_feaver?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">James Feaver</a> on <a href="https://unsplash.com/s/photos/real-estate-uk?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure>Real datasets are much more exciting. We are given some data and the task of modelling their underlying distribution. There are no right or wrong answers, and every approach is fair game. There are obviously some better and worse solutions which is precisely what we try to explain in this post.
Consider <a href="https://www.kaggle.com/hm-land-registry/uk-housing-prices-paid">this</a> dataset from UK’s HM Land Registry, hosted in Kaggle. It contains more than 22M real estate transactions for the period between 1995 and 2016, with individual information on the price of sale, date, location and type of property, among others. From the original dataset, we select the period between 2012-2014 and consider only the columns that describe the date of sale and price.
<h3>1. Objective and Approach</h3>
Our aim is to model the probability distribution of prices paid for real estate in the selected period. We examine four different approaches to tackle this problem:
<ul>
<li>Gamma fit</li>
<li>Log-normal fit</li>
<li>Gaussian Mixture for the log-price data</li>
<li>Gaussian Mixture</li>
</ul>
In this post, we review mixture models [1] to show how these can be useful when dealing with complex probability distributions. In particular, we apply a Gaussian mixture model to model the probability distribution of prices. The code to run these simulations can be found <a href="https://gist.github.com/kiko-datasparq/3070ad063dd3b69d240f23181c00126a">on github</a>.
<h4>Take-Home Messages</h4>
What worked? Gaussian Mixtures provided a great balance between mathematical complexity, difficulty in the implementation and results for the fit. What didn’t work? I initially tried fitting a mixture of Gammas because I thought it might give better results. This became a mathematical nightmare that I quickly dropped, sticking with the nice closed-form results for Gaussian mixtures. What was time-consuming? I spent a significant part of the time understanding and interpreting the components in each GM fit, as well as finding a nice way to demonstrate my insights. As a result, I decided on the log-log, log-x and linear plots to show each part of the distribution in detail. Why should we care about this? Modelling the underlying distribution is crucial to understanding our data. Although in many situations we might get away with a known distribution, this is not always the case, as we have clearly shown. Having mixture models in our toolbox allows us to work with more complex datasets.
<h3>2. Time-adjusted Prices</h3>
Let’s examine the evolution of the mean daily price paid per property in the selected period.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S9J8Wx4rhsC9UQesXhMA7Q.png"></figure>It is very clear that there is an increasing trend in this time series. The series starts with mean values around 230k and ends with mean values in the order of 280k, implying an increase of around 20%. Before modelling the probability distribution for the whole series, we adjust the prices for this increase.
We model the trend using an exponential function of the form
<figure><img alt="" src="https://cdn-images-1.medium.com/max/141/1*oA6BtHb9GmiRbP51JcLuIw.png"></figure>where a, b are the parameters we need to fit, t is the time (in days) since the beginning of 2012 (our selected period) and y is the target price (in £). The result of the fit is given below.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KoiNkzEgDmqieHyakpJCsg.png"></figure>An exponential fit provides a model with an intuitive interpretation in this context, where we have a geometric growth (similar to inflation) of prices. Note that a linear fit would give a very similar result but would lack such interpretability.
To adjust the prices, we multiply the price of the sale by exp(-bt), where b is the parameter obtained from the fit and t is the number of days between the beginning of 2012 and the actual day of the sale. As a result, we obtain an adjusted time series that looks as follows:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*phu1H0daPdfKQmCeMAK_1w.png"></figure>This result is far from perfect because we still observe periods where the adjusted price goes far away from the mean (which corresponds to parameter a from our exponential fit). However, it is a significant improvement with respect to our original data because we have eliminated its trend.
<h3>3. Modelling the Price using a known Distribution</h3>
We’ll assume the exponential adjustment is sufficiently good for our analysis. Now let’s examine the distribution of adjusted prices.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ddlZe5-sIS_g6IJul-S3iQ.png"></figure>This histogram has been truncated to prices below £1.2M and grouped in bins of £1k. The price distribution is relatively smooth and has a heavy tail with a maximum above £80M (not shown in the histogram).
<h4>3.1 Gamma Fit</h4>
As a first approach, we model the probability distribution using a Gamma. Note the pdf for this distribution is given by
<figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*AHDH3IHEJv5SlARCEkuH4g.png"></figure>where α, β and x₀ are the parameters to be estimated. It turns out that for our data a Gamma distribution is not a good model. The Maximum Likelihood Estimation (MLE) with the entire distribution does not give anything meaningful so we don’t include that result here. The fits with percentiles 95 and 99 (p95 and p99) of the data are given below.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aE_6qg_B-0-VIeih2NpeZw.png"></figure>The p95 fit captures well the mode of the empirical distribution but it does not explain a significant proportion of the data. As we try to include a larger part of the tail, the model breaks. We see that for p99, the match between the empirical and Gamma fitted distributions is much worse than for p95.
<blockquote>In conclusion, the Gamma distribution is poor at capturing the heavy tail of our data.</blockquote>
<h4>3.2 Log-normal Fit</h4>
The histograms for the price (with log-x axis) are given below. For the plot on the left, we have also used a log-y axis to clearly show the heavy tail of the distribution, with prices above 1M for tens of thousands of transactions.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eNqWOcLTniISmGyqDDhvVQ.png"><figcaption>(left) Log-log histogram for price data; (right) Log-x histogram for price data.</figcaption></figure>The plot on the right resembles a Gaussian distribution which motivates using a log-normal distribution to model the prices. To this end, we transform our original price distribution X as
<figure><img alt="" src="https://cdn-images-1.medium.com/max/125/1*PSF3VRG1rbKaRjPlajEqCw.png"></figure>where
<figure><img alt="" src="https://cdn-images-1.medium.com/max/132/1*DsYavMJHktPrdUjv_9QT_g.png"></figure>is distributed as a Gaussian with mean μ and standard deviation σ. We compute the MLE fit with the log transformation of the data and obtain the following results.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k27zcg74csn2exEdiD2qdg.png"><figcaption>(left) Log-log pdf; (centre) Log-price pdf; (right) pdf truncated at 1.2M.</figcaption></figure><ul>
<li>The plot on the left shows the log-log pdf for both the empirical distribution (i.e, the data) and the log-normal fit. We observe that for selling prices larger than £1M, the fit systematically underestimates the observed probability.</li>
<li>The centre plot shows the log-x pdf. The log-normal fit captures correctly the mode at around £100k. However, we see that it underestimates the probability allocated at the mode. We also observe this in the linear-x plot on the right.</li>
</ul>
<blockquote>We are interested in finding a model that can approximate better the pdf both at the mode and the tail of the distribution. One way to do this consists of using mixture models which we review next.</blockquote>
<h3>4. Modelling the Price using Gaussian Mixtures</h3>
As we saw, a log-normal fit for our property prices data presents certain limitations, since the two free parameters that we fit in the MLE cannot capture all the complexity present in the data. Mixture models provide a solution to this problem. We assume that our data X is distributed as the linear combination:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/202/1*kq0cNeP0FWJPs0JQZ_mJLQ.png"></figure>where the weights ωₖ satisfy
<figure><img alt="" src="https://cdn-images-1.medium.com/max/108/1*jz8vLVvvmKkxi76oqsAUJg.png"></figure>and the basis functions gₖ are probability density functions (pdf) from known families of distributions. In this case, we choose gₖ to be Gaussian, therefore:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/161/1*2mE9TpdbVzXd4EErJQCuxQ.png"></figure>Instead of estimating (μ, σ) as we did before, now we estimate (ωₖ, μₖ, σₖ) for all components in the mixture. In the numerical experiments below, we have used the <a href="http://pages.cs.wisc.edu/~jerryzhu/cs838/EM.pdf">Expectation-Maximisation (EM)</a> algorithm to obtain the parameters for the mixture which is very simple and easy to implement.
<h4>4.1 Gaussian Mixture for Log-price</h4>
We set the number of components in the mixture K = 6 and find the MLE solution for a Gaussian Mixture (GM) for Y = log(X), where X is the adjusted real estate prices. This value has been chosen heuristically to strike a good balance between the number of parameters (model complexity) and the capacity of the model to explain the data. We briefly address how to quantitatively select the number of components in the mixture in “Open Questions”.
<h4>4.2 Gaussian Mixture for Price</h4>
We set K = 6 and find the MLE solution for a Gaussian Mixture (GM) for X, the adjusted real estate prices.
<h3>5. Results</h3>
So far we have discussed four approaches to model the probability distribution of real estate prices, namely:
<ul>
<li>Gamma fit</li>
<li>Log-normal</li>
<li>Gaussian Mixture for the log-price data</li>
<li>Gaussian Mixture</li>
</ul>
We discard the Gamma fit from further analysis due to its severe limitations and compare the remaining three. In the graph below, we have plots with four lines, one corresponding to the empirically observed distribution (data pdf) and one for each of our fitted models.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lzQacZTTQoSh9goOjc4WKQ.png"><figcaption>(left) Log-Log pdfs; (centre) Log-x pdfs; (right) Pdfs truncated at 1.2M.</figcaption></figure>As we already analysed the log-normal fit, let’s discuss the GM for the log-price (green curves). This fit is clearly better. In the (left) log-log plot we observe that the GM-log fit is closer to the empirical pdf than the log-normal fit. Additionally, in the centre and right plots, we note that the mode is not only captured but also assigned the correct probability. The only limitation of this mixture is that it systematically underestimates probabilities for prices above £5M.
The GM fit for the price gives slightly different results. It is also preferable to the log-normal fit since it follows the tail of the distribution for large prices, and it assigns the mode the correct probability, as we see in the centre and right plots. However, we also observe that at the head and the tail of the distribution, there is still a discrepancy to the empirical pdf.
<h4>5.1 Analysis of Components</h4>
In both GM fits we chose K=6 components. We now have a look at how these components look and contribute to the final GM distribution.
In the graphs below, we have plotted a line for each component gₖ in the GM log fit, weighted by its corresponding ωₖ, each denoted by Cₖ for k=1…K (not sorted in any particular order).
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9SoDjcW83SG1KBOh63NatA.png"><figcaption>[GM log-price components] (left) Log-log pdfs; (centre) Log-x pdfs; (right) Pdfs truncated at 1.2M.</figcaption></figure>These plots are quite revealing. The (left) log-log plot shows that the components present a very sharp decrease at the tail of the distribution which is why this mixture struggles to capture probabilities for large prices. The plot on the right makes explicit that each component is a log-normal distribution in the linear space (as expected).
Similarly, we analyse the components for the GM fit for the price. In the graph below, we have plotted the pdf for the weighted components ωₖgₖ, each denoted by Cₖ.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lTc2GTJui_a2scydDLfpuw.png"><figcaption>[GM price components] (left) Log-log pdfs; (centre) Log-x pdfs; (right) Pdfs truncated at 1.2M.</figcaption></figure>The plot on the right shows that the pdf is decomposed using Gaussians in the linear space. In the plot on the right, we see why the pdf has wiggles, as we have a component (C1) with a very large mean (more than £5M) with a very small weight which contributes to allocating probability at the tail of the distribution. This is something we did not have either in the log-normal or the GM-log fits, as these fits systematically underestimated the probability at the tail of the distribution.
Additionally, the GM fit assigns a non-zero probability to relatively small, non-negative values, which are unrealistic (all price values in the dataset are strictly positive, as expected). However, for consistency, we have excluded prices lower than zero from the linear-x plot on the right.
<h4>5.2 Open Questions</h4>
There are different directions for improving this analysis. A fair question would be: how do we choose between models? As of now, We do not have a clear answer on whether the GM-log or GM (or even the log-normal) fit is preferred. We discussed the results qualitatively but did not provide a metric to make a quantitative decision. Additionally, we could ask how best to choose the number of components. Both these questions can be answered via the <a href="https://medium.com/@analyttica/what-is-bayesian-information-criterion-bic-b3396a894be6">Bayesian Information Criterion (BIC)</a> which allows choosing between models with a different number of parameters via computation of the likelihood.
<h3>Summary</h3>
<ul>
<li>
Mixture Models provide a flexible framework to model complex distributions coming from real data.</li>
<li>We used Gaussian Mixtures (GM) to model the distribution of real estate prices in the UK for the period 2012–2014. We compared GMs against more simple distributions such as Gamma or Log-normal which provided worse results for the fit.</li>
<li>
Gaussian Mixtures are simple and easy to implement in combination with the EM algorithm. There are analytical expressions for estimating all parameters which is not the case for other mixtures.</li>
</ul>
<h3>References</h3>
<a href="https://www.gwern.net/docs/statistics/2019-mclachlan.pdf">[1]</a> McLachlan, Geoffrey J., Sharon X. Lee, and Suren I. Rathnayake. “Finite mixture models.” Annual review of statistics and its application 6 (2019): 355–378.
Would you like to understand how AI can generate value for your business? Visit <a href="https://www.datasparq.ai/">DataSparQ </a>to find out more about the products we could build for you and the services we offer or get in touch with a member of the team to start your AI journey today.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=d70acb3bc9d" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/gaussian-mixtures-to-model-real-estate-prices-in-the-uk-d70acb3bc9d">Gaussian Mixtures to Model Real Estate Prices in the UK</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Gaussian Mixtures to Model Real Estate Prices in the UK

A small bit of web history
Some years ago in the 90s when I was doing my MSc in Computer Science and the web was the next big tech frontier, we had a problem.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fbQmx9uT6PuwrmE3Vv36BA.jpeg"><figcaption>Photo by <a href="https://unsplash.com/@dlerman6?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Daniel Lerman</a> on <a href="https://unsplash.com/s/photos/search?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure>That problem was search — it was broken, not just a little bit either but totally broken. The search engines of the day regarded search much as a word processor does now. They gave you a search dialogue and expected to return some or even all of the pages in their catalogue which contained your search term. If there was a tie between matching pages, then the engine might look to see if your search term appeared in a title or heading or appeared multiple times within the body of the page.
The inevitable happened — spam pages emerged with huge numbers of hidden words and likely search terms embedded in them. Overnight web search as a tool became completely useless. You couldn’t find anything.
That was of course until <a href="http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf">Sergei Brin and Larry Page wrote a paper</a> as part of their PhD detailing a neat piece of what would later become known as Data Science that could recover relevance and importance in web search. That piece of applied mathematics was PageRank and it became a $1tn company — you will know it as Google.
Job search
Well searching for a job is a bit like that now. Job postings appear containing all sorts of relevant or not so relevant information. They’re posted with wildly diverging titles even if they represent essentially the same job. So Primary Teacher role might get posted with the title: “Part-time Teacher (Primary) years 5–6 Ipswich-starts September” or equally “Primary Teacher (Fixed term maternity cover) £24k — Brent”.
All because jobs forums and recruiting companies think they can elevate their posting in a simple search box. Meanwhile applicants are reduced to searching for terms they think might match the sort of job they are interested in — often the job title. The result is a deeply unsatisfying user experience —lists of over specific jobs that do not represent the breadth of possibility for the applicant’s capability.
Job search is as broken now as searching for a web page in 1996.
The Job Finder Machine
We started work on the <a href="https://labs.datasparq.ai/job-finder-machine/">Job Finder Machine</a> to address this problem in December 2020. As my colleague <a href="https://medium.com/@oliverbartlett/1a7ffe6aa7ff">Oli Bartlett has written in his post</a>, this was part of a data-science-for-good project with the <a href="https://emergentalliance.org/products-and-services/job-finder-machine/">Emergent Alliance</a> to answer the challenge of getting people with skills into well-matched jobs after the pandemic.
Our Data Science Hypothesis
Our data set consisted of a set of job titles representing current or recent job openings and associated with each title was a list of skills that were required by that job.
As with all good Data Science projects, we started with a working hypothesis which we tried to disprove.
<blockquote>
Hypothesis: We can generate high quality job matches based on user-entered natural language descriptions of skills alone</blockquote>
There were the usual problems of skills being duplicated, spelt wrong, capitalised, not linked with the skill with exactly the same name. Also we were expecting user input of skills from the applicant with exactly the same issues. Some useful techniques from NLP came to hand to help address some of these issues — but not all.
How did we do it?
First we developed a conceptual model of the jobs-skills landscape: a bipartite graph connecting a set U of jobs with a set V of jobs.
<figure><img alt="A bipartite graph representing the relationship between job postings and skills" src="https://cdn-images-1.medium.com/max/1024/0*nPps9WEKeqlJ95Rd"><figcaption>A graph of U, the set of job postings, to V, the set of skills needed by a job.</figcaption></figure>Why bipartite? Because at this stage, we only had a relationship between jobs and skills or skills and jobs and there were no direct relationships between jobs (this would come later).
What did this represent? It showed simply that some jobs need a set of skills and those skills might also be required by some other jobs. Not earth shattering but still useful.
Imagine that an applicant highlights some skills in set V that are relevant to them, could we use that information together with the graphical model to return a list of jobs that were a good match for the user’s skills. Potentially.
Those sets, by the way, U might have as many as 180,000 current jobs in at any one time, and V might have 3 million skills listed.
Just returning all the jobs that happened to mention one of the skills the candidate mentioned was a non-starter — we would have many 1000s of matching jobs potentially.
Equally, we realised quite quickly that returning all the jobs that matched all the skills a user mentioned quickly reduced the the number of matches to zero and even if not, very clearly missed lots of jobs that were clearly very similar in nature but had happen not to mention one of the skills that other jobs with the same name had mentioned.
Step 1: Constructing a common skills description for a job
<figure><img alt="Job descriptions and their associated and likely different skills descriptions." src="https://cdn-images-1.medium.com/max/1024/0*DF9kAkcbepxq7d21"></figure>Many jobs with the same job title had slightly differing skills.
We were trying to give a good match to a set of likely jobs and not necessarily a precise match to a specific posting. Also it might be that a hiring manager had left off some skills or had assumed them to be implicit in the role.
Whereas other companies were more precise over how they described the skills that a job needed.
We could learn from the aggregate data to come to a common skills description of a job with a given title. It would fill in some of the gaps left by the individual postings. It would also allow us to capture phrases and synonyms for skills that different companies had used.
We could also start to pick out the most frequent skills for a job and use that to give an indication of the importance of a skill for each job title.
Step 2: Establishing a jobs network
With a common view of skills for each job title we could now look at how similar jobs are in terms of skills.
<figure><img alt="The overlap in skills between a Software Engineer and a Data Scientist — represented as a Venn diagram." src="https://cdn-images-1.medium.com/max/970/0*HHlhpdVtFjMHivCt"><figcaption>A Venn diagram showing the similarity between two related job titles</figcaption></figure>We constructed a similarity metric between job titles, by comparing their skill sets.
We can do this by computing the <a href="https://en.wikipedia.org/wiki/Jaccard_index">Jaccard Index</a> between each pair of job titles.
This is then used to assign a weight to the edge between a network of job titles. This can be visualised in the following example of Job Titles Network.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1dP9ZcVG_xjuZapz"><figcaption>A network of job titles with associated skills similarity measures</figcaption></figure>Step 3: Clustering a network of Job Titles
Having obtained a job titles network we are in the exciting position to solve the problem of having lots of very differently described jobs with very similar skills. These naturally cluster together and for a certain granularity we can create a higher or lower abstraction on the jobs graph.
<figure><img alt="Clustering a graph" src="https://cdn-images-1.medium.com/max/440/0*HPvRN4XnJFfBX0w4"><figcaption>A clustered graph of job titles</figcaption></figure>With this clustered representation we can have a single representation of a set of job titles with a set of similar skills.
Step 4: Ranking job clusters
Finally, we modified the Page and Brin algorithm from 1999 and created our own ranking algorithm across the skills-jobs network using a discrete-time Markov Chain. Now we can take into account how important skills are for a given job (how frequently they are mentioned) — and extend that to clusters of similar jobs.
Now when an applicant lights up some of the skills in the original data set, we are able to produce a probabilistic rank of jobs clusters and produce meaningful importance ranked output based on continuously-updated historical data.
What about algorithm bias?
We need to address the issue of potential bias in the algorithm. This is a massive topic for any recruitment or applicant matching algorithm. While this approach is not immune, the good news is that because the training data is not based on individual applicants we side-step the biggest single source of bias in job-matching — that of prior applicant selection bias.
So where can bias occur? Well two main areas:
<ol>
<li>The job descriptions themselves might contain biased perspective on skills, but unless this is a sector-wide problem, individual posting will get amalgamated with other jobs from the same sector and the effect of the bias will be significantly diluted if not removed altogether.</li>
<li>The candidate applying may be self-selective about how they describe their own abilities. However the algorithm looks for jobs that match partially and “fills in the blanks” for missing skills. This then gets displayed in the percentage match metric.</li>
</ol>
Summary
We have presented a brief overview of our approach to tackling the job search problem. We had a lot of fun doing it and were able to deploy on a GCP-BigQuery-Dataflow architecture two months after starting the project.
If you are interested in joining us for exciting projects like this or just finding out how your company can benefit from our Data Science and Engineering expertise, <a href="mailto:%20info@datasparq.ai">please get in touch</a>. We’ll be delighted to hear from you.
Huge credit goes to the team from Code First Girls who brilliantly innovated on the data science behind the Job Finder Machine with us at <a href="https://datasparq.ai/">DataSparQ</a>: <a href="https://www.linkedin.com/in/erika-nicole-gravina-42a72b176/">Erika Gravina</a>, <a href="https://www.linkedin.com/in/dehaja-senanayake/">Dehaja Senanajaka</a> and <a href="https://www.linkedin.com/in/rajwinderbhatoe/">Rajwinder Bhatoe</a>.
References
[1] The Job Finder Machine: <a href="https://labs.datasparq.ai/job-finder-machine/">https://labs.datasparq.ai/job-finder-machine/</a>
[2] Page, L., Brin, S., Motwani, R., and Winograd, T. <a href="http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf">The PageRank Citation Ranking: Bringing Order to the Web</a>. Technical Report. Stanford InfoLab. 1999.
[3] Oliver Bartlett. <a href="https://medium.com/@oliverbartlett/1a7ffe6aa7ff">Using Data Science to help people get back into work</a>. Medium, 2021.
[4] <a href="https://en.wikipedia.org/wiki/Hierarchical_clustering_of_networks">Hierarchical clustering of networks</a>. Wikipedia, April 2021.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=ad6f7f0fc358" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/job-matching-a-new-challenge-in-search-ad6f7f0fc358">Job matching: a new challenge in search</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Job matching: a new challenge in search

<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xSP0_BOiR-iv6tVCXLidOQ.jpeg"><figcaption>But some industries aren’t</figcaption></figure><a href="https://www.datasparq.ai/">DataSparQ</a> recently worked with <a href="http://reed.co.uk/">Reed</a>, <a href="https://codefirstgirls.org.uk/">Code First Girls</a> and <a href="https://cloud.google.com/">Google</a> as part of the <a href="https://emergentalliance.org/">Emergent Alliance</a> to help people get back into work post Covid.
One of the challenges put to the Emergent Alliance by Reed was to assess ways in which job and skills data might be used to help those made redundant during the pandemic get back into work.
<a href="https://www.ons.gov.uk/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/bulletins/uklabourmarket/april2021#employment-unemployment-and-economic-inactivity">Figures from the ONS</a> show UK unemployment at 4.9% in April 2021 (up from 4% a year earlier) with hospitality, retail and entertainment sectors especially impacted. Of around 800,000 payroll jobs to disappear since March 2020, 526,000 are from retail and hospitality alone.
What’s more, for some specific sectors like high street retail, it’s not clear how quickly (if ever) employment levels will return to their pre-covid levels.
For those who have been made redundant through this period there are a couple of significant challenges Reed identified:
(a) People whose sector or industry has been significantly disrupted may not know where to look for their next role
and
(b) they may be missing key skills which would allow them to move into other roles or sectors (and may not be aware what those skills are).
We set out to develop a proof of concept (PoC) that would demonstrate how job and skills data can be used to help solve these problems. Head over to the <a href="https://labs.datasparq.ai/job-finder-machine/">Job Finder Machine</a> to have a play.
So what’s going on here? To satisfy challenge (a), we wanted to reimagine job search as a less targeted activity. For example, rather than asking a search system what “Hospitality” jobs were available in your area, the idea was to allow people to tell the service what skills they have, and to have relevant roles suggested back. This way, search results would no longer be constrained to the roles or sectors the jobseeker was familiar with. We also wanted to experiment with the idea of returning generic job titles, rather than specific roles, to see if this could be a useful pre-search tool for job seekers to identify new roles or sectors before using more traditional job search tools.
The underlying data science going on here is much better <a href="https://medium.com/@jeremytbradley/ad6f7f0fc358">explained by Jeremy in his post on the project</a>, but at a high level we analysed thousands of job specifications to build a model which clustered job roles according to their skills. The job roles are returned according to how closely the skills required match the skills entered.
The approach has a number of benefits over a free text search, including:
<ul>
<li>Free text search will only return roles whose description uses the same words as have been entered into the search. Because our approach has learnt from many different descriptions of a role it means we know that a role which asks for a “team player” is likely to be suitable for someone who describes themselves as “collaborative”.</li>
<li>We can reduce the impact of <a href="https://www.totaljobs.com/insidejob/gender-bias-decoder/">gendered terms</a> when either used in job descriptions or as inputted skills. The model understands the similarity between skills and therefore it’s possible to automatically add non gendered (or oppositely gendered) skills to any input to increase the breadth of roles returned (this isn’t yet implemented in the machine).</li>
<li>The underlying model provides a navigable graph which could allow jobseekers to browse roles by skills in order to help them explore further outside of their normal comfort zone. For example, you could imagine for any roles presented back to the user, there could be another 5 roles which require 1 additional skill. Clicking on one of these would open up another 5 roles and so on.</li>
</ul>
The skills gaps problem (b) is something we didn’t have time to implement in the proof of concept, but the approach supports it. For example, it would be relatively easy to identify the top roles that require 1,2 or 3 more skills in order to match the user’s input, and then display those jobs in the results along with highlighted skills gaps.
The PoC has demonstrated that a relatively simple dataset can produce a data graph which provides a powerful, intuitive way to find jobs, while also opening up the opportunity to develop innovative new capabilities like the gender-bias-reducer or graph browsing.
If you’re interested in discussing how the work done here might support challenges you’re facing with recruitment, then please <a href="https://www.datasparq.ai/contact">get in touch</a>, or you can contact the Emergent Alliance through <a href="https://emergentalliance.org/products-and-services/job-finder-machine/">the feedback form on this page</a>.
About: The <a href="https://emergentalliance.org/">Emergent Alliance</a> was set up to aid post Covid Recovery through developing data-driven insight and tools to be made available to businesses and government. <a href="https://datasparq.ai/">DataSparQ</a> contributed engineering, data science and product management effort towards the <a href="https://emergentalliance.org/challenge-themes/labour-market-and-how-we-work/regional-re-skill-redeployment-challenge-statement/">Regional Re-skill and Redeployment challenge</a> submitted by <a href="https://www.reed.co.uk/">Reed</a>.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=1a7ffe6aa7ff" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/using-data-science-to-help-people-get-back-into-work-1a7ffe6aa7ff">Using Data Science to help people get back into work</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Using Data Science to help people get back into work

Anomalies and outliers and how to find them.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IPChOJ-xy78-HPr1.jpg"></figure>When I started to write this article the first thing I did was to look for formal definitions of anomaly and outlier. Turns out that there isn’t a consensus on the matter. Every field would have (annoying) a slightly different opinion:
<ul>
<li>Statisticians, for the most part, will use the two terms interchangeably. [<a href="http://dx.doi.org/10.1007/978-3-319-47578-3_1">ref</a>]</li>
<li>Climatologists would say that a (temperature) anomaly is the difference between value and the mean [<a href="https://www.ncdc.noaa.gov/monitoring-references/faq/anomalies.php">ref</a>]</li>
<li>In manufacturing, anomalies are defects (I have not seen anyone use the term outlier)</li>
<li>In banking and insurance anomalies are synonymous with fraud (but in some papers, they would use the term outlier interchangeably)</li>
<li>And don’t ask physicists on their opinion, unless you understand <a href="https://en.wikipedia.org/wiki/Anomaly_(physics)">quantum field theory</a> …</li>
</ul>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*Z3w3XP15GekuZe50FValPQ.png"><figcaption>Manufacturing defects (anomalies). Source:<a href="https://arxiv.org/abs/2102.01882">Kamoona et al.</a></figcaption></figure>At the risk of annoying some people, I’d like to draw a distinct line between anomaly and outlier in the context of Machine Learning. They are not original definitions, merely an attempt to translate the opinion of much more intelligent and experienced people, to simpler terms.
<h3>Outlier</h3>
<blockquote>An outlier is an improbable data point</blockquote>
Imagine that you have a shoe store and would like to know how many shoes of each size you need to order. You call your friendly neighborhood data scientists. He pulls some data and sends you the following:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*gYhB37eiKOQtGwJ6fB4gBg.png"><figcaption>Syntetic data (does not represent reality)</figcaption></figure>That looks exactly like you expected — a nice normal distribution; sizes 40and 41 being the most common. But wait, you look at the table to the right and see something really odd — size 50. You have been selling shoes for years, and have never heard of such monstrosity. You call back your data scientist and ask for your money back, because the data is incorrect. The friendly data scientists then explains to you that:
<blockquote>“1 in 100,000 people has a size 50 shoes.That is very rare, but still possible. That is an outlier!”
</blockquote>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-bqetyg1BRyhOHu6BBCT_Q.jpeg"><figcaption>A 6ft 5in bride-to-be with size 15 feet is unable to find wedding shoes big enough. Source: <a href="https://www.thesun.co.uk/fabulous/7882473/bride-big-feet-shoes-wedding/">The Sun</a></figcaption></figure><h3>Anomaly</h3>
<blockquote>An anomaly is something created by a different process entirely. There is a different underlying function or a distribution.</blockquote>
Back in the shoe store, we now want to run a campaign where we give you a free pair of jeans with every shoe purchase. We want to make sure that we order the correct jean sizes and tie those to shoe sizes (please just roll with it …).
<figure><img alt="" src="https://cdn-images-1.medium.com/max/390/0*jAFeQRMejtRj94bU.gif"><figcaption><a href="http://gph.is/1OYW0I0">Source</a></figcaption></figure>So who you gonna call? That’s right, back we go to that overworked and slightly underpaid data scientist. He now sends us something a bit more complicated:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*EGt5JNQkRWqWAhxs8HfFAA.png"><figcaption>Linear regression of shoe size and height</figcaption></figure>This is a scatter plot that shows how shoe size and height are correlated. The line in middle is a linear model that shows a very high correlation between the two variables.
However, your keen eye has spotted something wrong with this picture. Let me use the power of colour and show you what I am talking about:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*8GD8mE2YUYNQa7n_rPDlKQ.png"></figure>There are two orange dots on the chart that seem a bit out of place. I have pasted them below for closer investigation.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*wMj4CnspSnkxv2hY9ukI3g.png"></figure><ul>
<li>Both shoe sizes 35 and 47 are within our expected range.</li>
<li>Heights of 158 and 183 are also quite normal
</li>
</ul>
<blockquote>It is almost impossible for a human to be 183 cm tall and have size 35shoes. Or be158 cm with size 47 shoes (unless he is called Sideshow Bob). That is an anomaly.</blockquote>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*FrZL9usMmvTUnSUi.gif"><figcaption>Everyone knows that Sideshow Bob mas massive feet. <a href="http://gph.is/1TvNnp2">Source</a></figcaption></figure>Lets look a bit closer at the people that wear size 35 :
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*LMXTbD-HGiy2DvX0lvtK9w.png"></figure>As you can see a height of 183 is more than 10cm than the largest value in this group (shoe size 35). That data point is not realistically part of the population.
<h4>Recap</h4>
When looking at shoes size or height individually (univariate), we only see outliers — values that are rare but still probable. Analyzing the dataset as a whole (multivariate) we can see combination of values that are anomalies.
<h3>Anomaly detection</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/603/0*NN5M4j1j0AmGhcac.gif"><figcaption>No, not that kind of anomaly. <a href="http://gph.is/2vgaNul">Source</a></figcaption></figure>Now that we (hopefully) understand how this works, let’s take it up a notch and go looking for anomalies in a dataset with 3 variables. I will kick in an extra variable called weight (to our previous dataset) and boot out the shoe store allegory.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YnL5hHoye09ZDvoNBufwDQ.png"><figcaption>Pairplot that shows relationship between all the variables</figcaption></figure>These kind of plots are very useful to get a first glimpse into how the different variables interact with each other.:
<ul>
<li>All three variables are normally distributed</li>
<li>shoe_size and height are positively correlated</li>
<li>weight and height are also positively correlated</li>
<li>shoe_size and weight are not correlated</li>
</ul>
We can try and find the anomalies like we did in the previous section, but that will take a while. There we had one scatter plot that we could visually inspect, now we have three. A more systematic approach is required if you don’t want to spend the rest of the day staring at dots and histograms.
<h3>PCA</h3>
Enter <a href="https://www.sartorius.com/en/knowledge/science-snippets/what-is-principal-component-analysis-pca-and-how-it-is-used-507186">Principal Component Analysis</a>. I am not going to go into details about it, but you can follow the link if you want to know more or need a refresher.
<blockquote>
PCA uses fancy algebra to take any number of variables (dimensions) and reduce them to a more manageable number without losing much of the information stored in the data.</blockquote>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*9eyHMZIrbMI5fRBV.gif"></figure>In our example we take the 3 variables and reduce them 2 and we get to keep 99% of the explanatory power. Needless to say this is much more impressive (and useful) when you have even more variables, but let’s keep it simple.
One of the upsides of PCA is that you can more easily visualize your data. Now we can much more easily see those outlier/anomalies:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1gw7C8XHeC9WR-5_0-BS5Q.png"></figure>But before you get your crayons out and start colouring in those dots let me show you another trick from the data science hat. We are really deep into the rabbits hole now, please stay with me, we are almost in Wonderland.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/513/0*IkcEkcEQKZaExtVr.gif"><figcaption><a href="http://gph.is/1LQnssc">Source</a></figcaption></figure><h3>DBSCAN</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*c60aVwNf8mdOMnyt.gif"><figcaption>No, not that kind of scan. <a href="https://gph.is/2CDGoLz">Source</a></figcaption></figure><a href="https://scikit-learn.org/stable/modules/clustering.html#dbscan">DBSCAN</a> is an unsupervised clustering algorithm (you can get a nice intro <a href="https://towardsdatascience.com/dbscan-clustering-explained-97556a2ad556">here</a>). We can feed it the PCA data and some basic rules and have it figure out which is the main cluster. It does that by figuring out how close each point is to its neighbors. If it’s surrounded by lots of them — it’s part of a cluster. If a data point is too far — it’s an anomaly.
DBSCAN().labels_ returns an array where each cluster is given a number, starting from 0 and anything that doesn’t fit into a cluster is -1 . Since our dataset is very uniform, we only have one cluster.
In the image below you can see the PCA plot from before, with the anomalies coloured in salmon.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*60rtZJXZZNXpeaEB7sFmqw.png"></figure>The hardest thing in this kind of analysis is always finding a good boundary. And that is more of an art than a science despite the plethora of statistical and ML methods swimming out there.
For the sake of argument let’s agree that our salmon dots are quite the catch and move to the next step.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2epBcSty_2rpoqJd"><figcaption>Photo by <a href="https://unsplash.com/@noaa?utm_source=medium&amp;utm_medium=referral">NOAA</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure>PC1 and PC2 are not very useful numbers for us mortals, so we need to go back to the original data. Below we add the anomalies (DBSCAN clsuter) to the pairplot from before:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y7M58O3-SsW19XW1zwBe0A.png"></figure>With this visualization it is much easier to explain why certain points are anomalies.
What is even more interesting is that we have found anomalies that we would have missed if we used the method from the first part of the article. Look at the anomaly data points in the bottom left scatter plot (weight~height). We could have probably found the ones that are visually outside the main “cluster” (distribution), but we would have missed the ones that are “inside” (with a higher frequence/probability).
<figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*ylO30fdvTYuDXjKD.gif"><figcaption><a href="https://gph.is/g/4LN0wvL">Source</a></figcaption></figure>And for the viewing pleasure of the spreadsheet nerds, you can find a random sample of the anomalies and their values below:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*W_d5s5tMMa4vHm2jrvghPg.png"><figcaption>sample of 20 anomalies (out of 39)</figcaption></figure>Let’s analyze some of them:
<ul>
<li>76283 (2nd row) — height and shoe size seem acceptable, but the weight is too low</li>
<li>100024 (last row) — height and weight are fine, but the shoe size seems proposterous</li>
<li>100001 (somewhere in the middle) — that observation is the definition of anomaly. 184cm tall at 64 kg and size 35 feet? Sounds like a Slenderman to me.</li>
</ul>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*4gwa2QocKvJLEAFj.gif"><figcaption>Slenderman. <a href="http://gph.is/1c3ufZ3">Source</a></figcaption></figure><h4>Final words</h4>
<ul>
<li>The dataset I used was hand crafted by me for the purposes of this article, so don’t go around using those graphs to draw conclusions about the real world.</li>
<li>Real datasets are much more complicated and messy. Having a good grasp of the process or business logic behind them will go a long way in the quest of anomaly detection.</li>
<li>The notebook(s) for this article can be found <a href="https://github.com/ivan-sparq/blog-artefacts/tree/main/AnomalyDetection/notebooks">here</a>
</li>
</ul>
Hope you enjoyed reading this as much as me writing it. Questions and comments are as always, highly encouraged.
<h4>Sources:</h4>
<ul>
<li><a href="https://stats.stackexchange.com/questions/189664/difference-between-anomaly-and-outlier">https://stats.stackexchange.com/questions/189664/difference-between-anomaly-and-outlier</a></li>
<li><a href="https://akajuvonen.github.io/2017/10/31/outlier-vs-anomaly.html">https://akajuvonen.github.io/2017/10/31/outlier-vs-anomaly.html</a></li>
<li><a href="https://arxiv.org/abs/2102.01882">https://arxiv.org/abs/2102.01882</a></li>
</ul>
<ul>
<li><a href="https://medium.com/@mehulved1503/outlier-detection-and-anomaly-detection-with-machine-learning-caa96b34b7f6">Outlier Detection and Anomaly Detection with Machine Learning</a></li>
<li><a href="https://towardsdatascience.com/anomaly-detection-for-dummies-15f148e559c1">Anomaly Detection for Dummies</a></li>
<li><a href="https://towardsdatascience.com/a-brief-overview-of-outlier-detection-techniques-1e0b2c19e561">A Brief Overview of Outlier Detection Techniques</a></li>
<li><a href="https://towardsdatascience.com/outlier-detection-with-isolation-forest-3d190448d45e">Outlier Detection with Isolation Forest</a></li>
<li><a href="https://towardsdatascience.com/outlier-detection-theory-visualizations-and-code-a4fd39de540c">Outlier Detection — Theory, Visualizations, and Code</a></li>
<li><a href="https://towardsdatascience.com/5-ways-to-detect-outliers-that-every-data-scientist-should-know-python-code-70a54335a623">5 Ways to Detect Outliers That Every Data Scientist Should Know (Python Code)</a></li>
<li><a href="https://towardsdatascience.com/dbscan-clustering-explained-97556a2ad556">DBSCAN Clustering — Explained</a></li>
</ul>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=640c949d206" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/anomaly-detection-for-beginners-640c949d206">Anomaly Detection for Beginners</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Anomaly Detection for Beginners

<blockquote>An economy that puts back what it takes out, that is sustainable by design. — Matthew Bishop
</blockquote>
The circular economy is a foundational principal of sustainable commercial and industrial activity.
Circular economy requires that we account for the natural resources we take and use in everyday processes and, as far as possible, return them to their original state — everything from air and water to habitats. Further, every product — every single product — we consume needs to have a reuse or recycling pathway either directly or in an adjacent process or industry.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ek_90EPKNxiLOdcj.png"><figcaption>The circular economy in action (Geissdoerfer et al. 2020)</figcaption></figure>Be in no doubt, the transformation required by the circular economy will make the industrial revolution and the data revolution look like a walk in the park by comparison!
What does circularity entail
<blockquote>In its industrial processes, a truly circular economy creates no net pollution or waste. — Matthew Bishop
</blockquote>
Given this stringent definition, the goal of achieving circular industrial processes across a national and by extension, for it to make sense, a trans-national economy, is a huge challenge. A social challenge certainly, a commercial and investment challenge definitely, but also I argue, a data challenge and and a massive scientific challenge as well.
Circularity subsumes the carbon cycle neutrality that many nations are working towards and extends it towards other natural resource cycles such as water, air, minerals, and biodiversity. By definition though, circularity includes the products that are created in a thriving economy and all the materials that go into making them.
If circularity is to happen — and for ongoing sustainability it surely has to — this requires the a dramatic and disruptive change to the global economy.
Changing production/logistic networks
Currently many products and processes revolve around a linear production model. This does not work well as a model for reclaiming materials for recycling and reuse as many products end up in landfill.
<figure><img alt="A linear production model which see products go from manufacture to waste disposal at end-of-life" src="https://cdn-images-1.medium.com/max/414/1*yZRh5PJDxCnEApshDe1IWQ.png"><figcaption>A linear production model that sees products go from manufacture to disposal (jb, 2021)</figcaption></figure>A linear production model involves a supply chain of raw materials that see a product created at a production facility and then pushed into a supply chain distribution network. The use-life will sometimes have maintenance cycles embedded in it if the product is of sufficiently high value. When end-of-life is reached the product is disposed to landfill or incineration with some portion of the product being recycled depending on legislation, consumer habit and process availability.
For new products designed for circularity, we will see many different and potentially disruptive production and use models spring up.
<figure><img alt="A cyclic production-use-recycling process" src="https://cdn-images-1.medium.com/max/385/1*skKHxkUjhVqXfjUAtvGP2A.png"><figcaption>A cyclic production model where the production is located with or nearby the point-of-sale (jb, 2021)</figcaption></figure>Future circular logistic chains may see production — perhaps micro-production — move closer to the point-of-sale. Imagine bottling plants or food packing facilities embedded in supermarkets.
Similarly some outlets will become points-of-return for their products at end-of-life. Except that, what was previously end-of-life will become start-of-second-life after a suitable recycling process.
The distribution network will become a means for front-hauling raw components to the manufacturing facility and taking packaging and old products for refurbishment — so-called, back-haul.
All this requires unprecedented cooperation between logistics providers, manufacturers retailers and consumers. Undoubtedly incentive programs like bottle deposit schemes will be needed and the economics of even those industry schemes are not straightforward. For more expensive products, lease-and-return will likely be the better model replacing own-and-discard.
The scientific and engineering challenge
Let us for a moment look at the industries that will be swept up in this revolution:
<ul>
<li>Building and construction</li>
<li>Clothing and textiles</li>
<li>Metals (bulk metals like, iron and aluminium) as well as tech metals such as rare earth metals, copper and lithium</li>
<li>Farming and food production</li>
<li>Energy generation</li>
<li>Water processing</li>
</ul>
This is a scientific challenge at a materials level, and an engineering level at a process design level. Assuming we have a circular supply chain in place (see above) to collect the raw material: Can we physically or chemically reconstitute the materials once they have been used? Does an industrial scale process exist to support such a reclamation? Is it economically viable to carry out the recycling activity?
<figure><img alt="Recycling box full of glass bottles." src="https://cdn-images-1.medium.com/max/1024/0*ilTGUJZZE3WMQLNY.jpg"></figure>As an instance, thermal recycling of glass (melting recycled glass down and reforming it) cannot be done currently with 100% used feedstock or cullet (the name given in the industry to the broken glass obtained from bottle banks). It has to be mixed with new silica/soda-lime in order to produce a suitably high quality product (green glass can be formed from 90% feedstock, clear glass only 25%). Different but equally hard challenges exist in recycling PET plastics and other raw materials.
These are massive challenges by themselves and the good news that many research funders and universities are making inroads into these challenges.
Why a data challenge?
If this wasn’t exciting enough, the cross industry aspect is where the circular economy becomes a really interesting puzzle.
<blockquote>There is no such thing as ‘away’. When we throw anything away it must go somewhere. — Annie Leonard
</blockquote>
The challenge lies in joining up industries that have previously had no interrelationship.
For the sake of argument, say it were possible (I’ve no idea whether it is) to reuse the magnets in the motors of discarded washing machines and tumble dryers to help create generators for wind turbines and new motors for electric cars. This would be an example of how the white goods industry might find itself in the supply chain for fast growing green technologies.
If we now forbid those growing industries to use virgin raw materials from usually mined ores — we run into two problems straight away:
<ol>
<li>Are there enough discarded washing machines and tumble dryers to supply all the magnets need by the wind turbine and electric car industries</li>
<li>Are the recycled materials available in sufficient quantity at the right time to allow manufacturers to rely on them in their supply chains</li>
</ol>
Even if the answer to (1) is yes, the answer to (2) is almost certainly going to be no — at least to start with. These industries have never had cause to synchronise before — they are not aware of each others production flows, but they will need to be if there is to be synergy between them. This is where data science and smart supply chain processes can drive real innovation.
Data flows for circular economy
In supply chains, products, components and raw materials flow between mines, factories, warehouses and points of sale. In efficient supply chains, goods do not spend long at any one point in the supply chain, instead they are ordered as needed and in anticipation of a downstream demand.
What drives this efficient flow is data pipelining and accurate forecasting of demand. Without that, materials end up being stock piled in warehouses to mitigate the risk of material exhaustion. Storage of materials, just as with components, is inefficient and costly.
In a circular economy with many cross-industry dependencies, we will need data visibility on:
<ul>
<li>demand forecasts in consuming industries; and</li>
<li>flows of products reaching end-of-first-life in feeder industries</li>
<li>volumes of recycled material that is about to be available for reuse</li>
</ul>
Industries which have never talked to each other will need standardised APIs for interrogating each other or pushing notifications to each other to achieve this seamless integration.
Algorithms for the circular economy
If this sounds like it will be ripe for application of predictive data science, it absolutely will be. The better the forecast or anticipation of arrival of a feedstock, down to the day, hour or even minute, the more efficient the manufacture of the new product can be. Materials arriving early can at least be stored (if space allows) — materials arriving late though will potentially hold up a whole manufacturing pipeline.
Another key technology will be in the area of meta-heuristics, e.g. Silva et al. 2003. If you’ve never heard of these before in the context of data science problems, they are approximate but fast solution approaches to massive multi-scale optimisation problems. And if we are to minimise the wait time, maximise the throughput and minimise the waste in dynamic cross-industry supply chains, we will need the very latest state-of-the-art optimisation algorithms to enable this.
Models of flow of goods than can capture recycling streams at a cross industry level will be extremely important, for example, Bhaskar et al. 2010. These models will need to represent delay, storage and uncertainty but they will also need to learn distributional patterns that represent volume. Being able to identify key bottlenecks well ahead of them actually occurring will be necessary so that alternative sources or feedstocks can be planned for.
Conclusion
So what can we do now to address these challenges? For those of us who look for data-driven projects in a variety of industries — there is low-hanging fruit and much of the benefit can start within a company.
<ul>
<li>understanding and mapping the data flows and data services within a company is a health check that many companies could do and leave for another day</li>
<li>creating a sustainability call-to-action within a company which looks at the flows of natural resources as well as raw products and would allow for an audit of natural capital expenditure will likely expose circularity shortfalls</li>
<li>engaging with the data science to create high quality forecasts for many parts of the company to use and understanding the key impacts and risks of that forecast being right or wrong</li>
<li>engaging with universities and <a href="https://www.ukri.org/news/hub-to-provide-national-circular-economy-leadership/">circular economy hubs</a> at this early stage so that a company is aware of upcoming technical and business opportunities</li>
<li>creating an open data culture so that people who want to work with you can see your consumption and your production and forecasts of the same so they can understand what the time and volume challenge will be if they start to be your supplier/consumer for a critical material</li>
</ul>
If sustainability is to be sustainable then it has to drive value itself and the good news is that the actions that any company can take in preparation for the circular revolution will also drive efficiency, savings, internal reliability and external opportunities.
References
Geissdoerfer, M., Pieroni, M.P., Pigosso, D.C. and Soufani, K. “Circular business models: A review.” Journal of Cleaner Production, 2020.
Silva, C.A., Runkler, T.A., Sousa, J.M., Sá da Costa, J.M. “Optimization of Logistic Processes in Supply-Chains Using Meta-heuristics.” EPIA 2003: Progress in Artificial Intelligence. LNCS vol. 2902, pp 9–23. Dec. 2003.
Bhaskar, V., Lallement, P. “Modeling a supply chain using a network of queues.” Applied Mathematical Modelling 34(8):2074–2088, Aug. 2010. DOI: 10.1016/j.apm.2009.10.019
About the author
Dr. Jeremy Bradley is Chief Data Scientist to DataSparQ, a member of the Strategic Advisory Network for the UKRI/EPSRC Circular Economy Theme, a member of the Strategic Advisory Team for EPSRC Mathematical Sciences and Fellow of the Institute of Mathematics and its Applications. He lectured and researched Computer Science at Imperial College London for 14 years. He has industrial data science experience as part of the data science teams at Tesco, Royal Mail and now DataSparQ. Jeremy also co-hosts the <a href="http://datacafe.uk/">DataCafé data science podcast</a> with his friend Dr. Jason Byrne.
If you would like to know more about how sustainable data science can help your business then do <a href="mailto:info@datasparq.ai">get in touch</a>. We would be happy to share our learnings and see if we can help.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=9653a29d0c55" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/the-next-revolution-data-science-and-the-circular-economy-9653a29d0c55">The next revolution: Data Science and the Circular Economy</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Who we are

What we do

Work we've done

Contact us

Welcome

Turn data into better

Perform beyond your potential with AI and data solutions from Datasparq.

Who we work with

Our clients and partners

What we do

Data science meets design thinking

How we do it

Turning data into better solutions to real-life problems

Turning in-store purchase data into better retail locations

Turning traffic data into a better car parking experience

Turning salad sales data into less food waste

Turning project risk data into better use of tax-payers’ money

Turning supply and demand data into happier holidaymakers

Turning job vacancy data into better opportunities for jobseekers

Who we are

Passionate practitioners

Inspired? Join our team