Better organisations use AI and data to supercharge their people, allowing them to do things faster, cheaper, smarter. What does better mean for you?

Turn data into better

Our clients and partners

We identify opportunities for AI in your business and deliver solutions which transform day-to-day operations. This is about the practical application of data platforms, data science and machine learning to create real tools you’ll use every day to improve efficiency, effectiveness and, ultimately, create more value.

Data science meets design thinking

Turning data into better solutions to real-life problems

At Datasparq, you only get the A-team. We put together unique expertise for unique challenges, with the right blend of data science, product thinking, engineering and design.

Passionate practitioners

Datasparq people are smart, curious, skilled, experienced and creative. If you have what it takes to create data solutions that aren’t like everyone else’s, we’d love to hear from you.

Inspired? Join our team

Home

Welcome to Datasparq’s technology blog. At Datasparq, we design, deliver and run bespoke data science products to help organisations capture value from data. - Medium

Datasparq Technology - Medium

As technology and investment market trends converge, now is the time for PE firms to learn what AI really is.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mwnMr4VIcm2l4Dxka4ohhA.jpeg"><figcaption>Photo by <a href="https://unsplash.com/@yukato?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Yu Kato</a> on <a href="https://unsplash.com/s/photos/lens?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure>One clear trend in the investment world right now is the <a href="https://www.ft.com/content/cd9571a3-726c-4995-9954-23a8dcf12b19">growth of Private Equity deals</a>. The first half of 2021 has seen PE deals worth more than $500bn; helping to propel global mergers and acquisitions activity to an all-time high.
<a href="https://www.idc.com/getdoc.jsp?containerId=prUS47482321">The other trend is AI technology growth</a> — despite being around for decades, and growing significantly since the arrival of cloud computing earlier this millennium— AI is still growing. This trend isn’t particularly new, but it means that AI capabilities are becoming more common place in more businesses; and leading businesses are taking advantage of the benefits of AI to outperform their lagging competition.
Combining these trends, it’s no surprise that there’s an increasing amount of investment activity involving companies that have AI as a core component of their proposition and enterprise value. <a href="https://www.cbinsights.com/research/report/enterprise-ai-trends-2021/">AI companies raised a record $33 billion in equity funding in 2020</a>, and what’s more, an increasing number of companies claim to be “AI-first”; using cloud technology, data science and engineering to deploy AI into different aspects of their business to operate better.
Time for PE to learn about AI
If AI is deemed to be a crucial differentiator for a company’s proposition, or a major value lever given the rich, unique dataset the company has, then it’s important this is well understood when assessing the company’s valuation and investment potential. When considering a deal with an “AI-first” company, any investor needs to be clear on the existing and potential value of AI within that company.
<ul>
<li>Do buyers understand exactly what AI capabilities they’re buying?</li>
<li>How do buyers evaluate and leverage the claimed AI capabilities?</li>
<li>For data-rich companies that aren’t even claiming to use AI, how do buyers assess the potential value that AI could add to the acquisition?</li>
</ul>
Many investment brochures will advertise the organisation’s good data governance, rich data assets, scalable data platform, and leading AI capabilities that makes it a unique investment opportunity — but in many cases the truth may be it’s not so unique. After all, despite AI maturing in the last few years, it’s still like teenage sex — more companies are talking about it than actually doing it.
PE companies will use their preferred IT due-diligence and Commercial due-diligence partners to confirm a potential investment’s true capabilities, potential and risk; but these organisations rarely have the AI practitioner experience to robustly assess AI; and the remit of a typical IT DD and CDD exercise is too broad to provide any deep analysis within the AI domain.
The answer: AI Due-diligence
Given the increasing importance of rigorously evaluating a company’s AI capabilities before investing, and the lack of detail IT DD and CDD studies provide for this, DataSparQ is finding itself developing valuable AI Due-diligence reports on potential assets for PE investors. Whilst a PPT slide deck isn’t the traditional output of a DataSparQ project (we pride ourselves on developing and running operational AI solutions in the real world; our outputs are usually more Python than PPT), our deep AI practitioner expertise means we know what good looks like, and because we understand what it takes to deliver AI successfully in the real world, we’re in a privileged position to help others.
Our AI DD service consists of both a “backward-looking” confirmatory assessment of a potential investment’s AI capability, as well as a “forward-looking” analysis of how AI could be used to unlock further hidden value in that business. Different investments often have a different focus; for businesses with established AI capabilities that give them a unique competitive advantage, the confirmatory assessment is more important. For data-rich companies early in their analytics journey, the forward-looking analysis is often more valuable.
The areas of investigation during an AI DD include:
<ul>
<li>
Data diagnostic — a review of the data collected, stored and used by the business. Is it done systematically or ad-hoc? How accurate, relevant and complete is it? How is it catalogued and governed?</li>
<li>
Algorithm diagnostic — an assessment of the algorithms and analysis used. Are they appropriate (meeting requirements for accuracy, explainability, auditability, bias, customisation)? Are they robust? Will they scale and remain competitive as the business grows and the market matures? Are they compliant with industry rules and regulations?</li>
<li>
Technology diagnostic — a review of the technology systems and architecture used to build and run these AI capabilities. Are they resilient? Will they scale to support future growth requirements? Are they cost efficient? Are they secure?</li>
<li>
Capability diagnostic — an analysis of the business’s way of designing, building and running AI solutions. Do the right skill-sets exist? Is the capability unique and sustainable? Does the way of working encourage innovation?</li>
<li>
AI Opportunity review — a process to identify, validate and quantify high-value opportunities for AI yet to be exploited. What hidden value levers can AI unlock? What incremental value does AI offer over a more low-tech solution? How feasible is each opportunity? What’s a realistic delivery timetable for value capture?</li>
</ul>
Given a modest amount of time with the management team, and other key individuals in the business who might not normally be involved in traditional DD exercises, these questions can be answered within the timelines for most DD processes. It can be efficient for the AI DD provider to work alongside the IT DD and CDD providers so the same questions don’t have to be asked twice.
Bottom line
As AI matures and becomes a differentiating value lever in organisations, it’s increasingly important for investors to put this capability under the microscope when considering deals. This means PE companies need to start asking new questions, and learning more about AI than the buzzwords and headlines in order to know what questions to ask, let alone answer them.
The innovation of AI technologies and methods is fast and shows no sign of slowing. It will be an ongoing learning journey for investors to ensure their assessments remain relevant. As a result, AI practitioners like us may find ourselves increasingly doing PPT as much as Python as we become the go-to service providers for AI DD.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=e2da04f94b8e" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/why-private-equity-needs-to-learn-about-ai-to-make-better-investment-decisions-e2da04f94b8e">Why Private Equity needs to learn about AI to make better investment decisions</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why Private Equity needs to learn about AI to make better investment decisions

Humans don’t like risk and, as with any technology investment, a Machine Learning solution comes with its own set of risks. Will it predict the right thing, will it be accurate, will it be useable? In this post I’m going to take you through how we at Datasparq reduce risk through Proof of Value.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*IwpfqunBgcA0693VDTVP4g.jpeg"><figcaption>A machine learning solution to demand forecasting can lead to pretty, ordered, stocked shelves… and higher revenues! But will it work?</figcaption></figure>Our machine learning solutions are designed to allow people to perform beyond their potential, whether that’s making more accurate sales forecasts, creating effective retention strategies for customers, or focusing business development effort on leads likely to convert to wins. Machine learning solutions give people the benefit of the predictive power hidden away in data.
Machine learning has proven time and again to be effective at predicting the future by learning from the past, whether that’s <a href="https://www.bbc.co.uk/news/technology-57157566">identifying potential skin cancer</a>, or <a href="https://towardsdatascience.com/deep-dive-into-netflixs-recommender-system-341806ae3b48">helping you decide what to watch next on TV</a>. That’s not entirely surprising as these solutions are fundamentally doing what humans have evolved to do — observe, learn and predict, just with the advantage of being able to process vast amounts more data.
But just because there’s evidence something has worked before doesn’t necessarily mean it’s going to work in every situation.
This is why we start all of our machine learning projects with a Prove phase. It’s a short, targeted mission to reduce ambiguity and test assumptions to minimise risk. Over the course of a few weeks, we validate or test the following:
<ol>
<li>That there is a clear target variable for the prediction - we know what we’re predicting.</li>
<li>That there is a viable means for measuring value from the prediction</li>
<li>That there is predictive power in the data (signal in the noise)</li>
</ol>
1. Clear target variable for the prediction
The target variable is the variable in a dataset whose value is to be predicted using other variables. It’s pretty fundamental to any machine learning solution, and picking the wrong target variable can lead to significant wasted effort down the line.
Our goal here is to define which fields in the data will be used to form the ‘target variable’. This is an iterative process where we identify a target variable, then validate how a prediction of this target variable will be beneficial. This in turn raises new questions which allows us to loop back and idenfity a better target variable.
It sounds like it should be easy. Let’s look at demand forecasting as an example. Imagine you sell jeans in a high street store. In order to effectively manage your stock levels you need to make an estimate in advance of how many you’re going to sell. So a machine learning solution just needs to predict your sales. Simple, right?
In reality, defining the scope of prediction tends to be a significant chunk of work that requires us to deep dive into an organisation’s processes and value chain.
<ul>
<li>What product or service are we predicting demand for? Is it one product, or hundreds, each with different demand patterns?</li>
<li>For what period are we predicting demand — next week, month, year? Or maybe combinations of these, or maybe different products would benefit from different levels of prediction?</li>
<li>What is ‘demand’? Is it a placed order? A product taken off a shelf? What if that order was later cancelled, returned, or not paid for?</li>
<li>Do we need to care about who is making the demand — do we need to predict demand from a specific customer or location, or a demand that will be fulfilled by a specific fulfilment centre?</li>
<li>How far in advance would the prediction be required in order to be valuable to the business? Predicting tomorrow’s demand today is much easier than predicting for a day next month.</li>
</ul>
Some of these questions (the known unknowns) will have been answered prior to the Prove phase as part of our <a href="https://www.datasparq.ai/products/">SparQshop™</a>, others arise from deeper exploration of the data (the unknown unknowns!). All need to be answered if we’re to precisely define the target variable.
2. Measuring value
In order to optimise a model we need to know what metrics we’re using to measure performance. Understanding the key performance metric gets to the heart of some of the difficult questions around model optimisation, for example:
<ul>
<li>In a regression problem (where we are predicting a number, like product sales), what’s the relative value in over predicting vs under predicting? Underpredicting by 10% might have a significantly bigger cost impact than overpredicting by 10% (you can’t sell what you don’t have, you may though be able to store some excess stock for selling later).</li>
<li>In a classification problem (where we are predicting a binary outcome, like customer churn), what’s the impact of a true positive vs a false positive? If we predict a customer to a subscription service is going to leave (churn), they may get offered a discount to stay. If they take up this discounted offer that’s great if they would have otherwise left. However, if it was a false positive prediction and they were actually never going to leave, it’s a cost to the business.</li>
</ul>
Then there may be other factors specific to the problem being solved. In the demand forecasting example, is it more important to accurately predict demand for some products over others (maybe some have a shorter shelf life, or are more expensive to keep in stock)?
The goal here is to come up with a metric to measure model performance which relates directly to the value being created for the business. For example, for customer churn, that metric may be:
Profit ($) = R*TP — C*FP
Where R is the revenue from a true positive, C is the cost of a false positive, and TP &amp; FP are the true positive and false positive rates.
3. Predictive power in the data
This is where we create a baseline machine learning model and test the predictive power inherent in the data. This starts with some basic evaluation of the data — specifically is there enough historical data to be able to train and test a model?
Next we need to know where to start with the data. We’ll typically run a hypothesis workshop with people in your organisation who understand the data and the target variable. In the case of demand forecasting this would involve those close to the buying/selling operations to understand, from their experience, what factors affect demand. For example, we may learn that they see sales go up in summer, or are higher at shops in certain locations. From these hypotheses, we extract <a href="https://en.wikipedia.org/wiki/Feature_(machine_learning)">features</a> from the data, for example “store location”, or “average temperature over the previous 7 days”.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*Uny-ym1ashJxGKzn1Gmwhw.png"><figcaption>thanks <a href="https://xkcd.com/license.html">XKCD</a></figcaption></figure>From here we can do some univariate analysis. Are there features which show predictive power on their own? <a href="https://www.tylervigen.com/spurious-correlations">Correlation ≠ causation</a> so success here isn’t proof of signal in the noise, but can help steer focus to the most useful areas of the data.
Once we have a handful of features we believe to be predictive, we can put them into a baseline model and test it. We measure the performance of the model by testing it against data it hasn’t seen. This is both out-of-time (given complete data up to 2018, can it predict what happened in 2019) and out-of-sample (given data for all time but removing data for certain shops, or products or price points at random, can it predict the gaps?).
The Prove phase isn’t designed to deliver a fully optimised model, our goal here is to demonstrate evidence that there is predictive power in the data and to get a sense of how much value might be gained through an optimised model. It’s not an exact science, but working closely with the subject matter experts gives the model the best chance of succeeding early on and it’s usually possible to prove value in this first phase, or at least get a good sense of whether it’s worth pursuing.
We’re good to go!
…not quite. The above activities bring clarity and definition to the model and it’s potential value but even the most accurate demand forecast is of absolutely no value unless it’s used.
The final piece we look at in the prove phase is how the solution will be used in operations. Are the stakeholders bought in to the idea of having a machine help with their jobs? Is is feasible to integrate the predictions with the systems/tools they use every day? What changes might be needed to operational processes once the predictions are live?
Even if these questions aren’t fully answered at this stage, having the discussion early on is another way to reduce risk.
If you think there may be an opportunity for machine learning to deliver value in your organisation and you’d like to chat about a low risk way to get started and prove the value, please get in touch at <a href="mailto:info@datasparq.ai">info@datasparq.ai</a>.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=e61f9700ef58" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/proving-the-value-of-machine-learning-e61f9700ef58">How to prove the value of machine learning and avoid worthless models</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to prove the value of machine learning and avoid worthless models

Well you probably do, it’s not that hard.
Rather than test your knowledge of the language, I’ll suggest seven different things you could do or learn next that will make you a better, more capable, Python developer. After all, it’s not the language itself that’s powerful, but how it’s used!
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2Uw3ZfvoggULk3qj"><figcaption>I promise I’m not gatekeeping (photo by <a href="https://unsplash.com/@lailagebhard?utm_source=medium&amp;utm_medium=referral">Laila Gebhard</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><h3>1. Master Unix and Learn (Some More) Bash</h3>
Hopefully you already love Bash more than Python and use it every day, but for many devs who started out with Python, it may never have been a major part of everyday operations. If Bash seems scary or clunky to you, I strongly recommend experimenting with some core commands and understanding loops, piping, and links.
The command line is the universal standard for getting programs to talk to each other, and if you can’t get your Python to talk to other programs, you’ll repeatedly hit barriers.
Bash has helped me get around awkward situations like not being able to use pip (in a serverless code execution environment), and not having admin rights on Windows (Git Bash was installed, thankfully).
You can always rely on Bash. With a strong grasp of it you can work around any problem. Not to mention you can run Bash commands with Python and vice versa.
<h3>2. Learn a tiny amount of HTML/JS/CSS</h3>
There are some things Python should not be used for, and Front End development is one of those things.
I’d also say the same about data visualisation. My favourite plotting libraries for Python (<a href="https://plotly.com/python/">Plotly</a> and <a href="https://bokeh.org/">Bokeh</a>) render charts within a browser, using JavaScript. These achieve far better results than Matplotlib in terms of appearance, ease of use, and interactivity.
JavaScript is even eating into Java and C#’s market share. <a href="https://www.electronjs.org/">Electron</a> allows you to build desktop applications with just HTML and JS (building on top of Chromium). It might surprise you to learn that MS Teams is an Electron app.
Don’t pigeonhole yourself as a purely back-end dev! Learn the basics of a framework like <a href="https://reactjs.org/docs/create-a-new-react-app.html#create-react-app">React</a> and start showing off how great your work is with some stunning user interfaces. Plus, putting some thought into the user experience of your app will help streamline your back-end code.
Best of all, you get to call yourself a Full Stack developer.
<h3>3. Use Docker to industrialise your code</h3>
<blockquote>“Well it works on my machine” -Bad Python developer</blockquote>
Hopefully this one seems obvious. A Python developer running everything on their laptop, or on a VM with a lengthy setup script, is not a good developer.
Code that provides value should be able to run on its own, without being babysat, and should be able to be deployed and redeployed by anyone, otherwise it will end up being thrown away as soon as the person who wrote it isn’t around.
The first step to industrialising code is to move the code and all dependencies into a container. Docker makes this fairly straightforward with Dockerfiles, and allows anyone to run the container on any machine, provided they have Docker installed.
You can then make use of serverless tools like Google’s Cloud Run or AWS Lambda, which will run your container on request without the need to provision compute resources. This is the easiest way to turn your code into an API, and allow anyone to use without having to install anything. From here you can plug it into a web front-end and reach a much larger audience.
I’ve seen many git repos from universities and hobbyists alike that contain some amazing code but have absolutely no thought put into industrialisation. This is a great shame since it prevents most people from being able to play with (and therefore potentially contribute to) the code. If they had only provided a requirements.txt and Dockerfile, someone might have taken it and created something truly mind-blowing.
Disclaimer: I’m a huge fan of Cloud Functions/Lambdas which don’t require any Docker knowledge, but some tasks require containers.
<h3>4. Learn PostgreSQL
</h3>
Everybody knows that Python is great for transforming data but that doesn’t mean it’s a great choice for an application backend or data pipeline. In fact, I would strongly discourage the use of Python (or pandas specifically) for any scenario in which a database could also be used. Database technologies like Postgres and MySQL have been designed to deal with data, and they do it really well.
Next time you think about joining two datasets with pandas, consider spinning up a PostgreSQL instance, importing your data into tables, and running a join on those tables (scripted by Python, using <a href="https://pypi.org/project/psycopg2/">psycopg2</a>). The setup may take longer, but when you find yourself needing to repeat the join again with updated data, you will see the benefits in terms of speed and reliability. You can then easily parameterise your SQL with Python formatting or Jinja2, massively reducing the amount of code needed to carry out all of your transformations.
SQL is dead easy to learn and Postgres is a great starting place. Once your data outgrows Postgres, you can apply the same knowledge to other, more powerful tools like Google BigQuery or Snowflake.
<h3>5. Don’t repeat yourself… ever again in your entire career</h3>
Have you ever copied code from one project to another? Perhaps you do this at the start of every project.
Save others from repeating your work! Make packages.
The first step is to collect together all the useful functions you use regularly. It might be best to group them up and make lots of separate packages to keep requirements simple.
Secondly make a <a href="https://docs.python.org/3/distutils/setupscript.html">setup.py</a> so that you can pip install your package locally.
Thirdly, and most importantly, put your package into a git repo and share it with your team*. Python packages can be pip installed directly from git repos like so:
<pre>pip install "git+ssh://git@github.com/&lt;username&gt;/&lt;repo&gt;.git"</pre>
Once you have a whole company using the same code across multiple projects, you start to see huge time savings both when starting new projects and when upgrading/fixing bugs in commonly used tools. I dread to think how long it would take us at DataSparQ to patch each of our projects separately if we weren’t using the same packages across all of them.
*Or, if you’re happy for it to be public, upload it to PyPI. That way you can save everyone from repeating your work!
<h3>6. Become a Git Expert</h3>
I’ve met developers who use Git every day and don’t actually understand it. As soon as something unexpected happens they don’t know what to do, and don’t know how to avoid problems in the first place.
Every team needs a Git expert. Someone who can confidently handle tricky merges, rebases, cherry picks, etc., and keep repos tidy.
If your releases are difficult and technical debt is piling up, the problem might be here. Make sure you understand branching strategies like Gitflow, and how to use Git + Agile + good documentation in the most complementary and efficient way.
<h3>7. Learn C to remove bottlenecks</h3>
I have come across problems that Python was too slow for. The solution was C, wrapped in Python (so you could say Python wasn’t too slow).
Ok you don’t need to learn C. I would suggest Go instead as it has a lot of support from cloud providers and is much easier to compile for different architectures, which was always the biggest pain with C for me.
Some of the most powerful packages like numpy, pillow, and TensorFlow are all just C modules wrapped in Python. That level of performance can also be achieved with very little effort these days. One could write a simple program in Go and call it from Python as part of a larger script, for example:
<pre>import subprocess</pre>
<pre>def run(cmd): p = subprocess.Popen(cmd, stdout=subprocess.PIPE, text=True) return list(iter(p.stdout.readline, "")) </pre>
<pre>x = run("count_emojis 🚀 --file wikipedia.zip")</pre>
If anything, writing a hello_world program in C is worthwhile purely to appreciate how easy Python is.
Disclaimer 1: I only suggest this method for simplicity. It doesn’t benefit from shared memory but is much less painful to write, and less likely to make Python crash.
Disclaimer 2: Python is definitely powerful enough to count emojis.
<h3>Conclusion</h3>
There are 0 job roles that require Python skills and nothing else. Branching out and learning complementary skills will make a huge difference to how effective your code can be.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=9c0d8dc8451e" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/so-you-think-you-know-python-9c0d8dc8451e">So you think you know Python?</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

So you think you know Python?

Synthetic datasets are boring. In most cases, we just generate them by drawing samples from a multivariate distribution with components from families of distributions that we have set previously, adding some noise at the end to make them more realistic. And of course, modelling a synthetic dataset is rather easy, as we have access to the underlying distribution of the data.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rlI8fkpCKdzYg7ljfQlclA.jpeg"><figcaption>Photo by <a href="https://unsplash.com/@photography_by_feaver?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">James Feaver</a> on <a href="https://unsplash.com/s/photos/real-estate-uk?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure>Real datasets are much more exciting. We are given some data and the task of modelling their underlying distribution. There are no right or wrong answers, and every approach is fair game. There are obviously some better and worse solutions which is precisely what we try to explain in this post.
Consider <a href="https://www.kaggle.com/hm-land-registry/uk-housing-prices-paid">this</a> dataset from UK’s HM Land Registry, hosted in Kaggle. It contains more than 22M real estate transactions for the period between 1995 and 2016, with individual information on the price of sale, date, location and type of property, among others. From the original dataset, we select the period between 2012-2014 and consider only the columns that describe the date of sale and price.
<h3>1. Objective and Approach</h3>
Our aim is to model the probability distribution of prices paid for real estate in the selected period. We examine four different approaches to tackle this problem:
<ul>
<li>Gamma fit</li>
<li>Log-normal fit</li>
<li>Gaussian Mixture for the log-price data</li>
<li>Gaussian Mixture</li>
</ul>
In this post, we review mixture models [1] to show how these can be useful when dealing with complex probability distributions. In particular, we apply a Gaussian mixture model to model the probability distribution of prices. The code to run these simulations can be found <a href="https://gist.github.com/kiko-datasparq/3070ad063dd3b69d240f23181c00126a">on github</a>.
<h4>Take-Home Messages</h4>
What worked? Gaussian Mixtures provided a great balance between mathematical complexity, difficulty in the implementation and results for the fit. What didn’t work? I initially tried fitting a mixture of Gammas because I thought it might give better results. This became a mathematical nightmare that I quickly dropped, sticking with the nice closed-form results for Gaussian mixtures. What was time-consuming? I spent a significant part of the time understanding and interpreting the components in each GM fit, as well as finding a nice way to demonstrate my insights. As a result, I decided on the log-log, log-x and linear plots to show each part of the distribution in detail. Why should we care about this? Modelling the underlying distribution is crucial to understanding our data. Although in many situations we might get away with a known distribution, this is not always the case, as we have clearly shown. Having mixture models in our toolbox allows us to work with more complex datasets.
<h3>2. Time-adjusted Prices</h3>
Let’s examine the evolution of the mean daily price paid per property in the selected period.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S9J8Wx4rhsC9UQesXhMA7Q.png"></figure>It is very clear that there is an increasing trend in this time series. The series starts with mean values around 230k and ends with mean values in the order of 280k, implying an increase of around 20%. Before modelling the probability distribution for the whole series, we adjust the prices for this increase.
We model the trend using an exponential function of the form
<figure><img alt="" src="https://cdn-images-1.medium.com/max/141/1*oA6BtHb9GmiRbP51JcLuIw.png"></figure>where a, b are the parameters we need to fit, t is the time (in days) since the beginning of 2012 (our selected period) and y is the target price (in £). The result of the fit is given below.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KoiNkzEgDmqieHyakpJCsg.png"></figure>An exponential fit provides a model with an intuitive interpretation in this context, where we have a geometric growth (similar to inflation) of prices. Note that a linear fit would give a very similar result but would lack such interpretability.
To adjust the prices, we multiply the price of the sale by exp(-bt), where b is the parameter obtained from the fit and t is the number of days between the beginning of 2012 and the actual day of the sale. As a result, we obtain an adjusted time series that looks as follows:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*phu1H0daPdfKQmCeMAK_1w.png"></figure>This result is far from perfect because we still observe periods where the adjusted price goes far away from the mean (which corresponds to parameter a from our exponential fit). However, it is a significant improvement with respect to our original data because we have eliminated its trend.
<h3>3. Modelling the Price using a known Distribution</h3>
We’ll assume the exponential adjustment is sufficiently good for our analysis. Now let’s examine the distribution of adjusted prices.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ddlZe5-sIS_g6IJul-S3iQ.png"></figure>This histogram has been truncated to prices below £1.2M and grouped in bins of £1k. The price distribution is relatively smooth and has a heavy tail with a maximum above £80M (not shown in the histogram).
<h4>3.1 Gamma Fit</h4>
As a first approach, we model the probability distribution using a Gamma. Note the pdf for this distribution is given by
<figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*AHDH3IHEJv5SlARCEkuH4g.png"></figure>where α, β and x₀ are the parameters to be estimated. It turns out that for our data a Gamma distribution is not a good model. The Maximum Likelihood Estimation (MLE) with the entire distribution does not give anything meaningful so we don’t include that result here. The fits with percentiles 95 and 99 (p95 and p99) of the data are given below.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aE_6qg_B-0-VIeih2NpeZw.png"></figure>The p95 fit captures well the mode of the empirical distribution but it does not explain a significant proportion of the data. As we try to include a larger part of the tail, the model breaks. We see that for p99, the match between the empirical and Gamma fitted distributions is much worse than for p95.
<blockquote>In conclusion, the Gamma distribution is poor at capturing the heavy tail of our data.</blockquote>
<h4>3.2 Log-normal Fit</h4>
The histograms for the price (with log-x axis) are given below. For the plot on the left, we have also used a log-y axis to clearly show the heavy tail of the distribution, with prices above 1M for tens of thousands of transactions.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eNqWOcLTniISmGyqDDhvVQ.png"><figcaption>(left) Log-log histogram for price data; (right) Log-x histogram for price data.</figcaption></figure>The plot on the right resembles a Gaussian distribution which motivates using a log-normal distribution to model the prices. To this end, we transform our original price distribution X as
<figure><img alt="" src="https://cdn-images-1.medium.com/max/125/1*PSF3VRG1rbKaRjPlajEqCw.png"></figure>where
<figure><img alt="" src="https://cdn-images-1.medium.com/max/132/1*DsYavMJHktPrdUjv_9QT_g.png"></figure>is distributed as a Gaussian with mean μ and standard deviation σ. We compute the MLE fit with the log transformation of the data and obtain the following results.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k27zcg74csn2exEdiD2qdg.png"><figcaption>(left) Log-log pdf; (centre) Log-price pdf; (right) pdf truncated at 1.2M.</figcaption></figure><ul>
<li>The plot on the left shows the log-log pdf for both the empirical distribution (i.e, the data) and the log-normal fit. We observe that for selling prices larger than £1M, the fit systematically underestimates the observed probability.</li>
<li>The centre plot shows the log-x pdf. The log-normal fit captures correctly the mode at around £100k. However, we see that it underestimates the probability allocated at the mode. We also observe this in the linear-x plot on the right.</li>
</ul>
<blockquote>We are interested in finding a model that can approximate better the pdf both at the mode and the tail of the distribution. One way to do this consists of using mixture models which we review next.</blockquote>
<h3>4. Modelling the Price using Gaussian Mixtures</h3>
As we saw, a log-normal fit for our property prices data presents certain limitations, since the two free parameters that we fit in the MLE cannot capture all the complexity present in the data. Mixture models provide a solution to this problem. We assume that our data X is distributed as the linear combination:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/202/1*kq0cNeP0FWJPs0JQZ_mJLQ.png"></figure>where the weights ωₖ satisfy
<figure><img alt="" src="https://cdn-images-1.medium.com/max/108/1*jz8vLVvvmKkxi76oqsAUJg.png"></figure>and the basis functions gₖ are probability density functions (pdf) from known families of distributions. In this case, we choose gₖ to be Gaussian, therefore:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/161/1*2mE9TpdbVzXd4EErJQCuxQ.png"></figure>Instead of estimating (μ, σ) as we did before, now we estimate (ωₖ, μₖ, σₖ) for all components in the mixture. In the numerical experiments below, we have used the <a href="http://pages.cs.wisc.edu/~jerryzhu/cs838/EM.pdf">Expectation-Maximisation (EM)</a> algorithm to obtain the parameters for the mixture which is very simple and easy to implement.
<h4>4.1 Gaussian Mixture for Log-price</h4>
We set the number of components in the mixture K = 6 and find the MLE solution for a Gaussian Mixture (GM) for Y = log(X), where X is the adjusted real estate prices. This value has been chosen heuristically to strike a good balance between the number of parameters (model complexity) and the capacity of the model to explain the data. We briefly address how to quantitatively select the number of components in the mixture in “Open Questions”.
<h4>4.2 Gaussian Mixture for Price</h4>
We set K = 6 and find the MLE solution for a Gaussian Mixture (GM) for X, the adjusted real estate prices.
<h3>5. Results</h3>
So far we have discussed four approaches to model the probability distribution of real estate prices, namely:
<ul>
<li>Gamma fit</li>
<li>Log-normal</li>
<li>Gaussian Mixture for the log-price data</li>
<li>Gaussian Mixture</li>
</ul>
We discard the Gamma fit from further analysis due to its severe limitations and compare the remaining three. In the graph below, we have plots with four lines, one corresponding to the empirically observed distribution (data pdf) and one for each of our fitted models.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lzQacZTTQoSh9goOjc4WKQ.png"><figcaption>(left) Log-Log pdfs; (centre) Log-x pdfs; (right) Pdfs truncated at 1.2M.</figcaption></figure>As we already analysed the log-normal fit, let’s discuss the GM for the log-price (green curves). This fit is clearly better. In the (left) log-log plot we observe that the GM-log fit is closer to the empirical pdf than the log-normal fit. Additionally, in the centre and right plots, we note that the mode is not only captured but also assigned the correct probability. The only limitation of this mixture is that it systematically underestimates probabilities for prices above £5M.
The GM fit for the price gives slightly different results. It is also preferable to the log-normal fit since it follows the tail of the distribution for large prices, and it assigns the mode the correct probability, as we see in the centre and right plots. However, we also observe that at the head and the tail of the distribution, there is still a discrepancy to the empirical pdf.
<h4>5.1 Analysis of Components</h4>
In both GM fits we chose K=6 components. We now have a look at how these components look and contribute to the final GM distribution.
In the graphs below, we have plotted a line for each component gₖ in the GM log fit, weighted by its corresponding ωₖ, each denoted by Cₖ for k=1…K (not sorted in any particular order).
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9SoDjcW83SG1KBOh63NatA.png"><figcaption>[GM log-price components] (left) Log-log pdfs; (centre) Log-x pdfs; (right) Pdfs truncated at 1.2M.</figcaption></figure>These plots are quite revealing. The (left) log-log plot shows that the components present a very sharp decrease at the tail of the distribution which is why this mixture struggles to capture probabilities for large prices. The plot on the right makes explicit that each component is a log-normal distribution in the linear space (as expected).
Similarly, we analyse the components for the GM fit for the price. In the graph below, we have plotted the pdf for the weighted components ωₖgₖ, each denoted by Cₖ.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lTc2GTJui_a2scydDLfpuw.png"><figcaption>[GM price components] (left) Log-log pdfs; (centre) Log-x pdfs; (right) Pdfs truncated at 1.2M.</figcaption></figure>The plot on the right shows that the pdf is decomposed using Gaussians in the linear space. In the plot on the right, we see why the pdf has wiggles, as we have a component (C1) with a very large mean (more than £5M) with a very small weight which contributes to allocating probability at the tail of the distribution. This is something we did not have either in the log-normal or the GM-log fits, as these fits systematically underestimated the probability at the tail of the distribution.
Additionally, the GM fit assigns a non-zero probability to relatively small, non-negative values, which are unrealistic (all price values in the dataset are strictly positive, as expected). However, for consistency, we have excluded prices lower than zero from the linear-x plot on the right.
<h4>5.2 Open Questions</h4>
There are different directions for improving this analysis. A fair question would be: how do we choose between models? As of now, We do not have a clear answer on whether the GM-log or GM (or even the log-normal) fit is preferred. We discussed the results qualitatively but did not provide a metric to make a quantitative decision. Additionally, we could ask how best to choose the number of components. Both these questions can be answered via the <a href="https://medium.com/@analyttica/what-is-bayesian-information-criterion-bic-b3396a894be6">Bayesian Information Criterion (BIC)</a> which allows choosing between models with a different number of parameters via computation of the likelihood.
<h3>Summary</h3>
<ul>
<li>
Mixture Models provide a flexible framework to model complex distributions coming from real data.</li>
<li>We used Gaussian Mixtures (GM) to model the distribution of real estate prices in the UK for the period 2012–2014. We compared GMs against more simple distributions such as Gamma or Log-normal which provided worse results for the fit.</li>
<li>
Gaussian Mixtures are simple and easy to implement in combination with the EM algorithm. There are analytical expressions for estimating all parameters which is not the case for other mixtures.</li>
</ul>
<h3>References</h3>
<a href="https://www.gwern.net/docs/statistics/2019-mclachlan.pdf">[1]</a> McLachlan, Geoffrey J., Sharon X. Lee, and Suren I. Rathnayake. “Finite mixture models.” Annual review of statistics and its application 6 (2019): 355–378.
Would you like to understand how AI can generate value for your business? Visit <a href="https://www.datasparq.ai/">DataSparQ </a>to find out more about the products we could build for you and the services we offer or get in touch with a member of the team to start your AI journey today.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=d70acb3bc9d" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/gaussian-mixtures-to-model-real-estate-prices-in-the-uk-d70acb3bc9d">Gaussian Mixtures to Model Real Estate Prices in the UK</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Gaussian Mixtures to Model Real Estate Prices in the UK

A small bit of web history
Some years ago in the 90s when I was doing my MSc in Computer Science and the web was the next big tech frontier, we had a problem.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fbQmx9uT6PuwrmE3Vv36BA.jpeg"><figcaption>Photo by <a href="https://unsplash.com/@dlerman6?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Daniel Lerman</a> on <a href="https://unsplash.com/s/photos/search?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure>That problem was search — it was broken, not just a little bit either but totally broken. The search engines of the day regarded search much as a word processor does now. They gave you a search dialogue and expected to return some or even all of the pages in their catalogue which contained your search term. If there was a tie between matching pages, then the engine might look to see if your search term appeared in a title or heading or appeared multiple times within the body of the page.
The inevitable happened — spam pages emerged with huge numbers of hidden words and likely search terms embedded in them. Overnight web search as a tool became completely useless. You couldn’t find anything.
That was of course until <a href="http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf">Sergei Brin and Larry Page wrote a paper</a> as part of their PhD detailing a neat piece of what would later become known as Data Science that could recover relevance and importance in web search. That piece of applied mathematics was PageRank and it became a $1tn company — you will know it as Google.
Job search
Well searching for a job is a bit like that now. Job postings appear containing all sorts of relevant or not so relevant information. They’re posted with wildly diverging titles even if they represent essentially the same job. So Primary Teacher role might get posted with the title: “Part-time Teacher (Primary) years 5–6 Ipswich-starts September” or equally “Primary Teacher (Fixed term maternity cover) £24k — Brent”.
All because jobs forums and recruiting companies think they can elevate their posting in a simple search box. Meanwhile applicants are reduced to searching for terms they think might match the sort of job they are interested in — often the job title. The result is a deeply unsatisfying user experience —lists of over specific jobs that do not represent the breadth of possibility for the applicant’s capability.
Job search is as broken now as searching for a web page in 1996.
The Job Finder Machine
We started work on the <a href="https://labs.datasparq.ai/job-finder-machine/">Job Finder Machine</a> to address this problem in December 2020. As my colleague <a href="https://medium.com/@oliverbartlett/1a7ffe6aa7ff">Oli Bartlett has written in his post</a>, this was part of a data-science-for-good project with the <a href="https://emergentalliance.org/products-and-services/job-finder-machine/">Emergent Alliance</a> to answer the challenge of getting people with skills into well-matched jobs after the pandemic.
Our Data Science Hypothesis
Our data set consisted of a set of job titles representing current or recent job openings and associated with each title was a list of skills that were required by that job.
As with all good Data Science projects, we started with a working hypothesis which we tried to disprove.
<blockquote>
Hypothesis: We can generate high quality job matches based on user-entered natural language descriptions of skills alone</blockquote>
There were the usual problems of skills being duplicated, spelt wrong, capitalised, not linked with the skill with exactly the same name. Also we were expecting user input of skills from the applicant with exactly the same issues. Some useful techniques from NLP came to hand to help address some of these issues — but not all.
How did we do it?
First we developed a conceptual model of the jobs-skills landscape: a bipartite graph connecting a set U of jobs with a set V of jobs.
<figure><img alt="A bipartite graph representing the relationship between job postings and skills" src="https://cdn-images-1.medium.com/max/1024/0*nPps9WEKeqlJ95Rd"><figcaption>A graph of U, the set of job postings, to V, the set of skills needed by a job.</figcaption></figure>Why bipartite? Because at this stage, we only had a relationship between jobs and skills or skills and jobs and there were no direct relationships between jobs (this would come later).
What did this represent? It showed simply that some jobs need a set of skills and those skills might also be required by some other jobs. Not earth shattering but still useful.
Imagine that an applicant highlights some skills in set V that are relevant to them, could we use that information together with the graphical model to return a list of jobs that were a good match for the user’s skills. Potentially.
Those sets, by the way, U might have as many as 180,000 current jobs in at any one time, and V might have 3 million skills listed.
Just returning all the jobs that happened to mention one of the skills the candidate mentioned was a non-starter — we would have many 1000s of matching jobs potentially.
Equally, we realised quite quickly that returning all the jobs that matched all the skills a user mentioned quickly reduced the the number of matches to zero and even if not, very clearly missed lots of jobs that were clearly very similar in nature but had happen not to mention one of the skills that other jobs with the same name had mentioned.
Step 1: Constructing a common skills description for a job
<figure><img alt="Job descriptions and their associated and likely different skills descriptions." src="https://cdn-images-1.medium.com/max/1024/0*DF9kAkcbepxq7d21"></figure>Many jobs with the same job title had slightly differing skills.
We were trying to give a good match to a set of likely jobs and not necessarily a precise match to a specific posting. Also it might be that a hiring manager had left off some skills or had assumed them to be implicit in the role.
Whereas other companies were more precise over how they described the skills that a job needed.
We could learn from the aggregate data to come to a common skills description of a job with a given title. It would fill in some of the gaps left by the individual postings. It would also allow us to capture phrases and synonyms for skills that different companies had used.
We could also start to pick out the most frequent skills for a job and use that to give an indication of the importance of a skill for each job title.
Step 2: Establishing a jobs network
With a common view of skills for each job title we could now look at how similar jobs are in terms of skills.
<figure><img alt="The overlap in skills between a Software Engineer and a Data Scientist — represented as a Venn diagram." src="https://cdn-images-1.medium.com/max/970/0*HHlhpdVtFjMHivCt"><figcaption>A Venn diagram showing the similarity between two related job titles</figcaption></figure>We constructed a similarity metric between job titles, by comparing their skill sets.
We can do this by computing the <a href="https://en.wikipedia.org/wiki/Jaccard_index">Jaccard Index</a> between each pair of job titles.
This is then used to assign a weight to the edge between a network of job titles. This can be visualised in the following example of Job Titles Network.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1dP9ZcVG_xjuZapz"><figcaption>A network of job titles with associated skills similarity measures</figcaption></figure>Step 3: Clustering a network of Job Titles
Having obtained a job titles network we are in the exciting position to solve the problem of having lots of very differently described jobs with very similar skills. These naturally cluster together and for a certain granularity we can create a higher or lower abstraction on the jobs graph.
<figure><img alt="Clustering a graph" src="https://cdn-images-1.medium.com/max/440/0*HPvRN4XnJFfBX0w4"><figcaption>A clustered graph of job titles</figcaption></figure>With this clustered representation we can have a single representation of a set of job titles with a set of similar skills.
Step 4: Ranking job clusters
Finally, we modified the Page and Brin algorithm from 1999 and created our own ranking algorithm across the skills-jobs network using a discrete-time Markov Chain. Now we can take into account how important skills are for a given job (how frequently they are mentioned) — and extend that to clusters of similar jobs.
Now when an applicant lights up some of the skills in the original data set, we are able to produce a probabilistic rank of jobs clusters and produce meaningful importance ranked output based on continuously-updated historical data.
What about algorithm bias?
We need to address the issue of potential bias in the algorithm. This is a massive topic for any recruitment or applicant matching algorithm. While this approach is not immune, the good news is that because the training data is not based on individual applicants we side-step the biggest single source of bias in job-matching — that of prior applicant selection bias.
So where can bias occur? Well two main areas:
<ol>
<li>The job descriptions themselves might contain biased perspective on skills, but unless this is a sector-wide problem, individual posting will get amalgamated with other jobs from the same sector and the effect of the bias will be significantly diluted if not removed altogether.</li>
<li>The candidate applying may be self-selective about how they describe their own abilities. However the algorithm looks for jobs that match partially and “fills in the blanks” for missing skills. This then gets displayed in the percentage match metric.</li>
</ol>
Summary
We have presented a brief overview of our approach to tackling the job search problem. We had a lot of fun doing it and were able to deploy on a GCP-BigQuery-Dataflow architecture two months after starting the project.
If you are interested in joining us for exciting projects like this or just finding out how your company can benefit from our Data Science and Engineering expertise, <a href="mailto:%20info@datasparq.ai">please get in touch</a>. We’ll be delighted to hear from you.
Huge credit goes to the team from Code First Girls who brilliantly innovated on the data science behind the Job Finder Machine with us at <a href="https://datasparq.ai/">DataSparQ</a>: <a href="https://www.linkedin.com/in/erika-nicole-gravina-42a72b176/">Erika Gravina</a>, <a href="https://www.linkedin.com/in/dehaja-senanayake/">Dehaja Senanajaka</a> and <a href="https://www.linkedin.com/in/rajwinderbhatoe/">Rajwinder Bhatoe</a>.
References
[1] The Job Finder Machine: <a href="https://labs.datasparq.ai/job-finder-machine/">https://labs.datasparq.ai/job-finder-machine/</a>
[2] Page, L., Brin, S., Motwani, R., and Winograd, T. <a href="http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf">The PageRank Citation Ranking: Bringing Order to the Web</a>. Technical Report. Stanford InfoLab. 1999.
[3] Oliver Bartlett. <a href="https://medium.com/@oliverbartlett/1a7ffe6aa7ff">Using Data Science to help people get back into work</a>. Medium, 2021.
[4] <a href="https://en.wikipedia.org/wiki/Hierarchical_clustering_of_networks">Hierarchical clustering of networks</a>. Wikipedia, April 2021.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=ad6f7f0fc358" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/job-matching-a-new-challenge-in-search-ad6f7f0fc358">Job matching: a new challenge in search</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Job matching: a new challenge in search

<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xSP0_BOiR-iv6tVCXLidOQ.jpeg"><figcaption>But some industries aren’t</figcaption></figure><a href="https://www.datasparq.ai/">DataSparQ</a> recently worked with <a href="http://reed.co.uk/">Reed</a>, <a href="https://codefirstgirls.org.uk/">Code First Girls</a> and <a href="https://cloud.google.com/">Google</a> as part of the <a href="https://emergentalliance.org/">Emergent Alliance</a> to help people get back into work post Covid.
One of the challenges put to the Emergent Alliance by Reed was to assess ways in which job and skills data might be used to help those made redundant during the pandemic get back into work.
<a href="https://www.ons.gov.uk/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/bulletins/uklabourmarket/april2021#employment-unemployment-and-economic-inactivity">Figures from the ONS</a> show UK unemployment at 4.9% in April 2021 (up from 4% a year earlier) with hospitality, retail and entertainment sectors especially impacted. Of around 800,000 payroll jobs to disappear since March 2020, 526,000 are from retail and hospitality alone.
What’s more, for some specific sectors like high street retail, it’s not clear how quickly (if ever) employment levels will return to their pre-covid levels.
For those who have been made redundant through this period there are a couple of significant challenges Reed identified:
(a) People whose sector or industry has been significantly disrupted may not know where to look for their next role
and
(b) they may be missing key skills which would allow them to move into other roles or sectors (and may not be aware what those skills are).
We set out to develop a proof of concept (PoC) that would demonstrate how job and skills data can be used to help solve these problems. Head over to the <a href="https://labs.datasparq.ai/job-finder-machine/">Job Finder Machine</a> to have a play.
So what’s going on here? To satisfy challenge (a), we wanted to reimagine job search as a less targeted activity. For example, rather than asking a search system what “Hospitality” jobs were available in your area, the idea was to allow people to tell the service what skills they have, and to have relevant roles suggested back. This way, search results would no longer be constrained to the roles or sectors the jobseeker was familiar with. We also wanted to experiment with the idea of returning generic job titles, rather than specific roles, to see if this could be a useful pre-search tool for job seekers to identify new roles or sectors before using more traditional job search tools.
The underlying data science going on here is much better <a href="https://medium.com/@jeremytbradley/ad6f7f0fc358">explained by Jeremy in his post on the project</a>, but at a high level we analysed thousands of job specifications to build a model which clustered job roles according to their skills. The job roles are returned according to how closely the skills required match the skills entered.
The approach has a number of benefits over a free text search, including:
<ul>
<li>Free text search will only return roles whose description uses the same words as have been entered into the search. Because our approach has learnt from many different descriptions of a role it means we know that a role which asks for a “team player” is likely to be suitable for someone who describes themselves as “collaborative”.</li>
<li>We can reduce the impact of <a href="https://www.totaljobs.com/insidejob/gender-bias-decoder/">gendered terms</a> when either used in job descriptions or as inputted skills. The model understands the similarity between skills and therefore it’s possible to automatically add non gendered (or oppositely gendered) skills to any input to increase the breadth of roles returned (this isn’t yet implemented in the machine).</li>
<li>The underlying model provides a navigable graph which could allow jobseekers to browse roles by skills in order to help them explore further outside of their normal comfort zone. For example, you could imagine for any roles presented back to the user, there could be another 5 roles which require 1 additional skill. Clicking on one of these would open up another 5 roles and so on.</li>
</ul>
The skills gaps problem (b) is something we didn’t have time to implement in the proof of concept, but the approach supports it. For example, it would be relatively easy to identify the top roles that require 1,2 or 3 more skills in order to match the user’s input, and then display those jobs in the results along with highlighted skills gaps.
The PoC has demonstrated that a relatively simple dataset can produce a data graph which provides a powerful, intuitive way to find jobs, while also opening up the opportunity to develop innovative new capabilities like the gender-bias-reducer or graph browsing.
If you’re interested in discussing how the work done here might support challenges you’re facing with recruitment, then please <a href="https://www.datasparq.ai/contact">get in touch</a>, or you can contact the Emergent Alliance through <a href="https://emergentalliance.org/products-and-services/job-finder-machine/">the feedback form on this page</a>.
About: The <a href="https://emergentalliance.org/">Emergent Alliance</a> was set up to aid post Covid Recovery through developing data-driven insight and tools to be made available to businesses and government. <a href="https://datasparq.ai/">DataSparQ</a> contributed engineering, data science and product management effort towards the <a href="https://emergentalliance.org/challenge-themes/labour-market-and-how-we-work/regional-re-skill-redeployment-challenge-statement/">Regional Re-skill and Redeployment challenge</a> submitted by <a href="https://www.reed.co.uk/">Reed</a>.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=1a7ffe6aa7ff" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/using-data-science-to-help-people-get-back-into-work-1a7ffe6aa7ff">Using Data Science to help people get back into work</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Using Data Science to help people get back into work

Anomalies and outliers and how to find them.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IPChOJ-xy78-HPr1.jpg"></figure>When I started to write this article the first thing I did was to look for formal definitions of anomaly and outlier. Turns out that there isn’t a consensus on the matter. Every field would have (annoying) a slightly different opinion:
<ul>
<li>Statisticians, for the most part, will use the two terms interchangeably. [<a href="http://dx.doi.org/10.1007/978-3-319-47578-3_1">ref</a>]</li>
<li>Climatologists would say that a (temperature) anomaly is the difference between value and the mean [<a href="https://www.ncdc.noaa.gov/monitoring-references/faq/anomalies.php">ref</a>]</li>
<li>In manufacturing, anomalies are defects (I have not seen anyone use the term outlier)</li>
<li>In banking and insurance anomalies are synonymous with fraud (but in some papers, they would use the term outlier interchangeably)</li>
<li>And don’t ask physicists on their opinion, unless you understand <a href="https://en.wikipedia.org/wiki/Anomaly_(physics)">quantum field theory</a> …</li>
</ul>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*Z3w3XP15GekuZe50FValPQ.png"><figcaption>Manufacturing defects (anomalies). Source:<a href="https://arxiv.org/abs/2102.01882">Kamoona et al.</a></figcaption></figure>At the risk of annoying some people, I’d like to draw a distinct line between anomaly and outlier in the context of Machine Learning. They are not original definitions, merely an attempt to translate the opinion of much more intelligent and experienced people, to simpler terms.
<h3>Outlier</h3>
<blockquote>An outlier is an improbable data point</blockquote>
Imagine that you have a shoe store and would like to know how many shoes of each size you need to order. You call your friendly neighborhood data scientists. He pulls some data and sends you the following:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*gYhB37eiKOQtGwJ6fB4gBg.png"><figcaption>Syntetic data (does not represent reality)</figcaption></figure>That looks exactly like you expected — a nice normal distribution; sizes 40and 41 being the most common. But wait, you look at the table to the right and see something really odd — size 50. You have been selling shoes for years, and have never heard of such monstrosity. You call back your data scientist and ask for your money back, because the data is incorrect. The friendly data scientists then explains to you that:
<blockquote>“1 in 100,000 people has a size 50 shoes.That is very rare, but still possible. That is an outlier!”
</blockquote>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-bqetyg1BRyhOHu6BBCT_Q.jpeg"><figcaption>A 6ft 5in bride-to-be with size 15 feet is unable to find wedding shoes big enough. Source: <a href="https://www.thesun.co.uk/fabulous/7882473/bride-big-feet-shoes-wedding/">The Sun</a></figcaption></figure><h3>Anomaly</h3>
<blockquote>An anomaly is something created by a different process entirely. There is a different underlying function or a distribution.</blockquote>
Back in the shoe store, we now want to run a campaign where we give you a free pair of jeans with every shoe purchase. We want to make sure that we order the correct jean sizes and tie those to shoe sizes (please just roll with it …).
<figure><img alt="" src="https://cdn-images-1.medium.com/max/390/0*jAFeQRMejtRj94bU.gif"><figcaption><a href="http://gph.is/1OYW0I0">Source</a></figcaption></figure>So who you gonna call? That’s right, back we go to that overworked and slightly underpaid data scientist. He now sends us something a bit more complicated:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*EGt5JNQkRWqWAhxs8HfFAA.png"><figcaption>Linear regression of shoe size and height</figcaption></figure>This is a scatter plot that shows how shoe size and height are correlated. The line in middle is a linear model that shows a very high correlation between the two variables.
However, your keen eye has spotted something wrong with this picture. Let me use the power of colour and show you what I am talking about:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*8GD8mE2YUYNQa7n_rPDlKQ.png"></figure>There are two orange dots on the chart that seem a bit out of place. I have pasted them below for closer investigation.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*wMj4CnspSnkxv2hY9ukI3g.png"></figure><ul>
<li>Both shoe sizes 35 and 47 are within our expected range.</li>
<li>Heights of 158 and 183 are also quite normal
</li>
</ul>
<blockquote>It is almost impossible for a human to be 183 cm tall and have size 35shoes. Or be158 cm with size 47 shoes (unless he is called Sideshow Bob). That is an anomaly.</blockquote>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*FrZL9usMmvTUnSUi.gif"><figcaption>Everyone knows that Sideshow Bob mas massive feet. <a href="http://gph.is/1TvNnp2">Source</a></figcaption></figure>Lets look a bit closer at the people that wear size 35 :
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*LMXTbD-HGiy2DvX0lvtK9w.png"></figure>As you can see a height of 183 is more than 10cm than the largest value in this group (shoe size 35). That data point is not realistically part of the population.
<h4>Recap</h4>
When looking at shoes size or height individually (univariate), we only see outliers — values that are rare but still probable. Analyzing the dataset as a whole (multivariate) we can see combination of values that are anomalies.
<h3>Anomaly detection</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/603/0*NN5M4j1j0AmGhcac.gif"><figcaption>No, not that kind of anomaly. <a href="http://gph.is/2vgaNul">Source</a></figcaption></figure>Now that we (hopefully) understand how this works, let’s take it up a notch and go looking for anomalies in a dataset with 3 variables. I will kick in an extra variable called weight (to our previous dataset) and boot out the shoe store allegory.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YnL5hHoye09ZDvoNBufwDQ.png"><figcaption>Pairplot that shows relationship between all the variables</figcaption></figure>These kind of plots are very useful to get a first glimpse into how the different variables interact with each other.:
<ul>
<li>All three variables are normally distributed</li>
<li>shoe_size and height are positively correlated</li>
<li>weight and height are also positively correlated</li>
<li>shoe_size and weight are not correlated</li>
</ul>
We can try and find the anomalies like we did in the previous section, but that will take a while. There we had one scatter plot that we could visually inspect, now we have three. A more systematic approach is required if you don’t want to spend the rest of the day staring at dots and histograms.
<h3>PCA</h3>
Enter <a href="https://www.sartorius.com/en/knowledge/science-snippets/what-is-principal-component-analysis-pca-and-how-it-is-used-507186">Principal Component Analysis</a>. I am not going to go into details about it, but you can follow the link if you want to know more or need a refresher.
<blockquote>
PCA uses fancy algebra to take any number of variables (dimensions) and reduce them to a more manageable number without losing much of the information stored in the data.</blockquote>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*9eyHMZIrbMI5fRBV.gif"></figure>In our example we take the 3 variables and reduce them 2 and we get to keep 99% of the explanatory power. Needless to say this is much more impressive (and useful) when you have even more variables, but let’s keep it simple.
One of the upsides of PCA is that you can more easily visualize your data. Now we can much more easily see those outlier/anomalies:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1gw7C8XHeC9WR-5_0-BS5Q.png"></figure>But before you get your crayons out and start colouring in those dots let me show you another trick from the data science hat. We are really deep into the rabbits hole now, please stay with me, we are almost in Wonderland.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/513/0*IkcEkcEQKZaExtVr.gif"><figcaption><a href="http://gph.is/1LQnssc">Source</a></figcaption></figure><h3>DBSCAN</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*c60aVwNf8mdOMnyt.gif"><figcaption>No, not that kind of scan. <a href="https://gph.is/2CDGoLz">Source</a></figcaption></figure><a href="https://scikit-learn.org/stable/modules/clustering.html#dbscan">DBSCAN</a> is an unsupervised clustering algorithm (you can get a nice intro <a href="https://towardsdatascience.com/dbscan-clustering-explained-97556a2ad556">here</a>). We can feed it the PCA data and some basic rules and have it figure out which is the main cluster. It does that by figuring out how close each point is to its neighbors. If it’s surrounded by lots of them — it’s part of a cluster. If a data point is too far — it’s an anomaly.
DBSCAN().labels_ returns an array where each cluster is given a number, starting from 0 and anything that doesn’t fit into a cluster is -1 . Since our dataset is very uniform, we only have one cluster.
In the image below you can see the PCA plot from before, with the anomalies coloured in salmon.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*60rtZJXZZNXpeaEB7sFmqw.png"></figure>The hardest thing in this kind of analysis is always finding a good boundary. And that is more of an art than a science despite the plethora of statistical and ML methods swimming out there.
For the sake of argument let’s agree that our salmon dots are quite the catch and move to the next step.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2epBcSty_2rpoqJd"><figcaption>Photo by <a href="https://unsplash.com/@noaa?utm_source=medium&amp;utm_medium=referral">NOAA</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure>PC1 and PC2 are not very useful numbers for us mortals, so we need to go back to the original data. Below we add the anomalies (DBSCAN clsuter) to the pairplot from before:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y7M58O3-SsW19XW1zwBe0A.png"></figure>With this visualization it is much easier to explain why certain points are anomalies.
What is even more interesting is that we have found anomalies that we would have missed if we used the method from the first part of the article. Look at the anomaly data points in the bottom left scatter plot (weight~height). We could have probably found the ones that are visually outside the main “cluster” (distribution), but we would have missed the ones that are “inside” (with a higher frequence/probability).
<figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*ylO30fdvTYuDXjKD.gif"><figcaption><a href="https://gph.is/g/4LN0wvL">Source</a></figcaption></figure>And for the viewing pleasure of the spreadsheet nerds, you can find a random sample of the anomalies and their values below:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*W_d5s5tMMa4vHm2jrvghPg.png"><figcaption>sample of 20 anomalies (out of 39)</figcaption></figure>Let’s analyze some of them:
<ul>
<li>76283 (2nd row) — height and shoe size seem acceptable, but the weight is too low</li>
<li>100024 (last row) — height and weight are fine, but the shoe size seems proposterous</li>
<li>100001 (somewhere in the middle) — that observation is the definition of anomaly. 184cm tall at 64 kg and size 35 feet? Sounds like a Slenderman to me.</li>
</ul>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*4gwa2QocKvJLEAFj.gif"><figcaption>Slenderman. <a href="http://gph.is/1c3ufZ3">Source</a></figcaption></figure><h4>Final words</h4>
<ul>
<li>The dataset I used was hand crafted by me for the purposes of this article, so don’t go around using those graphs to draw conclusions about the real world.</li>
<li>Real datasets are much more complicated and messy. Having a good grasp of the process or business logic behind them will go a long way in the quest of anomaly detection.</li>
<li>The notebook(s) for this article can be found <a href="https://github.com/ivan-sparq/blog-artefacts/tree/main/AnomalyDetection/notebooks">here</a>
</li>
</ul>
Hope you enjoyed reading this as much as me writing it. Questions and comments are as always, highly encouraged.
<h4>Sources:</h4>
<ul>
<li><a href="https://stats.stackexchange.com/questions/189664/difference-between-anomaly-and-outlier">https://stats.stackexchange.com/questions/189664/difference-between-anomaly-and-outlier</a></li>
<li><a href="https://akajuvonen.github.io/2017/10/31/outlier-vs-anomaly.html">https://akajuvonen.github.io/2017/10/31/outlier-vs-anomaly.html</a></li>
<li><a href="https://arxiv.org/abs/2102.01882">https://arxiv.org/abs/2102.01882</a></li>
</ul>
<ul>
<li><a href="https://medium.com/@mehulved1503/outlier-detection-and-anomaly-detection-with-machine-learning-caa96b34b7f6">Outlier Detection and Anomaly Detection with Machine Learning</a></li>
<li><a href="https://towardsdatascience.com/anomaly-detection-for-dummies-15f148e559c1">Anomaly Detection for Dummies</a></li>
<li><a href="https://towardsdatascience.com/a-brief-overview-of-outlier-detection-techniques-1e0b2c19e561">A Brief Overview of Outlier Detection Techniques</a></li>
<li><a href="https://towardsdatascience.com/outlier-detection-with-isolation-forest-3d190448d45e">Outlier Detection with Isolation Forest</a></li>
<li><a href="https://towardsdatascience.com/outlier-detection-theory-visualizations-and-code-a4fd39de540c">Outlier Detection — Theory, Visualizations, and Code</a></li>
<li><a href="https://towardsdatascience.com/5-ways-to-detect-outliers-that-every-data-scientist-should-know-python-code-70a54335a623">5 Ways to Detect Outliers That Every Data Scientist Should Know (Python Code)</a></li>
<li><a href="https://towardsdatascience.com/dbscan-clustering-explained-97556a2ad556">DBSCAN Clustering — Explained</a></li>
</ul>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=640c949d206" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/anomaly-detection-for-beginners-640c949d206">Anomaly Detection for Beginners</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Anomaly Detection for Beginners

<blockquote>An economy that puts back what it takes out, that is sustainable by design. — Matthew Bishop
</blockquote>
The circular economy is a foundational principal of sustainable commercial and industrial activity.
Circular economy requires that we account for the natural resources we take and use in everyday processes and, as far as possible, return them to their original state — everything from air and water to habitats. Further, every product — every single product — we consume needs to have a reuse or recycling pathway either directly or in an adjacent process or industry.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ek_90EPKNxiLOdcj.png"><figcaption>The circular economy in action (Geissdoerfer et al. 2020)</figcaption></figure>Be in no doubt, the transformation required by the circular economy will make the industrial revolution and the data revolution look like a walk in the park by comparison!
What does circularity entail
<blockquote>In its industrial processes, a truly circular economy creates no net pollution or waste. — Matthew Bishop
</blockquote>
Given this stringent definition, the goal of achieving circular industrial processes across a national and by extension, for it to make sense, a trans-national economy, is a huge challenge. A social challenge certainly, a commercial and investment challenge definitely, but also I argue, a data challenge and and a massive scientific challenge as well.
Circularity subsumes the carbon cycle neutrality that many nations are working towards and extends it towards other natural resource cycles such as water, air, minerals, and biodiversity. By definition though, circularity includes the products that are created in a thriving economy and all the materials that go into making them.
If circularity is to happen — and for ongoing sustainability it surely has to — this requires the a dramatic and disruptive change to the global economy.
Changing production/logistic networks
Currently many products and processes revolve around a linear production model. This does not work well as a model for reclaiming materials for recycling and reuse as many products end up in landfill.
<figure><img alt="A linear production model which see products go from manufacture to waste disposal at end-of-life" src="https://cdn-images-1.medium.com/max/414/1*yZRh5PJDxCnEApshDe1IWQ.png"><figcaption>A linear production model that sees products go from manufacture to disposal (jb, 2021)</figcaption></figure>A linear production model involves a supply chain of raw materials that see a product created at a production facility and then pushed into a supply chain distribution network. The use-life will sometimes have maintenance cycles embedded in it if the product is of sufficiently high value. When end-of-life is reached the product is disposed to landfill or incineration with some portion of the product being recycled depending on legislation, consumer habit and process availability.
For new products designed for circularity, we will see many different and potentially disruptive production and use models spring up.
<figure><img alt="A cyclic production-use-recycling process" src="https://cdn-images-1.medium.com/max/385/1*skKHxkUjhVqXfjUAtvGP2A.png"><figcaption>A cyclic production model where the production is located with or nearby the point-of-sale (jb, 2021)</figcaption></figure>Future circular logistic chains may see production — perhaps micro-production — move closer to the point-of-sale. Imagine bottling plants or food packing facilities embedded in supermarkets.
Similarly some outlets will become points-of-return for their products at end-of-life. Except that, what was previously end-of-life will become start-of-second-life after a suitable recycling process.
The distribution network will become a means for front-hauling raw components to the manufacturing facility and taking packaging and old products for refurbishment — so-called, back-haul.
All this requires unprecedented cooperation between logistics providers, manufacturers retailers and consumers. Undoubtedly incentive programs like bottle deposit schemes will be needed and the economics of even those industry schemes are not straightforward. For more expensive products, lease-and-return will likely be the better model replacing own-and-discard.
The scientific and engineering challenge
Let us for a moment look at the industries that will be swept up in this revolution:
<ul>
<li>Building and construction</li>
<li>Clothing and textiles</li>
<li>Metals (bulk metals like, iron and aluminium) as well as tech metals such as rare earth metals, copper and lithium</li>
<li>Farming and food production</li>
<li>Energy generation</li>
<li>Water processing</li>
</ul>
This is a scientific challenge at a materials level, and an engineering level at a process design level. Assuming we have a circular supply chain in place (see above) to collect the raw material: Can we physically or chemically reconstitute the materials once they have been used? Does an industrial scale process exist to support such a reclamation? Is it economically viable to carry out the recycling activity?
<figure><img alt="Recycling box full of glass bottles." src="https://cdn-images-1.medium.com/max/1024/0*ilTGUJZZE3WMQLNY.jpg"></figure>As an instance, thermal recycling of glass (melting recycled glass down and reforming it) cannot be done currently with 100% used feedstock or cullet (the name given in the industry to the broken glass obtained from bottle banks). It has to be mixed with new silica/soda-lime in order to produce a suitably high quality product (green glass can be formed from 90% feedstock, clear glass only 25%). Different but equally hard challenges exist in recycling PET plastics and other raw materials.
These are massive challenges by themselves and the good news that many research funders and universities are making inroads into these challenges.
Why a data challenge?
If this wasn’t exciting enough, the cross industry aspect is where the circular economy becomes a really interesting puzzle.
<blockquote>There is no such thing as ‘away’. When we throw anything away it must go somewhere. — Annie Leonard
</blockquote>
The challenge lies in joining up industries that have previously had no interrelationship.
For the sake of argument, say it were possible (I’ve no idea whether it is) to reuse the magnets in the motors of discarded washing machines and tumble dryers to help create generators for wind turbines and new motors for electric cars. This would be an example of how the white goods industry might find itself in the supply chain for fast growing green technologies.
If we now forbid those growing industries to use virgin raw materials from usually mined ores — we run into two problems straight away:
<ol>
<li>Are there enough discarded washing machines and tumble dryers to supply all the magnets need by the wind turbine and electric car industries</li>
<li>Are the recycled materials available in sufficient quantity at the right time to allow manufacturers to rely on them in their supply chains</li>
</ol>
Even if the answer to (1) is yes, the answer to (2) is almost certainly going to be no — at least to start with. These industries have never had cause to synchronise before — they are not aware of each others production flows, but they will need to be if there is to be synergy between them. This is where data science and smart supply chain processes can drive real innovation.
Data flows for circular economy
In supply chains, products, components and raw materials flow between mines, factories, warehouses and points of sale. In efficient supply chains, goods do not spend long at any one point in the supply chain, instead they are ordered as needed and in anticipation of a downstream demand.
What drives this efficient flow is data pipelining and accurate forecasting of demand. Without that, materials end up being stock piled in warehouses to mitigate the risk of material exhaustion. Storage of materials, just as with components, is inefficient and costly.
In a circular economy with many cross-industry dependencies, we will need data visibility on:
<ul>
<li>demand forecasts in consuming industries; and</li>
<li>flows of products reaching end-of-first-life in feeder industries</li>
<li>volumes of recycled material that is about to be available for reuse</li>
</ul>
Industries which have never talked to each other will need standardised APIs for interrogating each other or pushing notifications to each other to achieve this seamless integration.
Algorithms for the circular economy
If this sounds like it will be ripe for application of predictive data science, it absolutely will be. The better the forecast or anticipation of arrival of a feedstock, down to the day, hour or even minute, the more efficient the manufacture of the new product can be. Materials arriving early can at least be stored (if space allows) — materials arriving late though will potentially hold up a whole manufacturing pipeline.
Another key technology will be in the area of meta-heuristics, e.g. Silva et al. 2003. If you’ve never heard of these before in the context of data science problems, they are approximate but fast solution approaches to massive multi-scale optimisation problems. And if we are to minimise the wait time, maximise the throughput and minimise the waste in dynamic cross-industry supply chains, we will need the very latest state-of-the-art optimisation algorithms to enable this.
Models of flow of goods than can capture recycling streams at a cross industry level will be extremely important, for example, Bhaskar et al. 2010. These models will need to represent delay, storage and uncertainty but they will also need to learn distributional patterns that represent volume. Being able to identify key bottlenecks well ahead of them actually occurring will be necessary so that alternative sources or feedstocks can be planned for.
Conclusion
So what can we do now to address these challenges? For those of us who look for data-driven projects in a variety of industries — there is low-hanging fruit and much of the benefit can start within a company.
<ul>
<li>understanding and mapping the data flows and data services within a company is a health check that many companies could do and leave for another day</li>
<li>creating a sustainability call-to-action within a company which looks at the flows of natural resources as well as raw products and would allow for an audit of natural capital expenditure will likely expose circularity shortfalls</li>
<li>engaging with the data science to create high quality forecasts for many parts of the company to use and understanding the key impacts and risks of that forecast being right or wrong</li>
<li>engaging with universities and <a href="https://www.ukri.org/news/hub-to-provide-national-circular-economy-leadership/">circular economy hubs</a> at this early stage so that a company is aware of upcoming technical and business opportunities</li>
<li>creating an open data culture so that people who want to work with you can see your consumption and your production and forecasts of the same so they can understand what the time and volume challenge will be if they start to be your supplier/consumer for a critical material</li>
</ul>
If sustainability is to be sustainable then it has to drive value itself and the good news is that the actions that any company can take in preparation for the circular revolution will also drive efficiency, savings, internal reliability and external opportunities.
References
Geissdoerfer, M., Pieroni, M.P., Pigosso, D.C. and Soufani, K. “Circular business models: A review.” Journal of Cleaner Production, 2020.
Silva, C.A., Runkler, T.A., Sousa, J.M., Sá da Costa, J.M. “Optimization of Logistic Processes in Supply-Chains Using Meta-heuristics.” EPIA 2003: Progress in Artificial Intelligence. LNCS vol. 2902, pp 9–23. Dec. 2003.
Bhaskar, V., Lallement, P. “Modeling a supply chain using a network of queues.” Applied Mathematical Modelling 34(8):2074–2088, Aug. 2010. DOI: 10.1016/j.apm.2009.10.019
About the author
Dr. Jeremy Bradley is Chief Data Scientist to DataSparQ, a member of the Strategic Advisory Network for the UKRI/EPSRC Circular Economy Theme, a member of the Strategic Advisory Team for EPSRC Mathematical Sciences and Fellow of the Institute of Mathematics and its Applications. He lectured and researched Computer Science at Imperial College London for 14 years. He has industrial data science experience as part of the data science teams at Tesco, Royal Mail and now DataSparQ. Jeremy also co-hosts the <a href="http://datacafe.uk/">DataCafé data science podcast</a> with his friend Dr. Jason Byrne.
If you would like to know more about how sustainable data science can help your business then do <a href="mailto:info@datasparq.ai">get in touch</a>. We would be happy to share our learnings and see if we can help.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=9653a29d0c55" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/the-next-revolution-data-science-and-the-circular-economy-9653a29d0c55">The next revolution: Data Science and the Circular Economy</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

The next revolution: Data Science and the Circular Economy

Everyone starts somewhere
When a data project gets too hot for a script to handle, data engineers, scientists and developers will start to look for a better way to handle multiple different types of data processes.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4-vdQ0McFfzdwzCl"><figcaption>Photo by <a href="https://unsplash.com/@ryanquintal?utm_source=medium&amp;utm_medium=referral">Ryan Quintal</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure>This is exactly what happened to me when I started, I found a great project called <a href="https://github.com/spotify/luigi">Luigi</a> (Open Source from Spotify). Luigi is a data orchestration framework which gives you the ability to both reuse and define dependencies between data processes. Using frameworks like Luigi allow developers to create performant and complex data pipelines (combinations of multiple processes / data sources). This concept isn’t new, but is key to building a successful data platform.
Data Platforms are more than a collection of processes
Data pipelines, in combination with other features (data warehouse modelling, monitoring, etc), can be used to create a modern data platform. However, a data platform is much more than this, from a technical, business and cultural perspective.
Some of the common mistakes (technical) include:
<ol>
<li>
Overhead of maintenance. In any system there is a degree of maintenance (yes, even serverless products), you can accept it or the consequences. Data processes are modified and contributed frequently. Regular security, library, monitoring and functionality updates are essential. Cloud providers reduce overheads but do not remove them entirely via managed applications (Google Cloud Composer) or serverless solutions (Azure Data Factory).</li>
<li>
Extendibility. How will we manage requirements from teams with different needs than ours? Change is inevitable. Business users will want to build valuable complex processes and share results with others.</li>
<li>
Slow to implement. Platform projects have the potential to run for long periods of time, by which point existing business processes may have moved on.</li>
<li>
Lack of documentation. A data platform needs use cases and business users to drive data driven decisions and create products in order to generate value. Without these it is powerless. If people can’t find and understand the data within, this seriously limits its potential to generate value.</li>
</ol>
Services break large problems into smaller, reusable parts
This isn’t always an advantage. When breaking monolithic systems apart, bear in mind parts may still need to communicate. This can add other forms of complexity, which in a number of situations may outweigh the benefit of separation. However…
Data processes are very loosely, if at all, coupled. This makes them ideal candidates for separation. Why should the environment that runs a SQL query be the same for training a machine learning model?
This has led the team at DataSparQ to create individual services for the cloud data products in our architecture. Combined with “single source of the truth” best practices, it’s enabled us to build new projects quickly.
Below shows how different types of data stakeholders interact with one of our services. The selected service “warehouse operations”, handles all processes carried out by the data warehouse (queries, table creation, truncation, etc):
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6s7hnV0NtYGHBxogOkNS8w.png"><figcaption>DataSparQ Service Architecture</figcaption></figure>No data solution is perfect
We’ve made a trade here. A slight increase in initial development of processes, some of which are pre-built in classic orchestration solutions.
However it’s been worth it, we can now:
<ul>
<li>Replicate key services between multiple projects instantly, reducing development time and cost
</li>
<li>Run services on cloud serverless products, reducing maintenance and running cost across projects
</li>
<li>
Innovate with new cloud products &amp; techniques the moment they are available, snapping new services into development pipelines</li>
<li>
Follow metadata “single-source” best practice, allowing business users access to data catalogues whilst using them for our development activities (transformations, schema definitions, etc)</li>
</ul>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=104db9dedfa3" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/moving-from-data-platform-to-data-services-104db9dedfa3">Moving from Data Platform to Data Services</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Moving from Data Platform to Data Services

<figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*2k4t71mbk_-LdwBeduVjOw.png"><figcaption>What pops up when you Google image search for Robotic Process Automation</figcaption></figure>Recently I assessed Ui Path’s artificial intelligence service, AI Fabric, and to be more precise, implement a basic Natural Language Processing (NLP) classification model within AI Fabric. This project had two sides, understanding AI fabric and all of its quirks, and creating my first ML model in python.
A quick aside on the model required. This was my first ML project that I had undertaken; creating TF-IDF features on a set of german email complaints with the goal of classifying the type of complaint within the text — a simple NLP problem. Using the normal techniques of stemming, stop words and vectorizing, I was able to create features which could then be used to train a <a href="https://towardsdatascience.com/naive-bayes-classifier-81d512f50a7c">Naive Bayes</a> model, as described in many <a href="https://towardsdatascience.com/build-and-compare-3-models-nlp-sentiment-prediction-67320979de61">online guides</a>.
However, the task was not only to create the model, but to deploy it to AI Fabric in order to demonstrate the capabilities of ML in Robotic Process Automation (RPA). The platform provided by Ui Path proved to be very easy to use, allowing for quick data uploads, and even had numerous ‘out of the box’ ML models available. There are still a few teething issues with some of the models at the time of writing, such as environment variables not able to be set at run time. However, what is particularly useful in AI Fabric is that a zip package with python scripts for training and predicting can be uploaded, along with any untrained model that is desired.
The zip package that is uploaded must be in the required format as specified in the <a href="https://docs.uipath.com/ai-fabric/docs/building-ml-packages">AI Fabric documentation</a>.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vg17S-6qv2VJJyZfMRrDrw.png"><figcaption>A new ML Package can be created from an uploaded zip file of bespoke code, or an Out of the box Package</figcaption></figure>As per the documentation, AI Fabric looks for a class, Main(), in both main.py and train.py. The class has the required functions to train, test and predict. When training and testing the model, it is useful to export information so that it can be reviewed to see the results of the new model. This can be done by creating an artifacts folder which text files and graphs can be saved to. This is a great feature within AI Fabric, with each run creating its own artifacts folder, allowing for in-depth metrics and information to be saved and viewed later.
I would recommend rigorous testing locally when creating the model. AI Fabric is easy to use once the code is uploaded, however the process of making a change, compressing all the code to a new folder and uploading the zipped folder is a time-consuming process. If there are any errors when deploying or running the code, ML Logs is very useful, and will highlight any errors just like whichever IDE you use.
Once all the working code is assembled, the zip can be uploaded to AI Fabric. Within a project, the ML code can be uploaded in the ‘ML Packages’ section, along with any information for the model. Here it is validated, checking that the correct scripts and requirements are available. Once complete, a pipeline can be run. There are three types of pipelines; train, evaluation and full. A full pipeline is just a combination of both the train and evaluation pipeline, allowing the model to be trained and tested in one run.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*18FsFV3YsLmb4OuQmEHGMg.png"><figcaption>Creating a train pipeline for an uploaded model</figcaption></figure>The model, version (1.0 if the model has not been trained before) and data must be selected for each pipeline. A train pipeline is used to create a new version of the model (i.e v1.0 is trained to v1.1), and an evaluation pipeline is created to test the model. If a completely new version of the model is created, it can be uploaded to the same ML package, and creates a new major version (v2.0).
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XX3gJQLFly8wupsxnGnaXw.png"><figcaption>Deploying an ML Skill, a trained machine learning model which can be dragged and dropped into Ui Path</figcaption></figure>Once the model has been trained, it can be deployed to Ui path for implementation in an RPA stream. An ML skill must be created but once it is fully deployed, it becomes available in Ui Fabric, as long as you are on the same Orchestrator Service. Here it can be easily dragged and dropped into any workflow, just like any other process.
Now, the powers of AI can be unlocked in RPA. At a first trial, this simple classifier can be implemented to divert email complaints with a high level of success (80–90% accuracy with limited data). However, this is very similar to the ‘out of the box models’ mentioned earlier. These models work fine, and can be easily implemented, however the beauty of the bespoke packages are their customisability. For example, the percentage confidence can be returned with each classification, allowing human intervention to be called if the confidence is below a set threshold.
With this package structure, more complex models can be implemented as needed by the use case. For example, random forest or gradient boosting algorithms can be utilised to complete tasks effectively and increase the productivity of any business. Some backend work is still required, of course. Some fine tuning of the model input parameters for specific tasks may be needed, as each job is different, but any ML package can be easily uploaded, trained and deployed in a few simple steps on UI Path.
Links:
<ul>
<li><a href="https://towardsdatascience.com/build-and-compare-3-models-nlp-sentiment-prediction-67320979de61">Beginners guide to creating a classification model with nlp</a></li>
<li><a href="https://towardsdatascience.com/naive-bayes-classifier-81d512f50a7c">Introduction to Naive Bayes model</a></li>
<li><a href="https://docs.uipath.com/ai-fabric/docs/building-ml-packages">AI Fabric documentation</a></li>
</ul>
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=6b96ce36f6cc" width="1" height="1" alt=""><hr>
<a href="https://medium.com/datasparq-technology/integrating-ml-with-rpa-using-ai-fabric-6b96ce36f6cc">Integrating ML with RPA using AI Fabric</a> was originally published in <a href="https://medium.com/datasparq-technology">Datasparq Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Who we are

What we do

Work we've done

Contact us

Welcome

Turn data into better

Perform beyond your potential with AI and data solutions from Datasparq.

Who we work with

Our clients and partners

What we do

Data science meets design thinking

How we do it

Turning data into better solutions to real-life problems

Turning in-store purchase data into better retail locations

Turning traffic data into a better car parking experience

Turning salad sales data into less food waste

Turning project risk data into better use of tax-payers’ money

Turning supply and demand data into happier holidaymakers

Turning job vacancy data into better opportunities for jobseekers

Who we are

Passionate practitioners

Inspired? Join our team