Q2, 2020 at Intent HQ has seen the teams successfully adapt to remote working as a result of the pandemic lockdown. Our product and development teams in London and Barcelona have overcome such exceptional circumstances, and we’re ready to shout about a host of new important releases.
Q4 2019 Release Notes
Here are the details on the significant feature releases.
RUNNING PYTHON SCRIPTS AS A PART OF THE DATA PROCESSING PIPELINE
The IHQ Platform uses a set of configuration to define steps of data processing and enrichment during a pipeline run.
Popular due to its versatile nature, Python is popular with users such as data scientists, data engineers etc. to generate new value by combining information from multiple data sources. Additionally, users can utilise the Intent HQ DSL to help Python reference data sources and process information efficiently.
ENABLING PIPELINE RUNS FOR A FIXED SAMPLE/SUBSET OF PROFILES
In most scenarios, the effort of testing and validating configuration and/or feature engineering on a production-sized dataset is not only expensive and time consuming, but also highly inefficient at scale.
In this release, we have added the capability to run a pipeline for a chosen set of customer profiles. E.g. this may be a subset of users eligible for a sample to be further used as test/validation datasets during feature engineering.
Whether to validate configuration or enrichments (in DSL or Python) on a smaller sample first before going to production, or with the intention of using the subset for further work, we encourage the use of sampling for data exploration given the associated cost and effort of using full datasets.
LAUNCHING PIPELINES INSTANTLY WHEN NEW DATA IS INGESTED
In order to increase the level of ease and automation in data transformation, Intent HQ has added functionality to trigger data processing (pipelines) after file ingestion has been completed.
This means the platform will automatically recognise and reference a given pipeline configuration as soon as ‘expected’ data has been received and ingested and will continue to generate enriched profiles.