Azure Data Factory V2
Hybrid data integration service that simpliflies ETL at scale
UI/UX Design
As data becoming more and more important the ability to transform data to useful information become more and more demanding. Azure Data Factory, a service built for all data integration needs and skill levels. Easily construct ETL and ELT processes code-free within the intuitive visual environment, or write your own code.
Company/
Microsoft
​
Role /
Lead Designer
​
Year /
2017-2019
Dusting off the sketchbook . . .
The goal is to create the tool that would enable user, including those with less coding knowledge to transform and orchestrate their data.
The Hybrid data integration service that simplifies extract, transform and load data at scale
A problem worth solving
Microsoft realized that the main user for big data are data analyst and data scientist, however the user in these two personas not alway be tech savy. The process of cleaning and transform data usually demand the user to have extensive technical knowledge and ability to code. There wasn't that many virtual data transformation/orchestration tool that exist in the market and certainly non from Microsoft. Before data factory V2 the non-technical have to rely on data engineer to clean the data for them which mean extra time needed.
​
Part of Azure portal​
Pure coding experience ​
Can only work on one pipeline at a time​
No guiding information​
Persona
We can separate the user into 6 personas. In ADF v1, however, we only targeted one persona - data engineer. After comprehensive interviews with data engineers we realised that they work closely with data analyst and data scientist. There is a lot of unnecessary data transfer between these roles.
Potentail User
Current User
Potentail User
Data analyst and data scientist are usually less tech savvy, most of them prefer not to code. This gave us a goal to enable these 3 personas to collaborate more effectively.
Challenges
Flexible Integration​
Accommodate all the unstructured data from anywhere​
Compute agnostic​
ADF can hook up all the different analytic platforms and drive BI reporting​
Code free visual tool​
Enable complex experiences to less tech savvy users by creating intuitive visual tools​
Corroborate work space​
Version control & protected editing for different personas​
Process
Agile design process​
Wireframe
I started making wireframe once I have some idea on the direction we want to preceded. There were 2 levels: High level wireframe which used to show stakeholder, engineers and PMs and confirm that they are happy with the direction we are moving toward. Detail wireframes were used by PM to present to users(partner) to get their feedback and reaction.​
Detailed wireframes were used by PM to present to users(partner) to get their feedback and reaction.​
Design Details​
I created visual designs after making the detailed wireframe. One of the goals for this project was to make the help center consistent with Microsoft’s component library style. I used colours, fonts, spacing, buttons, cards, icons, etc. from Microsoft branding guidelines. In visual design process, I kept in mind keywords for simple and clean.
Overview
Azure navigator bar with the factory name​
Introduction video and latest news about the product​
Other option​
Logout option​
Tutorial video​
Pipeline Canvas​
Activities menu (collapse able)​
Version control​
Coding option​
Canvas action bar​
Setting (resizable)​
Data Flow Canvas​
Guiding layout system​
Coding option​
Canvas action bar​
Setting (resizable)​
Dataset setting tab​
Dataset icon​
We spend a lot of time making sure that the settings make sense to the user​
Coding option​
Tab for switching between settings​
Pipeline Run​ Monitor
These buttons appear only in monitoring mode​
Action bar​
Status
User can copy their filter and share their filter with other user​
Data Flow Monitor
Outcome​
Azure Data Factory launched general availability in July 2018 and October 2019 for Data Flow feature.
"I’ve worked many years with SAP and with other clouds and Microsoft is clearly on the right track. Nothing matches it in terms of interface and user friendliness"
– Alexandre G.
“It's pretty good. Very satisfied. I thought this was easier than Nifi (compatitor). This was actually very minimal. It's only a 1 step thing”
-Participant 7
“ADF improved our productivity! It helps us a lot”
-Kapil