Data Loading into Snowflake using Matillion with a Meta data driven approach

About Client

The client discussed in the case is one of the famous fashion retail chains of Women and girls’ attire. It is widely known for its shift dresses, active-wear, shoes, accessories and print dresses. Being in the business for 60 years, the client has been a designer, marketer, and distributor of upscale collections for women. It has emerged as an affluent brand with a heritage and aesthetic based on the Palm Beach resort lifestyle.

The brand has demonstrated multi-generational appeal. Its products can be found in the company’s owned signature stores, e-commerce site, certain department stores, and a variety of independent specialty stores. Programmers.io (PIO) is supporting this fashion retail client in the field of Data analysis using business intelligence, EDI transactions and IBM i projects.

Retail and Consumer Goods

The Challenge

The client being into fashion retail needs real-time data of the customers to analyze and create its business strategy.
Following challenges were faced by the client:

As the customer data was stored on various data sources such as IBM i DB2, Oracle, SAP and Salesforce (SFDC, SFMC, SFSC), the history of the data could not be stored
The data had to be loaded in a centralised data warehouse for accessing and analysing real time data using generated visual reports, along with maintenance of data history
Data overload needed immediate integration with the Data Lake along with proper QA and validation of the data
The client expected the data loading and validation task duration to be around 6 months with a separate team of QA engineers and a greater number of resources that increased the overall cost of the project

Technology Stack

The Solution

diagram-flow
Programmers.io team handled the project immediately and optimized the turn-around time to just 1 week! We adopted integrating Matillion (as an engine) with our MetaData driven approach – which improved our time value.

Data experts at Programmers.io suggested to build a Data Lake first. The team performed end to end integration of the data sources such as Salesforce, IBM i DB2, Oracle, Flat files, and SAP with the data lake

Our team used the Metadata driven approach/configuration to run Matillion in loading 200+ SFDC tables into Snowflake using single job

The Data was exported from the source (Salesforce, IBM i DB2, Oracle and SAP) and staged in AWS S3. This data was ingested into Snowflake (Landing layer) where it was validated by the automated QA suite and transferred to Data Lake for comparison and history analysis

Using a metadata-driven approach this robust solution made the onboarding changes faster, streamlined development cycle and supported quick maintenance of data tables

Audit framework captured the error logs, data lineage etc. For each table with solution and helped in generating the health report of the entire process.

Cleaned and real time collected data was then transformed into visual reports for in-depth business insights and proper decision making for the client

Business Impact

Programmers.io team delivered the project from expected time frame of 6 months to 1 week with end-to-end development, integration, audit log mechanism, data validation, QA validation (generic Test case) etc.

The meta data driven approach ensured accuracy, flexibility and hold on optimized real time data. This saved a lot of time of the client and helped it focus on designing the right business strategy

The process involved just one job hence leading to low maintenance cost

The visual representation made the right information accessible and easy to understand, empowering the client to generate more leads and revenue.

Testimonials

"The team is performing really well. They are learning fast and work with our team and culture well."

‐ Applications Director

"The team at Programmers.io has been fantastic. I was surprised at how fast they learned our system and were producing code…"

‐ EDI Manager

"I have been working with Programmers IO for more than 5 years now and I have been pleased with all projects and developers that I have worked with. We have had a few issues here and there but they have always fixed and made it right. They have been an excellent addition to our business."

‐ Director Of Development

"Always a pleasure to work with."

‐ Project Manager

"It has been a pleasure. Everyone is very responsive, and we get good quality work."

‐ Corporate Project Manager

"The professionalism and work that was done is outstanding. The ability to collaborate and communicate about changes is exceptional. The team is very responsive and supportive even with questions that did not pertain to the e-info aspect. They are great to work with and were very helpful in making all this work effectively and efficiently."

‐ IT Manager

"Team is doing great. Very engaged, very skilled in the area of IBMi/AS400 capabilities. Works very well proposing solutions to presented problem statements or use cases."

‐ Director of Information Technology

"Great team to work with."

‐ Director of Information Technology

"Happy with the performance. The team is self-driving, which is something I’ve had struggles with in the past. Very knowledgeable on the IBMi and requires minimal guidance."

‐ Director of Information Technology

"We LOVE your solution!"

‐ Sales Director

"Very responsive and explain things to me in a way I can understand since I am not an AS400 person at all. The level of service and support with them is one of the best I have had with an outside contractor."

‐ Information Technology Supervisor

"We find your team to be true business partners. Exceptionally responsive, strong domain knowledge, and outstanding at execution."

‐ Chief Integration Officer

"You guys Rock!"

‐ EDI Coordinator

"Over the past year I have worked with them, they have worked very hard to understand our highly customized system and troubleshoot things with little or no documentation while managing to keep the business up and running normally. Without the hard work and dedication, they have shown, I know we would have had some issues causing downtime or lost production. I am looking forward to continuing working them in the next year as well."

‐ Information Technology Supervisor

"The team that works with us is outstanding and are excellent to work with."

‐ Information Technology Supervisor

"The team meets and exceeds expectations. There isn't opportunity to "deliver early" the way I assign work to the team, they deliver exactly as I need which is great."

‐ Senior Consultant, Project Manager

"Nice to deal with. Professional results."

‐ Manager of Information Technology

"Team is professional and skilled"

‐ System Engineer Analyst Principal - IT

"The Programmers.io team is very flexible and accommodating."

‐ IT Project Manager

"I have been extremely pleased with the partnership we have had with Programmers.io and the resources to assist us."

‐ IT Analyst

"Our experience with Programmers.io has been exceptional in quantity and quality of work."

‐ Lead Developer

"The programmers I have worked with have all been great! They think out of the box and make suggestions, and they research and figure things out on their own, which is a tremendous help to me. I would recommend them to any prospective clients you have."

‐ Project Manager

"Everyone we have dealt with is great and they have helped make my job in managing projects a lot easier! Not only do they do the work requested, they ask great questions and make great suggestions."

‐ Project Manager

"I just want to say thanks so much for the guys that have been working on the Agent Sync / Salesforce project. They literally saved my job this week. I really appreciate all they do!"

‐ Project Manager

"We have had an exceptional experience with this new team in 2021.They came up to speed very quickly and have handled every assignment professionally and delivered great value & quality."

‐ Director of Application Delivery

"I've been consistently impressed with Programmers IO. There have been a couple issues along the way, but we've been able to resolve that quickly with options for what to do. I think the team working with our company is successful and happy with their work, work schedules, having clear expectations, and peer relationships."

‐ Director of Engineering

"I appreciate that the contractors we have working with us have support from ProgrammersIO. The management team is always available and helps to resolve problems or provide needed information."

‐ Director of Engineering

"The team with PIO has done a wonderful job for us!"

‐ IT Manager

"The team we work with is great in every way. We feel confident in the support we get during the day to day to the yearly maintenance."

‐ CFO

"The team goes above and beyond."

‐ IT Global Service Desk Manager

"The best support group at ____ and one of the best I've ever worked with. The response is great, constantly keeping in touch and when I ask for changes, they are done promptly and accurately."

‐ President

"It's been very easy to work with the developers at Programmer.io. They do a great job!"

‐ IT Project Manager

"Changes are made quickly and correctly. My programmers here hate the way I test their code because I always find a way to break it. I’ve tried to break the PIO code and I couldn’t."

‐ Vice President of Operations

"Are "Very Happy" and "Very Effective" the highest I can go?"

‐ Vice President of Operations

"I am very comfortable knowing that I can count P.io as part of my team."

‐ IT Project Manager

"The entire programming team at Programmers.io is amazing to work with."

‐ Vice President of Operations

"We've been very happy with Programmers.IO. So much so, we are rotating our other teams to include Programmers.IO into the mix."

‐ Business Intelligence Mgr.

"The team has been both patient and extremely responsive. They have worked through all concerns and issues and are dedicated to making our goals work. I am very pleased."

‐ Human Resource Manager

"I have really enjoyed having _______ on our team. She has helped me much to organize the work of the developers."

‐ Senior Director of Application Development

"The team has done all that we've asked of them. We'd like to continue to increase the work scope as time goes on. Nothing but great things to say so far regarding the work completed by the team."

‐ Director, Support Manager

"I appreciate how dedicated the team is to working hard on projects and getting things done in a timely matter. They take every effort to try and completely understand not only the project but how it will be used in our business."

‐ Computer Programmer

"I wanted to take a moment to highlight and commemorate the efforts from our dedicated IO team. To preface, the projects we work on here are complex: with rigid objectives, budgets, and timelines. Expectations are always high and are ever changing. From the start of our relationship, the IO team was able to exceed expectations and make our business wishes a reality."

‐ Project Manager

"The PIO Team has been great. Very professional, helpful and with great attitudes. We enjoy working with them and get many compliments from within our IT department and from other areas of the company. Keep up the good work!"

‐ Director of Technical Business Systems

"Programmers.io has been so helpful. We have been using them for over 18 months and I am constantly looking for more projects to send them."

‐ Director, Implementation and Support, Land & Vitals

"I have been working with Programmers IO of years and they have always been able to provide the programmers we have needed to grow our business. They are honest and upfront with everything and have delivered what was promised. They are an asset to the growth and our business."

‐ Director Of Development

"We are very happy with the services programmers.io provides."

‐ Vice President Information Technology

"The team is extremely responsive, flexible, and produces quality work consistently."

‐ IT Project Manager

"Great team, very engaged and creative in enabling our organization to have operate efficiently and deploy new value add solutions."

‐ Director of Information Technology

"Very good on AS400 development. Account rep has always been helpful and responsive."

‐ Director of Information Technology

"They are very dependable and complete their tasks in a timely manner."

‐ Manager, Information Technology

"I continue to appreciate the consistently good work of Vipul Kumar as the developer on my project."

‐ Project Manager/Business Analyst

"I appreciate the communication and the quick turnaround from Programmers IO!"

‐ IT Business Analyst

"We have worked with PIO since early 2020 and it has been a great experience for us and allowed us to gain additional automation of manual processes for Accounting group. It is a great experience to work with knowledgeable developers like Kapil, Praveen and Tushar on our projects. Thank you!"

‐ IT Project Manager

"The response time and service Programmers.io provides is consistently excellent! It is much appreciated."

‐ Director of Engineering

"We are still early in the relationship with Programmers.io. So far you are doing a good job completing the requests we have given you. As we use your services on more complex projects I will have a more defined opinion I can share in the future."

‐ Corporate Director, Information Technology

"Guys are awesome."

‐ IT Global Service Desk Manager

"Every developer we have worked with has been great! They are great at analyzing and making great suggestions, in addition to coding all that we ask for. I just wish they were here LOL. Even with the time difference, they are always willing to work on "our hours" when needed for deadlines and projects as needed. Thanks for everything!"

‐ Chief Information Security Officer

Video Testimonials

Effectively Extending
Your Team

Ken Stewart, PMP, CSM

Vice president of operations & Engineering

PSigen Software Inc.

Play Video

Video Testimonials

Onboarding The Programmers.io Team

Brandy Churchill

Software Development Manager

Western National Insurance

Play Video

Video Testimonials

Effectively Extending
Your Team

Jackie Flummer

Director, Technical Business Systems

AmerCareRoyal

Play Video

Video Testimonials

Effectively Extending
Your Team

Luis Gonzalez

President

SalesScoreKeeper

Play Video

View All

We are programming experts in both legacy and modern platforms.

We are programming experts in both legacy and modern platforms.

We are programming experts in both legacy and modern platforms.

Data Loading into Snowflake using Matillion with a Meta data driven approach

About Client

Retail and Consumer Goods

The Challenge

Technology Stack

The Solution

Business Impact

Read More Case Studies

Testimonials

"The team is performing really well. They are learning fast and work with our team and culture well."

"The team at Programmers.io has been fantastic. I was surprised at how fast they learned our system and were producing code…"

"I have been working with Programmers IO for more than 5 years now and I have been pleased with all projects and developers that I have worked with. We have had a few issues here and there but they have always fixed and made it right. They have been an excellent addition to our business."

"Always a pleasure to work with."

"It has been a pleasure. Everyone is very responsive, and we get good quality work."

"Team is doing great. Very engaged, very skilled in the area of IBMi/AS400 capabilities. Works very well proposing solutions to presented problem statements or use cases."

"Great team to work with."

"Happy with the performance. The team is self-driving, which is something I’ve had struggles with in the past. Very knowledgeable on the IBMi and requires minimal guidance."

"We LOVE your solution!"

"Very responsive and explain things to me in a way I can understand since I am not an AS400 person at all. The level of service and support with them is one of the best I have had with an outside contractor."

"We find your team to be true business partners. Exceptionally responsive, strong domain knowledge, and outstanding at execution."

"You guys Rock!"

"The team that works with us is outstanding and are excellent to work with."

"The team meets and exceeds expectations. There isn't opportunity to "deliver early" the way I assign work to the team, they deliver exactly as I need which is great."

"Nice to deal with. Professional results."

"Team is professional and skilled"

"The Programmers.io team is very flexible and accommodating."

"I have been extremely pleased with the partnership we have had with Programmers.io and the resources to assist us."

"Our experience with Programmers.io has been exceptional in quantity and quality of work."

"The programmers I have worked with have all been great! They think out of the box and make suggestions, and they research and figure things out on their own, which is a tremendous help to me. I would recommend them to any prospective clients you have."

"Everyone we have dealt with is great and they have helped make my job in managing projects a lot easier! Not only do they do the work requested, they ask great questions and make great suggestions."

"I just want to say thanks so much for the guys that have been working on the Agent Sync / Salesforce project. They literally saved my job this week. I really appreciate all they do!"

"We have had an exceptional experience with this new team in 2021.They came up to speed very quickly and have handled every assignment professionally and delivered great value & quality."

"I appreciate that the contractors we have working with us have support from ProgrammersIO. The management team is always available and helps to resolve problems or provide needed information."

"The team with PIO has done a wonderful job for us!"

"The team we work with is great in every way. We feel confident in the support we get during the day to day to the yearly maintenance."

"The team goes above and beyond."

"The best support group at ____ and one of the best I've ever worked with. The response is great, constantly keeping in touch and when I ask for changes, they are done promptly and accurately."

"It's been very easy to work with the developers at Programmer.io. They do a great job!"

"Changes are made quickly and correctly. My programmers here hate the way I test their code because I always find a way to break it. I’ve tried to break the PIO code and I couldn’t."

"Are "Very Happy" and "Very Effective" the highest I can go?"

"I am very comfortable knowing that I can count P.io as part of my team."

"The entire programming team at Programmers.io is amazing to work with."

"We've been very happy with Programmers.IO. So much so, we are rotating our other teams to include Programmers.IO into the mix."

"The team has been both patient and extremely responsive. They have worked through all concerns and issues and are dedicated to making our goals work. I am very pleased."

"I have really enjoyed having _______ on our team. She has helped me much to organize the work of the developers."

"The team has done all that we've asked of them. We'd like to continue to increase the work scope as time goes on. Nothing but great things to say so far regarding the work completed by the team."

"I appreciate how dedicated the team is to working hard on projects and getting things done in a timely matter. They take every effort to try and completely understand not only the project but how it will be used in our business."

"The PIO Team has been great. Very professional, helpful and with great attitudes. We enjoy working with them and get many compliments from within our IT department and from other areas of the company. Keep up the good work!"

"Programmers.io has been so helpful. We have been using them for over 18 months and I am constantly looking for more projects to send them."

"I have been working with Programmers IO of years and they have always been able to provide the programmers we have needed to grow our business. They are honest and upfront with everything and have delivered what was promised. They are an asset to the growth and our business."

"We are very happy with the services programmers.io provides."

"The team is extremely responsive, flexible, and produces quality work consistently."

"Great team, very engaged and creative in enabling our organization to have operate efficiently and deploy new value add solutions."

"Very good on AS400 development. Account rep has always been helpful and responsive."

"They are very dependable and complete their tasks in a timely manner."

"I continue to appreciate the consistently good work of Vipul Kumar as the developer on my project."

"I appreciate the communication and the quick turnaround from Programmers IO!"

"We have worked with PIO since early 2020 and it has been a great experience for us and allowed us to gain additional automation of manual processes for Accounting group. It is a great experience to work with knowledgeable developers like Kapil, Praveen and Tushar on our projects. Thank you!"

"The response time and service Programmers.io provides is consistently excellent! It is much appreciated."

"We are still early in the relationship with Programmers.io. So far you are doing a good job completing the requests we have given you. As we use your services on more complex projects I will have a more defined opinion I can share in the future."

"Guys are awesome."

Video Testimonials

Effectively Extending Your Team

Ken Stewart, PMP, CSM

Play Video

Video Testimonials

Onboarding The Programmers.io Team

Brandy Churchill

Play Video

Video Testimonials

Effectively Extending Your Team

Jackie Flummer

Play Video

Video Testimonials

Effectively Extending Your Team

Luis Gonzalez

Play Video

Effectively Extending
Your Team

Effectively Extending
Your Team

Effectively Extending
Your Team

Find out how Programmers.io can help your company
get to the next level.