In research, gathering the required data, running analysis on it and deriving insightful and novel findings based on established theories or to create new ones is a common practice. However, data collection is a tedious and cumbersome activity, often taking up a notable amount of a researcher’s effort, severely limiting his available bandwidth to actually work on the data and find significant results.

We recently were tasked to automate the process of interacting with a defined platform to gather millions of records, finalizing the output into a defined format.

Project Brief

The project involved working with a defined platform that is slow to load, buggy, and has tons of errors when overloaded with server requests. Moreover, there was also a significant time gap from a comparison of input data, to being presented with the output data from the platform.

Project Execution

We dissected the site in its whole, understood the constraints of the site through multiple trials and errors, and created a workflow that was 70% automated (the structure of the site made it impossible to fully automate due to multiple checks and balances). Thereafter, our team worked through the week to deliver the data in its complete form.

Project Outcome

The client was very impressed by the final output data and the speed of delivery. We essentially were able to save them months of work, allowing them more time to expedite their research work and focus on deriving novel insights and supporting it with significant results.