In research, gathering the required data, running analysis on it and deriving insightful and novel findings based on established theories or to create new ones is
We recently were tasked to automate the process of interacting with a defined platform to gather millions of records, finalizing the output into a defined format.
The project involved working with a defined platform that is slow to load, buggy, and has tons of errors when overloaded with server requests. Moreover, there was also a significant time gap from a comparison of input data, to being presented with the output data from the platform.
We dissected the site in its whole, understood the constraints of the site through multiple trials and errors, and created a workflow that was 70% automated (the structure of the site made it impossible to fully automate due to multiple checks and balances). Thereafter, our team worked through the week to deliver the data in its complete form.
The client was very impressed by the final output data and the speed of delivery. We essentially were able to save them months of work, allowing them more time to expedite their research work and focus on deriving novel insights and supporting it with significant results.