1
2 Comments

Introducing DataPipelines, a visual UI for Apache Spark.

When I first used Apache Spark about 3 years ago I was blown away how fast and easy to learn it was. So I have been using it wherever I could ever since and gained quite a bit of experience with it.

Since day one I have also been looking for a visual front end for it to build ETL processes and data pipelines with. There are some products out there but most of them are expensive and bloated with features I will never use.

So a few months ago I decided to create my own visual data pipeline builder backed by Spark. It makes it a breeze to quickly inspect large amounts of data in Amazon S3.

It's still heavily under development but is fully usable. The key points are:
can be used to build pipelines step-by-step or just inspect data

  • visual UI so no coding or sql skills are necessary
  • input and output is CSV format from/to AWS S3
  • uses Apache Spark so can handle large datasets
  • sophisticated scheduler

I also managed to grab the perfect domain: https://www.datapipelines.com

With the MVP done comes the hard part. Marketing, business development, etc...
I'm very technically minded so currently looking for someone with business experience to team up with.

I'm also looking for a company who would be willing to give the product a go. I will provide it for free and ask only for feedback in return. If you are interested let me know.

posted to Icon for group Looking to Partner Up
Looking to Partner Up
on May 18, 2020
Trending on Indie Hackers
How we got 100+ clients in 48 hours from Product Hunt User Avatar 16 comments Meme marketing for startups 🔥 User Avatar 11 comments After 19,314 lines of code, i'm shutting down my project User Avatar 1 comment Need feedback for my product. User Avatar 1 comment We are live on Product Hunt User Avatar 1 comment Don't be a Jerk. Use this Tip Calculator. User Avatar 1 comment