Web Search is Broken: The Problem
Hi IH!
I've been having this idea for so long and it's actually a product that I wish exists to solve search!
Current issue is searching for days and days while only finding a few handful useful results for what you're looking for, other times you don't even find what you're looking for.
I want to know if anyone has this same problem, and wish that search was much simpler? What are your pain points that you want to solve regarding this main problem?
Trending on Indie Hackers
Protect your momentum like your life depends on It
Opsgenie vs. Splunk: Choosing the Right Incident Management Solution
Yes. I agree with Tyler Cowen's assertion that Google search is now less useful for learning than Twitter search is.
The problem is almost all searches result in someone trying to sell something to you. That incentive isn't well aligned with my incentive to discover what is true about something.
For example, if you search for a web hosting with a service for their price, you'll generally find almost nothing but SEO-optimized "review" sites which are being paid by web hosts for placement. If you search for an online programming course, you'll almost certainly land on Medium blog posts stuffed with affiliate links as opposed to honest reviews or relevant Reddit threads.
If you search for something advertisers really fight over like financial or legal services, the results are even worse.
Yes, I completely agree. Most search results are filled with total garbage affiliate content. There has to be a better way than this.
This sounds like a massive undertaking though.
Thanks for your reply.
How much time does it take you to find what you're looking for? And how many links/websites do you need to go through to find what your looking for?
What can make search better for you? What is search missing?
About being a massive undertaking, it actually is and have thought so much about. I have solution that I will share soon in another post. But know want to know if anyone is facing this same issue and learn more about what other people have on mind.
I was recently searching for a VPN and it looks like they are mostly ads or SEO optimized posts on each page. It made it impossible to trust any of the search results.
I have no idea how to fix this, but maybe discounting posts with affiliate links would be a good place to start.
That's a nice idea.
So if you think more closely to this example, since these sites are ranking in the top those websites are giving them access to those affiliate links. So if you change those websites's ranking they won't get as much affiliate links as they do now. I might be wrong about this assumption, but that's the outcome that came to mind.
Yes Alchemist! This is what I mean! You've really nailed what I had in mind.
People with bad SEO might have good content, but they aren't appearing in search results because of this bad low level competition.
As you have mentioned, I usually search for a problem that I want to learn more about and read as many good articles, blogs, or essays about the subject, but face the same issue you're having now.
That's the problem I want to solve, and not "search" in it's general terms as I have mentioned to a few other IH that I'm not trying to compete with google or the current search engines.
I really thank you for sharing your thoughts!
Incidentally, one of the most profound ideas I've ever read was on medium (career-altering-totally-nonobvious-I-wish-I-knew-this-in-college type)
I don't have an exact solution for this but we're building a place where you can follow curated list of resources which helps a lot of people to find great content. For instance you can follow a list about "Product-market fit" to learn about it, instead of searching all the content yourself.
www.jooseph.com
For search we had the exact same problem
Nice site and that's a nice way to solve this problem.
My solution might be a bit similar in some aspects, but it's not considered a curated list. I will still have the life of a search engine. I will share my solution in another post soon, would be great to have you join tge discussion.
But I do wish you all the best in your solution!
You too,
let us know if we can help on something. Here's couple of Quora questions about this subject, I think you already take a look at those but in case you don't;
Actually, I just saw these for the first time. Thanks very much for sharing it! It's glad to know that there are many people that have this issue.
Btw, how much time does it take you to find what you're looking for? And how many links/websites do you need to go through to find what your looking for?
What do you search for that takes days and days to find only a handful of results?
As an example, how to start a B2B blog in a certain industry. Which is very specific and not enough results to use with how to expand my knowledge in that area. I take days as i try other keywords to try to get the results that I'm looking for.
But basic things and searches, it's not difficult to find a handful of articles for what I'm looking for.
Thanks. Remain unconvinced. Seems many "How to start a B2B blog" guides are easy to find - some industry specific others generic, so need to add own material. Obviously was just example but don't see it. Unless looking to shrink-wrap every possible endeavour into a "how to pack a tent" guide.
You're welcome and thanks for joining the discussion. Some of the comments in this discussion made better, more detailed, and specific examples to the same I'm facing.
I know that not everyone will have this same problem, but it's nice having different point of views. So thanks again for joining the discussion!
Agree comments from @alchemist and @jetsetcitizen strike a chord - where signal is buried in noise of SEO-optimised review sites, that tend to recommend same sites, which are the ones with highest affiliate payments.
Yeah that's another issue I would want to solve and demote these sites from ranking high in the search results.
Not sure about web search as a whole, but Google definitely is becoming less and less useful. I search a lot related to marketing related topics as part of my job, and the number of times sites like CrazyEgg and NeilPatel show up is crazy. And many times, they don't even address my query. If you ask me, the new BERT algo update is all hogwash.
I like Bing - though I don't use it consistently yet, it returns more diverse results instead of clinging on to the handful of sites that the search engine considers is "authority"
Hi Anand, thanks for your very detailed reply. That is the same issue I face but trying to learn or expand my knowledge in a certain topic and not finding the right content easily.
How much time does it take you to find what you're looking for? And how many links/websites do you need to go through to find what your looking for?
What can make search better for you? What is search missing?
Hey Ali, I have imagined a solution to this. but not really sure if it's practical.
So you must have noticed how Google has this "Show fewer of this" option when you want to close an ad or don't want to see some source in Google News.
I think a Chrome extension that would do something similar for Google Search would be great. So if I don't like NeilPatel or CrazyEgg, I can click and choose this option with the extension.
This way, by default, the extension would hide all these websites from my search results. The extension will also allow me to toggle so with one click, I can view all the hidden results.
What do you think?
That's a really good solution that I haven't thought about!
My solution is different in implementation, but close to the same conclusion of your solution. But I will think more about your suggestion and if I can incorporate it with mine.
Again thanks Anand for sharing! Hope you also join the discussion when I propose my solution in another post.
Sure, have fun building it. Would love to check it out.
Thanks, I really appreciate it! Will hear soon about my solution hopefully with an MVP also.
Actually no? Maybe some very obscure stack traces from elastic search or something. But honestly I can find pretty much anything on Google or DuckDuckGo
Hi Klaas, thanks for your reply.
Can you please tell me on average how much time it takes you to find what you're looking for?
seconds, sorry I don't really know what your plan is. But I think google has pretty much nailed search for the past 20 years
One more question how many links do you find what you're looking for?
And true Google has it figured out, but I'm not going to do what they're doing and not think of competing with them at least not directly.
Soon I will share my solution, and hope you can join in the discussion.
Something like they do? https://www.visymo.com/
Thanks for your reply, Condor.
This is the first time I here of them. But no they're similar to the typical search model. My solution from what I know isn't typically used in search engines, but in general terms it's not something ground breaking as its used in other types of sites.
Soon I will share the solution in another post, and hope you can join the discussion.
My major pain point is looking for something and finding a competing brand as first result in ads, that's so annoying.
Also tracking although helpful narrow my information sources.
I've set https://duckduckgo.com/ as main search engine and I'm very happy with it. When the search results are poor, I just retry with !g which triggers Google.
Looking forward to heard about your solution, and best of luck on implementing it.
Hi Oscar, thanks very much for your reply.
I will share it soon, and I got some good points to take into consideration and many have interest in it which is a good thing. And thanks too and best of luck to you too!
You're welcome.
Btw and for more information I usually find must of my information in stackoverflow and Wikipedia
Yeah both are great resources that I use too.
Yes, plain old search can be hard. I've learned over the years to search like this:
(Replace with your own choice of a community you trust for your search topic, for example HN, SO, or your subreddit of choice)
Thanks OT! Actually I use that when I found a specific source of content for what I'm looking for.
So do you have any problems with search in general?
I agree. I seem to add "Reddit" more often than ever before to my searches as I find that Reddit threads have more accurate and more useful information than many of the sites that search engines list towards the top of their results.
I don't use Reddit unless google points me to it, but I think I should check it out more.
How much time does it take you to find what you're looking for in search engines? And how many links/websites do you need to go through to find what your looking for?
What can make search better for you? What is search missing?
imho - google does a really good job for concrete searches (like for someone looking for 'what is seo') but google can only do a decent job for abstract / contextual searches (like for someone who wants to improve seo his/her site). Because no one else can know about that particular situation better than the person him/herself.
Of course, google has lot of noise with ads - but its their lifeline for building & providing the search infra for free.
Am open to alternatives where I don't have to spend hours searching to find a solution for "my" problem (am sure almost all of the knowledge workers at some point - spend hours in searches).
And to me the alternative looks something like I could explain my problem to someone else (who is knowledgeable) - who can find me an answer while I work on my next problem. I think this is where sites like reddit, twitter, stack-overflow (basically a n/w site) comes in?
Thanks for the detailed post! Yes you're right about finding simple stuff from google but detailed and specific knowledge is difficult.
And the way you're solving the problem is also nice. Those three including also Quora are good to use for an question and answer platform. My idea is close enough to these but trying to implement the searching element and making it low barrier of using the service. I will share my solution in another post soon, and would be gald if you also joined the discussion.
But can you please tell me, how much time does it take you to find what you're looking for in search engines? And how many links/websites do you need to go through to find what your looking for?
What can make search better for you? What is search missing?
Totally agree.
Hi Shashank, thanks for your reply.
How much time does it take you to find what you're looking for? And how many links/websites do you need to go through to find what your looking for?
What can make search better for you? What is search missing?
Well, this was one the reason I built pipfeed.com
That's a really nice way to solve this issue by curating content.
My solution in mind isn't AI, but will share it soon in another post. Hope you can share your thoughts during that discussion.
And thanks again Shashank for joining the discussion!
Sure, looking forward to it.
Search works pretty well for me. Sure the trend of SEO facing content being what you see isn't great, but I can still pretty much always find legitimate reviews or content if I want to without much hassle.
Hi and thanks for your reply.
I'm going to ask you the same questions I asked the other users, how much time does it take you to find what you're looking for? And how many links/websites do you need to go through to find what your looking for?
Not long and not many websites.
Can you be more specific? (making a close assumption)
No really, I can't remember the last time I was unhappy with Google search. I find things quickly and easy enough that I can't remember the couple missteps it takes me to find something sometimes.
That's good and thanks again for sharing!
I'd like you you join the discussion when I propose my solution to my ptoblem. I know now it's not a problem you're facing, but it's good to take as much feedback as possible.
No thanks buddy, I'm pretty busy and not optimistic on someone trying to take on Google.
I understand and thanks again though!
The thing here is that, you have to search like a robot not a human because you're working with robots when searching. Mostly you don't have to type in sentences but a combination of keywords or phrases. That's how I do it
Hi Farhan, thanks for the reply.
How much time does it take you to find what you're looking for? And how many links/websites do you need to go through to find what your looking for?
Searching the web? Or your computer? I'd avoid taking on web search
Searching the web, sorry for not being specific.
And why do yout think I should avoid it?
Because Google! The only players that could dream of taking on Google are pretty much Microsoft (trying their best with Bing), Apple or Facebook.
Even if you could somehow create better tech than Google Search, the problem you'd need to overcome is human habit. Maybe Bing's results are already better? I have no idea because I just mindlessly search on Google like everyone else.
And yet Duckduck go has been doing it successfully and built a gigantic business in the process. It's still growing quickly, too.
Good point, completely forgot about DDG. You're right that a profitable business could exist in search. I was coming at it from the idea of being the new search engine in town, which probably wasn't what OP had in mind
I don't use DDG though, I use only google. But thanks for the mention of it.
That's a valid point you have there. And actually one that I have thought about so much.
My solution to this problem (which i will share in another post as I'm finalizing a few things regarding it) is different from what the current search engines are doing. Actually I'm not thinking of competing with them at least not directly, I want to provide something new which they aren't providing.
From this post, I want to understand if anyone else has this same problem and what can be done about it as a solution. Or what pain points you're facing from current search?
This comment was deleted 5 years ago.
It's not a small list and for the time being I don't have anything specific. But MOST times I do a search I face this issue.
Do you have this same problem? Do you have any solutions? Any other related pain points to search that you have?
This comment was deleted 5 years ago.
Thanks for passing by then.