Takeaways

• Across the top 5 positions, pages that fully answer more PAA questions tend to rank higher.
• Commercial and transactional queries show the strongest correlations; navigational and informational queries are weaker.
• Beyond page 1, the correlation drops sharply and can even reverse in lower page 2 positions — likely due to small sample sizes and SERP variability.
• Answering PAA questions alone isn't enough to rank well; technical quality, external signals, and user experience still matter.
• Covering a wide range of related questions can make content semantically richer, which may support ranking potential.

Methodology

Using SerpAPI, OpenAI's API, and a PAA question API, we can test how rankings relate to PAA questions answered. Here's how:

1. Start with a list of seed keywords.
2. Fetch the top 20 organic results.
3. For each keyword, pull related PAA questions via the PAA API.
4. For each organic result, extract the main readable content.
5. Send the content and each question to GPT-3.5 Turbo, asking, “Does this page answer this question? Fully, partially, or not at all?”.

This enables us to see:

• The number of PAA questions answered by results ranking within the same SERP.
• Whether the number of PAA questions answered correlates with rankings.
• How all of the above is impacted by search intent.

Note: During this study, we found several areas that could be improved upon in the future. You can find those opportunities and caveats following the conclusion section of this study.

The detailed process

If you want to replicate this study, we outline the major portions of the process below:

• Keyword selection
• Working pipeline
• API specifics

Keyword selection

For this test, we needed a significant number of keywords to smooth over potential biases or low-level data issues. We ran the test using:

• 563 keywords
• 4 search intent stages
• 3 niches

The keywords were seeded per niche using the DataForSEO Keyword Ideas API input, so they were broadly related, essentially random, and grouped around the four standard intent stages (informational, navigational, commercial, and transactional).

Working pipeline

To conduct our analysis, we followed the steps below:

1. Preparation and setup
   • Connect to data services by API.
   • Build a list of seed queries (keywords) to research.
2. For each keyword (seed query)
   • Load related PAA questions for each keyword.
   • Get Google results — uses SerpAPI to fetch the top Google search results for that keyword.
3. For each result URL
   • Extract main content using a headless Chrome browser and a 'readability' API.
   • If content is insufficient (too short or missing), skip the page.
4. For each question
   • Send the extracted content to GPT-3.5 Turbo and prompt it to evaluate whether the content answers the question “fully”, “partially”, or “not at all”, with reasoning.
   • Record results and store the status and reasoning for each keyword, question, and URL.
   • Save all collected results into a CSV file.
5. Logging and error management
   • Log any errors (API failures, extraction issues, etc.) with timestamps for troubleshooting.

API specifics

If you want to replicate this yourself (or conduct similar research) here are the specifics around the APIs used for the pipeline.

• SerpAPI fetches organic URLs from the top 20 results.
• OpenAI's GPT-3.5 Turbo is more than adequate for the task at hand, whilst keeping costs manageable.
• PAA fetches went to a depth of 20 questions to keep them closely associated with the seed term, whilst not being too limiting.

Results

The results below show the number of questions answered by the top 20, 10, and 5 positions across the four intent stages. For further analysis, we also ran the same test with partially answered questions (using a ‘question score’).

Because this methodology is binary (the content either does or doesn't answer the question), we're potentially writing off a lot of half-answered questions, which are valuable if there is no better answer.

Rather than only using a simple yes/no approach, we also assigned points based on whether a question is wholly or partially answered to see how this might play into search rankings:

• Answered “yes” = 1pt
• “partial” = 0.5pts
• “no” = 0pts

This gives less credit to partially answered questions, but may help us see if we're missing any nuance.

How questions answered/question score correlate with ranking position

Note: As the positions descend but the averages ascend, a negative correlation is actually a good thing.

One striking pattern is that the correlation scores climb when we look at the top 20 positions. It's also worth noting that we have far fewer results recorded from the bottom of page 2. The change in average could be impacted by a lower number of results skewing overall averages.

Key findings

Across the top 5 organic search positions, there is a strong, positive correlation between the number of fully answered questions and ranking.

When we added question scoring (which accounts not only for fully answered questions, but also for partial or in-depth snippets) it added useful nuance to the analysis. However, its correlation with rank is weaker than the raw ‘questions answered’ metric. This suggests that, while partial coverage and depth matter, outright completeness appears to be more important.

For navigational queries (i.e., where users seek a specific site or page), the relationships weaken further. There is still a clear trend, where higher questions-answered counts and stronger question scores tend to align with better rankings, but neither metric has as much influence here as it does for other intent query types.

Surprisingly, informational queries don't display the strongest correlations (despite our expectation that thoroughly informative content would naturally answer more questions). Several factors could be at play, but our sample size makes us cautious about drawing firm conclusions. It is also possible that there are more questions at play, or perhaps that answering fewer, more specific questions is more significant than a breadth of answers.

Finally, beyond the first page — and especially in the lower slots of page 1 and throughout page 2 — the link between either metric and ranking diminishes sharply. By the bottom half of page 2, the trend even reverses: pages with fewer answers or lower question scores sometimes rank better.

Part of this anomaly stems from a statistical element: because positions 18–20 appear less frequently in the dataset, each additional answered question (full or partial) disproportionately inflates the average, creating a misleading spike in the ‘questions answered’ metric without reflecting a real ranking advantage.

Answering questions is crucial, but you still need a robust user experience and external quality signals.

It seems as though questions answered correlate with ranking position — specifically with regard to the top 5 positions. This may not necessarily be because a number of questions is answered on the page, but because you are covering a lot of the subject with the content that could lead to a positive experience for the user. Clearly, covering questions around an area makes that page semantically richer, too.

More time spent on crawling, scraping, and analysis would likely yield more complete results, but the directionality of the results would be the same. Every step of this process (keyword selection, niche selection, intent designation, and analysis) would likely have more of an impact on results, so if we wanted to be more concrete, we would need to significantly increase the number of keywords, ensure the pipeline is more robust, and create a more nuanced view of the intent stages.

Put simply, if you take all the PAAs for a niche and answer all those questions, you still need to have stronger technical standards, well written copy, external quality signals, and evidence of user satisfaction to rank well. So in isolation answering PAA questions is not enough, but it could be a significant factor when all are compared.

Methodology: Room for improvement

Below are some obstacles we could not fully address during the first iteration of this study:

• Readability errors — Some pages may not extract correctly (due to paywalls, heavy JS, or unusual layouts).
• Low-content filtering — Skips any page with <100 words, which might discard pages that answer succinctly.
• Bot/scraping protection — Some websites actively block or serve different content to bots or headless browsers.
• There's no built-in manual review to check if ChatGPT 3.5-Turbo's answers are accurate.
• If a top Google result can't be scraped or the content is empty, that result was ignored.
• If the questions can't be loaded (i.e., because there are none), that keyword was skipped for the gap analysis.
• A lot of sites (YouTube, Spotify, etc.) do not have content to scrape in the same sense for this kind of analysis. They are judged to not answer a question, even if they may have.

Some more general caveats about the integrity of the data itself, keyword selection, and potential risks in using LLMs include:

• Correlation is not causation — Answering questions might not cause high rankings, but could be a side effect of good content.
• SERP features bias — Featured snippets or People Also Ask boxes themselves might affect which pages get selected and ranked.
• Sample bias — If the keywords/questions aren't representative, it may not be appropriate to generalize results.
• Reliability of "answer status" — As mentioned before, LLM judgment is strong, but not perfect.