A crawl instruction and an indexing instruction sound similar enough to be confusing. They act at different points.
I start with the intended outcome: prevent crawling, remove a public page from search, or keep a private site inaccessible. Those are different jobs. Checking the response from outside WordPress tells me whether the settings actually produce the intended result.
The short version
robots.txt tells compliant crawlers which requests they may make. A noindex instruction tells a supporting search engine not to retain a page in its index once it has fetched and read that instruction.
If a crawl block prevents that fetch, the search engine cannot see the page’s new noindex. Google spells this out in its noindex guidance. That is why an already indexed test page needs a different removal plan from a new test site that has never been public.
Use the setting the site already has
In WordPress, check the native search-visibility control and whether the SEO plugin honours it across page output, headers and sitemaps. Two competing site-wide switches make it harder to know which instruction will be served.
For a deliberate de-indexing process, make sure the crawler can reach the page carrying the removal instruction. Assign someone to check progress and restore the intended restrictions afterwards. A private development site also needs proper access controls.
Neither robots rules nor noindex provide access control. If information must stay private, I put authentication or an appropriate access restriction in front of it.
Read the response, not just the checkbox
A physical robots file can be served before WordPress gets involved. A page cache can return HTML generated before the setting changed. The admin screen can therefore be correct while the public response is still wrong.
- Fetch the actual robots file at the site root.
- Inspect a public page’s response headers and HTML.
- Check a second page type so a template-specific rule is not missed.
- Confirm the response is a real working page, not a server error dressed up as a passed check.
- Repeat after clearing the relevant cache and after deployment.
I keep the intended behaviour written down for each environment. A test site and a public site often need different outcomes, and a deployment should not silently swap one for the other.
The useful result is agreement between the setting, the served instructions and the purpose of the site. A green tick beside only one of those is not enough.