lawfirm.dev-bot

lawfirm.dev keeps a source-tracked directory of licensed attorneys and their firms. Our crawler, lawfirm.dev-bot/1.0, reads two kinds of public pages: state bar and court lawyer directories, and law firms' own websites.

On a firm's site it reads the attorney bios, vCards and contact page: names, titles, practice areas, office addresses, phone numbers and the email addresses the firm publishes. Every fact we keep records the page it came from. We don't read anything behind a login, submit forms, or work around a site that hides its addresses.

How it behaves

• It obeys robots.txt, including rules addressed to lawfirm.dev-bot.
• It waits at least 1.5 seconds between requests to the same site, and fetches pages, not assets.
• It revisits a site about once a month.

Opting out

To keep the crawler off your site, add this to your robots.txt:

User-agent: lawfirm.dev-bot
Disallow: /

Attorneys can also correct or claim their own listing, free; attorneys never pay. For anything else, write to scrape-contact@lawfirm.dev.