Eray Bartan
Eray Bartan Blog

Eray Bartan Blog

Web Scraping with Jsdom and NodeJS

Web Scraping with Jsdom and NodeJS

Eray Bartan's photo
Eray Bartan

Published on Aug 23, 2021

1 min read

Subscribe to my newsletter and never miss my upcoming articles

My goal is to find a data scraping tool compatible with the Nextjs pages/API folder.

The article helped me a lot.

I was able to capture live data with the following file.

const fs = require('fs');
import got from 'got';
import jsdom from "jsdom";
const { JSDOM } = jsdom;
const vgmUrl= 'https://mugla.eczaneleri.org/bodrum/nobetci-eczaneler.html';
export default (req, res) => {
    const eczanelistesi = []
    got(vgmUrl).then(response => {
        const dom = new JSDOM(response.body);
        dom.window.document.querySelectorAll('li.media').forEach(link => {
            const eczaneadi = link.querySelector('h4').textContent
            eczanelistesi.push({'eczaneadi':eczaneadi.toString().replace(/[\n\r]+|[\s]{2,}/g, ' ').trim()});
          });
        res.json({ 
            name: dom.window.document.querySelector('title').textContent,            
            eczanelistesi:eczanelistesi},
        )
      }).catch(err => {
        console.log(err);
      });
  }

pages/index.js like;

import Head from "next/head";
import Layout from "../components/layout";
export default function Home({data}) {
  return (
    <Layout>
      <Head>
        <title>Property for sale in Bodrum</title>
        <link rel="icon" href="/favicon.ico" />
      </Head>
      <div className="p-4 py-12">
        <img
          className="h-10"
          src="/house.svg"
          alt="properties bodrum turkey sale"
        ></img>
        <img
          className=" h-64 w-full mt-2 object-cover object-top rounded-xl shadow-xl"
          src="/properties-bodrum-work.jpg"
          alt="properties bodrum turkey sale"
        ></img>
      </div>
      <div className="text-3xl text-center text-red-900">
        Welcom Bodrum Properties
      </div>
      <div><ul>{data.eczanelistesi.map((eczane) => (
        <li>{eczane.eczaneadi}</li>

      ))}</ul></div>

      </Layout>
  );
}


export async function getStaticProps(context) {
   const res = await fetch(`https://properties.bodrum.work/api/eczane`)
  const data = await res.json()
  if (!data) {
    return {
      notFound: true,
    }
  } 

  console.log(data.eczanelistesi)
 return {
    props: { data }, // will be passed to the page component as props
  } 
}
 
Share this

Impressum

Technical Specialties

  • React Native App Development
  • Neo4j Graph Database Beginner Developer
  • Google Data Studio Dashboard Design
  • Micosoft Power BI Dashboard Design

Industry Experience

  • Municiapilty (IT Department, Data Analyst)
  • Real Estate (Web Design, Social Media Publisher, Mailchimp Automation)
  • Hotels & Restaurants (Web Design, Google Ads , Facebook Ads)
  • Sports (Web Design, Mailchimp Campaign)
Proudly part of