node + typescript + batch

간단한 웹 크롤링 성격의 배치성격의 프로젝트를 만들일이 있어 Spring Batch로 만들기는 좀 오버인듯해서 간단히 구축해봤다
node등 기본 설치가 되있다는 전제한다.

npm init을 통해 package.json 파일을 생성한다.

$ npm init

typescript가 설치 되있지 않을경우 타입스크립트를 설치한다.

npm install -g typescript

typescript를 초기화 한다.

 tsc --init

batch처리를 할수있도록 cron 할수있는 npm 모듈을 설치하고 관련 type 을 install 한다.

npm install --save node-cron
npm install --save @types/node-cron

package.json

{
  "name": "crawler_batch",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node ./main.js",
    "ts-start": "tsc && node dist"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@types/node-cron": "^2.0.4",
    "axios": "^0.21.1",
    "cheerio": "^1.0.0-rc.10",
    "node-cron": "^3.0.0"
  }
}

- scripts ts-start
- cheerio 는 추후 웹크롤링 적용시 사용 에정 

tsconfig.json

{
  "include": [
    "src/**/*"
  ],
  "compilerOptions": {
    /* Visit https://aka.ms/tsconfig.json to read more about this file */
    /* Basic Options */
    // "incremental": true,                         /* Enable incremental compilation */
    "target": "es6",                                /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017', 'ES2018', 'ES2019', 'ES2020', 'ES2021', or 'ESNEXT'. */
    "module": "commonjs",                           /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', 'es2020', or 'ESNext'. */
    // "lib": [],                                   /* Specify library files to be included in the compilation. */
    // "allowJs": true,                             /* Allow javascript files to be compiled. */
    // "checkJs": true,                             /* Report errors in .js files. */
    // "jsx": "preserve",                           /* Specify JSX code generation: 'preserve', 'react-native', 'react', 'react-jsx' or 'react-jsxdev'. */
    // "declaration": true,                         /* Generates corresponding '.d.ts' file. */
    // "declarationMap": true,                      /* Generates a sourcemap for each corresponding '.d.ts' file. */
    "sourceMap": true,                           /* Generates corresponding '.map' file. */
    // "outFile": "./",                             /* Concatenate and emit output to single file. */
    "outDir": "dist",                              /* Redirect output structure to the directory. */
    // "rootDir": "./",                             /* Specify the root directory of input files. Use to control the output directory structure with --outDir. */
    // "composite": true,                           /* Enable project compilation */
    // "tsBuildInfoFile": "./",                     /* Specify file to store incremental compilation information */
    // "removeComments": true,                      /* Do not emit comments to output. */
    // "noEmit": true,                              /* Do not emit outputs. */
    // "importHelpers": true,                       /* Import emit helpers from 'tslib'. */
    // "downlevelIteration": true,                  /* Provide full support for iterables in 'for-of', spread, and destructuring when targeting 'ES5' or 'ES3'. */
    // "isolatedModules": true,                     /* Transpile each file as a separate module (similar to 'ts.transpileModule'). */

    /* Strict Type-Checking Options */
    "strict": true,                                 /* Enable all strict type-checking options. */
    // "noImplicitAny": true,                       /* Raise error on expressions and declarations with an implied 'any' type. */
    // "strictNullChecks": true,                    /* Enable strict null checks. */
    // "strictFunctionTypes": true,                 /* Enable strict checking of function types. */
    // "strictBindCallApply": true,                 /* Enable strict 'bind', 'call', and 'apply' methods on functions. */
    // "strictPropertyInitialization": true,        /* Enable strict checking of property initialization in classes. */
    // "noImplicitThis": true,                      /* Raise error on 'this' expressions with an implied 'any' type. */
    // "alwaysStrict": true,                        /* Parse in strict mode and emit "use strict" for each source file. */

    /* Additional Checks */
    // "noUnusedLocals": true,                      /* Report errors on unused locals. */
    // "noUnusedParameters": true,                  /* Report errors on unused parameters. */
    // "noImplicitReturns": true,                   /* Report error when not all code paths in function return a value. */
    // "noFallthroughCasesInSwitch": true,          /* Report errors for fallthrough cases in switch statement. */
    // "noUncheckedIndexedAccess": true,            /* Include 'undefined' in index signature results */
    // "noImplicitOverride": true,                  /* Ensure overriding members in derived classes are marked with an 'override' modifier. */
    // "noPropertyAccessFromIndexSignature": true,  /* Require undeclared properties from index signatures to use element accesses. */

    /* Module Resolution Options */
    "moduleResolution": "node",                  /* Specify module resolution strategy: 'node' (Node.js) or 'classic' (TypeScript pre-1.6). */
    // "baseUrl": "./",                             /* Base directory to resolve non-absolute module names. */
    // "paths": {},                                 /* A series of entries which re-map imports to lookup locations relative to the 'baseUrl'. */
    // "rootDirs": [],                              /* List of root folders whose combined content represents the structure of the project at runtime. */
    // "typeRoots": [],                             /* List of folders to include type definitions from. */
    // "types": [],                                 /* Type declaration files to be included in compilation. */
    // "allowSyntheticDefaultImports": true,        /* Allow default imports from modules with no default export. This does not affect code emit, just typechecking. */
    "esModuleInterop": true,                        /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */
    // "preserveSymlinks": true,                    /* Do not resolve the real path of symlinks. */
    // "allowUmdGlobalAccess": true,                /* Allow accessing UMD globals from modules. */

    /* Source Map Options */
    // "sourceRoot": "",                            /* Specify the location where debugger should locate TypeScript files instead of source locations. */
    // "mapRoot": "",                               /* Specify the location where debugger should locate map files instead of generated locations. */
    // "inlineSourceMap": true,                     /* Emit a single file with source maps instead of having a separate file. */
    // "inlineSources": true,                       /* Emit the source alongside the sourcemaps within a single file; requires '--inlineSourceMap' or '--sourceMap' to be set. */

    /* Experimental Options */
    // "experimentalDecorators": true,              /* Enables experimental support for ES7 decorators. */
    // "emitDecoratorMetadata": true,               /* Enables experimental support for emitting type metadata for decorators. */

    /* Advanced Options */
    //"skipLibCheck": true,                           /* Skip type checking of declaration files. */
    //"forceConsistentCasingInFileNames": true        /* Disallow inconsistently-cased references to the same file. */
  }
}

타입스크립트 실행시 아래와 같은 오류가 발생하면 2021-07-11일 최신버전(Version 4.3.5)
에서는 include 옵션이 루트 레벨로 올라갔다
위처럼  루트레벨의 별도로 include 선언해서 설정 하면 된다.
tsconfig.json:70:5 - error TS5023: Unknown compiler option 'include'.

/src/index.ts

import cron from 'node-cron';
import {getWebCrawlerInfo} from './crawler.js'

async function crawlerSync() {
    const powerBall = await getWebCrawlerInfo()
    console.log("powerBall", powerBall)
}

cron.schedule("*/1 * * * *", async () => {
    await crawlerSync()
});

crawlerSync()

- npm modul node-cron을 이용해서 스케줄링을 한다.

/src/crawler.ts

import axios from 'axios'
import dateFormatUtil from './util/dateUtil'
// const axios = require("axios");
// const cheerio = require("cheerio");

async function getWebCrawlerInfo() {
    
    // const $ = cheerio.load(html.data); 추후 웹 크롤링 추가하자 지금은 api 샘플로 대신한다.
    const today = new Date();
    const toCurDateString:String = dateFormatUtil.toFormat(today,'-')
    console.log("TODAY = " + toCurDateString)
    const res = await axios.get(`https://www.test.co.kr/?date=${toCurDateString}&page=1`)

    if (!res.data.content) return;
    return  res.data.content[0]
}
export  { getWebCrawlerInfo }

/src/util/dateUtil.ts 추가한다.

const dateFormatUtil = {
    toFormat:  (date:Date, delimeter:String):String => {
        const year:number = date.getFullYear();
        const month:String = ('0' + (date.getMonth() + 1)).slice(-2);
        const day:String = ('0' + date.getDate()).slice(-2);
        return `${year}${delimeter}${month}${delimeter}${day}`
    }
}

export default dateFormatUtil

실행해본다.

npm run ts-start

추후 DB 연동 하자
git은 1차 완료후 올릴 예정