Datax離線數(shù)據(jù)抽取的實(shí)現(xiàn)方法是什么

本篇內(nèi)容主要講解“Datax離線數(shù)據(jù)抽取的實(shí)現(xiàn)方法是什么”，感興趣的朋友不妨來(lái)看看。本文介紹的方法操作簡(jiǎn)單快捷，實(shí)用性強(qiáng)。下面就讓小編來(lái)帶大家學(xué)習(xí)“Datax離線數(shù)據(jù)抽取的實(shí)現(xiàn)方法是什么”吧!

1.下載安裝Datax軟件(必須安裝jdk 1.8版本以上) 

[root@localhost ~]# tar xvf jdk-8u65-linux-x64.tar.gz 

[root@localhost ~]# mv jdk1.8.0_151  /usr/local/jdk1.8.0_151

[root@localhost ~]# vim /etc/profile

export PATH=$PATH:/usr/local/jdk1.8.0_151/bin

[root@localhost ~]# vim /etc/ld.so.conf.d/mysql-x86_64.conf

/usr/local/jdk1.8.0_151/lib

[root@localhost ~]# tar xvf  datax.tar.gz

[root@localhost ~]# cd datax/job/

2.編輯配置文件(首先用kettle將表結(jié)構(gòu)同步過(guò)去)

[root@localhost job]# vim job1.json

{
    "job": {
        "setting": {
            "speed": {
                "channel": 5
            }
        },
        "content": [
            {
                "reader": {
                    "name": "oraclereader",
                    "parameter": {
                        "username": "upcenter",
                        "password": "upcenter",
                        "column": ["*"],
                        "connection": [
                           {
                              "table": ["STOCK_CONC"],
                              "jdbcUrl": ["jdbc:oracle:thin:@192.168.7.7:1521:upqc"]
                           }
                        ]
                    }
                },
                "writer": {
                    "name": "mysqlwriter",
                    "parameter": {
                        "writeMode": "update",
                        "username": "wangying",
                        "password": "wangying",
                        "column": ["*"],
                        "connection": [
                            {
                               "jdbcUrl": "jdbc:mysql://172.16.8.93:3306/db_stktag",
                               "table": ["t3"]
                            }
                        ]
                      }
                    }
                }
        ]
    }
}

[root@localhost job]#

3.數(shù)據(jù)抽取

[root@localhost job]# python2 /root/datax/bin/datax.py job1.json

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !

Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

2019-01-29 14:23:53.177 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl

2019-01-29 14:23:53.186 [main] INFO  Engine - the machine info  => 

osInfo: Oracle Corporation 1.8 25.151-b12

jvmInfo: Linux amd64 3.10.0-123.el7.x86_64

cpu num: 8

totalPhysicalMemory: -0.00G

freePhysicalMemory: -0.00G

maxFileDescriptorCount: -1

currentOpenFileDescriptorCount: -1

。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。

。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。

2019-01-29 14:24:04.012 [job-0] INFO  JobContainer - 

任務(wù)啟動(dòng)時(shí)刻                    : 2019-01-29 14:23:53

任務(wù)結(jié)束時(shí)刻                    : 2019-01-29 14:24:04

任務(wù)總計(jì)耗時(shí)                    :                 10s

任務(wù)平均流量                    :          176.48KB/s

記錄寫入速度                    :           4047rec/s

讀出記錄總數(shù)                    :               40475

讀寫失敗總數(shù)                    :                   0

[root@localhost job]#

4.驗(yàn)證數(shù)據(jù)

mysql> select count(1) from t3;

+----------+

| count(1) |

+----------+

|    40475 |

+----------+

1 row in set (0.03 sec)

mysql> 

到此，相信大家對(duì)“Datax離線數(shù)據(jù)抽取的實(shí)現(xiàn)方法是什么”有了更深的了解，不妨來(lái)實(shí)際操作一番吧！這里是億速云網(wǎng)站，更多相關(guān)內(nèi)容可以進(jìn)入相關(guān)頻道進(jìn)行查詢，關(guān)注我們，繼續(xù)學(xué)習(xí)！