하이브로 mysql을 설정 한 후 하이브 메타 스토어 서비스 또는 하이브 - 쉘을 시작할 수 없습니다

나는이 질문이 이미 질문 된 것을 알고 있지만 그 대답은 어떤 식 으로든 도움이되지 못합니다.

하이브를 사용하여 mysql을 설정하는 데 더 많은 시간을 할애하고있다. 오류가 발생할 때마다 그리고 내가 잘못 읽은 곳을 이해하지 못한다. .....이 링크에서 볼 수 있듯이,

hive-site.xml 설정은 링크에서 볼 수 있습니다.

다음은 파일 구조입니다.

그래서이 질문을 만드는 이유는 .... 이것이 내가 해결하는 데 도움이되기를 바랍니다.

나는이 링크를 따라 갔다.

https://dzone.com/articles/how-configure-mysql-metastore

hive0.13.1의 hive-site.xml 경로

https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration

하이브 메타 스토어 연결을위한 Hive-Site.xml 파일의 구성 설정 방법

https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-RemoteMetastoreServer

해결법

==============================

1.hive-site.xml 파일에 문제가있어서이 구성으로이 문제를 해결할 수있었습니다.

hive-site.xml 파일에 문제가있어서이 구성으로이 문제를 해결할 수있었습니다.

  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost/metastore_db?createDatabaseIfNotExist=true</value>
    <description>metadata is stored in a MySQL server</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>MySQL JDBC driver class</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hiveuser</value>
    <description>user name for connecting to mysql server </description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hivepassword</value>
    <description>password for connecting to mysql server </description>
  </property>
  <property>
    <name>javax.jdo.PersistenceManagerFactoryClass</name>
    <value>org.datanucleus.api.jdo.JDOPersistenceManagerFactory</value>
    <description>class implementing the jdo persistence</description>
  </property>

  <property>
    <name>javax.jdo.option.DetachAllOnCommit</name>
    <value>true</value>
    <description>detaches all objects from session so that they can be used after transaction is committed</description>
  </property>

  <property>
    <name>javax.jdo.option.NonTransactionalRead</name>
    <value>true</value>
    <description>reads outside of transactions</description>
  </property>
  <property>
    <name>javax.jdo.option.Multithreaded</name>
    <value>true</value>
    <description>Set this to true if multiple threads access metastore through JDO concurrently.</description>
  </property>

  <property>
    <name>datanucleus.validateTables</name>
    <value>false</value>
    <description>validates existing schema against code. turn this on if you want to verify existing schema </description>
  </property>

  <property>
    <name>datanucleus.validateColumns</name>
    <value>false</value>
    <description>validates existing schema against code. turn this on if you want to verify existing schema </description>
  </property>

  <property>
    <name>datanucleus.validateConstraints</name>
    <value>false</value>
    <description>validates existing schema against code. turn this on if you want to verify existing schema </description>
  </property>

  <property>
    <name>datanucleus.storeManagerType</name>
    <value>rdbms</value>
    <description>metadata store type</description>
  </property>

  <property>
    <name>datanucleus.autoCreateSchema</name>
    <value>false</value>
  </property>

  <property>
    <name>datanucleus.autoStartMechanismMode</name>
    <value>checked</value>
    <description>throw exception if metadata tables are incorrect</description>
  </property>

  <property>
    <name>datanucleus.autoStartMechanism</name>
    <value>SchemaTable</value>
  </property>

  <property>
    <name>datanucleus.fixedDatastore</name>
    <value>true</value>
  </property>

  <property>
    <name>datanucleus.transactionIsolation</name>
    <value>read-committed</value>
    <description>Default transaction isolation level for identity generation. </description>
  </property>

  <property>
    <name>datanucleus.cache.level2</name>
    <value>false</value>
    <description>Use a level 2 cache. Turn this off if metadata is changed independently of Hive metastore server</description>
  </property>

  <property>
    <name>datanucleus.cache.level2.type</name>
    <value>SOFT</value>
    <description>SOFT=soft reference based cache, WEAK=weak reference based cache.</description>
  </property>

  <property>
    <name>datanucleus.identifierFactory</name>
    <value>datanucleus1</value>
    <description>Name of the identifier factory to use when generating table/column names etc. 'datanucleus1' is used for backward  compatibility with DataNucleus v1</description>
  </property>


  <property>
    <name>datanucleus.plugin.pluginRegistryBundleCheck</name>
    <value>LOG</value>
    <description>Defines what happens when plugin bundles are found and are duplicated [EXCEPTION|LOG|NONE]</description>
  </property>

  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
  </property>

  <property>
    <name>hive.metastore.execute.setugi</name>
    <value>false</value>
    <description>In unsecure mode, setting this property to true will cause the metastore to execute DFS operations using the client's reported user and group permissions. Note that this property must be set on both the client and server sides. Further  note that its best effort. If client sets its to true and server sets it to false, client setting will be ignored.</description>
  </property>

  <property>
    <name>hive.metastore.event.listeners</name>
    <value></value>
    <description>list of comma separated listeners for metastore events.</description>
  </property>

  <property>
    <name>hive.metastore.partition.inherit.table.properties</name>
    <value></value>
    <description>list of comma separated keys occurring in table properties which will get inherited to newly created partitions. *   implies all the keys will get inherited.</description>
  </property>

  <property>
    <name>hive.metadata.export.location</name>
    <value></value>
    <description>When used in conjunction with the org.apache.hadoop.hive.ql.parse.MetaDataExportListener pre event listener, it is the location to which the metadata will be exported. The default is an empty string, which results in the metadata being exported   to the current user's home directory on HDFS.</description>
  </property>

  <property>
    <name>hive.metadata.move.exported.metadata.to.trash</name>
    <value></value>
    <description>When used in conjunction with the org.apache.hadoop.hive.ql.parse.MetaDataExportListener pre event listener, this setting determines if the metadata that is exported will subsequently be moved to the user's trash directory alongside the   dropped table data. This ensures that the metadata will be cleaned up along with the dropped table data.</description>
  </property>

  <property>
    <name>hive.metastore.partition.name.whitelist.pattern</name>
    <value></value>
    <description>Partition names will be checked against this regex pattern and rejected if not matched.</description>
  </property>

  <property>
    <name>hive.metastore.disallow.incompatible.col.type.change</name>
    <value></value>
    <description>If true (default is false), ALTER TABLE operations which change the type of   a column (say STRING) to an incompatible type (say MAP&lt;STRING, STRING&gt;) are disallowed.    RCFile default SerDe (ColumnarSerDe) serializes the values in such a way that the  datatypes can be converted from string to any type. The map is also serialized as  a string, which can be read as a string as well. However, with any binary   serialization, this is not true. Blocking the ALTER TABLE prevents ClassCastExceptions  when subsequently trying to access old partitions.   Primitive types like INT, STRING, BIGINT, etc are compatible with each other and are   not blocked.  

  See HIVE-4409 for more details.
    </description>
  </property>

  <property>
    <name>hive.metastore.end.function.listeners</name>
    <value></value>
    <description>list of comma separated listeners for the end of metastore functions.</description>
  </property>

  <property>
    <name>hive.metastore.event.expiry.duration</name>
    <value>0</value>
    <description>Duration after which events expire from events table (in seconds)</description>
  </property>

  <property>
    <name>hive.metastore.event.clean.freq</name>
    <value>0</value>
    <description>Frequency at which timer task runs to purge expired events in metastore(in seconds).</description>
  </property>

  <property>
    <name>hive.metastore.connect.retries</name>
    <value>5</value>
    <description>Number of retries while opening a connection to metastore</description>
  </property>

  <property>
    <name>hive.metastore.failure.retries</name>
    <value>3</value>
    <description>Number of retries upon failure of Thrift metastore calls</description>
  </property>

  <property>
    <name>hive.metastore.client.connect.retry.delay</name>
    <value>1</value>
    <description>Number of seconds for the client to wait between consecutive connection attempts</description>
  </property>

  <property>
    <name>hive.metastore.client.socket.timeout</name>
    <value>20</value>
    <description>MetaStore Client socket timeout in seconds</description>
  </property>

  <property>
    <name>hive.metastore.rawstore.impl</name>
    <value>org.apache.hadoop.hive.metastore.ObjectStore</value>
    <description>Name of the class that implements org.apache.hadoop.hive.metastore.rawstore interface. This class is used to store   and retrieval of raw metadata objects such as table, database</description>
  </property>

  <property>
    <name>hive.metastore.batch.retrieve.max</name>
    <value>300</value>
    <description>Maximum number of objects (tables/partitions) can be retrieved from metastore in one batch. The higher the number, the less the number of round trips is needed to the Hive metastore server, but it may also cause higher memory requirement at the   client side.</description>
  </property>

  <property>
    <name>hive.metastore.batch.retrieve.table.partition.max</name>
    <value>1000</value>
    <description>Maximum number of table partitions that metastore internally retrieves in one batch.</description>
  </property>
  <property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9083</value>
    <description>Hive metastore Thrift server</description>
  </property>
  <property>
    <name>mapred.reduce.tasks</name>
    <value>-1</value>
    <description>The default number of reduce tasks per job.  Typically set to a prime close to the number of available hosts.  Ignored when mapred.job.tracker is "local". Hadoop set this to 1 by default, whereas Hive uses -1 as its default value. By setting this property to -1, Hive will automatically figure out what should be the number of reducers.
    </description>
  </property>
  <property>
    <name>hive.cli.print.header</name>
    <value>false</value>
    <description>Whether to print the names of the columns in query output.</description>
  </property>

  <property>
    <name>hive.cli.print.current.db</name>
    <value>false</value>
    <description>Whether to include the current database in the Hive prompt.</description>
  </property>

  <property>
    <name>hive.cli.prompt</name>
    <value>hive</value>
    <description>Command line prompt configuration value. Other hiveconf can be used in this configuration value. Variable substitution will only be invoked at the Hive
    CLI startup.</description>
  </property>

  <property>
    <name>hive.test.mode</name>
    <value>false</value>
    <description>Whether Hive is running in test mode. If yes, it turns on sampling and prefixes the output tablename.</description>
  </property>

  <property>
    <name>hive.test.mode.prefix</name>
    <value>test_</value>
    <description>if Hive is running in test mode, prefixes the output table by this string</description>
  </property>


  <property>
    <name>hive.test.mode.samplefreq</name>
    <value>32</value>
    <description>if Hive is running in test mode and table is not bucketed, sampling frequency</description>
  </property>

  <property>
    <name>hive.test.mode.nosamplelist</name>
    <value></value>
    <description>if Hive is running in test mode, don't sample the above comma separated list of tables</description>
  </property>

  <property>
    <name>hive.metastore.uris</name>
    <value></value>
    <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
  </property>
</configuration>

==============================

2.

            copy mysql sql driver in $HIVE_HOME/lib folder


            change hive-site.xml file properties under $HIVE_HOME/conf
            #########################################################

            <property>
            <name>javax.jdo.option.ConnectionDriverName</name>
            <value>com.mysql.jdbc.Driver</value>
            <description>Driver class name for a JDBC metastore</description>
            </property>



            <property>
                <name>javax.jdo.option.ConnectionURL</name>
                <value>jdbc:mysql://localhost:3306/hive_metastore_db?createDatabaseIfNotExist=true</value>
              </property>



            <property>
              <name>datanucleus.autoCreateSchema</name>
              <value>true</value>
            </property>

            <property>
              <name>datanucleus.fixedDatastore</name>
              <value>true</value>
            </property>

            <property>
             <name>datanucleus.autoCreateTables</name>
             <value>True</value>
             </property>

             <property>
                <name>javax.jdo.option.ConnectionPassword</name>
                <value>root</value>
                <description>password to use against metastore database</description>
              </property>


             <property>
                <name>javax.jdo.option.ConnectionUserName</name>
                <value>root</value>
                <description>Username to use against metastore database</description>
              </property>
          #############################

from https://stackoverflow.com/questions/36545207/unable-to-start-hive-metastore-service-or-hive-shell-after-configuring-mysql-wit by cc-by-sa and MIT license

'HADOOP' 카테고리의 다른 글

[HADOOP] 감속기가 <Text, IntWritable> 및 <Text, IntWritable>을 수신 할 매퍼를 내보내도록 설정하는 방법은 무엇입니까? (0)	2019.07.02
[HADOOP] HADOOP :: java.lang.ClassNotFoundException : WordCount (0)	2019.07.02
[HADOOP] 이미지의 하둡 및 3D 렌더링 (0)	2019.07.02
[HADOOP] hadoop 클러스터가 안전 모드로 실행되는 이유는 무엇입니까? (0)	2019.07.02
[HADOOP] spb sbt가 오류 libraryDependencies를 컴파일합니다. (0)	2019.07.02

복붙노트

[HADOOP] 하이브로 mysql을 설정 한 후 하이브 메타 스토어 서비스 또는 하이브 - 쉘을 시작할 수 없습니다

하이브로 mysql을 설정 한 후 하이브 메타 스토어 서비스 또는 하이브 - 쉘을 시작할 수 없습니다

해결법

1.hive-site.xml 파일에 문제가있어서이 구성으로이 문제를 해결할 수있었습니다.

2.

'HADOOP' 카테고리의 다른 글

티스토리툴바