[HADOOP] Hadoop의지도 축소 프레임에서 "Selenium Webdriver"를 실행하고지도 단계에서 고정합니다.

Hadoop의지도 축소 프레임에서 "Selenium Webdriver"를 실행하고지도 단계에서 고정합니다.

며칠 전, 나는 셀레늄 webdriver (타사 패키지) hadoop의 mapreduce 프레임에서 실행됩니다. 그리고 나는 문제를 만났다. 새로운 FirefoxDriver ();에서 맵 단계가 멈 춥니 다. FirefoxDriver 클래스는 selenium-server-standalone-2.38.0.jar이라는 써드 파티 항아리에있었습니다. 누군가가 경험이나 관심을 가지고 있다면, 나는 당신의 도움이 필요합니다!

    public class MapRunnerNewFirefox extends Configured implements Tool, MapRunnable{
        public static final Logger LOG = LoggerFactory.getLogger(MapRunnerNewFirefox.class);
        @Override
        public void configure(JobConf conf) {
        }
        @Override
        public void run(RecordReader recordReader,
            OutputCollector output, Reporter reporter) throws IOException {
            LongWritable key = new LongWritable(-1);// shouldn't be null ,otherwise the recordReader will report nullpointer err;
            Text val = new Text("begin text");      // same as up line;
            int i = 0;
            reporter.progress();
            while(recordReader.next(key, val)){         
                if(LOG.isInfoEnabled()){
                    LOG.info("key: "+key.toString()+" val: "+val.toString());
                }
                String temp = "ao";
                NewFirefox ff = new NewFirefox("/home/cc/firefox/firefox/firefox");
                output.collect(new Text("get-"+key.toString()), new Text(temp));
            }
        }
        @Override
        public int run(String[] args) throws Exception {
            if(LOG.isInfoEnabled()) {
                LOG.info("set maprunner conf");
            }
            Path urlDir = new Path(args[0]);
            Path resultDir = new Path(args[1] + System.currentTimeMillis());
            JobConf job = new JobConf(getConf());
            job.setNumMapTasks(1);
            job.setJobName("hello maprunners");
            job.setInputFormat(TextInputFormat.class);
            FileInputFormat.addInputPath(job, urlDir);      
            job.setMapRunnerClass(MapRunnerNewFirefox.class);
            FileOutputFormat.setOutputPath(job, resultDir);
            job.setOutputFormat(TextOutputFormat.class);
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(Text.class);
            JobClient.runJob(job);      
            return 0;
        }
        public static void main(String[] args) throws Exception{    
            Configuration conf = new Configuration();       
            int res = ToolRunner.run(conf, new MapRunnerNewFirefox(), args);
            System.exit(res);   
        }
    }
    public class NewFirefox {
        private WebDriver driver;
        private static final Logger LOG = LoggerFactory.getLogger(NewFirefox.class);
        public NewFirefox(String firefoxPath){
            if(LOG.isInfoEnabled()){
                LOG.info("firefox ****0");
            }
            System.setProperty("webdriver.firefox.bin", firefoxPath);
            if(LOG.isInfoEnabled()){
                LOG.info("firefox ****1");
            }
            ProfilesIni profile = new ProfilesIni();
            FirefoxProfile ffprofile = profile.getProfile("default");
            if(LOG.isInfoEnabled()){
                LOG.info("firefox ****2");
            }
            this.driver = new FirefoxDriver(ffprofile);
            if(LOG.isInfoEnabled()){
                LOG.info("firefox ****3");
            }
            this.driver.quit();
            if(LOG.isInfoEnabled()){
                LOG.info("firefox quit");
            }
        }
    }

해결법

from https://stackoverflow.com/questions/20581214/run-the-selenium-webdriver-in-the-mapreduce-frame-of-hadoop-freezed-in-the-ma by cc-by-sa and MIT license