본문 바로가기

Hadoop Ecosystem

(23)
namenode.SafeModeException [hadoop@h001 hadoop]$ ./bin/hadoop fs -put README.txt /user/hadoop/xxx/xx/RRRR.txtput: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file/user/hadoop/xxx/xx/RRRR.txt. Name node is in safe mode.[hadoop@h001 hadoop]$ ls1901 derby.log lib1902 docs libexec1988.csv FindMax.jar LICENSE.txt2007.csv hadoop-ant-1.0.4.jar logsair001.jar hadoop-client-1.0.4.jar metastore_dbair002...
ucloud server disk use summay http://deepblue28.tistory.com/34http://www.anycoding.com/bbs/board.php?bo_table=acProgram_Etc&wr_id=671http://www.lug.or.kr/docs/LINUX/others/01-05-4.htm [root@h002 ~]# fdisk -l Disk /dev/xvdb: 85.9 GB, 85899345920 bytes255 heads, 63 sectors/track, 10443 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 b..
snappy snappy tar
Job Configuration: JobId - job_201307111807_0008 Job Configuration: JobId - job_201307111807_0008 namevaluefs.s3n.implorg.apache.hadoop.fs.s3native.NativeS3FileSystemmapred.task.cache.levels2hadoop.tmp.dir/home/hadoop/hadoop/hadoop-data/hadoop.native.libtruemap.sort.classorg.apache.hadoop.util.QuickSortdfs.namenode.decommission.nodes.per.interval5dfs.https.need.client.authfalseipc.client.idlethreshold4000dfs.datanode.data.dir.perm755mapred.sys..
Map Output Key 변경변화 확인 Year, Month 에 UniqueCarrier 및 로직 추가 Year, Month 로만 Map Output Key 로 잡은 경우 13/07/11 18:30:42 INFO mapred.JobClient: Job complete: job_201307111807_000113/07/11 18:30:43 INFO mapred.JobClient: Counters: 3013/07/11 18:30:43 INFO mapred.JobClient: Job Counters 13/07/11 18:30:43 INFO mapred.JobClient: Launched reduce tasks=113/07/11 18:30:43 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=253143413/07/11 18:30:43 INFO mapred.JobClient: Total ..
cloud : kt ucloud vm - openAPI test 01 kt ucloud vm - openAPI test -- php version[ https://ucloudbiz.olleh.com/ ] ************************************************************************************* ucloudbiz r개발자센터 Open API http://developer.ucloudbiz.olleh.com*************************************************************************************api key : 45xxxxxxxxxxxxxxxxx your api key xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxyQsecr..
Partitioning - MapReduce 역시 diagram 을 보면 이해가 한결 수월해진다. 개인적인 느낌이지만 IT 기술분야를 학습해 나가면서 느끼는 것은 IT 는 똑같은 분야를 놓고 책을 3, 4권 정도 봐야 제대로 된 설명을 찾아내거나 상호간의 비교 및 조합을 통해서 너무 추상화된 개념을 또는 허공에 뜬 개념을 이해할 수 있는 땅바닥으로 끌어 내릴 수 있는 듯 하다. The partitioning pattern moves the records into categories (i.e., shards, partitions, orbins) but it doesn’t really care about the order of records. IntentThe intent is to take similar records in a data set and..
Shuffling - MapReduce Shuffling 의 목적과 왜 하는지에 대한 설명이다 . The total order sorting and shuffling patterns are opposites in terms of effect, but thelatter is also concerned with the order of data in records. IntentYou have a set of records that you want to completely randomize. MotivationThis whole chapter has been about applying some sort of order to your data set exceptfor this pattern which is instead about completely ..